When trying to use GPU mode in KeyShot make sure that your drivers are updated.

We recommend drivers version 531.79 and later. We can don't recommend drivers of version 531.29 and below as there are known issues in KeyShot and those drivers.
You can use NVidia's search tool for the most recent drivers to your GPU.

Click here for link to search tool.