I didn't find an exact solution for this in the forum and it seems some others have noticed this issues as well but the solution isn't clearly mentioned.
My guest OS is Windows 10 Pro (Build 10586) and I am using ASUS STRIX GTX 970 with pass-through.
When I add below to the config only my GPU...
