After (literally) months of trying to get a GPU to work with my VM's I have finally managed to do it.
I now have two VMs that each run their own GPU and can use them as expected. My main purpose for this was for remote gaming as my partner has a rather slow 2012 PC that can barely keep up with...