this is only possible for 'special' gpus and if the drivers are available (i am not sure nvidia gives out those drivers for generic linux/kvm)
for amd there is at least one card that should work, though there is a custom kernel module that needs to be compiled
https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
also some intel igpu have that feature also (called gvt-g)
all 'normal' gpus can only be passed-through to one vm at a time (i believe this is also true for xenserver. for vmware i do not know)