[SOLVED] Multi GPU Passthrough issues

[proxima]

Hello,

I am trying to pass through (via omvf) 2 video cards to two different VM's however I do not get video when passing through the card that gets bound to Proxmox (the one that ouputs the boot text). If I swap the cards on the motherboard slots I can pass through which ever one does not startup. When I attempt to pass-through the boot card the monitor output switches from the Proxmox login prompt (text) to "no signal" and the assigned VM refuses to start. In the Proxmox web interface the VM is labelled as started and the qm start command executes without issue however the cpu activity for the VM is near-zero and the machine does not actually boot (checked with ping and video).

Both cards are tested working with uefi bioses and their respective models are:
nvidia quadro 600
amd radeon 6570

The system is a Lenovo Thinkstation S20 and the CPU is an x5675 (Nehalem).

Any ideas?

Here is my Proxmox version:

root@pve:~# pveversion -v
proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.3-4 (running version: 4.3-4/ada7f750)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-91
pve-firmware: 1.1-10
libpve-common-perl: 4.0-75
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-66
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.7.0-2
pve-container: 1.0-78
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80

Here is the VM config:

balloon: 2
bios: ovmf
bootdisk: scsi0
cores: 2
hostpci0: 01:00,pcie=1,x-vga=on
machine: q35
memory: 2048
name: Media
net0: e1000=36:34:33:64:32:63,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local:102/vm-102-disk-1.qcow2,iothread=1,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=6ab803af-bfd0-4f29-9af0-be766d18b614
sockets: 1
usb0: host=046d:c526
usb1: host=050d:016a
vga: none

Boot/Grub parameters:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifb:off nofb nomodeset video=vesafb:off"
GRUB_CMDLINE_LINUX=""

/etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Blacklist config:

cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

 

[efeu]

Passthrough of primary VGA is not supported, maybe this will work in future like they implemented passing through Intel igp.

 

[proxima]

Thanks for letting me know.

 

[ziddey]

I've done a bit of testing wrt primary gpu passthrough and proxmox and have a working setup (radeon 7850).

I'm not familiar with the nvidia cards, although from what I've read, you shouldn't have to jump through hoops with a quadro.

The 6570 most likely does not have a UEFI rom. It may be possible to add UEFI support (I did with my 7850), but there's no sense worrying about that at the moment-- stick with seabios (you can eventually use ovmf-with-csm instead of ovmf-pure-efi as well).

AFAIK, there are no performance benefits to using q35 over the old 440fx. Most use cases seem to be for running osx.

For my 7850, even though I have flashed a uefi capable bios, I still need to pass the romfile in order for uefi to work (I mucked with QemuServer.pm to allow this). However, I've found that when using seabios, it happily uses the rom directly.

So, if you wanted to test passing through the 6570 as the primary card, let's try seabios without q35. You can remove the "machine: q35" line, and change "hostpci0: 01:00,pcie=1,x-vga=on" to "hostpci0: 01:00.0,x-vga=on". Note that without q35, there is not pcie. As well, add the .0 to remove the audio device (if applicable for the 6570..) for now. Later on, if you wanted to test support, use 01:00.0;01:00.1 instead of simply 01:00. I've found that the latter does not boot properly.

Since you're able to start the vm (although it doesn't display anything), I'm assuming you've bound the gpu to vfio-pci in /etc/modprobe.d/vfio.conf.

In my setup, proxmox does not utilize any framebuffer, so the extra config you have in your /etc/default/grub may not be necessary either:
video=efifbff nofb nomodeset video=vesafbff
When testing on ubuntu, I've found that nomodeset video=vga is enough to prevent 1. utilizing a video driver and 2. utilizing a framebuffer. However, my host is not uefi compatible.

I'm a little hazy on the seabios approach since I mainly focused on ovmf, but I believe you don't want the disable_vga=1 parameter in vfio.conf. This causes x-vga=on to not work in kvm (proxmox omits this line when booting ovmf, but not when using seabios).

Trying just the primary 6570 as a passthrough device, this should work (seabios/440fx).

Once this baseline is established (hopefully it works), if you really wanted to go ovmf, you could either look into modifying your vbios or change out the ovmf.fd that proxmox uses for one that has csm.

 

[Xenophon Costeas]

Ziddey, just a quick note to say: "thank you"!

I've been trying to passthrough a GT710 and a GTX660 on an ASUS H81 machine using Seabios (required by a legacy Linux guest), and after a week of troubleshooting, I stumbled on to your -generic sounding- post.

Thanks.

For the record, the critical addition to /etc/kernel/cmdline (6.2 on systemd) was "nomodeset video=vga".

Hope this helps someone!

Take care.

 

[rifaterdemsahin]

desperately trying this on windows would it work?