[SOLVED] PCIe passthrough works with one vm, fails on another

[oz1cw7yymn]

Hi,

It's probably something obvious but I'm stuck. I have a node with a pcie device available for passthrough (a sas adapter). The device has been attached to one vm, and I wanted to attach it to another vm instead. VM602 is switched off, the device is removed from the hardware configuration and added to the other VM (VM702, also switched off).

When I start VM702 the start job fails with the following

/dev/rbd2
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-702-disk-0: rbd: sysfs write failed
TASK ERROR: start failed: command '/usr/bin/kvm -id 702 <SNIP>'

If I remove the pcie device, the VM starts without an issue, readding it fails.

Moving the device back to 602 also works fine. Both VMs are running on the same node, have same BIOS, Display and Machine settings.

What am I missing?

 

[dcsapak]

can you post the vm configs (failing and non-failing) ?

 

[oz1cw7yymn]

I can, but they are on a disconnected network so a bit difficult. This are the diff rows:

602	702
balloon: 2048	balloon: 16384
hostpci0: 37:00,pcie=1	NA
ide2: cephfs an installer cd	ide2: cephfs another installer cd
memory: 8192	memory: 24576
name: name1	name: name2
net0: virtio=MAC,bridge=vmbr0,firewall=1 net1: virtio=MAC,bridge=vmbr1,firewall=1	net0: virtio=MAC,bridge=vmbr0,firewall=1
scsi0: rbd-ssd:vm-602-disk-0,cache=writeback,discard=on,size=50G	scsi0: rbd-ssd:9000-disk-0/vm-702-disk-0,cache=writeback,discard=on,size=50G
smbios1: uuid=UUID vmgenid: VMID	smbios1: uuid=UUID vmgenid: VMID

As I mentioned, 702 starts if I remove the pcie and doesn't if I add it. Maybe I should try adding it manually to the vm config to be sure it's added the same way. I'm now also moving the disk to create a full copy instead of the differencing disk it is now.

I'll try those things and report back if it helped.

 

[dcsapak]

is that the complete task log in your first post? if not, could you share it?

what may be a problem is the additional memory.. if you pass through a pci device, qemu has to allocate the whole memory beforehand, which can take a while and it maybe runs into a timeout
you can try to enable hugepages (see 'man qm.conf')

we generally try to increase the timeout with additional memory, but that only starts with 30GB, but maybe the 25G you configured comes close to the default 30s timeout...

in that case the unmapping of the rbd image is also not good, but a red herring since it happens after qemu run into a timeout

 

[oz1cw7yymn]

That's the complete task log in the Task viewer for the start job. The status tab show the duration to be 31s so it being a time-out seems reasonable.

The rbd image merging didn't help, as you wrote it was probably a red herring.

I actually had it at 16GB before and increased it to 24GB, but it also didn't work at 16GB. Changing it to 8GB actually works! (but not good for the work load of course).

man page for qm.conf shows options 1024 | 2 | any, so I searched for and found this post https://forum.proxmox.com/threads/vm-start-timeout-pci-pass-through-related.28876/post-145140

I put hugepages: 2 in but got hugepage allocation failed so I think I may need to enable it in GRUB first. I'll have to try that on Monday.

Thanks a lot!

 

[oz1cw7yymn]

Before I start messing too much with the boot parameters, are there any documentation about hugepages best practices and settings?

 

[oz1cw7yymn]

Got it working! Added "hugepagez=1G default_hugepagesz=2M" to grub and "hugepages: 2" to the vm config file and now it starts!

As I'm writing this, I see my typo (hugepagez) but it still worked..