release PCi-e ?

[unf0rg0tt3n]

Hi,

I am using proxmox for several years now, and passing through a PCi-e device always works.
But how do I release a device which is used by a guest (without rebooting)?
I have a HP H240 in Jbod mode, but I can't re-use it in another guest when the first guest is stopped.

Anyone a clue?
Thanks!

Dennis

 

[leesteken]

If the device resets properly, it should be possible (and easy) to use it in second VM after stopping the first.
Does your device work when you stop and then start the first VM? This should behave the same as stopping the first VM and the starting the second VM.
Does the device work in the second VM if you reboot the Proxmox host and make sure not to start the first VM?

 

[unf0rg0tt3n]

If the device resets properly, it should be possible (and easy) to use it in second VM after stopping the first.
Does your device work when you stop and then start the first VM? This should behave the same as stopping the first VM and the starting the second VM.
Does the device work in the second VM if you reboot the Proxmox host and make sure not to start the first VM?

Thanks for your reply!
It does work when I stop and start the VM, It doesn't work when I stop VM 1 and start VM 2.

 

[leesteken]

Thanks for your reply!
It does work when I stop and start the VM, It doesn't work when I stop VM 1 and start VM 2.

But does it work when you never start VM1 but start VM2? In that case the problem is with VM2 or inside it. Can you show us the VM configurations (from /etc/pve/qemu-server/) and tell us about the differences between VM1 and VM2?

 

[unf0rg0tt3n]

But does it work when you never start VM1 but start VM2? In that case the problem is with VM2 or inside it. Can you show us the VM configurations (from /etc/pve/qemu-server/) and tell us about the differences between VM1 and VM2?

It does work when I never start VM 1

The config from top to down VM1 - VM2

agent: 1
balloon: 16384
bootdisk: sata0
cores: 8
cpu: qemu64,flags=+pdpe1gb
hostpci0: 09:00,pcie=1
hostpci1: 06:00,pcie=1
hotplug: disk,network,usb
machine: q35
memory: 61440
name: NAS
net0: virtio=5E:A7:6B:DA:B3:6D,bridge=vmbr1
numa: 1
onboot: 1
ostype: l26
parent: working
sata0: VM:vm-102-disk-0,cache=writeback,size=32G,ssd=1
sata1: local-lvm:vm-102-disk-0,cache=writeback,size=32G,ssd=1
smbios1: uuid=48ddfaaa-7b97-4d49-918f-d1863952c091
sockets: 1
startup: order=2,up=180
vmgenid: ea3ded1f-021a-4981-a1a4-6f5dd5b24496




agent: 1
boot: order=sata0;ide2;net0
cores: 4
hostpci0: 09:00,pcie=1
hostpci1: 06:00,pcie=1
hotplug: disk,network,usb
ide2: local:iso/TrueNAS-SCALE-21.06-BETA.1.iso,media=cdrom
machine: q35
memory: 16384
name: NAS2
net0: virtio=62:02:1E:34:BE:BE,bridge=vmbr1,firewall=1
numa: 1
ostype: l26
sata0: VM:vm-104-disk-0,size=32G
smbios1: uuid=a248cbf4-9bb2-4de3-a367-6950b5e1ed7d
sockets: 1
vmgenid: 53a2ac92-bdff-4219-9a85-52e7b062e72e

 

[leesteken]

It sounds like the OS or driver from VM1 releases the device in a state from which it can use it after a reset, but the OS or driver from VM2 cannot. Would it be possible to use the same driver version in both VMs? Maybe dumping the ROM from the device and using the romfile option can help? Or maybe using hook scripts, you can pass the device back to the host and maybe its driver can bring it back to a reusable state? I have done such things for a GPU but I have no experience with a drive controller like yours.

 

[unf0rg0tt3n]

It sounds like the OS or driver from VM1 releases the device in a state from which it can use it after a reset, but the OS or driver from VM2 cannot. Would it be possible to use the same driver version in both VMs? Maybe dumping the ROM from the device and using the romfile option can help? Or maybe using hook scripts, you can pass the device back to the host and maybe its driver can bring it back to a reusable state? I have done such things for a GPU but I have no experience with a drive controller like yours.

It's quite possible it could be an OS thing. The driver in use is VFIO-PCI so the virtual driver is in use. I do think truenas needs the storage pool to be disconnected before any attempt to mount after a shutdown?

 

[leesteken]

I don't know about truenas, but I meant the drivers inside the VM. Can you make those the same? Or how are they different (inside each VM)?