[SOLVED] PCI passthrough causes Proxmox host to crash

Z

Zx4ACQQ7

Member
Feb 14, 2021
16
1
23
54
Hello,
I was trying to install an OPNsense VM on a host with Proxmox.
I installed it fine and then went to configure it and everything was still working.

I noticed that after I added a PCI device (NIC) as passthrough the host started crashing as soon as the VM was booting which means that if you enable it to start up on boot this would restart the host every ~2 minutes.
 
After a long investigation and trying different things from the internet I finally found the solution.
This happened because I had assigned all of the memory to the VM.
This works fine normally but for some reason the passthrough does not like that and requires some memory for itself.
Once I reduced the memory (16GB -> 12GB with ballooning) and there was some memory left for the host the problem was solved and everything was working.

A few things that I tried which were described as solutions that didn't have an effect in the end are:
1. Enabling/disabling ROM-BAR
2. Leaving one NIC for the host
3. Trying different NIC's just in case the host had earmarked one for itself and it was somehow clashing
 
You saved me. I can confirm the same behavior.

16 GB ram on proxmox host
  • vm1 min4Gb max8Gb
  • vm2 min6gb max8Gb
without pci pass-through on vm2, everything works, two VMs can run together on the Host.
Instead as soon as I enable a pci pass-through on any VM and two VMs run together, Proxmox crash.

It seems (confirm) that PCI pass-through requires that there is no over memory allocation. I would like to better understand this to know the exact rule that rules this
 
ronzino said:
16 GB ram on proxmox host
  • vm1 min4Gb max8Gb
  • vm2 min6gb max8Gb
without pci pass-through on vm2, everything works, two VMs can run together on the Host.
Instead as soon as I enable a pci pass-through on any VM and two VMs run together, Proxmox crash.

It seems (confirm) that PCI pass-through requires that there is no over memory allocation. I would like to better understand this to know the exact rule that rules this
Click to expand...
Two times 8GB on a 16GB is memory over-commit because Proxmox also requires memory: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_system_requirements .
It worked without PCI(e) passthrough because Proxmox will take memory away from the VMs as soon as the host memory usage is at least 80%: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_memory . Ballooning does not work with PCI(e) passthrough because PCI(e) devices that are passed through can do Direct Memory Access (DMA) at any time. Therefore all VM memory must be pinned into actual host RAM.
 
Thanks for the kudos, it is my pleasure.
This is a public service announcement.
Be diligent, if you find the solution to your problem remember to close the loop so that another poor soul may benefit down the line.
It is such a joy when you find that one post hidden in internet history that solves that head scratcher of yours.
 
leesteken said:
Two times 8GB on a 16GB is memory over-commit because Proxmox also requires memory: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_system_requirements .
It worked without PCI(e) passthrough because Proxmox will take memory away from the VMs as soon as the host memory usage is at least 80%: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_memory . Ballooning does not work with PCI(e) passthrough because PCI(e) devices that are passed through can do Direct Memory Access (DMA) at any time. Therefore all VM memory must be pinned into actual host RAM.
Click to expand...
Thanks for the explanation. Now this is complete.
 
leesteken said:
Two times 8GB on a 16GB is memory over-commit because Proxmox also requires memory: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_system_requirements .
It worked without PCI(e) passthrough because Proxmox will take memory away from the VMs as soon as the host memory usage is at least 80%: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_memory . Ballooning does not work with PCI(e) passthrough because PCI(e) devices that are passed through can do Direct Memory Access (DMA) at any time. Therefore all VM memory must be pinned into actual host RAM.
Click to expand...
Quick check so I don't misunderstand it.
In light of this explanation of how things work for this type of VM, no point in setting min and max memory, it won't baloon.
So makes more sense to have both values the same? That is what I have by the way, same values and ballon disabled.
 
pengu1n said:
Quick check so I don't misunderstand it.
In light of this explanation of how things work for this type of VM, no point in setting min and max memory, it won't baloon.
So makes more sense to have both values the same? That is what I have by the way, same values and ballon disabled.
Click to expand...
I don't understand what you are asking. You can have the balloon device enabled in the VM but it will never reduce the memory from max. It does not matter what you set the min to. You can also disable the balloon device (but maybe memory usage reports will be less accurate; I'm not sure). Your are free to do any of these things but it will not balloon when passthrough is used.
 
It would be nice that Proxmox would not allow to insert a min value when passthrough is on (or at least a callout warning message). Now i have another question related to this (about integrated intel GPU, vGPU an memory aperture size) but as it would be off-of-this-topic , I am going to open a new one.
 
You must log in or register to reply here.
Top Bottom