PCIe passthrough stops after suspends to RAM.

Tassir

Active Member
Feb 22, 2016
7
1
43
38
Hi,

I got Proxmox installed and set up just the way I envisioned it and I have been loving it. There is one thing that's left and I can't seem to get it sorted out.

Current set up is as follow:
- Proxmox installed with 4 NICs and an LSI HBA card.
- eth0 is bridged for the host itself.
- eth1 and the LSI card are passed through for the NAS.
- eth2 and eth3 are passed through for Pfsense.

The server works beautifully and is rock solid, the one thing that I would love to be able to do is to suspend it to RAM when not used and resume when I need it (through wake on LAN). I have had this working nicely when I had just the NAS running on bare metal, but now that I added PCIe passthrough, it doesn't seem to work.

I can still suspend the server as usual "rtcwake -m mem -s whatever" and then wake it up on time, the problem is that after resume, all the VMs lose access the devices that were passed in by Proxmox (no NICs and no drives). The only way to get the VMs up and running again is to reset them which is not an option as I want to keep the ZFS cache on the NAS (otherwise might as well just shut the host down).

I have spent days looking for this but I have reached a dead end and it doesn't seem like there is much info regarding this. I guess it's not a usual set up to build a server, load it up with VMs with devices being passed through all over the place and then wanting to suspend it!! But in my case, I am out for work all day and I literally get like 2 hours of use a day from this server so It doesn't really make sense to have it on 24/7.

Hardware set up:
- Asus Z9PA-U8.
- Intel E5 2660 v2.
- 64GB RAM.
- LSI HBA card (6 drives attached).
- 4 Intel NICs.

Thanks
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!