Strange behaviour PVE 7 with 4-port I350-T4 NIC

[fbkreon]

Hello all,
I am experiencing some strange behavior with my PVE 7.0 installation and I think I need your help.
I am using a Dell OEM version of the Intel I350-T4 quad port adapters. This device is recognized by PVE without any problems and assigned to 4 different groups in IOMMU. I have assigned 2 VMs each a port via PCIe passthrough and the VMs have static IP addresses. When I start the VMs, they are initially reachable. But after some time - usually some hours - the VMs are not reachable anymore. When I then start an SSH session, I get the PVE CLI. Does anyone have an idea why PVE seems to "hijack" the NICs?

Thanks for helping me!

 

[Dunuin]

As soon as a VM stops the NICs will be given back to the host. I would guess your host is running out of memory (VMs with enabled PCI passthrough will always use the full RAM you assign to these NICs so you chould make sure you are not overprovisioning your RAM) and the OOM Killer kicks in and kills your VMs. Because of that the VMs aren't reachable anymore and the NICs were given back to your host. You should search your syslog for OOM messages and look if that the VM is stopped or still running when this happens.

Btw...half of the i350-T4 out there are chinese fakes. There are alot of threads out there with pictures to compare original and fake ones. Maybe you should check that if you don't directly bought that NIC from Dell.

 

[fbkreon]

I got this NIC from a trustable vendor and I checked the NIC when I got it in my mail. It's original. The VMs are working flawlessly and I can still reach them through the PVE console. According to PVE dashboard the VMs use about 10% of the assigned RAM. The VMs are alpine linux with unboard / pihole installations and 4gb RAM. The host is using up to 16% of available RAM.

 

[fbkreon]

Quick update:
I guess I solved the problem by making little changes to my VM configs - which where 1) upgraded assigned amount of RAM to make sure the machines don't run into OOM (though I couldn't find any error messages in syslog regarding any OOM issues), 2) I changed the machine type to q35 instead of i440fx and 3) I disabled NUMA option. So far, all VMs keep their NICs up and running.