Odd network issue after upgrade to Proxmox6.2

[adamb]

4 node cluster of 2x HP DL 560's, 1x HP DL 380, 1x Supermicro and Nimble storage. This cluster was on 5.x for a couple years now with no issues.

The issue only exists on one of the four nodes. Along with that the issue seems to only exist for the VM.

At random times the VM looses network connectivity, some packets are lost or the latency is very high. Whats interesting is this only happens to the VM, it doesn't happen to the host. If I have pings going to the VM and the host at the same time, only the VM will see the increase in latency or packet loss.

If I move the VM to one of the other hosts, the issue never happens.

I am stumped, anyone have any ideas?

 

[Deleted member 93625]

@adamb Just wondering if the connection between your host and the storage is alright. I guess you are using a shared storage?

 

[adamb]

@adamb Just wondering if the connection between your host and the storage is alright. I guess you are using a shared storage?

Yep storage network is solid. Its a private dedicated 10G network.

 

[adamb]

The only other issue I can seem to find is the following.

2020-07-30 05:14:32 starting migration of VM 100 to node 'fpracprox1' (10.211.45.1)
can't deactivate LV '/dev/Data/vm-100-disk-1':   Logical volume Data/vm-100-disk-1 is used by another device.
2020-07-30 05:14:33 ERROR: volume deactivation failed: Data:vm-100-disk-1 at /usr/share/perl5/PVE/Storage.pm line 1076.
2020-07-30 05:14:33 ERROR: migration finished with problems (duration 00:00:01)
TASK ERROR: migration problems

Everything seems solid on my storage network. This is still the only cluster seeing this issue.

 

[adamb]

Finally pinned this issue down. At some point I installed the numad package when the cluster was on proxmox5. Once moved to proxmox6 the the numad package was causing the VM to hang because it was moving it around to different numa nodes. Basically causing the VM to hang for periods of time.