Dear all,
I have a cluster of 3 proxmox nodes, currently 45 VMs in a research setting. I have been operating this cluster for about 6 years now, started out with proxmox 4, and have been upgrading every time to the latest release according to the upgrade protocol without too much trouble. However, that changed this year in February when I upgraded from proxmox 6.4 to 7.1. Ever since I have been experiencing some issues with the stability of the VMs. First of, I was unable to live migrate some of the VMs back after the upgrade, because of the issue that was also mentioned here: https://forum.proxmox.com/threads/116315. This was unfortunate because I had already started the migration, so I had to force kill some of the VMs. According to that thread, this might be resolved by upgrading to the opt-in kernel 5.19, which indeed was the case. However, after that, some VMs would mysteriously get into this "stuck" state, and I have been unable to find out why, which is why I am posting this question here.
It happens relatively rarely, but often enough to become very annoying. I have been keeping track, and it has occurred exactly 17 times between February 9th, and May 22 (today), which is on average once a week. I am unable to find any pattern in which VMs crash, and which do not, except for the fact that it seems that the VMs that are more actively used, also seem to crash more often.
At first I used to force reset the stuck VMs, but I found out that live migrating the VMs to another node also seems to resolve the stuck state. I am unable to find any logging info, but I am also not sure where to look. On the guests, I find lots of timeout warnings after the VM is released, which makes sense, but nothing that I think points to a probable cause. I cannot find anything on the host machine, but please point me in the right direction if you think there could be something useful.
In attempts to fix this I already completely reinstalled the proxmox host OS on all three nodes. I have enough resources to completely drain a single node at a time, so I did exactly that to do a clean install of proxmox, including installing the opt-in kernel, which is now at version 6.2. I also recently installed an up-to-date bios firmware, and intel-microcode packages, but that seems to have no effect.
Finally some more info on the used hardware and software.
Hardware
Software
I have a cluster of 3 proxmox nodes, currently 45 VMs in a research setting. I have been operating this cluster for about 6 years now, started out with proxmox 4, and have been upgrading every time to the latest release according to the upgrade protocol without too much trouble. However, that changed this year in February when I upgraded from proxmox 6.4 to 7.1. Ever since I have been experiencing some issues with the stability of the VMs. First of, I was unable to live migrate some of the VMs back after the upgrade, because of the issue that was also mentioned here: https://forum.proxmox.com/threads/116315. This was unfortunate because I had already started the migration, so I had to force kill some of the VMs. According to that thread, this might be resolved by upgrading to the opt-in kernel 5.19, which indeed was the case. However, after that, some VMs would mysteriously get into this "stuck" state, and I have been unable to find out why, which is why I am posting this question here.
It happens relatively rarely, but often enough to become very annoying. I have been keeping track, and it has occurred exactly 17 times between February 9th, and May 22 (today), which is on average once a week. I am unable to find any pattern in which VMs crash, and which do not, except for the fact that it seems that the VMs that are more actively used, also seem to crash more often.
At first I used to force reset the stuck VMs, but I found out that live migrating the VMs to another node also seems to resolve the stuck state. I am unable to find any logging info, but I am also not sure where to look. On the guests, I find lots of timeout warnings after the VM is released, which makes sense, but nothing that I think points to a probable cause. I cannot find anything on the host machine, but please point me in the right direction if you think there could be something useful.
In attempts to fix this I already completely reinstalled the proxmox host OS on all three nodes. I have enough resources to completely drain a single node at a time, so I did exactly that to do a clean install of proxmox, including installing the opt-in kernel, which is now at version 6.2. I also recently installed an up-to-date bios firmware, and intel-microcode packages, but that seems to have no effect.
Finally some more info on the used hardware and software.
Hardware
The host OS is installed on a ZFS mirrored SSD, and the VMs are stored on a Ceph cluster that is internally connected with a 10GB SFP+ fibre network on the backend. The Ceph cluster is running quincy 17.6, and I have no reason to suspect that there are any problems there. Of course, if you think it is relevant I could provide more details.
Software