This sounds like there was something else, possibly completely different, broken...
Open a different thread - if stop mode backup isn't involved this is for sure not the same issue as this thread had.
VM doesn't start Proxmox 6 - timeout waiting on systemd
- Thread starter homozavrus
- Start date
This sounds like there was something else, possibly completely different, broken...
Open a different thread - if stop mode backup isn't involved this is for sure not the same issue as this thread had.
Why would I start a new thread, when this one contains all the history of the issue? A VM locks up. It can't be reset. qm stop <vmid> from command line sends a sigkill. The VM cannot be restarted due to systemd timeout. Migration works, but you have to click the stop button to get it to migrate. The VM will never restart on the original host until the host is reboot.
I have no reports in my logs, the VM load is light, and the NFS server has stayed online throughout.
Gerald
The web gui not responding properly seems to have been a one off problem. So it is probably unrelated.
But the timeout error remains. Just for this one VM. Even after rebooting the host. All other VMs work well.
I have one VM that has become permanently un-startable (I will have to delete it, if no solution is found) and one VM that had a timeout issue but a "reboot" fixed that for now.
When the second VM did not start because of the timeout issue, it was impossible to reboot PVE (it tried to reboot but had some issue, with stuff locking up, that I eventually resolved by turning it off and on again).
When the second VM did not start because of the timeout issue, it was impossible to reboot PVE (it tried to reboot but had some issue, with stuff locking up, that I eventually resolved by turning it off and on again).
I tried disabling the blk-mq framework and it worked fine at first. This problem occurred again last night and this time I was able to track in zabbix exactly when it happened. After checking the logs, I found the event:
It turned out to be old disks and apparently it's time to replace them. Anyway, the problem discussed here can be reduced to the disk subsystem (something may be configured incorrectly, or the hardware may be outdated)
Code:
Jul 16 04: 11:23 pve02 kernel: INFO: task md126_raid10:742 blocked for more than 120 seconds.So, I managed to catch a VM as it locked up. It's IOWAIT time on disk access was using all of the CPUs, which matches what the Proxmox people have mentioned as a potential problem. I have my disk images on NFS.
The thing is, none of the other VMs showed the same IOWAIT, including one that uses the disk quite a lot. So, what made this particular VM have an IOWAIT issue? The NFS server was responsive and working.
I've moved several VMs off Proxmox to Ubuntu 20.04 running KVM with virt-manager and cockpit. I'll monitor to see if the issue happens there as well. My guess is no, since (for me) this has been a Proxmox 6 issue that I never saw on Proxmox 5.4 with the same hardware and NFS server.
The thing is, none of the other VMs showed the same IOWAIT, including one that uses the disk quite a lot. So, what made this particular VM have an IOWAIT issue? The NFS server was responsive and working.
I've moved several VMs off Proxmox to Ubuntu 20.04 running KVM with virt-manager and cockpit. I'll monitor to see if the issue happens there as well. My guess is no, since (for me) this has been a Proxmox 6 issue that I never saw on Proxmox 5.4 with the same hardware and NFS server.
Just to bring this to the top. I'm still seeing this issue. I can't start VM's because of the systemd timeout. :/
I have had to move away from Proxmox. 25% of my servers are now on Ubuntu 20.04 with libvirtd and cockpit, Yes, I lose functionality, but at least my VMs stay up.
I'd move back to Proxmox in a heartbeat if they solved this issue.
Gerald
Guys, do you use Intel CPUs who have the timeout issue? Because I had some issues with my Hetzner Servers on Intel CPUs including this and seems was able to fix them by adding intel_idle.max_cstate=1 to the file /etc/default/grub
and
and rebooting a server as described here https://forum.proxmox.com/threads/r...mox-ve-6-1-auf-ex62-nvme-hetzner.63597/page-3
Code:
GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 intel_idle.max_cstate=1"
Code:
# update-grubI wanted to say I've solved this bug timeout waiting on systemd and permanent crashing by adding intel_idle.max_cstate=1
All Servers and VMs are stable now.
All Servers and VMs are stable now.
Just to be clear: Did you make this change on the PVE host or the guest VMs?
Edit: Changed that cstate setting on my PVE host, updated grub, and rebooted. I've got three VMs that won't start up already. No dice.
Last edited: