Ubuntu 20.04.04 machine freezes

[fiona]

Hi,

Good afternoon
We have two hosts that are united in a cluster.
1st host: 88 x Intel(R) Xeon(R) CPU E5-2696 v4
2nd host: 96 x AMD EPYC 7642
When migrating from one host to another, the virtual machines were rebooted, in the properties of the virtual machine I had to change the processor type to KVM64 - the problem with the migration disappeared.
Virtual machines migrated from one host to another occurred live.

Live-migration between CPUs from different vendors generally are not supported. Even if you use a generic CPU type, it's still luck if it works or not and might break upon (kernel/QEMU) updates.

Another point has arisen that some virtual machines on Ubuntu freeze after migration after a random time, and some freeze even without migration, you can’t do anything in the console, only hard reset

What could be the problem? Can anyone tell me?

For those that freeze without migration, please provide more information like requested here: https://forum.proxmox.com/threads/ubuntu-20-04-04-machine-freezes.112507/post-662750

 

[dimond44]

Hi,

Live-migration between CPUs from different vendors generally are not supported. Even if you use a generic CPU type, it's still luck if it works or not and might break upon (kernel/QEMU) updates.

For those that freeze without migration, please provide more information like requested here: https://forum.proxmox.com/threads/ubuntu-20-04-04-machine-freezes.112507/post-662750

 

[fiona]

pve-qemu-kvm: 8.1.5-2

Please upgrade to the latest version. You need to shutdown+start a VM or migrate it to an upgraded node to have it actually use the new QEMU version.

How exactly does the VM hang? I.e. black screen/stuck/not possible to ping/etc.? Is there anything in the host or guest logs around the time the issue happens?

 

[dimond44]

pve-qemu-kvm: 8.1.5-2

Please upgrade to the latest version. You need to shutdown+start a VM or migrate it to an upgraded node to have it actually use the new QEMU version.

How exactly does the VM hang? I.e. black screen/stuck/not possible to ping/etc.? Is there anything in the host or guest logs around the time the issue happens?

Yes, I'll try to transfer it to a host with version 8.2.2.
The virtual machine simply freezes, nothing can be done in the console, I am attaching a screenshot.

 

[fiona]

Yes, I'll try to transfer it to a host with version 8.2.2.
The virtual machine simply freezes, nothing can be done in the console, I am attaching a screenshot.

So there is a log but unfortunately it's cut off. Would be great if you could get the complete log somehow (e.g. monitor it via SSH or maybe it made it to the persistent log). Can you correlate the freezes with some other operation you are doing? E.g. backup, snapshot or replication?

 

[dimond44]

So there is a log but unfortunately it's cut off. Would be great if you could get the complete log somehow (e.g. monitor it via SSH or maybe it made it to the persistent log). Can you correlate the freezes with some other operation you are doing? E.g. backup, snapshot or replication?

Unfortunately, there is no full log, because... The console freezes and it is impossible to get the log.
The virtual machine freezes when idle, and nothing is even done on it.

 

[fiona]

Yes, I'll try to transfer it to a host with version 8.2.2.

I'd recommend not using that testing version, because it has issues in combination with iothread. We'll release QEMU 9.0 soonish and skip 8.2 because of that. Instead, please use version 8.1.5-6, i.e. the latest on no-subscription and enterprise repository. EDIT: To clarify, I'm talking about the pve-qemu-kvm version, not the pve-manager version.

Unfortunately, there is no full log, because... The console freezes and it is impossible to get the log.
The virtual machine freezes when idle, and nothing is even done on it.

You can connect via SSH from another machine and monitor the system log/journal via journalctl -f (EDIT: fixed typo, should be -f not -b). But is there some task on the outside of the machine (please check the Task History)?

 

[dimond44]

I'd recommend not using that testing version, because it has issues in combination with iothread. We'll release QEMU 9.0 soonish and skip 8.2 because of that. Instead, please use version 8.1.5-6, i.e. the latest on no-subscription and enterprise repository.


You can connect via SSH from another machine and monitor the system log/journal via journalctl -b. But is there some task on the outside of the machine (please check the Task History)?

We have 3 hosts in the cluster:
1st host - version 8.2.2 - there are no virtual machines on it now, we migrated everything to the 2nd host
2nd host - version 8.1.4 - there are about 20 virtual machines on it, and freezes are observed only on virtual machines on Ubuntu
3rd host - version 8.1.11 - it has only 2 virtual machines on Windows 10 - they work stably.
I can connect via SSH, but the session is interrupted if the machine freezes and kicks me out of it.
Tasks are not performed from outside. Freezes occur at different times.

 

[fiona]

We have 3 hosts in the cluster:
1st host - version 8.2.2 - there are no virtual machines on it now, we migrated everything to the 2nd host
2nd host - version 8.1.4 - there are about 20 virtual machines on it, and freezes are observed only on virtual machines on Ubuntu
3rd host - version 8.1.11 - it has only 2 virtual machines on Windows 10 - they work stably.

Oh, you are talking about the pve-manager version. To clarify, above I'm talking about the pve-qemu-kvm version. Then yes, you should try the upgraded node.

I can connect via SSH, but the session is interrupted if the machine freezes and kicks me out of it.

Ah, sorry, there was a typo in my posts. The command should be journalctl -f, to follow the log. Then you'll see the log up to the point where the connection drops.

 

[dimond44]

Oh, you are talking about the pve-manager version. To clarify, above I'm talking about the pve-qemu-kvm version. Then yes, you should try the upgraded node.

Ah, sorry, there was a typo in my posts. The command should be journalctl -f, to follow the log. Then you'll see the log up to the point where the connection drops.

Today I migrated a virtual machine from a host with version 8.1.4 to a host with version 8.2.2
15 minutes after the migration, the virtual machine simply froze; nothing can be done with it except reboot.
Virtual machines on Ubuntu freeze
Maybe something else can be done with the virtual machine itself?

 

[fiona]

Today I migrated a virtual machine from a host with version 8.1.4 to a host with version 8.2.2
15 minutes after the migration, the virtual machine simply froze; nothing can be done with it except reboot.
Virtual machines on Ubuntu freeze
Maybe something else can be done with the virtual machine itself?

Was the migration between hosts with physical CPUs from different vendors? Was there anything in the system logs (host or guest) around the time the issue happened?

 

[dimond44]

Была ли миграция между хостами с физическими процессорами разных производителей? Было ли что-нибудь в системных журналах (хостовой или гостевой) в момент возникновения проблемы?

Was the migration between hosts with physical CPUs from different vendors? Was there anything in the system logs (host or guest) around the time the issue happened?

During the migration process, the hosts have different processors, the virtual machine itself has a CPU set to KVM64
The crash after migration occurs only for Ubuntu virtual machines
There is no way to view the magazine, because... The virtual machine freezes completely and freezes randomly.

 

[fiona]

During the migration process, the hosts have different processors, the virtual machine itself has a CPU set to KVM64
The crash after migration occurs only for Ubuntu virtual machines
There is no way to view the magazine, because... The virtual machine freezes completely and freezes randomly.

As already written: https://forum.proxmox.com/threads/ubuntu-20-04-04-machine-freezes.112507/post-663455

Live-migration between CPUs from different vendors generally are not supported. Even if you use a generic CPU type, it's still luck if it works or not and might break upon (kernel/QEMU) updates.

 

[dimond44]

As already written: https://forum.proxmox.com/threads/ubuntu-20-04-04-machine-freezes.112507/post-663455

This only happens on virtual machines with Ubuntu 22 installed

 

[fiona]

This only happens on virtual machines with Ubuntu 22 installed

Might just mean the guest kernel is more sensitive to some change caused by the migration or doing something else to trigger it.