Kernel panic on migrated machines

[dmembibre]

Hello,

we have a cluster with several servers running version 7.4, and we have already updated a couple of them to version 8. When we move machines from a server with version 7.4 to one with version 8.0, the machines experience kernel panic. This happens with different operating systems and their versions. However, if we move the same machines between servers running version 8, this issue does not occur. Has anyone else experienced this issue?

EX:

Server 1: CPU 48 x AMD EPYC 7413 24-Core Processor (1 Socket) Proxmox version 7.4-15
Server 2: CPU 48 x AMD EPYC 7413 24-Core Processor (1 Socket) Proxmox version 8.0.3

VM: Ubuntu 22.04



BR

UPDATE: Any machine that we migrate experiences kernel panic or freezes.

 

[Philipp Hufnagl]

can you post the config of the vim with cat /etc/pve/qemu-server/<VMID>.conf ?

 

[fiona]

Hi,
yes, the VM configuration would be interesting. Is there anything interesting in the system logs (host or guest) or migration logs? Can you run (on the host)

apt install cpuid
cpuid | grep "PK[ER]"

This will check if a certain feature is present on your host CPU, that might related to the issue.

 

[dmembibre]

Hi,

Finally, it only happened with two nodes in the cluster, any machine that was migrated froze, so we made the decision to turn off, move, and turn on. Now with the entire cluster in version 8 there is no problem.

BR

 

[fiona]

Hi,

Finally, it only happened with two nodes in the cluster, any machine that was migrated froze, so we made the decision to turn off, move, and turn on. Now with the entire cluster in version 8 there is no problem.

BR

Could you still share the output of the cpuid command? Would be interesting to know if it was that issue (fix is on the way: https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=1559d22f3510d8532aa756fe5d69c74c76211e1e)

 

[dmembibre]

Here is the command output

grep "PK[ER]"

OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)
OSPKE CR4.PKE and RDPKRU/WRPKRU          = true
PKRU state                           = true
PKRU features (0xd/9):
PKRU save state byte size                = 0x00000008 (8)
PKRU save state byte offset              = 0x00000980 (2432)

 

[fiona]

Yes, then it's likely the same issue. It can only happen when migrating from a 5.15 kernel to a 6.2 kernel without the fix I mentioned.