We're looking into it and I already tried to backport that finicky fpu masking patch that is also linked here, I went for another try today and managed to get something that builds, boots and also seems to host VMs and allows migrating them, but it is definitively not battle tested enough for production, iow. like always we got the bootstrap problem, we need people to test it to (hopefully) declare it stable and release over the normal channel, but most only got production setups and thus want to wait out for other feedback and us to declare it stable.
All related packages are available at
http://download.proxmox.com/temp/pve-kernel-5.15.35-guest-fpu-mask/
The SHA256 sums are:
Code:
4f15a1e9fdcfae212d6279f26cc76f76300655c6dbfc2b53cf990deaedb4159d linux-tools-5.15_5.15.39-3_amd64.deb
85d029b6a27b541121a3a627c135caded4213fb97cb1c9a36ce1bf0eedf8da45 linux-tools-5.15-dbgsym_5.15.39-3_amd64.deb
bb2c7263e0699cb27cda64ac50d81f5956bf6a3de3e6ac228e9cb6f71f66b29b pve-headers-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb
5ba1c056d80f4cb7c05cbdf37524ee897302918a59e1dfdc2099e51e0e07efb1 pve-kernel-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb
cf2c24aea618a96fcff3a9a67a318feb04aa554bba0c40670cb51b2a117ae6fd pve-kernel-libc-dev_5.15.39-3_amd64.deb
For most setups it's enough to get the
pve-kernel-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb one:
Code:
wget http://download.proxmox.com/temp/pve-kernel-5.15.35-guest-fpu-mask/pve-kernel-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb
# verify the checksum
sha256sum pve-kernel-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb
5ba1c056d80f4cb7c05cbdf37524ee897302918a59e1dfdc2099e51e0e07efb1 pve-kernel-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb
apt install ./pve-kernel-5.15.39-3-pve-guest-fpu_5.15.39-3_amd64.deb
# optionally ensure that this specific kernel gets booted via a pin
proxmox-boot-tool kernel pin 5.15.39-3-pve-guest-fpu
systemctl reboot
As said, boot and basic tests went fine with above, not recommended for production setups.
Note also that even if this test kernel above will fix the specific regression, this is a problem that will reappear from time to time
for heterogeneous clusters, while different CPU vendors is def. making it more likely, bigger architectural leaps can sometimes be enough to trigger such issues between different generations from the same vendor, as a new kernel comes around and either it:
- unlocks features from the newer generation that the older don't have, trickling into the guest state; like probably here happened with enabling Intel AMX (aka TMUL) in between the 5.15.30 and 5.15.35 kernel.
- needs to be more restrictive, e.g., due to security issues
And the only guarantee that no such state leakage has any effect is to use a homogeneous cluster. All the steadily growing millions of HW combinations cannot realistic get tested on every kernel release, which often come along with time pressure of getting out a mitigation for a critical issue.
Platforms that state they can handle and guarantee that simply don't tell the truth or have a very limited set of HW supported in the first place. FWIW, having the complexity of the systems and the amount of change that happens on a smaller level in mind and the vast abstractions required to hide those away for VMs and migration, it works out quite well almost surprisingly often IMO.
tldr; if you don't want this to happen the easiest way is to:
- use enterprise HW
- use as identical HW as possible (cpu, mainboard, system vendor wise)
- keep firmware and µcode updated and at the same level
All else may work out often, but simply cannot be guaranteed for.
This cannot be "bought" away in software, get fitting HW if you want a production setup.
