PVE 6 to 7 - Call trace kvm

[glanc]

Hello just upgraded my VDS on Contabo from pve 6.4 to 7. After restarting all kvm guest are stuck at booting, lxc containers are working fine. In the boot log i see these messages: I could not find any info on the web. Help needed.

[Wed Sep 15 15:30:36 2021] [drm:vmw_probe.cold [vmwgfx]] *ERROR* Hardware has no pitchlock
[Wed Sep 15 15:30:36 2021] vmwgfx: probe of 0000:00:02.0 failed with error -38
[Wed Sep 15 15:30:36 2021] ZFS: Loaded module v2.0.5-pve1, ZFS pool version 5000, ZFS filesystem version 5
[Wed Sep 15 15:30:36 2021] cryptd: max_cpu_qlen set to 1000
[Wed Sep 15 15:30:36 2021] AVX2 version of gcm_enc/dec engaged.
[Wed Sep 15 15:30:36 2021] AES CTR mode by8 optimization enabled
[Wed Sep 15 15:30:36 2021] unchecked MSR access error: RDMSR from 0xda0 at rIP: 0xffffffff8b27d5f8 (native_read_msr+0x8/0x40)
[Wed Sep 15 15:30:36 2021] Call Trace:
[Wed Sep 15 15:30:36 2021] kvm_arch_hardware_setup+0x475/0x490 [kvm]
[Wed Sep 15 15:30:36 2021] kvm_init+0x9e/0x2b0 [kvm]
[Wed Sep 15 15:30:36 2021] ? svm_hardware_setup+0x518/0x518 [kvm_amd]
[Wed Sep 15 15:30:36 2021] svm_init+0x26/0x28 [kvm_amd]
[Wed Sep 15 15:30:36 2021] do_one_initcall+0x48/0x1d0
[Wed Sep 15 15:30:36 2021] ? kmem_cache_alloc_trace+0xf6/0x200
[Wed Sep 15 15:30:36 2021] ? do_init_module+0x28/0x290
[Wed Sep 15 15:30:36 2021] do_init_module+0x62/0x290
[Wed Sep 15 15:30:36 2021] load_module+0x25ca/0x2820
[Wed Sep 15 15:30:36 2021] ? security_kernel_post_read_file+0x5c/0x70
[Wed Sep 15 15:30:36 2021] __do_sys_finit_module+0xc2/0x120
[Wed Sep 15 15:30:36 2021] __x64_sys_finit_module+0x1a/0x20
[Wed Sep 15 15:30:36 2021] do_syscall_64+0x38/0x90
[Wed Sep 15 15:30:36 2021] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Wed Sep 15 15:30:36 2021] RIP: 0033:0x7fd6403bd9b9
[Wed Sep 15 15:30:36 2021] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a7 54 0c 00 f7 d8 64 89 01 48
[Wed Sep 15 15:30:36 2021] RSP: 002b:00007ffe842de4f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[Wed Sep 15 15:30:36 2021] RAX: ffffffffffffffda RBX: 000055dbfc81ad30 RCX: 00007fd6403bd9b9
[Wed Sep 15 15:30:36 2021] RDX: 0000000000000000 RSI: 00007fd640548e2d RDI: 0000000000000018
[Wed Sep 15 15:30:36 2021] RBP: 0000000000020000 R08: 0000000000000000 R09: 000055dbfc876240
[Wed Sep 15 15:30:36 2021] R10: 0000000000000018 R11: 0000000000000246 R12: 00007fd640548e2d
[Wed Sep 15 15:30:36 2021] R13: 0000000000000000 R14: 000055dbfc871980 R15: 000055dbfc81ad30
[Wed Sep 15 15:30:36 2021] kvm: Nested Virtualization enabled
[Wed Sep 15 15:30:36 2021] SVM: kvm: Nested Paging enabled
[Wed Sep 15 15:30:36 2021] Decoding supported only on Scalable MCA processors.

 

[Stefan_R]

MSR 0xda0 is MSR_IA32_XSS, which is related to the 'xsaves' CPU feature. This looks to me like your provider's host fails to emulate a specific CPU feature correctly, which is in use for KVM with the updated 5.11 kernel of PVE 7. You may want to indicate this to your service provider, so they can update their kernel version on the host or otherwise mitigate the issue.

As a workaround, it might be possible to use the noxsaves or even noxsave noxsaveopt noxsaves kernel parameters. Add these to your kernel commandline as explained in our documentation. See also the kernel documentation (CTRL-F "xsave").

 

[glanc]

thanks a lot Stefan for your reply, I'll contact Contabo and tell them about the issue.

 

[sachz]

@glanc : Did you get to solve it with Contabo?
When i tried it i had the same, and the vnc access was no longer working.
Since it was a clean install I just redeployed the proxmox image from Contabo and started over. But i've been running the vds for a while and am very happy with it, but i would love to upgrade to the latest version of proxmox.

thanks!

 

[glanc]

@glanc : Did you get to solve it with Contabo?
When i tried it i had the same, and the vnc access was no longer working.
Since it was a clean install I just redeployed the proxmox image from Contabo and started over. But i've been running the vds for a while and am very happy with it, but i would love to upgrade to the latest version of proxmox.

thanks!

no! contabo still does not list pve ver 7 as an os template when trying to reinstall vds so probably the problem i still there, We should ask them when they'll insert pve vers 7 as a template

 

[sachz]

no! contabo still does not list pve ver 7 as an os template when trying to reinstall vds so probably the problem i still there, We should ask them when they'll insert pve vers 7 as a template

yeah, i'd prefer an upgrade path.
I made a setup with a virtual firewall and proxmox behind it, if I have to backup everything and restore afterwards for the new template it will be a pain. I'm hoping there will be an upgrade path in the future so we can just upgrade from within the VDS with no issue. Because besides this I'm very happy with their service.

 

[cjangrist]

Did you guys figure out how to fix this? I'm deciding between the cheapest bare metal ($120/mo) at Vultr https://www.vultr.com/products/bare-metal/#pricing or the Cloud VDS XL at contabo

 

[sachz]

Did you guys figure out how to fix this? I'm deciding between the cheapest bare metal ($120/mo) at Vultr https://www.vultr.com/products/bare-metal/#pricing or the Cloud VDS XL at contabo

Honestly, I moved away from contabo. Support takes a while to answer, their systems don’t work with the kernel that comes with PVE7 so you have to run an old kernel or run pve6.
For the exact same price as a vds at contabo I got a dedicated server at herzner with more resources and diskspace for the same price, only difference is I get a reply from herzner within the hour, contabo’s max was leaving my vds broken without help for 5 days until I manually performed kernel downgrades and got my server back online. This left my mailserver and document server unreachable (including data) offline for an extended period of time.

Bad communication, bad service, bad performance due to extra virtualisation layer, and they don’t really seem to care.

Hope this helps!

 

[sachz]

Honestly, I moved away from contabo. Support takes a while to answer, their systems don’t work with the kernel that comes with PVE7 so you have to run an old kernel or run pve6.
For the exact same price as a vds at contabo I got a dedicated server at herzner with more resources and diskspace for the same price, only difference is I get a reply from herzner within the hour, contabo’s max was leaving my vds broken without help for 5 days until I manually performed kernel downgrades and got my server back online. This left my mailserver and document server unreachable (including data) offline for an extended period of time.

Bad communication, bad service, bad performance due to extra virtualisation layer, and they don’t really seem to care.

Hope this helps!

When I opened the case something became very clear.
With the issue they first said we will activate parameter on your vds to make it work. That did not work, even when giving all the information and told them their host OS needs to be on the correct kernel to be able to run proxmox 7 nested they did not reply, they just assumed I made a mistake. This while I tested the whole scenario at home on my lab environment and gave them detailed information about the issue, my tests and the results.
They also didn’t test PVE7 before only offering that option for installation on the VDS. Not acceptable for a datacenter provider imho.
Technical support is unreachable for customers. Even the support people can only reach technical support by e-mail.
This makes the whole support process a pain, I had to call multiple times and each time I would get a person that had no clue what was going on but asking for my patience. Internal notes in the support system are also barely made, that made it I had to explain the issue completely each time I called. And with each call i would say it’s very urgent because my production environment was not working and impacting business, yet my ticket did not get that status so it took a long time.

Keep in mind that i knew about the issues before the upgrade and contacted support ahead of time to ask if everything was ready and in order for the upgrade because it failed earlier in the year and i had to reinstall to pve6.

 

[cjangrist]

Oh man I literally just clicked buy! They wanted to verify my info before deploying so I'll tell the to confirm that proxmox 7 works before moving forward.

really hoping they've made some changes!

 

[sachz]

Oh man I literally just clicked buy! They wanted to verify my info before deploying so I'll tell the to confirm that proxmox 7 works before moving forward.

really hoping they've made some changes!

This happenend a month ago to give you an idea, if you went for a personal account you have the right to cancel within the first 14 days in europe, might be something to look into