I have some other issues with the 6.2 kernel and hardware. I see people downgraded to the 5.15 kernel. How does one get that on a Proxmox 8 system? This was a clean install, not an upgrade from 7.x.
Yes, this matches our experience as well. We also don't see issues with VMs of low memory or less vCPUs or even CPU sockets assigned. But as we usually use one proxmox host for a single w2019 RDS server vm with full CPU and RAM capacity (>200GB RAM) the issue is quite prominent. As said earlier, things improved with themitigations=off
approach you suggested (thx!) but today as users logged in we can clearly see the issue again. Not so prominent and with less probability, but especially running a constant ICMP ping against a vm shows that from time to time ping times raise up to 10-20 seconds and then the novnc proxmox console freezes for some time, the proxmox host itself then shows also 100% cpu spikes and then after some seconds everything returns to normal but spikes come back after some time again. As said, with mitigations disabled it works better now, but not as smoothly as with kernel 5.15 booted. So there must be something else. So can you please share a VM config of one of your w2019 VMs so that we can compare.
root@063-pve-04347:~# cat /etc/pve/qemu-server/6303.conf
agent: 1
bios: ovmf
boot: order=ide2;scsi0
sockets: 2
cores: 24
cpu: host
numa: 1
ide2: none,media=cdrom
machine: pc-i440fx-8.0
memory: 147456
meta: creation-qemu=8.0.2,ctime=1687984201
name: srv-term-mng.domain
net0: virtio=00:50:56:AD:3E:A5,bridge=vmbr0,firewall=1,tag=630
ostype: win10
scsi0: rbd:vm-6303-disk-0,size=1T,ssd=1T
scsihw: virtio-scsi-pci
smbios1: uuid=847d250b-c8a0-4a8f-9518-7993adf7e1ff
vmgenid: 32448db6-e370-48bf-bd13-d1b8122ac3ec
root@063-pve-04347:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel
Model name: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
BIOS Model name: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz CPU @ 2.6GHz
BIOS CPU family: 179
CPU family: 6
Model: 79
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
Stepping: 1
CPU(s) scaling MHz: 87%
CPU max MHz: 3500.0000
CPU min MHz: 1200.0000
BogoMIPS: 5200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx
fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts re
p_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx sm
x est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadli
ne_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 in
vpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_a
djust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cq
m_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 896 KiB (28 instances)
L1i: 896 KiB (28 instances)
L2: 7 MiB (28 instances)
L3: 70 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
Vulnerabilities:
Itlb multihit: KVM: Vulnerable
L1tf: Mitigation; PTE Inversion; VMX vulnerable
Mds: Vulnerable; SMT vulnerable
Meltdown: Vulnerable
Mmio stale data: Vulnerable
Retbleed: Not affected
Spec store bypass: Vulnerable
Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Srbds: Not affected
Tsx async abort: Vulnerable
on my nodes is "Y"cat /sys/module/kvm_*/parameters/nested ?
if Y switch off and test
root@063-pve-04347:~# cat /sys/module/kvm_*/parameters/nested
Y
cat /sys/module/kvm_*/parameters/nested ?
if Y switch off and test
options kvm_intel nested=0
things did not improve and we still see 100%CPU and ICMP ping spikes and novnc console temp stalling.Can you post output of
grep -H '' /sys/module/kvm/parameters/*
grep -H '' /sys/module/kvm_intel/parameters/*
in kernel 6.2
# grep -H '' /sys/module/kvm/parameters/*
grep -H '' /sys/module/kvm_intel/parameters/*
/sys/module/kvm/parameters/eager_page_split:Y
/sys/module/kvm/parameters/enable_pmu:Y
/sys/module/kvm/parameters/enable_vmware_backdoor:N
/sys/module/kvm/parameters/flush_on_reuse:N
/sys/module/kvm/parameters/force_emulation_prefix:0
/sys/module/kvm/parameters/halt_poll_ns:200000
/sys/module/kvm/parameters/halt_poll_ns_grow:0
/sys/module/kvm/parameters/halt_poll_ns_grow_start:10000
/sys/module/kvm/parameters/halt_poll_ns_shrink:0
/sys/module/kvm/parameters/ignore_msrs:N
/sys/module/kvm/parameters/kvmclock_periodic_sync:Y
/sys/module/kvm/parameters/lapic_timer_advance_ns:-1
/sys/module/kvm/parameters/min_timer_period_us:200
/sys/module/kvm/parameters/mitigate_smt_rsb:N
/sys/module/kvm/parameters/mmio_caching:Y
/sys/module/kvm/parameters/nx_huge_pages:N
/sys/module/kvm/parameters/nx_huge_pages_recovery_period_ms:0
/sys/module/kvm/parameters/nx_huge_pages_recovery_ratio:60
/sys/module/kvm/parameters/pi_inject_timer:0
/sys/module/kvm/parameters/report_ignored_msrs:Y
/sys/module/kvm/parameters/tdp_mmu:Y
/sys/module/kvm/parameters/tsc_tolerance_ppm:250
/sys/module/kvm/parameters/vector_hashing:Y
/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
/sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_ipiv:N
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
/sys/module/kvm_intel/parameters/enlightened_vmcs:N
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:Y
/sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:Y
/sys/module/kvm_intel/parameters/nested_early_check:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/ple_window_grow:2
/sys/module/kvm_intel/parameters/ple_window_max:4294967295
/sys/module/kvm_intel/parameters/ple_window_shrink:0
/sys/module/kvm_intel/parameters/pml:Y
/sys/module/kvm_intel/parameters/preemption_timer:Y
/sys/module/kvm_intel/parameters/pt_mode:0
/sys/module/kvm_intel/parameters/sgx:N
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmentry_l1d_flush:never
/sys/module/kvm_intel/parameters/vnmi:Y
/sys/module/kvm_intel/parameters/vpid:Y
That's exactly what we are seeing here as well when trying a fresh install. Can you try to downgrade to kernel 5.15 and see if this improves the situation? In addition, please state which underlying host hardware you are using and what are the VM config settings especially regarding number of assigned CPU sockets and cores. In recent tests it looks like that limiting the windows2019 VM to only a single CPU socket improves the VM performance also under kernel 6.2. That would explain why @Ramalana did not see any problem since he has just a single CPU on his host hardware.
$> dpkg --list | grep -i pmx1
ii libnss-myhostname:amd64 252.12-pmx1 amd64 nss module providing fallback resolution for the current hostname
ii libnss-systemd:amd64 252.12-pmx1 amd64 nss module providing dynamic user and group name resolution
ii libpam-systemd:amd64 252.12-pmx1 amd64 system and service manager - PAM module
ii libsystemd-shared:amd64 252.12-pmx1 amd64 systemd shared private library
ii libsystemd0:amd64 252.12-pmx1 amd64 systemd utility library
ii libudev-dev:amd64 252.12-pmx1 amd64 libudev development files
ii libudev1:amd64 252.12-pmx1 amd64 libudev shared library
ii systemd 252.12-pmx1 amd64 system and service manager
ii systemd-sysv 252.12-pmx1 amd64 system and service manager - SysV compatibility symlinks
ii udev 252.12-pmx1 amd64 /dev/ and hotplug management daemon
HPE ML350p Gen8
CPU(s)
16 x Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz (2 Sockets)
Kernel Version
Linux 6.2.16-4-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-5 (2023-07-14T17:53Z)
PVE Manager Version
pve-manager/8.0.3/bbf3993334bfa916
balloon: 0
boot: order=sata0;scsi1
sockets: 1
cores: 4
cpu: kvm64
machine: pc-q35-8.0
memory: 16384
meta: creation-qemu=8.0.2,ctime=1689529605
name: GAMES
net0: e1000=D2:96:4A:B2:F1:A6,bridge=vmbr0
numa: 0
ostype: other
sata0: local:iso/en_windows_server_2019_updated_jan_2021_x64_dvd_5ef22372.iso,media=cdrom,size=5406530K
scsi1: local:3000/vm-3000-disk-0.qcow2,aio=native,backup=0,size=150G
smbios1: uuid=44cacc68-1f15-49a2-a28d-1bc8254e8db6
vmgenid: 138ee4a9-1c44-40bd-b672-3911bb008ca3
vmstatestorage: local
Can you post output of
grep -H '' /sys/module/kvm/parameters/*
grep -H '' /sys/module/kvm_intel/parameters/*
in kernel 6.2
root@proxmox2:~# uname -a
Linux proxmox2 6.2.16-4-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-5 (2023-07-14T17:53Z) x86_64 GNU/Linux
root@proxmox2:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel
Model name: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
BIOS Model name: Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz CPU @ 3.3GHz
BIOS CPU family: 179
CPU family: 6
Model: 62
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
Stepping: 4
CPU(s) scaling MHz: 43%
CPU max MHz: 4000.0000
CPU min MHz: 1200.0000
BogoMIPS: 6584.03
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant
_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid
dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi fle
xpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 512 KiB (16 instances)
L1i: 512 KiB (16 instances)
L2: 4 MiB (16 instances)
L3: 50 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
Vulnerabilities:
Itlb multihit: KVM: Mitigation: Split huge pages
L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
Mds: Mitigation; Clear CPU buffers; SMT disabled
Meltdown: Mitigation; PTI
Mmio stale data: Unknown: No mitigations
Retbleed: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
root@proxmox2:~# grep -H '' /sys/module/kvm/parameters/*
/sys/module/kvm/parameters/eager_page_split:Y
/sys/module/kvm/parameters/enable_pmu:Y
/sys/module/kvm/parameters/enable_vmware_backdoor:N
/sys/module/kvm/parameters/flush_on_reuse:N
/sys/module/kvm/parameters/force_emulation_prefix:0
/sys/module/kvm/parameters/halt_poll_ns:200000
/sys/module/kvm/parameters/halt_poll_ns_grow:0
/sys/module/kvm/parameters/halt_poll_ns_grow_start:10000
/sys/module/kvm/parameters/halt_poll_ns_shrink:0
/sys/module/kvm/parameters/ignore_msrs:Y
/sys/module/kvm/parameters/kvmclock_periodic_sync:Y
/sys/module/kvm/parameters/lapic_timer_advance_ns:-1
/sys/module/kvm/parameters/min_timer_period_us:200
/sys/module/kvm/parameters/mitigate_smt_rsb:N
/sys/module/kvm/parameters/mmio_caching:Y
/sys/module/kvm/parameters/nx_huge_pages:Y
/sys/module/kvm/parameters/nx_huge_pages_recovery_period_ms:0
/sys/module/kvm/parameters/nx_huge_pages_recovery_ratio:60
/sys/module/kvm/parameters/pi_inject_timer:0
/sys/module/kvm/parameters/report_ignored_msrs:N
/sys/module/kvm/parameters/tdp_mmu:Y
/sys/module/kvm/parameters/tsc_tolerance_ppm:250
/sys/module/kvm/parameters/vector_hashing:Y
root@proxmox2:~# grep -H '' /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
/sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_ipiv:N
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:N
/sys/module/kvm_intel/parameters/enlightened_vmcs:N
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:N
/sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:Y
/sys/module/kvm_intel/parameters/nested_early_check:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/ple_window_grow:2
/sys/module/kvm_intel/parameters/ple_window_max:4294967295
/sys/module/kvm_intel/parameters/ple_window_shrink:0
/sys/module/kvm_intel/parameters/pml:N
/sys/module/kvm_intel/parameters/preemption_timer:Y
/sys/module/kvm_intel/parameters/pt_mode:0
/sys/module/kvm_intel/parameters/sgx:N
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmentry_l1d_flush:cond
/sys/module/kvm_intel/parameters/vnmi:Y
/sys/module/kvm_intel/parameters/vpid:Y
try with mitigations off and ksm disabled
Haven't tried to disable KSM yet. What is the exact procedure to do that?@jens-maus have you tried to disable KSM on top of "mitigations=off" before VM start?
in my setup there is no memory pressure within the VM, so KSM in inactive state
When I setup 1 VM per host I used to disable KSM on the node following this guide: https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM)Haven't tried to disable KSM yet. What is the exact procedure to do that?
I dont think this solve the "lagg" problem, the same settings on "Windows 2012R2" working flawlessly.try with mitigations off and ksm disabled
Links ? References ?E5-xxx v2 doesn't have many "mitigation problem" unlike the v3/v4 series.
What links?Links ? References ?
It's not about your words, i meant really lscpu....sorry for my words.
I mean what are your "sources" or "experiences" ? for your assumption :"E5-xxx v2 doesn't have many "mitigation problem" unlike the v3/v4 series.
Vulnerabilities:
Itlb multihit: KVM: Mitigation: Split huge pages
L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vul nerable
Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Meltdown: Mitigation; PTI
Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Retbleed: Mitigation; IBRS
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitizat ion
Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP conditional, RSB fillin g, PBRSB-eIBRS Not affected
Srbds: Mitigation; Microcode
Tsx async abort: Mitigation; TSX disabled
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitizat ion
Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-e IBRS SW sequence
Srbds: Not affected
Tsx async abort: Not affected
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitizat ion
Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-o n, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Vulnerable: No microcode
Vulnerability Tsx async abort: Not affected