Kernel 7.0.x - Windows VMs freezing with cpu high 60-70%, newer cpus?

[Maximiliano]

edited.

 

[glaeken2]

Do the freezes match the time where a guest-fsfreeze-freeze command is done? This can happen, for example when making backups.

Please take a look at [1] for a list of operations that trigger a qemu-guest-agent filesystem freeze via the guest-fsfreeze-freeze command.

Can you think of anything else that the affected affected guests have in common?

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_qga_fsfreeze

Nope, it's random, But I have one new information:

- attaching ptrace() always gets the VM unstuck. Judging by the trace, it takes ~1 second 100ms to get the vm "unstuck" mostly doing:
[pid [REMOVED]] [REMOVED].109866 madvise(0x7668fb600000, 4096, MADV_DONTNEED) = 0

i will analyze the ptrace, maybe i will find something more

This VM has a x86-64-v3, aio=native, scsi single, q35, seabios, 13gb ram, 1 NIC, all drivers virtio

The very first lines after attaching ptrace:

strace: Process [REMOVED] attached with 28 threads
[pid [REMOVED]] [REMOVED].106907 futex(0x6442491b2370, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 632, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid [REMOVED]] [REMOVED].107108 io_uring_enter(22, 0, 1, IORING_ENTER_GETEVENTS|0x80, NULL, 8 <unfinished ...>
[pid [REMOVED]] [REMOVED].107119 io_uring_enter(19, 0, 1, IORING_ENTER_GETEVENTS|0x80, NULL, 8 <unfinished ...>
[pid [REMOVED]] [REMOVED].107122 io_uring_enter(16, 0, 1, IORING_ENTER_GETEVENTS|0x80, NULL, 8 <unfinished ...>
[pid [REMOVED]] [REMOVED].107125 futex(0x644237cb0d28, FUTEX_WAIT, 4294967295, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107128 ppoll([{fd=5, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=24, events=POLLIN}, {fd=25, events=POLLIN}, {fd=26, events=POLLIN}, {fd=72,
 events=POLLIN}], 7, {tv_sec=2, tv_nsec=[REMOVED]}, NULL, 8 <unfinished ...>
[pid [REMOVED]] [REMOVED].107154 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107157 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107159 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107162 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107169 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107171 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid [REMOVED]] [REMOVED].107174 futex(0x644237c96200, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
-------------

[pid [REMOVED]] [REMOVED].107450 <... futex resumed>) = 0
[pid [REMOVED]] [REMOVED].107454 <... futex resumed>) = 1
[pid [REMOVED]] [REMOVED].107457 futex(0x644237c96200, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid [REMOVED]] [REMOVED].107461 ioctl(58, KVM_RUN <unfinished ...>
[pid [REMOVED]] [REMOVED].107565 <... futex resumed>) = 1
[pid [REMOVED]] [REMOVED].107567 <... futex resumed>) = 0

These are all different pids, probably different io threads waking up (probably going out of a stuck mutex)