VM's Freezing Randomly during VNC

[btipton1]

Having issue with my VM's freezing completely during a VNC session. Have to reboot the whole VM to get back to a functioning state. Using tiger vnc client. Any ideas on where to look for the logs or where the problem may be?

```
May 5 17:09:11 homelab pvestatd[1301]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - unable to connect to VM 103 qmp socket - timeout
after 31 retries
```
Quite a few attempts of above in /var/log/syslog when they crash

 

[fiona]

Hi,
did the issue start appearing after a QEMU upgrade? If so, which version are you running now and which version did you run before (/var/log/apt/history.log might help checking). This thread might be talking about same issue. If you want, you can check with a gdb backtrace as described there too. Even if it's not the same issue, that would be helpful information.

EDIT: Could you also tell us which version of tiger vnc you are using (might be relevant for reproducing the issue)?

 

[fiona]

I was able to reproduce a freeze using tiger vnc and sent a patch (taken from upstream QEMU which already fixed it). Let's hope it's the same issue.

 

[btipton1]

Currently running TigerVNC version 1.12.0. The history.log file is empty. No recent upgrades to QEMU. Honestly haven't done any updates to my system since I got it up and running in the beginning of the year.
No mentions of any seg faults in the syslog.
proxmox-ve 7.1-7
qemu-server 7.1-4

Will attempt to run a gdb session.

 

[btipton1]

Having trouble getting a gdb session working. This is the script I have.

                                                                                                      1,1           Top
  1 #!/bin/bash
  2 /usr/bin/kvm \
  3   -id 100 \
  4   -name ArchBox \
  5   -no-shutdown \
  6   -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' \
  7   -mon 'chardev=qmp,mode=control' \
  8   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
  9   -mon 'chardev=qmp-event,mode=control' \
 10   -pidfile /var/run/qemu-server/100.pid \
 11   -smbios 'type=1,uuid=22f3c344-45bb-474c-9c9a-851648d78b3a' \
 12   -smp '4,sockets=2,cores=2,maxcpus=4' \
 13   -nodefaults \
 14   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
 15   -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' \
 16   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
 17   -m 4096 \
 18   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
 19   -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \
 20   -device 'vmgenid,guid=10a11734-6c31-4de4-9a70-d4516eee52fa' \
 21   -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
 22   -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
 23   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
 24   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
 25   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:fa94959361f2' \
 26   -drive 'file=/var/lib/vz/template/iso/archlinux-2022.05.01-x86_64.iso,if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
 27   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' \
 28   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
 29   -drive 'file=/dev/vms/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' \
 30   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
 31   -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
 32   -device 'virtio-net-pci,mac=F2:F7:6A:2C:57:6D,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' \
 33   -machine 'type=pc+pve0' \
 34   -vnc 0.0.0.0:69

Running the script by itself

chmod +x debug.sh && ./debug.sh

outputs

kvm: -drive file=/dev/vms/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/vms/vm-100-disk-0': No such file or directory

If I run:

kvm ./debug.sh

It outputs this and acts like it has started. But I am unable to connect.

WARNING: Image format was not specified for 'debug.sh' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
VNC server running on 127.0.0.1:5900

Trying with gdb:

gdb /usr/bin/kvm

Outputs

### gdb default output

(gdb) run debug.sh

Starting program: /usr/bin/kvm debug.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffeb6a5700 (LWP 8503)]
[New Thread 0x7fffeabb1700 (LWP 8504)]
WARNING: Image format was not specified for 'debug.sh' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
[New Thread 0x7fffe9f8e700 (LWP 8507)]

Thread 4 "kvm" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffe9f8e700 (LWP 8507)]
futex_wait_cancelable (private=0, expected=0, futex_word=0x55555671e728) at ../sysdeps/nptl/futex-internal.h:186
186     ../sysdeps/nptl/futex-internal.h: No such file or directory.

This results in the vm not starting either.
Any advice on this issue?
Thanks,
Brian

 

[fiona]

Having trouble getting a gdb session working. This is the script I have.

                                                                                                      1,1           Top
  1 #!/bin/bash
  2 /usr/bin/kvm \
  3   -id 100 \
  4   -name ArchBox \
  5   -no-shutdown \
  6   -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' \
  7   -mon 'chardev=qmp,mode=control' \
  8   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
  9   -mon 'chardev=qmp-event,mode=control' \
 10   -pidfile /var/run/qemu-server/100.pid \
 11   -smbios 'type=1,uuid=22f3c344-45bb-474c-9c9a-851648d78b3a' \
 12   -smp '4,sockets=2,cores=2,maxcpus=4' \
 13   -nodefaults \
 14   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
 15   -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' \
 16   -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep \
 17   -m 4096 \
 18   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
 19   -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \
 20   -device 'vmgenid,guid=10a11734-6c31-4de4-9a70-d4516eee52fa' \
 21   -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
 22   -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
 23   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
 24   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
 25   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:fa94959361f2' \
 26   -drive 'file=/var/lib/vz/template/iso/archlinux-2022.05.01-x86_64.iso,if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
 27   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' \
 28   -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
 29   -drive 'file=/dev/vms/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' \
 30   -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
 31   -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
 32   -device 'virtio-net-pci,mac=F2:F7:6A:2C:57:6D,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' \
 33   -machine 'type=pc+pve0' \
 34   -vnc 0.0.0.0:69

Running the script by itself

chmod +x debug.sh && ./debug.sh

outputs

kvm: -drive file=/dev/vms/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/vms/vm-100-disk-0': No such file or directory

If I run:

kvm ./debug.sh

It outputs this and acts like it has started. But I am unable to connect.

WARNING: Image format was not specified for 'debug.sh' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
VNC server running on 127.0.0.1:5900

Trying with gdb:

gdb /usr/bin/kvm

Outputs

### gdb default output

(gdb) run debug.sh

Starting program: /usr/bin/kvm debug.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffeb6a5700 (LWP 8503)]
[New Thread 0x7fffeabb1700 (LWP 8504)]
WARNING: Image format was not specified for 'debug.sh' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
[New Thread 0x7fffe9f8e700 (LWP 8507)]

Thread 4 "kvm" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffe9f8e700 (LWP 8507)]
futex_wait_cancelable (private=0, expected=0, futex_word=0x55555671e728) at ../sysdeps/nptl/futex-internal.h:186
186     ../sysdeps/nptl/futex-internal.h: No such file or directory.

This results in the vm not starting either.
Any advice on this issue?
Thanks,
Brian

You can either start the VM normally and attach gdb to the process with gdb -p <PID> (then enter continue and when the hang happens, press Ctrl+C and execute backtrace) or add gdb --args to the beginning of your script, and use simply run, and when the hang happens, same as before.