[SOLVED] VM freezes when disconnecting USB Keyboard and Mouse.

[Steffenprox]

Hello, I have a problem when I disconnect my USB Keyboard and Mouse which i pass-through to a VM. I tested this with 2 different VMs booth have the same problem. In earlier versions of Proxmox I didn't have these problems. Today I tested with the older kernel (5.3.18-3-pve) but I also got freezes when these devices disconnect.

Actual Proxmox Versions:

proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

VM 1 Configuration:

agent: 1
balloon: 0
bootdisk: virtio0
cores: 4
cpu: host
hostpci0: 00:02.0,rombar=0
hostpci1: 00:0e.0,rombar=0
ide2: none,media=cdrom
machine: q35
memory: 8192
name: zorin3
net0: virtio=2E:74:61:B0:2F:BB,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=f6ffd6ef-0f6e-4e02-ba72-5addbec64b59
sockets: 1
startup: order=5,up=3
usb0: host=05dc:a203,usb3=1
usb1: host=0bda:0326,usb3=1
usb2: host=1b1c:1b33
usb3: host=046d:c332
vga: none
virtio0: local-lvm:vm-100-disk-0,cache=writethrough,size=32G
vmgenid: 542df39b-e008-4509-bbff-a88c74edb53f

VM2 test configuration:

agent: 1
balloon: 0
bootdisk: virtio0
cores: 4
cpu: host
hostpci0: 00:02.0,rombar=0
hostpci1: 00:0e.0,rombar=0
ide2: none,media=cdrom
machine: q35
memory: 8192
name: solus
net0: virtio=72:8C:F11:AD:40,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=688f2567-2a70-4487-b776-7138e73ad512
sockets: 1
usb2: host=1-1.1
usb3: host=1-1.3
vga: none
virtio0: local-lvm:vm-105-disk-0,cache=writethrough,size=32G
vmgenid: e712e7ab-4b5c-491b-90db-a29e6abb5784

 

[tomtomcapo]

Hello. I have the exact same problem on my PVE server. When I disconnect a serial port USB device (FTDI), the guest crash.
This problem has appeared when I upgraded Proxmox from 5 to 6.

To reproduce the bug:
1 - Start the VM.
2 - Connect the USB device and use it from the guest, at this point, all seems ok.
3 - Disconnect the USB device.
4 - The guest freezes and the VNC console is unreachable from the web gui, unlike a "generic" kernel panic from the guest.

proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-4.15: 5.4-17
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

 

[Steffenprox]

Yeah sounds like we have the same problem and i had think something of my hardware is broken.

 

[vizsla]

Cross-commenting here with my USB VM freezing issue to help raise awareness. https://forum.proxmox.com/threads/usb-drive-crashes-vm.70214/

 

[vizsla]

We had some PVE updates this morning. I won't be able to test the USB issue until this evening.

 

[Steffenprox]

I did a upgrade today. The problem is fixed for me.

pve-edk2-firmware: 2.20200531-1
pve-kernel-5.4: 6.2-3
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-helper: 6.2-3

 

[vizsla]

I did a upgrade today. The problem is fixed for me.

I’m curious, are your USB issues still resolved a day later? Mine didn’t go away after the latest upgrade.

 

[Steffenprox]

Yesterday I did test the disconnect and connect my USB devices and they work, today i get the same freezes like before. Something is curious. I add a new USB Stick to my VM an it works sometimes without freeze and sometimes not.

 

[Dark26]

Same problem here. If i unplug a usb, th vm freeze

 

[vizsla]

I updated to pve-kernel-5.4 (6.2-4). Ran my first test this morning by restarting my server with the USB flash drive plugged in. My VM recognized the flash drive. I unplugged it, the VM didn't freeze/crash, and I could see in the CentOS VM that it had been disconnected. I then plugged my next rotation USB flash drive in and CentOS showed it mount and the VM did not freeze. I'll keep trying this to see if the issue is gone or if this was a fluke.

 

[skytrooper09]

I updated to pve-kernel-5.4 (6.2-4). Ran my first test this morning by restarting my server with the USB flash drive plugged in. My VM recognized the flash drive. I unplugged it, the VM didn't freeze/crash, and I could see in the CentOS VM that it had been disconnected. I then plugged my next rotation USB flash drive in and CentOS showed it mount and the VM did not freeze. I'll keep trying this to see if the issue is gone or if this was a fluke.

Any verdict on this? Did it end up being a fluke? My system doesn't seem to care if it boots up with the USB attached - the VM will still freeze when it's unplugged.

 

[zaxmyth]

Any verdict on this? Did it end up being a fluke? My system doesn't seem to care if it boots up with the USB attached - the VM will still freeze when it's unplugged.

This definitely doesn't seem to be a fluke on my system. I think we may still be waiting for Proxmox to confirm this is an issue and that an investigation or fix is planned or in the works.

 

[dcsapak]

just to clarify, it seems this is the same issue as here https://bugzilla.proxmox.com/show_bug.cgi?id=2781
we were not able to reproduce this issue here (yet), and also the provided logs in the bug report were inconclusive
so if anyone has another sets of logs (or more insight/trigger conditions) please answer on the bug report with that info

 

[alexsahka]

VM Synology (customized Debian), attached USB HDD for Backup, if USB HDD connected or disconnected (after eject) 7 - 8 times out of 10 VM becomes frozen and the only way to reanimate VM is to forcing VM STOP and then Start VM again.
I have tried 4 different manufactured USB HDDs, and VM still freezes.

 

[leesteken]

It does not happen all the time, but when it does, it does not matter whether:
- I physically unplug a USB device or use echo {portid} >/sys/bus/usb/drivers/usb/unbind, when using usb0: host=<portid>;
- I use usb0: host=<portid> or I use device_add usb-host,bus=xhci.0,hostbus=.. and device_del ... using pvesh;
- I use Ryzen X470 and others use an Intel platform, which suggest it is not a hardware specific issue.

The first two points make that one can test for this problem using a software script, without any physical actions.

It appears that Proxmox 6.1 did not have this issue, according to other posts on this forum.
I did not find any report of this problem that did not mention using a recent Proxmox 6.2
I seem to have this issue then using onboard physical USB 3.x ports more often than using USB 2.0 ports. (I also noticed that using 3.0 causes keystrokes to be lost more often than 2.0.)

Maybe it happens on consumer hardware, and not on server platforms (really not sure about this)? Maybe it depends of the order of starting VMs?
Maybe it depends on the amount of memory used in the system. Maybe it depends on some uninitialized variable, somewhere.
It does appear to be a hit or miss issue, happening to multiple users (somewhat consistently) but not all. I assume that not many people use USB passthrough.

The good news is that it looks like a test can be scripted and repeated many times over. Unfortunately, it is not clear what the required conditions are for this to trigger.

 

[Steffenprox]

I have tested today again and noticed when i disconnect only 1 device the sound output stutters for 1 second and after that the VM is fine but when i disconnect 2 devices at same time the VM freezes. Seems like its to much for the VM to handle the action, the server is fine.

proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-2
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

VM Configuration:

agent: 1
balloon: 4096
bootdisk: scsi0
cores: 4
cpu: host
hostpci0: 00:02
hostpci1: 00:0e.0
ide2: none,media=cdrom
machine: q35
memory: 8192
name: manjaro3
net0: virtio=5E:30:36:64:5D:04,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local-lvm:vm-108-disk-0,cache=unsafe,discard=on,size=32G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=e445507f-22d3-4e2e-bf68-66c5a0ee5c2c
sockets: 1
startup: order=5,up=3
usb0: host=1-3
usb1: host=1-2
vga: none
vmgenid: 0f29ab91-6387-499f-93b8-210b45dd4144

Serverlog:

Aug 11 01:13:16 homeserver kernel: [  812.341260] usb 1-3: new low-speed USB device number 12 using xhci_hcd
Aug 11 01:13:17 homeserver kernel: [  812.512827] usb 1-3: New USB device found, idVendor=04d9, idProduct=1603, bcdDevice= 3.13
Aug 11 01:13:17 homeserver kernel: [  812.512831] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Aug 11 01:13:17 homeserver kernel: [  812.512833] usb 1-3: Product: USB Keyboard
Aug 11 01:13:17 homeserver kernel: [  812.512834] usb 1-3: Manufacturer:
Aug 11 01:13:17 homeserver kernel: [  812.531888] input:   USB Keyboard as /devices/pci0000:00/0000:00:15.0/usb1/1-3/1-3:1.0/0003:04D9:1603.0012/input/input25
Aug 11 01:13:17 homeserver kernel: [  812.593474] hid-generic 0003:04D9:1603.0012: input,hidraw0: USB HID v1.10 Keyboard [  USB Keyboard] on usb-0000:00:15.0-3/input0
Aug 11 01:13:17 homeserver kernel: [  812.608011] input:   USB Keyboard System Control as /devices/pci0000:00/0000:00:15.0/usb1/1-3/1-3:1.1/0003:04D9:1603.0013/input/input26
Aug 11 01:13:17 homeserver kernel: [  812.665374] input:   USB Keyboard Consumer Control as /devices/pci0000:00/0000:00:15.0/usb1/1-3/1-3:1.1/0003:04D9:1603.0013/input/input27
Aug 11 01:13:17 homeserver kernel: [  812.665461] hid-generic 0003:04D9:1603.0013: input,hidraw1: USB HID v1.10 Device [  USB Keyboard] on usb-0000:00:15.0-3/input1
Aug 11 01:13:18 homeserver kernel: [  813.637371] usb 1-3: reset low-speed USB device number 12 using xhci_hcd
Aug 11 01:13:23 homeserver kernel: [  819.240737] usb 1-2: new low-speed USB device number 13 using xhci_hcd
Aug 11 01:13:23 homeserver kernel: [  819.393573] usb 1-2: New USB device found, idVendor=093a, idProduct=2510, bcdDevice= 1.00
Aug 11 01:13:23 homeserver kernel: [  819.393576] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Aug 11 01:13:23 homeserver kernel: [  819.393578] usb 1-2: Product: USB OPTICAL MOUSE
Aug 11 01:13:23 homeserver kernel: [  819.393580] usb 1-2: Manufacturer: PIXART
Aug 11 01:13:23 homeserver kernel: [  819.400517] input: PIXART USB OPTICAL MOUSE as /devices/pci0000:00/0000:00:15.0/usb1/1-2/1-2:1.0/0003:093A:2510.0014/input/input28
Aug 11 01:13:23 homeserver kernel: [  819.401020] hid-generic 0003:093A:2510.0014: input,hidraw0: USB HID v1.10 Mouse [PIXART USB OPTICAL MOUSE] on usb-0000:00:15.0-2/input0
Aug 11 01:13:26 homeserver kernel: [  821.712825] usb 1-2: reset low-speed USB device number 13 using xhci_hcd
Aug 11 01:13:30 homeserver kernel: [  826.334273] usb 1-3: USB disconnect, device number 12
Aug 11 01:13:30 homeserver kernel: [  826.480340] usb 1-2: USB disconnect, device number 13
Aug 11 01:14:03 homeserver kernel: [  858.674411] vmbr0: port 7(tap108i0) entered disabled state
Aug 11 01:14:03 homeserver kernel: [  858.677715] vmbr0: port 7(tap108i0) entered disabled state

 

[Steffenprox]

Today i upgraded pve-qemu-kvm to version 5.1.0-1 and did try to get the bug several times but the VM didn't freeze.

Maybe is the bug fixed, others need to try.

The changelog for this qemu version don't have any changes. https://wiki.qemu.org/ChangeLog/5.1#USB

proxmox-ve: 6.2-1 (running kernel: 5.4.60-1-pve) pve-manager: 6.2-11 (running version: 6.2-11/22fb4983) pve-kernel-5.4: 6.2-6 pve-kernel-helper: 6.2-6 pve-kernel-5.4.60-1-pve: 5.4.60-2 pve-kernel-5.4.55-1-pve: 5.4.55-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph-fuse: 12.2.11+dfsg1-2.1+b1 corosync: 3.0.4-pve1 criu: 3.11-3 glusterfs-client: 5.5-3 ifupdown: 0.8.35+pve1 ksm-control-daemon: 1.3-1 libjs-extjs: 6.0.1-10 libknet1: 1.16-pve1 libproxmox-acme-perl: 1.0.5 libpve-access-control: 6.1-2 libpve-apiclient-perl: 3.0-3 libpve-common-perl: 6.2-2 libpve-guest-common-perl: 3.1-3 libpve-http-server-perl: 3.0-6 libpve-storage-perl: 6.2-6 libqb0: 1.0.5-1 libspice-server1: 0.14.2-4~pve6+1 lvm2: 2.03.02-pve4 lxc-pve: 4.0.3-1 lxcfs: 4.0.3-pve3 novnc-pve: 1.1.0-1 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.2-12 pve-cluster: 6.1-8 pve-container: 3.2-1 pve-docs: 6.2-5 pve-edk2-firmware: 2.20200531-1 pve-firewall: 4.1-2 pve-firmware: 3.1-3 pve-ha-manager: 3.1-1 pve-i18n: 2.2-1 pve-qemu-kvm: 5.1.0-1 pve-xtermjs: 4.7.0-2 qemu-server: 6.2-14 smartmontools: 7.1-pve2 spiceterm: 3.1-1 vncterm: 1.6-2 zfsutils-linux: 0.8.4-pve1

 

[chrzr]

Today i upgraded pve-qemu-kvm to version 5.1.0-1 and did try to get the bug several times but the VM didn't freeze.

Maybe is the bug fixed, others need to try.

The changelog for this qemu version don't have any changes. https://wiki.qemu.org/ChangeLog/5.1#USB

proxmox-ve: 6.2-1 (running kernel: 5.4.60-1-pve) pve-manager: 6.2-11 (running version: 6.2-11/22fb4983) pve-kernel-5.4: 6.2-6 pve-kernel-helper: 6.2-6 pve-kernel-5.4.60-1-pve: 5.4.60-2 pve-kernel-5.4.55-1-pve: 5.4.55-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph-fuse: 12.2.11+dfsg1-2.1+b1 corosync: 3.0.4-pve1 criu: 3.11-3 glusterfs-client: 5.5-3 ifupdown: 0.8.35+pve1 ksm-control-daemon: 1.3-1 libjs-extjs: 6.0.1-10 libknet1: 1.16-pve1 libproxmox-acme-perl: 1.0.5 libpve-access-control: 6.1-2 libpve-apiclient-perl: 3.0-3 libpve-common-perl: 6.2-2 libpve-guest-common-perl: 3.1-3 libpve-http-server-perl: 3.0-6 libpve-storage-perl: 6.2-6 libqb0: 1.0.5-1 libspice-server1: 0.14.2-4~pve6+1 lvm2: 2.03.02-pve4 lxc-pve: 4.0.3-1 lxcfs: 4.0.3-pve3 novnc-pve: 1.1.0-1 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.2-12 pve-cluster: 6.1-8 pve-container: 3.2-1 pve-docs: 6.2-5 pve-edk2-firmware: 2.20200531-1 pve-firewall: 4.1-2 pve-firmware: 3.1-3 pve-ha-manager: 3.1-1 pve-i18n: 2.2-1 pve-qemu-kvm: 5.1.0-1 pve-xtermjs: 4.7.0-2 qemu-server: 6.2-14 smartmontools: 7.1-pve2 spiceterm: 3.1-1 vncterm: 1.6-2 zfsutils-linux: 0.8.4-pve1

I can confirm that updating the pve-qemu-kvm package seems to have resolved the problem for me!
Since updating yesterday, I was able to remove several devices multiple times which are passed through to two VMs without any locking up. Before there was about a 50% chance of the VM locking.