Hi guys, I would like to show a small project and hear what you think: qdev-host. It adds PCIe devices to running VMs of a stock PVE 9 node and takes them out again, from the VM's Hardware panel or from the shell. Nothing in QEMU or Proxmox is patched or rebuilt. There are two kinds of devices:
The first plugin, tb-vga, is a hot-pluggable display: a Bochs-compatible PCIe VGA endpoint that the stock Linux bochs-drm driver binds without changes. Its picture does not stay inside qdev-host. It is exported over QEMU's own D-Bus display protocol (org.qemu.Display1), so a client of "-display dbus" can show it: the display probe of my WebRTC console, QSM Direct (thread 187042), receives its frames unchanged. It can also drive a real monitor of the host through DRM/KMS, with zero-copy scanout where the GPU can import the memory.
There is also a VirGL 3D GPU, the built-in plugin "virgl-gpu". It does not need a plugin of mine: pve-qemu-kvm already ships QEMU's own out-of-process virtio-gpu, /usr/lib/kvm/vhost-user-gpu, and qdev-host runs it per VM as a systemd service on the render node you choose, so VirGL rendering happens outside the QEMU process. A Linux guest sees a normal virtio-gpu with virgl, and with QSM Direct's VirGL display the QSM console shows it. It is on the QEMU command line from the next VM start and is not hot-plugged, because QEMU's D-Bus display only exports consoles that exist when QEMU starts. It needs shared guest memory: if the VM has neither hugepages nor a virtiofs device, qdev-host adds a memfd memory backend of the VM's size to args (save the GPU again after you change the VM's memory; the hookscript refuses a start with a stale size and says so).
The TB bus. qdev-host gives a VM up to eight hot-plug PCIe root ports (four by default), and its own devices go there too. PVE's own PCI Device, Network Device and USB Device dialogs get a Bus field, Standard or TB bus. On the TB bus a passed-through device, a NIC (virtio, e1000e, vmxnet3) or a QEMU USB controller goes into the running VM and comes out again, and USB devices of the node attach to that controller and detach. The ports and the devices present at VM start come from a -readconfig file that the hookscript writes at every start, so the VM's args change only once; an ACPI table built per VM can give the ports the Thunderbolt properties of the qemu-thunderbolt fork.
macOS took the most work. Its PCI driver waits for a port's own PCIe hot-plug interrupts, while q35 hands bridge hot-plug to ACPI since QEMU 6.1, so a macOS VM is switched to native PCIe hot-plug. And macOS releases a device only through the attention-button sequence of a slot that has a power controller, while it switches such a slot on only when the device's link is down at the hot-plug event; QEMU reports the link up at once. So every hot-plug into a macOS VM starts with a short handshake: a throwaway device with its link down makes macOS switch the slot on, it leaves again from the host side, and the real device arrives in a powered slot. Devices then arrive in the running macOS in about 15 s and leave in about 7 s, as often as you like.
On PVE it is two packages:
qdevctl add 109 tbvga0 --plugin tb-vga --args vram=16
qdevctl add 109 tbxhci0 --plugin tb-xhci
qdevctl add 109 tbusb0 --plugin tb-usb --args host=1a86:7523 --bus tbxhci0
qdevctl stop 109 tbxhci0 (pulls the controller out; qdevctl start puts it back)
qdevctl list
qdevctl remove 109 tbvga0
How it fits into stock PVE: qm for config changes, the node's own PVE::QemuServer::Monitor for device_add, device_del and query-pci, the standard hookscript to start the device processes with the VM, and systemd for the device processes. The TB bus goes into the VM's args (a -readconfig file and an ACPI table), so the first time it appears after one VM restart. The panel talks to a small HTTPS service on the node, which checks the browser's PVE ticket against the node's own /access/ticket. Because the operations edit args, only root@pam may use it by default. The page entry goes through pve-ui-overlays (separate thread), so it coexists with QSM Direct and other add-ons. On a node behind QSM's TCP 8006 gateway the panel reaches its service through port 8006 as well.
Measured overhead of vfio-user against the same device inside QEMU: a posted MMIO write (doorbell) costs 4.8-5.0 µs instead of 3.1, a register read 22-26 µs instead of 3.3, access to an mmap-ed BAR has no exit at all. For a display that does not matter: a few register writes per mode change, and the framebuffer is shared memory. For a fast device the rule is doorbells as posted writes and queues in guest memory.
Install, from the same signed APT repository as QSM Direct (its README has the key and the pin file, which already lists qdev-host, qdev-host-pve and pve-ui-overlays):
apt install qdev-host qdev-host-pve
Requirements: a q35 VM, and a directory storage with "snippets" content for the hookscript (usually local). qdev-host refuses, before changing anything, a VM whose config has settings the node's qemu-server cannot read, because qm set would drop them.
Verified so far, on PVE 9.2.21 with pve-qemu-kvm 11.0.3 (qdev-host 0.6.2):
- Debian 11 and 13 guests, from the Hardware panel and from the shell: tb-vga added, edited, hot-unplugged, replugged and removed, with bochs-drm binding in the guest; its frames reach QSM Direct's display probe; on a laptop, guest frames go to the real panel at 4K.
Not yet: Windows guests; a 3D application in a guest on virgl-gpu; the full QSM console on a tb-vga device end to end, with sound; a VM on another cluster node than the one serving the page; several devices per process. Known limit: with QEMU 11 a PCI device cannot be hot-plugged while a vfio-user device is attached (QEMU aborts in its legacy VFIO container code), so qdev-host refuses that and the device comes at the next VM start.
What I would like to hear: which devices would you want as plugins, and which ones would you hot-plug into a macOS or Windows guest? And does the hookscript plus args approach work with your setups, for example with your own hookscripts (qdev-host chains an existing one) or cluster nodes?
Source, docs and measurements: https://github.com/rdmitry0911/qdev-host
APT repository: https://rdmitry0911.github.io/qsm-rd/
Project sponsor: Takt (https://taktcycles.com/).
Thank you, Dmitry
- emulated devices that run in their own process on the host and are plugged into the VM through QEMU's vfio-user-pci, which the stock pve-qemu-kvm of PVE 9.2 already has;
- QEMU's own devices on a "TB bus" of hot-plug ports: PCI pass-through, network cards, and a USB controller with USB devices of the node on it.
The first plugin, tb-vga, is a hot-pluggable display: a Bochs-compatible PCIe VGA endpoint that the stock Linux bochs-drm driver binds without changes. Its picture does not stay inside qdev-host. It is exported over QEMU's own D-Bus display protocol (org.qemu.Display1), so a client of "-display dbus" can show it: the display probe of my WebRTC console, QSM Direct (thread 187042), receives its frames unchanged. It can also drive a real monitor of the host through DRM/KMS, with zero-copy scanout where the GPU can import the memory.
There is also a VirGL 3D GPU, the built-in plugin "virgl-gpu". It does not need a plugin of mine: pve-qemu-kvm already ships QEMU's own out-of-process virtio-gpu, /usr/lib/kvm/vhost-user-gpu, and qdev-host runs it per VM as a systemd service on the render node you choose, so VirGL rendering happens outside the QEMU process. A Linux guest sees a normal virtio-gpu with virgl, and with QSM Direct's VirGL display the QSM console shows it. It is on the QEMU command line from the next VM start and is not hot-plugged, because QEMU's D-Bus display only exports consoles that exist when QEMU starts. It needs shared guest memory: if the VM has neither hugepages nor a virtiofs device, qdev-host adds a memfd memory backend of the VM's size to args (save the GPU again after you change the VM's memory; the hookscript refuses a start with a stale size and says so).
The TB bus. qdev-host gives a VM up to eight hot-plug PCIe root ports (four by default), and its own devices go there too. PVE's own PCI Device, Network Device and USB Device dialogs get a Bus field, Standard or TB bus. On the TB bus a passed-through device, a NIC (virtio, e1000e, vmxnet3) or a QEMU USB controller goes into the running VM and comes out again, and USB devices of the node attach to that controller and detach. The ports and the devices present at VM start come from a -readconfig file that the hookscript writes at every start, so the VM's args change only once; an ACPI table built per VM can give the ports the Thunderbolt properties of the qemu-thunderbolt fork.
macOS took the most work. Its PCI driver waits for a port's own PCIe hot-plug interrupts, while q35 hands bridge hot-plug to ACPI since QEMU 6.1, so a macOS VM is switched to native PCIe hot-plug. And macOS releases a device only through the attention-button sequence of a slot that has a power controller, while it switches such a slot on only when the device's link is down at the hot-plug event; QEMU reports the link up at once. So every hot-plug into a macOS VM starts with a short handshake: a throwaway device with its link down makes macOS switch the slot on, it leaves again from the host side, and the real device arrives in a powered slot. Devices then arrive in the running macOS in about 15 s and leave in about 7 s, as often as you like.
On PVE it is two packages:
- qdev-host: the device host, the plugins and a private copy of libvfio-user
- qdev-host-pve: the qdevctl command, a hookscript, systemd units, and the Hardware panel part
qdevctl add 109 tbvga0 --plugin tb-vga --args vram=16
qdevctl add 109 tbxhci0 --plugin tb-xhci
qdevctl add 109 tbusb0 --plugin tb-usb --args host=1a86:7523 --bus tbxhci0
qdevctl stop 109 tbxhci0 (pulls the controller out; qdevctl start puts it back)
qdevctl list
qdevctl remove 109 tbvga0
How it fits into stock PVE: qm for config changes, the node's own PVE::QemuServer::Monitor for device_add, device_del and query-pci, the standard hookscript to start the device processes with the VM, and systemd for the device processes. The TB bus goes into the VM's args (a -readconfig file and an ACPI table), so the first time it appears after one VM restart. The panel talks to a small HTTPS service on the node, which checks the browser's PVE ticket against the node's own /access/ticket. Because the operations edit args, only root@pam may use it by default. The page entry goes through pve-ui-overlays (separate thread), so it coexists with QSM Direct and other add-ons. On a node behind QSM's TCP 8006 gateway the panel reaches its service through port 8006 as well.
Measured overhead of vfio-user against the same device inside QEMU: a posted MMIO write (doorbell) costs 4.8-5.0 µs instead of 3.1, a register read 22-26 µs instead of 3.3, access to an mmap-ed BAR has no exit at all. For a display that does not matter: a few register writes per mode change, and the framebuffer is shared memory. For a fast device the rule is doorbells as posted writes and queues in guest memory.
Install, from the same signed APT repository as QSM Direct (its README has the key and the pin file, which already lists qdev-host, qdev-host-pve and pve-ui-overlays):
apt install qdev-host qdev-host-pve
Requirements: a q35 VM, and a directory storage with "snippets" content for the hookscript (usually local). qdev-host refuses, before changing anything, a VM whose config has settings the node's qemu-server cannot read, because qm set would drop them.
Verified so far, on PVE 9.2.21 with pve-qemu-kvm 11.0.3 (qdev-host 0.6.2):
- Debian 11 and 13 guests, from the Hardware panel and from the shell: tb-vga added, edited, hot-unplugged, replugged and removed, with bochs-drm binding in the guest; its frames reach QSM Direct's display probe; on a laptop, guest frames go to the real panel at 4K.
- virgl-gpu on an AMD Raphael render node: the Debian 13 guest found virtio-vga with +virgl and its capability sets, and the QSM console showed it.
- TB bus with Debian 13: virtio and e1000e NICs and a USB controller with a USB serial adapter of the node, also through PVE's own dialogs, and on a second node an AMD USB 3 controller passed through; at the VM start, hot-plugged and removed.
- macOS 26.1 on OpenCore: a USB controller with a USB disk on it (mounted, a file written, ejected, the controller pulled out and put back, the file still there), a virtio NIC, a second USB controller and the passed-through AMD controller, all at the same time, each pulled out and put back more than once, also after a VM start with the devices present.
Not yet: Windows guests; a 3D application in a guest on virgl-gpu; the full QSM console on a tb-vga device end to end, with sound; a VM on another cluster node than the one serving the page; several devices per process. Known limit: with QEMU 11 a PCI device cannot be hot-plugged while a vfio-user device is attached (QEMU aborts in its legacy VFIO container code), so qdev-host refuses that and the device comes at the next VM start.
What I would like to hear: which devices would you want as plugins, and which ones would you hot-plug into a macOS or Windows guest? And does the hookscript plus args approach work with your setups, for example with your own hookscripts (qdev-host chains an existing one) or cluster nodes?
Source, docs and measurements: https://github.com/rdmitry0911/qdev-host
APT repository: https://rdmitry0911.github.io/qsm-rd/
Project sponsor: Takt (https://taktcycles.com/).
Thank you, Dmitry