Hi Christoph,
live migration of VMs with an NVIDIA vGPU (mapped device, mdev) fails reliably between two identical nodes.
ENVIRONMENT
- 2 nodes (nodeA, nodeB) in the same cluster, identical hardware, each with 2x NVIDIA RTX PRO 6000 Blackwell (ECC on, SR-IOV with GSP mode)
- NVIDIA vGPU host driver 595.91.04 on both nodes
- Proxmox VE 9.2 (pve-manager 9.2.21), kernel 7.0.14-24-pve, pve-qemu-kvm 11.0.3-4, qemu-server 9.2.11
- Storage: Ceph RBD / NVMe pool; guest: Windows 11, guest driver 596.86
- vGPU license: Licensed
- VM config (relevant lines): bios ovmf, machine pc-q35-11.0+pve2, cpu host, numa 0, hostpci0: mapping=gpu1,mdev=nvidia-<type>,pcie=1
- Kernel log on the destination: "vGPU migration enabled with upstream V2 migration protocol"
WHAT HAPPENS
Case 1: VM 109 (1 TiB RAM, vGPU type 1537 = DC-32Q), nodeB -> nodeA, 2026-10-15
Case 2: VM 113 (8 GiB RAM, freshly created, vGPU type 1534), nodeA -> nodeB, 2026-10-16
Case 3: VM 113 again, nodeA -> nodeB, 2026-10-19, mdev type [CONFIRM: nvidia-1521 or other]. Same QEMU error and the same 76 s timing as in the earlier cases (task log below). [ADD: whether the GPU on nodeA was lost again or not]
Pre-copy of RAM and VFIO state runs normally. The failure happens in stop-and-copy. For cases 1 and 2, timings relative to the start of stop-and-copy (taken from nvidia-vgpu-mgr) are identical:
+0 s nvidia-vgpu-mgr: Plugin migration stage change pre_copy -> stop_and_copy
+25 s kernel (source): [nvidia-vgpu-vfio] Failed to get plugin status for migration state. ret: -110 status: 0x65
kernel: Failed to set stop-and-copy state -110
+50 s kernel: Invalid state transition, should be in precopy state 0x3 (-22)
NVRM Xid 45 on all channels
+76 s Proxmox task log (VM 113):
migration status error: failed - 0001:15:06.1: Failed setting device state to STOP_COPY. Recover state is ERROR. Resetting device: Connection timed out
Proxmox task log (VM 109):
ERROR: online migrate failure - unable to parse migration status 'failing' - aborting
On the destination the vGPU only logs "Start restoring vGPU state ..." with no error; it never receives complete state.
On the source for VM 113 (case 2), nvidia-vgpu-mgr also logs "Copy sysmem tracking failed, 0x2" at +76 s.
ADDITIONAL FAILURE (VM 113, case 2 only, 2026-10-16)
At the moment QEMU resets the device (10:08:49), the source node logs a fatal PCIe error on the root port (Intel 8086:0db0, AER layer: Transaction Layer), "AER: device recovery failed", "can't recover (no error_detected callback)" for all functions of the GPU, then:
NVRM: GPU 0001:15:00.0: GPU has fallen off the bus.
NVRM: A GPU crash dump has been created.
Afterwards nvidia-smi reports "Unable to determine the device handle for GPU1: 0001:15:00.0: Unknown Error" and the node needs a reboot. This did NOT happen in case 1 (no AER/Hardware Error messages).
NOTE ON THE VM 113 CONFIG
The failed migration of VM 113 in case 2 used mdev type nvidia-1534. I changed the mdev type to nvidia-1521 afterwards, so the config below differs from the one in the case 2 log.
WHAT I RULED OUT
- VM size, dirty rate, network bandwidth (an 8 GiB VM fails exactly like a 1 TiB VM)
- vGPU profile and physical GPU (1537 and 1534, both GPUs fail)
- Migration direction (both directions fail)
- Primary GPU / x-vga (enabled on VM 109, not on VM 113)
- Destination memory (plenty free, no OOM)
- Network/storage (the timeout is raised inside the vGPU driver stack)
root@nodeA:~# qm config 113
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0;ide0
cores: 4
cpu: host
efidisk0: storage1:vm-113-disk-0,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=528K
hostpci0: mapping=gpu1,mdev=nvidia-1521,pcie=1
ide0: none,media=cdrom
machine: pc-q35-11.0+pve2
memory: 8128
meta: creation-qemu=11.0.3,ctime=<redacted>
name: vm113
net0: virtio=<mac>,bridge=vmbrX,firewall=1
numa: 0
ostype: win11
scsi0: storage1:vm-113-disk-1,backup=0,discard=on,iothread=1,size=100G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=<redacted>
sockets: 1
tpmstate0: storage1:vm-113-disk-2,size=4M,version=v2.0
Task log, case 3 (VM 113, 2026-10-19):
2026-10-19 11:18:39 migrating VM which uses mapped local devices
2026-10-19 11:18:39 starting migration of VM 113 to node 'nodeB' (x.x.x.x)
2026-10-19 11:18:39 starting VM 113 on remote node 'nodeB'
2026-10-19 11:18:43 [nodeB] kvm: -device vfio-pci,host=0001:15:06.1,enable-migration=on,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: warning: vfio 0001:15:06.1: Could not enable error recovery for the device
2026-10-19 11:18:43 [nodeB] kvm: -device vfio-pci,host=0001:15:06.1,enable-migration=on,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: warning: VFIO dma-buf not supported in kernel: PCI BAR IOMMU mappings may fail
2026-10-19 11:18:43 start remote tunnel
2026-10-19 11:18:44 ssh tunnel ver 1
2026-10-19 11:18:44 starting online/live migration on unix:/run/qemu-server/113.migrate
2026-10-19 11:18:44 set migration capabilities
2026-10-19 11:18:44 migration downtime limit: 100 ms
2026-10-19 11:18:44 migration cachesize: 1.0 GiB
2026-10-19 11:18:44 set migration parameters
2026-10-19 11:18:44 start migrate command to unix:/run/qemu-server/113.migrate
2026-10-19 11:18:45 migration active, transferred 645.6 MiB of 8.0 GiB VM-state, 1000.1 MiB/s (+ 18.0 MiB VFIO-state)
2026-10-19 11:18:46 migration active, transferred 1.6 GiB of 8.0 GiB VM-state, 997.7 MiB/s (+ 37.0 MiB VFIO-state)
2026-10-19 11:18:47 migration active, transferred 2.4 GiB of 8.0 GiB VM-state, 995.3 MiB/s (+ 56.0 MiB VFIO-state)
2026-10-19 11:18:48 migration active, transferred 3.4 GiB of 8.0 GiB VM-state, 974.1 MiB/s (+ 75.0 MiB VFIO-state)
2026-10-19 11:18:49 migration active, transferred 4.3 GiB of 8.0 GiB VM-state, 940.5 MiB/s (+ 94.0 MiB VFIO-state)
2026-10-19 11:18:50 migration active, transferred 5.3 GiB of 8.0 GiB VM-state, 0.0 B/s (+ 485.0 MiB VFIO-state)
2026-10-19 11:18:51 auto-increased downtime to continue migration: 200 ms
2026-10-19 11:18:51 migration active, transferred 6.1 GiB of 8.0 GiB VM-state, 653.1 MiB/s (+ 838.0 MiB VFIO-state)
2026-10-19 11:18:51 xbzrle: send updates to 13276 pages in 12.3 MiB encoded memory, overflow 549
2026-10-19 11:20:07 migration status error: failed - 0001:15:06.1: Failed setting device state to STOP_COPY. Recover state is ERROR. Resetting device: Connection timed out
2026-10-19 11:20:07 ERROR: online migrate failure - aborting
2026-10-19 11:20:07 aborting phase 2 - cleanup resources
2026-10-19 11:20:07 migrate_cancel
2026-10-19 11:20:10 ERROR: migration finished with problems (duration 00:01:31)
TASK ERROR: migration problems
