Proxmox live migration fail from PVE v8.4.11 to v9.0.5

M

m7medibrahem1

Member
May 23, 2023
1
0
6
i have a proxmox cluster (v8.4.11) and have running VMs , i migrated the running vms on 1 node to the other nodes in cluster to avoid downtime and upgraded it to 9.0.5 , the upgrade went successfully but when i tried to migrate the vms back to the upgraded node to upgrade the other nodes
the migrate failed , actually the vm migrated with the disk but not started (the live migration failed) i got the following error

2025-08-19 12:56:36 start remote tunnel
2025-08-19 12:56:38 ssh tunnel ver 1
2025-08-19 12:56:38 starting online/live migration on unix:/run/qemu-server/1234.migrate
2025-08-19 12:56:38 set migration capabilities
2025-08-19 12:56:38 migration downtime limit: 100 ms
2025-08-19 12:56:38 migration cachesize: 256.0 MiB
2025-08-19 12:56:38 set migration parameters
2025-08-19 12:56:38 start migrate command to unix:/run/qemu-server/1234.migrate
2025-08-19 12:56:39 migration active, transferred 113.3 MiB of 2.0 GiB VM-state, 130.8 MiB/s
2025-08-19 12:56:40 migration active, transferred 215.9 MiB of 2.0 GiB VM-state, 1.4 GiB/s
2025-08-19 12:56:41 average migration speed: 688.3 MiB/s - downtime 24 ms
2025-08-19 12:56:41 migration completed, transferred 246.2 MiB VM-state
2025-08-19 12:56:41 migration status: completed
2025-08-19 12:56:41 ERROR: tunnel replied 'ERR: resume failed - VM 1234 not running' to command 'resume 1234'
2025-08-19 12:56:45 ERROR: migration finished with problems (duration 00:00:12)
TASK ERROR: migration problems
 
I've the same issue with a test VM running OPNSense.

journalctl -e on the target node (upgraded to 9) got


Code: 
Sep 03 23:15:26 cr systemd[1]: Started 125.scope.
Sep 03 23:15:26 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 03 23:15:26 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 03 23:15:26 cr kernel: tap125i0: entered promiscuous mode
Sep 03 23:15:26 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-interface:lib/interface/state/mtu)
Sep 03 23:15:26 cr ovs-vsctl[5077]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap125i0
Sep 03 23:15:26 cr ovs-vsctl[5077]: ovs|00002|db_ctl_base|ERR|no port named tap125i0
Sep 03 23:15:26 cr ovs-vsctl[5079]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln125i0
Sep 03 23:15:26 cr ovs-vsctl[5079]: ovs|00002|db_ctl_base|ERR|no port named fwln125i0
Sep 03 23:15:26 cr ovs-vsctl[5080]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- add-port vmbr1 tap125i0 -- set Interface tap125i0 mtu_request=9000
Sep 03 23:15:26 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 03 23:15:26 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 03 23:15:27 cr kernel: tap125i1: entered promiscuous mode
Sep 03 23:15:27 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-interface:lib/interface/state/mtu)
Sep 03 23:15:27 cr ovs-vsctl[5106]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap125i1
Sep 03 23:15:27 cr ovs-vsctl[5106]: ovs|00002|db_ctl_base|ERR|no port named tap125i1
Sep 03 23:15:27 cr ovs-vsctl[5108]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln125i1
Sep 03 23:15:27 cr ovs-vsctl[5108]: ovs|00002|db_ctl_base|ERR|no port named fwln125i1
Sep 03 23:15:27 cr ovs-vsctl[5109]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- add-port vmbr1 tap125i1 -- set Interface tap125i1 mtu_request=9000
Sep 03 23:15:27 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 03 23:15:27 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 03 23:15:27 cr kernel: tap125i2: entered promiscuous mode
Sep 03 23:15:27 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-interface:lib/interface/state/mtu)
Sep 03 23:15:27 cr ovs-vsctl[5129]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap125i2
Sep 03 23:15:27 cr ovs-vsctl[5129]: ovs|00002|db_ctl_base|ERR|no port named tap125i2
Sep 03 23:15:27 cr ovs-vsctl[5131]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln125i2
Sep 03 23:15:27 cr ovs-vsctl[5131]: ovs|00002|db_ctl_base|ERR|no port named fwln125i2
Sep 03 23:15:27 cr ovs-vsctl[5132]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- add-port vmbr1 tap125i2 tag=4091 -- set Interface tap125i2 mtu_request=9000
Sep 03 23:15:27 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 03 23:15:27 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 03 23:15:28 cr kernel: tap125i3: entered promiscuous mode
Sep 03 23:15:28 cr ovs-vsctl[5151]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap125i3
Sep 03 23:15:28 cr ovs-vsctl[5151]: ovs|00002|db_ctl_base|ERR|no port named tap125i3
Sep 03 23:15:28 cr ovs-vsctl[5153]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln125i3
Sep 03 23:15:28 cr ovs-vsctl[5153]: ovs|00002|db_ctl_base|ERR|no port named fwln125i3
Sep 03 23:15:28 cr ovs-vsctl[5154]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- add-port vmbr0 tap125i3 -- set Interface tap125i3 mtu_request=1500
Sep 03 23:15:28 cr pve-ha-lrm[5049]: VM 125 started with PID 5061.
Sep 03 23:15:28 cr pve-ha-lrm[5048]: <root@pam> end task UPID:cr:000013B9:00006EBD:68B8AFEE:qmstart:125:root@pam: OK
Sep 03 23:15:28 cr pve-ha-lrm[5048]: service status vm:125 started
Sep 03 23:15:31 cr pvedaemon[2129]: VM 125 qmp command failed - VM 125 qmp command 'guest-ping' failed - got timeout
Sep 03 23:15:51 cr pvedaemon[2127]: VM 125 qmp command failed - VM 125 qmp command 'guest-ping' failed - got timeout
 
Hi,
@m7medibrahem1 your issue is probably the same as here: https://forum.proxmox.com/threads/qemu-virtio-issues-after-upgrade-to-9.169625/

@iopsthecloud do you also see a migration error? From your journal output it seems like the VM was successfully started, just that commands to the guest agent timed out afterwards. Still, please also see the thread linked above to check if it might be the same issue. Otherwise, please provide more information like the VM configuration qm config 125 as well as the full migration task log.
 
Hi @fiona

From my test, the vm was "restarted" so the live migration failed (resume).

I tried to do another test with a linux/debian guest, and I've got a more explicit message, with the VM not starting on the target host.

logs from src (pve8):
Code: 
Sep 05 18:31:29 ti pvedaemon[2775055]: <X@pve> starting task UPID:ti:002C2017:E5210A72:68BB1061:qmigrate:115:X@pve:
Sep 05 18:31:29 ti pvedaemon[2769699]: worker exit
Sep 05 18:31:29 ti pvedaemon[2010]: worker 2769699 finished
Sep 05 18:31:29 ti pvedaemon[2010]: starting 1 worker(s)
Sep 05 18:31:29 ti pvedaemon[2010]: worker 2891804 started
Sep 05 18:31:29 ti pmxcfs[430240]: [status] notice: received log
Sep 05 18:31:31 ti pmxcfs[430240]: [status] notice: received log
Sep 05 18:31:36 ti ovs-vsctl[2891860]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i0
Sep 05 18:31:36 ti ovs-vsctl[2891860]: ovs|00002|db_ctl_base|ERR|no port named fwln115i0
Sep 05 18:31:36 ti ovs-vsctl[2891861]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i0
Sep 05 18:31:36 ti bgpd[1446]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF tap115i0 in VRF 0
Sep 05 18:31:36 ti kernel: tap115i1: left allmulticast mode
Sep 05 18:31:36 ti kernel: fwbr115i1: port 1(tap115i1) entered disabled state
Sep 05 18:31:36 ti zebra[1439]: [WPPMZ-G9797] if_zebra_speed_update: fwbr115i1 old speed: 10000 new speed: 4294967295
Sep 05 18:31:36 ti ovs-vsctl[2891874]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115o1
Sep 05 18:31:36 ti kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
Sep 05 18:31:36 ti kernel: fwln115o1 (unregistering): left allmulticast mode
Sep 05 18:31:36 ti kernel: fwln115o1 (unregistering): left promiscuous mode
Sep 05 18:31:36 ti kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
Sep 05 18:31:36 ti bgpd[1446]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF fwln115o1 in VRF 0
Sep 05 18:31:36 ti ovs-vsctl[2891878]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i1
Sep 05 18:31:36 ti ovs-vsctl[2891878]: ovs|00002|db_ctl_base|ERR|no port named fwln115i1
Sep 05 18:31:36 ti bgpd[1446]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF fwbr115i1 in VRF 0
Sep 05 18:31:36 ti ovs-vsctl[2891882]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i1
Sep 05 18:31:36 ti ovs-vsctl[2891882]: ovs|00002|db_ctl_base|ERR|no port named tap115i1
Sep 05 18:31:36 ti bgpd[1446]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF tap115i1 in VRF 0
Sep 05 18:31:36 ti qmeventd[2842255]: read: Connection reset by peer
Sep 05 18:31:37 ti systemd[1]: 115.scope: Deactivated successfully.
Sep 05 18:31:37 ti systemd[1]: 115.scope: Consumed 13.808s CPU time.
Sep 05 18:31:37 ti systemd-logind[1327]: Session 644096 logged out. Waiting for processes to exit.
Sep 05 18:31:37 ti systemd[1]: session-644096.scope: Deactivated successfully.
Sep 05 18:31:37 ti systemd[1]: session-644096.scope: Consumed 1.484s CPU time.
Sep 05 18:31:37 ti systemd-logind[1327]: Removed session 644096.
Sep 05 18:31:37 ti pmxcfs[430240]: [status] notice: received log
Sep 05 18:31:37 ti pvedaemon[2891799]: migration problems
Sep 05 18:31:37 ti pvedaemon[2775055]: <X@pve> end task UPID:ti:002C2017:E5210A72:68BB1061:qmigrate:115:X@pve: migration problems
Sep 05 18:32:01 ti pmxcfs[430240]: [dcdb] notice: data verification successful

logs from target (pve9):
Code: 
Sep 05 18:31:29 cr systemd-logind[1645]: New session 355 of user root.
Sep 05 18:31:29 cr systemd[1]: Started session-355.scope - Session 355 of User root.
Sep 05 18:31:29 cr qm[1514653]: <root@pam> starting task UPID:cr:00171CA0:00EE1BF1:68BB1061:qmstart:115:root@pam:
Sep 05 18:31:29 cr qm[1514656]: start VM 115: UPID:cr:00171CA0:00EE1BF1:68BB1061:qmstart:115:root@pam:
Sep 05 18:31:30 cr systemd[1]: Started 115.scope.
Sep 05 18:31:30 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 05 18:31:30 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 05 18:31:30 cr kernel: tap115i0: entered promiscuous mode
Sep 05 18:31:30 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-interface:lib/interface/state/mtu)
Sep 05 18:31:30 cr ovs-vsctl[1514684]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i0
Sep 05 18:31:30 cr ovs-vsctl[1514684]: ovs|00002|db_ctl_base|ERR|no port named tap115i0
Sep 05 18:31:30 cr ovs-vsctl[1514686]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i0
Sep 05 18:31:30 cr ovs-vsctl[1514686]: ovs|00002|db_ctl_base|ERR|no port named fwln115i0
Sep 05 18:31:30 cr ovs-vsctl[1514687]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- add-port vmbr1 tap115i0 tag=4 -- set Interface tap115i0 mtu_request=9000
Sep 05 18:31:30 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 05 18:31:30 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 05 18:31:31 cr kernel: tap115i1: entered promiscuous mode
Sep 05 18:31:31 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-interface:lib/interface/state/mtu)
Sep 05 18:31:31 cr ovs-vsctl[1514708]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i1
Sep 05 18:31:31 cr ovs-vsctl[1514708]: ovs|00002|db_ctl_base|ERR|no port named tap115i1
Sep 05 18:31:31 cr ovs-vsctl[1514710]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i1
Sep 05 18:31:31 cr ovs-vsctl[1514710]: ovs|00002|db_ctl_base|ERR|no port named fwln115i1
Sep 05 18:31:31 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 05 18:31:31 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 05 18:31:31 cr kernel: fwbr115i1: port 1(tap115i1) entered blocking state
Sep 05 18:31:31 cr kernel: fwbr115i1: port 1(tap115i1) entered disabled state
Sep 05 18:31:31 cr kernel: tap115i1: entered allmulticast mode
Sep 05 18:31:31 cr kernel: fwbr115i1: port 1(tap115i1) entered blocking state
Sep 05 18:31:31 cr kernel: fwbr115i1: port 1(tap115i1) entered forwarding state
Sep 05 18:31:31 cr ovs-vsctl[1514720]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- add-port vmbr1 fwln115o1 -- set Interface fwln115o1 mtu_request=9000 -- set Interface fwln115o1 type=internal
Sep 05 18:31:31 cr zebra[1761]: libyang Unsatisfied pattern - "" does not conform to "[0-9a-fA-F]{2}(:[0-9a-fA-F]{2}){5}". (/frr-interface:lib/interface/state/phy-address)
Sep 05 18:31:31 cr kernel: fwln115o1: entered promiscuous mode
Sep 05 18:31:31 cr zebra[1761]: libyang Invalid boolean value "". (/frr-vrf:lib/vrf/state/active)
Sep 05 18:31:31 cr zebra[1761]: libyang Invalid type uint32 empty value. (/frr-vrf:lib/vrf/state/id)
Sep 05 18:31:31 cr kernel: fwbr115i1: port 2(fwln115o1) entered blocking state
Sep 05 18:31:31 cr kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
Sep 05 18:31:31 cr kernel: fwln115o1: entered allmulticast mode
Sep 05 18:31:31 cr kernel: fwbr115i1: port 2(fwln115o1) entered blocking state
Sep 05 18:31:31 cr kernel: fwbr115i1: port 2(fwln115o1) entered forwarding state
Sep 05 18:31:31 cr qm[1514656]: VM 115 started with PID 1514668.
Sep 05 18:31:31 cr qm[1514653]: <root@pam> end task UPID:cr:00171CA0:00EE1BF1:68BB1061:qmstart:115:root@pam: OK
Sep 05 18:31:31 cr systemd-logind[1645]: Session 355 logged out. Waiting for processes to exit.
Sep 05 18:31:31 cr systemd[1]: session-355.scope: Deactivated successfully.
Sep 05 18:31:31 cr systemd[1]: session-355.scope: Consumed 590ms CPU time, 132.8M memory peak.
Sep 05 18:31:31 cr systemd-logind[1645]: Removed session 355.
Sep 05 18:31:31 cr systemd-logind[1645]: New session 356 of user root.
Sep 05 18:31:31 cr systemd[1]: Started session-356.scope - Session 356 of User root.
Sep 05 18:31:35 cr QEMU[1514668]: kvm: Features 0x130afffaf unsupported. Allowed features: 0x1c0010179bfffe7
Sep 05 18:31:35 cr QEMU[1514668]: kvm: Failed to load virtio-net:virtio
Sep 05 18:31:35 cr QEMU[1514668]: kvm: error while loading state for instance 0x0 of device '0000:00:1e.0:01.0:12.0/virtio-net'
Sep 05 18:31:35 cr QEMU[1514668]: kvm: load of migration failed: Operation not permitted
Sep 05 18:31:35 cr kernel: tap115i1: left allmulticast mode
Sep 05 18:31:35 cr kernel: fwbr115i1: port 1(tap115i1) entered disabled state
Sep 05 18:31:35 cr ovs-vsctl[1514768]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115o1
Sep 05 18:31:35 cr kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
Sep 05 18:31:35 cr kernel: fwln115o1 (unregistering): left allmulticast mode
Sep 05 18:31:35 cr kernel: fwln115o1 (unregistering): left promiscuous mode
Sep 05 18:31:35 cr kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
Sep 05 18:31:35 cr zebra[1761]: libyang Unsatisfied pattern - "" does not conform to "[0-9a-fA-F]{2}(:[0-9a-fA-F]{2}){5}". (/frr-interface:lib/interface/state/phy-address)
Sep 05 18:31:35 cr ovs-vsctl[1514773]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i1
Sep 05 18:31:35 cr ovs-vsctl[1514773]: ovs|00002|db_ctl_base|ERR|no port named fwln115i1
Sep 05 18:31:35 cr ovs-vsctl[1514776]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i1
Sep 05 18:31:35 cr ovs-vsctl[1514776]: ovs|00002|db_ctl_base|ERR|no port named tap115i1
Sep 05 18:31:35 cr ovs-vsctl[1514778]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i0
Sep 05 18:31:35 cr ovs-vsctl[1514778]: ovs|00002|db_ctl_base|ERR|no port named fwln115i0
Sep 05 18:31:35 cr ovs-vsctl[1514779]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i0
Sep 05 18:31:35 cr qm[1514753]: VM 115 qmp command failed - VM 115 qmp command 'query-status' failed - client closed connection
Sep 05 18:31:35 cr sshd-session[1514745]: pam_unix(sshd:session): session closed for user root
Sep 05 18:31:35 cr systemd-logind[1645]: Session 356 logged out. Waiting for processes to exit.
Sep 05 18:31:35 cr systemd[1]: session-356.scope: Deactivated successfully.
Sep 05 18:31:35 cr systemd[1]: session-356.scope: Consumed 1.108s CPU time, 116.5M memory peak.
Sep 05 18:31:35 cr systemd-logind[1645]: Removed session 356.
Sep 05 18:31:35 cr systemd[1]: 115.scope: Deactivated successfully.
Sep 05 18:31:35 cr systemd[1]: 115.scope: Consumed 1.661s CPU time, 494.7M memory peak.
Sep 05 18:31:35 cr qmeventd[1514784]: Starting cleanup for 115
Sep 05 18:31:35 cr ovs-vsctl[1514786]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i1
Sep 05 18:31:35 cr ovs-vsctl[1514786]: ovs|00002|db_ctl_base|ERR|no port named fwln115i1
Sep 05 18:31:35 cr ovs-vsctl[1514787]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i1
Sep 05 18:31:35 cr ovs-vsctl[1514787]: ovs|00002|db_ctl_base|ERR|no port named tap115i1
Sep 05 18:31:35 cr ovs-vsctl[1514788]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i0
Sep 05 18:31:35 cr ovs-vsctl[1514788]: ovs|00002|db_ctl_base|ERR|no port named fwln115i0
Sep 05 18:31:35 cr ovs-vsctl[1514789]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i0
Sep 05 18:31:35 cr ovs-vsctl[1514789]: ovs|00002|db_ctl_base|ERR|no port named tap115i0
Sep 05 18:31:35 cr qmeventd[1514784]: Finished cleanup for 115

The funny thing is from the webui I saw the VM on the target host in running state (but only from the webui view of course). In the reallity nothing run on the target (pve9).

If I do a migrate from the pve9 => pv8 hosts, the webui show the VM on the target but in stop state, and I can start it.


If I do a cold migration, the vm start to pve9 without knowned issue, but I can't do a live migration to the pve8 target due a qmeu issue explicity print in the migration window. To put back my VM I've to do a cald migration.

Another thing, My network have MTU 9000, but the source and destination hosts have the same config. The target host is an upgrade from the cluster host from pve8 to pve9
 
Last edited:
iopsthecloud said:
I tried to do another test with a linux/debian guest, and I've got a more explicit message, with the VM not starting on the target host.
Click to expand...
Could you also share the VM 115 - Migrate task log as well as the VM 115 - Start task log (bottom panel in the web UI or check the node's Task History)?
 
fiona said:
Could you also share the VM 115 - Migrate task log as well as the VM 115 - Start task log (bottom panel in the web UI or check the node's Task History)?
Click to expand...

yes plese find it bellow

Code: 
2025-09-05 18:31:29 starting migration of VM 115 to node 'cr' (X)
2025-09-05 18:31:29 starting VM 115 on remote node 'cr'
2025-09-05 18:31:31 start remote tunnel
2025-09-05 18:31:31 ssh tunnel ver 1
2025-09-05 18:31:31 starting online/live migration on unix:/run/qemu-server/115.migrate
2025-09-05 18:31:31 set migration capabilities
2025-09-05 18:31:31 migration downtime limit: 100 ms
2025-09-05 18:31:31 migration cachesize: 512.0 MiB
2025-09-05 18:31:31 set migration parameters
2025-09-05 18:31:31 start migrate command to unix:/run/qemu-server/115.migrate
2025-09-05 18:31:32 migration active, transferred 137.3 MiB of 3.0 GiB VM-state, 117.2 MiB/s
2025-09-05 18:31:33 migration active, transferred 250.3 MiB of 3.0 GiB VM-state, 114.8 MiB/s
2025-09-05 18:31:35 average migration speed: 768.3 MiB/s - downtime 93 ms
2025-09-05 18:31:35 migration completed, transferred 378.1 MiB VM-state
2025-09-05 18:31:35 migration status: completed
2025-09-05 18:31:35 ERROR: tunnel replied 'ERR: resume failed - VM 115 qmp command 'query-status' failed - client closed connection' to command 'resume 115'
2025-09-05 18:31:37 ERROR: migration finished with problems (duration 00:00:08)
TASK ERROR: migration problems
 
iopsthecloud said:
Sep 05 18:31:35 cr QEMU[1514668]: kvm: Features 0x130afffaf unsupported. Allowed features: 0x1c0010179bfffe7
Sep 05 18:31:35 cr QEMU[1514668]: kvm: Failed to load virtio-net:virtio
Sep 05 18:31:35 cr QEMU[1514668]: kvm: error while loading state for instance 0x0 of device '0000:00:1e.0:01.0:12.0/virtio-net'
Sep 05 18:31:35 cr QEMU[1514668]: kvm: load of migration failed: Operation not permitted
Click to expand...
See the link already posted above, this is the same issue: https://forum.proxmox.com/threads/qemu-virtio-issues-after-upgrade-to-9.169625/

A fix is available with qemu-server >= 9.0.20 which is currently available in the pve-test repository.
 
Hello,

on the node pve9 I do these updates from test
2025-09-08_13-17.jpeg
journalctl -e (targeted node with pve9)
Code: 
sept. 08 13:18:36 cr systemd-logind[1645]: New session 842 of user root.
sept. 08 13:18:36 cr systemd[1]: Started session-842.scope - Session 842 of User root.
sept. 08 13:18:43 cr QEMU[3848347]: kvm: Features 0x130afffaf unsupported. Allowed features: 0x1c0010179bfffe7
sept. 08 13:18:43 cr QEMU[3848347]: kvm: Failed to load virtio-net:virtio
sept. 08 13:18:43 cr QEMU[3848347]: kvm: error while loading state for instance 0x0 of device '0000:00:1e.0:01.0:12.0/virtio-net'
sept. 08 13:18:43 cr QEMU[3848347]: kvm: load of migration failed: Operation not permitted
sept. 08 13:18:43 cr kernel: tap115i1: left allmulticast mode
sept. 08 13:18:43 cr kernel: fwbr115i1: port 1(tap115i1) entered disabled state
sept. 08 13:18:43 cr ovs-vsctl[3848525]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115o1
sept. 08 13:18:43 cr kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
sept. 08 13:18:43 cr kernel: fwln115o1 (unregistering): left allmulticast mode
sept. 08 13:18:43 cr kernel: fwln115o1 (unregistering): left promiscuous mode
sept. 08 13:18:43 cr kernel: fwbr115i1: port 2(fwln115o1) entered disabled state
sept. 08 13:18:43 cr zebra[1761]: libyang Unsatisfied pattern - "" does not conform to "[0-9a-fA-F]{2}(:[0-9a-fA-F]{2}){5}". (/frr-interface:lib/interface/state/phy->
sept. 08 13:18:43 cr ovs-vsctl[3848530]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i1
sept. 08 13:18:43 cr ovs-vsctl[3848530]: ovs|00002|db_ctl_base|ERR|no port named fwln115i1
sept. 08 13:18:43 cr ovs-vsctl[3848533]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i1
sept. 08 13:18:43 cr ovs-vsctl[3848533]: ovs|00002|db_ctl_base|ERR|no port named tap115i1
sept. 08 13:18:43 cr ovs-vsctl[3848535]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i0
sept. 08 13:18:43 cr ovs-vsctl[3848535]: ovs|00002|db_ctl_base|ERR|no port named fwln115i0
sept. 08 13:18:43 cr ovs-vsctl[3848537]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i0
sept. 08 13:18:43 cr corosync[2004]:   [KNET  ] link: host: 2 link: 0 is down
sept. 08 13:18:43 cr corosync[2004]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
sept. 08 13:18:43 cr corosync[2004]:   [KNET  ] host: host: 2 has no active links
sept. 08 13:18:43 cr qm[3848432]: VM 115 qmp command failed - VM 115 qmp command 'query-status' failed - client closed connection
sept. 08 13:18:43 cr sshd-session[3848424]: pam_unix(sshd:session): session closed for user root
sept. 08 13:18:43 cr systemd-logind[1645]: Session 842 logged out. Waiting for processes to exit.
sept. 08 13:18:43 cr systemd[1]: session-842.scope: Deactivated successfully.
sept. 08 13:18:43 cr systemd[1]: session-842.scope: Consumed 1.747s CPU time, 117.4M memory peak.
sept. 08 13:18:43 cr systemd-logind[1645]: Removed session 842.
sept. 08 13:18:43 cr systemd[1]: 115.scope: Deactivated successfully.
sept. 08 13:18:43 cr systemd[1]: 115.scope: Consumed 1.921s CPU time, 834.3M memory peak.
sept. 08 13:18:44 cr qmeventd[3848542]: Starting cleanup for 115
sept. 08 13:18:44 cr ovs-vsctl[3848545]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i0
sept. 08 13:18:44 cr ovs-vsctl[3848545]: ovs|00002|db_ctl_base|ERR|no port named fwln115i0
sept. 08 13:18:44 cr ovs-vsctl[3848546]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i0
sept. 08 13:18:44 cr ovs-vsctl[3848546]: ovs|00002|db_ctl_base|ERR|no port named tap115i0
sept. 08 13:18:44 cr ovs-vsctl[3848547]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln115i1
sept. 08 13:18:44 cr ovs-vsctl[3848547]: ovs|00002|db_ctl_base|ERR|no port named fwln115i1
sept. 08 13:18:44 cr ovs-vsctl[3848548]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap115i1
sept. 08 13:18:44 cr ovs-vsctl[3848548]: ovs|00002|db_ctl_base|ERR|no port named tap115i1
sept. 08 13:18:44 cr qmeventd[3848542]: Finished cleanup for 115


task.log (node source)
Code: 
()
2025-09-08 13:18:34 starting migration of VM 115 to node 'cr' (X)
2025-09-08 13:18:34 starting VM 115 on remote node 'cr'
2025-09-08 13:18:36 start remote tunnel
2025-09-08 13:18:37 ssh tunnel ver 1
2025-09-08 13:18:37 starting online/live migration on unix:/run/qemu-server/115.migrate
2025-09-08 13:18:37 set migration capabilities
2025-09-08 13:18:37 migration downtime limit: 100 ms
2025-09-08 13:18:37 migration cachesize: 512.0 MiB
2025-09-08 13:18:37 set migration parameters
2025-09-08 13:18:37 start migrate command to unix:/run/qemu-server/115.migrate
2025-09-08 13:18:38 migration active, transferred 137.6 MiB of 3.0 GiB VM-state, 116.1 MiB/s
2025-09-08 13:18:39 migration active, transferred 251.0 MiB of 3.0 GiB VM-state, 113.7 MiB/s
2025-09-08 13:18:40 migration active, transferred 363.9 MiB of 3.0 GiB VM-state, 150.1 MiB/s
2025-09-08 13:18:41 migration active, transferred 477.6 MiB of 3.0 GiB VM-state, 263.8 MiB/s
2025-09-08 13:18:42 migration active, transferred 591.6 MiB of 3.0 GiB VM-state, 410.6 MiB/s
2025-09-08 13:18:43 average migration speed: 512.2 MiB/s - downtime 65 ms
2025-09-08 13:18:43 migration completed, transferred 708.3 MiB VM-state
2025-09-08 13:18:43 migration status: completed
2025-09-08 13:18:43 ERROR: tunnel replied 'ERR: resume failed - VM 115 qmp command 'query-status' failed - client closed connection' to command 'resume 115'
2025-09-08 13:18:46 ERROR: migration finished with problems (duration 00:00:12)
TASK ERROR: migration problems

After the migration task in failed state, I see a task "start"
2025-09-08_13-23.jpeg

But as you seen, it's still failing.
 
You need to upgrade both nodes.
 
fiona said:
You need to upgrade both nodes.
Click to expand...
oh... but after upgrade do I need top stop and start (cold) the VM to be migrated from pve8 to pve9?

Because if it's mandatory it's easiest for me to migrate to pve9 because in all cases I'll need top stop the service....

If it's not mandatory to restart VM on the source nodes, can I upgrade only qemu ?
 
No, the VM does not need to be stopped in this case. Both source and target node need to have at least qemu-server >= 9.0.20 (or 8.4.2 for Proxmox VE 8), that is the relevant package for fixing the issue at hand.

EDIT: also mention version for Proxmox VE 8 here for completeness.
 
Last edited:
iopsthecloud said:
OK, but http://download.proxmox.com/debian/dists/bookworm/pvetest/binary-amd64/ show no qemu-server 9 at all.

My sous node is in pve8 / bookworm
Click to expand...
For Proxmox VE 8, the fix is in
Code: 
qemu-server (8.4.2) bookworm; urgency=medium

  * fix #6466: aarch64: pci: properly print higher-index PCI bridge addresses.

  * api: vm start: introduce nets-host-mtu parameter for live-migration
    compat. This fixes some issues with migrating to a Proxmox VE 9 host, e.g.
    during major upgrade, as that version started to inherit the MTU from the
    underlying Linux bridge if it's not explicitly set, this the chosen MTU
    for a virtual NIC might differ on source and target, which can result in a
    different (incompatible) virtual hardware layout, at least for i440fx
    machines.

 -- Proxmox Support Team <support@proxmox.com>  Thu, 04 Sep 2025 20:02:48 +0200
 
You must log in or register to reply here.
Top Bottom