As a cluster storage i use RADOS/RBD(ceph block storage)
When i try to migrate from node1 or node3 to node2 it fails with:
node1
node2
node3
cmd of one of the vms (they all fail)
different vm
my storage.cfg
If you need more information please tell me and ill provide them the best I can
When i try to migrate from node1 or node3 to node2 it fails with:
Code:
Header
Virtual Machine 100 (ubuntu-test-migration) on node 'nue-srv-pve2'
2024-04-11 07:10:56 starting migration of VM 100 to node 'nue-srv-pve2' (10.15.15.51)
2024-04-11 07:10:56 starting VM 100 on remote node 'nue-srv-pve2'
2024-04-11 07:10:58 start remote tunnel
2024-04-11 07:10:59 ssh tunnel ver 1
2024-04-11 07:10:59 starting online/live migration on unix:/run/qemu-server/100.migrate
2024-04-11 07:10:59 set migration capabilities
2024-04-11 07:10:59 migration downtime limit: 100 ms
2024-04-11 07:10:59 migration cachesize: 1.0 GiB
2024-04-11 07:10:59 set migration parameters
2024-04-11 07:10:59 start migrate command to unix:/run/qemu-server/100.migrate
2024-04-11 07:11:00 migration active, transferred 302.7 MiB of 8.0 GiB VM-state, 485.6 MiB/s
2024-04-11 07:11:01 migration active, transferred 762.5 MiB of 8.0 GiB VM-state, 555.9 MiB/s
2024-04-11 07:11:02 migration active, transferred 1.2 GiB of 8.0 GiB VM-state, 517.1 MiB/s
2024-04-11 07:11:03 migration active, transferred 1.6 GiB of 8.0 GiB VM-state, 347.2 MiB/s
2024-04-11 07:11:04 migration active, transferred 2.0 GiB of 8.0 GiB VM-state, 366.6 MiB/s
2024-04-11 07:11:05 migration active, transferred 2.5 GiB of 8.0 GiB VM-state, 435.8 MiB/s
2024-04-11 07:11:06 migration active, transferred 2.9 GiB of 8.0 GiB VM-state, 375.1 MiB/s
2024-04-11 07:11:07 migration active, transferred 3.3 GiB of 8.0 GiB VM-state, 444.8 MiB/s
2024-04-11 07:11:08 migration active, transferred 3.8 GiB of 8.0 GiB VM-state, 495.3 MiB/s
2024-04-11 07:11:09 migration active, transferred 4.3 GiB of 8.0 GiB VM-state, 514.6 MiB/s
2024-04-11 07:11:10 migration active, transferred 4.7 GiB of 8.0 GiB VM-state, 502.5 MiB/s
2024-04-11 07:11:11 migration active, transferred 5.3 GiB of 8.0 GiB VM-state, 524.1 MiB/s
2024-04-11 07:11:12 migration active, transferred 5.8 GiB of 8.0 GiB VM-state, 516.9 MiB/s
2024-04-11 07:11:13 migration active, transferred 6.2 GiB of 8.0 GiB VM-state, 497.7 MiB/s
2024-04-11 07:11:14 migration active, transferred 6.7 GiB of 8.0 GiB VM-state, 517.1 MiB/s
2024-04-11 07:11:15 migration active, transferred 7.2 GiB of 8.0 GiB VM-state, 461.2 MiB/s
2024-04-11 07:11:16 average migration speed: 483.1 MiB/s - downtime 278 ms
2024-04-11 07:11:16 migration status: completed
2024-04-11 07:11:16 ERROR: tunnel replied 'ERR: resume failed - VM 100 qmp command 'query-status' failed - client closed connection' to command 'resume 100'
VM quit/powerdown failed - terminating now with SIGTERM
2024-04-11 07:11:30 ERROR: migration finished with problems (duration 00:00:35)
TASK ERROR: migration problemsnode1
Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-3-pve)
pve-manager: 8.1.10 (running version: 8.1.10/4b06efb5db453f29)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph: 18.2.2-pve1
ceph-fuse: 18.2.2-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve1
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.10-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1node2
Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-5-pve)
pve-manager: 8.1.10 (running version: 8.1.10/4b06efb5db453f29)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph: 18.2.2-pve1
ceph-fuse: 18.2.2-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve1
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.10-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1node3
Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-3-pve)
pve-manager: 8.1.10 (running version: 8.1.10/4b06efb5db453f29)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph: 18.2.2-pve1
ceph-fuse: 18.2.2-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve1
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.10-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1cmd of one of the vms (they all fail)
Code:
/usr/bin/kvm \
-id 100 \
-name 'ubuntu-test-migration,debug-threads=on' \
-no-shutdown \
-chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' \
-mon 'chardev=qmp,mode=control' \
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
-mon 'chardev=qmp-event,mode=control' \
-pidfile /var/run/qemu-server/100.pid \
-daemonize \
-smbios 'type=1,uuid=d4f8d1ff-fef7-4b84-98cd-f43fb78daa72' \
-drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' \
-drive 'if=pflash,unit=1,id=drive-efidisk0,cache=writeback,format=raw,file=rbd:cluster-data/vm-100-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/cluster-data.keyring:rbd_cache_policy=writeback,size=540672' \
-smp '4,sockets=1,cores=4,maxcpus=4' \
-nodefaults \
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
-vnc 'unix:/var/run/qemu-server/100.vnc,password=on' \
-cpu host,+kvm_pv_eoi,+kvm_pv_unhalt \
-m 8192 \
-object 'iothread,id=iothread-virtioscsi0' \
-readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
-device 'vmgenid,guid=17554cb6-0dc7-4d32-945f-0ad6645439fc' \
-device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
-chardev 'socket,id=tpmchar,path=/var/run/qemu-server/100.swtpm' \
-tpmdev 'emulator,id=tpmdev,chardev=tpmchar' \
-device 'tpm-tis,tpmdev=tpmdev' \
-device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
-chardev 'socket,path=/var/run/qemu-server/100.qga,server=on,wait=off,id=qga0' \
-device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
-device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:1997c5a7e2ff' \
-drive 'file=/mnt/pve/isostore/template/iso/ubuntu-22.04.4-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' \
-device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' \
-drive 'file=rbd:cluster-data/vm-100-disk-1:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/cluster-data.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
-device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
-netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
-device 'virtio-net-pci,mac=BC:24:11:FF:4C:B8,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102' \
-machine 'type=q35+pve0'different vm
Code:
/usr/bin/kvm \
-id 101 \
-name 'NUE-CLI-WIN11TEST,debug-threads=on' \
-no-shutdown \
-chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server=on,wait=off' \
-mon 'chardev=qmp,mode=control' \
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
-mon 'chardev=qmp-event,mode=control' \
-pidfile /var/run/qemu-server/101.pid \
-daemonize \
-smbios 'type=1,uuid=437db651-5755-4af7-9f03-0ed7ccb24298' \
-drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' \
-drive 'if=pflash,unit=1,id=drive-efidisk0,cache=writeback,format=raw,file=rbd:cluster-data/vm-101-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/cluster-data.keyring:rbd_cache_policy=writeback,size=540672' \
-smp '4,sockets=1,cores=4,maxcpus=4' \
-nodefaults \
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
-vnc 'unix:/var/run/qemu-server/101.vnc,password=on' \
-cpu 'host,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt' \
-m 8192 \
-readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
-device 'vmgenid,guid=e040c197-a1a2-4ad6-89b4-352083867edf' \
-device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
-chardev 'socket,id=tpmchar,path=/var/run/qemu-server/101.swtpm' \
-tpmdev 'emulator,id=tpmdev,chardev=tpmchar' \
-device 'tpm-tis,tpmdev=tpmdev' \
-device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:1997c5a7e2ff' \
-drive 'file=/mnt/pve/isostore/template/iso/virtio-win-0.1.248.iso,if=none,id=drive-ide2,media=cdrom,aio=io_uring' \
-device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2' \
-device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' \
-drive 'file=rbd:cluster-data/vm-101-disk-1:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/cluster-data.keyring,if=none,id=drive-sata0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
-device 'ide-hd,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100' \
-netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' \
-device 'e1000,mac=BC:24:11:5F:4B:F9,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=101' \
-rtc 'driftfix=slew,base=localtime' \
-machine 'hpet=off,type=pc-q35-8.1+pve0' \
-global 'kvm-pit.lost_tick_policy=discard'my storage.cfg
Code:
dir: local
disable
path /var/lib/vz
content backup,iso,vztmpl
shared 0
lvmthin: local-lvm
disable
thinpool data
vgname pve
content rootdir,images
cephfs: isostore
path /mnt/pve/isostore
content vztmpl,iso
fs-name isostore
rbd: cluster-data
content images,rootdir
krbd 0
pool cluster-data
iscsi: temp
disable
portal 192.168.95.250
target iqn.2004-04.com.qnap:ts-431p:iscsi.proxmox.09eedb
content images
lvm: te
disable
vgname Cluster-Storage
content rootdir,images
saferemove 0
shared 0
cephfs: backup
path /mnt/pve/backup
content iso,backup,vztmpl
fs-name backupIf you need more information please tell me and ill provide them the best I can
