[SOLVED] VM doesn`t start after the server has been moved to the new DC.

[aindjov]

Hello,

As you can understand from the thread.
pveversion: pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.8.4-2-pve)

When i try to run the VM i get this error

TASK ERROR: start failed: command '/usr/bin/kvm -id 160 -name 'ODC-SIEM01,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/160.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/160.pid -daemonize -smbios 'type=1,uuid=d48c33d6-93b5-427b-a5f2-948629370acd' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/160.vnc,password=on' -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 8192 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=10b959b8-bea5-42ec-9bc0-71854e632f03' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:606130edafc0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:VMStore/vm-160-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=rbd:VMStore/vm-160-disk-3:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi1,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -drive 'file=rbd:VMStore/vm-160-disk-4:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi2,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2' -drive 'file=rbd:VMStore/vm-160-disk-5:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi3,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3' -netdev 'type=tap,id=net0,ifname=tap160i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'vmxnet3,mac=7A:D5:B5:14:D6:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=101' -machine 'type=pc+pve0'' failed: got timeout

any suggestion?

Thanks

 

[leesteken]

When i try to run the VM i get this error

TASK ERROR: start failed: command '/usr/bin/kvm -id 160 -name 'ODC-SIEM01,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/160.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/160.pid -daemonize -smbios 'type=1,uuid=d48c33d6-93b5-427b-a5f2-948629370acd' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/160.vnc,password=on' -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 8192 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=10b959b8-bea5-42ec-9bc0-71854e632f03' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:606130edafc0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:VMStore/vm-160-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=rbd:VMStore/vm-160-disk-3:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi1,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -drive 'file=rbd:VMStore/vm-160-disk-4:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi2,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2' -drive 'file=rbd:VMStore/vm-160-disk-5:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi3,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3' -netdev 'type=tap,id=net0,ifname=tap160i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'vmxnet3,mac=7A:D5:B5:14:D6:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=101' -machine 'type=pc+pve0'' failed: got timeout

any suggestion?

Check the Proxmox Syslog (or journalctl) from around the time that your started VM 160. If there are no message there about what went wrong, then it's a timeout because Proxmox cannot find 8GB of free memory quickly enough.

 

[bbgeek17]

I recommend adding VM config next time ("qm config [id]") to your report, its much easier to read than the single line you posted.

That said, was the server moved as is, or was the VM moved to new server? Is Ceph health ok?
If the VM was moved, the PCI devices may have changed. Additionally, CPU flags may no longer match.

You have a lot going on in the config, try to simplify it. And do look at the log.

Spoiler: vm line split

TASK ERROR: start failed: command '/usr/bin/kvm
-id 160
-name 'ODC-SIEM01,debug-threads=on'
-no-shutdown
-chardev 'socket,id=qmp,path=/var/run/qemu-server/160.qmp,server=on,wait=off'
-mon 'chardev=qmp,mode=control'
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5'
-mon 'chardev=qmp-event,mode=control'
-pidfile /var/run/qemu-server/160.pid
-daemonize
-smbios 'type=1,uuid=d48c33d6-93b5-427b-a5f2-948629370acd'
-smp '2,sockets=2,cores=1,maxcpus=2'
-nodefaults
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg'
-vnc 'unix:/var/run/qemu-server/160.vnc,password=on'
-cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
-m 8192
-device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e'
-device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f'
-device 'vmgenid,guid=10b959b8-bea5-42ec-9bc0-71854e632f03'
-device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2'
-device 'usb-tablet,id=tablet,bus=uhci.0,port=1'
-device 'VGA,id=vga,bus=pci.0,addr=0x2'
-device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on'
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:606130edafc0'
-device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5'
-drive 'file=rbd:VMStore/vm-160-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on'
-device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100'
-drive 'file=rbd:VMStore/vm-160-disk-3:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi1,format=raw,cache=none,aio=io_uring,detect-zeroes=on'
-device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1'
-drive 'file=rbd:VMStore/vm-160-disk-4:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi2,format=raw,cache=none,aio=io_uring,detect-zeroes=on'
-device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2'
-drive 'file=rbd:VMStore/vm-160-disk-5:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/VMStore.keyring,if=none,id=drive-scsi3,format=raw,cache=none,aio=io_uring,detect-zeroes=on'
-device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3'
-netdev 'type=tap,id=net0,ifname=tap160i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown'
-device 'vmxnet3,mac=7A5:B5:146:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=101'
-machine 'type=pc+pve0'' failed: got timeout

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[aindjov]

Thanks for your answers.
The virtual machine is on the same server, it's just that the servers themselves were shut down due to a lack of power.
And then moved physically to another place.
After we started them and everything went normally with the proxmox there are no problems, but there are problems with the virtual machines themselves.

logs from syslog

2024-06-01T18:16:56.856151+03:00 ODC-HV003 pvedaemon[496481]: VM 160 qmp command failed - VM 160 qmp command 'query-proxmox-support' failed - got timeout
2024-06-01T18:17:00.788467+03:00 ODC-HV003 pvestatd[4061]: VM 160 qmp command failed - VM 160 qmp command 'query-proxmox-support' failed - unable to connect to VM 160 qmp socket - timeout after 51 retries

qm config 160
agent: 0
boot: order=scsi0;net0
cores: 1
description: Template for Linux Server
hotplug: disk,network,usb
localtime: 0
memory: 8192
meta: creation-qemu=6.1.1,ctime=1657102774
name: ODC-SIEM01
net0: vmxnet3=7A:D5:B5:14:D6:87,bridge=vmbr3,firewall=1
numa: 0
ostype: l26
scsi0: VMStore:vm-160-disk-0,size=20G
scsi1: VMStore:vm-160-disk-3,size=4G
scsi2: VMStore:vm-160-disk-4,size=4G
scsi3: VMStore:vm-160-disk-5,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=d48c33d6-93b5-427b-a5f2-948629370acd
sockets: 2
vmgenid: 10b959b8-bea5-42ec-9bc0-71854e632f03
vmstatestorage: VMStore

 

[leesteken]

The virtual machine is on the same server, it's just that the servers themselves were shut down due to a lack of power.

Did you have an unexpected power loss? Maybe your VM storage drives became corrupted; please check them.

logs from syslog

2024-06-01T18:16:56.856151+03:00 ODC-HV003 pvedaemon[496481]: VM 160 qmp command failed - VM 160 qmp command 'query-proxmox-support' failed - got timeout
2024-06-01T18:17:00.788467+03:00 ODC-HV003 pvestatd[4061]: VM 160 qmp command failed - VM 160 qmp command 'query-proxmox-support' failed - unable to connect to VM 160 qmp socket - timeout after 51 retries

Surely there must be more log lines besides those two? Not all relevant lines with contain the number 160. Maybe there are drive errors also?

 

[bbgeek17]

The VM has modest resources. Absent any other information, given that all VM disks are on Ceph, I guess that the timeout is Ceph/storage related.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox