[SOLVED] VMs start error and console not working after upgrade 5.2 --> 5.3

[hans05]

Hi forum,

after updating my 4 server cluster from 5.2 to 5.3 I got a severe problem.
When starting a VM I get:

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name Xubuntu-Base -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=8f08ce50-14c2-4301-a08c-c32bff8a2a23' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:10227640133e' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:pool01/vm-100-disk-1:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/ceph01.keyring,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3A:04:E3:37:DA:89,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc'' failed: got timeout

The GUI nevertheless shows a green triangle indicating that the VM is running and I can select the VM and "Summary" will show normal operation with uptime, CPU and memory.

Opening a console to any "running" VM fails.

Does anyone have any hint as to where to start tackling the problem?

Thanks and greetings,

hans

 

[hans05]

Update:

After doing NOTHING at all, now it seems to be working again

Maybe there is something to look into:

I originally started to describe the problem here:
https://forum.proxmox.com/threads/task-error-failed-to-run-vncproxy.49954/#post-241858

As described there, after the update from 5.2 to 5.3 apparently I lost the entries for the cluster members in each /etc/hosts file of each server.
I noticed that because I could not ssh into any other server using server names (only IP-addresses worked).
I fixed that and then COULD ssh into the other servers. But the original problem of VMs not starting correctly and console not coming remained.

I restarted the entire cluster just to make sure, but even after that the problem remained. That is when I wrote this topic.

Now, about 1/2 hour later, everything seems to work correctly. Needless to say that I am a bit puzzled about the whole thing.
Assuming that maybe the cluster needed time to get used to the fact that /etc/hosts got updated (surprises me), still I do not understand how /etc/hosts got messed up in the first place....

Anyway, thanks a lot for quick reaction and BIG thumbs up for Proxmox, one of the best programs I know!

 

[Stoiko Ivanov]

Glad the issue resolved itself! Please mark the thread as SOLVED, so that others know what to expect - Thanks!

 

[hans05]

OK, marked as solved even though I repeat that it might be interesting to check whether or not the upgrade is the culprit for messing up the hosts file.
My system really worked fine before the upgrade....

Anyway, thanks again

 

[Stoiko Ivanov]

hm - had not seen a case where a regular package upgrade mangled the /etc/hosts (on PVE) - would be surprised if it did