Lost the Proxmox web interface and my vm refuse to start with command line (image is corrupt)

[Laurent97410]

Hello,
after an automatic package update on a dedicated server running Proxmox on Debian, my vm crached and refuse to start because the image is corrupted.

Anyone can help me to fix this, or help me to rebuild a vm from the snapshots I have ? I'm not very efficient in system administration (and english is not my native language).

Thanks a lot.

Laurent

 

[Stefan_R]

Can you post the actual error message (as well as the command that fails) and any related log files (e.g. journalctl -u 'pve*')?

 

[Laurent97410]

Hello, thanks for your answer, I succeed in restoring an old backup of the vm and made the site online again, lot of job since and that's why I made so long to answer you.
Here's the error message I had when, with the Proxmox web interface lost, I tried "qm start 101" :

Virtual Environment 5.1-51
Virtual Machine 101 (irc2018pmox2) on node 'xxxxxxxx'
Logs
()
kvm: -drive file=/var/lib/vz/images/101/vm-101-disk-2.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on: qcow2: Image is corrupt; cannot be opened read/write
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -name xxxxxxxxxx -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=dbed3870-a1be-4b39-a474-85d5c207e49d' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,enforce -m 8192 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:bb96ea23a5a' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/var/lib/vz/images/101/vm-101-disk-2.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:00:00:db:92:ee,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1"

The vm is started and online but I still do not have the Proxmox web interface online and I don't know if it can make the site down again if I try to reinstall Proxmox on the server ...

Thanks for your advises ,

Laurent

 

[Stefan_R]

To see why the web interface is not showing, logs from journalctl -u 'pve*' would be very helpful. Did you check your disks for hardware errors? Do you have enough free space (use df -h)?

 

[Laurent97410]

Hello, and thanks for your answer.
I'm not able to be all days on this problem but I do want to solve it.

Unfortunately

journalctl -u 'pve*'

open on the September 6th, lot of days after the problem.

The command :

df -h

Gives :

Sys. de fichiers     Taille Utilisé Dispo Uti% Monté sur
udev                    16G       0   16G   0% /dev
tmpfs                  3,2G    347M  2,8G  11% /run
/dev/md2                20G     11G  7,2G  61% /
tmpfs                   16G     25M   16G   1% /dev/shm
tmpfs                  5,0M       0  5,0M   0% /run/lock
tmpfs                   16G       0   16G   0% /sys/fs/cgroup
/dev/mapper/pve-data   1,8T     14G  1,7T   1% /var/lib/vz
/dev/fuse               30M     16K   30M   1% /etc/pve

and since the qm restore I don't see no snapshot and no backup, I downloaded those files before the restore command so I have been able to upload a snapshot file to rollback on it.

As I manage to have the server online again with

qm restore ...

The command

qm list

gives

 VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID
       101 ******         running    8192             200.00 2371

I tried to rollback a snapshot that I have saved via scp, and the result is an error :

qm rollback 101 /home/********/snapshots/vm-101-state-snap20190820.raw

400 Parameter verification failed.
snapname: invalid format - invalid configuration ID '/home/********/snapshots/vm-101-state-snap20190820.raw'

Thanks to tell me if the disk capacity is enough, and what really mean the error message on rollback, is the snapshot have to be linked to the restyored vm in configuration file, and why is the snapname format invalid ?

Thanks to guide me,
reagards,
Laurent

 

[dcsapak]

vm-101-state-snap20190820.raw

that is not the complete snapshot but only the memory part of it. with qcow2 files the snapshot is stored in the original file, since you restored over it, i guess the snapshot is lost...

 

[Laurent97410]

Thanks to reply.
Before the restore command I have downloaded also the vm-101-disk-1.qcow2 (4,5Go) and vm-101-disk-2.qcow2 (175 Mo) files before the restore command .. That can help to rollback to the most recent vm state ?

Thanks again,
Laurent

 

[dcsapak]

ok with those files you should be able to restore the disk snapshots (check the cli tool 'qemu-img'; especially the 'snapshot' parameter)
but not the config content of that snapshot, this would have to be recreated by hand

 

[Laurent97410]

Hello, hanks to reply,
the backup of the .qcow2 disk files happened after the vm crach with the error "image is corrupt", are those files still usable to have the vm in the before crach state ?
And what do you mean by 'config content' to be recreated by hand, is there a .conf file for the vm ?

I'm just beginner in vm and proxmox ...

Thanks again,
Laurent

 

[dcsapak]

re those files still usable to have the vm in the before crach state ?

this depends on how/where the file is corrupted, so you have to test

And what do you mean by 'config content' to be recreated by hand, is there a .conf file for the vm ?

yes for vms it is under /etc/pve/nodes/NODENAME/qemu-server/ID.conf
and for containers /etc/pve/nodes/NODENAME/lxc/ID.conf