All VM stopped working

[Estario]

Hello,
I had to reboot my server and after that I can log into the proxmox web interface, but I am unable to use any of the VM. When I try to start the I get this error:
Virtual Environment 4.3-1/e7cdc165
Virtual Machine 110 ('Doci-MB1' ) on node 'pve'
Logs

()
kvm: -drive file=/dev/pve/vm-103-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/pve/vm-103-disk-1': No such file or directory
TASK ERROR: start failed: command '/usr/bin/kvm -id 103 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=2ccd9aec-ac71-4e02-919d-7a7d404c4f5d' -name PrimaryDomainController -smp '8,sockets=2,cores=4,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/103.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 8192 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:189534a7056' -drive 'file=/var/lib/vz/template/iso/zentyal-5.0-development-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/pve/vm-103-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=AEC:BC:E7:E00,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1

And another one:
()
kvm: -drive file=/dev/pve/vm-112-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on: Could not open '/dev/pve/vm-112-disk-1': No such file or directory
TASK ERROR: start failed: command '/usr/bin/kvm -id 112 -chardev 'socket,id=qmp,path=/var/run/qemu-server/112.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/112.pid -daemonize -smbios 'type=1,uuid=96500499-614d-4b03-80a7-56604e372710' -name Doci-Client -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/112.vnc,x509,password -no-hpet -cpu 'kvm64,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce' -m 2048 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:189534a7056' -drive 'file=/var/lib/vz/template/iso/virtio-win-0.1.126.iso,if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' -drive 'file=/var/lib/vz/template/iso/Office-2013-SP1-PPVL-x64-en-US-Jun2014.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201' -drive 'file=/dev/pve/vm-112-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap112i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=4A:AD:18:33:7D:8B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1

I can't understand what the problem is. My knowledge is limited and I have no idea what to do to try solve the issue.

 

[wolfgang]

Hi,

the problem is the lvm.

What you get fore an output with
lvs
vgs

 

[Estario]

The output from lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotzM- 1.70t 92.97 99.95
root pve -wi-ao---- 96.00g
swap pve -wi-ao---- 8.00g
vm-100-disk-1 pve Vwi-i-tz-- 650.00g data 100.00 100.00
vm-100-disk-2 pve Vwi-i-tz-- 1.00t data 100.00 100.00
vm-101-disk-1 pve Vwi-i-tz-- 5.00g data 100.00 100.00
vm-101-disk-2 pve Vwi-i-tz-- 1.00t data 100.00 100.00
vm-102-disk-1 pve Vwi-i-tz-- 1.00t data 100.00 100.00
vm-103-disk-1 pve Vwi-i-tz-- 1.00t data 100.00 100.00
vm-104-disk-1 pve Vwi-i-tz-- 1.00t data 100.00 100.00
vm-106-disk-1 pve Vwi-i-tz-- 200.00g data 100.00 100.00
vm-107-disk-1 pve Vwi-i-tz-- 60.00g data 100.00 100.00
vm-108-disk-1 pve Vwi-i-tz-- 60.00g data 100.00 100.00
vm-109-disk-1 pve Vwi-i-tz-- 60.00g data 100.00 100.00
vm-110-disk-1 pve Vwi-i-tz-- 60.00g data 100.00 100.00
vm-111-disk-1 pve Vwi-i-tz-- 60.00g data 100.00 100.00
vm-112-disk-1 pve Vwi-i-tz-- 40.00g data 100.00 100.00
vm-113-disk-1 pve Vwi-i-tz-- 500.00g data 100.00 100.00
and VGS:
VG #PV #LV #SN Attr VSize VFree
pve 2 18 0 wz--n- 1.82t 15.79g

 

[wolfgang]

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotzM- 1.70t 92.97 99.95

Your have no space on this storage left.
This is the reason why you can't start the vm's.

 

[Estario]

Ok. I feel kind of stupid.
Next problem. When I try removing VM's I get this error. Nothing seems to be using any of the disks. No VM's are running.

()
Task viewer: VM 107 - Destroy

OutputStatus

Stop
Device pve-data-tpool (251:4) is used by another device.
TASK ERROR: lvremove 'pve/vm-107-disk-1' error: Unable to deactivate logical volume pve/vm-107-disk-1.

 

[wolfgang]

The problem is you have no metadata left try to erase snapshots.