VMs Not Starting After Hard Reboot

[jingjong]

Hi All,

After we had a hard reboot today. All VMs are not starting. Below is one of the VM error log when we tried to start the VM. Also for some reason I can't find the /dev/pve/vm-102-disk-1

And now the Storage 'local-lvm' says: Usage 0.00% (0 B of 326.88 GiB)

I am currently: Virtual Environment 4.4-1/eb2d6f1e

Error Log:
start failed: command '/usr/bin/kvm -id 102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=c1a20845-a998-466b-9f13-bb5c4f03628b' -name vpn -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3d6c78c4a2a2' -drive 'file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-1611.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=22:E7:B0:76:C2:E1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=F6:A8:7F:E2:3E:4E,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301'' failed: exit code 1

 

[dietmar]

what is the output if you start manually on the console:

# qm start 102

 

[jingjong]

what is the output if you start manually on the console:

# qm start 102

Thanks a lot Dietmar for the reply.

This is what I got

# qm start 102
kvm: -drive file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/pve/vm-102-disk-1': No such file or directory
start failed: command '/usr/bin/kvm -id 102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=c1a20845-a998-466b-9f13-bb5c4f03628b' -name vpn -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3d6c78c4a2a2' -drive 'file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-1611.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=22:E7:B0:76:C2:E1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=F6:A8:7F:E2:3E:4E,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301'' failed: exit code 1
ipcc_send_rec failed: File too large

 

[dietmar]

T
kvm: -drive file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/pve/vm-102-disk-1': No such file or directory

Indeed, seems the file is not there - what is the output of

# lvs

And what happened exactly (any hints in the syslog, what caused the hard reboot)? Do you use some kind of RAID or a single disk setup?

 

[jingjong]

Hi Dietmar,

# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi---tz-- 326.88g
root pve -wi-ao---- 96.00g
swap pve -wi-ao---- 8.00g
vm-100-disk-1 pve Vwi---tz-- 10.00g data
vm-101-disk-1 pve Vwi---tz-- 10.00g data
vm-102-disk-1 pve Vwi---tz-- 32.00g data

Single Drive and No Raid

 

[jingjong]

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                mwfufw-UNeX-fIgL-lekx-I6nX-cNxr-msUy1J
  LV Write Access        read/write
  LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
  LV Status              available
  # open                 2
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:1

  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                V0cSbF-oWO0-juSr-wxcO-HqPw-Ubu3-pgILFc
  LV Write Access        read/write
  LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
  LV Status              available
  # open                 1
  LV Size                96.00 GiB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0

  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                UL3eR6-ylB0-Rr5I-mwjE-DyQQ-VKC3-IaOq1p
  LV Write Access        read/write
  LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              NOT available
  LV Size                326.88 GiB
  Current LE             83682
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/pve/vm-100-disk-1
  LV Name                vm-100-disk-1
  VG Name                pve
  LV UUID                3p938q-sXPY-DOgf-a8d1-xraO-137b-UP0uQp
  LV Write Access        read/write
  LV Creation host, time main, 2017-03-21 16:15:38 -0400
  LV Pool name           data
  LV Status              NOT available
  LV Size                10.00 GiB
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/pve/vm-101-disk-1
  LV Name                vm-101-disk-1
  VG Name                pve
  LV UUID                zFEnq3-fg0A-RVZ1-Ayuu-IgtY-xmDB-ninqCP
  LV Write Access        read/write
  LV Creation host, time main, 2017-03-22 12:11:09 -0400
  LV Pool name           data
  LV Status              NOT available
  LV Size                10.00 GiB
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/pve/vm-102-disk-1
  LV Name                vm-102-disk-1
  VG Name                pve
  LV UUID                RvPdhS-lPrd-WoYc-Bend-coaC-mIXt-a7ezoI
  LV Write Access        read/write
  LV Creation host, time main, 2017-03-23 15:48:47 -0400
  LV Pool name           data
  LV Status              NOT available
  LV Size                32.00 GiB
  Current LE             8192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

Tried the following, but still the pve,data,vm-100-disk-1,vm-101-disk-1 and vm-102-disk-1 are NOT available

# vgchange -a y pve
  Check of pool pve/data failed (status:1). Manual repair required!
  2 logical volume(s) in volume group "pve" now active
# lvchange -a y pve/data
  Check of pool pve/data failed (status:1). Manual repair required!

 

[dietmar]

Seems volumes are damaged. You can try to repair:

https://forum.proxmox.com/threads/lvm-issue.29134/

But you did not answered my other questions, so it is hard to help.

 

[jingjong]

Hi Dietmar,

Thanks for the reply. We are still checking the logs what really happen, but what we know so far is that the server went unresponsive the other day and we decided for a hard reboot, after that this what happen.

I will try

lvconvert --repair pve/data

and wiil try see what happen.

But what about manual repair for VG?

# vgchange -a y pve
  Check of pool pve/data failed (status:1). Manual repair required!
  2 logical volume(s) in volume group "pve" now active

vgconvert --repair pve ? is that right?


Thanks

 

[jingjong]

Hi Dietmar,

Tried the lvconvert --repair pve/data but I am getting this.

#lvconvert --repair pve/data
bad checksum in superblock
  Repair of thin metadata volume of thin pool pve/data failed (status:1). Manual repair required!

What does Manual repair required means?

 

[jingjong]

Any help from anyone? Please....

Thank you in advance.

 

[jingjong]

OK I found some reference on how to manually fix but I think he must have missed something.

https://www.redhat.com/archives/linux-lvm/2014-October/msg00032.html

I think he doesn't have steps to move the old LVs to the new pool.

So what do you guys think about this? I created my version using what I have right with my ProxMox server.

1. create temporary small LV
# lvcreate -an -Zn -L327G --name temp pve

2. replace pool's metadata volume with this tempLV
# lvconvert --thinpool pve/data --poolmetadata temp
(say 'y' to swap)

3. activate & repair metadata from 'temp' volume - you will likely need another volume where to store repaire metadata -
so create:
# lvcreate -an -Zn -L329G --name repaired pve
# lvchage -ay pve/temp
# thin_repair -i /dev/pve/temp /dev/pve/repaired

if everything when fine - compare visualy 'transaction_id' of repaired metadata (thin_dump /dev/pve/repaired)

4. swap deactivated repaired volume back to your thin-pool
# lvchange -an pve/repaired
# lvconvert --thinpool pve/pool --poolmetadata repaired

Guys I badly need your help. Thanks