1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

VMs Not Starting After Hard Reboot

Discussion in 'Proxmox VE: Installation and configuration' started by jingjong, Jan 12, 2018.

  1. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    Hi All,

    After we had a hard reboot today. All VMs are not starting. Below is one of the VM error log when we tried to start the VM. Also for some reason I can't find the /dev/pve/vm-102-disk-1

    And now the Storage 'local-lvm' says: Usage 0.00% (0 B of 326.88 GiB)

    I am currently: Virtual Environment 4.4-1/eb2d6f1e

    Error Log:
    start failed: command '/usr/bin/kvm -id 102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=c1a20845-a998-466b-9f13-bb5c4f03628b' -name vpn -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3d6c78c4a2a2' -drive 'file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-1611.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=22:E7:B0:76:C2:E1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=F6:A8:7F:E2:3E:4E,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301'' failed: exit code 1
     
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    15,532
    Likes Received:
    186
    what is the output if you start manually on the console:

    # qm start 102
     
  3. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0

    Thanks a lot Dietmar for the reply.

    This is what I got

    # qm start 102
    kvm: -drive file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/pve/vm-102-disk-1': No such file or directory
    start failed: command '/usr/bin/kvm -id 102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=c1a20845-a998-466b-9f13-bb5c4f03628b' -name vpn -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3d6c78c4a2a2' -drive 'file=/dev/pve/vm-102-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-1611.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=22:E7:B0:76:C2:E1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=F6:A8:7F:E2:3E:4E,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301'' failed: exit code 1
    ipcc_send_rec failed: File too large
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    15,532
    Likes Received:
    186
    Indeed, seems the file is not there - what is the output of

    # lvs

    And what happened exactly (any hints in the syslog, what caused the hard reboot)? Do you use some kind of RAID or a single disk setup?
     
  5. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    Hi Dietmar,

    # lvs
    LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
    data pve twi---tz-- 326.88g
    root pve -wi-ao---- 96.00g
    swap pve -wi-ao---- 8.00g
    vm-100-disk-1 pve Vwi---tz-- 10.00g data
    vm-101-disk-1 pve Vwi---tz-- 10.00g data
    vm-102-disk-1 pve Vwi---tz-- 32.00g data

    Single Drive and No Raid
     
    #5 jingjong, Jan 12, 2018
    Last edited: Jan 12, 2018
  6. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    Code:
    # lvdisplay
      --- Logical volume ---
      LV Path                /dev/pve/swap
      LV Name                swap
      VG Name                pve
      LV UUID                mwfufw-UNeX-fIgL-lekx-I6nX-cNxr-msUy1J
      LV Write Access        read/write
      LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
      LV Status              available
      # open                 2
      LV Size                8.00 GiB
      Current LE             2048
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     256
      Block device           251:1
    
      --- Logical volume ---
      LV Path                /dev/pve/root
      LV Name                root
      VG Name                pve
      LV UUID                V0cSbF-oWO0-juSr-wxcO-HqPw-Ubu3-pgILFc
      LV Write Access        read/write
      LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
      LV Status              available
      # open                 1
      LV Size                96.00 GiB
      Current LE             24576
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     256
      Block device           251:0
    
      --- Logical volume ---
      LV Name                data
      VG Name                pve
      LV UUID                UL3eR6-ylB0-Rr5I-mwjE-DyQQ-VKC3-IaOq1p
      LV Write Access        read/write
      LV Creation host, time proxmox, 2017-03-21 11:28:39 -0400
      LV Pool metadata       data_tmeta
      LV Pool data           data_tdata
      LV Status              NOT available
      LV Size                326.88 GiB
      Current LE             83682
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-100-disk-1
      LV Name                vm-100-disk-1
      VG Name                pve
      LV UUID                3p938q-sXPY-DOgf-a8d1-xraO-137b-UP0uQp
      LV Write Access        read/write
      LV Creation host, time main, 2017-03-21 16:15:38 -0400
      LV Pool name           data
      LV Status              NOT available
      LV Size                10.00 GiB
      Current LE             2560
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-101-disk-1
      LV Name                vm-101-disk-1
      VG Name                pve
      LV UUID                zFEnq3-fg0A-RVZ1-Ayuu-IgtY-xmDB-ninqCP
      LV Write Access        read/write
      LV Creation host, time main, 2017-03-22 12:11:09 -0400
      LV Pool name           data
      LV Status              NOT available
      LV Size                10.00 GiB
      Current LE             2560
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    
      --- Logical volume ---
      LV Path                /dev/pve/vm-102-disk-1
      LV Name                vm-102-disk-1
      VG Name                pve
      LV UUID                RvPdhS-lPrd-WoYc-Bend-coaC-mIXt-a7ezoI
      LV Write Access        read/write
      LV Creation host, time main, 2017-03-23 15:48:47 -0400
      LV Pool name           data
      LV Status              NOT available
      LV Size                32.00 GiB
      Current LE             8192
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
    Tried the following, but still the pve,data,vm-100-disk-1,vm-101-disk-1 and vm-102-disk-1 are NOT available
    Code:
    # vgchange -a y pve
      Check of pool pve/data failed (status:1). Manual repair required!
      2 logical volume(s) in volume group "pve" now active
    # lvchange -a y pve/data
      Check of pool pve/data failed (status:1). Manual repair required!
     
  7. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    15,532
    Likes Received:
    186
  8. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    Hi Dietmar,

    Thanks for the reply. We are still checking the logs what really happen, but what we know so far is that the server went unresponsive the other day and we decided for a hard reboot, after that this what happen.

    I will try

    lvconvert --repair pve/data

    and wiil try see what happen.

    But what about manual repair for VG?
    Code:
    # vgchange -a y pve
      Check of pool pve/data failed (status:1). Manual repair required!
      2 logical volume(s) in volume group "pve" now active
    vgconvert --repair pve ? is that right?


    Thanks
     
    #8 jingjong, Jan 13, 2018 at 13:10
    Last edited: Jan 14, 2018 at 03:05
  9. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    Hi Dietmar,

    Tried the lvconvert --repair pve/data but I am getting this.

    Code:
    #lvconvert --repair pve/data
    bad checksum in superblock
      Repair of thin metadata volume of thin pool pve/data failed (status:1). Manual repair required!
    What does Manual repair required means?
     
  10. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    Any help from anyone? Please....

    Thank you in advance.
     
  11. jingjong

    jingjong New Member

    Joined:
    Jan 12, 2018
    Messages:
    9
    Likes Received:
    0
    OK I found some reference on how to manually fix but I think he must have missed something.
    Code:
    https://www.redhat.com/archives/linux-lvm/2014-October/msg00032.html
    I think he doesn't have steps to move the old LVs to the new pool.

    So what do you guys think about this? I created my version using what I have right with my ProxMox server.

    Code:
    1. create temporary small LV
    # lvcreate -an -Zn -L327G --name temp pve
    
    2. replace pool's metadata volume with this tempLV
    # lvconvert --thinpool pve/data --poolmetadata temp
    (say 'y' to swap)
    
    3. activate & repair metadata from 'temp' volume - you will likely need another volume where to store repaire metadata -
    so create:
    # lvcreate -an -Zn -L329G --name repaired pve
    # lvchage -ay pve/temp
    # thin_repair -i /dev/pve/temp /dev/pve/repaired
    
    if everything when fine - compare visualy 'transaction_id' of repaired metadata (thin_dump /dev/pve/repaired)
    
    4. swap deactivated repaired volume back to your thin-pool
    # lvchange -an pve/repaired
    # lvconvert --thinpool pve/pool --poolmetadata repaired
    
    Guys I badly need your help. Thanks
     

Share This Page