VM not starting after backup problem

[sistemas.defensorba]

Hi everyone! I'm here because i had a serious problem.
On weekend, there was a programmed backup job on our proxmox enviroment of just one VM.
This VM had it's data storaged on a ISCSI device (A storage array of disks) so the backup job really takes long time to finish.
Anyway, it started with suspend mode, on friday night and everything looks normal at least.
On sunday night the physical server reboots while the backup job was still running and there starts the problem!.

First at all the virtual machine wasn't starting because remain "locked" for backup job.... after dealing with that lock i was able to start the virtual machine but i never get it to run normally again.

The actual problem is: when the virtual machine starts, it tries to boot from disk, but this message is shown

Gave up waiting for boot device. Common problems:
-Boot args (cat/proc/cmdline)
-Check rootdelay= (did the system wait long enough?)
-Check root= (did the system wait for the right device?)
-Missing Modules (cat/proc/modules; Ls/dev)

ALERT! /dev/mapper/gestion-root does not exist. Dropping to a shell!

Busy Box v1.13.3 (Ubuntu 1:1.13.3-1 ubuntu11) built-in a shell (ash)
Enter 'help' for a list of built-in commands

(initramfs)

Obviously, the virtual machine NEVER boot and i can't use it.
Hope someone could bring me any help because i have to recover data of last week (i have done a manual backup before).
I could see the virtual disk of the virtual machine on the physical server but i can't mount it or something like that...

I need help really fast, hope this amazing community helps me....
Greetings from Argentina


PS: Sorry about my english... this is the best i can do about it...

 

[udo]

Hi,
looks like that the storage isn't ready. What is the output, if you start the vm from commandline (like "qm start 123")?
What show's "vgs"? And paste your vm-config (/etc/qemu-server/VMID.conf) and your storage.conf.

Any hint's in the logs?

Udo

 

[sistemas.defensorba]

Hi Udo! thanks for being here so quickly!!

Well... let's start answering you:

1) qm start 101 (VID = 101 in this case) says: virtual machine already running, start failed (obviously to me why)
2) on physical servers there's the following vg's:

server4:~# vgdisplay --- Volume group ---
VG Name LUN8TB
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 28
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 7.27 TB
PE Size 4.00 MB
Total PE 1905805
Alloc PE / Size 1902733 / 7.26 TB
Free PE / Size 3072 / 12.00 GB
VG UUID b9DTqE-vSNr-v0Gt-ekw6-9J5u-2VNP-gsDt59


--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 409.60 GB
PE Size 4.00 MB
Total PE 104857
Alloc PE / Size 103834 / 405.60 GB
Free PE / Size 1023 / 4.00 GB
VG UUID NUSRju-3j4D-rcf6-6X8b-nzfL-yGxk-eplXS8

Next... this is the VM conf file:

ostype: l26memory: 16384
sockets: 2
onboot: 1
name: Gestion
ide2: local:iso/ubuntu-10.04-netbook-i386.iso,media=cdrom (DONT TAKE CARE OF THIS, WAS JUST A TEST... ACTUALLY IT WAS INSTALLED UBUNTU SERVER 10.4 LTS)
vlan0: rtl8139=92:09A:BF:3E:B3
bootdisk: scsi0
scsi0: LUN8TB:vm-101-disk-1 (THIS VIRTUAL DISK IS STORED IN ISCSI ATACHED DEVICE USING open-iscsi initiator)
description: VM del sistema de gestion interna
cores: 4

Finally, this is the storage.conf configuration file:

dir: local
path /var/lib/vz
content images,iso,vztmpl,rootdir


iscsi: MSA2312i
portal 192.168.1.2
target iqn.1986-03.com.hp:storage.msa2312i.1042116a76
content none


lvm: LUN8TB (WHEN I CREATED THE VIRTUAL MACHINE, THIS WAS THE SELECTED DESTINATION FOR "DISK STORAGE")
vgname LUN8TB
base MSA2312i:0.0.1.scsi-3600c0ff00010e16fb4dfb64d01000000
shared
content images


dir: backup
path /backupVM
content backup

Well i think there's no much info about my problem....
Hope you Udo or anyone else who wants to join, could help me.

Again.... Thanks a lot!! this is too much important to me and i need this working.

 

[udo]

Hi Udo! thanks for being here so quickly!!

Well... let's start answering you:

1) qm start 101 (VID = 101 in this case) says: virtual machine already running, start failed (obviously to me why)
...

Hi,
is the VM in the GUI also marked as running? Can you stop them there?
Other way you can kill the kvm-process - look with "ps aux | grep 101.pid" for the right PID to kill - "cat /var/run/qemu-server/101.pid" should also work to see the PID.

Your volumegroups looks ok - if you select a linux-live-cd for this VM (like grml), do you see the disk of the VM? I assume yes, because you have only one and the grub-loader was found.
BTW, why do you use scsi? Normaly ide or virtio (for linux) is the better choice. If I'm right, ubuntu use UUIDs in the fstab, so you can edit your VM-config to:

bootdisk: virtio0
virtio0: LUN8TB:vm-101-disk-1

On one machine i have trouble the the default caching, and all run's fine with:

bootdisk: virtio0
virtio0: LUN8TB:vm-101-disk-1,cache=writethrough

You can try that also.

Udo

Remark: after changing the config-file you must stop/start the VM!