VM disk full, shutdown, now won't start

[Fortel]

I think I'm hosed. But I have an old (2 months) backup, so not completely hosed.

We have a Proxmox 3.3 host in a remote data center, running mostly OpenVZ instances. But we have 2 KVM machines running Windows. I have very little experience with KVM...

The drive on one of those filled to capacity. I was not able to resize it as I've done before, and shut the machine down thinking that might allow me to add capacity. But when I tried to start it again, a dreaded "TASK ERROR: volume 'Backups:105/vm-105-disk-1.qcow2' does not exist" appeared.

I'm not sure how this was functioning with a missing disk. I don't remember how I'd originally set this up (it's been a while!) but we have another KVM set up the same way, still running. I don't dare shut that one down until the disk has been migrated, if that's even possible.

So I'm wondering, is the VM at 105 gone for good?

Thanks a lot,

Peter

 

[wolfgang]

Hi,

I'm mot sure, but it sounds like your erase somehow the disk.
Check on the command-line if the disk is still there.

the path <Backups>/inmages/105/vm-105-disk-1.qcow2

 

[Fortel]

Hi,

I'm mot sure, but it sounds like your erase somehow the disk.
Check on the command-line if the disk is still there.

the path <Backups>/inmages/105/vm-105-disk-1.qcow2

 

[Fortel]

Thank you, Wolfgang...

The disk is definitely gone. This VM was set up a long time ago, along with another very similar VM. This one ran out of space- I shut it down, and discovered the disk is gone. The other VM is still running, but also has no disk!

Any chance of a recovery somehow? I've not rebooted the host...nor intentionally deleted anything.

Thanks again!

Peter

 

[wolfgang]

See
https://forum.proxmox.com/threads/qcow2-file-on-guest-suddenly-gone.26296/#post-132318

 

[Fortel]

See
https://forum.proxmox.com/threads/qcow2-file-on-guest-suddenly-gone.26296/#post-132318

 

[Fortel]

Thanks, Wolfgang.

I did find that post earlier today. But the difference is, my vm is not running. Seems my mistake (or one of my mistakes!) was to shut down the VM. That was before I realized the disk was missing.

I'm not sure if adding back an empty disk will bring life back to this VM... I've obviously got some study ahead of me.

Thanks again,

Peter

 

[udo]

Thanks, Wolfgang.

I did find that post earlier today. But the difference is, my vm is not running. Seems my mistake (or one of my mistakes!) was to shut down the VM. That was before I realized the disk was missing.

I'm not sure if adding back an empty disk will bring life back to this VM... I've obviously got some study ahead of me.

Thanks again,

Peter

Hi,
is the missing disk an important disk, or only configured for test some time ago? The psition on backup sound so...

Can you post your VM-config and storag-config?

cat /etc/pve/qemu-server/105.conf

cat /etc/pve/storage.cfg

Udo

 

[Fortel]

Hi Udo,

The missing disk is not important, but the VM appears to be dead without it. So, in that sense, perhaps the missing disk is important.

root@denpm1:~# cat /etc/pve/qemu-server/105.conf
bootdisk: ide1
cores: 2
ide1: Backups:105/vm-105-disk-1.qcow2,format=qcow2,cache=writeback,size=54G
ide2: cdrom,media=cdrom
memory: 4096
name: Fortel-Networks-Inc
net0: e1000=4A:9E:14:7F:64:63,bridge=vmbr0
ostype: win7
smbios1: uuid=8c55262b-a861-4c2a-906e-d3d8509d28f5
sockets: 2


root@denpm1:~# cat /etc/pve/storage.cfg
dir: backups
path /backups
content images,backup
maxfiles 1

dir: local
path /var/lib/vz
content images,iso,vztmpl,rootdir
maxfiles 0

nfs: Synology_RW_CAB1
path /mnt/pve/Synology_RW_CAB1
server <Proxmox IP address at a different location>
export /volume1/Proxmox_Backups
options vers=3
content images,backup
maxfiles 1

nfs: Proxmox_Backups_Denver
path /mnt/pve/Proxmox_Backups_Denver
server <Proxmox IP address at a different location>
export /volume1/Proxmox_Backups_Denver
options vers=3
content images,backup
maxfiles 3

dir: KVM_Backups
path /kvm_backups
shared
content images
maxfiles 1

lvm: KVM_Storage
vgname pve
content images

dir: Backups
path /var/lib/vz/backups
shared
content images,backup
maxfiles 2

nfs: Synology_Home
path /mnt/pve/Synology_Home
server <my IP address at home>
export /volume1/Home_Storage
options vers=3
content images,backup,rootdir
maxfiles 2

nfs: home_storage
path /mnt/pve/home_storage
server <my IP address at home>
export /volume1/home_storage
options vers=3
content images,rootdir,backup
maxfiles 2

nfs: Home
path /mnt/pve/Home
server <my IP address at home>
export /volume1/Essential_Backups/dump
options vers=3
content images,rootdir,backup
maxfiles 2

 

[udo]

Hi Udo,

The missing disk is not important, but the VM appears to be dead without it. So, in that sense, perhaps the missing disk is important.

the disks is the only disk of the vm - so they is important.

dir: Backups
path /var/lib/vz/backups
shared
content images,backup
maxfiles 2

Why is /var/lib/vz/backups shared? Is this an nfs-mount?
Look in /etc/fstab, perhaps you have unmount the share?
Or is the node part of an cluster and the VM was running on the other node? In this case, the data should be on the other node.

Udo

 

[Fortel]

When I first set these up, I followed a tutorial- and everything just worked. But clearly, something wasn't done correctly. Since the VM was running (and had been for a long time,) I thought it must be set up correctly. This is not a production machine, per se, but it did have data on it that is important to me. Not critical, but important. I am learning a lesson with this experience.

With OpenVZ containers, /var/lib/vz/backups is the directory I'd set up for shared local storage.

/etc/fstab has

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=ea030fab-ef0a-41d6-a649-01a9600e2d61 /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

Not sure of what to make of it.

If the VM is permanently gone because of the missing disk, I'll live. But it would be very nice to somehow resurrect...

Thanks Udo!!

Peter

 

[udo]

When I first set these up, I followed a tutorial- and everything just worked. But clearly, something wasn't done correctly. Since the VM was running (and had been for a long time,) I thought it must be set up correctly. This is not a production machine, per se, but it did have data on it that is important to me. Not critical, but important. I am learning a lesson with this experience.

With OpenVZ containers, /var/lib/vz/backups is the directory I'd set up for shared local storage.

/etc/fstab has

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=ea030fab-ef0a-41d6-a649-01a9600e2d61 /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

Not sure of what to make of it.

If the VM is permanently gone because of the missing disk, I'll live. But it would be very nice to somehow resurrect...

Thanks Udo!!

Peter

Hi Peter,
this sounds that someone delete the dir/file on the disk?! Look in the shell history.

BTW. it's make no dense to define an backup-dir on /var/lib/vz/backup, because /var/lib/vz are an allready defined storage (local).
And shared for an local dir make also no sense (the only exception is an manual synced iso-store to use live-migration with an cdrom-live-vm).

You can look, if the file was moved...

find /var/lib/vz -name vm-105-disk-1.qcow2 -ls

Udo

 

[Fortel]

I really appreciate your help, Udo. Even if this proves to be fruitless (which I'm suspecting will be the case.)

find /var/lib/vz -name vm-105-disk-1.qcow2 -ls <-- This turns up empty- nothing found.

The shell history is an interesting idea. My brief review of that file doesn't show any obvious clue. But I will study it more in the morning (just a few hours from now!) It's late here in California; I need some rest. I'll resume at daybreak.

Thanks again, Udo!

Peter