Accidentially deleted running VM with WinSCP - how to rstore image?

Nachtfalke

Member
Apr 7, 2013
46
0
6
Hi,

I was playing with WinSCP and went to my HDD where I stored the .qcow2 images.

/HDD500GB/images/100/

The VM was running and is still running at the moment - I think - on ProxmoxVE 3.0

Unfortunately I clicked delete on this folder because I wanted to delete something else. No the folder has gone. What to do to get this VM back?
I do not have any backup - this was something I want to do later...argh

I would really appreciate any help. I am a linux noob :(


--- edit ---
I found this chapter on the wiki:
http://pve.proxmox.com/wiki/Backup_-_Restore_-_Live_Migration

My VM is running as KVM - willt it be possible to do a Backup from the Proxmox GUI do do a complete Backup when going to "Backup --> Snapshot"?
 
Last edited:
Is the guest OS Linux? If so, are the mounted partitions managed by LVM?

If the data inside the VM is accessible, I'd create an LVM snapshot inside the VM, then use dd to make an image of the snapshot and transfer it over the network to another box, then restore it on a fresh VM and test:
http://www.tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

I don't think a migration would work because the source HDD file would be gone, so the migration script wouldn't be able to pick anything up, same with the VZdump scripts.
 
Well, you can try stopping the VM, unmounting the HDD500GB partition and undeleting the file:
http://extundelete.sourceforge.net/

I'd suggesting copying all information that you want to keep that's running inside the VM to another computer first. It's very likely that your disk image won't be recoverable and even if it is you may have corruption. It's also puzzling that you'd be able to delete the disk image of a running VM, it should be locked by the filesystem.
 
Hi Kyc,

thank you very much for your help and your suggestion.

We just started after I deleted the VM files to make backups of the database and system configuration files on the guest machine. And you are absolutly right - I don't know how it is possible to "delete" files which are in use. perhaps they aren't deleted until now but just invisible for me and if I shutdown the VM and proxmox will unlock the files then the system will delete the files.

Because we could login on the VM guest, we could start a tool which makes a backup of the running service config and database. We could copy this to an external storage and so on.

Your link offered some helpful info I am wollowing at the moment:
If you think the file may be still open by some program (for example, if it is a movie file currently being played by a movie player), and you know the filename, then first follow this procedure:
$ lsof|grep "/path/to/file"
progname 5559 user 22r REG 8,5 1282410 1294349 /path/to/file
Notice the number in the second column is 5559 and the number in the fourth column is 22. The command to restore that file is:
$ cp /proc/5559/fd/22 restored.file

For me it shows this:
root@ProxmoxVE1:~# lsof|grep "/HDD500GB/images/100/"
kvm 2850 root 19u REG 8,1 240724279296 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2
kvm 2850 2865 root 19u REG 8,1 240724279296 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2
kvm 2850 2868 root 19u REG 8,1 240724279296 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2

At the moment it is copying something with after I used this command:
root@ProxmoxVE1:~# cp /proc/2850/fd/19 /HDD500GB/restore.qcow2

Will see if I move this restored file to another ProxmoxVE server and try to create a VM with that if it will work and boot.
Will report back - probably tomorrow because I do not have access to all servers right now.
 
Hi Kyc,

thank you very much for your help and your suggestion.

We just started after I deleted the VM files to make backups of the database and system configuration files on the guest machine. And you are absolutly right - I don't know how it is possible to "delete" files which are in use. perhaps they aren't deleted until now but just invisible for me and if I shutdown the VM and proxmox will unlock the files then the system will delete the files.

Because we could login on the VM guest, we could start a tool which makes a backup of the running service config and database. We could copy this to an external storage and so on.

Your link offered some helpful info I am wollowing at the moment:


For me it shows this:


At the moment it is copying something with after I used this command:


Will see if I move this restored file to another ProxmoxVE server and try to create a VM with that if it will work and boot.
Will report back - probably tomorrow because I do not have access to all servers right now.
Hi,
this is the right way - see also here http://forum.proxmox.com/threads/9258-Deleted-image-of-running-vm

Don't know how reliable qcow can handle changes during copy... but perhaps an fsck do the right job after that.

Udo
 
Hmm...doesn't seem to work like I thought. When I created the VM I added 100GB as HDD space. The space used in realyti on the guest was 40GB.

When I started to copy from /proc I got a file size of 200GB still increasing...

Could this be because I restored the file on the same partition as the original, deleted file was?
 
Hi Udo,

yes - the machine seems to be very calm I think.

You say "stop" the machine but wouldn't this result in a deleted file and empty /proc ?
Or is this the same as the "Stop" option on the GUI?


I am asking this because we probably backuped everything and could restore this on a new VM. But this will cost some time and we need the server the next two weeks for education purposes and as long as it is running and "working" I would make sure that this will not "kill" my last hope ;-)

By the way:
It is very great to get such support from the community! :D
 
Hi Udo,

yes - the machine seems to be very calm I think.

You say "stop" the machine but wouldn't this result in a deleted file and empty /proc ?
Or is this the same as the "Stop" option on the GUI?


I am asking this because we probably backuped everything and could restore this on a new VM. But this will cost some time and we need the server the next two weeks for education purposes and as long as it is running and "working" I would make sure that this will not "kill" my last hope ;-)

By the way:
It is very great to get such support from the community! :D

Hi,
"kill -s stop" mean freeze the process - stop on the gui is power down (in your case the file is lost).
With "kill -s cont" the process is running again.

You can play with someting like this:
Code:
cp /dev/random xx&
rm xx

ps aux | grep  xx
udo   5941  0.0  0.0  22352   720 pts/0    S+   07:01   0:00 cp /dev/random xx

ls -lsa /proc/5941/fd/4
0 l-wx------ 1 udo udo 64 Jul 19 07:04 /proc/5941/fd/4 -> /home/udo/xx (deleted)

kill -s stop 5941
ps aux | grep  xx
udo   5941  0.0  0.0  22352   720 pts/0    T    07:01   0:00 cp /dev/random xx
Udo
 
Hi Udo,

I did what you said but it does not seem to work.
kill -s stop 2850
Then started the copy but still a file with 160GB increasing.

How could I make sure that the process really stopped?
Please remember - I am a linux n00b ;-)
 
Hi Udo,

I did what you said but it does not seem to work.

Then started the copy but still a file with 160GB increasing.

How could I make sure that the process really stopped?
Please remember - I am a linux n00b ;-)
Hi,
with "ps aux | grep 2850" you see the Flag T instead auf S for an stopped process:
Code:
# running:
root       76699  2.6  3.2 842372 535828 ?       Sl   Jun25 904:52 /usr/bin/kvm -id 102 ...

# stopped:
root       76699  2.6  3.2 842372 535828 ?       Tl   Jun25 904:55 /usr/bin/kvm -id 102 ...
I have copied an deleted qcow2-file from an stopped VM and it runs like expected:
Code:
/var/lib/vz/images/102# ls -lsa
total 1048280
      4 drwxr-xr-x  2 root root       4096 Aug  9  2012 .
      4 drwxr-xr-x 10 root root       4096 May 31 11:56 ..
1048272 -rw-r--r--  1 root root 1072497152 Jul 19 10:33 vm-102-disk-1.qcow2

rm vm-102-disk-1.qcow2
ls -lsa /proc/76699/fd/15
0 lrwx------ 1 root root 64 Jul 19 10:36 /proc/76699/fd/15 ->  (deleted)/var/lib/vz/images/102/vm-102-disk-1.qcow2
kill -s stop 76699
cp /proc/76699/fd/15 vm-102-disk-1.qcow2.rec
root@proxtest4:/var/lib/vz/images/102# ls -lsa
total 2096792
      4 drwxr-xr-x  2 root root       4096 Jul 19 10:38 .
      4 drwxr-xr-x 10 root root       4096 May 31 11:56 ..
1048392 -rw-r--r--  1 root root 1072497152 Jul 19 10:38 vm-102-disk-1.qcow2.rec

# after "kill -s cont 76699" + shutdown the VM
 mv vm-102-disk-1.qcow2.rec vm-102-disk-1.qcow2

# the VM start normal...
don't know why your new image produce so much data.

What is the output of
Code:
qemu-img info /proc/2850/fd/19
Udo
 
Hi Udo,

thank you very much for your help:

root@ProxmoxVE1:~# qemu-img info /proc/2850/fd/19
image: /proc/2850/fd/19
file format: qcow2
virtual size: 100G (107374182400 bytes)
disk size: 150G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 Basis 0 2013-03-18 16:58:12 00:00:00.000
2 Betriebsfertig 0 2013-04-22 13:47:53 615:43:03.900
3 Backup 0 2013-04-27 15:30:01 737:23:27.527
4 Backup2 0 2013-05-28 21:33:31 670:21:27.933
5 Backup3 0 2013-06-28 14:57:46 1407:43:07.477
root@ProxmoxVE1:~#


This is when doing ps -aux on the normal running machine:
root@ProxmoxVE1:~# ps aux | grep 2850
root 2850 10.1 37.2 3614156 1501316 ? Rl Jul16 457:26 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -name W7-MOTOROLA-HPA -smp sockets=1,cores=1 -cpu host -nodefaults -boot menu=on -vga std -no-hpet -k de -m 3072 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive file=/var/lib/vz/template/iso/VirtIO-Win-0.1-59.iso,if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/HDD500GB/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,mac=00:19:99:7F:42:44,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -global kvm-pit.lost_tick_policy=discard
root 2863 0.1 0.0 0 0 ? S Jul16 7:57 [vhost-2850]
root 93816 0.0 0.0 7784 912 pts/0 S+ 12:06 0:00 grep 2850
root@ProxmoxVE1:~#
This when try to freeze the process:
root@ProxmoxVE1:~# kill -s stop 2850
root@ProxmoxVE1:~#
root@ProxmoxVE1:~# ps aux | grep 2850
root 2850 10.1 37.2 3614156 1501316 ? Tl Jul16 457:30 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -name W7-MOTOROLA-HPA -smp sockets=1,cores=1 -cpu host -nodefaults -boot menu=on -vga std -no-hpet -k de -m 3072 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive file=/var/lib/vz/template/iso/VirtIO-Win-0.1-59.iso,if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/HDD500GB/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,mac=00:19:99:7F:42:44,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -global kvm-pit.lost_tick_policy=discard
root 2863 0.1 0.0 0 0 ? S Jul16 7:57 [vhost-2850]
root 93839 0.0 0.0 7784 912 pts/0 S+ 12:07 0:00 grep 2850
root@ProxmoxVE1:~#
Nothing changed, didn't it?

When I do lsof I get this:
root@ProxmoxVE1:~# lsof|grep "/HDD500GB/images/100/"
kvm 2850 root 19u REG 8,1 240753901568 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2
kvm 2850 2865 root 19u REG 8,1 240753901568 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2
kvm 2850 2868 root 19u REG 8,1 240753901568 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2
kvm 2850 93835 root 19u REG 8,1 240753901568 30359555 (deleted)/HDD500GB/images/100/vm-100-disk-1.qcow2
root@ProxmoxVE1:~#
Is there some other process 2868 and/or 2865 using this file also? Do I need to freeze more processes?

Thank you for your time!
 
Hi,
I guess the amoung of data came due the snapshots?!?

Your kvm-process is freezed - that's ok.

Try to extract the hdd as raw:
Code:
qemu-img convert -f qcow2 -O raw /proc/2850/fd/19 vm-100-disk-1.raw
this must produce a raw-image of the hdd of 100GB.

Udo
 
Hi,

will try with this command to the the process in percentage:
qemu-img convert -p -f qcow2 -O raw /proc/2850/fd/19 vm-100-disk-1.raw

--- edit ---
Looks different than before. The destination file has a size of ~100GB and does not increase even if the pecentage value increases. Will let it run now and try the rest from home using VPN.
Thank you so much intil now. Will report back the result or further questions :p
 
Last edited:
Hi again,

want to let you know that it worked. I got a 100GB raw image, converted it back to qcow2 and then startet it in a test VM with success.
After that I stopped the running VM and placed the qcow2 image in there and rebootet the machine. It worked like a charm.

Thank you very much for helping me with that. It will save me/us much time to reconfigure the server and do all the configuration.

Great to get such good and fast feedback here in the forum! :)

Thank you and have a nice weekend!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!