[SOLVED] missing VM disks images

M

MarcinR

New Member
Proxmox Subscriber
Feb 10, 2025
5
1
3
Hello,

My Proxmox lost all VM disk images. Folder /mnt/vm-storage/images/ contains only last created machine. All other machines work, but after stop/start I have f.e. "volume 'vm-storage:104/vm-104-disk-2.qcow2' does not exist". Machines wosrks, but I can't shutdown, backup.
Haw can I rescu working yet machines?

Best regards
Marcin
 
Hi @MarcinR , welcome to the forum.

The path mentioned in your post is not a standard path used by default PVE install. It sounds like you may have lost a disk? Perhaps it was externally connected?

What is the output of the following commands:
cat /etc/pve/storage.cfg
pvesm status
df
mount
lsbk

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
  • Like
Reactions: waltar
Code: 
$ sudo cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content iso,backup
    shared 0

Code: 
dir: vm-storage
    path /mnt/vm-storage
    content vztmpl,images,rootdir
    prune-backups keep-all=1
    shared 0

Code: 
$ sudo pvesm status
Name              Type     Status           Total            Used       Available        %
local              dir     active       759227464        10513712       716813952    1.38%
vm-storage         dir     active      9688270628      1364904504      7835029536   14.09%

Code: 
$ df
Filesystem            1K-blocks       Used  Available Use% Mounted on
udev                   65878528          0   65878528   0% /dev
tmpfs                  13182468       1832   13180636   1% /run
/dev/mapper/pve-root  759227464   10513736  716813928   2% /
tmpfs                  65912336      70608   65841728   1% /dev/shm
tmpfs                      5120          0       5120   0% /run/lock
efivarfs                    512         92        416  19% /sys/firmware/efi/efivars
/dev/sda2               1046508      11912    1034596   2% /boot/efi
/dev/sdb             9688270628 1364904512 7835029528  15% /mnt/vm-storage
/dev/fuse                131072         44     131028   1% /etc/pve
tmpfs                  13182464          0   13182464   0% /run/user/1000

Code: 
$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=65878528k,nr_inodes=16469632,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13182468k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro,stripe=64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15171)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdb on /mnt/vm-storage type ext4 (rw,relatime,stripe=64)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=13182464k,nr_inodes=3295616,mode=700,uid=1000,gid=1000,inode64)

Code: 
$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda            8:0    0 745.2G  0 disk
├─sda1         8:1    0  1007K  0 part
├─sda2         8:2    0     1G  0 part /boot/efi
└─sda3         8:3    0 744.2G  0 part
  ├─pve-swap 252:0    0     8G  0 lvm  [SWAP]
  └─pve-root 252:1    0 736.2G  0 lvm  /
sdb            8:16   0   9.1T  0 disk /mnt/vm-storage
 
Last edited:
Please use "CODE" tags when pasting information - it makes for easier human processing.

Do you see anything interesting in the output of :
dmesg
journalctl -n 500

Have you examined the context of the /mnt/vm-storage ? (ls -R) Are the files and relevant directory structure there?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
MarcinR said:
Can this be fixed?
Click to expand...
The easiest way is to restore from backup.

You can also try "ddrescue", although results and data integrity are not guaranteed.

For the VMs that are still running:
https://superuser.com/questions/283...ed-file-if-it-is-still-opened-by-some-process

Other options:
https://unix.stackexchange.com/questions/80270/recover-deleted-files-on-linux

Relevant google search : "linux recover deleted" and/or "linux recover deleted file still open"

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
I tried this solution:
https://www.geeksforgeeks.org/how-to-recover-a-deleted-file-in-linux/

I found the PID of the process that is holding open image file:

Code: 
$ ps -ef | grep cc-wiki-old | grep -v grep
root      747349       1  1  2024 ?        1-02:10:26 /usr/bin/kvm -id 102 -name cc-wiki-old,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/102.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/102.pid -daemonize -smbios type=1,uuid=7cbe185d-9513-49fd-88d1-20e52fc5f79c -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/102.vnc,password=on -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 2048 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device vmgenid,guid=6a3b96df-d68e-4320-807e-8f0de9d2a2d8 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:5bcf0c83347 -drive file=/mnt/vm-storage/images/102/vm-102-disk-0.raw,if=none,id=drive-ide0,format=raw,cache=none,aio=io_uring,detect-zeroes=on -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100 -drive if=none,id=drive-ide2,media=cdrom,aio=io_uring -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -netdev type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=86:1...2:F3,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=300 -machine type=pc+pve0

but when I copy descriptor to file:

Code: 
$ sudo cp /proc/747349/fd/3 /tmp/
cp: cannot open '/proc/747349/fd/3' for reading: No such device or address

descriptor points to socket no file:

Code: 
$ sudo ls -l /proc/747349/fd/3
lrwx------ 1 root root 64 Feb 10 17:12 /proc/747349/fd/3 -> 'socket:[6234978]'

Code: 
$ sudo lsof | grep -i vm-102
kvm        747349                         root   24u      REG               8,16   32212254720   62521346 /mnt/vm-storage/images/102/vm-102-disk-0.raw (deleted)
kvm        747349  747350 call_rcu        root   24u      REG               8,16   32212254720   62521346 /mnt/vm-storage/images/102/vm-102-disk-0.raw (deleted)
kvm        747349  747419 CPU\x200/       root   24u      REG               8,16   32212254720   62521346 /mnt/vm-storage/images/102/vm-102-disk-0.raw (deleted)
kvm        747349  747422 vnc_worke       root   24u      REG               8,16   32212254720   62521346 /mnt/vm-storage/images/102/vm-102-disk-0.raw (deleted)
kvm        747349  747705 iou-wrk-7       root   24u      REG               8,16   32212254720   62521346 /mnt/vm-storage/images/102/vm-102-disk-0.raw (deleted)


How restore file from socket?
 
Last edited:
Success!

Code: 
$ sudo ls -l /proc/747349/fd/
total 0
lrwx------ 1 root root 64 Feb 10 17:29 0 -> /dev/null
lrwx------ 1 root root 64 Feb 10 17:29 1 -> /dev/null
lrwx------ 1 root root 64 Feb 10 17:29 10 -> 'socket:[6225493]'
lrwx------ 1 root root 64 Feb 10 17:29 11 -> 'socket:[6222410]'
lrwx------ 1 root root 64 Feb 10 17:29 12 -> 'anon_inode:[io_uring]'
lr-x------ 1 root root 64 Feb 10 17:29 13 -> 'pipe:[6241802]'
lrwx------ 1 root root 64 Feb 10 17:29 14 -> /dev/kvm
lrwx------ 1 root root 64 Feb 10 17:29 15 -> anon_inode:kvm-vm
lrwx------ 1 root root 64 Feb 10 17:29 16 -> /dev/net/tun
lrwx------ 1 root root 64 Feb 10 17:29 17 -> /dev/vhost-net
lrwx------ 1 root root 64 Feb 10 17:29 18 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 19 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 2 -> 'socket:[6222411]'
lrwx------ 1 root root 64 Feb 10 17:29 20 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 21 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 22 -> anon_inode:kvm-vcpu:0
]lr-x------ 1 root root 64 Feb 10 17:29 23 -> anon_inode:kvm-vcpu-stats:0
lrwx------ 1 root root 64 Feb 10 17:29 24 -> '/mnt/vm-storage/images/102/vm-102-disk-0.raw (deleted)'
lrwx------ 1 root root 64 Feb 10 17:29 26 -> 'socket:[6222411]'
lrwx------ 1 root root 64 Feb 10 17:29 27 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 28 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 29 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:12 3 -> 'socket:[6234978]'
lrwx------ 1 root root 64 Feb 10 17:29 30 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 31 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 32 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 4 -> 'anon_inode:[eventfd]'
l-wx------ 1 root root 64 Feb 10 17:29 5 -> /run/qemu-server/102.pid
l-wx------ 1 root root 64 Feb 10 17:29 6 -> 'pipe:[6235793]'
lrwx------ 1 root root 64 Feb 10 17:29 7 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 Feb 10 17:29 8 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Feb 10 17:29 9 -> 'anon_inode:[eventfd]'

Code: 
$ sudo cp /proc/747349/fd/24 /tmp/
$ sudo ls -l /tmp/
total 9723912
-rw-r----- 1 root root 32212254720 Feb 11 12:21 24
$ sudo mv /tmp/24 /mnt/vm-storage/images/102/vm-102-disk-0.raw
 
Last edited:
  • Like
Reactions: leesteken
You must log in or register to reply here.
Top Bottom
Back