Volume does not exist - Please help me to not screw up

S

siege801

Member
Jun 9, 2022
6
1
8
Hello Everyone,

I'm quite a new Proxmox user having converted from ESXi. I have a homelab running on an Intel NUC with the following disk hardware, partition and usage formation:
  • 1 x M.2 1Tb (/dev/nvme0n1)
    • Full disk is xfs formatted, used to hold the Virtual Machines
  • 1 x SSD 2Tb (/dev/sda)
    • 80Gb LVM used for the Proxmox installation (along with relevant BIOS boot and EFI partitions)
    • /dev/sda4 -> ~500Gb xfs formatted, this was then added as a Directory in the GUI. Then "attached" (may be the wrong word) to the guest that was to use it - aka guest1).
    • /dev/sda5 -> ~1.3Tb xfs formatted, as above, this was then added as a Directory in the GUI. Then "attached" to a different guest - aka guest2.
It's the last two 'partitions' that I feel are now in danger. Here's the story.

I created the two guests, (guest1 / vmid 107, and guest2 / vmid 104).

I installed the guest OSes and started to configure them (they're both running independent Ubuntu/Nextcloud stacks fwiw). I came to a point where I needed to restart the two installations - so I created guest3 and guest4. I added the virtio lines to the two new guests' /etc/pve/qemu-server/xxx.conf files
  • For guest3 (which becomes the new guest1) -> virtio1: 500Gb_Dir:107/vm-107-disk-0.qcow2 (note, the new vmid for this guest is 110)
  • For guest4 (which becomes the new guest2) -> virtio1: 1.3Tb_Dir:104/vm-104-disk-0.qcow2 (note, the new vmid for this guest is 112)
I then detached the "Hard Disks" from the old guests (guest1 and guest2), and deleted guest1 (vmid 107) and guest2 (vmid 104). I confirmed the data was still accessible to guest3 and guest4 - and everything seemed (and still seems) to be ok.

However - My concern now is that when I attempt a backup of either guest3 or guest4 I get the error:
volume '500Gb_Dir:107/vm-107-disk-0.qcow2' does not exist

Both of the guests still have a disk attached - I just don't know where that disk is - if that makes sense. My concerns are:
  1. First and foremost - if I understand enough of what's going on - it's likely that if either of the two guests (or indeed the host) are rebooted - the data partitions won't be visible to the guest(s).
  2. Also bad news - I can't currently backup these guests. That said, I do have the data backed up to a guest on another pve node via borg.

As a result - I don't know what to do to remedy the situation. Accordingly - any help would be greatly appreciated. Information dumps below:


Host
Code: 
blkid
<snip>
/dev/sda4: UUID="e38ab4c4-41f4-4dda-8db7-xxxxxx" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="d59e19cb-239f-a644-8869-xxxxx"
/dev/sda5: UUID="82515bae-b3f1-41ba-95e7-xxxxxx" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="69977dd7-bbb4-8947-87a9-xxxxxx"
</snip>

cat /etc/pve/qemu-server/110.conf | grep virtio
virtio0: NUC_M2_1Tb:110/vm-110-disk-0.qcow2,size=11468M
virtio1: 500Gb_Dir:107/vm-107-disk-0.qcow2

cat /etc/pve/qemu-server/112.conf | grep virtio
virtio0: NUC_M2_1Tb:112/vm-112-disk-0.qcow2,size=11468M
virtio1: 1.3Tb_Dir:104/vm-104-disk-0.qcow2

# fdisk -l /dev/sda
Device          Start        End    Sectors  Size Type
/dev/sda1          34       2047       2014 1007K BIOS boot
/dev/sda2        2048    1050623    1048576  512M EFI System
/dev/sda3     1050624  157286400  156235777 74.5G Linux LVM
/dev/sda4   157288448 1205864447 1048576000  500G Linux filesystem
/dev/sda5  1205864448 3907029134 2701164687  1.3T Linux filesystem

guest3 (vmid 110)
Code: 
blkid
<snip>
/dev/mapper/500Gb: LABEL="500Gb" UUID="66a1950b-7029-4ba3-8c24-dc82" BLOCK_SIZE="512" TYPE="xf
s"
</snip>

df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/500G  500G   19G  482G   4% /mnt/ncdata_intra


guest4 (vmid 112)
Code: 
blkid
<snip>
/dev/mapper/1.3Tb: LABEL="1.3Tb" UUID="1e9ab81b-ebea-408c-b9d4-01fdfc" BLOCK_SIZE="512" TYPE="xfs"
</snip>

df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/1.3Tb  1.3T   18G  1.3T   2% /mnt/ncdata_ext
 
Thanks for replying!

Sure thing
Code: 
# cat /etc/pve/storage.cfg  
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

dir: NUC_M2_1Tb
        path /mnt/pve/NUC_M2_1Tb
        content iso,vztmpl,rootdir,backup,images,snippets
        is_mountpoint 1
        nodes nucpve

dir: HP_SATA_160G
        path /mnt/HP_SATA_160G
        content images,snippets,iso,rootdir,vztmpl,backup
        prune-backups keep-all=1
        shared 0

dir: 500Gb_Dir
        path /mnt/pve/500Gb_Dir
        content snippets,images,backup,rootdir,iso,vztmpl
        is_mountpoint 1
        nodes nucpve

dir: 1.3Tb_Dir
        path /mnt/pve/1.3Tb_Dir
        content images,snippets,iso,rootdir,vztmpl,backup
        is_mountpoint 1
        nodes nucpve

nfs: NFS_STORE
        export /mnt/1Tb_Dumps/NFS_STORE
        path /mnt/pve/NFS_STORE
        server 192.168.10.4
        content backup
prune-backups keep-all=1

And if it's helpful:
Code: 
/mnt/pve# tree
├── 1.3Tb_Dir
│   ├── dump
│   ├── images
│   ├── private
│   ├── snippets
│   └── template
│       ├── cache
│       └── iso
├── 500Gb_Dir
│   ├── dump
│   ├── images
│   ├── private
│   ├── snippets
│   └── template
│       ├── cache
│       └── iso

And this:
Code: 
# find / -type f -name *.qcow2
/var/lib/vz/template/qemu/haos_ova-8.2.qcow2
find: ‘/mnt/pve/HP_SATA_1Tb’: Stale file handle
/mnt/pve/NUC_M2_1Tb/images/2003/vm-2003-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/103/vm-103-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/103/vm-103-disk-0.qcow2
/mnt/pve/NUC_M2_1Tb/images/105/vm-105-disk-0.qcow2
/mnt/pve/NUC_M2_1Tb/images/105/vm-105-disk-1.qcow2
/mnt/pve/NUC_M2_1Tb/images/105/vm-105-disk-2.qcow2
/mnt/pve/NUC_M2_1Tb/images/106/vm-106-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/106/vm-106-disk-0.qcow2
/mnt/pve/NUC_M2_1Tb/images/2002/vm-2002-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/2002/base-2002-disk-0.qcow2
/mnt/pve/NUC_M2_1Tb/images/108/vm-108-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/110/vm-110-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/110/vm-110-disk-0.qcow2
/mnt/pve/NUC_M2_1Tb/images/112/vm-112-cloudinit.qcow2
/mnt/pve/NUC_M2_1Tb/images/112/vm-112-disk-0.qcow2
 
Hi,
can you check with lsof, if there's still a file descriptor to the image open? For example (for VM 102 in my case):
Code: 
root@pve701 ~ # lsof -p $(cat /var/run/qemu-server/102.pid) | grep qcow2
kvm     35218 root   14u      REG              253,4 4295884800 528283 /var/lib/vz/images/102/vm-102-disk-0.qcow2 (deleted)
Then you can use the PID (35218 in my case) and the number of the fd (14 in my case) to copy the image:
Code: 
root@pve701 ~ # cp /proc/35218/fd/14 /var/lib/vz/images/102/vm-102-disk-0.qcow2

Or save it to a different location and check the contents. Afterwards I'd rename the image so that the ID matches the VM it belongs to. You'll also need to adapt the configuration file then of course.
 
Hi again,

I think we're almost there - and I couldn't be more thankful for your help!

So far I've done (and found) the following:
Code: 
kvm     2532011 root   11r      REG              259,1      4521984 536871049 /mnt/pve/NUC_M2_1Tb/images/110/vm-110-cloudinit.qcow2
kvm     2532011 root   15u      REG              259,1  12027166720 536871050 /mnt/pve/NUC_M2_1Tb/images/110/vm-110-disk-0.qcow2
kvm     2532011 root   16u      REG                8,4 536953094144       134 /mnt/pve/500Gb_Dir/images/107/vm-107-disk-0.qcow2 (deleted)

The word deleted there had me worried. But I continued on.

I couldn't do exactly as you described, because /var/lib/vz/images is on the root partition which wasn't large enough for the qcow2 file. So instead I did:

Code: 
cp /proc/2532011/fd/16 /mnt/pve/500Gb_Dir/images/110/ncdata_intra-data.qcow2

/mnt/pve/500Gb_Dir/images# ls -alh
total 40G
drwxr-xr-x 2 root root   37 Aug 19 21:00 .
drwxr-xr-x 7 root root   79 Jul  4 21:07 ..
-rw-r----- 1 root root 501G Aug 19 21:18 ncdata_intra-data.qcow2

So this is looking pretty good. I'm just not positive on how to direct the vm config to this qcow2 file now. Currently the config has:

Code: 
virtio1: 500Gb_Dir:107/vm-107-disk-0.qcow2

Would I be correct thinking it should become this:

Code: 
virtio1: 500Gb_Dir:110/ncdata_intra-data.qcow2

I figure the config line works as follows:

virtio1: (the device name)
500Gb_Dir (the directory name within /mnt/pve
110 (the vmid sub directory)
ncdata[...]qcow2 (the file itself)


You also said I should check the contents, but I'm uncertain on how to do that.

Thank again for the help so far!
 
siege801 said:
Hi again,

I think we're almost there - and I couldn't be more thankful for your help!

So far I've done (and found) the following:
Code: 
kvm     2532011 root   11r      REG              259,1      4521984 536871049 /mnt/pve/NUC_M2_1Tb/images/110/vm-110-cloudinit.qcow2
kvm     2532011 root   15u      REG              259,1  12027166720 536871050 /mnt/pve/NUC_M2_1Tb/images/110/vm-110-disk-0.qcow2
kvm     2532011 root   16u      REG                8,4 536953094144       134 /mnt/pve/500Gb_Dir/images/107/vm-107-disk-0.qcow2 (deleted)

The word deleted there had me worried. But I continued on.
Click to expand...
It means that the file on the filesystem was deleted. But luckily, the kvm process still has the file descriptor open.

siege801 said:
I couldn't do exactly as you described, because /var/lib/vz/images is on the root partition which wasn't large enough for the qcow2 file. So instead I did:

Code: 
cp /proc/2532011/fd/16 /mnt/pve/500Gb_Dir/images/110/ncdata_intra-data.qcow2

/mnt/pve/500Gb_Dir/images# ls -alh
total 40G
drwxr-xr-x 2 root root   37 Aug 19 21:00 .
drwxr-xr-x 7 root root   79 Jul  4 21:07 ..
-rw-r----- 1 root root 501G Aug 19 21:18 ncdata_intra-data.qcow2

So this is looking pretty good. I'm just not positive on how to direct the vm config to this qcow2 file now. Currently the config has:

Code: 
virtio1: 500Gb_Dir:107/vm-107-disk-0.qcow2

Would I be correct thinking it should become this:

Code: 
virtio1: 500Gb_Dir:110/ncdata_intra-data.qcow2

I figure the config line works as follows:

virtio1: (the device name)
500Gb_Dir (the directory name within /mnt/pve
110 (the vmid sub directory)
ncdata[...]qcow2 (the file itself)
Click to expand...
Yes, but this can't be done while the VM is still running. And Proxmox VE expects the name of the file to be of a certain form, e.g. vm-110-disk-<number>.qcow2 (replacing number with something that's not in use yet). That's why I'd check that everything is in place first!

siege801 said:
You also said I should check the contents, but I'm uncertain on how to do that.

Thank again for the help so far!
Click to expand...
Easiest is probably to attach the drive to some other VM and mount it there.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom