f*ed up VM storage setup - need help recovering VM disk ZFS dataset

[dakazze]

First off, this is a super embarrassing fuckup that happened because I rushed setting up my backup server...

Creating some VMs I did not check the storage location which defaulted to a NAS NFS share. The NAS is still there and the NFS share is working but after rebooting the NAS while the VMs were running I cant start these VMs because "TASK ERROR: volume 'NAS:103/vm-103-disk-0.qcow2' does not exist".
One of these VMs involves a lot of work to setup again so I would really appreciate if there was a way to get it back up so I can move the disks where they belong.

There have been no major changes to the NAS dataset but rebooting the host did not help.

Thats the mountpoint set in the vm.conf:

scsi0: NAS:103/vm-103-disk-0.qcow2,iothread=1,size=32G

root@proxBu:/mnt# zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
rpool                          101G   728G    25K  /rpool
rpool/ROOT                    5.65G   728G    24K  /rpool/ROOT
rpool/ROOT/pve-1              5.65G   728G  5.65G  /
rpool/data                    57.1G   728G    25K  /rpool/data
rpool/data/subvol-100-disk-1   840M  99.2G   840M  /rpool/data/subvol-100-disk-1
rpool/data/subvol-104-disk-0  12.0M  3.99G  11.8M  /rpool/data/subvol-104-disk-0
rpool/data/vm-101-disk-0      35.5K   728G  35.5K  -
rpool/data/vm-101-disk-1      11.0G   728G  11.0G  -
rpool/data/vm-102-disk-0      81.5K   728G  44.5K  -
rpool/data/vm-103-disk-0      26.7G   728G  26.7G  -
rpool/data/vm-105-disk-0      10.7G   728G  10.2G  -
rpool/data/vm-105-state-uh    2.22G   728G  2.22G  -
rpool/data/vm-106-disk-0      5.66G   728G  5.66G  -
rpool/var-lib-vz              38.3G   728G  38.3G  /var/lib/vz

I am a noob when it comes to ZFS so my first thought was to use zfs set mountpoint for vm-103-disk which "does not apply to datasets of this type"

Any ideas?

 

[miguelsousa]

If I understood you probably have a locked nfs mount because of nas reboot. Did you reboot the host?

 

[dakazze]

If I understood you probably have a locked nfs mount because of nas reboot. Did you reboot the host?

Sorry, sometimes I have a hard time expressing myself in english.

The short Version:

1. Created VM and mistakenly set storage to NAS (different machine) which is a NFS share
2. I did not notice this mistake and everything worked fine for 2-3 weeks
3. I rebooted the NAS while the VM was still up because I did not know yet
4. Ever since I cant start the VM because 'NAS:103/vm-103-disk-0.qcow2' does not exist"
5. rebooted the host --> NFS share is accessible, path is the same, got R/W access
6. VM still cant boot because it cant find its boot disk
7. frustration / no idea what to do / angry at self for retarded mistake

 

[waltar]

1. Created VM and mistakenly set storage to NAS (different machine) which is a NFS share
2. I did not notice this mistake and everything worked fine for 2-3 weeks

That's not a bad mistake as we have all vm's and lxc's on nas also and works great.

3. I rebooted the NAS while the VM was still up because I did not know yet

That's ok as the kernel slow down (until 0) the app which is from kernel side the vm as it cannot temp. write I/O data to unavailable nfs-server. When nfs mount valid again outstanding data is written and vm get cpu slices (for generating new I/O) again.

4. Ever since I cant start the VM because 'NAS:103/vm-103-disk-0.qcow2' does not exist"
5. rebooted the host --> NFS share is accessible, path is the same, got R/W access
6. VM still cant boot because it cant find its boot disk

Problem on nfs server !! Maybe mostly (?) related to your zfs setup there.

7. frustration / no idea what to do / angry at self for retarded mistake

I understand you but still there's no mistake to vm storage definition.