VM with ZFS raw disks over iSCSI does not boot due to missing extents in TrueNAS

[linkstat]

Hello community.

I have a small environment of two Proxmox VE nodes pve-manager/7.2-7/d0dd0e85 (running kernel: 5.15.39-1-pve) + 1 TrueNAS storage (TrueNAS-12.0-U8.1 with freenas plugin 2.2.0-0 -beta1 from theGrandWazoo). I store my VMs using ZFS over iSCSI, and backup them using NFS.

Until a few days ago, everything was perfect. But one day, after a power outage, the TrueNAS server went "unconfigured" (maybe because I didn't save the configuration, but that doesn't matter) and most of the iSCSI extents are no longer listed, although the raw disks of the VMs are still there, intact.

Of course, this causes some VMs to start without problems, but others (those that do not have their corresponding extents in TrueNAS) do not, indicating with the error message:

TASK ERROR: Could not find lu_name for zvol vm-1020-disk-0 at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 118.

On TrueNAS, I can verify that there is no iSCSI extent for that VM, and I guess that is the cause of the problem. Because for the VMs that are listed, they can boot:




So, I decide to add an extent manually, indicating the corresponding device, for example:




Nice, now I have everything in order, I try to start the VM:




But the VM startup still fails...


So what am I missing? How do I link / edit the disk with its corresponding extent?

Now, to fix this, I'm creating new VMs with disks stored on TrueNAS via ZFS over iSCSI, and then from TrueNAS dd if=original_vm_image_disk of=new_vm_image_disk
but this takes a lot of time, and I'm sure there is a better way.

Any help will be appreciated.

Regards!

 

[LnxBil]

Now, to fix this, I'm creating new VMs with disks stored on TrueNAS via ZFS over iSCSI, and then from TrueNAS dd if=original_vm_image_disk of=new_vm_image_disk
but this takes a lot of time, and I'm sure there is a better way.

One way would be to create every disk new (exact! size) and just rename the underlying zvols on your TrueNAS server.

 

[eugenevdm]

I just had a situation similar to @linkstat .

In fact, I have it quite often when I have failed or aborted migrations between different disks using Proxmox and TrueNAS core. I often have a disk that 'stays behind' although the copy worked. I don't really worry about this these disks, until this evening.

I've found that when you use Proxmox to enlarge the disk, the Linux VM disk goes read-only. Of course I can shut down, but this is a real pain and not the same with local-lvm, where it always works perfectly online. Normally on some of my servers the read-only isn't a problem and I just reboot. This time however, the disk got disassociated from TrueNAS.

The big clue was this message when starting the VM:

TASK ERROR: Could not find lu_name for zvol vm-168-disk-0 at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 118.

At first I followed the path of maybe using `dd`. However, the output on TrueNAS is this:

root@truenas[~]# ls -la /dev/zvol/pool/vm-168-disk-0
lrwxrwxrwx 1 root root 11 Oct 24 23:22 /dev/zvol/pool/vm-168-disk-0 -> ../../zd240
root@truenas[~]# ls -lah /dev/zd240
brw-rw---- 1 root disk 230, 240 Oct 24 23:22 /dev/zd240

I believe this is a ZFS block device and `dd` works with it, but the syntax and aliasing looked too complicated for my skill set.

Then I followed @linkstat solution and recreated the extend. I then also created a target picking a random LUN ID that wasn't in use. Fortunately after recreating the extend and the target I was able to start the VM up again.