[SOLVED]: ZFS over iSCSI Connectivity Weirdness

[EagleTG]

Hi all,

Running into a strange issue with Proxmox VMs that connect to a FreeNAS device via ZFS over iSCSI.

It seems that my FreeNAS box had some issues early this morning, posted a thread in the FreeNAS forum to check out those issues.

I noticed some odd behavior on one of the Proxmox hosted VMs. It looked like there was some underlying EXT4 corruption:

Oct 29 06:55:30 [servername] kernel: [2292027.838643] EXT4-fs warning (device dm-0): ext4_dirent_csum_verify:367: inode #917781: comm unattended-upgr: No space for directory leaf checksum. Please run e2fsck -D.

And I was seeing odd errors when attempting to run certain binary executables:

bash: /sbin/e2fsck: cannot execute binary file: Exec format error

I shut down the VM to attempt running fsck from a Live Session, and was unable to get it to boot. Errors from the UI were 'TASK ERROR: start failed: QEMU exited with code 1' and further details indicated:

iscsiadm: No session found.
iscsiadm: No session found.
kvm: -drive file=iscsi://172.20.___.___/iqn.2005-10.org.freenas.ctl:freenasblk01/1,if=none,id=drive-scsi0,cache=writethrough,discard=on,format=raw,aio=threads,detect-zeroes=unmap: iSCSI: Failed to connect to LUN : SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:LOGICAL_UNIT_NOT_SUPPORTED(0x2500)
TASK ERROR: start failed: QEMU exited with code 1

It seems that connectivity is functioning, certain VMs are actually online and running (hosted on the FreeNAS machine). Strangely the Proxmox hosts report 'No active sessions.' when I run 'iscsiadm -m session'.

I attempted manually connecting with the following on one of my hosts:

iscsiadm -m node --loginall=automatic

Which succeeds:

Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:freenasblk01, portal: 172.20.___.___,3260] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:freenasblk01, portal: 172.20.___.___,3260] successful.

And shows the connection as active now:

____@____:/etc/pve/qemu-server# iscsiadm -m session
tcp: [1] 172.20.___.___:3260,1 iqn.2005-10.org.freenas.ctl:freenasblk01 (non-flash)

But still won't let me connect/start the VM (at least the error has changed):

Rescanning session [sid: 1, target: iqn.2005-10.org.freenas.ctl:freenasblk01, portal: 172.20.___.___,3260]
Rescanning session [sid: 1, target: iqn.2005-10.org.freenas.ctl:freenasblk01, portal: 172.20.___.___,3260]
kvm: -drive file=iscsi://172.20.___.___/iqn.2005-10.org.freenas.ctl:freenasblk01/1,if=none,id=drive-scsi0,cache=writethrough,discard=on,format=raw,aio=threads,detect-zeroes=unmap: iSCSI: Failed to connect to LUN : SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:LOGICAL_UNIT_NOT_SUPPORTED(0x2500)
TASK ERROR: start failed: QEMU exited with code 1

I'm not entirely sure what the proper next steps are. I am considering a full environment reboot, but that will take me some time to plan out. I've already tried rebooting one of the Proxmox hosts (after successfully migrating the running VMs off of it), and that has not made the situation any better.

Thanks.

 

[EagleTG]

Rebooting the environment, including the FreeNAS machine has brought everything back to "relatively normal". I'm still digging for a root cause, will be posting a thread here with another simple question.