ZFS over iSCSI partially stopped to work

[j4sdk]

Hello.

Just upgraded 3-node cluster with latest Proxmox 6.2, and now run into the problems.

pve-manager/6.2-11/22fb4983 (running kernel: 5.4.60-1-pve)

iSCSI target is targetcli on kvm-fs node (running proxmox with zfs). Was working fine past year.
Nodes are interconnected via Mikrotik 10Gbps switch, with Mellanox Connect-X4 NICs. There is no problems with connectivity currently.

1) I can't restore any VM...

restore vma archive: lzop -d -c /mnt/pve/transit/dump/vzdump-qemu-105-2020_09_16-14_16_13.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp5575.fifo - /var/tmp/vzdumptmp5575
CFG: size: 568 name: qemu-server.conf
DEV: dev_id=1 size: 131072 devname: drive-efidisk0
DEV: dev_id=2 size: 107374182400 devname: drive-scsi0
CTIME: Wed Sep 16 14:16:15 2020
new volume ID is 'scsi-01:vm-100-disk-2'
new volume ID is 'scsi-01:vm-100-disk-3'
map 'drive-efidisk0' to 'iscsi://172.16.32.2/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/2' (write zeros = 1)
map 'drive-scsi0' to 'iscsi://172.16.32.2/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/3' (write zeros = 1)
vma: can't open file iscsi://172.16.32.2/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/2 - iSCSI: Failed to connect to LUN : Failed to log in to target. Status: Authorization failure(514)
/bin/bash: line 1:  5577 Broken pipe             lzop -d -c /mnt/pve/transit/dump/vzdump-qemu-105-2020_09_16-14_16_13.vma.lzo
      5578 Trace/breakpoint trap   | vma extract -v -r /var/tmp/vzdumptmp5575.fifo - /var/tmp/vzdumptmp5575
temporary volume 'scsi-01:vm-100-disk-2' sucessfuly removed
temporary volume 'scsi-01:vm-100-disk-3' sucessfuly removed
no lock found trying to remove 'create'  lock
TASK ERROR: command 'set -o pipefail && lzop -d -c /mnt/pve/transit/dump/vzdump-qemu-105-2020_09_16-14_16_13.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp5575.fifo - /var/tmp/vzdumptmp5575' failed: exit code 133

2) I can't start any VM on iscsi-over-zfs disks, it's just saying "TASK ERROR: start failed: QEMU exited with code -1" (no more info), however qm start 105 works just fine (via ssh, thank god).
3) I can't backup disks on iscs-over-zfs... (it's just freezes, waited for hour for 32gb disk).

Any idea what's going on and what to do ? Thank you.

 

[j4sdk]

Also, unable to move-edit zvols from Proxmox interface all failed with:

error during cfs-locked 'storage-kvm-fs-hdd' operation: command '/usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/172.16.32.2_id_rsa root@172.16.32.2 -- /usr/bin/targetcli /iscsi/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/tpg1/luns/ delete lun0' failed: Insecure dependency in exec while running with -T switch at /usr/share/perl/5.28/IPC/Open3.pm line 178. ...propagated at /usr/share/perl5/PVE/Storage/LunCmd/LIO.pm line 337.

 

[Deleted member 93625]

@j4sdk

iSCSI: Failed to connect to LUN : Failed to log in to target. Status: Authorization failure(514)

might be the clue?