ZFS over iSCSI partially stopped to work

j4sdk

New Member
Sep 16, 2020
2
0
1
35
Hello.

Just upgraded 3-node cluster with latest Proxmox 6.2, and now run into the problems.
Code:
pve-manager/6.2-11/22fb4983 (running kernel: 5.4.60-1-pve)

iSCSI target is targetcli on kvm-fs node (running proxmox with zfs). Was working fine past year.
Nodes are interconnected via Mikrotik 10Gbps switch, with Mellanox Connect-X4 NICs. There is no problems with connectivity currently.

1) I can't restore any VM...
Code:
restore vma archive: lzop -d -c /mnt/pve/transit/dump/vzdump-qemu-105-2020_09_16-14_16_13.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp5575.fifo - /var/tmp/vzdumptmp5575
CFG: size: 568 name: qemu-server.conf
DEV: dev_id=1 size: 131072 devname: drive-efidisk0
DEV: dev_id=2 size: 107374182400 devname: drive-scsi0
CTIME: Wed Sep 16 14:16:15 2020
new volume ID is 'scsi-01:vm-100-disk-2'
new volume ID is 'scsi-01:vm-100-disk-3'
map 'drive-efidisk0' to 'iscsi://172.16.32.2/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/2' (write zeros = 1)
map 'drive-scsi0' to 'iscsi://172.16.32.2/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/3' (write zeros = 1)
vma: can't open file iscsi://172.16.32.2/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/2 - iSCSI: Failed to connect to LUN : Failed to log in to target. Status: Authorization failure(514)
/bin/bash: line 1:  5577 Broken pipe             lzop -d -c /mnt/pve/transit/dump/vzdump-qemu-105-2020_09_16-14_16_13.vma.lzo
      5578 Trace/breakpoint trap   | vma extract -v -r /var/tmp/vzdumptmp5575.fifo - /var/tmp/vzdumptmp5575
temporary volume 'scsi-01:vm-100-disk-2' sucessfuly removed
temporary volume 'scsi-01:vm-100-disk-3' sucessfuly removed
no lock found trying to remove 'create'  lock
TASK ERROR: command 'set -o pipefail && lzop -d -c /mnt/pve/transit/dump/vzdump-qemu-105-2020_09_16-14_16_13.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp5575.fifo - /var/tmp/vzdumptmp5575' failed: exit code 133

2) I can't start any VM on iscsi-over-zfs disks, it's just saying "TASK ERROR: start failed: QEMU exited with code -1" (no more info), however qm start 105 works just fine (via ssh, thank god).
3) I can't backup disks on iscs-over-zfs... (it's just freezes, waited for hour for 32gb disk).

Any idea what's going on and what to do ? Thank you.
 
Also, unable to move-edit zvols from Proxmox interface all failed with:
Code:
error during cfs-locked 'storage-kvm-fs-hdd' operation: command '/usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/172.16.32.2_id_rsa root@172.16.32.2 -- /usr/bin/targetcli /iscsi/iqn.2003-01.org.linux-iscsi.kvm-fs.x8664:sn.2a9f108b8cc6/tpg1/luns/ delete lun0' failed: Insecure dependency in exec while running with -T switch at /usr/share/perl/5.28/IPC/Open3.pm line 178. ...propagated at /usr/share/perl5/PVE/Storage/LunCmd/LIO.pm line 337.
 
@j4sdk
Code:
iSCSI: Failed to connect to LUN : Failed to log in to target. Status: Authorization failure(514)
might be the clue?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!