ZFS over iSCSI: ZVOLs created on iSCSI server but not being mounted on PVE box

Dec 4, 2023
45
12
13
Hello,

I'm prototyping the ZFS-over-iSCSI functionality on PVE 9.2.20 and running into trouble. Specifically, I followed these two blogposts to get it set up (I added auth on tpg1 instead of the ACLs).

PVE package list:
Code:
proxmox-ve: 9.2.0 (running kernel: 7.0.14-19-pve)
pve-manager: 9.2.20 (running version: 9.2.20/49318c671b82f31e)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-19-pve-signed: 7.0.14-19
proxmox-kernel-7.0: 7.0.14-19
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
ceph-fuse: 19.2.6-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve3
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.3
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.2
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.2
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.6-1
proxmox-backup-file-restore: 4.2.6-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.10
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.12
pve-edk2-firmware: 4.2026.08-1
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.6
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.8
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1

targetcli setup:
Code:
$ sudo targetcli

/> ls
/> /iscsi create
Created target iqn.STUFF:sn.HEXSTUFF.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/> cd /iscsi/iqn.STUFF:sn.HEXSTUFF/tpg1/
/iscsi/iqn.20...HEXSTUFF/tpg1> set attribute generate_node_acls=1
Parameter generate_node_acls is now '1'.
/iscsi/iqn.20...HEXSTUFF/tpg1> set attribute authentication=1
Parameter authentication is now '1'.
/iscsi/iqn.20...HEXSTUFF/tpg1> set auth userid=USERNAME
Parameter userid is now 'USERNAME'.
/iscsi/iqn.20...HEXSTUFF/tpg1> set auth password=PASSWORD
Parameter password is now 'PASSWORD'.
/iscsi/iqn.20...HEXSTUFF/tpg1> cd /
/> saveconfig
Configuration saved to /etc/rtslib-fb-target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/rtslib-fb-target/backup/.
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

storage.cfg:
Code:
zfs: iscsi-zfs
        blocksize 16k
        iscsiprovider LIO
        pool POOLNAME
        portal IP.ADD.RE.SS
        target iqn.STUFF:sn.HEXSTUFF
        content images
        lio_tpg tpg1
        nowritecache 1
        sparse 1
        zfs-base-path /dev

Added to /etc/iscsi/iscsid.conf:
Code:
node.session.auth.authmethod = CHAP
node.session.auth.chap_algs = SHA3-256,SHA256,SHA1,MD5
node.session.auth.username = USERNAMEGOESHERE
node.session.auth.password = PASSWORDGOESHERE

iscsiadm discovery shows one entry. Login is successful.

Proxmox is able to SSH in and I see the ZVOL being created in the pool, but VM creation fails with TASK ERROR: unable to create VM 996 - Could not open /dev/POOLNAME/vm-996-disk-0 and Proxmox doesn't seem to be mounting the zvol anywhere that I can find it.

What am I doing wrong?

Thanks in advance!
 
Last edited:
Proxmox is able to SSH in and I see the ZVOL being created in the pool, but VM creation fails with TASK ERROR: unable to create VM 996 - Could not open /dev/POOLNAME/vm-996-disk-0 and Proxmox doesn't seem to be mounting the zvol anywhere that I can find it.
I have not looked at the referenced blogs so I don't know whether they are incorrect or you may have skipped a step. The error most likely means you have wrong path/pool information, or the system you are using utilizes different organization than the plugin expects.

Use "pvesm allocate" to create a volume, then examine your storage target to find the path that is actually present, that may inform you whether any adjustments in the config are needed.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox