[SOLVED] Need help configuring ZFS over iSCSI

[jsalas424]

PVE 6.4-6 on both nodes

I'm trying to set up ZFS over iSCSI to share disks in my cluster.

I have followed the instructions from the official docs. I confirmed that I can log in to each node with ssh -i /etc/pve/priv/zfs/192.168.1.1_id_rsa root@192.168.1.1 replacing the IP with each node's address.

I have tried to use the solaris

zfs: solaris
    blocksize 4k
    target iqn.2010-08.org.illumos:02:b00c9870-6a97-6f0b-847e-bbfb69d2e581:tank1
    pool tank
    iscsiprovider comstar
    portal 192.168.1.129
    content images

and linux configs as such:

zfs: linux
    blocksize 4k
    target iqn.2001-04.com.example:tank1
    pool tank
    iscsiprovider iet
    portal 192.168.1.129
    content images

but neither storage's seem to work. What am I doing wrong here?

IP of PVE server that hosts the disks: 192.168.1.129
IP of node: 192.168.1.24

 

[jsalas424]

Here is what the error looks like on my node when attempting to connect to the ZFS host


This is how I configured the storage where 192.168.1.129 is the IP for the ZFS disk host

zfs: solaris
        blocksize 4k
        target iqn.2010-08.org.illumos:02:b00c9870-6a97-6f0b-847e-bbfb69d2e581:tank1
        pool new_ssd
        iscsiprovider comstar
        portal 192.168.1.129
        content images

 

[dcsapak]

what happens when you copy the command from the error and execute it manually?

 

[jsalas424]

Here is what the error looks like on my node when attempting to connect to the ZFS host
View attachment 26595

This is how I configured the storage where 192.168.1.129 is the IP for the ZFS disk host

zfs: solaris
        blocksize 4k
        target iqn.2010-08.org.illumos:02:b00c9870-6a97-6f0b-847e-bbfb69d2e581:tank1
        pool new_ssd
        iscsiprovider comstar
        portal 192.168.1.129
        content images

That was a great suggestion, now I can see that it's actually the publickey being denied.

I'm assuming because the key is password protected

root@NodeA:~# /usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/192.168.1.129_id_rsa root@192.168.1.129 zfs list -o name,volsize,origin,type,refquota -t volume,filesystem -Hrp
root@192.168.1.129: Permission denied (publickey).

So I tried ssh again to check ad it works, but its password protected and that's not delcared in the iSCSI command. Should I just make a new public key and not password protect it, or how do I proceed with a password protected public key?

root@NodeA:~# ssh -i /etc/pve/priv/zfs/192.168.1.129_id_rsa 192.168.1.129
Enter passphrase for key '/etc/pve/priv/zfs/192.168.1.129_id_rsa':

Linux Server 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100) x86_64


The programs included with the Debian GNU/Linux system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent

permitted by applicable law.

Last login: Mon Jun  7 22:10:23 2021

root@Server:~#

 

[jsalas424]

I made the key again and just didnt password protect it this time. It worked. Maybe this warrants an update to the proxmox ZFS over iSCSI docs?

Is it still inadvisable to use the linux setup as per docs?

"A word of caution. For enterprise usecases I would only recommend solaris based platforms with Comstar. Linux based platforms can IMHO be used in a non-enterprise setup which requires working HA."

 

[dcsapak]

I made the key again and just didnt password protect it this time. It worked. Maybe this warrants an update to the proxmox ZFS over iSCSI docs?

yes thanks i added a note.

Is it still inadvisable to use the linux setup as per docs?

honestly i do not know. the paragraph comes from the original (external) developer and is quite old at this point (from the original wiki entry from 2013)
i'd say i'd try and test it, but i cannot really give a concrete answer to this