[TUTORIAL] Guide: Setup ZFS-over-iSCSI with PVE 5x and FreeNAS 11+

PBoerst said:
Oddly though, the error says Undefined 'freenas_user" and/or "freenas_password"
but, in the storage.cfg, the variables are labeled as 'freenas_user' and 'truenas_secret'
should that 'truenas_secret' be labeled as 'freenas_password'?
Click to expand...
Thats a good guess and easy to try. You can examine the plugin source to see what values are actually used.
Did you install the plugin from official source or one of the clones? If its the former, you should file a bug with developer.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Thats a good guess and easy to try. You can examine the plugin source to see what values are actually used.
Did you install the plugin from official source or one of the clones? If its the former, you should file a bug with developer.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Official source, but now I receive the error I updated above.
If I view line 389 I see this.
freenas_api_log_error($freenas_server_list->{$apihost});
die "Unable to connect to the FreeNAS API service at '" . $apihost . vvv.
I'm not sure where to find the freenas_server_list to manually modify it though.
I believe that because I am connecting to the target with an initiator with a /30 network, I need to somehow tell this too look for that targets ip?
 
Last edited:
Hi, I just wanted to share the error I encountered and the solution that worked for me.

Code: 
update VM 183: -scsi1 iscsi-pve2-truenas:32,iothread=on
Use of uninitialized value $extent_id in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 651.
TASK ERROR: Unable to create lun /dev/zvol/*****************/recursos-iscsi/pve-dataset/disk-images-pve2/vm-183-disk-0 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 270

This error occurred because the name of the pool exceeded 64 characters.

Solution:I shortened the pool name to meet the 64-character limit, and the issue was resolved.

Best regards!
 
It worked on mine PVE 8.2.7.

"You dont need to create any extents as the FreeNAS-API plugin will do this automatically when the drive is created on the VM."

But I made my extent myself.

Thanks!

P.S I'm playing Proxmox in my lab (under free EXSi) and I really like PVE & PBS!
 
Last edited:
I got this error when I try to use the ISCSI...

1732163227886.png

My storage config:

1732163270190.png

1732163301369.png

On truenas (tail -f /var/log/auth.log), appears only:

1732163393873.png

I don't know more what to do :/

Somebody could help me, please? Thanks! :)
 
Try to run that command manually (up to "zfs list" the rest you can drop). You may get a more verbose error. If it works, the SSH is working, keep adding portions of the command until you hit an error. Troubleshoot from there.
If the ssh fails, then examine your password, ssh key etc. If you cant ssh - nothing will work.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: eduardoroedel
bbgeek17 said:
Try to run that command manually (up to "zfs list" the rest you can drop). You may get a more verbose error. If it works, the SSH is working, keep adding portions of the command until you hit an error. Troubleshoot from there.
If the ssh fails, then examine your password, ssh key etc. If you cant ssh - nothing will work.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
@bbgeek17, thank you for supporting. I solved the mistake and ssh was the problem... The official doc followed didn't solved and in this case, the winner was chatgpt hehehe.

In just one node from proxmox cluster, run:

--Generate ssh
ssh-keygen -t rsa -b 4096 -f /etc/pve/priv/zfs/<ip truenas>_id_rsa -N ''

--Copy to TrueNAS
ssh-copy-id -i /etc/pve/priv/zfs/<ip truenas>_id_rsa.pub root@<ip truenas>

--Test
ssh -i /etc/pve/priv/zfs/<ip truenas>_id_rsa root@<ip truenas>
 
eduardoroedel said:
@bbgeek17, thank you for supporting. I solved the mistake and ssh was the problem... The official doc followed didn't solved and in this case, the winner was chatgpt hehehe.

In just one node from proxmox cluster, run:

--Generate ssh
ssh-keygen -t rsa -b 4096 -f /etc/pve/priv/zfs/<ip truenas>_id_rsa -N ''

--Copy to TrueNAS
ssh-copy-id -i /etc/pve/priv/zfs/<ip truenas>_id_rsa.pub root@<ip truenas>

--Test
ssh -i /etc/pve/priv/zfs/<ip truenas>_id_rsa root@<ip truenas>
Click to expand...
the test part is important to do in all nodes to prevent fingertip issue on ssh connection :)
 
Warning to all who got here. TrueNAS 25.04 has deprecated the REST API that this plugin relies upon. ZFS over iSCSI using this plugin (TheGrandWazoo's repo version) WILL NOT FUNCTION on TrueNAS 25.04. I have rolled back to 24.10 for the time being until the plugin is adapted to whatever TrueNAS is favoring or I migrate to a different storage plugin.

See this GitHub issue for details and tracking. With no repo activity since 2023 and no activity whatsoever from TheGrandWazoo on GitHub since early 2024, I'm afraid it's going to take a fork to continue this work.


https://github.com/TheGrandWazoo/freenas-proxmox/issues/205
 
  • Like
Reactions: mir and linkstat
Plug4038 said:
Warning to all who got here. TrueNAS 25.04 has deprecated the REST API that this plugin relies upon. ZFS over iSCSI using this plugin (TheGrandWazoo's repo version) WILL NOT FUNCTION on TrueNAS 25.04. I have rolled back to 24.10 for the time being until the plugin is adapted to whatever TrueNAS is favoring or I migrate to a different storage plugin.

See this GitHub issue for details and tracking. With no repo activity since 2023 and no activity whatsoever from TheGrandWazoo on GitHub since early 2024, I'm afraid it's going to take a fork to continue this work.


https://github.com/TheGrandWazoo/freenas-proxmox/issues/205
Click to expand...
I am now very happy I never upgrade before the first point release is available ;)
I hope for native support in Truenas Scale so we can skip the plugin entirely and avoid the patching of Proxmox.
 
Plug4038 said:
Warning to all who got here. TrueNAS 25.04 has deprecated the REST API that this plugin relies upon. ZFS over iSCSI using this plugin (TheGrandWazoo's repo version) WILL NOT FUNCTION on TrueNAS 25.04. I have rolled back to 24.10 for the time being until the plugin is adapted to whatever TrueNAS is favoring or I migrate to a different storage plugin.

See this GitHub issue for details and tracking. With no repo activity since 2023 and no activity whatsoever from TheGrandWazoo on GitHub since early 2024, I'm afraid it's going to take a fork to continue this work.


https://github.com/TheGrandWazoo/freenas-proxmox/issues/205
Click to expand...
@TheGrandWazoo has woken up and is aware of the issue. We have a working patch that based on my limited testing has restored the REST API functionality in 25.04.0.

I've started an effort to convert the code to use Websocket API. There appears to be a bug in the iscsi query methods in TrueNAS 25.04.0 preventing me from testing all of the converted code.

It would be nice to see this functionality baked into proxmox nativity.

I never thought I'd ever learn to write perl... But here we are.
 
Has there been any work on this? I finally upgraded from pve 7 to 8 and iscsi no longer functions. I am using an API key. Tired regenerating a new one but no change.
Truenas scale still on 24, everything was working during upgrading my nodes, until I got the final node on pve8 then I only get this error on all nodes now.
Code: 
Could not find lu_name for zvol vm-103-disk-0 at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 118. (500)
Trying to make a new storage block gives this error, however it did make a new zvol on truenas so I suppose the API stuff is working just fine. Something with perl stuff on pve 8 I suppose... Should I just patch the perl files again?

Code: 
update VM 103: -virtio3 nasx:32,iothread=on
Warning: volblocksize (4096) is less than the default minimum block size (16384).
To reduce wasted space a volblocksize of 16384 is recommended.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 597.
Use of uninitialized value $target_id in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 787.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 618.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 597.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 597.
Use of uninitialized value $target_id in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 787.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 597.
Use of uninitialized value $target_id in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 787.
TASK ERROR: Unable to find the target id for iqn.2005-10.org.nasx.nya:pve at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 259.
 
Last edited:
  • Like
Reactions: linkstat
Hello. First and foremost, thank you very much for your dedicated work.


I'm writing to report that I installed the plugin today (following the instructions from the GitHub repository). The VMs that were previously configured with TrueNAS continue to work fine. However, today I created a new VM, added the disks, and it won't start:
Code: 
Header
Proxmox
Virtual Environment 9.0.10
Search
Virtual Machine 1033 (fms-wwvm) on node 'fms-pve2'
webserver
Server View
Logs

()
TrueNAS [INFO] : Disconnected
TrueNAS [WARN] : Connection timed out. Reconnecting...
TrueNAS [INFO] : Connected
TrueNAS [INFO] : Authenticated
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'path' found : /dev/zvol/trinity/pve/vm-1033-disk-0
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'lunid' found : 11
kvm: -blockdev {"detect-zeroes":"unmap","discard":"unmap","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"iscsi://10.47.27.1/iqn.2022-07.lan.fms:srv-nas1:storage/11","node-name":"e19c125fa03b991854467d1a88a0102","read-only":false},"node-name":"f19c125fa03b991854467d1a88a0102","read-only":false},"node-name":"drive-virtio0","read-only":false,"throttle-group":"throttle-drive-virtio0"}: Could not open 'iscsi://10.47.27.1/iqn.2022-07.lan.fms:srv-nas1:storage/11': No such file or directory
TASK ERROR: start failed: QEMU exited with code 1


So I tried deleting the disk in question (which completed successfully):
Code: 
Header
Proxmox
Virtual Environment 9.0.10
Search
Virtual Machine 1033 (fms-wwvm) on node 'fms-pve2'
webserver
Server View
Logs

()
update VM 1033: -delete unused0
TrueNAS [INFO] : Disconnected
TrueNAS [WARN] : Connection timed out. Reconnecting...
TrueNAS [INFO] : Connected
TrueNAS [INFO] : Authenticated
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'path' found : /dev/zvol/trinity/pve/vm-1033-disk-0
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'lunid' found : 11
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'path' found : /dev/zvol/trinity/pve/vm-1033-disk-0
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'lunid' found : 11
TrueNAS [INFO] : zvol/trinity/pve/vm-1033-disk-0 with key 'path' found : /dev/zvol/trinity/pve/vm-1033-disk-0
TrueNAS [INFO] : Deleted LUN: zvol/trinity/pve/vm-1033-disk-0
TASK OK


I then added a new disk with default options (except for Bus/Device, which I set to VirtIO Block), this time from another storage pool also configured as ZFS over iSCSI (also successful):
Code: 
Header
Proxmox
Virtual Environment 9.0.10
Search
Virtual Machine 1033 (fms-wwvm) on node 'fms-pve2'
webserver
Server View
Logs

()
update VM 1033: -virtio0 tank-zfs:32,discard=on,iothread=on
TrueNAS [INFO] : /dev/zvol/tank/pve/vm-1033-disk-0
TrueNAS [INFO] : Disconnected
TrueNAS [WARN] : Connection timed out. Reconnecting...
TrueNAS [INFO] : Connected
TrueNAS [INFO] : Authenticated
TrueNAS [INFO] : Created LUN: /dev/zvol/tank/pve/vm-1033-disk-0 : T1:E146:L11
virtio0: successfully created disk 'tank-zfs:vm-1033-disk-0,discard=on,iothread=1,size=32G'
TASK OK


I attempted to start the VM, but once again got an error:
Code: 
Header
Proxmox
Virtual Environment 9.0.10
Search
Virtual Machine 1033 (fms-wwvm) on node 'fms-pve2'
webserver
Server View
Logs

()
TrueNAS [INFO] : Connected
TrueNAS [INFO] : Authenticated
TrueNAS [INFO] : Version: TrueNAS-25.04.2.1
TrueNAS [INFO] : zvol/tank/pve/vm-1033-disk-0 with key 'path' found : /dev/zvol/tank/pve/vm-1033-disk-0
TrueNAS [INFO] : zvol/tank/pve/vm-1033-disk-0 with key 'lunid' found : 11
kvm: -blockdev {"detect-zeroes":"unmap","discard":"unmap","driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"iscsi://10.47.27.1/iqn.2022-07.lan.fms:srv-nas1:storage/11","node-name":"ed0eb8cdcef4275c7f0ada8d9365b49","read-only":false},"node-name":"fd0eb8cdcef4275c7f0ada8d9365b49","read-only":false},"node-name":"drive-virtio0","read-only":false,"throttle-group":"throttle-drive-virtio0"}: Could not open 'iscsi://10.47.27.1/iqn.2022-07.lan.fms:srv-nas1:storage/11': No such file or directory
TASK ERROR: start failed: QEMU exited with code 1


I don't understand why this is happening, especially since I've verified on the TrueNAS storage that the extent exists, with Device/File, Serial, and NAA properly assigned/generated.


The only difference I notice is in the naming convention between the old and new extents:
For example:
1759957438559.png
Notice the difference in the extent name? Previously it used hyphens (-), now it uses forward slashes (/)


I have the API configured, SSH keys set up, etc. (plus the VMs created earlier are starting fine). Any help or suggestions on what else I should check?
 
I'm reintroducing the conversion of / to - in the extent names for consistency. The name is not used for lookup in the plugin so has no effect on the operation of the native plugin.

Regarding this error:

Code: 
kvm: -blockdev {"detect-zeroes":"unmap","discard":"unmap","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"iscsi://10.47.27.1/iqn.2022-07.lan.fms:srv-nas1:storage/11","node-name":"e19c125fa03b991854467d1a88a0102","read-only":false},"node-name":"f19c125fa03b991854467d1a88a0102","read-only":false},"node-name":"drive-virtio0","read-only":false,"throttle-group":"throttle-drive-virtio0"}: Could not open 'iscsi://10.47.27.1/iqn.2022-07.lan.fms:srv-nas1:storage/11': No such file or directory

This error is seen when coming from the old plugin. Please run

Bash: 
apt reinstall pvemanager libpve-storage-perl
systemctl restart pvedaemon.service pveproxy.service pvestatd.service

You may have to also reinstall the plugin after completing the above.

The patch version of the plugin has major flaw when being uninstalled. It restores the backup files from when the patches were applied which could be out of date.

If you want to fix any extent names that still have the / you can run the following on TrueNAS

Code: 
sqlite3 /data/freenas-v1.db "UPDATE services_iscsitargetextent SET iscsi_target_extent_name = REPLACE(iscsi_target_extent_name, '/','-')"
 
Last edited:
  • Like
Reactions: uzumo
You must log in or register to reply here.
Top Bottom