[TUTORIAL] Guide: Setup ZFS-over-iSCSI with PVE 5x and FreeNAS 11+

[freebattie]

I have not done much testing yet,
but i have 1 truenas with 4ssd setup as 2vdev where disk are set to mirror
and also on the proxmox i have the same setup for local storage
porxmox and truenas are connected tru 10gbe for the iscsi
OS on each server is on two seperate ssd setup also as zfs
they both have same hardware, R620 2x 2650 and 125gb ram
disks are not enterprice so i think this will be the main bottleneck when testing.
plan is to run same vm on local and over iscsi and see how it preforms, also testing snapshoting etc.
creating new vms work like a charm, all done from proxmox like normal
also running with zfs set sync=disabled

 

[velocity08]

I have not done much testing yet,
but i have 1 truenas with 4ssd setup as 2vdev where disk are set to mirror
and also on the proxmox i have the same setup for local storage
porxmox and truenas are connected tru 10gbe for the iscsi
OS on each server is on two seperate ssd setup also as zfs
they both have same hardware, R620 2x 2650 and 125gb ram
disks are not enterprice so i think this will be the main bottleneck when testing.
plan is to run same vm on local and over iscsi and see how it preforms, also testing snapshoting etc.
creating new vms work like a charm, all done from proxmox like normal
also running with zfs set sync=disabled

i think for testing (non production) you can use normal SSD but just remember they will ware out fast so don't keep anything on there you cant afford to loose.

the performance IOP's will be similar but more latency over the iSCSI connection vs local disk its just the way it is as its another layer for data to travel over.

would love to keep a track of the testing please update this thread if you can

best of luck.

""Cheers
G

 

[Grunt]

I'm getting the following error when trying to create a VM on Proxmox 7.4-3:

TASK ERROR: unable to create VM 120 - Unable to connect to the FreeNAS API service at '10.1.0.252' using the 'http' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 380.

I followed the installation instructions on TheGrandWazoo's Github page, and then did the below on each of my nodes to generate the SSH keys

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/10.1.0.252_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/10.1.0.252_id_rsa.pub root@10.1.0.252

On each node, I confirmed that I can login successfully using

ssh root@10.1.0.252

At that point I figured I was done and created my ZFS over iSCSI storage:



It seems to be happy and is showing active. I tried to create a VM but received the above mentioned error. So I went back through and ran the three patch commands, the first two commands said the patch has already been applied. Copied FreeNAS.pm and restarted the requested services, still get the same error.

When I run iscsiadm -m session on a proxmox node, It appears I'm connected to TrueNAS via iSCSI.



I would really appreciate any help you guys can provide.

 

[Grunt]

My issue was ridiculous...after lots of testing and fiddling around, I figured out that Proxmox was connecting to TrueNAS over SSH and running the appropriate commands successfully. Afterwards, it connects to the API over 80/443 to perform additional functions. The API listens on the same interfaces that the WebGUI is configured to listen on. On TrueNAS, going to System>General and adding my storage network to the Web Interface IPv4 address, everything started working properly.

 

[MisterDeeds]

Hello all

I have created a small tutorial on how I successfully set up ZFS over iSCSI for Proxmox 7.4 and TrueNAS Scale. Maybe it will help someone:

Important, in my example it is a dedicated SAN network where neither clients nor other devices have access. Therefore, IP restrictions, authentication or initiator restrictions were not used.

First install the TrueNAS plugin on the proxmox: (https://github.com/TheGrandWazoo/freenas-proxmox):

curl https://ksatechnologies.jfrog.io/artifactory/ksa-repo-gpg/ksatechnologies-release.gpg -o /etc/apt/trusted.gpg.d/ksatechnologies-release.gpg
curl https://ksatechnologies.jfrog.io/artifactory/ksa-repo-gpg/ksatechnologies-repo.list -o /etc/apt/sources.list.d/ksatechnologies-repo.list
apt update
apt install freenas-proxmox -y

Quote from the Github page
"!!! ATTENTION 2023-07-13: JFrog has canceled my repo for all my projects. I am currently looking at other solutions. Please be patient while I remedy the situation!!!"

Now create a SSH key for the Proxmox and exchange it with the TrueNAS: https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/10.0.0.40_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/10.0.0.40_id_rsa.pub root@10.0.0.40
ssh -i /etc/pve/priv/zfs/10.0.0.40_id_rsa root@10.0.0.40

Now enable at TrueNAS under «System Setting > Services > SSH & iSCSI» Service:


Activate "Log in as Root with Password" in the SSH configuration:


Under «Credentials» define the password for the root user and deselect "Disable Password":


Create new dataset with the name "iSCSI" (name can be chosen freely):


Now activate and configure the iSCSI service under "Shares":


Continue to the iSCSI configuration:


Under "Portals", select the IP address on which the web interface is running. You can also select 0.0.0.0 for all IP addresses.


Under "Initiators Groups" select "Allow All Initiators":


Now create a target:


The target name can also be freely selected. Only lowercase letters may be used:


1/2

 

[MisterDeeds]

Now add the storage under Proxmox:


If the "FreeNAS-API" plugin does not appear, you need to clear your browser's cache. At Target you enter your Base Name, then colon and then your Target Name (in my case "vm").


If everything has worked, the iSCSI target can now be used:


Under TrueNAS it is now nice to see that the API has created a new ZVOL. This can now also be individually snapshotted, etc.


Best regards

2/2

 

[Moditproxer]

I would love to get some help. I have tried all of the options that have been suggested but still cannot get past this error. Any ideas? It creates the new vdev for the VM harddrive on the TrueNAS box but fails during the process.

failed to update VM 101: Unable to connect to the FreeNAS API service at '192.168.210.31' using the 'http' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 241. (500)

I have https redirect turned off and I can get to the web GUI for Truenas through http. The ip address for the gui is 0.0.0.0.

 

[MisterDeeds]

Since the installation via repository is currently not possible, here is the package file for the manual installation. As soon as the developer TheGrandWazoo has found a solution, please switch back to the online repository - otherwise there will be no updates!

Download the ZIP attached and move it to your "/tmp" folder of the Proxmox Host

Deletes the unrecoverable repositories

rm /etc/apt/trusted.gpg.d/ksatechnologies-release.gpg
rm /etc/apt/sources.list.d/ksatechnologies-repo.list

apt update -y && apt dist-upgrade -y && apt autoremove -y && apt autoclean -y

Install the package

cd /tmp
unzip freenas-proxmox_2.2.0-0-beta10_all.deb.zip
chmod +x freenas-proxmox_2.2.0-0-beta10_all.deb
apt install librest-client-perl && dpkg -i freenas-proxmox_2.2.0-0-beta10_all.deb

If everything worked, it should look like this

root@PVE01:~# dpkg -l |grep freenas
ii  freenas-proxmox                      2.2.0-0-beta10                       all          TrueNAS Plugin for Proxmox VE

Big thanks to TheGrandWazoo for the plugin!

 

[MisterDeeds]

I would love to get some help. I have tried all of the options that have been suggested but still cannot get past this error. Any ideas? It creates the new vdev for the VM harddrive on the TrueNAS box but fails during the process.

failed to update VM 101: Unable to connect to the FreeNAS API service at '192.168.210.31' using the 'http' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 241. (500)

I have https redirect turned off and I can get to the web GUI for Truenas through http. The ip address for the gui is 0.0.0.0.

Sometimes it resets this setting but still shows it, then password login is possible. Try removing the option and then reactivating it, including setting a new password. Then it worked for me again.

 

[AaShiyana]

Hello all

I have created a small tutorial on how I successfully set up ZFS over iSCSI for Proxmox 7.4 and TrueNAS Scale. Maybe it will help someone:

Important, in my example it is a dedicated SAN network where neither clients nor other devices have access. Therefore, IP restrictions, authentication or initiator restrictions were not used.

First install the TrueNAS plugin on the proxmox: (https://github.com/TheGrandWazoo/freenas-proxmox):

curl https://ksatechnologies.jfrog.io/artifactory/ksa-repo-gpg/ksatechnologies-release.gpg -o /etc/apt/trusted.gpg.d/ksatechnologies-release.gpg
curl https://ksatechnologies.jfrog.io/artifactory/ksa-repo-gpg/ksatechnologies-repo.list -o /etc/apt/sources.list.d/ksatechnologies-repo.list
apt update
apt install freenas-proxmox -y

Quote from the Github page
"!!! ATTENTION 2023-07-13: JFrog has canceled my repo for all my projects. I am currently looking at other solutions. Please be patient while I remedy the situation!!!"

Now create a SSH key for the Proxmox and exchange it with the TrueNAS: https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/10.0.0.40_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/10.0.0.40_id_rsa.pub root@10.0.0.40
ssh -i /etc/pve/priv/zfs/10.0.0.40_id_rsa root@10.0.0.40

Now enable at TrueNAS under «System Setting > Services > SSH & iSCSI» Service:
View attachment 50580

Activate "Log in as Root with Password" in the SSH configuration:
View attachment 50581

Under «Credentials» define the password for the root user and deselect "Disable Password":
View attachment 50582

Create new dataset with the name "iSCSI" (name can be chosen freely):
View attachment 50583

Now activate and configure the iSCSI service under "Shares":
View attachment 50584

Continue to the iSCSI configuration:
View attachment 50585

Under "Portals", select the IP address on which the web interface is running. You can also select 0.0.0.0 for all IP addresses.
View attachment 50586

Under "Initiators Groups" select "Allow All Initiators":
View attachment 50587

Now create a target:
View attachment 50588

The target name can also be freely selected. Only lowercase letters may be used:
View attachment 50589

1/2

A huge big Thank you to MisterDeeds for sharing the package and the steps to manually install the plugin whilst the issue exists on JFrog and the github page. I had this working previously but with the issue you too mentioned, it was no longer possible to get it working again. With your package file I have managed to install, configure and working again - this time on Proxmox 8.03 and TrueNAS-13.0-U5.2 - THANK YOU!​

​

 

[MisterDeeds]

A huge big Thank you to MisterDeeds for sharing the package and the steps to manually install the plugin whilst the issue exists on JFrog and the github page. I had this working previously but with the issue you too mentioned, it was no longer possible to get it working again. With your package file I have managed to install, configure and working again - this time on Proxmox 8.03 and TrueNAS-13.0-U5.2 - THANK YOU!​

​

you're very welcome, glad that you have made it work! BR

 

[Desktop4354]

A huge big Thank you to MisterDeeds for sharing the package and the steps to manually install the plugin whilst the issue exists on JFrog and the github page. I had this working previously but with the issue you too mentioned, it was no longer possible to get it working again. With your package file I have managed to install, configure and working again - this time on Proxmox 8.03 and TrueNAS-13.0-U5.2 - THANK YOU!​

​

Did you have to do anything special to get it to work on Proxmox 8? which version of the plugin did you use? I just got it working on 7.4, been thinking about upgrading to 8 for the newer kernel.

 

[linkstat]

Hello.
If I currently have the plugin running (both ZFS over iSCSI and NFS), and I upgrade the storage from TrueNAS CORE (whose base system is FreeBSD) to TrueNAS SCALE (which now comes with GNU/Linux), will everything still work fine?
TrueNAS offers this change of operating system as an "upgrade", but since the base system changes, I do not dare to carry it out, and I would like to know if anyone here has already tried going from CORE to SCALE.

Regards!

 

[ccundari]

Hello,
I followed step by step the installation instructions but I'm getting the following error when trying to execute qm rescan command on Proxmox 8.1 and Truesan Core
Unable to connect to the FreeNAS API service at '192.168.20.7' using the 'https' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 380.
Plugin version
pkg -l |grep freenas
ii freenas-proxmox 2.2.0-1 all FreeNAS Plugin for Proxmox VE
line 380 "die "Unable to connect to the FreeNAS API service at '" . $apihost . "' using the '" . $scheme . "' protocol";"
Any ideas?

 

[bbgeek17]

Unable to connect to the FreeNAS API service at '192.168.20.7' using the 'https' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 380.

The first thing to check is whether that IP is reachable on that port. You can try using curl.
Without examining the code, the error could be a catch-all, and might be due to authentication problem or other condition. If manual testing works, I'd reach out to plugin developer.

good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ccundari]

this is the output:
root@pve05:~# curl -k -I https://192.168.20.7/api/
HTTP/2 302
server: nginx
date: Thu, 21 Mar 2024 08:57:51 GMT
content-type: text/html
content-length: 138
location: https://192.168.20.7/ui/
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
permissions-policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
referrer-policy: strict-origin
x-frame-options: SAMEORIGIN

root@pve05:~#
From shell everything seems to work perfectly...

 

[bbgeek17]

I guess, start reading plugin code above line 380 to see what leads to the "die" method. Or open an issue on github with the developers of the plugin.

You can play around with adding "print" methods to see what commands/options are being called. I would probably start with reviewing the configuration steps carefully to ensure that nothing was missed.

Its also possible that something has changed on FreeNAS side and plugin is no longer compatible.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[PBoerst]

Hello everyone, hate to have to ask for help but I've been working on this for about 20 hours now, when creating the vm I receive
If anyone can provide any insight or hints I would greatly appreciate it.
Also, if there's more info that I can provide please let me know.
TASK ERROR: unable to create VM 102 - Undefined `freenas_user` and/or `freenas_password` variables. at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 136.

 

[bbgeek17]

The first question is - did you define those variables in /etc/pve/storage.cfg ?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[PBoerst]

The first question is - did you define those variables in /etc/pve/storage.cfg ?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

UPDATE TO BELOW: yes, I changed the 'truenas_secret' variable to be 'freenas_password' and then received a new error, posted below.

Yes, they are defined there. I believe the entry is made when I add the ZFS over ISCSI and it adds the entries that I type in the GUI.
Oddly though, the error says Undefined 'freenas_user" and/or "freenas_password"
but, in the storage.cfg, the variables are labeled as 'freenas_user' and 'truenas_secret'
should that 'truenas_secret' be labeled as 'freenas_password'?


---New issue---
TASK ERROR: unable to create VM 102 - Unable to connect to the FreeNAS API service at '10.20.0.1' using the 'http' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 389.