TrueNAS Storage Plugin

[warlocksyno]

Hey all, I'm sure a few of you have seen the plugin we are developing over Reddit, but I wanted to make a formal post on the forum here to hopefully get more testers and feedback.

The plugin, source code, and documentation is all available here on the Github:
https://github.com/WarlockSyno/TrueNAS-Proxmox-VE-Storage-Plugin/tree/main

Here is a quick rundown:

​

• iSCSI Block Storage - Direct integration with TrueNAS SCALE via iSCSI targets
• ZFS Snapshots - Instant, space-efficient snapshots via TrueNAS ZFS
• Live Snapshots - Full VM state snapshots including RAM (vmstate)
• Cluster Compatible - Full support for Proxmox VE clusters with shared storage
• Automatic Volume Management - Dynamic zvol creation and iSCSI extent mapping
• Configuration Validation - Pre-flight checks and validation prevent misconfigurations
• Dual API Support - WebSocket (JSON-RPC) and REST API transports
• Rate Limiting Protection - Automatic retry with exponential backoff for TrueNAS API limits
• Storage Efficiency - Thin provisioning and ZFS compression support
• Multi-path Support - Native support for iSCSI multipathing
• CHAP Authentication - Optional CHAP security for iSCSI connections
• Volume Resize - Grow-only resize with preflight space checks
• Error Recovery - Comprehensive error handling with actionable error messages
• Performance Optimization - Configurable block sizes and sparse volumes

In the future we plan on adding NVMeOF support, automatic snapshot management (similar to Pure Storage "SafeMode" snapshots), API key rotation, and a few other things that you can view here.

The aim of the project is to have TrueNAS be a seamless block storage available to PVE, without all the headaches of doing it manually. Plus to have well put together documentation and guides.

If you have any suggestions, feedback, or want to help out please leave comments here or in the discussions board on the Github repo. And for those willing to try it, please try to break it! The more testing that can be done, the more robust the plugin can be! We've tried creating tools to help us with development (as found in the tools folder) to make sure all features work as expected. If any part of the documentation is confusing or not clear enough, please let us know!



(Also, if any of you are good at making digital art, I have a fun idea for a "logo" to throw on the Github )

 

[DerekG]

Hey all, I'm sure a few of you have seen the plugin we are developing over Reddit, but I wanted to make a formal post on the forum here to hopefully get more testers and feedback.

The plugin, source code, and documentation is all available here on the Github:
https://github.com/WarlockSyno/TrueNAS-Proxmox-VE-Storage-Plugin/tree/main

Here is a quick rundown:

​

• iSCSI Block Storage - Direct integration with TrueNAS SCALE via iSCSI targets
• ZFS Snapshots - Instant, space-efficient snapshots via TrueNAS ZFS
• Live Snapshots - Full VM state snapshots including RAM (vmstate)
• Cluster Compatible - Full support for Proxmox VE clusters with shared storage
• Automatic Volume Management - Dynamic zvol creation and iSCSI extent mapping
• Configuration Validation - Pre-flight checks and validation prevent misconfigurations
• Dual API Support - WebSocket (JSON-RPC) and REST API transports
• Rate Limiting Protection - Automatic retry with exponential backoff for TrueNAS API limits
• Storage Efficiency - Thin provisioning and ZFS compression support
• Multi-path Support - Native support for iSCSI multipathing
• CHAP Authentication - Optional CHAP security for iSCSI connections
• Volume Resize - Grow-only resize with preflight space checks
• Error Recovery - Comprehensive error handling with actionable error messages
• Performance Optimization - Configurable block sizes and sparse volumes

In the future we plan on adding NVMeOF support, automatic snapshot management (similar to Pure Storage "SafeMode" snapshots), API key rotation, and a few other things that you can view here.

The aim of the project is to have TrueNAS be a seamless block storage available to PVE, without all the headaches of doing it manually. Plus to have well put together documentation and guides.

If you have any suggestions, feedback, or want to help out please leave comments here or in the discussions board on the Github repo. And for those willing to try it, please try to break it! The more testing that can be done, the more robust the plugin can be! We've tried creating tools to help us with development (as found in the tools folder) to make sure all features work as expected. If any part of the documentation is confusing or not clear enough, please let us know!



(Also, if any of you are good at making digital art, I have a fun idea for a "logo" to throw on the Github )

That looks like an extremely interesting project, I'm looking forward to testing on my homelab.

 

[warlocksyno]

As a heads up, currently MPIO is kinda meh on read performance, it's documented here:
https://github.com/WarlockSyno/True...ions.md#multipath-read-performance-limitation

I'll reach out to TrueNAS/iXSystems to see if there's anything that can be done on tuning the iSCSI knobs on the TrueNAS side. NVMe-TCP shouldn't have this issue, but it's not implemented at the moment, it's a planned feature though. We will need to source some hardware to test it properly though.

 

[DerekG]

As a heads up, currently MPIO is kinda meh on read performance, it's documented here:
https://github.com/WarlockSyno/True...ions.md#multipath-read-performance-limitation

I'll reach out to TrueNAS/iXSystems to see if there's anything that can be done on tuning the iSCSI knobs on the TrueNAS side. NVMe-TCP shouldn't have this issue, but it's not implemented at the moment, it's a planned feature though. We will need to source some hardware to test it properly though.

Thanks for the heads-up

I wouldn't even attempt such an install on 1Gbe networking, I just assumed that 10Gbe would by a part of the minimum specs, for this use case.

 

[warlocksyno]

I wouldn't even attempt such an install on 1Gbe networking, I just assumed that 10Gbe would by a part of the minimum specs, for this use case.

Oh for sure. It's mainly just to let people know that right now the read speeds don't scale up with interface speed at the moment, but it should scale out with more interfaces. Until the iSCSI settings can be configured on the TrueNAS side some more, it'll be limited in comparison to what the hardware can actually do.

 

[DerekG]

Oh for sure. It's mainly just to let people know that right now the read speeds don't scale up with interface speed at the moment, but it should scale out with more interfaces. Until the iSCSI settings can be configured on the TrueNAS side some more, it'll be limited in comparison to what the hardware can actually do.

Well, you'll be pleased to know that the TrueNAS team are currently revamping the API system so that developers will be able to gain far more control over the internals.

 

[bbgeek17]

Hi @warlocksyno

This is excellent progress on getting Proxmox support working and its really nice to see the effort and attention to detail going into it.

I just wanted to help clarify one concept you mentioned regarding MaxOutstandingR2T. Any MPIO limitations you might be encountering aren't related to that parameter. In fact, MaxOutstandingR2T=1 has no impact on the behavior of reads at all.

In iSCSI, R2T (Request To Transfer) is a protocol mechanism the target uses to request data from the initiator for SCSI commands that carry "OUT" data, meaning writes or the occasional bidirectional command. It's essentially a flow-control feature to prevent the initiator from sending too much write data at once.

So in short: whatever's causing the performance issue with reads, it's not related to MaxOutstandingR2T.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[warlocksyno]

Aye, the famous @bbgeek17 !

Good to know! I need to read up more on the iSCSI specific tunings. I know the TrueNAS SCALE side has a bunch of safe defaults. Much of what I learned with iSCSI was our migration from ESXi to Proxmox at work, but we needed to use our Pure Storage arrays. After much trial and error, using the Pure Storage plugin (which inspired me to begin work on this idea!) I was able to absolutely max out 2x25GbE connections on each node to the Pure Storage. There was a lot of tuning to be done. I kind of mention a little bit of it over on Reddit.

Do you have an recommendations on what could be looked at to improve the read performance? Given the limitations on the TrueNAS side that is.


Also, big fan of Blockbridge. Honestly, had I known about Blockbridge when we were hunting for storage at work, I would have been picking that over the Pure arrays.