Using ZFS Pool as "local" storage

cmonty14

Active Member
Mar 4, 2014
338
4
38
Hello,
I have a server with ECC RAM and multiple disks, means it's equipped like a NAS.

However, I want to install Proxmox and run a VM with TrueNAS Core.
All relevant disks (e.g. WD Red) will be configured as passthrough for this VM.

Then in TrueNAS Core I will configure these drives for ZFS.

My question is:
Can I use this ZFS storage in Proxmox directly w/o setting up a share in TrueNAS Core?
Because effectively the relevant disks are HW connected to bus controlled by Proxmox OS.

But I assume disk passthrough will prevent this.
How can I then utilize the storage provided by TrueNAS Core most efficient w/o any latency caused by network stack in Proxmox?

THX
 

leesteken

Famous Member
May 31, 2020
2,388
511
118
.... I want to install Proxmox and run a VM with TrueNAS Core.
All relevant disks (e.g. WD Red) will be configured as passthrough for this VM.

Then in TrueNAS Core I will configure these drives for ZFS.
Note that some WD Red use SMR which can give problems with ZFS. Only WD Red Plus is guaranteed to be CMR.
My question is:
Can I use this ZFS storage in Proxmox directly w/o setting up a share in TrueNAS Core?
Because effectively the relevant disks are HW connected to bus controlled by Proxmox OS.
No, you cannot share disks between VMs or between VMs and the Proxmox host. Only containers can share storage using mountpoints.
But I assume disk passthrough will prevent this.
There is nothing to stop you, but just as you cannot share disk between physical machines at the same time, this will corrupt the files and directories on the disk.
How can I then utilize the storage provided by TrueNAS Core most efficient w/o any latency caused by network stack in Proxmox?
In the same way as other physical or virtual machines can use TrueNAS shares.

I know it is good practice to abstract storage and separate it from the VM virtualization hosts (like a SAN), but sharing the ZFS storage from Proxmox directly is much simpler (especially for Linux containers). EDIT: You could use TurnKey Linux File Server in a container to share the ZFS storage from the Proxmox host and separate/abstract it in that way.
 
Last edited:

cmonty14

Active Member
Mar 4, 2014
338
4
38
So, if I deploy TrueNAS Core in a VM, I must accept the disadvantage that shared storage used by Proxmox, e.g. ISO, images, etc. will always be thwarted by the network stack?

If I setup ZoL in Proxmox, can I utilize this storage in a VM running TrueNAS Core "directly"?
And what about the features provided by TrueNAS Core for ZFS performance improvement? Would these features still work?

Or would I get the best storage performance if I install TrueNAS Scale on the server and deploy VMs in that OS?
 

leesteken

Famous Member
May 31, 2020
2,388
511
118
So, if I deploy TrueNAS Core in a VM, I must accept the disadvantage that shared storage used by Proxmox, e.g. ISO, images, etc. will always be thwarted by the network stack?
Yes. Unless you use containers (that run a Linux distro) on Proxmox.
If I setup ZoL in Proxmox, can I utilize this storage in a VM running TrueNAS Core "directly"?
No. There will be the same network stack as above, just the other way around.
That Wiki feels old, maybe have a look at the current manual?
And what about the features provided by TrueNAS Core for ZFS performance improvement? Would these features still work?
You can tune ZFS settings on Proxmox as much as you want manually. What are those features you are talking about?
Or would I get the best storage performance if I install TrueNAS Scale on the server and deploy VMs in that OS?
I don't know. Performance can mean different things to different people and I don't know what your other concerns are. Bare-metal or Proxmox with LVM could be faster.
 

Dunuin

Famous Member
Jun 30, 2020
8,227
2,068
151
Germany
No. There will be the same network stack as above, just the other way around.
That Wiki feels old, maybe have a look at the current manual?
There is virtfs which will allow to passthrough a folder into a VM, similar to bind-mounts into a LXC: https://www.linux-kvm.org/page/VirtFS
But Proxmox doesn't support that yet, so you would need to set it up yourself using the CLI editing the KVM parameters in the VMs config file.
 

sublightnova

New Member
May 30, 2022
21
2
3
Would getting a dedicated controller card and using full passthrough to truenas (and ONLY truenas) be stable enough to not have extreme worries about a ZFS Pool? I mean.. still have to build it up.. Currently only 1 drive.. Going to a mirror soon and probably a few more drives after that.

Currently I'm just using the soft passthrough to truenas which causes issues I believe.

Sorry if I should have opened up a seperate thread for this.. Still new here.
 

leesteken

Famous Member
May 31, 2020
2,388
511
118
Would getting a dedicated controller card and using full passthrough to truenas (and ONLY truenas) be stable enough to not have extreme worries about a ZFS Pool? I mean.. still have to build it up.. Currently only 1 drive.. Going to a mirror soon and probably a few more drives after that.

Currently I'm just using the soft passthrough to truenas which causes issues I believe.

Sorry if I should have opened up a seperate thread for this.. Still new here.
Maybe a separate thread that explains your specific issues might get more concrete suggestions for solutions. Please note that there are no guarantees that you can get PCI(e) passthrough to work and there are no guarantees that it will not break (temporarily) with updates.
 

cmonty14

Active Member
Mar 4, 2014
338
4
38
OK.
My conclusion of this discussion is:
It's not recommended to run TrueNAS Core in a VM if best disk performance is required for Proxmox because network stack will limit this.

If virtualization features are important then ZoL should be used in Proxmox, and NAS will not be virtualized.
Any network shared service (NFS, Samba, iSCSI) could be provided by containers running in Proxmox.

Would you agree with with?

Is there anything in TrueNAS Core (feature, function) that could not be build in Debian?
My assumption is that TrueNAS Core delivers a nice and fancy WebUI.
But I'm a fan of CLI and using the right software should deliver me the same features and functions as TrueNAS Core just w/o a WebUI.
 

sublightnova

New Member
May 30, 2022
21
2
3
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!