Using ZFS Pool as "local" storage

[cmonty14]

Hello,
I have a server with ECC RAM and multiple disks, means it's equipped like a NAS.

However, I want to install Proxmox and run a VM with TrueNAS Core.
All relevant disks (e.g. WD Red) will be configured as passthrough for this VM.

Then in TrueNAS Core I will configure these drives for ZFS.

My question is:
Can I use this ZFS storage in Proxmox directly w/o setting up a share in TrueNAS Core?
Because effectively the relevant disks are HW connected to bus controlled by Proxmox OS.

But I assume disk passthrough will prevent this.
How can I then utilize the storage provided by TrueNAS Core most efficient w/o any latency caused by network stack in Proxmox?

THX

 

[leesteken]

.... I want to install Proxmox and run a VM with TrueNAS Core.
All relevant disks (e.g. WD Red) will be configured as passthrough for this VM.

Then in TrueNAS Core I will configure these drives for ZFS.

Note that some WD Red use SMR which can give problems with ZFS. Only WD Red Plus is guaranteed to be CMR.

My question is:
Can I use this ZFS storage in Proxmox directly w/o setting up a share in TrueNAS Core?
Because effectively the relevant disks are HW connected to bus controlled by Proxmox OS.

No, you cannot share disks between VMs or between VMs and the Proxmox host. Only containers can share storage using mountpoints.

But I assume disk passthrough will prevent this.

There is nothing to stop you, but just as you cannot share disk between physical machines at the same time, this will corrupt the files and directories on the disk.

How can I then utilize the storage provided by TrueNAS Core most efficient w/o any latency caused by network stack in Proxmox?

In the same way as other physical or virtual machines can use TrueNAS shares.

I know it is good practice to abstract storage and separate it from the VM virtualization hosts (like a SAN), but sharing the ZFS storage from Proxmox directly is much simpler (especially for Linux containers). EDIT: You could use TurnKey Linux File Server in a container to share the ZFS storage from the Proxmox host and separate/abstract it in that way.

 

[cmonty14]

So, if I deploy TrueNAS Core in a VM, I must accept the disadvantage that shared storage used by Proxmox, e.g. ISO, images, etc. will always be thwarted by the network stack?

If I setup ZoL in Proxmox, can I utilize this storage in a VM running TrueNAS Core "directly"?
And what about the features provided by TrueNAS Core for ZFS performance improvement? Would these features still work?

Or would I get the best storage performance if I install TrueNAS Scale on the server and deploy VMs in that OS?

 

[leesteken]

So, if I deploy TrueNAS Core in a VM, I must accept the disadvantage that shared storage used by Proxmox, e.g. ISO, images, etc. will always be thwarted by the network stack?

Yes. Unless you use containers (that run a Linux distro) on Proxmox.

If I setup ZoL in Proxmox, can I utilize this storage in a VM running TrueNAS Core "directly"?

No. There will be the same network stack as above, just the other way around.
That Wiki feels old, maybe have a look at the current manual?

And what about the features provided by TrueNAS Core for ZFS performance improvement? Would these features still work?

You can tune ZFS settings on Proxmox as much as you want manually. What are those features you are talking about?

Or would I get the best storage performance if I install TrueNAS Scale on the server and deploy VMs in that OS?

I don't know. Performance can mean different things to different people and I don't know what your other concerns are. Bare-metal or Proxmox with LVM could be faster.

 

[Dunuin]

No. There will be the same network stack as above, just the other way around.
That Wiki feels old, maybe have a look at the current manual?

There is virtfs which will allow to passthrough a folder into a VM, similar to bind-mounts into a LXC: https://www.linux-kvm.org/page/VirtFS
But Proxmox doesn't support that yet, so you would need to set it up yourself using the CLI editing the KVM parameters in the VMs config file.

 

[sublightnova]

Would getting a dedicated controller card and using full passthrough to truenas (and ONLY truenas) be stable enough to not have extreme worries about a ZFS Pool? I mean.. still have to build it up.. Currently only 1 drive.. Going to a mirror soon and probably a few more drives after that.

Currently I'm just using the soft passthrough to truenas which causes issues I believe.

Sorry if I should have opened up a seperate thread for this.. Still new here.

 

[leesteken]

Would getting a dedicated controller card and using full passthrough to truenas (and ONLY truenas) be stable enough to not have extreme worries about a ZFS Pool? I mean.. still have to build it up.. Currently only 1 drive.. Going to a mirror soon and probably a few more drives after that.

Currently I'm just using the soft passthrough to truenas which causes issues I believe.

Sorry if I should have opened up a seperate thread for this.. Still new here.

Maybe a separate thread that explains your specific issues might get more concrete suggestions for solutions. Please note that there are no guarantees that you can get PCI(e) passthrough to work and there are no guarantees that it will not break (temporarily) with updates.

 

[cmonty14]

Currently I'm just using the soft passthrough to truenas which causes issues I believe.

What is a "soft passthrough"?

 

[Dunuin]

What is a "soft passthrough"?

I think he means passing trough individual disks using "qm set" like described here: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)
That is not a real physical passthrough as the VM still only sees virtual disks.

 

[cmonty14]

OK.
My conclusion of this discussion is:
It's not recommended to run TrueNAS Core in a VM if best disk performance is required for Proxmox because network stack will limit this.

If virtualization features are important then ZoL should be used in Proxmox, and NAS will not be virtualized.
Any network shared service (NFS, Samba, iSCSI) could be provided by containers running in Proxmox.

Would you agree with with?

Is there anything in TrueNAS Core (feature, function) that could not be build in Debian?
My assumption is that TrueNAS Core delivers a nice and fancy WebUI.
But I'm a fan of CLI and using the right software should deliver me the same features and functions as TrueNAS Core just w/o a WebUI.

 

[sublightnova]

I think he means passing trough individual disks using "qm set" like described here: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)
That is not a real physical passthrough as the VM still only sees virtual disks.

Correct.. I meant that method which I'm using now...

Thanks for the replies... It is not extremely critical data perse just more of a home(lab) situation but still would like to have it as stable as possible.