Looking for advise: zfs should belong to Proxmox or Truenas?

[lawf]

I have sufficient disks to create an HDD ZFS pool and a SSD ZFS pool, as well as a SSD/NVMe for boot drive. I'd like to install Proxmox as the hypervisor, and run some form of NAS software (TRueNAS or something) and Plex. Looking for advise on how that should be setup, from a storage perspective and VM/Container perspective.

I'm guessing I should let Proxmox "own" the SSD ZFS pool, and I'll put my VMs and Containers there? The HDD ZFS pool will be my data pool, where I'll store all my media, and my file shares (as NAS.) Should I install TrueNAS and passthrough the HDDs to TrueNAS? Or should Proxmox create the HDD ZFS pool? If so, how will the NAS and Plex have access to the Proxmox HDD ZFS pool? I can think of many ways to skin the cat, but which way is the most efficient (best performance)?

While I'm in the asking mode, I'll also ask the following questions:
Can VM access Proxmox ZFS natively? Can Container access Proxmox ZFS natively?
I need a NAS to shares my HDD storage. What should I use? TrueNAS? Or something else like Open Media Vault? Do I make it a VM or Container?
As for Plex, I believe a Container running the Plex server is sufficient?

Thank you all in advanced.

 

[rungekutta]

If you are going to use TrueNAS, it wants low-level access to the disks so you should pass them through, or actually ideally the whole controller. Some people do that by running the disks on a PCIe HBA which is then passed through to TrueNAS running in a VM. You may be able to pass through the controller on the motherboard too, depending.

In any case you can’t run TrueNAS in a Container as it’s FreeBSD based, not Linux. TrueNAS Scale (currently in alpha or beta) *may* work, but I wouldn’t do it myself, much too likely that something will go wrong.

The other option is to skip TrueNAS altogether and let Proxmox itself be your NAS. That gains you some efficiency as the hypervisor gets native access to the ZFS pool (as opposed to having to go through a network layer). And is a simpler setup so one less thing to go wrong. What you lose are some of the nice built-in features of TrueNAS, for example automatic/periodic snapshots, backup to cloud or synchronization of ZFS snapshots to remote site, monitoring. You can do all if that in Proxmox too but typically requires a bit more work and custom setup.

I have no experience of Open Media Vault. Plex runs fine in a container in Proxmox. Although I recommend Jellyfin instead.

 

[Dunuin]

If you are going to use TrueNAS, it wants low-level access to the disks so you should pass them through, or actually ideally the whole controller. Some people do that by running the disks on a PCIe HBA which is then passed through to TrueNAS running in a VM. You may be able to pass through the controller on the motherboard too, depending.

Jep, if you really want TrueNAS you should get a LSI PCIe HBA and use PCI Passthrough to bringt it into a TrueNAS VM including all Disks attached to it. That should give you the best performance and less problems because it is the only way how a TrueNAS VM can physically access the real disks.

I have no experience of Open Media Vault. Plex runs fine in a container in Proxmox. Although I recommend Jellyfin instead.

If you don't need all that advanced features of TrueNAS a OMV inside a LXC will work too. You can than create a ZFS pool on your host, bind-mount the datasets into that OVM LXC so you can manage users and shares using the OMV WebUI.

Can VM access Proxmox ZFS natively?

No.

Can Container access Proxmox ZFS natively?

Not ZFS itself, but mountpoints of ZFS datasets. So yes, you could access Data stored on ZFS from inside a LXC.

As for Plex, I believe a Container running the Plex server is sufficient?

That depends how secure your want your Plex. If you plan to open some ports in your router so you can access your Plex from the internet I would use a VM because this is more secure. If you just want to access Plex from your LAN a LXC would be totally fine.

 

[lawf]

Thank you all for your reply and suggestion. Some follow-up questions/comments after digestion:
I have an HP DL380Gen8 with 25 drive bays. Not sure how easy it is to find a LSI PCIe HBA to take in all 25 bays. It is currently running a P420i and I put it in HBA mode so a natively loaded OS (Proxmox/TrueNAS) can see all 25 drives without being RAID'ed. I also don't think it would be easy to do PCI passthrough of the P420i to either Proxmox or TrueNAS VM.

If I skip TrueNAS and let Proxmox run ZFS and let it be my NAS, how will I do that? NFS server is easy to setup under Proxmox/Linux, but do I need to install Samba to do Windows/SMB share? I would like a "simpler" and cleaner solution. Maybe PMV inside a LXC is my NAS solution? Or TrueNAS running as a VM an NFS mount the Proxmox ZFS? Humm...

Similarly I guess I can run Plex inside a LXC and bind-mount my media datasets from Proxmox ZFS. Heard of Jellyfin but not familiar with it. Why do you prefer it over Plex? How's it "better"? If on the other hand I run Plex as a VM (assume I install some kind of Linux and install Plex server on it), how will Plex have access to the media datasets? Share it out from Proxmox NFS server and NFS mount it to the Linux VM? Do I take a performance hit on that, understanding that both the NFS server and NFS client are on the same physical server, with one a native OS and the other VM? Maybe build the two to talk over the same (private) bridge?

Thanks.

 

[Dunuin]

Thank you all for your reply and suggestion. Some follow-up questions/comments after digestion:
I have an HP DL380Gen8 with 25 drive bays. Not sure how easy it is to find a LSI PCIe HBA to take in all 25 bays. It is currently running a P420i and I put it in HBA mode so a natively loaded OS (Proxmox/TrueNAS) can see all 25 drives without being RAID'ed. I also don't think it would be easy to do PCI passthrough of the P420i to either Proxmox or TrueNAS VM.

You could try to passthrough the P420i. Should work if it is a PCIe card. If its onboard it depends how your mainboard is handling IOMMU groups. But in both cases you need to run PVE from disks that are not connected to the P420i, because you can only passthrough the complete controller and not individual disks. And PVE shouldn't be run from USB Sticks/SD cards because it is writing too much. So if you don't got another SATA controller onboard or M.2 Slot onboard you might need to use a USB SSD for your host OS.

If I skip TrueNAS and let Proxmox run ZFS and let it be my NAS, how will I do that? NFS server is easy to setup under Proxmox/Linux, but do I need to install Samba to do Windows/SMB share? I would like a "simpler" and cleaner solution.

Yes, PVEs WebUI doesn't offer any NAS functionalities. So if you want to run NFS/SMB directly on your host itself you would need to manage everything using CLI and edit those config files by yourself.

Maybe PMV inside a LXC is my NAS solution?

Like I already said a OMV inside a LXC will work. But you still need to manage the ZFS yourself on the host because a OMV LXC can only access the files/folders of a ZFS dataset but can't manage your ZFS (so no snapshots, smart monitoring, replication and so on).

Or TrueNAS running as a VM an NFS mount the Proxmox ZFS? Humm...

ZFS ontop of ZFS is a bad idea because the overhead will multiply. And TrueNAS can only use ZFS. So best to run ZFS only inside your VM or only on your host.

Similarly I guess I can run Plex inside a LXC and bind-mount my media datasets from Proxmox ZFS.

Yes, you can do that but I would prefer the NAS way. Not sure how it works with plex but with emby you basically want your stuff on a SMB share because all local media will also be mapped to a SMB share. So If you want to stream a video from your LAN the emby client will first check if that video is also available on the SMB share. If it is the client will directly stream the video using SMB so there is no additional video encoding. It will just open the file from the SMB share and play it back. If you don't got that video on a SMB share the client will ask the emby server to send a stream. The emby server will open the local file, encode it in software (which will basically bring your CPU to 100% utilization) or in hardware (if you got a dedicated GPU and a subscription to be able to use hardware acceleration) and then steam this newly encoded video.
So basically why rip a video from one format to another if you can just open the file on a network share.
The other point is that you somehow need to manage your media. How do you store your videos on that ZFS pool? Easiest way to manage your media libraries would be to use SMB. And then you maybe also want other VMs to store/access files on all your HDDs. VMs can't bind-mount so they really need SMB/NFS to access them.

Heard of Jellyfin but not familiar with it. Why do you prefer it over Plex? How's it "better"? If on the other hand I run Plex as a VM (assume I install some kind of Linux and install Plex server on it), how will Plex have access to the media datasets? Share it out from Proxmox NFS server and NFS mount it to the Linux VM?

I tested plex, jellyfin and emby and I prefer emby. With plex you are way more limited if you don't buy a subscription because many of the interesting features aren't usable without it. And jellyfin wasn't that advanced in developement compared to plex or emby. With emby you get great support (the developer is answering every thread himself) you can nearly use all features without a licence. Only important features missing with the free version are PVR and hardware encoding. And if I remember right the media recognition of emby was better.

And yes, if you want a plex VM NFS/SMB will be fine.

Do I take a performance hit on that, understanding that both the NFS server and NFS client are on the same physical server, with one a native OS and the other VM? Maybe build the two to talk over the same (private) bridge?

You will always get overhead using NFS/SMB but as long as both VMs/LXCs are attached to the same bridge your network performance is only limited by the speed of your CPU. So the VMs/LXCs can communicate with 10+ Gbit/s if your CPU can handle all the packets even if your server only got 1 Gbit/s NICs.

 

[rungekutta]

Yes, you can do that but I would prefer the NAS way. Not sure how it works with plex but with emby you basically want your stuff on a SMB share because all local media will also be mapped to a SMB share. So If you want to stream a video from your LAN the emby client will first check if that video is also available on the SMB share. If it is the client will directly stream the video using SMB so there is no additional video encoding. It will just open the file from the SMB share and play it back. If you don't got that video on a SMB share the client will ask the emby server to send a stream. The emby server will open the local file, encode it in software (which will basically bring your CPU to 100% utilization) or in hardware (if you got a dedicated GPU and a subscription to be able to use hardware acceleration) and then steam this newly encoded video.

Minor correction. I don't think Emby will re-encode the file if the client is able to support it in its original format. In that case the file will just be streamed as-is. In other cases, the server will re-encode the file into something that the client can support (and this can be a killer without dedicated hardware). But yes, in both cases it will be read from storage and streamed out again to the client.

I tested plex, jellyfin and emby and I prefer emby. With plex you are way more limited if you don't buy a sucscription because many of the interesting features aren't usable without it. And jellyfin wasn't that advanced in developement compared to plex or emby. And yes, if you want a plex VM NFS/SMB will be fine.

Emby was once open-source, but when they decided to take the project closed-source Jellyfin was created as a fork, and remains free and open-source. The two projects have developed in parallell since then (couple of years).

 

[lawf]

Again, thank you all for your comments/suggestions.

I'll try to passthru my P420i and see if that is possible. During POST, it said it is an "embedded" controller, whatever that means.

I install PVE on a separate PCI card with a M.2 SATA drive (SATA connected to the MB.) It is totally "unrelated" to the P420i.

I think I've learned enough of ZFS to manage it between PVE GUI/CLI. That should be no issue. I'll "carve" out datasets in the ZFS pool(s) and mount the datasets to OMV. Currently I'm not sure I want/need snapshot, but I guess it's nice to have. I have no intension to do ZFS over ZFS with TrueNAS. If I do run a TrueNAS VM, I think I'll do PVE ZFS, create datasets, and then have TrueNAS VM mount the datasets via NFS. Wonder how bad the performance will be if they "talk" to each other over a bridge, which is virtualized? I hope I don't need to tie the bridge to a physical NIC and be (worse) limited by the NIC speed? (Edit: I didn't read Dunuin's later response until...later. )

I do intend to share my media ZFS dataset(s) out via SMB. Dunuin is correct in that I will manage the content on the dataset via Windows/SMB (unless I switch over to Linux as my primary desktop, but that's a totally different topic. ) I did not know Emby can/will go straight to the SMB share, and wonder if Plex will do the same. I was always concerned with the speed of media server (Plex/Emby) accessing the media on disk, but never really thought about the path to the TV is as good as its weakest link.

So maybe I'll do this:
PVE ZFS - either passthru P420i, or access as individual disk with P420i in HBA mode.

TrueNAS VM - mount PVE ZFS datasets via NFS from PVE, then share datasets out as SMB (as needed) is that even possible? If so, then this way I get snapshots and all advanced features of TrueNAS. I'll likely use TrueNAS core then as it is more matured. Since I won't be using TrueNAS's virtualization, it doesn't really matter to me whether it is BSD or Linux underneath. I don't want to wait for TrueNAS Scale for another year.
If I cannot utilize TrueNAS to "manage" NFS mounted dirs, then maybe I have to fall back to OMV, with OMV bind-mount PVE ZFS datasets and manage those (I assume that is possible?)
Or maybe just skip PVE ZFS, passthru P420i or HDDs to TrueNAS and let TrueNAS manage them with ZFS? TrueNAS can then share the datasets out as NFS and SMB.

Plex LXC - will play with the difference between bind-mount datasets vs. SMB from TrueNAS VM (build a bridge between Plex LXC and TrueNAS VM.) I do have a Plex lifetime subscription so it would be a waste if I don't use it.

Emby LXC - nothing prevents me from installing Emby and accessing the same media datasets as Plex, right? Then I get to play with both of them and see which one I prefer. If I have to get a Emby subscription later, it's not the end of the world.

Any fault you folks see with the above? Looking for a suggested setup if you are in my situation; what will you do? Thanks again for the advise.

 

[Dunuin]

I have no intension to do ZFS over ZFS with TrueNAS. If I do run a TrueNAS VM, I think I'll do PVE ZFS, create datasets, and then have TrueNAS VM mount the datasets via NFS. Wonder how bad the performance will be if they "talk" to each other over a bridge, which is virtualized? I hope I don't need to tie the bridge to a physical NIC and be (worse) limited by the NIC speed? (Edit: I didn't read Dunuin's later response until...later. )

Like I already said, as far as I know TrueNAS can really only use ZFS and needs direct physical access to the drives. I don't think you will be able to mount a NFS share to use that as your storage. TrueNAS is also an alliance so your can install/setup some individual stuff because it would be overwritten with the next update.
And it is no problem to tie a bridge to a physical NIC. As long as the packets don't need to leave your server you still get all the performance your CPU can handle. And if you don't got 10Gbit or 40Gbit NICs the speed of the bridge or virtual virtio NICs shoudn't be the bottleneck.

If so, then this way I get snapshots and all advanced features of TrueNAS. I'll likely use TrueNAS core then as it is more matured. Since I won't be using TrueNAS's virtualization, it doesn't really matter to me whether it is BSD or Linux underneath. I don't want to wait for TrueNAS Scale for another year.
If I cannot utilize TrueNAS to "manage" NFS mounted dirs, then maybe I have to fall back to OMV, with OMV bind-mount PVE ZFS datasets and manage those (I assume that is possible?)

Even if mounting NFS shares inside a TrueNAS VM would be possible, you wouldn't be able to use snapshots, replication or all the other features that ZFS offers. snapshots and so on are deeply build into the filesystem/disk management of ZFS. This is working on block level and not on file level and NFS is only filelevel and behind a abstraction layer on top of that.

Or maybe just skip PVE ZFS, passthru P420i or HDDs to TrueNAS and let TrueNAS manage them with ZFS? TrueNAS can then share the datasets out as NFS and SMB.

That is what I would prefer if your passthrough is working. If that doesn't work you can look for some cheap rebranded LSI raid controllers. I for example bought two Dell PERC H310 for 35€ each and crossflashed them to be a fully features LSI 9211-8i HBA working in IT-Mode. That also works with some other models. But for the newer PCIe 3.0 chipsets or 16 SATA versions you will pay way more (atlest 100€+).

Emby LXC - nothing prevents me from installing Emby and accessing the same media datasets as Plex, right? Then I get to play with both of them and see which one I prefer. If I have to get a Emby subscription later, it's not the end of the world.

Just make sure that Emby won't store metadata next to your media. You can tell emby to only store them in the DB and not write jpgs and xmls next to it movies and so on.

 

[lawf]

Thank you Dunuin.

I guess I'll passthru the HDDs over to TrueNAS VM, let TrueNAS manages the HDDs under ZFS. If I can get the P420i to PCI passthru, great. Otherwise P420i in HBA/IT mode.

Now, that means Proxmox will have to use TrueNAS shares for its VM and LXC boot and data disks. Is that possible? Or even desired? Or do I need to split the HDDs to give some to Proxmox ZFS and some to TrueNAS ZFS?

Plex and Emby will access the media files from TrueNAS shares as SMB. Should be no problem with that?

Great, the picture is getting clearer. Thanks.

 

[Dunuin]

I personally wouldn't use HDDs as a VM storage but you could run VMs from a NFS share (which isn't uncommon if you got a PVE cluster and need a shared storage). The problem with HDDs is that they only can handle 100-200 IOPS and a raidz pool won't increase the IOPS. If you for example got 8 HDDs in a raidz2 your complete pool still only got the 100-200 IOPS performance of one single disk. And because of virtualization, different filesystems ontop of each other and ZFS you get write amplification so everything causes way more reads/writes than usual. And you usually want to run multiple OSs at the same time and all will permanently access that pool. Remember how fast Windows got back then when everyone switched from HDDs to SSDs. Now imagine running multiple Wins at the same time of the same HDD. And that over network because of the NFS share...
So HDDs are basically only useful if you want to store few but big files that can be read/written sequentially. For everything else you want enterprise SSDs. A VM storage is the exactly opposite of what HDDs are good at. You could use alot of HDDs as a striped mirror (raid10) to increase the IOPS of the pool but that way you loose half of the capacity.
So I would use a dedicated SSD mirror as a VM storage or atleast a single SSD mirror that the system and VMs.

 

[lawf]

Understood. I was setting things up in my test Supermicro with only 4 HDDs to play with different options.

On my "production" (using the term loosely) DL380p, I have 5 SSDs which I will setup as a ZFS pool and one of its use is for VM and LXC storage. The rest of the HDDs will be in another ZFS pool for NAS, Plex/Emby medias...etc. I think it is pretty clear to me now that I'll passthru the HDDs to TrueNAS VM to build the NAS pool. The remaining question is what to do with the SSD ZFS pool? Keep it for Proxmox ZFS? If so, I can't store any NAS data on it/them. Also, since I am splitting my 25 drive bays to different "hosts", I won't be passing the entire P420i as PCI passthru to TrueNAS (assume I can even do that in the first place.)

Decision, decision, decision....

 

[Dunuin]

The remaining question is what to do with the SSD ZFS pool? Keep it for Proxmox ZFS? If so, I can't store any NAS data on it/them.

There are many option. Atleast 2 of them I would use locally on your PVE host a mirrored ZFS system + VM storage. If you need more storage for VMs use 4 of them as a striped mirror and keep one as a spare if one of the SSDs fails.
If your are fine with just 2 SSDs for PVE you could also passthrough the other 3 SSDs and create in your TrueNAS VM an additinal raidz1 pool so you got a small but fast SMB share. Or you could pass the 3 SSDs into your TrueNAS VM and use them as a SLOG or as a special device to speed up your big HDD pool.

Also, since I am splitting my 25 drive bays to different "hosts", I won't be passing the entire P420i as PCI passthru to TrueNAS (assume I can even do that in the first place.)

That isn't working. You can only passthrough the complete controller into a single VM. So if all 25 drives are attached to that controller, all 25 drives will only be available in your TrueNAS VM.

 

[lawf]

Hi Duniun, again, thank you for your suggestion. I'm going to experiment with the settings and I'll report what my final setup looks like. Thanks.

 

[lawf]

Okay, this is my current (final??) setup:
Proxmox 7 as hypervisor; a SATA SSD and a NVMe SSD on a PCI card for local/local-lvm VM storage.
TrueNAS core 12-U6 in a VM; a HDD data (media) pool and a SSD file pool from individual disk passthru. Many of the datasets in the pools are shared out via SMB.
Plex running in a Ubuntu VM (in preparation for a potential future Nvidia PCI passthru for transcoding), mounting the media SMB shares from TrueNAS.

So far so good. Thanks again for everyone's suggestion.