HDD setup for Plex and other VMS

[TheSpartan18k]

Hello,

I am new to Proxmox. I am looking for the best way to setup my HDDs for Plex. My setup is 128gb of ECC RAM. Dual e5-2560 v2s and 8 identical 3TB HDDs. Proxmox is installed in a ZFS raid on two SSDs. My problem is that I want to be able to move my media storage around from VM to VM. I read that using 6 HDDs in ZFS2 is ideal for performance, which I would also like because the hard drives are from 2011. So do I just pass through my HDDs to Ubuntu and create the raid pool there to be able to move it or do I create a pool in Proxmox? I plan to create my first media server and move all of my existing media from my Windows server and then create a second, more advanced media server that I would like to move my storage to. Additionally, I know that in the future I will outgrow this server and need to move the media storage again. In addition to the media server, I will have other VMs for things like a windows machine, minecraft server, and a website. Docker will be used for the media servers as well.

Thanks for any advice!

 

[Dunuin]

First I wouldn't run VMs on HDDs, HDDs are very bad as a VM storage because VMs need alot of IOPS and HDDs just can't handle that (expecially if these are not some enterprise 10K or 15K RPM disks). As a VM storage you really want some enterprise grade SSDs (look for some second hand ones with good SMART values if you are low on budget).

For your media I wouldn't use big virtual disks. Keep in mind that VMs can't share the same virtual disk. So you want to use SMB or NFS shares to bring your media into your VMs. So what you really need is some kind of NAS. Here you have two options:
1.) if you want a nice GUI and ZFS I would recommend buying a LSI HBA that supports 8x SATA ports (like a Dell PERC H310 that costs around 30-50€ and can be flashed into IT mode). Then you can create a TrueNAS VM and use PCI passthrough to passthorugh the complete HBA card with all 8 disks attached to it into your TrueNAS VM. That way TrueNAS can directly access the disks and you get no overhead. You can then create a 8 disk raidz2 inside that TrueNAS VM and the VM will share the folders using NFS, SMB or iSCSI.
By the way...as long as your packets don't leave your server (for example if a plex VM will access a TrueNAS VM and they are on the same bridge) your are not limited to the speed of your NIC but limited to how many packets your CPU can handle. So if your CPU is fast enough and you use jumboframes, you VM to VM network communication might for example woth with 10Gbit or more even if your server only got 1Gbit NICs.
2.) You use your mainboards SATA ports (you can't use HW raid controllers if you want to use ZFS, so make sure your mainboard isn't using any onboard raid controller) and create a 8 disk raidz2 directly on your host. Then you have two options to setup a NFS/SMB server. First you could install a SMB or NFS server directily on your host. But Proxmox got no GUI for that you you need to install and setup everything yourself using CLI. The better option would be to create a privileged debian LXC, bind-mount some dataset mountpoints from your hosts ZFS pool into your LXC and run the OpenMediaVault installer script inside that LXC. You then got a OpenMediaVault LXC that has access to the ZFS pool on your host and you can use the OMVs WebUI to manage your NFS/SMB shares and user accounts. Here you again shouldn't got overhead because there isn't anything virtualized.

Also keep in mind that a raidz1/2/3 can't be extended later. If you create a raidz2 of 6 disks you need to add atleat 6 more disks. There is a new feature that should allow this but I'm not sure if Proxmox already supports it. And that new feature will only increase the capacity. Adding new disks won't increase the performance if you don't destroy your pool and create it new from scratch. And raid never replaces a backup so you should buy some HDDs to be able to backup everything. And also keep in mind that resilvering a pool might take days or weeks and while the resilvering is rebuilding the pool the disks are tortured 24/7, get unusable slow and are very likely to fail. You could speed that up by striping smaller raidz1s or by using striped mirrors instead of raidz1/2/3. If you don't trust your HDDs because they are already 10 years old you really should consider using them as a striped mirror. That way up to 4 of 8 disks may fail without loosing data and it is way faster to resilver that pool if one of the disks fails.
And a pool gets slow if it is more than 80% filled and ZFS switches into panic mode if it gets over 90%. So you should set a quota of 80% or 90% for the complete pool so you are never able to accidentally write more then 80/90% to it.

 

[TheSpartan18k]

Thank you for the quick reply. I actually already have an HBA card with all 8 SATA drives connected and it is flashed to IT mode. I will create a TrueNAS server VM and set that up as my main storage for the media on Proxmox. Overall, how would you setup the TrueNAS raid pools then? I will eventually need more storage and I am not particularly worried about the safety of the data because it is just media files and nothing important, but I wanted some redundancy as to not have to start from scratch in the event of a failure.

 

[Dunuin]

You can always run into bugs that force you to destroy the compete pool or the complete pool just degrades and can't be restored. I for example once created a snapshot with a "/" in the name but a "/" is forbidden because it is used to define datasets like "mypool/mydataset/myotherdataset". But somehow TrueNAS allowed me to create that snapshot with that name. So I got a snapshot like "mypool/mydataset@snapshot/name". That snapshot worked fine but it was impossible to delete it. I even wasn't able to destroy the complete pool because ZFS complained that it can't destroy that snapshot. So my only option to delete that snapshot was to wipe all disks and create the pool from scratch. And just ignoring that snapshot isn't possible with ZFS because the snapshot will grow and grow the older it gets. Basically as long as there is a snapshot ZFS can't delete any data...

Like I already written above, I would use the disks as a striped mirror. Your disks will most likely fail soon if they are already 10 years old and with a striped mirror the resilvering is way faster. Its also way easier to extend a striped mirror because you will be able to add disks in pairs and get the full additional capacity and performance. With a 8 disk striped mirror 1 to 4 drives may fail. With a raidz2 only 2. Its a bit more gambling with a striped mirror because if the wrong 2 disks are failing at the same time all data is lost but statistically a striped mirror should be more safe. And the resilvering can be done in hours instead of days/weeks so your downtime is way smaller and the disks are not tortured for that long so the chance that they fail too while resilvering is lower.

A great feature of ZFS is the incremental replication. I got two identical TrueNAS servers in my homelab and once a week the main TrueNAS will replicate all pools to the backup TrueNAS. So if any of the pools may fail I won't loose any data. I could buy some new drives, recreate the pool and reverse the replication to sync everything back. And you don't need two servers for that. Another TrueNAS VM with some USB-HDDs should work too.