Proxmox Home Media Server, file system and HDD setup

[achimcc]

Hello,

I'm a bit lost on the optimal configuration for a Proxmox Homeserver running Jellyfin and other file and cloud services. I've an Asus C246M WU4 Mainboard with Xeon E2146 CPU. I use two NVMe SSDs in a mirrored setup with ZVS filesystem.

I own one 8TB HDD for media storage which I plan to extend. So far I was just running a single HDD in a NAS without redundancy, which I would like to add. I'm considering to use MergerFS and SnapRaid on three 8TB HDDs. Is this an acceptable setup, also in terms of performance with several users streaming media? Could I add SSD caching? Or should I go for BTRFS or even with ZFS, then probably with 4HDDs (without gaining extra storage in RaidZ2)?

Many thanks in advance,
Achim

 

[brb8two]

Hi Achim,

I assume that you have proceeded since your post 2 weeks ago. Here is some gerneral Advice;

ZFS is recommended with ECC RAM and typically requires 1GB RAM per 1TB Storage. You have a Xeon so that is a possibility but not a Given.

VM's on ProxMox work best when there is High performace Storage involved for VMs. i.e. a Single Spinning disk may not be suitiable for more than a couple VMs depending on the work load. I assuming the SSD will be used for this.

You have indicated you have NVMe SSDs in Mirrored RAID, do they have power loss protection? Consumer Grade SSD's don't Enterprise grade SSD's can, these type are suitiabe for RAID as they insure that the cache is written before power is lost, otherwise you end up with data corruption.

BTRFS and ZFS provide BitRot protection and RAID.

SnapRAID is not real time Protection, this may be suitiable for Media that isn't going to change often. Not generally the focus of Proxmox Forum.

Proxmox has no GUI support for BTRFS
Proxmox has no GUI support for SnapRAID
Proxmox has no GUI support for File sharing

Depends on how much you want to manage via the command line.

With 8TB HDD, Depending on the Brand Model, they Could have a read performance between 80MB/s and 180MB/s and Write Performance of 40MB/s to 180MB/s. (i.e. Segate Archive Drive vs Enterprise NAS Drive) As they are Spinning Disks, Fragmentation over time will also decreese their performence. Now with Streaming Media, I'm assuming Videos, the performance you need depends on the Encoded Bit Rate of the media, number of users and minimally double that to account for fragmentation or fast forwarding, jumping in time of the video. Example 2K Video at 10Mb/s x 4 users = 40Mb/s, double that = 80Mb/s = about 8 MB/s if only streaming videos. If you are also adding content at the same time and performing SNAPRAID snapshots, there could be performance issues (video pauses when the cahce runs out) if the drives are too slow.

Basically Beyond 2 or 4 TB HDD's using RAID other than Mirror for Data protection (redundancy) is not recommended. Basically the story is that if Drives fail after an amount of time of use and you build your RAID set from a new set of drives, they may fail at a similar point in time, days or months of each other. The time it takes to rebuild a RAID5/6 configuration, there is a higher chance of another drive failure during the re-build. I think this more to do with performance of the RAID5/6 cacluations of an operational RAID then just drive failures. i.e. a Mirror RAID or Mirror Stripe RAID will perform as fast as the Disk can perform but RAID5/6 is limited to Processor speed of the controller when there is avalibility for processing.

I have not used ZFS, so I don't know it's limitaitons but from what I have used of BTRFS in Mirror Stripe Mode, it is flexiable enough to add and remove drives to an operational array. I have not tested how it operates with failed drives but assuming you have capacity, you can re-balance before adding a new drive. (i.e. 6 Drive array, drop 1 drive you can re-configure to a 5 drive array before you add the 6th drive again) This may be desireable should you not have the hardware available for some time. BTRFS is still a work in Progress and from what I have read this is mostly to do with RAID5/6 configurations and is regarding memory loss due to power loss.

If you want to use BTRFS, what you can do with ProxMox is to Pass the Physical Drives through to a Guest VM and the Guest VM can handle the drives directly. However Most recommend to pass the Drive controller card to gain the full SMART monitoring capability. Then you can use a Guest VM for NAS like OMV or Rockstor. Rockstor has BTRFS intergrated in the GUI, OMV supports BTRFS but it is not intergrated in the GUI.

I'm trying Proxmox with Rockstor Guest 5 drive pass through BTRFS and OMV 32bit baremetal 2xSata Mirror 2xUSB plan to be mirror internal using with BTRFS.

Anyone feel free to correct me if I've mixed up anything regarding Storage, Proxmox, etc.

 

[Anti-Ctrl]

I only flew over brb8two's comment, but here is what i have:
ZFS RaidZ2 with 6x3TB Drives (mixed WD Red and Generic HDDs). Jellyfin as Media Playback. I transcode directly to ram, so if thats needed, then the Scrubbing Performance is really responsive, as far as the Server can keep up with Transcoding. Read Speed isnt really a Problem for me either, even with 5 or 6 People at once. If you are only Storing Media on that Pool, then maybe a RAIDZ would be plenty for 4 drives, as one can fail and a resilver would "only" take around 24 - 36 Hours (i guess), depending on the Disk. I would go for ZFS because bit rot can occur, even if its unlikely. Media Files are normally stored over a long period of time. BTRFS still has a Write Hole with RAID5/6, so that would be a problem in case of Powerloss. Keep in Mind, that, if you want to expand you Pool, you need to add an additional 4 drives in the same configuration. RAIDZ Expansion is still in Alpha Preview state.

If you use the SSDs for the VMs, then DONT put Jellyfin on there, as it constantly writes some stuff, for whatever reason. For me thats about 3TB a Month or so. For filesharing i just use SMB on the Host itself. You can use built in ZFS smb, which should have a bit better Performance and less overhead.