Recommendations on PVE environment

[donnib]

Hi,

I have following today :
- Synology in a RAID1 - 2 disk with all my photos/music etc. Total storage 3.6Tb and i have 35% left.
- PVE running on a Mac Mini with one VM (debian) with a bunch of docker containers, and all docker data is on the Synology and accessed over NFS.
- I do daily backup of the VM to the Synology.
- I do daily backup of the Synology to MS Azure Blob Storage <- free credit through my work

I have now bought following or had laying around :
- HP DL360 Gen8, with 8 bays
- I have 4 x 480gb Kingston SSD brand new ones laying around and a 160gb SSD intel (older one).
- A M1015 HBA card flashed in IT mode WITHOUT boot option.

What i'd like :
- Get rid of the Synology and replace with HP server
- Get rid of the Mac Mini and replace with HP server
- Move my important data to ZFS pool <- my photo/videos, important documents
- Move not important data to MergerFS / Snapraid <- Downloads, Music etc

Specifics around the HP server :
I understand that this server does not have UEFI and since i have not flashed boot on the controller (LSI) i need another way to boot. Was it a bad idea not to have boot on the controller ? I heard that if i have boot on it it requires to click a key during a boot to continue.

Questions :

1. Considering the specifics of this server how would you install PVE ? On a ZFS ? RAID 1 ? ext4 ? I read other places that i can install on a SATA disk, use CDROM connection or install on one of the disk on the HBA controller then move the boot to a USB key.
2. How would you organize the data in my case ? Is it a bad idea to use ZFS at all ? Should i use something else ?
3. I have the 4 x SSD 480gb disks, would you use them ? If so would you use them for ZFS or for the MergerFS/SnapRAID ?
4. My Synology has 2 x 3.5" 7200rpm drives, the new sever has only 2.5" bays and consumer drives runs at 5400rpm, would i notice a difference in speed compared to synology ? I don't do heavy transfers, i think only the backup made from the Mac Mini to the Synology is the only transfer that is large e.g 25gb daily when it does the backup.

Any other ideas or things i haven't thought of please do comment, i am just starting so i am sure there are things i haven't thought about.
Some might ask why do i even want to use PVE, well i like that i can back up the whole things and restore quickly if needed and i also like that i have the option to spin up a VM if i need to in the future. Another question some might ask would be why not drop the LSI controller and use the build in P420i RAID controller and run RAID....well i heard that's not a good option because you can't monitor the disks, please correct if i am wrong and that's a good idea.

/donnib

 

[Dunuin]

1. I have the 4 x SSD 480gb disks, would you use them ? If so would you use them for ZFS or for the MergerFS/SnapRAID ?

I wouldn't use them for ZFS. ZFS can kill your SSDs quite fast if you don't use high endurance enterprise SSDs.

1. My Synology has 2 x 3.5" 7200rpm drives, the new sever has only 2.5" bays and consumer drives runs at 5400rpm, would i notice a difference in speed compared to synology ? I don't do heavy transfers, i think only the backup made from the Mac Mini to the Synology is the only transfer that is large e.g 25gb daily when it does the backup.

Did you verified that the server can use SATA drives? These servers normally use backplanes and if it is a SAS backplane you can only use SAS drives.
2,5" HDDs are way slower. Not only because of the 5400rpm, they are also using less discs stacked inside and because the discs are smaller they are less fast at the edge of the disc.

 

[donnib]

I wouldn't use them for ZFS. ZFS can kill your SSDs quite fast if you don't use high endurance enterprise SSDs.

Did you verified that the server can use SATA drives? These servers normally use backplanes and if it is a SAS backplane you can only use SAS drives.
2,5" HDDs are way slower. Not only because of the 5400rpm, they are also using less discs stacked inside and because the discs are smaller they are less fast at the edge of the disc.

@Dunuin thx for replying. Ok so point taken, not good to use SSD for ZFS, would they be ok for MergerFS/Snapraid ?

Yes i know that i can use SATA disks, i can use both SATA and SAS disks. I have tested it and it works with SATA.

It's sad to hear about the speed of the 2.5" disks, what should i expect write/read speeds from such disks do you think ?

 

[Dunuin]

It's sad to hear about the speed of the 2.5" disks, what should i expect write/read speeds from such disks do you think ?

Should be around 1/2 to 2/3 of the Speed of your 3,5" HDDs. And the capacity is quite limited if you only use 2,5" disks. The biggest consumer disk you can buy without SMR is only 2TB and really expensive (140€). So the server really isn't great as a NAS.
So if you buy 8 of these and use them with raid10 you get 8TB usable capacity for 1120€. Two 8TB 3,5" HDDs as raid1 with also 8TB usable capacity would only cost you 280€.

 

[donnib]

Should be around 1/2 to 2/3 of the Speed of your 3,5" HDDs. And the size is quite limited if you only use 2,5" disks.

Well that's a bummer, so that's like 50mb/s ? Size yes, at most i can get 2tb disks.

Looking at the disks i have today in my Synology : https://hdd.userbenchmark.com/WD-Red-4TB-2013/Rating/3525 vs a 2.5 disk from WD vs https://hdd.userbenchmark.com/SpeedTest/420084/WDC-WD20SPZX-11CRAT0

91,3mb/s vs 119mb/s. a difference of 30% loss in performance. I could live with that, it's rare i do huge transfers i mean 15gb transfer is 13 vs 20s in time, not big issue i guess.

Regarding ZFS ans SSD, i seem to read many run with them and they are happy, do you have something i can read why this is a bad combination ?

 

[Dunuin]

To quote the Proxmox Teams ZFS SSD benchmark paper:

Can I use consumer or pro-sumer SSDs, as these are much cheaper than enterprise-class SSD?
No. Never. These SSDs wont provide the required performance, reliability or endurance.

You can use cheap consumer SSDs but each write will damage the SSD. And ZFS is a copy-on-write filesystem and will write alot. If consumer SSD will work or not depends on your workload and how high your write amplification is. Your SSD may live for years or die within months.
I for example once bought two Samsung 970 Evo 500GB SSDs. They only got 300 TBW so they are rated to survive 300TB of data written to them.
I got a write amplification of factor 20. So if I try to write 15TB of data that will result in 300TB written to that SSD. After writing 15TB of data OR after 5 years the warranty will expire. You will reach that 15TB after 5 years if you continously write with 100kb/s. And it's really not hard to do that if all your VMs are creating logs, storing metrics and so on. I looked at the write rates and calculated that my new bought SSDs won't survice a single year. So I removed them and replaced them with second hand enterprise SSD that are 30 times more durable.

So you can use your SSDs but don't wonder if you need to replace them after a year or so.

 

[donnib]

To quote the Proxmox Teams ZFS SSD benchmark paper:


You can use cheap consumer SSDs but each write will damage the SSD. And ZFS is a copy-on-write filesystem and will write alot. If consumer SSD will work or not depends on your workload and how high your write amplification is. Your SSD may live for years or die within months.
I for example once bought two Samsung 970 Evo 500GB SSDs. They only got 300 TBW so they are rated to survive 300TB of data written to them.
I got a write amplification of factor 20. So if I try to write 15TB of data that will result in 300TB written to that SSD. After writing 15TB of data OR after 5 years the warranty will expire. You will reach that 15TB after 5 years if you continously write with 100kb/s. And it's really not hard to do that if all your VMs are creating logs, storing metrics and so on. I looked at the write rates and calculated that my new bought SSDs won't survice a single year. So I removed them and replaced them with second hand enterprise SSD that are 30 times more durable.

So you can use your SSDs but don't wonder if you need to replace them after a year or so.

Got it, i can use it as normal disks then right ?