Recommendations on the best storage configuration

[sismondi]

Hello. I hope you are all okay.

I've been reading a lot these days. But the number of variables is very high.

I am migrating to proxmox from omv.

The idea is to create several virtual machines and containers for: web servers, nextcloud, radarr, sonarr, plex, omv, code server, qbittorrent, etc.

Some of the data is very important to me (nextcloud for example) while others I am not too interested in if they are lost in the future (movies and music).

For important things, I already bought a mini PC to run Proxmox Backup Server and make backups every night.
But for me it doesn't make much sense to make backups or have redundancy for movie storage.

Therefore, in addition to zfs, I am exploring other alternatives such as snapraid+mergefs, lvm-thin, etc.

My server has the following features:

amd ryzen 4750g pro
64gb ram (not ecc)
4 ssd intel dc s3710 400gb
1 960gb ssd (not enterprise grade)
1 720gb ssd (not enterprise grade)
1 nvme 256gb samsung 960 evo
4 hdd toshiba N300 6tb
3 hdd ironwolf 4tb
1 hdd ironwolf 3tb
lsi 9207i

What combinations of configurations do you recommend for storage?

In advance I thank you very much for the help you give me.

 

[Dunuin]

4 ssd intel dc s3710 400gb

Those in a striped mirror as a VM storage for everything that writes a lot or is doing sync writes.

4 hdd toshiba N300 6tb
3 hdd ironwolf 4tb
1 hdd ironwolf 3tb

Maybe something like unraid for cold storage which allows to mix different sized disks?
Or try to sell the 3 and 4 TB disks and get more 6TB disks instead for a raidz1 or raidz2?

1 960gb ssd (not enterprise grade)
1 720gb ssd (not enterprise grade)
1 nvme 256gb samsung 960 evo

No idea what to use those for. Without parity/mirroring I wouldn't put data on it that you rely on. And probably a waste to use them for toring stuff like ISO. My best idea would be to use them as PBS datastores but you said you already got another PBS.

 

[sismondi]

Those in a striped mirror as a VM storage for everything that writes a lot or is doing sync writes.

Would the operating system also go on those disks?

Maybe something like unraid for cold storage which allows to mix different sized disks?
Or try to sell the 3 and 4 TB disks and get more 6TB disks instead for a raidz1 or raidz2?

and make two raidz1?
one with the 4 toshiba drives and the other with the 3 4tb ironwolf drives?

How does ZFS deal with mechanical disks that are not enterprise grade?

No idea what to use those for. Without parity/mirroring I wouldn't put data on it that you rely on. And probably a waste to use them for toring stuff like ISO. My best idea would be to use them as PBS datastores but you said you already got another PBS.

On the mini PC I have an 800GB S3700. I suppose I could move it to the server and use these disks in the mini pc for pbs. would this be okay?

 

[Dunuin]

Would the operating system also go on those disks?

Thats possible but dedicated system disks would be better in case you got 2 small spare disks for a mirror. If not I would prefer to use the this striped mirror over a single system disk.

and make two raidz1?
one with the 4 toshiba drives and the other with the 3 4tb ironwolf drives?

Yes, thats what I would do if you have to stick with those diska you got. Maybe also add 2 additional SSDs in a mirror as special devices to speed the disks up.

How does ZFS deal with mechanical disks that are not enterprise grade?

You shoulsn't use SMR (shingled magnetic recording) disks. But as long as those consumer disks are CMR it should be fine. At least in a homelab.

But not a good idea to use consumer SSDs with ZFS because of the wear ans bad performance.

On the mini PC I have an 800GB S3700. I suppose I could move it to the server and use these disks in the mini pc for pbs. would this be okay?

Yes. But then the question would be what to do with that single 800GB SSD in case you care about your time or data as you then you probably want to but a second one for mirroring,

 

[sismondi]

Thats possible but dedicated system disks would be better in case you got 2 small spare disks for a mirror. If not I would prefer to use the this striped mirror over a single system disk.


Yes, thats what I would do if you have to stick with those diska you got. Maybe also add 2 additional SSDs in a mirror as special devices to speed the disks up.


You shoulsn't use SMR (shingled magnetic recording) disks. But as long as those consumer disks are CMR it should be fine. At least in a homelab.

But not a good idea to use consumer SSDs with ZFS because of the wear ans bad performance.


Yes. But then the question would be what to do with that single 800GB SSD in case you care about your time or data as you then you probably want to but a second one for mirroring,

Hi.

I've been thinking a lot.

I came to this conclusion. If I'm wrong please tell me.

I am going to buy 2 SSDs with a capacity of 120GB for the operating system using zfs mirror.

I will use the four SSDs with a capacity of 400GB disks in RAIDZ 10 for VMs.

I will use the four 6TB disks for data that I really cannot lose under any circumstances with zfs (although I don't know which is a better option, raidz10, raidz1 or raidz2).

For the three 4tb disks and the 3tb disk I will use snapraid+mergefs and I will only use them for movies and series.

Do you think this would be an optimal configuration?

 

[JSEHV]

I can really recommend the Samsung PM883 (eg. 960GB) to use for ZFS or Ceph on Proxmox.
They never let me down and they are still very affordable now the SSD prices are on the rise.

Cheapest new I've found for the 960GB version, €67 ex VAT: https://smicro.eu/samsung-ssd-960gb-2-5-sata-pm883-tlc-mz7lh960hajr-00005

 

[Dunuin]

I will use the four SSDs with a capacity of 400GB disks in RAIDZ 10 for VMs.

When doing that keep in mind that any raidz requires you to increase the blocksize if you don'T want to lose too much capacity due to padding overhead. In this case 16K or even 64K so running stuff like DBs won't be great that are doing small IO (8K = 50% capacity loss; 16K = 33% loss; 64K = 27% loss). And IOPS performance will also only be half of a raid10.
So for a VM storage you usually want a raid10.

 

[sismondi]

When doing that keep in mind that any raidz requires you to increase the blocksize if you don'T want to lose too much capacity due to padding overhead. In this case 16K or even 64K so running stuff like DBs won't be great that are doing small IO (8K = 50% capacity loss; 16K = 33% loss; 64K = 27% loss). And IOPS performance will also only be half of a raid10.
So for a VM storage you usually want a raid10.

I didn't understand what you want to tell me.

raidz 10 is different from raid 10?

what I want to do is a striped mirror

 

[Dunuin]

I didn't understand what you want to tell me.

raidz 10 is different from raid 10?

what I want to do is a striped mirror

Sorry, I've read "raidz1" which would be a raid5. Raid10 aka striped mirror would be fine. with the default 8K block size.

 

[nautilus7]

Hi, I would like some advice on a similar ssd configuration. I have:

3 x 400gb intel s3710
4 x 150gb intel s3520

It's not necessarily to use all of them and there's also a possibility to buy more discs. One problem is that I have only 4 sata ports, so they should be 4 ssds in total.

How would you recommend to setup them for a proxmox machine running opnsense, adguard home, frigate nvr, plex/jellyfin, qbittorrent, truenas, a small wordpress blog, home assistant and some other non critical lxcs/vms?

My hdds are passed through to the truenas vm for bulk storage.

I would like to know how to setup the ssds for proxmox installation and vm/lxc storage. Possible ideas:

1. Use 2 zfs mirrors, one for vm storage, one for proxmox installation. Which discs to select for each?

2. Use a zfs stripped mirror with 4 ssds. Either with the 3 existing s3710 plus another one I would buy, or the 4 existing s3520, for both vm storage and proxmox installation.

3. Use a zfs mirror with 2 of the existing ssds for vm storage and buy a single nvme for proxmox installation or the other way around (single nvme for vm storage and zfs mirror for proxmox installation).

 

[Dunuin]

With only 4x SATA available I would choose option 2 and buy a fourth S3710. I also got 4x S3710 400GB in striped mirror for VM storage for one node as well as 2x S3710 400GB for system+VMs on another node. Performance should be fine...problem is more running out of space...640GB of usable storage really isn't that much...especially when also using snapshots.

 

[nautilus7]

Thanks for the reply. What do you use for system storage on your other node?

I have done some research on power consumption of different hardware lately and how they behave when combined together regarding to c-states (cpu), aspm (pcie), devslp (sata). I noticed that most of the enterprise ssds, although having plp, they do not support devslp.

The intel s3710 and s3520 do not support devslp which limits my machine to only reach package c6 state. On the contrary with a consumer samsung ssd it could easily reach package c10 when idling. The only enterprise ssds that seem to support devslp are the seagate nytro series, but I haven't get any to test.

 

[Dunuin]

Thanks for the reply. What do you use for system storage on your other node?

2x S3700 100GB in mirror.

I have done some research on power consumption of different hardware lately and how they behave when combined together regarding to c-states (cpu), apsm (pcie), devslp (sata). I noticed that most of the enterprise ssds, although having plp, they do not support devslp.

Yes, and the power consumption of those S3710 isn't great either. According to datasheet 0.61W idle, up to 10.1W peak and averaging at 2.8/5.5W while reading/writing per disk. Easily adds up. Here they are busy all the time.