Which storage type is best for HDD disk?

left4pve

Member
Jul 12, 2021
34
3
13
32
Hey guys. I added a WD 16TB HDD to my PVE server via USB3.0 as a backup disk for photos, videos, downloaded files, etc. Since it's HDD I'm not expecting a very fast speed. When adding the disk to PVE datacenter as storage, there are many options that I can choose, which causes me decidophobia.. So can I get some advice on which one I should use for my use case?
Thanks in advance!



1639623695326.png
 
If you want additional data integrity checks format that disk with ZFS. But ZFS will use alot of RAM (20GB RAM in you case if you use the rule of thumb) and got alot of overhead. If you just want it simple format it with ext4 or xfs.
In both cases you want to add the mountpoint of that partition as a "Directory" storage.

PS: "Datacenter -> Storage -> Add" is just for adding a storage to your PVE that you already have created manually partitioned and formated using the CLI.
 
Last edited:
Thanks @Dunuin! I guess I don't need the fancy feature of ZFS let alone I don't have 20GB of free memory. So EXT4 it is. Thanks!
 
If you want additional data integrity checks format that disk with ZFS. But ZFS will use alot of RAM (20GB RAM in you case if you use the rule of thumb) and got alot of overhead. If you just want it simple format it with ext4 or xfs.
In both cases you want to add the mountpoint of that partition as a "Directory" storage.

PS: "Datacenter -> Storage -> Add" is just for adding a storage to your PVE that you already have created manually partitioned and formated using the CLI.
Would You explain, how to calculate ZFS need of RAM for certain SSD HDD size?
 
Would You explain, how to calculate ZFS need of RAM for certain SSD HDD size?
In short - as always - it depends.

(Open)ZFS - as well as any filesystem - is faster when it stores often used blocks in memory. The ext3/4 familiy is no exception, but the location where it stores this is relevant. Non-ZFS filesystems use the ordinary buffer cache for this, ZFS has its own memory due to some porting related issues from Solaris. Therefore in ZFS, the OS cannot as easily flush the buffer cache to make room for other things, so that no OOM occurs. The buffer cache in Linux grows automatically to fill your whole memory. That's why in Linux, there is normally no free RAM and that is a good thing. Windows does the same, but counts all the buffer stuff free, whereas Linux calls it used (this is so common, that someone created a website for this).
If the memory is needed, the least recently used (LRU) buffers are thrown away and its memory is used for other stuff. You can easiy drop this cache completely via drop_caches.

Back to ZFS: The amount of memory you need is heavily dependent on your needs. The ext3/4 familiy of filesystems have no settings to configure was is cached and what not. ZFS on the other hand has and that's why it depends on your needs. You can, e.g. create a dataset for backups that is (hopefully) only written to and does not have to be read (so often), therefore you can completely disable caching on it (in ZFS terminology: setting the primarycache to metadata only), which then does not cache written or read blocks in memory, reducing your memory footprint a lot. Of course repeatedly reading from the device is much slower, but that's the setup of this dataset. For other datasets, e.g. movies, that are also seldomly read, you also don't need a cache.

You can use ZFS with a very restricted guaranteed memory footprint if you want, but you have to find this yourself, because it is very dependend on your storage needs. A few years back, I ran ZFS on a 256MB Raspberry Pi and it worked. It was not fast, but it worked. You should at least use 1 GB of dedicated minimum space to ZFS (via arc parameters) and can use e.g. 2 GB as a upper bound. It will work, maybe not as fast as you want, but it will.
 
  • Like
Reactions: Dunuin and Sasha
It will run with less RAM, but the less RAM you allow ZFS to use the slower it will get. Also stuff like deduplication tables and so on might need alot of RAM.

Rule of thumb without deduplication is:
4GB + 1GB per 1 TB of RAW storage. So 4+16=20 GB for a 16TB disk.

Rule of thumb with deduplication would be:
4GB + 5GB per 1 TB of deduplicated RAW storage. So in that case 4+80GB=84 GB RAM for a 16TB disk.

Thats why I'm not using ZFS deduplication...doesn't got all that spare RAM.
 
  • Like
Reactions: Sasha
I have 1,73G HDD mirrored + 512G SSD mirrored on my system.

So what rule says:
4+1*1.73+4+1*0.512 to avoid a potential bottle neck, right?
Or i should add 4G once for all disk systems: 4+1*1.73+1*0.512?
 
Last edited:
Its 1GB per 1TB of raw storage, so if yout got two 2TB HDDs + two 0.5TB SSDs in mirror that would be: 4 + 2 * 2 + 2 * 0.5 = 9GB ARC size.
So 9GB would be a good starting point. From that you can try to lower the ARC size until you see the hit rates dropping. So let it run for a day and check with arc_summary the ARC utilization (free dnode/metadata cache, hit rates and so on). Then remove 1 GB from the ARC, let it run for a day and check arc_summary if you can see a negative effect compared to the previous run. Do that again until you think the performance drop is too bad.

By default ZFSs ARC will use UP TO 50% of your hosts total RAM if you don't manually limit that.
 
Last edited:
Rule of thumb without deduplication is:
4GB + 1GB per 1 TB of RAW storage. So 4+16=20 GB for a 16TB disk.

Rule of thumb with deduplication would be:
4GB + 5GB per 1 TB of deduplicated RAW storage. So in that case 4+80GB=84 GB RAM for a 16TB disk.
Yes, those numbers float around for years and I hate those numbers. They rely on so many assumptions that may not fit your needs:
- hot and cold storage relation unknown
- how many small and big files in use (varying actual recordsize and volblocksize - this is really HUGE)
- how much data in datasets and volumes
- storage layout (mirroring or raidzX)

So in short: There is no rule of thumb that fits your needs, therefore the 50% rule is often (not always) the better rule of thumb.
To get YOUR perfect answer, start with a reasonable low start value, e.g. 4 GB and monitor your cache hit ratio and optimize it.

RAM is not the only adjusting screw, yet the one all talk about. The performance is hugely dependend on
- storage layout. Don't use raidzX if you want performance. Ever. Period. - RAM will not fix this
- use storage tiering, e.g. metadata (special group) on mirrored SSD
- use SLOG
 
  • Like
Reactions: Neobin and Sasha

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!