ZFS vs RAM?

[lemel]

I have a 64GB RAM system with 16TB in a RAID HHD storage, so with 8TB used. The OS and VM/CTs are all in a 2TB SSD drive.

Proxmox is set-up to use ZFS and I've noticed that my system has been peaking the memory usage, even though individually the services don't even use 25%. I've also noticed certain services are a bit slow and lagging during the overall peaking.

Guess my question is if ZFS was a mistake for my system, is it just requiring too much memory and not allowing enough any for the services? As an example, Nextcloud is allowed 16GB RAM, but it slows to a crawl using only 2GB, but Proxmox is at 95%+ overall usage.

If I do switch course and transition away from ZFS is there another well recommended filesystem? And since I'm sure it's not a easy switch, any recommended guides on converting the filesystem?

Or is there still a bigger benefit from keeping ZFS?

 

[j0n1th4n]

ZFS is good but i guess you haven't set a limit for the ZFS cache.

cat /sys/module/zfs/parameters/zfs_arc_max

This will show you the limit for the zfs cache.

echo "$[8 * 1024*1024*1024]" >/sys/module/zfs/parameters/zfs_arc_max

This will set the limit to 8GB.
You should try this and then check the memory usage again.

More Info here: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

 

[Max Carrara]

As @j0n1th4n already mentioned, you can limit how much RAM ZFS uses. Per default ZFS likes to use at most 50% of the system's available RAM, but usually it doesn't "claim" it completely - if another process needs more RAM, ZFS should be able to hand it over and shrink its ARC, AFAIR (someone correct me if I'm wrong). Seems like that's not happening in your case, though.

Or is there still a bigger benefit from keeping ZFS?

Some of its benefits are that it supports filesystem-wide compression and error correction.

Let me tell you a little story I've already shared here before: I once upgraded my homelab's RAM without running a memtest beforehand. After some time I began to discover a series of kernel panics caused by the ZFS kernel module, which I first took as a random bug. I reported this to the ZFS guys on GitHub - they then suggested that my RAM might be faulty, because they had never seen such errors before. I checked zfs status again and saw that there were more than 5000 READ and WRITE errors.

Realizing that my new RAM was acting up, I shut off my server and took it out again. After a reboot, I ran a zfs scrub. Thankfully, not enough data was corrupted, and ZFS was able to repair everything on my 4x 8TB pool. No data was lost - my server's still happily chugging along to this day.

So yeah, without ZFS a good chunk of my archive / Nextcloud would've probably been corrupted. Since that was a rather new setup, I had no backups in place either. It pretty much saved some of my most essential data.

Been counting on it ever since and haven't looked back. Make of that story what you will!

Regarding compression: Compression should IMO always be enabled. A user on Reddit did a bit of testing to see what difference it makes, check out the post if you'd like to see some data. In short, ZFS compression will actually speed up your effective write speeds, because there's less to write once something has been compressed. So you won't just get more efficient storage usage, it's also going to run faster, because your CPU is being kept busy while writing to the filesystem (to put it simply). I personally am a big fan of just setting compression=zstd on all my datasets

I hope I could clear some things up!

 

[lemel]

ZFS is good but i guess you haven't set a limit for the ZFS cache.

cat /sys/module/zfs/parameters/zfs_arc_max

This will show you the limit for the zfs cache.

echo "$[8 * 1024*1024*1024]" >/sys/module/zfs/parameters/zfs_arc_max

This will set the limit to 8GB.
You should try this and then check the memory usage again.

More Info here: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

OK, so caching did work, thanks for the tip! I noticed that max memory code resets after a reboot. Any concerns to be aware of if using the other script to still keep the limit post reboots?
Ex: options zfs zfs_arc_max=8589934592

 

[j0n1th4n]

No, that's exactly what you want to use for persistence.