How much RAM for ZFS ARC + VMs on Proxmox?

speedbird

Member
Nov 3, 2017
45
5
13
Hello,
I'm currently close to building my new Proxmox rig and since ZFS is a first time for me, I was wondering about the need for RAM in my setup.

I will run a 4x480GB SSD setup in RAID10 with ZFS. System memory is 64GB ECC RAM.

Now, I will run 3 VMs for a start, needing 24GB, 8GB and 8GB of RAM respectively.

Proxmox wiki says, it's a good idea to restrict RAM usage of ARC to prevent the host from being exhausted. The page says:

options zfs zfs_arc_max=8589934592

to limit usage to 8GB of System RAM.

Question now is, how much does ZFS actually need to not run into OOM situations, begin swapping or something the like.

I really don't need any unexpected behavior on this machine. How will taking snapshots / backups etc be affected by available ARC memory? Will ZFS purge "old" entries to make room for new ones or will the pool simply fail at some point because there's not enough system memory?

Just to clarify, I do not plan on using Dedup! Only compression is going to be active.

Thanks guys!
 

guletz

Famous Member
Apr 19, 2017
1,523
238
83
Brasov, Romania
Hi,

If you reserv 8 Gb Ram for zfs I guess it will be no problem. I do not see any OOM situation because of zfs even with only 1 Gb Ram for zfs. Also you can use arc min like 4 Gb and arc max like 8Gb. And for zvol is better to cache only metadata if you do not want to waste your ram (at zvol level and at the guest VM level).
 

speedbird

Member
Nov 3, 2017
45
5
13
Hm what do you mean with Zvol caching? Haven't made any kind of setting changes towards that...

However I now tried with 16GB for ARC but when I run Atto Benchmark on a windows 7 vm with 64K size, after the bench is complete, I can see the RAM fill up in no time and the whole system crashes. Only a hard reset can bring it back then. I don't know why that happed. On my test system there's 36GB of RAM, 8GB being fed to windows and 16GB max to ARC. still it crashed the whole system. Strange. Any ideas?
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
6,496
480
103
Hi,

Do you have a separat zfs pool or do you use the rpool?
 

speedbird

Member
Nov 3, 2017
45
5
13
Do you have a separat zfs pool or do you use the rpool?
Hi Wolfgang,
well, I use the rpool that is created with the installation. Since I can only have 4 drives in the server and want to have RAID10, this seems to be the only option, isn't it?

Thanks for your input.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!