Just another ARC question

S

sj4318

New Member
Jul 18, 2024
13
0
1
Sydney
Hi all

I've just bought a nice new server with a good chunk of RAM in it and installed PVE 9 and I was wondering is there any harm in over sizing the ARC?

I know the formula is 4Gb + 1Gb for every TB of RAW storage so for example my server with 20TB of RAW storage the ARC max should be limited to 24Gb. The new server has 384Gb of RAM which the VMs will never come close to using up. So what if for example I made the max ARC limit 128Gb leaving 256Gb of RAM for VMs and system consumption?

I've seen plenty of posts and info on the formula for calculating ARC sizing but I couldn't find anything that indicated why I shouldn't give it an over sized amount aside from OoM scenarios.

Thanks :)
 
It changed in 9/ZFS 2.3. As I understand it’s much faster to release memory now. So I think the new default is 90 or 95% or something.

In prior (recent) versions Proxmox had limited it to 10% or 16 GB by default IIRC.
 
SteveITS said:
It changed in 9/ZFS 2.3. As I understand it’s much faster to release memory now. So I think the new default is 90 or 95% or something.

In prior (recent) versions Proxmox had limited it to 10% or 16 GB by default IIRC.
Click to expand...
I checked the zfs.conf after install but can't remember the max size that was preset. I'll know later this week when I do the second server that is identical.

What prompted me to start asking questions is that I ran up a Windows VM with two disks, the second disk I used for testing throughput within the VM and gave it a smacking with Crystalmark for science. I set the second disk to use write back caching and SSD emulation. In crystalmark I set the test count to 9, test size to 64Gb and let it loose on a 100Gb VD. Not long after I noticed the VM was unresponsive and these lines were logged in syslog.

VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries

So I'm wondering if I should tweak the ARC sizing a little more or tweak the disk throughput. The specs of the physical server are:

Dell PowerEdge R7725
AMD EPYC 9175F 16-Core Processor (x2)
Micron 6400MHz 32Gb (x12)
Dell NVMe ISE PE9010 GEN4 RI M.2 480GB (BOSS-N1)
Dell DC NVMe CD8P E3.S 3.84TB (x5)
Broadcom BCM57412 2x10GbT OCP Ethernet NIC
Broadcom BCM57412 4x10GbT OCP Ethernet NIC
 
You must log in or register to reply here.
Top Bottom