Swap space usage with ZFS

S

Sam T

New Member
Sep 24, 2025
3
0
1
Howdy!

Context: I've got 3 Proxmox 9.0.10 nodes clustered, each using a 256gb boot drive and a 2tb nvme drive with 32gb of ram. Guest storage sits on a single-disk zfs pool on each 2tb drive.

I'm not sure if this is just a 'local man doesn't understand swap' situation, but I've noticed some occasions of persistently high swap usage on my host nodes (50%-90%), which is surprising since ram on my host is mostly unallocated to guests. For example at time of writing, node 01 has 4gb of swap used while 23gb of ram is currently just being used for zfs' ARC. I'm under the impression swap should only be used if a machine is out of memory, while in this situation I'd just expect arc to be getting evicted to make space.

I've noticed several posts talking about swap as an optional thing - I was under the impression it was just 'a thing linux will have'. I guess questions here would be:

Are my assumptions right regarding how swap *should* work and how arc should be working here?

Assuming I have plenty of ram, would it make more sense here just to disable swap, if that's possible? And if so, how would I go about doing that on my hosts?
 
Sam T said:
Howdy!

Context: I've got 3 Proxmox 9.0.10 nodes clustered, each using a 256gb boot drive and a 2tb nvme drive with 32gb of ram. Guest storage sits on a single-disk zfs pool on each 2tb drive.

I'm not sure if this is just a 'local man doesn't understand swap' situation, but I've noticed some occasions of persistently high swap usage on my host nodes (50%-90%), which is surprising since ram on my host is mostly unallocated to guests. For example at time of writing, node 01 has 4gb of swap used while 23gb of ram is currently just being used for zfs' ARC. I'm under the impression swap should only be used if a machine is out of memory, while in this situation I'd just expect arc to be getting evicted to make space.

I've noticed several posts talking about swap as an optional thing - I was under the impression it was just 'a thing linux will have'. I guess questions here would be:

Are my assumptions right regarding how swap *should* work and how arc should be working here?

Assuming I have plenty of ram, would it make more sense here just to disable swap, if that's possible? And if so, how would I go about doing that on my hosts?
Click to expand...

In my humble opinion, what you’re seeing is just normal Linux behavior: it may swap idle pages even with free RAM to keep ARC/cache. Personally I’d suggest trying zram (with zram-tools or systemd-zram-generator) instead of disabling swap — it gives you compressed RAM-based swap, avoids NVMe wear, and still provides a safety net. A modest setup (≈ half RAM, zstd, low swappiness, ARC cap) has worked well for me..
 
  • Like
Reactions: UdoB and Johannes S
martinkubat said:
In my humble opinion, what you’re seeing is just normal Linux behavior: it may swap idle pages even with free RAM to keep ARC/cache. Personally I’d suggest trying zram (with zram-tools or systemd-zram-generator) instead of disabling swap — it gives you compressed RAM-based swap, avoids NVMe wear, and still provides a safety net. A modest setup (≈ half RAM, zstd, low swappiness, ARC cap) has worked well for me..
Click to expand...
That's fair, just... confusing. I'm not sure why linux would be prioritizing arc over keeping stuff in ram instead of swap. zram does look interesting though! I think I'd just rather keep stuff out of swap if possible however, I just don't think it's really needed with the amount of spare ram we're talking about.
 
Sam T said:
That's fair, just... confusing. I'm not sure why linux would be prioritizing arc over keeping stuff in ram instead of swap. zram does look interesting though! I think I'd just rather keep stuff out of swap if possible however, I just don't think it's really needed with the amount of spare ram we're talking about.
Click to expand...

Imho 32 GB of RAM is really quite low for Proxmox + ZFS, but of course I don’t know what exactly you’re running there and how many VMs/CTs you have per node. In any case, good luck resolving the issue!
 
martinkubat said:
Imho 32 GB of RAM is really quite low for Proxmox + ZFS, but of course I don’t know what exactly you’re running there and how many VMs/CTs you have per node. In any case, good luck resolving the issue!
Click to expand...
It works well in my case cuz it's 32GB *per node*, so 96 in all which works out pretty fine. Homelab, so nothing crazy in terms of what I'm running. In theory the zfs side shouldn't cause me issues cuz arc should intelligently free itself, in practice... we'll see! For now I've disabled swap altogether, but if needs be I'll re-enable it (possibly moving to zram...) and/or drop my swappiness levels down
 
You must log in or register to reply here.
Top Bottom