zram - why bother?

[BloodyIron]

On PVE Nodes with plenty of RAM (not even close to running out), why even bother with zram?

I've inherited an environment with zram present on some of the PVE Nodes in the cluster, and it seems completely redundant. RAM used to... provide RAM when you're out of RAM? What?

So far all the justifications I see are for memory starved scenarios, like laptops with 8GB of RAM or something. But there's no RAM resource contention at all that I can see. And the swap-function usage of the zram in this environment is barely anything.

It feels like adding unwarranted fragility to a base distro (Proxmox VE OS) that already addresses things like this with KSM and... swap as a file on disk (or partition?).

Am I somehow missing something here???

 

[LnxBil]

Compression ... that is the main goal and you'll always swap. Not so much with KVM, yet with containers. I don't know why, but there is always swapping going on in containers.

 

[BloodyIron]

I wasn't talking about running without swap, where did you get that impression? To me using RAM to replace RAM when you're in an OOM statement is a liability for operations. I'd rather use a local SSD for swap when absolutely necessary (last resort) and day to day just install more RAM or address capacity in similar regards.

What happens if, using ZRAM, I'm 100% out of RAM? Because to me that sounds like a scenario where everything dies horribly.

And also, doesn't KSM do compression already out of the box?

I'm still not seeing it as warranted.

 

[UdoB]

And also, doesn't KSM do compression already out of the box?

No. "Kernel Samepage Merging" doesn't do compression but it looks "only" for identical pages and cross-references them. Only one page will reside in Ram until one of the not-more-existing copies is going to get modified. https://docs.kernel.org/admin-guide/mm/ksm.html does not mention compression at all.

ZRAM does compression. If it could compress data down to 66% you gain 33% of memory. Of course this requires some CPU cycles. See https://docs.kernel.org/admin-guide/blockdev/zram.html

 

[BloodyIron]

No. "Kernel Samepage Merging" doesn't do compression but it looks "only" for identical pages and cross-references them. Only one page will reside in Ram until one of the not-more-existing copies is going to get modified. https://docs.kernel.org/admin-guide/mm/ksm.html does not mention compression at all.

ZRAM does compression. If it could compress data down to 66% you gain 33% of memory. Of course this requires some CPU cycles. See https://docs.kernel.org/admin-guide/blockdev/zram.html

Welp guess that was a misunderstanding on my part, thanks for clarifying.

Okay so apart from trying to get ahead of OOM scenarios... is there _any_ other reason to use zram instead of just swap on disk?

 

[UdoB]

is there _any_ other reason to use zram instead of just swap on disk?

Disclaimer: I am not in a position to give a good or ultimative answer.

For a Node Ram should not get over-committed (there should be plenty of "free" Ram used for caches, buffers, ARC...). While CPUs can, Disk space can, Bandwidth can - Ram should not. In my personal point of view. When the host starts swapping a lot of weird things can/will happen. This (my) statement would recommend to swap inside a guest. Generally I do not do this.

That said... and while I try hard to not over-commit Ram I use both methods. ZRam is easier to activate because you do not need to modify the partition table - but then there are Swap-files. On slow storage ZRam is a lot faster as it handles "just" Ram without the need to traverse the complex storage management APIs, possibly over the network...

In my dayjob I have enough Ram to avoid tinkering. In my Homelab (with Mini-PCs) Ram is always the limiting resource :-(

Ymmv - just try it!

 

[LnxBil]

For a Node Ram should not get over-committed (there should be plenty of "free" Ram used for caches, buffers, ARC...). While CPUs can, Disk space can, Bandwidth can - Ram should not. In my personal point of view. When the host starts swapping a lot of weird things can/will happen. This (my) statement would recommend to swap inside a guest. Generally I do not do this.

Totally agree!

is there _any_ other reason to use zram instead of just swap on disk?

Depsite the obvious: have compressed memory and reduce disk I/O, maybe easier disk size enhancement if the swap is in the way (default in debian is on first extended partition for BIOS) ... not that I can think of.
In my PVE hosts, I use it alongside regular swap, but with a higher priority, so that I hit compressed memory first and then go to disk if you need more. Yet as @UdoB already stated, the hypervisor should not swap at all - at least for KVM-only usage.

 

[BloodyIron]

Totally agree!


Depsite the obvious: have compressed memory and reduce disk I/O, maybe easier disk size enhancement if the swap is in the way (default in debian is on first extended partition for BIOS) ... not that I can think of.
In my PVE hosts, I use it alongside regular swap, but with a higher priority, so that I hit compressed memory first and then go to disk if you need more. Yet as @UdoB already stated, the hypervisor should not swap at all - at least for KVM-only usage.

Well to clarify the assumption to me is that swap should generally be empty day to day, and it should be available as "only last resort when things really start getting angry". So to me I would rather not use any RAM at all that is managed by a service (zram the service) and instead plan capacity so there's day-to-day always enough RAM to expand into, and swap is served by a file-on-disk (on the PVE Node) if you're asleep and things start getting out of control despite good planning.

In scenarios like that disk IO matters less than "things keep running". And to me running zram is one more thing I could forget to do when adding/replacing a PVE Node. I am a fan of keeping PVE Nodes as uncustomised as possible from an under-the-hood regard so that PVE Node replacement is as streamlined as possible.

I appreciate the insights here but I still am not seeing the value being worthwhile, more placebo, in my experience and opinion anyways.

 

[leesteken]

Well to clarify the assumption to me is that swap should generally be empty day to day, and it should be available as "only last resort when things really start getting angry".

Some software does allocate memory that happens not to be used (in your specific system or workload), since no software is perfect. If that memory that is (mostly) never touched gets swapped out, that a good thing since it make room for your VMs and cache and stuff that does matter.

Swap that is mostly only written to is not a problem (but a work-around to over-allocation by software). Swap that is often read (and rewritten and read again) is when it starts causing more problems than it solves (or works around) and is then called thrashing.

People get nervous about swap getting full because of the red color, but slowly increasing swap usage (in days or weeks, without high IO!) is in my opinion better than having less and less memory available over time because of sub-optimal software. Just reboot on a kernel update to restart the cycle.

EDIT: Maybe zram can help but I prefer to have that memory available to VMs instead and have swap use disk space. To me, keeping swap (compressed) in memory is preparing for thrashing, which should be avoided by swapping the unused stuff out instead.

 

[BloodyIron]

Some software does allocate memory that happens not to be used (in your specific system or workload), since no software is perfect. If that memory that is (mostly) never touched gets swapped out, that a good thing since it make room for your VMs and cache and stuff that does matter.

Swap that is mostly only written to is not a problem (but a work-around to over-allocation by software). Swap that is often read (and rewritten and read again) is when it starts causing more problems than it solves (or works around) and is then called thrashing.

People get nervous about swap getting full because of the red color, but slowly increasing swap usage (in days or weeks, without high IO!) is in my opinion better than having less and less memory available over time because of sub-optimal software. Just reboot on a kernel update to restart the cycle.

EDIT: Maybe zram can help but I prefer to have that memory available to VMs instead and have swap use disk space. To me, keeping swap (compressed) in memory is preparing for thrashing, which should be avoided by swapping the unused stuff out instead.

I've exhaustively worked through proving in multiple environments that leaving data in swap tangibly reduces performance of whatever system is doing it, as well as impacting other systems in the same environment that may or may not have anything in its own swap. Whether it's Windows, Linux, or hypervisors such as Proxmox VE.

I am aware that plenty of software likes to swap anyways as a matter of day to day operations, however since being able to actually observe the performance impact of allowing swap to be wild-west as it normally is, vs flushing swap with some regularity, I have no plans to ever go back to not flushing swap with some regularity (as in cron job stuff). The performance gains and responsiveness gains were so substantial it was night and day.

The premise that low amounts of swap has no negative impact is false, and I know a lot of people that are very smart (legitimately so) make the claim it has no impact. I spent exhaustive testing over multiple years to explore this topic.

Yes, I agree that certain percentages of RAM should be made available for elastic growth and other rationale. But I am completely convinced that swap should generally never be populated with data day to day, and that it does have performance impacts. Especially in a PVE Cluster. Even if you have many VMs with a little bit of swap, the swap of one VM impacts the performance of another VM, and this becomes a compounding problem.

So at this point, for me and the environments I architect for, and am otherwise responsible for, swap is a last resort, and I don't plan to ever touch zram.