[SOLVED] ZFS Backup and Restore massively fill RAM and eventually SWAP with ZST compression

[PT400C]

Hello,

recently I noticed a problem with ZFS and creating compressed backups (ZST compression) on the latest Proxmox VE 6.3-6. Every time I create a backup, my entire RAM + some SWAP (64 GB DDR4 + 32 GB SWAP) quickly fill up. This RAM won't (!) free up after the process has finished and will still be listed in htop as used. Eventhough I can't find any associated process.

---

Running this simple script, the RAM is released again:

Spoiler: Script:

sync && echo 3 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a

On the other hand, restoring a backup seems to not clutter the RAM directly but fills the RAM with cache contents (yellow in htop). I can't explain why this happens but it's less of a problem than the one when creating a backup.

For sure I could always clear the RAM caches and SWAP with the script above. But it doesn't seem to be safe to me in a production env and it isn't a proper solution.

Am I doing something wrong or is this an actual problem in the latest Proxmox version?

Thanks in advance

PT400C

EDIT: It happens on LXC as well as on actual QEMU VMs

 

[PT400C]

Update: Out of curiosity I now executed the backup command from the root user CLI instead of the webinterface. Command:

vzdump VMID --node NODE --storage local --mode stop --compress zstd

Exactly the same VM, same storage, same node. Everything identical. But this time no memory-leak occurs. Very interesting.

EDIT: Couldn't reliably reproduce that behaviour above. Sometimes backup tasks issued via the CLI still fill up the RAM consequently.

 

[PT400C]

After many restarts and tests the issue sadly pesists on my end.

 

[PT400C]

After many restarts and tests the issue sadly pesists on my end.

And I just checked if the same problem is still prominent with other compression algorithms - it seems to be! With the same ZFS storage, compressing with tar.gz also cause the ram to clutter up.

 

[leesteken]

And I just checked if the same problem is still prominent with other compression algorithms - it seems to be! With the same ZFS storage, compressing with tar.gz also cause the ram to clutter up.

I've seen this happening when the virtual disk cache was set to Writeback, but not with the cache set to Default or No cache. Does changing the cache setting help you?

 

[PT400C]

I've seen this happening when the virtual disk cache was set to Writeback, but not with the cache set to Default or No cache. Does changing the cache setting help you?

I'll test that on an actual VM (qemu) asap! But on the containers I don't have any control over the caching done it seems. And those fill up the ram 100% of the time I try.

EDIT: I just tried and after a few seconds I could already tell that the memory starts to fill up even with any caches disabled on the VMs settings.

Below is a graph to show what happens with the RAM. Btw, in the end it flattens due to SWAP taking over to not crash the Host server.

 

[PT400C]

https://bugzilla.proxmox.com/show_bug.cgi?id=3377