Higher memory usage 7.0.14-19-pve

[lucdg]

Good: 7.0.14-15-pve. Bad: 7.0.14-19-pve

I've noticed after updating from 7.0.14-15 to 7.0.14-19 higher memory usage than usual. It gradually grew over a few days before I noticed it becoming critical, nearing 100%. Shutting the VMs down returned their memory but did not return the extra memory, approx. 16Gb at the time. The unaccounted for memory appeared to grow at approx. 4-5gb/day. I have updated to 7.0.14-20 and rebooted the server about 12 hours ago, so far about 2.5Gb of memory is unaccounted for so it does not appear to have help. The graph below shows my memory usage under 7.0.14-15 up to upgrading and rebooting into 7.0.14-19 and then gradually increase in memory usage after.

I asked an AI for help and feed it some logs, this is what is spat back out at me.

• Growth comes in bursts correlated with activity
• The leak occurs in bursts tied to I/O on raw SATA HDDs passed through to a VM (cache=none, aio=io_uring).
• About 16 GiB of memory missing from /proc/meminfo and from all cgroups, growing about 4–5 GiB/day.
• Not released when any VM is stopped or restarted. It's orphaned until a host reboot.

 

[chrcoluk]

You can try aio=native? Something seems to have changed on io_uring lately, curious if it will help.

 

[lucdg]

Thanks for the suggestion. I've just updated to aio=native and shutdown/rebooted the vm. I'll post back in a few hours how the memory usage is going.

 

[lucdg]

That doesn't appear to have helped. The leak seems to have continued at the same pace since stopping and starting the vm.

 

[fabian]

please post /proc/meminfo for both kernels.

 

[lucdg]

Thanks for the response, Fabian. I have attached a /proc/meminfo from 7.0.14-19 yesterday and one from now on 7.0.14-20. I won't be in a position to revert to 7.0.14-15 and reboot the server for a few hours. Once I have, I will upload that meminfo.

 

[lucdg]

This might be incorrect information from an AI so sorry if it is, but this is what freed up the memory while running 7.0.14-20

• The unaccounted memory is the dm-bufio cache (/sys/module/dm_bufio/parameters/current_allocated_bytes): 6.0 GiB against max_cache_size_bytes 1.9 GiB, and it grows at ∼2% of bytes written to an LVM-thin pool.
• echo 2 > /proc/sys/vm/drop_caches frees it instantly (6.0 GiB → 2.4 MiB). So the buffers are reclaimable, but the automatic trim to max_cache_size_bytes isn't running.
• The metadata pool is 64216/4161600 blocks used. The cache grows toward the full tmeta size (15.9 GiB), consistent with ∼16 GiB after 3.5 days on -19.
• Good: 7.0.14-15. Bad: 7.0.14-19 and 7.0.14-20. Likely candidates are dm-bufio/dm-thin changes in the upstream stable updates pulled in by -17/-18.