Where is stuff from container's /run directory actually stored?

[r7s]

I'm in the process of changing journald config to solve a problem of journald eating through SWAP.

I know that on some of my containers, journald stores logs in memory. After I cleaned logs on one of the containers, SWAP usage went down from 100% to basically 0%.

Now I have another container and this is what's reported from inside the container:

root@container:~# journalctl --disk-usage
Archived and active journals take up 2.4G on disk.
root@container:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.0G        221M         44M        714M        758M        802M
Swap:          3.0G        1.8G        1.2G
root@container:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2      7.9G  3.3G  4.2G  44% /
none            492K     0  492K   0% /dev
tmpfs            32G  4.0K   32G   1% /dev/shm
tmpfs            32G  2.5G   29G   8% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
none             32G     0   32G   0% /run/shm

That's a single 2.4 GB log file stored in /run/log/journal, but that's a lot more than whan free -h reports in terms of memory and swap usage.

Where are the logs actually stored then?

I also don't know how to check if right now those journald are hogging memory or SWAP which could be utilized by other processes in the container.

Any help would be appreciated, along with some pointers toward what to google, because I have very liitle idea when it comes to that.

Spoiler: My pveversion -v

foo@host:~$ pveversion -v
proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve)
pve-manager: 5.4-3 (running version: 5.4-3/0a6eaa62)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-35
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.15.15-1-pve: 4.15.15-6
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-1-pve: 4.10.17-18
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: not correctly installed
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-50
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-41
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-25
pve-cluster: 5.0-36
pve-container: 2.0-37
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-19
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-3
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-50
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[Alwin]

Using swap is not a bad thing [0] but it needs to be managed as well. The journald size might be expected, at least from its manual (~10%), see here for details [0].

And I recommend to upgrade since Proxmox VE 5.4 will be EoL in July [2].

[0] https://chrisdown.name/2018/01/02/in-defence-of-swap.html
[1] https://unix.stackexchange.com/questions/482920/store-more-logs-in-memory-with-systemd-journald
[2] https://pve.proxmox.com/wiki/FAQ

 

[LnxBil]

Where are the logs actually stored then?

It's tmpfs so it is stored in a ram disk. If you have "old data" (data that has not been used for a while), it may be swapped out if you run out of memory. Therefore removing the stuff in a tmpfs mount will result in freeing up memory, swap or both.

 

[r7s]

Thanks LnxBil, that's what I ultimately realized as well.

/run is reported as 32 GB, idk where this figure is taken from as we have ~64 GB of RAM and 32 GB of SWAP on the main node. Even though it's reported as 32 GB, the amount of stuff the container can store there is still limited by memory and SWAP allocated to the container.

It is true that freeing up tmpfs mount will free up memory or SWAP. What I didn't realize is that the amount of memory taken by journald is reported by free not under used, but under the shared column. So the size reported by journalctl --disk-usage actually matches free -h.

 

[LnxBil]

I wasn't aware that journalctl has its own disk usage reporting. SystemD and JournalD is far from unix-like (do one thing only, but good).