log2ram getting full frequently and proxmox becomes unusable

[rkbest]

Hi all,
I have a proxmox 7 running from a USB drive 128 GB(fast) and VMs on a NVME drive. I have truenas running as VM that uses 3 spinning disk as zfs pool for NAS. As the OS is on USB, i installed log2ram to reduce the frequent read/writes. But, often, my VM's start throwing error that the /var/log is full or something similar. Is there a way to fix this or make it automatically clear the /var/log folder . the log2ram uses a 40MB that i tried to increase to larger value but i want to make this automatically handled to avoid future issues. I dont need last logs and want to limit the file size for syslog or similar file to limit this from swelling too much.

Sample of the issues is

root@vikings:~# df -h
df: /mnt/pve/vikingdata: Host is down
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 235M 2.1G 10% /run
/dev/mapper/pve-root 30G 3.0G 27G 11% /
tmpfs 12G 37M 12G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdd1 229G 26G 191G 12% /mnt/usb-drive250
/dev/sde2 511M 336K 511M 1% /boot/efi
VMD 297G 128K 297G 1% /VMD
VMD/subvol-105-disk-0 32G 3.1G 29G 10% /VMD/subvol-105-disk-0
log2ram 40M 40M 0 100% /var/log
/dev/fuse 128M 20K 128M 1% /etc/pve
VMD/subvol-103-disk-0 32G 1.3G 31G 4% /VMD/subvol-103-disk-0
/dev/sdg 3.6T 56K 3.4T 1% /mnt/USB-4TB
tmpfs 2.4G 0 2.4G 0% /run/user/0

Please help.

 

[rkbest]

Also, is it safe to delete the log x.gz as i see many archived logs.
like
-rw-r----- 1 root adm 55 Oct 26 00:00 pve-firewall.log
-rw-r----- 1 root adm 179 Oct 26 00:00 pve-firewall.log.1
-rw-r----- 1 root adm 121 Oct 25 00:00 pve-firewall.log.2.gz
-rw-r----- 1 root adm 129 Oct 24 00:00 pve-firewall.log.3.gz
-rw-r----- 1 root adm 163 Oct 23 20:07 pve-firewall.log.4.gz
-rw-r----- 1 root adm 138 Oct 17 00:00 pve-firewall.log.5.gz
-rw-r----- 1 root adm 137 Oct 16 00:00 pve-firewall.log.6.gz
-rw-r----- 1 root adm 118 Oct 15 00:00 pve-firewall.log.7.gz

 

[UdoB]

Also, is it safe to delete the log x.gz as i see many archived logs.

Yes. The were created by logrotate. See /etc/logrotate.d/pve-firewall for the controlling rules.

If you have another server running 24*7 you may log to a remote syslog. This way your local USB-Storage is not used and you get persistence nevertheless.


Best regards

 

[LnxBil]

If you have another server running 24*7 you may log to a remote syslog.

That's what I also would suggest.

Yet be advised, logs are not the only write-intensive task, you also have the pve-cluster filesystem itself, that is internally a sqlite database.

 

[Docop2]

The log2ram look interesting. But actually, is there a way to simply setup a limit on the log amount ? Like do not store anything over 3 days.
As i don,t need an external log server and don't need much log either. But try to limit writing to the unit.

 

[UdoB]

Like do not store anything over 3 days.

I'd told the name of the configuration file already. Just change the "7" to "3" in this file:

~# cat /etc/logrotate.d/pve-firewall
/var/log/pve-firewall.log {
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    sharedscripts
    create 640 root adm
    postrotate
        invoke-rc.d pvefw-logger restart 2>/dev/null >/dev/null || true
    endscript
}

Of course this will affect pve-firewall.log only. There are several similar files...


Best regards

 

[Docop2]

Oh thanks a lot, that's what i was looking for. appreciated.