[SOLVED] Proxmox HD space is getting full

[BlinTUZ]

Hi.
Proxmox VE HD spase is getting full.

There have been no problems for three years. The disk has been full for the last 5-6 days and it is impossible to use the console because of this.
Returns an error: error 500 closing file '/var/tmp/pve-reserved-ports.tmp.2150368' failed - no space left on device
I ask you to help solve this problem. There were no such problems before. Due to the fact that I don't have enough knowledge in linux, I can't identify the exact problem to root it out.

df -h


df -aTh


findmnt

 

[Dunuin]

9GB is very small for a root filesystem. I personally wouldn`t install PVE with less than 16GB.

Try to find the files that consume that space and delete them. Good place to start would be /var/tmp, /var/log and /var/lib/vz. After that run a fstrim -av. A journalctl --vacuum-size=50M and apt autoremove could also help.

 

[BlinTUZ]

9GB is very small for a root filesystem. I personally wouldn`t install PVE with less than 16GB.

Try to find the files that consume that space and delete them. Good place to start would be /var/tmp, /var/log and /var/lib/vz. After that run a fstrim -av. A journalctl --vacuum-size=50M and apt autoremove could also help.

Thank you . This command journalctl --vacuum-size=50M helped a little.
There are files in /var/log/ .

Please tell me if I can delete them and if so, how:
just delete them or is there a special command for smooth removal without harmful consequences something like cat /dev/null > filename ?

 

[Dunuin]

First you should analyze them to see what errors/warnings are spamming your logs and then try to fix those problems. If you don't fix the source of the problem your root filesystem will fill up again and again.

 

[BlinTUZ]

First you should analyze them to see what errors/warnings are spamming your logs and then try to fix those problems. If you don't fix the source of the problem your root filesystem will fill up again and again.

I have discovered a problem that is causing the log files to get bigger. This is due to the fact that a few days ago I turned on wire guard debugging mode, and after that these three files began to grow rapidly.
modprobe wireguard
echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control
touch /var/log/wireguard-dyndbg.log
I did everything following the instructions listed on the link

The instructions also specify the reverse command, which in theory should disable debugging mode, but it does not work.
echo module wireguard -p > /sys/kernel/debug/dynamic_debug/control

Since I am not well versed in Linux, I would like to ask you to tell me how to stop this debugging mode. Maybe I'm doing something wrong.

 

[Kingneutron]

Try rebooting, /sys is a dynamic virtual filesystem and should refresh at boot.

You should also gzip or rm /var/log/wireguard-dyndbg.log if you don't need it anymore.

EDIT: If you don't have sufficient space for log files on your filesystem, you can soft-symlink the log file to point somewhere else, such as a ZFS compressed dataset, filesystem on another partition or even a shared drive

 

[BlinTUZ]

Try rebooting, /sys is a dynamic virtual filesystem and should refresh at boot.

You should also gzip or rm /var/log/wireguard-dyndbg.log if you don't need it anymore.

EDIT: If you don't have sufficient space for log files on your filesystem, you can soft-symlink the log file to point somewhere else, such as a ZFS compressed dataset, filesystem on another partition or even a shared drive

Thanks for reply.

Rebooting is a bit problematic for me, since at the moment I have about 20 virtual machines actively working in production. Is there an option to do this without restarting the entire system, and restart only the appropriate service?

 

[Kingneutron]

Not that I know of. I would recommend scheduling a downtime window for Friday night or the weekend so you can reboot it (and maybe resize the root FS up a bit)

 

[BlinTUZ]

Not that I know of. I would recommend scheduling a downtime window for Friday night or the weekend so you can reboot it (and maybe resize the root FS up a bit)

Clear. Thanks to you and @Dunuin for your help