Insanely high writes on nvme

[sistemieva]

Hi all,
I'm using a Proxmox Backup Server instance as a backup pool of a PVE cluster.
Despite the PBS pool is mounted on a RAID5 4x16TB HDD, and the 256 nvme has only OS on it, it seems the nvme keeps doing a very very huge traffic (645TB over less than 1year period):



the command iostat 1 suggests a real-time usage of about 20MB/s of constant write speed used by proxmox-bakcup-proxy and systemd-journald processes.

The Proxmox Backup Server verison is 2.4.6.

Do someone have similar experience and could provide help in troubleshooting this? At the moment, we're buying another nvme unit but if we don't solve the problem this unit will die in another year.

Thanks in advance.

 

[Dunuin]

Probably the logs. You got millions of chunk files and when reading/writing those it will cause multiple log lines per chunk. Unluckily there is no built-in verbosity setting to only log warnings/error (but they are working on it).

You could try some stuff like moving journald from persistent disk to volatile RAM. There is an option i journald.conf for this.

 

[sistemieva]

Hi Dunuin and thanks for your reply. I tried your suggestion, setting in the journald.conf file the Storage=volatile variable, restarted the systemd-journald service and now the logs are written on RAM in /run/log/journal.
Unfortunately, the problem seems not resloved at all: despite the change, the command iostat 1 returns a high kB_wrtn/s speed that keeps to persist:



df -h not showing relevant infos about eventually wrong mount points on the nvme0n1 disk:



I'm trying to do a lot of research but I really don't know where other look into. Any other suggestion would be really appreciated!

Thanks alot

 

[Dunuin]

Maybe installing iotop could help to see whats going on?: https://github.com/Tomas-M/iotop

 

[sistemieva]

I already tried the iotop way: tons of proxmox-backup-proxy lines, high current disk write, but no clue on what (or at least, no clue to me):



What is strange is the really weird amount of disk usage: we're talking of 661TB of data written on a 256GB SSD in ~300 days of usage... Tomorrow I will switch the SSD with a new, bigger one (in order to set a nice overprovisioning), but I would avoid to break the new disk in another year

 

[sistemieva]

Maybe I solved the issue, based on this thread.
I stopped proxmox-backup-proxy service, then moved the entire /var/log/proxmox-backup folder to an HDD partition, then soft-linked the new folder to the original path:

systemctl stop proxmox-backup-proxy
mv /var/log/proxmox-backup/ /my/folder/on/hard/drive/
ln -s /my/folder/on/hard/drive /var/log/proxmox-backup
systemctl start proxmox-backup-proxy

Now the write speed seems significantly decreased.
Based on the thread, in the next future there should be the possibility to set log to OFF or at least to exclude infos, hope the patch arrives soon.

Thanks again for your help!