Reducing rrdcached writes

emox

Member
Dec 4, 2019
4
2
23
I didn't find any information about this, so I thought I'd post in case other were interested.

Some users would prefer to have reduced disk writes on their proxmox installation.
One source of writes is the very useful rrd data.

One way to continue to obtain the info, but reduce the disk writes can be directly configured by RRDcached.

1. Increase the write and flush timers

Increasing the write timer is easy
in /etc/defaults/rrdcached set the following:
# Optional override write interval (-w arg) , in seconds.
WRITE_TIMEOUT=3600


The flush timer is a bit trickier as there is no predefined variable for it, so I added this
FLUSH_TIMEOUT=7200
and added a line to /etc/init.d/rrdcached

RRDCACHED_OPTIONS="\
${BASE_OPTIONS} \
${NETWORK_OPTIONS} \
${WRITE_TIMEOUT:+-w ${WRITE_TIMEOUT}} \
${FLUSH_TIMEOUT:+-f ${FLUSH_TIMEOUT}} \
${WRITE_JITTER:+-z ${WRITE_JITTER}} \
${WRITE_THREADS:+-t ${WRITE_THREADS}} \
${BASE_PATH:+-b ${BASE_PATH}} \
${JOURNAL_PATH:+-j ${JOURNAL_PATH}} \
${DAEMON_GROUP:+-G ${DAEMON_GROUP}} \
${DAEMON_USER:+-U ${DAEMON_USER}} \
-p ${PIDFILE} \
${SOCKFILE:+${SOCKGROUP:+-s ${SOCKGROUP}} ${SOCKMODE:+-m ${SOCKMODE}} -l unix:${SOCKFILE}} \
"

2. Disable journaling
All the above is not so useful if rrdcached is still writing journal updates.
Note - removing journaling means the chance of recovering the data in case of a crash is lower. Best to read about what journaling does if you care about the data

in /etc/defaults/rrdcached set the following:
# Where journal files are placed. If left unset, journaling will
# be disabled.
#JOURNAL_PATH=/var/lib/rrdcached/journal/

Restart the rrdcached service:

systemctl restart rrdcached.service

And hopefully the writes will be reduced for you!

Note - if you have the proxmox webgui page open and you are viewing the rrd stats, i suspect somehow the system instructs the rrdcached to flush, so I would close the webpage if you are interested in viewing the effects with something like iotop.

Hope this helps - and if there are other adverse effects to this, please do let me know.
 
  • Like
Reactions: panchuz
As an aside, I was also successful in reducing the writes by using a persistent ramdisk. But since then I discovered that rrdcached has these built in settings so I figured those would be best to try first.
 
I have a similar issue, the writes appear every 2 seconds but very small. Are they necessary to be written so quickly?

I have a new proxmox setup and two new Samsung PRO drives running zfs mirror pool for the operating system. (no vms on these drives)
I've been online for about 2 weeks

my drives already show a 1% wear

zpool iostat 2 shows 1mb written every few seconds. Looks like its pmxcfs is writing but I am not certain.

Thanks for any suggestions?

PS. I tried the above cache changes and no change..

I also installed proxmox on a test machine and reproduced the writes. So this is confirms the effect the question now is it normal and can it be reduced. Thanks

zpool_iostat_2.png
 
Last edited:
Same problem , is there someone from proxmox which could explain whats is write every 2 seconds ? Thanks !
 
As an aside, I was also successful in reducing the writes by using a persistent ramdisk. But since then I discovered that rrdcached has these built in settings so I figured those would be best to try first.
Is it a simple configuration item to set the rrdcached writes to ram disk rather than disk? This approach could leave the WRITE_INTERVAL and FLUSH_INTERVAL as standard but write to ram instead of wearing out SSDs? I don’t fully understand why Proxmox writes so much to disk if it could easily use RAM disk?
 
  • Like
Reactions: TheRealMaN_
I didn't find any information about this, so I thought I'd post in case other were interested.

Some users would prefer to have reduced disk writes on their proxmox installation.
One source of writes is the very useful rrd data.

One way to continue to obtain the info, but reduce the disk writes can be directly configured by RRDcached.

1. Increase the write and flush timers

Increasing the write timer is easy
in /etc/defaults/rrdcached set the following:
# Optional override write interval (-w arg) , in seconds.
WRITE_TIMEOUT=3600


The flush timer is a bit trickier as there is no predefined variable for it, so I added this
FLUSH_TIMEOUT=7200
and added a line to /etc/init.d/rrdcached

RRDCACHED_OPTIONS="\
${BASE_OPTIONS} \
${NETWORK_OPTIONS} \
${WRITE_TIMEOUT:+-w ${WRITE_TIMEOUT}} \
${FLUSH_TIMEOUT:+-f ${FLUSH_TIMEOUT}} \
${WRITE_JITTER:+-z ${WRITE_JITTER}} \
${WRITE_THREADS:+-t ${WRITE_THREADS}} \
${BASE_PATH:+-b ${BASE_PATH}} \
${JOURNAL_PATH:+-j ${JOURNAL_PATH}} \
${DAEMON_GROUP:+-G ${DAEMON_GROUP}} \
${DAEMON_USER:+-U ${DAEMON_USER}} \
-p ${PIDFILE} \
${SOCKFILE:+${SOCKGROUP:+-s ${SOCKGROUP}} ${SOCKMODE:+-m ${SOCKMODE}} -l unix:${SOCKFILE}} \
"

2. Disable journaling
All the above is not so useful if rrdcached is still writing journal updates.
Note - removing journaling means the chance of recovering the data in case of a crash is lower. Best to read about what journaling does if you care about the data

in /etc/defaults/rrdcached set the following:
# Where journal files are placed. If left unset, journaling will
# be disabled.
#JOURNAL_PATH=/var/lib/rrdcached/journal/

Restart the rrdcached service:

systemctl restart rrdcached.service

And hopefully the writes will be reduced for you!

Note - if you have the proxmox webgui page open and you are viewing the rrd stats, i suspect somehow the system instructs the rrdcached to flush, so I would close the webpage if you are interested in viewing the effects with something like iotop.

Hope this helps - and if there are other adverse effects to this, please do let me know.
/etc/defaults/rrdcached does not currently exist and Google shows only this document. Is this a typo? Can you please provide the correct filename?
 
  • Like
Reactions: TheRealMaN_
Try /etc/default/rddcached without the s.
Nope; doesn't exist either. However, you put me on the right track. The correct path is
/etc/default/rrdcached
Thanks!

EDIT: replaced 'rdd' with 'rrd' (typo)
 
Last edited:
  • Like
Reactions: TheRealMaN_

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!