[SOLVED] ZFS pool IO when idle

[elterminatore]

hey there,

my current (test) PBS has 20 ssds in one pool with 2 raidz2 vdevs,
when the system is idle and no backup is running, there are permanently much io on the pool.



is this a normal behavior for zfs?
~800 write iops when idle?

regards
stefan

 

[dcsapak]

is that the rpool? in that case it may be, because of rrd/logging/etc.

 

[elterminatore]

is that the rpool? in that case it may be, because of rrd/logging/etc.

i only have one pool as the datastore (tank).
the system is not installed via the pbs installer. it is a standard debian installation with pbs on top.

 

[dcsapak]

do you have the ui open? certain actions/page loads on the ui can trigger some io on the datastores (e.g. the content listing)
do you have some other scheduled operations (gc/verify/sync?)

 

[elterminatore]

do you have the ui open? certain actions/page loads on the ui can trigger some io on the datastores (e.g. the content listing)
do you have some other scheduled operations (gc/verify/sync?)

no scheduled tasks are running during daytime. (only backups)
and yes... the UI was open and showed the datastore summary page.
when i show the dashboard, the IO is gone

take a look at the screenshot. during the time in the red frame the summary page of the datastore was shown. the peak at the right side is a normal backup task. but isn't it a little bit much IO for displaying four diagrams?

 

[dcsapak]

on the summary page, we read part of the snapshot data, my guess is that your zpool is not mounted with 'relatime' which means that everytime a file is accessed its metadata will get updated (and written to disk)
with 'relatime' the access time will only updated once a day. you can turn this on with 'zfs set relatime=on <pool>'

 

[tuxis]

I thought PBS uses atime for verification-purposes (or something like that). So it is safe to enable relatime?

 

[dcsapak]

I thought PBS uses atime for verification-purposes (or something like that). So it is safe to enable relatime?

yes we use atime for gc, but we assume 'relatime' (thus the 24 hours + 5 min grace period) since that is the default for most ext4/xfs systems (and we should probably set it as default for zfs as well)

 

[tuxis]

Thanks, that'll save us a whole lot of iops!

 

[elterminatore]

on the summary page, we read part of the snapshot data, my guess is that your zpool is not mounted with 'relatime' which means that everytime a file is accessed its metadata will get updated (and written to disk)
with 'relatime' the access time will only updated once a day. you can turn this on with 'zfs set relatime=on <pool>'

thanks! that did the trick
after turning relatime on, there is only a short disk access and permanent IO is gone.



why isn't it the defauilt behavior? or in my case because i created the zpool and datastore manually and not via the UI ?

 

[dcsapak]

why isn't it the defauilt behavior? or in my case because i created the zpool and datastore manually and not via the UI ?

no it is just not the zfs default, but as i said, i think we should make it the default when a pool is created with our tools...

 

[entilza]

Hmm reading up on relatime... so should we be turning this on everywhere?

 

[dcsapak]

Hmm reading up on relatime... so should we be turning this on everywhere?

if you have no application that depends on an precise atime, it should not hurt to turn on relatime. for ext4/xfs i think it is the default

 

[entilza]

if you have no application that depends on an precise atime, it should not hurt to turn on relatime. for ext4/xfs i think it is the default

Thanks,

I see it was off on the root rpool already by default.

I set atime=off for my VM pools and set relatime=on.

I left atime=on my pool that has the PBS data.

 

[RolandK]

> I set atime=off for my VM pools and set relatime=on.

this is wrong @entilza , you need to set atime=on for making relatime work at all

furthermore, you can safely set atime=on and relatime=on on pbs, this will increase performance

 

[entilza]

I only need it for proxmox backup server not my regular VMs.