Constant writes on backup storage (VZDump backup file)

[gadean]

Hey there,
i'm currently having the issue, that roughly every 10 seconds something is writing ~1.7M to my backup storage.

I have a dedicated file server and created an smb share there to store backups of my VMs.
After adding the share in PVE and enabling it, i noticed that every ~10 seconds something gets written to the storage.
When i disable the share in PVE, it stops.

Unfortunately, i'm unable to figure out which pve process is causing those writes, but it's pretty annoying and unnecessary - since no backup is running.
Any idea how to stop that behavior?

I'm currently running PVE v7.4-15 (Standalone)

 

[RolandK]

are you sure these writes originate from pve ?

what does cifsiostat tell on the pve side ?

 

[Dunuin]

PVE will poll every 8 seconds or so all storages to see if they are still available. Maybe updating the atime is causing small writes and these get amplified?

 

[gadean]

@RolandK At least it's causing the writes, since it stops when i disable the storage in pve.
cifsiostat shows from time to time 3 closed files, but that's it

@Dunuin I've set atime to no and it looks like that stopped it - i've ran zpool iostat now for 10 min and had 0 writes.
Do you know if it's possible to increase the check interval or even completely disable it?

 

[RolandK]

you should be able to use fatrace utility to check what is accessing and where, at least on the smb server side

on my smb server it looks like this - smbd is accessing these dirs every 10 seconds
root@backup-usb-filer:/backup# fatrace -c smbd(486): O /backup/s740_backup_vzdump smbd(486): C /backup/s740_backup_vzdump smbd(486): O /backup/s740_backup_vzdump/dump smbd(486): C /backup/s740_backup_vzdump/dump smbd(486): O /backup/s740_backup_vzdump smbd(486): C /backup/s740_backup_vzdump

the access to the smb mount can be traced on the pve server with strace

# strace -f -p 1711 2>&1 |grep mnt [pid 3710186] newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer", <unfinished ...> newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer/dump", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0 newfstatat(AT_FDCWD, "/mnt", {st_mode=S_IFDIR|0755, st_size=22, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(AT_FDCWD, "/mnt/pve", {st_mode=S_IFDIR|0755, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer", {st_mode=S_IFDIR|0755, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer/dump", {st_mode=S_IFDIR|0755, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 [pid 3710187] statfs("/mnt/pve/s740-vzdumps_auf_backup-usb-filer", <unfinished ...> read(8, "grep\0mnt\0", 8192) = 9

i'd really wonder when a simple atime update on 2 directories would cause 1.7mb writes - but zfs may surprise us sometimes and i would consider opening a tickat in zfs github for that

 

[Dunuin]

Do you know if it's possible to increase the check interval or even completely disable it?

No, not possible. Common workaround is to create a vzdump hook script that will enable the storage when a backup job starts and disables the storage when the backup job has finished. That way it is not polled when not needed. See for example here:
https://forum.proxmox.com/threads/vzdzump-script-enable-cifs-storage.106694/

 

[gadean]

@RolandK As far as i know fatrace isn't available for freebsd and other tools didn't work out.
For strace i need a pid that i'm unable to find

Overall, it feels like pve and zfs aren't really good for each other - at least in my experience
causing higher wearout on ssds with zfs compared to ext4 (18% vs 1% - same runtime, only for the OS) and now that

@Dunuin Thanks a lot, gonna check it out

 

[RolandK]

>causing higher wearout on ssds with zfs compared to ext4 (18% vs 1% - same runtime, only for the OS)

how did you measure that ?

 

[gadean]

With smart?
I have two system, one with ext4 and one with zfs - only for the OS, nothing else on thos disks / both have the same model of SSD installed.
And the system with the ext4 gets used most of the time, while the other just runs a few VMs.

Edit:

	Power_On_Hours	Total_LBAs_Written
ext4	34886	5.972.017.877
zfs	31334	58.062.916.752

 

[RolandK]

you are sure that BOTH are only OS disks and you don't have accidentaly a VM on the zfs OS disk ?

if you are right with your observation, maybe this is some pathological write-amplification bacause of sqlite config.db , i had some discussion on this a while ago at https://www.mail-archive.com/pve-devel@lists.proxmox.com/msg02986.html

maybe it's really worth to inspect this more thoroughly if zfs does 18x more write in comparison to ext4 when being used for the rootfs.

 

[gadean]

Yep, the "local" storage is disable, there are no VMs on it and no other storage on the disk - that was one of the first things i checked after i noticed that.

Edit: Haha, i just notice that i'm writing with you
I saw that post last year while i was searching for a hint regarding that

Not sure if that's a good idea, since i'm not that experienced, but would it be bad if i set atime to "no" on the OS-pool?
Just to see if that changes something

 

[Dunuin]

Not sure if that's a good idea, since i'm not that experienced, but would it be bad if i set atime to "no" on the OS-pool?
Just to see if that changes something

Check what zfs get relatime rpool/ROOT/pve-1 reports. If it is set to "on" then a "atime=off" won't help much, as the atime then will only be updated once per day.

And yes, write amplification is horrible with ZFS when doing small writes, especially small sync writes. And the OS is primarily doing small writes to log files or the cluster.db. So I wouldn't wonder if the SSD with ZFS would wear that much more.

 

[gadean]

It's set to "off" and i see every 5 seconds a small writes (~1.3M) with zpool iostat

Edit: just checked the ext4 system and there it's only a few kB (highest was ~300) every 5 seconds

 

[Dunuin]

Then I would set zfs set relatime=on rpool.

 

[gadean]

Unfortunately it doesn't make any difference