Constant writes on backup storage (VZDump backup file)

Feb 25, 2020
7
0
21
Hey there,
i'm currently having the issue, that roughly every 10 seconds something is writing ~1.7M to my backup storage.

I have a dedicated file server and created an smb share there to store backups of my VMs.
After adding the share in PVE and enabling it, i noticed that every ~10 seconds something gets written to the storage.
When i disable the share in PVE, it stops.

Unfortunately, i'm unable to figure out which pve process is causing those writes, but it's pretty annoying and unnecessary - since no backup is running.
Any idea how to stop that behavior?

I'm currently running PVE v7.4-15 (Standalone)
 

Attachments

  • zpool-iostat.txt
    18.8 KB · Views: 7
PVE will poll every 8 seconds or so all storages to see if they are still available. Maybe updating the atime is causing small writes and these get amplified?
 
  • Like
Reactions: gadean
@RolandK At least it's causing the writes, since it stops when i disable the storage in pve.
cifsiostat shows from time to time 3 closed files, but that's it

@Dunuin I've set atime to no and it looks like that stopped it - i've ran zpool iostat now for 10 min and had 0 writes.
Do you know if it's possible to increase the check interval or even completely disable it?
 
you should be able to use fatrace utility to check what is accessing and where, at least on the smb server side

on my smb server it looks like this - smbd is accessing these dirs every 10 seconds
root@backup-usb-filer:/backup# fatrace -c smbd(486): O /backup/s740_backup_vzdump smbd(486): C /backup/s740_backup_vzdump smbd(486): O /backup/s740_backup_vzdump/dump smbd(486): C /backup/s740_backup_vzdump/dump smbd(486): O /backup/s740_backup_vzdump smbd(486): C /backup/s740_backup_vzdump

the access to the smb mount can be traced on the pve server with strace

# strace -f -p 1711 2>&1 |grep mnt [pid 3710186] newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer", <unfinished ...> newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer/dump", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0 newfstatat(AT_FDCWD, "/mnt", {st_mode=S_IFDIR|0755, st_size=22, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(AT_FDCWD, "/mnt/pve", {st_mode=S_IFDIR|0755, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer", {st_mode=S_IFDIR|0755, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 newfstatat(AT_FDCWD, "/mnt/pve/s740-vzdumps_auf_backup-usb-filer/dump", {st_mode=S_IFDIR|0755, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 [pid 3710187] statfs("/mnt/pve/s740-vzdumps_auf_backup-usb-filer", <unfinished ...> read(8, "grep\0mnt\0", 8192) = 9

i'd really wonder when a simple atime update on 2 directories would cause 1.7mb writes - but zfs may surprise us sometimes and i would consider opening a tickat in zfs github for that
 
Last edited:
  • Like
Reactions: gadean
  • Like
Reactions: gadean and _gabriel
@RolandK As far as i know fatrace isn't available for freebsd and other tools didn't work out.
For strace i need a pid that i'm unable to find :(

Overall, it feels like pve and zfs aren't really good for each other - at least in my experience
causing higher wearout on ssds with zfs compared to ext4 (18% vs 1% - same runtime, only for the OS) and now that

@Dunuin Thanks a lot, gonna check it out :)
 
Last edited:
With smart?
I have two system, one with ext4 and one with zfs - only for the OS, nothing else on thos disks / both have the same model of SSD installed.
And the system with the ext4 gets used most of the time, while the other just runs a few VMs.

Edit:

Power_On_HoursTotal_LBAs_Written
ext4348865.972.017.877
zfs3133458.062.916.752
 
Last edited:
you are sure that BOTH are only OS disks and you don't have accidentaly a VM on the zfs OS disk ?

if you are right with your observation, maybe this is some pathological write-amplification bacause of sqlite config.db , i had some discussion on this a while ago at https://www.mail-archive.com/pve-devel@lists.proxmox.com/msg02986.html

maybe it's really worth to inspect this more thoroughly if zfs does 18x more write in comparison to ext4 when being used for the rootfs.
 
Last edited:
  • Like
Reactions: gadean
Yep, the "local" storage is disable, there are no VMs on it and no other storage on the disk - that was one of the first things i checked after i noticed that.

Edit: Haha, i just notice that i'm writing with you :D
I saw that post last year while i was searching for a hint regarding that

Not sure if that's a good idea, since i'm not that experienced, but would it be bad if i set atime to "no" on the OS-pool?
Just to see if that changes something
 
Last edited:
Not sure if that's a good idea, since i'm not that experienced, but would it be bad if i set atime to "no" on the OS-pool?
Just to see if that changes something
Check what zfs get relatime rpool/ROOT/pve-1 reports. If it is set to "on" then a "atime=off" won't help much, as the atime then will only be updated once per day.

And yes, write amplification is horrible with ZFS when doing small writes, especially small sync writes. And the OS is primarily doing small writes to log files or the cluster.db. So I wouldn't wonder if the SSD with ZFS would wear that much more.
 
Last edited:
It's set to "off" and i see every 5 seconds a small writes (~1.3M) with zpool iostat

Edit: just checked the ext4 system and there it's only a few kB (highest was ~300) every 5 seconds
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!