zfs_resilver_delay - Permission denied (parameter doesn't exist?)

[alpha754293]

I am trying to speed up my resilvering process as I am expanding one of my raidz2 vdevs from being 8x HGST 6 TB SATA 6 Gbps 7200 rpm HDDs to 8x WD DC520 12 TB SAS 12 Gbps 7200 rpm HDDs.

From this reddit post, it suggested that I tried to set these ZFS module parameters to achieve this goal:

echo 0 > /sys/module/zfs/parameters/zfs_resilver_delay
echo 512 > /sys/module/zfs/parameters/zfs_top_maxinflight
echo 8000 > /sys/module/zfs/parameters/zfs_resilver_min_time_ms

However, when I tried to execute the first step, I get:

-bash: /sys/module/zfs/parameters/zfs_resilver_delay: Permission denied

When I checked the /sys/module/zfs/parameters folder, that parameter doesn't exist.

Is that by design or is this an issue with my system?

Here is the output of my package versions:

proxmox-ve: 7.4-1 (running kernel: 5.15.149-1-pve)
pve-manager: 7.4-17 (running version: 7.4-17/513c62be)
pve-kernel-5.15: 7.4-12
pve-kernel-5.15.149-1-pve: 5.15.149-1
pve-kernel-5.15.143-1-pve: 5.15.143-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.3
libpve-apiclient-perl: 3.2-2
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.6-1
proxmox-backup-file-restore: 2.4.6-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+3
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.10-1
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-5
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.15-pve1

Any help or insight as to why this ZFS module parameter appearing to be absent from my system is greatly appreciated.

Thank you.

P.S. I'm running Proxmox 7.4-17 with the 5.15.149-1 Linux kernel.

Hardware specs:
2x Intel Xeon E5-2697A v4
Supermicro X10DRi-T4+
16x Crucial DDR4-2666(?) ECC Reg. RAM running at DDR4-2400 speed (I think? I can't remember anymore)
Broadcom/Avago/LSI 9361-8i MegaRAID SAS 12 Gbps HW SAS RAID HBA
RTX A2000 6 GB
Mellanox ConnectX-4 dual VPI port 100 Gbps IB NIC
4x HGST 3 TB SATA 3 Gbps 7200 rpm HDDs for the Proxmox OS itself (in HW RAID6)
8x HGST 6 TB SATA 6 Gbps 7200 rpm HDDs for VM/CT disks (raidz2)
8x HGST 6 TB + 8x 10 TB + 8x 10 TB HDDs for bulk storage (one pool, three raidz2 vdevs)
Supermicro 4U 36-bay server.

*edit*
It would appear that the ZFS module parameter zfs_top_maxinflight also doesn't exist in /sys/module/zfs/parameters, in Proxmox 7.4-17.

 

[waltar]

The available zfs parameters are release dependend so that's not a problem of your installed version 2.1.15.
Try this instead "echo 5 >/sys/module/zfs/parameters/zfs_scan_mem_lim_fact" .

 

[alpha754293]

The available zfs parameters are release dependend so that's not a problem of your installed version 2.1.15.
Try this instead "echo 5 >/sys/module/zfs/parameters/zfs_scan_mem_lim_fact" .

This will change the fraction of RAM that is used for I/O sorting from being (physical RAM / 20) to (physical RAM / 5), correct?

(P.S. I am not sure that I would want to touch any of the parameters that would also deal with RAM only because the system is a live server, and as such, it is currently also still serving services, so I don't want to set a ZFS module parameter that will adversely influence or have an adverse impact on the performance of said services that the server is serving.

As such, I have already set the ZFS ARC min size to be 8 GiB and a ZFS ARC max size to be only 64 GiB (out of a system with a total of 256 GB of RAM).

 

[waltar]

Right as it allows to use more mem of the arc while scrubbing/resilvering but going down again when it's done, see zpool status and zpool iostat -l 1 while running. As you save iops that way it should not influence zfs service more than without that.
Btw. your previous resilver parameters will more hurt zfs service performance.
Your arc min and max settings are very low for a host with 256GB and I won't do that, compare with top for displayed mem free.

 

[alpha754293]

Right as it allows to use more mem of the arc while scrubbing/resilvering but going down again when it's done

Thank you.

ARC utilisation is already at 100% of the 64 GiB max that has been allocated to it.

I would imagine that if it is using the ARC for resilvering, that it would be competing with ARC hits for non-resilvering reads as well, which then results in ARC contention, no?

see zpool status and zpool iostat -l 1 while running.

Yeah, I have two terminal tabs that are monitoring the resilvering performance -- one for zpool status and the other for watch -n1 zpool iostat -vy 1 1 to monitor the iostat performance.

As you save iops that way it should not influence zfs service more than without that.

Does it really save I/O/s though? (I am trying to figure out how this statement can be true.)

I mean, ARC would still have the fetch the data from disk, whether that's used for the resilvering process or the remaining non-resilvering reads, right?

I would agree with you if the data to be resilvered is already sitting in ARC, then I can see it saving on I/O/s because it's resilvering from ARC, but if the data isn't already in ARC, then it would have to fetch it from disk to ARC, for it to be effective, correct?

Please correct me if my logic is incorrect in terms of how I am thinking about its mode of operation and how that ZFS module parameter will impact performance (both resilvering and non-resilvering performance).

Btw. your previous resilver parameters will more hurt zfs service performance.

Understood. I don't mind sacrificing a little bit of ZFS service performance, prioritising resilver performance so long as it doesn't end up taking the service down (because if that happens, then it would be better for me to shut down the service altogether, so that it will stop the service from reading data from the pool entirely, which I have thought about, but haven't implemented yet).

Your arc min and max settings are very low for a host with 256GB and I won't do that, compare with top for displayed mem free.

Thank you.

Yes, I know.

This is deliberate.

The default ZFS ARC settings aren't optimal for my system because the max ARC is 1/2 of physical RAM installed (i.e. 128 GB), which is really, really high.

I have somewhere between 17-24 LXCs running, at any given point in time, as well as something like 8 VMs running as well. (Some of the larger LLMs need upwards of 48 GB of RAM to run by itself.)

Some of those VMs uses virtio-fs for VM <-> host communication, and as such, it mirrors/echos/shares data in RAM to facilitate said VM <-> host communication.

(PVE actually tells me that the size of the virtio-fs device/socket needs to equal the RAM that's allocated to the VM.)

When I first deployed the system, I actually originally set the ZFS ARC min/max to be 4 GiB and 32 GiB respectively, but then found that bumping that back up to 8 GiB and 64 GiB respective striked a balance between VM virtio-fs performance and host performance.

If I DIDN'T cap the ZFS ARC max to 64 GiB, then the VM's/LXC's performance will start to suffer as it competes with ARC for RAM, and ARC doesn't always release the RAM back to the VMs/LXCs in time, which has resulted in my VMs crashing due to "out of memory" errors.

So, I have purposely capped the ZFS ARC max, to mitigate this issue and I've been running it this way for almost 2 years now.

 

[alpha754293]

*update*
So, I ended up stopping all but 3 of the services that my Proxmox host was hosting/responsible.

As such, that allowed pretty much almost all of the RAM (256 GB) of the system to be repurposed for the ZFS resilvering process, and I set the zfs_arc_min and zfs_arc_max to be 75% of phsymem (or 192 GB) respectively.

I also set the zfs_scan_mem_lim_fact to be 5 as well (so that it has upto 1/5th of physmem) for the sorting algorithm to work.

It still took my system 3 days 16 hours and change to resilver the first 6 TB -> 12 TB replacement.

One down, 7 more to go.

This is going to take a while.

 

[leesteken]

As such, that allowed pretty much almost all of the RAM (256 GB) of the system to be repurposed for the ZFS resilvering process, and I set the zfs_arc_min and zfs_arc_max to be 75% of phsymem (or 192 GB) respectively.

There has been a report about ZFS using (much) more memory than zfs_arc_max before. It's unclear why this happens but appears to be rare. Maybe it's (amplified) writes that are taking long to process? I don't expect setting the zfs_arc_max high will give any benefits for resilvering or scrubbing, so maybe set it to a more reasonable size (like 10% or 20%) and reboot? I believe things got better with newer ZFS versions on newer Proxmox versions.

 

[waltar]

Yeah, that's one of the dark sides of zfs, having more than 30% used in a zpool and any non-zfs raid is much faster in scrub/rebuild/resilver than zfs.
I don't expect also - setting the zfs_arc_max that high will give any benefits for resilvering or scrubbing and stay it as before.

 

[alpha754293]

My thought process (for increasing the ARC) was to increase the amount that could be read and cached in RAM such that for the resilvering piece of it, it would be reaching from ARC rather than directly from disk.

Not sure if that worked at all, but I was also operating on the idea that giving ARC more RAM (during a resilver) can't hurt, right?

(I could be wrong though.)

so maybe set it to a more reasonable size (like 10% or 20%) and reboot?

So with my zfs_arc_max currently set to 64 GiB, that works out to be 25% of physmem, so that should work with this recommendation/suggestion.

I was watching Saso Kiselkov's video about ZFS resilvering performance on YouTube earlier together today (both the original video, uploaded 7 years ago, plus his follow-up the following year and it seemed like that his general recommendation was ~ 1 GB of physmem per TB of data. So with `zfs_scan__mem_lim_fact being set to the default 20, it should be plenty of RAM for the sorting algorithm that is needed for resilvering.