kernel 7.0 performance issue with zfs pools

D

drom

New Member
Proxmox Subscriber
Dec 2, 2025
3
1
3
Hello

I wish to report a drastic loss of performance with the 7.0 kernel as soon as it implies operating with zfs pools.

I run 2 of these, one is a single raidz2 array of 12 4TB sas hard disks, the other has 2 raidz2 arrays of 8 600GB sas hard drives and each is available to the backup server as a separate datastore. The server itself if grately overpowered with 16 cores, 130GB ram, 10GB ethernets and SSD mirrors for the system as well as the contents of the datastore namespaces (through symlinks rather direct than zfs cross mounts)

the upgrade to the 7.0 kernel made pbs verifications and garbage collecting take 3 to as much as 4 time as long as they used to, partly because the two mentionned operations were now running concurrently rather than during separated timeframes.

next the release of the first bugfix to the 7.0 kernel didn't address the problem so once I figured its origin (I must admit I was quite careless while proceding to the previous one without carefully checking the release notes, my bad) I switched back to the 6.17 kernel and now my backup server is back to its previous operationnal health.

ah about this maybe you also need to know that the zfs pools themselves haven't been upgraded to the current version shipped with the 4.0 pbs release and is still as the PBS 3 created them the reason for this beeing that the most valuable datastore is replicated using zfs snapshots and send/recv process on a remote installation still running a debian 12.5 runnng zfs 2.1.11 that we wish to fully upgrade before we do the same on the backup server. actually the full plan is even to install the proxmox backup server on that one as well and switch to PBS syncs rather than zfs transfers but that ain't the question of the thread.

I fiirst started searching the forums in order to see if someone else experienced and reported a similar issue but found nothing, but if I once again missed something I'd be glad to contribute there rather than in this new one and/or provide additionnal information if asked to.

best reagards !
 
Last edited:
I confirm the above.

Server:
40 x Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz (2 Sockets)
32G RAM
raid10 3 mirror

Stored data 40Tb

On the kernel 6.17.13 GC time was 3-4 hours
On the kernel 7.0.2 GC more than a day, up to two days

The RAM usage graph also shows that the system is not using as much memory as before.
1780134850046.png
1780134883675.png

kernel 7.0.6-2-pve

2026-05-30T11:19:51+05:00: starting garbage collection on store basket
2026-05-30T11:19:52+05:00: Access time update check successful, proceeding with GC.
2026-05-30T11:19:52+05:00: Using access time cutoff 1d 5m, minimum access time is 2026-05-29T06:14:51Z
2026-05-30T11:19:52+05:00: Start GC phase1 (mark used chunks)
2026-05-30T11:36:02+05:00: marked 1% (12 of 1165 index files)
2026-05-30T11:43:23+05:00: marked 2% (24 of 1165 index files)
2026-05-30T11:47:14+05:00: marked 3% (35 of 1165 index files)
2026-05-30T11:48:28+05:00: marked 4% (47 of 1165 index files)
2026-05-30T11:53:14+05:00: marked 5% (59 of 1165 index files)
2026-05-30T11:56:25+05:00: marked 6% (70 of 1165 index files)
2026-05-30T12:05:25+05:00: marked 7% (82 of 1165 index files)
2026-05-30T12:08:20+05:00: marked 8% (94 of 1165 index files)
2026-05-30T12:13:05+05:00: marked 9% (105 of 1165 index files)
2026-05-30T12:16:23+05:00: marked 10% (117 of 1165 index files)
2026-05-30T12:18:48+05:00: marked 11% (129 of 1165 index files)
2026-05-30T12:27:05+05:00: marked 12% (140 of 1165 index files)
2026-05-30T12:28:07+05:00: marked 13% (152 of 1165 index files)
2026-05-30T12:30:23+05:00: marked 14% (164 of 1165 index files)
2026-05-30T12:33:41+05:00: marked 15% (175 of 1165 index files)
2026-05-30T12:34:10+05:00: marked 16% (187 of 1165 index files)
2026-05-30T14:17:51+05:00: marked 17% (199 of 1165 index files)

kernel 6.17.13-13-pve

2026-05-30T15:02:08+05:00: starting garbage collection on store basket
2026-05-30T15:02:09+05:00: Access time update check successful, proceeding with GC.
2026-05-30T15:02:09+05:00: Using access time cutoff 1d 5m, minimum access time is 2026-05-29T09:57:08Z
2026-05-30T15:02:09+05:00: Start GC phase1 (mark used chunks)
2026-05-30T15:20:37+05:00: marked 1% (12 of 1165 index files)
2026-05-30T15:28:57+05:00: marked 2% (24 of 1165 index files)
2026-05-30T15:33:15+05:00: marked 3% (35 of 1165 index files)
2026-05-30T15:34:39+05:00: marked 4% (47 of 1165 index files)
2026-05-30T15:39:58+05:00: marked 5% (59 of 1165 index files)
2026-05-30T15:43:33+05:00: marked 6% (70 of 1165 index files)
2026-05-30T15:53:24+05:00: marked 7% (82 of 1165 index files)
2026-05-30T15:56:36+05:00: marked 8% (94 of 1165 index files)
2026-05-30T16:01:24+05:00: marked 9% (105 of 1165 index files)
2026-05-30T16:04:57+05:00: marked 10% (117 of 1165 index files)
2026-05-30T16:07:23+05:00: marked 11% (129 of 1165 index files)
2026-05-30T16:15:49+05:00: marked 12% (140 of 1165 index files)
2026-05-30T16:16:46+05:00: marked 13% (152 of 1165 index files)
2026-05-30T16:18:58+05:00: marked 14% (164 of 1165 index files)
2026-05-30T16:22:05+05:00: marked 15% (175 of 1165 index files)
2026-05-30T16:22:30+05:00: marked 16% (187 of 1165 index files)
2026-05-30T16:40:35+05:00: marked 17% (199 of 1165 index files)
 
Last edited:
Same here, garbage_collection tasks are (much) slower with 7.0.2-6 than with 6.17.13-11.
Fastest verification on 7.0 kernel (6h35) is slower than slowest verification on 6.17 kernel (4h15).


Christophe.
 
Seems (a little bit) better with latest kernel 7.0.6-2, latest garbage collect task : 6h46.
7.0.2-6 : average garbage collect : about 24h.
6.17.13 : (not easy to do the maths) : less than 3 hours.

Datastore with 95TiB backups.
 
I don't know why, but in kernel 7, the ARC gets flushed out and holds little metadata. Setting a value for `zfs_arc_min` prevents it from shrinking.
 
Finally got a chance to check on this. The problem seems to be related to a recently reported bug [1] that causes ARC to evict all/most of it's contents erroneously, even when there's plenty of RAM available.

This causes ARC to almost never reach it's arc_max setting, even less to keep it filled, increasing memory thrashing and disk I/O:
1787140488849.png

To mitigate this I've had to (check this [2] on how to make this values permanent):
  • Set a higher zfs_arc_min to limit the damage when ARC shrinks. The amount to use here depend on a number of factors, but I try to make it at least half zfs_arc_max (set VALUE to the amount of GiB to use):
    • echo $((VALUE*1024*1024*1024)) > /sys/module/zfs/parameters/zfs_arc_min
  • Tell ZFS to free ARC memory slower when required. Default is 0: free as much as needed on a single allocate request (set VALUE to the amount to use):
    • echo VALUE > /sys/module/zfs/parameters/zfs_arc_shrinker_limit
    • The VALUE to use here seems to depend of the amount of RAM in the host (the less ram, the often the bug will trigger). This table is working good for me:
      256 GiB → 5300
      192 GiB → 4000
      128 GiB → 2700
      96 GiB → 2000
      64 GiB → 1300
      32 GiB → 700
WARNING: be very careful on setting this tunables. The first will make ARC to always use at least that much RAM. The second will make ARC to return memory at a slower pace than default, making an OOM event more likely. Also, don't fill you RAM with ARC to the brim to avoid some edge cases related to memory fragmentation, specially if you don't have swap on your server.

The benefit is clear: even if dips still happen, they no longer invalidate the whole ARC and allow it to reach zfs_arc_max easily.
1787141429277.png

I've applied this on 5 servers of different sizes/workloads and will see in a week how beneficial has this been regarding to PBS task efficiency.

A couple extra tips somewhat related to the thread:
  • On a dedicated PBS host, disabling KSM is a good idea (once the first KSM passes merges whatever mergeable contents are there it will just waste CPU cyles. ARC isn't mergeable at all) :
    • systemctl disable --now ksmtuned
  • Biasing ZFS to keep more metadata in ARC usually helps too, specially with GC or browsing/listing backups (default is 500, this example would make it 4 times more probable to keep metadata in ARC):
    • echo 2000 > /sys/module/zfs/parameters/zfs_arc_meta_balance

[1] https://github.com/openzfs/zfs/issues/18893
[2] https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage
 
Last edited:
  • Like
Reactions: Onslow and waltar
Interesting.

IMHO the underlying issue is to rely on ARC for performance.
IMHO ARC should only be looked at as a "boost" or "(free) performance gain" and not as a given.
Not only because it is a little bit unpredictable, but also because it has to become hot and is lost after a reboot.
 
Check this point of view: there's no ZFS without ARC: it's and inherent part of ZFS architecture and ZFS performance is designed around it, so you do relay on it and expect it to behave correctly to get the most of your ZFS. Like why not raise all sails on the boat to make it faster.

IsThisThingOn said:
Not only because it is a little bit unpredictable, but also because it has to become hot and is lost after a reboot.
Click to expand...
Something that happens with any other disk cache, which is what ARC is. And for an always on PBS server losing ARC on reboot isn't really an issue, specially when datastore size is bigger than a few TB and it's whole metadata (even less data) won't ever fit in ARC.
 
VictorSTS said:
Like why not raise all sails on the boat to make it faster.
Click to expand...
IMHO it is more like me having a sailing boat and a 5PS small motor.
There might be wind, and I might be able to travel at 20km/h.
But I don't count on it. I know that worst case, there is no wind and I have to use my 5PS motor which will get me 4km/h.
BTW this is a very real example. You can see this often at Lake Constance :)
VictorSTS said:
Something that happens with any other disk cache, which is what ARC is.
Click to expand...
I don't agree, or maybe we are misundarstanding each other. For a good performance pool, I can rely on its performance. It is not based on a reboot.
A special vdev or L2ARC is a "form of disk cache" that is not affected by a reboot.

Don't get me wrong, of course the behavior of ZFS still seems to be wrong here. But again, you would not be really affected, if you would not rely on ARC to get performance to begin with. RAIDZ & HDDs & no L2ARC & no svdev is pretty much against the recommendations for PBS.
Which is fine. A TrueNAS forum member always said in such cases; "There might be dragons".
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back