Low ZFS read performance Disk->Tape

dcsapak · Apr 30, 2024

Lephisto said:
let me bump this up. Is there any news regarding tape write performance?

sorry for the late answer, but i did now send some patches that may improve the performance:

https://lists.proxmox.com/pipermail/pbs-devel/2024-April/009162.html
https://lists.proxmox.com/pipermail/pbs-devel/2024-April/009161.html

basically we read the data from the datastore in a single thread, which is probably not optimal for HDD especially when they're in some kind of raid (since then the reads cannot distribute across the disks in parallel)
my patches add a tuning option for the number of threads, and for tapes sets the default to 4 (the same value we currently use for verification threads)

we're going to test that in various configurations (also with a HDD raid array to an LTO-8 drive), i can post the result when we're done with that (no idea when this will be though)

Lephisto · Apr 30, 2024

This is awesome news, I am really looking forward to this..

Lephisto · May 2, 2024

Can I somehow test this without doing a complete PBS build myself? (eG. are there development deb's?)

dcsapak · May 2, 2024

no we don't have dev builds for patches that are not applied + package bumped yet, but we'll still evaluating if the patches are improving the situation
if it does, we'll normally apply + bump the version and then it should be available on test repository relatively shortly after

EDIT: in case you're really curious, you can build it yourself of course, here's a link to our dev documentation (that should include build instructions): https://pbs.proxmox.com/wiki/index.php/Developer_Documentation

Lephisto · May 6, 2024

Looking at the Thread again: If the gains are to visible for a single Spinner, the effect to have multiple IO threads should be even more Interessting on ZFS pools with several Disks...

Lephisto · May 6, 2024

Okay, was i impatient, built PBS myself and can conform significant improvements to the ZFS Performance on Spinners, especially when writing data to tape. Finally I can (at least if no other things are running) saturate the write performance of my LTO9 drive. Before the patch it maxed at 200MB/s, now with 8 iothreads I ceil at 300MB/s (maximum for that LTO9 drive).

Awesome, thank you.

Lephisto · May 6, 2024

More Update on the Topic:

Increased the Thread to 16 (which is core count of the backup machine):

300MB/s (was <=200MB/s volatile) when running a tape job only.
~160-200MB/s (was <= 60MB/s volatile) when tape job was running while a verify job was running.

//What I am noticing now is some sort of decline over a jobs runtime:

Code:

2024-05-06T14:27:34+02:00: wrote 1417 chunks (4295.23 MB at 229.53 MB/s)
2024-05-06T14:27:48+02:00: wrote 1452 chunks (4297.06 MB at 298.31 MB/s)
2024-05-06T14:28:03+02:00: wrote 1452 chunks (4298.11 MB at 298.78 MB/s)
2024-05-06T14:28:17+02:00: wrote 1433 chunks (4295.75 MB at 297.90 MB/s)
2024-05-06T14:28:32+02:00: wrote 1457 chunks (4297.33 MB at 291.27 MB/s)
2024-05-06T14:28:49+02:00: wrote 1523 chunks (4296.02 MB at 252.33 MB/s)
2024-05-06T14:29:04+02:00: wrote 1583 chunks (4297.59 MB at 290.71 MB/s)
2024-05-06T14:29:19+02:00: wrote 1648 chunks (4296.54 MB at 292.32 MB/s)
2024-05-06T14:29:36+02:00: wrote 1517 chunks (4295.49 MB at 250.63 MB/s)
2024-05-06T14:29:54+02:00: wrote 1362 chunks (4296.02 MB at 229.39 MB/s)
2024-05-06T14:30:08+02:00: wrote 1307 chunks (4297.33 MB at 305.63 MB/s)
2024-05-06T14:30:23+02:00: wrote 1286 chunks (4297.33 MB at 301.64 MB/s)
2024-05-06T14:30:37+02:00: wrote 1312 chunks (4297.85 MB at 296.39 MB/s)
2024-05-06T14:30:52+02:00: wrote 1290 chunks (4295.75 MB at 298.68 MB/s)
2024-05-06T14:31:06+02:00: wrote 1264 chunks (4297.06 MB at 299.40 MB/s)
2024-05-06T14:31:20+02:00: wrote 1283 chunks (4297.85 MB at 303.93 MB/s)
2024-05-06T14:31:36+02:00: wrote 1376 chunks (4296.02 MB at 276.17 MB/s)
2024-05-06T14:31:50+02:00: wrote 1226 chunks (4298.11 MB at 296.89 MB/s)
2024-05-06T14:32:05+02:00: wrote 1259 chunks (4297.85 MB at 293.44 MB/s)
2024-05-06T14:32:19+02:00: wrote 1283 chunks (4296.54 MB at 294.67 MB/s)
2024-05-06T14:32:36+02:00: wrote 1307 chunks (4296.28 MB at 254.66 MB/s)
2024-05-06T14:32:50+02:00: wrote 1359 chunks (4298.64 MB at 303.59 MB/s)
2024-05-06T14:33:05+02:00: wrote 1285 chunks (4296.28 MB at 298.23 MB/s)
2024-05-06T14:33:19+02:00: wrote 1220 chunks (4296.80 MB at 304.04 MB/s)
2024-05-06T14:33:33+02:00: wrote 1195 chunks (4298.64 MB at 302.95 MB/s)
2024-05-06T14:33:47+02:00: wrote 1201 chunks (4298.64 MB at 304.21 MB/s)
2024-05-06T14:34:01+02:00: wrote 1200 chunks (4297.06 MB at 313.64 MB/s)
2024-05-06T14:34:15+02:00: wrote 1243 chunks (4297.85 MB at 304.32 MB/s)
2024-05-06T14:34:29+02:00: wrote 1330 chunks (4296.02 MB at 303.70 MB/s)
2024-05-06T14:34:44+02:00: wrote 1422 chunks (4295.75 MB at 287.84 MB/s)
2024-05-06T14:35:02+02:00: wrote 1455 chunks (4296.28 MB at 237.75 MB/s)
2024-05-06T14:35:26+02:00: wrote 1455 chunks (4296.54 MB at 177.95 MB/s)
2024-05-06T14:35:49+02:00: wrote 1487 chunks (4295.23 MB at 199.38 MB/s)
2024-05-06T14:36:16+02:00: wrote 1431 chunks (4296.28 MB at 168.17 MB/s)
2024-05-06T14:36:39+02:00: wrote 1679 chunks (4298.38 MB at 198.55 MB/s)
2024-05-06T14:37:07+02:00: wrote 1605 chunks (4295.23 MB at 166.31 MB/s)
2024-05-06T14:37:30+02:00: wrote 1525 chunks (4296.54 MB at 193.95 MB/s)
2024-05-06T14:37:53+02:00: wrote 1529 chunks (4296.02 MB at 196.08 MB/s)
2024-05-06T14:38:13+02:00: wrote 1384 chunks (4296.80 MB at 238.34 MB/s)
2024-05-06T14:38:31+02:00: wrote 1346 chunks (4296.02 MB at 253.07 MB/s)
2024-05-06T14:38:49+02:00: wrote 1376 chunks (4298.11 MB at 254.48 MB/s)
2024-05-06T14:39:07+02:00: wrote 1374 chunks (4296.54 MB at 255.77 MB/s)
2024-05-06T14:39:26+02:00: wrote 1518 chunks (4297.33 MB at 248.65 MB/s)
2024-05-06T14:39:44+02:00: wrote 1437 chunks (4297.59 MB at 252.82 MB/s)
2024-05-06T14:40:03+02:00: wrote 1442 chunks (4296.02 MB at 252.92 MB/s)
2024-05-06T14:40:21+02:00: wrote 1488 chunks (4298.11 MB at 248.45 MB/s)
2024-05-06T14:40:40+02:00: wrote 1488 chunks (4297.06 MB at 250.81 MB/s)
2024-05-06T14:41:00+02:00: wrote 1465 chunks (4296.80 MB at 263.58 MB/s)
2024-05-06T14:41:22+02:00: wrote 1486 chunks (4296.28 MB at 207.28 MB/s)
2024-05-06T14:41:47+02:00: wrote 1543 chunks (4295.23 MB at 187.08 MB/s)
2024-05-06T14:42:12+02:00: wrote 1508 chunks (4296.80 MB at 179.75 MB/s)
2024-05-06T14:42:43+02:00: wrote 1517 chunks (4296.80 MB at 141.82 MB/s)
2024-05-06T14:43:08+02:00: wrote 1529 chunks (4295.23 MB at 183.25 MB/s)
2024-05-06T14:43:36+02:00: wrote 2199 chunks (4298.38 MB at 164.11 MB/s)
2024-05-06T14:44:05+02:00: wrote 2690 chunks (4296.02 MB at 161.29 MB/s)
2024-05-06T14:44:31+02:00: wrote 1692 chunks (4295.23 MB at 179.66 MB/s)
2024-05-06T14:44:55+02:00: wrote 1488 chunks (4295.23 MB at 187.12 MB/s)
2024-05-06T14:45:20+02:00: wrote 1509 chunks (4298.90 MB at 184.92 MB/s)
2024-05-06T14:45:46+02:00: wrote 1506 chunks (4298.90 MB at 182.74 MB/s)
2024-05-06T14:46:10+02:00: wrote 1470 chunks (4295.49 MB at 183.46 MB/s)
2024-05-06T14:46:35+02:00: wrote 1368 chunks (4296.54 MB at 188.28 MB/s)
2024-05-06T14:47:03+02:00: wrote 1367 chunks (4297.33 MB at 162.10 MB/s)
2024-05-06T14:47:29+02:00: wrote 1398 chunks (4295.75 MB at 167.12 MB/s)
2024-05-06T14:47:54+02:00: wrote 1474 chunks (4297.33 MB at 186.90 MB/s)
2024-05-06T14:48:18+02:00: wrote 1389 chunks (4298.38 MB at 189.76 MB/s)
2024-05-06T14:48:43+02:00: wrote 1375 chunks (4296.02 MB at 188.45 MB/s)
2024-05-06T14:49:12+02:00: wrote 1369 chunks (4295.75 MB at 160.23 MB/s)
2024-05-06T14:49:39+02:00: wrote 1379 chunks (4295.75 MB at 188.49 MB/s)

What could cause this? The tape job still is the only one...

This happened:

dcsapak · May 6, 2024

Lephisto said:
What could cause this? The tape job still is the only one...

mhmm.. how much the disks can read, depends probably largely on how the chunks are laid out on them. so i did further tests (will post in the commit message in a v2 of the patches) that show the chunk layout on disk
can have a large impact on read speed (e.g. are they sequential on disk, or scattered)

Lephisto said:
This happened:

my first seeing this graph would be that the backup now reads from disks that are slower e.g. if you have mixed disks in RAID10 array, then if the later disks in the array are slower this could happen
another thing could be some kind of cache that ran out of space/data (e.g. on a RAID controller)
also thermal limits could play a role here

but to be sure we'd need more data (e.g. atop or similar)

does the perf drop happen always after the same time? or after a specific amount of data ?

Lephisto · May 6, 2024

First I would like to provide you with some Specs:

AMD EPYC 7313 16-Core Processor
12x Seagate Exos 20tb
4x 4tb NVME

It's configured as raid-z3 with a 4-way mirror special device on nvme.

It - seems - that the drop comes in after some time, a few minutes. I made another job:

Code:

2024-05-06T16:30:05+02:00: Starting tape backup job 'px11:px11-mai:lto9:px11'
2024-05-06T16:30:05+02:00: update media online status
2024-05-06T16:30:07+02:00: media set uuid: 1b68f06c-183d-4f85-a40a-eefd1c71acc7
2024-05-06T16:30:08+02:00: found 134 groups (out of 134 total)
2024-05-06T16:30:08+02:00: latest-only: true (only considering latest snapshots)
2024-05-06T16:30:08+02:00: skip snapshot vm/9000/2023-11-05T22:00:01Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10100/2024-05-05T20:30:03Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10101/2024-05-05T20:30:05Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10104/2024-05-05T20:31:22Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10105/2023-08-30T02:00:01Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10106/2024-05-05T20:38:01Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10107/2024-05-05T20:30:05Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10108/2024-05-05T20:39:05Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10109/2024-05-05T20:30:06Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10110/2024-05-05T20:31:17Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10111/2024-05-05T20:31:21Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10112/2024-05-05T20:32:56Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10113/2024-05-05T20:34:13Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10114/2024-05-05T20:35:07Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10115/2024-05-05T20:40:54Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10116/2024-05-05T20:41:04Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10117/2024-05-05T20:45:09Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10118/2024-05-05T20:46:02Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10119/2024-05-05T20:46:07Z
2024-05-06T16:30:08+02:00: skip snapshot vm/10120/2024-05-05T20:46:12Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10151/2024-05-05T20:46:47Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10152/2024-05-05T20:30:12Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10153/2024-05-05T20:30:05Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10154/2024-05-05T20:53:36Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10155/2024-05-05T20:57:20Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10156/2024-05-05T20:59:35Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10157/2024-05-05T21:00:57Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10158/2024-05-05T21:02:28Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10159/2024-05-05T21:03:50Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10160/2024-05-05T21:05:29Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10161/2024-05-05T21:08:26Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10162/2024-05-05T21:10:29Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10163/2024-05-05T20:33:30Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10164/2024-05-05T21:13:24Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10165/2024-05-05T21:14:57Z
2024-05-06T16:30:09+02:00: skip snapshot vm/10245/2024-05-05T20:33:02Z
2024-05-06T16:30:09+02:00: backup snapshot "vm/10501/2024-05-05T20:33:12Z"
2024-05-06T16:30:09+02:00: allocated new writable media 'NSK125L9'
2024-05-06T16:30:09+02:00: trying to load media 'NSK125L9' into drive 'lto9'
2024-05-06T16:34:41+02:00: found media label NSK125L9 (d967a51f-49dd-4659-bf31-b5332e3272f5)
2024-05-06T16:34:42+02:00: moving to end of media
2024-05-06T16:37:14+02:00: arrived at end of media
2024-05-06T16:37:32+02:00: wrote 1564 chunks (4297.59 MB at 233.06 MB/s)
2024-05-06T16:37:46+02:00: wrote 1222 chunks (4296.54 MB at 304.79 MB/s)
2024-05-06T16:38:00+02:00: wrote 1384 chunks (4297.85 MB at 304.53 MB/s)
2024-05-06T16:38:15+02:00: wrote 1442 chunks (4295.75 MB at 304.76 MB/s)
2024-05-06T16:38:31+02:00: wrote 1632 chunks (4296.28 MB at 256.17 MB/s)
2024-05-06T16:38:45+02:00: wrote 1425 chunks (4296.80 MB at 304.32 MB/s)
2024-05-06T16:39:00+02:00: wrote 1287 chunks (4297.85 MB at 304.79 MB/s)
2024-05-06T16:39:14+02:00: wrote 1294 chunks (4296.28 MB at 304.94 MB/s)
2024-05-06T16:39:28+02:00: wrote 1325 chunks (4296.28 MB at 304.95 MB/s)
2024-05-06T16:39:49+02:00: wrote 1504 chunks (4296.02 MB at 206.78 MB/s)
2024-05-06T16:40:13+02:00: wrote 1562 chunks (4296.28 MB at 177.63 MB/s)
2024-05-06T16:40:37+02:00: wrote 1320 chunks (4297.33 MB at 178.40 MB/s)
2024-05-06T16:41:02+02:00: wrote 1271 chunks (4296.28 MB at 167.29 MB/s)
2024-05-06T16:41:23+02:00: wrote 1610 chunks (4295.23 MB at 209.41 MB/s)
2024-05-06T16:41:49+02:00: wrote 1319 chunks (4296.54 MB at 166.62 MB/s)
2024-05-06T16:42:09+02:00: wrote 1329 chunks (4297.06 MB at 211.22 MB/s)
2024-05-06T16:42:35+02:00: wrote 1536 chunks (4296.54 MB at 168.35 MB/s)
2024-05-06T16:43:00+02:00: wrote 2278 chunks (4295.49 MB at 168.34 MB/s)
2024-05-06T16:43:18+02:00: wrote 1625 chunks (4296.80 MB at 243.88 MB/s)
2024-05-06T16:43:38+02:00: wrote 1397 chunks (4295.49 MB at 215.55 MB/s)
2024-05-06T16:43:57+02:00: wrote 1379 chunks (4297.85 MB at 229.11 MB/s)
2024-05-06T16:44:11+02:00: wrote 1397 chunks (4296.80 MB at 298.83 MB/s)
2024-05-06T16:44:26+02:00: wrote 1376 chunks (4297.33 MB at 291.50 MB/s)
2024-05-06T16:44:43+02:00: wrote 1426 chunks (4296.28 MB at 251.02 MB/s)
2024-05-06T16:45:01+02:00: wrote 1325 chunks (4297.59 MB at 232.79 MB/s)
2024-05-06T16:45:15+02:00: wrote 1309 chunks (4296.54 MB at 303.61 MB/s)
2024-05-06T16:45:30+02:00: wrote 1443 chunks (4297.59 MB at 303.34 MB/s)
2024-05-06T16:45:45+02:00: wrote 1773 chunks (4296.28 MB at 271.36 MB/s)
2024-05-06T16:46:07+02:00: wrote 1490 chunks (4295.75 MB at 196.45 MB/s)
2024-05-06T16:46:29+02:00: wrote 1258 chunks (4296.54 MB at 194.41 MB/s)
2024-05-06T16:46:51+02:00: wrote 1181 chunks (4297.59 MB at 197.96 MB/s)
2024-05-06T16:47:06+02:00: wrote 1193 chunks (4298.11 MB at 284.05 MB/s)
2024-05-06T16:47:25+02:00: wrote 1292 chunks (4296.80 MB at 237.62 MB/s)
2024-05-06T16:47:43+02:00: wrote 1331 chunks (4297.33 MB at 239.11 MB/s)
2024-05-06T16:48:01+02:00: wrote 1267 chunks (4295.75 MB at 240.12 MB/s)
2024-05-06T16:48:22+02:00: wrote 1131 chunks (4297.06 MB at 214.19 MB/s)
2024-05-06T16:48:41+02:00: wrote 1237 chunks (4297.85 MB at 232.62 MB/s)
2024-05-06T16:49:07+02:00: wrote 1520 chunks (4296.02 MB at 201.01 MB/s)
2024-05-06T16:49:30+02:00: wrote 1458 chunks (4297.06 MB at 205.10 MB/s)
2024-05-06T16:50:00+02:00: wrote 1668 chunks (4295.75 MB at 170.34 MB/s)
2024-05-06T16:50:04+02:00: wrote 361 chunks (374.34 MB at 163.63 MB/s)
2024-05-06T16:50:11+02:00: end backup px11:"vm/10501/2024-05-05T20:33:12Z"
2024-05-06T16:50:11+02:00: percentage done: 27.61% (37/134 groups)
2024-05-06T16:50:11+02:00: backup snapshot "vm/10600/2024-05-05T21:15:42Z"
2024-05-06T16:50:34+02:00: wrote 1637 chunks (4298.38 MB at 294.39 MB/s)
2024-05-06T16:50:52+02:00: wrote 1158 chunks (4298.38 MB at 255.39 MB/s)
2024-05-06T16:51:10+02:00: wrote 1126 chunks (4298.64 MB at 255.59 MB/s)
2024-05-06T16:51:35+02:00: wrote 1100 chunks (4296.02 MB at 189.63 MB/s)
2024-05-06T16:51:53+02:00: wrote 1079 chunks (4298.64 MB at 255.55 MB/s)
2024-05-06T16:52:11+02:00: wrote 1105 chunks (4298.90 MB at 255.46 MB/s)
2024-05-06T16:52:30+02:00: wrote 1291 chunks (4296.28 MB at 255.59 MB/s)
2024-05-06T16:52:48+02:00: wrote 1593 chunks (4297.33 MB at 255.96 MB/s)
2024-05-06T16:53:07+02:00: wrote 1261 chunks (4295.23 MB at 256.40 MB/s)
2024-05-06T16:53:25+02:00: wrote 1119 chunks (4296.02 MB at 256.38 MB/s)
2024-05-06T16:53:46+02:00: wrote 1098 chunks (4297.33 MB at 221.40 MB/s)

Thats the Log of the Job currently running..

ZPOOL Layout Config:

Code:

    NAME                                           STATE     READ WRITE CKSUM
    datastore                                      ONLINE       0     0     0
      raidz3-0                                     ONLINE       0     0     0
        wwn-0x5000c500daa82a33                     ONLINE       0     0     0
        wwn-0x5000c500dad69eef                     ONLINE       0     0     0
        wwn-0x5000c500dad57c03                     ONLINE       0     0     0
        wwn-0x5000c500daba855f                     ONLINE       0     0     0
        wwn-0x5000c500dae104f3                     ONLINE       0     0     0
        wwn-0x5000c500dadcb543                     ONLINE       0     0     0
        wwn-0x5000c500dae034eb                     ONLINE       0     0     0
        wwn-0x5000c500d7f8276f                     ONLINE       0     0     0
        wwn-0x5000c500dab30c7b                     ONLINE       0     0     0
        wwn-0x5000c500daaf99df                     ONLINE       0     0     0
        wwn-0x5000c500dae0fbeb                     ONLINE       0     0     0
        wwn-0x5000c500daa8d887                     ONLINE       0     0     0
    special
      mirror-1                                     ONLINE       0     0     0
        nvme-eui.01000000000000008ce38ee30052f175  ONLINE       0     0     0
        nvme-eui.01000000000000008ce38ee30052f134  ONLINE       0     0     0
        nvme-eui.000000000000000100a07523417d6ced  ONLINE       0     0     0
        nvme-eui.000000000000000100a07523417d6ce4  ONLINE       0     0     0

atop while job is running:

Interessting tough: Full speed is there for the first few minutes after starting the job, after that read/write starts to decline and stays at a lower level..

//Edit:

Another interessting observation: After the initial decline it seemt to ceil at ~255MB/s. Coincidence?

Code:

024-05-06T18:44:41+02:00: moving to end of media
2024-05-06T18:44:41+02:00: arrived at end of media
2024-05-06T18:44:57+02:00: wrote 1039 chunks (4296.80 MB at 272.55 MB/s)
2024-05-06T18:45:11+02:00: wrote 1035 chunks (4298.64 MB at 302.98 MB/s)
2024-05-06T18:45:26+02:00: wrote 1034 chunks (4298.38 MB at 303.54 MB/s)
2024-05-06T18:45:42+02:00: wrote 1034 chunks (4298.38 MB at 255.64 MB/s)
2024-05-06T18:45:57+02:00: wrote 1034 chunks (4297.59 MB at 302.77 MB/s)
2024-05-06T18:46:11+02:00: wrote 1032 chunks (4295.75 MB at 304.10 MB/s)
2024-05-06T18:46:25+02:00: wrote 1035 chunks (4296.54 MB at 302.78 MB/s)
2024-05-06T18:46:39+02:00: wrote 1039 chunks (4298.64 MB at 303.58 MB/s)
2024-05-06T18:46:53+02:00: wrote 1118 chunks (4297.33 MB at 303.53 MB/s)
2024-05-06T18:47:07+02:00: wrote 1034 chunks (4296.28 MB at 303.47 MB/s)
2024-05-06T18:47:21+02:00: wrote 1051 chunks (4295.23 MB at 303.30 MB/s)
2024-05-06T18:47:36+02:00: wrote 1062 chunks (4296.80 MB at 303.49 MB/s)
2024-05-06T18:47:50+02:00: wrote 1040 chunks (4297.06 MB at 303.55 MB/s)
2024-05-06T18:48:04+02:00: wrote 1033 chunks (4296.28 MB at 302.81 MB/s)
2024-05-06T18:48:18+02:00: wrote 1031 chunks (4295.75 MB at 303.35 MB/s)
2024-05-06T18:48:32+02:00: wrote 1034 chunks (4296.02 MB at 303.50 MB/s)
2024-05-06T18:48:47+02:00: wrote 1036 chunks (4295.49 MB at 302.67 MB/s)
2024-05-06T18:49:01+02:00: wrote 1034 chunks (4296.28 MB at 303.47 MB/s)
2024-05-06T18:49:15+02:00: wrote 1034 chunks (4298.38 MB at 302.95 MB/s)
2024-05-06T18:49:32+02:00: wrote 1035 chunks (4298.11 MB at 255.11 MB/s)
2024-05-06T18:49:46+02:00: wrote 1034 chunks (4297.33 MB at 302.80 MB/s)
2024-05-06T18:50:00+02:00: wrote 1063 chunks (4296.28 MB at 303.27 MB/s)
2024-05-06T18:50:14+02:00: wrote 1066 chunks (4296.28 MB at 303.39 MB/s)
2024-05-06T18:50:28+02:00: wrote 1061 chunks (4297.06 MB at 303.44 MB/s)
2024-05-06T18:50:43+02:00: wrote 1045 chunks (4298.38 MB at 302.88 MB/s)
2024-05-06T18:50:57+02:00: wrote 1038 chunks (4295.23 MB at 303.31 MB/s)
2024-05-06T18:51:11+02:00: wrote 1033 chunks (4296.54 MB at 303.30 MB/s)
2024-05-06T18:51:25+02:00: wrote 1051 chunks (4295.23 MB at 303.26 MB/s)
2024-05-06T18:51:39+02:00: wrote 1054 chunks (4295.23 MB at 303.21 MB/s)
2024-05-06T18:51:53+02:00: wrote 1083 chunks (4296.02 MB at 303.22 MB/s)
2024-05-06T18:52:08+02:00: wrote 1057 chunks (4296.54 MB at 303.40 MB/s)
2024-05-06T18:52:22+02:00: wrote 1041 chunks (4296.54 MB at 302.68 MB/s)
2024-05-06T18:52:40+02:00: wrote 1036 chunks (4299.16 MB at 256.18 MB/s)
2024-05-06T18:52:59+02:00: wrote 1035 chunks (4296.80 MB at 255.92 MB/s)
2024-05-06T18:53:20+02:00: wrote 1035 chunks (4295.49 MB at 220.98 MB/s)
2024-05-06T18:53:38+02:00: wrote 1045 chunks (4298.64 MB at 255.97 MB/s)
2024-05-06T18:53:56+02:00: wrote 1035 chunks (4298.38 MB at 255.99 MB/s)
2024-05-06T18:54:15+02:00: wrote 1034 chunks (4297.33 MB at 256.12 MB/s)
2024-05-06T18:54:33+02:00: wrote 1035 chunks (4296.80 MB at 255.95 MB/s)
2024-05-06T18:54:51+02:00: wrote 1038 chunks (4295.23 MB at 255.94 MB/s)
2024-05-06T18:55:10+02:00: wrote 1035 chunks (4298.64 MB at 256.07 MB/s)
2024-05-06T18:55:28+02:00: wrote 1038 chunks (4298.11 MB at 255.75 MB/s)
2024-05-06T18:55:47+02:00: wrote 1056 chunks (4295.75 MB at 255.76 MB/s)
2024-05-06T18:56:05+02:00: wrote 1060 chunks (4298.90 MB at 255.57 MB/s)
2024-05-06T18:56:23+02:00: wrote 1055 chunks (4295.23 MB at 255.26 MB/s)
2024-05-06T18:56:42+02:00: wrote 1035 chunks (4296.02 MB at 255.55 MB/s)
2024-05-06T18:57:00+02:00: wrote 1033 chunks (4298.38 MB at 255.71 MB/s)
2024-05-06T18:57:19+02:00: wrote 1036 chunks (4298.64 MB at 255.37 MB/s)
2024-05-06T18:57:37+02:00: wrote 1037 chunks (4296.28 MB at 255.11 MB/s)
2024-05-06T18:57:58+02:00: wrote 1107 chunks (4296.28 MB at 219.94 MB/s)
2024-05-06T18:58:17+02:00: wrote 1042 chunks (4299.16 MB at 250.43 MB/s)
2024-05-06T18:58:35+02:00: wrote 1036 chunks (4296.28 MB at 255.43 MB/s)
2024-05-06T18:58:54+02:00: wrote 1031 chunks (4295.49 MB at 255.32 MB/s)
2024-05-06T18:59:12+02:00: wrote 1033 chunks (4298.11 MB at 254.72 MB/s)
2024-05-06T18:59:31+02:00: wrote 1032 chunks (4296.54 MB at 255.08 MB/s)
2024-05-06T18:59:49+02:00: wrote 1035 chunks (4299.16 MB at 255.51 MB/s)
2024-05-06T19:00:07+02:00: wrote 1044 chunks (4297.33 MB at 255.38 MB/s)

Lephisto · May 6, 2024

Another observation:

This is how it looks like in the Datastore graph - only the Tape backup job is running, but it constantly shows ~255MB/s:

vs:

Code:

2024-05-06T20:01:25+02:00: backup snapshot "vm/11222/2024-05-05T20:35:05Z"
2024-05-06T20:02:05+02:00: wrote 7322 chunks (4295.75 MB at 183.45 MB/s)
2024-05-06T20:02:24+02:00: wrote 13521 chunks (4295.49 MB at 264.32 MB/s)
2024-05-06T20:02:42+02:00: wrote 6795 chunks (4298.38 MB at 257.03 MB/s)
2024-05-06T20:03:03+02:00: wrote 5725 chunks (4296.28 MB at 222.06 MB/s)
2024-05-06T20:03:21+02:00: wrote 9569 chunks (4297.85 MB at 256.77 MB/s)
2024-05-06T20:03:40+02:00: wrote 8901 chunks (4295.75 MB at 256.67 MB/s)
2024-05-06T20:04:01+02:00: wrote 5037 chunks (4295.23 MB at 218.36 MB/s)
2024-05-06T20:04:37+02:00: wrote 8325 chunks (4296.28 MB at 129.19 MB/s)
2024-05-06T20:05:09+02:00: wrote 4828 chunks (4295.23 MB at 162.62 MB/s)
2024-05-06T20:05:42+02:00: wrote 7502 chunks (4295.23 MB at 141.38 MB/s)
2024-05-06T20:06:10+02:00: wrote 7738 chunks (4296.28 MB at 164.96 MB/s)
2024-05-06T20:06:44+02:00: wrote 10420 chunks (4297.06 MB at 127.16 MB/s)
2024-05-06T20:07:13+02:00: wrote 7108 chunks (4295.49 MB at 169.05 MB/s)
2024-05-06T20:07:31+02:00: wrote 2504 chunks (4295.49 MB at 256.61 MB/s)
2024-05-06T20:08:00+02:00: wrote 5025 chunks (4295.23 MB at 154.13 MB/s)
2024-05-06T20:08:44+02:00: wrote 11017 chunks (4295.49 MB at 102.12 MB/s)
2024-05-06T20:10:00+02:00: wrote 2959 chunks (4298.11 MB at 59.74 MB/s)
2024-05-06T20:10:20+02:00: wrote 1327 chunks (4296.02 MB at 255.63 MB/s)
2024-05-06T20:10:41+02:00: wrote 1329 chunks (4295.75 MB at 220.82 MB/s)
2024-05-06T20:11:00+02:00: wrote 1224 chunks (4299.16 MB at 255.69 MB/s)
2024-05-06T20:11:19+02:00: wrote 1149 chunks (4296.02 MB at 248.78 MB/s)
2024-05-06T20:11:37+02:00: wrote 1100 chunks (4298.90 MB at 255.86 MB/s)
2024-05-06T20:11:55+02:00: wrote 1175 chunks (4295.49 MB at 253.05 MB/s)
2024-05-06T20:12:13+02:00: wrote 1118 chunks (4297.33 MB at 256.47 MB/s)
2024-05-06T20:12:32+02:00: wrote 1452 chunks (4297.85 MB at 254.14 MB/s)
2024-05-06T20:12:50+02:00: wrote 1192 chunks (4296.80 MB at 255.89 MB/s)
2024-05-06T20:13:10+02:00: wrote 1348 chunks (4298.90 MB at 236.50 MB/s)
2024-05-06T20:13:35+02:00: wrote 1154 chunks (4297.06 MB at 171.22 MB/s)
2024-05-06T20:13:53+02:00: wrote 1326 chunks (4295.75 MB at 255.34 MB/s)
2024-05-06T20:14:12+02:00: wrote 1343 chunks (4298.64 MB at 256.01 MB/s)
2024-05-06T20:14:31+02:00: wrote 1480 chunks (4298.38 MB at 241.61 MB/s)
2024-05-06T20:14:49+02:00: wrote 1218 chunks (4297.59 MB at 255.85 MB/s)
2024-05-06T20:15:07+02:00: wrote 1205 chunks (4297.06 MB at 256.81 MB/s)
2024-05-06T20:15:25+02:00: wrote 1191 chunks (4296.02 MB at 256.46 MB/s)
2024-05-06T20:15:46+02:00: wrote 1196 chunks (4295.49 MB at 221.70 MB/s)
2024-05-06T20:16:05+02:00: wrote 1181 chunks (4296.02 MB at 256.65 MB/s)
2024-05-06T20:16:23+02:00: wrote 1299 chunks (4295.75 MB at 256.77 MB/s)
2024-05-06T20:16:41+02:00: wrote 1198 chunks (4295.75 MB at 256.65 MB/s)
2024-05-06T20:17:00+02:00: wrote 1147 chunks (4299.16 MB at 256.10 MB/s)
2024-05-06T20:17:18+02:00: wrote 1115 chunks (4296.02 MB at 255.87 MB/s)
2024-05-06T20:17:36+02:00: wrote 1232 chunks (4296.80 MB at 256.28 MB/s)
2024-05-06T20:17:55+02:00: wrote 1341 chunks (4295.75 MB at 256.40 MB/s)
2024-05-06T20:18:13+02:00: wrote 1362 chunks (4298.90 MB at 256.40 MB/s)
2024-05-06T20:18:31+02:00: wrote 1266 chunks (4296.54 MB at 256.44 MB/s)
2024-05-06T20:18:50+02:00: wrote 1330 chunks (4296.28 MB at 256.31 MB/s)
2024-05-06T20:19:08+02:00: wrote 1252 chunks (4296.02 MB at 255.89 MB/s)
2024-05-06T20:19:26+02:00: wrote 1162 chunks (4296.28 MB at 256.08 MB/s)
2024-05-06T20:19:45+02:00: wrote 1133 chunks (4295.23 MB at 255.85 MB/s)
2024-05-06T20:20:03+02:00: wrote 1131 chunks (4298.64 MB at 256.03 MB/s)
2024-05-06T20:20:24+02:00: wrote 1090 chunks (4297.85 MB at 220.54 MB/s)
2024-05-06T20:20:43+02:00: wrote 1119 chunks (4296.28 MB at 255.28 MB/s)
2024-05-06T20:21:01+02:00: wrote 1313 chunks (4298.38 MB at 255.44 MB/s)
2024-05-06T20:21:19+02:00: wrote 1233 chunks (4295.49 MB at 255.45 MB/s)
2024-05-06T20:21:38+02:00: wrote 1262 chunks (4295.75 MB at 255.80 MB/s)
2024-05-06T20:21:56+02:00: wrote 1266 chunks (4298.64 MB at 255.06 MB/s)
2024-05-06T20:22:15+02:00: wrote 1212 chunks (4296.54 MB at 254.94 MB/s)
2024-05-06T20:22:33+02:00: wrote 1130 chunks (4295.49 MB at 255.37 MB/s)
2024-05-06T20:22:52+02:00: wrote 1198 chunks (4296.80 MB at 255.45 MB/s)
2024-05-06T20:23:10+02:00: wrote 1205 chunks (4295.75 MB at 255.63 MB/s)
2024-05-06T20:23:28+02:00: wrote 1178 chunks (4298.64 MB at 256.00 MB/s)
2024-05-06T20:23:47+02:00: wrote 1103 chunks (4298.64 MB at 255.21 MB/s)
2024-05-06T20:24:05+02:00: wrote 1151 chunks (4296.02 MB at 255.63 MB/s)
2024-05-06T20:24:24+02:00: wrote 1191 chunks (4297.33 MB at 255.28 MB/s)
2024-05-06T20:24:42+02:00: wrote 1172 chunks (4297.85 MB at 255.70 MB/s)
2024-05-06T20:25:00+02:00: wrote 1194 chunks (4297.85 MB at 255.95 MB/s)
2024-05-06T20:25:21+02:00: wrote 1255 chunks (4297.06 MB at 221.19 MB/s)
2024-05-06T20:25:40+02:00: wrote 1260 chunks (4298.38 MB at 256.19 MB/s)
2024-05-06T20:25:58+02:00: wrote 1304 chunks (4297.06 MB at 256.44 MB/s)
2024-05-06T20:26:16+02:00: wrote 1198 chunks (4295.75 MB at 256.14 MB/s)
2024-05-06T20:26:35+02:00: wrote 1185 chunks (4295.75 MB at 256.55 MB/s)
2024-05-06T20:26:53+02:00: wrote 1151 chunks (4295.75 MB at 256.69 MB/s)
2024-05-06T20:27:11+02:00: wrote 1158 chunks (4296.54 MB at 256.77 MB/s)
2024-05-06T20:27:30+02:00: wrote 1183 chunks (4298.64 MB at 256.48 MB/s)
2024-05-06T20:27:48+02:00: wrote 1250 chunks (4296.28 MB at 256.42 MB/s)
2024-05-06T20:28:06+02:00: wrote 1245 chunks (4295.23 MB at 256.28 MB/s)
2024-05-06T20:28:25+02:00: wrote 1266 chunks (4298.64 MB at 256.23 MB/s)
2024-05-06T20:28:43+02:00: wrote 1413 chunks (4295.75 MB at 256.24 MB/s)
2024-05-06T20:29:01+02:00: wrote 1511 chunks (4295.23 MB at 255.90 MB/s)
2024-05-06T20:29:20+02:00: wrote 1266 chunks (4298.64 MB at 255.37 MB/s)
2024-05-06T20:29:38+02:00: wrote 1323 chunks (4298.11 MB at 255.44 MB/s)
2024-05-06T20:30:10+02:00: wrote 1198 chunks (4297.06 MB at 142.94 MB/s)
2024-05-06T20:30:28+02:00: wrote 1309 chunks (4296.54 MB at 255.09 MB/s)
2024-05-06T20:30:47+02:00: wrote 1287 chunks (4296.28 MB at 255.67 MB/s)
2024-05-06T20:31:05+02:00: wrote 1220 chunks (4299.16 MB at 256.13 MB/s)
2024-05-06T20:31:23+02:00: wrote 1130 chunks (4297.06 MB at 256.14 MB/s)
2024-05-06T20:31:42+02:00: wrote 1155 chunks (4296.02 MB at 256.32 MB/s)
2024-05-06T20:32:00+02:00: wrote 1209 chunks (4296.80 MB at 256.02 MB/s)
2024-05-06T20:32:19+02:00: wrote 1269 chunks (4296.80 MB at 256.13 MB/s)
2024-05-06T20:32:37+02:00: wrote 1234 chunks (4298.64 MB at 256.15 MB/s)
2024-05-06T20:32:55+02:00: wrote 1170 chunks (4296.02 MB at 256.74 MB/s)
2024-05-06T20:33:14+02:00: wrote 1172 chunks (4296.02 MB at 256.37 MB/s)
2024-05-06T20:33:32+02:00: wrote 1146 chunks (4297.06 MB at 256.18 MB/s)
2024-05-06T20:33:50+02:00: wrote 1138 chunks (4297.59 MB at 256.43 MB/s)
2024-05-06T20:34:09+02:00: wrote 1069 chunks (4296.54 MB at 255.73 MB/s)
2024-05-06T20:34:27+02:00: wrote 1162 chunks (4296.02 MB at 256.36 MB/s)
2024-05-06T20:34:45+02:00: wrote 1186 chunks (4297.59 MB at 256.49 MB/s)
2024-05-06T20:35:07+02:00: wrote 1198 chunks (4295.23 MB at 269.41 MB/s)
2024-05-06T20:35:25+02:00: wrote 1383 chunks (4296.54 MB at 256.30 MB/s)
2024-05-06T20:35:43+02:00: wrote 1311 chunks (4298.11 MB at 255.80 MB/s)
2024-05-06T20:36:02+02:00: wrote 1261 chunks (4295.49 MB at 256.00 MB/s)
2024-05-06T20:36:20+02:00: wrote 1248 chunks (4296.02 MB at 255.53 MB/s)
2024-05-06T20:36:39+02:00: wrote 1376 chunks (4297.33 MB at 256.42 MB/s)
2024-05-06T20:36:57+02:00: wrote 1387 chunks (4295.49 MB at 256.36 MB/s)
2024-05-06T20:37:15+02:00: wrote 1493 chunks (4296.54 MB at 256.22 MB/s)
2024-05-06T20:37:34+02:00: wrote 1401 chunks (4298.11 MB at 256.09 MB/s)
2024-05-06T20:37:52+02:00: wrote 1250 chunks (4297.85 MB at 256.34 MB/s)
2024-05-06T20:38:10+02:00: wrote 1134 chunks (4298.38 MB at 256.01 MB/s)
2024-05-06T20:38:29+02:00: wrote 1164 chunks (4296.28 MB at 256.39 MB/s)
2024-05-06T20:38:47+02:00: wrote 1173 chunks (4297.06 MB at 257.14 MB/s)
2024-05-06T20:39:05+02:00: wrote 1269 chunks (4297.59 MB at 255.86 MB/s)
2024-05-06T20:39:24+02:00: wrote 1216 chunks (4297.06 MB at 255.83 MB/s)
2024-05-06T20:39:45+02:00: wrote 1213 chunks (4295.23 MB at 218.73 MB/s)
2024-05-06T20:40:03+02:00: wrote 1134 chunks (4298.11 MB at 255.12 MB/s)
2024-05-06T20:40:22+02:00: wrote 1104 chunks (4296.28 MB at 255.66 MB/s)
2024-05-06T20:40:40+02:00: wrote 1165 chunks (4295.23 MB at 255.36 MB/s)
2024-05-06T20:40:59+02:00: wrote 1464 chunks (4295.23 MB at 255.71 MB/s)
2024-05-06T20:41:17+02:00: wrote 1277 chunks (4297.06 MB at 255.46 MB/s)
2024-05-06T20:41:35+02:00: wrote 1170 chunks (4296.80 MB at 256.21 MB/s)
2024-05-06T20:41:54+02:00: wrote 1158 chunks (4297.85 MB at 255.72 MB/s)
2024-05-06T20:42:12+02:00: wrote 1791 chunks (4298.64 MB at 256.26 MB/s)
2024-05-06T20:42:30+02:00: wrote 1025 chunks (4296.54 MB at 255.83 MB/s)
2024-05-06T20:42:49+02:00: wrote 1025 chunks (4295.23 MB at 255.68 MB/s)
2024-05-06T20:43:07+02:00: wrote 1070 chunks (4298.90 MB at 255.80 MB/s)
2024-05-06T20:43:26+02:00: wrote 1126 chunks (4298.38 MB at 256.31 MB/s)
2024-05-06T20:43:44+02:00: wrote 1352 chunks (4296.54 MB at 255.99 MB/s)
2024-05-06T20:44:02+02:00: wrote 1210 chunks (4296.54 MB at 256.31 MB/s)
2024-05-06T20:44:23+02:00: wrote 1298 chunks (4297.33 MB at 221.38 MB/s)
2024-05-06T20:44:42+02:00: wrote 1163 chunks (4298.90 MB at 256.30 MB/s)
2024-05-06T20:45:00+02:00: wrote 1098 chunks (4299.16 MB at 256.09 MB/s)
2024-05-06T20:45:18+02:00: wrote 1179 chunks (4296.28 MB at 255.88 MB/s)
2024-05-06T20:45:37+02:00: wrote 1212 chunks (4295.49 MB at 256.48 MB/s)
2024-05-06T20:45:55+02:00: wrote 1118 chunks (4298.11 MB at 256.02 MB/s)
2024-05-06T20:46:13+02:00: wrote 1027 chunks (4298.38 MB at 256.04 MB/s)
2024-05-06T20:46:32+02:00: wrote 1132 chunks (4297.06 MB at 255.70 MB/s)
2024-05-06T20:46:50+02:00: wrote 1029 chunks (4295.75 MB at 255.57 MB/s)
2024-05-06T20:47:09+02:00: wrote 1038 chunks (4296.54 MB at 255.51 MB/s)
2024-05-06T20:47:27+02:00: wrote 1028 chunks (4299.16 MB at 255.83 MB/s)
2024-05-06T20:47:45+02:00: wrote 1025 chunks (4298.11 MB at 255.66 MB/s)
2024-05-06T20:48:04+02:00: wrote 1111 chunks (4296.28 MB at 255.47 MB/s)
2024-05-06T20:48:22+02:00: wrote 1367 chunks (4295.49 MB at 255.82 MB/s)
2024-05-06T20:48:41+02:00: wrote 1298 chunks (4297.33 MB at 255.96 MB/s)
2024-05-06T20:49:04+02:00: wrote 1562 chunks (4295.23 MB at 198.23 MB/s)
2024-05-06T20:49:25+02:00: wrote 1351 chunks (4295.23 MB at 221.27 MB/s)
2024-05-06T20:49:43+02:00: wrote 1337 chunks (4296.80 MB at 255.44 MB/s)
2024-05-06T20:50:02+02:00: wrote 1308 chunks (4295.75 MB at 255.44 MB/s)
2024-05-06T20:50:20+02:00: wrote 1285 chunks (4295.75 MB at 255.33 MB/s)
2024-05-06T20:50:39+02:00: wrote 1392 chunks (4295.75 MB at 253.24 MB/s)

Might this be a speed calculation error?

regards.

dcsapak · May 7, 2024

i now sent a v2 of my series: https://lists.proxmox.com/pipermail/pbs-devel/2024-May/009188.html
note that the option on the datastore changed name: from 'read-threads' to 'tape-backup-read-threads'
but aside from that there hasn't changed anything significantly for the tape backup code

curiously i can reproduce the throttling behaviour (on a similar setup that you have). I'll look into it, but I fear that it's something out of our control (e.g. thermal limits, drive limits, cache, etc.) since there shouldn't be any change in behaviour based on time (the code that writes the chunks to tape basically does that only in a loop, i don't see how an additional amount of time could make an impact here code-wise)

Lephisto · May 7, 2024

What type of drive do you use?

dcsapak · May 7, 2024

the one in here: https://www.quantum.com/en/products/tape-storage/superloader-3/

one LTO-8 HH and one LTO-9 HH

Lephisto · May 7, 2024

I notice one thing, which is not totally logical:

Code:

2024-05-07T13:32:18+02:00: wrote 1272 chunks (4299.95 MB at 256.08 MB/s)
2024-05-07T13:32:37+02:00: wrote 1211 chunks (4305.72 MB at 255.80 MB/s) => 4305MB in 19s: 226,578947368 MB
2024-05-07T13:32:55+02:00: wrote 1490 chunks (4295.23 MB at 256.10 MB/s) => 4295MB in 18s: 238,611111111 MB
2024-05-07T13:33:13+02:00: wrote 1548 chunks (4297.06 MB at 256.09 MB/s)

I just picked two examples: This line doesn't make sense: 4305MB in 19s wouldn't be 255.8 MB. 1211*4 aren't 4305MB assuming all chunks are 4MB of size. Something is not totally right here

dcsapak · May 7, 2024

Lephisto said:
I notice one thing, which is not totally logical:

well between those two log files happens a bit more than just the writes, we currently do a few security checks in between the chunk archives (what each line represents)
e.g. checking if the tape is nearly full, we're still at the end of the tape etc.

but i admit it should not amount to over a second, i'll see if we can reduce that, but that has nothing do to with the actual write speed
which is what we measure (from shortly before writing, to shortly after) and that does not change magically after 10 minutes

in the meantime i have written a short tool that basically writes only from memory without any actions in between and there i get ~300MB/s without the drop to ~250MB/s after ~9-10 minutes...
i'll test further to see what the problem here is, but thanks for testing

Lephisto · May 7, 2024

Okay, so we can already rule out the Drive to not be able to sustain the specified 300MB/s.. so the bottleneck must either be the ZFS, or the Code that's pulling the data off.

dcsapak · May 7, 2024

ok thanks to you i looked a little closer on what exactly is happening between those write messages. I noticed that after a certain amount of time there was a gap of ~2 seconds, which would be enough to slow the data flow so that the drive had to slow down.
it turns out, we accidentally synced/wrote to disk on every archive (chunk archive and snapshot archive) after 128 GiB were written on tape (instead of doing that *every 128GiB once*)

i sent a fix for that: https://lists.proxmox.com/pipermail/pbs-devel/2024-May/009192.html

in my tests i still had slowdowns sometimes (but that's probably due to disk and drive limitations, e.g. when it has to reverse it's direction) but it continued on with ~300MB/s after that
since it seems you already have the setup to build your own packages/binaries, would you mind testing this too (in combination with the other patches to retain the high throughput) ?

Lephisto · May 7, 2024

.. build running. It just takes ages.

I have v1 of your series + the fix. will that work?

dcsapak · May 7, 2024

Lephisto said:
.. build running. It just takes ages.

I have v1 of your series + the fix. will that work?

yes that should still work

Lephisto · May 7, 2024

We're getting somewhere:

Code:

2024-05-07T16:56:14+02:00: Starting tape backup job 'zfs:cephfs-mai:lto9:cephfs'
2024-05-07T16:56:14+02:00: update media online status
2024-05-07T16:56:16+02:00: media set uuid: c3d64c07-c811-48f0-9845-086cead14e55
2024-05-07T16:56:16+02:00: found 9 groups (out of 9 total)
2024-05-07T16:56:16+02:00: latest-only: true (only considering latest snapshots)
2024-05-07T16:56:16+02:00: backup snapshot "ns/cephfs1/host/cephfs1n1/2024-05-03T08:30:01Z"
2024-05-07T16:56:16+02:00: allocated new writable media 'AAO809L9'
2024-05-07T16:56:16+02:00: trying to load media 'AAO809L9' into drive 'lto9'
2024-05-07T16:56:17+02:00: found media label AAO809L9 (598019aa-a771-4cf0-956b-42e47b695ef0)
2024-05-07T16:56:17+02:00: writing new media set label
2024-05-07T16:56:28+02:00: moving to end of media
2024-05-07T16:56:30+02:00: arrived at end of media
2024-05-07T16:57:07+02:00: wrote 1588 chunks (4296.02 MB at 116.27 MB/s)
2024-05-07T16:57:21+02:00: wrote 1422 chunks (4297.06 MB at 303.82 MB/s)
2024-05-07T16:57:35+02:00: wrote 1450 chunks (4296.02 MB at 302.19 MB/s)
2024-05-07T16:57:50+02:00: wrote 1469 chunks (4297.59 MB at 298.30 MB/s)
2024-05-07T16:58:04+02:00: wrote 1486 chunks (4299.95 MB at 296.36 MB/s)
2024-05-07T16:58:19+02:00: wrote 1594 chunks (4296.54 MB at 290.92 MB/s)
2024-05-07T16:58:36+02:00: wrote 1738 chunks (4296.54 MB at 252.95 MB/s)
2024-05-07T16:58:50+02:00: wrote 1777 chunks (4296.02 MB at 302.44 MB/s)
2024-05-07T16:59:06+02:00: wrote 2058 chunks (4295.75 MB at 280.02 MB/s)
2024-05-07T16:59:23+02:00: wrote 2938 chunks (4296.28 MB at 254.08 MB/s)
2024-05-07T16:59:37+02:00: wrote 2566 chunks (4295.23 MB at 288.55 MB/s)
2024-05-07T17:00:00+02:00: wrote 2428 chunks (4297.59 MB at 194.97 MB/s)
2024-05-07T17:00:14+02:00: wrote 2131 chunks (4296.28 MB at 287.30 MB/s)
2024-05-07T17:00:35+02:00: wrote 2347 chunks (4297.59 MB at 204.64 MB/s)
2024-05-07T17:00:51+02:00: wrote 2428 chunks (4295.75 MB at 284.96 MB/s)
2024-05-07T17:01:10+02:00: wrote 3094 chunks (4296.02 MB at 215.88 MB/s)
2024-05-07T17:01:26+02:00: wrote 1606 chunks (4301.00 MB at 281.77 MB/s)
2024-05-07T17:01:47+02:00: wrote 1721 chunks (4297.85 MB at 198.77 MB/s)
2024-05-07T17:02:02+02:00: wrote 1905 chunks (4297.33 MB at 288.59 MB/s)
2024-05-07T17:02:15+02:00: wrote 2086 chunks (4295.49 MB at 326.48 MB/s)
2024-05-07T17:02:30+02:00: wrote 1564 chunks (4297.06 MB at 303.65 MB/s)
2024-05-07T17:02:44+02:00: wrote 1658 chunks (4298.38 MB at 302.87 MB/s)
2024-05-07T17:02:58+02:00: wrote 1706 chunks (4301.78 MB at 304.80 MB/s)
2024-05-07T17:03:12+02:00: wrote 1825 chunks (4295.75 MB at 297.27 MB/s)
2024-05-07T17:03:26+02:00: wrote 1838 chunks (4298.64 MB at 310.33 MB/s)
2024-05-07T17:03:40+02:00: wrote 1846 chunks (4295.49 MB at 304.23 MB/s)
2024-05-07T17:03:54+02:00: wrote 1504 chunks (4303.88 MB at 304.14 MB/s)
2024-05-07T17:04:09+02:00: wrote 1489 chunks (4296.28 MB at 303.59 MB/s)
2024-05-07T17:04:23+02:00: wrote 1569 chunks (4295.23 MB at 303.49 MB/s)
2024-05-07T17:04:37+02:00: wrote 1630 chunks (4295.49 MB at 304.24 MB/s)
2024-05-07T17:04:51+02:00: wrote 1585 chunks (4296.02 MB at 303.50 MB/s)
2024-05-07T17:05:08+02:00: wrote 1544 chunks (4298.38 MB at 255.88 MB/s)
2024-05-07T17:05:26+02:00: wrote 1437 chunks (4295.23 MB at 255.41 MB/s)
2024-05-07T17:05:40+02:00: wrote 1448 chunks (4296.02 MB at 303.58 MB/s)
2024-05-07T17:05:55+02:00: wrote 1363 chunks (4295.75 MB at 303.44 MB/s)
2024-05-07T17:06:09+02:00: wrote 1359 chunks (4296.80 MB at 303.62 MB/s)
2024-05-07T17:06:23+02:00: wrote 1370 chunks (4296.02 MB at 304.17 MB/s)

At least it comes back to spec speed, I still would expect it to sustain 300MB/s.. I will keep you posted. thanks for the work.

regards.

Low ZFS read performance Disk->Tape

Proxmox Staff Member

Active Member

Active Member

Proxmox Staff Member

Active Member

Active Member

Active Member

Proxmox Staff Member

Active Member

Active Member

Proxmox Staff Member

Active Member

Proxmox Staff Member

Active Member

Proxmox Staff Member

Active Member

Proxmox Staff Member

Active Member

Proxmox Staff Member

Active Member