Proxmox Backup Server 3.4 released!
- Thread starter t.lamprecht
- Start date
Hi,
With PBS 3.4 I'm now testing restore from LTO-9 speed.
I first benchmarched using multiple
Read or write from our LTO-9 tape library (HPE MSL3040) is about 300 MByte/s per drive, we have three drives.
When I launched restore in parallel of three LTO-9 tapes - one per drive - to one datastore (sdc) I got a total output of about 1.6 TB/hour
When I launched restore in parallel of the same 3 tapes but this time first tape to first datastore (sdc), second tape to second datastore (sdd), third tape to third datastore (sde) I got a total output of about 2.5 TB/hour, so 56% higher.
tape-job now have a "worker-threads 8" option but there is no equivalent option for tape-restore jobs AFAIK, may be it would be useful and gain some LTO-9 to datastore write performance?
Thanks!
PS: for the record sdc sdd sde are ext4 formatted and the underlying storage for each is an ec42 ceph pool (56 SSD over 7 hosts, 4x10Gbit/s networking)
With PBS 3.4 I'm now testing restore from LTO-9 speed.
I first benchmarched using multiple
cat /dev/urandom > /mnt/datastore/xxx/yyy our three datastore (sdc, sdd, sde) in parallel, write speed results were from iostat -xmt 10 sdc sdd sde about 2 Gbyte/s (> 7 TB/hour) :
Code:
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdc 0.00 0.00 0.00 0.00 0.00 0.00 592.20 669.58 15.30 2.52 41.76 1157.80 0.00 0.00 0.00 0.00 0.00 0.00 0.40 47.25 24.75 76.52
sdd 0.00 0.00 0.00 0.00 0.00 0.00 540.10 610.81 11.80 2.14 48.51 1158.06 0.00 0.00 0.00 0.00 0.00 0.00 0.40 36.75 26.22 70.61
sde 0.00 0.00 0.00 0.00 0.00 0.00 683.60 716.32 16.40 2.34 42.44 1073.01 0.00 0.00 0.00 0.00 0.00 0.00 0.40 39.00 29.03 51.50Read or write from our LTO-9 tape library (HPE MSL3040) is about 300 MByte/s per drive, we have three drives.
Code:
2025-04-22T09:44:38+02:00: wrote 1440 chunks (4298.38 MB at 305.10 MB/s)
2025-04-22T09:44:52+02:00: wrote 1343 chunks (4296.80 MB at 305.20 MB/s)
2025-04-22T09:45:06+02:00: wrote 1170 chunks (4296.54 MB at 304.68 MB/s)
2025-04-22T09:45:20+02:00: wrote 1256 chunks (4298.38 MB at 304.09 MB/s)
Code:
2025-04-20T14:02:58+02:00: register 1083 chunks
2025-04-20T14:02:58+02:00: File 201: chunk archive for datastore 'datastore1'
2025-04-20T14:03:12+02:00: restored 4.299 GB (304.04 MB/s)When I launched restore in parallel of three LTO-9 tapes - one per drive - to one datastore (sdc) I got a total output of about 1.6 TB/hour
When I launched restore in parallel of the same 3 tapes but this time first tape to first datastore (sdc), second tape to second datastore (sdd), third tape to third datastore (sde) I got a total output of about 2.5 TB/hour, so 56% higher.
tape-job now have a "worker-threads 8" option but there is no equivalent option for tape-restore jobs AFAIK, may be it would be useful and gain some LTO-9 to datastore write performance?
Thanks!
PS: for the record sdc sdd sde are ext4 formatted and the underlying storage for each is an ec42 ceph pool (56 SSD over 7 hosts, 4x10Gbit/s networking)
I did the update and now the GC is extremely slow, much more than it was before. Before the update, I used to run the GC task once a week and it would take about 2 days to complete. After the update, the GC has already taken 3 days and it's still only at 6% of Phase 2. This is really bad because it's killing PBSe's performance and I'm running out of space.
After Update:
The GC is still running and I can't paste the logs here because there are too many lines with the following messages:
My setup is ZFS with RAID6, my RAM is at 60% of 128GB and I don't use SWAP
After Update:
Code:
2025-05-03T05:00:00-03:00: starting garbage collection on store pbs002
2025-05-03T05:00:00-03:00: task triggered by schedule 'sat 05:00'
2025-05-03T05:00:00-03:00: Start GC phase1 (mark used chunks)
2025-05-03T05:31:28-03:00: marked 1% (33 of 3209 index files)
2025-05-03T06:09:41-03:00: marked 2% (65 of 3209 index files)
2025-05-03T06:28:11-03:00: marked 3% (97 of 3209 index files)
2025-05-03T06:47:07-03:00: marked 4% (129 of 3209 index files)
2025-05-03T07:02:02-03:00: marked 5% (161 of 3209 index files)
2025-05-03T07:50:50-03:00: marked 6% (193 of 3209 index files)
2025-05-03T08:11:54-03:00: marked 7% (225 of 3209 index files)
2025-05-03T08:18:13-03:00: marked 8% (257 of 3209 index files)
2025-05-03T08:24:52-03:00: marked 9% (289 of 3209 index files)
2025-05-03T08:28:12-03:00: marked 10% (321 of 3209 index files)
2025-05-03T08:39:08-03:00: marked 11% (353 of 3209 index files)
2025-05-03T08:46:39-03:00: marked 12% (386 of 3209 index files)
2025-05-03T09:07:20-03:00: marked 13% (418 of 3209 index files)
2025-05-03T09:09:45-03:00: marked 14% (450 of 3209 index files)
2025-05-03T09:17:46-03:00: marked 15% (482 of 3209 index files)
2025-05-03T09:30:36-03:00: marked 16% (514 of 3209 index files)
2025-05-03T09:41:01-03:00: marked 17% (546 of 3209 index files)
2025-05-03T09:43:49-03:00: marked 18% (578 of 3209 index files)
2025-05-03T09:46:12-03:00: marked 19% (610 of 3209 index files)
2025-05-03T09:47:23-03:00: marked 20% (642 of 3209 index files)
2025-05-03T09:54:42-03:00: marked 21% (674 of 3209 index files)
2025-05-03T10:00:07-03:00: marked 22% (706 of 3209 index files)
2025-05-03T10:05:15-03:00: marked 23% (739 of 3209 index files)
2025-05-03T10:20:17-03:00: marked 24% (771 of 3209 index files)
2025-05-03T10:30:31-03:00: marked 25% (803 of 3209 index files)
2025-05-03T10:52:06-03:00: marked 26% (835 of 3209 index files)
2025-05-03T11:02:28-03:00: marked 27% (867 of 3209 index files)
2025-05-03T11:26:51-03:00: marked 28% (899 of 3209 index files)
2025-05-03T12:30:42-03:00: marked 29% (931 of 3209 index files)
2025-05-03T12:38:35-03:00: marked 30% (963 of 3209 index files)
2025-05-03T13:05:10-03:00: marked 31% (995 of 3209 index files)
2025-05-03T13:07:32-03:00: marked 32% (1027 of 3209 index files)
2025-05-03T13:20:10-03:00: marked 33% (1059 of 3209 index files)
2025-05-03T13:27:36-03:00: marked 34% (1092 of 3209 index files)
2025-05-03T13:32:37-03:00: marked 35% (1124 of 3209 index files)
2025-05-03T13:36:10-03:00: marked 36% (1156 of 3209 index files)
2025-05-03T13:54:26-03:00: marked 37% (1188 of 3209 index files)
2025-05-03T14:29:57-03:00: marked 38% (1220 of 3209 index files)
2025-05-03T14:52:44-03:00: marked 39% (1252 of 3209 index files)
2025-05-03T15:55:53-03:00: marked 40% (1284 of 3209 index files)
2025-05-03T15:57:55-03:00: marked 41% (1316 of 3209 index files)
2025-05-03T16:24:00-03:00: marked 42% (1348 of 3209 index files)
2025-05-03T16:47:32-03:00: marked 43% (1380 of 3209 index files)
2025-05-03T17:15:39-03:00: marked 44% (1412 of 3209 index files)
2025-05-03T17:19:46-03:00: marked 45% (1445 of 3209 index files)
2025-05-03T17:23:31-03:00: marked 46% (1477 of 3209 index files)
2025-05-03T17:38:44-03:00: marked 47% (1509 of 3209 index files)
2025-05-03T17:40:51-03:00: marked 48% (1541 of 3209 index files)
2025-05-03T17:49:44-03:00: marked 49% (1573 of 3209 index files)
2025-05-03T17:51:28-03:00: marked 50% (1605 of 3209 index files)
2025-05-03T17:55:32-03:00: marked 51% (1637 of 3209 index files)
2025-05-03T17:59:41-03:00: marked 52% (1669 of 3209 index files)
2025-05-03T18:05:03-03:00: marked 53% (1701 of 3209 index files)
2025-05-03T18:12:45-03:00: marked 54% (1733 of 3209 index files)
2025-05-03T18:15:25-03:00: marked 55% (1765 of 3209 index files)
2025-05-03T18:33:46-03:00: marked 56% (1798 of 3209 index files)
2025-05-03T18:46:33-03:00: marked 57% (1830 of 3209 index files)
2025-05-03T18:51:49-03:00: marked 58% (1862 of 3209 index files)
2025-05-03T19:02:25-03:00: marked 59% (1894 of 3209 index files)
2025-05-03T19:04:44-03:00: marked 60% (1926 of 3209 index files)
2025-05-03T19:05:40-03:00: marked 61% (1958 of 3209 index files)
2025-05-03T19:58:20-03:00: marked 62% (1990 of 3209 index files)
2025-05-03T19:59:31-03:00: marked 63% (2022 of 3209 index files)
2025-05-03T20:13:38-03:00: marked 64% (2054 of 3209 index files)
2025-05-03T20:16:13-03:00: marked 65% (2086 of 3209 index files)
2025-05-03T20:18:53-03:00: marked 66% (2118 of 3209 index files)
2025-05-03T20:20:46-03:00: marked 67% (2151 of 3209 index files)
2025-05-03T20:20:55-03:00: marked 68% (2183 of 3209 index files)
2025-05-03T20:21:05-03:00: marked 69% (2215 of 3209 index files)
2025-05-03T20:26:06-03:00: marked 70% (2247 of 3209 index files)
2025-05-03T20:27:36-03:00: marked 71% (2279 of 3209 index files)
2025-05-03T20:28:18-03:00: marked 72% (2311 of 3209 index files)
2025-05-03T20:28:42-03:00: marked 73% (2343 of 3209 index files)
2025-05-03T20:28:58-03:00: marked 74% (2375 of 3209 index files)
2025-05-03T20:30:41-03:00: marked 75% (2407 of 3209 index files)
2025-05-03T20:40:59-03:00: marked 76% (2439 of 3209 index files)
2025-05-03T21:04:15-03:00: marked 77% (2471 of 3209 index files)
2025-05-03T21:05:24-03:00: marked 78% (2504 of 3209 index files)
2025-05-03T21:06:10-03:00: marked 79% (2536 of 3209 index files)
2025-05-03T21:07:42-03:00: marked 80% (2568 of 3209 index files)
2025-05-03T21:08:21-03:00: marked 81% (2600 of 3209 index files)
2025-05-03T21:51:56-03:00: marked 82% (2632 of 3209 index files)
2025-05-03T21:56:02-03:00: marked 83% (2664 of 3209 index files)
2025-05-03T22:21:01-03:00: marked 84% (2696 of 3209 index files)
2025-05-03T22:37:29-03:00: marked 85% (2728 of 3209 index files)
2025-05-03T22:41:53-03:00: marked 86% (2760 of 3209 index files)
2025-05-03T22:46:42-03:00: marked 87% (2792 of 3209 index files)
2025-05-03T22:50:45-03:00: marked 88% (2824 of 3209 index files)
2025-05-04T00:37:41-03:00: marked 89% (2857 of 3209 index files)
2025-05-04T01:16:01-03:00: marked 90% (2889 of 3209 index files)
2025-05-04T01:26:04-03:00: marked 91% (2921 of 3209 index files)
2025-05-04T01:53:07-03:00: marked 92% (2953 of 3209 index files)
2025-05-04T01:56:29-03:00: marked 93% (2985 of 3209 index files)
2025-05-04T01:57:42-03:00: marked 94% (3017 of 3209 index files)
2025-05-04T02:05:34-03:00: marked 95% (3049 of 3209 index files)
2025-05-04T02:06:24-03:00: marked 96% (3081 of 3209 index files)
2025-05-04T02:07:21-03:00: marked 97% (3113 of 3209 index files)
2025-05-04T02:15:15-03:00: marked 98% (3145 of 3209 index files)
2025-05-04T02:18:19-03:00: marked 99% (3177 of 3209 index files)
2025-05-04T02:19:45-03:00: marked 100% (3209 of 3209 index files)
2025-05-04T02:19:45-03:00: Start GC phase2 (sweep unused chunks)
2025-05-04T02:55:48-03:00: processed 1% (245241 chunks)
2025-05-04T03:39:59-03:00: processed 2% (492340 chunks)
2025-05-04T04:01:34-03:00: processed 3% (740226 chunks)
2025-05-04T04:23:38-03:00: processed 4% (987592 chunks)
2025-05-04T04:44:38-03:00: processed 5% (1235042 chunks)
2025-05-04T05:05:13-03:00: processed 6% (1482572 chunks)
2025-05-04T05:24:47-03:00: processed 7% (1730247 chunks)
2025-05-04T05:43:36-03:00: processed 8% (1977945 chunks)
2025-05-04T06:02:40-03:00: processed 9% (2225663 chunks)
2025-05-04T06:22:10-03:00: processed 10% (2473145 chunks)
2025-05-04T06:41:35-03:00: processed 11% (2719525 chunks)
2025-05-04T07:01:16-03:00: processed 12% (2967621 chunks)
2025-05-04T07:21:33-03:00: processed 13% (3215253 chunks)
2025-05-04T07:39:33-03:00: processed 14% (3462307 chunks)
2025-05-04T07:57:56-03:00: processed 15% (3710343 chunks)
2025-05-04T08:19:21-03:00: processed 16% (3958111 chunks)
2025-05-04T08:37:18-03:00: processed 17% (4205764 chunks)
2025-05-04T08:55:10-03:00: processed 18% (4453407 chunks)
2025-05-04T09:15:40-03:00: processed 19% (4700600 chunks)
2025-05-04T09:32:54-03:00: processed 20% (4948745 chunks)
2025-05-04T09:50:11-03:00: processed 21% (5196225 chunks)
2025-05-04T10:07:23-03:00: processed 22% (5444214 chunks)
2025-05-04T10:24:57-03:00: processed 23% (5691966 chunks)
2025-05-04T10:42:33-03:00: processed 24% (5939963 chunks)
2025-05-04T11:00:05-03:00: processed 25% (6187963 chunks)
2025-05-04T11:17:33-03:00: processed 26% (6436525 chunks)
2025-05-04T11:34:51-03:00: processed 27% (6684464 chunks)
2025-05-04T11:52:03-03:00: processed 28% (6932546 chunks)
2025-05-04T12:13:14-03:00: processed 29% (7180007 chunks)
2025-05-04T12:31:17-03:00: processed 30% (7427916 chunks)
2025-05-04T12:49:13-03:00: processed 31% (7676269 chunks)
2025-05-04T13:08:16-03:00: processed 32% (7923135 chunks)
2025-05-04T13:27:19-03:00: processed 33% (8170646 chunks)
2025-05-04T13:46:11-03:00: processed 34% (8418444 chunks)
2025-05-04T14:04:54-03:00: processed 35% (8666514 chunks)
2025-05-04T14:22:36-03:00: processed 36% (8914241 chunks)
2025-05-04T14:39:53-03:00: processed 37% (9161717 chunks)
2025-05-04T14:57:26-03:00: processed 38% (9409313 chunks)
2025-05-04T15:15:29-03:00: processed 39% (9656600 chunks)
2025-05-04T15:33:29-03:00: processed 40% (9904630 chunks)
2025-05-04T15:51:30-03:00: processed 41% (10152628 chunks)
2025-05-04T16:09:42-03:00: processed 42% (10401520 chunks)
2025-05-04T16:27:36-03:00: processed 43% (10649463 chunks)
2025-05-04T16:45:26-03:00: processed 44% (10896730 chunks)
2025-05-04T17:03:50-03:00: processed 45% (11145307 chunks)
2025-05-04T17:21:55-03:00: processed 46% (11392734 chunks)
2025-05-04T17:40:10-03:00: processed 47% (11640456 chunks)
2025-05-04T17:58:01-03:00: processed 48% (11889427 chunks)
2025-05-04T18:15:58-03:00: processed 49% (12136878 chunks)
2025-05-04T18:37:17-03:00: processed 50% (12384373 chunks)
2025-05-04T19:00:03-03:00: processed 51% (12632686 chunks)
2025-05-04T19:17:57-03:00: processed 52% (12880584 chunks)
2025-05-04T19:44:02-03:00: processed 53% (13128827 chunks)
2025-05-04T20:01:56-03:00: processed 54% (13376527 chunks)
2025-05-04T20:22:30-03:00: processed 55% (13624544 chunks)
2025-05-04T20:46:15-03:00: processed 56% (13872999 chunks)
2025-05-04T21:06:17-03:00: processed 57% (14121903 chunks)
2025-05-04T21:23:55-03:00: processed 58% (14370032 chunks)
2025-05-04T21:43:45-03:00: processed 59% (14618632 chunks)
2025-05-04T22:11:19-03:00: processed 60% (14867524 chunks)
2025-05-04T22:30:19-03:00: processed 61% (15117038 chunks)
2025-05-04T22:47:54-03:00: processed 62% (15365930 chunks)
2025-05-04T23:33:46-03:00: processed 63% (15615636 chunks)
2025-05-04T23:51:08-03:00: processed 64% (15866485 chunks)
2025-05-05T00:15:53-03:00: processed 65% (16116858 chunks)
2025-05-05T00:37:34-03:00: processed 66% (16367529 chunks)
2025-05-05T00:54:29-03:00: processed 67% (16618371 chunks)
2025-05-05T01:27:01-03:00: processed 68% (16869564 chunks)
2025-05-05T02:06:46-03:00: processed 69% (17120929 chunks)
2025-05-05T02:25:58-03:00: processed 70% (17373964 chunks)
2025-05-05T03:26:22-03:00: processed 71% (17627813 chunks)
2025-05-05T03:44:42-03:00: processed 72% (17882431 chunks)
2025-05-05T04:02:10-03:00: processed 73% (18138423 chunks)
2025-05-05T04:19:09-03:00: processed 74% (18393874 chunks)
2025-05-05T04:36:08-03:00: processed 75% (18647269 chunks)
2025-05-05T04:52:58-03:00: processed 76% (18902209 chunks)
2025-05-05T05:09:46-03:00: processed 77% (19156340 chunks)
2025-05-05T05:26:40-03:00: processed 78% (19411951 chunks)
2025-05-05T05:43:59-03:00: processed 79% (19667565 chunks)
2025-05-05T06:01:02-03:00: processed 80% (19922095 chunks)
2025-05-05T06:18:21-03:00: processed 81% (20177659 chunks)
2025-05-05T06:35:25-03:00: processed 82% (20432768 chunks)
2025-05-05T06:52:36-03:00: processed 83% (20688947 chunks)
2025-05-05T07:09:48-03:00: processed 84% (20943605 chunks)
2025-05-05T07:28:03-03:00: processed 85% (21198150 chunks)
2025-05-05T07:45:12-03:00: processed 86% (21452311 chunks)
2025-05-05T08:03:42-03:00: processed 87% (21707855 chunks)
2025-05-05T08:22:39-03:00: processed 88% (21963546 chunks)
2025-05-05T08:41:13-03:00: processed 89% (22218631 chunks)
2025-05-05T08:58:53-03:00: processed 90% (22474207 chunks)
2025-05-05T09:16:28-03:00: processed 91% (22729248 chunks)
2025-05-05T09:35:23-03:00: processed 92% (22985163 chunks)
2025-05-05T09:54:08-03:00: processed 93% (23240299 chunks)
2025-05-05T10:12:43-03:00: processed 94% (23495988 chunks)
2025-05-05T10:31:19-03:00: processed 95% (23751451 chunks)
2025-05-05T10:50:27-03:00: processed 96% (24006242 chunks)
2025-05-05T11:08:35-03:00: processed 97% (24261401 chunks)
2025-05-05T11:25:38-03:00: processed 98% (24517450 chunks)
2025-05-05T11:42:49-03:00: processed 99% (24772420 chunks)
2025-05-05T11:59:49-03:00: Removed garbage: 9.458 TiB
2025-05-05T11:59:49-03:00: Removed chunks: 6149718
2025-05-05T11:59:49-03:00: Pending removals: 2.049 GiB (in 1970 chunks)
2025-05-05T11:59:49-03:00: Original data usage: 542.491 TiB
2025-05-05T11:59:49-03:00: On-Disk usage: 36.812 TiB (6.79%)
2025-05-05T11:59:49-03:00: On-Disk chunks: 18875319
2025-05-05T11:59:49-03:00: Deduplication factor: 14.74
2025-05-05T11:59:49-03:00: Average chunk size: 2.045 MiB
2025-05-05T11:59:49-03:00: TASK OKThe GC is still running and I can't paste the logs here because there are too many lines with the following messages:
Code:
2025-05-13T01:29:04-03:00: Encountered new index file '"/mnt/datastore/pbs/ns/stor01/vm/20288/2025-05-12T06:00:02Z/drive-scsi0.img.fidx"', increment total index file count
2025-05-13T01:37:01-03:00: Encountered new index file '"/mnt/datastore/pbs/ns/stor01/vm/20266/2025-05-12T02:26:25Z/drive-scsi0.img.fidx"', increment total index file count
2025-05-13T01:37:46-03:00: Encountered new index file '"/mnt/datastore/pbs/ns/stor01/vm/20266/2025-05-11T02:11:07Z/drive-scsi0.img.fidx"', increment total index file count
2025-05-12T13:39:01-03:00: warning: unable to access non-existent chunk 780f82546e91812e1b1c04815c2afbf73ad963bc99650502b8c2b379093e3407, required by "/mnt/datastore/pbs002/ns/cloudkvm02/vm/20510/2025-05-12T00:55:08Z/drive-scsi0.img.fidx"
2025-05-12T13:39:01-03:00: warning: unable to access non-existent chunk 6a8c3eea53c7a6f96418f0b48f7c47ac342497836d3c3c9b6fe6a5d7826ed353, required by "/mnt/datastore/pbs002/ns/cloudkvm02/vm/20510/2025-05-12T00:55:08Z/drive-scsi0.img.fidx"My setup is ZFS with RAID6, my RAM is at 60% of 128GB and I don't use SWAP
Hi,
please open a new thread for this (referencing your post here for completeness) and post the output of
proxmox-backup-manager version --verbose so we can investigate this further. Also share the prune settings cat /etc/proxmox-backup/prune.cfg and datastore config cat /etc/proxmox-backup/datastore.cfg.My guess is that you are writing and pruning backups faster than the GC is able to keep up with. New backups might be created faster (using fast incremental mode) than the GC phase 1can keep up with due to cache capacity limit. How big are such backup snapshots typically in your case? Are they in the TiB range? In previous versions new snapshot indices were not considered during GC, which could however lead to possibly not touching all chunks in very specific edge cases with long running GC and high frequency pruning setups.
You can try and increase the
gc-cache-capacity in the datastore tuning options to the maximum value of 8388608 and restart GC, see https://pbs.proxmox.com/docs/storage.html#tuningIn general, you should consider adding a special device for such a storage setup, see https://pbs.proxmox.com/docs/sysadmin.html#zfs-special-device
I've been trying to find it and can't...does gc-cache-capacity have a default value? Or is the default "off"?
Hi, no the default value is
1024*1024, see https://lore.proxmox.com/pbs-devel/20250404130713.376630-1-c.ebner@proxmox.com/Will send a patch to extend the docs in that regard.
Ah thank you. So if 8388608 is 32 TB data then 1/8 is good for 4 TB of backup data, good to know.