Slow Tape Backups on HP LTO4

[resoli]

Hello, I am testing pbs 2.0-9 on an old HP MSL6030 tape library with two LTO4 drives inside. The first backup seems quite slow compared with usual ones (made with dump command).

• dump: 2426.42GB in 47 hours, so more or less 14MB/s
• pbs: 135GB in 6 hours, so about 6MB/s

Hers's the job detail:

()
2021-09-16T09:28:26+02:00: Starting tape backup job 'pbsstore:mediapooltest:Drive0:testdump'
2021-09-16T09:28:26+02:00: update media online status
2021-09-16T09:28:26+02:00: starting new media set - reason: policy is AlwaysCreate
2021-09-16T09:28:26+02:00: media set uuid: 4dc8ae77-2a10-46b3-8e96-21ba3a4c34bb
2021-09-16T09:28:26+02:00: found 4 groups
2021-09-16T09:28:26+02:00: backup snapshot vm/150/2021-09-13T19:00:02Z
2021-09-16T09:28:26+02:00: allocated new writable media 'QT0791L4'
2021-09-16T09:28:26+02:00: loading media 'QT0791L4' into drive 'Drive0'
2021-09-16T09:28:26+02:00: found media label QT0791L4 (cfe1236b-66bf-43a5-82ef-8e0912bd970a)
2021-09-16T09:28:26+02:00: writing new media set label (overwrite '00000000-0000-0000-0000-000000000000/0')
2021-09-16T09:28:28+02:00: moving to end of media
2021-09-16T09:50:13+02:00: wrote 2217 chunks (4298.90 MB at 3.30 MB/s)
2021-09-16T10:10:58+02:00: wrote 2218 chunks (4295.23 MB at 3.45 MB/s)
2021-09-16T10:33:52+02:00: wrote 2196 chunks (4296.02 MB at 3.13 MB/s)
2021-09-16T11:07:09+02:00: wrote 2210 chunks (4295.49 MB at 2.15 MB/s)
2021-09-16T11:23:20+02:00: wrote 2109 chunks (4297.33 MB at 4.42 MB/s)
2021-09-16T11:32:22+02:00: wrote 1230 chunks (2487.22 MB at 4.59 MB/s)
2021-09-16T11:32:23+02:00: end backup pbsstore:vm/150/2021-09-13T19:00:02Z
2021-09-16T11:32:23+02:00: percentage done: 8.33% (0/4 groups, 1/3 snapshots in group #1)
2021-09-16T11:32:23+02:00: backup snapshot vm/150/2021-09-14T19:00:02Z
2021-09-16T12:03:24+02:00: wrote 2562 chunks (4295.49 MB at 2.31 MB/s)
2021-09-16T12:16:23+02:00: wrote 2095 chunks (4296.80 MB at 5.52 MB/s)
2021-09-16T12:21:35+02:00: wrote 893 chunks (1842.35 MB at 5.91 MB/s)
2021-09-16T12:21:36+02:00: end backup pbsstore:vm/150/2021-09-14T19:00:02Z
2021-09-16T12:21:36+02:00: percentage done: 16.67% (0/4 groups, 2/3 snapshots in group #1)
2021-09-16T12:21:36+02:00: backup snapshot vm/150/2021-09-15T19:00:02Z
2021-09-16T12:51:14+02:00: wrote 2604 chunks (4295.23 MB at 3.47 MB/s)
2021-09-16T13:04:21+02:00: wrote 2077 chunks (4295.75 MB at 5.46 MB/s)
2021-09-16T13:09:36+02:00: wrote 861 chunks (1766.06 MB at 5.62 MB/s)
2021-09-16T13:09:36+02:00: end backup pbsstore:vm/150/2021-09-15T19:00:02Z
2021-09-16T13:09:36+02:00: percentage done: 25.00% (1/4 groups)
2021-09-16T13:09:36+02:00: backup snapshot vm/205/2021-09-13T19:00:02Z
2021-09-16T13:31:08+02:00: wrote 1877 chunks (4296.80 MB at 3.33 MB/s)
2021-09-16T13:41:41+02:00: wrote 1900 chunks (4296.28 MB at 6.78 MB/s)
2021-09-16T13:53:34+02:00: wrote 1926 chunks (4298.38 MB at 6.04 MB/s)
2021-09-16T14:04:16+02:00: wrote 1879 chunks (4296.02 MB at 6.69 MB/s)
2021-09-16T14:18:27+02:00: wrote 1971 chunks (4296.02 MB at 5.05 MB/s)
2021-09-16T14:27:30+02:00: wrote 2108 chunks (4295.75 MB at 7.92 MB/s)
2021-09-16T14:31:47+02:00: wrote 950 chunks (1936.72 MB at 7.53 MB/s)
2021-09-16T14:31:48+02:00: end backup pbsstore:vm/205/2021-09-13T19:00:02Z
2021-09-16T14:31:48+02:00: percentage done: 33.33% (1/4 groups, 1/3 snapshots in group #2)
2021-09-16T14:31:48+02:00: backup snapshot vm/205/2021-09-14T19:00:02Z
2021-09-16T14:50:16+02:00: wrote 2615 chunks (4295.23 MB at 4.74 MB/s)
2021-09-16T14:59:09+02:00: wrote 2114 chunks (4296.28 MB at 8.07 MB/s)
2021-09-16T15:01:53+02:00: wrote 550 chunks (1121.19 MB at 6.89 MB/s)
2021-09-16T15:01:53+02:00: end backup pbsstore:vm/205/2021-09-14T19:00:02Z
2021-09-16T15:01:53+02:00: percentage done: 41.67% (1/4 groups, 2/3 snapshots in group #2)
2021-09-16T15:01:53+02:00: backup snapshot vm/205/2021-09-15T19:00:02Z
2021-09-16T15:18:55+02:00: wrote 2735 chunks (4295.23 MB at 4.20 MB/s)
2021-09-16T15:28:09+02:00: wrote 2119 chunks (4296.54 MB at 7.76 MB/s)
2021-09-16T15:31:24+02:00: wrote 801 chunks (1675.62 MB at 8.61 MB/s)
2021-09-16T15:31:24+02:00: end backup pbsstore:vm/205/2021-09-15T19:00:02Z
2021-09-16T15:31:24+02:00: percentage done: 50.00% (2/4 groups)
2021-09-16T15:31:24+02:00: group vm/206 was empty
2021-09-16T15:31:24+02:00: backup snapshot vm/211/2021-09-12T19:00:02Z
2021-09-16T15:43:34+02:00: wrote 3063 chunks (4295.49 MB at 8.09 MB/s)
2021-09-16T15:47:08+02:00: wrote 1051 chunks (1447.82 MB at 6.76 MB/s)
2021-09-16T15:47:09+02:00: end backup pbsstore:vm/211/2021-09-12T19:00:02Z
2021-09-16T15:47:09+02:00: percentage done: 81.25% (3/4 groups, 1/4 snapshots in group #4)
2021-09-16T15:47:09+02:00: backup snapshot vm/211/2021-09-13T19:00:02Z
2021-09-16T15:50:20+02:00: wrote 697 chunks (699.40 MB at 4.67 MB/s)
2021-09-16T15:50:20+02:00: end backup pbsstore:vm/211/2021-09-13T19:00:02Z
2021-09-16T15:50:20+02:00: percentage done: 87.50% (3/4 groups, 2/4 snapshots in group #4)
2021-09-16T15:50:20+02:00: backup snapshot vm/211/2021-09-14T19:00:01Z
2021-09-16T15:53:30+02:00: wrote 751 chunks (707.53 MB at 4.90 MB/s)
2021-09-16T15:53:30+02:00: end backup pbsstore:vm/211/2021-09-14T19:00:01Z
2021-09-16T15:53:30+02:00: percentage done: 93.75% (3/4 groups, 3/4 snapshots in group #4)
2021-09-16T15:53:30+02:00: backup snapshot vm/211/2021-09-15T19:00:02Z
2021-09-16T15:56:52+02:00: wrote 832 chunks (670.30 MB at 4.26 MB/s)
2021-09-16T15:56:52+02:00: end backup pbsstore:vm/211/2021-09-15T19:00:02Z
2021-09-16T15:56:52+02:00: percentage done: 100.00% (4/4 groups)
2021-09-16T15:56:56+02:00: append media catalog
2021-09-16T15:56:56+02:00: TASK OK

Is it normal? Any hint?

 

[dietmar]

I guess your PBS datastore is too slow. What kind of disks/storage do you use?

For LTO4, your datastore should be able to deliver 140MB/s to get maximal tape speed (your seem
to provide 3-5 MB/s).

 

[resoli]

Hello Dietmar

The PBS datastore is an ext4 filesystem on a LV in a large VG where other LV are used as BackupPC pools. The physical drives are a bunch of enterprise SAS disks in raid5. The tape library is FC connected.

You are right. These LV volumes are often under heavy utilization. I tried few minutes ago (BackupPC dump running) a dump command on a snapshot of the PBS datastore on the same tape drive and the same tape media and the speed is same or lower than PBS one. Will retry with a dedicated system or so.

Thanks,
rob