[SOLVED] tape backup write performance - shoe shining

butterblume78 · Oct 7, 2024

Hello together,

we try to backup datastore to tape, but the write stream falls below the minimum speed.
The required minimum write speed for Tandberg LTO-6 drive is 54 MB/s

System:
CPU: 2x Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz
RAM: 256G
Storage: mdadm, software raid6, 10 x SATA SSD 960GB

fio benchmark;
fio --filename=/mnt/storage/test --size=500M --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=1 --time_based --group_reporting --name=iops-test-job --eta-newline=1

result: ...read: IOPS=41.9k, BW=164MiB/s ...

The reported write speed is 16 MB/s - see attached screenshot. - whole time.
Interference from other utilization is excluded.
The backup task of 5TB lasts 3days.

The system requirements are met.
I don't see any possibilities for performance tuning.

My questions:

Is there a bug?
Can someone point me in the right direction?
Why is this so slow?

waltar · Oct 7, 2024

Is your LTO-6 directly connected to that mentioned host with sw-raid6, right ?
What does top show against the backup procxess while it's running ?
What does iostat -xm 1 show same time ? (apt install sysstat).

butterblume78 · Oct 7, 2024

Yes, directly connected LTO-6 drive

top # while tape writing

top - 10:21:11 up 205 days, 16:27, 2 users, load average: 0,04, 0,08, 0,03
Tasks: 624 total, 1 running, 623 sleeping, 0 stopped, 0 zombie
%CPU(s): 0,0 us, 0,1 sy, 0,0 ni, 99,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Spch: 257646,1 total, 220913,7 free, 9681,3 used, 28862,1 buff/cache
MiB Swap: 7807,0 total, 7804,2 free, 2,8 used. 247964,8 avail Spch

PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
1627 backup 20 0 17,2g 143432 28364 S 3,6 0,1 14d+10h proxmox-backup-
1469 root 20 0 3964296 34420 22848 S 0,7 0,0 4d+17h proxmox-backup-
...

iostat -xm 1 # while tape writing

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,02    0,00    0,09    0,02    0,00   99,88

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
md0              0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,00
md1              0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,00
md127           30,00     14,25     0,00   0,00    0,53   486,40    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,02   2,40
md2              0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,00
sda              0,00      0,00     0,00   0,00    0,00     0,00    1,00      0,00     0,00   0,00    9,00     0,50    0,00      0,00     0,00   0,00    0,00     0,00    1,00    9,00    0,02   1,20
sdb              0,00      0,00     0,00   0,00    0,00     0,00    1,00      0,00     0,00   0,00    9,00     0,50    0,00      0,00     0,00   0,00    0,00     0,00    1,00    9,00    0,02   1,20
sdc              6,00      1,37     0,00   0,00    0,83   233,33    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   1,20
sdd              6,00      1,50     0,00   0,00    0,83   256,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   1,20
sde              4,00      1,00     0,00   0,00    1,00   256,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,80
sdf              4,00      0,90     0,00   0,00    1,00   231,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,80
sdg              4,00      1,00     0,00   0,00    1,00   256,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,80
sdh              6,00      1,50     0,00   0,00    1,00   256,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   1,20
sdi              4,00      2,00     0,00   0,00    1,25   512,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   2,00
sdj              4,00      2,00     0,00   0,00    1,25   512,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   2,00
sdk              4,00      1,48     0,00   0,00    1,25   379,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   2,00
sdl              3,00      1,50     0,00   0,00    1,33   512,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,01   1,60

butterblume78 · Oct 7, 2024

no issues with controller, drive connection so far
I cannot see something from

Code:

journalctl
# or
dmesg -T

dietmar · Oct 7, 2024

Also, do you run the latest version of PBS?

butterblume78 · Oct 7, 2024

dietmar said:
Also, do you run the latest version of PBS?

Yes, before 3.1-4 now 3.2-7.
The behavior seems to be the same.

waltar · Oct 7, 2024

Run a cleaner tape ! Do you run cleaner tapes regulary, eg. every week ?

butterblume78 · Oct 7, 2024

waltar said:
Run a cleaner tape ! Do you run cleaner tapes regulary, eg. every week ?

Tape Library: "Auto Clean" configured
Drive Status: "No Cleaning Required"

There are no logged drive or write errors reported by the library.
The libary can be also successfully utilized by other hardware.

waltar · Oct 7, 2024

Maybe you can find out more with "strace -p <pid>" of your 2 backup pid's running as user root and backup (as shown on top picture above) ?
Maybe file locking problems occur ? Are you backup the running (and so permanently changing) vm/lxc files ?

butterblume78 · Oct 7, 2024

waltar said:
Maybe you can find out more with "strace -p <pid>" of your 2 backup pid's running as user root and backup (as shown on top picture above) ?
Maybe file locking problems occur ? Are you backup the running (and so permanently changing) vm/lxc files ?

There are no concurrent tasks.
There are no concurrent running backups.

Also mentioned, the slow write speed is the same - over days. It is not changing.
This does not appear to be an issue of technical utilization of the storage being read from.
There is no io delay , wait time, whatsoever.

waltar · Oct 7, 2024

Take a look on the pid's at next backup run.
What files exactly are you backing up ?

butterblume78 · Oct 7, 2024

Code:

 ps aux |grep proxmox
root        3654  6.3  0.0 4032956 23032 ?       Ssl  10:53   5:01 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-api
backup      3766  3.0  0.0 4597800 54400 ?       Ssl  10:53   2:25 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy



there are threads ..we should use 


with lsof 

and 
strace -fp <pid>
#  strace -fp 3766  in this case 

I see a lot of something.
But strace output file is immediately too big to upload.


[CODE] grep "sg2" /tmp/3766.txt
3850  openat(29, "sg2", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 28
3850  readlinkat(29, "sg2", "../../devices/pci0000:00/0000:00"..., 4096) = 95
3850  openat(29, "sg2", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 28
3850  openat(AT_FDCWD, "/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/host0/target0:0:18/0:0:18:0/scsi_generic/sg2/uevent", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 28
3850  read(28, "MAJOR=21\nMINOR=2\nDEVNAME=sg2\n", 4104) = 29
3850  readlinkat(AT_FDCWD, "/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/host0/target0:0:18/0:0:18:0/scsi_generic/sg2/subsystem", "../../../../../../../../../class"..., 4096) = 45
3849  openat(29, "sg2", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 28
3849  readlinkat(29, "sg2", "../../devices/pci0000:00/0000:00"..., 4096) = 95
3849  openat(29, "sg2", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 28
3849  openat(AT_FDCWD, "/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/host0/target0:0:18/0:0:18:0/scsi_generic/sg2/uevent", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 28
3849  read(28, "MAJOR=21\nMINOR=2\nDEVNAME=sg2\n", 4104) = 29
3849  readlinkat(AT_FDCWD, "/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/host0/target0:0:18/0:0:18:0/scsi_generic/sg2/subsystem", "../../../../../../../../../class"..., 4096) = 45

Code:

grep "/mnt/storage" /tmp/3766.txt
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/163b/163bc1e99c4d768737d86e3bbc21cf144bfa7a68a14565dab3254d520941b145", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/c6f8/c6f8047344ef137c6f87fa6dd8f0cfbe0ffc2e2356af7f00f84beba42025f8fc", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/fb2a/fb2a3d10266e3313ab5a0d04618186da6451d00f1d647285799317783b6e76dd", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/8176/8176ad8e67604df3475a14e04004f10245d02ebd01cf3acffea0ed6210e23c8d", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/7d27/7d27ae8e9526eb40c6e16cbf304a84a1f1a9986e5fd233807cf3d3e73ae5b5d4", O_RDONLY|O_CLOEXEC) = 18
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/4621/4621e40889b5b163e156e7c13716a011fcd5ded8c8a467c9885a4d95ba94a3da", O_RDONLY|O_CLOEXEC) = 18
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/c9b4/c9b4b56192cefb86fbc35c10ae8e2d1ff9d11d3a633a5578faf82647ff71a7e6", O_RDONLY|O_CLOEXEC) = 18
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/3f7b/3f7b5ed8e8b4ad7c858878a4b97f1a44d7cf8076a9def836347b2e2bd6daed41", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/100d/100ddf15cd66d8e9046ca27d008dcaf8c9e0eea7cc22bbf72f43c31761ec6275", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/34c8/34c8f68b22e6a68a932306cc124e1ba8fe72669c24513fe6f2be1448b5522705", O_RDONLY|O_CLOEXEC <unfinished ...>
11807 openat(AT_FDCWD, "/mnt/storage/.chunks/81db/81db5e31e9591c6085fae7f45a49ea663fdbb81de5a221d04762d0b859341692", O_RDONLY|O_CLOEXEC <unfinished ...>
10685 statfs("/mnt/storage", {f_type=XFS_SUPER_MAGIC, f_bsize=4096, f_blocks=1874594304, f_bfree=431549393, f_bavail=431549393, f_files=750046400, f_ffree=747481729, f_fsid={val=[0x10300, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
10685 newfstatat(AT_FDCWD, "/mnt/storage", {st_mode=S_IFDIR|0755, st_size=74, ...}, 0) = 0

I need help debugging.
What am I searching for?

butterblume78 · Oct 7, 2024

waltar said:
Take a look on the pid's at next backup run.
What files exactly are you backing up ?

I try to do a tape backup of a local datastore.
The datastore contains backups of 2 KVM virtual machines.

butterblume78 · Oct 7, 2024

If I do "read only" Maintenance Mode, the backup performance is the same.
If I mount the xfs storage with "noatime", the backup performance is the same.

waltar · Oct 7, 2024

With "noatime" your win will be just (nearly unmeasureable) minimal as default ist "relatime" (=on in batched writes).

grep "read(" /tmp/3766.txt

butterblume78 · Oct 7, 2024

waltar said:
With "noatime" your win will be just (nearly unmeasureable) minimal as default ist "relatime" (=on in batched writes).

grep "read(" /tmp/3766.txt

waltar · Oct 7, 2024

butterblume78 said:
<unfinished ...>

Wondering about these lines which send by you and in attached file also.
But lots of data is read which is ok and as it's quiet long strong error not assumed.
No idea so far (for this remote problem).

butterblume78 · Oct 8, 2024

ok, with iostat
I can see that the system is constantly reading with round about 18 MB/s

it seems, the the mdadm raid 6 with 10x SAMSUNG_MZ7LM960HMJP-00005 (960 GB SATA SSD)
is too slow for this task - at least in this configuration.

waltar · Oct 8, 2024

No, the mdadm raid isn't too slow !! First always let iostat "run" as the first output is wrong ! Second if you let it run with "-xm 1" you see disk utilization in % which is low as too less I/O requests were generated. In "top" you will (hopefully ?) not see md or xfs processes taking 100% or ?

butterblume78 · Oct 15, 2024

waltar said:
No, the mdadm raid isn't too slow !! First always let iostat "run" as the first output is wrong ! Second if you let it run with "-xm 1" you see disk utilization in % which is low as too less I/O requests were generated. In "top" you will (hopefully ?) not see md or xfs processes taking 100% or ?

Yes, it is. I have changed the hardware. The SSDs are the same, but it's now a hardware RAID6.
The installed system is the same, on the same disks.

old System:
CPU: 2x Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz
RAM: 256G
data storage: mdadm, software raid6, 10 x SATA SSD 960GB

new System:
CPU: 2x CPU Intel XEON E5-2630v4
RAM: 384GB
data storage: AVAGO 3108 MegaRAID, Hardware Raid6, 10 x SATA SSD 960GB

fio --filename=/mnt/storage/test --size=500M --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=1 --time_based --group_reporting --name=iops-test-job --eta-newline=1

old result: read: IOPS=41.9k, BW=164MiB/s ...
new result: read: IOPS=57.4k, BW=224MiB/s (235MB/s)

Code:

2024-10-15T16:26:08+02:00: wrote 6131 chunks (4296.80 MB at 78.49 MB/s)
2024-10-15T16:26:52+02:00: wrote 1705 chunks (4298.11 MB at 97.05 MB/s)
2024-10-15T16:27:36+02:00: wrote 1722 chunks (4297.06 MB at 97.09 MB/s)
2024-10-15T16:28:23+02:00: wrote 1861 chunks (4297.33 MB at 91.91 MB/s)
2024-10-15T16:28:49+02:00: wrote 1740 chunks (4295.75 MB at 166.68 MB/s)
2024-10-15T16:29:15+02:00: wrote 1677 chunks (4297.85 MB at 166.50 MB/s)
2024-10-15T16:29:40+02:00: wrote 1870 chunks (4297.33 MB at 166.75 MB/s)
2024-10-15T16:30:06+02:00: wrote 1815 chunks (4297.06 MB at 166.58 MB/s)

It is constantly writing to the tape, now.

FAZIT:
fio benchmark does not disclose that the hardware requirements are not met.

-> Ticket closed

[SOLVED] tape backup write performance - shoe shining

New Member

Attachments

Active Member

New Member

New Member

Proxmox Staff Member

New Member

Active Member

New Member

Active Member

New Member

Active Member

New Member

New Member

New Member

Active Member

New Member

Attachments

Active Member

New Member

Active Member

New Member