Slow Backup/restore Speed on PBS

[eude]

Hallo,
We run a 14 Node Proxmox Cluster (6.3.3) with an attached 7 Node Ceph Cluster (Nautilus 14.2.16) as Storage. Our Promox Backup Server (1.0.8 but tried also with 1.0.11) runs as Baremetal inside the Proxmox Cluster with a ca. 70TiB ZFS as Backupstorage (128GB Memory, has read+write cache SSDs). Everything is connected with 10GB network.

here is a actual write benchmark test from one of the Nodes to the Backupstorage:

proxmox-backup-client benchmark --repository root@pam@10.10.5.6:datastore


an rbd export test shows:



but Backup speed is compared to those numbers - slow:


any ideas where the problem might be ?

thanks in advance!

 

[scasrl]

My own problem
https://forum.proxmox.com/threads/slow-backup-reading-source.86107/

 

[eude]

^

 

[fabian]

how fast is your backup storage (for random access)? what kind of pool is it?

 

[eude]

root@pve06:~# zpool status -v backup
  pool: backup
 state: ONLINE
  scan: scrub repaired 0B in 0 days 14:41:04 with 0 errors on Sun Mar 14 15:05:06 2021
config:

        NAME                        STATE     READ WRITE CKSUM
        backup                      ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            scsi-2001b4d20594d3b83  ONLINE       0     0     0
            scsi-2001b4d20bb09162b  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            scsi-2001b4d204af52ea5  ONLINE       0     0     0
            scsi-2001b4d203bf1bdf4  ONLINE       0     0     0
          mirror-2                  ONLINE       0     0     0
            scsi-2001b4d20458efc5d  ONLINE       0     0     0
            scsi-2001b4d20cb081513  ONLINE       0     0     0
          mirror-3                  ONLINE       0     0     0
            scsi-2001b4d201487ec1d  ONLINE       0     0     0
            scsi-2001b4d20d44038dd  ONLINE       0     0     0
          mirror-4                  ONLINE       0     0     0
            scsi-2001b4d20038272af  ONLINE       0     0     0
            scsi-2001b4d20b2f02278  ONLINE       0     0     0
          mirror-5                  ONLINE       0     0     0
            scsi-2001b4d20768779cd  ONLINE       0     0     0
            scsi-2001b4d20cd8bc2c7  ONLINE       0     0     0
          mirror-6                  ONLINE       0     0     0
            scsi-2001b4d20c1d7c83d  ONLINE       0     0     0
            scsi-2001b4d205719b533  ONLINE       0     0     0
          mirror-7                  ONLINE       0     0     0
            scsi-2001b4d20479f7de2  ONLINE       0     0     0
            scsi-2001b4d20b950c15e  ONLINE       0     0     0
          mirror-8                  ONLINE       0     0     0
            scsi-2001b4d20ddc3c58f  ONLINE       0     0     0
            scsi-2001b4d2024997f7e  ONLINE       0     0     0
          mirror-10                 ONLINE       0     0     0
            scsi-2001b4d20fb6971c5  ONLINE       0     0     0
            scsi-2001b4d2030e1bc95  ONLINE       0     0     0
        logs
          mirror-9                  ONLINE       0     0     0
            scsi-2001b4d206c21d309  ONLINE       0     0     0
            scsi-2001b4d206ff145c8  ONLINE       0     0     0
        cache
          scsi-2001b4d207f25cbb4    ONLINE       0     0     0
          scsi-2001b4d20b8df435f    ONLINE       0     0     0

errors: No known data errors

Mirrors = SATA (20x 8TB), Logs+Cache = SSDs (4x 1TB)




#######################################
Fio Random Write test on NFS Export of ZFS Pool Backup from Prx Node to Storage Node:

fio --name=randwrite --rw=randwrite --direct=1 --ioengine=libaio --bs=64k --numjobs=8 --size=512m --runtime=600 --group_reporting

Jobs: 8 (f=8): [w(8)][100.0%][w=197MiB/s][w=3153 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=8): err= 0: pid=18806: Wed Apr  7 09:59:21 2021
  write: IOPS=4073, BW=255MiB/s (267MB/s)(4096MiB/16089msec); 0 zone resets
    slat (usec): min=7, max=118, avg=15.07, stdev= 3.96
    clat (usec): min=324, max=457464, avg=1945.55, stdev=11857.83
     lat (usec): min=335, max=457481, avg=1960.85, stdev=11858.32
    clat percentiles (usec):
     |  1.00th=[   465],  5.00th=[   515], 10.00th=[   545], 20.00th=[   586],
     | 30.00th=[   627], 40.00th=[   660], 50.00th=[   693], 60.00th=[   725],
     | 70.00th=[   775], 80.00th=[   832], 90.00th=[   947], 95.00th=[  1205],
     | 99.00th=[ 38011], 99.50th=[ 90702], 99.90th=[170918], 99.95th=[217056],
     | 99.99th=[455082]
   bw (  KiB/s): min=  512, max=81792, per=12.38%, avg=32271.66, stdev=26581.76, samples=256
   iops        : min=    8, max= 1278, avg=504.23, stdev=415.34, samples=256
  lat (usec)   : 500=3.53%, 750=62.11%, 1000=26.33%
  lat (msec)   : 2=4.60%, 4=0.94%, 10=0.50%, 20=0.62%, 50=0.49%
  lat (msec)   : 100=0.47%, 250=0.39%, 500=0.01%
  cpu          : usr=0.34%, sys=0.92%, ctx=65610, majf=0, minf=91
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=4096MiB (4295MB), run=16089-16089msec
root@prx01:/mnt/pve/backuptest10#

###################################
Fio random read test on NFS Export of ZFS Pool Backup from Prx Node to Storage Node

fio --name=randread --rw=randwrite --direct=1 --ioengine=libaio --bs=64k --numjobs=8 -
-size=512m --runtime=600 --group_reporting

Jobs: 8 (f=8): [w(8)][100.0%][w=491MiB/s][w=7855 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=8): err= 0: pid=22153: Wed Apr  7 10:01:31 2021
  write: IOPS=4242, BW=265MiB/s (278MB/s)(4096MiB/15447msec); 0 zone resets
    slat (nsec): min=7739, max=98143, avg=14860.35, stdev=3768.92
    clat (usec): min=364, max=383821, avg=1833.94, stdev=10370.58
     lat (usec): min=378, max=383836, avg=1849.02, stdev=10371.04
    clat percentiles (usec):
     |  1.00th=[   465],  5.00th=[   515], 10.00th=[   545], 20.00th=[   594],
     | 30.00th=[   627], 40.00th=[   660], 50.00th=[   693], 60.00th=[   734],
     | 70.00th=[   783], 80.00th=[   848], 90.00th=[   971], 95.00th=[  1270],
     | 99.00th=[ 35914], 99.50th=[ 63177], 99.90th=[120062], 99.95th=[225444],
     | 99.99th=[383779]
   bw (  KiB/s): min=  768, max=88192, per=12.83%, avg=34827.21, stdev=28972.91, samples=240
   iops        : min=   12, max= 1378, avg=544.12, stdev=452.73, samples=240
  lat (usec)   : 500=3.41%, 750=60.26%, 1000=27.35%
  lat (msec)   : 2=5.26%, 4=0.74%, 10=0.69%, 20=0.74%, 50=0.81%
  lat (msec)   : 100=0.57%, 250=0.13%, 500=0.05%
  cpu          : usr=0.39%, sys=0.93%, ctx=65607, majf=0, minf=88
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=4096MiB (4295MB), run=15447-15447msec
root@prx01:/mnt/pve/backuptest10#

 

[fabian]

is the PBS datastore directly on ZFS or also accessed via NFS? what does the PBS client benchmark command report when run directly on the PBS server?

 

[eude]

the PBS is directly on ZFS,

benchmark on the pbs:

 

[fabian]

so you have a theoretical max throughput of 264MB/s (neither your server's hashing performance, nor your ZFS pools random write performance is higher than that). in practice you seem to be getting 70MB/s.

how is the random read performance inside the VM? how is the backup performance when you run backups in parallel from multiple nodes? how is it when you generate random file (e.g., a few GB worth of /dev/urandom into a tmpfs) and back that up directly using proxmox-backup-client? how is the load on the PBS server side during a backup (CPU & disks)?

 

[eude]

yes, around 240-270MiB/s is what im expecting with this ZFS and old CPU im using on the PBS.

################
Random read and write inside VM:

which VM?



################
Backup parallel performance from multiple nodes:

The Speed dropped on all parallel Backups tried 3 Nodes parallel:





################
Random File - direct pbs backup:

Load on PBS (8 Cores,):
Load is Hardly over 3, averages at 1.8,
in iftop you read the MiB/s the Proxmox Status Windows shows (20-50MiB/s),

heres iotop while backup is running (what you see is pretty much the average +/- a bit):

vmstat shows no swapping

 

[fabian]

yes, around 240-270MiB/s is what im expecting with this ZFS and old CPU im using on the PBS.

################
Random read and write inside VM:

which VM?

one using the same storage as your logs so far

################
Backup parallel performance from multiple nodes:

The Speed dropped on all parallel Backups tried 3 Nodes parallel:

View attachment 25065

so this confirms there really is a bottle neck, and it's not lack of IO depth..

################
Random File - direct pbs backup:

Load on PBS (8 Cores,):
Load is Hardly over 3, averages at 1.8,
in iftop you read the MiB/s the Proxmox Status Windows shows (20-50MiB/s),

that is kinda surprising - did you put the random file on a tmpfs? how big was it? if this was from memory to PBS, then either your client system is really slow at hashing, or the bottle neck is the server/ZFS end..

heres iotop while backup is running (what you see is pretty much the average +/- a bit):
View attachment 25066
vmstat shows no swapping

you didn't give concrete models for you disks, but here we have a few with 60-80 writes/s, which might already be the limit for those disks if they are 7.2k or 5.4k spinners..

 

[guletz]

how is the random read performance inside the VM? how is the backup performance when you run backups in parallel from multiple nodes? how is it when you generate random file (e.g., a few GB worth of /dev/urandom into a tmpfs) and back that up directly using proxmox-backup-client? how is the load on the PBS server side during a backup (CPU & disks)?

Hi,

@fabian try to figure out if your problem is on your VMs side or not, because on pbs side it seems to be ok.

So try to do this test as he told you.

Good luck / Bafta !

 

[eude]

Inside VM:
Random write with Fio to the ZFS Backup Storage
Run status group 0 (all jobs):
WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=4096MiB (4295MB), run=14772-14772msec

iostat on the ZFS while writing:


Random readwith Fio to the ZFS Backup Storage
Run status group 0 (all jobs):
READ: bw=748MiB/s (784MB/s), 748MiB/s-748MiB/s (784MB/s-784MB/s), io=4096MiB (4295MB), run=5479-5479msec

iostat on the ZFS while reading



The same storage was previously used with rbd incremental snapshots (eve4barc) (of course its something else then chunks) and has reached steady 240+ MiB/s

Disks are all Seagate Ironwolf, 8TB, 7200u/min, 256MB Cache, SATA 6 Gb/s

 

[eude]

i will deploy other Hardware too, to keep testing.

Thanks for all your help so far!

 

[eude]

Freshly installed pbs on 12 Core, 32GB Ram, with 2 TB Raid configured with 2x10gb/NIC

testing showed:




Could the issue being connected to his?
https://forum.proxmox.com/threads/proxmox-backup-speed-extremely-slow-for-vms-stored-on-ceph.30294/

and this?
https://lists.proxmox.com/pipermail/pve-devel/2016-February/019267.html

implemete Ceph Snapshots in PBS?

 

[fabian]

that's trivial to find out - just do a backup of a VM not stored on Ceph?

################
Random File - direct pbs backup:

Load on PBS (8 Cores,):
Load is Hardly over 3, averages at 1.8,
in iftop you read the MiB/s the Proxmox Status Windows shows (20-50MiB/s),

that is kinda surprising - did you put the random file on a tmpfs? how big was it? if this was from memory to PBS, then either your client system is really slow at hashing, or the bottle neck is the server/ZFS end..

you didn't reply to this - if a backup from memory/tmpfs is slow, then it can't really be Ceph as source that is at fault?

 

[mscd]

Same/similar problem in my setup ... Proxmox VE nodes connected with 10 Gbit NICs (10 Gbit tested by iperf) ... proxmox-backup-client benchmark shows 250 MB/s TLS-speed ... backups to Proxmox Backup Server "only" in the area of 70-80 MB/s ...



... one (interesting) point ... at the end of the backup process read speed increases dramatically (perhaps due to the unused blocks in the disk image of the vm?)

 

[mscd]

One more information:

My Proxmox Backup Server is running inside a VM (on a Proxmox VE node) ... backup data is residing on a TrueNas device (connected over 10 GBit per CIFS share). I run fio benchmark on the Proxmox Server (on the CIFS share) once with "--direct=1" flag ... 80 MB/s ...



... and once without "--direct=1" ... 330 MB/s

 

[tom]

See also:

https://pbs.proxmox.com/docs/system-requirements.html

 

[fabian]

One more information:

My Proxmox Backup Server is running inside a VM (on a Proxmox VE node) ... backup data is residing on a TrueNas device (connected over 10 GBit per CIFS share). I run fio benchmark on the Proxmox Server (on the CIFS share) once with "--direct=1" flag ... 80 MB/s ...

View attachment 25101

... and once without "--direct=1" ... 330 MB/s

View attachment 25100

the second one likely benchmarks your cache (data size is only 4G).

 

[eude]

Testing with a VM on another Storage (full SSD, no ceph) with 2x10GB NIC, was slightly better but not good (was more like ~80MiBs read/write):



Network is ok:

and vice-versa:



Back to testing:

fio --rw=readwrite --name=testrand --size=5G --direct=1 --bs=64k --rwmixread=50
(read and write test, without Memory, Blocksize 64k read/write ratio 50/50)

shows:


only ~80MiBs !

while the same with 1024k Blocksize (fio --rw=readwrite --name=testrand --size=5G --direct=1 --bs=1024k --rwmixread=50) shows:


What Blocksize does pbs use? is there a posibility to change it - for testing issues?