PBS Backup from ceph storage extremely slow

[GabrieleV]

Is it normal that PBS backup of VMs from a CEPH storage are extremely slow ?
Backups start for the first steps with a 200MB/s transfer rate and slows down to 5MB/s ...

 

[tom]

Is it normal that PBS backup of VMs from a CEPH storage are extremely slow ?

No.

 

[GabrieleV]

What kind of information should I provide to help diagnose the problem ?

 

[Dunuin]

I guess an explanation of your ceph setup (OSDs, network, etc) would be useful.

Keep in mind that PBS needs good IOPS performance. Does your ceph on the PBS host got a local copy of your datastore? Maybe it accesses the datastore contents over the network resulting in bad latencies.

 

[GabrieleV]

Ceph cluster:

3nodes
Dual Xeon
16Gb Ram
7x600GB HDD OSD
1x480GB SSD DBWAL
2x1Gb NIC management
2x1Gb NIC cluster
2x10Gb NIC storage

1 OSD per disk with common DB/WAL on 1 single SSD
1 Pool with 512PG

Backup is done via the management LAN on 1 PBS node
1x Intel(R) Pentium(R) CPU G3220 @ 3.00GHz
20GB Ram
2x8TB ZFS mirrored pool
1x240 ZFS SSD zil 16GB+160GBcache

 

[tom]

Start reading on https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2020-09-hyper-converged-with-nvme.76516/

You use slow HDDs and quite old hardware in general (?) - 16 GB ram is not enough for a HCI Ceph cluster.

Backup Server:
Intel G3220 is a CPU from 2013.
You also run HDDs, not optimal. You can improve pbs performance with https://pbs.proxmox.com/docs/sysadmin.html#local-zfs-special-device

=> Use modern hardware: for small HCI cluster, only NVMe/SSDs if you want good performance.

 

[GabrieleV]

Thank you for your advice. Yes, performances are bad due to lack of good hardware.Another problem was autoscaler hitting hard on ceph ...