Network interface for backup

[m3a2r1]

Sorry, I can't find which interface is used for backup function, can you explain me?

 

[Chris]

Hi,
this depends on which route is used to connect to your remote storage. ip route might give you some more clues.

 

[m3a2r1]

All devices in the same network, bridges on interfaces eth/sfp. Which of them is selected default for backup?

 

[m3a2r1]

I'm asking because I've got slow backup speed.
Proxmox server with vm I want to backup is connected to 10Gbit fiber switch,
on other side is another Proxmox server with PBS, connected to 10Gbit switch, PBS storage is on QNAP NAS NVME SSD drives provided by NFS.
Both servers has sfp link as default route, PBS is connected to sfp linked bridge, iperf tested between first Proxmox & PBS showed about 8Gbit.
But transfers are like that:

 

[Chris]

Could it be that you are limited by the read speed of the storage the VM resides on? What kind of storage is this?

 

[m3a2r1]

HP DL380 server with RAID 60, below write/read tests:

 

[m3a2r1]

And write/read test directly on NFS share which is used on PBS (dramatically slow read speed via NFS, it's ssd drive):

 

[Chris]

Okay, so not the storage. Could you provide the full log of the backup task. Also do you see high cpu load, high IO delay or similar during backup?

 

[m3a2r1]

There is no issues during backup.

Backup log:
INFO: starting new backup job: vzdump 406 --storage qnap2ssd --remove 0 --node wb-pve1 --mode snapshot
INFO: Starting Backup of VM 406 (qemu)
INFO: Backup started at 2023-01-05 13:20:54
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ws-ssd
INFO: include disk 'scsi0' 'qnap2:406/vm-406-disk-0.qcow2' 60G
INFO: include disk 'scsi1' 'VMs:406/vm-406-disk-0.qcow2' 32G
INFO: include disk 'efidisk0' 'wb-storage:406/vm-406-disk-0.qcow2' 128K
INFO: creating Proxmox Backup Server archive 'vm/406/2023-01-05T12:20:54Z'
INFO: starting kvm to execute backup task
INFO: started backup task 'ddf0f1a6-65b5-4b31-8aa0-be95fead04fa'
INFO: efidisk0: dirty-bitmap status: created new
INFO: scsi0: dirty-bitmap status: created new
INFO: scsi1: dirty-bitmap status: created new
INFO: 0% (280.0 MiB of 92.0 GiB) in 3s, read: 93.3 MiB/s, write: 52.0 MiB/s
INFO: 1% (960.0 MiB of 92.0 GiB) in 14s, read: 61.8 MiB/s, write: 61.8 MiB/s
INFO: 2% (1.8 GiB of 92.0 GiB) in 29s, read: 61.9 MiB/s, write: 61.9 MiB/s
INFO: 3% (2.8 GiB of 92.0 GiB) in 44s, read: 64.3 MiB/s, write: 64.3 MiB/s
INFO: 4% (3.7 GiB of 92.0 GiB) in 58s, read: 67.7 MiB/s, write: 66.3 MiB/s
INFO: 5% (4.6 GiB of 92.0 GiB) in 1m 10s, read: 76.7 MiB/s, write: 76.7 MiB/s
INFO: 6% (5.6 GiB of 92.0 GiB) in 1m 24s, read: 70.9 MiB/s, write: 70.9 MiB/s
INFO: 7% (6.5 GiB of 92.0 GiB) in 1m 35s, read: 82.2 MiB/s, write: 82.2 MiB/s
INFO: 8% (7.4 GiB of 92.0 GiB) in 1m 48s, read: 72.9 MiB/s, write: 72.9 MiB/s
INFO: 9% (8.4 GiB of 92.0 GiB) in 2m 2s, read: 71.1 MiB/s, write: 71.1 MiB/s
INFO: 10% (9.2 GiB of 92.0 GiB) in 2m 12s, read: 86.4 MiB/s, write: 86.4 MiB/s
INFO: 11% (10.1 GiB of 92.0 GiB) in 2m 25s, read: 73.8 MiB/s, write: 73.8 MiB/s
INFO: 12% (11.1 GiB of 92.0 GiB) in 2m 39s, read: 68.6 MiB/s, write: 68.6 MiB/s
INFO: 13% (12.0 GiB of 92.0 GiB) in 2m 54s, read: 64.3 MiB/s, write: 64.3 MiB/s
INFO: 14% (12.9 GiB of 92.0 GiB) in 3m 7s, read: 69.8 MiB/s, write: 69.8 MiB/s
INFO: 15% (13.8 GiB of 92.0 GiB) in 3m 21s, read: 67.4 MiB/s, write: 67.4 MiB/s
INFO: 16% (14.8 GiB of 92.0 GiB) in 3m 35s, read: 70.3 MiB/s, write: 70.3 MiB/s
INFO: 17% (15.7 GiB of 92.0 GiB) in 3m 49s, read: 67.4 MiB/s, write: 67.4 MiB/s
INFO: 18% (16.6 GiB of 92.0 GiB) in 4m 2s, read: 67.4 MiB/s, write: 67.4 MiB/s
INFO: 19% (17.5 GiB of 92.0 GiB) in 4m 16s, read: 68.3 MiB/s, write: 68.3 MiB/s
INFO: 20% (18.4 GiB of 92.0 GiB) in 4m 29s, read: 73.5 MiB/s, write: 73.5 MiB/s
INFO: 21% (19.3 GiB of 92.0 GiB) in 4m 42s, read: 69.8 MiB/s, write: 69.8 MiB/s
INFO: 22% (20.3 GiB of 92.0 GiB) in 4m 55s, read: 73.5 MiB/s, write: 73.5 MiB/s
INFO: 23% (21.2 GiB of 92.0 GiB) in 5m 7s, read: 80.7 MiB/s, write: 80.7 MiB/s
INFO: 24% (22.1 GiB of 92.0 GiB) in 5m 19s, read: 75.7 MiB/s, write: 75.7 MiB/s
INFO: 25% (23.0 GiB of 92.0 GiB) in 5m 34s, read: 63.5 MiB/s, write: 63.5 MiB/s
INFO: 26% (24.0 GiB of 92.0 GiB) in 5m 49s, read: 65.3 MiB/s, write: 65.3 MiB/s
INFO: 27% (24.9 GiB of 92.0 GiB) in 6m 3s, read: 66.6 MiB/s, write: 66.6 MiB/s
INFO: 28% (25.8 GiB of 92.0 GiB) in 6m 17s, read: 68.9 MiB/s, write: 68.9 MiB/s
INFO: 29% (26.7 GiB of 92.0 GiB) in 6m 30s, read: 68.0 MiB/s, write: 68.0 MiB/s
INFO: 30% (27.6 GiB of 92.0 GiB) in 6m 43s, read: 74.8 MiB/s, write: 74.8 MiB/s
INFO: 31% (28.6 GiB of 92.0 GiB) in 6m 55s, read: 81.3 MiB/s, write: 81.3 MiB/s
INFO: 32% (29.5 GiB of 92.0 GiB) in 7m 8s, read: 71.4 MiB/s, write: 71.4 MiB/s
INFO: 33% (30.4 GiB of 92.0 GiB) in 7m 21s, read: 69.8 MiB/s, write: 69.8 MiB/s
INFO: 34% (31.3 GiB of 92.0 GiB) in 7m 35s, read: 68.9 MiB/s, write: 68.9 MiB/s
INFO: 35% (32.2 GiB of 92.0 GiB) in 7m 49s, read: 67.7 MiB/s, write: 58.3 MiB/s
INFO: 36% (33.2 GiB of 92.0 GiB) in 8m 7s, read: 51.3 MiB/s, write: 44.9 MiB/s
INFO: 37% (34.1 GiB of 92.0 GiB) in 8m 31s, read: 38.3 MiB/s, write: 38.3 MiB/s
INFO: 38% (35.0 GiB of 92.0 GiB) in 8m 55s, read: 40.2 MiB/s, write: 40.2 MiB/s
INFO: 39% (35.9 GiB of 92.0 GiB) in 9m 14s, read: 48.6 MiB/s, write: 45.7 MiB/s
INFO: 40% (36.8 GiB of 92.0 GiB) in 9m 35s, read: 44.4 MiB/s, write: 44.4 MiB/s
INFO: 41% (37.7 GiB of 92.0 GiB) in 9m 57s, read: 43.3 MiB/s, write: 43.3 MiB/s
INFO: 42% (38.6 GiB of 92.0 GiB) in 10m 19s, read: 42.2 MiB/s, write: 42.2 MiB/s
INFO: 43% (39.6 GiB of 92.0 GiB) in 10m 42s, read: 42.1 MiB/s, write: 42.1 MiB/s
INFO: 44% (40.5 GiB of 92.0 GiB) in 11m 4s, read: 43.3 MiB/s, write: 43.3 MiB/s
INFO: 45% (41.4 GiB of 92.0 GiB) in 11m 24s, read: 45.6 MiB/s, write: 45.6 MiB/s
INFO: 46% (42.3 GiB of 92.0 GiB) in 11m 44s, read: 47.2 MiB/s, write: 47.0 MiB/s
INFO: 47% (43.3 GiB of 92.0 GiB) in 12m 1s, read: 57.4 MiB/s, write: 57.4 MiB/s
INFO: 48% (44.2 GiB of 92.0 GiB) in 12m 12s, read: 86.2 MiB/s, write: 86.2 MiB/s
INFO: 49% (45.1 GiB of 92.0 GiB) in 12m 24s, read: 75.0 MiB/s, write: 75.0 MiB/s
INFO: 50% (46.1 GiB of 92.0 GiB) in 12m 37s, read: 77.2 MiB/s, write: 77.2 MiB/s
INFO: 51% (47.0 GiB of 92.0 GiB) in 12m 49s, read: 78.7 MiB/s, write: 78.7 MiB/s
INFO: 52% (47.8 GiB of 92.0 GiB) in 13m 1s, read: 73.3 MiB/s, write: 73.3 MiB/s
INFO: 53% (48.8 GiB of 92.0 GiB) in 13m 14s, read: 76.0 MiB/s, write: 76.0 MiB/s
INFO: 54% (49.7 GiB of 92.0 GiB) in 13m 26s, read: 76.0 MiB/s, write: 76.0 MiB/s
INFO: 55% (50.6 GiB of 92.0 GiB) in 13m 39s, read: 72.9 MiB/s, write: 72.9 MiB/s
INFO: 56% (51.5 GiB of 92.0 GiB) in 13m 51s, read: 78.3 MiB/s, write: 78.3 MiB/s
INFO: 57% (52.5 GiB of 92.0 GiB) in 14m 3s, read: 82.7 MiB/s, write: 82.7 MiB/s
INFO: 58% (53.4 GiB of 92.0 GiB) in 14m 14s, read: 81.5 MiB/s, write: 81.5 MiB/s
INFO: 59% (54.3 GiB of 92.0 GiB) in 14m 25s, read: 86.5 MiB/s, write: 86.5 MiB/s
INFO: 60% (55.2 GiB of 92.0 GiB) in 14m 35s, read: 91.2 MiB/s, write: 91.2 MiB/s
INFO: 61% (56.1 GiB of 92.0 GiB) in 14m 46s, read: 85.8 MiB/s, write: 85.8 MiB/s
INFO: 62% (57.1 GiB of 92.0 GiB) in 14m 56s, read: 95.6 MiB/s, write: 95.6 MiB/s
INFO: 63% (58.0 GiB of 92.0 GiB) in 15m 7s, read: 86.5 MiB/s, write: 86.5 MiB/s
INFO: 64% (58.9 GiB of 92.0 GiB) in 15m 17s, read: 96.4 MiB/s, write: 96.4 MiB/s
INFO: 65% (59.8 GiB of 92.0 GiB) in 15m 26s, read: 101.8 MiB/s, write: 101.8 MiB/s
INFO: 66% (60.8 GiB of 92.0 GiB) in 15m 36s, read: 96.0 MiB/s, write: 96.0 MiB/s
INFO: 67% (61.7 GiB of 92.0 GiB) in 15m 46s, read: 96.0 MiB/s, write: 96.0 MiB/s
INFO: 68% (62.6 GiB of 92.0 GiB) in 15m 56s, read: 88.8 MiB/s, write: 88.8 MiB/s
INFO: 69% (63.5 GiB of 92.0 GiB) in 16m 5s, read: 106.2 MiB/s, write: 106.2 MiB/s
INFO: 70% (64.4 GiB of 92.0 GiB) in 16m 14s, read: 105.8 MiB/s, write: 105.8 MiB/s
INFO: 71% (65.3 GiB of 92.0 GiB) in 16m 23s, read: 103.1 MiB/s, write: 103.1 MiB/s
INFO: 72% (66.3 GiB of 92.0 GiB) in 16m 32s, read: 104.4 MiB/s, write: 104.4 MiB/s
INFO: 73% (67.3 GiB of 92.0 GiB) in 16m 41s, read: 112.9 MiB/s, write: 112.9 MiB/s
INFO: 74% (68.1 GiB of 92.0 GiB) in 16m 51s, read: 85.6 MiB/s, write: 85.6 MiB/s
INFO: 75% (69.1 GiB of 92.0 GiB) in 17m, read: 109.3 MiB/s, write: 109.3 MiB/s
INFO: 76% (69.9 GiB of 92.0 GiB) in 17m 10s, read: 89.6 MiB/s, write: 89.6 MiB/s
INFO: 77% (70.9 GiB of 92.0 GiB) in 17m 30s, read: 49.8 MiB/s, write: 49.8 MiB/s
INFO: 78% (71.8 GiB of 92.0 GiB) in 17m 51s, read: 43.2 MiB/s, write: 43.2 MiB/s
INFO: 79% (72.7 GiB of 92.0 GiB) in 18m 15s, read: 39.0 MiB/s, write: 39.0 MiB/s
INFO: 80% (73.7 GiB of 92.0 GiB) in 18m 39s, read: 41.5 MiB/s, write: 41.5 MiB/s
INFO: 81% (74.6 GiB of 92.0 GiB) in 18m 47s, read: 113.5 MiB/s, write: 113.5 MiB/s
INFO: 82% (75.5 GiB of 92.0 GiB) in 18m 56s, read: 104.4 MiB/s, write: 104.4 MiB/s
INFO: 83% (76.5 GiB of 92.0 GiB) in 19m 5s, read: 112.4 MiB/s, write: 112.4 MiB/s
INFO: 84% (77.3 GiB of 92.0 GiB) in 19m 13s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 85% (78.2 GiB of 92.0 GiB) in 19m 22s, read: 105.8 MiB/s, write: 105.8 MiB/s
INFO: 86% (79.2 GiB of 92.0 GiB) in 19m 32s, read: 96.8 MiB/s, write: 96.8 MiB/s
INFO: 87% (80.0 GiB of 92.0 GiB) in 19m 41s, read: 97.8 MiB/s, write: 97.8 MiB/s
INFO: 88% (81.0 GiB of 92.0 GiB) in 19m 51s, read: 102.0 MiB/s, write: 102.0 MiB/s
INFO: 89% (81.9 GiB of 92.0 GiB) in 20m, read: 98.2 MiB/s, write: 98.2 MiB/s
INFO: 90% (82.8 GiB of 92.0 GiB) in 20m 9s, read: 102.2 MiB/s, write: 102.2 MiB/s
INFO: 91% (83.8 GiB of 92.0 GiB) in 20m 18s, read: 110.2 MiB/s, write: 110.2 MiB/s
INFO: 92% (84.7 GiB of 92.0 GiB) in 20m 27s, read: 107.1 MiB/s, write: 107.1 MiB/s
INFO: 93% (85.6 GiB of 92.0 GiB) in 20m 36s, read: 101.3 MiB/s, write: 101.3 MiB/s
INFO: 94% (86.5 GiB of 92.0 GiB) in 20m 46s, read: 92.8 MiB/s, write: 92.8 MiB/s
INFO: 95% (87.4 GiB of 92.0 GiB) in 20m 55s, read: 103.1 MiB/s, write: 103.1 MiB/s
INFO: 96% (88.3 GiB of 92.0 GiB) in 21m 4s, read: 102.7 MiB/s, write: 102.7 MiB/s
INFO: 97% (89.3 GiB of 92.0 GiB) in 21m 13s, read: 108.4 MiB/s, write: 108.4 MiB/s
INFO: 98% (90.2 GiB of 92.0 GiB) in 21m 21s, read: 115.5 MiB/s, write: 115.5 MiB/s
INFO: 99% (91.2 GiB of 92.0 GiB) in 21m 30s, read: 112.9 MiB/s, write: 112.9 MiB/s
INFO: 100% (92.0 GiB of 92.0 GiB) in 21m 39s, read: 94.7 MiB/s, write: 94.7 MiB/s
INFO: backup is sparse: 452.00 MiB (0%) total zero data
INFO: backup was done incrementally, reused 452.00 MiB (0%)
INFO: transferred 92.00 GiB in 1299 seconds (72.5 MiB/s)
INFO: stopping kvm after backup task
INFO: Finished Backup of VM 406 (00:21:44)
INFO: Backup finished at 2023-01-05 13:42:38
INFO: Backup job finished successfully
TASK OK

 

[Chris]

Since you are using PBS for backups, you can try the included benchmark by running proxmox-backup-client benchmark --repository mybackupstore.

Is the PBS installed on bare metal or running in a VM? If in a VM, make sure to set the CPU model to
host in order to get hardware acceleration for cryptographic operations. Also check that these are enabled in your BIOS.

See also https://forum.proxmox.com/threads/extremely-slow-pbs-speeds.102967/

 

[m3a2r1]

PBS is VM on another proxmox server. There is no CPU model set.

 

[m3a2r1]

 

[GTA_doum]

Found any solution? I have a similar setup with even higher speed on benchmark, but the backups run much slower. It seems the pbs in a VM does not see the network card being a 10G and treats it as 1G. Any suggestions to get this flying, so I could see all the money I've put down to get backups fast...

 

[m3a2r1]

I didn't find solution, I'm still working on it. With iperf, I get about 8Gbit, but when I try to copy file between nodes, I get about 120-160MB/s , This is very much under storage performance.

 

[m3a2r1]

I've created another topic about 8 months ago and didn't get help, too : https://forum.proxmox.com/threads/backup-restore-low-speed-what-is-wrong.124426/