10gb net

[gta]

Good day to everyone!
I have a bridge on my PVE for VMs with one 1gb/s network interface in it by which I have an access to this PVE (and VMs also).
PBS was installed in the same network. Backup jobs worked fine with 1gb/s speed.
After a while a did add 10gb cards in both PVE and PBS and also add PBS storage with new 10gb/s card IP (PBS and PVE connected directly).
Iperf shows ~10gb/s.
But PVE continues to backup only on 1gb/s.
Please tell me why? is there any restriction in PVE/PBS?
thank you

 

[dcsapak]

can you post such a task log?
are you sure you don't have a bottle neck somewhere else? e.g. your storage (reading or writing)

 

[gta]

INFO: starting new backup job: vzdump 111 --node pve01 --remove 0 --notes-template '{{guestname}}' --mode stop --storage pbs10gb
INFO: Starting Backup of VM 111 (qemu)
INFO: Backup started at 2023-12-14 13:21:00
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: AD-SRV2
INFO: include disk 'scsi0' 'local-zfs:vm-111-disk-0' 100G
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'vm/111/2023-12-14T10:21:00Z'
INFO: starting kvm to execute backup task
INFO: started backup task 'e2001269-4eed-44d2-acd7-4f9772491c44'
INFO: resuming VM again after 13 seconds
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (552.0 MiB of 100.0 GiB) in 3s, read: 184.0 MiB/s, write: 10.7 MiB/s
INFO: 1% (1.0 GiB of 100.0 GiB) in 9s, read: 82.0 MiB/s, write: 58.7 MiB/s
INFO: 2% (2.0 GiB of 100.0 GiB) in 19s, read: 104.0 MiB/s, write: 41.2 MiB/s
INFO: 3% (3.1 GiB of 100.0 GiB) in 28s, read: 116.0 MiB/s, write: 30.2 MiB/s
INFO: 4% (4.0 GiB of 100.0 GiB) in 35s, read: 140.0 MiB/s, write: 25.7 MiB/s
INFO: 5% (5.1 GiB of 100.0 GiB) in 43s, read: 140.5 MiB/s, write: 17.5 MiB/s
INFO: 6% (6.0 GiB of 100.0 GiB) in 51s, read: 117.5 MiB/s, write: 27.0 MiB/s
INFO: 7% (7.0 GiB of 100.0 GiB) in 59s, read: 126.0 MiB/s, write: 20.0 MiB/s
INFO: 8% (8.0 GiB of 100.0 GiB) in 1m 5s, read: 168.7 MiB/s, write: 6.0 MiB/s
INFO: 9% (9.0 GiB of 100.0 GiB) in 1m 12s, read: 149.7 MiB/s, write: 17.7 MiB/s
INFO: 10% (10.0 GiB of 100.0 GiB) in 1m 20s, read: 127.5 MiB/s, write: 35.0 MiB/s
INFO: 11% (11.0 GiB of 100.0 GiB) in 1m 31s, read: 94.9 MiB/s, write: 51.3 MiB/s
INFO: 12% (12.0 GiB of 100.0 GiB) in 1m 40s, read: 114.2 MiB/s, write: 53.3 MiB/s
INFO: 13% (13.1 GiB of 100.0 GiB) in 1m 49s, read: 116.4 MiB/s, write: 54.7 MiB/s
INFO: 14% (14.0 GiB of 100.0 GiB) in 1m 57s, read: 123.5 MiB/s, write: 38.0 MiB/s
INFO: 15% (15.1 GiB of 100.0 GiB) in 2m 6s, read: 116.9 MiB/s, write: 25.3 MiB/s
INFO: 16% (16.1 GiB of 100.0 GiB) in 2m 14s, read: 130.0 MiB/s, write: 37.0 MiB/s
INFO: 17% (17.1 GiB of 100.0 GiB) in 2m 23s, read: 111.6 MiB/s, write: 52.0 MiB/s
INFO: 18% (18.1 GiB of 100.0 GiB) in 2m 30s, read: 145.7 MiB/s, write: 32.0 MiB/s
INFO: 19% (19.1 GiB of 100.0 GiB) in 2m 38s, read: 136.5 MiB/s, write: 24.0 MiB/s
INFO: 20% (20.2 GiB of 100.0 GiB) in 2m 45s, read: 160.6 MiB/s, write: 26.3 MiB/s
INFO: 21% (21.1 GiB of 100.0 GiB) in 2m 50s, read: 172.8 MiB/s, write: 34.4 MiB/s
INFO: 22% (22.8 GiB of 100.0 GiB) in 2m 55s, read: 351.2 MiB/s, write: 26.4 MiB/s
INFO: 23% (23.8 GiB of 100.0 GiB) in 2m 58s, read: 344.0 MiB/s, write: 22.7 MiB/s
INFO: 24% (24.3 GiB of 100.0 GiB) in 3m 1s, read: 166.7 MiB/s, write: 14.7 MiB/s
INFO: 25% (25.2 GiB of 100.0 GiB) in 3m 6s, read: 186.4 MiB/s, write: 21.6 MiB/s
INFO: 29% (29.2 GiB of 100.0 GiB) in 3m 9s, read: 1.3 GiB/s, write: 2.7 MiB/s
INFO: 36% (36.7 GiB of 100.0 GiB) in 3m 12s, read: 2.5 GiB/s, write: 0 B/s
INFO: 44% (44.3 GiB of 100.0 GiB) in 3m 15s, read: 2.5 GiB/s, write: 0 B/s
INFO: 51% (51.6 GiB of 100.0 GiB) in 3m 18s, read: 2.5 GiB/s, write: 0 B/s
INFO: 59% (59.3 GiB of 100.0 GiB) in 3m 21s, read: 2.5 GiB/s, write: 0 B/s
INFO: 67% (67.2 GiB of 100.0 GiB) in 3m 24s, read: 2.6 GiB/s, write: 0 B/s
INFO: 74% (75.0 GiB of 100.0 GiB) in 3m 27s, read: 2.6 GiB/s, write: 0 B/s
INFO: 82% (82.9 GiB of 100.0 GiB) in 3m 30s, read: 2.6 GiB/s, write: 0 B/s
INFO: 90% (90.8 GiB of 100.0 GiB) in 3m 33s, read: 2.6 GiB/s, write: 0 B/s
INFO: 98% (98.4 GiB of 100.0 GiB) in 3m 36s, read: 2.5 GiB/s, write: 0 B/s
INFO: 100% (100.0 GiB of 100.0 GiB) in 3m 39s, read: 561.3 MiB/s, write: 0 B/s
INFO: Waiting for server to finish backup validation...
INFO: backup is sparse: 78.86 GiB (78%) total zero data
INFO: backup was done incrementally, reused 93.98 GiB (93%)
INFO: transferred 100.00 GiB in 221 seconds (463.3 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 111 (00:03:54)
INFO: Backup finished at 2023-12-14 13:24:54
INFO: Backup job finished successfully
TASK OK

 

[dcsapak]

the log seems to show that there is a write bottleneck somewhere, what is the target storage in pbs?

 

[gta]

hi, dcsapak!
thank you for quick answer and your help. appreciate!
there is a zfs on PBS with WD10EZEX with 150MB/s write speed
(((
sory

 

[gta]

its a result on pbs

 

[dcsapak]

ok but that seems to be sequential write speed, since pbs writes random chunks you won't get the same performance there.
you could test with the tool fio (Caution: some operations are destructive on the disk), e.g. with random write with about 2-4 MiB write size (4MiB chunks that get compressed) to check if that is the bottle neck

EDIT:

also you could do a benchmark with 'proxmox-backup-client benchmark' if you do that on the client and give a repository, you can see the theoretical limits

 

[gta]

 

[gta]

fio read
iodepth=32

 

[dcsapak]

View attachment 59845

it seems the client is limited somewhat by the cpu (190MiB/s verification is not really fast)

fio read
iodepth=32

but that is the read speed... for backup you need to test the write speed on the pbs (and as i said with 2-4 MiB size, not 4k, otherwise the speed will be very bad)

 

[gta]

Hi, Dominik!
I have two E5-2680 v3 @ 2.50GHz (12 core) in each socket (2) with 5% load on a client (PBS)
There is ZFS on PBS with 4 hdd. How can I safely use "fio" write tests, with no data lost.

 

[dcsapak]

instead of benchmarking you can also use a tool like 'atop' during a backup to see what part of the pbs is used how much (it's probably much faster to see bottlenecks this way)

 

[Testani]

did u solve the issue? i am in the same situation