PBS traffic control issue - does not seem to work

[David Thistlethwaite]

Once again thank you proxmox team for and excellent product and all your hard work.

I am trying to implement a global traffic limit on my primary pbs server using pbs traffic control and moving away from wondershaper.
I turned off wonder shaper and setup a pbs traffic control rule at 5MB/s to test the function.
I have backup current running so a good time to test.

This is the rule I created:
# proxmox-backup-manager traffic-control list
┌────────┬─────────┬──────────┬──────────┬───────────┬──────────────────────┬───────────┬─────────┐
│ name │ rate-in │ burst-in │ rate-out │ burst-out │ network │ timeframe │ comment │
╞════════╪═════════╪══════════╪══════════╪═══════════╪══════════════════════╪═══════════╪═════════╡
│ test-1 │ 5 MiB │ │ 5 MiB │ │ ["0.0.0.0/0","::/0"] │ │ test 5 │
└────────┴─────────┴──────────┴──────────┴───────────┴──────────────────────┴───────────┴─────────┘

it does affect the traffic in any way.

I am confused. Ideas ?

Thanks again

 

[dcsapak]

it does affect the traffic in any way.

i guess you mean 'does not affect'?

how did you measure the traffic, and what was it ?

 

[David Thistlethwaite]

Doh ! you are correct, I measured the NIC throughput with iftop in MB/s mode and the rate was around 30MB/s when it should have dropped to 5MB/s as the only network traffic were the backups.

 

[dcsapak]

mhmm.... works here, can you post your iftop command? (are you sure there is not the mismatch between 'megabits per second' (afaics the iftop default) and "megabytes per second" (what the traffic control configures)) ?

 

[David Thistlethwaite]

mhmm.... works here, can you post your iftop command? (are you sure there is not the mismatch between 'megabits per second' (afaics the iftop default) and "megabytes per second" (what the traffic control configures)) ?

I used iftop -B -i eno1
Last night I also disabled the wondershaper service just in case it was somehow interfering with PBS traffic control, same result, ran a backup rate shot up to 60 MB/s when I requested that PBS keep it to 5MB/s in/out.
I also set the rate explicitly on my LAN subnet rather than 0.0.0.0/0, so my network setting was 192.168.30.0/24

David

 

[dcsapak]

what usage does the gui show? (columns: Rate In used and Rate out Used) ?
can you also post the output of 'proxmox-backup-manager versions --verbose' ?

 

[David Thistlethwaite]

what usage does the gui show? (columns: Rate In used and Rate out Used) ?
can you also post the output of 'proxmox-backup-manager versions --verbose' ?

The traffic control rate in used/out used always shows zero's, like it somehow is not bound to the only NIC

root@pve-pbs-n1:~# proxmox-backup-manager versions --verbose
proxmox-backup 2.1-1 running kernel: 5.13.19-3-pve
proxmox-backup-server 2.1.3-1 running version: 2.1.3
pve-kernel-helper 7.1-8
pve-kernel-5.13 7.1-6
pve-kernel-5.11 7.0-10
pve-kernel-5.4 6.4-5
pve-kernel-5.13.19-3-pve 5.13.19-7
pve-kernel-5.13.19-2-pve 5.13.19-4
pve-kernel-5.13.19-1-pve 5.13.19-3
pve-kernel-5.11.22-7-pve 5.11.22-12
pve-kernel-5.11.22-5-pve 5.11.22-10
pve-kernel-5.11.22-4-pve 5.11.22-9
pve-kernel-5.4.128-1-pve 5.4.128-1
pve-kernel-5.4.65-1-pve 5.4.65-1
ifupdown2 3.1.0-1+pmx3
libjs-extjs 7.0.0-1
proxmox-backup-docs 2.1.3-1
proxmox-backup-client 2.1.3-1
proxmox-mini-journalreader 1.2-1
proxmox-widget-toolkit 3.4-5
pve-xtermjs 4.12.0-1
smartmontools 7.2-pve2
zfsutils-linux 2.1.2-pve1

 

[dcsapak]

ok the versions look ok...

i mean it's probably obvious, but i'll say it nonetheless: the traffic control only limits traffic from and to the proxmox-backup-proxy/api, all other traffic to the server (e.g. ssh) is not affected by this..
could you maybe try restarting the daemon? 'systemctl reload proxmox-backup-proxy proxmox-backup' ?

 

[David Thistlethwaite]

ok the versions look ok...

i mean it's probably obvious, but i'll say it nonetheless: the traffic control only limits traffic from and to the proxmox-backup-proxy/api, all other traffic to the server (e.g. ssh) is not affected by this..
could you maybe try restarting the daemon? 'systemctl reload proxmox-backup-proxy proxmox-backup' ?

I am using the command "proxmox-backup-client backup family-video.pxar:" to backup a file system and would assume this program uses the "proxmox-backup-proxy/api, if it does not then that would explain the issue. During these test this is the only traffic on the system.

I restarted the daemon and still the same results.

Thanks

 

[dcsapak]

I am using the command "proxmox-backup-client backup family-video.pxar:" to backup a file system and would assume this program uses the "proxmox-backup-proxy/api, if it does not then that would explain the issue. During these test this is the only traffic on the system.

I restarted the daemon and still the same results.

mhmm no that should work this way (the client uses the api). i'll have to think why it could fail for you but works here just fine...

 

[David Thistlethwaite]

mhmm no that should work this way (the client uses the api). i'll have to think why it could fail for you but works here just fine...

Also I have a friend that is also using PBS in the same way and he is experiencing the same results, different hardware but same backup methodology.

Thanks again

 

[dcsapak]

mhmm... can you post your network config? ('ip addr', 'ip route', 'ip link') as well as the ip address of the client that makes the backup?

 

[David Thistlethwaite]

mhmm... can you post your network config? ('ip addr', 'ip route', 'ip link') as well as the ip address of the client that makes the backup?

root@pve-pbs-n1:~# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
link/ether d4:ae:52:d5:37:af brd ff:ff:ff:ff:ff:ff
altname enp4s0
inet 192.168.10.237/24 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:fed5:37af/64 scope link
valid_lft forever preferred_lft forever
3: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN group default qlen 32
link/ether d2:3c:ea:c8:7c:2d brd ff:ff:ff:ff:ff:ff
inet6 fe80::d03c:eaff:fec8:7c2d/64 scope link
valid_lft forever preferred_lft forever
root@pve-pbs-n1:~# ip route
default via 192.168.10.1 dev eno1 proto kernel onlink
192.168.10.0/24 dev eno1 proto kernel scope link src 192.168.10.237
root@pve-pbs-n1:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP mode DEFAULT group default qlen 1000
link/ether d4:ae:52:d5:37:af brd ff:ff:ff:ff:ff:ff
altname enp4s0
3: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN mode DEFAULT group default qlen 32
link/ether d2:3c:ea:c8:7c:2d brd ff:ff:ff:ff:ff:ff


ip of client being backed up 192.168.10.243

 

[dcsapak]

that looks normal and ok. can you maybe send a part of the syslog while you are doing a backup? maybe we can see more there? you can
get the log with 'journalctl' e.g. since the last boot with 'journalctl -b'

 

[David Thistlethwaite]

that looks normal and ok. can you maybe send a part of the syslog while you are doing a backup? maybe we can see more there? you can
get the log with 'journalctl' e.g. since the last boot with 'journalctl -b'

The dump generated thousands of line, do you want that ?

 

[dcsapak]

if you want, you can limit it with the '--since' and '--until' parameters (best a few minutes before and after a backup)
but yes, you can also attach the full log if you want (maybe check it for sensitive information before, and zip it to get a smaller file)

 

[David Thistlethwaite]

if you want, you can limit it with the '--since' and '--until' parameters (best a few minutes before and after a backup)
but yes, you can also attach the full log if you want (maybe check it for sensitive information before, and zip it to get a smaller file)

ok, try this
thanks again

 

[dcsapak]

no, there is sadly nothing there either...
there are not really many options left to debug this...
could you maybe change the traffic control to match the network of your client (and post the config again) and test if that works?

 

[David Thistlethwaite]

no, there is sadly nothing there either...
there are not really many options left to debug this...
could you maybe change the traffic control to match the network of your client (and post the config again) and test if that works?

I tried that and it did not work, same result.
oh well, I will think about it and update you if I find the answer.

thanks again