PBS traffic control issue - does not seem to work

May 14, 2019
43
1
13
59
Once again thank you proxmox team for and excellent product and all your hard work.

I am trying to implement a global traffic limit on my primary pbs server using pbs traffic control and moving away from wondershaper.
I turned off wonder shaper and setup a pbs traffic control rule at 5MB/s to test the function.
I have backup current running so a good time to test.

This is the rule I created:
# proxmox-backup-manager traffic-control list
┌────────┬─────────┬──────────┬──────────┬───────────┬──────────────────────┬───────────┬─────────┐
│ name │ rate-in │ burst-in │ rate-out │ burst-out │ network │ timeframe │ comment │
╞════════╪═════════╪══════════╪══════════╪═══════════╪══════════════════════╪═══════════╪═════════╡
│ test-1 │ 5 MiB │ │ 5 MiB │ │ ["0.0.0.0/0","::/0"] │ │ test 5 │
└────────┴─────────┴──────────┴──────────┴───────────┴──────────────────────┴───────────┴─────────┘

it does affect the traffic in any way.

I am confused. Ideas ?

Thanks again
 
May 14, 2019
43
1
13
59
Doh ! you are correct, I measured the NIC throughput with iftop in MB/s mode and the rate was around 30MB/s when it should have dropped to 5MB/s as the only network traffic were the backups.
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
mhmm.... works here, can you post your iftop command? (are you sure there is not the mismatch between 'megabits per second' (afaics the iftop default) and "megabytes per second" (what the traffic control configures)) ?
 
May 14, 2019
43
1
13
59
mhmm.... works here, can you post your iftop command? (are you sure there is not the mismatch between 'megabits per second' (afaics the iftop default) and "megabytes per second" (what the traffic control configures)) ?
I used iftop -B -i eno1
Last night I also disabled the wondershaper service just in case it was somehow interfering with PBS traffic control, same result, ran a backup rate shot up to 60 MB/s when I requested that PBS keep it to 5MB/s in/out.
I also set the rate explicitly on my LAN subnet rather than 0.0.0.0/0, so my network setting was 192.168.30.0/24

David
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
what usage does the gui show? (columns: Rate In used and Rate out Used) ?
can you also post the output of 'proxmox-backup-manager versions --verbose' ?
 
May 14, 2019
43
1
13
59
what usage does the gui show? (columns: Rate In used and Rate out Used) ?
can you also post the output of 'proxmox-backup-manager versions --verbose' ?
The traffic control rate in used/out used always shows zero's, like it somehow is not bound to the only NIC

root@pve-pbs-n1:~# proxmox-backup-manager versions --verbose
proxmox-backup 2.1-1 running kernel: 5.13.19-3-pve
proxmox-backup-server 2.1.3-1 running version: 2.1.3
pve-kernel-helper 7.1-8
pve-kernel-5.13 7.1-6
pve-kernel-5.11 7.0-10
pve-kernel-5.4 6.4-5
pve-kernel-5.13.19-3-pve 5.13.19-7
pve-kernel-5.13.19-2-pve 5.13.19-4
pve-kernel-5.13.19-1-pve 5.13.19-3
pve-kernel-5.11.22-7-pve 5.11.22-12
pve-kernel-5.11.22-5-pve 5.11.22-10
pve-kernel-5.11.22-4-pve 5.11.22-9
pve-kernel-5.4.128-1-pve 5.4.128-1
pve-kernel-5.4.65-1-pve 5.4.65-1
ifupdown2 3.1.0-1+pmx3
libjs-extjs 7.0.0-1
proxmox-backup-docs 2.1.3-1
proxmox-backup-client 2.1.3-1
proxmox-mini-journalreader 1.2-1
proxmox-widget-toolkit 3.4-5
pve-xtermjs 4.12.0-1
smartmontools 7.2-pve2
zfsutils-linux 2.1.2-pve1
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
ok the versions look ok...

i mean it's probably obvious, but i'll say it nonetheless: the traffic control only limits traffic from and to the proxmox-backup-proxy/api, all other traffic to the server (e.g. ssh) is not affected by this..
could you maybe try restarting the daemon? 'systemctl reload proxmox-backup-proxy proxmox-backup' ?
 
May 14, 2019
43
1
13
59
ok the versions look ok...

i mean it's probably obvious, but i'll say it nonetheless: the traffic control only limits traffic from and to the proxmox-backup-proxy/api, all other traffic to the server (e.g. ssh) is not affected by this..
could you maybe try restarting the daemon? 'systemctl reload proxmox-backup-proxy proxmox-backup' ?
I am using the command "proxmox-backup-client backup family-video.pxar:" to backup a file system and would assume this program uses the "proxmox-backup-proxy/api, if it does not then that would explain the issue. During these test this is the only traffic on the system.

I restarted the daemon and still the same results.

Thanks
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
I am using the command "proxmox-backup-client backup family-video.pxar:" to backup a file system and would assume this program uses the "proxmox-backup-proxy/api, if it does not then that would explain the issue. During these test this is the only traffic on the system.

I restarted the daemon and still the same results.
mhmm no that should work this way (the client uses the api). i'll have to think why it could fail for you but works here just fine...
 
May 14, 2019
43
1
13
59
mhmm no that should work this way (the client uses the api). i'll have to think why it could fail for you but works here just fine...
Also I have a friend that is also using PBS in the same way and he is experiencing the same results, different hardware but same backup methodology.

Thanks again
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
mhmm... can you post your network config? ('ip addr', 'ip route', 'ip link') as well as the ip address of the client that makes the backup?
 
May 14, 2019
43
1
13
59
mhmm... can you post your network config? ('ip addr', 'ip route', 'ip link') as well as the ip address of the client that makes the backup?
root@pve-pbs-n1:~# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
link/ether d4:ae:52:d5:37:af brd ff:ff:ff:ff:ff:ff
altname enp4s0
inet 192.168.10.237/24 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:fed5:37af/64 scope link
valid_lft forever preferred_lft forever
3: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN group default qlen 32
link/ether d2:3c:ea:c8:7c:2d brd ff:ff:ff:ff:ff:ff
inet6 fe80::d03c:eaff:fec8:7c2d/64 scope link
valid_lft forever preferred_lft forever
root@pve-pbs-n1:~# ip route
default via 192.168.10.1 dev eno1 proto kernel onlink
192.168.10.0/24 dev eno1 proto kernel scope link src 192.168.10.237
root@pve-pbs-n1:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP mode DEFAULT group default qlen 1000
link/ether d4:ae:52:d5:37:af brd ff:ff:ff:ff:ff:ff
altname enp4s0
3: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN mode DEFAULT group default qlen 32
link/ether d2:3c:ea:c8:7c:2d brd ff:ff:ff:ff:ff:ff


ip of client being backed up 192.168.10.243
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
that looks normal and ok. can you maybe send a part of the syslog while you are doing a backup? maybe we can see more there? you can
get the log with 'journalctl' e.g. since the last boot with 'journalctl -b'
 
May 14, 2019
43
1
13
59
that looks normal and ok. can you maybe send a part of the syslog while you are doing a backup? maybe we can see more there? you can
get the log with 'journalctl' e.g. since the last boot with 'journalctl -b'
The dump generated thousands of line, do you want that ?
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
if you want, you can limit it with the '--since' and '--until' parameters (best a few minutes before and after a backup)
but yes, you can also attach the full log if you want (maybe check it for sensitive information before, and zip it to get a smaller file)
 
May 14, 2019
43
1
13
59
if you want, you can limit it with the '--since' and '--until' parameters (best a few minutes before and after a backup)
but yes, you can also attach the full log if you want (maybe check it for sensitive information before, and zip it to get a smaller file)
ok, try this
thanks again
 

Attachments

  • traffic_logs.zip
    20.8 KB · Views: 2

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,261
1,048
164
34
Vienna
no, there is sadly nothing there either...
there are not really many options left to debug this...
could you maybe change the traffic control to match the network of your client (and post the config again) and test if that works?
 
May 14, 2019
43
1
13
59
no, there is sadly nothing there either...
there are not really many options left to debug this...
could you maybe change the traffic control to match the network of your client (and post the config again) and test if that works?
I tried that and it did not work, same result.
oh well, I will think about it and update you if I find the answer.

thanks again
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!