PBS 3.2-7 Traffic Control not working ???

[ednt]

Hi,

due to 'wholes' in the administration diagrams during backups, and some failed backups, I tried to reduce the incoming traffic.
I reduced it to Rate In: 256.00 MiB/s.

But ....
in the Network traffic diagram I can still see a peek of netin: 33.85G

Is the traffic control not working?

The PBS is installed on bare metal.

The CPU load is maximum at 35%
Server load maximum at 40%
But all these loads are long before the dropout in the diagrams.

Best regards,

Bernd

 

[Chris]

Hi,

due to 'wholes' in the administration diagrams during backups, and some failed backups, I tried to reduce the incoming traffic.
I reduced it to Rate In: 256.00 MiB/s.

But ....
in the Network traffic diagram I can still see a peek of netin: 33.85G

Is the traffic control not working?

The PBS is installed on bare metal.

The CPU load is maximum at 35%
Server load maximum at 40%
But all these loads are long before the dropout in the diagrams.

Best regards,

Bernd

Hi,
how did you try to limit the traffic exactly? On the client side (PVE or promox-backup-client) or the PBS side? Please share the output of cat /etc/proxmox-backup/traffic-control.cfg and cat /etc/network/interfaces from the PBS host.

Also, what do you mean with administration diagrams? Are you referring to the network graphs on the PVE side?

 

[ednt]

Hi Chris,

I used the Traffic Control from the PBS.

traffic-control.cfg:

rule: Drosselung
        network 0.0.0.0/0
        network ::/0
        rate-in 256 MiB
        timeframe 00:00-23:59

interfaces:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address 192.168.248.91/23
        gateway 192.168.249.254

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface eno49 inet manual

iface eno50 inet manual

auto bond0
iface bond0 inet manual
        bond-mode 802.3ad
        bond_xmit_hash_policy layer2+3
        bond-slaves eno49 eno50
        mtu 9000

auto ceph_public
iface ceph_public inet static
        address 10.10.3.91/24
        mtu 9000
        vlan-id 55
        vlan-raw-device bond0

The traffic goes over the ceph_public interface.

The diagrams are on the PBS below 'Server Administration'



I don't know how this value is possible (Week (maximum)). The bond results in 20Gbit/s.
Maybe the calculation has a bug, or does it accumulate the traffic over a minute?

I think the traffic control works (in a way).
At the first peek the it was set to 1024Mbit/s.
At the second peek it was set to 512Mbit/s.
Now it is at 256Mbit/s.

But the peeks are changing, and I don't think that they are now always 'low'.


Best regards,

Bernd

 

[ednt]

The reason to reduce the traffic and to avoid that blank gaps on the diagrams are failing backups like this:

2024-08-01T01:35:19+02:00: starting new backup on datastore 'pk-pbs-01' from ::ffff:10.10.3.9: "ns/backup-41/vm/211/2024-07-31T23:35:16Z"
2024-08-01T01:35:19+02:00: download 'index.json.blob' from previous backup.
2024-08-01T01:37:55+02:00: register chunks in 'drive-virtio0.img.fidx' from previous backup.
2024-08-01T01:37:55+02:00: download 'drive-virtio0.img.fidx' from previous backup.
2024-08-01T01:37:55+02:00: created new fixed index 1 ("ns/backup-41/vm/211/2024-07-31T23:35:16Z/drive-virtio0.img.fidx")
2024-08-01T01:38:09+02:00: register chunks in 'drive-virtio1.img.fidx' from previous backup.
2024-08-01T01:38:09+02:00: download 'drive-virtio1.img.fidx' from previous backup.
2024-08-01T01:38:09+02:00: created new fixed index 2 ("ns/backup-41/vm/211/2024-07-31T23:35:16Z/drive-virtio1.img.fidx")
2024-08-01T01:38:22+02:00: register chunks in 'drive-virtio2.img.fidx' from previous backup.
2024-08-01T01:38:22+02:00: download 'drive-virtio2.img.fidx' from previous backup.
2024-08-01T01:38:23+02:00: created new fixed index 3 ("ns/backup-41/vm/211/2024-07-31T23:35:16Z/drive-virtio2.img.fidx")
2024-08-01T01:38:39+02:00: register chunks in 'drive-virtio3.img.fidx' from previous backup.
2024-08-01T01:38:39+02:00: download 'drive-virtio3.img.fidx' from previous backup.
2024-08-01T01:39:50+02:00: created new fixed index 4 ("ns/backup-41/vm/211/2024-07-31T23:35:16Z/drive-virtio3.img.fidx")
2024-08-01T01:40:02+02:00: add blob "/mnt/datastore/pk-pbs-01/ns/backup-41/vm/211/2024-07-31T23:35:16Z/qemu-server.conf.blob" (526 bytes, comp: 526)
2024-08-01T01:40:02+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-08-01T01:40:02+02:00: removing unfinished backup
2024-08-01T01:40:02+02:00: TASK ERROR: backup ended but finished flag is not set.

 

[Chris]

I don't know how this value is possible (Week (maximum)). The bond results in 20Gbit/s.
Maybe the calculation has a bug, or does it accumulate the traffic over a minute?

You might be interpreting the graphs incorrectly, the values show accumulated traffic, not the rate directly.

At the first peek the it was set to 1024Mbit/s.
At the second peek it was set to 512Mbit/s.
Now it is at 256Mbit/s.

Also the values configured via the WebUI are set to MiB/s not Mbit/s, please note that distinction.

Further note that the traffic limits will only apply to the connections to the PBS server itself, if you have other, independent traffic like e.g. Ceph traffic this will not be covered by the traffic limiter.

I would however recommend to clearly separate networks when possible.

2024-08-01T01:40:02+02:00: backup ended and finish failed: backup ended but finished flag is not set. 2024-08-01T01:40:02+02:00: removing unfinished backup 2024-08-01T01:40:02+02:00: TASK ERROR: backup ended but finished flag is not set.

This sounds more like the client connection vanished before it finished the backup. Can you share the tasklog for the backup job on the PVE side. That might give a clue on why this backup fails. Also, please post the output of proxmox-backup-manager version --verbose and pveversion -v for the Proxmox VE host.

 

[ednt]

Hi Chris,

PBS is up to date:

proxmox-backup                     3.2.0        running kernel: 6.8.8-3-pve
proxmox-backup-server              3.2.7-1      running version: 3.2.7     
proxmox-kernel-helper              8.1.0                                   
pve-kernel-5.15                    7.4-10                                 
proxmox-kernel-6.8                 6.8.8-3                                 
proxmox-kernel-6.8.8-3-pve-signed  6.8.8-3                                 
proxmox-kernel-6.8.8-2-pve-signed  6.8.8-2                                 
proxmox-kernel-6.8.8-1-pve-signed  6.8.8-1                                 
proxmox-kernel-6.5.13-5-pve-signed 6.5.13-5                               
proxmox-kernel-6.5                 6.5.13-5                               
pve-kernel-5.15.136-1-pve          5.15.136-1                             
pve-kernel-5.15.35-1-pve           5.15.35-3                               
ifupdown2                          3.2.0-1+pmx9                           
libjs-extjs                        7.0.0-4                                 
proxmox-backup-docs                3.2.7-1                                 
proxmox-backup-client              3.2.7-1                                 
proxmox-mail-forward               0.2.3                                   
proxmox-mini-journalreader         1.4.0                                   
proxmox-offline-mirror-helper      0.6.6                                   
proxmox-widget-toolkit             4.2.3                                   
pve-xtermjs                        5.3.0-3                                 
smartmontools                      7.3-pve1                               
zfsutils-linux                     2.2.4-pve1

PVE is not up to date it is the latest version of 7.4.18
I know, I have to update it.


the log on PVE looks like:

INFO: Starting Backup of VM 136 (qemu)
INFO: Backup started at 2024-08-07 01:16:45
INFO: status = running
INFO: VM Name: pk-xxxx-sd-01
INFO: include disk 'virtio0' 'local_images:136/vm-136-disk-0.qcow2' 10G
INFO: exclude disk 'virtio1' 'pk-pm-backup-03:136/vm-136-disk-1.qcow2' (backup=no)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/136/2024-08-06T23:16:45Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 136 qmp command 'backup' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 136 failed - VM 136 qmp command 'backup' failed - got timeout
INFO: Failed at 2024-08-07 01:19:19

It looks like the problem is on the PVE side.

 

[Chris]

PVE is not up to date it is the latest version of 7.4.18
I know, I have to update it.

That would be the first step, so we are not hunting ghosts of the past.

Please also share your VM config qm config 136 --current and the storage config cat /etc/pve/storage.cfg

 

[ednt]

config:

agent: 1
boot: order=ide2;virtio0;net0
cores: 4
description: pk-bareos-sd-01
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=7.0.0,ctime=1691072341
name: pk-bareos-sd-01
net0: virtio=72:EE:F0:76:F7:B0,bridge=vmbr1,tag=1200
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=90526676-3d97-401c-bc48-ef6628aa977d
sockets: 1
virtio0: local_images:136/vm-136-disk-0.qcow2,size=10G
virtio1: pk-pm-backup-03:136/vm-136-disk-1.qcow2,backup=0,size=5000G
vmgenid: c5f60392-0637-4f03-a2f3-1e6b6d54bce1

storage:

dir: local
        disable
        path /var/lib/vz
        content backup,vztmpl,iso
        prune-backups keep-last=1
        shared 0

lvmthin: local-lvm
        disable
        thinpool data
        vgname pve
        content rootdir,images

rbd: ssd-1
        content images,rootdir
        krbd 0
        pool ssd-1

rbd: hdd-1
        content rootdir,images
        krbd 0
        pool hdd-1

dir: local_images
        path /var/lib/vz
        content images
        shared 0

pbs: pk-pbs-02
        disable
        datastore backup-41
        server 10.10.3.92
        content backup
        fingerprint xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
        prune-backups keep-all=1
        username backup-41@pbs

dir: pk-pm-backup-01
        path /mnt/pve/pk-pm-backup-01
        content snippets,backup,vztmpl,iso,rootdir,images
        is_mountpoint 1
        nodes pk-pm-backup-01

dir: pk-pm-backup-02
        disable
        path /mnt/pve/pk-pm-backup-02
        content images,rootdir,snippets,vztmpl,iso,backup
        is_mountpoint 1
        nodes pk-pm-backup-02
        shared 0

dir: pk-pm-backup-03
        path /mnt/pve/pk-pm-backup-03
        content rootdir,images,backup,vztmpl,iso,snippets
        is_mountpoint 1
        nodes pk-pm-backup-03

lvmthin: cpu-01-raid10-spare
        thinpool cpu-01-raid10-spare
        vgname cpu-01-raid10-spare
        content rootdir,images
        nodes pk-pm-cpu-01

lvmthin: cpu-02-raid10-spare
        thinpool cpu-02-raid10-spare
        vgname cpu-02-raid10-spare
        content images,rootdir
        nodes pk-pm-cpu-02

zfspool: zpool1
        pool zpool1
        content images,rootdir
        mountpoint /zpool1
        nodes pk-pm-cpu-09,pk-pm-cpu-12,pk-pm-cpu-10
        sparse 0

pbs: pk-pbs-01
        datastore pk-pbs-01
        server 10.10.3.91
        content backup
        fingerprint xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
        namespace backup-41
        prune-backups keep-all=1
        username backup-41@pbs

btrfs: cpu-11-raid10-btrfs
        path /mnt/cpu-11-raid10-btrfs
        content images,rootdir
        nodes pk-pm-cpu-11
        prune-backups keep-all=1

cifs: pk-q-nas-01
        path /mnt/pve/pk-q-nas-01
        server 10.10.3.80
        share images
        content vztmpl,iso
        prune-backups keep-all=1
        username images

Backups go to pbs: pk-pbs-01


It is not so easy to find a timeslot to update everything and some big VMs are on fast local nvme storage, what does not allow to move the VMs.
But it is on th to do list.