Backup: Odd slowness on one host in cluster

[bferrell]

I'm getting some odd slowness in one member of my cluster. When the backup runs it shows about 3/3MB backup speed (even though iperf3 to the FreeNAS NFS store is 9GB/s) but after migrating to another cluster member it's much faster. Backup from below is on NODE2 (svr-02), slowness is on NODE1 (svr-01), 192.168.101.102 is the FreeNAS backup server.

Connecting to host 192.168.101.102, port 5201
[ 5] local 192.168.101.11 port 38302 connected to 192.168.101.102 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 974 MBytes 8.17 Gbits/sec 0 1.51 MBytes
[ 5] 1.00-2.00 sec 859 MBytes 7.20 Gbits/sec 0 1.51 MBytes
[ 5] 2.00-3.00 sec 980 MBytes 8.22 Gbits/sec 0 1.51 MBytes
[ 5] 3.00-4.00 sec 866 MBytes 7.27 Gbits/sec 850 916 KBytes
[ 5] 4.00-5.00 sec 1.08 GBytes 9.29 Gbits/sec 0 1.33 MBytes
[ 5] 5.00-6.00 sec 1.03 GBytes 8.87 Gbits/sec 0 1.36 MBytes
[ 5] 6.00-7.00 sec 1.01 GBytes 8.69 Gbits/sec 0 1.36 MBytes
[ 5] 7.00-8.00 sec 1.09 GBytes 9.33 Gbits/sec 0 1.36 MBytes
[ 5] 8.00-9.00 sec 1001 MBytes 8.40 Gbits/sec 584 1.04 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.88 GBytes 8.48 Gbits/sec 1434 sender
[ 5] 0.00-10.00 sec 9.88 GBytes 8.48 Gbits/sec receiver

iperf Done.
root@svr-01:~#

INFO: starting new backup job: vzdump 100 --remove 0 --node svr-02 --mode snapshot --storage FN2_BACKUP --compress lzo
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-04-13 22:23:10
INFO: status = running
INFO: update VM 100: -lock backup
INFO: VM Name: DNS
INFO: include disk 'scsi0' 'FN3_IMAGES:100/vm-100-disk-3.qcow2' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: pending configuration changes found (not included into backup)
INFO: creating archive '/mnt/pve/FN2_BACKUP/dump/vzdump-qemu-100-2020_04_13-22_23_10.vma.lzo'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '096cb49a-504c-40be-9565-8dc58f960411'
INFO: status: 0% (220987392/53687091200), sparse 0% (5505024), duration 3, read/write 73/71 MB/s
INFO: status: 1% (594018304/53687091200), sparse 0% (219504640), duration 7, read/write 93/39 MB/s
INFO: status: 2% (1098514432/53687091200), sparse 0% (517537792), duration 14, read/write 72/29 MB/s
INFO: status: 3% (1676279808/53687091200), sparse 0% (534294528), duration 24, read/write 57/56 MB/s
INFO: status: 4% (2189426688/53687091200), sparse 1% (545710080), duration 36, read/write 42/41 MB/s

 

[tim]

What's the backup speed of svr1, if this is the one where you had the reduced backup speed?

 

[bferrell]

4/1 - see below. I've moved all of my guests off of this machine so I can troubleshoot. The network config is the same as the other nodes, 1G .102 subnet for corosyn, 10G .100 for default network and 10G .101 for storage. The base network is working - another iperf run is below. I'm starting to suspect a problem with the NIC. I haven't dug into the switch logs, but the .100 port seems to be showing less activity than node2 for some reason.

INFO: starting new backup job: vzdump 204 --compress lzo --mode snapshot --node svr-01 --storage FN2_BACKUP --remove 0
INFO: Starting Backup of VM 204 (qemu)
INFO: Backup started at 2020-04-14 13:32:15
INFO: status = running
INFO: update VM 204: -lock backup
INFO: VM Name: Jitsi
INFO: include disk 'scsi0' 'FN3_IMAGES:204/vm-204-disk-0.qcow2' 100G
INFO: backup mode: snapshot
INFO: bandwidth limit: 9500 KB/s
INFO: ionice priority: 0
INFO: creating archive '/mnt/pve/FN2_BACKUP/dump/vzdump-qemu-204-2020_04_14-13_32_15.vma.lzo'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'e28f1fc2-94f8-4966-a489-cc9968446060'
INFO: status: 0% (12189696/107374182400), sparse 0% (7802880), duration 3, read/write 4/1 MB/s

iperf3

bferrell@jitsi:~$ sudo iperf3 -c 192.168.100.12
Connecting to host 192.168.100.12, port 5201
[ 4] local 192.168.100.120 port 36300 connected to 192.168.100.12 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.04 GBytes 8.89 Gbits/sec 1297 1.49 MBytes
[ 4] 1.00-2.00 sec 1.05 GBytes 9.07 Gbits/sec 512 997 KBytes
[ 4] 2.00-3.00 sec 995 MBytes 8.34 Gbits/sec 363 1.06 MBytes
[ 4] 3.00-4.00 sec 1.07 GBytes 9.19 Gbits/sec 377 1.04 MBytes
[ 4] 4.00-5.00 sec 1.03 GBytes 8.87 Gbits/sec 275 1021 KBytes
[ 4] 5.00-6.00 sec 1.07 GBytes 9.19 Gbits/sec 242 1.22 MBytes
[ 4] 6.00-7.00 sec 1.09 GBytes 9.34 Gbits/sec 0 1.74 MBytes
[ 4] 7.00-8.00 sec 1.09 GBytes 9.35 Gbits/sec 475 1.39 MBytes
[ 4] 8.00-9.00 sec 1.08 GBytes 9.31 Gbits/sec 572 1.11 MBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.34 Gbits/sec 260 1.35 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.6 GBytes 9.09 Gbits/sec 4373 sender
[ 4] 0.00-10.00 sec 10.6 GBytes 9.09 Gbits/sec receiver

iperf Done.

 

[tim]

There is a bandwidth limit configured on srvr1 -> INFO: bandwidth limit: 9500 KB/s

 

[bferrell]

Yea, I just put that and the ionice in there yesterday to see if it made any difference, and I've tried huge number and tiny numbers with really no effect. I've just rebooted the node to see if that makes any difference. I'll reset the options to default and rerun when it's back up.

 

[bferrell]

Rebooting seems to have resolved it for now, my previous hdparm result was about 10MB/s

bferrell@jitsi:~$ sudo hdparm -Tt /dev/sda
[sudo] password for bferrell:

/dev/sda:
Timing cached reads: 10680 MB in 1.99 seconds = 5357.38 MB/sec
Timing buffered disk reads: 776 MB in 3.01 seconds = 257.90 MB/sec