[SOLVED] Very poor LXC network performance

[ku4eto]

Hey folks. Im gonna need some assistance in debugging and possibly fixing this issue.

I have an Proxmox on a Lenovo SR550 host, with an LXC running OL 8.5.
The LXC has an Bacula Server on it.
We are observing:
1. Slow network transfer speeds of bacula - maximum 250Mbps.
2. Network failuers (timeouts/connection reset by peer) when running big network file transfers.
3. Affects ONLY ingress traffic towards the Bacula LXC. Egress traffic from it is mostly fine (or at least not 10x slower).
4. Network transfer speeds may start high (near 1Gbps), but go down pretty fast.

Host information:

root@pve:~# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz
Stepping:                        7

root@pve:~# free -m
               total        used        free      shared  buff/cache   available
Mem:           63996       21853       12703         847       29440       40573

1x occupied 1Gbps ethernet port to Access Mode.

PVE information:

pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve)
root@pve:~# uname -a
Linux pve 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100) x86_64 GNU/Linux
root@pve:~# cat /etc/debian_version
11.6

Storage information:
2x4TB SAS HDD, running in RAID1.
PVE OS and LVM storage are both located on the RAID1.


LXC Contaier info:

arch: amd64
cores: 4
features: fuse=1,mount=nfs;cifs,nesting=1
hostname: bacula.ZZZ
memory: 4096
mp0: local-lvm:vm-108-disk-3,mp=/mnt/storage,size=1400G
net0: name=eth0,bridge=vmbr0,firewall=1,gw=X.X.X.X.133,hwaddr=3E:D6:F9:B2:2D:68,ip=X.X.X.X.254/24,type=veth
net1: name=eth1,bridge=vmbr0,firewall=1,hwaddr=6E:98:5D:BA:8E:5D,ip=Y.Y.Y.89/16,type=veth
onboot: 1
ostype: centos
rootfs: local-lvm:vm-108-disk-2,size=8G
swap: 1024
tags: ol8

Performance tests:

Disk tests:
run directly on the LXC container. Performance is the same as the host.

[root@bacula /mnt/storage]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4096k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 --direct=1
test: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.19
clock setaffinity failed: Invalid argument
clock setaffinity failed: Invalid argument
clock setaffinity failed: Invalid argument
Starting 1 process
Jobs: 1 (f=1): [m(1)][94.4%][r=179MiB/s,w=59.8MiB/s][r=44,w=14 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=224836: Sat Jan 13 10:34:22 2024
  read: IOPS=45, BW=183MiB/s (192MB/s)(3000MiB/16398msec)
   bw (  KiB/s): min=130549, max=236621, per=98.92%, avg=185309.61, stdev=28990.91, samples=31
   iops        : min=   31, max=   57, avg=44.42, stdev= 7.10, samples=31
  write: IOPS=16, BW=66.8MiB/s (70.1MB/s)(1096MiB/16398msec); 0 zone resets
   bw (  KiB/s): min=24478, max=106071, per=98.84%, avg=67646.90, stdev=26073.74, samples=31
   iops        : min=    5, max=   25, avg=15.58, stdev= 6.39, samples=31
  cpu          : usr=0.28%, sys=8.03%, ctx=1917, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.1%, >=64=93.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=750,274,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=3000MiB (3146MB), run=16398-16398msec
  WRITE: bw=66.8MiB/s (70.1MB/s), 66.8MiB/s-66.8MiB/s (70.1MB/s-70.1MB/s), io=1096MiB (1149MB), run=16398-16398msec

Disk stats (read/write):
    dm-14: ios=48814/17879, merge=0/0, ticks=10439368/5696676, in_queue=16136044, util=98.55%, aggrios=49643/18043, aggrmerge=0/0, aggrticks=10520540/5791300, aggrin_queue=16311840, aggrutil=99.93%
    dm-4: ios=49643/18043, merge=0/0, ticks=10520540/5791300, in_queue=16311840, util=99.93%, aggrios=24822/9021, aggrmerge=0/0, aggrticks=5261990/2895632, aggrin_queue=8157622, aggrutil=99.94%
    dm-2: ios=2/0, merge=0/0, ticks=3536/0, in_queue=3536, util=21.18%, aggrios=12731/4569, aggrmerge=36916/13531, aggrticks=2612828/1442517, aggrin_queue=4055346, aggrutil=99.96%
  sda: ios=12731/4569, merge=36916/13531, ticks=2612828/1442517, in_queue=4055346, util=99.96%
  dm-3: ios=49643/18043, merge=0/0, ticks=10520444/5791264, in_queue=16311708, util=99.94%

Network performance tests were run from a VM located in a vCenter cluster. The VM has both public and private IP addresses on a single interface. Runs CentOS 7.9.

Network tests with IPERF3:
no issues when testing between vCenter VM > PVE host or vCenter VM > LXC, regardless of direction. Both were able to reach almost a full 1Gbps (around 950Mbps). Test time was 120s (so 120 samples and 2 min dur.). Tested with both interfaces/networks.

Network tests with SCP:
Here is where the big difference comes. Transferring with SCP between the vCenter VM > LXC on either interfaces comes with huge slowdown compared to a Ubuntu 22.04 (kernel 6.5) VM running on the same Proxmox host, with 2 interfaces both in the same networks as the LXC and vCenter VM.

vCenter VM > Proxmox host, private IP (the only thing the host has directly attached).

[root@gitss backups]# scp registry.tar.gz root@Y.Y.Y.1:/root/
registry.tar.gz             100% 7056MB 111.1MB/s   01:03

vCenter VM > LXC, private IP interface.

[root@gitss backups]# scp registry.tar.gz root@Y.Y.Y.89:/mnt/storage/
registry.tar.gz               100% 7056MB  18.0MB/s   06:33

vCenter VM > LXC, public IP interface.

[root@gitss backups]# scp registry.tar.gz root@X.X.X.254:/mnt/storage/
registry.tar.gz          100% 7056MB  12.2MB/s   09:38

vCenter VM > Proxmox Ubuntu VM, private IP interface (virtio drives, LSI controller, poor disk perf on scsi drives).

[root@gitss backups]# scp registry.tar.gz ubuntu@Y.Y.Y.79:/home/ubuntu/
registry.tar.gz             100% 7056MB 103.7MB/s   01:08

vCenter VM > Proxmox Ubuntu VM, public IP interface (virtio drives, LSI controller, poor disk perf on scsi drives).

[root@gitss backups]# scp registry.tar.gz ubuntu@X.X.X.226:/home/ubuntu/
registry.tar.gz              100% 7056MB 110.3MB/s   01:03

Things i tried:
Changing between firewall=0 and firewall=1, no difference.
Changing between mount=none and mount=cifs,nfs, no difference.
Changing kernel.tcp_keepalives_* on the vCenter VM, but no difference between values.

If we cant find a solution to this, we will have to move the Bacula service from the LXC to an VM. Which is undesirable, since its going to be more work.

 

[Pifouney]

Hey,
Primilarly, it s seems (for me) an network load error WHEN real load for compression /bckp/filetranfrt rate.

Can you launch a glances monitoring in your proxmox host when your lxc container is loaded by bacula ?

If you have a big io-wait, all explained by this ( process paused while io not properly wroten, same for network work)

If no io-wait, check pve host log for any line about reseting host adapter. If yes, very probably a hardware network prblm ( card or network road )
Hope this can help

 

[ku4eto]

Hey,
Primilarly, it s seems (for me) an network load error WHEN real load for compression /bckp/filetranfrt rate.

Can you launch a glances monitoring in your proxmox host when your lxc container is loaded by bacula ?

If you have a big io-wait, all explained by this ( process paused while io not properly wroten, same for network work)

If no io-wait, check pve host log for any line about reseting host adapter. If yes, very probably a hardware network prblm ( card or network road )
Hope this can help

Hello,
LXC does not report IO Wait in the GUI dashboard.
During the transfer, there is very small 3-4% IO wait reported by top from the host and the LXC.

What do you mean by "compression" in your first sentence?

The PVE kernel log is getting filled with some messages:

[3411663.559466] device tap114i0 entered promiscuous mode
[3411663.629271] vmbr0: port 7(fwpr114p0) entered blocking state
[3411663.629275] vmbr0: port 7(fwpr114p0) entered disabled state
[3411663.629343] device fwpr114p0 entered promiscuous mode
[3411663.629407] vmbr0: port 7(fwpr114p0) entered blocking state
[3411663.629409] vmbr0: port 7(fwpr114p0) entered forwarding state
[3411663.639516] fwbr114i0: port 1(fwln114i0) entered blocking state
[3411663.639520] fwbr114i0: port 1(fwln114i0) entered disabled state
[3411663.639593] device fwln114i0 entered promiscuous mode
[3411663.639636] fwbr114i0: port 1(fwln114i0) entered blocking state
[3411663.639638] fwbr114i0: port 1(fwln114i0) entered forwarding state
[3411663.648692] fwbr114i0: port 2(tap114i0) entered blocking state
[3411663.648696] fwbr114i0: port 2(tap114i0) entered disabled state
[3411663.648784] fwbr114i0: port 2(tap114i0) entered blocking state
[3411663.648786] fwbr114i0: port 2(tap114i0) entered forwarding state
[3411746.545817] fwbr114i0: port 2(tap114i0) entered disabled state
[3411746.592110] fwbr114i0: port 1(fwln114i0) entered disabled state
[3411746.592546] vmbr0: port 7(fwpr114p0) entered disabled state
[3411746.592922] device fwln114i0 left promiscuous mode
[3411746.592929] fwbr114i0: port 1(fwln114i0) entered disabled state
[3411746.632862] device fwpr114p0 left promiscuous mode
[3411746.632874] vmbr0: port 7(fwpr114p0) entered disabled state

I have not done anything during the period those were written (yesterday). They do not show up while i was testing the network transfer.

 

[ku4eto]

Tested on brand new stock LXC with some stock image. Same results.

 

[ku4eto]

Never mind.
Turns out, the issue is caused by a bug in PVE.
SCP while writing to disk also buffers to RAM cache.
Once the RAM cache is filled, it drops the speed significantly.
Not only that, but this is the reason for several OOMKillers activations, killing the Bacula Service processes.

Bug is fixed with PVE with "pve-container 4.4-4" version. 7.3.2 PVE has 4.4-2. There is an workaround targetting the LXC cgroup and memory.max values of it. Buts a dirty fix, not meant for production use.

 

[stavie131]

Experiencing the same issue, can you dumb down the dirty fix so I can try implementing it? I don't understand everything that is said.

 

[ku4eto]

Experiencing the same issue, can you dumb down the dirty fix so I can try implementing it? I don't understand everything that is said.

If you are on PVE 7.0-7.3.x, the best solution is to upgrade to 7.4 or 8.0. Thats the fix.

The workaround is to apply a custom value to a volatile file on the PVE host directly.

echo "${integer}" > /sys/fs/cgroup/lxc/"$(LXC_ID)"/memory.high
echo "${integer}" > /sys/fs/cgroup/lxc/$(LXC_ID)"/ns/memory.high

"${integer}" is a placeholder for a byte integer string, at which the RAM cache will stop being filled and start being cleared/evicted.
"${LXC_ID}" is a placeholder for the LXC container ID.
As mentioned, the files are volatile. Once you reboot the PVE or the LXC, the contents of the files will be restored to the default value.

 

[dlasher]

echo "${integer}" > /sys/fs/cgroup/lxc/"$(LXC_ID)"/memory.high
echo "${integer}" > /sys/fs/cgroup/lxc/$(LXC_ID)"/ns/memory.high

"${integer}" is a placeholder for a byte integer string, at which the RAM cache will stop being filled and start being cleared/evicted.
"${LXC_ID}" is a placeholder for the LXC container ID.
As mentioned, the files are volatile. Once you reboot the PVE or the LXC, the contents of the files will be restored to the default value.

So pulling this thread in an attempt to fix SMB performance issues under 8.0.x - my samba container has the following values:

root@pmx3:~# cat /sys/fs/cgroup/lxc/5209/memory.high
8522825728
root@pmx3:~# cat /sys/fs/cgroup/lxc/5209/ns/memory.high
max

How would you change that? It's a container with 8G of ram allocated. What would you change the values to? Is this even valid any more?

I also see : https://github.com/lxc/lxc/issues/4122

 

[ku4eto]

So pulling this thread in an attempt to fix SMB performance issues under 8.0.x - my samba container has the following values:

root@pmx3:~# cat /sys/fs/cgroup/lxc/5209/memory.high
8522825728
root@pmx3:~# cat /sys/fs/cgroup/lxc/5209/ns/memory.high
max

How would you change that? It's a container with 8G of ram allocated. What would you change the values to? Is this even valid any more?

I also see : https://github.com/lxc/lxc/issues/4122

Not sure, we are yet to upgrade our PVE from 7.3 to 8.0.
Will see next week if we can do it, to get it off our heads.

 

[mihaib]

I am on 8.2.7 and I have the same issue. the LXC container gets throttled when making traffic.
I tried the suggested workaround but no luck. same behaviour