Why would the bandwidth throttle down? (VM with wireguard) (video)

[jpdm]

10.0.0.1 - VPS outside LAN
10.0.0.2 - VM inside proxmox with wireguard (within LAN)
192.168.1.112 - NAS without wireguard (within LAN)

Video: https://reddit.com/link/wp26ec/video/85rm42pn7wh91/player

10.0.0.1 <--> 192.168.1.112 bandwidth is throttling as show in my video (traffic routes through 10.0.0.2)
10.0.0.1 <--> 10.0.0.2 bandwidth is OK (LAN to VPS, wireguard to wireguard)
10.0.0.2 <--> 192.168.1.112 bandwidth is OK (both within LAN)

I tried setting up a new VM outside proxmox (virtual box on a windows machine) and installed wireguard and used that as the tunnel, and the bandwidth speed is OK. The problem is when VM with wireguard is within proxmox.

What could be the causing the bandwidth to throttle down?

root@pve1:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.39-3-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-8
pve-kernel-helper: 7.2-8
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

 

[fabian]

check MTUs, retransmissions, error counters - but unless you explicitly configured a ratelimit in PVE there is nothing intentionally slowing down network traffic.

 

[jpdm]

check MTUs, retransmissions, error counters - but unless you explicitly configured a ratelimit in PVE there is nothing intentionally slowing down network traffic.

MTU on all nodes, router and NAS is 1500 except for both VPS and UBUNTU-VM, 1420 wireguard default. Tried different MTUs, nothing improved.

No rate limit set in PVE. I tested wireguard to wireguard and the bandwidth is okay. Only when traffic from LAN (NAS) routed through wireguard to VPS is the bandwidth slowed down, but this only happens when wireguard is in a PVE VM / Host. A regular VM outside PVE with wireguard has no bandwidth problem when routing traffic.

I also tried using the PVE host as the wireguard peer, and the bandwidth is still being slowed down. I removed proxmox and installed ubuntu bare metal then wireguard, used that as peer and the routed traffic bandwidth is okay.

The next thing I'll do is just install debian bare metal and wireguard on that same machine (previously proxmox). If there's no problem with the bandwidth with that setup, I think it's a most likely a proxmox bug or a PVE kernel issue.

The bandwidth throttling down only happens when proxmox is involved (host or vm).

 

[fabian]

that does seem strange.. do you have the pve-firewall active? could you include the network config of the PVE host when using it directly as WG peer?

 

[jpdm]

that does seem strange.. do you have the pve-firewall active? could you include the network config of the PVE host when using it directly as WG peer?

Both PVE and Datacenter firewalls are disabled.

/etc/network/interfaces

auto lo
iface lo inet loopback

iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.14/24
        gateway 192.168.1.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

iface wlp3s0 inet manual

/etc/wireguard/wg0.conf

[Interface]
#PVEHost
PrivateKey = <private key>
Address = 10.0.0.2/24
ListenPort = 123123
DNS = 1.1.1.1
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o vmbr0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o vmbr0 -j MASQUERADE

[Peer]
#VPS
PublicKey = <public key>
Endpoint = <VPS Public IP>:51820
AllowedIPs = 10.0.0.0/24
PersistentKeepalive = 25

 

[fabian]

you could double check that the same module is used for the NIC in both PVE and other distro cases, and verify the various offloading/.. settings are identical with ethtool.

 

[jpdm]

you could double check that the same module is used for the NIC in both PVE and other distro cases, and verify the various offloading/.. settings are identical with ethtool.

NIC comparison between PVE Host and Ubuntu 20.04 LTS (bare metal install using same PVE hardware):



Only difference I can see above that could be affecting this issue is the kernel driver, that's why I'm thinking this could be a kernel bug/issue.

Here's the ethtool settings comparison between PVE host and UBUNTU-VM (not the bare metal install above, but same ethtool settings where bandwidth is okay). To the left is the PVE host and to right is the UBUNTU-VM (where I can use wireguard peer to route and bandwidth is okay):



Here's the ethtool comparison between the PVE host and ubuntu bare metal install (using same hardware as PVE host):



I made sure to change the PVE host settings to be as similar as possible to the ubuntu-vm and ubuntu bare metal install. I then did another test but nothing changed. Bandwidth still being throttled down on PVE wireguard install.

Just to reiterate, both the VM (outside proxmox) and ubuntu bare metal install can route through wireguard without the bandwidth being throttled. Only when the PVE host or vm is involved is when the bandwidth throttling happens.

 

[fabian]

modinfo would also be interesting (to see firmware versions).

 

[jpdm]

modinfo would also be interesting (to see firmware versions).

modinfo comparison between PVE and the ubuntu bare metal install:



ethtool -i comparison between PVE and the ubuntu bare metal install:



I also installed a new Debian 11.4 bare metal on the same PVE hardware, installed wireguard, did the same tests in the video and the bandwidth throttling also happens. The next thing I did is install Debian 11.4 on virtual box (outside PVE), installed wireguard, did the same tests and there's no bandwidth throttling.

In summary:

PVE host with wireguard: Bandwidth throttled
PVE VM with wireguard: Bandwidth throttled
Bare metal Debian 11.4 with wireguard (on same PVE host hardware): Bandwidth throttled
Bare metal Ubuntu 20.04.4 LTS with wireguard (on same PVE host hardware): OK
Vbox VM Debian 11.4 with wireguard (on a windows PC): OK
Vbox VM Ubuntu 20.04.4 with wireguard (on a windows PC): OK

What do you think could be the issue? Because of the issue happening in a Debian bare metal install above, it's looking more like Debian driver related although I don't know how to tell if the driver/firmware is up to date or if it can be updated.

 

[fabian]

something common to Debian and PVE userspace (since they don't have the same kernel) that affects your NIC (since the Debian VM is fine on another host), but that's just guess work now.. I'd do a packet dump with tcpdump next and see whether anything stands out there between the working and non-working setups..

 

[jpdm]

something common to Debian and PVE userspace (since they don't have the same kernel) that affects your NIC (since the Debian VM is fine on another host), but that's just guess work now.. I'd do a packet dump with tcpdump next and see whether anything stands out there between the working and non-working setups..

I'll try tcpdump later, that's going to be a challenge to read for a noob like me.

I just noticed in the driver date from this image is dated 02/26/15. Is there a way to update this (specifically in Debian)?

 

[fabian]

both PVE (doesn't work) and Ubuntu (works) use the same firmware, so that can't be the cause..

 

[jpdm]

both PVE (doesn't work) and Ubuntu (works) use the same firmware, so that can't be the cause..

Ah yes, that's correct.

 

[jpdm]

something common to Debian and PVE userspace (since they don't have the same kernel) that affects your NIC (since the Debian VM is fine on another host), but that's just guess work now.. I'd do a packet dump with tcpdump next and see whether anything stands out there between the working and non-working setups..

I tried tcpdump though. The only thing I noticed (and possibly understood) is that the packet length/segment size (on Debian baremetal) mostly don't follow the MTU size settings. Packet length sent were in the range of 6000-11000 bytes, even though the MTU is set to 1500 (PC/NAS), and 1420 on the wireguard config. So I was thinking that could be it, so I tried MSS clamping and then confirmed that packets are sent in the fixed size I clamped it to. The result? Same, bandwidth still throttled.

I did some more googling and found a thread where they talk about long time issues about RealTek NICs and the shitty quality it provides (at least in some cases like this). The post also specifically mentions the NIC in my machine (RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller), and the drivers not being updated/fixed in some of the distros (so I'm guessing this is the case with Debian). I just think that there could be a difference in how it was implemented in Ubuntu compared to Debian. There's probably be some improvements in Ubuntu that somehow fixed this issue (I just don't know how to pinpoint or determine that fix)

Anyway, just to close this one, the solution that worked for me is to not use the built LAN port and instead bought a USB to LAN adapter as replacement. It now works as I want it to.

 

[Azzola]

Hi, i found this thread after many attempts, i think this is also my case, Proxmox 7 is up to date and i have a similar behavior... has been addressed this problem in a way not implying a NIC change?
I have to find a Proxmox compatible NIC with M.2 socket (shortest format), with wiring pins lying for lack of space. USB NIC is not an options for me, had too much unstable experiences in the past.

 

[jpdm]

Hi, i found this thread after many attempts, i think this is also my case, Proxmox 7 is up to date and i have a similar behavior... has been addressed this problem in a way not implying a NIC change?
I have to find a Proxmox compatible NIC with M.2 socket (shortest format), with wiring pins lying for lack of space. USB NIC is not an options for me, had too much unstable experiences in the past.

Unfortunately, that's the only thing that worked in my case (RTL8111/8168/8411 NIC + Debian + Wireguard = Bad). I'd suggest try it out with a USB NIC first to see if it's really the same case.

 

[Azzola]

Just found the main NIC is Intel (instead of the second, dedicated to the bridge, being Realtek RTL 8111), tried to swap the bridge so the VM uses the Intel NIC but it doesnt seem to solve my issue... could you provide your MTU configs? i've also tried to set MTUs at the different stages like this, holding a 30 byte for overhead between everyone:

NIC MTU 1500
Bridge MTU: 1470
VM NIC MTU: 1445
WG server config MTU: 1410

No gain...