pveversion:
Environment:
Problem:
Throughput to and from the VMs is very limited, but only on a per-connection basis it seems. Single connections cannot achieve over 10mbps; there is a hard limit where I see it pegged at 10.1 Mbps in iperf.
However, when I run the tests multi-threaded, I can get the full throughput of the server's connection.
Tests with iperf in both directions:
As a final test to rule out opnsense or wireguard, I momentarily assigned the public IP directly to the 'test server' and connected it on br0, where I still saw the same issue. Since then I have been investigating if the problem is in Proxmox and my virtualization settings... It's just really weird that it's only a per-thread cap with a very clear limit at 10mbps. It feels like a configuration setting but I haven't found what it could be.
I also disabled proxmox firewall globally during the tests.
Is there something weird about my configuration that's causing this behavior? I've found a few other people with similar setups in the last year who appeared to have run into the same situation, but I didn't see a resolution to any of those.
I'm really stumped on this, any ideas would be appreciated. Thanks!
Code:
# pveversion --verbose
proxmox-ve: 7.4-1 (running kernel: 5.15.107-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1Environment:
Code:
Proxmox VE - Dedicated 4c 8t, 32gb, 250mbps
1 Physical Nic
eth0: manual config
2 Linux Bridges
br0 - ports: eth0
No IP
br1 - ports: none
IP 10.10.10.10, Gateway 10.10.10.1 (opnsense LAN IP)
connects all the vms, opnsense dhcp server on this bridge
3 VMs (all use host CPU type)
opnsense
two virtio nics
WAN connected to br0 with host nic
IP Public
LAN connected to br1 with vms, dhcp serving 10.10.10.0/24
IP 10.10.10.1
wireguard interface wg0
IP 10.20.20.1
nat'd to lan
listen on WAN IP
file server
one virtio nic
eth0 connected to br1
DHCP IP 10.10.10.101
test server
one virtio nic
eth0 connected to br1
DHCP IP 10.10.10.102
Storage
Local storage 4TB LVM on Soft Raid10 (HDDs)
VMs all use virtio scsi with ide drives
VPN Client
Connected to wg0 through opnsense wan
IP 10.20.20.100, Gateway 10.10.10.1
Public Client
Connecting through internet to opnsense WAN IP, ports forwarded to LAN for iperf testsProblem:
Throughput to and from the VMs is very limited, but only on a per-connection basis it seems. Single connections cannot achieve over 10mbps; there is a hard limit where I see it pegged at 10.1 Mbps in iperf.
However, when I run the tests multi-threaded, I can get the full throughput of the server's connection.
Tests with iperf in both directions:
Code:
wireguard client <-> file server - BAD
Connections were initially not getting over ~7mbps for one thread
But if I run 25-30 threads I can get the full bandwidth
Tweaked MTUs on the interfaces and TCP windows/buffers in the VMs, and achieved exactly 10.1 Mbps on every thread.
file server <-> test server - GOOD (both within same lan)
public client <-> proxmox ve - GOOD (momentarily assigned the public IP to proxmox for this test)
public client <-> test server - BAD (port forward through opnsense WAN)
public client <-> opnsense WAN - BADI also disabled proxmox firewall globally during the tests.
Is there something weird about my configuration that's causing this behavior? I've found a few other people with similar setups in the last year who appeared to have run into the same situation, but I didn't see a resolution to any of those.
I'm really stumped on this, any ideas would be appreciated. Thanks!
Last edited: