VM with NAT and performance issues

[kbechler]

Hello,

I've got fairly simple infrastructure with a single PVE host (OK, it's a member of bigger cluster, but it's not a case) and two Debian-based VMs.
On of these VMs (called gw) has two network interfaces and acts as a network gateway for a second one. Second VM (called test) has a single interface only:

+==========proxmox host============+
|                                  |
|  +---vm1------+     +---vm2---+  |
|  |            |     |         |  |
|  |            |     |         |  |
|  |            |     |         |  |
|  +-eth0--eth1-+     +-eth0----+  |
|     |      |           |         |
|     |      |           |         |
|     |      +-vmbr1.200-+         |
|   vmbr1                          |
|     |                            |
|     |                            |
|     |                            |
+===enp2s0f1np1====================+

And here's my problem:

root@test:~# ./speedtest --progress=no
Idle Latency:     4.09 ms   (jitter: 0.09ms, low: 4.02ms, high: 4.22ms)
    Download:     8.12 Mbps (data used: 14.0 MB)                                                   
                  4.08 ms   (jitter: 0.14ms, low: 3.94ms, high: 5.42ms)
      Upload:   772.71 Mbps (data used: 588.0 MB)                                                   
                 15.57 ms   (jitter: 0.78ms, low: 3.98ms, high: 17.94ms)
 Packet Loss:     0.0%


root@gw:~# ./speedtest --progress=no
Idle Latency:     3.89 ms   (jitter: 0.02ms, low: 3.88ms, high: 3.93ms)
    Download:   830.90 Mbps (data used: 1.5 GB)                                                   
                  4.00 ms   (jitter: 0.14ms, low: 3.90ms, high: 6.82ms)
      Upload:   742.85 Mbps (data used: 484.8 MB)                                                   
                 16.29 ms   (jitter: 0.95ms, low: 4.00ms, high: 18.74ms)
 Packet Loss:     0.0%

My uplink should be around 1Gbps, so it works fine from gw. But download speed from "test" VM is terrbily slow.
Never seen such issue, and I cannot find a way to fix it :-(

Proxmox host was rebooted, VMs were re-created from a scratch (from Debian 12 ISO and Ubuntu 24.04 cloud image).
Without any change here.



Proxmox host:

# cat /etc/network/interfaces
iface enp2s0f0np0 inet manual
iface enp2s0f1np1 inet manual

auto vmbr0
iface vmbr0 inet static
    address a.b.c.d
    gateway e.f.g.h
    bridge-ports enp2s0f0np0
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp2s0f1np1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr1.200
iface vmbr1.200 inet static

First VM:

# cat /etc/network/interfaces
allow-hotplug ens18
iface ens18 inet static
    address x.y.z.105/27
    gateway x.y.z.126
    dns-nameservers 8.8.8.8

iface ens19 inet static
    address 10.10.10.1/24

# history
      iptables -t nat -A POSTROUTING -o ens18 -j MASQUERADE
      sysctl net.ipv4.conf.all.forwarding=1
      sysctl net.ipv4.conf.all.rp_filter=0

Second VM:

# cat /etc/network/interfaces
allow-hotplug ens18
iface ens18 inet static
    address 10.10.10.2/24
    gateway 10.10.10.1
    dns-nameservers 8.8.8.8

 

[kbechler]

Communication between VMs also looks fine:

root@test:~# iperf3 -c 10.10.10.1 --bidir
Connecting to host 10.10.10.1, port 5201
[  5] local 10.10.10.2 port 60972 connected to 10.10.10.1 port 5201
[  7] local 10.10.10.2 port 60986 connected to 10.10.10.1 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[...]           
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  34.3 GBytes  29.4 Gbits/sec    2             sender
[  5][TX-C]   0.00-10.00  sec  34.3 GBytes  29.4 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  35.3 GBytes  30.3 Gbits/sec    0             sender
[  7][RX-C]   0.00-10.00  sec  35.3 GBytes  30.3 Gbits/sec                  receiver

iperf Done.