Asymmetric Tailscale speed issue with Kernel 7 and multiple NIC types

bossanova808 · Saturday at 04:36

Trying here as my report in the kernel thread is not getting much attention. I know of at least 5 users reporting this issue here and on Reddit - but it definitely also doesn't affect everyone.

The issue is easily tested and consistently reproducible by me with Kernel 7 and using iperf3 across Tailscale. I have reproduced it both within my local network and across interstate connections.

The specific tests below tests are run with exactly the same hardware, same network, same Tailscale version - i.e the _only_ change is the kernel version.

Kernel 7.0.2-2-pve - clear, massive regression:

Code:

10:20 user@samba:~ > iperf3 -c 100.93.240.XX -t 30
Connecting to host 100.93.240.XX, port 5201
[  5] local 100.125.133.YY port 50050 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec   52   2.40 KBytes
[  5]   1.00-2.00   sec   640 KBytes  5.24 Mbits/sec   40   2.40 KBytes
[  5]   2.00-3.00   sec   640 KBytes  5.24 Mbits/sec   46   2.40 KBytes
[  5]   3.00-4.00   sec   256 KBytes  2.10 Mbits/sec   29   1.20 KBytes
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec   29   2.40 KBytes
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  16.0 MBytes  4.47 Mbits/sec  1200            sender
[  5]   0.00-30.00  sec  16.0 MBytes  4.47 Mbits/sec                  receiver
iperf Done.
10:20 user@samba:~ > iperf3 -c 100.93.240.XX -t 30 -R
Connecting to host 100.93.240.XX, port 5201
Reverse mode, remote host 100.93.240.XX is sending
[  5] local 100.125.133.YY port 45092 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   150 MBytes  1.26 Gbits/sec
[  5]   1.00-2.00   sec   148 MBytes  1.24 Gbits/sec
[  5]   2.00-3.00   sec   153 MBytes  1.28 Gbits/sec
[  5]   3.00-4.00   sec   146 MBytes  1.22 Gbits/sec
[  5]   4.00-5.00   sec   155 MBytes  1.30 Gbits/sec
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  4.27 GBytes  1.22 Gbits/sec   16            sender
[  5]   0.00-30.00  sec  4.26 GBytes  1.22 Gbits/sec                  receiver
iperf Done.

Vs. Kernel 6.17.13-7-pve results:

Code:

10:49 user@samba:~ > iperf3 -c 100.93.240.XX -t 5
Connecting to host 100.93.240.XX, port 5201
[  5] local 100.125.133.YY port 57468 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  64.2 MBytes   539 Mbits/sec  152    562 KBytes
[  5]   1.00-2.00   sec  64.0 MBytes   537 Mbits/sec    1    631 KBytes
[  5]   2.00-3.00   sec  58.9 MBytes   494 Mbits/sec    0    690 KBytes
[  5]   3.00-4.00   sec  61.4 MBytes   515 Mbits/sec   61    534 KBytes
[  5]   4.00-5.00   sec  51.6 MBytes   432 Mbits/sec    2    597 KBytes
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   300 MBytes   503 Mbits/sec  216            sender
[  5]   0.00-5.01   sec   298 MBytes   498 Mbits/sec                  receiver

iperf Done.
10:49 user@samba:~ > iperf3 -c 100.93.240.XX -t 5 -R
Connecting to host 100.93.240.XX port 5201
Reverse mode, remote host 100.93.240.XX is sending
[  5] local 100.125.133.11 port 54136 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   150 MBytes  1.26 Gbits/sec
[  5]   1.00-2.00   sec   150 MBytes  1.26 Gbits/sec
[  5]   2.00-3.00   sec   138 MBytes  1.16 Gbits/sec
[  5]   3.00-4.00   sec   104 MBytes   870 Mbits/sec
[  5]   4.00-5.00   sec   153 MBytes  1.28 Gbits/sec
..
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   698 MBytes  1.17 Gbits/sec   50            sender
[  5]   0.00-5.00   sec   695 MBytes  1.17 Gbits/sec                  receiver

iperf Done.

However tests between two Tailscale LXCs on the same machine show great performance:

Code:

9:42 user@samba:~ > iperf3 -c ts.ip.same.machine1 -t 5
Connecting to host ts.ip.same.machine1, port 5201
[  5] local ts.ip.same.machine2 port 59294 connected to ts.ip.same.machine1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.38 GBytes  11.9 Gbits/sec    0   4.16 MBytes
[  5]   1.00-2.00   sec  1.40 GBytes  12.0 Gbits/sec    0   4.16 MBytes
[  5]   2.00-3.00   sec  1.39 GBytes  11.9 Gbits/sec    0   4.16 MBytes
[  5]   3.00-4.00   sec  1.39 GBytes  12.0 Gbits/sec    0   4.16 MBytes
[  5]   4.00-5.00   sec  1.38 GBytes  11.8 Gbits/sec    0   4.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  6.94 GBytes  11.9 Gbits/sec    0            sender
[  5]   0.00-5.00   sec  6.94 GBytes  11.9 Gbits/sec                  receiver

iperf Done.
09:43 user@samba:~ > iperf3 -c ts.ip.same.machine1 -t 5 -R
Connecting to host ts.ip.same.machine1, port 5201
Reverse mode, remote host ts.ip.same.machine1 is sending
[  5] local ts.ip.same.machine2 port 59310 connected to ts.ip.same.machine1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.39 GBytes  11.9 Gbits/sec
[  5]   1.00-2.00   sec  1.40 GBytes  12.0 Gbits/sec
[  5]   2.00-3.00   sec  1.41 GBytes  12.2 Gbits/sec
[  5]   3.00-4.00   sec  1.42 GBytes  12.2 Gbits/sec
[  5]   4.00-5.00   sec  1.41 GBytes  12.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  7.03 GBytes  12.1 Gbits/sec    0            sender
[  5]   0.00-5.00   sec  7.03 GBytes  12.1 Gbits/sec                  receiver

iperf Done

Issue affects multiple NIC types with at least these reported so far:

AQtion AQC113CS
Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
& mine:

Code:

Device Type: ConnectX4LX
Part Number: MCX4121A-ACA_Ax
Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
PSID: MT_2420110034
12:22 root@pve-homeserver25:~/manually_installed/Mellanox $ ./mlxup --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  /dev/mst/mt4117_pciconf0
  Base MAC:         0c42a12d0cd2
  Versions:         Current        Available
     FW             14.32.1912     14.32.1010
     PXE            3.6.0502       3.6.0502
     UEFI           14.25.0017     14.25.0017

  Status:           Up to date

(this is the latest Mellanox FW from: https://network.nvidia.com/products/adapter-software/firmware-tools/ and https://network.nvidia.com/support/firmware/connectx4lxen/)

Network configuration:

Proxmox host uses a standard Linux bridge (vmbr0) — no SR-IOV, no VLAN-aware bridge, etc
Physical NIC (nic2, Mellanox ConnectX-4 Lx, mlx5_core) is a bridge member of vmbr0
LXC containers connect via veth pairs through the bridge, with Proxmox firewall enabled (fwbr/fwln/fwpr chain)
Tailscale runs inside the LXC (not on the Proxmox host), so WireGuard UDP packets egress via: tailscale0 (LXC) → veth → fwbr → vmbr0 → nic2 (mlx5_core) → physical
lxc.mount.entry: /dev/net/tun passthrough (required for Tailscale in LXC)
Proxmox firewall enabled on the LXC interface (firewall=1 in LXC config) - issue still occurs with firewall disabled, though
LXCs are unprivileged, Debian 13, running basically nothing but Tailscale and samba (also effects my Jellyfin container and iperf3 performance as above!).

The issue is present and reproducible with at least these kernels:
7.0.0-3-pve
7.0.2-2-pve

The issue is NOT present with all prior kernels including (i.e. the workaround is to pin to an older kernel):
6.17.13-6-pve
6.17.13-7-pve

I have done extensive testing (mostly Calude guided, for what that is worth) - and pretty much ruled out:
ECN, congestion control, TSO/GSO/GRO, tunnel offloads, conntrack, router and NIC firmware, ISP issues

It would be great to get some eyes on this, and I am happy to run tests/supply logs etc.

Also reported to Tailscale at: https://github.com/tailscale/tailscale/issues/19777

_gabriel · Saturday at 16:35

bossanova808 said:
tests between two Tailscale LXCs on the same machine show great performance

HW NIC is not used in this test.

As Tailscale uses wireguard, have you tried a pure wireguard ?

bossanova808 · 2026-05-17T03:03:06+0200

I have just done a test with pure WireGuard, and it does not show the issue:

Code:

10:58 user@samba:~ > iperf3 -c 192.168.99.2 -t 10
iperf3 -c 192.168.99.2 -t 10 -R
Connecting to host 192.168.99.2, port 5201
[  5] local 192.168.99.1 port 60466 connected to 192.168.99.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   137 MBytes  1.15 Gbits/sec  171    585 KBytes
[  5]   1.00-2.00   sec   134 MBytes  1.12 Gbits/sec   22    521 KBytes
[  5]   2.00-3.00   sec   134 MBytes  1.12 Gbits/sec    0    680 KBytes
[  5]   3.00-4.00   sec   128 MBytes  1.07 Gbits/sec    4    632 KBytes
[  5]   4.00-5.00   sec   135 MBytes  1.13 Gbits/sec   60    566 KBytes
[  5]   5.00-6.00   sec   118 MBytes   987 Mbits/sec    0    701 KBytes
[  5]   6.00-7.00   sec   122 MBytes  1.02 Gbits/sec   28    615 KBytes
[  5]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec   48    574 KBytes
[  5]   8.00-9.00   sec   134 MBytes  1.12 Gbits/sec   24    522 KBytes
[  5]   9.00-10.00  sec   130 MBytes  1.09 Gbits/sec    0    680 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.27 GBytes  1.09 Gbits/sec  357            sender
[  5]   0.00-10.01  sec  1.27 GBytes  1.09 Gbits/sec                  receiver

iperf Done.
Connecting to host 192.168.99.2, port 5201
Reverse mode, remote host 192.168.99.2 is sending
[  5] local 192.168.99.1 port 50946 connected to 192.168.99.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   123 MBytes  1.03 Gbits/sec
[  5]   1.00-2.00   sec   122 MBytes  1.02 Gbits/sec
[  5]   2.00-3.00   sec   123 MBytes  1.03 Gbits/sec
[  5]   3.00-4.00   sec   121 MBytes  1.01 Gbits/sec
[  5]   4.00-5.00   sec   115 MBytes   968 Mbits/sec
[  5]   5.00-6.00   sec   104 MBytes   871 Mbits/sec
[  5]   6.00-7.00   sec   103 MBytes   864 Mbits/sec
[  5]   7.00-8.00   sec   120 MBytes  1.01 Gbits/sec
[  5]   8.00-9.00   sec   124 MBytes  1.04 Gbits/sec
[  5]   9.00-10.00  sec   121 MBytes  1.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.15 GBytes   988 Mbits/sec    0            sender
[  5]   0.00-10.00  sec  1.15 GBytes   986 Mbits/sec                  receiver

iperf Done.

bossanova808 · 2026-05-17T03:22:29+0200

Some more info that might be useful:

Code:

ip -d link show tailscale0
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
    link/none  promiscuity 0 allmulti 0 minmtu 68 maxmtu 65535
    tun type tun pi off vnet_hdr on persist off addrgenmode random numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536

vs.

Code:

ip -d link show wg-test
4: wg-test: <POINTOPOINT,NOARP> mtu 1420 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/none  promiscuity 0 allmulti 0 minmtu 0 maxmtu 2147483552
    wireguard addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536

_gabriel · 2026-05-17T14:50:45+0200

Have you tried with VM instead LXC ?
Have you tried directly on the host ?

bossanova808 · 2026-05-18T03:01:23+0200

VM - same results (as others have also reported).

Host - I do not want Tailscale on the host (but iperf3 performance without Tailscale is fine).

Search

Search

Asymmetric Tailscale speed issue with Kernel 7 and multiple NIC types

bossanova808

New Member

_gabriel

Distinguished Member

bossanova808

New Member

bossanova808

New Member

_gabriel

Distinguished Member

bossanova808

New Member

We value your privacy