VirtIO and Realtek RTL8125: 23% TCP throughput improvement by adjusting GSO max size

antoinemartin26

New Member
Nov 14, 2026
3
1
3
Hello everyone,

I would like to share the results of a network performance investigation on my Proxmox VE server. After extensive testing, I identified a significant throughput improvement by adjusting the maximum GSO size inside a Linux guest to match the capabilities of the underlying physical network adapter.

I am sharing this in the hope that it may help other Proxmox users and possibly lead to an improvement in the default VirtIO network configuration.

1. Hardware and software environment

- Motherboard: MSI B85-G43
- CPU: Intel Core i7-4790K (4 cores / 8 threads)
- Proxmox kernel: 7.0.14-28-pve
- Physical NICs: TP-Link TX201 (Realtek RTL8125B)
- Host network driver: native Linux r8169
- Network speed: 2.5 Gbit/s, full duplex
- Guest: Ubuntu Server
- Guest CPU configuration: host, 8 vCPUs
- Network model: VirtIO
- VirtIO multiqueue: 8 queues per interface
- vhost-net: enabled
- Packed virtqueues: enabled
- Linux bridges: vmbr0 (LAN) and vmbr1 (WAN)
- MTU: 1500 throughout the network
- Client: Fedora Linux with another RTL8125-based NIC using r8169

The Ubuntu VM also acts as an Internet gateway for my local network.

2. Initial problem

I noticed asymmetric TCP throughput between my Ubuntu VM and my Fedora workstation.

Initial iperf3 results:

Direction| Throughput
Fedora to Proxmox| 2.34 Gbit/s
Fedora to Ubuntu VM| 2.34 Gbit/s
Proxmox to Fedora| 2.14 Gbit/s
Ubuntu VM to Fedora| 1.73–1.76 Gbit/s

The issue was reproducible with multiple TCP streams and remained present when directly connecting the workstation to the Proxmox server, bypassing the physical Ethernet switch.

3. Troubleshooting

I tested several potential causes:

- Physical switch and other LAN traffic
- Native r8169 versus out-of-tree r8125 on the client
- TCP congestion control algorithms
- VirtIO multiqueue configurations (1, 2, 4 and 8 queues)
- RPS and CPU load
- Connection tracking
- Queue disciplines
- Byte Queue Limits
- GRO and TSO settings

None of these tests alone provided a significant improvement in guest-to-client throughput.

CPU monitoring also showed no sustained saturation of individual vhost or vCPU threads at the sampling resolution used.

4. Identification of a GSO size mismatch

Using ip -d link show, I observed the following maximum GSO sizes:

Interface| GSO max size
Guest VirtIO interface (ens18)| 65536
Host TAP interface (tap100i0)| 65536
Linux bridge (vmbr0)| 64000
Physical RTL8125 interface| 64000

The r8169 driver defines the following limits for supported controller generations:

#define RTL_GSO_MAX_SIZE_V2 64000
#define RTL_GSO_MAX_SEGS_V2 64

I then used tcpdump on both the TAP interface and the physical NIC to compare TCP payload sizes before hardware segmentation.

With TSO enabled inside the Ubuntu guest and the default GSO maximum size of 65536 bytes:

- The TAP interface frequently received TCP payloads of 65160 bytes.
- On the physical NIC, 19101 out of 20000 captured packets had a TCP payload of only 1448 bytes.
- TCP throughput remained around 1.76 Gbit/s.

This suggests that oversized GSO buffers were being segmented in software before reaching the physical NIC.

5. Workaround

Inside the Ubuntu guest, I applied the following settings as root:

ethtool -K ens18 gro on gso on tso on
ip link set dev ens18 gso_max_size 63000
ip link set dev ens18 gso_ipv4_max_size 63000

The value of 63000 bytes was deliberately chosen to remain below the physical NIC's 64000-byte limit.

No MTU changes, physical hardware changes or host reboot were required.

After applying this configuration, the TCP payload size distributions captured on the TAP and physical interfaces became almost identical.

For example:

TCP payload size| TAP count| Physical NIC count
62264 bytes| 8772| 8777
2896 bytes| 6638| 6635
59368 bytes| 334| 332
60816 bytes| 279| 279
57920 bytes| 254| 253

These observations are consistent with large GSO buffers being preserved through the host networking path and hardware segmentation being used more effectively.

6. Performance improvements

All local tests used iperf3 with four parallel TCP streams.

Configuration| Guest-to-client throughput
TSO disabled, original configuration| 1.73 Gbit/s
TSO enabled, GSO size 65536| 1.76 Gbit/s
TSO enabled, GSO size 63000| 2.13 Gbit/s
Native Proxmox-to-client reference| 2.14 Gbit/s

The improvement was approximately 23% compared with the original configuration.

A subsequent 60-second iperf3 test confirmed 2.13 Gbit/s with zero TCP retransmissions in the guest-to-client direction.

Since the Ubuntu VM also acts as a router, I tested Internet throughput as well:

Measurement| Before| After
Download| 1761.56 Mbps| 2135.97 Mbps
Upload| 2117.67 Mbps| 2105.34 Mbps
Idle latency| 11 ms| 11 ms
Download loaded latency| 22 ms| 11 ms
Upload loaded latency| 11 ms| 11 ms

The Internet download improvement was approximately 21%.

I subsequently applied the same GSO limits to the WAN VirtIO interface for consistency, although this did not produce a measurable additional throughput improvement.

7. Questions for the Proxmox developers and community

- Is this GSO size mismatch between VirtIO and the underlying physical NIC a known performance issue?
- Could the guest's effective GSO limits be negotiated or configured to better accommodate the physical network path?
- Would it make sense to expose an optional maximum GSO size setting in Proxmox for VirtIO interfaces?
- Is this something that should instead be investigated upstream by the Linux networking, VirtIO or QEMU developers?

I understand that the physical network path can change and that automatically selecting a suitable GSO limit may not be straightforward.

I am not claiming that this is necessarily a Proxmox bug. However, the performance difference and packet captures suggest that this interaction is worth investigating.

I have kept the workaround as a persistent systemd service inside the Ubuntu guest.

I would be happy to provide additional configuration details, command outputs or testing results if useful.

Thank you for your work on Proxmox and for maintaining such a great open-source virtualization platform.

Best regards,
A Proxmox home server user from France
 
  • Like
Reactions: waltar
For completeness, here are the exact software versions used during my investigation.

Proxmox host:

  • Proxmox VE: 9.2.0
  • PVE Manager: 9.2.26
  • Kernel: 7.0.14-28-pve
  • pve-qemu-kvm: 11.1.3-1
  • qemu-server: 9.2.15
  • Debian: 13.7 (Trixie)
  • iperf3: 3.18
  • Network driver: r8169
Ubuntu guest:

  • Ubuntu Server 26.04.1 LTS (Resolute Raccoon)
  • Kernel: 7.0.0-43-generic
  • iperf3: 3.20
  • ethtool: 6.19
  • iproute2: 6.19.0
  • Network driver: virtio_net
Fedora workstation:

  • Fedora 45
  • CachyOS kernel: 7.3.3-1.cachyos1.fc45.x86_64
  • iperf3: 3.22
  • Network driver: r8169
  • NIC firmware: rtl8125b-2_0.0.2 (07/13/20)
The workaround is now persistent through a systemd service inside the Ubuntu guest, with the following settings applied to both LAN and WAN VirtIO interfaces:

  • TSO, GSO and GRO enabled
  • gso_max_size: 63000
  • gso_ipv4_max_size: 63000
  • MTU: 1500
  • VirtIO multiqueue: 8
No permanent changes to the Proxmox host were required.
 
VirtIO negotiates whether the virtual device can accept TCP segmentation offload from the guest. The transmit header’s gso_size is the desired TCP segment payload size (MSS). The physical uplink’s maximum GSO buffer size sits outside that negotiation. That explains why the guest’s 65,536-byte limit can coexist with the lower limit on your RTL8125 path. VirtIO 1.4, network device

For an upstream report, I’d repeat the guest-to-Fedora test at both caps and take a short net:net_dev_start_xmit trace filtered to the host’s RTL8125 interface. Compare len, gso_size and gso_segs at driver entry. If the default run shows large buffers at the TAP but MTU-sized buffers at driver entry, while the 63,000-byte run still hands GSO buffers to the driver, that narrows down where segmentation occurs. The host trace and the versions you posted would give maintainers a reproducible starting point. Linux trace fields · Driver handoff in Linux

OpenAI Codex assisted with drafting this reply.
 
Thank you again, sanoc_mars, for your suggestion.

I repeated the tests after rebooting my server and collected the net tracepoint using the kernel's built-in ftrace infrastructure.

Both tests were performed with TSO, GSO and GRO enabled. Only gso_max_size and gso_ipv4_max_size were changed on the Ubuntu guest's LAN VirtIO interface.

Test methodology

  • iperf3: four parallel TCP streams, 30 seconds, reverse mode (guest to workstation).
  • Tracepoint: net
  • Trace filter: name == "nic0" && protocol == 0x0800
  • Trace duration: 200 ms
  • Independent ftrace instance
  • Buffer size: 8192 KB per CPU
No trace buffer overwrites were observed in either capture.

Test A — GSO maximum size: 65536

TCP throughput:

  • Sender: 1.75 Gbit/s
  • Receiver: 1.74 Gbit/s
  • Retransmissions: 0
Total trace events: 16975

Most frequent trace entries (count, len, gso_size, gso_segs):

16432 len=1514 gso_size=0 gso_segs=0
26 len=63778 gso_size=1448 gso_segs=44
25 len=44954 gso_size=1448 gso_segs=31
24 len=55090 gso_size=1448 gso_segs=38
23 len=59434 gso_size=1448 gso_segs=41
Approximately 96.8% of all captured transmission events were MTU-sized non-GSO buffers.

Test B — GSO maximum size: 63000

TCP throughput:

  • Sender: 2.13 Gbit/s
  • Receiver: 2.13 Gbit/s
  • Retransmissions: 0
Total trace events: 1024

Most frequent trace entries:

711 len=62330 gso_size=1448 gso_segs=43
26 len=2962 gso_size=1448 gso_segs=2
21 len=60882 gso_size=1448 gso_segs=42
21 len=59434 gso_size=1448 gso_segs=41
20 len=66 gso_size=0 gso_segs=0
17 len=43506 gso_size=1448 gso_segs=30
Approximately 69.4% of all captured transmission events were 62330-byte GSO buffers containing 43 TCP segments.

Summary

MeasurementGSO 65536GSO 63000
TCP sender throughput1.75 Gbit/s2.13 Gbit/s
TCP retransmissions00
Driver-entry trace events169751024
Most frequent buffer length151462330
Most frequent gso_size01448
Most frequent gso_segs043
The throughput improvement was approximately 21.7%, while the number of observed driver-entry transmission events decreased by approximately 94%.

Combined with my previous tcpdump captures on the TAP and physical interfaces, these results strongly support the hypothesis that oversized GSO buffers are being segmented in software before reaching the r8169 driver.

The trace filter includes all IPv4 transmissions on nic0, not exclusively iperf3 traffic. Also, the earlier TAP captures were not collected simultaneously with these kernel traces.

The workaround remains effective after reboot, and I have restored the optimized guest configuration.

I have retained both raw trace files and can provide them if needed.

Do you think these results would be sufficient for an upstream Linux networking report?

Thank you for helping me investigate this further!