Hello everyone,
I would like to share the results of a network performance investigation on my Proxmox VE server. After extensive testing, I identified a significant throughput improvement by adjusting the maximum GSO size inside a Linux guest to match the capabilities of the underlying physical network adapter.
I am sharing this in the hope that it may help other Proxmox users and possibly lead to an improvement in the default VirtIO network configuration.
1. Hardware and software environment
- Motherboard: MSI B85-G43
- CPU: Intel Core i7-4790K (4 cores / 8 threads)
- Proxmox kernel: 7.0.14-28-pve
- Physical NICs: TP-Link TX201 (Realtek RTL8125B)
- Host network driver: native Linux r8169
- Network speed: 2.5 Gbit/s, full duplex
- Guest: Ubuntu Server
- Guest CPU configuration: host, 8 vCPUs
- Network model: VirtIO
- VirtIO multiqueue: 8 queues per interface
- vhost-net: enabled
- Packed virtqueues: enabled
- Linux bridges: vmbr0 (LAN) and vmbr1 (WAN)
- MTU: 1500 throughout the network
- Client: Fedora Linux with another RTL8125-based NIC using r8169
The Ubuntu VM also acts as an Internet gateway for my local network.
2. Initial problem
I noticed asymmetric TCP throughput between my Ubuntu VM and my Fedora workstation.
Initial iperf3 results:
Direction| Throughput
Fedora to Proxmox| 2.34 Gbit/s
Fedora to Ubuntu VM| 2.34 Gbit/s
Proxmox to Fedora| 2.14 Gbit/s
Ubuntu VM to Fedora| 1.73–1.76 Gbit/s
The issue was reproducible with multiple TCP streams and remained present when directly connecting the workstation to the Proxmox server, bypassing the physical Ethernet switch.
3. Troubleshooting
I tested several potential causes:
- Physical switch and other LAN traffic
- Native r8169 versus out-of-tree r8125 on the client
- TCP congestion control algorithms
- VirtIO multiqueue configurations (1, 2, 4 and 8 queues)
- RPS and CPU load
- Connection tracking
- Queue disciplines
- Byte Queue Limits
- GRO and TSO settings
None of these tests alone provided a significant improvement in guest-to-client throughput.
CPU monitoring also showed no sustained saturation of individual vhost or vCPU threads at the sampling resolution used.
4. Identification of a GSO size mismatch
Using ip -d link show, I observed the following maximum GSO sizes:
Interface| GSO max size
Guest VirtIO interface (ens18)| 65536
Host TAP interface (tap100i0)| 65536
Linux bridge (vmbr0)| 64000
Physical RTL8125 interface| 64000
The r8169 driver defines the following limits for supported controller generations:
#define RTL_GSO_MAX_SIZE_V2 64000
#define RTL_GSO_MAX_SEGS_V2 64
I then used tcpdump on both the TAP interface and the physical NIC to compare TCP payload sizes before hardware segmentation.
With TSO enabled inside the Ubuntu guest and the default GSO maximum size of 65536 bytes:
- The TAP interface frequently received TCP payloads of 65160 bytes.
- On the physical NIC, 19101 out of 20000 captured packets had a TCP payload of only 1448 bytes.
- TCP throughput remained around 1.76 Gbit/s.
This suggests that oversized GSO buffers were being segmented in software before reaching the physical NIC.
5. Workaround
Inside the Ubuntu guest, I applied the following settings as root:
ethtool -K ens18 gro on gso on tso on
ip link set dev ens18 gso_max_size 63000
ip link set dev ens18 gso_ipv4_max_size 63000
The value of 63000 bytes was deliberately chosen to remain below the physical NIC's 64000-byte limit.
No MTU changes, physical hardware changes or host reboot were required.
After applying this configuration, the TCP payload size distributions captured on the TAP and physical interfaces became almost identical.
For example:
TCP payload size| TAP count| Physical NIC count
62264 bytes| 8772| 8777
2896 bytes| 6638| 6635
59368 bytes| 334| 332
60816 bytes| 279| 279
57920 bytes| 254| 253
These observations are consistent with large GSO buffers being preserved through the host networking path and hardware segmentation being used more effectively.
6. Performance improvements
All local tests used iperf3 with four parallel TCP streams.
Configuration| Guest-to-client throughput
TSO disabled, original configuration| 1.73 Gbit/s
TSO enabled, GSO size 65536| 1.76 Gbit/s
TSO enabled, GSO size 63000| 2.13 Gbit/s
Native Proxmox-to-client reference| 2.14 Gbit/s
The improvement was approximately 23% compared with the original configuration.
A subsequent 60-second iperf3 test confirmed 2.13 Gbit/s with zero TCP retransmissions in the guest-to-client direction.
Since the Ubuntu VM also acts as a router, I tested Internet throughput as well:
Measurement| Before| After
Download| 1761.56 Mbps| 2135.97 Mbps
Upload| 2117.67 Mbps| 2105.34 Mbps
Idle latency| 11 ms| 11 ms
Download loaded latency| 22 ms| 11 ms
Upload loaded latency| 11 ms| 11 ms
The Internet download improvement was approximately 21%.
I subsequently applied the same GSO limits to the WAN VirtIO interface for consistency, although this did not produce a measurable additional throughput improvement.
7. Questions for the Proxmox developers and community
- Is this GSO size mismatch between VirtIO and the underlying physical NIC a known performance issue?
- Could the guest's effective GSO limits be negotiated or configured to better accommodate the physical network path?
- Would it make sense to expose an optional maximum GSO size setting in Proxmox for VirtIO interfaces?
- Is this something that should instead be investigated upstream by the Linux networking, VirtIO or QEMU developers?
I understand that the physical network path can change and that automatically selecting a suitable GSO limit may not be straightforward.
I am not claiming that this is necessarily a Proxmox bug. However, the performance difference and packet captures suggest that this interaction is worth investigating.
I have kept the workaround as a persistent systemd service inside the Ubuntu guest.
I would be happy to provide additional configuration details, command outputs or testing results if useful.
Thank you for your work on Proxmox and for maintaining such a great open-source virtualization platform.
Best regards,
A Proxmox home server user from France
I would like to share the results of a network performance investigation on my Proxmox VE server. After extensive testing, I identified a significant throughput improvement by adjusting the maximum GSO size inside a Linux guest to match the capabilities of the underlying physical network adapter.
I am sharing this in the hope that it may help other Proxmox users and possibly lead to an improvement in the default VirtIO network configuration.
1. Hardware and software environment
- Motherboard: MSI B85-G43
- CPU: Intel Core i7-4790K (4 cores / 8 threads)
- Proxmox kernel: 7.0.14-28-pve
- Physical NICs: TP-Link TX201 (Realtek RTL8125B)
- Host network driver: native Linux r8169
- Network speed: 2.5 Gbit/s, full duplex
- Guest: Ubuntu Server
- Guest CPU configuration: host, 8 vCPUs
- Network model: VirtIO
- VirtIO multiqueue: 8 queues per interface
- vhost-net: enabled
- Packed virtqueues: enabled
- Linux bridges: vmbr0 (LAN) and vmbr1 (WAN)
- MTU: 1500 throughout the network
- Client: Fedora Linux with another RTL8125-based NIC using r8169
The Ubuntu VM also acts as an Internet gateway for my local network.
2. Initial problem
I noticed asymmetric TCP throughput between my Ubuntu VM and my Fedora workstation.
Initial iperf3 results:
Direction| Throughput
Fedora to Proxmox| 2.34 Gbit/s
Fedora to Ubuntu VM| 2.34 Gbit/s
Proxmox to Fedora| 2.14 Gbit/s
Ubuntu VM to Fedora| 1.73–1.76 Gbit/s
The issue was reproducible with multiple TCP streams and remained present when directly connecting the workstation to the Proxmox server, bypassing the physical Ethernet switch.
3. Troubleshooting
I tested several potential causes:
- Physical switch and other LAN traffic
- Native r8169 versus out-of-tree r8125 on the client
- TCP congestion control algorithms
- VirtIO multiqueue configurations (1, 2, 4 and 8 queues)
- RPS and CPU load
- Connection tracking
- Queue disciplines
- Byte Queue Limits
- GRO and TSO settings
None of these tests alone provided a significant improvement in guest-to-client throughput.
CPU monitoring also showed no sustained saturation of individual vhost or vCPU threads at the sampling resolution used.
4. Identification of a GSO size mismatch
Using ip -d link show, I observed the following maximum GSO sizes:
Interface| GSO max size
Guest VirtIO interface (ens18)| 65536
Host TAP interface (tap100i0)| 65536
Linux bridge (vmbr0)| 64000
Physical RTL8125 interface| 64000
The r8169 driver defines the following limits for supported controller generations:
#define RTL_GSO_MAX_SIZE_V2 64000
#define RTL_GSO_MAX_SEGS_V2 64
I then used tcpdump on both the TAP interface and the physical NIC to compare TCP payload sizes before hardware segmentation.
With TSO enabled inside the Ubuntu guest and the default GSO maximum size of 65536 bytes:
- The TAP interface frequently received TCP payloads of 65160 bytes.
- On the physical NIC, 19101 out of 20000 captured packets had a TCP payload of only 1448 bytes.
- TCP throughput remained around 1.76 Gbit/s.
This suggests that oversized GSO buffers were being segmented in software before reaching the physical NIC.
5. Workaround
Inside the Ubuntu guest, I applied the following settings as root:
ethtool -K ens18 gro on gso on tso on
ip link set dev ens18 gso_max_size 63000
ip link set dev ens18 gso_ipv4_max_size 63000
The value of 63000 bytes was deliberately chosen to remain below the physical NIC's 64000-byte limit.
No MTU changes, physical hardware changes or host reboot were required.
After applying this configuration, the TCP payload size distributions captured on the TAP and physical interfaces became almost identical.
For example:
TCP payload size| TAP count| Physical NIC count
62264 bytes| 8772| 8777
2896 bytes| 6638| 6635
59368 bytes| 334| 332
60816 bytes| 279| 279
57920 bytes| 254| 253
These observations are consistent with large GSO buffers being preserved through the host networking path and hardware segmentation being used more effectively.
6. Performance improvements
All local tests used iperf3 with four parallel TCP streams.
Configuration| Guest-to-client throughput
TSO disabled, original configuration| 1.73 Gbit/s
TSO enabled, GSO size 65536| 1.76 Gbit/s
TSO enabled, GSO size 63000| 2.13 Gbit/s
Native Proxmox-to-client reference| 2.14 Gbit/s
The improvement was approximately 23% compared with the original configuration.
A subsequent 60-second iperf3 test confirmed 2.13 Gbit/s with zero TCP retransmissions in the guest-to-client direction.
Since the Ubuntu VM also acts as a router, I tested Internet throughput as well:
Measurement| Before| After
Download| 1761.56 Mbps| 2135.97 Mbps
Upload| 2117.67 Mbps| 2105.34 Mbps
Idle latency| 11 ms| 11 ms
Download loaded latency| 22 ms| 11 ms
Upload loaded latency| 11 ms| 11 ms
The Internet download improvement was approximately 21%.
I subsequently applied the same GSO limits to the WAN VirtIO interface for consistency, although this did not produce a measurable additional throughput improvement.
7. Questions for the Proxmox developers and community
- Is this GSO size mismatch between VirtIO and the underlying physical NIC a known performance issue?
- Could the guest's effective GSO limits be negotiated or configured to better accommodate the physical network path?
- Would it make sense to expose an optional maximum GSO size setting in Proxmox for VirtIO interfaces?
- Is this something that should instead be investigated upstream by the Linux networking, VirtIO or QEMU developers?
I understand that the physical network path can change and that automatically selecting a suitable GSO limit may not be straightforward.
I am not claiming that this is necessarily a Proxmox bug. However, the performance difference and packet captures suggest that this interaction is worth investigating.
I have kept the workaround as a persistent systemd service inside the Ubuntu guest.
I would be happy to provide additional configuration details, command outputs or testing results if useful.
Thank you for your work on Proxmox and for maintaining such a great open-source virtualization platform.
Best regards,
A Proxmox home server user from France