LAN network speeds are fine, but Proxmox and VM's have slow internet speeds.

[Galahad]

Hi all!

I have a very odd situation at the moment. The internet download speeds on both proxmox and any VM's are not at all what I should be getting. I have a 1000/100 connection and all of my other devices on my LAN are able to hit these speeds so I know it is not an issue with my router/switches. However proxmox and it's VM's are only able to get around 200/100. Oddly enough, if I reboot proxmox or a VM and do a speedtest straight away I can usually hit my full speeds and then it drops off to about 200/100, sometimes to around 100/100. I have run iperfs between proxmox to another lan machine, and from my VM's to another lan machine and I am getting transfers at gigabit speeds.

I have disabled the firewall on proxmox and on the VM's, all of the VM's are using virtio nics and I am utterly baffled as to what could be causing this.

Spoiler: pveversion -v

proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-7
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Spoiler: iperf from small debian VM to LAN machine

Spoiler: speedtest from same VM

This speedtest happens to be a bit faster than usual, but is usually no more than 200Mbps.

Spoiler: speedtest from a desktop

Any help at all is greatly appreciated

 

[makabazi]

i have got the same problem

 

[Fazzle]

same problem, have updated and upgraded everything except kernel 5.15

 

[Elias137]

Same here, Proxmox 7.2

 

[ph1823]

Same problem with Proxmox 6.4

 

[jermaine150]

I'm experiencing the exact same issue, any one found a resolution on this?

 

[crocobcc]

Hi there, I also have the same problem.

Only with VM not with LCX. Windows, Linux with virtIO or Intel or Realtek driver => same result.

Cannot go over 400-500mbits...on the Pve directly 1g ok.

Does anyone has found a solution for this ?

Regars

 

[wigor]

Hey,
maybe you can test with different settings for the offload features in the NIC Driver. I have no concrete advice, but you often find hints to problems with offloading & virtualization.

 

[crocobcc]

Hey,
maybe cou can test with different settings for the offload features in the NIC Driver. I have no concrete advice, but you often find hints to problems with offloading & virtualization.

I already tried to disable Offload but it's not changing anything....

And the facts are strange => LAN speed is OK but Internet speed is limited (passing trought the same Nic...)

 

[wigor]

You double checked mtu settings? Here (in Germany) using dsl the mtu is usally 1492 e.g. instead of 1500 for local ethernet.

 

[crocobcc]

You double checked mtu settings? Here (in Germany) using dsl the mtu is usally 1492 e.g. instead of 1500 for local ethernet.

Yes i also tried to change MTU => no change at all.

Cannot download at more than 36Mo/s => 300mbits

I have 2 pve and it's the same behaviour on both for all VM (linux and Windows) (LCX are ok)

 

[crocobcc]

Seems a lots of person (everyone ?) are having this problem....

I looked at a lot website ,test a lots of things => no one found a solution....

I just hope Proxomox will imporove this in the future... king of frustating to download at 200-300mbits with a 1gbits connexion ^^

Does anyone has normal network performance with a big connection (eg 1gbits) ?

 

[UdoB]

Does anyone has normal network performance with a big connection (eg 1gbits) ?

Okay, out of curiosity I've made two tests on this Friday afternoon:

1) from a DSL provider (Vodafon) in Germany. Crippled asymmetric 1GBit/s / 50 MBit/s
(Sidenode: I actually do get 1 GBit/s down sometimes - for example via Steam CDN.)

This in inside a VM on a AMD Ryzen Threadripper 2920X behind an AVM Fritz!Box:

Retrieving speedtest.net configuration...
Testing from Vodafone Germany Cable (77.20.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by wittenberg-net GmbH (Lutherstadt Wittenberg) [183.83 km]: 27.085 ms
Testing download speed................................................................................
Download: 743.80 Mbit/s
Testing upload speed.....................................................................................................
.Upload: 50.94 Mbit/s

Second run:

Retrieving speedtest.net configuration...
Testing from Vodafone Germany Cable (77.20.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by wittenberg-net GmbH (Lutherstadt Wittenberg) [183.83 km]: 26.627 ms
Testing download speed................................................................................
Download: 735.29 Mbit/s
Testing upload speed......................................................................................................
Upload: 58.12 Mbit/s

2) this a a German university. Theoretical up to 10 GBit/s symmetric
This is a VM on an AMD Epyc Processor behind a Software (Shorewall) Router:

Retrieving speedtest.net configuration...
Testing from Deutsches Forschungsnetz (134.76.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by myLoc managed IT AG (Dusseldorf) [221.62 km]: 16.809 ms
Testing download speed................................................................................
Download: 2395.03 Mbit/s
Testing upload speed......................................................................................................
Upload: 507.46 Mbit/s

speedtest --server 24887 
Retrieving speedtest.net configuration...
Testing from Deutsches Forschungsnetz (134.76.x.x)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by Studenten Net Twente (Enschede) [218.86 km]: 21.8 ms
Testing download speed................................................................................
Download: 2208.19 Mbit/s
Testing upload speed......................................................................................................
Upload: 261.74 Mbit/s

speedtest --server 29806 
Retrieving speedtest.net configuration...
Testing from Deutsches Forschungsnetz (134.76.x.x)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by Schlueter Onlinedienste (Ruethen) [103.43 km]: 11.763 ms
Testing download speed................................................................................
Download: 1900.23 Mbit/s
Testing upload speed......................................................................................................
Upload: 493.99 Mbit/s

All NICs = standard virtio.

 

[crocobcc]

Okay, out of curiosity I've made two tests on this Friday afternoon:

1) from a DSL provider (Vodafon) in Germany. Crippled asymmetric 1GBit/s / 50 MBit/s
(Sidenode: I actually do get 1 GBit/s down sometimes - for example via Steam CDN.)

....

Thank you for confirming it CAN work.

I keep searching and found a solution (for me at least don't know for others)

I just change CPU type to "host" + configure cpuunits/cpulimits => I passed from 300mb to 700-800mb !

Regards

 

[kguevarra]

Hello Guys,
Any update on this? I'm experiencing the same issues.
Here is my PVE version:

proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.4-1
proxmox-backup-file-restore: 2.2.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

Virtio's is being used

 

[xcelitek]

I too am having the issue. Proxmox host has 10Gb/s connection and my windows hosts, get the full 10Gbs bandwidth. with 6Gbs throttle. Speedtest's on windows run about 4-4.5Gbs consistantly. on my two linux hosts (are consistant and Ubuntu/Debian) with the same NIC settings, can barley get 400Mb/s

Tied all above suggestions and every NIC setting and still cannot get more from the Linux VM but Windows is just fine.

 

[SgtSid]

Bump. I am experiencing the same issues. I can saturate my 1Gbps LAN connection and my 10Gbps SAN connection on the host, but not my 800Mbps internet connection from a VM or the host. Downloads and updates take incredibly too long.

Updating, Google searching, iperf testing has yielded no results to fix this issue and the issue seems to have started around my update to PVE 7.2

 

[wigor]

Hey,

for tcp connections there are a lot of parameters, that can cause such sort of trouble. I myself observed through wireshark, that a server with problems did not scale tcp window size correctly.
Connection was bound to 400kbit/s therefore.
It was a typo in the sysctl settings for the tcp stack on an old linux machine. Your set up may differ, but tcp session can somehow be tricky and you can search at the wrong place for performance problems maybe.

just a further idea to dig deeper.

 

[SgtSid]

Anyone still seeing this issue?

 

[HidayatLahabu]

Anyone still seeing this issue?

yes, my LAN connection to ubuntu lxc is little bit slow, but not for my wondows 7 VM