Slow internet speed on pve but not on local network

LolekUK · Nov 11, 2021

Hi,

I've mentioned this on another post, but not exactly same issue so I've started this one here with more info troubleshooting notes.

I'm getting slow internet speeds on my host, vm's and containers. Initially I thought it was to recent 6to7 upgrade with whole network (not just internet). But looks as it wasn't the case.

Since then, I've:
* swapped ISP routers, defaulted settings etc.
* set up new host (fresh pve7 install) with 16GB RAM, 7th gen i5 CPU, Intel NICs, speeds were reporting fine
* moved 2 containers and 1 vm across to the new host - initially (for about 24 hours) all seemed ok, but then the speeds went down again,
* /I think/ once the machines were moved to new host, old host started reporting fast internet speeds
* local network traffic is fine (within pve and with local network devices)

CTs are: pi-hole and Plex server with google drive rclone (with no traffic on them while testing/troubleshooting).
VM is: VPN box (not a gateway for other machines)
Same speeds are reported even when the above machines are powered off.

Troubleshooting:

ethtool, all seems ok there with 1000Mb/s on NIC and 10000Mb/s on virtual interface,

iperf3, local LAN ok,

5GB/s on internal pve traffic (between containers and host)
up to 1GB/s on internal traffic between laptop (1gb nic) and host/containers.

internet speed tests - very slow, between 30-100Mbps, upload fine at 100Mbps,
wget 1gb files from various speed test websites, at around 20Mbps

speedtest

iperf3 - tried few online iperf3 servers with same slow result

Laptop plugged in directly to the network reporting correct internet speeds, using same gateway (router) and DNS (pi-hole)

I'm confused. Why local network works fine with all VMs and CTs but not the internet?
Could ISP (BT UK), block/traffic shape internal devices? if they e.g. don't like VPN and or rclone?
Any thoughts will be appreciated.

Thanks

LolekUK · Nov 11, 2021

All I can think of is perhaps something to do with the kernel and the Intel NICs I've got?
It has worked well for months, all I can think that changed was the pve6to7 upgrade and then new machine had fresh pve7 installed but then all updates installed as well (and this is perhaps when it started going slow again).

Just a thought...

beroliv · Nov 27, 2021

i got the same issue. Did you fix it already

https://forum.proxmox.com/threads/langsame-verbindung-ins-wan-im-lan-alles-gut.100500/

LolekUK · Nov 28, 2021

No I haven't, surprised nobody else is reporting/solving the issue here? I didn't try different drivers or anything but it looks as it's not only Intel NICs as someone else had Realtek chipset

beroliv · Nov 28, 2021

i think its a Kernel issue. Is it maybe possible change back to an older Kernel? My Problem is i have only this Server. So if i destroy them i have no light in the house anymore.
https://forums.debian.net/viewtopic.php?t=149955


ethtool -i eno1

driver: e1000e

version: 5.13.19-1-pve

firmware-version: 0.4-4

expansion-rom-version: 

bus-info: 0000:00:1f.6

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

beroliv · Nov 28, 2021

what about that??

https://gist.github.com/pepitooo/6c9f8ad5d48af24f33e614412a7e20b1

beroliv · Nov 28, 2021

can you check your syslog or kern.log if you see error msg like "Detected Hardware Unit Hang:"
i dont have any errors like that. i try updating the bios and order a usb-c Ethernet Adapter.
I also start with an older Kernel but dont solves this problem.
Hopefully i bring it to work with the adapter.

beroliv · Nov 28, 2021

Actually the Kernel pve 5.4.73-1-pve solves my issue

sajtos888 · Nov 28, 2021

beroliv said:
Actually the Kernel pve 5.4.73-1-pve solves my issue

Can you write a description of how to restore the kernel? currently Linux pve 5.11.22-4-pve. It is possible that I have this problem too!

beroliv · Nov 28, 2021

i just choose it in the Grub Menu while booting. Make some speedtest and you find it out.
to make it more visible i installed the Speedtracker Container in a Docker LXC.
https://github.com/henrywhitaker3/Speedtest-Tracker

sajtos888 · Nov 29, 2021

beroliv said:
i just choose it in the Grub Menu while booting. Make some speedtest and you find it out.
to make it more visible i installed the Speedtracker Container in a Docker Container.
https://github.com/henrywhitaker3/Speedtest-Tracker

View attachment 31845

In Grub you can only select 5.13.19-1-pve

beroliv · Nov 29, 2021

Kernel 5.4.124-1-pve works too. Newer Kernel are not working.
I set this Kernel now as Default.
@sajtos888 you have to choos them in the advanced Menü on the Grub start screen.

sajtos888 · Nov 29, 2021

beroliv said:
Kernel 5.4.124-1-pve works too. Newer Kernel are not working.
I set this Kernel now as Default.
@sajtos888 you have to choos them in the advanced Menü on the Grub start screen.

I mean, that's one option in the special menu

beroliv · Nov 29, 2021

you can install older Kernel with apt
search them with

Bash:

apt search pve-kernel | grep "pve-kernel*"

sajtos888 · Nov 29, 2021

beroliv said:
you can install older Kernel with apt
search them with

Bash:

apt search pve-kernel | grep "pve-kernel*"

Code:

root@pve:~# apt search pve-kernel                                               Sorting... Done
Full Text Search... Done
pve-firmware/stable,now 3.3-3 all [installed]
  Binary firmware code for the pve-kernel

pve-kernel-5.10.6-1-pve/stable 5.10.6-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11/stable,now 7.0-10 all [installed]
  Latest Proxmox VE Kernel Image

pve-kernel-5.11.0-1-pve/stable 5.11.0-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.12-1-pve/stable 5.11.12-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.17-1-pve/stable 5.11.17-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.21-1-pve/stable 5.11.21-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-1-pve/stable 5.11.22-2 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-2-pve/stable 5.11.22-4 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-3-pve/stable 5.11.22-7 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-4-pve/stable,now 5.11.22-9 amd64 [installed]
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-5-pve/stable 5.11.22-10 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-6-pve/stable 5.11.22-11 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.11.22-7-pve/stable,now 5.11.22-12 amd64 [installed,automatic]
  The Proxmox PVE Kernel Image

pve-kernel-5.11.7-1-pve/stable 5.11.7-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.13/stable,now 7.1-4 all [installed,automatic]
  Latest Proxmox VE Kernel Image

pve-kernel-5.13.14-1-pve/stable 5.13.14-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.13.18-1-pve/stable 5.13.18-1 amd64
  The Proxmox PVE Kernel Image

pve-kernel-5.13.19-1-pve/stable,now 5.13.19-3 amd64 [installed,automatic]
  The Proxmox PVE Kernel Image

pve-kernel-helper/stable,now 7.1-4 all [installed]
  Function for various kernel maintenance tasks.

pve-kernel-libc-dev/stable 5.13.19-3 amd64
  Linux support headers for userspace development

root@pve:~#

beroliv · Nov 29, 2021

@LolekUK did you try an older Kernel? i got it back working with PVE 5.4.124-1

LolekUK · Jan 5, 2022

ah, unsure how I've missed your posts @beroliv.

I've installed the edge kernels in December (newer ones) but still having same issue https://github.com/fabianishere/pve-edge-kernel

Another interesting finding, I've installed Windows 10 VM on my host yesterday.

Within Windows 10 VM, speeds are fine (600Mbps - speedtest.net and other broadband speed testers). What I have also noticed, speeds on my containers were also fine while the VM was up and running. Weird? I think so.

So ran few tests, and so far, while VM is up and running (or restarting) my speeds on containers are up to 900Mbps (full gigabit broadband).
It slows down after a while when VM is running but the restarting VM kicks it back in.

Once VM is turned off, the speeds go down to below 100Mbps.

I've repeated that few times and turning on/off the VM kicks the network about.

Weird!

I'll try to downgrade to older kernel.

beroliv · Jan 5, 2022

Hi @LolekUK,
I have the Kernel 5.15.7-1-pve running with the Kernel Parameters described in a other tread.
https://forum.proxmox.com/threads/langsame-verbindung-ins-wan-im-lan-alles-gut.100500/#post-436011

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_idle.max_cstate=1"

insert this in /etc/default/grub

To go back is not really a option. i want update my system and keep it up to date.
with out the parameters i still have this useu with the slow internet connection over all CT and VM.

LolekUK · Jan 5, 2022

beroliv said:
Hi @LolekUK,
I have the Kernel 5.15.7-1-pve running with the Kernel Parameters described in a other tread.
https://forum.proxmox.com/threads/langsame-verbindung-ins-wan-im-lan-alles-gut.100500/#post-436011

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_idle.max_cstate=1"

insert this in /etc/default/grub

To go back is not really a option. i want update my system and keep it up to date.
with out the parameters i still have this useu with the slow internet connection over all CT and VM.

Amazing @beroliv , thank you

beroliv · Jan 5, 2022

Hi,
is this test with the Kernel Parameter? And yes this issue is very wired.

Slow internet speed on pve but not on local network

Active Member

Attachments

Active Member

New Member

Active Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

Active Member

New Member

Active Member

New Member