Slow network speed on VMs, but not on host

[kevinsah]

Hey there!

I got a very strange issue;
Sometimes my network speed is throttled to 100 Mbps. But I got 1000 Mbps at home.
For example;
If i download something on host, its downloading in 110MB/s (normal speed) - but if i download from VM, it's downloading in 10MB/s.
It's like its changing from 1000 Mbps Full Duplex to 100 Mbps randomly... I got multiple IPs from my ISP, via DHCP - if this is necessary information!
I dont got Rate limit on the VM.
Sometimes it's back in "normal" speed if i restart my VM, but not always.

OS:
Debian 10, Linux 5.0.15-1-pve

 

[Moayad]

Hi,

Please post your network config, Debian VM config and pveversion -v as well

 

[kevinsah]

Hi!
here comes the configs

Hi,

Please post your network config, Debian VM config and pveversion -v as well

Spoiler: pveversion -v

proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-4 (running version: 6.0-4/2a719255)
pve-kernel-5.0: 6.0-5
pve-kernel-helper: 6.0-5
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 14.2.4.1-pve1
ceph-fuse: 14.2.4.1-pve1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-2
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-5
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-3
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-5
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-5
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

Spoiler: Debian VM Network Config

Spoiler: PVE Host Network Config

 

[Moayad]

Please Debian VM config with command:

qm config <VMID>

 

[kevinsah]

Please Debian VM config with command:

qm config <VMID>

Spoiler: QM CONFIG DEBIAN VM

 

[Moayad]

If you use virtio nic on a vmbr network the speed is limited by the CPU/Memory, also I wondering how your VM running with 115 ram?

So try to give more memory and core and and check the firewall if running - if running try to disable it and try test network again

 

[MarcMorgan]

Sorry to jump on a old thread but im getting the same, my machine (on same network (cable) hits around 380MBs and proxmos how hits around 280MBS which i guess im ok with (could be better but unsure on how) but my VM (windows) only hits 100MBS (if lucky

Proxmox is on Dell R710
VM - Windows 10
NIC for VM - Intel E1000

Tried the firewall suggestion (Not rebooted VM)

Installed speedtest-cli on proxmox host to do the tests.

Not sure what else to try really. it must be some configuration, see below for qm config

 

[yaboc]

Sorry to jump on a old thread but im getting the same, my machine (on same network (cable) hits around 380MBs and proxmos how hits around 280MBS which i guess im ok with (could be better but unsure on how) but my VM (windows) only hits 100MBS (if lucky

Proxmox is on Dell R710
VM - Windows 10
NIC for VM - Intel E1000

Tried the firewall suggestion (Not rebooted VM)

Installed speedtest-cli on proxmox host to do the tests.

Not sure what else to try really. it must be some configuration, see below for qm config

Use virtio nic. you seem to be hitting gigabit limit of E1000.

 

[MarcMorgan]

I changed this over this morning and speeds are going back up. Thank you

Not sure how to resolve this or mark as complete?

 

[LolekUK]

Hi all,

Seems as I'm having network problems with my network after pve6to7 upgrade, speedtest getting only 200+Mbps on gigabit FTTP and containers getting around 100Mbps. VM looks as getting similar speeds to the host. But still far away from the gigabit connection I've got. My laptop plugged in with a cable directly to the router is getting correct speeds. Any thoughts? Thank you

pveversion -v
root@pve:~# pveversion -v proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve) pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488) pve-kernel-helper: 7.1-2 pve-kernel-5.11: 7.0-8 pve-kernel-5.4: 6.4-6 pve-kernel-5.11.22-5-pve: 5.11.22-10 pve-kernel-5.4.140-1-pve: 5.4.140-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph-fuse: 14.2.21-1 corosync: 3.1.5-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: 0.8.36+pve1 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.22-pve1 libproxmox-acme-perl: 1.4.0 libproxmox-backup-qemu0: 1.2.0-1 libpve-access-control: 7.0-5 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-10 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-3 libpve-storage-perl: 7.0-12 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-4 lxcfs: 4.0.8-pve2 novnc-pve: 1.2.0-3 proxmox-backup-client: 2.0.11-1 proxmox-backup-file-restore: 2.0.11-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.3-6 pve-cluster: 7.0-3 pve-container: 4.1-1 pve-docs: 7.0-5 pve-edk2-firmware: 3.20210831-1 pve-firewall: 4.2-4 pve-firmware: 3.3-2 pve-ha-manager: 3.3-1 pve-i18n: 2.5-1 pve-qemu-kvm: 6.0.0-4 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-16 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.5-pve1

VM host network config
auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.1.10 netmask 255.255.255.0 gateway 192.168.1.1 bridge_ports eno1 bridge_stp off bridge_fd 0

Container network config:
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.3/24 gateway 192.168.1.1

 

[Dunuin]

Did you get full Gbit internet with another host? Maybe your ISP or model just can't handle more than 200 Mbit otherwise.

 

[LolekUK]

I had gigabit speeds on that host (and containers) before with no problems, all I can think of is (the only configuration change) that I've recently ran pve6to7 upgrade and perhaps didn't run any speedtests until now.

edit:
also, had a ubuntu VM with a docker and speedtest-tracker on, that was also reporting full ISP gigabit speeds. But not anymore.

 

[LolekUK]

installed speedtest for CLI from speedtest.net website and tested on host that I'm having issues with:


I have now also created new Proxmox host, and ran the same test which shows correct speeds:


Somethins has gone wrong somewhere, I assume during pve6to7 upgrade...

 

[Dunuin]

Are you sure that speedtest don't write to your drives when receiving data from the speedtest servers? Because upload is in both cases 100 Mbit, just the download is the drifferent. Most of the time the drives are bottlenecking if someone complains about bad network performance. You really should try iperf3 to check if your local network can handle Gbit. iperf3 is writing to RAM and not to disk, so if network performance is still slow then it should really be an network problem.

 

[LolekUK]

Are you sure that speedtest don't write to your drives when receiving data from the speedtest servers? Because upload is in both cases 100 Mbit, just the download is the drifferent. Most of the time the drives are bottlenecking if someone complains about bad network performance. You really should try iperf3 to check if your local network can handle Gbit. iperf3 is writing to RAM and not to disk, so if network performance is still slow then it should really be an network problem.

both hosts have nvme ssd drives in, no raid or anything. Decided to migrate all VMs to the new host anyway.

 

[Dunuin]

That doesn't mean much. By using virtualization, a bad storage layout and the right workload I can bring a NVMe SSD down to below 1 MB/s write bandwidth. If you want to test network performance you should write to RAM.

 

[LolekUK]

Interesting, migrated 2-3 machines and then it worked fine for about 24 hours on the new host but then internet speeds went down to just 200Mbps. On the other hand, the initial machine that had no longer hosted the VMs started reporting 900Mbps speeds on the broadband.

It's only a small home lab, with pi-hole, plex containers and vpn gateway VM. As said before it worked for over a year and only noticed the speed issue shortly after pve6to7 upgrade.

I will see if I can get iperf3 tests on local network.

 

[LolekUK]

it's all rather bizarre,

iperf3 on local network (with devices plugged in to 1G switch, proxmox and a laptop), all looking fine, 1G transfer between the two.
iperf3 between host and container - 5Gbps,

Laptop used for iperf3 server, multiple broadband speed tests reporting 900Mbps speeds which is correct.
Proxmox server ran iperf3 with clooud server and multiple speed tests, reporting only 35Mbps.

Tried the ISP (BT) router again today with hard reset, all the same.

I'm confused. Local network all fine but once trying to do anything on the internet, it's very slow. But all other devices on the network are fine.

 

[laz]

it's all rather bizarre,

iperf3 on local network (with devices plugged in to 1G switch, proxmox and a laptop), all looking fine, 1G transfer between the two.
iperf3 between host and container - 5Gbps,

Laptop used for iperf3 server, multiple broadband speed tests reporting 900Mbps speeds which is correct.
Proxmox server ran iperf3 with clooud server and multiple speed tests, reporting only 35Mbps.

Tried the ISP (BT) router again today with hard reset, all the same.

I'm confused. Local network all fine but once trying to do anything on the internet, it's very slow. But all other devices on the network are fine.

I have the same problem. The PVE is 7.0-2, speedtest show Download is only 100Mbps

proxmox-ve: 7.0-2 (running kernel: 5.11.22-7-pve)
pve-manager: 7.0-14+1 (running version: 7.0-14+1/08975a4c)
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-12
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-13
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.13-1
proxmox-backup-file-restore: 2.0.13-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.1.0-1
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-18
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

 

[LolekUK]

I have the same problem. The PVE is 7.0-2, speedtest show Download is only 100Mbps

proxmox-ve: 7.0-2 (running kernel: 5.11.22-7-pve)
pve-manager: 7.0-14+1 (running version: 7.0-14+1/08975a4c)
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-12
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-13
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.13-1
proxmox-backup-file-restore: 2.0.13-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.1.0-1
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-18
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

I keep reading up about it, and could be new kernel and intel NICs, I've got i219-v, my brother just realised he's got same issue with i218-v. Will try new driver sometime this week.

What's yours?

lspci