Network Speedtest-cli is very slow from vm trough VM to the internet

chrigiboy · Apr 9, 2024

Hello
i have build a Proxmox Server. i created a virtual machine with Sophos FireWall and a 2. virtual machine with Ubuntu.

1. When i run speedtest-cli directly from the Proxmox Server i got 25/25 Gbit. So the traffic goes from the Hostsystem through the virtual Machine without any issues. The sophos virtual machine looks fast enough.
2. When i run the speedtest on the Ubuntu VM, i got only 3/3 Gbit. First i disabled FireWall checkbox, now i got 5/5 gbit/s. i am very far away from 25/25 Gbit/s. I tested several network adapter but only VirtIO give me the best speed. So why i got from ubuntu VM through Sophos to the speedtest server so a slow speed? I can also see, when i run the speedtest directly from Proxmox host Server the CPU load on the sophos is nearly 2%, but when i run the speedtest on the virtual ubuntu system the CPU load of sophos goes up to 75%, but why and whats wrong? the name of the process on the sophos firewall is named ksoftirqd/2 which tooks 75% of the cpu load. On the Proxmox host system, i can only see high kvm load.

What can i do, so i got also 25/25 Gbit on the ubuntu system?

3. I also created a windows virtual machine. When i run the speedtest on those machine, the local screen on the proxmox gui is freezed during the test. i got 3/3 gbit with disabled firewall. During the test, the cpu load of this vm goes up to 40%.

sophos:

Code:

           CPU0       CPU1       CPU2       CPU3

  0:          1          0          0          0   IO-APIC   2-edge      timer

  1:          1          9          0          0   IO-APIC   1-edge      i8042

  8:          1          0          0          0   IO-APIC   8-edge      rtc0

  9:          0          0          0          0   IO-APIC   9-fasteoi   acpi

 11:          0          0          0         31   IO-APIC  11-fasteoi   uhci_hcd:usb1

 12:          5          0          0          0   IO-APIC  12-edge      i8042

 14:          0          0          0          0   IO-APIC  14-edge      ide0

 15:          0          0          0          0   IO-APIC  15-edge      ide1

 24:          0          0          0          0   PCI-MSI 540672-edge      virtio5-config

 25:          0          0          0          0   PCI-MSI 540673-edge      virtio5-control

 26:          0          0          0          0   PCI-MSI 540674-edge      virtio5-event

 27:      27881          0          0          0   PCI-MSI 540675-edge      virtio5-request

 28:          0      29455          0          0   PCI-MSI 540676-edge      virtio5-request

 29:          0          0      26917          0   PCI-MSI 540677-edge      virtio5-request

 30:          0          0          0      29442   PCI-MSI 540678-edge      virtio5-request

 31:          0          0          0          0   PCI-MSI 294912-edge      virtio2-config

 32:          0     639115          0          0   PCI-MSI 294913-edge      virtio2-input.0

 33:          0       6940          0          0   PCI-MSI 294914-edge      virtio2-output.0

 34:          0          0          0          0   PCI-MSI 311296-edge      virtio3-config

 35:          0          0    1910800          0   PCI-MSI 311297-edge      virtio3-input.0

 36:          0          0     314049          0   PCI-MSI 311298-edge      virtio3-output.0

 37:          0          0          0          0   PCI-MSI 327680-edge      virtio4-config

 38:          0          0    1341397          0   PCI-MSI 327681-edge      virtio4-input.0

 39:          0          0     348587          0   PCI-MSI 327682-edge      virtio4-output.0

NMI:          0          0          0          0   Non-maskable interrupts

LOC:    9044286    9059191    8993006    8989673   Local timer interrupts

SPU:          0          0          0          0   Spurious interrupts

PMI:          0          0          0          0   Performance monitoring interrupts

IWI:          0          0          0          0   IRQ work interrupts

RTR:          0          0          0          0   APIC ICR read retries

RES:     504524     501659     465353     516110   Rescheduling interrupts

CAL:      19352      18816      18639      18574   Function call interrupts

TLB:       2169       2049       2069       2034   TLB shootdowns

TRM:          0          0          0          0   Thermal event interrupts

THR:          0          0          0          0   Threshold APIC interrupts

MCE:          0          0          0          0   Machine check exceptions

MCP:         42         42         42         42   Machine check polls

HYP:          0          0          0          0   Hypervisor callback interrupts

ERR:          0

MIS:          0

PIN:          0          0          0          0   Posted-interrupt notification event

NPI:          0          0          0          0   Nested posted-interrupt event

PIW:          0          0          0          0   Posted-interrupt wakeup event

SFVH_SO01_SFOS 20.0.0 GA-Build222#

chrigiboy · Apr 9, 2024

proxmox:

Code:

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11
   0:         39          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    2-edge      timer
   6:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    6-edge      AMDI0010:03
   7:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    7-fasteoi   pinctrl_amd
   8:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    8-edge      rtc0
   9:          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC    9-fasteoi   acpi
  25:          0          0          0          0          0          0          0          0          0          0          0          0  IOMMU-MSI    0-edge      AMD-Vi
  37:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-0000:02:00.1    0-edge      ahci[0000:02:00.1]
  38:          0          0          0          0          0          0          0          0          0          0          0          1  IR-PCI-MSIX-0000:07:00.0    0-edge      enp7s0
  39:     280853       2445          0      11682          0          0          0       2783        359          0          0       9738  IR-PCI-MSIX-0000:07:00.0    1-edge      enp7s0-TxRx-0
  40:          0     394813       1394        655        214          0          0         69          0        940       2793          0  IR-PCI-MSIX-0000:07:00.0    2-edge      enp7s0-TxRx-1
  41:          0        430     324446        673        552          0          0        369       3771        846          0        549  IR-PCI-MSIX-0000:07:00.0    3-edge      enp7s0-TxRx-2
  42:          0        642        132     283616        759        384          0       1031          0        839       3583       1686  IR-PCI-MSIX-0000:07:00.0    4-edge      enp7s0-TxRx-3
  44:          1          0          0          0         79          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    0-edge      nvme1q0
  45:          0          0          0         79          0          0          0          0          0          0          0          1  IR-PCI-MSIX-0000:08:00.0    0-edge      nvme0q0
  46:      74725          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    1-edge      nvme1q1
  47:          0      75177          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    2-edge      nvme1q2
  48:          0          0      74161          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    3-edge      nvme1q3
  49:          0          0          0      74407          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    4-edge      nvme1q4
  50:          0          0          0          0      75828          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    5-edge      nvme1q5
  51:          0          0          0          0          0      74781          0          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    6-edge      nvme1q6
  52:          0          0          0          0          0          0      75818          0          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    7-edge      nvme1q7
  53:          0          0          0          0          0          0          0      76128          0          0          0          0  IR-PCI-MSIX-0000:05:00.0    8-edge      nvme1q8
  54:          0          0          0          0          0          0          0          0      74931          0          0          0  IR-PCI-MSIX-0000:05:00.0    9-edge      nvme1q9
  55:          0          0          0          0          0          0          0          0          0      73774          0          0  IR-PCI-MSIX-0000:05:00.0   10-edge      nvme1q10
  56:          0          0          0          0          0          0          0          0          0          0      76385          0  IR-PCI-MSIX-0000:05:00.0   11-edge      nvme1q11
  57:          0          0          0          0          0          0          0          0          0          0          0      75525  IR-PCI-MSIX-0000:05:00.0   12-edge      nvme1q12
  78:      80526          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    1-edge      nvme0q1
  79:          0      80064          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    2-edge      nvme0q2
  80:          0          0      78579          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    3-edge      nvme0q3
  81:          0          0          0      78565          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    4-edge      nvme0q4
  82:          0          0          0          0      80613          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    5-edge      nvme0q5
  83:          0          0          0          0          0      78882          0          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    6-edge      nvme0q6
  84:          0          0          0          0          0          0      80816          0          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    7-edge      nvme0q7
  85:          0          0          0          0          0          0          0      81537          0          0          0          0  IR-PCI-MSIX-0000:08:00.0    8-edge      nvme0q8
  86:          0          0          0          0          0          0          0          0      80842          0          0          0  IR-PCI-MSIX-0000:08:00.0    9-edge      nvme0q9
  87:          0          0          0          0          0          0          0          0          0      79208          0          0  IR-PCI-MSIX-0000:08:00.0   10-edge      nvme0q10
  88:          0          0          0          0          0          0          0          0          0          0      81945          0  IR-PCI-MSIX-0000:08:00.0   11-edge      nvme0q11
  89:          0          0          0          0          0          0          0          0          0          0          0      80750  IR-PCI-MSIX-0000:08:00.0   12-edge      nvme0q12
 110:          0          0          0          0          0         81          0          0          0          0          0          0  IR-PCI-MSIX-0000:02:00.0    0-edge      xhci_hcd

 118:          0        427          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.3    0-edge      xhci_hcd
 126:          0          0          0          0          0          0          0          0          0     216067          0          0  IR-PCI-MSIX-0000:09:00.4    0-edge      xhci_hcd
 127:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    1-edge      xhci_hcd
 128:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    2-edge      xhci_hcd
 129:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    3-edge      xhci_hcd
 130:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    4-edge      xhci_hcd
 131:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    5-edge      xhci_hcd
 132:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    6-edge      xhci_hcd
 133:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.4    7-edge      xhci_hcd
 135:          0          0          0          0          0       7699          0          0          0          0          0          0  IR-PCI-MSIX-0000:01:00.0    0-edge       mlx5_comp12@pci:0000:01:00.0
 149:          0      24699       3768       4682       2670          0     238324         52       6713      21354         62      18866  IR-PCI-MSIX-0000:01:00.1    0-edge      mlx5_async0@pci:0000:01:00.1
 150:    1700142          0          0          0      70192          0          0          0       1065          0     501932       1258  IR-PCI-MSIX-0000:01:00.1    1-edge      mlx5_comp1@pci:0000:01:00.1
 151:          0    1914620         48     447340       2493         20          0      27796          0        148          0          6  IR-PCI-MSIX-0000:01:00.1    2-edge      mlx5_comp2@pci:0000:01:00.1
 152:     202164         17    1951908         12         59          0          0          0          3          0         73     452188  IR-PCI-MSIX-0000:01:00.1    3-edge      mlx5_comp3@pci:0000:01:00.1
 153:          0          0         32    2049626          0        110          0          0     810163          0          8          0  IR-PCI-MSIX-0000:01:00.1    4-edge      mlx5_comp4@pci:0000:01:00.1
 154:          0      10220       4486          0    1936479     161341          0          0          0     345646      16621         45  IR-PCI-MSIX-0000:01:00.1    5-edge      mlx5_comp5@pci:0000:01:00.1
 155:          0          0          0          0          0    1622132     569612          0          0          0          0          0  IR-PCI-MSIX-0000:01:00.1    6-edge      mlx5_comp6@pci:0000:01:00.1
 156:          0          0          0          0          0          0    1810840     579422          0          0       1552      75043  IR-PCI-MSIX-0000:01:00.1    7-edge      mlx5_comp7@pci:0000:01:00.1
 157:          0        247          0       3195          0     629383          0    1726359          0        232          0       1270  IR-PCI-MSIX-0000:01:00.1    8-edge      mlx5_comp8@pci:0000:01:00.1
 158:          0      25734         52          0     514656         53          0          0    1644596      92546      57891        298  IR-PCI-MSIX-0000:01:00.1    9-edge      mlx5_comp9@pci:0000:01:00.1
 159:          0      62691          0       1350          0          0          0      20088          0    2216906        436     271371  IR-PCI-MSIX-0000:01:00.1   10-edge      mlx5_comp10@pci:0000:01:00.1
 160:     449014          0      58243          0          0          0          0          0          0     127866    2283429          0  IR-PCI-MSIX-0000:01:00.1   11-edge      mlx5_comp11@pci:0000:01:00.1
 161:          0         10     454891          6       5862      69843          0         30          0      14171     159304    1660639  IR-PCI-MSIX-0000:01:00.1   12-edge      mlx5_comp12@pci:0000:01:00.1
 162:          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSIX-0000:09:00.2    0-edge      psp-1
 165:          0          0          0          0          0          0          0          0         76          0          0          0  IR-PCI-MSI-0000:09:00.1    0-edge      snd_hda_intel:card0
 166:          0          0          0          0          0          0          0          0          0        838          0          0  IR-PCI-MSI-0000:09:00.6    0-edge      snd_hda_intel:card1
 167:          0      32739          0      34643       4798          0          0       9512          0          0    1479658          0  IR-PCI-MSIX-0000:09:00.0    0-edge      amdgpu
 NMI:        110        113        113        113        115        114        114        112        111        115        112        115   Non-maskable interrupts
 LOC:    6062013    6248193    6109838    6327101    6533994    6879595    6769204    6291622    6439308    6462287    6483487    6494026   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:        110        113        113        113        115        114        114        112        111        115        112        115   Performance monitoring interrupts
 IWI:          0          0          0          0          0          0          1          0          2          0          0          0   IRQ work interrupts
 RTR:          0          0          0          0          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:     675214     768761     737595     685941     747933     688340     695464     745089     666885     735710     666813     727258   Rescheduling interrupts
 CAL:    3137759    2963506    2833098    2782549    2715267    2724213    2740072    2740546    2727386    2691963    2786636    2731959   Function call interrupts
 TLB:      14487      16789      16728      15126      20179      23057      17450      17870      13186      18306      16815      12733   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 DFR:          0          0          0          0          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:         43         43         43         43         43         43         43         43         43         43         43         43   Machine check polls
 ERR:          0
 MIS:          0
 PIN:          0          0          0          0          0          0          0          0          0          0          0          0   Posted-interrupt notification event
 NPI:          0          0          0          0          0          0          0          0          0          0          0          0   Nested posted-interrupt event
 PIW:          0          0          0          0          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event

_gabriel · Apr 9, 2024

passing through a virtual router is slower than being connected directly to the gateway.
your virtual router is limited to pve cpu.
don't forget to set vCpu "type" to "host" to get all features of the pve cpu.

chrigiboy · Apr 10, 2024

Hello,
CPU ist set to host. Of course a VM is everytime slower then a dedicated machine. But i dont have this trouble with ESXI or Hyper-V with the same hardware. So i believe that is a driver problem or a software limitation of Proxmox.

_gabriel · Apr 10, 2024

are NIC pass through to the VM ? or do you use Linux bridges ?
fwiw, Linux bridges, so physical NIC aren't involved , iperf3 bandwith vm -> host is about 40 Gb/s with a Xeon E-2386G and iperf3 -R vm -> host about 20 Gb/s

chrigiboy · Apr 10, 2024

I use the proxmox Linux bridge. Host => VM is fast, but VM => VM is slow.

_gabriel · Apr 10, 2024

fwiw, iperf3 VM (WinSrv2019) -> VM (WinSrv2022) is 18 Gb/s with a Xeon E-2386G ( iperf3 -R is 12 Gb/s )
just tried over an ESXi vanilla 6.7 box with a Ryzen 1800X (2017 era), iperf3 VM (WinSrv2016) -> VM (WinSrv2016) is 4 Gb/s (ipef3 -R is 2 Gb/s)
edit: another PVE host, EPYC 7302 (2020) , iperf3 VM (WinSrv2019) -> VM (WinSrv2019) is 8 Gb/s (iperf3 -R is 4 Gb/s )
raw bandwith over the bridge, without routing/nat involved, iperf3 default tcp buffer 128KB.
with default network settings on PVE / ESXi and VM (standard 1500 MTU , no VLAN ) , cpu mitigations disabled.

spirit · Apr 10, 2024

iperf3 is not multithread (even with -P option), so check that you don't have 1core at 100%.

iperf2 is multithread is you use multiple stream with -P (and increase queue in vm nic options)

ksoftirqd is the handling of network interrupts. a vm can do around 2millions packet per second by core. (if you do bench with 64bytes packet size, you'll not reach full bandwith , vs a bench with 1500bytes or 1MB packet ...) The limit is always the number of pps.

Search

Search

Network Speedtest-cli is very slow from vm trough VM to the internet

chrigiboy

Well-Known Member

chrigiboy

Well-Known Member

_gabriel

Renowned Member

chrigiboy

Well-Known Member

_gabriel

Renowned Member

chrigiboy

Well-Known Member

_gabriel

Renowned Member

spirit

Distinguished Member