[SOLVED] Slow "upload" from VM running apache2

undoo99

New Member
May 3, 2022
5
1
3
Hello,

I have a VM running Ubuntu server 20.04 LTS with apache2 as a web server.

When I try to do a download from that VM, 1core used by apache is running at 100% and the download speed is capped at around 35MB/s.

1651572885751.png

Also, I tested other VMs on this host and they are also capped at this speed.

Doing a download from the internet to this VMs, I get around 99MB/s with an 1Gbps link.

The host is an HP DL360 gen8 with 1 CPU "16 x Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (1 Socket)" and it has 128GB of RAM.
As storage, I use 2x 1,92TB Samsung PM883 enterprise SSD in RAID-1.

The version of PVE is:
Code:
pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-7
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
openvswitch-switch: 2.15.0+ds1-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

And the VM settings are:
Code:
qm config 108
agent: 1
boot: order=scsi0;ide2;net0
cores: 4
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=6.1.1,ctime=1651488381
name: 10.com
net0: virtio=22:91:CF:0E:89:F1,bridge=vmbr0,tag=15
numa: 0
ostype: l26
parent: three
scsi0: local-lvm:vm-108-disk-0,discard=on,size=30G
scsihw: virtio-scsi-pci
smbios1: uuid=4af994ec-703a-401f-821c-cab0008c9763
sockets: 1
vmgenid: c17064c8-2faf-4ad8-9db7-7bbd42509384

Even VM to VM speed in the same host is the same.

There must be something that I did wrong, but I don't know what.

Please help me solve this issue.

Thank you,
Undoo99
 
Even VM to VM speed in the same host is the same.

Can you run pveperf on your proxmox server and post the output?

You can also install and run a iperf3 test between to VMs to differentiate if this is a problem within your internal network.

Best regards,
Marco
 
Hi Marco,

pveperf:
Code:
# pveperf
CPU BOGOMIPS:      86185.28
REGEX/SECOND:      1575017
HD SIZE:           93.93 GB (/dev/mapper/pve-root)
BUFFERED READS:    824.52 MB/sec
AVERAGE SEEK TIME: 0.11 ms
FSYNCS/SECOND:     2397.48
DNS EXT:           373.09 ms
DNS INT:           406.72 ms

iperf3 as server:
Code:
iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from x.x.x.x, port 58656
[  5] local x.x.x.x port 5201 connected to x.x.x.x port 58658
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.23 GBytes  10.5 Gbits/sec                 
[  5]   1.00-2.00   sec  1.51 GBytes  13.0 Gbits/sec                 
[  5]   2.00-3.00   sec  1.56 GBytes  13.4 Gbits/sec                 
[  5]   3.00-4.00   sec  1.64 GBytes  14.1 Gbits/sec                 
[  5]   4.00-5.00   sec  1.49 GBytes  12.8 Gbits/sec                 
[  5]   5.00-6.00   sec  1.54 GBytes  13.2 Gbits/sec                 
[  5]   6.00-7.00   sec  1.48 GBytes  12.7 Gbits/sec                 
[  5]   7.00-8.00   sec  1.45 GBytes  12.5 Gbits/sec                 
[  5]   8.00-9.00   sec  1.36 GBytes  11.7 Gbits/sec                 
[  5]   9.00-10.00  sec  1.48 GBytes  12.7 Gbits/sec                 
[  5]  10.00-10.04  sec  69.8 MBytes  13.4 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec  14.8 GBytes  12.7 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

and as client:
Code:
iperf3 -c x.x.x.x
Connecting to host x.x.x.x, port 5201
[  5] local x.x.x.x port 53444 connected to x.x.x.x port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.56 GBytes  13.4 Gbits/sec    0   2.31 MBytes       
[  5]   1.00-2.00   sec  1.59 GBytes  13.7 Gbits/sec    0   2.76 MBytes       
[  5]   2.00-3.00   sec  1.83 GBytes  15.7 Gbits/sec    0   2.76 MBytes       
[  5]   3.00-4.00   sec  1.76 GBytes  15.1 Gbits/sec    0   2.76 MBytes       
[  5]   4.00-5.00   sec  1.67 GBytes  14.4 Gbits/sec    0   2.76 MBytes       
[  5]   5.00-6.00   sec  1.61 GBytes  13.9 Gbits/sec    0   2.76 MBytes       
[  5]   6.00-7.00   sec  1.85 GBytes  15.9 Gbits/sec    0   2.76 MBytes       
[  5]   7.00-8.00   sec  1.87 GBytes  16.1 Gbits/sec    0   2.76 MBytes       
[  5]   8.00-9.00   sec  1.84 GBytes  15.8 Gbits/sec    0   2.76 MBytes       
[  5]   9.00-10.00  sec  1.75 GBytes  15.1 Gbits/sec    0   2.92 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  17.3 GBytes  14.9 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  17.3 GBytes  14.9 Gbits/sec                  receiver

iperf Done.

Thank you,
undoo99
 
Looks like you have some issues outside the proxmox infrastructure or between outside and inside proxmox ve.

Could you try a traceroute from the VM to your internal DNS server and aditionally a traceroute to the (external) google DNS?
 
Hi Marco,

This is a traceroute from the VM to the Google DNS:
Code:
traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  1.com (x.x.x.x)  0.885 ms  0.951 ms  1.047 ms
 2  x.x.x.x (x.x.x.x)  1.005 ms  0.984 ms  0.961 ms
 3  x.x.x.x (x.x.x.x)  28.857 ms  28.831 ms  28.827 ms
 4  x.x.x.x (x.x.x.x)  28.771 ms x.x.x.x (x.x.x.x)  28.778 ms x.x.x.x (x.x.x.x)  28.751 ms
 5  1.com (x.x.x.x)  28.730 ms  28.814 ms  28.795 ms
 6  * * *
 7  dns.google (8.8.8.8)  28.953 ms  28.793 ms  28.908 ms

I don't have an internal DNS server.

The VMs have public IPs and use public DNS servers (8.8.8.8 & 1.1.1.1) the host is behind a NAT and it uses an internal DNS (local) server.

How can a high DNS response time have something to do with the throughput between 2 VMs on the same host?

Thank you,
undoo99
 
I'm not sure, but could it be a problem with fragmented packets and a packet size limit on the gateway/firewall/router?

You might try to find out at what packet size you get fragmented packets from a VM on that system.
 
Both VMs have public IPs from the same subnet, so there is no firewall/bridge/router between them, only the open virtual switch.

I use Open Virtual Switch (OVS) and it has an MTU of 1500, also the VMs have an MTU of 1500.

Between the 2 VMs, the throughput with iperf3 is over 12Gbps.

But with apache2 as the server and wget as the client, the throughput is at around 250Mbps and 1 core, used by apache, is at 100%.

Where could be the problem?

Thank you,
undoo99
 
Hello,

The issue was with the CPU type.
It was set to "Default (kvm64)" after I changed it to "host" there was no performance issue.

Thank you Marco for your time.

Best regards,
undoo99
 
  • Like
Reactions: mgabriel

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!