Network - High Frequenz VM

[Baader-IT]

Hi,

we want to build a high frequenz VM which have about ~300.000.000 incoming Data-Packages á day.

Now we have a big problem because there is a big packet loss (~1,5% - 3.000.000) but the packages arrive at the host (the Switch is not the Problem).
(We guess perhaps network buffering ?!)

Info to the VM:
-64GB RAM
-32 CPU's (2 sockets, 16 cores)
- NVME Harddisk

What we already did:
- VM with virtio network device
- Multiqueues = 8 (+ ethtool -L eth1 combined 8)

1) Do anyone can give me some infos, how I can reduce the packet loss or improve my network performance for this VM?

2) Is there a way to set the Multiqueues higher than 8? I can only set 8...
[root@vm ~]# ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 8


Greetings,
Tobi

 

[Baader-IT]

If I set the Multiqueues over my VM config file I can set them to an other value.

net1: virtio=F6:55:F2:37:1B:11,bridge=vmbr10,queues=16


Is there a way to change them over the GUI ?

 

[David Herselman]

You could try the following:

• Disable Hyper-Threading in the host's BIOS
• Pin the vCPUs to cores that are on the same processor the host NIC connects to. Reference the motherboard's block datagram
• There was talk that it's better to assign CPUs rather than cores to VMs but I assume that relates to users defining VMs with eg 8 cores where they should rather assign 2 virtual CPUs with 4 cores each to better map to physical host
• Increase VMs NIC queue length to buffer bursts of packets. This increase latency but can avoid packet loss

Herewith notes on how to increase the virtual NIC transmit buffer for VMs:

Increasing tap interface buffer:
  pico /etc/udev/rules.d/60-tap.rules
    KERNEL=="tap*", RUN+="/sbin/ip link set %k txqueuelen 25000"
  Changing existing tap interfaces:
    for dev in `ip a | grep tap | perl -pe 's/.*(tap.*?):.*/\1/'`; do ifconfig $dev txqueuelen 25000; done                                                          
  Changing receive buffer:
    pico /etc/sysctl.conf
      # Increase receive buffer queue
      net.core.netdev_max_backlog = 25000
    sysctl -p;
    sysctl -a | grep max_backlog;

You can also consider increasing the physical NIC buffers:

/etc/rc.local:
# Increase ring buffers:
[ `cut -d. -f1 /proc/uptime` -le 600 ] && for f in eth0 eth1 eth2 eth3; do ethtool -G $f rx 4096 tx 4096; sleep 5; done

Check current and max NIC ring buffer:
ethtool -g eth0

Get NIC statistics:
ethtool -S eth0

PS: Physical NICs typically have 64 queues which are equally allocated to cores. You most probably don't have buffering problems on the host. You can review 'ifconfig' to review drop or overrun counters.

 

[David Herselman]

Interesting reading:
https://blog.cloudflare.com/how-to-receive-a-million-packets/
https://blog.cloudflare.com/how-to-achieve-low-latency/amp/

 

[spirit]

whats is the pps rate per second ?

if you don't need live migration, you could try pci passthrough too.

 

[Baader-IT]

if you don't need live migration, you could try pci passthrough too.

We need live migration because of our HA strategy.

The pps rate wasn't analysed.
Why do you need this info?
I will analyse the pps rate with our network team.

You could try the following.

Good Info!
We will try this asap and will give you a feedback.
Thanks!

 

[Baader-IT]

Pin the vCPUs to cores that are on the same processor the host NIC connects to.

How can I pin the vCPUs to cores ?
Didn't find anything in the Proxmox documentation.

 

[David Herselman]

How can I pin the vCPUs to cores ?

You should define NUMA regions to match the physical host's and draw CPUs from pools. Documentation is unfortunately very sparse (numaN):
https://pve.proxmox.com/wiki/Manual:_qm.conf

The following sample, most relevant somewhere like rc.local, would then need to run after starting the guest:

cpus='4-19,24-39';
for pid in `pidof kvm`; do
  taskset -a -cp $cpus $pid;
  for vhostpid in `pidof vhost-$pid`; do
    taskset -a -cp $cpus $vhostpid;
  done
done
for pid in `pidof ceph-fuse ceph-mon ceph-osd`; do
  taskset -a -cp $cpus $pid;
done
# Assign reserved CPUs to 'syrex-sip':
for pid in `qm list | grep -P '\ssyrex-sip\s' | awk '{print $6}'`; do
  taskset -a -cp 0-3,20-23 $pid
  for vhostpid in `pidof vhost-$pid`; do
    taskset -a -cp $cpus $vhostpid;
  done
done