> 3000 mSec Ping and packet drops with VirtIO under load

[mac.linux.free]

I just saw, there where a lot of fixes in NetKVM for Windows in the latest virtio drivers:

https://fedorapeople.org/groups/virt/virtio-win/CHANGELOG

 

[aderumier]

If it's really a proxmox upgrade bug, I think it can be only 2 thing : qemu version, or kvm version (host kernel module).

Maybe can you try to install proxmox 4 pve-qemu-kvm and pve-kernel deb package on your proxmox 5 installation

try first with qemu:
-----------------------------
wget http://download.proxmox.com/debian/...n/binary-amd64/pve-qemu-kvm_2.7.1-4_amd64.deb
wget http://download.proxmox.com/debian/...ion/binary-amd64/libiscsi4_1.15.0-1_amd64.deb
dpkg -i *.deb

(old libiscsi is needed as dependency)

then with kernel
---------------------------
wget http://download.proxmox.com/debian/...d64/pve-kernel-4.4.8-1-pve_4.4.8-52_amd64.deb
dpkg -i *.deb

 

[udo]

I thought about it and should add a few details about my network setup:
The NICs of both Windows Server 2016 VMs are connected to a bridge:

brctl show vmbr1
bridge name    bridge id        STP enabled    interfaces
vmbr1        8000.aace816c169a    no        tap0
                                            tap100i0
                                            tap101i0

tap0 is a OpenVPN server tap device and the other two are the VMs.
So there is no physical device attached to this particular bridge. The NICs are shows as 10 GBit devs inside the VMs. So could it just be that the VMs can send quicker than the bridge can handle the traffic?
Is it a good idea to us the "Rate Limit" on the NICs in the PVE configuration? Since the "outside" is connected via WAN / OpenVPN it would probably never exceed 300 MBits and this would still be enough for backups and everything else.

Someone with experience on this? Am I thinking in the wrong direction?

Hi Andreas,
your network has no real interface - does this mean all traffic (backup, Datev-DB, rdp is going through the openvpn-connection?
Or is the high network load between VM 100 + VM 101?

If the main traffic on openvpn, how looks the pings between VM 100 + VM 101 if this happens?!

Perhaps the openvpn-tunnel is the problem? (MTU?)

And like mac.linux.free allready wrote, OpenVswitch is perhaps a solution - I have measure an better performance with openvswitch against the normal linux-bridge.

Udo

 

[Andreas Piening]

Hi Andreas,
your network has no real interface - does this mean all traffic (backup, Datev-DB, rdp is going through the openvpn-connection?
Or is the high network load between VM 100 + VM 101?

If the main traffic on openvpn, how looks the pings between VM 100 + VM 101 if this happens?!

Perhaps the openvpn-tunnel is the problem? (MTU?)

And like mac.linux.free allready wrote, OpenVswitch is perhaps a solution - I have measure an better performance with openvswitch against the normal linux-bridge.

Hi Udo,

most of the traffic is between VM100 + VM101. VM100 is the Domain Controller / Fileserver and Datev DB server while VM101 is the Terminal Server. The "outside world" connects via OpenVPN (tap0) to connect to the Terminal Server via RDP. There is also some traffic targeting a locally installed printer but it's only active when someone prints and there is generally not much traffic going through the VPN at all.

The problem occurs even if there is no load on the VPN because no one is using RDP outside the office hours. When the scheduled backup starts, my monitoring system reports packet loss and after the backup is done the system responds to pings again.
The monitoring node runs on the host (PVE server) and is directly pinging the two systems over the bridge. No OpenVPN involved here.
So when the backup on VM100 is running, VM101 does still respond to pings (at least it does not have more than 3 ping losses out of 5 which is the alert I configured). But the DB check runs locally on VM100 and the system still looses network connectivity completely.

I have exactly the same setup for other clients, only the older PVE version and Windows Server 2012. I can work on the systems while the backup is running with RDP and do a DB check at the same time and update Applications at the same time. Of course the system gets slower then but it is still responsive and the ping responses are fluctuating just a little bit as it is normal for bare metal systems as well.

Another user "micro" mentioned in this thread that he uses OVS and has the same issue.
My issue is the IO load of the VM100 while it is starting is enough to get huge ping delays and packet loss. It takes like 10 minutes to boot the machine and start every service until the system is idling and I cannot connect to the system before.
I can connect to my Windows Server 2012 on slower hardware after around 60 seconds when I push the start button of the VM and it basically runs the same services.

My network requirements are pretty low, getting higher transfer speeds is always a good thing but my issue is the network is completely unreliable if there is disk IO on the guests. I don't see this as a issue that is related only to the network.

 

[micro]

I managed to pinpoint where exactly the problem in my system.
It is only happening if I'm using NFQUEUE target in my iptables. If the network packets are routed via nfqueue for packet decision they are severely delayed if any IO wait on the system is shown. Unfortunately I really need to use nfqueue for packet decision and now I'm wondering why IO wait is blocking nfqueue processing.

 

[Andreas Piening]

Interesting. So when you don't use NFQUEUE your network is reliable even under IO load in your VMs? And that's the only thing you've changed?

I wonder if we have the same issue then, since from my knowledge I don't use NFQUEUE. At least not on purpose.
I'm using shorewall on my PVE host, which basically manages iptables based on config files.
I have very basic rules: Drop / Reject policy on my interfaces and the Accept for the few ports I use.
How can I check if I use NFQUEUE?

iptables --list | grep queue

does not return anything.

 

[micro]

Interesting. So when you don't use NFQUEUE your network is reliable even under IO load in your VMs? And that's the only thing you've changed?

I wonder if we have the same issue then, since from my knowledge I don't use NFQUEUE. At least not on purpose.
I'm using shorewall on my PVE host, which basically manages iptables based on config files.
I have very basic rules: Drop / Reject policy on my interfaces and the Accept for the few ports I use.
How can I check if I use NFQUEUE?

iptables --list | grep queue

does not return anything.

try with:

iptables -nvL |grep NFQUEUE

it is in upper case.

I guess your rules are pretty simple and you know what they are doing actually (only accept/drop/...)

I'm wondering if shorewall or something else in your system is using something similar networking in userspace ?

I'm redirecting all traffic from few interfaces via nfqueue to a user space process for the decision making (should I let this packet to go or not). The IO load doesn't affect the ping to vms (or other hosts) which are not behind this router (with nfqueue moderation). If I disable the nfqueue decision and just accept the packets (without the verification) the latency is gone, even I start a dd if=/dev/zero of=test bs=10M load on the shared storage. The IO wait is there, the ping is normal (1-2ms).

 

[Andreas Piening]

So "iptables -nvL | grep NFQUEUE" does not return anything here.
Shorewall just compiles the rules from my config files and loads them with iptables on startup. It does not run in userspace.

It seems that we have different issues. Regarding to networking all I have done is I have defined the additional local bridge, configured OpenVPN and installed Shorewall to manage my firewall rules with iptables. In PVE I selected the local bridge to assign the Interfaces of my VMs.

It may still be possible that our problems are related to each other, but NFQUEUE does not seem to be the connection.

 

[micro]

I agree - it is not nfqueue in your case. But you are using OpenVPN. Is your bad network experience via the OpenVPN only?
Because OpenVPN is also an userspace daemon. And as such you are probably affected by the same problem - userspace network related process hanging because of IO load. Especially if you are using OpenVPN on TCP port (instead of UDP).
I think this network latency we are experiencing is because of userspace processes blocked from doing their network job if there is IO wait in the system.

 

[Andreas Piening]

No, my bad network experience is not limited to the communication over OpenVPN:
My monitoring daemon (monit) does ping tests from the PVE host to both Windows VMs directly via the bridge. There is no OpenVPN involved.
And these Pings are affected by the IO load in the guest VM. The backup software installed inside the VM is started by a scheduler and it communicates to another host on the internet via SMB. So this traffic is routed by PVE from the bridge, it does not use OpenVPN.
My understanding is that the disk IO caused by the backup process, has direct influence on the network in a way that the PVE host cannot ping the guest over the bridge while there is IO load.
The backup happens at night when no one using the OpenVPN connection. I then get Ping alerts via Email from the monitoring system that says that there has been no ping responses while the backup was running.
I analysed this while other IO intense tasks (Datev DB check) was running by pinging from the host to the VM over the bridge. I was connected with SSH, not using the OpenVPN connection. The host with the IO load does not respond to pings (or with huge delays) while the other host still does.

 

[manu]

if your storage is so slow that a ping request is blockend by a pending write on the storage, you should maybe start benchmarking the storage too
Are you using VirtiO Network drivers for the VM NIC ?
It would be interesting to see the vm.conf

 

[micro]

In my case the storage is not slow. It is achieving about 80-90MB/s and about 600 fsync/s (pveperf). My VMs are mostly idle, so I doubt more than 2-3% of the storage is utilized at any time. Those io wait are some spikes (probably some software inside the VMs are flushing their data) so I really didn't utilize the storage or it's capacity. Maybe there is something about how exactly the PVE nodes are flushing the data to the shared storage.

 

[manu]

@micro, can you post your vm.conf ?

 

[micro]

103.conf:

#XXXX
agent: 1
args: -cpu kvm64,+ssse3
balloon: 16384
bootdisk: virtio0
cores: 8
memory: 32768
name: XXXX.XXXXX.XXX
net0: virtio=52:16:78:785:10,bridge=vmbr1,tag=XXX
net1: virtio=4A:1D:FC:A4:BA:71,bridge=vmbr1,tag=XXX
net2: virtio=4A:C2:97:37:3A:92,bridge=vmbr1,tag=XXX
net3: virtio=BA:E0:74:68:003,bridge=vmbr1,tag=XXX
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=8b3488e0-2dc2-4308-b7fc-9901c2cc1e83
sockets: 1
startup: order=1
tablet: 0
vga: qxl
virtio0: airraid:vm-103-disk-1,size=20G

 

[micro]

I found this topic here: https://forum.proxmox.com/threads/storage-san-fibre-channel-and-load-sharing.26563/
I think maybe I'm having the same issue here. All my VMs right now are capped (their disk throttled) to 30MB/s and still I have the issue.
I found also that restoring from backup to the shared storage also loads heavily nodes (Load, IO wait).

 

[Andreas Piening]

if your storage is so slow that a ping request is blockend by a pending write on the storage, you should maybe start benchmarking the storage too
Are you using VirtiO Network drivers for the VM NIC ?
It would be interesting to see the vm.conf

Since the ping requests are handled mainly by CPU / RAM I didn't expect local storage IO to introduce this kind of latency. Even not with IOwait.

But doing some benchmarking on the storage was a good hint and honestly the results are confusing to me:
I downloaded CrystalDiskMark 5.2.2 x64 and did a run with the default settings (5 x 1 GiB):
Sequential, 32 queues, 1 Thread => 2588 MB/sec Read / 857.8 MB/s Write
Random, 4kiB, 32 queues, 1 Thread => 409.3 MB/sec Read / 358.0 MB/s Write
Sequential => 2109 MB/sec Read / 1506 MB/s Write
Random, 4kiB => 58.07 MB/sec Read / 46.60 MB/s Write

These high values are completely unreal. I have two SATA drives with 7.200 rpm in RAID 1 and I would expect something like 100 to 200 MB/s per second. Seems some weird caching is going on.
What tool / settings should I use to get realistic results?

But even more confusing was the ping results while doing the benchmark. 835 packets transmitted, 832 packets received, 0.4% packet loss and this was over OpenVPN / WAN so I can't even blame VirtIO for that.
So I wonder why the benchmark does not introduce the connectivity issues I had before while running the full backup with Veeam.

Right: I use VirtIO network drivers for the VM NIC.

VM config (MAC replaced with FF):

#MS Windows Server 2016 Std - Terminal Server
agent: 1
boot: dc
bootdisk: scsi0
cores: 6
ide2: none,media=cdrom
memory: 131072
name: winsrvts
net0: virtio=FF:FF:FF:FF:FF:FF,bridge=vmbr1
numa: 0
onboot: 1
ostype: win10
scsi0: local-zfs:vm-101-disk-1,discard=on,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=ae1a0841-b9a1-42e9-aea9-1fd0eff3af30
sockets: 1
startup: order=2

Kind regards

Andreas

 

[micro]

I did some more experiments this morning and found that:
- changing the vm storage from virtio to sata doesn't help
- changing the SCSI Controller type from VirtIO SCSI to Default (LSI 53C895A) doesn't help
+ changing the network cards type from virtio to e1000 helped a little.

It is still happening, not that often, the hiccups are shorter.
The biggest difference now with e1000 is If I throttle the disk of the other test VM to 1/3 of the FC SAN storage and try to inflict iowait to the storage (with dd) the packets are with normal latency. Previously, with virtio network card throttling the disks of the other VM still caused network latency and blocking the network for a second or two.

Of course this isn't resolving the issue, because for example restoring some VM from backup to the shared storage (SAN) of the cluster would cause again latency in this VM and it's routing functions.

P.S: e1000 is working little bit better but now it is generating 5x more software irqs which are not problem right now for me but it may be in the future. I don't like the idea so I decided to move this VM's storage from shared to the local storage of this particular cluster node and changed again to virtio for networking. Finally, no network hiccups, no huge latency anymore. Yeah, no hardware HA for this VM, sounds terrible, but I can't tolerate anymore the interrupts/latency in this VM.

 

[mac.linux.free]

today there where some updates on pve-kernel and qemu...

 

[micro]

I found some other thread and bug report filled about it. Looks like the same problem.
https://forum.proxmox.com/threads/k...ckup-restore-migrate.34362/page-2#post-174960
https://bugzilla.proxmox.com/show_bug.cgi?id=1453

 

[Andreas Piening]

I found some other thread and bug report filled about it. Looks like the same problem.
https://forum.proxmox.com/threads/k...ckup-restore-migrate.34362/page-2#post-174960
https://bugzilla.proxmox.com/show_bug.cgi?id=1453

Thank you. The more I read and think about it, the more confusing it gets.
I do backups from within VMs on PVE 4.4-17 and one machine has 1.8 TB of data and even while doing a full backup I have no packet drops and my ping responses are fine. Using ZFS + KVM + virtIO + OpenVPN there but it is a linux guest instead.
So to me it looked like this must be a PVE 5 issue, but in this bug report says the issue was also in PVE 4.

And you wrote in comment #37 that the issue was gone after you switched to local storage?
Is the issue really completely gone?
I am on local storage with these VMs I'm experiencing the issue with, right from the start.