> 3000 mSec Ping and packet drops with VirtIO under load

aderumier · Sep 26, 2017

Andreas Piening said:
So you suggested to install the pve-qemu-kvm package from PVE 4?

yes

But I don't think it is easy because there might be a lot of dependencies, but we'll see.

only dependency is the old libscsi (which don't conflict with current libiscsi from proxmox5).
See my previous post with links to .deb

if you have time to test ,it could be great to known if it's work or not with qemu 2.7 from proxmox4, to be 100% sure that regression is coming from qemu 2.9.

AlBundy · Sep 26, 2017

This also is going on with LXC for me (4 qemu server that are almost idle and 1 LXC container doing a lot of write actions)

mac.linux.free · Sep 27, 2017

so it is not the kernel and not qemu-kvm?

AlBundy · Sep 27, 2017

Im going to check kernel, because i upgraded from 4.4 (still installed) will report my results later today

mac.linux.free · Sep 27, 2017

perhaps someone could push kernel 4.13 to pvetest as mentioned in roadmap to 5.1

AlBundy · Sep 27, 2017

Its already feeling faster, will keep you guys up to date

AlBundy · Sep 29, 2017

The last day it's running more stable and faster! Even during the same actions as I did Tuesday the IO delay didn't go sky-high and the vm didn't became unresponsive

aderumier · Sep 29, 2017

mac.linux.free said:
perhaps someone could push kernel 4.13 to pvetest as mentioned in roadmap to 5.1

available in pvetest ! :

http://download.proxmox.com/debian/...64/pve-kernel-4.13.3-1-pve_4.13.3-2_amd64.deb

mac.linux.free · Sep 29, 2017

aderumier said:
available in pvetest ! :

http://download.proxmox.com/debian/...64/pve-kernel-4.13.3-1-pve_4.13.3-2_amd64.deb

installed and running fine

Andreas Piening · Oct 2, 2017

aderumier said:
http://download.proxmox.com/debian/...64/pve-kernel-4.13.3-1-pve_4.13.3-2_amd64.deb

Code:

#uname -a
Linux pve7 4.13.3-1-pve #1 SMP PVE 4.13.3-2 (Wed, 27 Sep 2017 14:01:40 +0200) x86_64 GNU/Linux

I have installed pve-kernel-4.13.3-1-pve_4.13.3-2_amd64.deb on my test system and I can't see any difference. I have 18% packet loss during the boot process and I was pinging from the host. Response times fluctuating and often over 300ms.
My test server has nothing running but this single windows server 2016 KVM VM so it is pretty good to compare the behavior between a single change on the configuration.

mac.linux.free said:
installed and running fine

What do you mean by that? The kernel was installed successfully and boots, or did it resolve the issue we're discussing here for you?

Andreas Piening · Oct 2, 2017

AlBundy said:
The last day it's running more stable and faster! Even during the same actions as I did Tuesday the IO delay didn't go sky-high and the vm didn't became unresponsive

Can you explain what you have changed, please?

Andreas Piening · Oct 2, 2017

aderumier said:
can you try to install pve-qemu 2.7 from proxmox4 on your proxmox5 install ?

wget http://download.proxmox.com/debian/...n/binary-amd64/pve-qemu-kvm_2.7.1-4_amd64.deb
wget http://download.proxmox.com/debian/...ion/binary-amd64/libiscsi4_1.15.0-1_amd64.deb
dpkg -i *.deb

(old libiscsi is needed as dependency)

I have tried to install these two .deb packages:

Code:

#dpkg -i *.deb
Selecting previously unselected package libiscsi4.
(Reading database ... 66968 files and directories currently installed.)
Preparing to unpack libiscsi4_1.15.0-1_amd64.deb ...
Unpacking libiscsi4 (1.15.0-1) ...
dpkg: warning: downgrading pve-qemu-kvm from 2.9.1-1 to 2.7.1-4
Preparing to unpack pve-qemu-kvm_2.7.1-4_amd64.deb ...
Unpacking pve-qemu-kvm (2.7.1-4) over (2.9.1-1) ...
Setting up libiscsi4 (1.15.0-1) ...
dpkg: dependency problems prevent configuration of pve-qemu-kvm:
 pve-qemu-kvm depends on libgnutls-deb0-28 (>= 3.3.0); however:
  Package libgnutls-deb0-28 is not installed.
 pve-qemu-kvm depends on libnettle4 (>= 2.6); however:
  Package libnettle4 is not installed.
 pve-qemu-kvm depends on libpng12-0 (>= 1.2.13-4); however:
  Package libpng12-0 is not installed.

dpkg: error processing package pve-qemu-kvm (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.7.6.1-2) ...
Errors were encountered while processing:
 pve-qemu-kvm

I have searched on http://download.proxmox.com/debian/pve/dists/jessie/pve-no-subscription/binary-amd64/ but can't find the mentioned packages there.
My guess is these are system packages from debian. Maybe I can find these in the debian repos.
If you know where I can get these packages or if there is a better way, please let me know.

aderumier · Oct 2, 2017

Andreas Piening said:

I have tried to install these two .deb packages:

Code:

#dpkg -i *.deb
Selecting previously unselected package libiscsi4.
(Reading database ... 66968 files and directories currently installed.)
Preparing to unpack libiscsi4_1.15.0-1_amd64.deb ...
Unpacking libiscsi4 (1.15.0-1) ...
dpkg: warning: downgrading pve-qemu-kvm from 2.9.1-1 to 2.7.1-4
Preparing to unpack pve-qemu-kvm_2.7.1-4_amd64.deb ...
Unpacking pve-qemu-kvm (2.7.1-4) over (2.9.1-1) ...
Setting up libiscsi4 (1.15.0-1) ...
dpkg: dependency problems prevent configuration of pve-qemu-kvm:
 pve-qemu-kvm depends on libgnutls-deb0-28 (>= 3.3.0); however:
  Package libgnutls-deb0-28 is not installed.
 pve-qemu-kvm depends on libnettle4 (>= 2.6); however:
  Package libnettle4 is not installed.
 pve-qemu-kvm depends on libpng12-0 (>= 1.2.13-4); however:
  Package libpng12-0 is not installed.

dpkg: error processing package pve-qemu-kvm (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.7.6.1-2) ...
Errors were encountered while processing:
 pve-qemu-kvm

I have searched on http://download.proxmox.com/debian/pve/dists/jessie/pve-no-subscription/binary-amd64/ but can't find the mentioned packages there.
My guess is these are system packages from debian. Maybe I can find these in the debian repos.
If you know where I can get these packages or if there is a better way, please let me know.

I have theses 3 packages installed, but It was on a proxmox5 after upgrade from proxmox4.
they are coming from jessie package.
You can install them safely, no conflict with newer packages

http://ftp.us.debian.org/debian/pool/main/g/gnutls28/libgnutls-deb0-28_3.3.8-6+deb8u7_amd64.deb
http://ftp.us.debian.org/debian/pool/main/n/nettle/libnettle4_2.7.1-5+deb8u2_amd64.deb
http://ftp.us.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u3_amd64.deb

Andreas Piening · Oct 2, 2017

aderumier said:
I have theses 3 packages installed, but It was on a proxmox5 after upgrade from proxmox4.
they are coming from jessie package.
You can install them safely, no conflict with newer packages

http://ftp.us.debian.org/debian/pool/main/g/gnutls28/libgnutls-deb0-28_3.3.8-6+deb8u7_amd64.deb
http://ftp.us.debian.org/debian/pool/main/n/nettle/libnettle4_2.7.1-5+deb8u2_amd64.deb
http://ftp.us.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u3_amd64.deb

Thank you!

I had to download one additional library "libhogweed2" but then it installed fine. I rebooted the host after the installation was done.

But: Still the same issue!

aderumier · Oct 3, 2017

Andreas Piening said:
Thank you!

I had to download one additional library "libhogweed2" but then it installed fine. I rebooted the host after the installation was done.

But: Still the same issue!

if with kernel 4.4 + pve-qemu-kvm 2.7 both from proxmox 4 on proxmox 5, you still have the problem, I really don't understand what it can be ....

Are you 100% sure that you don't have the same problem with proxmox 4 on this specific server ?

felipe · Oct 3, 2017

hi, yes we are also experiencing similar problems but with ide. the guest is a windows 2012 (not r2)
we allways used ide drivers because virtio was slower in win2012 (NOT r2) then ide. (with windows 2012 r2 virtio is faster)
after upgrading to proxmox 5 from 4.4 the io load from the hoste node changed from under 1% to about 5% .
once we also experienced a total network hang - and had to reactivate the network card of the guest

as far as i can read there is still no solution fot this?

aderumier · Oct 3, 2017

felipe said:
hi, yes we are also experiencing similar problems but with ide. the guest is a windows 2012 (not r2)
we allways used ide drivers because virtio was slower in win2012 (NOT r2) then ide. (with windows 2012 r2 virtio is faster)
after upgrading to proxmox 5 from 4.4 the io load from the hoste node changed from under 1% to about 5% .
once we also experienced a total network hang - and had to reactivate the network card of the guest

as far as i can read there is still no solution fot this?

maybe problem is different. (different storage,network configuration).
maybe you can try to install pve-kernel 4.4 , test. and if it don't work, test pve-qemu-kvm 2.7 from proxmox4. (I have posted link in previous posts).

AlBundy · Oct 3, 2017

Andreas Piening said:
Can you explain what you have changed, please?

Sure thing, nothing special though!

I still had kernel version 4.4 installed, booted from that one and the whole system became more stable! I don't know if all issues are resolved but my I/O is back at 0.5% except for backup times then it goes to 4%

Phinitris · Oct 3, 2017

@AlBundy did you also had the network issues under load like lags and packet loss/high latency before?

felipe · Oct 9, 2017

aderumier said:
maybe problem is different. (different storage,network configuration).
maybe you can try to install pve-kernel 4.4 , test. and if it don't work, test pve-qemu-kvm 2.7 from proxmox4. (I have posted link in previous posts).

hi,
i can confirm now that with kernel 4.4.79 the load ist a lot better! (then the actual and newet kernel from proxmox 5.0)
i hope that with proxmox 5.1 the new 5.12 kernel will resolve the load problems.

> 3000 mSec Ping and packet drops with VirtIO under load

Renowned Member

Active Member

Renowned Member

Active Member

Renowned Member

Active Member

Active Member

Renowned Member

Renowned Member

Well-Known Member

Well-Known Member

Well-Known Member

Renowned Member

Well-Known Member

Renowned Member

Well-Known Member

Renowned Member

Active Member

Active Member

Well-Known Member