KVM Hypervisor slows down over time

[squeeb]

Hi all,

I'm having a really weird issue in that a KVM hypervisor running CentOS 6.7 starts up fine and then anywhere between 3 - 12 hours later, anything that requires the opening of a new socket or file descriptor becomes very slow.

While it is slow, telnetting to port 22 from an external host (my laptop) to the hypervisor sshd results in a connection being made instantly, IE the syn-ack happens and a session is established but the "SSH-2.0-OpenSSH" header doesn't appear until around 20-40 seconds later.
Standard ssh connections take even longer to get a prompt.

It's not just ssh either, it's any service that's running on the host.

Once I am in the host via ssh however, everything looks fine. Load is normal, CPU usage is fine, I/O wait is nothing, memory usage is fine, no swap used.

I don't even know where to start debugging in this case and would love a few pointers.

The host node itself seems absolutely fine aswell with other Hypervisors running fine along side OpenVZ containers.

 

[spirit]

for ssh, it's look like a bad dns resolution on the host.

 

[squeeb]

for ssh, it's look like a bad dns resolution on the host.

Yes but I should still see the openssh header via telnet, regardless of host lookup.
This issue isn't just affecting SSH connections, it's any service that's running on the system.
Even launching 'top' takes a few seconds.

 

[sigxcpu]

strace a hanging process and find out where it stalls.

 

[squeeb]

This is the issue, none of the processes are 'hanging', they eventually operate and run but very slowly - stracing reveals nothing other than normal syscalls.

Even stracing the kvm process itself reveals nothing unusual..

 

[sigxcpu]

There is "perf" on Linux that can help you to see where a process spends most time. Search for "Brendan Gregg" on Google and you will find tons of ideas on how to monitor performance. You don't know if you are CPU bound, I/O bound or simply some network service is slow (e.g. DNS).