VM disturbing the whole machine

[greg]

Greetings
I've been using Proxmox for years, mainly form CTs, with a few VMs.
Recently I started developing for Docker, so I created various VMs and CTs for testing.

The last VM I created is problematic: when it's running, the whole machine (host + other guests) are slow and show messages like:

NMI watchdog: BUG:soft lockup - CPU#2 stuck for 21s!

The problematic VM itself show weird messages:

rcu_sched detected stalls on CPUs/tasks:
6-...0: (1 GPs behind) idle=52a/1/0x4000000000 softirq=2573/2574 fqs=2506
(detected by 3, t=5252 jiffies, g=3089, q=271)
rcu_sched kthread starved for 1366 fiffies! g3089 f0x0 RCU_Gp_DOING_FQS(6) ->state=0x0 ->cpu=0

There's nothing running in it beside the Docker daemon. The "summary" page show low CPU usage, low memory, low network.

I'm puzzled... any idea?

Thanks in advance

Regards

 

[dcsapak]

maybe it is saturation your storage? this would explain why everything else is slow

 

[greg]

Thanks for your answer. Storage has 200G left. Would that be a possible explanation for my problem? when the VM is not running, everything works fine.

 

[dcsapak]

no i meant that the vm maybe generates a high io load (iow. reads and writes much) so that the disk is saturated

 

[greg]

oh no, nothing runs in the VM, zero io

 

[dcsapak]

not much what it can be then... anything in the guest/host logs ?

 

[greg]

Only the weird CPU/irq message I posted.
I'm trying to free a lot of space to test if space is the problem, I know ZFS deals poorly with low disk space.

 

[greg]

It seems you were right: I casually let the zfs partition go behind 80% and it was probably the root cause. After a lot of cleanup, things seems to be back to normal. Thanks a lot!!

 

[greg]

Well it wasn't the cause... I know have 2T free disk, the VM is using 1G of 3G allowed, server load is between 0 and 4 (for 12 cores) and I see this kind of things:

Jun 1 17:18:24 dkr06 kernel: [10071.847256] clear_huge_page+0x110/0x200
Jun 1 17:19:43 dkr06 kernel: [10150.892113] clear_subpage+0x3b/0x50
Jun 1 17:19:43 dkr06 kernel: [10150.892114] clear_huge_page+0x110/0x200
Jun 1 17:19:43 dkr06 kernel: [10150.892114] do_huge_pmd_anonymous_page+0x1b5/0x740
Jun 1 17:19:43 dkr06 kernel: [10150.892114] __handle_mm_fault+0xdba/0x1270
Jun 1 17:19:43 dkr06 kernel: [10150.892115] handle_mm_fault+0xd6/0x200
Jun 1 17:19:43 dkr06 kernel: [10150.892115] __do_page_fault+0x249/0x4f0
Jun 1 17:19:43 dkr06 kernel: [10150.892115] ? async_page_fault+0x8/0x30
Jun 1 17:19:43 dkr06 kernel: [10150.892115] async_page_fault+0x1e/0x30
Jun 1 17:19:43 dkr06 kernel: [10150.892116] RIP: 0033:0x7f4ca5351346
Jun 1 17:19:43 dkr06 kernel: [10150.892116] Code: 00 00 00 0f 18 8e c0 00 00 00 0f 10 06 0f 10 4e 10 0f 10 56 20 0f 10 5e 30 48 83 c6 40 48 83 ea 40 66 0f e7 07 66 0f e7 4f 10 <66> 0f e7 57 20 66 0f e7 5f 30 48 83 c7 40 48 83 fa 40 77 be 0f ae
Jun 1 17:19:43 dkr06 kernel: [10150.892117] RSP: 002b:00007ffd9a10a4b8 EFLAGS: 00010202
Jun 1 17:19:43 dkr06 kernel: [10150.892117] RAX: 00007f4c8aa00018 RBX: 00007f4c8aa00000 RCX: 00007f4c8f0297b1
Jun 1 17:19:43 dkr06 kernel: [10150.892118] RDX: 00000000008297a1 RSI: 00007f4c93000008 RDI: 00007f4c8e7fffe0
Jun 1 17:19:43 dkr06 kernel: [10150.892118] RBP: 00007f4ca2212050 R08: fffffffffffffff8 R09: 0000000000000000
Jun 1 17:19:43 dkr06 kernel: [10150.892118] R10: 00007f4c8f0297c1 R11: 00007f4c8aa00018 R12: 00000000046297a9
Jun 1 17:19:43 dkr06 kernel: [10150.892119] R13: 00007f4c8f200000 R14: 00007f4ca2211ff0 R15: 00007f4c97b07290


What could have I done wrong for this VM to be able to bring the whole system down???

 

[greg]

Interesting follow-up.

I moved a working VM to the host where the problem occured, and the VM is running erratically. So it's probably a host-related problem.

Both hosts are running PVE 5.4-15, the working is Intel Xeon E3-1245v2@3,4GHz, the problematic is Intel Xeon E5-1650 v2 @3.5 GHz.

Could it be a BIOS configuration issue?

 

[greg]

As further testing, I moved the problematic VM to another node, and it's working.
So it's very likely a host-related issue.

Any idea??