PVE 6, Kernel 5.3, Minkube, Docker Kubernetes not working

C

CyManiac

Member
Apr 13, 2018
4
3
23
47
With kernel versions > 5.0 you will get general protection faults, sigsegs which affect other VMs on the same node as well.
Setup:
VM with
- ubuntu (tested 18-19)
- debian buster
- docker (tested 18-19)
- K8s 17, 16
- Minibkube 16, 15
After a while the apiserver starts crashing and didn't come back properly.
On hypervisor you get:
Dec 19 15:38:19 proxmox2 kernel: [ 165.923684] WARNING: CPU: 11 PID: 2342 at arch/x86/kvm/x86.c:8056 vcpu_enter_guest+0x134e/0x1570 [kvm]
Dec 19 15:38:19 proxmox2 kernel: [ 165.923699] snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_hda_codec snd_hda_core i915 snd_hwdep snd_pcm intel_rapl_perf snd_timer snd drm_kms_helper pcspkr wmi_bmof intel_wmi_thunderbolt mxm_wmi soundcore drm i2c_algo_bit mei_me fb_sys_fops syscopyarea sysfillrect sysimgblt mei intel_pch_thermal ie31200_edac mac_hid acpi_pad acpi_tad zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nct6775 hwmon_vid coretemp sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 e1000e i2c_i801 ahci libahci wmi video
Dec 19 15:38:19 proxmox2 kernel: [ 165.923714] CPU: 11 PID: 2342 Comm: kvm Tainted: P O 5.3.13-1-pve #1
Dec 19 15:38:19 proxmox2 kernel: [ 165.923714] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390 Extreme4, BIOS P2.30 12/25/2018
Dec 19 15:38:19 proxmox2 kernel: [ 165.923723] RIP: 0010:vcpu_enter_guest+0x134e/0x1570 [kvm]
Dec 19 15:38:19 proxmox2 kernel: [ 165.923724] Code: 00 00 e9 e1 f2 ff ff f0 80 63 33 ef 48 8b 43 78 45 31 e4 c7 40 08 18 00 00 00 48 8b 43 78 c7 40 20 02 00 00 00 e9 b1 ef ff ff <0f> 0b e9 1c ee ff ff f6 43 3a 02 0f 85 99 27 03 00 48 8b 05 d2 55
Dec 19 15:38:19 proxmox2 kernel: [ 165.923724] RAX: 0000000000004800 RBX: ffff8f0a0cfc0000 RCX: ffff8f0a0cc55000
Dec 19 15:38:19 proxmox2 kernel: [ 165.923725] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec 19 15:38:19 proxmox2 kernel: [ 165.923725] RBP: ffffa29dc21dbd88 R08: 0000000000000000 R09: 0000000000000000
Dec 19 15:38:19 proxmox2 kernel: [ 165.923726] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8f0a0cd90700
Dec 19 15:38:19 proxmox2 kernel: [ 165.923727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 19 15:38:19 proxmox2 kernel: [ 165.923728] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 19 15:38:19 proxmox2 kernel: [ 165.923739] ? restart_apic_timer+0xa0/0x140 [kvm]
Dec 19 15:38:19 proxmox2 kernel: [ 165.923752] ? do_futex+0xc4/0xc50
Dec 19 15:38:19 proxmox2 kernel: [ 165.923756] ? _copy_to_user+0x2b/0x40
Dec 19 15:38:19 proxmox2 kernel: [ 165.923760] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Dec 19 15:38:19 proxmox2 kernel: [ 165.923761] RSP: 002b:00007fa495379678 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Dec 19 15:38:19 proxmox2 kernel: [ 165.923762] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000019
Dec 19 15:38:19 proxmox2 kernel: [ 165.923763] R10: 0000000000000001 R11: 0000000000000246 R12: 00007fa495d0c200
Dec 19 15:38:19 proxmox2 kernel: [ 165.923764] ---[ end trace 6d0f3f298a50e567 ]---


On guests you will get:
syslog.1:Dec 25 03:34:23 minkube kernel: [29633.506742] traps: kubectl[27625] general protection ip:422741 sp:7ffc0f8ceb48 error:0 in kubectl[400000+1084000]
syslog.1:Dec 25 19:21:02 minkube kubelet[1550]: [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x719800]
syslog.1:Dec 25 21:34:07 minkube dockerd[757]: [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x5589f7643dcc]
syslog.1:Dec 25 15:30:11 minkube kernel: [72580.551635] traps: python3[11678] general protection ip:5b8aed sp:7fc648f307b0 error:0 in python3.7[421000+234000]
syslog.1:Dec 25 16:17:07 minkube kernel: [75396.639425] traps: docker-init[13610] general protection ip:4603b2 sp:7ffe5cf77728 error:0 in docker-init[400000+bd000]

Dec 20 02:56:07 emailarchiv kernel: [33168.356798] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Dec 20 02:56:07 emailarchiv kernel: [33168.356853] rcu: (detected by 1, t=5639 jiffies, g=774921, q=0)
Dec 20 02:56:07 emailarchiv kernel: [33168.356890] rcu: All QSes seen, last rcu_sched kthread activity 5639 (4303184458-4303178819), jiffies_till_next_fqs=1, root ->qsmask 0x0
Dec 20 02:56:07 emailarchiv kernel: [33168.356964] swapper/1 R running task 0 0 1 0x80000000
Dec 20 02:56:07 emailarchiv kernel: [33168.356967] Call Trace:
Dec 20 02:56:07 emailarchiv kernel: [33168.356980] <IRQ>
Dec 20 02:56:07 emailarchiv kernel: [33168.356994] sched_show_task.cold.85+0x9a/0xc6
Dec 20 02:56:07 emailarchiv kernel: [33168.356999] rcu_check_callbacks.cold.81+0x31e/0x335
Dec 20 02:56:07 emailarchiv kernel: [33168.357004] ? tick_sched_do_timer+0x60/0x60
Dec 20 02:56:07 emailarchiv kernel: [33168.357007] update_process_times+0x28/0x60
Dec 20 02:56:07 emailarchiv kernel: [33168.357009] tick_sched_handle+0x22/0x60
Dec 20 02:56:07 emailarchiv kernel: [33168.357010] tick_sched_timer+0x37/0x70
Dec 20 02:56:07 emailarchiv kernel: [33168.357011] __hrtimer_run_queues+0x100/0x280
Dec 20 02:56:07 emailarchiv kernel: [33168.357013] hrtimer_interrupt+0x100/0x220
Dec 20 02:56:07 emailarchiv kernel: [33168.357019] smp_apic_timer_interrupt+0x6a/0x140
Dec 20 02:56:07 emailarchiv kernel: [33168.357024] apic_timer_interrupt+0xf/0x20
Dec 20 02:56:07 emailarchiv kernel: [33168.357025] </IRQ>
Dec 20 02:56:07 emailarchiv kernel: [33168.357030] RIP: 0010:native_safe_halt+0xe/0x10
Dec 20 02:56:07 emailarchiv kernel: [33168.357032] Code: ff ff 7f c3 65 48 8b 04 25 40 5c 01 00 f0 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 90 e9 07 00 00 00 0f 00 2d 96 c0 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 86 c0 4d 00 f4 c3 90 90 0f 1f 44 00
Dec 20 02:56:07 emailarchiv kernel: [33168.357033] RSP: 0018:ffffb1d4401ffea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Dec 20 02:56:07 emailarchiv kernel: [33168.357034] RAX: ffffffffaa32b880 RBX: 0000000000000001 RCX: ffffffffaac4f350
Dec 20 02:56:07 emailarchiv kernel: [33168.357034] RDX: 0000000000292196 RSI: ffffffffaac4afb8 RDI: 0000000000000001
Dec 20 02:56:07 emailarchiv kernel: [33168.357035] RBP: 0000000000000001 R08: 00003651060a5723 R09: ffff896fb5a86400
Dec 20 02:56:07 emailarchiv kernel: [33168.357035] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
Dec 20 02:56:07 emailarchiv kernel: [33168.357036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec 20 02:56:07 emailarchiv kernel: [33168.357037] ? __sched_text_end+0x7/0x7
Dec 20 02:56:07 emailarchiv kernel: [33168.357042] default_idle+0x1c/0x140
Dec 20 02:56:07 emailarchiv kernel: [33168.357045] do_idle+0x1f1/0x280
Dec 20 02:56:07 emailarchiv kernel: [33168.357046] ? do_idle+0x18f/0x280
Dec 20 02:56:07 emailarchiv kernel: [33168.357047] cpu_startup_entry+0x6f/0x80
Dec 20 02:56:07 emailarchiv kernel: [33168.357053] start_secondary+0x1a4/0x1f0
Dec 20 02:56:07 emailarchiv kernel: [33168.357056] secondary_startup_64+0xa4/0xb0
Dec 20 02:56:07 emailarchiv kernel: [33168.357059] rcu: rcu_sched kthread starved for 5639 jiffies! g774921 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
Dec 20 02:56:07 emailarchiv kernel: [33168.357110] rcu: RCU grace-period kthread stack dump:
Dec 20 02:56:07 emailarchiv kernel: [33168.357143] rcu_sched R running task 0 10 2 0x80000000
Dec 20 02:56:07 emailarchiv kernel: [33168.357145] Call Trace:
Dec 20 02:56:07 emailarchiv kernel: [33168.357147] ? __schedule+0x2a2/0x870
Dec 20 02:56:07 emailarchiv kernel: [33168.357148] schedule+0x28/0x80
Dec 20 02:56:07 emailarchiv kernel: [33168.357149] schedule_timeout+0x16b/0x390
Dec 20 02:56:07 emailarchiv kernel: [33168.357151] ? __next_timer_interrupt+0xc0/0xc0
Dec 20 02:56:07 emailarchiv kernel: [33168.357152] rcu_gp_kthread+0x40d/0x850
Dec 20 02:56:07 emailarchiv kernel: [33168.357156] ? call_rcu_sched+0x20/0x20
Dec 20 02:56:07 emailarchiv kernel: [33168.357157] kthread+0x112/0x130
Dec 20 02:56:07 emailarchiv kernel: [33168.357158] ? kthread_bind+0x30/0x30
Dec 20 02:56:07 emailarchiv kernel: [33168.357160] ret_from_fork+0x35/0x40


This is _not_ a hardware fault. I migrated the affected machines on three different nodes and the error migrates with them.

Solution:

Boot hypervisor with 5.0 kernel and everything works as expected.

Maybe the guys from Proxmox are able to provide a reason why this happens and a fix?
 
Last edited:
Can you please provide the output of pveversion -v and tell us on which hardware you tried this?
CyManiac said:
I migrated the affected machines on three different nodes and the error migrates with them.
Click to expand...
Did you do a live migration or did you shut the VM down and then booted it fresh on the other node?
 
Hi Aaron,

thank you for you reply.

root@pve01:/home/chris# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 4.13.13-2-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-4.15: 5.4-6
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.15-1-pve: 5.0.15-1
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Base Board Information
Manufacturer: ASRock
Product Name: B85M Pro4
Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
4x Crucial Ballistic Sport 8GB DDR3-1600 UDIMM BLS8G3D1609DS1S00

---

root@proxmox2:/home/chris# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Manufacturer: ASRock
Product Name: Z390 Extreme4
4x Crucial BLS16G4D30AESE.M16FE
Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz

---


I use ansible to install a clean Debian, download and install the tools directly. So I restarted the nodes and did a clean installation based on the same playbooks. First I tested with Minikube. Later I installed a k8s cluster directly. In all cases I got the errors shown above with the latest kernel. When switching back to kernel 5.0 everything works as expected. The error is reproducible. If I boot with the latest kernel, the error comes back.

Cheers

Chris
 
I did a Kubernetes setup (for the first time) with Rancher. I used Proxmox 6 to create 10 KVMs, which I split into Control Plane, etcd, Rancher, and Worker servers. Debian Buster with upstream Docker was the base setup. I set "CPU Type" to "host" in Proxmox. I am running a recent PVE 5.3 kernel. I don't see those segfaults.
 
this bug only in PVE with kernel 5.3.
kernel 5.0 and older not have any problem with this issue.
I see problem with any configuration CPU type, except kvm*.
 
jebbam said:
I did a Kubernetes setup (for the first time) with Rancher. I used Proxmox 6 to create 10 KVMs, which I split into Control Plane, etcd, Rancher, and Worker servers. Debian Buster with upstream Docker was the base setup. I set "CPU Type" to "host" in Proxmox. I am running a recent PVE 5.3 kernel. I don't see those segfaults.
Click to expand...

If you will use VM OS CentOS 7/8 with PVE kernel 5.3 and CPU type "host" you will have this problem
 
Ok, it is not related to docker or Minikube. I just built a new server which shows the same problem,.
Happy to see that I'm not alone with this.
I use kvm64 as cputype and will give host a try.

Thx
 
  • Like
Reactions: apgpavel
CyManiac said:
Ok, it is not related to docker or Minikube. I just built a new server which shows the same problem,.
Happy to see that I'm not alone with this.
I use kvm64 as cputype and will give host a try.

Thx
Click to expand...

I also confirm that this problem persists in proxmox 6 + kernel 5.3
this behavior is related to the type of processor or subtype CPU which you are use and not related by functionality of proxmox.
this is BUG, whatever and whoever wrote here.
this problem is observed I think on all types of Intel Skylake Xeon processors and some Cascadelake

You can repeat this problem if You will use type of CPU "host" or some specefic type for example "Skylake server", and I can't repeat problem if use CPU type "kvm64" of VM. (This problem was tested from my side on more than 10 Intel CPU servers)
 
Last edited:
You must log in or register to reply here.
Top Bottom