Soft lockup on Proxmox 4.4

B

biddy

New Member
Apr 9, 2014
7
0
1
Hello,
I'm been running a Proxmox server for about three years now. A few months ago I started coming home to a crashed server. It boots fine after a restart, but after a few hours, maybe a day, it crashes again. Last night I logged into the system and just left the terminal up so I could see what kind of error message is on the screen when it crashes and this is what I'm seeing:

kernel:[93679.838175] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kvm:1610]

It's this, over and over. Every few seconds it throws up this message again. The CPU# changes but the message about a soft lockup stays.

Does anyone have any tips on chasing this down? My initial thoughts were perhaps a hardware problem. The PSU in this system was old and slightly underpowered to begin with so I went ahead and bought a new and higher wattage power supply. I'm still getting the errors.
 
anything suspicious in the syslog/journal?
 
dcsapak said:
anything suspicious in the syslog/journal?
Click to expand...
I'm not exactly sure where the error begins, but this is what repeats in syslog over and over once the lock up starts. These messages loop over and over on the screen while the VMs running become unresponsive after the lock up begins. The web GUI becomes inaccessible at this time as well.

Code: 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kvm:1659]
Feb 24 22:24:50 insley kernel: [35081.562317] Modules linked in: input_leds ip_set ip6table_filter ip6_tables iptable_filter ip_tables x_tables softdo
g nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scs
i_transport_iscsi nfnetlink_log nfnetlink snd_hda_codec_hdmi nouveau mxm_wmi wmi video ttm arc4 drm_kms_helper rt61pci snd_hda_codec_realtek drm rt2x0
0pci rt2x00mmio snd_hda_codec_generic rt2x00lib i2c_algo_bit mac80211 fb_sys_fops ppdev gpio_ich snd_hda_intel syscopyarea sysfillrect sysimgblt coret
emp snd_hda_codec snd_hda_core cfg80211 kvm_intel snd_hwdep snd_pcm kvm snd_timer eeprom_93cx6 snd crc_itu_t soundcore irqbypass shpchp pcspkr i7core_
edac serio_raw edac_core 8250_fintek parport_pc lpc_ich parport mac_hid vhost_net vhost macvtap macvlan autofs4 ses enclosure i2c_i801 pata_acpi uas p
ata_it8213 usb_storage r8169 ahci mii libahci fjes
Feb 24 22:24:50 insley kernel: [35081.562361] CPU: 2 PID: 1659 Comm: kvm Tainted: G             L  4.4.40-1-pve #1
Feb 24 22:24:50 insley kernel: [35081.562362] Hardware name: Gigabyte Technology Co., Ltd. P55A-UD3/P55A-UD3, BIOS F8 04/29/2010
Feb 24 22:24:50 insley kernel: [35081.562364] task: ffff88040a5272c0 ti: ffff88040be7c000 task.ti: ffff88040be7c000
Feb 24 22:24:50 insley kernel: [35081.562365] RIP: 0010:[<ffffffff811044b8>]  [<ffffffff811044b8>] smp_call_function_single+0xd8/0x130
Feb 24 22:24:50 insley kernel: [35081.562371] RSP: 0018:ffff88040be7fa98  EFLAGS: 00000202
Feb 24 22:24:50 insley kernel: [35081.562372] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
Feb 24 22:24:50 insley kernel: [35081.562373] RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000286
Feb 24 22:24:50 insley kernel: [35081.562374] RBP: ffff88040be7fae0 R08: ffffffff81f4e540 R09: 00001b2e79ad2b9d
Feb 24 22:24:50 insley kernel: [35081.562375] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffffc02df720
Feb 24 22:24:50 insley kernel: [35081.562376] R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000001
Feb 24 22:24:50 insley kernel: [35081.562378] FS:  00007fb5b1bff700(0000) GS:ffff88041fc80000(0000) knlGS:0000000000000000
Feb 24 22:24:50 insley kernel: [35081.562379] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 24 22:24:50 insley kernel: [35081.562380] CR2: 000056374d018100 CR3: 00000000db761000 CR4: 00000000000026e0
Feb 24 22:24:50 insley kernel: [35081.562381] Stack:
Feb 24 22:24:50 insley kernel: [35081.562382]  ffff88041fd0f000 ffff8800362a3110 0000000000000000 ffffffffc02df720
Feb 24 22:24:50 insley kernel: [35081.562384]  ffff880407d8c510 0000000000000003 00000000142ec320 ffff880407d88000
Feb 24 22:24:50 insley kernel: [35081.562385]  0000000000000002 ffff88040be7faf0 ffffffffc02dc735 ffff88040be7fc00
Feb 24 22:24:50 insley kernel: [35081.562387] Call Trace:
Feb 24 22:24:50 insley kernel: [35081.562393]  [<ffffffffc02df720>] ? vmx_adjust_tsc_offset_guest+0xb0/0xb0 [kvm_intel]
Feb 24 22:24:50 insley kernel: [35081.562397]  [<ffffffffc02dc735>] loaded_vmcs_clear+0x25/0x30 [kvm_intel]
Feb 24 22:24:50 insley kernel: [35081.562400]  [<ffffffffc02e172d>] vmx_vcpu_load+0x1ed/0x2d0 [kvm_intel]
Feb 24 22:24:50 insley kernel: [35081.562422]  [<ffffffffc01cdddb>] ? start_apic_timer+0x8b/0x1a0 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562424]  [<ffffffff810b7885>] ? update_curr+0x75/0x160
Feb 24 22:24:50 insley kernel: [35081.562426]  [<ffffffff810b8064>] ? dequeue_entity+0x414/0x9f0
Feb 24 22:24:50 insley kernel: [35081.562438]  [<ffffffffc01ad9eb>] kvm_arch_vcpu_load+0x3b/0x220 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562446]  [<ffffffffc0197639>] kvm_sched_in+0x39/0x40 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562449]  [<ffffffff810aa363>] finish_task_switch+0x163/0x240
Feb 24 22:24:50 insley kernel: [35081.562451]  [<ffffffff81857af1>] __schedule+0x371/0xa50
Feb 24 22:24:50 insley kernel: [35081.562464]  [<ffffffffc01d02d8>] ? kvm_apic_has_interrupt+0x28/0xd0 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562466]  [<ffffffff81858205>] schedule+0x35/0x80
Feb 24 22:24:50 insley kernel: [35081.562475]  [<ffffffffc01980c2>] kvm_vcpu_block+0x82/0x2e0 [kvm]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kvm:1659]
Feb 24 22:24:50 insley kernel: [35081.562317] Modules linked in: input_leds ip_set ip6table_filter ip6_tables iptable_filter ip_tables x_tables softdo
g nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scs
i_transport_iscsi nfnetlink_log nfnetlink snd_hda_codec_hdmi nouveau mxm_wmi wmi video ttm arc4 drm_kms_helper rt61pci snd_hda_codec_realtek drm rt2x0
0pci rt2x00mmio snd_hda_codec_generic rt2x00lib i2c_algo_bit mac80211 fb_sys_fops ppdev gpio_ich snd_hda_intel syscopyarea sysfillrect sysimgblt coret
emp snd_hda_codec snd_hda_core cfg80211 kvm_intel snd_hwdep snd_pcm kvm snd_timer eeprom_93cx6 snd crc_itu_t soundcore irqbypass shpchp pcspkr i7core_
edac serio_raw edac_core 8250_fintek parport_pc lpc_ich parport mac_hid vhost_net vhost macvtap macvlan autofs4 ses enclosure i2c_i801 pata_acpi uas p
ata_it8213 usb_storage r8169 ahci mii libahci fjes
Feb 24 22:24:50 insley kernel: [35081.562361] CPU: 2 PID: 1659 Comm: kvm Tainted: G             L  4.4.40-1-pve #1
Feb 24 22:24:50 insley kernel: [35081.562362] Hardware name: Gigabyte Technology Co., Ltd. P55A-UD3/P55A-UD3, BIOS F8 04/29/2010
Feb 24 22:24:50 insley kernel: [35081.562364] task: ffff88040a5272c0 ti: ffff88040be7c000 task.ti: ffff88040be7c000
Feb 24 22:24:50 insley kernel: [35081.562365] RIP: 0010:[<ffffffff811044b8>]  [<ffffffff811044b8>] smp_call_function_single+0xd8/0x130
Feb 24 22:24:50 insley kernel: [35081.562371] RSP: 0018:ffff88040be7fa98  EFLAGS: 00000202
Feb 24 22:24:50 insley kernel: [35081.562372] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
Feb 24 22:24:50 insley kernel: [35081.562373] RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000286
Feb 24 22:24:50 insley kernel: [35081.562374] RBP: ffff88040be7fae0 R08: ffffffff81f4e540 R09: 00001b2e79ad2b9d
Feb 24 22:24:50 insley kernel: [35081.562375] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffffc02df720
Feb 24 22:24:50 insley kernel: [35081.562376] R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000001
Feb 24 22:24:50 insley kernel: [35081.562378] FS:  00007fb5b1bff700(0000) GS:ffff88041fc80000(0000) knlGS:0000000000000000
Feb 24 22:24:50 insley kernel: [35081.562379] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb 24 22:24:50 insley kernel: [35081.562380] CR2: 000056374d018100 CR3: 00000000db761000 CR4: 00000000000026e0
Feb 24 22:24:50 insley kernel: [35081.562381] Stack:
Feb 24 22:24:50 insley kernel: [35081.562382]  ffff88041fd0f000 ffff8800362a3110 0000000000000000 ffffffffc02df720
Feb 24 22:24:50 insley kernel: [35081.562384]  ffff880407d8c510 0000000000000003 00000000142ec320 ffff880407d88000
Feb 24 22:24:50 insley kernel: [35081.562385]  0000000000000002 ffff88040be7faf0 ffffffffc02dc735 ffff88040be7fc00
Feb 24 22:24:50 insley kernel: [35081.562387] Call Trace:
Feb 24 22:24:50 insley kernel: [35081.562393]  [<ffffffffc02df720>] ? vmx_adjust_tsc_offset_guest+0xb0/0xb0 [kvm_intel]
Feb 24 22:24:50 insley kernel: [35081.562397]  [<ffffffffc02dc735>] loaded_vmcs_clear+0x25/0x30 [kvm_intel]
Feb 24 22:24:50 insley kernel: [35081.562400]  [<ffffffffc02e172d>] vmx_vcpu_load+0x1ed/0x2d0 [kvm_intel]
Feb 24 22:24:50 insley kernel: [35081.562422]  [<ffffffffc01cdddb>] ? start_apic_timer+0x8b/0x1a0 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562424]  [<ffffffff810b7885>] ? update_curr+0x75/0x160
Feb 24 22:24:50 insley kernel: [35081.562426]  [<ffffffff810b8064>] ? dequeue_entity+0x414/0x9f0
Feb 24 22:24:50 insley kernel: [35081.562438]  [<ffffffffc01ad9eb>] kvm_arch_vcpu_load+0x3b/0x220 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562446]  [<ffffffffc0197639>] kvm_sched_in+0x39/0x40 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562449]  [<ffffffff810aa363>] finish_task_switch+0x163/0x240
Feb 24 22:24:50 insley kernel: [35081.562451]  [<ffffffff81857af1>] __schedule+0x371/0xa50
Feb 24 22:24:50 insley kernel: [35081.562464]  [<ffffffffc01d02d8>] ? kvm_apic_has_interrupt+0x28/0xd0 [kvm]
Feb 24 22:24:50 insley kernel: [35081.562466]  [<ffffffff81858205>] schedule+0x35/0x80
Feb 24 22:24:50 insley kernel: [35081.562475]  [<ffffffffc01980c2>] kvm_vcpu_block+0x82/0x2e0 [kvm]
 
i would urge you to upgrade, 4.4 is EOL since some time
 
yes but it may be a simple kernel bug so i would at least upgrade to the latest 4.4 packages and reboot
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom