We've been using Proxmox now for a month I think, never had a problem.
Today we head a soft lockup, and no more responses from the server.
This is the error:
We are running:
Everything is up to date, updating to the latest kernel now, any suggestions?
Today we head a soft lockup, and no more responses from the server.
This is the error:
Code:
Aug 24 10:51:37 beast kernel: BUG: soft lockup - CPU#0 stuck for 67s! [events/0:323]Aug 24 10:51:37 beast kernel: Modules linked in: mptctl mptbase vzethdev vznetdev simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nfs lockd fscache nfs_acl au
th_rpcgss sunrpc nf_conntrack vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length xt_hl xt_tcpmss xt_TCPMSS vhost_net macvtap macvlan
tun iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp kvm_intel ipt_REJECT kvm ip_tables dlm configfs vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core
ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse ipv6 radeon snd_pcsp snd_pcm ttm drm_kms_helper snd_timer drm snd tpm_tis soundcore serio_raw i7core_
edac i2c_algo_bit tpm shpchp tpm_bios i2c_core edac_core hpilo hpwdt snd_page_alloc power_meter ext3 jbd mbcache sg ata_generic pata_acpi ata_piix igb dca hpsa netxen_n
ic [last unloaded: scsi_wait_scan]
Aug 24 10:51:37 beast kernel: CPU 0
Aug 24 10:51:37 beast kernel: Modules linked in: mptctl mptbase vzethdev vznetdev simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nfs lockd fscache nfs_acl au
th_rpcgss sunrpc nf_conntrack vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length xt_hl xt_tcpmss xt_TCPMSS vhost_net macvtap macvlan
tun iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp kvm_intel ipt_REJECT kvm ip_tables dlm configfs vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core
ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse ipv6 radeon snd_pcsp snd_pcm ttm drm_kms_helper snd_timer drm snd tpm_tis soundcore serio_raw i7core_
edac i2c_algo_bit tpm shpchp tpm_bios i2c_core edac_core hpilo hpwdt snd_page_alloc power_meter ext3 jbd mbcache sg ata_generic pata_acpi ata_piix igb dca hpsa netxen_n
ic [last unloaded: scsi_wait_scan]
Aug 24 10:51:37 beast kernel:
Aug 24 10:51:37 beast kernel: Pid: 323, comm: events/0 veid: 0 Tainted: G W --------------- 2.6.32-14-pve #1 042stab059_7 HP ProLiant DL980 G7
Aug 24 10:51:37 beast kernel: RIP: 0010:[<ffffffff815280c7>] [<ffffffff815280c7>] _spin_unlock_irqrestore+0x17/0x20
Aug 24 10:51:37 beast kernel: RSP: 0000:ffff880069803ba0 EFLAGS: 00000246
Aug 24 10:51:37 beast kernel: RAX: ffff880fe4bc08c0 RBX: ffff880069803ba0 RCX: 000000000000037a
Aug 24 10:51:37 beast kernel: RDX: ffff884ff809e8e0 RSI: 0000000000000246 RDI: 0000000000000246
Aug 24 10:51:37 beast kernel: RBP: ffffffff8100bcd3 R08: 0000000000000000 R09: 000000000000005d
Aug 24 10:51:37 beast kernel: R10: 0000000000000004 R11: 000000000000003c R12: ffff880069803b20
Aug 24 10:51:37 beast kernel: R13: ffff884ff809e8e0 R14: ffff880fb5039fc0 R15: ffffffff8152dbbb
Aug 24 10:51:37 beast kernel: FS: 0000000000000000(0000) GS:ffff880069800000(0000) knlGS:0000000000000000
Aug 24 10:51:37 beast kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 24 10:51:37 beast kernel: CR2: 00007f516c32fa98 CR3: 0000002f9de6f000 CR4: 00000000000026e0
Aug 24 10:51:37 beast kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 24 10:51:37 beast kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 24 10:51:37 beast kernel: Process events/0 (pid: 323, veid: 0, threadinfo ffff881078c06000, task ffff881078c04f00)
Aug 24 10:51:37 beast kernel: Stack:
Aug 24 10:51:37 beast kernel: ffff880069803c10 ffffffff812a70c2 ffff880fb5039fc0 000000000000037a
Aug 24 10:51:37 beast kernel: <d> 0000000000000246 0000000000000000 ffff881076b0a440 ffff880fb23477c0
Aug 24 10:51:37 beast kernel: <d> ffff8810742c2400 0000fffffffff000 0000000000000001 ffff884ff809e8e0
Aug 24 10:51:37 beast kernel: Call Trace:
Aug 24 10:51:37 beast kernel: <IRQ>
Aug 24 10:51:37 beast kernel: [<ffffffff812a70c2>] ? alloc_iova+0x1b2/0x240
Aug 24 10:51:37 beast kernel: [<ffffffff812a8d45>] ? intel_alloc_iova+0xb5/0xe0
Aug 24 10:51:37 beast kernel: [<ffffffff812abb8f>] ? __intel_map_single+0xbf/0x210
Aug 24 10:51:37 beast kernel: [<ffffffff812abd13>] ? intel_map_page+0x33/0x40
Aug 24 10:51:37 beast kernel: [<ffffffffa0045b22>] ? igb_alloc_rx_buffers+0x162/0x350 [igb]
Aug 24 10:51:37 beast kernel: [<ffffffff812abce0>] ? intel_map_page+0x0/0x40
Aug 24 10:51:37 beast kernel: [<ffffffffa0046334>] ? igb_poll+0x624/0x11c0 [igb]
Aug 24 10:51:37 beast kernel: [<ffffffff81081374>] ? mod_timer+0x144/0x230
Aug 24 10:51:37 beast kernel: [<ffffffff813bb4f0>] ? usb_hcd_poll_rh_status+0x140/0x180
Aug 24 10:51:37 beast kernel: [<ffffffff8107fc33>] ? run_timer_softirq+0x1f3/0x380
Aug 24 10:51:37 beast kernel: [<ffffffff8145f2d3>] ? net_rx_action+0x103/0x2e0
Aug 24 10:51:37 beast kernel: [<ffffffff813bbd10>] ? rh_timer_func+0x0/0x10
Aug 24 10:51:37 beast kernel: [<ffffffff81075363>] ? __do_softirq+0x103/0x260
Aug 24 10:51:37 beast kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 24 10:51:37 beast kernel: [<ffffffff8100df35>] ? do_softirq+0x65/0xa0
Aug 24 10:51:37 beast kernel: [<ffffffff8107518d>] ? irq_exit+0xcd/0xd0
Aug 24 10:51:37 beast kernel: [<ffffffff8152dad5>] ? do_IRQ+0x75/0xf0
Aug 24 10:51:37 beast kernel: [<ffffffff8100bb13>] ? ret_from_intr+0x0/0x11
Aug 24 10:51:37 beast kernel: <EOI>
Aug 24 10:51:37 beast kernel: [<ffffffff815269f4>] ? mutex_trylock+0x14/0x40
Aug 24 10:51:37 beast kernel: [<ffffffff8118292f>] ? cache_reap+0x2f/0x2e0
Aug 24 10:51:37 beast kernel: [<ffffffffa0043df2>] ? igb_watchdog_task+0x1d2/0x600 [igb]
Aug 24 10:51:37 beast kernel: [<ffffffff8114e440>] ? vmstat_update+0x0/0x40
Aug 24 10:51:37 beast kernel: [<ffffffff81182900>] ? cache_reap+0x0/0x2e0
Aug 24 10:51:37 beast kernel: [<ffffffff8108f3f0>] ? worker_thread+0x190/0x2d0
Aug 24 10:51:37 beast kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Aug 24 10:51:37 beast kernel: [<ffffffff8108f260>] ? worker_thread+0x0/0x2d0
Aug 24 10:51:37 beast kernel: [<ffffffff81095606>] ? kthread+0x96/0xa0
Aug 24 10:51:37 beast kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 24 10:51:37 beast kernel: [<ffffffff81095570>] ? kthread+0x0/0xa0
Aug 24 10:51:37 beast kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Aug 24 10:51:37 beast kernel: Code: 00 00 00 01 74 05 e8 f9 43 d5 ff c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 48 89 fa 66 ff 02 66 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 c9 c3 66 90 55 48 89 e5 0f 1f 44 00 00 f0 ff 07
Aug 24 10:51:37 beast kernel: Call Trace:
Aug 24 10:51:37 beast kernel: <IRQ> [<ffffffff812a70c2>] ? alloc_iova+0x1b2/0x240
Aug 24 10:51:37 beast kernel: [<ffffffff812a8d45>] ? intel_alloc_iova+0xb5/0xe0
Aug 24 10:51:37 beast kernel: [<ffffffff812abb8f>] ? __intel_map_single+0xbf/0x210
Aug 24 10:51:37 beast kernel: [<ffffffff812abd13>] ? intel_map_page+0x33/0x40
Aug 24 10:51:37 beast kernel: [<ffffffffa0045b22>] ? igb_alloc_rx_buffers+0x162/0x350 [igb]
Aug 24 10:51:37 beast kernel: [<ffffffff812abce0>] ? intel_map_page+0x0/0x40
Aug 24 10:51:37 beast kernel: [<ffffffffa0046334>] ? igb_poll+0x624/0x11c0 [igb]
Aug 24 10:51:37 beast kernel: [<ffffffff81081374>] ? mod_timer+0x144/0x230
Aug 24 10:51:37 beast kernel: [<ffffffff813bb4f0>] ? usb_hcd_poll_rh_status+0x140/0x180
Aug 24 10:51:37 beast kernel: [<ffffffff8107fc33>] ? run_timer_softirq+0x1f3/0x380
Aug 24 10:51:37 beast kernel: [<ffffffff8145f2d3>] ? net_rx_action+0x103/0x2e0
Aug 24 10:51:37 beast kernel: [<ffffffff813bbd10>] ? rh_timer_func+0x0/0x10
Aug 24 10:51:37 beast kernel: [<ffffffff81075363>] ? __do_softirq+0x103/0x260
Aug 24 10:51:37 beast kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Aug 24 10:51:37 beast kernel: [<ffffffff8100df35>] ? do_softirq+0x65/0xa0
Aug 24 10:51:37 beast kernel: [<ffffffff8107518d>] ? irq_exit+0xcd/0xd0
Aug 24 10:51:37 beast kernel: [<ffffffff8152dad5>] ? do_IRQ+0x75/0xf0
Aug 24 10:51:37 beast kernel: [<ffffffff8100bb13>] ? ret_from_intr+0x0/0x11
Aug 24 10:51:37 beast kernel: <EOI> [<ffffffff815269f4>] ? mutex_trylock+0x14/0x40
Aug 24 10:51:37 beast kernel: [<ffffffff8118292f>] ? cache_reap+0x2f/0x2e0
Aug 24 10:51:37 beast kernel: [<ffffffffa0043df2>] ? igb_watchdog_task+0x1d2/0x600 [igb]
Aug 24 10:51:37 beast kernel: [<ffffffff8114e440>] ? vmstat_update+0x0/0x40
Aug 24 10:51:37 beast kernel: [<ffffffff81182900>] ? cache_reap+0x0/0x2e0
Aug 24 10:51:37 beast kernel: [<ffffffff8108f3f0>] ? worker_thread+0x190/0x2d0
Aug 24 10:51:37 beast kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Aug 24 10:51:37 beast kernel: [<ffffffff8108f260>] ? worker_thread+0x0/0x2d0
Aug 24 10:51:37 beast kernel: [<ffffffff81095606>] ? kthread+0x96/0xa0
Aug 24 10:51:37 beast kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Aug 24 10:51:37 beast kernel: [<ffffffff81095570>] ? kthread+0x0/0xa0
Aug 24 10:51:37 beast kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20We are running:
Code:
pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-73
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-13-pve: 2.6.32-72
pve-kernel-2.6.32-14-pve: 2.6.32-73
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-48
pve-firmware: 1.0-18
libpve-common-perl: 1.0-30
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-30
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-7
ksm-control-daemon: 1.1-1
Code:
00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 22)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:0a.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 22)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
01:03.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI ES1000 (rev 02)
02:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 04)
02:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 04)
02:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 01)
03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
04:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
04:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
04:00.2 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
04:00.3 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
05:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
05:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
05:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
Code:
processor : 79
vendor_id : GenuineIntel
cpu family : 6
model : 47
model name : Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz
stepping : 2
cpu MHz : 2394.026
cache size : 30720 KB
physical id : 3
siblings : 20
core id : 25
cpu cores : 10
apicid : 243
initial apicid : 243
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 x2apic popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 4787.99
clflush size : 64
cache_alignment : 64
address sizes : 44 bits physical, 48 bits virtual
power management:Everything is up to date, updating to the latest kernel now, any suggestions?