Constant hanging Proxmox VE 2.3

[103]

Recently I'm having issues with a machine that never had any problems at all, the system hangs showing the log bellow. Right now there are no containers nor KVMs. I reinstalled/updated Proxmox VE about three times but the problem is constant. I post my pveversion and the syslog fragment. Best regards to all.

pveversion -v:

pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-96
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-18-pve: 2.6.32-88
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-20
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-7
vncterm: 1.0-4
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-10
ksm-control-daemon: 1.1-1


/var/log/syslog fragment:

May 17 01:46:22 proxmox kernel: WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G W --------------- )
May 17 01:46:22 proxmox kernel: Hardware name: System Product Name
May 17 01:46:22 proxmox kernel: list_add corruption. next->prev should be prev (ffff880078addca0), but was ffff8800793b7cf8. (next=ffff8800793b7c50).
May 17 01:46:22 proxmox kernel: Modules linked in: xt_mark vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nf_conntrack vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables vhost_net macvtap macvlan tun kvm_intel kvm vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi nfs lockd fscache nfs_acl auth_rpcgss sunrpc i915 drm_kms_helper shpchp tpm_tis tpm drm snd_pcsp i2c_algo_bit i2c_i801 snd_pcm snd_timer i2c_core video tpm_bios snd soundcore snd_page_alloc output ext3 jbd mbcache ata_generic pata_acpi 8139too 8139cp r8169 mii ata_piix [last unloaded: scsi_wait_scan]
May 17 01:46:22 proxmox kernel: Pid: 1305, comm: ntpd veid: 0 Tainted: G W --------------- 2.6.32-18-pve #1
May 17 01:46:22 proxmox kernel: Call Trace:
May 17 01:46:22 proxmox kernel: [<ffffffff8106d228>] ? warn_slowpath_common+0x88/0xc0
May 17 01:46:22 proxmox kernel: [<ffffffff8106d316>] ? warn_slowpath_fmt+0x46/0x50
May 17 01:46:22 proxmox kernel: [<ffffffff8106d235>] ? warn_slowpath_common+0x95/0xc0
May 17 01:46:22 proxmox kernel: [<ffffffff8128292d>] ? __list_add+0x6d/0xa0
May 17 01:46:22 proxmox kernel: [<ffffffff81096f1b>] ? add_wait_queue+0x3b/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad875>] ? __pollwait+0x75/0xf0
May 17 01:46:22 proxmox kernel: [<ffffffff8113650d>] ? free_hot_page+0x2d/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff8144d4b2>] ? datagram_poll+0x152/0x180
May 17 01:46:22 proxmox kernel: [<ffffffff814b6204>] ? udp_poll+0x24/0x70
May 17 01:46:22 proxmox kernel: [<ffffffff8143ca6a>] ? sock_poll+0x1a/0x20
May 17 01:46:22 proxmox kernel: [<ffffffff811ae5f2>] ? do_select+0x3a2/0x6d0
May 17 01:46:22 proxmox kernel: [<ffffffff81491660>] ? ip_finish_output+0x0/0x2e0
May 17 01:46:22 proxmox kernel: [<ffffffff811ad800>] ? __pollwait+0x0/0xf0
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff811ad8f0>] ? pollwake+0x0/0x60
May 17 01:46:22 proxmox kernel: [<ffffffff8109613b>] ? do_schedule_next_timer+0x4b/0xe0
May 17 01:46:22 proxmox kernel: [<ffffffff811aeaaa>] ? core_sys_select+0x18a/0x2d0
May 17 01:46:22 proxmox kernel: [<ffffffff81010b2e>] ? copy_user_generic+0xe/0x20
May 17 01:46:22 proxmox kernel: [<ffffffff81015dbd>] ? save_i387_xstate+0x15d/0x220
May 17 01:46:22 proxmox kernel: [<ffffffff8100a3dc>] ? do_signal+0x18c/0x800
May 17 01:46:22 proxmox kernel: [<ffffffff81010b2e>] ? copy_user_generic+0xe/0x20
May 17 01:46:22 proxmox kernel: [<ffffffff81015a4b>] ? check_for_xstate+0x3b/0x90
May 17 01:46:22 proxmox kernel: [<ffffffff81015bff>] ? restore_i387_xstate+0x15f/0x1c0
May 17 01:46:22 proxmox kernel: [<ffffffff811aee47>] ? sys_select+0x47/0x110
May 17 01:46:22 proxmox kernel: [<ffffffff8100b102>] ? system_call_fastpath+0x16/0x1b
May 17 01:46:22 proxmox kernel: ---[ end trace 007ea75c9ea15f0e ]---

 

[udo]

Recently I'm having issues with a machine that never had any problems at all, the system hangs showing the log bellow. Right now there are no containers nor KVMs. I reinstalled/updated Proxmox VE about three times but the problem is constant. I post my pveversion and the syslog fragment. Best regards to all.

I guess that the issue is an hardware-problem. Normaly pve are quite stable.

pveversion -v:

pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-96
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-18-pve: 2.6.32-88
...
May 17 01:46:22 proxmox kernel: Pid: 1305, comm: ntpd veid: 0 Tainted: G W --------------- 2.6.32-18-pve #1

strange,
running kernel is .32-19 but your trace show .32-18?!?

Do you have allways an entry in syslog/messages if the system hang?
How the issue resolved? Do you must powercycle the host, or is the console responsible?

Udo

 

[103]

Do you have allways an entry in syslog/messages if the system hang?

Yes I do. The syslog file grows with the same lines just changing the address of the trace.

How the issue resolved? Do you must powercycle the host, or is the console responsible?

It is temporarily resolved by rebooting the system, the console is unresponsible.
I'm also getting this message: 'AMI BIOS detected: BIOS may corrupt low RAM, working around it.'
So I'm suspecting the issue is a faulty RAM, but it is strange, how come the system works for a certain period of time and then hangs?

Thank you, best regards

 

[103]

I guess that the issue is an hardware-problem. Normaly pve are quite stable.

Yes you were right, CPU issues, thanks anyway for the suggestions.

Best regards