S
siddharth007
Guest
We have an IBM x3650 server blade where we are using the Proxmox virtual OS.We have done clustering using another IBM blade.The kernel version is 2.6.32-14-pve.Everything was running smooth until recently when our system suddenly halted causing the virtual containers to go down.We restarted the blade and everything was back to normal.However,we could not find a reason why this happened.The system generated no logs for the events which caused our system to go down.The system went down on Feb 18 15:56 IST.However the Kernel logs(kern.log,please check attachments) has the last entry on Feb 18 03:12:22 IST. Further logs were only generated after we powered off and powered on our system remotely through ipmitool.
I am attaching the kern.log file.Please help me out with this.
I am attaching the kern.log file.Please help me out with this.
Code:
[/COLOR][/SIZE][/FONT]
Feb 18 00:13:49 clust-sec1 kernel: WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G W --------------- )
Feb 18 00:13:49 clust-sec1 kernel: Hardware name: IBM System x3650 -[0007979]-
Feb 18 00:13:49 clust-sec1 kernel: list_add corruption. next->prev should be prev (ffff88023b48fc38), but was 7672655320726f20. (next=ffff880095308ae0).
Feb 18 00:13:49 clust-sec1 kernel: Modules linked in: vzethdev vznetdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_owner xt_mac ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss vhost_net xt_TCPMSS macvtap ipt_REJECT xt_DSCP xt_dscp xt_multiport macvlan tun xt_limit kvm_intel iptable_mangle kvm iptable_filter ip_tables dlm configfs vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi radeon ttm drm_kms_helper drm snd_pcsp ibmpex i2c_algo_bit i5000_edac snd_pcm ibmaem snd_timer ipmi_msghandler snd soundcore ics932s401 edac_core i2c_i801 ioatdma tpm_tis i5k_amb i2c_core snd_page_alloc tpm tpm_bios dca serio_raw shpchp ext4 mbcache jbd2 sg ses enclosure ata_generic pata_acpi ata_piix bnx2 aacraid [las
Feb 18 00:13:49 clust-sec1 kernel: t unloaded: configfs]
Feb 18 00:13:49 clust-sec1 kernel: Pid: 870426, comm: clamscan veid: 101 Tainted: G W --------------- 2.6.32-14-pve #1
Feb 18 00:13:49 clust-sec1 kernel: Call Trace:
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff8106c608>] ? warn_slowpath_common+0x88/0xc0
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff8106c6f6>] ? warn_slowpath_fmt+0x46/0x50
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff812816cd>] ? __list_add+0x6d/0xa0
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffffa072a4cc>] ? nfs_dq_prealloc_space+0x23c/0x2f0 [nfs]
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffffa07153aa>] ? nfs_file_write+0xba/0x210 [nfs]
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff8119489a>] ? do_sync_write+0xfa/0x140
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff8100984c>] ? __switch_to+0x1ac/0x320
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff81194b78>] ? vfs_write+0xb8/0x1a0
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff81195591>] ? sys_write+0x51/0x90
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff815288ae>] ? do_device_not_available+0xe/0x10
Feb 18 00:13:49 clust-sec1 kernel: [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Feb 18 00:13:49 clust-sec1 kernel: ---[ end trace 9389904522a4dd41 ]---
Feb 18 00:29:10 clust-sec1 kernel: ------------[ cut here ]------------
Feb 18 00:29:10 clust-sec1 kernel: WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G W --------------- )
Feb 18 00:29:10 clust-sec1 kernel: Hardware name: IBM System x3650 -[0007979]-
Feb 18 00:29:10 clust-sec1 kernel: list_add corruption. next->prev should be prev (ffff88023b48fc38), but was b4ce58e5e426e95b. (next=ffff880095308ae0).
Feb 18 00:29:10 clust-sec1 kernel: Modules linked in: vzethdev vznetdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_owner xt_mac ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss vhost_net xt_TCPMSS macvtap ipt_REJECT xt_DSCP xt_dscp xt_multiport macvlan tun xt_limit kvm_intel iptable_mangle kvm iptable_filter ip_tables dlm configfs vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi radeon ttm drm_kms_helper drm snd_pcsp ibmpex i2c_algo_bit i5000_edac snd_pcm ibmaem snd_timer ipmi_msghandler snd soundcore ics932s401 edac_core i2c_i801 ioatdma tpm_tis i5k_amb i2c_core snd_page_alloc tpm tpm_bios dca serio_raw shpchp ext4 mbcache jbd2 sg ses enclosure ata_generic pata_acpi ata_piix bnx2 aacraid [las
Feb 18 00:29:10 clust-sec1 kernel: t unloaded: configfs]
Feb 18 00:29:10 clust-sec1 kernel: Pid: 870426, comm: clamscan veid: 101 Tainted: G W --------------- 2.6.32-14-pve #1
Feb 18 00:29:10 clust-sec1 kernel: Call Trace:
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff8106c608>] ? warn_slowpath_common+0x88/0xc0
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff8106c6f6>] ? warn_slowpath_fmt+0x46/0x50
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff812816cd>] ? __list_add+0x6d/0xa0
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffffa072a4cc>] ? nfs_dq_prealloc_space+0x23c/0x2f0 [nfs]
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffffa07153aa>] ? nfs_file_write+0xba/0x210 [nfs]
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff8119489a>] ? do_sync_write+0xfa/0x140
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff8126ea51>] ? cpumask_any_but+0x31/0x50
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff81194b78>] ? vfs_write+0xb8/0x1a0
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff81195591>] ? sys_write+0x51/0x90
Feb 18 00:29:10 clust-sec1 kernel: [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Feb 18 00:29:10 clust-sec1 kernel: ---[ end trace 9389904522a4dd42 ]---
Feb 18 00:44:20 clust-sec1 kernel: ------------[ cut here ]------------
Feb 18 00:44:20 clust-sec1 kernel: WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G W --------------- )
Feb 18 00:44:20 clust-sec1 kernel: Hardware name: IBM System x3650 -[0007979]-
Feb 18 00:44:20 clust-sec1 kernel: list_add corruption. next->prev should be prev (ffff88023b48fc38), but was 00000000034b5aa0. (next=ffff880095308ae0).
Feb 18 00:44:20 clust-sec1 kernel: Modules linked in: vzethdev vznetdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_owner xt_mac ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss vhost_net xt_TCPMSS macvtap ipt_REJECT xt_DSCP xt_dscp xt_multiport macvlan tun xt_limit kvm_intel iptable_mangle kvm iptable_filter ip_tables dlm configfs vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi radeon ttm drm_kms_helper drm snd_pcsp ibmpex i2c_algo_bit i5000_edac snd_pcm ibmaem snd_timer ipmi_msghandler snd soundcore ics932s401 edac_core i2c_i801 ioatdma tpm_tis i5k_amb i2c_core snd_page_alloc tpm tpm_bios dca serio_raw shpchp ext4 mbcache jbd2 sg ses enclosure ata_generic pata_acpi ata_piix bnx2 aacraid [las
Feb 18 00:44:20 clust-sec1 kernel: t unloaded: configfs]
Feb 18 00:44:20 clust-sec1 kernel: Pid: 870426, comm: clamscan veid: 101 Tainted: G W --------------- 2.6.32-14-pve #1
Feb 18 00:44:20 clust-sec1 kernel: Call Trace:
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff8106c608>] ? warn_slowpath_common+0x88/0xc0
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff8106c6f6>] ? warn_slowpath_fmt+0x46/0x50
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff812816cd>] ? __list_add+0x6d/0xa0
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffffa072a4cc>] ? nfs_dq_prealloc_space+0x23c/0x2f0 [nfs]
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffffa07153aa>] ? nfs_file_write+0xba/0x210 [nfs]
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff8119489a>] ? do_sync_write+0xfa/0x140
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff8126ea51>] ? cpumask_any_but+0x31/0x50
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff81194b78>] ? vfs_write+0xb8/0x1a0
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff81195591>] ? sys_write+0x51/0x90
Feb 18 00:44:20 clust-sec1 kernel: [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Feb 18 00:44:20 clust-sec1 kernel: ---[ end trace 9389904522a4dd43 ]---
Feb 18 00:44:29 clust-sec1 kernel: ------------[ cut here ]------------
Feb 18 00:44:29 clust-sec1 kernel: WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G W --------------- )
Feb 18 00:44:29 clust-sec1 kernel: Hardware name: IBM System x3650 -[0007979]-
Feb 18 00:44:29 clust-sec1 kernel: list_add corruption. next->prev should be prev (ffff88023b48fc38), but was 00007f5fd6f5cc60. (next=ffff880095308ae0).
Feb 18 00:44:29 clust-sec1 kernel: Modules linked in: vzethdev vznetdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_owner xt_mac ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss vhost_net xt_TCPMSS macvtap ipt_REJECT xt_DSCP xt_dscp xt_multiport macvlan tun xt_limit kvm_intel iptable_mangle kvm iptable_filter ip_tables dlm configfs vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi radeon ttm drm_kms_helper drm snd_pcsp ibmpex i2c_algo_bit i5000_edac snd_pcm ibmaem snd_timer ipmi_msghandler snd soundcore ics932s401 edac_core i2c_i801 ioatdma tpm_tis i5k_amb i2c_core snd_page_alloc tpm tpm_bios dca serio_raw shpchp ext4 mbcache jbd2 sg ses enclosure ata_generic pata_acpi ata_piix bnx2 aacraid [las
Feb 18 00:44:29 clust-sec1 kernel: t unloaded: configfs]
Feb 18 00:44:29 clust-sec1 kernel: Pid: 870426, comm: clamscan veid: 101 Tainted: G W --------------- 2.6.32-14-pve #1
Feb 18 00:44:29 clust-sec1 kernel: Call Trace:
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff8106c608>] ? warn_slowpath_common+0x88/0xc0
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff8106c6f6>] ? warn_slowpath_fmt+0x46/0x50
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff812816cd>] ? __list_add+0x6d/0xa0
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffffa072a4cc>] ? nfs_dq_prealloc_space+0x23c/0x2f0 [nfs]
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffffa07153aa>] ? nfs_file_write+0xba/0x210 [nfs]
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff8119489a>] ? do_sync_write+0xfa/0x140
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff81095be0>] ? autoremove_wake_function+0x0/0x40
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff8126ea51>] ? cpumask_any_but+0x31/0x50
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff81194b78>] ? vfs_write+0xb8/0x1a0
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff81195591>] ? sys_write+0x51/0x90
Feb 18 00:44:29 clust-sec1 kernel: [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Feb 18 00:44:29 clust-sec1 kernel: ---[ end trace 9389904522a4dd44 ]---
Feb 18 00:44:49 clust-sec1 kernel: CT: 105: stopped
Feb 18 00:44:51 clust-sec1 kernel: CT: 105: started
Feb 18 01:38:00 clust-sec1 kernel: CPT ERR: ffff88003779b000,102 :foreign process 24716/953295(bash) inside CT (e.g. vzctl enter or vzctl exec).
Feb 18 01:38:00 clust-sec1 kernel: CPT ERR: ffff88003779b000,102 :suspend is impossible now.
Feb 18 03:12:22 clust-sec1 kernel: Holy Crap 1 0 910222,551(apache2)
---- This portion of the log is after our system was restarted -------
Feb 18 17:36:37 clust-sec1 kernel: imklog 4.6.4, log source = /proc/kmsg started.
Feb 18 17:36:37 clust-sec1 kernel: Initializing cgroup subsys cpuset
Feb 18 17:36:37 clust-sec1 kernel: Initializing cgroup subsys cpu
Feb 18 17:36:37 clust-sec1 kernel: Linux version 2.6.32-14-pve (root@maui) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Tue Aug 21 08:24:37 CEST 2012
Feb 18 17:36:37 clust-sec1 kernel: Command line: BOOT_IMAGE=/vmlinuz-2.6.32-14-pve root=/dev/mapper/pve-root ro quiet
Feb 18 17:36:37 clust-sec1 kernel: KERNEL supported cpus:
Feb 18 17:36:37 clust-sec1 kernel: Intel GenuineIntel
Feb 18 17:36:37 clust-sec1 kernel: AMD AuthenticAMD
Feb 18 17:36:37 clust-sec1 kernel: Centaur CentaurHauls
Feb 18 17:36:37 clust-sec1 kernel: BIOS-provided physical RAM map:
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 0000000000000000 - 000000000009ac00 (usable)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 000000000009ac00 - 00000000000a0000 (reserved)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 0000000000100000 - 00000000bffc74c0 (usable)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 00000000bffc74c0 - 00000000bffceac0 (ACPI data)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 00000000bffceac0 - 00000000c0000000 (reserved)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
Feb 18 17:36:37 clust-sec1 kernel: BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
Feb 18 17:36:37 clust-sec1 kernel: DMI 2.4 present.
.
.
.
.
.
.
.
.
.
--- Log truncated ----
[FONT=Tahoma][SIZE=1][COLOR=black]