Hi,
I have a Supermicro server running Proxmox several months really well since yesterday, the entire server hung (kernel panic) 3 times in the same day and I have to reboot it.
The server is running 10 containers and nothing has changed (I think)
This is what I see in syslog:
Any help would be appreciated
I have a Supermicro server running Proxmox several months really well since yesterday, the entire server hung (kernel panic) 3 times in the same day and I have to reboot it.
The server is running 10 containers and nothing has changed (I think)
Code:
pveversion -v
pve-manager: 2.3-13(pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-16-pve
proxmox-ve-2.6.32: 2.2-80
pve-kernel-2.6.32-16-pve: 2.6.32-82
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-20
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.0-36
vncterm: 1.0-4
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-10
ksm-control-daemon: 1.1-1
Code:
--
Jul 16 15:56:48 srv1 kernel:------------[ cut here ]------------
Jul 16 15:56:48 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 16 15:56:48 srv1 kernel: Hardwarename: X9DRW
Jul 16 15:56:48 srv1 kernel: list_delcorruption. prev->next should be ffff88034e2d5dc0, but wasffff88086eea30b0
Jul 16 15:56:48 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 15:56:48 srv1 kernel: Pid:108900, comm: httpd veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 15:56:48 srv1 kernel: CallTrace:
Jul 16 15:56:48 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 15:56:48 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 15:56:48 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 15:56:48 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 16 15:56:48 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 15:56:48 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 15:56:48 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 15:56:48 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 15:56:48 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 15:56:48 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 15:56:48 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 16 15:56:48 srv1 kernel: ---[ endtrace 1bbb5cda1f7c0281 ]---
Jul 16 15:56:50 srv1 snmpd[2037]:Connection from UDP: [192.168.0.8]:44494->[192.168.0.10]
Jul 16 15:56:54 srv1 snmpd[2037]:Connection from UDP: [192.168.0.8]:46413->[192.168.0.10]
--
Jul 16 16:14:15 srv1 kernel:------------[ cut here ]------------
Jul 16 16:14:15 srv1 kernel: WARNING:at lib/list_debug.c:51 list_del+0x8d/0xa0() (Tainted: G W --------------- )
Jul 16 16:14:15 srv1 kernel: Hardwarename: X9DRW
Jul 16 16:14:15 srv1 kernel: list_delcorruption. next->prev should be ffff8803f12f3880, but wasffff880861d03470
Jul 16 16:14:15 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 16:14:15 srv1 kernel: Pid:108788, comm: httpd veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 16:14:15 srv1 kernel: CallTrace:
Jul 16 16:14:15 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 16:14:15 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 16:14:15 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 16:14:15 srv1 kernel:[<ffffffff8128301d>] ? list_del+0x8d/0xa0
Jul 16 16:14:15 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 16:14:15 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 16:14:15 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 16:14:15 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 16:14:15 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 16:14:15 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 16:14:15 srv1 kernel:[<ffffffff81193635>] ? sys_close+0xa5/0x100
Jul 16 16:14:15 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 16 16:14:15 srv1 kernel: ---[ endtrace 1bbb5cda1f7c0282 ]---
Jul 16 16:14:19 srv1 snmpd[2037]:Connection from UDP: [192.168.0.8]:55878->[192.168.0.10]
--
Jul 16 16:22:22 srv1 kernel:------------[ cut here ]------------
Jul 16 16:22:22 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 16 16:22:22 srv1 kernel: Hardwarename: X9DRW
Jul 16 16:22:22 srv1 kernel: list_delcorruption. prev->next should be ffff88086eea30b0, but wasffff88034e42edb0
Jul 16 16:22:22 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 16:22:22 srv1 kernel: Pid:638942, comm: httpd veid: 4288424 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 16:22:22 srv1 kernel: CallTrace:
Jul 16 16:22:22 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 16:22:22 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 16:22:22 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 16:22:22 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 16 16:22:22 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 16:22:22 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 16:22:22 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 16:22:22 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 16:22:22 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 16:22:22 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 16:22:22 srv1 kernel:[<ffffffff810cc657>] ? compat_sys_wait4+0x97/0xe0
Jul 16 16:22:22 srv1 kernel:[<ffffffff811e6fbc>] ? poll_select_copy_remaining+0xcc/0x120
Jul 16 16:22:22 srv1 kernel:[<ffffffff811e9d79>] ? compat_sys_select+0x59/0x110
Jul 16 16:22:22 srv1 kernel:[<ffffffff8104b990>] ? sys32_waitpid+0x10/0x20
--
Jul 16 16:39:19 srv1 kernel:------------[ cut here ]------------
Jul 16 16:39:19 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 16 16:39:19 srv1 kernel: Hardwarename: X9DRW
Jul 16 16:39:19 srv1 kernel: list_delcorruption. prev->next should be ffff88034e42edb0, but was (null)
Jul 16 16:39:19 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 16:39:19 srv1 kernel: Pid:108202, comm: couriertcpd veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 16:39:19 srv1 kernel: CallTrace:
Jul 16 16:39:19 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 16:39:19 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 16:39:19 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 16:39:19 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 16 16:39:19 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 16:39:19 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 16:39:19 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 16:39:19 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 16:39:19 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 16:39:19 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 16:39:19 srv1 kernel:[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
Jul 16 16:39:19 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 16 16:39:19 srv1 kernel: ---[ endtrace 1bbb5cda1f7c0284 ]---
Jul 16 16:39:21 srv1 snmpd[2037]:Connection from UDP: [192.168.0.8]:54277->[192.168.0.10]
--
Jul 16 17:35:12 srv1 kernel:------------[ cut here ]------------
Jul 16 17:35:12 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 16 17:35:12 srv1 kernel: Hardwarename: X9DRW
Jul 16 17:35:12 srv1 kernel: list_delcorruption. prev->next should be ffff88052805cdb0, but was306e644e39623157
Jul 16 17:35:12 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 17:35:12 srv1 kernel: Pid:108202, comm: couriertcpd veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 17:35:12 srv1 kernel: CallTrace:
Jul 16 17:35:12 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 17:35:12 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 17:35:12 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 17:35:12 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 16 17:35:12 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 17:35:12 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 17:35:12 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 17:35:12 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 17:35:12 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 17:35:12 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 17:35:12 srv1 kernel:[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
Jul 16 17:35:12 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 16 17:35:12 srv1 kernel: ---[ endtrace 1bbb5cda1f7c0285 ]---
Jul 16 17:35:13 srv1 snmpd[2037]:Connection from UDP: [192.168.0.8]:60576->[192.168.0.10]
--
Jul 16 18:12:25 srv1 kernel:------------[ cut here ]------------
Jul 16 18:12:25 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 16 18:12:25 srv1 kernel: Hardwarename: X9DRW
Jul 16 18:12:25 srv1 kernel: list_delcorruption. prev->next should be ffff880c7ba42c70, but was (null)
Jul 16 18:12:25 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 18:12:25 srv1 kernel: Pid:108202, comm: couriertcpd veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 18:12:25 srv1 kernel: CallTrace:
Jul 16 18:12:25 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 18:12:25 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 18:12:25 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 18:12:25 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 16 18:12:25 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 18:12:25 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 18:12:25 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 18:12:25 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 18:12:25 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 18:12:25 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 18:12:25 srv1 kernel:[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
Jul 16 18:12:25 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 16 18:12:25 srv1 kernel: ---[ endtrace 1bbb5cda1f7c0286 ]---
Jul16 18:12:26 srv1 snmpd[2037]: Connection from UDP:[192.168.0.8]:45530->[192.168.0.10]
--
Jul 16 18:55:46 srv1 kernel:------------[ cut here ]------------
Jul 16 18:55:46 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 16 18:55:46 srv1 kernel: Hardwarename: X9DRW
Jul 16 18:55:46 srv1 kernel: list_delcorruption. prev->next should be ffff880673f3f700, but wasffffffff00000001
Jul 16 18:55:46 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap ipt_LOG macvlan tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd sb_edac soundcore tpm_tis tpm i2c_i801 tpm_biossnd_page_alloc serio_raw edac_core i2c_core ioatdma shpchpusb_storage ext3 jbd mbcache ses enclosure isci libsas igb ahci dcascsi_transport_sas aacraid [last unloaded: scsi_wait_scan]
Jul 16 18:55:46 srv1 kernel: Pid:108202, comm: couriertcpd veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 16 18:55:46 srv1 kernel: CallTrace:
Jul 16 18:55:46 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 16 18:55:46 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 16 18:55:46 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 16 18:55:46 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 16 18:55:46 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 16 18:55:46 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 16 18:55:46 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 16 18:55:46 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 16 18:55:46 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 16 18:55:46 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 16 18:55:46 srv1 kernel:[<ffffffff8100bfbb>] ? device_not_available+0x1b/0x20
Jul 16 18:55:46 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 16 18:55:46 srv1 kernel: ---[ endtrace 1bbb5cda1f7c0287 ]---
Jul 16 18:55:49 srv1 snmpd[2037]:Connection from UDP: [192.168.0.8]:50536->[192.168.0.10]
--
Jul 17 01:33:07 srv1 kernel:------------[ cut here ]------------
Jul 17 01:33:07 srv1 kernel: WARNING:at lib/list_debug.c:51 list_del+0x8d/0xa0() (Tainted: G W --------------- )
Jul 17 01:33:07 srv1 kernel: Hardwarename: X9DRW
Jul 17 01:33:07 srv1 kernel: list_delcorruption. next->prev should be ffff8801c7c89c00, but wasdead000000200200
Jul 17 01:33:07 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap macvlan ipt_LOG tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd soundcore sb_edac i2c_i801 tpm_tis tpmsnd_page_alloc tpm_bios serio_raw edac_core i2c_core ioatdma shpchpext3 jbd mbcache usb_storage ses enclosure isci libsas ahci aacraidigb scsi_transport_sas dca [last unloaded: scsi_wait_scan]
Jul 17 01:33:07 srv1 kernel: Pid:667078, comm: sh veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 17 01:33:07 srv1 kernel: CallTrace:
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 17 01:33:07 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 17 01:33:07 srv1 kernel:[<ffffffff8128301d>] ? list_del+0x8d/0xa0
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 17 01:33:07 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 17 01:33:07 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 17 01:33:07 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 17 01:33:07 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 17 01:33:07 srv1 kernel: ---[ endtrace 6c5dbae05996fd9a ]---
Jul 17 01:33:07 srv1 kernel:------------[ cut here ]------------
Jul 17 01:33:07 srv1 kernel: WARNING:at lib/list_debug.c:30 __list_add+0x8f/0xa0() (Tainted: G W --------------- )
Jul 17 01:33:07 srv1 kernel: Hardwarename: X9DRW
Jul 17 01:33:07 srv1 kernel: list_addcorruption. prev->next should be next (ffff8801c7f85bd0), but wasdead000000100100. (prev=ffff880bcb4c6d70).
Jul 17 01:33:07 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap macvlan ipt_LOG tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd soundcore sb_edac i2c_i801 tpm_tis tpmsnd_page_alloc tpm_bios serio_raw edac_core i2c_core ioatdma shpchpext3 jbd mbcache usb_storage ses enclosure isci libsas ahci aacraidigb scsi_transport_sas dca [last unloaded: scsi_wait_scan]
Jul 17 01:33:07 srv1 kernel: Pid:666860, comm: php-cgi veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 17 01:33:07 srv1 kernel: CallTrace:
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 17 01:33:07 srv1 kernel:[<ffffffff812830bf>] ? __list_add+0x8f/0xa0
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106bfb5>] ? copy_process+0x13b5/0x1400
Jul 17 01:33:07 srv1 kernel:[<ffffffff8118272e>] ? kmem_cache_alloc+0x1be/0x1e0
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106c09c>] ? do_fork_pid+0x9c/0x480
Jul 17 01:33:07 srv1 kernel:[<ffffffff81197f32>] ? get_empty_filp+0xa2/0x190
Jul 17 01:33:07 srv1 kernel:[<ffffffff8119804e>] ? alloc_file+0x2e/0xe0
Jul 17 01:33:07 srv1 kernel:[<ffffffff811b4523>] ? alloc_fd+0x53/0x140
Jul 17 01:33:07 srv1 kernel:[<ffffffff81193247>] ? fd_install+0x47/0x90
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106c49a>] ? do_fork+0x1a/0x20
Jul 17 01:33:07 srv1 kernel:[<ffffffff810090f8>] ? sys_clone+0x28/0x30
Jul 17 01:33:07 srv1 kernel:[<ffffffff8100b4a3>] ? stub_clone+0x13/0x20
Jul 17 01:33:07 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
--
Jul 17 01:33:07 srv1 kernel:------------[ cut here ]------------
Jul 17 01:33:07 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 17 01:33:07 srv1 kernel: Hardwarename: X9DRW
Jul 17 01:33:07 srv1 kernel: list_delcorruption. prev->next should be ffff880a12bc3230, but wasffff880c176d39c0
Jul 17 01:33:07 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap macvlan ipt_LOG tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd soundcore sb_edac i2c_i801 tpm_tis tpmsnd_page_alloc tpm_bios serio_raw edac_core i2c_core ioatdma shpchpext3 jbd mbcache usb_storage ses enclosure isci libsas ahci aacraidigb scsi_transport_sas dca [last unloaded: scsi_wait_scan]
Jul 17 01:33:07 srv1 kernel: Pid:666860, comm: php-cgi veid: 6548159 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 17 01:33:07 srv1 kernel: CallTrace:
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 17 01:33:07 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 17 01:33:07 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 17 01:33:07 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 17 01:33:07 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 17 01:33:07 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 17 01:33:07 srv1 kernel:[<ffffffff81193635>] ? sys_close+0xa5/0x100
Jul 17 01:33:07 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 17 01:33:07 srv1 kernel: ---[ endtrace 6c5dbae05996fd9c ]---
Jul 17 01:33:07 srv1 kernel:------------[ cut here ]------------
Jul 17 01:33:07 srv1 kernel: WARNING:at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G W --------------- )
Jul 17 01:33:07 srv1 kernel: Hardwarename: X9DRW
Jul 17 01:33:07 srv1 kernel: list_delcorruption. prev->next should be ffff880c176d39c0, but wasffff8801c7f85bd0
Jul 17 01:33:07 srv1 kernel: Moduleslinked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1ploop simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpcvzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filterip6_tables nf_nat_ftp nf_conntrack_ftp xt_state xt_length xt_hlxt_tcpmss xt_TCPMSS vhost_net xt_limit macvtap macvlan ipt_LOG tunxt_DSCP xt_dscp kvm_intel kvm ipt_REJECT ext4 jbd2 drbd vzeventib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi xt_u32iptable_mangle xt_multiport iptable_filter iptable_nat nf_natnf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables sg snd_pcspsnd_pcm snd_timer snd soundcore sb_edac i2c_i801 tpm_tis tpmsnd_page_alloc tpm_bios serio_raw edac_core i2c_core ioatdma shpchpext3 jbd mbcache usb_storage ses enclosure isci libsas ahci aacraidigb scsi_transport_sas dca [last unloaded: scsi_wait_scan]
Jul 17 01:33:07 srv1 kernel: Pid:85707, comm: couriertcpd veid: 119 Tainted: G W --------------- 2.6.32-16-pve #1
Jul 17 01:33:07 srv1 kernel: CallTrace:
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106cf48>] ? warn_slowpath_common+0x88/0xc0
Jul 17 01:33:07 srv1 kernel:[<ffffffff8106d036>] ? warn_slowpath_fmt+0x46/0x50
Jul 17 01:33:07 srv1 kernel:[<ffffffff810f0f05>] ? call_rcu_sched+0x15/0x20
Jul 17 01:33:07 srv1 kernel:[<ffffffff81282ffe>] ? list_del+0x6e/0xa0
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071122>] ? release_task+0x4d2/0x520
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071609>] ? wait_task_zombie+0x499/0x5f0
Jul 17 01:33:07 srv1 kernel:[<ffffffff810719d9>] ? wait_consider_task+0x209/0x550
Jul 17 01:33:07 srv1 kernel:[<ffffffff81071e16>] ? do_wait+0xf6/0x260
Jul 17 01:33:07 srv1 kernel:[<ffffffff81072008>] ? sys_wait4+0x88/0xd0
Jul 17 01:33:07 srv1 kernel:[<ffffffff81070290>] ? child_wait_callback+0x0/0x90
Jul 17 01:33:07 srv1 kernel:[<ffffffff8100b451>] ? int_signal+0x12/0x17
Jul 17 01:33:07 srv1 kernel:[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 17 01:33:07 srv1 kernel: ---[ endtrace 6c5dbae05996fd9d ]---
Jul 17 01:33:09 srv1 kernel:------------[ cut here ]------------
Jul 17 01:33:09 srv1 kernel: WARNING:at lib/list_debug.c:30 __list_add+0x8f/0xa0() (Tainted: G W --------------- )
--Any help would be appreciated