Call trace on proxmox node

K

Kaya

Member
Jun 20, 2012
111
2
18
Hi everybody,
Today I noticed a strange problem in one of two proxmox nodes:
On Proxmox node 2 there is a call trace like that:
Code: 
root@proxmox2:~# cat /var/log/kern.log.1Aug  1 23:03:33 proxmox2 kernel: INFO: task lzop:606425 blocked for more than 120 seconds.
Aug  1 23:03:33 proxmox2 kernel:      Not tainted 2.6.32-29-pve #1
Aug  1 23:03:33 proxmox2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  1 23:03:33 proxmox2 kernel: lzop          D ffff8806684724c0     0 606425 606412    0 0x00000000
Aug  1 23:03:33 proxmox2 kernel: ffff8804da045a68 0000000000000086 0000000000000000 0000000000000000
Aug  1 23:03:33 proxmox2 kernel: ffff8804da045a28 ffffffff8100983c 000000000002526e 0000000000100000
Aug  1 23:03:33 proxmox2 kernel: 0000000000000000 00000001b6069634 ffff880668472a88 000000000001ec80
Aug  1 23:03:33 proxmox2 kernel: Call Trace:
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff8100983c>] ? __switch_to+0x1ac/0x320
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02c71b0>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff8155c5d3>] io_schedule+0x73/0xc0
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02c71be>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff8155d5ff>] __wait_on_bit+0x5f/0x90
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02c71b0>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff8155d6a8>] out_of_line_wait_on_bit+0x78/0x90
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff810a27a0>] ? wake_bit_function+0x0/0x40
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02c719f>] nfs_wait_on_request+0x2f/0x40 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02cdf0b>] nfs_updatepage+0x22b/0x580 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02bbac2>] nfs_write_end+0x142/0x280 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff81162e64>] ? ii_iovec_copy_from_user_atomic+0x84/0x110
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff81135970>] generic_file_buffered_write_iter+0x170/0x2b0
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff811365f5>] __generic_file_write_iter+0x225/0x420
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff81136875>] __generic_file_aio_write+0x85/0xa0
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff81136918>] generic_file_aio_write+0x88/0x100
Aug  1 23:03:33 proxmox2 kernel: [<ffffffffa02bae6c>] nfs_file_write+0x10c/0x280 [nfs]
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff811abf72>] do_sync_write+0xf2/0x140
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff811ac258>] vfs_write+0xb8/0x1a0
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff811acb51>] sys_write+0x51/0x90
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff8155f60e>] ? do_device_not_available+0xe/0x10
Aug  1 23:03:33 proxmox2 kernel: [<ffffffff8100b102>] system_call_fastpath+0x16/0x1b
Aug  1 23:07:33 proxmox2 kernel: INFO: task lzop:606425 blocked for more than 120 seconds.
Aug  1 23:07:33 proxmox2 kernel:      Not tainted 2.6.32-29-pve #1
Aug  1 23:07:33 proxmox2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  1 23:07:33 proxmox2 kernel: lzop          D ffff8806684724c0     0 606425 606412    0 0x00000000
Aug  1 23:07:33 proxmox2 kernel: ffff8804da045a68 0000000000000086 0000000000000000 0000000000000000
Aug  1 23:07:33 proxmox2 kernel: ffff8804da045a28 ffffffff8100983c 000000000002526e 0000000000100000
Aug  1 23:07:33 proxmox2 kernel: 0000000000000000 00000001b60a7b96 ffff880668472a88 000000000001ec80
Aug  1 23:07:33 proxmox2 kernel: Call Trace:
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff8100983c>] ? __switch_to+0x1ac/0x320
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02c71b0>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff8155c5d3>] io_schedule+0x73/0xc0
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02c71be>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff8155d5ff>] __wait_on_bit+0x5f/0x90
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02c71b0>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff8155d6a8>] out_of_line_wait_on_bit+0x78/0x90
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff810a27a0>] ? wake_bit_function+0x0/0x40
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02c719f>] nfs_wait_on_request+0x2f/0x40 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02cdf0b>] nfs_updatepage+0x22b/0x580 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02bbac2>] nfs_write_end+0x142/0x280 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff81162e64>] ? ii_iovec_copy_from_user_atomic+0x84/0x110
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff81135970>] generic_file_buffered_write_iter+0x170/0x2b0
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff811365f5>] __generic_file_write_iter+0x225/0x420
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff81136875>] __generic_file_aio_write+0x85/0xa0
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff81136918>] generic_file_aio_write+0x88/0x100
Aug  1 23:07:33 proxmox2 kernel: [<ffffffffa02bae6c>] nfs_file_write+0x10c/0x280 [nfs]
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff811abf72>] do_sync_write+0xf2/0x140
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff811ac258>] vfs_write+0xb8/0x1a0
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff811acb51>] sys_write+0x51/0x90
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff8155f60e>] ? do_device_not_available+0xe/0x10
Aug  1 23:07:33 proxmox2 kernel: [<ffffffff8100b102>] system_call_fastpath+0x16/0x1b

I can't understand on what's happening, except that on 23:00:00 backup start.
Can someone help me to understand what's happened?

Thanks
 
I am guessing NFS was not quite happy during the backup process.

It might be because of many factors: hardware, networking, cabling, etc. Simple exclude one by one but one thing at the time.

Try to exercise mounted nfs with various tests like: (just vary size and number of files)
Code: 
[COLOR=#141414][FONT=Consolas]~# dd if=/dev/zero of=/nas-mount-point/samplefile bs=1M count=1024 oflag=direct
[/FONT][/COLOR][COLOR=#141414][FONT=Consolas]~# dd if=/nas-mount-point/samplefile of=/dev/null bs=1M count=1024 iflag=direct[/FONT][/COLOR][COLOR=#141414][FONT=Consolas]
[/FONT][/COLOR]

You might also try to play with nfs parameters in /etc/fstab Just google some examples.
 
You must log in or register to reply here.
Top Bottom