Strange errors on virtual machines

gerdnl

Member
Oct 21, 2013
114
0
16
What is likely to cause this?

[ 48.289800] EXT4-fs (vda1): re-mounted. Opts: errors=remount-ro

[ 50.133390] init: failsafe main process (629) killed by TERM signal

[ 65.420940] random: nonblocking pool is initialized

[ 71.605717] ip_tables: (C) 2000-2006 Netfilter Core Team

[ 71.610990] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

[ 71.638785] ip6_tables: (C) 2000-2006 Netfilter Core Team

[28361.820072] [sched_delayed] sched: RT throttling activated

[453601.060135] INFO: task jbd2/vda1-8:156 blocked for more than 120 seconds.

[453601.060398] Not tainted 3.13.0-35-generic #62-Ubuntu

[453601.060505] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[453601.060623] jbd2/vda1-8 D ffff8801bfc14440 0 156 2 0x00000000

[453601.060638] ffff8801b2adfbc8 0000000000000046 ffff8801b2a30000 ffff8801b2adffd8

[453601.060645] 0000000000014440 0000000000014440 ffff8801b2a30000 ffff8801bfc14cd8

[453601.060647] ffff8801bffbb098 0000000000000002 ffffffff811ee720 ffff8801b2adfc40

[453601.060650] Call Trace:

[453601.060750] [<ffffffff811ee720>] ? generic_block_bmap+0x50/0x50

[453601.060805] [<ffffffff81722a6d>] io_schedule+0x9d/0x140

[453601.060809] [<ffffffff811ee72e>] sleep_on_buffer+0xe/0x20

[453601.060812] [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90

[453601.060815] [<ffffffff811ee720>] ? generic_block_bmap+0x50/0x50

[453601.060818] [<ffffffff81722f97>] out_of_line_wait_on_bit+0x77/0x90

[453601.060864] [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40

[453601.060868] [<ffffffff811efa5a>] __wait_on_buffer+0x2a/0x30

[453601.060908] [<ffffffff812882e0>] jbd2_journal_commit_transaction+0xee0/0x1a70

[453601.060935] [<ffffffff8107550f>] ? try_to_del_timer_sync+0x4f/0x70

[453601.060939] [<ffffffff8128ce9d>] kjournald2+0xbd/0x250

[453601.060943] [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100

[453601.060946] [<ffffffff8128cde0>] ? commit_timeout+0x10/0x10

[453601.060956] [<ffffffff8108b4a2>] kthread+0xd2/0xf0

[453601.060960] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[453601.060963] [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0

[453601.060965] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[453601.060976] INFO: task postgres:949 blocked for more than 120 seconds.

[453601.061096] Not tainted 3.13.0-35-generic #62-Ubuntu

[453601.061200] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[453601.061317] postgres D ffff8801bfc14440 0 949 943 0x00000000

[453601.061320] ffff8801b5341a28 0000000000000082 ffff8801b4d33000 ffff8801b5341fd8

[453601.061323] 0000000000014440 0000000000014440 ffff8801b4d33000 ffff8801bfc14cd8

[453601.061325] ffff8801bffd9cb0 0000000000000002 ffffffff81284bb0 ffff8801b5341aa0

[453601.061327] Call Trace:

[453601.061340] [<ffffffff81284bb0>] ? start_this_handle+0x590/0x590

[453601.061344] [<ffffffff81722a6d>] io_schedule+0x9d/0x140

[453601.061347] [<ffffffff81284bbe>] sleep_on_shadow_bh+0xe/0x20

[453601.061350] [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90

[453601.061352] [<ffffffff81284bb0>] ? start_this_handle+0x590/0x590

[453601.061355] [<ffffffff81722f97>] out_of_line_wait_on_bit+0x77/0x90

[453601.061359] [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40

[453601.061362] [<ffffffff810aabe5>] ? wake_up_bit+0x25/0x30

[453601.061365] [<ffffffff81285fdd>] do_get_write_access+0x2ad/0x4f0

[453601.061368] [<ffffffff811f002d>] ? __getblk+0x2d/0x2e0

[453601.061372] [<ffffffff81286247>] jbd2_journal_get_write_access+0x27/0x40

[453601.061383] [<ffffffff8126c8cb>] __ext4_journal_get_write_access+0x3b/0x80

[453601.061409] [<ffffffff81242750>] ext4_reserve_inode_write+0x70/0xa0

[453601.061413] [<ffffffff812459e0>] ? ext4_dirty_inode+0x40/0x60

[453601.061416] [<ffffffff812427c4>] ext4_mark_inode_dirty+0x44/0x1f0

[453601.061419] [<ffffffff812459e0>] ext4_dirty_inode+0x40/0x60

[453601.061422] [<ffffffff811e6cca>] __mark_inode_dirty+0x10a/0x2d0

[453601.061432] [<ffffffff811d7db1>] update_time+0x81/0xd0

[453601.061435] [<ffffffff811d7fc0>] file_update_time+0x80/0xd0

[453601.061467] [<ffffffff8114fed0>] __generic_file_aio_write+0x180/0x3d0

[453601.061492] [<ffffffff813111f7>] ? apparmor_socket_sendmsg+0x17/0x20

[453601.061496] [<ffffffff81150178>] generic_file_aio_write+0x58/0xa0

[453601.061499] [<ffffffff8123a579>] ext4_file_write+0x99/0x400

[453601.061520] [<ffffffff811bca2a>] do_sync_write+0x5a/0x90

[453601.061524] [<ffffffff811bd1b4>] vfs_write+0xb4/0x1f0

[453601.061527] [<ffffffff811bdbe9>] SyS_write+0x49/0xa0

[453601.061529] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

[453601.061535] INFO: task supervisord:1107 blocked for more than 120 seconds.

[453601.061646] Not tainted 3.13.0-35-generic #62-Ubuntu

[453601.061755] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[453601.061875] supervisord D ffff8801bfc94440 0 1107 1 0x00000000

[453601.061878] ffff8800bb301a28 0000000000000086 ffff8801b486c800 ffff8800bb301fd8

[453601.061881] 0000000000014440 0000000000014440 ffff8801b486c800 ffff8801bfc94cd8

[453601.061883] ffff8801bffc64b0 0000000000000002 ffffffff81284bb0 ffff8800bb301aa0

[453601.061885] Call Trace:

[453601.061889] [<ffffffff81284bb0>] ? start_this_handle+0x590/0x590

[453601.061892] [<ffffffff81722a6d>] io_schedule+0x9d/0x140

[453601.061895] [<ffffffff81284bbe>] sleep_on_shadow_bh+0xe/0x20

[453601.061898] [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90

[453601.061900] [<ffffffff81284bb0>] ? start_this_handle+0x590/0x590

[453601.061903] [<ffffffff81722f97>] out_of_line_wait_on_bit+0x77/0x90

[453601.061907] [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40

[453601.061910] [<ffffffff810aabe5>] ? wake_up_bit+0x25/0x30

[453601.061913] [<ffffffff81285fdd>] do_get_write_access+0x2ad/0x4f0

[453601.061916] [<ffffffff811f002d>] ? __getblk+0x2d/0x2e0

[453601.061919] [<ffffffff81286247>] jbd2_journal_get_write_access+0x27/0x40

[453601.061922] [<ffffffff8126c8cb>] __ext4_journal_get_write_access+0x3b/0x80

[453601.061926] [<ffffffff81242750>] ext4_reserve_inode_write+0x70/0xa0

[453601.061929] [<ffffffff812459e0>] ? ext4_dirty_inode+0x40/0x60

[453601.061932] [<ffffffff812427c4>] ext4_mark_inode_dirty+0x44/0x1f0

[453601.061935] [<ffffffff812459e0>] ext4_dirty_inode+0x40/0x60

[453601.061937] [<ffffffff811e6cca>] __mark_inode_dirty+0x10a/0x2d0

[453601.061940] [<ffffffff811d7db1>] update_time+0x81/0xd0

[453601.061943] [<ffffffff811d7fc0>] file_update_time+0x80/0xd0

[453601.061946] [<ffffffff8114fed0>] __generic_file_aio_write+0x180/0x3d0

[453601.061949] [<ffffffff81150178>] generic_file_aio_write+0x58/0xa0

[453601.061952] [<ffffffff8123a579>] ext4_file_write+0x99/0x400

[453601.061971] [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20

[453601.061974] [<ffffffff811c238d>] ? cp_new_stat+0x13d/0x160

[453601.061977] [<ffffffff811bca2a>] do_sync_write+0x5a/0x90

[453601.061979] [<ffffffff811bd1b4>] vfs_write+0xb4/0x1f0

[453601.061982] [<ffffffff811bdbe9>] SyS_write+0x49/0xa0

[453601.061985] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

[453601.062000] INFO: task kworker/u8:2:28729 blocked for more than 120 seconds.

[453601.062112] Not tainted 3.13.0-35-generic #62-Ubuntu

[453601.062218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[453601.062333] kworker/u8:2 D ffff8801bfc14440 0 28729 2 0x00000000

[453601.062340] Workqueue: writeback bdi_writeback_workfn (flush-253:0)

[453601.062342] ffff88015c7fd658 0000000000000046 ffff8800b80de000 ffff88015c7fdfd8

[453601.062344] 0000000000014440 0000000000014440 ffff8800b80de000 ffff8801bfc14cd8

[453601.062347] ffff8801bffc64b0 0000000000000002 ffffffff81284bb0 ffff88015c7fd6d0

[453601.062349] Call Trace:

[453601.062352] [<ffffffff81284bb0>] ? start_this_handle+0x590/0x590

[453601.062356] [<ffffffff81722a6d>] io_schedule+0x9d/0x140

[453601.062358] [<ffffffff81284bbe>] sleep_on_shadow_bh+0xe/0x20

[453601.062361] [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90

[453601.062364] [<ffffffff81284bb0>] ? start_this_handle+0x590/0x590

[453601.062367] [<ffffffff81722f97>] out_of_line_wait_on_bit+0x77/0x90

[453601.062370] [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40

[453601.062373] [<ffffffff810aabe5>] ? wake_up_bit+0x25/0x30

[453601.062376] [<ffffffff81285fdd>] do_get_write_access+0x2ad/0x4f0

[453601.062379] [<ffffffff811f002d>] ? __getblk+0x2d/0x2e0

[453601.062382] [<ffffffff81286247>] jbd2_journal_get_write_access+0x27/0x40

[453601.062385] [<ffffffff8126c8cb>] __ext4_journal_get_write_access+0x3b/0x80

[453601.062388] [<ffffffff81242750>] ext4_reserve_inode_write+0x70/0xa0

[453601.062391] [<ffffffff8126619b>] ? __ext4_ext_dirty+0x6b/0x80

[453601.062394] [<ffffffff812427c4>] ext4_mark_inode_dirty+0x44/0x1f0

[453601.062405] [<ffffffff8126619b>] __ext4_ext_dirty+0x6b/0x80

[453601.062409] [<ffffffff81267c2f>] ext4_ext_insert_extent+0x43f/0x1250

[453601.062412] [<ffffffff81274972>] ? ext4_mb_new_blocks+0x172/0x580

[453601.062415] [<ffffffff8126b07a>] ext4_ext_map_blocks+0x8ca/0xf40

[453601.062419] [<ffffffff8123ff75>] ext4_map_blocks+0x2d5/0x540

[453601.062422] [<ffffffff81242f60>] ext4_writepages+0x5f0/0xcd0

[453601.062426] [<ffffffff8115a97e>] do_writepages+0x1e/0x40

[453601.062428] [<ffffffff811e6660>] __writeback_single_inode+0x40/0x210

[453601.062430] [<ffffffff811e7417>] writeback_sb_inodes+0x247/0x3e0

[453601.062433] [<ffffffff811e764f>] __writeback_inodes_wb+0x9f/0xd0

[453601.062435] [<ffffffff811e78c3>] wb_writeback+0x243/0x2c0

[453601.062438] [<ffffffff811e9282>] bdi_writeback_workfn+0x2a2/0x430

[453601.062441] [<ffffffff810839d2>] process_one_work+0x182/0x450

[453601.062444] [<ffffffff810847c1>] worker_thread+0x121/0x410

[453601.062447] [<ffffffff810846a0>] ? rescuer_thread+0x430/0x430

[453601.062450] [<ffffffff8108b4a2>] kthread+0xd2/0xf0

[453601.062452] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[453601.062455] [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0

[453601.062457] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[453601.062462] INFO: task postgres:16944 blocked for more than 120 seconds.

[453601.062572] Not tainted 3.13.0-35-generic #62-Ubuntu

[453601.062684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[453601.062802] postgres D ffff8801bfc14440 0 16944 943 0x00000004

[453601.062804] ffff8800b8b95d28 0000000000000086 ffff8800b80fe000 ffff8800b8b95fd8

[453601.062807] 0000000000014440 0000000000014440 ffff8800b80fe000 ffff8801bfc14cd8

[453601.062809] ffff8801bffd6de8 0000000000000002 ffffffff8114e420 ffff8800b8b95da0

[453601.062811] Call Trace:

[453601.062815] [<ffffffff8114e420>] ? wait_on_page_read+0x60/0x60

[453601.062818] [<ffffffff81722a6d>] io_schedule+0x9d/0x140

[453601.062821] [<ffffffff8114e42e>] sleep_on_page+0xe/0x20

[453601.062824] [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90

[453601.062827] [<ffffffff8114e1ef>] wait_on_page_bit+0x7f/0x90

[453601.062831] [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40

[453601.062843] [<ffffffff8115b9a1>] ? pagevec_lookup_tag+0x21/0x30

[453601.062847] [<ffffffff8114e2f9>] filemap_fdatawait_range+0xf9/0x190

[453601.062850] [<ffffffff8115a97e>] ? do_writepages+0x1e/0x40

[453601.062853] [<ffffffff8114fa89>] ? __filemap_fdatawrite_range+0x59/0x60

[453601.062856] [<ffffffff8114fb9f>] filemap_write_and_wait_range+0x3f/0x70

[453601.062859] [<ffffffff8123a99a>] ext4_sync_file+0xba/0x320

[453601.062862] [<ffffffff811ecf01>] do_fsync+0x51/0x80

[453601.062865] [<ffffffff811bdbe9>] ? SyS_write+0x49/0xa0

[453601.062868] [<ffffffff811ed1b3>] SyS_fdatasync+0x13/0x20

[453601.062870] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

[455281.060066] INFO: task jbd2/vda1-8:156 blocked for more than 120 seconds.

[455281.060120] Not tainted 3.13.0-35-generic #62-Ubuntu

[455281.060243] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[455281.060364] jbd2/vda1-8 D ffff8801bfc14440 0 156 2 0x00000000

[455281.060369] ffff8801b2adfca8 0000000000000046 ffff8801b2a30000 ffff8801b2adffd8

[455281.060373] 0000000000014440 0000000000014440 ffff8801b2a30000 ffff8801b2adfd98

[455281.060375] ffff8801b28928b8 ffff8801b2a30000 ffff88015c4aa000 ffff8801b2adfd80

[455281.060378] Call Trace:

[455281.060391] [<ffffffff81722749>] schedule+0x29/0x70

[455281.060399] [<ffffffff8128767f>] jbd2_journal_commit_transaction+0x27f/0x1a70

[455281.060404] [<ffffffff8109d27d>] ? sched_clock_local+0x1d/0x80

[455281.060410] [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100

[455281.060415] [<ffffffff8107465b>] ? lock_timer_base.isra.35+0x2b/0x50

[455281.060418] [<ffffffff8107550f>] ? try_to_del_timer_sync+0x4f/0x70

[455281.060422] [<ffffffff8128ce9d>] kjournald2+0xbd/0x250

[455281.060425] [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100

[455281.060443] [<ffffffff8128cde0>] ? commit_timeout+0x10/0x10

[455281.060451] [<ffffffff8108b4a2>] kthread+0xd2/0xf0

[455281.060456] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[455281.060460] [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0

[455281.060463] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[455281.060474] INFO: task postgres:949 blocked for more than 120 seconds.

[455281.060613] Not tainted 3.13.0-35-generic #62-Ubuntu

[455281.060727] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[455281.060846] postgres D ffff8801bfc14440 0 949 943 0x00000000

[455281.060850] ffff8801b5341ae8 0000000000000082 ffff8801b4d33000 ffff8801b5341fd8

[455281.060852] 0000000000014440 0000000000014440 ffff8801b4d33000 ffff8801b2892800

[455281.060854] ffff8801b2892870 000000000269e18e 0000000000000002 ffff8801b5578840

[455281.060857] Call Trace:

[455281.060861] [<ffffffff81722749>] schedule+0x29/0x70

[455281.060866] [<ffffffff81284575>] wait_transaction_locked+0x75/0xa0

[455281.060869] [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100

[455281.060872] [<ffffffff81284890>] start_this_handle+0x270/0x590

[455281.060922] [<ffffffff8165de45>] ? ip_local_out+0x25/0x30

[455281.060957] [<ffffffff811a2362>] ? kmem_cache_alloc+0x1b2/0x1e0

[455281.060962] [<ffffffff81284d50>] ? jbd2__journal_start+0x90/0x1e0

[455281.060964] [<ffffffff81284db3>] jbd2__journal_start+0xf3/0x1e0

[455281.060971] [<ffffffff812459ca>] ? ext4_dirty_inode+0x2a/0x60

[455281.060976] [<ffffffff8126c5d9>] __ext4_journal_start_sb+0x69/0xe0

[455281.060979] [<ffffffff812459ca>] ext4_dirty_inode+0x2a/0x60

[455281.060984] [<ffffffff811e6cca>] __mark_inode_dirty+0x10a/0x2d0

[455281.060988] [<ffffffff811d7db1>] update_time+0x81/0xd0

[455281.060991] [<ffffffff811d7fc0>] file_update_time+0x80/0xd0

[455281.060998] [<ffffffff8114fed0>] __generic_file_aio_write+0x180/0x3d0

[455281.061004] [<ffffffff813111f7>] ? apparmor_socket_sendmsg+0x17/0x20

[455281.061007] [<ffffffff81150178>] generic_file_aio_write+0x58/0xa0

[455281.061011] [<ffffffff8123a579>] ext4_file_write+0x99/0x400

[455281.061016] [<ffffffff811bca2a>] do_sync_write+0x5a/0x90

[455281.061019] [<ffffffff811bd1b4>] vfs_write+0xb4/0x1f0

[455281.061022] [<ffffffff811bdbe9>] SyS_write+0x49/0xa0

[455281.061025] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

[455281.061028] INFO: task postgres:953 blocked for more than 120 seconds.

[455281.061140] Not tainted 3.13.0-35-generic #62-Ubuntu

[455281.061263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[455281.061383] postgres D ffff8801bfd14440 0 953 943 0x00000000

[455281.061386] ffff8801b4d8b8f0 0000000000000082 ffff8800b8b9b000 ffff8801b4d8bfd8

[455281.061388] 0000000000014440 0000000000014440 ffff8800b8b9b000 ffff8801b2f1f440

[455281.061391] ffff8801bfd1c000 ffff8801b2f1f458 0000000000000002 0000000000000003

[455281.061393] Call Trace:

[455281.061397] [<ffffffff81722749>] schedule+0x29/0x70

[455281.061442] [<ffffffff81375546>] percpu_ida_alloc+0x186/0x390

[455281.061455] [<ffffffff81372535>] ? __bitmap_weight+0x35/0x80

[455281.061459] [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100

[455281.061472] [<ffffffff81342b8b>] blk_mq_wait_for_tags+0x1b/0x40

[455281.061476] [<ffffffff81341bf6>] blk_mq_alloc_request_pinned+0x56/0x120

[455281.061485] [<ffffffff81342777>] blk_mq_make_request+0x3a7/0x470

[455281.061502] [<ffffffff81335f22>] generic_make_request+0xc2/0x110

[455281.061508] [<ffffffff81335fe1>] submit_bio+0x71/0x150

[455281.061514] [<ffffffff81246755>] ext4_io_submit+0x25/0x50

[455281.061520] [<ffffffff81243163>] ext4_writepages+0x7f3/0xcd0

[455281.061526] [<ffffffff8115a97e>] do_writepages+0x1e/0x40

[455281.061532] [<ffffffff8114fa89>] __filemap_fdatawrite_range+0x59/0x60

[455281.061537] [<ffffffff8114fb3c>] filemap_flush+0x1c/0x20

[455281.061542] [<ffffffff81240798>] ext4_alloc_da_blocks+0x38/0x70

[455281.061545] [<ffffffff8124c96d>] ext4_rename+0x66d/0x9c0

[455281.061559] [<ffffffff81724b62>] ? mutex_lock+0x12/0x2f

[455281.061565] [<ffffffff811ca87a>] vfs_rename+0x57a/0x5d0

[455281.061569] [<ffffffff811cdab5>] SYSC_renameat+0x3b5/0x410

[455281.061581] [<ffffffff8117fb2c>] ? vm_munmap+0x4c/0x60

[455281.061586] [<ffffffff811ceccb>] SyS_rename+0x1b/0x20

[455281.061588] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

[455281.061593] INFO: task supervisord:1107 blocked for more than 120 seconds.

[455281.061715] Not tainted 3.13.0-35-generic #62-Ubuntu

[455281.061818] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[455281.061943] supervisord D ffff8801bfc94440 0 1107 1 0x00000000

[455281.061946] ffff8800bb301a18 0000000000000086 ffff8801b486c800 ffff8800bb301fd8

[455281.061949] 0000000000014440 0000000000014440 ffff8801b486c800 ffff8801bfc94cd8

[455281.061951] ffff8801bffd1a18 ffff8800bb301aa0 0000000000000002 ffffffff811ee720

[455281.061953] Call Trace:

[455281.061960] [<ffffffff811ee720>] ? generic_block_bmap+0x50/0x50

[455281.061964] [<ffffffff81722a6d>] io_schedule+0x9d/0x140

[455281.061966] [<ffffffff811ee72e>] sleep_on_buffer+0xe/0x20

[455281.061969] [<ffffffff81722ff8>] __wait_on_bit_lock+0x48/0xb0

[455281.061972] [<ffffffff811ee720>] ? generic_block_bmap+0x50/0x50

[455281.061975] [<ffffffff817230d7>] out_of_line_wait_on_bit_lock+0x77/0x90

[455281.061979] [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40

[455281.061983] [<ffffffff811efbca>] __lock_buffer+0x2a/0x30

[455281.061986] [<ffffffff81285f90>] do_get_write_access+0x260/0x4f0

[455281.061989] [<ffffffff811f002d>] ? __getblk+0x2d/0x2e0

[455281.061992] [<ffffffff81286247>] jbd2_journal_get_write_access+0x27/0x40

[455281.061996] [<ffffffff8126c8cb>] __ext4_journal_get_write_access+0x3b/0x80

[455281.061999] [<ffffffff81242750>] ext4_reserve_inode_write+0x70/0xa0

[455281.062002] [<ffffffff812459e0>] ? ext4_dirty_inode+0x40/0x60

[455281.062005] [<ffffffff812427c4>] ext4_mark_inode_dirty+0x44/0x1f0

[455281.062008] [<ffffffff812459e0>] ext4_dirty_inode+0x40/0x60

[455281.062011] [<ffffffff811e6cca>] __mark_inode_dirty+0x10a/0x2d0

[455281.062014] [<ffffffff811d7db1>] update_time+0x81/0xd0

[455281.062016] [<ffffffff811d7fc0>] file_update_time+0x80/0xd0

[455281.062020] [<ffffffff8114fed0>] __generic_file_aio_write+0x180/0x3d0

[455281.062023] [<ffffffff81150178>] generic_file_aio_write+0x58/0xa0

[455281.062026] [<ffffffff8123a579>] ext4_file_write+0x99/0x400

[455281.062031] [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20

[455281.062034] [<ffffffff811c238d>] ? cp_new_stat+0x13d/0x160

[455281.062037] [<ffffffff811bca2a>] do_sync_write+0x5a/0x90

[455281.062040] [<ffffffff811bd1b4>] vfs_write+0xb4/0x1f0

[455281.062042] [<ffffffff811bdbe9>] SyS_write+0x49/0xa0

[455281.062045] [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

[455281.062052] INFO: task kworker/u8:2:28729 blocked for more than 120 seconds.

[455281.062182] Not tainted 3.13.0-35-generic #62-Ubuntu

[455281.062288] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[455281.062421] kworker/u8:2 D ffff8801bfd94440 0 28729 2 0x00000000

[455281.062428] Workqueue: writeback bdi_writeback_workfn (flush-253:0)

[455281.062431] ffff88015c7fd780 0000000000000046 ffff8800b80de000 ffff88015c7fdfd8

[455281.062433] 0000000000014440 0000000000014440 ffff8800b80de000 ffff8801b2f1f440

[455281.062435] ffff8801bfd9c000 ffff8801b2f1f458 0000000000000002 0000000000000003

[455281.062437] Call Trace:

[455281.062441] [<ffffffff81722749>] schedule+0x29/0x70

[455281.062444] [<ffffffff81375546>] percpu_ida_alloc+0x186/0x390

[455281.062447] [<ffffffff81372535>] ? __bitmap_weight+0x35/0x80

[455281.062451] [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100

[455281.062454] [<ffffffff81342b8b>] blk_mq_wait_for_tags+0x1b/0x40

[455281.062457] [<ffffffff81341bf6>] blk_mq_alloc_request_pinned+0x56/0x120

[455281.062460] [<ffffffff81342777>] blk_mq_make_request+0x3a7/0x470

[455281.062463] [<ffffffff81335f22>] generic_make_request+0xc2/0x110

[455281.062476] [<ffffffff81335fe1>] submit_bio+0x71/0x150

[455281.062487] [<ffffffff811f48a6>] ? bio_alloc_bioset+0x196/0x2a0

[455281.062491] [<ffffffff81158e29>] ? account_page_writeback+0x29/0x30

[455281.062495] [<ffffffff811ef945>] _submit_bh+0x135/0x200

[455281.062498] [<ffffffff811f247a>] __block_write_full_page+0x12a/0x380

[455281.062501] [<ffffffff810aab74>] ? __wake_up+0x44/0x50

[455281.062504] [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10

[455281.062507] [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10

[455281.062512] [<ffffffff811f2891>] block_write_full_page_endio+0xe1/0xf0

[455281.062516] [<ffffffff811f28b5>] block_write_full_page+0x15/0x20

[455281.062521] [<ffffffff811f62f8>] blkdev_writepage+0x18/0x20

[455281.062524] [<ffffffff81158ba3>] __writepage+0x13/0x40

[455281.062529] [<ffffffff81159555>] write_cache_pages+0x235/0x480

[455281.062533] [<ffffffff81158b90>] ? global_dirtyable_memory+0x50/0x50

[455281.062538] [<ffffffff811597e0>] generic_writepages+0x40/0x60

[455281.062543] [<ffffffff8115a97e>] do_writepages+0x1e/0x40

[455281.062547] [<ffffffff811e6660>] __writeback_single_inode+0x40/0x210

[455281.062552] [<ffffffff811e7417>] writeback_sb_inodes+0x247/0x3e0

[455281.062556] [<ffffffff811e764f>] __writeback_inodes_wb+0x9f/0xd0

[455281.062560] [<ffffffff811e78c3>] wb_writeback+0x243/0x2c0

[455281.062562] [<ffffffff811e9282>] bdi_writeback_workfn+0x2a2/0x430

[455281.062568] [<ffffffff810839d2>] process_one_work+0x182/0x450

[455281.062571] [<ffffffff810847c1>] worker_thread+0x121/0x410

[455281.062574] [<ffffffff810846a0>] ? rescuer_thread+0x430/0x430

[455281.062576] [<ffffffff8108b4a2>] kthread+0xd2/0xf0

[455281.062579] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

[455281.062582] [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0

[455281.062584] [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0

root@pobbchat:~#
 

LnxBil

Well-Known Member
Feb 21, 2015
4,193
405
83
Germany
Looks like problems with the underlying storage backend. Is the VM currently in Backup mode and maybe stuck there?

Best use CODE tags for better formatting.
 

LnxBil

Well-Known Member
Feb 21, 2015
4,193
405
83
Germany
If the VM is trying to write and the backend is not responding. If the connection is reacquired, it should not be a problem, but if the virtual disk is dropped inside of the guest, you have to restart.

Is your VM responding correctly? Have a look at syslog (or dmesg -T) to see if the errors are recent or older.
 

gerdnl

Member
Oct 21, 2013
114
0
16
the vm is responding normaly , it seems only to happen when the backup is running, and then not even always.
 

gerdnl

Member
Oct 21, 2013
114
0
16
I changed my backup methode to local gzip and the same thing happends, 1 vm totaly frozen up in the morning and the backup still running

Code:
[  281.156270] Code: 00 55 89 f9 48 89 e5 0f 32 31 c9 89 c0 48 c1 e2 20 89 0e 48 09 c2 48 89 d0 5d c3 66 0f 1f 44 00 00 55 89 f0 89 f9 48 89 e5 0f 30 <31> c0 5d c3 66 90 55 89 f9 48 89 e5 0f 33 89 c0 48 c1 e2 20 48
[  259.809369] NMI backtrace for cpu 1
[  259.809369] CPU: 1 PID: 88 Comm: scsi_eh_1 Not tainted 3.13.0-35-generic #62-Ubuntu
[  259.809369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
[  259.809369] task: ffff8801b29f1800 ti: ffff8801b29da000 task.ti: ffff8801b29da000
[  259.809369] RIP: 0010:[<ffffffff8104f5a5>]  [<ffffffff8104f5a5>] native_halt+0x5/0x10
[  259.809369] RSP: 0018:ffff8801b29dbd28  EFLAGS: 00000046
[  259.809369] RAX: 000000000000056e RBX: ffff8801bfc4df80 RCX: 0000000000000001
[  259.809369] RDX: 0000000000000000 RSI: 0000000000000570 RDI: ffff8801bfff4040
[  259.809369] RBP: ffff8801b29dbd28 R08: 0000000000000004 R09: 05c0000000000000
[  259.809369] R10: dff8bd81afc000b8 R11: 0000000000000000 R12: 0000000000000001
[  259.809369] R13: 0000000000000000 R14: 0000003c8920cf3b R15: 0000000000000082
[  259.809369] FS:  0000000000000000(0000) GS:ffff8801bfc40000(0000) knlGS:0000000000000000
[  259.809369] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  259.809369] CR2: 00007f906caf0000 CR3: 00000000ae2d9000 CR4: 00000000000006e0
[  259.809369] Stack:
[  259.809369]  ffff8801b29dbd68 ffffffff8104ec25 0000057000000001 ffff8801b29dbe18
[  259.809369]  ffff8801b2a00000 ffff8801b2a00000 0000000000000000 0000000000000000
[  259.809369]  ffff8801b29dbdc0 ffffffff8104e381 0000000000000000 dff8bd81afc000b8
[  259.809369] Call Trace:
[  259.809369]  [<ffffffff8104ec25>] kvm_lock_spinning+0x125/0x1b0
[  259.809369]  [<ffffffff8104e381>] __raw_callee_save_kvm_lock_spinning+0x11/0x20
[  259.809369]  [<ffffffff817263e3>] ? _raw_spin_lock_irqsave+0x53/0x60
[  259.809369]  [<ffffffff81508651>] ata_scsi_cmd_error_handler+0x41/0x160
[  259.809369]  [<ffffffff817222d1>] ? __schedule+0x381/0x7d0
[  259.809369]  [<ffffffff8150e216>] ata_scsi_error+0x86/0xc0
[  259.809369]  [<ffffffff814df8f3>] scsi_error_handler+0x123/0x810
[  259.809369]  [<ffffffff814df7d0>] ? scsi_eh_get_sense+0x2b0/0x2b0
[  259.809369]  [<ffffffff8108b4a2>] kthread+0xd2/0xf0
[  259.809369]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  259.809369]  [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0
[  259.809369]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  259.809369] Code: 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 5d c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 <5d> c3 66 0f 1f 84 00 00 00 00 00 55 49 89 ca 49 89 d1 8b 07 48
[  281.156407] NMI backtrace for cpu 4
[  281.156411] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 3.13.0-35-generic #62-Ubuntu
[  281.156413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
[  281.156415] task: ffff8801b69b8000 ti: ffff8801b69c0000 task.ti: ffff8801b69c0000
[  281.156417] RIP: 0010:[<ffffffff8104f596>]  [<ffffffff8104f596>] native_safe_halt+0x6/0x10
[  281.156421] RSP: 0018:ffff8801b69c1ea8  EFLAGS: 00000286
[  281.156423] RAX: 00000000ffffffed RBX: ffff8801b69c1f38 RCX: 0100000000000000
[  281.156425] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[  281.156427] RBP: ffff8801b69c1ea8 R08: 0000000000000000 R09: 0000000000000000
[  281.156428] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000004
[  281.156430] R13: ffffffff81d13c40 R14: 0000000000000000 R15: ffff8801b69c1fd8
[  281.156432] FS:  0000000000000000(0000) GS:ffff8801bfd00000(0000) knlGS:0000000000000000
[  281.156434] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  281.156436] CR2: 00007f906ed83870 CR3: 00000001b251b000 CR4: 00000000000006e0
[  281.156440] Stack:
[  281.156441]  ffff8801b69c1ec8 ffffffff8101c5ff ffff8801b69c1f38 ffff8801b69c1fd8
[  281.156445]  ffff8801b69c1ed8 ffffffff8101cec6 ffff8801b69c1f28 ffffffff810bed85
[  281.156448]  ffff8801b69c1fd8 ffff8801b69c1fd8 ef6cbb843e70a7ca ffff8801b69c1f38
[  281.156451] Call Trace:
[  281.156457]  [<ffffffff8101c5ff>] default_idle+0x1f/0xc0
[  281.156460]  [<ffffffff8101cec6>] arch_cpu_idle+0x26/0x30
[  281.156464]  [<ffffffff810bed85>] cpu_startup_entry+0xc5/0x290
[  281.156467]  [<ffffffff81040fe8>] start_secondary+0x218/0x2c0
[  281.156469] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[  281.156007] NMI backtrace for cpu 3
[  281.156007] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-35-generic #62-Ubuntu
[  281.156007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
[  281.156007] task: ffff8801b69ae000 ti: ffff8801b69b6000 task.ti: ffff8801b69b6000
[  281.156007] RIP: 0010:[<ffffffff8104f596>]  [<ffffffff8104f596>] native_safe_halt+0x6/0x10
[  281.156007] RSP: 0018:ffff8801b69b7ea8  EFLAGS: 00000286
[  281.156007] RAX: 00000000ffffffed RBX: ffff8801b69b7f38 RCX: 0100000000000000
[  281.156007] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[  281.156007] RBP: ffff8801b69b7ea8 R08: 0000000000000000 R09: 0000000000000000
[  281.156007] R10: 0000000000000000 R11: 00000000000003ff R12: 0000000000000003
[  281.156007] R13: ffffffff81d13c40 R14: 0000000000000000 R15: ffff8801b69b7fd8
[  281.156007] FS:  0000000000000000(0000) GS:ffff8801bfcc0000(0000) knlGS:0000000000000000
[  281.156007] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  281.156007] CR2: 00007f904752aff4 CR3: 0000000001c0e000 CR4: 00000000000006e0
[  281.156007] Stack:
[  281.156007]  ffff8801b69b7ec8 ffffffff8101c5ff ffff8801b69b7f38 ffff8801b69b7fd8
[  281.156007]  ffff8801b69b7ed8 ffffffff8101cec6 ffff8801b69b7f28 ffffffff810bed85
[  281.156007]  ffff8801b69b7fd8 ffff8801b69b7fd8 06d56c7e253e92e2 ffff8801b69b7f38
[  281.156007] Call Trace:
[  281.156007]  [<ffffffff8101c5ff>] default_idle+0x1f/0xc0
[  281.156007]  [<ffffffff8101cec6>] arch_cpu_idle+0x26/0x30
[  281.156007]  [<ffffffff810bed85>] cpu_startup_entry+0xc5/0x290
[  281.156007]  [<ffffffff81040fe8>] start_secondary+0x218/0x2c0
[  281.156007] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[  281.156674] NMI backtrace for cpu 5
[  281.156678] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 3.13.0-35-generic #62-Ubuntu
[  281.156680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014

[  259.809369] ata2: lost interrupt (Status 0x58)
[  327.647165] ata2: drained 65536 bytes to clear DRQ
[  328.121023] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[  328.121155] sr 1:0:0:0: CDB:
[  328.121159] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
[  328.121169] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
[  328.121169]          res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[  328.121292] ata2.00: status: { DRDY }
[  328.121445] ata2: soft resetting link
[  328.283335] ata2.00: configured for MWDMA2
[  328.283683] ata2: EH complete
[  639.044082] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=15002 jiffies, g=32946, c=32945, q=0)
[  639.044209] sending NMI to all CPUs:
[  639.044223] NMI backtrace for cpu 2
[  639.044227] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.13.0-35-generic #62-Ubuntu
[  639.044229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
[  609.837592] task: ffff8801b29f1800 ti: ffff8801b29da000 task.ti: ffff8801b29da000
[  609.837592] RIP: 0010:[<ffffffff8104f5a5>]  [<ffffffff8104f5a5>] native_halt+0x5/0x10
[  609.837592] RSP: 0018:ffff8801b29dbd28  EFLAGS: 00000046
[  609.837592] RAX: 000000000000097e RBX: ffff8801bfccdf80 RCX: 0000000000000001
[  609.837592] RDX: 0000000000000000 RSI: 0000000000000980 RDI: ffff8801bfff40c0
[  609.837592] RBP: ffff8801b29dbd28 R08: 0000000000000006 R09: 05c0000000000000
[  609.837592] R10: dff8bd81afc000b8 R11: 0000000000000005 R12: 0000000000000003
[  609.837592] R13: 0000000000000000 R14: 0000008fe170cd1b R15: 0000000000000082
[  609.837592] FS:  0000000000000000(0000) GS:ffff8801bfcc0000(0000) knlGS:0000000000000000
[  609.837592] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  609.837592] CR2: 00007f904752aff4 CR3: 0000000001c0e000 CR4: 00000000000006e0
[  609.837592] Stack:
[  609.837592]  ffff8801b29dbd68 ffffffff8104ec25 0000098000000001 ffff8801b29dbe18
[  609.837592]  ffff8801b2a00000 ffff8801b2a00000 0000000000000000 0000000000000000
[  609.837592]  ffff8801b29dbdc0 ffffffff8104e381 0000000000000005 dff8bd81afc000b8
[  609.837592] Call Trace:
[  609.837592]  [<ffffffff8104ec25>] kvm_lock_spinning+0x125/0x1b0
[  609.837592]  [<ffffffff8104e381>] __raw_callee_save_kvm_lock_spinning+0x11/0x20
[  609.837592]  [<ffffffff817263e3>] ? _raw_spin_lock_irqsave+0x53/0x60
[  609.837592]  [<ffffffff81508651>] ata_scsi_cmd_error_handler+0x41/0x160
[  609.837592]  [<ffffffff817222d1>] ? __schedule+0x381/0x7d0
[  609.837592]  [<ffffffff8150e216>] ata_scsi_error+0x86/0xc0
[  609.837592]  [<ffffffff814df8f3>] scsi_error_handler+0x123/0x810
[  609.837592]  [<ffffffff814df7d0>] ? scsi_eh_get_sense+0x2b0/0x2b0
[  609.837592]  [<ffffffff8108b4a2>] kthread+0xd2/0xf0
[  609.837592]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  609.837592]  [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0
[  609.837592]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  609.837592] Code: 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 5d c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 <5d> c3 66 0f 1f 84 00 00 00 00 00 55 49 89 ca 49 89 d1 8b 07 48
[  720.800114] INFO: task kworker/u12:1:85 blocked for more than 120 seconds.
[  720.800216]       Not tainted 3.13.0-35-generic #62-Ubuntu
[  720.800266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.800338] kworker/u12:1   D ffff8801bfc54440     0    85      2 0x00000000
[  720.800348] Workqueue: writeback bdi_writeback_workfn (flush-253:0)
[  720.800351]  ffff8801b2a13740 0000000000000046 ffff8801b2a0b000 ffff8801b2a13fd8
[  720.800355]  0000000000014440 0000000000014440 ffff8801b2a0b000 ffff8801b2af8240
[  720.800357]  ffff8801bfc5c040 ffff8801b2af8258 0000000000000003 0000000000000004
[  720.800360] Call Trace:
[  720.800368]  [<ffffffff81722749>] schedule+0x29/0x70
[  720.800372]  [<ffffffff81375546>] percpu_ida_alloc+0x186/0x390
[  720.800377]  [<ffffffff81372535>] ? __bitmap_weight+0x35/0x80
[  720.800383]  [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100
[  720.800387]  [<ffffffff81342b8b>] blk_mq_wait_for_tags+0x1b/0x40
[  720.800390]  [<ffffffff81341bf6>] blk_mq_alloc_request_pinned+0x56/0x120
[  720.800393]  [<ffffffff81342777>] blk_mq_make_request+0x3a7/0x470
[  720.800398]  [<ffffffff81335f22>] generic_make_request+0xc2/0x110
[  720.800400]  [<ffffffff81335fe1>] submit_bio+0x71/0x150
[  720.800405]  [<ffffffff81159cb7>] ? __test_set_page_writeback+0x157/0x1b0
[  720.800412]  [<ffffffff81246755>] ext4_io_submit+0x25/0x50
[  720.800414]  [<ffffffff81246909>] ext4_bio_write_page+0x159/0x2e0
[  720.800418]  [<ffffffff8123decd>] mpage_submit_page+0x5d/0x80
[  720.800420]  [<ffffffff8123e000>] mpage_process_page_bufs+0x110/0x130
[  720.800423]  [<ffffffff8123ef59>] mpage_prepare_extent_to_map+0x219/0x2c0
[  720.800426]  [<ffffffff81242d50>] ext4_writepages+0x3e0/0xcd0
[  720.800429]  [<ffffffff8115a97e>] do_writepages+0x1e/0x40
[  720.800432]  [<ffffffff811e6660>] __writeback_single_inode+0x40/0x210
[  720.800434]  [<ffffffff811e7417>] writeback_sb_inodes+0x247/0x3e0
[  720.800436]  [<ffffffff811e764f>] __writeback_inodes_wb+0x9f/0xd0
[  720.800438]  [<ffffffff811e78c3>] wb_writeback+0x243/0x2c0
[  720.800440]  [<ffffffff811e9282>] bdi_writeback_workfn+0x2a2/0x430
[  720.800445]  [<ffffffff810839d2>] process_one_work+0x182/0x450
[  720.800448]  [<ffffffff810847c1>] worker_thread+0x121/0x410
[  720.800450]  [<ffffffff810846a0>] ? rescuer_thread+0x430/0x430
[  720.800453]  [<ffffffff8108b4a2>] kthread+0xd2/0xf0
[  720.800456]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  720.800458]  [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0
[  720.800460]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  720.800464] INFO: task jbd2/vda1-8:189 blocked for more than 120 seconds.
[  720.800519]       Not tainted 3.13.0-35-generic #62-Ubuntu
[  720.800575] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.800637] jbd2/vda1-8     D ffff8801bfd54440     0   189      2 0x00000000
[  720.800639]  ffff8801b257fbc8 0000000000000046 ffff8801b2489800 ffff8801b257ffd8
[  720.800642]  0000000000014440 0000000000014440 ffff8801b2489800 ffff8801bfd54cd8
[  720.800644]  ffff8801bffb9820 0000000000000002 ffffffff811ee720 ffff8801b257fc40
[  720.800646] Call Trace:
[  720.800650]  [<ffffffff811ee720>] ? generic_block_bmap+0x50/0x50
[  720.800653]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
[  720.800655]  [<ffffffff811ee72e>] sleep_on_buffer+0xe/0x20
[  720.800658]  [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90
[  720.800660]  [<ffffffff811ee720>] ? generic_block_bmap+0x50/0x50
[  720.800663]  [<ffffffff81722f97>] out_of_line_wait_on_bit+0x77/0x90
[  720.800666]  [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40
[  720.800669]  [<ffffffff811efa5a>] __wait_on_buffer+0x2a/0x30
[  720.800673]  [<ffffffff81288c1c>] jbd2_journal_commit_transaction+0x181c/0x1a70
[  720.800677]  [<ffffffff8107550f>] ? try_to_del_timer_sync+0x4f/0x70
[  720.800680]  [<ffffffff8128ce9d>] kjournald2+0xbd/0x250
[  720.800683]  [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100
[  720.800686]  [<ffffffff8128cde0>] ? commit_timeout+0x10/0x10
[  720.800688]  [<ffffffff8108b4a2>] kthread+0xd2/0xf0
[  720.800690]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  720.800692]  [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0
[  720.800694]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  720.800702] INFO: task postgres:1105 blocked for more than 120 seconds.
[  720.800755]       Not tainted 3.13.0-35-generic #62-Ubuntu
[  720.800803] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.800871] postgres        D ffff8801bfc54440     0  1105   1081 0x00000000
[  720.800874]  ffff8801b46878f0 0000000000000086 ffff8801b2726000 ffff8801b4687fd8
[  720.800876]  0000000000014440 0000000000014440 ffff8801b2726000 ffff8801b2af8240
[  720.800878]  ffff8801bfc5c040 ffff8801b2af8258 0000000000000003 0000000000000004
[  720.800880] Call Trace:
[  720.800883]  [<ffffffff81722749>] schedule+0x29/0x70
[  720.800886]  [<ffffffff81375546>] percpu_ida_alloc+0x186/0x390
[  720.800888]  [<ffffffff81372535>] ? __bitmap_weight+0x35/0x80
[  720.800891]  [<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100
[  720.800894]  [<ffffffff81342b8b>] blk_mq_wait_for_tags+0x1b/0x40
[  720.800897]  [<ffffffff81341bf6>] blk_mq_alloc_request_pinned+0x56/0x120
[  720.800899]  [<ffffffff81342777>] blk_mq_make_request+0x3a7/0x470
[  720.800902]  [<ffffffff81335f22>] generic_make_request+0xc2/0x110
[  720.800904]  [<ffffffff81335fe1>] submit_bio+0x71/0x150
[  720.800907]  [<ffffffff81246755>] ext4_io_submit+0x25/0x50
[  720.800910]  [<ffffffff81243163>] ext4_writepages+0x7f3/0xcd0
[  720.800913]  [<ffffffff8115a97e>] do_writepages+0x1e/0x40
[  720.800926]  [<ffffffff8114fa89>] __filemap_fdatawrite_range+0x59/0x60
[  720.800928]  [<ffffffff8114fb3c>] filemap_flush+0x1c/0x20
[  720.800931]  [<ffffffff81240798>] ext4_alloc_da_blocks+0x38/0x70
[  720.800933]  [<ffffffff8124c96d>] ext4_rename+0x66d/0x9c0
[  720.800936]  [<ffffffff81724b62>] ? mutex_lock+0x12/0x2f
[  720.800940]  [<ffffffff811ca87a>] vfs_rename+0x57a/0x5d0
[  720.800943]  [<ffffffff811cdab5>] SYSC_renameat+0x3b5/0x410
[  720.800948]  [<ffffffff8117fb2c>] ? vm_munmap+0x4c/0x60
[  720.800951]  [<ffffffff811ceccb>] SyS_rename+0x1b/0x20
[  720.800953]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
[  720.800962] INFO: task postgres:13779 blocked for more than 120 seconds.
[  720.801040]       Not tainted 3.13.0-35-generic #62-Ubuntu
[  720.801092] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.801153] postgres        D ffff8801bfd54440     0 13779   1081 0x00000000
[  720.801155]  ffff8801b5dc1d28 0000000000000082 ffff8801b3271800 ffff8801b5dc1fd8
[  720.801157]  0000000000014440 0000000000014440 ffff8801b3271800 ffff8801bfd54cd8
[  720.801160]  ffff8801bffd97e8 0000000000000002 ffffffff8114e420 ffff8801b5dc1da0
[  720.801162] Call Trace:
[  720.801165]  [<ffffffff8114e420>] ? wait_on_page_read+0x60/0x60
[  720.801168]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
[  720.801171]  [<ffffffff8114e42e>] sleep_on_page+0xe/0x20
[  720.801173]  [<ffffffff81722ef2>] __wait_on_bit+0x62/0x90
[  720.801176]  [<ffffffff8114e1ef>] wait_on_page_bit+0x7f/0x90
[  720.801179]  [<ffffffff810ab010>] ? autoremove_wake_function+0x40/0x40
[  720.801182]  [<ffffffff8115b9a1>] ? pagevec_lookup_tag+0x21/0x30
[  720.801185]  [<ffffffff8114e2f9>] filemap_fdatawait_range+0xf9/0x190
[  720.801188]  [<ffffffff8115a97e>] ? do_writepages+0x1e/0x40
[  720.801191]  [<ffffffff8114fa89>] ? __filemap_fdatawrite_range+0x59/0x60
[  720.801194]  [<ffffffff8114fb9f>] filemap_write_and_wait_range+0x3f/0x70
[  720.801197]  [<ffffffff8123a99a>] ext4_sync_file+0xba/0x320
[  720.801199]  [<ffffffff811ecf01>] do_fsync+0x51/0x80
[  720.801203]  [<ffffffff811bdbe9>] ? SyS_write+0x49/0xa0
[  720.801205]  [<ffffffff811ed1b3>] SyS_fdatasync+0x13/0x20
[  720.801207]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
[  586.305845] NMI backtrace for cpu 0
[  586.305845] CPU: 0 PID: 6 Comm: kworker/u12:0 Not tainted 3.13.0-35-generic #62-Ubuntu
[  586.305845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
[  586.305845] Workqueue: events_freezable_power_ disk_events_workfn
[  586.305845] task: ffff8801b6968000 ti: ffff8801b6970000 task.ti: ffff8801b6970000
[  586.305845] RIP: 0010:[<ffffffff8137683b>]  [<ffffffff8137683b>] iowrite8+0x1b/0x40
[  586.305845] RSP: 0018:ffff8801b69719a8  EFLAGS: 00000002
[  586.305845] RAX: 00000000000000a0 RBX: ffff8801b2a00000 RCX: 0000000001005900
[  586.305845] RDX: 0000000000000176 RSI: 0000000000010176 RDI: 00000000000000a0
[  586.305845] RBP: ffff8801b69719d0 R08: 000000100000014a R09: 0008000000100000
[  586.305845] R10: ffff8801b2a00198 R11: 0000000000000008 R12: ffff8801b2a00000
[  586.305845] R13: ffff8801b2a00178 R14: 0000000000000002 R15: ffff8801b2a00158
[  586.305845] FS:  0000000000000000(0000) GS:ffff8801bfc00000(0000) knlGS:0000000000000000
[  586.305845] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  586.305845] CR2: 00007f904752aff4 CR3: 00000000ad352000 CR4: 00000000000006f0
[  586.305845] Stack:
[  586.305845]  ffffffff8151221a ffff8801b2a00158 ffff8801b2a00000 ffff8801b2a01e70
[  586.305845]  ffff8801b2a00178 ffff8801b6971a00 ffffffff8151258d ffff8801b2a00158
[  586.305845]  ffff8801b2a00000 ffff8801b2a01e70 0000000000000001 ffff8801b6971a30
[  586.305845] Call Trace:
[  586.305845]  [<ffffffff8151221a>] ? ata_sff_tf_load+0xea/0x1b0
[  586.305845]  [<ffffffff8151258d>] ata_sff_qc_issue+0x6d/0x1c0
[  586.305845]  [<ffffffff81512713>] ata_bmdma_qc_issue+0x33/0x190
[  586.305845]  [<ffffffff814fe742>] ata_qc_issue+0x172/0x380
[  586.305845]  [<ffffffff815039db>] ? atapi_xlat+0x14b/0x170
[  586.305845]  [<ffffffff81507626>] ata_scsi_queuecmd+0x126/0x400
[  586.305845]  [<ffffffff81503890>] ? ata_scsiop_inq_std+0xa0/0xa0
[  586.305845]  [<ffffffff814da121>] scsi_dispatch_cmd+0xb1/0x290
[  586.305845]  [<ffffffff814e228a>] scsi_request_fn+0x2fa/0x5e0
[  586.305845]  [<ffffffff813360f3>] __blk_run_queue+0x33/0x40
[  586.305845]  [<ffffffff8133dc25>] blk_execute_rq_nowait+0xa5/0x150
[  586.305845]  [<ffffffff8133dd52>] blk_execute_rq+0x82/0x120
[  586.305845]  [<ffffffff813398e8>] ? blk_rq_bio_prep+0x68/0xc0
[  586.305845]  [<ffffffff8133daa9>] ? blk_rq_map_kern+0xb9/0x160
[  586.305845]  [<ffffffff8133826b>] ? blk_get_request+0x8b/0x110
[  586.305845]  [<ffffffff814e01a7>] scsi_execute+0xd7/0x160
[  586.305845]  [<ffffffff814e143c>] scsi_execute_req_flags+0x8c/0x100
[  586.305845]  [<ffffffff814f443b>] sr_check_events+0xbb/0x2d0
[  586.305845]  [<ffffffff815325c8>] cdrom_check_events+0x18/0x30
[  586.305845]  [<ffffffff814f48c9>] sr_block_check_events+0x39/0x50
[  586.305845]  [<ffffffff81345dec>] disk_check_events+0x4c/0x180
[  586.305845]  [<ffffffff81345f36>] disk_events_workfn+0x16/0x20
[  586.305845]  [<ffffffff810839d2>] process_one_work+0x182/0x450
[  586.305845]  [<ffffffff810847c1>] worker_thread+0x121/0x410
[  586.305845]  [<ffffffff810846a0>] ? rescuer_thread+0x430/0x430
[  586.305845]  [<ffffffff8108b4a2>] kthread+0xd2/0xf0
[  586.305845]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  586.305845]  [<ffffffff8172ecbc>] ret_from_fork+0x7c/0xb0
[  586.305845]  [<ffffffff8108b3d0>] ? kthread_create_on_node+0x1c0/0x1c0
[  586.305845] Code: 8b 07 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 81 fe ff ff 03 00 48 89 f2 77 2c 48 81 fe 00 00 01 00 76 0b 0f b7 d6 89 f8 ee <c3> 0f 1f 40 00 55 48 c7 c6 a9 ed a8 81 48 89 d7 48 89 e5 e8 9d
[  609.837592] ata2: lost interrupt (Status 0x58)
[  760.946919] ata2: drained 65536 bytes to clear DRQ
[  761.289846] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[  761.289896] sr 1:0:0:0: CDB:
[  761.289898] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
[  761.289907] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
[  761.289907]          res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[  761.289984] ata2.00: status: { DRDY }
[  761.290111] ata2: soft resetting link
[  761.445540] ata2.00: configured for MWDMA2
[  761.445886] ata2: EH complete
root@directopdate:~#
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!