Task blocked for more than 120 seconds => VM file corruption?

Miktash · Jul 8, 2015

I have a proxmox node that's currently running only one VM. It's running a mysql server inside and I have configured linux software raid10 in proxmox over 4 samsung 840 Pro SSD's. Speed is awesome, also in the VM. So mysql is running better than expected !
It's been running like that for a couple of weeks without any issues until today.

I noticed that one myisam table in the mysql VM got corrupted. I had to "repair table tbl" to fix it and everything is up and running again since. But then I started investigating and I noticed this in the VM's kernel log:

Code:

(my message got too long for this forum to accept it. So i'm going to post it right below in a reply)

So I logged in to proxmox and checked it's kernel logs and I see this:

Code:

[FONT=Menlo](my message got too long for thie forum to accept it. So I'm going to post it right below in a second reply)
[/FONT]

So the proxmox host was busy with lzop. If I'm not mistaken this is the compression process that runs while taking backups. And yes, the kernel log entries are at the time the backups are supposed to run.. So it seems related to the backups.

I have no idea why it suddenly became a problem? but I know for sure that disk-speed shouldn't be an issue as these ssd's aren't used that heavily. Especially not during the backup time...

The call trace talks about nfs, and it's true that our backups are getting stored on NAS that's available over NFS to proxmox. So i'm thinking that maybe the switch is failing and blocked for some reason?

That could explain the error on the proxmox host itself. But it doesn't explain why I get these 'blocked for 120 second' messages inside the VM.

Inside the VM it looks like the call trace is talking about the filesystem. So it makes me think the VM had troubles accessing it's disk?

If the VM had troubles accessing it's disk and the proxmox host too, then it could explain why I got that message in both the host and the VM.

But the host never talked about a disk issue or anything like that. It even did a complete software raid 10 data-check right before the backup started:

Code:

md: data-check of RAID array md0

Code:

[FONT=Menlo]md: minimum _guaranteed_  speed: 1000 KB/sec/disk.[/FONT]
[FONT=Menlo]md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.[/FONT]
[FONT=Menlo]md: using 128k window, over a total of 499855360k.[/FONT]
[FONT=Menlo]md: md0: data-check done.[/FONT]

So i'm puzzled on what happend here.

Can anyone help me find out what went wrong?

Miktash · Jul 8, 2015

So, the error I got inside the VM:

Code:

[FONT=Menlo][487201.436538] INFO: task jbd2/dm-0-8:387 blocked for more than 120 seconds.[/FONT][FONT=Menlo][487201.436597]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.436799] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.436995] jbd2/dm-0-8     D ffff880dffc53180     0   387      2 0x00000000[/FONT]
[FONT=Menlo][487201.437000]  ffff880036679bd8 0000000000000046 ffff880dafb2e000 ffff880036679fd8[/FONT]
[FONT=Menlo][487201.437368]  0000000000013180 0000000000013180 ffff880dafb2e000 ffff880dffc53a18[/FONT]
[FONT=Menlo][487201.437370]  ffff880dfff7a750 0000000000000002 ffffffff811efa80 ffff880036679c50[/FONT]
[FONT=Menlo][487201.437373] Call Trace:[/FONT]
[FONT=Menlo][487201.440026]  [<ffffffff811efa80>] ? generic_block_bmap+0x50/0x50[/FONT]
[FONT=Menlo][487201.441691]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][487201.441697]  [<ffffffff811efa8e>] sleep_on_buffer+0xe/0x20[/FONT]
[FONT=Menlo][487201.441699]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][487201.441701]  [<ffffffff811efa80>] ? generic_block_bmap+0x50/0x50[/FONT]
[FONT=Menlo][487201.441703]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][487201.442411]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][487201.442423]  [<ffffffff811f0dba>] __wait_on_buffer+0x2a/0x30[/FONT]
[FONT=Menlo][487201.443147]  [<ffffffff8128b520>] jbd2_journal_commit_transaction+0xf30/0x1ab0[/FONT]
[FONT=Menlo][487201.443161]  [<ffffffff8107562f>] ? try_to_del_timer_sync+0x4f/0x70[/FONT]
[FONT=Menlo][487201.443165]  [<ffffffff8129017d>] kjournald2+0xbd/0x250[/FONT]
[FONT=Menlo][487201.443167]  [<ffffffff810ab2c0>] ? prepare_to_wait_event+0x100/0x100[/FONT]
[FONT=Menlo][487201.443169]  [<ffffffff812900c0>] ? commit_timeout+0x10/0x10[/FONT]
[FONT=Menlo][487201.443185]  [<ffffffff8108b702>] kthread+0xd2/0xf0[/FONT]
[FONT=Menlo][487201.443187]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0[/FONT]
[FONT=Menlo][487201.443195]  [<ffffffff81733ca8>] ret_from_fork+0x58/0x90[/FONT]
[FONT=Menlo][487201.443197]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0[/FONT]
[FONT=Menlo][487201.443225] INFO: task mysqld:13509 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][487201.443432]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.443656] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.443871] mysqld          D ffff880dffc53180     0 13509  13057 0x00000000[/FONT]
[FONT=Menlo][487201.443875]  ffff88065d057e50 0000000000000086 ffff880db4398000 ffff88065d057fd8[/FONT]
[FONT=Menlo][487201.443877]  0000000000013180 0000000000013180 ffff880db4398000 ffff880daf365000[/FONT]
[FONT=Menlo][487201.443880]  0000000000261607 ffff880daf365088 ffff880daf365024 ffff88065d057e90[/FONT]
[FONT=Menlo][487201.443882] Call Trace:[/FONT]
[FONT=Menlo][487201.443884]  [<ffffffff81727699>] schedule+0x29/0x70[/FONT]
[FONT=Menlo][487201.443887]  [<ffffffff8128fba5>] jbd2_log_wait_commit+0x95/0x100[/FONT]
[FONT=Menlo][487201.443907]  [<ffffffff810ab2c0>] ? prepare_to_wait_event+0x100/0x100[/FONT]
[FONT=Menlo][487201.443911]  [<ffffffff81292143>] jbd2_complete_transaction+0x53/0xa0[/FONT]
[FONT=Menlo][487201.443930]  [<ffffffff8123c5f2>] ext4_sync_file+0x292/0x320[/FONT]
[FONT=Menlo][487201.443933]  [<ffffffff811ee261>] do_fsync+0x51/0x80[/FONT]
[FONT=Menlo][487201.443935]  [<ffffffff811ee4f0>] SyS_fsync+0x10/0x20[/FONT]
[FONT=Menlo][487201.443937]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]
[FONT=Menlo][487201.443946] INFO: task mysqld:13756 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][487201.444183]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.444404] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.448294] mysqld          D ffff880dffdd3180     0 13756  13057 0x00000000[/FONT]
[FONT=Menlo][487201.448303]  ffff8801615ffac8 0000000000000086 ffff880daf67b000 ffff8801615fffd8[/FONT]
[FONT=Menlo][487201.448306]  0000000000013180 0000000000013180 ffff880daf67b000 ffff880dffdd3a18[/FONT]
[FONT=Menlo][487201.448308]  ffff880dfff8eb68 0000000000000002 ffffffff81287da0 ffff8801615ffb40[/FONT]
[FONT=Menlo][487201.448311] Call Trace:[/FONT]
[FONT=Menlo][487201.448320]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.448324]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][487201.448326]  [<ffffffff81287dae>] sleep_on_shadow_bh+0xe/0x20[/FONT]
[FONT=Menlo][487201.448328]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][487201.448349]  [<ffffffff811d21a7>] ? poll_freewait+0x47/0xa0[/FONT]
[FONT=Menlo][487201.448352]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.448354]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][487201.448358]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][487201.448360]  [<ffffffff810aaed5>] ? wake_up_bit+0x25/0x30[/FONT]
[FONT=Menlo][487201.448363]  [<ffffffff812891cd>] do_get_write_access+0x2ad/0x4f0[/FONT]
[FONT=Menlo][487201.448375]  [<ffffffff811f138d>] ? __getblk+0x2d/0x2e0[/FONT]
[FONT=Menlo][487201.448378]  [<ffffffff81289437>] jbd2_journal_get_write_access+0x27/0x40[/FONT]
[FONT=Menlo][487201.448389]  [<ffffffff8126f13b>] __ext4_journal_get_write_access+0x3b/0x80[/FONT]
[FONT=Menlo][487201.448393]  [<ffffffff812443b0>] ext4_reserve_inode_write+0x70/0xa0[/FONT]
[FONT=Menlo][487201.448395]  [<ffffffff812476a0>] ? ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.448397]  [<ffffffff81244424>] ext4_mark_inode_dirty+0x44/0x1f0[/FONT]
[FONT=Menlo][487201.448399]  [<ffffffff812476a0>] ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.448402]  [<ffffffff811e7faa>] __mark_inode_dirty+0x10a/0x2d0[/FONT]
[FONT=Menlo][487201.448405]  [<ffffffff811d8fb1>] update_time+0x81/0xd0[/FONT]
[FONT=Menlo][487201.448408]  [<ffffffff811d90f9>] touch_atime+0xf9/0x140[/FONT]
[FONT=Menlo][487201.448820]  [<ffffffff81151d88>] generic_file_aio_read+0x598/0x700[/FONT]
[FONT=Menlo][487201.448842]  [<ffffffff811bda1a>] do_sync_read+0x5a/0x90[/FONT]
[FONT=Menlo][487201.448852]  [<ffffffff811be0b5>] vfs_read+0x95/0x160[/FONT]
[FONT=Menlo][487201.448862]  [<ffffffff811bed32>] SyS_pread64+0x72/0xb0[/FONT]
[FONT=Menlo][487201.448873]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]
[FONT=Menlo][487201.448900] INFO: task mysqld:2653 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][487201.449181]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.449390] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.453134] mysqld          D ffff880dffc93180     0  2653  13057 0x00000000[/FONT]
[FONT=Menlo][487201.453137]  ffff8808cf3cda18 0000000000000086 ffff8808cf85b000 ffff8808cf3cdfd8[/FONT]
[FONT=Menlo][487201.453147]  0000000000013180 0000000000013180 ffff8808cf85b000 ffff880dffc93a18[/FONT]
[FONT=Menlo][487201.453150]  ffff880dfff82dc0 0000000000000002 ffffffff81287da0 ffff8808cf3cda90[/FONT]
[FONT=Menlo][487201.453152] Call Trace:[/FONT]
[FONT=Menlo][487201.453155]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.453157]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][487201.453159]  [<ffffffff81287dae>] sleep_on_shadow_bh+0xe/0x20[/FONT]
[FONT=Menlo][487201.453161]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][487201.453163]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.453165]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][487201.453168]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][487201.453170]  [<ffffffff810aaed5>] ? wake_up_bit+0x25/0x30[/FONT]
[FONT=Menlo][487201.453172]  [<ffffffff812891cd>] do_get_write_access+0x2ad/0x4f0[/FONT]
[FONT=Menlo][487201.453174]  [<ffffffff811f138d>] ? __getblk+0x2d/0x2e0[/FONT]
[FONT=Menlo][487201.453177]  [<ffffffff81289437>] jbd2_journal_get_write_access+0x27/0x40[/FONT]
[FONT=Menlo][487201.453179]  [<ffffffff8126f13b>] __ext4_journal_get_write_access+0x3b/0x80[/FONT]
[FONT=Menlo][487201.453198]  [<ffffffff812443b0>] ext4_reserve_inode_write+0x70/0xa0[/FONT]
[FONT=Menlo][487201.453201]  [<ffffffff812476a0>] ? ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.453203]  [<ffffffff81244424>] ext4_mark_inode_dirty+0x44/0x1f0[/FONT]
[FONT=Menlo][487201.453206]  [<ffffffff812476a0>] ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.453208]  [<ffffffff811e7faa>] __mark_inode_dirty+0x10a/0x2d0[/FONT]
[FONT=Menlo][487201.453210]  [<ffffffff811d8fb1>] update_time+0x81/0xd0[/FONT]
[FONT=Menlo][487201.453212]  [<ffffffff811d91c0>] file_update_time+0x80/0xd0[/FONT]
[FONT=Menlo][487201.453214]  [<ffffffff81150cf0>] __generic_file_aio_write+0x180/0x3d0[/FONT]
[FONT=Menlo][487201.453215]  [<ffffffff81150f98>] generic_file_aio_write+0x58/0xa0[/FONT]
[FONT=Menlo][487201.453218]  [<ffffffff8123c012>] ext4_file_write+0xa2/0x3f0[/FONT]
[FONT=Menlo][487201.453220]  [<ffffffff811bdaaa>] do_sync_write+0x5a/0x90[/FONT]
[FONT=Menlo][487201.453222]  [<ffffffff811be234>] vfs_write+0xb4/0x1f0[/FONT]
[FONT=Menlo][487201.453224]  [<ffffffff811bede2>] SyS_pwrite64+0x72/0xb0[/FONT]
[FONT=Menlo][487201.453226]  [<ffffffff811bebfc>] ? SyS_read+0x7c/0xa0[/FONT]
[FONT=Menlo][487201.453228]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]
[FONT=Menlo][487201.453231] INFO: task mysqld:2663 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][487201.453463]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.453729] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.453935] mysqld          D ffff880dffdd3180     0  2663  13057 0x00000000[/FONT]
[FONT=Menlo][487201.453938]  ffff880636d1fa28 0000000000000086 ffff880c531c8000 ffff880636d1ffd8[/FONT]
[FONT=Menlo][487201.453941]  0000000000013180 0000000000013180 ffff880c531c8000 ffff880dffdd3a18[/FONT]
[FONT=Menlo][487201.453943]  ffff880dfff87b60 0000000000000002 ffffffff81287da0 ffff880636d1faa0[/FONT]
[FONT=Menlo][487201.453945] Call Trace:[/FONT]
[FONT=Menlo][487201.453948]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.453950]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][487201.453952]  [<ffffffff81287dae>] sleep_on_shadow_bh+0xe/0x20[/FONT]
[FONT=Menlo][487201.453954]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][487201.453956]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.453958]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][487201.453961]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][487201.453963]  [<ffffffff810aaed5>] ? wake_up_bit+0x25/0x30[/FONT]
[FONT=Menlo][487201.453965]  [<ffffffff812891cd>] do_get_write_access+0x2ad/0x4f0[/FONT]
[FONT=Menlo][487201.453967]  [<ffffffff811f138d>] ? __getblk+0x2d/0x2e0[/FONT]
[FONT=Menlo][487201.453975]  [<ffffffff81289437>] jbd2_journal_get_write_access+0x27/0x40[/FONT]
[FONT=Menlo][487201.453978]  [<ffffffff8126f13b>] __ext4_journal_get_write_access+0x3b/0x80[/FONT]
[FONT=Menlo][487201.453981]  [<ffffffff812443b0>] ext4_reserve_inode_write+0x70/0xa0[/FONT]
[FONT=Menlo][487201.453983]  [<ffffffff812476a0>] ? ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.453986]  [<ffffffff81244424>] ext4_mark_inode_dirty+0x44/0x1f0[/FONT]
[FONT=Menlo][487201.453988]  [<ffffffff812476a0>] ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.453990]  [<ffffffff811e7faa>] __mark_inode_dirty+0x10a/0x2d0[/FONT]
[FONT=Menlo][487201.453992]  [<ffffffff811d8fb1>] update_time+0x81/0xd0[/FONT]
[FONT=Menlo][487201.453995]  [<ffffffff811d91c0>] file_update_time+0x80/0xd0[/FONT]
[FONT=Menlo][487201.453997]  [<ffffffff81150cf0>] __generic_file_aio_write+0x180/0x3d0[/FONT]
[FONT=Menlo][487201.453999]  [<ffffffff81150f98>] generic_file_aio_write+0x58/0xa0[/FONT]
[FONT=Menlo][487201.454001]  [<ffffffff8123c012>] ext4_file_write+0xa2/0x3f0[/FONT]
[FONT=Menlo][487201.454004]  [<ffffffff8172f334>] ? __do_page_fault+0x204/0x570[/FONT]
[FONT=Menlo][487201.454007]  [<ffffffff811bdaaa>] do_sync_write+0x5a/0x90[/FONT]
[FONT=Menlo][487201.454009]  [<ffffffff811be234>] vfs_write+0xb4/0x1f0[/FONT]
[FONT=Menlo][487201.454011]  [<ffffffff811bec69>] SyS_write+0x49/0xa0[/FONT]
[FONT=Menlo][487201.454014]  [<ffffffff81089de1>] ? posix_clock_realtime_get+0x11/0x20[/FONT]
[FONT=Menlo][487201.454017]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]
[FONT=Menlo][487201.454031] INFO: task mysqld:23444 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][487201.454253]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.454475] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.454692] mysqld          D ffff880dffdd3180     0 23444  13057 0x00000000[/FONT]
[FONT=Menlo][487201.454695]  ffff880bcc5adac8 0000000000000086 ffff880a8efbe000 ffff880bcc5adfd8[/FONT]
[FONT=Menlo][487201.454698]  0000000000013180 0000000000013180 ffff880a8efbe000 ffff880dffdd3a18[/FONT]
[FONT=Menlo][487201.454700]  ffff880dfff89a80 0000000000000002 ffffffff81287da0 ffff880bcc5adb40[/FONT]
[FONT=Menlo][487201.454702] Call Trace:[/FONT]
[FONT=Menlo][487201.454716]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.454719]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][487201.454721]  [<ffffffff81287dae>] sleep_on_shadow_bh+0xe/0x20[/FONT]
[FONT=Menlo][487201.454723]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][487201.454725]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.454726]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][487201.454729]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][487201.454731]  [<ffffffff810aaed5>] ? wake_up_bit+0x25/0x30[/FONT]
[FONT=Menlo][487201.454733]  [<ffffffff812891cd>] do_get_write_access+0x2ad/0x4f0[/FONT]
[FONT=Menlo][487201.454735]  [<ffffffff811f138d>] ? __getblk+0x2d/0x2e0[/FONT]
[FONT=Menlo][487201.454738]  [<ffffffff81289437>] jbd2_journal_get_write_access+0x27/0x40[/FONT]
[FONT=Menlo][487201.454740]  [<ffffffff8126f13b>] __ext4_journal_get_write_access+0x3b/0x80[/FONT]
[FONT=Menlo][487201.454742]  [<ffffffff812443b0>] ext4_reserve_inode_write+0x70/0xa0[/FONT]
[FONT=Menlo][487201.454744]  [<ffffffff812476a0>] ? ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.454746]  [<ffffffff81244424>] ext4_mark_inode_dirty+0x44/0x1f0[/FONT]
[FONT=Menlo][487201.454748]  [<ffffffff812476a0>] ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.454750]  [<ffffffff811e7faa>] __mark_inode_dirty+0x10a/0x2d0[/FONT]
[FONT=Menlo][487201.454752]  [<ffffffff811d8fb1>] update_time+0x81/0xd0[/FONT]
[FONT=Menlo][487201.454754]  [<ffffffff811d90f9>] touch_atime+0xf9/0x140[/FONT]
[FONT=Menlo][487201.454756]  [<ffffffff81151d88>] generic_file_aio_read+0x598/0x700[/FONT]
[FONT=Menlo][487201.454758]  [<ffffffff811bda1a>] do_sync_read+0x5a/0x90[/FONT]
[FONT=Menlo][487201.454760]  [<ffffffff811be0b5>] vfs_read+0x95/0x160[/FONT]
[FONT=Menlo][487201.454762]  [<ffffffff811bed32>] SyS_pread64+0x72/0xb0[/FONT]
[FONT=Menlo][487201.454764]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]
[FONT=Menlo][487201.454851] INFO: task kworker/u16:2:23308 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][487201.455036]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][487201.455259] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][487201.455472] kworker/u16:2   D ffff880dffc93180     0 23308      2 0x00000000[/FONT]
[FONT=Menlo][487201.455479] Workqueue: writeback bdi_writeback_workfn (flush-252:0)[/FONT]
[FONT=Menlo][487201.455481]  ffff8802e3d55688 0000000000000046 ffff880db2210000 ffff8802e3d55fd8[/FONT]
[FONT=Menlo][487201.455483]  0000000000013180 0000000000013180 ffff880db2210000 ffff880dffc93a18[/FONT]
[FONT=Menlo][487201.455486]  ffff880dfff87b60 0000000000000002 ffffffff81287da0 ffff8802e3d55700[/FONT]
[FONT=Menlo][487201.455488] Call Trace:[/FONT]
[FONT=Menlo][487201.455491]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.455493]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][487201.455495]  [<ffffffff81287dae>] sleep_on_shadow_bh+0xe/0x20[/FONT]
[FONT=Menlo][487201.455496]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][487201.455525]  [<ffffffff81287da0>] ? start_this_handle+0x590/0x590[/FONT]
[FONT=Menlo][487201.455528]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][487201.455531]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][487201.455533]  [<ffffffff810aaed5>] ? wake_up_bit+0x25/0x30[/FONT]
[FONT=Menlo][487201.455536]  [<ffffffff812891cd>] do_get_write_access+0x2ad/0x4f0[/FONT]
[FONT=Menlo][487201.455538]  [<ffffffff811f138d>] ? __getblk+0x2d/0x2e0[/FONT]
[FONT=Menlo][487201.455541]  [<ffffffff81289437>] jbd2_journal_get_write_access+0x27/0x40[/FONT]
[FONT=Menlo][487201.455544]  [<ffffffff8126f13b>] __ext4_journal_get_write_access+0x3b/0x80[/FONT]
[FONT=Menlo][487201.455546]  [<ffffffff812443b0>] ext4_reserve_inode_write+0x70/0xa0[/FONT]
[FONT=Menlo][487201.455548]  [<ffffffff812476a0>] ? ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.455550]  [<ffffffff81244424>] ext4_mark_inode_dirty+0x44/0x1f0[/FONT]
[FONT=Menlo][487201.455552]  [<ffffffff812476a0>] ext4_dirty_inode+0x40/0x60[/FONT]
[FONT=Menlo][487201.455554]  [<ffffffff811e7faa>] __mark_inode_dirty+0x10a/0x2d0[/FONT]
[FONT=Menlo][487201.455557]  [<ffffffff812416f5>] ext4_da_update_reserve_space+0x1b5/0x270[/FONT]
[FONT=Menlo][487201.455559]  [<ffffffff8126dc36>] ext4_ext_map_blocks+0xc16/0xf40[/FONT]
[FONT=Menlo][487201.455562]  [<ffffffff81241a85>] ext4_map_blocks+0x2d5/0x540[/FONT]
[FONT=Menlo][487201.455564]  [<ffffffff81244bef>] ext4_writepages+0x61f/0xd20[/FONT]
[FONT=Menlo][487201.455577]  [<ffffffff8115b76e>] do_writepages+0x1e/0x40[/FONT]
[FONT=Menlo][487201.455580]  [<ffffffff811e7930>] __writeback_single_inode+0x40/0x220[/FONT]
[FONT=Menlo][487201.455582]  [<ffffffff811e86f7>] writeback_sb_inodes+0x247/0x3e0[/FONT]
[FONT=Menlo][487201.455584]  [<ffffffff811e892f>] __writeback_inodes_wb+0x9f/0xd0[/FONT]
[FONT=Menlo][487201.455586]  [<ffffffff811e8ba3>] wb_writeback+0x243/0x2c0[/FONT]
[FONT=Menlo][487201.455606]  [<ffffffff811ea479>] bdi_writeback_workfn+0x1b9/0x430[/FONT]
[FONT=Menlo][487201.455611]  [<ffffffff81083b22>] process_one_work+0x182/0x450[/FONT]
[FONT=Menlo][487201.455613]  [<ffffffff81084911>] worker_thread+0x121/0x410[/FONT]
[FONT=Menlo][487201.455615]  [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430[/FONT]
[FONT=Menlo][487201.455618]  [<ffffffff8108b702>] kthread+0xd2/0xf0[/FONT]
[FONT=Menlo][487201.455620]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0[/FONT]
[FONT=Menlo][487201.455622]  [<ffffffff81733ca8>] ret_from_fork+0x58/0x90[/FONT]
[FONT=Menlo][487201.455624]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0[/FONT]
[FONT=Menlo][659881.512855] INFO: task jbd2/dm-0-8:387 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][659881.514946]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][659881.515114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][659881.515265] jbd2/dm-0-8     D ffff880dffd13180     0   387      2 0x00000000[/FONT]
[FONT=Menlo][659881.515270]  ffff880036679bd8 0000000000000046 ffff880dafb2e000 ffff880036679fd8[/FONT]
[FONT=Menlo][659881.515279]  0000000000013180 0000000000013180 ffff880dafb2e000 ffff880dffd13a18[/FONT]
[FONT=Menlo][659881.515281]  ffff880dfff93170 0000000000000002 ffffffff811efa80 ffff880036679c50[/FONT]
[FONT=Menlo][659881.515284] Call Trace:[/FONT]
[FONT=Menlo][659881.515682]  [<ffffffff811efa80>] ? generic_block_bmap+0x50/0x50[/FONT]
[FONT=Menlo][659881.515703]  [<ffffffff817279bd>] io_schedule+0x9d/0x140[/FONT]
[FONT=Menlo][659881.515707]  [<ffffffff811efa8e>] sleep_on_buffer+0xe/0x20[/FONT]
[FONT=Menlo][659881.515709]  [<ffffffff81727e42>] __wait_on_bit+0x62/0x90[/FONT]
[FONT=Menlo][659881.515711]  [<ffffffff811efa80>] ? generic_block_bmap+0x50/0x50[/FONT]
[FONT=Menlo][659881.515713]  [<ffffffff81727ee7>] out_of_line_wait_on_bit+0x77/0x90[/FONT]
[FONT=Menlo][659881.515730]  [<ffffffff810ab300>] ? autoremove_wake_function+0x40/0x40[/FONT]
[FONT=Menlo][659881.515736]  [<ffffffff811f0dba>] __wait_on_buffer+0x2a/0x30[/FONT]
[FONT=Menlo][659881.515756]  [<ffffffff8128b520>] jbd2_journal_commit_transaction+0xf30/0x1ab0[/FONT]
[FONT=Menlo][659881.515765]  [<ffffffff8107562f>] ? try_to_del_timer_sync+0x4f/0x70[/FONT]
[FONT=Menlo][659881.515768]  [<ffffffff8129017d>] kjournald2+0xbd/0x250[/FONT]
[FONT=Menlo][659881.515770]  [<ffffffff810ab2c0>] ? prepare_to_wait_event+0x100/0x100[/FONT]
[FONT=Menlo][659881.515772]  [<ffffffff812900c0>] ? commit_timeout+0x10/0x10[/FONT]
[FONT=Menlo][659881.515783]  [<ffffffff8108b702>] kthread+0xd2/0xf0[/FONT]
[FONT=Menlo][659881.515785]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0[/FONT]
[FONT=Menlo][659881.515791]  [<ffffffff81733ca8>] ret_from_fork+0x58/0x90[/FONT]
[FONT=Menlo][659881.515793]  [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0[/FONT]
[FONT=Menlo][659881.515807] INFO: task mysqld:13509 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][659881.515953]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][659881.516499] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][659881.516682] mysqld          D ffff880dffc53180     0 13509  13057 0x00000000[/FONT]
[FONT=Menlo][659881.516685]  ffff88065d057e50 0000000000000086 ffff880db4398000 ffff88065d057fd8[/FONT]
[FONT=Menlo][659881.516688]  0000000000013180 0000000000013180 ffff880db4398000 ffff880daf365000[/FONT]
[FONT=Menlo][659881.516690]  00000000002ca26c ffff880daf365088 ffff880daf365024 ffff88065d057e90[/FONT]
[FONT=Menlo][659881.516692] Call Trace:[/FONT]
[FONT=Menlo][659881.516695]  [<ffffffff81727699>] schedule+0x29/0x70[/FONT]
[FONT=Menlo][659881.516697]  [<ffffffff8128fba5>] jbd2_log_wait_commit+0x95/0x100[/FONT]
[FONT=Menlo][659881.516700]  [<ffffffff810ab2c0>] ? prepare_to_wait_event+0x100/0x100[/FONT]
[FONT=Menlo][659881.516702]  [<ffffffff81292143>] jbd2_complete_transaction+0x53/0xa0[/FONT]
[FONT=Menlo][659881.516717]  [<ffffffff8123c5f2>] ext4_sync_file+0x292/0x320[/FONT]
[FONT=Menlo][659881.516719]  [<ffffffff811ee261>] do_fsync+0x51/0x80[/FONT]
[FONT=Menlo][659881.516721]  [<ffffffff811ee4f0>] SyS_fsync+0x10/0x20[/FONT]
[FONT=Menlo][659881.516723]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]
[FONT=Menlo][659881.516726] INFO: task mysqld:13511 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo][659881.517480]       Not tainted 3.13.0-55-generic #94-Ubuntu[/FONT]
[FONT=Menlo][659881.517620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo][659881.517769] mysqld          D ffff880dffc53180     0 13511  13057 0x00000000[/FONT]
[FONT=Menlo][659881.517772]  ffff880daf047e50 0000000000000086 ffff880db3cc1800 ffff880daf047fd8[/FONT]
[FONT=Menlo][659881.517774]  0000000000013180 0000000000013180 ffff880db3cc1800 ffff880daf365000[/FONT]
[FONT=Menlo][659881.517777]  00000000002ca26d ffff880daf365088 ffff880daf365024 ffff880daf047e90[/FONT]
[FONT=Menlo][659881.517779] Call Trace:[/FONT]
[FONT=Menlo][659881.517781]  [<ffffffff81727699>] schedule+0x29/0x70[/FONT]
[FONT=Menlo][659881.517796]  [<ffffffff8128fba5>] jbd2_log_wait_commit+0x95/0x100[/FONT]
[FONT=Menlo][659881.517799]  [<ffffffff810ab2c0>] ? prepare_to_wait_event+0x100/0x100[/FONT]
[FONT=Menlo][659881.517801]  [<ffffffff81292143>] jbd2_complete_transaction+0x53/0xa0[/FONT]
[FONT=Menlo][659881.517803]  [<ffffffff8123c5f2>] ext4_sync_file+0x292/0x320[/FONT]
[FONT=Menlo][659881.517805]  [<ffffffff811ee261>] do_fsync+0x51/0x80[/FONT]
[FONT=Menlo][659881.517807]  [<ffffffff811ee4f0>] SyS_fsync+0x10/0x20[/FONT]
[FONT=Menlo][659881.517809]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f[/FONT]

Miktash · Jul 8, 2015

And the error I got on the host:

Code:

[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT][FONT=Menlo]      Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 0000000000000000 0000000000000000[/FONT]
[FONT=Menlo]0000000000000001 ffff881029e7ad00 ffff880e4f6efa08 ffffffff810097ef[/FONT]
[FONT=Menlo]00028f8ad9e0288a ffff88002829de00 000000012af19919 00000000000005a7[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff810097ef>] ? __switch_to+0x1df/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05edd0f>] ? nfs_write_begin+0x8f/0x1f0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff880e4f6ef9a8 00000001810adf88[/FONT]
[FONT=Menlo]ffff88002839de00 ffff881029f36f80 ffff880e4f6efa08 ffffffff810097ef[/FONT]
[FONT=Menlo]ffffffffa05ef490 0000000000000006 000000012af9f146 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff810097ef>] ? __switch_to+0x1df/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05edd0f>] ? nfs_write_begin+0x8f/0x1f0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff880e4f6ef9a8 00000001810adf88[/FONT]
[FONT=Menlo]ffff88002839de00 ffff881029f36f80 ffff880e4f6efa08 ffffffff810097ef[/FONT]
[FONT=Menlo]ffffffffa05ef490 0000000000000006 000000012af9f146 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff810097ef>] ? __switch_to+0x1df/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05edd0f>] ? nfs_write_begin+0x8f/0x1f0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff8810095121c0 0000000000000000[/FONT]
[FONT=Menlo]0000000000000000 ffff8800282db940 ffff880e4f6efa08 ffffffff8100989e[/FONT]
[FONT=Menlo]ffff8800282dde00 ffff8800282dde00 0000000000000003 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff8100989e>] ? __switch_to+0x28e/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05edd0f>] ? nfs_write_begin+0x8f/0x1f0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff811c5f76>] ? pollwake+0x66/0x70[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff811b9356>] ? pipe_read+0x6/0x4c0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff81138dba>] ? fire_user_return_notifiers+0x3a/0x50[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8156556e>] ? do_device_not_available+0xe/0x10[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff8810095121c0 0000000000000000[/FONT]
[FONT=Menlo]0000000000000000 ffff8800282db940 ffff880e4f6efa08 ffffffff8100989e[/FONT]
[FONT=Menlo]ffff8800282dde00 ffff8800282dde00 0000000000000003 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff8100989e>] ? __switch_to+0x28e/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05edd0f>] ? nfs_write_begin+0x8f/0x1f0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff811c5f76>] ? pollwake+0x66/0x70[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff811b9356>] ? pipe_read+0x6/0x4c0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff81138dba>] ? fire_user_return_notifiers+0x3a/0x50[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8156556e>] ? do_device_not_available+0xe/0x10[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff8810095121c0 0000000000000000[/FONT]
[FONT=Menlo]0000000000000000 ffff8800282db940 ffff880e4f6efa08 ffffffff8100989e[/FONT]
[FONT=Menlo]ffff8800282dde00 ffff8800282dde00 0000000000000003 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff8100989e>] ? __switch_to+0x28e/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05edd0f>] ? nfs_write_begin+0x8f/0x1f0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff811c5f76>] ? pollwake+0x66/0x70[/FONT]
[FONT=Menlo][<ffffffff811b8a72>] ? pipe_lock+0x12/0x20[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff811b9356>] ? pipe_read+0x6/0x4c0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff81138dba>] ? fire_user_return_notifiers+0x3a/0x50[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8156556e>] ? do_device_not_available+0xe/0x10[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff8810095121c0 0000000000000000[/FONT]
[FONT=Menlo]0000000000000000 ffff88002829b940 ffff880e4f6efa08 ffffffff8100989e[/FONT]
[FONT=Menlo]ffff88002829de00 ffff88002829de00 0000000000000002 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff8100989e>] ? __switch_to+0x28e/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]
[FONT=Menlo]INFO: task lzop:632696 blocked for more than 120 seconds.[/FONT]
[FONT=Menlo]     Tainted: P           ---------------    2.6.32-39-pve #1[/FONT]
[FONT=Menlo]"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.[/FONT]
[FONT=Menlo]lzop          D ffff8810095121c0     0 632696 632686    0 0x00000000[/FONT]
[FONT=Menlo]ffff880e4f6efa48 0000000000000086 ffff8810095121c0 0000000000000000[/FONT]
[FONT=Menlo]0000000000000000 ffff88002829b940 ffff880e4f6efa08 ffffffff8100989e[/FONT]
[FONT=Menlo]ffff88002829de00 ffff88002829de00 0000000000000002 ffff881009512878[/FONT]
[FONT=Menlo]Call Trace:[/FONT]
[FONT=Menlo][<ffffffff8100989e>] ? __switch_to+0x28e/0x2f0[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81561823>] io_schedule+0x73/0xc0[/FONT]
[FONT=Menlo][<ffffffffa05fa50e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff81562830>] __wait_on_bit+0x60/0x90[/FONT]
[FONT=Menlo][<ffffffffa05fa500>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs][/FONT]
[FONT=Menlo][<ffffffff815628dc>] out_of_line_wait_on_bit+0x7c/0x90[/FONT]
[FONT=Menlo][<ffffffffa05ef490>] ? nfs_wait_bit_killable+0x0/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffff810a6980>] ? wake_bit_function+0x0/0x50[/FONT]
[FONT=Menlo][<ffffffffa05faa5f>] nfs_wait_on_request+0x2f/0x40 [nfs][/FONT]
[FONT=Menlo][<ffffffffa06011b2>] nfs_updatepage+0x122/0x530 [nfs][/FONT]
[FONT=Menlo][<ffffffffa05eea48>] nfs_write_end+0x138/0x2e0 [nfs][/FONT]
[FONT=Menlo][<ffffffff8113a6a5>] generic_file_buffered_write_iter+0x175/0x2a0[/FONT]
[FONT=Menlo][<ffffffff8113abd5>] __generic_file_write_iter+0x215/0x3e0[/FONT]
[FONT=Menlo][<ffffffff8113ae23>] __generic_file_aio_write+0x83/0xa0[/FONT]
[FONT=Menlo][<ffffffff8113aec8>] generic_file_aio_write+0x88/0x100[/FONT]
[FONT=Menlo][<ffffffffa05ee32e>] nfs_file_write+0x11e/0x220 [nfs][/FONT]
[FONT=Menlo][<ffffffff811ad76c>] do_sync_write+0xec/0x140[/FONT]
[FONT=Menlo][<ffffffff811adfa1>] vfs_write+0xa1/0x190[/FONT]
[FONT=Menlo][<ffffffff811ae2fa>] sys_write+0x4a/0x90[/FONT]
[FONT=Menlo][<ffffffff8100b182>] system_call_fastpath+0x16/0x1b[/FONT]

mstrent · Jul 8, 2015

Perhaps some similar issues:

http://forum.proxmox.com/threads/20372-Linux-guest-problems-on-new-Haswell-EP-processors
http://forum.proxmox.com/threads/22488-VM-blocked-due-to-hung_task_timeout_secs
http://forum.proxmox.com/threads/22573-NFS-freezes-Proxmox-WebGUI-and-fails-to-backup

What make/model of server, and can you post "pveversion -v" and .conf from /etc/pve/qemu-server?

Miktash · Jul 8, 2015

pveversion:

# pveversion -v
proxmox-ve-2.6.32: 3.4-156 (running kernel: 2.6.32-39-pve)
pve-manager: 3.4-6 (running version: 3.4-6/102d4547)
pve-kernel-2.6.32-39-pve: 2.6.32-156
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-17
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

.conf file:

bootdisk: virtio0
cores: 8
ide2: none,media=cdrom
memory: 56320
name: DB
net0: virtio=B2

6

7:8C:47:4D,bridge=vmbr20
net1: virtio=16:97:E3:31:73:63,bridge=vmbr60
numa: 0
onboot: 1
ostype: l26
smbios1: uuid=b2bfa821-9cdb-45a7-bef5-b27b43b5b328
sockets: 1
virtio0: LOCAL-RAID10-STORAGE:122/vm-122-disk-1.raw,format=raw,size=15G
virtio1: LOCAL-RAID10-STORAGE:122/vm-122-disk-2.raw,format=raw,size=400G

Miktash · Jul 8, 2015

Looks like proxmox 3.4 is still running a 2.6 kernel?

Isn't there a 3.X or 4.1 kernel available?

I remember having problems with a server a couple years ago. That server also had unexplained "blocked for more than 120 seconds" errors/warnings.
It went away by upgrading to a more recent kernel. Maybe the issue we're having right now is related.. I don't know...

I'm actually surprised proxmox 3.4 is still on a 2.6 kernel. I never bothered to check before...

spirit · Jul 8, 2015

Miktash said:
Isn't there a 3.X or 4.1 kernel available?

yes, kernel 3.10 is available (it's missing openvz support,that's why it's not the default)

#apt-get install pve-kernel-3.10.0-10-pve

spirit · Jul 8, 2015

Seem to be simply pending ios on the nfs client fo rmore than 120s.

what is your nfs client mount options ?

what is you nfs server ?

Miktash · Jul 8, 2015

If they are pending ios on the nfs client, then why is it affecting my VM?
The VM is running from local storage (software raid10 on 4 ssd's). The VM isn't using NFS. Not inside the VM and not for it's VM image.

The only NFS mount point there is is to the backup server which is running FreeNAS.
It's mountpoint options are:

10.10.10.103:/mnt/storage/VMs on /mnt/pve/Backup01 type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.10.103,mountvers=3,mountport=656,mountproto=udp,local_lock=none,addr=10.10.10.103)

Miktash · Jul 8, 2015

I'm planning on giving the 3.10 kernel a try, unless it's something nfs related like you said..
But I don't understand how that could interrupt the VM...

Miktash · Jul 8, 2015

Now while I'm thinking about it.. Is it the KVM process that's actually doing the backups? Or is it a separate process? I guess it's the KVM process (as it needs a snapshot for backups)

If it's the same process, then I can understand that if that process is having troubles accessing the NFS server it somehow gets interrupted/stalled resulting in VM troubles.

But then again.. what's going on

I'm thinking it might still be a problem with the switch. Maybe I should replace the switch and see if it helps?
Or replace FreeNAS with something else...

mstrent · Jul 8, 2015

I'm starting to wonder if my similar issues are NFS related as well. I'm not getting any errors on the host like you, but I do backups to an NFS mount, and crashes happen much more often during backups. I've un-mounted the NFS and haven't had any crashes since.

(Though I'm looking to move to OpenStack. To many issues with Proxmox lately and I need stability. It's really a shame; I really like Proxmox.)

spirit · Jul 9, 2015

About backup, I'll try to explain how they work exactly.

The "snapshot" backup mode, will do a backup of the state of the disk, at the state of the begin of backup start.

That mean that if a write occur in the vm, when the backup is running, the old block of the disk must be copy to the backup storage, before to be replace by the new write.

That mean that if you have a lot of new write, when backup is running, you need to have a fast backup slorage, or it'll slowdown writes.

(don't known how much write ios you have in your vm ?)

Code:

new write------------>old block------------>backup storage
              <----ack----newblock<----ack----

Miktash · Jul 9, 2015

Okay, that makes sense.
So, if the backup is having troubles (in my case with the NFS server, it seems) it's normal for the VM to have issues too, because the VM's writes are delayed...
So I'm pretty sure my problem is just the backups / nfs. But I'm not sure what can be the cause...

Miktash · Jul 9, 2015

@spirit, I think the problem is network related. Saturated gigabit port to the backupserver.

I read somewhere else that you have infiniband stuff in use? So I was wondering if you could help me out with that? I'm looking on ebay to find some (not too expensive) components to upgrade to 10Gigabit over infiniband.

I'm a bit confused here. Can infiniband be used natively for creating a 10G network or do I need to use something like IPoIB ?

spirit · Jul 10, 2015

Miktash said:
@spirit, I think the problem is network related. Saturated gigabit port to the backupserver.

I read somewhere else that you have infiniband stuff in use? So I was wondering if you could help me out with that? I'm looking on ebay to find some (not too expensive) components to upgrade to 10Gigabit over infiniband.

I'm a bit confused here. Can infiniband be used natively for creating a 10G network or do I need to use something like IPoIB ?

Hi, sorry , I don't use infiniband. I'm using mellanox ethernet switches (sx1012), 12port 40gbe or 48ports 10gbe with breakout cable. (around 4000€).

But yes, with infiniband, you need to use IPoIB (I think you can reach around 6-7gbit/s with ip over infiniband)

As workaround, maybe can you do backup locally on a temp drive, and do rsync after.

Search

Search

Task blocked for more than 120 seconds => VM file corruption?

Miktash

Active Member

Miktash

Active Member

Miktash

Active Member

mstrent

New Member

Miktash

Active Member

Miktash

Active Member

spirit

Distinguished Member

spirit

Distinguished Member

Miktash

Active Member

Miktash

Active Member

Miktash

Active Member

mstrent

New Member

spirit

Distinguished Member

Miktash

Active Member

Miktash

Active Member

spirit

Distinguished Member