qemu-nbd bug

[rootuser]

Hello everyone,

I have the following problem.

I setup a new KVM guest and added a qcow2 hdd. After that I mounted the qcow2 image on the host like this

qemu-nbd -c /dev/nbd0 /path/vm-image.qcow2

and tried to create a new filesystem like this

mkfs.ext4 /dev/nbd0

At this point the process got stuck and I saw this in the kern.log:

Feb 28 02:19:01 node1 kernel: nbd (pid 5708: qemu-nbd) got signal 9
Feb 28 02:21:16 node1 kernel: INFO: task qemu-nbd:5706 blocked for more than 120 seconds.
Feb 28 02:21:16 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 28 02:21:16 node1 kernel: qemu-nbd      D ffff881765516d90     0  5706      1    0 0x00000000
Feb 28 02:21:16 node1 kernel: ffff8817654f5ad8 0000000000000082 0000000000000000 ffff8817654f5dc0
Feb 28 02:21:16 node1 kernel: ffff8817654f5dc8 ffff8817654f5dd0 ffff8817654f5ea8 ffff8817654f5da8
Feb 28 02:21:16 node1 kernel: ffff8817654f5db0 ffff881765517340 ffff8817654f5fd8 ffff8817654f5fd8
Feb 28 02:21:16 node1 kernel: Call Trace:
Feb 28 02:21:16 node1 kernel: [<ffffffff81529085>] schedule_timeout+0x215/0x2e0
Feb 28 02:21:16 node1 kernel: [<ffffffff81030b4e>] ? physflat_send_IPI_mask+0xe/0x10
Feb 28 02:21:16 node1 kernel: [<ffffffff8102ade9>] ? native_smp_send_reschedule+0x49/0x60
Feb 28 02:21:16 node1 kernel: [<ffffffff81054a9d>] ? task_rq_lock+0x5d/0xa0
Feb 28 02:21:16 node1 kernel: [<ffffffff81528cf3>] wait_for_common+0x123/0x190
Feb 28 02:21:16 node1 kernel: [<ffffffff81059f10>] ? default_wake_function+0x0/0x20
Feb 28 02:21:16 node1 kernel: [<ffffffff810857c9>] ? signal_wake_up+0x39/0x40
Feb 28 02:21:16 node1 kernel: [<ffffffff81528e1d>] wait_for_completion+0x1d/0x20
Feb 28 02:21:16 node1 kernel: [<ffffffff8119ebb7>] do_coredump+0x3a7/0xbf0
Feb 28 02:21:16 node1 kernel: [<ffffffff81084835>] ? __sigqueue_free+0x45/0x50
Feb 28 02:21:16 node1 kernel: [<ffffffff810888ad>] get_signal_to_deliver+0x1ed/0x470
Feb 28 02:21:16 node1 kernel: [<ffffffff8100a335>] do_signal+0x75/0x800
Feb 28 02:21:16 node1 kernel: [<ffffffff8100984c>] ? __switch_to+0x1ac/0x320
Feb 28 02:21:16 node1 kernel: [<ffffffff81010000>] ? find_oprom+0x210/0x3b0
Feb 28 02:21:16 node1 kernel: [<ffffffff81043837>] ? is_prefetch+0xb7/0x240
Feb 28 02:21:16 node1 kernel: [<ffffffff8100ab50>] do_notify_resume+0x90/0xc0
Feb 28 02:21:16 node1 kernel: [<ffffffff8100bc1c>] retint_signal+0x48/0x8c
Feb 28 02:21:16 node1 kernel: INFO: task qemu-nbd:5708 blocked for more than 120 seconds.
Feb 28 02:21:16 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 28 02:21:16 node1 kernel: qemu-nbd      D ffff880ba3674b90     0  5708      1    0 0x00000000
...

Information about my server:

pve-manager: 2.2-32 (pve-manager/2.2/3089a616)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.2-83
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1

Please let me know if you need any more information.

 

[dietmar]

I setup a new KVM guest and added a qcow2 hdd. After that I mounted the qcow2 image on the host like this

While the VM is running?

 

[rootuser]

While the VM is running?

No, the VM was not running at this point.

 

[dietmar]

What version do you use:

# pveversion -v

 

[rootuser]

#pveversion -v

pve-manager: 2.2-32 (pve-manager/2.2/3089a616)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.2-83
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1

 

[docent]

Hi everyone,
I have exactly the same problem.

root@hq-vmc1-1:/var/lib/storages/csv1# qemu-nbd -c /dev/nbd0 disk.qcow2
root@hq-vmc1-1:/var/lib/storages/csv1# parted /dev/nbd0
GNU Parted 2.3
Using /dev/nbd0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print free
Error: /dev/nbd0: unrecognised disk label
(parted) mktable gpt
(parted) print free
Model: Unknown (unknown)
Disk /dev/nbd0: 55.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
        17.4kB  55.0TB  55.0TB  Free Space

(parted) mkpart primary xfs 1M 55T
(parted) quit
Information: You may need to update /etc/fstab.

root@hq-vmc1-1:/var/lib/storages/csv1# parted /dev/nbd0 print free
Model: Unknown (unknown)
Disk /dev/nbd0: 55.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
        17.4kB  1049kB  1031kB  Free Space
 1      1049kB  55.0TB  55.0TB               primary
        55.0TB  55.0TB  1032kB  Free Space

root@hq-vmc1-1:/var/lib/storages/csv1# partprobe
root@hq-vmc1-1:/mnt# mkfs.xfs /dev/nbd0p1
meta-data=/dev/nbd0p1            isize=256    agcount=50, agsize=268435455 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=13421772288, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

root@hq-vmc1-1:/mnt# mount /dev/nbd0p1 1

root@hq-vmc1-1:/mnt# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root   63G  3.4G   57G   6% /
tmpfs                  32G     0   32G   0% /lib/init/rw
udev                   32G  172K   32G   1% /dev
tmpfs                  32G   41M   32G   1% /dev/shm
tmpfs                  13G     0   13G   0% /run
/dev/mapper/pve-data  931G   33M  931G   1% /var/lib/vz
/dev/sda2             494M   53M  417M  12% /boot
/dev/mapper/pve-csv1  2.8T  2.1G  2.8T   1% /var/lib/storages/csv1
/dev/fuse              30M   32K   30M   1% /etc/pve
/dev/nbd0p1            50T   34M   50T   1% /mnt/1

root@hq-vmc1-1:/mnt/1# dd bs=100M count=100 if=/dev/zero of=1234
100+0 records in
100+0 records out
10485760000 bytes (10 GB) copied, 7.56993 s, 1.4 GB/s
Press any key to continue...

root@hq-vmc1-1:/mnt/1# dd bs=100M count=100 if=/dev/random of=1234
^C^C^C^C^C^C^C^C

Apr 10 13:57:26 hq-vmc1-1 kernel: [15335.620871] loop: module loaded
Apr 10 14:00:01 hq-vmc1-1 kernel: [15490.755643] nbd: registered device at major 43
Apr 10 14:00:16 hq-vmc1-1 kernel: [15506.286871]  nbd0: unknown partition table
Apr 10 14:03:42 hq-vmc1-1 kernel: [15711.869417] block nbd0: NBD_DISCONNECT
Apr 10 14:03:42 hq-vmc1-1 kernel: [15711.869556] block nbd0: Unexpected reply (ffff880fe7379c60)
Apr 10 14:03:42 hq-vmc1-1 kernel: [15711.869655] block nbd0: queue cleared
Apr 10 14:04:39 hq-vmc1-1 kernel: [15768.840228]  nbd0: unknown partition table
Apr 10 14:10:14 hq-vmc1-1 kernel: [16103.626841] XFS (nbd0p1): Mounting Filesystem
Apr 10 14:10:14 hq-vmc1-1 kernel: [16103.642247] XFS (nbd0p1): Ending clean mount
Apr 10 14:11:28 hq-vmc1-1 kernel: [16177.336994] qemu-nbd[9380]: segfault at f2f9a44a ip 00007f80ea8790b4 sp 00007f80e47c4730 error 4 in libglib-2.0.so.0.2400.2[7f80ea81d000+db000]
Apr 10 14:11:28 hq-vmc1-1 kernel: [16177.337066] nbd (pid 9382: qemu-nbd) got signal 9
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721683] INFO: task flush-43:0:9320 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721743] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721829] flush-43:0      D ffff880feae6f550     0  9320      2 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721837]  ffff880feae6f550 0000000000000046 ffff880f00000000 ffff880ffe8798d0
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721848]  0000000000013740 ffff880fe683bfd8 ffff880fe683bfd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721855]  ffff880feae6f550 ffff880fe683a010 ffff880fe683b7a0 00000001e683b7a0
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721864] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721879]  [<ffffffff8136786b>] ? io_schedule+0x84/0xc3
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721888]  [<ffffffff811ab390>] ? get_request_wait+0x104/0x199
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721897]  [<ffffffff81063b01>] ? wake_up_bit+0x20/0x20
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721903]  [<ffffffff811ab867>] ? blk_queue_bio+0x17a/0x2cf
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721909]  [<ffffffff811aad0d>] ? generic_make_request+0x8e/0xcd
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721914]  [<ffffffff811aae25>] ? submit_bio+0xd9/0xf7
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721923]  [<ffffffff81124df1>] ? __mark_inode_dirty+0x22/0x1a7
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721958]  [<ffffffffa03e6a12>] ? xfs_submit_ioend+0x94/0xd7 [xfs]
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721974]  [<ffffffffa03e7b23>] ? xfs_vm_writepage+0x36b/0x3d0 [xfs]
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721982]  [<ffffffff810c46ba>] ? __writepage+0xa/0x21
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721989]  [<ffffffff810c57ad>] ? write_cache_pages+0x226/0x31e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721994]  [<ffffffff810c46b0>] ? set_page_dirty+0x61/0x61
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.721999]  [<ffffffff810c58e3>] ? generic_writepages+0x3e/0x55
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722005]  [<ffffffff81125340>] ? writeback_single_inode+0x178/0x35e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722011]  [<ffffffff811257f2>] ? writeback_sb_inodes+0x169/0x1ff
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722016]  [<ffffffff811258f5>] ? __writeback_inodes_wb+0x6d/0xab
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722022]  [<ffffffff81125b1b>] ? wb_writeback+0x128/0x222
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722027]  [<ffffffff810c4e7e>] ? determine_dirtyable_memory+0xd/0x1d
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722032]  [<ffffffff810c4ef6>] ? global_dirty_limits+0x29/0x10b
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722038]  [<ffffffff81125dd3>] ? wb_do_writeback+0x1be/0x1de
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722043]  [<ffffffff813674fc>] ? __schedule+0x5a0/0x5cd
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722048]  [<ffffffff81125eb6>] ? bdi_writeback_thread+0xc3/0x1fe
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722054]  [<ffffffff81125df3>] ? wb_do_writeback+0x1de/0x1de
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722058]  [<ffffffff81125df3>] ? wb_do_writeback+0x1de/0x1de
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722064]  [<ffffffff810636b5>] ? kthread+0x7a/0x82
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722071]  [<ffffffff81370134>] ? kernel_thread_helper+0x4/0x10
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722077]  [<ffffffff8106363b>] ? kthread_worker_fn+0x147/0x147
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722082]  [<ffffffff81370130>] ? gs_change+0x13/0x13
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722087] INFO: task qemu-nbd:9380 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722142] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722226] qemu-nbd        D ffff880ffa96b160     0  9380      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722232]  ffff880ffa96b160 0000000000000086 ffff880ff8bd3dd0 ffff880ffece1610
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722240]  0000000000013740 ffff880ff8bd3fd8 ffff880ff8bd3fd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722248]  ffff880ffa96b160 ffff880ff8bd2010 ffffffff81770270 0000000000000400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722255] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722261]  [<ffffffff81367a32>] ? schedule_timeout+0x2d/0xd7
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722266]  [<ffffffff81367678>] ? wait_for_common+0xd1/0x14e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722274]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722282]  [<ffffffff8110a63b>] ? zap_process+0x80/0xaa
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722288]  [<ffffffff8110aa21>] ? do_coredump+0x2d8/0xafa
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722294]  [<ffffffff8105772c>] ? __dequeue_signal+0xeb/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722301]  [<ffffffff8105a45a>] ? get_signal_to_deliver+0x48c/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722309]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722318]  [<ffffffff8136ba92>] ? do_page_fault+0x215/0x34c
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722323]  [<ffffffff8100d6b0>] ? __switch_to+0x1c9/0x2b1
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722330]  [<ffffffff81139b6a>] ? eventfd_ctx_read+0x43/0x16c
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722335]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722341]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722347]  [<ffffffff81368dfc>] ? retint_signal+0x48/0x8c
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722351] INFO: task qemu-nbd:9382 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722490] qemu-nbd        D ffff880ff8a28180     0  9382      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722496]  ffff880ff8a28180 0000000000000086 0000000000000000 ffff880ffeebb5d0
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722504]  0000000000013740 ffff880ff9f47fd8 ffff880ff9f47fd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722511]  ffff880ff8a28180 ffff880ff9f46010 0000000000000246 00000001f9f47947
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722519] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722524]  [<ffffffff81367e7d>] ? __mutex_lock_common+0x10c/0x172
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722529]  [<ffffffff81367fa9>] ? mutex_lock+0x1a/0x2c
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722535]  [<ffffffffa05bb184>] ? sock_shutdown+0x18/0x62 [nbd]
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722541]  [<ffffffffa05bb429>] ? sock_xmit+0x25b/0x2cc [nbd]
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722546]  [<ffffffff81046020>] ? try_to_wake_up+0x181/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722554]  [<ffffffff8101432e>] ? native_sched_clock+0x28/0x32
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722562]  [<ffffffff811b4e19>] ? blkiocg_update_completion_stats+0x36/0xb2
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722568]  [<ffffffff811a99ed>] ? freed_request+0x36/0x5a
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722573]  [<ffffffff811aa9b9>] ? __blk_end_bidi_request+0x22/0x29
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722579]  [<ffffffffa05bb9e0>] ? __nbd_ioctl+0x315/0x76e [nbd]
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722586]  [<ffffffff8103ba17>] ? cpumask_next+0x18/0x1a
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722593]  [<ffffffff8103d33d>] ? select_task_rq_fair+0x55f/0x820
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722600]  [<ffffffff811bc33f>] ? cpumask_next_and+0x2b/0x37
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722608]  [<ffffffff810f7426>] ? ____cache_alloc+0xc6/0x20d
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722614]  [<ffffffff8106102b>] ? alloc_pid+0x277/0x368
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722619]  [<ffffffffa05bbf84>] ? nbd_ioctl+0x14b/0x168 [nbd]
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722625]  [<ffffffff81038833>] ? __wake_up_common+0x41/0x78
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722630]  [<ffffffff811affa2>] ? blkdev_ioctl+0x5e5/0x61b
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722636]  [<ffffffff8103b949>] ? __wake_up_sync_key+0x3f/0x57
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722643]  [<ffffffff8112edb1>] ? block_ioctl+0x32/0x36
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722650]  [<ffffffff811142c7>] ? do_vfs_ioctl+0x464/0x4b1
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722656]  [<ffffffff81118467>] ? dput+0x29/0xf2
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722661]  [<ffffffff81107f00>] ? fput+0x17a/0x1a2
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722667]  [<ffffffff8111435f>] ? sys_ioctl+0x4b/0x70
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722672]  [<ffffffff8136dfd2>] ? system_call_fastpath+0x16/0x1b
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722676] INFO: task qemu-nbd:9659 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722815] qemu-nbd        D ffff880ffc87d1a0     0  9659      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722821]  ffff880ffc87d1a0 0000000000000086 ffff880ff8aa3d48 ffff880ffed95750
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722828]  0000000000013740 ffff880ff8aa3fd8 ffff880ff8aa3fd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722836]  ffff880ffc87d1a0 ffff880ff8aa2010 ffff880ffc87d1a0 ffff880ffefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722843] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722851]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722857]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722864]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722869]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722875]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722880]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722886]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722891]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722896]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722901] INFO: task qemu-nbd:9665 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.722955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723040] qemu-nbd        D ffff880ff90157d0     0  9665      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723045]  ffff880ff90157d0 0000000000000086 ffff880f00000000 ffff880ffedb49f0
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723053]  0000000000013740 ffff880ff11fffd8 ffff880ff11fffd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723060]  ffff880ff90157d0 ffff880ff11fe010 ffff880ff90157d0 00000001fefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723068] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723073]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723079]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723085]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723090]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723096]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723101]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723107]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723112]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723117]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723121] INFO: task qemu-nbd:9666 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723176] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723260] qemu-nbd        D ffff880ff9f30080     0  9666      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723265]  ffff880ff9f30080 0000000000000086 ffff880ff9c0fd48 ffff880ffeca00c0
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723273]  0000000000013740 ffff880ff9c0ffd8 ffff880ff9c0ffd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723280]  ffff880ff9f30080 ffff880ff9c0e010 ffff880ff9f30080 ffff880ffefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723288] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723293]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723299]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723305]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723310]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723316]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723322]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723327]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723332]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723337]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723341] INFO: task qemu-nbd:9667 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723395] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723480] qemu-nbd        D ffff880ff92c0af0     0  9667      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723485]  ffff880ff92c0af0 0000000000000086 ffff880f00000000 ffff880ffede2340
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723492]  0000000000013740 ffff880fe58b3fd8 ffff880fe58b3fd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723500]  ffff880ff92c0af0 ffff880fe58b2010 ffff880ff92c0af0 00000001fefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723508] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723513]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723519]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723525]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723530]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723536]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723541]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723547]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723552]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723556]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723561] INFO: task qemu-nbd:9668 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723700] qemu-nbd        D ffff880ff99d4f20     0  9668      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723706]  ffff880ff99d4f20 0000000000000086 ffff880ff99b5d48 ffff880ffed30200
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723714]  0000000000013740 ffff880ff99b5fd8 ffff880ff99b5fd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723721]  ffff880ff99d4f20 ffff880ff99b4010 ffff880ff99d4f20 ffff880ffefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723729] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723735]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723741]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723747]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723752]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723758]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723763]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723769]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723774]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723779]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723783] INFO: task qemu-nbd:9669 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723922] qemu-nbd        D ffff880ff8b58970     0  9669      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723928]  ffff880ff8b58970 0000000000000086 ffff880f00000000 ffff880ffed12f60
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723936]  0000000000013740 ffff880ff985ffd8 ffff880ff985ffd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723943]  ffff880ff8b58970 ffff880ff985e010 ffff880ff8b58970 00000001fefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723951] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723957]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723962]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723968]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723974]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723980]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723985]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723991]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.723996]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724000]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724005] INFO: task qemu-nbd:9670 blocked for more than 120 seconds.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724144] qemu-nbd        D ffff880ff8592ee0     0  9670      1 0x00000000
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724149]  ffff880ff8592ee0 0000000000000086 ffff880f00000000 ffff880ffee27890
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724157]  0000000000013740 ffff880fe6949fd8 ffff880fe6949fd8 0000000000013740
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724165]  ffff880ff8592ee0 ffff880fe6948010 ffff880ff8592ee0 00000001fefcf400
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724173] Call Trace:
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724178]  [<ffffffff8104b933>] ? exit_mm+0x93/0x117
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724184]  [<ffffffff8104cf01>] ? do_exit+0x25b/0x799
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724190]  [<ffffffff8104d4b9>] ? do_group_exit+0x7a/0xa4
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724195]  [<ffffffff8105a462>] ? get_signal_to_deliver+0x494/0x4b6
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724202]  [<ffffffff8100df4b>] ? do_signal+0x38/0x635
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724207]  [<ffffffff81046030>] ? try_to_wake_up+0x191/0x191
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724213]  [<ffffffff8100e56d>] ? do_notify_resume+0x25/0x67
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724218]  [<ffffffff81106fcb>] ? sys_write+0x62/0x6e
Apr 10 14:13:43 hq-vmc1-1 kernel: [16312.724222]  [<ffffffff8136e2a0>] ? int_signal+0x12/0x17

# pveversion --verbose
pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 3.2.0-0.bpo.4-amd64
proxmox-ve-2.6.32: 2.3-93
pve-kernel-2.6.32-19-pve: 2.6.32-93
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-18
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-8
ksm-control-daemon: 1.1-1

# ps axfu | grep " D"
root        126  0.0  0.0      0     0 ?        D    09:41   0:02  \_ [kworker/9:1]
root       9709  0.0  0.0   9676   608 pts/1    D+   14:12   0:01  |               \_ dd bs=100M count=100 if=/dev/random of=1234
root      12881  0.0  0.0  12328   780 pts/4    D+   16:20   0:00  |               \_ shutdown -r 0 w
root       9380  0.1  0.0 290532  5616 ?        Dsl  14:04   0:16 qemu-nbd -c /dev/nbd0 disk.qcow2