Hello. We have Proxmox 4.4 with 4-node cluster. Total number of virtual machines is 40
There was a problem with backups on nfs.
Nov 12 15:37:04 vbox1 kernel: [1665093.318616] INFO: task lzop:55243 blocked for more than 120 seconds.
Nov 12 15:37:04 vbox1 kernel: [1665093.318631] Tainted: P O 4.4.67-1-pve #1
Nov 12 15:37:04 vbox1 kernel: [1665093.318640] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 12 15:37:04 vbox1 kernel: [1665093.318654] lzop D ffff881008953a48 0 55243 55234 0x00000000
Nov 12 15:37:04 vbox1 kernel: [1665093.318658] ffff881008953a48 0000000000000003 ffff88081bee0000 ffff881019d0e200
Nov 12 15:37:04 vbox1 kernel: [1665093.318660] ffff881008954000 ffff88081f657300 7fffffffffffffff ffffffff81860ec0
Nov 12 15:37:04 vbox1 kernel: [1665093.318662] ffff881008953ba8 ffff881008953a60 ffffffff818605d5 0000000000000000
Nov 12 15:37:04 vbox1 kernel: [1665093.318664] Call Trace:
Nov 12 15:37:04 vbox1 kernel: [1665093.318671] [<ffffffff81860ec0>] ? bit_wait_timeout+0xa0/0xa0
Nov 12 15:37:04 vbox1 kernel: [1665093.318673] [<ffffffff818605d5>] schedule+0x35/0x80
Nov 12 15:37:04 vbox1 kernel: [1665093.318675] [<ffffffff81863825>] schedule_timeout+0x235/0x2d0
Nov 12 15:37:04 vbox1 kernel: [1665093.318678] [<ffffffff810bdac7>] ? find_busiest_group+0x47/0x4e0
Nov 12 15:37:04 vbox1 kernel: [1665093.318680] [<ffffffff81860ec0>] ? bit_wait_timeout+0xa0/0xa0
Nov 12 15:37:04 vbox1 kernel: [1665093.318681] [<ffffffff8185facb>] io_schedule_timeout+0xbb/0x140
Nov 12 15:37:04 vbox1 kernel: [1665093.318683] [<ffffffff81860edb>] bit_wait_io+0x1b/0x70
Nov 12 15:37:04 vbox1 kernel: [1665093.318684] [<ffffffff8186098f>] __wait_on_bit+0x5f/0x90
Nov 12 15:37:04 vbox1 kernel: [1665093.318686] [<ffffffff81860ec0>] ? bit_wait_timeout+0xa0/0xa0
Nov 12 15:37:04 vbox1 kernel: [1665093.318687] [<ffffffff81860a41>] out_of_line_wait_on_bit+0x81/0xb0
Nov 12 15:37:04 vbox1 kernel: [1665093.318690] [<ffffffff810c4720>] ? autoremove_wake_function+0x40/0x40
Nov 12 15:37:04 vbox1 kernel: [1665093.318717] [<ffffffffc095ec54>] nfs_wait_on_request+0x34/0x40 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318727] [<ffffffffc09639dd>] nfs_updatepage+0x1bd/0x9f0 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318734] [<ffffffffc0953dbf>] nfs_write_end+0x14f/0x4d0 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318737] [<ffffffff8140e80f>] ? iov_iter_copy_from_user_atomic+0x8f/0x230
Nov 12 15:37:04 vbox1 kernel: [1665093.318741] [<ffffffff8118ec94>] generic_perform_write+0x114/0x1c0
Nov 12 15:37:04 vbox1 kernel: [1665093.318743] [<ffffffff81190f56>] __generic_file_write_iter+0x1a6/0x1f0
Nov 12 15:37:04 vbox1 kernel: [1665093.318745] [<ffffffff8122c9b3>] ? touch_atime+0x33/0xd0
Nov 12 15:37:04 vbox1 kernel: [1665093.318747] [<ffffffff81191084>] generic_file_write_iter+0xe4/0x1e0
Nov 12 15:37:04 vbox1 kernel: [1665093.318752] [<ffffffffc095353a>] nfs_file_write+0x9a/0x160 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318755] [<ffffffff8120ee1b>] new_sync_write+0x9b/0xe0
Nov 12 15:37:04 vbox1 kernel: [1665093.318757] [<ffffffff8120ee86>] __vfs_write+0x26/0x40
Nov 12 15:37:04 vbox1 kernel: [1665093.318759] [<ffffffff8120f4f9>] vfs_write+0xa9/0x190
Nov 12 15:37:04 vbox1 kernel: [1665093.318760] [<ffffffff8120f434>] ? vfs_read+0x114/0x130
Nov 12 15:37:04 vbox1 kernel: [1665093.318762] [<ffffffff812102d5>] SyS_write+0x55/0xc0
Nov 12 15:37:04 vbox1 kernel: [1665093.318765] [<ffffffff818646f6>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 12 15:38:50 vbox1 kernel: [1665199.496603] nfs: server 192.168.162.11 not responding, still trying
Nov 12 15:38:51 vbox1 kernel: [1665200.496849] nfs: server 192.168.162.11 not responding, still trying
bwlimit: 20000
# pveversion -v
proxmox-ve: 4.4-92 (running kernel: 4.4.67-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-100
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
There was a problem with backups on nfs.
Nov 12 15:37:04 vbox1 kernel: [1665093.318616] INFO: task lzop:55243 blocked for more than 120 seconds.
Nov 12 15:37:04 vbox1 kernel: [1665093.318631] Tainted: P O 4.4.67-1-pve #1
Nov 12 15:37:04 vbox1 kernel: [1665093.318640] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 12 15:37:04 vbox1 kernel: [1665093.318654] lzop D ffff881008953a48 0 55243 55234 0x00000000
Nov 12 15:37:04 vbox1 kernel: [1665093.318658] ffff881008953a48 0000000000000003 ffff88081bee0000 ffff881019d0e200
Nov 12 15:37:04 vbox1 kernel: [1665093.318660] ffff881008954000 ffff88081f657300 7fffffffffffffff ffffffff81860ec0
Nov 12 15:37:04 vbox1 kernel: [1665093.318662] ffff881008953ba8 ffff881008953a60 ffffffff818605d5 0000000000000000
Nov 12 15:37:04 vbox1 kernel: [1665093.318664] Call Trace:
Nov 12 15:37:04 vbox1 kernel: [1665093.318671] [<ffffffff81860ec0>] ? bit_wait_timeout+0xa0/0xa0
Nov 12 15:37:04 vbox1 kernel: [1665093.318673] [<ffffffff818605d5>] schedule+0x35/0x80
Nov 12 15:37:04 vbox1 kernel: [1665093.318675] [<ffffffff81863825>] schedule_timeout+0x235/0x2d0
Nov 12 15:37:04 vbox1 kernel: [1665093.318678] [<ffffffff810bdac7>] ? find_busiest_group+0x47/0x4e0
Nov 12 15:37:04 vbox1 kernel: [1665093.318680] [<ffffffff81860ec0>] ? bit_wait_timeout+0xa0/0xa0
Nov 12 15:37:04 vbox1 kernel: [1665093.318681] [<ffffffff8185facb>] io_schedule_timeout+0xbb/0x140
Nov 12 15:37:04 vbox1 kernel: [1665093.318683] [<ffffffff81860edb>] bit_wait_io+0x1b/0x70
Nov 12 15:37:04 vbox1 kernel: [1665093.318684] [<ffffffff8186098f>] __wait_on_bit+0x5f/0x90
Nov 12 15:37:04 vbox1 kernel: [1665093.318686] [<ffffffff81860ec0>] ? bit_wait_timeout+0xa0/0xa0
Nov 12 15:37:04 vbox1 kernel: [1665093.318687] [<ffffffff81860a41>] out_of_line_wait_on_bit+0x81/0xb0
Nov 12 15:37:04 vbox1 kernel: [1665093.318690] [<ffffffff810c4720>] ? autoremove_wake_function+0x40/0x40
Nov 12 15:37:04 vbox1 kernel: [1665093.318717] [<ffffffffc095ec54>] nfs_wait_on_request+0x34/0x40 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318727] [<ffffffffc09639dd>] nfs_updatepage+0x1bd/0x9f0 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318734] [<ffffffffc0953dbf>] nfs_write_end+0x14f/0x4d0 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318737] [<ffffffff8140e80f>] ? iov_iter_copy_from_user_atomic+0x8f/0x230
Nov 12 15:37:04 vbox1 kernel: [1665093.318741] [<ffffffff8118ec94>] generic_perform_write+0x114/0x1c0
Nov 12 15:37:04 vbox1 kernel: [1665093.318743] [<ffffffff81190f56>] __generic_file_write_iter+0x1a6/0x1f0
Nov 12 15:37:04 vbox1 kernel: [1665093.318745] [<ffffffff8122c9b3>] ? touch_atime+0x33/0xd0
Nov 12 15:37:04 vbox1 kernel: [1665093.318747] [<ffffffff81191084>] generic_file_write_iter+0xe4/0x1e0
Nov 12 15:37:04 vbox1 kernel: [1665093.318752] [<ffffffffc095353a>] nfs_file_write+0x9a/0x160 [nfs]
Nov 12 15:37:04 vbox1 kernel: [1665093.318755] [<ffffffff8120ee1b>] new_sync_write+0x9b/0xe0
Nov 12 15:37:04 vbox1 kernel: [1665093.318757] [<ffffffff8120ee86>] __vfs_write+0x26/0x40
Nov 12 15:37:04 vbox1 kernel: [1665093.318759] [<ffffffff8120f4f9>] vfs_write+0xa9/0x190
Nov 12 15:37:04 vbox1 kernel: [1665093.318760] [<ffffffff8120f434>] ? vfs_read+0x114/0x130
Nov 12 15:37:04 vbox1 kernel: [1665093.318762] [<ffffffff812102d5>] SyS_write+0x55/0xc0
Nov 12 15:37:04 vbox1 kernel: [1665093.318765] [<ffffffff818646f6>] entry_SYSCALL_64_fastpath+0x16/0x75
Nov 12 15:38:50 vbox1 kernel: [1665199.496603] nfs: server 192.168.162.11 not responding, still trying
Nov 12 15:38:51 vbox1 kernel: [1665200.496849] nfs: server 192.168.162.11 not responding, still trying
bwlimit: 20000
# pveversion -v
proxmox-ve: 4.4-92 (running kernel: 4.4.67-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-100
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80