... blocked for 120 sec

[mfg666]

Hi,

i am new to proxmox.
hw
intel xeon e3 v5 1235l
fujitsu 3417b2
4x 8tb ironwolf (zfs pool raidz1)
adata 6000 (vms and containers)
usb stick 3.0 (proxmox)
2x Samsung - DDR4 - 16 GB - DIMM 288-PIN - 2400 MHz

now i want to fill my pool with data.
i connected wd-red 8tb to mainboard, mount, cp, ...

5801.178361] RIP: 0033:0x7f30c13fc970
[ 5801.178362] RSP: 002b:00007ffd5820ff28 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 5801.178364] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f30c13fc970
[ 5801.178364] RDX: 0000000000020000 RSI: 000055dd2004f000 RDI: 0000000000000004
[ 5801.178365] RBP: 000055dd2004f000 R08: 0000000000000000 R09: 0000000000000000
[ 5801.178366] R10: 00007ffd58210160 R11: 0000000000000246 R12: 000055dd2004f000
[ 5801.178366] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000020000
[ 6428.517921] systemd[1]: apt-daily-upgrade.timer: Adding 32min 37.986642s random time.
[ 6428.518721] systemd[1]: apt-daily.timer: Adding 1h 20min 24.021844s random time.
[ 6428.518860] systemd[1]: pve-daily-update.timer: Adding 2h 46min 11.668386s random time.
[ 6429.542450] systemd[1]: apt-daily-upgrade.timer: Adding 30min 54.904711s random time.
[ 6429.543228] systemd[1]: apt-daily.timer: Adding 6h 42min 14.597045s random time.
[ 6429.543357] systemd[1]: pve-daily-update.timer: Adding 1h 10min 27.939276s random time.
[ 6429.882232] systemd[1]: apt-daily-upgrade.timer: Adding 41min 3.828876s random time.
[ 6429.883001] systemd[1]: apt-daily.timer: Adding 9h 52min 52.061771s random time.
[ 6429.883132] systemd[1]: pve-daily-update.timer: Adding 1h 37min 52.868600s random time.
[ 7619.711252] fwbr102i0: port 2(veth102i0) entered disabled state
[ 7619.896567] audit: type=1400 audit(1561130665.757:13): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=25887 comm="apparmor_parser"
[ 7621.054454] fwbr102i0: port 2(veth102i0) entered disabled state
[ 7621.054613] device veth102i0 left promiscuous mode
[ 7621.054615] fwbr102i0: port 2(veth102i0) entered disabled state
[ 7621.109070] fwbr102i0: port 1(fwln102i0) entered disabled state
[ 7621.109129] vmbr0: port 2(fwpr102p0) entered disabled state
[ 7621.109213] device fwln102i0 left promiscuous mode
[ 7621.109214] fwbr102i0: port 1(fwln102i0) entered disabled state
[ 7621.129906] device fwpr102p0 left promiscuous mode
[ 7621.129908] vmbr0: port 2(fwpr102p0) entered disabled state
[ 7855.217973] INFO: task txg_sync:1051 blocked for more than 120 seconds.
[ 7855.219171]       Tainted: P           O     4.15.18-12-pve #1
[ 7855.220483] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7855.221680] txg_sync        D    0  1051      2 0x80000000
[ 7855.221682] Call Trace:
[ 7855.221687]  __schedule+0x3e0/0x870
[ 7855.221689]  schedule+0x36/0x80
[ 7855.221690]  io_schedule+0x16/0x40
[ 7855.221696]  cv_wait_common+0xb2/0x140 [spl]
[ 7855.221698]  ? wait_woken+0x80/0x80
[ 7855.221702]  __cv_wait_io+0x18/0x20 [spl]
[ 7855.221733]  zio_wait+0x105/0x1b0 [zfs]
[ 7855.221754]  dsl_pool_sync+0xb8/0x430 [zfs]
[ 7855.221779]  spa_sync+0x42d/0xd50 [zfs]
[ 7855.221856]  txg_sync_thread+0x2d4/0x4a0 [zfs]
[ 7855.221911]  ? txg_quiesce_thread+0x3f0/0x3f0 [zfs]
[ 7855.221918]  thread_generic_wrapper+0x74/0x90 [spl]
[ 7855.221922]  kthread+0x105/0x140
[ 7855.221926]  ? __thread_exit+0x20/0x20 [spl]
[ 7855.221930]  ? kthread_create_worker_on_cpu+0x70/0x70
[ 7855.221934]  ret_from_fork+0x35/0x40
[ 7855.221945] INFO: task cp:5594 blocked for more than 120 seconds.
[ 7855.223200]       Tainted: P           O     4.15.18-12-pve #1
[ 7855.224344] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7855.225665] cp              D    0  5594   2857 0x00000000
[ 7855.225666] Call Trace:
[ 7855.225668]  __schedule+0x3e0/0x870
[ 7855.225669]  schedule+0x36/0x80
[ 7855.225673]  cv_wait_common+0x11e/0x140 [spl]
[ 7855.225674]  ? wait_woken+0x80/0x80
[ 7855.225677]  __cv_wait+0x15/0x20 [spl]
[ 7855.225701]  txg_wait_open+0xb0/0x100 [zfs]
[ 7855.225720]  dmu_tx_wait+0x389/0x3a0 [zfs]
[ 7855.225738]  dmu_tx_assign+0x176/0x470 [zfs]
[ 7855.225763]  zfs_write+0x43f/0xea0 [zfs]
[ 7855.225765]  ? blk_finish_plug+0x2c/0x40
[ 7855.225767]  ? touch_atime+0x36/0xe0
[ 7855.225806]  zpl_write_common_iovec+0x8c/0xe0 [zfs]
[ 7855.225865]  zpl_iter_write+0xae/0xe0 [zfs]
[ 7855.225871]  new_sync_write+0xe7/0x140
[ 7855.225874]  __vfs_write+0x29/0x40
[ 7855.225877]  vfs_write+0xb5/0x1a0
[ 7855.225880]  SyS_write+0x55/0xc0
[ 7855.225883]  do_syscall_64+0x73/0x130
[ 7855.225887]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 7855.225904] RIP: 0033:0x7f30c13fc970
[ 7855.225906] RSP: 002b:00007ffd5820ff28 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 7855.225911] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f30c13fc970
[ 7855.225913] RDX: 0000000000020000 RSI: 000055dd2004f000 RDI: 0000000000000004
[ 7855.225915] RBP: 000055dd2004f000 R08: 0000000000000000 R09: 0000000000000000
[ 7855.225920] R10: 00007ffd58210160 R11: 0000000000000246 R12: 000055dd2004f000
[ 7855.225921] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000020000

after 30-60 minutes, cp turns slow and "... blocked ... " message appears.

anyone knows how to fix this ?

bg
mfg

 

[fireon]

Can you do an pveperf on your Zpool?

pveperf /yourpool

And what should this machine do for you? Only data's or also VM's? Big datastore, or only some documents...? Have you ECC Memory, and how you conneced your discs? With an SAScontroller, or on SATA?

 

[mfg666]

Thanks for your answer.

sudo pveperf /pool/ZFS
CPU BOGOMIPS:      15936.00
REGEX/SECOND:      2990213
HD SIZE:           20578.33 GB (ZFSmfg)
FSYNCS/SECOND:     73.38
DNS EXT:           44.53 ms
DNS INT:           1.13 ms (fritz.box)

hw:
2x Samsung DIMM 16GB, DDR4-2400, CL17-17-17, ECC (M391A2K43BB1-CRC)
4x Seagate IronWolf NAS HDD 8TB, SATA 6Gb/s (ST8000VN0022)(zfs, connected by sata to mainboard)
Intel Xeon E3-1235L v5, 4x 2.00GHz, tray (CM8066201935807)
Fujitsu D3417-B2/D3417-B21 (S26361-F5110-V140)
ADATA XPG SX6000 512GB, M.2 (ASX6000NP-512GT-C) (VMs)
usb stick 3.0 64gb sandisk ultra fit (proxmox)

This machine should do VMs (2-3) and Data for my familiy.

 

[Vladimir Bulgaru]

Similar situation on the latest kernel.

 

[mfg666]

Thanks.

Similar situation on the latest kernel.

Can you give me more context. I don not know what you mean.

 

[Vladimir Bulgaru]

Got something akin of:

lzop 1121 blocked for more than 120 sec
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

No idea why. Just confirming that i do get similar weird messages. In my case VMs were not reachable for some time.

 

[fireon]

Your machine is to slow. FSYNCS/SECOND minimum for productive 3000. 32GB of Memory for ZFS and VM's will be to small (depends on the VM's). And RaidZ is not designed for virtualization. Use only Raid10. Install Proxmox on 2 separate SSD's. Create an Raid10 with your HDD's and add an SSD for Cache and Log to the Pool. You should addmore Memory (VM's need also Mem) and test again. Also the CPU is not really the big one.
And do never install Proxmox on USBstick. It is designed to run on HDD/SSD. On USBstick your system will not work within half a year.

 

[mfg666]

I have expanded my RAM to 64GB.
I have installed proxmox on SSD, VMs and Containers to another SSD.
I have added SSD for cache & log.
I have changed the Mainboard to Asus P10S. D3417 got problems with more then one nvme.

pveperf /pool/ZFS
CPU BOGOMIPS:      15936.00
REGEX/SECOND:      3157538
HD SIZE:           20980.58 GB (ZFSmfg)
FSYNCS/SECOND:     1198.17
DNS EXT:           34.16 ms
DNS INT:           1.27 ms (fritz.box)

At the moment, no Problems. ;-)