Proxmox 6 CIFS stuck

[Liviu Sas]

Hello,

Since upgrading to proxmox 6 I started having issues with my CIFS storage.
From time to time the CIFS storage becomes stuck and all processes trying to access it are stuck in IO waiting.

PID TTY STAT TIME COMMAND
86 ? D 12:33 \_ [kswapd0]
32589 ? D 0:13 \_ [cifsd]
4497 ? D 0:01 \_ [kworker/2:1+cifsiod]
14911 ? D 0:05 \_ [kworker/u16:4+flush-cifs-27]
29283 ? D 0:00 \_ [kworker/2:0+cifsiod]
3286 ? D 0:00 \_ [kworker/3:1+cifsiod]
4408 ? D 0:00 \_ [kworker/2:2+cifsiod]
24872 ? D 0:00 \_ [kworker/2:4+cifsiod]
5778 ? Ds 1:54 /usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f -p /run/snmpd.pid
25331 ? D 0:00 \_ pvestatd

The CIFS storage is being mounted using the PVE CLI. This how the mount looks like:
//media/PVE on /mnt/pve/shared_media type cifs (rw,relatime,vers=3.0,cache=strict,username=pve,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.86.231,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

These processes never recover (waited up to 10 hours).
unmount -l umounts the share but the processes are still stuck.

The only fix is to hard reboot the host.

I had the same issues with shares from a linux samba server and from a windows 10 pro.

I have never experienced these issues with proxmox 5.

 

[Liviu Sas]

A bit of extra info I could find, maybe it helps:

root@pvefast:~# cat /proc/fs/cifs/DebugData
Display Internal CIFS Data Structures for Debugging
---------------------------------------------------
CIFS Version 2.17
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 19
Servers:
Number of credits: 33 Dialect 0x300
1) Name: 192.168.86.231 Uses: 1 Capability: 0x300047 Session Status: 1 TCP status: 4 Instance: 6
Local Users To Server: 2 SecMode: 0x1 Req On Wire: 1
Shares:
0) IPC: \\media\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0
Share Capabilities: None Share Flags: 0x0
tid: 0x49cb83b9 Maximal Access: 0x1f00a9 DISCONNECTED
1) \\media\PVE Mounts: 1 DevInfo: 0x20 Attributes: 0x1006f
PathComponentMax: 255 Status: 2 type: DISK
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x6336dfcf Optimal sector size: 0x200 Maximal Access: 0x1f01ff DISCONNECTED
MIDs:


root@pvefast:~# cat /proc/fs/cifs/Stats
Resources in use
CIFS Session: 13
Share (unique mount targets): 26
SMB Request/Response Buffer: 13 Pool size: 17
SMB Small Req/Resp Buffer: 13 Pool size: 30
Operations (MIDs): 0
345 session 36 share reconnects
Total vfs operations: 2558280 maximum at one time: 21
1) \\media\PVE DISCONNECTED
SMBs: 398714
Bytes read: 1152 Bytes written: 187023634663
Open files: 1 total (local), 1 open on server
TreeConnects: 15 total 0 failed
TreeDisconnects: 0 total 0 failed
Creates: 107590 total 2 failed
Closes: 107585 total 2 failed
Flushes: 0 total 0 failed
Reads: 4 total 0 failed
Writes: 44677 total 11 failed
Locks: 0 total 0 failed
IOCTLs: 5 total 1 failed
QueryDirectories: 46 total 0 failed
ChangeNotifies: 0 total 0 failed
QueryInfos: 138751 total 11 failed
SetInfos: 52 total 0 failed
OplockBreaks: 0 sent 0 failed


root@pvefast:~# cat /proc/fs/cifs/open_files
# Version:1
# Format:
# <tree id> <persistent fid> <flags> <count> <pid> <uid> <filename>
0x6336dfcf 0x31fee18 0x8001 12 18138 0 vzdump-lxc-107-2019_08_22-02_38_26.tar.dat

 

[Liviu Sas]

And found this in dmesg:

[217962.233752] CIFS VFS: sends on sock 0000000056c94524 stuck for 15 seconds
[217962.234106] CIFS VFS: Error -11 sending data on socket to server
[217982.773313] CIFS VFS: Error -104 sending data on socket to server
[217982.786506] CIFS VFS: Error -32 sending data on socket to server
[218225.402976] INFO: task telegraf:23436 blocked for more than 120 seconds.
[218225.403394] Tainted: P W O 5.0.18-1-pve #1
[218225.403758] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.404109] telegraf D 0 23436 1 0x00000000
[218225.404110] Call Trace:
[218225.404115] __schedule+0x2d4/0x870
[218225.404117] ? __switch_to_asm+0x35/0x70
[218225.404118] ? __switch_to_asm+0x41/0x70
[218225.404119] schedule+0x2c/0x70
[218225.404120] schedule_preempt_disabled+0xe/0x10
[218225.404121] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.404121] ? __switch_to_asm+0x35/0x70
[218225.404124] __mutex_lock_slowpath+0x13/0x20
[218225.404124] mutex_lock+0x2c/0x30
[218225.404140] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.404142] ? mempool_free_slab+0x17/0x20
[218225.404143] ? mempool_free+0x2f/0x90
[218225.404152] ? cifs_mid_q_entry_release+0x34/0x50 [cifs]
[218225.404161] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.404170] ? compound_send_recv+0x681/0x790 [cifs]
[218225.404179] SMB2_open_init+0x69/0x760 [cifs]
[218225.404180] ? mempool_free_slab+0x17/0x20
[218225.404189] smb2_query_info_compound+0x17b/0x3a0 [cifs]
[218225.404190] ? _cond_resched+0x19/0x30
[218225.404192] ? dput.part.33+0x2e/0x120
[218225.404194] ? follow_managed+0x292/0x320
[218225.404195] ? lookup_fast+0x13d/0x310
[218225.404196] ? _cond_resched+0x19/0x30
[218225.404205] smb2_queryfs+0x6e/0xf0 [cifs]
[218225.404214] ? smb2_queryfs+0x6e/0xf0 [cifs]
[218225.404221] cifs_statfs+0xaf/0x2d0 [cifs]
[218225.404222] statfs_by_dentry+0x6d/0x90
[218225.404223] vfs_statfs+0x1b/0xc0
[218225.404224] user_statfs+0x5a/0xa0
[218225.404225] __do_sys_statfs+0x27/0x60
[218225.404226] __x64_sys_statfs+0x16/0x20
[218225.404228] do_syscall_64+0x5a/0x110
[218225.404229] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[218225.404230] RIP: 0033:0x4b2100
[218225.404233] Code: Bad RIP value.
[218225.404233] RSP: 002b:000000c0002b37a0 EFLAGS: 00000202 ORIG_RAX: 0000000000000089
[218225.404234] RAX: ffffffffffffffda RBX: 000000c00005c000 RCX: 00000000004b2100
[218225.404235] RDX: 0000000000000000 RSI: 000000c0002b3870 RDI: 000000c000471360
[218225.404235] RBP: 000000c0002b3800 R08: 0000000000000000 R09: 0000000000000000
[218225.404235] R10: 0000000000000000 R11: 0000000000000202 R12: ffffffffffffffff
[218225.404236] R13: 000000000000009c R14: 000000000000009b R15: 0000000000000100
[218225.404238] INFO: task snmpd:5778 blocked for more than 120 seconds.
[218225.404600] Tainted: P W O 5.0.18-1-pve #1
[218225.404963] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.405337] snmpd D 0 5778 1 0x00000000
[218225.405338] Call Trace:
[218225.405339] __schedule+0x2d4/0x870
[218225.405340] ? __switch_to_asm+0x35/0x70
[218225.405341] ? __switch_to_asm+0x41/0x70
[218225.405342] schedule+0x2c/0x70
[218225.405343] schedule_preempt_disabled+0xe/0x10
[218225.405344] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.405345] ? __switch_to_asm+0x35/0x70
[218225.405345] __mutex_lock_slowpath+0x13/0x20
[218225.405346] mutex_lock+0x2c/0x30
[218225.405355] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.405357] ? mempool_free_slab+0x17/0x20
[218225.405357] ? mempool_free+0x2f/0x90
[218225.405366] ? cifs_mid_q_entry_release+0x34/0x50 [cifs]
[218225.405375] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.405383] ? compound_send_recv+0x681/0x790 [cifs]
[218225.405392] SMB2_open_init+0x69/0x760 [cifs]
[218225.405393] ? mempool_free_slab+0x17/0x20
[218225.405402] smb2_query_info_compound+0x17b/0x3a0 [cifs]
[218225.405403] ? _cond_resched+0x19/0x30
[218225.405405] ? dput.part.33+0x2e/0x120
[218225.405406] ? follow_managed+0x292/0x320
[218225.405407] ? lookup_fast+0x13d/0x310
[218225.405408] ? _cond_resched+0x19/0x30
[218225.405417] smb2_queryfs+0x6e/0xf0 [cifs]
[218225.405426] ? smb2_queryfs+0x6e/0xf0 [cifs]
[218225.405433] cifs_statfs+0xaf/0x2d0 [cifs]
[218225.405434] statfs_by_dentry+0x6d/0x90
[218225.405435] vfs_statfs+0x1b/0xc0
[218225.405435] user_statfs+0x5a/0xa0
[218225.405436] __do_sys_statfs+0x27/0x60
[218225.405437] __x64_sys_statfs+0x16/0x20
[218225.405438] do_syscall_64+0x5a/0x110
[218225.405439] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[218225.405440] RIP: 0033:0x7f85f1a8be97
[218225.405441] Code: Bad RIP value.
[218225.405442] RSP: 002b:00007fff870d6098 EFLAGS: 00000246 ORIG_RAX: 0000000000000089
[218225.405442] RAX: ffffffffffffffda RBX: 000055ad591a0b60 RCX: 00007f85f1a8be97
[218225.405443] RDX: 00007f85f23af566 RSI: 00007fff870d60b0 RDI: 000055ad591a0b60
[218225.405443] RBP: 000055ad595ffcf0 R08: ffffffffffffffff R09: 00007f85f24f5317
[218225.405443] R10: 0000000000000001 R11: 0000000000000246 R12: 00007f85f24f5315
[218225.405444] R13: 000055ad591a0f61 R14: 000055ad591a0b50 R15: 00007f85f1b60140
[218225.405525] INFO: task kworker/2:1:4497 blocked for more than 120 seconds.
[218225.405903] Tainted: P W O 5.0.18-1-pve #1
[218225.406284] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.406677] kworker/2:1 D 0 4497 2 0x80000000
[218225.406686] Workqueue: cifsiod cifs_writev_complete [cifs]
[218225.406686] Call Trace:
[218225.406688] __schedule+0x2d4/0x870
[218225.406689] ? __switch_to_asm+0x35/0x70
[218225.406690] schedule+0x2c/0x70
[218225.406691] schedule_preempt_disabled+0xe/0x10
[218225.406692] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.406693] ? del_timer_sync+0x39/0x40
[218225.406694] __mutex_lock_slowpath+0x13/0x20
[218225.406695] mutex_lock+0x2c/0x30
[218225.406704] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.406706] ? __enqueue_entity+0x5c/0x60
[218225.406707] ? wait_woken+0x80/0x80
[218225.406716] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.406722] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.406731] smb2_async_writev+0x9a/0x420 [cifs]
[218225.406738] ? cifs_writedata_direct_alloc+0x2a/0xa0 [cifs]
[218225.406740] ? pmdp_collapse_flush+0x10/0x10
[218225.406746] cifs_writev_complete+0x35d/0x670 [cifs]
[218225.406755] ? SMB2_read+0x460/0x460 [cifs]
[218225.406762] ? cifs_writev_complete+0x35d/0x670 [cifs]
[218225.406764] process_one_work+0x20f/0x410
[218225.406764] worker_thread+0x34/0x400
[218225.406766] kthread+0x120/0x140
[218225.406767] ? process_one_work+0x410/0x410
[218225.406768] ? __kthread_parkme+0x70/0x70
[218225.406769] ret_from_fork+0x35/0x40
[218225.406770] INFO: task kworker/u16:4:14911 blocked for more than 120 seconds.
[218225.407239] Tainted: P W O 5.0.18-1-pve #1
[218225.407813] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.408424] kworker/u16:4 D 0 14911 2 0x80000000
[218225.408429] Workqueue: writeback wb_workfn (flush-cifs-27)
[218225.408430] Call Trace:
[218225.408434] __schedule+0x2d4/0x870
[218225.408435] ? __switch_to_asm+0x35/0x70
[218225.408437] schedule+0x2c/0x70
[218225.408438] schedule_preempt_disabled+0xe/0x10
[218225.408439] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.408441] ? del_timer_sync+0x39/0x40
[218225.408442] __mutex_lock_slowpath+0x13/0x20
[218225.408443] mutex_lock+0x2c/0x30
[218225.408458] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.408460] ? wait_woken+0x80/0x80
[218225.408471] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.408478] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.408488] smb2_async_writev+0x9a/0x420 [cifs]
[218225.408490] ? __fprop_inc_percpu_max+0x69/0xb0
[218225.408499] cifs_writepages+0x6df/0xab0 [cifs]
[218225.408509] ? SMB2_read+0x460/0x460 [cifs]
[218225.408517] ? cifs_writepages+0x6df/0xab0 [cifs]
[218225.408520] do_writepages+0x41/0xd0
[218225.408521] ? __wb_calc_thresh+0x3e/0x130
[218225.408522] __writeback_single_inode+0x40/0x350
[218225.408524] writeback_sb_inodes+0x211/0x500
[218225.408525] __writeback_inodes_wb+0x67/0xb0
[218225.408526] wb_writeback+0x25f/0x2f0
[218225.408527] ? 0xffffffff97000000
[218225.408529] wb_workfn+0x175/0x3f0
[218225.408530] ? __switch_to+0x96/0x4e0
[218225.408532] process_one_work+0x20f/0x410
[218225.408533] worker_thread+0x25e/0x400
[218225.408535] kthread+0x120/0x140
[218225.408535] ? process_one_work+0x410/0x410
[218225.408536] ? __kthread_parkme+0x70/0x70
[218225.408537] ret_from_fork+0x35/0x40
[218225.408542] INFO: task kworker/2:0:29283 blocked for more than 120 seconds.
[218225.409008] Tainted: P W O 5.0.18-1-pve #1
[218225.409475] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.409947] kworker/2:0 D 0 29283 2 0x80000000
[218225.409963] Workqueue: cifsiod smb2_reconnect_server [cifs]
[218225.409964] Call Trace:
[218225.409966] __schedule+0x2d4/0x870
[218225.409968] schedule+0x2c/0x70
[218225.409969] schedule_preempt_disabled+0xe/0x10
[218225.409970] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.409971] __mutex_lock_slowpath+0x13/0x20
[218225.409971] mutex_lock+0x2c/0x30
[218225.409981] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.409983] ? lock_timer_base+0x6b/0x90
[218225.409985] ? wait_woken+0x80/0x80
[218225.409995] smb2_reconnect_server+0x18c/0x2d0 [cifs]
[218225.409996] process_one_work+0x20f/0x410
[218225.409997] worker_thread+0x34/0x400
[218225.409999] kthread+0x120/0x140
[218225.410000] ? process_one_work+0x410/0x410
[218225.410001] ? __kthread_parkme+0x70/0x70
[218225.410002] ret_from_fork+0x35/0x40
[218225.410004] INFO: task kworker/3:1:3286 blocked for more than 120 seconds.
[218225.410526] Tainted: P W O 5.0.18-1-pve #1
[218225.411101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.411642] kworker/3:1 D 0 3286 2 0x80000000
[218225.411654] Workqueue: cifsiod cifs_writev_complete [cifs]
[218225.411654] Call Trace:
[218225.411656] __schedule+0x2d4/0x870
[218225.411657] ? __switch_to_asm+0x35/0x70
[218225.411659] schedule+0x2c/0x70
[218225.411660] schedule_preempt_disabled+0xe/0x10
[218225.411661] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.411662] ? del_timer_sync+0x39/0x40
[218225.411663] __mutex_lock_slowpath+0x13/0x20
[218225.411663] mutex_lock+0x2c/0x30
[218225.411674] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.411675] ? wait_woken+0x80/0x80
[218225.411685] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.411692] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.411699] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.411709] smb2_async_writev+0x9a/0x420 [cifs]
[218225.411717] ? cifs_writedata_direct_alloc+0x2a/0xa0 [cifs]
[218225.411718] ? pmdp_collapse_flush+0x10/0x10
[218225.411725] cifs_writev_complete+0x35d/0x670 [cifs]
[218225.411735] ? SMB2_read+0x460/0x460 [cifs]
[218225.411742] ? cifs_writev_complete+0x35d/0x670 [cifs]
[218225.411743] process_one_work+0x20f/0x410
[218225.411744] worker_thread+0x34/0x400
[218225.411746] kthread+0x120/0x140
[218225.411746] ? process_one_work+0x410/0x410
[218225.411747] ? __kthread_parkme+0x70/0x70
[218225.411748] ret_from_fork+0x35/0x40
[218225.411752] INFO: task lzop:18138 blocked for more than 120 seconds.
[218225.412230] Tainted: P W O 5.0.18-1-pve #1
[218225.412711] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.413200] lzop D 0 18138 18136 0x80000002
[218225.413201] Call Trace:
[218225.413202] __schedule+0x2d4/0x870
[218225.413203] schedule+0x2c/0x70
[218225.413204] schedule_preempt_disabled+0xe/0x10
[218225.413205] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.413206] ? del_timer_sync+0x39/0x40
[218225.413207] __mutex_lock_slowpath+0x13/0x20
[218225.413208] mutex_lock+0x2c/0x30
[218225.413218] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.413220] ? wait_woken+0x80/0x80
[218225.413230] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.413237] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.413246] smb2_async_writev+0x9a/0x420 [cifs]
[218225.413248] ? percpu_counter_add_batch+0x4b/0x60
[218225.413257] cifs_writepages+0x6df/0xab0 [cifs]
[218225.413266] ? SMB2_read+0x460/0x460 [cifs]
[218225.413275] ? cifs_writepages+0x6df/0xab0 [cifs]
[218225.413276] do_writepages+0x41/0xd0
[218225.413277] ? free_pcp_prepare+0x4f/0xd0
[218225.413279] __filemap_fdatawrite_range+0xc5/0x100
[218225.413280] filemap_write_and_wait+0x31/0x90
[218225.413288] cifs_flush+0x43/0x90 [cifs]
[218225.413289] filp_close+0x36/0x70
[218225.413290] put_files_struct+0x6c/0xc0
[218225.413291] exit_files+0x49/0x50
[218225.413292] do_exit+0x2be/0xb30
[218225.413298] ? handle_mm_fault+0xe1/0x210
[218225.413299] do_group_exit+0x43/0xb0
[218225.413300] __x64_sys_exit_group+0x18/0x20
[218225.413308] do_syscall_64+0x5a/0x110
[218225.413309] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[218225.413311] RIP: 0033:0x7fa2da05f9d6
[218225.413314] Code: Bad RIP value.
[218225.413314] RSP: 002b:00007ffc0d4a2ff8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
[218225.413315] RAX: ffffffffffffffda RBX: 00007fa2da150760 RCX: 00007fa2da05f9d6
[218225.413316] RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
[218225.413316] RBP: 0000000000000000 R08: 00000000000000e7 R09: ffffffffffffff80
[218225.413316] R10: 00007ffc0d4a2eb6 R11: 0000000000000246 R12: 00007fa2da150760
[218225.413317] R13: 0000000000000001 R14: 00007fa2da159428 R15: 0000000000000000
[218225.413319] INFO: task kworker/2:2:4408 blocked for more than 120 seconds.
[218225.413824] Tainted: P W O 5.0.18-1-pve #1
[218225.414325] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.414843] kworker/2:2 D 0 4408 2 0x80000000
[218225.414852] Workqueue: cifsiod cifs_writev_complete [cifs]
[218225.414853] Call Trace:
[218225.414854] __schedule+0x2d4/0x870
[218225.414855] ? __switch_to_asm+0x35/0x70
[218225.414857] schedule+0x2c/0x70
[218225.414858] schedule_preempt_disabled+0xe/0x10
[218225.414858] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.414860] ? del_timer_sync+0x39/0x40
[218225.414861] __mutex_lock_slowpath+0x13/0x20
[218225.414861] mutex_lock+0x2c/0x30
[218225.414871] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.414872] ? wait_woken+0x80/0x80
[218225.414882] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.414889] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.414899] smb2_async_writev+0x9a/0x420 [cifs]
[218225.414906] ? cifs_writedata_direct_alloc+0x2a/0xa0 [cifs]
[218225.414907] ? pmdp_collapse_flush+0x10/0x10
[218225.414915] cifs_writev_complete+0x35d/0x670 [cifs]
[218225.414924] ? SMB2_read+0x460/0x460 [cifs]
[218225.414931] ? cifs_writev_complete+0x35d/0x670 [cifs]
[218225.414932] process_one_work+0x20f/0x410
[218225.414933] worker_thread+0x34/0x400
[218225.414935] kthread+0x120/0x140
[218225.414935] ? process_one_work+0x410/0x410
[218225.414936] ? __kthread_parkme+0x70/0x70
[218225.414937] ret_from_fork+0x35/0x40
[218225.414940] INFO: task kworker/2:4:24872 blocked for more than 120 seconds.
[218225.414952] Tainted: P W O 5.0.18-1-pve #1
[218225.415998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.416535] kworker/2:4 D 0 24872 2 0x80000000
[218225.416557] Workqueue: cifsiod cifs_writev_complete [cifs]
[218225.416557] Call Trace:
[218225.416559] __schedule+0x2d4/0x870
[218225.416560] ? __switch_to_asm+0x35/0x70
[218225.416561] schedule+0x2c/0x70
[218225.416562] schedule_preempt_disabled+0xe/0x10
[218225.416562] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.416563] ? del_timer_sync+0x39/0x40
[218225.416564] __mutex_lock_slowpath+0x13/0x20
[218225.416565] mutex_lock+0x2c/0x30
[218225.416574] smb2_reconnect+0x102/0x7d0 [cifs]
[218225.416575] ? wait_woken+0x80/0x80
[218225.416584] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.416591] ? cifs_echo_callback+0x30/0x30 [cifs]
[218225.416600] smb2_async_writev+0x9a/0x420 [cifs]
[218225.416601] ? __switch_to_asm+0x35/0x70
[218225.416601] ? __switch_to_asm+0x35/0x70
[218225.416602] ? __switch_to_asm+0x41/0x70
[218225.416603] ? pmdp_collapse_flush+0x10/0x10
[218225.416610] cifs_writev_complete+0x35d/0x670 [cifs]
[218225.416619] ? SMB2_read+0x460/0x460 [cifs]
[218225.416625] ? cifs_writev_complete+0x35d/0x670 [cifs]
[218225.416626] process_one_work+0x20f/0x410
[218225.416627] worker_thread+0x34/0x400
[218225.416628] kthread+0x120/0x140
[218225.416629] ? process_one_work+0x410/0x410
[218225.416630] ? __kthread_parkme+0x70/0x70
[218225.416631] ret_from_fork+0x35/0x40
[218225.416632] INFO: task pvestatd:25331 blocked for more than 120 seconds.
[218225.417141] Tainted: P W O 5.0.18-1-pve #1
[218225.417686] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[218225.418201] pvestatd D 0 25331 15648 0x80000004
[218225.418202] Call Trace:
[218225.418203] __schedule+0x2d4/0x870
[218225.418212] ? SMB2_sess_setup+0x16b/0x2b0 [cifs]
[218225.418213] schedule+0x2c/0x70
[218225.418214] schedule_preempt_disabled+0xe/0x10
[218225.418215] __mutex_lock.isra.10+0x2e4/0x4c0
[218225.418215] __mutex_lock_slowpath+0x13/0x20
[218225.418216] mutex_lock+0x2c/0x30
[218225.418223] cifs_mark_open_files_invalid+0x5b/0xa0 [cifs]
[218225.418232] smb2_reconnect+0x149/0x7d0 [cifs]
[218225.418241] smb2_plain_req_init+0x34/0x260 [cifs]
[218225.418250] SMB2_open_init+0x69/0x760 [cifs]
[218225.418259] SMB2_open+0x148/0x510 [cifs]
[218225.418268] open_shroot+0x170/0x210 [cifs]
[218225.418277] ? open_shroot+0x170/0x210 [cifs]
[218225.418286] smb2_query_path_info+0x137/0x1c0 [cifs]
[218225.418287] ? _cond_resched+0x19/0x30
[218225.418289] ? _cond_resched+0x19/0x30
[218225.418290] ? kmem_cache_alloc_trace+0x153/0x1d0
[218225.418298] cifs_get_inode_info+0x283/0xb40 [cifs]
[218225.418306] ? build_path_from_dentry_optional_prefix+0xc4/0x410 [cifs]
[218225.418314] cifs_revalidate_dentry_attr+0xdd/0x3a0 [cifs]
[218225.418323] cifs_getattr+0x5a/0x1a0 [cifs]
[218225.418324] vfs_getattr_nosec+0x73/0x90
[218225.418324] vfs_getattr+0x36/0x40
[218225.418325] vfs_statx+0x8d/0xe0
[218225.418326] __do_sys_newstat+0x3d/0x70
[218225.418327] __x64_sys_newstat+0x16/0x20
[218225.418328] do_syscall_64+0x5a/0x110
[218225.418329] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[218225.418330] RIP: 0033:0x7fa10c4b1aa5
[218225.418332] Code: Bad RIP value.
[218225.418332] RSP: 002b:00007ffcfdb4cd08 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[218225.418333] RAX: ffffffffffffffda RBX: 0000563e189a4260 RCX: 00007fa10c4b1aa5
[218225.418333] RDX: 0000563e189a4478 RSI: 0000563e189a4478 RDI: 0000563e1a2cc990
[218225.418333] RBP: 0000563e1cce38d0 R08: ffffffffffffffff R09: 0000000000000000
[218225.418334] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[218225.418334] R13: 0000563e18267818 R14: 0000563e1a2cc990 R15: 0000000000000000

 

[JohnD]

I have the same issue.... any ideas?