Proxmox 8.1 NFS Connection Issues

[Alex000ch]

Ive set up a new Standalone Proxmox Host for testing purposes and got some strange timeouts with the NFS Shares.

My current Setup:

PM Testserver (10.10.250.10) stands on a Site which is connected with an S2S VPN. Stable connection and low latency (5-7ms). My NFS Storage (Synology NAS 10.10.0.220) is located on my other Site. Ive mounted 4 NFS Shares (3 from x.0.220 and 1 from x.0.221) 2x datastore for the VM's (1x HDD, 1x SSD) 1 Share and 1 Backup where i backup my production VMs. For testing i started to restore some VMs to my Test PM (10.10.250.10) (From Backup NFS 10.10.0.221 to Datastore 10.10.0.220) and at this point everything starts going crazy.

As soon as i start the Restore my NFS paths going offline and various Logs appear in the journal. Following a part from the start of the restore:

Dec 05 14:11:22 pm2 pvedaemon[28810]: <root@pam> successful auth for user 'root@pam'
Dec 05 14:11:58 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:12:58 pm2 pvedaemon[28808]: <root@pam> starting task UPID:pm2:00007AAE:00075CC1:656F21D9:vzrestore:200:root@pam:
Dec 05 14:13:00 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:13:09 pm2 kernel: loop0: detected capacity change from 0 to 41943040
Dec 05 14:13:10 pm2 kernel: EXT4-fs (loop0): mounted filesystem 5fbcecf2-2014-4006-a401-99abe2a1f7d6 r/w with ordered data mode. Quota mode: none.
Dec 05 14:13:13 pm2 kernel: operation not supported error, dev loop0, sector 8488 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:15 pm2 kernel: operation not supported error, dev loop0, sector 12576 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:19 pm2 kernel: operation not supported error, dev loop0, sector 16672 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:21 pm2 kernel: operation not supported error, dev loop0, sector 20768 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:23 pm2 kernel: operation not supported error, dev loop0, sector 24864 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:25 pm2 kernel: operation not supported error, dev loop0, sector 28960 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:33 pm2 kernel: operation not supported error, dev loop0, sector 33056 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:35 pm2 kernel: operation not supported error, dev loop0, sector 37152 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:37 pm2 kernel: operation not supported error, dev loop0, sector 41248 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:39 pm2 kernel: operation not supported error, dev loop0, sector 45344 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:41 pm2 kernel: operation not supported error, dev loop0, sector 49440 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:43 pm2 kernel: operation not supported error, dev loop0, sector 53536 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:51 pm2 kernel: operation not supported error, dev loop0, sector 57632 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:58 pm2 kernel: operation not supported error, dev loop0, sector 61728 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:13:59 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:14:00 pm2 kernel: operation not supported error, dev loop0, sector 65824 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:14:02 pm2 kernel: operation not supported error, dev loop0, sector 69920 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 05 14:14:15 pm2 pvestatd[1343]: got timeout
Dec 05 14:14:17 pm2 pvestatd[1343]: got timeout
Dec 05 14:14:19 pm2 pvestatd[1343]: got timeout
Dec 05 14:14:19 pm2 pvestatd[1343]: status update time (6.377 seconds)
Dec 05 14:14:58 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:16:00 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:16:59 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:17:01 pm2 CRON[32038]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 05 14:17:01 pm2 CRON[32039]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Dec 05 14:17:01 pm2 CRON[32038]: pam_unix(cron:session): session closed for user root
Dec 05 14:17:05 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 05 14:17:05 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 05 14:17:09 pm2 kernel: INFO: task kworker/u40:1:31234 blocked for more than 120 seconds.
Dec 05 14:17:09 pm2 kernel:       Tainted: P           O       6.5.11-4-pve #1
Dec 05 14:03:44 pm2 spiceproxy[1401]: server closing
Dec 05 14:03:44 pm2 spiceproxy[1401]: server shutdown (r>
Dec 05 14:03:44 pm2 systemd[1]: Reloaded spiceproxy.serv>
Dec 05 14:03:44 pm2 systemd[1]: Reloading pvescheduler.s>
Dec 05 14:03:44 pm2 spiceproxy[1401]: restarting server
Dec 05 14:03:44 pm2 spiceproxy[1401]: starting 1 worker(>
Dec 05 14:03:44 pm2 spiceproxy[1401]: worker 28817 start>
Dec 05 14:03:44 pm2 spiceproxy[1401]: server closing
Dec 05 14:03:44 pm2 spiceproxy[1401]: server shutdown (restart)
Dec 05 14:03:44 pm2 systemd[1]: Reloaded spiceproxy.service - PVE SPICE Proxy Server.
Dec 05 14:03:44 pm2 systemd[1]: Reloading pvescheduler.service - Proxmox VE scheduler...

Followed by:

Dec 05 14:17:13 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 05 14:17:13 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 05 14:17:28 pm2 pvedaemon[28810]: got timeout
Dec 05 14:17:28 pm2 pvedaemon[28810]: unable to activate storage 'N2_VmStoreSSD_NFS' - directory '/mnt/pve/N2_VmStoreSSD_NFS' does not exist or is unreachable
Dec 05 14:17:29 pm2 nfsidmap[32104]: nss_getpwnam: name 'root@localdomain' does not map into domain 'seclua2.local'
Dec 05 14:17:29 pm2 nfsidmap[32105]: nss_name_to_gid: name 'root@localdomain' does not map into domain 'seclua2.local'
Dec 05 14:17:30 pm2 kernel: nfs: server 10.10.0.220 not responding, timed out
Dec 05 14:17:31 pm2 pvedaemon[28810]: got timeout
Dec 05 14:17:31 pm2 pvedaemon[28810]: unable to activate storage 'N2_Share_NFS' - directory '/mnt/pve/N2_Share_NFS' does not exist or is unreachable
Dec 05 14:17:33 pm2 pvedaemon[28810]: got timeout
Dec 05 14:17:33 pm2 pvedaemon[28810]: unable to activate storage 'N2_VmStoreHDD_NFS' - directory '/mnt/pve/N2_VmStoreHDD_NFS' does not exist or is unreachable
Dec 05 14:17:49 pm2 pvedaemon[28808]: got timeout
Dec 05 14:17:49 pm2 pvedaemon[28808]: unable to activate storage 'N2_VmStoreSSD_NFS' - directory '/mnt/pve/N2_VmStoreSSD_NFS' does not exist or is unreachable
Dec 05 14:17:51 pm2 pvedaemon[28808]: got timeout
Dec 05 14:17:51 pm2 pvedaemon[28808]: unable to activate storage 'N2_Share_NFS' - directory '/mnt/pve/N2_Share_NFS' does not exist or is unreachable
Dec 05 14:17:53 pm2 pvedaemon[28808]: got timeout
Dec 05 14:17:53 pm2 pvedaemon[28808]: unable to activate storage 'N2_VmStoreHDD_NFS' - directory '/mnt/pve/N2_VmStoreHDD_NFS' does not exist or is unreachable
Dec 05 14:17:59 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:17:59 pm2 pvedaemon[28810]: got timeout
Dec 05 14:18:43 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 05 14:18:43 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 05 14:19:00 pm2 kernel: usb 4-1: reset high-speed USB device number 2 using xhci_hcd
Dec 05 14:19:10 pm2 kernel: INFO: task kworker/u40:1:31234 blocked for more than 241 seconds.
Dec 05 14:19:10 pm2 kernel:       Tainted: P           O       6.5.11-4-pve #1
Dec 05 14:19:10 pm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

First i thought that the connection drops while sending an high load of data but ive tested with iperf from Site A to Site B and had no problems while stressing the 100mbit/s up/download from the remote site. Also its kinda strange that all NFS Connections from the Dest. NAS drips while the NFS path to my backup NAS has no problems at all ( NFS is identically configured on both NAS).

Can anyone give me any hint or stumbled upon similar problems?
Following the Log file from the Journal. And yes, my HDD sda has some faulty sectors but sda is not in use in any way.

 

[sb-jw]

You should limit the bandwidth of the restore, that could possibly solve your problem because your uplinks are probably just full. Apart from that, such a setup is definitely not recommended and 5 - 7ms in storage is far from low latency.

 

[Alex000ch]

Well, i just tried to limit trafic to 50% of actual traffic but i still have the same issues. About the latency, this is another thing, its just a testserver and i dont expect hyper speed when running a couple of machines but copiing files should definitly not a problem caused by a couple of ms delay.


Just tested:

Dec 06 00:01:45 pm2 nfsidmap[117450]: nss_getpwnam: name 'root@localdomain' does not map into domain 'seclua2.local'
Dec 06 00:01:45 pm2 nfsidmap[117451]: nss_name_to_gid: name 'root@localdomain' does not map into domain 'seclua2.local'
Dec 06 00:04:25 pm2 pvedaemon[1196]: <root@pam> successful auth for user 'root@pam'
Dec 06 00:06:27 pm2 nfsidmap[118475]: nss_getpwnam: name 'guest' does not map into domain 'seclua2.local'
Dec 06 00:06:27 pm2 nfsidmap[118476]: nss_name_to_gid: name 'users' does not map into domain 'seclua2.local'
Dec 06 00:07:11 pm2 pvedaemon[1195]: <root@pam> starting task UPID:pm2:0001CF70:00340426:656FAD1F:vzrestore:201:root@pam:
Dec 06 00:07:18 pm2 kernel: loop0: detected capacity change from 0 to 31457280
Dec 06 00:07:18 pm2 kernel: EXT4-fs (loop0): mounted filesystem 7944b3d4-d5ba-4a10-af31-1942ed418000 r/w with ordered data mode. Quota mode: none.
Dec 06 00:07:20 pm2 kernel: operation not supported error, dev loop0, sector 8480 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 06 00:07:24 pm2 kernel: operation not supported error, dev loop0, sector 12568 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 06 00:07:27 pm2 kernel: operation not supported error, dev loop0, sector 16664 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 2
Dec 06 00:07:37 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:39 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:41 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:41 pm2 pvestatd[1165]: status update time (6.404 seconds)
Dec 06 00:07:47 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:49 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:51 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:52 pm2 pvestatd[1165]: status update time (6.389 seconds)
Dec 06 00:07:57 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:59 pm2 pvestatd[1165]: got timeout
Dec 06 00:07:59 pm2 pvestatd[1165]: unable to activate storage 'N2_VmStoreSSD_NFS' - directory '/mnt/pve/N2_VmStoreSSD_NFS' does not exist or is unreachable
Dec 06 00:08:01 pm2 pvestatd[1165]: got timeout
Dec 06 00:08:01 pm2 pvestatd[1165]: unable to activate storage 'N2_Share_NFS' - directory '/mnt/pve/N2_Share_NFS' does not exist or is unreachable
Dec 06 00:08:01 pm2 pvestatd[1165]: status update time (6.386 seconds)
Dec 06 00:08:07 pm2 pvestatd[1165]: got timeout
Dec 06 00:08:07 pm2 pvestatd[1165]: unable to activate storage 'N2_VmStoreSSD_NFS' - directory '/mnt/pve/N2_VmStoreSSD_NFS' does not exist or is unreachable
Dec 06 00:08:09 pm2 pvestatd[1165]: got timeout
Dec 06 00:08:09 pm2 pvestatd[1165]: unable to activate storage 'N2_Share_NFS' - directory '/mnt/pve/N2_Share_NFS' does not exist or is unreachable
Dec 06 00:08:11 pm2 pvestatd[1165]: got timeout
Dec 06 00:08:11 pm2 pvestatd[1165]: unable to activate storage 'N2_VmStoreHDD_NFS' - directory '/mnt/pve/N2_VmStoreHDD_NFS' does not exist or is unreachable
Dec 06 00:08:11 pm2 pvestatd[1165]: status update time (6.409 seconds)
<<>>>
Dec 06 00:10:29 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 06 00:10:29 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 06 00:10:29 pm2 pvestatd[1165]: got timeout
Dec 06 00:10:29 pm2 pvestatd[1165]: unable to activate storage 'N2_VmStoreSSD_NFS' - directory '/mnt/pve/N2_VmStoreSSD_NFS' does not exist or is unreachable
Dec 06 00:10:31 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
Dec 06 00:10:31 pm2 kernel: nfs: server 10.10.0.220 not responding, still trying
<<>>>
Dec 06 00:11:00 pm2 kernel: INFO: task kworker/u40:3:118604 blocked for more than 120 seconds.
Dec 06 00:11:00 pm2 kernel:       Tainted: P           O       6.5.11-6-pve #1
Dec 06 00:11:00 pm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 06 00:11:00 pm2 kernel: task:kworker/u40:3   state:D stack:0     pid:118604 ppid:2      flags:0x00004000
Dec 06 00:11:00 pm2 kernel: Workqueue: loop0 loop_rootcg_workfn
Dec 06 00:11:00 pm2 kernel: Call Trace:
Dec 06 00:11:00 pm2 kernel:  <TASK>
Dec 06 00:11:00 pm2 kernel:  __schedule+0x3fd/0x1450
Dec 06 00:11:00 pm2 kernel:  ? nfs_pageio_complete+0xee/0x140 [nfs]
Dec 06 00:11:00 pm2 kernel:  schedule+0x63/0x110
Dec 06 00:11:00 pm2 kernel:  io_schedule+0x46/0x80
Dec 06 00:11:00 pm2 kernel:  folio_wait_bit_common+0x136/0x330
Dec 06 00:11:00 pm2 kernel:  ? __pfx_wake_page_function+0x10/0x10
Dec 06 00:11:00 pm2 kernel:  folio_wait_bit+0x18/0x30
Dec 06 00:11:00 pm2 kernel:  folio_wait_writeback+0x2c/0xa0
Dec 06 00:11:00 pm2 kernel:  __filemap_fdatawait_range+0x90/0x100
Dec 06 00:11:00 pm2 kernel:  file_write_and_wait_range+0x93/0xc0
Dec 06 00:11:00 pm2 kernel:  nfs_file_fsync+0x99/0x1d0 [nfs]
Dec 06 00:11:00 pm2 kernel:  vfs_fsync+0x51/0xa0
Dec 06 00:11:00 pm2 kernel:  loop_process_work+0x3f0/0xa80
Dec 06 00:11:00 pm2 kernel:  ? __schedule+0x405/0x1450
Dec 06 00:11:00 pm2 kernel:  loop_rootcg_workfn+0x1b/0x30
Dec 06 00:11:00 pm2 kernel:  process_one_work+0x23e/0x450
Dec 06 00:11:00 pm2 kernel:  worker_thread+0x50/0x3f0
Dec 06 00:11:00 pm2 kernel:  ? __pfx_worker_thread+0x10/0x10
Dec 06 00:11:00 pm2 kernel:  kthread+0xf2/0x120
Dec 06 00:11:00 pm2 kernel:  ? __pfx_kthread+0x10/0x10
Dec 06 00:11:00 pm2 kernel:  ret_from_fork+0x47/0x70
Dec 06 00:11:00 pm2 kernel:  ? __pfx_kthread+0x10/0x10
Dec 06 00:11:00 pm2 kernel:  ret_from_fork_asm+0x1b/0x30
Dec 06 00:11:00 pm2 kernel:  </TASK>

 

[Alex000ch]

I also just tested the case when i try to "upload" any data from the local PM Disk to the NFS share i get the same results.

Also i noticed that occasionally the web gui starts to grey out all machines, the host included.
Somehow it works great for 30 mins and then i see lots of "operation not supported errors" and after a while (approx. 10 min) the promlems starts again.

After some more hours of testing, i tested the same Setup with SMB and i have absolutly the same problems, even with the same logs apart from way less error messages.

Dec 09 01:46:41 pm2 pveupdate[4684]: <root@pam> starting task UPID:pm2:00001255:00012161:6573B8F1:aptupdate::root@pam:
Dec 09 01:46:42 pm2 pveupdate[4693]: update new package list: /var/lib/pve-manager/pkgupdates
Dec 09 01:46:45 pm2 pveupdate[4684]: <root@pam> end task UPID:pm2:00001255:00012161:6573B8F1:aptupdate::root@pam: OK
Dec 09 01:46:45 pm2 systemd[1]: pve-daily-update.service: Deactivated successfully.
Dec 09 01:46:45 pm2 systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
Dec 09 01:46:45 pm2 systemd[1]: pve-daily-update.service: Consumed 3.891s CPU time.
Dec 09 01:46:56 pm2 pvestatd[1493]: status update time (15.162 seconds)
Dec 09 01:46:57 pm2 pvedaemon[1523]: <root@pam> starting task UPID:pm2:00001407:0001274C:6573B900:vzdestroy:5000:root@pam:
Dec 09 01:47:12 pm2 pvestatd[1493]: status update time (15.422 seconds)
Dec 09 01:47:28 pm2 pvestatd[1493]: status update time (16.692 seconds)
Dec 09 01:47:35 pm2 pveproxy[1533]: worker exit
Dec 09 01:47:35 pm2 pveproxy[1530]: worker 1533 finished
Dec 09 01:47:35 pm2 pveproxy[1530]: starting 1 worker(s)
Dec 09 01:47:35 pm2 pveproxy[1530]: worker 5276 started
Dec 09 01:47:45 pm2 pvestatd[1493]: status update time (16.787 seconds)
Dec 09 01:47:45 pm2 pvedaemon[1523]: got timeout
Dec 09 01:47:45 pm2 pvedaemon[1523]: unable to activate storage 'N1_VmBackup_SMB' - directory '/mnt/pve/N1_VmBackup_SMB' does not exist or is unreachable
Dec 09 01:48:00 pm2 pvestatd[1493]: status update time (15.297 seconds)
Dec 09 01:48:13 pm2 pvedaemon[1523]: <root@pam> end task UPID:pm2:00001407:0001274C:6573B900:vzdestroy:5000:root@pam: OK
Dec 09 01:48:17 pm2 pvestatd[1493]: status update time (16.595 seconds)
Dec 09 01:48:33 pm2 pvestatd[1493]: status update time (15.803 seconds)
Dec 09 01:48:46 pm2 pvestatd[1493]: status update time (13.638 seconds)
Dec 09 01:48:47 pm2 pvedaemon[1523]: <root@pam> successful auth for user 'root@pam'


<<<>>>

Dec 09 02:10:24 pm2 pveproxy[1530]: worker 12078 started
Dec 09 02:11:22 pm2 pvedaemon[1523]: <root@pam> starting task UPID:pm2:00003046:000363B6:6573BEBA:vzcreate:5000:root@pam:
Dec 09 02:11:25 pm2 kernel: loop0: detected capacity change from 0 to 104857600
Dec 09 02:11:25 pm2 kernel: EXT4-fs (loop0): mounted filesystem 364dd2b7-c73f-49c7-b1fa-1f42d1abab80 r/w with ordered data mode. Quota mode: none.
Dec 09 02:11:37 pm2 systemd[1]: var-lib-lxc-5000-rootfs.mount: Deactivated successfully.
Dec 09 02:11:50 pm2 pvestatd[1493]: status update time (16.367 seconds)
Dec 09 02:12:11 pm2 pveproxy[5276]: worker exit
Dec 09 02:12:11 pm2 pveproxy[1530]: worker 5276 finished
Dec 09 02:12:11 pm2 pveproxy[1530]: starting 1 worker(s)
Dec 09 02:12:11 pm2 pveproxy[1530]: worker 12580 started
Dec 09 02:12:17 pm2 pvestatd[1493]: status update time (27.472 seconds)
Dec 09 02:12:19 pm2 kernel: EXT4-fs (loop0): unmounting filesystem 364dd2b7-c73f-49c7-b1fa-1f42d1abab80.
Dec 09 02:12:20 pm2 pvedaemon[1523]: <root@pam> end task UPID:pm2:00003046:000363B6:6573BEBA:vzcreate:5000:root@pam: OK
Dec 09 02:12:22 pm2 kernel: CIFS: VFS: No writable handle in writepages rc=-9
Dec 09 02:12:36 pm2 pvedaemon[1524]: <root@pam> starting task UPID:pm2:0000319F:000380A2:6573BF04:vzstart:5000:root@pam:
Dec 09 02:12:36 pm2 pvedaemon[12703]: starting CT 5000: UPID:pm2:0000319F:000380A2:6573BF04:vzstart:5000:root@pam:
Dec 09 02:14:29 pm2 pveproxy[1531]: worker exit
Dec 09 02:14:29 pm2 pveproxy[1530]: worker 1531 finished
Dec 09 02:14:29 pm2 pveproxy[1530]: starting 1 worker(s)
Dec 09 02:14:29 pm2 pveproxy[1530]: worker 13254 started
Dec 09 02:14:39 pm2 kernel: INFO: task task UPID:pm2:0:12703 blocked for more than 120 seconds.
Dec 09 02:14:39 pm2 kernel:       Tainted: P           O       6.5.11-6-pve #1
Dec 09 02:14:39 pm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 09 02:14:39 pm2 kernel: task:task UPID:pm2:0 state:D stack:0     pid:12703 ppid:1524   flags:0x00004002
Dec 09 02:14:39 pm2 kernel: Call Trace:
Dec 09 02:14:39 pm2 kernel:  <TASK>
Dec 09 02:14:39 pm2 kernel:  __schedule+0x3fd/0x1450
Dec 09 02:14:39 pm2 kernel:  ? page_counter_cancel+0x33/0x90
Dec 09 02:14:39 pm2 kernel:  schedule+0x63/0x110
Dec 09 02:14:39 pm2 kernel:  io_schedule+0x46/0x80
Dec 09 02:14:39 pm2 kernel:  folio_wait_bit_common+0x136/0x330
Dec 09 02:14:39 pm2 kernel:  ? __pfx_wake_page_function+0x10/0x10
Dec 09 02:14:39 pm2 kernel:  __folio_lock+0x17/0x30
Dec 09 02:14:39 pm2 kernel:  invalidate_inode_pages2_range+0x178/0x460
Dec 09 02:14:39 pm2 kernel:  invalidate_inode_pages2+0x17/0x30
Dec 09 02:14:39 pm2 kernel:  cifs_invalidate_mapping+0x3b/0x80 [cifs]
Dec 09 02:14:39 pm2 kernel:  cifs_revalidate_mapping+0xc2/0xe0 [cifs]
Dec 09 02:14:39 pm2 kernel:  cifs_revalidate_dentry+0x1f/0x30 [cifs]
Dec 09 02:14:39 pm2 kernel:  cifs_d_revalidate+0x5f/0x180 [cifs]
Dec 09 02:14:39 pm2 kernel:  lookup_fast+0x83/0x100
Dec 09 02:14:39 pm2 kernel:  walk_component+0x2c/0x190
Dec 09 02:14:39 pm2 kernel:  path_lookupat+0x67/0x1a0
Dec 09 02:14:39 pm2 kernel:  ? __alloc_pages+0x254/0x12e0
Dec 09 02:14:39 pm2 kernel:  filename_lookup+0xe4/0x200
Dec 09 02:14:39 pm2 kernel:  ? strncpy_from_user+0x50/0x170
Dec 09 02:14:39 pm2 kernel:  vfs_statx+0xa1/0x180
Dec 09 02:14:39 pm2 kernel:  vfs_fstatat+0x58/0x80
Dec 09 02:14:39 pm2 kernel:  __do_sys_newfstatat+0x44/0x90
Dec 09 02:14:39 pm2 kernel:  __x64_sys_newfstatat+0x1c/0x30
Dec 09 02:14:39 pm2 kernel:  do_syscall_64+0x5b/0x90
Dec 09 02:14:39 pm2 kernel:  ? handle_mm_fault+0xad/0x360
Dec 09 02:14:39 pm2 kernel:  ? exit_to_user_mode_prepare+0x39/0x190
Dec 09 02:14:39 pm2 kernel:  ? irqentry_exit_to_user_mode+0x17/0x20
Dec 09 02:14:39 pm2 kernel:  ? irqentry_exit+0x43/0x50
Dec 09 02:14:39 pm2 kernel:  ? exc_page_fault+0x94/0x1b0
Dec 09 02:14:39 pm2 kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Dec 09 02:14:39 pm2 kernel: RIP: 0033:0x7fa38212765a
Dec 09 02:14:39 pm2 kernel: RSP: 002b:00007ffec4641f68 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Dec 09 02:14:39 pm2 kernel: RAX: ffffffffffffffda RBX: 000055bfd338e2a0 RCX: 00007fa38212765a
Dec 09 02:14:39 pm2 kernel: RDX: 000055bfd338e4a8 RSI: 000055bfdb2095c0 RDI: 00000000ffffff9c
Dec 09 02:14:39 pm2 kernel: RBP: 000055bfd33c2330 R08: 000055bfdb2d94a0 R09: 000055bfdb2d94a0
Dec 09 02:14:39 pm2 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055bfdb2095c0
Dec 09 02:14:39 pm2 kernel: R13: 000055bfd1d5a16b R14: 0000000000000000 R15: 00007fa38232b020
Dec 09 02:14:39 pm2 kernel:  </TASK>