LXCs with NFS mounts failing to start after reboot or restore

Luxebeara

New Member
Apr 21, 2025
5
0
1
Hi,

Running a cluster on pve-manager/9.1.5/80cf92a64bef6889 / Linux 6.17.9-1-pve

Powered down a node to move it, on reboot, many but not all the LXCs with NFS mount points fail to start giving:



Code:
run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "109"
__lxc_start: 2046 Failed to initialize container "109"
TASK ERROR: startup for container '109' failed

an LXC that was still working failed after I backed it up and restored it.

Tested a freshly installed PVE machine not in the cluster and once I add the mount point it fails to start with the same error.

Very weird, anyone encountered similar in the past few days that can enlighten me?

NAS is running TrueNas Scale 25.10.1

We had a power cut recently, maybe that's a factor???

I'm baffled.

Many thanks
 
Last edited:
Hello All,

The issue has been fixed today. In case you have not upgraded already, you can do it now!
By the way, here are the commands for you:

Bash:
apt-mark unhold pve-container
apt update
apt full-upgrade

there is also a new Kernel to download. This will cost you an node reboot.

Here the commands

Bash:
# Dist Upgrade
apt-get dist-upgrade

# Cleanup some Storage
apt-get autoremove

Have a nice Day!
Chris
 
Last edited:
Unfortunately this is not fixed.

My infra es home lab with
proxmox-ve: 9.1.0 (running kernel: 6.17.9-1-pve)pve-manager: 9.1.5 (running version: 9.1.5/80cf92a64bef6889) 2 nodes.

Node has an NFS exported storage and HA VM, among other things.

I backup HA VM via the storage that is actually mounted/exported via NFS in the same machine. Not sure if that should create any issues. I assume no.

My logs show

Code:
Feb 21 03:00:02 pve pvescheduler[2273299]: <root@pam> starting task UPID:pve:0022B014:024213CA:699911A2:vzdump:114:root@pam:
Feb 21 03:00:07 pve pvescheduler[2273300]: INFO: starting new backup job: vzdump 114 --notification-mode notification-system --node pve --fleecing 0 --compress zstd --notes-template '{{guestname}}' --storage T3_2_VM --prune-backups 'keep-monthly=1,keep-weekly=1' --quiet 1 --mode snapshot
Feb 21 03:00:07 pve pvescheduler[2273300]: INFO: Starting Backup of VM 114 (qemu)
Feb 21 03:00:07 pve pvestatd[1204]: status update time (6.732 seconds)
Feb 21 03:01:08 pve pvestatd[1204]: status update time (6.894 seconds)
Feb 21 03:01:13 pve pvestatd[1204]: got timeout
Feb 21 03:03:59 pve kernel: INFO: task nfsd:1002813 blocked for more than 122 seconds.
Feb 21 03:03:59 pve kernel:       Tainted: P S         O        6.17.9-1-pve #1
Feb 21 03:03:59 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 21 03:03:59 pve kernel: task:nfsd            state:D stack:0     pid:1002813 tgid:1002813 ppid:2      task_flags:0x300040 flags:0x00004000
Feb 21 03:03:59 pve kernel: Call Trace:
Feb 21 03:03:59 pve kernel:  <TASK>
Feb 21 03:03:59 pve kernel:  __schedule+0x468/0x1310
Feb 21 03:03:59 pve kernel:  schedule+0x27/0xf0
Feb 21 03:03:59 pve kernel:  schedule_preempt_disabled+0x15/0x30
Feb 21 03:03:59 pve kernel:  rwsem_down_write_slowpath+0x268/0x6d0
Feb 21 03:03:59 pve kernel:  ? nfsd_current_rqst+0x3b/0x50 [nfsd]
Feb 21 03:03:59 pve kernel:  ? aa_file_perm+0x60/0x3b0
Feb 21 03:03:59 pve kernel:  down_write+0x5b/0x80
Feb 21 03:03:59 pve kernel:  ext4_buffered_write_iter+0x36/0x150
Feb 21 03:03:59 pve kernel:  ext4_file_write_iter+0xb1/0x900
Feb 21 03:03:59 pve kernel:  ? apparmor_file_permission+0x1f/0x30
Feb 21 03:03:59 pve kernel:  ? security_file_permission+0x36/0x60
Feb 21 03:03:59 pve kernel:  ? rw_verify_area+0x57/0x190
Feb 21 03:03:59 pve kernel:  vfs_iocb_iter_write+0xd7/0x230
Feb 21 03:03:59 pve kernel:  nfsd_vfs_write+0x264/0x670 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd4_write+0x124/0x1b0 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd4_proc_compound+0x3b0/0x770 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd_dispatch+0xcd/0x220 [nfsd]
Feb 21 03:03:59 pve kernel:  svc_process_common+0x4d3/0x6c0 [sunrpc]
Feb 21 03:03:59 pve kernel:  ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
Feb 21 03:03:59 pve kernel:  svc_process+0x136/0x1f0 [sunrpc]
Feb 21 03:03:59 pve kernel:  svc_recv+0x7e8/0x9c0 [sunrpc]
Feb 21 03:03:59 pve kernel:  ? __pfx_nfsd+0x10/0x10 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd+0x90/0xf0 [nfsd]
Feb 21 03:03:59 pve kernel:  kthread+0x108/0x220
Feb 21 03:03:59 pve kernel:  ? __pfx_kthread+0x10/0x10
Feb 21 03:03:59 pve kernel:  ret_from_fork+0x205/0x240
Feb 21 03:03:59 pve kernel:  ? __pfx_kthread+0x10/0x10
Feb 21 03:03:59 pve kernel:  ret_from_fork_asm+0x1a/0x30
Feb 21 03:03:59 pve kernel:  </TASK>
Feb 21 03:03:59 pve kernel: INFO: task nfsd:1002813 <writer> blocked on an rw-semaphore likely owned by task nfsd:1002819 <writer>
Feb 21 03:03:59 pve kernel: task:nfsd            state:D stack:0     pid:1002819 tgid:1002819 ppid:2      task_flags:0x300840 flags:0x00004000
Feb 21 03:03:59 pve kernel: Call Trace:
Feb 21 03:03:59 pve kernel:  <TASK>
Feb 21 03:03:59 pve kernel:  __schedule+0x468/0x1310
Feb 21 03:03:59 pve kernel:  ? rpc_run_task+0x10f/0x180 [sunrpc]
Feb 21 03:03:59 pve kernel:  schedule+0x27/0xf0
Feb 21 03:03:59 pve kernel:  wait_on_commit+0x9e/0xe0 [nfs]
Feb 21 03:03:59 pve kernel:  ? __pfx_var_wake_function+0x10/0x10
Feb 21 03:03:59 pve kernel:  __nfs_commit_inode+0xd2/0x1d0 [nfs]
Feb 21 03:03:59 pve kernel:  nfs_wb_folio+0xbb/0x1e0 [nfs]
Feb 21 03:03:59 pve kernel:  nfs_release_folio+0x72/0x110 [nfs]
Feb 21 03:03:59 pve kernel:  filemap_release_folio+0x62/0xa0
Feb 21 03:03:59 pve kernel:  __folio_split+0x679/0x1800
Feb 21 03:03:59 pve kernel:  ? __isolate_free_page+0x1dc/0x430
Feb 21 03:03:59 pve kernel:  split_huge_page_to_list_to_order+0x2e/0xb0
Feb 21 03:03:59 pve kernel:  split_folio_to_list+0x10/0x20
Feb 21 03:03:59 pve kernel:  migrate_pages_batch+0x463/0xec0
Feb 21 03:03:59 pve kernel:  ? __pfx_compaction_alloc+0x10/0x10
Feb 21 03:03:59 pve kernel:  ? __pfx_compaction_free+0x10/0x10
Feb 21 03:03:59 pve kernel:  migrate_pages+0x9a7/0xda0
Feb 21 03:03:59 pve kernel:  ? folio_add_lru+0x49/0x60
Feb 21 03:03:59 pve kernel:  ? __pfx_compaction_free+0x10/0x10
Feb 21 03:03:59 pve kernel:  ? __pfx_compaction_alloc+0x10/0x10
Feb 21 03:03:59 pve kernel:  compact_zone+0xb8e/0x1150
Feb 21 03:03:59 pve kernel:  compact_zone_order+0xa4/0x100
Feb 21 03:03:59 pve kernel:  try_to_compact_pages+0xde/0x2b0
Feb 21 03:03:59 pve kernel:  __alloc_pages_direct_compact+0x90/0x210
Feb 21 03:03:59 pve kernel:  __alloc_frozen_pages_noprof+0x5ba/0x12a0
Feb 21 03:03:59 pve kernel:  ? xas_load+0x15/0xf0
Feb 21 03:03:59 pve kernel:  ? policy_nodemask+0x111/0x190
Feb 21 03:03:59 pve kernel:  alloc_pages_mpol+0x80/0x180
Feb 21 03:03:59 pve kernel:  folio_alloc_noprof+0x5b/0xc0
Feb 21 03:03:59 pve kernel:  filemap_alloc_folio_noprof+0xe1/0xf0
Feb 21 03:03:59 pve kernel:  __filemap_get_folio+0x18c/0x340
Feb 21 03:03:59 pve kernel:  ext4_da_write_begin+0x143/0x360
Feb 21 03:03:59 pve kernel:  generic_perform_write+0x134/0x2d0
Feb 21 03:03:59 pve kernel:  ext4_buffered_write_iter+0x6d/0x150
Feb 21 03:03:59 pve kernel:  ext4_file_write_iter+0xb1/0x900
Feb 21 03:03:59 pve kernel:  ? apparmor_file_permission+0x1f/0x30
Feb 21 03:03:59 pve kernel:  ? security_file_permission+0x36/0x60
Feb 21 03:03:59 pve kernel:  ? rw_verify_area+0x57/0x190
Feb 21 03:03:59 pve kernel:  vfs_iocb_iter_write+0xd7/0x230
Feb 21 03:03:59 pve kernel:  nfsd_vfs_write+0x264/0x670 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd4_write+0x124/0x1b0 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd4_proc_compound+0x3b0/0x770 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd_dispatch+0xcd/0x220 [nfsd]
Feb 21 03:03:59 pve kernel:  svc_process_common+0x4d3/0x6c0 [sunrpc]
Feb 21 03:03:59 pve kernel:  ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
Feb 21 03:03:59 pve kernel:  svc_process+0x136/0x1f0 [sunrpc]
Feb 21 03:03:59 pve kernel:  svc_recv+0x7e8/0x9c0 [sunrpc]
Feb 21 03:03:59 pve kernel:  ? __pfx_nfsd+0x10/0x10 [nfsd]
Feb 21 03:03:59 pve kernel:  nfsd+0x90/0xf0 [nfsd]
Feb 21 03:03:59 pve kernel:  kthread+0x108/0x220
Feb 21 03:03:59 pve kernel:  ? __pfx_kthread+0x10/0x10
Feb 21 03:03:59 pve kernel:  ret_from_fork+0x205/0x240
Feb 21 03:03:59 pve kernel:  ? __pfx_kthread+0x10/0x10
Feb 21 03:03:59 pve kernel:  ret_from_fork_asm+0x1a/0x30
Feb 21 03:03:59 pve kernel:  </TASK>

.
.
.
.
Feb 21 03:04:00 pve kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Feb 21 03:10:01 pve CRON[2274974]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 21 03:10:01 pve CRON[2274976]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Feb 21 03:10:01 pve CRON[2274974]: pam_unix(cron:session): session closed for user root
Feb 21 03:17:01 pve CRON[2275933]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 21 03:17:01 pve CRON[2275935]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 21 03:17:01 pve CRON[2275933]: pam_unix(cron:session): session closed for user root
Feb 21 03:17:11 pve systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
Feb 21 03:17:12 pve pveupdate[2275960]: <root@pam> starting task UPID:pve:0022BA7D:0243A5FE:699915A8:aptupdate::root@pam:
Feb 21 03:17:13 pve pveupdate[2275965]: update new package list: /var/lib/pve-manager/pkgupdates
Feb 21 03:17:16 pve pveupdate[2275960]: <root@pam> end task UPID:pve:0022BA7D:0243A5FE:699915A8:aptupdate::root@pam: OK
Feb 21 03:17:16 pve systemd[1]: pve-daily-update.service: Deactivated successfully.
Feb 21 03:17:16 pve systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
Feb 21 03:17:16 pve systemd[1]: pve-daily-update.service: Consumed 4.503s CPU time, 324.8M memory peak.
Feb 21 03:20:48 pve pmxcfs[1003]: [status] notice: received log
Feb 21 03:20:51 pve pmxcfs[1003]: [status] notice: received log
Feb 21 03:21:57 pve pmxcfs[1003]: [dcdb] notice: data verification successful
Feb 21 03:31:36 pve postfix/qmgr[1149]: 6DB46360609: from=<adminno@pve.hp>, size=450, nrcpt=1 (queue active)
Feb 21 03:31:36 pve postfix/local[2277951]: error: open database /etc/aliases.db: No such file or directory
Feb 21 03:31:36 pve postfix/local[2277951]: warning: hash:/etc/aliases is unavailable. open database /etc/aliases.db: No such file or directory
Feb 21 03:31:36 pve postfix/local[2277951]: warning: hash:/etc/aliases: lookup of 'root' failed
Feb 21 03:31:36 pve postfix/local[2277951]: 6DB46360609: to=<root@pve.hp>, orig_to=<root>, relay=local, delay=382028, delays=382028/0.01/0/0.01, dsn=4.3.0, status=deferred (alias database unavailable)
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:37:14 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:46:39 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:46:39 pve kernel: nfs: server pve.lan not responding, timed out
Feb 21 03:46:39 pve kernel: rpc_check_timeout: 261 callbacks suppressed

Feb 21 03:47:44 pve pvestatd[1204]: got timeout
Feb 21 03:47:44 pve pvestatd[1204]: unable to activate storage 'T3_2_VM' - directory '/mnt/pve/T3_2_VM' does not exist or is unreachable
Feb 21 03:47:54 pve pvestatd[1204]: got timeout
Feb 21 03:47:54 pve pvestatd[1204]: unable to activate storage 'T3_2_VM' - directory '/mnt/pve/T3_2_VM' does not exist or is unreachable
Feb 21 03:48:09 pve pvestatd[1204]: got timeout
Feb 21 03:48:09 pve pvestatd[1204]: unable to activate storage 'T3_2_VM' - directory '/mnt/pve/T3_2_VM' does not exist or is unreachable
Feb 21 03:48:09 pve pvestatd[1204]: status update time (6.773 seconds)
Feb 21 03:48:15 pve pvestatd[1204]: got timeout

I do have the infamouns process in D state....so I am in my way to physically kill the manchine as not even reboot the node would work.
1771661403552.png
Hope this helps to nail down the issue. Let me know if I can help on anything providing more context

JC
 
Last edited:
Hello Julian33,

I think you have a different issue.

I have these software versions:

Code:
Header
Proxmox
Virtual Environment 9.1.5
Search
... / ...
Kernel Version Linux 6.17.9-1-pve (2026-01-12T16:25Z)
Boot Mode EFI
Manager Version pve-manager/9.1.5/80cf92a64bef6889
Repository Status Proxmox VE updates Non production-ready repository enabled!
Server View
Logs
()
proxmox-ve: 9.1.0 (running kernel: 6.17.9-1-pve)
pve-manager: 9.1.5 (running version: 9.1.5/80cf92a64bef6889)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17.9-1-pve-signed: 6.17.9-1
proxmox-kernel-6.17: 6.17.9-1
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.14.11-5-pve-signed: 6.14.11-5
proxmox-kernel-6.14: 6.14.11-5
proxmox-kernel-6.8: 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
.../...

I see you have timeout errors. This is a strange other issue. Is your NAS up and running? Firewall? ...

Best Regards

Chris