Why doesn't LXC fstab mount NFS mounts on startup?

A

agster27

Member
Proxmox Subscriber
Feb 27, 2023
17
3
8
Boston
Hello!

I've searched and searched and no answer or fix. Why doesn't NFS mounts in FStab mount when the container boots up. It is a privileged container, and when I run mount -a at the command line they will mount no problem.

Please help!

-Michael
 
agster27 said:
Hello!

I've searched and searched and no answer or fix. Why doesn't NFS mounts in FStab mount when the container boots up. It is a privileged container, and when I run mount -a at the command line they will mount no problem.

Please help!

-Michael
Click to expand...
Hi,
what container template are you using? Maybe there is something in the systemd journal/syslog of the container which indicates why the share is not mounted (could be e.g. missing network connectivity)?
 
Chris said:
Hi,
what container template are you using? Maybe there is something in the systemd journal/syslog of the container which indicates why the share is not mounted (could be e.g. missing network connectivity)?
Click to expand...


LXC Template is Ubuntu 22.04

So I can reproduce this very easily. If I reboot the LXC the mounts do not mount and the syslog does show "network is unreachable" errors. The weird thing is there is no issue with the network. I've included the Syslog and interfaces below. Not sure where to troubleshoot next.


auto lo iface lo inet loopback iface enp108s0 inet manual iface enp109s0 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.1.11/24 gateway 192.168.1.1 bridge-ports enp108s0 bridge-stp off bridge-fd 0 iface wlp0s20f3 inet manual auto vmbr1 iface vmbr1 inet static address 192.168.2.2/30 bridge-ports enp109s0 bridge-stp off bridge-fd 0



Jan 30 21:06:38 plex systemd-resolved[110]: Using system hostname 'plex'. Jan 30 21:06:38 plex systemd[1]: Started Network Name Resolution. Jan 30 21:06:38 plex systemd[1]: Reached target Network. Jan 30 21:06:38 plex systemd[1]: Reached target Network is Online. Jan 30 21:06:38 plex systemd[1]: Reached target Host and Network Name Lookups. Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/NZBGet_intermediate... Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/docker... Jan 30 21:06:38 plex systemd-networkd[88]: eth0: Gained carrier Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/downloads... Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/movies... Jan 30 21:06:38 plex mount[120]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/music... Jan 30 21:06:38 plex mount[122]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/photos... Jan 30 21:06:38 plex mount[124]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/software... Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/temp... Jan 30 21:06:38 plex mount[126]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex mount[128]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Mounting /mnt/tv... Jan 30 21:06:38 plex mount[130]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Starting Plex Media Server... Jan 30 21:06:38 plex mount[132]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Starting Postfix Mail Transport Agent (instance -)... Jan 30 21:06:38 plex mount[134]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Starting Notify NFS peers of a restart... Jan 30 21:06:38 plex systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped. Jan 30 21:06:38 plex systemd[1]: Condition check resulted in Ubuntu Pro Background Auto Attach being skipped. Jan 30 21:06:38 plex mount[137]: mount.nfs4: Network is unreachable Jan 30 21:06:38 plex systemd[1]: Started Dispatcher daemon for systemd-networkd. Jan 30 21:06:38 plex sm-notify[138]: Version 2.6.1 starting Jan 30 21:06:38 plex systemd[1]: mnt-NZBGet_intermediate.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-NZBGet_intermediate.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/NZBGet_intermediate. Jan 30 21:06:38 plex systemd[1]: Dependency failed for Remote File Systems. Jan 30 21:06:38 plex systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'. Jan 30 21:06:38 plex systemd[1]: mnt-docker.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-docker.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/docker. Jan 30 21:06:38 plex systemd[1]: mnt-downloads.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-downloads.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/downloads. Jan 30 21:06:38 plex systemd[1]: mnt-movies.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-movies.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/movies. Jan 30 21:06:38 plex systemd[1]: mnt-music.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-music.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/music. Jan 30 21:06:38 plex systemd[1]: mnt-photos.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-photos.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/photos. Jan 30 21:06:38 plex systemd[1]: mnt-software.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-software.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/software. Jan 30 21:06:38 plex systemd[1]: mnt-temp.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-temp.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/temp. Jan 30 21:06:38 plex systemd[1]: mnt-tv.mount: Mount process exited, code=exited, status=32/n/a Jan 30 21:06:38 plex systemd[1]: mnt-tv.mount: Failed with result 'exit-code'. Jan 30 21:06:38 plex systemd[1]: Failed to mount /mnt/tv. Jan 30 21:06:38 plex systemd[1]: Started Notify NFS peers of a restart. Jan 30 21:06:38 plex systemd[1]: Started Regular background program processing daemon. Jan 30 21:06:38 plex systemd[1]: Starting Permit User Sessions... Jan 30 21:06:38 plex cron[143]: (CRON) INFO (pidfile fd = 3) Jan 30 21:06:38 plex cron[143]: (CRON) INFO (Running @reboot jobs) Jan 30 21:06:38 plex systemd[1]: Started Plex Media Server. Jan 30 21:06:38 plex systemd[1]: Finished Permit User Sessions. Jan 30 21:06:38 plex systemd[1]: Started Console Getty. Jan 30 21:06:38 plex systemd[1]: Started Container Getty on /dev/tty1. Jan 30 21:06:38 plex systemd[1]: Started Container Getty on /dev/tty2. Jan 30 21:06:38 plex systemd[1]: Reached target Login Prompts. Jan 30 21:06:38 plex systemd[1]: Condition check resulted in Set console scheme being skipped. Jan 30 21:06:38 plex postfix[211]: Postfix is running with backwards-compatible default settings Jan 30 21:06:38 plex postfix[211]: See http://www.postfix.org/COMPATIBILITY_README.html for details Jan 30 21:06:38 plex postfix[211]: To disable backwards compatibility use "postconf compatibility_level=3.6" and "postfix reload" Jan 30 21:06:38 plex systemd-networkd[88]: eth0: DHCPv4 address 192.168.1.102/24 via 192.168.1.1 Jan 30 21:06:38 plex dbus-daemon[104]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.0' (uid=105 pid=88 comm="/lib/systemd/systemd-networkd " label="unconfined") Jan 30 21:06:38 plex systemd[1]: Starting Hostname Service... Jan 30 21:06:38 plex dbus-daemon[104]: [system] Successfully activated service 'org.freedesktop.hostname1' Jan 30 21:06:38 plex systemd[1]: Started Hostname Service. Jan 30 21:06:38 plex systemd-networkd[88]: Could not set hostname: Access denied Jan 30 21:06:38 plex postfix/postfix-script[329]: starting the Postfix mail system Jan 30 21:06:38 plex postfix/master[331]: daemon started -- version 3.6.4, configuration /etc/postfix Jan 30 21:06:38 plex systemd[1]: Started Postfix Mail Transport Agent (instance -). Jan 30 21:06:38 plex systemd[1]: Starting Postfix Mail Transport Agent... Jan 30 21:06:38 plex systemd[1]: Finished Postfix Mail Transport Agent. Jan 30 21:06:38 plex systemd[1]: Reached target Multi-User System. Jan 30 21:06:38 plex systemd[1]: Reached target Graphical Interface. Jan 30 21:06:38 plex systemd[1]: Starting Record Runlevel Change in UTMP... Jan 30 21:06:38 plex systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully. Jan 30 21:06:38 plex systemd[1]: Finished Record Runlevel Change in UTMP. Jan 30 21:06:38 plex systemd[1]: Startup finished in 317ms. Jan 30 21:06:38 plex systemd[1]: dmesg.service: Deactivated successfully. Jan 30 21:06:40 plex systemd-networkd[88]: eth0: Gained IPv6LL Jan 30 21:06:50 plex systemd[1]: Created slice User Slice of UID 0. Jan 30 21:06:50 plex systemd[1]: Starting User Runtime Directory /run/user/0... Jan 30 21:06:50 plex systemd[1]: Finished User Runtime Directory /run/user/0. Jan 30 21:06:50 plex systemd[1]: Starting User Manager for UID 0... Jan 30 21:06:50 plex systemd[447]: Queued start job for default target Main User Target. Jan 30 21:06:50 plex systemd[447]: Created slice User Application Slice. Jan 30 21:06:50 plex systemd[447]: Reached target Paths. Jan 30 21:06:50 plex systemd[447]: Reached target Timers. Jan 30 21:06:50 plex systemd[447]: Listening on GnuPG network certificate management daemon. Jan 30 21:06:50 plex systemd[447]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers). Jan 30 21:06:50 plex systemd[447]: Listening on GnuPG cryptographic agent and passphrase cache (restricted). Jan 30 21:06:50 plex systemd[447]: Listening on GnuPG cryptographic agent (ssh-agent emulation). Jan 30 21:06:50 plex systemd[447]: Listening on GnuPG cryptographic agent and passphrase cache. Jan 30 21:06:50 plex systemd[447]: Reached target Sockets. Jan 30 21:06:50 plex systemd[447]: Reached target Basic System. Jan 30 21:06:50 plex systemd[447]: Reached target Main User Target. Jan 30 21:06:50 plex systemd[447]: Startup finished in 10ms. Jan 30 21:06:50 plex systemd[1]: Started User Manager for UID 0. Jan 30 21:06:50 plex systemd[1]: Started Session 34 of User root. Jan 30 21:06:51 plex : Critical: libusb_init failed Jan 30 21:06:51 plex Plex Media Server[392]: Critical: libusb_init failed Jan 30 21:07:08 plex systemd[1]: systemd-hostnamed.service: Deactivated successfully.
 
agster27 said:
If I reboot the LXC the mounts do not mount and the syslog does show "network is unreachable" errors. The weird thing is there is no issue with the network.
Click to expand...
agster27 said:
systemd-networkd[88]: eth0: DHCPv4 address 192.168.1.102/24 via 192.168.1.1
Click to expand...

This log line indicates that the IPv4 address for the container is only obtained after the attempts to mount the nfs share. So the network is not reachable at the time of the attempted mounts, only after that.
 
Chris said:
This log line indicates that the IPv4 address for the container is only obtained after the attempts to mount the nfs share. So the network is not reachable at the time of the attempted mounts, only after that.
Click to expand...


Two questions:

1.) Is there a way to delay the NFS mount?
2.) If I configure a static IP I would assume this problem would go away.
 
agster27 said:
Is there a way to delay the NFS mount?
Click to expand...
Have a look at systemd mount options, e.g. you could delay the mount and/or make sure it depends on the network.target, which should be the one to bring up the interfaces and configure them. systemd-analyze critical-chain and systemctl list-dependencies might help getting the dependencies right. See also https://wiki.archlinux.org/title/Fstab#Remote_file_system

agster27 said:
If I configure a static IP I would assume this problem would go away.
Click to expand...
Not necessarily, this depends on whether the IP is configured before or after the mount unit is executed. Check your dependency graph, this should tell you more.

Edit: Oh I overlooked that systemd-networkd is setting up the IP address in your case, so the dependency should be on that I guess.
 
Last edited:
Chris said:
Have a look at systemd mount options, e.g. you could delay the mount and/or make sure it depends on the network.target, which should be the one to bring up the interfaces and configure them. systemd-analyze critical-chain and systemctl list-dependencies might help getting the dependencies right. See also https://wiki.archlinux.org/title/Fstab#Remote_file_system


Not necessarily, this depends on whether the IP is configured before or after the mount unit is executed. Check your dependency graph, this should tell you more.

Edit: Oh I overlooked that systemd-networkd is setting up the IP address in your case, so the dependency should be on that I guess.
Click to expand...


I so tried this in /etc/fstab and the same issue.

192.168.1.101:/volume1/downloads /mnt/downloads nfs4 auto,defaults,x-systemd.requires=systemd-networkd

Still same issue. Below is the output of:
systemd-analyze critical-chain and systemctl list-dependencies
Click to expand...

Any thoughts would be appreciated... This is driving me nuts..

Code: 
root@plex:/etc# systemctl list-dependencies
default.target
* |-display-manager.service
* |-systemd-update-utmp-runlevel.service
* `-multi-user.target
*   |-console-setup.service
*   |-cron.service
*   |-dbus.service
*   |-dmesg.service
*   |-e2scrub_reap.service
*   |-irqbalance.service
*   |-networkd-dispatcher.service
*   |-plexmediaserver.service
*   |-postfix.service
*   |-rpcbind.service
*   |-rsync.service
*   |-rsyslog.service
*   |-ssh.service
*   |-systemd-ask-password-wall.path
*   |-systemd-logind.service
*   |-systemd-networkd.service
*   |-systemd-resolved.service
*   |-systemd-update-utmp-runlevel.service
*   |-systemd-user-sessions.service
*   |-ua-reboot-cmds.service
*   |-ubuntu-advantage.service
*   |-ufw.service
*   |-basic.target
*   | |-tmp.mount
*   | |-paths.target
*   | |-slices.target
*   | | |--.slice
*   | | `-system.slice
*   | |-sockets.target
*   | | |-dbus.socket
*   | | |-rpcbind.socket
*   | | |-ssh.socket
*   | | |-systemd-initctl.socket
*   | | |-systemd-journald-audit.socket
*   | | |-systemd-journald-dev-log.socket
*   | | |-systemd-journald.socket
*   | | |-systemd-networkd.socket
*   | | |-systemd-udevd-control.socket
*   | | |-systemd-udevd-kernel.socket
*   | | `-uuidd.socket
*   | |-sysinit.target
*   | | |-apparmor.service
*   | | |-dev-hugepages.mount
*   | | |-dev-mqueue.mount
*   | | |-keyboard-setup.service
*   | | |-kmod-static-nodes.service
*   | | |-nftables.service
*   | | |-proc-sys-fs-binfmt_misc.automount
*   | | |-setvtrgb.service
*   | | |-sys-fs-fuse-connections.mount
*   | | |-sys-kernel-config.mount
*   | | |-sys-kernel-debug.mount
*   | | |-sys-kernel-tracing.mount
*   | | |-systemd-ask-password-console.path
*   | | |-systemd-binfmt.service
*   | | |-systemd-boot-system-token.service
*   | | |-systemd-journal-flush.service
*   | | |-systemd-journald.service
*   | | |-systemd-machine-id-commit.service
*   | | |-systemd-modules-load.service
*   | | |-systemd-network-generator.service
*   | | |-systemd-pstore.service
*   | | |-systemd-random-seed.service
*   | | |-systemd-sysctl.service
*   | | |-systemd-sysext.service
*   | | |-systemd-sysusers.service
*   | | |-systemd-timesyncd.service
*   | | |-systemd-tmpfiles-setup-dev.service
*   | | |-systemd-tmpfiles-setup.service
*   | | |-systemd-udev-trigger.service
*   | | |-systemd-udevd.service
*   | | |-systemd-update-utmp.service
*   | | |-cryptsetup.target
*   | | |-local-fs.target
*   | | | `-systemd-remount-fs.service
*   | | |-swap.target
*   | | `-veritysetup.target
*   | `-timers.target
*   |   |-apt-daily-upgrade.timer
*   |   |-apt-daily.timer
*   |   |-dpkg-db-backup.timer
*   |   |-e2scrub_all.timer
*   |   |-fstrim.timer
*   |   |-logrotate.timer
*   |   |-man-db.timer
*   |   |-motd-news.timer
*   |   |-systemd-tmpfiles-clean.timer
*   |   `-ua-timer.timer
*   |-getty.target
*   | |-console-getty.service
*   | |-container-getty@1.service
*   | |-container-getty@2.service
*   | `-getty-static.service
*   |-nfs-client.target
*   | |-auth-rpcgss-module.service
*   | |-rpc-statd-notify.service
*   | `-remote-fs-pre.target
*   |-remote-cryptsetup.target
*   |-remote-fs.target
*   | |-mnt-docker.mount
*   | |-mnt-downloads.mount
*   | |-mnt-movies.mount
*   | |-mnt-music.mount
*   | |-mnt-NZBGet_intermediate.mount
*   | |-mnt-photos.mount
*   | |-mnt-software.mount
*   | |-mnt-temp.mount
*   | |-mnt-tv.mount
*   | `-nfs-client.target
*   |   |-auth-rpcgss-module.service
*   |   |-rpc-statd-notify.service
*   |   `-remote-fs-pre.target
*   `-remote-veritysetup.target
root@plex:/etc# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @313ms
`-multi-user.target @313ms
  `-postfix.service @312ms +681us
    `-postfix@-.service @136ms +144ms
      `-nss-lookup.target @130ms
        `-systemd-resolved.service @99ms +31ms
          `-systemd-networkd.service @81ms +16ms
            `-network-pre.target @81ms
              `-ufw.service @80ms +786us
                `-local-fs.target @78ms
                  `-local-fs-pre.target @78ms
                    `-systemd-tmpfiles-setup-dev.service @76ms +2ms
                      `-systemd-sysusers.service @73ms +2ms
                        `-systemd-remount-fs.service @70ms +2ms
                          `-systemd-journald.socket @66ms
                            `-system.slice @62ms
                              `--.slice @62ms
 
Last edited:
agster27 said:
I so tried this in /etc/fstab and the same issue.

192.168.1.101:/volume1/downloads /mnt/downloads nfs4 auto,defaults,x-systemd.requires=systemd-networkd

Still same issue. Below is the output of:


Any thoughts would be appreciated... This is driving me nuts..

Code: 
root@plex:/etc# systemctl list-dependencies
default.target
* |-display-manager.service
* |-systemd-update-utmp-runlevel.service
* `-multi-user.target
*   |-console-setup.service
*   |-cron.service
*   |-dbus.service
*   |-dmesg.service
*   |-e2scrub_reap.service
*   |-irqbalance.service
*   |-networkd-dispatcher.service
*   |-plexmediaserver.service
*   |-postfix.service
*   |-rpcbind.service
*   |-rsync.service
*   |-rsyslog.service
*   |-ssh.service
*   |-systemd-ask-password-wall.path
*   |-systemd-logind.service
*   |-systemd-networkd.service
*   |-systemd-resolved.service
*   |-systemd-update-utmp-runlevel.service
*   |-systemd-user-sessions.service
*   |-ua-reboot-cmds.service
*   |-ubuntu-advantage.service
*   |-ufw.service
*   |-basic.target
*   | |-tmp.mount
*   | |-paths.target
*   | |-slices.target
*   | | |--.slice
*   | | `-system.slice
*   | |-sockets.target
*   | | |-dbus.socket
*   | | |-rpcbind.socket
*   | | |-ssh.socket
*   | | |-systemd-initctl.socket
*   | | |-systemd-journald-audit.socket
*   | | |-systemd-journald-dev-log.socket
*   | | |-systemd-journald.socket
*   | | |-systemd-networkd.socket
*   | | |-systemd-udevd-control.socket
*   | | |-systemd-udevd-kernel.socket
*   | | `-uuidd.socket
*   | |-sysinit.target
*   | | |-apparmor.service
*   | | |-dev-hugepages.mount
*   | | |-dev-mqueue.mount
*   | | |-keyboard-setup.service
*   | | |-kmod-static-nodes.service
*   | | |-nftables.service
*   | | |-proc-sys-fs-binfmt_misc.automount
*   | | |-setvtrgb.service
*   | | |-sys-fs-fuse-connections.mount
*   | | |-sys-kernel-config.mount
*   | | |-sys-kernel-debug.mount
*   | | |-sys-kernel-tracing.mount
*   | | |-systemd-ask-password-console.path
*   | | |-systemd-binfmt.service
*   | | |-systemd-boot-system-token.service
*   | | |-systemd-journal-flush.service
*   | | |-systemd-journald.service
*   | | |-systemd-machine-id-commit.service
*   | | |-systemd-modules-load.service
*   | | |-systemd-network-generator.service
*   | | |-systemd-pstore.service
*   | | |-systemd-random-seed.service
*   | | |-systemd-sysctl.service
*   | | |-systemd-sysext.service
*   | | |-systemd-sysusers.service
*   | | |-systemd-timesyncd.service
*   | | |-systemd-tmpfiles-setup-dev.service
*   | | |-systemd-tmpfiles-setup.service
*   | | |-systemd-udev-trigger.service
*   | | |-systemd-udevd.service
*   | | |-systemd-update-utmp.service
*   | | |-cryptsetup.target
*   | | |-local-fs.target
*   | | | `-systemd-remount-fs.service
*   | | |-swap.target
*   | | `-veritysetup.target
*   | `-timers.target
*   |   |-apt-daily-upgrade.timer
*   |   |-apt-daily.timer
*   |   |-dpkg-db-backup.timer
*   |   |-e2scrub_all.timer
*   |   |-fstrim.timer
*   |   |-logrotate.timer
*   |   |-man-db.timer
*   |   |-motd-news.timer
*   |   |-systemd-tmpfiles-clean.timer
*   |   `-ua-timer.timer
*   |-getty.target
*   | |-console-getty.service
*   | |-container-getty@1.service
*   | |-container-getty@2.service
*   | `-getty-static.service
*   |-nfs-client.target
*   | |-auth-rpcgss-module.service
*   | |-rpc-statd-notify.service
*   | `-remote-fs-pre.target
*   |-remote-cryptsetup.target
*   |-remote-fs.target
*   | |-mnt-docker.mount
*   | |-mnt-downloads.mount
*   | |-mnt-movies.mount
*   | |-mnt-music.mount
*   | |-mnt-NZBGet_intermediate.mount
*   | |-mnt-photos.mount
*   | |-mnt-software.mount
*   | |-mnt-temp.mount
*   | |-mnt-tv.mount
*   | `-nfs-client.target
*   |   |-auth-rpcgss-module.service
*   |   |-rpc-statd-notify.service
*   |   `-remote-fs-pre.target
*   `-remote-veritysetup.target
root@plex:/etc# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @313ms
`-multi-user.target @313ms
  `-postfix.service @312ms +681us
    `-postfix@-.service @136ms +144ms
      `-nss-lookup.target @130ms
        `-systemd-resolved.service @99ms +31ms
          `-systemd-networkd.service @81ms +16ms
            `-network-pre.target @81ms
              `-ufw.service @80ms +786us
                `-local-fs.target @78ms
                  `-local-fs-pre.target @78ms
                    `-systemd-tmpfiles-setup-dev.service @76ms +2ms
                      `-systemd-sysusers.service @73ms +2ms
                        `-systemd-remount-fs.service @70ms +2ms
                          `-systemd-journald.socket @66ms
                            `-system.slice @62ms
                              `--.slice @62ms
Click to expand...
Have you also tried the mount option _netdev? See man mount:
Code: 
       _netdev
           The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

Otherwise, I would suggest to also try the x-systemd.after=network-online.target.
 
Chris said:
Have you also tried the mount option _netdev? See man mount:
Code: 
       _netdev
           The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

Otherwise, I would suggest to also try the x-systemd.after=network-online.target.
Click to expand...

I've tried _netdev and x-systemd.after=network-online.target. Same issue. Thank you, really appreciate the help. Any ideas what to try next? This is driving me nuts. :)

Code: 
# UNCONFIGURED FSTAB FOR BASE SYSTEM
192.168.1.101:/volume1/downloads /mnt/downloads nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume1/docker   /mnt/docker     nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume1/temp     /mnt/temp       nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume1/movies   /mnt/movies     nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume1/tv       /mnt/tv         nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume2/music    /mnt/music      nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume1/photos   /mnt/photos     nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume1/software /mnt/software   nfs4    defaults,x-systemd.requires=network-online.target
192.168.1.101:/volume2/NZBGet_intermediate      /mnt/NZBGet_intermediate        nfs4    defaults,x-systemd.after=network-online.target

Code: 
Feb  1 21:26:39 plex systemd[1]: Started Network Name Resolution.
Feb  1 21:26:39 plex systemd[1]: Reached target Network.
Feb  1 21:26:39 plex systemd[1]: Reached target Network is Online.
Feb  1 21:26:39 plex systemd[1]: Reached target Host and Network Name Lookups.
Feb  1 21:26:39 plex networkd-dispatcher[107]: No valid path found for iwconfig
Feb  1 21:26:39 plex networkd-dispatcher[107]: No valid path found for iw
Feb  1 21:26:39 plex systemd-networkd[91]: eth0: Gained carrier
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/NZBGet_intermediate...
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/docker...
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/downloads...
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/movies...
Feb  1 21:26:39 plex mount[120]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/music...
Feb  1 21:26:39 plex mount[122]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/photos...
Feb  1 21:26:39 plex mount[124]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/software...
Feb  1 21:26:39 plex mount[126]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/temp...
Feb  1 21:26:39 plex mount[128]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Mounting /mnt/tv...
Feb  1 21:26:39 plex mount[130]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Starting Plex Media Server...
Feb  1 21:26:39 plex mount[132]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Starting Postfix Mail Transport Agent (instance -)...
Feb  1 21:26:39 plex mount[134]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Starting Notify NFS peers of a restart...
Feb  1 21:26:39 plex systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped.
Feb  1 21:26:39 plex systemd[1]: Condition check resulted in Ubuntu Pro Background Auto Attach being skipped.
Feb  1 21:26:39 plex mount[136]: mount.nfs4: Network is unreachable
Feb  1 21:26:39 plex systemd[1]: Started Dispatcher daemon for systemd-networkd.
Feb  1 21:26:39 plex systemd[1]: mnt-NZBGet_intermediate.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-NZBGet_intermediate.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex sm-notify[138]: Version 2.6.1 starting
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/NZBGet_intermediate.
Feb  1 21:26:39 plex systemd[1]: Dependency failed for Remote File Systems.
Feb  1 21:26:39 plex systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
Feb  1 21:26:39 plex systemd[1]: mnt-docker.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-docker.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/docker.
Feb  1 21:26:39 plex systemd[1]: mnt-downloads.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-downloads.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/downloads.
Feb  1 21:26:39 plex systemd[1]: mnt-movies.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-movies.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/movies.
Feb  1 21:26:39 plex systemd[1]: mnt-music.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-music.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/music.
Feb  1 21:26:39 plex systemd[1]: mnt-photos.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-photos.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/photos.
Feb  1 21:26:39 plex systemd[1]: mnt-software.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-software.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/software.
Feb  1 21:26:39 plex systemd[1]: mnt-temp.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-temp.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/temp.
Feb  1 21:26:39 plex systemd[1]: mnt-tv.mount: Mount process exited, code=exited, status=32/n/a
Feb  1 21:26:39 plex systemd[1]: mnt-tv.mount: Failed with result 'exit-code'.
Feb  1 21:26:39 plex systemd[1]: Failed to mount /mnt/tv.
Feb  1 21:26:39 plex systemd[1]: Started Notify NFS peers of a restart.
Feb  1 21:26:39 plex systemd[1]: Started Regular background program processing daemon.
Feb  1 21:26:39 plex systemd[1]: Starting Permit User Sessions...
Feb  1 21:26:39 plex cron[143]: (CRON) INFO (pidfile fd = 3)
Feb  1 21:26:39 plex systemd[1]: Started Plex Media Server.
Feb  1 21:26:39 plex cron[143]: (CRON) INFO (Running @reboot jobs)
Feb  1 21:26:39 plex systemd[1]: Finished Permit User Sessions.
Feb  1 21:26:39 plex systemd[1]: Started Console Getty.
Feb  1 21:26:39 plex systemd[1]: Started Container Getty on /dev/tty1.
Feb  1 21:26:39 plex systemd[1]: Started Container Getty on /dev/tty2.
Feb  1 21:26:39 plex systemd[1]: Reached target Login Prompts.
Feb  1 21:26:39 plex systemd[1]: Condition check resulted in Set console scheme being skipped.
Feb  1 21:26:39 plex systemd-networkd[91]: eth0: DHCPv4 address 192.168.1.102/24 via 192.168.1.1
Feb  1 21:26:39 plex dbus-daemon[104]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.0' (uid=105 pid=91 comm="/lib/systemd/systemd-networkd " label="unconfined")
Feb  1 21:26:39 plex systemd[1]: Starting Hostname Service...
Feb  1 21:26:39 plex postfix[213]: Postfix is running with backwards-compatible default settings
Feb  1 21:26:39 plex postfix[213]: See http://www.postfix.org/COMPATIBILITY_README.html for details
Feb  1 21:26:39 plex postfix[213]: To disable backwards compatibility use "postconf compatibility_level=3.6" and "postfix reload"
Feb  1 21:26:39 plex dbus-daemon[104]: [system] Successfully activated service 'org.freedesktop.hostname1'
Feb  1 21:26:39 plex systemd[1]: Started Hostname Service.
Feb  1 21:26:39 plex systemd-networkd[91]: Could not set hostname: Access denied
Feb  1 21:26:39 plex postfix/postfix-script[327]: starting the Postfix mail system
Feb  1 21:26:39 plex postfix/master[329]: daemon started -- version 3.6.4, configuration /etc/postfix
Feb  1 21:26:39 plex systemd[1]: Started Postfix Mail Transport Agent (instance -).
Feb  1 21:26:39 plex systemd[1]: Starting Postfix Mail Transport Agent...
Feb  1 21:26:39 plex systemd[1]: Finished Postfix Mail Transport Agent.
Feb  1 21:26:39 plex systemd[1]: Reached target Multi-User System.
Feb  1 21:26:39 plex systemd[1]: Reached target Graphical Interface.
Feb  1 21:26:39 plex systemd[1]: Starting Record Runlevel Change in UTMP...
Feb  1 21:26:39 plex systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.
Feb  1 21:26:39 plex systemd[1]: Finished Record Runlevel Change in UTMP.
Feb  1 21:26:39 plex systemd[1]: Startup finished in 419ms.
Feb  1 21:26:39 plex systemd[1]: dmesg.service: Deactivated successfully.
Feb  1 21:26:40 plex systemd-networkd[91]: eth0: Gained IPv6LL
Feb  1 21:26:52 plex : Critical: libusb_init failed
Feb  1 21:26:52 plex Plex Media Server[389]: Critical: libusb_init failed
Feb  1 21:27:09 plex systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Feb  1 21:27:15 plex systemd[1]: Created slice User Slice of UID 0.
Feb  1 21:27:15 plex systemd[1]: Starting User Runtime Directory /run/user/0...
Feb  1 21:27:15 plex systemd[1]: Finished User Runtime Directory /run/user/0.
Feb  1 21:27:15 plex systemd[1]: Starting User Manager for UID 0...
Feb  1 21:27:15 plex systemd[460]: Queued start job for default target Main User Target.
Feb  1 21:27:15 plex systemd[460]: Created slice User Application Slice.
Feb  1 21:27:15 plex systemd[460]: Reached target Paths.
Feb  1 21:27:15 plex systemd[460]: Reached target Timers.
Feb  1 21:27:15 plex systemd[460]: Listening on GnuPG network certificate management daemon.
Feb  1 21:27:15 plex systemd[460]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Feb  1 21:27:15 plex systemd[460]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Feb  1 21:27:15 plex systemd[460]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Feb  1 21:27:15 plex systemd[460]: Listening on GnuPG cryptographic agent and passphrase cache.
Feb  1 21:27:15 plex systemd[460]: Reached target Sockets.
Feb  1 21:27:15 plex systemd[460]: Reached target Basic System.
Feb  1 21:27:15 plex systemd[460]: Reached target Main User Target.
Feb  1 21:27:15 plex systemd[460]: Startup finished in 14ms.
Feb  1 21:27:15 plex systemd[1]: Started User Manager for UID 0.
Feb  1 21:27:15 plex systemd[1]: Started Session 176 of User root.
 
Last edited:
Okay, so from the logs we see that the network-online.target is already reached before the systemd-networkd.service sets up the network address via dhcp. So something is still not right with your service dependencies.

According to https://wiki.archlinux.org/title/Systemd-networkd#systemd-networkd-wait-online and also https://systemd.io/NETWORK_ONLINE/ you also require the systemd-networkd-wait-online.service dependency, as your network interfaces are managed by systemd-networkd.
This should make sure the NFS related mountpoints in the fstab entries are mouted afterwards, as they are mounted by the remote-fs.target, which depends on the network-online.target, so even without additional fstab parameters.

Please check if the service is enabled systemctl status systemd-networkd-wait-online.service and enable it if not systemctl enable systemd-networkd-wait-online.service. Then your NFS mounts should be delayed until after the systemd-networkd sets up the network.
 
You must log in or register to reply here.
Top Bottom