NFS Server stopped before VMs during shutdown

[masipcat]

After upgrading to Proxmox v8, I noticed that powering off / rebooting the host stops the nfs-server immediately, while VM are still running and connected to the NFS server.

I compared the systemd unit and this is what changed, but I'm not sure if the problem is here:

--- /tmp/pve1/lib/systemd/system/nfs-server.service    2021-06-28 09:15:06.000000000 +0200
+++ new-nfs-server.conf    2023-07-03 22:09:54.490884616 +0200
@@ -3,36 +3,31 @@
 DefaultDependencies=no
 Requires=network.target proc-fs-nfsd.mount
 Requires=nfs-mountd.service
-Wants=rpcbind.socket
-Wants=nfs-idmapd.service
+Wants=rpcbind.socket network-online.target
+Wants=rpc-statd.service nfs-idmapd.service
+Wants=rpc-statd-notify.service
+Wants=nfsdcld.service
 
-After=local-fs.target
-After=network.target proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service
+After=network-online.target local-fs.target
+After=proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service
 After=nfs-idmapd.service rpc-statd.service
+After=nfsdcld.service
 Before=rpc-statd-notify.service
 
 # GSS services dependencies and ordering
-Wants=auth-rpcgss-module.service
+Wants=auth-rpcgss-module.service rpc-svcgssd.service
 After=rpc-gssd.service gssproxy.service rpc-svcgssd.service
 
-# start/stop server before/after client
-Before=remote-fs-pre.target
-
-Wants=nfs-config.service
-After=nfs-config.service
-
 [Service]
-EnvironmentFile=-/run/sysconfig/nfs-utils
-
 Type=oneshot
 RemainAfterExit=yes
-ExecStartPre=/usr/sbin/exportfs -r
-ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
+ExecStartPre=-/usr/sbin/exportfs -r
+ExecStart=/usr/sbin/rpc.nfsd
 ExecStop=/usr/sbin/rpc.nfsd 0
 ExecStopPost=/usr/sbin/exportfs -au
 ExecStopPost=/usr/sbin/exportfs -f
 
-ExecReload=/usr/sbin/exportfs -r
+ExecReload=-/usr/sbin/exportfs -r
 
 [Install]
 WantedBy=multi-user.target

Thx

 

[Golum]

I'm having the same issue and it prevents vm's from shutting down which ends with a dead lock and thus total crash of the system.

 

[cweakland]

I got it sorted out, I watched the journalctl log and found nfs-server.service was stopping before pve-manager.service.

I made the following edit with:

systemctl edit pve-manager.service

#Add this to the file (be sure to read how this works, you need to add it in the empty space specified.)

[Unit]
After=nfs-server.service

Save and exit and then perform a:
systemctl daemon-reload

Reboots are now smooth, all my VMs shutdown before the nfs-server ends.

Good luck!

 

[Golum]

This still needs to be addressed by Proxmox staff, the service file will be overwritten on system upgrades.

Ping @tom @dietmar

 

[cweakland]

I agree, however, will it remove a systemd override file?

 

[BobhWasatch]

Or you could run your NFS server in a VM. That allows you to control the boot & shutdown order relative to the other VM's.

 

[masipcat]

Thank you @cweakland for your solution!

 

[Golum]

This Problem still exists and results in DATALOSS!

Why is it not picked up by the Proxmox staff ?

@t.lamprecht @martin

 

[BobhWasatch]

This Problem still exists and results in DATALOSS!

Why is it not picked up by the Proxmox staff ?

Maybe because professional operations don't run an NFS server directly on the hypervisor. They run it on separate hardware or in a VM. And maybe also because they don't' want to patch Debian's systemd files.

 

[t.lamprecht]

Yeah, we cannot hedge against all possible configurations users can make, and we certainly do not want to limit power users that understand their systems by restricting access to create NFS servers or the like (which is rather impossible to do for certain any way).

Anyhow, if you add custom services where your VMs then depend on that without telling PVE it won't work, the simplest way to tell PVE that it needs to stop the NFS server unit after the guests is to add an ordering constraint for pve-guests.service

systemctl edit nfs-server.service (or whatever your NFS server unit is named)

Then add

[Unit]
Before=pve-guests.service

This ensures that the NFS server unit is started before the pve-guests.service, as systemd reversed this on shutdown it will then ensure that it's stopped after pve-guests.service there.

Note that if we would add such a blanket statement in there we might break other use cases, e.g. where network is provided through a VM and NFS should be stopped before that VM is shut down.

 

[bjtmosfet]

I got it sorted out, I watched the journalctl log and found nfs-server.service was stopping before pve-manager.service.

I made the following edit with:

systemctl edit pve-manager.service

#Add this to the file (be sure to read how this works, you need to add it in the empty space specified.)

[Unit]
After=nfs-server.service

Save and exit and then perform a:
systemctl daemon-reload

Reboots are now smooth, all my VMs shutdown before the nfs-server ends.

Good luck!

TOP! thanks, now reboot time go back from 12min to 2min as before start using nfs