nvmf-autoconnect.service Not working

Ironhide526

New Member
Apr 19, 2025
2
1
1
I have configured shared storage for Proxmox using a Dell PowerStore 1200T. The protocol in use is NVMe over TCP. Everything works fine, except the connection does not persist after a reboot.

discovery.conf Configuration:

--transport=tcp --traddr=192.168.5.5 --trsvcid=4420
--transport=tcp --traddr=192.168.5.6 --trsvcid=4420
--transport=tcp --traddr=192.168.6.5 --trsvcid=4420
--transport=tcp --traddr=192.168.6.6 --trsvcid=4420


nvmf-autoconnect.service Configuration:


[Unit]
Description=Connect NVMe-oF subsystems automatically during boot
ConditionPathExists=/etc/nvme/discovery.conf
After=network-online.target
Before=remote-fs-pre.target

[Service]
Type=oneshot
ExecStartPre=/sbin/modprobe nvme_tcp
ExecStart=/usr/sbin/nvme connect-all

[Install]
WantedBy=default.target


The issue appears to be that the service is ignoring After=network-online.target and attempts to connect before the network is fully up.

I found a similar thread in the forums, but there were no answers.

Maybe there's a known bug that I'm not aware of.

Proxmox version: 8.4.1 (Latest)
 
  • Like
Reactions: zatanas
Hi,

this doesn't really have anything to do with Proxmox VE itself, but rather is normal Debian/systemd administration.

The issue appears to be that the service is ignoring After=network-online.target and attempts to connect before the network is fully up.
In any case, you probably also want Wants=network-online.target.

After= is a "ordering-only" directive and does not imply any dependencies on other services.

I'd suggest reading https://systemd.io/NETWORK_ONLINE/, esp. the paragraph How do I make sure that my service starts after the network is really online?
 
  • Like
Reactions: zatanas
Hi, thanks for the tip. However, I have already tried that. I added:
Wants=network-online.target to nvmf-autoconnect.service configuration


I also enabled the systemd-networkd-wait-online.service, but this service is failing at boot with the message: "Timeout occurred while waiting for network connectivity."

I modified the service's ExecStart to wait for specific interfaces:

ExecStart=/lib/systemd/systemd-networkd-wait-online --interface=vmbr1:routable --interface=vmbr2:routable

But it still fails during boot. Then, I removed the ":routable" part, and the service started successfully during boot, but the network storage still failed to attach automatically.
 
Did you find a resolution to this? I have recently connected some nvmeof (tcp) storage to my proxmox cluster, I notice it fails to reconnect after a reboot. However, once the host is up, I can ssh to the device and run nvme connect-all and the filesystem will connect.
 
Hi everyone,

Not the original poster here but I was able to resolve a similar issue.

In my scenario, the nvmf-autoconnect.service unit is running at boot before the network interface vmbr0, where my NVMe/TCP connections/fabric are traversing through, is actually up. This means that my NVMe/TCP target at 192.168.0.100 is unreachable at the time that nvmf-autoconnect attempts to reestablish the NVMe/TCP connections during the boot process. Since what I ultimately need is for nvmf-autoconnect.service to run after the NVMe/TCP target is reachable, instead of modifying network-online.target or nvmf-autoconnect.service directly, I created a new systemd service unit in /etc/systemd/system/my-new-unit.service as a dependency of nvmf-autoconnect.service, ensuring that my-new-unit.service runs before nvmf-autoconnect.service. I got the idea from the link shared by cheiss.

Here are the details of my-new-unit.service:

Bash:
[Unit]
DefaultDependencies=no
After=nss-lookup.target
Before=nvmf-autoconnect.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=sh -c 'until ping -c 1 192.168.0.100; do sleep 1; done'

[Install]
WantedBy=nvmf-autoconnect.service

The variable/setting Before=nvmf-autoconnect.service helps to ensure that my-new-unit.service runs before nvmf-autoconnect.service. The command at ExecStart= creates a loop which pings the IP listed once every second and exits once there is a successfull ping. The variable/setting WantedBy=nvmf-autoconnect.service sets my-new-unit.service as a dependecy of nvmf-autoconnect.service.

By forcing nvmf-autoconnect.service to wait until the intended NVMe/TCP target is reachable on the network it ensures that nvmf-autoconnect.service will be able to reestablish the NVMe/TCP connections during boot.

Please note: /etc/systemd/system/my-new-unit.service will need to have execute permissions in order to be able to run at boot. Once you create the service, you need to enable it via systemctl enable my-new-unit.service which creates the appropriate symlinks needed by systemd to run the service at boot as well as marking it as a dependency of nvmf-autoconnect.service.

I hope this helps someone out there.
 
Last edited: