nvmf-autoconnect.service Not working

[Ironhide526]

I have configured shared storage for Proxmox using a Dell PowerStore 1200T. The protocol in use is NVMe over TCP. Everything works fine, except the connection does not persist after a reboot.

discovery.conf Configuration:

--transport=tcp --traddr=192.168.5.5 --trsvcid=4420
--transport=tcp --traddr=192.168.5.6 --trsvcid=4420
--transport=tcp --traddr=192.168.6.5 --trsvcid=4420
--transport=tcp --traddr=192.168.6.6 --trsvcid=4420


nvmf-autoconnect.service Configuration:


[Unit]
Description=Connect NVMe-oF subsystems automatically during boot
ConditionPathExists=/etc/nvme/discovery.conf
After=network-online.target
Before=remote-fs-pre.target

[Service]
Type=oneshot
ExecStartPre=/sbin/modprobe nvme_tcp
ExecStart=/usr/sbin/nvme connect-all

[Install]
WantedBy=default.target


The issue appears to be that the service is ignoring After=network-online.target and attempts to connect before the network is fully up.

I found a similar thread in the forums, but there were no answers.

Maybe there's a known bug that I'm not aware of.

Proxmox version: 8.4.1 (Latest)

 

[cheiss]

Hi,

this doesn't really have anything to do with Proxmox VE itself, but rather is normal Debian/systemd administration.

The issue appears to be that the service is ignoring After=network-online.target and attempts to connect before the network is fully up.

In any case, you probably also want Wants=network-online.target.

After= is a "ordering-only" directive and does not imply any dependencies on other services.

I'd suggest reading https://systemd.io/NETWORK_ONLINE/, esp. the paragraph How do I make sure that my service starts after the network is really online?

 

[Ironhide526]

Hi, thanks for the tip. However, I have already tried that. I added:
Wants=network-online.target to nvmf-autoconnect.service configuration


I also enabled the systemd-networkd-wait-online.service, but this service is failing at boot with the message: "Timeout occurred while waiting for network connectivity."

I modified the service's ExecStart to wait for specific interfaces:

ExecStart=/lib/systemd/systemd-networkd-wait-online --interface=vmbr1:routable --interface=vmbr2:routable

But it still fails during boot. Then, I removed the ":routable" part, and the service started successfully during boot, but the network storage still failed to attach automatically.

 

[mmercier]

Did you find a resolution to this? I have recently connected some nvmeof (tcp) storage to my proxmox cluster, I notice it fails to reconnect after a reboot. However, once the host is up, I can ssh to the device and run nvme connect-all and the filesystem will connect.

 

[zatanas]

Hi everyone,

Not the original poster here but I was able to resolve a similar issue.

In my scenario, the nvmf-autoconnect.service unit is running at boot before the network interface vmbr0, where my NVMe/TCP connections/fabric are traversing through, is actually up. This means that my NVMe/TCP target at 192.168.0.100 is unreachable at the time that nvmf-autoconnect attempts to reestablish the NVMe/TCP connections during the boot process. Since what I ultimately need is for nvmf-autoconnect.service to run after the NVMe/TCP target is reachable, instead of modifying network-online.target or nvmf-autoconnect.service directly, I created a new systemd service unit in /etc/systemd/system/my-new-unit.service as a dependency of nvmf-autoconnect.service, ensuring that my-new-unit.service runs before nvmf-autoconnect.service. I got the idea from the link shared by cheiss.

Here are the details of my-new-unit.service:

[Unit]
DefaultDependencies=no
After=nss-lookup.target
Before=nvmf-autoconnect.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=sh -c 'until ping -c 1 192.168.0.100; do sleep 1; done'

[Install]
WantedBy=nvmf-autoconnect.service

The variable/setting Before=nvmf-autoconnect.service helps to ensure that my-new-unit.service runs before nvmf-autoconnect.service. The command at ExecStart= creates a loop which pings the IP listed once every second and exits once there is a successfull ping. The variable/setting WantedBy=nvmf-autoconnect.service sets my-new-unit.service as a dependecy of nvmf-autoconnect.service.

By forcing nvmf-autoconnect.service to wait until the intended NVMe/TCP target is reachable on the network it ensures that nvmf-autoconnect.service will be able to reestablish the NVMe/TCP connections during boot.

Please note: /etc/systemd/system/my-new-unit.service will need to have execute permissions in order to be able to run at boot. Once you create the service, you need to enable it via systemctl enable my-new-unit.service which creates the appropriate symlinks needed by systemd to run the service at boot as well as marking it as a dependency of nvmf-autoconnect.service.

I hope this helps someone out there.

 

[Scheme]

Hi everyone,

Not the original poster here but I was able to resolve a similar issue.

In my scenario, the nvmf-autoconnect.service unit is running at boot before the network interface vmbr0, where my NVMe/TCP connections/fabric are traversing through, is actually up. This means that my NVMe/TCP target at 192.168.0.100 is unreachable at the time that nvmf-autoconnect attempts to reestablish the NVMe/TCP connections during the boot process. Since what I ultimately need is for nvmf-autoconnect.service to run after the NVMe/TCP target is reachable, instead of modifying network-online.target or nvmf-autoconnect.service directly, I created a new systemd service unit in /etc/systemd/system/my-new-unit.service as a dependency of nvmf-autoconnect.service, ensuring that my-new-unit.service runs before nvmf-autoconnect.service. I got the idea from the link shared by cheiss.

Here are the details of my-new-unit.service:

[Unit]
DefaultDependencies=no
After=nss-lookup.target
Before=nvmf-autoconnect.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=sh -c 'until ping -c 1 192.168.0.100; do sleep 1; done'

[Install]
WantedBy=nvmf-autoconnect.service

The variable/setting Before=nvmf-autoconnect.service helps to ensure that my-new-unit.service runs before nvmf-autoconnect.service. The command at ExecStart= creates a loop which pings the IP listed once every second and exits once there is a successfull ping. The variable/setting WantedBy=nvmf-autoconnect.service sets my-new-unit.service as a dependecy of nvmf-autoconnect.service.

By forcing nvmf-autoconnect.service to wait until the intended NVMe/TCP target is reachable on the network it ensures that nvmf-autoconnect.service will be able to reestablish the NVMe/TCP connections during boot.

Please note: /etc/systemd/system/my-new-unit.service will need to have execute permissions in order to be able to run at boot. Once you create the service, you need to enable it via systemctl enable my-new-unit.service which creates the appropriate symlinks needed by systemd to run the service at boot as well as marking it as a dependency of nvmf-autoconnect.service.

I hope this helps someone out there.

Im just about to add this as part of my ansible script for rebuilding my homelab from scratch from total failure. Just curious if you were able to find a different fix for this? Still unsure if there was a change somewhere causing it to fail.

 

[zatanas]

Im just about to add this as part of my ansible script for rebuilding my homelab from scratch from total failure. Just curious if you were able to find a different fix for this? Still unsure if there was a change somewhere causing it to fail.

Hey Scheme,

Unfortunately, i have not found a different way to resolve this issue. I saw this issue from the very first time I implemented NVMe/TCP earlier this year. On average, the my-new-unit.service runs for approximately 44 seconds before a successful ping is made.

Something to keep in mind: During a boot, if the NVMe/TCP target is not available on the network, the systemd service my-new-unit.service will run indefinitely, as it is configured in my post above, preventing the system from booting up. if you run into this scenario, you can have systemd skip loading the service by adding the following line as a kernel parameter at the grub screen:

systemd.mask=my-new-unit.service

This can come in handy while troubleshooting.