[SOLVED] Host network not starting (PVE 6.2-11, udevadm settle)

[markus-r]

Hi,

since today, one of the servers in my cluster hangs on boot for two minutes while starting the units ifupdown2-pre and systemd-udev-settle.

Once, the system is booted, I have no networking available - I can start it using systemctl start networking.
If I mask ifupdown2-pre, networking works after the boot, but this is not a solution.

Since both, ifupdown2-pre and systemd-udev-settle are waiting for the same (udevadm settle), I think that at least one device is not initializing fast enough - and therefore indicating a problem. Unfortunately I can not find anything in the system log or dmesg which points to a device causing this issue.

I already switched on debug logging in udev.conf before booting, but it did not help me finding the device causing udev not to settle.

The underlying server is a Dell machine - I did a system self test and found no problem, there.

Any ideas?

Thanks,

Markus

 

[t.lamprecht]

Hi,

what where the recent changes on that server? As you imply it worked just fine until today.
Did you booted a new kernel version, if so which one and does booting an older one fixes this behaviour?

Anything in dmesg (kernel log) during bootup, errors or other things related to network?

 

[markus-r]

I wanted to upgrade another host in the cluster. Therefore I did a live migration of a bunch of VMs to the server having the problem, now. Obviously too many of them. During this migration the server's memory went low, it had a kernel panic (Bad RIP value), rebooted and since then, there is the problem.

The kernel version did not change. Later, I tried an older one - and even the most recent one - the same in all cases.

No errors in dmesg. Only an offset of about 120 sec between these two entries:

[   13.445257] ZFS: Loaded module v0.8.4-pve1, ZFS pool version 5000, ZFS filesystem version 5
[  132.022459] alua: device handler registered

 

[t.lamprecht]

During this migration the server's memory went low, it had a kernel panic (Bad RIP value), rebooted and since then, there is the problem.

So, a memory low should in general result in an Out-Of-Memory (OOM) situation where the OOM killer then reaps a process hogging lots of memory. A kernel panic with "Bad RIP values" is a bit weird for such a scenario.
It could point to a HW related issue, maybe failing memory. But I hardly see a connection between that issue and a failing udevadm settle on the next boots...

If really nothing changed in the software environment, I'd start witch checking HW for now - maybe run a longer memtest.

You could also try to edit the kernel boot comandline and add loglevel=7 to enable debug level logging.

 

[markus-r]

In the meantime I have been able to narrow the issue down to multipathd.

If I disable the unit multipathd, the system boots without any delay in udevadm settle.

The Server has two FC HBAs directly connected to the two controllers of a Dell MD38xxf. There are two LUNs on the storage array managed through multipathd. One of them comes up as on the other node, the other one fails. /etc/multipath.conf is identical on both servers (see below).

If I compare the output of "multipath -tt -v3" on both servers, the most obvious difference is that for some reason the failing server tries to use alua to determine the prio - just for the failing LUN.

Failing node:

Sep 23 15:35:04 | sde: detect_prio = yes (setting: multipath.conf defaults/devices section)
Sep 23 15:35:04 | sde: prio = alua (setting: storage device autodetected)

Intact node:

Sep 23 15:34:44 | sde: detect_prio = yes (setting: multipath.conf defaults/devices section)
Sep 23 15:34:44 | sde: prio = rdac (setting: storage device configuration)

My multipath.conf looks like this:

defaults {
    verbosity 2
    polling_interval 5
    max_polling_interval 20
    reassign_maps "no"
    multipath_dir "//lib/multipath"
    path_selector "service-time 0"
    path_grouping_policy "failover"
    uid_attribute "ID_SERIAL"
    prio "const"
    prio_args ""
    features "0"
    path_checker "tur"
    alias_prefix "mpath"
    failback "manual"
    rr_min_io 1000
    rr_min_io_rq 1
    max_fds "max"
    rr_weight "uniform"
    queue_without_daemon "no"
    flush_on_last_del "no"
    user_friendly_names "no"
    fast_io_fail_tmo 5
    bindings_file "/etc/multipath/bindings"
    wwids_file "/etc/multipath/wwids"
    prkeys_file "/etc/multipath/prkeys"
    log_checker_err always
    all_tg_pt "no"
    retain_attached_hw_handler "yes"
    detect_prio "yes"
    detect_checker "yes"
    force_sync "no"
    strict_timing "no"
    deferred_remove "no"
    config_dir "/etc/multipath/conf.d"
    delay_watch_checks "no"
    delay_wait_checks "no"
    marginal_path_err_sample_time "no"
    marginal_path_err_rate_threshold "no"
    marginal_path_err_recheck_gap_time "no"
    marginal_path_double_failed_time "no"
    find_multipaths "strict"
    uxsock_timeout 4000
    retrigger_tries 0
    retrigger_delay 10
    missing_uev_wait_timeout 30
    skip_kpartx "no"
    disable_changed_wwids "yes"
    remove_retries 0
    ghost_delay "no"
    find_multipaths_timeout -10
}
blacklist {
    wwid .*
}

blacklist_exceptions {
    wwid "3600a098000585058000007485f062ad7"
    wwid "3600a09800057790e0000086c5f062606"
}

devices {
    device {
        vendor "DELL"
        product "^MD3"
        product_blacklist "Universal Xport"
        path_grouping_policy "group_by_prio"
        path_checker "rdac"
        path_selector "round-robin 0"
        features "2 pg_init_retries 50"
        hardware_handler "1 rdac"
        prio "rdac"
        failback "immediate"
        no_path_retry 30
    }
}
overrides {
}
multipaths {
    multipath {
        wwid "3600a098000585058000007485f062ad7"
        alias "san-ssd"
    }
    multipath {
        wwid "3600a09800057790e0000086c5f062606"
        alias "san-hdd"
    }
}

Any ideas why multipath is acting differently on both nodes?

 

[markus-r]

Forget about my last post. I had to run update-initrd and now multipath is running just fine.

The news:
I left the cluster with my problematic node.
Then I reinstalled PVE on the node (yes, I am so desperate , updated everything and re-applied local configuration.
System boot still worked quickly.

Then I installed multipath-tools and configured it. After figuring our that I need tu update the initrd, multipathd works as expected.

But: The boot hangs like it was before the reinstallation / after the last system update.

In journalctl I see:

Sep 24 13:42:02 gvc02n01 systemd-udevd[743]: Spawned process '/bin/systemd-run /sbin/lvm pvscan --cache 8:65' [940] is taking longer than 59s to complete
Sep 24 13:42:02 gvc02n01 systemd-udevd[759]: Spawned process '/bin/systemd-run /sbin/lvm pvscan --cache 8:33' [941] is taking longer than 59s to complete
Sep 24 13:42:02 gvc02n01 systemd-udevd[730]: Spawned process '/bin/systemd-run /sbin/lvm pvscan --cache 8:49' [939] is taking longer than 59s to complete
Sep 24 13:42:02 gvc02n01 systemd-udevd[763]: Spawned process '/bin/systemd-run /sbin/lvm pvscan --cache 8:17' [942] is taking longer than 59s to complete
Sep 24 13:42:02 gvc02n01 systemd-udevd[718]: sdd1: Worker [730] processing SEQNUM=4080 is taking a long time
Sep 24 13:42:02 gvc02n01 systemd-udevd[718]: sde1: Worker [743] processing SEQNUM=4087 is taking a long time
Sep 24 13:42:02 gvc02n01 systemd-udevd[718]: sdc1: Worker [759] processing SEQNUM=4060 is taking a long time
Sep 24 13:42:02 gvc02n01 systemd-udevd[718]: sdb1: Worker [763] processing SEQNUM=4053 is taking a long time
Sep 24 13:43:01 gvc02n01 systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 13:43:01 gvc02n01 systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'.

sdd1, sde1, sdc1 and sdb1 are the four partitions behind my two multipaths.

If I disable the unit multipathd.service, the system boots quickly without the wait time.

Ideas?

 

[RokaKen]

I suggest you enable multipathd and DISABLE systemd-udev-settle -- it is essentially deprecated since it never worked reliably and isn't recommended.[0]

[0] systemd-udev-settle.service

 

[markus-r]

Thanks for this hint, RokaKen!

Since I have been able to narrow down my problem to multipath-tools, I downgraded from 0.7.9-3+deb10u1 to 0.7.9-3 which solves my boot problem. I will try to find out which change in this release could have caused my issue.

 

[t.lamprecht]

The single change between those versions seems to be:

kpartx: use correct path to partx in udev rule (Closes: #959727)

-- https://tracker.debian.org/media/packages/m/multipath-tools/changelog-0.7.9-3deb10u1
The bug report which was closed by this change: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959727

And the patch: https://salsa.debian.org/linux-bloc...mmit/775fe682dabb69e7a51d6fb84f3d172f29054ad0

This means that probably previously the partx call failed as the wrong path was used, now since that newer multipath-tools version partx is called correctly, and hangs at that stage.

IMO: The real fix for your issue is as @RokaKen recommended: disable the systemd-udev-settle.service

 

[markus-r]

Thanks, @t.lamprecht and @RokaKen. I finally deacitvated/masked systemd-udev-settle.service and my systems are running smoothly, again.

As soon as I find the time, I will investigate the origin of the kpartx hangs my system seems to experience since kpartx knows the correct path of partx.

Marking this as solved, now.

 

[Alex2801]

I can confirm that this issue is still present in a fresh 7.0-13 installation with multipath and 2 fc hbas connected to the two controllers of a Dell MD3620f without a switch. Masking the systemd-udev-settle.service also fixed the issue here.

Alex

 

[julz22x4]

7.2-8, network sill not starting. But masking the systemd-udev-settle.service did not fix the issue!
after 'systemctl restart networking' i have an ip.
what else could i do?

Edit: Solved: i missed there is an 2 in ifupdown2-pre.service. i only disabled ifupdown-pre.service