systemd reexecuted, now having pam(?) issues (cron, web logins)

[petemcdonnell]

This morning, one node of a 3 node cluster decided to restart systemd. I haven't figured out why yet - don't see any updates having been installed or the like. The result is that we are unable to log in to the web interface and cron jobs are failing with a "failure setting user credentials". I would guess the issue is with access to pam for authentication?

How can I get this node back on track without a full node restart?

systemd restart:

Sep 11 06:43:02 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:02 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:02 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:04 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:05 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:05 node1 dbus-daemon[3160]: [system] Reloaded configuration
Sep 11 06:43:05 node1 systemd[1]: Reexecuting.
Sep 11 06:43:05 node1 systemd[1]: systemd 247.3-7+deb11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unifie>
Sep 11 06:43:05 node1 systemd[1]: Detected architecture x86-64.

cron errors:

Sep 11 07:05:01 node1 cron[2711151]: Failure setting user credentials
Sep 11 07:05:01 node1 CRON[2711151]: Failure setting user credentials
Sep 11 07:05:01 node1 cron[2711152]: Failure setting user credentials
Sep 11 07:05:01 node1 CRON[2711152]: Failure setting user credentials
Sep 11 07:05:01 node1 cron[2711153]: Failure setting user credentials
Sep 11 07:05:01 node1 CRON[2711153]: Failure setting user credentials

 

[petemcdonnell]

Looks like updates were automatically applied. Some of the updates included systemd components which is what triggered the systemd restart. I imagine we'll have to reboot the node, but are there any tricks to fixing this type of issue without a reboot?

Thanks!