can't restart service (maridadb): systemd not booted as pid 1

mathx

Renowned Member
Jan 15, 2014
186
5
83
Very curious situation, which occured as well on another node of mine which I've been slowly unloading to reinstall completely, but has now happened on a more recently installed node.

Mariadb crashed for some reason (it is doing this much more frequently under pve 5.3 and .4 with maria 10.2 an 10.3, it used to never...) , but I could not restart it:
Code:
# service mysql start
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

#systemctl status mariadb.service
* mariadb.service - MariaDB 10.3.18 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-12-04 16:09:27 EST; 41s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 11181 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
  Process: 11184 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=1/FAILURE)

Dec 04 16:09:26 test systemd[1]: Starting MariaDB 10.3.18 database server...
Dec 04 16:09:27 test sh[11184]: System has not been booted with systemd as init system (PID 1). Can't operate.
Dec 04 16:09:27 test sh[11184]: Failed to connect to bus: Host is down
Dec 04 16:09:27 test systemd[1]: mariadb.service: Control process exited, code=exited, status=1/FAILURE
Dec 04 16:09:27 test systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 04 16:09:27 test systemd[1]: Failed to start MariaDB 10.3.18 database server.

#ps auxww | grep systemd
root         1  0.0  0.3 170504  7940 ?        Ss   Oct02   2:48 /lib/systemd/systemd --system --deserialize 18
root        83  0.0  0.3  19508  6740 ?        Ss   Oct02   0:12 /lib/systemd/systemd-logind
message+    90  0.0  0.1   9424  3556 ?        Ss   Oct02   8:39 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root      5293  0.0  3.9 124144 83100 ?        Ss   Nov18   4:56 /lib/systemd/systemd-journald
gizmo     5508  0.0  0.2  21024  4332 ?        Ss   14:38   0:00 /lib/systemd/systemd --user
root     11225  0.0  0.0   3084   880 pts/2    S+   16:10   0:00 grep --color systemd
The odd thing is I couldnt also run ps auxww from that shell actually, because I was pct enter $ID'd into it, it gave:

Code:
# ps auxww | grep systemd
Error: /proc must be mounted
  To mount /proc at boot you need an /etc/fstab line like:
      proc   /proc   proc    defaults
  In the meantime, run "mount proc /proc -t proc"

and I had to ssh into it instead. This absolutely was happening frequently on other CTs on that other host I figure is broken somehow. I think this has to do with some sort of systemd restart or reinit or something that isnt working on containers under new versions of PVE.

Ultimately, to fix mariadb, I had to restart the container entirely. And I can now pct enter and ps auxwwf, until whenever this happens again.

I suppose I should upgrade the PVE host, but upgrading and rebooting entire hosts to get systemd working is tricky with production sites on them.


Code:
#pveversion -V
proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve)
pve-manager: 5.3-12 (running version: 5.3-12/5fbbbaf6)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-35
pve-kernel-4.15.18-11-pve: 4.15.18-34
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-48
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-12
libpve-storage-perl: 5.0-39
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-24
pve-cluster: 5.0-34
pve-container: 2.0-35
pve-docs: 5.3-3
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-18
pve-firmware: 2.0-6
pve-ha-manager: 2.0-8
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-2
pve-xtermjs: 3.10.1-2
qemu-server: 5.0-47
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

The container is debian 10.2

This problem is similar to or the same as the one I mentioned here: https://forum.proxmox.com/threads/pct-list-not-working.59820/#post-276406
 
Last edited:
Hi,

pleas update to current supported PVE 5.4 or PVE 6.1.
 
If that's the solution, ok, thanks. Curious people were running 5.3 at all with mariadb, it's a pretty popular setup!

The other server with issues from the other thread with this missing /proc problem and similar systemd problems is:
Code:
proxmox-ve: 5.4-2 (running kernel: 4.15.18-20-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-8
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-10-pve: 4.15.18-32
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-55
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-6
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

Is this not sufficiently up todate?