Multiple containers fail to auto start services

[Justin Toler]

Steps taken until issue happens:
- pveam update
- download template
- create container
- install apache
- systemctl status apache -> it is running
- restart container
- systemctl status apache -> it no longer starts at start up

I have tried running 'systemctl enable apache2' but no luck.

I have been struggling for months to figure this out and I simply do not understand why this keeps happening on a fresh container. I even decided to install Proxmox 5.0 completely fresh and start over but still the same issue. I have tried researching everything I could think of but came across nothing that works. Any suggestions?

 

[Justin Toler]

I did forget to mention I have tried Ubuntu 16.04, 16.10, 17.04 templates, along with Debian 8 template.

 

[Oliver R]

Hi Justin!

Do you have any kind of syslog or apache error log? Without I can't really help you.

 

[Justin Toler]

I believe I figured it out. 'apt-get install apache2' seemed to not install a systemd service file.

Running

systemctl is-enabled apache2.service

I would get

apache2.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install is-enabled apache2

Creating a service file in /etc/systemd/system and then enabling the service seemed to fix the issue so far.

 

[Justin Toler]

Well it worked until I ran apt upgrade and rebooted.... apache and mysql no longer wants to start with the container. I do know the upgrade did upgrade systemd with version 229 it looks like.

journalctl appears to show nothing out of the norm. Apache2 and mysql logs do not show anything.

systemctl is-enabled apache2

still shows just 'enabled'

Trying to start anything with systemctl command just hangs, can use the service command and it works. I want to post logs but I'm not sure what to look at?

This is happening with more than 1 container. Have another container running Nextcloud with Nginx, php, etc. None of those service will start on boot.

 

[Justin Toler]

Weird... if I run

apt install - - reinstall systemd

I can then start any service via systemctl

systemctl start apache2

As soon as I exit and run the following on Proxmox

pct stop 103 && pct start 103 && pct enter 103

I am unable start any service via systemctl and nothing installed outside the distro/template starts at boot. Running systemctl start xxxx just hangs and never gives an error or response. This happens with fresh containers no matter what I do.

Can someone please help me diagnose what is going on?

 

[Oliver R]

I did a bit of testing but I can't recreate your problem. I tried Deb 8 and 9 and never had any kind of failure. Have you tried Debian 9? As you can see on my deb 9 install the symlinks were created without any flaw.

After installing Apache2 Deb 9

Systemd status after install Deb 9

Systemd status after reboot Deb 9

 

[Justin Toler]

I tried a fresh Debian 9 container and I just don't know what is going on.

This is the summary of my container settings:

Here is a copy of the commands I took right after creating the template. This is everything from start to finish while ssh into Proxmox:

root@proxmox:~# pct enter 111
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
root@DebianTest:~# dpkg-reconfigure locales
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = "en_US.UTF-8",
LC_CTYPE = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
Generating locales (this might take a while)...
en_US.UTF-8... done
Generation complete.
root@DebianTest:~# apt update
Ign:1 http://ftp.debian.org/debian stretch InRelease
Get:2 http://ftp.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:3 http://ftp.debian.org/debian stretch Release [118 kB]
Get:4 http://ftp.debian.org/debian stretch-updates/main amd64 Packages.diff/Index [736 B]
Get:5 http://security.debian.org stretch/updates InRelease [62.9 kB]
Get:6 http://ftp.debian.org/debian stretch Release.gpg [2,373 B]
Get:7 http://security.debian.org stretch/updates/main amd64 Packages [176 kB]
Get:8 http://security.debian.org stretch/updates/main Translation-en [81.3 kB]
Get:9 http://ftp.debian.org/debian stretch-updates/main Translation-en [3,040 B]
Get:10 http://ftp.debian.org/debian stretch-updates/main amd64 Packages [5,168 B]
Get:11 http://security.debian.org stretch/updates/contrib amd64 Packages [1,352 B]
Get:12 http://security.debian.org stretch/updates/contrib Translation-en [1,023 B]
Get:13 http://ftp.debian.org/debian stretch/main amd64 Packages [7,095 kB]
Get:14 http://ftp.debian.org/debian stretch/main Translation-en [5,393 kB]
Get:15 http://ftp.debian.org/debian stretch/contrib Translation-en [45.9 kB]
Fetched 13.1 MB in 3s (4,188 kB/s)
Reading package lists... Done
Building dependency tree... Done
43 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@DebianTest:~# apt-get install apache2
Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
apache2-bin apache2-data apache2-utils libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap liblua5.2-0
Suggested packages:
www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom
The following NEW packages will be installed:
apache2 apache2-bin apache2-data apache2-utils libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap liblua5.2-0
0 upgraded, 9 newly installed, 0 to remove and 43 not upgraded.
Need to get 1,794 kB/2,123 kB of archives.
After this operation, 7,073 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.debian.org stretch/updates/main amd64 apache2-bin amd64 2.4.25-3+deb9u3 [1,181 kB]
Get:2 http://security.debian.org stretch/updates/main amd64 apache2-utils amd64 2.4.25-3+deb9u3 [217 kB]
Get:3 http://security.debian.org stretch/updates/main amd64 apache2-data all 2.4.25-3+deb9u3 [162 kB]
Get:4 http://security.debian.org stretch/updates/main amd64 apache2 amd64 2.4.25-3+deb9u3 [235 kB]
Fetched 1,794 kB in 0s (6,587 kB/s)
Selecting previously unselected package libapr1:amd64.
(Reading database ... 20254 files and directories currently installed.)
Preparing to unpack .../0-libapr1_1.5.2-5_amd64.deb ...
Unpacking libapr1:amd64 (1.5.2-5) ...
Selecting previously unselected package libaprutil1:amd64.
Preparing to unpack .../1-libaprutil1_1.5.4-3_amd64.deb ...
Unpacking libaprutil1:amd64 (1.5.4-3) ...
Selecting previously unselected package libaprutil1-dbd-sqlite3:amd64.
Preparing to unpack .../2-libaprutil1-dbd-sqlite3_1.5.4-3_amd64.deb ...
Unpacking libaprutil1-dbd-sqlite3:amd64 (1.5.4-3) ...
Selecting previously unselected package libaprutil1-ldap:amd64.
Preparing to unpack .../3-libaprutil1-ldap_1.5.4-3_amd64.deb ...
Unpacking libaprutil1-ldap:amd64 (1.5.4-3) ...
Selecting previously unselected package liblua5.2-0:amd64.
Preparing to unpack .../4-liblua5.2-0_5.2.4-1.1+b2_amd64.deb ...
Unpacking liblua5.2-0:amd64 (5.2.4-1.1+b2) ...
Selecting previously unselected package apache2-bin.
Preparing to unpack .../5-apache2-bin_2.4.25-3+deb9u3_amd64.deb ...
Unpacking apache2-bin (2.4.25-3+deb9u3) ...
Selecting previously unselected package apache2-utils.
Preparing to unpack .../6-apache2-utils_2.4.25-3+deb9u3_amd64.deb ...
Unpacking apache2-utils (2.4.25-3+deb9u3) ...
Selecting previously unselected package apache2-data.
Preparing to unpack .../7-apache2-data_2.4.25-3+deb9u3_all.deb ...
Unpacking apache2-data (2.4.25-3+deb9u3) ...
Selecting previously unselected package apache2.
Preparing to unpack .../8-apache2_2.4.25-3+deb9u3_amd64.deb ...
Unpacking apache2 (2.4.25-3+deb9u3) ...
Setting up libapr1:amd64 (1.5.2-5) ...
Setting up apache2-data (2.4.25-3+deb9u3) ...
Processing triggers for libc-bin (2.24-11) ...
Setting up libaprutil1:amd64 (1.5.4-3) ...
Processing triggers for systemd (232-25) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up liblua5.2-0:amd64 (5.2.4-1.1+b2) ...
Setting up libaprutil1-ldap:amd64 (1.5.4-3) ...
Setting up libaprutil1-dbd-sqlite3:amd64 (1.5.4-3) ...
Setting up apache2-utils (2.4.25-3+deb9u3) ...
Setting up apache2-bin (2.4.25-3+deb9u3) ...
Setting up apache2 (2.4.25-3+deb9u3) ...
Enabling module mpm_event.
Enabling module authz_core.
Enabling module authz_host.
Enabling module authn_core.
Enabling module auth_basic.
Enabling module access_compat.
Enabling module authn_file.
Enabling module authz_user.
Enabling module alias.
Enabling module dir.
Enabling module autoindex.
Enabling module env.
Enabling module mime.
Enabling module negotiation.
Enabling module setenvif.
Enabling module filter.
Enabling module deflate.
Enabling module status.
Enabling module reqtimeout.
Enabling conf charset.
Enabling conf localized-error-pages.
Enabling conf other-vhosts-access-log.
Enabling conf security.
Enabling conf serve-cgi-bin.
Enabling site 000-default.
Created symlink /etc/systemd/system/multi-user.target.wants/apache2.service → /lib/systemd/system/apache2.service.
Created symlink /etc/systemd/system/multi-user.target.wants/apache-htcacheclean.service → /lib/systemd/system/apache-htcacheclean.service.
Processing triggers for libc-bin (2.24-11) ...
Processing triggers for systemd (232-25) ...
root@DebianTest:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2017-10-01 16:14:10 UTC; 1min 34s ago
Main PID: 1484 (apache2)
CGroup: /system.slice/apache2.service
├─1484 /usr/sbin/apache2 -k start
├─1485 /usr/sbin/apache2 -k start
└─1487 /usr/sbin/apache2 -k start

Oct 01 16:14:10 DebianTest systemd[1]: Starting The Apache HTTP Server...
Oct 01 16:14:10 DebianTest apachectl[1473]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Oct 01 16:14:10 DebianTest systemd[1]: Started The Apache HTTP Server.
root@DebianTest:~# exit
exit
root@proxmox:~# pct stop 111 && pct start 111 && pct enter 111
root@DebianTest:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: inactive (dead)

 

[dcsapak]

did you also try with 'pct shutdown' instead of 'pct stop' ? maybe the sudden stop breaks something?

 

[fabian]

also, maybe your containers are still waiting for boot to finish (check with "journalctl -b" / "systemctl status")? or you have a dependency cycle? otherwise, enabled services should start on boot..

 

[Oliver R]

Does your CT got an ipv6 address? Last time my boot never finished because ipv6 was configured on dhcp and I hadn't one setup at this time.

 

[Justin Toler]

also, maybe your containers are still waiting for boot to finish (check with "journalctl -b" / "systemctl status")? or you have a dependency cycle? otherwise, enabled services should start on boot..

Wow... your right! I ran the journalctl -b command and noticed the dhclient looked like it was hanging. As soon as it finished Apache started! The reason I never discovered this is, as you can see below, it took 5 mins for the dhclient to time out I guess and the boot continue. What is this? Is there a way I can fix this?

Oct 02 16:52:11 DebianTest dhclient[70]: Listening on LPF/eth0/XX:XX:XX:XX:XX
Oct 02 16:52:11 DebianTest ifup[47]: Listening on LPF/eth0/XX:XX:XX:XX:XX
Oct 02 16:52:11 DebianTest ifup[47]: Sending on LPF/eth0/ee:XX:XX:XX:XX:XX
Oct 02 16:52:11 DebianTest ifup[47]: Sending on Socket/fallback
Oct 02 16:52:11 DebianTest ifup[47]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5
Oct 02 16:52:11 DebianTest dhclient[70]: Sending on LPF/eth0/ee:XX:XX:XX:XX:XX
Oct 02 16:52:11 DebianTest dhclient[70]: Sending on Socket/fallback
Oct 02 16:52:11 DebianTest dhclient[70]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5
Oct 02 16:52:12 DebianTest dhclient[70]: DHCPREQUEST of 10.0.1.60 on eth0 to 255.255.255.255 port 67
Oct 02 16:52:12 DebianTest ifup[47]: DHCPREQUEST of 10.0.1.60 on eth0 to 255.255.255.255 port 67
Oct 02 16:52:12 DebianTest ifup[47]: DHCPOFFER of 10.0.1.60 from 10.0.1.1
Oct 02 16:52:12 DebianTest dhclient[70]: DHCPOFFER of 10.0.1.60 from 10.0.1.1
Oct 02 16:52:12 DebianTest dhclient[70]: DHCPACK of 10.0.1.60 from 10.0.1.1
Oct 02 16:52:12 DebianTest ifup[47]: DHCPACK of 10.0.1.60 from 10.0.1.1
Oct 02 16:52:12 DebianTest dhclient[70]: bound to 10.0.1.60 -- renewal in 3442 seconds.
Oct 02 16:52:12 DebianTest ifup[47]: bound to 10.0.1.60 -- renewal in 3442 seconds.
Oct 02 16:52:13 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 1060ms.
Oct 02 16:52:14 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 2210ms.
Oct 02 16:52:17 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 4240ms.
Oct 02 16:52:21 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 8340ms.
Oct 02 16:52:29 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 16180ms.
Oct 02 16:52:45 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 31770ms.
Oct 02 16:53:17 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 62200ms.
Oct 02 16:54:19 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 108930ms.
Oct 02 16:56:08 DebianTest dhclient[134]: XMT: Solicit on eth0, interval 118710ms.
Oct 02 16:57:11 DebianTest systemd[1]: networking.service: Start operation timed out. Terminating.
Oct 02 16:57:11 DebianTest systemd[1]: networking.service: Main process exited, code=killed, status=15/TERM
Oct 02 16:57:11 DebianTest systemd[1]: Failed to start Raise network interfaces.
Oct 02 16:57:11 DebianTest systemd[1]: networking.service: Unit entered failed state.
Oct 02 16:57:11 DebianTest systemd[1]: networking.service: Failed with result 'timeout'.
Oct 02 16:57:11 DebianTest systemd[1]: Reached target Network.
Oct 02 16:57:11 DebianTest systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Oct 02 16:57:11 DebianTest systemd[1]: Starting The Apache HTTP Server...
Oct 02 16:57:11 DebianTest systemd[1]: systemd-user-sessions.service: Failed to reset devices.list: Operation not permitted
Oct 02 16:57:11 DebianTest systemd[1]: Starting Permit User Sessions...
Oct 02 16:57:11 DebianTest systemd[1]: ssh.service: Failed to reset devices.list: Operation not permitted
Oct 02 16:57:11 DebianTest systemd[1]: Starting OpenBSD Secure Shell server...
Oct 02 16:57:11 DebianTest systemd[1]: Reached target Network is Online.
Oct 02 16:57:11 DebianTest systemd[1]: Starting Postfix Mail Transport Agent (instance -)...
Oct 02 16:57:11 DebianTest systemd[1]: apt-daily.timer: Adding 6h 51min 24.096736s random time.
Oct 02 16:57:11 DebianTest systemd[1]: Started Daily apt download activities.
Oct 02 16:57:11 DebianTest systemd[1]: apt-daily-upgrade.timer: Adding 10min 37.859612s random time.
Oct 02 16:57:11 DebianTest systemd[1]: Started Daily apt upgrade and clean activities.
Oct 02 16:57:11 DebianTest systemd[1]: Reached target Timers.
Oct 02 16:57:11 DebianTest systemd[1]: Started Permit User Sessions.
Oct 02 16:57:11 DebianTest systemd[1]: console-getty.service: Failed to reset devices.list: Operation not permitted
Oct 02 16:57:11 DebianTest systemd[1]: Started Console Getty.
Oct 02 16:57:11 DebianTest systemd[1]: Started Container Getty on /dev/tty1.
Oct 02 16:57:11 DebianTest systemd[1]: Reached target Login Prompts.
Oct 02 16:57:11 DebianTest systemd[1]: systemd-user-sessions.service: Failed to reset devices.list: Operation not permitted
Oct 02 16:57:11 DebianTest sshd[170]: Server listening on 0.0.0.0 port 22.
Oct 02 16:57:11 DebianTest sshd[170]: Server listening on :: port 22.
Oct 02 16:57:11 DebianTest systemd[1]: Started OpenBSD Secure Shell server.
Oct 02 16:57:11 DebianTest apachectl[168]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Oct 02 16:57:11 DebianTest systemd[1]: Started The Apache HTTP Server.

 

[Justin Toler]

The issue is with IPv6 like Oliver suggested. In Proxmox I had the container settings set to DHCP for IPv6 - changed to static and everything starts correctly without hanging now. I don't have my firewall set up for IPv6 so I guess that is the main issue that I need to fix someday.

 

[ibasaw]

i got the same problem, can't start nginx or php-fpm in a container... the command hang in a loop, no log, no error.

/etc/init.d/nginx status

Failed to get D-Bus connection: No such file or directory

i just use ipv4, how to disabled ipv6 inside the container ?

proxmox 5.2.2

 

[ibasaw]

i got the same problem, can't start nginx or php-fpm in a container... the command hang in a loop, no log, no error.

/etc/init.d/nginx status

Failed to get D-Bus connection: No such file or directory

i just use ipv4, how to disabled ipv6 inside the container ?

proxmox 5.2.2

problem solved, my container was corrupted...i deleted it, restore again, upgrade the container OS and it works now !