[SOLVED] Web Proxmox, all LXC and VM's are down after update and reboot (PVE8)

[Admiral Awesome]

Hi everyone,
After applying latest updates all of my containers, VM's and proxmox web instance are down.
I've already done some research on Forums on what could help, but no luck there. Here are some logs and findings.

I'm able to ssh to machine. I'm also getting ILO connection.

:~# lxc-ls -f
NAME STATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
100  STOPPED 0         -      -    -    true
101  STOPPED 0         -      -    -    false
102  STOPPED 0         -      -    -    true
103  STOPPED 0         -      -    -    false
104  STOPPED 0         -      -    -    true
106  STOPPED 0         -      -    -    false
107  STOPPED 0         -      -    -    false
108  STOPPED 0         -      -    -    false
109  STOPPED 0         -      -    -    true
110  STOPPED 0         -      -    -    false
111  STOPPED 0         -      -    -    false
112  STOPPED 0         -      -    -    true
113  STOPPED 0         -      -    -    true
115  STOPPED 0         -      -    -    true
116  STOPPED 0         -      -    -    true
117  STOPPED 0         -      -    -    true

:~# lxc-start 100
lxc-start: 100: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 870 No such file or directory - Failed to receive the container state
lxc-start: 100: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: 100: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode
lxc-start: 100: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

:~# pveupdate
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

:~# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=18.5 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=19.0 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=18.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=19.0 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=59 time=18.8 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 18.498/18.753/18.990/0.209 ms

:~# uname -a
Linux pandora 6.2.16-10-pve #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-10 (2023-08-18T11:42Z) x86_64 GNU/Linux

:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-10-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
proxmox-kernel-6.2: 6.2.16-10
proxmox-kernel-6.2.16-6-pve: 6.2.16-7
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.8
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-4
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

 

[Admiral Awesome]

After executing:
lxc-start -n 100 -F -l DEBUG -o /tmp/lxc-100.log

Log file contents:

lxc-start 100 20230830092947.348 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20230830092947.348 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20230830092947.349 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 100 20230830092947.349 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20230830092947.832 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: cluster not ready - no quorum?


lxc-start 100 20230830092947.842 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 2
lxc-start 100 20230830092947.842 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20230830092947.842 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "100"
lxc-start 100 20230830092947.842 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20230830092948.345 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20230830092948.823 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 100 20230830092948.823 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

 

[Admiral Awesome]

Abot the no quorum?

:~# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

Also - server is just one server, there should be no quorum.

EDIT:

I've might have a problem with /etc/hosts host name is pandora

I don't know what was the original file and now it looks modified, here is the current version:

127.0.0.1 localhost.localdomain localhost
10.1.0.3 pandora

It's true that server address in local network is 10.1.0.3. What am I missing here?

 

[Admiral Awesome]

Closing and solving,

Resolution:
nano /etc/hosts

127.0.0.1 localhost.localdomain localhost
<local IP> <servername>.<domain> <fqdn>

reboot

did the trick.

Worth noticing: systemctl restart networking was not able to fix it, so reboot was in order.

But hey - it's so fun to write on this forum!