[SOLVED] I can't start any LXC Node

[ionai]

Hello,

I am very beginner on proxmox and linux administration.
We have this Node nsXXXXXXX with 6 lxc. All these 6 lxc are down, right now, and can't start.
On the browser, when I try to start any of these VM, I get the error below :

Job for pve-container@148.service failed because the control process exited with error code.
See "systemctl status pve-container@148.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@148' failed: exit code 1

The command :~# systemctl status pve-container@148.service prints:

pve-container@148.service - PVE LXC Container: 148
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2020-07-02 09:52:48 UTC; 5h 22min ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 3910 ExecStart=/usr/bin/lxc-start -n 148 (code=exited, status=1/FAILURE)

Jul 02 09:52:48 nsXXXXXXX systemd[1]: Starting PVE LXC Container: 148...
Jul 02 09:52:48 nsXXXXXXX lxc-start[3910]: lxc-start: 148: lxccontainer.c: wait_on_daemonized_start: 856 No such
Jul 02 09:52:48 nsXXXXXXX lxc-start[3910]: lxc-start: 148: tools/lxc_start.c: main: 330 The container failed to s
Jul 02 09:52:48 nsXXXXXXX lxc-start[3910]: lxc-start: 148: tools/lxc_start.c: main: 333 To get more details, run
Jul 02 09:52:48 nsXXXXXXX lxc-start[3910]: lxc-start: 148: tools/lxc_start.c: main: 336 Additional information ca
Jul 02 09:52:48 nsXXXXXXX systemd[1]: pve-container@148.service: Control process exited, code=exited status=1
Jul 02 09:52:48 nsXXXXXXX systemd[1]: Failed to start PVE LXC Container: 148.
Jul 02 09:52:48 nsXXXXXXX systemd[1]: pve-container@148.service: Unit entered failed state.
Jul 02 09:52:48 nsXXXXXXX systemd[1]: pve-container@148.service: Failed with result 'exit-code'.

and the command :~# journalctl -xe

Jul 02 15:17:02 nsXXXXXXXX sshd[6088]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh
Jul 02 15:17:03 nsXXXXXXXX sshd[6088]: Failed password for invalid user kk from <IP> port 52068 ssh2
Jul 02 15:17:03 nsXXXXXXXX sshd[6088]: Received disconnect from <IP> port 52068:11: Bye Bye [preauth]
Jul 02 15:17:03 nsXXXXXXXX sshd[6088]: Disconnected from <IP> port 52068 [preauth]
Jul 02 15:17:08 nsXXXXXXXX sshd[6090]: Invalid user kali from <IP> port 37832
Jul 02 15:17:08 nsXXXXXXXX sshd[6090]: input_userauth_request: invalid user kali [preauth]
Jul 02 15:17:08 nsXXXXXXXX sshd[6090]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 15:17:08 nsXXXXXXXX sshd[6090]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh

When I execute :~pct list it shows:

user@nsXXXXXXX:~# pct list
VMID       Status     Lock         Name
148        stopped    mounted      example.org
149        stopped    mounted      anotherexample.com
150        stopped    mounted      thatexample.org
151        stopped                 thisexample.org
152        stopped                 moreexample.com
153        stopped                 lastexample.com

The lxc-start command shows the error below:

user@nsXXXXXXX:~# lxc-start -n 148 -F
lxc-start: 148: conf.c: run_buffer: 335 Script exited with status 2
lxc-start: 148: start.c: lxc_init: 861 Failed to run lxc.hook.pre-start for container "148"
lxc-start: 148: start.c: __lxc_start: 1944 Failed to initialize container "148"
lxc-start: 148: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: 148: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

The following informations might be helpful:

user@nsXXXXXXX:~# pveversion -v
proxmox-ve: 5.4-2 (running kernel: 4.15.18-30-pve)
pve-manager: 5.4-15 (running version: 5.4-15/d0ec33c6)
pve-kernel-4.15: 5.4-19
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-26-pve: 4.15.18-54
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-56
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-42
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-56
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3

How can I overcome that problem, please ?
Could someone help me, please ?

Thank you in advance.

 

[oguz]

hi,

can you provide debug logs? [0]

proxmox-ve: 5.4-2 (running kernel: 4.15.18-30-pve)

also i suggest that you update your installation to the latest 6.x when you can, as the newest bugfixes are there and the 5.x will be EOL soon...

[0]: https://pve.proxmox.com/pve-docs/chapter-pct.html#_obtaining_debugging_logs

 

[ionai]

Hi,

please, find enclosed file.
Yes, we will update to the latest version as soon as possible.

Thank you very much for your reply.
Best regards,
Ionai

 

[oguz]

in your log file i see:

lxc-start 148 20200706110218.482 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 148 lxc pre-start with output: volume 'data:148/vm-148-disk-0.raw' does not exist

it looks like the disk image of the container might be missing/moved?

 

[ionai]

Yes, it was moved.
2 weeks ago, my college in charge of the infra, has created RAID arrays with mdadm. And, it seems like everything were working well. Last week,
that node stopped and we cannot restart it (as usual via Browser neither via command line), and my college is on vacation

I am going to verify the others lxc behavior then investigate that disk image issue.

Thank you for helping.
Best regards,
Ionai

 

[ionai]

We did fix the disk image problem. Should I change the thread to resolved ?

Best regards,
Ionai

 

[oguz]

We did fix the disk image problem. Should I change the thread to resolved ?

Best regards,
Ionai

glad your issue is solved.

i think it's pretty clear so we can mark it [SOLVED]