[SOLVED] Container won't start - corrupt image

[alpha112]

Hi,

I'm getting an error whenever I try to start one of my containers. It started after a hard drive crash temporarily took down half of my array (1 drive failed, 2 went offline with it, 3 stayed up), so I imagine something ended up corrupt.

The RAID array is now back online and everything is up to date, and the other containers are working fine.

I can recreate the container (I thought I had a snapshot/backup, but must have missed this one), but if it's a quick fix I'd much rather do that.

Thanks for any help!

This is the output of journalctl -xn:

Mar 06 11:01:06 server kernel: JBD2: recovery failed
Mar 06 11:01:06 server kernel: EXT4-fs (loop3): error loading journal
Mar 06 11:01:11 server lxc-start[2477]: lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
Mar 06 11:01:11 server lxc-start[2477]: lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
Mar 06 11:01:11 server lxc-start[2477]: lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.
Mar 06 11:01:11 server systemd[1]: lxc@100.service: control process exited, code=exited status=1
Mar 06 11:01:11 server systemd[1]: Failed to start LXC Container: 100.
-- Subject: Unit lxc@100.service has failed
-- Defined-By: systemd
-- Support: [[link]]
-- 
-- Unit lxc@100.service has failed.
-- 
-- The result is failed.
Mar 06 11:01:11 server systemd[1]: Unit lxc@100.service entered failed state.
Mar 06 11:01:11 server pvedaemon[2475]: command 'systemctl start lxc@100' failed: exit code 1
Mar 06 11:01:11 server pvedaemon[1633]: <root@pam> end task UPID:server:000009AB:005CD641:58BD4171:vzstart:100:root@pam: command 'systemctl start lxc@100' failed: exit code 1

The conf file for this container:

arch: amd64
cpulimit: 2
cpuunits: 1024
hostname: crashplan
memory: 8192
net0: bridge=vmbr0,hwaddr=36:30:66:39:39:64,ip=dhcp,ip6=dhcp,name=eth0,type=veth
onboot: 1
ostype: ubuntu
rootfs: VMs:100/vm-100-disk-1.raw,size=8G
swap: 4096
#mp0: /mnt/Documents,mp=/mnt/Documents
#mp1: /mnt/External,mp=/mnt/External
#mp2: /mnt/Media,mp=/mnt/Media
#mp3: /mnt/Documents/Backup,mp=/mnt/Backup

And the output of lxc-start -n 100 --logfile ~/debug.log --logpriority=INFO

      lxc-start 20170306110745.344 INFO     lxc_start_ui - tools/lxc_start.c:main:275 - using rcfile /var/lib/lxc/100/config
      lxc-start 20170306110745.346 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
      lxc-start 20170306110745.363 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
      lxc-start 20170306110745.363 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
      lxc-start 20170306110745.407 INFO     lxc_container - lxccontainer.c:do_lxcapi_start:804 - Attempting to set proc title to [lxc monitor] /var/lib/lxc 100
      lxc-start 20170306110745.412 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
      lxc-start 20170306110745.414 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for reject_force_umount action 0.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for reject_force_umount action 0.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .[all].
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .kexec_load errno 1.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for kexec_load action 327681.
      lxc-start 20170306110745.415 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for kexec_load action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .open_by_handle_at errno 1.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for open_by_handle_at action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for open_by_handle_at action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .init_module errno 1.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for init_module action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for init_module action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .finit_module errno 1.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for finit_module action 327681.
      lxc-start 20170306110745.416 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:270 - Seccomp: got negative for syscall: -10085: finit_module.
      lxc-start 20170306110745.416 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:271 - This syscall will NOT be blacklisted.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for finit_module action 327681.
      lxc-start 20170306110745.416 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:270 - Seccomp: got negative for syscall: -10085: finit_module.
      lxc-start 20170306110745.416 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:271 - This syscall will NOT be blacklisted.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .delete_module errno 1.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for delete_module action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for delete_module action 327681.
      lxc-start 20170306110745.416 INFO     lxc_seccomp - seccomp.c:parse_config_v2:580 - Merging in the compat Seccomp ctx into the main one.
      lxc-start 20170306110745.417 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc".
      lxc-start 20170306110745.419 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
      lxc-start 20170306110745.420 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 5.
      lxc-start 20170306110745.464 INFO     lxc_monitor - monitor.c:lxc_monitor_sock_name:185 - Using monitor socket name "lxc/ad055575fe28ddd5//var/lib/lxc".
      lxc-start 20170306110745.778 ERROR    lxc_conf - conf.c:run_buffer:405 - Script exited with status 255.
      lxc-start 20170306110745.778 ERROR    lxc_start - start.c:lxc_init:450 - Failed to run lxc.hook.pre-start for container "100".
      lxc-start 20170306110745.778 ERROR    lxc_start - start.c:__lxc_start:1321 - Failed to initialize container "100".
      lxc-start 20170306110745.778 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - Command get_cgroup failed to receive response: Connection reset by peer.
      lxc-start 20170306110750.783 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
      lxc-start 20170306110750.783 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - To get more details, run the container in foreground mode.
      lxc-start 20170306110750.783 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.

 

[alpha112]

I solved it myself by doing an fsck on the raw image:

fsck.ext4 -v vm-100-disk-1.raw

Luckily there was only one error and it all seems ok now.