LXC container won't start

[Robertas]

I am using lxc containers on nfs storage. It was working well, until yesterday I had to restart my NFS server and proxmox won't boot my existing containers. I can create a new container which does boot up, but it won't start existing containers.

I am getting:

lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.
TASK ERROR: command 'lxc-start -n 104' failed: exit code 1

I've tried starting them via lxc-start -n 444 -F --logfile=lxc.log --logpriority=debug

lxc-start 1463041070.075 INFO  lxc_start_ui - lxc_start.c:main:264 - using rcfile /var/lib/lxc/104/config
  lxc-start 1463041070.075 WARN  lxc_confile - confile.c:config_pivotdir:1817 - lxc.pivotdir is ignored.  It will soon become an error.
  lxc-start 1463041070.077 WARN  lxc_cgmanager - cgmanager.c:cgm_get:994 - do_cgm_get exited with error
  lxc-start 1463041070.077 INFO  lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for reject_force_umount action 0
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:do_resolve_add_rule:216 - Setting seccomp rule to reject force umounts

  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for reject_force_umount action 0
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:do_resolve_add_rule:216 - Setting seccomp rule to reject force umounts

  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .[all].
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .kexec_load errno 1.
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for kexec_load action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for kexec_load action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .open_by_handle_at errno 1.
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for open_by_handle_at action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for open_by_handle_at action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .init_module errno 1.
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for init_module action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for init_module action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .finit_module errno 1.
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for finit_module action 327681
  lxc-start 1463041070.078 WARN  lxc_seccomp - seccomp.c:do_resolve_add_rule:233 - Seccomp: got negative # for syscall: finit_module
  lxc-start 1463041070.078 WARN  lxc_seccomp - seccomp.c:do_resolve_add_rule:234 - This syscall will NOT be blacklisted
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for finit_module action 327681
  lxc-start 1463041070.078 WARN  lxc_seccomp - seccomp.c:do_resolve_add_rule:233 - Seccomp: got negative # for syscall: finit_module
  lxc-start 1463041070.078 WARN  lxc_seccomp - seccomp.c:do_resolve_add_rule:234 - This syscall will NOT be blacklisted
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .delete_module errno 1.
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for delete_module action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for delete_module action 327681
  lxc-start 1463041070.078 INFO  lxc_seccomp - seccomp.c:parse_config_v2:436 - Merging in the compat seccomp ctx into the main one
  lxc-start 1463041070.078 INFO  lxc_conf - conf.c:run_script_argv:362 - Executing script '/usr/share/lxc/hooks/lxc-pve-prestart-hook' for container '104', config section 'lxc'
  lxc-start 1463041070.910 ERROR  lxc_conf - conf.c:run_buffer:342 - Script exited with status 255
  lxc-start 1463041070.910 ERROR  lxc_start - start.c:lxc_init:436 - failed to run pre-start hooks for container '104'.
  lxc-start 1463041070.910 ERROR  lxc_start - start.c:__lxc_start:1170 - failed to initialize the container
  lxc-start 1463041070.910 ERROR  lxc_start_ui - lxc_start.c:main:344 - The container failed to start.
  lxc-start 1463041070.910 ERROR  lxc_start_ui - lxc_start.c:main:348 - Additional information can be obtained by setting the --logfile and --logpriority options.

 

[dietmar]

I can create a new container which does boot up, but it won't start existing containers.

Maybe there are still hanging NFS mounts.. Does it help if you reboot the node?

 

[Robertas]

Maybe there are still hanging NFS mounts.. Does it help if you reboot the node?

Already tried that, and checked that the storage is actually mounted, which it is.

 

[dietmar]

Any hint in /var/log/syslog?

 

[Robertas]

It seems my problem is the nfs server itself, my zfs filesystem is faulty which does return generic io error even if I try read the type of container disk, by running `file vm-xxx.raw`. Thanks for help!