LVM2 broken - please help!

I

Iventa Markus Kaefer

New Member
Nov 19, 2017
7
0
1
41
Hello!

I have a problem after a crash on my server. My RAID Controller crashed and was replaced by the server provider. (it is a dedicated server)

I am running Debian / Proxmox 4.4-12/e71b7a74.

After the replacement Proxmox booted normally and mounted root and swap filesystem.
the vg pve was completly inactive and needed manual repair to activate them.

First I was not able to activate the LVs but after
lvconvert --repair pve/data
a data_meta0 was created and I was able to activate the LVs with the command
lvchange -a y pve/data

BUT my lxc's are not booting. The web interface only gives me the message that I should look at the systemctl.
the Systemctl shows:

Code: 
root@david:~# systemctl status lxc@101.service
● lxc@101.service - LXC Container: 101
   Loaded: loaded (/lib/systemd/system/lxc@.service; disabled)
  Drop-In: /usr/lib/systemd/system/lxc@.service.d
          └─pve-reboot.conf
   Active: failed (Result: exit-code) since Sun 2017-11-19 22:07:26 CET; 39min ago
    Docs: man:lxc-start
          man:lxc
  Process: 6452 ExecStart=/usr/bin/lxc-start -n %i (code=exited, status=1/FAILURE)

Nov 19 22:07:26 david lxc-start[6452]: lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
Nov 19 22:07:26 david lxc-start[6452]: lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
Nov 19 22:07:26 david lxc-start[6452]: lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.
Nov 19 22:07:26 david systemd[1]: lxc@101.service: control process exited, code=exited status=1
Nov 19 22:07:26 david systemd[1]: Failed to start LXC Container: 101.
Nov 19 22:07:26 david systemd[1]: Unit lxc@101.service entered failed state.

If I turn on DEBUG the following entries are in the logfile:

Code: 
     lxc-start 20171119221009.708 INFO     lxc_start_ui - tools/lxc_start.c:main:275 - using rcfile /var/lib/lxc/100/config
     lxc-start 20171119221009.708 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
     lxc-start 20171119221009.708 WARN     lxc_start - start.c:lxc_check_inherited:238 - Inherited fd: 3.
     lxc-start 20171119221009.708 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for reject_force_umount action 0.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for reject_force_umount action 0.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .[all].
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .kexec_load errno 1.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for kexec_load action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for kexec_load action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .open_by_handle_at errno 1.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for open_by_handle_at action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for open_by_handle_at action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .init_module errno 1.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for init_module action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for init_module action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .finit_module errno 1.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for finit_module action 327681.
     lxc-start 20171119221009.708 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:270 - Seccomp: got negative for syscall: -10085: finit_module.
     lxc-start 20171119221009.708 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:271 - This syscall will NOT be blacklisted.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for finit_module action 327681.
     lxc-start 20171119221009.708 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:270 - Seccomp: got negative for syscall: -10085: finit_module.
     lxc-start 20171119221009.708 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:271 - This syscall will NOT be blacklisted.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .delete_module errno 1.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for delete_module action 327681.
     lxc-start 20171119221009.708 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for delete_module action 327681.
     lxc-start 20171119221009.709 INFO     lxc_seccomp - seccomp.c:parse_config_v2:580 - Merging in the compat Seccomp ctx into the main one.
     lxc-start 20171119221009.709 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc".
     lxc-start 20171119221010.237 DEBUG    lxc_start - start.c:setup_signal_fd:273 - Set SIGCHLD handler with file descriptor: 6.
     lxc-start 20171119221010.238 DEBUG    console - console.c:lxc_console_peer_default:430 - opening /dev/tty for console peer
     lxc-start 20171119221010.238 DEBUG    console - console.c:lxc_console_peer_default:436 - using '/dev/tty' as console
     lxc-start 20171119221010.238 DEBUG    console - console.c:lxc_console_sigwinch_init:144 - 16592 got SIGWINCH fd 10
     lxc-start 20171119221010.238 DEBUG    console - console.c:lxc_console_winsz:71 - set winsz dstfd:7 cols:209 rows:48
     lxc-start 20171119221010.238 INFO     lxc_start - start.c:lxc_init:475 - Container "100" is initialized.
     lxc-start 20171119221010.238 DEBUG    lxc_start - start.c:__lxc_start:1336 - Not dropping CAP_SYS_BOOT or watching utmp.
     lxc-start 20171119221010.239 INFO     lxc_conf - conf.c:run_script:475 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "100", config section "net".
     lxc-start 20171119221010.576 DEBUG    lxc_conf - conf.c:instantiate_veth:2694 - instantiated veth 'veth100i0/vethV1DPDW', index is '8'
     lxc-start 20171119221010.576 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for 100
     lxc-start 20171119221010.576 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:filter_and_set_cpus:475 - No isolated cpus detected.
     lxc-start 20171119221010.576 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:handle_cpuset_hierarchy:645 - "cgroup.clone_children" was already set to "1".
     lxc-start 20171119221010.577 INFO     lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWNS.
     lxc-start 20171119221010.577 INFO     lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWPID.
     lxc-start 20171119221010.577 INFO     lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWUTS.
     lxc-start 20171119221010.577 INFO     lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWIPC.
     lxc-start 20171119221010.577 INFO     lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWNET.
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.deny' set to 'a'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c *:* m'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'b *:* m'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:3 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:5 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:7 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 5:0 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 5:1 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 5:2 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:8 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:9 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 136:* rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:229 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 254:0 rm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:200 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:228 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:232 rwm'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'memory.limit_in_bytes' set to '8589934592'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'memory.memsw.limit_in_bytes' set to '10737418240'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'cpu.shares' set to '1024'
     lxc-start 20171119221010.577 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'cpuset.cpus' set to '2-4,6'
     lxc-start 20171119221010.577 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2013 - cgroup has been setup
     lxc-start 20171119221010.623 DEBUG    lxc_conf - conf.c:lxc_assign_network:3190 - move 'vethV1DPDW'/'eth0' to '16635': .
     lxc-start 20171119221010.625 DEBUG    lxc_conf - conf.c:setup_rootfs:1273 - mounted '/var/lib/lxc/100/rootfs' on '/usr/lib/x86_64-linux-gnu/lxc/rootfs'
     lxc-start 20171119221010.625 INFO     lxc_conf - conf.c:setup_utsname:901 - 'web' hostname has been setup
     lxc-start 20171119221010.639 DEBUG    lxc_conf - conf.c:setup_hw_addr:2225 - mac address '56:C2:08:8A:9F:CB' on 'eth0' has been setup
     lxc-start 20171119221010.639 DEBUG    lxc_conf - conf.c:setup_netdev:2452 - 'eth0' has been setup
     lxc-start 20171119221010.639 INFO     lxc_conf - conf.c:setup_network:2473 - network has been setup
     lxc-start 20171119221010.639 INFO     lxc_conf - conf.c:mount_autodev:1130 - Mounting container /dev
     lxc-start 20171119221010.639 INFO     lxc_conf - conf.c:mount_autodev:1153 - Mounted tmpfs onto /usr/lib/x86_64-linux-gnu/lxc/rootfs/dev
     lxc-start 20171119221010.639 INFO     lxc_conf - conf.c:mount_autodev:1171 - Mounted container /dev
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1715 - remounting /sys/fs/fuse/connections on /usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections to respect bind or remount options
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1730 - (at remount) flags for /sys/fs/fuse/connections was 4096, required extra flags are 0
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1739 - mountflags already was 4096, skipping remount
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1765 - mounted '/sys/fs/fuse/connections' on '/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections', type 'none'
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1715 - remounting /sys/kernel/debug on /usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/kernel/debug to respect bind or remount options
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1730 - (at remount) flags for /sys/kernel/debug was 4096, required extra flags are 0
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1739 - mountflags already was 4096, skipping remount
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1765 - mounted '/sys/kernel/debug' on '/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/kernel/debug', type 'none'
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1715 - remounting /sys/kernel/security on /usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/kernel/security to respect bind or remount options
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1730 - (at remount) flags for /sys/kernel/security was 4110, required extra flags are 14
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1765 - mounted '/sys/kernel/security' on '/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/kernel/security', type 'none'
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1715 - remounting /sys/fs/pstore on /usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/pstore to respect bind or remount options
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1730 - (at remount) flags for /sys/fs/pstore was 4110, required extra flags are 14
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1765 - mounted '/sys/fs/pstore' on '/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/pstore', type 'none'
     lxc-start 20171119221010.663 DEBUG    lxc_conf - conf.c:mount_entry:1765 - mounted 'mqueue' on '/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/mqueue', type 'mqueue'
     lxc-start 20171119221010.663 INFO     lxc_conf - conf.c:mount_file_entries:1985 - mount points have been setup
     lxc-start 20171119221010.663 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "100", config section "lxc".
     lxc-start 20171119221010.686 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxc/hooks/lxc-pve-autodev-hook" for container "100", config section "lxc".
     lxc-start 20171119221010.807 INFO     lxc_conf - conf.c:fill_autodev:1199 - Creating initial consoles under container /dev
     lxc-start 20171119221010.807 INFO     lxc_conf - conf.c:fill_autodev:1210 - Populating container /dev
     lxc-start 20171119221010.807 INFO     lxc_conf - conf.c:fill_autodev:1247 - Populated container /dev
     lxc-start 20171119221010.807 INFO     lxc_conf - conf.c:setup_ttydir_console:1516 - created /usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc
     lxc-start 20171119221010.807 INFO     lxc_conf - conf.c:setup_ttydir_console:1562 - console has been setup on lxc/console
     lxc-start 20171119221010.807 INFO     lxc_utils - utils.c:mount_proc_if_needed:1785 - I am 1, /proc/self points to '1'
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_rootfs_pivot_root:1108 - pivot_root syscall to '/usr/lib/x86_64-linux-gnu/lxc/rootfs' successful
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:lxc_create_tty:3434 - allocated pty '/dev/pts/0' (12/15)
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:lxc_create_tty:3434 - allocated pty '/dev/pts/1' (16/17)
     lxc-start 20171119221010.819 INFO     lxc_conf - conf.c:lxc_create_tty:3445 - tty's configured
     lxc-start 20171119221010.819 INFO     lxc_conf - conf.c:setup_tty:1053 - 2 tty(s) has been setup
     lxc-start 20171119221010.819 INFO     lxc_conf - conf.c:setup_personality:1451 - set personality to '0x0'
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_caps:2135 - drop capability 'mac_admin' (33)
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_caps:2135 - drop capability 'mac_override' (32)
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_caps:2135 - drop capability 'sys_time' (25)
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_caps:2135 - drop capability 'sys_module' (16)
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_caps:2135 - drop capability 'sys_rawio' (17)
     lxc-start 20171119221010.819 DEBUG    lxc_conf - conf.c:setup_caps:2144 - capabilities have been setup
     lxc-start 20171119221010.819 NOTICE   lxc_conf - conf.c:lxc_setup:3982 - '100' is setup.
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.deny' set to 'a'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c *:* m'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'b *:* m'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:3 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:5 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:7 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 5:0 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 5:1 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 5:2 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:8 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 1:9 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 136:* rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:229 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 254:0 rm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:200 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:228 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'devices.allow' set to 'c 10:232 rwm'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'memory.limit_in_bytes' set to '8589934592'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'memory.memsw.limit_in_bytes' set to '10737418240'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'cpu.shares' set to '1024'
     lxc-start 20171119221010.819 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2009 - cgroup 'cpuset.cpus' set to '2-4,6'
     lxc-start 20171119221010.819 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2013 - cgroup has been setup
     lxc-start 20171119221010.839 INFO     lxc_start - start.c:do_start:836 - Unshared CLONE_NEWCGROUP.
     lxc-start 20171119221010.839 INFO     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:238 - changed apparmor profile to lxc-container-default-cgns
     lxc-start 20171119221010.840 NOTICE   lxc_start - start.c:start:1447 - Exec'ing "/sbin/init".
     lxc-start 20171119221010.896 NOTICE   lxc_start - start.c:post_start:1458 - Started "/sbin/init" with pid "16635".
     lxc-start 20171119221010.896 WARN     lxc_start - start.c:signal_handler:322 - Invalid pid for SIGCHLD. Received pid 16601, expected pid 16635.
     lxc-start 20171119221018.636 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:935 - Peer has disconnected.

This last messages is repeated endlessly.
Can any1 please help! I dont know were to look next. What am I missing?

Thanks in advance

Best regards
Markus
 
Dear LnxBil!

Thanks for you help in advance. Here is the requested Output:

The machines which are not working are 100, 101 and 102.

I restored my backups to the IDs 200, 201 and 202 so this machines were created after the outage.

data_meta0 was created when I issued the command "lvconvert --repair pve/data"

Code: 
root@david:~# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda3  pve  lvm2 a--  278.62g 15.68g
root@david:~#

Code: 
root@david:~# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1  11   0 wz--n- 278.62g 15.68g
root@david:~#

Code: 
root@david:~# lvs
  LV                        VG   Attr       LSize   Pool Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  data                      pve  twi-aotz-- 210.62g                    54.65  41.64
  data_meta0                pve  -wi-a----- 108.00m
  root                      pve  -wi-ao----  50.00g
  snap_vm-100-disk-1_vzdump pve  Vri---tz-k 100.00g data vm-100-disk-1
  swap                      pve  -wi-ao----   2.00g
  vm-100-disk-1             pve  Vwi-a-tz-- 100.00g data               62.36
  vm-101-disk-1             pve  Vwi-a-tz--  20.00g data               44.49
  vm-102-disk-1             pve  Vwi-a-tz--  20.00g data               34.00
  vm-200-disk-1             pve  Vwi-aotz-- 100.00g data               24.17
  vm-201-disk-1             pve  Vwi-aotz--  20.00g data               30.76
  vm-202-disk-1             pve  Vwi-aotz--  20.00g data               33.33
root@david:~#

Best Regards

Markus
 
Thank you. LVM seems fine. Could you try to mount a container via

Code: 
pct mount 100

and inspect the files? You can also chroot into the system and run internal consistency checks. Please refer to the guest os system manual.
 
Dear LnxBil!

I get the following error message when entering var/www/ (it is a Apache webserver, with about 40 Websites running)
Code: 
root@david:/var/lib/lxc/100/rootfs/var# cd www
-bash: cd: www: Structure needs cleaning
root@david:/var/lib/lxc/100/rootfs/var# ls -la
ls: cannot access www: Structure needs cleaning
total 44
drwxr-xr-x 12 root root   4096 Jan 25  2017 .
drwxr-xr-x 22 root root   4096 Nov 20 11:33 ..
drwxr-xr-x  3 root root   4096 Oct 20 08:58 backups
drwxr-xr-x 10 root root   4096 Jan 25  2017 cache
drwxr-xr-x 44 root root   4096 Oct 19 11:29 lib
drwxrwsr-x  2 root staff  4096 Feb  3  2015 local
lrwxrwxrwx  1 root root      9 Jun 30  2015 lock -> /run/lock
drwxrwxr-x 12 root netdev 4096 Nov 19 21:43 log
drwxrwsr-x  2 root mail   4096 Jul 25 08:58 mail
drwxr-xr-x  2 root root   4096 Jun 30  2015 opt
lrwxrwxrwx  1 root root      4 Jun 30  2015 run -> /run
drwxr-xr-x  6 root root   4096 Jun 30  2015 spool
drwxrwxrwt  2 root root   4096 Jul  3 11:17 tmp
d?????????  ? ?    ?         ?            ? www
root@david:/var/lib/lxc/100/rootfs/var#

Other directories like etc look fine.

Do I have to look also for other corrupted files or can I repair them all by one? AND can I repair them?

Best regards

Markus
 
As I suspected errors in/on the filesystem. Besides the mount command, there is also

Code: 
pct fsck 100

You need to unmount the volume before to get it working. If everything else failes, just restore from a previous backup.
 
Its as that easy ... Sorry.
I now get the message:
Code: 
root@david:~# pct fsck 100
fsck from util-linux 2.25.2
/dev/mapper/pve-vm--100--disk--1 contains a file system with errors, check forced.
/dev/mapper/pve-vm--100--disk--1: Inodes that were part of a corrupted orphan linked list found.

/dev/mapper/pve-vm--100--disk--1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
command 'fsck -a -l /dev/pve/vm-100-disk-1' failed: exit code 4
root@david:~#

I found that I should run:
Code: 
e2fsck -E journal_only PATH_to_VMDisk.raw
but my virtual disk is on the LVM there is no path right?
 
But I am able to run it on the device ... Sorry silly question ... just running the repairs with
Code: 
e2fsck -E journal_only /dev/mapper/pve-vm--100--disk--1
for VM 100 that worked.
101 and 102 had additionally the following error.
Code: 
root@david:~# pct fsck 102
fsck from util-linux 2.25.2
/dev/mapper/pve-vm--102--disk--1:
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

fsck.ext2: Bad magic number in super-block while trying to open /dev/mapper/pve-vm--102--disk--1
command 'fsck -a -l /dev/pve/vm-102-disk-1' failed: exit code 8
root@david:~#

So I tried to repair it with:
Code: 
e2fsck -y journal_only /dev/mapper/pve-vm--100--disk--1
 
So after trying to repair the filesystems of the VMs. Here is the result.
The disk of the VM 100 which was the only one which was repairable with the Journal. The problem persists the folder /var/www still repairing. Result data loss.

The other two VM disks 101 & 102 are empty. Result data loss.

My conclusion is if your RAID controller with Memory messages, restore the backup.

Which in my case worked perfectly.
Thank you very much for your help.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back