Move LXCs to new server - errors

M

MnM

Renowned Member
May 12, 2016
16
1
68
Hi,

While testing ProxMox 4.2 (and waiting for the proper storage) I used a small SSD (as local storage) and created a few LXCs.
All worked well, no issues. I shut down the LXCs and backed them up with the GUI. Transferred them to new PC via WinSCP.
After I got my new SSDs I removed the small SSD and built ProxMox from scratch (same version or the previous one = 4.2 + the updates available).
I added one of the new SSDs (LV, formatted wiht ext4 and mounted via /etc/fstab)
The old storage was called TempStorage while on the newly built server it is called LocalStorage
I then used WinSCP to move the backed up LXCs form my PC to the newProxMox build.
Next I have used the GUI to restore the backup (only tested one so far).LXC server starts but is doesnt run properly. In the logs (on the LXC and ProxMox) I see these errors:

Code: 
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.197:50): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/pstore/" pid=24622 comm="mount" fstype="pstore" srcname="pstore"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.197:51): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/pstore/" pid=24622 comm="mount" fstype="pstore" srcname="pstore" flags="ro"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.245:52): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=24824 comm="mount" flags="rw, remount, silent"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.245:53): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=24825 comm="mount" flags="rw, remount, relatime"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.261:54): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/" pid=24880 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.261:55): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/lock/" pid=24891 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.265:56): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/proc/" pid=24898 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.265:57): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/" pid=24906 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
May 13 15:51:03 proxmox kernel: audit: type=1400 audit(1463125863.281:58): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=24972 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"

So while the server starts and I can login, required services are not running due to the above.
Am I doing something wrong? In the forums here it is mentioned that if the LXCs are shut down when backed up they should import on the new server OK. I double checked this as I have the small SSD - LXC run OK, shut down one, back it up, move the file, etc = same errors.
Can I use the old installation and LXCs and attach the new SSD and migrate them somehow?

Thanks for your help.
 
does this issue also occur with a container created from scratch? you can get more meaningful debug logs from lxc by starting a container with "lxc-start -n ID -F -ldebug -o /tmp/lxc-debug.log" instead of "pct start ID". note that the lxc-start command does not terminate until you shut down the container (e.g., with "pct shutdown ID" in a second shell). the debug log will be in /tmp/lxc-debug.log
 
Thanks for helping.
Tried a new container (same TurnKey template for the one with issues). Works fine.

Below is the log file:

Code: 
root@proxmox:~# lxc-start -n 100 -F -ldebug -o /tmp/lxc-debug.log
readline() on closed filehandle $fd at /usr/share/lxc/hooks/lxc-pve-autodev-hook line 32.
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
mount: pstore is write-protected, mounting read-only
mount: cannot mount pstore read-only
[warn] udev does not support containers, not started ... (warning).
[ ok ] Setting parameters of disc: (none).
[info] Setting the system clock.
hwclock: Cannot access the Hardware Clock via any known method.
hwclock: Use the --debug option to see the details of our search for an access method.
[warn] Unable to set System Clock to: Fri May 13 08:52:30 UTC 2016 ... (warning).
[....] Setting up LVM Volume Groups...File descriptor 9 (anon_inode:[signalfd]) leaked on lvm invocation. Parent PID 309: /bin/sh
  No volume groups found
done.
[ ok ] Activating swap...done.
mount: cannot remount /dev/loop0 read-write, is write-protected
mount: cannot remount tmpfs read-write, is write-protected
mount: cannot remount tmpfs read-write, is write-protected
mount: cannot remount proc read-write, is write-protected
mount: cannot mount sysfs read-only
mount: cannot remount tmpfs read-write, is write-protected
mount: cannot remount devpts read-write, is write-protected
[warn] Fast boot enabled, so skipping file system check. ... (warning).
[ ok ] Cleaning up temporary files... /tmp.
[ ok ] Mounting local filesystems...done.
[ ok ] Activating swapfile swap...done.
[ ok ] Cleaning up temporary files....
[....] Setting kernel variables ...sysctl: setting key "kernel.printk": Read-only file system
failed.
[....] Configuring network interfaces...udhcpc: SIOCGIFINDEX: No such device
Failed to bring up eth1.
done.
[ ok ] Cleaning up temporary files....
[info] Setting console screen modes and fonts.
/etc/init.d/kbd: 162: /etc/init.d/kbd: cannot create /sys/module/vt/parameters/default_utf8: Read-only file system
setterm: cannot (un)set powersave mode: Inappropriate ioctl for device
Couldn't get a file descriptor referring to the console
Couldn't get a file descriptor referring to the console
[ ok ] Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
[info] Using makefile-style concurrent boot in runlevel 2.
[....] Starting cgroup management daemon: cgmanager[....] Starting cgroup management proxy daemon: cgproxy[....] Starting Initialization hooksRedirecting output to /var/log/inithooks.log
done.
[ ok ] Starting cgroup management proxy daemon: cgproxy[....] Starting uuid generator: uuidd.
[ ok ] Starting enhanced syslogd: rsyslogd.
[....] Starting ACPI services...RTNETLINK1 answers: No such file or directory
acpid: error talking to the kernel via netlink
. ok
[FAIL] Starting web server: apache2 failed!
[warn] The apache2 configtest failed. ... (warning).
Output of config test was:
(2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for main error log
AH00014: Configuration check failed
Action 'configtest' failed.
The Apache error log may have more information.
Starting SSL tunnels: [Started: /etc/stunnel/stunnel.conf] stunnel.
[ ok ] Starting periodic command scheduler: cron.
[ ok ] Starting OpenBSD Secure Shell server: sshd.
[FAIL] Starting MySQL database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
[ ok ] Starting Postfix Mail Transport Agent: postfix.
[ ok ] Starting daemon monitor: monit.
[ ok ] Starting webmindone.
[FAIL] startpar: service(s) returned failure: apache2 mysql ... failed!
^C
INIT: Sending processes the TERM signal
[info] Using makefile-style concurrent boot in runlevel 0.
[ ok ] Stopping daemon monitor: monit.
[....] Stopping cgroup management daemon: cgmanagerStopping SSL tunnels: [stopped: /etc/stunnel/stunnel.conf] stunnel.
[ ok ] Stopping uuidd generator: uuidd.
[ ok ] Stopping quota service: rpc.rquotad.
[....] Stopping webminStopping Webmin server in /usr/share/webmin
done.
[ ok ] Stopping web server: apache2.
[ ok ] Turning off quotas...done.
[ ok ] Stopping cgroup management proxy daemon: cgproxy[....] Stopping Postfix Mail Transport Agent: postfix.
[ ok ] Stopping MySQL database server: mysqld.
[ ok ] Asking all remaining processes to terminate...done.
[ ok ] All processes ended within 1 seconds...done.
[ ok ] Stopping enhanced syslogd: rsyslogd.
[info] Saving the system clock.
hwclock: Cannot access the Hardware Clock via any known method.
hwclock: Use the --debug option to see the details of our search for an access method.
[ ok ] Deconfiguring network interfaces...done.
[warn] Deactivating swap...failed.
mount: cannot mount /dev/loop0 read-only
[info] Will now halt.
root@proxmox:~# vi /tmp/lxc-debug.log
 
And part 2 of the log:

Code: 
root@proxmox:~# vi /tmp/lxc-debug.log
      lxc-start 1463129549.730 INFO     lxc_start_ui - lxc_start.c:main:264 - using rcfile /var/lib/lxc/100/config
      lxc-start 1463129549.730 WARN     lxc_confile - confile.c:config_pivotdir:1817 - lxc.pivotdir is ignored.  It will soon become an error.
      lxc-start 1463129549.731 WARN     lxc_cgmanager - cgmanager.c:cgm_get:994 - do_cgm_get exited with error
      lxc-start 1463129549.731 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for reject_force_umount action 0
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:216 - Setting seccomp rule to reject force umounts

      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for reject_force_umount action 0
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:216 - Setting seccomp rule to reject force umounts

      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .[all].
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .kexec_load errno 1.
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for kexec_load action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for kexec_load action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .open_by_handle_at errno 1.
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for open_by_handle_at action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for open_by_handle_at action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .init_module errno 1.
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for init_module action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for init_module action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .finit_module errno 1.
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for finit_module action 327681
      lxc-start 1463129549.731 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:233 - Seccomp: got negative # for syscall: finit_module
      lxc-start 1463129549.731 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:234 - This syscall will NOT be blacklisted
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for finit_module action 327681
      lxc-start 1463129549.731 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:233 - Seccomp: got negative # for syscall: finit_module
      lxc-start 1463129549.731 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:234 - This syscall will NOT be blacklisted
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:324 - processing: .delete_module errno 1.
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:426 - Adding native rule for delete_module action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:429 - Adding compat rule for delete_module action 327681
      lxc-start 1463129549.731 INFO     lxc_seccomp - seccomp.c:parse_config_v2:436 - Merging in the compat seccomp ctx into the main one
      lxc-start 1463129549.731 INFO     lxc_conf - conf.c:run_script_argv:362 - Executing script '/usr/share/lxc/hooks/lxc-pve-prestart-hook' for container '100', config section 'lxc'
      lxc-start 1463129549.999 DEBUG    lxc_start - start.c:setup_signal_fd:264 - sigchild handler set
      lxc-start 1463129549.999 DEBUG    lxc_console - console.c:lxc_console_peer_default:500 - opening /dev/tty for console peer
      lxc-start 1463129549.999 DEBUG    lxc_console - console.c:lxc_console_peer_default:506 - using '/dev/tty' as console
      lxc-start 1463129549.999 DEBUG    lxc_console - console.c:lxc_console_sigwinch_init:179 - 12445 got SIGWINCH fd 9
      lxc-start 1463129549.999 DEBUG    lxc_console - console.c:lxc_console_winsz:88 - set winsz dstfd:6 cols:184 rows:24
      lxc-start 1463129549.999 INFO     lxc_start - start.c:lxc_init:460 - '100' is initialized
      lxc-start 1463129549.999 DEBUG    lxc_start - start.c:__lxc_start:1184 - Not dropping cap_sys_boot or watching utmp
      lxc-start 1463129550.000 INFO     lxc_conf - conf.c:run_script:412 - Executing script '/usr/share/lxc/lxcnetaddbr' for container '100', config section 'net'
      lxc-start 1463129550.221 DEBUG    lxc_conf - conf.c:instantiate_veth:2826 - instantiated veth 'veth100i0/vethUYWV96', index is '12'
      lxc-start 1463129550.221 INFO     lxc_cgroup - cgroup.c:cgroup_init:65 - cgroup driver cgmanager initing for 100
      lxc-start 1463129550.223 DEBUG    lxc_cgmanager - cgmanager.c:cgm_setup_limits:1394 - cgroup 'memory.limit_in_bytes' set to '536870912'
      lxc-start 1463129550.223 DEBUG    lxc_cgmanager - cgmanager.c:cgm_setup_limits:1394 - cgroup 'memory.memsw.limit_in_bytes' set to '1073741824'
      lxc-start 1463129550.223 DEBUG    lxc_cgmanager - cgmanager.c:cgm_setup_limits:1394 - cgroup 'cpu.cfs_period_us' set to '100000'
      lxc-start 1463129550.223 DEBUG    lxc_cgmanager - cgmanager.c:cgm_setup_limits:1394 - cgroup 'cpu.cfs_quota_us' set to '100000'
      lxc-start 1463129550.223 DEBUG    lxc_cgmanager - cgmanager.c:cgm_setup_limits:1394 - cgroup 'cpu.shares' set to '1024'
      lxc-start 1463129550.223 INFO     lxc_cgmanager - cgmanager.c:cgm_setup_limits:1398 - cgroup limits have been setup
      lxc-start 1463129550.268 DEBUG    lxc_conf - conf.c:lxc_assign_network:3247 - move 'eth0' to '12499'
 
OK hopefully this will help someone else.
After a lot of reading and research this was the solution to fix my issue:

vi /etc/vzdump.conf and add
Code: 
stdexcludes: 0
reboot ProxMox host

You will have to take new backups after adding the above. If you have old backups they will not work. I still had all my original installation with working LXCs so I could take new backups.

without the above some dirs are not backed up and when restoring services will fail as the folders are missing.
In regards to the apparmor yes they are still there in the logs every time I restart my LXCs ...but all works as it should so not really sure what it means..
 
  • Like
Reactions: Ovidiu
You must log in or register to reply here.
Top Bottom
Back