Problems with Containers after upgrade to PVE 4

[Einar Stenberg]

I have just upgraded my servers to PVE 4.
I had some containers and some VM's. THe normal vm's are working fine, but the container seems to be hanging somhow when i start them.
They are all converted from openvz with the backup/restore as described on wiki.

When i start one they hang with 13-14 mb of ram in use and 0% cpu activity, no output in console.

In the syslog of server I get these error messages when container is starting; (something wrong with apparmor?)

Oct 07 23:56:29 Freya pvedaemon[14850]: starting CT 104: UPID:Freya:00003A02:000506C1:5615950D:vzstart:104:root@pam:
Oct 07 23:56:29 Freya kernel: EXT4-fs (loop1): couldn't mount as ext3 due to feature incompatibilities
Oct 07 23:56:29 Freya kernel: EXT4-fs (loop1): couldn't mount as ext2 due to feature incompatibilities
Oct 07 23:56:29 Freya kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)
Oct 07 23:56:30 Freya pvedaemon[15390]: command 'lxc-attach -n 104 -- df -P -B 1 /' failed: got timeout
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.101:98): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=15508 comm="mount" flags="rw, remount"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.101:99): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=15509 comm="mount" flags="rw, remount"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.437:100): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/" pid=15761 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.445:101): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/lock/" pid=15779 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.453:102): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/proc/" pid=15789 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.465:103): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/" pid=15800 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.565:104): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=15964 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Oct 07 23:56:32 Freya kernel: audit: type=1400 audit(1444254992.569:105): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/dev/pts/" pid=15975 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
Oct 07 23:58:44 Freya pvedaemon[11632]: <root@pam> starting task UPID:Freya:00004A89:00053BC7:56159594:vncproxy:104:root@pam:
Oct 07 23:58:44 Freya pvedaemon[19081]: starting lxc vnc proxy UPID:Freya:00004A89:00053BC7:56159594:vncproxy:104:root@pam:
Oct 08 00:00:34 Freya pveproxy[7830]: worker exit
Oct 08 00:00:34 Freya pveproxy[2592]: worker 7830 finished
Oct 08 00:00:34 Freya pveproxy[2592]: starting 1 worker(s)
Oct 08 00:00:34 Freya pveproxy[2592]: worker 21104 started
Oct 08 00:01:48 Freya rrdcached[889]: flushing old values
Oct 08 00:01:48 Freya rrdcached[889]: rotating journals
Oct 08 00:01:48 Freya rrdcached[889]: started new journal /var/lib/rrdcached/journal/rrd.journal.1444255308.639706
Oct 08 00:02:34 Freya pveproxy[3588]: worker exit
Oct 08 00:02:34 Freya pveproxy[2592]: worker 3588 finished
Oct 08 00:02:34 Freya pveproxy[2592]: starting 1 worker(s)
Oct 08 00:02:34 Freya pveproxy[2592]: worker 23155 started
Oct 08 00:02:36 Freya pvedaemon[15390]: worker exit
Oct 08 00:02:36 Freya pvedaemon[1257]: worker 15390 finished
Oct 08 00:02:36 Freya pvedaemon[1257]: starting 1 worker(s)
Oct 08 00:02:36 Freya pvedaemon[1257]: worker 23185 started

 

[dietmar]

In the syslog of server I get these error messages when container is starting; (something wrong with apparmor?)

Those warnings are quite normal. Those things are already mounted by LXC, so that is OK.

What is the output of

# pct exec 104 ps auxww

 

[Einar Stenberg]

output of pct exec 104 ps auxww

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 10660 1580 ? Ss Oct07 0:00 init boot
root 66 0.0 0.0 10660 120 console Ss+ Oct07 0:00 init boot
root 67 0.0 0.0 4192 1392 console S+ Oct07 0:00 /bin/sh /etc/init.d/rc S
root 1653 0.0 0.0 4192 1528 console S+ Oct07 0:00 /bin/sh /etc/rcS.d/S08checkfs.sh start
root 1744 0.0 0.0 19016 1684 console S+ Oct07 0:00 sulogin /dev/console
root 2473 0.0 0.1 16848 2384 ? R+ 07:17 0:00 ps auxww

 

[dietmar]

What OS is that exactly (inside container)?

 

[Einar Stenberg]

The containers are all Debian 7 x64

 

[dietmar]

The containers are all Debian 7 x64

I uploaded a bug fix to pvetest repository. Please can you test?

# wget http://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-container_1.0-7_all.deb
# dpkg -i pve-container_1.0-7_all.deb

 

[Einar Stenberg]

Great! Now they are starting and running properly.
But they seem to take quite a long time to start, before in openvz these would be up in 10 seconds or less (running small websites in apache2) now they take a approx 1 minute. (looks like they just stopped at the 13-14 mb ram for this time). I created a new ct base on a fresh template and installed apache2 in it, and that one appears to be starting a lot faster. is this normal?

 

[dietmar]

is this normal?

No, this is not normal. Maybe you can try to find the blocking process during startup using 'ps' again.