Conversion Issue

[carlosmp]

Hi,

I moved several machines from our Proxmox 3.x cluster to our new 4.1. Most of our containers are having odd issues related to databases. It appears that during the conversion, the /usr/sbin and /usr/local/bin paths are not available any more. The /etc/profile has the correct settings, and don't know where else to look to try and fix this. The source machines are all different versions of centos 6, as we could not convert v5. Has anyone else run into similar issues?

Thanks in advance,
Carlos.

 

[carlosmp]

This is happenning to every single Centos machine we convert. Is there another way to migrate from backup? Thinking something is not getting converted, as other services as Mysql are not creating the socks correctly. These pbx's were working correctly. I'm thinking something in the permissions during conversion is not correct. Permissions inside the container appear fine, but not sure what could be the case...

Any ideas of what to look for? Not looking forward to have to manually re-create all of our customer pbx systems...

THanks,
Carlos.

 

[carlosmp]

Here's what I'm seeing:

[root@cmp ~]# locate amportal
/etc/amportal.conf
/usr/local/sbin/amportal
/usr/sbin/sysadmin_amportal_restart
/usr/src/freepbx/amportal.conf
/usr/src/freepbx/amp_conf/bin/retrieve_parse_amportal_conf.pl
/usr/src/freepbx/amp_conf/sbin/amportal
/usr/src/freepbx/upgrades/1.10.010beta1/amportal_conf_options.php
/var/lib/asterisk/bin/retrieve_parse_amportal_conf.pl
/var/spool/asterisk/sysadmin/amportal_restart
/var/www/html/admin/modules/framework/amportal.conf
/var/www/html/admin/modules/framework/amp_conf/bin/retrieve_parse_amportal_conf.pl
/var/www/html/admin/modules/framework/amp_conf/sbin/amportal
/var/www/html/admin/modules/framework/upgrades/1.10.010beta1/amportal_conf_options.php

[root@cmp ~]# echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin

[root@cmp ~]# amportal
bash: amportal: command not found

[root@cmp ~]# /usr/local/sbin/amportal  

Please wait...

-------------FreePBX Control Script-----------------------------------------------

Usage:  amportal start|stop|restart|kill|chown

start:  Starts Asterisk and Flash Operator Panel server if enabled
stop:  Gracefully stops Asterisk
restart:  Stop and Starts
kill:  Kills Asterisk
chown:  Sets appropriate permissions on files

[root@cmp ~]# ls -l /usr/local/sbin
total 4
-rwxr-xr-x 1 root root 2226 May 28  2014 amportal

[root@cmp ~]#

 

[carlosmp]

So after noticing that the /usr/local/sbin is not there, I looked at the etc/profile file, and it appears that the /usr/local/sbin is not getting processed, (by pathmung) becuase the $EUID is not 0?. that should be valid for the user root, which when using pct enter xxxx should be connecting as root. THis is confirmed by running whoami, which shows root...

 

[windinternet]

@carlosmp,

It is not completely clear to me how you are entering the machine here. But I do think that it doesn't start a login shell, because then /etc/profile would run. You can start login shells with a -l to your shell or with a - in su or -i in sudo.

Also see http://askubuntu.com/questions/247738/why-is-etc-profile-not-invoked-for-non-login-shells

 

[carlosmp]

Hi - I'm logging in to the lxc using pct enter <vmid>. Interestingly, this seems to be occuring on Centos machines. It does not appear to be an issue on debian machines. I only have 2 debian machines compared to 20 centos, but I'd expect to be seeing similar issues as most of these are all setup the same...

Thanks,
Carlos.

 

[carlosmp]

Also, the problem isn't just limited to the path, we're seeing all sorts of issues - on a mail relay server, getting error message can't connect to mail, on sql servers, applications can't connect to the mysql.sock. Something must be off in the conversion process...I just don't know enough about the process to be able to pinpoint it...

 

[windinternet]

Ok, so you are trying to migrate from OpenVZ to LXC containers or from KVM? The single biggest difference between Debian/Ubuntu and Centos/RedHat is selinux. Could it be that your CentOS containers are now trying to enforce selinux permissions, or that they already did, but selinux attributes got lost?

 

[windinternet]

I looked some things up. It seems that pct does not create a login shell, so that is a little different from the situation with openVZ on proxmox 3.4.

Anyway, there also seem to be problems concering unix socketfiles and AppArmor on newer kernels like the Proxmox 4 series has. Can you try adding a line to your /etc/pve/lxc/{vmid}.conf file for testing this?

lxc.aa_profile: unconfined

...And shutdown and start the container after that..

aa-status should show that it is not confined. If you lookup the pid of the init process of your container, it should not be listed in the enforced section.

This bugreport seems to trace some current work on this: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1446906

 

[carlosmp]

wind,

Tried that and no change in the container behavior. I'm using pct enter <vmid>, similar to the vzctl enter <vmid> we used in Proxmox 3.x.

Ok, so you are trying to migrate from OpenVZ to LXC containers or from KVM? The single biggest difference between Debian/Ubuntu and Centos/RedHat is selinux. Could it be that your CentOS containers are now trying to enforce selinux permissions, or that they already did, but selinux attributes got lost?

We are migrating from centos 6 OpenVZ to LXC (proxmox 3.2 to 4.1) using the backup/restore process. The major change is Storage, moving from a local backed to a ceph/rbd. Also noticing this issue on a newly created templates.

SELinux - We had explicitly disabled previously, and sestatus confirms it's still disabled in the migrated containers.

The issue only appears to affect certain services. Bind/named don't seem to be affected, mysql, postfix seem to be affected. Even without logging on, or logging on through a direct ssh connection, echo $PATH still returns an incomplete path.

 

[windinternet]

@carlosmp,

The issue seems to revolve around programs creating a normal Unix path connected socket inside a chroot inside the container. Like the helper programs of postfix do in the Debian 7 template. And there also is a tiny difference if the container is loop mounted on the file system or if it is actually a dir on the filesystem like with the ZFS storage type. In the latter case it does a little bit better, but still fails on close of a socket.

Can you confirm that your mysql install uses a chroot?

 

[carlosmp]

Hi,

Mysql was not configured to run in a chroot originally. Not sure that would have been changed during the conversion process...

Thanks,
Carlos.

 

[windinternet]

Hello Carlos,

So, if you look in /proc/net/unix, you see the mysql socket starting with a / from the root of the container? /var/run/mysqld/mysql.sock or something?

Do you see any errors in the syslog of the host with apparmor DENIED messages at the moment of using mysql? What is the error you see with mysql?