Conversion Issue

carlosmp

Member
Jun 2, 2010
44
1
6
Hi,

I moved several machines from our Proxmox 3.x cluster to our new 4.1. Most of our containers are having odd issues related to databases. It appears that during the conversion, the /usr/sbin and /usr/local/bin paths are not available any more. The /etc/profile has the correct settings, and don't know where else to look to try and fix this. The source machines are all different versions of centos 6, as we could not convert v5. Has anyone else run into similar issues?

Thanks in advance,
Carlos.
 

carlosmp

Member
Jun 2, 2010
44
1
6
This is happenning to every single Centos machine we convert. Is there another way to migrate from backup? Thinking something is not getting converted, as other services as Mysql are not creating the socks correctly. These pbx's were working correctly. I'm thinking something in the permissions during conversion is not correct. Permissions inside the container appear fine, but not sure what could be the case...

Any ideas of what to look for? Not looking forward to have to manually re-create all of our customer pbx systems...

THanks,
Carlos.
 

carlosmp

Member
Jun 2, 2010
44
1
6
Here's what I'm seeing:

Code:
[root@cmp ~]# locate amportal
/etc/amportal.conf
/usr/local/sbin/amportal
/usr/sbin/sysadmin_amportal_restart
/usr/src/freepbx/amportal.conf
/usr/src/freepbx/amp_conf/bin/retrieve_parse_amportal_conf.pl
/usr/src/freepbx/amp_conf/sbin/amportal
/usr/src/freepbx/upgrades/1.10.010beta1/amportal_conf_options.php
/var/lib/asterisk/bin/retrieve_parse_amportal_conf.pl
/var/spool/asterisk/sysadmin/amportal_restart
/var/www/html/admin/modules/framework/amportal.conf
/var/www/html/admin/modules/framework/amp_conf/bin/retrieve_parse_amportal_conf.pl
/var/www/html/admin/modules/framework/amp_conf/sbin/amportal
/var/www/html/admin/modules/framework/upgrades/1.10.010beta1/amportal_conf_options.php

[root@cmp ~]# echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin

[root@cmp ~]# amportal
bash: amportal: command not found

[root@cmp ~]# /usr/local/sbin/amportal  

Please wait...

-------------FreePBX Control Script-----------------------------------------------

Usage:  amportal start|stop|restart|kill|chown

start:  Starts Asterisk and Flash Operator Panel server if enabled
stop:  Gracefully stops Asterisk
restart:  Stop and Starts
kill:  Kills Asterisk
chown:  Sets appropriate permissions on files

[root@cmp ~]# ls -l /usr/local/sbin
total 4
-rwxr-xr-x 1 root root 2226 May 28  2014 amportal

[root@cmp ~]#
 

carlosmp

Member
Jun 2, 2010
44
1
6
So after noticing that the /usr/local/sbin is not there, I looked at the etc/profile file, and it appears that the /usr/local/sbin is not getting processed, (by pathmung) becuase the $EUID is not 0?. that should be valid for the user root, which when using pct enter xxxx should be connecting as root. THis is confirmed by running whoami, which shows root...
 

carlosmp

Member
Jun 2, 2010
44
1
6
Hi - I'm logging in to the lxc using pct enter <vmid>. Interestingly, this seems to be occuring on Centos machines. It does not appear to be an issue on debian machines. I only have 2 debian machines compared to 20 centos, but I'd expect to be seeing similar issues as most of these are all setup the same...

Thanks,
Carlos.
 

carlosmp

Member
Jun 2, 2010
44
1
6
Also, the problem isn't just limited to the path, we're seeing all sorts of issues - on a mail relay server, getting error message can't connect to mail, on sql servers, applications can't connect to the mysql.sock. Something must be off in the conversion process...I just don't know enough about the process to be able to pinpoint it...
 

windinternet

Member
Oct 8, 2015
159
7
18
Ok, so you are trying to migrate from OpenVZ to LXC containers or from KVM? The single biggest difference between Debian/Ubuntu and Centos/RedHat is selinux. Could it be that your CentOS containers are now trying to enforce selinux permissions, or that they already did, but selinux attributes got lost?
 

windinternet

Member
Oct 8, 2015
159
7
18
I looked some things up. It seems that pct does not create a login shell, so that is a little different from the situation with openVZ on proxmox 3.4.

Anyway, there also seem to be problems concering unix socketfiles and AppArmor on newer kernels like the Proxmox 4 series has. Can you try adding a line to your /etc/pve/lxc/{vmid}.conf file for testing this?

Code:
lxc.aa_profile: unconfined
...And shutdown and start the container after that..

aa-status should show that it is not confined. If you lookup the pid of the init process of your container, it should not be listed in the enforced section.

This bugreport seems to trace some current work on this: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1446906
 

carlosmp

Member
Jun 2, 2010
44
1
6
wind,

Tried that and no change in the container behavior. I'm using pct enter <vmid>, similar to the vzctl enter <vmid> we used in Proxmox 3.x.

Ok, so you are trying to migrate from OpenVZ to LXC containers or from KVM? The single biggest difference between Debian/Ubuntu and Centos/RedHat is selinux. Could it be that your CentOS containers are now trying to enforce selinux permissions, or that they already did, but selinux attributes got lost?
We are migrating from centos 6 OpenVZ to LXC (proxmox 3.2 to 4.1) using the backup/restore process. The major change is Storage, moving from a local backed to a ceph/rbd. Also noticing this issue on a newly created templates.

SELinux - We had explicitly disabled previously, and sestatus confirms it's still disabled in the migrated containers.

The issue only appears to affect certain services. Bind/named don't seem to be affected, mysql, postfix seem to be affected. Even without logging on, or logging on through a direct ssh connection, echo $PATH still returns an incomplete path.
 

windinternet

Member
Oct 8, 2015
159
7
18
@carlosmp,

The issue seems to revolve around programs creating a normal Unix path connected socket inside a chroot inside the container. Like the helper programs of postfix do in the Debian 7 template. And there also is a tiny difference if the container is loop mounted on the file system or if it is actually a dir on the filesystem like with the ZFS storage type. In the latter case it does a little bit better, but still fails on close of a socket.

Can you confirm that your mysql install uses a chroot?
 

carlosmp

Member
Jun 2, 2010
44
1
6
Hi,

Mysql was not configured to run in a chroot originally. Not sure that would have been changed during the conversion process...

Thanks,
Carlos.
 

windinternet

Member
Oct 8, 2015
159
7
18
Hello Carlos,

So, if you look in /proc/net/unix, you see the mysql socket starting with a / from the root of the container? /var/run/mysqld/mysql.sock or something?

Do you see any errors in the syslog of the host with apparmor DENIED messages at the moment of using mysql? What is the error you see with mysql?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!