Upgraded to Proxmox 8.2 - LXC running MySQL busted

[Speedy059]

I upgraded our nodes to 8.2, and now I'm running into a very weird problem on a container running CentOS 8 with mysqld.service. Ever since the upgrade, it did something to the permissions of the container that is messing with mysqld. Anyone run into this?

Errors:
systemctl start mysqld

[root@cloud ~]# systemctl start mysqld
Job for mysqld.service failed because the control process exited with error code.
See "systemctl status mysqld.service" and "journalctl -xe" for details.

systemctl status mysqld.service

[root@cloud ~]# systemctl status mysqld
● mysqld.service - MySQL 8.0 database server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2024-04-25 15:38:46 MST; 32s ago
  Process: 2317 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
  Process: 2316 ExecStart=/usr/libexec/mysqld --basedir=/usr (code=exited, status=203/EXEC)
  Process: 2280 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS)
  Process: 2256 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 2316 (code=exited, status=203/EXEC)

Apr 25 15:38:46 cloud.opticip.com systemd[1]: mysqld.service: Service RestartSec=100ms expired, scheduling restart.
Apr 25 15:38:46 cloud.opticip.com systemd[1]: mysqld.service: Scheduled restart job, restart counter is at 5.
Apr 25 15:38:46 cloud.opticip.com systemd[1]: Stopped MySQL 8.0 database server.
Apr 25 15:38:46 cloud.opticip.com systemd[1]: mysqld.service: Start request repeated too quickly.
Apr 25 15:38:46 cloud.opticip.com systemd[1]: mysqld.service: Failed with result 'exit-code'.
Apr 25 15:38:46 cloud.opticip.com systemd[1]: Failed to start MySQL 8.0 database server.

journalctl -xe

Apr 25 15:40:27 cloud.opticip.com systemd[2801]: mysqld.service: Failed to execute command: Operation not permitted
Apr 25 15:40:27 cloud.opticip.com systemd[2801]: mysqld.service: Failed at step EXEC spawning /usr/libexec/mysqld: Operation not permitted
-- Subject: Process /usr/libexec/mysqld could not be executed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The process /usr/libexec/mysqld could not be executed and failed.
--
-- The error number returned by this process is 1.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: mysqld.service: Main process exited, code=exited, status=203/EXEC
Apr 25 15:40:27 cloud.opticip.com systemd[1]: mysqld.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit mysqld.service has entered the 'failed' state with result 'exit-code'.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: Failed to start MySQL 8.0 database server.
-- Subject: Unit mysqld.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit mysqld.service has failed.
--
-- The result is failed.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: mysqld.service: Service RestartSec=100ms expired, scheduling restart.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: mysqld.service: Scheduled restart job, restart counter is at 5.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Automatic restarting of the unit mysqld.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: Stopped MySQL 8.0 database server.
-- Subject: Unit mysqld.service has finished shutting down
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit mysqld.service has finished shutting down.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: mysqld.service: Start request repeated too quickly.
Apr 25 15:40:27 cloud.opticip.com systemd[1]: mysqld.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--

Permission of the mysqld
[root@cloud ~]# ls -la /usr/sbin/mysqld
lrwxrwxrwx 1 root root 17 Sep 1 2021 /usr/sbin/mysqld -> ../libexec/mysqld

[root@cloud libexec]# stat mysqld
  File: mysqld
  Size: 67443384        Blocks: 57865      IO Block: 131072 regular file
Device: 2ch/44d Inode: 2978        Links: 1
Access: (0751/-rwxr-x--x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-09-01 12:45:10.000000000 -0700
Modify: 2021-09-01 12:45:10.000000000 -0700
Change: 2024-04-25 15:33:07.705573522 -0700
 Birth: 2024-04-25 14:53:27.116253587 -0700

SELINUX: disabled

[root@cloud selinux]# sestatus
SELinux status:                 disabled

Container Config Info:

arch: amd64
cores: 8
cpuunits: 9000
features: fuse=1,mount=nfs;cifs,nesting=1
hostname: cloud.opticip.com
memory: 24576
mp0: rpool2:subvol-100-disk-0,mp=/storage1,backup=1,size=9T
nameserver: 4.2.2.2 8.8.4.4 4.2.2.3
net0: name=eth0,bridge=vmbr0,firewall=1,gw=74.221.218.65,hwaddr=D6:E0:96:F2:5B:4A,ip=74.221.218.66/26,tag=410,type=veth
onboot: 1
ostype: centos
parent: mysqlfix
rootfs: local-zfs:subvol-100-disk-0,size=160G
swap: 4096

Any assistance is greatly appreciated. Little bit of a loss on how to fix this, as I'm not sure if the issue is from inside the container, or on the host machine.

 

[slyuen]

I have literally upgraded proxmox to 8.2.4 from 7 and have encountered exactly the same problem you had (on both CentOS 8 and Rocky Linux 9). the LXC container I am using is privileged, with all the nested features enabled etc but still the same :-(

Although I haven't found the cause, I have found a quick/temporary "workaround" to get mysql server going (again, my lxc container is privileged, not tested on unprivileged mode yet).

as root:
cd /usr/libexec
cp mysqld mysqld2

edit /usr/lib/systemd/system/mysqld.service
Find:
ExecStart=/usr/libexec/mysqld --basedir=/usr

Replace with:
ExecStart=/usr/libexec/mysqld2 --basedir=/usr

systemctl daemon-reload
systemctl start mysqld (it should start this time)

Repeat the workaround after mysql upgrades......

This "workaround" can/should be undone if/when the official fix for the root cause gets released.

 

[slyuen]

Found a better workaround/mitigation: Use MySQL Community Server's repo to install MySQL Server instead - https://dev.mysql.com/downloads/repo/yum. It seems to just work out of the box then.

 

[garrettmills]

I'm running into this issue after upgrading as well. I tried updating the mysql package to the latest version from the main repos to no avail. I also ran into the issue w/ a Clickhouse server I have in an LXC container (it couldn't calculate its available disk space because of a permission issue).

Not sure how the permissions got so mangled during the upgrade, but hopefully we'll get some more insight. Thanks @slyuen for the workaround.

 

[garrettmills]

UPDATE: I was able to work around the mysqld problem by:

cd /usr/libexec
cp mysql mysql2
mv mysql2 mysql

 

[litdelta]

UPDATE: I was able to work around the mysqld problem by:

cd /usr/libexec
cp mysql mysql2
mv mysql2 mysql

I don't know why, any suggestion works for me :/

 

[litdelta]

I don't know why, any suggestion works for me :/

L8