lxc - opensuse15.6 - zypper, lsof - "Operation not permitted" error

[Bernd Senf]

Hi all

Maybe someone run's into same issue - here my workaround:

"zypper ps -sss" don't work as expected after upgrade 15.5 to 15.6

My system:
* proxmox 8.2 up-to-date
* LXC unprivileged
* LXC nesting=1
* upgrade opensuse 15.5 -> 15.6, and upgraded, minimal system (238 rpm packages)
Important news:
- systemd 249 -> 254
- glibc 2.31 -> 2.38
- SE-Linux disabled (default)

After upgrade some containers (LXC) based on openSUSE 15.5 to openSUSE 15.6 all works fine and all services running as expected. Seems perfect.
After some days later and newer updates "zypper ps -sss| xargs systemctl restart " always tried to restart same services again and again , eg systemd-logind,postfix
on this minimalistic environment.

(loop)
# zypper ps -sss| xargs -r systemctl restart

# zypper ps -s
postfix
systemd-logind

# zypper ps -s

The following running processes use deleted files:

PID | PPID | UID | User | Command | Service | Files
------+------+-----+---------+----------------+----------------+----------------------------------------------------------------------------
96 | 1 | 0 | root | systemd-logind | systemd-logind | /usr/lib64/systemd/libsystemd-shared-254.so (stat: Operation not permitted)
| | | | | | /lib64/libc.so.6 (stat: Operation not permitted)
| | | | | | /lib64/ld-linux-x86-64.so.2 (stat: Operation not permitted)
| | | | | | /usr/lib64/libblkid.so.1.1.0 (stat: Operation not permitted)
| | | | | | /usr/lib64/libcap.so.2.63 (stat: Operation not permitted)
| | | | | | /usr/lib64/libmount.so.1.1.0 (stat: Operation not permitted)
....
637 | 1 | 0 | root | master | postfix | /lib64/ld-linux-x86-64.so.2 (stat: Operation not permitted)
| | | | | | /lib64/libc.so.6 (stat: Operation not permitted)
| | | | | | /usr/lib64/libpostfix-util.so (stat: Operation not permitted)
| | | | | | /usr/lib64/libicuuc.so.suse65.1 (stat: Operation not permitted)
| | | | | | /usr/lib64/libicudata.so.suse65.1 (stat: Operation not permitted)
| | | | | | /lib64/libdl.so.2 (stat: Operation not permitted)
| | | | | | /usr/lib64/libk5crypto.so.3.1 (stat: Operation not permitted)
| | | | | | /usr/lib64/libpostfix-global.so (stat: Operation not permitted)
| | | | | | /usr/lib64/libcrypto.so.3.1.4 (stat: Operation not permitted)
| | | | | | /lib64/libgcc_s.so.1 (stat: Operation not permitted)

....

I find out, that package only "lsof" (changed during upgrade on openSUSE from 4.91 to 4.99, some small changes against libselinux ??) produced these output.

# id -a
uid=0(root) gid=0(root) groups=0(root)

# lsof -p $(pgrep master) ... > "Operation not permitted" ... !!????

# rpm -q --changelog lsof
* Sat Nov 11 2023 andreas.stieger@gmx.de
- lsof 4.99.0:
* Do not hard-code fd numbers in epoll test
* --with-selinux configure option.
* Improve performance by using closefrom()
* Introduce liblsof for programmatic access over spawning lsof
in a subprocess
- build with libtirpc
- switch to upstream tarball again as it dropped proprietary code
....

Workaround:
Backport LSOF from 4.99 (openSUSE15.6) to 4.91 (openSUSE15.5)

Btw:
VMs not affected / bare-metal systems not affected

Mfg/Regards

 

[Exzellius]

Hey there,

thank you for the analysis, running into the same issue in my Ansible playbooks.
Could you specify the steps to install the backported version of lsof?
I only get version 4.99 when I run zypper se -s lsof.

EDIT:
For reference my repos:

ansible:/ansible # zypper lr
Repository priorities are without effect. All enabled repositories share the same priority.

#  | Alias                       | Name                                                                                        | Enabled | GPG Check | Refresh
---+-----------------------------+---------------------------------------------------------------------------------------------+---------+-----------+--------
 1 | repo-backports-debug-update | Update repository with updates for openSUSE Leap debuginfo packages from openSUSE Backports | No      | ----      | ----
 2 | repo-backports-update       | Update repository of openSUSE Backports                                                     | Yes     | (r ) Yes  | Yes
 3 | repo-debug                  | Debug Repository                                                                            | No      | ----      | ----
 4 | repo-debug-non-oss          | Debug Repository (Non-OSS)                                                                  | No      | ----      | ----
 5 | repo-debug-update           | Update Repository (Debug)                                                                   | No      | ----      | ----
 6 | repo-debug-update-non-oss   | Update Repository (Debug, Non-OSS)                                                          | No      | ----      | ----
 7 | repo-non-oss                | Non-OSS Repository                                                                          | Yes     | (r ) Yes  | Yes
 8 | repo-oss                    | Main Repository                                                                             | Yes     | (r ) Yes  | Yes
 9 | repo-sle-debug-update       | Update repository with debuginfo for updates from SUSE Linux Enterprise 15                  | No      | ----      | ----
10 | repo-sle-update             | Update repository with updates from SUSE Linux Enterprise 15                                | Yes     | (r ) Yes  | Yes
11 | repo-source                 | Source Repository                                                                           | No      | ----      | ----
12 | repo-update                 | Main Update Repository                                                                      | Yes     | (r ) Yes  | Yes
13 | repo-update-non-oss         | Update Repository (Non-Oss)                                                                 | Yes     | (r ) Yes  | Yes
ansible:/ansible # zypper se -xs lsof
Loading repository data...
Reading installed packages...

S  | Name | Type    | Version            | Arch   | Repository
---+------+---------+--------------------+--------+----------------
i+ | lsof | package | 4.99.0-150600.1.15 | x86_64 | Main Repository
ansible:/ansible #

Best regards,
Ex

 

[Exzellius]

Hello,

for somebody to search the same question how to backport this ... here is the solution:

echo -e '[lsof-bug]\nname=lsof is bugged, that is why this repo is here\nenabled=1\nautorefresh=1\nbaseurl=http://download.opensuse.org/distribution/leap/15.5/repo/oss/' > /etc/zypp/repos.d/lsof-bug.repo
zypper ref
zypper install --oldpackage lsof-4.91-1.11.x86_64
zypper al lsof

Best regards,
Ex