[SOLVED] Proxmox 7 to 8 (specifically ifupdown2) breaks ifup and ifdown commands (Python)

Power2All

Active Member
Sep 13, 2017
19
0
41
42
www.power2all.com
Hi guys,

I have 2 servers running Proxmox.
Sadly, one of my servers after upgrading 7 to 8 didn't came back online anymore.
After troubleshooting I found very quickly it was ifupdown2 (3.2) after patching was breaking the whole network interface controls through ifup and ifdown.
When I downgraded ifupdown2 through the cache of DEB files from 3.2 to 3.1, the commands were working again.
I could repeat the same issue if I force upgrade the package again, so something is up with that.

For now I've put a "hold" on the package, everything else patched fine/
Did anybody else got confronted with this issue that their ifupdown2 package was broken after patching ?
The exact error it makes, is that the "systemd" python module could not be found, even though it's installed (python3-systemd).

Cheers !
 
Hi, this is weird. Maybe we can find out what happened. Could you please post the following?
  • the exact ifupdown2 version that broke networking, should look something like 3.2.N-N+XXXX. You can probably find it in /var/log/apt/history.log entry corresponding to the package downgrade -- you can also post the entry here.
  • the exact ifupdown2 version you have installed now (e.g. via dpkg -l ifupdown2)
  • the output of python3 -c "import systemd; print(systemd.__path__)"
  • if possible, the ifupdown2 debug log from when the network broke. Every time it tries to bring up networking, ifupdown2 creates a directory in /var/log/ifupdown2 with the current date and time. Could you try to find the directory corresponding to the time when the network broke? In this directory, there is a file ifupdown2.debug.log, it would be great if you could attach it. It should contain the systemd error you mentioned.
 
Hi, this is weird. Maybe we can find out what happened. Could you please post the following?
  • the exact ifupdown2 version that broke networking, should look something like 3.2.N-N+XXXX. You can probably find it in /var/log/apt/history.log entry corresponding to the package downgrade -- you can also post the entry here.
  • the exact ifupdown2 version you have installed now (e.g. via dpkg -l ifupdown2)
  • the output of python3 -c "import systemd; print(systemd.__path__)"
  • if possible, the ifupdown2 debug log from when the network broke. Every time it tries to bring up networking, ifupdown2 creates a directory in /var/log/ifupdown2 with the current date and time. Could you try to find the directory corresponding to the time when the network broke? In this directory, there is a file ifupdown2.debug.log, it would be great if you could attach it. It should contain the systemd error you mentioned.
Upgraded ifupdown2: ifupdown2_3.2.0-1+pmx3_all.deb
Downgraded ifupdown2: ifupdown2_3.1.0-1+pmx4_all.deb

Installed now: 3.1.0-1+pmx4

Python3 code output Downgraded (3.1):
Code:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'systemd'

Python3 code output Upgraded (3.2):
Code:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'systemd'


ifup command with Upgraded (3.2) error:
Code:
Traceback (most recent call last):
  File "/usr/sbin/ifup", line 30, in <module>
    from ifupdown2.lib.log import LogManager, root_logger
ModuleNotFoundError: No module named 'ifupdown2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/ifup", line 33, in <module>
    from lib.log import LogManager, root_logger
  File "/usr/share/ifupdown2/lib/log.py", line 32, in <module>
    from systemd.journal import JournalHandler
ModuleNotFoundError: No module named 'systemd'

ifup command with Downgraded (3.1) error:
Code:
error: '-a' option or interface list are required

^- Which means it works, no Python error.


Hope this gives you some insight.

/var/log/ifupdown2 doesn't exist, so probably logged in syslog ?
 
Thanks!
Python3 code output Downgraded (3.1):
Code:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'systemd'

Python3 code output Upgraded (3.2):
Code:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'systemd'
This does not seem right -- there seems to be something wrong with your python3-systemd installation, causing ifupdown2 3.2 to fail when importing the systemd package. ifupdown2 depends on python3-systemd from ifupdown2 version 3.2 on [1], which is why ifupdown2 3.1 still works fine.

In order to find out what's wrong with python3-systemd, could you please post the output of the following commands?
Code:
dpkg -l python3-systemd
dpkg -L python3-systemd
python3 -c 'import sys; print(sys.path)'

[1] https://github.com/CumulusNetworks/...27768d7dc08c9354406ae875/debian/changelog#L13
 
Thanks!

This does not seem right -- there seems to be something wrong with your python3-systemd installation, causing ifupdown2 3.2 to fail when importing the systemd package. ifupdown2 depends on python3-systemd from ifupdown2 version 3.2 on [1], which is why ifupdown2 3.1 still works fine.

In order to find out what's wrong with python3-systemd, could you please post the output of the following commands?
Code:
dpkg -l python3-systemd
dpkg -L python3-systemd
python3 -c 'import sys; print(sys.path)'

[1] https://github.com/CumulusNetworks/...27768d7dc08c9354406ae875/debian/changelog#L13

dpkg -l python3-systemd
Code:
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version      Architecture Description
+++-===============-============-============-=================================
ii  python3-systemd 235-1+b2     amd64        Python 3 bindings for systemd

dpkg -L python3-systemd
Code:
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/systemd
/usr/lib/python3/dist-packages/systemd/__init__.py
/usr/lib/python3/dist-packages/systemd/_daemon.cpython-311-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/systemd/_journal.cpython-311-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/systemd/_reader.cpython-311-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/systemd/daemon.py
/usr/lib/python3/dist-packages/systemd/id128.cpython-311-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/systemd/journal.py
/usr/lib/python3/dist-packages/systemd/login.cpython-311-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/systemd/test
/usr/lib/python3/dist-packages/systemd/test/test_daemon.py
/usr/lib/python3/dist-packages/systemd/test/test_id128.py
/usr/lib/python3/dist-packages/systemd/test/test_journal.py
/usr/lib/python3/dist-packages/systemd/test/test_login.py
/usr/lib/python3/dist-packages/systemd_python-235.egg-info
/usr/lib/python3/dist-packages/systemd_python-235.egg-info/PKG-INFO
/usr/lib/python3/dist-packages/systemd_python-235.egg-info/dependency_links.txt
/usr/lib/python3/dist-packages/systemd_python-235.egg-info/top_level.txt
/usr/share
/usr/share/doc
/usr/share/doc/python3-systemd
/usr/share/doc/python3-systemd/changelog.Debian.amd64.gz
/usr/share/doc/python3-systemd/changelog.Debian.gz
/usr/share/doc/python3-systemd/copyright

python3 -c 'import sys; print(sys.path)'
Code:
['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']
 
Thanks!
python3 -c 'import sys; print(sys.path)'
Code:
['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']
This looks like your python3 binary points to a custom (non-Debian) installation of Python 3.7 in /usr/local/, which does not have /usr/lib/python3/dist-packages/ in its module search path, and hence fails to import the systemd module from there. Is it possible that you manually installed Python 3.7 at some point? If yes and if it's not needed, I'd recommend to uninstall it, and use the Debian-provided python3.11 package instead.
 
Thanks!

This looks like your python3 binary points to a custom (non-Debian) installation of Python 3.7 in /usr/local/, which does not have /usr/lib/python3/dist-packages/ in its module search path, and hence fails to import the systemd module from there. Is it possible that you manually installed Python 3.7 at some point? If yes and if it's not needed, I'd recommend to uninstall it, and use the Debian-provided python3.11 package instead.
Right, I'll give that a try, thanks.
How can I uninstall old Python version, and install latest Python, without apt removing PXE components ?

Code:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  gnome-keyring gnome-keyring-pkcs11 libgnutlsxx30 libjs-sencha-touch libnfsidmap1 libpam-gnome-keyring libprotobuf32 liburcu8 p11-kit p11-kit-modules
  proxmox-default-kernel proxmox-kernel-helper python-babel-localedata python3.11-dev
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  ifupdown libxmuu1 xauth
Suggested packages:
  rdnssd
The following packages will be REMOVED:
  ansible ansible-core apt-listchanges brz bzr ceph-common ceph-fuse certbot cifs-utils cockpit-packagekit cockpit-storaged criu debsecan dh-python duplicity duply fail2ban
  glusterfs-client glusterfs-common ifupdown2 iotop kernelcare libpve-guest-common-perl libpve-storage-perl lxc-pve mycli nfs-common nfs-kernel-server ntp ntpsec
  ntpsec-ntpdig proxmox-ve pve-container pve-ha-manager pve-manager pve-qemu-kvm pyflakes3 python3 python3-acme python3-anyio python3-apt python3-argcomplete python3-attr
  python3-babel python3-bcrypt python3-blinker python3-breezy python3-ceph-argparse python3-ceph-common python3-cephfs python3-certbot python3-certifi python3-cffi-backend
  python3-chardet python3-charset-normalizer python3-cli-helpers python3-click python3-colorama python3-configargparse python3-configobj python3-cryptography python3-dbus
  python3-debconf python3-debian python3-debianbts python3-deprecated python3-dev python3-distro python3-distro-info python3-distutils python3-dnspython python3-dulwich
  python3-fastbencode python3-fasteners python3-fastimport python3-flake8 python3-future python3-gi python3-github python3-gitlab python3-gpg python3-h11 python3-h2
  python3-hpack python3-httpcore python3-httplib2 python3-httpx python3-hyperframe python3-icu python3-idna python3-importlib-metadata python3-iniconfig python3-invoke
  python3-jaraco.classes python3-jeepney python3-jinja2 python3-jmespath python3-josepy python3-jwt python3-kerberos python3-keyring python3-launchpadlib
  python3-lazr.restfulclient python3-lazr.uri python3-ldb python3-lib2to3 python3-libcloud python3-libvirt python3-libxml2 python3-lockfile python3-markdown-it
  python3-markupsafe python3-mccabe python3-mdurl python3-merge3 python3-mock python3-monotonic python3-more-itertools python3-nacl python3-netaddr python3-ntlm-auth
  python3-ntp python3-oauthlib python3-openssl python3-packaging python3-paramiko python3-parsedatetime python3-patiencediff python3-pbr python3-pexpect python3-pip
  python3-pkg-resources python3-pluggy python3-prettytable python3-prompt-toolkit python3-protobuf python3-psutil python3-ptyprocess python3-py python3-pyaes
  python3-pycodestyle python3-pycryptodome python3-pycurl python3-pyflakes python3-pygments python3-pyinotify python3-pymysql python3-pyparsing python3-pyperclip
  python3-pysimplesoap python3-pytest python3-rados python3-rbd python3-reportbug python3-requests python3-requests-kerberos python3-requests-ntlm python3-requests-toolbelt
  python3-resolvelib python3-rfc3339 python3-rfc3986 python3-rich python3-secretstorage python3-selinux python3-setuptools python3-simplejson python3-six python3-sniffio
  python3-software-properties python3-sqlglot python3-sqlparse python3-systemd python3-tabulate python3-talloc python3-terminaltables python3-toml python3-tz
  python3-urllib3 python3-wadllib python3-wcwidth python3-wheel python3-winrm python3-wrapt python3-xmltodict python3-yaml python3-zipp python3-zope.component
  python3-zope.event python3-zope.hookable python3-zope.interface qemu-server reportbug sntp software-properties-common speedtest-cli spiceterm unattended-upgrades virtinst
  xfsprogs zfs-initramfs zfs-zed zfsutils-linux
The following NEW packages will be installed:
  ifupdown libxmuu1 xauth
0 upgraded, 3 newly installed, 202 to remove and 0 not upgraded.
Need to get 122 kB of archives.
After this operation, 750 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.
 
Last edited:
Hello Everyone,

I updated my server today and I have very similar issue - all network configuration has gone.

Attached you can find the output for the same commands.

Screenshot 2023-12-29 at 11.23.27.jpg

And here are the error messages while networking start.


Screenshot 2023-12-29 at 11.32.07.jpg

Please be so kind and help me out of it.

UPDATE:
I found the solution in a different topic: "Removing "/etc/network/if-up.d/vzifup-post" fixed it"
 
Last edited:
I had the same issue today after restarting my server. I did the apt update yesterday and only restarted it this afternoon.
Same issue with all the network configuration still intact, but none of the networks came up. Same error messages about the vmbr0 that could not be recognized.

Code:
Jan 14 16:43:03 proxmox networking[51848]:     cls.run_iface_graph(ifupdownobj, ifacename, ops, parent,
Jan 14 16:43:03 proxmox networking[51848]:   File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 302, in run_iface_graph
Jan 14 16:43:03 proxmox networking[51848]:     cls.run_iface_list(ifupdownobj, dlist, ops,
Jan 14 16:43:03 proxmox networking[51848]:   File "/usr/share/ifupdown2/ifupdown/scheduler.py", line 337, in run_iface_list
Jan 14 16:43:03 proxmox networking[51848]:     raise Exception('%s : (%s)' %(ifacename, str(e)))
Jan 14 16:43:03 proxmox networking[51848]: error: vmbr0 : eth0 : (eth0: up cmd '/etc/network/if-up.d/vzifup-post' failed: returned 127)
Jan 14 16:43:03 proxmox /usr/sbin/ifup[51848]: error: vmbr0 : eth0 : (eth0: up cmd '/etc/network/if-up.d/vzifup-post' failed: returned 127)
Jan 14 16:43:03 proxmox networking[51848]: warning: vmbr0: interface not recognized - please check interface configuration

I tried many solutions and had no luck, until I found this post:
https://forum.proxmox.com/threads/no-network-on-server-unless-i-ifdown-ifup-vmbr0.62733/post-292150
where a bug was introduced in ifupdown2. The person downgraded to ifupdown before the bug was fixed and it resolved his problem. So I tried to just
Code:
apt install ifupdown
and it fixed my issue as well. It does uninstall ifupdown2, through.
So, i guess not ideal.

My interfaces file just has this:
Code:
iface lo inet loopback

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.6/24
        gateway 192.168.10.1
        bridge-ports eth0
        bridge-stp off
        bridge-fd 0