Cloud-Init suddenly stopped working for CentOS/Fedora templates

hohl

Member
Jun 18, 2020
33
2
13
30
Hi there,

I installed a few minor updates on two (so far) Proxmox hypervisors and suddenly all RHEL-related templates used seemed to have stopped working. The templates get cloned, boot up and they learn username+password from cloud-init, but then they don't learn any of the provided static IPv4 and IPv6 addresses. The exact same templates still work as before on the nodes that haven't yet been updated. On the non-updated nodes, you would just see the IPs in those ASCII-tables. On the updated nodes you see the below error output getting written to `/var/log/cloud-init-output.log` and just localhost IPs in the table. Seems somehow as if cloudinit doesn't know anymore how to handle static IPv6 addresses? The Debian/Ubuntu templates on the updated nodes however still work without any issues.

Code:
Cloud-init v. 19.4 running 'init-local' at Sat, 10 Apr 2021 16:17:44 +0000. Up 11.89 seconds.
2021-04-10 16:17:44,742 - util.py[WARNING]: failed stage init-local
failed run of stage init-local
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 362, in main_init
    init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 711, in apply_network_config
    return self.distro.apply_network_config(netcfg, bring_up=bring_up)
  File "/usr/lib/python3.6/site-packages/cloudinit/distros/__init__.py", line 178, in apply_network_config
    dev_names = self._write_network_config(netconfig)
  File "/usr/lib/python3.6/site-packages/cloudinit/distros/rhel.py", line 65, in _write_network_config
    return self._supported_write_network_config(netconfig)
  File "/usr/lib/python3.6/site-packages/cloudinit/distros/__init__.py", line 93, in _supported_write_network_config
    renderer.render_network_config(network_config)
  File "/usr/lib/python3.6/site-packages/cloudinit/net/renderer.py", line 56, in render_network_config
    templates=templates, target=target)
  File "/usr/lib/python3.6/site-packages/cloudinit/net/sysconfig.py", line 731, in render_network_state
    templates=templates).items():
  File "/usr/lib/python3.6/site-packages/cloudinit/net/sysconfig.py", line 704, in _render_sysconfig
    cls._render_physical_interfaces(network_state, iface_contents)
  File "/usr/lib/python3.6/site-packages/cloudinit/net/sysconfig.py", line 549, in _render_physical_interfaces
    iface_cfg, iface_subnets, network_state.has_default_route
  File "/usr/lib/python3.6/site-packages/cloudinit/net/sysconfig.py", line 399, in _render_subnets
    iface_cfg.name))
ValueError: Unknown subnet type 'static6' found for interface 'eth0'
------------------------------------------------------------
Cloud-init v. 19.4 running 'init' at Sat, 10 Apr 2021 16:18:15 +0000. Up 43.73 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+-----------+-------+-------------------+
ci-info: | Device |  Up  |           Address           |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------------------------+-----------+-------+-------------------+
ci-info: |  eth0  | True | fe80::81c:8dff:fefc:335c/64 |     .     |  link | 0a:1c:8d:fc:33:5c |
ci-info: |   lo   | True |          127.0.0.1          | 255.0.0.0 |  host |         .         |
ci-info: |   lo   | True |           ::1/128           |     .     |  host |         .         |
ci-info: +--------+------+-----------------------------+-----------+-------+-------------------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++

Did anybody else see such an issue and maybe has a fix/workaround for it?

Thanks,
Michael

EDIT: Looking at the source of `cloudinit/net/sysconfig.py`, `static6` seems to be a legit config value for cloudinit. Still have no clue whats going on here, so if there is anybody with an idea; I would really appreciate it.
 
Last edited:
As I probably left out some important information: below is the log of which packages where installed/upgraded. Full DPKG log is attached too.

Code:
$ sudo grep -E " (install|upgrade) " /var/log/dpkg.log.1
2021-03-09 13:14:49 upgrade base-files:amd64 10.3+deb10u6 10.3+deb10u8
2021-03-09 13:14:49 upgrade libnss-systemd:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:14:49 upgrade libsystemd0:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:14:50 upgrade libpam-systemd:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:14:50 upgrade systemd:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:14:51 upgrade udev:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:14:51 upgrade libudev1:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:16:24 upgrade systemd-sysv:amd64 241-7~deb10u4 241-7~deb10u6
2021-03-09 13:16:24 upgrade libgnutls-openssl27:amd64 3.6.7-4+deb10u5 3.6.7-4+deb10u6
2021-03-09 13:16:24 upgrade libgnutlsxx28:amd64 3.6.7-4+deb10u5 3.6.7-4+deb10u6
2021-03-09 13:16:24 upgrade libp11-kit0:amd64 0.23.15-2 0.23.15-2+deb10u1
2021-03-09 13:16:24 upgrade libgnutls30:amd64 3.6.7-4+deb10u5 3.6.7-4+deb10u6
2021-03-09 13:16:24 upgrade libzstd1:amd64 1.3.8+dfsg-3 1.3.8+dfsg-3+deb10u2
2021-03-09 13:16:25 upgrade libapt-pkg5.0:amd64 1.8.2.1 1.8.2.2
2021-03-09 13:16:25 upgrade libapt-inst2.0:amd64 1.8.2.1 1.8.2.2
2021-03-09 13:16:25 upgrade apt:amd64 1.8.2.1 1.8.2.2
2021-03-09 13:16:28 upgrade apt-utils:amd64 1.8.2.1 1.8.2.2
2021-03-09 13:16:28 upgrade tzdata:all 2020d-0+deb10u1 2021a-0+deb10u1
2021-03-09 13:16:28 upgrade iproute2:amd64 4.20.0-2 4.20.0-2+deb10u1
2021-03-09 13:16:29 upgrade libssl1.1:amd64 1.1.1d-0+deb10u3 1.1.1d-0+deb10u5
2021-03-09 13:16:29 upgrade libxml2:amd64 2.9.4+dfsg1-7+b3 2.9.4+dfsg1-7+deb10u1
2021-03-09 13:16:29 upgrade dnsutils:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:29 upgrade bind9-host:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:29 upgrade libbind9-161:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:29 upgrade libisccfg163:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:29 upgrade libisccc161:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:29 upgrade libirs161:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:29 upgrade libdns1104:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:30 upgrade libisc1100:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:30 upgrade liblwres161:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:30 upgrade file:amd64 1:5.35-4+deb10u1 1:5.35-4+deb10u2
2021-03-09 13:16:30 upgrade libmagic1:amd64 1:5.35-4+deb10u1 1:5.35-4+deb10u2
2021-03-09 13:16:30 upgrade libmagic-mgc:amd64 1:5.35-4+deb10u1 1:5.35-4+deb10u2
2021-03-09 13:16:30 upgrade libldap-common:all 2.4.47+dfsg-3+deb10u4 2.4.47+dfsg-3+deb10u6
2021-03-09 13:16:30 upgrade libldap-2.4-2:amd64 2.4.47+dfsg-3+deb10u4 2.4.47+dfsg-3+deb10u6
2021-03-09 13:16:30 upgrade openssl:amd64 1.1.1d-0+deb10u3 1.1.1d-0+deb10u5
2021-03-09 13:16:31 upgrade ca-certificates:all 20200601~deb10u1 20200601~deb10u2
2021-03-09 13:16:31 upgrade libcorosync-common4:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:31 upgrade libcfg7:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:31 upgrade libcmap4:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:31 upgrade libcpg4:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:31 upgrade libknet1:amd64 1.16-pve1 1.20-pve1
2021-03-09 13:16:31 upgrade libquorum5:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:31 upgrade libvotequorum8:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:31 upgrade corosync:amd64 3.0.4-pve1 3.1.0-pve1
2021-03-09 13:16:32 upgrade libcairo2:amd64 1.16.0-4 1.16.0-4+deb10u1
2021-03-09 13:16:32 upgrade libcupsimage2:amd64 2.2.10-6+deb10u3 2.2.10-6+deb10u4
2021-03-09 13:16:32 upgrade libcups2:amd64 2.2.10-6+deb10u3 2.2.10-6+deb10u4
2021-03-09 13:16:32 upgrade libisc-export1100:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:32 upgrade libdns-export1104:amd64 1:9.11.5.P4+dfsg-5.1+deb10u2 1:9.11.5.P4+dfsg-5.1+deb10u3
2021-03-09 13:16:32 upgrade libefivar1:amd64 37-2 37-2+deb10u1
2021-03-09 13:16:32 upgrade libefiboot1:amd64 37-2 37-2+deb10u1
2021-03-09 13:16:33 upgrade libjpeg62-turbo:amd64 1:1.5.2-2+b1 1:1.5.2-2+deb10u1
2021-03-09 13:16:33 upgrade libpve-apiclient-perl:all 3.0-3 3.1-3
2021-03-09 13:16:33 upgrade proxmox-backup-client:amd64 1.0.5-1 1.0.8-1
2021-03-09 13:16:33 upgrade libpve-storage-perl:all 6.3-1 6.3-7
2021-03-09 13:16:33 upgrade libpve-guest-common-perl:all 3.1-3 3.1-5
2021-03-09 13:16:33 upgrade libproxmox-acme-perl:all 1.0.5 1.0.7
2021-03-09 13:16:33 upgrade libpve-common-perl:all 6.2-6 6.3-4
2021-03-09 13:16:34 upgrade pve-qemu-kvm:amd64 5.1.0-7 5.2.0-2
2021-03-09 13:16:38 upgrade libproxmox-backup-qemu0:amd64 1.0.2-1 1.0.3-1
2021-03-09 13:16:38 upgrade libpve-http-server-perl:all 3.0-6 3.1-1
2021-03-09 13:16:38 upgrade sqlite3:amd64 3.27.2-3 3.27.2-3+deb10u1
2021-03-09 13:16:39 upgrade libsqlite3-0:amd64 3.27.2-3 3.27.2-3+deb10u1
2021-03-09 13:16:39 upgrade libtsk13:amd64 4.6.5-1 4.6.5-1+deb10u1
2021-03-09 13:16:39 upgrade linux-libc-dev:amd64 4.19.152-1 4.19.171-2
2021-03-09 13:16:39 upgrade lxcfs:amd64 4.0.3-pve3 4.0.6-pve1
2021-03-09 13:16:39 upgrade lxc-pve:amd64 4.0.3-1 4.0.6-2
2021-03-09 13:16:40 upgrade proxmox-widget-toolkit:all 2.4-3 2.4-5
2021-03-09 13:16:40 upgrade pve-container:all 3.3-1 3.3-4
2021-03-09 13:16:40 upgrade pve-firmware:all 3.1-3 3.2-2
2021-03-09 13:16:50 install pve-kernel-5.4.101-1-pve:amd64 <none> 5.4.101-1
2021-03-09 13:16:59 upgrade pve-kernel-5.4:all 6.3-1 6.3-6
2021-03-09 13:16:59 upgrade pve-kernel-helper:all 6.3-1 6.3-6
2021-03-09 13:16:59 upgrade qemu-server:amd64 6.3-1 6.3-5
2021-03-09 13:17:00 upgrade zstd:amd64 1.3.8+dfsg-3 1.3.8+dfsg-3+deb10u2
2021-03-09 13:17:00 upgrade pve-manager:amd64 6.3-2 6.3-4
2021-03-09 13:17:02 upgrade python-apt-common:all 1.8.4.1 1.8.4.3
2021-03-09 13:17:02 upgrade python3-apt:amd64 1.8.4.1 1.8.4.3
2021-03-09 13:17:02 upgrade sleuthkit:amd64 4.6.5-1 4.6.5-1+deb10u1
2021-03-09 13:17:02 upgrade spl:all 0.8.5-pve1 2.0.3-pve2
2021-03-09 13:17:02 upgrade sudo:amd64 1.8.27-1+deb10u2 1.8.27-1+deb10u3
2021-03-09 13:17:02 upgrade tcpdump:amd64 4.9.3-1~deb10u1 4.9.3-1~deb10u2

Seems to be unrelated to my CentOS images, as it happens to the untampered cloud reference images of CentOS too. So either something is off with how I run Proxmox/Cloudinit (and how others run it) or something broke in those updated packages.
 

Attachments

  • dpkg.log
    36.5 KB · Views: 1
In case anybody stumbles over this post in search for a solution: I do believe I have found the source of this issue. The latest Proxmox version relies on a CloudInit keyword (`static6`) that is only supported by the most recent CloudInit release, but CentOS, RHEL, Fedora, AlmaLinux, Oracle Linux, openSuse and many other distributions do still rely on older CloudInit versions. That's why they stopped working after installing the Proxmox update.

More details in the Proxmox Bug Tracker: https://bugzilla.proxmox.com/show_bug.cgi?id=3384
 
Hello,

Im using the latest CentOS 7, which have cloud-init-19.4-7.el7.centos.5.x86_64 packages, and still not working.
As mentioned in the ticket, the 19.4-11 release does not yet support the "static6" keyword. You will need to upgrade to newer images that ship with CloudInit 21.4 by default or edit your existing images to include a newer version. In general, that shouldn't be an issue: most Linux distributions already got updates to CloudInit 21.4, just CentOS 7 will stay unsupported by default as it will not receive any major package updates anymore. No clue whether there is an easy way to get CloudInit 21.4 running on CentOS 7 though.
 
Last edited:
As mentioned in the ticket, the 19.4-11 release does not yet support the "static6" keyword. You will need to upgrade to newer images that ship with CloudInit 21.4 by default or edit your existing images to include a newer version. In general, that shouldn't be an issue: most Linux distributions already got updates to CloudInit 21.4, just CentOS 7 will stay unsupported by default as it will not receive any major package updates anymore. No clue whether there is an easy way to get CloudInit 21.4 running on CentOS 7 though.
What kind of newer images? The latest version of cloudinit in CentOS 7 is 19.4-7.el7. There is no newer version. And im not really thing they will ever add 21.4 cloudinit, because not this is how centos working.
 
What kind of newer images? The latest version of cloudinit in CentOS 7 is 19.4-7.el7. There is no newer version. And im not really thing they will ever add 21.4 cloudinit, because not this is how centos working.
Exactly, CentOS 7 only gets security updates and fixes for mission critical bugs since August 2020, so I wouldn't count on that either.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!