[SOLVED] Problems upgrading Fedora to v26 / LXC

[ojaksch]

Upgrading from v24 to v25 works flawlessy, but neither from v25 to v26 nor from v24 to v26.
As decribed in https://fedoraproject.org/wiki/DNF_system_upgrade the last step is a "dnf system-upgrade reboot" in which it installs all new packages. I can see the progress by entering the container (pct enter VMID) and having a look to the logfiles.
I assume that when upgrade finishes it tries a final reboot; as mentioned earlier this works from v24 to v25 but when upgrading to v26 the Container seems to be dead atfer this very final step/reboot.
Also, when killing this lxc instance it won't start anymore saying

Aug 14 14:39:58 proxmox1 lxc-start[7976]: lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
Aug 14 14:39:58 proxmox1 lxc-start[7976]: lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
Aug 14 14:39:58 proxmox1 lxc-start[7976]: lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

When trying to run this in foreground the logs are giving me

lxc-start 20170814134330.190 WARN lxc_start - start.c:lxc_check_inherited:238 - Inherited fd: 3.
lxc-start 20170814134330.190 WARN lxc_start - start.c:lxc_check_inherited:238 - Inherited fd: 7.
lxc-start 20170814134330.190 INFO lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
lxc-start 20170814134330.190 DEBUG lxc_start - start.c:setup_signal_fd:273 - Set SIGCHLD handler with file descriptor: 6.
lxc-start 20170814134330.190 INFO lxc_start - start.c:lxc_init:475 - Container "ast-fed" is initialized.
lxc-start 20170814134330.191 DEBUG lxc_start - start.c:__lxc_start:1336 - Not dropping CAP_SYS_BOOT or watching utmp.
lxc-start 20170814134330.191 INFO lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for ast-fed
lxc-start 20170814134330.191 DEBUG lxc_cgfsng - cgroups/cgfsng.c:filter_and_set_cpus:475 - No isolated cpus detected.
lxc-start 20170814134330.191 DEBUG lxc_cgfsng - cgroups/cgfsng.c:handle_cpuset_hierarchy:645 - "cgroup.clone_children" was already set to "1".
lxc-start 20170814134330.195 INFO lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWNS.
lxc-start 20170814134330.195 INFO lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWPID.
lxc-start 20170814134330.195 INFO lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWUTS.
lxc-start 20170814134330.195 INFO lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWIPC.
lxc-start 20170814134330.195 INFO lxc_start - start.c:lxc_spawn:1155 - Cloned CLONE_NEWNET.
lxc-start 20170814134330.207 INFO lxc_conf - conf.c:mount_autodev:1130 - Mounting container /dev
lxc-start 20170814134330.207 INFO lxc_conf - conf.c:mount_autodev:1153 - Mounted tmpfs onto /dev
lxc-start 20170814134330.207 INFO lxc_conf - conf.c:mount_autodev:1171 - Mounted container /dev
lxc-start 20170814134330.207 INFO lxc_conf - conf.c:fill_autodev:1199 - Creating initial consoles under container /dev
lxc-start 20170814134330.207 INFO lxc_conf - conf.c:fill_autodev:1210 - Populating container /dev
lxc-start 20170814134330.207 INFO lxc_conf - conf.c:fill_autodev:1247 - Populated container /dev
lxc-start 20170814134330.207 INFO lxc_utils - utils.c:mount_proc_if_needed:1785 - I am 1, /proc/self points to '1117'
lxc-start 20170814134330.222 INFO lxc_utils - utils.c:mount_proc_if_needed:1812 - Mounted /proc in container for security transition
lxc-start 20170814134330.222 DEBUG lxc_conf - conf.c:setup_caps:2144 - capabilities have been setup
lxc-start 20170814134330.222 NOTICE lxc_conf - conf.c:lxc_setup:3982 - 'ast-fed' is setup.
lxc-start 20170814134330.223 INFO lxc_start - start.c:do_start:836 - Unshared CLONE_NEWCGROUP.
lxc-start 20170814134330.223 INFO lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:238 - changed apparmor profile to lxc-container-default-cgns
lxc-start 20170814134330.238 NOTICE lxc_start - start.c:start:1447 - Exec'ing "/sbin/init".
lxc-start 20170814134330.239 NOTICE lxc_start - start.cost_start:1458 - Started "/sbin/init" with pid "1117".
lxc-start 20170814134330.239 INFO console - console.c:lxc_console_mainloop_add:220 - no rootfs, no console.
lxc-start 20170814134330.239 WARN lxc_start - start.c:signal_handler:322 - Invalid pid for SIGCHLD. Received pid 1115, expected pid 1117.

...and thats all. What I'm missing here? Anybody else having trouble to upgrade Fedore to v26?

 

[pabernethy]

You may want to post the commands you entered, as well. I'm currently stuck after

dnf system-upgrade download --refresh --releasever=26

as I keep getting this:

# dnf system-upgrade reboot
Error: system is not ready for upgrade

 

[ojaksch]

Sure. They were

# dnf upgrade --refresh
# dnf install dnf-plugin-system-upgrade
# dnf system-upgrade download --refresh --releasever=26
# dnf system-upgrade reboot

Same for v24->v25 with the appropriate releasever which worked like a charm.

 

[pabernethy]

Figured it out. I had to install a kernel package, which is usually not required in a container, but the installer checks for it. With

# lxc-start -Fn 104

I could see the container booting to

[  OK  ] Reached target System Update.

After some time it rebooted and failed to start with the following messages

unsupported fedora release 'Fedora release 26 (Twenty Six)'
lxc-start: conf.c: run_buffer: 464 Script exited with status 25.
lxc-start: start.c: lxc_init: 450 Failed to run lxc.hook.pre-start for container "104".
lxc-start: start.c: __lxc_start: 1337 Failed to initialize container "104".
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

This is caused by our custom startup/setup script, which takes care of setting the network and tty config (which is normally done from inside the container). I looked over the changeset for Fedora 26 and didn't see any related changes. So I bumped the version in our setup script, after which the container came up fine.

--- a/usr/share/perl5/PVE/LXC/Setup/Fedora.pm    2017-08-16 13:45:24.662182945 +0200
+++ b/usr/share/perl5/PVE/LXC/Setup/Fedora.pm    2017-08-16 13:45:47.982065549 +0200
@@ -16,7 +16,7 @@
     my $version;

     if ($release =~ m/release\s+(\d+(?:\.\d+)?)(\.\d+)?/) {
-       if ($1 >= 22 && $1 < 26) {
+       if ($1 >= 22 && $1 < 27) {
            $version = $1;
        }
     }

Keep in mind that Fedora 26 is still unsupported and the fact that it worked for my, on a pretty much fresh container, doesn't necessarily mean it'll work for you, with an actually used container.

 

[ojaksch]

Ah, these are great news. Will try and report when system comes to end of work today.
Thank you for investigating into this!

 

[ojaksch]

I'm impatient - and it's always a good idea to create and keep backups, especially for experiments
Changed the mentioned line in /usr/share/perl5/PVE/LXC/Setup/Fedora.pm , restarted the "dead" (upgraded) container/Fedora v26 and voila, it starts and is working as if nothing other had happened.
WIll have a deeper look into this "unsupported" system, but I'm sure that everthing will be ok as there a only some programs/tasks in this container.

 

[ojaksch]

Can confirm that Fedora (upgraded from v24 to) v26 is working stable for at least 12 hours now. This a an asterisk/hylafax container and it works as before without any issues (yet).

 

[CRCinAU]

@fabian Any chance we can bump this for Fedora 33 seeing as its released now?

In fact, I wonder if there's any reason to have an upper limit at all? Given that unless you keep bumping this max number with every fedora release, it'll break all the time...

 

[fabian]

@fabian Any chance we can bump this for Fedora 33 seeing as its released now?

In fact, I wonder if there's any reason to have an upper limit at all? Given that unless you keep bumping this max number with every fedora release, it'll break all the time...

the reason is that it takes a bit of time after a distro release to make sure it works as expected. once that is done, we bump the upper limit. in this case, already before your post, but not yet released in the package: https://git.proxmox.com/?p=pve-container.git;a=commit;h=4160e22dafb7982bdf595def55bcfd8d45e3ee80

it's trivial enough to bump it manually if you want to test yourself whether it works, it's just changing a single number and restarting pveproxy/pvedaemon.

 

[CRCinAU]

Ah - I hadn't realised that, thanks...