Updating LXC Debian 10 to 11, issues

J

Joytidalis

Member
Proxmox Subscriber
Jun 27, 2019
16
1
23
47
I am having a lot of issues trying to upgrade my Debian 10 containers to Debian 11, after having upgraded the host OS from Proxmox 6 to 7.

Overall the upgrades take long, I see a lot of these type of messages combined with a lot of very long timeouts:

Code: 
Failed to reload daemon: Transport endpoint is not connected
Failed to get properties: Transport endpoint is not connected
Failed to reload daemon: Transport endpoint is not connected

or

Code: 
Installing new version of config file /etc/apt/apt.conf.d/01autoremove ...
Installing new version of config file /etc/kernel/postinst.d/apt-auto-removal ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for apt-daily-upgrade.timer: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
apt-daily-upgrade.timer is a disabled or a static unit not running, not starting it.
Failed to get unit file state for apt-daily.timer: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected

And at some point it sort of just stops, can force a reboot on the container, after which it's impossible to login.

I have tried restoring one of the containers from a backup, unprivileged, with nesting enabled. Same issue.
I see some people with similar issues, but not exactly the same either.
 
Hi,

I tried now upgrade the LXC Debian 10 to 11 without any issue. could you please post the LXC config pct config CTID?

What storage do you use in your PVE node?

Joysaab said:
And at some point it sort of just stops, can force a reboot on the container, after which it's impossible to login.
Click to expand...
Could you please generate the lxc log [0] using the below command and attach it
Bash: 
lxc-start -n CTID -F -l DEBUG -o /tmp/lxc-CTID.log
[0] https://pve.proxmox.com/wiki/Linux_...ers_with_tt_span_class_monospaced_pct_span_tt
 
I finished the Debian 11 upgrade now (took about 8 hours), and now the LXC will not start.

Debug log:
https://paste.gg/p/anonymous/140f14...3e/files/48616fbaefe249a08b821fb7cc123c6f/raw

Config:
Code: 
#Nextcloud
arch: amd64
features: keyctl=1,nesting=1
hostname: holt
memory: 4096
mp0: /storage/media,mp=/mnt/media
mp1: /storage/other,mp=/mnt/other
net0: name=eth0,bridge=vmbr0,hwaddr=B2:E3:0C:EA:DA:09,ip=dhcp,type=veth
onboot: 0
ostype: debian
rootfs: vms:subvol-100-disk-1,size=8G
swap: 4096
unprivileged: 1

Also note that the end of the Debian 11 upgrade didn't look good itself:
Code: 
Processing triggers for ca-certificates (20210119) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Processing triggers for libgdk-pixbuf-2.0-0:amd64 (2.42.2+dfsg-1) ...
Errors were encountered while processing:
 sudo
 openssh-server
 ssh

Error: Timeout was reached
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
Last edited:
I'm getting similar issue with Ubuntu 22.04 containers.

Created a new container from ubuntu-22.04-standard_22.04-1_amd64.tar.zst and ran

apt install xubuntu-desktop

Mostly it is fast and installing fine, but gets to things like this and hangs (Ctrl-c seems to help it proceed).

Setting up alsa-utils (1.2.6-1ubuntu1) ... Failed to reload daemon: Transport endpoint is not connected Failed to reload daemon: Transport endpoint is not connected Failed to get unit file state for alsa-restore.service: Transport endpoint is not connected Failed to retrieve unit state: Transport endpoint is not connected alsa-restore.service is a disabled or a static unit not running, not starting it.

Then everything fine for a while and then again...

Setting up libio-html-perl (1.004-2) ... Setting up libclucene-contribs1v5:amd64 (2.3.3.4+dfsg-1ubuntu5) ... Setting up gstreamer1.0-tools (1.20.1-1) ... Setting up libgarcon-1-0:amd64 (4.16.1-1) ... Setting up plocate (1.1.15-1ubuntu2) ... update-alternatives: using /usr/bin/plocate to provide /usr/bin/locate (locate) in auto mode Adding group `plocate' (GID 122) ... Done. Initializing plocate database; this may take some time... done Failed to preset unit: Transport endpoint is not connected /usr/bin/deb-systemd-helper: error: systemctl preset failed on plocate-updatedb.timer: No such file or directory

And SSH to the new container stops working, it just does this for several minutes before I kill it.

d7ewyetx@localhost:~$ ssh -v remotehost OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: Connecting to remotehost [192.168.1.100] port 22. debug1: Connection established. debug1: identity file /home/d7ewyetx/.ssh/id_rsa type 0 debug1: identity file /home/d7ewyetx/.ssh/id_rsa-cert type -1 debug1: identity file /home/d7ewyetx/.ssh/id_dsa type -1 debug1: identity file /home/d7ewyetx/.ssh/id_dsa-cert type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ecdsa type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ed25519 type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ed25519-cert type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ed25519_sk type -1 debug1: identity file /home/d7ewyetx/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/d7ewyetx/.ssh/id_xmss type -1 debug1: identity file /home/d7ewyetx/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
 
Last edited:
vesalius said:
Have you read through the threads below for ideas? I had issues back then on buster to bullseye LXC updates.

https://forum.proxmox.com/threads/l...-and-apparmor-errors.93064/page-2#post-423249
https://forum.proxmox.com/threads/l...-login-and-apparmor-errors.93064/#post-409030
Click to expand...
Nesting is enabled on my containers. For the rest I can't even get as far as to be able to change systemd files. During the Deb 10 --> 11 upgrade I get:

Installing new version of config file /etc/systemd/journald.conf ...
Installing new version of config file /etc/systemd/logind.conf ...
Installing new version of config file /etc/systemd/networkd.conf ...
Installing new version of config file /etc/systemd/resolved.conf ...
Installing new version of config file /etc/systemd/system.conf ...
Installing new version of config file /etc/systemd/user.conf ...
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service -> /lib/systemd/system/systemd-pstore.service.
Failed to get load state of systemd-networkd.socket: Connection timed out
Failed to try-restart systemd-networkd.service: Transport endpoint is not connected
See system logs and 'systemctl status systemd-networkd.service' for details.
Connection to champlain closed by remote host.
Connection to champlain closed.
Click to expand...

After that, the LXC is broken.
 
I ran into this while updating an LXC from Ubuntu 20.04 to 22.04. Almost two hours in there was a failure:
Code: 
Failed to get load state of systemd-networkd.socket: Connection timed out
Failed to try-restart systemd-networkd.service: Transport endpoint is not connected
and though the upgrade tried to continue it became glacially slow and timed out before finishing. I rolled back to a snapshot from just prior to the attempt.

On a second upgrade attempt when it reached the failing point, in a different tmux window I found a neverending series of ACPI errors in journalctl, as fast as it could log them:
Code: 
Sep 24 10:36:59 ubuntu systemd[1]: Condition check resulted in ACPI event daemon being skipped.
Sep 24 10:36:59 ubuntu systemd[1]: Condition check resulted in ACPI event daemon being skipped.
Sep 24 10:36:59 ubuntu systemd[1]: Condition check resulted in ACPI event daemon being skipped.
Sep 24 10:36:59 ubuntu systemd[1]: Condition check resulted in ACPI event daemon being skipped.
Sep 24 10:36:59 ubuntu systemd[1]: Condition check resulted in ACPI event daemon being skipped.

I did two things:
  1. sudo mv /etc/acpi/events /etc/acpi/events.bak
  2. sudo kill -HUP 1
I don't actually know if moving /etc/acpi/events is necessary, it was inspired by https://unix.stackexchange.com/ques...uck-at-100-inside-my-new-ubuntu-lxc-container

After sending SIGHUP to systemd it recovered, stopped logging ACPI errors, and the Ubuntu 22.04 update resumed its normal pace and succeeded. The LXC is now successfully running Ubuntu 22.04.
 
  • Like
Reactions: jahquubel, Datenfalke and mow
Hmm so this could be my issue on Debian 10 --> 11 as well?

Thanks for the tip anyway, I will try this. Just means having to monitor the upgrade process and do a timely kill before the entire thing comes to a grinding halt, right?
 
Joytidalis said:
Hmm so this could be my issue on Debian 10 --> 11 as well?
Click to expand...
Mine too. Took about five hours of false leads (systemd-journald-audit.socket, lxc.cap.drop = sys_rawio, which turned out to be rather cosmetical errors) until I found this thread.
Joytidalis said:
Thanks for the tip anyway, I will try this. Just means having to monitor the upgrade process and do a timely kill before the entire thing comes to a grinding halt, right?
Click to expand...
JFTR, you can get into the container with pct enter <vmid> even when it doesn't fully start (it stopped at
[ OK ] Reached target Basic System.
for me) and neither ssh nor console work.
And if even that fails, there's still pct mount <vmid>, which allows you to edit stuff below /var/lib/lxc/vmid/rootfs.
 
Joytidalis said:
Halleluja, after 10 months I am finally able to upgrade a Debian 10 LXC to Debian 11!
This simple thing seems to have done the trick. Oddly I didn't have to send any signal.

Thanks!
Click to expand...

At the end it seemed a really good idea to wait with the Debian upgrade so I could find this very helpful piece of information!
 
  • Like
Reactions: jahquubel
You must log in or register to reply here.
Top Bottom