Upgrade from 8.2.7 to 8.3 failed ... I think

idic

Member
Mar 13, 2020
9
1
23
52
I issued the upgrade command from the GUI on my single node non-clustered system. The upgrade completed but I did get a lot of odd messages starting with 'Setting up pve-cluster (8.0.10) ... Reload daemon failed: Connection timed out' . It wont let me reboot from command line or IPML console. Plus there are issues trying to run commands from the command line. I don't want to do a hard reboot on this system with the upgrade just happening and with my zfs pools.

Here is the upgrade. i've removed "normal" lines as I cant get this posted being this long.

Starting system upgrade: apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
proxmox-headers-6.5.13-5-pve proxmox-headers-6.8.12-1-pve
proxmox-headers-6.8.12-3-pve proxmox-headers-6.8.8-2-pve
proxmox-kernel-6.5.13-5-pve-signed proxmox-kernel-6.8.4-3-pve-signed
proxmox-kernel-6.8.8-2-pve-signed
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
libpve-network-api-perl proxmox-headers-6.8.12-4-pve
proxmox-kernel-6.8.12-4-pve-signed
The following packages will be upgraded:
ifupdown2 libjs-extjs libmpg123-0 libnss3 libpve-access-control
libpve-cluster-api-perl libpve-cluster-perl libpve-common-perl
libpve-guest-common-perl libpve-network-perl libpve-notify-perl libpve-rs-perl
libpve-storage-perl linux-libc-dev novnc-pve proxmox-archive-keyring
proxmox-backup-client proxmox-backup-file-restore proxmox-firewall
proxmox-headers-6.8 proxmox-kernel-6.8 proxmox-mail-forward proxmox-ve
proxmox-widget-toolkit pve-cluster pve-container pve-docs pve-firewall
pve-ha-manager pve-headers pve-i18n pve-manager pve-qemu-kvm qemu-server tzdata
35 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 184 MB of archives.
After this operation, 682 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.debian.org/debian-security bookworm-security/main amd64 libmpg123-0 amd64 1.31.2-1+deb12u1 [147 kB]
...
Get:38 http://download.proxmox.com/debian/pve bookworm/pve-no-subscription amd64 pve-headers all 8.3.0 [2868 B]
Fetched 184 MB in 46s (4011 kB/s)
Reading changelogs... Done
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 227368 files and directories currently installed.)
Preparing to unpack .../00-tzdata_2024b-0+deb12u1_all.deb ...
Unpacking tzdata (2024b-0+deb12u1) over (2024a-0+deb12u1) ...
...
Unpacking pve-headers (8.3.0) over (8.2.0) ...

Setting up proxmox-backup-file-restore (3.3.0-1) ...
Updating file-restore initramfs...
14166 blocks
Setting up proxmox-widget-toolkit (4.3.3) ...
Setting up libmpg123-0:amd64 (1.31.2-1+deb12u1) ...
Setting up proxmox-headers-6.8.12-4-pve (6.8.12-4) ...
Setting up proxmox-headers-6.8 (6.8.12-4) ...
Setting up pve-qemu-kvm (9.0.2-4) ...
Setting up proxmox-kernel-6.8.12-4-pve-signed (6.8.12-4) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/dkms 6.8.12-4-pve /boot/vmlinuz-6.8.12-4-pve
dkms: running auto installation service for kernel 6.8.12-4-pve.
Sign command: /lib/modules/6.8.12-4-pve/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module:
Cleaning build area...
'make' -j16 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=6.8.12-4-pve modules...............
Signing module /var/lib/dkms/nvidia/550.100/build/nvidia.ko
Signing module /var/lib/dkms/nvidia/550.100/build/nvidia-uvm.ko
Signing module /var/lib/dkms/nvidia/550.100/build/nvidia-modeset.ko
Signing module /var/lib/dkms/nvidia/550.100/build/nvidia-drm.ko
Signing module /var/lib/dkms/nvidia/550.100/build/nvidia-peermem.ko
Cleaning build area...

nvidia.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.8.12-4-pve/updates/dkms/

nvidia-uvm.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.8.12-4-pve/updates/dkms/

nvidia-modeset.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.8.12-4-pve/updates/dkms/

nvidia-drm.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.8.12-4-pve/updates/dkms/

nvidia-peermem.ko:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.8.12-4-pve/updates/dkms/
depmod...
dkms: autoinstall for kernel: 6.8.12-4-pve.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.8.12-4-pve /boot/vmlinuz-6.8.12-4-pve
update-initramfs: Generating /boot/initrd.img-6.8.12-4-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/0FEC-7CAF
Copying kernel and creating boot-entry for 6.5.13-6-pve
Copying kernel and creating boot-entry for 6.8.12-3-pve
Copying kernel and creating boot-entry for 6.8.12-4-pve
Removing old version 6.8.12-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/0FED-1308
Copying kernel and creating boot-entry for 6.5.13-6-pve
Copying kernel and creating boot-entry for 6.8.12-3-pve
Copying kernel and creating boot-entry for 6.8.12-4-pve
Removing old version 6.8.12-1-pve
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.8.12-4-pve /boot/vmlinuz-6.8.12-4-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.8.12-4-pve /boot/vmlinuz-6.8.12-4-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/0FEC-7CAF
Copying kernel and creating boot-entry for 6.5.13-6-pve
Copying kernel and creating boot-entry for 6.8.12-3-pve
Copying kernel and creating boot-entry for 6.8.12-4-pve
Copying and configuring kernels on /dev/disk/by-uuid/0FED-1308
Copying kernel and creating boot-entry for 6.5.13-6-pve
Copying kernel and creating boot-entry for 6.8.12-3-pve
Copying kernel and creating boot-entry for 6.8.12-4-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.8.12-4-pve /boot/vmlinuz-6.8.12-4-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.12-4-pve
Found initrd image: /boot/initrd.img-6.8.12-4-pve
Found linux image: /boot/vmlinuz-6.8.12-3-pve
Found initrd image: /boot/initrd.img-6.8.12-3-pve
Found linux image: /boot/vmlinuz-6.8.12-1-pve
Found initrd image: /boot/initrd.img-6.8.12-1-pve
Found linux image: /boot/vmlinuz-6.8.8-2-pve
Found initrd image: /boot/initrd.img-6.8.8-2-pve
Found linux image: /boot/vmlinuz-6.8.4-3-pve
Found initrd image: /boot/initrd.img-6.8.4-3-pve
Found linux image: /boot/vmlinuz-6.5.13-6-pve
Found initrd image: /boot/initrd.img-6.5.13-6-pve
Found linux image: /boot/vmlinuz-6.5.13-5-pve
Found initrd image: /boot/initrd.img-6.5.13-5-pve
Adding boot menu entry for UEFI Firmware Settings ...
done
Setting up proxmox-mail-forward (0.3.1) ...
Setting up linux-libc-dev:amd64 (6.1.119-1) ...
Setting up libpve-common-perl (8.2.9) ...
Setting up libnss3:amd64 (2:3.87.1-1+deb12u1) ...
Setting up proxmox-archive-keyring (3.1) ...
Setting up libjs-extjs (7.0.0-5) ...
Setting up ifupdown2 (3.2.0-1+pmx11) ...
Setting up pve-headers (8.3.0) ...
Setting up pve-docs (8.3.1) ...
Setting up tzdata (2024b-0+deb12u1) ...

Current default time zone: 'America/Edmonton'
Local time is now: Sun Dec 1 19:27:46 MST 2024.
Universal Time is now: Mon Dec 2 02:27:46 UTC 2024.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Setting up pve-cluster (8.0.10) ...
Reload daemon failed: Connection timed out
Failed to get unit file state for pve-cluster.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-cluster.service is a disabled or a static unit not running, not starting it.
Setting up novnc-pve (1.5.0-1) ...
Setting up proxmox-backup-client (3.3.0-1) ...
Setting up libpve-rs-perl (0.9.1) ...
Setting up pve-i18n (3.3.2) ...
Setting up libpve-cluster-perl (8.0.10) ...
Setting up proxmox-kernel-6.8 (6.8.12-4) ...
Setting up libpve-notify-perl (8.0.10) ...
Setting up libpve-network-perl (0.10.0) ...
Setting up libpve-access-control (8.2.0) ...
Setting up libpve-cluster-api-perl (8.0.10) ...
Setting up libpve-storage-perl (8.2.9) ...
Setting up pve-firewall (5.1.0) ...
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for pvefw-logger.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pvefw-logger.service is a disabled or a static unit not running, not starting it.
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for pve-firewall.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-firewall.service is a disabled or a static unit, not starting it.
Reload daemon failed: Transport endpoint is not connected
Failed to reload-or-try-restart pve-firewall.service: Transport endpoint is not connected
See system logs and 'systemctl status pve-firewall.service' for details.
Setting up proxmox-firewall (0.6.0) ...
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for proxmox-firewall.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
proxmox-firewall.service is a disabled or a static unit not running, not starting it.
Setting up libpve-guest-common-perl (5.1.6) ...
Setting up libpve-network-api-perl (0.10.0) ...
Setting up pve-container (5.2.2) ...
Setting up pve-ha-manager (4.0.6) ...
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for watchdog-mux.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
watchdog-mux.service is a disabled or a static unit, not starting it.
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for pve-ha-lrm.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-ha-lrm.service is a disabled or a static unit not running, not starting it.
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for pve-ha-crm.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-ha-crm.service is a disabled or a static unit not running, not starting it.
Setting up qemu-server (8.3.0) ...
Created symlink /etc/systemd/system/multi-user.target.wants/pve-query-machine-capabilities.service -> /lib/systemd/system/pve-query-machine-capabilities.service.
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for pve-query-machine-capabilities.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-query-machine-capabilities.service is a disabled or a static unit not running, not starting it.
Failed to get unit file state for qmeventd.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
qmeventd.service is a disabled or a static unit not running, not starting it.
Setting up pve-manager (8.3.0) ...
Reload daemon failed: Transport endpoint is not connected
Failed to get unit file state for pvedaemon.service: Transport endpoint is not connected
Failed to get unit file state for pveproxy.service: Transport endpoint is not connected
Failed to get unit file state for spiceproxy.service: Transport endpoint is not connected
Failed to get unit file state for pvestatd.service: Transport endpoint is not connected
Failed to get unit file state for pvebanner.service: Transport endpoint is not connected
Failed to get unit file state for pvescheduler.service: Transport endpoint is not connected
Failed to get unit file state for pve-daily-update.timer: Transport endpoint is not connected
Setting up proxmox-ve (8.3.0) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for pve-ha-manager (4.0.6) ...
Failed to retrieve unit state: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
Removing subscription nag from UI...

Your System is up-to-date


Seems you installed a kernel update - Please consider rebooting
this node to activate the new kernel.

starting shell


=================================================
This server is running


=================================================
dircolors: no SHELL environment variable, and no shell type option given

#

Trying to get on to the shell there is long waits between hitting enter and something happening. Due to the timeouts, I changed the DNS entry in my GUI to my router from my internal Pihole container. That did seem to speed things up a little but I think that is just a red herring.

I saw some posts that sounded similar-ish about bash-completion and the timezone, so I have changed my timezone to UTC and uninstalled bash-completion. No changes that I can see with that.

Here are some command messages happening now

$ sudo reboot
Call to Reboot failed: Connection timed out

Broadcast message from root@ggprox01 on pts/1 (Sun 2024-12-01 21:16:06 MST):

The system will reboot now!

^C
ggprox01:/var/log# systemctl
Failed to list units: Transport endpoint is not connected

Any idea on how to move forward with this safely?
 
forced a reboot with this crazy magic

echo s > /proc/sysrq-trigger
echo u > /proc/sysrq-trigger
echo s > /proc/sysrq-trigger
echo b > /proc/sysrq-trigger


and now everything seems to be back to normal. But it's worrying that after a normal GUI upgrade, I've was left with a malfunctioning and non-rebooting system.

Any input from the Proxmox team/community as to what happened would be great.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!