Can't open web UI due to issues with openssl after upgrading server with backports?

[AveryFreeman]

Hi, I'm noticing other people in the forum appear to be installing from debian backports without issue, but I ran into a problem with one of my servers that appears to have been fixed by removing the backports line in my sources.list and downgrading the packages.

On proxmox server, was having issues with accessing the web UI with the hostname I had set in my LAN DNS. So I tried logging into the UI using the IP address and disabled the firewall temporarily in case that was causing issues. After that, I still couldn't access the web UI, so I ssh-ed into the server and checked all the usual places - hostname -i, /etc/hostname, /etc/hosts, all were consistent with LAN settings. so I went looking further:

journalctl | tail -n 50

was littered with repeated instances of this failure:

Aug 03 10:47:31 proxmox pveproxy[143652]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2025.

I noticed at least 5 forum questions referencing that failure, but the bulk of them had to do with disks being full - note: I used Google to search for "proxmox <exact failure line>" as I couldn't find any results using the search on the forums. However, I did remember that I'd added the backports repo and ran apt upgrade before the last time I'd rebooted the system, so I started to suspect that might have something to do with it.

I kept digging;

systemctl --failed

showed services failing that were all generally related to openssl:

  UNIT                        LOAD   ACTIVE SUB    DESCRIPTION
● postfix@-.service           loaded failed failed Postfix Mail Transport Agent (instance -)
● pve-firewall.service        loaded failed failed Proxmox VE firewall
● pve-guests.service          loaded failed failed PVE guests
● pvescheduler.service        loaded failed failed Proxmox VE scheduler
● pvestatd.service            loaded failed failed PVE Status Daemon
● systemd-hostnamed.service   loaded failed failed Hostname Service
● systemd-random-seed.service loaded failed failed Load/Save Random Seed
● systemd-update-utmp.service loaded failed failed Record System Boot/Shutdown in UTMP

I found apt downgrading instructions on this external site: https://askubuntu.com/questions/115...orce-downgrade-of-all-its-needed-dependencies

in /etc/apt/sources.list I commented out the backports line.

substituted ftp.us.debian.org with the debian source listed in my sources.list file

cat << EOF > /etc/apt/preferences.d/99tmp
Package: *
Pin: origin "ftp.us.debian.org"
Pin-Priority: 1001
EOF

Afterwards, ran:

apt-get --allow-downgrades --allow-change-held-packages upgrade

and got an output of:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  proxmox-kernel-6.5.13-5-pve-signed proxmox-kernel-6.8.4-3-pve-signed
Use 'apt autoremove' to remove them.
The following packages will be DOWNGRADED:
  bind9-dnsutils bind9-host bind9-libs ceph-fuse corosync grub-common grub-efi-amd64 grub-efi-amd64-bin
  grub-efi-amd64-signed grub-pc-bin grub2-common ifupdown2 krb5-locales libcfg7 libcmap4 libcorosync-common4
  libcpg4 libgssapi-krb5-2 libk5crypto3 libknet1 libkrb5-3 libkrb5support0 libnozzle1 libopeniscsiusr
  libquorum5 libtpms0 libvotequorum8 linux-libc-dev lxcfs open-iscsi openssh-client openssh-server
  openssh-sftp-server python3-pyvmomi smartmontools ssh swtpm swtpm-libs swtpm-tools
0 upgraded, 0 newly installed, 39 downgraded, 0 to remove and 0 not upgraded.
Need to get 18.7 MB of archives.
After this operation, 1,842 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

rebooted the server after the downgrade, and then checked for failed services after it booted:

systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

Was relieved to see there were no longer failed services. Logged into the web UI, looked like everything was good again.

However, I was met with a bit of a curve ball:

After I deleted the /etc/apt/preferences.d/99tmp and I ran apt update, there were actually 38 of them, so I upgraded understanding I could do the same process if necessary, but I was a little scared upgrading might make the issue return

ran the apt upgrade, rebooted, systemctl --failed still reports no errors, and I can still log into the web UI.

Has anyone else had issues like this by adding the backports repo?

One other thing, I noticed in one of the other threads a rep from proxmox saying only to use apt full-upgrade - I wasn't aware of that until now, and hadn't been using it, so perhaps that led to my issues, using apt upgrade after adding the backports repo?

Thanks

 

[BobhWasatch]

It isn't clear to me what you did here, but you normally do not use the entire backports repository but rather just add specific packages from there. For example I use it on a couple of my machines because the hardware requires a newer kernel, but the rest of the system is still bookworm.

I also don't use backports on my PVE machine at all because I try to keep the base OS as stock as possible. I don't actually run anything directly on the Proxmox OS, everything is containers or virtual machines.

Is there something specific you need from backports? You don't need to switch to backports to get the latest CVE patches. The security team back-ports those to the stable repository.

IOW if you are surprised by what got installed from backports you might be doing it wrong.

 

[AveryFreeman]

It isn't clear to me what you did here, but you normally do not use the entire backports repository but rather just add specific packages from there. For example I use it on a couple of my machines because the hardware requires a newer kernel, but the rest of the system is still bookworm.

I also don't use backports on my PVE machine at all because I try to keep the base OS as stock as possible. I don't actually run anything directly on the Proxmox OS, everything is containers or virtual machines.

Is there something specific you need from backports? You don't need to switch to backports to get the latest CVE patches. The security team back-ports those to the stable repository.

IOW if you are surprised by what got installed from backports you might be doing it wrong.

No, nothing specific, just an interest in have the latest packages in general. Most other distros I've run one can set them to get the latest packages, update them, and reboot them, and not have to worry about breakage, but since it's an amalgam of Proxmox and Debian, I'll have to be more mindful going forward.

I thought what I wrote was clear, but the synopsis is: I enabled backports, ran apt upgrade, and on next boot couldn't access the proxmox UI. The rest was my troubleshooting, and getting it working again, along with a reference or two. Rather than be content with solving my issues and leaving it at that, I wanted to share my experience with the forum since I could reasonably expect others might make the same error.

 

[BobhWasatch]

I thought what I wrote was clear, but the synopsis is: I enabled backports, ran apt upgrade, and on next boot couldn't access the proxmox UI.

Yeah, that's not normal. Enabling it doesn't cause any upgrades by itself, you have to specifically install packages from there.
https://backports.debian.org/Instructions/

All backports are deactivated by default so that the normal operation of a stable installation will not be compromised with potentially disruptive changes (such as incompatible configuration schema).Release files make this possible, and all backported packages arepinned to priority 100 via ButAutomaticUpgrades: yes.