Upgrade Warning: Prevent proxmox-ve Removal, Firmware Conflicts, and Broken Kernels — Full Explanation and Safe Script

F

fredu

New Member
Dec 2, 2023
14
2
3
Hi all,

Firstly, I apologise for the sterile reading. Its chatgpt sanitised as I am poo at typing stuff that is readable!

After running into (and nearly falling victim to) a risky upgrade path while moving from Proxmox VE 8.3 to 8.4, I wanted to share a fully worked-out solution and explanation to save others time, stress, and potentially a full reinstall.


Important:
This problem mainly occurs if you installed Proxmox VE on top of Debian (rather than via the official Proxmox ISO). If you did a clean install with the ISO, you are unlikely to encounter these issues.

For Debian installs, removing Debian kernels and pinning firmware-linux-free is critical to avoid conflicts with Proxmox’s kernel and firmware packages.

⚠️ The Problem: GUI or CLI Upgrades May Attempt to Remove​


When upgrading via:
  • The GUI “Upgrade” button, or
  • Running apt-get dist-upgrade or apt full-upgrade manually
…it may try to remove:
  • proxmox-ve (meta-package)
  • pve-firmware
  • proxmox-kernel-6.8
  • proxmox-default-kernel

Instead, it installs:
  • firmware-linux-free (Debian firmware package)
  • linux-image-6.1.x (Debian generic kernel)

This breaks the Proxmox kernel stack. If blindly accepted, it can remove critical functionality — I had to recover a node in the past due to exactly this scenario.



Why This Happens​

Thanks to excellent insights from community contributors like @dietmar and @aaron, we now know:
  • pve-firmware is specific to Proxmox
  • Debian’s firmware-linux-free conflicts with it
  • Installing the Debian firmware forces apt to remove the entire kernel chain
  • GUI and apt-get dist-upgrade do not handle this well
  • pve-apt-hook throws a warning — but many users don’t know how to interpret or resolve it


✅ The Definitive Fix (Consolidated)​

1. Prevent Debian firmware from being installed or upgraded​

echo -e "Package: firmware-linux-free\nPin: release *\nPin-Priority: -1" > /etc/apt/preferences.d/no-debian-firmware

2. Clean and refresh APT​

apt clean
rm -rf /var/lib/apt/lists/*
apt update

3. Reinstall the correct Proxmox meta-packages (important!)​

The proxmox-ve package is the core meta-package that ensures all essential Proxmox components and kernels remain installed and up-to-date.​

apt install --reinstall proxmox-ve proxmox-kernel-6.8 proxmox-kernel-helper
This step prevents apt from removing essential Proxmox packages.

4. Now do a safe upgrade​

apt full-upgrade

Note: Avoid using apt upgrade alone in this context, as it does not handle removals required for dependency changes. Use apt full-upgrade or apt-get dist-upgrade which handle this properly.



Safe, Repeatable Upgrade Scripts​

To help others avoid this completely, I created and tested two bash scripts (I have a three node Cluster):
  • ✅ proxmox-upgrade-node.sh — for regular cluster nodes
  • ✅ proxmox-upgrade-final.sh — for the last node in the cluster
    Includes quorum checks and reboot prompts

Features:
  • Logs to /root/proxmox-upgrade-*.log
  • Verifies proxmox-ve is present
  • Pins firmware packages
  • Prompts before reboots
  • Allows --noreboot flag for staging upgrades
Note on scripts:
The provided bash scripts are designed to help recover or safely upgrade existing clusters affected by these conflicts. They are not recommended to be run blindly on fresh installations. Always verify your system state, presence of proxmox-ve, and repository configuration before use.

For regular nodes: proxmox-upgrade-node.sh
Bash: 
#!/bin/bash

LOGFILE="/root/proxmox-upgrade-$(date +%Y%m%d-%H%M%S).log"
REBOOT=true

# Optional: --noreboot
while [[ "$#" -gt 0 ]]; do
  case $1 in
    --noreboot) REBOOT=false ;;
  esac
  shift
done

echo "Logging to $LOGFILE"
exec > >(tee -a "$LOGFILE") 2>&1
set -e

if ! dpkg -l | grep -q '^ii.*proxmox-ve'; then
  echo "ERROR: proxmox-ve is not installed. Aborting."
  exit 1
fi

echo "=== STEP 1: Clean APT ==="
apt clean
rm -rf /var/lib/apt/lists/*
apt update

echo "=== STEP 2: Block firmware-linux-free ==="
echo -e "Package: firmware-linux-free\nPin: release *\nPin-Priority: -1" > /etc/apt/preferences.d/no-debian-firmware
apt update

echo "=== STEP 3: Reinstall Proxmox meta-packages ==="
apt install --reinstall proxmox-ve proxmox-kernel-6.8 proxmox-kernel-helper

echo "=== STEP 4: Full upgrade ==="
apt full-upgrade

if $REBOOT; then
  echo "=== Rebooting now ==="
  reboot
else
  echo "=== SKIPPED REBOOT. Please reboot manually later. ==="
fi

For the final node: proxmox-upgrade-final.sh
Bash: 
#!/bin/bash

LOGFILE="/root/proxmox-upgrade-final-$(date +%Y%m%d-%H%M%S).log"
REBOOT=true

while [[ "$#" -gt 0 ]]; do
  case $1 in
    --noreboot) REBOOT=false ;;
  esac
  shift
done

echo "Logging to $LOGFILE"
exec > >(tee -a "$LOGFILE") 2>&1
set -e

if ! dpkg -l | grep -q '^ii.*proxmox-ve'; then
  echo "ERROR: proxmox-ve is not installed. Aborting."
  exit 1
fi

# Check for quorum
echo "Checking cluster quorum..."
QUORUM=$(pvecm status | awk '/Quorate:/ {print $2}')
if [[ "$QUORUM" != "Yes" && "$QUORUM" != "yes" ]]; then
  echo "⚠️ WARNING: This node reports no quorum."
  read -p "Are you ABSOLUTELY sure this is the last node? [y/N] " confirm
  if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then
    echo "Aborting upgrade for safety."
    exit 1
  fi
fi

echo "=== STEP 1: Clean APT ==="
apt clean
rm -rf /var/lib/apt/lists/*
apt update

echo "=== STEP 2: Block firmware-linux-free ==="
echo -e "Package: firmware-linux-free\nPin: release *\nPin-Priority: -1" > /etc/apt/preferences.d/no-debian-firmware
apt update

echo "=== STEP 3: Reinstall core packages ==="
apt install --reinstall proxmox-ve proxmox-kernel-6.8 proxmox-kernel-helper

echo "=== STEP 4: Full upgrade ==="
apt full-upgrade

if $REBOOT; then
  echo "=== STEP 5: Prompting for final node reboot ==="
  read -p "Reboot this final node now? [y/N] " confirm
  if [[ "$confirm" =~ ^[Yy]$ ]]; then
    reboot
  else
    echo "Reboot skipped. Please do it manually when ready."
  fi
else
  echo "=== SKIPPED REBOOT. Manual reboot still required. ==="
fi


Why I didn't Use the GUI?​

  • I did previously (a year ago) and ended up having to completely rebuild my entire cluster from scratch. At the time I suspected it was related to CEPH but it was actually due to kernel/firmware conflicts.
  • The Proxmox GUI upgrade currently uses apt-get dist-upgrade which behaves similarly to full-upgrade but does not handle package conflicts or warn about firmware package pinning.
  • It also does not provide clear feedback about pve-apt-hook errors, which are crucial for diagnosing these problems.
  • Until the GUI logic is improved, using CLI with explicit package pinning and apt full-upgrade is safer and smarter.


✅ TL;DR​

Pin firmware, reinstall proxmox-ve, always use apt full-upgrade.
Always ensure proxmox-ve is installed and never allowed to be removed — your system depends on it
 
Last edited:
  • Like
Reactions: meyergru
Appreciate the effort, but giving this kind of script has it's risks. You can receive similar apt errors "attempting to remove proxmox-ve package" for many different reasons. As mentioned, this will never happen if you use the correct PVE repositories and follow the installation instructions either when using the PVE iso or when installing over Debian. In this later case, every single step in the instructions must be followed carefully.

fredu said:
⚠️ Never use dist-upgrade in this context — full-upgrade is smarter with dependency retention.
Click to expand...
No, full-upgrade and dist-upgrade are essentially synonyms and do the same thing: upgrading packages, dependencies and removing packages if needed to fulfill the upgrade. The command that you should avoid is apt upgrade, which does not remove packages even if required by the upgrade and in some cases may fail to upgrade dependencies, resulting in a misbehaving system that may not be trivial to get back in order.
 
You must log in or register to reply here.
Top Bottom