Upgrade from 7 to 8 failed due to custom kernel

Reliant8275

New Member
Sep 4, 2023
14
2
3
A while back I did my first PVE install on a mini PC with four Intel i226V Ethernet ports. They weren't supported by the current PVE kernel but somehow I managed to put in a custom kernel. Today I tried to update from PVE 7 to 8. The default kernel wouldn't update and then the system would hang on reboot. It works now after a manual choice on boot (5.19), but that's not a good choice for a headless server. Point being, I need to either fix this so I can sucessfully boot from the new kernel or pull my VM snapshots from my NAS after a complete reinstall.

Here's my output from pve7to8
Code:
root@pve1:~# pve7to8
= CHECKING VERSION INFORMATION FOR PVE PACKAGES =

Checking for package updates..
WARN: updates for the following packages are available:
  dpkg, fontconfig, telnet, inetutils-telnet, librados2, libfmt9, liblttng-ust1, libssl3, cifs-utils, libssh2-1, libssl3, tcpdump, libssl3, dmidecode, libcups2, libwrap0, libxtables12, libcurl4, libldap-2.5-0, libssl3, dmeventd, libgomp1, libpulse0, libx11-xcb1, libsmartcols1, zstd, udev, coreutils, libgdbm-compat4, libnetaddr-ip-perl, ceph-fuse, libfmt9, hdparm, libext2fs2, bind9-host, pciutils, libnftnl11, libgssapi-krb5-2, libnet-ssleay-perl, libssl3, libcurl3-gnutls, libldap-2.5-0, open-iscsi, libssl3, openssh-client, libssl3, iptables, python3-samba, libpython3.11, iputils-ping, fdutils, libglusterfs0, libssl3, libpve-rs-perl, libssl3, libauthen-pam-perl, libnl-3-200, libgfapi0, apt, corosync, libtemplate-perl, libtsan0, gcc-11-base, cpp, cpp-12, libnftables1, g++, g++-12, gcc-12, libfuse2, gcc, gcc-12, cryptsetup-bin, libacl1, gpg, perl, perl-modules-5.36, libperl5.36, libtinfo6, rrdcached, zfs-zed, libunwind8, bridge-utils, libldb2, libldap-2.5-0, libmagic-mgc, libpcre3, libpango-1.0-0, libunistring2, libtevent0, kbd, python3-cffi-backend, libffi8, proxmox-websocket-tunnel, libssl3, python3-minimal, chrony, libpam-systemd, samba-vfs-modules, liburing2, busybox, libleveldb1d, nmap-common, libnl-route-3-200, libqt5core5a, libicu72, zfs-initramfs, libselinux1, libarchive13, python3-lib2to3, liblzma5, sed, lsb-base, ssh, pve-qemu-kvm, liburing2, tar, proxmox-mail-forward, libssl3, libgfrpc0, libavahi-common-data, libcap2-bin, whiptail, base-passwd, libpve-cluster-api-perl, libsasl2-2, libcairo2, libgbm1, xxd, python3-ceph-common, libibverbs1, librbd1, libfmt9, liblttng-ust1, libssl3, pve-ha-manager, libpve-notify-perl, grub-pc-bin, libdb5.3, libbinutils, libfakeroot, libp11-kit0, libffi8, libapt-pkg6.0, procmail, lxcfs, swtpm-libs, libssl3, pve-lxc-syscalld, libposix-strptime-perl, apparmor, libproxmox-backup-qemu0, libssl3, swtpm-tools, libssl3, librgw2, libicu72, libldap-2.5-0, libssl3, libthrift-0.17.0, libfido2-1, libcbor0.8, libssl3, kmod, libssl3, libuutil3linux, libwayland-server0, libffi8, libopts25, libmagic1, liblua5.3-0, librrds-perl, faketime, libpve-storage-perl, ceph-common, libfmt9, libicu72, libldap-2.5-0, libssl3, libthrift-0.17.0, vim-common, liblvm2cmd2.03, libip4tc2, libsystemd0, libqt5network5, libssl3, libfreetype6, libmount1, nfs-common, libevent-core-2.1-7, libnfsidmap1, samba, libldap-2.5-0, libzpool5linux, librrd8, binutils-x86-64-linux-gnu, libgprofng0, file, bzip2, genisoimage, libpipeline1, libnss-systemd, debianutils, libpopt0, libcrypt-openssl-bignum-perl, libssl3, libpve-guest-common-perl, libslirp0, logrotate, libapparmor1, libwbclient0, vncterm, libnuma1, xfsprogs, libicu72, liburcu8, python3, python3.11, openssh-server, libssl3, libdbus-1-3, libqt5dbus5, gnupg, gpg-wks-server, libkrb5support0, libsmbclient, swtpm, libxml2, libicu72, wireguard-tools, pve-cluster, fuse, xz-utils, libzstd1, util-linux, util-linux-extra, libbsd0, less, libxpm4, libfontconfig1, libgpgme11, psmisc, libasound2, systemd, libsystemd-shared, libproxmox-rs-perl, samba-dsdb-modules, libudev1, gpg-agent, libcrypt-openssl-rsa-perl, libssl3, libgpg-error0, cstream, wget, nvme-cli, libnvme1, shared-mime-info, liblocale-gettext-perl, fdisk, libspice-server1, libssl3, libreadline8, proxmox-ve, proxmox-default-kernel, python3-pycurl, lxc-pve, libgnutlsxx30, gettext-base, libmaxminddb0, samba-common-bin, libldap-2.5-0, diffutils, strace, libfdisk1, gdisk, librdkafka1, libssl3, libassuan0, python3-talloc, libpython3.11, eject, usbutils, attr, lshw, lsof, libcap2, proxmox-backup-file-restore, libssl3, libfile-fcntllock-perl, libfuse3-3, python3-cephfs, lvm2, libc6, locales, dpkg-dev, libasan6, gcc-11-base, libpcap0.8, libpcre2-16-0, libpangocairo-1.0-0, libcrypt-openssl-random-perl, libssl3, libcephfs2, libfmt9, libkrb5-3, libssl3, libpam-modules, ifupdown2, python3-systemd, ethtool, pve-xtermjs, proxmox-termproxy, tdb-tools, python3-openssl, man-db, qemu-server, libcommon-sense-perl, gpgv, libtext-charwidth-perl, libpve-access-control, libuuid-perl, bash, traceroute, pve-container, libgcrypt20, isc-dhcp-client, grep, libcpg4, libfaketime, libstatgrab10, uuid-runtime, libipset13, libcap-ng0, vim-tiny, libgfchangelog0, libctf0, bind9-dnsutils, libasyncns0, libnss3, libfilesys-df-perl, rsyslog, libk5crypto3, libpangoft2-1.0-0, ncurses-base, gzip, keyutils, libtdb1, python3-apt, dmsetup, libsndfile1, libflac12, libmp3lame0, libmpg123-0, login, libtpms0, libssl3, libqb100, gnutls-bin, rpcbind, libcrypt-ssleay-perl, libssl3, libsqlite3-0, libevent-2.1-7, hostname, rsync, libssl3, pinentry-curses, python3-gpg, libjson-c5, libopeniscsiusr, gpgsm, glusterfs-common, libssl3, liburcu8, libhtml-parser-perl, libterm-readline-gnu-perl, python3-markupsafe, libxslt1.1, libunbound8, libradosstriper1, libfmt9, proxmox-backup-client, libssl3, libdevmapper-event1.02.1, libcrypt-dev, libcryptsetup12, libssl3, libasound2-data, findutils, python3-ldb, libpython3.11, libip6tc2, libpcre2-8-0, libcrypt1, libgstreamer1.0-0, libavahi-common3, binutils-common, libitm1, libpam-modules-bin, perl-openssl-defaults, libssl3, liblinux-inotify2-perl, init-system-helpers, usrmerge, grub-efi-amd64-bin, grub2-common, thin-provisioning-tools, libsemanage-common, proxmox-mini-journalreader, smartmontools, python3-rbd, python3-rgw, smbclient, python3-protobuf, libprotobuf32, uidmap, libsubid4, libpve-http-server-perl, procps, libproc2-0, python3-tdb, libdpkg-perl, python3-markdown, libpython3-stdlib, libpython3.11-stdlib, libc-dev-bin, mount, libglib2.0-0, libffi8, pve-manager, fonts-font-logos, libpve-notify-perl, python3-yaml, libpve-common-perl, perl-base, openssh-sftp-server, libpam0g, nano, btrfs-progs, grub-common, bind9-libs, libjemalloc2, libssl3, liboath0, libdrm2, samba-libs, libicu72, libldap-2.5-0, sudo, libu2f-server0, libssl3, libc-bin, ipset, libc-devtools, glusterfs-client, librabbitmq4, libssl3, librados2-perl, python3-distutils, libattr1, efibootmgr, libaudit1, libkmod2, libssl3, libclone-perl, ebtables, libyaml-libyaml-perl, bsd-mailx, nmap, libssl3, libc6-dev, rpcsvc-proto, parted, cpio, dirmngr, libldap-2.5-0, libzfs4linux, libssl3, socat, libssl3, libksba8, libharfbuzz0b, libtext-iconv-perl, libisns0, libssl3, criu, samba-common, cron, apt-utils, gnupg-utils, libblkid1, libgnutls30, curl, libubsan1, libslang2, liblsan0, libargon2-1, sysvinit-utils, logsave, libvirglrenderer1, libva-drm2, libva2, libvulkan1, libalgorithm-diff-xs-perl, libhttp-message-perl, pve-firewall, libjson-xs-perl, gnupg-l10n, libdw1, librdmacm1, gpg-wks-client, libpci3, libgd3, libavif15, libheif1, libtiff6, libwebp7, libxml-parser-perl, libncursesw6, sqlite3, python3-cryptography, libssl3, ncurses-bin, libbz2-1.0, libnozzle1, time, libx11-6, libgstreamer-plugins-base1.0-0, libsqlite3-mod-ceph, libfmt9, libanyevent-perl, python3-ceph-argparse, libelf1, groff-base, libmd0, ncurses-term, libnet-dbus-perl, dash, libtirpc3, libtalloc2, dbus, wireguard, libknet1, libssl3, libedit2, grub-pc, libpve-u2f-server-perl, bsdutils, iproute2, libbpf1, libavahi-client3, gpgconf, libdevmapper1.02.1, libgnutls-dane0, libxml-libxml-perl, libfstrm0, libss2, libtirpc-dev, libnewt0.52, spiceterm, libgdbm6, binutils, libuv1, fakeroot, libpsl5, bsdextrautils, libstdc++6, libbabeltrace1, libncurses6, zfsutils-linux, libssl3, postfix, libicu72, libssl3, libparted2, openssl, libssl3, e2fsprogs, passwd, libsemanage2, libpve-cluster-perl, python3-rados, libapt-pkg-perl, nftables, libnspr4, libgcc-s1

Checking proxmox-ve package version..
PASS: proxmox-ve package has version >= 7.4-1

Checking running kernel version..
WARN: a suitable kernel (pve-kernel-5.15) is intalled, but an unsuitable (5.19.17-2-pve) is booted, missing reboot?!

= CHECKING CLUSTER HEALTH/SETTINGS =

PASS: systemd unit 'pve-cluster.service' is in state 'active'
PASS: systemd unit 'corosync.service' is in state 'active'
PASS: Cluster Filesystem is quorate.

Analzying quorum settings and state..
INFO: configured votes - nodes: 3
INFO: configured votes - qdevice: 0
INFO: current expected votes: 3
INFO: current total votes: 3

Checking nodelist entries..
PASS: nodelist settings OK

Checking totem settings..
PASS: totem settings OK

INFO: run 'pvecm status' to get detailed cluster status..

= CHECKING HYPER-CONVERGED CEPH STATUS =

SKIP: no hyper-converged ceph setup detected!

= CHECKING CONFIGURED STORAGES =

storage 'synology' is not online
PASS: storage 'local' enabled and active.
PASS: storage 'local-lvm' enabled and active.
WARN: storage 'synology' enabled but not active!
INFO: Checking storage content type configuration..
PASS: no storage content problems found
WARN: activating 'synology' failed - storage 'synology' is not online

PASS: no storage re-uses a directory for multiple content types.

= MISCELLANEOUS CHECKS =

INFO: Checking common daemon services..
PASS: systemd unit 'pveproxy.service' is in state 'active'
PASS: systemd unit 'pvedaemon.service' is in state 'active'
PASS: systemd unit 'pvescheduler.service' is in state 'active'
PASS: systemd unit 'pvestatd.service' is in state 'active'
INFO: Checking for supported & active NTP service..
PASS: Detected active time synchronisation unit 'chrony.service'
INFO: Checking for running guests..
WARN: 3 running guest(s) detected - consider migrating or stopping them.
INFO: Checking if the local node's hostname 'pve1' is resolvable..
INFO: Checking if resolved IP is configured on local node..
PASS: Resolved node IP '192.168.1.20' configured and active on single interface.
INFO: Check node certificate's RSA key size
PASS: Certificate 'pve-root-ca.pem' passed Debian Busters (and newer) security level for TLS connections (4096 >= 2048)
PASS: Certificate 'pve-ssl.pem' passed Debian Busters (and newer) security level for TLS connections (2048 >= 2048)
INFO: Checking backup retention settings..
PASS: no backup retention problems found.
INFO: checking CIFS credential location..
PASS: no CIFS credentials at outdated location found.
INFO: Checking permission system changes..
INFO: Checking custom role IDs for clashes with new 'PVE' namespace..
PASS: no custom roles defined, so no clash with 'PVE' role ID namespace enforced in Proxmox VE 8
INFO: Checking if LXCFS is running with FUSE3 library, if already upgraded..
SKIP: not yet upgraded, no need to check the FUSE library version LXCFS uses
INFO: Checking node and guest description/note length..
PASS: All node config descriptions fit in the new limit of 64 KiB
PASS: All guest config descriptions fit in the new limit of 8 KiB
INFO: Checking container configs for deprecated lxc.cgroup entries
PASS: No legacy 'lxc.cgroup' keys found.
INFO: Checking if the suite for the Debian security repository is correct..
PASS: found no suite mismatch
INFO: Checking for existence of NVIDIA vGPU Manager..
PASS: No NVIDIA vGPU Service found.
INFO: Checking bootloader configuration...
SKIP: not yet upgraded, no need to check the presence of systemd-boot
INFO: Check for dkms modules...
/sbin/dkms: line 2497: echo: write error: Broken pipe
WARN: dkms modules found, this might cause issues during upgrade.
SKIP: NOTE: Expensive checks, like CT cgroupv2 compat, not performed without '--full' parameter

= SUMMARY =

TOTAL:    36
PASSED:   26
SKIPPED:  4
WARNINGS: 6
FAILURES: 0

ATTENTION: Please check the output for detailed information!

I wouldn't mind killing the cluster as I only have one other reliable system and the Synology NAS' VM is flaky and unreliable. I see there's a long list of packages to be upgraded. When I run apt upgrade it tries to upgrade but fails on the kernel. Here's the snippet:

Code:
dkms: autoinstall for kernel: 6.5.13-1-pve failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.5.13-1-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-6.5.13-1-pve-signed (--configure):
 installed proxmox-kernel-6.5.13-1-pve-signed package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.5:
 proxmox-kernel-6.5 depends on proxmox-kernel-6.5.13-1-pve-signed | proxmox-kernel-6.5.13-1-pve; however:
  Package proxmox-kernel-6.5.13-1-pve-signed is not configured yet.
  Package proxmox-kernel-6.5.13-1-pve is not installed.
  Package proxmox-kernel-6.5.13-1-pve-signed which provides proxmox-kernel-6.5.13-1-pve is not configured yet.

I tried to blindly install the above packages but either one errored out with
Code:
Error! The /var/lib/dkms/wireguard/1.0.20210219/6.5.13-1-pve/x86_64/dkms.conf for module wireguard includes a BUILD_EXCLUSIVE directive which does not match this kernel/arch/config.
This indicates that it should not be built.
Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.
dkms: autoinstall for kernel: 6.5.13-1-pve failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.5.13-1-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-6.5.13-1-pve-signed (--configure):
 installed proxmox-kernel-6.5.13-1-pve-signed package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.5:
 proxmox-kernel-6.5 depends on proxmox-kernel-6.5.13-1-pve-signed | proxmox-kernel-6.5.13-1-pve; however:
  Package proxmox-kernel-6.5.13-1-pve-signed is not configured yet.
  Package proxmox-kernel-6.5.13-1-pve is not installed.
  Package proxmox-kernel-6.5.13-1-pve-signed which provides proxmox-kernel-6.5.13-1-pve is not configured yet.

dpkg: error processing package proxmox-kernel-6.5 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 proxmox-kernel-6.5.13-1-pve-signed
 proxmox-kernel-6.5
E: Sub-process /usr/bin/dpkg returned an error code (1)

What did I do wrong? Would it be easier to do a fresh install and reinstall my 3 VMs from the snapshots on the NFS share/Synology NAS?
 
  • Like
Reactions: Kingneutron
A while back I did my first PVE install on a mini PC with four Intel i226V Ethernet ports. They weren't supported by the current PVE kernel but somehow I managed to put in a custom kernel. Today I tried to update from PVE 7 to 8. The default kernel wouldn't update and then the system would hang on reboot. It works now after a manual choice on boot (5.19), but that's not a good choice for a headless server. Point being, I need to either fix this so I can sucessfully boot from the new kernel or pull my VM snapshots from my NAS after a complete reinstall.

Here's my output from pve7to8
Code:
root@pve1:~# pve7to8
= CHECKING VERSION INFORMATION FOR PVE PACKAGES =

Checking for package updates..
WARN: updates for the following packages are available:
  dpkg, fontconfig, telnet, inetutils-telnet, librados2, libfmt9, liblttng-ust1, libssl3, cifs-utils, libssh2-1, libssl3, tcpdump, libssl3, dmidecode, libcups2, libwrap0, libxtables12, libcurl4, libldap-2.5-0, libssl3, dmeventd, libgomp1, libpulse0, libx11-xcb1, libsmartcols1, zstd, udev, coreutils, libgdbm-compat4, libnetaddr-ip-perl, ceph-fuse, libfmt9, hdparm, libext2fs2, bind9-host, pciutils, libnftnl11, libgssapi-krb5-2, libnet-ssleay-perl, libssl3, libcurl3-gnutls, libldap-2.5-0, open-iscsi, libssl3, openssh-client, libssl3, iptables, python3-samba, libpython3.11, iputils-ping, fdutils, libglusterfs0, libssl3, libpve-rs-perl, libssl3, libauthen-pam-perl, libnl-3-200, libgfapi0, apt, corosync, libtemplate-perl, libtsan0, gcc-11-base, cpp, cpp-12, libnftables1, g++, g++-12, gcc-12, libfuse2, gcc, gcc-12, cryptsetup-bin, libacl1, gpg, perl, perl-modules-5.36, libperl5.36, libtinfo6, rrdcached, zfs-zed, libunwind8, bridge-utils, libldb2, libldap-2.5-0, libmagic-mgc, libpcre3, libpango-1.0-0, libunistring2, libtevent0, kbd, python3-cffi-backend, libffi8, proxmox-websocket-tunnel, libssl3, python3-minimal, chrony, libpam-systemd, samba-vfs-modules, liburing2, busybox, libleveldb1d, nmap-common, libnl-route-3-200, libqt5core5a, libicu72, zfs-initramfs, libselinux1, libarchive13, python3-lib2to3, liblzma5, sed, lsb-base, ssh, pve-qemu-kvm, liburing2, tar, proxmox-mail-forward, libssl3, libgfrpc0, libavahi-common-data, libcap2-bin, whiptail, base-passwd, libpve-cluster-api-perl, libsasl2-2, libcairo2, libgbm1, xxd, python3-ceph-common, libibverbs1, librbd1, libfmt9, liblttng-ust1, libssl3, pve-ha-manager, libpve-notify-perl, grub-pc-bin, libdb5.3, libbinutils, libfakeroot, libp11-kit0, libffi8, libapt-pkg6.0, procmail, lxcfs, swtpm-libs, libssl3, pve-lxc-syscalld, libposix-strptime-perl, apparmor, libproxmox-backup-qemu0, libssl3, swtpm-tools, libssl3, librgw2, libicu72, libldap-2.5-0, libssl3, libthrift-0.17.0, libfido2-1, libcbor0.8, libssl3, kmod, libssl3, libuutil3linux, libwayland-server0, libffi8, libopts25, libmagic1, liblua5.3-0, librrds-perl, faketime, libpve-storage-perl, ceph-common, libfmt9, libicu72, libldap-2.5-0, libssl3, libthrift-0.17.0, vim-common, liblvm2cmd2.03, libip4tc2, libsystemd0, libqt5network5, libssl3, libfreetype6, libmount1, nfs-common, libevent-core-2.1-7, libnfsidmap1, samba, libldap-2.5-0, libzpool5linux, librrd8, binutils-x86-64-linux-gnu, libgprofng0, file, bzip2, genisoimage, libpipeline1, libnss-systemd, debianutils, libpopt0, libcrypt-openssl-bignum-perl, libssl3, libpve-guest-common-perl, libslirp0, logrotate, libapparmor1, libwbclient0, vncterm, libnuma1, xfsprogs, libicu72, liburcu8, python3, python3.11, openssh-server, libssl3, libdbus-1-3, libqt5dbus5, gnupg, gpg-wks-server, libkrb5support0, libsmbclient, swtpm, libxml2, libicu72, wireguard-tools, pve-cluster, fuse, xz-utils, libzstd1, util-linux, util-linux-extra, libbsd0, less, libxpm4, libfontconfig1, libgpgme11, psmisc, libasound2, systemd, libsystemd-shared, libproxmox-rs-perl, samba-dsdb-modules, libudev1, gpg-agent, libcrypt-openssl-rsa-perl, libssl3, libgpg-error0, cstream, wget, nvme-cli, libnvme1, shared-mime-info, liblocale-gettext-perl, fdisk, libspice-server1, libssl3, libreadline8, proxmox-ve, proxmox-default-kernel, python3-pycurl, lxc-pve, libgnutlsxx30, gettext-base, libmaxminddb0, samba-common-bin, libldap-2.5-0, diffutils, strace, libfdisk1, gdisk, librdkafka1, libssl3, libassuan0, python3-talloc, libpython3.11, eject, usbutils, attr, lshw, lsof, libcap2, proxmox-backup-file-restore, libssl3, libfile-fcntllock-perl, libfuse3-3, python3-cephfs, lvm2, libc6, locales, dpkg-dev, libasan6, gcc-11-base, libpcap0.8, libpcre2-16-0, libpangocairo-1.0-0, libcrypt-openssl-random-perl, libssl3, libcephfs2, libfmt9, libkrb5-3, libssl3, libpam-modules, ifupdown2, python3-systemd, ethtool, pve-xtermjs, proxmox-termproxy, tdb-tools, python3-openssl, man-db, qemu-server, libcommon-sense-perl, gpgv, libtext-charwidth-perl, libpve-access-control, libuuid-perl, bash, traceroute, pve-container, libgcrypt20, isc-dhcp-client, grep, libcpg4, libfaketime, libstatgrab10, uuid-runtime, libipset13, libcap-ng0, vim-tiny, libgfchangelog0, libctf0, bind9-dnsutils, libasyncns0, libnss3, libfilesys-df-perl, rsyslog, libk5crypto3, libpangoft2-1.0-0, ncurses-base, gzip, keyutils, libtdb1, python3-apt, dmsetup, libsndfile1, libflac12, libmp3lame0, libmpg123-0, login, libtpms0, libssl3, libqb100, gnutls-bin, rpcbind, libcrypt-ssleay-perl, libssl3, libsqlite3-0, libevent-2.1-7, hostname, rsync, libssl3, pinentry-curses, python3-gpg, libjson-c5, libopeniscsiusr, gpgsm, glusterfs-common, libssl3, liburcu8, libhtml-parser-perl, libterm-readline-gnu-perl, python3-markupsafe, libxslt1.1, libunbound8, libradosstriper1, libfmt9, proxmox-backup-client, libssl3, libdevmapper-event1.02.1, libcrypt-dev, libcryptsetup12, libssl3, libasound2-data, findutils, python3-ldb, libpython3.11, libip6tc2, libpcre2-8-0, libcrypt1, libgstreamer1.0-0, libavahi-common3, binutils-common, libitm1, libpam-modules-bin, perl-openssl-defaults, libssl3, liblinux-inotify2-perl, init-system-helpers, usrmerge, grub-efi-amd64-bin, grub2-common, thin-provisioning-tools, libsemanage-common, proxmox-mini-journalreader, smartmontools, python3-rbd, python3-rgw, smbclient, python3-protobuf, libprotobuf32, uidmap, libsubid4, libpve-http-server-perl, procps, libproc2-0, python3-tdb, libdpkg-perl, python3-markdown, libpython3-stdlib, libpython3.11-stdlib, libc-dev-bin, mount, libglib2.0-0, libffi8, pve-manager, fonts-font-logos, libpve-notify-perl, python3-yaml, libpve-common-perl, perl-base, openssh-sftp-server, libpam0g, nano, btrfs-progs, grub-common, bind9-libs, libjemalloc2, libssl3, liboath0, libdrm2, samba-libs, libicu72, libldap-2.5-0, sudo, libu2f-server0, libssl3, libc-bin, ipset, libc-devtools, glusterfs-client, librabbitmq4, libssl3, librados2-perl, python3-distutils, libattr1, efibootmgr, libaudit1, libkmod2, libssl3, libclone-perl, ebtables, libyaml-libyaml-perl, bsd-mailx, nmap, libssl3, libc6-dev, rpcsvc-proto, parted, cpio, dirmngr, libldap-2.5-0, libzfs4linux, libssl3, socat, libssl3, libksba8, libharfbuzz0b, libtext-iconv-perl, libisns0, libssl3, criu, samba-common, cron, apt-utils, gnupg-utils, libblkid1, libgnutls30, curl, libubsan1, libslang2, liblsan0, libargon2-1, sysvinit-utils, logsave, libvirglrenderer1, libva-drm2, libva2, libvulkan1, libalgorithm-diff-xs-perl, libhttp-message-perl, pve-firewall, libjson-xs-perl, gnupg-l10n, libdw1, librdmacm1, gpg-wks-client, libpci3, libgd3, libavif15, libheif1, libtiff6, libwebp7, libxml-parser-perl, libncursesw6, sqlite3, python3-cryptography, libssl3, ncurses-bin, libbz2-1.0, libnozzle1, time, libx11-6, libgstreamer-plugins-base1.0-0, libsqlite3-mod-ceph, libfmt9, libanyevent-perl, python3-ceph-argparse, libelf1, groff-base, libmd0, ncurses-term, libnet-dbus-perl, dash, libtirpc3, libtalloc2, dbus, wireguard, libknet1, libssl3, libedit2, grub-pc, libpve-u2f-server-perl, bsdutils, iproute2, libbpf1, libavahi-client3, gpgconf, libdevmapper1.02.1, libgnutls-dane0, libxml-libxml-perl, libfstrm0, libss2, libtirpc-dev, libnewt0.52, spiceterm, libgdbm6, binutils, libuv1, fakeroot, libpsl5, bsdextrautils, libstdc++6, libbabeltrace1, libncurses6, zfsutils-linux, libssl3, postfix, libicu72, libssl3, libparted2, openssl, libssl3, e2fsprogs, passwd, libsemanage2, libpve-cluster-perl, python3-rados, libapt-pkg-perl, nftables, libnspr4, libgcc-s1

Checking proxmox-ve package version..
PASS: proxmox-ve package has version >= 7.4-1

Checking running kernel version..
WARN: a suitable kernel (pve-kernel-5.15) is intalled, but an unsuitable (5.19.17-2-pve) is booted, missing reboot?!

= CHECKING CLUSTER HEALTH/SETTINGS =

PASS: systemd unit 'pve-cluster.service' is in state 'active'
PASS: systemd unit 'corosync.service' is in state 'active'
PASS: Cluster Filesystem is quorate.

Analzying quorum settings and state..
INFO: configured votes - nodes: 3
INFO: configured votes - qdevice: 0
INFO: current expected votes: 3
INFO: current total votes: 3

Checking nodelist entries..
PASS: nodelist settings OK

Checking totem settings..
PASS: totem settings OK

INFO: run 'pvecm status' to get detailed cluster status..

= CHECKING HYPER-CONVERGED CEPH STATUS =

SKIP: no hyper-converged ceph setup detected!

= CHECKING CONFIGURED STORAGES =

storage 'synology' is not online
PASS: storage 'local' enabled and active.
PASS: storage 'local-lvm' enabled and active.
WARN: storage 'synology' enabled but not active!
INFO: Checking storage content type configuration..
PASS: no storage content problems found
WARN: activating 'synology' failed - storage 'synology' is not online

PASS: no storage re-uses a directory for multiple content types.

= MISCELLANEOUS CHECKS =

INFO: Checking common daemon services..
PASS: systemd unit 'pveproxy.service' is in state 'active'
PASS: systemd unit 'pvedaemon.service' is in state 'active'
PASS: systemd unit 'pvescheduler.service' is in state 'active'
PASS: systemd unit 'pvestatd.service' is in state 'active'
INFO: Checking for supported & active NTP service..
PASS: Detected active time synchronisation unit 'chrony.service'
INFO: Checking for running guests..
WARN: 3 running guest(s) detected - consider migrating or stopping them.
INFO: Checking if the local node's hostname 'pve1' is resolvable..
INFO: Checking if resolved IP is configured on local node..
PASS: Resolved node IP '192.168.1.20' configured and active on single interface.
INFO: Check node certificate's RSA key size
PASS: Certificate 'pve-root-ca.pem' passed Debian Busters (and newer) security level for TLS connections (4096 >= 2048)
PASS: Certificate 'pve-ssl.pem' passed Debian Busters (and newer) security level for TLS connections (2048 >= 2048)
INFO: Checking backup retention settings..
PASS: no backup retention problems found.
INFO: checking CIFS credential location..
PASS: no CIFS credentials at outdated location found.
INFO: Checking permission system changes..
INFO: Checking custom role IDs for clashes with new 'PVE' namespace..
PASS: no custom roles defined, so no clash with 'PVE' role ID namespace enforced in Proxmox VE 8
INFO: Checking if LXCFS is running with FUSE3 library, if already upgraded..
SKIP: not yet upgraded, no need to check the FUSE library version LXCFS uses
INFO: Checking node and guest description/note length..
PASS: All node config descriptions fit in the new limit of 64 KiB
PASS: All guest config descriptions fit in the new limit of 8 KiB
INFO: Checking container configs for deprecated lxc.cgroup entries
PASS: No legacy 'lxc.cgroup' keys found.
INFO: Checking if the suite for the Debian security repository is correct..
PASS: found no suite mismatch
INFO: Checking for existence of NVIDIA vGPU Manager..
PASS: No NVIDIA vGPU Service found.
INFO: Checking bootloader configuration...
SKIP: not yet upgraded, no need to check the presence of systemd-boot
INFO: Check for dkms modules...
/sbin/dkms: line 2497: echo: write error: Broken pipe
WARN: dkms modules found, this might cause issues during upgrade.
SKIP: NOTE: Expensive checks, like CT cgroupv2 compat, not performed without '--full' parameter

= SUMMARY =

TOTAL:    36
PASSED:   26
SKIPPED:  4
WARNINGS: 6
FAILURES: 0

ATTENTION: Please check the output for detailed information!

I wouldn't mind killing the cluster as I only have one other reliable system and the Synology NAS' VM is flaky and unreliable. I see there's a long list of packages to be upgraded. When I run apt upgrade it tries to upgrade but fails on the kernel. Here's the snippet:

Code:
dkms: autoinstall for kernel: 6.5.13-1-pve failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.5.13-1-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-6.5.13-1-pve-signed (--configure):
 installed proxmox-kernel-6.5.13-1-pve-signed package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.5:
 proxmox-kernel-6.5 depends on proxmox-kernel-6.5.13-1-pve-signed | proxmox-kernel-6.5.13-1-pve; however:
  Package proxmox-kernel-6.5.13-1-pve-signed is not configured yet.
  Package proxmox-kernel-6.5.13-1-pve is not installed.
  Package proxmox-kernel-6.5.13-1-pve-signed which provides proxmox-kernel-6.5.13-1-pve is not configured yet.

I tried to blindly install the above packages but either one errored out with
Code:
Error! The /var/lib/dkms/wireguard/1.0.20210219/6.5.13-1-pve/x86_64/dkms.conf for module wireguard includes a BUILD_EXCLUSIVE directive which does not match this kernel/arch/config.
This indicates that it should not be built.
Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.
dkms: autoinstall for kernel: 6.5.13-1-pve failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/proxmox-kernel-6.5.13-1-pve-signed.postinst line 20.
dpkg: error processing package proxmox-kernel-6.5.13-1-pve-signed (--configure):
 installed proxmox-kernel-6.5.13-1-pve-signed package post-installation script subprocess returned error exit status 2
dpkg: dependency problems prevent configuration of proxmox-kernel-6.5:
 proxmox-kernel-6.5 depends on proxmox-kernel-6.5.13-1-pve-signed | proxmox-kernel-6.5.13-1-pve; however:
  Package proxmox-kernel-6.5.13-1-pve-signed is not configured yet.
  Package proxmox-kernel-6.5.13-1-pve is not installed.
  Package proxmox-kernel-6.5.13-1-pve-signed which provides proxmox-kernel-6.5.13-1-pve is not configured yet.

dpkg: error processing package proxmox-kernel-6.5 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 proxmox-kernel-6.5.13-1-pve-signed
 proxmox-kernel-6.5
E: Sub-process /usr/bin/dpkg returned an error code (1)

What did I do wrong? Would it be easier to do a fresh install and reinstall my 3 VMs from the snapshots on the NFS share/Synology NAS?


I think the problem is here:
WARN: a suitable kernel (pve-kernel-5.15) is intalled, but an unsuitable (5.19.17-2-pve) is booted, missing reboot?!

you are currently using kernel 5.19.17-2 which in not suitable for the upgrade. You need to run kernel 5.15. have you pinned the kernel to boot to 5.19? If yes, delete kernel pin. If not try to boot using 5.15
 
I do believe 5.15 was one of the manual choices on GRUB. I'll give it a shot.

There are 2 ways to change boot kernel:

1. attach monitor to your server and keyboard then reboot and choose the kernel
2. a more comfortable way is to ssh into proxmox and execute proxmox-boot-tool kernel pin <VERSION HERE>-pve --next-boot when prompted verify with y
 
  • Like
Reactions: Kingneutron
It'll have to be local because the 5.15 kernel didn't support the i226v ports. If I have any sense I'll wait until Friday night to start this since I have people actually using this server (virtualized router, FML) all week.
 
Looks like you used a optional kernel version (which might no longer be maintained) instead of the regular PVE 7.4 version. This should not be a real problem for upgrading to PVE 8.1.
Looks also like you manually installed a driver (via DKMS) that is not compatible with newer kernel versions. This is a serious problem for upgrading to PVE 8.1.

If you go back to the regular kernel 5.15, your special driver might not work (just as it cannot be build for kernel 6.5) and you probably lose the network device(s). If the system is headless, this might be a chicken and egg problem. Without running the new kernel, you cannot install (a newer, compatible version of) the driver and without the driver, you have no network to install the driver or even reach the system...
 
Looks like you used a optional kernel version (which might no longer be maintained) instead of the regular PVE 7.4 version. This should not be a real problem for upgrading to PVE 8.1.
Looks also like you manually installed a driver (via DKMS) that is not compatible with newer kernel versions. This is a serious problem for upgrading to PVE 8.1.

If you go back to the regular kernel 5.15, your special driver might not work (just as it cannot be build for kernel 6.5) and you probably lose the network device(s). If the system is headless, this might be a chicken and egg problem. Without running the new kernel, you cannot install (a newer, compatible version of) the driver and without the driver, you have no network to install the driver or even reach the system...
Headless is not worth the trouble here. If I use a USB network adapter that works with kernel 5.15 I can at least get connectivity to install the new kernel. That assumes that I can remove the manual driver and that the i226v works on 8.1/kernel 6.2, which I believe it does.

The test for that is to disconnect the current SSD and install on a spare SSD to see if everything works from the standard PVE 8.1 ISO. Although at that point I might just reimport my VMs and move on with my life.
 
  • Like
Reactions: Kingneutron
Headless is not worth the trouble here. If I use a USB network adapter that works with kernel 5.15 I can at least get connectivity to install the new kernel. That assumes that I can remove the manual driver and that the i226v works on 8.1/kernel 6.2, which I believe it does.
Good to hear. Just one thing: PVE 8.1 uses kernel 6.5 and kernel 6.2 (PVE 8.0) is no longer maintained (so better not use it).
 
Now i understand you problem... Sounds like you got cornered there mate. But if you have a cluster as you said, then I would migrate all the vm to the reliable server and remove that node from the cluster you want to upgrade first then do a fresh reinstall then add that back to the cluster. Remember to backup your network config to make it easier to restore your network. After that you can migrate it back to that server. This way you have zero to very little downtime.
 
Headless is not worth the trouble here. If I use a USB network adapter that works with kernel 5.15 I can at least get connectivity to install the new kernel. That assumes that I can remove the manual driver and that the i226v works on 8.1/kernel 6.2, which I believe it does.

The test for that is to disconnect the current SSD and install on a spare SSD to see if everything works from the standard PVE 8.1 ISO. Although at that point I might just reimport my VMs and move on with my life.


It should work on i226v. As I am using i226LM. I have i226v but i do not need it yet so i have not tested it yet.
 
I finished the upgrade. 5.15 and 5.19 both accessed the i226v NICs fine but it still wouldn't update until I found a Wireguard "dkms" that wasn't compatible. I don't even use Wireguard. That removed, it dist-upgraded without issues, with a proper 6.x kernel bootable. I'm just working to get GRUB to automatically choose the new kernel, since I changed the defaults to the older (booting) version.

Edit: that kernel pin problem was tough to track down until I read a few messages up.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!