[SOLVED] PVE 8 to 9 upgrade failing - Compilation errors

[dmpm]

I followed the checklist https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#Continuously_use_the_pve8to9_checklist_script , but when doing apt dist-upgrade it ends with these errors:

Processing triggers for pve-manager (8.4.13) ...
Job for pvedaemon.service failed.
See "systemctl status pvedaemon.service" and "journalctl -xeu pvedaemon.service" for details.
Job for pvestatd.service failed.
See "systemctl status pvestatd.service" and "journalctl -xeu pvestatd.service" for details.
Job for pveproxy.service failed.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.
Job for pvescheduler.service failed.
See "systemctl status pvescheduler.service" and "journalctl -xeu pvescheduler.service" for details.
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for pve-ha-manager (5.0.4) ...
Removing subscription nag from UI...

and now when I type in the console it doesn't show what I'm typing.

Checking pvedaemon.service shows:

● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-08-08 02:13:07 BST; 1 month 9 days ago
Process: 183711 ExecReload=/usr/bin/pvedaemon restart (code=exited, status=17)
Main PID: 1938 (pvedaemon)
Tasks: 6 (limit: 38226)
Memory: 229.1M
CPU: 18min 13.795s
CGroup: /system.slice/pvedaemon.service
├─ 1938 pvedaemon
├─ 122338 "task UPID:pve-oakwood:0001DDE2:1470E120:68C9A324:vncshell::root@pam:"
├─ 122339 /usr/bin/termproxy 5900 --path /nodes/pve-oakwood --perm Sys.Console -- /bin/login -f root
├─3866889 "pvedaemon worker"
├─3878898 "pvedaemon worker"
└─3949221 "pvedaemon worker"

Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 14.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 14.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 15.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 15.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Sep 16 18:54:38 pve-oakwood systemd[1]: pvedaemon.service: Control process exited, code=exited, status=17/n/a
Sep 16 18:54:38 pve-oakwood systemd[1]: Reload failed for pvedaemon.service - PVE API Daemon.
root@pve-oakwood:~#
Sep 16 18:49:03 pve-oakwood pvedaemon[3878898]: <root@pam> end task UPID:pve-oakwood:0001DD44:1470D82B:68C9A30D:vzstop:108:root@pam: OK
Sep 16 18:49:05 pve-oakwood pvedaemon[3878898]: <root@pam> starting task UPID:pve-oakwood:0001DD65:1470D980:68C9A311:vzstop:120:root@pam:
Sep 16 18:49:05 pve-oakwood pvedaemon[122213]: stopping CT 120: UPID:pve-oakwood:0001DD65:1470D980:68C9A311:vzstop:120:root@pam:
Sep 16 18:49:05 pve-oakwood pvedaemon[3949221]: unable to get PID for CT 120 (not running?)
Sep 16 18:49:06 pve-oakwood pvedaemon[3878898]: <root@pam> end task UPID:pve-oakwood:0001DD65:1470D980:68C9A311:vzstop:120:root@pam: OK
Sep 16 18:49:08 pve-oakwood pvedaemon[3878898]: <root@pam> starting task UPID:pve-oakwood:0001DD8C:1470DA9D:68C9A314:vzstop:121:root@pam:
Sep 16 18:49:08 pve-oakwood pvedaemon[122252]: stopping CT 121: UPID:pve-oakwood:0001DD8C:1470DA9D:68C9A314:vzstop:121:root@pam:
Sep 16 18:49:09 pve-oakwood pvedaemon[3878898]: <root@pam> end task UPID:pve-oakwood:0001DD8C:1470DA9D:68C9A314:vzstop:121:root@pam: OK
Sep 16 18:49:16 pve-oakwood pvedaemon[122290]: stop VM 123: UPID:pve-oakwood:0001DDB2:1470DDD6:68C9A31C:qmstop:123:root@pam:
Sep 16 18:49:16 pve-oakwood pvedaemon[3878898]: <root@pam> starting task UPID:pve-oakwood:0001DDB2:1470DDD6:68C9A31C:qmstop:123:root@pam:
Sep 16 18:49:19 pve-oakwood pvedaemon[3878898]: <root@pam> end task UPID:pve-oakwood:0001DDB2:1470DDD6:68C9A31C:qmstop:123:root@pam: OK
Sep 16 18:49:24 pve-oakwood pvedaemon[3878898]: <root@pam> starting task UPID:pve-oakwood:0001DDE2:1470E120:68C9A324:vncshell::root@pam:
Sep 16 18:49:24 pve-oakwood pvedaemon[122338]: starting termproxy UPID:pve-oakwood:0001DDE2:1470E120:68C9A324:vncshell::root@pam:
Sep 16 18:49:24 pve-oakwood pvedaemon[3949221]: <root@pam> successful auth for user 'root@pam'
Sep 16 18:49:24 pve-oakwood login[122341]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
Sep 16 18:54:37 pve-oakwood systemd[1]: Reloading pvedaemon.service - PVE API Daemon...
░░ Subject: A reload job for unit pvedaemon.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A reload job for unit pvedaemon.service has begun execution.
░░
░░ The job identifier is 43475.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Attempt to reload PVE/HA/Config.pm aborted.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 14.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 14.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 15.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 15.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedaemon.pm line 8.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Sep 16 18:54:38 pve-oakwood pvedaemon[183711]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Sep 16 18:54:38 pve-oakwood systemd[1]: pvedaemon.service: Control process exited, code=exited, status=17/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecReload= process belonging to unit pvedaemon.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 17.
Sep 16 18:54:38 pve-oakwood systemd[1]: Reload failed for pvedaemon.service - PVE API Daemon.
░░ Subject: A reload job for unit pvedaemon.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A reload job for unit pvedaemon.service has finished.
░░
░░ The job identifier is 43475 and the job result is failed.

I can't paste the output for the other services due to the post size limit, so I've put them on pastebin here https://pastebin.com/TvbpPijm

How can I fix this and get it updated? I'm too scared to try rebooting in case that doesn't work anymore.

 

[BobhWasatch]

Removing subscription nag from UI..

There's the problem. You likely used one of the "community scripts" to remove the subscription nag and it installed a hook to re-apply the patch on updates, But when you go between major versions the patch is wrong because the code has changed and said patched services will fail.

Look in /etc/apt/apt.conf.d and hopefully you'll be able to identify and remove the script that was installed. I've never used any of those scripts so I can't give an exact fix.

Then do "apt install --reinstall" the affected packages (pvedaemon, etc).

 

[dmpm]

Removing subscription nag from UI..

There's the problem. You likely used one of the "community scripts" to remove the subscription nag and it installed a hook to re-apply the patch on updates, But when you go between major versions the patch is wrong because the code has changed and said patched services will fail.

Look in /etc/apt/apt.conf.d and hopefully you'll be able to identify and remove the script that was installed. I've never used any of those scripts so I can't give an exact fix.

Then do "apt install --reinstall" the affected packages (pvedaemon, etc).

Thanks. I found a file there called "no-nag-script", which was installed using this community script https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/tools/pve/post-pve-install.sh so I moved that to /root and did "apt install -reinstall" but it just says:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 81 not upgraded.

 

[leesteken]

I find it a bit rude to ask for support while removing the reminder to get a support subscription, which possible cause the problem. Then again, it's open source and you are free to do with it whatever you want. Support for community scripts needs to be provided by the other community that maintains the scripts: https://github.com/community-scripts/ProxmoxVE/discussions

 

[BobhWasatch]

I think you need to give it a package name to reinstall.

ETA: Just so you are aware, there are so-called "man pages" that document most commands. See "man apt" for example.

 

[dmpm]

I find it a bit rude to ask for support while removing the reminder to get a support subscription, which possible cause the problem. Then again, it's open source and you are free to do with it whatever you want. Support for community scripts needs to be provided by the other community that maintains the scripts: https://github.com/community-scripts/ProxmoxVE/discussions

That seems a bit extreme. It's like saying its rude to ask for support on a Debian community forum because you've installed something that isn't officially supported by Debian and you haven't bought a subscription.

It's not even a specific "no-nag" community script, it's just this "Post Install" one https://community-scripts.github.io/ProxmoxVE/scripts?id=post-pve-install and people like me who are running single user or household servers aren't going to buy support subscriptions, especially if we're going to be told to go away because we've installed a community script that hasn't been officially approved. It's not like I'm asking for official support from the company. I note that the cheapest official support tier is 355EUR/year and for that they only let you open 3 support tickets a year, which might be fine for businesses but is no good for people like me who are learning to use PVE and experimenting.

 

[dmpm]

I think you need to give it a package name to reinstall.

ETA: Just so you are aware, there are so-called "man pages" that document most commands. See "man apt" for example.

pvedaemon, pvestatd and the other stuff that gave the compilation errors aren't packages that can be installed though.

apt list --upgradable shows

apparmor/stable 4.1.1-pmx1 amd64 [upgradable from: 3.0.8-3]
ceph-common/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
ceph-fuse/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
corosync/stable 3.1.9-pve2 amd64 [upgradable from: 3.1.9-pve1]
dmeventd/stable 2:1.02.205-2+pmx1 amd64 [upgradable from: 2:1.02.185-2]
dmsetup/stable 2:1.02.205-2+pmx1 amd64 [upgradable from: 2:1.02.185-2]
frr-pythontools/stable 10.3.1-1+pve4 all [upgradable from: 10.2.3-1+pve1]
frr/stable 10.3.1-1+pve4 amd64 [upgradable from: 10.2.3-1+pve1]
grub-common/stable 2.12-9+pmx2 amd64 [upgradable from: 2.06-13+pmx7]
grub-efi-amd64-bin/stable 2.12-9+pmx2 amd64 [upgradable from: 2.06-13+pmx7]
grub-efi-amd64-signed/stable 1+2.12+9+pmx2 amd64 [upgradable from: 1+2.06+13+pmx7]
grub-efi-amd64/stable 2.12-9+pmx2 amd64 [upgradable from: 2.06-13+pmx7]
grub-pc-bin/stable 2.12-9+pmx2 amd64 [upgradable from: 2.06-13+pmx7]
grub2-common/stable 2.12-9+pmx2 amd64 [upgradable from: 2.06-13+pmx7]
libapparmor1/stable 4.1.1-pmx1 amd64 [upgradable from: 3.0.8-3]
libcephfs2/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
libcrypt-openssl-rsa-perl/stable 0.35-1.1 amd64 [upgradable from: 0.33-3+b1]
libdevmapper-event1.02.1/stable 2:1.02.205-2+pmx1 amd64 [upgradable from: 2:1.02.185-2]
libdevmapper1.02.1/stable 2:1.02.205-2+pmx1 amd64 [upgradable from: 2:1.02.185-2]
liblvm2cmd2.03/stable 2.03.31-2+pmx1 amd64 [upgradable from: 2.03.16-2]
libnss-systemd/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
libnvpair3linux/stable 2.3.4-pve1 amd64 [upgradable from: 2.2.8-pve1]
libpam-systemd/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
libproxmox-backup-qemu0/stable 2.0.1 amd64 [upgradable from: 1.5.2]
libproxmox-rs-perl/stable 0.4.1 all [upgradable from: 0.3.5]
libpve-http-server-perl/stable 6.0.4 all [upgradable from: 5.2.2]
libpve-network-api-perl/stable 1.1.7 all [upgradable from: 0.11.2]
libpve-network-perl/stable 1.1.7 all [upgradable from: 0.11.2]
libpve-rs-perl/stable 0.10.10 amd64 [upgradable from: 0.9.4]
libpve-u2f-server-perl/stable 1.2.1 amd64 [upgradable from: 1.2.0]
librados2-perl/stable 1.5.0 amd64 [upgradable from: 1.4.1]
librados2/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
libradosstriper1/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
librbd1/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
librgw2/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
librrds-perl/stable 1.7.2-4.2+pve3 amd64 [upgradable from: 1.7.2-4+b8]
libsystemd-shared/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
libsystemd0/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
libtpms0/stable 0.9.7+pve2 amd64 [upgradable from: 0.9.7+pve1]
libudev1/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
libuutil3linux/stable 2.3.4-pve1 amd64 [upgradable from: 2.2.8-pve1]
lvm2/stable 2.03.31-2+pmx1 amd64 [upgradable from: 2.03.16-2]
lxc-pve/stable 6.0.5-1 amd64 [upgradable from: 6.0.0-1]
lxcfs/stable 6.0.4-pve1 amd64 [upgradable from: 6.0.0-pve2]
proxmox-backup-client/stable 4.0.14-1 amd64 [upgradable from: 3.4.6-1]
proxmox-backup-file-restore/stable 4.0.14-1 amd64 [upgradable from: 3.4.6-1]
proxmox-firewall/stable 1.1.2 amd64 [upgradable from: 0.7.1]
proxmox-grub/stable 2.12-9+pmx2 amd64 [upgradable from: 2.06-13+pmx7]
proxmox-mail-forward/stable 1.0.2 amd64 [upgradable from: 0.3.3]
proxmox-mini-journalreader/stable 1.6 amd64 [upgradable from: 1.5]
proxmox-offline-mirror-helper/stable 0.7.1 amd64 [upgradable from: 0.6.7]
proxmox-termproxy/stable 2.0.2 amd64 [upgradable from: 1.1.0]
proxmox-ve/stable 9.0.0 all [upgradable from: 8.4.0]
proxmox-websocket-tunnel/stable 1.0.0 amd64 [upgradable from: 0.2.0-1]
pve-cluster/stable 9.0.6 amd64 [upgradable from: 8.1.2]
pve-container/stable 6.0.12 all [upgradable from: 5.3.2]
pve-esxi-import-tools/stable 1.0.1 amd64 [upgradable from: 0.7.4]
pve-firewall/stable 6.0.3 amd64 [upgradable from: 5.1.2]
pve-lxc-syscalld/stable 2.0.2 amd64 [upgradable from: 1.3.0]
pve-manager/stable 9.0.10 all [upgradable from: 8.4.13]
pve-qemu-kvm/stable 10.0.2-4 amd64 [upgradable from: 9.2.0-7]
python3-ceph-argparse/stable,stable 19.2.3-pve1 all [upgradable from: 17.2.8-pve2]
python3-ceph-common/stable,stable 19.2.3-pve1 all [upgradable from: 17.2.8-pve2]
python3-cephfs/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
python3-rados/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
python3-rbd/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
python3-rgw/stable,stable 19.2.3-pve1 amd64 [upgradable from: 17.2.8-pve2]
qemu-server/stable 9.0.21 amd64 [upgradable from: 8.4.3]
rrdcached/stable 1.7.2-4.2+pve3 amd64 [upgradable from: 1.7.2-4+b8]
smartmontools/stable 7.4-pve1 amd64 [upgradable from: 7.3-pve1]
spiceterm/stable 3.4.0 amd64 [upgradable from: 3.3.0]
swtpm-libs/stable 0.8.0+pve2 amd64 [upgradable from: 0.8.0+pve1]
swtpm-tools/stable 0.8.0+pve2 amd64 [upgradable from: 0.8.0+pve1]
swtpm/stable 0.8.0+pve2 amd64 [upgradable from: 0.8.0+pve1]
systemd-sysv/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
systemd/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
udev/stable 257.8-1~deb13u2 amd64 [upgradable from: 252.39-1~deb12u1]
vncterm/stable 1.9.0 amd64 [upgradable from: 1.8.0]
zfs-initramfs/stable 2.3.4-pve1 all [upgradable from: 2.2.8-pve1]
zfs-zed/stable 2.3.4-pve1 amd64 [upgradable from: 2.2.8-pve1]
zfsutils-linux/stable 2.3.4-pve1 amd64 [upgradable from: 2.2.8-pve1]

but if I try to reinstall those I get errors about unmet dependencies, like so:

root@pve-oakwood:~# apt install --reinstall pve-cluster
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
pve-cluster : Depends: perl (>= 5.40.1-6) but 5.36.0-7+deb12u3 is to be installed
Depends: perlapi-5.40.1 but it is not installable
Depends: libfuse2t64 (>= 2.9.2) but it is not installable
Depends: libglib2.0-0t64 (>= 2.76.0) but it is not installable
Depends: librrd8t64 (>= 1.4~rc2) but it is not going to be installed

root@pve-oakwood:~# apt install --reinstall pve-container
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
pve-firewall : Depends: conntrack but it is not installable
Depends: libglib2.0-0t64 (>= 2.32.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

root@pve-oakwood:~# apt install --reinstall pve-firewall
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
pve-firewall : Depends: conntrack but it is not installable
Depends: libglib2.0-0t64 (>= 2.32.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

apt dist-upgrade shows:

The following packages have been kept back:
apparmor ceph-common ceph-fuse corosync dmeventd dmsetup frr frr-pythontools grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common libapparmor1 libcephfs2
libcrypt-openssl-rsa-perl libdevmapper-event1.02.1 libdevmapper1.02.1 liblvm2cmd2.03 libnss-systemd libnvpair3linux libpam-systemd libproxmox-backup-qemu0 libproxmox-rs-perl
libpve-http-server-perl libpve-network-api-perl libpve-network-perl libpve-rs-perl libpve-u2f-server-perl librados2 librados2-perl libradosstriper1 librbd1 librgw2 librrds-perl
libsystemd-shared libsystemd0 libtpms0 libudev1 libuutil3linux lvm2 lxc-pve lxcfs proxmox-backup-client proxmox-backup-file-restore proxmox-firewall proxmox-grub proxmox-mail-forward
proxmox-mini-journalreader proxmox-offline-mirror-helper proxmox-termproxy proxmox-ve proxmox-websocket-tunnel pve-cluster pve-container pve-esxi-import-tools pve-firewall pve-lxc-syscalld
pve-manager pve-qemu-kvm python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados python3-rbd python3-rgw qemu-server rrdcached smartmontools spiceterm swtpm swtpm-libs
swtpm-tools systemd systemd-sysv udev vncterm zfs-initramfs zfs-zed zfsutils-linux
0 upgraded, 0 newly installed, 0 to remove and 81 not upgraded.

 

[dmpm]

OK, turns out the problem was that I'd somehow managed to delete sources.list. It seems to be working now I've replaced that.