After upgrade to PVE 9, qm, ha-manager, pvestatd, and more fail to start (unknown file 'ha/rules.cfg')

[nathen419]

After upgrading to PVE9 vms fail to start, qm, ha-manager, services like pvestatd, web gui, and more fail to start with the common error:

root@pve3:/etc/pve# qm
unknown file 'ha/rules.cfg' at /usr/share/perl5/PVE/Cluster.pm line 524, <DATA> line 960.
Compilation failed in require at /usr/share/perl5/PVE/QemuServer.pm line 36, <DATA> line 960.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/QemuServer.pm line 36, <DATA> line 960.
Compilation failed in require at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 8, <DATA> line 960.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 8, <DATA> line 960.
Compilation failed in require at /usr/share/perl5/PVE/CLI/qm.pm line 30, <DATA> line 960.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/qm.pm line 30, <DATA> line 960.
Compilation failed in require at /usr/sbin/qm line 6, <DATA> line 960.
BEGIN failed--compilation aborted at /usr/sbin/qm line 6, <DATA> line 960

root@pve3:~# systemctl status pvestatd.service
× pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Fri 2025-08-08 01:24:17 CDT; 4s ago
    Process: 12648 ExecStart=/usr/bin/pvestatd start (code=exited, status=255/EXCEPTION)
        CPU: 1.335s

Aug 08 01:24:17 pve3 pvestatd[12648]: Compilation failed in require at /usr/share/perl5/PVE/QemuServer.pm line 36.
Aug 08 01:24:17 pve3 pvestatd[12648]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/QemuServer.pm line 36.
Aug 08 01:24:17 pve3 pvestatd[12648]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvestatd.pm line 21.
Aug 08 01:24:17 pve3 pvestatd[12648]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvestatd.pm lin>
Aug 08 01:24:17 pve3 pvestatd[12648]: Compilation failed in require at /usr/bin/pvestatd line 9.
Aug 08 01:24:17 pve3 pvestatd[12648]: BEGIN failed--compilation aborted at /usr/bin/pvestatd line 9.
Aug 08 01:24:17 pve3 systemd[1]: pvestatd.service: Control process exited, code=exited, status=255/EXCEPTION
Aug 08 01:24:17 pve3 systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Aug 08 01:24:17 pve3 systemd[1]: Failed to start pvestatd.service - PVE Status Daemon.
Aug 08 01:24:17 pve3 systemd[1]: pvestatd.service: Consumed 1.335s CPU time.

No files exist in /etc/pve/ha, and it was not configured on the host prior to the upgrade.


For those who ask, yes i did run the pve8to9 --full with no issues.

What gives? I cant find any documentation on what should be in that directory or why this is failing.

 

[dakralex]

Hi!

On which versions are the packages on pve3 (output of pveversion -v)?

Edit: It seems like an error from a partial upgrade where pve-cluster is not up-to-date yet.

 

[nathen419]

forgot to attach this to the initial post, my bad.


pveversion -v

root@pve3:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.14.8-2-pve)
pve-manager: 8.4.9 (running version: 8.4.9/649acf70aab54798)
proxmox-kernel-helper: 9.0.3
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
proxmox-kernel-6.14: 6.14.8-2
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.8-pve2
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
dnsmasq: 2.91-1
frr-pythontools: 10.2.3-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.3.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.1-1
lvm2: 2.03.31-2
lxc-pve: 6.0.4-2
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 3.4.3-1
proxmox-backup-file-restore: 3.4.3-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.1.1
proxmox-kernel-helper: 9.0.3
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.6
proxmox-widget-toolkit: 5.0.5
pve-cluster: 8.1.2
pve-container: 5.3.0
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 5.0.4
pve-i18n: 3.5.2
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.1
smartmontools: 7.4-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.8-pve1

 

[dakralex]

forgot to attach this to the initial post, my bad.


pveversion -v

Thanks, according to the output this looks like that there was an error while upgrading the packages and the packages were only upgraded partially. Were there any issues during the upgrade when running apt update and apt dist-upgrade? Were the bookworm/pve8 repositories removed/replaced in the apt sources list?

If there was a problem, does apt --fix-broken install / dpkg --configure -a solve the problem (make sure that you understand what these commands do before running them)?

 

[wrobelda]

Thanks, according to the output this looks like that there was an error while upgrading the packages and the packages were only upgraded partially. Were there any issues during the upgrade when running apt update and apt dist-upgrade? Were the bookworm/pve8 repositories removed/replaced in the apt sources list?

If there was a problem, does apt --fix-broken install / dpkg --configure -a solve the problem (make sure that you understand what these commands do before running them)?

Same issue here and no packages were reported broken:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  glusterfs-client libpve-u2f-server-perl libu2f-server0
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  proxmox-kernel-6.14 proxmox-kernel-6.14.8-2-pve-signed
The following packages have been kept back:
  apparmor ceph-common ceph-fuse corosync dmeventd dmsetup grub-common grub-efi-amd64-bin grub-efi-amd64-signed grub-pc grub-pc-bin grub2-common libapparmor1 libcephfs2 libcrypt-openssl-rsa-perl
  libdevmapper-event1.02.1 libdevmapper1.02.1 liblvm2cmd2.03 libnvpair3linux libproxmox-backup-qemu0 libproxmox-rs-perl libpve-http-server-perl libpve-network-api-perl libpve-network-perl libpve-rs-perl
  libpve-u2f-server-perl librados2 librados2-perl libradosstriper1 librbd1 librrds-perl libtpms0 libuutil3linux lvm2 lxc-pve lxcfs proxmox-backup proxmox-backup-client proxmox-backup-file-restore
  proxmox-backup-server proxmox-firewall proxmox-mail-forward proxmox-mini-journalreader proxmox-network-interface-pinning proxmox-offline-mirror-helper proxmox-termproxy proxmox-ve
  proxmox-websocket-tunnel pve-cluster pve-container pve-esxi-import-tools pve-firewall pve-lxc-syscalld pve-manager pve-qemu-kvm python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados
  python3-rbd qemu-server rrdcached sg3-utils smartmontools spiceterm swtpm swtpm-libs swtpm-tools vncterm zfs-zed zfsutils-linux
The following packages will be upgraded:
  ifupdown2 libcfg7 libcmap4 libcorosync-common4 libcpg4 libproxmox-acme-perl libproxmox-acme-plugins libpve-access-control libpve-apiclient-perl libpve-cluster-api-perl libpve-cluster-perl
  libpve-common-perl libpve-guest-common-perl libpve-notify-perl libpve-storage-perl libquorum5 libvotequorum8 novnc-pve pbs-i18n proxmox-archive-keyring proxmox-backup-docs proxmox-backup-restore-image
  proxmox-default-kernel proxmox-kernel-helper proxmox-offline-mirror-docs proxmox-widget-toolkit pve-docs pve-edk2-firmware-legacy pve-edk2-firmware-ovmf pve-ha-manager pve-i18n shim-signed
  shim-signed-common
33 upgraded, 2 newly installed, 0 to remove and 71 not upgraded.
Need to get 167 MB of archives.
After this operation, 638 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
[..]
Setting up proxmox-widget-toolkit (5.0.5) ...
Setting up proxmox-backup-docs (4.0.14-1) ...
Setting up pve-edk2-firmware-ovmf (4.2025.02-4) ...
Setting up libcorosync-common4:amd64 (3.1.9-pve2) ...
Setting up proxmox-offline-mirror-docs (0.7.0) ...
Setting up proxmox-archive-keyring (4.0) ...
Setting up ifupdown2 (3.3.0-1+pmx9) ...
Setting up pve-docs (9.0.8) ...
Setting up proxmox-kernel-helper (9.0.3) ...
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
Setting up novnc-pve (1.6.0-3) ...
Setting up pbs-i18n (3.5.2) ...
Setting up libpve-apiclient-perl (3.4.0) ...
Setting up proxmox-backup-restore-image (1.0.0) ...
Setting up libproxmox-acme-plugins (1.7.0) ...
Setting up proxmox-kernel-6.14.8-2-pve-signed (6.14.8-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.14.8-2-pve /boot/vmlinuz-6.14.8-2-pve
update-initramfs: Generating /boot/initrd.img-6.14.8-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 6.14.8-2-pve /boot/vmlinuz-6.14.8-2-pve
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 6.14.8-2-pve /boot/vmlinuz-6.14.8-2-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 6.14.8-2-pve /boot/vmlinuz-6.14.8-2-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.14.8-2-pve /boot/vmlinuz-6.14.8-2-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.14.8-2-pve
Found initrd image: /boot/initrd.img-6.14.8-2-pve
Found linux image: /boot/vmlinuz-6.8.12-13-pve
Found initrd image: /boot/initrd.img-6.8.12-13-pve
Found linux image: /boot/vmlinuz-6.8.12-9-pve
Found initrd image: /boot/initrd.img-6.8.12-9-pve
done
Setting up shim-signed-common (1.47+pmx1+15.8-1+pmx1) ...
No DKMS packages installed: not changing Secure Boot validation state.
Setting up pve-i18n (3.5.2) ...
Setting up libcfg7:amd64 (3.1.9-pve2) ...
Setting up pve-edk2-firmware-legacy (4.2025.02-4) ...
Setting up proxmox-kernel-6.14 (6.14.8-2) ...
Setting up libvotequorum8:amd64 (3.1.9-pve2) ...
Setting up libcpg4:amd64 (3.1.9-pve2) ...
Setting up libcmap4:amd64 (3.1.9-pve2) ...
Setting up libquorum5:amd64 (3.1.9-pve2) ...
Setting up shim-signed:amd64 (1.47+pmx1+15.8-1+pmx1) ...
Setting up proxmox-default-kernel (2.0.0) ...
Setting up libproxmox-acme-perl (1.7.0) ...
Setting up libpve-common-perl (9.0.9) ...
Setting up libpve-cluster-perl (9.0.6) ...
Setting up libpve-notify-perl (9.0.6) ...
Setting up libpve-access-control (9.0.3) ...
Setting up pve-ha-manager (5.0.4) ...
watchdog-mux.service is a disabled or a static unit, not starting it.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
Setting up libpve-cluster-api-perl (9.0.6) ...
Setting up libpve-storage-perl (9.0.13) ...
Setting up libpve-guest-common-perl (6.0.2) ...
Processing triggers for grub-cloud-amd64 (0.0.5) ...
Installing for i386-pc platform.
Installation finished. No error reported.
Installing for x86_64-efi platform.
Installation finished. No error reported.
Processing triggers for proxmox-backup-file-restore (3.4.6-1) ...
Updating file-restore initramfs...
14343 blocks
Processing triggers for initramfs-tools (0.142+deb12u3) ...
update-initramfs: Generating /boot/initrd.img-6.14.8-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Processing triggers for libc-bin (2.36-9+deb12u10) ...
Processing triggers for pve-manager (8.4.11) ...
Job for pvedaemon.service failed.
See "systemctl status pvedaemon.service" and "journalctl -xeu pvedaemon.service" for details.
Job for pvestatd.service failed.
See "systemctl status pvestatd.service" and "journalctl -xeu pvestatd.service" for details.
Job for pveproxy.service failed.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.
Job for pvescheduler.service failed.
See "systemctl status pvescheduler.service" and "journalctl -xeu pvescheduler.service" for details.
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for pve-ha-manager (5.0.4) ...

However, do notice that the crucial packages are being kept back for whatever reason. I double checked my apt/sources and:

 cat /etc/apt/sources.list
# See /etc/apt/sources.list.d/debian.sources

I am running this in a cloud KVM, the provider probably opted into using debian.sources instead of sources.list, which is now default in Trixie but was already available in Bookworm. I believe upgrade instructions for both PVE and PBS should instruct to double check whether /etc/apt/sources.list or /etc/apt/sources.list.d/debian.sources is in use and sed replace in-place `bookworm` with `trixie` as needed.

 

[whitetigerswt]

Hi,

Just respond to say that I have run into the same issue and have the same error posted by wrobelda and nathen419 while upgrading to PVE9. Have you guys resolved this issue?

Thanks in advance.

EDIT: Sorry, was unclear from the last post that I needed to modify the sources.list to remove the comments. I followed sockx's advice from this post: https://forum.proxmox.com/threads/pve-8to9-failed-qm-had-some-errors.169678/ "I commented out the contents of `/etc/apt/sources.list` too early. I removed the `#` and now the upgrade is normal."

For the record, the reason I thought I should modify that sources.list file was because while running apt update before the upgrade, it told me the sources are duplicated in places after adding the new pve9 repositories. I figured I should remove the old references before running the upgrade. Maybe we can make this clear in the documentation to ignore this warning before running the upgrade, but after running it, you should remove them?

after running apt dist-upgrade again in this weird state, grub-efi-amd64 was uninstalled. While it didn't break my grub, it brings back bad memories of your system becoming unbootable. Almost fell out of my chair .

 

[dakralex]

I am running this in a cloud KVM, the provider probably opted into using debian.sources instead of sources.list, which is now default in Trixie but was already available in Bookworm. I believe upgrade instructions for both PVE and PBS should instruct to double check whether /etc/apt/sources.list or /etc/apt/sources.list.d/debian.sources is in use and sed replace in-place `bookworm` with `trixie` as needed.

Hm, please look into whether /etc/apt/sources.list.d/ contains the Debian Trixie repositories, Proxmox VE 9 / Trixie repositories, Ceph Trixie repositories and that you run apt dist-upgrade. Other packages might be held back because Proxmox VE 9 package dependencies might not be available.

 

[wrobelda]

Hm, please look into whether /etc/apt/sources.list.d/ contains the Debian Trixie repositories, Proxmox VE 9 / Trixie repositories, Ceph Trixie repositories and that you run apt dist-upgrade. Other packages might be held back because Proxmox VE 9 package dependencies might not be available.

That's exactly what was happening, as I explained. Problem is the instructions assume debian repos are in /etc/apt/sources.list and there only, meanwhile mine were in /etc/apt/sources.list.d/