node failure after 8 to 9 upgrade

I

imstillalive

New Member
Sep 7, 2025
2
2
3
I attempted to upgrade 1 of my nodes to proxmox 9 today but have had issues every step of the way. Once i finally worked through the initial issues and got it seemingly pushing the upgrade to 9, the version did not reflect 9 once completed. Now I have multiple daemons failing and the node is unavailable.

Ive been digging through other posts and messing with repos alot to test but unable to find something that quite matches up with what im experiencing.
I will provide everything I can think of but can always any other info that might help resolve this issue.

almost all of them are complaining about a file (ha/rules.cfg) being unknown but im not seeing that file.

Any help is appreciated!

● pve-guests.service loaded failed failed PVE guests
● pve-ha-crm.service loaded failed failed PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service loaded failed failed PVE Local HA Resource Manager Daemon
● pvedaemon.service loaded failed failed PVE API Daemon
● pveproxy.service loaded failed failed PVE API Proxy Server
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service loaded failed failed PVE Status Daemon


root@DonkeyKong:/etc/apt# systemctl status pve-guests.service
× pve-guests.service - PVE guests
Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:43 MDT; 2h 30min ago
Process: 1343 ExecStartPre=/usr/share/pve-manager/helpers/pve-startall-delay (code=exited, status=0/SUCCESS)
Process: 1346 ExecStart=/usr/bin/pvesh --nooutput create /nodes/localhost/startall (code=exited, status=17)
Main PID: 1346 (code=exited, status=17)
CPU: 950ms

Sep 06 16:48:43 DonkeyKong pvesh[1346]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 14.
Sep 06 16:48:43 DonkeyKong pvesh[1346]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 15.
Sep 06 16:48:43 DonkeyKong pvesh[1346]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 15.
Sep 06 16:48:43 DonkeyKong pvesh[1346]: Compilation failed in require at /usr/share/perl5/PVE/CLI/pvesh.pm line 16.
Sep 06 16:48:43 DonkeyKong pvesh[1346]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/pvesh.pm line 16.
Sep 06 16:48:43 DonkeyKong pvesh[1346]: Compilation failed in require at /usr/bin/pvesh line 6.
Sep 06 16:48:43 DonkeyKong pvesh[1346]: BEGIN failed--compilation aborted at /usr/bin/pvesh line 6.
Sep 06 16:48:43 DonkeyKong systemd[1]: pve-guests.service: Main process exited, code=exited, status=17/n/a
Sep 06 16:48:43 DonkeyKong systemd[1]: pve-guests.service: Failed with result 'exit-code'.
Sep 06 16:48:43 DonkeyKong systemd[1]: Failed to start pve-guests.service - PVE guests.




root@DonkeyKong:/etc/apt# systemctl status pve-ha-crm.service
× pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon
Loaded: loaded (/lib/systemd/system/pve-ha-crm.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:36 MDT; 2h 31min ago
Process: 1316 ExecStart=/usr/sbin/pve-ha-crm start (code=exited, status=255/EXCEPTION)
CPU: 352ms

Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: unknown file 'ha/rules.cfg' at /usr/share/perl5/PVE/Cluster.pm line 524, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: Compilation failed in require at /usr/share/perl5/PVE/Service/pve_ha_crm.pm line 9, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pve_ha_crm.pm line 9, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: Compilation failed in require at /usr/sbin/pve-ha-crm line 6, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong pve-ha-crm[1316]: BEGIN failed--compilation aborted at /usr/sbin/pve-ha-crm line 6, <DATA> line 960.
Sep 06 16:48:36 DonkeyKong systemd[1]: pve-ha-crm.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 06 16:48:36 DonkeyKong systemd[1]: pve-ha-crm.service: Failed with result 'exit-code'.
Sep 06 16:48:36 DonkeyKong systemd[1]: Failed to start pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon.





root@DonkeyKong:/etc/apt# systemctl status pve-ha-lrm.service
× pve-ha-lrm.service - PVE Local HA Resource Manager Daemon
Loaded: loaded (/lib/systemd/system/pve-ha-lrm.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:41 MDT; 2h 32min ago
Process: 1336 ExecStart=/usr/sbin/pve-ha-lrm start (code=exited, status=255/EXCEPTION)
CPU: 351ms

Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: unknown file 'ha/rules.cfg' at /usr/share/perl5/PVE/Cluster.pm line 524, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 20, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: Compilation failed in require at /usr/share/perl5/PVE/Service/pve_ha_lrm.pm line 9, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pve_ha_lrm.pm line 9, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: Compilation failed in require at /usr/sbin/pve-ha-lrm line 6, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong pve-ha-lrm[1336]: BEGIN failed--compilation aborted at /usr/sbin/pve-ha-lrm line 6, <DATA> line 960.
Sep 06 16:48:41 DonkeyKong systemd[1]: pve-ha-lrm.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 06 16:48:41 DonkeyKong systemd[1]: pve-ha-lrm.service: Failed with result 'exit-code'.
Sep 06 16:48:41 DonkeyKong systemd[1]: Failed to start pve-ha-lrm.service - PVE Local HA Resource Manager Daemon.
Click to expand...


root@DonkeyKong:/etc/apt# systemctl status pvedaemon.service
× pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:51 MDT; 2h 32min ago
Process: 1387 ExecStart=/usr/bin/pvedaemon start (code=exited, status=17)
CPU: 666ms

Sep 06 16:48:51 DonkeyKong systemd[1]: Failed to start pvedaemon.service - PVE API Daemon.
Sep 06 16:48:52 DonkeyKong systemd[1]: pvedaemon.service: Start request repeated too quickly.
Sep 06 16:48:52 DonkeyKong systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Sep 06 16:48:52 DonkeyKong systemd[1]: Failed to start pvedaemon.service - PVE API Daemon.
Sep 06 16:48:53 DonkeyKong systemd[1]: pvedaemon.service: Start request repeated too quickly.
Sep 06 16:48:53 DonkeyKong systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Sep 06 16:48:53 DonkeyKong systemd[1]: Failed to start pvedaemon.service - PVE API Daemon.
Sep 06 16:48:54 DonkeyKong systemd[1]: pvedaemon.service: Start request repeated too quickly.
Sep 06 16:48:54 DonkeyKong systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Sep 06 16:48:54 DonkeyKong systemd[1]: Failed to start pvedaemon.service - PVE API Daemon.






root@DonkeyKong:/etc/apt# systemctl status pveproxy.service
× pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:54 MDT; 2h 33min ago
Process: 1396 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 1398 ExecStart=/usr/bin/pveproxy start (code=exited, status=17)
CPU: 1.023s

Sep 06 16:48:54 DonkeyKong systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 10.
Sep 06 16:48:54 DonkeyKong systemd[1]: Stopped pveproxy.service - PVE API Proxy Server.
Sep 06 16:48:54 DonkeyKong systemd[1]: pveproxy.service: Consumed 1.023s CPU time.
Sep 06 16:48:54 DonkeyKong systemd[1]: pveproxy.service: Start request repeated too quickly.
Sep 06 16:48:54 DonkeyKong systemd[1]: pveproxy.service: Failed with result 'exit-code'.
Sep 06 16:48:54 DonkeyKong systemd[1]: Failed to start pveproxy.service - PVE API Proxy Server.





root@DonkeyKong:/etc/apt# systemctl status pvescheduler.service
× pvescheduler.service - Proxmox VE scheduler
Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:43 MDT; 2h 34min ago
Process: 1350 ExecStart=/usr/bin/pvescheduler start (code=exited, status=255/EXCEPTION)
CPU: 542ms

Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Jobs/VZDump.pm line 10, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: Compilation failed in require at /usr/share/perl5/PVE/Jobs.pm line 9, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Jobs.pm line 9, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvescheduler.pm line 8, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvescheduler.pm line 8, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: Compilation failed in require at /usr/bin/pvescheduler line 6, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong pvescheduler[1350]: BEGIN failed--compilation aborted at /usr/bin/pvescheduler line 6, <DATA> line 960.
Sep 06 16:48:43 DonkeyKong systemd[1]: pvescheduler.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 06 16:48:43 DonkeyKong systemd[1]: pvescheduler.service: Failed with result 'exit-code'.
Sep 06 16:48:43 DonkeyKong systemd[1]: Failed to start pvescheduler.service - Proxmox VE scheduler.






root@DonkeyKong:/etc/apt# systemctl status pvestatd.service
× pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-09-06 16:48:36 MDT; 2h 34min ago
Process: 1280 ExecStart=/usr/bin/pvestatd start (code=exited, status=255/EXCEPTION)
CPU: 709ms

Sep 06 16:48:36 DonkeyKong pvestatd[1280]: unknown file 'ha/rules.cfg' at /usr/share/perl5/PVE/Cluster.pm line 524.
Sep 06 16:48:36 DonkeyKong pvestatd[1280]: Compilation failed in require at /usr/share/perl5/PVE/QemuServer.pm line 36.
Sep 06 16:48:36 DonkeyKong pvestatd[1280]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/QemuServer.pm line 36.
Sep 06 16:48:36 DonkeyKong pvestatd[1280]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvestatd.pm line 21.
Sep 06 16:48:36 DonkeyKong pvestatd[1280]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvestatd.pm line 21.
Sep 06 16:48:36 DonkeyKong pvestatd[1280]: Compilation failed in require at /usr/bin/pvestatd line 9.
Sep 06 16:48:36 DonkeyKong pvestatd[1280]: BEGIN failed--compilation aborted at /usr/bin/pvestatd line 9.
Sep 06 16:48:36 DonkeyKong systemd[1]: pvestatd.service: Control process exited, code=exited, status=255/EXCEPTION
Sep 06 16:48:36 DonkeyKong systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Sep 06 16:48:36 DonkeyKong systemd[1]: Failed to start pvestatd.service - PVE Status Daemon.
Click to expand...





head -n -0 /etc/apt/sources.list /etc/apt/sources.list.d/*
==> /etc/apt/sources.list <==
deb http://security.debian.org trixie-security main contrib

==> /etc/apt/sources.list.d/ceph.list <==
deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription

==> /etc/apt/sources.list.d/ceph.sources <==
Types: deb
URIs: http://download.proxmox.com/debian/ceph-squid
Suites: trixie
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

==> /etc/apt/sources.list.d/proxmox.sources <==
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

==> /etc/apt/sources.list.d/pve-enterprise.list <==
# deb https://enterprise.proxmox.com/debian/pve trixie pve-enterprise
Click to expand...
 
UdoB said:
Try to follow https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 carefully. As long as your specific system is not heavily modified it should lead to success.

In your sources.list I can see at least one bookworm (should be trixie) and the fundamental "main" Debian source is... missing?
Click to expand...
Thanks for looking into it!
I think the main issue was the remaining bookworm and missing repo. i ended up copying the repos from my 2nd server and following the 8 to 9 instructions again. Ran into a duplicate repo error but once that was fixed it seemed fine.

I was able to get it to perform the upgrade with a successful outcome and everything appears normal.

Thanks again!
 
  • Like
Reactions: UdoB and SteveITS
You must log in or register to reply here.
Top Bottom