[SOLVED] pve services start failed

[oomaxim]

Have cluster with 4 nodes.

Yesterday at 4 node, some of pve services failed to start.

pveversion -v output

proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-network-perl: 0.6.1
libpve-storage-perl: 7.0-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

systemctl output:

â— pve-storage.target - PVE Storage Target
     Loaded: loaded (/lib/systemd/system/pve-storage.target; static)
     Active: active since Mon 2021-11-01 23:31:37 CET; 10h ago

Nov 01 23:31:37 pve4 systemd[1]: Reached target PVE Storage Target.

â— pvedaemon.service - PVE API Daemon
     Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2021-11-01 23:31:43 CET; 10h ago
    Process: 2715 ExecStart=/usr/bin/pvedaemon start (code=exited, status=255/EXCEPTION)
        CPU: 610ms

Nov 01 23:31:43 pve4 systemd[1]: Failed to start PVE API Daemon.
Nov 01 23:31:44 pve4 systemd[1]: pvedaemon.service: Start request repeated too quickly.
Nov 01 23:31:44 pve4 systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Nov 01 23:31:44 pve4 systemd[1]: Failed to start PVE API Daemon.
Nov 01 23:31:45 pve4 systemd[1]: pvedaemon.service: Start request repeated too quickly.
Nov 01 23:31:45 pve4 systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Nov 01 23:31:45 pve4 systemd[1]: Failed to start PVE API Daemon.
Nov 01 23:31:47 pve4 systemd[1]: pvedaemon.service: Start request repeated too quickly.
Nov 01 23:31:47 pve4 systemd[1]: pvedaemon.service: Failed with result 'exit-code'.
Nov 01 23:31:47 pve4 systemd[1]: Failed to start PVE API Daemon.

â— pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-11-01 23:31:39 CET; 10h ago
    Process: 2642 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
   Main PID: 2669 (pvestatd)
      Tasks: 1 (limit: 629145)
     Memory: 119.5M
        CPU: 12min 25.316s
     CGroup: /system.slice/pvestatd.service
             └─2669 pvestatd

Nov 01 23:31:38 pve4 systemd[1]: Starting PVE Status Daemon...
Nov 01 23:31:39 pve4 pvestatd[2669]: starting server
Nov 01 23:31:39 pve4 systemd[1]: Started PVE Status Daemon.

â— pvenetcommit.service - Commit Proxmox VE network changes
     Loaded: loaded (/lib/systemd/system/pvenetcommit.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2021-11-01 23:31:27 CET; 10h ago
    Process: 1855 ExecStartPre=/bin/rm -f /etc/openvswitch/conf.db (code=exited, status=0/SUCCESS)
    Process: 1857 ExecStartPre=/bin/mv /etc/network/interfaces.new /etc/network/interfaces (code=exited, status=1/FAILURE)
    Process: 1862 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
   Main PID: 1862 (code=exited, status=0/SUCCESS)
        CPU: 5ms

Nov 01 23:31:27 pve4 systemd[1]: Starting Commit Proxmox VE network changes...
Nov 01 23:31:27 pve4 mv[1857]: /bin/mv: cannot stat '/etc/network/interfaces.new': No such file or directory
Nov 01 23:31:27 pve4 systemd[1]: Finished Commit Proxmox VE network changes.

â— pvesr.timer - Proxmox VE replication runner
     Loaded: loaded (/lib/systemd/system/pvesr.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2021-11-01 23:31:29 CET; 10h ago
    Trigger: Tue 2021-11-02 10:07:00 CET; 23s left
   Triggers: â— pvesr.service

Nov 01 23:31:29 pve4 systemd[1]: Started Proxmox VE replication runner.

â— pve-lxc-syscalld.service - Proxmox VE LXC Syscall Daemon
     Loaded: loaded (/lib/systemd/system/pve-lxc-syscalld.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-11-01 23:31:27 CET; 10h ago
   Main PID: 1890 (pve-lxc-syscall)
      Tasks: 105 (limit: 629145)
     Memory: 6.2M
        CPU: 11ms
     CGroup: /system.slice/pve-lxc-syscalld.service
             └─1890 /usr/lib/x86_64-linux-gnu/pve-lxc-syscalld/pve-lxc-syscalld --system /run/pve/lxc-syscalld.sock

Nov 01 23:31:27 pve4 systemd[1]: Starting Proxmox VE LXC Syscall Daemon...
Nov 01 23:31:27 pve4 systemd[1]: Started Proxmox VE LXC Syscall Daemon.

â— pve-firewall.service - Proxmox VE firewall
     Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-11-01 23:31:39 CET; 10h ago
    Process: 2641 ExecStartPre=/usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy (code=exited, status=0/SUCCESS)
    Process: 2643 ExecStartPre=/usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (code=exited, status=0/SUCCESS)
    Process: 2644 ExecStartPre=/usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (code=exited, status=0/SUCCESS)
    Process: 2645 ExecStart=/usr/sbin/pve-firewall start (code=exited, status=0/SUCCESS)
   Main PID: 2670 (pve-firewall)
      Tasks: 1 (limit: 629145)
     Memory: 88.4M
        CPU: 3min 3.400s
     CGroup: /system.slice/pve-firewall.service
             └─2670 pve-firewall

Nov 01 23:31:38 pve4 systemd[1]: Starting Proxmox VE firewall...
Nov 01 23:31:39 pve4 pve-firewall[2670]: starting server
Nov 01 23:31:39 pve4 systemd[1]: Started Proxmox VE firewall.

â— pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2021-11-01 23:31:47 CET; 10h ago
    Process: 2725 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 2727 ExecStart=/usr/bin/pveproxy start (code=exited, status=255/EXCEPTION)
        CPU: 916ms

Nov 01 23:31:47 pve4 systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 5.
Nov 01 23:31:47 pve4 systemd[1]: Stopped PVE API Proxy Server.
Nov 01 23:31:47 pve4 systemd[1]: pveproxy.service: Start request repeated too quickly.
Nov 01 23:31:47 pve4 systemd[1]: pveproxy.service: Failed with result 'exit-code'.
Nov 01 23:31:47 pve4 systemd[1]: Failed to start PVE API Proxy Server.

â— pve-daily-update.timer - Daily PVE download activities
     Loaded: loaded (/lib/systemd/system/pve-daily-update.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2021-11-01 23:31:29 CET; 10h ago
    Trigger: Wed 2021-11-03 04:52:56 CET; 18h left
   Triggers: â— pve-daily-update.service

Nov 01 23:31:29 pve4 systemd[1]: Started Daily PVE download activities.

â— pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon
     Loaded: loaded (/lib/systemd/system/pve-ha-crm.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2021-11-01 23:31:40 CET; 10h ago
    Process: 2694 ExecStart=/usr/sbin/pve-ha-crm start (code=exited, status=255/EXCEPTION)
        CPU: 611ms

Nov 01 23:31:40 pve4 pve-ha-crm[2694]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Resources/PVEVM.pm line 19.
Nov 01 23:31:40 pve4 pve-ha-crm[2694]: Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 21.
Nov 01 23:31:40 pve4 pve-ha-crm[2694]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 21.
Nov 01 23:31:40 pve4 pve-ha-crm[2694]: Compilation failed in require at /usr/share/perl5/PVE/Service/pve_ha_crm.pm line 9.
Nov 01 23:31:40 pve4 pve-ha-crm[2694]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pve_ha_crm.pm line 9.
Nov 01 23:31:40 pve4 pve-ha-crm[2694]: Compilation failed in require at /usr/sbin/pve-ha-crm line 6.
Nov 01 23:31:40 pve4 pve-ha-crm[2694]: BEGIN failed--compilation aborted at /usr/sbin/pve-ha-crm line 6.
Nov 01 23:31:40 pve4 systemd[1]: pve-ha-crm.service: Control process exited, code=exited, status=255/EXCEPTION
Nov 01 23:31:40 pve4 systemd[1]: pve-ha-crm.service: Failed with result 'exit-code'.
Nov 01 23:31:40 pve4 systemd[1]: Failed to start PVE Cluster HA Resource Manager Daemon.

â— pve-guests.service - PVE guests
     Loaded: loaded (/lib/systemd/system/pve-guests.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2021-11-01 23:31:43 CET; 10h ago
    Process: 2714 ExecStartPre=/usr/share/pve-manager/helpers/pve-startall-delay (code=exited, status=0/SUCCESS)
    Process: 2716 ExecStart=/usr/bin/pvesh --nooutput create /nodes/localhost/startall (code=exited, status=255/EXCEPTION)
   Main PID: 2716 (code=exited, status=255/EXCEPTION)
        CPU: 829ms

Nov 01 23:31:43 pve4 pvesh[2716]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 14.
Nov 01 23:31:43 pve4 pvesh[2716]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
Nov 01 23:31:43 pve4 pvesh[2716]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Nov 01 23:31:43 pve4 pvesh[2716]: Compilation failed in require at /usr/share/perl5/PVE/CLI/pvesh.pm line 16.
Nov 01 23:31:43 pve4 pvesh[2716]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/pvesh.pm line 16.
Nov 01 23:31:43 pve4 pvesh[2716]: Compilation failed in require at /usr/bin/pvesh line 6.
Nov 01 23:31:43 pve4 pvesh[2716]: BEGIN failed--compilation aborted at /usr/bin/pvesh line 6.
Nov 01 23:31:43 pve4 systemd[1]: pve-guests.service: Main process exited, code=exited, status=255/EXCEPTION
Nov 01 23:31:43 pve4 systemd[1]: pve-guests.service: Failed with result 'exit-code'.
Nov 01 23:31:43 pve4 systemd[1]: Failed to start PVE guests.

â— pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-11-01 23:31:38 CET; 10h ago
    Process: 2614 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
   Main PID: 2617 (pmxcfs)
      Tasks: 5 (limit: 629145)
     Memory: 23.7M
        CPU: 21.079s
     CGroup: /system.slice/pve-cluster.service
             └─2617 /usr/bin/pmxcfs

Nov 02 08:28:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 08:43:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 08:58:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 09:10:10 pve4 pmxcfs[2617]: [dcdb] notice: data verification successful
Nov 02 09:13:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 09:28:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 09:43:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 09:58:52 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 10:03:01 pve4 pmxcfs[2617]: [status] notice: received log
Nov 02 10:03:01 pve4 pmxcfs[2617]: [status] notice: received log

â— pvefw-logger.service - Proxmox VE firewall logger
     Loaded: loaded (/lib/systemd/system/pvefw-logger.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-11-02 00:00:00 CET; 10h ago
    Process: 8709 ExecStart=/usr/sbin/pvefw-logger (code=exited, status=0/SUCCESS)
   Main PID: 8710 (pvefw-logger)
      Tasks: 2 (limit: 629145)
     Memory: 376.0K
        CPU: 1.343s
     CGroup: /system.slice/pvefw-logger.service
             └─8710 /usr/sbin/pvefw-logger

Nov 02 00:00:00 pve4 systemd[1]: Starting Proxmox VE firewall logger...
Nov 02 00:00:00 pve4 systemd[1]: Started Proxmox VE firewall logger.
Nov 02 00:00:00 pve4 pvefw-logger[8710]: starting pvefw logger

â— pve-ha-lrm.service - PVE Local HA Resource Manager Daemon
     Loaded: loaded (/lib/systemd/system/pve-ha-lrm.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2021-11-01 23:31:41 CET; 10h ago
    Process: 2701 ExecStart=/usr/sbin/pve-ha-lrm start (code=exited, status=255/EXCEPTION)
        CPU: 609ms

Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Resources/PVEVM.pm line 19.
Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 21.
Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 21.
Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: Compilation failed in require at /usr/share/perl5/PVE/Service/pve_ha_lrm.pm line 9.
Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pve_ha_lrm.pm line 9.
Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: Compilation failed in require at /usr/sbin/pve-ha-lrm line 6.
Nov 01 23:31:41 pve4 pve-ha-lrm[2701]: BEGIN failed--compilation aborted at /usr/sbin/pve-ha-lrm line 6.
Nov 01 23:31:41 pve4 systemd[1]: pve-ha-lrm.service: Control process exited, code=exited, status=255/EXCEPTION
Nov 01 23:31:41 pve4 systemd[1]: pve-ha-lrm.service: Failed with result 'exit-code'.
Nov 01 23:31:41 pve4 systemd[1]: Failed to start PVE Local HA Resource Manager Daemon.

â— pvebanner.service - Proxmox VE Login Banner
     Loaded: loaded (/lib/systemd/system/pvebanner.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2021-11-01 23:31:27 CET; 10h ago
    Process: 1849 ExecStart=/usr/bin/pvebanner (code=exited, status=0/SUCCESS)
   Main PID: 1849 (code=exited, status=0/SUCCESS)
        CPU: 203ms

Nov 01 23:31:27 pve4 systemd[1]: Starting Proxmox VE Login Banner...
Nov 01 23:31:27 pve4 systemd[1]: Finished Proxmox VE Login Banner.

This node is updated and no updates is available.
Tried to re install the failed services, but with out success , still same errors.
Will glad to get some suggestion how to fix it.

 

[oguz]

hi,

does this happen on all nodes?

Tried to re install the failed services, but with out success , still same errors.

have you tried apt install pve-ha-manager --reinstall?

from the errors it looks like there might be modifications in your files? did you make any changes to the source at all?

 

[oomaxim]

have you tried apt install pve-ha-manager --reinstall?

Yes, still the same problem.

did you make any changes to the source at all?

yes, I did an modification at /usr/share/perl5/PVE/API2/Qemu.pm and /usr/share/perl5/PVE/QemuServer/Cloudinit.pm to make cloud init for windows work.
Also was installed the "sudo" packet from repo.

does this happen on all nodes?

All other nodes working perfect with these modification with out any problems.

 

[oguz]

yes, I did an modification at /usr/share/perl5/PVE/API2/Qemu.pm and /usr/share/perl5/PVE/QemuServer/Cloudinit.pm to make cloud init for windows work.

then maybe you've made a typo somewhere?

Also was installed the "sudo" packet from repo.

that should be fine

if you run apt install qemu-server --reinstall does it work normally?

 

[oomaxim]

if you run apt install qemu-server --reinstall does it work normally

No, it cannot download it.

apt install qemu-server --reinstall
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Reinstallation of qemu-server is not possible, it cannot be downloaded.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I checked the repository list and seems it as should be.

/etc/apt/sources.list

deb http://ftp.de.debian.org/debian bullseye main contrib

deb http://ftp.de.debian.org/debian bullseye-updates main contrib

# security updates
deb http://security.debian.org bullseye-security main contrib

# deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

/etc/apt/sources.list.d/pve-enterprise.list

deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

 

[oguz]

No, it cannot download it.

you can try running apt update before and try one more time.

All other nodes working perfect with these modification with out any problems.

if you're certain then you can try copying the modified files over from the working nodes and restart the services, see if it works.

 

[oomaxim]

Hi Oguz,

I enabled the no-subscription repository and this packet was successfully re-installed and all pve services back to work.

Thank you!

 

[Ernesto Martinez]

Hi, I have the same issue but doesn't work for me, Help Me please...



● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2022-02-21 08:25:43 CST; 22min ago
Process: 35847 ExecStop=/usr/bin/pveproxy stop (code=exited, status=255)
Process: 34190 ExecStart=/usr/bin/pveproxy start (code=exited, status=255)
Main PID: 33586 (code=exited, status=0/SUCCESS)
CPU: 597ms



● pvedaemon.service - PVE API Daemon
Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2022-02-21 08:25:43 CST; 22min ago
Process: 14072 ExecStop=/usr/bin/pvedaemon stop (code=exited, status=255)
Process: 34187 ExecStart=/usr/bin/pvedaemon start (code=exited, status=255)
Main PID: 22268 (code=exited, status=0/SUCCESS)
CPU: 576ms

Feb 21 08:25:43 R740 pvedaemon[34187]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 13.
Feb 21 08:25:43 R740 pvedaemon[34187]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 1
Feb 21 08:25:43 R740 pvedaemon[34187]: Compilation failed in require at /usr/share/perl5/PVE/Service/pvedaemon.
Feb 21 08:25:43 R740 pvedaemon[34187]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pvedae
Feb 21 08:25:43 R740 pvedaemon[34187]: Compilation failed in require at /usr/bin/pvedaemon line 11.
Feb 21 08:25:43 R740 pvedaemon[34187]: BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 11.
Feb 21 08:25:43 R740 systemd[1]: pvedaemon.service: Control process exited, code=exited status=255
Feb 21 08:25:43 R740 systemd[1]: Failed to start PVE API Daemon.
Feb 21 08:25:43 R740 systemd[1]: pvedaemon.service: Unit entered failed state.
Feb 21 08:25:43 R740 systemd[1]: pvedaemon.service: Failed with result 'exit-code'.




apt install pve-ha-manager --reinstall
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
pve-ha-manager : Depends: libpve-cluster-perl but it is not going to be installed
Depends: qemu-server (>= 6.0-15) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.



apt install qemu-server --reinstall
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
qemu-server : Depends: libpve-cluster-perl but it is not going to be installed
Depends: libpve-common-perl (>= 6.3-3) but 5.0-56 is to be installed
Depends: libpve-guest-common-perl (>= 3.1-3) but 2.0-20 is to be installed
Depends: libpve-storage-perl (>= 6.3-8) but 5.0-44 is to be installed
Depends: pve-ha-manager (>= 3.0-9) but 2.0-9 is to be installed
Depends: pve-qemu-kvm (>= 3.0.1-62) but 3.0.1-4 is to be installed
Breaks: pve-ha-manager (<= 3.0-4) but 2.0-9 is to be installed
E: Unable to correct problems, you have held broken packages.



cat /etc/apt/sources.list.d/pve-enterprise.list
deb http://download.proxmox.com/debian/pve buster pve-no-subscription



# apt install --reinstall $(apt list --installed | grep pve | sed 's/\/.*$//g')

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists... Done
Building dependency tree
Reading state information... Done
Reinstallation of pve-libspice-server1 is not possible, it cannot be downloaded.
Reinstallation of libcfg6 is not possible, it cannot be downloaded.
Reinstallation of tar is not possible, it cannot be downloaded.
Reinstallation of pve-kernel-4.15.18-21-pve is not possible, it cannot be downloaded.
Reinstallation of liblvm2cmd2.02 is not possible, it cannot be downloaded.
Reinstallation of liblvm2app2.2 is not possible, it cannot be downloaded.
Reinstallation of libtotem-pg5 is not possible, it cannot be downloaded.
Reinstallation of pve-kernel-4.15 is not possible, it cannot be downloaded.
Reinstallation of pve-kernel-4.15.18-10-pve is not possible, it cannot be downloaded.
Reinstallation of pve-kernel-4.15.18-14-pve is not possible, it cannot be downloaded.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
corosync : Depends: libknet1 (>= 1.18) but it is not going to be installed
dmeventd : Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
Depends: liblvm2cmd2.03 (>= 2.03.02) but it is not going to be installed
grub-common : Depends: libefiboot1 (>= 37) but 30-2 is to be installed
Depends: libefivar1 (>= 37) but 30-2 is to be installed
grub2-common : Depends: libefiboot1 (>= 37) but 30-2 is to be installed
Depends: libefivar1 (>= 37) but 30-2 is to be installed
libdevmapper1.02.1 : Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
libpve-common-perl : Depends: libproxmox-acme-perl but it is not going to be installed
libpve-http-server-perl : Depends: libanyevent-perl (>= 7.140-3) but 7.130-2 is to be installed
libpve-storage-perl : Depends: ceph-common (>= 12.2~) but 10.2.11-2+deb9u1 is to be installed
Depends: proxmox-backup-client (>= 0.8.2~) but it is not going to be installed
Depends: proxmox-backup-file-restore but it is not going to be installed
Recommends: zfs-zed but it is not going to be installed
libpve-u2f-server-perl : Depends: perl (>= 5.28.1-6) but 5.24.1-3+deb9u7 is to be installed
Depends: perlapi-5.28.1 but it is not installable
libzfs2linux : Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
libzpool2linux : Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
lvm2 : Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
lxc-pve : Depends: libc6 (>= 2.27) but 2.24-11+deb9u4 is to be installed
lxcfs : Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
pve-cluster : Depends: libsqlite3-0 (>= 3.20.0) but 3.16.2-5+deb9u3 is to be installed
Depends: perl (>= 5.28.1-6+deb10u1) but 5.24.1-3+deb9u7 is to be installed
Depends: perlapi-5.28.1 but it is not installable
pve-manager : Depends: libproxmox-acme-perl but it is not going to be installed
pve-qemu-kvm : Depends: libgfapi0 but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libgfchangelog0 but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libgfdb0 but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libgfrpc0 but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libgfxdr0 but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libglusterfs-dev but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libglusterfs0 but it is not installable or
glusterfs-common (>= 5.6) but 3.8.8-1+deb9u1 is to be installed
Depends: libiscsi7 (>= 1.18.0) but 1.17.0-1.1 is to be installed
Depends: libjemalloc2 but it is not installable
Depends: libspice-server1 (>= 0.14.2)
Depends: libusb-1.0-0 (>= 2:1.0.22) but 2:1.0.21-1 is to be installed
Depends: libc6 (>= 2.28) but 2.24-11+deb9u4 is to be installed
Depends: libcap-ng0 (>= 0.7.9) but 0.7.7-3+b1 is to be installed
Depends: libglib2.0-0 (>= 2.57.2) but 2.50.3-2+deb9u2 is to be installed
Depends: libgnutls30 (>= 3.6.6) but 3.5.8-5+deb9u6 is to be installed
Depends: libproxmox-backup-qemu0 (>= 1.0.3) but it is not going to be installed
Depends: librbd1 (>= 12.2.8) but 10.2.11-2+deb9u1 is to be installed
smartmontools : Depends: libc6 (>= 2.27) but 2.24-11+deb9u4 is to be installed
Depends: libcap-ng0 (>= 0.7.9) but 0.7.7-3+b1 is to be installed
zfsutils-linux : Depends: libnvpair3linux (>= 2.0.7) but it is not going to be installed
Depends: libuutil3linux (>= 2.0.7) but it is not going to be installed
Depends: libzfs4linux (>= 2.0.7) but it is not going to be installed
Depends: libzpool4linux (>= 2.0.7) but it is not going to be installed
Recommends: zfs-zed but it is not going to be installed
E: Unable to correct problems, you have held broken packages.