Power failure and now GUI doesn't work

E

Enrique Garcia Polo

Well-Known Member
Jun 29, 2018
37
1
48
43
Hi everybody
I have a Proxmox 8.3.3 node working well until a powerfailure. When I have recovered the power I got an error on Boot, I use fsck to fix inodes and continue booting but GUI.
Error "Failed to start pveproxy.service" al boot

If I do journalctl -xeu pveproxy.service

Feb 20 23:18:49 pve systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
░░ Subject: A start job for unit pveproxy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has begun execution.
░░
░░ The job identifier is 1381.
Feb 20 23:18:50 pve pveproxy[2558]: Can't locate Template.pm in @INC (you may need to install the Template module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_>
Feb 20 23:18:50 pve pveproxy[2558]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pveproxy.pm line 9.
Feb 20 23:18:50 pve pveproxy[2558]: Compilation failed in require at /usr/bin/pveproxy line 11.
Feb 20 23:18:50 pve pveproxy[2558]: BEGIN failed--compilation aborted at /usr/bin/pveproxy line 11.
Feb 20 23:18:50 pve systemd[1]: pveproxy.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit pveproxy.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 2.
Feb 20 23:18:50 pve systemd[1]: pveproxy.service: Failed with result 'exit-code'.

The vms work well but I havn't got WEB GUI

Any idea?
 
Hello Enrique! Could you please try to run apt reinstall pve-manager?

If that doesn't help, please post the output of pveversion -v, as this command should also show if some other packages are not correctly installed.
 
  • Like
Reactions: Kingneutron
l.leahu-vladucu said:
Hello Enrique! Could you please try to run apt reinstall pve-manager?

If that doesn't help, please post the output of pveversion -v, as this command should also show if some other packages are not correctly installed.
Click to expand...
Thank you l.leahu-vladucu


root@pve:~# apt reinstall pve-manager
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 549 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://download.proxmox.com/debian bookworm/pve-no-subscription amd64 pve-manager all 8.3.4 [549 kB]
Fetched 549 kB in 1s (740 kB/s)
dpkg: warning: files list file for package 'libavfilter8:amd64' missing; assuming package has no files currently installed
(Reading database ... 95919 files and directories currently installed.)
Preparing to unpack .../pve-manager_8.3.4_all.deb ...
Unpacking pve-manager (8.3.4) over (8.3.4) ...
Setting up pve-manager (8.3.4) ...
Job for pveproxy.service failed because the control process exited with error code.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.
Processing triggers for man-db (2.11.2-2) ...

-------------------------------------------


root@pve:~# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-8-pve)
pve-manager: 8.3.4 (running version: 8.3.4/65224a0f9cd294a3)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-8
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-7-pve-signed: 6.8.12-7
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
dnsmasq: 2.90-4~deb12u1
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.2.0
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
openvswitch-switch: 3.1.0-2+deb12u1
proxmox-backup-client: 3.3.3-1
proxmox-backup-file-restore: 3.3.3-1
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.4
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.3.3
pve-qemu-kvm: 9.0.2-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve1
 
Do you still get the same error message as before when calling journalctl -xeu pveproxy.service? Please also try to do apt reinstall libavfilter8.
 
I strongly recommend investing in a UPS. And setup NUT client.

https://www.thesmarthomebook.com/2022/09/02/setting-up-monitor-your-ups-proxmox-home-assistant/

.

If all else fails, backup your config, reinstall PVE ( NOTE the target disk for boot/root will be wiped! ) and restore LXC/VMs from backup (if you don't have it yet, setup PBS on separate hardware and take advantage of dedup)

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, point it to external disk / NAS, run it nightly in cron.
 
l.leahu-vladucu said:
Do you still get the same error message as before when calling journalctl -xeu pveproxy.service? Please also try to do apt reinstall libavfilter8.
Click to expand...
The problem remainds:

root@pve:~# apt reinstall libavfilter8
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/3,706 kB of archives.
After this operation, 0 B of additional disk space will be used.
dpkg: warning: files list file for package 'libavfilter8:amd64' missing; assuming package has no files currently installed
(Reading database ... 95919 files and directories currently installed.)
Preparing to unpack .../libavfilter8_7%3a5.1.6-0+deb12u1_amd64.deb ...
Unpacking libavfilter8:amd64 (7:5.1.6-0+deb12u1) over (7:5.1.6-0+deb12u1) ...
Setting up libavfilter8:amd64 (7:5.1.6-0+deb12u1) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
root@pve:~# systemctl start pveproxy.service
Job for pveproxy.service failed because the control process exited with error code.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.


root@pve:~# systemctl status pveproxy.service
× pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Fri 2025-02-21 18:51:11 CET; 49s ago
Process: 93171 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 93173 ExecStart=/usr/bin/pveproxy start (code=exited, status=2)
CPU: 453ms

Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 5.
Feb 21 18:51:11 pve systemd[1]: Stopped pveproxy.service - PVE API Proxy Server.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Start request repeated too quickly.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Failed with result 'exit-code'.
Feb 21 18:51:11 pve systemd[1]: Failed to start pveproxy.service - PVE API Proxy Server.


root@pve:~# journalctl -xeu pveproxy.service
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has begun execution.
░░
░░ The job identifier is 5340.
Feb 21 18:51:11 pve pveproxy[93173]: Can't locate Template.pm in @INC (you may need to install the Template module) (@INC co>
Feb 21 18:51:11 pve pveproxy[93173]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pveproxy.pm line 9.
Feb 21 18:51:11 pve pveproxy[93173]: Compilation failed in require at /usr/bin/pveproxy line 11.
Feb 21 18:51:11 pve pveproxy[93173]: BEGIN failed--compilation aborted at /usr/bin/pveproxy line 11.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit pveproxy.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 2.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit pveproxy.service has entered the 'failed' state with result 'exit-code'.
Feb 21 18:51:11 pve systemd[1]: Failed to start pveproxy.service - PVE API Proxy Server.
░░ Subject: A start job for unit pveproxy.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has finished with a failure.
░░
░░ The job identifier is 5340 and the job result is failed.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Automatic restarting of the unit pveproxy.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Feb 21 18:51:11 pve systemd[1]: Stopped pveproxy.service - PVE API Proxy Server.
░░ Subject: A stop job for unit pveproxy.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit pveproxy.service has finished.
░░
░░ The job identifier is 5459 and the job result is done.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Start request repeated too quickly.
Feb 21 18:51:11 pve systemd[1]: pveproxy.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit pveproxy.service has entered the 'failed' state with result 'exit-code'.
Feb 21 18:51:11 pve systemd[1]: Failed to start pveproxy.service - PVE API Proxy Server.
░░ Subject: A start job for unit pveproxy.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has finished with a failure.
░░
░░ The job identifier is 5459 and the job result is failed.
root@pve:~#
 
Last edited:
Kingneutron said:
I strongly recommend investing in a UPS. And setup NUT client.

https://www.thesmarthomebook.com/2022/09/02/setting-up-monitor-your-ups-proxmox-home-assistant/

.

If all else fails, backup your config, reinstall PVE ( NOTE the target disk for boot/root will be wiped! ) and restore LXC/VMs from backup (if you don't have it yet, setup PBS on separate hardware and take advantage of dedup)

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, point it to external disk / NAS, run it nightly in cron.
Click to expand...
Tanks,

I have an APC BACK-UPS 950 and it shutdown the server but when I connect the power the UPS was empty and the power fail again without shutdown system.

I also have a XPenology in differnet hardware but connected to the same UPS and It it broken too, DSM corrupted but data is OK I checked booting the Xpenology with live linux and the data is there.
 
alexskysilk said:
what does systemctl status pvestatd show? also, whats the state of your server (any hung mounts, other errors in journalctl, etc)
Click to expand...
Hi AlexSkySilk,
The pvestatd seems OK

root@pve:~# systemctl status pvestatd
● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
Active: active (running) since Sun 2025-02-23 13:47:25 CET; 44s ago
Process: 2138 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 2161 (pvestatd)
Tasks: 1 (limit: 19044)
Memory: 150.8M
CPU: 2.862s
CGroup: /system.slice/pvestatd.service
└─2161 pvestatd

Feb 23 13:47:24 pve systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Feb 23 13:47:25 pve pvestatd[2161]: starting server
Feb 23 13:47:25 pve systemd[1]: Started pvestatd.service - PVE Status Daemon.



I haven't noticed any additional problems other than the GUI
 
Ah, my bad, Template.pm is part of libtemplate-perl. Could you please try apt reinstall libtemplate-perl? This should finally fix the issue, at least as long as there are no further errors in other dependencies.
 
  • Like
Reactions: Enrique Garcia Polo
l.leahu-vladucu said:
Ah, my bad, Template.pm is part of libtemplate-perl. Could you please try apt reinstall libtemplate-perl? This should finally fix the issue, at least as long as there are no further errors in other dependencies.
Click to expand...

That works for my GUI, Thanks a lot!

But, I'm experimenting hungs when I try to update the server:

Bash: 
root@pve:~# apt dist-upgrade
Segmentation faultsts... 0%
root@pve:~# apt update
Hit:1 http://security.debian.org bookworm-security InRelease
Hit:2 http://ftp.es.debian.org/debian bookworm InRelease                                 
Get:3 http://download.proxmox.com/debian bookworm InRelease [2,768 B]                   
Fetched 2,768 B in 0s (6,803 B/s)   
Reading package lists... 0%







Bash: 
Feb 24 19:18:51 pve kernel: EXT4-fs (dm-6): error count since last fsck: 1
Feb 24 19:18:51 pve kernel: EXT4-fs (dm-6): initial error at time 1740066897: ext4_validate_block_bitmap:423
Feb 24 19:18:51 pve kernel: EXT4-fs (dm-6): last error at time 1740066897: ext4_validate_block_bitmap:423

So I think that my filesystem is broken
 
It seems your filesystem is more corrupted than I thought.
Enrique Garcia Polo said:
I use fsck to fix inodes
Click to expand...
May I ask how you did this, exactly? Which command(s) did you use? Did you unmount the filesystem before?

It is very important to unmount the filesystem before running fsck. For example, you could run any live Linux distribution and do the following:
Bash: 
fsck -pvcf

The command above will enable verbose mode (-v), force a file check (-f), try to find bad blocks (-c) and attempt an automatic repair if possible (-p). See man e2fsck (or the online version) for more information.

However, in the worst case, the filesystem might be too corrupted to repair it. But it's worth trying.
 
l.leahu-vladucu said:
It seems your filesystem is more corrupted than I thought.

May I ask how you did this, exactly? Which command(s) did you use? Did you unmount the filesystem before?

It is very important to unmount the filesystem before running fsck. For example, you could run any live Linux distribution and do the following:
Bash: 
fsck -pvcf

The command above will enable verbose mode (-v), force a file check (-f), try to find bad blocks (-c) and attempt an automatic repair if possible (-p). See man e2fsck (or the online version) for more information.

However, in the worst case, the filesystem might be too corrupted to repair it. But it's worth trying.
Click to expand...
I repaired the filesystem the first boot after power failure, when it was booting, there was a i/o error and ask me for repair filesystem and I acepted, it was correcting inodes and finally booted.

Yesterday when i get the segmentation error, I shutdown the system and I booted with a linux mint live CD. Then I passed check disk with the disk utility to every system logical volume and it doesn't locate any error, I get a clean system so... I don't know how to repair it.

The Vms are migrated to a miniPC with proxmox and now I was thinking about reinstall Proxmox on my HP Proliant Microserver Gen8 but I'd like to make a backup of main configuration. Is there a procedure for this or I have to copy the files individually like interfaces, fstab, etc...?
 
It seems the filesystem is too corrupted, I'm afraid. The fact that it doesn't find errors at least means that the disks are probably not corrupted otherwise (e.g. bad blocks), so at least that's good. While I don't expect the disks themselves to have issues, it might also be a good idea to check whether S.M.A.R.T. reports any errors (smartctl -a /dev/DISKNAME), just in case.

Enrique Garcia Polo said:
The Vms are migrated to a miniPC with proxmox and now I was thinking about reinstall Proxmox on my HP Proliant Microserver Gen8 but I'd like to make a backup of main configuration. Is there a procedure for this or I have to copy the files individually like interfaces, fstab, etc...?
Click to expand...
There is no automatic procedure for backing up configuration files, so you'll have to do that manually. Proxmox VE builds on top of Debian, so you'll need to backup both PVE and Debian configuration files. This effectively means you'll need to backup the following:
  1. The Proxmox VE configuration files, which are in /etc/pve.
  2. Any other configuration files from /etc that you want to restore on the newly installed machine (e.g. /etc/network/interfaces for the network configuration).
  3. VMs and LXC containers and disks using vzdump. I'm just writing this for the sake of completeness, because you wrote that you already migrated those.
Last but not least, while it is too late in your case, I also want to mention that to prevent such issues from happening in the future, it's a good idea to invest in datacenter SSDs which have power-loss protection.
 
Last edited:
You must log in or register to reply here.
Top Bottom