[SOLVED] After changing mountpoints /etc/pve empty and no web

T

taumeister

Member
Aug 16, 2022
31
5
8
49
Germany
Dear Team,
I am managing a Proxmox server for a small business (2 people) in Bonn. After reinstalling it yesterday and successfully importing all machines from the backup, everything is running smoothly again. This was standard procedure, with no issues.

This morning, I updated and corrected a few backup paths.
In the process, I made changes to already set storages [directories, mountpoints] in the file system (unmount, mount to another location, only for backups ( satabackup, smbbackup, usbbackup), which was probably not a good idea.
I suddenly received a message about a connection error in the web interface (something about RPC connections, but I'm not sure) and then I couldn't open the Proxmox website anymore.
On the CLI:
Code: 
systemctl status pveproxy:
/etc/pve/local/pve-ssl.key: failed to load...

This makes sense, as the /etc/pve folder is suddenly empty...?
The command qm list results in the following error messages:
Code: 
qm list
    ipcc_send_rec[1] failed: Connection refused
    ipcc_send_rec[2] failed: Connection refused
    ipcc_send_rec[3] failed: Connection refused
    Unable to load access control list: Connection refused

The virtual machines are currently running, but I am hesitant to restart the pve-cluster service or the host itself.
Additionaly I see some mysterious firewall alarms in pve-firewall.service.

Can you help me with how to proceed?
Thanks a lot in advance, I appreciate any time you´ll spend

/etc/hosts:
Code: 
127.0.0.1 localhost.localdomain localhost
192.168.1.1 pve.sp.intern pve

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

systemctl status pve*

Code: 
● pvenetcommit.service - Commit Proxmox VE network changes
     Loaded: loaded (/lib/systemd/system/pvenetcommit.service; enabled; preset: enabled)
     Active: active (exited) since Sun 2023-12-03 13:09:46 CET; 22h ago
   Main PID: 1329 (code=exited, status=0/SUCCESS)
        CPU: 1ms

Dec 03 13:09:46 pve systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes...
Dec 03 13:09:46 pve systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes.

● pve-firewall.service - Proxmox VE firewall
     Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-12-03 13:09:49 CET; 22h ago
   Main PID: 1758 (pve-firewall)
      Tasks: 1 (limit: 76002)
     Memory: 37.4M
        CPU: 6min 16.028s
     CGroup: /system.slice/pve-firewall.service
             └─1758 pve-firewall

Dec 04 11:34:11 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:34:21 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:34:31 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:34:41 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:34:51 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:35:01 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:35:11 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:35:21 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:35:31 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:35:41 pve pve-firewall[1758]: status update error: Connection refused

● pve-storage.target - PVE Storage Target
     Loaded: loaded (/lib/systemd/system/pve-storage.target; static)
     Active: active since Sun 2023-12-03 13:09:47 CET; 22h ago

Dec 03 13:09:47 pve systemd[1]: Reached target pve-storage.target - PVE Storage Target.

● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-12-03 13:09:48 CET; 22h ago
   Main PID: 1657 (pmxcfs)
      Tasks: 2 (limit: 76002)
     Memory: 10.0M

journalct -xe:
Code: 
Dec 04 11:21:28 pve pveproxy[327527]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:31 pve pve-firewall[1758]: status update error: Connection refused
Dec 04 11:21:31 pve pvestatd[1760]: ipcc_send_rec[1] failed: Connection refused
Dec 04 11:21:31 pve pvestatd[1760]: ipcc_send_rec[2] failed: Connection refused
Dec 04 11:21:31 pve pvestatd[1760]: ipcc_send_rec[3] failed: Connection refused
Dec 04 11:21:31 pve pvestatd[1760]: ipcc_send_rec[4] failed: Connection refused
Dec 04 11:21:31 pve pvestatd[1760]: status update error: Connection refused
Dec 04 11:21:33 pve pveproxy[327525]: worker exit
Dec 04 11:21:33 pve pveproxy[327526]: worker exit
Dec 04 11:21:33 pve pveproxy[325197]: worker 327525 finished
Dec 04 11:21:33 pve pveproxy[325197]: worker 327526 finished
Dec 04 11:21:33 pve pveproxy[325197]: starting 2 worker(s)
Dec 04 11:21:33 pve pveproxy[325197]: worker 327536 started
Dec 04 11:21:33 pve pveproxy[325197]: worker 327537 started
Dec 04 11:21:33 pve pveproxy[327527]: worker exit
Dec 04 11:21:33 pve pveproxy[327536]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:33 pve pveproxy[327537]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:33 pve pveproxy[325197]: worker 327527 finished
Dec 04 11:21:33 pve pveproxy[325197]: starting 1 worker(s)
Dec 04 11:21:33 pve pveproxy[325197]: worker 327538 started
Dec 04 11:21:33 pve pveproxy[327538]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:38 pve pveproxy[327536]: worker exit
Dec 04 11:21:38 pve pveproxy[327537]: worker exit
Dec 04 11:21:38 pve pveproxy[325197]: worker 327536 finished
Dec 04 11:21:38 pve pveproxy[325197]: starting 1 worker(s)
Dec 04 11:21:38 pve pveproxy[325197]: worker 327537 finished
Dec 04 11:21:38 pve pveproxy[325197]: worker 327544 started
Dec 04 11:21:38 pve pveproxy[327538]: worker exit
Dec 04 11:21:38 pve pveproxy[327544]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:38 pve pveproxy[325197]: worker 327538 finished
Dec 04 11:21:38 pve pveproxy[325197]: starting 2 worker(s)
Dec 04 11:21:38 pve pveproxy[325197]: worker 327545 started
Dec 04 11:21:38 pve pveproxy[325197]: worker 327546 started
Dec 04 11:21:38 pve pveproxy[327545]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:38 pve pveproxy[327546]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Dec 04 11:21:41 pve pve-firewall[1758]: status update error: Connection refused
 
Last edited:
taumeister said:
This makes sense, as the /etc/pve folder is suddenly empty...?
Click to expand...
Hi,
could it be that you unmounted the pmxcfs on /etc/pve, is it listed in the output of mount? Is there anything in the systemd journal for the journalctl -u pve-cluster.service? Try to restart the service by running systemctl restart pve-cluster.service
 
Hi Chris,
thanks for your reply.

Yes, that seems logical to me, as I executed umount -a. The command journalctl -xeu pve-cluster shows:

Code: 
systemctl -xeu pve-cluster
[main] notice: teardown filesystem


could this be the issue?
Can I simply remount into /etc/pve?
Are you certain that I can simply restart this service during operation?
Will the VMs lose their connection, or is it safe to do so?
 
taumeister said:
Hi Chris,
thanks for your reply.

Yes, that seems logical to me, as I executed umount -a. The command journalctl -xeu pve-cluster shows:

Code: 
systemctl -xeu pve-cluster
[main] notice: teardown filesystem


could this be the issue?
Can I simply remount into /etc/pve?
Are you certain that I can simply restart this service during operation?
Will the VMs lose their connection, or is it safe to do so?
Click to expand...
Okay, yes this explains why the pmxcfs got unmounted. Best is to restart pve-cluster service. As this is not a cluster node and the pmxcfs is already unmounted anyway, a restart in this state should not influence other services any further.

If unsure, and you don't want to risk operation you can do this in a maintenance window.
 
Ok, Chris,
super fast responses, thanks for that.

I will do so in the maintenance after 18:00 O´clock and will come back if there is any other problem.
Thanks for your help
Greetings
Thomas
 
just to make sure:

you should probably never run umount -a on a production system!

the only time that makes sense is as part of shutting down, or something like manually setting up/fixing a system using a livecd..
 
  • Like
Reactions: Chris
taumeister said:
Ok, Chris,
super fast responses, thanks for that.

I will do so in the maintenance after 18:00 O´clock and will come back if there is any other problem.
Thanks for your help
Greetings
Thomas
Click to expand...
In that case it is adviced to perform a full reboot, as other important filesystems might have been unmouned as well. The reboot should bring you back to a clean mount state.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom