Problem with web interface Proxmox

[MAX&GURGEN]

Hi! We have some problems w web interface. it isnt working after deleting cluster group and change hostname one of the our proxmox not well. There is some logs with errors.
Dec 19 08:39:30 pve00 pveproxy[2593]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
root@pve00:~# systemctl status -l pve-cluster × pve-cluster.service - The Proxmox VE cluster filesystem Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Tue 2023-12-19 08:47:44 +04; 10min ago Process: 3007 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION) CPU: 13ms Dec 19 08:47:44 pve00 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5. Dec 19 08:47:44 pve00 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem. Dec 19 08:47:44 pve00 systemd[1]: pve-cluster.service: Start request repeated too quickly. Dec 19 08:47:44 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. Dec 19 08:47:44 pve00 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

journalctl -u pve-cluster
Dec 18 08:46:24 pve pmxcfs[824157]: [quorum] crit: quorum_initialize failed: 2 Dec 18 08:46:24 pve pmxcfs[824157]: [quorum] crit: can't initialize service Dec 18 08:46:24 pve pmxcfs[824157]: [confdb] crit: cmap_initialize failed: 2 Dec 18 08:46:24 pve pmxcfs[824157]: [confdb] crit: can't initialize service Dec 18 08:46:24 pve pmxcfs[824157]: [dcdb] crit: cpg_initialize failed: 2 Dec 18 08:46:24 pve pmxcfs[824157]: [dcdb] crit: can't initialize service Dec 18 08:46:24 pve pmxcfs[824157]: [status] crit: cpg_initialize failed: 2 Dec 18 08:46:24 pve pmxcfs[824157]: [status] crit: can't initialize service Dec 18 10:09:54 pve00 pmxcfs[1153]: [main] crit: Unable to resolve node name 'pve00' to a non-loopback IP address - missing entry in '/etc/hosts' or DNS? Dec 18 10:26:39 pve00 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION Dec 18 10:26:39 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. ec 18 14:45:42 pve00 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem. Dec 18 14:45:42 pve00 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem... Dec 18 14:45:42 pve00 pmxcfs[2894]: [main] notice: resolved node name 'pve00' to '192.168.69.55' for default node IP address Dec 18 14:45:42 pve00 pmxcfs[2894]: [main] notice: resolved node name 'pve00' to '192.168.69.55' for default node IP address Dec 18 14:45:42 pve00 pmxcfs[2894]: fuse: mountpoint is not empty Dec 18 14:45:42 pve00 pmxcfs[2894]: fuse: if you are sure this is safe, use the 'nonempty' mount option Dec 18 14:45:42 pve00 pmxcfs[2894]: [main] crit: fuse_mount error: File exists Dec 18 14:45:42 pve00 pmxcfs[2894]: [main] crit: fuse_mount error: File exists Dec 18 14:45:42 pve00 pmxcfs[2894]: [main] notice: exit proxmox configuration filesystem (-1) Dec 18 14:45:42 pve00 pmxcfs[2894]: [main] notice: exit proxmox configuration filesystem (-1) Dec 18 14:45:42 pve00 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION Dec 18 14:45:42 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. Dec 18 14:45:42 pve00 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem. Dec 18 14:45:42 pve00 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5. Dec 18 14:45:42 pve00 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem. Dec 18 14:45:42 pve00 systemd[1]: pve-cluster.service: Start request repeated too quickly. Dec 18 14:45:42 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. Dec 18 14:45:42 pve00 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
/etc/hosts
#127.0.0.1 localhost.localdomain localhost 192.168.69.55 pve00.servtest pve00 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

hostname - pve00

root@pve00:~# cat /etc/postfix/main.cf # See /usr/share/postfix/main.cf.dist for a commented, more complete version myhostname=pve00.servtest smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) biff = no # appending .domain is the MUA's job. append_dot_mydomain = no # Uncomment the next line to generate "delayed mail" warnings #delay_warning_time = 4h alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases mydestination = $myhostname, localhost.$mydomain, localhost relayhost = mynetworks = 127.0.0.0/8 inet_interfaces = loopback-only recipient_delimiter = + compatibility_level = 2

TY for answering!

 

[Folke]

Hi,
unfortunately, more than your web interface seems to be broken. It seems like the cluster fs fails to mount, because there is already something in the mount target. Could you backup the content of /etc/pve/, empty the folder and try to start pve-cluster.service again?

 

[MAX&GURGEN]

ty for anwsering, there is still same error, we delete all from /etc/pve/ root@pve00:~# ^C root@pve00:~# cd /etc/pve/ root@pve00:/etc/pve# ls root@pve00:/etc/pve#
and try to start pve-cluster.service same problem root@pve00:~# systemctl status pve-cluster.service × pve-cluster.service - The Proxmox VE cluster filesystem Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Tue 2023-12-19 13:41:03 +04; 10min ago Process: 1294 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION) CPU: 16ms Dec 19 13:41:03 pve00 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5. Dec 19 13:41:03 pve00 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem. Dec 19 13:41:03 pve00 systemd[1]: pve-cluster.service: Start request repeated too quickly. Dec 19 13:41:03 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. Dec 19 13:41:03 pve00 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem. root@pve00:~# systemctl start pve-cluster.service Job for pve-cluster.service failed because the control process exited with error code. See "systemctl status pve-cluster.service" and "journalctl -xeu pve-cluster.service" for details. root@pve00:~# systemctl restart pve-cluster.service Job for pve-cluster.service failed because the control process exited with error code. See "systemctl status pve-cluster.service" and "journalctl -xeu pve-cluster.service" for details.
jornalctl wrote this Dec 19 13:51:41 pve00 pmxcfs[1976]: [main] notice: exit proxmox configuration filesystem (-1) Dec 19 13:51:41 pve00 systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ An ExecStart= process belonging to unit pve-cluster.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 255. Dec 19 13:51:41 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ The unit pve-cluster.service has entered the 'failed' state with result 'exit-code'. Dec 19 13:51:41 pve00 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem. ░░ Subject: A start job for unit pve-cluster.service has failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit pve-cluster.service has finished with a failure. ░░ ░░ The job identifier is 3223 and the job result is failed. Dec 19 13:51:41 pve00 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5. ░░ Subject: Automatic restarting of a unit has been scheduled ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ Automatic restarting of the unit pve-cluster.service has been scheduled, as the result for ░░ the configured Restart= setting for the unit. Dec 19 13:51:41 pve00 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem. ░░ Subject: A stop job for unit pve-cluster.service has finished ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A stop job for unit pve-cluster.service has finished. ░░ ░░ The job identifier is 3417 and the job result is done. Dec 19 13:51:41 pve00 systemd[1]: pve-cluster.service: Start request repeated too quickly. Dec 19 13:51:41 pve00 systemd[1]: pve-cluster.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ The unit pve-cluster.service has entered the 'failed' state with result 'exit-code'. Dec 19 13:51:41 pve00 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem. ░░ Subject: A start job for unit pve-cluster.service has failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit pve-cluster.service has finished with a failure. ░░ ░░ The job identifier is 3417 and the job result is failed.

 

[Folke]

Could you manually run pmxcfs -fd? It's the executable started by pve-cluster in debug mode.

 

[MAX&GURGEN]

can u please help us with the problem. we are students can't understand what is it mean. Can u navigate us
root@pve00:~# pmxcfs -fd [main] notice: resolved node name 'pve00' to '192.168.69.55' for default node IP address (pmxcfs.c:859:main) [database] debug: name __version__ (inode = 0000000000000000, parent = 0000000000000000) (database.c:370:bdb_backend_load_index) [main] debug: memdb open '/var/lib/pve-cluster/config.db' successful (version = 0000000000000001) (memdb.c:536:memdb_open) [main] debug: using local mode (corosync.conf does not exist) (pmxcfs.c:931:main) fuse: mountpoint is not empty fuse: if you are sure this is safe, use the 'nonempty' mount option [main] crit: fuse_mount error: File exists (pmxcfs.c:954:main) [main] notice: exit proxmox configuration filesystem (-1) (pmxcfs.c:1109:main)

 

[Folke]

Two problems, 1) it seems that the database backing pmxcfs is empty, which should not be the case and 2) something is still preventing the fuse filesystem from mounting.

At this point, if possible, I would definitely recommend reinstalling proxmox.

To understand how you got into this situation, could you tell me what you did to remove the node from the cluster and rename it?

 

[wbravin]

Hello all

I believe i have a major issue

It started with me not being able to reach my proxmox server once booted,

I thought that it was the NIC that had the issue because i could not see that the nic light were on. I replaced the NIC from my dell 720

Now when i boot, the nic is active when the server starts,. however when i get the proxmox screen to select the proxmox boot, the NIC looses connection.

When Proxmox starts I get an IP address however i cannot access this address from another pc.

I will need to reinstall proxmox OK


I did the following

root@pve00:~# ^Cr
oot@pve00:~# cd /etc/pve/
root@pve00:/etc/pve# ls

I see files in this folder should i backup these files to a usb?
Are there other files i should copy to usb?

This should copy all VM and critical date to the USB

Once this is donne i would reinstall proxmox from scratch correct
I can live with reinstalling opnsense and truenas scale vms however i will need to save if possible the Home assistant VM and its configuration

Thank you for your help