“killall pmxcfs”How to recover from mistakenly executing this command

[Zhuy Chin]

After executing this command, the connection to the virtual machine under the PVE server is normal. Then, restart the PVE server using the reboot command. At present, we are unable to connect to the virtual machine on the server and access the page on port 8006 of the PVE platform. How should I recover

 

[Chris]

Hi,
please create a dump of the systemd journal since boot on node pve1 via journalctl -b > $(hostname)-journal.txt and attach it here. Is this node part of a cluster or a standalone node?

With systemctl reset-failed pve-cluster.service you can reset the failed counter for the service, which should allow you to restart it via systemctl restart pve-cluster.service

 

[Zhuy Chin]

Thanks for your reply！
is a standalone node，

 

[Chris]

Thanks for your reply！
is a standalone node，
View attachment 61579
View attachment 61577

The errors you show are expected if the pmxcfs is not mounted correctly at /etc/pve. These are just consequences, not the root cause.

You will have to provide the requested output to see what exactly the problem is, but from your screenshots I got the suspicion you were trying to change the hostname and might have missed some configuration changes? Please follow the steps here if that is the case https://pve.proxmox.com/wiki/Renaming_a_PVE_node

Edit: Also make sure to not pollute you mountpoint by creating files/directories there while the pmxcfs is not mounted. Seems like you dumped the journal to exactly that location. This might impede the correct mounting!

 

[Zhuy Chin]

pmxcfs the following prompt appeared:

Afterwards, I replaced the undamaged files from other PVE servers here，the following prompt appeared:

The host name has not been changed

pmxcfs,Can it be reinstalled separately?

 

[Chris]

pmxcfs the following prompt appeared:
View attachment 61581
Afterwards, I replaced the undamaged files from other PVE servers here，the following prompt appeared:
View attachment 61582
The host name has not been changed
View attachment 61588
pmxcfs,Can it be reinstalled separately?

pmxcfs is provided by the package pve-cluster. You should not copy around executables ecc. from other hosts, but rather reinstall the packages if the files provided by them might have been corrupted. Please verify the package file integrity via dpkg -V pve-cluster and/or reinstallapt install --reinstall pve-cluster.

The Unable to get local IP address error indicates that the hostname is not resolved to the IP address, so something is off. What is the output of hostname --ip-address? What exactly did you change on the host?

 

[Zhuy Chin]

Attempt to delete the cluster created in the PVE management page from the command line, no modifications were made to the IP.
Here are some commands that have been executed:

 

[rj45]

Maybe you forgot some steps in https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

Section:

Separate a Node Without Reinstalling​

 

[Chris]

Okay, so this was a cluster. Try the following:

systemctl stop pve-cluster
systemctl stop corosync
# Make sure both services are stopped via
systemctl status pve-cluster corosync
# Make sure you have no pmxcfs processes running that you started manually
killall pmxcfs
# Check the /etc/pve mountpoint is empty, if not you must move the files there to a different location
ls -l /etc/pve
# Start pmxcfs in local mode
pmxcfs -l
# Remove possibly lingering corosync config
rm /etc/pve/corosync.conf
rm -r /etc/corosync/*
# Stop pmxcfs again and finally start pve-cluster service
killall pmxcfs
systemctl start pve-cluster

Also, your IP address configured on the network interface does not match the one configured in /etc/hosts. You will have to fix that as well, even better to do this before anything else, so that you should be able to download and reinstall the pve-cluster package, which does not seem possible at the moment.

 

[Zhuy Chin]

Is this related to not being able to reinstall pve-cluster ?

 

[rj45]

reboot now

 

[Chris]

Is this related to not being able to reinstall pve-cluster ?
View attachment 61602

You did not follow the commands and comments I posted, as you can see from the output, the pmxcfs is not mounted because the mountpoint is not empty.

 

[Zhuy Chin]

Thanks for the guidance, the fault has been handled.