Removed proxmox host and now all of my VMs seem to have disappeared

I

iwannabfishn

New Member
Aug 13, 2024
21
1
3
I had an issue with upgrading my first pve host in the cluster. I decided to just install 9.0.3 from the ISO. As a result, I needed to manually remove the host from the cluster. Here are the steps I ran.
Code: 
1. Ran pvecm delnode (nodename) from one of the other hosts in the cluster
2. Ran "rm -rf /etc/pve/nodes/(nodename)"
3. Ran "systemclt restart pve-cluster pvedaemon pvestatd"

Now I can not see any of my VMs. They are actually still running. I can ping and connect to them. It was after I ran step 3 that I lost them in the gui. I tried running "qm list" and I get an error "user config - ignore invalid privilege 'VM.Monitor'".

Any idea how to get my VMs back. I currently have some 8.4 hosts and 2 version 9 hosts. I am not sure if that is related.

Edit: After a little while, the gui stopped working too. I am guessing there is something wrong with the clustering, but I am not sure how to troubleshoot this. SSH is still working and the VMS are still working on the network. Any help would be appreciated.
Thanks!

Edit 2: I ran to following commands and now the gui works again and the error messages are gone in the gui, but I still cannot see any of my VMs even though they are functioning. I can RDP to them, but I can't see or manage them in the proxmox gui.
 
Last edited:
dakralex said:
Hi!

Have you moved all guests from the host and powered off the host before issuing the pvecm delnode $nodename command [0]? What is the output of pvecm status and an output/excerpt of journalctl would be helpful to diagnose the issue further.

[0] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_remove_a_cluster_node
Click to expand...
Code: 
root@pve-ucs2-02:~# pvecm status
Cluster information
-------------------
Name:             PM-UCS2
Config Version:   8
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Sep  5 08:16:19 2025
Quorum provider:  corosync_votequorum
Nodes:            6
Node ID:          0x00000002
Ring ID:          2.1cc
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   6
Highest expected: 6
Total votes:      6
Quorum:           4 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 172.19.2.237 (local)
0x00000003          1 172.19.2.238
0x00000004          1 172.19.2.239
0x00000005          1 172.19.2.245
0x00000006          1 172.19.2.246
0x00000007          1 172.19.2.247

root@pve-ucs2-02:~# journalctl
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: pam_env(sshd:session): deprecated reading o>
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: Received disconnect from 172.19.37.171 port>
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: Disconnected from user root 172.19.37.171 p>
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: pam_unix(sshd:session): session closed for >
Jun 13 22:20:43 pve-ucs2-02 systemd[1]: session-49697.scope: Deactivated successfully.
Jun 13 22:20:43 pve-ucs2-02 systemd-logind[1311]: Session 49697 logged out. Waiting f>
Jun 13 22:20:43 pve-ucs2-02 systemd-logind[1311]: Removed session 49697.
Jun 13 22:20:44 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp co>
Jun 13 22:20:45 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:20:45 pve-ucs2-02 pvestatd[1746]: status update time (9.404 seconds)
Jun 13 22:20:53 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp co>
Jun 13 22:20:53 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:20:54 pve-ucs2-02 pvestatd[1746]: status update time (9.299 seconds)
Jun 13 22:21:03 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp co>
Jun 13 22:21:04 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:04 pve-ucs2-02 pvestatd[1746]: status update time (9.267 seconds)
Jun 13 22:21:14 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp co>
Jun 13 22:21:15 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:15 pve-ucs2-02 pvestatd[1746]: status update time (9.263 seconds)
Jun 13 22:21:23 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp co>
Jun 13 22:21:24 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:24 pve-ucs2-02 pvestatd[1746]: status update time (9.341 seconds)
Jun 13 22:21:33 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp co>
lines 1-23...skipping...
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: pam_env(sshd:session): deprecated reading of user environment enabled
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: Received disconnect from 172.19.37.171 port 8133:11: Connection terminated by the client.
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: Disconnected from user root 172.19.37.171 port 8133
Jun 13 22:20:43 pve-ucs2-02 sshd[788363]: pam_unix(sshd:session): session closed for user root
Jun 13 22:20:43 pve-ucs2-02 systemd[1]: session-49697.scope: Deactivated successfully.
Jun 13 22:20:43 pve-ucs2-02 systemd-logind[1311]: Session 49697 logged out. Waiting for processes to exit.
Jun 13 22:20:43 pve-ucs2-02 systemd-logind[1311]: Removed session 49697.
Jun 13 22:20:44 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:20:45 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:20:45 pve-ucs2-02 pvestatd[1746]: status update time (9.404 seconds)
Jun 13 22:20:53 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:20:53 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:20:54 pve-ucs2-02 pvestatd[1746]: status update time (9.299 seconds)
Jun 13 22:21:03 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:21:04 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:04 pve-ucs2-02 pvestatd[1746]: status update time (9.267 seconds)
Jun 13 22:21:14 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:21:15 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:15 pve-ucs2-02 pvestatd[1746]: status update time (9.263 seconds)
Jun 13 22:21:23 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:21:24 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:24 pve-ucs2-02 pvestatd[1746]: status update time (9.341 seconds)
Jun 13 22:21:33 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:21:38 pve-ucs2-02 pvestatd[1746]: VM 135 qmp command failed - VM 135 qmp command 'query-proxmox-support' failed - got timeout
Jun 13 22:21:39 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:39 pve-ucs2-02 pvestatd[1746]: status update time (14.304 seconds)
Jun 13 22:21:42 pve-ucs2-02 pmxcfs[1648]: [status] notice: received log
Jun 13 22:21:43 pve-ucs2-02 sshd[790447]: Accepted password for root from 172.19.37.171 port 8311 ssh2
Jun 13 22:21:43 pve-ucs2-02 sshd[790447]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jun 13 22:21:43 pve-ucs2-02 systemd-logind[1311]: New session 49698 of user root.
Jun 13 22:21:43 pve-ucs2-02 systemd[1]: Started session-49698.scope - Session 49698 of User root.
Jun 13 22:21:43 pve-ucs2-02 sshd[790447]: pam_env(sshd:session): deprecated reading of user environment enabled
Jun 13 22:21:43 pve-ucs2-02 sshd[790447]: Received disconnect from 172.19.37.171 port 8311:11: Connection terminated by the client.
Jun 13 22:21:43 pve-ucs2-02 sshd[790447]: Disconnected from user root 172.19.37.171 port 8311
Jun 13 22:21:43 pve-ucs2-02 sshd[790447]: pam_unix(sshd:session): session closed for user root
Jun 13 22:21:43 pve-ucs2-02 systemd[1]: session-49698.scope: Deactivated successfully.
Jun 13 22:21:43 pve-ucs2-02 systemd-logind[1311]: Session 49698 logged out. Waiting for processes to exit.
Jun 13 22:21:43 pve-ucs2-02 systemd-logind[1311]: Removed session 49698.
Jun 13 22:21:51 pve-ucs2-02 pvestatd[1746]: VM 135 qmp command failed - VM 135 qmp command 'query-proxmox-support' failed - unable to connect to VM 135 qmp socket - timeout after 51 retries
Jun 13 22:21:56 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:21:57 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:21:57 pve-ucs2-02 pmxcfs[1648]: [status] notice: received log
Jun 13 22:21:57 pve-ucs2-02 pvestatd[1746]: status update time (17.882 seconds)
Jun 13 22:22:08 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:22:13 pve-ucs2-02 pvestatd[1746]: VM 135 qmp command failed - VM 135 qmp command 'query-proxmox-support' failed - unable to connect to VM 135 qmp socket - timeout after 51 retries
Jun 13 22:22:13 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:22:15 pve-ucs2-02 pvestatd[1746]: status update time (17.325 seconds)
Jun 13 22:22:26 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:22:31 pve-ucs2-02 pvestatd[1746]: VM 135 qmp command failed - VM 135 qmp command 'query-proxmox-support' failed - unable to connect to VM 135 qmp socket - timeout after 51 retries
Jun 13 22:22:32 pve-ucs2-02 pvestatd[1746]: storage 'esxiprod2-05' is not online
Jun 13 22:22:32 pve-ucs2-02 pvestatd[1746]: status update time (17.359 seconds)
Jun 13 22:22:43 pve-ucs2-02 pvestatd[1746]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 51 retries
Jun 13 22:22:44 pve-ucs2-02 sshd[792394]: Accepted password for root from 172.19.37.171 port 8502 ssh2
Jun 13 22:22:44 pve-ucs2-02 sshd[792394]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jun 13 22:22:44 pve-ucs2-02 systemd-logind[1311]: New session 49699 of user root.
lines 1-55


You can ignore the problem connecting to the esxi storage, that server isn't online anymore. Also, I verified if I install a new VM, the VM is visible. Weird thing is all my old VMs that I can't see, they are all functioning normally. I can RDP into them and they are working. So, the host that I removed was the first host in the cluster. I am not sure if that has something to do with this. I am really hoping there is something that will scan the running hosts and rebuild those config files.
 
Sorry, I forgot to answer your first question. The host that got removed had all the VMs removed prior to me trying to upgrade it. I just can figure out why I can't see the configs on any of the other hosts that have running VMs on them.
 
Last edited:
I was looking around and I found the information on the running VMs by running the command "ps -aux |grep kvm". Now I know the information and which host they were running on. I believe that can be used to recreate the config files. I am unsure of what impact that will have on my running vm, though.
 
You must log in or register to reply here.
Top Bottom