/etc/pve disappeared after an update

[rgfincher]

Hi, upgraded our "proxmox3" server last night, ( which is used only to run openVZ VMs) and all my virtual containers disappeared. This turned out to be caused by the entire /etc/pve directory disappearing, comparing it with our working "proxmox5" server, which was already the latest version, I saw that this directory is a "fuse" volume. I found that typing "pmxcfs" restored the missing directory, but couldn't find in the boot sequence where this is run, even by comparing to my good server. The problem server will work at the moment, but I have to run pmxcfs manually after each reboot, and then start all the containers manually one by one. We are not using clustering at this time.

We have community support subscriptions for all our servers.

Many thanks for any input.


Sent from my iPad using Tapatalk

 

[rgfincher]

Hi, anyone have any ideas on this?
I would like to hear from anyone who's had this problem before.

 

[dietmar]

Does it help if you reboot the node? Or restart pve services?

# service pve-cluster restart
# service pvedaemon restart
# service pveproxy restart
# service pvestatd restart

 

[rgfincher]

Hi, no the problem continues after rebooting the node.
I also found a lock file which I thought shouldn't be there and removed it, but that didn't help.
It does seem to fail to start the 'cluster' process, which I assume is related.

 

[dietmar]

It does seem to fail to start the 'cluster' process, which I assume is related.

What is the output of

# service pve-cluster restart

any further messages in /var/log/syslog?

 

[rgfincher]

Hi, it restarts fine using the command you specified.
i think the issue is the order in which the boot up sequence attempts to start the various services.

 

[rgfincher]

Apr 11 20:22:02 proxmox3 pmxcfs[3878]: [main] notice: teardown filesystem
Apr 11 20:22:08 proxmox3 pvestatd[3784]: WARNING: ipcc_send_rec failed: Transport endpoint is not connected
Apr 11 20:22:08 proxmox3 pvestatd[3784]: WARNING: ipcc_send_rec failed: Connection refused
Apr 11 20:22:08 proxmox3 pvestatd[3784]: WARNING: ipcc_send_rec failed: Connection refused
Apr 11 20:22:08 proxmox3 pvestatd[3784]: WARNING: ipcc_send_rec failed: Connection refused
Apr 11 20:22:08 proxmox3 pvestatd[3784]: WARNING: ipcc_send_rec failed: Connection refused
Apr 11 20:22:08 proxmox3 pvestatd[3784]: WARNING: ipcc_send_rec failed: Connection refused
Apr 11 20:22:14 proxmox3 pmxcfs[116429]: [status] notice: update cluster info (cluster name PVE21cluster, version = 3)
Apr 11 20:22:14 proxmox3 pmxcfs[116429]: [status] notice: node has quorum
Apr 11 20:22:14 proxmox3 pmxcfs[116429]: [dcdb] notice: members: 1/116429
Apr 11 20:22:14 proxmox3 pmxcfs[116429]: [dcdb] notice: all data is up to date
Apr 11 20:22:14 proxmox3 pmxcfs[116429]: [dcdb] notice: members: 1/116429
Apr 11 20:22:14 proxmox3 pmxcfs[116429]: [dcdb] notice: all data is up to date