[SOLVED] Recovering VM configuration

[Mex]

Two proxmox servers (pve1 and pve2) pve1 - v4.4, pve2 - v5.0
Pve1 - cluster
Pve1 - node

On pve2, there were several virtual machines in the ZFS repository.

After adding the node to the pve1 cluster (pvecm add pve1 command), errors occurred:

root@pve:/var/log# pvecm add pve1 --force
can't create shared ssh key database '/etc/pve/priv/authorized_keys'
node pve already defined
copy corosync auth key
stopping pve-cluster service
backup old database
Job for corosync.service failed. See 'systemctl status corosync.service' and 'journalctl -xn' for details.
waiting for quorum...

systemctl status corosync.service:

root@pve:/var/log# systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: failed (Result: exit-code) since Sat 2017-08-19 19:19:53 MSK; 23min ago
Process: 4362 ExecStart=/usr/share/corosync/corosync start (code=exited, status=1/FAILURE)

Aug 19 19:18:53 pve corosync[4371]: [QB ] server name: quorum
Aug 19 19:18:53 pve corosync[4371]: [TOTEM ] JOIN or LEAVE message was thrown away during flush operation.
Aug 19 19:18:53 pve corosync[4371]: [TOTEM ] A new membership (192.168.25.11:88) was formed. Members joined: 1
Aug 19 19:18:53 pve corosync[4371]: [QUORUM] Members[1]: 1
Aug 19 19:18:53 pve corosync[4371]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 19 19:18:53 pve corosync[4371]: [TOTEM ] A new membership (192.168.25.10:92) was formed. Members joined: 1
Aug 19 19:19:53 pve corosync[4362]: Starting Corosync Cluster Engine (corosync): [FAILED]
Aug 19 19:19:53 pve systemd[1]: corosync.service: control process exited, code=exited status=1
Aug 19 19:19:53 pve systemd[1]: Failed to start Corosync Cluster Engine.
Aug 19 19:19:53 pve systemd[1]: Unit corosync.service entered failed state.

syslog pve1:

Aug 19 19:18:53 pve1 corosync[1650]: notice [TOTEM ] A new membership (192.168.25.10:92) was formed. Members joined: 1
Aug 19 19:18:53 pve1 corosync[1650]: [TOTEM ] A new membership (192.168.25.10:92) was formed. Members joined: 1
Aug 19 19:18:54 pve1 corosync[1650]: notice [TOTEM ] A processor failed, forming new configuration.
Aug 19 19:18:54 pve1 corosync[1650]: [TOTEM ] A processor failed, forming new configuration.
Aug 19 19:18:54 pve1 pmxcfs[1641]: [status] notice: cpg_send_message retry 10
Aug 19 19:18:55 pve1 pmxcfs[1641]: [status] notice: cpg_send_message retry 20
Aug 19 19:18:55 pve1 corosync[1650]: notice [TOTEM ] A new membership (192.168.25.10:96) was formed. Members left: 1
Aug 19 19:18:55 pve1 corosync[1650]: notice [TOTEM ] Failed to receive the leave message. failed: 1
Aug 19 19:18:55 pve1 corosync[1650]: [TOTEM ] A new membership (192.168.25.10:96) was formed. Members left: 1
Aug 19 19:18:55 pve1 corosync[1650]: notice [QUORUM] Members[1]: 1
Aug 19 19:18:55 pve1 corosync[1650]: notice [MAIN ] Completed service synchronization, ready to provide service.
Aug 19 19:18:55 pve1 corosync[1650]: [TOTEM ] Failed to receive the leave message. failed: 1
Aug 19 19:18:55 pve1 corosync[1650]: [QUORUM] Members[1]: 1
Aug 19 19:18:55 pve1 corosync[1650]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 19 19:18:55 pve1 pmxcfs[1641]: [status] notice: cpg_send_message retried 21 times
Aug 19 19:19:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Aug 19 19:19:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Aug 19 19:19:58 pve1 pveproxy[7903]: worker exit
Aug 19 19:19:58 pve1 pveproxy[1679]: worker 7903 finished
Aug 19 19:19:58 pve1 pveproxy[1679]: starting 1 worker(s)
Aug 19 19:19:58 pve1 pveproxy[1679]: worker 10760 started

syslog pve2:

Aug 19 19:18:49 pve2 pmxcfs[3136]: [main] notice: teardown filesystem
Aug 19 19:18:49 pve2 systemd[1]: Stopping The Proxmox VE cluster filesystem...
Aug 19 19:18:50 pve2 systemd[1]: Starting Cleanup of Temporary Directories...
Aug 19 19:18:50 pve2 systemd[1]: Started Cleanup of Temporary Directories.
Aug 19 19:18:51 pve2 pmxcfs[3136]: [quorum] crit: quorum_finalize failed: 9
Aug 19 19:18:51 pve2 pmxcfs[3136]: [confdb] crit: cmap_finalize failed: 9
Aug 19 19:18:51 pve2 pve-ha-crm[2584]: ipcc_send_rec failed: Transport endpoint is not connected
Aug 19 19:18:51 pve2 pve-ha-crm[2584]: ipcc_send_rec failed: Connection refused
Aug 19 19:18:51 pve2 pve-ha-crm[2584]: ipcc_send_rec failed: Connection refused
Aug 19 19:18:52 pve2 pmxcfs[3136]: [main] notice: exit proxmox configuration filesystem (0)
Aug 19 19:18:52 pve2 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Aug 19 19:18:52 pve2 systemd[1]: Starting The Proxmox VE cluster filesystem...
Aug 19 19:18:52 pve2 pmxcfs[4349]: [quorum] crit: quorum_initialize failed: 2
Aug 19 19:18:52 pve2 pmxcfs[4349]: [quorum] crit: can't initialize service
Aug 19 19:18:52 pve2 pmxcfs[4349]: [confdb] crit: cmap_initialize failed: 2
Aug 19 19:18:52 pve2 pmxcfs[4349]: [confdb] crit: can't initialize service
Aug 19 19:18:52 pve2 pmxcfs[4349]: [dcdb] crit: cpg_initialize failed: 2
Aug 19 19:18:52 pve2 pmxcfs[4349]: [dcdb] crit: can't initialize service
Aug 19 19:18:52 pve2 pmxcfs[4349]: [status] crit: cpg_initialize failed: 2
Aug 19 19:18:52 pve2 pmxcfs[4349]: [status] crit: can't initialize service
Aug 19 19:18:52 pve2 pve-ha-lrm[2589]: ipcc_send_rec failed: Transport endpoint is not connected
Aug 19 19:18:52 pve2 pve-ha-lrm[2589]: ipcc_send_rec failed: Connection refused
Aug 19 19:18:52 pve2 pve-ha-lrm[2589]: ipcc_send_rec failed: Connection refused
Aug 19 19:18:52 pve2 pveproxy[4281]: worker exit
Aug 19 19:18:52 pve2 pveproxy[2590]: worker 4281 finished
Aug 19 19:18:52 pve2 pveproxy[2590]: starting 1 worker(s)
Aug 19 19:18:52 pve2 pveproxy[2590]: worker 4353 started
Aug 19 19:18:52 pve2 pveproxy[4282]: worker exit
Aug 19 19:18:52 pve2 pveproxy[4283]: worker exit
Aug 19 19:18:52 pve2 pveproxy[2590]: worker 4282 finished
Aug 19 19:18:52 pve2 pveproxy[2590]: starting 1 worker(s)
Aug 19 19:18:52 pve2 pveproxy[2590]: worker 4354 started
Aug 19 19:18:52 pve2 pveproxy[2590]: worker 4283 finished
Aug 19 19:18:52 pve2 pveproxy[2590]: starting 1 worker(s)
Aug 19 19:18:52 pve2 pveproxy[2590]: worker 4355 started
Aug 19 19:18:53 pve2 pveproxy[4355]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:18:53 pve2 pveproxy[4353]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:18:53 pve2 pveproxy[4354]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:18:53 pve2 systemd[1]: Started The Proxmox VE cluster filesystem.
Aug 19 19:18:53 pve2 systemd[1]: Starting Corosync Cluster Engine...
Aug 19 19:18:53 pve2 corosync[4369]: [MAIN ] Corosync Cluster Engine ('2.4.2'): started and ready to provide service.
Aug 19 19:18:53 pve2 corosync[4369]: [MAIN ] Corosync built-in features: augeas systemd pie relro bindnow
Aug 19 19:18:53 pve2 corosync[4371]: [TOTEM ] Initializing transport (UDP/IP Multicast).
Aug 19 19:18:53 pve2 corosync[4371]: [TOTEM ] Initializing transmit/receive security (NSS) crypto: aes256 hash: sha1
Aug 19 19:18:53 pve2 corosync[4371]: [TOTEM ] The network interface [192.168.25.11] is now up.
Aug 19 19:18:53 pve2 corosync[4371]: [SERV ] Service engine loaded: corosync configuration map access [0]
Aug 19 19:18:53 pve2 corosync[4371]: [QB ] server name: cmap
Aug 19 19:18:53 pve2 corosync[4371]: [SERV ] Service engine loaded: corosync configuration service [1]
Aug 19 19:18:53 pve2 corosync[4371]: [QB ] server name: cfg
Aug 19 19:18:53 pve2 corosync[4371]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Aug 19 19:18:53 pve2 corosync[4371]: [QB ] server name: cpg
Aug 19 19:18:53 pve2 corosync[4371]: [SERV ] Service engine loaded: corosync profile loading service [4]
Aug 19 19:18:53 pve2 corosync[4371]: [QUORUM] Using quorum provider corosync_votequorum
Aug 19 19:18:53 pve2 corosync[4371]: [QUORUM] This node is within the primary component and will provide service.
Aug 19 19:18:53 pve2 corosync[4371]: [QUORUM] Members[0]:
Aug 19 19:18:53 pve2 corosync[4371]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Aug 19 19:18:53 pve2 corosync[4371]: [QB ] server name: votequorum
Aug 19 19:18:53 pve2 corosync[4371]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Aug 19 19:18:53 pve2 corosync[4371]: [QB ] server name: quorum
Aug 19 19:18:53 pve2 corosync[4371]: [TOTEM ] JOIN or LEAVE message was thrown away during flush operation.
Aug 19 19:18:53 pve2 corosync[4371]: [TOTEM ] A new membership (192.168.25.11:88) was formed. Members joined: 1
Aug 19 19:18:53 pve2 corosync[4371]: [QUORUM] Members[1]: 1
Aug 19 19:18:53 pve2 corosync[4371]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 19 19:18:53 pve2 corosync[4371]: [TOTEM ] A new membership (192.168.25.10:92) was formed. Members joined: 1
Aug 19 19:18:54 pve2 pvestatd[2283]: ipcc_send_rec failed: Transport endpoint is not connected
Aug 19 19:18:58 pve2 pveproxy[4353]: worker exit
Aug 19 19:18:58 pve2 pveproxy[4355]: worker exit
Aug 19 19:18:58 pve2 pveproxy[4354]: worker exit
Aug 19 19:18:58 pve2 pmxcfs[4349]: [quorum] crit: quorum_initialize failed: 2
Aug 19 19:18:58 pve2 pmxcfs[4349]: [confdb] crit: cmap_initialize failed: 2
Aug 19 19:18:58 pve2 pmxcfs[4349]: [dcdb] crit: cpg_initialize failed: 2
Aug 19 19:18:58 pve2 pmxcfs[4349]: [status] crit: cpg_initialize failed: 2
Aug 19 19:18:58 pve2 pveproxy[2590]: worker 4354 finished
Aug 19 19:18:58 pve2 pveproxy[2590]: starting 1 worker(s)
Aug 19 19:18:58 pve2 pveproxy[2590]: worker 4353 finished
Aug 19 19:18:58 pve2 pveproxy[2590]: worker 4355 finished
Aug 19 19:18:58 pve2 pveproxy[2590]: worker 4400 started
Aug 19 19:18:58 pve2 pveproxy[4400]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:19:01 pve2 cron[2259]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Aug 19 19:19:03 pve2 pveproxy[2590]: starting 2 worker(s)
Aug 19 19:19:03 pve2 pveproxy[2590]: worker 4421 started
Aug 19 19:19:03 pve2 pveproxy[2590]: worker 4422 started
Aug 19 19:19:03 pve2 pveproxy[4400]: worker exit
Aug 19 19:19:03 pve2 pveproxy[4421]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:19:03 pve2 pveproxy[4422]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:19:03 pve2 pveproxy[2590]: worker 4400 finished
Aug 19 19:19:03 pve2 pveproxy[2590]: starting 1 worker(s)
Aug 19 19:19:03 pve2 pveproxy[2590]: worker 4423 started
Aug 19 19:19:03 pve2 pveproxy[4423]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:19:04 pve2 pmxcfs[4349]: [quorum] crit: quorum_initialize failed: 2
Aug 19 19:19:04 pve2 pmxcfs[4349]: [confdb] crit: cmap_initialize failed: 2
Aug 19 19:19:04 pve2 pmxcfs[4349]: [dcdb] crit: cpg_initialize failed: 2
Aug 19 19:19:04 pve2 pmxcfs[4349]: [status] crit: cpg_initialize failed: 2
Aug 19 19:19:08 pve2 pveproxy[4421]: worker exit
Aug 19 19:19:08 pve2 pveproxy[4422]: worker exit
Aug 19 19:19:08 pve2 pveproxy[2590]: worker 4421 finished
Aug 19 19:19:08 pve2 pveproxy[2590]: starting 1 worker(s)
Aug 19 19:19:08 pve2 pveproxy[2590]: worker 4422 finished
Aug 19 19:19:08 pve2 pveproxy[2590]: worker 4450 started
Aug 19 19:19:08 pve2 pveproxy[4423]: worker exit
Aug 19 19:19:08 pve2 pveproxy[4450]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEven
t.pm line 1618.
Aug 19 19:19:08 pve2 pveproxy[2590]: worker 4423 finished
Aug 19 19:19:08 pve2 pveproxy[2590]: starting 2 worker(s)

How can I return the configuration back to pve2?
Web interface on pve2 is not available.

In /etc/pve:

root@pve:/etc/pve# ls -al
total 9
drwxr-xr-x 2 root www-data 0 Jan 1 1970 .
drwxr-xr-x 100 root root 195 Aug 19 17:00 ..
-r--r----- 1 root www-data 574 Jan 1 1970 .clusterlog
-r--r----- 1 root www-data 349 Aug 19 19:18 corosync.conf
-rw-r----- 1 root www-data 2 Jan 1 1970 .debug
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 local -> nodes/pve
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 lxc -> nodes/pve/lxc
-r--r----- 1 root www-data 36 Jan 1 1970 .members
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 openvz -> nodes/pve/openvz
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 qemu-server -> nodes/pve/qemu-server
-r--r----- 1 root www-data 214 Jan 1 1970 .rrd
-r--r----- 1 root www-data 377 Jan 1 1970 .version
-r--r----- 1 root www-data 18 Jan 1 1970 .vmlist

What can I do to restore this? I see in /dev/rpool/data this:

root@pve:/dev/rpool/data# ls -al
total 0
drwxr-xr-x 2 root root 240 Aug 19 19:04 .
drwxr-xr-x 3 root root 80 Aug 19 19:04 ..
lrwxrwxrwx 1 root root 10 Aug 19 19:04 vm-100-disk-1 -> ../../zd48
lrwxrwxrwx 1 root root 12 Aug 19 19:04 vm-100-disk-1-part1 -> ../../zd48p1
lrwxrwxrwx 1 root root 12 Aug 19 19:04 vm-100-disk-1-part2 -> ../../zd48p2
lrwxrwxrwx 1 root root 10 Aug 19 19:04 vm-100-state-Snapshot_091216 -> ../../zd80
lrwxrwxrwx 1 root root 10 Aug 19 19:04 vm-100-state-Snapshot1 -> ../../zd32
lrwxrwxrwx 1 root root 10 Aug 19 19:04 vm-101-disk-1 -> ../../zd64
lrwxrwxrwx 1 root root 12 Aug 19 19:04 vm-101-disk-1-part1 -> ../../zd64p1
lrwxrwxrwx 1 root root 12 Aug 19 19:04 vm-101-disk-1-part2 -> ../../zd64p2
lrwxrwxrwx 1 root root 12 Aug 19 19:04 vm-101-disk-1-part5 -> ../../zd64p5
lrwxrwxrwx 1 root root 10 Aug 19 19:04 vm-101-state-Snapshot1 -> ../../zd16

I think that this is the way.

 

[Nascire]

I had a similar issue recently, this thread helped in my case:
https://forum.proxmox.com/threads/all-vm-lost-after-added-to-cluster.32051/

 

[Mex]

Thanks! It work for me!