Need some help, locked out by firewall, disaster recovery :-)

[Ovidiu]

###edit###
forgot to mention, running proxmox ve 4 as offered by soyoustart

Hi there,

I'm just getting started using proxmox on a remote server so I can't access it locally.
I decided to enable the firewall on proxmox, the node/host and the only guest and locked myself out as I had not defined any rules to allow me access.

so I booted from a rescue mode, mounted the / and did the following:

edited /etc/pve/firewall/cluster.fw and inserted

[OPTIONS]
# enable firewall (cluster wide setting, default is disabled)
enable: 0

[IPSET management]
MYIP

After inserting below content and restarting from disk as usual, I can now SSH into my host system where proxmox ve runs but I still can't access its management via web.

on the host I tried

pct enter 100

just for a test and I can't enter that LXC container.

netstat -a | grep 8006
tcp        0      0 *:8006                  *:*                     LISTEN
tcp      424      0 james.mydomain.:8006 p5xxCAF3A.dip0.t-:53269 CLOSE_WAIT
tcp      423      0 james.mydomain.:8006 p5xxCAF3A.dip0.t-:53252 ESTABLISHED
tcp        1      0 james.mydomain.:8006 p5xxCAF3A.dip0.t-:53266 CLOSE_WAIT
tcp        1      0 james.mydomain.:8006 p5xxCAF3A.dip0.t-:53254 CLOSE_WAIT
tcp        1      0 james.mydomain.:8006 p5xxCAF3A.dip0.t-:53253 CLOSE_WAIT

And yet I can't access the management console :-(

pve-firewall status
ipcc_send_rec failed: Connection refused
ipcc_send_rec failed: Connection refused
ipcc_send_rec failed: Connection refused
Status: disabled/stopped

iptables-save
# Generated by iptables-save v1.4.21 on Fri Nov  6 20:45:56 2015
*filter
:INPUT ACCEPT [1605:216894]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1889:243514]
COMMIT
# Completed on Fri Nov  6 20:45:56 2015

 

[Ovidiu]

started digging deeper:

inside /var/log/daemon.log

Nov 6 21:20:51 james pveproxy[5132]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1631.
Nov 6 21:20:51 james pveproxy[5133]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1631.

tried this:

root@james:~# pvecm updatecerts --force
ipcc_send_rec failed: Connection refused
ipcc_send_rec failed: Connection refused
ipcc_send_rec failed: Connection refused
pve configuration filesystem not mounted

tried setting up a tunnel via SSH,

ssh -D 88 myserver

then set localhost:88 as socks server in browser, can browse anywhere except to my proxmox interface.

root@james:~# ls -l /etc/pve
total 32
drwxr-xr-x 2 root root 4096 Nov 6 20:13 firewall
drwxr-xr-x 3 root root 4096 Nov 6 20:23 nodes
-rw-r--r-- 1 root root 23625 Nov 6 20:22 ystemctl
root@james:~# service pve-cluster restart
Job for pve-cluster.service failed. See 'systemctl status pve-cluster.service' and 'journalctl -xn' for details.
root@james:~#

root@james:~# systemctl status pve-cluster.service● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: failed (Result: exit-code) since Fri 2015-11-06 21:26:28 CET; 1min 48s ago
Process: 5703 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=255)


Nov 06 21:26:28 james pmxcfs[5703]: fuse: mountpoint is not empty
Nov 06 21:26:28 james pmxcfs[5703]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Nov 06 21:26:28 james pmxcfs[5703]: [main] crit: fuse_mount error: File exists
Nov 06 21:26:28 james pmxcfs[5703]: [main] notice: exit proxmox configuration filesystem (-1)
Nov 06 21:26:28 james pmxcfs[5703]: [main] crit: fuse_mount error: File exists
Nov 06 21:26:28 james pmxcfs[5703]: [main] notice: exit proxmox configuration filesystem (-1)
Nov 06 21:26:28 james systemd[1]: pve-cluster.service: control process exited, code=exited status=255
Nov 06 21:26:28 james systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Nov 06 21:26:28 james systemd[1]: Unit pve-cluster.service entered failed state.

journalctl -xn-- Logs begin at Fri 2015-11-06 20:51:28 CET, end at Fri 2015-11-06 21:51:54 CET. --
Nov 06 21:51:53 james pveproxy[1368]: worker 8254 started
Nov 06 21:51:53 james pveproxy[1368]: worker 8255 started
Nov 06 21:51:53 james pveproxy[8254]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1631.
Nov 06 21:51:53 james pveproxy[8255]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1631.
Nov 06 21:51:54 james pve-ha-lrm[1377]: ipcc_send_rec failed: Connection refused
Nov 06 21:51:54 james pve-ha-lrm[1377]: ipcc_send_rec failed: Connection refused
Nov 06 21:51:54 james pve-ha-lrm[1377]: ipcc_send_rec failed: Connection refused
Nov 06 21:51:54 james pve-ha-crm[1366]: ipcc_send_rec failed: Connection refused
Nov 06 21:51:54 james pve-ha-crm[1366]: ipcc_send_rec failed: Connection refused
Nov 06 21:51:54 james pve-ha-crm[1366]: ipcc_send_rec failed: Connection refused

Really no idea what to do now. Please anyone got any ideas?

 

[Ovidiu]

root@james:/# pmxcfs
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
[main] crit: fuse_mount error: File exists
[main] notice: exit proxmox configuration filesystem (-1)
root@james:/#

I'm pretty sure I screwed up with my rescue mode boot.

The point is, there is absolutely nothing critical on this server yet I'd like to understand what I did wrong and how to fix it.

 

[dietmar]

I'm pretty sure I screwed up with my rescue mode boot.

The point is, there is absolutely nothing critical on this server yet I'd like to understand what I did wrong and how to fix it.

Seems you (or some command) wrote data into /etc/pve/ while pmxcfs was not mounted?

# ls -lR /etc/pve/

(backup and) remove those files, then try again.

 

[Ovidiu]

woohoo, that was easy!

ls -lR /etc/pve/
mv /etc/pve/* /tmp/bckp/
pmxcfs

all back and running

 

[Muns.io]

A small tipp for everyone who is reading this.
Before you run "pmxcfs" to load the config files be sure to turn off all proxmox services cause if they are running you lock yourself out instantly.