Unable to start firewall on node

[Arthur Stephens]

unable to open file '/etc/pve/local/host.fw.tmp.2630' - Permission denied (500)

Firewall has started on the other two nodes of the cluster.
But when I try to add a rule I get the error message above.
I also tried rebooting the node.
The datacenter says the node is offline and yet I can log into it.


Virtual Environment 4.4-1/eb2d6f1e
CPU usage0.30% of 32 CPU(s)
IO delay0.08%
Load average0.00,0.00,0.00
RAM usage1.80% (2.27 GiB of 125.87 GiB)
KSM sharing0 B
HD space(root)3.48% (2.37 GiB of 68.28 GiB)
SWAP usage0.00% (0 B of 8.00 GiB)
CPUs32 x Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2 Sockets)
Kernel VersionLinux 4.4.35-1-pve #1 SMP Fri Dec 9 11:09:55 CET 2016
PVE Manager Version
pve-manager/4.4-1/eb2d6f1e

 

[Arthur Stephens]

More info
root@pteranode3:/etc/pve/local# ls -al
total 2
dr-xr-xr-x 2 root www-data 0 Dec 26 2016 .
dr-xr-xr-x 2 root www-data 0 Dec 26 2016 ..
-r--r----- 1 root www-data 22 Jan 22 2018 host.fw
-r--r----- 1 root www-data 83 Mar 23 03:11 lrm_status
dr-xr-xr-x 2 root www-data 0 Dec 26 2016 lxc
dr-xr-xr-x 2 root www-data 0 Dec 26 2016 openvz
dr-x------ 2 root www-data 0 Dec 26 2016 priv
-r--r----- 1 root www-data 1675 Dec 26 2016 pve-ssl.key
-r--r----- 1 root www-data 1720 Dec 26 2016 pve-ssl.pem
dr-xr-xr-x 2 root www-data 0 Dec 26 2016 qemu-server

Um why are those set to read only?
The same directory on the other two node shows the same files but they are -rw-r.

 

[Arthur Stephens]

So I thought I would change the permissions so I could manually put the commands in but the system will not let me chmod either

root@pteranode3:/etc/pve/local# chmod 600 host.fw
chmod: changing permissions of âhost.fwâ: Function not implemented

This system is being used for NTP DDOS attacks and looks like my only other recourse is to shut down the system.
Not preferred since this is part of a ceph drive and HA setup also, in production running our CMS and Email.

 

[dcsapak]

The same directory on the other two node shows the same files but they are -rw-r.

your network is disrupted, only the quorate network partition can write to the cluster filesystem
this node cannot reach the other two via corosync

unless you fix the network, you cannot write to /etc/pve

 

[Arthur Stephens]

Yes I thought about that - eveything is pingable.
The 10.10.10.0 subnet is 10gig modules plugged into Dell 10gig switch
The 69.28.32.0 subnet is a cisco gig switch

root@pteranode3:/etc/pve/local# ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.180 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.211 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=0.145 ms
^C
--- 10.10.10.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.145/0.178/0.211/0.031 ms
root@pteranode3:/etc/pve/local# ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
64 bytes from 10.10.10.2: icmp_seq=1 ttl=64 time=0.262 ms
64 bytes from 10.10.10.2: icmp_seq=2 ttl=64 time=0.218 ms
64 bytes from 10.10.10.2: icmp_seq=3 ttl=64 time=0.230 ms
^C
--- 10.10.10.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.218/0.236/0.262/0.025 ms
root@pteranode3:/etc/pve/local# ping 10.10.10.3
PING 10.10.10.3 (10.10.10.3) 56(84) bytes of data.
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.035 ms
^C
--- 10.10.10.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.032/0.033/0.035/0.006 ms

root@pteranode3:/etc/pve/local# ping 69.28.32.120
PING 69.28.32.120 (69.28.32.120) 56(84) bytes of data.
64 bytes from 69.28.32.120: icmp_seq=1 ttl=64 time=0.130 ms
64 bytes from 69.28.32.120: icmp_seq=2 ttl=64 time=0.193 ms
64 bytes from 69.28.32.120: icmp_seq=3 ttl=64 time=0.160 ms
^C
--- 69.28.32.120 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.130/0.161/0.193/0.025 ms
root@pteranode3:/etc/pve/local# ping 69.28.32.121
PING 69.28.32.121 (69.28.32.121) 56(84) bytes of data.
64 bytes from 69.28.32.121: icmp_seq=1 ttl=64 time=0.197 ms
64 bytes from 69.28.32.121: icmp_seq=2 ttl=64 time=0.108 ms
64 bytes from 69.28.32.121: icmp_seq=3 ttl=64 time=0.111 ms
64 bytes from 69.28.32.121: icmp_seq=4 ttl=64 time=0.224 ms
^C
--- 69.28.32.121 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.108/0.160/0.224/0.051 ms
root@pteranode3:/etc/pve/local# ping 69.28.32.122
PING 69.28.32.122 (69.28.32.122) 56(84) bytes of data.
64 bytes from 69.28.32.122: icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from 69.28.32.122: icmp_seq=2 ttl=64 time=0.051 ms
64 bytes from 69.28.32.122: icmp_seq=3 ttl=64 time=0.040 ms
64 bytes from 69.28.32.122: icmp_seq=4 ttl=64 time=0.031 ms
^C
--- 69.28.32.122 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.031/0.038/0.051/0.010 ms

How to do I find the disruption?

Thanks

 

[Arthur Stephens]

root@pteranode3:/etc/pve/local# pvecm status
Quorum information
------------------
Date: Thu Aug 23 11:04:53 2018
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000002
Ring ID: 2/1704
Quorate: No

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000002 1 69.28.32.122 (local)
root@pteranode3:/etc/pve/local# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
2 1 pteranode3 (local)

 

[Arthur Stephens]

root@pteracluster:~# pvecm status
Quorum information
------------------
Date: Thu Aug 23 11:11:01 2018
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000001
Ring ID: 1/308
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 69.28.32.120 (local)
0x00000003 1 69.28.32.121

root@pteranode3:/etc/pve/local# pvecm add 69.28.32.120
authentication key already exists

 

[Arthur Stephens]

Aug 23 11:31:45 pteranode3 systemd[1]: Starting The Proxmox VE cluster filesystem...
Aug 23 11:31:45 pteranode3 pmxcfs[43643]: [status] notice: update cluster info (cluster name pteracluster, version = 3)
Aug 23 11:31:45 pteranode3 pmxcfs[43643]: [dcdb] notice: members: 2/43643
Aug 23 11:31:45 pteranode3 pmxcfs[43643]: [dcdb] notice: all data is up to date
Aug 23 11:31:45 pteranode3 pmxcfs[43643]: [status] notice: members: 2/43643
Aug 23 11:31:45 pteranode3 pmxcfs[43643]: [status] notice: all data is up to date
Aug 23 11:31:45 pteranode3 pveproxy[33342]: ipcc_send_rec failed: Transport endpoint is not connected
Aug 23 11:31:45 pteranode3 pveproxy[38499]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pveproxy[38499]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pveproxy[33342]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pveproxy[33342]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pveproxy[38499]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pveproxy[37928]: ipcc_send_rec failed: Transport endpoint is not connected
Aug 23 11:31:45 pteranode3 pveproxy[37928]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pvedaemon[29140]: ipcc_send_rec failed: Transport endpoint is not connected
Aug 23 11:31:45 pteranode3 pveproxy[37928]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pvedaemon[29140]: ipcc_send_rec failed: Connection refused
Aug 23 11:31:45 pteranode3 pvedaemon[29140]: ipcc_send_rec failed: Connection refused

 

[Arthur Stephens]

• your network is disrupted, only the quorate network partition can write to the cluster filesystem
• this node cannot reach the other two via corosync
• 
  
• unless you fix the network, you cannot write to /etc/pve

OK the issue was two of them were plugged into a cisco gig switch and the third one was plugged into nexus switch with a jumper to the cisco switch. Ran a cable from node3 to the switch that the other two were plugged into and all is well.

Thanks for pointing me in the right direction - though it took me awhile to find it.