Timeout in Cluster

P

punjprateek

Member
Jul 26, 2021
71
1
13
30
India
I have 6 server in cluster, when i enabled the firewall on Datacenter and Node Level, out of 6 server 1 server stopped responding to ssh, it says connection timeout.
i am also attaching the datacenter fw rules and node fw rule for reference. it would be great if someone can help me fix the problem

[OPTIONS]

enable: 1
ebtables: 1

[IPSET api-ips] # IP Sets of UAT and Prod apiservers

10.250.0.103 # Prod Orchestrator
10.250.0.108 # UAT Orchestrator
103.182.65.1 # UAT Orchestrator (Extn. IP)
103.182.65.2 # Prod Orchestrator (Extn. IP)

[IPSET dns]

10.250.0.2

[IPSET gateway]

10.250.0.1
10.250.50.5

[IPSET mgmt-ip] # 0.x Segment

10.250.0.0/23 # Mangement Segment

[IPSET nfs-ips] # Access Login

10.250.0.10

[IPSET vm-ips] # IP Range of Antplay VMs

10.250.50.0/23 # IP Address of Antplay VMs

[IPSET vpn-ip] # VPN segment

10.250.1.0/24

[RULES]

GROUP basic-port
GROUP moonlight
|IN ACCEPT -source +vm-ips -dest +dns -log nolog
|IN RDP(ACCEPT) -dest +vm-ips -log nolog
|OUT DROP -source +vm-ips -dest +mgmt-ip -log debug # disable play seg to mgmt seg

[group basic-port] # communication port

IN ACCEPT -source +vpn-ip -dest +vm-ips -log nolog
IN ACCEPT -p tcp -dport 22,8006 -log nolog # ssh and web interface ports

[group moonlight]

IN DROP -source +vm-ips -dest +vm-ips -log warning # Disable VM to VM access.
IN ACCEPT -p udp -dport 47998,47999,48000,48002,48010 -log nolog
IN ACCEPT -p tcp -dport 47984,47989,48010 -log nolog
IN ACCEPT -source +api-ips -dest +vm-ips -p udp -log nolog
[OPTIONS]

enable: 1

[RULES]

IN DROP -source +vm-ips -dest +vm-ips -log nolog
|OUT DROP -source +vm-ips -dest +nfs-ips -log nolog
|GROUP basic-port
GROUP moonlight
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.3-1
proxmox-backup-file-restore: 2.4.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1
 
Hi,
please also share the network configuration cat /etc/network/interfaces and the output of iptables-save for the host not reachable (of course you will have to establish a connection first, before enabling the firewall, so that you can execute these commands with the firewall rules enabled via the established connection).
 
Chris said:
Hi,
please also share the network configuration cat /etc/network/interfaces and the output of iptables-save for the host not reachable (of course you will have to establish a connection first, before enabling the firewall, so that you can execute these commands with the firewall rules enabled via the established connection).
Click to expand...
Hi Attached is the result, i am not able to access through GUI but through ssh i am able to reach host using cmd.

auto lo
iface lo inet loopback

iface enp28s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.250.50.42/23
gateway 10.250.50.5
bridge-ports enp28s0f1
bridge-stp off
bridge-fd 0

iface eno1 inet manual

iface eno2 inet manual

iface enx3a5631568dd4 inet manual

iface enp28s0f0 inet manual
# Generated by iptables-save v1.8.7 on Sun Aug 13 23:40:52 2023
*raw
:PREROUTING ACCEPT [204995371:639080509840]
:OUTPUT ACCEPT [171453589:863741416324]
COMMIT
# Completed on Sun Aug 13 23:40:52 2023
# Generated by iptables-save v1.8.7 on Sun Aug 13 23:40:52 2023
*filter
:INPUT ACCEPT [2:208]
:FORWARD ACCEPT [3879365:456865439]
:OUTPUT ACCEPT [42186:2552176]
:GROUP-basic-port-IN - [0:0]
:GROUP-basic-port-OUT - [0:0]
:GROUP-moonlight-IN - [0:0]
:GROUP-moonlight-OUT - [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
:tap100i0-IN - [0:0]
:tap100i0-OUT - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A GROUP-basic-port-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-basic-port-IN -m set --match-set PVEFW-0-vpn-ip-v4 src -m set --match-set PVEFW-0-vm-ips-v4 dst -g PVEFW-SET-ACCEPT-MARK
-A GROUP-basic-port-IN -p tcp -m multiport --dports 22,8006 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-basic-port-IN -m comment --comment "PVESIG:wSZb4KgmJ0oA8RZy7zDqEWzkagg"
-A GROUP-basic-port-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-basic-port-OUT -m comment --comment "PVESIG:xxVsq70UXLT9ZzIBtsVFZssSqPw"
-A GROUP-moonlight-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-moonlight-IN -p udp -m multiport --dports 47998,47999,48000,48002,48010 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-moonlight-IN -p tcp -m multiport --dports 47984,47989,48010 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-moonlight-IN -p udp -m set --match-set PVEFW-0-api-ips-v4 src -m set --match-set PVEFW-0-vm-ips-v4 dst -g PVEFW-SET-ACCEPT-MARK
-A GROUP-moonlight-IN -m comment --comment "PVESIG:JLD/rtIXVmgHt8IRriLBMAUmRUI"
-A GROUP-moonlight-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-moonlight-OUT -m comment --comment "PVESIG:SfZYLZS+rTi+GGMoTRCDJXMWUdw"
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m physdev --physdev-out tap100i0 --physdev-is-bridged -j tap100i0-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:/naDZxJ06t8Dx9DQtmus9NvdHEA"
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap100i0 --physdev-is-bridged -j tap100i0-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:wA3mj3VIKyC/rlY95PCFN7paR5s"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -m set --match-set PVEFW-0-vm-ips-v4 src -m set --match-set PVEFW-0-vm-ips-v4 dst -j DROP
-A PVEFW-HOST-IN -j GROUP-moonlight-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -j GROUP-basic-port-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -j GROUP-moonlight-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -s 10.250.50.43/32 -d 10.250.50.42/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 10.250.50.44/32 -d 10.250.50.42/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 10.250.50.45/32 -d 10.250.50.42/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 10.250.50.47/32 -d 10.250.50.42/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 10.250.50.48/32 -d 10.250.50.42/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:+yK7ANSk8duUHGM9ysRa6QhZYKc"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -j GROUP-moonlight-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -j GROUP-basic-port-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -j GROUP-moonlight-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -d 10.250.50.0/23 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 10.250.50.0/23 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 10.250.50.0/23 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 10.250.50.0/23 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -s 10.250.50.42/32 -d 10.250.50.43/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 10.250.50.42/32 -d 10.250.50.44/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 10.250.50.42/32 -d 10.250.50.45/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 10.250.50.42/32 -d 10.250.50.47/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 10.250.50.42/32 -d 10.250.50.48/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:8Eqd+qocOlIZ3SY6fjdLu41APMg"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A tap100i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap100i0-IN -j PVEFW-Drop
-A tap100i0-IN -j DROP
-A tap100i0-IN -m comment --comment "PVESIG:dyBT7vHzkJ/DuuTmaQj+ta/b9fk"
-A tap100i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap100i0-OUT -m mac ! --mac-source d2:77:bc:5f:a3:d5 -j DROP
-A tap100i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap100i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap100i0-OUT -m comment --comment "PVESIG:G56q5Mk2voOV53Vae5oMUhx/U3g"
COMMIT
# Completed on Sun Aug 13 23:40:52 2023
 
punjprateek said:
server 1 server stopped responding to ssh, it says connection timeout.
Click to expand...
punjprateek said:
i am not able to access through GUI but through ssh i am able to reach host using cmd.
Click to expand...
Please clarify what the exact issue is. Are you able to ssh to the node with firewall enabled or not? Is the connection issue limited to the WebUI? If so, from where are you trying to connect to? Provide also the journal from around the time of the connection issue journalctl --since <DATETIME> --until <DATETIME> > journal.txt as attachment.
 
Chris said:
Please clarify what the exact issue is. Are you able to ssh to the node with firewall enabled or not? Is the connection issue limited to the WebUI? If so, from where are you trying to connect to? Provide also the journal from around the time of the connection issue journalctl --since <DATETIME> --until <DATETIME> > journal.txt as attachment.
Click to expand...
The connection issue is limited to webGUI only, from CMD i am able to SSH my server's.
 

Attachments

punjprateek said:
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
Click to expand...
punjprateek said:
-A GROUP-basic-port-IN -p tcp -m multiport --dports 22,8006 -g PVEFW-SET-ACCEPT-MARK
Click to expand...
As far as I can see, above rules should allow the traffic to the WebUI. Can you verify that the WebUI works with the firewall disabled for that node?
 
punjprateek said:
-A PVEFW-HOST-IN -m set --match-set PVEFW-0-vm-ips-v4 src -m set --match-set PVEFW-0-vm-ips-v4 dst -j DROP
Click to expand...
Is your source maybe within the IP range of the vm ips? As this is evaluated before your basic port rules. If that is not the case, I suggest to selectively disable rules until you get connectivity again (you will always have to wait a few seconds until the changed rules get applied by the PVE firewall) and start adapting the rules from there.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom