Can't contact DHCP after firewall is re-enabled

A

amulalic

New Member
Jan 31, 2025
4
0
1
Hi everyone, I have recently created a cluster of 2 nodes that I use to host local media (jellyfin, kubernetes, open media vault and such). I have never used firewalls before and I thought that this might be the perferct opportunity to test it out. I have set up most of the firewall (basic HTTPS, SSH and similar rules) and I have left it over night.
However, In the morning, I noticed that nothing works, I can't access anything, I can't ping anything. After some digging I found out that my VMs and containers can't get new IPs from the DHCP server. I realised that I had to allow inbound from those ports too (67 and 68) and I tried it again. Again, dhclient won't resolve, can't ping the gateway but I noticed that the traffic is leaving the container (throught verbose dhclient) and contacting the main node (pve1) through tcpdump.
The output, from container, looks like this:
Code: 
root@wireguard:~# ifdown eth0 && ifup eth0
Killed old client process
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/bc:24:11:c4:bd:1e
Sending on   LPF/eth0/bc:24:11:c4:bd:1e
Sending on   Socket/fallback
DHCPRELEASE of 192.168.98.27 on eth0 to 192.168.98.1 port 67
send_packet: Network is unreachable
send_packet: please consult README file regarding broadcast address.
dhclient.c:3124: Failed to send 300 byte long packet over fallback interface.
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/bc:24:11:c4:bd:1e
Sending on   LPF/eth0/bc:24:11:c4:bd:1e
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
        DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6
No DHCPOFFERS received.
No working leases in persistent database - sleeping.

And tcp dump from the node (which has network interface):
Code: 
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
00:28:08.732842 veth101i0 B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from bc:24:11:c4:bd:1e, length 300
00:28:14.034167 veth101i0 B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from bc:24:11:c4:bd:1e, length 300
00:28:27.834191 veth101i0 B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from bc:24:11:c4:bd:1e, length 300

I have tried everything that I found online, restart the interface, restart the container,... And the only thing that worked eventually is restarting the node itself. Just to verify that I am not going insane, I have did the same procedure:
- Ping the wireguard container (192.168.98.27)
- Observe the pings
- Turn datacenter firewall
- Pings stop eventually
- Disable datacenter firewall
- Try to ping - fail
- Try to manually get the IP - fail

I can easily disable the firewall and move on with this but I am really interested in what is going on here, why does the restart fix it and what do I need to do to restart the dhcp client without restarting the node (I am mostly away from the PC). Also I don't understand why disabling the firewall doesn't "undo" everything.

Any advice will be greatly appreciated.

Pve01/node/192.168.98.9 network:
Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.98.9/24
    gateway 192.168.98.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

source /etc/network/interfaces.d/*

Container/wireguard/192.168.98.27 network (configured to listen to vmbr0):
Code: 
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

Best regards,
Almir

EDIT 1:
- Removing and adding the interface back doesn't work
- I have added a new container, same issue, same output in journalctl as the failed dhclient command
- sudo ifdown vmbr0 && sudo ifup vmbr0 on host doesn't work
 
Last edited:
I intercepted all trafic on host node that is comming from the container and it looks like that its acknowlaged but there is no response back to the container.
Code: 
╰─ sudo tcpdump -i any src host 192.168.98.100                                                                                                       ─╯

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
12:46:23.182989 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 38, length 64
12:46:24.206912 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 39, length 64
12:46:25.231976 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 40, length 64
12:46:26.254996 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 41, length 64
12:46:27.278987 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 42, length 64
12:46:28.302913 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 43, length 64
12:46:29.326910 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 44, length 64
12:46:30.350990 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 45, length 64
12:46:31.374908 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 46, length 64
12:46:32.398978 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 47, length 64
12:46:33.422912 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 48, length 64
12:46:34.447919 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 49, length 64
12:46:35.470939 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 50, length 64
12:46:36.494953 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 51, length 64
12:46:37.518933 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 52, length 64
12:46:38.542955 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 53, length 64
12:46:39.567059 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 54, length 64
12:46:40.590946 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 55, length 64
12:46:41.615892 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 56, length 64
12:46:42.639983 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 57, length 64
12:46:43.662884 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 58, length 64
12:46:44.686917 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 59, length 64
12:46:45.710996 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 60, length 64
12:46:46.735008 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 61, length 64
12:46:47.758900 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 62, length 64
12:46:48.782900 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 63, length 64
12:46:49.807983 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 64, length 64
12:46:50.830983 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 65, length 64
12:46:51.854944 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 66, length 64
12:46:52.878953 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 67, length 64
12:46:53.902932 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 68, length 64
12:46:54.927007 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 69, length 64
12:46:55.886956 veth101i0 P   ARP, Request who-has 192.168.98.1 tell 192.168.98.100, length 28
12:46:55.887008 eno1  Out ARP, Request who-has 192.168.98.1 tell 192.168.98.100, length 28
12:46:55.951001 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 70, length 64
12:46:56.974998 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 71, length 64
12:46:57.999985 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 72, length 64
12:46:59.023907 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 73, length 64
12:47:00.047006 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 74, length 64
12:47:01.071982 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 75, length 64
12:47:02.094992 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 76, length 64
12:47:03.119002 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 77, length 64
12:47:04.143970 veth101i0 P   IP 192.168.98.100 > 192.168.98.1: ICMP echo request, id 1447, seq 78, length 64
^C
43 packets captured
46 packets received by filter
0 packets dropped by kernel

But the ping in the container hangs, it looks like something is blocking the requests even tho the firewall is disabled and stopped
 
Running the following on the host did the trick, still not sure why. I had a open TCP listner and as soon as I ran these commands everything started to go through, my best guess is that there was a stale OUTPUT or FORWARD rule even tho I checked multiple times to see if the iptables are empty:

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

I guess from now on manual reset of the iptables will be my first guess
 
FYI what you did in your most recent post was essentially drop all the firewall rules. Once you get them setup again (according to what you had in the first post) try the following for DHCP:

Code: 
iptables  -I INPUT -i <interface name> -p udp --dport 67:68 --sport 67:68 -j ACCEPT
 
I did that through UI but still resulted in same way, now it works.

I was under the impression that disabling the firewall (on the datacenter or node level) would in theory do this:

Code: 
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

But it looks like the rules stay
 
amulalic said:
I did that through UI but still resulted in same way, now it works.

I was under the impression that disabling the firewall (on the datacenter or node level) would in theory do this:

Code: 
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

But it looks like the rules stay
Click to expand...
That's my mistake. You can absolutely do this, the rules won't persist for iptables unless you save them.
 
You must log in or register to reply here.
Top Bottom