PVE listens on port 53 udp, why?

[gctwnl]

I have a pve running. The Debian on which PVE (PVE full install) has IP 192.168.2.50. On PVE runs a VM which serves an unbound server, which listens on port 53. The IP address of the unbound server is 192.168.2.125.

From the shell of the PVE Debian host, I get this (which I understand):

root@pve:~# nc -v -z -u 192.168.2.50 53
pve.rna.nl [192.168.2.50] 53 (domain) : Connection refused
root@pve:~# nc -v -z -u 127.0.0.1 53
localhost.localdomain [127.0.0.1] 53 (domain) : Connection refused
root@pve:~# nc -v -z -u 192.168.2.125 53
foo.rna.nl [192.168.2.125] 53 (domain) open

From the unbound server client VM inside the PVE host itself I get (which I also understand):

root@foo:/home/me# nc -v -z -u 192.168.2.125 53 && echo hello
Connection to 192.168.2.125 53 port [udp/domain] succeeded!
hello
root@foo:/home/me# nc -v -z -u 192.168.2.50 53 || echo hello
hello

But when I go to the PVE Debian host and the client VM from the outside I get:

gerben@bar% nc -v -z -u 192.168.2.50 53
Connection to 192.168.2.50 port 53 [udp/domain] succeeded!
gerben@bar% nc -v -z -u 192.168.2.125 53
Connection to 192.168.2.125 port 53 [udp/domain] succeeded!

The second one is as it should be. By why do I get the report that something is listening on 192.168.2.50 on UDP port 53 when I connect via the LAN, and not when I do this locally (inside the PVE machine)?

 

[LnxBil]

Have you checked the actual opened ports on your PVE host with netstat?

 

[bobmc]

I don't know this for a fact, but I would surmise that the host listens to port 53 to handle DNS replies from external servers?

 

[gctwnl]

I don't know this for a fact, but I would surmise that the host listens to port 53 to handle DNS replies from external servers?

There is a DNS (`unbound`) running on the VM with IP address 192.168.2.125 which listens on UDP 53. What I do not expect is that when I try from the LAN to connect to the IP of the PVE host itself, I get that UDP Port 53 is open, while it should not be. It seems that somehow the fact that the client VM has port 53 open makes the host react.

There is no `netstat` on a standard PVE install. But what I noted with nc above is that I get different results, depending on if I come from outside (ethernet) or if I am inside the VM/PVE-env.

ss on the host tells me nothing is listening on UDP 53

 

[Chris]

There is a DNS (`unbound`) running on the VM with IP address 192.168.2.125 which listens on UDP 53. What I do not expect is that when I try from the LAN to connect to the IP of the PVE host itself, I get that UDP Port 53 is open, while it should not be. It seems that somehow the fact that the client VM has port 53 open makes the host react.

sstells me nothing is listening on UDP 53

Hi,
are you sure that you are not forwarding the traffic to the VM? Can you check with tcpdump udp port 53 -i <interface> on the different interfaces where the traffic is going trough? Also check the rules with iptables

 

[gctwnl]

Hi,
are you sure that you are not forwarding the traffic to the VM? Can you check with tcpdump udp port 53 -i <interface> on the different interfaces where the traffic is going trough? Also check the rules with iptables

How do I find out which interfaces to check with tcpdump?

root@pve:~# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

This is all on the PVE host

 

[Chris]

How do I find out which interfaces to check with tcpdump?

root@pve:~# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination       

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination       

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

This is all on the PVE host

Ok, so no firewall rules forwarding traffic. I would first of all check the interface to the LAN of the PVE host, test with a few requests from the different clients and save the output, then repeat the same for the bridge the unbound VM is assigned to.

What is the output of ss -ulen on the PVE host? Can you also post the network configuration of the PVE host: cat /etc/network/interfaces

 

[gctwnl]

Hi,
are you sure that you are not forwarding the traffic to the VM? Can you check with tcpdump udp port 53 -i <interface> on the different interfaces where the traffic is going trough? Also check the rules with iptables

Using tcpdump -n -Q in udp port 53 on the PVE host (192.168.2.50) I get

When running nc -v -z -u 192.168.2.50 53 from an outside host (192.168.2.86) on the LAN to the PVE host (192.168.2.50):

17:41:15.580674 IP 192.168.2.86.57879 > 192.168.2.50.53: domain [length 1 < 12] (invalid)
17:41:15.580702 IP 192.168.2.86.57879 > 192.168.2.50.53: domain [length 1 < 12] (invalid)
17:41:15.580704 IP 192.168.2.86.57879 > 192.168.2.50.53: domain [length 1 < 12] (invalid)
17:41:15.580706 IP 192.168.2.86.57879 > 192.168.2.50.53: domain [length 1 < 12] (invalid)

When running nc -v -z -u 192.168.2.50 53 from the client (192.168.2.125) to the PVE host (192.168.2.50): nothing is seen in tcpdump on the PVE host (192.168.2.50).

When DNS traffic is passed on by PVE to the client (192.168.2.125) running the unbound DNS server:

17:41:36.842424 IP 192.168.2.86.53 > 192.168.2.125.44086: 16212 1/0/0 PTR router.rna.nl. (69)
17:41:36.842584 IP 192.168.2.66.53 > 192.168.2.125.44086: 16212 1/0/0 PTR router.rna.nl. (69)
17:41:36.842586 IP 192.168.2.86.53 > 192.168.2.125.58407: 7018 1/0/0 A 192.168.2.2 (47)
17:41:36.842642 IP 192.168.2.66.53 > 192.168.2.125.58407: 7018 1/0/0 A 192.168.2.2 (47)

 

[gctwnl]

Ok, so no firewall rules forwarding traffic. I would first of all check the interface to the LAN of the PVE host, test with a few requests from the different clients and save the output, then repeat the same for the bridge the unbound VM is assigned to.

What is the output of ss -ulen on the PVE host? Can you also post the network configuration of the PVE host: cat /etc/network/interfaces

root@pve:~# ss -ulen
State         Recv-Q        Send-Q               Local Address:Port                Peer Address:Port        Process                                                                                                     
UNCONN        0             0                          0.0.0.0:111                      0.0.0.0:*            ino:22971 sk:1003 cgroup:/system.slice/rpcbind.socket <->                                                 
UNCONN        0             0                        127.0.0.1:323                      0.0.0.0:*            ino:21067 sk:1004 cgroup:/system.slice/chrony.service <->                                                 
UNCONN        0             0                             [::]:111                         [::]:*            ino:22974 sk:1005 cgroup:/system.slice/rpcbind.socket v6only:1 <->                                         
UNCONN        0             0                            [::1]:323                         [::]:*            ino:21068 sk:1006 cgroup:/system.slice/chrony.service v6only:1 <->

root@pve:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.2.50/24
    gateway 192.168.2.2
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

iface wlp0s20f3 inet manual

 

[gctwnl]

It feels like nc -z directed at the PVE host when at least one of the client VMs listens on a port gets a positive response. That feels like PVE too late decides not to react on the en0 interface when the traffic is directed at itself when there is any client VM listening on the port. The traffic for client VM 192.168.2.125:53 ends up on the same physical en0 as 192.168.2.50:53. But PVE doesn't check if it is directed at the actual IP address of the client VM that is listening, reacts to it anyway. A real connection to the PVE host on UDP 53 will still fail, because at that stage PVE wants to pass it on to 192.168.2.50 (the PVE host itself) where no-one is listening. Feels like something deep in the PVE networking stack and checking the destination IP address too late. But I'm just guessing.

 

[Chris]

It feels like nc -z directed at the PVE host when at least one of the client VMs listens on a port gets a positive response. That feels like PVE too late decides not to react on the en0 interface when the traffic is directed at itself when there is any client VM listening on the port. The traffic for client VM 192.168.2.125:53 ends up on the same physical en0 as 192.168.2.50:53. But PVE doesn't check if it is directed at the actual IP address of the client VM that is listening, reacts to it anyway. A real connection to the PVE host on UDP 53 will still fail, because at that stage PVE wants to pass it on to 192.168.2.50 (the PVE host itself) where no-one is listening. Feels like something deep in the PVE networking stack and checking the destination IP address too late. But I'm just guessing.

I was able to reproduce this. This probably stems from different paths the packets take while going to the network stack (see https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg).

Edit: What kernel version are you running? uname -a

 

[gctwnl]

Edit: What kernel version are you running? uname -a

Linux pve 5.15.85-1-pve #1 SMP PVE 5.15.85-1 (2023-02-01T00:00Z) x86_64 GNU/Linux

 

[Chris]

Linux pve 5.15.85-1-pve #1 SMP PVE 5.15.85-1 (2023-02-01T00:00Z) x86_64 GNU/Linux

Okay, so after a goods night sleep, this seems to be more of a misleading netcat output. The same scan with nmap -sU -v <IP> -p 53 shows the port as open|filtered, which is according to nmap No response received (even after retransmissions) (see https://nmap.org/book/scan-methods-udp-scan.html).

 

[LnxBil]

Okay, so after a goods night sleep, this seems to be more of a misleading netcat output. The same scan with nmap -sU -v <IP> -p 53 shows the port as open|filtered, which is according to nmap No response received (even after retransmissions) (see https://nmap.org/book/scan-methods-udp-scan.html).

This is speculation and I did not find any evidence, just my own experience:
AFAIK this changed a few ehmm ... decades back. IIRC "in former times", I got a "port unreachable" back if a port was closed, but nowadays the package just got dropped unless you explicitely have an iptables rule that rejects the package instead of drop to that you will get the "port unreachable" back. This correspeonds to the "open|filered" logic.

 

[Chris]

This is speculation and I did not find any evidence, just my own experience:
AFAIK this changed a few ehmm ... decades back. IIRC "in former times", I got a "port unreachable" back if a port was closed, but nowadays the package just got dropped unless you explicitely have an iptables rule that rejects the package instead of drop to that you will get the "port unreachable" back. This correspeonds to the "open|filered" logic.

@LnxBil there is actually a quite nice explanation in the nmap book i linked to, right under table 5.3 they describe the nmap output behavior in more detail.

 

[gctwnl]

This is speculation and I did not find any evidence, just my own experience:
AFAIK this changed a few ehmm ... decades back. IIRC "in former times", I got a "port unreachable" back if a port was closed, but nowadays the package just got dropped unless you explicitely have an iptables rule that rejects the package instead of drop to that you will get the "port unreachable" back. This correspeonds to the "open|filered" logic.

Hmm.

The PVE host 192.168.2.50 does not listen on UDP 50-55
The PVE VM client 192.168.2.125 does listen on UDP 53 and not on the rest of UDP 50-55

It turns out there is a difference between UDP and TCP.

gerben@hermione% nc -v -z -u 192.168.2.50 50-55
Connection to 192.168.2.50 port 50 [udp/re-mail-ck] succeeded!
Connection to 192.168.2.50 port 51 [udp/la-maint] succeeded!
Connection to 192.168.2.50 port 52 [udp/xns-time] succeeded!
Connection to 192.168.2.50 port 53 [udp/domain] succeeded!
Connection to 192.168.2.50 port 54 [udp/xns-ch] succeeded!
Connection to 192.168.2.50 port 55 [udp/isi-gl] succeeded!
gerben@hermione% nc -v -z 192.168.2.50 50-55
nc: connectx to 192.168.2.50 port 50 (tcp) failed: Connection refused
nc: connectx to 192.168.2.50 port 51 (tcp) failed: Connection refused
nc: connectx to 192.168.2.50 port 52 (tcp) failed: Connection refused
nc: connectx to 192.168.2.50 port 53 (tcp) failed: Connection refused
nc: connectx to 192.168.2.50 port 54 (tcp) failed: Connection refused
nc: connectx to 192.168.2.50 port 55 (tcp) failed: Connection refused

 

[LnxBil]

@LnxBil there is actually a quite nice explanation in the nmap book i linked to, right under table 5.3 they describe the nmap output behavior in more detail.

Yes, this reflects the current state of the art and it states that in 2004 the nmap behaviour changed. I had a similar response with the ssh client that now times out if there is no SSH server running, this was different in pre-wheezy clients in which it directly said "connection refused".

 

[Chris]

Yes, this reflects the current state of the art and it states that in 2004 the nmap behaviour changed. I had a similar response with the ssh client that now times out if there is no SSH server running, this was different in pre-wheezy clients in which it directly said "connection refused".

Yes, so if one wants the rejection response, the iptables rule as you suggested is the way to go