[SOLVED] Hight rate of dns queries for PBS from PVE

E

Etienne Charlier

Well-Known Member
Oct 29, 2018
75
17
48
22
Dear,

I installed adguardhome as my home network dns "server"

I have a homelab with a pve server hosting a couple of vm ( nextcloud, influxdb...)
those vm are backuped to remote PBD instance ( kindly offered by tuxis.nl) and to a local PBS ( lorath-adm.phi8.ovh)

the statistics page of adguardhome show that almos 45% of DNS requests served by adguardhome are for the two PBS hostnames.

Is this normal ?
How/where can I start investigating/diagnosing this issue ?

Thanks in adance
EC

Capture d’écran 2022-11-05 à 12.21.31.png
12:26:18 11/5/2022 monitor.phi8.ovh Type: AAAA, Plain DNS Processed 0.56 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.21%22']172.19.1.21[/URL] 12:26:13 11/5/2022 lorath-adm.phi8.ovh Type: AAAA, Plain DNS Processed 0.41 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:13 11/5/2022 lorath-adm.phi8.ovh Type: A, Plain DNS Processed 0.12 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:13 11/5/2022 pbs001.tuxis.nl Type: AAAA, Plain DNS Processed 0.21 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:13 11/5/2022 pbs001.tuxis.nl Type: A, Plain DNS Processed 0.17 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:13 11/5/2022 pbs001.tuxis.nl Type: AAAA, Plain DNS Processed 0.08 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:13 11/5/2022 pbs001.tuxis.nl Type: A, Plain DNS Processed 0.12 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:08 11/5/2022 monitor.phi8.ovh Type: AAAA, Plain DNS Processed 0.44 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.21%22']172.19.1.21[/URL] 12:26:05 11/5/2022 pbs001.tuxis.nl Type: A, Plain DNS Processed 0.08 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:05 11/5/2022 pbs001.tuxis.nl Type: AAAA, Plain DNS Processed 0.16 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] 12:26:05 11/5/2022 pbs001.tuxis.nl Type: AAAA, Plain DNS Processed 0.08 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL] Block 12:26:05 11/5/2022 pbs001.tuxis.nl Type: A, Plain DNS Processed 0.13 ms [URL='http://thewall.phi8.ovh:3080/#logs?search=%22172.19.1.100%22']172.19.1.100[/URL]
 
  • Like
Reactions: GNMYT and Nightman
PVE keeps a graph of the storage space and contacts the PBS regularly (and rather often). The PBS is configured with an URL and PVE does not know how often the PBS changes its IP address. Therefore it does a DNS request every time, which is not uncommon (browsers do it all the time when you refresh a webpage). If you are using a caching DNS server then this should not be a problem.
 
I've observed exatly the same (4 req every 10 sec). I've put an entry for the pbs server to the /etc/hosts file of the pve and it is not polluting the AdGuardHome log/stat anymore. I guess that pbs server IP is not changing frequently.
 
  • Like
Reactions: ESMP
I solved this with a new entry on /etc/hosts and a script that executes every hour, just in case Tuxis change there IP.

Just change the website value on the script... chmod +x and add it to a cron job...

Bash: 
#!/bin/bash

# Define the website URL and the hosts file path
website="XXXXXXXXXXX.tuxis.nl"
hosts_file="/etc/hosts"

# Use dig to retrieve the current IP address
current_ip=$(dig +short "$website" | grep '^[.0-9]*$')

# Check if the IP address is already in the hosts file
if grep -q "$website" "$hosts_file"; then
    # Get the current IP address from the hosts file
    hosts_ip=$(grep "$website" "$hosts_file" | awk '{print $1}')

    # Compare the current IP with the one in the hosts file
    if [ "$current_ip" != "$hosts_ip" ]; then
        # Update the hosts file with the new IP
        sed -i "s/$hosts_ip.*/$current_ip $website/" "$hosts_file"
        echo "Updated $website in $hosts_file from $hosts_ip to $current_ip"
    else
        echo "No change in IP address."
    fi
else
    # Append the new entry to the hosts file
    echo "$current_ip $website" >> "$hosts_file"
    echo "Added $website to $hosts_file with IP $current_ip"
fi
 
Last edited:
Hi @vdias,
thank you for the script.

Unfortunately I'm pretty new to the proxmox and Linux world. I understand the concept of what you are doing with this script but can't figure out how to create and run it.
Could you please specify that line for a total newbie:
chmod +x and add it to a cron job
Click to expand...
What do the shell commands look like?

Thank you,
Hans
 
So a DNS server is supposed to answer DNS queries. The Adguard or PiHole is usually a local DNS server and caches the queries anyway so that they never reach public DNS servers in bulk.

Now building a script to avoid DNS requests is somehow really absurd. You can also avoid creating backups because it takes up traffic and storage space. It's best not to set up any services in the internal network if you're afraid that they can also be used.
 
  • Like
Reactions: HansJerkov
First op: Thank you for this registered on the forums to reply here. This one entry has accounted for 78k requests in the past 24 hours.

Second, It's not absurd when it's requiring extra processing power and 40x the load on that ONE request hitting 78k times. How is this OK!? I run a 4 cluster and this was the result.

It's a design flaw and constantly puts my network under a heavy load.

Uptime Kuma is the second at 6296 in the past 24 hours checking every 10 minutes.

1732649989490.png
sb-jw said:
So a DNS server is supposed to answer DNS queries. The Adguard or PiHole is usually a local DNS server and caches the queries anyway so that they never reach public DNS servers in bulk.

Now building a script to avoid DNS requests is somehow really absurd. You can also avoid creating backups because it takes up traffic and storage space. It's best not to set up any services in the internal network if you're afraid that they can also be used.
Click to expand...
 
Last edited:
  • Like
Reactions: Jeff Chien and Nightman
From my point of view this behavior (4 DNS request [2x A record, 2x AAA record] every 10 seconds for each pbs connection) seems from my point of view not to be a good design. I would assume that DNS queries for pbs are only made initially and then each time a connection attempt using the IP failed. Then we need to know the current IP by asking DNS again.

Could some people from proxmox dev look into this please.

I am impacted by this, too.
 
Last edited:
  • Like
Reactions: gr3n
I just stumbled upon this "solved" threat. Can someone explain to me, how this is solved?
This is obviously a design flaw. Probably one you can work around but obviously one that can be resolved as well.

Why would 4 DNS requests per 10 seconds be a good design decission? I understand, that there is no DNS caching implemented on default on proxmox, but perhaps it's a good idea to spend a few minutes to think about this. I can't see how this is an expected behavior.

I have 2 PBS instanced configured and the DNS lookups for them are 40% of my DNS traffic overall.
 
So PVE sends 0.4 request/sec, x2 = 0.8 request/sec. If that is 40% of your DNS traffic, then we're talking 2 requests/sec in total. Truly an enterprise-grade setup. How will your server withstand the onslaught. What is it, an Altair 8080 or something?

BTW, have you noticed how Windows behaves on a network? I think maybe a chill pill is in order.

More seriously, see @leesteken post above for why it might be doing this. I'm sure your patches to improve the situation are welcome.
 
Last edited:
alanshore said:
First op: Thank you for this registered on the forums to reply here. This one entry has accounted for 78k requests in the past 24 hours.
Click to expand...
There are 86400 seconds in a day, so that is less than one per second.

alanshore said:
It's a design flaw and constantly puts my network under a heavy load.
Click to expand...

One request per second is a "heavy load"? WTH?
 
In Linux systems there is a specific order in which DNS resolution is done:
Code: 
cat /etc/nsswitch.conf|grep hosts
hosts:          files dns

Each "service" is a caching device of a sort. The files (/etc/hosts) can cache your most used entry, the DNS service caches remote entries for TTL specified. If you want to avoid DNS traffic on your network you have two options:
- As mentioned previously - plug your PBS into /etc/hosts file
- Implement a local DNS caching mechanism. Keep in mind that the cache will only be alive for TTL.

PVE is a hypervisor suite, DNS is outside of its intended purpose. If you want to optimize your network - Linux is very flexible.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
BobhWasatch said:
So PVE sends 0.4 request/sec, x2 = 0.8 request/sec. If that is 40% of your DNS traffic, then we're talking 2 requests/sec in total. Truly an enterprise-grade setup. How will your server withstand the onslaught. What is it, an Altair 8080 or something?

BTW, have you noticed how Windows behaves on a network? I think maybe a chill pill is in order.

More seriously, see @leesteken post above for why it might be doing this. I'm sure your patches to improve the situation are welcome.
Click to expand...
So, "whataboutism" is the solution to bad design decissions?

I get what you mean - it's probably not that important. Still it's very weird if your stats look like this:

1754940192120.png

With 1 and 2 being PBS with factor 20 DNS requests over the rest.

Closing this as "solved" where nothing is solved is not what I would expect from any dev/admin that has any respect for his code.

Is it a serious flaw? Probably not. Is it bad design? Yes it is. And whoever defends this with "whataboutism" is prooving it. There's no need for those many requests, there's no design or architecture behind it. It was just implemented with a "works for me" mindset and that's what gets ppl. upset.
 
Nightman said:
From my point of view this behavior (4 DNS request [2x A record, 2x AAA record] every 10 seconds for each pbs connection) seems from my point of view not to be a good design. I would assume that DNS queries for pbs are only made initially and then each time a connection attempt using the IP failed. Then we need to know the current IP by asking DNS again.
Click to expand...
Yes, it sounds easy, but it complicates the logic. There are lots of ways for a network connection to fail, various cases to handle. Maybe the target is on a round-robin DNS so you _should_ do the query each time for that to work right, to give one example of a failure case you may not have considered.

It is much easier and more robust to just do the lookup each time. It is easier to code, it is easier to test, and it just doesn't matter if it a lookup happens only once a second. It is a trivial load for any DNS compared to what the rest of the network will be hitting it with (on a real network, not something at home).

Nightman said:
I am impacted by this, too.
Click to expand...
You are not "impacted" in any measurable way.
 
You must log in or register to reply here.
Top Bottom