[Solved] PVE 4.0 DNS settings disapearing

[auser]

[Solved]

[TLDR] When installing Debian Minimal do not select the option for IPv6 DNS

In a default Debian installation /etc/resolv.conf is autogenerated by the package named 'resolvconf' for IPv4
We already know not to install packages like network-manager or resolvconf because they will conflict with PVE
Similarly for IPv6 the package that overwrites /etc/resolv.conf is 'rdnssd'

apt-get remove rdnssd resolvconf network-manager

[/TLDR]

I can configure DNS settings via the GUI DNS / Edit / Dialog box 'Search Domain' and 'DNS Server'
and DNS resolution then works for a time:

root@deb82:~# cat /etc/resolv.conf  
search wnr2200.lan  
nameserver 192.168.11.1  
nameserver 192.168.5.1

However at some point /etc/resolv.conf is regenerated and the details are lost.
I have searched the forums and google and the 2 most helpful answers were:
Thread: Hostname Issue
https://forum.proxmox.com/threads/8715-Hostname-Issue
and
How do I include lines in resolv.conf that won't get lost on reboot? https://askubuntu.com/questions/157...s-in-resolv-conf-that-wont-get-lost-on-reboot

There are 2 NICs for which I configured 2 Linux Bridges:
and have tried adding to the bridges iface stanzas
dns-search wnr2200.lan
dns-nameservers 192.168.5.1

in /etc/network/interfaces

But cannot resolve this problem.

root@deb82:~# ls -al /etc/network/interfaces.d/   
total 8   
drwxr-xr-x 2 root root 4096 Mar 13  2015 .  
drwxr-xr-x 7 root root 4096 Oct 16 21:44 ..

root@deb82:~# pveversion    
pve-manager/4.0-50/d3a6b7e5 (running kernel: 4.2.2-1-pve)

I have been testing PVE4 since Beta1 and Beta2 and DNS worked ok on other boxes but now I see this problem on a box that I recently freshly installed.

I followed the guide: Install Proxmox VE on Debian Jessie after doing a fresh install of Debian 8.2

 

[auser]

Re: PVE 4.0 DNS settings disapearing

The forum ate all my formatting. :-( I tried to fix it twice now , but it keeps appearing mangled without newlines. :-( Any help?

 

[t.lamprecht]

Re: PVE 4.0 DNS settings disapearing

Can you give directions to exactly reproduce the problem?
Does
1. Edit DNS entries via GUI
2. Reboot
3. now the changes are lost

works for you?



btw.:
[noparse]

 log, command line output or text here

[/noparse]
this may help.

Also look in the forum editors panel on the left side if you are in the "WYSIWYG" mode.

 

[tom]

Re: PVE 4.0 DNS settings disapearing

The forum ate all my formatting. :-( I tried to fix it twice now , but it keeps appearing mangled without newlines. :-( Any help?

do not use https.

(a known bug and we work on a fix).

 

[auser]

Re: PVE 4.0 DNS settings disapearing

Can you give directions to exactly reproduce the problem?
Does
1. Edit DNS entries via GUI
2. Reboot
3. now the changes are lost

works for you?

1. Edit DNS entries via GUI
2. Wait a few minutes ( or reboot )
3. now the changes are lost

Exactly. Even if I reboot immediately upon entering DNS details, they are gone.

btw.:
[noparse]

 log, command line output or text here

[/noparse]
this may help.

Also look in the forum editors panel on the left side if you are in the "WYSIWYG" mode.

Thanks, changing to the simpler editor seems to have helped

 

[t.lamprecht]

Re: PVE 4.0 DNS settings disapearing

Hmm, did some test here, cannot reproduce that behaviour, did several reboots and waited some time...
Waiting and reboot let the changes, and everything is working stay as intended.
I also tried to reproduce it on a stock Jessie.

After you save the changes via GUI the changes are always visible in the file?
Proxmox writes the config only in one place and that's the 'update_dns' API call which the GUI (and only the gui) also uses.

So the rewrite of the config should not happen from Proxmox side.

/etc/network/interfaces although will be regenerated sometimes (from Debian also), there I would believe such lost changes a bit more.

Does "Wait a few minutes" mean that even when you do nothing on the system the file gets reset? That would be strange, do the logs say something?

 

[auser]

Re: PVE 4.0 DNS settings disapearing

Hi Thomas

After you save the changes via GUI the changes are always visible in the file?
Proxmox writes the config only in one place and that's the 'update_dns' API call which the GUI (and only the gui) also uses.

Yes - the new details are visible in /etc/resolv.conf as expected.
I can see the details disappear from the text fields in my browser when the gui updates.

/etc/network/interfaces although will be regenerated sometimes (from Debian also), there I would believe such lost changes a bit more.

Yeah, but the rewriting of /etc/network/interfaces has been fine so far
My commented out lines are still there. ;-)

Does "Wait a few minutes" mean that even when you do nothing on the system the file gets reset? That would be strange, do the logs say something?

Yes. Even when I do nothing but watch the async update of the browser GUI.
Details vanish in a few minutes. (e.g. 2 minutes, don't need to wait 10 minutes)

I have not noticed anything useful.
I monitored systemd log but didn't notice anything related.

Are there any other logs would you suggest?

Also - and I am a total novice at this - but isn't there a clever way to put a 'monitor' on a file descriptor and that way we could get a notification when the problematic update occurs and thus see what process was responsible ?

Thanks for your help.

 

[auser]

Re: PVE 4.0 DNS settings disapearing

So I tried to write a script to get notified when /etc/resolv.conf gets updated using:
http://entrproject.org/

root@deb82:~/admin# apt-cache show entr

apt-get install entr
...
Description-en: Run arbitrary commands when files change
The Event Notify Test Runner (entr) runs arbitrary commands when
files change. Changes are detected through the kqueue/inotify
kernel interface.

myscript.sh

#!/bin/bash

# do something when called by entr
#less
timeNow=$(date +"%F_%H:%M:%S")
echo "date: $timeNow"

pid="$$"
echo "pid: $pid"

#ps aux | grep "$pid"

#pwd
filepath='/etc/network'
filename="$timeNow.txt"

#ps aux >> myscript.out
ps aux > "$filepath/$filename"

called from a simple loop:
root@deb82:/etc# while sleep 1; do ls -d /etc/resolv.conf | entr /etc/network/myscript.sh ; done

date: 2015-10-19_17:35:46
pid: 7912

So now I can see when the (bad) update happens.
I couldn't actually think of a really useful thing to put in the script :-(
so I made it capture process information using 'ps aux' to a timestamped file.

The script does get triggered, and I can attach the file of dumped process information.
I don't know if this is of much use, but maybe it is a start.

I had to zip it to get past the forums tiny file upload size limit.

EDIT:
So I left it run for a while

date: 2015-10-19_17:37:23
pid: 8054
date: 2015-10-19_17:42:17
pid: 8445
date: 2015-10-19_17:49:27
pid: 9517
date: 2015-10-19_17:56:10
pid: 10039
date: 2015-10-19_18:03:30
pid: 10630
date: 2015-10-19_18:06:52
pid: 10909

Edit2: no GUI interaction at all.

root@deb82:/etc# ^C
root@deb82:/etc# while sleep 1; do ls -d /etc/resolv.conf | entr /etc/network/myscript.sh ; done
date: 2015-10-19_18:12:32
pid: 11365
date: 2015-10-19_18:22:10
pid: 12116
date: 2015-10-19_18:31:06
pid: 12829
date: 2015-10-19_18:35:07
pid: 13153
date: 2015-10-19_18:43:53
pid: 13861
date: 2015-10-19_18:47:28
pid: 14143
date: 2015-10-19_18:54:17
pid: 14677

I do not see any corelation between the timestamps and
/var/log/messages or /var/log/syslog

 

[auser]

Re: PVE 4.0 DNS settings disapearing

To recap:
I have a test box which displays undesiraeble behaviour whereby all DNS information is deleted every few minutes.

From the URLs I gave above, one can read that overwriting (autogeneration) of /etc/resolv.conf
is expected behaviour when using the package named 'resolvconf'

I have used dpkg to confirm that resolvconf is NOT installed.

root@deb82:~/admin# dpkg-query -l 'resolv*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version        Architecture   Description
+++-===================-==============-==============-============================================
un  resolvconf          <none>         <none>         (no description available)
root@deb82:~/admin#

Confirmed by

root@deb82:~/admin# apt-get remove resolvconf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package 'resolvconf' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@deb82:~/admin#

While the DNS malfunction does makes the box effectively unuseable because it cannot find sites on the internet and thus apt cannot function,
this is not an immediate problem for me as the test box was built to be part of a 3 node cluster, and I could just reinstall it.

I am scheduled to do this tomorrow, as new hardware for the 3rd cluster node is due to arrive Wednesday 21st October.

So if this issue is to be investigated further, the test machine is available today and tomorrow.

Last night I left the test script running and collected some information:

while sleep 1; do ls -d /etc/resolv.conf | entr /etc/network/myscript.sh ; done

[colin@k8 deb82]$ grep '^date:' ~/DEV/deb82/myscript.out | wc -l
22

This tells me that /etc/resolv.conf was modified 21 times in a period of 140 minutes
which is ~ once every 7 minutes

This morning I saw another perhaps interesting feature.

When I started up the testbox and browsed the GUI to the tab 'DNS'

I saw:
'DNS Server1 fd1d:e747:7c7e::1' ???

root@deb82:~/temp_test# cat /etc/resolv.conf
nameserver fd1d:e747:7c7e::1

I ran a quick system wide check, for this string

root@deb82:~/temp_test# find / -xdev -type f -exec grep -l "fd1d:e747:7c7e::1" /dev/null {} \;
/var/log/installer/syslog
/etc/resolv.conf
root@deb82:~/temp_test# 


root@deb82:/var/log/installer# ls -alrt
total 1672
-rw-------  1 root root   30807 Oct 14 12:53 Xorg.0.log
-rw-------  1 root root  448441 Oct 14 12:53 syslog
-rw-------  1 root root 1095570 Oct 14 12:53 partman
-rw-r--r--  1 root root   70138 Oct 14 12:53 status
-rw-r--r--  1 root root     158 Oct 14 12:53 lsb-release
-rw-r--r--  1 root root   39987 Oct 14 12:53 hardware-summary
drwxr-xr-x  2 root root    4096 Oct 14 12:53 cdebconf
drwxr-xr-x  3 root root    4096 Oct 14 12:53 .
drwxr-xr-x 13 root root    4096 Oct 19 09:49 ..
root@deb82:/var/log/installer# 

root@deb82:/var/log/installer# grep 'fd1d:e747:7c7e::1' /var/log/installer/syslog
Oct 14 11:06:04 netcfg[5203]: DEBUG: rdisc6 line:  Recursive DNS server     : fd1d:e747:7c7e::1
Oct 14 11:06:04 netcfg[5203]: DEBUG: Read nameserver fd1d:e747:7c7e::1
root@deb82:/var/log/installer#

To aid in automated diagnosis, I have created a simple test script which I have used to collect some data.

I improved the small script quoted above:

apt-get install entr


mkdir /root/temp_test   # script requires an absolute path.
cd /root/temp_test

onUpdate.sh

#!/bin/bash

## do something when called by entr

timeNow=$(date +"%F_%H:%M:%S")
pid="$$"

echo "date: $timeNow | pid: $pid"

if [ "$#" -gt 0 ]; then

   filename="$1"

   if [ -s "$filename" ]; then
   # file exists and is not empty.
      stat "$1"
   fi

   blocks=$(stat --format="%b" "$1")

   if [ "$blocks" -gt 0 ]; then
      sha1sum "$1"
   else
   # file contains no data
   # so dont fill the console with repeated stat data
      inode=$(stat --format="%i" "$1")
      access=$(stat --format="%x" "$1")
      modify=$(stat --format="%y" "$1")
      change=$(stat --format="%z" "$1")

      echo "Inode: $inode"
      echo "Access: $access"
      echo "Modify: $modify"
      echo "Change: $change"
   fi

fi

#filepath='~/temp_test'
filepath='/root/temp_test'
filename="$timeNow.txt"

ps aux > "$filepath/$filename"

run this:

root@deb82:~/admin# while sleep 1; do ls /etc/resolv.conf | entr /root/temp_test/onUpdate.sh /_; done

From a brief run of the updated script this morning

$ grep '^date:' ~/DEV/deb82/onUpdate.sh.out 
date: 2015-10-20_11:38:01 | pid: 23916
date: 2015-10-20_11:39:05 | pid: 24005
date: 2015-10-20_11:44:38 | pid: 24445
date: 2015-10-20_11:47:14 | pid: 24666

$ grep '^date:' ~/DEV/deb82/onUpdate.sh.out | wc -l
4

This shows that /etc/resolv.conf has been updated 3 times in 9 minutes.

 

[auser]

Re: PVE 4.0 DNS settings disapearing

I saw:
'DNS Server1 fd1d:e747:7c7e::1' ???

 

[auser]

Re: PVE 4.0 DNS settings disapearing

So I may have found the package causing the problem.

root@deb82:~/admin# apt-cache search rdnssd
rdnssd - IPv6 recursive DNS server discovery daemon

root@deb82:~/admin# apt-cache show rdnssd
Package: rdnssd
Source: ndisc6 (1.0.1-1)
Version: 1.0.1-1+b1
Installed-Size: 108
Maintainer: Rémi Denis-Courmont <remi@remlab.net>
Architecture: amd64
Depends: libc6 (>= 2.4)
Pre-Depends: adduser
Recommends: resolvconf
Suggests: ndisc6
Description-en: IPv6 recursive DNS server discovery daemon
 rdnssd autoconfigures recursive DNS servers on IPv6 networks
 using ICMPv6 Neighbor Discovery (RFC 5006), and [B]can update the
 DNS resolvers configuration (/etc/resolv.conf)[/B] accordingly.
Description-md5: 7f6af1fa6e3e8a6e206f1e1d3aa62701
Homepage: [url]http://www.remlab.net/ndisc6/[/url]
Tag: implemented-in::c, interface::daemon, network::server, protocol::dns,
 role::program
Section: net
Priority: optional
Filename: pool/main/n/ndisc6/rdnssd_1.0.1-1+b1_amd64.deb
<snip>

Solution

root@deb82:~/admin# apt-get remove rdnssd

I have left the monitoring script running.
So far the correct /etc/resolv.conf has not been overwritten.

I will hope that the contents of this thread may serve as a useful breadcrumb trail for anyone else who
may encounter this problem in the future. Much as the forum thread I listed above did for me.

Also this might be a good point to suggest a small addition to the wiki
Install Proxmox VE on Debian Jessie
At the bottom, perhaps after the section

'Make sure network-manager is not used, else pve-cluster will not start in some cases'

It might be useful to add something like:
"The PVE4 GUI expects to control DNS management and will no longer take its DNS settings from /etc/network/interfaces
Any package that autogenerates (overwrites) /etc/resolv.conf will cause DNS to fail.
e.g. packages 'resolvconf' for IPv4 and 'rdnssd' for IPv6

Also, wouldn't it be possible to add this information into the appropriate Proxmox package(s) when they are composed?

i.e. in the DEB metadata add 'conflicts: rdnssd' (and perhaps similarly 'conflicts: resolvconf') ?

Should this maybe be filed as an RFE somewhere?

p.s. I cannot see how to edit the thread title.
Thomas - perhaps you could change it to [Solved]

 

[auser]

Re: PVE 4.0 DNS settings disapearing

RFE created in the proxmox bugzilla

RFE: installer should warn if DNS cannot function

 

[t.lamprecht]

Re: PVE 4.0 DNS settings disapearing

Good that you solved this, sorry had a day off, I added your proposed line to the wiki. Thanks for the report!

AFAIK, you can mark the thread as solved, at the top of the thread page there should be a "Thread tools" button where the respective options is.