[Solved] PVE 4.0 DNS settings disapearing

Discussion in 'Proxmox VE: Installation and configuration' started by auser, Oct 19, 2015.

  1. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    [Solved]

    [TLDR] When installing Debian Minimal do not select the option for IPv6 DNS

    In a default Debian installation /etc/resolv.conf is autogenerated by the package named 'resolvconf' for IPv4
    We already know not to install packages like network-manager or resolvconf because they will conflict with PVE
    Similarly for IPv6 the package that overwrites /etc/resolv.conf is 'rdnssd'

    apt-get remove rdnssd resolvconf network-manager

    [/TLDR]

    I can configure DNS settings via the GUI DNS / Edit / Dialog box 'Search Domain' and 'DNS Server'
    and DNS resolution then works for a time:
    Code:
    root@deb82:~# cat /etc/resolv.conf  
    search wnr2200.lan  
    nameserver 192.168.11.1  
    nameserver 192.168.5.1
    However at some point /etc/resolv.conf is regenerated and the details are lost.
    I have searched the forums and google and the 2 most helpful answers were:
    Thread: Hostname Issue
    https://forum.proxmox.com/threads/8715-Hostname-Issue
    and
    How do I include lines in resolv.conf that won't get lost on reboot? https://askubuntu.com/questions/157...s-in-resolv-conf-that-wont-get-lost-on-reboot

    There are 2 NICs for which I configured 2 Linux Bridges:
    and have tried adding to the bridges iface stanzas
    dns-search wnr2200.lan
    dns-nameservers 192.168.5.1

    in /etc/network/interfaces

    But cannot resolve this problem.
    Code:
    root@deb82:~# ls -al /etc/network/interfaces.d/   
    total 8   
    drwxr-xr-x 2 root root 4096 Mar 13  2015 .  
    drwxr-xr-x 7 root root 4096 Oct 16 21:44 ..
    
    root@deb82:~# pveversion    
    pve-manager/4.0-50/d3a6b7e5 (running kernel: 4.2.2-1-pve)
    I have been testing PVE4 since Beta1 and Beta2 and DNS worked ok on other boxes but now I see this problem on a box that I recently freshly installed.

    I followed the guide: Install Proxmox VE on Debian Jessie after doing a fresh install of Debian 8.2
     
    #1 auser, Oct 19, 2015
    Last edited: Oct 21, 2015
  2. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    The forum ate all my formatting. :-( I tried to fix it twice now , but it keeps appearing mangled without newlines. :-( Any help?
     
  3. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,292
    Likes Received:
    187
    Re: PVE 4.0 DNS settings disapearing

    Can you give directions to exactly reproduce the problem?
    Does
    1. Edit DNS entries via GUI
    2. Reboot
    3. now the changes are lost

    works for you?



    btw.:
    [noparse]
    Code:
     log, command line output or text here
    [/noparse]
    this may help.

    Also look in the forum editors panel on the left side if you are in the "WYSIWYG" mode.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    13,635
    Likes Received:
    418
    Re: PVE 4.0 DNS settings disapearing

    do not use https.

    (a known bug and we work on a fix).
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    1. Edit DNS entries via GUI
    2. Wait a few minutes ( or reboot )
    3. now the changes are lost

    Exactly. Even if I reboot immediately upon entering DNS details, they are gone.

    btw.:
    [noparse]
    Code:
     log, command line output or text here
    [/noparse]
    this may help.

    Thanks, changing to the simpler editor seems to have helped :)
     
  6. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,292
    Likes Received:
    187
    Re: PVE 4.0 DNS settings disapearing

    Hmm, did some test here, cannot reproduce that behaviour, did several reboots and waited some time...
    Waiting and reboot let the changes, and everything is working stay as intended.
    I also tried to reproduce it on a stock Jessie.

    After you save the changes via GUI the changes are always visible in the file?
    Proxmox writes the config only in one place and that's the 'update_dns' API call which the GUI (and only the gui) also uses.

    So the rewrite of the config should not happen from Proxmox side.

    /etc/network/interfaces although will be regenerated sometimes (from Debian also), there I would believe such lost changes a bit more.

    Does "Wait a few minutes" mean that even when you do nothing on the system the file gets reset? That would be strange, do the logs say something?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    Hi Thomas
    Yes - the new details are visible in /etc/resolv.conf as expected.
    I can see the details disappear from the text fields in my browser when the gui updates.

    Yeah, but the rewriting of /etc/network/interfaces has been fine so far
    My commented out lines are still there. ;-)

    Yes. Even when I do nothing but watch the async update of the browser GUI.
    Details vanish in a few minutes. (e.g. 2 minutes, don't need to wait 10 minutes)

    I have not noticed anything useful.
    I monitored systemd log but didn't notice anything related.

    Are there any other logs would you suggest?

    Also - and I am a total novice at this - but isn't there a clever way to put a 'monitor' on a file descriptor and that way we could get a notification when the problematic update occurs and thus see what process was responsible ?

    Thanks for your help. :)
     
  8. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    So I tried to write a script to get notified when /etc/resolv.conf gets updated using:
    http://entrproject.org/

    root@deb82:~/admin# apt-cache show entr

    apt-get install entr
    ...
    Description-en: Run arbitrary commands when files change
    The Event Notify Test Runner (entr) runs arbitrary commands when
    files change. Changes are detected through the kqueue/inotify
    kernel interface.

    myscript.sh
    Code:
    #!/bin/bash
    
    # do something when called by entr
    #less
    timeNow=$(date +"%F_%H:%M:%S")
    echo "date: $timeNow"
    
    pid="$$"
    echo "pid: $pid"
    
    #ps aux | grep "$pid"
    
    #pwd
    filepath='/etc/network'
    filename="$timeNow.txt"
    
    #ps aux >> myscript.out
    ps aux > "$filepath/$filename"
    
    called from a simple loop:
    root@deb82:/etc# while sleep 1; do ls -d /etc/resolv.conf | entr /etc/network/myscript.sh ; done

    date: 2015-10-19_17:35:46
    pid: 7912

    So now I can see when the (bad) update happens.
    I couldn't actually think of a really useful thing to put in the script :-(
    so I made it capture process information using 'ps aux' to a timestamped file.

    The script does get triggered, and I can attach the file of dumped process information.
    I don't know if this is of much use, but maybe it is a start.

    I had to zip it to get past the forums tiny file upload size limit.

    EDIT:
    So I left it run for a while
    Code:
    date: 2015-10-19_17:37:23
    pid: 8054
    date: 2015-10-19_17:42:17
    pid: 8445
    date: 2015-10-19_17:49:27
    pid: 9517
    date: 2015-10-19_17:56:10
    pid: 10039
    date: 2015-10-19_18:03:30
    pid: 10630
    date: 2015-10-19_18:06:52
    pid: 10909
    
    Edit2: no GUI interaction at all.
    Code:
    root@deb82:/etc# ^C
    root@deb82:/etc# while sleep 1; do ls -d /etc/resolv.conf | entr /etc/network/myscript.sh ; done
    date: 2015-10-19_18:12:32
    pid: 11365
    date: 2015-10-19_18:22:10
    pid: 12116
    date: 2015-10-19_18:31:06
    pid: 12829
    date: 2015-10-19_18:35:07
    pid: 13153
    date: 2015-10-19_18:43:53
    pid: 13861
    date: 2015-10-19_18:47:28
    pid: 14143
    date: 2015-10-19_18:54:17
    pid: 14677
    
    I do not see any corelation between the timestamps and
    /var/log/messages or /var/log/syslog
     

    Attached Files:

    #8 auser, Oct 19, 2015
    Last edited: Oct 19, 2015
  9. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    To recap:
    I have a test box which displays undesiraeble behaviour whereby all DNS information is deleted every few minutes.

    From the URLs I gave above, one can read that overwriting (autogeneration) of /etc/resolv.conf
    is expected behaviour when using the package named 'resolvconf'

    I have used dpkg to confirm that resolvconf is NOT installed.
    Code:
    root@deb82:~/admin# dpkg-query -l 'resolv*'
    Desired=Unknown/Install/Remove/Purge/Hold
    | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    ||/ Name                Version        Architecture   Description
    +++-===================-==============-==============-============================================
    un  resolvconf          <none>         <none>         (no description available)
    root@deb82:~/admin# 
    
    Confirmed by
    Code:
    root@deb82:~/admin# apt-get remove resolvconf
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Package 'resolvconf' is not installed, so not removed
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    root@deb82:~/admin# 
    
    While the DNS malfunction does makes the box effectively unuseable because it cannot find sites on the internet and thus apt cannot function,
    this is not an immediate problem for me as the test box was built to be part of a 3 node cluster, and I could just reinstall it.

    I am scheduled to do this tomorrow, as new hardware for the 3rd cluster node is due to arrive Wednesday 21st October.

    So if this issue is to be investigated further, the test machine is available today and tomorrow.

    Last night I left the test script running and collected some information:
    Code:
    while sleep 1; do ls -d /etc/resolv.conf | entr /etc/network/myscript.sh ; done
    
    [colin@k8 deb82]$ grep '^date:' ~/DEV/deb82/myscript.out | wc -l
    22
    
    This tells me that /etc/resolv.conf was modified 21 times in a period of 140 minutes
    which is ~ once every 7 minutes

    This morning I saw another perhaps interesting feature.

    When I started up the testbox and browsed the GUI to the tab 'DNS'

    I saw:
    'DNS Server1 fd1d:e747:7c7e::1' ???
    Code:
    root@deb82:~/temp_test# cat /etc/resolv.conf
    nameserver fd1d:e747:7c7e::1
    
    I ran a quick system wide check, for this string

    Code:
    root@deb82:~/temp_test# find / -xdev -type f -exec grep -l "fd1d:e747:7c7e::1" /dev/null {} \;
    /var/log/installer/syslog
    /etc/resolv.conf
    root@deb82:~/temp_test# 
    
    
    root@deb82:/var/log/installer# ls -alrt
    total 1672
    -rw-------  1 root root   30807 Oct 14 12:53 Xorg.0.log
    -rw-------  1 root root  448441 Oct 14 12:53 syslog
    -rw-------  1 root root 1095570 Oct 14 12:53 partman
    -rw-r--r--  1 root root   70138 Oct 14 12:53 status
    -rw-r--r--  1 root root     158 Oct 14 12:53 lsb-release
    -rw-r--r--  1 root root   39987 Oct 14 12:53 hardware-summary
    drwxr-xr-x  2 root root    4096 Oct 14 12:53 cdebconf
    drwxr-xr-x  3 root root    4096 Oct 14 12:53 .
    drwxr-xr-x 13 root root    4096 Oct 19 09:49 ..
    root@deb82:/var/log/installer# 
    
    root@deb82:/var/log/installer# grep 'fd1d:e747:7c7e::1' /var/log/installer/syslog
    Oct 14 11:06:04 netcfg[5203]: DEBUG: rdisc6 line:  Recursive DNS server     : fd1d:e747:7c7e::1
    Oct 14 11:06:04 netcfg[5203]: DEBUG: Read nameserver fd1d:e747:7c7e::1
    root@deb82:/var/log/installer# 
    
    To aid in automated diagnosis, I have created a simple test script which I have used to collect some data.

    I improved the small script quoted above:
    Code:
    apt-get install entr
    
    
    mkdir /root/temp_test   # script requires an absolute path.
    cd /root/temp_test
    
    onUpdate.sh
    Code:
    #!/bin/bash
    
    ## do something when called by entr
    
    timeNow=$(date +"%F_%H:%M:%S")
    pid="$$"
    
    echo "date: $timeNow | pid: $pid"
    
    if [ "$#" -gt 0 ]; then
    
       filename="$1"
    
       if [ -s "$filename" ]; then
       # file exists and is not empty.
          stat "$1"
       fi
    
       blocks=$(stat --format="%b" "$1")
    
       if [ "$blocks" -gt 0 ]; then
          sha1sum "$1"
       else
       # file contains no data
       # so dont fill the console with repeated stat data
          inode=$(stat --format="%i" "$1")
          access=$(stat --format="%x" "$1")
          modify=$(stat --format="%y" "$1")
          change=$(stat --format="%z" "$1")
    
          echo "Inode: $inode"
          echo "Access: $access"
          echo "Modify: $modify"
          echo "Change: $change"
       fi
    
    fi
    
    #filepath='~/temp_test'
    filepath='/root/temp_test'
    filename="$timeNow.txt"
    
    ps aux > "$filepath/$filename"
    
    run this:

    root@deb82:~/admin# while sleep 1; do ls /etc/resolv.conf | entr /root/temp_test/onUpdate.sh /_; done

    From a brief run of the updated script this morning
    Code:
    $ grep '^date:' ~/DEV/deb82/onUpdate.sh.out 
    date: 2015-10-20_11:38:01 | pid: 23916
    date: 2015-10-20_11:39:05 | pid: 24005
    date: 2015-10-20_11:44:38 | pid: 24445
    date: 2015-10-20_11:47:14 | pid: 24666
    
    $ grep '^date:' ~/DEV/deb82/onUpdate.sh.out | wc -l
    4
    
    This shows that /etc/resolv.conf has been updated 3 times in 9 minutes.
     
    #9 auser, Oct 20, 2015
    Last edited: Oct 21, 2015
  10. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    PVE5_DNS_showing_hex4.png PVE5_DNS_showing_hex3.png
     

    Attached Files:

  11. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
    Re: PVE 4.0 DNS settings disapearing

    So I may have found the package causing the problem.
    Code:
    root@deb82:~/admin# apt-cache search rdnssd
    rdnssd - IPv6 recursive DNS server discovery daemon
    
    root@deb82:~/admin# apt-cache show rdnssd
    Package: rdnssd
    Source: ndisc6 (1.0.1-1)
    Version: 1.0.1-1+b1
    Installed-Size: 108
    Maintainer: Rémi Denis-Courmont <remi@remlab.net>
    Architecture: amd64
    Depends: libc6 (>= 2.4)
    Pre-Depends: adduser
    Recommends: resolvconf
    Suggests: ndisc6
    Description-en: IPv6 recursive DNS server discovery daemon
     rdnssd autoconfigures recursive DNS servers on IPv6 networks
     using ICMPv6 Neighbor Discovery (RFC 5006), and [B]can update the
     DNS resolvers configuration (/etc/resolv.conf)[/B] accordingly.
    Description-md5: 7f6af1fa6e3e8a6e206f1e1d3aa62701
    Homepage: [url]http://www.remlab.net/ndisc6/[/url]
    Tag: implemented-in::c, interface::daemon, network::server, protocol::dns,
     role::program
    Section: net
    Priority: optional
    Filename: pool/main/n/ndisc6/rdnssd_1.0.1-1+b1_amd64.deb
    <snip>
    
    Solution
    Code:
    root@deb82:~/admin# apt-get remove rdnssd 
    
    I have left the monitoring script running.
    So far the correct /etc/resolv.conf has not been overwritten. :)

    I will hope that the contents of this thread may serve as a useful breadcrumb trail for anyone else who
    may encounter this problem in the future. Much as the forum thread I listed above did for me. :)

    Also this might be a good point to suggest a small addition to the wiki
    Install Proxmox VE on Debian Jessie
    At the bottom, perhaps after the section

    'Make sure network-manager is not used, else pve-cluster will not start in some cases'

    It might be useful to add something like:
    "The PVE4 GUI expects to control DNS management and will no longer take its DNS settings from /etc/network/interfaces
    Any package that autogenerates (overwrites) /etc/resolv.conf will cause DNS to fail.
    e.g. packages 'resolvconf' for IPv4 and 'rdnssd' for IPv6

    Also, wouldn't it be possible to add this information into the appropriate Proxmox package(s) when they are composed?

    i.e. in the DEB metadata add 'conflicts: rdnssd' (and perhaps similarly 'conflicts: resolvconf') ?

    Should this maybe be filed as an RFE somewhere?

    p.s. I cannot see how to edit the thread title.
    Thomas - perhaps you could change it to [Solved]
     
    #11 auser, Oct 20, 2015
    Last edited: Oct 20, 2015
  12. auser

    auser New Member

    Joined:
    Jul 17, 2014
    Messages:
    28
    Likes Received:
    1
  13. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,292
    Likes Received:
    187
    Re: PVE 4.0 DNS settings disapearing

    Good that you solved this, sorry had a day off, I added your proposed line to the wiki. Thanks for the report!

    AFAIK, you can mark the thread as solved, at the top of the thread page there should be a "Thread tools" button where the respective options is.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice