Best practice for *extra* high availability of critical service?

[n8ur]

I have a three node cluster with HA and glusterfs and thought I would move my internal DNS from two raspberry pis to it. I set up three LXC containers running named to provide lookups for my internal domain, and three containers running pihole to serve as caching resolvers. Each node normally runs one pair of DNS and pihole, and the containers are configured with HA and stored on glusterFS. One of the DNS servers is master and the others pull their zone files from it. The three piholes all look to the three DNS servers for local lookups. My DHCP server gives clients the addresses of the three piholes. All this worked fine until I had some glusterfs networking issues. The result was that my whole DNS system came crashing down and while I was able to recover it, the downtime was quite annoying.

I'm now rebuilding the cluster with a minor change -- this time, putting one of the DNA/pihole pairs on local storage (call them the "standalone" servers. The idea is that even if glusterfs goes out, the third node can still provide DNS service independently. The one concern I have with this is what happens if the standalone DNS restarts and is unable to download a zone file from the master on restart, or the zone TTL expires. That would cause it to stop serving local data. My current thinking is to change the third DNS server from a slave to a master, and use a cron job to download the zone files from the real master every hour or two. That way the third pair would be truly independent.

Are there other, better, ways to deal with this sort of thing?

Thanks,
John

 

[alexskysilk]

https://kb.isc.org/docs/managing-manual-multi-master

 

[Johannes S]

Hello,

up to now I didn't had time to try it out but I can remeber that keepalived was recommended together with pi-hole here.
Google-Results seems promising: https://www.google.com/search?q=keepalived+pihole

If I understand it correctly you would point your DHCP to the keepalived virtual IP while keepalived will talk to the pi-holes, which are synced with gravity.

Or did I miss something in your usecase?

Hope this helps and best regards, Johannes.

 

[n8ur]

Thanks for the pointers to the two different (but both helpful) topics! I'm already using keepalived on the pve nodes to establish a virtual IP so I can get to the GUI without having to test nodes. I'd seen an earlier reference to it being used to keep pihole blocker databases syncd, so need to follow that up.

My DNS configuration is "simple" in that I only manage one zone, it's only used internally, and the zone file updates very infrequently unless I'm messing around (and I'm the only one who updates it). I had a chance to test this setup today when I had to power down the cluster and it actually worked as planned... the named and pihole containers on node 3 came up despite the fact the glusterfs filesystem didn't automount and the HA services were throwing errors as a result.

I'm just running an hourly cron script that copies the zone files from the "real" primary to named on the third host, and then reloads named. Given how static the files are, hourly seems plenty often. The pihole already has all three named instances listed as conditional forwarders, so it doesn't need any special configuration.

Now I need to figure out how to get the glusterfs filesystem to properly mount at boot. From what I've been able to research, it's a sequence issue between glusterd and some other proxmox service starting. A cure is to use a systemd service override to cause the filesystem to be mounted from localhost before the rest of proxmox comes up, but I haven't found enough details about that to feel comfortable trying it. Details at https://stanislas.blog/2018/10/how-to-mount-local-glusterfs-volume-boot-fstab-systemd-fix/

 

[waltar]

Take a look at https://forum.proxmox.com/threads/condition-vm-start-on-glusterfs-reachability.154087/ hopefully helping you.

 

[n8ur]

Thank you, Johannes! That will be really helpful info.