Proxmox 8 - Possible issue with DNS resolution and LXC

[societus]

Good evening guys,

I recently tried out Proxmox 8 on a couple spare machines before upgrading my production rigs, and may have discovered a possible issue, but cannot totally determine where the problem lies.

Issue: When host is connected to a Tailscale network, with or without accepting DNS, LXC containers created or started after it is connected fail to resolve DNS, and changes to DNS addresses from the Prox web-ui or resolv.conf in the LXC client are either not being recognized or otherwise ineffective.

Replication: Connected 3 nodes Proxmox 8 to a tailnet, both with the official Tailscale control server, or using a self hosted Headscale instance in the cloud. Debian 11,12, and Alpine Linux LXC containers fail to resolve DNS if they are started after the host is on the tailnet. The only resolution is to disable connection at startup and rebooting. While issue occurs, the host is still able to resolve DNS, and changes to resolv.conf and the web-ui for the host are accepted and register the changes correctly.

Additional changes attempted: Steps on guide for Tailscale in LXC were followed, all LXC containers intended to have Tailscale on directly have proper cgroup/tun access. Connecting a container to a tailnet while host is not seems to work okay, until the host is connected and DNS breaks immediately.

I have this setup on my production cluster using Prox 7, with hosts and containers separately connected as nodes in my tailnet, with no issues occurring to DNS. Not sure what else might help here, but hoping that this might bring attention to something that could be an issue, as I cannot risk this problem should I upgrade my main servers.

Thanks!

 

[antonionardella]

Hello @societus

I am having the same issue on Proxmox 8

Tailscale is not set up on the Proxmox host, but the LXC container stops DNS resolution as soon as tailscale is connected.

Did you find any solution to this?

 

[societus]

Hello @societus

I am having the same issue on Proxmox 8

Tailscale is not set up on the Proxmox host, but the LXC container stops DNS resolution as soon as tailscale is connected.

Did you find any solution to this?

Not directly a solution, but I have found a procedural workaround. As of the latest update, I have been able to get a test environment working past the bug, but it requires that the host first be attached to the tailnet in userspace, accepting subnet routes on the subnets the interfaces for VM/LXC uses. Afterwards the LXC containers have working networking long enough to connect IT to the tailnet, and not consistently, but most of the time, networking still works after logging the proxmox host out of the tailnet. It leads me to believe it could be tied to the uid/gid changes as it registers /dev/tun devices in the kernel, since LXC is not really designed for networking changes and negotiating native networking with partially virtualized or convergent bridges (that is more of a fat containers thing)

 

[fruchtzwerg]

I ran into the same problem several times. Is there any solution? Does this problem also attach using headscale?

 

[societus]

I ran into the same problem several times. Is there any solution? Does this problem also attach using headscale?

I’ve tried fresh installs every few months to see if the issue resolved, but have not found a proper fix yet, nor am I absolutely sure what the problem originates from. It doesn’t matter if it is headscale either, the problem replicates either way.

 

[fruchtzwerg]

Good to know that there are people out recognizing the issues.
I thought I had misconfigured the host twice and was beeing desparetly searching for my mistake.

The point ist: There are the limits of combining tailscale and proxmox. That's a pitty because I was so happy using the easy tailscale software for easy linking to backupdevices and pointer.
Is there any alternative out there for VPN-connecting?

 

[societus]

Good to know that there are people out recognizing the issues.
I thought I had misconfigured the host twice and was beeing desparetly searching for my mistake.

The point ist: There are the limits of combining tailscale and proxmox. That's a pitty because I was so happy using the easy tailscale software for easy linking to backupdevices and pointer.
Is there any alternative out there for VPN-connecting?

I think the thing to make note of is specific just to LXC when the prox hosts are also on the tailnet (at least in my attempts). I have my theories that it has to do with cgroups and the UID/GID shifts of processes running on Linux Containers. While not very convenient, I have worked around some of my constraints for my particular situation. I am using a jump host that is a standalone bare metal Rocky Linux Host. It is the Tailscale node my services can be accessed from (running nginx), and it has it's own subnet on a separate interface and VLAN going to my proxmox hosts, and all the hosts I want to host services have their respective ports served to the jumpbox with Stunnel. Since it is all internal networking, that's kind of overkill, but I wanted an extra degree of separation so the threat landscape is minimal without going directly through the tunnel (protecting the prox hosts, and limiting damage unless the LXC can break containment)

Your mileage may vary, but having an outside entity to be your access/reverse proxy does give you an in, despite the extra work.