[SOLVED] Understanding the logic with IP of LXC containers coming from Debian

[Flight6334]

Hi,

Very sorry if my questions are stupid, I'm just trying to figure it out.

I had a one node server running on Debian with Docker on it. Very simple, I had all my different services running on one machine (plex, nextcloud, etc.).

• I had a Pi-hole service for my DNS, this way, *.mydomain.com requests return my server IP (192.168.1.175).
• Then, once my server received it, Traefik looks the request like plex.mydomain.com and forwards to the correct Docker service. I didn't had to register each service, I added labels to my Docker Compose file to auto register it, and Traefik magically knew it like so:
  

  services:  home-assistant:
      ...
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.home-assistant.entrypoints=https"
        - "traefik.http.routers.home-assistant.rule=Host(`home-assistant.mydomain.com`)"
        - "traefik.http.routers.home-assistant.tls=true"
        - "traefik.http.services.home-assistant.loadbalancer.server.port=8123"
        - "traefik.http.services.home-assistant.loadbalancer.passhostheader=true"

But now, with Proxmox the logic feels completely different.

I think the best practice is to set up one LXC for one Docker service (previously, my plex, my nextcloud, etc.), but the problem is each new LXC asks a new IP to my DHCP server, it is a small home network with the 192.168.1.0/24 network, I can't change it and with all my services, I'm going to exceed IP address capacity.

So, I would like directions to help me about this issue.

My idea is to have all my LXC on the same IP but I don't understand the Proxmox bridge mode.
I don't know if it's a problem or anything but my server IP configuration is connected to the bridge but my physical is empty. I don't understand (see picture)



Why Proxmox has its IP in the Linux bridge and not, here, on my enp5s0 device?
I screwed up this part so many time I had to reinstall Proxmox two times (can't SSH) so I'm afraid to touch this part now...

My setup was simple but yet, it seems super confusing to do with Proxmox.

Thanks

 

[Dunuin]

I think the best practice is to set up one LXC for one Docker service

No, best practice would be to run docker in a VM. For better security and more granular backups/restores/migrations one VM for each docker stack.

I can't change it and with all my services, I'm going to exceed IP address capacity.

Are you sure you can't change it? If it doesn't even allow you to change the most basic configs like using a 10.0.0.0/8 subnet you should think about setting up a proper router, like a OPNsense/pfsense box (or VM).

My idea is to have all my LXC on the same IP but I don't understand the Proxmox bridge mode.

Thats not how it works. Each guests needs its own IP. But you could create another bigger subnet locally on your PVE on another bridge, run a router VM and let it route between a 10.0.0.0/8 and your 192.168.1.0/24 subnet.

Why Proxmox has its IP in the Linux bridge and not, here, on my enp5s0 device?

Otherwise guests couldn't communicate with the PVE host.

 

[Flight6334]

No, best practice would be to run docker in a VM. For better security and more granular backups/restores/migrations one VM for each docker stack.

Okay, to be sure, a VM, right, not a CT?

Are you sure you can't change it? If it doesn't even allow you to change the most basic configs like using a 10.0.0.0/8 subnet you should think about setting up a proper router, like a OPNsense/pfsense box (or VM).

Yeah, it is shit. But with your next point, I think It won't be a problem.

Thats not how it works. Each guests needs its own IP. But you could create another bigger subnet locally on your PVE on another bridge, run a router VM and let it route between a 10.0.0.0/8 and your 192.168.1.0/24 subnet.

Yes, I think it could be suitable for my case.

Otherwise guests couldn't communicate with the PVE host.

I see, but why the IP of my Proxmox server is written (192.168.1.175/24) on the bridge, and not onto the enp?

Thanks for your reply, it's a bit less confusing, I'll try with the info I already have.

 

[Dunuin]

Okay, to be sure, a VM, right, not a CT?

Both have advantages and disadvantages. But official recommendation according to Wiki is to use a VM:
https://pve.proxmox.com/wiki/Linux_Container

If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

While being way more ressource efficient and allowing to share host ressources like filesystems, GPU and so on LXCs are known to cause problems when running docker. If you don't want to fear that an update will break your docker services or you care about security, a VM with its proper isolation would be a better choice.

Yes, I think it could be suitable for my case.

Have a look at threads in this forum or find some tutorials online. Running a pfsense/OPNsense VM between your VMs/LXCs and LAN is a very common practice for better security.

I see, but why the IP of my Proxmox server is written (192.168.1.175/24) on the bridge, and not onto the enp?

Because the bridge is on top of that NIC. You don't use that NIC directly.