Hi all,
So this might be a bit basic, but I'm having trouble setting up a network for my homelab (single-node for now) proxmox cluster. I've made a little progress, but networking is still a mystery to me.
I'll start with my goal, as I'm not sure what I've done so far is even in the right direction. I'd like to host a couple of LXC containers, one running an http reverse proxy like nginx, another with sshd for remote login access. Behind these, I'd like to have VMs or containers for different projects (e.g. one running Jupyter for a data analysis project, another 3 for playing with kubernetes, etc). I can't do real HA now, but someday I'd like to try.
I have only one public IP (dynamic), so I'd like to forward 80/443 requests to any subdomain *.mydomain.net toward the backend host with the same hostname, for example project1.mydomain.net:80 --> nginx container --> project1 VM. (As far as I know it isn't possible to proxy ssh this way [?], so mydomain.net:22 can just go directly to the sshd bastion.) The less config needed to spin up and route a new project VM the better
(I'll be looking into something like a dynamic nginx config, or ceryx if needed, and I'll need to think about wildcard certs etc...)
My current setup looks like the following:
Router IP 10.1.10.1; I set the netmask to 255.255.0.0, thinking I can use the third octet to distinguish physical hosts from virtuals, and different types of virtuals. The router also serves wireless devices around the house via DHCP in the range 10.1.10.{2..254}. Putting all of these in the same subnet may not be wise, I just don't know enough to really weigh the options.
Physical Host: 10.1.11.1/16 (future servers could be 10.1.11.{2, 3, 4, ...}/16)
I was thinking of putting "control plane" containers (nginx, sshd bastion, ddclient for dynamic dns, local nameserver) in the 10.1.12.0/16 space. To help facilitate that, I setup a basic dnsmasq install at 10.1.12.4 to which I can point other VMs for easy VM/VM communication. This seems to work ok, though I was hoping to get DHCP running with dnsmasq (serving out 10.1.1.{100..200}) in such a way that name lookups for new VMs would be handled automagically; I think my bridge configuration is getting in the way of this. I am also thinking that rather than setup port forwarding on the virtual bridge, I do it directly on the router; that way in an HA setup the container could be moved without any special adjustments.
Here's my /etc/network/interfaces on the physical host; I had initially thought I'd put the dnsmasq container on vmbr1 to serve dhcp IPs, but that doesn't seem to go well with putting other things on vmbr0.
root@alpha:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface eno1 inet manual
#real IP address
auto vmbr0
iface vmbr0 inet static
address 10.1.11.1
netmask 255.255.0.0
gateway 10.1.10.1
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vmbr1
#private sub network
iface vmbr1 inet static
address 10.1.1.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
##########
Currently my sshbastion is 10.1.12.1/16 on vmbr0, nginx container 10.1.12.2/16 on vmbr0, with both using the router 10.1.10.1 as the gateway. dnsmasq is at 10.1.12.4/16 and I'm pointing the others to it for DNS; also trying to get it to serve 10.1.1.0/24 addresses (NAT). Here's the dnsmasq.conf if it helps:
root@dnsmasq:~# cat /etc/dnsmasq.conf
# global options
domain-needed
bogus-priv
no-resolv
filterwin2k
expand-hosts
domain=mydomain.net
local=/mydomain.net/
listen-address=127.0.0.1
listen-address=10.1.12.4
# DHCP range
dhcp-range=10.1.1.100,10.1.1.200,12h
dhcp-lease-max=100
# google nameservers
server=8.8.4.4
server=8.8.8.8
and /etc/hosts on that container:
root@dnsmasq:~# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# --- BEGIN PVE ---
10.1.12.4 dnsmasq.local dnsmasq
# --- END PVE ---
10.1.12.1 sshbastion
10.1.12.2 nginx
10.1.12.4 dnsmasq
################
Anyway, thanks for any direction anyone can provide - my networking experience is quite limited, but I'd like to learn more about putting together sane and robust layouts.
So this might be a bit basic, but I'm having trouble setting up a network for my homelab (single-node for now) proxmox cluster. I've made a little progress, but networking is still a mystery to me.
I'll start with my goal, as I'm not sure what I've done so far is even in the right direction. I'd like to host a couple of LXC containers, one running an http reverse proxy like nginx, another with sshd for remote login access. Behind these, I'd like to have VMs or containers for different projects (e.g. one running Jupyter for a data analysis project, another 3 for playing with kubernetes, etc). I can't do real HA now, but someday I'd like to try.
I have only one public IP (dynamic), so I'd like to forward 80/443 requests to any subdomain *.mydomain.net toward the backend host with the same hostname, for example project1.mydomain.net:80 --> nginx container --> project1 VM. (As far as I know it isn't possible to proxy ssh this way [?], so mydomain.net:22 can just go directly to the sshd bastion.) The less config needed to spin up and route a new project VM the better
(I'll be looking into something like a dynamic nginx config, or ceryx if needed, and I'll need to think about wildcard certs etc...)My current setup looks like the following:
Router IP 10.1.10.1; I set the netmask to 255.255.0.0, thinking I can use the third octet to distinguish physical hosts from virtuals, and different types of virtuals. The router also serves wireless devices around the house via DHCP in the range 10.1.10.{2..254}. Putting all of these in the same subnet may not be wise, I just don't know enough to really weigh the options.
Physical Host: 10.1.11.1/16 (future servers could be 10.1.11.{2, 3, 4, ...}/16)
I was thinking of putting "control plane" containers (nginx, sshd bastion, ddclient for dynamic dns, local nameserver) in the 10.1.12.0/16 space. To help facilitate that, I setup a basic dnsmasq install at 10.1.12.4 to which I can point other VMs for easy VM/VM communication. This seems to work ok, though I was hoping to get DHCP running with dnsmasq (serving out 10.1.1.{100..200}) in such a way that name lookups for new VMs would be handled automagically; I think my bridge configuration is getting in the way of this. I am also thinking that rather than setup port forwarding on the virtual bridge, I do it directly on the router; that way in an HA setup the container could be moved without any special adjustments.
Here's my /etc/network/interfaces on the physical host; I had initially thought I'd put the dnsmasq container on vmbr1 to serve dhcp IPs, but that doesn't seem to go well with putting other things on vmbr0.
root@alpha:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface eno1 inet manual
#real IP address
auto vmbr0
iface vmbr0 inet static
address 10.1.11.1
netmask 255.255.0.0
gateway 10.1.10.1
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vmbr1
#private sub network
iface vmbr1 inet static
address 10.1.1.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.1.1.0/24' -o eno1 -j MASQUERADE
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
##########
Currently my sshbastion is 10.1.12.1/16 on vmbr0, nginx container 10.1.12.2/16 on vmbr0, with both using the router 10.1.10.1 as the gateway. dnsmasq is at 10.1.12.4/16 and I'm pointing the others to it for DNS; also trying to get it to serve 10.1.1.0/24 addresses (NAT). Here's the dnsmasq.conf if it helps:
root@dnsmasq:~# cat /etc/dnsmasq.conf
# global options
domain-needed
bogus-priv
no-resolv
filterwin2k
expand-hosts
domain=mydomain.net
local=/mydomain.net/
listen-address=127.0.0.1
listen-address=10.1.12.4
# DHCP range
dhcp-range=10.1.1.100,10.1.1.200,12h
dhcp-lease-max=100
# google nameservers
server=8.8.4.4
server=8.8.8.8
and /etc/hosts on that container:
root@dnsmasq:~# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# --- BEGIN PVE ---
10.1.12.4 dnsmasq.local dnsmasq
# --- END PVE ---
10.1.12.1 sshbastion
10.1.12.2 nginx
10.1.12.4 dnsmasq
################
Anyway, thanks for any direction anyone can provide - my networking experience is quite limited, but I'd like to learn more about putting together sane and robust layouts.