3 nodes without cluster sharing a subnet?

[HerrMarauder]

Hello everyone,

first off, i am new to proxmox if i dont know some stuff i am sorry.
I wanted to build a HomeLab to Learn and have some usable services, heard good stuff about Proxmox. My Hardware consists of 3 old Laptops (HP 2x i7 11. Gen 2x. i7 10 Gen. all 16gb Ram, 1 LAN each) ans a TP Link Switch. My first idea was a cluster between 3 nodes but i cant get that to run and i gave up on that.

Now too my question. Is it possible to achive the drawn network infrastructure (Lines meaning 1 ethernet connection /dotted lines meaning association) in a way that i can connect to my containers form my .69 network to theri .70 addresses?
I tried having a virtual interface with the subnet on each node but that doesnt give me connections between the nodes. Is there an fix / trick to achive this or is this not possible and i need a diffrent solution.

Kind regards

 

[janus57]

Hi,

Your issue is not proxmox but pure network missing knowledge.

If you want to communicate between networks, you need a router + 192.169.X.X range is not private; you should check rfc1918.

Best regards,

 

[HerrMarauder]

Hey,
thanks for the quick reply.
The FritzBox is the Router of the Network, but it doesnt support subnets, is there something doable using static routes? Thats the most it can do.
Saw that i mistyped to 192.169.X.X, should be 192.168.X.X, i think thats a private one right?.

Kind regards

 

[janus57]

Hi,

i think thats a private one right?.

yes, see REF1918 for completeness

is there something doable using static routes?

Yes, if your router can do a static route

But at this point the question will be more "Why would you do this" instead of putting it all in the same subnet ?

Best regards,

 

[LnxBil]

The FritzBox is the Router of the Network, but it doesnt support subnets, is there something doable using static routes?

You may need to look deeper into how networks work.

If you want to have a setup like the one you explained in the image above, you need to have a router with two interface, one in the FritzBox and one in the subnet.

 

[UdoB]

Theoretically a workaround could be to let PVE masquerade the .70.x network. --> https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_masquerading

Note that masquerading destroys meta-information by replacing the source address with its own (on the PVE-XX). And alone this aspect makes me hate that approach. If you get a router in between you could specify rules to allow/forbid traffic from/to each VM. This is what you usually want. SNAT destroys this capability completely.

One way is to get a used router - there are literally millions out there. (Even an old DSL-router (like most Fritz!Boxes) would be sufficient as most of them have some LAN-copper ports and one of them being the uplink. But you need to check that in detail...)

If you can buy or build something from scratch I would opt for an OpnSense capable device --> https://docs.opnsense.org/manual/hardware.html Note that OpnSense is a large topic in itself.

You have already shown an OpnSense VM in your diagram. Just adapt the information from the other posts: this one needs a NIC in the .69.xx-network and one NIC in the .70.x network; this VM is placed between both networks. Problem solved ;-)

----
Edit: the OpnSense-VM-approach makes this one a SPOF, a Single-Point-Of-Failure. To eliminate this run two of them on two different nodes, without HA and look at https://docs.opnsense.org/manual/how-tos/carp.html

 

[HerrMarauder]

Okay i still have a lot to learn.
Would something like in this picture work if i get a used router (found a cheap i can get nearby) or would i set myself up for problems down the line.
Or would it be better to get a 2nd nic for 2 of the Laptops and use them as Firewall with a OPNSense CARP config?

Thanks for the help

 

[UdoB]

Okay i still have a lot to learn.

That's normal - and it is true for most users and admins, including me.

Would something like in this picture work if i get a used router (found a cheap i can get nearby) or would i set myself up for problems down the line.

Yes, that drawing looks better. PVE-03 still has the "wrong" network, but that's probably just an oversight.

I probably would introduce one more network for the PVE-nodes exclusively - to separate them from the VMs. But that would either require another NIC on the nodes or a VLAN-capable setup. Both variants would increase the complexity to the next level. We are talking about a homelab, right? So KISS (Keep-it-simple, stupid) is probably more important than a full blown, but more complex, solution.

Or would it be better to get a 2nd nic for 2 of the Laptops and use them as Firewall with a OPNSense CARP config?

"It depends." It is definitely the much more ambitious approach = more difficult but more resilient. I would leave it on the board for... next year or so.

Don't forget: have fun! And this does only work as long as the current task is not too complex to achieve

 

[Johannes S]

Edit: the OpnSense-VM-approach makes this one a SPOF, a Single-Point-Of-Failure. To eliminate this run two of them on two different nodes, without HA and look at https://docs.opnsense.org/manual/how-tos/carp.html

Then you would also need a way for both OPNsense nodes to get Internet access though otherwise your Internet uplink gets your SPOF. Otherwise you would put both of your sense nodes behind your Internet uplink (e.G. a Fritzbox DSL or fibre router) which will result in a double-NAT-setup which isn't a good way to setup.
OP: I advise to read following pieces @meyergru posted in the OPNsense forum:

• https://forum.opnsense.org/index.php?topic=42985.0 Read this first for beginners
• https://forum.opnsense.org/index.php?topic=44159.0 OPNsense and Virtualization

Afterwards reconsider what you actually want to achieve with your setup and whether you actually need this complex setup for it in the first place. If I see correctly you are planning with two routers (so generating a double-NAT) and an OPNsense VM on your ProxmoxVE cluster. This is a rather complex setup and nothing I would recommend to start with. In case of an error troubleshooting will need to be done at several points and not in a fun way. So start with something simpler: Just the Fritzbox, the switch and your PVE nodes directly attached to the Fritzbox LAN. Then start with virtual networks and routers (doesn't matter whether OpenWRT/OPNsense or whatever) on your PVE-cluster but nothing important so you can playaround and learn. If you feel confident enough you can then adapt your learnings to your production environment

 

[UdoB]

Then you would also need a way for both OPNsense nodes to get Internet access though otherwise your Internet uplink gets your SPOF.

Correct. Some part of the actual infrastructure is always a SPOF. For example a normal UPS does not really help if my single energy provider goes down. And while I have some redundancy on different levels like in storage (ZFS) and some services (DNS etc.) and I have multiple nodes to failover to, my network stack at home is completely not redundant. (Which puts me into hell when a switch dies... :-( )

Otherwise you would put both of your sense nodes behind your Internet uplink (e.G. a Fritzbox DSL or fibre router) which will result in a double-NAT-setup

Does it? The only device which does masquerading is my DSL router (a normal Fritz!Box) for IPv4. The inner side of that box is my DMZ. Then I have a software router with access to some other local networks. I have an extravagant number (read: too many) of IP-networks / VLANs. All of them are routed.

(( Unfortunately my IPv6 is crippled (/62 instead of /56) and does not work as advertised - this would remove that single point of NAT completely. ))