Proxmox (host) adding NIC with pfSense (vm)... I am confused...

[virtManager]

Hi

I've been running proxmox and pfsense virtualized for some months now and am relatively happy about it. The proxmox-server is headless so when I once in a while screw things up, I physically move it and lose all internet connectivity, then plugin HDMI and usb keyboard so I can see what is going on - but without DHCP-server/network, it's a hazzle (home setup). The last week I was playing with openvpn and wireguard server and suddenly everything crashed and became unresponsive. This is where I realized that I need a better solution and a better setup to avoid moving the server and have the whole network down, until I've sorted things out... On the proxmox system, I currently have:

Network configuration:

Proxmox (host):

cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet dhcp

iface enp1s0f0 inet manual

iface enp1s0f1 inet manual

iface enp1s0f2 inet manual


auto enp1s0f3
iface enp1s0f3 inet static
    address 192.168.100.3/24
#STATIC_EMERGENCY_RESCUE


auto vmbr0
iface vmbr0 inet static
    address 192.168.100.2/24
    gateway 192.168.100.1
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up /sbin/ethtool -s enp2s0 wol g

Remarks:

1. I can currently successfully connect to the proxmox webUI via https://192.168.1.2:8006 - but this is only because pfSense knows the MAC address so it statically assigns the IP address via DHCP.
2. Furthermore, I can access the proxmox webUI via (vmbr0) https://192.168.100.2:8006/ - I this this because I wanted a static IP address in case pfsense misbehaves, so I should still be able to start/stop VMs and perform emergency tasks.
3. The problem with (vmbr0) and access via https://192.168.100.2:8006/ is that (I think) it relies on pfSense being up and acting as a router/gateway. I created vmbr0 so VMs and a LXC fileserver can talk to each other (for config-backups, smb/cifs and nfs) and the whole 192.168.100.0/24 is an interface in pfSense, with a corresponding DHCP-server (responsible for handing out a statically assigned ip to the LXC fileserver on the same subnet, via DHCP).

I tried to add a new NIC to proxmox: enp1s0f3: I would like to be able to manually plug in a cable to this port and in the other end connect it directly to a laptop, without going through all the tedious VLAN-stuff from the managed switches I have and then I shoudl be able to access the webUI. But it does not work - I guess there isn't any DHCP-server, but is that a problem to connect just a laptop *directly* to the proxmox NIC without any DHCP-server?

I think it's a problem that the gateway is 192.168.100.1, which is the ip address to the pfSense-router. Furthermore I do not wish to passthrough "enp1s0f3" to pfSense, because I would like to be able to access proxmox, even if pfSense crashes or is turned off. I experience some issues I don't understand, e.g. if try to create a whole new interface for "enp1s0f3" with ip 192.168.90.2/24 and gateway 192.168.90.1, then I get "Error: Parameter verification failed. (400) gateway: Default gateway already exists on interface 'vmbr0'... But vmbr0 depends on pfSense, right? I also don't understand why enp1s0f3 is not "active", but maybe it is waiting for a DHCP-server? I tried putting a network cable in my laptop, manually assigning the IP address to the correct subnet, but then I guess it does not make sense to talk about a gateway?

I'm just really confused and trying to learn more about networks. Maybe I need to bridge something - I hope someone can help me straighten things out and I hope my explanation of my thoughts is understandable, otherwise I'll try to elaborate where needed...

 

[vesalius]

The problem with (vmbr0) and access via https://192.168.100.2:8006/ is that (I think) it relies on pfSense being up and acting as a router/gateway.

This is not strictly the case. although by your design this false premise end up true. iface vmbr0 inet static means that proxmox statically gives itself the address 192.168.100.2/24 and will not change this regardless of what a DHCP server on this subnet tries to do. The gateway 192.168.100.1 line is basically telling proxmox that this address is the next hop to ask for help when proxmox tries to reach an address outside the 192.168.100.0/24 subnet. However, because of the vmbr0 line bridge-ports none vmbr0 is a virtual network only and has no physical connection to your switch. The post-up /sbin/ethtool -s enp2s0 wol g sets Wake-on-LAN for enp2s0 but is otherwise noncontributory for vmbr0. As for iface enp2s0 inet dhcp, I don't think this doing you any good or what you expect.

Another place where things may be going wrong for you is that once you stop the pfsense VM, whatever computer/device you use to try and connect with proxmox at 192.168.100.2 would also need a static IP somewhere within the 192.168.100.0/24 subnet. Typically, once the DHCP server/pfsense goes down, all the DHCP endpoints will eventually self-assign themselves a link-local IP in the 169.254.0.0/16 subnet.

So any PC/MAC using DHCP ends up with an address of 169.254.0.1 for example and proxmox still uses 192.168.100.2 and neither can see the other.

The cleanest option to try.

iface enp2s0 inet manual

...

iface enp1s0f3 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.100.2/24
    gateway 192.168.100.1
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0

AND set up a PC/MAC to always have a Static address in the 192.168.100.0/24 subnet, 192.168.100.4 for example. After that shutdown the pfsense VM and as long as the 2 computers are connected to your switch (and vlan/routing at the switch level allows direct communication) you should still have access to the proxmox webgui.

Later if you decide you want redundant connections to your switch/lan from proxmox you can explore setting up a bond0 using both enp2s0 & enp1s0f3 followed by changing the vmbr0 Bridgeport to that bond0. IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP) is the best option, but will only work if the switch supports it and is harder to set up. Active-backup (active-backup) is the easiest to set up and will give you fault tolerance with zero set up on your switch required.

 

[virtManager]

iface vmbr0 inet static means that proxmox statically gives itself the address 192.168.100.2/24 and will not change this regardless of what a DHCP server on this subnet tries to do. The gateway 192.168.100.1 line is basically telling proxmox that this address is the next hop to ask for help when proxmox tries to reach an address outside the 192.168.100.0/24 subnet.

Right, at least that's how it works when there's a DHCP-server in the system. When there's no DHCP-server, I'm confused about what it means to have a gateway ip-address? Maybe it doesn't mean anything at all, without a DHCP-server?

However, because of the vmbr0 line bridge-ports none vmbr0 is a virtual network only and has no physical connection to your switch.

Right, vmbr0 is passed through to pfSense which knows that this interface should be VLAN 100 and it should also act as a DHCP-server - there are no cables. I can however connect to 192.168.100.0/24 via all my other VLANs (and cabled connections), but you're right VLAN 100 is meant as a "Proxmox & pfSense & all VMs & all LXC's"-network, although I'm happy that at least while pfSense is up and running normally, I can connect via the pfSense-VM router.

The post-up /sbin/ethtool -s enp2s0 wol g sets Wake-on-LAN for enp2s0 but is otherwise noncontributory for vmbr0. As for iface enp2s0 inet dhcp, I don't think this doing you any good or what you expect.

Yeah, sorry, I cut out this part of the description to make the original post shorter, quicker and easier to read/understand. In a previous version of my original post, I had some extra details about this but decided to remove it because it made the post long and more complicated. But the reason for using enp2s0 is that it's the realtek physical port and then I have 4 intel NIC physical ports. I read intel NICs was more reliable than realtek NICs, so I didn't want to use enp2s0 for daily use. I added this "wol"-stuff so that I could turn on the Proxmox machine using Wake-On-Lan - it actually worked, at least some months ago when I experimented with it. But at that time I think I manually had to statically assign my laptop a proper IP address and then send out the WOL-packet. It's actually a bit confusing that the line

post-up /sbin/ethtool -s enp2s0 wol g

is in the "vmbr0"-section because I do not believe WOL for enp2s0 has anything to do with vmbr0 - at least not as I remember it. In that sense I agree with you, it's noncontributory for vmbr0, but it does (at least did) have a purpose: To be able to turn on the proxmox-machine (and the VMs) using WOL from a laptop. So I prefer to keep this line in the config.

Another place where things may be going wrong for you is that once you stop the pfsense VM, whatever computer/device you use to try and connect with proxmox at 192.168.100.2 would also need a static IP somewhere within the 192.168.100.0/24 subnet. Typically, once the DHCP server/pfsense goes down, all the DHCP endpoints will eventually self-assign themselves a link-local IP in the 169.254.0.0/16 subnet.

So any PC/MAC using DHCP ends up with an address of 169.254.0.1 for example and proxmox still uses 192.168.100.2 and neither can see the other.

Thanks a lot for this explanation!

The cleanest option to try.

iface enp2s0 inet manual

...

iface enp1s0f3 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.100.2/24
    gateway 192.168.100.1
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0

AND set up a PC/MAC to always have a Static address in the 192.168.100.0/24 subnet, 192.168.100.4 for example. After that shutdown the pfsense VM and as long as the 2 computers are connected to your switch (and vlan/routing at the switch level allows direct communication) you should still have access to the proxmox webgui.

I first tried as you proposed - except I didn't shutdown the pfSense VM (maybe I should), I was hoping for a solution that would be stable enough to both work when pfSense was turned on - and when it's either not or unresponsive. In my case, with pfSense still turned on, everything became non-responsive - I lost all internet access on the network, so I turned off proxmox and turned it back on. Maybe this is where I need bond-interfaces?

Then I started thinking: That enp2s0-interface is connected to my managed switch (physical port 6 = VLAN 1), which is fed back again to pfSense via that same switch, but now via physical port 7 (trunk port). I don't fully understand the meaning of configuring "vmbr0" with "bridge-ports enp2s0": Without bridge-ports, they would be 2 interfaces with each their ip address, right? With the "bridge-ports" line will both vmbr0 and enp2s0 still have each their IP address? They were on different subnets and enp2s0 had it's IP configured statically with the pfSense DHCP-server (set to 192.168.1.2). I then thought more about it and perhaps you meant that I in the "vmbr0"-section should use "bridge-ports enp1s0f3" (which corresponds more to my original plan, because this port is not connected to any switches at all, so I'll just be plugging in a network cable directly to enp1s0f3 and the other end to a laptop, i.e. no "looping" or "feeback" issues with packets going back and forth between the router and VM, when pfSense is turned on) ? But I also tried that - just took my laptop, assigned it (statically, no DHCP) to 192.168.100.5/24 with gateway 192.168.100.1 - but no luck. Also, "ip link show" didn't even tell that "enp1s0f3" was up and in the proxmox GUI it didn't turn of as "active" although "auto-start" was chosen...

Later if you decide you want redundant connections to your switch/lan from proxmox you can explore setting up a bond0 using both enp2s0 & enp1s0f3

Thanks for that link. I also found an excellent description of this: bonding example step-by-step - I'll definately try those things one day, to get experience and a better understanding, thanks!

followed by changing the vmbr0 Bridgeport to that bond0. IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP) is the best option, but will only work if the switch supports it and is harder to set up. Active-backup (active-backup) is the easiest to set up and will give you fault tolerance with zero set up on your switch required.

My problem is also that I can see there are several bonding modes. I'll need to experiment with several of these before it is completely clear what I should use and I have to think about the managed switch-setup too. At least the first switch I have understands 802.3ad LACP according to TP-link switch specs (Link Aggregation, Static link aggregation, 802.3ad LACP, Up to 8 aggregation groups and up to 8 ports). But later I have another switch, they don't specifically mention 802.3ad LACP for it - second-switch-DLink - oh, I guess I must mess with that another time...

In any case, I'm grateful for the comments you gave me, it should be enough for me to better search the forum and google for things to try and then another time in near future I'll be experimenting more with this. For now, unfortunately I'll go away for some days and out of reach of the equipment - but thanks a lot for your help, advice and comments!

 

[vesalius]

Ok sorry, I was unaware of your VLAN setup with pfsense or that enp2s0 was in vlan1 and enp1s0f3 was in vlan100. Lots of unaccounted variables in this setup.

Maybe you can list the proxmox interfaces have you given pfsense? What from proxmox does pfsense use for Wan, Lan and/or other vlans? a screenshot of the proxmox-> pfsense VM hardware->network devices and/or the INTERFACES: ASSIGNMENTS from the pfsense webgui might help unravel this. The managed switch and whether ports or tagged, untagged with vlans will matter.

The gateway IP address is not the same as the DHCP server. They can be at the same IP number as in the case with pfsense or they can have different IP addresses and be on different machines altogether. the gateway address can work perfectly fine without a DHCP server. You seem to be thinking of a static IP address only from the perspective of setting them up in the pfsense dhcp settings.

1. Your enp2s0 for example will only get the IP 192.168.1.2 when it request an IP from pfsense over vlan1. However, If pfsense/dhcp goes down then enp2s0 will eventually self-assign an IP in the 169.254.0.0/16 subnet. If you want enp2s0 to always keep 192.168.1.2 when/if pfsense is shutdown then you should also change it to the below in /etc/network/interfaces:
   

   auto enp2s0
   iface enp2s0 inet static
          address 192.168.2.1/24

2. vmbr0 on the other hand never needs a DHCP server because you also assigned it a static IP 192.168.100.2/24 in the proxmox /etc/network/interfaces file. However as it stood initially vmbr0 was without a physical connection so it still becomes unavailable when pfsense is down. Using enp1s0f3 as a bridge port sounds like the right direction.

the gateway is where/what proxmox will depend on to route traffic from/to the internet or basically anything outside the vlan100 subnet. A DHCP server is not necessary for this function.

 

[virtManager]

Ok sorry, I was unaware of your VLAN setup with pfsense or that enp2s0 was in vlan1 and enp1s0f3 was in vlan100. Lots of unaccounted variables in this setup.

No problem, to begin with I couldn't figure out how to make an "easy" representation or drawing - but now I think I have. And enp1s0f3 is "free" to use and configure in any way possible, as it's not meant to be used for anything else, except that I would like to be able to contact proxmox via a direct cable, but without DHCP-server (it should hopefully work both if pfSense is running normally and if it crashed or is turned off).

Maybe you can list the proxmox interfaces have you given pfsense? What from proxmox does pfsense use for Wan, Lan and/or other vlans? a screenshot of the proxmox-> pfsense VM hardware->network devices and/or the INTERFACES: ASSIGNMENTS from the pfsense webgui might help unravel this. The managed switch and whether ports or tagged, untagged with vlans will matter.

That's a really good idea with these screenshots. I think I've managed to make something useful:



One more detail I forgot to put in the figure above (had to think about it for a while to remember this): enp2s0 is the realtek NIC which I didn't want to use daily - but I remember it has been working for WOL. The cable from this port is directly connected to port 6 on the first switch, which have been used for different VLAN-tests earlier but currently it's assigned to VLAN 1. That means although you cannot see it, from the screenshots above, enp2s0 is on VLAN 1 and connected to pfSense, but via cables and the first switch...

About the managed switch VLAN config: If I post it, it begins to be a bit complicated and since everything is actually (more or less) working ok, I only need to take into account the situation when pfSense crashes or becomes unresponsive and takes the whole network down. And in this case, I prefer (if possible) to configure enp1s0f3 directly - i.e. without any traffic going through switches. I need to just plugin a cable from a laptop and access Proxmox so I can e.g. either reboot pfSense or in the worst case, restore a backup. Once that works, I can configure the switch traffic myself, that part is not a problem.

The gateway IP address is not the same as the DHCP server. They can be at the same IP number as in the case with pfsense or they can have different IP addresses and be on different machines altogether. the gateway address can work perfectly fine without a DHCP server. You seem to be thinking of a static IP address only from the perspective of setting them up in the pfsense dhcp settings.

I understand you, but I know static IP address can be setup either on the DHCP-server - or manually on the computer (in Windows, there's a network icon and in linux I have a configuration I switch to using "nmtui"). But also when I use "nmtui", to directly connect to another machine, I've never understood the meaning of "gateway", when I/we bypass the DHCP-server and statically assign the IP address on e.g. a laptop using "nmtui". Let me elaborate a bit, this is what I do (in this example it's 192.168.1.0/24 but I know I can change that to whatever):



If I directly connect my laptop to the Proxmox physical port 3 (=enp1s0f3), then there's no router involved. And there can be no router involved, because we want to make things so robust that even if pfSense crashes, becomes unresponsive or is turned off, then I can access the admin interface of proxmox. What is the meaning of the "gateway" here?

I should add that I've also considered, if I need to install a DHCP-server on Proxmox, so it on one of the physical ports (port 3 = enp1s0f3 here) can be 100% independent of pfSense... Not sure if that's the way to go (that would in any case make it easier to connect a laptop, because then I just plugin the cable and don't even have to change the IP address manually)? And then I should avoid the 2 DHCP-servers from seeing each other (proxmox, if I install a DHCP-server directly on it and pfSense, when it's running normally).

1. Your enp2s0 for example will only get the IP 192.168.1.2 when it request an IP from pfsense over vlan1. However, If pfsense/dhcp goes down then enp2s0 will eventually self-assign an IP in the 169.254.0.0/16 subnet. If you want enp2s0 to always keep 192.168.1.2 when/if pfsense is shutdown then you should also change it to the below in /etc/network/interfaces:
   

   auto enp2s0
   iface enp2s0 inet static
          address 192.168.2.1/24

Right. hmm. Maybe I should just do that instead of trying to make it work for enp1s0f3... hmm, unless I already tried that (cannot remember anymore). I think the issue is that proxmox only listens on one interface, is that right? So when I try to add the enp1s0f3-interface, I don't think the proxmox webUI is listening on that (or on multiple interfaces, is that assumption correct?

I have a feeling that in order to make proxmox listen on multiple interfaces, I need bridging or something, is that correct or wrong?

1. vmbr0 on the other hand never needs a DHCP server because you also assigned it a static IP 192.168.100.2/24 in the proxmox /etc/network/interfaces file. However as it stood initially vmbr0 was without a physical connection so it still becomes unavailable when pfsense is down. Using enp1s0f3 as a bridge port sounds like the right direction.

the gateway is where/what proxmox will depend on to route traffic from/to the internet or basically anything outside the vlan100 subnet. A DHCP server is not necessary for this function.

Ah, so when to computers are talking to each other directly via a LAN-cable, because they're in the same subnet, they gateway ip address is never used, is that correct?

Anyway, I'll go on summer holiday later today and be travelling for a while, so it'll take a while before I will test more on this setup, unfortunately. But it's still really great for me to get this feedback and better understanding. It's my own mistake that things are a bit complicated, when things are going out of interfaces to switches and then again fed back via a trunk port - although I have a small home network, the complexity with the VLANs has grown, as I'm experimenting with things I'll read the answers and I'm very grateful for the answers - hopefully by reading the answers, I'll have enough knowledge to fix things when I'm back home again, thanks!