How can I configure Proxmox and pfSense vm?

[hodo]

You are welcome...

Lets start to configure your preferred setup:

1. Empty the settings on vmbr0 in proxmox gui -> let the bridge-port on eth0
2. Set IP address:192.168.1.140 Subnet mask:255.255.255.0 Gateway:192.168.1.1 and let the bridge-port (eth1) on vmbr1 in proxmox gui
3. connect first vNIC (net0) to vmbr0 on pfsense-vm at the VM-hardware tab in proxmox gui.
4. connect second vNIC (net1) to vmbr1 on pfsense-vm at the VM-hardware tab in proxmox gui.
5. reboot proxmox host and switch network cables behind the machine.
6. test connection to proxmox and pfsense

kind regards
hodo

 

[eiger3970]

Thank you again, worked a treat!

Not quite clear how you knew to switch the cables. I haven’t clearly matched up how NIC0 for WAN is where it is and vice versa.

Looks great, I can configure pfSense now. Actually have backups on the cloud, so as soon as I have Internet again, I can restore all the settings.

Unclear if Proxmox breaks how to get Internet or lan access? I think that is the case for a 3rd NIC. However, I think I will build a 2nd machine to run ASAP if machine1 breaks.

Also, I think I will try to update the Netgate guide which is misleading to me that it is for 2 NICs, not 3. Caused a lot of time delays.

 

[eiger3970]

Hmm, cannot ping or browse Proxmox now? I’ve also changed the router IP back to 192.168.1.170.

 

[eiger3970]

Hmm, cannot ping or browse Proxmox now? I’ve also changed the router IP back to 192.168.1.170.

Hmm, I unplugged the ISP’s bridged modem and the Proxmox server NIC0 lights stopped blinking.

However on the LAN PC 192.168.1.120, the pfSense GUI at 192.168.1.170 shows the pfSense Dashboard WAN as still up?

 

[eiger3970]

PfSense details:
WAN: up 10Gbase-T <full-duplex> 0.0.0.0
vtnet0
MAC Address: 16:62:51:c6:e7::de
(unplugging Ethernet cable from Proxmox eth0 does not change up status to down)

LAN: up 10Gbase-T <full-duplex> 192.168.1.1
vtnet1
MAC Address: ee:1a:d6:89:dc:97
(unplugging Ethernet cable from Proxmox does not change up status to down)

LAN PC:
IP address: 192.168.1.120
Subnet Mask: 255.255.255.0
Gateway: 192.68.1.1

arp -a ->
? (192.168.1.141) at <incomplete> on enp3s0
? (192.168.1.1) at ee:1a:d6:89:dc:97 [erher] on enp3s0
? (192.168.1.140) at <incomplete> on enp3s0
? (192.168.1.170) at <incomplete> on enp3s0
? (192.168.1.141) at <incomplete> on enp3s0

ping and browse 192.168.1.1 connects.
ping 192.168.1.140 Destination Host Unreachable
ping 192.168.1.141 Destination Host Unreachable
ping 192.168.1.170 Destination Host Unreachable
ping and browse 192.168.1.180 connects.

 

[eiger3970]

Maybe I need a pfSense VPN to access Proxmox and Internet...still reading various guides.

 

[eiger3970]

192.168.1.120 LAN PC > tcpdump > IP 192.168.1.170 > 192.168.1.120: ICMP host 8.8.8.8 unreachable
IP 192.168.1.120.34818 > 8.8.8.8.domain: 3434+ A? ssl.gstatic.com

192.168.1.140 Proxmox > tcpdump > ARP, Request who-has 192.168.1.170 tell Proxmox.com
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)

192.168.1.170 pfSense vm > tcpdump > ARP, Request who-has pfSense.localdomain tell 192.168.1.140
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)

 

[hodo]

Hello eiger3970,

before we talk about VPNs, you should establish a basic configuration.

First i have the following assumptions:

1. your server have 2 physical nics (NIC0 and NIC1)
2. NIC0 is solely connected to your wan modem?
3. NIC1 is connected to a switch, on this switch there are various other clients connected
4. NIC0 is eth0 is connected to vmbr0
5. NIC1 is eth1 is connected to vmbr1
6. You want to switch your router from a physical netgate device to a virtual pfsense-vm

Are these assumptions correct?

Hmm, I unplugged the ISP’s bridged modem and the Proxmox server NIC0 lights stopped blinking.

However on the LAN PC 192.168.1.120, the pfSense GUI at 192.168.1.170 shows the pfSense Dashboard WAN as still up?

When you remove the WAN Modem, your NIC0 has no connection -> no lights....
192.168.1.170 is your old netgate router, right? Cant say anything about it, because you have given no information about the connection type it is / was using...

PfSense details:
WAN: up 10Gbase-T <full-duplex> 0.0.0.0
vtnet0
MAC Address: 16:62:51:c6:e7::de
(unplugging Ethernet cable from Proxmox eth0 does not change up status to down)

LAN: up 10Gbase-T <full-duplex> 192.168.1.1
vtnet1
MAC Address: ee:1a:d6:89:dc:97
(unplugging Ethernet cable from Proxmox does not change up status to down)

This works as intended. Linux bridges are a kind of virtual switches. When you connect a client to it, the client has a "connection" and is considered as up. The client (pfsense-vm) knows nothing about eth0 or eth1 and the state of the physical port.

Monitoring your internet connection in pfsense is done with gateway monitoring. Think about this, like you would ping constantly the first gateway which is not under your control (maybe ISPs first hop). You can configure this monitoring later. First you should work on the basics.

kind regards
hodo

 

[eiger3970]

Yes, assumptions are correct. I’m working through the other steps for the basic configuration. Not interested in VPNs, just reading information that might solve the problem.

The old router had an Ethernet RJ45 connection.

Thanks for clarifying the Linux Bridge Up/down thing.

Continuing to work for the basics, getting Internet.

 

[eiger3970]

Yes, assumptions are correct. I’m working through the other steps for the basic configuration. Not interested in VPNs, just reading information that might solve the problem.

The old router had an Ethernet RJ45 connection.

Thanks for clarifying the Linux Bridge Up/down thing.

Continuing to work for the basics, getting Internet.

So, my understanding from the tcpdump is that Proxmox 192.168.1.140 needs the MAC address of the pfSense vm
And
PfSense 192.168.1.170 needs the MAC address of the Proxmox machine.

I thought pfSense’s MAC address was ee:1a:d6:89:dc:97 (LAN).
Need to be sure. So how to check MAC addresses (ifconfig shows this but not 100% sure vtnet1 is pfSense).

Then, how to add MAC address to respective machines.

 

[eiger3970]

So, my understanding from the tcpdump is that Proxmox 192.168.1.140 needs the MAC address of the pfSense vm
And
PfSense 192.168.1.170 needs the MAC address of the Proxmox machine.

I thought pfSense’s MAC address was ee:1a:d6:89:dc:97 (LAN).
Need to be sure. So how to check MAC addresses (ifconfig shows this but not 100% sure vtnet1 is pfSense).

Then, how to add MAC address to respective machines.

I’m confused on what the Proxmox machine’s MAC address is in ‘ip-a’?

 

[eiger3970]

Ok, I’m taking a step back, to clarify the interface allocation, which is probably the cause of issue.

I still don’t understand which interface is which and what to allocate the interface to. Then I need to allocate in Proxmox via CLI, as I have no GUI access.

ISP Modem with WAN NIC.
Proxmox machine with WAN NIC and LAN NIC.
pfSense vm in Proxmox.
24 port switch to devices.

Proxmox:
IP 192.168.1.140
Subnet Mask: 255.255.255.0
Gateway: 192.168.1.170

arp -a: ? (192.168.1.170) at <incomplete> on vmbr0

ping 8.8.8.8: Destination Host Unreachable

tcpdump: ARP, Request who-has 192.168.1.170 tell Proxmox.com
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)

pfSense vm:
IP 192.168.1.170

arp -a: pfSense.localdomain (192.168.1.170) at ee:1a:d6:89:dc:97 on vtnet1 permanent [ethernet]
? (192.168.1.110) at 40:6c:8f:2e:7b:18 on vtnet1 expires in 871 seconds [ethernet]
? (192.168.1.100) at 1c:ca:e3:77:c5:53 on vtnet1 expires in 1198 seconds [ethernet]
? (192.168.1.120) at 88:d7:f6:c9:08:eb on vtnet1 expires in 1196 seconds [ethernet]
? (192.168.1.180) at 00:ea:21:62:38:ce on vtnet1 expires in 1180 seconds [ethernet]

ping 8.8.8.8: No route to host

tcpdump: ARP, Request who-has pfSense.localdomain tell 192.168.1.140
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)


LAN PC:
IP 192.168.1.120
Subnet Mask: 255.255.255.0
Gateway: 192.168.1.170

arp -a: ? (192.168.1.170) at ee:1a:d6:89:dc:97 [ether] on enp3s0
? (192.168.1.140) at <incomplete> on enp3s0
? (192.168.1.141) at <incomplete> on enp3s0
? (192.168.1.180) at 00:ea:21:62:38:ce [ether] on enp3s0
? (192.168.1.17) at <incomplete> on enp3s0
? (192.168.1.1) at <incomplete> on enp3s0

ping 8.8.8.8: Destination Host Unreachable

tcpdump: (connects to 192.168.1.120 and 192.168.1.180).


NIC0: 16:62:51:c6:e7:de (WAN)
NIC1: ee:1a:d6:89:dc:97 (LAN)
eth0: 40:16:7e:37:21:af
eth1: 40:16:7e:37:21:b0
vmbr0: 40:16:7e:37:21:af
vmbr1: 40:16:7e:37:21:b0
vtnet0: 16:62:51:c6:e7:de
vtnet1: ee:1a:d6:89:dc:97

 

[eiger3970]

So, what is the PVE and VM network topology?
ISP > ISP modem (bridged) > ISP modem NIC WAN > PVE NIC WAN > VM pfSense vNIC WAN > VM pfSense vNIC LAN > PVE NIC LAN > Switch LAN > 24 ports of LAN devices.

So, how do the Interfaces and MAC addresses fit into this?
PVE NIC WAN: don’t know...confused where I sourced the NIC MAC addresses from?

 

[hodo]

Hello eiger3970,

lets work on the gui access.

I need a photo of 'cat /etc/network/interfaces' on the proxmox host.

I dont mean informations about the physical connection of your netgate router. What type of connection used it to connect to your isp over them wan-modem?

What has your netgate router configured at the wan interface? (maybe PPPOE?)

kind regards
hodo

 

[eiger3970]

Here’s the photo, attached.

The ISP modem has a coaxial cable to the wall.

There is no physical hardware router.
Pfsense is a virtual machine in Proxmox.

 

[hodo]

Okay, i can see the problem.

Your Proxmox management ip settings are set on vmbr0. vmbr0 is your wan....

Set the following like described in post #21:

1. Empty the settings on vmbr0 in proxmox gui -> let the bridge-port on eth0
2. Set IP address:192.168.1.140 Subnet mask:255.255.255.0 Gateway:192.168.1.1 and let the bridge-port (eth1) on vmbr1 in proxmox gui

Without proxmox gui you have to edit the /etc/network/interfaces:

auto vmbr0
iface vmbr0 inet static
bridge_ports eth0
bridge_stp off
bridge_fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.1.140
netmask 255.255.255.0
gateway 192.168.1.170
bridge_ports eth1
bridge_stp off
bridge_fd 0

What is the device with ip 192.168.1.170?


You said:

192.168.1.170 was the hardware pfSense router’s default gateway address.

Then you said:

There is no physical hardware router.

Iam confused...

I would suggest the following setup:

WAN MODEM -> eth0(vmbr0)->nic0->pfsense-vm (WAN IP)
LAN-Switch->eth1(vmbr1)->nic1->pfsense-vm (192.168.1.1)

Then all your devices on the LAN-Switch and the vmbr1 should have the efault gateway of 192.168.1.1.

Could it be, that 192.168.1.170 is yur modem, and your modem is more than only a modem? Then we have to think about other ips, because pfsense can not operate withe wan-subnet = lan-subnet.

First ste the settings above and check proxmox gui access.

kind regards
hodo

 

[eiger3970]

Thank you, I updated vmbr0/vmbr1 settings, as per photo attached.

My LAN PC 192.168.1.120 can now ping Proxmox 192.168.1.140, however cannot browse 192.168.1.140? See photo of 192.168.1.120’s settings.

Not sure why I can’t browse Proxmox GUI, so I added a photo of the vm pfSense settings.

Please forget/ignore the OLD hardware router, which no longer exists. It is past tense. There is only a virtual machine, qemu-143.conf.

192.168.1.170 is my preferred IP for the router gateway, rather than using 192.168.1.1.

The ISP modem is bridged and has no IP.

I am still confused with what NIC0 and eth0 are?

I can’t ping or browse the vm pfSense 192.168.1.170 from Proxmox 192.168.1.140 or from the LAN PC 192.168.1.120.

 

[eiger3970]

Finally fixed...see https://forum.proxmox.com/threads/cannot-connect-to-proxmox.50903/#post-236537