How can I configure Proxmox and pfSense vm?

Discussion in 'Proxmox VE: Networking and Firewall' started by eiger3970, Jan 8, 2019.

  1. hodo

    hodo New Member

    Joined:
    Jun 22, 2016
    Messages:
    28
    Likes Received:
    4
    You are welcome...

    Lets start to configure your preferred setup:

    1. Empty the settings on vmbr0 in proxmox gui -> let the bridge-port on eth0
    2. Set IP address:192.168.1.140 Subnet mask:255.255.255.0 Gateway:192.168.1.1 and let the bridge-port (eth1) on vmbr1 in proxmox gui
    3. connect first vNIC (net0) to vmbr0 on pfsense-vm at the VM-hardware tab in proxmox gui.
    4. connect second vNIC (net1) to vmbr1 on pfsense-vm at the VM-hardware tab in proxmox gui.
    5. reboot proxmox host and switch network cables behind the machine.
    6. test connection to proxmox and pfsense

    kind regards
    hodo
     
  2. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Thank you again, worked a treat!

    Not quite clear how you knew to switch the cables. I haven’t clearly matched up how NIC0 for WAN is where it is and vice versa.

    Looks great, I can configure pfSense now. Actually have backups on the cloud, so as soon as I have Internet again, I can restore all the settings.

    Unclear if Proxmox breaks how to get Internet or lan access? I think that is the case for a 3rd NIC. However, I think I will build a 2nd machine to run ASAP if machine1 breaks.

    Also, I think I will try to update the Netgate guide which is misleading to me that it is for 2 NICs, not 3. Caused a lot of time delays.
     
  3. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Hmm, cannot ping or browse Proxmox now? I’ve also changed the router IP back to 192.168.1.170.
     
  4. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Hmm, I unplugged the ISP’s bridged modem and the Proxmox server NIC0 lights stopped blinking.

    However on the LAN PC 192.168.1.120, the pfSense GUI at 192.168.1.170 shows the pfSense Dashboard WAN as still up?
     
  5. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    PfSense details:
    WAN: up 10Gbase-T <full-duplex> 0.0.0.0
    vtnet0
    MAC Address: 16:62:51:c6:e7::de
    (unplugging Ethernet cable from Proxmox eth0 does not change up status to down)

    LAN: up 10Gbase-T <full-duplex> 192.168.1.1
    vtnet1
    MAC Address: ee:1a:d6:89:dc:97
    (unplugging Ethernet cable from Proxmox does not change up status to down)

    LAN PC:
    IP address: 192.168.1.120
    Subnet Mask: 255.255.255.0
    Gateway: 192.68.1.1

    arp -a ->
    ? (192.168.1.141) at <incomplete> on enp3s0
    ? (192.168.1.1) at ee:1a:d6:89:dc:97 [erher] on enp3s0
    ? (192.168.1.140) at <incomplete> on enp3s0
    ? (192.168.1.170) at <incomplete> on enp3s0
    ? (192.168.1.141) at <incomplete> on enp3s0

    ping and browse 192.168.1.1 connects.
    ping 192.168.1.140 Destination Host Unreachable
    ping 192.168.1.141 Destination Host Unreachable
    ping 192.168.1.170 Destination Host Unreachable
    ping and browse 192.168.1.180 connects.
     
    #25 eiger3970, Jan 10, 2019
    Last edited: Jan 10, 2019
  6. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Maybe I need a pfSense VPN to access Proxmox and Internet...still reading various guides.
     
  7. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    192.168.1.120 LAN PC > tcpdump > IP 192.168.1.170 > 192.168.1.120: ICMP host 8.8.8.8 unreachable
    IP 192.168.1.120.34818 > 8.8.8.8.domain: 3434+ A? ssl.gstatic.com

    192.168.1.140 Proxmox > tcpdump > ARP, Request who-has 192.168.1.170 tell Proxmox.com
    IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)

    192.168.1.170 pfSense vm > tcpdump > ARP, Request who-has pfSense.localdomain tell 192.168.1.140
    IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)
     
  8. hodo

    hodo New Member

    Joined:
    Jun 22, 2016
    Messages:
    28
    Likes Received:
    4
    Hello eiger3970,

    before we talk about VPNs, you should establish a basic configuration.

    First i have the following assumptions:

    1. your server have 2 physical nics (NIC0 and NIC1)
    2. NIC0 is solely connected to your wan modem?
    3. NIC1 is connected to a switch, on this switch there are various other clients connected
    4. NIC0 is eth0 is connected to vmbr0
    5. NIC1 is eth1 is connected to vmbr1
    6. You want to switch your router from a physical netgate device to a virtual pfsense-vm

    Are these assumptions correct?

    When you remove the WAN Modem, your NIC0 has no connection -> no lights....
    192.168.1.170 is your old netgate router, right? Cant say anything about it, because you have given no information about the connection type it is / was using...


    This works as intended. Linux bridges are a kind of virtual switches. When you connect a client to it, the client has a "connection" and is considered as up. The client (pfsense-vm) knows nothing about eth0 or eth1 and the state of the physical port.

    Monitoring your internet connection in pfsense is done with gateway monitoring. Think about this, like you would ping constantly the first gateway which is not under your control (maybe ISPs first hop). You can configure this monitoring later. First you should work on the basics.

    kind regards
    hodo
     
  9. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Yes, assumptions are correct. I’m working through the other steps for the basic configuration. Not interested in VPNs, just reading information that might solve the problem.

    The old router had an Ethernet RJ45 connection.

    Thanks for clarifying the Linux Bridge Up/down thing.

    Continuing to work for the basics, getting Internet.
     
    #29 eiger3970, Jan 12, 2019
    Last edited: Jan 12, 2019
  10. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    So, my understanding from the tcpdump is that Proxmox 192.168.1.140 needs the MAC address of the pfSense vm
    And
    PfSense 192.168.1.170 needs the MAC address of the Proxmox machine.

    I thought pfSense’s MAC address was ee:1a:d6:89:dc:97 (LAN).
    Need to be sure. So how to check MAC addresses (ifconfig shows this but not 100% sure vtnet1 is pfSense).

    Then, how to add MAC address to respective machines.
     
  11. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    I’m confused on what the Proxmox machine’s MAC address is in ‘ip-a’?
     
  12. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Ok, I’m taking a step back, to clarify the interface allocation, which is probably the cause of issue.

    I still don’t understand which interface is which and what to allocate the interface to. Then I need to allocate in Proxmox via CLI, as I have no GUI access.

    ISP Modem with WAN NIC.
    Proxmox machine with WAN NIC and LAN NIC.
    pfSense vm in Proxmox.
    24 port switch to devices.

    Proxmox:
    IP 192.168.1.140
    Subnet Mask: 255.255.255.0
    Gateway: 192.168.1.170

    arp -a: ? (192.168.1.170) at <incomplete> on vmbr0

    ping 8.8.8.8: Destination Host Unreachable

    tcpdump: ARP, Request who-has 192.168.1.170 tell Proxmox.com
    IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)

    pfSense vm:
    IP 192.168.1.170

    arp -a: pfSense.localdomain (192.168.1.170) at ee:1a:d6:89:dc:97 on vtnet1 permanent [ethernet]
    ? (192.168.1.110) at 40:6c:8f:2e:7b:18 on vtnet1 expires in 871 seconds [ethernet]
    ? (192.168.1.100) at 1c:ca:e3:77:c5:53 on vtnet1 expires in 1198 seconds [ethernet]
    ? (192.168.1.120) at 88:d7:f6:c9:08:eb on vtnet1 expires in 1196 seconds [ethernet]
    ? (192.168.1.180) at 00:ea:21:62:38:ce on vtnet1 expires in 1180 seconds [ethernet]

    ping 8.8.8.8: No route to host

    tcpdump: ARP, Request who-has pfSense.localdomain tell 192.168.1.140
    IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 16:62:51:c6:e7:de (oui Unknown)


    LAN PC:
    IP 192.168.1.120
    Subnet Mask: 255.255.255.0
    Gateway: 192.168.1.170

    arp -a: ? (192.168.1.170) at ee:1a:d6:89:dc:97 [ether] on enp3s0
    ? (192.168.1.140) at <incomplete> on enp3s0
    ? (192.168.1.141) at <incomplete> on enp3s0
    ? (192.168.1.180) at 00:ea:21:62:38:ce [ether] on enp3s0
    ? (192.168.1.17) at <incomplete> on enp3s0
    ? (192.168.1.1) at <incomplete> on enp3s0

    ping 8.8.8.8: Destination Host Unreachable

    tcpdump: (connects to 192.168.1.120 and 192.168.1.180).


    NIC0: 16:62:51:c6:e7:de (WAN)
    NIC1: ee:1a:d6:89:dc:97 (LAN)
    eth0: 40:16:7e:37:21:af
    eth1: 40:16:7e:37:21:b0
    vmbr0: 40:16:7e:37:21:af
    vmbr1: 40:16:7e:37:21:b0
    vtnet0: 16:62:51:c6:e7:de
    vtnet1: ee:1a:d6:89:dc:97
     
  13. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    So, what is the PVE and VM network topology?
    ISP > ISP modem (bridged) > ISP modem NIC WAN > PVE NIC WAN > VM pfSense vNIC WAN > VM pfSense vNIC LAN > PVE NIC LAN > Switch LAN > 24 ports of LAN devices.

    So, how do the Interfaces and MAC addresses fit into this?
    PVE NIC WAN: don’t know...confused where I sourced the NIC MAC addresses from?
     
  14. hodo

    hodo New Member

    Joined:
    Jun 22, 2016
    Messages:
    28
    Likes Received:
    4
    Hello eiger3970,

    lets work on the gui access.

    I need a photo of 'cat /etc/network/interfaces' on the proxmox host.

    I dont mean informations about the physical connection of your netgate router. What type of connection used it to connect to your isp over them wan-modem?

    What has your netgate router configured at the wan interface? (maybe PPPOE?)

    kind regards
    hodo
     
  15. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Here’s the photo, attached.

    The ISP modem has a coaxial cable to the wall.

    There is no physical hardware router.
    Pfsense is a virtual machine in Proxmox.
     

    Attached Files:

  16. hodo

    hodo New Member

    Joined:
    Jun 22, 2016
    Messages:
    28
    Likes Received:
    4
    Okay, i can see the problem.

    Your Proxmox management ip settings are set on vmbr0. vmbr0 is your wan....

    Set the following like described in post #21:

    Without proxmox gui you have to edit the /etc/network/interfaces:

    auto vmbr0
    iface vmbr0 inet static
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

    auto vmbr1
    iface vmbr1 inet static
    address 192.168.1.140
    netmask 255.255.255.0
    gateway 192.168.1.170
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0

    What is the device with ip 192.168.1.170?


    You said:

    Then you said:

    Iam confused...

    I would suggest the following setup:

    WAN MODEM -> eth0(vmbr0)->nic0->pfsense-vm (WAN IP)
    LAN-Switch->eth1(vmbr1)->nic1->pfsense-vm (192.168.1.1)

    Then all your devices on the LAN-Switch and the vmbr1 should have the efault gateway of 192.168.1.1.

    Could it be, that 192.168.1.170 is yur modem, and your modem is more than only a modem? Then we have to think about other ips, because pfsense can not operate withe wan-subnet = lan-subnet.

    First ste the settings above and check proxmox gui access.

    kind regards
    hodo
     
  17. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
    Thank you, I updated vmbr0/vmbr1 settings, as per photo attached.

    My LAN PC 192.168.1.120 can now ping Proxmox 192.168.1.140, however cannot browse 192.168.1.140? See photo of 192.168.1.120’s settings.

    Not sure why I can’t browse Proxmox GUI, so I added a photo of the vm pfSense settings.

    Please forget/ignore the OLD hardware router, which no longer exists. It is past tense. There is only a virtual machine, qemu-143.conf.

    192.168.1.170 is my preferred IP for the router gateway, rather than using 192.168.1.1.

    The ISP modem is bridged and has no IP.

    I am still confused with what NIC0 and eth0 are?

    I can’t ping or browse the vm pfSense 192.168.1.170 from Proxmox 192.168.1.140 or from the LAN PC 192.168.1.120.
     

    Attached Files:

    #37 eiger3970, Jan 13, 2019
    Last edited: Jan 13, 2019
  18. eiger3970

    eiger3970 Member

    Joined:
    Sep 9, 2012
    Messages:
    234
    Likes Received:
    1
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice