Hetzner, ProxMox and pfSense

Z

zumo

New Member
Mar 11, 2023
8
1
1
Hello all,

I'm installed ProxMox in Hetzner server with 2 IP to the internet, one of them being is virtual.
The idea here is to set one of the IP for management for ProxMox and pfSense and the other for traffic with pfSense monitoring that traffic from the VMs.

I already have pfSense installed and configure the virtual IP to WAN and a normal internal internal network.

I've followed instructions from Hetzner documentation and I have no clue why the pfSense can't reach the any external links.
https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve

Is there something wrong with this configuration?
I'm I missing something in the bridge networking?

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

# ProxMox Management
auto enp3s0
iface enp3s0 inet static
        address xxx.xxx.xxx.145/26
        gateway xxx.xxx.xxx.129
        up route add -net xxx.xxx.xxx.128 netmask 255.255.255.192 gw xxx.xxx.xxx.129 dev enp3s0
# route xxx.xxx.xxx.128/26 via xxx.xxx.xxx.129


iface enp3s0 inet6 static
        address 2a01:4f8:212:2490::2/64
        gateway fe80::1

iface eth0 inet manual

# Bridge configuration with extra IP WAN pfSense
auto vmbr0
iface vmbr0 inet static
  address xxx.xxx.xxx.190/26
  bridge_ports none
  bridge_stp off
  bridge_fd 0

# Internal network
auto vmbr1
        iface vmbr1 inet static
        address 192.168.30.1
        netmask 255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0


Any help would be highly appreciated.
 

Attachments

  • Captura de ecrã 2023-03-11 101758.png
    Captura de ecrã 2023-03-11 101758.png
    38.9 KB · Views: 54
Hello,

so you want to give your two public IPs to your pfSense? Then you can't configure them on your Proxmox too. A bridge in Proxmox is both a linux bridge (= virtual Switch) and an interface held by the Proxmox OS.

For example, your vmbr0 has the IP xxx.xxx.xxx.190/26, and when you connect your pfSense to that same bridge and give that virtual interface that IP xxx.xxx.xxx.190/26 too, you have an IP address conflict.

I guess we have to set up first what you want to do. I think it is clear that xxx.xxx.xxx.190 should be given to your pfSense. But how do you want to do the Proxmox management?
  • Option1: xxx.xxx.xxx.145 is held by Proxmox. This exposes the Proxmox host to the internet directly.
  • Option2: xxx.xxx.xxx.145 is held by your pfSense, and in order to connect to your Proxmox management you have to go through pfSense. But you can only reach the Proxmox Management if your pfSense VM is running.
Kind regards,
Benedikt
 
Hello Benedikt,

Thank you for you input, although I have planed what is in the schema I have currently option 1 and I believe that I will stick with that one..
Although I'm facing the issue that the pfSense can not reach the internet. I've created a VM with the internal network (vmbr1) and I'm able to reach pfsense. Going to the internet is another problem. Summarizing, ProxMox is reacheable from the internet with the IP .145. pfSense is using the .190 to reach to the internet not working, but the internal bridge is working fine.

At the moment I need to understand what am I doing wrong in the configuration in order for the pfSense not to reach the internet.

Thank you.

Kind regards,
Zumo
 
zumo said:
Hello Benedikt,

Thank you for you input, although I have planed what is in the schema I have currently option 1 and I believe that I will stick with that one..
Although I'm facing the issue that the pfSense can not reach the internet. I've created a VM with the internal network (vmbr1) and I'm able to reach pfsense. Going to the internet is another problem. Summarizing, ProxMox is reacheable from the internet with the IP .145. pfSense is using the .190 to reach to the internet not working, but the internal bridge is working fine.

At the moment I need to understand what am I doing wrong in the configuration in order for the pfSense not to reach the internet.

Thank you.

Kind regards,
Zumo
Click to expand...
Yeah from your schema it was not clear whether Proxmox or pfSense holds the 145 IP, since there is a connection from 'MGMT' to your pfSense.

Well, I would be surprised if your pfsense managed to connect to the internet with your posted interfaces config. There is no way packets on vmbr0 can reach your 'WAN-interface' enp3s0.

In order to present both a Proxmox and a VM interface to your port, you have to restructure it a bit. Instead of giving your enp3s0 directly to Proxmox, you have to set it as a bridge-port for your 'WAN-bridge' vmbr0. The Proxmox-IP xxx.xxx.xxx.145 will then be assigned to vmbr0 and the virtual WAN-interface of your pfSense is then connected to vmbr0. You also have to adjust the pre-configured if up routing rules to adhere to the new interface.

Before you start, do note that what you are doing is changing the way you connect to your Proxmox, so make sure that you have a way to access to your Proxmox if something in the network config fails.

I would try something along this (I'll omit the IPv6 stuff):

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
  address xxx.xxx.xxx.145/26
  gateway xxx.xxx.xxx.129
  up route add -net xxx.xxx.xxx.128 netmask 255.255.255.192 gw xxx.xxx.xxx.129 dev vmbr0
  bridge-ports enp3s0
  bridge-stp off
  bridge-fd 0
# WAN-Bridge

auto vmbr1
iface vmbr1 inet static
  bridge-ports none
  bridge-stp off
  bridge-fd 0
# Bridge for internal network

I'll suppose your internal subnet would be 192.168.30.0/24 with Gateway 192.168.30.1 held by your pfSense. Your pfSense would be configured like such:
  • WAN-Interface
    • connected to vmbr0
    • IP: xxx.xxx.xxx.190/26, Gateway xxx.xxx.xxx.129
  • Internal Interface
    • connected to vmbr1
    • IP: 192.168.30.1/24
  • The usual Firewall rules like NAT/Masquerading, I assume you know how to configure them in your pfSense
Do note that Bridges can also be configured without IP adresses since they work on Layer2.

Kind regards,
Benedikt
 
Last edited:
Benedikt,

thank you for assist.
I've made the suggested changes.

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp3s0
iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address xxx.xxx.xxx.145/26
        gateway xxx.xxx.xxx.129
        up route add -net xxx.xxx.xxx.128 netmask 255.255.255.192 gw xxx.xxx.xxx.129 dev enp3s0
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 00::1

auto vmbr1
iface vmbr1 inet static
        address 192.168.30.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

getting this error when doing the #ifreload.
Code: 
root@Proxmox-VE ~ # ifreload --all
error: enp3s0: cmd '/bin/ip route add default via fe80::1 proto kernel dev enp3s0 onlink' failed: returned 2 (Error: IPv6 is disabled on nexthop device.)
warning: vmbr0: up cmd 'route add -net xxx.xxx.xxx.128 netmask 255.255.255.192 gw xxx.xxx.xxx.129 dev enp3s0' failed: returned 7 (SIOCADDRT: Network is unreachable)

Is it mandatory to reboot the host?

Kind regards,
Zumo
 
I suppose the enp3s0 error has something to do with your ipv6 config. Generally I disable IPv6, so I cant help you with that.

The warning is related to the up route command. I think I made a mistake in the replacing, it should route it via the interface vmbr0, not enp3s0. Also I don't think we should assign the gw here:
Code: 
up route add -net xxx.xxx.xxx.128 netmask 255.255.255.192 dev vmbr0
 
Made the change as requested also change the IP address of pfSense WAN, no go.

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp3s0
iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address xxx.xxx.xxx.145/26
        gateway xxx.xxx.xxx.129
        up route add -net xxx.xxx.xxx..128 netmask 255.255.255.192 dev vmbr0
# route xxx.xxx.xxx..128/26 via xxx.xxx.xxx..129
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.1.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Also getting a warning about the vnbr0.
root@Proxmox-VE ~ # ifreload --all warning: netlink: vmbr0: cannot delete address 136.243.70.145/26 dev vmbr0: operation failed with 'Cannot assign requested address' (99)

This is so frustrating this should be simple I mean I have 2 outgoing IP one for ProxMox and the other of pfSense VM traffic.
I cannot even make pfsense reach google DNS or any other website.
 
Changed a few parameters in the interface file but still no go.
Also configure the NAT rule in pfSense and GW, same outcome.

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address xxx.xxx.xxx.145/26
        gateway xxx.xxx.xxx.129
        up route add -net xxx.xxx.xxx.128 netmask 255.255.255.192 dev vmbr0
# route xxx.xxx.xxx.128/26 via xxx.xxx.xxx.129
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.1.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

No error nor warning when running the ifreload.
 
  • Like
Reactions: yashodhan
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom