proxmox ve bridged setup - guest no internet connection (hetzner)

[Frankenstein]

Hey guys,

actually setting up a dedicated from hetzner with bridged setup, cauz we wanna later set a OPNsense VM in front of it as gateway. For the first we are using a little debian vm to test the network setup.

Hostsystem /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.2/26
        gateway 10.0.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 1
        pointopoint 10.0.0.1
        bridge_hello 2
        bridge_maxage 12

auto vmbr1
iface vmbr1 inet static
        address 10.0.0.3/26
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr2
iface vmbr2 inet static
        address 10.0.0.4/26
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr192
iface vmbr192 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Guest system /etc/network/interfaces:

auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
    address 10.0.0.4/26
    pointopoint 10.0.0.1
    gateway 10.0.0.1

auto ens19
iface ens19 inet manual

As for the MAC addresses for each additional ip address, i have set the mac address which we have ordered for the additional ip address in the Proxmox interface under the VM Hardware as network device the vmbr2 with the generated mac address

Proxmox, VM and Hetzner Firewall took off for testing purposes, but still nothing on the tcpdump output:

myuser@hypervisor:~$ sudo tcpdump -i vmbr2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr2, link-type EN10MB (Ethernet), capture size 262144 bytes
15:00:22.466034 ARP, Request who-has static.1.0.0.10.clients.your-server.de tell vm-hostname.domain.tld, length 46
15:00:25.437667 ARP, Request who-has static.1.0.0.10.clients.your-server.de tell vm-hostname.domain.tld, length 46

myuser@hypervisor:~$ sudo tcpdump -i vmbr0 | grep vm-hostname.domain.tld
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
196 packets captured
266 packets received by filter
0 packets dropped by kernel

myuser@hypervisor:~$ sudo tcpdump -i vmbr0 | grep 10.0.0.4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
19 packets captured
34 packets received by filter
0 packets dropped by kernel

myuser@hypervisor:~$ netstat -ai
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eno1      1500    96419      0      0 0         37925      0      0      0 BMRU
lo       65536    41996      0      0 0         41996      0      0      0 LRU
tap102i0  1500      521      0      0 0             2      0      0      0 BMPRU
tap102i1  1500       14      0      0 0             2      0      0      0 BMPRU
vmbr0     1500    22850      0      0 0         28960      0      0      0 BMRU
vmbr1     1500        0      0      0 0             8      0      0      0 BMU
vmbr2     1500      626      0      0 0            12      0      0      0 BMRU
vmbr192   1500      150      0      0 0            12      0      0      0 BMRU

myuser@hypervisor:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2 peer 10.0.0.1/32 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aaaa:aaaa:aaaa:aaaa/64 scope link
       valid_lft forever preferred_lft forever
11: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/26 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 aa:aa:aa:aa:aa:aa/64 scope link
       valid_lft forever preferred_lft forever
12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bb:bb:bb:bb:bb:bb brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.4/26 scope global vmbr2
       valid_lft forever preferred_lft forever
    inet6 aa:aa:aa:aa:aa:aa/64 scope link
       valid_lft forever preferred_lft forever
13: vmbr192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether cc:cc:cc:cc:cc:cc brd ff:ff:ff:ff:ff:ff
    inet6 aa:aa:aa:aa:aa:aa/64 scope link
       valid_lft forever preferred_lft forever
18: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether dd:dd:dd:dd:dd:dd brd ff:ff:ff:ff:ff:ff
19: tap102i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr192 state UNKNOWN group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff

• 10.0.0.1 gateway of our main and additional ip addresses
• 10.0.0.2 proxmox host public ip
• 10.0.0.3 & 10.0.0.4 the additional ip addresses
• proxmox firewall is completely turned off
• hetzner firewall is completely turned off
• mac address of net0 (vmbr2 -> ens18) under "Hardware" of the VM is set to the additional mac address for additional ip address

Anybody a idea what i'm doing wrong?

 

[angelkiller]

Hi Frankenstein,

we also have a hetzner root server with proxmox in our company.
Did you set the additional ip adresses on vm bridges on the proxmox host? If yes this is wrong. You have to setup a router vm and give this vm for every additional ip a nic with the corresponding mac adress and linked to the bridge where your main ip is connected with the real nic. Than this router vm nic' s can dhcp for their ip settings into the hetzner network. When this is successfull you can set the settings as static settings.

 

[angelkiller]

Hi Frankenstein,

i looked closer to your config file from host. you can answer yourself the problem with this info:

auto vmbr0
iface vmbr0 inet static
address 10.0.0.2/26
gateway 10.0.0.1
bridge-ports eno1
bridge-stp off
bridge-fd 1
pointopoint 10.0.0.1
bridge_hello 2
bridge_maxage 12

auto vmbr1
iface vmbr1 inet static
address 10.0.0.3/26
bridge-ports none
bridge-stp off
bridge-fd 0

A VM NIC that is connected to vmbr0 has a chance to communicate outside into the hetzner network, while there is a real port (eno1) connected to.
Any VM that has a NIC that is not connected to vmbr0 (vmbr1, vmbr2) can only communicate to other VM Nics that also connected to these bridges. With that you can only communicate from VM1 to VM2 inside your Proxmox datacenter.

So you need to delete the IP settings from vmbr1 and vmbr2 and link your test vm to vmbr0 with the correct mac address from your additional ip under vm settings - network interface - mac address. Than your test vm will become the correct ip settings from dhcp and can talk outside into the WWW.

Best Regards
Klaus

 

[Frankenstein]

Hey Klaus,

i think i understand the most what you mean - thank you for investigating. So if i understand it correct and setup like this, i think their would be a little misconfiguration. The vmbr0 in this case, is bridged with eno1, which is the real NIC from the host. vmbr0 is used by the host. So for the actually setup, which we wanna change to a OPNsense gateway as DHCP (which was thought for vmbr1)

But when i read it again in this second...did u mean the following?

1. remove the ip addresses from vmbr1 and vmbr2
2. add the vmbr0 with the configured static ip and mac from host to the additional VM
3. add the vmbr1 with their own additional mac address from additional ip to the additional VM to get DHCP working (for their own pub ip)

If yes, i missed something more in network technology at school than i thougt.

Further, would u give a little kick (i dont like direct solutions) how the configuration need to get changed to get the OPNsense VM with additional ip as gateway and dhcp for the proxmox host and all the other additional VMs? In this time i will remove the dust from my network tech books and try to get a fresh up

Best Regards
Frankenstein

 

[angelkiller]

Hi Frankenstein,

i think the best thing in your case is to remove vmbr1 and vmbr2 for better understanding. Reboot the server after removing.
Than you only have vmbr0 with the main ip from your hetzner root server. Than build a vm and in the nic configuration set the mac address of one additional ip that hetzner was giving to you. Mac adresses have to generate under your server setup, but i think that was done. This mac address for that ip you have to use in the vm nic settings. Now the vm can only be bridged to vmbr0 and will successfull get the additional ip from hetzner dhcp server. If this is working, than you can plan to setup your router vm with opnsense or whatever you like.
The main ip you can' t bring to your router vm. The main ip must stay at the vmbr0 bridge for accessing the root server. It is possible to setup szenario with nat that unused ports (not ssh and not 8006 WebUI) can pass to a router vm. It is possible but need much more configuration. I read about that in a forum or saw a youtube video explaining that. But for better security we setup our root with additional ip' s and a watchguard firebox v as router vm.

Write if you need further help!

BR
Klaus

 

[Frankenstein]

The main ip you can' t bring to your router vm. The main ip must stay at the vmbr0 bridge for accessing the root server.

We discussed about that - what would you say to block the Proxmox WebUI over the Hetzner FW from external access, just over the VPN specific IP (VPN coming from router vm) gets access. So, if i didnt be wrong, the WebGUI should be safe and just accessible when we are connected to our VPN Network.

 

[angelkiller]

Yes that can be done and we only allow webui and ssh from our public company ip in hetzner firewall to be secure our root server main ip. With this config we can reach the webui and ssh even when the router vm has any issues over the public main ip. Than we can fix any problem. And only we can connect from our static public company ip. You can configure 10 rules in hetzner firewall.
Than the fireboxV with additional ip connect our internal company net with vm net on proxmox host with ipsec vpn.

 

[Frankenstein]

Hey Klaus,

after short break after a accident its now going forward:

- the Proxmox UI actually is only reachable from our static addresses at home (Hetzner FW)
- the OPNsense has now the WAN with DHCP and correct ip address, which we are able to "ping"

Its a dumb, but serious question. So cauz its not recommended to bind the route vm web ui to the WAN, we created again the vmbr1, attached it to the OPNsense, configured the interface and OPNsense got the ip 192.168.1.1/24 from it. The OPNsense now showing the WAN and LAN with the named ip addresses. Default the web ui working the LAN - but how to access?

I installed a ubuntu 20.04 with KDE, ssh and system-default-worktools - i created in the NetworkManager two connections. The first was the WAN, setup as DHCP to the ens18 which is the vmbr0 with correct mac address in hardware details, the second is the LAN setup as DHCP to the ens19 which is the vmbr1.

I'm able to open websites, find my ip over such websites, get a source list update and dist-upgrade without issues on the VMs firefox. PING the device from external isnt possible. A traceroute follows until the device was reached. A ssh connection try didnt getting any response (ssh running and configured).

If i understand it correct, the OPNsense should generally work as router/dhcp after initial setup for the LAN so without further configuration the LAN of the additional ubuntu VM should DHCP a IP from it and the WebUI should be available over 192.168.1.1 over the firefox e.g.

But it isnt...I will start to research - eventually the issue is already readable in my setup. I will edit this post when i got some logs for interest.

 

[angelkiller]

Hi, one ip is the main ip from your root server. And how many additional ip you ordered?
Best practice i think is to have first bridge for all external network access and one or two bridges only for internal things.
Router VM with nic1 connect to vmbr0 (external internet connect) nic2 connect to vmbr1 for internal vm network and nic3 connect to vmbr2 for a second internal network.
We did that to connect our webserver in network1 and our mail server into network2. And every traffic need from webserver net to mail server net must go over the firewall vm. So we can filter the traffic for better security.
When you have setup opensense it will be reachable from a vm that is connected to lan from the opensense internal net. Than if you need to reach services on vm in lan network you must configure static nat rules in opensense firewall. Are you familar with configuring such things?

BR
Klaus

 

[Frankenstein]

Actually we have 2 additional ip addresses (with seperate macs) - so the proxmox with his own, the router vm with his own and the maintainer (desktop for web ui opnsense) with his own.

I have setup the vmbr0 with eno1 as bridge port - nothing else. Then i configured the OPNsense with vmbr0 (with seperate mac) and vmbr1. Then i started the OPNsense, configured the ens18 (vmbr0) as DHCP WAN and the ens19 (vmbr1) as DHCP LAN - it got the public additional ip/26 as WAN and 192.168.1.1/24 as LAN.

Then i created a debian with ens18 DHCP (vmbr0 and additional mac) and ens19 DHCP (vmbr1). After setup the device got the public additional ip address/26 as WAN (ens18) and 192.168.1.101 as LAN (ens19).

For the first, the second additional VM is our maintainer to access the web ui - later when the OPNsense is configured well, we use the OPNsense internal VPN to connect to the web ui from home.

So sounds like the same as you said, just without the vmbr2 (cauz actually no services running - for the first we wanna just configure the router vm well now before setting up other vms with services).

About natting - i'm not alone with that little learning project - my colluege is more fit than me in nat and will explain when we are ready to access the web ui.

So - my thought was, that the second additional vm should be in the internal net from opensense, cauz both connected with DHCP to the vmbr1. But seems that the OPNsense or the second VM need additional configuration to access the same internal net.

I will take the night to read the documentation more - i think their is again any misunderstanding from my side while config.

Best regards
Frankenstein

 

[angelkiller]

Setup your debian/management vm with only one nic to vmbr1 internal opensense network. Setup this vm with ip from internal opensense net like 192.168.1.5/24 and gateway the opensense ip 192.168.1.1
Now connect via proxmox management console to this debian vm and try to get with firefox to opensense webui. When this works, you can setup your firewal rules vpn and all your needs.

 

[Frankenstein]

Setup your debian/management vm with only one nic to vmbr1 internal opensense network. Setup this vm with ip from internal opensense net like 192.168.1.5/24 and gateway the opensense ip 192.168.1.1
Now connect via proxmox management console to this debian vm and try to get with firefox to opensense webui. When this works, you can setup your firewal rules vpn and all your needs.

Done, but still no access to the ui. Hmmm...Its getting late, will research tomorrow.

 

[angelkiller]

Can you Ping the opensense ip 192.168.1.1 from Debian vm?
Did you use https instead of http://192.168.1.1?
Opensense lan nic and debian nic connected to the same vmbr1?

If nothing helps i can have a look over teamviewer to your setup?

 

[angelkiller]

I read your earlier post more in detail now. When you get i ip configuration in your debian/maintenance vm from opensense dhcp server (192.168.1.101) what you wrote, than the network configuration is correct. Than it must be possible to open the opensense webui. Otherwise i think there is a misconfiguration in opensense. You can login to your opensense via proxmox terminal and get into the command line. If you enter the following, the firewall is temp offline and you can login to the webui from every network address. Also over WAN with the public ip. So be aware that you setup a good pw and after setting the correct rules reboot the vm to bring up normal state.

Enter:
pfctl -d

Or you delete the opensense vm and try to setup a new one with correct nic setup from the beginning.

 

[Frankenstein]

So i'm back over here - after a fresh setup of a new kubuntu (going faster) in minimal setup the OPNsense was available.

Actually, the Hetzner FW looks like:


the thing is - if we turn off the hetzner fw, the maintainer (kubuntu) vm has a internet connection. Checked "wieistmeineip.de" and got the public ip from the gateway shown - seems like the vm getting routed over the gateway from default?

If we turn up the hetzner fw, the maintainer hasnt a internet connection.

------does this seems normal? - damn, first day feelings.

About the nat - actually we did not understand how to, but will read into it. Thanks for your helpful and friendly help

 

[brandonlea]

Did you manage to fix it because I have the same issue.

 

[angelkiller]

Please setup a additional rule with the following data in hetzner Firewall and test again:

Name: allow ack
Source IP: No entry
Destination IP: No entry
Source port: No entry
Destination port: 1-65535
Protocol: tcp
TCP flags: ack
Action: accept

When this works you can setup static nat in OPNsense firewall. If you need help write back.