Proxmox on Dedicated Hetzner, on the VM with pfSense ping works but TCP only works sometimes

[blanco]

Hello everybody,

I have a dedicated on Hetzner on which I have installed Proxmox.
The problem I have is that with both Bridge and Routed configuration, I only have ping which works 100% while TCP connections only work sometimes (3/10 or so).

Configuration:
Host with Debian 10 and Proxmox (public and private IP)
VM: Pfsense (with only private IP and gateway on private HOST IP)


HOST Network: (/etc/network/interfaces):
auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp4s0
iface enp4s0 inet static
address 144.7x.xx.xx
netmask 255.255.255.224
pointopoint 144.7x.xx.xx
gateway 144.7x.xx.xx

up route add -net 144.7x.xx.xx netmask 255.255.255.224 gw 144.7x.xx.xx dev enp4s0
auto vmbr0
iface vmbr0 inet static
address 144.7x.xx.xx
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0

# LAN
auto vmbr1
iface vmbr1 inet manual
address 192.168.10.254
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 1



VM NETWORK:

192.168.10.1/24
Gateway 192.168.10.254 (HOST Private IP)


Forward on HOST is enabled and there is also a masquerade rule for the enp4s0 interfaces:
15585 1083K MASQUERADE all -- * enp4s0 0.0.0.0/0 0.0.0.0/0

TEST:
on the VM ping always work.
but curl (TCP,. HTTP), works only 3-4 times on 10):

When is KO:
[2.5.2-RELEASE][root@xxxxxx]/root: curl -L -I https://1.1.1.1


When is OK:
[2.5.2-RELEASE][root@xxxxxxxx/root: curl -L -I https://1.1.1.1
HTTP/2 200
date: Thu, 15 Jul 2021 07:46:55 GMT
content-type: text/html
last-modified: Thu, 17 Jun 2021 15:48:54 GMT
x-amz-request-id: tx00000000000003f1c71ae-0060efe665-424d0ed-default
strict-transport-security: max-age=31536000
served-in-seconds: 0.002
cache-control: public, max-age=14400
cf-cache-status: HIT
age: 394
expires: Thu, 15 Jul 2021 11:46:55 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 66f161376854dfd7-FRA



On the HOST ping and TCP are always ok.

I also opened a ticket to Hetzner but they told me that there are no problems on their side.

What am I doing wrong?
Thank you!

 

[blanco]

I tried with Linux (Ubuntu 20.04) as VM and Ping and TCP (curl) works at 100%.

Consequently the problem is on pfSense.
Anyone know more?

 

[flames]

Hi,
PVE7.x Linux Bridge does not work currently on Hetzner. You could fix your installation following way:

remove vmbr (all vmbrs) from /etc/network/interfaces. leave only the physical interface. name might be different for you (to list all available interfaces try ls -d /sys/class/net/*)

auto lo
iface lo inet loopback


auto enp0s1
iface enp0s1 inet static
        address xxx.xxx.xxx.xxx/xx #your main IP + Mask in CIDR notation
        gateway xxx.xxx.xxx.xxx #your main ip gateway as you see in hetzner robot
        pontopoint xxx.xxx.xxx.xxx #your main ip gateway as you see in hetzner robot

reload interface ifreload -a

now you can access your host via ssh and gui on the main ip.

install OVS apt update && apt install openvswitch-switch

now in network settings you can via gui create a new OVS Bridge! Not Linux Bridge!

the only setting you need in the new OVS Bridge is one of the IPs and mask of your Ripe block or your extra single IP (CIDR notation again), you can leave all other fields empty (even Bridge Ports).

If you created your OVS Bridge with a different name (vmbr0/vmbr1/...) edit the virtual NICs of your VMs and Containers in the corresponding hardware options.

Edit: I have bridged setup, this is how i fixed my hetzner host after upgrade to pve7.0.1
I don't know, how to use routed setup at all, never used it on hetzner

edit2: you might need to order (free) the kvm console.

my /etc/network/interfaces looks like this now, everything is working:

auto lo
iface lo inet loopback


auto enp0s1
iface enp0s1 inet static
        address main.ip.adress/26
        gateway main.gateway.ip
        pontopoint main.gateway.ip


auto vmbr0
iface vmbr0 inet static
        address first.usable.block.ip/29
        ovs_type OVSBridge

 

[blanco]

Thanks flames!

Just one more question, I only have one additional IP that I should assign to the VM firewall (pfSense).
pfSense should therefore have two IPs: the additional public IP and another LAN.

Hosr is configured as you told me:

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp4s0
iface enp4s0 inet static
        address 144.xxxxxx/27
        gateway 144.xxxxxx
        pointopoint 144.xxxxxx
        up route add -net 144.xxxxxx netmask 255.255.255.224 gw 144.xxxxxxx dev enp4s0

auto vmbr0
iface vmbr0 inet static
        address 144.xxxxx/32 (Additional IP)
        ovs_type OVSBridge

# LAN
auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge

In pfSense I have

vtnet0 with  144.xxxxx/27 (Additional IP)
and
vtnet1 with a private IP for VMs LAN

gateway: Main IP (HOST)

But that way neither ping nor TCP works.
If I sniff the traffic with tcpdump on the HOST I don't see the packets passing through.

Where am I doing wrong?
Thank you so much for your help!

 

[flames]

Hi @blanco,
is the additional ip from the same subnet or different?
if its in same subnet as the main ip, you could just assign the main ip to your vmbr0 instead of enp4s0 (leave enp4s0 all fields empty) and the additional ip configure as wan ip in the pfsense.
obviously, if you assigned your main ip to enp4s0 and the additional ip to vmbr0, you can't assign it to vtnet0 in pfsense anymore, since it is now in use by vmbr0 on the host.
if the additional ip is in a different subnet, then you can only go with routed network on host side. sadly, i can't help you with routed network, since i never used such setup and have absolutely no clue how to configure it right.

 

[blanco]

Hello flames!

I tried your configuration, but with pfSense I have same problem... Ping perfect but TCP connections sometimes KO

auto lo
iface lo inet loopback

iface lo inet6 loopback

allow-vmbr0 enp4s0
iface enp4s0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0

allow-ovs vmbr0
iface vmbr0 inet static
        address 144.xxxxxx/27
        gateway 144.xxxxxx
        ovs_type OVSBridge
        ovs_ports enp4s0

allow-ovs vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge

Do you have other ideas?

Very Thanks for your help and your time!

 

[blanco]

I ordered a new dedicated in Hetzner (in another DC) and now with it, all works.
It's a Hetzner issue.
Thanks to flames for the help!