GRE broken after upgrading from 3.3-5 to 3.4-6

[richardvd]

After upgrading from 3.3-5 to 3.4-6 I can't get GRE traffic to/from pfSense VM. Other traffic (ICMP, TCP, UDP) is working fine.
The firewall is disabled at the VM level, so all traffic should pass.

As soon as I disable the firewall at the datacenter level, then GRE traffic to/from the VM is restored.

I want to keep the firewall enabled at the datacenter level, to limit access to the Proxmox nodes from the outside. But I need GRE tunnels to pfSense as well. How to solve?

 

[dietmar]

Does it work if you set VM firewall input policy to 'ACCEPT'?

 

[richardvd]

No, it still blocks GRE to/from the VM with input policy ACCEPT at the Datacenter level and the VM level.

 

[avn]

I think it can be related to this issue http://forum.proxmox.com/threads/21...event-bridge-VM-comunication-on-the-same-node

 

[dietmar]

We need more info - what traffic is blocked exactly (outside to VM, VM to VM, same bridge, ...?)

 

[richardvd]

@avn: No, I don't think it's related to your problem. I tried to reproduce your situation. With the firewall on, I can ping from a host having eth0 linked to vmbrX (vlan N) to another host's eth0 linked to vmbrY (vlan M), with a pfSense router in between to route the two subnets. Both VMs and pfSense are on the same node.

 

[richardvd]

What is blocked with the firewall on:
GRE packets from internet to the public IP of my pfSense VM on vtnet0 (vmbr0), and vice versa.

Using tcpdump I can see on eth0.1 and vmbr0 the GRE packets coming from internet, but the packets do not arrive on vtnet0 in the pfSense VM.

The GRE packets from the pfSense VM to internet are visible using tcpdump on pfSense vtnet0, and on Proxmox vmbr0, but not on Proxmox eth0.1.

By the way: TCP, UDP and ICMP are not affected.

 

[dietmar]

What is eth0.1 exactly? Maybe you can post your /etc/network/interfaces and the VM config of the pfsense VM?

 

[richardvd]

eth0.1 is eth0 with VLAN tag 1, just as eth0.123 would be VLAN 123 on eth0. No problem for Linux and Netgear switches.

/etc/network/interfaces:

[COLOR=#000000][FONT=Consolas]# network interface settings[/FONT]
[FONT=Consolas]auto lo[/FONT]
[FONT=Consolas]iface lo inet loopback[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]iface eth0 inet manual[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]iface eth1 inet manual[/FONT]
[FONT=Consolas]
auto eth2[/FONT]
[FONT=Consolas]iface eth2 inet static[/FONT]
[FONT=Consolas]    address  192.168.18.2[/FONT]
[FONT=Consolas]    netmask  255.255.255.252[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr0[/FONT]
[FONT=Consolas]iface vmbr0 inet static[/FONT]
[FONT=Consolas]    address  x.x.x.252[/FONT]
[FONT=Consolas]    netmask  255.255.255.192[/FONT]
[FONT=Consolas]    gateway  x.x.x.193[/FONT]
[FONT=Consolas]    bridge_ports eth0.1[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr4[/FONT]
[FONT=Consolas]iface vmbr4 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.4[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr6[/FONT]
[FONT=Consolas]iface vmbr6 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.6[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr8[/FONT]
[FONT=Consolas]iface vmbr8 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.8[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr10[/FONT]
[FONT=Consolas]iface vmbr10 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.10[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr11[/FONT]
[FONT=Consolas]iface vmbr11 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.11[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr12[/FONT]
[FONT=Consolas]iface vmbr12 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.12[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr13[/FONT]
[FONT=Consolas]iface vmbr13 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.13[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr14[/FONT]
[FONT=Consolas]iface vmbr14 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.14[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[FONT=Consolas]
[/FONT]
[FONT=Consolas]auto vmbr3[/FONT]
[FONT=Consolas]iface vmbr3 inet manual[/FONT]
[FONT=Consolas]    bridge_ports eth1.3[/FONT]
[FONT=Consolas]    bridge_stp off[/FONT]
[FONT=Consolas]    bridge_fd 0[/FONT]
[/COLOR]

/etc/pve/qemu-server/202.conf (pfSense):

[FONT=Consolas]boot: dc[/FONT]
[FONT=Consolas]bootdisk: virtio0[/FONT]
[FONT=Consolas]cores: 2[/FONT]
[FONT=Consolas]ide2: none,media=cdrom[/FONT]
[FONT=Consolas]memory: 4096[/FONT]
[FONT=Consolas]name: pfSense[/FONT]
[FONT=Consolas]net0: virtio=7A:31:7C:75:1C:4C,bridge=vmbr0[/FONT]
[FONT=Consolas]net1: virtio=F2:A7:DD:B6:95:A1,bridge=vmbr0[/FONT]
[FONT=Consolas]net10: virtio=5E:A7:E8:67:84:3C,bridge=vmbr13[/FONT]
[FONT=Consolas]net2: virtio=B2:AE:FC:DE:54:DC,bridge=vmbr12[/FONT]
[FONT=Consolas]net3: virtio=56:CE:97:C7:19:A7,bridge=vmbr6[/FONT]
[FONT=Consolas]net4: virtio=8A:7B:27:62:BF:B4,bridge=vmbr14[/FONT]
[FONT=Consolas]net5: virtio=22:57:AA:D6:55:A6,bridge=vmbr10[/FONT]
[FONT=Consolas]net6: virtio=AE:8D:B3:00:23:5C,bridge=vmbr11[/FONT]
[FONT=Consolas]net7: virtio=82:6A:8A:ED:6A:32,bridge=vmbr4[/FONT]
[FONT=Consolas]net8: virtio=4A:19:53:DE:04:2D,bridge=vmbr8[/FONT]
[FONT=Consolas]net9: virtio=AA:7D:96:97:6A:55,bridge=vmbr3[/FONT]
[FONT=Consolas]onboot: 1[/FONT]
[FONT=Consolas]ostype: other[/FONT]
[FONT=Consolas]smbios1: uuid=71565135-fb4a-4c89-be8f-6252d45b61e7[/FONT]
[FONT=Consolas]sockets: 1[/FONT]
[FONT=Consolas]virtio0: vm2_data:vm-202-disk-1,cache=writeback,size=4G[/FONT]

pfSense vtnet0 has IP x.x.x.237/26, gateway x.x.x.193.

IP x.x.x.193 is the router of my internet provider.

 

[dietmar]

GRE packets from internet to the public IP of my pfSense VM on vtnet0 (vmbr0), and vice versa.

Are those packets correctly tagged (with vlan id)?

 

[richardvd]

For eth0 you mean? It must be, because the switch port is set to accept and send only tagged frames and it's working with the firewall disabled in 3.4-6 (and with the firewall enabled in 3.3-5). And also, all incoming and outgoing GRE packets are visible using tcpdump on vmbr0, so that rules out any tagging problems on eth0.

Firewall enabled:
outgoing GRE: vtnet0 (OK) -> vmbr0 (OK) -> eth0.1 (missing)
incoming GRE: eth0.1 (OK) -> vmbr0 (OK) -> vtnet0 (missing)

Firewall disabled:
outgoing GRE: vtnet0 (OK) -> vmbr0 (OK) -> eth0.1 (OK)
incoming GRE: eth0.1 (OK) -> vmbr0 (OK) -> vtnet0 (OK)

 

[Kovalchuk Evgeny]

Confirm on 3.4.-9.
GRE traffic on Vms blocked after enabling firewall on datacenter level.

 

[nevakee]

Problem still exists in version 3.4-11.

If I delete the following iptables rule, it works.

iptables -D PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP

 

[pero]

Problem still exists in version 3.4-11.

If I delete the following iptables rule, it works.

iptables -D PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP

I can confirm this bug(?) in proxmox 4.1. Deleting the rule above solves this issue.
But how can I make this change permament? And is there any side effect removing this rule?
Thx

 

[Andrii]

Need to add:

iptables -I FORWARD -p gre -j ACCEPT

 

[wbumiller]

Try loading the nf_conntrack_proto_gre module instead if it's the conntrack rule that causes your problems.

 

[Andrii]

It was not loaded. So in this way I can remove this rule of firewall?

 

[wbumiller]

I think so. Try it. If it works add an entry to /etc/modules-load.d to autoload it on boot.