pfSense VLAN - partially broken internet

[shorez]

Hey folks!
I've successfully created something very weird.
First things first, this is my current Setup

Hardware:

• Managed Switch
• Intel NUC running Proxmox (only one Network Interface Card)
• several Clients

2 physical networks:

• WAN: 192.168.178.0/24
• LAN: 172.19.10.0/24

Managed Switch:

• WAN on VLAN5-untagged
• Intel NUC on VLAN10-untagged, on VLAN5-tagged and VLAN11-tagged
• Several Clients on VLAN10-untagged and VLAN11-untagged

Proxmox Setup:

• vmbr0: static IP 172.19.10.5, VLAN-aware
• pfSense VM
  • net0: virtio, bridge=vmbr0,tag=5
  • net1: virtio, bridge=vmbr0
  • net2: virtio, bridge=vmbr0,tag=11

pfSense Configuration:

• WAN: net0 (set to dhcp)
• LAN: net1 (static IP 172.19.10.1)
• IOT: net2 (static IP 172.19.11.1)

Problem:
Internet access is partially broken. ALL CLIENTS and the pfSense VM itself are able to access the internet. They can ping, dnsresolve and http, etc.
The proxmox node itself and every other VM I create have a little problem: They are still able to ping and dnsresolve. But http is .. weird. Here a wget example:

In the following, the URL of german google is used. Cannot write it as I'm not allowed to by the forum!?

PROXMOX HOST:

root@proxmox:~# wget URL
--2018-05-21 17:27:43--  URL
Resolving URL (URL)... 216.58.213.227, 2a00:1450:4005:80a::2003
Connecting to URL (URL)|216.58.213.227|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: URL [following]
--2018-05-21 17:27:43--  URL
Resolving URL (URL... 216.58.213.227, 2a00:1450:4005:80a::2003
Reusing existing connection to URL:80.
HTTP request sent, awaiting response...

At this point it gets stuck forever.
But the clients work fine:

CLIENT:

$ wget URL
--2018-05-21 17:27:19--  URL
Resolving URL (URL)... 216.58.213.227, 2a00:1450:4005:80a::2003
Connecting to URL(URL)|216.58.213.227|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: URL [following]
--2018-05-21 17:27:19--  URL
Resolving URL (URL)... 216.58.213.227, 2a00:1450:4005:80a::2003
Reusing existing connection to google.de:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html              [ <=>                ]  10,26K  --.-KB/s    in 0,005s 

2018-05-21 17:27:19 (1,93 MB/s) - ‘index.html’ saved [10511]

But something internal works fine as well on the Proxmox Host:


Here I used the pfSense address.
PROXMOX HOST:

root@proxmox:~# wget --no-check-certificate pfsense.kss19.de
URL transformed to HTTPS due to an HSTS policy
--2018-05-21 17:31:53--  URL
Resolving URL (URL)... 172.19.10.1
Connecting to URL (URL)|172.19.10.1|:443... connected.
WARNING: The certificate of 'URL' is not trusted.
WARNING: The certificate of 'URL' hasn't got a known issuer.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'

index.html                             [ <=>                                                           ]   8.82K  --.-KB/s    in 0s     

2018-05-21 17:31:53 (156 MB/s) - 'index.html' saved [9032]

The pfSense firewall is set to block anything incoming on the WAN and allowing the LAN from everywhere to everywhere.
There are no "BLOCK" logs. So the firewall shouldn't get in the way, right?

I think I'm missing anything in the network setup that causes these problems.
Hopefully one of you can help me with this mess,
~shorez

 

[bobmc]

I think a major part of your problem is the fact you are trying to use a single LAN connection on the Proxmox host for both WAN and LAN

Lots of people successfully run pfSense on proxmox as an internet gateway but you really need two physical nics to make it work.

 

[shorez]

I think a major part of your problem is the fact you are trying to use a single LAN connection on the Proxmox host for both WAN and LAN

Lots of people successfully run pfSense on proxmox as an internet gateway but you really need two physical nics to make it work.

Exactly this is what I think as well.
The passthrough setup is no problem, I've it up and running for a long time on a different network.
But I didn't see a problem with the single NIC thing .. it sounds crazy but I wanted to know whether it's possible. And so far it looks like I'm close to success, because early everything works .. even the WAN on everything but Proxmox.

But I don't know enough about how networks work on a hardware level that I could tell what's killing me here.
A solution would be to buy an USBtoEthernet thing for the NUC .. I'm gonna do this as a last thing, first I want to know why this isn't possible what I have here right now

~shorez

 

[shorez]

Exactly this is what I think as well.
The passthrough setup is no problem, I've it up and running for a long time on a different network.
But I didn't see a problem with the single NIC thing .. it sounds crazy but I wanted to know whether it's possible. And so far it looks like I'm close to success, because early everything works .. even the WAN on everything but Proxmox.

But I don't know enough about how networks work on a hardware level that I could tell what's killing me here.
A solution would be to buy an USBtoEthernet thing for the NUC .. I'm gonna do this as a last thing, first I want to know why this isn't possible what I have here right now

~shorez

btw single NIC setups seem to work .. at least with bare-metal pfSense
blog.spirotot.com/2016/06/28/pfsense-vlans-with-one-nic-nuc-a-tp-link-tl-sg108e/

I don't know what happens to Proxmox, but the problem relies on it's side I guess