[SOLVED] Containers cant communicate through Pfsense VPN client VM

Navity

New Member
Nov 16, 2019
2
0
1
26
Hey guys,
I have been trying to get this to work for several days now, for the life of me i cant figure out whats going wrong.
To summaries before details, in Proxmox I have a PFsense VPN client VM that is only for a specific network, If i connect to that network with an AP my phone/laptop can use the VPN just fine; and pfsense can connect through the VPN. If i connect a Proxmox VM/LXC it can send traffic out but never receives (except Pings). Note, these VM/LXC can connect on the normal network fine.

Proxmox interfaces:
Code:
auto lo
iface lo inet loopback
iface enp3s0 inet manual
iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address  192.168.8.8
    netmask  24
    gateway  192.168.8.1
    bridge-ports enp3s0
    bridge-stp off
    bridge-fd 0
#Normal network

auto vmbr5
iface vmbr5 inet static
     address  192.168.9.0
     netmask  24
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
#VPN Network
Container Interfaces
Code:
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

iface eth0 inet6 auto
Pfsense
Code:
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether fe:7d:cc:a9:f7:ab
    hwaddr fe:7d:cc:a9:f7:ab
    inet6 fe80::fc7d:ccff:fea9:f7ab%vtnet0 prefixlen 64 scopeid 0x1
    inet 192.168.8.3 netmask 0xffffff00 broadcast 192.168.8.255
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6c00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether c2:5d:9a:fc:91:e6
    hwaddr c2:5d:9a:fc:91:e6
    inet6 fe80::c05d:9aff:fefc:91e6%vtnet1 prefixlen 64 scopeid 0x2
    inet 192.168.9.1 netmask 0xffffff00 broadcast 192.168.9.255
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
enc0: flags=0<> metric 0 mtu 1536
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: enc
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
pflog0: flags=100<PROMISC> metric 0 mtu 33160
    groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
    groups: pfsync
    syncpeer: 224.0.0.240 maxupd: 128 defer: on
    syncok: 1
ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    inet6 fe80::9814:5b1d:7bbd:2479%ovpnc1 prefixlen 64 scopeid 0x7
    inet6 fdda:d0d0:cafe:1301::1009 prefixlen 64
    inet 10.15.0.11 --> 10.15.0.1 netmask 0xffff0000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: tun openvpn
    Opened by PID 26607
If i ping our of the LXC container I can follow the packets all the way out of my router to the VPN server, however it just never receives anything back. At first i thought it was a DNS issue, after using PFsense as the DNS and he upstreams to the VPN the LXC container can resolve hostnames correctly, and even ping the IP addresses it just fails to make any other connection to the addresses.

From LXC Container over PFsense VPN Client
Code:
root@lxc-transmission-02:~# dig reddit.com

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> reddit.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11803
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;reddit.com.            IN    A

;; ANSWER SECTION:
reddit.com.        300    IN    A    151.101.65.140
reddit.com.        300    IN    A    151.101.1.140
reddit.com.        300    IN    A    151.101.129.140
reddit.com.        300    IN    A    151.101.193.140

;; Query time: 95 msec
;; SERVER: 192.168.9.1#53(192.168.9.1)
;; WHEN: Sat Nov 16 15:59:36 UTC 2019
;; MSG SIZE  rcvd: 103

root@lxc-transmission-02:~# ping 151.101.65.140
PING 151.101.65.140 (151.101.65.140) 56(84) bytes of data.
64 bytes from 151.101.65.140: icmp_seq=1 ttl=60 time=11.2 ms
64 bytes from 151.101.65.140: icmp_seq=2 ttl=60 time=11.1 ms
64 bytes from 151.101.65.140: icmp_seq=3 ttl=60 time=11.3 ms
^C
--- 151.101.65.140 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 5ms
rtt min/avg/max/mdev = 11.067/11.193/11.288/0.153 ms

root@lxc-transmission-02:~# wget reddit.com
--2019-11-16 16:01:24--  http://reddit.com/
Resolving reddit.com (reddit.com)... 151.101.65.140, 151.101.1.140, 151.101.129.140, ...
Connecting to reddit.com (reddit.com)|151.101.65.140|:80... failed: Connection timed out.
Connecting to reddit.com (reddit.com)|151.101.1.140|:80... failed: Connection timed out.
Connecting to reddit.com (reddit.com)|151.101.129.140|:80...^C
root@lxc-transmission-02:~#
What is so confusing is I know the PFsense router works correctly since when connected to an AP connected to PFsense using a physical device it works as expected, it only fails when it's a proxmox VM/LXC thats connected to the PFsense VM. However its more confusing since is it appears that the network is fine since it's sent all the way through correctly out my router. Just never see anything coming back in IF it's for a proxmox LXC/VM I see messages being responded too if its for a physical device.

From Laptop over PFsense VPN Client
Code:
~ »» ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 98:fa:9b:21:d3:4d brd ff:ff:ff:ff:ff:ff
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:c6:37:d4:43:88 brd ff:ff:ff:ff:ff:ff
    inet 192.168.9.57/24 brd 192.168.9.255 scope global dynamic noprefixroute wlp0s20f3
       valid_lft 4309sec preferred_lft 4309sec
    inet6 fe80::898a:7c39:527a:4728/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
~ »» dig reddit.com

; <<>> DiG 9.14.7 <<>> reddit.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52637
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;reddit.com.            IN    A

;; ANSWER SECTION:
reddit.com.        300    IN    A    151.101.193.140
reddit.com.        300    IN    A    151.101.129.140
reddit.com.        300    IN    A    151.101.1.140
reddit.com.        300    IN    A    151.101.65.140

;; Query time: 14 msec
;; SERVER: 192.168.9.1#53(192.168.9.1)
;; WHEN: Sat Nov 16 16:09:06 GMT 2019
;; MSG SIZE  rcvd: 103

~ »» ip r g 151.101.193.140
151.101.193.140 via 192.168.9.1 dev wlp0s20f3 src 192.168.9.57 uid 1000
    cache
~ »» ping 151.101.193.140
PING 151.101.193.140 (151.101.193.140) 56(84) bytes of data.
64 bytes from 151.101.193.140: icmp_seq=1 ttl=60 time=13.0 ms
64 bytes from 151.101.193.140: icmp_seq=2 ttl=60 time=13.1 ms
^C
--- 151.101.193.140 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.980/13.019/13.059/0.039 ms
~ »» wget reddit.com
--2019-11-16 16:09:32--  http://reddit.com/
Resolving reddit.com (reddit.com)... 151.101.193.140, 151.101.129.140, 151.101.1.140, ...
Connecting to reddit.com (reddit.com)|151.101.193.140|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.reddit.com/ [following]
--2019-11-16 16:09:32--  https://www.reddit.com/
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.reddit.com (www.reddit.com)... 151.101.17.140
Connecting to www.reddit.com (www.reddit.com)|151.101.17.140|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 654021 (639K) [text/html]
Saving to: ‘index.html’

index.html                        100%[===========================================================>] 638.69K  3.66MB/s    in 0.2s   

2019-11-16 16:09:34 (3.66 MB/s) - ‘index.html’ saved [654021/654021]
 
Last edited:

Navity

New Member
Nov 16, 2019
2
0
1
26
Solved:
I was doing more googling and testing, what I find while doing more networking debugging is that the packets from container was failing the checksum.

Container
Code:
1:27:45.839647 IP (tos 0x0, ttl 64, id 53960, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.9.80.37448 > 151.101.193.140.80: Flags [S], cksum 0x2319 (incorrect -> 0xef3f), seq 3873543622, win 64240, options [mss 1460,sackOK,TS val 854653107 ecr 0,nop,wscale 7], length 0
21:27:46.869663 IP (tos 0x0, ttl 64, id 53961, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.9.80.37448 > 151.101.193.140.80: Flags [S], cksum 0x2319 (incorrect -> 0xeb39), seq 3873543622, win 64240, options [mss 1460,sackOK,TS val 854654137 ecr 0,nop,wscale 7], length 0
Laptop
Code:
21:25:15.658608 IP (tos 0x0, ttl 64, id 53456, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.9.57.56150 > 151.101.1.140.443: Flags [S], cksum 0x977e (correct), seq 3540121396, win 29200, options [mss 1460,sackOK,TS val 190523643 ecr 0,nop,wscale 7], length 0
21:25:15.669427 IP (tos 0x0, ttl 60, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    151.101.1.140.443 > 192.168.9.57.56150: Flags [S.], cksum 0x03ed (correct), seq 1828719889, ack 3540121397, win 28960, options [mss 1357,sackOK,TS val 494796102 ecr 190523643,nop,wscale 9], length 0
21:25:15.672111 IP (tos 0x0, ttl 64, id 53457, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.9.57.56150 > 151.101.1.140.443: Flags [.], cksum 0xa282 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 190523656 ecr 494796102], length 0
This of course does not affect ICMP packets which is why the container could ping out, and why he could communicate for DNS from the router.

To resolve I then found Pfsense Virtualizing which indicated to "Disable hardware checksum offload" Doing so resolved my issue.

Im not 100% sure why this never affected me when connecting to Pfsense from a physical device, could be a timing issue with the container being connected virtually only.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!