PVE Firewall Datacenter Security Groups and Impossible to understand Logs !

S

silverstone

Well-Known Member
Apr 28, 2018
85
5
48
35
I really like the Proxmox VE Product.

However I am getting at my wits end with regards to the Firewall Logging Capability. It might as well be Chinese :( .

Some Traffic to/from the OPNSense VM works (e.g. from my Home pinging Proxmox Host AND OPNSense VM works correctly), same from Proxmox VE Host to e.g. Google Servers. BUT from OPNSense VM I cannot do much: ping fails, traceroute fails, tcp traceroute fails, udp traceroute fails, icmp traceroute fails, ...

Unfortunately right now I'm at my second Day of diagnozing why some traffic works and some does not. In particular, the OPNSense VM 100 seem unable to send ANYTHING out (but is able to get a WAN DHCP IP and I can connect via OpenVPN without issues).

If I create a Rule in Datacenter -> Firewall -> Security Groups, this is what I get in the logs:
Code: 
0 6 GROUP-allow_traceroute-IN 20/May/2024:21:24:36 +0200 IN=vmbr0 PHYSIN=eth0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=54.241.234.32 DST=XX.XX.XX.proxmox LEN=36 TOS=0x00 PREC=0x00 TTL=242 ID=39734 DF PROTO=ICMP TYPE=8 CODE=0 ID=6 SEQ=6698
0 7 GROUP-proxmox_block-IN 20/May/2024:21:24:37 +0200 IN=vmbr0 PHYSIN=eth0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=162.142.125.232 DST=XX.XX.XX.proxmox LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=35589 PROTO=TCP SPT=40695 DPT=8272 SEQ=1889855200 ACK=0 WINDOW=42340 SYN
100 6 tap100i0-IN 20/May/2024:21:24:37 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.104 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=40013 DF PROTO=TCP SPT=51775 DPT=49843 SEQ=3294490980 ACK=0 WINDOW=64240 SYN
0 6 GROUP-allow_traceroute-IN 20/May/2024:21:24:38 +0200 IN=vmbr0 PHYSIN=eth0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=54.183.93.203 DST=XX.XX.XX.proxmox LEN=36 TOS=0x00 PREC=0x00 TTL=242 ID=19833 DF PROTO=ICMP TYPE=8 CODE=0 ID=25 SEQ=19130
100 6 tap100i0-IN 20/May/2024:21:24:40 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.249 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=45807 DF PROTO=TCP SPT=59475 DPT=1217 SEQ=4040425590 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:24:40 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=45.142.182.70 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=57 ID=57110 DF PROTO=TCP SPT=55188 DPT=81 SEQ=1424128710 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:24:41 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=103.149.12.28 DST=XX.XX.XX.opnsense LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=58379 PROTO=TCP SPT=49460 DPT=3925 SEQ=1564744633 ACK=0 WINDOW=1024 SYN
0 6 GROUP-allow_ntp_client-OUT 20/May/2024:21:24:42 +0200 OUT=vmbr0 SRC=XX.XX.XX.proxmox DST=213.239.239.164 LEN=76 TOS=0x18 PREC=0xA0 TTL=64 ID=39456 DF PROTO=UDP SPT=123 DPT=123 LEN=56
0 7 GROUP-anti_lockout-IN 20/May/2024:21:24:41 +0200 IN=vmbr0 PHYSIN=eth0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=xx.xx.xx.xx DST=XX.XX.XX.proxmox LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=33850 DF PROTO=TCP SPT=46641 DPT=8006 SEQ=760661967 ACK=0 WINDOW=32120 SYN
0 7 GROUP-anti_lockout-IN 20/May/2024:21:24:43 +0200 IN=vmbr0 PHYSIN=eth0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=xx.xx.xx.xx DST=XX.XX.XX.proxmox LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=46515 DF PROTO=TCP SPT=60015 DPT=8006 SEQ=1406464688 ACK=0 WINDOW=32120 SYN
100 6 tap100i0-IN 20/May/2024:21:24:44 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=104.156.155.2 DST=XX.XX.XX.opnsense LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=6329 PROTO=TCP SPT=42180 DPT=8830 SEQ=2633026080 ACK=0 WINDOW=1024 SYN
0 7 GROUP-proxmox_block-IN 20/May/2024:21:24:44 +0200 IN=vmbr0 PHYSIN=eth0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=79.110.62.77 DST=XX.XX.XX.proxmox LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=25948 PROTO=TCP SPT=43617 DPT=28470 SEQ=3630824071 ACK=0 WINDOW=1024 SYN
0 7 GROUP-anti_lockout-IN 20/May/2024:21:24:46 +0200 IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=xx.xx.xx.xx DST=XX.XX.XX.opnsense LEN=84 TOS=0x00 PREC=0x00 TTL=55 ID=51398 DF PROTO=ICMP TYPE=8 CODE=0 ID=54872 SEQ=1
100 6 tap100i0-IN 20/May/2024:21:24:50 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=3.144.187.139 DST=XX.XX.XX.opnsense LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54321 PROTO=TCP SPT=45555 DPT=488 SEQ=1915114513 ACK=0 WINDOW=65535 SYN

Whereas if I create a Rule in VM -> Firewall, this is what I get in the logs:
Code: 
00 6 tap100i0-IN 20/May/2024:21:25:03 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=112.93.119.35 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=36125 DF PROTO=TCP SPT=52804 DPT=6379 SEQ=3316736422 ACK=0 WINDOW=29200 SYN
100 6 tap100i0-IN 20/May/2024:21:25:13 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.76 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=17523 DF PROTO=TCP SPT=43007 DPT=49843 SEQ=789400401 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:16 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.220 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=54018 DF PROTO=TCP SPT=45769 DPT=1217 SEQ=1209725229 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:20 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=45.142.182.70 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=57 ID=48106 DF PROTO=TCP SPT=56098 DPT=81 SEQ=3845166040 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:21 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=49.229.100.123 DST=XX.XX.XX.opnsense LEN=52 TOS=0x00 PREC=0x00 TTL=111 ID=19553 DF PROTO=TCP SPT=60110 DPT=3389 SEQ=2243072329 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:21 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=84.54.51.115 DST=XX.XX.XX.opnsense LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=7255 PROTO=TCP SPT=47873 DPT=24086 SEQ=1103606438 ACK=0 WINDOW=1024 SYN
100 6 tap100i0-IN 20/May/2024:21:25:25 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=206.168.34.168 DST=XX.XX.XX.opnsense LEN=420 TOS=0x00 PREC=0x00 TTL=40 ID=46968 PROTO=UDP SPT=59810 DPT=500 LEN=400
100 6 tap100i0-IN 20/May/2024:21:25:25 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=160.155.192.235 DST=XX.XX.XX.opnsense LEN=52 TOS=0x00 PREC=0x00 TTL=114 ID=11112 DF PROTO=TCP SPT=59882 DPT=5985 SEQ=3221345359 ACK=0 WINDOW=8192 SYN
100 6 tap100i0-IN 20/May/2024:21:25:32 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.92 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=11072 DF PROTO=TCP SPT=50249 DPT=49843 SEQ=1376079353 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:34 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.251 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19473 DF PROTO=TCP SPT=56457 DPT=1217 SEQ=1987375271 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:38 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.92 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=11826 DF PROTO=TCP SPT=56359 DPT=49843 SEQ=4191047535 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:40 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=45.142.182.70 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=57 ID=41371 DF PROTO=TCP SPT=56672 DPT=81 SEQ=910509197 ACK=0 WINDOW=64240 SYN
100 6 tap100i0-IN 20/May/2024:21:25:41 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=79.110.62.77 DST=XX.XX.XX.opnsense LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=14486 PROTO=TCP SPT=43617 DPT=33677 SEQ=3916319740 ACK=0 WINDOW=1024 SYN
100 6 tap100i0-IN 20/May/2024:21:25:41 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=167.94.138.139 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=58 ID=19883 PROTO=TCP SPT=38200 DPT=591 SEQ=3203854416 ACK=0 WINDOW=42340 SYN
100 6 tap100i0-IN 20/May/2024:21:25:52 +0200 ACCEPT: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=tap100i0 MAC=XX:XX:XX:XX:XX:XX:40:71:83:a5:f3:ea:08:00 SRC=87.236.176.242 DST=XX.XX.XX.opnsense LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=2043 DF PROTO=TCP SPT=53507 DPT=1217 SEQ=1827974883 ACK=0 WINDOW=64240 SYN

Basically for the "Group" there is no POLICY in the logs (ACCEPT, DROP, REJECT) :(.
The lines beginning with "100" is ONLY after I created a DEDICATED / DIRECT Rule in VM 100 -> Firewall (IN ALLOW ALL & OUT ALLOW ALL).

It's just extremely hard to diagnose ... In OPNSense Firewall everything that should be green is Green (default "Allow everything out of this firewall").

But obviously something is not right ...

I also posted this on the OPNSense Forum (results of traceroute etc): https://forum.opnsense.org/index.php?topic=40585.0

But I'd say that it's PVE Firewall that's doing something wrong ... I just cannot see what.

And the Security Group Policy is something very Practical, I'd like to do EVERYTHING with it in order to avoid manually defining each rule.
But if I do that, the logs of EVERYTHING just go into the Node, while the VM Logs stay empty. Plugs the aforementioned issue that I cannot see what the Policy was (ACCEPT, DROP, etc).

In particular: applying a Group Security Policy to a VM made the rule only visible in the Node Firewall Logs (NOT the VM) AND it was TAGGED as "0" (HOST) !

Furthermore, just to make sure I understand correctly (I'm using Bridged Setup): I ONLY need to define the Rules that apply to the OPNSense VM in the OPNSense VM -> Firewall TAB, or do I **ALSO** need to define the OPNSense Rules in the Node -> Firewall Tab ?

Thank you for your help :). I'd really like to go forward with this. It's taking a toll on me losing so much time for this stupid issue.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom