Enabling Datacenter firewall blocks DNS/Ping outbound traffic

R

reetp

Renowned Member
Aug 19, 2013
67
11
73
I have a Proxmox server at online.net

It runs a few VMs with linux servers.

I thought I would try and run some firewalling - specifically to block/allow some IPs to all VMs rather than have to do it individually.

Datacenter has 3 security groups

remotemanagement - allows access for ping, ssh https etc
vm-generalports - allows access t all VMs - http/https/ssh/mail etc
clusterblocklist - blocks some IPs via IPSet blacklist

Intention:

The actual proxmox server itself uses remotemanagement + clusterblocklist

Individual VMs use vm-generalports + clusterblocklist

Default Output policy is set to ALLOW at Datacenter and VM level - it is not an option at Proxmox server level?

No matter what I try and do, it seems when the firewall is up I can access VM services eg https, ping, mail etc

However, the VMs themselves cannot ping or dig out?

Been driving me crazy for days now!

One point I noted is despite setting output policy ALLOW it only shows "policy_in: DROP" in cluster.fw below? No "policy_out: ALLOW" ?

Any ideas appreciated. Feels ever so much like a bug but wondering if devs are more focussed on nftables than this?

(Would be nice to have geoip blocking built in to Prox - rather than on each VM. Save a lot of overhead.)

Proxmox v8.2.2 patched with licence.
Standalone mode

cat /etc/network/interfaces


Code: 
source /etc/network/interfaces.d/*


auto lo
iface lo inet loopback


iface eno1 inet manual
#External IP


iface eno2 inet manual
#Internal IP


auto vmbr0
iface vmbr0 inet static
    address x.x.x.x/24
    gateway x.x.x.x
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0


auto vmbr1
iface vmbr1 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#For dummy SME ports


auto vmbr3
iface vmbr3 inet static
    address 10.90.44.13/26
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
#Internal IP


cat cluster.fw

Code: 
[OPTIONS]


enable: 0
policy_in: DROP


[IPSET clusterblacklist] # ClusterBlacklist


103.145.253.0/24
183.129.178.0/24


[RULES]


GROUP clusterblocklist
GROUP vm-generalports
GROUP remotemanagement


[group clusterblocklist] # uses DC/ClusterBlacklist


IN DROP -source +dc/clusterblacklist -log info


[group remotemanagement] # Remote Management


IN ACCEPT -p tcp -dport 3128 -log nolog # Port 3128
IN ACCEPT -p tcp -dport 8006 -log nolog # Prox HTTPS
IN ACCEPT -p tcp -dport 2222 -log nolog # SSH
IN Ping(ACCEPT) -source 1.2.3.4 -log nolog # Remote Ping 
IN Ping(ACCEPT) -source 5.6.7.8 -log nolog # Remote Ping


[group vm-generalports] # http https imap smtp ssh etc


OUT DNS(ACCEPT) -log info # Desperate attmpts
OUT Ping(ACCEPT) -log info # Desperate attmpts
IN ACCEPT -p tcp -dport 2229 -log nolog # SSH 2229
IN Ping(ACCEPT) -log nolog
IN OpenVPN(ACCEPT) -log nolog
IN Mail(ACCEPT) -log nolog
IN HKP(ACCEPT) -log nolog
IN IPsecah(ACCEPT) -log nolog
IN IPsecnat(ACCEPT) -log nolog
IN IPsec(ACCEPT) -log nolog
IN IMAPS(ACCEPT) -log nolog
IN ACCEPT -p tcp -dport 993 -log nolog # IMAPS Manual
IN SMTPS(ACCEPT) -log nolog
IN SMTP(ACCEPT) -log nolog
IN HTTPS(ACCEPT) -log nolog
IN DNS(ACCEPT) -log nolog
IN HTTP(ACCEPT) -log nolog
|IN DROP -source 0.0.0.0/0 -log nolog # Disabled for testing


cat 101.fw

Code: 
[OPTIONS]


enable: 1
log_level_in: nolog
ipfilter: 1
log_level_out: info


[RULES]


GROUP vm-generalports -i net0
 
reetp said:
However, the VMs themselves cannot ping or dig out?
Click to expand...
Where are you trying to ping / dig to? The host or somewhere else? Does this work when the firewall is disabled?

reetp said:
One point I noted is despite setting output policy ALLOW it only shows "policy_in: DROP" in cluster.fw below? No "policy_out: ALLOW" ?
Click to expand...
If it is not explicitly set, the firewall uses the default values for all options (which is in this case ACCEPT)

Can you post the output of the following commands?

Code: 
systemctl status {pve,proxmox}-firewall
iptables-save
qm config 101
 
Hi and thanks for replying.

Where are you trying to ping / dig to? The host or somewhere else? Does this work when the firewall is disabled?
Click to expand...

From any VM out to anywhere on the interwebs. It works perfectly with the firewall disabled at the 'Cluster/Datacenter' level eg this is with the firewall disabled. If I enable it this stops.

[root@vm-main ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=1.12 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=1.18 ms


If it is not explicitly set, the firewall uses the default values for all options (which is in this case ACCEPT)
Click to expand...

OK - just a bit disconcerting....! For the sake of a line it would be nicer to see it there as confirmation the system knows what it is doing ;-)

With the firewall off at Datacenter level we have the following - I can turn it on and get the output later.

Code: 
root@vps:~# systemctl status {pve,proxmox}-firewall

For a moment I thought it was this that I just noticed - interesting that the firewall service is running and not actually stopped.
Jul 07 15:24:33 sd-90195 pve-firewall[3938670]: /etc/pve/firewall/cluster.fw (line 5): can't parse option 'poluicy_out: ALLOW'
Jul 07 15:24:33 sd-90195 pve-firewall[3938670]: /etc/pve/firewall/cluster.fw (line 5): can't parse option 'poluicy_out: ALLOW'
Click to expand...

So I restarted the proxmox-firewall service, enabled the Datacenter firewall and it again failed. Got this output:

● pve-firewall.service - Proxmox VE firewall
Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; preset: enabled)
Active: active (running) since Mon 2024-07-08 17:04:26 CEST; 1min 15s ago
Process: 169139 ExecStartPre=/usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy (code=exited, status=0/SUCCESS)
Process: 169142 ExecStartPre=/usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (code=exited, status=0/SUCCESS)
Process: 169143 ExecStartPre=/usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (code=exited, status=0/SUCCESS)
Process: 169144 ExecStart=/usr/sbin/pve-firewall start (code=exited, status=0/SUCCESS)
Main PID: 169145 (pve-firewall)
Tasks: 1 (limit: 38239)
Memory: 95.2M
CPU: 1.098s
CGroup: /system.slice/pve-firewall.service
└─169145 pve-firewall

Jul 08 17:04:25 sd-90195 systemd[1]: Starting pve-firewall.service - Proxmox VE firewall...
Jul 08 17:04:26 sd-90195 pve-firewall[169145]: starting server
Jul 08 17:04:26 sd-90195 systemd[1]: Started pve-firewall.service - Proxmox VE firewall.

● proxmox-firewall.service - Proxmox nftables firewall
Loaded: loaded (/lib/systemd/system/proxmox-firewall.service; enabled; preset: enabled)
Active: active (running) since Sat 2024-06-29 15:36:02 CEST; 1 week 2 days ago
Main PID: 1268 (proxmox-firewal)
Tasks: 1 (limit: 38239)
Memory: 2.5M
CPU: 4.824s
CGroup: /system.slice/proxmox-firewall.service
└─1268 /usr/libexec/proxmox/proxmox-firewall

Jun 29 15:36:02 sd-90195 systemd[1]: Started proxmox-firewall.service - Proxmox nftables firewall.
Click to expand...



Code: 
root@vps:~# iptables-save

Code: 
# Generated by iptables-save v1.8.9 on Mon Jul  8 16:57:02 2024
*raw
:PREROUTING ACCEPT [137561624:130066502028]
:OUTPUT ACCEPT [999213:649381707]
COMMIT
# Completed on Mon Jul  8 16:57:02 2024
# Generated by iptables-save v1.8.9 on Mon Jul  8 16:57:02 2024
*filter
:INPUT ACCEPT [146:51662]
:FORWARD ACCEPT [4468:1168678]
:OUTPUT ACCEPT [101:44859]
:f2b-sshd - [0:0]
-A INPUT -p tcp -m multiport --dports 22,2222 -j f2b-sshd
-A INPUT -s 103.145.253.165/32 -j DROP
-A f2b-sshd -s 120.26.90.97/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.83/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.84/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 103.232.122.33/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.37/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.34/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.101/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.115/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
COMMIT
# Completed on Mon Jul  8 16:57:02 2024

Code: 
root@vps:~# qm config 101

Code: 
agent: 1,fstrim_cloned_disks=1
balloon: 0
boot: order=scsi0
cores: 4
cpu: host
memory: 8192
name: vm-main
net0: virtio=52:54:00:00:15:74,bridge=vmbr0,firewall=1
net1: virtio=A2:07:AF:69:76:18,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsi0: pve:vm-101-disk-0,discard=on,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=66156652-373e-4edf-ac20-444a8428862a
sockets: 1
vmgenid: c5773ea2-1373-4c60-b52e-b9917b2802ca
 
OK, long post with info.
Ping tests to godaddy.co.uk

101 Fail
102 Fail
103 OK
120 OK
160 OK
200 Fail

On VMs with two network devices they are linux servers running their own DNS in a server/gateway mode with the net1 device set as a dummy 'internal' interface.
They could be connected to the providers internal interface for internal comms, but they aren't.


Code: 
qm config 101
agent: 1,fstrim_cloned_disks=1
balloon: 0
boot: order=scsi0
cores: 4
cpu: host
memory: 8192
name: mail-crm
net0: virtio=52:54:00:00:15:74,bridge=vmbr0,firewall=1
net1: virtio=A2:07:AF:69:76:18,bridge=vmbr1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsi0: pve:vm-101-disk-0,discard=on,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=66156652-373e-4edf-ac20-444a8428862a
sockets: 1
vmgenid: c5773ea2-1373-4c60-b52e-b9917b2802ca




qm config 102
agent: 1,fstrim_cloned_disks=1
balloon: 0
boot: order=virtio1
cores: 2
memory: 4096
name: voip
net0: virtio=52:54:00:00:15:82,bridge=vmbr0,firewall=1
net1: virtio=32:FB:06:61:A9:6D,bridge=vmbr1
numa: 0
onboot: 1
ostype: l26
protection: 1
sockets: 1
startup: order=2
virtio1: pve:vm-102-disk-1,discard=on,size=64G




qm config 103
agent: 1,fstrim_cloned_disks=1
boot: order=scsi1
cores: 2
memory: 4096
name: nextcloud
net0: virtio=00:50:56:01:d3:7d,bridge=vmbr0,firewall=1
net1: virtio=F6:0C:6F:A1:17:3B,bridge=vmbr1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsi1: pve:vm-103-disk-0,size=300G
scsihw: virtio-scsi-pci
smbios1: uuid=84007432-d701-423b-8322-151c22079cd6
sockets: 1
vmgenid: 8bcc12e5-9791-48b7-9ef2-1c59967e1ec9


qm config 120
agent: 1,fstrim_cloned_disks=1
boot: order=scsi0;net0
cores: 2
memory: 4096
name: Gitea
net0: virtio=52:54:00:00:15:81,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: pve:vm-120-disk-0,discard=on,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=c5e3fa10-e89c-43b9-9cb2-b2bee93b3468
sockets: 1
vga: qxl
vmgenid: d8400402-f0e6-44be-97f6-d3c24a511158


qm config 160
agent: 1,fstrim_cloned_disks=1
balloon: 0
boot: order=scsi0;net0;net1
cores: 2
cpu: host
memory: 6144
name: Rocket
net0: virtio=52:54:00:00:1A:DD,bridge=vmbr0,firewall=1
net1: virtio=2A:6E:BD:4E:01:1B,bridge=vmbr1
numa: 0
ostype: l26
protection: 1
scsi0: pve:vm-160-disk-0,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=2641d718-0082-4451-96d1-14bedb63711c
sockets: 1
vga: virtio
vmgenid: ed0d508b-6ea1-4880-9752-1161f5de187e


qm config 200
agent: 1,fstrim_cloned_disks=1
balloon: 0
bootdisk: scsi0
cores: 2
memory: 4096
name: BuildBox
net0: virtio=52:54:00:00:1A:DC,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: pve:vm-200-disk-0,discard=on,size=150G
scsihw: virtio-scsi-pci
smbios1: uuid=48ed1c9a-1962-4d55-b934-4600ddbb545a
sockets: 1
vmgenid: 57a6bfd6-b92a-496e-b962-372fe5903081



systemctl status {pve,proxmox}-firewall
Code: 
● pve-firewall.service - Proxmox VE firewall
     Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-07-08 17:04:26 CEST; 1 week 3 days ago
    Process: 169139 ExecStartPre=/usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy (code=exited, status=0/SUCCESS)
    Process: 169142 ExecStartPre=/usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (code=exited, status=0/SUCCESS)
    Process: 169143 ExecStartPre=/usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (code=exited, status=0/SUCCESS)
    Process: 169144 ExecStart=/usr/sbin/pve-firewall start (code=exited, status=0/SUCCESS)
   Main PID: 169145 (pve-firewall)
      Tasks: 1 (limit: 38239)
     Memory: 96.0M
        CPU: 1h 14min 59.386s
     CGroup: /system.slice/pve-firewall.service
             └─169145 pve-firewall


Jul 08 17:04:25 sd-90195 systemd[1]: Starting pve-firewall.service - Proxmox VE firewall...
Jul 08 17:04:26 sd-90195 pve-firewall[169145]: starting server
Jul 08 17:04:26 sd-90195 systemd[1]: Started pve-firewall.service - Proxmox VE firewall.


● proxmox-firewall.service - Proxmox nftables firewall
     Loaded: loaded (/lib/systemd/system/proxmox-firewall.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-06-29 15:36:02 CEST; 2 weeks 5 days ago
   Main PID: 1268 (proxmox-firewal)
      Tasks: 1 (limit: 38239)
     Memory: 2.5M
        CPU: 10.177s
     CGroup: /system.slice/proxmox-firewall.service
             └─1268 /usr/libexec/proxmox/proxmox-firewall


Jun 29 15:36:02 sd-90195 systemd[1]: Started proxmox-firewall.service - Proxmox nftables firewall.


IPtables to follow as I hit the message limit
 
iptables-save

Code: 
# Generated by iptables-save v1.8.9 on Thu Jul 18 19:07:45 2024
*raw
:PREROUTING ACCEPT [330917937:301970615511]
:OUTPUT ACCEPT [3912156:2099052544]
COMMIT
# Completed on Thu Jul 18 19:07:45 2024
# Generated by iptables-save v1.8.9 on Thu Jul 18 19:07:45 2024
*filter
:INPUT ACCEPT [13:784]
:FORWARD ACCEPT [644:33190]
:OUTPUT ACCEPT [2:152]
:GROUP-clusterblocklist-IN - [0:0]
:GROUP-clusterblocklist-OUT - [0:0]
:GROUP-remotemanagement-IN - [0:0]
:GROUP-remotemanagement-OUT - [0:0]
:GROUP-vm-generalports-IN - [0:0]
:GROUP-vm-generalports-OUT - [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
:f2b-sshd - [0:0]
:tap101i0-IN - [0:0]
:tap101i0-OUT - [0:0]
:tap102i0-IN - [0:0]
:tap102i0-OUT - [0:0]
:tap200i0-IN - [0:0]
:tap200i0-OUT - [0:0]
-A INPUT -p tcp -m multiport --dports 22,2222 -j f2b-sshd
-A INPUT -s 103.145.253.165/32 -j DROP
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A GROUP-clusterblocklist-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-clusterblocklist-IN -m set --match-set PVEFW-3700B5D8 src -m limit --limit 1/sec -j NFLOG --nflog-prefix ":0:6:GROUP-clusterblocklist-IN: "
-A GROUP-clusterblocklist-IN -m set --match-set PVEFW-3700B5D8 src -j DROP
-A GROUP-clusterblocklist-IN -m comment --comment "PVESIG:OgKsFz9l97esJKqlVSGfscndhPg"
-A GROUP-clusterblocklist-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-clusterblocklist-OUT -m comment --comment "PVESIG:KvVvAcbu7fxAOcEKcZV8MrigwCc"
-A GROUP-remotemanagement-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-remotemanagement-IN -p tcp -m tcp --dport 3128 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-remotemanagement-IN -p tcp -m tcp --dport 8006 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-remotemanagement-IN -p tcp -m tcp --dport 2222 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-remotemanagement-IN -s 80.24.1.207/32 -p icmp -m icmp --icmp-type 8 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-remotemanagement-IN -s 81.137.216.70/32 -p icmp -m icmp --icmp-type 8 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-remotemanagement-IN -m comment --comment "PVESIG:cgrRXOBmAIjJ9L+Ujk2v8Q7nhjE"
-A GROUP-remotemanagement-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-remotemanagement-OUT -m comment --comment "PVESIG:PEnB39dzhaYQmiqyZ4MuobUpSbc"
-A GROUP-vm-generalports-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 2222 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p icmp -m icmp --icmp-type 8 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p udp -m udp --dport 1194 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 25 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 465 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 587 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 11371 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p udp -m udp --sport 500 --dport 500 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p ah -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p udp -m udp --dport 500 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p udp -m udp --dport 4500 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p esp -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p udp -m udp --sport 500 --dport 500 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p esp -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 993 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 993 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 465 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 25 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p udp -m udp --dport 53 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 53 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -p tcp -m tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-IN -m comment --comment "PVESIG:9UWLtGX58RUO8GkFyOHMbmPIwck"
-A GROUP-vm-generalports-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-vm-generalports-OUT -p udp -m udp --dport 53 -m limit --limit 1/sec -j NFLOG --nflog-prefix ":0:6:GROUP-vm-generalports-OUT: "
-A GROUP-vm-generalports-OUT -p udp -m udp --dport 53 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-OUT -p tcp -m tcp --dport 53 -m limit --limit 1/sec -j NFLOG --nflog-prefix ":0:6:GROUP-vm-generalports-OUT: "
-A GROUP-vm-generalports-OUT -p tcp -m tcp --dport 53 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-OUT -p icmp -m icmp --icmp-type 8 -m limit --limit 1/sec -j NFLOG --nflog-prefix ":0:6:GROUP-vm-generalports-OUT: "
-A GROUP-vm-generalports-OUT -p icmp -m icmp --icmp-type 8 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-OUT -m limit --limit 1/sec -j NFLOG --nflog-prefix ":0:6:GROUP-vm-generalports-OUT: "
-A GROUP-vm-generalports-OUT -g PVEFW-SET-ACCEPT-MARK
-A GROUP-vm-generalports-OUT -m comment --comment "PVESIG:4PAcW7m2wtCG/fOKesv8UFij/yc"
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m physdev --physdev-out tap101i0 --physdev-is-bridged -j tap101i0-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap102i0 --physdev-is-bridged -j tap102i0-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap200i0 --physdev-is-bridged -j tap200i0-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:ovMMyMwniQBaQ59zh20REGXH94o"
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap101i0 --physdev-is-bridged -j tap101i0-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap102i0 --physdev-is-bridged -j tap102i0-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap200i0 --physdev-is-bridged -j tap200i0-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:gEL0+u13Q1xGxS6dfoYMGjNrOWE"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -i eno1 -j GROUP-remotemanagement-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i vmbr0 -j GROUP-remotemanagement-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i vmbr1 -j GROUP-remotemanagement-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -j GROUP-clusterblocklist-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -j GROUP-vm-generalports-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -j GROUP-remotemanagement-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:tntmvSxKKf4NDT9zNl759JuApTs"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -o eno1 -j GROUP-remotemanagement-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -o vmbr0 -j GROUP-remotemanagement-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -o vmbr1 -j GROUP-remotemanagement-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -j GROUP-clusterblocklist-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -j GROUP-vm-generalports-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -j GROUP-remotemanagement-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -d 195.154.176.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 195.154.176.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 195.154.176.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 195.154.176.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:FTq/62+1+fG2Dsm9Vlim9YlsS7g"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A f2b-sshd -s 93.123.39.184/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.100/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 124.232.197.15/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.37/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.101/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.152/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.83/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.36/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.118.39.115/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 182.66.79.118/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
-A tap101i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap101i0-IN -j GROUP-vm-generalports-IN
-A tap101i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
-A tap101i0-IN -j PVEFW-Drop
-A tap101i0-IN -j DROP
-A tap101i0-IN -m comment --comment "PVESIG:jpL+IXwVdBoeaL6k/WQonP1GdxQ"
-A tap101i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap101i0-OUT -m mac ! --mac-source 52:54:00:00:15:74 -j DROP
-A tap101i0-OUT -m set ! --match-set PVEFW-101-ipfilter-net0-v4 src -j DROP
-A tap101i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap101i0-OUT -j GROUP-vm-generalports-OUT
-A tap101i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A tap101i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap101i0-OUT -m comment --comment "PVESIG:bxecZvgfeNVLrIPhi0AtqNlluXo"
-A tap102i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap102i0-IN -p udp -m udp --dport 1194 -j ACCEPT
-A tap102i0-IN -j GROUP-vm-generalports-IN
-A tap102i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
-A tap102i0-IN -j PVEFW-Drop
-A tap102i0-IN -j DROP
-A tap102i0-IN -m comment --comment "PVESIG:L1xR2s2nrWMLSGDErqpyoWmHbcc"
-A tap102i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap102i0-OUT -m mac ! --mac-source 52:54:00:00:15:82 -j DROP
-A tap102i0-OUT -m set ! --match-set PVEFW-102-ipfilter-net0-v4 src -j DROP
-A tap102i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap102i0-OUT -j GROUP-vm-generalports-OUT
-A tap102i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A tap102i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap102i0-OUT -m comment --comment "PVESIG:o5dSY2n0Sp8AqCdBrLIsFqvxAkQ"
-A tap200i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap200i0-IN -p tcp -m tcp --dport 2223 -m limit --limit 1/sec -j NFLOG --nflog-prefix ":200:6:tap200i0-IN: ACCEPT: "
-A tap200i0-IN -p tcp -m tcp --dport 2223 -j ACCEPT
-A tap200i0-IN -j GROUP-vm-generalports-IN
-A tap200i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
-A tap200i0-IN -j PVEFW-Drop
-A tap200i0-IN -j DROP
-A tap200i0-IN -m comment --comment "PVESIG:hJ9ftNjYa+6rFPCab9S2g9rgGMA"
-A tap200i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap200i0-OUT -m mac ! --mac-source 52:54:00:00:1a:dc -j DROP
-A tap200i0-OUT -m set ! --match-set PVEFW-200-ipfilter-net0-v4 src -j DROP
-A tap200i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap200i0-OUT -j GROUP-vm-generalports-OUT
-A tap200i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A tap200i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap200i0-OUT -m comment --comment "PVESIG:Al1BTaSASgP6BA15wnOrg5fRbos"
COMMIT
# Completed on Thu Jul 18 19:07:45 2024
 
I have wondered if I have done something stupid here - won't surprise me - but can't see anything glaringly obvious.

Any pointers appreciated.
 
In your firewall rules you are using --icmp-type 8 for incoming traffic as well as outgoing. ICMP code 8 is for echo requests. The ICMP code for echo reply is 0, so I think you are blocking the echo replies because of this firewall rule. I'll check the ping macro, but it seems like this is intended

edit: nvm, conntrack should handle that of course - i'll look further
 
Last edited:
You have the IP Filter activated, but I don't see a IPFilter set configured in the VM firewall config, does disabling the IP Filter work?
 
OK - hunting through for clues.

I realised that these were affected if the firewall was enabled for the VM as follows:

101 Fail - Firewall enable: 1 ipfilter: 1
102 Fail - Firewall enable: 1 ipfilter: 1
103 OK - Firewall enable: 0 ipfilter: 1
120 OK - Firewall enable: 0 ipfilter: 1
160 OK - Firewall enable: 0 ipfilter: 1
200 Fail - Firewall enable: 1 ipfilter: 1

At Datacenter level I have a IPSet clusterblacklist that should blacklist 2 IPs cluster wide to all VMs? But that is only for incoming?
 
ipfilter setting means that the firewall will look for an IPSet named ipfilter-net<id> in the VM firewall configuration. Outgoing connections on the interface <id> will then only be allowed to have an IP from this IPset as the source address. If it doesn't exist it will get autogenerated [1]:

This is equivalent to adding an empty ipfilter-net<id> ipset for every interface. Such ipsets implicitly contain sane default restrictions such as restricting IPv6 link local addresses to the one derived from the interface’s MAC address. For containers the configured IP addresses will be implicitly added.
Click to expand...

Judging from your 101.fw config you do not have such an ipfilter IPset configured explicitly. So outgoing connection will get blocked. You can check if that is the case by disabling the ipfilter setting and check if outgoing connections work then.


[1] https://pve.proxmox.com/wiki/Firewall#pve_firewall_vm_container_configuration
 
OK - that seems to have cured it. many thanks!!

So my understanding of IP Sets is less than clear - the documentation isn't as clear as it seems :-(

shanreich said:
Outgoing connections on the interface <id> will then only be allowed to have an IP from this IPset as the source address. If it doesn't exist it will get autogenerated
Click to expand...

I can now see for a VM we can:

Create an IP Set "Test"
Add rule
Set Source or Destination > IP Set "Test"
Direction, Action etc etc.


But I thought that it meant:

Datacenter
App IPSet 'clusterblocklist'
Add to DAtacenter Security group

Then

VM
Insert Security Group 'clusterblocklist'
IPSet -> enable (because....it's an IPSet!)


I presume you ONLY use IPSet at VM level if you specifically have an IPSet for that VM?

It is more confusing because at the VM level you can either:

Add, Source/Destination IPSet 'clusterblocklist'

Or

Insert Security Group 'clusterblocklist'

Possibly a user should not be able to set IP FIlter unless there is something in IPSet?

I was also confused as the default OUT policy is ALLOW and I didn't think I had set any blocks on that!

Once again, many thanks.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back