CT firewall rules with source ip

Golhum · Jan 2, 2020

Hello !
I've a fresh install of proxmox 6.
I've create a CT debian 10, with external FO ip
I want to use the firewall to limit access to the CT, and filtering on source ips.
but If I set source IP in the container, the traffic was not allowed. I need to set the source as the ip of the promox host !
A rule without source is OK.
A rule with source different than the host is bocking.

As is proxmox was a router and the only visible source ip was the proxmox.

But I want to use my external ip directly.

Maybe a config of the bridge interface ?

wolfgang · Jan 3, 2020

Hi,

are the firewall on datacenter level and Container,NIC level turned on?

Golhum · Jan 3, 2020

Hi,
Yes, all turned ON (datacenter, host, container and on network interface).

I just want to filtering ex : port 22 to my ips only.
I've reinstall 2 times, always the same pb.
I'm on a dedicated server by OVH, with proxmox preconfigured, and range of failover ips.

Golhum · Jan 3, 2020

The only source I can use in the rule in the container was the ip of the host. All other source will not work.

wolfgang · Jan 3, 2020

can you please send the fw files of the Container
they are located at.

/etc/pve/firewall/<GuestID>.fw
/etc/pve/firewall//cluster.fw

jozwior · Jan 3, 2020

I've similar issue, but with disabled all firewalls and noticed that proxmox overwrite source IP to own public IP address. Couldn't find solution yet. Will paste dumps here from tcpdump later

Golhum · Jan 3, 2020

My files ...
I've make somes tests. here, an iplist +group, but not working too.
In the container, no rule keep connection pass from ip in list or ip in source
I need to create a rule with ip of the host in source (so no ip filtration).
I've try the 'ip_filter' to yes, but not changing anything

root@xxxx:/# cat /etc/pve/firewall/cluster.fw
[OPTIONS]

enable: 1

[IPSET nico]

xx.xx.xx.140 # vpn
xx.xx.xx.50 # freebox
xx.xx.xx.221 # orange
xx.xx.xx.30 # bbox
xx.xx.xx.241 # az

[group grnico]

IN ACCEPT -source +nico -log nolog

root@xxx:/# cat /etc/pve/firewall/101.fw
[OPTIONS]

ipfilter: 1
enable: 1

[RULES]

IN ACCEPT -source xx.xx.xx.221 -log nolog
|GROUP grnico

jozwior · Jan 3, 2020

tcpdumps:
From proxmox:

17:38:01.689454 IP 89.XX.XX.41 > 51.ZZ.AAA.88: ICMP echo request, id 1, seq 1, length 64
17:38:01.689479 IP 51.ZZ.AAA.156 > 51.BB.YYY.88: ICMP echo request, id 1, seq 1, length 64
17:38:01.689515 IP 51.BB.YYY.88 > 51.ZZ.AAA.156: ICMP echo reply, id 1, seq 1, length 64
17:38:01.689522 IP 51.BB.YYY.88 > 89.XX.XX.41: ICMP echo reply, id 1, seq 1, length 64

From VM(doesn't matter if LXC or KVM):

17:43:45.549781 IP 51.ZZ.AAA.156 > 51.BB.YYY.88: ICMP echo request, id 2846, seq 1, length 12
17:43:45.549796 IP 51.BB.YYY.88 > 51.ZZ.AAA.156: ICMP echo reply, id 2846, seq 1, length 12

Where:
89.XX.XX.41 - my public IP addr
51.BB.YYY.88 - VM public IP addr
51.ZZ.AAA.156 - proxmox public IP addr

Golhum · Jan 3, 2020

Yes, I think we have the same problem.
I don't have dump the packets, but all traffic in the VM like to get from the proxmox host.
I try some configuration, and flags ... but no result

jozwior · Jan 5, 2020

@Golhum I found the cause of the problem, but unfortunately I didn't find the correct solution. Solution(temporary) which works for me is just disable apparmor:

systemctl stop apparmor
systemctl disable apparmor

I also restart the server to make sure apparmor not impact to any existing process.

Hope it will work for you too

Golhum · Jan 5, 2020

jozwior said:
@Golhum I found the cause of the problem, but unfortunately I didn't find the correct solution. Solution(temporary) which works for me is just disable apparmor:

I also restart the server to make sure apparmor not impact to any existing process.

Hope it will work for you too

Yes !!!!!
It works like a charm.
I've reboot too, and no problem.
Some tests, and sources in the firewall works perfectly. Tomorrow I'll test more.

As I'm the only user of the system and the containers, I have no really risk.
But if anyone know what the disable of apparmor can perturb proxmox or create significant security breach
Is it safe to production environment ?

Thanks jozwior

wolfgang · Jan 7, 2020

I can't reproduce this behavior here.
Is there any special setting on this host or do you install extra packages?

Golhum · Jan 7, 2020

Hi !
No, install from OVH template on dedicated server.
If you want, I can send you an access as it was not on prod now and can easily reset.

wolfgang · Jan 8, 2020

No this is not necessary.

please send me a complete list of installed packages.
and the running network config.

Code:

dpkg -l
ip a
ip r

for the ip command, the container must run.
You can send me the IP output private or mask the public IP's.
But if you mask them, keep in mind I have to know how the network works.

Golhum · Jan 8, 2020

OK ...
So, the config, with 2 container running. and apparmor disabled
The only package I've manually install on the host was curlftpfs (but I've install it after)

dpkg -l :
see attached file

ip a :

Code:

root@toto:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 08:_____ brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 08:_____ brd ff:ff:ff:ff:ff:ff
    inet 46._._._/24 brd 46._._._ scope global dynamic vmbr0
       valid_lft 69369sec preferred_lft 69369sec
    inet6 fe80::__/64 scope link
       valid_lft forever preferred_lft forever
5: veth100i0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether fe:8d:c8:ec:32:ec brd ff:ff:ff:ff:ff:ff link-netnsid 0
6: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ea:92:ba:b1:f4:29 brd ff:ff:ff:ff:ff:ff
7: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 4e:eb:bd:c4:ed:66 brd ff:ff:ff:ff:ff:ff
8: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether ea:92:ba:b1:f4:29 brd ff:ff:ff:ff:ff:ff
10: veth101i0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether fe:d2:dc:7a:0b:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 76:18:01:72:85:fa brd ff:ff:ff:ff:ff:ff
12: fwpr101p0@fwln101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether be:a9:b4:d9:74:11 brd ff:ff:ff:ff:ff:ff
13: fwln101i0@fwpr101p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether 76:18:01:72:85:fa brd ff:ff:ff:ff:ff:ff

and ip r :

Code:

default via 46._._._ dev vmbr0
46._._._/24 dev vmbr0 proto kernel scope link src 46._._._

And in the container 100 :

Code:

root@iii:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:____ brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 51._._.80/32 brd 51._._.80 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ff:_:_/64 scope link
       valid_lft forever preferred_lft forever


root@iii:~# ip r
default via 51._._.254 dev eth0
51._._.254 dev eth0 scope link

container 101 :

Code:

root@test:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:00:00:___ brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 51._._.81/32 brd 51._._.81 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ff:_:_/64 scope link
       valid_lft forever preferred_lft forever

root@test:~# ip r
default via 51._._.254 dev eth0
51._._.254 dev eth0 scope link

wolfgang · Jan 9, 2020

There are additional AppArmor packages (profiles) at this installation.
Please try to remove the following packages.
But be careful, please check which packages will automatically be removed too.

Code:

apparmor-easyprof
apparmor-profiles 
apparmor-profiles-extra
apparmor-utils

reboot the system

Golhum · Jan 9, 2020

Hi !
I've remove package you suggest, enable apparmor and reboot ! And I cannot access to the server. Try to disable firewall, apparmor ... no way to retrive the service working.

(I've no KVM on the hardware, so I've try with netboot rescue system to change settings)

So,
I reinstall the server with a fresh system.
Remove the 4 packages, and reboot.
Create a container.

And now, it works perfectly.
apparmor is active on the host.
firewall filtering was OK for me (ip source works in the container).

Are this 4 packages important for proxmox ?

Maybe I can now use the server in prod now. Need to try with real container and services before.

Big Thanks Wolfgang !

wolfgang · Jan 9, 2020

These 4 packages are not part of Proxmox VE.
Why OVH decide to install them I don't know.
This are extra AppArmor profiles that are not compatible with or set.

Golhum · Jan 9, 2020

OK !
Very thanks for your help.

I'll try to send the information to OVH ...

I need some times to check all before mark this thread solved.

wolfgang · Jan 9, 2020

I will check if we set these packages as conflicted with our proxmox-ve firewall packages.

CT firewall rules with source ip

New Member

Attachments

Proxmox Retired Staff

New Member

New Member

Proxmox Retired Staff

New Member

New Member

New Member

New Member

New Member

New Member

Proxmox Retired Staff

New Member

Proxmox Retired Staff

New Member

Attachments

Proxmox Retired Staff

New Member

Proxmox Retired Staff

New Member

Proxmox Retired Staff