[SOLVED] Proxmox - IP Hijacking module

[Egner]

Hi,

I have searched here on the proxmox forum for a couple of hours without finding what i looking for.

For example so have both solusvm and Opennebula created a feature called IP Hijacking, it prevent users to be able to add a different IP-adress on their virtual machine that's not belongs to them.


For example KVM vm number one:

eth0 have ip : 185.166.133.2/24 and it belongs to the user of the vm. But then the user comes up with the brilliant idea and make a alias upon the current eth0 network.
He/or she added an extra alias like this on their virtual machine : "ifconfig eth0:0 185.166.133.3 netmask 255.255.255.0 up"

Now the user been able to add a different public IP from the current /24, and this is not how it should work. If the user not own the current IP it should not be possible to add it as an extra alias and have it working out of the public internet.

It looks like OpenNebula have created an extra feature added to OpenVswitch: http://docs.opennebula.io/4.10/administration/networking/openvswitch.html

IP hijacking

These rules prevent any traffic to come out of the port for IPv4 IP’s not configured for a VM

in_port=<PORT>,arp,dl_src=<MAC>priority=45000,actions=drop
in_port=<PORT>,arp,dl_src=<MAC>,nw_src=<IP>,priority=46000,actions=normal

Both SolusVM and OpenNebula have comes up with a solution for this and i wonder if Proxmox have anything to protect against this? And if something that is relative easy to implement.

Let me know if you need more information.

 

[Alwin]

For example so have both solusvm and Opennebula created a feature called IP Hijacking, it prevent users to be able to add a different IP-adress on their virtual machine that's not belongs to them.

So you mean IP spoofing? If so, see our docs.
https://pve.proxmox.com/pve-docs/chapter-pve-firewall.html#pve_firewall_ipfilter_section

 

[Egner]

Hi Alwin,

Thank you for your quick reply.

I mean both for mac spoofing and IP spoofing.

Ok so you have to edit the file /etc/pve/firewall/<VMID>.fw for each of the virtual machine.

Is this also included KVM/LXC?

When i try to add it like this:

[IPSET ipfilter-net0] # only allow specified IPs on net0
178.21.75.31




Then i still can add the second IP 178.21.75.32/24 without problems, i have enable ipset on the vm is there anything more i have to do?

 

[Alwin]

Can i still add the second IP 178.21.75.32/24 without problems, i have enable ipset on the vm is there anything more i have to do?

IPSETs can go by interface and will need to be enabled. Additionally the DC firewall and VM/LXC firewall needs to be activated. Otherwise nothing will be set.

 

[Egner]

IPset is enable and firewall is activated on the virtual machine.

See attached files:




root@pve1:~# pve-firewall status
Status: enabled/running




And i can ensure the firewall is working, when i set input to DROP it totally block all incoming traffic. So there must be something more with the IPset that i don't understand how to get working.

 

[Alwin]

And i can ensure the firewall is working, when i set input to DROP it totally block all incoming traffic. So there must be something more with the IPset that i don't understand how to get working.

But what does not work? To clarify, this is iptables and does nothing inside the VM.

 

[Egner]

But what does not work? To clarify, this is iptables and does nothing inside the VM.

Let me clarify, i know this is iptables rules added to the hosts, and that not have nothing to do with the actual firewall on the virtual machine itself.

Somehow i get it working now, but you need to restart the virtual machine each time you remove a IP to the ipset list. (and i tried to remove and start the virtual machine again, the extra IP-address is still working even if it's not in the ipset list. How can it be so? does it take time to apply the settings? or is it a way to force the config to take action ?


And is there a way instantly to added the rule in to action without the need of restart the virtual machine, if you understand what i mean?

 

[Alwin]

Somehow i get it working now, but you need to restart the virtual machine each time you remove a IP to the ipset list. (and i tried to remove and start the virtual machine again, the extra IP-address is still working even if it's not in the ipset list. How can it be so? does it take time to apply the settings? or is it a way to force the config to take action ?

This is probably due to connection tracking. As long as a connection is open the kernel will keep the rule, even if it has been removed already. You would need to delete the entries in the conntrack table by hand.

 

[Egner]

I found a way to do it, and that is to restart pve-firewall to get it to take action, is there a API command for this available ?

 

[Alwin]

No. It also means that the whole iptables will be flushed and re-set again. You could change the conntrack timeout to a lower value, so when the IP has been removed the state is dropped earlier.

 

[Egner]

Ok thank you,

I guess this is the one you are talking about ? net.netfilter.nf_conntrack_generic_timeout = 600

 

[Alwin]

I guess this is the one you are talking about ? net.netfilter.nf_conntrack_generic_timeout = 600

From top of my head, yes.

 

[Egner]

Well it's not working so it must be another setting in sysctl.conf that need to be adjusted.

maybe this one net.netfilter.nf_conntrack_tcp_timeout_established =

nf_conntrack_tcp_timeout_established

This node only keeps track of the netfilter connections if they live. Dead connections are deleted automatically from the table. This deletion happens based on the set timeout period. The longer the timeout period, the longer the record of the connection will stay in the tracking table. The value of this option is in seconds. By default, the value is set to 4,32,000 seconds or 12 hours

 

[Alwin]

Sounds like it. 12h is sure a long time.

 

[Egner]

Alwin, I have setup the value in the sysctl.conf and rebooted the host, but the standard value exists, is it not possible to override the setting?

Proxmox does not have a setting that prevent the change?

See here:
first i run systemct -p
net.netfilter.nf_conntrack_tcp_timeout_established = 3600

root@pve3:~# sysctl -a | grep nf_conntrack_tcp_timeout_established
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
root@pve3:~# sysctl -a | grep nf_conntrack_tcp_timeout_established
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
root@pve3:~# sysctl -a | grep nf_conntrack_tcp_timeout_established
net.netfilter.nf_conntrack_tcp_timeout_established = 432000 (Here it jumps back to the default value automatic after a few seconds)

 

[Alwin]

Proxmox does not have a setting that prevent the change?

No. It's Debian with an Ubuntu-based Kernel.

Alwin, I have setup the value in the sysctl.conf and rebooted the host, but the standard value exists, is it not possible to override the setting?

Is the nf_conntrack module loaded? Otherwise there will be no sysctl setting.

 

[Egner]

Yup i have added it to /etc/modules

cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
nf_conntrack

 

[Alwin]

Yup i have added it to /etc/modules

But is it loaded?

 

[Egner]

But is it loaded?

Yup

root@pve3:/etc/pve# lsmod | grep nf_conntrack
nf_conntrack 139264 4 xt_conntrack,nf_nat,openvswitch,nf_conncount
nf_defrag_ipv6 24576 2 nf_conntrack,openvswitch
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 8 nf_conntrack,nf_nat,dm_persistent_data,bnx2x,openvswitch,btrfs,raid456,sctp

 

[Egner]

Hi Alwin,
It took about 20 minutes until iptables release the state, so i think i can live with that.

Thank you so much.