martian source from itself

[TheForumTroll]

Hello experts!

I'm seeing a lot of martian sources in my pve syslog log which has me rather confused as it seems to be from a host to the same host:

IPv4: martian source 192.168.20.102 from 192.168.20.102, on dev eth1 ll header: 00000000: 4e a8 57 59 29 13 e4 11 5b 9b 99 81 08 00 IPv4: martian source 192.168.20.102 from 192.168.20.102, on dev eth1 ll header: 00000000: 4e a8 57 59 29 13 e4 11 5b 9b 99 81 08 00 IPv4: martian source 192.168.20.102 from 192.168.20.102, on dev eth1 ll header: 00000000: 4e a8 57 59 29 13 e4 11 5b 9b 99 81 08 00

It happens approx. 25 times over a minute and then again after something like 20 minutes, over and over. Everything in proxmox has firewall turned on.

 

[mira]

Can you provide the output of ip -details a?

Does network work reliably all the time?

 

[TheForumTroll]

As far as I can tell the network works fine.

~# ip -details a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:00:ac:ca:d3:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth numtxqueues 32 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
    inet 192.168.0.102/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6800:acff:feca:d380/64 scope link
       valid_lft forever preferred_lft forever
3: eth1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4e:a8:57:59:29:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth numtxqueues 32 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
    inet 192.168.20.102/24 brd 192.168.20.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::4ca8:57ff:fe59:2913/64 scope link
       valid_lft forever preferred_lft forever

 

[mira]

So you're using veths. That means we would need the output of ip -details link as well.

Do you have other guests with the same IP configured by any chance?

 

[TheForumTroll]

~# ip -details link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
2: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 6a:00:ac:ca:d3:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth addrgenmode eui64 numtxqueues 32 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
3: eth1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 4e:a8:57:59:29:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth addrgenmode eui64 numtxqueues 32 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535

No other guests with the same IP, it is also the only CT on 192.168.20.*

Edit: Only other device on .20.* is the OPNsense firewall at .20.1

 

[spirit]

sound like a network loop or something like that, where you received traffic that you have send.

(you can verifiy with a tcpdump the mac address of incoming packet with your own ip, to see if it's same mac or not)

 

[TheForumTroll]

I'm not very knowledgable in networking, but I interpret it as data is in fact between eth0 and eth1 on the same host. The header MAC addresses from the syslog fit the MAC addresses from the ip command. But I have no idea why this happens and why it isn't blocked when the firewall is turned on in the CT and eth1/192.168.20.102 is a VLAN and should be blocked from eth0/192.168.0.102

 

[spirit]

if it's occur sometimes around 20 minutes, They are chance it's coming from mac-address-table ageing timeout (on your physical switch) or arp timeout.

If you have gateway (maybe pfsense here ?), you need to check that arp timeout is lower than mac-address-table ageing of the switches, to be sure than switches mac table is refreshed and mac entries don't expire, or the switch will flood traffic to all ports until it receive a packet.



(Personnaly, in production, I'm using 2h pour ageing mac address timeout on my switches).

 

[TheForumTroll]

The switches are at 300 seconds timeout. I'm not sure where I see what timeout OPNsense use but a quick Google search tell me it is 20 minutes. I have no idea if that is correct though. I'll see what I can find.

 

[spirit]

The switches are at 300 seconds timeout. I'm not sure where I see what timeout OPNsense use but a quick Google search tell me it is 20 minutes. I have no idea if that is correct though. I'll see what I can find.

try to reduce opensense arp timeout or increase switches mac timeout.

it should fix your problem.

(300s for switches seem really low, until you have low memory in your switch asic for mac address table in your switches and a lot of macs in your network, it shouldn't be a problem to increase it).

 

[TheForumTroll]

300 seconds (5 hours) is the default in both Cisco and HP switches. It is much higher than the approx. 15-25 minutes intervals I'm seeing.

 

[_gabriel]

300 seconds = 5 MINUTES

 

[TheForumTroll]

Oh, brainfart. Minutes vs seconds

 

[TheForumTroll]

Small update: Changing the ageing timeout didn't fix it. Still have a log full of IPv4: martian source 192.168.20.102 from 192.168.20.102, on dev eth1

Eth1 is 192.168.20.102 so why it is a martian package I don't know.

 

[TheForumTroll]

As it seems I can't stop this warning from happening I need to turn it off until I find a fix as it spams my logging server and might hide more important alerts. However I can't seem to make it stop:

/proc/sys/net/ipv4/conf/all/log_martians = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.vmbr0.log_martians = 0
net.ipv4.conf.enp2s0f0.log_martians = 0

Am I missing something?

 

[Onyx]

In case this could help someone:

I had a similar problem with kernel martian messages in syslog on the proxmox hosts.
I have pfsense vm's where internet traffic arrives and routes it to containers and vm's on the same and other proxmox hosts. The proxmox hosts are only out-of-band accessible but do host the L2 stuff (vlans, bridges...).

In my case I discovered that those martian messages had mainly a source IP that not existed in that network. They came with arp requests where there was no response and those happen constantly from bots scanning that network (if traffic is not explicitly blocked to non-existing IP's).

The solution was to disable rp filter on the proxmox hosts. (/proc/sys/net/ipv4/conf/<interfacename>/rp_filter or in the settings of the firewall you use on that host, Shorewall in my case)