You are correct.
It did not work, because of how it's designed. I have default DROP policy and when such is set, ipfilter does not work correctly.
I decided to go with my firewall rules, which block in both ways (ipfilter does only egres filtering, relying on mac filter) and mac filter.
Works...