Not sure I understand your requests 100% - do you want to skip processing mails for users who are not on a list?
That should be possible - just create a Who Object with all addresses that you want to process mails for (Subscriber list" and then a list with high priority (above all other rules):
* Who - "Does not match any" - "Subscriber list"
* Action Accept
please test this before putting it in production
I hope this helps!