[TUTORIAL] Use OpenRouter to get Spamscore

ivenae

Renowned Member
Feb 11, 2022
206
101
63
42
I built an app that performs an AI check via a SpamAssassin filter.

What data is transmitted?
- Sending server
- Sender
- Subject
- Recipient

Sender, recipient, and subject may be a data protection issue!

What does it cost?
Depends on the chosen model and mailtraffic. There are free models on OpenRouter, but they change frequently (and you have to update them in your config to keep things working). You need to create an account and enter an API key.
It is possible to configure both free and paid models. The tool keeps track of models that do not work and pauses their usage for a configurable period of time. As long as free, non-blocked models are available, no paid models will be used. If you want to use a paid model, you can choose very inexpensive ones, e.g. DeepSeek v4 Flash.

Doesn't that generate a lot of traffic?
That depends. The integration is designed to conserve resources. First, all SpamAssassin checks are performed. The AI script runs last and evaluates the eventually score total. If it falls within a specific range (in this case: between 2 and 9; I reject emails starting from 7), the AI script becomes active and sends the data to an AI.
I suggest using something like my policyguard to filter many mails very early. In my case, 1000 daily mails will produce approx. 50 AI scans per day.

Can I use a local AI?
Yes. Local AIs use the same interfaces as OpenRouter and generally work. Qwen, for example, would probably work well, although I haven’t tested it myself yet.


Code:
# Install requirements
apt install python3-openai

# Create Folder for Statistics
mkdir -p /t/
touch /t/ai-spamcheck.log
touch /t/ai-spamcheck-stats.json
chmod -R 777 /t/

Paste your API Key into /etc/ai-spamcheck.toml
Remove the txt file ending from attached files and copy them to
Code:
/etc/mail/spamassassin/aicheck.cf
/usr/local/lib/AICheck.pm
/usr/local/bin/ai-spamcheck.py
/etc/ai-spamcheck.toml

Restart Spamassassin
Code:
systemctl restart pmg-smtp-filter.service

Code:
# Test with E-Mail with score between 2 and 9
spamassassin -t < test.eml


Changes you can made in AICheck.pm
Code:
    SCRIPT   => "/usr/local/bin/ai-spamcheck.py",
    TIMEOUT  => 15,          # hard kill for the external call (seconds)
    RULENAME => "AI_SCORE",
    MIN_GATE => 2.0,         # run AI check only if SA score so far is >= this
    MAX_GATE => 9.0,         # ... and below this (PMG block level)

Changes you can made in ai-spamcheck.toml
Code:
API_KEY = "sk-or-v1-1<your API key>"                    # your OpenAI API key= "sk-or-v1-REPLACE-ME"
API_URL = "https://openrouter.ai/api/v1"                                                                 # base URL
TIMEOUT = 4.0                               # API request timeout in seconds

# List of model IDs. One is chosen at random per run.  Free before paid.
MODEL = [
    "minimax/minimax-m2.7:free",
    "minimax/minimax-m3:free",
    "google/gemma-4-31b-it:free",
    "deepseek/deepseek-v4-flash-0731",
    "upstage/solar-pro4",
]

# JSON dict tracking usage and errors per model.
# The directory must exist and be writable by the script's user.
STATISTIC_FILE = "/t/ai-spamcheck-stats.json"

# Table log file.
LOG_FILE = "/t/ai-spamcheck.log"


Code:
Content preview:  Servus Herr XXXXX, sollten Sie noch Leute für Ihre Autowerkstatt
   suchen, kann ich Ihnen hierfür gerade eine Einstellungsgarantie aussprechen.
   Wäre das grundsätzlich für Sie interessant?

Content analysis details:   (9.2 points,  5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 8.0 URIBL_ABUSE_SURBL      Contains an URL listed in the ABUSE SURBL blocklist
                            [URI: getrfmarketingagency.com]
-3.0 POS_XXXXX              BODY: No description available.
 1.2 AI_SCORE               HEADER: AI-based spam score
 3.0 HTML_MIME_NO_HTML_TAG  HTML-only message, but there is no HTML tag


Code:
#cat /t/ai-spamcheck.log
2026-09-07 03:26:57 | -1   | nvidia/nemotron-3.5-lightni… | TYSPR03MB9145.apcprd03.prod… | Jofar Ben <jofararcherben@outlook.com>   | Jofar Ben <jofararcherben@o… | Re: Android app development?
2026-09-07 03:27:26 | 2.9  | nvidia/nemotron-3.5-lightni… | sjc04p1infpx012.cb2.sjc04.t… | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX          | "info@XXXXXXXXXXXXXXXXXX" <… | Funds transfer remittance advice
2026-09-07 03:27:58 | -2   | nvidia/nemotron-3.5-lightni… | gu.d.sender-sib.com          | XXXXXXXXXXXXXXXXXXXXXXXXX <no-reply@ser… | abwicklung@XXXXXXXXX, artur… | XXXXX / Eigenversand FZ-Dokumente / 14.08.2026
2026-09-07 03:28:00 | -1   | minimax/minimax-m3:free      | localhost                    | Clara Becker <becker@getrfmarketingagen… | info@XXXXXXXXXXXXXXXXXX      | Bewerbung
2026-09-07 03:28:03 | 2.5  | minimax/minimax-m3:free      | localhost                    | Clara Becker <becker@getrfmarketingagen… | info@XXXXXXXXXXXXXXXXXX      | Bewerbung
2026-09-07 03:28:06 |  3   | minimax/minimax-m3:free      | by mail-oi1-f169.google.com… | Safiya Magnumikz | E & D KOKORIKO <safi… | undisclosed-recipients:;     | Inquiry About Product Purchase and Shipping to Cyprus
2026-09-07 03:28:15 | -2   | nvidia/nemotron-3.5-lightni… | gu.d.sender-sib.com          | XXXXXXXXXXXXXXXXXXXXXXXXX <no-reply@ser… | abwicklung@XXXXXXXXX, artur… | XXXXX / Eigenversand FZ-Dokumente / 14.08.2026
 

Attachments

Last edited:
Feedback:

Conclusion: Not recommended without further improvement

Code:
2026-09-07 06:00:28 | 2      | qwen2.5:3b                   | by mailer101-1.mailemm.de f… | Digitaler Vertragsabschluss bei Bank11: So legitimieren Sie…
2026-09-07 06:01:15 | 0      | qwen2.5:3b                   | client.hidden.invalid        | Volland3
2026-09-07 06:01:24 | 2      | qwen2.5:3b                   | smtpclient.apple             | Arbeitsgrößen Philipp R.
2026-09-07 06:01:38 | 2      | qwen2.5:3b                   | [61.8.138.53]                | Dringende Terminanfrage- Verkehrsunfall vom 14.08.2026
2026-09-07 06:02:45 | 2      | qwen2.5:3b                   | MI3PR05MB13040.eurprd05.pro… | AW: Vertragsanpassung
2026-09-07 06:02:58 | 2      | qwen2.5:3b                   | B0D0N4U2.zurich.com          | Zurich Schadennr.: 26-15xxxxx
2026-09-07 06:27:40 | 0      | minimax/minimax-m3:free      | by mail-wr1-f41.google.com … | SEHR DRINGEND!!!!!! Re: 312/26 Krause ./. Schulze
2026-09-07 06:28:11 | -0     | google/gemma-4-31b-it:free   | sonic.gate.mail.ne1.yahoo.c… | Brandschutztür im Keller
2026-09-07 06:28:39 | -2     | minimax/minimax-m2.7:free    | sonic.gate.mail.ne1.yahoo.c… | Brandschutztür im Keller
2026-09-07 07:05:26 | 0.8    | minimax/minimax-m3:free      | localhost                    | 17 kritische Bewertungen für ihr Unternehmen absch…
2026-09-07 07:06:10 | -2     | minimax/minimax-m2.7:free    | localhost                    | 17 kritische Bewertungen für ihr Unternehmen absch…
2026-09-07 07:43:10 | 4.2    | minimax/minimax-m3:free      | localhost                    | Ihr Unternehmen: Was Ihre 7 kritischen Bewertungen…
2026-09-07 08:44:48 | -0     | minimax/minimax-m3:free      | mail195.atl241.mcsv.net      | Ein Impuls für den Wirtschaftsstandort Düsseldorf – und ein…
2026-09-07 08:45:19 | -2     | minimax/minimax-m2.7:free    | mail195.atl241.mcsv.net      | Ein Impuls für den Wirtschaftsstandort Düsseldorf – und ein…
2026-09-07 09:05:40 | -2     | minimax/minimax-m2.7:free    | mss02.controlexpert.eu       | Prüfbericht zu Ihrer Rechnung Nummer 100580 (xxxxxxxxx)
2026-09-07 09:07:46 | -2     | minimax/minimax-m2.7:free    | ip-10-11-42-86.ca-central-1… | Kurzer Gedanke zur Website von Ihr Unternehmen…
2026-09-07 09:23:08 | -2     | minimax/minimax-m2.7:free    | geopod-ismtpd-20             | 4 neue Bildstile zum Ausprobieren
2026-09-07 10:01:18 | -1     | minimax/minimax-m2.7:free    | kitex1.myintellisource.com   | Mehr Effizienz für Ihr Unternehmen - mit der Cloud-Softwar…
2026-09-07 10:18:54 | -2     | minimax/minimax-m2.7:free    | geopod-ismtpd-5              | VW Golf 1.5 eTSI LIFE, available !
2026-09-07 10:20:08 | -2     | minimax/minimax-m2.7:free    | s4-141.my-control-panel.com  | Initiativbewerbung als Full-Stack Entwickler / Technischer …
2026-09-07 10:20:26 | -1     | minimax/minimax-m2.7:free    | s4-141.my-control-panel.com  | Initiativbewerbung als Full-Stack Entwickler / Technischer …
2026-09-07 10:31:13 | -2     | minimax/minimax-m2.7:free    | mail162.suw12.mcsv.net       | Zur Vorbereitung in der Versammlung
2026-09-07 10:31:13 | -2     | minimax/minimax-m2.7:free    | mail162.suw12.mcsv.net       | Zur Vorbereitung in der Versammlung
2026-09-07 10:34:01 | 2      | minimax/minimax-m2.7:free    | geopod-ismtpd-14             | VW Tiguan 1.5 eTSI LIFE, available!
2026-09-07 13:02:48 | -0     | google/gemma-4-31b-it:free   | a9-110.smtp-out.amazonses.c… | ALLDATA Rechnung - INVC0739xxxxx


Local models Qwen2.5 3B:

  • Qwen2.5 3B is a local model. Mail content was also passed to it. Nevertheless, the model keeps producing the same score. I’ve also found similar reports online with the same behavior.
    Implication: Use significantly larger models and much more prompt engineering. But that also means regular adjustments. If I define today: “No ‘job applications’,” then even legitimate candidates like Anna Müller applying for an accounting position would get filtered out.
Regarding the online models:

  • The score “-0” is assigned due to an error. In those cases, no actual check was performed (e.g., rate limit or timeout).
  • You can see the model temperature effect. Identical inputs sometimes produce noticeably different results. That’s fundamentally undesirable for a spam filter (e.g., “17 critical reviews”).
  • The results are too positive. IHK spam is rated positively, vehicle UCE is rated positively for the VW Golf, correctly negative for the Tiguan, “4 new image styles to try” is more of a newsletter than spam—but the negative score is clearly nonsense.
  • Overall, I’d say it’s not really usable. Because of this “positivity bias,” one could at least try counting only negative scores and ignoring positive ones to push some emails toward spam.
  • Conclusion: 8 hours of effort, unfortunately unusable results.
Potential improvement: Include the full email content, not just headers—but that raises data protection concerns.

Next step: I’ll constrain the system to only assign scores between 0 and 5 and let it run for a while. I will also try different models. I would say the minimax3 was better than minimax 2.7. Now that i have proven, that there are not many requests, i will try paid models like deepseek v4 and optimize prompt engineering.

There is another project on my list: project "first encounter", to scan all e-mails from never-received domains, too. ... catch phishing from hacked websites.
Maybe you just try yourself...
 
Last edited: