Interfacing PMG with a Neural Network for spam score alteration

[AFrenchCompany]

Hi,

I have trained a small model with highly curated spam and ham from our own PMG instance.

The model is behaving extremely well and I would like to use it to alter the spam score from PMG's spamassassin slightly.

Currently,
* the model can be ran in CLI in a conda env
* it accepts an .eml file as the input
* answers with a float score for spamassassin [-2,+2].

What would be my best options to interface PMG with this model ?

Thanks !


edit: If that makes it easier, I could expose the inference via a web API, taking the .eml as body and answering with JSON or plaintext.

 

[AFrenchCompany]

There seems to be a way by creating a SpamAssassin Perl plugin, which in turn would call the python script or API.

If no recommendations are made around here, I will try this method and update the thread.

Hopefully that could help others.

 

[AFrenchCompany]

I used the Custom Check Interface feature of PMG.
I have a script that properly queries my model and returns a score.

But I cannot see that score anywhere in the X-SPAM-LEVEL.
Is this expected ?

Script is in /usr/local/bin/pmg-custom-check
`custom_check 1` has been changed to /etc/pmg/pmg.conf
and I can see that my API is being hit.

Edit: OK, something's wrong with my custom check apparently.
This is visible in the tracking center `pmg-smtp-filter[572]: 31D8366705279DBE8F custom check error: got unexpected output!`

 

[AFrenchCompany]

The output seems aligned with what the documentation states.
Yet I still get custom check error: got unexpected output!

root@spamfilter:~# /usr/local/bin/pmg-custom-check v1 /var/spool/pmg/spam/A0/2F9A966700405B62A0
v1
SCORE: 1.91

Am I missing something regarding what the custom check is supposed to return ?

Edit: this is extremely odd, despite the errors in the logs, when emails are classified as spam I can see the CustomCheck score.
Edit2: the errors aren't systematic, some emails do get the score [...] autolearn=no autolearn_force=no hits=CustomCheck(1.37)

 

[Stoiko Ivanov]

The output seems aligned with what the documentation states.
Yet I still get custom check error: got unexpected output!

root@spamfilter:~# /usr/local/bin/pmg-custom-check v1 /var/spool/pmg/spam/A0/2F9A966700405B62A0
v1
SCORE: 1.91

Am I missing something regarding what the custom check is supposed to return ?

This should work - here a link to the relevant source-code:
https://git.proxmox.com/?p=pmg-api....0e2aa33e6c2b286ab92;hb=refs/heads/master#l381
could you show the logs of one e-mail where it fails?

does your script also provide negative values (with fractional parts)? - I think there might be a glitch there in the source...

 

[Stoiko Ivanov]

does your script also provide negative values (with fractional parts)? - I think there might be a glitch there in the source...

In any case - the glitch was there - I checked that the issue was indeed there and wrote a patch:
https://lists.proxmox.com/pipermail/pmg-devel/2024-June/002928.html

once this (or an improved version) gets applied and released also scores of the form '-1.91' will work.

Thanks for reporting!

 

[AFrenchCompany]

Thank you for investigating

Indeed, the script returns in the [-2,2] range which include negative values for emails it deems to be probably not spam.

 

[AFrenchCompany]

I've applied the patch and can confirm that it works great.



If I may suggest something though, CustomCheck doesn't follow the same casing as other rules.
For consistency sake it could be more appropriate to display it as CUSTOM_CHECK