[SOLVED] VIRUSTOTAL API integration.

itNGO

Well-Known Member
Jun 12, 2020
776
176
53
45
Germany
it-ngo.com
Is it possible to integrate the VirusTotal-API for FILE, URL or Domain-Scan as a rule?
We are evaluating this in rspamd on a dedicated host and like the idea to use the external service to enhance detection rate by a massive amount....
 
hi,

you could take a look at implementing custom checks [0]
if you can call virustotal API from a script then you should be able to do it.

to use the external service to enhance detection rate by a massive amount
also be aware that you'll get more delays on your emails, since they will be scanned by virustotal (and you'll have to wait for API response), not to mention that everything will be uploaded to their servers (just a consideration).

[0]: https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#pmgconfig_custom_check
 
Is it possible to integrate the VirusTotal-API for FILE, URL or Domain-Scan as a rule?
Currently the way to do this would be to write a custom_check_script which interacts with the virustotal api:
https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#pmgconfig_custom_check

We are evaluating this in rspamd on a dedicated host and like the idea to use the external service to enhance detection rate by a massive amount....
The idea sounds good in general - but as far as I could find out - directly integrating it in PMG could be problematic:
https://developers.virustotal.com/reference/public-vs-premium-api
* the free API is explicitly not for commercial products and services (so we at least would need to make this very explicit for the users)
* the rate-limits are a bit too restrictive for a mail-gateway (4/minute, 500/day)

I would be curious how the premium API is priced (could not find anything on a quick search) - if it's somewhat reasonable, I could imagine implementing some kind of integration like PMG has with avast.
So should you have a premium API account or have contacted them - I would be grateful if you share your experiences
 
  • Like
Reactions: itNGO
Currently the way to do this would be to write a custom_check_script which interacts with the virustotal api:
https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#pmgconfig_custom_check


The idea sounds good in general - but as far as I could find out - directly integrating it in PMG could be problematic:
https://developers.virustotal.com/reference/public-vs-premium-api
* the free API is explicitly not for commercial products and services (so we at least would need to make this very explicit for the users)
* the rate-limits are a bit too restrictive for a mail-gateway (4/minute, 500/day)

I would be curious how the premium API is priced (could not find anything on a quick search) - if it's somewhat reasonable, I could imagine implementing some kind of integration like PMG has with avast.
So should you have a premium API account or have contacted them - I would be grateful if you share your experiences
Code:
VirusTotal premium services are meant for companies and public sector organizations, not individuals. Popular threat intelligence packages lie in the range $(XX)K-(XXX)K, although we can also work on custom solutions for your use case.



Whatever that means... I will try to get more about this, once we have a custom-check-prototype to see if it is of any use with more realistic E-Mail-Workload....
 
$(XX)K-(XXX)K
if that's 10k - 999k then I'd assume it's out of the price-range for most our users ;)

. I will try to get more about this, once we have a custom-check-prototype to see if it is of any use with more realistic E-Mail-Workload....

nice - keep us posted on your experiences - Thanks!
 
Well I made a more or less script for custom_check but get in trouble with "pmg-smtp-filter" spawning new threads every time my custom-script sends a hash/file to Virustotal. Is there a setting to allow more threads as for now this leads very fast to following error in log.

Code:
Aug 4 13:36:35 RZ-PMG postfix/smtpd[1884]: warning: timeout talking to proxy 127.0.0.1:10023
Aug 4 13:36:35 RZ-PMG postfix/smtpd[1884]: proxy-reject: END-OF-MESSAGE: 451 4.3.0 Error: queue file write error; from=<REMOVESD> to=<REMOVED> proto=ESMTP helo=<SOMETARGETSERVER>
Aug 4 13:36:35 RZ-PMG postfix/smtpd[1884]: NOQUEUE: client=unknown[10.255.14.30]
 
Is there a setting to allow more threads as for now this leads very fast to following error in log.
the number of pmg-smtp-filter processes is based on the available memory in the system - and is not configurable (short of editing the code):
https://git.proxmox.com/?p=pmg-api....b4005610ba0b72362efadba52b4531158;hb=HEAD#l94

However I wouldn't expect that this is the main fault - what are the other processess doing at that point?
How long does your custom check script need for interacting with virustotal?
Could you please share more of the logs (especially anything from pmg-smtp-filter itself)...
 
the number of pmg-smtp-filter processes is based on the available memory in the system - and is not configurable (short of editing the code):
https://git.proxmox.com/?p=pmg-api....b4005610ba0b72362efadba52b4531158;hb=HEAD#l94

However I wouldn't expect that this is the main fault - what are the other processess doing at that point?
How long does your custom check script need for interacting with virustotal?
Could you please share more of the logs (especially anything from pmg-smtp-filter itself)...
Never mind... this was just a script bug on our site.
So far we made a simple script to query the API.

Just waiting if it is capturing..... so far it look promising... will post my script when it has proven to work....
 
Script is working so far... Uploads every MAIL for analytic to Virustotal waits 77 seconds and gets result.
Then adds spamscore depending on the result....

However there came a question.... GOBD or in Germany the so called (DSGVO).... I think this can not be used at all when you have to follow law about data privacy, cause you can not select what to "scan" and "upload" for now.... we have to discuss this further until we can continue....
 
However there came a question.... GOBD or in Germany the so called (DSGVO).... I think this can not be used at all when you have to follow law about data privacy, cause you can not select what to "scan" and "upload" for now.... we have to discuss this further until we can continue....
thought it's called GDPR in English (but am not too well versed on these topics ....).
In any case - again only by quickly glancing over the API - I thought you upload md5 hashes of files only?!
If that's the case I would doubt that GOBD/GDPR/DSGVO would be violated?
 
thought it's called GDPR in English (but am not too well versed on these topics ....).
In any case - again only by quickly glancing over the API - I thought you upload md5 hashes of files only?!
If that's the case I would doubt that GOBD/GDPR/DSGVO would be violated?
That's the alternative to just use Hashes.... still heavily work in progress while staff is complaining about the don't and does.... ;-)
 
So here we are.
Hash-Check for E-Mail and Attachment against Virustotal API. This is really not good BASH-Code, but it works. Open for suggestions and optimizations.

E-Mails and Attachments are extracted with MUNPACK into a Temp-Folder below /tmp and removed again after generating HASH and Check against Virustotal-API is done.

For now only "Malicious Count" is used, but this can be enhanced to also look for "Suspicious" or any other result later....
If the File-Hash has never been seen at Virustotal it will also return "0" which is considered as clean for now.
So you have Hash-Check against 50 or more AV-Engines.

Every check is delayed by 7 seconds. Next is to contact VIRUSTOTAL to get pricing for this usage. Currently I would guess, this is not allowed in free API except only for private usage!

#!/bin/sh vt_api_key=***PUT_YOUR_API_KEY_HERE*** max_mail_size=2M echo "called with $*" 1>&2 if [ "$#" -ne 2 ]; then echo "usage: $0 APIVERSION QUEUEFILENAME" 1>&2 exit 1 fi apiver="$1" shift if [ "$apiver" != "v1" ]; then echo "wrong APIVERSION: $apiver" 1>&2 exit 2 fi queue_file="$1" echo "v1" #Extract Mail and Attachments mkdir -p /tmp$queue_file > /dev/null cd /tmp$queue_file > /dev/null munpack -fqt $queue_file > /dev/null FILES="/tmp$queue_file/*" rm -f /tmp$queue_file/*.txt > /dev/null rm -f /tmp$queue_file/*.png > /dev/null rm -f /tmp$queue_file/*.jpg > /dev/null rm -f /tmp$queue_file/*.desc > /dev/null rm -f /tmp$queue_file/part* > /dev/null if [ -z "$(ls -A /tmp$queue_file/)" ] ; then logger "VIRUSTOTAL Nothing to do for $queue_file... EXIT" echo OK exit fi for f in $FILES do logger "VIRUSTOTAL Processing $f file..." # take action on each file. $f store current file name sha256=$(sha256sum $f) logger "VIRUSTOTAL Send Hash for File $f" data=$(curl -s --request GET --url https://www.virustotal.com/api/v3/files/$sha256 --header "x-apikey: $vt_api_key" | jq '.[]' | grep -oP '(?<="malicious": ).*?(?=,)') choice=$(echo $data) re='^[0-9]+$' if ! [ "$choice" = "$re" ] ; then choice=0 logger "VIRUSTOTAL File $queue_file Malicious Count: $choice - $f" fi if ! [ "$choice" = "0" ] ; then logger "VIRUSTOTAL File $queue_file Malicious Count: $choice - $f" break fi sleep $((RANDOM % 17)) done rm -fr /tmp$queue_file/ case "$choice" in 0) echo OK ;; 1) echo SCORE: 1 ;; 2) echo SCORE: 2 ;; 3) echo SCORE: 3 ;; 4) echo SCORE: 4 ;; 5) echo SCORE: 5 ;; 6) echo SCORE: 6 ;; 7) echo SCORE: 8 ;; *) echo SCORE: 10 ;; esac exit 0
 
Last edited:
Well... got some pricing.... forget it.... nice idea... but what they call is nowhere affordable for small providers or companies.....
Pricing for API:
1'000 Daily API lookups - €7'920 / year
5'000 Daily API lookups - €19'800 / year
10'000 Daily API lookups - €31'680 / year
(there are bigger quotas available) :oops::eek::rolleyes:o_O
 
got some pricing....
Thanks for the script and for sharing that information - and I'd agree that this will probably be beyond most users' budget!
 
Thanks for the script and for sharing that information - and I'd agree that this will probably be beyond most users' budget!
But if someone really needs it... he can now do it.... and for private personal home usage, it is also allowed and the free key should be enough for around 1000 mails with attachment a day.... will upgrade script in the posting, as I made some fine tuning....
 
  • Like
Reactions: Stoiko Ivanov

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!