[SOLVED] Maximum amount of who-objects?

Mr_Diba · Oct 10, 2022

I was wondering if there is a maximum amount of who-objects PMG can handle?

The reason for asking is that I would like to redirect most of the TLD's to the quarantine for assessment.
I have a rule that looks like this:

Code:

Found RULE 15 (prio: 80, in, active): Graylist
  FOUND FROM GROUP 29: GraylistTLD
    OBJECT 2814: .*\.aarp
    OBJECT 2815: .*\.abarth
    OBJECT 2816: .*\.abb
    OBJECT 2817: .*\.abbott
    OBJECT 2818: .*\.abbvie
    OBJECT 2819: .*\.abc
    ...
    <insert around 1300 TLDs more here>
    ...
    OBJECT 3997: .*\.today
    ...
    OBJECT 4135: .*\.zm
    OBJECT 4136: .*\.zone
    OBJECT 4137: .*\.zuerich
    OBJECT 4138: .*\.zw

  FOUND FROM GROUP 33: GraylistOther
    OBJECT 4144: .*kvk.emailing@.*
    OBJECT 4145: .*kvk.mailingservice@.*
    OBJECT 4146: online.nl
    OBJECT 4147: onsmail.nl
    OBJECT 4148: onsnet.nl
  FOUND ACTION GROUP 19: Quarantine
    OBJECT 32: Move to quarantine.

But I still receive mail from:

Code:

Return-Path: <56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today>

And when I look in the Tracking Center I see the following line:

Code:

Oct  8 16:46:15 <pmg-server> pmg-smtp-filter[443346]: 30015D63418D36D1F0A: accept mail to <<receiver>@<domain>.com> (D3693205DDE) (rule: default-accept)

Which makes me wonder if the 1330 objects are to much to handle resulting in a timeout and just accepting the e-mail?

pmgversion: pmg-api/7.1-7/4d02e400 (running kernel: 5.15.60-1-pve)

Stoiko Ivanov · Oct 10, 2022

Mr_Diba said:
Which makes me wonder if the 1330 objects are to much to handle resulting in a timeout and just accepting the e-mail?

There is no built-in hard limit in the rulesystem (and I'm not aware of anyone running into any) , plus 1330 regexes should easily fit in memory and work out quite smoothly afaict

What's the Action on rule 15 'Graylist' ? (the snippet of pmgdb dump you posted only shows the action for GraylistOther

also the complete log for this mail might help

do you have any odd messages in the journal from pmg-smtp-filter, postfix (or any other pmg service)?

Mr_Diba · Oct 10, 2022

Both GraylistOther and GraylistTLD should go to the Quarantine, but reading your reply I might misunderstand how multiple "From" objects operate together. I assume that multiple "From" objects are just a logical "OR"?

Full maillog:

Code:

Oct 8 16:46:14 <pmg-server> postfix/smtpd[443486]: connect from <mail-relay>[<mail-relay-ip>]
Oct 8 16:46:14 <pmg-server> postfix/smtpd[443486]: C46B4205DCF: client=<mail-relay>[<mail-relay-ip>]
Oct 8 16:46:14 <pmg-server> postfix/cleanup[443143]: C46B4205DCF: message-id=<rt4yacj781bgvvkt-5pbwzksya9b0nuxs-16649-20a4c@brainc13.today>
Oct 8 16:46:14 <pmg-server> postfix/qmgr[1072]: C46B4205DCF: from=<56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today>, size=8280, nrcpt=1 (queue active)
Oct 8 16:46:14 <pmg-server> postfix/smtpd[443486]: disconnect from <mail-relay>[<mail-relay-ip>] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Oct 8 16:46:14 <pmg-server> pmg-smtp-filter[443346]: 30015D63418D36D1F0A: new mail message-id=<rt4yacj781bgvvkt-5pbwzksya9b0nuxs-16649-20a4c@brainc13.today>#012
Oct 8 16:46:15 <pmg-server> pmg-smtp-filter[443346]: 30015D63418D36D1F0A: SA score=1/5 time=0.962 bayes=0.50 autolearn=no autolearn_force=no hits=BAYES_50(0.8),DKIM_INVALID(0.1),DKIM_SIGNED(0.1),HTML_FONT_LOW_CONTRAST(0.001),HTML_MESSAGE(0.001),KAM_DMARC_NONE(0.25),KAM_DMARC_STATUS(0.01),LONGLN_LOW_CONTRAST(1.596),RAZOR2_CF_RANGE_51_100(1.886),RAZOR2_CHECK(0.922),RCVD_IN_DNSWL_HI(-5),SPF_HELO_NONE(0.001),T_SPF_PERMERROR(0.01),URIBL_ABUSE_SURBL(1.25),URIBL_BLOCKED(0.001)
Oct 8 16:46:15 <pmg-server> postfix/smtpd[443148]: connect from localhost.localdomain[127.0.0.1]
Oct 8 16:46:15 <pmg-server> postfix/smtpd[443148]: D3693205DDE: client=localhost.localdomain[127.0.0.1], orig_client=<mail-relay>[<mail-relay-ip>]
Oct 8 16:46:15 <pmg-server> postfix/cleanup[443143]: D3693205DDE: message-id=<rt4yacj781bgvvkt-5pbwzksya9b0nuxs-16649-20a4c@brainc13.today>
Oct 8 16:46:15 <pmg-server> postfix/smtpd[443148]: disconnect from localhost.localdomain[127.0.0.1] ehlo=1 xforward=1 mail=1 rcpt=1 data=1 commands=5
Oct 8 16:46:15 <pmg-server> postfix/qmgr[1072]: D3693205DDE: from=<56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today>, size=9844, nrcpt=1 (queue active)
Oct 8 16:46:15 <pmg-server> pmg-smtp-filter[443346]: 30015D63418D36D1F0A: accept mail to <<receiver>@<domain>.com> (D3693205DDE) (rule: default-accept)
Oct 8 16:46:15 <pmg-server> postfix/smtp[443352]: Trusted TLS connection established to <imap-server-ip>[<imap-server-ip>]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)
Oct 8 16:46:15 <pmg-server> pmg-smtp-filter[443346]: 30015D63418D36D1F0A: processing time: 1.055 seconds (0.962, 0.024, 0)
Oct 8 16:46:15 <pmg-server> postfix/lmtp[443144]: C46B4205DCF: to=<<receiver>@<domain>.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.02/0/0.04/1.1, dsn=2.5.0, status=sent (250 2.5.0 OK (30015D63418D36D1F0A))
Oct 8 16:46:15 <pmg-server> postfix/qmgr[1072]: C46B4205DCF: removed
Oct 8 16:46:15 <pmg-server> postfix/smtp[443352]: D3693205DDE: to=<<receiver>@<domain>.com>, relay=<imap-server-ip>[<imap-server-ip>]:25, delay=0.09, delays=0.05/0/0/0.03, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as E6294100FC4)
Oct 8 16:46:15 <pmg-server> postfix/qmgr[1072]: D3693205DDE: removed

Mail header:

Code:

Return-Path: <56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today>
Received: from gws.<domain>.com ([unix socket])
     by mail02 (Cyrus 3.2.6-Debian-3.2.6-2+deb11u2) with LMTPA;
     Sat, 08 Oct 2022 16:46:15 +0200
X-Cyrus-Session-Id: cyrus-1665240375-3545936-2-11630560560653337076
X-Sieve: CMU Sieve 3.0
Received: from mail.<domain>.com (unknown [<pmg-server-ip>])
    by gws.<domain>.com (Postfix) with ESMTPS id E6294100FC4
    for <<receiver>@<domain>.com>; Sat,  8 Oct 2022 16:46:15 +0200 (CEST)
Received: from <pmg-server>.<domain>.com (localhost.localdomain [127.0.0.1])
    by mail.<domain>.com (Proxmox) with ESMTP id D3693205DDE
    for <<receiver>@<domain>.com>; Sat,  8 Oct 2022 16:46:15 +0200 (CEST)
Received-SPF: permerror (mail.brainc13.today: Junk encountered in mechanism 'ip4:104.148.0.124~all') receiver=<pmg-server>.<domain>.com; identity=mailfrom; envelope-from="#-#-#-#-<receiver>=<domain>.com@mail.brainc13.today"; helo=relay.<domain>.com; client-ip=<mail-relay-ip>
Received: from relay.<domain>.com (<mail-relay> [<mail-relay-ip>])
    by mail.<domain>.com (Proxmox) with ESMTP id C46B4205DCF
    for <<receiver>@<domain>.com>; Sat,  8 Oct 2022 16:46:14 +0200 (CEST)
Received-SPF: Permerror (mailfrom) identity=mailfrom; client-ip=104.148.0.124; helo=simi.brainc13.today; envelope-from=56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today; receiver=<UNKNOWN>
Received: from simi.brainc13.today (unknown [104.148.0.124])
    by relay.<domain>.com (Postfix) with ESMTP id 205453003F0
    for <<receiver>@<domain>.com>; Sat,  8 Oct 2022 16:38:17 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=k1; d=brainc13.today;
 h=Mime-Version:Content-Type:Date:From:Reply-To:Subject:To:Message-ID; i=FadingMemory@brainc13.today;
 bh=Z6l173P/PkM5KfsgNGu+V2StvaM=;
 b=ld1K3FKGCBoq5LKoHh6F7/Kad4jK9N1rU9JWIn4MEcwfErbu0hErPtnhRQYh+zSGls+d6TDViR3O
   aNkCjB15U8i7uKlfed9OD/k4vOEXdlimeIcQI32J2MgjOAamQh4m3+gh6ZPvD8/FQqmD39cDS16j
   hG1wTiCv9uMVF7ztX50=
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=k1; d=brainc13.today;
 b=USs8SlbrnAyjcAQRAwTLVsXH6f5kTfb4YzRC4HESrX7kXpRsoTG9XZi2mFUuePhcAx3HEg7UcfHE
   /mBc+rP2QyNt9EhodWwVDMxnuKJtwlST6k2/i/T8a5QwcF9y3EuotOG1m8oMklRXQeOg62z3BxVx
   bmzq+KQfNyNDyf83eQs=;
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary="f7ad85702c49dc09aa105346b0a8c9e0_16649_20a4c"
Date: Sat, 8 Oct 2022 10:38:07 -0400
From: "Memory Problems" <FadingMemory@brainc13.today>
Reply-To: "Fading Memory" <MemoryProblems@brainc13.today>
Subject: Fading memory? Eat THIS 'Mental Vitamin' tonight
To: <<receiver>@<domain>.com>
Message-ID: <rt4yacj781bgvvkt-5pbwzksya9b0nuxs-16649-20a4c@brainc13.today>
X-SPAM-LEVEL: Spam detection results:  1
    BAYES_50                  0.8 Bayes spam probability is 40 to 60%
    DKIM_INVALID              0.1 DKIM or DK signature exists, but is not valid
    DKIM_SIGNED               0.1 Message has a DKIM or DK signature, not necessarily valid
    HTML_FONT_LOW_CONTRAST  0.001 HTML font color similar or identical to background
    HTML_MESSAGE            0.001 HTML included in message
    KAM_DMARC_NONE           0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy
    KAM_DMARC_STATUS         0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
    LONGLN_LOW_CONTRAST     1.596 Excessively long line + hidden text
    RAZOR2_CF_RANGE_51_100  1.886 Razor2 gives confidence level above 50%
    RAZOR2_CHECK            0.922 Listed in Razor2 (http://razor.sf.net/)
    RCVD_IN_DNSWL_HI           -5 Sender listed at https://www.dnswl.org/, high trust
    SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
    T_SPF_PERMERROR          0.01 SPF: test of record failed (permerror)
    URIBL_ABUSE_SURBL        1.25 Contains an URL listed in the ABUSE SURBL blocklist [brainc13.today]
    URIBL_BLOCKED           0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked.  See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [brainc13.today]

<start of message>
...

I could not find anything strange in the pmg service journals.

Stoiko Ivanov · Oct 10, 2022

Mr_Diba said:
oth GraylistOther and GraylistTLD should go to the Quarantine, but reading your reply I might misunderstand how multiple "From" objects operate together. I assume that multiple "From" objects are just a logical "OR"?

No - you understood it correctly - I just misread the pmgdb dump (thought it's 2 rules) output - sorry for the confusion caused!

Mr_Diba said:
<56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today>, size=8280, nrcpt=1 (queue active)

hm - since you anonymized the output - does the actual log also contain '<' and '>' characters as such for receiver and domain?
e.g is it:

Code:

56810-91721-133708-13739-<s.ivanov>=<proxmox.com>.com@mail.brainc13.today>

or

Code:

56810-91721-133708-13739-s.ivanov=proxmox.com.com@mail.brainc13.today>

Mr_Diba · Oct 11, 2022

Stoiko Ivanov said:
No - you understood it correctly - I just misread the pmgdb dump (thought it's 2 rules) output - sorry for the confusion caused!

hm - since you anonymized the output - does the actual log also contain '<' and '>' characters as such for receiver and domain?
e.g is it:

Code:

56810-91721-133708-13739-<s.ivanov>=<proxmox.com>.com@mail.brainc13.today>

or

Code:

56810-91721-133708-13739-s.ivanov=proxmox.com.com@mail.brainc13.today>

All the anonymization has been done using "<something>", so:

Code:

envelope-from=56810-91721-133708-13739-<receiver>=<domain>.com@mail.brainc13.today

equals

Code:

envelope-from=56810-91721-133708-13739-s.ivanov=proxmox.com@mail.brainc13.today

and

Code:

for <<receiver>@<domain>.com>; Sat,  8 Oct 2022 16:46:14 +0200 (CEST)

equals

Code:

for <s.ivanov@proxmox.com>; Sat,  8 Oct 2022 16:46:14 +0200 (CEST)

Stoiko Ivanov · Oct 11, 2022

thanks - still could not reproduce it - also created a who object with 1500 domain entries (automated - so it looks like '.*\.tb863day' - but also '.*\.today' )
sent a mail with forged smtp-sender of : "56810-91721-133708-13739-s.ivanov=proxmox.com@mail.testthisnonexitent.today"
-> it got put into quarantine by the rule with 2 from objects (one of them is the one described above, the other did not match the mail)

so I'm a bit unsure where the issue is actually rooted - maybe check for spaces or other glitches in the .today regex...

Mr_Diba · Oct 12, 2022

I might have found something:

To get all of the 1300 entries in PMG I created a script that creates a SQL statement from a list of TLDs. The output of that script for ".*\.today"is as follow:

Code:

INSERT INTO object(objecttype,objectgroup_id,value) VALUES('1000','29','\x2e2a5c2e746f6461790a');

When I take a look in the database I have indeed the following entry:

Code:

Proxmox_ruledb=> select * from object where id = '3997';
  id  | objecttype | objectgroup_id |         value          
------+------------+----------------+------------------------
 3997 |       1000 |             29 | \x2e2a5c2e746f6461790a
(1 row)

(I do not know if a restart of the service is necessary, but I did a full reboot to make sure)

When I enter the same ".*\.today" in the GUI and read the database it has the following entry (Ignore the group_id):

Code:

Proxmox_ruledb=> select * from object where id = '4153';
  id  | objecttype | objectgroup_id |        value         
------+------------+----------------+----------------------
 4153 |       1000 |             33 | \x2e2a5c2e746f646179
(1 row)

The difference being a hex 0a or a "LF" in ASCII. Which could be the problem.

It does explain the output of the pmgdb dump before I sanitized it:

Code:

Found RULE 15 (prio: 80, in, active): Graylist
  FOUND FROM GROUP 29: GraylistTLD
    OBJECT 2814: .*\.aarp

    OBJECT 2815: .*\.abarth

    OBJECT 2816: .*\.abb

    OBJECT 2817: .*\.abbott

    OBJECT 2818: .*\.abbvie

    OBJECT 2819: .*\.abc
...

Funny thing is that the GUI is not aware of the LF and passing the string test on the imported TLD without any problem using your forged smtp-sender domain.

Stoiko Ivanov · Oct 12, 2022

Mr_Diba said:
To get all of the 1300 entries in PMG I created a script that creates a SQL statement from a list of TLDs. The output of that script for ".*\.today"is as follow

ok - that sounds somewhat like a likely cause.
I'd suggest using the REST-API for such things - or if you prefer pmgsh - then you get the validation that is done by the backend:
https://pmg.proxmox.com/pmg-docs/api-viewer/index.html

Mr_Diba said:
It does explain the output of the pmgdb dump before I sanitized it:

yes that explains it

I hope this helps!

Mr_Diba · Oct 12, 2022

I can confirm the "LF" in the database was indeed the issue. I changed all the TLD entries with an entry without "LF" and now I see the mails getting to the Quarantine.

Code:

Oct 12 11:08:51 <pmg-server> pmg-smtp-filter[1313598]: 3027626346842278187: moved mail for <<receiver>@<domain>.com> to spam quarantine - 92F16346842397D72 (rule: Graylist)

Thank you for all the support and a great tool. I will try to use the API in the future when possible

Search

Search

[SOLVED] Maximum amount of who-objects?

Mr_Diba

New Member

Stoiko Ivanov

Proxmox Staff Member

Mr_Diba

New Member

Stoiko Ivanov

Proxmox Staff Member

Mr_Diba

New Member

Stoiko Ivanov

Proxmox Staff Member

Mr_Diba

New Member

Stoiko Ivanov

Proxmox Staff Member

Mr_Diba

New Member