How to log subjects of all incoming email?

Miktash

Active Member
Mar 6, 2015
67
1
28
How do I log all subjects of incoming email?

It doesn't matter if I can't see them in the gui but I need the log in case I need to debug things.
 
Anyone has an idea?

I could probably reconfigure postfix to do this but isn't that going to cause problems with pmg-log-tracker or any other part of the system?
 
Never mind. I modified pmg-smtp-filter to log both the From and Subject and will keep track of my changes so I can re-do this after updates.
It's logging perfectly fine now and it's shown in the tracking center too. Subject is RFC1522 decoded too.

Nice ;)
 
Hi,

can you provide, how you did. Looking at my Advancing PMG thread, you can see, that I did it with postfix header checks using the template copy mechanism. As I didn't change the PMG internal files and I used the official template adjustment, it's update-safe (just need to check changes to the templates from time to time). However, UTF8-encoded headers are not decoded, as I did not found a solution on how to do that with postfix. For sure would be great, if PMG would directly also show the subjects and envelope senders of the message, you may create a feature request for that on the PMG bugzilla. You also may consider to contribute your adjustments to PMG development mailing list, maybe it will be integrated in future updates (so you aren't require to redo).

As you were able to adjust pmg-smtp-filter, I still have some adjustment looking for help on pmg-log-tracker to be able to well log my milter-rejects, I have just an adjustment to show them at all, but they are missing the additional lines with from, subject, to etc.

Regards,
Christian
 
Well, /usr/bin/pmg-smtp-filter is a python script which you can edit with your favorite editor.

What i changed:

First, add this somewhere between the 'use' statements at the beginning of the file:

use PVE::Tools;

Then search for this line:

$self->log (3, "$queue->{logid}: new mail message-id=%s", $queue->{msgid});

Add the following below that line:

$self->log (3, "$queue->{logid}: Subject: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($entity->head->get('subject'))) || 'No Subject');
my @fromarray = split('\s*,\s*', $entity->head->get('from') || $msginfo->{sender});
$self->log (3, "$queue->{logid}: From: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($fromarray[0])) );


This will log both the Subject and the From header from the mail.

pmg-log-tracker just works with that. I didn't have to adjust it.
 
  • Like
Reactions: Kenny Huynh
I haven't looked at the pmg-log-tracker code yet but might need to do that later if I make more changes.
If I'm going to read into that code I'll get back to you.
 
I haven't looked at the pmg-log-tracker code yet but might need to do that later if I make more changes.
If I'm going to read into that code I'll get back to you.

Great, many thanks. And please consider to provide your adjustments to integrate subjects into PMG to contribute to Proxmox Development Mailing list, so it may be integrated into the "official" code. I recently also saw again a wishlist to have this information inside of PMG as well as the envelope sender (as addition to the the header sender) as this may differ because of using ESPs like Amazon SES, Mailchimp and others or be technical different because of bounce check addresses.
 
Maybe you will need to think also on the regulation/law that are in place for your country/region. In my own opinion to log any subject for any mail a bit more intrusive. Any mail is a privet communication if at least one side is a human. Think for a moment, that sameone as a person receive a mail with subject "you have X illness ". And you store this info or you use this info, because you can. At least in EU you can not do it legal because you do not work let say in madical/health field. But you do it because you can. So to be complied with EU GDPR you will need (I am not a expert in this field) to declare that you store medical / helth information because you want to do this.

So maybe I am wrong, but I suggest to speak with a lawyer about this.

And as a personal ideea, I have not think that if I have o problem to solve about mails, I must invade the privacy of my clients because I can do it !

Many managers ask me to show the mail from/to X . I say for any request that I can not do this. The subject of mails could be the first stpep.

For sure my response is not objective, because I liveing in a former communist country, where is was very common that our mail envelop that you send to any country to be inspected and rediding by communists employees. At the first step they read the receive address from the envelop (like subject for mails),

So the question is you must do it? I am pretty sure that are many proxmox users from countries where this kind of feature will be bery useful.... ;) But in the end we must think if we want to destroy the life of such unlucky persons.

And this is not a supposition from my point of view. I see this with my own eyes a long time ago. But I care about this ... so I dare to say all of this.

Do you care? This is the question!

Sorry to all if I say bad things, it is not my intention at all.
 
Maybe you will need to think also on the regulation/law that are in place for your country/region. In my own opinion to log any subject for any mail a bit more intrusive. Any mail is a privet communication if at least one side is a human. Think for a moment, that sameone as a person receive a mail with subject "you have X illness ". And you store this info or you use this info, because you can. At least in EU you can not do it legal because you do not work let say in madical/health field. But you do it because you can. So to be complied with EU GDPR you will need (I am not a expert in this field) to declare that you store medical / helth information because you want to do this.

So maybe I am wrong, but I suggest to speak with a lawyer about this.

And as a personal ideea, I have not think that if I have o problem to solve about mails, I must invade the privacy of my clients because I can do it !

Many managers ask me to show the mail from/to X . I say for any request that I can not do this. The subject of mails could be the first stpep.

For sure my response is not objective, because I liveing in a former communist country, where is was very common that our mail envelop that you send to any country to be inspected and rediding by communists employees. At the first step they read the receive address from the envelop (like subject for mails),

So the question is you must do it? I am pretty sure that are many proxmox users from countries where this kind of feature will be bery useful.... ;) But in the end we must think if we want to destroy the life of such unlucky persons.

And this is not a supposition from my point of view. I see this with my own eyes a long time ago. But I care about this ... so I dare to say all of this.

Do you care? This is the question!

Sorry to all if I say bad things, it is not my intention at all.

Sure, as Privacy Professional I‘m aware of Privacy. So this option should be available in three options: 1. disable (default), 2. enable, 3. 4-eyes-principle (requires two login phrases to access information), that may also include senders and recipients (per option). Being able to access the sender and recipient as well is a privacy issue. So depends on setup, if it’s a company with no private usage allowed, you can also check this details and they are helpful for decision on spam or no-spam. Administrators always have deeper access to perform their job, however, need to be well defined and ensured, that access is not misused.
 
Maybe it's an better solution to log only the subject if the email is blocked or Quarantined? I think there is no need to store such Infos, if the mail was correctly forwarded to the recipient.

SpamExperts logged all information from a mail, but you are only able to see the whole mail if it is blocked, otherwise you can't see the whole mail.
 
Maybe it's an better solution to log only the subject if the email is blocked or Quarantined? I think there is no need to store such Infos, if the mail was correctly forwarded to the recipient.

SpamExperts logged all information from a mail, but you are only able to see the whole mail if it is blocked, otherwise you can't see the whole mail.

Don't need to see the whole mail but subject helps a lot. For sure, if blocked (illegal in Germany and should not be used) or quarantined (better for me "spam tagged") would be more important, but if I check new blacklists, which just add notify lines, if I just tag messages as possible spam and check the spam assassin scores, I would also like to check the envelope from, to and subject to see, if this mail is legit or spam, fake, ... and that's impossible without the subject, just the combination of all three helps out. So there are too much conditions to be required to see this information than vice versa and it's hard to introduce a filter of subject information suppression on such rare conditions than have the three options as being privacy complaint.
 
Well, /usr/bin/pmg-smtp-filter is a python script which you can edit with your favorite editor.

What i changed:

First, add this somewhere between the 'use' statements at the beginning of the file:

use PVE::Tools;

Then search for this line:

$self->log (3, "$queue->{logid}: new mail message-id=%s", $queue->{msgid});

Add the following below that line:

$self->log (3, "$queue->{logid}: Subject: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($entity->head->get('subject'))) || 'No Subject');
my @fromarray = split('\s*,\s*', $entity->head->get('from') || $msginfo->{sender});
$self->log (3, "$queue->{logid}: From: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($fromarray[0])) );


This will log both the Subject and the From header from the mail.

pmg-log-tracker just works with that. I didn't have to adjust it.

Hello,

Thanks for your script. It works great.

Best Regards,
 
Hi, is the script work for 6.1.2? I am looking the the same, applied but seems no luck.
 
  • Like
Reactions: kez
Well, /usr/bin/pmg-smtp-filter is a python script which you can edit with your favorite editor.

What i changed:

First, add this somewhere between the 'use' statements at the beginning of the file:

use PVE::Tools;

Then search for this line:

$self->log (3, "$queue->{logid}: new mail message-id=%s", $queue->{msgid});

Add the following below that line:

$self->log (3, "$queue->{logid}: Subject: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($entity->head->get('subject'))) || 'No Subject');
my @fromarray = split('\s*,\s*', $entity->head->get('from') || $msginfo->{sender});
$self->log (3, "$queue->{logid}: From: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($fromarray[0])) );


This will log both the Subject and the From header from the mail.

pmg-log-tracker just works with that. I didn't have to adjust it.
Well, /usr/bin/pmg-smtp-filter is a python script which you can edit with your favorite editor.

What i changed:

First, add this somewhere between the 'use' statements at the beginning of the file:

use PVE::Tools;

Then search for this line:

$self->log (3, "$queue->{logid}: new mail message-id=%s", $queue->{msgid});

Add the following below that line:

$self->log (3, "$queue->{logid}: Subject: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($entity->head->get('subject'))) || 'No Subject');
my @fromarray = split('\s*,\s*', $entity->head->get('from') || $msginfo->{sender});
$self->log (3, "$queue->{logid}: From: %s", PMG::Utils::decode_rfc1522(PVE::Tools::trim($fromarray[0])) );


This will log both the Subject and the From header from the mail.

pmg-log-tracker just works with that. I didn't have to adjust it.

Hello ,
Is this script is working with Ver 6.2.
I did the following with no lack.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!