Retention in tracking center

Apr 9, 2018
15
0
21
46
Hi, by law I have to keep maillogs for 2 years. Can I set a retention time in proxmox? It would be ideal if lookup in the tracking center would go that far back
 
Not really, for such a use case you need to implement a powerful log server where you can store this huge amount of logs.
 
Not really, for such a use case you need to implement a powerful log server where you can store this huge amount of logs.

Actually, everyone in Belgium using proxmail would be obligated to do that, a lot smaller providers like myself probably ignore it though.
Still, the current retention in the tracking center is not that long, I started the gateway on april 10 and the first log entry I can still access is april 26...

Guy
 
The tracking center is a front-end to the syslog and the syslog files are available for 7 days, just standard Debian Linux.

If you want to store gigabytes of logs from every day for 2 years, you need a huge server. You will find a lot of tools exactly for this task.
 
So, for my own comfort, I'd like a retention of at least a month in the message center. If I change the logrotation, it will be visible in tracking center?

Secondly, I will probably push the logs to an elasticsearch server to (unfortunatly) comply with draconian law.
 
So, for my own comfort, I'd like a retention of at least a month in the message center. If I change the logrotation, it will be visible in tracking center?
...

yes, if you syslogs are availabe for 30 days, you can also query them via the message tracking (check Debian howto for changing the log rotation)
 
yes, if you syslogs are availabe for 30 days, you can also query them via the message tracking (check Debian howto for changing the log rotation)
Exactly. Increase rotate count at /etc/logrotate.d/rsyslog on /var/log/syslog {}
 
The tracking center is a front-end to the syslog and the syslog files are available for 7 days, just standard Debian Linux.

If you want to store gigabytes of logs from every day for 2 years, you need a huge server. You will find a lot of tools exactly for this task.
Sorry for the noob question, but how can I increase the limit?
I would like to keep at least one month.
 
As posted above: Increase rotate count at /etc/logrotate.d/rsyslog on /var/log/syslog

However, ensure that longer logging comply as well with GDPR
Do I have to restart sth later?
And if I will update my PMG, do I have to change again that setting? Can I do sth like a custom file?

ps: Damn, I hate GDPR :/
 
GDPR doesn't say nothing about logging. But could be a law specifically in your country, and in this case you must be follow.

Yes and no, GDPR doesn’t say anything about logging but says many things about processing personal data and they are also processed in logs. GDPR then also says many things about storage limitation. So you can‘t read anything directly but the data protection supervisory bodies like to reduce the log files storage, if containing personal data and it does: sender, recipient, ...
 
Can someone give some guidance what are the correct settings if we want to keep logs for 1 month, 3 months, 6 months, 1 year? We have the storage space how much log can we realistically store on proxmox mail gateway before it breaks?

Looking at /etc/logrotate.d/rsyslog I am guessing we have to increase this value rotate 8??
What are the best values to keep the tracking center search speed as fast as possible? Should we rotate daily, monthly instead?

/etc/logrotate.d/rsyslog
Code:
/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 8
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
}
 
Last edited:
Can someone please respond to my message above? How we should configure this and what is the maximum recommended value until pmg breaks? Thank you!
 
You need to do for /var/log/syslog similar to messages below. You can use rotate value to increase and/or weekly attribute to different attribute like monthly, yearly, ...

However, it depend on your storage capability on one hand and your memory as well as your cpu performance for the tracking center on the other hand. You may expect, as larger the log file get, as longer tracking center will take. I believe, there is no limit at all, however tracking center isn't meant for huge logs, so other logging processing tools would be much more performant and you should store and process logs there. It also depend on what you're using the server for. I use PMG on my private environment (for commercial my IT staff is now taking over the job), so it's very low volume, I could also store the logs for a year without any issues, but if you're providing mail services for a company with thousands of users and huge amount of spam, you may get to your resource limits just by a logging amount of a week or so.
 
Can someone give some guidance what are the correct settings if we want to keep logs for 1 month, 3 months, 6 months, 1 year? We have the storage space how much log can we realistically store on proxmox mail gateway before it breaks?

Looking at /etc/logrotate.d/rsyslog I am guessing we have to increase this value rotate 8??
What are the best values to keep the tracking center search speed as fast as possible? Should we rotate daily, monthly instead?

/etc/logrotate.d/rsyslog
Code:
/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 8
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
}
To keep logs for 1 month, 3 months, 6 months, and 1 year, you can change the "rotate" value to the number of logs you want to keep. For example, if you want to keep logs for 1 month, you can set the "rotate" value to 4 (4 weeks in a month), and for 1 year, you can set it to 52 (52 weeks in a year).
It's difficult to say how much log you can store on the proxmox mail gateway without it breaking, as it depends on several factors such as the size of the logs, the available storage space, and the performance of the server. It's best to monitor the storage usage and adjust the log retention accordingly.
As for the best values to keep the tracking center search speed as fast as possible, it's generally recommended to rotate the logs daily or weekly. This helps to keep the log files small, which can improve the search speed. However, it's also important to consider the log retention requirements and adjust the rotation frequency accordingly.
In the /etc/logrotate.d/rsyslog configuration file, you can adjust the "rotate" value and the "weekly" frequency to meet your log retention and performance requirements. It's also recommended to keep the other settings such as "missingok", "notifempty", and "compress" to ensure the logs are rotated and compressed properly.
 
  • Like
Reactions: poetry
To keep logs for 1 month, 3 months, 6 months, and 1 year, you can change the "rotate" value to the number of logs you want to keep. For example, if you want to keep logs for 1 month, you can set the "rotate" value to 4 (4 weeks in a month), and for 1 year, you can set it to 52 (52 weeks in a year).
It's difficult to say how much log you can store on the proxmox mail gateway without it breaking, as it depends on several factors such as the size of the logs, the available storage space, and the performance of the server. It's best to monitor the storage usage and adjust the log retention accordingly.
As for the best values to keep the tracking center search speed as fast as possible, it's generally recommended to rotate the logs daily or weekly. This helps to keep the log files small, which can improve the search speed. However, it's also important to consider the log retention requirements and adjust the rotation frequency accordingly.
In the /etc/logrotate.d/rsyslog configuration file, you can adjust the "rotate" value and the "weekly" frequency to meet your log retention and performance requirements. It's also recommended to keep the other settings such as "missingok", "notifempty", and "compress" to ensure the logs are rotated and compressed properly.

Hello @rason thanks for the comment. This makes some sense but I am still unclear what to do. For example I want to have more logs available let's say for start for 16 weeks and searching to be as fast as possible.

If I set rotate value to 84 and daily what would that mean? My guess is this means 84 log files and rotated daily. Does this mean I can only search log for 1 day interval at the time? Will the search speed be faster then weekly rotation?
If I set rotate value to 8 and weekly what would that mean? My guess is this means 8 week log files and rotated weekly. Does this means I can only search logs for 1 week interval at the time? Will the search speed be slower or faster?
 
Last edited:
Currently the implementation has a few things hard-coded (mostly due to speed improvments).
The logfiles considered are /var/log/syslog, /var/log/syslog.1, /var/log/syslog[2-31].gz - see:
https://git.proxmox.com/?p=pmg-log-...62995ed065b49d5039379bc485405aa;hb=HEAD#l2157

This means the maximal `rotate` value that makes sense is 32

speedwise I think it's a tradeoff between CPU speed, vs. disk-speed (if you keep weekly logs around each log will be ~7 times as large as for daily (more diskreads), if you rotate more often more logs will need to get gunzipped (more cpu-cycles)

pmg-log-tracker is built to only look at the beginning of the files and to only go through the ones that cover the provided timeframe - so I think the difference should not be too noticeable (if you have any concrete numbers in one direction or the other it'd be still very interesting to know)

all logfiles that fit the pattern from above are considered.

there are a few enhancement requests to make this all a bit more flexible (without sacrificing performance) e.g.:
https://bugzilla.proxmox.com/show_bug.cgi?id=3657

I hope this explains it!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!