Hi, by law I have to keep maillogs for 2 years. Can I set a retention time in proxmox? It would be ideal if lookup in the tracking center would go that far back
Retention in tracking center
- Thread starter taurix
- Start date
Actually, everyone in Belgium using proxmail would be obligated to do that, a lot smaller providers like myself probably ignore it though.
Still, the current retention in the tracking center is not that long, I started the gateway on april 10 and the first log entry I can still access is april 26...
Guy
The tracking center is a front-end to the syslog and the syslog files are available for 7 days, just standard Debian Linux.
If you want to store gigabytes of logs from every day for 2 years, you need a huge server. You will find a lot of tools exactly for this task.
If you want to store gigabytes of logs from every day for 2 years, you need a huge server. You will find a lot of tools exactly for this task.
So, for my own comfort, I'd like a retention of at least a month in the message center. If I change the logrotation, it will be visible in tracking center?
Secondly, I will probably push the logs to an elasticsearch server to (unfortunatly) comply with draconian law.
Secondly, I will probably push the logs to an elasticsearch server to (unfortunatly) comply with draconian law.
yes, if you syslogs are availabe for 30 days, you can also query them via the message tracking (check Debian howto for changing the log rotation)
As posted above: Increase rotate count at /etc/logrotate.d/rsyslog on /var/log/syslog
However, ensure that longer logging comply as well with GDPR
Do I have to restart sth later?
And if I will update my PMG, do I have to change again that setting? Can I do sth like a custom file?
ps: Damn, I hate GDPR :/
This setting seems to be outside the template system. I also won’t believe, you need to restart anything to get it running.
GDPR doesn't say nothing about logging. But could be a law specifically in your country, and in this case you must be follow.
Yes and no, GDPR doesn’t say anything about logging but says many things about processing personal data and they are also processed in logs. GDPR then also says many things about storage limitation. So you can‘t read anything directly but the data protection supervisory bodies like to reduce the log files storage, if containing personal data and it does: sender, recipient, ...
Can someone give some guidance what are the correct settings if we want to keep logs for 1 month, 3 months, 6 months, 1 year? We have the storage space how much log can we realistically store on proxmox mail gateway before it breaks?
Looking at /etc/logrotate.d/rsyslog I am guessing we have to increase this value rotate 8??
What are the best values to keep the tracking center search speed as fast as possible? Should we rotate daily, monthly instead?
/etc/logrotate.d/rsyslog
Looking at /etc/logrotate.d/rsyslog I am guessing we have to increase this value rotate 8??
What are the best values to keep the tracking center search speed as fast as possible? Should we rotate daily, monthly instead?
/etc/logrotate.d/rsyslog
Code:
/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 8
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
Last edited:
You need to do for /var/log/syslog similar to messages below. You can use rotate value to increase and/or weekly attribute to different attribute like monthly, yearly, ...
However, it depend on your storage capability on one hand and your memory as well as your cpu performance for the tracking center on the other hand. You may expect, as larger the log file get, as longer tracking center will take. I believe, there is no limit at all, however tracking center isn't meant for huge logs, so other logging processing tools would be much more performant and you should store and process logs there. It also depend on what you're using the server for. I use PMG on my private environment (for commercial my IT staff is now taking over the job), so it's very low volume, I could also store the logs for a year without any issues, but if you're providing mail services for a company with thousands of users and huge amount of spam, you may get to your resource limits just by a logging amount of a week or so.
However, it depend on your storage capability on one hand and your memory as well as your cpu performance for the tracking center on the other hand. You may expect, as larger the log file get, as longer tracking center will take. I believe, there is no limit at all, however tracking center isn't meant for huge logs, so other logging processing tools would be much more performant and you should store and process logs there. It also depend on what you're using the server for. I use PMG on my private environment (for commercial my IT staff is now taking over the job), so it's very low volume, I could also store the logs for a year without any issues, but if you're providing mail services for a company with thousands of users and huge amount of spam, you may get to your resource limits just by a logging amount of a week or so.
To keep logs for 1 month, 3 months, 6 months, and 1 year, you can change the "rotate" value to the number of logs you want to keep. For example, if you want to keep logs for 1 month, you can set the "rotate" value to 4 (4 weeks in a month), and for 1 year, you can set it to 52 (52 weeks in a year).
It's difficult to say how much log you can store on the proxmox mail gateway without it breaking, as it depends on several factors such as the size of the logs, the available storage space, and the performance of the server. It's best to monitor the storage usage and adjust the log retention accordingly.
As for the best values to keep the tracking center search speed as fast as possible, it's generally recommended to rotate the logs daily or weekly. This helps to keep the log files small, which can improve the search speed. However, it's also important to consider the log retention requirements and adjust the rotation frequency accordingly.
In the /etc/logrotate.d/rsyslog configuration file, you can adjust the "rotate" value and the "weekly" frequency to meet your log retention and performance requirements. It's also recommended to keep the other settings such as "missingok", "notifempty", and "compress" to ensure the logs are rotated and compressed properly.
Hello @rason thanks for the comment. This makes some sense but I am still unclear what to do. For example I want to have more logs available let's say for start for 16 weeks and searching to be as fast as possible.
If I set rotate value to 84 and daily what would that mean? My guess is this means 84 log files and rotated daily. Does this mean I can only search log for 1 day interval at the time? Will the search speed be faster then weekly rotation?
If I set rotate value to 8 and weekly what would that mean? My guess is this means 8 week log files and rotated weekly. Does this means I can only search logs for 1 week interval at the time? Will the search speed be slower or faster?
Last edited:
Currently the implementation has a few things hard-coded (mostly due to speed improvments).
The logfiles considered are /var/log/syslog, /var/log/syslog.1, /var/log/syslog[2-31].gz - see:
https://git.proxmox.com/?p=pmg-log-...62995ed065b49d5039379bc485405aa;hb=HEAD#l2157
This means the maximal `rotate` value that makes sense is 32
speedwise I think it's a tradeoff between CPU speed, vs. disk-speed (if you keep weekly logs around each log will be ~7 times as large as for daily (more diskreads), if you rotate more often more logs will need to get gunzipped (more cpu-cycles)
pmg-log-tracker is built to only look at the beginning of the files and to only go through the ones that cover the provided timeframe - so I think the difference should not be too noticeable (if you have any concrete numbers in one direction or the other it'd be still very interesting to know)
all logfiles that fit the pattern from above are considered.
there are a few enhancement requests to make this all a bit more flexible (without sacrificing performance) e.g.:
https://bugzilla.proxmox.com/show_bug.cgi?id=3657
I hope this explains it!
The logfiles considered are /var/log/syslog, /var/log/syslog.1, /var/log/syslog[2-31].gz - see:
https://git.proxmox.com/?p=pmg-log-...62995ed065b49d5039379bc485405aa;hb=HEAD#l2157
This means the maximal `rotate` value that makes sense is 32
speedwise I think it's a tradeoff between CPU speed, vs. disk-speed (if you keep weekly logs around each log will be ~7 times as large as for daily (more diskreads), if you rotate more often more logs will need to get gunzipped (more cpu-cycles)
pmg-log-tracker is built to only look at the beginning of the files and to only go through the ones that cover the provided timeframe - so I think the difference should not be too noticeable (if you have any concrete numbers in one direction or the other it'd be still very interesting to know)
all logfiles that fit the pattern from above are considered.
there are a few enhancement requests to make this all a bit more flexible (without sacrificing performance) e.g.:
https://bugzilla.proxmox.com/show_bug.cgi?id=3657
I hope this explains it!