[SOLVED] Proxmox-backup logs

[franko5]

Hi,

I just noticed that /var/log/proxmox-backup is using quite a lot of space.
It seems as log rotation is not explicitly defined in logrotate.
Is there any other setting that is taking care of this log rotation or can I just use logrotate as for any other logs?

I didn’t find anything related in documentation, so probably rotation can be defined in logrotate as for any other service?

Thanks in advance.

 

[dcsapak]

we recently implmented a cleanup of old task logs https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=0e1edf19b1c3f4e68e1340b340a6c655846ce095
it's not currently packaged, so it may take a bit until you can update to it.

if you really cannot wait, you can manually delete task logs e.g. older than 90 days with

find /var/log/proxmox-backup/tasks -type f  -name 'UPID*' -mtime +90 -delete

(replace the '90' with a value you find appropriate)

if you do not give the '-delete' parameter it gives you a list of files that matched

note that it will no longer work to view those task logs in the webui if you delete them

just out of curiosity, how much space is used? (you can check e.g. with 'du -sh /var/log/proxmox-backup/*')

 

[franko5]

Hi @dcsapak ,

Thanks for the answer, I will implement this cleaning as you suggested until the feature is available in update.
I guess I can clean the /var/log/proxmox-backup/api as well, actually, better to use logrotate here as the logs are moved daily to zst compressed access.log and auth.log?

The used space of proxmox-backup logs:

$ du -sh /var/log/proxmox-backup/*
4.8G    /var/log/proxmox-backup/api
4.5G    /var/log/proxmox-backup/tasks

Server uses about 22 Tb of backup data.

 

[dcsapak]

I guess I can clean the /var/log/proxmox-backup/api as well, actually, better to use logrotate here as the logs are moved daily to zst compressed access.log and auth.log?

we do not use logrotate but use our own logrotation (since we can then use zstd which should be better than gzip) in the proxmox-backup-proxy (which is not configurable at the moment, though you can open a feature request here if you want: https://bugzilla.proxmox.com)
deleting old logs manually should not be a problem though

 

[dcsapak]

also, can you show the content of /var/log/proxmox-backup/api ? it may be that the daemon did not rotate because of a bug (fixed in git but also not yet rolled out). that could be fixed by restarting proxmox-backup-proxy (systemctl restart proxmox-backup-proxy) but beware, that will abort running tasks (backup/restore/etc)

 

[franko5]

Sure, this was output (I removed archived one manually):

total 4.6G
-rw-r--r-- 1 backup backup 4.3G Oct 19 10:26 access.log.1
-rw-r--r-- 1 backup backup 2.6M Jun  5 00:00 access.log.10.zst
-rw-r--r-- 1 backup backup 2.7M Jun  4 00:00 access.log.11.zst
-rw-r--r-- 1 backup backup 2.6M Jun  3 00:00 access.log.12.zst
-rw-r--r-- 1 backup backup 1.7M Jun  2 00:00 access.log.13.zst
-rw-r--r-- 1 backup backup  34M Aug 21 00:00 access.log.2.zst
-rw-r--r-- 1 backup backup 166M Aug 11 00:00 access.log.3.zst
-rw-r--r-- 1 backup backup 3.4M Jun 11 00:00 access.log.4.zst
-rw-r--r-- 1 backup backup 2.7M Jun 10 00:00 access.log.5.zst
-rw-r--r-- 1 backup backup 2.7M Jun  9 00:00 access.log.6.zst
-rw-r--r-- 1 backup backup 2.7M Jun  8 00:00 access.log.7.zst
-rw-r--r-- 1 backup backup 2.6M Jun  7 00:00 access.log.8.zst
-rw-r--r-- 1 backup backup 2.6M Jun  6 00:00 access.log.9.zst
-rw-r--r-- 1 backup backup 7.2M Oct 19 10:26 auth.log
-rw-r--r-- 1 backup backup  33M Oct 18 23:59 auth.log.1
-rw-r--r-- 1 backup backup 286K Oct  3 00:00 auth.log.10.zst
-rw-r--r-- 1 backup backup 284K Oct  1 00:00 auth.log.11.zst
-rw-r--r-- 1 backup backup 309K Sep 29 00:00 auth.log.12.zst
-rw-r--r-- 1 backup backup 286K Sep 27 00:00 auth.log.13.zst
-rw-r--r-- 1 backup backup 305K Oct 19 00:00 auth.log.2.zst
-rw-r--r-- 1 backup backup 285K Oct 17 00:00 auth.log.3.zst
-rw-r--r-- 1 backup backup 300K Oct 15 00:00 auth.log.4.zst
-rw-r--r-- 1 backup backup 311K Oct 13 00:00 auth.log.5.zst
-rw-r--r-- 1 backup backup 289K Oct 11 00:00 auth.log.6.zst
-rw-r--r-- 1 backup backup 295K Oct  9 00:00 auth.log.7.zst
-rw-r--r-- 1 backup backup 261K Oct  7 00:00 auth.log.8.zst
-rw-r--r-- 1 backup backup 268K Oct  5 00:00 auth.log.9.zst

Well, seems that restart of proxmox-backup-proxy did some rotate as I had this:

ll api/
total 4.6G
-rw-r--r-- 1 backup backup 4.6G Oct 19 13:45 access.log.1
-rw-r--r-- 1 backup backup  11M Oct 19 13:45 auth.log
-rw-r--r-- 1 backup backup  35M Oct 18 23:59 auth.log.1

(so access.log was missing), after restart of the service it is here:

$ ll api/
total 4.6G
-rw-r--r-- 1 backup backup  407 Oct 19 13:45 access.log
-rw-r--r-- 1 backup backup 4.6G Oct 19 13:45 access.log.1
-rw-r--r-- 1 backup backup  11M Oct 19 13:45 auth.log
-rw-r--r-- 1 backup backup  35M Oct 18 23:59 auth.log.1

BTW, the proxmox-backup-manager is still on the version 1.1.13. We are planning upgrade.

Thanks for this clarification.

 

[DerDanilo]

@dcsapak Was this fixed already?

On our server the following logs are quiet huge. We currently solved this by switching log rotate to daily.

- proxmox-backup = 10 GiB
- daemon.log = 4 GiB
- syslog = 2 GiB

 

[dcsapak]

we did fix some log rotation related things, namely the cleanup would not continue if it encountered an error... but this only affects the proxmox-backup logs, the syslog + daemon.log are managed by logrotate

 

[DerDanilo]

Any other systems that cause big daemon.log files?
Could you build a command to manually force log rotation for PBS related logs?

 

[dcsapak]

Any other systems that cause big daemon.log files?

i'd check the content of the log to determine what it fills...

Could you build a command to manually force log rotation for PBS related logs?

we could do that, but there is (since recently) a config to set the (maximum) days to keep logs

proxmox-backup-manager node update --task-log-max-days N

it will keep at least N days of task logs (a bit more actually)

task log rotation for now is triggered daily at midnight
does that help you?

 

[NetUser]

Hi,
I'm facing the same issue, in particular, my garbage collection tasks log occupies around 1.1GB of space (there are like 5 million warning inside, which i can't see cause webgui crashes).

Where can i use this "proxmox-backup-manager node update --task-log-max-days N" code? And when will it delete the older logs to give me a bit of time to investigate further the Warning issue?

Thanks in advance

 

[dcsapak]

I'm facing the same issue, in particular, my garbage collection tasks log occupies around 1.1GB of space (there are like 5 million warning inside, which i can't see cause webgui crashes).

sounds like there is some serious issue here, you can check the task log manually with
'proxmox-backup-manager task log <UPID>'

Where can i use this "proxmox-backup-manager node update --task-log-max-days N" code? And when will it delete the older logs to give me a bit of time to investigate further the Warning issue?

that is simply a command you can enter on the cli (via ssh or the web console), but this will probably not help if you have few task logs that are big
the log-rotation is triggered every day at midnight

 

[NetUser]

yup, I've entered 60 days and it will delete a bit of logs which will give me a bit of space/time for the analysis.
About the 1.1GB logs instead, i've this situation:

I used to have a daily backup schedule with like 200 VMs
I moved two of em into a new backup schedule (with dedicated datastore on PBS) without deleting the old ones and the warnings are all like this.

2022-05-13T02:26:11+02:00: WARN: warning: unable to access non-existent chunk 56e479d7042cb81f6dd00db0b80660f8f24b971b092962329f3fb3886757b956, required by "/backup/proxmox/SQLDumps/vm/11273/2022-04-24T21:12:02Z/drive-virtio0.img.fidx"

Is it safe to delete my backups on the older datastore? I think this might solve the issue cause the garbage collection task which gives me pain is the on on the older datastore, while the one on the newer is ok and properly dimensioned

 

[dcsapak]

do you have nested datastores? what does your datastore config look like? (/etc/proxmox-backup/datastore.cfg)

normally chunks should not vanish, or what did you move exactly?

 

[NetUser]

Yes, I've nested datastores, this is the config:

datastore: nfs-em1nas-volume1
    gc-schedule thu 22:00
    keep-daily 7
    keep-last 7
    keep-monthly 6
    keep-weekly 6
    keep-yearly 1
    path /backup/proxmox
    prune-schedule daily

datastore: nfs-truenas-db
    comment
    gc-schedule daily
    keep-daily 11
    keep-last 72
    keep-monthly 6
    keep-weekly 6
    keep-yearly 1
    path /backup/proxmox/SQLDumps
    prune-schedule daily

I didn't "move" anything, i just removed the 2 VMs from the daily backup job on PVE and inserted em in the hourly one

The daily backup is configured on nfs-em1nas-volume1 datastore and the hourly one in the nfs-truenas-db datastore

Right now i have older backups on the daily datastore and the newer ones on the hourly one.

 

[NetUser]

Hi,

despite i've given the "proxmox-backup-manager node update --task-log-max-days 60" i still have older tasks in my folder, here are the examples:

-rw-r--r--   1 backup backup  1.1G Feb 26 01:56 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.1G Mar  4 13:03 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.1G Mar 11 04:50 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Mar 18 04:14 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Mar 25 04:26 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Apr  1 04:21 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Apr  8 04:59 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Apr 15 05:46 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Apr 22 05:25 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G Apr 29 05:03 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.2G May  6 05:16 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:
-rw-r--r--   1 backup backup  1.3G May 13 06:05 UPID:[edited]:garbage_collection:nfs\x2dem1nas\x2dvolume1:root@pam:

If i check the /etc/proxmox-backup/node.cfg file it's empty, how can i check that the PBS has correctly taken/understood the above command?
This is my backup manager version:

proxmox-backup             2.1-1        running kernel: 5.13.19-2-pve
proxmox-backup-server      2.1.5-1      running version: 2.1.2       
pve-kernel-helper          7.1-14                                    
pve-kernel-5.13            7.1-9                                     
pve-kernel-5.4             6.4-10                                    
pve-kernel-5.13.19-2-pve   5.13.19-4                                 
pve-kernel-5.4.151-1-pve   5.4.151-1                                 
pve-kernel-5.4.65-1-pve    5.4.65-1                                  
ifupdown2                  3.1.0-1+pmx3                              
libjs-extjs                7.0.0-1                                   
proxmox-backup-docs        2.1.5-1                                   
proxmox-backup-client      2.1.5-1                                   
proxmox-mini-journalreader 1.2-1                                     
proxmox-widget-toolkit     3.4-7                                     
pve-xtermjs                4.16.0-1                                  
smartmontools              7.2-pve2                                  
zfsutils-linux             2.1.4-pve1

 

[Neobin]

Yes, I've nested datastores

This should be avoided. [1] [2] [3]

Consider changing it, to stop all those warnings in your logs in the first place.

[1] https://forum.proxmox.com/threads/additional-datastore-on-same-zfs-pool.99751/#post-430553
[2] https://forum.proxmox.com/threads/root-disk-filling-up.84215/#post-370332
[3] https://forum.proxmox.com/threads/root-disk-filling-up.84215/#post-370415

 

[NetUser]

thanks for the advice, I'll work on it.

Aside from that the main issue is that the task log rotation is not working, see #16 post for more details, this is the log in my "Log rotation" task

2022-05-15T00:00:00+02:00: starting task log rotation
2022-05-15T00:00:00+02:00: task log archive was not rotated
2022-05-15T00:00:00+02:00: API access log was not rotated
2022-05-15T00:00:00+02:00: API authentication log was rotated
2022-05-15T00:00:00+02:00: TASK OK

 

[dcsapak]

the log rotation only works on our 'index' (/var/log/proxmox-backup/tasks/archive.*) files, so only if those rotate out, then will task logs be removed
thats why i said:

but this will probably not help if you have few task logs that are big

if i do here:

proxmox-backup-manager node update --task-log-max-days 60

my /etc/proxmox-backup/node.cfg contains:

task-log-max-days: 60

instead of trying to solve it that way, i'd rather fix the underlying issue of yours that you nested datastores

you can of course manually delete those task logs for now, ofc you won't be able to view those tasks logs anymore

 

[NetUser]

Ok, thanks for the info, a couple more questions, given that i'm already pursuing the double, not nested, datastores.

1. Why if i type "proxmox-backup-manager node update --task-log-max-days 60" it doesn't appear in my node.cfg config?
my version on another server is the latest and it still doesn't appear, any idea on how i can troubleshoot that? Can i type it directly in the node.cfg and restart proxmox-backup-manager?

2. When does a log task ends up in the archive? Cause some of those logs are definitely older than 60 days, and after I've a clean situation with the double datastore, I'd like to slowly cleanup those bigger logs

Many thanks for the help