Where are cluster and task logs stored? (For shipping to log server)

victorhooi

Well-Known Member
Apr 3, 2018
251
20
58
38
I have a 4-node Proxmox cluster.

I'd like ship my Proxmox Cluster logs and Task logs to an offsite logging server (e.g. Graylog, Loki etc.) - this allows us to keep track of which users did what actions, and when VMs were created (creation time isn't currently exposed - FR to add this - but for now, we'll try to do log scraping):

Screen Shot 2020-07-11 at 9.51.16 am.png

Screen Shot 2020-07-11 at 9.54.59 am.png

I believe the cluster log is at /etc/pve/.clusterlog on each node in our cluster. However, this only stores the most recent entries? Is there another location where these are stored, or where we can get older, or rotated logs?

For the task log - is there a place where historical tasks are logged?
 
hi,

/var/log/pve/ directory has the tasks subdirectory which holds the logs for all the tasks.

also /var/log/pveproxy holds the web ui logs (access time, IP, like a regular webserver)

hope this helps
 
Thanks oguz for the info!

However, I'm having trouble finding the info I need in that folder - or maybe I'm not parsing it correctly?

My goal is to get a time-ordered list of tasks (i.e. a logfile of tasks), with the timestamp, action, and username for each line. Basically, like you would see in say, syslog.

I took a look in /var/log/pve/tasks.

There is an index file, which has lines like this:

Code:
UPID:example-kvm02:003D2ED6:01835485:5F09980F:vncproxy:524:example@example.io: 5F099B97 OK

There is also a file called active, which has lines like this:
Code:
UPID:example-kvm02:003A5FA7:02FDD3D5:5F0D6103:qmstart:371:user1@example.io: 1 5F0D6104 OK
UPID:example-kvm02:0031D74E:02C5D2CA:5F0CD1A4:qmdestroy:544:user2@example.io: 1 5F0CD1A7 OK
UPID:example-kvm02:0031D703:02C5D22A:5F0CD1A3:qmstop:544:user3@example.io: 1 5F0CD1A4 OK

Then there are directories called 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E and F. These contain files like UPID:example-kvm02:00032AB1:0197985D:5F09CBF0:vncproxy:524:user1@anguslab.io:, which contains text like:
Code:
TASK OK

So my questions is:

1. Where is the timestamp for each task (e.g. start/stop time)?
2. Is there some documentation that lists what each of the colon-delimited fields represents?
2. Does this tasks directory only contain active/recent tasks? How do you control what tasks get rotated out?
3. Is there a historical record already somewhere, that I should turn to? It seems like that might be easier - otherwise I will have to periodically walk this directory, and do diffs on the files, right?
 
  • Like
Reactions: Tmanok
general format:
UPID:$node:$pid:$pstart:$starttime:$dtype:$id:$user
pid, pstart and starttime are hex encoded

also the stat command can reveal valuable information about tasks:

Code:
  File: UPID:gaia:000014C6:000009EA:5EC295B0:startall::root@pam:
  Size: 8               Blocks: 1          IO Block: 512    regular file
Device: 1ah/26d Inode: 884811      Links: 1
Access: (0640/-rw-r-----)  Uid: (   33/www-data)   Gid: (    0/    root)
Access: 2020-05-18 16:03:28.830346490 +0200
Modify: 2020-05-18 16:03:28.850346490 +0200
Change: 2020-05-18 16:03:28.850346490 +0200
 Birth: -

also for example stat -c %x filename will reveal last access time of the file.

more information can be found about other flags in man stat.

there's also pvenode task list command which will print out a nice table :)


2. Does this tasks directory only contain active/recent tasks? How do you control what tasks get rotated out?

on my testing machine i have task logs from almost 6 months ago. the active file will show the tasks in the current session.

for example if you run: find /var/log/pve/tasks -exec stat -c %x {} \; | sort you'll see the dates


also read the previous thread[0] about this.

[0]: https://forum.proxmox.com/threads/api-to-read-qemu-vm-creation-time-or-uptime.45092/
 
Hmm for myself - I seem to have files mostly from 2020-07-08 (i.e. around a week ago) up to today - along with some random files from around 2020-03-23:

Code:
# find /var/log/pve/tasks -exec stat -c %x {} \; | sort 
2020-03-23 07:09:47.468022365 +1100
2020-03-23 07:13:52.528000551 +1100
2020-03-23 07:13:52.536000551 +1100
2020-03-23 08:05:38.086395843 +1100
2020-03-23 08:07:49.324000512 +1100
2020-03-23 08:09:55.428798260 +1100
2020-03-23 08:16:43.388188930 +1100
2020-03-23 08:16:58.908643849 +1100
2020-03-23 08:21:21.337692048 +1100
2020-03-23 08:21:40.841973413 +1100
2020-03-23 08:21:59.942243947 +1100
2020-03-23 08:22:32.786699535 +1100
2020-03-24 17:55:59.792008229 +1100
2020-03-26 20:55:05.117488060 +1100
2020-03-27 04:57:33.226832909 +1100
2020-03-28 07:33:32.129257399 +1100
2020-03-28 08:00:14.272680001 +1100
2020-03-29 05:14:30.794226868 +1100
2020-07-08 06:22:04.187974055 +1000
2020-07-08 06:25:46.703387054 +1000
2020-07-08 06:32:04.266757388 +1000
2020-07-08 06:42:28.544930615 +1000
2020-07-08 06:45:56.309916914 +1000
2020-07-08 07:17:35.239886741 +1000
2020-07-08 08:50:57.304121776 +1000
2020-07-08 08:51:48.684303977 +1000
2020-07-08 08:53:17.548622354 +1000
2020-07-08 08:54:11.324816784 +1000
2020-07-08 08:58:27.281756362 +1000
2020-07-08 09:01:26.358423909 +1000
2020-07-08 14:34:17.328257249 +1000
2020-07-08 14:34:36.540285146 +1000
2020-07-08 15:32:10.728831825 +1000
2020-07-08 15:45:08.448514079 +1000
2020-07-08 15:45:20.680558353 +1000
2020-07-08 15:48:56.813300368 +1000
2020-07-08 15:49:14.557358401 +1000
2020-07-08 16:13:06.123028808 +1000
2020-07-08 16:39:51.927929459 +1000
2020-07-08 16:40:20.639973637 +1000
2020-07-08 16:43:57.640105807 +1000
....<shortened>....
2020-07-14 17:38:43.482189882 +1000
2020-07-14 17:38:44.306208656 +1000

So it seems in my case, it retains about a week of task logs.

What would be the best practice way of "tailing" this hierarchy, so to speak?
 
well, you can watch the log output of pvedaemon and pveproxy, it will tell you when tasks are started and finished including the UPID that is used as reference to retrieve the task metadata and logs:
Code:
$ journalctl -u pveproxy -u pvedaemon
[...]
Feb 04 10:17:30 $HOST pvedaemon[51844]:  starting task UPID:$HOST:000318A5:0007A69E:5E3936AA:qmcreate:101:$USERID: 
Feb 04 10:17:30 $HOST pvedaemon[51844]:  end task UPID:$HOST:000318A5:0007A69E:5E3936AA:qmcreate:101:$USERID: OK 
[...]

the rest can then be done via 'pvenode' or the API..
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!