Where are cluster and task logs stored? (For shipping to log server)

[victorhooi]

I have a 4-node Proxmox cluster.

I'd like ship my Proxmox Cluster logs and Task logs to an offsite logging server (e.g. Graylog, Loki etc.) - this allows us to keep track of which users did what actions, and when VMs were created (creation time isn't currently exposed - FR to add this - but for now, we'll try to do log scraping):





I believe the cluster log is at /etc/pve/.clusterlog on each node in our cluster. However, this only stores the most recent entries? Is there another location where these are stored, or where we can get older, or rotated logs?

For the task log - is there a place where historical tasks are logged?

 

[oguz]

hi,

/var/log/pve/ directory has the tasks subdirectory which holds the logs for all the tasks.

also /var/log/pveproxy holds the web ui logs (access time, IP, like a regular webserver)

hope this helps

 

[victorhooi]

Thanks oguz for the info!

However, I'm having trouble finding the info I need in that folder - or maybe I'm not parsing it correctly?

My goal is to get a time-ordered list of tasks (i.e. a logfile of tasks), with the timestamp, action, and username for each line. Basically, like you would see in say, syslog.

I took a look in /var/log/pve/tasks.

There is an index file, which has lines like this:

UPID:example-kvm02:003D2ED6:01835485:5F09980F:vncproxy:524:example@example.io: 5F099B97 OK

There is also a file called active, which has lines like this:

UPID:example-kvm02:003A5FA7:02FDD3D5:5F0D6103:qmstart:371:user1@example.io: 1 5F0D6104 OK
UPID:example-kvm02:0031D74E:02C5D2CA:5F0CD1A4:qmdestroy:544:user2@example.io: 1 5F0CD1A7 OK
UPID:example-kvm02:0031D703:02C5D22A:5F0CD1A3:qmstop:544:user3@example.io: 1 5F0CD1A4 OK

Then there are directories called 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E and F. These contain files like UPID:example-kvm02:00032AB1:0197985D:5F09CBF0:vncproxy:524:user1@anguslab.io:, which contains text like:

TASK OK

So my questions is:

1. Where is the timestamp for each task (e.g. start/stop time)?
2. Is there some documentation that lists what each of the colon-delimited fields represents?
2. Does this tasks directory only contain active/recent tasks? How do you control what tasks get rotated out?
3. Is there a historical record already somewhere, that I should turn to? It seems like that might be easier - otherwise I will have to periodically walk this directory, and do diffs on the files, right?

 

[oguz]

general format:
UPID:$node:$pid:$pstart:$starttime:$dtype:$id:$user
pid, pstart and starttime are hex encoded

also the stat command can reveal valuable information about tasks:

  File: UPID:gaia:000014C6:000009EA:5EC295B0:startall::root@pam:
  Size: 8               Blocks: 1          IO Block: 512    regular file
Device: 1ah/26d Inode: 884811      Links: 1
Access: (0640/-rw-r-----)  Uid: (   33/www-data)   Gid: (    0/    root)
Access: 2020-05-18 16:03:28.830346490 +0200
Modify: 2020-05-18 16:03:28.850346490 +0200
Change: 2020-05-18 16:03:28.850346490 +0200
 Birth: -

also for example stat -c %x filename will reveal last access time of the file.

more information can be found about other flags in man stat.

there's also pvenode task list command which will print out a nice table

2. Does this tasks directory only contain active/recent tasks? How do you control what tasks get rotated out?

on my testing machine i have task logs from almost 6 months ago. the active file will show the tasks in the current session.

for example if you run: find /var/log/pve/tasks -exec stat -c %x {} \; | sort you'll see the dates


also read the previous thread[0] about this.

[0]: https://forum.proxmox.com/threads/api-to-read-qemu-vm-creation-time-or-uptime.45092/

 

[victorhooi]

Hmm for myself - I seem to have files mostly from 2020-07-08 (i.e. around a week ago) up to today - along with some random files from around 2020-03-23:

# find /var/log/pve/tasks -exec stat -c %x {} \; | sort 
2020-03-23 07:09:47.468022365 +1100
2020-03-23 07:13:52.528000551 +1100
2020-03-23 07:13:52.536000551 +1100
2020-03-23 08:05:38.086395843 +1100
2020-03-23 08:07:49.324000512 +1100
2020-03-23 08:09:55.428798260 +1100
2020-03-23 08:16:43.388188930 +1100
2020-03-23 08:16:58.908643849 +1100
2020-03-23 08:21:21.337692048 +1100
2020-03-23 08:21:40.841973413 +1100
2020-03-23 08:21:59.942243947 +1100
2020-03-23 08:22:32.786699535 +1100
2020-03-24 17:55:59.792008229 +1100
2020-03-26 20:55:05.117488060 +1100
2020-03-27 04:57:33.226832909 +1100
2020-03-28 07:33:32.129257399 +1100
2020-03-28 08:00:14.272680001 +1100
2020-03-29 05:14:30.794226868 +1100
2020-07-08 06:22:04.187974055 +1000
2020-07-08 06:25:46.703387054 +1000
2020-07-08 06:32:04.266757388 +1000
2020-07-08 06:42:28.544930615 +1000
2020-07-08 06:45:56.309916914 +1000
2020-07-08 07:17:35.239886741 +1000
2020-07-08 08:50:57.304121776 +1000
2020-07-08 08:51:48.684303977 +1000
2020-07-08 08:53:17.548622354 +1000
2020-07-08 08:54:11.324816784 +1000
2020-07-08 08:58:27.281756362 +1000
2020-07-08 09:01:26.358423909 +1000
2020-07-08 14:34:17.328257249 +1000
2020-07-08 14:34:36.540285146 +1000
2020-07-08 15:32:10.728831825 +1000
2020-07-08 15:45:08.448514079 +1000
2020-07-08 15:45:20.680558353 +1000
2020-07-08 15:48:56.813300368 +1000
2020-07-08 15:49:14.557358401 +1000
2020-07-08 16:13:06.123028808 +1000
2020-07-08 16:39:51.927929459 +1000
2020-07-08 16:40:20.639973637 +1000
2020-07-08 16:43:57.640105807 +1000
....<shortened>....
2020-07-14 17:38:43.482189882 +1000
2020-07-14 17:38:44.306208656 +1000

So it seems in my case, it retains about a week of task logs.

What would be the best practice way of "tailing" this hierarchy, so to speak?

 

[fabian]

well, you can watch the log output of pvedaemon and pveproxy, it will tell you when tasks are started and finished including the UPID that is used as reference to retrieve the task metadata and logs:

$ journalctl -u pveproxy -u pvedaemon
[...]
Feb 04 10:17:30 $HOST pvedaemon[51844]:  starting task UPID:$HOST:000318A5:0007A69E:5E3936AA:qmcreate:101:$USERID: 
Feb 04 10:17:30 $HOST pvedaemon[51844]:  end task UPID:$HOST:000318A5:0007A69E:5E3936AA:qmcreate:101:$USERID: OK 
[...]

the rest can then be done via 'pvenode' or the API..