Proxmox Logs to ElasticSearch ... any advice?

N

numpty_boy

New Member
Jul 14, 2025
3
0
1
Hi Folks.

we're looking at using proxmox to replace vmware [like millions of others, I suspect]. We've always sent our ESX logs to ElasticSearch for enhancement and storage, like we do for just about everything.

I appreciate that Proxmox already has a great front-end and what I just wrote may be heresy to some, but that's the way 'the suits' want to go.

I'm happy with Elasticsearch [less so with proxmox, but a log is a log ... right?], but before heading off on this journey just wondered if anyone had already done this and had any advice they'd be kind enough to share with me?

Thanks & All the BEst

ChIP.
 
Install filebeat and read from /var/log, same as any debian. But you didn't say what else do you have,like ceph,etc. Maybe you need more collectors inside .yml file.
 
  • Like
Reactions: Kingneutron
Thanks for your speedy replies. Much appreciated.

I've found the files you mention and already started looking at their structure, etc 'by hand', before my post.

Let me ask some slightly more specific questions:
  1. Does anyone have or know where I can find 'grok' strings for the log files proxmox produces? It would be a shame to produce my own, only to find that someone had done so before me [almost certainly better]. Yes, I've looked, but aside from a very old discussion re- firewall logs, nothing much out there.
  2. Are the file formats defined anywhere?
  3. Has anyone done what I'm proposing to do before, either with filebeat, logstash, or ES Agent? If so, how did it go? Any 'gochas'?
My starting point here is that I'm really surprised that there aren't loads of folks doing this. Since there don't seem to be, I'm wondering if there's a reason for that ..?

Thanks again for the great replies already received.

ChIP.
 
I'm forwarding logs directly to Eventlog, instead of elastic or filebeat or logstash. But if you want to enrich the data,or split onto some more fields, than i don't have anything. What is the primary goal, what kind of messages do you want/need?
 
ness1602 said:
I'm forwarding logs directly to Eventlog, instead of elastic or filebeat or logstash. But if you want to enrich the data,or split onto some more fields, than i don't have anything. What is the primary goal, what kind of messages do you want/need?
Click to expand...
Thanks for this.

The objective here is to basically get an overview of the proxmox estate, in terms of physical machine health [we can do this via existing ES agents] and the individual VMs hosted on those machines [so we can see VM 'X' is hosted on proxmox server 'Y' and these are the resources/state/errors, etc associated with that VM]. We're also interested in the 'proxmox internals' that aren't readily available from standard *ix integrations already available from ES.

For example - I'm sure there's a lot of really helpful information in /var/log/pve, but as to what it is ...

In a nutshell, I guess what I'm looking to do is produce the equivalent of the ES VSphere agent [ https://www.elastic.co/docs/reference/integrations/vsphere ] for Proxmox.

Thanks again

ChIP.
 
You must log in or register to reply here.
Top Bottom