PVE Native Install of Grafana Alloy Client

[tssrshot]

Is it possible or possibly in advisable to install Grafana Alloy client natively in each PVE Physical Server?

Alloy vs Prometheus Exporter has flipped the script on remote write of logs and metrics, but people are suggesting it as a container…however, that lacks permissions to some key functions like storage and logs to be universally forwarded.

There are Debian repos for Alloy, and there is a small config file element to it.

https://grafana.com/docs/alloy/latest/set-up/install/linux/

 

[cloudsherd]

Hi,

Try this:
1. Create small container: 1 CPU, 256 MB RAM
2. Install alloy in it.
3. In the PVE host console run:

#!/bin/bash

container=<put-your-container-name-here>

pct set $container -mp0 /,mp=/host/rootfs,ro=1
pct set $container -mp1 /proc,mp=/host/proc,ro=1
pct set $container -mp2 /sys,mp=/host/sys,ro=1
pct set $container -mp3 /run/udev/data,mp=/host/run/udev/data,ro=1

4. Restart container
5. Configure alloy in container:

//config.alloy example
// journald
loki.source.journal "journal" {
  forward_to = [loki.write.default.receiver]
  relabel_rules = loki.relabel.journal.rules
  max_age    = "12h"
  path       = "/host/rootfs/var/log/journal"
  labels = {
    job  = "systemd-journal",
    host = "pve",
  }
}
prometheus.exporter.unix "pve" {
  // Disable unnecessary collectors to reduce overhead
  disable_collectors = ["ipvs", "infiniband", "xfs", "btrfs"]
  rootfs_path = "/host/rootfs"
  procfs_path = "/host/proc"
  sysfs_path = "/host/sys"
  udev_data_path = "/host/run/udev/data"
  ...
}

6. In container, execute: systemctl start alloy
7. Having rootfs mounted inside the container isn't safe even it's read only.

Good luck.

 

[stoye]

Not OP, but thank you for this! Works perfectly. I'm just using the native prometheus exporter for metrics and just wanted alloy for log collection, so simply mounting `/var/log` to `/host` read-only and tweaking your config worked a treat.

 

[deebsr]

I'm curious about the guidance from someone from Proxmox itself on this.
Is this advisable? can it scale? We are looking to also possibly use Alloy but we are a much bigger environment than a home lab.

It would be nice if there was a built in setup similar to the Metrics ( via Otel ) setup specifically for sending logs to an external collector.

 

[tssrshot]

We've actually gotten some good metrics from the new OTel Exporter directly to Alloy.

 

[louie1961]

I have Grafana Alloy installed natively on my Proxmox nodes. Pretty easy to do:

apt update && apt install -y gpg wget
mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list

apt update && apt install -y alloy

The config for Alloy lives at /etc/alloy/config.alloy. I assume you know how to configure it for your stack?

 

[deebsr]

We've actually gotten some good metrics from the new OTel Exporter directly to Alloy.

I've implemented this but there seems to be some questions I have about the metrics that get gathered: https://forum.proxmox.com/threads/questions-about-otel-metrics-listing.184949/

I ended up installing Alloy on an external system ( acting like a gateway or collector to Mimir ) and used OTEL on the PVE cluster to send to that.

While the above comment you can install directly onto the host I think this is not advisable on an enterprise scale ( extra thing to manage and install ).
What would be nice is that Alloy could be included into the PVE distro or perhaps we could all the alloy repo to something like POM so we can gate the versioning that is rolled out?

 

[louie1961]

While the above comment you can install directly onto the host I think this is not advisable on an enterprise scale ( extra thing to manage and install ).

That's what tools like Ansible are for. I manage my entire monitoring stack with Ansible, and I used Ansible for the original deployment across my fleet of VMs and physical hosts. I also used Ansible to update from Promtail to Alloy, and do my version updates for the entire stack using Ansible.

As far as versioning you can pin versions with either apt or with ansible.

 

[deebsr]

That's what tools like Ansible are for. I manage my entire monitoring stack with Ansible, and I used Ansible for the original deployment across my fleet of VMs and physical hosts. I also used Ansible to update from Promtail to Alloy, and do my version updates for the entire stack using Ansible.

As far as versioning you can pin versions with either apt or with ansible.

yeah that works.
IMO a hypervisor should not need to have anything installed inside it. It just makes support that much more difficult since you have to take into account any of the extra software that's installed. I'm coming from the Vmware world where there was generally nothing "installed" on a host.
things like metrics and logs was just a config that was set. the act of installing an managing software to get metrics was built into the hypervisor.

PVE has the OTEL config for metrics but what would be nice is if they have some way of also just configuring the hosts to send their logs without having to resort to installing xyz software agent into every host and configuring it to do such thing.

 

[louie1961]

IMO a hypervisor should not need to have anything installed inside it.

Perhaps, but that doesn't seem to matter to the OP. Plus Proxmox is just Debian with a custom kernel and a few extra bits thrown on top. It's pretty easy to manage.