Possible to bind mount PVE "/var/logs" as read-only inside LXC?

[snowturtle]

I'm looking to setup a light SIEM in my homelab, well it's more just an "observability" setup really. I'm planning to put both logs and metrics into this across a variety of systems. I know I can export metrics via the native influxdb/graphite options but there's no built-in way to export logs.

I'm going to use Grafana Alloy | Grafana Alloy documentation to collect system logs, but I know the advice is to put a minimal amount of stuff on top of proxmox to prevent any issues with updating or dependency conflicts. My idea was that I would spin up an lxc on each pve node and bind mount proxmox's `/var/logs/` directory into that lxc, hopefully as read-only, and then install the Alloy agent inside the lxc.

Would I run into any issues doing that? I think there might be, hopefully minor, permission conflicts as it should be read-only to the LXC, so alloy isn't going to be writing to it on either the PVE side or the LXC side.

If I can get away with running the Alloy agent 'bare metal' on the PVE nodes directly, then that could be useful as Alloy also collects system metrics, as well as traces, profiling and "frontend user monitoring". I'm not sure how useful those last 3 would actually be for proxmox monitoring, but it could be cool to play with.

If I do go the LXC route for Alloy then I won't be able to collect system metrics, but I can probably just use the promethis-pve-exporter project for those (if I don't fall back on influxdb, which is an option).

 

[leesteken]

My idea was that I would spin up an lxc on each pve node and bind mount proxmox's `/var/logs/` directory into that lxc, hopefully as read-only, and then install the Alloy agent inside the lxc.

Have you tried using a bind mount: lxc.mount.entry: /var/log path_without_starting_slash/inside/the/container none bind 0 0? I don't know how to configure Alloy to look at the path_without_starting_slash/inside/the/container directory.

 

[snowturtle]

Have you tried using a bind mount: lxc.mount.entry: /var/log path_without_starting_slash/inside/the/container none bind 0 0? I don't know how to configure Alloy to look at the path_without_starting_slash/inside/the/container directory.

That's kinda the question I had; it was less about alloy and more about bind mounting the log directory as read only - which I get the impression is possible by the `path none bind 0 0` you mentioned. I tried to look it up on the docs but I didn't find it particularly enlightening.

Also along a similar vein; would it be better to run alloy on the host or in an lxc?

 

[leesteken]

Also along a similar vein; would it be better to run alloy on the host or in an lxc?

Do you want to monitor your host when no guests are running (and therefore your Alloy agent is not active)?

 

[snowturtle]

Do you want to monitor your host when no guests are running (and therefore your Alloy agent is not active)?

It'd be nice to be able to do that, but I'm in-different to it. The lxc that alloy will run in will be dedicated to alloy so then so long as that containers running then that's fine. I'll probably update the start order to make alloy start first.

And I'm right in thinking I wouldn't be able to pass host metrics through to the lxc? Things like host cpu ussge, mem ussge, storage usage. I'd assume that's not possible and it makes sense why it wouldn't be.