[SOLVED] Odd behavior - node live system log blank - after delay then appears (only on primary cluster node)

[rfox]

Running latest 9.2.4 with all updates - cluster of 4 nodes and one extra node standalone. I've recently noticed very odd behavior when I check the node logs under System > System Logs

In live mode, nothing is showing, blank screen - when I select a time range - it says "nothing logged or no timespan selected" - but if I drop to a shell and run "journalctl -f" - I see entries ?!? Tried with latest Firefox & Chrome browsers

After testing further - the primary node in the cluster (node 1) shows data in the GUI in Live Mode - but only after a long delay - the other nodes show nothing under System Logs

The single node also shows this behavior - I click on system logs, it's blank and about 20-30 seconds later it's populated ?!?

Is this normal ? Any hints would be welcome . . .

 

[woma]

Not normal ...

Every node shows data instantly after selecting "System Logs" in live mode.
Does not matter if I am logged in directly on that node to be queried or on any other cluster node.

Live Mode is active by default when entering that "System Logs" page, so message "nothing logged or no timespan selected" never appears first.
It would show up after clicking on "Time Span" button.

Cluster has 6 nodes, Enterprise is currently at 9.2.3, Firefox 151.0.4 on Ubuntu 26.04

 

[rfox]

Thanks woma for the prompt response - you are right with the "nothing logged" message - only shows when timespan is chosen - my mistake

But the other three nodes in my cluster are not showing any data in Live Mode or timespan - just the node 1and after a long delay
When I click on system logs on node one - I can hear he system get louder briefly like it's doing something - then the data appears

Nodes 2,3 & 4 show NO data - no matter how long I wait. In the console windows of those nodes when I check "journalctl -f" I see data

 

[rfox]

After further research I found that I have to vacuum out journalctl logs with the following commands on each node:

journalctl --vacuum-time=2d
systemctl restart systemd-journald

Now, when I choose the Live Mode - I get instant result (as expected!)