Cluster Tasks vs. node's Tasks History

[lubr]

Hi,

I thought cluster tasks (Tasks tab) is a merge of all nodes' Tasks History. But sometimes it does not include all tasks that have been performed over node. After reading note in https://pve.proxmox.com/wiki/Graphical_User_Interface#_log_panel I realized it is shrinked when there are lots of same tasks at the same time. I would understand that it is a kind of optimization for GUI but I found out the same behavior is applied to https://pve.proxmox.com/pve-docs/api-viewer/index.html#/cluster/tasks API method as well. Why?

Another sentence in that notice describing Log panel (Tasks tab + /cluster/tasks method) ensures "But you can still find those tasks within the node panel in theTask History." meaning a group of clusters tasks related to specific node should be found there anytime. But that is not true because eg. vncproxy task on any VM in that node is a part of cluster tasks only but not in node's Tasks History so when I want to check if vncproxy has been active on some VM, I need to check cluster tasks with manual filtering of VM instead of node tasks history where it should belong, I suppose.

Why eg. vncproxy is not a part of node tasks history but is a part of cluster tasks only? Are there any other VM tasks that can be found in cluster tasks only? What if cluster tasks are shorted so that vncproxy of VM I am seeking for is not found, and therfore I am not able to catch VNC operation anyhow?

Thx.

 

[dcsapak]

but I found out the same behavior is applied to https://pve.proxmox.com/pve-docs/api-viewer/index.html#/cluster/tasks API method as well. Why?

The cluster task list is distributed over the cluster file system, and to keep the load small, only a certain amount (i think 32k bytes, but i'm not sure, don't quote me on that ) is retained there (otherwise it could grow quite quickly, increasing the load on the cluster network that's also responsible for things like HA

But that is not true because eg. vncproxy task on any VM in that node is a part of cluster tasks only but not in node's Tasks History so when I want to check if vncproxy has been active on some VM, I need to check cluster tasks with manual filtering of VM instead of node tasks history where it should belong, I suppose.

mhmm i can see the vncproxy tasks in the nodes task history, are you sure it's not there for you? what are the commands/views you did check?

note that only about 2000 tasks (IIRC) are retained, so older ones are deleted automatically...

 

[lubr]

Thx for you response. See attachment please where Task History vs. Tasks is shown - the line is placed into the beginning of the list where I would expect Console (vncproxy) command after all those prunebackups commands as well. Note that Reload was pressed several times + page reloaded as well.



And regarding 2000 tasks limit: when I store task ID (that UPIDxxxx) in to DB and query this task few days later, does it mean it will not exist anymore?

 

[dcsapak]

regarding the screenshot, that's interesting indeed. Does it show the task on the vm task history itself?
Was the vm moved between hosts? (the task won't migrate with the vm, but stays on the host where it was started)

can you open the status pane of the task log window and post a screenshot ? (sans sensitive info of course)

And regarding 2000 tasks limit: when I store task ID (that UPIDxxxx) in to DB and query this task few days later, does it mean it will not exist anymore?

if it's rotated out, yes it does not exists anymore, but usually, this limit is so high that it does not matter for most people. if you feel it's not enough for you, you can open a feature request, e.g. to be able to tune it (https://bugzilla.proxmox.com)

 

[lubr]

Yes, in VM tasks it is not shown as well, see:



Status pane of the Console task:



But I remember that PVE 8.1.x had no such issue and since 8.2.4, on which we updated then, it suddenly became missing in both node Tasks History and was included in cluster's Tasks only (so we updated our implementation but now we realize we need it in rather in Task History). I thought it was a feature but it seems as a bug, or? Does it mean that cluster's Tasks should really be a merge of nodes' Tasks History over the same cluster?

 

[dcsapak]

ah ok i guess i see what's happening

you connect to *-VPSLAB01 but the vm is on *-VPSLAB02

the console task is actually started on the node where you connect to, not the one where the vm is (the latter part of the connection is then tunneled to the right server)

so you should see the task on the other node (*-VPSLAB01)

 

[lubr]

You are absolutely right, I have missed that. Now I have realized that node Task History is related to node URL I am connected to, so when I open console on VMs on the node I am connected to, I see that task in node Tasks History which means Tasks History of VM is being build in node am I operating on, no matter on which node VM is placed. But once I migration the VM on the node I am connected to, suddenly I have this history available but lose history from the old node.

Therefore, working with cluster Tasks is more reliable since besides that optimization shortening of the same tasks on some peak occasions, so I am gonna post feature request to eg. pass parameter 'full-list=<true|false>' to /cluster/tasks which would return really the most recent tasks over that 32 k bytes limit with understanding it could have performance impact.

One last question: can you explain again that 2000 rows limit? Is it node Tasks History limit, or cluster Tasks one? If it is nodes' limit, what is the rows limit of cluster Tasks then?

Thx for your support.

PS: Can you, please, respond to my second topic on https://forum.proxmox.com/threads/delete-all-vm-backups-via-api-prunebackups.168414/ as well?