Maximal Workers/bulk-action default (4) appears to behave like 1.

[leesteken]

I noticed on a tiny PVE setup (no-subscription 9.2.11 with only two cores in a VM) that when Maximal Workers/bulk-action (under Datacenter, Options) is set to 4 or left empty (the GUI shows 4 in both cases), the Bulk shutdown VMs and Containers consistently seems to behave as if it is set to 1. When set to 3 or 5, if does shut down 3 or 5 at a time and finishes much faster while shutting down 4 containers. I cannot explain this from the manual.

I noticed this because a reboot of the PVE VM took too long to shut down when using tight timeouts. I would not notice this otherwise and just assume that PVE is slow to shut down, as the default setting seems to trigger this behavior.

The work-around is easy but maybe someone could check if this is a reproducible bug affecting others as well?

 

[jklocker]

I couldn't reproduce this. On my side, with 4 containers (no Start/Shutdown order set) and max-workers at the default of 4, all four kick off at once and finish in a couple of seconds - no serial behavior. I also printed the worker count in the code and it correctly resolves to 4.

To narrow down what's different about your setup, could you provide:
- Do any of the guests have a Start/Shutdown order configured (guest -> Options)?
- The task log and the journal of a slow run (default/4) and a fast run (3 or 5)
- Are the guests VMs, CTs or mixed? From what I understand they are only CTs, correct?

 

[leesteken]

I couldn't reproduce this. On my side, with 4 containers (no Start/Shutdown order set) and max-workers at the default of 4, all four kick off at once and finish in a couple of seconds - no serial behavior. I also printed the worker count in the code and it correctly resolves to 4.

Thank you for looking into this.

- Do any of the guests have a Start/Shutdown order configured (guest -> Options)?

They all have order any and time-out 20.

- The task log and the journal of a slow run (default/4) and a fast run (3 or 5)

They are the same. While it is processing, I can see whether it does multiple at once (like 3) or one at a time (when set to default or 4) by the speed the lines appear in the task log.

Stopping CT 128 (timeout = 20 seconds)
Stopping CT 126 (timeout = 20 seconds)
Stopping CT 124 (timeout = 20 seconds)
Stopping CT 122 (timeout = 20 seconds)

This is Bulk Shutdown with 3:

This is Bulk Shutdown with default (4):

- Are the guests VMs, CTs or mixed? From what I understand they are only CTs, correct?

All up to date Debian and Ubuntu containers, four in total. EDIT: I see the same behavior when doing a Bulk Start (or starting/shutting down the nested Proxmox VM).

 

[leesteken]

A different PVE 9.2 setup with 4 cores (on a VPS) seems to stop/start 3 at a time when set to default (4). Almost as if PVE does (number of cores - 1) at a time when set to default (or 4)? Which would be only noticeable on very small and rather slow systems?

 

[jklocker]

A different PVE 9.2 setup with 4 cores (on a VPS) seems to stop/start 3 at a time when set to default (4). Almost as if PVE does (number of cores - 1) at a time when set to default (or 4)?

That's a good observation. Turns out I was looking at the wrong bulk action -- the endpoints, and therefore code executed, is different if you use the `Bulk Actions` on the Datacenter level (Datacenter->Bulk Action) or the node level (<node>->Bulk Action).

As you correctly observed, when the default option is set it falls back to cpu cores - 1 for the node level bulk action.
Following function is responsible for this [0].

[0] https://git.proxmox.com/?p=pve-mana...c5e08c165703671f5ed1172cb649e0a;hb=HEAD#l2114