Default Startup Delay

[cybran]

For a virtual machine, I can configure startup order and startup delay (see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_virtual_machines_settings).

However, what is the default startup delay and how can I configure it (as a node-wide setting?).

(I'm not referring to the start-on-boot delay.)

 

[VictorSTS]

Not sure if I'm understanding your question, but maybe you are refering to this:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#first_guest_boot_delay

The default is 0, meaning that VMs and CTs will start as soon as pve-manager service starts.

 

[Gavino]

@VictorSTS I would say that @cybran isn't referring to the "--startall-onboot-delay" you linked to because they stated, "I'm not referring to the start-on-boot delay", which while not exact wording, is very close.

I think @cybran came here for the same reason I did just now. I'm almost certain they mean this part (highlighted):



The "Help" button links here:
https://gtpve03.owenmg.org:8006/pve-docs/chapter-qm.html#qm_startup_and_shutdown

The "shutdown timeout" default is given (180 seconds), but unfortunately, nowhere that i can see says what the Startup delay default is. It merely states:

"

• Startup delay: Defines the interval between this VM start and subsequent VMs starts. For example, set it to 240 if you want to wait 240 seconds before starting other VMs.

"

No mention of default. Anyone know what the default is, and if it can be changed? Ideally I'd like to be able to change the default (like I can in vSphere), so I don't have to remember to update this for each VM, if the default is not to my liking.

Thanks,
Gavin

PS - Proxmox folk - please update online help. Thank you.

 

[LukasInCloud]

Gavinо’s right: the Help section explains what the setting does, but not what the default value is (if any) or whether it can be configured globally.
As far as I know, there is no node-wide default setting for the per-VM startup delay. If the delay is left empty in the UI, it defaults to 0 (i.e., no delay). You’d need to set it individually per VM. I agree that it would be useful to have a global/default setting like in vSphere.
Could you share if you're using HA or clustering? That might open up other config options.

 

[VictorSTS]

@Gavino What are you trying to archive exactly? Don't know what is the behavior of that vSphere delay setting you mention.

Remember that those order and start/shutdown delays aren't honored if the VMs are under HA and that those order and delays only apply to each node separately (i.e. you can't set orders among nodes of a cluster).

Edit: AFAIR, there's no way to set a global startup delay, which defaults to 0. Makes sense, as it is intended to delay the start of the next VM, i.e. if you know that this VM will take say 1 minute to boot and start a database needed by a VM that will be started in the next order.

 

[Gavino]

OK so it does seem clear that the per-VM startup delay default value is zero, based on what Lukas and Victor have stated above, paired with my own experience. Good Lord - zero is such a bad default to have!

I am setting up this one customer who has 6 VMs and 1 container on a standalone PVE host (standalone for now). I set the Windows domain controller to start up first with a 180 second shutdown and startup delay. It is the DHCP/DNS and other important "must start first" things.

I didn't touch the startup/shutdown values for other VMs, as ordering doesn't matter for them, and I naively assumed startup staggering. In vSphere, after the "Automatic Ordered" VMs are started, the unordered ones start randomly but staggered on a default amount that can be tweaked. See here for vCenter Server Administrator, in the section for "Automatic" VMs:

I can't recall if 120 seconds is the value a fresh install gets, or if it was what I set (I'd say probably 120 seconds). But regardless, it's staggered and can be changed. The beauty of it is you can elect to start up the next one when the VMware Tools start responding to the ESXi host. Proxmox should look at doing similar.

So I was quite surprised when watching a Proxmox PVE host boot up for this build I'm doing that after the DC started, and then waiting the 180 seconds I set - all the other VMs started up at the same time! Because of the disk load, I noticed that some of the automatic services in one of the Windows VMs failed to start. That's never happened before (when on ESXi) because the disks (HDDs) have never copped so much parallel load on startup before. All these VMs all requesting mass disk I/O at the same time - it's crazy.

Remember that those order and start/shutdown delays aren't honored if the VMs are under HA and that those order and delays only apply to each node separately (i.e. you can't set orders among nodes of a cluster).

That information is not in the "Help" section. It's all new info to me - so thanks for the heads up. Care to link to the docs for that? (if indeed it's in the docs, and not just knowledge acquired through the cuts and bruises of past experience)
EDIT: OK that part is in the Help section in a note. My bad. But I'm not concerned about HA at this stage - info is moot for this use case.


My pertinent advice to Proxmox devs, as someone wanting to improve the product for the benefit of all:

1. List what the default value is in parenthesis, directly in the web UI. e.g. "Default (0)"
2. Make the default configurable, as zero is for sure not a sane default. The UI would then update. e.g. "Default (90)"
3. When adding the above, then also add a "Continue if Qemu Tools is started"
4. Update the linked help section, as information is sorely lacking in this area

I'll be setting this value manually from here on in, and hope that Proxmox can take onboard the suggestions I've given above.

 

[Neobin]

Remember that those order and start/shutdown delays aren't honored if the VMs are under HA and that those order and delays only apply to each node separately (i.e. you can't set orders among nodes of a cluster).

That information is not in the "Help" section. It's all new info to me - so thanks for the heads up. Care to link to the docs for that? (if indeed it's in the docs, and not just knowledge acquired through the cuts and bruises of past experience)

VMs managed by the HA stack do not follow the start on boot and boot order options currently. Those VMs will be skipped by the startup and shutdown algorithm as the HA manager itself ensures that VMs get started and stopped.

Please note that machines without a Start/Shutdown order parameter will always start after those where the parameter is set. Further, this parameter can only be enforced between virtual machines running on the same host, not cluster-wide.

https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_startup_and_shutdown

all the other VMs started up at the same time!

Never tested it myself, but maybe setting max_workers to (e.g.) 1 takes also effect in this case?
But will also limit other cases where it might not be wanted...

max_workers: <integer> (1 - N)
Defines how many workers (per node) are maximal started on actions like stopall VMs or task from the ha-manager.

https://pve.proxmox.com/pve-docs/datacenter.cfg.5.html

 

[Gavino]

Thanks the HA part is mentioned - my bad. Edited my note above. Since I'm not doing HA for this one customer, that info is not relevant for this particular use case. I aim to add some HA in future though, so it is still interesting none the less. I wouldn't want the "HA manager" to start multiple VMs on a particular PVE host at the exact same time.

It looks like PVE could do with a fair amount of polish in this area for both HA and non-HA VM startup configuration. I will test the "max_workers" setting when I start doing some HA. Hopefully it achieves what I want without any negative impacts.

 

[VictorSTS]

This is an area that is very lacking in PVE, unfortunately. If not using HA, simply setting startup delay and some ordering will suit your use case. I mean, it's just 7 VMs, its doable. But once HA quicks in, currently there is no way to easily such order. Not even mention cluster wide orquestation [1].

When I needed something like this, I ended up creating a dummy diskless VM and using hook script [2] to boot the other VMs in the chain, sometimes even dealing with HA enable/disable. Somewhat tedious, but at least you only have to edit a single script.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=1462
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_hookscripts