Maintenance mode - script / process

[mtis]

Hi,

I manage two Proxmox clusters and am looking to document the maintenance process. Both are running Ceph. I'm getting mixed information. Options include:

1. Set noout. Maybe nodown and norebalance too?
2. ha-manager crm-command node-maintenance [enable|disable] [nodename]

Is there clear guidance on this? I can't see an option in the GUI, or a script.

 

[SteveITS]

That’s more or less what I’ve put together from Proxmox posts here. I use norecover and noscrub also because it sometimes starts a scrub when a node comes up.

I uncheck norecover after each node boots up to let it recover to 100% which is a few seconds.

Maintenance will migrate all VMs, and bring them back when turned off. There isn’t a GUI option for it.

 

[mtis]

Will keep an eye on this

 

[sinwar]

Will keep an eye on this

There has been an update to this, rather technical, though.

Following up on the last comment with more code inspection, I found that the endpoint implementation is not only easy, but already existing in PVE::CLI::ha_manager. Specifically, this part:

__PACKAGE__->register_method({
    name => 'node-maintenance-set',
    path => 'node-maintenance-set',
    method => 'POST',
    description => "Change the node-maintenance request state.",
    permissions => {
        check => ['perm', '/', ['Sys.Console']],
    },
    parameters => {
        additionalProperties => 0,
        properties => {
            node => get_standard_option('pve-node'),
            disable => {
                description => "Requests disabling or enabling maintenance-mode.",
                type => 'boolean',
            },
        },
    },
    returns => { type => 'null' },
    code => sub {
        my ($param) = @_;

        PVE::Cluster::check_node_exists($param->{node});

        my $cmd = $param->{disable} ? 'disable-node-maintenance' : 'enable-node-maintenance';
        PVE::HA::Config::queue_crm_commands("$cmd $param->{node}");

        return undef;
    },
});

Which is then used as Perl invocation using the name field for the crm-command like this:

our $cmddef = {
    # Other "ha-manager" commands
    'crm-command' => {
        # Other "ha-manager crm-command" commands
        'node-maintenance' => {
            enable => [__PACKAGE__, 'node-maintenance-set', ['node'], { disable => 0 }],
            disable => [__PACKAGE__, 'node-maintenance-set', ['node'], { disable => 1 }],
        },
    },
};

From what I read, the path field should be REST API endpoint name. The problem is that this implementation is under PVE::CLI and not PVE::API2. Those under the latter are exposed under /api2/json/cluster/ha, but not node-maintenance-set.

Therefore, if I understood the code correctly, either the endpoint does not make it to the production, or there is a hidden internal API endpoint POST /something/something/node-maintenance-set with two parameters (node<string>, disable<true|false>) for setting node maintenance that is either undocumented, unaccessible from outside, or both.

This is as far as my little investigation has led me to. It is up to the developers to clarify further from here whether there is a hidden endpoint that we can use or that it should be exposed under the PVE::API2 package, and when do we expect that to happen, if ever. I am willing to implement it myself, but I will need some hand holding from the developers with the development environment setup.