Remove third host without reboot others

artisan1 · Jun 16, 2023

Hello last year i removed the fourthy host, i dont remember how but doesn't had this impact, this week I rebooted one host (now have 3 on) with this just 2 on but at the same time the others 2 rebooted and lost qorum and resynced again, I turned on the other host again...

I tried reboot and put other ISO to format but this happened, whats the best method? shutdown using some commands? to remove cables from switch before shutdown? shutdown via iDRAC? what can i do to not impact on the other 2 nodes?

PVECM STATUS node that i wanna turn off and migrate to another infra.

OBS: IPs invented.

Cluster information
-------------------
Name: xxxxxxxxxxxx
Config Version: 6
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Fri Jun 16 XXXXXXX 2023
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 2.d0a
Quorate: Yes

Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 3
Quorum: 3
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000002 1 10.10.10.1
0x00000003 1 10.10.10.2 (local)
0x00000004 1 10.10.10.3

logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: HOSTA-1
nodeid: 1
quorum_votes: 1
ring0_addr: 10.10.10.0
ring1_addr: 192.168.100.1
}
node {
name: HOSTB-2
nodeid: 2
quorum_votes: 1
ring0_addr: 10.10.10.1
ring1_addr: 192.168.100.2
}
node {
name: pve-iig-dc-r9-r720-1
nodeid: 3
quorum_votes: 1
ring0_addr: 10.10.10.2
ring1_addr: 192.168.100.3
}
node {
name: pve-iig-dc-r9-r720-2
nodeid: 4
quorum_votes: 1
ring0_addr: 10.10.10.3
ring1_addr: 192.168.100.4
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: xxxxxxxx
config_version: 6
interface {
linknumber: 0
}
interface {
linknumber: 1
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

shanreich · Jun 16, 2023

You still have 4 nodes configured in your corosync configuration - that means corosync is expecting 4 votes, even though you only have 3 nodes.

Now, when you shut down a 2nd node, the other two nodes lose quorum since they only have 2 / 4 votes (which is not more than 50%, which is required) and they also fence [1] themselves since they have no quorum.

Did you make sure to follow the whole procedure for removing a node [2] when removing the fourth node last year? It seems like you didn't run pvecm delnode, which would remove the node from the nodelist.

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#ha_manager_fencing
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

artisan1 · Jun 16, 2023

ohhhh thats illuminating, can i set this delnode from this node without vm or container now without impact?

shanreich · Jun 16, 2023

artisan1 said:
ohhhh thats illuminating, can i set this delnode from this node without vm or container now without impact?

Yes you should be able to run it on any of the 3 left nodes. Please make sure to heed the warning in the documentation:

As mentioned above, it is critical to power off the nodebefore removal, and make sure that it will not power on again(in the existing cluster network) with its current configuration.If you power on the node as it is, the cluster could end up broken,and it could be difficult to restore it to a functioning state.

artisan1 · Jun 16, 2023

understand the first node removed last year doesnt will turn back, after this can I safely shutdown the other node and remove the vote from him from another 2 hosts?

shanreich · Jun 16, 2023

artisan1 said:
understand the first node removed last year doesnt will turn back, after this can I safely shutdown the other node and remove the vote from him from another 2 hosts?

If you permanently remove the fourth host from your config, then you will be able to shutdown the other node - yes.

Do you want to remove the other host permanently as well? Then I would suggest looking into setting up a QDevice [1]

[1] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

artisan1 · Jun 16, 2023

ok, fourth host already removed, just run pvecm delnode?

after this, can i shutdown full the another host, i can wait the completly shutdown and just after run pvecm delnode, correct?

i think that's dont have a time limit for this delnode, correct?

my error was that the node second host that i wanna remove was just rebooted (tried boot with another ISO) and the qorum was losted, fortunately all cluster came back and qorum "ok" with 3 / 4 votes.

OBS: i'm now reading the documentation, sorry, i'm just afraid may it happen again.

shanreich · Jun 16, 2023

artisan1 said:
ok, fourth host already removed, just run pvecm delnode?

yes

artisan1 said:
after this, can i shutdown full the another host, i can wait the completly shutdown and just after run pvecm delnode, correct?

Only if you want to PERMANENTLY remove the second host. If you only want to shut it down temporarily, then you do NOT need to run pvecm delnode - this command is only for PERMANENTLY removing a node from a cluster

artisan1 · Jun 16, 2023

shanreich said:
yes

Only if you want to PERMANENTLY remove the second host. If you only want to shut it down temporarily, then you do NOT need to run pvecm delnode - this command is only for PERMANENTLY removing a node from a cluster

yep, but doesn't have problem to keep the node at the cluster config?

edited: oh the problem is the votes, i understand know

shanreich · Jun 16, 2023

artisan1 said:
edited: oh the problem is the votes, i understand know

Yes, exactly! If you only have 3 nodes, then there are only 3 votes. So it is not a problem to shutdown one node then since there are still 2/3 (> 50%) votes left.

artisan1 · Jun 16, 2023

great support man, i will delnode now the host removed last year and i will shutdown the other host to make some changes are, very thankful

artisan1 · Jun 16, 2023

another question, to shutdown this second host, can i just power off the button or shutdown via iDRAC? the safest mode i think is shutdown physically... idk if "shutdown -h now" at the terminal of the host can be secure for this...

shanreich · Jun 16, 2023

It shouldn't really matter which one you use afaict.

artisan1 · Jun 19, 2023

@shanreich i can't see my first node removed last year, if i execute this command: pvecm delnode pve-iig-dc-r9-r630-1, the proxmox will be able to remove the nodeid 1?

After removed the 1, i will shutdown node3 and after remove him too.

shanreich · Jun 19, 2023

~~The node has been successfully removed if it doesn't show up anymore in the pvecm nodes output. (~~edit: see below)

Please note again that we do not recommend running a cluster with 2 nodes alone, you should at least utilize a QDevice [1].

[1] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

artisan1 · Jun 19, 2023

shanreich said:
The node has been successfully removed if it doesn't show up anymore in the pvecm nodes output.

Please note again that we do not recommend running a cluster with 2 nodes alone, you should at least utilize a QDevice [1].

[1] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

but yet shows 4 expected votes =/

shanreich · Jun 19, 2023

How does the corosync config look like?

Code:

cat /etc/pve/corosync.conf

What's in the syslog of the node?

Code:

journalctl --since '1 day ago' > syslog.txt

artisan1 · Jun 19, 2023

shanreich said:
How does the corosync config look like?

Code:

cat /etc/pve/corosync.conf

What's in the syslog of the node?

Code:

journalctl --since '1 day ago' > syslog.txt

Code:

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-iig-dc-r9-630-1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: XXX.XX.XXX.X4
    ring1_addr: XXX.XX.XXX.X4
  }
  node {
    name: pve-iig-dc-r9-r630-2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: XXX.XX.XXX.X5
    ring1_addr: XXX.XX.XXX.X5
  }
  node {
    name: pve-iig-dc-r9-r720-1
    nodeid: 3
    quorum_votes: 1
    ring0_addr: XXX.XX.XXX.X6
    ring1_addr: XXX.XX.XXX.X6
  }
  node {
    name: pve-iig-dc-r9-r720-2
    nodeid: 4
    quorum_votes: 1
    ring0_addr: XXX.XX.XXX.X7
    ring1_addr: XXX.XX.XXX.X7
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: XXX.XX.XXX.X-DC
  config_version: 6
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

shanreich · Jun 19, 2023

Ah sorry, I thought the node would still show up if it hasn't been removed - but it also doesn't show up in the output of both commands if it isn't online.

On which node did you run pvecm delnode pve-iig-dc-r9-630-1 ? Are there any errors when you run this command?

artisan1 · Jun 19, 2023

shanreich said:
Ah sorry, I thought the node would still show up if it hasn't been removed - but it also doesn't show up in the output of both commands if it isn't online.

On which node did you run pvecm delnode pve-iig-dc-r9-630-1 ? Are there any errors when you run this command?

i will discover this more late, because i can't execute this right now, scare of another fencing hehe

i wil try run this at the nodeid 3, if removed and votes back to 3 / 3, I will shutdown the nodeid3 too.

Remove third host without reboot others

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

Attachments

Proxmox Staff Member

New Member