Hello Everyone,
Right now I am finally jumping into my Proxmox Ceph Cluster Project I have been waiting to work on for months now. So far, things have been going smoothly in terms of getting the Cluster created, however I am somewhat unsure as to whether I have a proper understanding of the configuration needed to meet my networking requirements.
As detailed in my past post, each of my Nodes has a total of 5 Network ports in use:
I have two "linknumber" entries defined under the Cluster Totem, where each corresponds to a ring0/1_address on each of the nodes. However, when I run a "pvecm status" to check on things, under "Membership Information" I only see the IP's of the ring0/linknumber0 corosync network (not sure if ring1/linknumber1 should also be present). I have tried fully removing the switch that controls the small ring0 network, but when I run the command it still does not show the IP's of the ring1 network.
When I run the "journalctl -b -u corosync" command on each node I see entries that "host X: Link 0/1 is up" for all the other nodes in the cluster, so I have a feeling that things are configured properly, but I just have a suspicion like I may be missing something with the ring1 network by not seeing any command output confirmation.
Any input would be greatly appreciated
. I know this will probably come off as a dumb post in a day or two, but I am slowly getting the hang of PVE. After I know I have the Corosync stuff working I will be rolling directly into testing the Replication Network and confirm that all replication is being done via that network and not the Public haha.
Right now I am finally jumping into my Proxmox Ceph Cluster Project I have been waiting to work on for months now. So far, things have been going smoothly in terms of getting the Cluster created, however I am somewhat unsure as to whether I have a proper understanding of the configuration needed to meet my networking requirements.
As detailed in my past post, each of my Nodes has a total of 5 Network ports in use:
- Ethernet Port 1: Management Port on Management VLAN (10.10.100.0/24)
- Ethernet Port 2: Corosync Ring 1 (Connected to Switch #1 to act as a heartbeat) (172.16.1.0/24)
- Ethernet Port 3: Corosync Ring 2 (Connected to Switch #2 to act as a backup heartbeat) (172.16.2.0/24)
- 10GB Port 1: Proxmox Public Network (LAN Access) (10.10.150.0/24)
- 10GB Port 2: Proxmox Private Network (Cluster Replication) (172.16.100.0/24)
Code:
root@Banner-PVE03:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: Banner-PVE03
nodeid: 3
quorum_votes: 1
ring0_addr: 172.16.1.113
ring1_addr: 172.16.2.113
}
node {
name: Rogers-PVE01
nodeid: 1
quorum_votes: 1
ring0_addr: 172.16.1.111
ring1_addr: 172.16.2.111
}
node {
name: Stark-PVE02
nodeid: 2
quorum_votes: 1
ring0_addr: 172.16.1.112
ring1_addr: 172.16.2.112
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: Avengers
config_version: 4
interface {
linknumber: 0
}
interface {
linknumber: 1
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}I have two "linknumber" entries defined under the Cluster Totem, where each corresponds to a ring0/1_address on each of the nodes. However, when I run a "pvecm status" to check on things, under "Membership Information" I only see the IP's of the ring0/linknumber0 corosync network (not sure if ring1/linknumber1 should also be present). I have tried fully removing the switch that controls the small ring0 network, but when I run the command it still does not show the IP's of the ring1 network.
Code:
root@Banner-PVE03:~# pvecm status
Cluster information
-------------------
Name: Avengers
Config Version: 4
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Tue Feb 15 12:45:40 2022
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 1.af
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 172.16.1.111
0x00000002 1 172.16.1.112
0x00000003 1 172.16.1.113 (local)When I run the "journalctl -b -u corosync" command on each node I see entries that "host X: Link 0/1 is up" for all the other nodes in the cluster, so I have a feeling that things are configured properly, but I just have a suspicion like I may be missing something with the ring1 network by not seeing any command output confirmation.
Any input would be greatly appreciated
. I know this will probably come off as a dumb post in a day or two, but I am slowly getting the hang of PVE. After I know I have the Corosync stuff working I will be rolling directly into testing the Replication Network and confirm that all replication is being done via that network and not the Public haha.
