[SOLVED] Setting up ceph, getting an error in the gui

[rsr911]

Yes do the whole networking before starting the clustering. Do it web-ui only.

Perfect.

No 5) does not need connection to anything else.

So I'm 100% clear the ceph public network does not need a connection to anything outside in this arrangement?

As I said no 3node cluster via two rooms


What do mean with that? theres no issue using 3 nodes for production?

I read somewhere that three nodes will work but is not ideal.

This does also not work later on. You need to think about every possible downtime-outcome: if you use three nodes in one room, and this room fails you still have the same problem I already explained Two room setups start with Room 1: 2 nodes | Room 2: 2 nodes | Room 3: Quorum Node doing a SIZE=4 MINSIZE=2 setup (with custom crushrule) or 3 Rooms with each having a full pve-ceph node (including storage not only quorum purpose host) and SIZE=3 MINSIZE=2.

I still have a Xeon powered workstation that was to be my witness host in VMware. I also have a barebones server. So could I, and should I setup a fourth server, two in each building, as a five node cluster using this Xeon workstation (or even PVE with PBS on top) in a separate third room for quorum? If so that would need (I think) connections to web UI, corosync, public and cluster network. That would be 2 data nodes per building and 5 monitor nodes with the fifth just a low power machine with no storage for quorum. Is it safe to assume I can start with the three I have, everything in one rack, and when I get time and money add the fourth node and go 2, 2, 1? I'm asking because my current sfp switches wouldn't have enough ports for 5 machines.

By default (without using custome crushrules you still write 4 copies (and they can be anywhere if you dont define room entities in crushmap). Doing a 5way replication (you say: each server has a copy) is absolute OVERKILL. Go for a 4way mirror (2 copies per room) while having the quorum node in room3. Quorum Node can be something really cheap, aslong as you have network in corosync and ceph.

I believe this kind of answers my above question. Corosync and ceph have to attach to the quorum node.

Yeah go for a single room setup and get familiar with ceph and do lots of testing before going live with anything important. As I said above, you would need 2 more servers (one with osds) and the second just for quorum (third room).

Be careful with the wording. Proxmox VE does not offer mirroring for two complete seperated clusters. The example we talked about here is ONE Cluster but with 6 Nodes, that are fully usable and avaiable, but they use intelligten replica-placement, so you can loose up to a room, without having lots of downtime.

Theres a way to setup ceph-mirror between two seperated cluster, but this is a lot more complex then ceph already is for beginners.


Your Welcome, ceph is complex in the beginning but youll love it once you have a perfect setup and see how easy and secure it is (if setup correct)

Thank you Jonas!

I am going to go three for now as long as I can add nodes later, which it appears I can. I'd rather spend money on enterprise SSDs right now than 5 nodes which requires larger sfp+ switches, more cable to third room and an entire server build out. One room/building will get me going.

 

[jsterr]

I still have a Xeon powered workstation that was to be my witness host in VMware. I also have a barebones server. So could I, and should I setup a fourth server, two in each building, as a five node cluster using this Xeon workstation (or even PVE with PBS on top) in a separate third room for quorum?

It would be possible, if this workstation can run 24/7 and fits the performance you need for pbs. for quorum purposes it does not need to be something special. for a backupserver there is a need for lot of disk slots and you should also think about the correct amount of MEMORY and CPU. But yeah as you are going for a 3node setup in room now, this is not something to talk about atm ;-)

If so that would need (I think) connections to web UI, corosync, public and cluster network. That would be 2 data nodes per building and 5 monitor nodes with the fifth just a low power machine with no storage for quorum.

Not sure why you are saying 5 monitor nodes. Proxmox VE and Ceph = HCI so no separate nodes, all service are running hyperconverged (means on the same machine). But yeah I would go for 2 pveceph with osds per room, third room only pveceph without osds. You are correct with the monitors: min 3 max 5 - but you would need 5

Is it safe to assume I can start with the three I have, everything in one rack, and when I get time and money add the fourth node and go 2, 2, 1? I'm asking because my current sfp switches wouldn't have enough ports for 5 machines.

Yes this is possible even without downtime if done correctly.

I am going to go three for now as long as I can add nodes later, which it appears I can. I'd rather spend money on enterprise SSDs right now than 5 nodes which requires larger sfp+ switches, more cable to third room and an entire server build out. One room/building will get me going.

Good luck! Hope your doing fine with the setup and you are ok with the performance. 10Gbit is the minimum for ceph but for a small company this should be sufficient.

Just one thing that is most important: read about mon.osd fullratio this can cause complete ceph downtime. In short: make sure that a ceph osd never reaches 90-95% .. if this happens, ceph will shutdown the pool access. This happens no matter how many nodes etc. if one of the nodes have one osd with 95% usage, you have complete storage downtime for this pool. That means: monitor the osd disk usage from all disks! By default youll get a error at 75% usage, but this is to late. Why? Because you also need to calculate possible failures of disk. If you have 4 disks and all 4 disks are 75% usage, you would assume your fine! No you arent .. Why? Because if one of the four disks fails, ceph will recover the 75% data that is now missing because the disk failed, it will recover to the same SERVER the failed disk was located too. Which means ... your other three disks get 25% additional data each ... --> they will reach 95% usage because of that -> this causes complete storage downtime over all nodes.

 

[rsr911]

It would be possible, if this workstation can run 24/7 and fits the performance you need for pbs. for quorum purposes it does not need to be something special. for a backupserver there is a need for lot of disk slots and you should also think about the correct amount of MEMORY and CPU. But yeah as you are going for a 3node setup in room now, this is not something to talk about atm ;-)

Let me clarify. I have the three Lenovo servers for the current three nodes, a fourth to run PBS (which could run on top of PVE and be the 5th quorum), a barebones Lenovo, and a Xeon workstation. If I used the workstation it would ONLY be for the quorum. Alternately I can run the backup Lenovo server as the quorum IF I install PBS on top of PVE.

Am I correct that the quorum machine needs to connect to the ceph front and back end and web UI?

But yes, no need right now.

Not sure why you are saying 5 monitor nodes. Proxmox VE and Ceph = HCI so no separate nodes, all service are running hyperconverged (means on the same machine). But yeah I would go for 2 pveceph with osds per room, third room only pveceph without osds. You are correct with the monitors: min 3 max 5 - but you would need 5


Yes this is possible even without downtime if done correctly.


Good luck! Hope your doing fine with the setup and you are ok with the performance. 10Gbit is the minimum for ceph but for a small company this should be sufficient.

Just one thing that is most important: read about mon.osd fullratio this can cause complete ceph downtime. In short: make sure that a ceph osd never reaches 90-95% .. if this happens, ceph will shutdown the pool access. This happens no matter how many nodes etc. if one of the nodes have one osd with 95% usage, you have complete storage downtime for this pool. That means: monitor the osd disk usage from all disks! By default youll get a error at 75% usage, but this is to late. Why? Because you also need to calculate possible failures of disk. If you have 4 disks and all 4 disks are 75% usage, you would assume your fine! No you arent .. Why? Because if one of the four disks fails, ceph will recover the 75% data that is now missing because the disk failed, it will recover to the same SERVER the failed disk was located too. Which means ... your other three disks get 25% additional data each ... --> they will reach 95% usage because of that -> this causes complete storage downtime over all nodes.

Duly noted on the disk usage. Ive read this elsewhere as well.

Here is our current plan of attack:

Three nodes in one room. Running on current SSDs for testing while we source and buy the correct drives. Place PBS on top of PVE and locate it in another room far away but in the same building. Install the Optane drives, likely in ZFS raidZ (raid 5) one in each server for use as database only storage. Database will be fault tolerant, not HA. Once we have enterprise SSDs for the rest of the data and VMs I will go HA on the windows servers.
Going this route because it seems like the easier way to setup the database and it doesnt need to be HA long term since a power outage would shutdown production anyway. However being able to live migrate VMs will be nice for physical maintenance like cleaning. Windows VMs are my DCs and DNS/DHCP. I'd like at least one of those up all the time.

Later on build out my spare chassis, two nodes per room, one quorum node in a third room. Once I prove concept to ownership funding the next steps will be easy.

Wrong forum but for licenses I believe I need 4 2cpu right now plus a PBS license. (All servers are dual socket)

 

[rsr911]

I have one thing I'm confused about. The ceph public network. If it's sfp+ connected to one of my 8 port switches only how do clients connect? Or is this not an issue because the VMs are running on the servers so the "clients" are internal to the cluster and the VMs will be installed on Cephfs? How about backups, as listed above all three servers will have backup ports that connect to the backup server thru the main switch. Does the cluster handle connecting to the external backup server, or is it essentially a client that needs to connect to ceph?

 

[rsr911]

Sitting here thinking. All of my 10g cards are dual port. Normally I would make bonds across ports on a card. However what if I bond port0 to port 0 and port 1 to port 1, spanning the cards. This way in the event of a nic failure none of my networks lose complete connectivity.

For example card 0 port 0 bonded to card 1 port 0 for ceph cluster bond. Card 0 port 1 bonded to card 1 port 1 for ceph public.

Do the same on my RJ48 cards for VM network and backup traffic.

Doing it this way I could lose one RJ 48 and one sfp+ card in each server and still have a link.

Is this a sound plan? I've never bonded that way before. It sure does seem to be a good idea.

 

[jsterr]

Sitting here thinking. All of my 10g cards are dual port. Normally I would make bonds across ports on a card. However what if I bond port0 to port 0 and port 1 to port 1, spanning the cards. This way in the event of a nic failure none of my networks lose complete connectivity.

For example card 0 port 0 bonded to card 1 port 0 for ceph cluster bond. Card 0 port 1 bonded to card 1 port 1 for ceph public.

Do the same on my RJ48 cards for VM network and backup traffic.

Doing it this way I could lose one RJ 48 and one sfp+ card in each server and still have a link.

Is this a sound plan? I've never bonded that way before. It sure does seem to be a good idea.

Yes, thats how you do it.

I have one thing I'm confused about. The ceph public network. If it's sfp+ connected to one of my 8 port switches only how do clients connect? Or is this not an issue because the VMs are running on the servers so the "clients" are internal to the cluster and the VMs will be installed on Cephfs? How about backups, as listed above all three servers will have backup ports that connect to the backup server thru the main switch. Does the cluster handle connecting to the external backup server, or is it essentially a client that needs to connect to ceph?

Your nodes, they are hyperconverged, the client in your case is proxmox-ve which runs on the same machine as the storage itself. but vms wont run on cephfs, they will use RBD via librados or krbd library. btw: you are still having network in ceph public from every node, so every node you put into this network can access ceph if you give it the correct ceph-key and install the right packages.

Backups are unindependent from ceph, they just communicate via ip that u use for integrating the pbs into your pve-cluster(s) and grab the backup whatever storage you use for your vms.

 

[rsr911]

Yes, thats how you do it.

Awesome, see I am capable of learning lol!

Yes thats because they are hyperconverged, the client in your case is proxmox-ve which runs on the same machine as the storage itself. but vms wont run on cephfs, they will use RBD via librados or krbd library.

I knew it worked something like that. What I meant was VM storage is internal to the cluster. Hyper converged as you say. Compute and storage on the same machines.

Backups are unindependent from ceph, they just communicate via ip that u use for integrating the pbs into your pve-cluster(s) and grab the backup whatever storage you use for your vms.

Well this presents an interesting opportunity. If I install PBS on top of PVE it can be the fifth quorum member later on correct? All I would have to do is add NICs to it for connection to ceph public and ceph cluster. Locate it in room three. Then populate my barebones server with CPUs, ram, NICs, and drives to be intentical to my current three nodes. Then I can have my two nodes per building with the PVE/PBS quorum server at the far end of building 1 so its well isolated from fire and other issues. To make things even better I can power it with a different leg of the three phase power coming into the buildings. I actually already have a perfect place, our QC lab. Its almost exactly the same distance from rack 0 room 0 as rack 1 room 1 is. I know this because I just recently pulled a cable there. That room will become rack 2 room 2 on my chart. I would, of course, need larger switches to connect all 10 ports on each of the ceph networks.

Now this presents it's own question. Earlier it was recommended I use enterprise drives. I am finding refurbished 1.92tb drives for a decent price. 8tb is far and away more than enough. However its not in the cards at the moment and neither is the fourth server. Could I just put four of my current 1tb drives in each of the nodes for now and later change them out one by one with the 1.92tb enterprise drives? This would be cheaper for now and also leave four bays per server open for future storage expansion. I am certain once I get this cluster up and going, do some testing to make sure it all works I can then have the conversation with ownership about adding a fourth server, the real need for enterprise drives, and the extra network gear I would need. If they see it running in one rack and I explain why that will be about all it would take. We had an actual fire in one building last year. They want the data spread out. But due to my debacle with VMware I way over budget right now. If they can see it running and have me explain how it all works I'm certain they will be in a rush to get a fourth data node up and running.

 

[rsr911]

Well same issue. I used the GUI to set everything up. Got this error during installation on host 1.

Multiple IPs for ceph public network '192.168.0.6/24' detected on host1: 192.168.0.6 192.168.0.11 use 'mon-address' to specify one of them. (500)

From Ceph Configuration tab:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster network = 192.168.0.11/24
     cluster_network = 192.168.0.11/24
     fsid = 0243c2b3-149a-4fc0-9b16-e68d9f86b1d1
     mon_allow_pool_delete = true
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public network = 192.168.0.6/24
     public_network = 192.168.0.6/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

Commented out the extra networks:

[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
#        cluster network = 192.168.0.11/24
         cluster_network = 192.168.0.11/24
         fsid = 0243c2b3-149a-4fc0-9b16-e68d9f86b1d1
         mon_allow_pool_delete = true
         osd_pool_default_min_size = 2
         osd_pool_default_size = 3
#        public network = 192.168.0.6/24
         public_network = 192.168.0.6/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

This error is displayed over config tab with spinning thing"

rados_connect failed - No such file or directory (500)

 

[rsr911]

So basically things stall during the install with that error. I don't know why.

Here is the output of "ip a"

root@host1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp6s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond2 state UP group default qlen 1000
    link/ether 98:b7:85:1e:bf:75 brd ff:ff:ff:ff:ff:ff permaddr 40:a6:b7:3c:dd:24
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:0f:41:fd:36:15 brd ff:ff:ff:ff:ff:ff
    altname enp10s0f0
    inet 192.168.1.141/24 scope global eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::fa0f:41ff:fefd:3615/64 scope link
       valid_lft forever preferred_lft forever
4: enp6s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond3 state UP group default qlen 1000
    link/ether 86:b8:4a:e4:5a:ab brd ff:ff:ff:ff:ff:ff permaddr 40:a6:b7:3c:dd:25
5: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:0f:41:fd:36:16 brd ff:ff:ff:ff:ff:ff
    altname enp10s0f1
    inet 192.168.1.156/24 scope global eno3
       valid_lft forever preferred_lft forever
    inet6 fe80::fa0f:41ff:fefd:3616/64 scope link
       valid_lft forever preferred_lft forever
6: enp4s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether de:52:2a:bf:c0:60 brd ff:ff:ff:ff:ff:ff permaddr 98:b7:85:00:f1:72
7: enp4s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 66:5d:bc:5b:5d:86 brd ff:ff:ff:ff:ff:ff permaddr 98:b7:85:00:f1:73
8: enp132s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether de:52:2a:bf:c0:60 brd ff:ff:ff:ff:ff:ff permaddr 98:b7:85:00:f0:c6
9: enp132s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 66:5d:bc:5b:5d:86 brd ff:ff:ff:ff:ff:ff permaddr 98:b7:85:00:f0:c7
10: enp135s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond2 state UP group default qlen 1000
    link/ether 98:b7:85:1e:bf:75 brd ff:ff:ff:ff:ff:ff
11: enp135s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond3 state UP group default qlen 1000
    link/ether 86:b8:4a:e4:5a:ab brd ff:ff:ff:ff:ff:ff permaddr 98:b7:85:1e:bf:76
12: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 66:5d:bc:5b:5d:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.161/24 scope global bond1
       valid_lft forever preferred_lft forever
    inet6 fe80::645d:bcff:fe5b:5d86/64 scope link
       valid_lft forever preferred_lft forever
13: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:b7:85:1e:bf:75 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.6/24 scope global bond2
       valid_lft forever preferred_lft forever
    inet6 fe80::9ab7:85ff:fe1e:bf75/64 scope link
       valid_lft forever preferred_lft forever
14: bond3: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 86:b8:4a:e4:5a:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.11/24 scope global bond3
       valid_lft forever preferred_lft forever
    inet6 fe80::84b8:4aff:fee4:5aab/64 scope link
       valid_lft forever preferred_lft forever
15: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether de:52:2a:bf:c0:60 brd ff:ff:ff:ff:ff:ff
16: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether de:52:2a:bf:c0:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.151/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::dc52:2aff:febf:c060/64 scope link
       valid_lft forever preferred_lft forever
root@host1:~#

Attached picture of my network setup for host1

 

[rsr911]

Removed ceph from node 1, rebooted it. Tried to install ceph from the datacenter tab. Same error.

I have no clue what I am doing wrong. All the networking looks right. I ran pings and iperf3 to check. Made sure all of that worked and that the proper speeds. Cabling has been double checked. Everything "talks" fine. Running iperf3 with --bind-dev on the ceph networks both show 9.4x gbit speeds.

Confused....

 

[rsr911]

Just to rehash what I did.

1) Made sure all three servers were identical in hardware and where cards were placed.
2) Clean installs on all three.
3) Reserved IP range on my network for all the ports I would need
4) Checked and rechecked cabling and that ports were lit on the switches and cards
5) Set switch LAG'd ports to passive LACP, numbered everything so I knew which cables go where.
6) Configured networking on all three machines identical with incremental increase in IP ie .141, .142 .143 for managment etc.
7) Built the cluster, set corosync links in the gui
8) Started ceph install, then I get the error.

 

[jsterr]

Hi @rsr911 Im not sure but I guess you cant use the same network 192.168.0.0/24 for both ceph networks. Seperate it and try again. You need to have two different networks one for ceph public and one for ceph-cluster. Ceph uses the network in its config (/24) means it just refers to the subnet not to direct ip-addresses: public network = 192.168.0.6/24 thats why it might get confused because you are als using the subnet for cluster network cluster_network = 192.168.0.11/24 - separate the networks, guess that should fix it. But you will need to reinstall because this is a misconfiguration that is not that easy to fix.

 

[rsr911]

Hi @rsr911 Im not sure but I guess you cant use the same network 192.168.0.0/24 for both ceph networks. Seperate it and try again. You need to have two different networks one for ceph public and one for ceph-cluster.

Up early here in the US, stayed late last night. As you may have noticed in my IP ranges I've left space for a fourth data node and fifth node running PVE with PBS on top. However to get there I need to show the owners I can get this thing running.

Anyway what you are suggesting is both reasonable and easy to do since those networks are on separate smart switches. I'm going into work in about 2 hours. I'll jump on that. Change one switch over to 192.168.2.* and then change the bond IPs in the hosts and report back.

Basically this is my situation. If I can show Proxmox as a viable solution I can order another server, all the networking gear AND enterprise drives. Basically it's a "show me it works and we'll invest in the hardware and subscriptions." Lol.

Looking at your response I feel kinda dumb for not thinking of that myself. I do know the cluster is supposed to be a separate subnet. The servers, and proxmox, have no clue these are independent networks physically, they just see IP ranges that COULD be on the same network.

That's a long way of saying I bet you're right!!!

Otherwise everything is setup and cabled the way we discussed. I bonded across NICs rather than within NICs, 1g connections are on the original switch which is uplinked to the main 10g switch. Ceph public and ceph cluster are each on their own switches and not physically connected to anything else so your idea here makes a lot of sense.

I now also have a detailed spreadsheet of all my hardware, serial numbers, Mac addresses, what switch and ports they connect to. Even labeled things inside the switch guis. It kind of looks professional

I'll try the network thing and respond back.

 

[jsterr]

Up early here in the US, stayed late last night. As you may have noticed in my IP ranges I've left space for a fourth data node and fifth node running PVE with PBS on top. However to get there I need to show the owners I can get this thing running.

Anyway what you are suggesting is both reasonable and easy to do since those networks are on separate smart switches. I'm going into work in about 2 hours. I'll jump on that. Change one switch over to 192.168.2.* and then change the bond IPs in the hosts and report back.

Basically this is my situation. If I can show Proxmox as a viable solution I can order another server, all the networking gear AND enterprise drives. Basically it's a "show me it works and we'll invest in the hardware and subscriptions." Lol.

Looking at your response I feel kinda dumb for not thinking of that myself. I do know the cluster is supposed to be a separate subnet. The servers, and proxmox, have no clue these are independent networks physically, they just see IP ranges that COULD be on the same network.

That's a long way of saying I bet you're right!!!

Otherwise everything is setup and cabled the way we discussed. I bonded across NICs rather than within NICs, 1g connections are on the original switch which is uplinked to the main 10g switch. Ceph public and ceph cluster are each on their own switches and not physically connected to anything else so your idea here makes a lot of sense.

I now also have a detailed spreadsheet of all my hardware, serial numbers, Mac addresses, what switch and ports they connect to. Even labeled things inside the switch guis. It kind of looks professional

I'll try the network thing and respond back.

Changing the ips on Proxmox VE wont do the thing, as ceph already uses the configuration internaly. You might check: https://forum.proxmox.com/threads/how-to-change-ceph-internal-cluster-network.132513/

Im happy to help.

 

[rsr911]

Changing the ips on Proxmox VE wont do the thing, as ceph already uses the configuration internaly. You might check: https://forum.proxmox.com/threads/how-to-change-ceph-internal-cluster-network.132513/

Im happy to help.

I am thinking:

Ceph_public 192.168.0.*/8
Ceph_cluster 192.168.2.*/10

Or something like that. Different subnets.

Or I could do completely different networks like

10.10.10.*/8
10.100.10.*/10

I can do anything I want really as these are independent switches. What do you think Proxmox would like to see?

I'm not really up to speed on subnets. I just know 10. , 172. , and 192. Are the private network ranges.

What I meant by changing IPs is I will set the correct IPs for each network on the hosts after reconfiguring the switches. I like to reserve .1-.5 on switches.

For all intents and purposes could I just do 10.0.0.6/24 on public switch and 172.0.0.6/24 on the cluster switch and not mess with subnet size? This way once I have 5 hosts they will be .6-.10 on each switch. I'll set the switch management IP to .2 or something.

Like this:

Ceph public
10.0.0.6/24
10.0.0.7/24
10.0.0.8/24

Ceph cluster
172.0.0.6/24
172.0.0.7/24
172.0.0.8/24

Then later add .9 and .10 when I get two more servers up?

Any reason to add this PVE/PBS-VM server to the cluster now? I don't have its sfp+ cards yet. I am reasonably certain I'll get this working and be ordering the fifth server by end of the week. I just sort of thought the easy way was to get the first three up and running, prove concept, and move forward.

 

[Azunai333]

Those netmasks

Ceph_public 192.168.0.*/8
Ceph_cluster 192.168.2.*/10

and these IPs won't work.

Ceph cluster
172.0.0.6/24
172.0.0.7/24
172.0.0.8/24

You have public IP addresses mixed in.

These are the private IPs for IPv4 (source: [1])
10.0.0.0 – 10.255.255.255
172.16.0.0 – 172.31.255.255
192.168.0.0 – 192.168.255.255

10.10.10.*/8
10.100.10.*/10

Won't work ether. Your /10 network is part of your /8 network.

Maybe look at [2] for infos how to subnet. (Or use an subnet calculator, those are useful to check if the netmasks aren't overlapping.)

[1] https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses
[2] https://www.subnetting.net/Tutorial.aspx

 

[jsterr]

Just use something like:

• Ceph_public 192.168.0.0/24
• Ceph_cluster 192.168.2.0/24

third octet needs to be different. And no other network card should use this networks.

 

[rsr911]

Just use something like:

• Ceph_public 192.168.0.0/24
• Ceph_cluster 192.168.2.0/24

third octet needs to be different. And no other network card should use this networks.

Jonas!!! How do I buy you a beer??? It's working!

I just used 10.0.0.0/8 public and 172.0.0.0/10 cluster. Everything went just fine. I didn't even touch the switches. Got my OSD's and monitors setup. Only weird thing I noticed is for some odd reason one of my boot drives is sde and it's sda on the other two. Probably a bios post issue. This cluster is just a test. I'm going to play around with it a bit, set up some VMs, failure test by pulling power and network cables and drives.

Now I get to read up on setting up a second PG with my NVMEs (one per server) as a fault tolerant pool for the eventual database. But huge hurdle overcome. I have a working cluster! Thank you so so much!

 

[rsr911]

Those netmasks


and these IPs won't work.

You have public IP addresses mixed in.

These are the private IPs for IPv4 (source: [1])
10.0.0.0 – 10.255.255.255
172.16.0.0 – 172.31.255.255
192.168.0.0 – 192.168.255.255


Won't work ether. Your /10 network is part of your /8 network.

Maybe look at [2] for infos how to subnet. (Or use an subnet calculator, those are useful to check if the netmasks aren't overlapping.)

[1] https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses
[2] https://www.subnetting.net/Tutorial.aspx

It's working. Will it be a problem to just go to 192.168.0.0/24 and 192.168.2.0/24 as jsterr mentioned?

 

[jsterr]

Jonas!!! How do I buy you a beer??? It's working!

Nice! Its ok I always like connecting to people, so if you want you can add me on linkedin (business) if you have! Greetings