Review appreciated/best practise: new pve environment with ceph

[ViciousVermicelli]

Hi everyone,
this is my first forum post here. Hope this post is not against any rules.

We are currently in the process of setting up a new proxmox environment to replace our vsphere setup. This obviously also includes proxmox support.

We're planning a completly new environment with pve, pbs, ceph, 100G Ethernet and new Firewalls.
The current plan looks like this:


We are starting with five DELL R7715 Servers with 1TB RAM. Each device has 4x100Gbit/s and 4x25Gbit/s Ethernet Interfaces.
This Host OS will be on a 500GB SSD Raid1. For Ceph storage we have 8x8TB NVMe SSDs

For the network side we're planning to connect each server with multiple links to multiple Switches. We do not have MLAG.
Each proxmox-function (ceph, backup, mgmt, corosync, etc) gets its own vlan. The corosync-vlans are not shared between switches.

My main question is: Would the network setup shown in the image above work and is it best-practise "compliant" or are there any improvements we could implement?

 

[UdoB]

Nice approach

(ceph, backup, mgmt, corosync, etc) gets its own vlan.

Even though you have redundant switches I would recommend to prepare a separate wire (not a virtual LAN, 1 GBit/s is sufficient) for one of your multiple corosync rings.

Disclaimer: your approach is multiple levels above my usual stuff

 

[Steven-b]

Is it going to be exactly like the drawing or is the setup going to be separated across different rooms?
You have 8x 8TB, is this per server or in total? that's not entirely clear to me, I am guessing that this per server but just checking I am very interested in following your setup and how it would perform.

 

[ViciousVermicelli]

@UdoB: this was our plan aswell, but our Servers don't even have Copper-Ethernet enymore. Another problem would be rackspace. If we would want to connect the servers via 1Gbit/s we would need two additional "slow" switches. Obviously dedicating one 25Gbit/s Link for corosync would be possible but somehow way overkill and we figured out mgmt-traffic would be really low aswell.

@Steven-b: Each server has 8x8TB Storage (64TB per Server total) - 320TB Raw capacity. All servers are in one location - one Rack acutually

 

[Steven-b]

@UdoB: this was our plan aswell, but our Servers don't even have Copper-Ethernet enymore. Another problem would be rackspace. If we would want to connect the servers via 1Gbit/s we would need two additional "slow" switches. Obviously dedicating one 25Gbit/s Link for corosync would be possible but somehow way overkill and we figured out mgmt-traffic would be really low aswell.

@Steven-b: Each server has 8x8TB Storage (64TB per Server total) - 320TB Raw capacity. All servers are in one location - one Rack acutually

OK, 320TB raw capacity, you are going to use Ceph with N-replication? leaving a +/- 106TB net capacity if it would be a 3-way replication setup.
Back-ups are going offsite? just in case? cause you mention everything in one rack.

 

[ViciousVermicelli]

OK, 320TB raw capacity, you are going to use Ceph with N-replication? leaving a +/- 106TB net capacity if it would be a 3-way replication setup.
Back-ups are going offsite? just in case? cause you mention everything in one rack.

This correct and also the amount we calculated.

We're planning on one PBS in the same location and one PBS replicated offsite

 

[ness1602]

My recommendation is to use physical links to corosync, not vlan ones. As for the other things this is okay, all links should be active-passive in proxmox, so that you don't care if something dies ,and this is it. CEPH will work great in that case, those ssds work really great in practice.