[SOLVED] NVME/100GB Ceph network config advice needed

[teddyh2o]

Hello,
I was looking into Proxmox set up with Ceph on my full NVME servers. At first, I was looking into 40GB but that wasn't enough with SSD.
Used following documents as guide line but wanted to get some feedback on my setup/settings (not implemented yet)

https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2020-09-hyper-converged-with-nvme.76516/
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

Here is the config

OSD node(1-5)
ens3f0 & eno3f1 Bond0 - 192.168.1.101 to 105 (Mesh - 2 x 100GB) - Connected to other nodes directly
eno1 - 10.123.123.101 to 105 (proxmox mgmt - 1GB) - Connected to 1GB switch
ens5f0np0 & ens5f0np0 Bond1 - 10.123.122.101 to 105 (proxmox cluster - 2 x 10GB) - Connected to 10GB switch
ens1f0np0 & ens1f1np1 Bond2 - 10.123.0.0/24 (VM network - 2 x 10GB) - Connected to 10GB switch

MON node(1-3)
eno3 - 10.123.123.201 to203 (proxmox mgmt - 1GB) - Connected to 1GB switch
eno1 & eno2 - Bond0 - 10.123.122.201 to 203 (proxmox cluster - 2 x 10GB) - Connected to 10GB switch

 

[teddyh2o]

any suggestion?

 

[Klaus Steinberger]

Hello,
I was looking into Proxmox set up with Ceph on my full NVME servers. At first, I was looking into 40GB but that wasn't enough with SSD.
Used following documents as guide line but wanted to get some feedback on my setup/settings (not implemented yet)

https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2020-09-hyper-converged-with-nvme.76516/
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

Here is the config

OSD node(1-5)
ens3f0 & eno3f1 Bond0 - 192.168.1.101 to 105 (Mesh - 2 x 100GB) - Connected to other nodes directly
eno1 - 10.123.123.101 to 105 (proxmox mgmt - 1GB) - Connected to 1GB switch
ens5f0np0 & ens5f0np0 Bond1 - 10.123.122.101 to 105 (proxmox cluster - 2 x 10GB) - Connected to 10GB switch
ens1f0np0 & ens1f1np1 Bond2 - 10.123.0.0/24 (VM network - 2 x 10GB) - Connected to 10GB switch

MON node(1-3)
eno3 - 10.123.123.201 to203 (proxmox mgmt - 1GB) - Connected to 1GB switch
eno1 & eno2 - Bond0 - 10.123.122.201 to 203 (proxmox cluster - 2 x 10GB) - Connected to 10GB switch

For a Mesh you need 4 Ports per node (not bonded!) for 5 nodes (with two ports bonded you need 8 ports per node)
it is not necessary to run MON's on separate nodes, load is minimal, I usually run them on every node

 

[teddyh2o]

doh. I thought i could do daisy chain. (eg: 01-02->03->04->05->01)
what is the performance like if 100GB connection goes through the switch instead?

 

[Klaus Steinberger]

You will add more latency by daisy chaining, than by a switch

-> Switch: two hops, but modern switches can forward already when they have read the header
so in most cases latency of one hop

-> daisy chain: openvswitch with RSTP will find shortest path, in most cases 2 hops
-> as it is no dedicated switching hardware packets will probably be received completly before forwarding
-> added load to the server, as the server has to do the packet forwarding instead of a dedicated switching engine
so latency can vary widely

-> full mesh: always one hop

 

[teddyh2o]

thank you that make sense now.