Hello,
i am TotallyInOverMyHead - i hope you can help me.
I come here by suggestion of users in this reddit thread:
www_reddit.com/r/networking/comments/3w62gt/been_assigned_to_completely_redo_the_enterprise/
Backstory
I recently got hired to maintain an existing Proxmox/Ceph infrastructure straight out of university. I use Proxmox on home-Lab with 3 nodes and a mobile lab on my notebook since ages. At university i had about 6 month volunteering at the DC with Proxmox and Ceph specifically.
During my first week at work and while still getting my bearings i get informed that their "network admin" was fired. During a meeting with the company owner i get "incentivised/promoted/roped in" to take that over, until (maybe) they can find a replacement. I asked for em to bring in a consultant and that gets denied. I then start the above reddit Thread.
It should be said that i have rudimentary networking skills and do not know what to look for in switches.
www_dropbox.com/s/kknnh0d7gonf4kf/Screen%20Shot%202015-12-10%20at%2023.45.39.png
The Mayor painpoints (where i am asking for help with on this forum) are as follows:
- We have 50 Nodes running Proxmox with ceph. They are separated into 10 Racks in 7 different buildings. 5 Nodes per rack.
- There are 5 Proxmox-Clusters in total with 10 Nodes per. Each Cluster has a Node in each Rack. There is 10 HE unused per Rack. There TOTAL Vlan separation going on. There is separate Vlans per Cluster per backend.
- There is a total of 3 Ceph-nodes (EC-pool with SSD caching tier + a Replicated 4/2 pool.) Each Ceph-Cluster has Nodes in each of the Racks and in each of the Proxmox-Clusters. There is separate Vlans per Cluster per Client frontend and Backend.
- There is exclusively OpenVswitch (2.3.2-2) in use.
- The big issue is the network. Each Node has 2x10G onboard nics. They are bonded via LACP Balance-rr. But the links are only 1G at present. They use jumboframes (9000 MTU). What makes this worse is that there are some 100 Mbit switches creating 100 Mbit bottlenecks. Not like that matters at this point tho.
I know the following questions may be loaded, so please bear with me.
Question 1: Switches
What type of TOR switches should i use ? I know i need 10 Ports per TOR-switch right now + Uplinks. Half the Rack is empty so that might be filled with some more nodes in the future.
What type of features do i need to look at ?
Do Jumbo Frames make a difference in my scenario ? Mtu 9000 i've seen switches supporting MTU 16000.
Wha type of switches do you guys use ? what do you use to aggregate your TOR-Switches ?
Question 2: QOS + OVS/OVN - basically SDN
I spent the better part of my work-day looking into how to do "load-balancing". This is mostly because i'm sure some nodes will need to have the 10G links bonded (there are virtual NAS running on it that might easily see 20 Gbit/s traffic. Those servers have no free pcie-Slots so i could stick some more Nics in there.
How would you do QOS if separate nics are no option ? (i know you require that in your wiki)
Some redditors have pointed me towards OVS's sFlow features and i spent the better part of my free-time tonight watching the "Fall conference Video's for '14/'15". There they explain that you can use OVS in conjunction with OVN (which should be included in OVS 2.3.2) to tackle your issue of mice (a lot of flows with small bandwith) and elephants (small number of Flows with massive bandwith) by using QOS with OVS / OVN. Me not beeing a network guy i understood only half it, but the gist is that its better then doing QOS end-to-end this way, then doing it on teh switches. In fact some switch vendors actually support this sort of thing in their devices (sFlow/openFlow/SFF)
Edit: What i am looking for is SDN Software Defined Network.
i hope you can help me (with the more then indepth questions then you are probably used to).
sincerely,
TotallyInOverMyHead
i am TotallyInOverMyHead - i hope you can help me.
I come here by suggestion of users in this reddit thread:
www_reddit.com/r/networking/comments/3w62gt/been_assigned_to_completely_redo_the_enterprise/
Backstory
I recently got hired to maintain an existing Proxmox/Ceph infrastructure straight out of university. I use Proxmox on home-Lab with 3 nodes and a mobile lab on my notebook since ages. At university i had about 6 month volunteering at the DC with Proxmox and Ceph specifically.
During my first week at work and while still getting my bearings i get informed that their "network admin" was fired. During a meeting with the company owner i get "incentivised/promoted/roped in" to take that over, until (maybe) they can find a replacement. I asked for em to bring in a consultant and that gets denied. I then start the above reddit Thread.
It should be said that i have rudimentary networking skills and do not know what to look for in switches.
www_dropbox.com/s/kknnh0d7gonf4kf/Screen%20Shot%202015-12-10%20at%2023.45.39.png
The Mayor painpoints (where i am asking for help with on this forum) are as follows:
- We have 50 Nodes running Proxmox with ceph. They are separated into 10 Racks in 7 different buildings. 5 Nodes per rack.
- There are 5 Proxmox-Clusters in total with 10 Nodes per. Each Cluster has a Node in each Rack. There is 10 HE unused per Rack. There TOTAL Vlan separation going on. There is separate Vlans per Cluster per backend.
- There is a total of 3 Ceph-nodes (EC-pool with SSD caching tier + a Replicated 4/2 pool.) Each Ceph-Cluster has Nodes in each of the Racks and in each of the Proxmox-Clusters. There is separate Vlans per Cluster per Client frontend and Backend.
- There is exclusively OpenVswitch (2.3.2-2) in use.
- The big issue is the network. Each Node has 2x10G onboard nics. They are bonded via LACP Balance-rr. But the links are only 1G at present. They use jumboframes (9000 MTU). What makes this worse is that there are some 100 Mbit switches creating 100 Mbit bottlenecks. Not like that matters at this point tho.
I know the following questions may be loaded, so please bear with me.
Question 1: Switches
What type of TOR switches should i use ? I know i need 10 Ports per TOR-switch right now + Uplinks. Half the Rack is empty so that might be filled with some more nodes in the future.
What type of features do i need to look at ?
Do Jumbo Frames make a difference in my scenario ? Mtu 9000 i've seen switches supporting MTU 16000.
Wha type of switches do you guys use ? what do you use to aggregate your TOR-Switches ?
Question 2: QOS + OVS/OVN - basically SDN
I spent the better part of my work-day looking into how to do "load-balancing". This is mostly because i'm sure some nodes will need to have the 10G links bonded (there are virtual NAS running on it that might easily see 20 Gbit/s traffic. Those servers have no free pcie-Slots so i could stick some more Nics in there.
How would you do QOS if separate nics are no option ? (i know you require that in your wiki)
Some redditors have pointed me towards OVS's sFlow features and i spent the better part of my free-time tonight watching the "Fall conference Video's for '14/'15". There they explain that you can use OVS in conjunction with OVN (which should be included in OVS 2.3.2) to tackle your issue of mice (a lot of flows with small bandwith) and elephants (small number of Flows with massive bandwith) by using QOS with OVS / OVN. Me not beeing a network guy i understood only half it, but the gist is that its better then doing QOS end-to-end this way, then doing it on teh switches. In fact some switch vendors actually support this sort of thing in their devices (sFlow/openFlow/SFF)
Edit: What i am looking for is SDN Software Defined Network.
i hope you can help me (with the more then indepth questions then you are probably used to).
sincerely,
TotallyInOverMyHead
Last edited: