Is my network setup is correct ?

S

Saxophone

New Member
Sep 3, 2024
17
0
1
I am trying to make a proxmox cluster that uses the built in ceph storage to create a pool where I can store all of my files, photos, videos, movies, music and the lot. Basically one giant hard drive that I can connect my home computers to via 10g network. I would also like the ability to remote into the pool to watch movies on a tablet or laptop when away from home. I am new to proxmox and ceph and have never worked with vm's before. My guess is that there will be one vm for serving out the ceph pool, and another vm for hosting plex. But I really don't know what would be considered best practices.

I have 4 supermicro servers with 12 hdd's each, supermicro motherboard, 128g ram, 2 ssd's for OS.

I am using the 1g motherboard nic for administering the proxmox cluster through the web interface on 192.168.1.x. Proxmox calls this nic eno1 and made it a bridge on vmbr0

I have a 2 port 10g nic that I am using one of the ports to connect to 10.10.30.x using a proxmox bridge vmbr1 that is for my ceph frontend network. Ceph documentation suggested this.

The second port is currently not used.

I have a second 2 port 10g nic that I have bonded together in proxmox with bond0 to connect to 10.10.40.x with bridge vmbr2 that is my ceph backend network. Also suggested in ceph documentation.

I want to make sure this is correct because I want 10g networking to my home computer.
 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual
#1G for pve & ceph managment

iface eno2 inet manual
#1g not used

iface enxbe3af2b6059f inet manual
#motherboard ipmi

auto enp65s0f0
iface enp65s0f0 inet manual
#10g for ceph backend

auto enp65s0f1
iface enp65s0f1 inet manual
#10g for ceph backend

auto enp66s0f0
iface enp66s0f0 inet manual
#10g for ceph frontend

iface enp66s0f1 inet manual
#10g not used

auto bond0
iface bond0 inet manual
bond-slaves enp65s0f0 enp65s0f1
bond-miimon 100
bond-mode balance-alb
#ceph backend bond

auto vmbr0
iface vmbr0 inet static
address 192.168.1.25/24
gateway 192.168.1.1
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.10.30.35/24
bridge-ports enp66s0f0
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet static
address 10.10.40.45/24
bridge-ports bond0
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*
 
floh8 said:
Config looks good. U need also a vm and cluster network.
Click to expand...
I don't know what that means. Could you elaborate please. I thought I have a cluster network one for frontend and one for backend. This is all new to me. What should that look like? Under Datacenter>cluster>I have 4 nodes with link0 in the 192.168.1.x range.
 
Last edited:
The cluster network for the pve should be a separate 1G Network. AS failover cluster network u can use the management network. Additional u need a network where your VMS are connected to for access Form your normal LAN.
 
Last edited:
floh8 said:
The cluster network for the pve should be a separate 1G Network. AS failover cluster network u can use the management network. Additional u need a network where your VMS are connected to for access Form your normal LAN.
Click to expand...
So let me see if I am understanding you correctly. When I first installed proxmox the setup utility asked for an ip address for each node that is the management network. Now I should use the currently unused eno2 device to give each of the nodes a new network address under the Datacenter>Cluster>link0 column, and this will be the proxmox cluster network, also it will not have a gateway if I am correct. I might be able to change the addresses or I might have to de-cluster the nodes and redo the cluster formation. I should be able to figure this out.

I do not understand the VM network please explain further. Please add detail and an example.
Is this a virtual network or a wired one ?
Is this so the VM can talk to the node it is on or to talk to other VM's on the same node or maybe different nodes ?

Another question:

I originally set up one of my 10g nic's on an address 10.10.30.35 ,36,37,38 on the 4 nodes so that I could get 10g speeds between the ceph clustered storage through my 10g switch to my computers 10g nic. I can ping these addresses from and to each other in a Datacenter>Node>Shell and also my home computer. But from my home computer I can only ping the management network nodes but I cannot ping the 10.10.30.x ports. What have I configured wrong here ?
 
Saxophone said:
I do not understand the VM network please explain further. Please add detail and an example.
Is this a virtual network or a wired one ?
Is this so the VM can talk to the node it is on or to talk to other VM's on the same node or maybe different nodes ?
Click to expand...
Sorry, I would differ the VM network from the management network in production environment. Of course u can use the management network for access the VM and your PC, but u have to change the management network to 10 GB Port if u Wanne access your data with 10GBit/s.
Saxophone said:
originally set up one of my 10g nic's on an address 10.10.30.35 ,36,37,38 on the 4 nodes so that I could get 10g speeds between the ceph clustered storage through my 10g switch to my computers 10g nic. I can ping these addresses from and to each other in a Datacenter>Node>Shell and also my home computer. But from my home computer I can only ping the management network nodes but I cannot ping the 10.10.30.x ports. What have I configured wrong here ?
Click to expand...

The frontend and back end net for ceph need no PC access. I think u should read more about ceph network or watch some videos for better understanding what u are doing.
 
Last edited:
floh8 said:
Sorry, I would differ the VM network from the management network in production environment. Of course u can use the management network for access the VM and your PC, but u have to change the management network to 10 GB Port if u Wanne access your data with 10GBit/s.


The frontend and back end net for ceph need no PC access. I think u should read more about ceph network or watch some videos for better understanding what u are doing.
Click to expand...
I have been and I am at my wits end
 
If u are a beginner with proxmox its better to start with a simple installation and not with ceph.
 
Since this is a home-setup, I take it that downtime isn't that big of an "issue", right? You of course would want to avoid it, but if it happens, it happens.
The reason floh8 seems to hammer on separate networks is because in "production" setups where downtime needs to be avoid at all cost, having the different types of traffic (being cluster-traffic, management-traffic, storage-traffic, vm/container-traffic) is strongly encouraged, especially keeping the cluster-traffic separate, and only having it set as failover to the other networks to keep the cluster intact in case of a network-failure. It is still possible to add more links to the existing cluster, but it will have to be done manually [1]
On top of that, something floh8 didn't seem to have noticed, was you mentioning having 4 nodes, this is another thing for production-clusters, keep to an uneven set of nodes, because you need MORE then half of the cluster alive to have it be able to make changes (now if 2 fall out, you're at exactly half, so it would stop working. You could consider adding a vote-deamon on an external device to break this tie [2] )

That said, if you followed the documentation like you said, and from my own, limited, experience with CEPH (did have it during the training but it has fallen to the background some since we couldn't use it in our own production-setup), it looks good. That said though, did you check that all your switches as well as the network-port on your PC is actually 10GB? And are you sure you want to interface with CEPH directly, or instead have a storage-VM serving you the data. My suggestion would be to do the second, let everyone "do what they're good at" basically, the storage-VM providing you the data, while proxmox is the one talking to ceph to handle the read/write requests.

Finally, there is one more thing you need to consider, are you going to go for speed or redundancy of data. A per-server pool of data is fast but not redundant (without backups), a shared pool of data is redundant, but to be so redundant writes need to go to multiple sources and have multiple check-layers to might be slower (even on a 10GB network). So if speed is your main focus, you might want to instead consider using just the local storage and building a pool on that (perhaps adding some RAID to have some in-device redundancy for disk-failure) and then using backups or replication to other node(s) for the redundancy (with the possibility of some data-loss since the last sync/backup)

[1] https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_redundancy
[2] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
 
floh8 said:
If u are a beginner with proxmox its better to start with a simple installation and not with ceph.
Click to expand...
What I am looking for is a Ceph cluster. While ZFS maybe easier I think it is a weaker system. I am not fond of loosing 1 zpool causing the loss of everything. I like Synology but they have a 108Tb single pool limit that I have outgrown. Ceph being integrated with Proxmox looked like an easier situation than using a Linux distro with Ceph installed and then using Cockpit to create the VM's
 
sw-omit said:
Since this is a home-setup, I take it that downtime isn't that big of an "issue", right? You of course would want to avoid it, but if it happens, it happens.
The reason floh8 seems to hammer on separate networks is because in "production" setups where downtime needs to be avoid at all cost, having the different types of traffic (being cluster-traffic, management-traffic, storage-traffic, vm/container-traffic) is strongly encouraged, especially keeping the cluster-traffic separate, and only having it set as failover to the other networks to keep the cluster intact in case of a network-failure. It is still possible to add more links to the existing cluster, but it will have to be done manually [1]
On top of that, something floh8 didn't seem to have noticed, was you mentioning having 4 nodes, this is another thing for production-clusters, keep to an uneven set of nodes, because you need MORE then half of the cluster alive to have it be able to make changes (now if 2 fall out, you're at exactly half, so it would stop working. You could consider adding a vote-deamon on an external device to break this tie [2] )

That said, if you followed the documentation like you said, and from my own, limited, experience with CEPH (did have it during the training but it has fallen to the background some since we couldn't use it in our own production-setup), it looks good. That said though, did you check that all your switches as well as the network-port on your PC is actually 10GB? And are you sure you want to interface with CEPH directly, or instead have a storage-VM serving you the data. My suggestion would be to do the second, let everyone "do what they're good at" basically, the storage-VM providing you the data, while proxmox is the one talking to ceph to handle the read/write requests.

Finally, there is one more thing you need to consider, are you going to go for speed or redundancy of data. A per-server pool of data is fast but not redundant (without backups), a shared pool of data is redundant, but to be so redundant writes need to go to multiple sources and have multiple check-layers to might be slower (even on a 10GB network). So if speed is your main focus, you might want to instead consider using just the local storage and building a pool on that (perhaps adding some RAID to have some in-device redundancy for disk-failure) and then using backups or replication to other node(s) for the redundancy (with the possibility of some data-loss since the last sync/backup)

[1] https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_redundancy
[2] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
Click to expand...
So if I am understanding floh8 correctly
I should have one of my 1g ports on the motherboard being used for proxmox management. ie the web gui interface
The second 1g port on the motherboard should be for VM network. Does this mean corosync ?

I am going for redundancy however 3 copies of everything uses too much space so I am looking at an EC pool of 4 + 2 yielding 66% usable space over the 33% usable space of a 3 copy pool. I will have backups on different equipment and some cold storage.

I agree with floh8 that the backend ceph network needs no PC access and thus no gateway. But the frontend should have PC access other wise I won't have a 10g link.

My network hardware was not in my original question I didn't think it was needed but I am always wrong?

Internet coming in to my house and going to a Unifi Dream Machine Pro via 1g connection

The dream machine is plugged into a 10g unifi switch with 12 ports
one 10g port goes to my home computer
one 10g port goes to a 24 x 1g port unifi poe+ switch
one 10g port goes to a 24 x 10g port unifi switch
one 10g port goes to a synology nas
one 10g port goes to an old btrfs system I had but has since been changed to truenas
four of the 10g ports go to my proxmox cluster and is intended to be the frontend network

The 24 port poe+ switch has 4 x 1g cables going to the first port on each of the proxmox host motherboards. I call this my management port.
It also has 4 x 1g cables going to the ipmi ports on the proxmox host motherboards.

The 24 port 10g switch has 8 cables 2 of which are going to each of the 4 proxmox hosts both going to the same nic. Inside of proxmox I have bonded these 2 connections. This is intended to be the Ceph backend network traffic.

I don't know much about networking. I have no VLAN's. I don't know how to use them but will set up everything however it needs to be done.

I have read that proxmox also has software defined networking that I have not looked into.

I want to make sure this is correct because I want 10g transfer speeds to my home computer.

Most videos I have watched put everything on one port to make the video eaiser to understand. Ceph and Proxmox documentation suggests different ports and networks because of high traffic but don't explain how it is done.

Can you tell me how it should be properly set up. I want to do it right.
 
You must log in or register to reply here.
Top Bottom