my network setup need TLC

Saxophone

New Member
Sep 3, 2024
17
0
1
I am new to Proxmox and Ceph. I have two 2port 10g network cards. One for the frontend network and one for the backend network.

Does this setup look ok ?

I'm having problems setting up Ceph with it and had to run the setup with only the 1g vmbr0

Can I change it in Ceph later, if so how ?

1725478059439.png
 
Your issue most likely is that you're using the same IP-range for all network-adapters.
Like you have it now, the server will not be able to know which port to use for what.
Change your network for the bond and vmbr1 to something else, for example 10.0.0.X/24 for the bond and 10.0.1.X/24 for the vmbr1
That way, if you send traffic from one server to the other with the 10.0.1.X IP's, it knows to use the vmbr1/the first ceph backend adaptor.

You don't need a default gateway on those ports/ranges, as you wouldn't need to get to internet through those, only to eachother, and the vmbr0 is for your internet-access.

(Also, sorry, haven't used CEPH much, so can't really help you with that other question)
 
Your issue most likely is that you're using the same IP-range for all network-adapters.
Like you have it now, the server will not be able to know which port to use for what.
Change your network for the bond and vmbr1 to something else, for example 10.0.0.X/24 for the bond and 10.0.1.X/24 for the vmbr1
That way, if you send traffic from one server to the other with the 10.0.1.X IP's, it knows to use the vmbr1/the first ceph backend adaptor.

You don't need a default gateway on those ports/ranges, as you wouldn't need to get to internet through those, only to eachother, and the vmbr0 is for your internet-access.

(Also, sorry, haven't used CEPH much, so can't really help you with that other question)
I thought that by making it 192.168.1.35 I was be giving it a static ip address. What if I changed it to 192.168.1.35/32
 
Yes, with 1.35 it is making it a static IP, but that does nothing about the routing
if you set it to 1.35/32, then you won't be able to connect to anything else on the network through that port (as there are no other devices in a "range" of 32).

The subnet tells it what IP's / Devices are within the "local" network, so it can just "ask around" on the network, and for anything outside of that range, it knows that it isn't local and it needs to ask the "default gateway" device, as that device should know where to get to that other device.

So in your example you have 3 IP's all in the same "local" network, the network 192.168.1.X
Now a packet needs to go to 192.168.1.222 for example, how would the PC know which of the 3 devices to send that traffic out from? Since they are all local, they could go out from each of the 3 ports at random, although usually, because that port has the default gateway and lowest IP, it will probably pick the vmbr0 port.

There are ways to tell it "for traffic to IP A, always use port/IP B", but just using separate ranges is more stable, more easily set up, and more frequently seen as as such understood.

If you set your IPs to
vmbr0 192.168.1.25/24
bond 10.0.0.25/24
vmbr1 10.0.1.25/24
on server 1 and
vmbr0 192.168.1.27/24
bond 10.0.0.27/24
vmbr1 10.0.1.27/24
on server 2

If you say "Server 1, send this traffic to 10.0.0.27", the Server will check if it has an IP in the range that is requested and finds the 10.0.0.25 on bond, so it's uses the bond and neither of the other 2.
Similarly, if you say "Server 1, send this traffic to 10.0.1.27" it will find 10.0.1.25 and use vmbr1, and if you want to go to 8.8.8.8, it will not find that IP on any of it's devices, so it goes and ask it's default gateway, which it knows to find on ip 192.168.1.1, which it then can find from 192.168.1.25
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!