[SOLVED] Isolate the management network from the ceph network

[able]

auto lo
iface lo inet loopback

iface eno3 inet manual

iface eno4 inet manual

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.34/24
    gateway 192.168.1.1
    bridge-ports eno3
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet static
    address 192.168.2.34/24
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

I want to use vmbr0 as the management network and vmbr1 as the ceph network，eno1 connects to the trunk port of vlan 2 on the switch with other ceph servers.
I tried to set the gateway for vmbr1, but pve did not allow it. How can ceph servers communicate with each other?

 

[Guillaume Delanoy]

Hello,

You can only set *one* gateway in a system using IPv4, and you already set it for vmbr0.

For your Ceph network through vmbr1, you'll have to set a route.
See "ip route" and/or "route -n".


Kind regards,


Guillaume Delanoy

 

[ce3rd]

"How can ceph servers communicate with each other?" -> as long as the ceph servers are in the same IP subnet, they don't need a gateway (or no route) to talk to each other... ;-)

 

[able]

"How can ceph servers communicate with each other?" -> as long as the ceph servers are in the same IP subnet, they don't need a gateway (or no route) to talk to each other... ;-)

Thank you very much for your reply，this is the first time I have used proxmox in combination with ceph.
Until now, I had always thought of ceph as a process similar to vm under the linux bridge.

 

[able]

Hello,

You can only set *one* gateway in a system using IPv4, and you already set it for vmbr0.

For your Ceph network through vmbr1, you'll have to set a route.
See "ip route" and/or "route -n".


Kind regards,


Guillaume Delanoy

I was wrong earlier，the ceph servers don't need a gateway (or no route) to talk to each other.
But as you said, if I need to communicate with an interface that has no gateway, I need to set up a route.
Thanks a lot.