Hi,
I have a 3 node Proxmox cluster in a data center setup. They are connected via public gateway to internet over 1 GBPS connection. Also these hosts are equipped with 10GBPS connection and connected to a private switch. Please see attached diagram.
I am planning to configure CEPH private network over 10 GPBS private link.
What I am trying to achieve:
A redundancy for CEPH private network. If 10 GBPS link fails, then the traffic should move over to the 1 GBPS link automatically.
After reading through various threads, I tried implementing FRR and OpenFabric to achieve this.
Here is my configuration of Open Fabric
Node 2
Node 1
Here is the /etc/network/interfaces
Node 2
Node 1
And they these nodes are showing up on the network topology
from Node 2
But when I ping from Node2 to Node1, I get destination host unreachable error
If someone can help point out the issue in my configuration, that will be really appreciated.
Perhaps, the way I am trying to configure will not work as these machines are connected to switches. Is that what is wrong here? If so, what else can I use to achieve desired effect?
Thanks
I have a 3 node Proxmox cluster in a data center setup. They are connected via public gateway to internet over 1 GBPS connection. Also these hosts are equipped with 10GBPS connection and connected to a private switch. Please see attached diagram.
I am planning to configure CEPH private network over 10 GPBS private link.
What I am trying to achieve:
A redundancy for CEPH private network. If 10 GBPS link fails, then the traffic should move over to the 1 GBPS link automatically.
After reading through various threads, I tried implementing FRR and OpenFabric to achieve this.
Here is my configuration of Open Fabric
Node 2
Code:
Current configuration:
!
frr version 8.4.4
frr defaults traditional
hostname CE-FS-Node2
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface eno4
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
exit
!
interface enp129s0
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
exit
!
interface lo
ip router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0001.2222.2222.2222.00
lsp-gen-interval 1
max-lsp-lifetime 600
lsp-refresh-interval 180
exit
!
end
Node 1
Code:
Current configuration:
!
frr version 8.4.4
frr defaults traditional
hostname CE-FS-Node1
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface eno3
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
exit
!
interface enp129s0
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
exit
!
interface lo
ip router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0001.1111.1111.1111.00
lsp-gen-interval 1
max-lsp-lifetime 600
lsp-refresh-interval 180
exit
!
end
Here is the /etc/network/interfaces
Node 2
Code:
auto lo
iface lo inet loopback
auto lo:0
iface lo:0 inet static
address 10.15.15.6
iface eno4 inet manual
iface eno3 inet manual
iface eno1 inet manual
iface eno2 inet manual
auto enp129s0
iface enp129s0 inet manual
auto vmbr0
iface vmbr0 inet static
address xx.xxx.xxx.xx/29
gateway xx.xxx.xxx.xx
bridge-ports eno4
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 10.0.0.6/24
bridge-ports enp129s0
bridge-stp off
bridge-fd 0
#10 GPBPS Private LAN
auto vmbr2
iface vmbr2 inet static
address 192.168.1.10/24
bridge-ports none
bridge-stp off
bridge-fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -o vmbr0 -j MASQUERADE
Node 1
Code:
auto lo
iface lo inet loopback
auto lo:0
iface lo:0 inet static
address 10.15.15.3
iface eno4 inet manual
iface eno3 inet manual
iface eno1 inet manual
iface eno2 inet manual
auto enp129s0
iface enp129s0 inet manual
auto vmbr0
iface vmbr0 inet static
address xx.xxx.xxx.99/29
gateway xx.xxx.xxx.97
bridge-ports eno4
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 10.0.0.6/24
bridge-ports enp129s0
bridge-stp off
bridge-fd 0
#10 GPBPS Private LAN
auto vmbr2
iface vmbr2 inet static
address 192.168.1.10/24
bridge-ports none
bridge-stp off
bridge-fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -o vmbr0 -j MASQUERADE
And they these nodes are showing up on the network topology
from Node 2
Code:
root@CE-FS-Node2:~# vtysh -c "show openfabric topology"
Area 1:
IS-IS paths to level-2 routers that speak IP
Vertex Type Metric Next-Hop Interface Parent
CE-FS-Node2
10.15.15.6/32 IP internal 0 CE-FS-Node2(4)
CE-FS-Node1 TE-IS 10 CE-FS-Node1 enp129s0 CE-FS-Node2(4)
CE-FS-Node1 eno4
10.15.15.3/32 IP TE 20 CE-FS-Node1 enp129s0 CE-FS-Node1(4)
CE-FS-Node1 eno4
IS-IS paths to level-2 routers with hop-by-hop metric
Vertex Type Metric Next-Hop Interface Parent
But when I ping from Node2 to Node1, I get destination host unreachable error
Code:
root@CE-FS-Node2:~# ping 10.15.15.3
PING 10.15.15.3 (10.15.15.3) 56(84) bytes of data.
From 10.15.15.6 icmp_seq=1 Destination Host Unreachable
From 10.15.15.6 icmp_seq=2 Destination Host Unreachable
From 10.15.15.6 icmp_seq=3 Destination Host Unreachable
If someone can help point out the issue in my configuration, that will be really appreciated.
Perhaps, the way I am trying to configure will not work as these machines are connected to switches. Is that what is wrong here? If so, what else can I use to achieve desired effect?
Thanks