[SOLVED] Full mesh with 3 Proxmox and IPv6!

G

Gilberto Ferreira

Renowned Member
Aug 21, 2014
390
51
93
Joinville, Santa Catarina, Brazil, Brazil
Hi there.

I am trying to implement a Full Mesh network with 3 servers.
I am using this guide: https://packetpushers.net/blog/proxmox-ceph-full-mesh-hci-cluster-w-dynamic-routing/

My 3 nodes has:
2x 1G NIC
Port1 = eno8303
Port2 = eno8403
for the LAN and PVE access. This is my vmbr0 in each server.
4x 1G NIC
Port1 = eno12399
Port2 = eno12409
Port3 = eno12419 - using this one to cluster
Port4 = eno12429 - using this one to cluster
4x 10G NIC
Port1 = ens1f0np0
Port2 = ens1f1np1
Port3 = ens3f0np0
Port4 = ens3f1np1
I want to use port3 and port4 to CEPH.

Now, I successfully created the cluster between 3 server, using localhost interface over IPV6.
It's works fine.
In each server I had created like this, in the interfaces network file:
...
auto lo
iface lo inet loopback

auto lo:0
iface lo:0 inet static
address fc00::1/128 -----------------> of course, I changed to each server like: fc00::2, fc00::3 and so on.
...

I have had used this configuration in vtysh:

ipv6 forwarding
!
router ospf6
ospf6 router-id 0.0.0.1
log-adjacency-changes
exit
!
interface lo
ipv6 ospf6 area 0
exit
!
interface eno12419
ipv6 ospf6 area 0
ipv6 ospf6 network point-to-point
exit
!
interface eno12429
ipv6 ospf6 area 0
ipv6 ospf6 network point-to-point
exit
!

That's works fine to use this 2 eno12419 and eno12429.

But now, I want to create other routes using the other interfaces.
So, I went aready and create this configuration in the vtysh:
ipv6 forwarding
!
router ospf6
ospf6 router-id 0.0.0.1
log-adjacency-changes
exit
!
interface lo
ipv6 ospf6 area 0
exit
!
interface lo
ipv6 ospf6 area 1
exit
!
interface lo
ipv6 ospf6 area 2
exit
!
interface eno12399
ipv6 ospf6 area 0
ipv6 ospf6 network point-to-point
exit
!
interface eno12409
ipv6 ospf6 area 0
ipv6 ospf6 network point-to-point
exit
!
interface eno12419
ipv6 ospf6 area 1
ipv6 ospf6 network point-to-point
exit
!
interface eno12429
ipv6 ospf6 area 1
ipv6 ospf6 network point-to-point
exit
!
interface ens3f0np0
ipv6 ospf6 area 2
ipv6 ospf6 network point-to-point
exit
!
interface ens3f1np1
ipv6 ospf6 area 2
ipv6 ospf6 network point-to-point
exit
!

But I am in doubt how I can sen traffic over that specific network.
I have had create others localhost definitions like that:
iface lo:0 inet static
address fc00::1/128
# CLUSTER

iface lo:1 inet static
address fc00::110/128
# MIGRATE

iface lo:2 inet static
address fc00::210/128
# CEPH

In the vtysh I have this route:
show ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

O fc00::1/128 [110/10] is directly connected, lo, weight 1, 00:05:17
C>* fc00::1/128 is directly connected, lo, 01:48:06
O>* fc00::2/128 [110/110] via fe80::d604:e6ff:fe84:9361, eno12399, weight 1, 00:05:15
* via fe80::d604:e6ff:fe84:9363, eno12419, weight 1, 00:05:15
O>* fc00::3/128 [110/110] via fe80::d604:e6ff:fe84:99cc, eno12409, weight 1, 00:05:15
* via fe80::d604:e6ff:fe84:99ce, eno12429, weight 1, 00:05:15
O fc00::110/128 [110/10] is directly connected, lo, weight 1, 00:05:17
C>* fc00::110/128 is directly connected, lo, 01:01:14
O>* fc00::120/128 [110/110] via fe80::d604:e6ff:fe84:9361, eno12399, weight 1, 00:05:15
* via fe80::d604:e6ff:fe84:9363, eno12419, weight 1, 00:05:15
O>* fc00::130/128 [110/110] via fe80::d604:e6ff:fe84:99cc, eno12409, weight 1, 00:05:15
* via fe80::d604:e6ff:fe84:99ce, eno12429, weight 1, 00:05:15
O fc00::210/128 [110/10] is directly connected, lo, weight 1, 00:05:17
C>* fc00::210/128 is directly connected, lo, 01:01:14
O>* fc00::220/128 [110/110] via fe80::d604:e6ff:fe84:9361, eno12399, weight 1, 00:05:15
* via fe80::d604:e6ff:fe84:9363, eno12419, weight 1, 00:05:15
C * fe80::/64 is directly connected, lan200, 01:07:09
C * fe80::/64 is directly connected, eno12409, 01:42:03
C * fe80::/64 is directly connected, eno12429, 01:42:45
C * fe80::/64 is directly connected, ens3f0np0, 01:48:06
C * fe80::/64 is directly connected, eno12429, 01:48:06
C * fe80::/64 is directly connected, eno12419, 01:48:06
C * fe80::/64 is directly connected, eno12409, 01:48:06
C>* fe80::/64 is directly connected, eno12399, 01:48:06

And I can see the neighbor

show ipv6 ospf6 neighbor
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
0.0.0.2 1 00:00:34 Full/PointToPoint 00:05:56 eno12419[PointToPoint]
0.0.0.3 1 00:00:35 Full/PointToPoint 00:05:54 eno12429[PointToPoint]
0.0.0.2 1 00:00:34 Full/PointToPoint 00:05:56 eno12399[PointToPoint]
0.0.0.3 1 00:00:35 Full/PointToPoint 00:05:54 eno12409[PointToPoint]
0.0.0.3 1 00:00:35 Full/PointToPoint 00:05:54 ens3f0np0[PointToPoint]
0.0.0.2 1 00:00:34 Full/PointToPoint 00:05:56 ens3f1np1[PointToPoint]

But I can't figure out how to send traffic to the others interfaces.

I hope some one can help me.

Thanks

Best regards.

Note: attached is the draw of the network. Very simple, but I hope it can clarify things.
 

Attachments

  • novas-conexoes-atualizado.png
    novas-conexoes-atualizado.png
    23.1 KB · Views: 2
Hi folks... It's me again.
Just a little update.
I figure out that doing this way will be a better approach:
frr version 8.5.2
frr defaults traditional
hostname proxmox01
log syslog informational
service integrated-vtysh-config
!
interface lo
ipv6 ospf6 area 0.0.0.0
exit
!
router ospf6
ospf6 router-id 0.0.0.1
log-adjacency-changes
exit
!
interface eno12419
ipv6 ospf6 area 0.0.0.0
ipv6 ospf6 network point-to-point
exit
!
interface eno12429
ipv6 ospf6 area 0.0.0.0
ipv6 ospf6 network point-to-point
exit
!
interface eno12419
ipv6 address fd69:beef:cafe::551/128
ipv6 ospf6 area 1.1.1.1
ipv6 ospf6 network point-to-point
exit
!
interface eno12429
ipv6 address fd69:beef:cafe::552/128
ipv6 ospf6 area 1.1.1.1
ipv6 ospf6 network point-to-point
exit
!
interface ens3f0np0
ipv6 address fd69:beef:cafe::553/128
ipv6 ospf6 area 2.2.2.2
ipv6 ospf6 network point-to-point
exit
!
interface ens3f1np1
ipv6 address fd69:beef:cafe::554/128
ipv6 ospf6 area 2.2.2.2
ipv6 ospf6 network point-to-point
exit
!

ip -6 route
fc00::1 dev lo proto kernel metric 256 pref medium
fc00::2 nhid 46 via fe80::d604:e6ff:fe84:9363 dev eno12419 proto ospf metric 20 pref medium
fc00::3 nhid 52 via fe80::d604:e6ff:fe84:99ce dev eno12429 proto ospf metric 20 pref medium
fd69:beef:cafe::351 nhid 52 via fe80::d604:e6ff:fe84:99ce dev eno12429 proto ospf metric 20 pref medium
fd69:beef:cafe::352 nhid 52 via fe80::d604:e6ff:fe84:99ce dev eno12429 proto ospf metric 20 pref medium
fd69:beef:cafe::353 nhid 51 via fe80::262:bff:fe74:2e71 dev ens3f0np0 proto ospf metric 20 pref medium
fd69:beef:cafe::354 nhid 51 via fe80::262:bff:fe74:2e71 dev ens3f0np0 proto ospf metric 20 pref medium
fd69:beef:cafe::451 nhid 46 via fe80::d604:e6ff:fe84:9363 dev eno12419 proto ospf metric 20 pref medium
fd69:beef:cafe::452 nhid 46 via fe80::d604:e6ff:fe84:9363 dev eno12419 proto ospf metric 20 pref medium
fd69:beef:cafe::453 nhid 48 via fe80::262:bff:fe7e:6fc0 dev ens3f1np1 proto ospf metric 20 pref medium
fd69:beef:cafe::454 nhid 48 via fe80::262:bff:fe7e:6fc0 dev ens3f1np1 proto ospf metric 20 pref medium
fd69:beef:cafe::551 dev eno12419 proto kernel metric 256 pref medium
fd69:beef:cafe::552 dev eno12429 proto kernel metric 256 pref medium
fd69:beef:cafe::553 dev ens3f0np0 proto kernel metric 256 pref medium
fd69:beef:cafe::554 dev ens3f1np1 proto kernel metric 256 pref medium
fe80::/64 dev lan200 proto kernel metric 256 pref medium
fe80::/64 dev eno12409 proto kernel metric 256 pref medium
fe80::/64 dev eno8303 proto kernel metric 256 linkdown pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev eno12419 proto kernel metric 256 pref medium
fe80::/64 dev ens3f0np0 proto kernel metric 256 pref medium
fe80::/64 dev ens3f1np1 proto kernel metric 256 pref medium
fe80::/64 dev eno12429 proto kernel metric 256 pref medium
fe80::/64 dev eno12399 proto kernel metric 256 pref medium

If anybody has a better idea, that could share, I will be appreciated.

Thanks a lot.
 
So, I tried to combine both configuration, and works ok.
But now I am trying to use one 10g nic to ceph-net and the other to ceph-osd.
Here is the frr.conf config:
frr version 8.5.2
frr defaults traditional
ipv6 forwarding
hostname proxmox01
log syslog informational
service integrated-vtysh-config
!
interface eno12419
ipv6 ospf6 area 0
ipv6 ospf6 network point-to-point
exit
!
interface eno12429
ipv6 ospf6 area 0
ipv6 ospf6 network point-to-point
exit
!
interface lo
ipv6 ospf6 area 0
exit
!
router ospf6
ospf6 router-id 0.0.0.1
log-adjacency-changes
exit
!
interface lo
ip address 10.15.15.50/32
ip router openfabric 1
openfabric passive
!
interface lo
ip address 10.15.15.60/32
ip router openfabric 2
openfabric passive
!
interface ens3f0np0
ip router openfabric 1
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
!
interface ens3f1np1
ip router openfabric 2
openfabric csnp-interval 2
openfabric hello-interval 1
openfabric hello-multiplier 2
!
line vty
!
router openfabric 1
net 49.0001.1111.1111.1111.00
lsp-gen-interval 1
max-lsp-lifetime 600
lsp-refresh-interval 180
!
router openfabric 2
net 50.0001.1111.1111.1111.00
lsp-gen-interval 1
max-lsp-lifetime 600
lsp-refresh-interval 180

But when I checked in
vtysh

Hello, this is FRRouting (version 8.5.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

proxmox01# show openfabric route
Area 1:
IS-IS L2 IPv4 routing table:

Prefix Metric Interface Nexthop Label(s)
------------------------------------------------------
10.15.15.50/32 0 - - -
10.15.15.60/32 0 - - -

Area 2:
IS-IS L2 IPv4 routing table:

Any clue?
 
You must log in or register to reply here.
Top Bottom