[SOLVED] SDN nodes out of sync.

volga629

New Member
May 11, 2022
16
0
1
Hello Everyones,
Scenario: cluster 4 nodes
Issue: Some nodes get out of sync ( impossible push any configuration )


Log: pvestatd[1971268]: local sdn network configuration is too old, please reload


I checked SDN configuration files and they missing configuration.

How to restore good state ?
 

spirit

Famous Member
Apr 2, 2010
5,779
674
133
www.odiso.com
Hello Everyones,
Scenario: cluster 4 nodes
Issue: Some nodes get out of sync ( impossible push any configuration )


Log: pvestatd[1971268]: local sdn network configuration is too old, please reload


I checked SDN configuration files and they missing configuration.

How to restore good state ?
Hi,

do you have included
"source /etc/network/interfaces.d/*"

in /etc/network/interfaces ?

do you see the reload task for theses nodes ?

if yes, can you send the result of "ifquery -c -a" ?
 

volga629

New Member
May 11, 2022
16
0
1
Thank you for reply
do you have included
"source /etc/network/interfaces.d/*"
Yes it included. I see in ifquery
do you see the reload task for theses nodes ?
No I don't see the running task, because in web ui, mode is pending and it will execute task only on available nodes.

if yes, can you send the result of "ifquery -c -a" ?

ifquery output


Code:
root@aitdalpve02:/etc/network# ifquery -c -a
auto Vnet1
iface Vnet1                                                         [pass]
    bridge-ports vxlan_Vnet1                                    [pass]
    bridge-fd 0                                                 [pass]
    bridge-stp no                                               [pass]
    vrf vrf_vmbr2                                               [pass]
    mtu 9000                                                    [pass]
    hwaddress d6:ad:ac:b0:d9:35                                 [pass]
    ip-forward on                                               [pass]
    address                                                     [pass]

auto Vnet2
iface Vnet2                                                         [pass]
    bridge-ports vxlan_Vnet2                                    [pass]
    bridge-fd 0                                                 [pass]
    bridge-stp no                                               [pass]
    vrf vrf_vmbr2                                               [pass]
    mtu 9000                                                    [pass]
    hwaddress d6:ad:ac:b0:d9:35                                 [pass]
    alias MirtaPBX                                              [pass]
    ip-forward on                                               [pass]
    address                                                     [pass]

auto vrf_vmbr2
iface vrf_vmbr2                                                     [pass]
    post-up ip route add vrf vrf_vmbr2 unreachable default metric 4278198272         []
    vrf-table 1001                                              [pass]

auto vrfbr_vmbr2
iface vrfbr_vmbr2                                                   [pass]
    bridge-ports vrfvx_vmbr2                                    [pass]
    bridge-fd 0                                                 [pass]
    bridge-stp no                                               [pass]
    vrf vrf_vmbr2                                               [pass]
    mtu 9000                                                    [pass]

auto vrfvx_vmbr2
iface vrfvx_vmbr2                                                   [pass]
    vxlan-id 10000                                              [pass]
    vxlan-local-tunnelip                                      [pass]
    bridge-learning off                                         [pass]
    mtu 9000                                                    [pass]

auto vxlan_Vnet1
iface vxlan_Vnet1                                                   [pass]
    vxlan-id 1010                                               [pass]
    vxlan-local-tunnelip                                        [pass]
    bridge-learning off                                         [pass]
    mtu 9000                                                    [pass]

auto vxlan_Vnet2
iface vxlan_Vnet2                                                   [pass]
    vxlan-id 1011                                               [pass]
    vxlan-local-tunnelip                                        [pass]
    bridge-learning off                                         [pass]
    mtu 9000                                                    [pass]

auto lo
iface lo inet loopback

auto Mgmt-VRF
iface Mgmt-VRF                                                      [pass]
    post-up ip route add vrf Mgmt-VRF unreachable default metric 4278198272         []
    vrf-table 1199                                              [pass]

auto enp10s0
iface enp10s0 inet manual                                           [pass]
    post-up ethtool -G enp10s0 rx 4096 tx 4096                      []
    post-up ip link set dev enp10s0 txqueuelen 10000                []
    mtu 9000                                                    [pass]

auto enp11s0
iface enp11s0 inet manual                                           [pass]
    post-up ethtool -G enp11s0 rx 4096 tx 4096                      []
    post-up ip link set dev enp11s0 txqueuelen 10000                []
    mtu 9000                                                    [pass]

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto enp137s0
iface enp137s0 inet manual                                          [pass]
    post-up ethtool -G enp137s0 rx 4096 tx 4096                     []
    post-up ip link set dev enp137s0 txqueuelen 10000               []
    mtu 9000                                                    [pass]

auto enp138s0
iface enp138s0 inet manual                                          [pass]
    post-up ethtool -G enp138s0 rx 4096 tx 4096                     []
    post-up ip link set dev enp138s0 txqueuelen 10000               []
    mtu 9000                                                    [pass]

auto vlan99
iface vlan99 inet dhcp                                              [pass]
    vrf Mgmt-VRF                                                [pass]

auto vlan995
iface vlan995 inet static                                           [pass]
    address                                                         [pass]

auto vlan998
iface vlan998 inet static                                           [pass]
    address                                                        [pass]

auto bond0
iface bond0 inet manual                                             [pass]
    vrf Mgmt-VRF                                                [pass]

auto bond1
iface bond1 inet manual

auto bond2
iface bond2 inet manual

auto vmbr0
iface vmbr0 inet manual                                             [pass]
    vrf Mgmt-VRF                                                [pass]

auto vmbr1
iface vmbr1 inet manual

auto vmbr2
iface vmbr2 inet manual

auto vlan991
iface vlan991 inet manual                                           [pass]
    address                                                        [pass]
 

spirit

Famous Member
Apr 2, 2010
5,779
674
133
www.odiso.com
can you send me the log of the task "reloadnetworkall", when you click on the "apply" button on main section section ?

also, can you send me the content of /etc/pve/sdn/*.cfg files ?
 

volga629

New Member
May 11, 2022
16
0
1
can you send me the log of the task "reloadnetworkall", when you click on the "apply" button on main section section ?

also, can you send me the content of /etc/pve/sdn/*.cfg files ?
That interesting


Code:
proxy handler failed: ssh: connect to host 172.22.100.6 port 22: Connection refused
command 'pvesh set /nodes/aitdalpve03/network' failed: exit code 255
proxy handler failed: ssh: connect to host 172.22.100.5 port 22: Connection refused
command 'pvesh set /nodes/aitdalpve02/network' failed: exit code 255
TASK OK

not sure yet why connection refused . SSH listen on all interfaces

Code:
root@aitdalpve02:~# lsof -i :22
COMMAND     PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
sshd       3068 root    3u  IPv4     57770      0t0  TCP *:ssh (LISTEN)
sshd       3068 root    4u  IPv6     57772      0t0  TCP *:ssh (LISTEN)
 
Last edited:

volga629

New Member
May 11, 2022
16
0
1
Thank you for all help.
Issue has been resolved .
Ssh daemon was configured to use Mgmt VRF.
I configured secondary ssh instance and everything is reloads and applies as expected.

Code:
ExecStart=/usr/sbin/ip vrf exec Mgmt-VRF
 

spirit

Famous Member
Apr 2, 2010
5,779
674
133
www.odiso.com
Thank you for all help.
Issue has been resolved .
Ssh daemon was configured to use Mgmt VRF.
I configured secondary ssh instance and everything is reloads and applies as expected.

Code:
ExecStart=/usr/sbin/ip vrf exec Mgmt-VRF
ok thanks !

be carefull, because for vm live migration for example, proxmox also use ssh tunnel, and it's really run in the main vrf.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!