corosync[1108]: [KNET ] rx: Packet rejected from 192.168.1.82:5405

[scyto]

I have a 3 node proxmox 8 cluster i setup in the last 3 days (this is my first time).

It was working, but I have been messing around with networking a lot and now one of the nodes refuses to participate
pve1 and pve3 are up
pve2 (192.168.1.82) doesn't seem to get any corosync updated from pve1 / pve3

on both pve1 and pve3 the syslog is getting spammed with corosync[1108]: [KNET ] rx: Packet rejected from 192.168.1.82:5405

/etc/pve/corosync/conf is the same on all nodes - it previously had an issue in that the ring0_addr for pve2 was mis-typed by me to 129.168.1.82
i corrected this on the working nodes

for pve2 i had to set `pvecm expected 1` to be able to edit - this didn't fix it

i also checked the hosts file entries are correct and interfaces file is correct

what do i do next to solve? or should i just remove node 2 and rebuild/rejoin?

on pve2 i notice the Ring ID is different - i can't find how to change it to match the other two nodes...

root@pve2:~# pvecm status
Cluster information
-------------------
Name:             pve-cluster1
Config Version:   4
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sun Aug 13 00:26:13 2023
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000002
Ring ID:          2.e7
Quorate:          No

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      1
Quorum:           2 Activity blocked
Flags:         

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 192.168.1.82 (local)

on pve 1/3

root@pve1:~# pvecm status
Cluster information
-------------------
Name:             pve-cluster1
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sun Aug 13 00:26:57 2023
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.c2
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.1.81 (local)
0x00000003          1 192.168.1.83

This is my corosync on each of the nodes

root@pve2:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.1.81
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.1.82
  }
  node {
    name: pve3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.1.83
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: pve-cluster1
  config_version: 4
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}