Unable to Update Proxmox

[Cryptonym]

Good day all,
I'm in the process of setting up a 3 Node HA cluster for my homelab.
Everything WAS going well until i tried updating Node 2 (pve2) which keeps failing.
Nodes 1 (pve 1) and 3 (pve 3) update without issue.
All nodes were setup together using the same steps (so can't see any mistakes made by myself)
When using the UI to 'update' the output is as follows:
starting apt-get update
Ign:1 http://ftp.nl.debian.org/debian bookworm InRelease
Ign:2 http://download.proxmox.com/debian/pve bookworm InRelease
Ign:3 http://download.proxmox.com/debian/ceph-reef bookworm InRelease
Ign:4 http://ftp.nl.debian.org/debian bookworm-updates InRelease
Ign:5 http://security.debian.org bookworm-security InRelease
Ign:5 http://security.debian.org bookworm-security InRelease
Err:6 http://security.debian.org bookworm-security Release
Cannot initiate the connection to security.debian.org:80 (2a04:4e42:400::644). - connect (101: Network is unreachable) Cannot initiate the connection to security.debian.org:80 (2a04:4e42:200::644). - connect (101: Network is unreachable) Cannot initiate the connection to security.debian.org:80 (2a04:4e42:600::644). - connect (101: Network is unreachable) Cannot initiate the connection to security.debian.org:80 (2a04:4e42::644). - connect (101: Network is unreachable)

I have 2 NICs per Node (1x onboard 1GB and 1x usb 2.5GB connected to a 2.5GBE switch).
All nodes show they are able to communicate with the rest of the Network and each other.
I ran a dig for 'download.proxmox.com' :

~# dig download.proxmox.com

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> download.proxmox.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30466
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;download.proxmox.com. IN A

;; ANSWER SECTION:
download.proxmox.com. 89 IN CNAME download.cdn.proxmox.com.
download.cdn.proxmox.com. 60 IN CNAME nl.eu.cdn.proxmox.com.
nl.eu.cdn.proxmox.com. 60 IN CNAME de.cdn.proxmox.com.
de.cdn.proxmox.com. 60 IN A 212.224.123.70

;; Query time: 196 msec
;; SERVER: 192.168.1.103#53(192.168.1.103) (UDP)
;; WHEN: Tue Mar 05 16:54:17 CET 2024
;; MSG SIZE rcvd: 129

When checking the configuration of the network :

~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
link/ether 10:e7:c6:0b:a9:61 brd ff:ff:ff:ff:ff:ff
altname enp0s31f6
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 10:e7:c6:0b:a9:61 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.109/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::12e7:c6ff:fe0b:a961/64 scope link
valid_lft forever preferred_lft forever
4: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 56:36:fe:f1:4c:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.119/24 scope global vmbr1
valid_lft forever preferred_lft forever
inet6 fe80::5436:feff:fef1:4c82/64 scope link
valid_lft forever preferred_lft forever

~# cat /etc/resolv.conf
search cryptonym.space
nameserver 192.168.1.103

~# cat /etc/resolv.conf
search cryptonym.space
nameserver 192.168.1.103

~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface enx5c857e3f7595 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.109/24
gateway 192.168.1.1
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
address 192.168.1.119/24
gateway 192.168.1.1
bridge-ports enx5c857e3f7595
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*

I've seen in other searches suggested that IPv6 be disabled but since they were not disabled on the other 2 nodes and they work fine i didn't see any need to do so on this one.
I am able to ping everything on my network but nothing externally i.e:
My dns server (pihole) is successful
My gateway is successful
'google.com' is unsuccessful
and 1.1.1.1 is also unsuccessful.

It goes without saying that all 3 nodes are 100% identical in terms of hardware and software (running Proxmox VE 8.1.4) on 3x HP 800 G3 SFF (1ltr) machines.
My experience with Proxmox is 'zero' but my Linux experience is somewhat better but limited to what ive picked up along the way.
My desperation to resolve this is at an all time high and short of scrapping Node 2 and starting from fresh and re adding it to the cluster, I'm not sure what else to do.
Only thing stopping me is i'm worried this will bork the cluster completely and all the work invested already.

If anyone can help me I would greatly appreciate it, Thanks in advance.

 

[Dunuin]

auto vmbr0
iface vmbr0 inet static
address 192.168.1.109/24
gateway 192.168.1.1
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
address 192.168.1.119/24
gateway 192.168.1.1
bridge-ports enx5c857e3f7595
bridge-stp off
bridge-fd 0

There can only be one default gateway per host. And a host shouldn't have two IPs in the same subnet or this might screw up routing. Try to remove the "address 192.168.1.119/24" and
"gateway 192.168.1.1" from your vmbr1 which is set to not use a IP anyway because of the "manual".
If you want your PVE host to be accessible on both NICs you need to create two different subnets (and maybe an additional virtual/physical router in case you want both subnets to be able to communicate with each other).

 

[Cryptonym]

There can only be one default gateway per host. And a host shouldn't have two IPs in the same subnet or this might screw up routing. Try to remove the "address 192.168.1.119/24" and
"gateway 192.168.1.1" from your vmbr1 which is set to not use a IP anyway because of the "manual".
If you want your PVE host to be accessible on both NICs you need to create two different subnets (and maybe an additional virtual/physical router in case you want both subnets to be able to communicate with each other).

Ok, ill give that a try, but it doesn't seem to affect the other nodes and they are set up the same way.

 

[Cryptonym]

There can only be one default gateway per host. And a host shouldn't have two IPs in the same subnet or this might screw up routing. Try to remove the "address 192.168.1.119/24" and
"gateway 192.168.1.1" from your vmbr1 which is set to not use a IP anyway because of the "manual".
If you want your PVE host to be accessible on both NICs you need to create two different subnets (and maybe an additional virtual/physical router in case you want both subnets to be able to communicate with each other).

Hi apologies for taking so long to get back to you, i had more pressing things to get to before this.
I did eventually follow your suggestion and backups are now working again.
Thank you for the help.