After upgrade v6 -> v7 no more web gui, no more ssh

[Mark Tetrode]

I have a cluster of 3 nodes on v6. I upgraded one node to v7 according to the upgrade guide https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0

The upgrade process went well. However after booting

• no more ssh access
• no more web access
• DNS resolving does not work
• v7 did not join cluster
• no containers started

So after all things did not go very well :-(

I have looked at different forum threads but have not yet found a solution. What can I do, what information do I need to provide?

 

[bbgeek17]

Most likely your network interface naming changed with new Kernel.
Look at the output of "ip a"
Compare the interface names to what you have in /etc/network/interfaces
Make adjustments as necessary.

If you dont see any differences, then provide the output of both of the commands above.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Mark Tetrode]

Sorry for the screenshot, but I cannot copy / paste text

I think that the output of ip a and /etc/network/interfaces is correct?

 

[bbgeek17]

I suspect the subnet is incorrect. By not specifying it, the subnet was set to /32. Which would prevent access to gateway.

How many IP addresses are in a 32?
A /32 only has one ip address so you don't have enough addresses for a subnet id, subnet broadcast or hosts. A /31 will give you two ip addresses, so you can have a subnet id and a subnet broadcast.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Mark Tetrode]

From a still working V6 with the same type of config:

# cat /etc/network/interfaces

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad

auto vmbr0
iface vmbr0 inet static
    address 51.91.30.11
    gateway 51.91.30.254
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0


# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether ac:1f:6b:71:2b:3c brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether ac:1f:6b:71:2b:3c brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ac:1f:6b:71:2b:3c brd ff:ff:ff:ff:ff:ff
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:71:2b:3c brd ff:ff:ff:ff:ff:ff
    inet 51.91.30.11/32 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe71:2b3c/64 scope link
       valid_lft forever preferred_lft forever

 

[bbgeek17]

well, your interfaces on 7x are up. Can you ping the gateway from the console?
If not, do you have anything in arp table : arp -a (you may not have arp installed), in that case try: "ip neigh show dev vmbr0"
Check routing table "ip route".

There is also possibility that network issue is outside of PVE and it just got exposed at the same time as upgrade/reboot. Proxmox is based on Debian and its networking is vanilla Linux. Check your bonds, try with one interface down, etc.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Mark Tetrode]

Thanks for your insight

Again a screenshot of the output



I could not ping the gateway on the v7 but on the v6 this is possible

So definately network related, I will look further into this.

If anybody has still some pointers, they are welcome

 

[bbgeek17]

The screenshot shows that your host was able to receive MAC of the router, which is an indication that Layer2 is (or was) functional.
There is a difference in working and non-working host - bond hashing. Try to remove it.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Mark Tetrode]

When I configure this



The networking starts working, web interface works, but of course the containers do not start because no vmbr0.

When I remove the bond hash policy, it still does not work

Any more insights?

 

[bbgeek17]

may be there is no bond on the other side? May be its misconfigured?

https://supportportal.juniper.net/s...-interfaces-on-RHEL-CentOS-7-5?language=en_US

Try bond with single interface in bond eno1
Try bond with single interface in bond eno2
Finally, or may be first - reach out to your network admin to confirm switch configuration...
If all else fails, remove bond and update vmbr to point to eno1...

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Stoiko Ivanov]

On a hunch - maybe you're affected by the changed mac-address assignment to virtual interfaces (bonds bridges):
https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0#Network

make sure that you do not have a duplicate Mac-address in the network.

 

[Mark Tetrode]

Hi Stoiko

Thanks a bunch - and also to bbgeek17

Finally adding the correct hwaddress ether xx:yy:zz... to interfaces caused the network to start working again

reason: the switches expect a specific MAC to be bound to a specific IP

final configuration

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad

auto vmbr0
iface vmbr0 inet static
    hwaddress ether ac:1f:6b:d2:92:c8
    address 51.91.247.33
    netmask 255.255.255.0
    network 51.91.247.0
    gateway 51.91.247.254
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0

removed:
bond-xmit-hash-policy layer2+3
added
hwaddress ether ac:1f:6b:d2:92:c8

Again thanks everyone for your help