After upgrade to PVE 8.0 no network access

[speedbird]

Hi there,
I just upgraded to 8.0 using the upgrade guide and pve7to 8 said everything is fine. The update went all smooth, no errors and it rebooted without issues but the machine can't be reached anymore and I can't access anything from the machine. No ping is going out, nothing.

The Interface names have not changed, neither did the network config. Everything is as it was before and all links are up, but still no connectivity.

This is how my interfaces look.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
  address xx.xx.xx.xx
  netmask xx.xx.xx.xx
  gateway xx.xx.xx.xx
  bridge_ports enp4s0
  bridge_stp off
  bridge_fd 0

Redacted the IPs.

What can I do now?

 

[bbgeek17]

did you retype the whole file? I suspect so, as enp4so should be enp4s0 (last character is zero).
What does your "ip a" look like? Did interface name change with new kernel? If it did - update the interfaces file, paying careful attention to using proper characters.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[speedbird]

Yeah typo, sorry. It's a 0 of course. iconfig says the adapter still is called enp4s0

 

[bbgeek17]

Is LOWER_UP and is "state" UP for the interface?
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

Do you have any messages in journalctl or dmesg? Did you upgrade Kernel along with OS? Can you boot to older kernel and does that fix the issue?

Rather than retyping you can plug a usb stick and redirect output to a file that you can then copy/paste to avoid confusion.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Chris]

What does your "ip a" look like?

Please provide the output as requested by @bbgeek17 and further also systemctl status networking.service

 

[speedbird]

That was all good. I ended up reinstalling, pasting my exact same network config again and everything is running again.

Guess I'll never know what caused this? Strange.

 

[Chris]

To bad you did not attach the journal first, might have been a clue there on what has happened, but I guess it is to late now.

 

[speedbird]

Sorry but I have no physical access to the machine just a screen capture remote console. Would've not been possible with the journal I guess.

 

[AprilMan7]

Hi,

I have the exact same issue. I upgraded earlier today from 7.4 to 8.0 and my server don't have any connectivity.

I'm stuck at boot on "Job networking.service/start running"


Here's my current network interface configuration (other interfaces are commented so not included, I'm trying to get connectivity from the main interface for now):

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
    address 217.x.x.x/23
    gateway 217.x.x.x

iface eno2 inet manual

Service have no interesting logs:

-- Boot 280a2d41903941b4ae267e8c8bebc5b5 --
Jun 22 19:55:03 proxmox systemd[1]: Starting networking.service - Network initialization...
Jun 22 19:55:03 proxmox networking[806]: networking: Configuring network interfaces
Jun 22 19:55:09 proxmox systemd[1]: Finished networking.service - Network initialization.
-- Boot 01d708ebdefa47af9fd4bcea31e7ce3f --
Jun 22 20:10:55 proxmox systemd[1]: Starting networking.service - Network initialization...
Jun 22 20:10:55 proxmox networking[676]: networking: Configuring network interfaces
Jun 22 20:11:02 proxmox systemd[1]: Finished networking.service - Network initialization.
-- Boot 2e0f0dafb59e4910af5af356ccfc3704 --
Jun 22 20:22:51 proxmox systemd[1]: Starting networking.service - Network initialization...
Jun 22 20:22:52 proxmox networking[667]: networking: Configuring network interfaces

Do note that I don't have physical access to the machine, I only can edit things from a rescue livecd.

Any idea of what's the issue? Settings everything to manual gets me to the login screen so it's definetly something with the interfaces.

Edit:

I've been able to access my proxmox server with ssh by editing the networking service to kill it after 30 seconds. Here's the full journalctl logs coming from "networking.service" (I switched to eth0 while I was testing things, that's why it's not eno1 anymore): https://pastebin.com/WFamyYu2

 

[rfox]

I have the same issue as well - did two nothing special "test" machines to test the upgrade process - went just fine! Then I tried the least important node on a 4 node cluster and it fails to start networking - see attached

Not sure where dhclient is getting eth0 from - I don't have an eth0 ?!?

hwinfo --network --short
network interface:
enp2s0 Ethernet network interface
enp1s0 Ethernet network interface
lo Loopback network interface
wlp3s0 Ethernet network interface

 

[Chris]

Please also provide the /var/log/ifupdown2/network_config_ifupdown2_34_Jun-23-2023_11:40:09.385986 and the journal since boot journalctl -b > journal.txt as attachment.

 

[fiona]

Hi,

I have the same issue as well - did two nothing special "test" machines to test the upgrade process - went just fine! Then I tried the least important node on a 4 node cluster and it fails to start networking - see attached

Not sure where dhclient is getting eth0 from - I don't have an eth0 ?!?

hwinfo --network --short
network interface:
enp2s0 Ethernet network interface
enp1s0 Ethernet network interface
lo Loopback network interface
wlp3s0 Ethernet network interface

could it be that the interface name simply changed? It's unfortunately, possible with certain devices, as mentioned in the upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Network_Interface_Name_Change

 

[rfox]

here are the requested files from that directory

 

[rfox]

Hi,

could it be that the interface name simply changed? It's unfortunately, possible with certain devices, as mentioned in the upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_7_to_8#Network_Interface_Name_Change

I double checked that - and NO - the interface names are the same as before - but oddly "eth0" seems to appear from somewhere and that's what brings the networking service to a halt

 

[Chris]

I double checked that - and NO - the interface names are the same as before - but oddly "eth0" seems to appear from somewhere and that's what brings the networking service to a halt

Please check your dhclient.conf for possible interface configurations for eth0.

 

[rfox]

I just checked dhclient.conf - everything is commented out - and I check the other three active nodes running 7.4 - the same. So nothing has changed at all:

root@FoxProxMini:/etc/dhcp# cat dhclient.conf
# Configuration file for /sbin/dhclient.
#
# This is a sample configuration file for dhclient. See dhclient.conf's
# man page for more information about the syntax of this file
# and a more comprehensive list of the parameters understood by
# dhclient.
#
# Normally, if the DHCP server provides reasonable information and does
# not leave anything out (like the domain name, for example), then
# few changes must be made to this file, if any.
#

option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;

send host-name = gethostname();
request subnet-mask, broadcast-address, time-offset, routers,
domain-name, domain-name-servers, domain-search, host-name,
dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers,
netbios-name-servers, netbios-scope, interface-mtu,
rfc3442-classless-static-routes, ntp-servers;

#send dhcp-client-identifier 1:0:a0:24:ab:fb:9c;
#send dhcp-lease-time 3600;
#supersede domain-name "fugue.com home.vix.com";
#prepend domain-name-servers 127.0.0.1;
#require subnet-mask, domain-name-servers;
#timeout 60;
#retry 60;
#reboot 10;
#select-timeout 5;
#initial-interval 2;
#script "/sbin/dhclient-script";
#media "-link0 -link1 -link2", "link0 link1";
#reject 192.33.137.209;

#alias {
# interface "eth0";
# fixed-address 192.5.5.213;
# option subnet-mask 255.255.255.255;
#}

#lease {
# interface "eth0";
# fixed-address 192.33.137.200;
# medium "link0 link1";
# option host-name "andare.swiftmedia.com";
# option subnet-mask 255.255.255.0;
# option broadcast-address 192.33.137.255;
# option routers 192.33.137.250;
# option domain-name-servers 127.0.0.1;
# renew 2 2000/1/12 00:00:01;
# rebind 2 2000/1/12 00:00:01;
# expire 2 2000/1/12 00:00:01;
#}

 

[fiona]

here are the requested files from that directory

You have a interfaces.d/setup.old file referencing eth0. I'm not sure the .old extension is ignored here. Can you try and remove it?

 

[rfox]

OK - funny story - during my investigations - I came across that setup file - then cross checked with another running system which didn't have it - and thought it might be a leftover from the upgrade - so I renamed it to setup.old hoping that would fix it

Now that I have removed it - I got to a step further - networking service starts now and the virtual bridges are now seen - also with correct IP addresses BUT, I can't ping out on ANY of the interfaces (or in) and I can only ping local 127.0.0.1 loopback

I see network link status from the lights on the interfaces - but NO network still Then I checked the interfaces file and somehow the bridge-ports we set to none on the critical interfaces !?! After changing that and restarting - all seems well

Just not sure how that setup file appeared in the first place . . . . No on to the next (least important) node in the cluster!!

BTW - is there any issues in a cluster if some nodes are 7.4 and some 8 ?!? Long term affects?

Thx,
Robert

 

[rfox]

Is it advisable to shutdown all running VMs and containers before the upgrade?

Bonus question - should the "main" cluster node be upgraded first??

 

[fiona]

Is it advisable to shutdown all running VMs and containers before the upgrade?

Yes, it's highly recommended to have no running guests on the node during upgrade as mentioned in the upgrade guide https://pve.proxmox.com/wiki/Upgrade_from_7_to_8 and checker script pve7to8. Otherwise, the guests might run into issues! If you have a cluster and want to avoid downtime, migrate the guests away before upgrade and back after the upgrade.