Strange Proxmox networking: VM are alive but hardware is not

[Paul McBets]

Hello all.
I have a problem with network.
Server: Dell R540 Xeon Silver 4210R 2.4G*1 etc...

#grep Linux /var/log/messages
Aug 24 10:39:22 dvn-proxmox01 kernel: [    0.000000] Linux version 5.4.106-1-pve (build@pve) (gcc version 8.3.0 (Debian 8.3.0-6) ...

After a hard reset the Proxmox web console disappeared. I tried to connect to server by ssh, but failed. So i connected by iDRAC and saw that no ping came. The eno1 inteface must get ip-address from dhcp but don't get it.
At the same time both virtual machines are alive and run correctly.
In the log i found following messages:

kernel: [] tg3 0000:04:00.0 eno1: renamed from eth0
kernel: [] tg3 0000:04:00.0 eno1: Link is up at 1000 Mbps, full duplex
kernel: [] tg3 0000:04:00.0 eno1: EEE is disabled
kernel: [] vmbr0: port 1 (eno1) entered blocking state
kernel: [] vmbr0: port 1 (eno1) entered disabled state
kernel: [] device eno1 entered promicuous mode
kernel: [] vmbr0: port 1 (eno1) entered blocking state
kernel: [] vmbr0: port 1 (eno1) entered forwarding state
...
kernel: [] device tap100i0 entered promicuous mode
kernel: [] fwbr100i0: port 1 (fwln100i0) entered blocking state
kernel: [] fwbr100i0: port 1 (fwln100i0) entered disabled state
kernel: [] device fwln100i0 entered promicuous mode
kernel: [] fwbr100i0: port 1 (fwln100i0) entered blocking state
kernel: [] fwbr100i0: port 1 (fwln100i0) entered forwarding state
... and so on with fwbrXX and tapXX devices.

But as i understand it's normal.

/etc/network/interfaces:

auto lo
iface lo inet loopback

iface idrac inet manual

auto eno1
iface eno1 inet dhcp

auto vmbr0
iface vmbr0 inet dhcp
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

iface ens1f0np0 inet manual

iface eno2 inet manual

iface ens1f1np1 inet manual

ip a:

#ip a
...
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 10.18.22.63/24 brd 10.18.22.255 scope global dynamic eno1
        valid_lft 687259sec preferred_lft 687259sec
...
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 10.18.22.71/24 brd 10.18.22.255 scope global dynamic vmbr0
        valid_lft 687259sec preferred_lft 687259sec

other interfaces have no ip addresses.

Self-ping runs correctly:

#ping 10.18.22.63
PING 10.18.22.63 (10.18.22.63) 56 (84) bytes of data.
64 bytes from 10.18.22.63: icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from 10.18.22.63: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from 10.18.22.63: icmp_seq=3 ttl=64 time=0.041 ms
^C

Ping someone else doesn'n work:

#ping 10.18.22.1
PING 10.18.22.1 (10.18.22.1) 56 (84) bytes of data.
From 10.18.22.63 icmp_seq=1 Destination host unreachable
From 10.18.22.63 icmp_seq=2 Destination host unreachable
From 10.18.22.63 icmp_seq=3 Destination host unreachable
^C

What should i do to return network interfaces to life?

 

[aaron]

What is the content of your /etc/hosts file?
Setting the main NIC to DHCP can be problematic if the IP changes at some point, as PVE assumes a fixed IP address that must match the entry in the /etc/hosts referencing the system itself.

Why do you set both, the physical interface and the vmbr using that interface, to DHCP? You should set the physical interface to manual, e.g.:

iface eno1 inet manual

You only need one IP address assigned to the server.

 

[Paul McBets]

What is the content of your /etc/hosts file?
Setting the main NIC to DHCP can be problematic if the IP changes at some point, as PVE assumes a fixed IP address that must match the entry in the /etc/hosts referencing the system itself.

My DHCP gives static IPs to each server. Address did not change. /etc/hosts

Why do you set both, the physical interface and the vmbr using that interface, to DHCP? You should set the physical interface to manual, e.g.:

iface eno1 inet manual

You only need one IP address assigned to the server.

It's the company policy: not to set IPs manually. So DHCP technology allows to set constant addresses from one control center. But if it doesn't work, i'll set it manually
I installed and set Proxmox the first time, so i don't know the features. When i read manuals i understood it that way.
Should i set the same IPs on eno1 and vmbr0 or delete IP from eno1?

 

[Paul McBets]

So, i could repair networking, but the cause is still incomprehensible.
When i delete ip on eno1, ping to vmbr0 appeared, but both VMs fell out of network. Changing network settings on VM interfaces had no effect. Deleting and adding back network interfaces in VM hardware had no effect too. Nothing worked while eno1 had "manual" in /etc/network/interfaces.
Only after me add an ip address on eno1, delete and add back interface on VM and set ip address on it inside VM operating system, the ping begin to run correctly.
I googled the same problem, but found the only incident in Russia happened before 2018 with Dell PE R720 (BTW i have the R540) that was solved by changing the network hardware.

 

[aaron]

This does sound curious.
Again, it should be okay to only have an IP configured on vmbr0 and have eno1 set to manual. Make sure that the IP address that is assigned to vmbr0 matches the IP address in the second line of the /etc/hosts file which should look something like this (depending on what you named you server):

10.18.22.71 pve.mydomain.com pve

Did you reboot the server to apply the network changes?

 

[Paul McBets]

This does sound curious.
Again, it should be okay to only have an IP configured on vmbr0 and have eno1 set to manual. Make sure that the IP address that is assigned to vmbr0 matches the IP address in the second line of the /etc/hosts file which should look something like this (depending on what you named you server):

10.18.22.71 pve.mydomain.com pve

Did you reboot the server to apply the network changes?

First of all, /etc/hosts must RIP in previous century. Clever people developed DHCP to let ITsts manage IP addresses centrally. So i'm surprised that someone still uses such garbage like /etc/hosts.

My work yesterday was:
1. Delete ip from eno1. VMs died, vmbr0 revived immediately. No need to reboot smthg.
2. Write right ip and right name in /etc/hosts. Nothing changes even after reboot hardware server. Honestly i tried it.
3. Change ip in VM. Doesn't work.
4. Delete and create network interface in VM. Set ip in VM. Doesn't work.
5. Set ip in eno1. Doesn't work.
6. Delete and create network interface in VM again. Set ip in VM. VM revival. No need to reboot smth again.
What am i doing wrong?


BTW why should i reboot the whole server? Does #service network restart not work? Even Microsoft OSs have no need to reboot often nowadays, so *nix systems never needed it. Restarting separate service was enough always.