Network STOPPED working, after I fixed it in the first place

[dmattox10]

When I installed proxmox (both times) /etc/network/interfaces contained only this:

auto vmbr0
iface vmbr0 inet static
address 10.0.0.20
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0

I had no entries for the quad port GBe NIC, eth 0-3, and no entry for eth4, my Mellanox Connect-X2.

through a series of edits, reboots, and luck/magic, I managed to get all of my interfaces entered, and working, even using iSCSI over the 10GBe Mellanox.

I made a few more networking changes after installing a few WORKING VM's, to create new bridges for other VM's I was about to install. The server suggested I restart to apply changes, and now NOTHING works. I cannot reach the web interface, I cannot ping google standing at the rack.

/etc/network/interfaces contains ALL of the following:

auto lo
auto lo inet loopback

auto eth0
iface eth0 inet static
address 10.0.0.12
netmask 255.255.255.0
gateway 10.0.0.2
broadcast 10.0.0.255
network 10.0.0.0

auto eth1
iface eth1 inet static
address 10.0.0.13
netmask 255.255.255.0
broadcast 10.0.0.255
network 10.0.0.0

auto eth2
iface eth2 inet static
address 10.0.0.14
netmask 255.255.255.0
broadcast 10.0.0.255
network 10.0.0.0

auto eth3
iface eth3 inet static
address 10.0.0.15
netmask 255.255.255.0
broadcast 10.0.0.255
network 10.0.0.0

auto eth4
iface eth4 inet static
address 10.0.10.2
netmask 255.255.255.0
broadcast 10.0.10.255
network 10.0.10.0

auto vmbr0
iface vmbr0 inet static
address 10.0.0.20
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0
#WINDOWS-DESKTOPS

auto vmbr1
iface vmbr1 inet static
address 10.0.0.40
netmask 255.255.255.0
bridge_ports eth1
bridge_stp off
bridge_fd 0
#LINUX-SERVERS

auto vmbr2
iface vmbr2 inet static
address 10.0.0.50
netmask 255.255.255.0
bridge_ports eth2
bridge_stp off
bridge_fd 0
#LINUX-DESKTOPS

What finally made everything work last night, was:
ifconfig eth0 10.0.0.12 netmask 255.255.255.0 up
ifconfig eth1 10.0.0.13 netmask 255.255.255.0 up
ifconfig eth2 10.0.0.14 netmask 255.255.255.0 up
ifconfig eth3 10.0.0.15 netmask 255.255.255.0 up
ifconfig eth4 10.0.10.2 netmask 255.255.255.0 up
ifconfig vmbr0 10.0.0.20 netmask 255.255.255.0 up
I noticed in the networking configurator, that none of them were set to auto, so, I changed that.

I could ping google, I used the web interface to set up my vm's, everything worked.
Now that I've rebooted, I can't load the web interface, or ping google.

I'm so very close to going back to ESXi.

 

[Richard]

When I installed proxmox (both times) /etc/network/interfaces contained only this:

auto vmbr0
iface vmbr0 inet static
address 10.0.0.20
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0

I had no entries for the quad port GBe NIC, eth 0-3, and no entry for eth4, my Mellanox Connect-X2.

through a series of edits, reboots, and luck/magic, I managed to get all of my interfaces entered, and working, even using iSCSI over the 10GBe Mellanox.

I made a few more networking changes after installing a few WORKING VM's, to create new bridges for other VM's I was about to install. The server suggested I restart to apply changes, and now NOTHING works. I cannot reach the web interface, I cannot ping google standing at the rack.

/etc/network/interfaces contains ALL of the following:

auto lo
auto lo inet loopback

auto eth0
iface eth0 inet static
address 10.0.0.12
netmask 255.255.255.0
gateway 10.0.0.2
broadcast 10.0.0.255
network 10.0.0.0

auto eth1
iface eth1 inet static
address 10.0.0.13
netmask 255.255.255.0
broadcast 10.0.0.255
network 10.0.0.0

auto eth2
iface eth2 inet static
address 10.0.0.14
netmask 255.255.255.0
broadcast 10.0.0.255
network 10.0.0.0

auto eth3
iface eth3 inet static
address 10.0.0.15
netmask 255.255.255.0
broadcast 10.0.0.255
network 10.0.0.0

auto eth4
iface eth4 inet static
address 10.0.10.2
netmask 255.255.255.0
broadcast 10.0.10.255
network 10.0.10.0

auto vmbr0
iface vmbr0 inet static
address 10.0.0.20
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0
#WINDOWS-DESKTOPS

auto vmbr1
iface vmbr1 inet static
address 10.0.0.40
netmask 255.255.255.0
bridge_ports eth1
bridge_stp off
bridge_fd 0
#LINUX-SERVERS

auto vmbr2
iface vmbr2 inet static
address 10.0.0.50
netmask 255.255.255.0
bridge_ports eth2
bridge_stp off
bridge_fd 0
#LINUX-DESKTOPS

What finally made everything work last night, was:
ifconfig eth0 10.0.0.12 netmask 255.255.255.0 up
ifconfig eth1 10.0.0.13 netmask 255.255.255.0 up
ifconfig eth2 10.0.0.14 netmask 255.255.255.0 up
ifconfig eth3 10.0.0.15 netmask 255.255.255.0 up
ifconfig eth4 10.0.10.2 netmask 255.255.255.0 up
ifconfig vmbr0 10.0.0.20 netmask 255.255.255.0 up
I noticed in the networking configurator, that none of them were set to auto, so, I changed that.

I could ping google, I used the web interface to set up my vm's, everything worked.
Now that I've rebooted, I can't load the web interface, or ping google.

I'm so very close to going back to ESXi.

Important:

- do not assign addresses in the same subnet (10.0.0.0/24 in your case) to different interfaces

- do not assign any address to a bridge port (eth0, 1, 2 bin your case).


After installation you will have usually bridge vmbr0 with just one bridge port (usually eth0) configured.

After booting the first time all physically existing NICs will appear as inactive in the WEB GUI Datacenter -> <host> -> Network

It's recommended to use then GUI for configuring all the other interfaces, create bridges etc.

 

[dmattox10]

I won't pretend to understand in depth Linux networking, but on vsphere, those ports ARE on the same subnet, and I need that in my use case, my VMS communicate with each other. Thanks for your reply, I'll stick to vsphere, a mod can close this thread.