Bridge won't start on new install - any way to check what's wrong?

J

joechssn

New Member
Mar 4, 2023
5
0
1
Bridge won't come up. Please explain what I'm doing wrong.

Current environment:
  • PVE host
    • Bridge Interface (vmbr0) 192.168.1.20/24
    • Physical Interface (eno1) 192.168.1.22/24
  • PfSense VM
    • Intel I350-T4 PCI Passthrough (vfio-pci driver)
      • enp1s0f0 WAN - public IP from ISP
      • enp1s0f1 LAN - 192.168.1.1 (GW)


Problem description:
  1. PVE can't ping domain names or 8.8.8.8
  2. PVE can ping the GW (PfSense LAN) but nothing past it
  3. Bridge vmbr0 will not come up

Things I've tried:
  1. I can ping domain names or 8.8.8.8 from PfSense VM and other PCs on LAN
  2. I can ping PVE & eno1 from PfSense VM
  3. verified eno1 & enp1s0f1 physical interfaces are UP
  4. verified /etc/network/interfaces is correct
  5. verified PCI Passthrough is correct (to the best of my knowledge)

vmbr0 refuses to start and I don't know what else to do.



Code: 
/etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address 192.168.1.22/24
# backup way to access PVE in case pfSense ever goes down

auto enp1s0f0
iface enp1s0f0 inet manual

auto enp1s0f1
iface enp1s0f1 inet manual

auto enp1s0f2
iface enp1s0f2 inet manual

auto enp1s0f3
iface enp1s0f3 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.20/24
        gateway 192.168.1.1
        bridge-ports enp1s0f1
        bridge-stp off
        bridge-fd 0

Noticed my first clue: bridge is down
Code: 
ip route
default via 192.168.1.1 dev vmbr0 proto kernel onlink linkdown 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.22 
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.20 linkdown

So I checked to verify if enp1s0f1 was up:
1686372108168.png

So I tried to bring vmbr0 up:
Code: 
ip link set vmbr0 up
root@pve:~# ip link show vmbr07: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000    link/ether a0:36:9f:2f:27:6d brd ff:ff:ff:ff:ff:ff


Other info:
Code: 
lspci | grep Ethernet
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Code: 
lspci -nnk
01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection
[8086:1521] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 [8086:5001]
        Kernel driver in use: vfio-pci
        Kernel modules: igb
01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection 
[8086:1521] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 [8086:5001]
        Kernel driver in use: vfio-pci
        Kernel modules: igb0
1:00.2 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection 
[8086:1521] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 [8086:5001]
        Kernel driver in use: vfio-pci
        Kernel modules: igb0
1:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection 
[8086:1521] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4 [8086:5001]
        Kernel driver in use: vfio-pci
        Kernel modules: igb

Code: 
/proc/cmdline
initrd=\EFI\proxmox\5.15.107-2-pve\initrd.img-5.15.107-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

Code: 
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done;
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0d)
IOMMU group 10 01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 11 01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 12 01:00.2 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 13 01:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
IOMMU group 2 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU group 3 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU group 4 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU group 5 00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 [8086:a368] (rev 10)
IOMMU group 6 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU group 6 00:16.3 Serial controller [0700]: Intel Corporation Cannon Lake PCH Active Management Technology - SOL [8086:a363] (rev 10)
IOMMU group 7 00:17.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 10)
IOMMU group 8 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
IOMMU group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Q370 Chipset LPC/eSPI Controller [8086:a306] (rev 10)
IOMMU group 9 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU group 9 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU group 9 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU group 9 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)


Code: 
lsmod | grep vfio
vfio_pci               16384  4
vfio_pci_core          73728  1 vfio_pci
vfio_virqfd            16384  1 vfio_pci_core
irqbypass              16384  27 vfio_pci_core,kvm
vfio_iommu_type1       45056  1
vfio                   45056  11 vfio_pci_core,vfio_iommu_type1


Am I doing something wrong? Any ideas how to troubleshoot this?
TIA!
 
you can passthrough a physical nic to the vm, and at same time use it on the hypervisor.
(the nic is "detached" from hypervisor, you should be able to see it with #ip addrr" on hypervisor after the vm have started

If you passthourgh you wan/lan nic to your pfsense, it need to go back to your physical switch, then go back to your hypervisor vms through another phyiscal nics
 
spirit said:
you can passthrough a physical nic to the vm, and at same time use it on the hypervisor.
(the nic is "detached" from hypervisor, you should be able to see it with #ip addrr" on hypervisor after the vm have started

If you passthourgh you wan/lan nic to your pfsense, it need to go back to your physical switch, then go back to your hypervisor vms through another phyiscal nics
Click to expand...
Thanks for the reply, spirit - it is just as you described in the 2nd paragraph: WAN & LAN are passthrough. WAN to ISP router, LAN to switch and another phyiscal nic from host to switch.

Host can ping 1.1 GW but nothing past the GW. I don't know enough to figure out why
 
I think I fixed it. The host cannot see the enp1s0f1 interface since it is passthrough so the vmbr0 could not start. I removed enp1s0f1 from the bridge as a bridge port and replaced it with eno1. Pings to 192.168.1.22 (eno1) stopped though.

Code: 
**New** /etc/network/interfaces

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address 192.168.1.22/24
        gateway 192.168.1.1

auto enp1s0f0
iface enp1s0f0 inet manual

auto enp1s0f1
iface enp1s0f1 inet manual

auto enp1s0f2
iface enp1s0f2 inet manual

auto enp1s0f3
iface enp1s0f3 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.20/24
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

up ip route add default via 192.168.1.1 dev vmbr0

There was no IP Route so I added this to /etc/network/interfaces:

Code: 
up ip route add default via 192.168.1.1 dev vmbr0

  1. Is there a better way to do this so I can keep 192.168.1.22?
  2. Should I set a persistent IP Route?
 
you can't have an ip address on a physical interface, if this interface is plugged in a bridge. (linux kernel refuse it)

(but you can have multiple "address ..." lines in vmbr0 if you want)

Code: 
auto eno1
iface eno1 inet manual


auto vmbr0
iface vmbr0 inet static
        address 192.168.1.20/24
        address 192.168.1.22/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
 
spirit said:
you can't have an ip address on a physical interface, if this interface is plugged in a bridge. (linux kernel refuse it)

(but you can have multiple "address ..." lines in vmbr0 if you want)

Code: 
auto eno1
iface eno1 inet manual


auto vmbr0
iface vmbr0 inet static
        address 192.168.1.20/24
        address 192.168.1.22/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
Click to expand...

Is there any benefit to having 192.168.1.22? It's an emergency address in case PfSense is down and is not handing out DHCP
 
Should I leave the IP Route in /etc/network/interfaces or create a persistent route?

Code: 
up ip route add default via 192.168.1.1 dev vmbr0
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom