Hi there.
Since I'm using 3 VM's on a Proxmox I'm trying to use more than just one "vmbr'ed" ethernet device for network traffic.
DELL R710 has 4x1 gigabit NICs. However PROXMOX shows them in a same PCIE group. So far forcing passthrough is a fail.
# lspci | grep Ethernet
01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Any ideas on successful PCIE passthrough ?
I get the "vfio: failed to set iommu for container: Operation not permitted" and fail vm startup while trying any pcie passthrough (even with passthrough entire group).
On the other hand... bonding NICs into one interface might also do the job (at least for now).
I'm thinking of using 1st Broadcom as a Proxmox management and other 3 for bond/stack and use it for VM's.
BTW: The network has single router/gateway with DHCP on it.
It apears I can't define same gateway on more than 1 PM Host interface. If so - how can I allow internet access to both eno1 (Proxmox host) and bond0 interface?
So far I have:
eno1 = static IP proxmox host ethernet interface
bond0 = bond interface with eno2, eno3 and eno4
vmbr0 = static IP bridge with bond0 attached
It seems even if my VMs have DHCP enabled and VM get IP from DHCP server I have no internet connection (just LAN available/browseable). I've tested connections on Win2016 VM. With network diagnostics I get the info: default gateway is inaccessible (or something), while successfully pinging lan devices. Funny thing is... when loop a ping to lets say 8.8.8.8 I sometimes get some packet responds.
/etc/network/interfaces:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address 192.168.1.193
netmask 255.255.255.0
auto eno2
iface eno2 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto bond0
iface bond0 inet manual
slaves eno2 eno3 eno4
bond_miimon 100
bond_mode balance-rr
auto vmbr0
iface vmbr0 inet static
address 192.168.1.192
netmask 255.255.255.0
gateway 192.168.1.252
bridge_ports bond0
bridge_stp off
bridge_fd 0
So even with network settings shown above I'm (usually) unable to get any internet on vm's (no matter DHCP or static IP setup on vm's. I'm able to ping lan but no response on router/gateway ping).
PS:
# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d4:ae:52:be:63:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.193/24 brd 192.168.1.255 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:febe:6358/64 scope link
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
10: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether ea:1a:b3:08:43:b4 brd ff:ff:ff:ff:ff:ff
11: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.192/24 brd 192.168.1.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:febe:635a/64 scope link
valid_lft forever preferred_lft forever
Since I'm using 3 VM's on a Proxmox I'm trying to use more than just one "vmbr'ed" ethernet device for network traffic.
DELL R710 has 4x1 gigabit NICs. However PROXMOX shows them in a same PCIE group. So far forcing passthrough is a fail.
# lspci | grep Ethernet
01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Any ideas on successful PCIE passthrough ?
I get the "vfio: failed to set iommu for container: Operation not permitted" and fail vm startup while trying any pcie passthrough (even with passthrough entire group).
On the other hand... bonding NICs into one interface might also do the job (at least for now).
I'm thinking of using 1st Broadcom as a Proxmox management and other 3 for bond/stack and use it for VM's.
BTW: The network has single router/gateway with DHCP on it.
It apears I can't define same gateway on more than 1 PM Host interface. If so - how can I allow internet access to both eno1 (Proxmox host) and bond0 interface?
So far I have:
eno1 = static IP proxmox host ethernet interface
bond0 = bond interface with eno2, eno3 and eno4
vmbr0 = static IP bridge with bond0 attached
It seems even if my VMs have DHCP enabled and VM get IP from DHCP server I have no internet connection (just LAN available/browseable). I've tested connections on Win2016 VM. With network diagnostics I get the info: default gateway is inaccessible (or something), while successfully pinging lan devices. Funny thing is... when loop a ping to lets say 8.8.8.8 I sometimes get some packet responds.
/etc/network/interfaces:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address 192.168.1.193
netmask 255.255.255.0
auto eno2
iface eno2 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto bond0
iface bond0 inet manual
slaves eno2 eno3 eno4
bond_miimon 100
bond_mode balance-rr
auto vmbr0
iface vmbr0 inet static
address 192.168.1.192
netmask 255.255.255.0
gateway 192.168.1.252
bridge_ports bond0
bridge_stp off
bridge_fd 0
So even with network settings shown above I'm (usually) unable to get any internet on vm's (no matter DHCP or static IP setup on vm's. I'm able to ping lan but no response on router/gateway ping).
PS:
# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d4:ae:52:be:63:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.193/24 brd 192.168.1.255 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:febe:6358/64 scope link
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
10: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether ea:1a:b3:08:43:b4 brd ff:ff:ff:ff:ff:ff
11: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:ae:52:be:63:5a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.192/24 brd 192.168.1.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:febe:635a/64 scope link
valid_lft forever preferred_lft forever
Last edited: