proxmox on zotac zbox nano with wireless and realtek ethernet dual ports

T

twerp71

New Member
Sep 18, 2022
3
0
1
I have a zotac xbox C133 nano that has wireless as well as dual ethernet ports, and would like to use this as my firewall with opnsense. Ideally, I would like to use the wireless connection for management access to promox only, and passthrough to the dual ethernet ports for firewall wan and lan
I have been able to get the wireless connection going and am able to access proxmox as admin. I have also been able to set up passthrough using the IOMMU related posts, but have noticed that all my traffic seems to be going through the wireless connection. Using iperf, I am getting decent (GB/s) between machines on my lan, but consistently getting 24MB/s up / 18 MB/s down for all traffice to the zbox - i.e.
- proxmox shell to another machine on the LAN
- proxmox shell to VM within zbox
- VM within zbox to another machine on the LAN

I am getting the same speeds when bridging the ethernet ports
Here are the interfaces configurations

passthrough
auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet static
address 192.168.6.30/24
gateway 192.168.6.1

auto enp1s0
iface enp1s0 inet static
address 192.168.6.31/24
gateway 192.168.6.1

auto wlan0
iface wlan0 inet static
address 192.168.6.29/24
gateway 192.168.6.1


bridged
auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.6.30/24
gateway 192.168.6.1
bridge-ports enp1s0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.6.31/24
bridge-ports enp2s0
bridge-stp off
bridge-fd 0

auto wlan0
iface wlan0 inet static

address 192.168.6.29/24
gateway 192.168.6.1

Any pointers to what I am doing wrong here would be greatly appreciated.
 
It is probably routing all the traffic through the Wi-FI Interface, since you have the same CIDR configured for all 3 interfaces. Try using different subnets for the other interfaces e.g. 192.168.7.x/24 . You can verify this by running route on the host. There it probably shows only one route.
 
Thanks for the response - I will look into subnets, as that seems to be the better approach here
 
@twerp71,
It looks like you are trying to achieve very similar result (which is I am planning).
Have you succeeded in getting opnsense to work on this machine under proxmox?
Did you use pass through?
Do you by the way know which of the LAN ports is 01:00.0 and which one is 02:00.0?
My thinking was to use one of the LAN ports as a WAN and pass it through to the pfsense directly (no any other "risky" exposure), and another one to be used for anything else.
 
You need to check the IOMMU groups. If those NICs are sharing a group with other chipset stuff, you won't be able to use PCI passthrough (atleast not without acs override which is on it own a security concern, because it will break the PCIe isolation).
 
I gave up on wireless and tried a USB-C ethernet adapter - speeds were good with the adapter, but still really slow within the VMs using either bridged or passthrough ethernet ports (max around 35mbps down, 50mbps up). This was tested by installing regular ubuntu within the VM.

I did read up on a lot of articles pointing to realtek driver updates for proxmox/debian, but never got round to it. I gave up and wiped proxmox and installed linux mint instead to use as a regular desktop - both ethernet ports are now working fine (>350mbps)

Will continue to search for a replacement box, but will make sure the ethernet ports are intel.
 
Dunuin said:
You need to check the IOMMU groups. If those NICs are sharing a group with other chipset stuff, you won't be able to use PCI passthrough (atleast not without acs override which is on it own a security concern, because it will break the PCIe isolation).
Click to expand...
this is what I get from the system

Code: 
root@pve1001:~# lspci
00:00.0 Host bridge: Intel Corporation Device 4e12
00:02.0 VGA compatible controller: Intel Corporation Device 4e61 (rev 01)
00:08.0 System peripheral: Intel Corporation Device 4e11
00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
00:14.3 Network controller: Intel Corporation Device 4df0 (rev 01)
00:15.0 Serial bus controller [0c80]: Intel Corporation Device 4de8 (rev 01)
00:15.2 Serial bus controller [0c80]: Intel Corporation Device 4dea (rev 01)
00:16.0 Communication controller: Intel Corporation Device 4de0 (rev 01)
00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
00:19.0 Serial bus controller [0c80]: Intel Corporation Device 4dc5 (rev 01)
00:19.1 Serial bus controller [0c80]: Intel Corporation Device 4dc6 (rev 01)
00:1c.0 PCI bridge: Intel Corporation Device 4dbc (rev 01)
00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev 01)
00:1e.3 Serial bus controller [0c80]: Intel Corporation Device 4dab (rev 01)
00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
00:1f.4 SMBus: Intel Corporation Device 4da3 (rev 01)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device 4da4 (rev 01)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
root@pve1001:~# lspci -t
-[0000:00]-+-00.0
           +-02.0
           +-08.0
           +-14.0
           +-14.2
           +-14.3
           +-15.0
           +-15.2
           +-16.0
           +-17.0
           +-19.0
           +-19.1
           +-1c.0-[01]----00.0
           +-1c.5-[02]----00.0
           +-1e.0
           +-1e.3
           +-1f.0
           +-1f.3
           +-1f.4
           \-1f.5

and

Code: 
root@pve1001:/sys/class/net# ls -lart
total 0
drwxr-xr-x 82 root root    0 Dec 29 12:57 ..
lrwxrwxrwx  1 root root    0 Dec 29 12:57 lo -> ../../devices/virtual/net/lo
drwxr-xr-x  2 root root    0 Dec 29 12:57 .
lrwxrwxrwx  1 root root    0 Dec 29 12:57 enp2s0 -> ../../devices/pci0000:00/0000:00:1c.5/0000:02:00.0/net/enp2s0
lrwxrwxrwx  1 root root    0 Dec 29 12:57 enp1s0 -> ../../devices/pci0000:00/0000:00:1c.0/0000:01:00.0/net/enp1s0
-rw-r--r--  1 root root 4096 Dec 29 12:57 bonding_masters
lrwxrwxrwx  1 root root    0 Dec 29 12:57 vmbr0 -> ../../devices/virtual/net/vmbr0
lrwxrwxrwx  1 root root    0 Dec 29 12:57 vmbr101 -> ../../devices/virtual/net/vmbr101
lrwxrwxrwx  1 root root    0 Dec 29 12:57 vmbr2 -> ../../devices/virtual/net/vmbr2
lrwxrwxrwx  1 root root    0 Dec 29 12:59 wlo1 -> ../../devices/pci0000:00/0000:00:14.3/net/wlo1

Does it looks like it is a group, which cannot "separate" individual NICs?
 
Last edited:
it gives nothing
Code: 
root@pve1001:~# find /sys/kernel/iommu_groups/ -type l
root@pve1001:~#
The respective folder is empty
 
I feel something is missing... or?... your, @Dunuin, help / advise how to decrypt this would be appreciated.
Do I understand correctly, all these steps are essential to get pass through for PCI devices, right?
Does it mean the pass through for individual LAN adapter is enabled and should be possible?

GRUB settings are changed:
Code: 
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

GRUB is updated
Code: 
root@pve1001:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.83-1-pve
Found initrd image: /boot/initrd.img-5.15.83-1-pve
Found linux image: /boot/vmlinuz-5.15.74-1-pve
Found initrd image: /boot/initrd.img-5.15.74-1-pve
Found linux image: /boot/vmlinuz-5.15.64-1-pve
Found initrd image: /boot/initrd.img-5.15.64-1-pve
Found linux image: /boot/vmlinuz-5.13.19-6-pve
Found initrd image: /boot/initrd.img-5.13.19-6-pve
Found linux image: /boot/vmlinuz-5.13.19-2-pve
Found initrd image: /boot/initrd.img-5.13.19-2-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done

refresh is executed:
Code: 
root@pve1001:~# proxmox-boot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

Modules have been activated:
Code: 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Remapping is supported:
Code: 
root@pve1001:~# dmesg | grep 'remapping'
[    0.138090] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.139826] DMAR-IR: Enabled IRQ remapping in x2apic mode

The line "DMAR: IOMMU enabled" is there:
Code: 
root@pve1001:~# dmesg | grep -e DMAR -e IOMMU
[    0.019090] ACPI: DMAR 0x00000000726D2000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.019126] ACPI: Reserving DMAR table memory at [mem 0x726d2000-0x726d2087]
[    0.050343] DMAR: IOMMU enabled
[    0.137957] DMAR: Host address width 39
[    0.137958] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.137964] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e
[    0.137967] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.137972] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.137974] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
[    0.137977] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.137979] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.137980] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.139717] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.320987] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.395462] DMAR: No ATSR found
[    0.395463] DMAR: No SATC found
[    0.395465] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.395466] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.395467] DMAR: IOMMU feature nwfs inconsistent
[    0.395468] DMAR: IOMMU feature pds inconsistent
[    0.395468] DMAR: IOMMU feature eafs inconsistent
[    0.395469] DMAR: IOMMU feature prs inconsistent
[    0.395470] DMAR: IOMMU feature nest inconsistent
[    0.395470] DMAR: IOMMU feature mts inconsistent
[    0.395471] DMAR: IOMMU feature sc_support inconsistent
[    0.395472] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.395473] DMAR: dmar0: Using Queued invalidation
[    0.395476] DMAR: dmar1: Using Queued invalidation
[    0.395996] DMAR: Intel(R) Virtualization Technology for Directed I/O

and yes, now this folder is not empty
Code: 
root@pve1001:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
/sys/kernel/iommu_groups/5/devices/0000:00:15.2
/sys/kernel/iommu_groups/5/devices/0000:00:15.0
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/11/devices/0000:00:1e.3
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:19.0
/sys/kernel/iommu_groups/8/devices/0000:00:19.1
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.5
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.5
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0

to add a bit more information
Code: 
root@pve1001:~# lspci -v
00:00.0 Host bridge: Intel Corporation Device 4e12
        DeviceName: Onboard - Other
        Subsystem: ZOTAC International (MCO) Ltd. Device b437
        Flags: bus master, fast devsel, latency 0, IOMMU group 0
        Capabilities: [e0] Vendor Specific Information: Len=10 <?>

<lines removed to save space, and as assumed being not relevant>

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
        Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 13
        I/O ports at 4000 [size=256]
        Memory at 7fd00000 (64-bit, non-prefetchable) [size=4K]
        Memory at 6000100000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [170] Latency Tolerance Reporting
        Kernel driver in use: r8169
        Kernel modules: r8169

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
        Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 14
        I/O ports at 3000 [size=256]
        Memory at 7fc00000 (64-bit, non-prefetchable) [size=4K]
        Memory at 6000000000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [170] Latency Tolerance Reporting
        Kernel driver in use: r8169
        Kernel modules: r8169
 
Last edited:
AndroGen said:
Do I understand correctly, all these steps are essential to get pass through for PCI devices, right?
Click to expand...
Jup.
AndroGen said:
Does it mean the pass through for individual LAN adapter is enabled and should be possible?
Click to expand...
Your NICs are alone in group 13 and 14. So yes, passthrough should work. You should now be able to add one of the NICs to a VM using the webUI.
 
  • Like
Reactions: AndroGen
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back