So I finally figured this out. And for anyone who might be interested in doing the same thing, I'll list the steps I did to make this work.
For some reason, the version 8 ISO of Proxmox would not work for me. It booted the installer GUI just fine, but while it was expanding the contents of the ISO to the local volume (I left everything at default settings), it would fail and say it couldn't find a command or something like that. I tried making that boot drive using *dd* on the Mac, then I tried Etcher on the Mac ... then I tried a different SD card and a different USB to SD gadget ... re-tried both of those methods, then I tried Etcher in Windows and then Rufis in dd mode in Windows, nothing worked, the install failed every time at the exact same place.
SO... I decided to install version 7 and do an in-place upgrade. Which worked fine.
When installing version 7, at the point where it asks for the local IP address, I chose the second NIC in the machine because the first NIC is plugged directly into the cable modem. I gave that NIC an address of
10.10.10.2 (because
.1 will be the LAN side of OPNsense).
Once Proxmox was installed and booted, I made all of these changes via command line and I didn't use the web GUI until after the upgrade to version 8 was complete.
I started by editing the network interfaces file so that the Internet would be accessible to Debian
pico /etc/network/interfaces
auto lo
iface lo inet loopback
auto enp2s0
iface enp2s0 inet dhcp
auto vmbr0
iface vmbr0 inet static
address 10.10.11.2/24
gateway 10.10.11.1
bridge_ports enp3s0
bridge_stp off
bridge_fd 0
iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
vmbr0 was created by Proxmox and so I left that part untouched. I did, however, move the manual lines to the bottom of the file and I put the lines that establish the Internet connection to the top so that the configs that create working objects are at the top and the nics that are there for sake of existing are at the bottom.
Save this file
Normally you would just reset the NICs to get this config working. However, in my case, resetting the NICs did give that Interface a public IP address from the cable modem, but routing wasn't working (no access to the Internet) so I rebooted and then the Internet was accessible.
Somewhere in this config, the grub file needs to be changed so that PCI Passthru will work - may as well change it now:
Changed this line to add the intel value at the end:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
It is necessary to fully update version 7 before upgrading to version 8. However, we need to add a line to the repo lists file so that we pull down a program that will let us confirm that the box is ready for the upgrade.
First, you edit the lists file
pico /etc/apt/sources.list
And make sure that file looks like this:
deb
http://ftp.debian.org/debian bullseye main contrib
deb
http://ftp.debian.org/debian bullseye-updates main contrib
# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb
http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# security updates
deb
http://security.debian.org/debian-security bullseye-security main contrib
Then we update and upgrade. Don't worry that we're doing a dist-upgrade here, this is necessary so that it pulls down the next command that we run.
apt update
apt dist-upgrade -y
The documents I read said to run this command, but I'm sure it's only necessary if you're upgrading a version 7 box that has been in a production. environment. Since this is a clean install, this step should not be necessary though it is certainly harmless and it will point anything that needs to be resolved before doing the upgrade. I think the main issue is to make sure any running VMs are stopped and backed up before the upgrade.
Next, we need to change the repositories lists file so that we are now pulling from the Proxmox 8 repos, and the only thing we need to do for that change is replace the word `bullseye` in that file with the word `bookworm`:
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list
Then, we update the local repo to match the upstream repo
The system is now ready for the upgrade to version 8. This next command will take some time to complete. I have a gigabit Internet connection and it took about 20 minutes or so to finish, and it will ask you some questions along the way. Basically it's going to ask you if it is OK to replace some of the config files with the upgrade version, because it determines that these config files have changed from their default after installing version 7.
When it asked about replacing the network interfaces file, I told it NO, but I said YES to all of the others, INCLUDING the grub settings file, because it keeps the changes we made, in the replacement file.
Upgrade to version 8:
You'll want to babysit the upgrade or periodically check back with it so you can respond to the prompts along the way.
When the upgrade finishes, the guides said to run this first command, but again, I think it's only necessary when upgrading a production environment and if you have errors during the upgrade. The command is harmless either way. The second command is just something I do when I commit major changes through a package manager.
apt -f install
apt autoremove
Finally, we need to remove the Internet connection from Debian because we are going to use OPNsense for Internet access. Make sure you have the OPNsense ISO file downloaded (not the IMG file - select DVD from their download page to get the ISO).
To remove the Internet from Debian:
pico /etc/network/interfaces
Replace these two lines:
auto enp2s0
iface enp2s0 inet dhcp
With this one line:
Your NICs name might be different than mine, so don't copy and paste that step.
Next:
Version 8 Steps
Now that the upgrade is complete, and version 8 is running, We need to create our bridge.
If you want to skip past my long-winded explanation of why this config is possible in Proxmox, scroll down to Steps.
This is the part where I really can appreciate Proxmox and how it is implemented, where it is essentially a hypervisor running on a standard installation of Debian, where Debian itself was never altered so that Proxmox would be a typical "appliance" like so many others. There is tremendous power in having done it this way in the sense that we have the flexibility of being able to work with Debian where Proxmox simply must allow whatever changes we make to Debian (within reason of course). And from what I can tell, Proxmox creates a virtual network layer in the environment which includes virtual network interface cards that we then assign to our virtual machines. Proxmox then handles all of the traffic between virtual interfaces and the hardware NIC provided to it from Debian.
This is why we are able to do what we are going to do next. In ESX, this is not possible because ESX is a fully hardened appliance which is essentially a hypervisor that is also a fully contained , custom build of an underlying *nix OS of some kind. But the user of ESX has no access to anything at the OS level and they did not design ESX so that a config like this could be done. Their answer to my problem was to use PCI passthru for all of my NICs, but that then would not have given me the ability to share that bridge with any other VMs without leveraging an external switch.
My praise of Proxmox also deserves a bit of balance. The downside to implementing the hypervisor on top of a standard install of Debian means that Proxmox is vulnerable to any weaknesses in Debian. Now Debian might not have any weaknesses and its attack surface might be impenetrable … but the unavoidable reality is that the attack surface on a default OS install like Debian is LARGE. In fact, it is significantly larger than ESX because ESX has basically stripped out all attack surfaces in the OS, leaving only the ESX virtual layers in the environment as the attack surface. This is also why ESX doesn't really ever need to be patched or updated outside of major version upgrades. The only time you typically ever do any updates to the ESX OS is when you need to add drivers for unsupported hardware.
Steps
This last part is simple and straightforward. Edit your network interfaces file and change it so it looks similar to what I have here. The ethernet port that is connected to the cable modem becomes a blind spot to Debian - BECAUSE of this config and also because we are going to assign that ethernet port to the OPNSense virtual machine using PCI passthru, which further removes that connection from Debian and thus Proxmox. After this, Proxmox can only access the Internet by using the LAN side of OPNSense as its gateway.
pico /etc/network/interfaces
Proxmox should have already created a bridge that has a single NIC in it. You will want to keep that bridge name the same so that Proxmox continues to use it without needing to change anything within the Proxmox environment. In this example, Proxmox created the name `vmbr0`.
Your file should look something like this:
auto lo
iface lo inet loopback
auto vmbr0
iface vmbr0 inet static
address 10.10.10.2/24
gateway 10.10.10.1
bridge_ports enp3s0 enp4s0 enp5s0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
iface enp2s0 inet manual
iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
In this config, NIC `enp2s0 ` is the NIC connected to my cable modem directly.
Save the file
Then reboot
Now, for the first time, we need to open the web interface for Proxmox.
Upload the OPNSense ISO to Proxmox by expanding
Storage and selecting
local (NOT local-lvm).
Click on
ISO Images, then click on
upload and go find the ISO you downloaded and extracted. Wait for it to upload into Proxmox.
Create a new Virtual Machine for OPNSense (leave all settings at default except where noted).
Give it a name then click next
Chose the ISO you uploaded then click Next
Click next again
I set the disk size to 25 (gigs). Click Next
I gave it 2 CPU sockets with 2 cores each, click next
I gave it 4096 memory (4 gigs), click next
Check the box next to No network device, then click next
Click on Finish.
Once the VM is built, click on it and click on
Hardware.
Add a PCI device and in the window that pops up, select
Raw Device and also check the
All Functions box.
Next, add a network device and change the model to
VMWare vmxnet3.
That should be it. At this point when you install OPNSense.
Once the OPNSense install is done, STAY IN THE COMMAND LINE INTERFACE and log into it and assign the ports, with the WAN port being the PCI passthru device. If you let Proxmox create your virtual nic name, then it will be the NIC that is not called vmx0, where vmx0 will be the LAN inside OPNSense. If you don’t use default private IP addys then you will also need to change the IP address of the LAN nic and establish DHCP etc. etc.
You should have a working environment at this point.
What I find interesting about this setup, is that even though OPNSense has been given ONE virtual NIC that Proxmox understands as being the physical nic called `vmbr0`, Debian is able to access the Internet by using 10.10.10.1 as its default gateway, and it does this by sending traffic through the virtual NIC that IT created from the above bridge config and that NIC has an ip address of 10.10.10.2.
So we have three physically isolated NICs that exist at the OS level as a bridged NIC and so that NIC is a software ethernet port having its own IP address. Debian does not have any awareness of the ProxMox virtual NIC because Debian is hosting Proxmox, it's not one of Proxmox's clients …
Yet in spite of that, Proxmox still knows that any traffic coming from that bridge NIC that Debian created needs to get sent through to the virtual nic that is assigned to OPNSense and I have to imagine that it does this at layer 2, where layer 3 is being handled by OPNSense.
It's all smoke and mirrors I suppose under the hood … lol.
My only criticism of Proxmox is that it does tend to get lost when you have a config in the GUI but then you make a change in the OS config files for whatever ... ProxMox can't seem to reconcile those external changes and it gets kinda stupid after that... there are other aspects of it that could use some improvement, but overall its a great hypervisor.
