Forgive me in advance, I've been a VMware admin for over 20 years and am moving to Proxmox in my home lab. Having some mind-wrapping issues around some things that I'm hoping to bounce off a few folks.
First, the setup, My vSphere cluster was backed by a TrueNAS Scale box running on some overpowered hardware, so my first thought was to just "migrate" the VMs from vSphere over to TrueNAS Scale. After some pain of exporting OVF's, converting to qcow then writing that to the raw ZVOLs, I have almost all of my VMs running on my TrueNAS box. Now I know what some are going to say, but keep in mind the primary purpose for the TrueNAS box was as a iSCSI NAS for the vSphere, and some other lightweight things. Here, let me give you the specs of the TrueNAS box, which is also the same as the other box I'm looking to build Proxmox on.
Dell R440
(x2) Intel Xeon Silver 4208 CPU @ 2.10GHz
256GB DDR4 2666 RAM
Dell HBA330 in IT mode
(x2) 120GB Intel Datacenter SSDs with PLP
(x6) 3.4TB Samsung Datacenter SSDs
BCM57810 Dual 10GB SFP+ NIC
(x2) BCM5720 1GB LOM NIC
So yeah, way overkill for TrueNAS and now that I've pretty much dumped the iSCSI load, it's a little bit of NFS storage and a hypervisor. So my plan is to build the second box as Proxmox, install TrueNAS Scale as a VM there, move the non-VM storage over to this new VM, then figure out how to move/convert the VMs from the bare-metal TrueNAS box to Proxmox.
So, after a few hours of tinkering, I got the two 10GB NICs in a LACP bond and got a bridge attached to that then several VLAN interfaces off the bridge. I probably won't stick with this in the long-run, probably just keep the two NICs in a bond with Round-Robin load-balancing and remove the LACP config. The plan is to use that bond for service delivery, container networking, and VM networking, basically anything not management. Oh yeah, I have docker running in Dockage and in VMs on TrueNAS, want to make those native on Proxmox, but that's like Phase 2 at this point. The management will be on one of the 1GB LOMs, with the other LOM going to eventually be connected to my ISP router to feed the WAN side of a pfSense box. I have a R440 running my primary pfSense box, the virtual one would be a backup to the main on in a HA configuration.
My immediate question is around the storage. So I've got the two 120GB disks in a ZFS mirror and installed the Proxmox OS there. As for the remaining 6 drives, originally I was going to make a 3-wide ZFS RAID10, but then I started thinking that may not be the best choice. My total usage on the bare-metal box is 3.56TiB, with 1.2TiB being VM storage and the rest being NFS and miscellaneous storage. My thought is to use two of the 3.4TB disks for a single two-wide ZFS mirror for the VM storage pool. Then, for the data side of the TrueNAS VM, pass through two of the disks directly to the VM and then the system disk will live in the VM storage pool and the data will pool will have direct access to the underlying disks.
With that said, is it even possible to share the single HBA between Proxmox and a VM like that? Is it even necessary to do PCI passthrough on the HBA or is just presenting the disks directly to the TrueNAS VM acceptable?
The only other thing in the mix I will need to figure out is the Nvidia T400 card that I have in the TrueNAS server passed through to my Frigate install running in a container on Dockage in a Jailmkr Jail. I am definitely looking to simplify that house of cards for sure, but the GPU and associated USB TPU is critical for the Frigate container when it get's moved to Proxmox, but one thing at a time I suppose.
Just wanting to bounce off the community and get a little feedback and learn a bit about this whole new ecosystem in the process. Also, I won't be running a cluster, just a stand-alone Proxmox at my main house and then rebuild the existing bare-metal TrueNAS box into a duplicate Proxmox box at my second home to replace the vSphere boxes there.
First, the setup, My vSphere cluster was backed by a TrueNAS Scale box running on some overpowered hardware, so my first thought was to just "migrate" the VMs from vSphere over to TrueNAS Scale. After some pain of exporting OVF's, converting to qcow then writing that to the raw ZVOLs, I have almost all of my VMs running on my TrueNAS box. Now I know what some are going to say, but keep in mind the primary purpose for the TrueNAS box was as a iSCSI NAS for the vSphere, and some other lightweight things. Here, let me give you the specs of the TrueNAS box, which is also the same as the other box I'm looking to build Proxmox on.
Dell R440
(x2) Intel Xeon Silver 4208 CPU @ 2.10GHz
256GB DDR4 2666 RAM
Dell HBA330 in IT mode
(x2) 120GB Intel Datacenter SSDs with PLP
(x6) 3.4TB Samsung Datacenter SSDs
BCM57810 Dual 10GB SFP+ NIC
(x2) BCM5720 1GB LOM NIC
So yeah, way overkill for TrueNAS and now that I've pretty much dumped the iSCSI load, it's a little bit of NFS storage and a hypervisor. So my plan is to build the second box as Proxmox, install TrueNAS Scale as a VM there, move the non-VM storage over to this new VM, then figure out how to move/convert the VMs from the bare-metal TrueNAS box to Proxmox.
So, after a few hours of tinkering, I got the two 10GB NICs in a LACP bond and got a bridge attached to that then several VLAN interfaces off the bridge. I probably won't stick with this in the long-run, probably just keep the two NICs in a bond with Round-Robin load-balancing and remove the LACP config. The plan is to use that bond for service delivery, container networking, and VM networking, basically anything not management. Oh yeah, I have docker running in Dockage and in VMs on TrueNAS, want to make those native on Proxmox, but that's like Phase 2 at this point. The management will be on one of the 1GB LOMs, with the other LOM going to eventually be connected to my ISP router to feed the WAN side of a pfSense box. I have a R440 running my primary pfSense box, the virtual one would be a backup to the main on in a HA configuration.
My immediate question is around the storage. So I've got the two 120GB disks in a ZFS mirror and installed the Proxmox OS there. As for the remaining 6 drives, originally I was going to make a 3-wide ZFS RAID10, but then I started thinking that may not be the best choice. My total usage on the bare-metal box is 3.56TiB, with 1.2TiB being VM storage and the rest being NFS and miscellaneous storage. My thought is to use two of the 3.4TB disks for a single two-wide ZFS mirror for the VM storage pool. Then, for the data side of the TrueNAS VM, pass through two of the disks directly to the VM and then the system disk will live in the VM storage pool and the data will pool will have direct access to the underlying disks.
With that said, is it even possible to share the single HBA between Proxmox and a VM like that? Is it even necessary to do PCI passthrough on the HBA or is just presenting the disks directly to the TrueNAS VM acceptable?
The only other thing in the mix I will need to figure out is the Nvidia T400 card that I have in the TrueNAS server passed through to my Frigate install running in a container on Dockage in a Jailmkr Jail. I am definitely looking to simplify that house of cards for sure, but the GPU and associated USB TPU is critical for the Frigate container when it get's moved to Proxmox, but one thing at a time I suppose.
Just wanting to bounce off the community and get a little feedback and learn a bit about this whole new ecosystem in the process. Also, I won't be running a cluster, just a stand-alone Proxmox at my main house and then rebuild the existing bare-metal TrueNAS box into a duplicate Proxmox box at my second home to replace the vSphere boxes there.