This is a thread to share experience and information regarding my latest project: virtulize Unraid(maybe 7.0) on PVE on a brand new Terramaster F6-424 Max(Max from here on) that will replace my main pve-node. I will update the post as I go. My hope is that this can help others in the future.
Background
Im running a two-node cluster with a qdevice. I virtulize everything because of the flexibility it gives me to test new and different OSs and applications with ease. On the cluster I run Pfsense FW, Fileservers, media-apps(*arrs, plex etc), prod-apps(nextcloud, vaultwarden etc), utils(unifi, pihole etc.) in a total of ~10 VMs and ~10 lxcs. A couple of the vms serves as dockerhosts and runs docker containers, all of which is administrated through a single Portainer instance.
Normally I run everything on my main node(pve1), and using the secondary node(pve2) as a replication and backup target for pve1. The only time I run services on pve2(except for Proxmox Backup Server which always runs here), is when I need to take down pve1 for some reason. Then I Migrate all services from pve1 to pve2.
Todays Hardware
PVE1: DIY-rig.
Why on earth do you do ZFS on Luks??? Fair question...
But the answer is quite simple: Proxmox does not (yet?) support replicating vm/ct on zfs-encrypted filesets, so to be able to keep all (more or less) important data encrypted all the time, Luks is a simple way to go, and works really well. I have automated the boot-time unlocking so Its impossible to unlock unless servers are on my network.
Drawbacks with this setup is:
pve1 power consumption ~40W
pve1 only supports 4 SATA, unless I put in another HBA with higher power consumption.
I did try the F4-424 Pro as main node, but it was not really up to par when it comes to passthrough sata-controller...
Here we have lspci:
As we see, the integrated Intel SATA Controller is 00:17.0, and then we have the ASM1062 on 03:00.0
We do a ls -al /sys/class/ata_port/ata* and gets
Here we see that the two first SATA-ports connects to Intel-SATA, which cant be passthroughed(is that a word?), and SATA port 3,4 is connected to the ASM1062, which CAN be passed trough to VM. So only 2 of 4 SATAs. The two first can be used with disk-passthrough, although Unraid really wants full control over disks to work as intended.
Allright...what about the brand new Max(that is so brand new, It has not been delivered by the time I write this)?
With the help of @Piece on Unraid.net We got this:
And:
Joy! Here we see that all SATA-ports are connected to the ASM1166, and that means that we can give Unraid(or any other VM) full control over all 6 SATA slots in the TM. In theory...we will see for real in a couple of days.
I guess the reason we see 32 SATA-ports here, is because the ASM1166 supports Port-multiplying. Exactly how that works I leave to someone else to explain.
Setting up the new Hardware
Opened the Max, and replaced the USB-drive containing TOS with my Unraid USB drive. I also moved my drives from my DIY-rig to the Max.
The Max comes with reasonable BIOS-settings, just press "del"-key at boot to enter BIOS. This is the changes I have done(as far as I can remember...
First Boot:
You really got to love Proxmox! I simply moved all disks, modified boot-order in BIOS, and everything works as before without any issues! The only thing that you might have to do is edit /etc/network/interfaces and change i/f. In my case I didn't have to.
When Idling the Max draws 12-13W after issuing
Edit: After a few weeks after writing the above, I have a really strange problem...After a reboot none, or sometimes a few, of my 5 SSDs shows up, and only a a complete Shutdown makes the disks usable again. I have tested with a HDD, and that one always works. I have tried to change everythinig in BIOS, but to no avail.
Next Step: Follow link below.
Part 2: Get Unraid up and running
Background
Im running a two-node cluster with a qdevice. I virtulize everything because of the flexibility it gives me to test new and different OSs and applications with ease. On the cluster I run Pfsense FW, Fileservers, media-apps(*arrs, plex etc), prod-apps(nextcloud, vaultwarden etc), utils(unifi, pihole etc.) in a total of ~10 VMs and ~10 lxcs. A couple of the vms serves as dockerhosts and runs docker containers, all of which is administrated through a single Portainer instance.
Normally I run everything on my main node(pve1), and using the secondary node(pve2) as a replication and backup target for pve1. The only time I run services on pve2(except for Proxmox Backup Server which always runs here), is when I need to take down pve1 for some reason. Then I Migrate all services from pve1 to pve2.
Todays Hardware
PVE1: DIY-rig.
MB: Gigabyte B760I AORUS Pro DDR4
CPU: Intel 14500T
RAM: 64GB ram
Chassi: Fractal node 804
Disks:
2xWD RED 2TB NVME - Part1-Proxmox ZFS Mirror, Part2-VM/CT. ZFS Mirror on Luks
3xSamsung QVO 8TB SSD - Media and other "WORM" content. ZFS RAIDZ1 on Luks
PVE2: Terramaster F4-424 ProCPU: Intel N305
Ram: 32GB
Disks:
1xWD RED 2TB NVME - Part1-Proxmox ZFS, Part2-VM/CT replication. ZFS on Luks
1xSamsung 2TB EVO SSD - Backup target. ZFS on Luks
1xSeagate IronWolf 12TB - Backup target. ZFS on Luks
Why on earth do you do ZFS on Luks??? Fair question...
But the answer is quite simple: Proxmox does not (yet?) support replicating vm/ct on zfs-encrypted filesets, so to be able to keep all (more or less) important data encrypted all the time, Luks is a simple way to go, and works really well. I have automated the boot-time unlocking so Its impossible to unlock unless servers are on my network.
Drawbacks with this setup is:
pve1 power consumption ~40W
pve1 only supports 4 SATA, unless I put in another HBA with higher power consumption.
I did try the F4-424 Pro as main node, but it was not really up to par when it comes to passthrough sata-controller...
Here we have lspci:
root@pve2:~# lspci
00:00.0 Host bridge: Intel Corporation Device 4617
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:14.0 USB controller: Intel Corporation Alder Lake-N PCH USB 3.2 xHCI Host Controller
00:14.2 RAM memory: Intel Corporation Alder Lake-N PCH Shared SRAM
00:16.0 Communication controller: Intel Corporation Alder Lake-N PCH HECI Controller
00:17.0 SATA controller: Intel Corporation Alder Lake-N SATA AHCI Controller
00:1c.0 PCI bridge: Intel Corporation Device 54ba
00:1c.3 PCI bridge: Intel Corporation Device 54bb
00:1c.6 PCI bridge: Intel Corporation Device 54be
00:1d.0 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port
00:1d.1 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port
00:1f.0 ISA bridge: Intel Corporation Alder Lake-N PCH eSPI Controller
00:1f.4 SMBus: Intel Corporation Alder Lake-N SMBus
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-N SPI (flash) Controller
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
04:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD
As we see, the integrated Intel SATA Controller is 00:17.0, and then we have the ASM1062 on 03:00.0
We do a ls -al /sys/class/ata_port/ata* and gets
/sys/class/ata_port/ata1 -> ../../devices/pci0000:00/0000:00:17.0/ata1/ata_port/ata1
/sys/class/ata_port/ata2 -> ../../devices/pci0000:00/0000:00:17.0/ata2/ata_port/ata2
/sys/class/ata_port/ata3 -> ../../devices/pci0000:00/0000:00:1c.6/0000:03:00.0/ata3/ata_port/ata3
/sys/class/ata_port/ata4 -> ../../devices/pci0000:00/0000:00:1c.6/0000:03:00.0/ata4/ata_port/ata4
Here we see that the two first SATA-ports connects to Intel-SATA, which cant be passthroughed(is that a word?), and SATA port 3,4 is connected to the ASM1062, which CAN be passed trough to VM. So only 2 of 4 SATAs. The two first can be used with disk-passthrough, although Unraid really wants full control over disks to work as intended.
Allright...what about the brand new Max(that is so brand new, It has not been delivered by the time I write this)?
With the help of @Piece on Unraid.net We got this:
00:00.0 Host bridge: Intel Corporation Alder Lake-U15 Host and DRAM Controller (rev 04)
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-UP3 GT2 [Iris Xe Graphics] (rev 0c)
00:04.0 Signal processing controller: Intel Corporation Alder Lake Innovation Platform Framework Processor Participant (rev 04)
00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 04)
00:06.2 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #2 (rev 04)
00:14.0 USB controller: Intel Corporation Alder Lake PCH USB 3.2 xHCI Host Controller (rev 01)
00:14.1 USB controller: Intel Corporation Device 51ee (rev 01)
00:14.2 RAM memory: Intel Corporation Alder Lake PCH Shared SRAM (rev 01)
00:15.0 Serial bus controller: Intel Corporation Alder Lake PCH Serial IO I2C Controller #0 (rev 01)
00:15.1 Serial bus controller: Intel Corporation Alder Lake PCH Serial IO I2C Controller #1 (rev 01)
00:16.0 Communication controller: Intel Corporation Alder Lake PCH HECI Controller (rev 01)
00:1c.0 PCI bridge: Intel Corporation Device 51bc (rev 01)
00:1c.6 PCI bridge: Intel Corporation Device 51be (rev 01)
00:1d.0 PCI bridge: Intel Corporation Device 51b2 (rev 01)
00:1e.0 Communication controller: Intel Corporation Alder Lake PCH UART #0 (rev 01)
00:1e.3 Serial bus controller: Intel Corporation Alder Lake SPI Controller (rev 01)
00:1f.0 ISA bridge: Intel Corporation Alder Lake PCH eSPI Controller (rev 01)
00:1f.3 Audio device: Intel Corporation Alder Lake PCH-P High Definition Audio Controller (rev 01)
00:1f.4 SMBus: Intel Corporation Alder Lake PCH-P SMBus Host Controller (rev 01)
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-P PCH SPI Controller (rev 01)
01:00.0 Non-Volatile memory controller: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD
02:00.0 Non-Volatile memory controller: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD
03:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
04:00.0 Ethernet controller: Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
05:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02)
And:
/sys/class/ata_port/ata1 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata1/ata_port/ata1/
/sys/class/ata_port/ata10 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata10/ata_port/ata10/
/sys/class/ata_port/ata11 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata11/ata_port/ata11/
/sys/class/ata_port/ata12 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata12/ata_port/ata12/
/sys/class/ata_port/ata13 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata13/ata_port/ata13/
/sys/class/ata_port/ata14 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata14/ata_port/ata14/
/sys/class/ata_port/ata15 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata15/ata_port/ata15/
/sys/class/ata_port/ata16 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata16/ata_port/ata16/
/sys/class/ata_port/ata17 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata17/ata_port/ata17/
/sys/class/ata_port/ata18 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata18/ata_port/ata18/
/sys/class/ata_port/ata19 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata19/ata_port/ata19/
/sys/class/ata_port/ata2 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata2/ata_port/ata2/
/sys/class/ata_port/ata20 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata20/ata_port/ata20/
/sys/class/ata_port/ata21 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata21/ata_port/ata21/
/sys/class/ata_port/ata22 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata22/ata_port/ata22/
/sys/class/ata_port/ata23 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata23/ata_port/ata23/
/sys/class/ata_port/ata24 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata24/ata_port/ata24/
/sys/class/ata_port/ata25 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata25/ata_port/ata25/
/sys/class/ata_port/ata26 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata26/ata_port/ata26/
/sys/class/ata_port/ata27 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata27/ata_port/ata27/
/sys/class/ata_port/ata28 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata28/ata_port/ata28/
/sys/class/ata_port/ata29 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata29/ata_port/ata29/
/sys/class/ata_port/ata3 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata3/ata_port/ata3/
/sys/class/ata_port/ata30 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata30/ata_port/ata30/
/sys/class/ata_port/ata31 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata31/ata_port/ata31/
/sys/class/ata_port/ata32 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata32/ata_port/ata32/
/sys/class/ata_port/ata4 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata4/ata_port/ata4/
/sys/class/ata_port/ata5 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata5/ata_port/ata5/
/sys/class/ata_port/ata6 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata6/ata_port/ata6/
/sys/class/ata_port/ata7 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata7/ata_port/ata7/
/sys/class/ata_port/ata8 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata8/ata_port/ata8/
/sys/class/ata_port/ata9 -> ../../devices/pci0000:00/0000:00:1d.0/0000:05:00.0/ata9/ata_port/ata9/
Joy! Here we see that all SATA-ports are connected to the ASM1166, and that means that we can give Unraid(or any other VM) full control over all 6 SATA slots in the TM. In theory...we will see for real in a couple of days.
I guess the reason we see 32 SATA-ports here, is because the ASM1166 supports Port-multiplying. Exactly how that works I leave to someone else to explain.
Setting up the new Hardware
Opened the Max, and replaced the USB-drive containing TOS with my Unraid USB drive. I also moved my drives from my DIY-rig to the Max.
The Max comes with reasonable BIOS-settings, just press "del"-key at boot to enter BIOS. This is the changes I have done(as far as I can remember...
- Disabled:
- Nic 1
- HD Audio
- Thunderbolt
- Wifi/BT(although no module's present)
- Boot TOS first
- Enabled:
- ASPM generally
- Enabled ASPM on the two NVME-PCIe slots(also disabled AER on those because otherwise my WD Reds floods syslog with "Corrected errors"
- Enabled ASPM on three "chipset"-PCIe-Lanes
- Settings for the Max to start after power is connected
- Set correct boot order
First Boot:
You really got to love Proxmox! I simply moved all disks, modified boot-order in BIOS, and everything works as before without any issues! The only thing that you might have to do is edit /etc/network/interfaces and change i/f. In my case I didn't have to.
When Idling the Max draws 12-13W after issuing
echo "powersave" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powertop --auto-tune
Edit: After a few weeks after writing the above, I have a really strange problem...After a reboot none, or sometimes a few, of my 5 SSDs shows up, and only a a complete Shutdown makes the disks usable again. I have tested with a HDD, and that one always works. I have tried to change everythinig in BIOS, but to no avail.
Next Step: Follow link below.
Part 2: Get Unraid up and running
Last edited:
