[SOLVED] Proxmox - TrueNAs, Disks were not passing to the TrueNas from Proxmox.
- Thread starter eshrath
- Start date
The reason i am asking, since there are so many cards and i do not want to waste any funds, getting the wrong card. i see a 10GTEk card on Amazon (https://www.amazon.com/10Gtek-Inter...-lpcontext&ref_=fplfs&smid=AE2OZG2NN3099&th=1) and it mentions LSI SAS, but no mention of IT mode.
This: [1] is a good video that gives a nice overview as well as explanations what card/generation to use (at least) for which scenario.
I do not know, if you can flash those third-party cards with the original LSI/Broadcom firmware. So you have to ask the manufacturer with which firmware he delivers the cards and if it is not already in IT-mode, if they provide the firmware to flash them into it yourself.
Or simply go with an original/full LSI/Broadcom one.
And be aware to not get a counterfeit one; especially if you buy from e.g. Ebay... (And no, I do not know how to identify them for sure. But you might find one or the other video regarding this also on the YouTube channel I linked. (And no², that is not me!))
[1] https://youtu.be/hTbKzQZk21w
Interesting video. Got 3x SAS2008 because they are well supported, reliable and cheap (I think I payed between 30-40€ per card incl. shipping) but the 10W per card was a bit annoying for a homelab. Thought about replacing a SAS2008 with a SAS2308 for my 8 SATA SSDs, but a bit shocked that they even consume more power. Hoped the newer models would be a bit more power efficient. But they are usually even second hand 100+€.
Porxmox to Truenas Scale
What I have
Motherboard - MSI MAG X570S Tomahawk WiFi Motherboard - AMD AM4, DDR4, PCIe 4.0
CPU AMD Ryzen 7 5700G 8-Core, 16-Thread
32 GB - Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 3200MHz – yes non ECC ram
2 - Crucial MX500 500GB (boot for proxmox)
2 - SAMSUNG 980 SSD 1TB PCle 4.0 NVMe M.2 Internal Solid State Hard Drive
5 - Seagate IronWolf Pro 4TB NAS Internal Hard Drive HDD – CMR 3.5 Inch SATA
Dell 27NFF LSI 9207-8i HBA SAS controller card
Under Proxmox all disks show up
What I did so far
Installed proxmox – No problems I can see.
When I install Proxmox I can see all my drives (see screen shot above)– sdf and sdg are boot drive for proxmox.
What I want to do is to passthrough to truenas scale the hard drive sda, sdb, sdc, sdd, sde and use the nvme0n1 and 1n1 as cache drive. Open to suggestion also
Some of the question I have even before I installed true NAS are:
Create Virtual Machine – Under System -SCSI controller – do I just use the default Virtlo SCSI Single, or the LSI cards listed as default. I added the Qemu Agent
General
Create Virtual Machine – Under System -SCSI controller – do I just use the default Virtlo SCSI Single, or the LSI cards listed as default. I added the Qemu Agent. When I remove LSI 53c895A and go back to Virtlo SCSI single – I get this but the booting from hard drive hangs
Add 4 cores
Added 24 Gig of ram – turned off Ballooning device
And confirm all above. VM 102 created (true)
Clicked on pve01 – 102 (True) – Hardware – Add PCI Device 10:00
root@pve01:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166a
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166b
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166c
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166d
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166e
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166f
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1670
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1671
10:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
21:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
23:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
26:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
28:00.0 Network controller: Intel Corporation Device 2725 (rev 1a)
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
2a:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2d:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
30:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c8)
30:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637
30:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
30:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
30:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
30:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
31:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
What I have
Motherboard - MSI MAG X570S Tomahawk WiFi Motherboard - AMD AM4, DDR4, PCIe 4.0
CPU AMD Ryzen 7 5700G 8-Core, 16-Thread
32 GB - Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 3200MHz – yes non ECC ram
2 - Crucial MX500 500GB (boot for proxmox)
2 - SAMSUNG 980 SSD 1TB PCle 4.0 NVMe M.2 Internal Solid State Hard Drive
5 - Seagate IronWolf Pro 4TB NAS Internal Hard Drive HDD – CMR 3.5 Inch SATA
Dell 27NFF LSI 9207-8i HBA SAS controller card
Under Proxmox all disks show up
What I did so far
Installed proxmox – No problems I can see.
When I install Proxmox I can see all my drives (see screen shot above)– sdf and sdg are boot drive for proxmox.
What I want to do is to passthrough to truenas scale the hard drive sda, sdb, sdc, sdd, sde and use the nvme0n1 and 1n1 as cache drive. Open to suggestion also
Some of the question I have even before I installed true NAS are:
Create Virtual Machine – Under System -SCSI controller – do I just use the default Virtlo SCSI Single, or the LSI cards listed as default. I added the Qemu Agent
General
Create Virtual Machine – Under System -SCSI controller – do I just use the default Virtlo SCSI Single, or the LSI cards listed as default. I added the Qemu Agent. When I remove LSI 53c895A and go back to Virtlo SCSI single – I get this but the booting from hard drive hangs
Add 4 cores
Added 24 Gig of ram – turned off Ballooning device
And confirm all above. VM 102 created (true)
Clicked on pve01 – 102 (True) – Hardware – Add PCI Device 10:00
root@pve01:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166a
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166b
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166c
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166d
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166e
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 166f
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1670
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1671
10:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
21:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
23:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
26:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
28:00.0 Network controller: Intel Corporation Device 2725 (rev 1a)
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
2a:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2d:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
30:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c8)
30:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637
30:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
30:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
30:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
30:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
31:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
I block out the backend of the IP address
I go back to 102 to shut down to add PCI Card
Restart 102 – get this again let it go for a bit
After a while I stop 102 remove the PCI card. Restart 102 and then I see TrueNas Scale
After I install TrueNas Scale (22.12.0) I do not see the hard drives sda, sdb, sdc, sdd, sde or the nvme drives on the true nas scale software.
I tried addition suggestion in the Proxmox forum - I updated the grub file and added amd_iomu=on
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quite amd_iomu=on"
GRUB_CMDLINE_LINUX=""
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
dmesg | grep -e DMAP -e IOMMU
[ 0.353743] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.359173] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.532387] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 13.772818] AMD-Vi: AMD IOMMUv2 loaded and initialized
root@pve01:~# ls /dev/disk/by-id/
ata-CT500MX500SSD1_2222E6373B7C nvme-Samsung_SSD_980_PRO_1TB_S5P2NL0T900967W
ata-CT500MX500SSD1_2222E6373B7C-part1 nvme-Samsung_SSD_980_PRO_1TB_S5P2NL0T904639Y
ata-CT500MX500SSD1_2222E6373B7C-part2 wwn-0x5000c500e54ef459
ata-CT500MX500SSD1_2222E6373B7C-part3 wwn-0x5000c500e54ef7a0
ata-CT500MX500SSD1_2223E63ACDB6 wwn-0x5000c500e54ef9a6
ata-CT500MX500SSD1_2223E63ACDB6-part1 wwn-0x5000c500e54efb2c
ata-CT500MX500SSD1_2223E63ACDB6-part2 wwn-0x5000c500e54efe59
ata-CT500MX500SSD1_2223E63ACDB6-part3 wwn-0x500a0751e6373b7c
ata-ST4000NE001-2MA101_WS24BSJH wwn-0x500a0751e6373b7c-part1
ata-ST4000NE001-2MA101_WS24BSR5 wwn-0x500a0751e6373b7c-part2
ata-ST4000NE001-2MA101_WS24BTFZ wwn-0x500a0751e6373b7c-part3
ata-ST4000NE001-2MA101_WS24BTW4 wwn-0x500a0751e63acdb6
ata-ST4000NE001-2MA101_WS24BTZV wwn-0x500a0751e63acdb6-part1
nvme-eui.002538b921a0042e wwn-0x500a0751e63acdb6-part2
nvme-eui.002538b921a01286 wwn-0x500a0751e63acdb6-part3
Please let me know if you need additional information that might help.
Thank you for your time
Chris
I go back to 102 to shut down to add PCI Card
Restart 102 – get this again let it go for a bit
After a while I stop 102 remove the PCI card. Restart 102 and then I see TrueNas Scale
After I install TrueNas Scale (22.12.0) I do not see the hard drives sda, sdb, sdc, sdd, sde or the nvme drives on the true nas scale software.
I tried addition suggestion in the Proxmox forum - I updated the grub file and added amd_iomu=on
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quite amd_iomu=on"
GRUB_CMDLINE_LINUX=""
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
dmesg | grep -e DMAP -e IOMMU
[ 0.353743] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.359173] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.532387] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 13.772818] AMD-Vi: AMD IOMMUv2 loaded and initialized
root@pve01:~# ls /dev/disk/by-id/
ata-CT500MX500SSD1_2222E6373B7C nvme-Samsung_SSD_980_PRO_1TB_S5P2NL0T900967W
ata-CT500MX500SSD1_2222E6373B7C-part1 nvme-Samsung_SSD_980_PRO_1TB_S5P2NL0T904639Y
ata-CT500MX500SSD1_2222E6373B7C-part2 wwn-0x5000c500e54ef459
ata-CT500MX500SSD1_2222E6373B7C-part3 wwn-0x5000c500e54ef7a0
ata-CT500MX500SSD1_2223E63ACDB6 wwn-0x5000c500e54ef9a6
ata-CT500MX500SSD1_2223E63ACDB6-part1 wwn-0x5000c500e54efb2c
ata-CT500MX500SSD1_2223E63ACDB6-part2 wwn-0x5000c500e54efe59
ata-CT500MX500SSD1_2223E63ACDB6-part3 wwn-0x500a0751e6373b7c
ata-ST4000NE001-2MA101_WS24BSJH wwn-0x500a0751e6373b7c-part1
ata-ST4000NE001-2MA101_WS24BSR5 wwn-0x500a0751e6373b7c-part2
ata-ST4000NE001-2MA101_WS24BTFZ wwn-0x500a0751e6373b7c-part3
ata-ST4000NE001-2MA101_WS24BTW4 wwn-0x500a0751e63acdb6
ata-ST4000NE001-2MA101_WS24BTZV wwn-0x500a0751e63acdb6-part1
nvme-eui.002538b921a0042e wwn-0x500a0751e63acdb6-part2
nvme-eui.002538b921a01286 wwn-0x500a0751e63acdb6-part3
Please let me know if you need additional information that might help.
Thank you for your time
Chris