SAS3008 (e.g. LSI 9300-8i) incompatible with Proxmox

[brightrgb]

Hallo,
I bought a new HBA with SAS3008. I want to passthroug it to a TrueNAS VM.
When I boot the VM. I get the following error message:

'
Unable to load Avago Technologies MPT BIOS
MTP BIOS Fault 02h encountered at adapter PCI(01h,00h,00h)
Firmware Fault Code: 2667h

Press any key to continue ...
'

If I press any key afer the error, the TrueNAS VM starts and everything looks fine.
But there seems something extremly wrong.
Is the SAS3008 incompatible with Proxmox/Debian?

 

[mira]

Latest firmware installed?

This doesn't seem to be an issue with PVE/Debian if you use PCI Passthrough.

 

[brightrgb]

Latest firmware installed?

I think so.
It is:
Controller: SAS3008
Firmware Product ID: 0x2221 (IT)
Firmware Version: 16.00.10.00

Could it a problem with passthrough, because Proxmox is here (passthrough) in a experimental status?

 

[mira]

That message is from inside the VM?
Is this by any chance your post? https://www.truenas.com/community/threads/is-rombar-1-important-for-truenas-vm.98242/

If it is your post, and it works fine with `rombar=0`, then use that [0]:
[B]rombar=on|off[/B] makes the firmware ROM visible for the guest. Default is on. Some PCI(e) devices need this disabled.

PCI Passthrough is mainly still experimental, because of hardware support. Even now, after more than a decade, it's still not reliable, beginning with IOMMU and the groups.
There's a wide range of issues with all kinds of hardware. Sometimes a firmware upgrade helps, other times different hardware is required to get it working.


[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

 

[brightrgb]

Is this by any chance your post?

Yes, it is.
I came from ESXi and my master server is still running on ESXi, but I'am not really happy with the restrictions in the ESXi Free version.

I tested a lot with proxmox so far and it is very good in my opinion.
But maybe I have the wrong concept.
I have to test, if I can configure the ZFS-Pools directly in Proxmox without the GUI of TrueNAS (if my skills are enough) .
Then I need the VM with TrueNas only to share the Dataset in my network.

 

[LnxBil]

I have to test, if I can configure the ZFS-Pools directly in Proxmox without the GUI of TrueNAS (if my skills are enough) .
Then I need the VM with TrueNas only to share the Dataset in my network.

Pool on PVE has to be shared from the PVE side in order to get the "real" ZFS. You cannot passthrough the Pool to a VM, only a zvol.

Having these Storage VMs is so VMware ... I did dat a decade ago too. It is the VMware way of thinking, glad I could get that all out of my system.

 

[brightrgb]

Pool on PVE has to be shared from the PVE side in order to get the "real" ZFS.

This would be the best way to do that, but the problem is, that PVE don't support SMB for example.
So I will need a VM, Container or whatever to share my ZFS-Storage Pool to my VMs (Windows, Linux, etc.)
Or is there a opportunity to do that in PVE?

Having these Storage VMs is so VMware ... I did dat a decade ago too. It is the VMware way of thinking, glad I could get that all out of my system.

Yes, I think it is not the best way, too.
Maybe they want no native ZFS in their system to hold it extremly small.

 

[bobmc]

Unable to load Avago Technologies MPT BIOS
MTP BIOS Fault 02h encountered at adapter PCI(01h,00h,00h)
Firmware Fault Code: 2667h

I think this just indicates that there was a problem loading the BIOS rom for the card which is only needed if you need to boot from an attached drive which you obviously don't in your case. This error can be ignored, or you could reflash the card in the hope of getting a valid boot rom onto it or possibly just leaving off the boot rom entirely which would resolve the error message and give you a quicker system boot as a bonus (btw was the card imported from a chinese seller on ebay by any chance?)

alternatively, you could pass the drives through to TrueNas
https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)

or you could run your Pool on the PVE host and just install samba to provide SMB shares to network clients. A bit more overhead to manage as you either need to edit a text file to define the shares or install Webmin which has a module for managing Samba fileshares. This also has an advantage that the whole filesystem is accessible and configurable for other VM's and containers - for example, I run Plex as a container and passthrough the media folders as a Plex library.

 

[brightrgb]

(btw was the card imported from a chinese seller on ebay by any chance?)

No, it was bought from a german it-store, but it was a refurbished card.
Do you think I have a chinese fake card?

or you could run your Pool on the PVE host and just install samba to provide SMB shares to network clients. A bit more overhead to manage as you either need to edit a text file to define the shares or install Webmin which has a module for managing Samba fileshares.

This sounds great.
What Container would you recommend for samba?
And is there maybe a tutorial or a help in the wiki, how I can set my samba share with an simple text file?

 

[Neobin]

My 9300-8i runs flawless passed through to a TrueNAS Core VM.
But as @bobmc mentioned, I removed the bios completely. Only thing flashed on the card is the it-firmware.

 

[LnxBil]

or you could run your Pool on the PVE host and just install samba to provide SMB shares to network clients. A bit more overhead to manage as you either need to edit a text file to define the shares or install Webmin which has a module for managing Samba fileshares.

Why don't you use use ZFS for that? There is the sharesmb propery exactly for that.

 

[brightrgb]

Why don't you use use ZFS for that? There is the sharesmb propery exactly for that.

Do you mean this: https://pve.proxmox.com/wiki/Storage:_CIFS

There is an example in it:

cifs: backup # can I set as I want
path /mnt/pve/backup # is the folder on my Dataset?
server 10.0.0.11 # IP of my PVE host?
share VMData # name of the Dataset on my zPool
content backup # not important I think
username anna # name of the user to enter the cifs-share
smbversion 3 # used smb version

I make some comments, where I'am not sure how to configure it.
And where can I set a PW for the user anna or set more diffrent users?

 

[bobmc]

Why don't you use use ZFS for that? There is the sharesmb propery exactly for that.

I don't think this, on it's own, would be much use. I believe the OS still needs to be setup for SMB shares so SAMBA will still be required. Besides, there are a whole bunch of features and use cases supported by SAMBA and configuring them does take a bit more effort.

@brightrgb - are your users (anna etc) on Windows, Mac or Linux systems and is that your goal to have personal shares for each user?

 

[brightrgb]

@brightrgb - are your users (anna etc) on Windows, Mac or Linux systems and is that your goal to have personal shares for each user?

Windows and Linux I need.
I want to give diffrent users (aprox. 8) diffrent rights, that the users only could read or write diffrent folders on the storage pool.
With TrueNAS this is not so a big problem, but with the editor I don't know!

 

[bobmc]

Assume you have proxmox installed (named pve01) and you have a ZFS pool called MyPool

Create a usershare
zfs create MyPool/UserShare
zfs create MyPool/UserShare/Home

Install samba
apt install samba
mv /etc/samba/smb.conf /etc/samba/smb.conf.orig
create a new smb.conf file
nano /etc/samba/smb.conf
to be like this
[homes] comment = Home Directories path = /MyPool/UserShare/Home/%S valid users = %S read only = No create mask = 0700 directory mask = 0700 browseable = No

now create a user and password - for example 'anna'
useradd -s /bin/true anna smbpasswd -a anna

create a user directory for anna
mkdir /MyPool/UserShare/Home/anna chown anna:anna /MyPool/UserShare/Home/anna chmod 775 /MyPool/UserShare/Home/anna

restart samba
service smbd restart

from a client pc map a network share to
\\pve01\anna with username=anna and password as required

 

[LnxBil]

I don't think this, on it's own, would be much use. I believe the OS still needs to be setup for SMB shares so SAMBA will still be required. Besides, there are a whole bunch of features and use cases supported by SAMBA and configuring them does take a bit more effort.

Of course, you need samba to do this and the sharesmb property is just a simpler way to do this. For complex stuff you need a configuration file with a good templating setup so that you have a KISS file.

 

[brightrgb]

Assume you have proxmox installed (named pve01) and you have a ZFS pool called MyPool

Create a usershare
zfs create MyPool/UserShare
zfs create MyPool/UserShare/Home

Thank you very much for your great help to set up a samba share.
I will try this at the weekend, but it looks not so difficult with the information you gave me.

But two thing I don't understand.

1. Why so many people want to run a TrueNAS-VM with a HBA on PVE, when PVE can do this by it's own?
   In my case I have bought the not really cheap HBA in vain, because if I don't need passthrouh I don't need necessarily the HBA. Otherwise more SATA and SAS ports are allways a benefit.
   
   
2. Why is there nearly a bashing on PVE in the TrueNAS forum in case of using passthrough?
   They only see ESXi as significant and usable for a virtualized storage VM. But in my opinion the hypervisor couldn't affect anything, that is direct passtroughed to a VM. Or ist this a mistake of me?

 

[Dunuin]

But two thing I don't understand.

1. Why so many people want to run a TrueNAS-VM with a HBA on PVE, when PVE can do this by it's own?
   In my case I have bought the not really cheap HBA in vain, because if I don't need passthrouh I don't need necessarily the HBA. Otherwise more SATA and SAS ports are allways a benefit.

Because it is very annoying and time consuming to do everything in CLI. And you don't need TrueNAS webUI for just SMB. For example what I use the webUI for:
- monitoring
- UPS
- managing NFS/SMB/iSCSI/FTP/rsync shares
- managing ALCs and rights
- creating users and groups
- creating snapshot/scrub/SMART/sync/replication tasks
- doing replications
- disk management
- ...

And another point is that in general you want your hypervisor to be as default as possible with as less as possible config changes and installed packages so its easier to reinstall it and less that could create problems preventing the guests from running.

 

[bobmc]

And another point is that in general you want your hypervisor to be as default as possible with as less as possible config changes and installed packages so its easier to reinstall it and less that could create problems preventing the guests from running.

there's nothing to stop you running a container or vm just to do samba shares, although personally I have no problem with installing a few useful packages on the host. If you document your setup, you should have no problem with re-installing.


Some people prefer cli to gui (sometimes it can actually be quicker and easier )

 

[tebers]

Assume you have proxmox installed (named pve01) and you have a ZFS pool called MyPool

Create a usershare
zfs create MyPool/UserShare
zfs create MyPool/UserShare/Home

Install samba
apt install samba
mv /etc/samba/smb.conf /etc/samba/smb.conf.orig
create a new smb.conf file
nano /etc/samba/smb.conf
to be like this
[homes] comment = Home Directories path = /MyPool/UserShare/Home/%S valid users = %S read only = No create mask = 0700 directory mask = 0700 browseable = No

now create a user and password - for example 'anna'
useradd -s /bin/true anna smbpasswd -a anna

create a user directory for anna
mkdir /MyPool/UserShare/Home/anna chown anna:anna /MyPool/UserShare/Home/anna chmod 775 /MyPool/UserShare/Home/anna

restart samba
service smbd restart

from a client pc map a network share to
\\pve01\anna with username=anna and password as required

Hi I am temping to do this. but regarding the recent exploit on samba shares I would like to do this in a container ? is there a way to pass the datasets MyPool/UserShare and MyPool/UserShare/Home to the container ? Asking this as i am new to pve and zfs .