No SAS2008 after upgrade

proxminent said:
hmmm ... same for me... but not booting. module loaded but sas2008 not detected .... funny ...
Click to expand...
Mine boots in both cases with slight change to the fstab in 6.2 as I have to comment out mounting the disks connected to the sas controllers. I'll try to boot live distro with kernel 6.2 or newer on that server and see if I'll the same problem as with pve kernel.
 
OK so I've checked live distros.

Endeavour OS with 6.3 arch kernel sees all the disks connected to the sas2008 cards and Ubuntu with 6.2 doesn't.
 
Running multiple systems with sas2008 cards too, no issues with 6.2.x opt-in kernels since becoming available.
Another system upgraded to pve8 with 6.2 kernel running fine too.

You could investigate your boot logs and compare the working and not working kernel messages.
So far you haven't posted much detail from (boot)logs, that could be helpful, something like spirit posted.
And listing from lspci for all your cards and what kernel driver they can and are using: lspci -nnk

Something like this will output:
Code: 
03:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
    Subsystem: Dell 6Gbps SAS HBA Adapter [1028:1f1c]
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas
 
Here is lspci -nnk from 7.4 current.
Code: 
05:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
    DeviceName: Integrated SAS                         
    Subsystem: Dell PERC H200 Integrated [1028:1f1e]
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas
    
22:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] [1000:0064] (rev 02)
    Subsystem: Broadcom / LSI 9201-16e 6Gb/s SAS/SATA PCIe x8 External HBA [1000:30d0]
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

Still afraid to hit the upgrade.
 
For what it's worth, I have an LSI Broadcom SAS 9300-8i 8-port 12Gb/s SATA+SAS PCI-Express 3.0, which uses the SAS 3008 IO controller. It worked fine under pve 7.4

I did the upgrade to pve 8, kernel 6.2.x no luck. I had to blacklist the mpt3sas driver. I just moved my VMs to another drive for now, but the disk array on this controller isn't usable. If I can't figure out a fix, I'm looking for alternative controllers that are actually supported.

post-upgrade -
lspci -nnk
Code: 
02:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)        Subsystem: Broadcom / LSI SAS9300-8i [1000:30e0]
        Kernel modules: mpt3sas
 
The last working kernel for my SAS2008 was 6.2.6-1-pve from PVE 7.4
Then 6.2.9, 6.2.11 are both broken.

So it's probably something between 6.2.6 and 6.2.9

Code: 
25:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: Broadcom / LSI 9211-8i [1000:3020]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
 
As there are multiple reports of non working cards (but not all) I think it's wise to file a bug report at https://bugzilla.proxmox.com/
There it can be tracked. The forum is not the best place for this.

Some quick searching on the kernel.org and ubuntu bugtrackers did not show relevant results, as far as I can see. So maybe this bug hasn't been reported yet.
@t.lamprecht: is there an opt-in 6.3 pve kernel in the works? one user reported in this thread that 6.3 arch kernel is working.
 
Last edited:
janssensm said:
@t.lamprecht: is there an opt-in 6.3 pve kernel in the works? one user reported in this thread that 6.3 arch kernel is working.
Click to expand...
Not at the moment, we'll only go for a newer (opt-in) major kernel release in Q3, to avoid to many jumps and work.

If there's a clear fix missing (or to be reverted), we can naturally backport that to our 6.2 kernel.
 
  • Like
Reactions: janssensm
justea said:
The last working kernel for my SAS2008 was 6.2.6-1-pve from PVE 7.4
Then 6.2.9, 6.2.11 are both broken.

So it's probably something between 6.2.6 and 6.2.9

Code: 
25:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: Broadcom / LSI 9211-8i [1000:3020]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
Click to expand...
My
Code: 
Linux 6.2.11-2-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.11-2 (2023-05-10T09:13Z)
pve-manager/7.4-16/0f39f621
is working fine. I was having random crashes with the opt-in kernel until I added "mitigations=off".
https://forum.proxmox.com/threads/opt-in-kernel-panics.122589/post-570225
Maybe some mitigation is causing the problem?
Thanks

FYI my processor is "32 x AMD Opteron(tm) Processor 6366 HE (2 Sockets)" on a dell r715 server.
 
Last edited:
donhwyo said:
My
Code: 
Linux 6.2.11-2-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.11-2 (2023-05-10T09:13Z)
pve-manager/7.4-16/0f39f621
is working fine. I was having random crashes with the opt-in kernel until I added "mitigations=off".
https://forum.proxmox.com/threads/opt-in-kernel-panics.122589/post-570225
Maybe some mitigation is causing the problem?
Thanks

FYI my processor is "32 x AMD Opteron(tm) Processor 6366 HE (2 Sockets)" on a dell r715 server.
Click to expand...

It was always mitigations=off
Code: 
BOOT_IMAGE=/boot/vmlinuz-6.2.6-1-pve root=/dev/mapper/pve-root ro quiet mitigations=off iommu=pt intel_iommu=on apparmor=0
AMD X3300 on a B450 board

6.2.6 works fine and stable (all vm are lxc, no kvm).
6.2.9 and 6.2.11 fail to boot with volume group pve errors.
pve can't boot, no ssh, I had to attach keyboard and monitor to find out what's going on.
The data drives are on the SAS2008, the boot drive is on the mainboard's SATA port, PVE should boot itself and then fail to start vm but it didn't.

It's now running PVE 8.0.3 with kernel pinned to 6.2.6 and seems stable.

Code: 
pve-manager/8.0.3/bbf3993334bfa916 (running kernel: 6.2.6-1-pve)
 
There is a newer opt-in kernel available.
Linux 6.2.16-4-bpo11-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-4~bpo11+1 (2023-07-07T15:05Z
Don't know if it can be installed on pve 8.Probably not but 8 version can't be far off.
 
  • Like
Reactions: oha and luckyluk83
Hi,

Any news on that ? I need to convert a windows server to proxmox, where the datastore is connected via a AVAGO LSI MegaRaid 9361-i8.
Should i install PVE 7 instead till this issue is resovled ?

Thx in advance
Martin
 
I'm running on PVE 8 with 6.1.10-1-pve kernel for a week now with no issues. I've tested few live distros with different kernels and I can say that all was fine on those with 6.1 or 6.3. The 6.2 kernel didn't see any disks connected to lsi 9211-8i.
 
Retro1982 said:
Hi,

Any news on that ? I need to convert a windows server to proxmox, where the datastore is connected via a AVAGO LSI MegaRaid 9361-i8.
Should i install PVE 7 instead till this issue is resovled ?

Thx in advance
Martin
Click to expand...
Sounds like you have a clean system. Try with version 8 and see if you can see the disks on the 9361. If so you are good to go. If not and you have time, see if you can help determine why not. Or if no time just install version 7.
Let us know either way.
Thanks
 
donhwyo said:
Sounds like you have a clean system. Try with version 8 and see if you can see the disks on the 9361. If so you are good to go. If not and you have time, see if you can help determine why not. Or if no time just install version 7.
Let us know either way.
Thanks
Click to expand...
I will, but the server is still in production (not all services are migrated to linux, but working on that). As soon as i can try i will and report back!

Thanks!
 
  • Like
Reactions: donhwyo
donhwyo said:
There is a newer opt-in kernel available.
Linux 6.2.16-4-bpo11-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-4~bpo11+1 (2023-07-07T15:05Z
Don't know if it can be installed on pve 8.Probably not but 8 version can't be far off.
Click to expand...
6.2.16-4 is now available for pve 8. Does it help?
 
sas2008 under pve 8 with kernel kernel-6.1 working ...

#>apt-get install pve-kernel-6.1
#>proxmox-boot-tool kernel pin 6.1.10-1-pve

#>proxmox-boot-tool kernel list

Automatically selected kernels:
6.1.10-1-pve # sas2008 not recognized
6.2.16-3-pve # sas2008 not recognized
6.2.16-4-pve # sas2008 not recognized

Pinned kernel:
6.1.10-1-pve # # sas2008 working !!!


#>uname -a
Linux haXX 6.1.10-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.1.10-1 (2023-02-07T13:10Z) x86_64 GNU/Linux
 
There is something wrong with;
Linux pve 6.2.16-4-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-5 (2023-07-14T17:53Z) x86_64 GNU/Linux
See below lspci -nnk;
Code: 
06:00.0 RAID bus controller [0104]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: Fujitsu Technology Solutions HBA Ctrl SAS 6G 0/1 [D2607] [1734:1177]
        Kernel modules: mpt3sas

and with older kernel;
Linux pve 6.1.10-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.1.10-1 (2023-02-07T13:10Z) x86_64 GNU/Linux
See below lspci -nnk;
Code: 
06:00.0 RAID bus controller [0104]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: Fujitsu Technology Solutions HBA Ctrl SAS 6G 0/1 [D2607] [1734:1177]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

I am getting below dmesg output on the 6.2.16 kernel;
Code: 
root@pve:~# dmesg | grep mpt
[    0.019013]   Device   empty
[    0.437121] Dynamic Preempt: voluntary
[    0.437200] rcu: Preemptible hierarchical RCU implementation.
[    3.320292] mpt3sas version 43.100.00.00 loaded
[    3.320618] mpt3sas 0000:06:00.0: BAR 1: can't reserve [mem 0x804c0000-0x804c3fff 64bit]
[    3.320630] mpt2sas_cm0: pci_request_selected_regions: failed
[    3.320683] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()!
 
Last edited:
Clean install Proxmox 8.0-2 and apt-get update/upgrade.

When I run older kernel everything is fine;
Linux pve 6.1.10-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.1.10-1 (2023-02-07T13:10Z) x86_64 GNU/Linux
Code: 
root@pve:~# dmesg | grep mpt
[    0.022100]   Device   empty
[    0.438838] Dynamic Preempt: voluntary
[    0.438938] rcu: Preemptible hierarchical RCU implementation.
[    2.781954] mpt3sas version 43.100.00.00 loaded
[    2.782415] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (65811472 kB)
[    2.836594] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    2.836609] mpt2sas_cm0: MSI-X vectors supported: 1
[    2.836616] mpt2sas_cm0:  0 1 1
[    2.836698] mpt2sas_cm0: High IOPs queues : disabled
[    2.836703] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 47
[    2.836707] mpt2sas_cm0: iomem(0x00000000904c0000), mapped(0x(____ptrval____)), size(16384)
[    2.836713] mpt2sas_cm0: ioport(0x000000000000d000), size(256)
[    2.891062] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    2.918599] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[    2.918705] mpt2sas_cm0: request pool(0x(____ptrval____)) - dma(0x118700000): depth(3492), frame_size(128), pool_size(436 kB)
[    2.934573] mpt2sas_cm0: sense pool(0x(____ptrval____)) - dma(0x118d80000): depth(3367), element_size(96), pool_size (315 kB)
[    2.934583] mpt2sas_cm0: sense pool(0x(____ptrval____))- dma(0x118d80000): depth(3367),element_size(96), pool_size(0 kB)
[    2.934686] mpt2sas_cm0: reply pool(0x(____ptrval____)) - dma(0x118e00000): depth(3556), frame_size(128), pool_size(444 kB)
[    2.934697] mpt2sas_cm0: config page(0x(____ptrval____)) - dma(0x118d34000): size(512)
[    2.934701] mpt2sas_cm0: Allocated physical memory: size(7579 kB)
[    2.934705] mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432)
[    2.934709] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[    2.979056] mpt2sas_cm0: overriding NVDATA EEDPTagMode setting
[    2.979759] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03), BiosVersion(07.39.02.00)
[    2.979769] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    2.981050] mpt2sas_cm0: sending port enable !!
[    4.581465] mpt2sas_cm0: hba_port entry: (____ptrval____), port: 255 is added to hba_port list
[    4.583494] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x5003005700dbf104), phys(8)
[    4.584173] mpt2sas_cm0: handle(0x9) sas_address(0x5000c50095dd0c2d) port_type(0x1)
[    4.584669] mpt2sas_cm0: handle(0xa) sas_address(0x4433221107000000) port_type(0x1)
[    4.832717] mpt2sas_cm0: handle(0xb) sas_address(0x4433221103000000) port_type(0x1)
[    4.833722] mpt2sas_cm0: handle(0xc) sas_address(0x5000c50095d5a9c9) port_type(0x1)
[    4.834700] mpt2sas_cm0: handle(0xd) sas_address(0x5000c5008e1b1cf9) port_type(0x1)
[    4.835674] mpt2sas_cm0: handle(0xe) sas_address(0x5000c50095daff09) port_type(0x1)
[    4.836630] mpt2sas_cm0: handle(0xf) sas_address(0x5000c50095dd1c41) port_type(0x1)
[    4.837592] mpt2sas_cm0: handle(0x10) sas_address(0x5000c50095dcfc85) port_type(0x1)
[    9.709609] mpt2sas_cm0: port enable: SUCCESS

Any ideas?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom