Backup on Supermicro, Broadcom SAS3008

Z

Zephrant

Member
Sep 12, 2021
34
4
8
124
I'm trying to get backup running on a Supermicro SSG-6048R-E1CR36L chassis. This has a SAS card:
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

I can't seem to get the disks on that card to be available to Proxmox:

root@test1:~/raid# proxmox-backup-manager disk zpool create zpool1 --devices sda,sdb,sdc,sdd,sde,sdf,sdg,sdh --raidlevel raidz2 --add-datastore
Error: disk 'sda' is already in use.

I've tried to disable the card in BIOS and in the POST setup, but no luck so far.

I'm booting off of a NVMe drive, so all SAS drives are available for a datastore.

Any tips on how to release the drives from the SAS so I can use them?
 
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: Super Micro Computer Inc AOC-S3008L-L8e Physical Slot: 0-2 Flags: bus master, fast devsel, latency 0, IRQ 75, NUMA node 0 I/O ports at 6000 [size=256] Memory at c7240000 (64-bit, non-prefetchable) [size=64K] Memory at c7200000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at c7100000 [disabled] [size=1M] Capabilities: [50] Power Management version 3 Capabilities: [68] Express Endpoint, MSI 00 Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [c0] MSI-X: Enable+ Count=96 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [1e0] Secondary PCI Express Capabilities: [1c0] Power Budgeting <?> Capabilities: [190] Dynamic Power Allocation <?> Capabilities: [148] Alternative Routing-ID Interpretation (ARI) Kernel driver in use: mpt3sas Kernel modules: mpt3sas
 
can you post the output of 'lsblk' ?

Zephrant said:
Error: disk 'sda' is already in use.
Click to expand...
it seems that the disk 'sda' is already used by something (partitions/mounts/lvm/zfs/etc)
 
root@test1:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 3.6T 0 disk sdb 8:16 0 3.6T 0 disk sdc 8:32 0 3.6T 0 disk sdd 8:48 0 3.6T 0 disk sde 8:64 0 3.6T 0 disk sdf 8:80 0 3.6T 0 disk sdg 8:96 0 3.6T 0 disk sdh 8:112 0 3.6T 0 disk sdi 8:128 0 3.6T 0 disk sdj 8:144 0 3.6T 0 disk sdk 8:160 0 3.6T 0 disk sdl 8:176 0 3.6T 0 disk sdm 8:192 0 3.6T 0 disk sdn 8:208 0 3.6T 0 disk sdo 8:224 0 3.6T 0 disk sdp 8:240 0 3.6T 0 disk sr0 11:0 1 9.5M 0 rom sdq 65:0 0 3.6T 0 disk sdr 65:16 0 3.6T 0 disk sds 65:32 0 3.6T 0 disk sdt 65:48 0 3.6T 0 disk sdu 65:64 0 3.6T 0 disk sdv 65:80 0 3.6T 0 disk sdw 65:96 0 3.6T 0 disk sdx 65:112 0 3.6T 0 disk sdy 65:128 0 3.6T 0 disk sdz 65:144 0 3.6T 0 disk sdaa 65:160 0 3.6T 0 disk sdab 65:176 0 3.6T 0 disk sdac 65:192 0 3.6T 0 disk sdad 65:208 0 3.6T 0 disk sdae 65:224 0 3.6T 0 disk sdaf 65:240 0 3.6T 0 disk sdag 66:0 0 3.6T 0 disk sdah 66:16 0 3.6T 0 disk sdai 66:32 0 3.6T 0 disk sdaj 66:48 0 3.6T 0 disk nvme0n1 259:0 0 372.6G 0 disk âânvme0n1p1 259:1 0 1007K 0 part âânvme0n1p2 259:2 0 512M 0 part /boot/efi âânvme0n1p3 259:3 0 372.1G 0 part ââpbs-swap 253:0 0 8G 0 lvm [SWAP] ââpbs-root 253:1 0 348.1G 0 lvm /

I've tried the first 5-6 drives, they all report as in-use.
 
Last edited:
Code: 
root@test1:~# sas3ircu list
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.


         Adapter      Vendor  Device                       SubSys  SubSys
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------    ------  ------
   0     SAS3008       1000h   97h    00h:03h:00h:00h      15d9h   0808h
SAS3IRCU: Utility Completed Successfully.

Code: 
root@test1:~# sas3ircu 0 display |more
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.29.01.00
  Firmware version                        : 12.00.02.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 255
  Concurrent commands supported           : 3072
  Slot                                    : Unknown
  Segment                                 : 0
  Bus                                     : 3
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 0
  SAS Address                             : 5000c50-0-85ed-55ad
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 3815447/7814037167
  Manufacturer                            : SEAGATE
  Model Number                            : ST4000NM0125  
  Firmware Revision                       : E002
  Serial No                               : ZC107HEL
  Unit Serial No(VPD)                     : ZC107HEL0000R707JFTJ
  GUID                                    : 5000c50085ed55af
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 1
  SAS Address                             : 5000c50-0-85e6-37e5
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 3815447/7814037167
  Manufacturer                            : SEAGATE
  Model Number                            : ST4000NM0125
  Firmware Revision                       : E002
  Serial No                               : ZC107G22
  Unit Serial No(VPD)                     : ZC107G220000C7077ZM5
  GUID                                    : 5000c50085e637e7
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

(and so on for the rest of the drives)
 
mhm.. what does the gui on the disk tab show for those disks (the usage?)

we probably do misdetect some usage that isn't there... in any case, you can ofc use the zfs binaries themselves to create a pools (zfs,zpool) and setup a datastore after that
 
Screen Shot 2021-10-20 at 8.36.06 AM.png

The Initialize Disk with GPT is greyed out. I'll see if I can get the binaries to talk to the disks.
 

Attachments

  • Screen Shot 2021-10-20 at 8.35.27 AM.png
    Screen Shot 2021-10-20 at 8.35.27 AM.png
    49.3 KB · Views: 3
This completed normally:
Code: 
zpool create zpool1 raidz2 sda sdb sdc sdd sde sdf sdg sdh sdi
root@test1:~# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0   3.6T  0 disk
├─sda1         8:1    0   3.6T  0 part
└─sda9         8:9    0     8M  0 part
sdb            8:16   0   3.6T  0 disk
├─sdb1         8:17   0   3.6T  0 part
└─sdb9         8:25   0     8M  0 part
sdc            8:32   0   3.6T  0 disk
├─sdc1         8:33   0   3.6T  0 part
└─sdc9         8:41   0     8M  0 part
sdd            8:48   0   3.6T  0 disk
├─sdd1         8:49   0   3.6T  0 part
└─sdd9         8:57   0     8M  0 part
sde            8:64   0   3.6T  0 disk
├─sde1         8:65   0   3.6T  0 part
└─sde9         8:73   0     8M  0 part
sdf            8:80   0   3.6T  0 disk
├─sdf1         8:81   0   3.6T  0 part
└─sdf9         8:89   0     8M  0 part
sdg            8:96   0   3.6T  0 disk
├─sdg1         8:97   0   3.6T  0 part
└─sdg9         8:105  0     8M  0 part
sdh            8:112  0   3.6T  0 disk
├─sdh1         8:113  0   3.6T  0 part
└─sdh9         8:121  0     8M  0 part
sdi            8:128  0   3.6T  0 disk
├─sdi1         8:129  0   3.6T  0 part
└─sdi9         8:137  0     8M  0 part
 
For anyone following along from home, this worked:
Code: 
zpool create -m /mnt/datastore/zpool1 zpool1 raidz2 sda sdb sdc sdd sde sdf sdg sdh sdi
zpool create -m /mnt/datastore/zpool2 zpool2 raidz2 sdj sdk sdl sdm sdn sdo sdp sdq sdr
zpool create -m /mnt/datastore/zpool3 zpool3 raidz2 sds sdt sdu sdv sdw sdx sdy sdz sdaa
zpool create -m /mnt/datastore/zpool4 zpool4 raidz2 sdab sdac sdad sdae sdaf sdag sdah sdai sdaj

proxmox-backup-manager datastore create store1 /mnt/datastore/zpool1
proxmox-backup-manager datastore create store2 /mnt/datastore/zpool2
proxmox-backup-manager datastore create store3 /mnt/datastore/zpool3
proxmox-backup-manager datastore create store4 /mnt/datastore/zpool4

I want multiple pools for resiliency, but am I going to have to treat these as separate backup pools now?
Should I have put all the drives in one big raidz2 pool?
 
Last edited:
For even more fun, one big pool of all the drives:
Code: 
zpool create -m /mnt/datastore/storage storage raidz2 sda sdb sdc sdd sde sdf sdg sdh sdi raidz2 sdj sdk sdl sdm sdn sdo sdp sdq sdr raidz2 sds sdt sdu sdv sdw sdx sdy sdz sdaa raidz2 sdab sdac sdad sdae sdaf sdag sdah sdai sdaj
zpool set autoreplace=on storage
zpool set autoexpand=on storage
proxmox-backup-manager datastore create store1 /mnt/datastore/storage
 
I believe this controller is not fully compatible with Proxymox/Debian.

In particular, I have problems with the SMART query because, as far as I know, the controller does not really allow a full transparent HBA mode.

Have you install the broadcom tools?
 
Yep, sas3ircu works just fine. I've got backups running, have 1188 backups on it already, and just restored from it this morning.
AFAIK, everything is working just fine.

As part of my testing, I spared out a drive, watched the raid degrade, wiped the drive with dd, then added it back in. It resilvered and came back up clean.

I am still also backing up my VMs to a NFS store, and will do so for a month or longer, as I'm super paranoid. My machine I re-purposed with this SAS card was not new, nor were the drives, so not ideal as a backup solution.

Driver:
https://www.ibm.com/support/pages/s...orage-management-v17000000-linux-ibm-system-x


Code: 
#
# sas3ircu list
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02) 
Copyright (c) 2009-2018 Avago Technologies. All rights reserved. 


         Adapter      Vendor  Device                       SubSys  SubSys 
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID 
 -----  ------------  ------  ------  -----------------    ------  ------ 
   0     SAS3008       1000h   97h    00h:03h:00h:00h      15d9h   0808h 
SAS3IRCU: Utility Completed Successfully.


# sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.29.01.00
  Firmware version                        : 12.00.02.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 255
  Concurrent commands supported           : 3072
  Slot                                    : Unknown
  Segment                                 : 0
  Bus                                     : 3
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

(snip)
 
Since I have two clusters to backup, and I can't backup them both to the same backup server, I setup a small pool on the boot drive for my small cluster, and am using it until Proxmox Backup supports multiple clusters natively.

The errors in this are from my testing and all. My OCD wants to figure out how to remove the red X and yellow ! though...

So far I'm impressed with the dedup- These backups are taking about 1/2 the space as the same backups on my NetApp NFS, which has full on-the-fly dedup/compression enabled.

So well done Proxmox Backup team, thank you.

1635959695170.png
 
Zephrant said:
Yep, sas3ircu works just fine. I've got backups running, have 1188 backups on it already, and just restored from it this morning.
AFAIK, everything is working just fine.

As part of my testing, I spared out a drive, watched the raid degrade, wiped the drive with dd, then added it back in. It resilvered and came back up clean.

I am still also backing up my VMs to a NFS store, and will do so for a month or longer, as I'm super paranoid. My machine I re-purposed with this SAS card was not new, nor were the drives, so not ideal as a backup solution.

Driver:
https://www.ibm.com/support/pages/s...orage-management-v17000000-linux-ibm-system-x


Code: 
#
# sas3ircu list
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.


         Adapter      Vendor  Device                       SubSys  SubSys
 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------    ------  ------
   0     SAS3008       1000h   97h    00h:03h:00h:00h      15d9h   0808h
SAS3IRCU: Utility Completed Successfully.


# sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.29.01.00
  Firmware version                        : 12.00.02.00
Click to expand...
BIOS and Firmware version is very old. may be a new version is needed for newer hard drives?
 
A good thought- The drives in my system are as old as the firmware.
Have you found instructions on updating the firmware on this card?

I just checked again, and I'm running the newest BIOS available for my system from Supermicro, 3.4a, but I've not tried to update the card's firmware.
 
Last edited:
Zephrant said:
For even more fun, one big pool of all the drives:
Code: 
zpool create -m /mnt/datastore/storage storage raidz2 sda sdb sdc sdd sde sdf sdg sdh sdi raidz2 sdj sdk sdl sdm sdn sdo sdp sdq sdr raidz2 sds sdt sdu sdv sdw sdx sdy sdz sdaa raidz2 sdab sdac sdad sdae sdaf sdag sdah sdai sdaj
zpool set autoreplace=on storage
zpool set autoexpand=on storage
proxmox-backup-manager datastore create store1 /mnt/datastore/storage
Click to expand...
I have always been told by Solaris and BSD administrators to use /dev/disk/by-id/disk-nameX instead of /dev/sdX

:)
 
I've read that as well, and sounds like good advice, but I couldn't get it to work. I tried to remove/replace a drive by ID, and it wouldn't take it, but maybe that's because I didn't create the pool that way.

Managing 36 disk-name IDs on one command line when they are each 22+ chars would be a mess.

Also wouldn't be useful to others here, as all the IDs are unique.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom