[SOLVED] Proxmox installer not seeing my sas drives

[Dieter Toews]

Hi everybody,

I'm a proxmox newbie.

I'm trying to install Proxmox onto a x8 supermicro system with a dell perc H310 flashed to P.20 lsi 9211-8i firmware. i have an identical card running my freenas system. i belive it should be fine : https://forum.proxmox.com/threads/recommended-hba-what-plays-nice-with-zfs.32517/#post-162216 For both cards In both cases i used the flashing guide at https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/
https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/
When i launch either the 4.3 or the 4.4 installer it says no hard drives detected. A debian 8.6 live .iso sees the drives just fine...

my intention is to create a zfs raid 10 setup and use zfs replication to backup to the freenas box. Is one array for boot and vm storage ok? Why doesn't the installer see the drives behind my HBA?

Thanks, Dieter

 

[fabian]

Hi everybody,

I'm a proxmox newbie.

I'm trying to install Proxmox onto a x8 supermicro system with a dell perc H310 flashed to P.20 lsi 9211-8i firmware. i have an identical card running my freenas system. i belive it should be fine : https://forum.proxmox.com/threads/recommended-hba-what-plays-nice-with-zfs.32517/#post-162216 For both cards In both cases i used the flashing guide at https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/
When i launch either the 4.3 or the 4.4 installer it says no hard drives detected. A debian 8.6 live .iso sees the drives just fine...

my intention is to create a zfs raid 10 setup and use zfs replication to backup to the freenas box. Is one array for boot and vm storage ok? Why doesn't the installer see the drives behind my HBA?

Thanks, Dieter

could you start in debug mode, continue up to the (first) licence screen and hit abort, and then collect the output of "lsblk" ? then do the same, but with a debian live cd / installer and post that as well. thanks!

 

[Dieter Toews]

Thanks for the suggestions Fabian! Are you the same Fabian that is on the opnsense forums?

I'll run the commands (this evening) and get back to you.

I did an experiment last night with freebsd and it didn't see the drives either. As for my Debian experiment I didn't boot to the Debian installer - just the live cd environment - there it enumerated the drives as sda to sdd (i've got 4). I'll also double check the cards bios settings and the MB bios.

Does the installer load the msplsi driver automatically? i found a document which talks about using lsi/avago/brodcom cards to boot in freebsd apparently as of freebsd 8.2 you needed to do some acrobatics to get it all to work.

 

[Dieter Toews]

Hi Fabian,

unfortunately - the debug installer gets as far as the "hit cntl-d to go to the installer" but freezes there (no interactive prompt). I have some screen shots from ipmi but the forum doesn't seem to want to let me post them. Debian 8.6 doesn't seem to have a debug mode installer (or is that just the live cd version?)

I also have a screen shot showing installer boot screen seeing the hba (but again can't post it)...

 

[fabian]

Hi Fabian,

unfortunately - the debug installer gets as far as the "hit cntl-d to go to the installer" but freezes there (no interactive prompt). I have some screen shots from ipmi but the forum doesn't seem to want to let me post them. Debian 8.6 doesn't seem to have a debug mode installer (or is that just the live cd version?)

I also have a screen shot showing installer boot screen seeing the hba (but again can't post it)...

I am not sure about the screenshots - did you try uploading them to the forum or link to them somewhere else? Uploading should work..

The PVE debug installer not working over some IKVM software is a known issue unfortunately..

Anyhow, could you post the following from a linux system that sees the disk(s) - replace DEV with the block device, like "sda":

lsblk
lsmod
readlink /sys/block/DEV/device/driver
readlink /sys/block/DEV/device/driver/module

if the last one gives you a module name (the part after the last "/"), also include the output of "modinfo MODULENAME"

 

[Dieter Toews]

Hi Fabian,

here is the output:

user@debian:~$ lsblk
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0    11:0    1 1024M  0 rom 
sr1    11:1    1  1.3G  0 rom  /lib/live/mount/medium
loop0   7:0    0  1.2G  1 loop /lib/live/mount/rootfs/
filesystem.squashfs
user@debian:~$ lsmod
Module                  Size  Used by
cfg80211              413730  0
rfkill                 18867  2 cfg80211
nfsd                  262938  2
auth_rpcgss            51209  1 nfsd
oid_registry           12419  1 auth_rpcgss
nfs_acl                12511  1 nfsd
nfs                   192232  0
lockd                  83389  2 nfs,nfsd
fscache                45542  1 nfs
sunrpc                237364  6 nfs,nfsd,auth_rpcgss,lockd,nfs_acl
joydev                 17063  0
intel_powerclamp       17159  0
coretemp               12820  0
kvm_intel             139116  0
kvm                   392862  1 kvm_intel
iTCO_wdt               12831  0
iTCO_vendor_support    12649  1 iTCO_wdt
pcspkr                 12595  0
serio_raw              12849  0
tpm_tis                17231  0
evdev                  17445  6
tpm                    31511  1 tpm_tis
ipmi_si                48709  0
acpi_cpufreq           17218  1
i7core_edac            22278  0
ipmi_msghandler        39917  1 ipmi_si
lpc_ich                20768  0
processor              28221  1 acpi_cpufreq
edac_core              47321  2 i7core_edac
mfd_core               12601  1 lpc_ich
ioatdma                57654  0
thermal_sys            27642  2 intel_powerclamp,processor
shpchp                 31121  0
i2c_i801               16965  0
vmhgfs                 65266  0
vmw_vmci               55383  1 vmhgfs
fuse                   83350  3
parport_pc             26300  0
ppdev                  16782  0
lp                     17074  0
parport                35749  3 lp,ppdev,parport_pc
autofs4                35529  2
squashfs               43856  1
loop                   26605  3
aufs                  199570  3699
nls_utf8               12456  1
isofs                  38965  1
hid_generic            12393  0
usbhid                 44460  0
hid                   102264  2 hid_generic,usbhid
sd_mod                 44356  0
crc_t10dif             12431  1 sd_mod
sg                     29973  0
crct10dif_generic      12581  1
crct10dif_common       12356  2 crct10dif_generic,crc_t10dif
sr_mod                 21903  1
cdrom                  47424  1 sr_mod
usb_storage            56215  1
ata_generic            12490  0
mgag200                40352  2
crc32c_intel           21809  0
ttm                    77862  1 mgag200
drm_kms_helper         49210  1 mgag200
ata_piix               33592  0
psmouse                99249  0
ehci_pci               12512  0
uhci_hcd               43499  0
igb                   171921  0
ehci_hcd               69837  1 ehci_pci
libata                177508  2 ata_generic,ata_piix
drm                   249998  5 ttm,drm_kms_helper,mgag200
dca                    13168  2 igb,ioatdma
mpt2sas               151328  0
ptp                    17692  1 igb
usbcore               195468  5 uhci_hcd,usb_storage,ehci_hcd,ehci_pci,usbhid
raid_class             12788  1 mpt2sas
pps_core               17225  1 ptp
usb_common             12440  1 usbcore
i2c_algo_bit           12751  2 igb,mgag200
megaraid_sas           97499  0
scsi_transport_sas     33531  1 mpt2sas
i2c_core               46012  6 drm,igb,i2c_i801,drm_kms_helper,mgag200,i2c_algo_bit
scsi_mod              191405  9 sg,scsi_transport_sas,usb_storage,mpt2sas,libata,sd_mod,sr_mod,raid_class,megaraid_sas
button                 12944  0

user@debian:~$ readlink /sys/block/sda/device
../../../1:0:0:0
user@debian:~$ readlink /sys/block/sda/device/driver
../../../../../../../../../
bus/scsi/drivers/sd
user@debian:~$ readlink /sys/block/sdb/device/driver
../../../../../../../../../bus/scsi/drivers/sd
user@debian:~$ readlink /sys/block/sdc/device/driver
../../../../../../../../../bus/scsi/drivers/sd
user@debian:~$ readlink /sys/block/sdd/device/driver
../../../../../../../../../bus/scsi/drivers/sd
user@debian:~$ readlink /sys/block/sda/device/driver/module

and here is the output of smartctl in case that illuminates anything...

user@debian:~$ sudo smartctl -x /dev/sda
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, [URL='http://www.smartmontools.org']www.smartmontools.org[/URL]

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              DKR5C-J600SS
Revision:             D028
Compliance:           SPC-4
User Capacity:        585,399,674,080 bytes [585 GB]
Logical block size:   520 bytes
Rotation Rate:        10020 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca012bb9088
Serial number:        PPKA4SXB
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Feb 18 23:11:31 2017 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     29 C
Drive Trip Temperature:        85 C

Manufactured in week 40 of year 2011
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  230
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  1435
Elements in grown defect list: 29

Vendor (Seagate) cache information
  Blocks sent to initiator = 6435681186349056

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   350667319   372085         0  351039404    4386841       7464.373          96
write:   3648586  1107827         0   4756413    1982003      14855.257           0
verify: 5425996644   941631         0  5426938275     591188     115662.072          23

Non-medium error count:        0

No self-tests have been logged

Background scan results log
  Status: no scans active
    Accumulated power on time, hours:minutes 29252:01 [1755121 minutes]
    Number of background scans performed: 0,  scan progress: 0.00%
    Number of background medium scans performed: 0

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 1
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca012bb9089
    attached SAS address = 0x5c81f660f54f7500
    attached phy identifier = 4
    Invalid DWORD count = 8
    Running disparity error count = 8
    Loss of DWORD synchronization = 2
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 8
     Running disparity error count: 8
     Loss of dword synchronization count: 2
     Phy reset problem count: 0
relative target port id = 2
  generation code = 1
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca012bb908a
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

user@debian:~$
user@debian:~$ sudo smartctl -x /dev/sdb
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, [URL='http://www.smartmontools.org']www.smartmontools.org[/URL]

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              DKR5C-J600SS
Revision:             D028
Compliance:           SPC-4
User Capacity:        585,399,674,080 bytes [585 GB]
Logical block size:   520 bytes
Rotation Rate:        10020 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca025139b0c
Serial number:        PVGAT7XB
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Feb 18 23:13:36 2017 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     28 C
Drive Trip Temperature:        85 C

Manufactured in week 02 of year 2012
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  53
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  1159
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 1100789883863040

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   249325478   364400         0  249689878    5076560      10639.914           0
write:   3069202   193648         0   3262850     961439       4840.152           0
verify: 5450775891  1742418         0  5452518309     846624     105966.777           0

Non-medium error count:        1

No self-tests have been logged

Background scan results log
  Status: no scans active
    Accumulated power on time, hours:minutes 26723:20 [1603400 minutes]
    Number of background scans performed: 0,  scan progress: 0.00%
    Number of background medium scans performed: 0

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 2
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca025139b0d
    attached SAS address = 0x5c81f660f54f7501
    attached phy identifier = 6
    Invalid DWORD count = 20
    Running disparity error count = 20
    Loss of DWORD synchronization = 5
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 20
     Running disparity error count: 20
     Loss of dword synchronization count: 5
     Phy reset problem count: 0
relative target port id = 2
  generation code = 2
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca025139b0e
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

user@debian:~$
user@debian:~$ sudo smartctl -x /dev/sdc
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, [URL='http://www.smartmontools.org']www.smartmontools.org[/URL]

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              DKR5C-J600SS
Revision:             D028
Compliance:           SPC-4
User Capacity:        585,399,674,080 bytes [585 GB]
Logical block size:   520 bytes
Rotation Rate:        10020 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca012c05f34
Serial number:        PPKDTRNB
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Feb 18 23:14:15 2017 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     28 C
Drive Trip Temperature:        85 C

Manufactured in week 39 of year 2011
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  200
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  1467
Elements in grown defect list: 31

Vendor (Seagate) cache information
  Blocks sent to initiator = 6577825310048256

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   369035510   442850         0  369478360    4954024       8608.436          75
write:   5495868   848586         0   6344454    2474597      15371.095           0
verify: [URL='https://forum.proxmox.com/tel:%28507%29%20604-2112']5076042112[/URL]  1099750         0  [URL='https://forum.proxmox.com/tel:%28507%29%20714-1862']5077141862[/URL]     652861     122287.120          26

Non-medium error count:        0

No self-tests have been logged

Background scan results log
  Status: no scans active
    Accumulated power on time, hours:minutes 30701:20 [1842080 minutes]
    Number of background scans performed: 0,  scan progress: 0.00%
    Number of background medium scans performed: 0

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 2
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca012c05f35
    attached SAS address = 0x5c81f660f54f7502
    attached phy identifier = 7
    Invalid DWORD count = 20
    Running disparity error count = 20
    Loss of DWORD synchronization = 5
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 20
     Running disparity error count: 20
     Loss of dword synchronization count: 5
     Phy reset problem count: 0
relative target port id = 2
  generation code = 2
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca012c05f36
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

user@debian:~$
user@debian:~$ sudo smartctl -x /dev/sdd
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, [URL='http://www.smartmontools.org']www.smartmontools.org[/URL]

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              DKR5C-J600SS
Revision:             D028
Compliance:           SPC-4
User Capacity:        585,399,674,080 bytes [585 GB]
Logical block size:   520 bytes
Rotation Rate:        10020 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca012bf30ac
Serial number:        PPKD4L2B
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Feb 18 23:15:47 2017 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     30 C
Drive Trip Temperature:        85 C

Manufactured in week 40 of year 2011
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  192
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  1294
Elements in grown defect list: 10

Vendor (Seagate) cache information
  Blocks sent to initiator = 5449549145440256

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   523678540   674815         0  524353355    5248832       7242.215           0
write:   9655775   630279         0  10286054    2153154      13641.075           0
verify: [URL='https://forum.proxmox.com/tel:%28873%29%20415-5144']8734155144[/URL]  2252038         0  [URL='https://forum.proxmox.com/tel:%28873%29%20640-7182']8736407182[/URL]    1172150     106752.532           0

Non-medium error count:        0

No self-tests have been logged

Background scan results log
  Status: no scans active
    Accumulated power on time, hours:minutes 26715:36 [1602936 minutes]
    Number of background scans performed: 0,  scan progress: 0.00%
    Number of background medium scans performed: 0

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 2
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca012bf30ad
    attached SAS address = 0x5c81f660f54f7503
    attached phy identifier = 5
    Invalid DWORD count = 20
    Running disparity error count = 20
    Loss of DWORD synchronization = 5
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 20
     Running disparity error count: 20
     Loss of dword synchronization count: 5
     Phy reset problem count: 0
relative target port id = 2
  generation code = 2
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca012bf30ae
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

user@debian:~$

as far as the posting pictures go - whenever i try to post a link the forum lets me that i can't because i'm too new and might be a spammer. It weird cuz it let me post links on the 1st message...[/code]

 

[fabian]

okay.. once more, sorry:

udevadm info --path /sys/block/sda --query all
cat /sys/block/sda/dev
cat /sys/block/sda/size
cat /sys/block/sda/device/model

 

[Dieter Toews]

voila!

root@debian:~# udevadm info --path /sys/block/sda --query all
P: /devices/pci0000:00/0000:00:
07.0/0000:04:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda
N: sda
S: disk/by-id/scsi-35000cca012c05f34
S: disk/by-id/wwn-0x5000cca012c05f34
S: disk/by-path/pci-0000:04:00.0-sas-0x5000cca012c05f35-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-35000cca012c05f34 /dev/disk/by-id/wwn-0x5000cca012c05f34 /dev/disk/by-path/pci-0000:04:00.0-sas-0x5000cca012c05f35-lun-0
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/block/sda
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=DKR5C-J600SS
E: ID_MODEL_ENC=DKR5C-J600SS\x20\x20\x20\x20
E: ID_PATH=pci-0000:04:00.0-sas-0x5000cca012c05f35-lun-0
E: ID_PATH_TAG=pci-0000_04_00_0-sas-0x5000cca012c05f35-lun-0
E: ID_REVISION=D028
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PPKDTRNB
E: ID_SERIAL=35000cca012c05f34
E: ID_SERIAL_SHORT=5000cca012c05f34
E: ID_TYPE=disk
E: ID_VENDOR=HITACHI
E: ID_VENDOR_ENC=HITACHI\x20
E: ID_WWN=0x5000cca012c05f34
E: ID_WWN_WITH_EXTENSION=0x5000cca012c05f34
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=73928

root@debian:~# udevadm info --path /sys/block/sdb --query all
P: /devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb
N: sdb
S: disk/by-id/scsi-35000cca012bb9088
S: disk/by-id/wwn-0x5000cca012bb9088
S: disk/by-path/pci-0000:04:00.0-sas-0x5000cca012bb9089-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-35000cca012bb9088 /dev/disk/by-id/wwn-0x5000cca012bb9088 /dev/disk/by-path/pci-0000:04:00.0-sas-0x5000cca012bb9089-lun-0
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:1/end_device-0:1/target0:0:1/0:0:1:0/block/sdb
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=DKR5C-J600SS
E: ID_MODEL_ENC=DKR5C-J600SS\x20\x20\x20\x20
E: ID_PATH=pci-0000:04:00.0-sas-0x5000cca012bb9089-lun-0
E: ID_PATH_TAG=pci-0000_04_00_0-sas-0x5000cca012bb9089-lun-0
E: ID_REVISION=D028
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PPKA4SXB
E: ID_SERIAL=35000cca012bb9088
E: ID_SERIAL_SHORT=5000cca012bb9088
E: ID_TYPE=disk
E: ID_VENDOR=HITACHI
E: ID_VENDOR_ENC=HITACHI\x20
E: ID_WWN=0x5000cca012bb9088
E: ID_WWN_WITH_EXTENSION=0x5000cca012bb9088
E: MAJOR=8
E: MINOR=16
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=75153

root@debian:~# udevadm info --path /sys/block/sdc --query all
P: /devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:2/end_device-0:2/target0:0:2/0:0:2:0/block/sdc
N: sdc
S: disk/by-id/scsi-35000cca012bf30ac
S: disk/by-id/wwn-0x5000cca012bf30ac
S: disk/by-path/pci-0000:04:00.0-sas-0x5000cca012bf30ad-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-35000cca012bf30ac /dev/disk/by-id/wwn-0x5000cca012bf30ac /dev/disk/by-path/pci-0000:04:00.0-sas-0x5000cca012bf30ad-lun-0
E: DEVNAME=/dev/sdc
E: DEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:2/end_device-0:2/target0:0:2/0:0:2:0/block/sdc
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=DKR5C-J600SS
E: ID_MODEL_ENC=DKR5C-J600SS\x20\x20\x20\x20
E: ID_PATH=pci-0000:04:00.0-sas-0x5000cca012bf30ad-lun-0
E: ID_PATH_TAG=pci-0000_04_00_0-sas-0x5000cca012bf30ad-lun-0
E: ID_REVISION=D028
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PPKD4L2B
E: ID_SERIAL=35000cca012bf30ac
E: ID_SERIAL_SHORT=5000cca012bf30ac
E: ID_TYPE=disk
E: ID_VENDOR=HITACHI
E: ID_VENDOR_ENC=HITACHI\x20
E: ID_WWN=0x5000cca012bf30ac
E: ID_WWN_WITH_EXTENSION=0x5000cca012bf30ac
E: MAJOR=8
E: MINOR=32
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=72833

root@debian:~# udevadm info --path /sys/block/sdd --query all
P: /devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:3/end_device-0:3/target0:0:3/0:0:3:0/block/sdd
N: sdd
S: disk/by-id/scsi-35000cca025139b0c
S: disk/by-id/wwn-0x5000cca025139b0c
S: disk/by-path/pci-0000:04:00.0-sas-0x5000cca025139b0d-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-35000cca025139b0c /dev/disk/by-id/wwn-0x5000cca025139b0c /dev/disk/by-path/pci-0000:04:00.0-sas-0x5000cca025139b0d-lun-0
E: DEVNAME=/dev/sdd
E: DEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.0/host0/port-0:3/end_device-0:3/target0:0:3/0:0:3:0/block/sdd
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=DKR5C-J600SS
E: ID_MODEL_ENC=DKR5C-J600SS\x20\x20\x20\x20
E: ID_PATH=pci-0000:04:00.0-sas-0x5000cca025139b0d-lun-0
E: ID_PATH_TAG=pci-0000_04_00_0-sas-0x5000cca025139b0d-lun-0
E: ID_REVISION=D028
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PVGAT7XB
E: ID_SERIAL=35000cca025139b0c
E: ID_SERIAL_SHORT=5000cca025139b0c
E: ID_TYPE=disk
E: ID_VENDOR=HITACHI
E: ID_VENDOR_ENC=HITACHI\x20
E: ID_WWN=0x5000cca025139b0c
E: ID_WWN_WITH_EXTENSION=0x5000cca025139b0c
E: MAJOR=8
E: MINOR=48
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=78484

root@debian:~# cat /sys/block/sda/dev
8:0
root@debian:~# cat /sys/block/sdb/dev
8:16
root@debian:~# cat /sys/block/sdc/dev
8:32
root@debian:~# cat /sys/block/sdd/dev
8:48
root@debian:~# cat /sys/block/sda/size
0
root@debian:~# cat /sys/block/sdb/size
0
root@debian:~# cat /sys/block/sdc/size
0
root@debian:~# cat /sys/block/sdd/size
0
root@debian:~# cat /sys/block/sda/device/model
DKR5C-J600SS   
root@debian:~# cat /sys/block/sdb/device/model
DKR5C-J600SS   
root@debian:~# cat /sys/block/sdc/device/model
DKR5C-J600SS   
root@debian:~# cat /sys/block/sdd/device/model
DKR5C-J600SS   
root@debian:~#

i'm also going to take a shot at updating my bios and see if that gets me anywhere....

 

[fabian]

root@debian:~# cat /sys/block/sda/size
0
root@debian:~# cat /sys/block/sdb/size
0
root@debian:~# cat /sys/block/sdc/size
0
root@debian:~# cat /sys/block/sdd/size
0

that is the reason, because we only put disks with a size > 0 into the disk list. I don't think we can skip this check since we use the disk size later on throughout the installer..

potentially upgrading bios and raid firmware or fiddling with some settings could help? you could also try with a live cd with a newer kernel (e.g., the yet unreleased Ubuntu 17.04 should have 4.9 or 4.10) and check the output of the commands / files we tried, maybe there is some kernel / driver issue as well (but with IT firmware, this should not really matter as the OS "sees" the individual drives).

 

[Dieter Toews]

Thanks for all you help Fabian.

I've already re-flashed my motherboard bios but i still don't see the boot drives i marked in the card bios when i boot into the bios so yes i think it is something in the hardware. I think I'll just breakdown and get some sata drives and do a mirrored boot volume that is separate from VM storage. The backplane i have will happily talk to my motherboards sata ports i was just hoping i could get away with only having one drive pool to manage (and not having to buy more drives).

I'd still like to get it working and I've found a few more things i can try...

here is my plan:

• try the right slot of my riser card - some people say they have had success
• try fildling with more of the HBAs bios setttings
  
• try disabling the disable the on-board controller by setting a jumper: motherboard has a built in lsi 2108 which sadly is useless for zfs because there is no it mode firmware for it. This isn't supposed to be in conflict (you should be able to have more than one card installed without problems but some people report having a problem with this.
• try flashing the 2008 (9122-8i card) to p19 firmware instead of p20 firmware. back in 2014 the 1st revisions of that firmware caused a lot of problems and had bugs. It has since stabilized but could it be that p20 firmware is still not very well supported by linux side drivers?

I'll update this thread with my results...

 

[Dieter Toews]

*** Update ***

I realize the probable cause of my problem is drifting away from proxmox but for the sake of the record I'll try to finish off this thread one way or another.

so far, and without success, I've tried:

• riser card swap: couldn't : it was too long
• hba bios fiddle: nada
• hba bios format drive: drives started showing at bootable from the motherboard bios!!!!!!
• drive "repair" utilities: using fdisk, gdisk and a constellation of other drive formatting tools from the command line. they all seem to just complain and exit. gdisk had some interesting output which seemed to indicate that the MBR of these drives was messed up. Could that be real? I found a utility called sg_format which seems like it might be used to do some extreme surgery but i don't think i know enough about the innards of hard drives to use it.

still to do:

• try fabians suggestion of ubuntu daily builds - next step
  
• try win server 2012 / 2016?? see if i can format the drives and then drop back into linux after to write over top of the NTFS file system with ZFS??
  
• do the firmware downgrade to p.19 of the HBA - it is a PITA so I've been putting it off.

*** Update 2 ****

ubuntu 17 says the same thing that debain did:

ubuntu@ubuntu:~$ sudo -i
root@ubuntu:~# lsblk
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0   7:0    0  1.5G  1 loop /rofs
sr0    11:0    1 1024M  0 rom  
sr1    11:1    1  1.5G  0 rom  /cdrom
root@ubuntu:~# lsmod
Module                  Size  Used by
xt_CHECKSUM            16384  1
iptable_mangle         16384  1
ipt_MASQUERADE         16384  1
nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
iptable_nat            16384  1
nf_conntrack_ipv4      16384  2
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 28672  2 nf_nat_masquerade_ipv4,nf_nat_
ipv4
nf_conntrack          131072  5 nf_conntrack_ipv4,ipt_MASQUERADE,nf_nat_masquerade_ipv4,nf_nat_ipv4,nf_nat
libcrc32c              16384  1 nf_nat
xt_tcpudp              16384  5
bridge                139264  0
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
iptable_filter         16384  1
intel_powerclamp       16384  0
coretemp               16384  0
kvm_intel             200704  0
joydev                 20480  0
input_leds             16384  0
kvm                   593920  1 kvm_intel
gpio_ich               16384  0
ipmi_ssif              24576  0
ipmi_si                57344  0
shpchp                 36864  0
serio_raw              16384  0
ipmi_devintf           20480  0
i7core_edac            24576  0
ioatdma                53248  0
irqbypass              16384  1 kvm
ipmi_msghandler        45056  3 ipmi_ssif,ipmi_devintf,ipmi_si
i5500_temp             16384  0
lpc_ich                24576  0
edac_core              53248  2 i7core_edac
intel_cstate           20480  0
mac_hid                16384  0
parport_pc             32768  0
ppdev                  20480  0
lp                     20480  0
parport                49152  3 lp,parport_pc,ppdev
ip_tables              24576  3 iptable_mangle,iptable_filter,iptable_nat
x_tables               36864  6 iptable_mangle,ip_tables,iptable_filter,xt_tcpudp,ipt_MASQUERADE,xt_CHECKSUM
autofs4                40960  2
aufs                  237568  5181
nls_utf8               16384  1
isofs                  40960  1
dm_mirror              24576  0
dm_region_hash         20480  1 dm_mirror
dm_log                 20480  2 dm_mirror,dm_region_hash
uas                    24576  0
usb_storage            69632  2 uas
hid_generic            16384  0
psmouse               139264  0
igb                   208896  0
dca                    16384  2 ioatdma,igb
pata_acpi              16384  0
mpt3sas               221184  0
ptp                    20480  1 igb
usbhid                 53248  0
pps_core               20480  1 ptp
hid                   114688  2 hid_generic,usbhid
i2c_algo_bit           16384  1 igb
raid_class             16384  1 mpt3sas
scsi_transport_sas     40960  1 mpt3sas
fjes                   73728  0
root@ubuntu:~# readlink /sys/block/sda/device/driver
../../../../../../../../../bus/scsi/drivers/sd
root@ubuntu:~# readlink /sys/block/sdb/device/driver
../../../../../../../../../bus/scsi/drivers/sd
root@ubuntu:~# readlink /sys/block/sdc/device/driver
../../../../../../../../../bus/scsi/drivers/sd
root@ubuntu:~# readlink /sys/block/sdd/device/driver
../../../../../../../../../bus/scsi/drivers/sd
root@ubuntu:~# readlink /sys/block/sda/device/driver/module
root@ubuntu:~# udevadm info --path /sys/block/sda --query all
P: /devices/pci0000:00/0000:00:09.0/0000:05:00.0/host4/port-4:0/end_device-4:0/target4:0:0/4:0:0:0/block/sda
N: sda
S: disk/by-id/scsi-35000cca012bb9088
S: disk/by-id/wwn-0x5000cca012bb9088
S: disk/by-path/pci-0000:05:00.0-sas-phy0-lun-0
E: DEVLINKS=/dev/disk/by-path/pci-0000:05:00.0-sas-phy0-lun-0 /dev/disk/by-id/scsi-35000cca012bb9088 /dev/disk/by-id/wwn-0x5000cca012bb9088
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/pci0000:00/0000:00:09.0/0000:05:00.0/host4/port-4:0/end_device-4:0/target4:0:0/4:0:0:0/block/sda
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=DKR5C-J600SS
E: ID_MODEL_ENC=DKR5C-J600SS\x20\x20\x20\x20
E: ID_PATH=pci-0000:05:00.0-sas-phy0-lun-0
E: ID_PATH_TAG=pci-0000_05_00_0-sas-phy0-lun-0
E: ID_REVISION=D028
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PPKA4SXB
E: ID_SERIAL=35000cca012bb9088
E: ID_SERIAL_SHORT=5000cca012bb9088
E: ID_TYPE=disk
E: ID_VENDOR=HITACHI
E: ID_VENDOR_ENC=HITACHI\x20
E: ID_WWN=0x5000cca012bb9088
E: ID_WWN_WITH_EXTENSION=0x5000cca012bb9088
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=9857892

root@ubuntu:~# udevadm info --path /sys/block/sdc --query all
P: /devices/pci0000:00/0000:00:09.0/0000:05:00.0/host4/port-4:2/end_device-4:2/target4:0:2/4:0:2:0/block/sdc
N: sdc
S: disk/by-id/scsi-35000cca025139b0c
S: disk/by-id/wwn-0x5000cca025139b0c
S: disk/by-path/pci-0000:05:00.0-sas-phy2-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-35000cca025139b0c /dev/disk/by-id/wwn-0x5000cca025139b0c /dev/disk/by-path/pci-0000:05:00.0-sas-phy2-lun-0
E: DEVNAME=/dev/sdc
E: DEVPATH=/devices/pci0000:00/0000:00:09.0/0000:05:00.0/host4/port-4:2/end_device-4:2/target4:0:2/4:0:2:0/block/sdc
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=DKR5C-J600SS
E: ID_MODEL_ENC=DKR5C-J600SS\x20\x20\x20\x20
E: ID_PATH=pci-0000:05:00.0-sas-phy2-lun-0
E: ID_PATH_TAG=pci-0000_05_00_0-sas-phy2-lun-0
E: ID_REVISION=D028
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PVGAT7XB
E: ID_SERIAL=35000cca025139b0c
E: ID_SERIAL_SHORT=5000cca025139b0c
E: ID_TYPE=disk
E: ID_VENDOR=HITACHI
E: ID_VENDOR_ENC=HITACHI\x20
E: ID_WWN=0x5000cca025139b0c
E: ID_WWN_WITH_EXTENSION=0x5000cca025139b0c
E: MAJOR=8
E: MINOR=32
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=9858141

root@ubuntu:~# cat /sys/block/sda/dev
8:0
root@ubuntu:~# cat /sys/block/sda/size
0
root@ubuntu:~# cat /sys/block/sda/device/model
DKR5C-J600SS    
root@ubuntu:~#

 

[Dieter Toews]

Drive Size no longer reports 0!!!!!

after the failure of unbuntu17.x and while waiting for windoz to download i started reviewing the output of smartctl and saw this...

Logical block size:   520 bytes

that fired a neuron that had been activated when i was reading up on sg_format. Apparently storage vendors sometimes use non standard drive sector size... my drives were 520 bytes instead of 512 or 4k!!! who knew? i certainly didn't. A bit more googling and i found a guide on how to actually use sg_format to change the sector size of sas drives. The only difference between what i did and the guide is the use of apt-get instead of yum to get the "sg3_utils" (called sg3-utils in the Debian repo) due to my use of Debian. I've done one drive completely and the size is now:

root@debian:~# cat /sys/block/sda/size
1172123568

My other 3 drives are formatting now... takes about 1.25 hrs... I'll post here and mark solved if i can get the proxmox installer recognizing these things.Now that i know about the sg_format thing i know two things 1. some vendors of enterprise hardware use non standard sector sizes and 2. sh*t is always more complicated and takes longer that you thought it would.

 

[jorgemop]

root@debian:~# cat /sys/block/sda/size
0
root@debian:~# cat /sys/block/sdb/size
0
root@debian:~# cat /sys/block/sdc/size
0
root@debian:~# cat /sys/block/sdd/size
0

that is the reason, because we only put disks with a size > 0 into the disk list. I don't think we can skip this check since we use the disk size later on throughout the installer..

potentially upgrading bios and raid firmware or fiddling with some settings could help? you could also try with a live cd with a newer kernel (e.g., the yet unreleased Ubuntu 17.04 should have 4.9 or 4.10) and check the output of the commands / files we tried, maybe there is some kernel / driver issue as well (but with IT firmware, this should not really matter as the OS "sees" the individual drives).

Hi Fabian, with the proxmox 6.3 installer or with debian 10 iso, have the message "disks no compatible" or "no disk detected" respectively. I would like install proxmox in IBM x3650 M3 with raid controller for sas disks, no raid hardware configure just pass the disk whitout config; the idea is to managed the reduncancy whit ZFS on top level. Do you have some idea or and advice about what to do?

 

[fabian]

Hi Fabian, with the proxmox 6.3 installer or with debian 10 iso, have the message "disks no compatible" or "no disk detected" respectively. I would like install proxmox in IBM x3650 M3 with raid controller for sas disks, no raid hardware configure just pass the disk whitout config; the idea is to managed the reduncancy whit ZFS on top level. Do you have some idea or and advice about what to do?

you need to find out how to make the disks visible/available with their proper size.. we don't have such a controller in our testlab so I am afraid we can't help you with that..

 

[jorgemop]

you need to find out how to make the disks visible/available with their proper size.. we don't have such a controller in our testlab so I am afraid we can't help you with that..

Yes, i do the old trick, since the raid controller don't have the capability for JBOD, in the RAID Bios i create a RAID 0 for each disk "-CfgEachDskRaid0 -aAll" at this point the installation was succefull. Now let's check performance. Thanks in advance