Search results

  1. N

    [Ceph] unable to run OSDs

    Hello Alwin, Unfortunately we are unable to get the drives to function at all in the office. I am assuming that the HBA/cables we have are unable to correctly initialise the drives, and they do not spin up... (does SAS require a signal saying "ok, you can fully power on now"??) So we are...
  2. N

    [Ceph] unable to run OSDs

    Hello Alwin, I have 4 of the drives back at our office, but am unable to connect them to any of our spare HBA cards as the drives appear to use SFF-8482 connectors. We have therefore ordered the correct cable set, and I will be attempting to test the drives when this arrives next week...
  3. N

    [Ceph] unable to run OSDs

    Hello Alwin, While this is a logical step, it would be difficult for us to attempt this due to the drives being SAS. All previous drives we have used for Ceph have been SATA. (it's only because Dell did not offer SATA 8TB drives in this platform that we have ended up with SAS here!) Trying to...
  4. N

    [Ceph] unable to run OSDs

    We've stepped through the commands shown in the logs by hand, and managed to generate an strace of the ceph-osd command that fails (attached). This bit looks interesting: ... madvise(0x55aaa41de000, 1990656, MADV_FREE) = 0 pread64(20, 0x55aaa41de000, 1048576, 3840699006976) = -1 EILSEQ...
  5. N

    [Ceph] unable to run OSDs

    Results of physical investigation: The HDDs and SSDs are all connected to the same Storage Backplane in the server, which is linked back to the onboard controller (the HBA330 Mini) via two cables with "unusual" cables on each end. Therefore the HDDs and SSDs are all connected to the same...
  6. N

    [Ceph] unable to run OSDs

    Allegedly, the SSDs and HDDs are on the same controller. This will be confirmed when we physically look inside the server :) Well, I gave it a go: root@ceph1m-2:/opt/MegaRAID/perccli# dd if=/dev/zero of=/dev/sde bs=1M count=2048 oflag=direct 2048+0 records in 2048+0 records out 2147483648...
  7. N

    [Ceph] unable to run OSDs

    %@£(%*^*, I didn't think of trying that! :) Bluestore OSD on an SSD works fine... and the SSDs respond correctly to 'hdparm -I'. We're going to look very closely at the HDDs and controller. My apologies for possibly having wasted your time today; I will come back to this ticket once we have...
  8. N

    [Ceph] unable to run OSDs

    Hello Alwin, Well, we have 3 servers exhibiting this behaviour so if all 3 have broken HBAs I would consider myself to be quite unlucky ;) A firmware bug is possible, but there is no updated firmware available. Dell call the controller "Dell HBA330 Mini (Embedded)", and we are currently using...
  9. N

    [Ceph] unable to run OSDs

    Hello Alwin, I've tried again after blanking the device with urandom: root@ceph1m-2:~# dd if=/dev/urandom of=/dev/sde bs=1M count=500 conv=fdatasync oflag=direct 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 10.6133 s, 49.4 MB/s but the results are the same -...
  10. N

    [Ceph] unable to run OSDs

    Hello Alwin, While I appreciate that you probably see a lot of these sort of posts, unfortunately the forum thread that you linked to does not give me any more information than I already have. As you will see in line 5 of the first CODE block in my post, I am already using dd to wipe the OSD...
  11. N

    [Ceph] unable to run OSDs

    My apologies in advance for the length of this post! During a new hardware install, our Ceph node/server is: Dell PowerEdge R7415: 1x AMD EPYC 7251 8-Core Processor 128GB RAM HBA330 disk controller (LSI/Broadcom SAS3008, running FW 15.17.09.06 in IT mode) 4x Toshiba THNSF8200CCS 200GB...

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!