Proxmox with Ceph - Disk crashed rate is too high

[viisking]

My cluster has 5 HP DL380G9, P440ar card, 3x4TB Seagate HDD, 3x4TB Samsung QVO each server for Ceph.
I got quite many OSD crash recent weeks (~15%), on both HDD and SSD. Anyone has exp on that?

 

[gurubert]

The log shows hardware issues. You should exchange the affected drives.

 

[ness1602]

Or maybe bad cable,that could also be the problem.

 

[viisking]

This problem happened random, on all servers, so might not be cable issue

 

[Falk R.]

I think the QLC NAND SSDs are not so good for Ceph. Have you any other SSD to test?

 

[viisking]

I think the QLC NAND SSDs are not so good for Ceph. Have you any other SSD to test?

I also think consumer HDD/SSD are not good for Ceph. Any work-around for that, or replace all disks to SAS/Enterprise disk grades?

 

[czechsys]

P440ar ? It's not real HBA controller, maybe problem is there...

 

[Falk R.]

P440ar ? It's not real HBA controller, maybe problem is there...

You can set the P440ar to HBA Mode. Then is the Raid Chip deactivated.

 

[p.jahn]

Did you find a solution?

I get the same errors with Samsung SM863a drives connected to a Broadcom/LSI 9305 HBA.

Happens random on 4 nodes as soon as I boot Kernel 5.11 or 5.13 but 5.4 seems fine.