Help, Ceph Configuration Showing Warnings

[Kratus]

Hello,
I am setting up Proxmox VE in my IT environment and encountering issues with Ceph configuration. My current setup is as follows:

Environment Details:​

• Server Model: Dell R640
• Disks:
  • 2 SSDs configured in RAID 1 (dedicated to Proxmox VE installation).
  • 4 SAS disks (7.2TB each, exclusively used for Ceph).
• Network:
  • 10Gbps 2x SFP connections on a private network.
  • Ceph cluster and VMs are isolated on different subnets.
• Proxmox VE Version: 8.3.0
• Ceph Version: Reef 18.2.4

Issue:​

When configuring Ceph, I see warnings in the Proxmox dashboard, such as:

• "OSD count 4 < osd_pool_default_size 2"
• "Reduced data availability."

I have configured each SAS disk as RAID 0 (one virtual disk per physical disk) to make them compatible with Ceph OSDs. However, these warnings persist even after proper network and disk configurations, when I include the OSD with 4 disks, an error occurs after creating the .mgr pool. I don’t know what else to do.

What I need help with:​

I would like guidance on how to properly configure Ceph in this environment to ensure

• Functional failover between nodes.
• Optimal performance and storage utilization.
• Reduced replication factor for better capacity without compromising failover.

Any advice or suggestions to resolve these warnings would be greatly appreciated, thanks

 

[Weehooey-HSS]

Hello!

First I recommend against setting up single-drive raid 0s in production. This can cause issues, and drives will not report proper data to Proxmox.

Functional failover between nodes.

Have you added other nodes to the cluster yet?
From the looks of things, you are setting up CEPH with only one node.
Ceph requires a minimum 3 nodes

 

[Kratus]

Hello!

First I recommend against setting up single-drive raid 0s in production. This can cause issues, and drives will not report proper data to Proxmox.


Have you added other nodes to the cluster yet?
From the looks of things, you are setting up CEPH with only one node.
Ceph requires a minimum 3 nodes

Hello,

I’m building my first node, and I was wondering if this error is common when there’s only one node? On my bare-metal server with the PERC controller, I only have the option for HBA or RAID. In this case, would it be ideal to set everything to HBA mode and run Proxmox directly on the SAS disks?

Thank you!

 

[Weehooey-HSS]

The errors are caused by only having one node setup.

If your PERC can do HBA, then go with HBA. We lean towards mirrored vdevs with ZFS for our Proxmox OS drives.

If you plan on using CEPH and ZFS on your boot drives, be sure they are enterprise-class drives with more than 1 DWPD!

Thanks!

 

[UdoB]

Any advice or suggestions to resolve these warnings would be greatly appreciated, thanks

@Weehooey-HSS already mentioned the requirement of three nodes.

On last Christmas I had written some more hints here: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

 

[Kratus]

Hello friends,

I followed Weehooey-HSS suggestion regarding HBA and then implemented ZFS with two disks on Proxmox, moving forward to four disks with Ceph. I am aware that Ceph requires more nodes, however, I am adding nodes one at a time. Since I am not experienced with Proxmox, I'm not sure if the yellow flag will "disappear" after having at least 5 nodes as UdoB suggested. I managed to set it up and created an RBD storage pool. Everything seemed normal, but now I can't even create a VM. The PVE management interface keeps loading... and the OSD disks have also disappeared... I would like to share the images I have for your opinions if you could help me...

 

[gurubert]

You have to have at least three nodes active before being able to do anything with Ceph storage.

 

[ceelight]

Yeah, Ceph on a one node system is more or less useless (sorry). Go for LVM instead for homelab use as ZFS is consuming RAM.