[SOLVED] ceph mix osd type ssd (sas,nvme(u2),nvme(pcie))

[ilia987]

I asked similar question around a year ago but i did not find it so ill ask it here again.

Our system:
proxmox cluster based on 6.3-2 10 node,
ceph pool based on 24 osds sas3 (4 or 8 TB ) more will be added soon. (split across 3 nodes, 1 more node will be added this week )

we plan to add more drives (some HDDs and some nvme(u2) nvme(pcie)

i want to make dedicated pools on each drive type .
for example:

• sas3 (replication 3) (read intensive )
  • pool 1 (existing)
  • pool 2 (existing)
• hdd (replication 3 ) low access data, mainly archive
  • pool 3
• nvme u2 (replication 3 ) (lxc host )(high performance iops)
  • pool 4
• nvme pcie (replication 1 ) (network shared tmp folder for grid computational task ) (very low latancy and high iops)
  • pool 5

is it possible?
is it safe to do it? because this is our main\production environment

 

[aaron]

Ceph will autodetect the device type and set the device class accordingly (hdd, ssd, nvme). But you can manually specify different device classes of you choosing and then create crush rules that limit to these classes. These rules can then be assigned to the pools so they will be stored only on that device class.

https://docs.ceph.com/en/octopus/rados/operations/crush-map/#device-classes

 

[ilia987]

is it safe to do on a running ceph cluster?

 

[aaron]

Yes. If you assign a specific rule to a pool, you will see that a rebalance will take place to move the pools data to the device types configured in the rule. Just make sure to have enough space with the limited device classes as the total available space for the pool will be limited to these device classes.
Having plenty of space left helps in case a node fails and the 3rd replicas need to be created on the remaining nodes.

 

[ilia987]

Yes. If you assign a specific rule to a pool, you will see that a rebalance will take place to move the pools data to the device types configured in the rule. Just make sure to have enough space with the limited device classes as the total available space for the pool will be limited to these device classes.
Having plenty of space left helps in case a node fails and the 3rd replicas need to be created on the remaining nodes.

Thanks.

Once ill install all the new hardware ill post here the commands i am planning to do for a short review.