[SOLVED] ceph mix osd type ssd (sas,nvme(u2),nvme(pcie))

ilia987

Member
Sep 9, 2019
228
9
23
35
I asked similar question around a year ago but i did not find it so ill ask it here again.

Our system:
proxmox cluster based on 6.3-2 10 node,
ceph pool based on 24 osds sas3 (4 or 8 TB ) more will be added soon. (split across 3 nodes, 1 more node will be added this week )

we plan to add more drives (some HDDs and some nvme(u2) nvme(pcie)

i want to make dedicated pools on each drive type .
for example:
  • sas3 (replication 3) (read intensive )
    • pool 1 (existing)
    • pool 2 (existing)
  • hdd (replication 3 ) low access data, mainly archive
    • pool 3
  • nvme u2 (replication 3 ) (lxc host )(high performance iops)
    • pool 4
  • nvme pcie (replication 1 ) (network shared tmp folder for grid computational task ) (very low latancy and high iops)
    • pool 5
is it possible?
is it safe to do it? because this is our main\production environment
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
2,843
435
88
Ceph will autodetect the device type and set the device class accordingly (hdd, ssd, nvme). But you can manually specify different device classes of you choosing and then create crush rules that limit to these classes. These rules can then be assigned to the pools so they will be stored only on that device class.

https://docs.ceph.com/en/octopus/rados/operations/crush-map/#device-classes
 
  • Like
Reactions: ilia987

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
2,843
435
88
Yes. If you assign a specific rule to a pool, you will see that a rebalance will take place to move the pools data to the device types configured in the rule. Just make sure to have enough space with the limited device classes as the total available space for the pool will be limited to these device classes.
Having plenty of space left helps in case a node fails and the 3rd replicas need to be created on the remaining nodes.
 
  • Like
Reactions: ilia987

ilia987

Member
Sep 9, 2019
228
9
23
35
Yes. If you assign a specific rule to a pool, you will see that a rebalance will take place to move the pools data to the device types configured in the rule. Just make sure to have enough space with the limited device classes as the total available space for the pool will be limited to these device classes.
Having plenty of space left helps in case a node fails and the 3rd replicas need to be created on the remaining nodes.
Thanks.

Once ill install all the new hardware ill post here the commands i am planning to do for a short review.
 
  • Like
Reactions: lmj and aaron

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!