Manually create OSD

[David Herselman]

We've added some NVMe drives to our production cluster and get maximal IOPS (100k) and throughput (1GB/s) when benchmarking with 8 threads.

Could someone please detail the commands that Proxmox issues when partitioning and formatting drives?

We essentially want to partition our drives (3.6TB) in to 7 x 500GB BlueStore volumes, to take advantage of concurrent capabilities on the NVMe units and additionally use them as write ahead logs for FileStore spinners.

 

[fabian]

https://git.proxmox.com/?p=pve-mana...5b4816e8cbccd7d6f97;hb=refs/heads/master#l230

it basically boils down to this (stuff in CAPS has to be replaced accordingly, stuff in [] is optional):

1) ensure everything is setup as expected (config symlinks, bootstrap keys, devices are valid block devices, ...)
2) prepare OSD
a) bluestore:
ceph-disk prepare --zap-disk --cluster CLUSTER --cluster-uid FSID --bluestore [--block.db JOURNAL_DEV] [--block.wal WAL_DEV] OSD_DEV
b) filestore:
ceph-disk prepare --zap-disk --cluster CLUSTER --cluster-uid FSID --filestore --fs-type FS [--journal-dev] OSD_DEV [JOURNAL_DEV]