[SOLVED] FYI: do not extend Ceph with OSDs connected via USB

[UdoB]

Just written down for your amusement, on a lazy, dark and rainy Sunday afternoon:

Someone (me) might try to extend a Ceph cluster by adding NVMe (or other SSDs) via USB3 Adapters. For a small homelab this should be feasable, isn't it? My three PVE nodes already run a single Ceph OSD on an internal NVMe; two other (slightly slower) nodes w/o VMs add another two OSDs. So five OSDs in total.

All nodes are connected via a 2.5 GBit/s switch; the NICs are mixed Intel+Realtek. The following tests ignore the fact that the PVE cluster is actively in use, with low load. In this specific setup my personal standard variant of a fio test gives me ~3000 IOPS on a mounted cephfs:

root@pveg:/mnt/pve/cephtestc/fio# fio --name=randrw --ioengine=libaio --direct=1  --rw=randrw --bs=4k --numjobs=1 --iodepth=16 --size=8G --runtime=60 --time_based --rwmixread=75 
  read: IOPS=2501, BW=9.77MiB/s (10.2MB/s)(586MiB/60011msec)
  write: IOPS=833, BW=3335KiB/s (3415kB/s)(195MiB/60011msec); 0 zone resets

Now I added one Samsung 2 TB NVMe disk and one 1 TB Intenso NVMe on different nodes. Via USB 3, so with 5 GBit/s. This is what I got:

root@pveg:/mnt/pve/cephtestc/fio# fio --name=randrw --ioengine=libaio --direct=1  --rw=randrw --bs=4k --numjobs=1 --iodepth=16 --size=8G --runtime=60 --time_based --rwmixread=75 
 read: IOPS=659, BW=2639KiB/s (2702kB/s)(155MiB/60126msec)
 write: IOPS=219, BW=880KiB/s (901kB/s)(51.7MiB/60126msec); 0 zone resets

One step back: removing the 2TB Samsung OSD gives me:

root@pveg:/mnt/pve/cephtestc/fio# fio --name=randrw --ioengine=libaio --direct=1  --rw=randrw --bs=4k --numjobs=1 --iodepth=16 --size=8G --runtime=60 --time_based --rwmixread=75 
 read: IOPS=1341, BW=5365KiB/s (5494kB/s)(315MiB/60031msec)
 write: IOPS=447, BW=1788KiB/s (1831kB/s)(105MiB/60031msec); 0 zone resets

Second step back: remove the remaining 1 TB Intenso OSD:

root@pveg:/mnt/pve/cephtestc/fio# fio --name=randrw --ioengine=libaio --direct=1  --rw=randrw --bs=4k --numjobs=1 --iodepth=16 --size=8G --runtime=60 --time_based --rwmixread=75 
 read: IOPS=2161, BW=8647KiB/s (8854kB/s)(507MiB/60022msec)
 write: IOPS=720, BW=2881KiB/s (2950kB/s)(169MiB/60022msec); 0 zone resets

So I am back to where I came from

Lesson learned: a single OSD with a very high latency (=low IOPS) might slow down the complete cluster. The limitations of Mini PCs are hard to circumvent, USB is NOT a valid solution for THIS problem in my specific situation.

---

Btw: above tests use a blocksize of 4 kB - the goal was to check IOPS. With 32 kB I get nearly the same IOPS but - of course - higher transfer rates:

root@pveg:/mnt/pve/cephtestc/fio# fio --name=randrw --ioengine=libaio --direct=1  --rw=randrw --bs=32k --numjobs=1 --iodepth=16 --size=8G --runtime=60 --time_based --rwmixread=75 
randrw: (groupid=0, jobs=1): err= 0: pid=2462444: Sun Feb  4 13:18:37 2024
  read: IOPS=2207, BW=69.0MiB/s (72.3MB/s)(4141MiB/60015msec)
  write: IOPS=736, BW=23.0MiB/s (24.1MB/s)(1382MiB/60015msec); 0 zone resets