Failure using SPDK and ceph-osd_15.2.5-pve1_amd64.deb

[Trilom]

Greetings,

I have been fighting through trying to enable spdk with ceph and have ran into a block and I'm not sure where to go from here and was curious if your team had some input.

The attempt here is to use spdk to present an nvme device to ceph. When using spdk to test the nvme device it works without a problem locally however when attempting to coerce ceph to use the device there are failures that I can't find much information regarding but came across some build files that might point to the problem potentially. Since the version of ceph* I am using is the pve built versions I am not 100% sure if certain build flags have been passed at build time, nor am I sure how to validate. The error I receive when creating an osd is below and has a failure when it attempts to call the "ust_nvme" backend type that doesn't exist. When doing some searching in the repo there appear to be come cmake (build flags?) that might need to be added in order to include this backend type. Here is a link to the BlockDevice.cc in the octopus head. From what I understand it should call the NVMEDevice header if HAVE_SPDK is included so my assumption is this build flag is not included and it can't find this file?

Any guidance would be nice, my next step is to open an issue with ceph but I'm afraid they'll punt me here or tell me to use their build. I plan on starting down the road of trying the debain buster octopus build to see if there are any changes.

Thanks

root@frank:~/spdk# ceph-osd -i 2 --mkfs -c /etc/pve/ceph.conf --debug_osd 10 --setuser ceph --setgroup ceph --cluster ceph --osd-uuid 7e5d5d56-80a1-4db4-b743-972c7f1e1f96
2020-11-17T09:09:14.747-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
2020-11-17T09:09:14.747-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
2020-11-17T09:09:14.747-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
2020-11-17T09:09:14.751-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid
2020-11-17T09:09:14.751-0500 7f6eb7768e00 -1 bdev create unknown backend ust-nvme
/build/ceph/ceph-15.2.5/src/os/bluestore/BlockDevice.cc: In function 'static BlockDevice* BlockDevice::create(ceph::common::CephContext*, const string&, BlockDevice::aio_callback_t, void*, BlockDevice::aio_callback_t, void*)' thread 7f6eb7768e00 time 2020-11-17T09:09:14.755541-0500
/build/ceph/ceph-15.2.5/src/os/bluestore/BlockDevice.cc: 135: ceph_abort_msg("abort() called")
 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x55b7667809d0]
 2: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 3: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 4: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 5: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 6: (main()+0x1443) [0x55b7667c0253]
 7: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 8: (_start()+0x2a) [0x55b7667d56fa]
2020-11-17T09:09:14.775-0500 7f6eb7768e00 -1 /build/ceph/ceph-15.2.5/src/os/bluestore/BlockDevice.cc: In function 'static BlockDevice* BlockDevice::create(ceph::common::CephContext*, const string&, BlockDevice::aio_callback_t, void*, BlockDevice::aio_callback_t, void*)' thread 7f6eb7768e00 time 2020-11-17T09:09:14.755541-0500
/build/ceph/ceph-15.2.5/src/os/bluestore/BlockDevice.cc: 135: ceph_abort_msg("abort() called")

 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x55b7667809d0]
 2: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 3: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 4: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 5: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 6: (main()+0x1443) [0x55b7667c0253]
 7: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 8: (_start()+0x2a) [0x55b7667d56fa]

*** Caught signal (Aborted) **
 in thread 7f6eb7768e00 thread_name:ceph-osd
 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (()+0x12730) [0x7f6eb81b9730]
 2: (gsignal()+0x10b) [0x7f6eb7c9a7bb]
 3: (abort()+0x121) [0x7f6eb7c85535]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x55b766780aa1]
 5: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 6: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 7: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 8: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 9: (main()+0x1443) [0x55b7667c0253]
 10: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 11: (_start()+0x2a) [0x55b7667d56fa]
2020-11-17T09:09:14.787-0500 7f6eb7768e00 -1 *** Caught signal (Aborted) **
 in thread 7f6eb7768e00 thread_name:ceph-osd

 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (()+0x12730) [0x7f6eb81b9730]
 2: (gsignal()+0x10b) [0x7f6eb7c9a7bb]
 3: (abort()+0x121) [0x7f6eb7c85535]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x55b766780aa1]
 5: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 6: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 7: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 8: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 9: (main()+0x1443) [0x55b7667c0253]
 10: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 11: (_start()+0x2a) [0x55b7667d56fa]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

    -9> 2020-11-17T09:09:14.747-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
    -8> 2020-11-17T09:09:14.747-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
    -7> 2020-11-17T09:09:14.747-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
    -6> 2020-11-17T09:09:14.751-0500 7f6eb7768e00 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid
    -2> 2020-11-17T09:09:14.751-0500 7f6eb7768e00 -1 bdev create unknown backend ust-nvme
    -1> 2020-11-17T09:09:14.775-0500 7f6eb7768e00 -1 /build/ceph/ceph-15.2.5/src/os/bluestore/BlockDevice.cc: In function 'static BlockDevice* BlockDevice::create(ceph::common::CephContext*, const string&, BlockDevice::aio_callback_t, void*, BlockDevice::aio_callback_t, void*)' thread 7f6eb7768e00 time 2020-11-17T09:09:14.755541-0500
/build/ceph/ceph-15.2.5/src/os/bluestore/BlockDevice.cc: 135: ceph_abort_msg("abort() called")

 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x55b7667809d0]
 2: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 3: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 4: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 5: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 6: (main()+0x1443) [0x55b7667c0253]
 7: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 8: (_start()+0x2a) [0x55b7667d56fa]

     0> 2020-11-17T09:09:14.787-0500 7f6eb7768e00 -1 *** Caught signal (Aborted) **
 in thread 7f6eb7768e00 thread_name:ceph-osd

 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (()+0x12730) [0x7f6eb81b9730]
 2: (gsignal()+0x10b) [0x7f6eb7c9a7bb]
 3: (abort()+0x121) [0x7f6eb7c85535]
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x55b766780aa1]
 5: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 6: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 7: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 8: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 9: (main()+0x1443) [0x55b7667c0253]
 10: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 11: (_start()+0x2a) [0x55b7667d56fa]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

*** Caught signal (Segmentation fault) **
 in thread 7f6eb7768e00 thread_name:ceph-osd
 ceph version 15.2.5 (73e4b91c0ea5263d266983f379539340dea672fa) octopus (stable)
 1: (()+0x12730) [0x7f6eb81b9730]
 2: (pthread_getname_np()+0x44) [0x7f6eb81ba8e4]
 3: (ceph::logging::Log::dump_recent()+0x428) [0x55b766fcf7d8]
 4: (()+0x1032385) [0x55b766df0385]
 5: (()+0x12730) [0x7f6eb81b9730]
 6: (gsignal()+0x10b) [0x7f6eb7c9a7bb]
 7: (abort()+0x121) [0x7f6eb7c85535]
 8: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x55b766780aa1]
 9: (BlockDevice::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (*)(void*, void*), void*, void (*)(void*, void*), void*)+0x4d4) [0x55b766d812c4]
 10: (BlueStore::_open_bdev(bool)+0x6d) [0x55b766c91efd]
 11: (BlueStore::mkfs()+0x416) [0x55b766cfced6]
 12: (OSD::mkfs(ceph::common::CephContext*, ObjectStore*, uuid_d, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x1ae) [0x55b7667fff9e]
 13: (main()+0x1443) [0x55b7667c0253]
 14: (__libc_start_main()+0xeb) [0x7f6eb7c8709b]
 15: (_start()+0x2a) [0x55b7667d56fa]
root@frank:~/spdk# apt list --installed | grep ceph

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

ceph-base/stable,now 15.2.5-pve1 amd64 [installed]
ceph-common/stable,now 15.2.5-pve1 amd64 [installed]
ceph-fuse/stable,now 15.2.5-pve1 amd64 [installed]
ceph-mds/stable,now 15.2.5-pve1 amd64 [installed]
ceph-mgr-modules-core/stable,now 15.2.5-pve1 all [installed,automatic]
ceph-mgr/stable,now 15.2.5-pve1 amd64 [installed]
ceph-mon/stable,now 15.2.5-pve1 amd64 [installed]
ceph-osd/stable,now 15.2.5-pve1 amd64 [installed]
libcephfs2/stable,now 15.2.5-pve1 amd64 [installed]
python-ceph-argparse/now 14.2.11-pve1 all [installed,local]
python3-ceph-argparse/stable,now 15.2.5-pve1 all [installed,automatic]
python3-ceph-common/stable,now 15.2.5-pve1 all [installed,automatic]
python3-cephfs/stable,now 15.2.5-pve1 amd64 [installed,automatic]

 

[Alwin]

Ahead of things, best try the ceph mailing list. This is a very very special topic and may be broken to beginn with.

Since the version of ceph* I am using is the pve built versions I am not 100% sure if certain build flags have been passed at build time, nor am I sure how to validate.

We leave the defaults, otherwise there are patches in the patches directory.

Any guidance would be nice, my next step is to open an issue with ceph but I'm afraid they'll punt me here or tell me to use their build. I plan on starting down the road of trying the debain buster octopus build to see if there are any changes.

Whichever, this setup isn't supported anyway.

I have been fighting through trying to enable spdk with ceph and have ran into a block and I'm not sure where to go from here and was curious if your team had some input.

Have you checked their documentation?
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#spdk-usage

When doing some searching in the repo there appear to be come cmake (build flags?) that might need to be added in order to include this backend type.

The SPDK flag needs to be enabled in CMake.

 

[Trilom]

Thank you very much for the prompt reply Alwin.

Being that the flag needs to be enabled, is there a place where I can pull, enable, build and test or can you provide me a build with this enabled for testing?

Their documentation is very sparse and fluid however I believe everything is in the place it needs to be. There is a good example here that you can follow if you'd like. Basically once the host is configured (I am able to test the host independently of ceph and it works fine as root, the files ceph should be using are perm correctly) you should be able to set some osd config in the ceph.conf. Based on what I've interpreted from the vstart.sh script I have added:

[global]
...
     bluestore_block_db_create = false
     bluestore_block_db_path = ""
     bluestore_block_db_size = 0
     bluestore_block_wal_create = false
     bluestore_block_wal_path = ""
     bluestore_block_wal_size = 0
...
[osd.2]
     bluestore_block_path = spdk:0000:04:00.0
     host = frank

When I attempt to create the osd the files do get created in the directory but the fsid is unpopulated, and the error given leads me to believe it didn't understand the nvme device correctly.

$uuid=$(uuidgen)
# create osd with keyfile
ceph -c /etc/pve/ceph.conf osd $uuid new -i test.json
# make osd dir
rm -rf /var/lib/ceph/osd/ceph-2 && mkdir /var/lib/ceph/osd/ceph-2/ && chown ceph:ceph /var/lib/ceph/osd/ceph-2/
# create keyring file
ceph auth get-or-create osd.2 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -o /var/lib/ceph/osd/ceph-2/keyring && chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring
# create osd
ceph-osd -i 2 --mkfs -c /etc/pve/ceph.conf --debug_osd 10 --setuser ceph --setgroup ceph --cluster ceph --osd-uuid $uuid
# inspect fs
ls -lah /var/lib/ceph/osd/ceph-2/
total 20K
drwxr-xr-x  2 ceph ceph  7 Nov 17 09:09 .
drwxr-xr-x 21 ceph ceph 21 Nov 17 09:08 ..
lrwxrwxrwx  1 ceph ceph 17 Nov 17 09:09 block -> spdk:0000:04:00.0
-rw-r--r--  1 ceph ceph  0 Nov 17 09:09 fsid
-rw-r--r--  1 ceph ceph 56 Nov 17 09:09 keyring
-rw-r--r--  1 ceph ceph 31 Nov 17 09:09 spdk:0000:04:00.0
-rw-------  1 ceph ceph 10 Nov 17 09:09 type

 

[Alwin]

Being that the flag needs to be enabled, is there a place where I can pull, enable, build and test or can you provide me a build with this enabled for testing?

All our source can be found on our public git.
https://git.proxmox.com/

Basically once the host is configured (I am able to test the host independently of ceph and it works fine as root, the files ceph should be using are perm correctly) you should be able to set some osd config in the ceph.conf.

They also adjusted the bluestore_spdk_mem = 2048" .

When I attempt to create the osd the files do get created in the directory but the fsid is unpopulated, and the error given leads me to believe it didn't understand the nvme device correctly.

I don't know how you create the OSD, but you might use ceph-volume raw prepare --bluestore --data {device} to tell the OSD to access a raw device.

 

[Trilom]

Thank you for the link to the source, I found it just a little ago and am in the process of building OSD now with SPDK enabled and will let you know how it goes.

Using ceph-volume raw prepare --bluestore --data {device} might not be a bad idea now but I'm not 100% sure. Prior to this when I would try to pass something that doesn't have a "/dev" prefix it complains about that. I'm not sure if this has changed in octopus but I can try this. Another idea is that I can pass the /dev/vfio/31 device to the above raw command to see if it knows what to do with it, but there isn't any documentation so I'll poke and let you know what happens!

 

[Trilom]

Okay this is looking a lot better.

While a build attempt was firing off I tried mounting the /dev/vfio/31 device with no luck, and had to umount /dev/vfio/31 to make it go away. This was somewhat expected. Trying the ceph-volume raw prepare --bluestore command gave me the same results as above, and as in nautulis.

After a successful build I am getting a failure, but it is initializing the spdk as expected. The issue below seems to point to it not being able to access hugepages at 2048 but I have mine set to 1G, so this is a configuration issue in ceph/spdk at this point.

Being that it isn't supported by Proxmox that is fine for my needs, but is there a way to include it in the build for unsupported usage? This is driving a decision to continue using the hyper-converged portion of Proxmox for my workload at this point as this adds steps to deployment. Here are some performance numbers that I have been tracking to "juice" these nvme disks. As you can see the gains are very substantial during these 60 second tests and ultimately will be rewarded soon (hopefully). I've been tracking all of my disks to see how their performance is and this by far has been the best tree to shake.



Thanks

"Successful" result

root@frank:~/spdk# ceph-osd -i 2 --mkfs -c /etc/pve/ceph.conf --debug_osd 10 --setuser ceph --setgroup ceph --cluster ceph --osd-uuid 7e5d5d56-80a1-4db4-b743-972c7f1e1f96
2020-11-17T15:30:20.657-0500 7fc9ccf37180 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
2020-11-17T15:30:20.657-0500 7fc9ccf37180 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
2020-11-17T15:30:20.657-0500 7fc9ccf37180 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory
2020-11-17T15:30:20.661-0500 7fc9ccf37180 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid
Starting SPDK v19.07-pre / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: nvme-device-manager --no-shconf -c 0x1 -m 2048 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1260976 ]
EAL: VFIO support initialized
EAL: DPDK is running on a NUMA system, but is compiled without NUMA support.
EAL: This will have adverse consequences for performance and usability.
EAL: Please use --legacy-mem option, or recompile with NUMA support.
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
Failed to initialize DPDK
EAL: Couldn't get fd on hugepage file
nvme.c: 337:nvme_driver_init: *ERROR*: primary process failed to reserve memory
nvme.c: 624:spdk_nvme_probe: *ERROR*: Create probe context failed
2020-11-17T15:30:20.729-0500 7fc9cc6da700 -1 bdev operator() device probe nvme failed
2020-11-17T15:30:20.729-0500 7fc9ccf37180 -1 bdev() open failed to get nvme device with transport address 0000:04:00.0
2020-11-17T15:30:20.729-0500 7fc9ccf37180 -1 bluestore(/var/lib/ceph/osd/ceph-2) mkfs failed, (1) Operation not permitted
2020-11-17T15:30:20.729-0500 7fc9ccf37180 -1 OSD::mkfs: ObjectStore::mkfs failed with error (1) Operation not permitted
2020-11-17T15:30:20.729-0500 7fc9ccf37180 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-2: (1) Operation not permitted

To build I simply cloned the repo and added the following cmake flag then ran these commands from a debian10 LXC container:

git clone git://git.proxmox.com/git/ceph.git
apt install git rsync dgit quilt -y
cd ceph
make all

PATCH:

root@build:~/ceph# git format-patch -1 origin/master ./ceph/debian/rules
0001-import-15.2.1-Octopus-source.patch
root@build:~/ceph# cat 0001-import-15.2.1-Octopus-source.patch
From 801d13919e6e45e802ad11351165f68b8a7a3869 Mon Sep 17 00:00:00 2001
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
Date: Wed, 15 Apr 2020 13:24:17 +0200
Subject: [PATCH] import 15.2.1 Octopus source

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 ceph/debian/rules | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ceph/debian/rules b/ceph/debian/rules
index 84529306b..8f2e9afa4 100755
--- a/ceph/debian/rules
+++ b/ceph/debian/rules
@@ -23,6 +23,7 @@ extraopts += -DWITH_PYTHON3=3
 extraopts += -DWITH_CEPHFS_JAVA=ON
 extraopts += -DWITH_CEPHFS_SHELL=ON
 extraopts += -DWITH_SYSTEMD=ON -DCEPH_SYSTEMD_ENV_DIR=/etc/default
+extraopts += -DWITH_GRAFANA=ON
 # assumes that ceph is exmpt from multiarch support, so we override the libdir.
 extraopts += -DCMAKE_INSTALL_LIBDIR=/usr/lib
 extraopts += -DCMAKE_INSTALL_LIBEXECDIR=/usr/lib
@@ -65,6 +66,8 @@ override_dh_auto_install:
 
        install -m 755 src/cephadm/cephadm $(DESTDIR)/usr/sbin/cephadm
 
+       install -m 644 -D monitoring/prometheus/alerts/ceph_default_alerts.yml $(DESTDIR)/etc/prometheus/ceph/ceph_default_alerts.yml
+
 # doc/changelog is a directory, which confuses dh_installchangelogs
 override_dh_installchangelogs:
        dh_installchangelogs --exclude doc/changelog
--
2.20.1

 

[Alwin]

Being that it isn't supported by Proxmox that is fine for my needs, but is there a way to include it in the build for unsupported usage?

No, as you see yourself. You need to compile SPDK/DPDK and Ceph yourself for the version(s) you will use. This is far away from a generally working solution.

 

[Trilom]

We leave the defaults, otherwise there are patches in the patches directory.

Given this is the case, then you are right that makes sense. My assumption was that ceph provides this by default and it was explicitly removed however the path to setting this up is very loose and this piece of information was assumed by me. I understand that even if I were to pull ceph from their packages it wouldn't have SPDK included because it's off by default therefore it makes sense not to include it with your packaging.

If anyone else stumbles upon this then this allows me to understand more why people want to deploy crimson-osd, for me I didn't understand the differences and considered them both to have SPDK included.

Thank you for answering my questions and providing support!