pveceph stop and pveceph purge

[breakaway9000]

So I completely messed up my ceph config in my test environment.

I read that running "pveceph stop" then "pveceph purge" will remove all config & data. So I did.

Then I went to the Ceph guide (https://pve.proxmox.com/wiki/Ceph_Server#Ceph_on_Proxmox_VE_5.1) and began creating my cluster again.

I ran "pveceph init --network 10.10.10.0/24" then "pveceph createmon" but then got this:

creating /etc/pve/priv/ceph.client.admin.keyring
monmaptool: monmap file /tmp/monmap
monmaptool: generated fsid 4b9a65e6-1eb7-43fc-b44f-1a7b4d0e7881
epoch 0
fsid 4b9a65e6-1eb7-43fc-b44f-1a7b4d0e7881
last_changed 2018-03-17 16:46:59.543608
created 2018-03-17 16:46:59.543608
0: 10.10.10.2:6789/0 mon.node2
monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
Job for ceph-mon@node2.service failed because the control process exited with error code.
See "
node2.service" and "journalctl -xe" for details.
command '/bin/systemctl start ceph-mon@node2' failed: exit code 1

The output of systemctl status is below.

 ceph-mon@node2.service - Ceph cluster monitor daemon
   Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
           └─ceph-after-pve-cluster.conf
   Active: failed (Result: exit-code) since Sat 2018-03-17 16:37:47 NZDT; 12min ago
  Process: 248197 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id node22 --setuser ceph --setgroup ceph (code=exited, status=1/
 Main PID: 248197 (code=exited, status=1/FAILURE)

Mar 17 16:37:47 node2 systemd[1]: ceph-mon@node2.service: Start request repeated too quickly.
Mar 17 16:37:47 node2 systemd[1]: Failed to start Ceph cluster monitor daemon.
Mar 17 16:37:47 node2 systemd[1]: ceph-mon@node2.service: Unit entered failed state.
Mar 17 16:37:47 node2 systemd[1]: ceph-mon@node2.service: Failed with result 'exit-code'.
Mar 17 16:38:46 node2 systemd[1]: ceph-mon@node2.service: Start request repeated too quickly.
Mar 17 16:38:46 node2 systemd[1]: Failed to start Ceph cluster monitor daemon.
Mar 17 16:38:46 node2 systemd[1]: ceph-mon@node2.service: Failed with result 'exit-code'.
Mar 17 16:46:59 node2 systemd[1]: ceph-mon@node2.service: Start request repeated too quickly.
Mar 17 16:46:59 node2 systemd[1]: Failed to start Ceph cluster monitor daemon.
Mar 17 16:46:59 node2 systemd[1]: ceph-mon@node2.service: Failed with result 'exit-code'.

What is going on here? It looks like the "purge" command didn't really purge everything?

 

[breakaway9000]

Alright... I've managed to get the monitors and managers created (using ceph luminous)

But when I go to view OSDs, I don't see my host names in there like I did earlier; I only see "default".

My crush map doesn't have any of my hosts in it either:

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
root default {
    id -1        # do not change unnecessarily
    # weight 0.000
    alg straw2
    hash 0    # rjenkins1
}

# rules
rule replicated_rule {
    id 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type host
    step emit
}

# end crush map

I'd rather not edit the crush map by hand; What do I need to do to get Proxmox/Ceph to re-build the ceph crush map with "safe" defaults?

 

[Ken Woods]

Reinstall.

You're going down a rabbit hole that *might* get you what you want, but it'd be faster to reinstall and start over.

 

[breakaway9000]

I want to figure out what went wrong here

The whole reason I'm here now is that I had a working ceph setup, I deleted it but when I went to re-add the OSDs, one of the OSDs just wouldn't show up.

Who/what is responsible for creating the initial CRUSH map?

 

[Ken Woods]

so....what happens when you start that OSD?

 

[breakaway9000]

Nothing, the OSD is running just fine.

When I list all my ceph OSDs, the OSD is shown at the bottom of the list; it has no relation to anything else in the cluster.

In fact it’s so broken now that if I go into ceph cluster view, I can’t even see my hosts in there. It just shows one node- “default”. A healthy system should show all your nodes in there.

The CRUSH map doesn’t show my nodes either.

I’m sure invoking whatever it is builds the initial CEPH will fix it - that’s why I thought to run pveceph stop & pveceph purge but all I’ve succeeded in doing is breaking it even more.

 

[Ken Woods]

Yeah...at this point, I'd reinstall.
This isn't in production.......right?

 

[Alwin]

You deleted your monitor store, hence not auth keys are left. And while the OSDs are starting, they don't have the right key to authenticate. Check the logs /var/log/ceph, you should see entries matching that. To get your OSDs back into the cluster, you need to create the proper auth permissions and get the key. This key can then be added to the OSD, after a restart of the OSD, it should show up again on the map. You may need to edit the crush map location of the OSD.

http://docs.ceph.com/docs/master/rados/configuration/auth-config-ref/
http://docs.ceph.com/docs/master/rados/operations/user-management/
http://docs.ceph.com/docs/master/architecture/

 

[breakaway9000]

Hi Alwin, I have completely disabled ceph authentication for now for troubleshooting. Our Ceph network is also isolated (its a daisy chained network) an we're looking for max performance. I followed the instructions at the end of this howTo https://pve.proxmox.com/wiki/Ceph_Server

But when I add a OSD, the process completes successfully but it's still not showing up in the "ceph > OSD" in the Web GUI. Here's what I see in my Ceph > OSD page on the Web GUI:

Note how it does not show any of my hosts? The "base" config should show the hosts at least shouldn't it?

Before taking the above screenshot I had just completed adding a OSD to the 1st host in the cluster. Below is the log from that.

create OSD on /dev/sdb (bluestore)
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/sdb1              isize=2048   agcount=4, agsize=6400 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=25600, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=1608, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
TASK OK

When I use ceph osd crush tree, here's what it shows:

ID CLASS WEIGHT TYPE NAME
-1            0 root default

I checked "/var/log/ceph" and there is nothing in there showing anything bad - ceph-mgr.nodename.log just shows

2018-03-26 17:07:11.989335 7f82a1931700  1 mgr send_beacon standby
2018-03-26 17:07:13.989790 7f82a1931700  1 mgr send_beacon standby
2018-03-26 17:07:15.990162 7f82a1931700  1 mgr send_beacon standby
2018-03-26 17:07:17.990546 7f82a1931700  1 mgr send_beacon standby

ceph.log itsef just shows

cluster [INF] Active manager daemon node4 restarted
cluster [INF] Activating manager daemon node4
mon.node2 mon.0 10.10.10.2:6789/0 17 : cluster [INF] Manager daemon node4 is now available
mon.node2 mon.0 10.10.10.2:6789/0 64 : cluster [INF] overall HEALTH_OK

What should be my next steps? I just want to get this back to a "clean slate" stage.

 

[Alwin]

Clean slate, as in new installation? Then you purge any ceph configs and delete (zap & dd) the OSDs.

 

[breakaway9000]

I thought that’s what pveceph stop and pveceph purge are supposed to do - but I digress.

So after I run those 2 commands - what don I need to do to well and truely purge ALL the config? OSDs, Crush maps, data on disks - it can all go. I’m happy to start setup from scratch.

 

[Alwin]

If you purged the packages, then there should be no config left. The only thing, to do as an extra setp, is to run dd on the first 200 MB, as a sgdisk zap leaves some leftovers and your OSD will not be added.

 

[breakaway9000]

I didn't purge any packages, because I don't know which packages to purge. I installed these packages by typing "pveceph install --version luminous" This "pveceph" command is a black box.

Is it possible to get a list of things this command does?

The doc page for this command (https://pve.proxmox.com/pve-docs/pveceph.1.html) even says

pveceph purge
Destroy ceph related data and configuration files
​

But clearly if it leaves all that stuff above behind, then it doesn't work as intended.

Anyway, we are making some progress now.

I have managed to re-add my hosts into the cluster by doing the following

ceph osd crush add-bucket node2 host
ceph osd crush move node2 root=default

After repeating for the rest of the hosts in the cluster, I can now correctly see these devices in the proxmox "Ceph > OSD" page on the Web UI.

But effectively what I am doing is adding hosts, OSDs etc manually now... I don't want to do this, I want to maintain compatibility with whatever the proxmox ceph/gui commands are doing. What can I do to figure out why this is broken?

The added OSDs don't show up either (see output of pveceph create osd below)

# pveceph createosd /dev/sdc
create OSD on /dev/sdc (bluestore)
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/sdc1              isize=2048   agcount=4, agsize=6400 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=25600, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=1608, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.

After doing this, OSDs are not showing up in the proxmox web gui, nor are they showing up in the output of ceph osd stat.

# ceph osd stat
0 osds: 0 up, 0 in
flags noout

None of the log files in /var/log/ceph are showing any errors.

Here are all my log files, if someone could look through and see if anything looks out of place I would really appreciate it.

ceph.log:

2018-03-28 01:23:25.582370 mon.host2 mon.0 10.10.10.2:6789/0 1 : cluster [INF] mon.host2 calling monitor election
2018-03-28 01:23:25.593082 mon.host2 mon.0 10.10.10.2:6789/0 2 : cluster [INF] mon.host2 is new leader, mons host2,host3,host4,host5 in quorum (ranks 0,1,2,3)
2018-03-28 01:23:25.687169 mon.host2 mon.0 10.10.10.2:6789/0 7 : cluster [WRN] overall HEALTH_WARN noout flag(s) set

ceph-mgr.log

2018-03-28 01:23:25.331129 7fb34841f6c0  0 set uid:gid to 64045:64045 (ceph:ceph)
2018-03-28 01:23:25.331144 7fb34841f6c0  0 ceph version 12.2.4 (4832b6f0acade977670a37c20ff5dbe69e727416) luminous (stable), process (unknown), pid 978669
2018-03-28 01:23:25.333540 7fb34841f6c0  0 pidfile_write: ignore empty --pid-file
2018-03-28 01:23:25.338899 7fb34841f6c0  1 mgr send_beacon standby
2018-03-28 01:23:25.348273 7fb33e9f6700  1 mgr init Loading python module 'balancer'
2018-03-28 01:23:25.375081 7fb33e9f6700  1 mgr init Loading python module 'restful'
2018-03-28 01:23:25.646243 7fb33e9f6700  1 mgr init Loading python module 'status'
2018-03-28 01:23:27.339370 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:29.340050 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:31.340486 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:33.340873 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:35.341264 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:37.341689 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:39.342169 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:41.342653 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:43.343165 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:45.343631 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:47.344054 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:49.344528 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:51.345028 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:53.345463 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:55.345926 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:57.346344 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:23:59.346827 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:24:01.347223 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:24:03.347715 7fb33b9f0700  1 mgr send_beacon standby
2018-03-28 01:24:05.348220 7fb33b9f0700  1 mgr send_beacon standby

ceph-mon.log

2018-03-28 01:23:25.558420 7f3ce17e7f80  0 set uid:gid to 64045:64045 (ceph:ceph)
2018-03-28 01:23:25.558438 7f3ce17e7f80  0 ceph version 12.2.4 (4832b6f0acade977670a37c20ff5dbe69e727416) luminous (stable), process (unknown), pid 978670
2018-03-28 01:23:25.558496 7f3ce17e7f80  0 pidfile_write: ignore empty --pid-file
2018-03-28 01:23:25.567522 7f3ce17e7f80  0 load: jerasure load: lrc load: isa 
2018-03-28 01:23:25.567820 7f3ce17e7f80  0  set rocksdb option compression = kNoCompression
2018-03-28 01:23:25.567830 7f3ce17e7f80  0  set rocksdb option write_buffer_size = 33554432
2018-03-28 01:23:25.567854 7f3ce17e7f80  0  set rocksdb option compression = kNoCompression
2018-03-28 01:23:25.567856 7f3ce17e7f80  0  set rocksdb option write_buffer_size = 33554432
2018-03-28 01:23:25.568000 7f3ce17e7f80  4 rocksdb: RocksDB version: 5.4.0

2018-03-28 01:23:25.568020 7f3ce17e7f80  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2018-03-28 01:23:25.568021 7f3ce17e7f80  4 rocksdb: Compile date Feb 28 2018
2018-03-28 01:23:25.568023 7f3ce17e7f80  4 rocksdb: DB SUMMARY

2018-03-28 01:23:25.568064 7f3ce17e7f80  4 rocksdb: CURRENT file:  CURRENT

2018-03-28 01:23:25.568068 7f3ce17e7f80  4 rocksdb: IDENTITY file:  IDENTITY

2018-03-28 01:23:25.568072 7f3ce17e7f80  4 rocksdb: MANIFEST file:  MANIFEST-000098 size: 326 Bytes

2018-03-28 01:23:25.568074 7f3ce17e7f80  4 rocksdb: SST files in /var/lib/ceph/mon/ceph-host2/store.db dir, Total Num: 1, files: 000104.sst 

2018-03-28 01:23:25.568076 7f3ce17e7f80  4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-host2/store.db: 000102.log size: 3976 ; 

2018-03-28 01:23:25.568077 7f3ce17e7f80  4 rocksdb:                         Options.error_if_exists: 0
2018-03-28 01:23:25.568078 7f3ce17e7f80  4 rocksdb:                       Options.create_if_missing: 0
2018-03-28 01:23:25.568079 7f3ce17e7f80  4 rocksdb:                         Options.paranoid_checks: 1
2018-03-28 01:23:25.568079 7f3ce17e7f80  4 rocksdb:                                     Options.env: 0x55998f77cc00
2018-03-28 01:23:25.568080 7f3ce17e7f80  4 rocksdb:                                Options.info_log: 0x5599913b6e40
2018-03-28 01:23:25.568081 7f3ce17e7f80  4 rocksdb:                          Options.max_open_files: -1
2018-03-28 01:23:25.568082 7f3ce17e7f80  4 rocksdb:                Options.max_file_opening_threads: 16
2018-03-28 01:23:25.568083 7f3ce17e7f80  4 rocksdb:                               Options.use_fsync: 0
2018-03-28 01:23:25.568084 7f3ce17e7f80  4 rocksdb:                       Options.max_log_file_size: 0
2018-03-28 01:23:25.568085 7f3ce17e7f80  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
2018-03-28 01:23:25.568086 7f3ce17e7f80  4 rocksdb:                   Options.log_file_time_to_roll: 0
2018-03-28 01:23:25.568086 7f3ce17e7f80  4 rocksdb:                       Options.keep_log_file_num: 1000
2018-03-28 01:23:25.568087 7f3ce17e7f80  4 rocksdb:                    Options.recycle_log_file_num: 0
2018-03-28 01:23:25.568088 7f3ce17e7f80  4 rocksdb:                         Options.allow_fallocate: 1
2018-03-28 01:23:25.568088 7f3ce17e7f80  4 rocksdb:                        Options.allow_mmap_reads: 0
2018-03-28 01:23:25.568089 7f3ce17e7f80  4 rocksdb:                       Options.allow_mmap_writes: 0
2018-03-28 01:23:25.568099 7f3ce17e7f80  4 rocksdb:                        Options.use_direct_reads: 0
2018-03-28 01:23:25.568100 7f3ce17e7f80  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
2018-03-28 01:23:25.568100 7f3ce17e7f80  4 rocksdb:          Options.create_missing_column_families: 0
2018-03-28 01:23:25.568101 7f3ce17e7f80  4 rocksdb:                              Options.db_log_dir: 
2018-03-28 01:23:25.568102 7f3ce17e7f80  4 rocksdb:                                 Options.wal_dir: /var/lib/ceph/mon/ceph-host2/store.db
2018-03-28 01:23:25.568103 7f3ce17e7f80  4 rocksdb:                Options.table_cache_numshardbits: 6
2018-03-28 01:23:25.568104 7f3ce17e7f80  4 rocksdb:                      Options.max_subcompactions: 1
2018-03-28 01:23:25.568105 7f3ce17e7f80  4 rocksdb:                  Options.max_background_flushes: 1
2018-03-28 01:23:25.568106 7f3ce17e7f80  4 rocksdb:                         Options.WAL_ttl_seconds: 0
2018-03-28 01:23:25.568107 7f3ce17e7f80  4 rocksdb:                       Options.WAL_size_limit_MB: 0
2018-03-28 01:23:25.568108 7f3ce17e7f80  4 rocksdb:             Options.manifest_preallocation_size: 4194304
2018-03-28 01:23:25.568108 7f3ce17e7f80  4 rocksdb:                     Options.is_fd_close_on_exec: 1
2018-03-28 01:23:25.568109 7f3ce17e7f80  4 rocksdb:                   Options.advise_random_on_open: 1
2018-03-28 01:23:25.568110 7f3ce17e7f80  4 rocksdb:                    Options.db_write_buffer_size: 0
2018-03-28 01:23:25.568110 7f3ce17e7f80  4 rocksdb:         Options.access_hint_on_compaction_start: 1
2018-03-28 01:23:25.568111 7f3ce17e7f80  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 0
2018-03-28 01:23:25.568111 7f3ce17e7f80  4 rocksdb:               Options.compaction_readahead_size: 0
2018-03-28 01:23:25.568112 7f3ce17e7f80  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
2018-03-28 01:23:25.568112 7f3ce17e7f80  4 rocksdb:           Options.writable_file_max_buffer_size: 1048576
2018-03-28 01:23:25.568113 7f3ce17e7f80  4 rocksdb:                      Options.use_adaptive_mutex: 0
2018-03-28 01:23:25.568113 7f3ce17e7f80  4 rocksdb:                            Options.rate_limiter: (nil)
2018-03-28 01:23:25.568114 7f3ce17e7f80  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
2018-03-28 01:23:25.568115 7f3ce17e7f80  4 rocksdb:                          Options.bytes_per_sync: 0
2018-03-28 01:23:25.568116 7f3ce17e7f80  4 rocksdb:                      Options.wal_bytes_per_sync: 0
2018-03-28 01:23:25.568116 7f3ce17e7f80  4 rocksdb:                       Options.wal_recovery_mode: 2
2018-03-28 01:23:25.568117 7f3ce17e7f80  4 rocksdb:                  Options.enable_thread_tracking: 0
2018-03-28 01:23:25.568117 7f3ce17e7f80  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
2018-03-28 01:23:25.568118 7f3ce17e7f80  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
2018-03-28 01:23:25.568118 7f3ce17e7f80  4 rocksdb:             Options.write_thread_max_yield_usec: 100
2018-03-28 01:23:25.568119 7f3ce17e7f80  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
2018-03-28 01:23:25.568119 7f3ce17e7f80  4 rocksdb:                               Options.row_cache: None
2018-03-28 01:23:25.568120 7f3ce17e7f80  4 rocksdb:                              Options.wal_filter: None
2018-03-28 01:23:25.568121 7f3ce17e7f80  4 rocksdb:             Options.avoid_flush_during_recovery: 0
2018-03-28 01:23:25.568121 7f3ce17e7f80  4 rocksdb:             Options.base_background_compactions: 1
2018-03-28 01:23:25.568122 7f3ce17e7f80  4 rocksdb:             Options.max_background_compactions: 1
2018-03-28 01:23:25.568122 7f3ce17e7f80  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2018-03-28 01:23:25.568123 7f3ce17e7f80  4 rocksdb:             Options.delayed_write_rate : 16777216
2018-03-28 01:23:25.568124 7f3ce17e7f80  4 rocksdb:             Options.max_total_wal_size: 0
2018-03-28 01:23:25.568124 7f3ce17e7f80  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
2018-03-28 01:23:25.568125 7f3ce17e7f80  4 rocksdb:                   Options.stats_dump_period_sec: 600
2018-03-28 01:23:25.568126 7f3ce17e7f80  4 rocksdb: Compression algorithms supported:
2018-03-28 01:23:25.568126 7f3ce17e7f80  4 rocksdb:     Snappy supported: 0
2018-03-28 01:23:25.568127 7f3ce17e7f80  4 rocksdb:     Zlib supported: 0
2018-03-28 01:23:25.568128 7f3ce17e7f80  4 rocksdb:     Bzip supported: 0
2018-03-28 01:23:25.568128 7f3ce17e7f80  4 rocksdb:     LZ4 supported: 0
2018-03-28 01:23:25.568129 7f3ce17e7f80  4 rocksdb:     ZSTD supported: 0
2018-03-28 01:23:25.568129 7f3ce17e7f80  4 rocksdb: Fast CRC32 supported: 0
2018-03-28 01:23:25.568550 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000098

2018-03-28 01:23:25.568633 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

2018-03-28 01:23:25.568639 7f3ce17e7f80  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
2018-03-28 01:23:25.568640 7f3ce17e7f80  4 rocksdb:           Options.merge_operator: 
2018-03-28 01:23:25.568641 7f3ce17e7f80  4 rocksdb:        Options.compaction_filter: None
2018-03-28 01:23:25.568642 7f3ce17e7f80  4 rocksdb:        Options.compaction_filter_factory: None
2018-03-28 01:23:25.568643 7f3ce17e7f80  4 rocksdb:         Options.memtable_factory: SkipListFactory
2018-03-28 01:23:25.568644 7f3ce17e7f80  4 rocksdb:            Options.table_factory: BlockBasedTable
2018-03-28 01:23:25.568661 7f3ce17e7f80  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55999107e138)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x559991390550
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 134217728
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  format_version: 2

2018-03-28 01:23:25.568666 7f3ce17e7f80  4 rocksdb:        Options.write_buffer_size: 33554432
2018-03-28 01:23:25.568667 7f3ce17e7f80  4 rocksdb:  Options.max_write_buffer_number: 2
2018-03-28 01:23:25.568669 7f3ce17e7f80  4 rocksdb:          Options.compression: NoCompression
2018-03-28 01:23:25.568671 7f3ce17e7f80  4 rocksdb:                  Options.bottommost_compression: Disabled
2018-03-28 01:23:25.568671 7f3ce17e7f80  4 rocksdb:       Options.prefix_extractor: nullptr
2018-03-28 01:23:25.568673 7f3ce17e7f80  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
2018-03-28 01:23:25.568674 7f3ce17e7f80  4 rocksdb:             Options.num_levels: 7
2018-03-28 01:23:25.568674 7f3ce17e7f80  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
2018-03-28 01:23:25.568675 7f3ce17e7f80  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
2018-03-28 01:23:25.568676 7f3ce17e7f80  4 rocksdb:            Options.compression_opts.window_bits: -14
2018-03-28 01:23:25.568676 7f3ce17e7f80  4 rocksdb:                  Options.compression_opts.level: -1
2018-03-28 01:23:25.568677 7f3ce17e7f80  4 rocksdb:               Options.compression_opts.strategy: 0
2018-03-28 01:23:25.568677 7f3ce17e7f80  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
2018-03-28 01:23:25.568678 7f3ce17e7f80  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
2018-03-28 01:23:25.568678 7f3ce17e7f80  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2018-03-28 01:23:25.568679 7f3ce17e7f80  4 rocksdb:              Options.level0_stop_writes_trigger: 36
2018-03-28 01:23:25.568681 7f3ce17e7f80  4 rocksdb:                   Options.target_file_size_base: 67108864
2018-03-28 01:23:25.568682 7f3ce17e7f80  4 rocksdb:             Options.target_file_size_multiplier: 1
2018-03-28 01:23:25.568683 7f3ce17e7f80  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
2018-03-28 01:23:25.568684 7f3ce17e7f80  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2018-03-28 01:23:25.568684 7f3ce17e7f80  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
2018-03-28 01:23:25.568686 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2018-03-28 01:23:25.568688 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2018-03-28 01:23:25.568689 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2018-03-28 01:23:25.568690 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2018-03-28 01:23:25.568691 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2018-03-28 01:23:25.568692 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2018-03-28 01:23:25.568693 7f3ce17e7f80  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2018-03-28 01:23:25.568694 7f3ce17e7f80  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
2018-03-28 01:23:25.568695 7f3ce17e7f80  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
2018-03-28 01:23:25.568696 7f3ce17e7f80  4 rocksdb:                        Options.arena_block_size: 4194304
2018-03-28 01:23:25.568697 7f3ce17e7f80  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
2018-03-28 01:23:25.568698 7f3ce17e7f80  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
2018-03-28 01:23:25.568699 7f3ce17e7f80  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
2018-03-28 01:23:25.568699 7f3ce17e7f80  4 rocksdb:                Options.disable_auto_compactions: 0
2018-03-28 01:23:25.568700 7f3ce17e7f80  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
2018-03-28 01:23:25.568702 7f3ce17e7f80  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
2018-03-28 01:23:25.568702 7f3ce17e7f80  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
2018-03-28 01:23:25.568703 7f3ce17e7f80  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2018-03-28 01:23:25.568704 7f3ce17e7f80  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2018-03-28 01:23:25.568704 7f3ce17e7f80  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2018-03-28 01:23:25.568705 7f3ce17e7f80  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2018-03-28 01:23:25.568706 7f3ce17e7f80  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2018-03-28 01:23:25.568706 7f3ce17e7f80  4 rocksdb:                   Options.table_properties_collectors: 
2018-03-28 01:23:25.568707 7f3ce17e7f80  4 rocksdb:                   Options.inplace_update_support: 0
2018-03-28 01:23:25.568716 7f3ce17e7f80  4 rocksdb:                 Options.inplace_update_num_locks: 10000
2018-03-28 01:23:25.568717 7f3ce17e7f80  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
2018-03-28 01:23:25.568718 7f3ce17e7f80  4 rocksdb:   Options.memtable_huge_page_size: 0
2018-03-28 01:23:25.568719 7f3ce17e7f80  4 rocksdb:                           Options.bloom_locality: 0
2018-03-28 01:23:25.568719 7f3ce17e7f80  4 rocksdb:                    Options.max_successive_merges: 0
2018-03-28 01:23:25.568720 7f3ce17e7f80  4 rocksdb:                Options.optimize_filters_for_hits: 0
2018-03-28 01:23:25.568720 7f3ce17e7f80  4 rocksdb:                Options.paranoid_file_checks: 0
2018-03-28 01:23:25.568721 7f3ce17e7f80  4 rocksdb:                Options.force_consistency_checks: 0
2018-03-28 01:23:25.568722 7f3ce17e7f80  4 rocksdb:                Options.report_bg_io_stats: 0
2018-03-28 01:23:25.570101 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-host2/store.db/MANIFEST-000098 succeeded,manifest_file_number is 98, next_file_number is 106, last_sequence is 29743, log_number is 0,prev_log_number is 0,max_column_family is 0

2018-03-28 01:23:25.570110 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 102

2018-03-28 01:23:25.570500 7f3ce17e7f80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1522153405570490, "job": 1, "event": "recovery_started", "log_files": [102]}
2018-03-28 01:23:25.570506 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #102 mode 2
2018-03-28 01:23:25.572684 7f3ce17e7f80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1522153405572675, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 106, "file_size": 4755, "table_properties": {"data_size": 3784, "index_size": 28, "filter_size": 33, "raw_key_size": 211, "raw_average_key_size": 23, "raw_value_size": 3586, "raw_average_value_size": 398, "num_data_blocks": 1, "num_entries": 9, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
2018-03-28 01:23:25.572710 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 107

2018-03-28 01:23:25.574432 7f3ce17e7f80  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1522153405574429, "job": 1, "event": "recovery_finished"}
2018-03-28 01:23:25.577462 7f3ce17e7f80  4 rocksdb: [/home/builder/source/ceph-12.2.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5599914ac000
2018-03-28 01:23:25.578410 7f3ce17e7f80  0 starting mon.host2 rank 0 at public addr 10.10.10.2:6789/0 at bind addr 10.10.10.2:6789/0 mon_data /var/lib/ceph/mon/ceph-host2 fsid 5526ca13-287b-46ff-a302-9b1853a5fb25
2018-03-28 01:23:25.578533 7f3ce17e7f80  0 starting mon.host2 rank 0 at 10.10.10.2:6789/0 mon_data /var/lib/ceph/mon/ceph-host2 fsid 5526ca13-287b-46ff-a302-9b1853a5fb25
2018-03-28 01:23:25.579093 7f3ce17e7f80  1 mon.host2@-1(probing) e6 preinit fsid 5526ca13-287b-46ff-a302-9b1853a5fb25
2018-03-28 01:23:25.579236 7f3ce17e7f80  1 mon.host2@-1(probing).mds e0 Unable to load 'last_metadata'
2018-03-28 01:23:25.579291 7f3ce17e7f80  1 mon.host2@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
2018-03-28 01:23:25.579314 7f3ce17e7f80  1 mon.host2@-1(probing).pg v0 on_upgrade discarding in-core PGMap
2018-03-28 01:23:25.579412 7f3ce17e7f80  0 mon.host2@-1(probing).mds e1 print_map
e1
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
legacy client fscid: -1

No filesystems configured

2018-03-28 01:23:25.579543 7f3ce17e7f80  0 mon.host2@-1(probing).osd e18 crush map has features 288514050185494528, adjusting msgr requires
2018-03-28 01:23:25.579550 7f3ce17e7f80  0 mon.host2@-1(probing).osd e18 crush map has features 288514050185494528, adjusting msgr requires
2018-03-28 01:23:25.579552 7f3ce17e7f80  0 mon.host2@-1(probing).osd e18 crush map has features 1009089990564790272, adjusting msgr requires
2018-03-28 01:23:25.579553 7f3ce17e7f80  0 mon.host2@-1(probing).osd e18 crush map has features 288514050185494528, adjusting msgr requires
2018-03-28 01:23:25.579765 7f3ce17e7f80  1 mon.host2@-1(probing).paxosservice(auth 251..312) refresh upgraded, format 0 -> 2
2018-03-28 01:23:25.581369 7f3cd4246700  0 -- 10.10.10.2:0/978670 >> 10.10.10.4:6800/2156616 conn(0x5599915a9000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2018-03-28 01:23:25.581400 7f3ce17e7f80  0 mon.host2@-1(probing) e6  my rank is now 0 (was -1)
2018-03-28 01:23:25.581463 7f3cd4246700  0 -- 10.10.10.2:0/978670 >> 10.10.10.4:6800/2156616 conn(0x5599915a9000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2018-03-28 01:23:25.582265 7f3cd3a45700  0 -- 10.10.10.2:6789/0 >> 10.10.10.4:6789/0 conn(0x5599915d2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=227 cs=1 l=0).process missed message?  skipped from seq 0 to 188115002
2018-03-28 01:23:25.582312 7f3cd3244700  0 -- 10.10.10.2:6789/0 >> 10.10.10.3:6789/0 conn(0x5599915aa800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=242 cs=1 l=0).process missed message?  skipped from seq 0 to 169582390
2018-03-28 01:23:25.582366 7f3cd724c700  0 log_channel(cluster) log [INF] : mon.host2 calling monitor election
2018-03-28 01:23:25.582455 7f3cd724c700  1 mon.host2@0(electing).elector(130) init, last seen epoch 130
2018-03-28 01:23:25.582469 7f3cd4246700  0 -- 10.10.10.2:6789/0 >> 10.10.10.5:6789/0 conn(0x5599915d3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=253 cs=1 l=0).process missed message?  skipped from seq 0 to 637204290
2018-03-28 01:23:25.593078 7f3cd724c700  0 log_channel(cluster) log [INF] : mon.host2 is new leader, mons host2,host3,host4,host5 in quorum (ranks 0,1,2,3)
2018-03-28 01:23:25.686850 7f3cd724c700  0 log_channel(cluster) log [DBG] : monmap e6: 4 mons at {host2=10.10.10.2:6789/0,host3=10.10.10.3:6789/0,host4=10.10.10.4:6789/0,host5=10.10.10.5:6789/0}
2018-03-28 01:23:25.686893 7f3cd724c700  0 log_channel(cluster) log [DBG] : fsmap 
2018-03-28 01:23:25.686964 7f3cd724c700  0 log_channel(cluster) log [DBG] : osdmap e18: 0 total, 0 up, 0 in
2018-03-28 01:23:25.687040 7f3cd724c700  0 log_channel(cluster) log [DBG] : mgrmap e21: host4(active), standbys: host3, host5, host2
2018-03-28 01:23:25.687168 7f3cd724c700  0 log_channel(cluster) log [WRN] : overall HEALTH_WARN noout flag(s) set
2018-03-28 01:23:25.711520 7f3cd724c700  0 log_channel(cluster) log [DBG] : Standby manager daemon host2 restarted
2018-03-28 01:23:25.711547 7f3cd724c700  0 log_channel(cluster) log [DBG] : Standby manager daemon host2 started
2018-03-28 01:23:25.769399 7f3cd2a43700  0 log_channel(cluster) log [DBG] : mgrmap e22: host4(active), standbys: host3, host5, host2

 

[breakaway9000]

Ok, I now have this working (mostly). It appears the pveceph create <diskname> command simply aliases to sgdisk -Z <diskname> then ceph-disk prepare --bluestore <diskname> then finally ceph-disk activiate <diskname>

Here are the commands I've used. In this example I'm preparing "sda" for use as an osd.

zgdisk -Z /dev/sda
ceph-disk prepare /dev/sda
ceph-disk activate /dev/sda1

After repating this for all the disks on all of the hosts, the "Ceph > OSD" page is populated correctly. The output of ceph osd crush tree is also correct now. Note - it is normal for "node4" below to not have any osds set up since I haven't added any disks to this chassis just yet.

# ceph osd crush tree
ID CLASS WEIGHT   TYPE NAME
-1       49.12097 root default
-2       16.37366     host node2
 0   hdd  5.45789         osd.0
 1   hdd  5.45789         osd.1
 2   hdd  5.45789         osd.2
-3       16.37366     host node3
 3   hdd  5.45789         osd.3
 4   hdd  5.45789         osd.4
 5   hdd  5.45789         osd.5
-4              0     host node4
-5       16.37366     host node5
 6   hdd  5.45789         osd.6
 7   hdd  5.45789         osd.7
 8   hdd  5.45789         osd.8

 

[Anthony Di Leonardo]

If you purged the packages, then there should be no config left. The only thing, to do as an extra setp, is to run dd on the first 200 MB, as a sgdisk zap leaves some leftovers and your OSD will not be added.

Hi, How would I purge the packages? I am new to proxmox and ceph, I am seeing the same behavior as breakaway9000. I had to remove and add some OSD drives and that is when this started happening. I did the pveceph purge, zap and dd after shutting down the ceph services. When it came back up I could add drives and they would not appear in the osd tree. The only thing I did not do was purge the ceph packages, is there a process or a command to purge ceph luminous packages that I am unaware of?

 

[breakaway9000]

After much searching & trial and error, I don't believe there is any way to actually purge the packages.

The problem is this "pveceph" command is a black box. We have no idea what it is actually doing in the background.

In the end, I removed the OSDs one by one. Here is the procedure to do that:

First, figure out which OSDs and Hosts you have in your cluster. For hosts & OSDs, use this command

ceph osd crush tree

And for mons/mgrs, use this command:

ceph -s

Before you start, prevent the cluster from rebalancing/healing:

ceph osd set noout

Now, remove the OSDs one by one. To do this, first stop all the ceph services by doing

systemctl stop ceph\*.service ceph\*.target

Now, delete the OSDs (substitute <OSDNO> with the OSD ID of course):

# Mark the OSD Out
ceph osd out osd.<OSDNO>
# Stop the service (this shouldn't be needed because we stopped all services above anyways)
systemctl stop ceph-osd@<OSDNO>
# Down the OSD
ceph osd down osd.<OSDNO>
# Remove OSD from CRUSH Map
ceph osd crush rm osd.<OSDNO>
# Remove the OSD
ceph osd rm osd.<OSDNO>
# Finally remove the auth keys since they are no longer needed
ceph auth del osd.<OSDNO>

At this point all your OSDs are gone, but hosts & monitor/manager still remain. Depending on what you are trying to accomplish, it may be enough to stop here and re-add the OSDs (As was my case).

Prep your disks (this must be done for each)

# Wipe out the partition table on the disk, prepping it for Ceph Use
sgdisk -Z /dev/<DEVICENAME>
# Prep the disk for use with Ceph - as a bluestore OSD
ceph-disk prepare --bluestore /dev/<DEVICENAME>
# Activate this newly created disk (OSD) into our cluster
ceph-disk activate /dev/<DEVICENAME>

Finally, start up all ceph services. Everything should be all good now.

systemctl start ceph.target

On some occasions, I found that the "default" tree in the Ceph config in the WebGUI wouldn't show my hosts or OSDs added on that host, despite the ceph -s command showing them. Turns out for some reason they were added by ceph but not actually added to the cluster. To fix this you need to introduce the host to the cluster, then add it to the "default" cluster so it becomes part of the cluster.

# Add the node into the crush map
ceph osd crush add-bucket <NODEHOSTNAME> host
# Move it to the correct location in the hierarchy
ceph osd crush move <NODEHOSTNAME> root=default

If you jump onto the #ceph channel on OFTC using your favourite IRC channel, someone is always around to lend a helping hand.

My advice is to not rely on proxmox built-in ceph management commands and tools because getting support is a real problem (unless you pay for a license).

 

[Anthony Di Leonardo]

After much searching & trial and error, I don't believe there is any way to actually purge the packages.

The problem is this "pveceph" command is a black box. We have no idea what it is actually doing in the background.

In the end, I removed the OSDs one by one. Here is the procedure to do that:

First, figure out which OSDs and Hosts you have in your cluster. For hosts & OSDs, use this command

ceph osd crush tree

And for mons/mgrs, use this command:

ceph -s

Before you start, prevent the cluster from rebalancing/healing:

ceph osd set noout

Now, remove the OSDs one by one. To do this, first stop all the ceph services by doing

systemctl stop ceph\*.service ceph\*.target

Now, delete the OSDs (substitute <OSDNO> with the OSD ID of course):

# Mark the OSD Out
ceph osd out osd.<OSDNO>
# Stop the service (this shouldn't be needed because we stopped all services above anyways)
systemctl stop ceph-osd@<OSDNO>
# Down the OSD
ceph osd down osd.<OSDNO>
# Remove OSD from CRUSH Map
ceph osd crush rm osd.<OSDNO>
# Remove the OSD
ceph osd rm osd.<OSDNO>
# Finally remove the auth keys since they are no longer needed
ceph auth del osd.<OSDNO>

At this point all your OSDs are gone, but hosts & monitor/manager still remain. Depending on what you are trying to accomplish, it may be enough to stop here and re-add the OSDs (As was my case).

Prep your disks (this must be done for each)

# Wipe out the partition table on the disk, prepping it for Ceph Use
sgdisk -Z /dev/<DEVICENAME>
# Prep the disk for use with Ceph - as a bluestore OSD
ceph-disk prepare --bluestore /dev/<DEVICENAME>
# Activate this newly created disk (OSD) into our cluster
ceph-disk activate /dev/<DEVICENAME>

Finally, start up all ceph services. Everything should be all good now.

systemctl start ceph.target

On some occasions, I found that the "default" tree in the Ceph config in the WebGUI wouldn't show my hosts or OSDs added on that host, despite the ceph -s command showing them. Turns out for some reason they were added by ceph but not actually added to the cluster. To fix this you need to introduce the host to the cluster, then add it to the "default" cluster so it becomes part of the cluster.

# Add the node into the crush map
ceph osd crush add-bucket <NODEHOSTNAME> host
# Move it to the correct location in the hierarchy
ceph osd crush move <NODEHOSTNAME> root=default

If you jump onto the #ceph channel on OFTC using your favourite IRC channel, someone is always around to lend a helping hand.

My advice is to not rely on proxmox built-in ceph management commands and tools because getting support is a real problem (unless you pay for a license).

Thanks, your process worked perfect for me on the first node. The second node, for some odd reason was giving me a headache. I kept getting the following error:

root@pve2:/var/lib/ceph/osd# ceph-disk activate /dev/sdb1
got monmap epoch 3
2018-04-14 09:48:14.304225 7fab529dbe00 -1 bluestore(/var/lib/ceph/tmp/mnt.X5fxVt/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.X5fxVt/block fsid 480f8068-2ee4-4e1b-826c-75cb4438bb98 does not match our fsid eacc7ace-4ba4-46a1-bb8a-d3adf9b3c260
2018-04-14 09:48:14.593897 7fab529dbe00 -1 bluestore(/var/lib/ceph/tmp/mnt.X5fxVt) mkfs fsck found fatal error: (5) Input/output error
2018-04-14 09:48:14.593918 7fab529dbe00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2018-04-14 09:48:14.593976 7fab529dbe00 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.X5fxVt: (5) Input/output error
mount_activate: Failed to activate

So I read several blogs on multiple sites in different areas. I ran into this process which suddenly made this error go away.

dd if=/dev/zero of=/dev/sdb bs=4096k count=100

Then..

sgdisk --zap-all --clear --mbrtogpt -g -- /dev/sdb

The third node had no issues whatsoever following your recommended process.

Thanks again, ceph OSD services are now restored.

 

[Alwin]

The problem is this "pveceph" command is a black box. We have no idea what it is actually doing in the background.

Perl code. You find the source for it here -> https://git.proxmox.com/?p=pve-manager.git;a=tree;hb=HEAD

My advice is to not rely on proxmox built-in ceph management commands and tools because getting support is a real problem (unless you pay for a license).

The license is GNU AGPL, v3 and you can find it here -> https://www.gnu.org/licenses/agpl-3.0.en.html

To get support directly by the proxmox devs or if you just want to support the project monetary, then our subscriptions are for you. https://www.proxmox.com/en/proxmox-ve/pricing

But this is not the only way to help out or get support. You can join the community on the IRC #proxmox channel, the proxmox forum or on the mailing lists. Bug/feature reports on our bugzilla (https://bugzilla.proxmox.com/) and patches (https://pve.proxmox.com/wiki/Developer_Documentation) are also welcome.

Hi, How would I purge the packages?

To purge packages on debian/ubuntu you do 'apt purge <package>'. Taken from 'man apt':

Removing a package removes all packaged data, but leaves usually small (modified) user configuration files behind, in case the remove was an accident. Just issuing an installation request for the
accidentally removed package will restore its function as before in that case. On the other hand you can get rid of these leftovers by calling purge even on already removed packages. Note that
this does not affect any data or configuration stored in your home directory.

dd if=/dev/zero of=/dev/sdb bs=4096k count=100

There are leftover pieces when it was an OSD before and a 'sgdisk -Z' isn't removing them.

 

[Anthony Di Leonardo]

To purge packages on debian/ubuntu you do 'apt purge <package>'. Taken from 'man apt':

Is there a way to remove all of the ceph luminous packages from a single command or is this not possible? How would I determine what all of the names of the packages are, or is there a shortcut way to do so?
​

 

[Alwin]

dpkg -l | grep -i ceph

To get a list of packages with the name ceph, also you can go and have a look with apt show ceph, what packages it depends on.