Cannot create jewel OSDs in 4.4.5

A

athompso

Renowned Member
Sep 13, 2013
129
8
83
Running pve-manager/4.4-5/c43015a5 (running kernel: 4.4.35-1-pve), with CEPH jewel installed (via "pveceph install -version jewel"), I find that I cannot create any OSDs.

Digging through journalctl output previously, I saw a "permission denied" error, but of course now I can't find it... it looked like maybe the permissions on the temporary mountpoint were wrong so the CEPH structures couldn't be created.

Same behaviour experienced from CLI (pveceph) and from GUI.
 
Have you run ceph hammer before you install jewel?
 
No, this was a fresh install of CEPH.
I recently completed the PVE cluster, finished migrating all local-storage VMs to NFS, went to create CEPH storage and ran into this problem.
 
Can you send the output of this commands

Code: 
ls -ahl /var/lib/ceph/osd
ls -ahl /etc/ceph/
ls -ahl /var/lib/ceph/
[CODE]
 
Sure...
Code: 
root@pve1:~# ls -ahl /var/lib/ceph/osd
total 8.0K
drwxr-xr-x 2 ceph ceph 4.0K Dec  9 15:03 .
drwxr-x--- 8 ceph ceph 4.0K Dec 30 13:07 ..
root@pve1:~# ls -ahl /etc/ceph/
total 16K
drwxr-xr-x  2 root root 4.0K Jan  1 10:57 .
drwxr-xr-x 98 root root 4.0K Jan  1 09:08 ..
-rw-------  1 ceph ceph  137 Jan  1 10:47 ceph.client.admin.keyring
-rw-r--r--  1 root root   92 Dec  9 15:04 rbdmap
root@pve1:~# ls -ahl /var/lib/ceph/
total 32K
drwxr-x---  8 ceph ceph 4.0K Dec 30 13:07 .
drwxr-xr-x 45 root root 4.0K Dec 30 13:07 ..
drwxr-xr-x  2 ceph ceph 4.0K Jan  1 10:57 bootstrap-mds
drwxr-xr-x  2 ceph ceph 4.0K Jan  1 10:57 bootstrap-osd
drwxr-xr-x  2 ceph ceph 4.0K Dec 30 13:08 bootstrap-rgw
drwxr-xr-x  2 ceph ceph 4.0K Jan  1 10:57 mon
drwxr-xr-x  2 ceph ceph 4.0K Dec  9 15:03 osd
drwxr-xr-x  2 ceph ceph 4.0K Jan  1 10:48 tmp
root@pve1:~#
 
How exactly you create your PVE ceph-cluster because the ceph.config missing?
Have you create an normal PVE cluster?
 
PVE cluster is happy:
Code: 
root@pve1:~# pvecm status
Quorum information
------------------
Date:             Tue Jan  3 06:38:07 2017
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1/212
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   7
Highest expected: 7
Total votes:      7
Quorum:           4 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          4 192.168.158.31 (local)
0x00000002          1 192.168.158.32
0x00000003          1 192.168.158.33
0x00000004          1 192.168.158.34

(yes, the 4 votes for the first node is deliberate)

And I have tried creating CEPH OSDs both using "pveceph createosd" and through the GUI. I've tried both using a separate journal device and not using a separate journal device.
It looks like *most* of the process completes, but not all of it. I have the correct partitions and filesystems on the CEPH target disk and on the SSD journal disk, but no files or directories on them.
I note that I do have to zap the GPT before I can re-attempt creating an OSD on any given target disk, so something went right, up to a point.
Dmesg output shows me the side-effects of partition creation, and I believe they do get mounted for a split second.
 
Missing some details:
first I did
Code: 
pveceph install -version jewel
then
Code: 
pveceph init -pg_bits 14 -size 3
then
Code: 
pveceph createmon
Switching to the GUI, I then created MONs on the other 3 nodes (for a total of 4 MONs in my 4-node cluster).
...all of this worked well up to this point.

using
Code: 
pveceph createosd /dev/sdd
fails.
Using the GUI to do the same thing fails.[/CODE]
 
Last edited:
Matthieu Le Corre said:
hum,

I've got the same issue, it seems that the bootstrap keys were losts.

Check with :
Code: 
 ceph auth list
Click to expand...

I merely get:
Code: 
root@pve1:~# ceph auth list
Error initializing cluster client: Error('error calling conf_read_file: error code 22',)[CODE]

UPDATE: silly me, that was because I was back to a "purged" state. After re-doing the initialization and monitor creation, "ceph auth list" shows me a list of keys as it should.
 
Last edited:
Journalctl output from "pveceph createosd /dev/sdc -journal_dev /dev/sb":
Code: 
Jan 03 07:04:18 pve1 systemd[1]: [/lib/systemd/system/ceph-mon@.service:24] Unknown lvalue 'TasksMax' in section 'Service'
Jan 03 07:04:18 pve1 systemd[1]: [/lib/systemd/system/ceph-osd@.service:18] Unknown lvalue 'TasksMax' in section 'Service'
Jan 03 07:04:18 pve1 pveceph[26062]: <root@pam> end task UPID:pve1:000065CF:00FC0491:586BA152:cephcreatemon:mon.0:root@pam: OK
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: no valid command found; 10 closest matches:
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config set <var> <val> [<val>...]
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: version
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: git_version
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: help
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config show
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: get_command_descriptions
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config get <var>
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: perfcounters_dump
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: 2
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config diff
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: admin_socket: invalid command
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
Jan 03 07:04:18 pve1 ceph-mon[26090]: starting mon.0 rank 0 at 192.168.158.31:6789/0 mon_data /var/lib/ceph/mon/ceph-0 fsid ef3ee364-6a34-47ae-955e-9c393939c4d8
Jan 03 07:04:19 pve1 ceph-create-keys[26088]: INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
Jan 03 07:04:19 pve1 ceph-create-keys[26088]: INFO:ceph-create-keys:Talking to monitor...
Jan 03 07:04:19 pve1 ceph-create-keys[26088]: INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-rgw/ceph.keyring
Jan 03 07:04:19 pve1 ceph-create-keys[26088]: INFO:ceph-create-keys:Talking to monitor...
Jan 03 07:04:47 pve1 pveceph[26238]: <root@pam> starting task UPID:pve1:00006692:00FC0FF1:586BA16F:cephcreateosd:sdd:root@pam:
Jan 03 07:04:47 pve1 kernel: Alternate GPT is invalid, using primary GPT.
Jan 03 07:04:47 pve1 kernel:  sdd:
Jan 03 07:04:47 pve1 kernel: Alternate GPT is invalid, using primary GPT.
Jan 03 07:04:47 pve1 kernel:  sdd:
Jan 03 07:04:49 pve1 kernel:  sdd:
Jan 03 07:04:49 pve1 kernel:  sdb:
Jan 03 07:04:49 pve1 kernel:  sdb:
Jan 03 07:04:51 pve1 kernel:  sdb: sdb1
Jan 03 07:04:51 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdb1...
-- Subject: Unit ceph-disk@dev-sdb1.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ceph-disk@dev-sdb1.service has begun starting up.
Jan 03 07:04:51 pve1 systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
Jan 03 07:04:51 pve1 systemd[1]: Failed to reset devices.list on /system.slice/system-ceph\x2dmon.slice: Invalid argument
Jan 03 07:04:51 pve1 sh[26348]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdb1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7fd9df0b86e0>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
Jan 03 07:04:51 pve1 sh[26348]: command: Running command: /sbin/init --version
Jan 03 07:04:51 pve1 sh[26348]: command: Running command: /sbin/blkid -o udev -p /dev/sdb1
Jan 03 07:04:51 pve1 sh[26348]: command: Running command: /sbin/blkid -o udev -p /dev/sdb1
Jan 03 07:04:51 pve1 sh[26348]: main_trigger: trigger /dev/sdb1 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 1b2a33a9-f3ef-476f-8e7e-09ed1b6e9c1c
Jan 03 07:04:51 pve1 sh[26348]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdb1
Jan 03 07:04:51 pve1 sh[26348]: main_trigger:
Jan 03 07:04:51 pve1 sh[26348]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
Jan 03 07:04:51 pve1 sh[26348]: command: Running command: /sbin/blkid -o udev -p /dev/sdb1
Jan 03 07:04:51 pve1 sh[26348]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdb1
Jan 03 07:04:51 pve1 sh[26348]: get_space_osd_uuid: Journal /dev/sdb1 has OSD UUID 00000000-0000-0000-0000-000000000000
Jan 03 07:04:51 pve1 sh[26348]: main_activate_space: activate: OSD device not present, not starting, yet
Jan 03 07:04:51 pve1 kernel:  sdd:
Jan 03 07:04:51 pve1 systemd[1]: Started Ceph disk activation: /dev/sdb1.
-- Subject: Unit ceph-disk@dev-sdb1.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ceph-disk@dev-sdb1.service has finished starting up.
-- 
-- The start-up result is done.
Jan 03 07:04:52 pve1 kernel:  sdd: sdd1
Jan 03 07:04:54 pve1 kernel: XFS (sdd1): Mounting V4 Filesystem
Jan 03 07:04:54 pve1 kernel: XFS (sdd1): Ending clean mount
Jan 03 07:04:54 pve1 kernel: XFS (sdd1): Unmounting Filesystem
Jan 03 07:04:54 pve1 kernel:  sdd: sdd1
Jan 03 07:04:55 pve1 kernel:  sdd: sdd1
Jan 03 07:04:56 pve1 pveceph[26238]: <root@pam> end task UPID:pve1:00006692:00FC0FF1:586BA16F:cephcreateosd:sdd:root@pam: OK
 
athompso said:
Code: 
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: no valid command found; 10 closest matches:
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config set <var> <val> [<val>...]
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: version
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: git_version
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: help
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config show
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: get_command_descriptions
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config get <var>
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: perfcounters_dump
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: 2
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: config diff
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: admin_socket: invalid command
Jan 03 07:04:18 pve1 ceph-create-keys[26088]: INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
Click to expand...


look like a typo with the ceph-create-keys command, are your bootstrap key owned by ceph:ceph ?
 
Matthieu Le Corre said:
look like a typo with the ceph-create-keys command, are your bootstrap key owned by ceph:ceph ?
Click to expand...

Don't know. Where are they stored? If you're referring to the files that I listed way back in the 5th post to this thread, then yes, they are owned by ceph/ceph.
 
That's not working either:
Code: 
root@pve1:~# cat /proc/partitions 
major minor  #blocks  name
   1        0      65536 ram0
   1        1      65536 ram1
   1        2      65536 ram2
   1        3      65536 ram3
   1        4      65536 ram4
   1        5      65536 ram5
   1        6      65536 ram6
   1        7      65536 ram7
   1        8      65536 ram8
   1        9      65536 ram9
   1       10      65536 ram10
   1       11      65536 ram11
   1       12      65536 ram12
   1       13      65536 ram13
   1       14      65536 ram14
   1       15      65536 ram15
   8        0  143374744 sda
   8        1       1007 sda1
   8        2     130048 sda2
   8        3  143243655 sda3
   8       32  292968750 sdc
   8       48  292968750 sdd
   8       49  292967709 sdd1
   8       64  292968750 sde
   8       80  292968750 sdf
   8       16   58615704 sdb
   8       17    5242880 sdb1
 251        0   35651584 dm-0
 251        1    8388608 dm-1
 251        2      81920 dm-2
 251        3   82427904 dm-3
 251        4   82427904 dm-4
root@pve1:~# ceph-disk prepare --journal-dev /dev/sdb1 /dev/sdc
prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
set_data_partition: incorrect partition UUID: 45b0969e-9b03-4f30-b4c6-b4b80ceff106, expected ['4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', '4fbd7e29-8ae0-4982-bf9d-5a8d867af560', '4fbd7e29-9d25-41b8-afd0-35865ceff05d']
meta-data=/dev/sdb1              isize=2048   agcount=4, agsize=327680 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@pve1:~# journalctl -f &
[1] 8237

root@pve1:~# -- Logs begin at Sun 2017-01-01 09:12:00 CST. --
Jan 03 10:00:08 pve1 pveproxy[22861]: worker 8070 started
Jan 03 10:00:08 pve1 pveproxy[7983]: Clearing outdated entries from certificate cache
Jan 03 10:00:20 pve1 sshd[8076]: Accepted password for root from 192.168.158.247 port 53644 ssh2
Jan 03 10:00:20 pve1 sshd[8076]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jan 03 10:00:45 pve1 kernel:  sdc:
Jan 03 10:00:45 pve1 kernel:  sdc:
Jan 03 10:00:46 pve1 kernel:  sdc: sdc1
Jan 03 10:00:47 pve1 kernel: XFS (sdb1): Mounting V4 Filesystem
Jan 03 10:00:47 pve1 kernel: XFS (sdb1): Ending clean mount
Jan 03 10:00:47 pve1 kernel: XFS (sdb1): Unmounting Filesystem

root@pve1:~# systemctl start ceph<TAB><TAB>
ceph-create-keys@0.service  ceph-disk@dev-sdc1.service  ceph-disk@dev-sdd1.service  ceph-disk@dev-sde1.service  ceph-disk@dev-sdf1.service  ceph-osd@0.service          ceph.service

root@pve1:~# systemctl start ceph-disk@dev-sdc1.service 
Jan 03 10:01:15 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdc1...
Jan 03 10:01:15 pve1 sh[8313]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdc1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7fa2710846e0>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
Jan 03 10:01:15 pve1 sh[8313]: command: Running command: /sbin/init --version
Jan 03 10:01:15 pve1 sh[8313]: command: Running command: /sbin/blkid -o udev -p /dev/sdc1
Jan 03 10:01:15 pve1 sh[8313]: command: Running command: /sbin/blkid -o udev -p /dev/sdc1
Jan 03 10:01:15 pve1 sh[8313]: main_trigger: trigger /dev/sdc1 parttype None uuid None
Jan 03 10:01:15 pve1 sh[8313]: Traceback (most recent call last):
Jan 03 10:01:15 pve1 sh[8313]: File "/usr/sbin/ceph-disk", line 9, in <module>
Jan 03 10:01:15 pve1 sh[8313]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Jan 03 10:01:15 pve1 sh[8313]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5009, in run
Jan 03 10:01:15 pve1 sh[8313]: main(sys.argv[1:])
Jan 03 10:01:15 pve1 sh[8313]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4960, in main
Jan 03 10:01:15 pve1 sh[8313]: args.func(args)
Jan 03 10:01:15 pve1 sh[8313]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4392, in main_trigger
Jan 03 10:01:15 pve1 sh[8313]: raise Error('unrecognized partition type %s' % parttype)
Jan 03 10:01:15 pve1 sh[8313]: ceph_disk.main.Error: Error: unrecognized partition type None
Job for ceph-disk@dev-sdc1.service failed. See 'systemctl status ceph-disk@dev-sdc1.service' and 'journalctl -xn' for details.
Jan 03 10:01:15 pve1 systemd[1]: ceph-disk@dev-sdc1.service: main process exited, code=exited, status=1/FAILURE
Jan 03 10:01:15 pve1 systemd[1]: Failed to start Ceph disk activation: /dev/sdc1.
Jan 03 10:01:15 pve1 systemd[1]: Unit ceph-disk@dev-sdc1.service entered failed state.

root@pve1:~# Jan 03 10:01:55 pve1 pveproxy[8070]: Clearing outdated entries from certificate cache

that looks like a bug to me, but my CEPH knowledge (and time available) is insufficient to track it down.
 
ok it seems to fail to create the journal (line 167 of your log).

Can you please :
- mount /dev/sdc on a temporary dir dans check the journal file (sould be a symlink to a dev )
- try to see if the dev id is the same a in
Code: 
lsblk /dev/sdp -o NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID"

It should point to /dev/sdc2 if you don't use external journal.
You should be able to write to this directory and everything should be owned by ceph:ceph user/group

you can re-create the journal :
Code: 
 ceph-osd --mkjournal -i <OSD_NUMBER>"
with OSD_NUMBER = 0 I guess for your system
 
1. "activate.monmap" is not owned by ceph/ceph, it's owned by root/root.
2. "journal" points to a partition UUID that doesn't exist.
3. "ceph-osd --mkjournal -i 0" fails, with this error:
Code: 
root@pve1:~# ceph-osd --mkjournal -i 0
2017-01-03 10:45:15.398375 7f22b6992800 -1 filestore(/var/lib/ceph/osd/ceph-0) FileStore::mkjournal: open error: (2) No such file or directory
2017-01-03 10:45:15.398385 7f22b6992800 -1  ** ERROR: error creating fresh journal /var/lib/ceph/osd/ceph-0/journal for object store /var/lib/ceph/osd/ceph-0: (2) No such file or directory
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom