[SOLVED] Journal was not prepared with ceph-disk

Firm

Renowned Member
Oct 27, 2015
40
1
73
Hi,

I use Proxmox 4.2 and wonder if things are ok:
Code:
# pveceph createosd /dev/sdc -journal_dev /dev/sda3
create OSD on /dev/sdc (xfs)
using device '/dev/sda3' for journal
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data
WARNING:ceph-disk:Old blkid does not support ID_PART_ENTRY_* fields, trying sgdisk; may not correctly identify ceph volumes with dmcrypt
WARNING:ceph-disk:Journal /dev/sda3 was not prepared with ceph-disk. Symlinking directly.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
meta-data=/dev/sdc1  isize=2048  agcount=4, agsize=244188597 blks
  =  sectsz=512  attr=2, projid32bit=1
  =  crc=0  finobt=0
data  =  bsize=4096  blocks=976754385, imaxpct=5
  =  sunit=0  swidth=0 blks
naming  =version 2  bsize=4096  ascii-ci=0 ftype=0
log  =internal log  bsize=4096  blocks=476930, version=2
  =  sectsz=512  sunit=0 blks, lazy-count=1
realtime =none  extsz=4096  blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
Before reboot I have the following partitions on /dev/sda:
Code:
sda  8:0  0 186.3G  0 disk  
├─sda1  8:1  0  1M  0 part  
├─sda2  8:2  0  244M  0 part  
│ └─md0  9:0  0 243.8M  0 raid1 /boot
├─sda3  8:3  0  5G  0 part  
└─sda4  8:4  0 181.1G  0 part  
  └─md1  9:1  0  181G  0 raid1
  └─vg0-root 251:0  0  15.3G  0 lvm  /

But after reboot /dev/sda3 is missing:
Code:
sda  8:0  0 186.3G  0 disk  
├─sda1  8:1  0  1M  0 part  
├─sda2  8:2  0  244M  0 part  
│ └─md0  9:0  0 243.8M  0 raid1 /boot
└─sda4  8:4  0 181.1G  0 part  
  └─md1  9:1  0  181G  0 raid1
  └─vg0-root 251:0  0  15.3G  0 lvm  /

But osd.0 disk is up and running in Proxmox. And I have
Code:
ls -l /var/lib/ceph/osd/ceph-0/journal
lrwxrwxrwx 1 root root 9 Jun 28 18:00 /var/lib/ceph/osd/ceph-0/journal -> /dev/sda3
which looks broken as /dev/sda3 is missing.

Questions:
1. Should I recreate partition manually?
2. Even if I try to set journal partition GUID manually, name it 'ceph journal', use ceph-disk to create OSD I get the same warning.

In both cases (pveceph and ceph-disk) osd.0 appears in Proxmox UI.

P.S. It seems to me that this warning can be silently ignored in my case but want to be sure.

Regards,
Alex
 
Very strange/bad. Even running
Code:
ceph-disk prepare --fs-type xfs --cluster ceph --cluster-uuid 908ceb45-91b6-4c31-8ede-00acab17c9ef --journal-dev /dev/sdc /dev/sda3
deletes /dev/sda3 :(
 
For those who are interested in creating osd journal on separate partition (not disk) here are the steps:
(assume that sda3 and sdb3 - 5GB partitions for journal, sdc and sdd - disks for osd data)

1. Create partition with correct size. If using fdisk partition's size should be 10483712 sectors.
2. Create osd disk with file-based journal
Code:
ceph-disk prepare --fs-type xfs --cluster ceph --cluster-uuid 908ceb45-91b6-4c31-8ede-00acab17c9ef --journal-file /dev/sdc /tmp/journal
3. Stop this osd.0 in Proxmox UI
4. Flush journal (as precaution)
Code:
ceph-osd --flush-journal -i 0
5. Relink journal to partition:
Code:
ln -sf /dev/sda3 /var/lib/ceph/osd/ceph-0/journal
6. Create journal
Code:
ceph-osd --mkjournal -i 0
7. Start osd.0 in Proxmox UI
8. Delete /tmp/journal and repeat steps above for sdb3 and sdd
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!