pveceph question

N

nethfel

Member
Dec 26, 2014
151
0
16
Hi all,
In talking with Udo on another post, he had mentioned suggesting using ext4 instead of xfs for the osd file system. I'm using pveceph, and:

pveceph createosd /dev/<DRIVEID>

will build a osd using xfs. Is there a way to get it to build it as ext4? Did I do something wrong at initial install of proxmox that now limits me to using xfs when dealing with ceph?
 
Pveceph creates XFS by default. ext4 is very stable tried and true filesystem. XFS is newer but also tried and true. Currently Ceph recommends (http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/) XFS.
If you want to use ext4, i think you have to go for manual Ceph deployment.
Udo probably can shed some light on ext4 over XFS. I personally dont have any issue in my all Ceph deployments on XFS. If you have seen my other post about recent issue, the root cause was human error.
 
Well, the reason I asked was in another thread in my discussion with Udo, he mentioned an improvement on latency with ext4 over xfs which was why I was looking to see if there was a way to handle it.

ps - yes, I had read some of the more recent threads too - btw, bought the book and really enjoyed it!
 
# pveceph help createosd -v
USAGE: pveceph createosd <dev> [OPTIONS]


Create OSD


<dev> string


Block device name.


-fstype (btrfs | ext4 | xfs) (default=xfs)


File system type.


-journal_dev string


Block device name for journal.
 
AH HA! Thank you Dietmar! I was doing the help wrong.

I was using more of a Cisco style of navigating help and it wasn't showing me suboptions, obviously if I had the words in the right order I would have seen that!
 
symmcom said:
Pveceph creates XFS by default. ext4 is very stable tried and true filesystem. XFS is newer but also tried and true.
Click to expand...
Hi,
sorry but xfs is much older than ext4!
Currently Ceph recommends (http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/) XFS.
If you want to use ext4, i think you have to go for manual Ceph deployment.
Udo probably can shed some light on ext4 over XFS. I personally dont have any issue in my all Ceph deployments on XFS. If you have seen my other post about recent issue, the root cause was human error.
Click to expand...
how good xfs work are highly depending from the kernel. Under normal condition the filesystem fragmentation is high - I had up to 20% fragmentation on some disks (the right mount option makes it better).
You can run defragmentation, but I had also an issue with an ubuntu-kernel which produce an kernel panic during this...
For me an filled ext4-filesystem performs better than an filled xfs-filesystem (for ceph). With filled I mean 60-70%.

BTW: If someone would try ext4 I use following options:
Code: 
[osd]
osd_mount_options_ext4 = "user_xattr,rw,noatime,nodiratime"
osd_mkfs_type = ext4
osd_mkfs_options_ext4 = -J size=1024 -E lazy_itable_init=0,lazy_journal_init=0
Udo
 
udo said:
sorry but xfs is much older than ext4!
Click to expand...
You are right! It appears that i have mixed up all my filesystem info. XFS is even older than ext3. I can not even remember where i found the info XFS was newer. What are we doing using XFS??

udo said:
BTW: If someone would try ext4 I use following options:
Code: 
[osd]
osd_mount_options_ext4 = "user_xattr,rw,noatime,nodiratime"
osd_mkfs_type = ext4
osd_mkfs_options_ext4 = -J size=1024 -E lazy_itable_init=0,lazy_journal_init=0
Udo
Click to expand...
Since all of my OSDs are using XFS, can i start replacing OSDs with ext4 one by one? I am assuming the HDD needs to recreate the filesystem. I still need to add 8 more OSDs in the recent incident cluster. I am thinking this would be a good time to introduce ext4 if i really want to do it.
 
udo said:
Hi,
sorry but xfs is much older than ext4!

how good xfs work are highly depending from the kernel. Under normal condition the filesystem fragmentation is high - I had up to 20% fragmentation on some disks (the right mount option makes it better).
You can run defragmentation, but I had also an issue with an ubuntu-kernel which produce an kernel panic during this...
For me an filled ext4-filesystem performs better than an filled xfs-filesystem (for ceph). With filled I mean 60-70%.
Click to expand...

Udo, could you provide some more details on your configuration?
- do you use SSDs of HDDs for CEPH OSDs?
- do you use dedicated SSD for journal with ext4 (file or partition based?) ?

Thanks in advance,
 
Whatever said:
Udo, could you provide some more details on your configuration?
- do you use SSDs of HDDs for CEPH OSDs?
- do you use dedicated SSD for journal with ext4 (file or partition based?) ?

Thanks in advance,
Click to expand...
Hi,
most of the OSDs are 4TB hdds. First I try filebased journaling on SSDs, but the performance was better with partition-based journal.
Since a short time I have an EC-pool (reasure coding) and therefore you need an cache tier in front of. For this I use SSDs OSDs (second root, defined in the crushmap).
Every osd-node has one ssd for journal and one ssd for the cache tier pool.

I will try the performance with an cache tier in front of an normal pool later.

Udo
 
You must log in or register to reply here.
Top Bottom