Ceph Performance inside Client

[udo]

Hi,
I have an 3-node ceph cluster with ssd-journal and 10GE-connection.
Inside an VM (wheezy) I got > 100MB/s troughput with dd - but if I cp (or dd) an big file on the same rbd-disk, the performance drop down to 12MB/s.
The networkperformance between host and ceph-nodes is 9.7 GB/s (iperf).

I have tried different cache options, different OS, different filesystems inside VM...

Any hint?

Udo

 

[spirit]

Hi,
I have an 3-node ceph cluster with ssd-journal and 10GE-connection.
Inside an VM (wheezy) I got > 100MB/s troughput with dd - but if I cp (or dd) an big file on the same rbd-disk, the performance drop down to 12MB/s.
The networkperformance between host and ceph-nodes is 9.7 GB/s (iperf).

I have tried different cache options, different OS, different filesystems inside VM...

Any hint?

Udo

maybe it's related to filesystem ? (but the overhead seem to be huge)
Try with a simple filesystem without journal like ext2 by example to compare.

also are your partition correctly aligned ? (alignment on 2048sector (1MB) is best for ssd).

Maybe Stefan Priebe can help you pve-devel mailing, he's using rbd in production and have done a lot of tests.

 

[udo]

maybe it's related to filesystem ? (but the overhead seem to be huge)
Try with a simple filesystem without journal like ext2 by example to compare.

also are your partition correctly aligned ? (alignment on 2048sector (1MB) is best for ssd).

Maybe Stefan Priebe can help you pve-devel mailing, he's using rbd in production and have done a lot of tests.

Hi Spirit,
thanks for the answer.
The aligning of the ssd is ok (with parted 2048s as start) - I guess that the used lvm don't break the aligment?!

But anyway - I have done another tests, and if I read with one VM on a rbd-disk and write with another VM to another rbd-disk, the performance don't stall.

The same happens inside one VM if I change the scheduler from cfq to deadline! With this scheduler I got transferrates around 60MB/s (reading and writing). Not perfect but much better than 12MB/s.

I will do further test, try to speedup the rbd-storage.

Udo

 

[mir]

Hi Spirit,
thanks for the answer.
The aligning of the ssd is ok (with parted 2048s as start) - I guess that the used lvm don't break the aligment?!

But anyway - I have done another tests, and if I read with one VM on a rbd-disk and write with another VM to another rbd-disk, the performance don't stall.

The same happens inside one VM if I change the scheduler from cfq to deadline! With this scheduler I got transferrates around 60MB/s (reading and writing). Not perfect but much better than 12MB/s.

I will do further test, try to speedup the rbd-storage.

Udo

For scheduler using SSD cfg is not recommended. noop or deadline is the recommended. Read a lot more here: https://wiki.archlinux.org/index.php/Solid_State_Drives

As regard to LVM then there is no problem since LVM will auto detect the alignment of the underlying disk. See
- md_chunk_alignment
- default_data_alignment
- data_alignment_detection
- data_alignment
- data_alignment_offset_detection

in /etc/lvm/lvm.conf

 

[udo]

For scheduler using SSD cfg is not recommended. noop or deadline is the recommended. Read a lot more here: https://wiki.archlinux.org/index.php/Solid_State_Drives

As regard to LVM then there is no problem since LVM will auto detect the alignment of the underlying disk. See
- md_chunk_alignment
- default_data_alignment
- data_alignment_detection
- data_alignment
- data_alignment_offset_detection

in /etc/lvm/lvm.conf

Hi mir,
thanks for the info - I will try deadline tomorrow on the three ceph-nodes.

Udo