Proxmox VE Ceph Server released (beta)

martin

martin

Proxmox Staff Member
Staff member
Apr 28, 2005
754
1,742
223
We already have a full featured Ceph Storage plugin in our Proxmox VE solution and now - BRAND NEW - it is now possible to install and manage the Ceph Server directly on Proxmox VE - integrated in our management stack (GUI and CLI via Proxmox VE API).

Documentation
http://pve.proxmox.com/wiki/Ceph_Server

Video Tutorial
http://youtu.be/ImyRUyMBrwo

Any comment and feedback is welcome!
__________________
Best regards,

Martin Maurer
Proxmox VE project leader
 
Hi to all.
I'm interested in this plugin but I have some questions:

1. Can I install Proxmox VE in both virtual and phisical servers? For example, I would like to use some virtual machines (not managed by proxmox) as MONs and some phisical servers as OSDs

2. On each OSD, can I configure a cluster network directly from Proxmox interface?

3. RGW is supported? If not, how can I sync ceph configuration managed by Proxmox to my RGW node? On every change from proxmox web interface should I manually sync the configuration on each rgw node?
 
martin said:
We already have a full featured Ceph Storage plugin in our Proxmox VE solution and now - BRAND NEW - it is now possible to install and manage the Ceph Server directly on Proxmox VE - integrated in our management stack (GUI and CLI via Proxmox VE API).

Documentation
http://pve.proxmox.com/wiki/Ceph_Server

Video Tutorial
http://youtu.be/ImyRUyMBrwo

Any comment and feedback is welcome!
__________________
Best regards,

Martin Maurer
Proxmox VE project leader
Click to expand...
Hi,
I'm wonder about theceph-wiki-article, that you use Samsung 840 for journaling.
I have made an test because I use different SSDs for the journal (to prevent an dead SSD on more than one node at the time).
On an Corsair Force GS I got following output:
Code: 
root@ceph-01:~# dd if=/root/randfile of=/srv/journal1/test bs=350k count=10000 oflag=direct,dsync                                                            
10000+0 Datensätze ein                                                                                                                                       
10000+0 Datensätze aus                                                                                                                                       
3584000000 Bytes (3,6 GB) kopiert, 38,3221 s, 93,5 MB/s
randfile is random content.

With an INTEL SSDSC2CW12 144MB/s but with Samsung 840 app. 30MB/s only.

Udo
 
udo said:
With an INTEL SSDSC2CW12 144MB/s but with Samsung 840 app. 30MB/s only.
Click to expand...

I get (sdb6 is a partition on the journal disk):

# dd if=/root/randfile of=/dev/sdb6 bs=350k count=10000 oflag=direct,dsync
3584000000 bytes (3.6 GB) copied, 41.2994 s, 86.8 MB/s
 
Something to compare against:
Code: 
=== START OF INFORMATION SECTION ===
Device Model:     OCZ-AGILITY3
Serial Number:    OCZ-QMZN8K4967DA9NGO
LU WWN Device Id: 5 e83a97 e560aff4d
Firmware Version: 2.22
User Capacity:    120.034.123.776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-2 revision 3
Local Time is:    Sat Jan 25 17:07:59 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

dd if=randfile of=/tmp/dd.img bs=350k count=10000 oflag=direct,dsync
10000+0 records in
10000+0 records out
3584000000 bytes (3,6 GB) copied, 44,9944 s, 79,7 MB/s
Code: 
=== START OF INFORMATION SECTION ===
Device Model:     Corsair Force GT
Serial Number:    120282010000075301AF
LU WWN Device Id: 0 000000 000000000
Firmware Version: 1.3.3
User Capacity:    90,028,302,336 bytes [90.0 GB]
Sector Size:      512 bytes logical/physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-2 revision 3
Local Time is:    Sat Jan 25 17:21:20 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


sudo dd if=randfile of=/var/lib/vz/dd.img bs=350k count=10000 oflag=direct,dsync
10000+0 records in
10000+0 records out
3584000000 bytes (3.6 GB) copied, 31.5023 s, 114 MB/s
Code: 
=== START OF INFORMATION SECTION ===
Model Family:     Intel 330/335 Series SSDs
Device Model:     INTEL SSDSC2CT120A3
Serial Number:    CVMP2325027B120BGN
LU WWN Device Id: 5 001517 bb2a00814
Firmware Version: 300i
User Capacity:    120.034.123.776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan 25 17:24:47 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


dd if=/tmp/randfile of=/home/mir/dd.img bs=350k count=10000 oflag=direct,dsync
10000+0 records in
10000+0 records out
3584000000 bytes (3,6 GB) copied, 54,2376 s, 66,1 MB/s
 
gandalf said:
1. Can I install Proxmox VE in both virtual and phisical servers? For example, I would like to use some virtual machines (not managed by proxmox) as MONs and some phisical servers as OSDs
Click to expand...

No, you can only use physical servers as Monitors (at least when you use the GUI)

gandalf said:
2. On each OSD, can I configure a cluster network directly from Proxmox interface?
Click to expand...

You want to use different cluster networks for each OSD?

gandalf said:
3. RGW is supported? If not, how can I sync ceph configuration managed by Proxmox to my RGW node? On every change from proxmox web interface should I manually sync the configuration on each rgw node?
Click to expand...

The whole idea of this project is to run ceph services on PVE nodes (so all nodes gets config update automatically).
 
dietmar said:
You want to use different cluster networks for each OSD?
Click to expand...

No, one cluster network for all OSD but I would like to split cluster network and primary network for performances.

The whole idea of this project is to run ceph services on PVE nodes (so all nodes gets config update automatically).
Click to expand...

But AFAICS, there is no RGW support...
 
This is great news, I was going to look into CEPH and now it is melting together with Proxmox, very good idea. Looking forward to this. :)

As I understand it there’s is no need to split cluster network and primary network for performance. Since Proxmox are writing directly to the CEPH nodes and then sync the cluster trough the dedicated 10 GBit network. Or am I missing something?

For performance it would be nice with support for Cache Pool and later Tiering. http://ceph.com/docs/master/dev/cache-pool/
 
X1M said:
As I understand it there’s is no need to split cluster network and primary network for performance. Since Proxmox are writing directly to the CEPH nodes and then sync the cluster trough the dedicated 10 GBit network. Or am I missing something?
Click to expand...

It is always good to use a separate network for ceph (see 'pveceph init --network ...'). If you split public and cluster network you can gain additional bandwidth.
 
Can this be integrated with existing CEPH cluster? or do i have to recreate a new cluster in Proxmox then migrate VM disks to the new cluster?
 
Without any issue I was able to setup Proxmox-Ceph cluster to see this amazing feature in action! This is by far the best feature of Proxmox. In my honest opinion of course. :)

1. Ceph Status
1-ceph.png

2. CEPH Monitors
2-ceph-mon.png

3. CEPH OSDs
3-osd.png

4. RBD attached to the cluster
4-rbd.png
 
dietmar said:
No, this only works if all ceph services runs on pve nodes.
Click to expand...
Hi Dietmar,
how does it work with an "Production Environment"? Ceph recomment 3 monitor nodes and several OSD-Nodes.

See http://ceph.com/planet/zero-to-hero-guide-for-ceph-cluster-planning/ : "Run Monitor on a separate dedicated hardware or coexists with ceph client nodes".

Are the pveceph osd nodes part of the ceph-cluster or indepenend storage nodes? An osd-node should not run virtual machines but if they part of the cluster the cpu socket will add for the licensing?!

Other Question - is it usefull to run virtual machines on the monitoring nodes? How much load is acceptable?

Udo
 
udo said:
how does it work with an "Production Environment"? Ceph recomment 3 monitor nodes and several OSD-Nodes.

See http://ceph.com/planet/zero-to-hero-guide-for-ceph-cluster-planning/ : "Run Monitor on a separate dedicated hardware or coexists with ceph client nodes".

Are the pveceph osd nodes part of the ceph-cluster or indepenend storage nodes? An osd-node should not run virtual machines but if they part of the cluster the cpu socket will add for the licensing?!
Click to expand...

The idea is to run OSDs/Monitors/VMs on the same nodes. So far this setup is not recommended by ceph.com.

But we claim that we can run up to 50VMs on a single node, so it is hard to see why it is not possible to run
a view OSDs ...


udo said:
Other Question - is it usefull to run virtual machines on the monitoring nodes?
Click to expand...

yes, we do that (see our example setup).

udo said:
How much load is acceptable?
Click to expand...

I guess this totally depends on your hardware. But an OSD needs about 1GB RAM, and usually not more than 0.5 cores.
So a recent hardware with 12 or 24 cores, and more than 32GB RAM give you enough
resources to run everything on the same host.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom