Proxmox VE Ceph Server released (beta)

Discussion in 'Proxmox VE: Installation and configuration' started by martin, Jan 24, 2014.

  1. martin

    martin Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    639
    Likes Received:
    363
    We already have a full featured Ceph Storage plugin in our Proxmox VE solution and now - BRAND NEW - it is now possible to install and manage the Ceph Server directly on Proxmox VE - integrated in our management stack (GUI and CLI via Proxmox VE API).

    Documentation
    http://pve.proxmox.com/wiki/Ceph_Server

    Video Tutorial

    http://youtu.be/ImyRUyMBrwo

    Any comment and feedback is welcome!
    __________________
    Best regards,

    Martin Maurer
    Proxmox VE project leader
     
  2. gandalf

    gandalf New Member

    Joined:
    Jan 24, 2014
    Messages:
    2
    Likes Received:
    0
    Hi to all.
    I'm interested in this plugin but I have some questions:

    1. Can I install Proxmox VE in both virtual and phisical servers? For example, I would like to use some virtual machines (not managed by proxmox) as MONs and some phisical servers as OSDs

    2. On each OSD, can I configure a cluster network directly from Proxmox interface?

    3. RGW is supported? If not, how can I sync ceph configuration managed by Proxmox to my RGW node? On every change from proxmox web interface should I manually sync the configuration on each rgw node?
     
  3. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,845
    Likes Received:
    159
    Hi,
    I'm wonder about theceph-wiki-article, that you use Samsung 840 for journaling.
    I have made an test because I use different SSDs for the journal (to prevent an dead SSD on more than one node at the time).
    On an Corsair Force GS I got following output:
    Code:
    root@ceph-01:~# dd if=/root/randfile of=/srv/journal1/test bs=350k count=10000 oflag=direct,dsync                                                            
    10000+0 Datensätze ein                                                                                                                                       
    10000+0 Datensätze aus                                                                                                                                       
    3584000000 Bytes (3,6 GB) kopiert, 38,3221 s, 93,5 MB/s
    
    randfile is random content.

    With an INTEL SSDSC2CW12 144MB/s but with Samsung 840 app. 30MB/s only.

    Udo
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    I get (sdb6 is a partition on the journal disk):

    # dd if=/root/randfile of=/dev/sdb6 bs=350k count=10000 oflag=direct,dsync
    3584000000 bytes (3.6 GB) copied, 41.2994 s, 86.8 MB/s
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,845
    Likes Received:
    159
    Hi Dietmar,
    sounds like I need an firmware-update for the samsung ssd.

    Udo
     
  6. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    Maybe you tested with smaller disk (<240GB)?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,845
    Likes Received:
    159
    Right (128GB) - and they also active - but my other measurements are also on active disks.

    Udo
     
  8. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,483
    Likes Received:
    97
    Something to compare against:
    Code:
    === START OF INFORMATION SECTION ===
    Device Model:     OCZ-AGILITY3
    Serial Number:    OCZ-QMZN8K4967DA9NGO
    LU WWN Device Id: 5 e83a97 e560aff4d
    Firmware Version: 2.22
    User Capacity:    120.034.123.776 bytes [120 GB]
    Sector Size:      512 bytes logical/physical
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   8
    ATA Standard is:  ACS-2 revision 3
    Local Time is:    Sat Jan 25 17:07:59 2014 CET
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    dd if=randfile of=/tmp/dd.img bs=350k count=10000 oflag=direct,dsync
    10000+0 records in
    10000+0 records out
    3584000000 bytes (3,6 GB) copied, 44,9944 s, 79,7 MB/s
    
    
    Code:
    === START OF INFORMATION SECTION ===
    Device Model:     Corsair Force GT
    Serial Number:    120282010000075301AF
    LU WWN Device Id: 0 000000 000000000
    Firmware Version: 1.3.3
    User Capacity:    90,028,302,336 bytes [90.0 GB]
    Sector Size:      512 bytes logical/physical
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   8
    ATA Standard is:  ACS-2 revision 3
    Local Time is:    Sat Jan 25 17:21:20 2014 CET
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    sudo dd if=randfile of=/var/lib/vz/dd.img bs=350k count=10000 oflag=direct,dsync
    10000+0 records in
    10000+0 records out
    3584000000 bytes (3.6 GB) copied, 31.5023 s, 114 MB/s
    
    Code:
    === START OF INFORMATION SECTION ===
    Model Family:     Intel 330/335 Series SSDs
    Device Model:     INTEL SSDSC2CT120A3
    Serial Number:    CVMP2325027B120BGN
    LU WWN Device Id: 5 001517 bb2a00814
    Firmware Version: 300i
    User Capacity:    120.034.123.776 bytes [120 GB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    Solid State Device
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   ACS-2 T13/2015-D revision 3
    SATA Version is:  SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
    Local Time is:    Sat Jan 25 17:24:47 2014 CET
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    dd if=/tmp/randfile of=/home/mir/dd.img bs=350k count=10000 oflag=direct,dsync
    10000+0 records in
    10000+0 records out
    3584000000 bytes (3,6 GB) copied, 54,2376 s, 66,1 MB/s
    
     
  9. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    No, you can only use physical servers as Monitors (at least when you use the GUI)

    You want to use different cluster networks for each OSD?

    The whole idea of this project is to run ceph services on PVE nodes (so all nodes gets config update automatically).
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. gandalf

    gandalf New Member

    Joined:
    Jan 24, 2014
    Messages:
    2
    Likes Received:
    0
    No, one cluster network for all OSD but I would like to split cluster network and primary network for performances.

    But AFAICS, there is no RGW support...
     
  11. X1M

    X1M New Member

    Joined:
    Apr 19, 2010
    Messages:
    19
    Likes Received:
    0
    This is great news, I was going to look into CEPH and now it is melting together with Proxmox, very good idea. Looking forward to this. :)

    As I understand it there’s is no need to split cluster network and primary network for performance. Since Proxmox are writing directly to the CEPH nodes and then sync the cluster trough the dedicated 10 GBit network. Or am I missing something?

    For performance it would be nice with support for Cache Pool and later Tiering. http://ceph.com/docs/master/dev/cache-pool/
     
  12. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    You can do that editing the configuration file.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    It is always good to use a separate network for ceph (see 'pveceph init --network ...'). If you split public and cluster network you can gain additional bandwidth.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    INCREDIBLE!! Thats all i have to say!

    This creates another milestone for Proxmox and a giant push forward! Cant wait to give it a try!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  15. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    Can this be integrated with existing CEPH cluster? or do i have to recreate a new cluster in Proxmox then migrate VM disks to the new cluster?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    No, this only works if all ceph services runs on pve nodes.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    Without any issue I was able to setup Proxmox-Ceph cluster to see this amazing feature in action! This is by far the best feature of Proxmox. In my honest opinion of course. :)

    1. Ceph Status
    1-ceph.png

    2. CEPH Monitors
    2-ceph-mon.png

    3. CEPH OSDs
    3-osd.png

    4. RBD attached to the cluster
    4-rbd.png
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. badji

    badji Member

    Joined:
    Jan 14, 2011
    Messages:
    193
    Likes Received:
    11
  19. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,845
    Likes Received:
    159
    Hi Dietmar,
    how does it work with an "Production Environment"? Ceph recomment 3 monitor nodes and several OSD-Nodes.

    See http://ceph.com/planet/zero-to-hero-guide-for-ceph-cluster-planning/ : "Run Monitor on a separate dedicated hardware or coexists with ceph client nodes".

    Are the pveceph osd nodes part of the ceph-cluster or indepenend storage nodes? An osd-node should not run virtual machines but if they part of the cluster the cpu socket will add for the licensing?!

    Other Question - is it usefull to run virtual machines on the monitoring nodes? How much load is acceptable?

    Udo
     
  20. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    The idea is to run OSDs/Monitors/VMs on the same nodes. So far this setup is not recommended by ceph.com.

    But we claim that we can run up to 50VMs on a single node, so it is hard to see why it is not possible to run
    a view OSDs ...


    yes, we do that (see our example setup).

    I guess this totally depends on your hardware. But an OSD needs about 1GB RAM, and usually not more than 0.5 cores.
    So a recent hardware with 12 or 24 cores, and more than 32GB RAM give you enough
    resources to run everything on the same host.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice