Proxmox VE Ceph Server released (beta)

Discussion in 'Proxmox VE: Installation and configuration' started by martin, Jan 24, 2014.

  1. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Hello,

    What is the linus kernel Proxmox VE Ceph Server released (beta) is on? Is it RHEL 6.4?

    I am installing a set of new QLogic inifiniband NICs for the Ceph nodes. However, Proxmox is not recognizing the card.

    QLogic support is stating they do not have drivers for Debian. Only for RHEL. I understand Proxmox was using RHEL kernal 6.3 as of last year. Is it still the same with the current version 3.2?

    Is there a list of compatible hardware NIC cards and mother boards that we can reference to? This would be helpful in selecting the hardware for the Ceph nodes. Thank you.
     
  2. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,481
    Likes Received:
    96
    Current version must be RHEL 6.5.
     
  3. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,323
    Likes Received:
    135

    pve-kernel-2.6.32 : rhel 6.5
    pve-kernel-3.10 : rhel 7
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
  5. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,463
    Likes Received:
    311
    yes
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,070
    Likes Received:
    24
    I can confirm that HA setup for VMs still works on Ceph nodes. It is expected to work since Ceph integration in Proxmox does not really change anything to the hypervisor itself. Proxmox mainly uses API for Ceph to do basic tasks and show information on GUI.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you. Do I need to do anything to switch to the Redhat Kernel?

    Tried to install a Qlogic adapter card with the driver provided by manufacturer. Got the error with pve kernel and not finding Redhat rpm.

    "Kernel version 2.6.32-29-pve
    Binary rpm not found for the above adapters"
     
  8. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Has anyone successfully ran an Infiniband network of 20gbps or 40gbps for the CEPH nodes? If so, I would greatly appreciate feedback on what servers (mother boards/processors) you are using that is capable of pushing such high speed? Thank you.
     
  9. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Proxmox is using RHEL kernel. It was stated in the forum that any device support RHEL would be compatible with Proxmox.

    When I tried to install a QLOGIC NIC card driver, it gave me an error:

    "Kernel version 2.6.32-29-pve
    Binary rpm not found for the above adapters"

    This mean it's recognizing Proxmox's own kernel and not RHEL?

    Can anyone please help to point me in the right direction?

    Thank you very much in advance for your help.
     
  10. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,835
    Likes Received:
    159
    Hi,
    I have "only" 10GB Ethernet (with dedicated osd-network).
    While the osd-network partially use the speed (after inserting additional disks, the ceph-cluster use up to 1.4GByte/s to reorginized the cluster) the speed for the client is not so good... I guess that 20 or 40GB don't change much.

    But perhaps someone can show better values?

    Udo
     
  11. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,323
    Likes Received:
    135
    Hi, I don't think you can reach 20gbits with IP over infiband. (I think maybe 4 gigabits max)
    Full rdma support is coming soon in ceph, so I think you'll be able to reach native infiband speed.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,835
    Likes Received:
    159
    Hi,
    a little bit more is possible.

    This shows an infiniband-connection (40GB/s) tested with iperf:
    Code:
    oot@proxtest3:~# iperf -c 172.20.1.82
    ------------------------------------------------------------
    Client connecting to 172.20.1.82, TCP port 5001
    TCP window size:  645 KByte (default)
    ------------------------------------------------------------
    [  3] local 172.20.1.83 port 51412 connected with 172.20.1.82 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  15.7 GBytes  13.4 Gbits/sec
    
    lspci
    07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
    
    Udo
     
  13. Florent

    Florent Member

    Joined:
    Apr 3, 2012
    Messages:
    91
    Likes Received:
    2
    I think I have found a bug in pveceph tool : when we have some custom sections in ceph.conf (like mds...), and pveceph needs to rewrite the file, it deletes sections it doesn't know (like ceph mds !).

    Hopefully I had a backup copy of my config file :)
     
  14. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    I am drooling while looking at the result of your iperf. Man, that is some serious speed for the Ceph nodes. An IT guy's wet dreams.

    May I ask for the brand/model of the:

    1) Switches and NIC cards you are using?

    2) Cables type (I think Infiniband CX4 cables only support up to 10Gbps)

    If you don't mind, please share your server config and hard drive type, etc. I know it takes a lot of CPUs workhorse to push that type of speed. Furthermore, the hard drive transfer speed should also be the bottle neck. I am perplexed as to how you can push this type of speed?

    Thanks in advance for your help.
     
    #214 impire, Jul 1, 2014
    Last edited: Jul 1, 2014
  15. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,481
    Likes Received:
    96
    From Udo's copy/paste:
    Nic: 07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

    http://www.mellanox.com/page/products_dyn?product_family=27
     
  16. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    I tried to find info on rdma with Ceph but it's very vague. What exactly is RDMA and how will it help Ceph to improve speed? Thanks.
     
  17. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,835
    Likes Received:
    159
    Hi,
    like mir allready wrote: Mellanox 40GB Card (without switch - it's an direct connection for drbd). For ceph I use 10GB Ethernet.
    I'm not sure how this cable named... QSFP?!
    In this case are the server testserver (to find an issue with an drbd-connection). Two amd-boxes - one fx8350 and one 965.

    Normaly the cards are in dual-Opterons server.+

    Udo
     
  18. felipe

    felipe Member

    Joined:
    Oct 28, 2013
    Messages:
    152
    Likes Received:
    1
     
  19. felipe

    felipe Member

    Joined:
    Oct 28, 2013
    Messages:
    152
    Likes Received:
    1
    ok i gout it: virtio makes a BIG! speed difference now.
    ide: 110 mb read
    virtio: 500mb read

    what is funny is that both give me 500mb/write! virtio and also ide
    also the cache modes make no difference. writeback or none is both 500mb
    i am using 4GB tests Crystal Diskmark so i dont hit the cache. with 1gb or 500mb it gets a lot faster because of the cache....

    what is also very strange is that i get better speeds inside the vm then with rados bench! (even with 32 or 64 threads) and also atop has more disk load. and the ceph log also says that disk load is higher...

    but still have the bottleneck of the 500mb/sec write for the whole cluster... (with 45 disks journals on disks - 3 server with 15 disks each) 10gig for external and 10gig for osd traffic...
     
  20. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,323
    Likes Received:
    135
    I you should try to bench from a linux vm with virtio disk, "fio" tools is really good for this.
    virtio on windows is known to be slower than linux
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice