Ceph with two nodes....

Discussion in 'Proxmox VE: Installation and configuration' started by Gilberto Ferreira, Sep 12, 2014.

  1. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    Hello!

    Is that possible?? I meant. in Proxmox way...

    Thank you
     
  2. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    13,635
    Likes Received:
    418
    its possible but it makes no sense. minimum is 3 nodes.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    Thank to answer Tom... Can you point me some docs?? I try it through web interface, but no way! I try create a pool with size/min like 2/1 but after poweroff node 2, ceph stop to work...

    Any idea?

    Thanks a lot
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    quorum...

    If you run 2 ceph monitors, you will loose quorum if one dies.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    Hum... I see... But can I use the quorum cluster as based to manage ceph?
     
  6. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    Sorry, I do not understand that question. But it is quite simple - you need 3 nodes to run a ceph server.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. Gilberto Ferreira

    Joined:
    Aug 21, 2014
    Messages:
    47
    Likes Received:
    0
    perhaps I do not myself clear
    can I use the cluster quorum configuration to provide quorum to a ceph server, or cluster quorum and ceph quorum are different things....
     
  8. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    13,635
    Likes Received:
    418
    No. I suggest you start reading here - http://pve.proxmox.com/wiki/Ceph_Server#Further_readings_about_Ceph

    It makes no sense to help to setup a useless setup and to point others to this. If you want Ceph, use at least 3 nodes.

    It also possible to experiment with virtual Proxmox VE Ceph servers - especially if you do not have the 3 nodes for testing.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. fibo_fr

    fibo_fr Member

    Joined:
    Apr 21, 2014
    Messages:
    40
    Likes Received:
    2
    Would 2 (nodes with) OSDs and 3 (nodes with) MONs work?
     
  10. gosha

    gosha Member

    Joined:
    Oct 20, 2014
    Messages:
    274
    Likes Received:
    15
    Yes, of course. If quorum present it will work.
    If You don,t need HA, even two nodes will work. But in this case if one node from two will down, the other node don't will work. Quorum - more then half nodes. This will be - 2 nodes for 2-node cluster (and 2 nodes too for 3-node cluster, for example). 3-node cluster may be HA, but 2-node cluster - don't.

    ---
    Best regards!
    Gosha
     
    #10 gosha, Apr 21, 2016
    Last edited: Apr 21, 2016
    fibo_fr likes this.
  11. markusd

    markusd Member
    Proxmox Subscriber

    Joined:
    Apr 20, 2015
    Messages:
    89
    Likes Received:
    1
    Hi,
    three mon & 2 osd-server will work and give you a bit HA.
    You need to set size "2" to your pool.
    So you have a second copy from your data.
    Its not perfect and not recommended, but works good.
    You have to look carefully, that your osds have enough space left for taking copys from other osds if some fail.
    I have used this a time. I could restart a osd-server without stopping vms on the other nodes..
    But now we have three node-setup...
    Markus
     
    fibo_fr likes this.
  12. fibo_fr

    fibo_fr Member

    Joined:
    Apr 21, 2014
    Messages:
    40
    Likes Received:
    2
    Thx.
    Got it working after some fights, trials and errors due to my network config... which pveceph DOES NOT LIKE if you have anything special.

    RTFM is correct... but slightly obscure. Secret weapon is to manually edit ceph.conf and give explicitly all the values needed in public network / cluster network in [global] (2 sets of 3 addresses/32) and in the [osd.x] (2 sets of 2 addresses)
     
  13. Kei

    Kei Member

    Joined:
    May 29, 2016
    Messages:
    86
    Likes Received:
    2
    Hi Mark, could you please provide us with additional info about why having only two OSD nodes would not give you full HA? If I understand correctly...
     
  14. markusd

    markusd Member
    Proxmox Subscriber

    Joined:
    Apr 20, 2015
    Messages:
    89
    Likes Received:
    1
    Hi,
    if one node goes down all pg - copies will be placed on the remaining node.
    This can result in a very high load for a long time & evtl. (too) full disks.
    If then an disk dies your data could be lost / hard to repair.
    (For maintenance you can set your osds "noout")

    Markus
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice