Sharing Storage (Ceph or Gluster) for VM's in a 3 node scenario...

Discussion in 'Proxmox VE: Installation and configuration' started by andrea68, May 21, 2019.

  1. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    Hi,

    I have a 3 dell Poweredge R610 24GB RAM and 6 SAS 300GB 10K rpm each.

    I wish to build a cluster (mostly KVM) with shared storage system between this 3 nodes, and I will use internal storage to do. I was thinking to use Ceph or GlusterFS, but I'm not sure what is the best choice.

    Each server has 4 ethernet port 1GB and I was thinking to use 2 ethernet port with separate switch to use for storage and the other one (or two) to networking.
    Then in internal storage I will use one disk (or two in raid 1?), maybe smaller of 300GB sas for proxmox os and the other ones to build the shared storage.

    Do you think I'm going in the right direction?
    Anyone has already trying similar solution?
    In this scenario what is the best choice, Ceph or Gluster?

    Thanx in advance!
     
  2. Romsch

    Romsch Member
    Proxmox Subscriber

    Joined:
    Feb 14, 2019
    Messages:
    57
    Likes Received:
    2
    Hi!

    In your case i would choose the option to build a cluster with ceph. Here you have the snapshot option available, and i think - thats my experience - that ceph is faster than other scalable network storage solutions, depending on hardware and so on.. thats clear.

    You should use for proxmox raid1 with two small ssds, and the 6 sas 300 GB 10k harddrives for ceph - but not in raid - directly connect the ceph harddrive disk to sata port on board.
    Two nic´s for the cluster network, maybe with bonding config, and also the other two nics for ceph. In a small environment the 1 GB nic for ceph is ok, but if you go to production or something else, 10GB nic are much better.

    best regards,
    Roman
     
  3. ness1602

    ness1602 New Member

    Joined:
    Oct 28, 2014
    Messages:
    26
    Likes Received:
    2
    For only 3 hosts,i would recommend Gluster
     
  4. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    Tnx Ness.

    I will looking into it.

    It's a little cluster we talking about, with slow performance... maybe I will upgrade to 10Gb SFP network card for every host (+ a 10Gb switch), to boost perfomances (if we use a shared storage) ...
    There will be few VM's for internal test and use (Win and Linux).

    Now I have the cluster with no shared storage whatsoever and all the VM's are saved nightly in a NAS connected to cluster with NFS (very slow but good enough for backup purpose)
    It will be nice to have not a really HA environment, but a simple shared storage to improuve migration from node-to-node.

    This will be my goal...

    Tnx for your advice!
     
  5. Romsch

    Romsch Member
    Proxmox Subscriber

    Joined:
    Feb 14, 2019
    Messages:
    57
    Likes Received:
    2
    and why? maybe a reason why it is "better" than ceph with only three nodes?
    i think, ceph is easier to setup, thats my opinion.
     
  6. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    I never use Ceph so I don't know if is it's easier or not.
    I use just a couple of time Gluster and I'll found very easy to setup but not simple to recover in case of failure of a node...

    Ceph seems to me a little bit complicated because use more components (OSD, Networking, Journaling etc...) but maybe more rock solid?

    Just asking...
     
  7. Romsch

    Romsch Member
    Proxmox Subscriber

    Joined:
    Feb 14, 2019
    Messages:
    57
    Likes Received:
    2
    ok. if you have more experience with Gluster, you should choose it.
    ceph is really easy if you know it.
    You should take a look at it anyway; Install Ceph on Proxmox
    With ceph you have "both"; block level storage for the VMs in raw format, and file level storage for iso´s, backups,...
     
  8. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    I will definetively do.

    Do you think that my hardware is good enough for a starting point or I will think to upgrade some components right now, before I start?
     
  9. Romsch

    Romsch Member
    Proxmox Subscriber

    Joined:
    Feb 14, 2019
    Messages:
    57
    Likes Received:
    2
    For a start i would say the DELL R610´s are of course ok! Also your maximum RAM is good. Storage... with GlusterFS - i dont realy know the distribution here - you have more Storage then ceph, because with ceph you only have 1/3 of all SAS Disks in GB for VMs, in your case maximum storage for ceph is round about 1 TB (4x300GB (max 6 SAS per DELL - 2 for Proxmox Raid1) x 3 DELL Server = 3,6 TB - Ceph Pool Size 3/2 = 1,2 TB ~ 1 TB for RAW images, VMs).
    For GlusterFS as i said, i dont know the maximum TB Storage.

    Best regards, roman
     
    andrea68 likes this.
  10. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    Tnx Roman...

    How about network cards?
    It's mandatory to upgrade from 1GB to 10Gb?
     
  11. alexskysilk

    alexskysilk Active Member

    Joined:
    Oct 16, 2015
    Messages:
    556
    Likes Received:
    58
    ceph is happiest with a greater number of nodes/osds, and with 1Gbit connections your experience may be less then ideal. Gluster is better suited for this size of deployment. If you have the time and inclination, try both and see how either performs on your HW.

    No, its not mandatory, but it will help your perceived performance quite a bit, especially as your VM count increases. With only 3 nodes you can deploy mesh networking which means you wont need a 10g switch.
     
  12. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    Tnx alexskysilk.
    The starting point is 3 node minimum but I don't exclude that we will increase the nodes from 3 to 4 or 5... In time...
    It depends.

    I will try both systems but at some point I need to decide what is the right direction...


    The cost of a 10g card is very affordable, what concern me is the cost of the switch, configuration, maintenance etc ...
    I will looking into it...
     
  13. ness1602

    ness1602 New Member

    Joined:
    Oct 28, 2014
    Messages:
    26
    Likes Received:
    2
    Gluster is very easy to setup , and performs okay with 3 nodes(better than CEPH,that is).
     
  14. andrea68

    andrea68 Member

    Joined:
    Jun 30, 2010
    Messages:
    91
    Likes Received:
    0
    Have you tried to recover from from a node fail without problems?
    I try to, but with some issues...
     
  15. Romsch

    Romsch Member
    Proxmox Subscriber

    Joined:
    Feb 14, 2019
    Messages:
    57
    Likes Received:
    2
    Please try ceph if you want to plan to increase the cluster with more nodes.. and 10GB nics are not so expensive. But try it with 1 GB, thats really ok
     
  16. ness1602

    ness1602 New Member

    Joined:
    Oct 28, 2014
    Messages:
    26
    Likes Received:
    2
    NO,unfortunately.
     
  17. Dark26

    Dark26 Member

    Joined:
    Nov 27, 2017
    Messages:
    71
    Likes Received:
    3
    i think it's depend of the configuration, replica, distribute, number of brick...

    for example, with a replica 3 arbiter 1 or full replica 3 with 3 bricks, the recover is easy because the all the data is on the brick itself.

    if you have a problem, the data is visible on the storage.

    if you have distribute ...
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice