Proxmox VE Ceph Server released (beta)

Discussion in 'Proxmox VE: Installation and configuration' started by martin, Jan 24, 2014.

  1. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    13,551
    Likes Received:
    405
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. lynn_yudi

    lynn_yudi Member

    Joined:
    Nov 27, 2011
    Messages:
    80
    Likes Received:
    0
  3. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,520
    Likes Received:
    21
    Can an existing ceph cluster be upgraded using ' pveceph ' ?
     
  4. lynn_yudi

    lynn_yudi Member

    Joined:
    Nov 27, 2011
    Messages:
    80
    Likes Received:
    0
    yes, refresh install pve and ceph by pveceph

    #ceph -v
    ceph version 0.80 (b78644e7dee100e48dfeca32c9270a6b210d3003)

    #ceph -s
    cluster b82584ba-4461-4117-a797-6a41f7f1be14
    health HEALTH_WARN 198 pgs degraded; 368 pgs stuck unclean; clock skew detected on mon.1
    monmap e3: 3 mons at {0=192.168.11.2:6789/0,1=192.168.11.3:6789/0,2=192.168.11.4:6789/0}, election epoch 4, quorum 0,1,2 0,1,2
    osdmap e20: 3 osds: 3 up, 3 in
    pgmap v34: 368 pgs, 3 pools, 0 bytes data, 0 objects
    105 MB used, 11155 GB / 11156 GB avail
    62 active
    140 active+degraded
    108 active+remapped
    58 active+degraded+remapped

    but, delete the rbd pool, no error! and create it, it's normal.

    in cli:
    # ceph osd pool delete metadata metadata --yes-i-really-really-mean-it
    Error EBUSY: pool 'metadata' is in use by CephFS
     
    #84 lynn_yudi, May 11, 2014
    Last edited: May 17, 2014
  5. MimCom

    MimCom Member

    Joined:
    Apr 22, 2011
    Messages:
    202
    Likes Received:
    3
  6. lynn_yudi

    lynn_yudi Member

    Joined:
    Nov 27, 2011
    Messages:
    80
    Likes Received:
    0
  7. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Hello,

    I am not sure what the problem is. I had to keep running the command many times before ceph can install. Almost every single time it give me the below error:

    root@server1:~# pveceph install -version firefly
    download and import ceph reqpository keys
    update available package list
    Reading package lists...
    Building dependency tree...
    Reading state information...
    gdisk is already the newest version.
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.
    The following information may help to resolve the situation:

    The following packages have unmet dependencies:
    ceph : Depends: binutils but it is not installable
    Depends: xfsprogs but it is not installable
    Depends: libgoogle-perftools4 but it is not installable
    ceph-common : Depends: libgoogle-perftools4 but it is not installable
    E: Unable to correct problems, you have held broken packages.
    command 'apt-get -q --assume-yes --no-install-recommends -o 'Dpkg::Options::=--f
    orce-confnew' install -- ceph ceph-common gdisk' failed: exit code 100
    root@server1:~#
     
  8. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,520
    Likes Received:
    21
    On Saturday I reinstalled ceph to 3 nodes, one of them had that issue.
    It left me having to reinstall. i forget the exact error msg.

    I reinstalled and had the exact same issue.

    I test installed just one of the pgms because if i let ' pveceph ' command run it'd install just some of the pgms and not let me install the rest.

    As far as I remember running ' apt-get update ' a few times until this worked.
    Code:
    apt-get update
    apt-get install xfsprogs
    
    I did that 3 or 4 times.

    Also fiddled with sources,list and ended up with
    Code:
    deb http://ftp.us.debian.org/debian wheezy main contrib
    deb http://security.debian.org/ wheezy/updates main contrib
    
    # wheezy-updates, previously known as 'volatile'
    deb http://ftp.us.debian.org/debian/ wheezy-updates main
    
    and /etc/apt/sources.list.d/pve.list
    Code:
    deb [arch=amd64] http://download.proxmox.com/debian wheezy pve-no-subscription
    


    So I test installed just one of the pgms because if i let ' pveceph ' command run it'd install just some of the pgms and not let me install the rest.

    The weird thing is that I installed all 3 nodes withing the same hour and just one of them had the issue.
     
  9. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    I read everywhere that SAS is supposed to be better and faster than SATA.

    I have 3 servers filed with SATA drives for OSD and also for boot. On the 4th server, I filled it with SAS drives.

    After creation of OSDs for all 4 servers, on the "OSD" section it show all the SAS drives on the 4th server have latency.

    I also notice that during the install, boot, and OSD creation, the SATA seems to be faster than the SAS.

    Am I not supposed to mix SATA and SAS drives for CEPH?

    When I want to add more hard drive, does it have to be identical to the existing?
     
  10. dswartz

    dswartz Member
    Proxmox Subscriber

    Joined:
    Dec 13, 2010
    Messages:
    257
    Likes Received:
    4
    Without more info, it's hard to say, but all other things being equal SAS should certainly not be slower than SATA!
     
  11. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,066
    Likes Received:
    24
    It is not good idea to mix and match different speed hard drive. SAS certainly faster than SATA. In your case SATA seems faster because your majority drives are SATA and they work together about same speed. CEPH will try to write in all drives equally. So your SAS might be faster but they have to wait for the slower drives to finish before they get their share. You can still mix, but do it eqaully in all nodes. Instead of having all SAS in one node, spread them over 4 nodes. Take out some SATA from other nodes and fill the 4th one. When you want to replace any SATA with SAS, do it in a set of 4. Hope this makes sense.

    How many replicas are you using? Whats your PG count?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. felipe

    felipe Member

    Joined:
    Oct 28, 2013
    Messages:
    152
    Likes Received:
    1
    isnt it possible to mixe sas,sata sas disk using weights for the disks?
    example weight 1 for all sata disks and weight 2 for all sas disks?
     
  13. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,066
    Likes Received:
    24
    If i understand correctly Weight in CEPH defines how much data that HDD going to hold, not the speed. For example, a 2TB with weight 2will keep writing till it is full. If i have mix of 1 TB and 2TB in the cluster and i want to evenly distribute data among all of them, i would set weight 1 for all the 2TB HDD so that do not hold more than 1 TB. Weight allows use of multiple sizes HDD so certain HDDs do not write more than others.
    I guess with this in a way we can set all SATA 2TBs weight to 1.90 while keep all SAS 2TB HDD to say 2. So they will be writing more while SATA 2TBs catches up. Cannot set SAS 2TB beyond 2 then it will run out of space and cause backfill. Actually the weight of 2TB is 1.81 by default in CEPH. I just used rounded figures here of weight 2.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you very much. It makes a whole lot of sense.

    I have 3 replicas. PG count is 1024 (16 x 2TB HD)

    If I want to add more hard drive in the future, can I use 4TB-6TB drives or am I stuck with 2TB drives to match with the current? It would be silly if the drives we add to the cepth nodes has to match the current capacity. Please advise. Thank you for your help.
     
  15. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Hello everyone,

    1) Is there any detailed documentation for Promox and Ceph?

    2) Per this link: http://pve.proxmox.com/wiki/Storage:_Ceph

    I've completed install of ceph nodes, created the necessary steps, then created the VMs successsfully.

    Now I want to add ISO and installation images. Where do I go to upload them? I've tried to upload but keep running to an error. I can't find any documentation show me how to do it.

    Any help pointing in the right direction is greatly appreciated.
     
  16. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,066
    Likes Received:
    24
    Sure you can add any sizes HDD later in future. It doesnt have to 2TB always. What i was saying, Since you have 3 replicas on 3 nodes, to balance writing you should replace in a set of 3 HDDs. If you replacing a 2TB with 4TB, try to replace 3 TBs with 3 4TBs. Note that it "DOES NOT" have to be that way. You can mix and match any sizes, CEPH will automatically set weight based on their capacity, but by using same sets it just gives you good balance of writes thus little more performance.



    You cannot upload ISO on CEPH RBD storage, if that is what you are trying to do. RBD only supports RAW image. If you want to use CEPH to store ISO and other disk images such as qcow2, vmdk then you have to setup CephFS. Here are some simplified steps to create CephFS on CEPH cluster. This steps needs to be done all proxmox nodes that you want to use CephFS:
    1. Install ceph-fuse Proxmox node: #apt-get install ceph-fuse
    2. Create a separate pool: #ceph osd pool create <a_name> 512 512
    3. Create a mount folder on Proxmox nodes: #mkdir /mnt/cephfs or <anything>
    4. #cd /etc/pve
    5. #ceph-fuse /mnt/<a_folder> -o nonempty
    6. Goto Proxmox GUI and add the storage as local directory: /mnt/<a_folder> and you will see you can use any image types.

    To umount a CephFS simply run: #fusermount -u /mnt/<a_folder>

    To mount the CephFS during Proxmox reboot automatically add this to /etc/fstab:
    #DEVICE PATH TYPE OPTIONS
    id=admin,conf=/etc/pve/ceph.conf /mnt/<a_folder> fuse.ceph defaults 0 0

    That is it for creating CephFS.

    Keep in mind that, CephFS is not considered as "Production Ready" yet. But i have been using it for last 11 months without issue. I use it primarily to store ISO, templates and other test VMs with qcow2. All my production VMs are on RBD, so even if CephFS crashes it wont be big loss.
    Hope this helps.



     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,323
    Likes Received:
    135
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. symmcom

    symmcom Active Member

    Joined:
    Oct 28, 2012
    Messages:
    1,066
    Likes Received:
    24
    As spirit mentioned, that is the link for official documentation on Proxmox+CEPH.

    But if you want to details of CEPH, this is the official link: https://ceph.com/docs/master/

    Proxmox wiki shows how to setup CEPH with Proxmox for obvious reason.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. sdutremble

    sdutremble Member

    Joined:
    Sep 29, 2011
    Messages:
    85
    Likes Received:
    0
    I thought it was also necessary to have at least one MDS? Could you modify your steps to have this added if I am correct? Thanks, Serge
     
  20. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you. The documentation fall short of explaining what to do next to store the ISO images for installation to the VMs.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice