Proxmox VE Ceph Server released (beta)

Discussion in 'Proxmox VE: Installation and configuration' started by martin, Jan 24, 2014.

  1. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Hello,

    I would like to run the OpenVZ container on the Ceph Nodes. Is this possible. I don't see a way to do that? When I tried to create the container, ProxMox only give option for local drive.

    It would be great to be able to put the container on the Ceph storage.
     
  2. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,845
    Likes Received:
    159
    Hi,
    I use ceph-dash as gui for the ceph-cluster.
    But shows the statiscic of the whole cluster, not for single nodes...


    Udo
     
  3. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    Openvz cannot be used on RBD. You have to setup cephfs for that.

    Sent from my SGH-I747M using Tapatalk
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you Udo. Screenshot looks great, but I couldn't install it on the Proxmox (debian host). Tried to follow the documentation but didn't work. Any pointer on how to install this would be much appreciated.
     
  5. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you symmcom. Appreciate it. Since Cephfs is still in beta and not production ready, is there an alternative? Have any one tried GlusterFS?
     
  6. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    In addition to what Udo shared about ceph-dash, you can also use nload command line tool to monitor a network interface. The display looks as the following.
    nload.png

    The command format is : # nload ethX

    This shows bandwidth usage of a particular network interface in real time. In the example above it is showing bandwidth usage for eth2 which is part of Ceph network.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    I think Beta is relative word here. More like a precautionary measure from Ceph developers side and thats my opinion. I have been using CephFS for a while now without loss of data or inconsistency. I use it mostly to store ISOs, Templates and OpenVZs. I will not put KVMs on CephFS even if it was very stable and production ready only because RBD has better performance than CephFS. CephFS is a file system whereas RBD is blocks without additional software layer for file system like CephFS.

    Gluster is excellent storage system. But it somewhat gets messy to maintain to separate storage system. Gluster and Ceph both are very powerful storage technology. Using both would be like using Proxmox and VMWare side by side. In reality this does not make very much sense unless specifically required.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. jleg

    jleg Member

    Joined:
    Nov 24, 2009
    Messages:
    105
    Likes Received:
    2
    Does quota and Proxmox-UI (show disk usage etc.) still work with OpenVZ on CephFS?
    I once tried XFS, and had all sorts of problems, even with patched PVE perl modules (ok, that was about Prox V1.9 or so).
     
  9. mo_

    mo_ Member

    Joined:
    Oct 27, 2011
    Messages:
    399
    Likes Received:
    3
    I was wondering, with ceph-server being a tech preview still, would it possibly be more stable to just install the normal ceph packages and forgo the proxmox GUI integration? Possibly when the RHEL7 3.10 kernel comes around...? I know theres a deadlock kernel issue between locally mapping an RBD on a host thats running a monitor, but afaik qemu uses librbd (userspace, therefore "slow") to interface with the RBDs instead of mapping them in kernel space, so the vanilla ceph packages should just work I presume?

    Also, while we're at it: does proxmox actually use the 'ceph' CLI to interface with ceph clusters you have added as storage (not ceph-server, a distant cluster)? As in: does ceph-common on the proxmox nodes need to be of the same branch as the ceph cluster? Asking since the ceph CLI has changed significantly between cuttlefish and dumpling
     
  10. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,484
    Likes Received:
    314
    we use the 'normal' ceph packages.

    We use standard ceph packages, and standard ceph API.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    I couldn't find any Mellanox cards in quantity so opted for Qlogic Infiniband cards. What a nightmare. Proxmox won't recognize and Qlogic said driver only for Redhad Linux or SUSE. I am out of luck with 14 of these.
     
  12. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Hello,

    Not sure what I am doing wrong here but Ceph nodes keep getting the "can't connect to cluster" message.

    3 Proxmox hosts each have 3 NIC cards.

    vmbr0 = Eth0 = Proxmox host (assigned ip 5.5.5.10/24, gateway 5.5.5.10
    vmbr1 = Eth1 = VMs (no ip assigned to subnet)
    vmbr2 = Eth3 = Ceph nodes (assigned ip 10.10.10.3/24 -- .4 for second Ceph node, .5 for third Ceph node).

    I went through the installation procedure as suggested by Proxmox wiki doc. Then ran the command:
    pveceph init --network 10.10.10.0/24

    Everything seems fine. I was able to create the monitors, pool, storage, and copied keyring.

    However, when tried to create the VMs, it said cannot connect to rdb.

    Browsing to the storage name on each host give "cannot communicate" or "cannot connect to the cluster".

    Any help would be greatly appreciated. Thank you.
     
  13. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    I know you already said you created keyring. But did you make sure the name of the keyring matches the name of the storage you created? For example, if you named the ceph storage "ceph-rbd' through Proxmox GUI, then the ceph keyring should be copied as ceph-rbd.keyring and copied into /etc/pve/priv/ceph
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you. That was one of the steps which I carefully have done. I notice even before creating the VMs, I tried to browse the storage name and the error was already there.

    I am reinstalling the servers for the 5th times. Will make sure I have the key ring correctly copied. The reason for multiple installs is every time it does this, the OSDs needed to be wiped out from the hard drive controller. I've tried fdisk (option d, option w), it did not clear out the partition. The only solution was to reload.

    Does it matter if I name the storage name same as the pool and the cluster name?
     
    #194 impire, Jun 25, 2014
    Last edited: Jun 25, 2014
  15. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    It does not matter at all. Even with same name they are still separate entities.

    On of the Proxmox node if you run #ceph -s what result do you get?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0

    root@prox2:~# ceph -s
    2014-06-25 09:40:24.597315 7f44704fc700 0 -- :/1010268 >> 10.10.10.3:6789/0 p ipe(0xc990d0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0xc99340).fault
    2014-06-25 09:40:27.597304 7f44703fb700 0 -- :/1010268 >> 10.10.10.4:6789/0 p ipe(0xc9c9e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0xc9cc50).fault
    2014-06-25 09:40:30.597619 7f44704fc700 0 -- :/1010268 >> 10.10.10.3:6789/0 p ipe(0x7f446c001d90 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f446c002000).fault
    2014-06-25 09:40:33.597912 7f44703fb700 0 -- :/1010268 >> 10.10.10.4:6789/0 pipe(0x7f446c004010 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f446c004280).fault
    2014-06-25 09:40:36.598206 7f44704fc700 0 -- :/1010268 >> 10.10.10.3:6789/0 pipe(0xc9ba30 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0xc9bca0).fault
    2014-06-25 09:40:39.598356 7f44703fb700 0 -- :/1010268 >> 10.10.10.4:6789/0 pipe(0xc9e720 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0xc9e990).fault
    2014-06-25 09:40:42.598711 7f44704fc700 0 -- :/1010268 >> 10.10.10.3:6789/0 pipe(0x7f446c003ca0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f446c003f10).fault
    ^CError connecting to cluster: InterruptedOrTimeoutError

    I am reinstalling the server. This time will do what works before. Use the same subnet as the proxmox host.
     
  17. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    Does not seem like your Ceph nodes are talking to each other. Before you fully reinstall, i would suggest to try to make it work. There might be some underlying issue preventing this.

    Can you ping other Ceph nodes on Ceph subnet?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    Thank you. I already did the reinstall before seeing your message. But I can easily simulate that environment again.

    The strange thing is if I use the subnet same at the proxmox hosts, it works like a charm.

    Yes, I was able to ping all nodes on the Ceph subnet. For the life of me, I could not figure out what it is. If I exchange that subnet and use for the VMs, they also works fine.

    I even tried other subnets. Same problem. The only subnet it like is the proxmox host subnet.

    Thanks for your help.
     
  19. symmcom

    symmcom Well-Known Member

    Joined:
    Oct 28, 2012
    Messages:
    1,075
    Likes Received:
    25
    How many MONs have you created ?
    Ceph subnet on different switch?
    1 GB LAN or 10GB ?
    Whats the content of ceph.conf?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. impire

    impire Member

    Joined:
    Jun 10, 2010
    Messages:
    106
    Likes Received:
    0
    3 MONs created.
    Ceph subnet on different switch. Also tried on the same switch under separated VLAN.
    1GB LAN.
    I already reloaded so the content looks different now. I will need to reloaded it again to simulate the error one more time.

    It is working like a charm under the same subnet as the proxmox host. But I need to find out why it isn't working.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice