Understanding Proxmox Storage and CEPH

[briceburg]

I'm currently planning the architecture for a private cloud, and am leaning towards a Proxmox + Ceph setup. Basically Proxmox for the VMs and Ceph to provide an *expandable*, cost effective, and shared storage pool (as well as REST based S3 compatible object storage). Before committing to the hardware I want to fully understand how Proxmox uses storage on the compute nodes [ not answered via: http://pve.proxmox.com/wiki/Storage_Model ]. Specifically;

1. Under a SAN environment, is the KVM "image" copied to the compute node from the SAN before it is started up (similarly to Open Stack)? This would require at least enough local storage to cover the running VMs on a particular node. Or, alternatively, are the KVM images run directly from the SAN (e.g. via a NFS or CephFS mount)?

2. Do Proxmox KVM instances feature "ephemeral" root storage (the instance itself + runtime changes), and rely on mounted "volumes" [e.g. Amazon EBS] for persistent storage (similarly to Open Stack)?

3. Can CEPH's RBD be expanded (e.g. storage capacity increased by adding additional disks/OSDs) while Proxmox is actively using it as a Disk? I'm still unsure as to what aspects of Ceph Proxmox is leveraging via: http://pve.proxmox.com/wiki/Storage:_Ceph but assume it attaches to the RBD as an iSCSI target??

My preliminary setup looks like;

3 node proxmox cluster (each node: 96GB RAM, 2x E5-2609, 1x128GB SSD [OS], 2x2TB 7200RPM [Instance Storage??])
1 node database server (SSD backed local storage)

3 node ceph cluster (each node: 1x128GB SSD [OS], 6x2TB 7200RPM [Ceph OSD])
1 node Ceph Rados Gatway + Front End Proxy

Please let me know if something looks awry. Thanks for your pointers!

~ Brice

 

[tom]

See http://ceph.com/ceph-storage/block-storage/

Ceph RBD support on Proxmox VE is experimental but promising.

 

[spirit]

>>1. Under a SAN environment, is the KVM "image" copied to the compute node from the SAN before it is started up (similarly to Open Stack)? This would require at least enough local storage to cover the running VMs on a particular node. Or, alternatively, are the KVM >>images run directly from the SAN (e.g. via a NFS or CephFS mount)?
kvm running directly from san. They are no copy vm from image source/template for the moment.
>>2. Do Proxmox KVM instances feature "ephemeral" root storage (the instance itself + runtime changes), and rely on mounted "volumes" [e.g. Amazon EBS] for persistent storage (similarly to Open Stack)?
like EBS,permanent storage.

>>3. Can CEPH's RBD be expanded (e.g. storage capacity increased by adding additional disks/OSDs) while Proxmox is actively using it as a Disk? I'm still unsure as to what aspects of Ceph Proxmox is leveraging via: http://pve.proxmox.com/wiki/Storage:_Ceph but >>assume it attaches to the RBD as an iSCSI target??
No, RBD use his own protocol,include in kvm, no iscsi

 

[briceburg]

OK Thanks for the responses. From what I gather; running the VMs over the SAN would allow me to drop the 2TB drives from the Proxmox Nodes and just have the 128GB SSD drive for Operating System. Is this correct? It also puts further emphasis on SAN throughput/latency as there will be more I/O between compute node and SAN than there would be under compute node and local storage...

From this I also gather that my #2 question is irrelevant in that there is no "ephemeral" storage. Is this correct?

I am a bit worried about SAN (ceph) performance to go this route... has anyone done this?

~ Brice

 

[spirit]

I have benched around 20000io/s with a 3 nodes x 6 ssd ceph cluster

 

[jkirker]

@briceburg, as I resurrect an old thread I'm curious what you ended up doing and what your experiences have been so far? I'm also in the process of evaluating and rolling out this sort of model. Please let me know. Thanks...