Proxmox 4, LXC and GlusterFS

[ianux]

Hello,

I just installed Proxmox 4 and I have a few question about a setup I thought of.

I have a few servers with a lot of disk space (hardware RAID, showing 1 physical volume) and I am wondering how to share this space among the cluster, thinking about HA and space optimization. I thought about distributed replicated GlusterFS volumes with 2 replicas, like some sort of RAID 5. The idea is to have one partition dedicated to LXC raw images and the other for backup (snapshot) purpose. Is it safe to have 2 GlusterFS on one disk ?

I have 2 NICs 1Gb/s, one for public traffic and the other used by PVE cluster. Can I use the later for GlusterFS ? Also, what is the overhead of using GlusterFS ? I think I have enough RAM and CPU power for running about 50 CTs amoung 5 or 6 servers, but what could be the bottleneck here ? Bandwidth ? Disk speed ?

My servers have about 8TB disk space each and I only need 2 or 3TB purely for production, backups excepted. What could be a good HA solution without wasting too much disk space ? Ceph is only documented with 2 nodes, and DRBD9 is presented as a technology preview, so I think GlusterFS is the only distributed filesystem that could be used like some sort of SAN. Is anyone has good production experience with that kind of setup ?

 

[wolfgang]

If you what to use shared or distributed storage you should have a dedicated network for this.
If you have cluster communication and storage on one network you get problems with quorum on load.
And you should have 10GBit or faster for this network.
Because the network storage have to transport minimum the written data form and this is approximately the same amount like the vm's write.
This is for all distributed storage types true.
So mostly the bottleneck are the network.
I would recommend you ceph for full feature LXC (snapshot are not working on GlusterFS).
Here in this wiki it is documented with 3 nodes.
https://pve.proxmox.com/wiki/Ceph_Server

 

[gosha]

You offer place LXC-container on CEPH block device? Are you sure?
Or may be You mean the CEPH FS?

 

[wolfgang]

LXC on Ceph with krdb as blk.

 

[gosha]

LXC on Ceph with krdb as blk.

What is "krbd"? May be drbd?

 

[wolfgang]

http://kernelnewbies.org/OutreachyProjects/CephCleanup

 

[gosha]

http://kernelnewbies.org/OutreachyProjects/CephCleanup

Thnks for link to Kernel implementation of the RADOS Block Device.
But I still do not understand how you propose to place the LXC-container on RADOS Block Device?
I've always believed that LXC-container is placed on the any File System, but not on the block device. I was wrong?

 

[wolfgang]

make a filesystem on a blk and you have a filesystem what you can mount everywhere you like.
it is the same way hdd and ssd work only you have a rbd blk and no real one.

 

[gosha]

Thnks. A good option.
And KRBD suitable for production cluster?

 

[dietmar]

And KRBD suitable for production cluster?

There is one open bug which makes problem on migration:

http://tracker.ceph.com/issues/13560

 

[gosha]

There is one open bug which makes problem on migration:

http://tracker.ceph.com/issues/13560

Thank you. I've tried. Interesting. Live migration can not do.

 

[tom]

LXC containers cannot be live migrated.

 

[gosha]

LXC containers cannot be live migrated.

It will always be so, or only temporarily?

 

[dietmar]

It will always be so, or only temporarily?

We will try to implement CRIU when that project is more stable (that enables live migration).

 

[gosha]

make a filesystem on a blk and you have a filesystem what you can mount everywhere you like.
it is the same way hdd and ssd work only you have a rbd blk and no real one.

I tried to implement this idea.
for testing I created rbd-image (size 40GiB):

rbd create backup-store --size 40960 --pool ceph_stor

rbd ls -l ceph_stor

NAME SIZE PARENT FMT PROT LOCK
backup-store 40960M 1
vm-100-disk-1 5120M 2
...

The image "backup-store" is present.

I'm trying to determine which disk corresponds to the created image via "fdisk -l".
And I do not see any drive (among all /dev/rbd*) of this 40GiB size...

I misunderstood something?

 

[MACscr]

Doesnt LXD offer some additional great features and also support live migration?

 

[dietmar]

Doesnt LXD offer some additional great features and also support live migration?

I think we have more features - at least our toolkit is better integrated into the proxmox cluster stack, and has better support
for different storage types, HA, full features network setup/integration ...