Proxmox VE 5.1 and Ceph Filesystem

[Saiki]

Hello everyone,

Context :
I need to setup HA architecture between two datacenters for virtualization and distributed storage system. This latter would be used as block devices and filesystems for Proxmox virtual machines and also our legacy infrastructure.

Implementation :
- 2 datacenters
- 4 hosts (2 by datacenters)
- 4 OSD on each hosts
- Proxmox 5.0
- Ceph Luminous

Questioning :
I have the feeling Proxmox with Ceph is very suited for the case where the storage would be used as block devices. However, it seems that Ceph Filesystem is not very well integrated within the current Proxmox VE version 5.0. Further installation and configuration must be done in order to enable Ceph Filesystem use (example of thread). In addition, ceph-deploy tool is not compatible (Proxmox wiki), deployment of ceph client is not facilitated.

I have the feeling Proxmox with integrated Ceph seems to not completely fulfill my needs for my case.
I am currently thinking about installing Proxmox VE and independently deploy Ceph on the 4 hosts in order to take full advantages of both systems.

Will the Proxmox VE 5.1 release integrate these elements ?
Please, feel free to share your advices.

Thank you very much.

Best regards.

 

[jeffwadsworth]

You are going to want 3 nodes at each datacenter.

 

[guletz]

Hi,

Maybe you ask the wrong question . The wright question maybe is what is your goal, like if my primary DC is down, I want to start my VMs in the secondary DC.
What if your both DC lose the nework link? Split brain? In my opinion HA with only 2 resurces (aka data center) is only a dream. Any serious HA system has 3 resources(proxmox, databases, whatever )

So, put the Wright questions if you want the correct answers

 

[MewBie]

While it is relatively easy to run CephFS in Proxmox, I got it to work in a couple of evenings the first time, I wouldn't use it in production without extensive testing and research.

And as already been mentioned you want at least 3 nodes in each datacentre.

 

[udo]

And as already been mentioned you want at least 3 nodes in each datacentre.

Hi,
but this isn't enough!

If the connection between the DCs are lost, no DC has an quorum. E.G. all rbd-volumes are read only and all VMs hang after a very short time.

You can work with two nodes in each DC - with three DCs...

But the question is, if this is usefull? I think the latency between the DCs will kill any performance so that the VMs feel laggy (the write ack will given, if all journal writes are acknoledged).

Udo

 

[MewBie]

Hi,
but this isn't enough!

If the connection between the DCs are lost, no DC has an quorum. E.G. all rbd-volumes are read only and all VMs hang after a very short time.

You can work with two nodes in each DC - with three DCs...

But the question is, if this is usefull? I think the latency between the DCs will kill any performance so that the VMs feel laggy (the write ack will given, if all journal writes are acknoledged).

Udo

I don't know if it's possible with Proxmox by default, but if you use an active-passive type setup it might work with 2 DCs.

Clients always connect to DC1. DC1 syncs with DC2, if they lose the connection between each other that's not a problem they just don't sync temporarily. If the clients can't reach DC1 they fail over to DC2. You will have a problem if the DCs don't sync and your primary DC goes down.

 

[udo]

I don't know if it's possible with Proxmox by default, but if you use an active-passive type setup it might work with 2 DCs.

Clients always connect to DC1. DC1 syncs with DC2, if they lose the connection between each other that's not a problem they just don't sync temporarily. If the clients can't reach DC1 they fail over to DC2. You will have a problem if the DCs don't sync and your primary DC goes down.

Hi,
how should that work with ceph?
Perhaps you can build this with drbd, but afaik not with ceph. But I'm happy to learn, if there are an posibility.

Udo

 

[MewBie]

Hmm... I'm not a network architect but as far as PVE and Ceph would be concerned there is only 1 DC with 3 nodes. The failover would be outside of PVE. The sync between the DCs I don't know. Something like that might work.

 

[aderumier]

Hi,
how should that work with ceph?
Perhaps you can build this with drbd, but afaik not with ceph. But I'm happy to learn, if there are an posibility.

Udo

for rbd, you can use rbd mirror with async replication to another ceph cluster
for radosgw, you can mirror objets to a remote ceph cluster
but for cephfs, they a no async replication currently.

(I have they are rados async replication on the ceph roadmap), but currently it's done client side (rbd && radosgw)

 

[guletz]

... but if ceph is setup on top of zfs, then a zsync can be done for async replication.

 

[udo]

... but if ceph is setup on top of zfs, then a zsync can be done for async replication.

Hi,
I think it's an bad idea to use zfs as underlying FS. ceph tried to minimised the supported FS (cancel ext4 (afaik btrfs also)), only xfs + (nofs) bluestore are left.
I assume the goal is to support bluestore only.

Udo

 

[udo]

for rbd, you can use rbd mirror with async replication to another ceph cluster

Hi,
right - an async replication make sense. But all VM storage-access must done through the DC-Interconnection...

Udo

 

[aderumier]

Hi,
right - an async replication make sense. But all VM storage-access must done through the DC-Interconnection...

Udo

it's active/backup , for disaster recovery for example.

you have vms on DC1 with ceph1 , and mirroring to DC2 with ceph2 (standby).
it's per pool, so it's possible to do dual active-backup with 2 pools, with vms running on their master pool on each side.

 

[guletz]

Hi,
I think it's an bad idea to use zfs as underlying FS. ceph tried to minimised the supported FS (cancel ext4 (afaik btrfs also)), only xfs + (nofs) bluestore are left.
I assume the goal is to support bluestore only.

Udo

Thx Udo, my knowledge about ceph are like /dev/null .

 

[alexskysilk]

how should that work with ceph?

You said the magic word (three.) mind you, the third DC does not have to have a full complement of nodes or even disk- just one node running a monitor. that would be sufficient to provide quorum. You could (theortically) run the third node from a PC at home.

The greater point is- to what end? without pretty substantial bandwidth between the three data centers latency would go though the roof. see http://tracker.ceph.com/projects/ceph/wiki/Can_Ceph_Support_Multiple_Data_Centers