HA storage for web cluster on PVE/Ceph

[hansm]

ATM we're running a webserver cluster for a customer with 3 web vm's and 1 NFS fileserver VM for the doc root and dynamic user files, uploaded through the app of our client. This NFS server is the SPOF in the setup and we want to make the file storage HA. Off course the current NFS vm is on pve with HA enabled, but still it can crash or it needs maintenance and we want to reduce the chance on downtime of the central web cluster storage.

I tried GlusterFS a few years ago for this but it's performance is poor and I had some issues with data corruption. The solution I'm looking for needs to be a lot faster and reliable.

Our current PVE cluster consists of multiple PVE nodes and a separate Ceph cluster without CephFS but we're planning a new cluster based on PVE6 with Ceph on the PVE nodes.

I'm thinking of solving my "issue" with CephFS but I'm unsure about the possibilities and how this would fit in. This customer isn't the only one using our PVE cluster and data seperation is very important. Connecting directly to the Ceph(FS) cluster doesn't feel very safe in the context of data seperation/security.

Is CephFS on the PVE nodes usable for this and how would I set this up? Or do you advice another option? I rather don't use DRBD/corosync/pacemaker as this adds complexity, I want to keep is as simple as possible to avoid problems.

 

[ertanerbek]

ZFS+GlusterFS+GlusterFS-Tier System+ Some modification for more IO for small file.. By the way where you want use that storage system. on virtualization side or in WEB server ? If in WEB server DRBD+OCFS2

 

[hansm]

It's for the web servers. For VM storage we use Ceph RBD.
My experiences with OCFS2 aren't that great, I can't recall the problems I had but as far as I remember it had something to do with locking. Besides that, our Ceph cluster is configured with 3 replicas and it feels very inefficient to use DRBD where 2, or more servers store the same data. In the current situation 3 VM's store the same data 3 times, that means I got 9 copies of the data. Also, I think there will be 5 web servers next year and maybe more in the next few years. In my opinion the storage should be centralized and redundant (HA). I would like to know if CephFS on the Proxmox nodes can be used for this and how I can do that best.
Thanks.

 

[czechsys]

Options:
1] 2x HW raid on the servers + drbd + nfs. Raid10 + drbd = replica 4
2] drbd9 - use diskless nodes for webservers, drbd nodes on pve (need check sanity and performance)
3] cephfs - faster than linux nfs over ceph rbd, but harder to maintain
4] galera nfs?
etc.

 

[ertanerbek]

It's for the web servers. For VM storage we use Ceph RBD.
My experiences with OCFS2 aren't that great, I can't recall the problems I had but as far as I remember it had something to do with locking. Besides that, our Ceph cluster is configured with 3 replicas and it feels very inefficient to use DRBD where 2, or more servers store the same data. In the current situation 3 VM's store the same data 3 times, that means I got 9 copies of the data. Also, I think there will be 5 web servers next year and maybe more in the next few years. In my opinion the storage should be centralized and redundant (HA). I would like to know if CephFS on the Proxmox nodes can be used for this and how I can do that best.
Thanks.

I think I miss understand you. First Question, your all WEB server will work active+active am I right ? If yes you need ClusterAvare system on your WEB server. Ceph,GlusterFS,OCFS2,NFS etc... all of tham ClusterAvare system.. For Proxmox Ceph solition that is for virtualization means you QCOW file...

 

[hansm]

Options:
1] 2x HW raid on the servers + drbd + nfs. Raid10 + drbd = replica 4
2] drbd9 - use diskless nodes for webservers, drbd nodes on pve (need check sanity and performance)
3] cephfs - faster than linux nfs over ceph rbd, but harder to maintain
4] galera nfs?
etc.

1 and 2) I don't think this is a viable option. It's one of our customers on our platform, we can't modify our whole PVE cluster setup for just this customer. Our new setup consists of 5 PVE nodes with NVMe disks for Ceph VM storage. That means all our VM's are assigned a virtual disk on our Ceph storage cluster by RBD. On this cluster we run virtual machines for this specific customer, for example the current 3 web servers. These web servers need shared (cluster aware) storage, that's the whole point of my question. I'm using NFS for this at the moment and am happy with it but it isn't HA enough. If the NFS VM goed down, all web servers can't be used anymore. That's why I'm looking for a more redundant setup. The question could be, "how to make my NFS server redundant", but the answer to that question would be to use DRBD with 2 NFS VM's. That's not what I want, just like GlusterFS.

3) is more what I'm looking for, but how?
4) I think you mean NFS-Ganesha? I'm not sure why that would be a better fit than using CephFS directly? If CephFS can be used directly from the web servers than the current NFS server can be removed.

I think I miss understand you. First Question, your all WEB server will work active+active am I right ? If yes you need ClusterAvare system on your WEB server. Ceph,GlusterFS,OCFS2,NFS etc... all of tham ClusterAvare system.. For Proxmox Ceph solition that is for virtualization means you QCOW file...

1) Yes, all web servers work active/active, there's a (redundant) load balancer in front which balances the load evenly across the web servers.
I know I need a cluster aware file system, that's why I'm using a NFS server but that's the only part of the cluster which isn't redundant. The NFS server VM is on our PVE cluster and is HA, so when a PVE node fails it's migrated directly to a running one but it needs to be more fault tolerant. You're right that GlusterFS and OCFS2 are possible solutions for this problem, have you tried them yourself in a real life setup? Were you happy with it? I wasn't, performance wise it's not great and is also has some data corruption issues. NFS isn't redundant by design and it's a pain to do this right and for manageability it isn't great either. CephFS is what I want to use and our PVE cluster uses Ceph RBD already so this would be a logical choice. But how do I use this and how to bind it to the web servers?

 

[ertanerbek]

Why you do not continue with NFS ? cause of IO, if yes grow up your "RPCNFSDCOUNT"

 

[hansm]

Why you do not continue with NFS ? cause of IO, if yes grow up your "RPCNFSDCOUNT"

It's not for more I/O. I already wrote this:

This NFS server is the SPOF in the setup and we want to make the file storage HA

And this:

I'm using a NFS server but that's the only part of the cluster which isn't redundant

I'm looking for the solution that comes closest to 100% uptime.

Thanks for your suggestions anyway.

 

[ertanerbek]

GlusterFS+NFS Genesha + VRRP on your network