Existing Node - Create new Ceph cluster

[puntoboy]

Hello,

I have an existing PVE node that has been working for a few years. Single node, pretty basic, single 128GB SSD. It currently runs two VMs and backs up to my Synology NAS.



I have just bought a second node, same hardware, with only an extra SSD (I want to add another SSD to the first node as well).



Node1 (existing) us running Proxmox 7.3-4.
Node2 (new) has a fresh install of 8.2.2

I would like to create a Ceph cluster for both nodes (unless you have a better suggestion - not using the NAS for shared storage). What is the best way to do this?

Create cluster
Configure Ceph
Migrate VMs
Upgrade node1 to 8.2.2

Does that sound the best way? Or is there a better way? Ideally I'd like to keep the VMs online a much as possible.

 

[gurubert]

Do not run Ceph on just two nodes.

 

[puntoboy]

Do not run Ceph on just two nodes.

So what's the recommendation when using local storage?

 

[puntoboy]

Since you mentioned it, I've been looking into it some more. Seems I need a third device to act as Quorum or a CoroSync QDevice so there are 3 nodes in the cluster.

This is an image I found on this forum which is similar to my setup (just without the WAN doing to OPNSense).



Would this setup work?

 

[alexskysilk]

This is an image I found on this forum which is similar to my setup

that describes how to do pve clustering with two real nodes and a quorum. this doesnt address ceph.

 

[puntoboy]

that describes how to do pve clustering with two real nodes and a quorum. this doesnt address ceph.

OK, so what else do I need?

 

[gurubert]

With only two nodes and the need for shared storage on local disks the only option is to integrate DRBD as far as I know.

Linbit offers a Proxmox plugin but this is not officially supported by Proxmox.

 

[puntoboy]

Thanks.

Is GlusterFS with two nodes not a viable solution for a small home lab? https://jon.sprig.gs/blog/post/2885

 

[alexskysilk]

Is GlusterFS with two nodes not a viable solution for a small home lab? https://jon.sprig.gs/blog/post/2885

It really depends on your skill, comfort level operating without support, and you goal. There are 2 "proper" method to provide shared storage in your scenario:
1. syncronized zfs- https://pve.proxmox.com/wiki/PVE-zsync. advantages are its easy and performant; the downside is that its asynchronous.
2. external filer such as trunas, sharing either iscsi or nfs.

glusterfs CAN be used, but it's a second class citizen under pve. You can expect stability and potential data corruption issues that you'll have to detangle on your own.

 

[puntoboy]

Since my last post I have been looking at ZFS replication, is that the same as synchronised ZFS? On the face of it they look like the same thing.

I have Synology and UNRaid NAS already for other things, but at the moment I want to keep that separate to Proxmox.

So my intention is a 2-node Proxmox cluster with ZFS replication (zsync) and a Corosync QDevice running as a Docker/VM on Synology NAS. 15 minute replicas should be fine for what I want. I do daily backups to PBS so that's not a problem.