Storage config for replication

J

jackdaw

New Member
Jun 23, 2022
6
0
1
Dear Proxmox staff and community,

Just want to say first what an excellent piece of software this is and I look forward to using it more and more.

Now for my question. I have a 3 node cluster with the following specs:

node 1 and node 2:
2x AMD Epyc 7453
256GB RAM
2x 256GB NVMe (ZFS Mirror for Proxmox OS)
1x 1TB NVMe (ext4, directory for ISOs, templates, etc)
10x 2TB NVMe (currently configured in RAIDZ2 with different names for each node)

node 3: same as nodes 1 and 2 except instead of the 10x 2TB NVMe drives, there are 9x 4TB SSDs (RAIDZ2).

I understand that for the replication to work, the pools need to have the same name, and since the setup is new I have no problem reconfiguring it.

I would just like to know can I add all of this storage to the Datacentre even though the pool on node 3 is different in size and drive layout?

Any help would be greatly appreciated.

Thank you!
 
The size and drive layout isn't important.
Just make sure if storages are only available on a subset of nodes, that you limit it to those in the storage configuration.

You can export the pool and import it again under a different name, see man zpool-export and man zpool-import.
For this to work, the pool can't be in use at that moment. So disable it in the storage config and maybe reboot afterwards to make sure nothing accesses it.
 
  • Like
Reactions: jackdaw and Moayad
Many thanks for your reply, mira.

Am I correct in saying that if I want all of my storage to be available in all of my nodes, I should select all three nodes when configuring the storage in the Datacenter -> Storage section?
 
Yes, or if you select none, all nodes in the cluster will be assumed.
 
LnxBil said:
Why did you go the ZFS route instead of CEPH? Seems for the superior solution for HA setups.
Click to expand...
I found the ZFS route to be less complicated as I don’t know much about CEPH. And also I was just looking for a simple way to replicate VMs onto another node. However I would be happy to read up on it if you have any good beginner guides to link to.
 
jackdaw said:
And also I was just looking for a simple way to replicate VMs onto another node.
Click to expand...
I lofe ZFS but I would never use ZFS asynchronous replication for my datacenter. I have requirements that are only fullfilled by synchronous replication, which CEPH fullfills.

jackdaw said:
However I would be happy to read up on it if you have any good beginner guides to link to.
Click to expand...
You mean besides the PVE documentation?
 
LnxBil said:
I lofe ZFS but I would never use ZFS asynchronous replication for my datacenter. I have requirements that are only fullfilled by synchronous replication, which CEPH fullfills.


You mean besides the PVE documentation?
Click to expand...

That is interesting, I should learn more about it.
Could you describe the system config and use case where you are running CEPH?

While the PVE documentation is excellent, I was looking more for something like a brief primer or a getting started guide. I know it’s pretty trivial to do a web search but it’s always nice to see what experienced users recommend.
 
LnxBil said:
I lofe ZFS
Click to expand...
Damn, I should more carefully read what I write :mad:

jackdaw said:
Could you describe the system config and use case where you are running CEPH?
Click to expand...
Anything hyper-converged-related like you have with nodes WITH a lot of storage (distributed shared storage). In contrast the non-hyper-converged setup would require some kind of HA storage outside of your nodes (e.g. iSCSI or FC SAN, which then is dedicated shared storage) to yield the same data availability and consistency requirements.

There is a CEPH benchmark on the Proxmox website that described the setup well.
 
LnxBil said:
Damn, I should more carefully read what I write :mad:


Anything hyper-converged-related like you have with nodes WITH a lot of storage (distributed shared storage). In contrast the non-hyper-converged setup would require some kind of HA storage outside of your nodes (e.g. iSCSI or FC SAN, which then is dedicated shared storage) to yield the same data availability and consistency requirements.

There is a CEPH benchmark on the Proxmox website that described the setup well.
Click to expand...
I see what you mean. And it is definitely worth investigating further. One of the options for us was shared storage but our budget didn’t allow it.

Many thanks for that link. I will have a more detailed look at it.

Also many thanks for the explanation.
 
You must log in or register to reply here.
Top Bottom
Back