[SOLVED] Advice for my setup (ZFS 3 nodes, storage?)

woutwoot

Active Member
Jul 17, 2017
8
0
41
28
Hi everyone. I'm looking for some advice on how I should set up my storage and backups. I'll first share what hardware I currently have and what I need to be able to do with it.

This is what I have as for storage. I've got 4 disks (+ 1 small SSD for proxmox) per node, each 2 matching disks configured as a ZFS mirror for redundancy.

Physically the blades look like this:
Node1: 500GB SSD 500GB SSD 1TB SSD 1TB SSD
Node2: 500GB SSD 500GB SSD 2TB HDD 2TB HDD
Node3: 500GB SSD 500GB SSD 2TB HDD 2TB HDD

Each node has 128GB of RAM, while I'll need about 80GB for VMs per node (probably less). There should be more than enough left for ZFS.

What I want to be able to do is have some containers running on one node, and use 2 nodes to each host a big KVM VM. If something goes down that's not a big issue, but I should be able to recover it on another node in less than an hour. My ZFS mirrors should already protect against disk failure.

It seems I need to be able to use replication between nodes, as I don't have actual shared storage available. Because of the same reason HA is probably not a good idea, but it's also not strictly required.

I've currently configured part of this setup (2 nodes) but already I can't migrate a container between the two nodes because my storage names don't match? How should I set this up, do you have any advice/suggestions?

I have 2 10Gb connections per node, one for cluster communication and one for the rest.
 
Hi,

I would make a symmetric zfs setup and use pvesr for the replication.
In this case, you have your guest data on two of this three nodes and in case of a node failure, you can start the guest.
But keep in mind pvesr is an asyncron replica so the data state on the target node can be older than the origin one.
 
Hi,

I would make a symmetric zfs setup and use pvesr for the replication.
In this case, you have your guest data on two of this three nodes and in case of a node failure, you can start the guest.
But keep in mind pvesr is an asyncron replica so the data state on the target node can be older than the origin one.
Does this mean I need to rename my storages so they have the same name? Size difference does not matter? I could not name them the same via the GUI. (for the 2TB HDD vs 1TB SSD raid 0) Some data being lost is acceptable as I'm hoping that having to recover on another node will be a very rare event.
 
Does this mean I need to rename my storages so they have the same name?
Yes
Size difference does not matter?
No, all nodes storages are handled separately.

You can rename the zpool on the command line.
Code:
zpool export <poolname>
zpool import <poolname> <new poolname>
 
Yes

No, all nodes storages are handled separately.

You can rename the zpool on the command line.
Code:
zpool export <poolname>
zpool import <poolname> <new poolname>
Thank you, If I understand correctly this means I need to rename my ZFS pools as well as my storage entries in /etc/pve/storage.cfg? Final result would be a "left" and "right" pool and storage entry with the same name on each node representing the two drives on the left/right of the blade. Is that correct?

Edit: this is working great. Thanks!
 
Last edited:
In this case, you have your guest data on two of this three nodes and in case of a node failure, you can start the guest.

In case of a node failure one has to do what exactly? According to pve the vms are still running on the defect node.
How can one moved instances from one node to another if the source is dead.

Does the cluster allow to change vm config files in /etc/pve without quorum in this case?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!