Add ZFS storage

[dixie2000]

Hello,

First I am relatively new to Proxmox. I have a 2 node PVE setup and want to be able to replicate the VM's and CT's on node 1 to node 2. I have reviewed many documents and done several Google searches and can't seem to find a solution.

1. Do I need a ZFS pool or something else?
2. Can I add what might be required (ZFS) to my existing configuration or do I need to start over?

Any help is certainly appreciated!

 

[Dunuin]

Do I need a ZFS pool or something else?

Yes, replication is based on ZFS native features. So this is a requirement.

Can I add what might be required (ZFS) to my existing configuration or do I need to start over?

If you got some unused Enterprise SSDs or CMR HDDs you could add them and create new ZFS pools. Make sure they got the same name on both nodes.

I have a 2 node PVE setup

I hope you also got a third machine working as your qdevice, as a proper cluster needs 3+ machines.

 

[dixie2000]

@Dunuin

Thank you for your reply.

My second node is a duplicate, hardware wise, of node one. I have nothing running on it at the moment. It has a 500GB NVME drive in it.

1. Can I utilize that drive and configure as a ZFS drive?
2. Does Proxmox need to be reinstalled on this node or can I somehow just reformat the NVME drive and configure as ZFS?

Thanks!

 

[Dunuin]

1. Can I utilize that drive and configure as a ZFS drive?
2. Does Proxmox need to be reinstalled on this node or can I somehow just reformat the NVME drive and configure as ZFS?

You would need to reinstall PVE on both nodes using ZFS if they only got a single disk that isn't used with ZFS yet.
And again, there are no 2-node clusters. You need 3 nodes or 2 nodes + some third physical machine as third voter for quorum. Please read the full wiki article for cluster basics, design considerations and limitations: https://pve.proxmox.com/wiki/Cluster_Manager

 

[dixie2000]

You would need to reinstall PVE on both nodes using ZFS if they only got a single disk that isn't used with ZFS yet. Not looking forward to that. I wish there was some way to backup all the configuration etc. I do have Proxmox Backup Server running so I assume I can restore all my VM's and CT's from it.

And again, there are no 2-node clusters. You need 3 nodes or 2 nodes + some third physical machine as third voter for quorum - I have a two node cluster currently setup and using an old Raspberry Pi as a Qdevice.

 

[Dunuin]

Not looking forward to that. I wish there was some way to backup all the configuration etc. I do have Proxmox Backup Server running so I assume I can restore all my VM's and CT's from it.

Yes, but only the VMs/LXCs and not all the node/datacenter/host configs.

I have a two node cluster currently setup and using an old Raspberry Pi as a Qdevice.

Ok, great

 

[dixie2000]

@Dunuin
Is there by chance any way to "backup" the node/datacenter/host configs? Like maybe a copy of the etc/pve directory.

 

[Dunuin]

Its not that easy as PVE isn't an appliance but a full-fledged Linux distribution with thousands of packages and config files. How hard this gets depends on how well you optimized, secured and monitored your PVE node. The more you do, the harder it gets. Similar to Windows, where copying the C:\Users folder isn't enough to backup/restore the whole OS with all installed programs and configs.

One option would be to copy the config.db: https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)#_recovery
Another one to google for some third-party backup/restore scripts. There are tons on github like this: https://github.com/DerDanilo/proxmox-stuff

 

[dixie2000]

@Dunuin

Well thanks again for all your help, guess I need to do some in depth thinking and planning. When I started my Home Lab with Proxmox a year ago I had one PVE Server, I then added another and a backup server as well. Now, and obviously after the fact, I want to have some fail over functionality, either HA or replication process. I suppose I could go on the way I am and if node one fails just power up node 2 and restore all the VM's and CT's and move on from there.

I may just setup Proxmox on an old PC I have around, move all the VM's and CT's to it and then start from scratch with my current setup.

Live and learn I guess

 

[Dunuin]

I suppose I could go on the way I am and if node one fails just power up node 2 and restore all the VM's and CT's and move on from there.

Yes, for a homelab where downtime isn't critical thats a valid option. Not doing it differently here with 5x stand- alone PVE nodes and 3x PBS.
With raid, UPS and daily backups it's pretty secure and easy to manage and if a node runs into a problem I could always restore the last daily backup to some other node or boot a rescue ISO to fix stuff or move the disks to some other machine to get access to the latest data.

The annoying part without a cluster is to keep config files like IP sets, security groups, aliases and so on in sync between the unclustered nodes so guests restored to another host won't stop working because of missing firewall definitions. Unluckily those are all part of the datacenter config and not included in the VM/LXC backups and there is no way to sync the datacenter firewall configs between different datacenters.

 

[dixie2000]

@Dunuin

Well after a good bit of reading, research and thinking (resulting in a good bit of confusion) here is what I hope to be final question to you on this topic.

Goal:
To be able to replicate VM's and CT's to the second node for redundancy and possibly use HA.

Facts:
I have a 2 node cluster and a raspberry pi as a Q device
All VM's and CT' are backed up to a Proxmox Backup Server
Each node has an SSD that has Proxmox OS on it
Each node has an NVME drive (ext4)

Question:
Can I wipe and initialize with GPT each NVME and then configure the NVME drives using ZFS and then restore by VM's and CT's as needed?

Again, thank you for taking the time to help me on this, it is appreciated!
Al...

 

[Dunuin]

Can I wipe and initialize with GPT each NVME and then configure the NVME drives using ZFS and then restore by VM's and CT's as needed?

Yes. With dedicated disks for VM storage and proper backups this is super easy.
Keep in mind that PVE nodes should be empty when adding them to a cluster.
And if you want to move guest via backup+restore between nodes you better shut dem down on node a, after that do the backup and only start them on the new node.
I would avoid snapshot mode backups so you don't kind of "crash" the guestOS.

 

[dixie2000]

@Dunuin

When you say "Keep in mind that PVE nodes should be empty when adding them to a cluster." The NVME drives will be empty but the SSD with the PVE OS and VM's and CT's will be on its "local-lvm", is this OK?

 

[Dunuin]

When you say "Keep in mind that PVE nodes should be empty when adding them to a cluster." The NVME drives will be empty but the SSD with the PVE OS and VM's and CT's will be on its "local-lvm", is this OK?

No, the node shouldn't contain any VMs/LXCs when adding it. See the Wiki: https://pve.proxmox.com/wiki/Cluster_Manager

All existing configuration in /etc/pve is overwritten when joining a cluster. In particular, a joining node cannot hold any guests, since guest IDs could otherwise conflict, and the node will inherit the cluster’s storage configuration. To join a node with existing guest, as a workaround, you can create a backup of each guest (using vzdump) and restore it under a different ID after joining. If the node’s storage layout differs, you will need to re-add the node’s storages, and adapt each storage’s node restriction to reflect on which nodes the storage is actually available.

 

[dixie2000]

You said "All existing configuration in /etc/pve is overwritten when joining a cluster."

I already have the cluster setup, several weeks ago. I was hoping I could just reconfigure the NVME drives in each machine as they currently have nothing on them.