Requirements for PBR/PVE/Storage Replication/Cluster - am I missing something?

[markf1301]

Current config for reference:
new 8.x pve server with ZFS for storage and hosting approx 21 machines.
old server getting repurposed with 8.x pve, ZFS, and PBR.
no san/external nas used for 'data' - only on the 2 'main' boxes.
misc pc to be loaded with 8.x pve and used for quorum purposes.
DR site will have a 8.x pve/pbr for obvious reasons and long term storage of backups.


I see in the documentation that a cluster is required for HA (3 node minimum for Corosync) but it seems like what I've found for Storage Replication is that only 2 devices are required. I'll be using 3 devices for the quorum so that's not an issue. Regarding Storage Replication I've only found the following requirement: ZFS. I currently, and will have, a ZFS partition on the 'box to be replicated' to so that won't be an issue either. That said, does the names of the ZFS partition need to be the same or does that matter as long as it is configured with ZFS and the volume/space is large enough to accommodate the 'replicas'? I presume the 'tools' in ZFS are whats keeping track of the data (snapshots and qcow2 files) but wanted to be sure before i started.

I've seen in the documentation that you can either add the PBR to your existing PVE installation, or run it as a vm to manage your backups. I'm hoping I can do file level restores with the PBR and not have to do 'full' reloads as well as reduced snap/backups but haven't gotten that far yet. What do most people do and what seems the most stable or practical from a management and data access perspective?

With replication set and working - will HA work or will I have to migrate the 'config' data to the new server and then be able to bring it back up then? If i copy that data from Server1 to Server2 (/etc/pve/nodes/file.conf) will it work, or will the config still be showing as 'attached' to server 1 whenever server 2 comes back up?

 

[Chris]

That said, does the names of the ZFS partition need to be the same or does that matter as long as it is configured with ZFS and the volume/space is large enough to accommodate the 'replicas'? I presume the 'tools' in ZFS are whats keeping track of the data (snapshots and qcow2 files) but wanted to be sure before i started.

A ZFS backed storage with the same name needs to exist on both nodes in order for ZFS replication to work. Of course, there needs to be enough space to store the volumes and snapshots to sync on both.

I've seen in the documentation that you can either add the PBR to your existing PVE installation, or run it as a vm to manage your backups.

Please note that a co-installation of Proxmox VE and Proxmox Backup Server is not recommended for disaster recovery reasons. It is recommended to use a dedicated host [0].

With replication set and working - will HA work or will I have to migrate the 'config' data to the new server and then be able to bring it back up then? If i copy that data from Server1 to Server2 (/etc/pve/nodes/file.conf) will it work, or will the config still be showing as 'attached' to server 1 whenever server 2 comes back up?

HA will work with storage replication, but with the potential of some data loss. Therefore a shared storage would be better if that should be avoided. Also, manual recovering the replicated guest is an option, see further details described in the docs [1].

[0] https://pbs.proxmox.com/docs/installation.html#install-proxmox-backup-server-on-proxmox-ve
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvesr

 

[markf1301]

A ZFS backed storage with the same name needs to exist on both nodes in order for ZFS replication to work. Of course, there needs to be enough space to store the volumes and snapshots to sync on both.


Please note that a co-installation of Proxmox VE and Proxmox Backup Server is not recommended for disaster recovery reasons. It is recommended to use a dedicated host [0].


HA will work with storage replication, but with the potential of some data loss. Therefore, a shared storage would be better if that should be avoided. Also, manual recovering the replicated guest is an option, see further details described in the docs [1].

[0] https://pbs.proxmox.com/docs/installation.html#install-proxmox-backup-server-on-proxmox-ve
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvesr

Chris - thanks for the info but still have a few questions.

From reading the document on your link (1) - the documentation only states the Storage Replication process will replicate guest volumes (the actual server info) from server-1 to server -2. Further in the documentation it states that you have to have a ZFS pool on both sides.

The pvesr command-line tool manages the Proxmox VE storage replication framework. Storage replication brings redundancy for guests using local storage and reduces migration time. It replicates guest volumes to another node so that all data is available without using shared storage. Replication uses snapshots to minimize traffic sent over the network.

.​

Makes sense as the 'zfs tools' and environment needs to be the same in both places. It doesn't state that the ZFS pool has to be the named the same on both servers. Probably not an issue for me as I'm just starting with this environment but did I miss something within the document, or another 'sub-link' somewhere that states that the ZFS volume NAME - to be replicated (source 1 to source 2) has to be called the same on both servers? The info being replicated is just a 'dataset' name anyway, isn't it? I have been using ZFS replication from Server1 to Server2 within another environment - with different ZFS pools w/o issues for many years. Isn't the 'datastore' just a name on the file system anyway? It seems like this process has multiple layers to it: one is the location of the data, and the 2nd layer, possible since you are using ZFS, then you are using the ZFS tools - to replicate 'location 1 to location 2' - which would seem like it's 1 process or item but is actually 2 different processes to actually complete the task.

Regarding the 'dual' installation - I get it. Multiple devices serving different functions makes sense except for the fact that utilizing Virtual systems why companies virtualize everything to reduce costs and management. No company wants to say that you can potentially do 2 or 3 things on the same box (pve and pbr) from a support process, but your documentation states that it can be done in either environment. "Separate physical servers are recommended due to the potential failure of a single box - but isn't this also or the same issue if you do have 2 physical boxes (pve and pbr) and the pbr box goes down? It would seem that in either solution (dedicated PBR and PVE servers) there are potential issues and processes that need to be managed. Whether you use a 2 physical or 1 with both servers on it, pve and pbr as a virtual box server - aren't you potentially having the same issues if one of your servers goes down?

Again, thanks for the info and please let me know if I'm wrong - or trying to do to much with your solutions.

mark

 

[Chris]

Makes sense as the 'zfs tools' and environment needs to be the same in both places. It doesn't state that the ZFS pool has to be the named the same on both servers.

Actually, you will have to have the same storage layout for each VM disk to replicate, e.g. you have 2 disks, located on two different pools, then you will need to have both of these pools available on the target node as well.

The info being replicated is just a 'dataset' name anyway, isn't it?

The info being replicated is a zfs snapshot, send to the target, but the datasets and snapshots live on zpools, so you will have to have the underlying storage structure for the snapshots to be send over to the target.

Otherwise it could grow rather complex, as then you would need to tell the job which disk resides on what storage on the target node. As is, it is a simple one to one mapping.

but your documentation states that it can be done in either environment

Yes it is possible to visualize your PBS instance or co-install it on the PVE hypervisor, in fact a lot of people do that. But you must be aware of possible limitations, as the documentation tells you:

Installing the backup server directly on the hypervisor is not recommended. It is safer to use a separate physical server to store backups. Should the hypervisor server fail, you can still access the backups.

Whether you use a 2 physical or 1 with both servers on it, pve and pbr as a virtual box server - aren't you potentially having the same issues if one of your servers goes down?

But if the node with your mission critical VM goes down, and your PBS VM resides on the same host, you cannot restore the mission critical VM from backup, unless you bring the host up again.

 

[markf1301]

It was mentioned that I needed a 'like' pool for replication between 2 different nodes within the cluster. To recap, I have 2 pve servers without a shared pool on the back end, meaning that each server is essentially separate - but replication should happen between the primary and secondary, if replication works. I have set up the 2nd pve/pbr server with a different zfs-pool name and got the following error:

I would think that the primary server is looking for a 'named pool' called 'localzfs' in order to replicate the data between the servers. Is this correct, and if I create the zfs pool, on the secondary with the same name - should the replication work? Am I missing something?

Also - for others thinking about pbs backups - make the 'volume' you want to replicate the size you need before you start. I'd created a pool to test, which worked for a couple of servers - then expanded my job to include more servers. Although on the 'back end' I had expanded the size - the original size still thinks the backup location is to small, and now I have to remove that dataset - recreate it with a larger size, and then point my backups back to it. AARRGGG !!!!

 

[Chris]

2nd pve/pbr server with a different zfs-pool name

should the replication work? Am I missing something?

As the error states, there is not pool with the same name for storage replication available. I am not sure I understand what you did set up, so please share the following (in code tags):

• output of cat /etc/pve/storage.cfg
• output of the VM config you are trying to replicate qm config <VMID> --current
• zpool list for both, source and target node

Also - for others thinking about pbs backups - make the 'volume' you want to replicate the size you need before you start.

Not sure I understand what you are trying to do here either. Are you trying to replicate the Proxmox Backup Server VM? If so, what filesystem is used within the PBS VM? You will have to expand not only the virtual disk of a VM, but also the filesystem (and volume groups if you are using LVMs) inside of the VM.

 

[markf1301]

I've attached the pve/zpool list.
As you can see the storage on the pve is listed different on the primary pve than the secondary server (listed as pve/pbr). The zpool names are different and that might be the cause. I think this is the cause as the primary box is looking for a similar naming zpool name to replicate to - but I'd have thought that the 'mappings' between different servers - once the storage is known, wouldn't make a difference.

 

[Chris]

Well, as the output of zfs list for the source and target host shows, you do not have the same storage layout! Both nodes require a zpool localzfs to be available, on each node respectively.

 

[markf1301]

Well, as the output of zfs list for the source and target host shows, you do not have the same storage layout! Both nodes require a zpool localzfs to be available, on each node respectively.

I'd suspected as much after reviewing the zfs-list pools.
That said, and since this server is part of the cluster - what is the best method of fixing my issue? Should I remove it from the cluster (how or what is the recommended method of doing that) or can I just delete the 'local store' on the pve/pbr server called pbsdata (nothing there yet I need now) and recreate the zfs store with the same name as the primary server? The primary server zfs pool is called localzfs.

 

[Chris]

I'd suspected as much after reviewing the zfs-list pools.
That said, and since this server is part of the cluster - what is the best method of fixing my issue? Should I remove it from the cluster (how or what is the recommended method of doing that) or can I just delete the 'local store' on the pve/pbr server called pbsdata (nothing there yet I need now) and recreate the zfs store with the same name as the primary server? The primary server zfs pool is called localzfs.

You do not need to remove the node from the cluster. If you are sure that you do not have any data of value on the zfs pool (it seems there are 993G of data allocated), what you can do is remove the storage called pvepbs-datastore on your node (also make sure that this is not used by PBS as datastore). Then destroy the zpool, wipe the disks related to that pool and recreate the zpool with the same name as the one on the other node.

 

[markf1301]

Thanks. I remember whenever I created the cluster, I got an error about the 'localzfs' zfs pool - which is on the main/original pve server and it was something about 'shared storage'. The cluster process was able to complete but it shows up with a ? in the cluster interface. See below for the screen shot.
Yes - you are correct in that there is 'data' on the pvepbs-datastore because that is related to the 2nd server in the cluster (pvepbr) which contains my pbs server and some temporary backup files. I'll remove the pbr server vm (have a backup of it) and the datastore to 'clean up' the datastores on the 2nd server in the cluster. Once I do that and format the zfspool on the 2nd server (pvepbr) and recreate the zfspool with the same name as in the original server (localzfs) should I expect the error message in the console to go away and auto-correct itself? I inadvertently started another backup of the proxmoxbackupserver image and forgot to remove the drive from it so i have to wait till it finished or 'force-restart' that server to break the backup before I can 'clean things' up.

 

[Chris]

You get the question mark because pvestatd is not able to access this storage on this node (which is because it is not there). So you will have to either create that storage also for that node (which is what you want since you want to have storage replication via this storage) or limit that configured storage to the subset of nodes where it is actually available, see the nodes property in the storage configuration https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_common_storage_properties

 

[markf1301]

I think I'm in a 'loop' and not sure how to get out of it. I removed the corpproxmox and pvepbs-datastore storage from the server. The cluster looks like this now -

I still have 1 server, after removing the hard-drives from the server so there's essentially 'nothing' there and it looks like this now -

I was able to remove the server - as long as I didn't 'check' to remove it from the configuration and now that server/cluster looks like this -


Whenever I try to 'wipe' any of the disks - I get an error -


Is there a better method of doing this - perhaps from the cli in which i can remove the zfs pool - from the system - then 'locate' them back in the console to 'wipe disk' and rebuild the zfs pool as to what it's supposed to be? I think the zpool destroy "name" will destroy the pool am uncertain about whether the system will identify the drives again (in the gui) for me to rebuild it again. Is there an option to take the pool offline (I boot from another disk in the system) and while it's offline - rename it, and then bring the pool back in, maybe with a import command?

 

[markf1301]

Ok - must be 'pick on me day' ...
Got rid of the zfs pool that was named wrong and destroyed that zfs pool.
Chris - you mentioned that I'd have to create the pool on that server - so I can do the storage replication.
If 'limiting' the pool to a specific might work, I'm unsure how to do this from reading the reference you sent. I'll try to review it more.
I did try to create that zfs pool but got a (500) error message because that pool already is in the cluster. ?? Limiting it - since the storage replication is looking for that naming schema would hopefully resolve the issue.

 

[Chris]

I did try to create that zfs pool but got a (500) error message because that pool already is in the cluster. ?? Limiting it - since the storage replication is looking for that naming schema would hopefully resolve the issue.

You are almost there, the issue here is that the storage configuration for the storage called localzfs already exists on both nodes, since they are joined in a cluster. As you can see in the Create ZFS window, there is an Add Storage checkmark, which controls whether to create or not create the storage config. So if you uncheck that, the zpool should be created just fine and be online afterwards.

s there an option to take the pool offline (I boot from another disk in the system) and while it's offline - rename it, and then bring the pool back in, maybe with a import command?

What would have been possible is to export the zpool via zpool export <poolname> and reimport it using a different name via zpool import <old-poolname> <new-poolname>.

 

[markf1301]

Never mind - figured it out. Got to learn 'where' to click.
That said - it seems like on the 'new' pvepbr server I don't have access to any of the previous 'shared' resources - like the storage items. I'll play around some more as maybe I'm missing something..somewhere. Since changing or limiting the storage from 1 server - to itself, and not the cluster I presume that I have "2" different storage locations - localzfs on pve and localzfs on pvepbr?

 

[Chris]

Never mind - figured it out. Got to learn 'where' to click.
That said - it seems like on the 'new' pvepbr server I don't have access to any of the previous 'shared' resources - like the storage items. I'll play around some more as maybe I'm missing something..somewhere. Since changing or limiting the storage from 1 server - to itself, and not the cluster I presume that I have "2" different storage locations - localzfs on pve and localzfs on pvepbr?

This is not a shared storage, you have 2 local storages called localzfs, one for each node. Please see the table of available storage types and their properties [0]. Contents are node bound and you will have to transfer contents if they should be available also on the other node, e.g. the VM disks via the storage replication feature which is what you were trying to achieve.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_storage_types

Edit: Fixed link

 

[UdoB]

[0] oxymoron.blochwave.org

??? ;-)

nslookup oxymoron.blochwave.org 9.9.9.9
Server:         9.9.9.9
Address:        9.9.9.9#53

** server can't find oxymoron.blochwave.org: NXDOMAIN

 

[markf1301]

Thanks Chris -

That was what I was suspecting as I truly have 2 different storage pools. Wish I could get mgmt to splurge for another server to use as my 'real' shared backend storage. Perhaps if I can do that in the future - I can go this process again and get a real back shared storage. Fun..Fun...

I'll deal with that for now. That said - I got the storage to replicate from 1 server to another w/o issues. I also learned that on the 2nd server that I also need to have the networking configured the same as I did a 'migrate' and the vm wouldn't start (this was due to vlans on the primary server not configured on/in the pve environment of the 2nd server). Checking and adjusting the network on the 2nd server - got the other server to start and works. Will have to make some adjustments there. The process of storage migration worked, but what do I do that info now?

Can I import them on the new server and 'mount' them if needed or have I just duplicated my images and data from 1 server to another? What I was hoping, thru this process, is that I could use the storage motion process to 'duplicate' the data from 1 server to another and it be available - and visible - on the 2nd server so that I could quickly mount/power them up. Is this possible - or is this part of a HA set up that I haven't done?

For my DR site - If after I have migrated the servers (A->B) - is there a way to 'import' the servers into the new environment thru the gui - or will i have to use some type of scp and qmrestore or import process to get this done? I am currently setting up a remote DR server in which I'd like to have the following:

1. VM image from production replicated daily to the DR site.
2. Daily snapshots replicated or replicated servers visible in the DR site so I can quickly (previous days info if that's all they require) start the servers if needed.

I think I will need to 'seed' the data from corp to DR and don't think the storage migration will work in this scenario.
What would you recommend for this process? PBR, SCP with some type of restore/import process, ? I want the replication process to be automated (replication - that daily data be updated with current info, email confirmation of processes, etc) and am not looking to utilize a lot of 'external or bash' scripts ?? I learned, the hard way, thru this process that my zfs storage needs to be the same on the 2 servers in production. I presume that the DR side would need to have the same 'storage name' on it to 'match' otherwise the configuration for the vm's won't boot w/o manipulate (network changes). I did purchase the PBR server for both environments and am setting it up now so any suggestions would be appreciated.

 

[Chris]

Can I import them on the new server and 'mount' them if needed or have I just duplicated my images and data from 1 server to another? What I was hoping, thru this process, is that I could use the storage motion process to 'duplicate' the data from 1 server to another and it be available - and visible - on the 2nd server so that I could quickly mount/power them up. Is this possible - or is this part of a HA set up that I haven't done?

The procedure to recover the guests replicated to a different node is rather straight forward, please let me refer you to the section in the docs describing the error recovery steps [0]. You can also use HA in order to achieve this as mentioned at the start of the section [1].

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_error_handling
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvesr