Backup to ZFS array

[Analius]

Hi,

I have recently gotten into the whole cluster thing, I've got it all worked out, except for backing up to zfs.

Previously, when using standalone nodes, I could simply create a storage type "directory" and point the location to a dir on a zfs array.
However, now in a cluster, if I allow multiple nodes to that directory, the directory gets created on all nodes. This is an issue as I dont have the exact same zfs arrays on all nodes.

How do I go around this problem? I've tried enabling/disabling the "Shared" flag, but the only way to allow a node to backup to another node's zfs array, is to allow it access with the "nodes" flag, but this then creates the same directory on the original node (the one where that array does not exist).

Thanks

 

[IT Hero]

Hey Analius,

TLDR: Directly ZFS on a shared LUN won't work because ZFS is not cluster-aware.

Explanation:
you should have a look at: https://pve.proxmox.com/wiki/Storage

ZFS is a filesystem so it operates on the file level.

For what you want to achieve you would need a shared LUN (all nodes see the same local drive)

The problem in this case is that each individual node doesn't know that others are accessing this LUN, so it will get corrupted.

A shared LUN can be created using SCSI (Fiber,SAS) or iSCSI (over LAN), then you can set up LVM on it. With LVM, Proxmox will ensure that no corruption occurs.

If you want ZFS, you can have another machine with a zpool serving zvols over iSCSI which will be setup by Proxmox (one zvol per VM/container).

You can have a look at my thread: https://forum.proxmox.com/threads/s...an-fiber-for-virtual-disks.89007/#post-390302

 

[Analius]

Hey Analius,

TLDR: Directly ZFS on a shared LUN won't work because ZFS is not cluster-aware.

Explanation:
you should have a look at: https://pve.proxmox.com/wiki/Storage

ZFS is a filesystem so it operates on the file level.

For what you want to achieve you would need a shared LUN (all nodes see the same local drive)

The problem in this case is that each individual node doesn't know that others are accessing this LUN, so it will get corrupted.

A shared LUN can be created using SCSI (Fiber,SAS) or iSCSI (over LAN), then you can set up LVM on it. With LVM, Proxmox will ensure that no corruption occurs.

If you want ZFS, you can have another machine with a zpool serving zvols over iSCSI which will be setup by Proxmox (one zvol per VM/container).

You can have a look at my thread: https://forum.proxmox.com/threads/s...an-fiber-for-virtual-disks.89007/#post-390302

That is a bummer but it does make sense, thank you for your reply.

That leaves me with the following problem though:

Let's say I have setup a backup job, set to backup a bunch of containers/vms. This backup job, writes to a storage that is available to the node I set the job up from. What happens if I now migrate one of those vms/containers to another node?
How would you go about this?

Maybe it would help if I mention what it is that I am trying to achieve, perhaps you can give me some insights: Like I mentioned, I have 2 nodes, what I want is for all containers/vms that are running on both nodes, to be backed up to both nodes, so a backup exists in 2 places.

 

[IT Hero]

You have two possibilities:

1) The easy solution is to have a NAS on your network with a network share that you will mount on all nodes at once (in Datacenter/storage/add CIFS or NFS). Then you can use this as target for your backups. You'll have to consider your network speek, the sizes of your VMs and the frequency of your backups.

2) Set up replication between your two nodes: https://pve.proxmox.com/wiki/Storage_Replication and this uses indeed ZFS.
Sadly I cannot tell you anything about it as I never tried it. But it seems to be exactly what you want to do.

 

[Analius]

Thank you, I will have to rethink this a bit.