fastest way to copy zfs volume from server to another

[emanuelebruno]

Good morning everyone.

I have two Proxmox servers (the first one in version 7.4-15 and the second in version 7.1-7) and I need to move two KVMs that are on a ZFS disk. I was looking on the web to check the fastest solution and I came across some posts talking about snapshots, hot cloning, full backup, synchronization of volumes at pre-set minutes, etc.

It's all very nice and wonderful but I believe the simplest thing for me would be to shut down the KVMs and start the synchronization of the ZFS volumes from one server to another.

I have already successfully migrated a KVM from a "thin" volume from one server to another through an SSH channel.

Thank you in advance for your responses."

 

[sterzy]

You could make use of Remote Migration, too, though that is still experimental [1] and may not be available on your older PVE host (tho, you should probably update that anyway).

Alternatively, to move the disks, you can use zfs send/receive like so:

• Shut down the VM on the sending node and create a snapshot of each disk: zfs snapshot rpool/vm-X-disk-Y@now.
• Create the volume on the receiving side.
• Send the volume via: zfs send -Rpv rpool/vm-X-disk-Y@now | ssh -o BatchMode=yes root@$IP zfs recv -Fv rpool/vm-X-disk-Y.

If you want, you can destroy the snapshot after like so: zfs destroy rpool/vm-X-disk-Y@now.

[1]: https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/remote_migrate

 

[emanuelebruno]

You could make use of Remote Migration, too, though that is still experimental [1] and may not be available on your older PVE host (tho, you should probably update that anyway).

Alternatively, to move the disks, you can use zfs send/receive like so:

• Shut down the VM on the sending node and create a snapshot of each disk: zfs snapshot rpool/vm-X-disk-Y@now.
• Create the volume on the receiving side.
• Send the volume via: zfs send -Rpv rpool/vm-X-disk-Y@now | ssh -o BatchMode=yes root@$IP zfs recv -Fv rpool/vm-X-disk-Y.

If you want, you can destroy the snapshot after like so: zfs destroy rpool/vm-X-disk-Y@now.

[1]: https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/remote_migrate

Hello Sterzy,

I've reviewed your discussion on the Proxmox forum about transferring ZFS volumes from one server to another and would like to share the approach I'm planning to use for minimal downtime VM migration. Here are the steps:

1. Initial Snapshot Creation:
   • While the VM is still running, I'll create an initial snapshot of the current state of the ZFS volume. This allows capturing the current state without having to shut down the virtual machine.
   • Command: qm snapshot my_kvm_number initial_snapshot
2. Initial Snapshot Transfer:
   • I will transfer this first snapshot to the new server using the zfs send and zfs receive commands. This step might take some time depending on the volume size.
   • Command: zfs send -Rpv rpool/vm-X-disk-Y@initial_snapshot | ssh -o BatchMode=yes root@new_server_ip zfs recv -Fv rpool/vm-X-disk-Y
3. VM Shutdown and Second Snapshot Creation:
   • After completing the initial transfer, I'll shut down the VM to ensure data consistency and then create a second snapshot. This will capture only the changes made since the first snapshot, allowing for a quicker transfer.
   • Commands:
     • Stop VM: qm stop my_kvm_number
     • Create snapshot: qm snapshot my_kvm_number final_snapshot
4. Incremental Transfer:
   • Now, I'll transfer the second incremental snapshot to the new server.
   • Command: zfs send -Rpv -I rpool/vm-X-disk-Y@initial_snapshot rpool/vm-X-disk-Y@final_snapshot | ssh -o BatchMode=yes root@new_server_ip zfs recv -Fv rpool/vm-X-disk-Y
5. Starting the VM on the New Server:
   • Once the incremental transfer is complete, I'll start the VM on the new server and perform a series of tests to ensure everything is functioning correctly.

I hope this procedure is clear and helpful. Looking forward to your feedback or any questions!

 

[emanuelebruno]

[...]

Hello Sterzy,

I've investigated the possibility of limiting the data transfer rate between the two servers to avoid potential overloads, especially concerning disk and network usage during the ZFS send/receive operation. I found a solution that can effectively control the transfer speed.

The command used for this purpose is as follows:

zfs send [options] | pv -L 50m | ssh [destination] zfs recv [options]

In this command, pv -L 50m is the key part. It uses the pv (Pipe Viewer) tool to limit the data transfer rate to 50 megabytes per second. This can be particularly useful to prevent high disk I/O and network bandwidth consumption from impacting other operations on the servers.

Remember, the [options] placeholders in the zfs send and zfs recv commands need to be replaced with the actual options appropriate for your specific use case.

I hope this information is helpful to you or anyone else facing similar challenges.

Best regards,
Emanuele Bruno.