Best way to migrate large volume from Ceph to ZFS

[valeech]

I have an older Proxmox and Ceph cluster running 5.3-11 that I need to decomm. I have a new host running 7.3-3 that I want to move a VM to that is running ZFS.

The current VM has 2 virtio drives. One on local storage and a large 20TB volume on Ceph.

I would like to live migrate this VM to the new host but am not sure that is the best method. Backups for this environment are not great. The data is backed up but only at a file level. It would be a large task to rebuild the VM.

I have shared out the new ZFS pool over NFS and mounted that on the old Proxmox cluster. I have tested a live migration of a smaller VM and it worked.

My question is - does anyone see any issue with live migrating this large disk from Ceph to the the ZFS pool over NFS? My plan is to _not_ delete the source when I migrate so if there are issues I can simply reconfigure the VM on the old cluster to use the Ceph volume.

Thanks!

 

[aaron]

My question is - does anyone see any issue with live migrating this large disk from Ceph to the the ZFS pool over NFS?

VM disk images are not stored as files on ZFS, but in so-called volume (ZVOL) datasets that provide a block device.

So overall your plan would be to expose the ZFS storage of the new host to the old one, that it can do disk move while the VM is running. Then power down the VM on the old host and configure it on the new host, using that disk image? As this should keep the downtime rather short, compared to exporting/importing that 20TB disk image?

If both were running Proxmox VE 7.3 you could try to do it via the new (and experimental) cross cluster live migration. Otherwise, I don't have a good idea on how to make the ZFS storage of the new host available to the old.

If you do have plenty of space, you could create a NFS share on the new host. Configure it as a storage on the old, and as a "Directory" Storage on the new one. Then, once the switch has been made, you could move the disk to the final ZFS storage as it should be. But with a 20TB disk image, I am not sure that you have enough space :-/

 

[danielb]

I guess the other way around is an option : add the Ceph storage of the old cluster as a remote Ceph on the new host, stop the VM on the old cluster and start it on the new host from the Ceph storage, then live move the disk to the ZFS storage

 

[aaron]

I guess the other way around is an option : add the Ceph storage of the old cluster as a remote Ceph on the new host, stop the VM on the old cluster and start it on the new host from the Ceph storage, then live move the disk to the ZFS storage

Good point Though i would test it with a smaller test VM first because the Ceph versions might have drifted too far apart.

 

[danielb]

If Ceph version is an issue, then you might be able to
- install targetcli on the new host
- configure the ZFS storage of the new host as a ZFS over iSCSI storage on the old cluster (ZFS over iSCSI uses zvol just like a local ZFS)
- live move the disk from Ceph to the ZFS over iSCSI storage on the old cluster
- stop the VM on the old cluster
- start it on the new host
- remove targetcli on the new host

 

[alexskysilk]

this question is posted and reposted constantly (in various combinations.)

the fastest (and least safe) is to dd | tar | ssh
the safest (and smartest generally) is to BACK UP the original vm, and restore it on the new hv. You say to dont backup/dont have a backup device? get one and start.

if you need to ask why, see taobackup.com

 

[spirit]

you can use the new "qm remote-migrate" beta command.

It's allow to migrate cross-cluster, live, with storage migration too.

"
qm remote-migrate <vmidsrc> <vmiddst> 'apitoken=PVEAPIToken=root@pve!APItoken=12345...,host=targethost.fdn' --target-bridge vmbr0:vmbrx --target-storage oldstorage:newstorage --online"


you need to create an api token on destination cluster first.

 

[valeech]

VM disk images are not stored as files on ZFS, but in so-called volume (ZVOL) datasets that provide a block device.

So overall your plan would be to expose the ZFS storage of the new host to the old one, that it can do disk move while the VM is running. Then power down the VM on the old host and configure it on the new host, using that disk image? As this should keep the downtime rather short, compared to exporting/importing that 20TB disk image?

If both were running Proxmox VE 7.3 you could try to do it via the new (and experimental) cross cluster live migration. Otherwise, I don't have a good idea on how to make the ZFS storage of the new host available to the old.

If you do have plenty of space, you could create a NFS share on the new host. Configure it as a storage on the old, and as a "Directory" Storage on the new one. Then, once the switch has been made, you could move the disk to the final ZFS storage as it should be. But with a 20TB disk image, I am not sure that you have enough space :-/

Thanks for the feedback!!

Sharing out the new server ZFS pool via NFS to the older cluster is currently working. I tested a live disk move on a smaller VM and everything went fine. I was then able to move it, as you suggest, from a local directory storage on the new server to ZFS pool on the new server to clean things up. I do have enough space to do this with the large disk too so I think this is my way forward.

It is my experience with Proxmox that if a live disk move is interrupted for any reason, operations continue on the source host and the move is "rolled back". Just verifying that should be the experience? Any concerns with doing this type of disk move From Ceph to NFS with such a large volume?

 

[alexskysilk]

you can use the new "qm remote-migrate" beta command.

hmm. Do you know if this option is available (backportable) to the old hypervisor? Op stated it was pve 5.3

 

[jdancer]

If you have a spare machine with capacity another option is Promox Backup Server.

Old -> PBS -> New