[SOLVED] ZFS replication for VMs with multiple disks

M

macleod

Well-Known Member
Aug 3, 2017
66
8
48
47
Hello

I was just testing Proxmox 6.2 and I was wondering why only the second disk of the vm was transferred fully, and first one was just incremental (and the zvol was there). Well, actually it seems that I've just set a replication (from GUI) and forgot about it :p. But the question remains:

Why only the first disk of the VM was replicated ? It this the intended behavior ? Or for my scenario (multiple zvols - actually on the same storage) I must dig and use the CLI of pve-zsync ?

Also please say YES or NO to the following algorithm (let's see if I've understand the basics):
1. live migrating VMs with local disks on zfs (no replication):
- creating zvols on destination (on lvm got thicker from thin, but zfs is smart enough to "ignore" holes)
- "starting" vm on destination (kvm/qemu "mirroring")
- snapshot on source
- transfer zfs data from source to destination (from snapshot above)
- memory transfer
- delta disks and delta memory transfer (small amounts just before the switchover) - better say "dirty" than "delta"
- switchover the vm on destination
- deleting the zvols on source

2. live migrating VMs with local disks on zfs (with replication to destination):
- zvols are already on destination (with some old snapshots) - actually like I've said, only the first vm disk is there
- "starting" vm on destination (kvm/qemu "mirroring")
- snapshot on source (x)
- transfer zfs data (delta) from last replicated snapshot to the snapshot above (x)
- memory transfer
- delta disks and delta memory transfer (small amounts just before the switchover) - better say "dirty" than "delta"
- switchover the vm on destination
- NOT deleting the zvols on source (probably because replication exists) - actually the secondary disk get deleted

I belived that doing a live migration to a non-replica destination it's like (1), but probably without deleting the zvols on source.

Thanks and sorry for such a long text :p
 
Last edited:
Just a test machine:

agent: 1
boot: c
bootdisk: scsi0
cores: 1
cpu: cputype=host
memory: 1024
name: test
net0: virtio=xxxxxxxxxxxxxxxx,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-101-disk-0,discard=on,format=raw,size=20G
scsi1: local-zfs:vm-101-disk-1,backup=0,discard=on,format=raw,replicate=0,size=2G
scsihw: virtio-scsi-pci
smbios1: uuid=xxx
sockets: 2
tablet: 0
vmgenid: xxxx

But thanks for the hint, now i see there is a replicate=0 parameter (I must see from where it came from and why). Most probably this is the cause, so I will test later and give some feedback.

Couple of years ago there was a problem on migration with multiple vms (with lvm storage - don't remember exactly), but then was a bug. Now probably is a PEBKAC :-P
 
definitely the problem was between keyboard and chair :-P
after unchecking "skip replication" for the second disk everything went fine, just ping-pong-ed the vm back and forth and only the memory was full transferred (as it should)
so ... many thanks for the hint!
 
  • Like
Reactions: fireon
Hello
I have a strange problem with a VM which had temporarilly 2 disks, but really only only one during the 6 latest month at least.
The remaining disk is disk-1 and has the correct replication settings.
My problem is that a ghost of the disk-0 seems to be kept somewhere.

During the replication the logs confirm that disk-1 is replicated (and only this disk which is good)
But when I do a migration, the process quickly synchronizes disk-1, then do a full copy of a useless disk-0 which is nearly as big as the real one (and takes a long time).

Disk-0 does not appear in the web console ... do I need to find the file and remove it in CLI ?

My proxmox cluster is using the latest pve-manager/7.2-4/ca9d43cc with Kernel Linux 5.15.35-2-pve
but the problem was already the same for several months

If someone has a clue ...
Apart from this the ZFS replication has been very helpfull for this simple 2 node configuration
 
Pierre DEDIEU said:
During the replication the logs confirm that disk-1 is replicated (and only this disk which is good)
But when I do a migration, the process quickly synchronizes disk-1, then do a full copy of a useless disk-0 which is nearly as big as the real one (and takes a long time).
Click to expand...
most probably it should appear as 'unused disk', and can be erased (if really you are 100% sure that is not needed anymore
also it is possible that the volume exists even if doesn't appear in config, so very careful you could delete that volume using zfs commands in console
as deleting is a destruction-type operation, beware and use it only when you are really sure you want to do that
but ... it's quite strange that full replication only for that disk-0, which suggest something fishy about that vm/zfs/...etc
i think the vm config (stripped by some sensitive info) and some listings of zfs volumes, also replication.cfg from /etc/pve (regarding that vm) should bring some light to this problem
 
Good news, I overcame these difficulties today:
I post what occured to me, in case it can help someone.
- First disk-0 was not showing at all in the VM / Hardware
- I detected thru the pve shell that zfs list was showing the unused old disk, so I decided to use zfs destroy disk
- The situation was improved, but the was still a copy of this unused disk on the replication destination, so I proceded with another zfs destroy on the second pve.
- Then the next migration failed ... and I used a known recovery: Deleted the vm replication, recreate it and then it successed
- The migration was quicker, but the log showed that it also transfered 2 snapshots, which from my knowledge were legitimate but supposed to have been deleted (they didn't show in the WM / Replication screen). So I used zfs destroy on both pve and TADAAA...
- the situation is stabilized: Last migration time was 1 minute instead of 16 minutes, replication is quick

Thank you for your confirmation of the path to the light !

ZFS snapshots and replication really permits great possibilities for those without real share storage.
I'll use Ceph for my next cluster, but ZFS prooved very good for my test platform
 
Doing a qm rescan --vmid VMID should have found the disk in your ZFS storage and add it to the VM as "unused", allowing you to remove it both from the storage and from the VM config using the WebUI. It should also remove the replication settings for that disk and remove it from "the other" host.
 
  • Like
Reactions: i_am_jam, Neobin and macleod
You must log in or register to reply here.
Top Bottom