LVM Snapshot script for shared storage LVM over iSCSI only

[rustine22]

Hi,

i have made a script to permit PVE users to do snapshots by SSH on VM node only for VM located on on shared storage LVM over iSCSI Storage (Tests on TrueNas).

What do you think of it ?

Usage: ./pve_iscsi_lvm_snapshot.sh <action> <vmid> [<snapshotname>]
action : Action à effectuer (list, create, delete, revert)
vmid : ID de la machine virtuelle
snapshotname : Nom du snapshot (obligatoire pour action create,delete,revert)


#list snapshot of VM 100
./pve_iscsi_lvm_snapshot.sh list 100

#create snapshot snap1 on VM100
./pve_iscsi_lvm_snapshot.sh create 100 snap1

#delete snapshot snap1 on VM 100 (forget it)
./pve_iscsi_lvm_snapshot.sh delete 100 snap1

#revert to snapshot snap1 on VM 100
./pve_iscsi_lvm_snapshot.sh revert 100 snap1

Best regards

 

[einhirn]

Looks interesting - I'm not sure it will be found easily here... Maybe I'll link this in some other discussions.

Perhaps you could translate the comments and prompts in the script to English, that might help some people to understand your script better.

Also, please think about hosting the script on github etc. because you need to log in to download attachments and it's not very accessible inside a .zip file. This would also allow people to report issues or suggest changes to the script.

Personally, in the code, I would have liked an overview like "the following functions are defined, this one does that and calls the next one" etc. as a description, but I was able to figure out what it did in general.

There was some commented out stuff there, looks like preparation for Version 2 with LVM-Snapshot autoextension or something? Currently it'll probably need the same amount of space allocated for VM disks as unallocated space in the LVM VG...

Did you try it with multiple hosts? Like moving a VM after you did a Snapshot? Will the script find the snapshot on the new host?

Can the script freeze the VM before taking the snapshot? Maybe leverage something like "qemu snapshot" as with backup until you've created the LVM snap? Maybe not needed, but I don't know if it could help with a VM having multiple disks, so that it achieves a "crash consistent" snapshot state with all disks snapshotted at the same point in time or something.

 

[rustine22]

Looks interesting - I'm not sure it will be found easily here... Maybe I'll link this in some other discussions.

Thanks

Perhaps you could translate the comments and prompts in the script to English, that might help some people to understand your script better.

Yes i will dot that asap

Also, please think about hosting the script on github etc. because you need to log in to download attachments and it's not very accessible inside a .zip file. This would also allow people to report issues or suggest changes to the script.

exact, il will do that

Personally, in the code, I would have liked an overview like "the following functions are defined, this one does that and calls the next one" etc. as a description, but I was able to figure out what it did in general.

There was some commented out stuff there, looks like preparation for Version 2 with LVM-Snapshot autoextension or something? Currently it'll probably need the same amount of space allocated for VM disks as unallocated space in the LVM VG...

The script allocate 50% of original disk as snapshot, but it could be less for example 1% or 5% then autoextend would do extend if write io rate remain not too high. It depend of VM activity and if big modifications on disk will occur.

Did you try it with multiple hosts? Like moving a VM after you did a Snapshot? Will the script find the snapshot on the new host?

Not tested on several hosts, but with shared LVM iscsi, the script will probably execute right on server that hosts the VM.

Can the script freeze the VM before taking the snapshot? Maybe leverage something like "qemu snapshot" as with backup until you've created the LVM snap? Maybe not needed, but I don't know if it could help with a VM having multiple disks, so that it achieves a "crash consistent" snapshot state with all disks snapshotted at the same point in time or something.

No, no freeze implemented on script sorry. Script make lvm snapshot on all disk of the vm.

 

[einhirn]

Great - I just thought that some more background information would be nice, like:

Script will find LVM volume for each Disk of given VM
It will then create a LVM snapshot with half the size of original disk - this is the amount of free space required in LVM.

The LVM snapshot tracks changes to original Volume, so when a snapshot is no longer needed, it can just be discarded (no additional I/O load).

Should a VM be very active writing changes to disk, the snapshot might fill up (What happens then?)

When a VM needs to be rolled back to a snapshot, it needs to be shut down first. To start the rollback, that is for some reason called "merge" in LVM, the VMs disk volumes will be deactivated and activated again. This will instantly allow the VM to boot with the previous state of the disks, but in the background LVM will move blocks around to revert the changes in original volume. (Additional space requirements?)

Edge cases: What about moving a VM to another host while a merge is underway? Is there a way to prevent that? But then you'd need to undo whatever you did to prevent the move after "LVM merge" is completed. Hmm.

 

[rustine22]

Great - I just thought that some more background information would be nice, like:

Script will find LVM volume for each Disk of given VM
It will then create a LVM snapshot with half the size of original disk - this is the amount of free space required in LVM.

yes

The LVM snapshot tracks changes to original Volume, so when a snapshot is no longer needed, it can just be discarded (no additional I/O load).

Should a VM be very active writing changes to disk, the snapshot might fill up (What happens then?)

As i understand of LVM Snapshot, max size of snaphot size is the original disk size. So i think, the LVM snapshot track every change on every byte of the original disk, so if all disk is changed, max size of snapshot will be the original size. But it's a my theory.

When a VM needs to be rolled back to a snapshot, it needs to be shut down first. To start the rollback, that is for some reason called "merge" in LVM, the VMs disk volumes will be deactivated and activated again. This will instantly allow the VM to boot with the previous state of the disks, but in the background LVM will move blocks around to revert the changes in original volume. (Additional space requirements?)

Yes it is. I hav added an ehancement on that script that verify the status of lv just after deleting snapshot, to wait for end of merging task before end the script

Edge cases: What about moving a VM to another host while a merge is underway? Is there a way to prevent that? But then you'd need to undo whatever you did to prevent the move after "LVM merge" is completed. Hmm.

there are some tests to do for this. in my memories, the vm is shutdown and lvm is merging so perhaps the lv status prevent the vm to be moved

 

[rustine22]

last version with lvm configuration directives on top, and wait for merging task before end the script

 

[waltar]

Here is somethink to sync just the snapshot incrementals between hosts but never tested so if any has interests and time to understand what happens there (like zfs send/recv) take a look into and perhaps find useful inside or as whole : https://github.com/mpalmer/lvmsync .

 

[einhirn]

I think the sync of snapshots isn't needed, because it's LVM on a shared iSCSI LUN - the other hosts also should be able to see the snapshots.

 

[waltar]

Yes I know not used in this case directly but if you want a DR storage of that shared storage that lvmsync go further for solution

 

[LnxBil]

Just found this thread ... is this not just the almost the same as the ZFS-over-iSCSI plugin, yet instead LVM-over-iSCSI? Have you tried to copy and adapt the plugin?

 

[rustine22]

No i hav written this script from scratch

 

[LnxBil]

No i hav written this script from scratch

Yes, I know. Therefore I said that it would be best if you would try to merge it to a copy of the ZFS-over-iSCSI plugin.

 

[rustine22]

Sorry, it's an external script, i don't know pve script, i will try asap

 

[einhirn]

One issue I just came across: LVM itself isn't cluster aware and "shared LVM thick" just works because proxmox orchestrates it across the cluster, doing things that will change LVM metadata (like creating a new volume or adding a snapshot) on one host at a time using a locking mechanism. It also automatically refreshes LVM infos on the other hosts, so if this script should work reliably, it should somehow hitch a ride on those mechanisms, I guess.

Post in thread 'Shared LVM on iSCSI: how safe is it?'

Mar 28, 2024

You can also try to setup proper LVM clustering, which is not done automatically with PVE, at least not for years. In PVE 5 and before, this was neccessary, but as long as only PVE is changing the LVM metadata, you won't need LVM clustering.

• LnxBil

• 

"as long as only PVE is changing the LVM metadata ..."​

Also: LVM-Thin just isn't supported because it will change LVM metadata without callback or warning when a volume grows automatically - so there's no way to communicate changes like that to the other cluster nodes...

 

[rustine22]

main LVM Thick volume attributes aren't changed except snapshot presence or not.

LVM metadata are stored with LVM volume in PV so are shared with others hosts ? Or you are speaking about PVE infos about LVM volume ? Main attributes known by PVE are not changed in my opinion.

 

[LnxBil]

LVM metadata are stored with LVM volume in PV so are shared with others hosts ? Or you are speaking about PVE infos about LVM volume ? Main attributes known by PVE are not changed in my opinion.

Yes, they are updated on the disk, but the other nodes will not read them, because they think, they have not been altered. Therefore the cluster locking as introduced and made sure that every metadata change was communicated across the lvm cluster to all nodes. This was before LVM thin was introduced and AFAIK, there is no cluster version of LVM thin, so it will corrupt your data if you try to cluster it.

 

[rustine22]

OK. So to resolve this problem,

1- prevent / not move a VM that hav lvm thick snapshot => very uncomfortable on a cluster environnment..
2- the script should trigger lvm re-read on other nodes of cluster. Do you think it is possible over SSH ?

Thanks

 

[LnxBil]

OK. So to resolve this problem,

1- prevent / not move a VM that hav lvm thick snapshot => very uncomfortable on a cluster environnment..
2- the script should trigger lvm re-read on other nodes of cluster. Do you think it is possible over SSH ?

Unless they changed the implementation of thick snapshots, those need to have the same size as the volume in order to not run out of space. Effectively doubling the space requirement.
In general, if this would be a viable and workable solution, Proxmox would have implemented it. They haven't so this might not be as easy as synchronizing some stuff.

 

[rustine22]

For LVM Thick snapshot, i think the size of snapshot is a setting that you can change : in the actual script it is 50% of disk size with autogrow. You can begin at 1% of disk size with autogrow. but there is a risk if there is massive write when snapshot is active and if autogrow is not enough reactive ..

# Prerequesties on all PVE nodes :
# 
# in etc/lvm/lvm.conf
#
# enable settings :
#
#         snapshot_autoextend_threshold = 70
#         snapshot_autoextend_percent = 20
#