What about LVM+qcow2

[carragom]

Hi,
This is an idea that makes a lot of sense to me, yet I'm not able to find people talking about it so it's probably a bad idea. Still I would love to know why it's bad. Before anyone asks, no I'm not planing on using this in production.

According to the docs PVE supports shared LVM storage when connected with iSCSI or FC which is great. But there is no snapshot support on it, and this makes sense since clustered LVM does not support snapshots, so far so good but... What if we can use an LV as a file in qcow2 format and let the format handle the snapshots?. AFIAK XenServer/XCP-ng does something similar with the vhd format.

So I decided to do some testing in the lab with a single PVE node and a local (not shared) LVM. And to my surprise I was able to do the following

• Boot the VM with a manually modified lv+qcow2
• Partition the disk into 2 partitions
• mkfs.ext4 each partition
• Write data to each with multiple files and directories
• Take a snapshot
• Add new files and dirs
• Change existing file content
• Revert to the snapshot previously taken

How did I do this?

1. Create a VM with a disk in the LVM storage without starting it
2. Enable the LV if needed with lvchange -ay /dev/vg1/vm-100-disk-0
3. Format the LV with qemu-img create -f qcow2 /dev/vd1/vm-100-disk-0 32G
4. Use qm show 100 to see the command that proxmox would use to start the VM
5. Change the value from format=raw to format=qcow2 in the -drive parameter
6. Manually start the VM with the modified command

At this point the web ui will show the VM booting, hopefully with a live cd where you can start changing things in the disk. Every time the VM is shutdown, the LV gets deactivated so use step 2 in the previous list to activate it so you can start the vm again or play with snapshots.
To manage the snapshots I used

1. Create snapshot qemu-img snapshot -c snap1 /dev/vd1/vm-100-disk-0
2. List the snapshots with qemu-img snapshot -l /dev/vd1/vm-100-disk-0
3. Revert snapshot qemu-img snapshot -a snap1 /dev/vd1/vm-100-disk-0
4. Delete a snapshot qemu-img snapshot -d snap1 /dev/vd1/vm-100-disk-0

The desired end here would be to have multi-path enabled iscsi with shared LVM and snapshots like XenServer/XCP. Posibly better because these would be thin snapshots. But again, all this sounds too good to be true so could anyone tell me why this is a bad idea?

 

[iwik]

I think it is great idea! It should work and be perfect solution for SAN environment (ideal vmware replacement). Actually I have found this article which is about implementing kubernetes CSI driver on SAN using qcow2 on LVM. They mention this document about how this is implemented in ovirt/vdsm which have snapshots on SAN. So this solution is already used in other virtualization solution. Vdsm is doing thin provision, but to keep it easy, it is not needed, just to have snapshots on LVM. Thin provison and dedup can be done on SAN storage side. Can somebody from Proxmox look at this?

 

[LnxBil]

Thin provison [...] can be done on SAN storage side.

Despite what all vendors advertise ... without proper guest support ist just a plain and simple no, they're lying. Once the data is written and deleted but not unmapped from the guest disk, you will end up with an almost thick-provisioned volume. Just had this case with a 6/7-figure 3PAR.

 

[iwik]

Despite what all vendors advertise ... without proper guest support ist just a plain and simple no, they're lying. Once the data is written and deleted but not unmapped from the guest disk, you will end up with an almost thick-provisioned volume. Just haEd this case with a 6/7-figure 3PAR.

I have experience with larger vmware env backed by multiple all flash storages. It works well, also dedup.

 

[carragom]

I think it is great idea! It should work and be perfect solution for SAN environment (ideal vmware replacement). Actually I have found this article which is about implementing kubernetes CSI driver on SAN using qcow2 on LVM. They mention this document about how this is implemented in ovirt/vdsm which have snapshots on SAN. So this solution is already used in other virtualization solution. Vdsm is doing thin provision, but to keep it easy, it is not needed, just to have snapshots on LVM. Thin provison and dedup can be done on SAN storage side. Can somebody from Proxmox look at this?

Glad to see I'm not the only one that thinks there is value behind this. Thanks for the links, good to know others are trying to work on this. To be honest I had almost forgotten about this, It's been 4 months without a reply after all. My current setup for multi-path iSCSI uses OCFS2 and I would love to be rid of that unnecessary layer.

 

[iwik]

Out of topic, but how is ocfs2 with qcow2 performing? And it is reliable? I can't find any benchmarks or tutorials and my past experience (10+ years) with ocfs2 is only bad (kernel panics).

 

[carragom]

Pretty solid for the past couple of years on a v7 six-node cluster with around 200 VMs. I did encounter a kernel panic once, it was fixed pretty quickly with an update. Also there is a problem with v8 where the VM won't start if you use Async IO = io_uring on the hard drive. Maybe it's fixed by now, have not tested it lately. Native and threads options work fine. As for performance, it feels pretty good and since NFS is out of the question for me because it lacks multi-path support, I guess the only other option to compare it to would be GFS2, I might try that someday. Management is a bit clunky though, that's why I would love to have shared LVM with snapshots, even if the snapshots are thick.