Cloning from a KVM VM snapshot fails: "Failed to find logical volume..."

[johnnyutahh]

I'm seeking suggestions for debugging the following.

After running a "clone from snapshot" in the web gui rev 6.2-11, I receiving the following output:

TASK ERROR: clone failed: activate_volume 'pve/snap_vm-199-cloudinit_baseline_1' error:   Failed to find logical volume "pve/snap_vm-199-cloudinit_baseline_1"

In case this is helpful:

$ qm listsnapshot 199
`-> U20_04_1_fresh_qemuguestagent 2020-09-22 15:43:05     no-description
    `-> baseline_1      2020-09-22 16:00:28     no-description
        `-> baseline_5_ipaclient 2020-09-22 16:29:17     no-description
            `-> gitlab_1 2020-09-22 19:31:12     no-description
                `-> gitlab_2 2020-09-23 15:07:50     no-description
                    `-> gitlab_3 2020-09-23 15:17:46     no-description
                        `-> gitlab_test1_4 2020-09-23 16:15:22     no-description
                            `-> current                         You are here!
$
$ pvesm list local-lvm
Volid                      Format  Type             Size VMID
local-lvm:base-499-disk-0  raw     images     2361393152 499
local-lvm:vm-100-disk-0    raw     images    68719476736 100
local-lvm:vm-100-disk-1    raw     images        4194304 100
local-lvm:vm-199-cloudinit raw     images        4194304 199
local-lvm:vm-199-disk-1    raw     images    53901000704 199
local-lvm:vm-200-disk-0    raw     images    21474836480 200
local-lvm:vm-299-disk-0    raw     images    21474836480 299
local-lvm:vm-399-disk-0    raw     images     8589934592 399
local-lvm:vm-499-cloudinit raw     images        4194304 499
$

 

[johnnyutahh]

[DISCLAIMER: rant follows.]

So..... how do I figure this out?

I do not want to sound disrespectful. Or non-appreciative. For $0, I love what I _hope_ I can get from ProxmoxVE. Very promising.

But I'm an experienced (several decades) techie in unix, windows, linux, macOS, VMS server and storage systems. Note though I'm OLD school; I learned and did most of my when zfs was still "super experimental." So granted, my experience is dated.

But still, I know how to learn, as I've learned (and designed and commercialized) MANY software systems over the years. And (for example) it was NOT clear (from pvesm docs) to me and took way too long to discover that one gets volume IDs (I guess?) by getting the STORAGE_ID from `/etc/pve/storage.cfg` and then running `pvesm list STORAGE_ID`. And then I get the volume IDs... and I'm not sure what to do with them to solve the problem above.

I'd like to say the above is a one-time problem, but it's not. Most times I'm digging deep into PVE to find how things work I have to sift through too many docs and decode too many things to understand why/how/what.

I have to ask, at risk of seeming a bit too flippant: is Proxmox's biz model partly supported by offering few if any good introductory docs/videos so that their new users/admins are forced to buy their subscriptions?

I'd like to employ a consultant to provide me a super-techie training session for a couple hours. So far I can not get any to respond on the job boards (like upwork.com). Maybe they're so busy keeping existing clients afloat?

Apologies: I'm being too snarky here. But golly, it sure feels like most of my learnings for PVE does not have to be this hard.
[/rant]

The following was one of my tinkering trying to find stuff. No dice. Thus far.

$ file /dev/pve/vm-199-disk-1.raw
/dev/pve/vm-199-disk-1.raw: cannot open `/dev/pve/vm-199-disk-1.raw' (No such file or directory)
$
$
$ pvesm path local-lvm:vm-199-disk-1.raw
/dev/pve/vm-199-disk-1.raw
$
$ file /dev/pve/vm-199-disk-1
/dev/pve/vm-199-disk-1: symbolic link to ../dm-14
$
$
$ file /dev/dm-14
/dev/dm-14: block special (253/14)
$
$
$
$ pvesm path local-lvm:vm-199-cloudinit
/dev/pve/vm-199-cloudinit
$
$ file /dev/pve/vm-199-cloudinit
/dev/pve/vm-199-cloudinit: symbolic link to ../dm-13
$
$ file /dev/dm-13
/dev/dm-13: block special (253/13)
$
$ cd !$
cd /dev/dm-13
-bash: cd: /dev/dm-13: Not a directory
$
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   32G     0   32G   0% /dev
tmpfs                 6.3G   34M  6.3G   1% /run
/dev/mapper/pve-root   94G   24G   66G  27% /
tmpfs                  32G   43M   32G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  32G     0   32G   0% /sys/fs/cgroup
/dev/nvme0n1p2        511M   15M  497M   3% /boot/efi
/dev/fuse              30M   28K   30M   1% /etc/pve
tmpfs                 6.3G     0  6.3G   0% /run/user/0
$
$ file /dev/pve/vm-199-cloudinit.raw
/dev/pve/vm-199-cloudinit.raw: cannot open `/dev/pve/vm-199-cloudinit.raw' (No such file or directory)
$

 

[johnnyutahh]

Why are my above kvm image formats 'raw' and not 'qcow2'?

I thought qemu/kvm images would default to their more-featureful (_maybe_?) copy-on-write formats?

(See rant above. Yet another thing that does not make sense and feels like it takes longer than it should to properly understand.)

 

[johnnyutahh]

Hunting for `images` dirs does not seem to yield much. :-/

$ pveversion
pve-manager/6.2-11/22fb4983 (running kernel: 5.4.60-1-pve)
$
$ cd /
$ pwd
/
$ find . -iname images
./var/lib/vz/images
./usr/share/pve-manager/touch/resources/themes/images
./usr/share/pve-manager/images
./usr/share/javascript/proxmox-widget-toolkit/images
./usr/share/javascript/extjs/theme-crisp/resources/theme-neptune/images
./usr/share/javascript/extjs/theme-crisp/resources/theme-neutral/images
./usr/share/javascript/extjs/theme-crisp/resources/images
./usr/share/novnc-pve/app/images
./usr/share/pve-docs/images
./usr/lib/python2.7/dist-packages/pycriu/images
$
$ cd /var/lib/vz
$ tree -F
.
├── dump/
│   ├── vzdump-qemu-100-2020_09_12-23_41_59.log
│   ├── vzdump-qemu-100-2020_09_12-23_41_59.vma.zst
│   ├── vzdump-qemu-199-2020_09_21-20_17_23.log
│   └── vzdump-qemu-199-2020_09_21-20_17_23.vma.zst
├── images/
└── template/
    ├── cache/
    │   └── ubuntu-20.04-standard_20.04-1_amd64.tar.gz
    ├── iso/
    │   ├── clover-r4920.iso
    │   ├── clover-r5070.iso
    │   ├── Mojave-installer.iso
    │   └── ubuntu-20.04.1-live-server-amd64.iso
    └── qemu/

6 directories, 9 files
$

 

[mira]

Thank you for bringing this bug to our attention. The cloudinit disk should be ignored completely when cloning as it is recreated on VM start. But currently it seems no snapshot is taken for the cloudinit disk, but it is then expected to be there.

 

[johnnyutahh]

Okay, thanks for your timely reply. I'm not sure exactly what the following quote means to me (I'm not yet versed enough in PVE) or how to solve the above problem. I'm not clear on how it's a bug. A few more details/pointers might help. eg: what is the expected, proper behavior?

If I buy a suitable support package subscription can and will someone (at Proxmox corporate support) help me solve this problem? Is this a good example representing the value of what the PVE support subscription can provide?

Thank you for bringing this bug to our attention. The cloudinit disk should be ignored completely when cloning as it is recreated on VM start. But currently it seems no snapshot is taken for the cloudinit disk, but it is then expected to be there.

 

[johnnyutahh]

If I buy a suitable support package subscription can and will someone (at Proxmox corporate support) help me solve this problem? Is this a good example representing the value of what the PVE support subscription can provide?

To be clear: my team is happy to subscribe to paid support. (ie: we will not "hold it against" Proxmox if they do not offer the full, corporate-tech-support assistance here, in this community-only forum, without payment. It makes business-model sense.)

We're just looking for an answer, either way.

If this is something that paid-tech support _can not_ provide us further assistance, this will be a factor in our decision to buy a tech-support subscription(s). (We are considering one or more Premium subscriptions.)

 

[mira]

Please take a look at the subscription agreement to see what is covered by a subscription: https://www.proxmox.com/en/downloads/item/proxmox-ve-subscription-agreement

Regarding the issue at hand, the expected behaviour is that cloning of a VM from a snapshot works, even with cloud-init. As this currently does not work, a patch was sent to the pve-devel mailing list that fixes the issue: https://lists.proxmox.com/pipermail/pve-devel/2020-September/045261.html

If you're using the default 'local-lvm' it should be LVM-thin, which is a block level storage (see https://pve.proxmox.com/pve-docs/chapter-pvesm.html). On block level storages only the 'raw' format is supported as 'qcow2' files require a filesystem which is only available on file level storages. LVM-thin supports both snapshots and thin-provisioning, same as qcow2 files. If you want a copy-on-write file system, use ZFS instead of LVM.

As you're looking for a consultant, have you seen the list of our partners? https://www.proxmox.com/en/partners
Also, if you're looking for an in-depth training, please take a look at our offerings: https://www.proxmox.com/en/training

 

[johnnyutahh]

Great, thanks @mira.

1. updating my (non-production) server with patches

a. is there a build-from-source (if needed?) full procedure somewhere I can read? (I'm assuming this is needed to apply patches.) The referenced patch shows changes to files in /PVE (I think?), a directory which is not on my system.

b. alternatively, are there pre-release downloads/modules I can acquire to effectively install executable software with the patch? (Somewhere in here, perhaps?)

2. what are my available workarounds in lieu of the above?

a. Do I simply make a KVM-vm without cloud-init?

Regarding the issue at hand, the expected behaviour is that cloning of a VM from a snapshot works, even with cloud-init. As this currently does not work, a patch was sent to the pve-devel mailing list that fixes the issue: https://lists.proxmox.com/pipermail/pve-devel/2020-September/045261.html

If you're using the default 'local-lvm' it should be LVM-thin, which is a block level storage (see https://pve.proxmox.com/pve-docs/chapter-pvesm.html). On block level storages only the 'raw' format is supported as 'qcow2' files require a filesystem which is only available on file level storages. LVM-thin supports both snapshots and thin-provisioning, same as qcow2 files. If you want a copy-on-write file system, use ZFS instead of LVM.

 

[mira]

A workaround would be to remove the cloudinit disk, then clone the VM and afterwards add it again on both the original and the cloned VM.

You can apply the patch in-place in /usr/share/perl5/PVE but the files in there will be replaced once the corresponding packages are upgraded or reinstalled.

If you want a guide on how to check out and build the packages, take a look at the following: https://pve.proxmox.com/wiki/Developer_Documentation . Once you've checked out the repository and applied the patch, you can run 'make deb' to get a package to install.

Every package goes through multiple stages. First a package is released to our internal staging repository where it is initially tested. Once it is tested and no errors are found it then goes on to pvetest which is a public repository. There it will be further tested and then released to pve-no-subscription. Once it has received lots of testing and is deemed stable, it is then moved to pve-enterprise.

So in this case the patch has not been applied yet to the repository. The only way for you to test it is to apply it locally, either directly on the files or in your local repository of qemu-server which you then build and install.

 

[johnnyutahh]

For the point at hand:

1. workaround for my current problem

A workaround would be to remove the cloudinit disk, then clone the VM and afterwards add it again on both the original and the cloned VM.

I might be able to figure out "remove the cloudinit disk," but "add it again on both the original and the cloned VM" is not clear to me. What is "it" -- the VM, and if so which one (the original or cloned... or whatever)? or the cloudinit disk, and if so, how do I re-add a "disk" (whatever that means in this in this context) if it was "removed" (whatever that means in this context)?

Maybe a "removed disk" is still available for future use (despite the word "removed")?

2. Patch applications and developer docs

We much appreciate these references, super helpful @mira.

More generally, and PVE training, documentation, and knowledge transfer:

[ @mira - I'm not picking on you specifically here; the following comments are addressed at Proxmox corporate in general. ]

I'm not trying to be mean, I'm really not. Rather, I'm trying to point out that the PVE community often has embedded, presumed knowledge that's not clear when sentences like that above are presented in forum posts, wiki pages, documents, man pages, etc.

And it makes it rather hard for newbies like me (even newbies with lots of experience in related, computing-software domains) to understand.

I ask Proxmox: where is there a "top down" set of docs (better yet: videos?) that tends to describe fundamental concepts and terminology, preferably with visuals, of the PVE system? I'm more than happy to read those in depth.

Most of what I see is "bottom up" docs and verbiage (like above) that does not provide context and presumes you already understand the concepts. And things makes things hard for me, and possibly other newbies.

I'm hoping that when I buy a support subscription(s) that some of these above, knowledge resources are provided. If not, maybe a sufficiently-capable support person can spend a small amount of focused time with me to help get me over the "concepts hump." (It usually does not take long; rather, it's typically more dependent on the quality of the knowledge source.)