[SOLVED] Error: no such logical volume

[scubaninja]

Hi,

I changed something months ago, broke everything, tried to fix it, and I think I got part of the way back, but I'm still stuck trying to get one VM up. Sadly, it is the only VM that I actually want, the others are all disposable. I'm new to proxmox, so this is 100% user error.

There are 3 nodes (Marvin, elmer, foghorn) and the VM that I want is 'puffin' on node foghorn.
When trying to start puffin, the description shows "VM 100 - Start" with status of "Error: no such logical volume pve/vm-100-disk-0"
It then lists description of "HA 100 - start" with status of "Error: command 'ha-manager set vm:100 --state started' failed: exit code 255"

Somewhere, I tried teaching myself high availability and I think I made things more complicated than I should have. At this point, I don't even care if I can't get the VM operational. I'm willing to start over with a new build, I just want to get the website data out of Apache.

I have tried to do my homework before asking for help and I see that there are often requests for the following:
lvs

  LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotz-- <348.82g             4.37   0.62                          
  root          pve -wi-ao----   96.00g                                                  
  swap          pve -wi-ao----    8.00g                                                  
  vm-102-disk-0 pve Vwi-a-tz--  100.00g data        15.23

vgs

  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1   4   0 wz--n- <475.94g 16.00g

lsblk

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                            8:0    0 476.9G  0 disk
├─sda1                         8:1    0  1007K  0 part
├─sda2                         8:2    0     1G  0 part /boot/efi
└─sda3                         8:3    0 475.9G  0 part
  ├─pve-swap                 252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta           252:2    0   3.6G  0 lvm
  │ └─pve-data-tpool         252:4    0 348.8G  0 lvm
  │   ├─pve-data             252:5    0 348.8G  1 lvm
  │   └─pve-vm--102--disk--0 252:6    0   100G  0 lvm
  └─pve-data_tdata           252:3    0 348.8G  0 lvm
    └─pve-data-tpool         252:4    0 348.8G  0 lvm
      ├─pve-data             252:5    0 348.8G  1 lvm
      └─pve-vm--102--disk--0 252:6    0   100G  0 lvm

/etc/pve/storage.conf

dir: local
        path /var/lib/vz
        content backup,iso,vztmpl
        shared 1

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

zfspool: SSD-ZFS
        disable
        pool SSD-ZFS
        content rootdir,images
        mountpoint /SSD-ZFS
        sparse 0

lvm: foghorn-SSD
        vgname pve
        content rootdir,images
        nodes elmer,foghorn,Marvin
        saferemove 0
        shared 1

lvm: Marvin-SSD
        vgname pve
        content rootdir,images
        nodes Marvin
        saferemove 0
        shared 1

lvm: elmer-ssd
        vgname pve
        content rootdir,images
        nodes foghorn,elmer
        saferemove 0
        shared 1

lvscan

  ACTIVE            '/dev/pve/data' [<348.82 GiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  ACTIVE            '/dev/pve/vm-102-disk-0' [100.00 GiB] inherit

Please let this be an easy one for the proxmox wizards. I would be forever grateful for assistance.
Thanks in advance!

 

[fba]

The storage setup seems quirky to me with locally backed LVM accessible for multiple nodes.
Nevertheless, what I would do first:
Login to each of the nodes and check with lvs command which is holding the vm disk. If found, move conf-file of the vm manually to this node (/etc/pve/nodes/<nodename>/qemuserver)

 

[fiona]

Hi,
please note what the shared flag for storages means (from man pvesm):

--shared <boolean>
Indicate that this is a single storage with the same contents on all nodes (or all listed in the nodes option). It will not make the contents of a local storage automatically accessible to other nodes, it just marks an already shared storage as such!

 

[scubaninja]

The storage setup seems quirky to me with locally backed LVM accessible for multiple nodes.
Nevertheless, what I would do first:
Login to each of the nodes and check with lvs command which is holding the vm disk. If found, move conf-file of the vm manually to this node (/etc/pve/nodes/<nodename>/qemuserver)

Hi fba,

I followed your instructions and it worked! I now have the ability to start-up the desired vm and access the system. Now I just need to find the safest way to remove all the complication and bring it back to a single server.

Thank you so much for your expertise!
As promised, I am eternally grateful!

 

[fba]

I am glad you found the vm disk and it is running again.
Btw. why did you create these <node>-SSD entries in the storage configuration at all? Assuming each node provides the default LVM-Thinpool anyway, you're not gaining anything from that. If no vm/lxc has it's disk stored there, removing the entries would be a good start for a cleanup.
If you really want to configure HA you will need either a shared storage or use ZFS and replication. With the current configuration it wont work.

 

[scubaninja]

If I remember correctly, I added separate SSDs to two of the nodes. So those machines had NVME and an SSD. My thought was to aggregate the SSDs into one large pool and have it be shared storage for all nodes. But as I mentioned, I'm new to proxmox and the distributed nature of VMs was all new to me. I'm familiar with running standalone LAMP servers and implementing failover across multiple machines like DNS uses, but I bit off more than I could chew with this one.

I found a set of commands to separate a node from a cluster and make it standalone which is what I did to each machine. I may try to migrate the websites to a clean install and just start fresh with the cluster project again from scratch. Mini PCs are [were] so cheap that putting together a cluster was affordable to just experiment and learn on. But then I went and used it for more than experimentation and screwed myself. Any suggestions for solid tutorials on clusters and HA?