wrong thinlvm getting mounted / boot issues

[spacebass]

Howdy! I have a head-scratcher.
I installed 7.2-4 on new hardware and added it to my cluster. The install created three LVMs:

LV     VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data pve twi-aotz-- <794.79g             0.00   0.24                          
  root   pve -wi-ao----   96.00g                                                  
  swap   pve -wi-ao----    8.00g

However, data wasn't formatted or mounted. So I:
1. mkfs.ext4 /dev/pve/data
2. added /dev/pve/data to fstab:

/dev/pve/data  /mnt/pve/data ext4 errors=remount-ro 0 2

3. added /dev/pve/data as a directory only for this one node

What worked in giving me a place to put disk images, ISOs, etc.

But, it doesn't survive a reboot. I get an issue about dependancies failing. I suspect that's because /dev/pve isn't created yet?

Then I tried:
1. removing /dev/pve/data from fstab
2. adding the thinlvm in the GUI and assigning to only this one specific node

That results in Proxmox just mounting an empty 96g thinlvm calling it data. It is really strange. Here's what that looks like in /etc/pve/storage.conf

lvmthin: data
        thinpool data
        vgname pve
        nodes teatons
        content images,snippets,rootdir,iso,vztmpl,backup
        nodes teatons
        prune-backups keep-all=1
        shared 0

Next I tried renaming the thinlvm to something unique, thinking it was a namespace conflict with other nodes' data LVMs.
I tried both adding the LVM via the GUI (same result as above, small, empty). Then I tried mounting it via fstab in /mnt/data_unique rather than /mnt/pve/data_unique

I have to add it as a directory to the Proxmox GUI or I can't use it as storage. Again, this works in user space but doesn't survive a reboot.

It seems I either get a very small, empty data thinlvm that survives reboots, or I can have my true data lvm which doesn't survive reboots.

Anyone have any tips or suggestions?

root@teatons:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[bbgeek17]

You've conflated a lot of things here, mostly incorrectly:

1) The default PVE installation does create "data" volume. It does not format or mount it, because it is used for BLOCK storage, ie lvm volumes
2) Your configuration file specifically defines pool called "data" and uses "lvmthin" storage plugin - it expects to slice and dice the "data" PVS into raw LVs.
3) Your configuration is incorrectly marking "lvmthin" pool as capable of storing "snippets,iso,vztmpl,backup" - these are all files that need _file_ storage.
4) You should not be formatting or mounting "data" unless you plan to forgo "lvmthin" in favor "directory" storage.

The results you are getting are expectedly unpredictable as the system was not designed to function like this.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[spacebass]

I understand I've broken quite a bit. The root cause is that data, as a thinlvm, despite being created, was not available to this new install.
So, now that I have data on it I have two choices:
1. move those to another store temporarily and re-create data as it should be
2. keep it as a directory

I'm perfectly happy to do the later. However, I can't get the system to boot when it is mounted as and defined in /etc/storage.conf as a directory.

Any tips for accomplishing either path?

 

[bbgeek17]

I understand I've broken quite a bit. The root cause is that data, as a thinlvm, despite being created, was not available to this new install.

I find it hard to believe, hundreds of people install PVE daily using default option (as you appear to have done) and have no issues with availability of thinlvm/data pool. You are likely misremembering the initial state, before you started to make changes.

So, now that I have data on it I have two choices:
1. move those to another store temporarily and re-create data as it should be

should be pretty simple, use standard Linux tools to unmount any filesystem that somehow made its way to VG data, remove any partitions from it, LVs so its in raw empty state. Edit your storage configuration and remove file based capabilities. Restart your host.

2. keep it as a directory

Start with removing lvmthin pool - as its broken. Make sure there is an LV created on VG, format and prepare it. Edit fstab as desired, reboot. You have not provided any detailed information about your prior mount failures. You should start from scratch anyway.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[spacebass]

I find it hard to believe, hundreds of people install PVE daily using default option (as you appear to have done) and have no issues with availability of thinlvm/data pool. You are likely misremembering the initial state, before you started to make changes.

thanks for the condescension, super helpful. Maybe you're right and, on this my 20th...?30th? install I somehow got misremembered. And even if I had, mistakes happen. That's what I'm here to sort out. I open it, I screwed it up, I get it. I'm just looking for some help in getting things straightened out.

I'll back up the VMs and reinstall