Individual nodes lost (do not see) "own, local" thin disk after joining cluster

[AndroGen]

I'm new to Proxmox and this question might be basic, still need help, as could not find the answer by myself.

The situation:
3 Proxmox systems have been installed on individual (rather small boxes).
2 systems (pve01, pve02) are based on N5100 (2LAN ports) and another one is a bit beefier (4LAN ports) (pve03), also low power Celeron server system.
2 first N510 systems have only one, single 2TB SSD - and this is planned being used for the Proxmox and local VM / container storage (with some replication or back-up to other systems). 3rd system has more disks and will have more complicated disks configuration.

Installation was fine, on the first system pve01 one VM was created and stored on the "local-lvm" - all was fine all these systems have been combined into the cluster.

What I've understood. The cluster has the same /etc/pve/storage.cfg

Indeed, the file looks identical on all three systems, and all three point to the third system: pve03

cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve03
        content rootdir,images

and of course, I have figured this out only when pve03 has been stopped.
The intention is that this system pve03 will not be up and running all the time, just periodically, to replicate/back-up the data.

What I could not figure out: how my configuration should be properly reflected in /etc/pve/storage.cfg that each system has access to local-lvm

Following resources have been already checked (but could not grasp the idea out of it yet)
https://pve.proxmox.com/wiki/Storage
https://pve.proxmox.com/wiki/Storage:_LVM_Thin
https://forum.proxmox.com/threads/cluster-with-both-zfs-and-lvm.50399/

Any idea, guidance would be appreciated.
Any reference to a good video would be also a help.

As a background info, I'm have rather long (maybe not always too deep, but still) experience with IT, Linux, VMware, and many other product, but new to Proxmox.
More systems to be added to the cluster when the confidence / experience reaches expected level.

 

[fabian]

the storage.cfg file is shared for the whole cluster - so any definition contained there that is not restricted to a subset of nodes must be valid on ALL nodes. for your local-lvm this means
- either having the same VG name on all nodes, and a single local-lvm entry
- splitting local-lvm and have one entry for each VG, and restrictring those entries to the nodes where they actually exist
-- e.g., local-lvm with VG 'pve' and restricted to nodes pve01 and pve02
-- lvm-pve03 with VG 'pve03' and restricted to node pve03

 

[AndroGen]

Thanks @fabian, I guess in my case (please help me where I am going into a wrong direction).
Since the set-up is rather "unstable" and cannot rely on "always on" capability I would need to have each server pointing out to the own local-lvm.
In this set-up there is no "sharing" possible.
That would be my understanding.

Now the question (what I could not get from the documentation) - how such scenarios need to be reflected in the /etc/pve/storage.cfg
and if I change this file on one system - will it be replicated to all or I would need to edit each individually and manually?

would this be correct to have:

cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm01
        thinpool data
        vgname pve01
        content rootdir,images
        nodes pve01

lvmthin: local-lvm02
        thinpool data
        vgname pve02
        content rootdir,images
        nodes pve02

lvmthin: local-lvm03
        thinpool data
        vgname pve03
        content rootdir,images
        nodes pve03

What else should be changed in the system, that each system stores the VM / container image "locally"?
I guess, I might need to adjust already created VM that it points out to the right place / storage, right? if yes, where?

 

[fabian]

no, that is the opposite of what you want. if you can, you want a single entry pointing to a single VG name and thin pool that exists on each node. the contents will be different on each node, that is okay and expected and handled by PVE. the whole cluster has a single storage.cfg file that is automatically synced to all nodes (like all files in /etc/pve).

so what you want:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

you'd only need another entry if for example pve03 has an extra disk that the other nodes do not have, and that disk also contains a PV/VG/thin pool, in which case the following would be the normal way to configure it:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

lvmthin: pve03-extra
        thinpool data
        vgname extra
        nodes pve03
        content rootdir,images

note how the storage ID and vgname are different, and the pve03-extra storage is being limited to just the pve03 node.

 

[AndroGen]

Thanks, this has resolved the issue.

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

It worked well.
I guess, I need to explore the local-lvm a bit more.
It looks like local-lvm has been created with the system install, and allocated the rest of SSD already (I used the "default" settings during the install).

What this vgname stands for? Is it points to something, e.g. folder on the file system??

 

[fabian]

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_lvm
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_lvmthin