How do I set up a flexible storage with LVM thin pools for backups?

[pancakes]

I'm trying to set up storage configuration as follows

1. Built on two physical disks, 1TB & 4TB
2. PVE host OS itself having a guaranteed amount of space ('root', 32 GB)
3. PVE host having access to storage location ('bulk' - max 5TB) for
   1. large files (pictures etc) for use across VMs - propose to do this via SMB sharing on PVE host
   2. VM/container backups/snapshots - only accessible by PVE host
4. Containers/VMs with flexible storage capacity ('guest1' max 5TB)

Below is a visual representation of my idea (the arrow indicating the flexibility part of the storage):



I tried getting this to work with LVM LVs but I'm confused between the LVM nomenclature / configuration and the Proxmox storage representation in the GUI.

1. What kind of storage is 'local'? Where does it reside in the hierarchy of LVM?
2. What are the 'root' and 'swap' lvs? I presume this is where Proxmox is installed, correct? If so, can I make this a thin LV?
3. (How) can I store (VZDump) backup files on a thin LVM? Via the GUI I can only store these on 'local' which is a directory storage.
4. How do I give PVE access to a LVM thin LV?

Hope somebody can help

Current config

root@pve:/etc/network# pvs
  PV             VG  Fmt  Attr PSize    PFree 
  /dev/nvme0n1p3 pve lvm2 a--  <931.01g 818.81g
  /dev/sda1      pve lvm2 a--    <3.64t  <3.64t

root@pve:~# vgs
  VG  #PV #LV #SN Attr   VSize  VFree 
  pve   2   3   0 wz--n- <4.55t <4.44t

root@pve:~# lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data pve twi-a-tz-- 100.00g             0.00   10.43                           
  root pve -wi-ao----   8.00g                                                    
  swap pve -wi-ao----   4.00g

root@pve:~# lsblk 
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   3.6T  0 disk 
└─sda1               8:1    0   3.6T  0 part 
nvme0n1            259:0    0 931.5G  0 disk 
├─nvme0n1p1        259:1    0  1007K  0 part 
├─nvme0n1p2        259:2    0   512M  0 part /boot/efi
└─nvme0n1p3        259:3    0   931G  0 part 
  ├─pve-swap       253:0    0     4G  0 lvm  [SWAP]
  ├─pve-root       253:1    0     8G  0 lvm  /
  ├─pve-data_tmeta 253:2    0   100M  0 lvm  
  │ └─pve-data     253:4    0   100G  0 lvm  
  └─pve-data_tdata 253:3    0   100G  0 lvm  
    └─pve-data     253:4    0   100G  0 lvm

 

[Dunuin]

I'm trying to set up storage configuration as follows

1. Built on two physical disks, 1TB & 4TB
2. PVE host OS itself having a guaranteed amount of space ('root', 32 GB)
3. PVE host having access to storage location ('bulk' - max 5TB) for
   1. large files (pictures etc) for use across VMs - propose to do this via SMB sharing on PVE host
   2. VM/container backups/snapshots - only accessible by PVE host
4. Containers/VMs with flexible storage capacity ('guest1' max 5TB)

Below is a visual representation of my idea (the arrow indicating the flexibility part of the storage):

You really shouldn't store your backups on the same disks you store your guests on. Lose one of those two disks and you will lose your VMs/LXCs as well as their backups. That only could make sense when also syncing all the backups to another storage afterwards.

What kind of storage is 'local'? Where does it reside in the hierarchy of LVM?

"local" is a normal LV called "root" on VG "pve". It stores your root filesystem where PVE s installed and is the only place where you can store files/folders.

What are the 'root' and 'swap' lvs? I presume this is where Proxmox is installed, correct? If so, can I make this a thin LV?

The "swap" LV is used as a swap "partition". Not sure if you can make it a thin volume.

(How) can I store (VZDump) backup files on a thin LVM? Via the GUI I can only store these on 'local' which is a directory storage.

The LVM-Thin storage doesn't support storing files, as LVM-Thin is a block type storage. You would need to manually create a new thin volume using the CLI, manually format that thin volume, manually mount that thin filesystem and then create a new directory storage pointing to the mountpoint of that thin volumes filesystem.

How do I give PVE access to a LVM thin LV?

You mean for example for storing files on it for the SMB server? Same as above, just skip the part with creating a directory storage. So manually creating, formating and mounting a thin volume.
PVE also got no NAS functionlities. So you would need to manually install and configure your SMB server.

 

[pancakes]

Thanks for your reply @Dunuin !

You really shouldn't store your backups on the same disks you store your guests on. Lose one of those two disks and you will lose your VMs/LXCs as well as their backups. That only could make sense when also syncing all the backups to another storage afterwards.

Yes fair point, I was planning on moving them afterwards, but first wanted to solve the volume architecture

The LVM-Thin storage doesn't support storing files, as LVM-Thin is a block type storage. You would need to manually create a new thin volume using the CLI, manually format that thin volume, manually mount that thin filesystem and then create a new directory storage pointing to the mountpoint of that thin volumes filesystem.

Thanks, I think I missed two things:

1. There's an additional filesystem layer on top of the LVM
2. LVM thin consists of two layers, a thin pool and a thin volume

With this in mind, I think my target config looks more like this, which I'll try later today.

 

[pancakes]

Ok it seems to work . Some learnings:

1. Proxmox creates a layer around storage by itself, which is tracked in /etc/pve/storage.cfg, and using lv* CLI tools does not automatically populate this configuration.
2. For VMs/Containers Proxmox creates the final layer itself, e.g. adding a LVM-thin pool via pvesm or Datacenter -> Storage creates a reference to a pre-existing (pre-created) LMV-thin pool. When creating a new VM or Container, it creates a thin volume itself.

New question: I'm now serving the bulk data share through Samba on the Proxmox host. Can I also share this via Samba on the Guest OS while keeping it out of backups/snapshots? Ideally I'd have as little running on the Proxmox server to maximise stability/reduce attack surface.

I now have this approach, any feedback or thoughts are welcome

Create thin pool & 2 thin volumes on remaining free data

lvcreate --thin -l 100%FREE pve/thinpool1
lvcreate --thinpool pve/thinpool1 --name lv_bulk --virtualsize 2.5T
lvcreate --thinpool pve/thinpool1 --name lv_backup --virtualsize 2.25T

Create fs on thin volumes

mkfs.ext4 /dev/mapper/pve-lv_bulk
mkfs.ext4 /dev/mapper/pve-lv_backup

Mount in Proxmox:

mkdir /mnt/{bulk,backup}
mount /dev/mapper/pve-lv_bulk /mnt/bulk/
mount /dev/mapper/pve-lv_backup /mnt/backup/
mkdir -p /mnt/backup/{pve,mba,mbp,remote}

Add /mnt/backup/pve directory to PVE storage manager to be used for VM backup

pvesm add dir backup --path /mnt/backup/pve --content vztmpl,iso,backup

Add LVM thin pool to PVE storage manager for VM storage

pvesm add lvmthin thinpool1 --vgname pve --thinpool thinpool1

From here I can create VMs in PVE on thinpool1, and store VM/container backups on /mnt/backup/pve. I will separately setup SMB to share /mnt/bulk to VMs/Containers

The architecture looks as follows (color coding indicates which configuration is affected - NB excludes the 'backup' thin volume but it's analogous to the 'bulk' volume):


Configuration looks as follows (vm-200-disk-0 was already created automatically by Proxmox when creating a VM on storage thinpool1):

root@pve:~# cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content iso,vztmpl,backup

dir: backup
    path /mnt/backup/pve
    content backup,vztmpl,iso

lvmthin: thinpool1
    thinpool thinpool1
    vgname pve

root@pve:~# lvs -a
  LV                VG  Attr       LSize  Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_backup         pve Vwi-aotz--  2.25t thinpool1        1.81                                  
  lv_bulk           pve Vwi-aotz--  2.50t thinpool1        1.80                                  
  [lvol0_pmspare]   pve ewi------- 76.00m                                                        
  root              pve -wi-ao----  8.00g                                                        
  swap              pve -wi-ao----  4.00g                                                        
  thinpool1         pve twi-aotz-- <4.54t                  1.91   11.05                          
  [thinpool1_tdata] pve Twi-ao---- <4.54t                                                        
  [thinpool1_tmeta] pve ewi-ao---- 76.00m                                                        
  vm-200-disk-0     pve Vwi-a-tz--  8.00g thinpool1        11.62