Storage limit

I

inder441

New Member
Dec 17, 2024
10
0
1
Hello Team,

I am running my production datacenter on 7 nodes proxmox cluster, with HBA SAN shared LUNs. on top of which I created LVM.

I create PV and then VG on each LUN then add then created LVM (shared) in proxmox cluster.

So one LUN------------> PV------------->VG--------------->shared LVM

Not at all combining PVs in VGs and VGs in LVM.

I am not combining PVs or VGs together at all to make things simple. As of now I am creating 20 TB LVMs. It all works beautifully.

Yesterday, I expanded two LUNs to 30TB and 40TB. However, they still show 20TB in pvs and vgs output and in LVM as well.

I know very well how to expand the LUNs and I have done that in the past successfully. However, I have never expanded them above 20 TB. This is the first time.

So I would like to understand what is the recommended size of LVM LUNs in proxmox env. Also does proxmox support move then 20TB LUN in the scenario what I have configured.

Expedite response will be highly appreciated.

Regards.
 
Hi @inder441 , there is no particular recommended size that PVE enforces.

This is entirely up to you and your storage vendor. We have customers using luns ranging from 64K to 64TB (ESXi recommended limit).

Perhaps if you can share the command line output and specific steps you used to expand all layers, someone may be able to provide a recommendation.

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thank
bbgeek17 said:
Hi @inder441 , there is no particular recommended size that PVE enforces.

This is entirely up to you and your storage vendor. We have customers using luns ranging from 64K to 64TB (ESXi recommended limit).

Perhaps if you can share the command line output and specific steps you used to expand all layers, someone may be able to provide a recommendation.

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Thank you for quick response. Here is the output of PVdisplay and VGdisplay:
----------------------------------------------------------------------------------------
root@mtlvipve007:~# pvdisplay
--- Physical volume ---
PV Name /dev/mapper/360000970000197604069533030343134
VG Name vg343134
PV Size 20.00 TiB / not usable 4.62 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 5242879
Free PE 5242879
Allocated PE 0
PV UUID et0oiB-IPlL-a0zl-Zpse-cl5j-fNzn-YhKddB

--- Physical volume ---
PV Name /dev/mapper/360000970000197604069533030343142
VG Name vg343142
PV Size 20.00 TiB / not usable 4.62 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 5242879
Free PE 719929
Allocated PE 4522950
PV UUID eSkwua-bKnb-pqtn-MVq6-y4zM-ISF6-zIwE4w

--- Physical volume ---
PV Name /dev/mapper/360000970000197604069533030343138
VG Name vg343138
PV Size 20.00 TiB / not usable 4.62 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 5242879
Free PE 778237
Allocated PE 4464642
PV UUID c44DKm-kTgZ-fUZi-hYgc-8waF-RKSC-yAKvEA
-------------------------------------------------
root@mtlvipve007:~# vgdisplay
--- Volume group ---
VG Name vg343134
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <20.00 TiB
PE Size 4.00 MiB
Total PE 5242879
Alloc PE / Size 0 / 0
Free PE / Size 5242879 / <20.00 TiB
VG UUID 3pf35H-mw6l-9fgO-X9CK-28Qu-ulpW-TpxxNl

--- Volume group ---
VG Name vg343142
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 26
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 11
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size <20.00 TiB
PE Size 4.00 MiB
Total PE 5242879
Alloc PE / Size 4522950 / 17.25 TiB
Free PE / Size 719929 / <2.75 TiB
VG UUID nEkb1t-ZWWY-DOiI-7Opd-xulc-W66l-jG5uXg

--- Volume group ---
VG Name vg343138
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 96
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 66
Open LV 22
Max PV 0
Cur PV 1
Act PV 1
VG Size <20.00 TiB
PE Size 4.00 MiB
Total PE 5242879
Alloc PE / Size 4464642 / 17.03 TiB
Free PE / Size 778237 / <2.97 TiB
VG UUID eLdT9K-JiFK-j37a-gYxB-0jO3-1tyF-yqJb3x

-----------------------------------------------------------------------------------------------

For now I have removed the LUNs which I extended. As this is my production. Also all my PVs and VGs are configured similar way with similar config.

If I extend them beyond 20TB multipath and other places it shows extended size but not GUI, PVs and VGs.
 
It seems things are fluid at the moment?

Please use CODE tags when pasting info for ease of reading. Also, please incorporate output of "lsblk" as that is the basis for PV and VG layers.
But since you have removed the LUNs, I guess there is not much to troubleshoot at this point.

Do make sure that size update propagates through each layer in order : lsblk>multipath>pv>vg

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
It seems things are fluid at the moment?

Please use CODE tags when pasting info for ease of reading. Also, please incorporate output of "lsblk" as that is the basis for PV and VG layers.
But since you have removed the LUNs, I guess there is not much to troubleshoot at this point.

Do make sure that size update propagates through each layer in order : lsblk>multipath>pv>vg

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Thank you for the guidance. Will keep this in mind.

What you suggested is what was performed:

lsblk. It reflected right size.
multipath -r
pvresize
vgresize

it all worked well. Still it showed 20TB for both the expanded disks.
 
inder441 said:
it all worked well. Still it showed 20TB for both the expanded disks.
Click to expand...
When you get into this situation next time, please collect and provide all data and try keep the system in that state while troubleshooting.
I don't think much can be done now if anything at all.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: inder441
You must log in or register to reply here.
Top Bottom