Extending LVM on iSCSI

EC2_Jase · Jun 19, 2024

My DEV Storage ran out of space, I've already extended it on the Storage array. But unsure on how to extend within Proxmox without lossing 6TB of VMs.

lsblk

on sdb, sde, sdq, sdr

pvscan

Any direction would be greatly recieved.

gfngfn256 · Jun 19, 2024

Before you start: make sure you have backups - you've been WARNED!

I assume you have already resized the underlying iSCSI file extents (in the target storage array).

1. Enter pvdisplay to identify the relevant volume mount (lets assume for this example /dev/sdX)
2. Lets resize the volume to fill the disk with pvresize /dev/sdX

You may need to now resize the LV/s (logical volume/s) but this depends on your use case.

Disclaimer: I don't use iSCSI

EC2_Jase · Jun 19, 2024

Thanks for your reply @gfngfn256.

pvdisplay shows me.

So the command I ran was:
pvresize /dev/mapper/3624a937037dc4aba9bff4b94000113ea

lvs just shows me the VM HDD

gfngfn256 · Jun 19, 2024

So what does pvdisplay now show?

EC2_Jase · Jun 19, 2024

pvdisplay shows no change.

gfngfn256 · Jun 19, 2024

gfngfn256 said:
I assume you have already resized the underlying iSCSI file extents (in the target storage array).

?

skgle · Jun 19, 2024

what ford fdiak -l show you? you might need to add a new partition to /dev/sdb to extend

Before doing any of the below make sure you have a backup!

Code:

# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x2c457faa.
Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-50331647, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-50331647, default 50331647):
Created a new partition 1 of type 'Linux' and of size 24 GiB.
Command (m for help): t
Selected partition 1      
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

once you add a new LVM partition you need to use

Code:

pvcreate /dev/sdbX

then add the new PV to the LVM

Code:

vgextend V_GROUP /dev/sdbX

then extend the LVM

Code:

lvextend -l +100%FREE {path to the LVM}

after that you grow the filesystem

bbgeek17 · Jun 19, 2024

What are the outputs (in TEXT with CODE tags without masking) of:
lsscsi
lsblk
pvs
vgs
lvs

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

EC2_Jase · Jun 19, 2024

gfngfn256 said:
?

I can confirm.
On the Storage Array.

On the Proxmox Host

bbgeek17 · Jun 19, 2024

Try:
echo 1 > /sys/block/path_device/device/rescan
i.e.

Code:

# echo 1 > /sys/block/sda/device/rescan
# echo 1 > /sys/block/sdb/device/rescan
# echo 1 > /sys/block/sde/device/rescan
# echo 1 > /sys/block/sdf/device/rescan

multipathd resize map multipath_device
see: https://access.redhat.com/documenta...inux/7/html/dm_multipath/online_device_resize

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

EC2_Jase · Jun 19, 2024

bbgeek17 said:
What are the outputs (in TEXT with CODE tags without masking) of:
lsscsi
lsblk
pvs
vgs
lvs

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

lssci ( not issued )

lsblk

Code:

root@apo:~# lsblk
NAME                                            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sdb                                               8:16   0     8T  0 disk 
└─3624a937037dc4aba9bff4b94000113ea             252:7    0     6T  0 mpath
  ├─Hidden_Dev_Storage-vm--131--disk--0         252:12   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--100--disk--0         252:13   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--101--disk--0         252:14   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--102--disk--0         252:15   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--103--disk--0         252:16   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--142--disk--0         252:17   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--148--disk--0         252:18   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--146--disk--0         252:64   0   250G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--1         252:65   0   100G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--2         252:66   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--3         252:67   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--0         252:68   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--1         252:69   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--126--disk--0         252:70   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--129--disk--0         252:73   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--0         252:75   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--136--disk--0         252:76   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--2         252:83   0     1T  0 lvm   
  ├─Hidden_Dev_Storage-vm--143--disk--0         252:85   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--151--disk--0         252:86   0     8G  0 lvm   
  ├─Hidden_Dev_Storage-vm--149--disk--0         252:88   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--150--disk--0         252:90   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--145--disk--0         252:91   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--125--disk--0         252:92   0   128G  0 lvm   
  └─Hidden_Dev_Storage-vm--147--disk--0         252:93   0   256G  0 lvm   
sde                                               8:64   0     8T  0 disk 
└─3624a937037dc4aba9bff4b94000113ea             252:7    0     6T  0 mpath
  ├─Hidden_Dev_Storage-vm--131--disk--0         252:12   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--100--disk--0         252:13   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--101--disk--0         252:14   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--102--disk--0         252:15   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--103--disk--0         252:16   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--142--disk--0         252:17   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--148--disk--0         252:18   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--146--disk--0         252:64   0   250G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--1         252:65   0   100G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--2         252:66   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--3         252:67   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--0         252:68   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--1         252:69   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--126--disk--0         252:70   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--129--disk--0         252:73   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--0         252:75   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--136--disk--0         252:76   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--2         252:83   0     1T  0 lvm   
  ├─Hidden_Dev_Storage-vm--143--disk--0         252:85   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--151--disk--0         252:86   0     8G  0 lvm   
  ├─Hidden_Dev_Storage-vm--149--disk--0         252:88   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--150--disk--0         252:90   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--145--disk--0         252:91   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--125--disk--0         252:92   0   128G  0 lvm   
  └─Hidden_Dev_Storage-vm--147--disk--0         252:93   0   256G  0 lvm   
sdq                                              65:0    0     8T  0 disk 
└─3624a937037dc4aba9bff4b94000113ea             252:7    0     6T  0 mpath
  ├─Hidden_Dev_Storage-vm--131--disk--0         252:12   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--100--disk--0         252:13   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--101--disk--0         252:14   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--102--disk--0         252:15   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--103--disk--0         252:16   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--142--disk--0         252:17   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--148--disk--0         252:18   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--146--disk--0         252:64   0   250G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--1         252:65   0   100G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--2         252:66   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--3         252:67   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--0         252:68   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--1         252:69   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--126--disk--0         252:70   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--129--disk--0         252:73   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--0         252:75   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--136--disk--0         252:76   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--2         252:83   0     1T  0 lvm   
  ├─Hidden_Dev_Storage-vm--143--disk--0         252:85   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--151--disk--0         252:86   0     8G  0 lvm   
  ├─Hidden_Dev_Storage-vm--149--disk--0         252:88   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--150--disk--0         252:90   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--145--disk--0         252:91   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--125--disk--0         252:92   0   128G  0 lvm   
  └─Hidden_Dev_Storage-vm--147--disk--0         252:93   0   256G  0 lvm   
sdr                                              65:16   0     8T  0 disk 
└─3624a937037dc4aba9bff4b94000113ea             252:7    0     6T  0 mpath
  ├─Hidden_Dev_Storage-vm--131--disk--0         252:12   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--100--disk--0         252:13   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--101--disk--0         252:14   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--102--disk--0         252:15   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--103--disk--0         252:16   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--142--disk--0         252:17   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--148--disk--0         252:18   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--146--disk--0         252:64   0   250G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--1         252:65   0   100G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--2         252:66   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--3         252:67   0    10G  0 lvm   
  ├─Hidden_Dev_Storage-vm--124--disk--0         252:68   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--1         252:69   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--126--disk--0         252:70   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--129--disk--0         252:73   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--0         252:75   0     4M  0 lvm   
  ├─Hidden_Dev_Storage-vm--136--disk--0         252:76   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--132--disk--2         252:83   0     1T  0 lvm   
  ├─Hidden_Dev_Storage-vm--143--disk--0         252:85   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--151--disk--0         252:86   0     8G  0 lvm   
  ├─Hidden_Dev_Storage-vm--149--disk--0         252:88   0   256G  0 lvm   
  ├─Hidden_Dev_Storage-vm--150--disk--0         252:90   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--145--disk--0         252:91   0   128G  0 lvm   
  ├─Hidden_Dev_Storage-vm--125--disk--0         252:92   0   128G  0 lvm   
  └─Hidden_Dev_Storage-vm--147--disk--0         252:93   0   256G  0 lvm   
nvme0n1                                         259:1    0 447.1G  0 disk 
├─nvme0n1p1                                     259:2    0  1007K  0 part 
├─nvme0n1p2                                     259:3    0     1G  0 part  /boot/efi
└─nvme0n1p3                                     259:4    0 446.1G  0 part 
  ├─pve-swap                                    252:0    0     8G  0 lvm   [SWAP]
  ├─pve-root                                    252:1    0    96G  0 lvm   /
  ├─pve-data_tmeta                              252:2    0   3.3G  0 lvm   
  │ └─pve-data-tpool                            252:4    0 319.5G  0 lvm   
  │   └─pve-data                                252:5    0 319.5G  1 lvm   
  └─pve-data_tdata                              252:3    0 319.5G  0 lvm   
    └─pve-data-tpool                            252:4    0 319.5G  0 lvm   
      └─pve-data                                252:5    0 319.5G  1 lvm   
root@apo:~#

pvs

Code:

root@apo:~# pvs
  PV                                            VG                         Fmt  Attr PSize     PFree    
  /dev/mapper/3624a937037dc4aba9bff4b94000113e8 Testbed_VMs                lvm2 a--  <1024.00g <1024.00g
  /dev/mapper/3624a937037dc4aba9bff4b94000113ea Hidden_Dev_Storage         lvm2 a--     <6.00t   161.98g
  /dev/mapper/3624a937037dc4aba9bff4b94000113ef Hidden_Argonaut_Storage    lvm2 a--    <25.00t   <25.00t
  /dev/mapper/3624a937037dc4aba9bff4b94000113f0 Hidden_DataShare_Storage   lvm2 a--    <12.00t  <218.99g
  /dev/mapper/3624a937037dc4aba9bff4b94000113f1 Hidden_Infrastrure_Storage lvm2 a--     <9.00t     5.13t
  /dev/mapper/3624a937037dc4aba9bff4b94000113f2 Hidden_Vane_Datastore      lvm2 a--     <2.00t  <298.00g
  /dev/mapper/3624a937037dc4aba9bff4b9400011b2b Hidden_Insight                lvm2 a--     <8.00t  <798.00g
  /dev/nvme0n1p3                                pve                        lvm2 a--   <446.07g    16.00g
root@apo:~#

vgs

Code:

root@apo:~# vgs
  VG                         #PV #LV #SN Attr   VSize     VFree    
  Hidden_Insight                  1   8   0 wz--n-    <8.00t  <798.00g
  Hidden_Argonaut_Storage      1   0   0 wz--n-   <25.00t   <25.00t
  Hidden_DataShare_Storage     1   9   0 wz--n-   <12.00t  <218.99g
  Hidden_Dev_Storage           1  29   0 wz--n-    <6.00t   161.98g
  Hidden_Infrastrure_Storage   1  36   0 wz--n-    <9.00t     5.13t
  Hidden_Vane_Datastore        1   7   0 wz--n-    <2.00t  <298.00g
  Testbed_VMs                  1   0   0 wz--n- <1024.00g <1024.00g
  pve                          1   3   0 wz--n-  <446.07g    16.00g
root@apo:~#

lvs

Code:

root@apo:~# 
root@apo:~# lvs
  LV            VG                         Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                                       
  vm-100-disk-0 Hidden_Dev_Storage         -wi-ao----  128.00g                                                    
  vm-101-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-102-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-103-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-124-disk-0 Hidden_Dev_Storage         -wi-ao----    4.00m                                                    
  vm-124-disk-1 Hidden_Dev_Storage         -wi-ao----  100.00g                                                    
  vm-124-disk-2 Hidden_Dev_Storage         -wi-ao----   10.00g                                                    
  vm-124-disk-3 Hidden_Dev_Storage         -wi-ao----   10.00g                                                    
  vm-125-disk-0 Hidden_Dev_Storage         -wi-ao----  128.00g                                                    
  vm-126-disk-0 Hidden_Dev_Storage         -wi-ao----  128.00g                                                    
  vm-129-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-131-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-132-disk-0 Hidden_Dev_Storage         -wi-ao----    4.00m                                                    
  vm-132-disk-1 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-132-disk-2 Hidden_Dev_Storage         -wi-ao----    1.00t                                                    
  vm-134-disk-0 Hidden_Dev_Storage         -wi-------    4.00m                                                    
  vm-134-disk-1 Hidden_Dev_Storage         -wi-------  100.00g                                                    
  vm-135-disk-0 Hidden_Dev_Storage         -wi-------  256.00g                                                    
  vm-136-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-142-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-143-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-144-disk-0 Hidden_Dev_Storage         -wi-------  512.00g                                                    
  vm-145-disk-0 Hidden_Dev_Storage         -wi-ao----  128.00g                                                    
  vm-146-disk-0 Hidden_Dev_Storage         -wi-ao----  250.00g                                                    
  vm-147-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-148-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-149-disk-0 Hidden_Dev_Storage         -wi-ao----  256.00g                                                    
  vm-150-disk-0 Hidden_Dev_Storage         -wi-ao----  128.00g                                                    
  vm-151-disk-0 Hidden_Dev_Storage         -wi-ao----    8.00g                                                                                    
  data          pve                        twi-aotz-- <319.55g             0.00   0.52                            
  root          pve                        -wi-ao----   96.00g                                                    
  swap          pve                        -wi-ao----    8.00g                                                    
root@apo:~#

EC2_Jase · Jun 19, 2024

quick update...

multipath -ll shows:

Code:

3624a937037dc4aba9bff4b94000113ea dm-7 PURE,FlashArray
size=6.0T features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:2 sdb 8:16  active ready running
  |- 5:0:0:2 sde 8:64  active ready running
  |- 6:0:0:2 sdq 65:0  active ready running
  `- 7:0:0:2 sdr 65:16 active ready running

after restarting the multipath service it now shows:

Code:

3624a937037dc4aba9bff4b94000113ea dm-7 PURE,FlashArray
size=8.0T features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 4:0:0:2 sdb 8:16  active ready running
  |- 5:0:0:2 sde 8:64  active ready running
  |- 6:0:0:2 sdq 65:0  active ready running
  `- 7:0:0:2 sdr 65:16 active ready running

no change in the GUI yet when looking at the disks on the host

EC2_Jase · Jun 19, 2024

Its fixed and now resized.

I ran pvdisplay to check the size. (before)

Code:

--- Physical volume ---
  PV Name               /dev/mapper/3624a937037dc4aba9bff4b94000113ea
  VG Name               Hidden_Dev_Storage
  PV Size               6.00 TiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              1572863
  Free PE               205308
  Allocated PE          1367555
  PV UUID               EzrR03-gJZS-kGBw-AnvO-YfO3-X3B6-dO50Tw

Then reran the command:
pvresize /dev/mapper/3624a937037dc4aba9bff4b94000113ea

then pvdisplay showed: and confirmed in the GUI

Code:

--- Physical volume ---
  PV Name               /dev/mapper/3624a937037dc4aba9bff4b94000113ea
  VG Name               Apollo_Dev_Storage
  PV Size               8.00 TiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              2097151
  Free PE               729596
  Allocated PE          1367555
  PV UUID               EzrR03-gJZS-kGBw-AnvO-YfO3-X3B6-dO50Tw

looks like restarting the multipath service allowed promox to detect the new size and then the resize worked.

Thanks all

bbgeek17 · Jun 19, 2024

Now that you have the underlying physical device properly updated you need to move up the stack into LVM.
Use the commands provided earlier to expand the Physical LVM volume, followed by LVM Volume Group extension.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Search

Search

Extending LVM on iSCSI

EC2_Jase

New Member

gfngfn256

Renowned Member

EC2_Jase

New Member

gfngfn256

Renowned Member

EC2_Jase

New Member

gfngfn256

Renowned Member

skgle

New Member

bbgeek17

Distinguished Member

EC2_Jase

New Member

Attachments

bbgeek17

Distinguished Member

EC2_Jase

New Member

EC2_Jase

New Member

EC2_Jase

New Member

bbgeek17

Distinguished Member