iSCSI Shared Storage Resize whilst system still live

[Luca Malaspina]

Morning,


I have successfully resize shared storage before on Proxmox BEFORE my cluster went to host many production servers.


I need to expand my shared storage again and run pvresize, is this safe to do whilst VM's are still running?


If not, could I live migrate all VMs to one host, resize the host with no VMs, move them all over and repeat?


Is this possible or do all VM's need to be switched off?


Thanks,

Luca

 

[manu]

What volume do you want to extend with pvresize ?

 

[Luca Malaspina]

What volume do you want to extend with pvresize ?

I would like to extend my main iSCSI Array Storage volume where my VM disk's are stored.

 

[manu]

Enlarging the PV while the logical volumes should work, but I would advise you to test this on a test storage.
Procedure is more or less as such:
* enlarge the LUN on your SAN
* rescan the iscsi targets on the PVE host
iscsiadm --mode session --rescan
* check with lsblk that the linux block device has been resized
(you can find with lsscsi --transport the mapping between target and block device )
lsblk /dev/sdX
* then proceed with
pveresize /dev/sdX

the Volume Group using the /dev/sdX Physical Volume shoud automatically pick up the new available space

PVE does use the lvmetad, so there is not in memory cache of the metadata, so the operation should be safe, as long as you do not try to call pveresize on the same LUN form multiple nodes at the same time

after that you can also Live enlarge your VM disk images, as explained in
https://pve.proxmox.com/wiki/Resize_disks

 

[Luca Malaspina]

Hi,

That's great thank you.

I tried this at the weekend and all seemed to be fine, but now when I try to restore a VM to this storage space, I get the following error message:


'WARNING: Device /dev/mapper/dell-md-lun has size of 4294967296 sectors which is smaller than corresponding PV size of 7516190720 sectors. Was device resized?
One or more devices used as PVs in VG vg_iscsi have changed sizes.
WARNING: Device /dev/mapper/dell-md-lun has size of 4294967296 sectors which is smaller than corresponding PV size of 7516190720 sectors. Was device resized?
One or more devices used as PVs in VG vg_iscsi have changed sizes.
WARNING: Device /dev/mapper/dell-md-lun has size of 4294967296 sectors which is smaller than corresponding PV size of 7516190720 sectors. Was device resized?
One or more devices used as PVs in VG vg_iscsi have changed sizes.
WARNING: Device /dev/mapper/dell-md-lun has size of 4294967296 sectors which is smaller than corresponding PV size of 7516190720 sectors. Was device resized?
One or more devices used as PVs in VG vg_iscsi have changed sizes.
device-mapper: resume ioctl on (253:22) failed: Invalid argument
Unable to resume vg_iscsi-vm--199--disk--1 (253:22)
TASK ERROR: lvcreate 'vg_iscsi/pve-vm-199' error: Failed to activate new LV.'


I have two hosts servers in my cluster, I only seem to get this problem whilst restoring to one of them and not the other (which lets me restore fine)


Have you ever seen anything like this before and know what I could have done incorrectly?

Thankyou,

Luca

 

[manu]

Hi Luca
Probably you're using multipath and you need a way to multipath daemon ti rescan the the underlying block device since they have changed. Check the multipath documentation this is not PVE specific here.
YMMV, but I disabled multipath in my testlab because it was causing me more configurations headaches that anything.

 

[Luca Malaspina]

Hi Luca
Probably you're using multipath and you need a way to multipath daemon ti rescan the the underlying block device since they have changed. Check the multipath documentation this is not PVE specific here.
YMMV, but I disabled multipath in my testlab because it was causing me more configurations headaches that anything.

Hi,

I have just rebooted my host server on the hope that that might miraculously resolve the problem, and it turns out it has done.

I am not 100% why it has fixed it but it seems to be working okay at the moment.

Thanks for your help

Luca

 

[lungao]

Hi,

I want to directly enlarge /dev/sda3，but the command pvresize /dev/sda3 does not work
Can you tell me the way to do it ?

 

[aasami]

Sorry to revive an old thread, but I've come here looking for the same answer. Here is is how you can do an online resize with multipath.
1. Extend LUN on your SAN.

Then, do on every host in a cluster and extend multipath device (this steps are NOT propagated to all nodes in a cluster):
2. Pick which multipath device you wand to enlarge and check it's size for comparison
multipath -l my_shared_lun
3. Rescan ALL paths to the multipath device (change your device names in following commands)
echo 1 > /sys/block/sdA/device/rescan
echo 1 > /sys/block/sdB/device/rescan
echo 1 > /sys/block/sdC/device/rescan
echo 1 > /sys/block/sdD/device/rescan
3. Resize multipath device
multipathd resize map my_shared_lun
4. Check that the system picked up new device size
lsblk|grep my_shared_lun
5. Resize physical volume on it
pvresize /dev/mapper/my_shared_lun
6. Check that corresponding volume group changed ist's size
vgdisplay my_vg
7. Now you can see, in Proxmox GUI the new available space for VMs.

I hope this helps someone.

 

[sfi-nate]

Hi;

Also not trying to kick an old post here, but I followed Manu's instructions, resized the iscsi vol, resized the LUN, did an iscsiadm rescan, and then I only had to do a pvreize on one out of my three hosts in the cluster that all share the same LUN. (Proxmox 6).

Just sharing my experience.

 

[VGE]

Enlarging the PV while the logical volumes should work, but I would advise you to test this on a test storage.
Procedure is more or less as such:
* enlarge the LUN on your SAN
* rescan the iscsi targets on the PVE host
iscsiadm --mode session --rescan
* check with lsblk that the linux block device has been resized
(you can find with lsscsi --transport the mapping between target and block device )
lsblk /dev/sdX
* then proceed with
pveresize /dev/sdX

the Volume Group using the /dev/sdX Physical Volume shoud automatically pick up the new available space

PVE does use the lvmetad, so there is not in memory cache of the metadata, so the operation should be safe, as long as you do not try to call pveresize on the same LUN form multiple nodes at the same time

after that you can also Live enlarge your VM disk images, as explained in
https://pve.proxmox.com/wiki/Resize_disks

Sorry to revive. But

pveresize /dev/sdX

currently is:

pvresize /dev/sdX

 

[bbgeek17]

Sorry to revive. But

pveresize /dev/sdX

currently is:

pvrsize /dev/sdX

that appears to be incorrect: Better way to put it - its neither pveresize nor pvrsize It is and has always been pvresize, as it stands for "Resize Physical Volume" and is not related to Proxmox Virtual Environment.

root@pve8demo1:~# apt install lvm2

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
lvm2 is already the newest version (2.03.16-2).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

root@pve8demo1:~# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.13-1-pve)

root@pve8demo1:~# /usr/sbin/pvresize --version
  LVM version:     2.03.16(2) (2022-05-18)
  Library version: 1.02.185 (2022-05-18)
  Driver version:  4.48.0

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[VGE]

that appears to be incorrect: Better way to put it - its neither pveresize nor pvrsize It is and has always been pvresize, as it stands for "Resize Physical Volume" and is not related to Proxmox Virtual Environment.

root@pve8demo1:~# apt install lvm2

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
lvm2 is already the newest version (2.03.16-2).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

root@pve8demo1:~# pveversion
pve-manager/8.1.4/ec5affc9e41f1d79 (running kernel: 6.5.13-1-pve)

root@pve8demo1:~# /usr/sbin/pvresize --version
  LVM version:     2.03.16(2) (2022-05-18)
  Library version: 1.02.185 (2022-05-18)
  Driver version:  4.48.0

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Sorry! Fixed my fault. Forgot the 'e' pvrEsize is what I meant.

 

[dsmteam]

Old topic but even with multipath, scanning only one path was necessary before running
multipathd resize map my_shared_lun
and
pvresize /dev/mapper/my_shared_lun