Proxmox GUI question marks when adjusting/resizing storage with enterprise SANs

N

NoriCopperBrand

New Member
Aug 27, 2026
4
0
1
Hi all,

I'm running into an issue with a 6-node Proxmox cluster connected to a Dell PowerStore and an older Compellent 5020. We're about to add a second cluster of 4 hosts, and we don't have HA set up yet.

While migrating VMs recently, a couple of our datastores got tight on space. To fix it, I went host-by-host to adjust the storage, running rescan-scsi-bus.sh and tweaking LVM sizes.
Every time I did this across multiple hosts within a short time frame of maybe minutes, the Proxmox GUI completely locked up. The VMs themselves kept running fine, but the web UI showed gray question marks, and my open shell sessions froze up.

I don't have logs from when it happened a month ago, but I've narrowed the triggers down to running rescan-scsi-bus.sh and adjusting LVM sizes on multiple hosts consecutively. For context, we currently have our multipath config set with user_friendly_names yes, though we plan to switch to tracking by WWIDs soon.

Does anyone have advice on best practices for expanding storage on the fly with enterprise SANs in Proxmox, or ideas on what might be causing these GUI/storage freezes?

Thanks!
 
Hey NoriCopperBrand,

Sounds like a storage slow down to me. Something is unhappy about the rescan as you point out. When PVE shows a question mark on a storage device it means it has lost contact with it- or more specifically pvestatd is unable to collect statistics and therefore can't display them on the web gui, which generally means one of the two is expecting an answer and not getting one. What was the iowait in the PVE GUI for the hosts whose VMs were locking up? Click on the node and go to the summary tab, you'll see CPU usage and then iowait.

To be honest, it sounds like rescan-scsi-bus.sh is provoking a SCSI/multipath stall on one or more SAN paths. pvestatd then blocks while interrogating the affected LVM/SAN storage, so Proxmox stops receiving status updates and paints nodes/VMs/storage with grey ? marks.

Just to be extra clear because your methodology isn't all that clear to me, you're applying the following, right?
  • Grow the LUN on the SAN.
  • Determine the individual paths belonging to that LUN with multipath -ll.
  • Rescan those path devices specifically:
    • echo 1 > /sys/block/sdX/device/rescan
  • Tell multipath to resize that map:
    • multipathd resize map <map>
  • Verify the new size.
  • Then perform the LVM operation.
If you're rescanning everything over and over, you're probably causing some huge io pressure and stalls. One more thing, IDK about your specific SANs but some of them will spectacularly fail or hang if you query something with broken paths. You should send over your /etc/multipath.conf if you want me to take a look. Certified storage badass over here /s hah I'm actually coming back to technical challenges after three years as a high level IT planner, I'm pretty rusty.
Cheers,


Tmanok
 
  • Like
Reactions: Onslow
I have never used the rescan-scsi-bus.sh command you are referring to and have never experienced the issue you are experiencing. So, I am guessing your issue is related to this command.
Here are the steps I use. Obtained from various sources. Might be some steps that aren't necessary, but it works without issue.
How to resize an iSCSI LUN in Proxmox.
This process assumes you have already set up your Proxmox cluster with multipath and have already set up the resize in your SAN.

Display information about all physical volumes (PVs) with additional tag details, your LUN_name will be in /dev/mapper/Your_LUN_name. It will also show your device identifier.
pvs -o +tags

Show detailed information about the specified multipath device 'Your_LUN_name'.
multipath -ll Your_LUN_name

List information about all available block devices in a tree-like format, so you can check the volume size.
lsblk

Trigger a rescan of the SCSI device 'sdX' (Inform your device identifier) to detect any changes (e.g., increased LUN size).
echo 1 >/sys/block/sdX/device/rescan

Rescan Multipath device
multipathd resize map <mpathname>
EX: multipathd resize map mpatha

Reload the multipath configuration to apply any changes detected during the rescan.
multipath -r

Show updated detailed information about the specified multipath device 'LUN_name'.
multipath -ll Your_LUN_name

STOP
Repeat above process on other hosts in cluster so they have detected the new size. Then do remaining step below.

Resize the physical volume to use any additional space detected on the multipath device. Do on only one host.
pvresize /dev/mapper/Your_LUN_name

After this then it should be automatically updated in Proxmox.

-Craig
 
Cutthroat said:
I have never used the rescan-scsi-bus.sh command you are referring to and have never experienced the issue you are experiencing. So, I am guessing your issue is related to this command.
Here are the steps I use. Obtained from various sources. Might be some steps that aren't necessary, but it works without issue.
How to resize an iSCSI LUN in Proxmox.
This process assumes you have already set up your Proxmox cluster with multipath and have already set up the resize in your SAN.

Display information about all physical volumes (PVs) with additional tag details, your LUN_name will be in /dev/mapper/Your_LUN_name. It will also show your device identifier.
pvs -o +tags

Show detailed information about the specified multipath device 'Your_LUN_name'.
multipath -ll Your_LUN_name

List information about all available block devices in a tree-like format, so you can check the volume size.
lsblk

Trigger a rescan of the SCSI device 'sdX' (Inform your device identifier) to detect any changes (e.g., increased LUN size).
echo 1 >/sys/block/sdX/device/rescan

Rescan Multipath device
multipathd resize map <mpathname>
EX: multipathd resize map mpatha

Reload the multipath configuration to apply any changes detected during the rescan.
multipath -r

Show updated detailed information about the specified multipath device 'LUN_name'.
multipath -ll Your_LUN_name

STOP
Repeat above process on other hosts in cluster so they have detected the new size. Then do remaining step below.

Resize the physical volume to use any additional space detected on the multipath device. Do on only one host.
pvresize /dev/mapper/Your_LUN_name

After this then it should be automatically updated in Proxmox.

-Craig
Click to expand...



For resizing and general creation I was performing the following:

Resizing:

extend lun on san

rescan-scsi-bus.sh

multipath -ll (to see the changes in storage size or find the new lun and friendly name)

lsblk -l (make sure the lun and disks line up)
fdisk -l (make sure the lun and disks line up)

multipathd resize map mpathX
pvresize /dev/mapper/mpathX

From there I was going to each host and running the rescan-scsi-bus.sh command to get the hosts to recognize the storage change but realized too only after the fact when trying to scope out the issue further that the mpath friendly name letters were different on each host.

When moving forward with this next cluster we won't be using the friendly names and will instead identiy by the WWID instead and will have to change later the friendly names to no in our multipath statement for the current cluster.

I didn't realize I was supposed to be doing individual scans when making storage changes and will have to try that out soon.
 
Tmanok said:
Hey NoriCopperBrand,

Sounds like a storage slow down to me. Something is unhappy about the rescan as you point out. When PVE shows a question mark on a storage device it means it has lost contact with it- or more specifically pvestatd is unable to collect statistics and therefore can't display them on the web gui, which generally means one of the two is expecting an answer and not getting one. What was the iowait in the PVE GUI for the hosts whose VMs were locking up? Click on the node and go to the summary tab, you'll see CPU usage and then iowait.

To be honest, it sounds like rescan-scsi-bus.sh is provoking a SCSI/multipath stall on one or more SAN paths. pvestatd then blocks while interrogating the affected LVM/SAN storage, so Proxmox stops receiving status updates and paints nodes/VMs/storage with grey ? marks.

Just to be extra clear because your methodology isn't all that clear to me, you're applying the following, right?
  • Grow the LUN on the SAN.
  • Determine the individual paths belonging to that LUN with multipath -ll.
  • Rescan those path devices specifically:
    • echo 1 > /sys/block/sdX/device/rescan
  • Tell multipath to resize that map:
    • multipathd resize map <map>
  • Verify the new size.
  • Then perform the LVM operation.
If you're rescanning everything over and over, you're probably causing some huge io pressure and stalls. One more thing, IDK about your specific SANs but some of them will spectacularly fail or hang if you query something with broken paths. You should send over your /etc/multipath.conf if you want me to take a look. Certified storage badass over here /s hah I'm actually coming back to technical challenges after three years as a high level IT planner, I'm pretty rusty.
Cheers,


Tmanok
Click to expand...


cat /etc/multipath.conf
defaults {
polling_interval 5
checker_timeout 15
# disable_changed_wwids yes
find_multipaths no
user_friendly_names yes
enable_foreign nvme
}
devices {
device {
vendor DellEMC
product PowerStore
detect_prio "yes"
path_selector "queue-length 0"
path_grouping_policy "group_by_prio"
path_checker tur
failback immediate
fast_io_fail_tmo 5
no_path_retry 3
rr_min_io_rq 1
max_sectors_kb 1024
dev_loss_tmo 10
hardware_handler "1 alua"
}
}



As I said below I was running the rescan each time I made adjustments with the information below to storage to get the question marks to disappear, or I was realizing I could run pvscan --cache or vgscan --cache to get the hosts to recognize the storage adjustments.

I need to create new storage on the same cluster and adjust whats already there but don't want to crash the hypervisor gui again so I need to make sure I handle the changes intelligently as best I can. Im rather new to linux and proxmox but enjoying what I've been learning so far.
 
Hey NoriCopperBrand,

Good news, nothing wrong with your equipment! In other news, you are in for some pleasant learning. First major thing: Do not run these commands on every PVE node:
Code: 
rescan-scsi-bus.sh
multipathd resize map mpathX
pvresize /dev/mapper/mpathX

Also, after you run the resize map, you do not need to force a reload: Don't run multipath -r after multipathd resize map mpathX. multipath -r means force a reload of all existing multipath maps, not just the LUN being expanded.

There's another big eyebrow raiser here which is that your configuration goes directly against Debian's manpages. The Debian multipath documentation says fast_io_fail_tmo should be smaller than dev_loss_tmo, but yours is inverted:
Code: 
fast_io_fail_tmo 5
dev_loss_tmo 10

Dell also agrees with this and recommends PowerStore connected Linux systems use fast_io_fail_tmo 15. What was the reasoning for your configuration change? Maybe don't do what Dell recommends just yet, first send us some multipath info as requested below.
https://www.dell.com/support/kbdoc/...e-to-missing-mpio-configuration-on-hosts-side
https://manpages.debian.org/testing/multipath-tools/multipath.conf.5.en.html

You also have friendly names on, which is fine, but the PVE cluster as a whole may not like that and when you rescan your LUNs sometimes specific nodes may have struggled. There is an interesting behaviour in PVE related to which node you're logged into which will show different behaviour under some very specific circumstances... This might be one of those, but anyway let's take a look at what multipath sees first:
Code: 
multipath -ll
multipathd show paths
multipath -t

Also, can you send your compellent configuration? This is fun!
Cheers!


Tmanok
 
Tmanok said:
Hey NoriCopperBrand,

Good news, nothing wrong with your equipment! In other news, you are in for some pleasant learning. First major thing: Do not run these commands on every PVE node:
Code: 
rescan-scsi-bus.sh
multipathd resize map mpathX
pvresize /dev/mapper/mpathX

Also, after you run the resize map, you do not need to force a reload: Don't run multipath -r after multipathd resize map mpathX. multipath -r means force a reload of all existing multipath maps, not just the LUN being expanded.

There's another big eyebrow raiser here which is that your configuration goes directly against Debian's manpages. The Debian multipath documentation says fast_io_fail_tmo should be smaller than dev_loss_tmo, but yours is inverted:
Code: 
fast_io_fail_tmo 5
dev_loss_tmo 10

Dell also agrees with this and recommends PowerStore connected Linux systems use fast_io_fail_tmo 15. What was the reasoning for your configuration change? Maybe don't do what Dell recommends just yet, first send us some multipath info as requested below.
https://www.dell.com/support/kbdoc/...e-to-missing-mpio-configuration-on-hosts-side
https://manpages.debian.org/testing/multipath-tools/multipath.conf.5.en.html

You also have friendly names on, which is fine, but the PVE cluster as a whole may not like that and when you rescan your LUNs sometimes specific nodes may have struggled. There is an interesting behaviour in PVE related to which node you're logged into which will show different behaviour under some very specific circumstances... This might be one of those, but anyway let's take a look at what multipath sees first:
Code: 
multipath -ll
multipathd show paths
multipath -t

Also, can you send your compellent configuration? This is fun!
Cheers!


Tmanok
Click to expand...


Im going to take the weekend to check some of this out and digest it a bit.

So far since the GUI crashes I haven't touched my storage at all and we've been 100% stable which is great though I am at a point where I need to either adjust or create some new datastores for the purposes of snapshots for specific machines needing break fixes.

I will feel much better when I get can my veeam up and running for the whole cluster in the coming week or start looking into the ProxMox Backup Server. Thank you very much so far for the education!
 
You must log in or register to reply here.
Top Bottom
Back