Disks have disappeared from the SAN(!)

[yatesco]

(URGENT!!) Disks have disappeared from the SAN

Hi all,

Has anyone experienced SAN backed disks just disappearing?

Yesterday I created about 40 small VMs whose disks were backed by a SAN. Everything was fine until I apt-get update ; apt-get dist-upgrade this morning and then rebooted.

Now some of the VMs won't start and pvesm list --all shows an absence of their disks. Other VMs are fine. I cannot find any pattern to the missing ones either.

Background: cluster of two identical machines each with 2x1GB connections to the SAN. I created the SAN storage (marked shared but not one LVM per machine) on the master and then created an LVM group (marked shared) for the entire size of the SAN storage. I then created the VMs - nothing different between the broken and working ones.

The cluster is running 1.5 - 6.2.18

One very *very* bizarre thing is that the master can ping the SAN however the client node (which has the broken VMs) cannot. Even though it cannot ping the SAN it can still start up some of the VMs with SAN backed disks....

I previously used ESXi on this SAN and have had no problems.

Help would be gratefully received - this is very critical for me.

 

[udo]

Re: (URGENT!!) Disks have disappeared from the SAN

Hi,
i'm not very firm with iscsi - i tried some times and use than fc.
To track down the problem you can use following commands:

dmsetup info
pvdisplay
fdisk -l
iscsiadm --mode node
dmesg | grep sd

Udo

 

[yatesco]

Re: (URGENT!!) Disks have disappeared from the SAN

I think it might have been my fault (isn't it always ). I started using the SAN within seconds of creating the VDisk which was minutes after the RAID 10 was created on brand new disks. The SAN said "initialising" but I thought hey - it is letting me use it so it must be safe.

I repeated the experiment yesterday and the problem seems to have gone away now it is full initialised. Thanks Udo.

(P.S. Also had a moment of panic when the said 'media scrub starting' after I had installed many many GBs of VMs onto it(!). Turns out to be a non-destructive maintenance routine. Phew!)