Bug : iscsi + multipath : luns are not showing in web interface

S

spirit

Distinguished Member
Apr 2, 2010
7,019
1,125
273
www.groupe-cyllene.com
Hi,

i'm trying multipath with iscsi, multipath is ok on host

3600144f0f62f0e0000004cc6f3fe0001dm-3 NEXENTA ,COMSTAR
[size=10G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 5:0:0:0 sdc 8:32 [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 6:0:0:0 sde 8:64 [active][undef]


but luns are not showing in web interface. (without multipath, luns are showing).

i need to use luns directly (no lvm on top).

Can someone can help me ?

i'ill try to hack the perl files, but code is a little complex ;)
 
some progress,
i found the problem,
in file /usr/share/perl5/PVE/Storage.pm ,
sub sub iscsi_device_list

line :

my $blockdev = find_dev_by_id ("/dev/$bdev");
return if !$blockdev;

the $bdev param, are the real iscsi disk (sda,sdb,sdc...).

with multipath, devices in /dev/disk/by-id/ are dm-1,dm-2,dm-3 and not sda,sdb,sdc.

so i need to find a way, to translate sdx to dm-x.

i can do it with multipath -l command with some grep, but it's not very clean.

can someone tell me where in can found multipath mapping informations ?
 
Ok, i found a proper way, here the modified function:
(maybe it can be push upstream ? i don't know how to contact proxmox staff)

Code: 
sub iscsi_device_list {

    my $res = {};

    my $dirname = '/sys/class/iscsi_session';

    dir_glob_foreach ($dirname, 'session(\d+)', sub {
    my ($ent, $session) = @_;

    my $target = file_read_firstline ("$dirname/$ent/targetname");
    return if !$target;

    my (undef, $host) = dir_glob_regex ("$dirname/$ent/device", 'target(\d+):.*');
    return if !defined($host);

    dir_glob_foreach ("/sys/bus/scsi/devices", "$host:" . '(\d+):(\d+):(\d+)', sub {
        my ($tmp, $channel, $id, $lun) = @_;

        my $type = file_read_firstline ("/sys/bus/scsi/devices/$tmp/type");
        return if !defined($type) || $type ne '0'; # list disks only


        my $bdev;
        if (-d "/sys/bus/scsi/devices/$tmp/block") { # newer kernels
        (undef, $bdev) = dir_glob_regex ("/sys/bus/scsi/devices/$tmp/block/", '([A-Za-z]\S*)');
        } else {
        (undef, $bdev) = dir_glob_regex ("/sys/bus/scsi/devices/$tmp", 'block:(\S+)');
        }
        return if !$bdev;

  [B]      #begin check multipath
        my $multipathdev;
        
        if (-d "/sys/block/$bdev/holders") 
        { 
        (undef, $multipathdev) = dir_glob_regex ("/sys/block/$bdev/holders", '([A-Za-z]\S*)');
           if($multipathdev ne '')
           {
              $bdev=$multipathdev;
           }
        }
        #end check multipath[/B]
        
        my $blockdev = find_dev_by_id ("/dev/$bdev");
        return if !$blockdev;

        my $size = file_read_firstline ("/sys/block/$bdev/size");
        return if !$size;
      

        my $volid = "$channel.$id.$lun.$blockdev";

        $res->{$target}->{$volid} = {
        'format' => 'raw', 
        'size' => int($size / 2), 
        'vmid' => 0, # not assigned to any vm
        'channel' => $channel,
        'id' => $id,
        'lun' => $lun,
        };

        #print "TEST: $target $session $host,$bus,$tg,$lun $blockdev\n"; 
    });

    });

    return $res;
}
 
Last edited by a moderator:
spirit said:
Ok, i found a proper way, here the modified function:
(maybe it can be push upstream ? i don't know how to contact proxmox staff)
Click to expand...

Please can you post a correct diff (see 'man diff'). And please use a 'code' tag when you post code to this forum.
 
here the diff (/usr/share/perl5/PVE/Storage.pm)

Code: 
1208a1209,1220
>           #check multipath
>           my $multipathdev;
>           
>           if (-d "/sys/block/$bdev/holders") 
>           { 
>               (undef, $multipathdev) = dir_glob_regex ("/sys/block/$bdev/holders", '([A-Za-z]\S*)');
>               if($multipathdev ne '')
>               {
>                     $bdev=$multipathdev;
>               }
>           }
>
 
V2: we need to modify iscsi_session_rescan sub, because with multipath we have 2(or more) sessions for 1 target.


Code: 
1153c1153
<     my $session = shift;
---
>     my $target = shift;
1172,1174c1172,1191
<     my $cmd = [$ISCSIADM, '--mode', 'session', '-r', $session, '-R'];
<     eval { run_command ($cmd, outfunc => sub {}); };
<     warn $@ if $@;
---
>     
>     my $cmd = [$ISCSIADM, '--mode', 'session'];
> 
>     my $res = {};
> 
>     run_command ($cmd, outfunc => sub {
>       my $line = shift;
> 
>       if ($line =~ m/^tcp:\s+\[(\S+)\]\s+\S+\s+(\S+)\s*$/) {
>           if($2 eq $target)
>           {
>                        my $cmd = [$ISCSIADM, '--mode', 'session', '-r', $1, '-R'];
>                       eval { run_command ($cmd, outfunc => sub {}); };
>                       warn $@ if $@;
> 
>           }
> 
>       }
>     });
> 
1208a1226,1237
>           #check multipath
>           my $multipathdev;
>           
>           if (-d "/sys/block/$bdev/holders") 
>           { 
>               (undef, $multipathdev) = dir_glob_regex ("/sys/block/$bdev/holders", '([A-Za-z]\S*)');
>               if($multipathdev ne '')
>               {
>                     $bdev=$multipathdev;
>               }
>           }
> 
2191c2220
<           iscsi_session_rescan ($iscsi_sess);
---
>           iscsi_session_rescan ($scfg->{target});
 
What do you think about that? Please can you test?

Code: 
Index: Storage.pm
===================================================================
--- Storage.pm  (revision 5120)
+++ Storage.pm  (working copy)
@@ -1096,7 +1096,8 @@
 
        if ($line =~ m/^tcp:\s+\[(\S+)\]\s+\S+\s+(\S+)\s*$/) {
            my ($session, $target) = ($1, $2);
-           $res->{$target} = $session;
+           # there can be several sessions per target (multipath)
+           push @{$res->{$target}}, $session;
        }
     });
 
@@ -1150,7 +1151,7 @@
 my $rescan_filename = "/var/run/pve-iscsi-rescan.lock";
 
 sub iscsi_session_rescan {
-    my $session = shift;
+    my $session_list = shift;
 
     check_iscsi_support ();
 
@@ -1169,9 +1170,11 @@
        utime undef, undef, $rescan_filename;
     }
 
-    my $cmd = [$ISCSIADM, '--mode', 'session', '-r', $session, '-R'];
-    eval { run_command ($cmd, outfunc => sub {}); };
-    warn $@ if $@;
+    foreach my $session (@$session_list) {
+       my $cmd = [$ISCSIADM, '--mode', 'session', '-r', $session, '-R'];
+       eval { run_command ($cmd, outfunc => sub {}); };
+       warn $@ if $@;
+    }
 }
 
 sub iscsi_device_list {
@@ -1203,6 +1206,12 @@
            }
            return if !$bdev;
 
+           #check multipath           
+           if (-d "/sys/block/$bdev/holders") { 
+               my $multipathdev = dir_glob_regex ("/sys/block/$bdev/holders", '[A-Za-z]\S*');
+               $bdev = $multipathdev if $multipathdev;
+           }
+
            my $blockdev = find_dev_by_id ("/dev/$bdev");
            return if !$blockdev;

BTW, please use 'diff -u' when you generate patches (that is more readable)
 
it's ok for me, works great and better code !




next thing i have in mind, is the not working volume_is_used sub with iscsi.(we can mount 1 lun to 2 vm...very dangerous).

it can be easy with 1 server (just parse vm config files to see if the lun is used),
but i don't see how it can be done in a cluster for the moment , because we don't have a centralized configuration repository ....
maybe with a soap query on each cluster member ?
 
spirit said:
it's ok for me, works great and better code !
Click to expand...

OK, I uploaded the fix to:

ftp://download.proxmox.com/debian/d...nary-amd64/libpve-storage-perl_1.0-15_all.deb

spirit said:
next thing i have in mind, is the not working volume_is_used sub with iscsi.(we can mount 1 lun to 2 vm...very dangerous).

it can be easy with 1 server (just parse vm config files to see if the lun is used),
but i don't see how it can be done in a cluster for the moment , because we don't have a centralized configuration repository ....
maybe with a soap query on each cluster member ?
Click to expand...

May I suggest you wait with that until we have 2.0. In 2.0 we have a full featured cluster communication stack (corosync), and a replicated, distributed configuration storage. I guess those things will be helpful to solve such problems.

Besides, do you know how libvirt solves such problems?
 
yes, with 2.0 , i think it can be more easy . but the question is, when ? (i know when it'll be done ;).
More seriously, could you tell us if you have planned to release it in 6 month, 1 year, 2 years ?

for libvirt,
i have found this informations here:

http://berrange.com/topics/libvirt/

"In particular, notice how the guest uses the huge paths under /dev/disk/by-path to refer to the LUNs, and also that the second disk has the <shareable/> flag set. This ensures the SELinux labelling allows multiple guests to access the disk and that all I/O caching is disabled on the host. Both critically important if you want the disk to be safely shared between guests."

I'm new with kvm, so maybe it's possible to mount a disk to 2 vm ? (with a cluster filesystem like gfs,ocfs2,...)
 
spirit said:
More seriously, could you tell us if you have planned to release it in 6 month, 1 year, 2 years ?
Click to expand...

The plan is to release within 6 month.

spirit said:
I'm new with kvm, so maybe it's possible to mount a disk to 2 vm ? (with a cluster filesystem like gfs,ocfs2,...)
Click to expand...

Yes, that is perfectly valid (although i never tested that).
 
new patch, we need to modify the iscsi_login and iscsi_discovery subs, because we can have more than 1 portal by target. (1 portal by path).

so just use the first portal ip (configured in webinterface), to find the others portals of the target and to log in this portals.


Code: 
--- /root/origprox/PVE/Storage.pm       2010-10-28 12:40:32.000000000 +0200
+++ /usr/share/perl5/PVE/Storage.pm     2010-11-01 17:35:39.000000000 +0100
@@ -1119,7 +1119,7 @@
        if ($line =~ m/^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d+)\,\S+\s+(\S+)\s*$/) {
            my $portal = $1;
            my $target = $2;
-           $res->{$target} = $portal;
+           push @{$res->{$target}}, $portal;
        }
     });
 
@@ -1127,15 +1127,18 @@
 }
 
 sub iscsi_login {
-    my ($target, $portal) = @_;
+    my ($target, $portal_in) = @_;
 
     check_iscsi_support ();
 
-    iscsi_discovery ($portal);
+    my $res=iscsi_discovery ($portal_in);
+    my $portal_list=$res->{$target};
 
+    foreach my $portal (@$portal_list) {
     my $cmd = [$ISCSIADM, '--mode', 'node', '--portal', $portal, 
               '--targetname',  $target, '--login'];
     run_command ($cmd);
+    }
 }
 
 sub iscsi_logout {
 
spirit said:
new patch, we need to modify the iscsi_login and iscsi_discovery subs, because we can have more than 1 portal by target. (1 portal by path).
Click to expand...

I am finally confused by those functionallity. Please can you describe how you want to handle multipathing with pve? I though multipath-tools manage that for us? Please can you post an example config?

spirit said:
so just use the first portal ip (configured in webinterface), to find the others portals of the target and to log in this portals.
Click to expand...

I don't think that we should run iscsi_discovery() each time we want do iscsi_login(). Instead that should be stored in the configuration file.
 
dietmar said:
I am finally confused by those functionallity. Please can you describe how you want to handle multipathing with pve? I though multipath-tools manage that for us? Please can you post an example config?
Click to expand...

the only thing that multipath-tools do , is to agregate 2 devices (ex : sda from path1, and sdb from path2) in one like dm-0.

Code: 
#multipath -l

3600144f0f62f0e0000004cc98584000edm-0 NEXENTA ,COMSTAR       
[size=20G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 8:0:0:0 sda 8:80  [active][undef]
 \_ 7:0:0:0 sdb 8:64  [active][undef]


But for do that, we need to do iscsi_login to each portal (ips) of the target.

for the moment, we only have 1 portal in target configuration,so we have only 1 login to 1 path.

so we have /dev/sda , but not /dev/sdb.

so multipathing is working, but in dm-0 , we have only /dev/sda.

Code: 
#multipath -l

3600144f0f62f0e0000004cc98584000edm-0 NEXENTA ,COMSTAR       
[size=20G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 8:0:0:0 sda 8:80  [active][undef]


so in my patch, i use iscsi_discovery to the first portal, to find others portal of the target.

ex: 1 have 2 portals for 1 target, 10.5.0.18 and 10.6.0.18.

Code: 
# iscsiadm --mode discovery --type sendtargets --portal  10.5.0.18

10.5.0.18:3260,3 iqn.1986-03.com.sun:02:316dd6a9-76bc-62ea-93fa-d0140e876a4b
10.6.0.18:3260,2 iqn.1986-03.com.sun:02:316dd6a9-76bc-62ea-93fa-d0140e876a4b

but if you want, we can defined all portals (ip) of the target in the configuration file;
I just reused iscsi_discovery because it was already use in the iscsi_login code.

is it ok for you ?



there is another case i cannot test, is multipathing with 2 active targets with 2 portals each.
(2 active san controller on a shared jbod disk storage array).
in these case, we need to make a targetgroup, an login on each target and each portal of theses targets.
this need more modifications in the code I think.
 
spirit said:
I just reused iscsi_discovery because it was already use in the iscsi_login code.

is it ok for you ?
Click to expand...

Sure - I just need to understand before I make such modifications ;-)


spirit said:
there is another case i cannot test, is multipathing with 2 active targets with 2 portals each.
(2 active san controller on a shared jbod disk storage array).
in these case, we need to make a targetgroup, an login on each target and each portal of theses targets.
this need more modifications in the code I think.
Click to expand...

What would be needed to test that?
 
i need hardware ;)

i'have a nexenta iscsi san (in ha cluster, so 2 controller with a shared jbod), be only active/passive.
so when the first controller fail, the target is migrated on second controller.

some expensive san have active/active controller, so 2 differents targets with 1,2,3.... paths.

on linux side, we have /dev/sda (target1-path1) , /dev/sdb (target1-path2) , /dev/sdc (target2-path1), /dev/sdd (target2-path2),
all are the same disk, and multipath aggregate them.
 
You must log in or register to reply here.
Top Bottom