PVE LVM striping on iSCSI storage (with multipath)

[adis]

We are testing Dell storage (MD3620i) with dual active iSCSI controllers using 1Gbit switch
and dual port 1Gbit NICs on Proxmox hosts (no 10Gbit stuff yet)

Unfortunately, only one controller can access to one virtual disk on storage, so we did the following:
- Configured and tested multipath setup using vendor documentation
- Created two identical "virtual disks" on storage
- Manualy created Volume Group with two physical volumes (from storage "virtual disks")
- Added that volume group to the web interface (as an "existing volume group")

To achieve the final goal (IO over two iSCSI controllers), LVs must be created
with --stripes option (see patch at the bottom).

The result is rather reasonable (180MB/s-200MB/s) from a Debian VPS installed
on "striped" volume group.

Is it possible to add this option to the Proxmox itself, something along the lines:
- "number of stripes" option when creating LV groups
- More elaborate patch for /usr/share/perl5/PVE/Storage.pm, to detect stripes > 0

--- Storage.pm.orig	2011-08-22 12:49:32.000000000 +0200
+++ Storage.pm	2011-08-22 12:51:20.000000000 +0200
@@ -1621,7 +1621,7 @@
 	    die "unable to allocate an image name for VM $vmid in storage '$storeid'\n"
 		if !$name;
 
-	    my $cmd = ['/sbin/lvcreate', '--addtag', "pve-vm-$vmid", '--size', "${size}k", '--name', $name, $vg];
+	    my $cmd = ['/sbin/lvcreate', '--addtag', "pve-vm-$vmid", '--size', "${size}k", '--stripes', '2', '--name', $name, $vg];
 
 	    run_command ($cmd);

 

[dietmar]

To achieve the final goal (IO over two iSCSI controllers), LVs must be created
with --stripes option (see patch at the bottom).

You want that to improve performance?

Is it possible to add this option to the Proxmox itself, something along the lines:
- "number of stripes" option when creating LV groups
- More elaborate patch for /usr/share/perl5/PVE/Storage.pm, to detect stripes > 0

I will add that to the todo list for 2.x

 

[adis]

Yes, striping LV over two different virtual disks/controllers improves performance significantly

 

[dietmar]

Yes, striping LV over two different virtual disks/controllers improves performance significantly

Well, normally this is done by the raid controller. If you use iscsi, a faster net (10GB) should do the same job.

 

[willy1009]

hi,

i try to setup a MD3200i with proxmox cluster. But it seems that don't work.

adis could you show me you lvm.conf and your multipath.conf just for the configuration.

Thanks

 

[lauigo]

Hi adis,
I have also purchased a MD3600i storage device and I'm trying to balance the controllers. I created 2 Virtual Disks(VDs) and assigned VD1 to controller 1 and VD2 to controller2. I built a lvm stripped volume from those 2 VDs(just as you did) but i get terrible slow read performance(2-10MB/s) while write rates are just fine(limited by 1Gb network currently). Can you tell me how did you obtain such high read rates? I tried various segment/stripe sizes but without noticeable improvement.

Looking forward for you answer.
Laurentiu.

We are testing Dell storage (MD3620i) with dual active iSCSI controllers using 1Gbit switch
and dual port 1Gbit NICs on Proxmox hosts (no 10Gbit stuff yet)

Unfortunately, only one controller can access to one virtual disk on storage, so we did the following:
- Configured and tested multipath setup using vendor documentation
- Created two identical "virtual disks" on storage
- Manualy created Volume Group with two physical volumes (from storage "virtual disks")
- Added that volume group to the web interface (as an "existing volume group")

To achieve the final goal (IO over two iSCSI controllers), LVs must be created
with --stripes option (see patch at the bottom).

The result is rather reasonable (180MB/s-200MB/s) from a Debian VPS installed
on "striped" volume group.

Is it possible to add this option to the Proxmox itself, something along the lines:
- "number of stripes" option when creating LV groups
- More elaborate patch for /usr/share/perl5/PVE/Storage.pm, to detect stripes > 0

--- Storage.pm.orig    2011-08-22 12:49:32.000000000 +0200
+++ Storage.pm    2011-08-22 12:51:20.000000000 +0200
@@ -1621,7 +1621,7 @@
         die "unable to allocate an image name for VM $vmid in storage '$storeid'\n"
         if !$name;
 
-        my $cmd = ['/sbin/lvcreate', '--addtag', "pve-vm-$vmid", '--size', "${size}k", '--name', $name, $vg];
+        my $cmd = ['/sbin/lvcreate', '--addtag', "pve-vm-$vmid", '--size', "${size}k", '--stripes', '2', '--name', $name, $vg];
 
         run_command ($cmd);

 

[adis]

1) Could you check your network settings? MTU, flow control on switches etc.

2) Also, consider changing IO scheduler on guests (noop instead of cfq) and on hypervisor.

3) Check your cache settings for virtual disks, by default Proxmox ads cache=none to disks, but the default cache=writethrough works much better for iSCSI
(I'll start another topic on this subject)

 

[lauigo]

Hi,
1. MTU is 9000 on MD, host and Switch(cisco 3750X)
2. i set IO scheduler to all possible values without improvment
3. I'm currently using Centos 5.7 not Proxmox....and i also played with various caches of the MD
Anything else to check?(i already open a ticket to dell but it is very weird that your MD works well while mine it is not)

1) Could you check your network settings? MTU, flow control on switches etc.

2) Also, consider changing IO scheduler on guests (noop instead of cfq) and on hypervisor.

3) Check your cache settings for virtual disks, by default Proxmox ads cache=none to disks, but the default cache=writethrough works much better for iSCSI
(I'll start another topic on this subject)

 

[adis]

Hmm, I don't know how well Centos 5 works with this hardware.

We are using cheap 3COM^^HP switch here (v1910). cfq io scheduler for /dev/sd? and /dev/dm-? devices, cache=writethrough and noop io scheduler for guest.

What's your multipath configuration like?

devices {
	device {
		vendor "DELL"
		product "MD36xxi"
		path_grouping_policy group_by_prio
		prio rdac
		polling_interval 5
		path_checker rdac
		path_selector "round-robin 0"
		hardware_handler "1 rdac"
		failback immediate
		features "2 pg_init_retries 50"
		no_path_retry 30
		rr_min_io 100
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
	}
}

 

[lauigo]

Hi,
I tracked down the issue, it was not the IO scheduler although it point it me to the right direction to search. It was read_ahead_kb parameter; on the /dev/sdX devices was 128 but on the lvm striped volume it was 2048 (this is the way lvm creates it). I switched to 128 on the lvm => read performance was ok. Now i'm struggling with lvm stripe size to get the best value for paralel read from different places on disk; it seems this value is 4096(maximum extent size).
As a side note i have exactly the same multipath conf as you do. I remember that i had to manually add "/sbin/mpath_prio_rdac" file for device-mapper-multipath-0.4.9-23 while device-mapper-multipath-0.4.7X had it by default; but 0.4.7 did not know the "prio" tag... Any idea why?
And another question: are you using this device in production or just testing around?
Thank you for your help .

 

[adis]

Great

In our system (Debian 5.0/Proxmox 1.9) read_ahead_kb parameter is set to 128 on both, LVM devices and iSCSI disks. And LVM stripe size is 64k
(I figured that there is not point in "optimizing" since workloads will vary greatly)


Our version of multipath-tools is 0.4.8.
The only problem we had was system (hypervisor) freezing during shutdown, and we resolved that by hacking multipath init script (stop mode),
adding something like this _before_ '/sbin/multipath -F' line:

lvchange -an /dev/$STORAGE_NAME/vm-$VMID-disk-$DISK_NO

(for every vm disk)

 

[dups]

Have you guys looked at this yet for Proxmox 2.0? I have pretty much the EXACT same setup (except an MD3600i instead of the 3620) and was wanting to do the same thing. I tried modifying that Storage.pm file in PVE 2.0 but it does not seem to have an affect. Checking the LV after its creation indicates a linear stripe instead of a "striped" stripe which it should show if the above modifications still worked. Anybody have any ideas?

TIA!

We are testing Dell storage (MD3620i) with dual active iSCSI controllers using 1Gbit switch
and dual port 1Gbit NICs on Proxmox hosts (no 10Gbit stuff yet)

Unfortunately, only one controller can access to one virtual disk on storage, so we did the following:
- Configured and tested multipath setup using vendor documentation
- Created two identical "virtual disks" on storage
- Manualy created Volume Group with two physical volumes (from storage "virtual disks")
- Added that volume group to the web interface (as an "existing volume group")

To achieve the final goal (IO over two iSCSI controllers), LVs must be created
with --stripes option (see patch at the bottom).

The result is rather reasonable (180MB/s-200MB/s) from a Debian VPS installed
on "striped" volume group.

Is it possible to add this option to the Proxmox itself, something along the lines:
- "number of stripes" option when creating LV groups
- More elaborate patch for /usr/share/perl5/PVE/Storage.pm, to detect stripes > 0

--- Storage.pm.orig    2011-08-22 12:49:32.000000000 +0200
+++ Storage.pm    2011-08-22 12:51:20.000000000 +0200
@@ -1621,7 +1621,7 @@
         die "unable to allocate an image name for VM $vmid in storage '$storeid'\n"
         if !$name;
 
-        my $cmd = ['/sbin/lvcreate', '--addtag', "pve-vm-$vmid", '--size', "${size}k", '--name', $name, $vg];
+        my $cmd = ['/sbin/lvcreate', '--addtag', "pve-vm-$vmid", '--size', "${size}k", '--stripes', '2', '--name', $name, $vg];
 
         run_command ($cmd);

 

[dietmar]

Checking the LV after its creation indicates a linear stripe instead of a "striped" stripe which it should show if the above modifications still worked. Anybody have any ideas?

Nobody worked on that so far.