iSCSI Best Practice

G

GoatMaster

Active Member
Oct 4, 2017
41
1
28
35
Is there a Best Practice (Whitepaper) how to connect a iSCSI SAN to Proxmox ?

Including multipathing, jumbo frames...
 
  • Like
Reactions: tomtomz
For connecting a SAN to Proxmox.
* create a large LUN on the SAN side
* add an ISCSI LUN using Datacenter -> Add storage
* add a Volume Group on the LUN using Add VG -> Use existing Base Volume
* this way you can use a on the large LUN LVs for VMs, without having to create LUNs for each VMs.

For Configuring Multipath:
* you should use use a Multipath conf provided by your SAN Vendor. A proper Multipath configuration is too complicated to do general advices.

For MTU 9000:
* IMHO a lot of headaches for a minimal 10 % througput/latency benefit (see http://longwhiteclouds.com/2013/09/10/the-great-jumbo-frames-debate/ for benchmarks) After connecting to your SAN, benchmark it with FIO and based on that you can choose to configure your ICSI path to use 9000 bytes frames.
 
So I set everything up with the vendor settings.

I still have a message that is shown in the kvm of the node.

device-mapper: multipath: blk_get_request() returned -12 - requeuing

Where is this coming from and what can I do?
 
GoatMaster said:
I still have a message that is shown in the kvm of the node.

device-mapper: multipath: blk_get_request() returned -12 - requeuing
Click to expand...

Wait ... do you have the multipathed iSCSI on the host or on the guest? If you have it on the host, the guest should not see multipath and if the error is on the very same guest you should not have multipath enabled.
 
GoatMaster said:
I only have it on the host. The host is giving this message.
Click to expand...

Ah, so KVM does not mean kernel virtual machine (VM), but some kind of terminal output. Understood.

Does the error occur only once or does it occur frequently? Have you tested your setup with fio to benchmark it properly? If not, do so. Examples about fio are everywhere in the forums.
 
I have the same messages in the syslog.
Code: 
Dec 29 16:36:44 pve1 kernel: [ 2169.882847] device-mapper: multipath: blk_get_request() returned -12 - requeuing
Dec 29 16:36:48 pve1 kernel: [ 2173.776888] multipath_clone_and_map: 12 callbacks suppressed
Dec 29 16:36:48 pve1 kernel: [ 2173.776889] device-mapper: multipath: blk_get_request() returned -12 - requeuing
my storage 10Gb iSCSI - MD3800i with dual RAID controller modules
server with two 10Gb cards
Code: 
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 03)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 03)
08:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
my conf
proxmox-ve: 5.1-32 (running kernel: 4.13.13-2-pve)
pve-manager: 5.1-41 (running version: 5.1-41/0b958203)
pve-kernel-4.13.13-2-pve: 4.13.13-32
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-18
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9
Code: 
auto lo
iface lo inet loopback

iface eno3 inet manual
iface eno4 inet manual

auto eno1
iface eno1 inet static
        address  192.168.130.3
        netmask  255.255.255.0
        mtu     9000

auto eno2
iface eno2 inet static
        address  192.168.131.3
        netmask  255.255.255.0
        mtu     9000

auto bond0
iface bond0 inet manual
        slaves eno3 eno4
        bond_miimon 100
        bond_mode balance-rr

auto vmbr0
iface vmbr0 inet static
        address  192.168.5.41
        netmask  255.255.255.0
        gateway  192.168.5.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
Code: 
tcp: [1] 192.168.130.2:3260,2 iqn.1984-05.com.dell:powervault.md3800i.600axxxxxxx (non-flash)
tcp: [2] 192.168.131.1:3260,1 iqn.1984-05.com.dell:powervault.md3800i.600axxxxxxx (non-flash)
tcp: [3] 192.168.130.1:3260,1 iqn.1984-05.com.dell:powervault.md3800i.600axxxxxxx (non-flash)
tcp: [4] 192.168.131.2:3260,2 iqn.1984-05.com.dell:powervault.md3800i.600axxxxxxx (non-flash)
Code: 
defaults {
        polling_interval        2
        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        uid_attribute           ID_SERIAL
        rr_min_io               100
        failback                immediate
        no_path_retry           queue
        user_friendly_names     yes
}
blacklist {
 wwid .*
}
blacklist_exceptions {
wwid 3600a098000bd35ab000004e15a415ccb
wwid 3600a098000bd352d000006bb5a415c69
wwid 3600a098000bd352d000006bd5a415d80
}
devices {
        device {
                vendor                  "DELL"
                product                 "MD38xxi"
                path_grouping_policy    group_by_prio
                prio                    rdac
                polling_interval        5
                path_checker            rdac
                path_selector           "round-robin 0"
                hardware_handler        "1 rdac"
                failback                immediate
                features                "2 pg_init_retries 50"
                no_path_retry           30
                rr_min_io               100
        }
}
multipaths {
        multipath {
                wwid 3600a098000bd352d000006bd5a415d80
                alias dell-backups
        }
        multipath {
                wwid 3600a098000bd352d000006bb5a415c69
                alias dell-data
        }
        multipath {
                wwid 3600a098000bd35ab000004e15a415ccb
                alias dell-db
        }
}
Code: 
dell-data (3600a098000bd352d000006bb5a415c69) dm-6 DELL,MD38xxi
size=1.0T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 12:0:0:1 sdf 8:80  active ready running
| `- 13:0:0:1 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 14:0:0:1 sdg 8:96  active ready running
  `- 11:0:0:1 sdh 8:112 active ready running
dell-db (3600a098000bd35ab000004e15a415ccb) dm-5 DELL,MD38xxi
size=444G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=12 status=enabled
| |- 11:0:0:0 sdb 8:16  active ready running
| `- 14:0:0:0 sdd 8:48  active ready running
`-+- policy='round-robin 0' prio=11 status=active
  |- 13:0:0:0 sdc 8:32  active ready running
  `- 12:0:0:0 sde 8:64  active ready running
dell-backups (3600a098000bd352d000006bd5a415d80) dm-7 DELL,MD38xxi
size=7.0T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 13:0:0:2 sdj 8:144 active ready running
| `- 12:0:0:2 sdk 8:160 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 11:0:0:2 sdl 8:176 active ready running
  `- 14:0:0:2 sdm 8:192 active ready running
Code: 
        global_filter = [ "a|/dev/sda3|", "a|/dev/mapper/dell-.*|", "r|.*|" ]
For Dell storage need use specific rdac options in multipath.conf
https://pve.proxmox.com/wiki/ISCSI_Multipath#Dell
otherwise LUN thrashing problem will occur in cluster configuration.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back