updated dm-switch

Hi

This is updated dm-switch. It has better documentation and better error reporting in message function.

It also fixes bugs introduced when someone reworked my original the code. The code placed here
http://people.redhat.com/agk/patches/linux/editing/dm-add-switch-target.patch
is very buggy - it has argument parsing bugs that prevent it from being loaded at all and uninitialized sctx->nr_paths variable that results in memory corruption.

Mikulas

---

From: Jim Ramsay <[email protected]>

dm-switch is a new target that maps IO to underlying block devices efficiently when there are a large number of fixed-sized address regions but there is no simple pattern to allow for a compact mapping representation such as dm-stripe.

Motivation
----------

Dell EqualLogic and some other iSCSI storage arrays use a distributed frameless architecture. In this architecture, the storage group consists of a number of distinct storage arrays ("members"), each having independent controllers, disk storage and network adapters. When a LUN is created it is spread across multiple members. The details of the spreading are hidden from initiators connected to this storage system. The storage group exposes a single target discovery portal, no matter how many members are being used. When iSCSI sessions are created, each session is connected to an eth port on a single member. Data to a LUN can be sent on any iSCSI session, and if the blocks being accessed are stored on another member the IO will be forwarded as required. This forwarding is invisible to the initiator. The storage layout is also dynamic, and the blocks stored on disk may be moved from member to member as needed to balance the load.

This architecture simplifies the management and configuration of both the storage group and initiators. In a multipathing configuration, it is possible to set up multiple iSCSI sessions to use multiple network interfaces on both the host and target to take advantage of the increased network bandwidth. An initiator could use a simple round robin algorithm to send IO across all paths and let the storage array members forward it as necessary, but there is a performance advantage to sending data directly to the correct member.

The Device Mapper table architecture already supports designating different address regions with different targets, but in our architecture the LUN is spread with an address region size on the order of 10s of MBs, which means the resulting DM table could have more than a million entries and consume far too much memory.

Solution
--------
Based on earlier discussion with the dm-devel contributors, we have solved this problem by using Device Mapper to build a two-layer device hierarchy:

Upper Tier – Determine which array member the IO should be sent to. Lower Tier – Load balance amongst paths to a particular member.

The lower tier consists of a single dm multipath device for each member. Each of these multipath devices contains the set of paths directly to the array member in one priority group, and leverages existing path selectors to load balance amongst these paths. We also build a non-preferred priority group containing paths to other array members for failover reasons.

The upper tier consists of a single dm switch device, using the new DM target module. This device uses a bitmap to look up the location of the IO and choose the appropriate lower tier device to route the IO. By using a bitmap we are able to use 4 bits for each address range in a 16 member group (which is very large for us). This is a much denser representation than the DM table B-tree can achieve.

Though we have developed this target for a specific storage device, we have made an effort to keep it as general purpose as possible in the hope that others may benefit.

Originally developed by Jim Ramsay. Simplified by Mikulas Patocka.

FIXME reformatted new-style documentation file, complete code review (table storage/message parsing), finish variable/fn naming review

---
 Documentation/device-mapper/switch.txt |   83 +++++
 drivers/md/Kconfig                     |   14 
 drivers/md/Makefile                    |    1 
 drivers/md/dm-switch.c                 |  539 +++++++++++++++++++++++++++++++++
 4 files changed, 637 insertions(+)

Index: linux-3.9.3-fast/Documentation/device-mapper/switch.txt
===================================================================
@@ -0,0 +1,83 @@
+dm-switch
+=========
+
+dm-switch target is suitable for Dell EqualLogic storage system.
+
+The EqualLogic storage consists of several nodes. Each host is connected
+to each node. The host may send I/O requests to any node, the node that
+received the requests forwards it to the node where the data is stored.
+
+However, there is a performance advantage of sending I/O requests to the
+node where the data is stored to avoid forwarding. The dm-switch targets
+is created to use this performance advantage.
+
+The dm-switch target splits the device to fixed-size regions. It maintains
+a redirection table that maps regions to storage nodes. Every request is
+forwarded to the corresponding storage node specified in the redirection
+table. The table may be changed with messages while the dm-switch target
+is running.
+
+
+Construction Parameters
+=======================
+
+    <num_nodes> <region_size> <num_optional_arguments>
+    for each node: <device_name> <offset>
+
+<num_nodes>
+    The number of nodes in the EqualLogic storage.
+
+<region_size>
+    The number of 512-byte sectors in a page. Each page can be redirected
+    to a specified node.
+
+<num_optional_arguments>
+    The number of optional arguments. Currently, there are no optional
+    arguments possible and it must be zero.
+
+<device_name>
+    The block device that is used to access the specific node.
+
+<offset>
+    The offset of data start on the specific node (in the unit of
+    512-byte sectors). This number is added to the sector number when
+    forwarding the request to the specific node. Typically, it is zero.
+
+
+Messages
+========
+
+set_region_mappings <index>:<node> [<index>]:<node> [<index>]:<node>...
+
+Modify the region table - set which regions are redirected to which nodes.
+
+<index>
+    Hexadecimal argument.
+    The region number (region size was specified in constructor parameters).
+    If index is omitted, a next region (previous index + 1) is used
+
+<node>
+    Hexadecimal argument.
+    The number number (in the range 0 ... <num_nodes> - 1)
+
+
+Status
+======
+
+There is no status line.
+
+
+Example
+=======
+
+Assume that you have volumes vg1/switch0 vg1/switch1 vg1/switch2 with
+the same size.
+
+Create a switch device with 64kB region size:
+    dmsetup create switch --table "0 `blockdev --getsize /dev/vg1/switch0`
+	switch 3 128 0 /dev/vg1/switch0 0 /dev/vg1/switch1 0 /dev/vg1/switch2 0"
+
+Set mappings for the first 7 entries to point to devices switch0, switch1,
+switch2, switch0, switch1, switch2, switch1:
+    dmsetup message switch 0 set_region_mappings 0:1 :2 :0 :1 :2 :0 :1
+

Index: linux-3.9.3-fast/drivers/md/Kconfig
===================================================================
@@ -428,4 +428,18 @@ config DM_VERITY
 
 	  If unsure, say N.
 
+config DM_SWITCH
+	tristate "Switch target support (EXPERIMENTAL)"
+	depends on BLK_DEV_DM
+	---help---
+	  This device-mapper target creates a device that supports an arbitrary
+	  mapping of fixed-size regions of I/O across a fixed set of paths.
+	  The path uses for any specific region can be switched dynamically
+	  by sending the target a message.
+
+	  To compile this code as a module, choose M here: the module will
+	  be called dm-switch.
+
+	  If unsure, say N.
+
 endif # MD

Index: linux-3.9.3-fast/drivers/md/Makefile
===================================================================
@@ -53,6 +53,7 @@ obj-$(CONFIG_DM_CACHE)		+= dm-cache.o
 obj-$(CONFIG_DM_CACHE_MQ)	+= dm-cache-mq.o
 obj-$(CONFIG_DM_CACHE_CLEANER)	+= dm-cache-cleaner.o
 obj-$(CONFIG_DM_ZEROED)		+= dm-zeroed.o
+obj-$(CONFIG_DM_SWITCH)		+= dm-switch.o
 
 ifeq ($(CONFIG_DM_UEVENT),y)
 dm-mod-objs			+= dm-uevent.o

...

Interesting. I have been working on something similar recently, for not dissimilar reasons. However the approach I had been taking was to extend __choose_pgpath to take a pos parameter indicating the block being requested, and then add a new dm-pref-hint multipath driver, which effectively does a round-robin but prefers any path that has the "pref" bit. Currently the dm-pref-hint driver is extremely hacky and reaches down into the scsi_device to do an appropriate CDB along each path to get the pref-map, or I'd post it for comments.

I'm interested in why the decision was made to have dm-switch as a separate device-mapper layer? Are there a corresponding set of patches to multipath-tools which know how to do the setup of each individual path?

J.

Is this real - that's the first thing I think every morning.
This .sig brought to you by the letter N and the number 24
Product of the Republic of HuggieTag

It is a separate target because it was originally developed as such by Dell.

It could be possible to turn it into a plugin to dm-mpath but that would require changes in the userspace code that is driving it.

Mikulas

Based on my own experience I would venture that the reason it was developed as such is that currently there's no way to ship a dm-mpath plugin that can achieve this, because the plugin interface doesn't have the appropriate information (the request offset) to make the decision. So if you want to ship something that'll work with whatever enterprise kernel the server is running you have to layer on top.

I present the patch below as potential solution. It adds a pos parameter providing this information, and is against 3.10-rc4.

I think this allows a cleaner solution. multipath-tools can continue to manage the paths (including things like dealing ALUA infomation), and then the underlying dm-mpath plugin gets to worry about which of the deemed available paths is the best one to use for that particular offset.

As a separate part, is there a way to feed information into a dm-mpath plugin, after its up and running? I'd like to take my pref-hints driver and make it suitable for submission too, but first I need a cleaner way of getting it the information it needs than it reaching into the SCSI layer.

-----

index bdf26f5..3cbf9d5 100644
@@ -292,11 +292,11 @@ static void __switch_pg(struct multipath *m, struct pgpath *pgpath)
 }
 
 static int __choose_path_in_pg(struct multipath *m, struct priority_group *pg,
-			       size_t nr_bytes)
+			       size_t pos, size_t nr_bytes)
 {
 	struct dm_path *path;
 
-	path = pg->ps.type->select_path(&pg->ps, &m->repeat_count, nr_bytes);
+	path = pg->ps.type->select_path(&pg->ps, &m->repeat_count, pos, nr_bytes);
 	if (!path)
 		return -ENXIO;
 

@@ -308,7 +308,7 @@ static int __choose_path_in_pg(struct multipath *m, struct priority_group *pg,
 	return 0;
 }
 
-static void __choose_pgpath(struct multipath *m, size_t nr_bytes)
+static void __choose_pgpath(struct multipath *m, size_t pos, size_t nr_bytes)
 {
 	struct priority_group *pg;
 	unsigned bypassed = 1;

@@ -320,12 +320,12 @@ static void __choose_pgpath(struct multipath *m, size_t nr_bytes)
 	if (m->next_pg) {
 		pg = m->next_pg;
 		m->next_pg = NULL;
-		if (!__choose_path_in_pg(m, pg, nr_bytes))
+		if (!__choose_path_in_pg(m, pg, pos, nr_bytes))
 			return;
 	}
 
 	/* Don't change PG until it has no remaining paths */
-	if (m->current_pg && !__choose_path_in_pg(m, m->current_pg, nr_bytes))
+	if (m->current_pg && !__choose_path_in_pg(m, m->current_pg, pos, nr_bytes))
 		return;
 
 	/*

@@ -338,7 +338,7 @@ static void __choose_pgpath(struct multipath *m, size_t nr_bytes)
 		list_for_each_entry(pg, &m->priority_groups, list) {
 			if (pg->bypassed == bypassed)
 				continue;
-			if (!__choose_path_in_pg(m, pg, nr_bytes)) {
+			if (!__choose_path_in_pg(m, pg, pos, nr_bytes)) {
 				if (!bypassed)
 					m->pg_init_delay_retry = 1;
 				return;

@@ -373,6 +373,7 @@ static int map_io(struct multipath *m, struct request *clone,
 {
 	int r = DM_MAPIO_REMAPPED;
 	size_t nr_bytes = blk_rq_bytes(clone);
+	size_t pos = blk_rq_pos(clone);
 	unsigned long flags;
 	struct pgpath *pgpath;
 	struct block_device *bdev;

@@ -383,7 +384,7 @@ static int map_io(struct multipath *m, struct request *clone,
 	/* Do we need to select a new pgpath? */
 	if (!m->current_pgpath ||
 	    (!m->queue_io && (m->repeat_count && --m->repeat_count == 0)))
-		__choose_pgpath(m, nr_bytes);
+		__choose_pgpath(m, pos, nr_bytes);
 
 	pgpath = m->current_pgpath;
 

@@ -489,7 +490,7 @@ static void process_queued_ios(struct work_struct *work)
 	spin_lock_irqsave(&m->lock, flags);
 
 	if (!m->current_pgpath)
-		__choose_pgpath(m, 0);
+		__choose_pgpath(m, 0, 0);
 
 	pgpath = m->current_pgpath;
 

@@ -1569,7 +1570,7 @@ again:
 	spin_lock_irqsave(&m->lock, flags);
 
 	if (!m->current_pgpath)
-		__choose_pgpath(m, 0);
+		__choose_pgpath(m, 0, 0);
 
 	pgpath = m->current_pgpath;
 

index e7d1fa8..e06839a 100644
@@ -57,6 +57,7 @@ struct path_selector_type {
 	 */
 	struct dm_path *(*select_path) (struct path_selector *ps,
 					unsigned *repeat_count,
+					size_t pos,
 					size_t nr_bytes);
 
 	/*

index 3941fae..b574fea1 100644
@@ -169,7 +169,7 @@ static int ql_reinstate_path(struct path_selector *ps, struct dm_path *path)
  * Select a path having the minimum number of in-flight I/Os
  */
 static struct dm_path *ql_select_path(struct path_selector *ps,
-				      unsigned *repeat_count, size_t nr_bytes)
+				      unsigned *repeat_count, size_t pos, size_t nr_bytes)
 {
 	struct selector *s = ps->context;
 	struct path_info *pi = NULL, *best = NULL;

index 6ab1192..5b57e07 100644
@@ -163,7 +163,7 @@ static int rr_reinstate_path(struct path_selector *ps, struct dm_path *p)
 }
 
 static struct dm_path *rr_select_path(struct path_selector *ps,
-				      unsigned *repeat_count, size_t nr_bytes)
+				      unsigned *repeat_count, size_t pos, size_t nr_bytes)
 {
 	struct selector *s = (struct selector *) ps->context;
 	struct path_info *pi = NULL;

index 9df8f6b..5ccf2d1 100644
@@ -256,7 +256,7 @@ static int st_compare_load(struct path_info *pi1, struct path_info *pi2,
 }
 
 static struct dm_path *st_select_path(struct path_selector *ps,
-				      unsigned *repeat_count, size_t nr_bytes)
+				      unsigned *repeat_count, size_t pos, size_t nr_bytes)
 {
 	struct selector *s = ps->context;
 	struct path_info *pi = NULL, *best = NULL;

-----

J.

] http://www.earth.li/~noodles/ []   Mistakes aren't always regrets.   [
]  PGP/GPG Key @ the.earth.li   []                                     [
] via keyserver, web or email.  []                                     [
] RSA: 4096/2DA8B985            []                                     [