Skip to contentSkip to conversation

dm: alternate solution to reloading with failed paths

PATCH In review Benjamin Marzinski 2 participants 12 messages Aug 13, 2014 – Sep 18, 2014 Device mapper

lore
Share & tools
lore

Benjamin Marzinski

author maintainer
#0 lore raw

cover dm: alternate solution to reloading with failed paths

This is different solution to the problem of multipath not being able to reload devices with failed paths than the one that Hannes first proposed. It adds a list of dm_devs to the mapped_device structure. All of the table lists point to devices on this list. The list keeps track of how many tables are accessing these devices, and locks them to deal with tables being created and destroyed at the same time. This avoids the issue of having tables that list devices for which the kernel is not holding any reference, and which can disappear and come back as a completely different device if userspace drops its reference. This also allows these devices to be reinstated without another table reload.

The other patch deals with issues I discovered while tracking down why multipath devices with no valid paths would hang on table reload. It turns out that multipath_map can no longer be called after a table reload until queue_io is cleared, but multipath map was previously the function that usually cleared queue_io.

Benjamin Marzinski (2):
dm-multipath: cleanup IO queueing after table load
device-mapper: allow tables to share dm_devs

 drivers/md/dm-ioctl.c |   2 +-
 drivers/md/dm-mpath.c |   6 ++-
 drivers/md/dm-table.c | 101 ++++++++++++++---------------------------
 drivers/md/dm.c       | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/md/dm.h       |   5 +-
 5 files changed, 165 insertions(+), 72 deletions(-)
Signature · 1 line
1.8.3.1
author maintainer
#1 lore raw

PATCH 1/2 dm-multipath: cleanup IO queueing after table load

Commit e809917735ebf1b9a56c24e877ce0d320baee2ec changed the code in multipath_busy that checked if a multipath device was initializing a path. It returned busy if queue_io was set on the device. However queue_io is set when the table is loaded, and as long as the device was busy, multipath_map wouldn't run to clear it upon receiving requsts. multipath_ioctl and reinstate_path were still able to clear queue_io. But if no paths were reinstated and the multipath device received no ioctls after the table reload, all requests to it would simply queue.

This patch switches the multipath_busy code to check for a pending or in progress path initialization, without using queue_io. This means incoming requests will allow multipath_map to initialize paths and clear queue_io.

This patch also changes __choose_pgpath to make it clear queue_io if there are no valid paths, since there are obviously no paths that can be initialized.

---
 drivers/md/dm-mpath.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/md/dm-mpath.c +4 -2
index 3f6fd9d..a0b83cb 100644
@@ -317,8 +317,10 @@ static void __choose_pgpath(struct multipath *m, size_t nr_bytes)
 	struct priority_group *pg;
 	unsigned bypassed = 1;
 
-	if (!m->nr_valid_paths)
+	if (!m->nr_valid_paths) {
+		m->queue_io = 0;
 		goto failed;
+	}
 
 	/* Were we instructed to switch PG? */
 	if (m->next_pg) {
@@ -1612,7 +1614,7 @@ static int multipath_busy(struct dm_target *ti)
 	spin_lock_irqsave(&m->lock, flags);
 
 	/* pg_init in progress, requeue until done */
-	if (!pg_ready(m)) {
+	if (m->pg_init_required || m->pg_init_in_progress) {
 		busy = 1;
 		goto out;
 	}
Signature · 1 line
1.8.3.1
#2 lore raw

Re: [PATCH 1/2] dm-multipath: cleanup IO queueing after table load

Quoted 9 lines · Benjamin Marzinski

Commit e809917735ebf1b9a56c24e877ce0d320baee2ec changed the code in multipath_busy that checked if a multipath device was initializing a path. It returned busy if queue_io was set on the device. However queue_io is set when the table is loaded, and as long as the device was busy, multipath_map wouldn't run to clear it upon receiving requsts. multipath_ioctl and reinstate_path were still able to clear queue_io. But if no paths were reinstated and the multipath device received no ioctls after the table reload, all requests to it would simply queue.

Quoted 3 lines

This patch switches the multipath_busy code to check for a pending or in progress path initialization, without using queue_io. This means incoming requests will allow multipath_map to initialize paths and clear queue_io.

This was already addressed with commit 7a7a3b45fe ("dm mpath: fix IO hang due to logic bug in multipath_busy").

Quoted 3 lines

This patch also changes __choose_pgpath to make it clear queue_io if there are no valid paths, since there are obviously no paths that can be initialized.

Can you elaborate on why this is needed? If it still is then it needs to be rebased.

author maintainer
#3 lore raw

Re: [PATCH 1/2] dm-multipath: cleanup IO queueing after table load

Quoted 17 lines · Mike Snitzer
Quoted 9 lines · Benjamin Marzinski

Commit e809917735ebf1b9a56c24e877ce0d320baee2ec changed the code in multipath_busy that checked if a multipath device was initializing a path. It returned busy if queue_io was set on the device. However queue_io is set when the table is loaded, and as long as the device was busy, multipath_map wouldn't run to clear it upon receiving requsts. multipath_ioctl and reinstate_path were still able to clear queue_io. But if no paths were reinstated and the multipath device received no ioctls after the table reload, all requests to it would simply queue.

Quoted 3 lines

This patch switches the multipath_busy code to check for a pending or in progress path initialization, without using queue_io. This means incoming requests will allow multipath_map to initialize paths and clear queue_io.

This was already addressed with commit 7a7a3b45fe ("dm mpath: fix IO hang due to logic bug in multipath_busy").

Quoted 6 lines
Quoted 3 lines

This patch also changes __choose_pgpath to make it clear queue_io if there are no valid paths, since there are obviously no paths that can be initialized.

Can you elaborate on why this is needed? If it still is then it needs to be rebased.

For a while now, it's been possible to create a multipath table with no devices. With this patch set, it will be even easier to reload a multipath device with no valid paths (since multipath now supports reloading failed paths). In these scenarios, we still need to disable queue_io, so that IOs to the device will not just back up.

-Ben

Quoted 5 lines

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

author maintainer
#4 lore raw

PATCH 2/2 device-mapper: allow tables to share dm_devs

In reply to Benjamin Marzinski

When a new table is created, in needs to open all of the devices that it will use. For multipathi, this can cause problems because it may need to reload a table while some devices are unusable. To get around this, device mapper now stores the dm_devs list for all table devices in the mapped-device object itself, and the tables' device lists just point to the main list.

The device list code in dm.c isn't a straight copy/paste form the code in dm-table.c, but it's very close. One subtle difference is that find_devices for the mapped-device list will only match devices with the same name and mode. This is because we don't want to upgrade a device's mode because of a new inactive table.

Access to the main list requires a mutex, so that tables can be created and destroyed concurrently.

---
 drivers/md/dm-ioctl.c |   2 +-
 drivers/md/dm-table.c | 101 ++++++++++++++---------------------------
 drivers/md/dm.c       | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/md/dm.h       |   5 +-
 4 files changed, 161 insertions(+), 70 deletions(-)
drivers/md/dm-ioctl.c +1 -1
index 5152142..0be9381 100644
@@ -1418,7 +1418,7 @@ static void retrieve_deps(struct dm_table *table,
 	deps->count = count;
 	count = 0;
 	list_for_each_entry (dd, dm_table_get_devices(table), list)
-		deps->dev[count++] = huge_encode_dev(dd->dm_dev.bdev->bd_dev);
+		deps->dev[count++] = huge_encode_dev(dd->dm_dev->bdev->bd_dev);
 
 	param->data_size = param->data_start + needed;
 }
drivers/md/dm-table.c +33 -68

193 lines not on this page · raw message

drivers/md/dm.c +123 -0

159 lines not on this page · raw message

drivers/md/dm.h +4 -1
index ed76126..e6aa368 100644
@@ -44,7 +44,7 @@
 struct dm_dev_internal {
 	struct list_head list;
 	atomic_t count;
-	struct dm_dev dm_dev;
+	struct dm_dev *dm_dev;
 };
 
 struct dm_table;
@@ -189,6 +189,9 @@ int dm_cancel_deferred_remove(struct mapped_device *md);
 int dm_request_based(struct mapped_device *md);
 sector_t dm_get_size(struct mapped_device *md);
 struct request_queue *dm_get_md_queue(struct mapped_device *md);
+int dm_get_table_device(struct mapped_device *md, dev_t dev, fmode_t mode,
+			struct dm_dev **result);
+void dm_put_table_device(struct mapped_device *md, struct dm_dev *d);
 struct dm_stats *dm_get_stats(struct mapped_device *md);
 
 int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,
Signature · 1 line
1.8.3.1
#5 lore raw
In reply to Benjamin Marzinski
Quoted 11 lines · Benjamin Marzinski

This is different solution to the problem of multipath not being able to reload devices with failed paths than the one that Hannes first proposed. It adds a list of dm_devs to the mapped_device structure. All of the table lists point to devices on this list. The list keeps track of how many tables are accessing these devices, and locks them to deal with tables being created and destroyed at the same time. This avoids the issue of having tables that list devices for which the kernel is not holding any reference, and which can disappear and come back as a completely different device if userspace drops its reference. This also allows these devices to be reinstated without another table reload.

Quoted 5 lines

The other patch deals with issues I discovered while tracking down why multipath devices with no valid paths would hang on table reload. It turns out that multipath_map can no longer be called after a table reload until queue_io is cleared, but multipath map was previously the function that usually cleared queue_io.

I've done a first pass review of the code changes and overlayed a cleanup patch in this branch:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=devel

I'll do another pass of review tomorrow and perform a full regression test as well as a targetted test of loading various multipath tables with no devices.

If all looks good I'll get it staged for 3.18.

author maintainer
#6 lore raw
Quoted 24 lines · Mike Snitzer
Quoted 11 lines · Benjamin Marzinski

This is different solution to the problem of multipath not being able to reload devices with failed paths than the one that Hannes first proposed. It adds a list of dm_devs to the mapped_device structure. All of the table lists point to devices on this list. The list keeps track of how many tables are accessing these devices, and locks them to deal with tables being created and destroyed at the same time. This avoids the issue of having tables that list devices for which the kernel is not holding any reference, and which can disappear and come back as a completely different device if userspace drops its reference. This also allows these devices to be reinstated without another table reload.

Quoted 5 lines

The other patch deals with issues I discovered while tracking down why multipath devices with no valid paths would hang on table reload. It turns out that multipath_map can no longer be called after a table reload until queue_io is cleared, but multipath map was previously the function that usually cleared queue_io.

I've done a first pass review of the code changes and overlayed a cleanup patch in this branch:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=devel

Quoted 5 lines

I'll do another pass of review tomorrow and perform a full regression test as well as a targetted test of loading various multipath tables with no devices.

If all looks good I'll get it staged for 3.18.

You changes look fine by me.

-Ben

#7 lore raw

v2 cover dm: alternate solution to reloading with failed paths

In reply to Benjamin Marzinski

Hi Hannes,

Here is v2 of Ben's approach to fixing the inability to reload a multipath table that includes failed paths. I've staged it in linux-dm.git's 'for-next' branch.

But I wanted to send this v2 out to verify this approach works for you. This approach is focused on allowing failed paths to be pushed down in a multipath table.

It does _not_ allow non-existent devices to be listed in an mpath table like your patchset did. So the aggressive dev_loss_tmo case isn't covered... but Ben said that case is already handled by multipathd getting a change event and pushing the table down without the removed SCSI device(s).

Thoughts?
Mike

Benjamin Marzinski (2):
dm mpath: stop queueing IO when no valid paths exist
dm: allow active and inactive tables to share dm_devs

 drivers/md/dm-ioctl.c         |   2 +-
 drivers/md/dm-mpath.c         |   4 +-
 drivers/md/dm-table.c         | 102 ++++++++++++----------------------
 drivers/md/dm.c               | 126 ++++++++++++++++++++++++++++++++++++++++++
 drivers/md/dm.h               |   5 +-
 include/uapi/linux/dm-ioctl.h |   4 +-
 6 files changed, 170 insertions(+), 73 deletions(-)
Signature · 1 line
1.9.3
#8 lore raw

PATCH v2 1/2 dm mpath: stop queueing IO when no valid paths exist

From: Benjamin Marzinski <[email protected]>

'queue_io' is set so that IO is queued while paths are being initialized. Clear queue_io in __choose_pgpath if there are no valid paths, since there are obviously no paths that can be initialized. Otherwise IOs to the device will back up.

---
 drivers/md/dm-mpath.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/md/dm-mpath.c +3 -1
index 833d7e7..7b6b0f0 100644
@@ -317,8 +317,10 @@ static void __choose_pgpath(struct multipath *m, size_t nr_bytes)
 	struct priority_group *pg;
 	unsigned bypassed = 1;
 
-	if (!m->nr_valid_paths)
+	if (!m->nr_valid_paths) {
+		m->queue_io = 0;
 		goto failed;
+	}
 
 	/* Were we instructed to switch PG? */
 	if (m->next_pg) {
Signature · 1 line
1.9.3
#9 lore raw

PATCH v2 2/2 dm: allow active and inactive tables to share dm_devs

In reply to Mike Snitzer

From: Benjamin Marzinski <[email protected]>

When a new table is created, in needs to open all of the devices that it will use. For multipath, this can cause problems because it may need to reload a table while some devices are unusable (failed). To get around this, device mapper now stores the dm_devs list for all table devices in the mapped_device structure, and the tables' device lists just point to this main table_devices list.

The device list code in dm.c isn't a straight copy/paste from the code in dm-table.c, but it's very close (aside from some variable renames). One subtle difference is that find_table_device for the tables_devices list will only match devices with the same name and mode. This is because we don't want to upgrade a device's mode in the active table when an inactive table is loaded.

Access to the mapped_device structure's tables_devices list requires a mutex (tables_devices_lock), so that tables cannot be created and destroyed concurrently.

---
 drivers/md/dm-ioctl.c         |   2 +-
 drivers/md/dm-table.c         | 102 ++++++++++++----------------------
 drivers/md/dm.c               | 126 ++++++++++++++++++++++++++++++++++++++++++
 drivers/md/dm.h               |   5 +-
 include/uapi/linux/dm-ioctl.h |   4 +-
 5 files changed, 167 insertions(+), 72 deletions(-)
drivers/md/dm-ioctl.c +1 -1
index 5152142..0be9381 100644
@@ -1418,7 +1418,7 @@ static void retrieve_deps(struct dm_table *table,
 	deps->count = count;
 	count = 0;
 	list_for_each_entry (dd, dm_table_get_devices(table), list)
-		deps->dev[count++] = huge_encode_dev(dd->dm_dev.bdev->bd_dev);
+		deps->dev[count++] = huge_encode_dev(dd->dm_dev->bdev->bd_dev);
 
 	param->data_size = param->data_start + needed;
 }
drivers/md/dm-table.c +34 -68

194 lines not on this page · raw message

drivers/md/dm.c +126 -0

162 lines not on this page · raw message

drivers/md/dm.h +4 -1
index e81d215..988c7fb 100644
@@ -44,7 +44,7 @@
 struct dm_dev_internal {
 	struct list_head list;
 	atomic_t count;
-	struct dm_dev dm_dev;
+	struct dm_dev *dm_dev;
 };
 
 struct dm_table;
@@ -188,6 +188,9 @@ int dm_cancel_deferred_remove(struct mapped_device *md);
 int dm_request_based(struct mapped_device *md);
 sector_t dm_get_size(struct mapped_device *md);
 struct request_queue *dm_get_md_queue(struct mapped_device *md);
+int dm_get_table_device(struct mapped_device *md, dev_t dev, fmode_t mode,
+			struct dm_dev **result);
+void dm_put_table_device(struct mapped_device *md, struct dm_dev *d);
 struct dm_stats *dm_get_stats(struct mapped_device *md);
 
 int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,
include/uapi/linux/dm-ioctl.h +2 -2
index c8a4302..06614f6 100644
@@ -267,9 +267,9 @@ enum {
 #define DM_DEV_SET_GEOMETRY	_IOWR(DM_IOCTL, DM_DEV_SET_GEOMETRY_CMD, struct dm_ioctl)
 
 #define DM_VERSION_MAJOR	4
-#define DM_VERSION_MINOR	27
+#define DM_VERSION_MINOR	28
 #define DM_VERSION_PATCHLEVEL	0
-#define DM_VERSION_EXTRA	"-ioctl (2013-10-30)"
+#define DM_VERSION_EXTRA	"-ioctl (2014-9-17)"
 
 /* Status bits */
 #define DM_READONLY_FLAG	(1 << 0) /* In/Out */
Signature · 1 line
1.9.3
#10 lore raw

Re: [PATCH v2 2/2] dm: allow active and inactive tables to share dm_devs

I've revised the header for this patch:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=d379c2886c48b32d5f4cf70a27ba2409e127b6f7

dm: allow active and inactive tables to share dm_devs

Until this change, when loading a new DM table, DM core would re-open all of the devices in the DM table. Now, DM core will avoid redundant device opens (and closes when destroying the old table) if the old table already has a device open using the same mode. This is achieved by managing reference counts on the table_devices that DM core now stores in the mapped_device structure (rather than in the dm_table structure). So a mapped_device's active and inactive dm_tables' dm_dev lists now just point to the dm_devs stored in the mapped_device's table_devices list.

This improvement in DM core's device reference counting has the side-effect of fixing a long-standing limitation of the multipath
target: a DM multipath table couldn't include any paths that were unusable (failed). For example: if all paths have failed and you add a new, working, path to the table; you can't use it since the table load would fail due to it still containing failed paths. Now a re-load of a multipath table can include failed devices and when those devices become active again they can be used instantly.

The device list code in dm.c isn't a straight copy/paste from the code in dm-table.c, but it's very close (aside from some variable renames). One subtle difference is that find_table_device for the tables_devices list will only match devices with the same name and mode. This is because we don't want to upgrade a device's mode in the active table when an inactive table is loaded. Access to the mapped_device structure's tables_devices list requires a mutex (tables_devices_lock), so that tables cannot be created and destroyed concurrently.

#11 lore raw
In reply to Mike Snitzer
Quoted 10 lines · Mike Snitzer

Hi Hannes,

Here is v2 of Ben's approach to fixing the inability to reload a multipath table that includes failed paths. I've staged it in linux-dm.git's 'for-next' branch.

But I wanted to send this v2 out to verify this approach works for you. This approach is focused on allowing failed paths to be pushed down in a multipath table.

Quoted 5 lines

It does _not_ allow non-existent devices to be listed in an mpath table like your patchset did. So the aggressive dev_loss_tmo case isn't covered... but Ben said that case is already handled by multipathd getting a change event and pushing the table down without the removed SCSI device(s).

It also doesn't allow the initial multipath table load to include failed devices...