dm: alternate solution to reloading with failed paths
Benjamin Marzinski
maintainercover dm: alternate solution to reloading with failed paths
This is different solution to the problem of multipath not being able to reload devices with failed paths than the one that Hannes first proposed. It adds a list of dm_devs to the mapped_device structure. All of the table lists point to devices on this list. The list keeps track of how many tables are accessing these devices, and locks them to deal with tables being created and destroyed at the same time. This avoids the issue of having tables that list devices for which the kernel is not holding any reference, and which can disappear and come back as a completely different device if userspace drops its reference. This also allows these devices to be reinstated without another table reload.
The other patch deals with issues I discovered while tracking down why multipath devices with no valid paths would hang on table reload. It turns out that multipath_map can no longer be called after a table reload until queue_io is cleared, but multipath map was previously the function that usually cleared queue_io.
Benjamin Marzinski (2):
dm-multipath: cleanup IO queueing after table load
device-mapper: allow tables to share dm_devs
drivers/md/dm-ioctl.c | 2 +- drivers/md/dm-mpath.c | 6 ++- drivers/md/dm-table.c | 101 ++++++++++++++--------------------------- drivers/md/dm.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++ drivers/md/dm.h | 5 +- 5 files changed, 165 insertions(+), 72 deletions(-)
Signature · 1 line
1.8.3.1
Benjamin Marzinski
maintainerPATCH 1/2 dm-multipath: cleanup IO queueing after table load
Commit e809917735ebf1b9a56c24e877ce0d320baee2ec changed the code in multipath_busy that checked if a multipath device was initializing a path. It returned busy if queue_io was set on the device. However queue_io is set when the table is loaded, and as long as the device was busy, multipath_map wouldn't run to clear it upon receiving requsts. multipath_ioctl and reinstate_path were still able to clear queue_io. But if no paths were reinstated and the multipath device received no ioctls after the table reload, all requests to it would simply queue.
This patch switches the multipath_busy code to check for a pending or in progress path initialization, without using queue_io. This means incoming requests will allow multipath_map to initialize paths and clear queue_io.
This patch also changes __choose_pgpath to make it clear queue_io if there are no valid paths, since there are obviously no paths that can be initialized.
--- drivers/md/dm-mpath.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/md/dm-mpath.c +4 -2
@@ -317,8 +317,10 @@ static void __choose_pgpath(struct multipath *m, size_t nr_bytes)struct priority_group *pg;unsigned bypassed =1 ; -if (!m->nr_valid_paths) +if (!m->nr_valid_paths) { + m->queue_io =0 ; goto failed; + }/* Were we instructed to switch PG? */ if (m->next_pg) {
@@ -1612,7 +1614,7 @@ static int multipath_busy(struct dm_target *ti)spin_lock_irqsave (&m->lock, flags);/* pg_init in progress, requeue until done */ -if (!pg_ready (m)) { +if (m->pg_init_required || m->pg_init_in_progress) { busy =1 ; goto out; }
Signature · 1 line
1.8.3.1
Re: [PATCH 1/2] dm-multipath: cleanup IO queueing after table load
Quoted 9 lines · Benjamin Marzinski
Commit
e809917735ebf1b9a56c24e877ce0d320baee2ecchanged the code inmultipath_busythat checked if a multipath device was initializing a path. It returned busy ifqueue_iowas set on the device. Howeverqueue_iois set when the table is loaded, and as long as the device was busy,multipath_mapwouldn't run to clear it upon receiving requsts.multipath_ioctlandreinstate_pathwere still able to clearqueue_io. But if no paths were reinstated and the multipath device received no ioctls after the table reload, all requests to it would simply queue.
Quoted 3 lines
This patch switches the
multipath_busycode to check for a pending or in progress path initialization, without usingqueue_io. This means incoming requests will allowmultipath_mapto initialize paths and clearqueue_io.
This was already addressed with commit 7a7a3b45fe ("dm mpath: fix IO hang due to logic bug in multipath_busy").
Quoted 3 lines
This patch also changes
__choose_pgpathto make it clearqueue_ioif there are no valid paths, since there are obviously no paths that can be initialized.
Can you elaborate on why this is needed? If it still is then it needs to be rebased.
Benjamin Marzinski
maintainerRe: [PATCH 1/2] dm-multipath: cleanup IO queueing after table load
Quoted 17 lines · Mike Snitzer
Quoted 9 lines · Benjamin Marzinski
Commit
e809917735ebf1b9a56c24e877ce0d320baee2ecchanged the code inmultipath_busythat checked if a multipath device was initializing a path. It returned busy ifqueue_iowas set on the device. Howeverqueue_iois set when the table is loaded, and as long as the device was busy,multipath_mapwouldn't run to clear it upon receiving requsts.multipath_ioctlandreinstate_pathwere still able to clearqueue_io. But if no paths were reinstated and the multipath device received no ioctls after the table reload, all requests to it would simply queue.Quoted 3 lines
This patch switches the
multipath_busycode to check for a pending or in progress path initialization, without usingqueue_io. This means incoming requests will allowmultipath_mapto initialize paths and clearqueue_io.This was already addressed with commit
7a7a3b45fe("dm mpath: fix IO hang due to logic bug inmultipath_busy").
Quoted 6 lines
Quoted 3 lines
This patch also changes
__choose_pgpathto make it clearqueue_ioif there are no valid paths, since there are obviously no paths that can be initialized.Can you elaborate on why this is needed? If it still is then it needs to be rebased.
For a while now, it's been possible to create a multipath table with no devices. With this patch set, it will be even easier to reload a multipath device with no valid paths (since multipath now supports reloading failed paths). In these scenarios, we still need to disable queue_io, so that IOs to the device will not just back up.
-Ben
Quoted 5 lines
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Benjamin Marzinski
maintainerPATCH 2/2 device-mapper: allow tables to share dm_devs
In reply to Benjamin MarzinskiWhen a new table is created, in needs to open all of the devices that it will use. For multipathi, this can cause problems because it may need to reload a table while some devices are unusable. To get around this, device mapper now stores the dm_devs list for all table devices in the mapped-device object itself, and the tables' device lists just point to the main list.
The device list code in dm.c isn't a straight copy/paste form the code in dm-table.c, but it's very close. One subtle difference is that find_devices for the mapped-device list will only match devices with the same name and mode. This is because we don't want to upgrade a device's mode because of a new inactive table.
Access to the main list requires a mutex, so that tables can be created and destroyed concurrently.
--- drivers/md/dm-ioctl.c | 2 +- drivers/md/dm-table.c | 101 ++++++++++++++--------------------------- drivers/md/dm.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++ drivers/md/dm.h | 5 +- 4 files changed, 161 insertions(+), 70 deletions(-)
drivers/md/dm-ioctl.c +1 -1
@@ -1418,7 +1418,7 @@ static void retrieve_deps(struct dm_table *table, deps->count = count; count =0 ;list_for_each_entry (dd,dm_table_get_devices (table), list) - deps->dev[count++] = huge_encode_dev(dd->dm_dev.bdev->bd_dev); + deps->dev[count++] = huge_encode_dev(dd->dm_dev->bdev->bd_dev); param->data_size = param->data_start + needed; }
drivers/md/dm-table.c +33 -68
193 lines not on this page · raw message
drivers/md/dm.c +123 -0
159 lines not on this page · raw message
drivers/md/dm.h +4 -1
@@ -44,7 +44,7 @@struct dm_dev_internal {struct list_head list;atomic_t count; -struct dm_dev dm_dev; +struct dm_dev *dm_dev; };struct dm_table ;
@@ -189,6 +189,9 @@ int dm_cancel_deferred_remove(struct mapped_device *md);int dm_request_based (struct mapped_device *md);sector_t dm_get_size (struct mapped_device *md);struct request_queue *dm_get_md_queue (struct mapped_device *md); +int dm_get_table_device (struct mapped_device *md,dev_t dev,fmode_t mode, +struct dm_dev **result); +void dm_put_table_device (struct mapped_device *md,struct dm_dev *d);struct dm_stats *dm_get_stats (struct mapped_device *md); int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,
Signature · 1 line
1.8.3.1
Quoted 11 lines · Benjamin Marzinski
This is different solution to the problem of multipath not being able to reload devices with failed paths than the one that Hannes first proposed. It adds a list of
dm_devsto themapped_devicestructure. All of the table lists point to devices on this list. The list keeps track of how many tables are accessing these devices, and locks them to deal with tables being created and destroyed at the same time. This avoids the issue of having tables that list devices for which the kernel is not holding any reference, and which can disappear and come back as a completely different device if userspace drops its reference. This also allows these devices to be reinstated without another table reload.
Quoted 5 lines
The other patch deals with issues I discovered while tracking down why multipath devices with no valid paths would hang on table reload. It turns out that
multipath_mapcan no longer be called after a table reload untilqueue_iois cleared, but multipath map was previously the function that usually clearedqueue_io.
I've done a first pass review of the code changes and overlayed a cleanup patch in this branch:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=devel
I'll do another pass of review tomorrow and perform a full regression test as well as a targetted test of loading various multipath tables with no devices.
If all looks good I'll get it staged for 3.18.
Benjamin Marzinski
maintainerQuoted 24 lines · Mike Snitzer
Quoted 11 lines · Benjamin Marzinski
This is different solution to the problem of multipath not being able to reload devices with failed paths than the one that Hannes first proposed. It adds a list of
dm_devsto themapped_devicestructure. All of the table lists point to devices on this list. The list keeps track of how many tables are accessing these devices, and locks them to deal with tables being created and destroyed at the same time. This avoids the issue of having tables that list devices for which the kernel is not holding any reference, and which can disappear and come back as a completely different device if userspace drops its reference. This also allows these devices to be reinstated without another table reload.Quoted 5 lines
The other patch deals with issues I discovered while tracking down why multipath devices with no valid paths would hang on table reload. It turns out that
multipath_mapcan no longer be called after a table reload untilqueue_iois cleared, but multipath map was previously the function that usually clearedqueue_io.I've done a first pass review of the code changes and overlayed a cleanup patch in this branch:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=devel
Quoted 5 lines
I'll do another pass of review tomorrow and perform a full regression test as well as a targetted test of loading various multipath tables with no devices.
If all looks good I'll get it staged for 3.18.
You changes look fine by me.
-Ben
v2 cover dm: alternate solution to reloading with failed paths
In reply to Benjamin MarzinskiHi Hannes,
Here is v2 of Ben's approach to fixing the inability to reload a multipath table that includes failed paths. I've staged it in linux-dm.git's 'for-next' branch.
But I wanted to send this v2 out to verify this approach works for you. This approach is focused on allowing failed paths to be pushed down in a multipath table.
It does _not_ allow non-existent devices to be listed in an mpath table like your patchset did. So the aggressive dev_loss_tmo case isn't covered... but Ben said that case is already handled by multipathd getting a change event and pushing the table down without the removed SCSI device(s).
Thoughts?
Mike
Benjamin Marzinski (2):
dm mpath: stop queueing IO when no valid paths exist
dm: allow active and inactive tables to share dm_devs
drivers/md/dm-ioctl.c | 2 +- drivers/md/dm-mpath.c | 4 +- drivers/md/dm-table.c | 102 ++++++++++++---------------------- drivers/md/dm.c | 126 ++++++++++++++++++++++++++++++++++++++++++ drivers/md/dm.h | 5 +- include/uapi/linux/dm-ioctl.h | 4 +- 6 files changed, 170 insertions(+), 73 deletions(-)
Signature · 1 line
1.9.3
PATCH v2 1/2 dm mpath: stop queueing IO when no valid paths exist
From: Benjamin Marzinski <[email protected]>
'queue_io' is set so that IO is queued while paths are being initialized. Clear queue_io in __choose_pgpath if there are no valid paths, since there are obviously no paths that can be initialized. Otherwise IOs to the device will back up.
--- drivers/md/dm-mpath.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/md/dm-mpath.c +3 -1
@@ -317,8 +317,10 @@ static void __choose_pgpath(struct multipath *m, size_t nr_bytes)struct priority_group *pg;unsigned bypassed =1 ; -if (!m->nr_valid_paths) +if (!m->nr_valid_paths) { + m->queue_io =0 ; goto failed; + }/* Were we instructed to switch PG? */ if (m->next_pg) {
Signature · 1 line
1.9.3
PATCH v2 2/2 dm: allow active and inactive tables to share dm_devs
In reply to Mike SnitzerFrom: Benjamin Marzinski <[email protected]>
When a new table is created, in needs to open all of the devices that it will use. For multipath, this can cause problems because it may need to reload a table while some devices are unusable (failed). To get around this, device mapper now stores the dm_devs list for all table devices in the mapped_device structure, and the tables' device lists just point to this main table_devices list.
The device list code in dm.c isn't a straight copy/paste from the code in dm-table.c, but it's very close (aside from some variable renames). One subtle difference is that find_table_device for the tables_devices list will only match devices with the same name and mode. This is because we don't want to upgrade a device's mode in the active table when an inactive table is loaded.
Access to the mapped_device structure's tables_devices list requires a mutex (tables_devices_lock), so that tables cannot be created and destroyed concurrently.
--- drivers/md/dm-ioctl.c | 2 +- drivers/md/dm-table.c | 102 ++++++++++++---------------------- drivers/md/dm.c | 126 ++++++++++++++++++++++++++++++++++++++++++ drivers/md/dm.h | 5 +- include/uapi/linux/dm-ioctl.h | 4 +- 5 files changed, 167 insertions(+), 72 deletions(-)
drivers/md/dm-ioctl.c +1 -1
@@ -1418,7 +1418,7 @@ static void retrieve_deps(struct dm_table *table, deps->count = count; count =0 ;list_for_each_entry (dd,dm_table_get_devices (table), list) - deps->dev[count++] = huge_encode_dev(dd->dm_dev.bdev->bd_dev); + deps->dev[count++] = huge_encode_dev(dd->dm_dev->bdev->bd_dev); param->data_size = param->data_start + needed; }
drivers/md/dm-table.c +34 -68
194 lines not on this page · raw message
drivers/md/dm.c +126 -0
162 lines not on this page · raw message
drivers/md/dm.h +4 -1
@@ -44,7 +44,7 @@struct dm_dev_internal {struct list_head list;atomic_t count; -struct dm_dev dm_dev; +struct dm_dev *dm_dev; };struct dm_table ;
@@ -188,6 +188,9 @@ int dm_cancel_deferred_remove(struct mapped_device *md);int dm_request_based (struct mapped_device *md);sector_t dm_get_size (struct mapped_device *md);struct request_queue *dm_get_md_queue (struct mapped_device *md); +int dm_get_table_device (struct mapped_device *md,dev_t dev,fmode_t mode, +struct dm_dev **result); +void dm_put_table_device (struct mapped_device *md,struct dm_dev *d);struct dm_stats *dm_get_stats (struct mapped_device *md); int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,
include/uapi/linux/dm-ioctl.h +2 -2
@@ -267,9 +267,9 @@ enum {#define DM_DEV_SET_GEOMETRY _IOWR(DM_IOCTL, DM_DEV_SET_GEOMETRY_CMD, struct dm_ioctl)#define DM_VERSION_MAJOR 4 -#define DM_VERSION_MINOR 27 +#define DM_VERSION_MINOR 28#define DM_VERSION_PATCHLEVEL 0 -#define DM_VERSION_EXTRA "-ioctl (2013-10-30)" +#define DM_VERSION_EXTRA "-ioctl (2014-9-17)"/* Status bits */ #define DM_READONLY_FLAG (1 << 0)/* In/Out */
Signature · 1 line
1.9.3
Re: [PATCH v2 2/2] dm: allow active and inactive tables to share dm_devs
I've revised the header for this patch:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=d379c2886c48b32d5f4cf70a27ba2409e127b6f7
dm: allow active and inactive tables to share dm_devs
Until this change, when loading a new DM table, DM core would re-open all of the devices in the DM table. Now, DM core will avoid redundant device opens (and closes when destroying the old table) if the old table already has a device open using the same mode. This is achieved by managing reference counts on the table_devices that DM core now stores in the mapped_device structure (rather than in the dm_table structure). So a mapped_device's active and inactive dm_tables' dm_dev lists now just point to the dm_devs stored in the mapped_device's table_devices list.
This improvement in DM core's device reference counting has the side-effect of fixing a long-standing limitation of the multipath
target: a DM multipath table couldn't include any paths that were unusable (failed). For example: if all paths have failed and you add a new, working, path to the table; you can't use it since the table load would fail due to it still containing failed paths. Now a re-load of a multipath table can include failed devices and when those devices become active again they can be used instantly.
The device list code in dm.c isn't a straight copy/paste from the code in dm-table.c, but it's very close (aside from some variable renames). One subtle difference is that find_table_device for the tables_devices list will only match devices with the same name and mode. This is because we don't want to upgrade a device's mode in the active table when an inactive table is loaded. Access to the mapped_device structure's tables_devices list requires a mutex (tables_devices_lock), so that tables cannot be created and destroyed concurrently.
Quoted 10 lines · Mike Snitzer
Hi Hannes,
Here is v2 of Ben's approach to fixing the inability to reload a multipath table that includes failed paths. I've staged it in linux-dm.git's 'for-next' branch.
But I wanted to send this v2 out to verify this approach works for you. This approach is focused on allowing failed paths to be pushed down in a multipath table.
Quoted 5 lines
It does _not_ allow non-existent devices to be listed in an mpath table like your patchset did. So the aggressive
dev_loss_tmocase isn't covered... but Ben said that case is already handled by multipathd getting a change event and pushing the table down without the removed SCSI device(s).
It also doesn't allow the initial multipath table load to include failed devices...