Skip to contentSkip to conversation

dm-mpath: fix for race condition between multipath_dtr and pg_init_done.

PATCH RESEND In review Merla, ShivaKrishna 4 participants 8 messages Oct 30, 2013 – Nov 5, 2013 Device mapper

lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 dm-mpath: fix for race condition between multipath_dtr and pg_init_done.

Whenever multipath_dtr is happening, we should prevent queueing any further path activation work. There was a kernel panic where after pg_init_done() decrements pg_init_in_progress to 0, wait_for_pg_init_completion call assumes there are no more pending path management commands. But if pg_init_required is set by pg_init_done call due to retriable mode_select errors , then process_queued_ios() will again queue the path activation work. If free_multipath call has been completed by the time activate_path work is called, kernel panic was seen on accessing multipath members.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
RIP: 0010:[<ffffffffa003db1b>]  [<ffffffffa003db1b>] activate_path+0x1b/0x30 [dm_multipath]
[<ffffffff81090ac0>] worker_thread+0x170/0x2a0
[<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

---

drivers/md/dm-mpath.c +9 -2
@@ -73,6 +73,7 @@ struct multipath {
 
 	wait_queue_head_t pg_init_wait;	/* Wait for pg_init completion */
 
+	unsigned dtr_in_progress;	/* multipath destroy in progress */
 	unsigned pg_init_required;	/* pg_init needs calling? */
 	unsigned pg_init_in_progress;	/* Only one pg_init allowed at once */
 	unsigned pg_init_delay_retry;	/* Delay pg_init retry? */
@@ -498,7 +499,8 @@ static void process_queued_ios(struct wo
 	    (!pgpath && !m->queue_if_no_path))
 		must_queue = 0;
 
-	if (m->pg_init_required && !m->pg_init_in_progress && pgpath)
+	if (m->pg_init_required && !m->pg_init_in_progress && pgpath &&
+	    !m->dtr_in_progress)
 		__pg_init_all_paths(m);
 
 	spin_unlock_irqrestore(&m->lock, flags);
@@ -951,6 +953,11 @@ static void flush_multipath_work(struct 
 static void multipath_dtr(struct dm_target *ti)
 {
 	struct multipath *m = ti->private;
+	unsigned long flags;
+
+	spin_lock_irqsave(&m->lock, flags);
+	m->dtr_in_progress = 1;
+	spin_unlock_irqrestore(&m->lock, flags);
 
 	flush_multipath_work(m);
 	free_multipath(m);
@@ -1164,7 +1171,7 @@ static int pg_init_limit_reached(struct 
 
 	spin_lock_irqsave(&m->lock, flags);
 
-	if (m->pg_init_count <= m->pg_init_retries)
+	if (m->pg_init_count <= m->pg_init_retries && !m->dtr_in_progress)
 		m->pg_init_required = 1;
 	else
 		limit_reached = 1;

--

#1 lore raw
Quoted 40 lines · ShivaKrishna

Whenever multipath_dtr is happening, we should prevent queueing any further path activation work. There was a kernel panic where after pg_init_done() decrements pg_init_in_progress to 0, wait_for_pg_init_completion call assumes there are no more pending path management commands. But if pg_init_required is set by pg_init_done call due to retriable mode_select errors , then process_queued_ios() will again queue the path activation work. If free_multipath call has been completed by the time activate_path work is called, kernel panic was seen on accessing multipath members.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
RIP: 0010:[<ffffffffa003db1b>]  [<ffffffffa003db1b>] activate_path+0x1b/0x30 [dm_multipath]
[<ffffffff81090ac0>] worker_thread+0x170/0x2a0
[<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

---

drivers/md/dm-mpath.c +3 -1
@@ -73,6 +73,7 @@ struct multipath {
 
 	wait_queue_head_t pg_init_wait;	/* Wait for pg_init completion */
 
+	unsigned dtr_in_progress;	/* multipath destroy in progress */
 	unsigned pg_init_required;	/* pg_init needs calling? */
 	unsigned pg_init_in_progress;	/* Only one pg_init allowed at once */
 	unsigned pg_init_delay_retry;	/* Delay pg_init retry? */
@@ -498,7 +499,8 @@ static void process_queued_ios(struct wo
 	    (!pgpath && !m->queue_if_no_path))
 		must_queue = 0;
 
-	if (m->pg_init_required && !m->pg_init_in_progress && pgpath)
+	if (m->pg_init_required && !m->pg_init_in_progress && pgpath &&
+	    !m->dtr_in_progress)
 		__pg_init_all_paths(m);
 
 	spin_unlock_irqrestore(&m->lock, flags);

drivers/md/dm-mpath.c static void flush_multipath_work(struct · line 960

Quoted 8 lines
drivers/md/dm-mpath.c +5 -0
@@ -951,6 +953,11 @@ static void flush_multipath_work(struct 
 static void multipath_dtr(struct dm_target *ti)
 {
 	struct multipath *m = ti->private;
+	unsigned long flags;
+
+	spin_lock_irqsave(&m->lock, flags);
+	m->dtr_in_progress = 1;
+	spin_unlock_irqrestore(&m->lock, flags);

Hi,

how about moving this to flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work() {
<disable pg_init retry>
...
<enable pg_init retry>
}

Then it not only fixes the crash you hit, it also fixes the hidden bug that pg_init continues retrying while the device is suspended.

Signature · 1 line
Jun'ichi Nomura, NEC Corporation
#2 lore raw
Quoted 82 lines

-----Original Message-----
From: Junichi Nomura [mailto:[email protected]]
Sent: Thursday, October 31, 2013 4:04 AM
To: device-mapper development; Merla, ShivaKrishna
Cc: [email protected]; [email protected]
Subject: Re: [dm-devel] [PATCH][RESEND]dm-mpath: fix for race condition between multipath_dtr and pg_init_done.

Quoted 1 line · ShivaKrishna

Whenever multipath_dtr is happening, we should prevent queueing any

further path

Quoted 1 line

activation work. There was a kernel panic where after pg_init_done()

decrements

Quoted 1 line

pg_init_in_progress to 0, wait_for_pg_init_completion call assumes there

are no

Quoted 1 line

more pending path management commands. But if pg_init_required is set

by

Quoted 1 line

pg_init_done call due to retriable mode_select errors , then

process_queued_ios()

Quoted 2 lines

will again queue the path activation work. If free_multipath call has been completed by the time activate_path work is called, kernel panic was seen

on

Quoted 3 lines

accessing multipath members.

BUG: unable to handle kernel NULL pointer dereference at

0000000000000090

Quoted 1 line
RIP: 0010:[<ffffffffa003db1b>]  [<ffffffffa003db1b>]
activate_path+0x1b/0x30 [dm_multipath]
> [<ffffffff81090ac0>] worker_thread+0x170/0x2a0
> [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40
Quoted 3 lines
Reviewed-by Krishnasamy

Somasundaram<[email protected]>

drivers/md/dm-mpath.c struct multipath { · line 74

Quoted 8 lines

---

drivers/md/dm-mpath.c +0 -0
@@ -73,6 +73,7 @@ struct multipath {
 
 	wait_queue_head_t pg_init_wait;	/* Wait for pg_init

completion */

drivers/md/dm-mpath.c struct multipath {

Quoted 4 lines
drivers/md/dm-mpath.c +1 -0
+	unsigned dtr_in_progress;	/* multipath destroy in progress */
 	unsigned pg_init_required;	/* pg_init needs calling? */
 	unsigned pg_init_in_progress;	/* Only one pg_init allowed at once

*/

Quoted 12 lines
drivers/md/dm-mpath.c +2 -1
 	unsigned pg_init_delay_retry;	/* Delay pg_init retry? */
@@ -498,7 +499,8 @@ static void process_queued_ios(struct wo
 	    (!pgpath && !m->queue_if_no_path))
 		must_queue = 0;
 
-	if (m->pg_init_required && !m->pg_init_in_progress && pgpath)
+	if (m->pg_init_required && !m->pg_init_in_progress && pgpath &&
+	    !m->dtr_in_progress)
 		__pg_init_all_paths(m);
 
 	spin_unlock_irqrestore(&m->lock, flags);

drivers/md/dm-mpath.c static void flush_multipath_work(struct · line 960

Quoted 8 lines
drivers/md/dm-mpath.c +5 -0
@@ -951,6 +953,11 @@ static void flush_multipath_work(struct
 static void multipath_dtr(struct dm_target *ti)
 {
 	struct multipath *m = ti->private;
+	unsigned long flags;
+
+	spin_lock_irqsave(&m->lock, flags);
+	m->dtr_in_progress = 1;
+	spin_unlock_irqrestore(&m->lock, flags);

Hi,

how about moving this to flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work() {
<disable pg_init retry>
...
<enable pg_init retry>
}

Quoted 2 lines

Then it not only fixes the crash you hit, it also fixes the hidden bug that pg_init continues retrying while the device is suspended.

We looked into adding checks for pg_init_required as well in multipath_wait_for_pg_init_completion(), but none of these will prevent pg_init_limit_reached() i.e from pg_init_done() from setting pg_init_required. From code I see we end up waiting indefinitely for its completion. The reason for adding check in process_queued_ios() is __choose_pgpath() might again trigger pg_init_required and end up calling __pg_init_all_paths() which will unset pg_init_required and sets pg_init_in_progress within a lock. Also I believe the patch given above will prevent further path activations and no work will be queued again with dtr_in_progress. With Hannes patch IO's will not be queued in when pg_init_in_progress so we are clean there as well. Let me know your thoughts.

#3 lore raw

PATCH v2 1/1 dm mpath: fix race condition between multipath_dtr and pg_init_done

In reply to Junichi Nomura
Quoted 12 lines · Junichi Nomura

Hi,

how about moving this to flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work() {
<disable pg_init retry>
...
<enable pg_init retry>
}

Quoted 2 lines

Then it not only fixes the crash you hit, it also fixes the hidden bug that pg_init continues retrying while the device is suspended.

I ran with your suggestion. Please see below.

To be clear, pg_init isn't disabled while mpath device is suspended (meaning m->pg_init_disabled isn't set until the device is resumed). But flush_multipath_work() will no longer start pg_init during suspend -- which could otherwise occur while the mpath device is suspended. So in practice it accomplishes the stated goal.

Thanks for the suggestion Junichi. Are you OK with this? If so please provide your Ack.

Shiva, can you please verify that this patch resolves the race, should accomplish the same: just pushes the disabling of pg_init inside flush_multipath_work().

From: Shiva Krishna Merla <[email protected]>
Subject: dm mpath: fix race condition between multipath_dtr and pg_init_done
Date: Wed, 30 Oct 2013 03:26:38 +0000

Whenever multipath_dtr() is happening we must prevent queueing any further path activation work. Implement this by adding a new 'pg_init_disabled' flag to the multipath structure that denotes future path activation work should be skipped if it is set. By disabling pg_init and then re-enabling in flush_multipath_work() we also avoid the potential for pg_init to be initiated while suspending an mpath device.

Without this patch a race condition exists that may result in a kernel panic:

1) If after pg_init_done() decrements pg_init_in_progress to 0, a call to wait_for_pg_init_completion() assumes there are no more pending path management commands.
2) If pg_init_required is set by pg_init_done(), due to retryable mode_select errors, then process_queued_ios() will again queue the path activation work.
3) If free_multipath() completes before activate_path() work is called a NULL pointer dereference like the following can be seen when accessing members of the recently destructed multipath:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
RIP: 0010:[<ffffffffa003db1b>]  [<ffffffffa003db1b>] activate_path+0x1b/0x30 [dm_multipath]
[<ffffffff81090ac0>] worker_thread+0x170/0x2a0
[<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40
[switched to disabling pg_init in flush_multipath_work, header edited by Mike Snitzer]
---
 drivers/md/dm-mpath.c |   18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/md/dm-mpath.c +15 -3
Index: linux/drivers/md/dm-mpath.c
===================================================================
@@ -87,6 +87,7 @@ struct multipath {
 	unsigned queue_if_no_path:1;	/* Queue I/O if last path fails? */
 	unsigned saved_queue_if_no_path:1; /* Saved state during suspension */
 	unsigned retain_attached_hw_handler:1; /* If there's already a hw_handler present, don't change it. */
+	unsigned pg_init_disabled:1;	/* pginit is currently not allowed */
 
 	unsigned pg_init_retries;	/* Number of times to retry pg_init */
 	unsigned pg_init_count;		/* Number of times pg_init called */
@@ -497,7 +498,8 @@ static void process_queued_ios(struct wo
 	    (!pgpath && !m->queue_if_no_path))
 		must_queue = 0;
 
-	if (m->pg_init_required && !m->pg_init_in_progress && pgpath)
+	if (m->pg_init_required && !m->pg_init_in_progress && pgpath &&
+	    !m->pg_init_disabled)
 		__pg_init_all_paths(m);
 
 	spin_unlock_irqrestore(&m->lock, flags);
@@ -942,10 +944,20 @@ static void multipath_wait_for_pg_init_c
 
 static void flush_multipath_work(struct multipath *m)
 {
+	unsigned long flags;
+
+	spin_lock_irqsave(&m->lock, flags);
+	m->pg_init_disabled = 1;
+	spin_unlock_irqrestore(&m->lock, flags);
+
 	flush_workqueue(kmpath_handlerd);
 	multipath_wait_for_pg_init_completion(m);
 	flush_workqueue(kmultipathd);
 	flush_work(&m->trigger_event);
+
+	spin_lock_irqsave(&m->lock, flags);
+	m->pg_init_disabled = 0;
+	spin_unlock_irqrestore(&m->lock, flags);
 }
 
 static void multipath_dtr(struct dm_target *ti)
@@ -1164,7 +1176,7 @@ static int pg_init_limit_reached(struct
 
 	spin_lock_irqsave(&m->lock, flags);
 
-	if (m->pg_init_count <= m->pg_init_retries)
+	if (m->pg_init_count <= m->pg_init_retries && !m->pg_init_disabled)
 		m->pg_init_required = 1;
 	else
 		limit_reached = 1;
@@ -1714,7 +1726,7 @@ out:
  *---------------------------------------------------------------*/
 static struct target_type multipath_target = {
 	.name = "multipath",
-	.version = {1, 5, 1},
+	.version = {1, 6, 0},
 	.module = THIS_MODULE,
 	.ctr = multipath_ctr,
 	.dtr = multipath_dtr,
#4 lore raw

Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done

Quoted 35 lines

-----Original Message-----
From: Mike Snitzer [mailto:[email protected]]
Sent: Thursday, October 31, 2013 9:48 AM
To: Junichi Nomura; Merla, ShivaKrishna
Cc: device-mapper development; [email protected]
Subject: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done

Quoted 12 lines · Junichi Nomura

Hi,

how about moving this to flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work() {
<disable pg_init retry>
...
<enable pg_init retry>
}

Quoted 2 lines

Then it not only fixes the crash you hit, it also fixes the hidden bug that pg_init continues retrying while the device is suspended.

I ran with your suggestion. Please see below.

To be clear, pg_init isn't disabled while mpath device is suspended (meaning m->pg_init_disabled isn't set until the device is resumed). But flush_multipath_work() will no longer start pg_init during suspend -- which could otherwise occur while the mpath device is suspended. So in practice it accomplishes the stated goal.

Quoted 6 lines

Thanks for the suggestion Junichi. Are you OK with this? If so please provide your Ack.

Shiva, can you please verify that this patch resolves the race, should accomplish the same: just pushes the disabling of pg_init inside flush_multipath_work().

Yes Mike, This looks good. Thanks Junichi for the suggestion.

Acked-by on v2 1/1
#5 lore raw

Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done

In reply to Mike Snitzer
Quoted 16 lines · Mike Snitzer
Quoted 11 lines · Junichi Nomura

how about moving this to flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work() {
<disable pg_init retry>
...
<enable pg_init retry>
}

Then it not only fixes the crash you hit, it also fixes the hidden bug that pg_init continues retrying while the device is suspended.

I ran with your suggestion. Please see below.

Quoted 8 lines

To be clear, pg_init isn't disabled while mpath device is suspended (meaning m->pg_init_disabled isn't set until the device is resumed). But flush_multipath_work() will no longer start pg_init during suspend -- which could otherwise occur while the mpath device is suspended. So in practice it accomplishes the stated goal.

Thanks for the suggestion Junichi. Are you OK with this? If so please provide your Ack.

Yes, that's perfect. Thank you.

And thanks Shiva Krishna for finding/fixing this.

Quoted 4 lines

Shiva, can you please verify that this patch resolves the race, should accomplish the same: just pushes the disabling of pg_init inside flush_multipath_work().

Signature · 1 line
Jun'ichi Nomura, NEC Corporation
#6 lore raw

Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done

In reply to Mike Snitzer

drivers/md/dm-mpath.c static void process_queued_ios(struct wo · line 502

Quoted 7 lines · Mike Snitzer
drivers/md/dm-mpath.c +3 -0
 
+	unsigned pg_init_disabled:1;	/* pginit is currently not allowed */
 
+	if (m->pg_init_required && !m->pg_init_in_progress && pgpath &&
+	    !m->pg_init_disabled)

So we're accepting that pg_init might be both "disabled" and "required" simultaneously (otherwise we wouldn't need this test)?

drivers/md/dm-mpath.c static void multipath_wait_for_pg_init_c · line 953

Quoted 11 lines
drivers/md/dm-mpath.c +6 -0
@@ -942,10 +944,20 @@ static void multipath_wait_for_pg_init_c
 static void flush_multipath_work(struct multipath *m)
 {
+	unsigned long flags;
+
+	spin_lock_irqsave(&m->lock, flags);
+	m->pg_init_disabled = 1;
+	spin_unlock_irqrestore(&m->lock, flags);
+
 	flush_workqueue(kmpath_handlerd);
 	multipath_wait_for_pg_init_completion(m);

So this implies pg_init could even still be running while it is disabled?

If the logic is correct, then I find the new variable name quite confusing and think it should be changed, or as a minimum have inline comments explaining the apparent contradictions!

Alasdair

#7 lore raw

Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done

Quoted 11 lines · Alasdair G Kergon

drivers/md/dm-mpath.c static void process_queued_ios(struct wo · line 502

Quoted 7 lines · Mike Snitzer
drivers/md/dm-mpath.c +3 -0
 
+	unsigned pg_init_disabled:1;	/* pginit is currently not allowed */
 
+	if (m->pg_init_required && !m->pg_init_in_progress && pgpath &&
+	    !m->pg_init_disabled)

So we're accepting that pg_init might be both "disabled" and "required" simultaneously (otherwise we wouldn't need this test)?

Yes, needs to be independent of pg_init_required.

Quoted 5 lines
Quoted 5 lines
drivers/md/dm-mpath.c +2 -0
@@ -942,10 +944,20 @@ static void multipath_wait_for_pg_init_c
 static void flush_multipath_work(struct multipath *m)
 {
+	unsigned long flags;
+
Quoted 8 lines

drivers/md/dm-mpath.c static void multipath_wait_for_pg_init_c · line 954

Quoted 6 lines
drivers/md/dm-mpath.c +4 -0
+	spin_lock_irqsave(&m->lock, flags);
+	m->pg_init_disabled = 1;
+	spin_unlock_irqrestore(&m->lock, flags);
+
 	flush_workqueue(kmpath_handlerd);
 	multipath_wait_for_pg_init_completion(m);

So this implies pg_init could even still be running while it is disabled?

pg_init_disabled doesn't allow pg_init to be initiated once set (even if pg_init_required is). Hence pg_init is disabled.

Quoted 3 lines

If the logic is correct, then I find the new variable name quite confusing and think it should be changed, or as a minimum have inline comments explaining the apparent contradictions!

I'm not seeing a contradiction.