dm-mpath: fix for race condition between multipath_dtr and pg_init_done.
PATCH RESEND 1/1 dm-mpath: fix for race condition between multipath_dtr and pg_init_done.
Whenever multipath_dtr is happening, we should prevent queueing any further path activation work. There was a kernel panic where after pg_init_done() decrements pg_init_in_progress to 0, wait_for_pg_init_completion call assumes there are no more pending path management commands. But if pg_init_required is set by pg_init_done call due to retriable mode_select errors , then process_queued_ios() will again queue the path activation work. If free_multipath call has been completed by the time activate_path work is called, kernel panic was seen on accessing multipath members.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000090 RIP: 0010:[<ffffffffa003db1b>] [<ffffffffa003db1b>] activate_path+0x1b/0x30 [dm_multipath] [<ffffffff81090ac0>] worker_thread+0x170/0x2a0 [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40
---
drivers/md/dm-mpath.c +9 -2
@@ -73,6 +73,7 @@ struct multipath {wait_queue_head_t pg_init_wait;/* Wait for pg_init completion */ +unsigned dtr_in_progress;/* multipath destroy in progress */ unsigned pg_init_required;/* pg_init needs calling? */ unsigned pg_init_in_progress;/* Only one pg_init allowed at once */ unsigned pg_init_delay_retry;/* Delay pg_init retry? */
@@ -498,7 +499,8 @@ static void process_queued_ios(struct wo (!pgpath && !m->queue_if_no_path)) must_queue =0 ; -if (m->pg_init_required && !m->pg_init_in_progress && pgpath) +if (m->pg_init_required && !m->pg_init_in_progress && pgpath && + !m->dtr_in_progress)__pg_init_all_paths (m);spin_unlock_irqrestore (&m->lock, flags);
@@ -951,6 +953,11 @@ static void flush_multipath_work(structstatic void multipath_dtr (struct dm_target *ti) {struct multipath *m = ti->private; +unsigned long flags; + +spin_lock_irqsave (&m->lock, flags); + m->dtr_in_progress =1 ; +spin_unlock_irqrestore (&m->lock, flags);flush_multipath_work (m);free_multipath (m);
@@ -1164,7 +1171,7 @@ static int pg_init_limit_reached(structspin_lock_irqsave (&m->lock, flags); -if (m->pg_init_count <= m->pg_init_retries) +if (m->pg_init_count <= m->pg_init_retries && !m->dtr_in_progress) m->pg_init_required =1 ;else limit_reached =1 ;
--
Quoted 40 lines · ShivaKrishna
Whenever
multipath_dtris happening, we should prevent queueing any further path activation work. There was a kernel panic where afterpg_init_done()decrementspg_init_in_progressto 0,wait_for_pg_init_completioncall assumes there are no more pending path management commands. But ifpg_init_requiredis set bypg_init_donecall due to retriablemode_selecterrors , thenprocess_queued_ios()will again queue the path activation work. Iffree_multipathcall has been completed by the timeactivate_pathwork is called, kernel panic was seen on accessing multipath members.BUG: unable to handle kernel NULL pointer dereference at 0000000000000090 RIP: 0010:[<ffffffffa003db1b>] [<ffffffffa003db1b>] activate_path+0x1b/0x30 [dm_multipath] [<ffffffff81090ac0>] worker_thread+0x170/0x2a0 [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40Signed-off-by Shiva Krishna Merla<[email protected]>Reviewed-by Krishnasamy Somasundaram<[email protected]>Tested-by Speagle Andy<[email protected]>---
drivers/md/dm-mpath.c +3
-1@@ -73,6 +73,7 @@ struct multipath {wait_queue_head_t pg_init_wait;/* Wait for pg_init completion */ +unsigned dtr_in_progress;/* multipath destroy in progress */ unsigned pg_init_required;/* pg_init needs calling? */ unsigned pg_init_in_progress;/* Only one pg_init allowed at once */ unsigned pg_init_delay_retry;/* Delay pg_init retry? */ @@ -498,7 +499,8 @@ static void process_queued_ios(struct wo (!pgpath && !m->queue_if_no_path)) must_queue =0 ; -if (m->pg_init_required && !m->pg_init_in_progress && pgpath) +if (m->pg_init_required && !m->pg_init_in_progress && pgpath && + !m->dtr_in_progress)__pg_init_all_paths (m);spin_unlock_irqrestore (&m->lock, flags);
drivers/md/dm-mpath.c static void flush_multipath_work(struct · line 960
Quoted 8 lines
drivers/md/dm-mpath.c +5
-0@@ -951,6 +953,11 @@ static void flush_multipath_work(structstatic void multipath_dtr (struct dm_target *ti) {struct multipath *m = ti->private; +unsigned long flags; + +spin_lock_irqsave (&m->lock, flags); + m->dtr_in_progress =1 ; +spin_unlock_irqrestore (&m->lock, flags);
Hi,
how about moving this to flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work() {
<disable pg_init retry>
...
<enable pg_init retry>
}
Then it not only fixes the crash you hit, it also fixes the hidden bug that pg_init continues retrying while the device is suspended.
Signature · 1 line
Jun'ichi Nomura, NEC Corporation
Quoted 82 lines
-----Original Message-----
From: Junichi Nomura [mailto:[email protected]]
Sent: Thursday, October 31, 2013 4:04 AM
To: device-mapper development; Merla, ShivaKrishna
Cc: [email protected]; [email protected]
Subject: Re: [dm-devel] [PATCH][RESEND]dm-mpath: fix for race condition betweenmultipath_dtrandpg_init_done.Quoted 1 line · ShivaKrishna
Whenever
multipath_dtris happening, we should prevent queueing anyfurther path
Quoted 1 line
activation work. There was a kernel panic where after
pg_init_done()decrements
Quoted 1 line
pg_init_in_progressto 0,wait_for_pg_init_completioncall assumes thereare no
Quoted 1 line
more pending path management commands. But if
pg_init_requiredis setby
Quoted 1 line
pg_init_donecall due to retriablemode_selecterrors , then
process_queued_ios()Quoted 2 lines
will again queue the path activation work. If
free_multipathcall has been completed by the timeactivate_pathwork is called, kernel panic was seenon
Quoted 3 lines
accessing multipath members.
BUG: unable to handle kernel NULL pointer dereference at0000000000000090
Quoted 1 line
RIP: 0010:[<ffffffffa003db1b>] [<ffffffffa003db1b>]activate_path+0x1b/0x30 [dm_multipath] > [<ffffffff81090ac0>] worker_thread+0x170/0x2a0 > [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40Quoted 3 lines
Signed-off-by Shiva Krishna Merla<[email protected]>Reviewed-by KrishnasamySomasundaram<[email protected]>
drivers/md/dm-mpath.c struct multipath { · line 74
Quoted 8 lines
Tested-by Speagle Andy<[email protected]>---
drivers/md/dm-mpath.c +0
-0@@ -73,6 +73,7 @@ struct multipath {wait_queue_head_t pg_init_wait;/* Wait for pg_init completion */
drivers/md/dm-mpath.c struct multipath {
Quoted 4 lines
drivers/md/dm-mpath.c +1
-0+unsigned dtr_in_progress;/* multipath destroy in progress */ unsigned pg_init_required;/* pg_init needs calling? */ unsigned pg_init_in_progress;/* Only one pg_init allowed at once */
Quoted 12 lines
drivers/md/dm-mpath.c +2
-1unsigned pg_init_delay_retry;/* Delay pg_init retry? */ @@ -498,7 +499,8 @@ static void process_queued_ios(struct wo (!pgpath && !m->queue_if_no_path)) must_queue =0 ; -if (m->pg_init_required && !m->pg_init_in_progress && pgpath) +if (m->pg_init_required && !m->pg_init_in_progress && pgpath && + !m->dtr_in_progress)__pg_init_all_paths (m);spin_unlock_irqrestore (&m->lock, flags);drivers/md/dm-mpath.c static void flush_multipath_work(struct · line 960
Quoted 8 lines
drivers/md/dm-mpath.c +5
-0@@ -951,6 +953,11 @@ static void flush_multipath_work(structstatic void multipath_dtr (struct dm_target *ti) {struct multipath *m = ti->private; +unsigned long flags; + +spin_lock_irqsave (&m->lock, flags); + m->dtr_in_progress =1 ; +spin_unlock_irqrestore (&m->lock, flags);Hi,
how about moving this to
flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work(){
<disablepg_initretry>
...
<enablepg_initretry>
}
Quoted 2 lines
Then it not only fixes the crash you hit, it also fixes the hidden bug that
pg_initcontinues retrying while the device is suspended.
We looked into adding checks for pg_init_required as well in multipath_wait_for_pg_init_completion(), but none of these will prevent pg_init_limit_reached() i.e from pg_init_done() from setting pg_init_required. From code I see we end up waiting indefinitely for its completion. The reason for adding check in process_queued_ios() is __choose_pgpath() might again trigger pg_init_required and end up calling __pg_init_all_paths() which will unset pg_init_required and sets pg_init_in_progress within a lock. Also I believe the patch given above will prevent further path activations and no work will be queued again with dtr_in_progress. With Hannes patch IO's will not be queued in when pg_init_in_progress so we are clean there as well. Let me know your thoughts.
PATCH v2 1/1 dm mpath: fix race condition between multipath_dtr and pg_init_done
In reply to Junichi NomuraQuoted 12 lines · Junichi Nomura
Hi,
how about moving this to
flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work(){
<disablepg_initretry>
...
<enablepg_initretry>
}
Quoted 2 lines
Then it not only fixes the crash you hit, it also fixes the hidden bug that
pg_initcontinues retrying while the device is suspended.
I ran with your suggestion. Please see below.
To be clear, pg_init isn't disabled while mpath device is suspended (meaning m->pg_init_disabled isn't set until the device is resumed). But flush_multipath_work() will no longer start pg_init during suspend -- which could otherwise occur while the mpath device is suspended. So in practice it accomplishes the stated goal.
Thanks for the suggestion Junichi. Are you OK with this? If so please provide your Ack.
Shiva, can you please verify that this patch resolves the race, should accomplish the same: just pushes the disabling of pg_init inside flush_multipath_work().
From: Shiva Krishna Merla <[email protected]>
Subject: dm mpath: fix race condition between multipath_dtr and pg_init_done
Date: Wed, 30 Oct 2013 03:26:38 +0000
Whenever multipath_dtr() is happening we must prevent queueing any further path activation work. Implement this by adding a new 'pg_init_disabled' flag to the multipath structure that denotes future path activation work should be skipped if it is set. By disabling pg_init and then re-enabling in flush_multipath_work() we also avoid the potential for pg_init to be initiated while suspending an mpath device.
Without this patch a race condition exists that may result in a kernel panic:
1) If after pg_init_done() decrements pg_init_in_progress to 0, a call to wait_for_pg_init_completion() assumes there are no more pending path management commands.
2) If pg_init_required is set by pg_init_done(), due to retryable mode_select errors, then process_queued_ios() will again queue the path activation work.
3) If free_multipath() completes before activate_path() work is called a NULL pointer dereference like the following can be seen when accessing members of the recently destructed multipath:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000090 RIP: 0010:[<ffffffffa003db1b>] [<ffffffffa003db1b>] activate_path+0x1b/0x30 [dm_multipath] [<ffffffff81090ac0>] worker_thread+0x170/0x2a0 [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40
--- drivers/md/dm-mpath.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
drivers/md/dm-mpath.c +15 -3
@@ -87,6 +87,7 @@ struct multipath { unsigned queue_if_no_path:1; /* Queue I/O if last path fails? */ unsigned saved_queue_if_no_path:1; /* Saved state during suspension */ unsigned retain_attached_hw_handler:1; /* If there's already a hw_handler present, don't change it. */ + unsigned pg_init_disabled:1; /* pginit is currently not allowed */unsigned pg_init_retries;/* Number of times to retry pg_init */ unsigned pg_init_count;/* Number of times pg_init called */
@@ -497,7 +498,8 @@ static void process_queued_ios(struct wo (!pgpath && !m->queue_if_no_path)) must_queue =0 ; -if (m->pg_init_required && !m->pg_init_in_progress && pgpath) +if (m->pg_init_required && !m->pg_init_in_progress && pgpath && + !m->pg_init_disabled)__pg_init_all_paths (m);spin_unlock_irqrestore (&m->lock, flags);
@@ -942,10 +944,20 @@ static void multipath_wait_for_pg_init_cstatic void flush_multipath_work (struct multipath *m) { +unsigned long flags; + +spin_lock_irqsave (&m->lock, flags); + m->pg_init_disabled =1 ; +spin_unlock_irqrestore (&m->lock, flags); +flush_workqueue (kmpath_handlerd);multipath_wait_for_pg_init_completion (m);flush_workqueue (kmultipathd);flush_work (&m->trigger_event); + +spin_lock_irqsave (&m->lock, flags); + m->pg_init_disabled =0 ; +spin_unlock_irqrestore (&m->lock, flags); }static void multipath_dtr (struct dm_target *ti)
@@ -1164,7 +1176,7 @@ static int pg_init_limit_reached(structspin_lock_irqsave (&m->lock, flags); -if (m->pg_init_count <= m->pg_init_retries) +if (m->pg_init_count <= m->pg_init_retries && !m->pg_init_disabled) m->pg_init_required =1 ;else limit_reached =1 ;
@@ -1714,7 +1726,7 @@ out:*---------------------------------------------------------------*/ static struct target_type multipath_target = { .name ="multipath" , - .version = {1 ,5 ,1 }, + .version = {1 ,6 ,0 }, .module =THIS_MODULE , .ctr = multipath_ctr, .dtr = multipath_dtr,
Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done
Quoted 35 lines
-----Original Message-----
From: Mike Snitzer [mailto:[email protected]]
Sent: Thursday, October 31, 2013 9:48 AM
To: Junichi Nomura; Merla, ShivaKrishna
Cc: device-mapper development; [email protected]
Subject: [PATCH v2] dm mpath: fix race condition betweenmultipath_dtrandpg_init_doneQuoted 12 lines · Junichi Nomura
Hi,
how about moving this to
flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work(){
<disablepg_initretry>
...
<enablepg_initretry>
}Quoted 2 lines
Then it not only fixes the crash you hit, it also fixes the hidden bug that
pg_initcontinues retrying while the device is suspended.I ran with your suggestion. Please see below.
To be clear,
pg_initisn't disabled while mpath device is suspended (meaningm->pg_init_disabledisn't set until the device is resumed). Butflush_multipath_work()will no longer startpg_initduring suspend -- which could otherwise occur while the mpath device is suspended. So in practice it accomplishes the stated goal.
Quoted 6 lines
Thanks for the suggestion Junichi. Are you OK with this? If so please provide your Ack.
Shiva, can you please verify that this patch resolves the race, should accomplish the same: just pushes the disabling of
pg_initinsideflush_multipath_work().
Yes Mike, This looks good. Thanks Junichi for the suggestion.
Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done
In reply to Mike SnitzerQuoted 16 lines · Mike Snitzer
Quoted 11 lines · Junichi Nomura
how about moving this to
flush_multipath_work(), which is supposed to silence background activities?
I.e.
flush_multipath_work(){
<disablepg_initretry>
...
<enablepg_initretry>
}Then it not only fixes the crash you hit, it also fixes the hidden bug that
pg_initcontinues retrying while the device is suspended.I ran with your suggestion. Please see below.
Quoted 8 lines
To be clear,
pg_initisn't disabled while mpath device is suspended (meaningm->pg_init_disabledisn't set until the device is resumed). Butflush_multipath_work()will no longer startpg_initduring suspend -- which could otherwise occur while the mpath device is suspended. So in practice it accomplishes the stated goal.Thanks for the suggestion Junichi. Are you OK with this? If so please provide your Ack.
Yes, that's perfect. Thank you.
And thanks Shiva Krishna for finding/fixing this.
Quoted 4 lines
Shiva, can you please verify that this patch resolves the race, should accomplish the same: just pushes the disabling of
pg_initinsideflush_multipath_work().
Signature · 1 line
Jun'ichi Nomura, NEC Corporation
Alasdair G Kergon
maintainerRe: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done
In reply to Mike Snitzerdrivers/md/dm-mpath.c static void process_queued_ios(struct wo · line 502
Quoted 7 lines · Mike Snitzer
drivers/md/dm-mpath.c +3
-0+ unsigned pg_init_disabled:1; /* pginit is currently not allowed */ +if (m->pg_init_required && !m->pg_init_in_progress && pgpath && + !m->pg_init_disabled)
So we're accepting that pg_init might be both "disabled" and "required" simultaneously (otherwise we wouldn't need this test)?
drivers/md/dm-mpath.c static void multipath_wait_for_pg_init_c · line 953
Quoted 11 lines
drivers/md/dm-mpath.c +6
-0@@ -942,10 +944,20 @@ static void multipath_wait_for_pg_init_cstatic void flush_multipath_work (struct multipath *m) { +unsigned long flags; + +spin_lock_irqsave (&m->lock, flags); + m->pg_init_disabled =1 ; +spin_unlock_irqrestore (&m->lock, flags); +flush_workqueue (kmpath_handlerd);multipath_wait_for_pg_init_completion (m);
So this implies pg_init could even still be running while it is disabled?
If the logic is correct, then I find the new variable name quite confusing and think it should be changed, or as a minimum have inline comments explaining the apparent contradictions!
Alasdair
Re: [PATCH v2] dm mpath: fix race condition between multipath_dtr and pg_init_done
Quoted 11 lines · Alasdair G Kergon
drivers/md/dm-mpath.c static void process_queued_ios(struct wo · line 502
Quoted 7 lines · Mike Snitzer
drivers/md/dm-mpath.c +3
-0+ unsigned pg_init_disabled:1; /* pginit is currently not allowed */ +if (m->pg_init_required && !m->pg_init_in_progress && pgpath && + !m->pg_init_disabled)So we're accepting that
pg_initmight be both "disabled" and "required" simultaneously (otherwise we wouldn't need this test)?
Yes, needs to be independent of pg_init_required.
Quoted 5 lines
Quoted 5 lines
drivers/md/dm-mpath.c +2
-0@@ -942,10 +944,20 @@ static void multipath_wait_for_pg_init_cstatic void flush_multipath_work (struct multipath *m) { +unsigned long flags; +
Quoted 8 lines
drivers/md/dm-mpath.c static void multipath_wait_for_pg_init_c · line 954
Quoted 6 lines
drivers/md/dm-mpath.c +4
-0+spin_lock_irqsave (&m->lock, flags); + m->pg_init_disabled =1 ; +spin_unlock_irqrestore (&m->lock, flags); +flush_workqueue (kmpath_handlerd);multipath_wait_for_pg_init_completion (m);So this implies
pg_initcould even still be running while it is disabled?
pg_init_disabled doesn't allow pg_init to be initiated once set (even if pg_init_required is). Hence pg_init is disabled.
Quoted 3 lines
If the logic is correct, then I find the new variable name quite confusing and think it should be changed, or as a minimum have inline comments explaining the apparent contradictions!
I'm not seeing a contradiction.