multipathd: fix __delayed_reconfig logic
cover multipathd: fix __delayed_reconfig logic
From: Martin Wilck <[email protected]>
Hi Ben, hi Christophe, hi Guozhonghua,
here's my new take at the race condition issue reported by Guozhonghua. This patch set is meant as an alternative to Ben's recent series "fix looping when reconfigure is delayed". I believe that by removing the special case from __post_config_state(), the state change logic becomes somewhat easier to understand again.
I hope I got it right this time.
Regards
Martin
Martin Wilck (5):
multipathd: child(): remove superfluous if condition
multipathd: set reload_type in when calling reconfigure()
multipathd: avoid busy loop in child() for delayed reconfigure
multipathd: reset __delayed_reconfig from ev_add_map()
multipathd: remove volatile qualifier from running_state
multipathd/main.c | 146 +++++++++++++++++++++------------------------- 1 file changed, 67 insertions(+), 79 deletions(-)
Signature · 6 lines
2.35.1 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 1/5 multipathd: child(): remove superfluous if condition
From: Martin Wilck <[email protected]>
No need to test for state == DAEMON_CONFIGURE at this point, we know that this is the case.
--- multipathd/main.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
multipathd/main.c +16 -16
@@ -3395,6 +3395,8 @@ child (__attribute__((unused)) void *param)pthread_attr_destroy (&misc_attr);while (1 ) { +int rc =0 ; +pthread_cleanup_push (config_cleanup,NULL );pthread_mutex_lock (&config_lock); while (running_state != DAEMON_CONFIGURE &&
@@ -3404,23 +3406,21 @@ child (__attribute__((unused)) void *param)pthread_cleanup_pop (1 );if (state ==DAEMON_SHUTDOWN )break ; -if (state ==DAEMON_CONFIGURE ) { -int rc =0 ; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (&vecs->lock); -pthread_testcancel (); -if (!need_to_delay_reconfig (vecs)) - rc =reconfigure (vecs); -else -enable_delayed_reconfig (); -lock_cleanup_pop (vecs->lock); -if (!rc) -post_config_state (DAEMON_IDLE ); -else { -condlog (0 ,"fatal error applying configuration - aborting" ); -exit_daemon (); - } +/* handle DAEMON_CONFIGURE */ +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (&vecs->lock); +pthread_testcancel (); +if (!need_to_delay_reconfig (vecs)) + rc =reconfigure (vecs); +else +enable_delayed_reconfig (); +lock_cleanup_pop (vecs->lock); +if (!rc) +post_config_state (DAEMON_IDLE ); +else { +condlog (0 ,"fatal error applying configuration - aborting" ); +exit_daemon (); } }
Signature · 6 lines
2.35.1 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [PATCH 1/5] multipathd: child(): remove superfluous if condition
Quoted 1 line · On Mon, Mar 14, 2022 at 10:30:32PM +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
Quoted 62 lines
No need to test for state ==
DAEMON_CONFIGUREat this point, we know that this is the case.Signed-off-by Martin Wilck <[email protected]>--- multipathd/main.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)multipathd/main.c +16
-16@@ -3395,6 +3395,8 @@ child (__attribute__((unused)) void *param)pthread_attr_destroy (&misc_attr);while (1 ) { +int rc =0 ; +pthread_cleanup_push (config_cleanup,NULL );pthread_mutex_lock (&config_lock); while (running_state != DAEMON_CONFIGURE &&@@ -3404,23 +3406,21 @@ child (__attribute__((unused)) void *param)pthread_cleanup_pop (1 );if (state ==DAEMON_SHUTDOWN )break ; -if (state ==DAEMON_CONFIGURE ) { -int rc =0 ; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (&vecs->lock); -pthread_testcancel (); -if (!need_to_delay_reconfig (vecs)) - rc =reconfigure (vecs); -else -enable_delayed_reconfig (); -lock_cleanup_pop (vecs->lock); -if (!rc) -post_config_state (DAEMON_IDLE ); -else { -condlog (0 ,"fatal error applying configuration - aborting" ); -exit_daemon (); - } +/* handle DAEMON_CONFIGURE */ +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (&vecs->lock); +pthread_testcancel (); +if (!need_to_delay_reconfig (vecs)) + rc =reconfigure (vecs); +else +enable_delayed_reconfig (); +lock_cleanup_pop (vecs->lock); +if (!rc) +post_config_state (DAEMON_IDLE ); +else { +condlog (0 ,"fatal error applying configuration - aborting" ); +exit_daemon (); } }Signature · 1 line
2.35.1
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 2/5 multipathd: set reload_type in when calling reconfigure()
In reply to [email protected]From: Martin Wilck <[email protected]>
Only set reload_type (and reset reconfigure_pending) immediately before we actually call reconfigure(). This allows us to get rid of the reload_type global variable, and makes sure that reconfigure() is called with the reload type that was last requested.
While at it, convert configure() and reconfigure() to static functions.
--- multipathd/main.c | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
multipathd/main.c +17 -18
@@ -287,14 +287,10 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,/* Don't access this variable without holding config_lock */ static enum force_reload_types reconfigure_pending =FORCE_RELOAD_NONE ; -/* Only set while changing to DAEMON_CONFIGURE, and only access while -* reconfiguring or scheduling a delayed reconfig in DAEMON_CONFIGURE */ -static volatile enum force_reload_types reload_type =FORCE_RELOAD_NONE ;static void enable_delayed_reconfig (void ) {pthread_mutex_lock (&config_lock); - reconfigure_pending = reload_type; __delayed_reconfig = true;pthread_mutex_unlock (&config_lock); }
@@ -324,11 +320,6 @@ static void __post_config_state(enum daemon_status state) old_state =DAEMON_IDLE ; state =DAEMON_CONFIGURE ; } -if (state ==DAEMON_CONFIGURE ) { - reload_type = (reconfigure_pending ==FORCE_RELOAD_YES ) ?FORCE_RELOAD_YES :FORCE_RELOAD_WEAK ; - reconfigure_pending =FORCE_RELOAD_NONE ; - __delayed_reconfig = false; - } running_state = state;pthread_cond_broadcast (&config_cond);do_sd_notify (old_state, state);
@@ -2714,8 +2705,8 @@ checkerloop (void *ap)return NULL ; } -int -configure (struct vectors * vecs) +static int +configure (struct vectors * vecs,enum force_reload_types reload_type) {struct multipath * mpp;struct path * pp;
@@ -2846,8 +2837,8 @@ void rcu_free_config(struct rcu_head *head)free_config (conf); } -int -reconfigure (struct vectors * vecs) +static int +reconfigure (struct vectors *vecs,enum force_reload_types reload_type) {struct config * old, *conf;int old_marginal_pathgroups;
@@ -2894,8 +2885,7 @@ reconfigure (struct vectors * vecs)#ifdef FPIN_EVENT_HANDLER fpin_clean_marginal_dev_list (NULL );#endif -configure (vecs); - +configure (vecs, reload_type);return 0 ; }
@@ -3411,9 +3401,18 @@ child (__attribute__((unused)) void *param)pthread_cleanup_push (cleanup_lock, &vecs->lock);lock (&vecs->lock);pthread_testcancel (); -if (!need_to_delay_reconfig (vecs)) - rc =reconfigure (vecs); -else +if (!need_to_delay_reconfig (vecs)) { +enum force_reload_types reload_type; + +pthread_mutex_lock (&config_lock); + reload_type = reconfigure_pending ==FORCE_RELOAD_YES ? +FORCE_RELOAD_YES :FORCE_RELOAD_WEAK ; + reconfigure_pending =FORCE_RELOAD_NONE ; + __delayed_reconfig = false; +pthread_mutex_unlock (&config_lock); + + rc =reconfigure (vecs, reload_type); + }else enable_delayed_reconfig ();lock_cleanup_pop (vecs->lock);if (!rc)
Signature · 6 lines
2.35.1 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 3/5 multipathd: avoid busy loop in child() for delayed reconfigure
In reply to [email protected]From: Martin Wilck <[email protected]>
If need_to_delay_reconfig() returned true, the logic introduced by 250708c ("multipathd: improve delayed reconfigure") and 4b104e6 ("multipathd: pass in the type of reconfigure") could cause child() to run in a tight loop, switching back and forth between DAEMON_CONFIGURE and DAEMON_IDLE states without actually calling reconfigure().
Move the logic to postpone reconfigure() calls from __post_config_state() into child(), entirely, to avoid this situation. This means that child() needs to poll for reconfigure_pending besides running_state changes.
--- multipathd/main.c | 48 +++++++++++++++++++---------------------------- 1 file changed, 19 insertions(+), 29 deletions(-)
multipathd/main.c +19 -29
@@ -288,38 +288,12 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,/* Don't access this variable without holding config_lock */ static enum force_reload_types reconfigure_pending =FORCE_RELOAD_NONE ; -static void enable_delayed_reconfig (void ) -{ -pthread_mutex_lock (&config_lock); - __delayed_reconfig = true; -pthread_mutex_unlock (&config_lock); -} -/* must be called with config_lock held */ static void __post_config_state (enum daemon_status state) {if (state != running_state && running_state !=DAEMON_SHUTDOWN ) {enum daemon_status old_state = running_state; -/* -* Handle a pending reconfigure request. -* DAEMON_IDLE is set from child() after reconfigure(), -* or from checkerloop() after completing checkers. -* In either case, child() will see DAEMON_CONFIGURE -* again and start another reconfigure cycle. -*/ -if (reconfigure_pending !=FORCE_RELOAD_NONE && - state ==DAEMON_IDLE && - (old_state ==DAEMON_CONFIGURE || - old_state ==DAEMON_RUNNING )) { -/* -* notify systemd of transient idle state, lest systemd -* thinks the reload lasts forever. -*/ -do_sd_notify (old_state,DAEMON_IDLE ); - old_state =DAEMON_IDLE ; - state =DAEMON_CONFIGURE ; - } running_state = state;pthread_cond_broadcast (&config_cond);do_sd_notify (old_state, state);
@@ -3390,8 +3364,21 @@ child (__attribute__((unused)) void *param)pthread_cleanup_push (config_cleanup,NULL );pthread_mutex_lock (&config_lock);while (running_state !=DAEMON_CONFIGURE && - running_state !=DAEMON_SHUTDOWN ) + running_state !=DAEMON_SHUTDOWN && +/* +* Check if another reconfigure request was scheduled +* while we last ran reconfigure(). +* We have to test __delayed_reconfig here +* to avoid a busy loop +*/ + (reconfigure_pending ==FORCE_RELOAD_NONE + || __delayed_reconfig))pthread_cond_wait (&config_cond, &config_lock); + +if (running_state !=DAEMON_CONFIGURE && + running_state !=DAEMON_SHUTDOWN ) +/* This sets running_state to DAEMON_CONFIGURE */ +__post_config_state (DAEMON_CONFIGURE ); state = running_state;pthread_cleanup_pop (1 );if (state ==DAEMON_SHUTDOWN )
@@ -3412,8 +3399,11 @@ child (__attribute__((unused)) void *param)pthread_mutex_unlock (&config_lock); rc =reconfigure (vecs, reload_type); - }else -enable_delayed_reconfig (); + }else { +pthread_mutex_lock (&config_lock); + __delayed_reconfig = true; +pthread_mutex_unlock (&config_lock); + }lock_cleanup_pop (vecs->lock);if (!rc)post_config_state (DAEMON_IDLE );
Signature · 6 lines
2.35.1 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [PATCH 3/5] multipathd: avoid busy loop in child() for delayed reconfigure
Quoted 9 lines · On Mon, Mar 14, 2022 at 10:30:34PM +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
If
need_to_delay_reconfig()returned true, the logic introduced by 250708c ("multipathd: improve delayed reconfigure") and 4b104e6 ("multipathd: pass in the type of reconfigure") could causechild()to run in a tight loop, switching back and forth betweenDAEMON_CONFIGUREandDAEMON_IDLEstates without actually callingreconfigure().
Quoted 4 lines
Move the logic to postpone
reconfigure()calls from__post_config_state()intochild(), entirely, to avoid this situation. This means thatchild()needs to poll forreconfigure_pendingbesidesrunning_statechanges.
With the understanding that this fix still leaves a bug that needs to be resolved.
Quoted 89 lines
Fixes 250708c ("multipathd: improve delayed reconfigure")Fixes 4b104e6 ("multipathd: pass in the type of reconfigure")Signed-off-by Martin Wilck <[email protected]>--- multipathd/main.c | 48 +++++++++++++++++++---------------------------- 1 file changed, 19 insertions(+), 29 deletions(-)multipathd/main.c +19
-29@@ -288,38 +288,12 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,/* Don't access this variable without holding config_lock */ static enum force_reload_types reconfigure_pending =FORCE_RELOAD_NONE ; -static void enable_delayed_reconfig (void ) -{ -pthread_mutex_lock (&config_lock); - __delayed_reconfig = true; -pthread_mutex_unlock (&config_lock); -} -/* must be called with config_lock held */ static void __post_config_state (enum daemon_status state) {if (state != running_state && running_state !=DAEMON_SHUTDOWN ) {enum daemon_status old_state = running_state; -/* -* Handle a pending reconfigure request. -* DAEMON_IDLE is set from child() after reconfigure(), -* or from checkerloop() after completing checkers. -* In either case, child() will see DAEMON_CONFIGURE -* again and start another reconfigure cycle. -*/ -if (reconfigure_pending !=FORCE_RELOAD_NONE && - state ==DAEMON_IDLE && - (old_state ==DAEMON_CONFIGURE || - old_state ==DAEMON_RUNNING )) { -/* -* notify systemd of transient idle state, lest systemd -* thinks the reload lasts forever. -*/ -do_sd_notify (old_state,DAEMON_IDLE ); - old_state =DAEMON_IDLE ; - state =DAEMON_CONFIGURE ; - } running_state = state;pthread_cond_broadcast (&config_cond);do_sd_notify (old_state, state);@@ -3390,8 +3364,21 @@ child (__attribute__((unused)) void *param)pthread_cleanup_push (config_cleanup,NULL );pthread_mutex_lock (&config_lock);while (running_state !=DAEMON_CONFIGURE && - running_state !=DAEMON_SHUTDOWN ) + running_state !=DAEMON_SHUTDOWN && +/* +* Check if another reconfigure request was scheduled +* while we last ran reconfigure(). +* We have to test __delayed_reconfig here +* to avoid a busy loop +*/ + (reconfigure_pending ==FORCE_RELOAD_NONE + || __delayed_reconfig))pthread_cond_wait (&config_cond, &config_lock); + +if (running_state !=DAEMON_CONFIGURE && + running_state !=DAEMON_SHUTDOWN ) +/* This sets running_state to DAEMON_CONFIGURE */ +__post_config_state (DAEMON_CONFIGURE ); state = running_state;pthread_cleanup_pop (1 );if (state ==DAEMON_SHUTDOWN )@@ -3412,8 +3399,11 @@ child (__attribute__((unused)) void *param)pthread_mutex_unlock (&config_lock); rc =reconfigure (vecs, reload_type); - }else -enable_delayed_reconfig (); + }else { +pthread_mutex_lock (&config_lock); + __delayed_reconfig = true; +pthread_mutex_unlock (&config_lock); + }lock_cleanup_pop (vecs->lock);if (!rc)post_config_state (DAEMON_IDLE );Signature · 1 line
2.35.1
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 4/5 multipathd: reset __delayed_reconfig from ev_add_map()
In reply to [email protected]From: Martin Wilck <[email protected]>
With the previous patch, one race condition between child() and the uevent handler (ev_add_map()) remains: 1. child() sets __delayed_reconfig, 2. ev_add_map() calls schedule_reconfigure() and thus DAEMON_CONFIGURE, 3. child() sets DAEMON_IDLE. This would cause the pending reconfigure request to be missed.
To fix this, reset __delayed_reconfig immediately from ev_add_map() (and likewise, missing_uev_wait_tick()). This way the wait loop in child() terminates on the reconfigure_pending condition.
Also, these schedule_reconfigure() callers don't really request a reconfigure() call; they just unblock processing previously requested reconfigure() calls. Add a new helper, unblock_reconfigure(), that does just that.
--- multipathd/main.c | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-)
multipathd/main.c +20 -21
@@ -155,16 +155,6 @@ int should_exit(void)return get_running_state () ==DAEMON_SHUTDOWN ; } -static bool get_delayed_reconfig (void ) -{ -bool val; - -pthread_mutex_lock (&config_lock); - val = __delayed_reconfig; -pthread_mutex_unlock (&config_lock); -return val; -} -/* * global copy of vecs for use in sig handlers */
@@ -308,6 +298,22 @@ void post_config_state(enum daemon_status state)pthread_cleanup_pop (1 ); } +static bool unblock_reconfigure (void ) +{ +bool was_delayed; + +pthread_mutex_lock (&config_lock); + was_delayed = __delayed_reconfig; +if (was_delayed) { + __delayed_reconfig = false; +__post_config_state (DAEMON_CONFIGURE ); + } +pthread_mutex_unlock (&config_lock); +if (was_delayed) +condlog (2 ,"reconfigure (delayed)" ); +return was_delayed; +} +void schedule_reconfigure (enum force_reload_types requested_type) {pthread_mutex_lock (&config_lock);
@@ -790,12 +796,9 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)dm_get_info (mpp->alias, &mpp->dmi);if (mpp->wait_for_udev) { mpp->wait_for_udev =0 ; -if (get_delayed_reconfig () && - !need_to_delay_reconfig (vecs)) { -condlog (2 ,"reconfigure (delayed)" ); -schedule_reconfigure (FORCE_RELOAD_WEAK ); +if (!need_to_delay_reconfig (vecs) && +unblock_reconfigure ())return 0 ; - } }/* * Not really an error -- we generate our own uevent
@@ -1899,11 +1902,8 @@ missing_uev_wait_tick(struct vectors *vecs) } } -if (timed_out &&get_delayed_reconfig () && - !need_to_delay_reconfig (vecs)) { -condlog (2 ,"reconfigure (delayed)" ); -schedule_reconfigure (FORCE_RELOAD_WEAK ); - } +if (timed_out && !need_to_delay_reconfig (vecs)) +unblock_reconfigure (); } static void
@@ -3395,7 +3395,6 @@ child (__attribute__((unused)) void *param) reload_type = reconfigure_pending ==FORCE_RELOAD_YES ?FORCE_RELOAD_YES :FORCE_RELOAD_WEAK ; reconfigure_pending =FORCE_RELOAD_NONE ; - __delayed_reconfig = false;pthread_mutex_unlock (&config_lock); rc =reconfigure (vecs, reload_type);
Signature · 6 lines
2.35.1 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Benjamin Marzinski
maintainerRe: [dm-devel] [PATCH 4/5] multipathd: reset __delayed_reconfig from ev_add_map()
Quoted 8 lines · On Mon, Mar 14, 2022 at 10:30:35PM +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
With the previous patch, one race condition between
child()and the uevent handler (ev_add_map()) remains: 1.child()sets__delayed_reconfig, 2.ev_add_map()callsschedule_reconfigure()and thusDAEMON_CONFIGURE, 3.child()setsDAEMON_IDLE. This would cause the pending reconfigure request to be missed.
Quoted 9 lines
To fix this, reset
__delayed_reconfigimmediately fromev_add_map()(and likewise,missing_uev_wait_tick()). This way the wait loop inchild()terminates on thereconfigure_pendingcondition.Also, these
schedule_reconfigure()callers don't really request areconfigure()call; they just unblock processing previously requestedreconfigure()calls. Add a new helper,unblock_reconfigure(), that does just that.
Doesn't unblock_reconfigure() allow us to set the state to DAEMON_CONFIGURE, right after the checkerloop() has set it to DAEMON_RUNNING. schedule_reconfigure() didn't update the state if it was DAEMON_RUNNING. Instead, it would wait till checkerloop() set the state back to DAEMON_IDLE for the reconfigure to start.
I have also come up with a different bug that effects these fixes, including mine. But it will effect this fix worse. This fix assumes that when a reconfigure is delayed, it should remain delayed until either a change event happens on the multipath device (ev_add_map) or multipathd times out waiting for one (missing_uev_wait_tick). However the map could be removed before either of those things happen. It's possible that multipathd could get a remove uevent after the add event but before the change event. Either that or a multipathd client command could remove it, or a dm event could happen either removing the device, or updating it, but with __setup_multipath() removing it. Those are just the examples I came up with off the top of my head.
So there's a specific problem where we don't handle clearing __delayed_reconfig if a map was removed. That can be handled in a different patch. But this, or any other oversight we might have here can be mitigated by being more proactive in removing __delayed_reconfig. For instance, when then main thread tries to reconfigure, it checks need_to_delay_reconfig(), and only does the reconfigure if this returns that it's ok. __delayed_reconfig mostly exists so that the main thread doesn't need to grab the vecs lock and loop through the the multipath devices to know if it need to delay, but need_to_delay_reconfig() is the definitive answer. When the main thread checks this, we should be updating __delayed_reconfig to match the result. This patch removes the "__delayed_reconfig = false;" line, which I think is a mistake.
Imagine if, because a map got removed before ev_add_map() was called, __delayed_reconfig was true, even though there were no maps with mpp->wait_for_udev set. Any existing config request would still be delayed, but if another reconfigure was requested not only would it happen, but if the "__delayed_reconfig = false;" line was back, __delayed_reconfig would go back to the correct value.
-Ben
Quoted 89 lines
Suggested-by Benjamin Marzinski <[email protected]>Signed-off-by Martin Wilck <[email protected]>--- multipathd/main.c | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-)multipathd/main.c +20
-21@@ -155,16 +155,6 @@ int should_exit(void)return get_running_state () ==DAEMON_SHUTDOWN ; } -static bool get_delayed_reconfig (void ) -{ -bool val; - -pthread_mutex_lock (&config_lock); - val = __delayed_reconfig; -pthread_mutex_unlock (&config_lock); -return val; -} -/* * global copy of vecs for use in sig handlers */ @@ -308,6 +298,22 @@ void post_config_state(enum daemon_status state)pthread_cleanup_pop (1 ); } +static bool unblock_reconfigure (void ) +{ +bool was_delayed; + +pthread_mutex_lock (&config_lock); + was_delayed = __delayed_reconfig; +if (was_delayed) { + __delayed_reconfig = false; +__post_config_state (DAEMON_CONFIGURE ); + } +pthread_mutex_unlock (&config_lock); +if (was_delayed) +condlog (2 ,"reconfigure (delayed)" ); +return was_delayed; +} +void schedule_reconfigure (enum force_reload_types requested_type) {pthread_mutex_lock (&config_lock);@@ -790,12 +796,9 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)dm_get_info (mpp->alias, &mpp->dmi);if (mpp->wait_for_udev) { mpp->wait_for_udev =0 ; -if (get_delayed_reconfig () && - !need_to_delay_reconfig (vecs)) { -condlog (2 ,"reconfigure (delayed)" ); -schedule_reconfigure (FORCE_RELOAD_WEAK ); +if (!need_to_delay_reconfig (vecs) && +unblock_reconfigure ())return 0 ; - } }/* * Not really an error -- we generate our own uevent @@ -1899,11 +1902,8 @@ missing_uev_wait_tick(struct vectors *vecs) } } -if (timed_out &&get_delayed_reconfig () && - !need_to_delay_reconfig (vecs)) { -condlog (2 ,"reconfigure (delayed)" ); -schedule_reconfigure (FORCE_RELOAD_WEAK ); - } +if (timed_out && !need_to_delay_reconfig (vecs)) +unblock_reconfigure (); } static void@@ -3395,7 +3395,6 @@ child (__attribute__((unused)) void *param) reload_type = reconfigure_pending ==FORCE_RELOAD_YES ?FORCE_RELOAD_YES :FORCE_RELOAD_WEAK ; reconfigure_pending =FORCE_RELOAD_NONE ; - __delayed_reconfig = false;pthread_mutex_unlock (&config_lock); rc =reconfigure (vecs, reload_type);Signature · 1 line
2.35.1
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [PATCH 4/5] multipathd: reset __delayed_reconfig from ev_add_map()
Quoted 22 lines · Benjamin Marzinski
Quoted 14 lines · On Mon, Mar 14, 2022 at 10:30:35PM +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
With the previous patch, one race condition between
child()and the uevent handler (ev_add_map()) remains: 1.child()sets__delayed_reconfig, 2.ev_add_map()callsschedule_reconfigure()and
thusDAEMON_CONFIGURE, 3.child()setsDAEMON_IDLE. This would cause
the pending reconfigure request to be missed.To fix this, reset
__delayed_reconfigimmediately fromev_add_map()(and likewise,missing_uev_wait_tick()). This way the wait loop inchild()terminates on thereconfigure_pendingcondition.Quoted 6 lines
Also, these
schedule_reconfigure()callers don't really request areconfigure()call; they just unblock processing previously requestedreconfigure()calls. Add a new helper,unblock_reconfigure(), that does just that.
Quoted 3 lines
Doesn't
unblock_reconfigure()allow us to set the state toDAEMON_CONFIGURE, right after thecheckerloop()has set it toDAEMON_RUNNING.
Right. I added the __post_config_state() there in order to be sure that child() would be woken up. But unblock_reconfigure() should only call __post_config_state if running_state is DAEMON_IDLE, like schedule_reconfigure(). Otherwise, it can be sure that some other process will wake up child().
Quoted 24 lines
schedule_reconfigure()didn't update the state if it wasDAEMON_RUNNING. Instead, it would wait tillcheckerloop()set the state
back toDAEMON_IDLEfor the reconfigure to start.I have also come up with a different bug that effects these fixes, including mine. But it will effect this fix worse. This fix assumes that
when a reconfigure is delayed, it should remain delayed until either a
change event happens on the multipath device (ev_add_map) or multipathd
times out waiting for one (missing_uev_wait_tick). However the map could
be removed before either of those things happen. It's possible that multipathd could get a remove uevent after the add event but before the
change event. Either that or a multipathd client command could remove it, or a dm event could happen either removing the device, or updating
it, but with__setup_multipath()removing it. Those are just the examples I came up with off the top of my head.
Quoted 16 lines
So there's a specific problem where we don't handle clearing
__delayed_reconfigif a map was removed. That can be handled in a different patch. But this, or any other oversight we might have here can be mitigated by being more proactive in removing__delayed_reconfig.
For instance, when then main thread tries to reconfigure, it checksneed_to_delay_reconfig(), and only does the reconfigure if this returns
that it's ok.__delayed_reconfigmostly exists so that the main thread
doesn't need to grab the vecs lock and loop through the the multipath devices to know if it need to delay, butneed_to_delay_reconfig()is the definitive answer. When the main thread checks this, we should be updating__delayed_reconfigto match the result. This patch removes the
"__delayed_reconfig= false;" line, which I think is a mistake.
Agreed. My thinking was that this is done by the uevent handler when the state of need_to_delay_reconfig() changes. But you're right with the "remove" special case. We should re-add this, it definitely doesn't hurt.
Quoted 6 lines
Imagine if, because a map got removed before
ev_add_map()was called,__delayed_reconfigwas true, even though there were no maps withmpp->wait_for_udevset. Any existing config request would still be delayed, but if another reconfigure was requested not only would it happen, but if the "__delayed_reconfig= false;" line was back,__delayed_reconfigwould go back to the correct value.
So we should call unblock_reconfigure() from remove_map(). Unfortunately that requires another callback from libmultipath into multipathd. Trying to figure it out...
Thanks,
Martin
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
PATCH 5/5 multipathd: remove volatile qualifier from running_state
In reply to [email protected]From: Martin Wilck <[email protected]>
running_state is only accessed while holding the config_lock, the volatile qualifier is superfluous.
--- multipathd/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
multipathd/main.c +1 -1
@@ -127,7 +127,7 @@ static int poll_dmevents = 0;static int poll_dmevents =1 ;#endif /* Don't access this variable without holding config_lock */ -static volatile enum daemon_status running_state =DAEMON_INIT ; +static enum daemon_status running_state =DAEMON_INIT ;/* Don't access this variable without holding config_lock */ static bool __delayed_reconfig;pid_t daemon_pid;
Signature · 6 lines
2.35.1 -- dm-devel mailing list [email protected] https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] [PATCH 5/5] multipathd: remove volatile qualifier from running_state
Quoted 1 line · On Mon, Mar 14, 2022 at 10:30:36PM +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
Quoted 25 lines
running_stateis only accessed while holding theconfig_lock, the volatile qualifier is superfluous.Suggested-by Benjamin Marzinski <[email protected]>Signed-off-by Martin Wilck <[email protected]>--- multipathd/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)multipathd/main.c +1
-1@@ -127,7 +127,7 @@ static int poll_dmevents = 0;static int poll_dmevents =1 ;#endif /* Don't access this variable without holding config_lock */ -static volatile enum daemon_status running_state =DAEMON_INIT ; +static enum daemon_status running_state =DAEMON_INIT ;/* Don't access this variable without holding config_lock */ static bool __delayed_reconfig;pid_t daemon_pid;Signature · 1 line
2.35.1
--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel