Skip to contentSkip to conversation

multipathd: fix __delayed_reconfig logic

PATCH Superseded [email protected] 2 participants 11 messages Mar 14, 2022 – Mar 16, 2022 Device mapper

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover multipathd: fix __delayed_reconfig logic

From: Martin Wilck <[email protected]>

Hi Ben, hi Christophe, hi Guozhonghua,

here's my new take at the race condition issue reported by Guozhonghua. This patch set is meant as an alternative to Ben's recent series "fix looping when reconfigure is delayed". I believe that by removing the special case from __post_config_state(), the state change logic becomes somewhat easier to understand again.

I hope I got it right this time.

Regards
Martin

Martin Wilck (5):
multipathd: child(): remove superfluous if condition
multipathd: set reload_type in when calling reconfigure()
multipathd: avoid busy loop in child() for delayed reconfigure
multipathd: reset __delayed_reconfig from ev_add_map()
multipathd: remove volatile qualifier from running_state

 multipathd/main.c | 146 +++++++++++++++++++++-------------------------
 1 file changed, 67 insertions(+), 79 deletions(-)
Signature · 6 lines
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#1 lore raw

PATCH 1/5 multipathd: child(): remove superfluous if condition

From: Martin Wilck <[email protected]>

No need to test for state == DAEMON_CONFIGURE at this point, we know that this is the case.

---
 multipathd/main.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
multipathd/main.c +16 -16
index f2c0b28..1c8839d 100644
@@ -3395,6 +3395,8 @@ child (__attribute__((unused)) void *param)
 	pthread_attr_destroy(&misc_attr);
 
 	while (1) {
+		int rc = 0;
+
 		pthread_cleanup_push(config_cleanup, NULL);
 		pthread_mutex_lock(&config_lock);
 		while (running_state != DAEMON_CONFIGURE &&
@@ -3404,23 +3406,21 @@ child (__attribute__((unused)) void *param)
 		pthread_cleanup_pop(1);
 		if (state == DAEMON_SHUTDOWN)
 			break;
-		if (state == DAEMON_CONFIGURE) {
-			int rc = 0;
 
-			pthread_cleanup_push(cleanup_lock, &vecs->lock);
-			lock(&vecs->lock);
-			pthread_testcancel();
-			if (!need_to_delay_reconfig(vecs))
-				rc = reconfigure(vecs);
-			else
-				enable_delayed_reconfig();
-			lock_cleanup_pop(vecs->lock);
-			if (!rc)
-				post_config_state(DAEMON_IDLE);
-			else {
-				condlog(0, "fatal error applying configuration - aborting");
-				exit_daemon();
-			}
+		/* handle DAEMON_CONFIGURE */
+		pthread_cleanup_push(cleanup_lock, &vecs->lock);
+		lock(&vecs->lock);
+		pthread_testcancel();
+		if (!need_to_delay_reconfig(vecs))
+			rc = reconfigure(vecs);
+		else
+			enable_delayed_reconfig();
+		lock_cleanup_pop(vecs->lock);
+		if (!rc)
+			post_config_state(DAEMON_IDLE);
+		else {
+			condlog(0, "fatal error applying configuration - aborting");
+			exit_daemon();
 		}
 	}
 
Signature · 6 lines
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
maintainer Reviewed-by on v1 1/5
#2 lore raw

Re: [dm-devel] [PATCH 1/5] multipathd: child(): remove superfluous if condition

Quoted 1 line · On Mon, Mar 14, 2022 at 10:30:32PM +0100, [email protected] wrote:

From: Martin Wilck <[email protected]>

Quoted 62 lines

No need to test for state == DAEMON_CONFIGURE at this point, we know that this is the case.

---
 multipathd/main.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
multipathd/main.c +16 -16
index f2c0b28..1c8839d 100644
@@ -3395,6 +3395,8 @@ child (__attribute__((unused)) void *param)
 	pthread_attr_destroy(&misc_attr);
 
 	while (1) {
+		int rc = 0;
+
 		pthread_cleanup_push(config_cleanup, NULL);
 		pthread_mutex_lock(&config_lock);
 		while (running_state != DAEMON_CONFIGURE &&
@@ -3404,23 +3406,21 @@ child (__attribute__((unused)) void *param)
 		pthread_cleanup_pop(1);
 		if (state == DAEMON_SHUTDOWN)
 			break;
-		if (state == DAEMON_CONFIGURE) {
-			int rc = 0;
 
-			pthread_cleanup_push(cleanup_lock, &vecs->lock);
-			lock(&vecs->lock);
-			pthread_testcancel();
-			if (!need_to_delay_reconfig(vecs))
-				rc = reconfigure(vecs);
-			else
-				enable_delayed_reconfig();
-			lock_cleanup_pop(vecs->lock);
-			if (!rc)
-				post_config_state(DAEMON_IDLE);
-			else {
-				condlog(0, "fatal error applying configuration - aborting");
-				exit_daemon();
-			}
+		/* handle DAEMON_CONFIGURE */
+		pthread_cleanup_push(cleanup_lock, &vecs->lock);
+		lock(&vecs->lock);
+		pthread_testcancel();
+		if (!need_to_delay_reconfig(vecs))
+			rc = reconfigure(vecs);
+		else
+			enable_delayed_reconfig();
+		lock_cleanup_pop(vecs->lock);
+		if (!rc)
+			post_config_state(DAEMON_IDLE);
+		else {
+			condlog(0, "fatal error applying configuration - aborting");
+			exit_daemon();
 		}
 	}
 
Signature · 1 line
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#3 lore raw

PATCH 2/5 multipathd: set reload_type in when calling reconfigure()

In reply to [email protected]

From: Martin Wilck <[email protected]>

Only set reload_type (and reset reconfigure_pending) immediately before we actually call reconfigure(). This allows us to get rid of the reload_type global variable, and makes sure that reconfigure() is called with the reload type that was last requested.

While at it, convert configure() and reconfigure() to static functions.

---
 multipathd/main.c | 35 +++++++++++++++++------------------
 1 file changed, 17 insertions(+), 18 deletions(-)
multipathd/main.c +17 -18
index 1c8839d..7ecf3bd 100644
@@ -287,14 +287,10 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 
 /* Don't access this variable without holding config_lock */
 static enum force_reload_types reconfigure_pending = FORCE_RELOAD_NONE;
-/* Only set while changing to DAEMON_CONFIGURE, and only access while
- * reconfiguring or scheduling a delayed reconfig in DAEMON_CONFIGURE */
-static volatile enum force_reload_types reload_type = FORCE_RELOAD_NONE;
 
 static void enable_delayed_reconfig(void)
 {
 	pthread_mutex_lock(&config_lock);
-	reconfigure_pending = reload_type;
 	__delayed_reconfig = true;
 	pthread_mutex_unlock(&config_lock);
 }
@@ -324,11 +320,6 @@ static void __post_config_state(enum daemon_status state)
 			old_state = DAEMON_IDLE;
 			state = DAEMON_CONFIGURE;
 		}
-		if (state == DAEMON_CONFIGURE) {
-			reload_type = (reconfigure_pending == FORCE_RELOAD_YES) ? FORCE_RELOAD_YES : FORCE_RELOAD_WEAK;
-			reconfigure_pending = FORCE_RELOAD_NONE;
-			__delayed_reconfig = false;
-		}
 		running_state = state;
 		pthread_cond_broadcast(&config_cond);
 		do_sd_notify(old_state, state);
@@ -2714,8 +2705,8 @@ checkerloop (void *ap)
 	return NULL;
 }
 
-int
-configure (struct vectors * vecs)
+static int
+configure (struct vectors * vecs, enum force_reload_types reload_type)
 {
 	struct multipath * mpp;
 	struct path * pp;
@@ -2846,8 +2837,8 @@ void rcu_free_config(struct rcu_head *head)
 	free_config(conf);
 }
 
-int
-reconfigure (struct vectors * vecs)
+static int
+reconfigure (struct vectors *vecs, enum force_reload_types reload_type)
 {
 	struct config * old, *conf;
 	int old_marginal_pathgroups;
@@ -2894,8 +2885,7 @@ reconfigure (struct vectors * vecs)
 #ifdef FPIN_EVENT_HANDLER
 	fpin_clean_marginal_dev_list(NULL);
 #endif
-	configure(vecs);
-
+	configure(vecs, reload_type);
 
 	return 0;
 }
@@ -3411,9 +3401,18 @@ child (__attribute__((unused)) void *param)
 		pthread_cleanup_push(cleanup_lock, &vecs->lock);
 		lock(&vecs->lock);
 		pthread_testcancel();
-		if (!need_to_delay_reconfig(vecs))
-			rc = reconfigure(vecs);
-		else
+		if (!need_to_delay_reconfig(vecs)) {
+			enum force_reload_types reload_type;
+
+			pthread_mutex_lock(&config_lock);
+			reload_type = reconfigure_pending == FORCE_RELOAD_YES ?
+				FORCE_RELOAD_YES : FORCE_RELOAD_WEAK;
+			reconfigure_pending = FORCE_RELOAD_NONE;
+			__delayed_reconfig = false;
+			pthread_mutex_unlock(&config_lock);
+
+			rc = reconfigure(vecs, reload_type);
+		} else
 			enable_delayed_reconfig();
 		lock_cleanup_pop(vecs->lock);
 		if (!rc)
Signature · 6 lines
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#4 lore raw

PATCH 3/5 multipathd: avoid busy loop in child() for delayed reconfigure

In reply to [email protected]

From: Martin Wilck <[email protected]>

If need_to_delay_reconfig() returned true, the logic introduced by 250708c ("multipathd: improve delayed reconfigure") and 4b104e6 ("multipathd: pass in the type of reconfigure") could cause child() to run in a tight loop, switching back and forth between DAEMON_CONFIGURE and DAEMON_IDLE states without actually calling reconfigure().

Move the logic to postpone reconfigure() calls from __post_config_state() into child(), entirely, to avoid this situation. This means that child() needs to poll for reconfigure_pending besides running_state changes.

Fixes 250708c ("multipathd: improve delayed reconfigure")
Fixes 4b104e6 ("multipathd: pass in the type of reconfigure")
---
 multipathd/main.c | 48 +++++++++++++++++++----------------------------
 1 file changed, 19 insertions(+), 29 deletions(-)
multipathd/main.c +19 -29
index 7ecf3bd..d033a9a 100644
@@ -288,38 +288,12 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 /* Don't access this variable without holding config_lock */
 static enum force_reload_types reconfigure_pending = FORCE_RELOAD_NONE;
 
-static void enable_delayed_reconfig(void)
-{
-	pthread_mutex_lock(&config_lock);
-	__delayed_reconfig = true;
-	pthread_mutex_unlock(&config_lock);
-}
-
 /* must be called with config_lock held */
 static void __post_config_state(enum daemon_status state)
 {
 	if (state != running_state && running_state != DAEMON_SHUTDOWN) {
 		enum daemon_status old_state = running_state;
 
-		/*
-		 * Handle a pending reconfigure request.
-		 * DAEMON_IDLE is set from child() after reconfigure(),
-		 * or from checkerloop() after completing checkers.
-		 * In either case, child() will see DAEMON_CONFIGURE
-		 * again and start another reconfigure cycle.
-		 */
-		if (reconfigure_pending != FORCE_RELOAD_NONE &&
-		    state == DAEMON_IDLE &&
-		    (old_state == DAEMON_CONFIGURE ||
-		     old_state == DAEMON_RUNNING)) {
-			/*
-			 * notify systemd of transient idle state, lest systemd
-			 * thinks the reload lasts forever.
-			 */
-			do_sd_notify(old_state, DAEMON_IDLE);
-			old_state = DAEMON_IDLE;
-			state = DAEMON_CONFIGURE;
-		}
 		running_state = state;
 		pthread_cond_broadcast(&config_cond);
 		do_sd_notify(old_state, state);
@@ -3390,8 +3364,21 @@ child (__attribute__((unused)) void *param)
 		pthread_cleanup_push(config_cleanup, NULL);
 		pthread_mutex_lock(&config_lock);
 		while (running_state != DAEMON_CONFIGURE &&
-		       running_state != DAEMON_SHUTDOWN)
+		       running_state != DAEMON_SHUTDOWN &&
+		       /*
+			* Check if another reconfigure request was scheduled
+			* while we last ran reconfigure().
+			* We have to test __delayed_reconfig here
+			* to avoid a busy loop
+			*/
+		       (reconfigure_pending == FORCE_RELOAD_NONE
+			 || __delayed_reconfig))
 			pthread_cond_wait(&config_cond, &config_lock);
+
+		if (running_state != DAEMON_CONFIGURE &&
+		    running_state != DAEMON_SHUTDOWN)
+			/* This sets running_state to DAEMON_CONFIGURE */
+			__post_config_state(DAEMON_CONFIGURE);
 		state = running_state;
 		pthread_cleanup_pop(1);
 		if (state == DAEMON_SHUTDOWN)
@@ -3412,8 +3399,11 @@ child (__attribute__((unused)) void *param)
 			pthread_mutex_unlock(&config_lock);
 
 			rc = reconfigure(vecs, reload_type);
-		} else
-			enable_delayed_reconfig();
+		} else {
+			pthread_mutex_lock(&config_lock);
+			__delayed_reconfig = true;
+			pthread_mutex_unlock(&config_lock);
+		}
 		lock_cleanup_pop(vecs->lock);
 		if (!rc)
 			post_config_state(DAEMON_IDLE);
Signature · 6 lines
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
maintainer Reviewed-by on v1 3/5
#5 lore raw

Re: [dm-devel] [PATCH 3/5] multipathd: avoid busy loop in child() for delayed reconfigure

Quoted 9 lines · On Mon, Mar 14, 2022 at 10:30:34PM +0100, [email protected] wrote:

From: Martin Wilck <[email protected]>

If need_to_delay_reconfig() returned true, the logic introduced by 250708c ("multipathd: improve delayed reconfigure") and 4b104e6 ("multipathd: pass in the type of reconfigure") could cause child() to run in a tight loop, switching back and forth between DAEMON_CONFIGURE and DAEMON_IDLE states without actually calling reconfigure().

Quoted 4 lines

Move the logic to postpone reconfigure() calls from __post_config_state() into child(), entirely, to avoid this situation. This means that child() needs to poll for reconfigure_pending besides running_state changes.

With the understanding that this fix still leaves a bug that needs to be resolved.

Quoted 89 lines
Fixes 250708c ("multipathd: improve delayed reconfigure")
Fixes 4b104e6 ("multipathd: pass in the type of reconfigure")
---
 multipathd/main.c | 48 +++++++++++++++++++----------------------------
 1 file changed, 19 insertions(+), 29 deletions(-)
multipathd/main.c +19 -29
index 7ecf3bd..d033a9a 100644
@@ -288,38 +288,12 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 /* Don't access this variable without holding config_lock */
 static enum force_reload_types reconfigure_pending = FORCE_RELOAD_NONE;
 
-static void enable_delayed_reconfig(void)
-{
-	pthread_mutex_lock(&config_lock);
-	__delayed_reconfig = true;
-	pthread_mutex_unlock(&config_lock);
-}
-
 /* must be called with config_lock held */
 static void __post_config_state(enum daemon_status state)
 {
 	if (state != running_state && running_state != DAEMON_SHUTDOWN) {
 		enum daemon_status old_state = running_state;
 
-		/*
-		 * Handle a pending reconfigure request.
-		 * DAEMON_IDLE is set from child() after reconfigure(),
-		 * or from checkerloop() after completing checkers.
-		 * In either case, child() will see DAEMON_CONFIGURE
-		 * again and start another reconfigure cycle.
-		 */
-		if (reconfigure_pending != FORCE_RELOAD_NONE &&
-		    state == DAEMON_IDLE &&
-		    (old_state == DAEMON_CONFIGURE ||
-		     old_state == DAEMON_RUNNING)) {
-			/*
-			 * notify systemd of transient idle state, lest systemd
-			 * thinks the reload lasts forever.
-			 */
-			do_sd_notify(old_state, DAEMON_IDLE);
-			old_state = DAEMON_IDLE;
-			state = DAEMON_CONFIGURE;
-		}
 		running_state = state;
 		pthread_cond_broadcast(&config_cond);
 		do_sd_notify(old_state, state);
@@ -3390,8 +3364,21 @@ child (__attribute__((unused)) void *param)
 		pthread_cleanup_push(config_cleanup, NULL);
 		pthread_mutex_lock(&config_lock);
 		while (running_state != DAEMON_CONFIGURE &&
-		       running_state != DAEMON_SHUTDOWN)
+		       running_state != DAEMON_SHUTDOWN &&
+		       /*
+			* Check if another reconfigure request was scheduled
+			* while we last ran reconfigure().
+			* We have to test __delayed_reconfig here
+			* to avoid a busy loop
+			*/
+		       (reconfigure_pending == FORCE_RELOAD_NONE
+			 || __delayed_reconfig))
 			pthread_cond_wait(&config_cond, &config_lock);
+
+		if (running_state != DAEMON_CONFIGURE &&
+		    running_state != DAEMON_SHUTDOWN)
+			/* This sets running_state to DAEMON_CONFIGURE */
+			__post_config_state(DAEMON_CONFIGURE);
 		state = running_state;
 		pthread_cleanup_pop(1);
 		if (state == DAEMON_SHUTDOWN)
@@ -3412,8 +3399,11 @@ child (__attribute__((unused)) void *param)
 			pthread_mutex_unlock(&config_lock);
 
 			rc = reconfigure(vecs, reload_type);
-		} else
-			enable_delayed_reconfig();
+		} else {
+			pthread_mutex_lock(&config_lock);
+			__delayed_reconfig = true;
+			pthread_mutex_unlock(&config_lock);
+		}
 		lock_cleanup_pop(vecs->lock);
 		if (!rc)
 			post_config_state(DAEMON_IDLE);
Signature · 1 line
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#6 lore raw

PATCH 4/5 multipathd: reset __delayed_reconfig from ev_add_map()

In reply to [email protected]

From: Martin Wilck <[email protected]>

With the previous patch, one race condition between child() and the uevent handler (ev_add_map()) remains: 1. child() sets __delayed_reconfig, 2. ev_add_map() calls schedule_reconfigure() and thus DAEMON_CONFIGURE, 3. child() sets DAEMON_IDLE. This would cause the pending reconfigure request to be missed.

To fix this, reset __delayed_reconfig immediately from ev_add_map() (and likewise, missing_uev_wait_tick()). This way the wait loop in child() terminates on the reconfigure_pending condition.

Also, these schedule_reconfigure() callers don't really request a reconfigure() call; they just unblock processing previously requested reconfigure() calls. Add a new helper, unblock_reconfigure(), that does just that.

---
 multipathd/main.c | 41 ++++++++++++++++++++---------------------
 1 file changed, 20 insertions(+), 21 deletions(-)
multipathd/main.c +20 -21
index d033a9a..2424db7 100644
@@ -155,16 +155,6 @@ int should_exit(void)
 	return get_running_state() == DAEMON_SHUTDOWN;
 }
 
-static bool get_delayed_reconfig(void)
-{
-	bool val;
-
-	pthread_mutex_lock(&config_lock);
-	val = __delayed_reconfig;
-	pthread_mutex_unlock(&config_lock);
-	return val;
-}
-
 /*
  * global copy of vecs for use in sig handlers
  */
@@ -308,6 +298,22 @@ void post_config_state(enum daemon_status state)
 	pthread_cleanup_pop(1);
 }
 
+static bool unblock_reconfigure(void)
+{
+	bool was_delayed;
+
+	pthread_mutex_lock(&config_lock);
+	was_delayed = __delayed_reconfig;
+	if (was_delayed) {
+		__delayed_reconfig = false;
+		__post_config_state(DAEMON_CONFIGURE);
+	}
+	pthread_mutex_unlock(&config_lock);
+	if (was_delayed)
+		condlog(2, "reconfigure (delayed)");
+	return was_delayed;
+}
+
 void schedule_reconfigure(enum force_reload_types requested_type)
 {
 	pthread_mutex_lock(&config_lock);
@@ -790,12 +796,9 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)
 		dm_get_info(mpp->alias, &mpp->dmi);
 		if (mpp->wait_for_udev) {
 			mpp->wait_for_udev = 0;
-			if (get_delayed_reconfig() &&
-			    !need_to_delay_reconfig(vecs)) {
-				condlog(2, "reconfigure (delayed)");
-				schedule_reconfigure(FORCE_RELOAD_WEAK);
+			if (!need_to_delay_reconfig(vecs) &&
+			    unblock_reconfigure())
 				return 0;
-			}
 		}
 		/*
 		 * Not really an error -- we generate our own uevent
@@ -1899,11 +1902,8 @@ missing_uev_wait_tick(struct vectors *vecs)
 		}
 	}
 
-	if (timed_out && get_delayed_reconfig() &&
-	    !need_to_delay_reconfig(vecs)) {
-		condlog(2, "reconfigure (delayed)");
-		schedule_reconfigure(FORCE_RELOAD_WEAK);
-	}
+	if (timed_out && !need_to_delay_reconfig(vecs))
+		unblock_reconfigure();
 }
 
 static void
@@ -3395,7 +3395,6 @@ child (__attribute__((unused)) void *param)
 			reload_type = reconfigure_pending == FORCE_RELOAD_YES ?
 				FORCE_RELOAD_YES : FORCE_RELOAD_WEAK;
 			reconfigure_pending = FORCE_RELOAD_NONE;
-			__delayed_reconfig = false;
 			pthread_mutex_unlock(&config_lock);
 
 			rc = reconfigure(vecs, reload_type);
Signature · 6 lines
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
#7 lore raw

Re: [dm-devel] [PATCH 4/5] multipathd: reset __delayed_reconfig from ev_add_map()

Quoted 8 lines · On Mon, Mar 14, 2022 at 10:30:35PM +0100, [email protected] wrote:

From: Martin Wilck <[email protected]>

With the previous patch, one race condition between child() and the uevent handler (ev_add_map()) remains: 1. child() sets __delayed_reconfig, 2. ev_add_map() calls schedule_reconfigure() and thus DAEMON_CONFIGURE, 3. child() sets DAEMON_IDLE. This would cause the pending reconfigure request to be missed.

Quoted 9 lines

To fix this, reset __delayed_reconfig immediately from ev_add_map() (and likewise, missing_uev_wait_tick()). This way the wait loop in child() terminates on the reconfigure_pending condition.

Also, these schedule_reconfigure() callers don't really request a reconfigure() call; they just unblock processing previously requested reconfigure() calls. Add a new helper, unblock_reconfigure(), that does just that.

Doesn't unblock_reconfigure() allow us to set the state to DAEMON_CONFIGURE, right after the checkerloop() has set it to DAEMON_RUNNING. schedule_reconfigure() didn't update the state if it was DAEMON_RUNNING. Instead, it would wait till checkerloop() set the state back to DAEMON_IDLE for the reconfigure to start.

I have also come up with a different bug that effects these fixes, including mine. But it will effect this fix worse. This fix assumes that when a reconfigure is delayed, it should remain delayed until either a change event happens on the multipath device (ev_add_map) or multipathd times out waiting for one (missing_uev_wait_tick). However the map could be removed before either of those things happen. It's possible that multipathd could get a remove uevent after the add event but before the change event. Either that or a multipathd client command could remove it, or a dm event could happen either removing the device, or updating it, but with __setup_multipath() removing it. Those are just the examples I came up with off the top of my head.

So there's a specific problem where we don't handle clearing __delayed_reconfig if a map was removed. That can be handled in a different patch. But this, or any other oversight we might have here can be mitigated by being more proactive in removing __delayed_reconfig. For instance, when then main thread tries to reconfigure, it checks need_to_delay_reconfig(), and only does the reconfigure if this returns that it's ok. __delayed_reconfig mostly exists so that the main thread doesn't need to grab the vecs lock and loop through the the multipath devices to know if it need to delay, but need_to_delay_reconfig() is the definitive answer. When the main thread checks this, we should be updating __delayed_reconfig to match the result. This patch removes the "__delayed_reconfig = false;" line, which I think is a mistake.

Imagine if, because a map got removed before ev_add_map() was called, __delayed_reconfig was true, even though there were no maps with mpp->wait_for_udev set. Any existing config request would still be delayed, but if another reconfigure was requested not only would it happen, but if the "__delayed_reconfig = false;" line was back, __delayed_reconfig would go back to the correct value.

-Ben

Quoted 89 lines
---
 multipathd/main.c | 41 ++++++++++++++++++++---------------------
 1 file changed, 20 insertions(+), 21 deletions(-)
multipathd/main.c +20 -21
index d033a9a..2424db7 100644
@@ -155,16 +155,6 @@ int should_exit(void)
 	return get_running_state() == DAEMON_SHUTDOWN;
 }
 
-static bool get_delayed_reconfig(void)
-{
-	bool val;
-
-	pthread_mutex_lock(&config_lock);
-	val = __delayed_reconfig;
-	pthread_mutex_unlock(&config_lock);
-	return val;
-}
-
 /*
  * global copy of vecs for use in sig handlers
  */
@@ -308,6 +298,22 @@ void post_config_state(enum daemon_status state)
 	pthread_cleanup_pop(1);
 }
 
+static bool unblock_reconfigure(void)
+{
+	bool was_delayed;
+
+	pthread_mutex_lock(&config_lock);
+	was_delayed = __delayed_reconfig;
+	if (was_delayed) {
+		__delayed_reconfig = false;
+		__post_config_state(DAEMON_CONFIGURE);
+	}
+	pthread_mutex_unlock(&config_lock);
+	if (was_delayed)
+		condlog(2, "reconfigure (delayed)");
+	return was_delayed;
+}
+
 void schedule_reconfigure(enum force_reload_types requested_type)
 {
 	pthread_mutex_lock(&config_lock);
@@ -790,12 +796,9 @@ ev_add_map (char * dev, const char * alias, struct vectors * vecs)
 		dm_get_info(mpp->alias, &mpp->dmi);
 		if (mpp->wait_for_udev) {
 			mpp->wait_for_udev = 0;
-			if (get_delayed_reconfig() &&
-			    !need_to_delay_reconfig(vecs)) {
-				condlog(2, "reconfigure (delayed)");
-				schedule_reconfigure(FORCE_RELOAD_WEAK);
+			if (!need_to_delay_reconfig(vecs) &&
+			    unblock_reconfigure())
 				return 0;
-			}
 		}
 		/*
 		 * Not really an error -- we generate our own uevent
@@ -1899,11 +1902,8 @@ missing_uev_wait_tick(struct vectors *vecs)
 		}
 	}
 
-	if (timed_out && get_delayed_reconfig() &&
-	    !need_to_delay_reconfig(vecs)) {
-		condlog(2, "reconfigure (delayed)");
-		schedule_reconfigure(FORCE_RELOAD_WEAK);
-	}
+	if (timed_out && !need_to_delay_reconfig(vecs))
+		unblock_reconfigure();
 }
 
 static void
@@ -3395,7 +3395,6 @@ child (__attribute__((unused)) void *param)
 			reload_type = reconfigure_pending == FORCE_RELOAD_YES ?
 				FORCE_RELOAD_YES : FORCE_RELOAD_WEAK;
 			reconfigure_pending = FORCE_RELOAD_NONE;
-			__delayed_reconfig = false;
 			pthread_mutex_unlock(&config_lock);
 
 			rc = reconfigure(vecs, reload_type);
Signature · 1 line
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#8 lore raw

Re: [dm-devel] [PATCH 4/5] multipathd: reset __delayed_reconfig from ev_add_map()

Quoted 22 lines · Benjamin Marzinski
Quoted 14 lines · On Mon, Mar 14, 2022 at 10:30:35PM +0100, [email protected] wrote:

From: Martin Wilck <[email protected]>

With the previous patch, one race condition between child() and the uevent handler (ev_add_map()) remains: 1. child() sets __delayed_reconfig, 2. ev_add_map() calls schedule_reconfigure() and
thus DAEMON_CONFIGURE, 3. child() sets DAEMON_IDLE. This would cause
the pending reconfigure request to be missed.

To fix this, reset __delayed_reconfig immediately from ev_add_map() (and likewise, missing_uev_wait_tick()). This way the wait loop in child() terminates on the reconfigure_pending condition.

Quoted 6 lines

Also, these schedule_reconfigure() callers don't really request a reconfigure() call; they just unblock processing previously requested
reconfigure() calls. Add a new helper, unblock_reconfigure(), that does just that.

Quoted 3 lines

Doesn't unblock_reconfigure() allow us to set the state to DAEMON_CONFIGURE, right after the checkerloop() has set it to DAEMON_RUNNING.

Right. I added the __post_config_state() there in order to be sure that child() would be woken up. But unblock_reconfigure() should only call __post_config_state if running_state is DAEMON_IDLE, like schedule_reconfigure(). Otherwise, it can be sure that some other process will wake up child().

Quoted 24 lines

schedule_reconfigure() didn't update the state if it was
DAEMON_RUNNING. Instead, it would wait till checkerloop() set the state
back to DAEMON_IDLE for the reconfigure to start.

I have also come up with a different bug that effects these fixes, including mine. But it will effect this fix worse. This fix assumes that
when a reconfigure is delayed, it should remain delayed until either a
change event happens on the multipath device (ev_add_map) or multipathd
times out waiting for one (missing_uev_wait_tick). However the map could
be removed before either of those things happen.  It's possible that multipathd could get a remove uevent after the add event but before the
change event. Either that or a multipathd client command could remove it, or a dm event could happen either removing the device, or updating
it, but with __setup_multipath() removing it. Those are just the examples I came up with off the top of my head.

Quoted 16 lines

So there's a specific problem where we don't handle clearing __delayed_reconfig if a map was removed. That can be handled in a different patch.  But this, or any other oversight we might have here can be mitigated by being more proactive in removing __delayed_reconfig.
For instance, when then main thread tries to reconfigure, it checks need_to_delay_reconfig(), and only does the reconfigure if this returns
that it's ok.  __delayed_reconfig mostly exists so that the main thread
doesn't need to grab the vecs lock and loop through the the multipath devices to know if it need to delay, but need_to_delay_reconfig() is the definitive answer. When the main thread checks this, we should be updating __delayed_reconfig to match the result. This patch removes the
"__delayed_reconfig = false;" line, which I think is a mistake.

Agreed. My thinking was that this is done by the uevent handler when the state of need_to_delay_reconfig() changes. But you're right with the "remove" special case. We should re-add this, it definitely doesn't hurt.

Quoted 6 lines

Imagine if, because a map got removed before ev_add_map() was called, __delayed_reconfig was true, even though there were no maps with mpp->wait_for_udev set. Any existing config request would still be delayed, but if another reconfigure was requested not only would it happen, but if the "__delayed_reconfig = false;" line was back, __delayed_reconfig would go back to the correct value.

So we should call unblock_reconfigure() from remove_map(). Unfortunately that requires another callback from libmultipath into multipathd. Trying to figure it out...

Thanks,
Martin

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel

#9 lore raw

PATCH 5/5 multipathd: remove volatile qualifier from running_state

In reply to [email protected]

From: Martin Wilck <[email protected]>

running_state is only accessed while holding the config_lock, the volatile qualifier is superfluous.

---
 multipathd/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
multipathd/main.c +1 -1
index 2424db7..b47a457 100644
@@ -127,7 +127,7 @@ static int poll_dmevents = 0;
 static int poll_dmevents = 1;
 #endif
 /* Don't access this variable without holding config_lock */
-static volatile enum daemon_status running_state = DAEMON_INIT;
+static enum daemon_status running_state = DAEMON_INIT;
 /* Don't access this variable without holding config_lock */
 static bool __delayed_reconfig;
 pid_t daemon_pid;
Signature · 6 lines
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel
maintainer Reviewed-by on v1 5/5
#10 lore raw

Re: [dm-devel] [PATCH 5/5] multipathd: remove volatile qualifier from running_state

Quoted 1 line · On Mon, Mar 14, 2022 at 10:30:36PM +0100, [email protected] wrote:

From: Martin Wilck <[email protected]>

Quoted 25 lines

running_state is only accessed while holding the config_lock, the volatile qualifier is superfluous.

---
 multipathd/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
multipathd/main.c +1 -1
index 2424db7..b47a457 100644
@@ -127,7 +127,7 @@ static int poll_dmevents = 0;
 static int poll_dmevents = 1;
 #endif
 /* Don't access this variable without holding config_lock */
-static volatile enum daemon_status running_state = DAEMON_INIT;
+static enum daemon_status running_state = DAEMON_INIT;
 /* Don't access this variable without holding config_lock */
 static bool __delayed_reconfig;
 pid_t daemon_pid;
Signature · 1 line
2.35.1

--
dm-devel mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/dm-devel