Skip to contentSkip to conversation

Possibly lost multipath-tools patches

PATCH RESEND In review Martin Wilck 1 participant 5 messages Apr 11, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Possibly lost multipath-tools patches

Hello Christophe,

lest they be forgotten, the patches in this series have been previously submitted to dm-devel, and got positive review. Please apply at your convenience. Ben's latest series "[PATCH v3 00/17] Misc Multipath patches" is IMO also ready for merging.

Patches 2/3 and 3/3 were part of my "[RFC PATCH 0/6] multipath-tools: Fix remaining shutdown delay issues" series in January. These two got acked by Ben. They are independent of the rest of the series, which still needs more work. I made a small change to patch 3/3: the original patch had renamed "running_state" to "_running_state" to make the compiler catch all accesses to this variable. After fixing all references, I believe that the renaming isn't necessary any more, and not doing it improves the readability of the patch.

Regards
Martin

Chongyun Wu (1):
multipathd: fix daemon not really shutdown

Dominique Leuenberger (1):
multipath-tools: Build: properly parse systemd's version

Martin Wilck (1):
multipathd: protect all access to running_state

 Makefile.inc      |  9 +++++--
 multipathd/main.c | 60 ++++++++++++++++++++++++++++++-----------------
 2 files changed, 46 insertions(+), 23 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH RESEND 1/3 multipath-tools: Build: properly parse systemd's version

From: Dominique Leuenberger <[email protected]>

Since systemd 241, systemctl --version no longer 'just' prints out the version, but gives more information like git commit ref and whatnot. In it's shortest form, it gives something like "systemd 241 (241)", which when passed as parameter "-DUSE_SYSTEMD=241 (241)" results in shell errors.

Try to retrieve the version from pkg-config instead, and if that fails, discard anything after the first number in "systemctl --version" output.

---
 Makefile.inc | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Makefile.inc +7 -2
index fc728ca9..56c3eda0 100644
@@ -36,8 +36,13 @@ ifndef RUN
 endif
 
 ifndef SYSTEMD
-	ifeq ($(shell systemctl --version > /dev/null 2>&1 && echo 1), 1)
-		SYSTEMD = $(shell systemctl --version 2> /dev/null |  sed -n 's/systemd \([0-9]*\)/\1/p')
+	ifeq ($(shell pkg-config --modversion libsystemd >/dev/null 2>&1 && echo 1), 1)
+		SYSTEMD = $(shell pkg-config --modversion libsystemd)
+	else
+		ifeq ($(shell systemctl --version >/dev/null 2>&1 && echo 1), 1)
+			SYSTEMD = $(shell systemctl --version 2> /dev/null | \
+				sed -n 's/systemd \([0-9]*\).*/\1/p')
+		endif
 	endif
 endif
 
Signature · 1 line
2.21.0
#2 lore raw

PATCH RESEND 2/3 multipathd: fix daemon not really shutdown

In reply to Martin Wilck

From: Chongyun Wu <[email protected]>

Test environment: 25 hosts, each host have more than 100 luns, each lun have two paths.
Some times when we try to ceate new multipath will encounter "could not create uxsock:98" but the multipathd still running not shutdown and can't response any multipathd commands also.

After reproduce this issue and debug, found below fixes might work:
(1) set_config_state() after pthread_cond_timedwait() other threads might changed the running_state from DAEMON_SHUTDOWN to other status like DAEMON_IDLE, which will make the shutdown process stopped. I found logs to prove this really happened, so we need add judgement here too.

(2) [this part removed by mwilck]

---
 multipathd/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
multipathd/main.c +1 -1
index 1413c6d6..47aff2d8 100644
@@ -249,7 +249,7 @@ int set_config_state(enum daemon_status state)
 							    &config_lock, &ts);
 			}
 		}
-		if (!rc) {
+		if (!rc && (running_state != DAEMON_SHUTDOWN)) {
 			running_state = state;
 			pthread_cond_broadcast(&config_cond);
 #ifdef USE_SYSTEMD
Signature · 1 line
2.21.0
#3 lore raw

PATCH RESEND 3/3 multipathd: protect all access to running_state

In reply to Martin Wilck

Chonyun Wu's latest patch has shown that the handling of the daemon state variable running_state is racy and difficult to get right. It's not a good candidate for a "benign race" annotation. So, as a first step to sanitizing it, make sure all accesses to the state variable are protected by config_lock.

The patch also replaces "if" with "while" in several places where the code was supposed to wait until a certain state was reached. It's important that DAEMON_SHUTDOWN terminates all loops of this kind.

---
 multipathd/main.c | 58 +++++++++++++++++++++++++++++++----------------
 1 file changed, 38 insertions(+), 20 deletions(-)
multipathd/main.c +38 -20
index 47aff2d8..c8a8529c 100644
@@ -127,11 +127,22 @@ int poll_dmevents = 0;
 #else
 int poll_dmevents = 1;
 #endif
+/* Don't access this variable without holding config_lock */
 enum daemon_status running_state = DAEMON_INIT;
 pid_t daemon_pid;
 pthread_mutex_t config_lock = PTHREAD_MUTEX_INITIALIZER;
 pthread_cond_t config_cond;
 
+static inline enum daemon_status get_running_state(void)
+{
+	enum daemon_status st;
+
+	pthread_mutex_lock(&config_lock);
+	st = running_state;
+	pthread_mutex_unlock(&config_lock);
+	return st;
+}
+
 /*
  * global copy of vecs for use in sig handlers
  */
@@ -149,7 +160,7 @@ static volatile sig_atomic_t log_reset_sig;
 const char *
 daemon_status(void)
 {
-	switch (running_state) {
+	switch (get_running_state()) {
 	case DAEMON_INIT:
 		return "init";
 	case DAEMON_START:
@@ -169,10 +180,10 @@ daemon_status(void)
 /*
  * I love you too, systemd ...
  */
-const char *
-sd_notify_status(void)
+static const char *
+sd_notify_status(enum daemon_status state)
 {
-	switch (running_state) {
+	switch (state) {
 	case DAEMON_INIT:
 		return "STATUS=init";
 	case DAEMON_START:
@@ -189,17 +200,18 @@ sd_notify_status(void)
 }
 
 #ifdef USE_SYSTEMD
-static void do_sd_notify(enum daemon_status old_state)
+static void do_sd_notify(enum daemon_status old_state,
+			 enum daemon_status new_state)
 {
 	/*
 	 * Checkerloop switches back and forth between idle and running state.
 	 * No need to tell systemd each time.
 	 * These notifications cause a lot of overhead on dbus.
 	 */
-	if ((running_state == DAEMON_IDLE || running_state == DAEMON_RUNNING) &&
+	if ((new_state == DAEMON_IDLE || new_state == DAEMON_RUNNING) &&
 	    (old_state == DAEMON_IDLE || old_state == DAEMON_RUNNING))
 		return;
-	sd_notify(0, sd_notify_status());
+	sd_notify(0, sd_notify_status(new_state));
 }
 #endif
 
@@ -208,6 +220,7 @@ static void config_cleanup(void *arg)
 	pthread_mutex_unlock(&config_lock);
 }
 
+/* must be called with config_lock held */
 static void __post_config_state(enum daemon_status state)
 {
 	if (state != running_state && running_state != DAEMON_SHUTDOWN) {
@@ -216,7 +229,7 @@ static void __post_config_state(enum daemon_status state)
 		running_state = state;
 		pthread_cond_broadcast(&config_cond);
 #ifdef USE_SYSTEMD
-		do_sd_notify(old_state);
+		do_sd_notify(old_state, state);
 #endif
 	}
 }
@@ -253,7 +266,7 @@ int set_config_state(enum daemon_status state)
 			running_state = state;
 			pthread_cond_broadcast(&config_cond);
 #ifdef USE_SYSTEMD
-			do_sd_notify(old_state);
+			do_sd_notify(old_state, state);
 #endif
 		}
 	}
@@ -1397,17 +1410,20 @@ uev_trigger (struct uevent * uev, void * trigger_data)
 	int r = 0;
 	struct vectors * vecs;
 	struct uevent *merge_uev, *tmp;
+	enum daemon_status state;
 
 	vecs = (struct vectors *)trigger_data;
 
 	pthread_cleanup_push(config_cleanup, NULL);
 	pthread_mutex_lock(&config_lock);
-	if (running_state != DAEMON_IDLE &&
-	    running_state != DAEMON_RUNNING)
+	while (running_state != DAEMON_IDLE &&
+	       running_state != DAEMON_RUNNING &&
+	       running_state != DAEMON_SHUTDOWN)
 		pthread_cond_wait(&config_cond, &config_lock);
+	state = running_state;
 	pthread_cleanup_pop(1);
 
-	if (running_state == DAEMON_SHUTDOWN)
+	if (state == DAEMON_SHUTDOWN)
 		return 0;
 
 	/*
@@ -2755,6 +2771,7 @@ child (void * param)
 	struct config *conf;
 	char *envp;
 	int queue_without_daemon;
+	enum daemon_status state;
 
 	mlockall(MCL_CURRENT | MCL_FUTURE);
 	signal_init();
@@ -2852,6 +2869,7 @@ child (void * param)
 		/* Wait for uxlsnr startup */
 		while (running_state == DAEMON_IDLE)
 			pthread_cond_wait(&config_cond, &config_lock);
+		state = running_state;
 	}
 	pthread_cleanup_pop(1);
 
@@ -2859,7 +2877,7 @@ child (void * param)
 		condlog(0, "failed to create cli listener: %d", rc);
 		goto failed;
 	}
-	else if (running_state != DAEMON_CONFIGURE) {
+	else if (state != DAEMON_CONFIGURE) {
 		condlog(0, "cli listener failed to start");
 		goto failed;
 	}
@@ -2899,15 +2917,17 @@ child (void * param)
 	}
 	pthread_attr_destroy(&misc_attr);
 
-	while (running_state != DAEMON_SHUTDOWN) {
+	while (1) {
 		pthread_cleanup_push(config_cleanup, NULL);
 		pthread_mutex_lock(&config_lock);
-		if (running_state != DAEMON_CONFIGURE &&
-		    running_state != DAEMON_SHUTDOWN) {
+		while (running_state != DAEMON_CONFIGURE &&
+		       running_state != DAEMON_SHUTDOWN)
 			pthread_cond_wait(&config_cond, &config_lock);
-		}
+		state = running_state;
 		pthread_cleanup_pop(1);
-		if (running_state == DAEMON_CONFIGURE) {
+		if (state == DAEMON_SHUTDOWN)
+			break;
+		if (state == DAEMON_CONFIGURE) {
 			pthread_cleanup_push(cleanup_lock, &vecs->lock);
 			lock(&vecs->lock);
 			pthread_testcancel();
@@ -3077,8 +3097,6 @@ main (int argc, char *argv[])
 
 	ANNOTATE_BENIGN_RACE_SIZED(&multipath_conf, sizeof(multipath_conf),
 				   "Manipulated through RCU");
-	ANNOTATE_BENIGN_RACE_SIZED(&running_state, sizeof(running_state),
-		"Suppress complaints about unprotected running_state reads");
 	ANNOTATE_BENIGN_RACE_SIZED(&uxsock_timeout, sizeof(uxsock_timeout),
 		"Suppress complaints about this scalar variable");
 
Signature · 1 line
2.21.0
#4 lore raw
In reply to Martin Wilck

Hi Christophe,

Quoted 6 lines · Martin Wilck

lest they be forgotten, the patches in this series have been previously
submitted to dm-devel, and got positive review. Please apply at your convenience. Ben's latest series "[PATCH v3 00/17] Misc Multipath patches"
is IMO also ready for merging.

I'd like to remind you also of my series "[PATCH v3 00/10] multipath-
tools: avoid I/O during uevent processing" which had been Ack'd by Ben on March 19th:

https://www.redhat.com/archives/dm-devel/2019-March/msg00184.html

As usual, the patches that I consider ready for upstream inclusion can be found here:

https://github.com/openSUSE/multipath-tools/commits/upstream-queue

Regards,
Martin

Signature · 9 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)


--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel