Possibly lost multipath-tools patches
Subsystems (2)
cover Possibly lost multipath-tools patches
Hello Christophe,
lest they be forgotten, the patches in this series have been previously submitted to dm-devel, and got positive review. Please apply at your convenience. Ben's latest series "[PATCH v3 00/17] Misc Multipath patches" is IMO also ready for merging.
Patches 2/3 and 3/3 were part of my "[RFC PATCH 0/6] multipath-tools: Fix remaining shutdown delay issues" series in January. These two got acked by Ben. They are independent of the rest of the series, which still needs more work. I made a small change to patch 3/3: the original patch had renamed "running_state" to "_running_state" to make the compiler catch all accesses to this variable. After fixing all references, I believe that the renaming isn't necessary any more, and not doing it improves the readability of the patch.
Regards
Martin
Chongyun Wu (1):
multipathd: fix daemon not really shutdown
Dominique Leuenberger (1):
multipath-tools: Build: properly parse systemd's version
Martin Wilck (1):
multipathd: protect all access to running_state
Makefile.inc | 9 +++++-- multipathd/main.c | 60 ++++++++++++++++++++++++++++++----------------- 2 files changed, 46 insertions(+), 23 deletions(-)
Signature · 1 line
2.21.0
PATCH RESEND 1/3 multipath-tools: Build: properly parse systemd's version
From: Dominique Leuenberger <[email protected]>
Since systemd 241, systemctl --version no longer 'just' prints out the version, but gives more information like git commit ref and whatnot. In it's shortest form, it gives something like "systemd 241 (241)", which when passed as parameter "-DUSE_SYSTEMD=241 (241)" results in shell errors.
Try to retrieve the version from pkg-config instead, and if that fails, discard anything after the first number in "systemctl --version" output.
--- Makefile.inc | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
Makefile.inc +7 -2
@@ -36,8 +36,13 @@ ifndef RUN endif ifndef SYSTEMD - ifeq ($(shell systemctl --version > /dev/null 2>&1 && echo 1), 1) - SYSTEMD = $(shell systemctl --version 2> /dev/null | sed -n 's/systemd \([0-9]*\)/\1/p') + ifeq ($(shell pkg-config --modversion libsystemd >/dev/null 2>&1 && echo 1), 1) + SYSTEMD = $(shell pkg-config --modversion libsystemd) + else + ifeq ($(shell systemctl --version >/dev/null 2>&1 && echo 1), 1) + SYSTEMD = $(shell systemctl --version 2> /dev/null | \ + sed -n 's/systemd \([0-9]*\).*/\1/p') + endif endif endif
Signature · 1 line
2.21.0
PATCH RESEND 2/3 multipathd: fix daemon not really shutdown
In reply to Martin WilckFrom: Chongyun Wu <[email protected]>
Test environment: 25 hosts, each host have more than 100 luns, each lun have two paths.
Some times when we try to ceate new multipath will encounter "could not create uxsock:98" but the multipathd still running not shutdown and can't response any multipathd commands also.
After reproduce this issue and debug, found below fixes might work:
(1) set_config_state() after pthread_cond_timedwait() other threads might changed the running_state from DAEMON_SHUTDOWN to other status like DAEMON_IDLE, which will make the shutdown process stopped. I found logs to prove this really happened, so we need add judgement here too.
(2) [this part removed by mwilck]
--- multipathd/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
multipathd/main.c +1 -1
@@ -249,7 +249,7 @@ int set_config_state(enum daemon_status state) &config_lock, &ts); } } -if (!rc) { +if (!rc && (running_state !=DAEMON_SHUTDOWN )) { running_state = state;pthread_cond_broadcast (&config_cond);#ifdef USE_SYSTEMD
Signature · 1 line
2.21.0
PATCH RESEND 3/3 multipathd: protect all access to running_state
In reply to Martin WilckChonyun Wu's latest patch has shown that the handling of the daemon state variable running_state is racy and difficult to get right. It's not a good candidate for a "benign race" annotation. So, as a first step to sanitizing it, make sure all accesses to the state variable are protected by config_lock.
The patch also replaces "if" with "while" in several places where the code was supposed to wait until a certain state was reached. It's important that DAEMON_SHUTDOWN terminates all loops of this kind.
--- multipathd/main.c | 58 +++++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 20 deletions(-)
multipathd/main.c +38 -20
@@ -127,11 +127,22 @@ int poll_dmevents = 0;#else int poll_dmevents =1 ;#endif +/* Don't access this variable without holding config_lock */ enum daemon_status running_state =DAEMON_INIT ;pid_t daemon_pid;pthread_mutex_t config_lock =PTHREAD_MUTEX_INITIALIZER ;pthread_cond_t config_cond; +static inline enum daemon_status get_running_state (void ) +{ +enum daemon_status st; + +pthread_mutex_lock (&config_lock); + st = running_state; +pthread_mutex_unlock (&config_lock); +return st; +} +/* * global copy of vecs for use in sig handlers */
@@ -149,7 +160,7 @@ static volatile sig_atomic_t log_reset_sig;const char *daemon_status (void ) { -switch (running_state) { +switch (get_running_state ()) {case DAEMON_INIT :return "init" ;case DAEMON_START :
@@ -169,10 +180,10 @@ daemon_status(void)/* * I love you too, systemd ... */ -const char * -sd_notify_status (void ) +static const char * +sd_notify_status (enum daemon_status state) { -switch (running_state) { +switch (state) {case DAEMON_INIT :return "STATUS=init" ;case DAEMON_START :
@@ -189,17 +200,18 @@ sd_notify_status(void) }#ifdef USE_SYSTEMD -static void do_sd_notify (enum daemon_status old_state) +static void do_sd_notify (enum daemon_status old_state, +enum daemon_status new_state) {/* * Checkerloop switches back and forth between idle and running state. * No need to tell systemd each time. * These notifications cause a lot of overhead on dbus. */ -if ((running_state ==DAEMON_IDLE || running_state ==DAEMON_RUNNING ) && +if ((new_state ==DAEMON_IDLE || new_state ==DAEMON_RUNNING ) && (old_state ==DAEMON_IDLE || old_state ==DAEMON_RUNNING ))return ; -sd_notify (0 ,sd_notify_status ()); +sd_notify (0 ,sd_notify_status (new_state)); }#endif
@@ -208,6 +220,7 @@ static void config_cleanup(void *arg)pthread_mutex_unlock (&config_lock); } +/* must be called with config_lock held */ static void __post_config_state (enum daemon_status state) {if (state != running_state && running_state !=DAEMON_SHUTDOWN ) {
@@ -216,7 +229,7 @@ static void __post_config_state(enum daemon_status state) running_state = state;pthread_cond_broadcast (&config_cond);#ifdef USE_SYSTEMD -do_sd_notify (old_state); +do_sd_notify (old_state, state);#endif } }
@@ -253,7 +266,7 @@ int set_config_state(enum daemon_status state) running_state = state;pthread_cond_broadcast (&config_cond);#ifdef USE_SYSTEMD -do_sd_notify (old_state); +do_sd_notify (old_state, state);#endif } }
@@ -1397,17 +1410,20 @@ uev_trigger (struct uevent * uev, void * trigger_data)int r =0 ;struct vectors * vecs;struct uevent *merge_uev, *tmp; +enum daemon_status state; vecs = (struct vectors *)trigger_data;pthread_cleanup_push (config_cleanup,NULL );pthread_mutex_lock (&config_lock); -if (running_state !=DAEMON_IDLE && - running_state !=DAEMON_RUNNING ) +while (running_state !=DAEMON_IDLE && + running_state !=DAEMON_RUNNING && + running_state !=DAEMON_SHUTDOWN )pthread_cond_wait (&config_cond, &config_lock); + state = running_state;pthread_cleanup_pop (1 ); -if (running_state ==DAEMON_SHUTDOWN ) +if (state ==DAEMON_SHUTDOWN )return 0 ;/*
@@ -2755,6 +2771,7 @@ child (void * param)struct config *conf;char *envp;int queue_without_daemon; +enum daemon_status state;mlockall (MCL_CURRENT |MCL_FUTURE );signal_init ();
@@ -2852,6 +2869,7 @@ child (void * param)/* Wait for uxlsnr startup */ while (running_state ==DAEMON_IDLE )pthread_cond_wait (&config_cond, &config_lock); + state = running_state; }pthread_cleanup_pop (1 );
@@ -2859,7 +2877,7 @@ child (void * param)condlog (0 ,"failed to create cli listener: %d" , rc); goto failed; } -else if (running_state !=DAEMON_CONFIGURE ) { +else if (state !=DAEMON_CONFIGURE ) {condlog (0 ,"cli listener failed to start" ); goto failed; }
@@ -2899,15 +2917,17 @@ child (void * param) }pthread_attr_destroy (&misc_attr); -while (running_state !=DAEMON_SHUTDOWN ) { +while (1 ) {pthread_cleanup_push (config_cleanup,NULL );pthread_mutex_lock (&config_lock); -if (running_state !=DAEMON_CONFIGURE && - running_state !=DAEMON_SHUTDOWN ) { +while (running_state !=DAEMON_CONFIGURE && + running_state !=DAEMON_SHUTDOWN )pthread_cond_wait (&config_cond, &config_lock); - } + state = running_state;pthread_cleanup_pop (1 ); -if (running_state ==DAEMON_CONFIGURE ) { +if (state ==DAEMON_SHUTDOWN ) +break ; +if (state ==DAEMON_CONFIGURE ) {pthread_cleanup_push (cleanup_lock, &vecs->lock);lock (&vecs->lock);pthread_testcancel ();
@@ -3077,8 +3097,6 @@ main (int argc, char *argv[])ANNOTATE_BENIGN_RACE_SIZED (&multipath_conf,sizeof (multipath_conf),"Manipulated through RCU" ); -ANNOTATE_BENIGN_RACE_SIZED (&running_state,sizeof (running_state), -"Suppress complaints about unprotected running_state reads" );ANNOTATE_BENIGN_RACE_SIZED (&uxsock_timeout,sizeof (uxsock_timeout),"Suppress complaints about this scalar variable" );
Signature · 1 line
2.21.0
Hi Christophe,
Quoted 6 lines · Martin Wilck
lest they be forgotten, the patches in this series have been previously
submitted to dm-devel, and got positive review. Please apply at your convenience. Ben's latest series "[PATCH v3 00/17] Misc Multipath patches"
is IMO also ready for merging.
I'd like to remind you also of my series "[PATCH v3 00/10] multipath-
tools: avoid I/O during uevent processing" which had been Ack'd by Ben on March 19th:
https://www.redhat.com/archives/dm-devel/2019-March/msg00184.html
As usual, the patches that I consider ready for upstream inclusion can be found here:
https://github.com/openSUSE/multipath-tools/commits/upstream-queue
Regards,
Martin
Signature · 9 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- dm-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/dm-devel