Skip to contentSkip to conversation

multipath-tools: generic async threads for TUR checker

PATCH Superseded Martin Wilck 3 participants 11 messages Apr 23 – Apr 28 Device mapper

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover multipath-tools: generic async threads for TUR checker

Motivated by the recent submission of an ALUA checker for multipath-tools by Brian Bunker, I am proposing a generic framework for asynchronous checker threads in multipath-tools.

The first patch is a small fix I came up with while testing this. 2/5 is the actual implementation, 3/5 test code, and 4/4 modifies the TUR checker to the new library code. 5/5 is new, it simplifies testing the timeout handling code.

This makes the logic of the TUR checker easier to understand, and should make it easier for Brian to implement the ALUA checker based on the same framework.

Further improvements on top of this are possible. The new TUR code, except for the tur_check() function itself, is pretty generic and would allow abstracting an "async checker" model with just a few changes. This would make it possible to switch also the legacy checkers to an asynchronous mode of operation.

Comments and reviews welcome.
Martin

Changes v1 -> v2 (based on the review by Ben Marzinski):

- The design of the runner code has been changed such that no caller-side references are dropped automatically any more. The caller needs
to call release_runner() explicitly to drop its reference, independent of the runner status. This simplifies the code and allows implementing the MAX_TIMEOUTS logic in tur.c as it used to be. The overall API is not more complex, as release_runner() replaces cancel_runner().
- An additional state RUNNER_DEAD is introduced, denoting a state in which the callback result is unavailable, but the thread has finished.
- The naming of variables and struct members has been cleaned up.
- 02/05: specified alignment for struct runner_context.data
- 02/05: removed most assert() statements
- 02/05: removed superfluous cmm_smp_wmb()
- 02/05: perform all state modifications in the runner's cleanup function
- 03/05: fixed signal handler name
- 03/05: reduced the run time of the runner test script
- 04/05: revert back to the original MAX_TIMEOUTS logic, keeping a ref the "last" runner context until the thread eventually terminates
- 04/05: consistent handling of state and checker message
- 05/05: new

Martin Wilck (5):
multipathd: get_new_state: map PATH_TIMEOUT to PATH_DOWN
libmpathutil: add generic implementation for checker thread runners multipath-tools tests: add test program for thread runners
libmultipath: TUR checker: use runner threads
libmultipath: tur checker: improve tur_deep_sleep() test

 libmpathutil/Makefile             |   2 +-
 libmpathutil/libmpathutil.version |   7 +-
 libmpathutil/runner.c             | 221 ++++++++++++
 libmpathutil/runner.h             | 102 ++++++
 libmultipath/checkers/tur.c       | 369 +++++++-------------
 multipathd/main.c                 |   4 +-
 tests/Makefile                    |  15 +-
 tests/runner-test.sh              |  55 +++
 tests/runner-test.supp            |  15 +
 tests/runner.c                    | 552 ++++++++++++++++++++++++++++++
 10 files changed, 1081 insertions(+), 261 deletions(-)
 create mode 100644 libmpathutil/runner.c
 create mode 100644 libmpathutil/runner.h
 create mode 100755 tests/runner-test.sh
 create mode 100644 tests/runner-test.supp
 create mode 100644 tests/runner.c
Signature · 1 line
2.53.0
#1 lore raw

PATCH v2 1/5 multipathd: get_new_state: map PATH_TIMEOUT to PATH_DOWN

We map PATH_TIMEOUT to PATH_DOWN in pathinfo(), but not in get_new_state(). Do it there, too, to treat the states consistently.
This avoids logging "checker timed out" twice in update_path_state(), even if log_checker_err is set to "once".

---
 multipathd/main.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
multipathd/main.c +3 -1
index d87c7b1..9bce4d6 100644
@@ -2503,8 +2503,10 @@ get_new_state(struct path *pp)
 	 * Wait for uevent for removed paths;
 	 * some LLDDs like zfcp keep paths unavailable
 	 * without sending uevents.
+	 * Also, map PATH_TIMEOUT to PATH_DOWN here, like we do in
+	 * pathinfo().
 	 */
-	if (newstate == PATH_REMOVED)
+	if (newstate == PATH_REMOVED || newstate == PATH_TIMEOUT)
 		newstate = PATH_DOWN;
 
 	/*
Signature · 1 line
2.53.0
#2 lore raw

PATCH v2 2/5 libmpathutil: add generic implementation for checker thread runners

In reply to Martin Wilck

This code adds a generic, abstract implementation of the kind of threads we need for checkers: detached threads that may time out. The design is such that these threads never access any memory of the calling program, simplifying the management of lifetimes of objects.

See the documentation of the API in "runner.h".

---
 libmpathutil/Makefile             |   2 +-
 libmpathutil/libmpathutil.version |   7 +-
 libmpathutil/runner.c             | 221 ++++++++++++++++++++++++++++++
 libmpathutil/runner.h             | 102 ++++++++++++++
 4 files changed, 330 insertions(+), 2 deletions(-)
 create mode 100644 libmpathutil/runner.c
 create mode 100644 libmpathutil/runner.h
libmpathutil/Makefile +1 -1
index e1cc2c6..bdd7063 100644
@@ -13,7 +13,7 @@ LIBDEPS += -lpthread -ldl -ludev -L$(mpathcmddir) -lmpathcmd $(SYSTEMD_LIBDEPS)
 
 # other object files
 OBJS := mt-libudev.o parser.o vector.o util.o debug.o time-util.o \
-	uxsock.o log_pthread.o log.o strbuf.o globals.o msort.o
+	uxsock.o log_pthread.o log.o strbuf.o globals.o msort.o runner.o
 
 all:	$(DEVLIB)
 
libmpathutil/libmpathutil.version +6 -1
index 22ef994..c69120d 100644
@@ -43,7 +43,7 @@ LIBMPATHCOMMON_1.0.0 {
 	put_multipath_config;
 };
 
-LIBMPATHUTIL_5.0 {
+LIBMPATHUTIL_6.0 {
 global:
 	alloc_bitfield;
 	alloc_strvec;
@@ -62,6 +62,7 @@ global:
 	cleanup_vector;
 	cleanup_vector_free;
 	convert_dev;
+	check_runner;
 	dlog;
 	filepresent;
 	fill_strbuf;
@@ -72,6 +73,8 @@ global:
 	free_strvec;
 	get_linux_version_code;
 	get_monotonic_time;
+	get_persistent_runner;
+	get_runner;
 	get_strbuf_buf__;
 	get_next_string;
 	get_strbuf_len;
@@ -161,7 +164,9 @@ global:
 	process_file;
 	pthread_cond_init_mono;
 	recv_packet;
+	release_runner;
 	reset_strbuf;
+	runner_state_name;
 	safe_write;
 	send_packet;
 	set_max_fds;
libmpathutil/runner.c new file +221 -0

221 lines not on this page · raw message

libmpathutil/runner.h new file +102 -0

102 lines not on this page · raw message

Signature · 1 line
2.53.0
#3 lore raw

Re: [PATCH v2 2/5] libmpathutil: add generic implementation for checker thread runners

Quoted 154 lines · Martin Wilck

This code adds a generic, abstract implementation of the kind of threads we need for checkers: detached threads that may time out. The design is such that these threads never access any memory of the calling program, simplifying the management of lifetimes of objects.

See the documentation of the API in "runner.h".

---
 libmpathutil/Makefile             |   2 +-
 libmpathutil/libmpathutil.version |   7 +-
 libmpathutil/runner.c             | 221 ++++++++++++++++++++++++++++++
 libmpathutil/runner.h             | 102 ++++++++++++++
 4 files changed, 330 insertions(+), 2 deletions(-)
 create mode 100644 libmpathutil/runner.c
 create mode 100644 libmpathutil/runner.h
libmpathutil/Makefile +1 -1
index e1cc2c6..bdd7063 100644
@@ -13,7 +13,7 @@ LIBDEPS += -lpthread -ldl -ludev -L$(mpathcmddir) -lmpathcmd $(SYSTEMD_LIBDEPS)
 
 # other object files
 OBJS := mt-libudev.o parser.o vector.o util.o debug.o time-util.o \
-	uxsock.o log_pthread.o log.o strbuf.o globals.o msort.o
+	uxsock.o log_pthread.o log.o strbuf.o globals.o msort.o runner.o
 
 all:	$(DEVLIB)
 
libmpathutil/libmpathutil.version +6 -1
index 22ef994..c69120d 100644
@@ -43,7 +43,7 @@ LIBMPATHCOMMON_1.0.0 {
 	put_multipath_config;
 };
 
-LIBMPATHUTIL_5.0 {
+LIBMPATHUTIL_6.0 {
 global:
 	alloc_bitfield;
 	alloc_strvec;
@@ -62,6 +62,7 @@ global:
 	cleanup_vector;
 	cleanup_vector_free;
 	convert_dev;
+	check_runner;
 	dlog;
 	filepresent;
 	fill_strbuf;
@@ -72,6 +73,8 @@ global:
 	free_strvec;
 	get_linux_version_code;
 	get_monotonic_time;
+	get_persistent_runner;
+	get_runner;
 	get_strbuf_buf__;
 	get_next_string;
 	get_strbuf_len;
@@ -161,7 +164,9 @@ global:
 	process_file;
 	pthread_cond_init_mono;
 	recv_packet;
+	release_runner;
 	reset_strbuf;
+	runner_state_name;
 	safe_write;
 	send_packet;
 	set_max_fds;
libmpathutil/runner.c new file +78 -0
new file mode 100644
index 0000000..f5c5482
@@ -0,0 +1,221 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+// Copyright (c) 2026 SUSE LLC
+#include <assert.h>
+#include <time.h>
+#include <pthread.h>
+#include <urcu/uatomic.h>
+#include "util.h"
+#include "debug.h"
+#include "time-util.h"
+#include "runner.h"
+
+#define STACK_SIZE (4 * 1024)
+#define MILLION 1000000
+
+const char *runner_state_name(int state)
+{
+	// clang-format off
+	static const char * const state_name_[] = {
+		[RUNNER_IDLE] = "idle",
+		[RUNNER_RUNNING] = "running",
+		[RUNNER_DONE] = "done",
+		[RUNNER_CANCELLED] = "cancelled",
+		[RUNNER_DEAD] = "dead"
+	};
+	// clang-format on
+
+	if (state < RUNNER_IDLE || state > RUNNER_DEAD)
+		return "unknown";
+	return state_name_[state];
+}
+
+struct runner_context {
+	int refcount;
+	int status;
+	struct timespec deadline;
+	pthread_t thr;
+	void (*func)(void *data);
+	/* User data will be copied into this area */
+	char __attribute__((aligned(sizeof(void *)))) data[];
+};
+
+static void release_context(struct runner_context *rctx)
+{
+	int n;
+
+	n = uatomic_sub_return(&rctx->refcount, 1);
+	assert(n >= 0);
+
+	if (n == 0)
+		free(rctx);
+}
+
+static void cleanup_context(struct runner_context **prctx)
+{
+	struct runner_context *rctx = *prctx;
+	int st;
+
+	if (!rctx)
+		return;
+
+	st = uatomic_cmpxchg(&rctx->status, RUNNER_RUNNING, RUNNER_DONE);
+	if (st != RUNNER_RUNNING) {
+		uatomic_cmpxchg(&rctx->status, st, RUNNER_DEAD);
+		if (st != RUNNER_CANCELLED)
+			condlog(2, "%s: runner %p finished in state %s",
+				__func__, rctx, runner_state_name(st));
+	}
+	release_context(rctx);
+}
+
+static void *runner_thread(void *arg)
+{
+	int st, refcount;
+	/*
+	 * The cleanup function makes sure memory is freed if the thread is
+	 * cancelled (-fexceptions).
+	 */
+	struct runner_context *rctx __attribute__((cleanup(cleanup_context))) = arg;

libmpathutil/runner.c line 88

Quoted 10 lines
libmpathutil/runner.c new file +10 -0
+
+	refcount = uatomic_add_return(&rctx->refcount, 1);
+	assert(refcount == 2);
+
+	st = uatomic_cmpxchg(&rctx->status, RUNNER_IDLE, RUNNER_RUNNING);
+	/*
+	 * The thread has already been cancelled before it was even started.
+	 * In this case, cancel_runner() doesn't release the context.
+	 * Do it here. See comments for RUNNER_IDLE in cancel_runner().
+	 */

I'm confused about this, and AFAICS, it doesn't appear to be true. If you call check_runner() and it returns RUNNER_CANCELLED, you have no idea if the runner was previously in RUNNER_IDLE, which means that you cannot touch the context anymore, or if the runner was in RUNNER_RUNNING, in which case it's your job to release the context. In the TUR checker, check_runner_state() just calls release_runner(), which will cause a UAF error if the runner was in RUNNER_IDLE before it got cancelled.

It seems to me that the correct way to do this would be to set rctx->refcount to 2 before creating the runner thread, and not even have a RUNNER_IDLE state. Perhaps I'm just being stupid, but I don't see the necessity. If you set a thread to deferred cancellation, I don't believe it can get cancelled between when you create it, and when it hits it's first cancellation point, so you shouldn't have to worry about getting cancelled before the runner's cleanup_context handler isset up. I'm pretty sure that if the thread gets created successfully, it should always drop its reference count when it exits, regardless of if/when it gets cancelled. And if you fail to create the thread, you can just drop the refcount back to 1 before calling release_context() in get_runner().

-Ben

Quoted 243 lines
libmpathutil/runner.c new file +133 -0
+	if (st != RUNNER_IDLE) {
+		release_context(rctx);
+		return NULL;
+	}
+
+	(*rctx->func)(rctx->data);
+	return NULL;
+}
+
+static int cancel_runner(struct runner_context *rctx)
+{
+	int st, st_new;
+	int level = 4, retry = 1;
+
+repeat:
+	st = uatomic_cmpxchg(&rctx->status, RUNNER_RUNNING, RUNNER_CANCELLED);
+	st_new = st;
+	switch (st) {
+	case RUNNER_IDLE:
+		/*
+		 * Race with thread startup.
+		 *
+		 * If after the following cmpxchg st is still IDLE, the cmpxchg
+		 * in runner_thread() will return CANCELLED, and the context
+		 * will be relased there. Otherwise, the thread has switched
+		 * to RUNNING in the meantime, and we will be able to cancel
+		 * it regularly if we retry.
+		 */
+		if (retry--) {
+			st = uatomic_cmpxchg(&rctx->status, RUNNER_IDLE,
+					     RUNNER_CANCELLED);
+			if (st == RUNNER_IDLE)
+				st_new = RUNNER_CANCELLED;
+			else
+				goto repeat;
+		}
+		break;
+	case RUNNER_RUNNING:
+		pthread_cancel(rctx->thr);
+		st_new = RUNNER_CANCELLED;
+		/* fallthrough */
+	case RUNNER_CANCELLED:
+		break;
+	case RUNNER_DONE:
+		st_new = RUNNER_DEAD;
+		/* fallthrough */
+	case RUNNER_DEAD:
+		level = 3;
+		break;
+	}
+	condlog(level, "%s: runner %p cancelled in state '%s'", __func__, rctx,
+		runner_state_name(st));
+	return st_new;
+}
+
+void release_runner(struct runner_context *rctx)
+{
+	cancel_runner(rctx);
+	release_context(rctx);
+}
+
+int check_runner(struct runner_context *rctx, void *data, unsigned int size)
+{
+	int st = uatomic_read(&rctx->status);
+
+	switch (st) {
+	case RUNNER_DONE:
+		if (data)
+			/* hand back the data to the caller */
+			memcpy(data, rctx->data, size);
+		/* fallthrough */
+	case RUNNER_DEAD:
+	case RUNNER_CANCELLED:
+		return st;
+	case RUNNER_IDLE:
+	case RUNNER_RUNNING:
+		if (rctx->deadline.tv_sec != 0 || rctx->deadline.tv_nsec != 0) {
+			struct timespec now;
+
+			get_monotonic_time(&now);
+			if (timespeccmp(&rctx->deadline, &now) <= 0)
+				return cancel_runner(rctx);
+		}
+		/* don't bother the caller with RUNNER_IDLE */
+		return RUNNER_RUNNING;
+	default:
+		condlog(1, "%s: runner in impossible state '%s'", __func__,
+			runner_state_name(st));
+		assert(false);
+		return st;
+	}
+}
+
+struct runner_context *get_runner(runner_func func, void *data,
+				  unsigned int size, unsigned long timeout_usec)
+{
+	static const struct timespec time_zero = {.tv_sec = 0};
+	struct runner_context *rctx;
+	pthread_attr_t attr;
+	int rc;
+
+	if (!func || !data || size <= 0) {
+		condlog(0, "%s: illegal arguments", __func__);
+		return NULL;
+	}
+
+	rctx = malloc(sizeof(*rctx) + size);
+	if (!rctx)
+		return NULL;
+
+	rctx->func = func;
+	uatomic_set(&rctx->refcount, 1);
+	uatomic_set(&rctx->status, RUNNER_IDLE);
+	memcpy(rctx->data, data, size);
+
+	if (timeout_usec) {
+		get_monotonic_time(&rctx->deadline);
+		rctx->deadline.tv_sec += timeout_usec / MILLION;
+		rctx->deadline.tv_nsec += (timeout_usec % MILLION) * 1000;
+	} else
+		rctx->deadline = time_zero;
+
+	setup_thread_attr(&attr, STACK_SIZE, 1);
+	rc = pthread_create(&rctx->thr, &attr, runner_thread, rctx);
+	pthread_attr_destroy(&attr);
+
+	if (rc) {
+		condlog(1, "%s: pthread_create(): %s", __func__, strerror(rc));
+		release_context(rctx);
+		return NULL;
+	}
+	return rctx;
+}
libmpathutil/runner.h new file +102 -0
new file mode 100644
index 0000000..cb2ad94
@@ -0,0 +1,102 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+// Copyright (c) 2026 SUSE LLC
+#ifndef RUNNER_H_INCLUDED
+#define RUNNER_H_INCLUDED
+
+enum runner_status {
+	/**
+	 * Initial state. Runner thread has not started yet.
+	 */
+	RUNNER_IDLE,
+	/**
+	 * The runner thread is running in @runner_func (see below).
+	 */
+	RUNNER_RUNNING,
+	/**
+	 * @runner_func has terminated. This is the only only state
+	 * in which @check_runner can obtain user data in the @data
+	 * parameter.
+	 */
+	RUNNER_DONE,
+	/**
+	 * The runner thread has been cancelled (usually because of a timeout),
+	 * but @runner_func is still running.
+	 */
+	RUNNER_CANCELLED,
+	/**
+	 * The runner thread has terminated without providing user data
+	 * (usually after a timeout).
+	 */
+	RUNNER_DEAD,
+};
+
+typedef void (*runner_func)(void *data);
+struct runner_context;
+
+/**
+ * runner_state_name(): helper for printing runner states
+ *
+ * @param state: a valid @runner_status value
+ * @returns: a string describing the status
+ */
+const char *runner_state_name(int state);
+
+/**
+ * get_runner(): start a runner thread
+ *
+ * This function starts a runner thread that calls @func(@data).
+ * The thread is created as detached thread.
+ * @data will be copied to thread-private memory, which will be freed when
+ * the runner terminates.
+ * Output values can be retrieved with @check_runner().
+ *
+ * @param func: the worker function to invoke
+ *        This parameter must not be NULL.
+ * @param data: pointer the data to pass to the function (input and output)
+ *        This parameter must not be NULL.
+ * @param size: the size (in bytes) of the data passed to the function
+ *        This parameter must be positive.
+ * @param timeout_usec: timeout (in microseconds) after which to cancel the
+ * runner. If it is 0, the runner will not time out.
+ * @returns: a runner context that must be passed to the functions below.
+ */
+struct runner_context *get_runner(runner_func func, void *data,
+				  unsigned int size, unsigned long timeout_usec);
+
+/**
+ * release_runner(): release a runner context
+ *
+ * This function should be called when the caller has no interest to
+ * operate on the given @runner_context any more.
+ * If necessary, the runner thread will be cancelled.
+ * Upon return of this function, @rctx becomes stale and shouldn't accessed
+ * any more.
+ *
+ * @param rctx: the context of the runner to be released.
+ */
+void release_runner(struct runner_context *rctx);
+
+/**
+ * check_runner(): query the state of a runner thread and obtain results
+ *
+ * Check the state of a runner previously started with @get_runner. If the
+ * thread function has completed, @RUNNER_DONE will be returned, and the
+ * user data will be copied into the @data argument. If @check_runner returns
+ * anything else, @data contains no valid data. The @size argument
+ * will typically be the same as the @size passed to @get_runner (meaning
+ * that @data represents an object of the same type as the @data argument
+ * passed to @get_runner previously).
+ *
+ * Side effect: If the runner has timed out, the thread will be cancelled.
+ *
+ * @param rctx: the context of the runner to be queried
+ * @param data: memory pointer that will receive results of the worker
+ * function. Can be NULL, in which case no data will be copied.
+ * @param size: size of the memory pointed to by data. It must be no bigger
+ *        than the size of the memory passed to @get_runner for this runner.
+ *        It can be smaller, but no more than @size bytes will be copied.
+ * @returns: @RUNNER_RUNNING, @RUNNER_DONE, @RUNNER_CANCELLED, or @RUNNER_DEAD.
+ */
+int check_runner(struct runner_context *rctx, void *data, unsigned int size);
+
+#endif /* RUNNER_H_INCLUDED */
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH v2 2/5] libmpathutil: add generic implementation for checker thread runners

Hi Ben,

Quoted 22 lines · Benjamin Marzinski
Quoted 21 lines · Martin Wilck

This code adds a generic, abstract implementation of the kind of threads we
need for checkers: detached threads that may time out. The design is such
that these threads never access any memory of the calling program, simplifying the management of lifetimes of objects.

See the documentation of the API in "runner.h".

---
 libmpathutil/Makefile             |   2 +- libmpathutil/libmpathutil.version |   7 +- libmpathutil/runner.c             | 221
++++++++++++++++++++++++++++++
 libmpathutil/runner.h             | 102 ++++++++++++++

 4 files changed, 330 insertions(+), 2 deletions(-)

 create mode 100644 libmpathutil/runner.c
 create mode 100644 libmpathutil/runner.h

[...]

Quoted 5 lines
Quoted 5 lines
libmpathutil/runner.c new file +0 -0
new file mode 100644
index 0000000..f5c5482

[...]

Quoted 26 lines
Quoted 23 lines
libmpathutil/runner.c new file +5 -0
+static void *runner_thread(void *arg)
+{
+	int st, refcount;
+	/*
+	 * The cleanup function makes sure memory is freed if the

thread is

libmpathutil/runner.c new file +3 -0
+	 * cancelled (-fexceptions).
+	 */
+	struct runner_context *rctx

__attribute__((cleanup(cleanup_context))) = arg;

libmpathutil/runner.c new file +5 -0
+
+	refcount = uatomic_add_return(&rctx->refcount, 1);
+	assert(refcount == 2);
+
+	st = uatomic_cmpxchg(&rctx->status, RUNNER_IDLE,

RUNNER_RUNNING);

libmpathutil/runner.c new file +2 -0
+	/*
+	 * The thread has already been cancelled before it was

even started.

libmpathutil/runner.c new file +1 -0
+	 * In this case, cancel_runner() doesn't release the

context.

libmpathutil/runner.c new file +1 -0
+	 * Do it here. See comments for RUNNER_IDLE in

cancel_runner().

libmpathutil/runner.c

Quoted 1 line
libmpathutil/runner.c new file +1 -0
+	 */

I'm confused about this, and AFAICS, it doesn't appear to be true. If

Quoted 8 lines

you call check_runner() and it returns RUNNER_CANCELLED, you have no idea if the runner was previously in RUNNER_IDLE, which means that you
cannot touch the context anymore, or if the runner was in RUNNER_RUNNING, in which case it's your job to release the context. In the TUR checker, check_runner_state() just calls release_runner(), which will cause a UAF error if the runner was in RUNNER_IDLE before it got cancelled.

Right, good catch, thanks for finding it. This was a remnant of the previous code, where the caller didn't have to release the context after RUNNER_CANCELLED.

Releasing the context here is just wrong, I'll drop the line.

I had expected my tests to capture an UAF like this, but I made a dumb
mistake: I used a timeout of 0 in my test code, which means "infinite" timeout for the runner, and thus my first test case, which was intended to capture thus this kind of issue, was ineffective. I will change my test program now to use 1us timeout if "-t 0" is specified. I will also add a compile-time option to artificially delay thread startup in order to test this.

Quoted 4 lines

It seems to me that the correct way to do this would be to set rctx->refcount to 2 before creating the runner thread, and not even have
a RUNNER_IDLE state.

Correct, too. I was totally naïve to rely on the thread itself increasing the refcount if I allow the thread to be cancelled before it even starts :-/

However, I think I'll keep RUNNER_IDLE state around. It doesn't hurt and it provides possible insight into whether or not the runner function was actually called.

Quoted 7 lines

Perhaps I'm just being stupid, but I don't see the necessity. If you set a thread to deferred cancellation, I don't believe
it can get cancelled between when you create it, and when it hits it's
first cancellation point, so you shouldn't have to worry about getting

Quoted 8 lines

cancelled before the runner's cleanup_context handler isset up. I'm pretty sure that if the thread gets created successfully, it should always drop its reference count when it exits, regardless of if/when it
gets cancelled. And if you fail to create the thread, you can just drop
the refcount back to 1 before calling release_context() in get_runner().

I believe that it's possible for pthread_create() to succeed and the program to terminate before the thread is started. In which case the thread's cleanup code won't be set up, and the context won't be freed. But we explicitly don't want to wait for these threads, so I think that's a cosmetic problem in a corner case that we don't need to bother with.

Thanks,
Martin

#5 lore raw

PATCH v2 3/5 multipath-tools tests: add test program for thread runners

In reply to Martin Wilck

Add a test program for the "runner" thread implementation from the previous commit. The test program runs simulated "hanging" threads that may time out, and optionally kills all threads at an arbitrary point in time. See the comments at the top of the file for details.

Also add a test driver script (runner-test.sh) with a few reasonable combinations of command line arguments.

The test program has been used to test the "runner" implementation extensively on different architectures (x86_64, aarch64, ppc64le, s390x), using both valgrind and the gcc address sanitizer (libasan) for detection of memory leaks and use-after-free errors.

For valgrind, a suppression file needs to be added, as valgrind doesn't seem to capture the deallocation of thread local storage for detached threads in the test case where the test program is killed. The suppression affects only memory allocated by glibc. This leak has not been seen with libasan, only with valgrind.

---
 tests/Makefile         |  15 +-
 tests/runner-test.sh   |  55 ++++
 tests/runner-test.supp |  15 ++
 tests/runner.c         | 552 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 633 insertions(+), 4 deletions(-)
 create mode 100755 tests/runner-test.sh
 create mode 100644 tests/runner-test.supp
 create mode 100644 tests/runner.c
tests/Makefile +11 -4
index 9f1b950..f0e5dd3 100644
@@ -9,12 +9,12 @@ CFLAGS += $(BIN_CFLAGS) -Wno-unused-parameter -Wno-unused-function $(W_MISSING_I
 LIBDEPS += -L. -L $(mpathutildir) -L$(mpathcmddir) -lmultipath -lmpathutil -lmpathcmd -lcmocka
 
 TESTS := uevent parser util dmevents hwtable blacklist unaligned vpd pgpolicy \
-	 alias directio valid devt mpathvalid strbuf sysfs features cli mapinfo
+	 alias directio valid devt mpathvalid strbuf sysfs features cli mapinfo runner
 HELPERS := test-lib.o test-log.o
 
 .PRECIOUS: $(TESTS:%=%-test)
 
-all:	$(TESTS:%=%.out)
+all:	$(TESTS:%=%.out) runner-test.out
 progs:	$(TESTS:%=%-test) lib/libchecktur.so
 valgrind:	$(TESTS:%=%.vgr)
 
@@ -65,6 +65,7 @@ features-test_LIBDEPS := -ludev -lpthread
 features-test_OBJDEPS := $(mpathutildir)/mt-libudev.o
 cli-test_OBJDEPS := $(daemondir)/cli.o
 mapinfo-test_LIBDEPS = -lpthread -ldevmapper
+runner-test_LIBDEPS = -lpthread
 
 %.o: %.c
 	@echo building $@ because of $?
@@ -81,12 +82,18 @@ lib/libchecktur.so:
 %.vgr:  %-test lib/libchecktur.so
 	@echo == running valgrind for $< ==
 	@LD_LIBRARY_PATH=.:$(mpathutildir):$(mpathcmddir) \
-		valgrind --leak-check=full --error-exitcode=128 ./$< >$@ 2>&1
+		valgrind --leak-check=full --error-exitcode=128 --suppressions=./runner-test.supp ./$< >$@ 2>&1
 
 OBJS = $(TESTS:%=%.o) $(HELPERS)
 
+runner-test.out: runner-test
+	$(Q)./runner-test.sh >$@ 2>&1
+
+runner-test.vgr: runner-test
+	$(Q)VALGRIND=1 ./runner-test.sh >$@ 2>&1
+
 test_clean:
-	$(Q)$(RM) $(TESTS:%=%.out) $(TESTS:%=%.vgr) *.so*
+	$(Q)$(RM) $(TESTS:%=%.out) $(TESTS:%=%.vgr) *.so* runner-test.out runner-test.vgr
 
 valgrind_clean:
 	$(Q)$(RM) $(TESTS:%=%.vgr)
tests/runner-test.sh new file +55 -0

55 lines not on this page · raw message

tests/runner-test.supp new file +15 -0
new file mode 100644
index 0000000..e099ec9
@@ -0,0 +1,15 @@
+{
+   glibc TLS for detached threads if program is terminated
+   Memcheck:Leak
+   match-leak-kinds: possible
+   fun:calloc
+   ...
+   fun:allocate_dtv
+   fun:_dl_allocate_tls
+   fun:allocate_stack
+   ...
+   fun:get_runner
+   fun:start_runner
+   ...
+   fun:main
+}
tests/runner.c new file +552 -0

552 lines not on this page · raw message

Signature · 1 line
2.53.0
maintainer Reviewed-by on v2 3/5
#6 lore raw

Re: [PATCH v2 3/5] multipath-tools tests: add test program for thread runners

Quoted 13 lines · Martin Wilck

Add a test program for the "runner" thread implementation from the previous commit. The test program runs simulated "hanging" threads that may time out, and optionally kills all threads at an arbitrary point in time. See the comments at the top of the file for details.

Also add a test driver script (runner-test.sh) with a few reasonable combinations of command line arguments.

The test program has been used to test the "runner" implementation extensively on different architectures (x86_64, aarch64, ppc64le, s390x), using both valgrind and the gcc address sanitizer (libasan) for detection of memory leaks and use-after-free errors.

Quoted 7 lines

For valgrind, a suppression file needs to be added, as valgrind doesn't seem to capture the deallocation of thread local storage for detached threads in the test case where the test program is killed. The suppression affects only memory allocated by glibc. This leak has not been seen with libasan, only with valgrind.

#7 lore raw

PATCH v2 4/5 libmultipath: TUR checker: use runner threads

In reply to Martin Wilck

Use the generic runners to simplify the TUR checker code.

---
 libmultipath/checkers/tur.c | 363 +++++++++++-------------------------
 1 file changed, 112 insertions(+), 251 deletions(-)
libmultipath/checkers/tur.c +112 -251
index ba4ca68..0a312c7 100644
@@ -3,7 +3,6 @@
  *
  * Copyright (c) 2004 Christophe Varoqui
  */
-#include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <sys/types.h>
@@ -14,16 +13,11 @@
 #include <sys/sysmacros.h>
 #include <errno.h>
 #include <sys/time.h>
-#include <pthread.h>
-#include <urcu.h>
-#include <urcu/uatomic.h>
 
 #include "checkers.h"
-
 #include "debug.h"
 #include "sg_include.h"
-#include "util.h"
-#include "time-util.h"
+#include "runner.h"
 
 #define TUR_CMD_LEN 6
 #define HEAVY_CHECK_COUNT       10
@@ -45,70 +39,47 @@ const char *libcheck_msgtable[] = {
 	NULL,
 };
 
-struct tur_checker_context {
-	dev_t devt;
-	int state;
-	int running; /* uatomic access only */
+struct tur_data {
 	int fd;
+	dev_t devt;
 	unsigned int timeout;
-	time_t time;
-	pthread_t thread;
-	pthread_mutex_t lock;
-	pthread_cond_t active;
-	int holders; /* uatomic access only */
-	int msgid;
-	struct checker_context ctx;
-	unsigned int nr_timeouts;
-	bool checked_state;
+	int state;
+	short msgid;
 };
 
-int libcheck_init (struct checker * c)
+struct tur_checker_context {
+	struct checker_context chkr;
+	int last_runner_state;
+	unsigned int nr_timeouts;
+	struct runner_context *rtx;
+	struct tur_data tdata;
+};
+
+int libcheck_init(struct checker *c)
 {
-	struct tur_checker_context *ct;
+	struct tur_checker_context *tcc;
 	struct stat sb;
 
-	ct = malloc(sizeof(struct tur_checker_context));
-	if (!ct)
-		return 1;
-	memset(ct, 0, sizeof(struct tur_checker_context));
-
-	ct->state = PATH_UNCHECKED;
-	ct->fd = -1;
-	uatomic_set(&ct->holders, 1);
-	pthread_cond_init_mono(&ct->active);
-	pthread_mutex_init(&ct->lock, NULL);
+	tcc = calloc(1, sizeof(*tcc));
+	tcc->tdata.state = PATH_UNCHECKED;
+	tcc->tdata.fd = -1;
 	if (fstat(c->fd, &sb) == 0)
-		ct->devt = sb.st_rdev;
-	ct->ctx.cls = c->cls;
-	c->context = ct;
-
+		tcc->tdata.devt = sb.st_rdev;
+	tcc->chkr.cls = c->cls;
+	c->context = tcc;
 	return 0;
 }
 
-static void cleanup_context(struct tur_checker_context *ct)
-{
-	pthread_mutex_destroy(&ct->lock);
-	pthread_cond_destroy(&ct->active);
-	free(ct);
-}
-
 void libcheck_free (struct checker * c)
 {
-	if (c->context) {
-		struct tur_checker_context *ct = c->context;
-		int holders;
-		int running;
+	struct tur_checker_context *tcc = c->context;
 
-		running = uatomic_xchg(&ct->running, 0);
-		if (running)
-			pthread_cancel(ct->thread);
-		ct->thread = 0;
-		holders = uatomic_sub_return(&ct->holders, 1);
-		if (!holders)
-			cleanup_context(ct);
-		c->context = NULL;
-	}
-	return;
+	if (!tcc)
+		return;
+	c->context = NULL;
+	if (tcc->rtx)
+		release_runner(tcc->rtx);
+	free(tcc);
 }
 
 static int
@@ -216,19 +187,6 @@ retry:
 	return PATH_UP;
 }
 
-#define tur_thread_cleanup_push(ct) pthread_cleanup_push(cleanup_func, ct)
-#define tur_thread_cleanup_pop(ct) pthread_cleanup_pop(1)
-
-static void cleanup_func(void *data)
-{
-	int holders;
-	struct tur_checker_context *ct = data;
-
-	holders = uatomic_sub_return(&ct->holders, 1);
-	if (!holders)
-		cleanup_context(ct);
-}
-
 /*
  * Test code for "zombie tur thread" handling.
  * Compile e.g. with CFLAGS=-DTUR_TEST_MAJOR=8
@@ -249,13 +207,13 @@ static void cleanup_func(void *data)
 #define TUR_SLEEP_SECS 60
 #endif
 
-static void tur_deep_sleep(const struct tur_checker_context *ct)
+static void tur_deep_sleep(const struct tur_data *tdata)
 {
 	static int sleep_cnt;
 	const struct timespec ts = { .tv_sec = TUR_SLEEP_SECS, .tv_nsec = 0 };
 	int oldstate;
 
-	if (ct->devt != makedev(TUR_TEST_MAJOR, TUR_TEST_MINOR) ||
+	if (tdata->devt != makedev(TUR_TEST_MAJOR, TUR_TEST_MINOR) ||
 	    ++sleep_cnt % TUR_SLEEP_INTERVAL != 0)
 		return;
 
@@ -273,110 +231,92 @@ static void tur_deep_sleep(const struct tur_checker_context *ct)
 #define tur_deep_sleep(x) do {} while (0)
 #endif /* TUR_TEST_MAJOR */
 
-void *libcheck_thread(struct checker_context *ctx)
+void runner_callback(void *arg)
 {
-	struct tur_checker_context *ct =
-		container_of(ctx, struct tur_checker_context, ctx);
-	int state, running;
-	short msgid;
+	struct tur_data *tdata = arg;
+	int state;
 
-	/* This thread can be canceled, so setup clean up */
-	tur_thread_cleanup_push(ct);
+	condlog(4, "%d:%d : tur checker starting up", major(tdata->devt),
+		minor(tdata->devt));
 
-	condlog(4, "%d:%d : tur checker starting up", major(ct->devt),
-		minor(ct->devt));
-
-	tur_deep_sleep(ct);
-	state = tur_check(ct->fd, ct->timeout, &msgid);
+	tur_deep_sleep(tdata);
+	state = tur_check(tdata->fd, tdata->timeout, &tdata->msgid);
+	tdata->state = state;
 	pthread_testcancel();
-
-	/* TUR checker done */
-	pthread_mutex_lock(&ct->lock);
-	ct->state = state;
-	ct->msgid = msgid;
-	pthread_cond_signal(&ct->active);
-	pthread_mutex_unlock(&ct->lock);
-
-	condlog(4, "%d:%d : tur checker finished, state %s", major(ct->devt),
-		minor(ct->devt), checker_state_name(state));
-
-	running = uatomic_xchg(&ct->running, 0);
-	if (!running)
-		pause();
-
-	tur_thread_cleanup_pop(ct);
-
-	return ((void *)0);
+	condlog(4, "%d:%d : tur checker finished, state %s", major(tdata->devt),
+		minor(tdata->devt), checker_state_name(state));
 }
 
-static void tur_set_async_timeout(struct checker *c)
+static int check_runner_state(struct tur_checker_context *tcc)
 {
-	struct tur_checker_context *ct = c->context;
-	struct timespec now;
+	struct runner_context *rtx = tcc->rtx;
+	int rc;
 
-	get_monotonic_time(&now);
-	ct->time = now.tv_sec + c->timeout;
-}
-
-static int tur_check_async_timeout(struct checker *c)
-{
-	struct tur_checker_context *ct = c->context;
-	struct timespec now;
-
-	get_monotonic_time(&now);
-	return (now.tv_sec > ct->time);
-}
-
-int check_pending(struct checker *c)
-{
-	struct tur_checker_context *ct = c->context;
-	int tur_status = PATH_PENDING;
-
-	pthread_mutex_lock(&ct->lock);
-
-	if (ct->state != PATH_PENDING || ct->msgid != MSG_TUR_RUNNING)
-	{
-		tur_status = ct->state;
-		c->msgid = ct->msgid;
-	}
-	pthread_mutex_unlock(&ct->lock);
-	if (tur_status == PATH_PENDING && c->msgid == MSG_TUR_RUNNING) {
+	rc = check_runner(rtx, &tcc->tdata, sizeof(tcc->tdata));
+	switch (rc) {
+	case RUNNER_DEAD:
+		tcc->tdata.state = PATH_TIMEOUT;
+		tcc->tdata.msgid = MSG_TUR_TIMEOUT;
+		/* fallthrough */
+	case RUNNER_DONE:
+		release_runner(tcc->rtx);
+		tcc->rtx = NULL;
+		tcc->last_runner_state = rc;
+		tcc->nr_timeouts = 0;
+		condlog(rc == RUNNER_DONE ? 4 : 3,
+			"%d:%d : tur checker finished, state %s, runner state %s",
+			major(tcc->tdata.devt), minor(tcc->tdata.devt),
+			checker_state_name(tcc->tdata.state),
+			runner_state_name(rc));
+		break;
+	case RUNNER_CANCELLED:
+		tcc->last_runner_state = rc;
+		tcc->tdata.state = PATH_TIMEOUT;
+		tcc->tdata.msgid = MSG_TUR_TIMEOUT;
+		if (tcc->nr_timeouts < MAX_NR_TIMEOUTS) {
+			condlog(3, "%d:%d : tur checker timed out, releasing it",
+				major(tcc->tdata.devt), minor(tcc->tdata.devt));
+			tcc->nr_timeouts++;
+			release_runner(tcc->rtx);
+			tcc->rtx = NULL;
+		} else if (tcc->nr_timeouts == MAX_NR_TIMEOUTS) {
+			tcc->nr_timeouts++;
+			condlog(3, "%d:%d : tur checker timed out, waiting for it",
+				major(tcc->tdata.devt), minor(tcc->tdata.devt));
+		}
+		break;
+	default:
 		condlog(4, "%d:%d : tur checker still running",
-			major(ct->devt), minor(ct->devt));
-	} else {
-		int running = uatomic_xchg(&ct->running, 0);
-		if (running)
-			pthread_cancel(ct->thread);
-		ct->thread = 0;
+			major(tcc->tdata.devt), minor(tcc->tdata.devt));
+		tcc->tdata.msgid = MSG_TUR_RUNNING;
+		break;
 	}
-
-	ct->checked_state = true;
-	return tur_status;
+	return rc;
 }
 
 bool libcheck_need_wait(struct checker *c)
 {
 	struct tur_checker_context *ct = c->context;
-	return (ct && ct->thread && uatomic_read(&ct->running) != 0 &&
-		!ct->checked_state);
+
+	return ct && ct->rtx;
 }
 
 int libcheck_pending(struct checker *c)
 {
 	struct tur_checker_context *ct = c->context;
-
 	/* The if path checker isn't running, just return the exiting value. */
-	if (!ct || !ct->thread)
+	if (!ct || !ct->rtx)
 		return c->path_state;
 
-	return check_pending(c);
+	/* This may nullify ct->rtx */
+	check_runner_state(ct);
+	c->msgid = ct->tdata.msgid;
+	return ct->tdata.state;
 }
 
 int libcheck_check(struct checker * c)
 {
 	struct tur_checker_context *ct = c->context;
-	pthread_attr_t attr;
-	int tur_status, r;
 
 	if (!ct)
 		return PATH_UNCHECKED;
@@ -384,109 +324,30 @@ int libcheck_check(struct checker * c)
 	if (checker_is_sync(c))
 		return tur_check(c->fd, c->timeout, &c->msgid);
 
-	/*
-	 * Async mode
-	 */
-	if (ct->thread) {
-		ct->checked_state = true;
-		if (tur_check_async_timeout(c)) {
-			int running = uatomic_xchg(&ct->running, 0);
-			if (running) {
-				pthread_cancel(ct->thread);
-				condlog(3, "%d:%d : tur checker timeout",
-					major(ct->devt), minor(ct->devt));
-				c->msgid = MSG_TUR_TIMEOUT;
-				tur_status = PATH_TIMEOUT;
-			} else {
-				pthread_mutex_lock(&ct->lock);
-				tur_status = ct->state;
-				c->msgid = ct->msgid;
-				pthread_mutex_unlock(&ct->lock);
-			}
-			ct->thread = 0;
-		} else if (uatomic_read(&ct->running) != 0) {
-			condlog(3, "%d:%d : tur checker not finished",
-				major(ct->devt), minor(ct->devt));
-			tur_status = PATH_PENDING;
-			c->msgid = MSG_TUR_RUNNING;
-		} else {
-			/* TUR checker done */
-			ct->thread = 0;
-			pthread_mutex_lock(&ct->lock);
-			tur_status = ct->state;
-			c->msgid = ct->msgid;
-			pthread_mutex_unlock(&ct->lock);
-		}
-	} else {
-		if (uatomic_read(&ct->holders) > 1) {
-			/* The thread has been cancelled but hasn't quit. */
-			if (ct->nr_timeouts == MAX_NR_TIMEOUTS) {
-				condlog(2, "%d:%d : waiting for stalled tur thread to finish",
-					major(ct->devt), minor(ct->devt));
-				ct->nr_timeouts++;
-			}
-			/*
-			 * Don't start new threads until the last once has
-			 * finished.
-			 */
-			if (ct->nr_timeouts > MAX_NR_TIMEOUTS) {
-				c->msgid = MSG_TUR_TIMEOUT;
-				return PATH_TIMEOUT;
-			}
-			ct->nr_timeouts++;
-			/*
-			 * Start a new thread while the old one is stalled.
-			 * We have to prevent it from interfering with the new
-			 * thread. We create a new context and leave the old
-			 * one with the stale thread, hoping it will clean up
-			 * eventually.
-			 */
-			condlog(3, "%d:%d : tur thread not responding",
-				major(ct->devt), minor(ct->devt));
-
-			/*
-			 * libcheck_init will replace c->context.
-			 * It fails only in OOM situations. In this case, return
-			 * PATH_UNCHECKED to avoid prematurely failing the path.
-			 */
-			if (libcheck_init(c) != 0) {
-				c->msgid = MSG_TUR_FAILED;
-				return PATH_UNCHECKED;
-			}
-			((struct tur_checker_context *)c->context)->nr_timeouts = ct->nr_timeouts;
-
-			if (!uatomic_sub_return(&ct->holders, 1)) {
-				/* It did terminate, eventually */
-				cleanup_context(ct);
-				((struct tur_checker_context *)c->context)->nr_timeouts = 0;
-			}
-
-			ct = c->context;
-		} else
-			ct->nr_timeouts = 0;
-		/* Start new TUR checker */
-		pthread_mutex_lock(&ct->lock);
-		tur_status = ct->state = PATH_PENDING;
-		c->msgid = ct->msgid = MSG_TUR_RUNNING;
-		pthread_mutex_unlock(&ct->lock);
-		ct->fd = c->fd;
-		ct->timeout = c->timeout;
-		ct->checked_state = false;
-		uatomic_add(&ct->holders, 1);
-		uatomic_set(&ct->running, 1);
-		tur_set_async_timeout(c);
-		setup_thread_attr(&attr, 32 * 1024, 1);
-		r = start_checker_thread(&ct->thread, &attr, &ct->ctx);
-		pthread_attr_destroy(&attr);
-		if (r) {
-			uatomic_sub(&ct->holders, 1);
-			uatomic_set(&ct->running, 0);
-			ct->thread = 0;
-			condlog(3, "%d:%d : failed to start tur thread, using"
-				" sync mode", major(ct->devt), minor(ct->devt));
-			return tur_check(c->fd, c->timeout, &c->msgid);
-		}
+	/* Handle the case that the checker just completed */
+	if (ct->rtx) {
+		check_runner_state(ct);
+		c->msgid = ct->tdata.msgid;
+		return ct->tdata.state;
 	}
 
-	return tur_status;
+	/* create new checker thread */
+	ct->tdata.fd = c->fd;
+	ct->tdata.timeout = c->timeout;
+
+	ct->tdata.state = PATH_PENDING;
+	ct->tdata.msgid = MSG_TUR_RUNNING;
+	condlog(3, "%d:%d : starting checker", major(ct->tdata.devt),
+		minor(ct->tdata.devt));
+	ct->rtx = get_runner(runner_callback, &ct->tdata, sizeof(ct->tdata),
+			     1000000 * c->timeout);
+
+	if (ct->rtx) {
+		c->msgid = ct->tdata.msgid;
+		return ct->tdata.state;
+	} else {
+		condlog(3, "%d:%d : failed to start tur thread, using sync mode",
+			major(ct->tdata.devt), minor(ct->tdata.devt));
+		return tur_check(c->fd, c->timeout, &c->msgid);
+	}
 }
Signature · 1 line
2.53.0
#9 lore raw

PATCH v2 5/5 libmultipath: tur checker: improve tur_deep_sleep() test

In reply to Martin Wilck

The tur_deep_sleep() test serves to test the behavior of the tur checker when checkers time out and cannot be cancelled.

Up to now the checker timed out on every TUR_SLEEP_INTERVALth invocation. That made it impossible to test the MAX_TIMEOUTS logic, which requires multiple consecutive timeouts. To fix it, invert the logic, so that the checker just succeeds on every TUR_SLEEP_INTERVALth call.

Also, decrease loglevel of the messages from tur_deep_sleep().

---
 libmultipath/checkers/tur.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
libmultipath/checkers/tur.c +3 -3
index 0a312c7..6791172 100644
@@ -214,15 +214,15 @@ static void tur_deep_sleep(const struct tur_data *tdata)
 	int oldstate;
 
 	if (tdata->devt != makedev(TUR_TEST_MAJOR, TUR_TEST_MINOR) ||
-	    ++sleep_cnt % TUR_SLEEP_INTERVAL != 0)
+	    ++sleep_cnt % TUR_SLEEP_INTERVAL == 0)
 		return;
 
-	condlog(1, "tur thread going to sleep for %ld seconds", ts.tv_sec);
+	condlog(3, "tur thread going to sleep for %ld seconds", ts.tv_sec);
 	if (pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate) != 0)
 		condlog(0, "pthread_setcancelstate: %m");
 	if (nanosleep(&ts, NULL) != 0)
 		condlog(0, "nanosleep: %m");
-	condlog(1, "tur zombie thread woke up");
+	condlog(3, "tur zombie thread woke up");
 	if (pthread_setcancelstate(oldstate, NULL) != 0)
 		condlog(0, "pthread_setcancelstate (2): %m");
 	pthread_testcancel();
Signature · 1 line
2.53.0
maintainer Reviewed-by on v2 5/5
#10 lore raw

Re: [PATCH v2 5/5] libmultipath: tur checker: improve tur_deep_sleep() test

Quoted 11 lines · Martin Wilck

The tur_deep_sleep() test serves to test the behavior of the tur checker when checkers time out and cannot be cancelled.

Up to now the checker timed out on every TUR_SLEEP_INTERVALth invocation. That made it impossible to test the MAX_TIMEOUTS logic, which requires multiple consecutive timeouts. To fix it, invert the logic, so that the checker just succeeds on every TUR_SLEEP_INTERVALth call.

Also, decrease loglevel of the messages from tur_deep_sleep().