Skip to contentSkip to conversation

membarrier: add MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU

PATCH Superseded Peter Oskolkov 4 participants 8 messages Aug 6, 2020 – Aug 7, 2020

Subsystems (4)

v1 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 membarrier: add MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU

This patchset is based on Google-internal RSEQ work done by Paul Turner and Andrew Hunter.

When working with per-CPU RSEQ-based memory allocations, it is sometimes important to make sure that a global memory location is no longer accessed from RSEQ critical sections. For example, there can be two per-CPU lists, one is "active" and accessed per-CPU, while another one is inactive and worked on asynchronously "off CPU" (e.g. garbage collection is performed). Then at some point the two lists are swapped, and a fast RCU-like mechanism is required to make sure that the previously active list is no longer accessed.

This patch introduces such a mechanism: in short, membarrier() syscall issues an IPI to a CPU, restarting any potentially active RSEQ critical sections on the CPU, with an option to restart RSEQ CSs on all CPUs.

The second patch in the patchset adds a selftest of this feature.

---
 include/uapi/linux/membarrier.h |  8 ++++++
 kernel/sched/membarrier.c       | 51 +++++++++++++++++++++++++++++++--
 2 files changed, 57 insertions(+), 2 deletions(-)
include/uapi/linux/membarrier.h +8 -0
index 5891d7614c8c..169ffb613397 100644
@@ -114,6 +114,13 @@
  *                          If this command is not implemented by an
  *                          architecture, -EINVAL is returned.
  *                          Returns 0 on success.
+ * @MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU:
+ *                          If a thread belonging to the current process
+ *                          is currently in an RSEQ critical section on the
+ *                          CPU identified by flags parameter, restart it.
+ *                          @flags: if @flags >= 0, identifies the CPU to
+ *                                  restart RSEQ CS on; if == -1, restarts
+ *                                  RSEQ CSs on all CPUs.
  * @MEMBARRIER_CMD_SHARED:
  *                          Alias to MEMBARRIER_CMD_GLOBAL. Provided for
  *                          header backward compatibility.
@@ -131,6 +138,7 @@ enum membarrier_cmd {
 	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED		= (1 << 4),
 	MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE		= (1 << 5),
 	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE	= (1 << 6),
+	MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU		= (1 << 7),
 
 	/* Alias for header backward compatibility. */
 	MEMBARRIER_CMD_SHARED			= MEMBARRIER_CMD_GLOBAL,
kernel/sched/membarrier.c +49 -2

83 lines not on this page · raw message

Signature · 1 line
2.28.0.163.g6104cc2f0b6-goog
#1 lore raw

PATCH 2/2 selftests/rseq: test MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU

Based on Google-internal RSEQ work done by
Paul Turner and Andrew Hunter.

This patch adds a selftest for MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU. The test quite often fails without the previous patch in this patchset, but consistently passes with it.

---
 .../selftests/rseq/basic_percpu_ops_test.c    | 181 ++++++++++++++++++
 1 file changed, 181 insertions(+)
tools/testing/selftests/rseq/basic_percpu_ops_test.c +181 -0
index eb3f6db36d36..147c80deac19 100644
@@ -3,16 +3,21 @@
 #include <assert.h>
 #include <pthread.h>
 #include <sched.h>
+#include <stdatomic.h>
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <stddef.h>
+#include <syscall.h>
+#include <unistd.h>
 
 #include "rseq.h"
 
 #define ARRAY_SIZE(arr)	(sizeof(arr) / sizeof((arr)[0]))
 
+#define MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU	(1<<7)
+
 struct percpu_lock_entry {
 	intptr_t v;
 } __attribute__((aligned(128)));
@@ -289,6 +294,180 @@ void test_percpu_list(void)
 	assert(sum == expected_sum);
 }
 
+struct test_membarrier_thread_args {
+	int stop;
+	intptr_t percpu_list_ptr;
+};
+
+/* Worker threads modify data in their "active" percpu lists. */
+void *test_membarrier_worker_thread(void *arg)
+{
+	struct test_membarrier_thread_args *args =
+		(struct test_membarrier_thread_args *)arg;
+	const int iters = 10 * 1000 * 1000;
+	int i;
+
+	if (rseq_register_current_thread()) {
+		fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s\n",
+			errno, strerror(errno));
+		abort();
+	}
+
+	for (i = 0; i < iters; ++i) {
+		while (true) {
+			int cpu, ret;
+			struct percpu_list *list_ptr = (struct percpu_list *)
+				atomic_load(&args->percpu_list_ptr);
+
+			if (!list_ptr)
+				continue;  /* Not yet initialized. */
+
+			cpu = rseq_cpu_start();
+			struct percpu_list_node *node = list_ptr->c[cpu].head;
+			const intptr_t prev = node->data;
+
+			ret = rseq_cmpeqv_cmpeqv_storev(&node->data, prev,
+					&args->percpu_list_ptr,
+					(intptr_t)list_ptr, prev + 1, cpu);
+			if (!ret)
+				break;  /* Success. */
+		}
+	}
+
+	if (rseq_unregister_current_thread()) {
+		fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): %s\n",
+			errno, strerror(errno));
+		abort();
+	}
+	return NULL;
+}
+
+void test_membarrier_init_percpu_list(struct percpu_list *list)
+{
+	int i;
+
+	memset(list, 0, sizeof(*list));
+	for (i = 0; i < CPU_SETSIZE; i++) {
+		struct percpu_list_node *node;
+
+		node = malloc(sizeof(*node));
+		assert(node);
+		node->data = 0;
+		node->next = NULL;
+		list->c[i].head = node;
+	}
+}
+
+void test_membarrier_free_percpu_list(struct percpu_list *list)
+{
+	int i;
+
+	for (i = 0; i < CPU_SETSIZE; i++)
+		free(list->c[i].head);
+}
+
+static int sys_membarrier(int cmd, int flags)
+{
+	return syscall(__NR_membarrier, cmd, flags);
+}
+
+/*
+ * The manager thread swaps per-cpu lists that worker threads see,
+ * and validates that there are no unexpected modifications.
+ */
+void *test_membarrier_manager_thread(void *arg)
+{
+	struct test_membarrier_thread_args *args =
+		(struct test_membarrier_thread_args *)arg;
+	struct percpu_list list_a, list_b;
+	intptr_t expect_a = 0, expect_b = 0;
+	int cpu_a = 0, cpu_b = 0;
+
+	if (rseq_register_current_thread()) {
+		fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s\n",
+			errno, strerror(errno));
+		abort();
+	}
+
+	/* Init lists. */
+	test_membarrier_init_percpu_list(&list_a);
+	test_membarrier_init_percpu_list(&list_b);
+
+	atomic_store(&args->percpu_list_ptr, (intptr_t)&list_a);
+
+	while (!atomic_load(&args->stop)) {
+		/* list_a is "active". */
+		cpu_a = rand() % CPU_SETSIZE;
+		/*
+		 * As list_b is "inactive", we should never see changes
+		 * to list_b.
+		 */
+		if (expect_b != atomic_load(&list_b.c[cpu_b].head->data)) {
+			fprintf(stderr, "Membarrier test failed\n");
+			abort();
+		}
+
+		/* Make list_b "active". */
+		atomic_store(&args->percpu_list_ptr, (intptr_t)&list_b);
+		sys_membarrier(MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU, cpu_a);
+		/*
+		 * Cpu A should now only modify list_b, so we values
+		 * in list_a should be stable.
+		 */
+		expect_a = atomic_load(&list_a.c[cpu_a].head->data);
+
+		cpu_b = rand() % CPU_SETSIZE;
+		/*
+		 * As list_a is "inactive", we should never see changes
+		 * to list_a.
+		 */
+		if (expect_a != atomic_load(&list_a.c[cpu_a].head->data)) {
+			fprintf(stderr, "Membarrier test failed\n");
+			abort();
+		}
+
+		/* Make list_a "active". */
+		atomic_store(&args->percpu_list_ptr, (intptr_t)&list_a);
+		sys_membarrier(MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU, cpu_b);
+		/* Remember a value from list_b. */
+		expect_b = atomic_load(&list_b.c[cpu_b].head->data);
+	}
+
+	test_membarrier_free_percpu_list(&list_a);
+	test_membarrier_free_percpu_list(&list_b);
+
+	if (rseq_unregister_current_thread()) {
+		fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): %s\n",
+			errno, strerror(errno));
+		abort();
+	}
+	return NULL;
+}
+
+/* Test MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU membarrier command. */
+void test_membarrier(void)
+{
+	struct test_membarrier_thread_args thread_args;
+	pthread_t worker_threads[CPU_SETSIZE];
+	pthread_t manager_thread;
+	int i;
+
+	thread_args.stop = 0;
+	thread_args.percpu_list_ptr = 0;
+	pthread_create(&manager_thread, NULL,
+		       test_membarrier_manager_thread, &thread_args);
+
+	for (i = 0; i < CPU_SETSIZE; i++)
+		pthread_create(&worker_threads[i], NULL,
+		       test_membarrier_worker_thread, &thread_args);
+
+	for (i = 0; i < CPU_SETSIZE; i++)
+		pthread_join(worker_threads[i], NULL);
+
+	atomic_store(&thread_args.stop, 1);
+	pthread_join(manager_thread, NULL);
+}
+
 int main(int argc, char **argv)
 {
 	if (rseq_register_current_thread()) {
@@ -300,6 +479,8 @@ int main(int argc, char **argv)
 	test_percpu_spinlock();
 	printf("percpu_list\n");
 	test_percpu_list();
+	printf("membarrier\n");
+	test_membarrier();
 	if (rseq_unregister_current_thread()) {
 		fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): %s\n",
 			errno, strerror(errno));
Signature · 1 line
2.28.0.163.g6104cc2f0b6-goog
#2 lore raw
In reply to Peter Oskolkov

On Wed, Aug 05, 2020 at 05:08:58PM -0700, Peter Oskolkov wrote:

Thanks for the Cc!

kernel/sched/membarrier.c static int membarrier_register_private_expedited(int flags) · line 345

Quoted 11 lines

+ * @MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU:

+ *                          If a thread belonging to the current process
+ *                          is currently in an RSEQ critical section on the
+ *                          CPU identified by flags parameter, restart it.
+ *                          @flags: if @flags >= 0, identifies the CPU to

+ * restart RSEQ CS on; if == -1, restarts
+ * RSEQ CSs on all CPUs.

kernel/sched/membarrier.c +3 -0
+	} else if (cpu_id == -1) {
+		on_each_cpu(membarrier_rseq_ipi,
+			    current->group_leader, true);

This is an unpriv IPI the world. That's a big no-no.

Double so because all you want to target is the current process, which you're defining as CLONE_THREAD, where the rest of this file uses CLONE_VM to define a process.

#3 lore raw
Quoted 4 lines · On Thu, Aug 6, 2020 at 6:48 AM <[email protected]> wrote:

On Wed, Aug 05, 2020 at 05:08:58PM -0700, Peter Oskolkov wrote:

Thanks for the Cc!

Always a pleasure!

(Sorry, included only membarrier maintainers in v1; in v2 included both membarrier and rseq maintainers).

Quoted 9 lines
Quoted 8 lines

+ * @MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU:

+ *                          If a thread belonging to the current process
+ *                          is currently in an RSEQ critical section on the
+ *                          CPU identified by flags parameter, restart it.
+ *                          @flags: if @flags >= 0, identifies the CPU to

+ * restart RSEQ CS on; if == -1, restarts
+ * RSEQ CSs on all CPUs.

Quoted 5 lines

kernel/sched/membarrier.c static int membarrier_register_private_expedited(int flags) · line 345

Quoted 3 lines
kernel/sched/membarrier.c +3 -0
+     } else if (cpu_id == -1) {
+             on_each_cpu(membarrier_rseq_ipi,
+                         current->group_leader, true);

This is an unpriv IPI the world. That's a big no-no.

removed in v2.

Quoted 4 lines

Double so because all you want to target is the current process, which you're defining as CLONE_THREAD, where the rest of this file uses CLONE_VM to define a process.

Use current->mm in v2 instead of current->group_leader. Is it better this way?

#4 lore raw
Quoted 20 lines · ----- On Aug 6, 2020, at 1:07 PM, Peter Oskolkov [email protected] wrote:
Quoted 4 lines · On Thu, Aug 6, 2020 at 6:48 AM <[email protected]> wrote:

On Wed, Aug 05, 2020 at 05:08:58PM -0700, Peter Oskolkov wrote:

Thanks for the Cc!

Always a pleasure!

(Sorry, included only membarrier maintainers in v1; in v2 included both membarrier and rseq maintainers).

Quoted 9 lines
Quoted 8 lines

+ * @MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU:

+ *                          If a thread belonging to the current process
+ *                          is currently in an RSEQ critical section on the
+ *                          CPU identified by flags parameter, restart it.
+ *                          @flags: if @flags >= 0, identifies the CPU to

+ * restart RSEQ CS on; if == -1, restarts
+ * RSEQ CSs on all CPUs.

Quoted 7 lines
Quoted 5 lines

kernel/sched/membarrier.c static int membarrier_register_private_expedited(int flags) · line 345

Quoted 3 lines
kernel/sched/membarrier.c +3 -0
+     } else if (cpu_id == -1) {
+             on_each_cpu(membarrier_rseq_ipi,
+                         current->group_leader, true);

This is an unpriv IPI the world. That's a big no-no.

removed in v2.

I don't think the feature must be removed, but its implementation needs adjustment.

How about we simply piggy-back on the membarrier schemes we already have, and implement:

membarrier_register_private_expedited(MEMBARRIER_FLAG_RSEQ) membarrier_private_expedited(MEMBARRIER_FLAG_RSEQ)

All the logic is there to prevent sending IPIs to runqueues which are not running threads associated with the same mm. Considering that preemption does an rseq abort, running a thread belonging to a different mm should mean that this CPU is not currently executing an rseq critical section, or if it was, it has already been aborted, so it is quiescent.

Then you'll probably want to change membarrier_private_expedited so it takes an extra "cpu" argument. If cpu=-1, iterate on all runqueues like we currently do. If cpu >= 0, only IPI that CPU if the thread currently running has the same mm.

Also, should this belong to the membarrier or the rseq system call ? It just looks like the membarrier happens to implement very similar things for barriers, but arguably this is really about rseq. I wonder if we should expose this through rseq instead, even if we end up using membarrier code.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#5 lore raw
Quoted 21 lines · Mathieu Desnoyers
Quoted 6 lines
Quoted 4 lines

This is an unpriv IPI the world. That's a big no-no.

removed in v2.

I don't think the feature must be removed, but its implementation needs adjustment.

How about we simply piggy-back on the membarrier schemes we already have, and implement:

membarrier_register_private_expedited(MEMBARRIER_FLAG_RSEQ) membarrier_private_expedited(MEMBARRIER_FLAG_RSEQ)

All the logic is there to prevent sending IPIs to runqueues which are not running threads associated with the same mm. Considering that preemption does an rseq abort, running a thread belonging to a different mm should mean that this CPU is not currently executing an rseq critical section, or if it was, it has already been aborted, so it is quiescent.

Quoted 4 lines

Then you'll probably want to change membarrier_private_expedited so it takes an extra "cpu" argument. If cpu=-1, iterate on all runqueues like we currently do. If cpu >= 0, only IPI that CPU if the thread currently running has the same mm.

Thanks, Mathieu! I'll prepare something based on your and Peter's feedback.

Quoted 4 lines

Also, should this belong to the membarrier or the rseq system call ? It just looks like the membarrier happens to implement very similar things for barriers, but arguably this is really about rseq. I wonder if we should expose this through rseq instead, even if we end up using membarrier code.

Yes, this is more about rseq; on the other hand, the high-level API/behavior looks closer to that membarrier, and a lot of code will be shared.

As you are the maintainer for both rseq and membarrier, this is for you to decide, I guess... :)

#6 lore raw
Quoted 3 lines · ----- On Aug 7, 2020, at 1:48 PM, Peter Oskolkov [email protected] wrote:
Quoted 1 line · Mathieu Desnoyers

[...]

Quoted 11 lines
Quoted 5 lines

Also, should this belong to the membarrier or the rseq system call ? It just looks like the membarrier happens to implement very similar things for barriers, but arguably this is really about rseq. I wonder if we should expose this through
rseq instead, even if we end up using membarrier code.

Yes, this is more about rseq; on the other hand, the high-level API/behavior looks closer to that membarrier, and a lot of code will be shared.

As you are the maintainer for both rseq and membarrier, this is for you to decide, I guess... :)

Considering that membarrier has been made extensible with the cmd argument, and on the other hand rseq can be extended with "flags", but is currently only about registration/unregistration, I think adding a command to membarrier is indeed a natural approach.

I am not very fond on re-purposing the membarrier flags parameter into a cpu number though. Maybe we should tweak the membarrier system call so it can expect 3 arguments instead ?

int membarrier(int cmd, int flags, int cpu);

where cpu is only used for specific commands.

One thing I find weird about Peter's patch is that it adds a MEMBERRIER_CMD_PRIVATE_EXPEDITED_RSEQ without a corresponding MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ. Considering that the SYNC_CORE variant already has its own register command, I find it weird that the RSEQ counterpart does not have one.

Also, do we want to allow a RSEQ | SYNC_CORE private expedited membarrier as well ? If that is the case, then we might want to investigate exposing RSEQ-membarrier as a new membarrier flag rather than as a stand-alone command.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#7 lore raw
Quoted 5 lines · Mathieu Desnoyers

One thing I find weird about Peter's patch is that it adds a MEMBERRIER_CMD_PRIVATE_EXPEDITED_RSEQ without a corresponding MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ. Considering that the SYNC_CORE variant already has its own register command, I find it weird that the RSEQ counterpart does not have one.

I thought the register thing was about global, not private membarriers.

Anyway, it was just a quick pseudo thing to show how one can go about adding the rseq to the mm iteration we already have.