membarrier: allow cpu_id to be set on more commands
Subsystems (3)
cover membarrier: allow cpu_id to be set on more commands
Hi,
I noticed that only the RSEQ membarrier command allows specifying a specific cpu. I have a (extremely toy) lib I was playing around with [1] and noticed this and that being able to specify the cpu would be useful to me.
I'm by no means an expert in this code though - and so could be totally missing something.
Additionally this seems really difficult to actually test. I added a self test that just proces "some" interrupts are being sent, but nothing more than that. I don't know what else can be done there.
Patch 1 is the main change
Patch 2 is the self-test. Which maybe you don't want at all.
[1]: https://github.com/DylanZA/rseqlock/commit/be7bc7214fd5aacec47e26126118f8bbdda0f0ea
Thanks,
Dylan
Dylan Yudaken (2):
membarrier: allow cpu_id to be set on more commands
membarrier: self test for cpu specific calls
kernel/sched/membarrier.c | 6 +- tools/testing/selftests/membarrier/.gitignore | 1 + tools/testing/selftests/membarrier/Makefile | 3 +- .../membarrier/membarrier_test_expedited.c | 135 ++++++++++++++++++ .../membarrier/membarrier_test_impl.h | 5 + 5 files changed, 148 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/membarrier/membarrier_test_expedited.c
base-commit: ee88bddf7f2f5d1f1da87dd7bedc734048b70e88
Signature · 1 line
2.49.0
PATCH 1/2 membarrier: allow cpu_id to be set on more commands
No reason to not allow MEMBARRIER_CMD_FLAG_CPU onMEMBARRIER_CMD_PRIVATE_EXPEDITED orMEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE.
If it is known specifically what cpu you want to interrupt then there is a decent efficiency saving in not interrupting all the other ones.
Also - the code already works as is for them.
--- kernel/sched/membarrier.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/sched/membarrier.c +5 -1
@@ -595,7 +595,9 @@ static int membarrier_get_registrations(void)* contains the CPU on which to interrupt (= restart) * the RSEQ critical section. * @cpu_id: if @flags == MEMBARRIER_CMD_FLAG_CPU, indicates the cpu on which -* RSEQ CS should be interrupted (@cmd must be +* RSEQ CS should be interrupted (@cmd must be one of +* MEMBARRIER_CMD_PRIVATE_EXPEDITED, +* MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE, * MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ). * * If this system call is not implemented, -ENOSYS is returned. If the
@@ -625,6 +627,8 @@ static int membarrier_get_registrations(void) SYSCALL_DEFINE3(membarrier, int, cmd, unsigned int, flags, int, cpu_id) {switch (cmd) { +case MEMBARRIER_CMD_PRIVATE_EXPEDITED : +case MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE :case MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ :if (unlikely (flags && flags !=MEMBARRIER_CMD_FLAG_CPU ))return -EINVAL ;
Signature · 1 line
2.49.0
Mathieu Desnoyers
maintainerQuoted 8 lines · Dylan Yudaken
No reason to not allow
MEMBARRIER_CMD_FLAG_CPUonMEMBARRIER_CMD_PRIVATE_EXPEDITEDorMEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE.If it is known specifically what cpu you want to interrupt then there is a decent efficiency saving in not interrupting all the other ones.
Also - the code already works as is for them.
Can you elaborate on a concrete use-case justifying adding this ?
Thanks,
Mathieu
Quoted 30 lines
Signed-off-by Dylan Yudaken <[email protected]>--- kernel/sched/membarrier.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)kernel/sched/membarrier.c +5
-1@@ -595,7 +595,9 @@ static int membarrier_get_registrations(void)* contains the CPU on which to interrupt (= restart) * the RSEQ critical section. * @cpu_id: if @flags == MEMBARRIER_CMD_FLAG_CPU, indicates the cpu on which -* RSEQ CS should be interrupted (@cmd must be +* RSEQ CS should be interrupted (@cmd must be one of +* MEMBARRIER_CMD_PRIVATE_EXPEDITED, +* MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE, * MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ). * * If this system call is not implemented, -ENOSYS is returned. If the @@ -625,6 +627,8 @@ static int membarrier_get_registrations(void) SYSCALL_DEFINE3(membarrier, int, cmd, unsigned int, flags, int, cpu_id) {switch (cmd) { +case MEMBARRIER_CMD_PRIVATE_EXPEDITED : +case MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE :case MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ :if (unlikely (flags && flags !=MEMBARRIER_CMD_FLAG_CPU ))return -EINVAL ;
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Quoted 9 lines · Mathieu Desnoyers
Quoted 7 lines · Dylan Yudaken
No reason to not allow
MEMBARRIER_CMD_FLAG_CPUonMEMBARRIER_CMD_PRIVATE_EXPEDITEDorMEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE.If it is known specifically what cpu you want to interrupt then there is a decent efficiency saving in not interrupting all the other ones.
Quoted 8 lines
Quoted 1 line
Also - the code already works as is for them.
Can you elaborate on a concrete use-case justifying adding this ?
Thanks,
Mathieu
So my use case is for core-local data such as performance counters.
I have a library that allows a fast thread to "lock" a core -> do some work (probably incrementing some performance counters) -> unlock. The "lock" uses restartable sequences (ie no serializing instructions), and the unlock just writes a 0 to memory (again, no serializing instructions).
A slow thread will occasionally (say every few minutes) try and read data computed in the work section.
It does this by disabling locking and firing off a membarrier(RSEQ) on that core to be sure that the core is either "locked" or "unlocked". It then spins waiting for it to be unlocked.
At this point my understanding is a bit fuzzy - but I believe you need that core to have a memory barrier since there is no serializing instruction and the processor would happily reorder some "work" after the "unlock" instruction.
That serializing instruction is what I want from this. But since I know the cpu_id that I am working with I don't need to do a barrier on _all_ the cores.
To be clear: (1) I don't have a current real world use case, and (2) my library/design/understanding might be buggy.
(3) I don't have a use case for the SYNC_CORE part, but again it seemed easy enough to add and I presume others might have a use case.
PATCH 2/2 membarrier: self test for cpu specific calls
In reply to Dylan YudakenAdd a self test for the cpu specific calls to membarrier.
This works by figuring out the number of interrupts on a given core before/after calling membarrier(2) to assert that at least some interrupts have happened.
This feels like it might be a bit flaky if for example the worker thread was switched out. To mitigate this there are some checks such as making sure it stays on one core, and also it asserts only 1 interrupt for every 2 calls to membarrier(2)
--- tools/testing/selftests/membarrier/.gitignore | 1 + tools/testing/selftests/membarrier/Makefile | 3 +- .../membarrier/membarrier_test_expedited.c | 135 ++++++++++++++++++ .../membarrier/membarrier_test_impl.h | 5 + 4 files changed, 143 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/membarrier/membarrier_test_expedited.c
tools/testing/selftests/membarrier/.gitignore +1 -0
@@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only membarrier_test_multi_thread membarrier_test_single_thread +membarrier_test_expedited
tools/testing/selftests/membarrier/Makefile +2 -1
@@ -3,6 +3,7 @@ CFLAGS += -g $(KHDR_INCLUDES) LDLIBS += -lpthread TEST_GEN_PROGS := membarrier_test_single_thread \ - membarrier_test_multi_thread + membarrier_test_multi_thread \ + membarrier_test_expedited include ../lib.mk
tools/testing/selftests/membarrier/membarrier_test_expedited.c new file +135 -0
135 lines not on this page · raw message
tools/testing/selftests/membarrier/membarrier_test_impl.h +5 -0
@@ -16,6 +16,11 @@ static int sys_membarrier(int cmd, int flags)return syscall (__NR_membarrier, cmd, flags); } +static int sys_membarrier_cpu (int cmd,int cpu) +{ +return syscall (__NR_membarrier, cmd,MEMBARRIER_CMD_FLAG_CPU , cpu); +} +static int test_membarrier_get_registrations (int cmd) {int ret, flags =0 ;
Signature · 1 line
2.49.0