Restartable sequences and CPU op vector
Subsystems (16)
Mathieu Desnoyers
maintainerv12 cover Restartable sequences and CPU op vector
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
This series contains:
- Restartable sequences system call (x86 32/64, powerpc 32/64, arm 32),
- CPU operation vector system call (x86 32/64, powerpc 32/64, arm 32),
- membarrier shared expedited command.
Compared to v11, I've removed the "sync core" membarrier command, now queued for 4.16.
I have also fixed a missing page fault-in in cpu_opv, and added a selftest test-case to cover this.
This series applies on top of current Linus' master as of
commit e1d1ea549b57 "Merge tag 'fbdev-v4.15' of git://github.com/bzolnier/linux"
The git tag including this series can be found at
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git
tag: v4.14+-rseq-20171121
Thanks,
Mathieu
Boqun Feng (2):
powerpc: Add support for restartable sequences
powerpc: Wire up restartable sequences system call
Mathieu Desnoyers (20):
uapi headers: Provide types_32_64.h
rseq: Introduce restartable sequences system call (v12)
arm: Add restartable sequences support
arm: Wire up restartable sequences system call
x86: Add support for restartable sequences
x86: Wire up restartable sequence system call
sched: Implement push_task_to_cpu
cpu_opv: Provide cpu_opv system call (v4)
x86: Wire up cpu_opv system call
powerpc: Wire up cpu_opv system call
arm: Wire up cpu_opv system call
cpu_opv: selftests: Implement selftests (v3)
rseq: selftests: Provide self-tests (v3)
rseq: selftests: arm: workaround gcc asm size guess
Fix: membarrier: add missing preempt off around smp_call_function_many
membarrier: selftest: Test private expedited cmd (v2)
powerpc: membarrier: Skip memory barrier in switch_mm() (v7)
membarrier: Document scheduler barrier requirements (v5)
membarrier: provide SHARED_EXPEDITED command (v2)
membarrier: selftest: Test shared expedited cmd
MAINTAINERS | 21 + arch/Kconfig | 7 + arch/arm/Kconfig | 1 + arch/arm/kernel/signal.c | 7 + arch/arm/tools/syscall.tbl | 2 + arch/powerpc/Kconfig | 2 + arch/powerpc/include/asm/membarrier.h | 26 + arch/powerpc/include/asm/systbl.h | 2 + arch/powerpc/include/asm/unistd.h | 2 +- arch/powerpc/include/uapi/asm/unistd.h | 2 + arch/powerpc/kernel/signal.c | 3 + arch/powerpc/mm/mmu_context.c | 7 + arch/x86/Kconfig | 1 + arch/x86/entry/common.c | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 2 + arch/x86/entry/syscalls/syscall_64.tbl | 2 + arch/x86/kernel/signal.c | 6 + arch/x86/mm/tlb.c | 5 + fs/exec.c | 1 + include/linux/sched.h | 102 ++ include/linux/sched/mm.h | 21 +- include/linux/syscalls.h | 6 + include/trace/events/rseq.h | 56 + include/uapi/linux/cpu_opv.h | 114 ++ include/uapi/linux/membarrier.h | 34 +- include/uapi/linux/rseq.h | 141 +++ include/uapi/linux/types_32_64.h | 67 + init/Kconfig | 31 + kernel/Makefile | 2 + kernel/cpu_opv.c | 1060 ++++++++++++++++ kernel/fork.c | 2 + kernel/rseq.c | 338 +++++ kernel/sched/core.c | 88 +- kernel/sched/membarrier.c | 125 +- kernel/sched/sched.h | 9 + kernel/sys_ni.c | 4 + tools/testing/selftests/Makefile | 2 + tools/testing/selftests/cpu-opv/.gitignore | 1 + tools/testing/selftests/cpu-opv/Makefile | 17 + .../testing/selftests/cpu-opv/basic_cpu_opv_test.c | 1189 ++++++++++++++++++ tools/testing/selftests/cpu-opv/cpu-op.c | 348 ++++++ tools/testing/selftests/cpu-opv/cpu-op.h | 68 ++ tools/testing/selftests/lib.mk | 4 + .../testing/selftests/membarrier/membarrier_test.c | 162 ++- tools/testing/selftests/rseq/.gitignore | 4 + tools/testing/selftests/rseq/Makefile | 23 + .../testing/selftests/rseq/basic_percpu_ops_test.c | 333 +++++ tools/testing/selftests/rseq/basic_test.c | 55 + tools/testing/selftests/rseq/param_test.c | 1285 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 568 +++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 567 +++++++++ tools/testing/selftests/rseq/rseq-x86.h | 898 ++++++++++++++ tools/testing/selftests/rseq/rseq.c | 116 ++ tools/testing/selftests/rseq/rseq.h | 154 +++ tools/testing/selftests/rseq/run_param_test.sh | 124 ++ 55 files changed, 8166 insertions(+), 52 deletions(-) create mode 100644 arch/powerpc/include/asm/membarrier.h create mode 100644 include/trace/events/rseq.h create mode 100644 include/uapi/linux/cpu_opv.h create mode 100644 include/uapi/linux/rseq.h create mode 100644 include/uapi/linux/types_32_64.h create mode 100644 kernel/cpu_opv.c create mode 100644 kernel/rseq.c create mode 100644 tools/testing/selftests/cpu-opv/.gitignore create mode 100644 tools/testing/selftests/cpu-opv/Makefile create mode 100644 tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.h create mode 100644 tools/testing/selftests/rseq/.gitignore create mode 100644 tools/testing/selftests/rseq/Makefile create mode 100644 tools/testing/selftests/rseq/basic_percpu_ops_test.c create mode 100644 tools/testing/selftests/rseq/basic_test.c create mode 100644 tools/testing/selftests/rseq/param_test.c create mode 100644 tools/testing/selftests/rseq/rseq-arm.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc.h create mode 100644 tools/testing/selftests/rseq/rseq-x86.h create mode 100644 tools/testing/selftests/rseq/rseq.c create mode 100644 tools/testing/selftests/rseq/rseq.h create mode 100755 tools/testing/selftests/rseq/run_param_test.sh
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 1/22 uapi headers: Provide types_32_64.h
Provide helper macros for fields which represent pointers in kernel-userspace ABI. This facilitates handling of 32-bit user-space by 64-bit kernels by defining those fields as 32-bit 0-padding and 32-bit integer on 32-bit architectures, which allows the kernel to treat those as 64-bit integers. The order of padding and 32-bit integer depends on the endianness.
--- include/uapi/linux/types_32_64.h | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 include/uapi/linux/types_32_64.h
include/uapi/linux/types_32_64.h new file +67 -0
@@ -0,0 +1,67 @@ +#ifndef _UAPI_LINUX_TYPES_32_64_H +#define _UAPI_LINUX_TYPES_32_64_H + +/* +* linux/types_32_64.h +* +* Integer type declaration for pointers across 32-bit and 64-bit systems. +* +* Copyright (c) 2015-2017 Mathieu Desnoyers <[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#ifdef __KERNEL__ +# include <linux/types.h> +#else +# include <stdint.h> +#endif + +#include <asm/byteorder.h> + +#ifdef __BYTE_ORDER +# if (__BYTE_ORDER == __BIG_ENDIAN) +# define LINUX_BYTE_ORDER_BIG_ENDIAN +# else +# define LINUX_BYTE_ORDER_LITTLE_ENDIAN +# endif +#else +# ifdef __BIG_ENDIAN +# define LINUX_BYTE_ORDER_BIG_ENDIAN +# else +# define LINUX_BYTE_ORDER_LITTLE_ENDIAN +# endif +#endif + +#ifdef __LP64__ +# define LINUX_FIELD_u32_u64 (field) uint64_t field +# define LINUX_FIELD_u32_u64_INIT_ONSTACK (field, v) field = (intptr_t)v +#else +# ifdef LINUX_BYTE_ORDER_BIG_ENDIAN +# define LINUX_FIELD_u32_u64 (field) uint32_t field ## _padding, field +# define LINUX_FIELD_u32_u64_INIT_ONSTACK (field, v) \ + field ## _padding = 0, field = (intptr_t)v +# else +# define LINUX_FIELD_u32_u64 (field) uint32_t field, field ## _padding +# define LINUX_FIELD_u32_u64_INIT_ONSTACK (field, v) \ + field = (intptr_t)v, field ## _padding = 0 +# endif +#endif + +#endif /* _UAPI_LINUX_TYPES_32_64_H */
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 2/22 rseq: Introduce restartable sequences system call
In reply to Mathieu DesnoyersExpose a new system call allowing each thread to register one userspace memory area to be used as an ABI between kernel and user-space for two
purposes: user-space restartable sequences and quick access to read the current CPU number value from user-space.
* Restartable sequences (per-cpu atomics)
Restartables sequences allow user-space to perform update operations on per-cpu data without requiring heavy-weight atomic operations.
The restartable critical sections (percpu atomics) work has been started by Paul Turner and Andrew Hunter. It lets the kernel handle restart of critical sections. [1] [2] The re-implementation proposed here brings a few simplifications to the ABI which facilitates porting to other architectures and speeds up the user-space fast path. A second system call, cpu_opv(), is proposed as fallback to deal with debugger single-stepping. cpu_opv() executes a sequence of operations on behalf of user-space with preemption disabled.
Here are benchmarks of various rseq use-cases.
Test hardware:
arm32: ARMv7 Processor rev 4 (v7l) "Cubietruck", 2-core
x86-64: Intel E5-2630 [email protected], 16-core, hyperthreading
The following benchmarks were all performed on a single thread.
* Per-CPU statistic counter increment
getcpu+atomic (ns/op) rseq (ns/op) speedup
arm32: 344.0 31.4 11.0 x86-64: 15.3 2.0 7.7
* LTTng-UST: write event 32-bit header, 32-bit payload into tracer per-cpu buffer
getcpu+atomic (ns/op) rseq (ns/op) speedup
arm32: 2502.0 2250.0 1.1 x86-64: 117.4 98.0 1.2
* liburcu percpu: lock-unlock pair, dereference, read/compare word
getcpu+atomic (ns/op) rseq (ns/op) speedup
arm32: 751.0 128.5 5.8 x86-64: 53.4 28.6 1.9
* jemalloc memory allocator adapted to use rseq
Using rseq with per-cpu memory pools in jemalloc at Facebook (based on rseq 2016 implementation):
The production workload response-time has 1-2% gain avg. latency, and the P99 overall latency drops by 2-3%.
* Reading the current CPU number
Speeding up reading the current CPU number on which the caller thread is running is done by keeping the current CPU number up do date within the cpu_id field of the memory area registered by the thread. This is done by making scheduler preemption set the TIF_NOTIFY_RESUME flag on the current thread. Upon return to user-space, a notify-resume handler updates the current CPU value within the registered user-space memory area. User-space can then read the current CPU number directly from memory.
Keeping the current cpu id in a memory area shared between kernel and user-space is an improvement over current mechanisms available to read the current CPU number, which has the following benefits over alternative approaches:
- 35x speedup on ARM vs system call through glibc
- 20x speedup on x86 compared to calling glibc, which calls vdso executing a "lsl" instruction,
- 14x speedup on x86 compared to inlined "lsl" instruction,
- Unlike vdso approaches, this cpu_id value can be read from an inline assembly, which makes it a useful building block for restartable sequences.
- The approach of reading the cpu id through memory mapping shared between kernel and user-space is portable (e.g. ARM), which is not the case for the lsl-based x86 vdso.
On x86, yet another possible approach would be to use the gs segment selector to point to user-space per-cpu data. This approach performs similarly to the cpu id cache, but it has two disadvantages: it is not portable, and it is incompatible with existing applications already using the gs segment selector for other purposes.
Benchmarking various approaches for reading the current CPU number:
ARMv7 Processor rev 4 (v7l)
Machine model: Cubietruck
- Baseline (empty loop): 8.4 ns
- Read CPU from rseq cpu_id: 16.7 ns
- Read CPU from rseq cpu_id (lazy register): 19.8 ns
- glibc 2.19-0ubuntu6.6 getcpu: 301.8 ns
- getcpu system call: 234.9 ns
x86-64 Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz:
- Baseline (empty loop): 0.8 ns - Read CPU from rseq cpu_id: 0.8 ns - Read CPU from rseq cpu_id (lazy register): 0.8 ns - Read using gs segment selector: 0.8 ns - "lsl" inline assembly: 13.0 ns - glibc 2.19-0ubuntu6 getcpu: 16.6 ns - getcpu system call: 53.9 ns
- Speed (benchmark taken on v8 of patchset)
Running 10 runs of hackbench -l 100000 seems to indicate, contrary to expectations, that enabling CONFIG_RSEQ slightly accelerates the scheduler:
Configuration: 2 sockets * 8-core Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (directly on hardware, hyperthreading disabled in BIOS, energy saving disabled in BIOS, turboboost disabled in BIOS, cpuidle.off=1 kernel parameter), with a Linux v4.6 defconfig+localyesconfig, restartable sequences series applied.
* CONFIG_RSEQ=n
avg.: 41.37 s std.dev.: 0.36 s
* CONFIG_RSEQ=y
avg.: 40.46 s std.dev.: 0.33 s
- Size
On x86-64, between CONFIG_RSEQ=n/y, the text size increase of vmlinux is 567 bytes, and the data size increase of vmlinux is 5696 bytes.
On x86-64, between CONFIG_CPU_OPV=n/y, the text size increase of vmlinux is 5576 bytes, and the data size increase of vmlinux is 6164 bytes.
[1] https://lwn.net/Articles/650333/
[2] http://www.linuxplumbersconf.org/2013/ocw/system/presentations/1695/original/LPC%20-%20PerCpu%20Atomics.pdf
---
Changes since v1:
- Return -1, errno=EINVAL if cpu_cache pointer is not aligned on sizeof(int32_t).
- Update man page to describe the pointer alignement requirements and update atomicity guarantees.
- Add MAINTAINERS file GETCPU_CACHE entry.
- Remove dynamic memory allocation: go back to having a single getcpu_cache entry per thread. Update documentation accordingly.
- Rebased on Linux 4.4.
Changes since v2:
- Introduce a "cmd" argument, along with an enum with GETCPU_CACHE_GET and GETCPU_CACHE_SET. Introduce a uapi header linux/getcpu_cache.h defining this enumeration.
- Split resume notifier architecture implementation from the system call wire up in the following arch-specific patches.
- Man pages updates.
- Handle 32-bit compat pointers.
- Simplify handling of getcpu_cache GETCPU_CACHE_SET compiler barrier:
set the current cpu cache pointer before doing the cache update, and set it back to NULL if the update fails. Setting it back to NULL on error ensures that no resume notifier will trigger a SIGSEGV if a migration happened concurrently.
Changes since v3:
- Fix __user annotations in compat code,
- Update memory ordering comments.
- Rebased on kernel v4.5-rc5.
Changes since v4:
- Inline getcpu_cache_fork, getcpu_cache_execve, and getcpu_cache_exit.
- Add new line between if() and switch() to improve readability.
- Added sched switch benchmarks (hackbench) and size overhead comparison to change log.
Changes since v5:
- Rename "getcpu_cache" to "thread_local_abi", allowing to extend this system call to cover future features such as restartable critical sections. Generalizing this system call ensures that we can add features similar to the cpu_id field within the same cache-line without having to track one pointer per feature within the task struct.
- Add a tlabi_nr parameter to the system call, thus allowing to extend the ABI beyond the initial 64-byte structure by registering structures with tlabi_nr greater than 0. The initial ABI structure is associated with tlabi_nr 0.
- Rebased on kernel v4.5.
Changes since v6:
- Integrate "restartable sequences" v2 patchset from Paul Turner.
- Add handling of single-stepping purely in user-space, with a fallback to locking after 2 rseq failures to ensure progress, and by exposing a __rseq_table section to debuggers so they know where to put breakpoints when dealing with rseq assembly blocks which can be aborted at any point.
- make the code and ABI generic: porting the kernel implementation simply requires to wire up the signal handler and return to user-space hooks, and allocate the syscall number.
- extend testing with a fully configurable test program. See param_spinlock_test -h for details.
- handling of rseq ENOSYS in user-space, also with a fallback to locking.
- modify Paul Turner's rseq ABI to only require a single TLS store on the user-space fast-path, removing the need to populate two additional registers. This is made possible by introducing struct rseq_cs into the ABI to describe a critical section start_ip, post_commit_ip, and abort_ip.
- Rebased on kernel v4.7-rc7.
Changes since v7:
- Documentation updates.
- Integrated powerpc architecture support.
- Compare rseq critical section start_ip, allows shriking the user-space fast-path code size.
- Added Peter Zijlstra, Paul E. McKenney and Boqun Feng as co-maintainers.
- Added do_rseq2 and do_rseq_memcpy to test program helper library.
- Code cleanup based on review from Peter Zijlstra, Andy Lutomirski and Boqun Feng.
- Rebase on kernel v4.8-rc2.
Changes since v8:
- clear rseq_cs even if non-nested. Speeds up user-space fast path by removing the final "rseq_cs=NULL" assignment.
- add enum rseq_flags: critical sections and threads can set migration, preemption and signal "disable" flags to inhibit rseq behavior.
- rseq_event_counter needs to be updated with a pre-increment: Otherwise misses an increment after exec (when TLS and in-kernel states are initially 0).
Changes since v9:
- Update changelog.
- Fold instrumentation patch.
- check abort-ip signature: Add a signature before the abort-ip landing address. This signature is also received as a new parameter to the rseq system call. The kernel uses it ensures that rseq cannot be used as an exploit vector to redirect execution to arbitrary code.
- Use rseq pointer for both register and unregister. This is more symmetric, and eventually allow supporting a linked list of rseq struct per thread if needed in the future.
- Unregistration of a rseq structure is now done with RSEQ_FLAG_UNREGISTER.
- Remove reference counting. Return "EBUSY" to the caller if rseq is already registered for the current thread. This simplifies implementation while still allowing user-space to perform lazy registration in multi-lib use-cases. (suggested by Ben Maurer)
- Clear rseq_cs upon unregister.
- Set cpu_id back to -1 on unregister, so if rseq user libraries follow an unregister, and they expect to lazily register rseq, they can do so.
- Document rseq_cs clear requirement: JIT should reset the rseq_cs pointer before reclaiming memory of rseq_cs structure.
- Introduce rseq_len syscall parameter, rseq_cs version field:
Allow keeping track of the registered rseq struct length, for future extensions. Add rseq_cs version as first field. Will allow future extensions.
- Use offset and unsigned arithmetic to save a branch: Save a conditional branch when comparing instruction pointer against a rseq_cs descriptor's address range by having post_commit_ip as an offset from start_ip, and using unsigned integer comparison. Suggested by Ben Maurer.
- Remove event counter from ABI. Suggested by Andy Lutomirski.
- Add INIT_ONSTACK macro: Introduce the RSEQ_FIELD_u32_u64_INIT_ONSTACK() macros to ensure that users correctly initialize the upper bits of RSEQ_FIELD_u32_u64() on their stack to 0 on 32-bit architectures.
- Select MEMBARRIER: Allows user-space rseq fast-paths to use the value of cpu_id field (inherently required by the rseq algorithm) to figure out whether membarrier can be expected to be available. This effectively allows user-space fast-paths to remove extra comparisons and branch testing whether membarrier is enabled, and thus whether a full barrier is required (e.g. in userspace RCU implementation after rcu_read_lock/before rcu_read_unlock).
- Expose cpu_id_start field: Checking whether the (cpu_id < 0) in the C preparation part of the rseq fast-path brings significant overhead at least on arm32. We can remove this extra comparison by exposing two distinct cpu_id fields in the rseq TLS:
The field cpu_id_start always contain a *possible* cpu number, although it may not be the current one if, for instance, rseq is not initialized for the current thread. cpu_id_start is meant to be used in the C code for the pointer chasing to figure out which per-cpu data structure should be passed to the rseq asm sequence.
The field cpu_id values -1 means rseq is not initialized, and -2 means initialization failed. That field is used in the rseq asm sequence to confirm that the cpu_id_start value was indeed the current cpu number. It also ends up confirming that rseq is initialized for the current thread, because values -1 and -2 will never match the cpu_id_start possible cpu number values.
This allows checking the current CPU number and rseq initialization state with a single comparison on the fast-path.
Changes since v10:
- Update rseq.c comment, removing reference to event_counter.
Changes since v11:
- Replace task struct rseq_preempt, rseq_signal, and rseq_migrate bool by u32 rseq_event_mask.
- Add missing sys_rseq() asmlinkage declaration to include/linux/syscalls.h.
- copy event mask on process fork, set to 0 on exec and thread-fork.
- Cleanups based on review from Peter Zijlstra.
- Cleanups based on review from Thomas Gleixner.
Man page associated:
RSEQ(2) Linux Programmer's Manual RSEQ(2)
NAME
rseq - Restartable sequences and cpu number cache
SYNOPSIS
#include <linux/rseq.h>
int rseq(struct rseq * rseq, uint32_t rseq_len, int flags, uint32_t sig);DESCRIPTION
The rseq() ABI accelerates user-space operations on per-cpu data by defining a shared data structure ABI between each user- space thread and the kernel.
It allows user-space to perform update operations on per-cpu data without requiring heavy-weight atomic operations.
Restartable sequences are atomic with respect to preemption (making it atomic with respect to other threads running on the same CPU), as well as signal delivery (user-space execution contexts nested over the same thread).
It is suited for update operations on per-cpu data.
It can be used on data structures shared between threads within a process, and on data structures shared between threads across different processes.
Some examples of operations that can be accelerated or improved by this ABI:
· Memory allocator per-cpu free-lists,
· Querying the current CPU number,
· Incrementing per-CPU counters,
· Modifying data protected by per-CPU spinlocks,
· Inserting/removing elements in per-CPU linked-lists,
· Writing/reading per-CPU ring buffers content.
· Accurately reading performance monitoring unit counters with
respect to thread migration.
The rseq argument is a pointer to the thread-local rseq struc‐ ture to be shared between kernel and user-space. A NULL rseq value unregisters the current thread rseq structure.
The layout of struct rseq is as follows:
Structure alignment
This structure is aligned on multiples of 32 bytes.
Structure size
This structure is extensible. Its size is passed as parameter to the rseq system call.
Fields
cpu_id_start
Optimistic cache of the CPU number on which the current thread is running. Its value is guaranteed to always be a possible CPU number, even when rseq is not initial‐ ized. The value it contains should always be confirmed by reading the cpu_id field.
cpu_id
Cache of the CPU number on which the current thread is running. -1 if uninitialized.
rseq_cs
The rseq_cs field is a pointer to a struct rseq_cs. Is is NULL when no rseq assembly block critical section is active for the current thread. Setting it to point to a critical section descriptor (struct rseq_cs) marks the beginning of the critical section.
flags
Flags indicating the restart behavior for the current thread. This is mainly used for debugging purposes. Can be either:
· RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT
· RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL
· RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE
The layout of struct rseq_cs version 0 is as follows:
Structure alignment
This structure is aligned on multiples of 32 bytes.
Structure size
This structure has a fixed size of 32 bytes.
Fields
version
Version of this structure.
flags
Flags indicating the restart behavior of this structure. Can be either:
· RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT
· RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL
· RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE
start_ip
Instruction pointer address of the first instruction of the sequence of consecutive assembly instructions.
post_commit_offset
Offset (from start_ip address) of the address after the last instruction of the sequence of consecutive assembly instructions.
abort_ip
Instruction pointer address where to move the execution flow in case of abort of the sequence of consecutive assembly instructions.
The rseq_len argument is the size of the struct rseq to regis‐ ter.
The flags argument is 0 for registration, and RSEQ_FLAG_UNREG‐ ISTER for unregistration.
The sig argument is the 32-bit signature to be expected before the abort handler code.
A single library per process should keep the rseq structure in a thread-local storage variable. The cpu_id field should be initialized to -1, and the cpu_id_start field should be ini‐ tialized to a possible CPU value (typically 0).
Each thread is responsible for registering and unregistering its rseq structure. No more than one rseq structure address can be registered per thread at a given time.
In a typical usage scenario, the thread registering the rseq structure will be performing loads and stores from/to that structure. It is however also allowed to read that structure from other threads. The rseq field updates performed by the kernel provide relaxed atomicity semantics, which guarantee that other threads performing relaxed atomic reads of the cpu number cache will always observe a consistent value.
RETURN VALUE
A return value of 0 indicates success. On error, -1 is returned, and errno is set appropriately.
ERRORS
EINVAL Either flags contains an invalid value, or rseq contains
an address which is not appropriately aligned, or rseq_len contains a size that does not match the size received on registration.
ENOSYS The rseq() system call is not implemented by this ker‐
nel.
EFAULT rseq is an invalid address.
EBUSY Restartable sequence is already registered for this
thread.
EPERM The sig argument on unregistration does not match the
signature received on registration.
VERSIONS
The rseq() system call was added in Linux 4.X (TODO).
CONFORMING TO
rseq() is Linux-specific.
SEE ALSO
sched_getcpu(3)
Linux 2017-11-06 RSEQ(2)
--- MAINTAINERS | 11 ++ arch/Kconfig | 7 + fs/exec.c | 1 + include/linux/sched.h | 102 +++++++++++++ include/linux/syscalls.h | 3 + include/trace/events/rseq.h | 56 ++++++++ include/uapi/linux/rseq.h | 141 ++++++++++++++++++ init/Kconfig | 14 ++ kernel/Makefile | 1 + kernel/fork.c | 2 + kernel/rseq.c | 338 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/core.c | 4 + kernel/sys_ni.c | 3 + 13 files changed, 683 insertions(+) create mode 100644 include/trace/events/rseq.h create mode 100644 include/uapi/linux/rseq.h create mode 100644 kernel/rseq.c
MAINTAINERS +11 -0
@@ -11616,6 +11616,17 @@ F: include/dt-bindings/reset/ F: include/linux/reset.h F: include/linux/reset-controller.h +RESTARTABLE SEQUENCES SUPPORT +M: Mathieu Desnoyers <[email protected]> +M: Peter Zijlstra <[email protected]> +M: "Paul E. McKenney" <[email protected]> +M: Boqun Feng <[email protected]> +L: [email protected] +S: Supported +F: kernel/rseq.c +F: include/uapi/linux/rseq.h +F: include/trace/events/rseq.h + RFKILL M: Johannes Berg <[email protected]> L: [email protected]
arch/Kconfig +7 -0
@@ -258,6 +258,13 @@ config HAVE_REGS_AND_STACK_ACCESS_API declared in asm/ptrace.h For example the kprobes-based event tracer needs this API. +config HAVE_RSEQ + bool + depends on HAVE_REGS_AND_STACK_ACCESS_API + help + This symbol should be selected by an architecture if it + supports an implementation of restartable sequences. + config HAVE_CLK bool help
fs/exec.c +1 -0
@@ -1803,6 +1803,7 @@ static int do_execveat_common(int fd, struct filename *filename, current->fs->in_exec =0 ; current->in_execve =0 ;membarrier_execve (current); +rseq_execve (current);acct_update_integrals (current);task_numa_free (current);free_bprm (bprm);
include/linux/sched.h +102 -0
118 lines not on this page · raw message
include/linux/syscalls.h +3 -0
@@ -66,6 +66,7 @@ struct old_linux_dirent;struct perf_event_attr ;struct file_handle ;struct sigaltstack ; +struct rseq ;union bpf_attr ;#include <linux/types.h>
@@ -940,5 +941,7 @@ asmlinkage long sys_pkey_alloc(unsigned long flags, unsigned long init_val);asmlinkage long sys_pkey_free (int pkey);asmlinkage long sys_statx (int dfd,const char __user *path,unsigned flags,unsigned mask,struct statx __user *buffer); +asmlinkage long sys_rseq (struct rseq __user *rseq,uint32_t rseq_len, +int flags,uint32_t sig);#endif
include/trace/events/rseq.h new file +56 -0
56 lines not on this page · raw message
include/uapi/linux/rseq.h new file +141 -0
141 lines not on this page · raw message
init/Kconfig +14 -0
@@ -1400,6 +1400,20 @@ config USERFAULTFD Enable the userfaultfd() system call that allows to intercept and handle page faults in userland. +config RSEQ + bool "Enable rseq() system call" if EXPERT + default y + depends on HAVE_RSEQ + select MEMBARRIER + help + Enable the restartable sequences system call. It provides a + user-space cache for the current CPU number value, which + speeds up getting the current CPU number from user-space, + as well as an ABI to speed up user-space operations on + per-CPU data. + + If unsure, say Y. + config EMBEDDED bool "Embedded system" option allnoconfig_y
kernel/Makefile +1 -0
@@ -112,6 +112,7 @@ obj-$(CONFIG_CONTEXT_TRACKING) += context_tracking.o obj-$(CONFIG_TORTURE_TEST) += torture.o obj-$(CONFIG_HAS_IOMEM) += memremap.o +obj-$(CONFIG_RSEQ) += rseq.o $(obj)/configs.o: $(obj)/config_data.h
kernel/fork.c +2 -0
@@ -1858,6 +1858,8 @@ static __latent_entropy struct task_struct *copy_process(*/ copy_seccomp (p); +rseq_fork (p, clone_flags); +/* * Process group and session signals need to be delivered to just the * parent before the fork or both the parent and the child after the
kernel/rseq.c new file +338 -0
338 lines not on this page · raw message
kernel/sched/core.c +4 -0
@@ -1188,6 +1188,8 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)WARN_ON_ONCE (!cpu_online (new_cpu));#endif +rseq_migrate (p); +trace_sched_migrate_task (p, new_cpu);if (task_cpu (p) != new_cpu) {
@@ -2590,6 +2592,7 @@ prepare_task_switch(struct rq *rq, struct task_struct *prev, {sched_info_switch (rq, prev, next);perf_event_task_sched_out (prev, next); +rseq_sched_out (prev);fire_sched_out_preempt_notifiers (prev, next);prepare_lock_switch (rq, next);prepare_arch_switch (next);
@@ -3350,6 +3353,7 @@ static void __sched notrace __schedule(bool preempt)clear_preempt_need_resched ();if (likely (prev != next)) { +rseq_preempt (prev); rq->nr_switches++; rq->curr = next;/*
kernel/sys_ni.c +3 -0
@@ -259,3 +259,6 @@ cond_syscall(sys_membarrier);cond_syscall (sys_pkey_mprotect);cond_syscall (sys_pkey_alloc);cond_syscall (sys_pkey_free); + +/* restartable sequence */ +cond_syscall (sys_rseq);
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 3/22 arm: Add restartable sequences support
In reply to Mathieu DesnoyersCall the rseq_handle_notify_resume() function on return to userspace if TIF_NOTIFY_RESUME thread flag is set.
Increment the event counter and perform fixup on the pre-signal frame when a signal is delivered on top of a restartable sequence critical section.
--- arch/arm/Kconfig | 1 + arch/arm/kernel/signal.c | 7 +++++++ 2 files changed, 8 insertions(+)
arch/arm/Kconfig +1 -0
@@ -88,6 +88,7 @@ config ARM select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE if (SMP && ARM_LPAE) select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_UID16 select HAVE_VIRT_CPU_ACCOUNTING_GEN
arch/arm/kernel/signal.c +7 -0
@@ -541,6 +541,12 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)int ret;/* +* Increment event counter and perform fixup for the pre-signal +* frame. +*/ +rseq_signal_deliver (regs); + +/* * Set up the stack frame */ if (ksig->ka.sa.sa_flags &SA_SIGINFO )
@@ -660,6 +666,7 @@ do_work_pending(struct pt_regs *regs, unsigned int thread_flags, int syscall) }else {clear_thread_flag (TIF_NOTIFY_RESUME );tracehook_notify_resume (regs); +rseq_handle_notify_resume (regs); } }local_irq_disable ();
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 4/22 arm: Wire up restartable sequences system call
In reply to Mathieu DesnoyersWire up the rseq system call on 32-bit ARM.
This provides an ABI improving the speed of a user-space getcpu operation on ARM by skipping the getcpu system call on the fast path, as well as improving the speed of user-space operations on per-cpu data compared to using load-linked/store-conditional.
--- arch/arm/tools/syscall.tbl | 1 + 1 file changed, 1 insertion(+)
arch/arm/tools/syscall.tbl +1 -0
@@ -412,3 +412,4 @@ 395 common pkey_alloc sys_pkey_alloc 396 common pkey_free sys_pkey_free 397 common statx sys_statx +398 common rseq sys_rseq
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 5/22 x86: Add support for restartable sequences
In reply to Mathieu DesnoyersCall the rseq_handle_notify_resume() function on return to userspace if TIF_NOTIFY_RESUME thread flag is set.
Increment the event counter and perform fixup on the pre-signal frame when a signal is delivered on top of a restartable sequence critical section.
--- arch/x86/Kconfig | 1 + arch/x86/entry/common.c | 1 + arch/x86/kernel/signal.c | 6 ++++++ 3 files changed, 8 insertions(+)
arch/x86/Kconfig +1 -0
@@ -174,6 +174,7 @@ config X86 select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && UNWINDER_FRAME_POINTER && STACK_VALIDATION select HAVE_STACK_VALIDATION if X86_64 + select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_UNSTABLE_SCHED_CLOCK select HAVE_USER_RETURN_NOTIFIER
arch/x86/entry/common.c +1 -0
@@ -160,6 +160,7 @@ static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags)if (cached_flags & _TIF_NOTIFY_RESUME) {clear_thread_flag (TIF_NOTIFY_RESUME );tracehook_notify_resume (regs); +rseq_handle_notify_resume (regs); }if (cached_flags & _TIF_USER_RETURN_NOTIFY)
arch/x86/kernel/signal.c +6 -0
@@ -687,6 +687,12 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)sigset_t *set =sigmask_to_save ();compat_sigset_t *cset = (compat_sigset_t *) set; +/* +* Increment event counter and perform fixup for the pre-signal +* frame. +*/ +rseq_signal_deliver (regs); +/* Set up the stack frame */ if (is_ia32_frame (ksig)) {if (ksig->ka.sa.sa_flags &SA_SIGINFO )
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 6/22 x86: Wire up restartable sequence system call
In reply to Mathieu DesnoyersWire up the rseq system call on x86 32/64.
This provides an ABI improving the speed of a user-space getcpu operation on x86 by removing the need to perform a function call, "lsl" instruction, or system call on the fast path, as well as improving the speed of user-space operations on per-cpu data.
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + 2 files changed, 2 insertions(+)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -391,3 +391,4 @@ 382 i386 pkey_free sys_pkey_free 383 i386 statx sys_statx 384 i386 arch_prctl sys_arch_prctl compat_sys_arch_prctl +385 i386 rseq sys_rseq
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -339,6 +339,7 @@ 330 common pkey_alloc sys_pkey_alloc 331 common pkey_free sys_pkey_free 332 common statx sys_statx +333 common rseq sys_rseq # # x32-specific system call numbers start at 512 to avoid cache impact
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 7/22 powerpc: Add support for restartable sequences
In reply to Mathieu DesnoyersFrom: Boqun Feng <[email protected]>
Call the rseq_handle_notify_resume() function on return to userspace if TIF_NOTIFY_RESUME thread flag is set.
Increment the event counter and perform fixup on the pre-signal when a signal is delivered on top of a restartable sequence critical section.
--- arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/signal.c | 3 +++ 2 files changed, 4 insertions(+)
arch/powerpc/Kconfig +1 -0
@@ -221,6 +221,7 @@ config PPC select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING select HAVE_IRQ_TIME_ACCOUNTING + select HAVE_RSEQ select IRQ_DOMAIN select IRQ_FORCED_THREADING select MODULES_USE_ELF_RELA
arch/powerpc/kernel/signal.c +3 -0
@@ -133,6 +133,8 @@ static void do_signal(struct task_struct *tsk)/* Re-enable the breakpoints for the signal stack */ thread_change_pc (tsk, tsk->thread.regs); +rseq_signal_deliver (tsk->thread.regs); +if (is32) {if (ksig.ka.sa.sa_flags &SA_SIGINFO ) ret =handle_rt_signal32 (&ksig, oldset, tsk);
@@ -161,6 +163,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags)if (thread_info_flags & _TIF_NOTIFY_RESUME) {clear_thread_flag (TIF_NOTIFY_RESUME );tracehook_notify_resume (regs); +rseq_handle_notify_resume (regs); }if (thread_info_flags & _TIF_PATCH_PENDING)
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 8/22 powerpc: Wire up restartable sequences system call
In reply to Mathieu DesnoyersFrom: Boqun Feng <[email protected]>
Wire up the rseq system call on powerpc.
This provides an ABI improving the speed of a user-space getcpu operation on powerpc by skipping the getcpu system call on the fast path, as well as improving the speed of user-space operations on per-cpu data compared to using load-reservation/store-conditional atomics.
--- arch/powerpc/include/asm/systbl.h | 1 + arch/powerpc/include/asm/unistd.h | 2 +- arch/powerpc/include/uapi/asm/unistd.h | 1 + 3 files changed, 3 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/systbl.h +1 -0
@@ -389,3 +389,4 @@ COMPAT_SYS_SPU(preadv2) COMPAT_SYS_SPU(pwritev2) SYSCALL(kexec_file_load) SYSCALL(statx) +SYSCALL(rseq)
arch/powerpc/include/asm/unistd.h +1 -1
@@ -12,7 +12,7 @@#include <uapi/asm/unistd.h> -#define NR_syscalls 384 +#define NR_syscalls 385#define __NR__exit __NR_exit
arch/powerpc/include/uapi/asm/unistd.h +1 -0
@@ -395,5 +395,6 @@#define __NR_pwritev2 381#define __NR_kexec_file_load 382#define __NR_statx 383 +#define __NR_rseq 384#endif /* _UAPI_ASM_POWERPC_UNISTD_H_ */
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 9/22 sched: Implement push_task_to_cpu
In reply to Mathieu DesnoyersImplement push_task_to_cpu(), which moves the task received as argument to the destination cpu's runqueue. It only does so if the CPU is within the CPU allowed mask of the task, else it returns -EINVAL.
It does not change the CPU allowed mask, and can therefore be used within applications which rely on owning the sched_setaffinity() state.
It does not pin the task to the destination CPU, which means that the scheduler may choose to move the task away from that CPU before the task executes. Code invoking push_task_to_cpu() must be prepared to retry in that case.
--- kernel/sched/core.c | 37 +++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 9 +++++++++ 2 files changed, 46 insertions(+)
kernel/sched/core.c +37 -0
@@ -1061,6 +1061,43 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)set_curr_task (rq, p); } +int push_task_to_cpu (struct task_struct *p,unsigned int dest_cpu) +{ +struct rq_flags rf; +struct rq *rq; +int ret =0 ; + + rq =task_rq_lock (p, &rf); +update_rq_clock (rq); + +if (!cpumask_test_cpu (dest_cpu, &p->cpus_allowed)) { + ret = -EINVAL ; + goto out; + } + +if (task_cpu (p) == dest_cpu) + goto out; + +if (task_running (rq, p) || p->state ==TASK_WAKING ) { +struct migration_arg arg = { p, dest_cpu }; +/* Need help from migration thread: drop lock and wait. */ +task_rq_unlock (rq, p, &rf); +stop_one_cpu (cpu_of (rq), migration_cpu_stop, &arg); +tlb_migrate_finish (p->mm); +return 0 ; + }else if (task_on_rq_queued (p)) { +/* +* OK, since we're going to drop the lock immediately +* afterwards anyway. +*/ + rq =move_queued_task (rq, &rf, p, dest_cpu); + } +out: +task_rq_unlock (rq, p, &rf); + +return ret; +} +/* * Change a given task's CPU affinity. Migrate the thread to a * proper CPU and schedule it away if the CPU it's executing on
kernel/sched/sched.h +9 -0
@@ -1223,6 +1223,15 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)#endif } +#ifdef CONFIG_SMP +int push_task_to_cpu (struct task_struct *p,unsigned int dest_cpu); +#else +static inline int push_task_to_cpu (struct task_struct *p,unsigned int dest_cpu) +{ +return 0 ; +} +#endif +/* * Tunables that become constants when CONFIG_SCHED_DEBUG is off: */
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v4 10/22 cpu_opv: Provide cpu_opv system call
In reply to Mathieu DesnoyersThe cpu_opv system call executes a vector of operations on behalf of user-space on a specific CPU with preemption disabled. It is inspired by readv() and writev() system calls which take a "struct iovec" array as argument.
The operations available are: comparison, memcpy, add, or, and, xor, left shift, right shift, and memory barrier. The system call receives a CPU number from user-space as argument, which is the CPU on which those operations need to be performed. All pointers in the ops must have been set up to point to the per CPU memory of the CPU on which the operations should be executed. The "comparison" operation can be used to check that the data used in the preparation step did not change between preparation of system call inputs and operation execution within the preempt-off critical section.
The reason why we require all pointer offsets to be calculated by user-space beforehand is because we need to use get_user_pages_fast() to first pin all pages touched by each operation. This takes care of faulting-in the pages. Then, preemption is disabled, and the operations are performed atomically with respect to other thread execution on that CPU, without generating any page fault.
An overall maximum of 4216 bytes in enforced on the sum of operation length within an operation vector, so user-space cannot generate a too long preempt-off critical section (cache cold critical section duration measured as 4.7µs on x86-64). Each operation is also limited a length of 4096 bytes, meaning that an operation can touch a maximum of 4 pages (memcpy: 2 pages for source, 2 pages for destination if addresses are not aligned on page boundaries).
If the thread is not running on the requested CPU, it is migrated to it.
**** Justification for rseq ****
Here are a few reasons justifying why the cpu_opv system call is needed in addition to rseq:
1) Handling single-stepping from tools
Tools like debuggers, and simulators use single-stepping to run through existing programs. If core libraries start to use restartable sequences for e.g. memory allocation, this means pre-existing programs cannot be single-stepped, simply because the underlying glibc or jemalloc has changed.
The rseq user-space does expose a __rseq_table section for the sake of debuggers, so they can skip over the rseq critical sections if they want. However, this requires upgrading tools, and still breaks single-stepping in case where glibc or jemalloc is updated, but not the tooling.
Having a performance-related library improvement break tooling is likely to cause a big push-back against wide adoption of rseq.
2) Forward-progress guarantee
Having a piece of user-space code that stops progressing due to external conditions is pretty bad. Developers are used to think of fast-path and slow-path (e.g. for locking), where the contended vs uncontended cases have different performance characteristics, but each need to provide some level of progress guarantees.
There are concerns about proposing just "rseq" without the associated slow-path (cpu_opv) that guarantees progress. It's just asking for trouble when real-life will happen: page faults, uprobes, and other unforeseen conditions that would seldom cause a rseq fast-path to never progress.
3) Handling page faults
It's pretty easy to come up with corner-case scenarios where rseq does not progress without the help from cpu_opv. For instance, a system with swap enabled which is under high memory pressure could trigger page faults at pretty much every rseq attempt. Although this scenario is extremely unlikely, rseq becomes the weak link of the chain.
4) Comparison with LL/SC
The layman versed in the load-link/store-conditional instructions in RISC architectures will notice the similarity between rseq and LL/SC critical sections. The comparison can even be pushed further: since debuggers can handle those LL/SC critical sections, they should be able to handle rseq c.s. in the same way.
First, the way gdb recognises LL/SC c.s. patterns is very fragile:
it's limited to specific common patterns, and will miss the pattern in all other cases. But fear not, having the rseq c.s. expose a __rseq_table to debuggers removes that guessing part.
The main difference between LL/SC and rseq is that debuggers had to support single-stepping through LL/SC critical sections from the get go in order to support a given architecture. For rseq, we're adding critical sections into pre-existing applications/libraries, so the user expectation is that tools don't break due to a library optimization.
5) Perform maintenance operations on per-cpu data
rseq c.s. are quite limited feature-wise: they need to end with a *single* commit instruction that updates a memory location. On the other hand, the cpu_opv system call can combine a sequence of operations that need to be executed with preemption disabled. While slower than rseq, this allows for more complex maintenance operations to be performed on per-cpu data concurrently with rseq fast-paths, in cases where it's not possible to map those sequences of ops to a rseq.
6) Use cpu_opv as generic implementation for architectures not implementing rseq assembly code
rseq critical sections require architecture-specific user-space code to be crafted in order to port an algorithm to a given architecture. In addition, it requires that the kernel architecture implementation adds hooks into signal delivery and resume to user-space.
In order to facilitate integration of rseq into user-space, cpu_opv can provide a (relatively slower) architecture-agnostic implementation of rseq. This means that user-space code can be ported to all architectures through use of cpu_opv initially, and have the fast-path use rseq whenever the asm code is implemented.
7) Allow libraries with multi-part algorithms to work on same per-cpu data without affecting the allowed cpu mask
The lttng-ust tracer presents an interesting use-case for per-cpu
buffers: the algorithm needs to update a "reserve" counter, serialize data into the buffer, and then update a "commit" counter _on the same per-cpu buffer_. Using rseq for both reserve and commit can bring significant performance benefits.
Clearly, if rseq reserve fails, the algorithm can retry on a different per-cpu buffer. However, it's not that easy for the commit. It needs to be performed on the same per-cpu buffer as the reserve.
The cpu_opv system call solves that problem by receiving the cpu number on which the operation needs to be performed as argument. It can push the task to the right CPU if needed, and perform the operations there with preemption disabled.
Changing the allowed cpu mask for the current thread is not an acceptable alternative for a tracing library, because the application being traced does not expect that mask to be changed by libraries.
8) Ensure that data structures don't need store-release/load-acquire semantic to handle fall-back
cpu_opv performs the fall-back on the requested CPU by migrating the task to that CPU. Executing the slow-path on the right CPU ensures that store-release/load-acquire semantic is not required neither on the fast-path nor slow-path.
**** rseq and cpu_opv use-cases ****
1) per-cpu spinlock
A per-cpu spinlock can be implemented as a rseq consisting of a comparison operation (== 0) on a word, and a word store (1), followed by an acquire barrier after control dependency. The unlock path can be performed with a simple store-release of 0 to the word, which does not require rseq.
The cpu_opv fallback requires a single-word comparison (== 0) and a single-word store (1).
2) per-cpu statistics counters
A per-cpu statistics counters can be implemented as a rseq consisting of a final "add" instruction on a word as commit.
The cpu_opv fallback can be implemented as a "ADD" operation.
Besides statistics tracking, these counters can be used to implement user-space RCU per-cpu grace period tracking for both single and multi-process user-space RCU.
3) per-cpu LIFO linked-list (unlimited size stack)
A per-cpu LIFO linked-list has a "push" and "pop" operation, which respectively adds an item to the list, and removes an item from the list.
The "push" operation can be implemented as a rseq consisting of a word comparison instruction against head followed by a word store (commit) to head. Its cpu_opv fallback can be implemented as a word-compare followed by word-store as well.
The "pop" operation can be implemented as a rseq consisting of loading head, comparing it against NULL, loading the next pointer at the right offset within the head item, and the next pointer as a new head, returning the old head on success.
The cpu_opv fallback for "pop" differs from its rseq algorithm:
considering that cpu_opv requires to know all pointers at system call entry so it can pin all pages, so cpu_opv cannot simply load head and then load the head->next address within the preempt-off critical section. User-space needs to pass the head and head->next addresses to the kernel, and the kernel needs to check that the head address is unchanged since it has been loaded by user-space. However, when accessing head->next in a ABA situation, it's possible that head is unchanged, but loading head->next can result in a page fault due to a concurrently freed head object. This is why the "expect_fault" operation field is introduced: if a fault is triggered by this access, "-EAGAIN" will be returned by cpu_opv rather than -EFAULT, thus indicating the the operation vector should be attempted again. The "pop" operation can thus be implemented as a word comparison of head against the head loaded by user-space, followed by a load of the head->next pointer (which may fault), and a store of that pointer as a new head.
4) per-cpu LIFO ring buffer with pointers to objects (fixed-sized stack)
This structure is useful for passing around allocated objects by passing pointers through per-cpu fixed-sized stack.
The "push" side can be implemented with a check of the current offset against the maximum buffer length, followed by a rseq consisting of a comparison of the previously loaded offset against the current offset, a word "try store" operation into the next ring buffer array index (it's OK to abort after a try-store, since it's not the commit, and its side-effect can be overwritten), then followed by a word-store to increment the current offset (commit).
The "push" cpu_opv fallback can be done with the comparison, and two consecutive word stores, all within the preempt-off section.
The "pop" side can be implemented with a check that offset is not 0 (whether the buffer is empty), a load of the "head" pointer before the offset array index, followed by a rseq consisting of a word comparison checking that the offset is unchanged since previously loaded, another check ensuring that the "head" pointer is unchanged, followed by a store decrementing the current offset.
The cpu_opv "pop" can be implemented with the same algorithm as the rseq fast-path (compare, compare, store).
5) per-cpu LIFO ring buffer with pointers to objects (fixed-sized stack) supporting "peek" from remote CPU
In order to implement work queues with work-stealing between CPUs, it is useful to ensure the offset "commit" in scenario 4) "push" have a store-release semantic, thus allowing remote CPU to load the offset with acquire semantic, and load the top pointer, in order to check if work-stealing should be performed. The task (work queue item) existence should be protected by other means, e.g. RCU.
If the peek operation notices that work-stealing should indeed be performed, a thread can use cpu_opv to move the task between per-cpu workqueues, by first invoking cpu_opv passing the remote work queue cpu number as argument to pop the task, and then again as "push" with the target work queue CPU number.
6) per-cpu LIFO ring buffer with data copy (fixed-sized stack) (with and without acquire-release)
This structure is useful for passing around data without requiring memory allocation by copying the data content into per-cpu fixed-sized stack.
The "push" operation is performed with an offset comparison against the buffer size (figuring out if the buffer is full), followed by a rseq consisting of a comparison of the offset, a try-memcpy attempting to copy the data content into the buffer (which can be aborted and overwritten), and a final store incrementing the offset.
The cpu_opv fallback needs to same operations, except that the memcpy is guaranteed to complete, given that it is performed with preemption disabled. This requires a memcpy operation supporting length up to 4kB.
The "pop" operation is similar to the "push, except that the offset is first compared to 0 to ensure the buffer is not empty. The copy source is the ring buffer, and the destination is an output buffer.
7) per-cpu FIFO ring buffer (fixed-sized queue)
This structure is useful wherever a FIFO behavior (queue) is needed. One major use-case is tracer ring buffer.
An implementation of this ring buffer has a "reserve", followed by serialization of multiple bytes into the buffer, ended by a "commit". The "reserve" can be implemented as a rseq consisting of a word comparison followed by a word store. The reserve operation moves the producer "head". The multi-byte serialization can be performed non-atomically. Finally, the "commit" update can be performed with a rseq "add" commit instruction with store-release semantic. The ring buffer consumer reads the commit value with load-acquire semantic to know whenever it is safe to read from the ring buffer.
This use-case requires that both "reserve" and "commit" operations be performed on the same per-cpu ring buffer, even if a migration happens between those operations. In the typical case, both operations will happens on the same CPU and use rseq. In the unlikely event of a migration, the cpu_opv system call will ensure the commit can be performed on the right CPU by migrating the task to that CPU.
On the consumer side, an alternative to using store-release and load-acquire on the commit counter would be to use cpu_opv to ensure the commit counter load is performed on the right CPU. This effectively allows moving a consumer thread between CPUs to execute close to the ring buffer cache lines it will read.
---
Changes since v1:
- handle CPU hotplug,
- cleanup implementation using function pointers: We can use function pointers to implement the operations rather than duplicating all the user-access code.
- refuse device pages: Performing cpu_opv operations on io map'd pages with preemption disabled could generate long preempt-off critical sections, which leads to unwanted scheduler latency. Return EFAULT if a device page is received as parameter
- restrict op vector to 4216 bytes length sum: Restrict the operation vector to length sum of:
- 4096 bytes (typical page size on most architectures, should be enough for a string, or structures)
- 15 * 8 bytes (typical operations on integers or pointers). The goal here is to keep the duration of preempt off critical section short, so we don't add significant scheduler latency.
- Add INIT_ONSTACK macro: Introduce the CPU_OP_FIELD_u32_u64_INIT_ONSTACK() macros to ensure that users correctly initialize the upper bits of CPU_OP_FIELD_u32_u64() on their stack to 0 on 32-bit architectures.
- Add CPU_MB_OP operation:
Use-cases with:
- two consecutive stores,
- a mempcy followed by a store,
require a memory barrier before the final store operation. A typical use-case is a store-release on the final store. Given that this is a slow path, just providing an explicit full barrier instruction should be sufficient.
- Add expect fault field:
The use-case of list_pop brings interesting challenges. With rseq, we can use rseq_cmpnev_storeoffp_load(), and therefore load a pointer, compare it against NULL, add an offset, and load the target "next" pointer from the object, all within a single req critical section.
Life is not so easy for cpu_opv in this use-case, mainly because we need to pin all pages we are going to touch in the preempt-off critical section beforehand. So we need to know the target object (in which we apply an offset to fetch the next pointer) when we pin pages before disabling preemption.
So the approach is to load the head pointer and compare it against NULL in user-space, before doing the cpu_opv syscall. User-space can then compute the address of the head->next field, *without loading it*.
The cpu_opv system call will first need to pin all pages associated with input data. This includes the page backing the head->next object, which may have been concurrently deallocated and unmapped. Therefore, in this case, getting -EFAULT when trying to pin those pages may
happen: it just means they have been concurrently unmapped. This is an expected situation, and should just return -EAGAIN to user-space, to user-space can distinguish between "should retry" type of situations and actual errors that should be handled with extreme prejudice to the program (e.g. abort()).
Therefore, add "expect_fault" fields along with op input address pointers, so user-space can identify whether a fault when getting a field should return EAGAIN rather than EFAULT.
- Add compiler barrier between operations: Adding a compiler barrier between store operations in a cpu_opv sequence can be useful when paired with membarrier system call.
An algorithm with a paired slow path and fast path can use sys_membarrier on the slow path to replace fast-path memory barriers by compiler barrier.
Adding an explicit compiler barrier between operations allows cpu_opv to be used as fallback for operations meant to match the membarrier system call.
Changes since v2:
- Fix memory leak by introducing struct cpu_opv_pinned_pages. Suggested by Boqun Feng.
- Cast argument 1 passed to access_ok from integer to void __user *, fixing sparse warning.
Changes since v3:
- Fix !SMP by adding push_task_to_cpu() empty static inline.
- Add missing sys_cpu_opv() asmlinkage declaration to include/linux/syscalls.h.
Changes since v4:
- Cleanup based on Thomas Gleixner's feedback.
- Fault-in pages which are not faulted in yet (e.g. zero pages).
- Handle retry in case where the scheduler migrates the thread away from the target CPU after migration within the syscall rather than returning EAGAIN to user-space.
- Move push_task_to_cpu() to its own patch.
---
Man page associated:
CPU_OPV(2) Linux Programmer's Manual CPU_OPV(2)
NAME
cpu_opv - CPU preempt-off operation vector system call
SYNOPSIS
#include <linux/cpu_opv.h>
int cpu_opv(struct cpu_op * cpu_opv, int cpuopcnt, int cpu, int flags);DESCRIPTION
The cpu_opv system call executes a vector of operations on behalf of user-space on a specific CPU with preemption disabled.
The operations available are: comparison, memcpy, add, or, and, xor, left shift, right shift, and memory barrier. The system call receives a CPU number from user-space as argument, which is the CPU on which those operations need to be performed. All pointers in the ops must have been set up to point to the per CPU memory of the CPU on which the operations should be executed. The "com‐ parison" operation can be used to check that the data used in the preparation step did not change between preparation of system call inputs and operation execution within the preempt-off criti‐ cal section.
An overall maximum of 4216 bytes in enforced on the sum of opera‐ tion length within an operation vector, so user-space cannot gen‐ erate a too long preempt-off critical section. Each operation is also limited a length of 4096 bytes. A maximum limit of 16 opera‐ tions per cpu_opv syscall invocation is enforced.
If the thread is not running on the requested CPU, it is migrated to it.
The layout of struct cpu_opv is as follows:
Fields
op Operation of type enum cpu_op_type to perform. This opera‐
tion type selects the associated "u" union field.
len
Length (in bytes) of data to consider for this operation.
u.compare_op
For a CPU_COMPARE_EQ_OP , and CPU_COMPARE_NE_OP , contains the a and b pointers to compare. The expect_fault_a and expect_fault_b fields indicate whether a page fault should be expected for each of those pointers. If expect_fault_a , or expect_fault_b is set, EAGAIN is returned on fault, else EFAULT is returned. The len field is allowed to take values from 0 to 4096 for comparison operations.
u.memcpy_op
For a CPU_MEMCPY_OP , contains the dst and src pointers, expressing a copy of src into dst. The expect_fault_dst and expect_fault_src fields indicate whether a page fault
should be expected for each of those pointers. If
expect_fault_dst , or expect_fault_src is set, EAGAIN is returned on fault, else EFAULT is returned. The len field is allowed to take values from 0 to 4096 for memcpy opera‐ tions.
u.arithmetic_op
For a CPU_ADD_OP , contains the p , count , and
expect_fault_p fields, which are respectively a pointer to the memory location to increment, the 64-bit signed inte‐ ger value to add, and whether a page fault should be
expected for p . If expect_fault_p is set, EAGAIN is
returned on fault, else EFAULT is returned. The len field is allowed to take values of 1, 2, 4, 8 bytes for arith‐ metic operations.
u.bitwise_op
For a CPU_OR_OP , CPU_AND_OP , and CPU_XOR_OP , contains the p , mask , and expect_fault_p fields, which are respectively a pointer to the memory location to target, the mask to apply, and whether a page fault should be expected for p . If expect_fault_p is set, EAGAIN is returned on fault, else EFAULT is returned. The len field is allowed to take values of 1, 2, 4, 8 bytes for bitwise operations.
u.shift_op
For a CPU_LSHIFT_OP , and CPU_RSHIFT_OP , contains the p , bits , and expect_fault_p fields, which are respectively a pointer to the memory location to target, the number of bits to shift either left of right, and whether a page fault should be expected for p . If expect_fault_p is set, EAGAIN is returned on fault, else EFAULT is returned. The len field is allowed to take values of 1, 2, 4, 8 bytes for shift operations. The bits field is allowed to take values between 0 and 63.
The enum cpu_op_types contains the following operations:
· CPU_COMPARE_EQ_OP: Compare whether two memory locations are
equal,
· CPU_COMPARE_NE_OP: Compare whether two memory locations differ,
· CPU_MEMCPY_OP: Copy a source memory location into a destina‐
tion,
· CPU_ADD_OP: Increment a target memory location of a given
count,
· CPU_OR_OP: Apply a "or" mask to a memory location,
· CPU_AND_OP: Apply a "and" mask to a memory location,
· CPU_XOR_OP: Apply a "xor" mask to a memory location,
· CPU_LSHIFT_OP: Shift a memory location left of a given number
of bits,
· CPU_RSHIFT_OP: Shift a memory location right of a given number
of bits.
· CPU_MB_OP: Issue a memory barrier.
All of the operations above provide single-copy atomicity guar‐ antees for word-sized, word-aligned target pointers, for both loads and stores.
The cpuopcnt argument is the number of elements in the cpu_opv array. It can take values from 0 to 16.
The cpu argument is the CPU number on which the operation sequence needs to be executed.
The flags argument is expected to be 0.
RETURN VALUE
A return value of 0 indicates success. On error, -1 is returned, and errno is set appropriately. If a comparison operation fails, execution of the operation vector is stopped, and the return value is the index after the comparison operation (values between 1 and 16).
ERRORS
EAGAIN cpu_opv() system call should be attempted again.
EINVAL Either flags contains an invalid value, or cpu contains an
invalid value or a value not allowed by the current thread's allowed cpu mask, or cpuopcnt contains an invalid value, or the cpu_opv operation vector contains an invalid op value, or the cpu_opv operation vector contains an invalid len value, or the cpu_opv operation vector sum of len values is too large.
ENOSYS The cpu_opv() system call is not implemented by this ker‐
nel.
EFAULT cpu_opv is an invalid address, or a pointer contained
within an operation is invalid (and a fault is not expected for that pointer).
VERSIONS
The cpu_opv() system call was added in Linux 4.X (TODO).
CONFORMING TO
cpu_opv() is Linux-specific.
SEE ALSO
membarrier(2), rseq(2)
Linux 2017-11-10 CPU_OPV(2)
--- MAINTAINERS | 7 + include/linux/syscalls.h | 3 + include/uapi/linux/cpu_opv.h | 114 +++++ init/Kconfig | 14 + kernel/Makefile | 1 + kernel/cpu_opv.c | 1060 ++++++++++++++++++++++++++++++++++++++++++ kernel/sys_ni.c | 1 + 7 files changed, 1200 insertions(+) create mode 100644 include/uapi/linux/cpu_opv.h create mode 100644 kernel/cpu_opv.c
MAINTAINERS +7 -0
@@ -3728,6 +3728,13 @@ B: https://bugzilla.kernel.org F: drivers/cpuidle/* F: include/linux/cpuidle.h +CPU NON-PREEMPTIBLE OPERATION VECTOR SUPPORT +M: Mathieu Desnoyers <[email protected]> +L: [email protected] +S: Supported +F: kernel/cpu_opv.c +F: include/uapi/linux/cpu_opv.h + CRAMFS FILESYSTEM M: Nicolas Pitre <[email protected]> S: Maintained
include/linux/syscalls.h +3 -0
@@ -67,6 +67,7 @@ struct perf_event_attr;struct file_handle ;struct sigaltstack ;struct rseq ; +struct cpu_op ;union bpf_attr ;#include <linux/types.h>
@@ -943,5 +944,7 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags, unsigned mask, struct statx __user *buffer);asmlinkage long sys_rseq (struct rseq __user *rseq,uint32_t rseq_len,int flags,uint32_t sig); +asmlinkage long sys_cpu_opv (struct cpu_op __user *ucpuopv,int cpuopcnt, +int cpu,int flags);#endif
include/uapi/linux/cpu_opv.h new file +114 -0
114 lines not on this page · raw message
init/Kconfig +14 -0
@@ -1404,6 +1404,7 @@ config RSEQ bool "Enable rseq() system call" if EXPERT default y depends on HAVE_RSEQ + select CPU_OPV select MEMBARRIER help Enable the restartable sequences system call. It provides a
@@ -1414,6 +1415,19 @@ config RSEQ If unsure, say Y. +config CPU_OPV + bool "Enable cpu_opv() system call" if EXPERT + default y + help + Enable the CPU preempt-off operation vector system call. + It allows user-space to perform a sequence of operations on + per-cpu data with preemption disabled. Useful as + single-stepping fall-back for restartable sequences, and for + performing more complex operations on per-cpu data that would + not be otherwise possible to do with restartable sequences. + + If unsure, say Y. + config EMBEDDED bool "Embedded system" option allnoconfig_y
kernel/Makefile +1 -0
@@ -113,6 +113,7 @@ obj-$(CONFIG_TORTURE_TEST) += torture.o obj-$(CONFIG_HAS_IOMEM) += memremap.o obj-$(CONFIG_RSEQ) += rseq.o +obj-$(CONFIG_CPU_OPV) += cpu_opv.o $(obj)/configs.o: $(obj)/config_data.h
kernel/cpu_opv.c new file +1060 -0
1,060 lines not on this page · raw message
kernel/sys_ni.c +1 -0
@@ -262,3 +262,4 @@ cond_syscall(sys_pkey_free);/* restartable sequence */ cond_syscall (sys_rseq); +cond_syscall (sys_cpu_opv);
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 11/22 x86: Wire up cpu_opv system call
In reply to Mathieu Desnoyers--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + 2 files changed, 2 insertions(+)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -392,3 +392,4 @@ 383 i386 statx sys_statx 384 i386 arch_prctl sys_arch_prctl compat_sys_arch_prctl 385 i386 rseq sys_rseq +386 i386 cpu_opv sys_cpu_opv
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -340,6 +340,7 @@ 331 common pkey_free sys_pkey_free 332 common statx sys_statx 333 common rseq sys_rseq +334 common cpu_opv sys_cpu_opv # # x32-specific system call numbers start at 512 to avoid cache impact
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 12/22 powerpc: Wire up cpu_opv system call
In reply to Mathieu Desnoyers--- arch/powerpc/include/asm/systbl.h | 1 + arch/powerpc/include/asm/unistd.h | 2 +- arch/powerpc/include/uapi/asm/unistd.h | 1 + 3 files changed, 3 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/systbl.h +1 -0
@@ -390,3 +390,4 @@ COMPAT_SYS_SPU(pwritev2) SYSCALL(kexec_file_load) SYSCALL(statx) SYSCALL(rseq) +SYSCALL(cpu_opv)
arch/powerpc/include/asm/unistd.h +1 -1
@@ -12,7 +12,7 @@#include <uapi/asm/unistd.h> -#define NR_syscalls 385 +#define NR_syscalls 386#define __NR__exit __NR_exit
arch/powerpc/include/uapi/asm/unistd.h +1 -0
@@ -396,5 +396,6 @@#define __NR_kexec_file_load 382#define __NR_statx 383#define __NR_rseq 384 +#define __NR_cpu_opv 385#endif /* _UAPI_ASM_POWERPC_UNISTD_H_ */
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 13/22 arm: Wire up cpu_opv system call
In reply to Mathieu Desnoyers--- arch/arm/tools/syscall.tbl | 1 + 1 file changed, 1 insertion(+)
arch/arm/tools/syscall.tbl +1 -0
@@ -413,3 +413,4 @@ 396 common pkey_free sys_pkey_free 397 common statx sys_statx 398 common rseq sys_rseq +399 common cpu_opv sys_cpu_opv
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v3 14/22 cpu_opv: selftests: Implement selftests
In reply to Mathieu DesnoyersImplement cpu_opv selftests. It needs to express dependencies on header files and .so, which require to override the selftests lib.mk targets. Introduce a new OVERRIDE_TARGETS define for this.
---
Changes since v1:
- Expose similar library API as rseq: Expose library API closely matching the rseq APIs, following removal of the event counter from the rseq kernel API.
- Update makefile to fix make run_tests dependency on "all".
- Introduce a OVERRIDE_TARGETS.
Changes since v2:
- Test page faults.
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpu-opv/.gitignore | 1 + tools/testing/selftests/cpu-opv/Makefile | 17 + .../testing/selftests/cpu-opv/basic_cpu_opv_test.c | 1189 ++++++++++++++++++++ tools/testing/selftests/cpu-opv/cpu-op.c | 348 ++++++ tools/testing/selftests/cpu-opv/cpu-op.h | 68 ++ tools/testing/selftests/lib.mk | 4 + 8 files changed, 1629 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/.gitignore create mode 100644 tools/testing/selftests/cpu-opv/Makefile create mode 100644 tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.h
MAINTAINERS +1 -0
@@ -3734,6 +3734,7 @@ L: [email protected] S: Supported F: kernel/cpu_opv.c F: include/uapi/linux/cpu_opv.h +F: tools/testing/selftests/cpu-opv/ CRAMFS FILESYSTEM M: Nicolas Pitre <[email protected]>
tools/testing/selftests/Makefile +1 -0
@@ -5,6 +5,7 @@ TARGETS += breakpoints TARGETS += capabilities TARGETS += cpufreq TARGETS += cpu-hotplug +TARGETS += cpu-opv TARGETS += efivarfs TARGETS += exec TARGETS += firmware
tools/testing/selftests/cpu-opv/.gitignore new file +1 -0
@@ -0,0 +1 @@ +basic_cpu_opv_test
tools/testing/selftests/cpu-opv/Makefile new file +17 -0
@@ -0,0 +1,17 @@ +CFLAGS += -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_cpu_opv_test + +TEST_GEN_PROGS_EXTENDED = libcpu-op.so + +include ../lib.mk + +$(OUTPUT)/libcpu-op.so: cpu-op.c cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) cpu-op.h + $(CC) $(CFLAGS) $< -lcpu-op -o $@
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +1189 -0
1,189 lines not on this page · raw message
tools/testing/selftests/cpu-opv/cpu-op.c new file +348 -0
348 lines not on this page · raw message
tools/testing/selftests/cpu-opv/cpu-op.h new file +68 -0
68 lines not on this page · raw message
tools/testing/selftests/lib.mk +4 -0
@@ -105,6 +105,9 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) endif +# Selftest makefiles can override those targets by setting +# OVERRIDE_TARGETS = 1. +ifeq ($(OVERRIDE_TARGETS),) $(OUTPUT)/%:%.c $(LINK.c) $^ $(LDLIBS) -o $@
@@ -113,5 +116,6 @@ $(OUTPUT)/%.o:%.S $(OUTPUT)/%:%.S $(LINK.S) $^ $(LDLIBS) -o $@ +endif .PHONY: run_tests all clean install emit_tests
Signature · 1 line
2.11.0
Shuah Khan
maintainerRe: [RFC PATCH for 4.15 v3 14/22] cpu_opv: selftests: Implement selftests
Quoted 40 lines · Mathieu Desnoyers
Implement
cpu_opvselftests. It needs to express dependencies on header files and .so, which require to override the selftests lib.mk targets. Introduce a newOVERRIDE_TARGETSdefine for this.CC Russell King <linux-lFZ/[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <andi-Vw/[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <torvalds-de/[email protected]>CC Andrew Morton <akpm-de/[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>---
Changes since v1:- Expose similar library API as rseq: Expose library API closely matching the rseq APIs, following removal of the event counter from the rseq kernel API.
- Update makefile to fix makerun_testsdependency on "all".
- Introduce aOVERRIDE_TARGETS.Changes since v2:
- Test page faults.
Quoted 9 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpu-opv/.gitignore | 1 + tools/testing/selftests/cpu-opv/Makefile | 17 + .../testing/selftests/cpu-opv/basic_cpu_opv_test.c | 1189 ++++++++++++++++++++ tools/testing/selftests/cpu-opv/cpu-op.c | 348 ++++++ tools/testing/selftests/cpu-opv/cpu-op.h | 68 ++ tools/testing/selftests/lib.mk | 4 +
Please make the change to add OVERRIDE_TARGETS lib.mk a separate patch. This is most likely going to be the first patch in this series.
Quoted 1,246 lines
8 files changed, 1629 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/.gitignore create mode 100644 tools/testing/selftests/cpu-opv/Makefile create mode 100644 tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.hMAINTAINERS +1
-0@@ -3734,6 +3734,7 @@ L: [email protected] S: Supported F: kernel/cpu_opv.c F: include/uapi/linux/cpu_opv.h +F: tools/testing/selftests/cpu-opv/ CRAMFS FILESYSTEM M: Nicolas Pitre <[email protected]>tools/testing/selftests/Makefile +1
-0@@ -5,6 +5,7 @@ TARGETS += breakpoints TARGETS += capabilities TARGETS += cpufreq TARGETS += cpu-hotplug +TARGETS += cpu-opv TARGETS += efivarfs TARGETS += exec TARGETS += firmwaretools/testing/selftests/cpu-opv/.gitignore new file +1
-0@@ -0,0 +1 @@ +basic_cpu_opv_testtools/testing/selftests/cpu-opv/Makefile new file +17
-0@@ -0,0 +1,17 @@ +CFLAGS += -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_cpu_opv_test + +TEST_GEN_PROGS_EXTENDED = libcpu-op.so + +include ../lib.mk + +$(OUTPUT)/libcpu-op.so: cpu-op.c cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) cpu-op.h + $(CC) $(CFLAGS) $< -lcpu-op -o $@tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +1177
-0@@ -0,0 +1,1189 @@ +/* +* Basic test coverage for cpu_opv system call. +*/ + +#define _GNU_SOURCE +#include <assert.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <string.h> +#include <sys/time.h> +#include <errno.h> +#include <stdlib.h> + +#include "cpu-op.h" + +#define ARRAY_SIZE (arr) (sizeof(arr) / sizeof((arr)[0])) + +#define TESTBUFLEN 4096 +#define TESTBUFLEN_CMP 16 + +#define TESTBUFLEN_PAGE_MAX 65536 + +#define NR_PF_ARRAY 16384 +#define PF_ARRAY_LEN 4096 + +/* 64 MB arrays for page fault testing. */ +char pf_array_dst[NR_PF_ARRAY ][PF_ARRAY_LEN ]; +char pf_array_src[NR_PF_ARRAY ][PF_ARRAY_LEN ]; + +static int test_compare_eq_op (char *a,char *b,size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.a, a), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.b, b), + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_compare_eq_same (void ) +{ +int i, ret; +char buf1[TESTBUFLEN ]; +char buf2[TESTBUFLEN ]; +const char *test_name ="test_compare_eq same" ; + +printf ("Testing %s\n" , test_name); + +/* Test compare_eq */ +for (i =0 ; i <TESTBUFLEN ; i++) + buf1[i] = (char )i; +for (i =0 ; i <TESTBUFLEN ; i++) + buf2[i] = (char )i; + ret =test_compare_eq_op (buf2, buf1,TESTBUFLEN ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret >0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,0 ); +return -1 ; + } +return 0 ; +} + +static int test_compare_eq_diff (void ) +{ +int i, ret; +char buf1[TESTBUFLEN ]; +char buf2[TESTBUFLEN ]; +const char *test_name ="test_compare_eq different" ; + +printf ("Testing %s\n" , test_name); + +for (i =0 ; i <TESTBUFLEN ; i++) + buf1[i] = (char )i; +memset (buf2,0 ,TESTBUFLEN ); + ret =test_compare_eq_op (buf2, buf1,TESTBUFLEN ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret ==0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,1 ); +return -1 ; + } +return 0 ; +} + +static int test_compare_ne_op (char *a,char *b,size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_NE_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.a, a), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.b, b), + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_compare_ne_same (void ) +{ +int i, ret; +char buf1[TESTBUFLEN ]; +char buf2[TESTBUFLEN ]; +const char *test_name ="test_compare_ne same" ; + +printf ("Testing %s\n" , test_name); + +/* Test compare_ne */ +for (i =0 ; i <TESTBUFLEN ; i++) + buf1[i] = (char )i; +for (i =0 ; i <TESTBUFLEN ; i++) + buf2[i] = (char )i; + ret =test_compare_ne_op (buf2, buf1,TESTBUFLEN ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret ==0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,1 ); +return -1 ; + } +return 0 ; +} + +static int test_compare_ne_diff (void ) +{ +int i, ret; +char buf1[TESTBUFLEN ]; +char buf2[TESTBUFLEN ]; +const char *test_name ="test_compare_ne different" ; + +printf ("Testing %s\n" , test_name); + +for (i =0 ; i <TESTBUFLEN ; i++) + buf1[i] = (char )i; +memset (buf2,0 ,TESTBUFLEN ); + ret =test_compare_ne_op (buf2, buf1,TESTBUFLEN ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,0 ); +return -1 ; + } +return 0 ; +} + +static int test_2compare_eq_op (char *a,char *b,char *c,char *d, +size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.a, a), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.b, b), + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_COMPARE_EQ_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.a, c), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.b, d), + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_2compare_eq_index (void ) +{ +int i, ret; +char buf1[TESTBUFLEN_CMP ]; +char buf2[TESTBUFLEN_CMP ]; +char buf3[TESTBUFLEN_CMP ]; +char buf4[TESTBUFLEN_CMP ]; +const char *test_name ="test_2compare_eq index" ; + +printf ("Testing %s\n" , test_name); + +for (i =0 ; i <TESTBUFLEN_CMP ; i++) + buf1[i] = (char )i; +memset (buf2,0 ,TESTBUFLEN_CMP ); +memset (buf3,0 ,TESTBUFLEN_CMP ); +memset (buf4,0 ,TESTBUFLEN_CMP ); + +/* First compare failure is op[0], expect 1. */ + ret =test_2compare_eq_op (buf2, buf1, buf4, buf3,TESTBUFLEN_CMP ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=1 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,1 ); +return -1 ; + } + +/* All compares succeed. */ +for (i =0 ; i <TESTBUFLEN_CMP ; i++) + buf2[i] = (char )i; + ret =test_2compare_eq_op (buf2, buf1, buf4, buf3,TESTBUFLEN_CMP ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,0 ); +return -1 ; + } + +/* First compare failure is op[1], expect 2. */ +for (i =0 ; i <TESTBUFLEN_CMP ; i++) + buf3[i] = (char )i; + ret =test_2compare_eq_op (buf2, buf1, buf4, buf3,TESTBUFLEN_CMP ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=2 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,2 ); +return -1 ; + } + +return 0 ; +} + +static int test_2compare_ne_op (char *a,char *b,char *c,char *d, +size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_NE_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.a, a), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.b, b), + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_COMPARE_NE_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.a, c), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.compare_op.b, d), + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_2compare_ne_index (void ) +{ +int i, ret; +char buf1[TESTBUFLEN_CMP ]; +char buf2[TESTBUFLEN_CMP ]; +char buf3[TESTBUFLEN_CMP ]; +char buf4[TESTBUFLEN_CMP ]; +const char *test_name ="test_2compare_ne index" ; + +printf ("Testing %s\n" , test_name); + +memset (buf1,0 ,TESTBUFLEN_CMP ); +memset (buf2,0 ,TESTBUFLEN_CMP ); +memset (buf3,0 ,TESTBUFLEN_CMP ); +memset (buf4,0 ,TESTBUFLEN_CMP ); + +/* First compare ne failure is op[0], expect 1. */ + ret =test_2compare_ne_op (buf2, buf1, buf4, buf3,TESTBUFLEN_CMP ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=1 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,1 ); +return -1 ; + } + +/* All compare ne succeed. */ +for (i =0 ; i <TESTBUFLEN_CMP ; i++) + buf1[i] = (char )i; +for (i =0 ; i <TESTBUFLEN_CMP ; i++) + buf3[i] = (char )i; + ret =test_2compare_ne_op (buf2, buf1, buf4, buf3,TESTBUFLEN_CMP ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,0 ); +return -1 ; + } + +/* First compare failure is op[1], expect 2. */ +for (i =0 ; i <TESTBUFLEN_CMP ; i++) + buf4[i] = (char )i; + ret =test_2compare_ne_op (buf2, buf1, buf4, buf3,TESTBUFLEN_CMP ); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret !=2 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,2 ); +return -1 ; + } + +return 0 ; +} + +static int test_memcpy_op (void *dst,void *src,size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.dst, dst), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.src, src), + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_memcpy (void ) +{ +int i, ret; +char buf1[TESTBUFLEN ]; +char buf2[TESTBUFLEN ]; +const char *test_name ="test_memcpy" ; + +printf ("Testing %s\n" , test_name); + +/* Test memcpy */ +for (i =0 ; i <TESTBUFLEN ; i++) + buf1[i] = (char )i; +memset (buf2,0 ,TESTBUFLEN ); + ret =test_memcpy_op (buf2, buf1,TESTBUFLEN ); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +for (i =0 ; i <TESTBUFLEN ; i++) { +if (buf2[i] != (char )i) { +printf ("%s failed. Expecting '%d', found '%d' at offset %d\n" , + test_name, (char )i, buf2[i], i); +return -1 ; + } + } +return 0 ; +} + +static int test_memcpy_u32 (void ) +{ +int ret; +uint32_t v1, v2; +const char *test_name ="test_memcpy_u32" ; + +printf ("Testing %s\n" , test_name); + +/* Test memcpy_u32 */ + v1 =42 ; + v2 =0 ; + ret =test_memcpy_op (&v2, &v1,sizeof (v1)); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (v1 != v2) { +printf ("%s failed. Expecting '%d', found '%d'\n" , + test_name, v1, v2); +return -1 ; + } +return 0 ; +} + +static int test_memcpy_mb_memcpy_op (void *dst1,void *src1, +void *dst2,void *src2,size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.dst, dst1), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.src, src1), + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [1 ] = { + .op =CPU_MB_OP , + }, + [2 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.dst, dst2), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.src, src2), + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_memcpy_mb_memcpy (void ) +{ +int ret; +int v1, v2, v3; +const char *test_name ="test_memcpy_mb_memcpy" ; + +printf ("Testing %s\n" , test_name); + +/* Test memcpy */ + v1 =42 ; + v2 = v3 =0 ; + ret =test_memcpy_mb_memcpy_op (&v2, &v1, &v3, &v2,sizeof (int )); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (v3 != v1) { +printf ("%s failed. Expecting '%d', found '%d'\n" , + test_name, v1, v3); +return -1 ; + } +return 0 ; +} + +static int test_add_op (int *v,int64_t increment) +{ +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_op_add (v, increment,sizeof (*v), cpu); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_add (void ) +{ +int orig_v =42 , v, ret; +int increment =1 ; +const char *test_name ="test_add" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_add_op (&v, increment); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != orig_v + increment) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v); +return -1 ; + } +return 0 ; +} + +static int test_two_add_op (int *v,int64_t *increments) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_ADD_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.arithmetic_op.p, v), + .u.arithmetic_op.count = increments[0 ], + .u.arithmetic_op.expect_fault_p =0 , + }, + [1 ] = { + .op =CPU_ADD_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.arithmetic_op.p, v), + .u.arithmetic_op.count = increments[1 ], + .u.arithmetic_op.expect_fault_p =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_two_add (void ) +{ +int orig_v =42 , v, ret; +int64_t increments[2 ] = {99 ,123 }; +const char *test_name ="test_two_add" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_two_add_op (&v, increments); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != orig_v + increments[0 ] + increments[1 ]) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v); +return -1 ; + } +return 0 ; +} + +static int test_or_op (int *v,uint64_t mask) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_OR_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.bitwise_op.p, v), + .u.bitwise_op.mask = mask, + .u.bitwise_op.expect_fault_p =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_or (void ) +{ +int orig_v =0xFF00000 , v, ret; +uint32_t mask =0xFFF ; +const char *test_name ="test_or" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_or_op (&v, mask); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != (orig_v | mask)) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v | mask); +return -1 ; + } +return 0 ; +} + +static int test_and_op (int *v,uint64_t mask) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_AND_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.bitwise_op.p, v), + .u.bitwise_op.mask = mask, + .u.bitwise_op.expect_fault_p =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_and (void ) +{ +int orig_v =0xF00 , v, ret; +uint32_t mask =0xFFF ; +const char *test_name ="test_and" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_and_op (&v, mask); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != (orig_v & mask)) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v & mask); +return -1 ; + } +return 0 ; +} + +static int test_xor_op (int *v,uint64_t mask) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_XOR_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.bitwise_op.p, v), + .u.bitwise_op.mask = mask, + .u.bitwise_op.expect_fault_p =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_xor (void ) +{ +int orig_v =0xF00 , v, ret; +uint32_t mask =0xFFF ; +const char *test_name ="test_xor" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_xor_op (&v, mask); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != (orig_v ^ mask)) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v ^ mask); +return -1 ; + } +return 0 ; +} + +static int test_lshift_op (int *v,uint32_t bits) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_LSHIFT_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.shift_op.p, v), + .u.shift_op.bits = bits, + .u.shift_op.expect_fault_p =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_lshift (void ) +{ +int orig_v =0xF00 , v, ret; +uint32_t bits =5 ; +const char *test_name ="test_lshift" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_lshift_op (&v, bits); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != (orig_v << bits)) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v << bits); +return -1 ; + } +return 0 ; +} + +static int test_rshift_op (int *v,uint32_t bits) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_RSHIFT_OP , + .len =sizeof (*v), +LINUX_FIELD_u32_u64_INIT_ONSTACK ( + .u.shift_op.p, v), + .u.shift_op.bits = bits, + .u.shift_op.expect_fault_p =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_rshift (void ) +{ +int orig_v =0xF00 , v, ret; +uint32_t bits =5 ; +const char *test_name ="test_rshift" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_rshift_op (&v, bits); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return -1 ; + } +if (v != (orig_v >> bits)) { +printf ("%s unexpected value: %d. Should be %d.\n" , + test_name, v, orig_v >> bits); +return -1 ; + } +return 0 ; +} + +static int test_cmpxchg_op (void *v,void *expect,void *old,void *n, +size_t len) +{ +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_op_cmpxchg (v, expect, old, n, len, cpu); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + + +static int test_cmpxchg_success (void ) +{ +int ret; +uint64_t orig_v =1 , v, expect =1 , old =0 , n =3 ; +const char *test_name ="test_cmpxchg success" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_cmpxchg_op (&v, &expect, &old, &n,sizeof (uint64_t )); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,0 ); +return -1 ; + } +if (v != n) { +printf ("%s v is %lld, expecting %lld\n" , + test_name, (long long )v, (long long )n); +return -1 ; + } +if (old != orig_v) { +printf ("%s old is %lld, expecting %lld\n" , + test_name, (long long )old, (long long )orig_v); +return -1 ; + } +return 0 ; +} + +static int test_cmpxchg_fail (void ) +{ +int ret; +uint64_t orig_v =1 , v, expect =123 , old =0 , n =3 ; +const char *test_name ="test_cmpxchg fail" ; + +printf ("Testing %s\n" , test_name); + + v = orig_v; + ret =test_cmpxchg_op (&v, &expect, &old, &n,sizeof (uint64_t )); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +if (ret ==0 ) { +printf ("%s returned %d, expecting %d\n" , + test_name, ret,1 ); +return -1 ; + } +if (v == n) { +printf ("%s v is %lld, expecting %lld\n" , + test_name, (long long )v, (long long )orig_v); +return -1 ; + } +if (old != orig_v) { +printf ("%s old is %lld, expecting %lld\n" , + test_name, (long long )old, (long long )orig_v); +return -1 ; + } +return 0 ; +} + +static int test_memcpy_expect_fault_op (void *dst,void *src,size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.dst, dst), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.src, src), + .u.memcpy_op.expect_fault_dst =0 , +/* Return EAGAIN on fault. */ + .u.memcpy_op.expect_fault_src =1 , + }, + }; +int cpu; + + cpu =cpu_op_get_current_cpu (); +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +static int test_memcpy_fault (void ) +{ +int ret; +char buf1[TESTBUFLEN ]; +const char *test_name ="test_memcpy_fault" ; + +printf ("Testing %s\n" , test_name); + +/* Test memcpy */ + ret =test_memcpy_op (buf1,NULL ,TESTBUFLEN ); +if (!ret || (ret <0 && errno !=EFAULT )) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +/* Test memcpy expect fault */ + ret =test_memcpy_expect_fault_op (buf1,NULL ,TESTBUFLEN ); +if (!ret || (ret <0 && errno !=EAGAIN )) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +return 0 ; +} + +static int do_test_unknown_op (void ) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =-1 ,/* Unknown */ + .len =0 , + }, + }; +int cpu; + + cpu =cpu_op_get_current_cpu (); +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +static int test_unknown_op (void ) +{ +int ret; +const char *test_name ="test_unknown_op" ; + +printf ("Testing %s\n" , test_name); + + ret =do_test_unknown_op (); +if (!ret || (ret <0 && errno !=EINVAL )) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +return 0 ; +} + +static int do_test_max_ops (void ) +{ +struct cpu_op opvec[] = { + [0 ] = { .op =CPU_MB_OP , }, + [1 ] = { .op =CPU_MB_OP , }, + [2 ] = { .op =CPU_MB_OP , }, + [3 ] = { .op =CPU_MB_OP , }, + [4 ] = { .op =CPU_MB_OP , }, + [5 ] = { .op =CPU_MB_OP , }, + [6 ] = { .op =CPU_MB_OP , }, + [7 ] = { .op =CPU_MB_OP , }, + [8 ] = { .op =CPU_MB_OP , }, + [9 ] = { .op =CPU_MB_OP , }, + [10 ] = { .op =CPU_MB_OP , }, + [11 ] = { .op =CPU_MB_OP , }, + [12 ] = { .op =CPU_MB_OP , }, + [13 ] = { .op =CPU_MB_OP , }, + [14 ] = { .op =CPU_MB_OP , }, + [15 ] = { .op =CPU_MB_OP , }, + }; +int cpu; + + cpu =cpu_op_get_current_cpu (); +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +static int test_max_ops (void ) +{ +int ret; +const char *test_name ="test_max_ops" ; + +printf ("Testing %s\n" , test_name); + + ret =do_test_max_ops (); +if (ret <0 ) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +return 0 ; +} + +static int do_test_too_many_ops (void ) +{ +struct cpu_op opvec[] = { + [0 ] = { .op =CPU_MB_OP , }, + [1 ] = { .op =CPU_MB_OP , }, + [2 ] = { .op =CPU_MB_OP , }, + [3 ] = { .op =CPU_MB_OP , }, + [4 ] = { .op =CPU_MB_OP , }, + [5 ] = { .op =CPU_MB_OP , }, + [6 ] = { .op =CPU_MB_OP , }, + [7 ] = { .op =CPU_MB_OP , }, + [8 ] = { .op =CPU_MB_OP , }, + [9 ] = { .op =CPU_MB_OP , }, + [10 ] = { .op =CPU_MB_OP , }, + [11 ] = { .op =CPU_MB_OP , }, + [12 ] = { .op =CPU_MB_OP , }, + [13 ] = { .op =CPU_MB_OP , }, + [14 ] = { .op =CPU_MB_OP , }, + [15 ] = { .op =CPU_MB_OP , }, + [16 ] = { .op =CPU_MB_OP , }, + }; +int cpu; + + cpu =cpu_op_get_current_cpu (); +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +static int test_too_many_ops (void ) +{ +int ret; +const char *test_name ="test_too_many_ops" ; + +printf ("Testing %s\n" , test_name); + + ret =do_test_too_many_ops (); +if (!ret || (ret <0 && errno !=EINVAL )) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +return 0 ; +} + +/* Use 64kB len, largest page size known on Linux. */ +static int test_memcpy_single_too_large (void ) +{ +int i, ret; +char buf1[TESTBUFLEN_PAGE_MAX +1 ]; +char buf2[TESTBUFLEN_PAGE_MAX +1 ]; +const char *test_name ="test_memcpy_single_too_large" ; + +printf ("Testing %s\n" , test_name); + +/* Test memcpy */ +for (i =0 ; i <TESTBUFLEN_PAGE_MAX +1 ; i++) + buf1[i] = (char )i; +memset (buf2,0 ,TESTBUFLEN_PAGE_MAX +1 ); + ret =test_memcpy_op (buf2, buf1,TESTBUFLEN_PAGE_MAX +1 ); +if (!ret || (ret <0 && errno !=EINVAL )) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +return 0 ; +} + +static int test_memcpy_single_ok_sum_too_large_op (void *dst,void *src,size_t len) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.dst, dst), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.src, src), + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.dst, dst), + LINUX_FIELD_u32_u64_INIT_ONSTACK(.u.memcpy_op.src, src), + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; +int ret, cpu; + +do { + cpu =cpu_op_get_current_cpu (); + ret =cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); + }while (ret ==-1 && errno ==EAGAIN ); + +return ret; +} + +static int test_memcpy_single_ok_sum_too_large (void ) +{ +int i, ret; +char buf1[TESTBUFLEN ]; +char buf2[TESTBUFLEN ]; +const char *test_name ="test_memcpy_single_ok_sum_too_large" ; + +printf ("Testing %s\n" , test_name); + +/* Test memcpy */ +for (i =0 ; i <TESTBUFLEN ; i++) + buf1[i] = (char )i; +memset (buf2,0 ,TESTBUFLEN ); + ret =test_memcpy_single_ok_sum_too_large_op (buf2, buf1,TESTBUFLEN ); +if (!ret || (ret <0 && errno !=EINVAL )) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +exit (-1 ); + } +return 0 ; +} + +/* +* Iterate over large uninitialized arrays to trigger page faults. +*/ +int test_page_fault (void ) +{ +int ret =0 ; +uint64_t i; +const char *test_name ="test_page_fault" ; + +printf ("Testing %s\n" , test_name); + +for (i =0 ; i <NR_PF_ARRAY ; i++) { + ret =test_memcpy_op (pf_array_dst[i], + pf_array_src[i], +PF_ARRAY_LEN ); +if (ret) { +printf ("%s returned with %d, errno: %s\n" , + test_name, ret,strerror (errno)); +return ret; + } + } +return 0 ; +} + +int main (int argc,char **argv) +{ +int ret =0 ; + + ret |=test_compare_eq_same (); + ret |=test_compare_eq_diff (); + ret |=test_compare_ne_same (); + ret |=test_compare_ne_diff (); + ret |=test_2compare_eq_index (); + ret |=test_2compare_ne_index (); + ret |=test_memcpy (); + ret |=test_memcpy_u32 (); + ret |=test_memcpy_mb_memcpy (); + ret |=test_add (); + ret |=test_two_add (); + ret |=test_or (); + ret |=test_and (); + ret |=test_xor (); + ret |=test_lshift (); + ret |=test_rshift ();
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c line 1187
Quoted 10 lines
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +10
-0+ ret |=test_cmpxchg_success (); + ret |=test_cmpxchg_fail (); + ret |=test_memcpy_fault (); + ret |=test_unknown_op (); + ret |=test_max_ops (); + ret |=test_too_many_ops (); + ret |=test_memcpy_single_too_large (); + ret |=test_memcpy_single_ok_sum_too_large (); + ret |=test_page_fault (); +
Where do pass counts get printed. I am seeing error messages when tests fail, not seeing any pass messages. It would be nice to use ksft framework for counting pass/fail for these series of tests that get run.
Quoted 446 lines
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +2
-0+return ret; +}tools/testing/selftests/cpu-opv/cpu-op.c new file +348
-0@@ -0,0 +1,348 @@ +/* +* cpu-op.c +* +* Copyright (C) 2017 Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/[email protected]> +* +* This library is free software; you can redistribute it and/or +* modify it under the terms of the GNU Lesser General Public +* License as published by the Free Software Foundation; only +* version 2.1 of the License. +* +* This library is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* Lesser General Public License for more details. +*/ + +#define _GNU_SOURCE +#include <errno.h> +#include <sched.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <syscall.h> +#include <assert.h> +#include <signal.h> + +#include "cpu-op.h" + +#define ARRAY_SIZE (arr) (sizeof(arr) / sizeof((arr)[0])) + +int cpu_opv (struct cpu_op *cpu_opv,int cpuopcnt,int cpu,int flags) +{ +return syscall (__NR_cpu_opv, cpu_opv, cpuopcnt, cpu, flags); +} + +int cpu_op_get_current_cpu (void ) +{ +int cpu; + + cpu =sched_getcpu (); +if (cpu <0 ) { +perror ("sched_getcpu()" ); +abort (); + } +return cpu; +} + +int cpu_op_cmpxchg (void *v,void *expect,void *old,void *n, +size_t len,int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + .u.memcpy_op.dst = (unsigned long )old, + .u.memcpy_op.src = (unsigned long )v, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [1 ] = { + .op =CPU_COMPARE_EQ_OP , + .len = len, + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [2 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )n, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_add (void *v,int64_t count,size_t len,int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_ADD_OP , + .len = len, + .u.arithmetic_op.p = (unsigned long )v, + .u.arithmetic_op.count = count, + .u.arithmetic_op.expect_fault_p =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )&newv, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +static int cpu_op_cmpeqv_storep_expect_fault (intptr_t *v,intptr_t expect, +intptr_t *newp,int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )newp, + .u.memcpy_op.expect_fault_dst =0 , +/* Return EAGAIN on src fault. */ + .u.memcpy_op.expect_fault_src =1 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu) +{ +intptr_t oldv =READ_ONCE (*v); +intptr_t *newp = (intptr_t *)(oldv + voffp); +int ret; + +if (oldv == expectnot) +return 1 ; + ret =cpu_op_cmpeqv_storep_expect_fault (v, oldv, newp, cpu); +if (!ret) { + *load = oldv; +return 0 ; + } +if (ret >0 ) { + errno =EAGAIN ; +return -1 ; + } +return -1 ; +} + +int cpu_op_cmpeqv_storev_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v2, + .u.memcpy_op.src = (unsigned long )&newv2, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [2 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )&newv, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_cmpeqv_storev_mb_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v2, + .u.memcpy_op.src = (unsigned long )&newv2, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [2 ] = { + .op =CPU_MB_OP , + }, + [3 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )&newv, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_cmpeqv_cmpeqv_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t expect2,intptr_t newv, +int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v2, + .u.compare_op.b = (unsigned long )&expect2, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [2 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )&newv, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_cmpeqv_memcpy_storev (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + .u.memcpy_op.dst = (unsigned long )dst, + .u.memcpy_op.src = (unsigned long )src, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [2 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )&newv, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_cmpeqv_memcpy_mb_storev (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu) +{ +struct cpu_op opvec[] = { + [0 ] = { + .op =CPU_COMPARE_EQ_OP , + .len =sizeof (intptr_t ), + .u.compare_op.a = (unsigned long )v, + .u.compare_op.b = (unsigned long )&expect, + .u.compare_op.expect_fault_a =0 , + .u.compare_op.expect_fault_b =0 , + }, + [1 ] = { + .op =CPU_MEMCPY_OP , + .len = len, + .u.memcpy_op.dst = (unsigned long )dst, + .u.memcpy_op.src = (unsigned long )src, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + [2 ] = { + .op =CPU_MB_OP , + }, + [3 ] = { + .op =CPU_MEMCPY_OP , + .len =sizeof (intptr_t ), + .u.memcpy_op.dst = (unsigned long )v, + .u.memcpy_op.src = (unsigned long )&newv, + .u.memcpy_op.expect_fault_dst =0 , + .u.memcpy_op.expect_fault_src =0 , + }, + }; + +return cpu_opv (opvec,ARRAY_SIZE (opvec), cpu,0 ); +} + +int cpu_op_addv (intptr_t *v,int64_t count,int cpu) +{ +return cpu_op_add (v, count,sizeof (intptr_t ), cpu); +}tools/testing/selftests/cpu-opv/cpu-op.h new file +68
-0@@ -0,0 +1,68 @@ +/* +* cpu-op.h +* +* (C) Copyright 2017 - Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#ifndef CPU_OPV_H +#define CPU_OPV_H + +#include <stdlib.h> +#include <stdint.h> +#include <linux/cpu_opv.h> + +#define likely (x) __builtin_expect(!!(x), 1) +#define unlikely (x) __builtin_expect(!!(x), 0) +#define barrier () __asm__ __volatile__("" : : : "memory") + +#define ACCESS_ONCE (x) (*(__volatile__ __typeof__(x) *)&(x)) +#define WRITE_ONCE (x, v) __extension__ ({ ACCESS_ONCE(x) = (v); }) +#define READ_ONCE (x) ACCESS_ONCE(x) + +int cpu_opv (struct cpu_op *cpuopv,int cpuopcnt,int cpu,int flags); +int cpu_op_get_current_cpu (void ); + +int cpu_op_cmpxchg (void *v,void *expect,void *old,void *_new, +size_t len,int cpu); +int cpu_op_add (void *v,int64_t count,size_t len,int cpu); + +int cpu_op_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu); +int cpu_op_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu); +int cpu_op_cmpeqv_storev_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu); +int cpu_op_cmpeqv_storev_mb_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu); +int cpu_op_cmpeqv_cmpeqv_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t expect2,intptr_t newv, +int cpu); +int cpu_op_cmpeqv_memcpy_storev (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu); +int cpu_op_cmpeqv_memcpy_mb_storev (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu); +int cpu_op_addv (intptr_t *v,int64_t count,int cpu); + +#endif /* CPU_OPV_H_ */ tools/testing/selftests/lib.mk +3
-0@@ -105,6 +105,9 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) endif +# Selftest makefiles can override those targets by setting +# OVERRIDE_TARGETS = 1. +ifeq ($(OVERRIDE_TARGETS),) $(OUTPUT)/%:%.c $(LINK.c) $^ $(LDLIBS) -o $@
tools/testing/selftests/lib.mk $(OUTPUT)/%.o:%.S · line 121
Quoted 6 lines
tools/testing/selftests/lib.mk +1
-0@@ -113,5 +116,6 @@ $(OUTPUT)/%.o:%.S $(OUTPUT)/%:%.S $(LINK.S) $^ $(LDLIBS) -o $@ +endif .PHONY: run_tests all clean install emit_tests
As I said before, please do this change in a separate patch.
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 14/22] cpu_opv: selftests: Implement selftests
----- On Nov 21, 2017, at 10:17 AM, shuah [email protected] wrote:
[...]
Quoted 26 lines
Quoted 26 lines
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +26
-0+int main (int argc,char **argv) +{ +int ret =0 ; + + ret |=test_compare_eq_same (); + ret |=test_compare_eq_diff (); + ret |=test_compare_ne_same (); + ret |=test_compare_ne_diff (); + ret |=test_2compare_eq_index (); + ret |=test_2compare_ne_index (); + ret |=test_memcpy (); + ret |=test_memcpy_u32 (); + ret |=test_memcpy_mb_memcpy (); + ret |=test_add (); + ret |=test_two_add (); + ret |=test_or (); + ret |=test_and (); + ret |=test_xor (); + ret |=test_lshift (); + ret |=test_rshift (); + ret |=test_cmpxchg_success (); + ret |=test_cmpxchg_fail (); + ret |=test_memcpy_fault (); + ret |=test_unknown_op (); + ret |=test_max_ops (); + ret |=test_too_many_ops ();
Quoted 8 lines
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c line 1187
Quoted 4 lines
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +4
-0+ ret |=test_memcpy_single_too_large (); + ret |=test_memcpy_single_ok_sum_too_large (); + ret |=test_page_fault (); +Where do pass counts get printed. I am seeing error messages when tests fail, not seeing any pass messages. It would be nice to use ksft framework for counting pass/fail for these series of tests that get run.
done. New output:
TAP version 13
(standard_in) 1: syntax error
selftests: basic_cpu_opv_test
========================================
TAP version 13
ok 1 test_compare_eq same test
ok 2 test_compare_eq different test
ok 3 test_compare_ne same test
ok 4 test_compare_ne different test
ok 5 test_2compare_eq index test
ok 6 test_2compare_ne index test
ok 7 test_memcpy test
ok 8 test_memcpy_u32 test
ok 9 test_memcpy_mb_memcpy test
ok 10 test_add test
ok 11 test_two_add test
ok 12 test_or test
ok 13 test_and test
ok 14 test_xor test
ok 15 test_lshift test
ok 16 test_rshift test
ok 17 test_cmpxchg success test
ok 18 test_cmpxchg fail test
ok 19 test_memcpy_fault test
ok 20 test_unknown_op test
ok 21 test_max_ops test
ok 22 test_too_many_ops test
ok 23 test_memcpy_single_too_large test
ok 24 test_memcpy_single_ok_sum_too_large test
ok 25 test_page_fault test
Pass 25 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..25
ok 1.. selftests: basic_cpu_opv_test [PASS]
(note the "(standard_in) 1: syntax error" for which I provided a fix in a separate thread still appears with my make version)
[...]
Quoted 16 lines
Quoted 16 lines
tools/testing/selftests/lib.mk +3
-0@@ -105,6 +105,9 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) endif +# Selftest makefiles can override those targets by setting +# OVERRIDE_TARGETS = 1. +ifeq ($(OVERRIDE_TARGETS),) $(OUTPUT)/%:%.c $(LINK.c) $^ $(LDLIBS) -o $@@@ -113,5 +116,6 @@ $(OUTPUT)/%.o:%.S
Quoted 8 lines
tools/testing/selftests/lib.mk $(OUTPUT)/%.o:%.S · line 121
Quoted 6 lines
tools/testing/selftests/lib.mk +1
-0$(OUTPUT)/%:%.S $(LINK.S) $^ $(LDLIBS) -o $@ +endif .PHONY: run_tests all clean install emit_testsAs I said before, please do this change in a separate patch.
Sorry, it appears that I missed this comment last time. Will move this change to a separate patch.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRFC PATCH v3 15/22 rseq: selftests: Provide self-tests
In reply to Mathieu DesnoyersImplements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works.
"basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness.
"param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.
A run_param_test.sh script runs many variants of the parametrizable tests.
As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the cpu_opv system call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the __rseq_table section. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.
The rseq library expose APIs that present the fast-path operations. The new from userspace is, e.g. for a counter increment:
cpu = rseq_cpu_start();
ret = rseq_addv(&data->c[cpu].count, 1, cpu);
if (likely(!ret))
return 0; /* Success. */
do {
cpu = rseq_current_cpu();
ret = cpu_op_addv(&data->c[cpu].count, 1, cpu);
if (likely(!ret))
return 0; /* Success. */
} while (ret > 0 || errno == EAGAIN);
perror("cpu_op_addv");
return -1; /* Unexpected error. */PowerPC tests have been implemented by Boqun Feng.
---
Changes since v1:
- Provide abort-ip signature: The abort-ip signature is located just before the abort-ip target. It is currently hardcoded, but a user-space application could use the __rseq_table to iterate on all abort-ip targets and use a random value as signature if needed in the future.
- Add rseq_prepare_unload(): Libraries and JIT code using rseq critical sections need to issue rseq_prepare_unload() on each thread at least once before reclaim of struct rseq_cs.
- Use initial-exec TLS model, non-weak symbol: The initial-exec model is signal-safe, whereas the global-dynamic model is not. Remove the "weak" symbol attribute from the __rseq_abi in rseq.c. The rseq.so library will have ownership of that symbol, and there is not reason for an application or user library to try to define that symbol. The expected use is to link against libreq.so, which owns and provide that symbol.
- Set cpu_id to -2 on register error
- Add rseq_len syscall parameter, rseq_cs version
- Ensure disassember-friendly signature: x86 32/64 disassembler have a hard time decoding the instruction stream after a bad instruction. Use a nopl instruction to encode the signature. Suggested by Andy Lutomirski.
- Exercise parametrized tests variants in a shell scripts.
- Restartable sequences selftests: Remove use of event counter.
- Use cpu_id_start field: With the cpu_id_start field, the C preparation phase of the fast-path does not need to compare cpu_id < 0 anymore.
- Signal-safe registration and refcounting: Allow libraries using librseq.so to register it from signal handlers.
- Use OVERRIDE_TARGETS in makefile.
- Use "m" constraints for rseq_cs field.
Changes since v2:
- Update based on Thomas Gleixner's comments.
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 + tools/testing/selftests/rseq/Makefile | 23 + .../testing/selftests/rseq/basic_percpu_ops_test.c | 333 +++++ tools/testing/selftests/rseq/basic_test.c | 55 + tools/testing/selftests/rseq/param_test.c | 1285 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 535 ++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 567 +++++++++ tools/testing/selftests/rseq/rseq-x86.h | 898 ++++++++++++++ tools/testing/selftests/rseq/rseq.c | 116 ++ tools/testing/selftests/rseq/rseq.h | 154 +++ tools/testing/selftests/rseq/run_param_test.sh | 124 ++ 13 files changed, 4096 insertions(+) create mode 100644 tools/testing/selftests/rseq/.gitignore create mode 100644 tools/testing/selftests/rseq/Makefile create mode 100644 tools/testing/selftests/rseq/basic_percpu_ops_test.c create mode 100644 tools/testing/selftests/rseq/basic_test.c create mode 100644 tools/testing/selftests/rseq/param_test.c create mode 100644 tools/testing/selftests/rseq/rseq-arm.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc.h create mode 100644 tools/testing/selftests/rseq/rseq-x86.h create mode 100644 tools/testing/selftests/rseq/rseq.c create mode 100644 tools/testing/selftests/rseq/rseq.h create mode 100755 tools/testing/selftests/rseq/run_param_test.sh
MAINTAINERS +1 -0
@@ -11634,6 +11634,7 @@ S: Supported F: kernel/rseq.c F: include/uapi/linux/rseq.h F: include/trace/events/rseq.h +F: tools/testing/selftests/rseq/ RFKILL M: Johannes Berg <[email protected]>
tools/testing/selftests/Makefile +1 -0
@@ -26,6 +26,7 @@ TARGETS += nsfs TARGETS += powerpc TARGETS += pstore TARGETS += ptrace +TARGETS += rseq TARGETS += seccomp TARGETS += sigaltstack TARGETS += size
tools/testing/selftests/rseq/.gitignore new file +4 -0
@@ -0,0 +1,4 @@ +basic_percpu_ops_test +basic_test +basic_rseq_op_test +param_test
tools/testing/selftests/rseq/Makefile new file +23 -0
@@ -0,0 +1,23 @@ +CFLAGS += -O2 -Wall -g -I./ -I../cpu-opv/ -I../../../../usr/include/ -L./ -Wl,-rpath=./ +LDLIBS += -lpthread + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_test basic_percpu_ops_test param_test + +TEST_GEN_PROGS_EXTENDED = librseq.so libcpu-op.so + +TEST_PROGS = run_param_test.sh + +include ../lib.mk + +$(OUTPUT)/librseq.so: rseq.c rseq.h rseq-*.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/libcpu-op.so: ../cpu-opv/cpu-op.c ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) rseq.h rseq-*.h ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) $< $(LDLIBS) -lrseq -lcpu-op -o $@
tools/testing/selftests/rseq/basic_percpu_ops_test.c new file +333 -0
333 lines not on this page · raw message
tools/testing/selftests/rseq/basic_test.c new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/rseq/param_test.c new file +1285 -0
1,285 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-arm.h new file +535 -0
535 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-ppc.h new file +567 -0
567 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-x86.h new file +898 -0
184 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.c new file +116 -0
116 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.h new file +154 -0
154 lines not on this page · raw message
tools/testing/selftests/rseq/run_param_test.sh new file +124 -0
124 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.11.0
Part of the diff is not shown here. View the raw message.
Shuah Khan
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 99 lines · Mathieu Desnoyers
Implements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "
basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works."
basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness."
param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.A
run_param_test.shscript runs many variants of the parametrizable tests.As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the
cpu_opvsystem call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the__rseq_tablesection. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.The rseq library expose APIs that present the fast-path operations. The new from userspace is, e.g. for a counter increment:
cpu = rseq_cpu_start(); ret = rseq_addv(&data->c[cpu].count, 1, cpu); if (likely(!ret)) return 0; /* Success. */ do { cpu = rseq_current_cpu(); ret = cpu_op_addv(&data->c[cpu].count, 1, cpu); if (likely(!ret)) return 0; /* Success. */ } while (ret > 0 || errno == EAGAIN); perror("cpu_op_addv"); return -1; /* Unexpected error. */PowerPC tests have been implemented by Boqun Feng.
CC Russell King <linux-lFZ/[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <andi-Vw/[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <torvalds-de/[email protected]>CC Andrew Morton <akpm-de/[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>---
Changes since v1:
- Provide abort-ip signature: The abort-ip signature is located just before the abort-ip target. It is currently hardcoded, but a user-space application could use the__rseq_tableto iterate on all abort-ip targets and use a random value as signature if needed in the future.
- Addrseq_prepare_unload(): Libraries and JIT code using rseq critical sections need to issuerseq_prepare_unload()on each thread at least once before reclaim of structrseq_cs.
- Use initial-exec TLS model, non-weak symbol: The initial-exec model is signal-safe, whereas the global-dynamic model is not. Remove the "weak" symbol attribute from the__rseq_abiinrseq.c. The rseq.so library will have ownership of that symbol, and there is not reason for an application or user library to try to define that symbol. The expected use is to link against libreq.so, which owns and provide that symbol.
- Setcpu_idto -2 on register error
- Addrseq_lensyscall parameter,rseq_csversion
- Ensure disassember-friendly signature: x86 32/64 disassembler have a hard time decoding the instruction stream after a bad instruction. Use a nopl instruction to encode the signature. Suggested by Andy Lutomirski.
- Exercise parametrized tests variants in a shell scripts.
- Restartable sequences selftests: Remove use of event counter.
- Usecpu_id_startfield: With thecpu_id_startfield, the C preparation phase of the fast-path does not need to comparecpu_id< 0 anymore.
- Signal-safe registration and refcounting: Allow libraries using librseq.so to register it from signal handlers.
- UseOVERRIDE_TARGETSin makefile.
- Use "m" constraints forrseq_csfield.
Quoted 6 lines
Changes since v2:
- Update based on Thomas Gleixner's comments.--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 +
Thanks for the .gitignore files. It is commonly missed change, I end up adding one to clean things up after tests get in.
Quoted 838 lines
tools/testing/selftests/rseq/Makefile | 23 + .../testing/selftests/rseq/basic_percpu_ops_test.c | 333 +++++ tools/testing/selftests/rseq/basic_test.c | 55 + tools/testing/selftests/rseq/param_test.c | 1285 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 535 ++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 567 +++++++++ tools/testing/selftests/rseq/rseq-x86.h | 898 ++++++++++++++ tools/testing/selftests/rseq/rseq.c | 116 ++ tools/testing/selftests/rseq/rseq.h | 154 +++ tools/testing/selftests/rseq/run_param_test.sh | 124 ++ 13 files changed, 4096 insertions(+) create mode 100644 tools/testing/selftests/rseq/.gitignore create mode 100644 tools/testing/selftests/rseq/Makefile create mode 100644 tools/testing/selftests/rseq/basic_percpu_ops_test.c create mode 100644 tools/testing/selftests/rseq/basic_test.c create mode 100644 tools/testing/selftests/rseq/param_test.c create mode 100644 tools/testing/selftests/rseq/rseq-arm.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc.h create mode 100644 tools/testing/selftests/rseq/rseq-x86.h create mode 100644 tools/testing/selftests/rseq/rseq.c create mode 100644 tools/testing/selftests/rseq/rseq.h create mode 100755 tools/testing/selftests/rseq/run_param_test.shMAINTAINERS +1
-0@@ -11634,6 +11634,7 @@ S: Supported F: kernel/rseq.c F: include/uapi/linux/rseq.h F: include/trace/events/rseq.h +F: tools/testing/selftests/rseq/ RFKILL M: Johannes Berg <[email protected]>tools/testing/selftests/Makefile +1
-0@@ -26,6 +26,7 @@ TARGETS += nsfs TARGETS += powerpc TARGETS += pstore TARGETS += ptrace +TARGETS += rseq TARGETS += seccomp TARGETS += sigaltstack TARGETS += sizetools/testing/selftests/rseq/.gitignore new file +4
-0@@ -0,0 +1,4 @@ +basic_percpu_ops_test +basic_test +basic_rseq_op_test +param_testtools/testing/selftests/rseq/Makefile new file +23
-0@@ -0,0 +1,23 @@ +CFLAGS += -O2 -Wall -g -I./ -I../cpu-opv/ -I../../../../usr/include/ -L./ -Wl,-rpath=./ +LDLIBS += -lpthread + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_test basic_percpu_ops_test param_test + +TEST_GEN_PROGS_EXTENDED = librseq.so libcpu-op.so + +TEST_PROGS = run_param_test.sh + +include ../lib.mk + +$(OUTPUT)/librseq.so: rseq.c rseq.h rseq-*.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/libcpu-op.so: ../cpu-opv/cpu-op.c ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) rseq.h rseq-*.h ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) $< $(LDLIBS) -lrseq -lcpu-op -o $@tools/testing/selftests/rseq/basic_percpu_ops_test.c new file +333
-0@@ -0,0 +1,333 @@ +#define _GNU_SOURCE +#include <assert.h> +#include <pthread.h> +#include <sched.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <stddef.h> + +#include "rseq.h" +#include "cpu-op.h" + +#define ARRAY_SIZE (arr) (sizeof(arr) / sizeof((arr)[0])) + +struct percpu_lock_entry { +intptr_t v; +} __attribute__((aligned (128 ))); + +struct percpu_lock { +struct percpu_lock_entry c[CPU_SETSIZE ]; +}; + +struct test_data_entry { +intptr_t count; +} __attribute__((aligned (128 ))); + +struct spinlock_test_data { +struct percpu_lock lock; +struct test_data_entry c[CPU_SETSIZE ]; +int reps; +}; + +struct percpu_list_node { +intptr_t data; +struct percpu_list_node *next; +}; + +struct percpu_list_entry { +struct percpu_list_node *head; +} __attribute__((aligned (128 ))); + +struct percpu_list { +struct percpu_list_entry c[CPU_SETSIZE ]; +}; + +/* A simple percpu spinlock. Returns the cpu lock was acquired on. */ +int rseq_percpu_lock (struct percpu_lock *lock) +{ +int cpu; + +for (;;) { +int ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_cmpeqv_storev (&lock->c[cpu].v, +0 ,1 , cpu); +if (likely (!ret)) +break ; +if (ret >0 ) +continue ;/* Retry. */ +#endif + slowpath: + __attribute__((unused)); +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_cmpeqv_storev (&lock->c[cpu].v,0 ,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +/* +* Acquire semantic when taking lock after control dependency. +* Matches rseq_smp_store_release(). +*/ +rseq_smp_acquire__after_ctrl_dep (); +return cpu; +} + +void rseq_percpu_unlock (struct percpu_lock *lock,int cpu) +{ +assert (lock->c[cpu].v ==1 ); +/* +* Release lock, with release semantic. Matches +* rseq_smp_acquire__after_ctrl_dep(). +*/ +rseq_smp_store_release (&lock->c[cpu].v,0 ); +} + +void *test_percpu_spinlock_thread (void *arg) +{ +struct spinlock_test_data *data = arg; +int i, cpu; + +if (rseq_register_current_thread ()) { +fprintf (stderr,"Error: rseq_register_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); +abort (); + } +for (i =0 ; i < data->reps; i++) { + cpu =rseq_percpu_lock (&data->lock); + data->c[cpu].count++; +rseq_percpu_unlock (&data->lock, cpu); + } +if (rseq_unregister_current_thread ()) { +fprintf (stderr,"Error: rseq_unregister_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); +abort (); + } + +return NULL ; +} + +/* +* A simple test which implements a sharded counter using a per-cpu +* lock. Obviously real applications might prefer to simply use a +* per-cpu increment; however, this is reasonable for a test and the +* lock can be extended to synchronize more complicated operations. +*/ +void test_percpu_spinlock (void ) +{ +const int num_threads =200 ; +int i; +uint64_t sum; +pthread_t test_threads[num_threads]; +struct spinlock_test_data data; + +memset (&data,0 ,sizeof (data)); + data.reps =5000 ; + +for (i =0 ; i < num_threads; i++) +pthread_create (&test_threads[i],NULL , + test_percpu_spinlock_thread, &data); + +for (i =0 ; i < num_threads; i++) +pthread_join (test_threads[i],NULL ); + + sum =0 ; +for (i =0 ; i <CPU_SETSIZE ; i++) + sum += data.c[i].count; + +assert (sum == (uint64_t )data.reps * num_threads); +} + +int percpu_list_push (struct percpu_list *list,struct percpu_list_node *node) +{ +intptr_t *targetptr, newval, expect; +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t )READ_ONCE (list->c[cpu].head); + newval = (intptr_t )node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret =rseq_cmpeqv_storev (targetptr, expect, newval, cpu); +if (likely (!ret)) +return cpu; +#endif +/* Fallback on cpu_opv system call. */ + slowpath: + __attribute__((unused)); +for (;;) { + cpu =rseq_current_cpu (); +/* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t )READ_ONCE (list->c[cpu].head); + newval = (intptr_t )node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret =cpu_op_cmpeqv_storev (targetptr, expect, newval, cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +return cpu; +} + +/* +* Unlike a traditional lock-less linked list; the availability of a +* rseq primitive allows us to implement pop without concerns over +* ABA-type races. +*/ +struct percpu_list_node *percpu_list_pop (struct percpu_list *list) +{ +struct percpu_list_node *head; +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_cmpnev_storeoffp_load ((intptr_t *)&list->c[cpu].head, + (intptr_t )NULL , + offsetof(struct percpu_list_node , next), + (intptr_t *)&head, cpu); +if (likely (!ret)) +return head; +if (ret >0 ) +return NULL ; +#endif +/* Fallback on cpu_opv system call. */ + slowpath: + __attribute__((unused)); +for (;;) { + cpu =rseq_current_cpu (); + ret =cpu_op_cmpnev_storeoffp_load ( + (intptr_t *)&list->c[cpu].head, + (intptr_t )NULL , + offsetof(struct percpu_list_node , next), + (intptr_t *)&head, cpu); +if (likely (!ret)) +break ; +if (ret >0 ) +return NULL ; +assert (ret >=0 || errno ==EAGAIN ); + } +return head; +} + +void *test_percpu_list_thread (void *arg) +{ +int i; +struct percpu_list *list = (struct percpu_list *)arg; + +if (rseq_register_current_thread ()) { +fprintf (stderr,"Error: rseq_register_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); +abort (); + } + +for (i =0 ; i <100000 ; i++) { +struct percpu_list_node *node =percpu_list_pop (list); + +sched_yield ();/* encourage shuffling */ +if (node) +percpu_list_push (list, node); + } + +if (rseq_unregister_current_thread ()) { +fprintf (stderr,"Error: rseq_unregister_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); +abort (); + } + +return NULL ; +} + +/* Simultaneous modification to a per-cpu linked list from many threads. */ +void test_percpu_list (void ) +{ +int i, j; +uint64_t sum =0 , expected_sum =0 ; +struct percpu_list list; +pthread_t test_threads[200 ]; +cpu_set_t allowed_cpus; + +memset (&list,0 ,sizeof (list)); + +/* Generate list entries for every usable cpu. */ +sched_getaffinity (0 ,sizeof (allowed_cpus), &allowed_cpus); +for (i =0 ; i <CPU_SETSIZE ; i++) { +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; +for (j =1 ; j <=100 ; j++) { +struct percpu_list_node *node; + + expected_sum += j; + + node =malloc (sizeof (*node)); +assert (node); + node->data = j; + node->next = list.c[i].head; + list.c[i].head = node; + } + } + +for (i =0 ; i <200 ; i++) +assert (pthread_create (&test_threads[i],NULL , + test_percpu_list_thread, &list) ==0 ); + +for (i =0 ; i <200 ; i++) +pthread_join (test_threads[i],NULL ); + +for (i =0 ; i <CPU_SETSIZE ; i++) { +cpu_set_t pin_mask; +struct percpu_list_node *node; + +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; + +CPU_ZERO (&pin_mask); +CPU_SET (i, &pin_mask); +sched_setaffinity (0 ,sizeof (pin_mask), &pin_mask); + +while ((node =percpu_list_pop (&list))) { + sum += node->data; +free (node); + } + } + +/* +* All entries should now be accounted for (unless some external +* actor is interfering with our allowed affinity while this +* test is running). +*/ +assert (sum == expected_sum); +} + +int main (int argc,char **argv) +{ +if (rseq_register_current_thread ()) { +fprintf (stderr,"Error: rseq_register_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); + goto error; + } +printf ("spinlock\n" ); +test_percpu_spinlock (); +printf ("percpu_list\n" ); +test_percpu_list (); +if (rseq_unregister_current_thread ()) { +fprintf (stderr,"Error: rseq_unregister_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); + goto error; + } +return 0 ; + +error: +return -1 ; +} +tools/testing/selftests/rseq/basic_test.c new file +55
-0@@ -0,0 +1,55 @@ +/* +* Basic test coverage for critical regions and rseq_current_cpu(). +*/ + +#define _GNU_SOURCE +#include <assert.h> +#include <sched.h> +#include <signal.h> +#include <stdio.h> +#include <string.h> +#include <sys/time.h> + +#include "rseq.h" + +void test_cpu_pointer (void ) +{ +cpu_set_t affinity, test_affinity; +int i; + +sched_getaffinity (0 ,sizeof (affinity), &affinity); +CPU_ZERO (&test_affinity); +for (i =0 ; i <CPU_SETSIZE ; i++) { +if (CPU_ISSET (i, &affinity)) { +CPU_SET (i, &test_affinity); +sched_setaffinity (0 ,sizeof (test_affinity), + &test_affinity); +assert (sched_getcpu () == i); +assert (rseq_current_cpu () == i); +assert (rseq_current_cpu_raw () == i); +assert (rseq_cpu_start () == i); +CPU_CLR (i, &test_affinity); + } + } +sched_setaffinity (0 ,sizeof (affinity), &affinity); +} + +int main (int argc,char **argv) +{ +if (rseq_register_current_thread ()) { +fprintf (stderr,"Error: rseq_register_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); + goto init_thread_error; + } +printf ("testing current cpu\n" ); +test_cpu_pointer (); +if (rseq_unregister_current_thread ()) { +fprintf (stderr,"Error: rseq_unregister_current_thread(...) failed(%d): %s\n" , + errno,strerror (errno)); + goto init_thread_error; + } +return 0 ; + +init_thread_error: +return -1 ; +}tools/testing/selftests/rseq/param_test.c new file +346
-0@@ -0,0 +1,1285 @@ +#define _GNU_SOURCE +#include <assert.h> +#include <pthread.h> +#include <sched.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <syscall.h> +#include <unistd.h> +#include <poll.h> +#include <sys/types.h> +#include <signal.h> +#include <errno.h> +#include <stddef.h> + +#include "cpu-op.h" + +static inline pid_t gettid (void ) +{ +return syscall (__NR_gettid); +} + +#define NR_INJECT 9 +static int loop_cnt[NR_INJECT +1 ]; + +static int opt_modulo, verbose; + +static int opt_yield, opt_signal, opt_sleep, + opt_disable_rseq, opt_threads =200 , + opt_disable_mod =0 , opt_test ='s' , opt_mb =0 ; + +static long long opt_reps =5000 ; + +static __thread __attribute__((tls_model ("initial-exec" )))unsigned int signals_delivered; + +#ifndef BENCHMARK + +static __thread __attribute__((tls_model ("initial-exec" )))unsigned int yield_mod_cnt, nr_abort; + +#define printf_verbose (fmt, ...) \ + do { \ + if (verbose) \ + printf(fmt, ## __VA_ARGS__); \ + } while (0) + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#if defined(__x86_64__) || defined(__i386__) + +#define INJECT_ASM_REG "eax" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM (n) \ + "mov %[loop_cnt_" #n "], %%" INJECT_ASM_REG "\n\t" \ + "test %%" INJECT_ASM_REG ",%%" INJECT_ASM_REG "\n\t" \ + "jz 333f\n\t" \ + "222:\n\t" \ + "dec %%" INJECT_ASM_REG "\n\t" \ + "jnz 222b\n\t" \ + "333:\n\t" + +#elif defined(__ARMEL__) + +#define INJECT_ASM_REG "r4" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM (n) \ + "ldr " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "cmp " INJECT_ASM_REG ", #0\n\t" \ + "beq 333f\n\t" \ + "222:\n\t" \ + "subs " INJECT_ASM_REG ", #1\n\t" \ + "bne 222b\n\t" \ + "333:\n\t" + +#elif __PPC__ +#define INJECT_ASM_REG "r18" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM (n) \ + "lwz %%" INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "cmpwi %%" INJECT_ASM_REG ", 0\n\t" \ + "beq 333f\n\t" \ + "222:\n\t" \ + "subic. %%" INJECT_ASM_REG ", %%" INJECT_ASM_REG ", 1\n\t" \ + "bne 222b\n\t" \ + "333:\n\t" +#else +#error unsupported target +#endif + +#define RSEQ_INJECT_FAILED \ + nr_abort++; + +#define RSEQ_INJECT_C (n) \ +{ \ + int loc_i, loc_nr_loops = loop_cnt[n]; \ + \ + for (loc_i = 0; loc_i < loc_nr_loops; loc_i++) { \ + barrier(); \ + } \ + if (loc_nr_loops == -1 && opt_modulo) { \ + if (yield_mod_cnt == opt_modulo - 1) { \ + if (opt_sleep > 0) \ + poll(NULL, 0, opt_sleep); \ + if (opt_yield) \ + sched_yield(); \ + if (opt_signal) \ + raise(SIGUSR1); \ + yield_mod_cnt = 0; \ + } else { \ + yield_mod_cnt++; \ + } \ + } \ +} + +#else + +#define printf_verbose (fmt, ...) + +#endif /* BENCHMARK */ + +#include "rseq.h" + +struct percpu_lock_entry { +intptr_t v; +} __attribute__((aligned (128 ))); + +struct percpu_lock { +struct percpu_lock_entry c[CPU_SETSIZE ]; +}; + +struct test_data_entry { +intptr_t count; +} __attribute__((aligned (128 ))); + +struct spinlock_test_data { +struct percpu_lock lock; +struct test_data_entry c[CPU_SETSIZE ]; +}; + +struct spinlock_thread_test_data { +struct spinlock_test_data *data; +long long reps; +int reg; +}; + +struct inc_test_data { +struct test_data_entry c[CPU_SETSIZE ]; +}; + +struct inc_thread_test_data { +struct inc_test_data *data; +long long reps; +int reg; +}; + +struct percpu_list_node { +intptr_t data; +struct percpu_list_node *next; +}; + +struct percpu_list_entry { +struct percpu_list_node *head; +} __attribute__((aligned (128 ))); + +struct percpu_list { +struct percpu_list_entry c[CPU_SETSIZE ]; +}; + +#define BUFFER_ITEM_PER_CPU 100 + +struct percpu_buffer_node { +intptr_t data; +}; + +struct percpu_buffer_entry { +intptr_t offset; +intptr_t buflen; +struct percpu_buffer_node **array; +} __attribute__((aligned (128 ))); + +struct percpu_buffer { +struct percpu_buffer_entry c[CPU_SETSIZE ]; +}; + +#define MEMCPY_BUFFER_ITEM_PER_CPU 100 + +struct percpu_memcpy_buffer_node { +intptr_t data1; +uint64_t data2; +}; + +struct percpu_memcpy_buffer_entry { +intptr_t offset; +intptr_t buflen; +struct percpu_memcpy_buffer_node *array; +} __attribute__((aligned (128 ))); + +struct percpu_memcpy_buffer { +struct percpu_memcpy_buffer_entry c[CPU_SETSIZE ]; +}; + +/* A simple percpu spinlock. Returns the cpu lock was acquired on. */ +static int rseq_percpu_lock (struct percpu_lock *lock) +{ +int cpu; + +for (;;) { +int ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_cmpeqv_storev (&lock->c[cpu].v, +0 ,1 , cpu); +if (likely (!ret)) +break ; +if (ret >0 ) +continue ;/* Retry. */ +#endif> + slowpath: + __attribute__((unused)); +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_cmpeqv_storev (&lock->c[cpu].v,0 ,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +/* +* Acquire semantic when taking lock after control dependency. +* Matches rseq_smp_store_release(). +*/ +rseq_smp_acquire__after_ctrl_dep (); +return cpu; +} + +static void rseq_percpu_unlock (struct percpu_lock *lock,int cpu) +{ +assert (lock->c[cpu].v ==1 ); +/* +* Release lock, with release semantic. Matches +* rseq_smp_acquire__after_ctrl_dep(). +*/ +rseq_smp_store_release (&lock->c[cpu].v,0 ); +} + +void *test_percpu_spinlock_thread (void *arg) +{ +struct spinlock_thread_test_data *thread_data = arg; +struct spinlock_test_data *data = thread_data->data; +int cpu; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps; +for (i =0 ; i < reps; i++) { + cpu =rseq_percpu_lock (&data->lock); + data->c[cpu].count++; +rseq_percpu_unlock (&data->lock, cpu); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif + } +printf_verbose ("tid %d: number of rseq abort: %d, signals delivered: %u\n" , + (int )gettid (), nr_abort, signals_delivered); +if (!opt_disable_rseq && thread_data->reg + &&rseq_unregister_current_thread ()) +abort (); +return NULL ; +} + +/* +* A simple test which implements a sharded counter using a per-cpu +* lock. Obviously real applications might prefer to simply use a +* per-cpu increment; however, this is reasonable for a test and the +* lock can be extended to synchronize more complicated operations. +*/ +void test_percpu_spinlock (void ) +{ +const int num_threads = opt_threads; +int i, ret; +uint64_t sum; +pthread_t test_threads[num_threads]; +struct spinlock_test_data data; +struct spinlock_thread_test_data thread_data[num_threads]; + +memset (&data,0 ,sizeof (data)); +for (i =0 ; i < num_threads; i++) { + thread_data[i].reps = opt_reps; +if (opt_disable_mod <=0 || (i % opt_disable_mod)) + thread_data[i].reg =1 ; +else + thread_data[i].reg =0 ; + thread_data[i].data = &data; + ret =pthread_create (&test_threads[i],NULL , + test_percpu_spinlock_thread, &thread_data[i]); +if (ret) { + errno = ret; +perror ("pthread_create" ); +abort (); + } + } + +for (i =0 ; i < num_threads; i++) { +pthread_join (test_threads[i],NULL ); +if (ret) { + errno = ret; +perror ("pthread_join" ); +abort (); + } + } + + sum =0 ; +for (i =0 ; i <CPU_SETSIZE ; i++) + sum += data.c[i].count; + +assert (sum == (uint64_t )opt_reps * num_threads); +} + +void *test_percpu_inc_thread (void *arg) +{ +struct inc_thread_test_data *thread_data = arg; +struct inc_test_data *data = thread_data->data; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps;
tools/testing/selftests/rseq/param_test.c line 356
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+for (i =0 ; i < reps; i++) { +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) + goto next; +#endif
So the test needs to compiled with this enabled? I think it would be better to make this an argument to be abel to select at test start time as opposed to making this compile time option. Remember that these tests get run in automated test rings. Making this a compile time otpion pertty much ensures that this path will not be tested.
So I would reccommend adding a paratemer.
Quoted 6 lines
tools/testing/selftests/rseq/param_test.c new file +6
-0+ slowpath: + __attribute__((unused)); +for (;;) { +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_addv (&data->c[cpu].count,1 , cpu);
tools/testing/selftests/rseq/param_test.c line 373
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } + next: + __attribute__((unused)); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif
Same comment as before. Avoid compile time options.
tools/testing/selftests/rseq/run_param_test.sh
Quoted 3,343 lines
tools/testing/selftests/rseq/param_test.c new file +912
-0+ } +printf_verbose ("tid %d: number of rseq abort: %d, signals delivered: %u\n" , + (int )gettid (), nr_abort, signals_delivered); +if (!opt_disable_rseq && thread_data->reg + &&rseq_unregister_current_thread ()) +abort (); +return NULL ; +} + +void test_percpu_inc (void ) +{ +const int num_threads = opt_threads; +int i, ret; +uint64_t sum; +pthread_t test_threads[num_threads]; +struct inc_test_data data; +struct inc_thread_test_data thread_data[num_threads]; + +memset (&data,0 ,sizeof (data)); +for (i =0 ; i < num_threads; i++) { + thread_data[i].reps = opt_reps; +if (opt_disable_mod <=0 || (i % opt_disable_mod)) + thread_data[i].reg =1 ; +else + thread_data[i].reg =0 ; + thread_data[i].data = &data; + ret =pthread_create (&test_threads[i],NULL , + test_percpu_inc_thread, &thread_data[i]); +if (ret) { + errno = ret; +perror ("pthread_create" ); +abort (); + } + } + +for (i =0 ; i < num_threads; i++) { +pthread_join (test_threads[i],NULL ); +if (ret) { + errno = ret; +perror ("pthread_join" ); +abort (); + } + } + + sum =0 ; +for (i =0 ; i <CPU_SETSIZE ; i++) + sum += data.c[i].count; + +assert (sum == (uint64_t )opt_reps * num_threads); +} + +int percpu_list_push (struct percpu_list *list,struct percpu_list_node *node) +{ +intptr_t *targetptr, newval, expect; +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t )READ_ONCE (list->c[cpu].head); + newval = (intptr_t )node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret =rseq_cmpeqv_storev (targetptr, expect, newval, cpu); +if (likely (!ret)) +return cpu; +#endif +/* Fallback on cpu_opv system call. */ +slowpath: + __attribute__((unused)); +for (;;) { + cpu =rseq_current_cpu (); +/* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t )READ_ONCE (list->c[cpu].head); + newval = (intptr_t )node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret =cpu_op_cmpeqv_storev (targetptr, expect, newval, cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +return cpu; +} + +/* +* Unlike a traditional lock-less linked list; the availability of a +* rseq primitive allows us to implement pop without concerns over +* ABA-type races. +*/ +struct percpu_list_node *percpu_list_pop (struct percpu_list *list) +{ +struct percpu_list_node *head; +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_cmpnev_storeoffp_load ((intptr_t *)&list->c[cpu].head, + (intptr_t )NULL , + offsetof(struct percpu_list_node , next), + (intptr_t *)&head, cpu); +if (likely (!ret)) +return head; +if (ret >0 ) +return NULL ; +#endif +/* Fallback on cpu_opv system call. */ + slowpath: + __attribute__((unused)); +for (;;) { + cpu =rseq_current_cpu (); + ret =cpu_op_cmpnev_storeoffp_load ( + (intptr_t *)&list->c[cpu].head, + (intptr_t )NULL , + offsetof(struct percpu_list_node , next), + (intptr_t *)&head, cpu); +if (likely (!ret)) +break ; +if (ret >0 ) +return NULL ; +assert (ret >=0 || errno ==EAGAIN ); + } +return head; +} + +void *test_percpu_list_thread (void *arg) +{ +long long i, reps; +struct percpu_list *list = (struct percpu_list *)arg; + +if (!opt_disable_rseq &&rseq_register_current_thread ()) +abort (); + + reps = opt_reps; +for (i =0 ; i < reps; i++) { +struct percpu_list_node *node =percpu_list_pop (list); + +if (opt_yield) +sched_yield ();/* encourage shuffling */ +if (node) +percpu_list_push (list, node); + } + +printf_verbose ("tid %d: number of rseq abort: %d, signals delivered: %u\n" , + (int )gettid (), nr_abort, signals_delivered); +if (!opt_disable_rseq &&rseq_unregister_current_thread ()) +abort (); + +return NULL ; +} + +/* Simultaneous modification to a per-cpu linked list from many threads. */ +void test_percpu_list (void ) +{ +const int num_threads = opt_threads; +int i, j, ret; +uint64_t sum =0 , expected_sum =0 ; +struct percpu_list list; +pthread_t test_threads[num_threads]; +cpu_set_t allowed_cpus; + +memset (&list,0 ,sizeof (list)); + +/* Generate list entries for every usable cpu. */ +sched_getaffinity (0 ,sizeof (allowed_cpus), &allowed_cpus); +for (i =0 ; i <CPU_SETSIZE ; i++) { +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; +for (j =1 ; j <=100 ; j++) { +struct percpu_list_node *node; + + expected_sum += j; + + node =malloc (sizeof (*node)); +assert (node); + node->data = j; + node->next = list.c[i].head; + list.c[i].head = node; + } + } + +for (i =0 ; i < num_threads; i++) { + ret =pthread_create (&test_threads[i],NULL , + test_percpu_list_thread, &list); +if (ret) { + errno = ret; +perror ("pthread_create" ); +abort (); + } + } + +for (i =0 ; i < num_threads; i++) { +pthread_join (test_threads[i],NULL ); +if (ret) { + errno = ret; +perror ("pthread_join" ); +abort (); + } + } + +for (i =0 ; i <CPU_SETSIZE ; i++) { +cpu_set_t pin_mask; +struct percpu_list_node *node; + +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; + +CPU_ZERO (&pin_mask); +CPU_SET (i, &pin_mask); +sched_setaffinity (0 ,sizeof (pin_mask), &pin_mask); + +while ((node =percpu_list_pop (&list))) { + sum += node->data; +free (node); + } + } + +/* +* All entries should now be accounted for (unless some external +* actor is interfering with our allowed affinity while this +* test is running). +*/ +assert (sum == expected_sum); +} + +bool percpu_buffer_push (struct percpu_buffer *buffer, +struct percpu_buffer_node *node) +{ +intptr_t *targetptr_spec, newval_spec; +intptr_t *targetptr_final, newval_final; +int cpu, ret; +intptr_t offset; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset == buffer->c[cpu].buflen) { +if (unlikely (cpu !=rseq_current_cpu_raw ())) + goto slowpath; +return false; + } + newval_spec = (intptr_t )node; + targetptr_spec = (intptr_t *)&buffer->c[cpu].array[offset]; + newval_final = offset +1 ; + targetptr_final = &buffer->c[cpu].offset; +if (opt_mb) + ret =rseq_cmpeqv_trystorev_storev_release (targetptr_final, + offset, targetptr_spec, newval_spec, + newval_final, cpu); +else + ret =rseq_cmpeqv_trystorev_storev (targetptr_final, + offset, targetptr_spec, newval_spec, + newval_final, cpu); +if (likely (!ret)) +return true; +#endif +slowpath: + __attribute__((unused)); +/* Fallback on cpu_opv system call. */ +for (;;) { + cpu =rseq_current_cpu (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset == buffer->c[cpu].buflen) +return false; + newval_spec = (intptr_t )node; + targetptr_spec = (intptr_t *)&buffer->c[cpu].array[offset]; + newval_final = offset +1 ; + targetptr_final = &buffer->c[cpu].offset; +if (opt_mb) + ret =cpu_op_cmpeqv_storev_mb_storev (targetptr_final, + offset, targetptr_spec, newval_spec, + newval_final, cpu); +else + ret =cpu_op_cmpeqv_storev_storev (targetptr_final, + offset, targetptr_spec, newval_spec, + newval_final, cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +return true; +} + +struct percpu_buffer_node *percpu_buffer_pop (struct percpu_buffer *buffer) +{ +struct percpu_buffer_node *head; +intptr_t *targetptr, newval; +int cpu, ret; +intptr_t offset; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset ==0 ) { +if (unlikely (cpu !=rseq_current_cpu_raw ())) + goto slowpath; +return NULL ; + } + head = buffer->c[cpu].array[offset -1 ]; + newval = offset -1 ; + targetptr = (intptr_t *)&buffer->c[cpu].offset; + ret =rseq_cmpeqv_cmpeqv_storev (targetptr, offset, + (intptr_t *)&buffer->c[cpu].array[offset -1 ], (intptr_t )head, + newval, cpu); +if (likely (!ret)) +return head; +#endif +slowpath: + __attribute__((unused)); +/* Fallback on cpu_opv system call. */ +for (;;) { + cpu =rseq_current_cpu (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset ==0 ) +return NULL ; + head = buffer->c[cpu].array[offset -1 ]; + newval = offset -1 ; + targetptr = (intptr_t *)&buffer->c[cpu].offset; + ret =cpu_op_cmpeqv_cmpeqv_storev (targetptr, offset, + (intptr_t *)&buffer->c[cpu].array[offset -1 ], + (intptr_t )head, newval, cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +return head; +} + +void *test_percpu_buffer_thread (void *arg) +{ +long long i, reps; +struct percpu_buffer *buffer = (struct percpu_buffer *)arg; + +if (!opt_disable_rseq &&rseq_register_current_thread ()) +abort (); + + reps = opt_reps; +for (i =0 ; i < reps; i++) { +struct percpu_buffer_node *node =percpu_buffer_pop (buffer); + +if (opt_yield) +sched_yield ();/* encourage shuffling */ +if (node) { +if (!percpu_buffer_push (buffer, node)) { +/* Should increase buffer size. */ +abort (); + } + } + } + +printf_verbose ("tid %d: number of rseq abort: %d, signals delivered: %u\n" , + (int )gettid (), nr_abort, signals_delivered); +if (!opt_disable_rseq &&rseq_unregister_current_thread ()) +abort (); + +return NULL ; +} + +/* Simultaneous modification to a per-cpu buffer from many threads. */ +void test_percpu_buffer (void ) +{ +const int num_threads = opt_threads; +int i, j, ret; +uint64_t sum =0 , expected_sum =0 ; +struct percpu_buffer buffer; +pthread_t test_threads[num_threads]; +cpu_set_t allowed_cpus; + +memset (&buffer,0 ,sizeof (buffer)); + +/* Generate list entries for every usable cpu. */ +sched_getaffinity (0 ,sizeof (allowed_cpus), &allowed_cpus); +for (i =0 ; i <CPU_SETSIZE ; i++) { +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; +/* Worse-case is every item in same CPU. */ + buffer.c[i].array = +malloc (sizeof (*buffer.c[i].array) *CPU_SETSIZE + *BUFFER_ITEM_PER_CPU ); +assert (buffer.c[i].array); + buffer.c[i].buflen =CPU_SETSIZE *BUFFER_ITEM_PER_CPU ; +for (j =1 ; j <=BUFFER_ITEM_PER_CPU ; j++) { +struct percpu_buffer_node *node; + + expected_sum += j; + +/* +* We could theoretically put the word-sized +* "data" directly in the buffer. However, we +* want to model objects that would not fit +* within a single word, so allocate an object +* for each node. +*/ + node =malloc (sizeof (*node)); +assert (node); + node->data = j; + buffer.c[i].array[j -1 ] = node; + buffer.c[i].offset++; + } + } + +for (i =0 ; i < num_threads; i++) { + ret =pthread_create (&test_threads[i],NULL , + test_percpu_buffer_thread, &buffer); +if (ret) { + errno = ret; +perror ("pthread_create" ); +abort (); + } + } + +for (i =0 ; i < num_threads; i++) { +pthread_join (test_threads[i],NULL ); +if (ret) { + errno = ret; +perror ("pthread_join" ); +abort (); + } + } + +for (i =0 ; i <CPU_SETSIZE ; i++) { +cpu_set_t pin_mask; +struct percpu_buffer_node *node; + +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; + +CPU_ZERO (&pin_mask); +CPU_SET (i, &pin_mask); +sched_setaffinity (0 ,sizeof (pin_mask), &pin_mask); + +while ((node =percpu_buffer_pop (&buffer))) { + sum += node->data; +free (node); + } +free (buffer.c[i].array); + } + +/* +* All entries should now be accounted for (unless some external +* actor is interfering with our allowed affinity while this +* test is running). +*/ +assert (sum == expected_sum); +} + +bool percpu_memcpy_buffer_push (struct percpu_memcpy_buffer *buffer, +struct percpu_memcpy_buffer_node item) +{ +char *destptr, *srcptr; +size_t copylen; +intptr_t *targetptr_final, newval_final; +int cpu, ret; +intptr_t offset; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset == buffer->c[cpu].buflen) { +if (unlikely (cpu !=rseq_current_cpu_raw ())) + goto slowpath; +return false; + } + destptr = (char *)&buffer->c[cpu].array[offset]; + srcptr = (char *)&item; + copylen =sizeof (item); + newval_final = offset +1 ; + targetptr_final = &buffer->c[cpu].offset; +if (opt_mb) + ret =rseq_cmpeqv_trymemcpy_storev_release (targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); +else + ret =rseq_cmpeqv_trymemcpy_storev (targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); +if (likely (!ret)) +return true; +#endif +slowpath: + __attribute__((unused)); +/* Fallback on cpu_opv system call. */ +for (;;) { + cpu =rseq_current_cpu (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset == buffer->c[cpu].buflen) +return false; + destptr = (char *)&buffer->c[cpu].array[offset]; + srcptr = (char *)&item; + copylen =sizeof (item); + newval_final = offset +1 ; + targetptr_final = &buffer->c[cpu].offset; +/* copylen must be <= PAGE_SIZE. */ +if (opt_mb) + ret =cpu_op_cmpeqv_memcpy_mb_storev (targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); +else + ret =cpu_op_cmpeqv_memcpy_storev (targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +return true; +} + +bool percpu_memcpy_buffer_pop (struct percpu_memcpy_buffer *buffer, +struct percpu_memcpy_buffer_node *item) +{ +char *destptr, *srcptr; +size_t copylen; +intptr_t *targetptr_final, newval_final; +int cpu, ret; +intptr_t offset; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset ==0 ) { +if (unlikely (cpu !=rseq_current_cpu_raw ())) + goto slowpath; +return false; + } + destptr = (char *)item; + srcptr = (char *)&buffer->c[cpu].array[offset -1 ]; + copylen =sizeof (*item); + newval_final = offset -1 ; + targetptr_final = &buffer->c[cpu].offset; + ret =rseq_cmpeqv_trymemcpy_storev (targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); +if (likely (!ret)) +return true; +#endif +slowpath: + __attribute__((unused)); +/* Fallback on cpu_opv system call. */ +for (;;) { + cpu =rseq_current_cpu (); +/* Load offset with single-copy atomicity. */ + offset =READ_ONCE (buffer->c[cpu].offset); +if (offset ==0 ) +return false; + destptr = (char *)item; + srcptr = (char *)&buffer->c[cpu].array[offset -1 ]; + copylen =sizeof (*item); + newval_final = offset -1 ; + targetptr_final = &buffer->c[cpu].offset; +/* copylen must be <= PAGE_SIZE. */ + ret =cpu_op_cmpeqv_memcpy_storev (targetptr_final, + offset, destptr, srcptr, copylen, + newval_final, cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + } +return true; +} + +void *test_percpu_memcpy_buffer_thread (void *arg) +{ +long long i, reps; +struct percpu_memcpy_buffer *buffer = (struct percpu_memcpy_buffer *)arg; + +if (!opt_disable_rseq &&rseq_register_current_thread ()) +abort (); + + reps = opt_reps; +for (i =0 ; i < reps; i++) { +struct percpu_memcpy_buffer_node item; +bool result; + + result =percpu_memcpy_buffer_pop (buffer, &item); +if (opt_yield) +sched_yield ();/* encourage shuffling */ +if (result) { +if (!percpu_memcpy_buffer_push (buffer, item)) { +/* Should increase buffer size. */ +abort (); + } + } + } + +printf_verbose ("tid %d: number of rseq abort: %d, signals delivered: %u\n" , + (int )gettid (), nr_abort, signals_delivered); +if (!opt_disable_rseq &&rseq_unregister_current_thread ()) +abort (); + +return NULL ; +} + +/* Simultaneous modification to a per-cpu buffer from many threads. */ +void test_percpu_memcpy_buffer (void ) +{ +const int num_threads = opt_threads; +int i, j, ret; +uint64_t sum =0 , expected_sum =0 ; +struct percpu_memcpy_buffer buffer; +pthread_t test_threads[num_threads]; +cpu_set_t allowed_cpus; + +memset (&buffer,0 ,sizeof (buffer)); + +/* Generate list entries for every usable cpu. */ +sched_getaffinity (0 ,sizeof (allowed_cpus), &allowed_cpus); +for (i =0 ; i <CPU_SETSIZE ; i++) { +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; +/* Worse-case is every item in same CPU. */ + buffer.c[i].array = +malloc (sizeof (*buffer.c[i].array) *CPU_SETSIZE + *MEMCPY_BUFFER_ITEM_PER_CPU ); +assert (buffer.c[i].array); + buffer.c[i].buflen =CPU_SETSIZE *MEMCPY_BUFFER_ITEM_PER_CPU ; +for (j =1 ; j <=MEMCPY_BUFFER_ITEM_PER_CPU ; j++) { + expected_sum +=2 * j +1 ; + +/* +* We could theoretically put the word-sized +* "data" directly in the buffer. However, we +* want to model objects that would not fit +* within a single word, so allocate an object +* for each node. +*/ + buffer.c[i].array[j -1 ].data1 = j; + buffer.c[i].array[j -1 ].data2 = j +1 ; + buffer.c[i].offset++; + } + } + +for (i =0 ; i < num_threads; i++) { + ret =pthread_create (&test_threads[i],NULL , + test_percpu_memcpy_buffer_thread, &buffer); +if (ret) { + errno = ret; +perror ("pthread_create" ); +abort (); + } + } + +for (i =0 ; i < num_threads; i++) { +pthread_join (test_threads[i],NULL ); +if (ret) { + errno = ret; +perror ("pthread_join" ); +abort (); + } + } + +for (i =0 ; i <CPU_SETSIZE ; i++) { +cpu_set_t pin_mask; +struct percpu_memcpy_buffer_node item; + +if (!CPU_ISSET (i, &allowed_cpus)) +continue ; + +CPU_ZERO (&pin_mask); +CPU_SET (i, &pin_mask); +sched_setaffinity (0 ,sizeof (pin_mask), &pin_mask); + +while (percpu_memcpy_buffer_pop (&buffer, &item)) { + sum += item.data1; + sum += item.data2; + } +free (buffer.c[i].array); + } + +/* +* All entries should now be accounted for (unless some external +* actor is interfering with our allowed affinity while this +* test is running). +*/ +assert (sum == expected_sum); +} + +static void test_signal_interrupt_handler (int signo) +{ + signals_delivered++; +} + +static int set_signal_handler (void ) +{ +int ret =0 ; +struct sigaction sa; +sigset_t sigset; + + ret =sigemptyset (&sigset); +if (ret <0 ) { +perror ("sigemptyset" ); +return ret; + } + + sa.sa_handler = test_signal_interrupt_handler; + sa.sa_mask = sigset; + sa.sa_flags =0 ; + ret =sigaction (SIGUSR1 , &sa,NULL ); +if (ret <0 ) { +perror ("sigaction" ); +return ret; + } + +printf_verbose ("Signal handler set for SIGUSR1\n" ); + +return ret; +} + +static void show_usage (int argc,char **argv) +{ +printf ("Usage : %s <OPTIONS>\n" , + argv[0 ]); +printf ("OPTIONS:\n" ); +printf (" [-1 loops] Number of loops for delay injection 1\n" ); +printf (" [-2 loops] Number of loops for delay injection 2\n" ); +printf (" [-3 loops] Number of loops for delay injection 3\n" ); +printf (" [-4 loops] Number of loops for delay injection 4\n" ); +printf (" [-5 loops] Number of loops for delay injection 5\n" ); +printf (" [-6 loops] Number of loops for delay injection 6\n" ); +printf (" [-7 loops] Number of loops for delay injection 7 (-1 to enable -m)\n" ); +printf (" [-8 loops] Number of loops for delay injection 8 (-1 to enable -m)\n" ); +printf (" [-9 loops] Number of loops for delay injection 9 (-1 to enable -m)\n" ); +printf (" [-m N] Yield/sleep/kill every modulo N (default 0: disabled) (>= 0)\n" ); +printf (" [-y] Yield\n" ); +printf (" [-k] Kill thread with signal\n" ); +printf (" [-s S] S: =0: disabled (default), >0: sleep time (ms)\n" ); +printf (" [-t N] Number of threads (default 200)\n" ); +printf (" [-r N] Number of repetitions per thread (default 5000)\n" ); +printf (" [-d] Disable rseq system call (no initialization)\n" ); +printf (" [-D M] Disable rseq for each M threads\n" ); +printf (" [-T test] Choose test: (s)pinlock, (l)ist, (b)uffer, (m)emcpy, (i)ncrement\n" ); +printf (" [-M] Push into buffer and memcpy buffer with memory barriers.\n" ); +printf (" [-v] Verbose output.\n" ); +printf (" [-h] Show this help.\n" ); +printf ("\n" ); +} + +int main (int argc,char **argv) +{ +int i; + +for (i =1 ; i < argc; i++) { +if (argv[i][0 ] !='-' ) +continue ; +switch (argv[i][1 ]) { +case '1' : +case '2' : +case '3' : +case '4' : +case '5' : +case '6' : +case '7' : +case '8' : +case '9' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + loop_cnt[argv[i][1 ] -'0' ] =atol (argv[i +1 ]); + i++; +break ; +case 'm' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + opt_modulo =atol (argv[i +1 ]); +if (opt_modulo <0 ) { +show_usage (argc, argv); + goto error; + } + i++; +break ; +case 's' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + opt_sleep =atol (argv[i +1 ]); +if (opt_sleep <0 ) { +show_usage (argc, argv); + goto error; + } + i++; +break ; +case 'y' : + opt_yield =1 ; +break ; +case 'k' : + opt_signal =1 ; +break ; +case 'd' : + opt_disable_rseq =1 ; +break ; +case 'D' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + opt_disable_mod =atol (argv[i +1 ]); +if (opt_disable_mod <0 ) { +show_usage (argc, argv); + goto error; + } + i++; +break ; +case 't' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + opt_threads =atol (argv[i +1 ]); +if (opt_threads <0 ) { +show_usage (argc, argv); + goto error; + } + i++; +break ; +case 'r' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + opt_reps =atoll (argv[i +1 ]); +if (opt_reps <0 ) { +show_usage (argc, argv); + goto error; + } + i++; +break ; +case 'h' : +show_usage (argc, argv); + goto end; +case 'T' : +if (argc < i +2 ) { +show_usage (argc, argv); + goto error; + } + opt_test = *argv[i +1 ]; +switch (opt_test) { +case 's' : +case 'l' : +case 'i' : +case 'b' : +case 'm' : +break ; +default : +show_usage (argc, argv); + goto error; + } + i++; +break ; +case 'v' : + verbose =1 ; +break ; +case 'M' : + opt_mb =1 ; +break ; +default : +show_usage (argc, argv); + goto error; + } + } + +if (set_signal_handler ()) + goto error; + +if (!opt_disable_rseq &&rseq_register_current_thread ()) + goto error; +switch (opt_test) { +case 's' : +printf_verbose ("spinlock\n" ); +test_percpu_spinlock (); +break ; +case 'l' : +printf_verbose ("linked list\n" ); +test_percpu_list (); +break ; +case 'b' : +printf_verbose ("buffer\n" ); +test_percpu_buffer (); +break ; +case 'm' : +printf_verbose ("memcpy buffer\n" ); +test_percpu_memcpy_buffer (); +break ; +case 'i' : +printf_verbose ("counter increment\n" ); +test_percpu_inc (); +break ; + } +if (!opt_disable_rseq &&rseq_unregister_current_thread ()) +abort (); +end: +return 0 ; + +error: +return -1 ; +}tools/testing/selftests/rseq/rseq-arm.h new file +535
-0@@ -0,0 +1,535 @@ +/* +* rseq-arm.h +* +* (C) Copyright 2016 - Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#define RSEQ_SIG 0x53053053 + +#define rseq_smp_mb () __asm__ __volatile__ ("dmb" : : : "memory") +#define rseq_smp_rmb () __asm__ __volatile__ ("dmb" : : : "memory") +#define rseq_smp_wmb () __asm__ __volatile__ ("dmb" : : : "memory") + +#define rseq_smp_load_acquire (p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + rseq_smp_mb(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep () rseq_smp_rmb() + +#define rseq_smp_store_release (p, v) \ +do { \ + rseq_smp_mb(); \ + WRITE_ONCE(*p, v); \ +} while (0) + +#define RSEQ_ASM_DEFINE_TABLE (section, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".pushsection " __rseq_str(section) ", \"aw\"\n\t" \ + ".balign 32\n\t" \ + ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + ".word " __rseq_str(start_ip) ", 0x0, " __rseq_str(post_commit_offset) ", 0x0, " __rseq_str(abort_ip) ", 0x0\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_STORE_RSEQ_CS (label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "adr r0, " __rseq_str(cs_label) "\n\t" \ + "str r0, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t" + +#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "ldr r0, %[" __rseq_str(current_cpu_id) "]\n\t" \ + "cmp %[" __rseq_str(cpu_id) "], r0\n\t" \ + "bne " __rseq_str(label) "\n\t" + +#define RSEQ_ASM_DEFINE_ABORT (table_label, label, section, sig, \ + teardown, abort_label, version, flags, start_ip,\ + post_commit_offset, abort_ip) \ + __rseq_str(table_label) ":\n\t" \ + ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + ".word " __rseq_str(start_ip) ", 0x0, " __rseq_str(post_commit_offset) ", 0x0, " __rseq_str(abort_ip) ", 0x0\n\t" \ + ".word " __rseq_str(RSEQ_SIG) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(abort_label) "]\n\t" + +#define RSEQ_ASM_DEFINE_CMPFAIL (label, section, teardown, cmpfail_label) \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(cmpfail_label) "]\n\t" + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expect], r0\n\t" +"bne 5f\n\t" +RSEQ_INJECT_ASM (4 ) + /* final store */ +"str %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(5) +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG ,"" , abort, + 0x0, 0x0, 1b, 2b-1b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, __rseq_failure, "", cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expectnot], r0\n\t" +"beq 5f\n\t" + RSEQ_INJECT_ASM(4) +"str r0, %[load]\n\t" +"add r0, %[voffp]\n\t" +"ldr r0, [r0]\n\t" +/* final store */ +"str r0, %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(5) +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG ,"" , abort, + 0x0, 0x0, 1b, 2b-1b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, __rseq_failure, "", cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expectnot]"r"(expectnot), + [voffp]"Ir"(voffp), + [load]"m"(*load) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_addv (intptr_t *v,intptr_t count,int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"add r0, %[count]\n\t" +/* final store */ +"str r0, %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(4) +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG ,"" , abort, + 0x0, 0x0, 1b, 2b-1b, 4f) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [count]"Ir"(count) + RSEQ_INJECT_INPUT + :"r0" ,"memory" ,"cc" +RSEQ_INJECT_CLOBBER + : abort + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expect], r0\n\t" +"bne 5f\n\t" +RSEQ_INJECT_ASM (4 ) + /* try store */ +"str %[newv2], %[v2]\n\t" +RSEQ_INJECT_ASM (5 ) + /* final store */ +"str %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(6) +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG ,"" , abort, + 0x0, 0x0, 1b, 2b-1b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, __rseq_failure, "", cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* try store input */ + [v2]"m"(*v2), + [newv2]"r"(newv2), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expect], r0\n\t" +"bne 5f\n\t" +RSEQ_INJECT_ASM (4 ) + /* try store */ +"str %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) +"dmb\n\t" /* full mb provides store-release */ +/* final store */ +"str %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(6) +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG ,"" , abort, + 0x0, 0x0, 1b, 2b-1b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, __rseq_failure, "", cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* try store input */ + [v2]"m"(*v2), + [newv2]"r"(newv2), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t expect2,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expect], r0\n\t" +"bne 5f\n\t" + RSEQ_INJECT_ASM(4) +"ldr r0, %[v2]\n\t" +"cmp %[expect2], r0\n\t" +"bne 5f\n\t" +RSEQ_INJECT_ASM (5 ) + /* final store */ +"str %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(6) +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG ,"" , abort, + 0x0, 0x0, 1b, 2b-1b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, __rseq_failure, "", cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* cmp2 input */ + [v2]"m"(*v2), + [expect2]"r"(expect2), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu) +{ +uint32_t rseq_scratch[3 ]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +"str %[src], %[rseq_scratch0]\n\t" +"str %[dst], %[rseq_scratch1]\n\t" +"str %[len], %[rseq_scratch2]\n\t" + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expect], r0\n\t" +"bne 5f\n\t" +RSEQ_INJECT_ASM (4 ) + /* try memcpy */ +"cmp %[len], #0\n\t" \ +"beq 333f\n\t" \ +"222:\n\t" \ +"ldrb %%r0, [%[src]]\n\t" \ +"strb %%r0, [%[dst]]\n\t" \ +"adds %[src], #1\n\t" \ +"adds %[dst], #1\n\t" \ +"subs %[len], #1\n\t" \ +"bne 222b\n\t" \ +"333:\n\t" \ +RSEQ_INJECT_ASM (5 ) + /* final store */ +"str %[newv], %[v]\n\t" +"2:\n\t" +RSEQ_INJECT_ASM (6 ) + /* teardown */ +"ldr %[len], %[rseq_scratch2]\n\t" +"ldr %[dst], %[rseq_scratch1]\n\t" +"ldr %[src], %[rseq_scratch0]\n\t" +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG , +/* teardown */ +"ldr %[len], %[rseq_scratch2]\n\t" +"ldr %[dst], %[rseq_scratch1]\n\t" +"ldr %[src], %[rseq_scratch0]\n\t" , + abort, 0x0, 0x0, 1b, 2b-1b, 4f) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure, +/* teardown */ +"ldr %[len], %[rseq_scratch2]\n\t" +"ldr %[dst], %[rseq_scratch1]\n\t" +"ldr %[src], %[rseq_scratch0]\n\t" , + cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv), +/* try memcpy input */ + [dst]"r"(dst), + [src]"r"(src), + [len]"r"(len), + [rseq_scratch0]"m"(rseq_scratch[0]), + [rseq_scratch1]"m"(rseq_scratch[1]), + [rseq_scratch2]"m"(rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu) +{ +uint32_t rseq_scratch[3 ]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +"str %[src], %[rseq_scratch0]\n\t" +"str %[dst], %[rseq_scratch1]\n\t" +"str %[len], %[rseq_scratch2]\n\t" + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"ldr r0, %[v]\n\t" +"cmp %[expect], r0\n\t" +"bne 5f\n\t" +RSEQ_INJECT_ASM (4 ) + /* try memcpy */ +"cmp %[len], #0\n\t" \ +"beq 333f\n\t" \ +"222:\n\t" \ +"ldrb %%r0, [%[src]]\n\t" \ +"strb %%r0, [%[dst]]\n\t" \ +"adds %[src], #1\n\t" \ +"adds %[dst], #1\n\t" \ +"subs %[len], #1\n\t" \ +"bne 222b\n\t" \ +"333:\n\t" \ + RSEQ_INJECT_ASM(5) +"dmb\n\t" /* full mb provides store-release */ +/* final store */ +"str %[newv], %[v]\n\t" +"2:\n\t" +RSEQ_INJECT_ASM (6 ) + /* teardown */ +"ldr %[len], %[rseq_scratch2]\n\t" +"ldr %[dst], %[rseq_scratch1]\n\t" +"ldr %[src], %[rseq_scratch0]\n\t" +"b 6f\n\t" +RSEQ_ASM_DEFINE_ABORT (3 ,4 , __rseq_failure,RSEQ_SIG , +/* teardown */ +"ldr %[len], %[rseq_scratch2]\n\t" +"ldr %[dst], %[rseq_scratch1]\n\t" +"ldr %[src], %[rseq_scratch0]\n\t" , + abort, 0x0, 0x0, 1b, 2b-1b, 4f) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure, +/* teardown */ +"ldr %[len], %[rseq_scratch2]\n\t" +"ldr %[dst], %[rseq_scratch1]\n\t" +"ldr %[src], %[rseq_scratch0]\n\t" , + cmpfail) +"6:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv), +/* try memcpy input */ + [dst]"r"(dst), + [src]"r"(src), + [len]"r"(len), + [rseq_scratch0]"m"(rseq_scratch[0]), + [rseq_scratch1]"m"(rseq_scratch[1]), + [rseq_scratch2]"m"(rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +}tools/testing/selftests/rseq/rseq-ppc.h new file +567
-0@@ -0,0 +1,567 @@ +/* +* rseq-ppc.h +* +* (C) Copyright 2016 - Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/[email protected]> +* (C) Copyright 2016 - Boqun Feng <[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#define RSEQ_SIG 0x53053053 + +#define rseq_smp_mb () __asm__ __volatile__ ("sync" : : : "memory") +#define rseq_smp_lwsync () __asm__ __volatile__ ("lwsync" : : : "memory") +#define rseq_smp_rmb () rseq_smp_lwsync() +#define rseq_smp_wmb () rseq_smp_lwsync() + +#define rseq_smp_load_acquire (p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + rseq_smp_lwsync(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep () rseq_smp_lwsync() + +#define rseq_smp_store_release (p, v) \ +do { \ + rseq_smp_lwsync(); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + +/* +* The __rseq_table section can be used by debuggers to better handle +* single-stepping through the restartable critical sections. +*/ + +#ifdef __PPC64__ + +#define STORE_WORD "std " +#define LOAD_WORD "ld " +#define LOADX_WORD "ldx " +#define CMP_WORD "cmpd " + +#define RSEQ_ASM_DEFINE_TABLE (label, section, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".pushsection " __rseq_str(section) ", \"aw\"\n\t" \ + ".balign 32\n\t" \ + __rseq_str(label) ":\n\t" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + ".quad " __rseq_str(start_ip) ", " __rseq_str(post_commit_offset) ", " __rseq_str(abort_ip) "\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_STORE_RSEQ_CS (label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "lis %%r17, (" __rseq_str(cs_label) ")@highest\n\t" \ + "ori %%r17, %%r17, (" __rseq_str(cs_label) ")@higher\n\t" \ + "rldicr %%r17, %%r17, 32, 31\n\t" \ + "oris %%r17, %%r17, (" __rseq_str(cs_label) ")@high\n\t" \ + "ori %%r17, %%r17, (" __rseq_str(cs_label) ")@l\n\t" \ + "std %%r17, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t" + +#else /* #ifdef __PPC64__ */ + +#define STORE_WORD "stw " +#define LOAD_WORD "lwz " +#define LOADX_WORD "lwzx " +#define CMP_WORD "cmpw " + +#define RSEQ_ASM_DEFINE_TABLE (label, section, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".pushsection " __rseq_str(section) ", \"aw\"\n\t" \ + ".balign 32\n\t" \ + __rseq_str(label) ":\n\t" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ +/* 32-bit only supported on BE */ \ + ".long 0x0, " __rseq_str(start_ip) ", 0x0, " __rseq_str(post_commit_offset) ", 0x0, " __rseq_str(abort_ip) "\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "lis %%r17, (" __rseq_str(cs_label) ")@ha\n\t" \ + "addi %%r17, %%r17, (" __rseq_str(cs_label) ")@l\n\t" \ + "stw %%r17, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t" + +#endif /* #ifdef __PPC64__ */ + +#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "lwz %%r17, %[" __rseq_str(current_cpu_id) "]\n\t" \ + "cmpw cr7, %[" __rseq_str(cpu_id) "], %%r17\n\t" \ + "bne- cr7, " __rseq_str(label) "\n\t" + +#define RSEQ_ASM_DEFINE_ABORT (label, section, sig, teardown, abort_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ + ".long " __rseq_str(sig) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(abort_label) "]\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_DEFINE_CMPFAIL (label, section, teardown, cmpfail_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(cmpfail_label) "]\n\t" \ + ".popsection\n\t" + + +/* +* RSEQ_ASM_OPs: asm operations for rseq +* RSEQ_ASM_OP_R_*: has hard-code registers in it +* RSEQ_ASM_OP_* (else): doesn't have hard-code registers(unless cr7) +*/ +#define RSEQ_ASM_OP_CMPEQ (var, expect, label) \ + LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ + CMP_WORD "cr7, %%r17, %[" __rseq_str(expect) "]\n\t" \ + "bne- cr7, " __rseq_str(label) "\n\t" + +#define RSEQ_ASM_OP_CMPNE (var, expectnot, label) \ + LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ + CMP_WORD "cr7, %%r17, %[" __rseq_str(expectnot) "]\n\t" \ + "beq- cr7, " __rseq_str(label) "\n\t" + +#define RSEQ_ASM_OP_STORE (value, var) \ + STORE_WORD "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" + +/* Load @var to r17 */ +#define RSEQ_ASM_OP_R_LOAD (var) \ + LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" + +/* Store r17 to @var */ +#define RSEQ_ASM_OP_R_STORE (var) \ + STORE_WORD "%%r17, %[" __rseq_str(var) "]\n\t" + +/* Add @count to r17 */ +#define RSEQ_ASM_OP_R_ADD (count) \ + "add %%r17, %[" __rseq_str(count) "], %%r17\n\t" + +/* Load (r17 + voffp) to r17 */ +#define RSEQ_ASM_OP_R_LOADX (voffp) \ + LOADX_WORD "%%r17, %[" __rseq_str(voffp) "], %%r17\n\t" + +/* TODO: implement a faster memcpy. */ +#define RSEQ_ASM_OP_R_MEMCPY () \ + "cmpdi %%r19, 0\n\t" \ + "beq 333f\n\t" \ + "addi %%r20, %%r20, -1\n\t" \ + "addi %%r21, %%r21, -1\n\t" \ + "222:\n\t" \ + "lbzu %%r18, 1(%%r20)\n\t" \ + "stbu %%r18, 1(%%r21)\n\t" \ + "addi %%r19, %%r19, -1\n\t" \ + "cmpdi %%r19, 0\n\t" \ + "bne 222b\n\t" \ + "333:\n\t" \ + +#define RSEQ_ASM_OP_R_FINAL_STORE (var, post_commit_label) \ + STORE_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ + __rseq_str(post_commit_label) ":\n\t" + +#define RSEQ_ASM_OP_FINAL_STORE (value, var, post_commit_label) \ + STORE_WORD "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \ + __rseq_str(post_commit_label) ":\n\t" + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, 5f) +RSEQ_INJECT_ASM (4 ) + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v not equal to @expectnot */ + RSEQ_ASM_OP_CMPNE(v, expectnot, 5f) +RSEQ_INJECT_ASM (4 ) + /* load the value of @v */ +RSEQ_ASM_OP_R_LOAD (v) +/* store it in @load */ +RSEQ_ASM_OP_R_STORE (load ) +/* dereference voffp(v) */ +RSEQ_ASM_OP_R_LOADX (voffp) +/* final store the value at voffp(v) */ +RSEQ_ASM_OP_R_FINAL_STORE (v,2 ) +RSEQ_INJECT_ASM (5 ) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) + RSEQ_ASM_DEFINE_CMPFAIL(5, __rseq_failure, "", cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expectnot]"r"(expectnot), + [voffp]"b"(voffp), + [load]"m"(*load) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_addv (intptr_t *v,intptr_t count,int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* load the value of @v */ +RSEQ_ASM_OP_R_LOAD (v) +/* add @count to it */ +RSEQ_ASM_OP_R_ADD (count ) +/* final store */ +RSEQ_ASM_OP_R_FINAL_STORE (v,2 ) +RSEQ_INJECT_ASM (4 ) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [count]"r"(count) + RSEQ_INJECT_INPUT + :"memory" ,"cc" ,"r17" +RSEQ_INJECT_CLOBBER + : abort + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, 5f) +RSEQ_INJECT_ASM (4 ) + /* try store */ + RSEQ_ASM_OP_STORE(newv2, v2) +RSEQ_INJECT_ASM (5 ) + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* try store input */ + [v2]"m"(*v2), + [newv2]"r"(newv2), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t newv2,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, 5f) +RSEQ_INJECT_ASM (4 ) + /* try store */ + RSEQ_ASM_OP_STORE(newv2, v2) +RSEQ_INJECT_ASM (5 ) + /* for 'release' */ +"lwsync\n\t" +/* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* try store input */ + [v2]"m"(*v2), + [newv2]"r"(newv2), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev (intptr_t *v,intptr_t expect, +intptr_t *v2,intptr_t expect2,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, 5f) +RSEQ_INJECT_ASM (4 ) + /* cmp @v2 equal to @expct2 */ + RSEQ_ASM_OP_CMPEQ(v2, expect2, 5f) +RSEQ_INJECT_ASM (5 ) + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* cmp2 input */ + [v2]"m"(*v2), + [expect2]"r"(expect2), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +/* setup for mempcy */ +"mr %%r19, %[len]\n\t" \ +"mr %%r20, %[src]\n\t" \ +"mr %%r21, %[dst]\n\t" \ +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, 5f) +RSEQ_INJECT_ASM (4 ) + /* try memcpy */ + RSEQ_ASM_OP_R_MEMCPY() +RSEQ_INJECT_ASM (5 ) + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) +RSEQ_INJECT_ASM (6 ) + /* teardown */ + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv), +/* try memcpy input */ + [dst]"r"(dst), + [src]"r"(src), + [len]"r"(len) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17", "r18", "r19", "r20", "r21" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release (intptr_t *v,intptr_t expect, +void *dst,void *src,size_t len,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +/* setup for mempcy */ +"mr %%r19, %[len]\n\t" \ +"mr %%r20, %[src]\n\t" \ +"mr %%r21, %[dst]\n\t" \ +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, 5f) +RSEQ_INJECT_ASM (4 ) + /* try memcpy */ + RSEQ_ASM_OP_R_MEMCPY() +RSEQ_INJECT_ASM (5 ) + /* for 'release' */ +"lwsync\n\t" +/* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) +RSEQ_INJECT_ASM (6 ) + /* teardown */ + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv), +/* try memcpy input */ + [dst]"r"(dst), + [src]"r"(src), + [len]"r"(len) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17", "r18", "r19", "r20", "r21" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +#undef STORE_WORD +#undef LOAD_WORD +#undef LOADX_WORD +#undef CMP_WORDtools/testing/selftests/rseq/rseq-x86.h new file +898
-0@@ -0,0 +1,898 @@ +/* +* rseq-x86.h +* +* (C) Copyright 2016 - Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#include <stdint.h> + +#define RSEQ_SIG 0x53053053 + +#ifdef __x86_64__ + +#define rseq_smp_mb () __asm__ __volatile__ ("mfence" : : : "memory") +#define rseq_smp_rmb () barrier() +#define rseq_smp_wmb () barrier() + +#define rseq_smp_load_acquire (p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + barrier(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep () rseq_smp_rmb() + +#define rseq_smp_store_release (p, v) \ +do { \ + barrier(); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + +#define RSEQ_ASM_DEFINE_TABLE (label, section, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".pushsection " __rseq_str(section) ", \"aw\"\n\t" \ + ".balign 32\n\t" \ + __rseq_str(label) ":\n\t" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + ".quad " __rseq_str(start_ip) ", " __rseq_str(post_commit_offset) ", " __rseq_str(abort_ip) "\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_STORE_RSEQ_CS (label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "leaq " __rseq_str(cs_label) "(%%rip), %%rax\n\t" \ + "movq %%rax, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t" + +#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "cmpl %[" __rseq_str(cpu_id) "], %[" __rseq_str(current_cpu_id) "]\n\t" \ + "jnz " __rseq_str(label) "\n\t" + +#define RSEQ_ASM_DEFINE_ABORT (label, section, sig, teardown, abort_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ +/* Disassembler-friendly signature: nopl <sig>(%rip). */ \ +".byte 0x0f, 0x1f, 0x05\n\t" \ + ".long " __rseq_str(sig) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "jmp %l[" __rseq_str(abort_label) "]\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_DEFINE_CMPFAIL (label, section, teardown, cmpfail_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \ + ".popsection\n\t" + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"cmpq %[v], %[expect]\n\t" +"jnz 5f\n\t" +RSEQ_INJECT_ASM (4 ) + /* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +"cmpq %[v], %[expectnot]\n\t" +"jz 5f\n\t" + RSEQ_INJECT_ASM(4) +"movq %[v], %%rax\n\t" +"movq %%rax, %[load]\n\t" +"addq %[voffp], %%rax\n\t" +"movq (%%rax), %%rax\n\t" +/* final store */ +"movq %%rax, %[v]\n\t" +"2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expectnot]"r"(expectnot), + [voffp]"er"(voffp), + [load]"m"(*load) + RSEQ_INJECT_INPUT + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail + ); +return 0 ; +abort: +RSEQ_INJECT_FAILED + return -1; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_addv (intptr_t *v,intptr_t count,int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +RSEQ_INJECT_ASM (3 ) + /* final store */ +"addq %[count], %[v]\n\t" +"2:\n\t" Lines of this file are not shown. View the raw message.
tools/testing/selftests/rseq/rseq.c new file +116
-0Lines of this file are not shown. View the raw message.
tools/testing/selftests/rseq/rseq.h new file +154
-0Lines of this file are not shown. View the raw message.
tools/testing/selftests/rseq/run_param_test.sh new file +124
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
thanks,
-- Shuah
Part of the diff is not shown here. View the raw message.
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
----- On Nov 21, 2017, at 10:34 AM, shuah [email protected] wrote:
[...]
Quoted 7 lines
Quoted 4 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 +Thanks for the .gitignore files. It is commonly missed change, I end up adding one to clean things up after tests get in.
I'm used to receive patches where contributors forget to add new files to gitignore within my own projects, which may contribute to my awareness of this pain point. :)
[...]
Quoted 40 lines
Quoted 11 lines
tools/testing/selftests/rseq/param_test.c new file +11
-0+ +void *test_percpu_inc_thread (void *arg) +{ +struct inc_thread_test_data *thread_data = arg; +struct inc_test_data *data = thread_data->data; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps;tools/testing/selftests/rseq/param_test.c line 357
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+for (i =0 ; i < reps; i++) { +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) + goto next; +#endif So the test needs to compiled with this enabled? I think it would be better to make this an argument to be abel to select at test start time as opposed to making this compile time option. Remember that these tests get run in automated test rings. Making this a compile time otpion pertty much ensures that this path will not be tested.
So I would reccommend adding a paratemer.
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+ slowpath: + __attribute__((unused)); +for (;;) { +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + }
Quoted 8 lines
tools/testing/selftests/rseq/param_test.c line 373
Quoted 6 lines
tools/testing/selftests/rseq/param_test.c new file +6
-0+ next: + __attribute__((unused)); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif Same comment as before. Avoid compile time options.
The goal of those compiler define are to generate the altered code without adding branches into the fast-paths.
Here is an alternative solution that should take care of your concern: I'll build multiple targets for param_test.c:
param_testparam_test_skip_fastpath (built with -DSKIP_FASTPATH) param_test_benchmark (build with -DBENCHMARK)
I'll update run_param_test.sh to run both param_test and param_test_skip_fastpath.
Note that "param_test_benchmark" is only useful for benchmarking, so I don't plan to run it from run_param_test.sh which is meant to track regressions.
Is that approach OK with you ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Shuah Khan
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 66 lines · Mathieu Desnoyers
----- On Nov 21, 2017, at 10:34 AM, shuah [email protected] wrote:
[...]
Quoted 7 lines
Quoted 4 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 +Thanks for the .gitignore files. It is commonly missed change, I end up adding one to clean things up after tests get in.
I'm used to receive patches where contributors forget to add new files to gitignore within my own projects, which may contribute to my awareness of this pain point. :)
[...]
Quoted 40 lines
Quoted 11 lines
tools/testing/selftests/rseq/param_test.c new file +11
-0+ +void *test_percpu_inc_thread (void *arg) +{ +struct inc_thread_test_data *thread_data = arg; +struct inc_test_data *data = thread_data->data; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps;tools/testing/selftests/rseq/param_test.c line 357
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+for (i =0 ; i < reps; i++) { +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) + goto next; +#endif So the test needs to compiled with this enabled? I think it would be better to make this an argument to be abel to select at test start time as opposed to making this compile time option. Remember that these tests get run in automated test rings. Making this a compile time otpion pertty much ensures that this path will not be tested.
So I would reccommend adding a paratemer.
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+ slowpath: + __attribute__((unused)); +for (;;) { +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + }Quoted 8 lines
tools/testing/selftests/rseq/param_test.c line 373
Quoted 6 lines
tools/testing/selftests/rseq/param_test.c new file +6
-0+ next: + __attribute__((unused)); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif Same comment as before. Avoid compile time options.
Quoted 2 lines
The goal of those compiler define are to generate the altered code without adding branches into the fast-paths.
That makes sense. You are looking to not add any overhead.
Quoted 8 lines
Here is an alternative solution that should take care of your concern: I'll build multiple targets for
param_test.c:
param_testparam_test_skip_fastpath(built with -DSKIP_FASTPATH)param_test_benchmark(build with -DBENCHMARK)
Quoted 8 lines
I'll update
run_param_test.shto run bothparam_testandparam_test_skip_fastpath.Note that "
param_test_benchmark" is only useful for benchmarking, so I don't plan to run it fromrun_param_test.shwhich is meant to track regressions.Is that approach OK with you ?
Yes. This approach addresses my concern about coverage for both paths.
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 83 lines · ----- On Nov 21, 2017, at 12:40 PM, shuah [email protected] wrote:
Quoted 66 lines · Mathieu Desnoyers
----- On Nov 21, 2017, at 10:34 AM, shuah [email protected] wrote:
[...]
Quoted 7 lines
Quoted 4 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 +Thanks for the .gitignore files. It is commonly missed change, I end up adding one to clean things up after tests get in.
I'm used to receive patches where contributors forget to add new files to gitignore within my own projects, which may contribute to my awareness of this pain point. :)
[...]
Quoted 40 lines
Quoted 11 lines
tools/testing/selftests/rseq/param_test.c new file +11
-0+ +void *test_percpu_inc_thread (void *arg) +{ +struct inc_thread_test_data *thread_data = arg; +struct inc_test_data *data = thread_data->data; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps;tools/testing/selftests/rseq/param_test.c line 357
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+for (i =0 ; i < reps; i++) { +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) + goto next; +#endif So the test needs to compiled with this enabled? I think it would be better to make this an argument to be abel to select at test start time as opposed to making this compile time option. Remember that these tests get run in automated test rings. Making this a compile time otpion pertty much ensures that this path will not be tested.
So I would reccommend adding a paratemer.
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+ slowpath: + __attribute__((unused)); +for (;;) { +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + }Quoted 8 lines
tools/testing/selftests/rseq/param_test.c line 373
Quoted 6 lines
tools/testing/selftests/rseq/param_test.c new file +6
-0+ next: + __attribute__((unused)); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif Same comment as before. Avoid compile time options.
Quoted 2 lines
The goal of those compiler define are to generate the altered code without adding branches into the fast-paths.
That makes sense. You are looking to not add any overhead.
Quoted 11 lines
Here is an alternative solution that should take care of your concern: I'll build multiple targets for
param_test.c:
param_testparam_test_skip_fastpath(built with -DSKIP_FASTPATH)param_test_benchmark(build with -DBENCHMARK)I'll update
run_param_test.shto run bothparam_testandparam_test_skip_fastpath.
Quoted 8 lines
Quoted 6 lines
Note that "
param_test_benchmark" is only useful for benchmarking, so I don't plan to run it fromrun_param_test.shwhich is meant to track regressions.Is that approach OK with you ?
Yes. This approach addresses my concern about coverage for both paths.
fyi, the updated patches can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=a0b8eb0eb5d4d8a280969370aa1dcf51801139c6
"selftests: lib.mk: Introduce OVERRIDE_TARGETS"
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=4ef0214e19bb7415fe7aed6852859b8d66e09a45
"cpu_opv: selftests: Implement selftests (v4)"
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=7d7530b843c7ecb50bea5a136c776cf3e9155d43
"rseq: selftests: Provide self-tests (v4)"
Thanks for the feedback!
Mathieu
Quoted 3 lines
thanks,
-- Shuah
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 99 lines · Mathieu Desnoyers
Quoted 83 lines · ----- On Nov 21, 2017, at 12:40 PM, shuah…
Quoted 66 lines · Mathieu Desnoyers
----- On Nov 21, 2017, at 10:34 AM, shuah [email protected] wrote:
[...]
Quoted 7 lines
Quoted 4 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 +Thanks for the .gitignore files. It is commonly missed change, I end up adding one to clean things up after tests get in.
I'm used to receive patches where contributors forget to add new files to gitignore within my own projects, which may contribute to my awareness of this pain point. :)
[...]
Quoted 40 lines
Quoted 11 lines
tools/testing/selftests/rseq/param_test.c new file +11
-0+ +void *test_percpu_inc_thread (void *arg) +{ +struct inc_thread_test_data *thread_data = arg; +struct inc_test_data *data = thread_data->data; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps;tools/testing/selftests/rseq/param_test.c line 357
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+for (i =0 ; i < reps; i++) { +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) + goto next; +#endif So the test needs to compiled with this enabled? I think it would be better to make this an argument to be abel to select at test start time as opposed to making this compile time option. Remember that these tests get run in automated test rings. Making this a compile time otpion pertty much ensures that this path will not be tested.
So I would reccommend adding a paratemer.
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+ slowpath: + __attribute__((unused)); +for (;;) { +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + }Quoted 8 lines
tools/testing/selftests/rseq/param_test.c line 373
Quoted 6 lines
tools/testing/selftests/rseq/param_test.c new file +6
-0+ next: + __attribute__((unused)); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif Same comment as before. Avoid compile time options.
Quoted 2 lines
The goal of those compiler define are to generate the altered code without adding branches into the fast-paths.
That makes sense. You are looking to not add any overhead.
Quoted 11 lines
Here is an alternative solution that should take care of your concern: I'll build multiple targets for
param_test.c:
param_testparam_test_skip_fastpath(built with -DSKIP_FASTPATH)param_test_benchmark(build with -DBENCHMARK)I'll update
run_param_test.shto run bothparam_testandparam_test_skip_fastpath.Quoted 8 lines
Quoted 6 lines
Note that "
param_test_benchmark" is only useful for benchmarking, so I don't plan to run it fromrun_param_test.shwhich is meant to track regressions.Is that approach OK with you ?
Yes. This approach addresses my concern about coverage for both paths.
fyi, the updated patches can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=a0b8eb0eb5d4d8a280969370aa1dcf51801139c6
"selftests: lib.mk: IntroduceOVERRIDE_TARGETS"
Quoted 8 lines
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=4ef0214e19bb7415fe7aed6852859b8d66e09a45
"cpu_opv: selftests: Implement selftests (v4)"https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=7d7530b843c7ecb50bea5a136c776cf3e9155d43
"rseq: selftests: Provide self-tests (v4)"Thanks for the feedback!
Are you going to send these to the mailing list? That way I can do a final review and give my Ack if they look good.
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 103 lines · ----- On Nov 21, 2017, at 4:24 PM, Shuah Khan [email protected] wrote:
Quoted 102 lines · Mathieu Desnoyers
Quoted 83 lines · ----- On Nov 21, 2017, at 12:40 PM, shuah [email protected] wrote:
Quoted 66 lines · Mathieu Desnoyers
----- On Nov 21, 2017, at 10:34 AM, shuah [email protected] wrote:
[...]
Quoted 7 lines
Quoted 4 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 +Thanks for the .gitignore files. It is commonly missed change, I end up adding one to clean things up after tests get in.
I'm used to receive patches where contributors forget to add new files to gitignore within my own projects, which may contribute to my awareness of this pain point. :)
[...]
Quoted 40 lines
Quoted 11 lines
tools/testing/selftests/rseq/param_test.c new file +11
-0+ +void *test_percpu_inc_thread (void *arg) +{ +struct inc_thread_test_data *thread_data = arg; +struct inc_test_data *data = thread_data->data; +long long i, reps; + +if (!opt_disable_rseq && thread_data->reg + &&rseq_register_current_thread ()) +abort (); + reps = thread_data->reps;tools/testing/selftests/rseq/param_test.c line 357
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+for (i =0 ; i < reps; i++) { +int cpu, ret; + +#ifndef SKIP_FASTPATH +/* Try fast path. */ + cpu =rseq_cpu_start (); + ret =rseq_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) + goto next; +#endif So the test needs to compiled with this enabled? I think it would be better to make this an argument to be abel to select at test start time as opposed to making this compile time option. Remember that these tests get run in automated test rings. Making this a compile time otpion pertty much ensures that this path will not be tested.
So I would reccommend adding a paratemer.
Quoted 10 lines
tools/testing/selftests/rseq/param_test.c new file +10
-0+ slowpath: + __attribute__((unused)); +for (;;) { +/* Fallback on cpu_opv system call. */ + cpu =rseq_current_cpu (); + ret =cpu_op_addv (&data->c[cpu].count,1 , cpu); +if (likely (!ret)) +break ; +assert (ret >=0 || errno ==EAGAIN ); + }Quoted 8 lines
tools/testing/selftests/rseq/param_test.c line 373
Quoted 6 lines
tools/testing/selftests/rseq/param_test.c new file +6
-0+ next: + __attribute__((unused)); +#ifndef BENCHMARK +if (i !=0 && !(i % (reps /10 ))) +printf_verbose ("tid %d: count %lld\n" , (int )gettid (), i); +#endif Same comment as before. Avoid compile time options.
Quoted 2 lines
The goal of those compiler define are to generate the altered code without adding branches into the fast-paths.
That makes sense. You are looking to not add any overhead.
Quoted 11 lines
Here is an alternative solution that should take care of your concern: I'll build multiple targets for
param_test.c:
param_testparam_test_skip_fastpath(built with -DSKIP_FASTPATH)param_test_benchmark(build with -DBENCHMARK)I'll update
run_param_test.shto run bothparam_testandparam_test_skip_fastpath.Quoted 8 lines
Quoted 6 lines
Note that "
param_test_benchmark" is only useful for benchmarking, so I don't plan to run it fromrun_param_test.shwhich is meant to track regressions.Is that approach OK with you ?
Yes. This approach addresses my concern about coverage for both paths.
fyi, the updated patches can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=a0b8eb0eb5d4d8a280969370aa1dcf51801139c6
"selftests: lib.mk: IntroduceOVERRIDE_TARGETS"https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=4ef0214e19bb7415fe7aed6852859b8d66e09a45
"cpu_opv: selftests: Implement selftests (v4)"
Quoted 8 lines
Quoted 5 lines
https://git.kernel.org/pub/scm/linux/kernel/git/rseq/linux-rseq.git/commit/?h=rseq/dev&id=7d7530b843c7ecb50bea5a136c776cf3e9155d43
"rseq: selftests: Provide self-tests (v4)"Thanks for the feedback!
Are you going to send these to the mailing list? That way I can do a final review and give my Ack if they look good.
Sure, I can do one hopefully last round of RFC with those selftests updates.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
In reply to Mathieu DesnoyersQuoted 16 lines · Mathieu Desnoyers
Implements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "
basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works."
basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness."
param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.A
run_param_test.shscript runs many variants of the parametrizable tests.
Quoted 8 lines
As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the
cpu_opvsystem call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the__rseq_tablesection. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.
Could I ask you to split this in smaller bits?
I'd start with just the rseq library, using only the rseq interface. Then add the whole cpu_opv fallback stuff.
Then add the selftests using librseq.
As is this is a tad much to read in a single go.
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 26 lines · ----- On Nov 22, 2017, at 2:38 PM, Peter Zijlstra [email protected] wrote:
Quoted 16 lines · Mathieu Desnoyers
Implements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "
basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works."
basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness."
param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.A
run_param_test.shscript runs many variants of the parametrizable tests.Quoted 8 lines
As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the
cpu_opvsystem call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the__rseq_tablesection. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.
Quoted 7 lines
Could I ask you to split this in smaller bits?
I'd start with just the rseq library, using only the rseq interface. Then add the whole
cpu_opvfallback stuff.
Then add the selftests using librseq.As is this is a tad much to read in a single go.
Sure, will do! And I plan to change the selftests to use the new "percpu_*()" API that removes the need to duplicate code in the caller code.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
In reply to Mathieu DesnoyersQuoted 27 lines · Mathieu Desnoyers
tools/testing/selftests/rseq/rseq-x86.h new file +21
-0@@ -0,0 +1,898 @@ +/* +* rseq-x86.h +* +* (C) Copyright 2016 - Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
tools/testing/selftests/rseq/rseq-x86.h line 31
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h new file +10
-0+* SOFTWARE. +*/ + +#include <stdint.h> + +#define RSEQ_SIG 0x53053053 + +#ifdef __x86_64__ + +#define rseq_smp_mb () __asm__ __volatile__ ("mfence" : : : "memory")
See commit:
450cbdd0125c ("locking/x86: Use LOCK ADD for smp_mb() instead of MFENCE")
Quoted 16 lines
tools/testing/selftests/rseq/rseq-x86.h new file +16
-0+#define rseq_smp_rmb () barrier() +#define rseq_smp_wmb () barrier() + +#define rseq_smp_load_acquire (p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + barrier(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep () rseq_smp_rmb() + +#define rseq_smp_store_release (p, v) \ +do { \ + barrier(); \ + RSEQ_WRITE_ONCE(*p, v); \
tools/testing/selftests/rseq/rseq-x86.h line 57
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h new file +10
-0+} while (0) + +#define RSEQ_ASM_DEFINE_TABLE (label, section, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".pushsection " __rseq_str(section) ", \"aw\"\n\t" \ + ".balign 32\n\t" \ + __rseq_str(label) ":\n\t" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + ".quad " __rseq_str(start_ip) ", " __rseq_str(post_commit_offset) ", " __rseq_str(abort_ip) "\n\t" \ + ".popsection\n\t"
OK, so this creates table entry, but why is @section an argument, AFAICT its _always_ the same thing, no?
tools/testing/selftests/rseq/rseq-x86.h line 63
Quoted 5 lines
tools/testing/selftests/rseq/rseq-x86.h new file +5
-0+#define RSEQ_ASM_STORE_RSEQ_CS (label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "leaq " __rseq_str(cs_label) "(%%rip), %%rax\n\t" \ + "movq %%rax, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t"
And this sets the TLS variable to point to the table entry from the previous macro, no? But again @rseq_cs seems to always be the very same, why is that an argument?
tools/testing/selftests/rseq/rseq-x86.h line 68
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h new file +4
-0+#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "cmpl %[" __rseq_str(cpu_id) "], %[" __rseq_str(current_cpu_id) "]\n\t" \ + "jnz " __rseq_str(label) "\n\t"
more things that are always the same it seems..
tools/testing/selftests/rseq/rseq-x86.h line 78
Quoted 9 lines
tools/testing/selftests/rseq/rseq-x86.h new file +9
-0+#define RSEQ_ASM_DEFINE_ABORT (label, section, sig, teardown, abort_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ +/* Disassembler-friendly signature: nopl <sig>(%rip). */ \ +".byte 0x0f, 0x1f, 0x05\n\t" \ + ".long " __rseq_str(sig) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "jmp %l[" __rseq_str(abort_label) "]\n\t" \ +".popsection\n\t"
@section and @sig seem to always be the same...
tools/testing/selftests/rseq/rseq-x86.h line 85
Quoted 6 lines
tools/testing/selftests/rseq/rseq-x86.h new file +6
-0+#define RSEQ_ASM_DEFINE_CMPFAIL (label, section, teardown, cmpfail_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \ + ".popsection\n\t"
Somewhat failing to see the point of this macro, it seems to just obfuscate the normal failure path.
tools/testing/selftests/rseq/rseq-x86.h line 84
Quoted 3 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu)
I find this a very confusing name for what is essentially compare-and-exchange or compare-and-swap, no?
tools/testing/selftests/rseq/rseq-x86.h line 87
Quoted 3 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)
So we set up the section, but unreadably so... reducing the number of arguments would help a lot.
Rename the current one to __RSEQ_ASM_DEFINE_TABLE() and then use:
#define RSEQ_ASM_DEFINE_TABLE(label, start_ip, post_commit_ip, abort_ip) \ __RSEQ_ASM_DEFINE_TABLE(label, __rseq_table, 0x0, 0x0, start_ip, \ (post_commit_ip - start_ip), abort_ip)
or something, such that we can write:
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
tools/testing/selftests/rseq/rseq-x86.h line 88
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs)
And here we open start the rseq by storing the table entry pointer into the TLS thingy.
tools/testing/selftests/rseq/rseq-x86.h line 99
Quoted 3 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +"cmpq %[v], %[expect]\n\t" +"jnz 5f\n\t"
"jnz %l[cmpfail]\n\t"
was too complicated?
tools/testing/selftests/rseq/rseq-x86.h line 113
Quoted 12 lines
tools/testing/selftests/rseq/rseq-x86.h new file +12
-0+/* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv)
: [cpu_id] "r" (cpu), [current_cpu_id] "m" (__rseq_abi.cpu_id), [rseq_cs] "m" (__rseq_abi.rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
or something does read much better
tools/testing/selftests/rseq/rseq-x86.h line 129
Quoted 13 lines
tools/testing/selftests/rseq/rseq-x86.h new file +13
-0+ : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu)
so this thing does what now? It compares @v to @expectnot, when _not_ matching it will store @voffp into @v and load something..?
tools/testing/selftests/rseq/rseq-x86.h line 139
Quoted 7 lines
tools/testing/selftests/rseq/rseq-x86.h new file +7
-0+{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +"cmpq %[v], %[expectnot]\n\t" +"jz 5f\n\t"
So I would prefer "je" in this context, or rather:
je %l[cmpfail]
tools/testing/selftests/rseq/rseq-x86.h line 141
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"movq %[v], %%rax\n\t"
loads @v in A
But it could already have changed since the previous load from cmp, no? Would it not make sense to put this load before the cmp and use A instead?
tools/testing/selftests/rseq/rseq-x86.h line 142
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"movq %%rax, %[load]\n\t"
stores A in @load
tools/testing/selftests/rseq/rseq-x86.h line 143
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"addq %[voffp], %%rax\n\t"
adds @off to A
tools/testing/selftests/rseq/rseq-x86.h line 144
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"movq (%%rax), %%rax\n\t"
loads (A) in A
tools/testing/selftests/rseq/rseq-x86.h line 129
Quoted 2 lines
tools/testing/selftests/rseq/rseq-x86.h new file +2
-0+/* final store */ +"movq %%rax, %[v]\n\t"
stores A in @v
So the whole thing loads @v into @load, adds and offset, dereferences and adds that back in @v, provided @v doesn't match @expected.. whee.
Quoted 24 lines
tools/testing/selftests/rseq/rseq-x86.h new file +23
-0+"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expectnot]"r"(expectnot), + [voffp]"er"(voffp), + [load]"m"(*load) + : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ; +cmpfail: + return 1; +} +#elif __i386__ +
tools/testing/selftests/rseq/rseq-x86.h line 411
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h new file +10
-0+/* +* Support older 32-bit architectures that do not implement fence +* instructions. +*/ +#define rseq_smp_mb () \ + __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory") +#define rseq_smp_rmb () \ + __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory") +#define rseq_smp_wmb () \ + __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory")
Oh shiny, you're supporting that OOSTORE and PPRO_FENCE nonsense?
Going by commit:
09df7c4c8097 ("x86: Remove CONFIG_X86_OOSTORE")
That smp_wmb() one was an 'optimization' (forced store buffer flush) but not a correctness thing. And we dropped that stuff from the kernel a _long_ time ago.
Ideally we'd kill that PPRO_FENCE crap too.
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 45 lines · ----- On Nov 22, 2017, at 4:48 PM, Peter Zijlstra [email protected] wrote:
Quoted 33 lines · Mathieu Desnoyers
tools/testing/selftests/rseq/rseq-x86.h new file +7
-0@@ -0,0 +1,898 @@ +/* +* rseq-x86.h +* +* (C) Copyright 2016 - Mathieu Desnoyers <[email protected]> +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */tools/testing/selftests/rseq/rseq-x86.h line 31
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h new file +8
-0+ +#include <stdint.h> + +#define RSEQ_SIG 0x53053053 + +#ifdef __x86_64__ + +#define rseq_smp_mb () __asm__ __volatile__ ("mfence" : : : "memory")See commit:
Quoted 1 line
450cbdd0125c("locking/x86: Use LOCK ADD forsmp_mb()instead of MFENCE")
OK, will use:
#define rseq_smp_mb() \
__asm__ __volatile__ ("lock; addl $0,-128(%%rsp)" ::: "memory", "cc")as done in tools/virtio/ringtest/main.h
Quoted 23 lines
Quoted 23 lines
tools/testing/selftests/rseq/rseq-x86.h new file +22
-0+#define rseq_smp_rmb () barrier() +#define rseq_smp_wmb () barrier() + +#define rseq_smp_load_acquire (p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + barrier(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep () rseq_smp_rmb() + +#define rseq_smp_store_release (p, v) \ +do { \ + barrier(); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + +#define RSEQ_ASM_DEFINE_TABLE (label, section, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".pushsection " __rseq_str(section) ", \"aw\"\n\t" \ + ".balign 32\n\t" \
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h
Quoted 5 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+ __rseq_str(label) ":\n\t" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + ".quad " __rseq_str(start_ip) ", " __rseq_str(post_commit_offset) ", "
__rseq_str(abort_ip)"\n\t" \tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+".popsection\n\t" OK, so this creates table entry, but why is @section an argument, AFAICT its _always_ the same thing, no?
I agree that section names don't need to be passed as arguments, since it is not useful information to understand the flow of asm using those macros. Will remove.
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h line 63
Quoted 6 lines
tools/testing/selftests/rseq/rseq-x86.h new file +5
-0+#define RSEQ_ASM_STORE_RSEQ_CS (label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "leaq " __rseq_str(cs_label) "(%%rip), %%rax\n\t" \ + "movq %%rax, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t"And this sets the TLS variable to point to the table entry from the previous macro, no? But again @rseq_cs seems to always be the very same, why is that an argument?
I don't want to hide anything within macros that would prevent someone looking that the overall assembly to understand the flow.
So all labels and references to named input operands, are passed as arguments to those macros.
Quoted 7 lines
tools/testing/selftests/rseq/rseq-x86.h line 68
Quoted 5 lines
tools/testing/selftests/rseq/rseq-x86.h new file +4
-0+#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "cmpl %[" __rseq_str(cpu_id) "], %[" __rseq_str(current_cpu_id) "]\n\t" \ + "jnz " __rseq_str(label) "\n\t"more things that are always the same it seems..
See this as passing arguments to functions. It makes the overall assembly using those macros easier to read.
Quoted 12 lines
tools/testing/selftests/rseq/rseq-x86.h line 78
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h new file +9
-0+#define RSEQ_ASM_DEFINE_ABORT (label, section, sig, teardown, abort_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ +/* Disassembler-friendly signature: nopl <sig>(%rip). */ \ +".byte 0x0f, 0x1f, 0x05\n\t" \ + ".long " __rseq_str(sig) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "jmp %l[" __rseq_str(abort_label) "]\n\t" \ +".popsection\n\t" @section and @sig seem to always be the same...
Good point for the @section, will move it into the macro because it is not useful to understand the flow.
Same for @sig. Will move into the macro, given that it's not useful to understand the flow.
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h line 85
Quoted 7 lines
tools/testing/selftests/rseq/rseq-x86.h new file +6
-0+#define RSEQ_ASM_DEFINE_CMPFAIL (label, section, teardown, cmpfail_label) \ + ".pushsection " __rseq_str(section) ", \"ax\"\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \ + ".popsection\n\t"Somewhat failing to see the point of this macro, it seems to just obfuscate the normal failure path.
It's needed to hold the "teardown" code needed on error, for the memcpy loops. You're right that I don't need it for all the other cases, and I can directly jump to the cmpfail_label. Will do.
Quoted 7 lines
tools/testing/selftests/rseq/rseq-x86.h
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu)I find this a very confusing name for what is essentially compare-and-exchange or compare-and-swap, no?
A compare-and-exchange will load the original value and return it. With rseq, it appears that all we need is to compare a value, and store the new value if the comparison succeeded. We seldom care about returning the old value, so we might as well skip that load. But then it's _not_ a cmpxchg, because it does not return the prior value.
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)So we set up the section, but unreadably so... reducing the number of arguments would help a lot.
Rename the current one to
__RSEQ_ASM_DEFINE_TABLE()and then use:
Quoted 8 lines
#define RSEQ_ASM_DEFINE_TABLE(label, start_ip, post_commit_ip, abort_ip) \ __RSEQ_ASM_DEFINE_TABLE(label, __rseq_table, 0x0, 0x0, start_ip, \ (post_commit_ip - start_ip), abort_ip)or something, such that we can write:
RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */
Good point, it removes @version, @flags, and the 2f-1f calculation from the asm, which distract from the flow.
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs)And here we open start the rseq by storing the table entry pointer into the TLS thingy.
I'll add a comment if that's what you're pointing to.
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h line 99
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +"cmpq %[v], %[expect]\n\t" +"jnz 5f\n\t" "jnz %l[cmpfail]\n\t"was too complicated?
Good point. It's a leftover from the prior iterations where I needed to clear the rseq_cs field before exiting from the critical section. Now that the kernel does it lazily, I don't need a "teardown" code in those cases, and we can directly jump to the cmpfail label. Will do.
Quoted 21 lines
tools/testing/selftests/rseq/rseq-x86.h line 113
Quoted 13 lines
tools/testing/selftests/rseq/rseq-x86.h new file +12
-0+/* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv): [cpu_id] "r" (cpu), [current_cpu_id] "m" (__rseq_abi.cpu_id), [rseq_cs] "m" (__rseq_abi.rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
Quoted 1 line
or something does read much better
done.
Quoted 17 lines
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+ : "memory", "cc", "rax" + : abort, cmpfail + );tools/testing/selftests/rseq/rseq-x86.h line 129
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h new file +10
-0+return 0 ; +abort: +return -1 ; +cmpfail: +return 1 ; +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, +off_t voffp,intptr_t *load,int cpu)so this thing does what now? It compares @v to @expectnot, when _not_ matching it will store @voffp into @v and load something..?
Not quite. I'll add this comment:
/*
* Compare @v against @expectnot. When it does _not_ match, load @v
* into @load, and store the content of *@v + voffp into @v. */
Quoted 11 lines
tools/testing/selftests/rseq/rseq-x86.h line 139
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h new file +7
-0+{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +"cmpq %[v], %[expectnot]\n\t" +"jz 5f\n\t" So I would prefer "je" in this context, or rather:
Quoted 1 line
je %l[cmpfail]
ok
Quoted 6 lines
tools/testing/selftests/rseq/rseq-x86.h line 141
Quoted 2 lines
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"movq %[v], %%rax\n\t" loads @v in A
But it could already have changed since the previous load from cmp, no?
No, given that it should only be touched by rseq or cpu_opv, which don't allow concurrent accesses on the same per-cpu data.
Quoted 2 lines
Would it not make sense to put this load before the cmp and use A instead?
Sure, I could do the load once, and compare against a register instead. It's not needed for correctness, but seems cleaner nevertheless. I'll need to use some other register than rax though, because I don't want to clash with the RSEQ_INJECT_ASM() delay-injection loops, which are using eax. Perhaps rbx then.
Quoted 13 lines
tools/testing/selftests/rseq/rseq-x86.h line 142
Quoted 2 lines
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"movq %%rax, %[load]\n\t" stores A in @load
tools/testing/selftests/rseq/rseq-x86.h line 143
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"addq %[voffp], %%rax\n\t" adds @off to A
tools/testing/selftests/rseq/rseq-x86.h line 144
Quoted 1 line
tools/testing/selftests/rseq/rseq-x86.h new file +1
-0+"movq (%%rax), %%rax\n\t" loads (A) in A
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h
Quoted 2 lines
tools/testing/selftests/rseq/rseq-x86.h new file +2
-0+/* final store */ +"movq %%rax, %[v]\n\t" stores A in @v
So the whole thing loads @v into @load, adds and offset, dereferences and adds that back in @v, provided @v doesn't match @expected.. whee.
Yes, exactly! ;)
Quoted 23 lines
Quoted 22 lines
tools/testing/selftests/rseq/rseq-x86.h new file +21
-0+"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), +/* final store input */ + [v]"m"(*v), + [expectnot]"r"(expectnot), + [voffp]"er"(voffp), + [load]"m"(*load) + : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ; +cmpfail: +return 1 ; +}
Quoted 14 lines
tools/testing/selftests/rseq/rseq-x86.h line 411
Quoted 12 lines
tools/testing/selftests/rseq/rseq-x86.h new file +12
-0+#elif __i386__ + +/* +* Support older 32-bit architectures that do not implement fence +* instructions. +*/ +#define rseq_smp_mb () \ + __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory") +#define rseq_smp_rmb () \ + __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory") +#define rseq_smp_wmb () \ + __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory")Oh shiny, you're supporting that OOSTORE and
PPRO_FENCEnonsense?
Well, what's the point in supporting only a subset of the x86 architecture ? ;)
Quoted 8 lines
Going by commit:
09df7c4c8097("x86: RemoveCONFIG_X86_OOSTORE")That
smp_wmb()one was an 'optimization' (forced store buffer flush) but not a correctness thing. And we dropped that stuff from the kernel a _long_ time ago.
Perhaps from a kernel code perspective it's not a correctness thing, but this is a correctness issue for user-space code.
I have the following comments in liburcu near the x86-32 barriers:
* We leave smp_rmb/smp_wmb as full barriers for processors that do not have
* fence instructions.
*
* An empty cmm_smp_rmb() may not be enough on old PentiumPro multiprocessor
* systems, due to an erratum. The Linux kernel says that "Even distro
* kernels should think twice before enabling this", but for now let's
* be conservative and leave the full barrier on 32-bit processors. Also,
* IDT WinChip supports weak store ordering, and the kernel may enable it
* under our feet; cmm_smp_wmb() ceases to be a nop for these processors.
So it looks like from a user-space perspective, having the smp_wmb() would be a correctness issue for older kernels on IDT WinChip processors.
Now you could argue that the recent kernel that will implement the "rseq" system call will _never_ enable this IDT WinChip OOSTORE stuff. So we would end up relying on success of the rseq cpu_id field comparison to ensure rseq is indeed available, and in those cases the OOSTORE-aware smp_wmb is not needed on x86-32.
Am I understanding this correctly ?
In that case, we should really document that those rseq_smp_wmb() are only valid when running on kernels that provide the rseq system call.
Quoted 1 line
Ideally we'd kill that
PPRO_FENCEcrap too.
AFAIU, this one is part of the PPRO architecture, not really enabled explicitly by the kernel, right ? I'm not sure what we can do about this other than politely asking everyone to throw away these old pieces of hardware... ;)
Thanks for the review!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
In reply to Mathieu DesnoyersQuoted 21 lines · Mathieu Desnoyers
tools/testing/selftests/rseq/basic_percpu_ops_test.c new file +17
-0+int percpu_list_push (struct percpu_list *list,struct percpu_list_node *node) +{ +intptr_t *targetptr, newval, expect; +int cpu, ret; + +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t )READ_ONCE (list->c[cpu].head); + newval = (intptr_t )node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret =rseq_cmpeqv_storev (targetptr, expect, newval, cpu); +if (likely (!ret)) +return cpu; +return cpu; +}
tools/testing/selftests/rseq/basic_percpu_ops_test.c
Quoted 10 lines
tools/testing/selftests/rseq/basic_percpu_ops_test.c new file +8
-0+static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) +RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id,4f )
So the actual C part of the RSEQ is subject to an ABA, right? We can get migrated to another CPU and back again without then failing here.
It used to be that this was caught by the sequence count, but that is now gone.
The thing that makes it work is the compare against @v:
Review
Quoted 2 lines
(quoted diff) +2
-0+ "cmpq %[v], %[expect]\n\t" + "jnz 5f\n\t"
That then ensures things are still as we observed them before (although this itself is also subject to ABA).
This means all RSEQ primitives that have a C part must have a cmp-and- form, but I suppose that was already pretty much the case anyway. I just don't remember seeing that spelled out anywhere. Then again, I've not yet read that manpage.
Quoted 21 lines
tools/testing/selftests/rseq/rseq-x86.h new file +21
-0+/* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ; +cmpfail: +return 1 ; +}
Peter Zijlstra
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Mathieu Desnoyers
tools/testing/selftests/rseq/rseq-ppc.h new file +7
-0+static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs)
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h line 99
Quoted 4 lines
tools/testing/selftests/rseq/rseq-x86.h new file +3
-0+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) +"cmpq %[v], %[expect]\n\t" +"jnz 5f\n\t"
Also, I'm confused between the abort and cmpfail cases.
In would expect the cpu_id compare to also result in cmpfail, that is, I would only expect the kernel to result in abort.
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/rseq/rseq-x86.h new file +10
-0+/* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v),
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h line 122
Quoted 8 lines
tools/testing/selftests/rseq/rseq-x86.h new file +8
-0+ [expect]"r"(expect), + [newv]"r"(newv) + : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ;
Which then would suggest this be -EINTR or something like that.
Quoted 3 lines
Quoted 3 lines
tools/testing/selftests/rseq/rseq-arm.h new file +3
-0+cmpfail: +return 1 ; +}
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
Quoted 12 lines · ----- On Nov 23, 2017, at 3:57 AM, Peter Zijlstra [email protected] wrote:
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers
tools/testing/selftests/rseq/rseq-ppc.h new file +8
-0+static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) +RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id,4f )
Quoted 7 lines
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ "cmpq %[v], %[expect]\n\t" + "jnz 5f\n\t"Also, I'm confused between the abort and cmpfail cases.
In would expect the
cpu_idcompare to also result in cmpfail, that is, I would only expect the kernel to result in abort.
Let's take the per-cpu spinlock as an example to explain why we need the "compare fail" and "cpu_id compare fail" to return different values.
Getting this lock involves doing:
cpu = rseq_cpu_start();
ret = rseq_cmpeqv_storev(&lock->c[cpu].v, 0, 1, cpu);
Now based on the "ret" value:
if ret == 0, it means that @v was indeed 0, and that rseq executed the commit (stored 1).
if ret > 0, it means the comparison of @v against 0 failed, which means the lock was already held. We therefore need to postpone and retry later. A "try_lock" operation would return that the lock is currently busy.
if ret < 0, then we have either been aborted by the kernel, or the comparison of @cpu against cpu_id failed. If we think about it, having @cpu != cpu_id will happen if we are migrated before we enter the rseq critical section, which is pretty similar to being aborted by the kernel within the critical section. So I don't see any reason for making the branch target of the cpu_id comparison anything else than the abort_ip. In that situation, the caller needs to either re-try with an updated @cpu value (except for multi-part algorithms e.g. reserve+commit, which don't allow changing the @cpu number on commit), or use cpu_opv to perform the operation.
Note that another cause why the @cpu == cpu_id test may fail is if rseq is not registered for the current thread. Again, just branching to the abort_ip and letting the caller fallback to cpu_opv solves this.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
tools/testing/selftests/rseq/rseq-x86.h new file +12
-0+/* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv)
Quoted 8 lines
Quoted 6 lines
tools/testing/selftests/rseq/rseq-x86.h line 122
Quoted 6 lines
tools/testing/selftests/rseq/rseq-x86.h new file +6
-0+ : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ;Which then would suggest this be
-EINTRor something like that.
I'm not so sure returning kernel error codes is the expected practice for user-space libraries.
Thoughts ?
Thanks!
Mathieu
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/rseq/rseq-arm.h new file +3
-0+cmpfail: +return 1 ; +}
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH for 4.15 v3 15/22] rseq: selftests: Provide self-tests
In reply to Peter ZijlstraQuoted 31 lines · ----- On Nov 23, 2017, at 3:55 AM, Peter Zijlstra [email protected] wrote:
Quoted 29 lines · Mathieu Desnoyers
tools/testing/selftests/rseq/basic_percpu_ops_test.c new file +24
-0+int percpu_list_push (struct percpu_list *list,struct percpu_list_node *node) +{ +intptr_t *targetptr, newval, expect; +int cpu, ret; + +/* Try fast path. */ + cpu =rseq_cpu_start (); +/* Load list->c[cpu].head with single-copy atomicity. */ + expect = (intptr_t )READ_ONCE (list->c[cpu].head); + newval = (intptr_t )node; + targetptr = (intptr_t *)&list->c[cpu].head; + node->next = (struct percpu_list_node *)expect; + ret =rseq_cmpeqv_storev (targetptr, expect, newval, cpu); +if (likely (!ret)) +return cpu; +return cpu; +} +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv, +int cpu) +{ + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, __rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f) +RSEQ_ASM_STORE_RSEQ_CS (1 ,3b , rseq_cs)
Quoted 4 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)So the actual C part of the RSEQ is subject to an ABA, right? We can get migrated to another CPU and back again without then failing here.
Yes, that's correct. All algorithms preparing something in C and then using a compare-and-other-stuff sequence need to ensure they do not have ABA situations. For instance, a list push does not care if the list head is reclaimed and re-inserted concurrently, because none of the preparation steps in C involve the head next pointer.
Quoted 3 lines
It used to be that this was caught by the sequence count, but that is now gone.
The sequence count introduced other weirdness: although it would catch those migration cases, it is a sequence read-lock, which means the C code "protected" by this sequence read-lock needed to be extremely careful about not accessing reclaimed memory.
The sequence lock ensures consistency of the data when they comparison matches, but it does not protect against other side-effects.
So removing this sequence lock is actually a good thing: it removes any expectation that users may have about that sequence counter being anything stronger than a read seqlock.
Quoted 8 lines
The thing that makes it work is the compare against @v:
Review
Quoted 2 lines
(quoted diff) +2
-0+ "cmpq %[v], %[expect]\n\t" + "jnz 5f\n\t"That then ensures things are still as we observed them before (although this itself is also subject to ABA).
Yes.
Quoted 5 lines
This means all RSEQ primitives that have a C part must have a cmp-and- form, but I suppose that was already pretty much the case anyway. I just don't remember seeing that spelled out anywhere. Then again, I've not yet read that manpage.
Yes, pretty much. The only primitives that don't have the compare are things like "rseq_addv()", which does not have much in the C part (it's just incrementing a counter).
I did not state anything like "typical rseq c.s. do a compare and other stuff" in rseq(2), given that the role of this man page, AFAIU, is to explain how to interact with the kernel system call, and not really a document about user-space implementation guide lines.
But let me know if I should expand it with a user-space sequence implementation guide lines, which would include notes about being careful about ABA. I'm not sure it belongs there though.
Thanks!
Mathieu
Quoted 22 lines
Quoted 22 lines
tools/testing/selftests/rseq/rseq-x86.h new file +21
-0+/* final store */ +"movq %[newv], %[v]\n\t" +"2:\n\t" + RSEQ_ASM_DEFINE_ABORT(4, __rseq_failure, RSEQ_SIG, "", abort) +RSEQ_ASM_DEFINE_CMPFAIL (5 , __rseq_failure,"" , cmpfail) + : /* gcc asm goto does not allow outputs */ + : [cpu_id]"r"(cpu), + [current_cpu_id]"m"(__rseq_abi.cpu_id), + [rseq_cs]"m"(__rseq_abi.rseq_cs), + [v]"m"(*v), + [expect]"r"(expect), + [newv]"r"(newv) + : "memory", "cc", "rax" + : abort, cmpfail + ); +return 0 ; +abort: +return -1 ; +cmpfail: +return 1 ; +}
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRFC PATCH v12 16/22 rseq: selftests: arm: workaround gcc asm size guess
In reply to Mathieu DesnoyersFixes assembler errors:/tmp/cceKwI9a.s: Assembler messages:/tmp/cceKwI9a.s:849: Error: co-processor offset out of range
with gcc prior to gcc-7. This can trigger if multiple rseq inline asm are used within the same function.
My best guess on the cause of this issue is that gcc has a hard time figuring out the actual size of the inline asm, and therefore does not compute the offsets at which literal values can be placed from the program counter accurately.
--- tools/testing/selftests/rseq/rseq-arm.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
tools/testing/selftests/rseq/rseq-arm.h +33 -0
@@ -79,12 +79,15 @@ do { \ teardown \ "b %l[" __rseq_str(cmpfail_label) "]\n\t" +#define rseq_workaround_gcc_asm_size_guess () __asm__ __volatile__("") +static inline __attribute__((always_inline))int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv,int cpu) { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)
@@ -115,11 +118,14 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
@@ -129,6 +135,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)
@@ -164,11 +171,14 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
@@ -177,6 +187,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)
@@ -203,8 +214,10 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) RSEQ_INJECT_CLOBBER : abort ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; }
@@ -216,6 +229,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)
@@ -253,11 +267,14 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
@@ -268,6 +285,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)
@@ -306,11 +324,14 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
@@ -321,6 +342,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)
@@ -359,11 +381,14 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
@@ -376,6 +401,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)"str %[src], %[rseq_scratch0]\n\t"
@@ -442,11 +468,14 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
@@ -459,6 +488,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)"str %[src], %[rseq_scratch0]\n\t"
@@ -526,10 +556,13 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
Signature · 1 line
2.11.0
Re: [RFC PATCH for 4.15 16/22] rseq: selftests: arm: workaround gcc asm size guess
Quoted 224 lines · Mathieu Desnoyers
Fixes assembler errors:
/tmp/cceKwI9a.s: Assembler messages:/tmp/cceKwI9a.s:849: Error: co-processor offset out of rangewith gcc prior to gcc-7. This can trigger if multiple rseq inline asm are used within the same function.
My best guess on the cause of this issue is that gcc has a hard time figuring out the actual size of the inline asm, and therefore does not compute the offsets at which literal values can be placed from the program counter accurately.
CC "Paul E. McKenney" <[email protected]>CC Peter Zijlstra <[email protected]>CC Paul Turner <[email protected]>CC Thomas Gleixner <[email protected]>CC Andrew Hunter <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <andi-Vw/[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <torvalds-de/[email protected]>CC Andrew Morton <akpm-de/[email protected]>CC Russell King <linux-lFZ/[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Michael Kerrisk <[email protected]>CC Boqun Feng <[email protected]>CC Florian Weimer <[email protected]>CC Shuah Khan <[email protected]>--- tools/testing/selftests/rseq/rseq-arm.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)tools/testing/selftests/rseq/rseq-arm.h +30
-0@@ -79,12 +79,15 @@ do { \ teardown \ "b %l[" __rseq_str(cmpfail_label) "]\n\t" +#define rseq_workaround_gcc_asm_size_guess () __asm__ __volatile__("") +static inline __attribute__((always_inline))int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv,int cpu) { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)@@ -115,11 +118,14 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }@@ -129,6 +135,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)@@ -164,11 +171,14 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }@@ -177,6 +187,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)@@ -203,8 +214,10 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) RSEQ_INJECT_CLOBBER : abort ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; }@@ -216,6 +229,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)@@ -253,11 +267,14 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }@@ -268,6 +285,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)@@ -306,11 +324,14 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }@@ -321,6 +342,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)RSEQ_ASM_STORE_RSEQ_CS (1 ,3f , rseq_cs)@@ -359,11 +381,14 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }@@ -376,6 +401,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)"str %[src], %[rseq_scratch0]\n\t" @@ -442,11 +468,14 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail ); +rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }@@ -459,6 +488,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) +rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(__rseq_table, 0x0, 0x0, 1f, 2f-1f, 4f)"str %[src], %[rseq_scratch0]\n\t" @@ -526,10 +556,13 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_CLOBBER : abort, cmpfail );
tools/testing/selftests/rseq/rseq-arm.h int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, · line 568
Quoted 10 lines
tools/testing/selftests/rseq/rseq-arm.h +3
-0+rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: +rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_workaround_gcc_asm_size_guess ();return 1 ; }
Looks fine to me. For this patch.
I have comments on other patches in this series.
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRFC PATCH v12 17/22 Fix: membarrier: add missing preempt off around smp_call_function_many
In reply to Mathieu Desnoyerssmp_call_function_many requires disabling preemption around the call.
--- kernel/sched/membarrier.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/membarrier.c +2 -0
@@ -89,7 +89,9 @@ static int membarrier_private_expedited(void)rcu_read_unlock (); }if (!fallback) { +preempt_disable ();smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); +preempt_enable ();free_cpumask_var (tmpmask); }cpus_read_unlock ();
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 18/22 membarrier: selftest: Test private expedited cmd
In reply to Mathieu DesnoyersTest the new MEMBARRIER_CMD_PRIVATE_EXPEDITED and MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED commands.
Add checks expecting specific error values on system calls expected to fail.
---
Changes since v1:
- return result of ksft_exit_pass from main(), silencing compiler warning about missing return value.
--- .../testing/selftests/membarrier/membarrier_test.c | 111 ++++++++++++++++++--- 1 file changed, 95 insertions(+), 16 deletions(-)
tools/testing/selftests/membarrier/membarrier_test.c +95 -16
@@ -16,49 +16,119 @@ static int sys_membarrier(int cmd, int flags)static int test_membarrier_cmd_fail (void ) {int cmd =-1 , flags =0 ; +const char *test_name ="sys membarrier invalid command" ;if (sys_membarrier (cmd, flags) !=-1 ) {ksft_exit_fail_msg ( -"sys membarrier invalid command test: command = %d, flags = %d. Should fail, but passed\n" , - cmd, flags); +"%s test: command = %d, flags = %d. Should fail, but passed\n" , + test_name, cmd, flags); + } +if (errno !=EINVAL ) { +ksft_exit_fail_msg ( +"%s test: flags = %d. Should return (%d: \"%s\"), but returned (%d: \"%s\").\n" , + test_name, flags,EINVAL ,strerror (EINVAL ), + errno,strerror (errno)); }ksft_test_result_pass ( -"sys membarrier invalid command test: command = %d, flags = %d. Failed as expected\n" , - cmd, flags); +"%s test: command = %d, flags = %d, errno = %d. Failed as expected\n" , + test_name, cmd, flags, errno);return 0 ; }static int test_membarrier_flags_fail (void ) {int cmd =MEMBARRIER_CMD_QUERY , flags =1 ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_QUERY invalid flags" ;if (sys_membarrier (cmd, flags) !=-1 ) {ksft_exit_fail_msg ( -"sys membarrier MEMBARRIER_CMD_QUERY invalid flags test: flags = %d. Should fail, but passed\n" , - flags); +"%s test: flags = %d. Should fail, but passed\n" , + test_name, flags); + } +if (errno !=EINVAL ) { +ksft_exit_fail_msg ( +"%s test: flags = %d. Should return (%d: \"%s\"), but returned (%d: \"%s\").\n" , + test_name, flags,EINVAL ,strerror (EINVAL ), + errno,strerror (errno)); }ksft_test_result_pass ( -"sys membarrier MEMBARRIER_CMD_QUERY invalid flags test: flags = %d. Failed as expected\n" , - flags); +"%s test: flags = %d, errno = %d. Failed as expected\n" , + test_name, flags, errno);return 0 ; } -static int test_membarrier_success (void ) +static int test_membarrier_shared_success (void ) {int cmd =MEMBARRIER_CMD_SHARED , flags =0 ; -const char *test_name ="sys membarrier MEMBARRIER_CMD_SHARED\n" ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_SHARED" ; + +if (sys_membarrier (cmd, flags) !=0 ) { +ksft_exit_fail_msg ( +"%s test: flags = %d, errno = %d\n" , + test_name, flags, errno); + } + +ksft_test_result_pass ( +"%s test: flags = %d\n" , test_name, flags); +return 0 ; +} + +static int test_membarrier_private_expedited_fail (void ) +{ +int cmd =MEMBARRIER_CMD_PRIVATE_EXPEDITED , flags =0 ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_PRIVATE_EXPEDITED not registered failure" ; + +if (sys_membarrier (cmd, flags) !=-1 ) { +ksft_exit_fail_msg ( +"%s test: flags = %d. Should fail, but passed\n" , + test_name, flags); + } +if (errno !=EPERM ) { +ksft_exit_fail_msg ( +"%s test: flags = %d. Should return (%d: \"%s\"), but returned (%d: \"%s\").\n" , + test_name, flags,EPERM ,strerror (EPERM ), + errno,strerror (errno)); + } + +ksft_test_result_pass ( +"%s test: flags = %d, errno = %d\n" , + test_name, flags, errno); +return 0 ; +} + +static int test_membarrier_register_private_expedited_success (void ) +{ +int cmd =MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED , flags =0 ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED" ;if (sys_membarrier (cmd, flags) !=0 ) {ksft_exit_fail_msg ( -"sys membarrier MEMBARRIER_CMD_SHARED test: flags = %d\n" , - flags); +"%s test: flags = %d, errno = %d\n" , + test_name, flags, errno); }ksft_test_result_pass ( -"sys membarrier MEMBARRIER_CMD_SHARED test: flags = %d\n" , - flags); +"%s test: flags = %d\n" , + test_name, flags); +return 0 ; +} + +static int test_membarrier_private_expedited_success (void ) +{ +int cmd =MEMBARRIER_CMD_PRIVATE_EXPEDITED , flags =0 ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_PRIVATE_EXPEDITED" ; + +if (sys_membarrier (cmd, flags) !=0 ) { +ksft_exit_fail_msg ( +"%s test: flags = %d, errno = %d\n" , + test_name, flags, errno); + } + +ksft_test_result_pass ( +"%s test: flags = %d\n" , + test_name, flags);return 0 ; }
@@ -72,7 +142,16 @@ static int test_membarrier(void) status =test_membarrier_flags_fail ();if (status)return status; - status =test_membarrier_success (); + status =test_membarrier_shared_success (); +if (status) +return status; + status =test_membarrier_private_expedited_fail (); +if (status) +return status; + status =test_membarrier_register_private_expedited_success (); +if (status) +return status; + status =test_membarrier_private_expedited_success ();if (status)return status;return 0 ;
@@ -108,5 +187,5 @@ int main(int argc, char **argv)test_membarrier_query ();test_membarrier (); -ksft_exit_pass (); +return ksft_exit_pass (); }
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v7 19/22 powerpc: membarrier: Skip memory barrier in switch_mm()
In reply to Mathieu DesnoyersAllow PowerPC to skip the full memory barrier in switch_mm(), and only issue the barrier when scheduling into a task belonging to a process that has registered to use expedited private.
Threads targeting the same VM but which belong to different thread groups is a tricky case. It has a few consequences:
It turns out that we cannot rely on get_nr_threads(p) to count the number of threads using a VM. We can use (atomic_read(&mm->mm_users) == 1 && get_nr_threads(p) == 1) instead to skip the synchronize_sched() for cases where the VM only has a single user, and that user only has a single thread.
It also turns out that we cannot use for_each_thread() to set thread flags in all threads using a VM, as it only iterates on the thread group.
Therefore, test the membarrier state variable directly rather than relying on thread flags. This means membarrier_register_private_expedited() needs to set the MEMBARRIER_STATE_PRIVATE_EXPEDITED flag, issue synchronize_sched(), and only then set MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY which allows private expedited membarrier commands to succeed. membarrier_arch_switch_mm() now tests for the MEMBARRIER_STATE_PRIVATE_EXPEDITED flag.
Changes since v1:
- Use test_ti_thread_flag(next, ...) instead of test_thread_flag() in powerpc membarrier_arch_sched_in(), given that we want to specifically check the next thread state.
- Add missing ARCH_HAS_MEMBARRIER_HOOKS in Kconfig.
- Use task_thread_info() to pass thread_info from task to *_ti_thread_flag().
Changes since v2:
- Move membarrier_arch_sched_in() call to finish_task_switch().
- Check for NULL t->mm in membarrier_arch_fork().
- Use membarrier_sched_in() in generic code, which invokes the arch-specific membarrier_arch_sched_in(). This fixes allnoconfig build on PowerPC.
- Move asm/membarrier.h include under CONFIG_MEMBARRIER, fixing allnoconfig build on PowerPC.
- Build and runtime tested on PowerPC.
Changes since v3:
- Simply rely on copy_mm() to copy the membarrier_private_expedited mm field on fork.
- powerpc: test thread flag instead of reading membarrier_private_expedited in membarrier_arch_fork().
- powerpc: skip memory barrier in membarrier_arch_sched_in() if coming from kernel thread, since mmdrop() implies a full barrier.
- Set membarrier_private_expedited to 1 only after arch registration code, thus eliminating a race where concurrent commands could succeed when they should fail if issued concurrently with process registration.
- Use READ_ONCE() for membarrier_private_expedited field access in membarrier_private_expedited. Matches WRITE_ONCE() performed in process registration.
Changes since v4:
- Move powerpc hook from sched_in() to switch_mm(), based on feedback from Nicholas Piggin.
Changes since v5:
- Rebase on v4.14-rc6.
- Fold "Fix: membarrier: Handle CLONE_VM + !CLONE_THREAD correctly on powerpc (v2)"
Changes since v6:
- Rename MEMBARRIER_STATE_SWITCH_MM to MEMBARRIER_STATE_PRIVATE_EXPEDITED.
--- MAINTAINERS | 1 + arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/membarrier.h | 25 +++++++++++++++++++++++++ arch/powerpc/mm/mmu_context.c | 7 +++++++ include/linux/sched/mm.h | 12 +++++++++++- init/Kconfig | 3 +++ kernel/sched/core.c | 10 ---------- kernel/sched/membarrier.c | 9 +++++++++ 8 files changed, 57 insertions(+), 11 deletions(-) create mode 100644 arch/powerpc/include/asm/membarrier.h
MAINTAINERS +1 -0
@@ -8938,6 +8938,7 @@ L: [email protected] S: Supported F: kernel/sched/membarrier.c F: include/uapi/linux/membarrier.h +F: arch/powerpc/include/asm/membarrier.h MEMORY MANAGEMENT L: [email protected]
arch/powerpc/Kconfig +1 -0
@@ -140,6 +140,7 @@ config PPC select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_PMEM_API if PPC64 + select ARCH_HAS_MEMBARRIER_HOOKS select ARCH_HAS_SCALED_CPUTIME if VIRT_CPU_ACCOUNTING_NATIVE select ARCH_HAS_SG_CHAIN select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST
arch/powerpc/include/asm/membarrier.h new file +25 -0
@@ -0,0 +1,25 @@ +#ifndef _ASM_POWERPC_MEMBARRIER_H +#define _ASM_POWERPC_MEMBARRIER_H + +static inline void membarrier_arch_switch_mm (struct mm_struct *prev, +struct mm_struct *next,struct task_struct *tsk) +{ +/* +* Only need the full barrier when switching between processes. +* Barrier when switching from kernel to userspace is not +* required here, given that it is implied by mmdrop(). Barrier +* when switching from userspace to kernel is not needed after +* store to rq->curr. +*/ +if (likely (!(atomic_read (&next->membarrier_state) + &MEMBARRIER_STATE_PRIVATE_EXPEDITED ) || !prev)) +return ; + +/* +* The membarrier system call requires a full memory barrier +* after storing to rq->curr, before going back to user-space. +*/ +smp_mb (); +} + +#endif /* _ASM_POWERPC_MEMBARRIER_H */
arch/powerpc/mm/mmu_context.c +7 -0
@@ -12,6 +12,7 @@#include <linux/mm.h> #include <linux/cpu.h> +#include <linux/sched/mm.h> #include <asm/mmu_context.h>
@@ -58,6 +59,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,* * On the read side the barrier is in pte_xchg(), which orders * the store to the PTE vs the load of mm_cpumask. +* +* This full barrier is needed by membarrier when switching +* between processes after store to rq->curr, before user-space +* memory accesses. */ smp_mb ();
@@ -80,6 +85,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,if (new_on_cpu)radix_kvm_prefetch_workaround (next); +else +membarrier_arch_switch_mm (prev, next, tsk);/* * The actual HW switching method differs between the various
include/linux/sched/mm.h +11 -1
@@ -215,14 +215,24 @@ static inline void memalloc_noreclaim_restore(unsigned int flags)#ifdef CONFIG_MEMBARRIER enum {MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY = (1U <<0 ), -MEMBARRIER_STATE_SWITCH_MM = (1U <<1 ), +MEMBARRIER_STATE_PRIVATE_EXPEDITED = (1U <<1 ), }; +#ifdef CONFIG_ARCH_HAS_MEMBARRIER_HOOKS +#include <asm/membarrier.h> +#endif +static inline void membarrier_execve (struct task_struct *t) {atomic_set (&t->mm->membarrier_state,0 ); }#else +#ifdef CONFIG_ARCH_HAS_MEMBARRIER_HOOKS +static inline void membarrier_arch_switch_mm (struct mm_struct *prev, +struct mm_struct *next,struct task_struct *tsk) +{ +} +#endif static inline void membarrier_execve (struct task_struct *t) { }
init/Kconfig +3 -0
@@ -1400,6 +1400,9 @@ config USERFAULTFD Enable the userfaultfd() system call that allows to intercept and handle page faults in userland. +config ARCH_HAS_MEMBARRIER_HOOKS + bool + config RSEQ bool "Enable rseq() system call" if EXPERT default y
kernel/sched/core.c +0 -10
@@ -2693,16 +2693,6 @@ static struct rq *finish_task_switch(struct task_struct *prev) prev_state = prev->state;vtime_task_switch (prev);perf_event_task_sched_in (prev, current); -/* -* The membarrier system call requires a full memory barrier -* after storing to rq->curr, before going back to user-space. -* -* TODO: This smp_mb__after_unlock_lock can go away if PPC end -* up adding a full barrier to switch_mm(), or we should figure -* out if a smp_mb__after_unlock_lock is really the proper API -* to use. -*/ -smp_mb__after_unlock_lock ();finish_lock_switch (rq, prev);finish_arch_post_lock_switch ();
kernel/sched/membarrier.c +9 -0
@@ -118,6 +118,15 @@ static void membarrier_register_private_expedited(void)if (atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY )return ; +atomic_or (MEMBARRIER_STATE_PRIVATE_EXPEDITED , + &mm->membarrier_state); +if (!(atomic_read (&mm->mm_users) ==1 &&get_nr_threads (p) ==1 )) { +/* +* Ensure all future scheduler executions will observe the +* new thread flag state for this process. +*/ +synchronize_sched (); + }atomic_or (MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY , &mm->membarrier_state); }
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v5 20/22 membarrier: Document scheduler barrier requirements
In reply to Mathieu DesnoyersDocument the membarrier requirement on having a full memory barrier in __schedule() after coming from user-space, before storing to rq->curr. It is provided by smp_mb__after_spinlock() in __schedule().
Document that membarrier requires a full barrier on transition from kernel thread to userspace thread. We currently have an implicit barrier from atomic_dec_and_test() in mmdrop() that ensures this.
The x86 switch_mm_irqs_off() full barrier is currently provided by many cpumask update operations as well as write_cr3(). Document that write_cr3() provides this barrier.
Changes since v1:
- Update comments to match reality for code paths which are after storing to rq->curr, before returning to user-space, based on feedback from Andrea Parri.
Changes since v2:
- Update changelog (smp_mb__before_spinlock -> smp_mb__after_spinlock). Based on feedback from Andrea Parri.
Changes since v3:
- Clarify comments following feeback from Peter Zijlstra. Changes since v4:
- Update comment regarding powerpc barrier.
--- arch/x86/mm/tlb.c | 5 +++++ include/linux/sched/mm.h | 5 +++++ kernel/sched/core.c | 37 ++++++++++++++++++++++++++----------- 3 files changed, 36 insertions(+), 11 deletions(-)
arch/x86/mm/tlb.c +5 -0
@@ -146,6 +146,11 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,#endif this_cpu_write (cpu_tlbstate.is_lazy, false); +/* +* The membarrier system call requires a full memory barrier +* before returning to user-space, after storing to rq->curr. +* Writing to CR3 provides that full memory barrier. +*/ if (real_prev == next) {VM_WARN_ON (this_cpu_read (cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id);
include/linux/sched/mm.h +5 -0
@@ -39,6 +39,11 @@ static inline void mmgrab(struct mm_struct *mm)extern void __mmdrop (struct mm_struct *);static inline void mmdrop (struct mm_struct *mm) { +/* +* The implicit full barrier implied by atomic_dec_and_test is +* required by the membarrier system call before returning to +* user-space, after storing to rq->curr. +*/ if (unlikely (atomic_dec_and_test (&mm->mm_count)))__mmdrop (mm); }
kernel/sched/core.c +26 -11
@@ -2697,6 +2697,12 @@ static struct rq *finish_task_switch(struct task_struct *prev)finish_arch_post_lock_switch ();fire_sched_in_preempt_notifiers (current); +/* +* When transitioning from a kernel thread to a userspace +* thread, mmdrop()'s implicit full barrier is required by the +* membarrier system call, because the current active_mm can +* become the current mm without going through switch_mm(). +*/ if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) {
@@ -2802,6 +2808,13 @@ context_switch(struct rq *rq, struct task_struct *prev,*/ arch_start_context_switch (prev); +/* +* If mm is non-NULL, we pass through switch_mm(). If mm is +* NULL, we will pass through mmdrop() in finish_task_switch(). +* Both of these contain the full memory barrier required by +* membarrier after storing to rq->curr, before returning to +* user-space. +*/ if (!mm) { next->active_mm = oldmm;mmgrab (oldmm);
@@ -3338,6 +3351,9 @@ static void __sched notrace __schedule(bool preempt)* Make sure that signal_pending_state()->signal_pending() below * can't be reordered with __set_current_state(TASK_INTERRUPTIBLE) * done by the caller to avoid the race with signal_wake_up(). +* +* The membarrier system call requires a full memory barrier +* after coming from user-space, before storing to rq->curr. */ rq_lock (rq, &rf);smp_mb__after_spinlock ();
@@ -3386,17 +3402,16 @@ static void __sched notrace __schedule(bool preempt)/* * The membarrier system call requires each architecture * to have a full memory barrier after updating -* rq->curr, before returning to user-space. For TSO -* (e.g. x86), the architecture must provide its own -* barrier in switch_mm(). For weakly ordered machines -* for which spin_unlock() acts as a full memory -* barrier, finish_lock_switch() in common code takes -* care of this barrier. For weakly ordered machines for -* which spin_unlock() acts as a RELEASE barrier (only -* arm64 and PowerPC), arm64 has a full barrier in -* switch_to(), and PowerPC has -* smp_mb__after_unlock_lock() before -* finish_lock_switch(). +* rq->curr, before returning to user-space. +* +* Here are the schemes providing that barrier on the +* various architectures: +* - mm ? switch_mm() : mmdrop() for x86, s390, sparc, PowerPC. +* switch_mm() rely on membarrier_arch_switch_mm() on PowerPC. +* - finish_lock_switch() for weakly-ordered +* architectures where spin_unlock is a full barrier, +* - switch_to() for arm64 (weakly-ordered, spin_unlock +* is a RELEASE barrier), */ ++*switch_count;
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v2 21/22 membarrier: provide SHARED_EXPEDITED command
In reply to Mathieu DesnoyersAllow expedited membarrier to be used for data shared between processes (shared memory).
Processes wishing to receive the membarriers register with MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED. Those which want to issue membarrier invoke MEMBARRIER_CMD_SHARED_EXPEDITED.
This allows extremely simple kernel-level implementation: we have almost everything we need with the PRIVATE_EXPEDITED barrier code. All we need to do is to add a flag in the mm_struct that will be used to check whether we need to send the IPI to the current thread of each CPU.
There is a slight downside of this approach compared to targeting specific shared memory users: when performing a membarrier operation, all registered "shared" receivers will get the barrier, even if they don't share a memory mapping with the "sender" issuing MEMBARRIER_CMD_SHARED_EXPEDITED.
This registration approach seems to fit the requirement of not disturbing processes that really deeply care about real-time: they simply should not register with MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED.
---
Changes since v1:
- Add missing preempt disable around smp_call_function_many().
--- arch/powerpc/include/asm/membarrier.h | 3 +- include/linux/sched/mm.h | 6 +- include/uapi/linux/membarrier.h | 34 ++++++++-- kernel/sched/membarrier.c | 114 ++++++++++++++++++++++++++++++++-- 4 files changed, 143 insertions(+), 14 deletions(-)
arch/powerpc/include/asm/membarrier.h +2 -1
@@ -12,7 +12,8 @@ static inline void membarrier_arch_switch_mm(struct mm_struct *prev,* store to rq->curr. */ if (likely (!(atomic_read (&next->membarrier_state) - &MEMBARRIER_STATE_PRIVATE_EXPEDITED ) || !prev)) + & (MEMBARRIER_STATE_PRIVATE_EXPEDITED + |MEMBARRIER_STATE_SHARED_EXPEDITED )) || !prev))return ;/*
include/linux/sched/mm.h +4 -2
@@ -219,8 +219,10 @@ static inline void memalloc_noreclaim_restore(unsigned int flags)#ifdef CONFIG_MEMBARRIER enum { -MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY = (1U <<0 ), -MEMBARRIER_STATE_PRIVATE_EXPEDITED = (1U <<1 ), +MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY = (1U <<0 ), +MEMBARRIER_STATE_PRIVATE_EXPEDITED = (1U <<1 ), +MEMBARRIER_STATE_SHARED_EXPEDITED_READY = (1U <<2 ), +MEMBARRIER_STATE_SHARED_EXPEDITED = (1U <<3 ), };#ifdef CONFIG_ARCH_HAS_MEMBARRIER_HOOKS
include/uapi/linux/membarrier.h +28 -6
@@ -40,6 +40,28 @@* (non-running threads are de facto in such a * state). This covers threads from all processes * running on the system. This command returns 0. +* @MEMBARRIER_CMD_SHARED_EXPEDITED: +* Execute a memory barrier on all running threads +* part of a process which previously registered +* with MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED. +* Upon return from system call, the caller thread +* is ensured that all running threads have passed +* through a state where all memory accesses to +* user-space addresses match program order between +* entry to and return from the system call +* (non-running threads are de facto in such a +* state). This only covers threads from processes +* which registered with +* MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED. +* This command returns 0. Given that +* registration is about the intent to receive +* the barriers, it is valid to invoke +* MEMBARRIER_CMD_SHARED_EXPEDITED from a +* non-registered process. +* @MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED: +* Register the process intent to receive +* MEMBARRIER_CMD_SHARED_EXPEDITED memory +* barriers. Always returns 0. * @MEMBARRIER_CMD_PRIVATE_EXPEDITED: * Execute a memory barrier on each running * thread belonging to the same process as the current
@@ -70,12 +92,12 @@* the value 0. */ enum membarrier_cmd { -MEMBARRIER_CMD_QUERY =0 , -MEMBARRIER_CMD_SHARED = (1 <<0 ), -/* reserved for MEMBARRIER_CMD_SHARED_EXPEDITED (1 << 1) */ -/* reserved for MEMBARRIER_CMD_PRIVATE (1 << 2) */ -MEMBARRIER_CMD_PRIVATE_EXPEDITED = (1 <<3 ), -MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED = (1 <<4 ), +MEMBARRIER_CMD_QUERY =0 , +MEMBARRIER_CMD_SHARED = (1 <<0 ), +MEMBARRIER_CMD_SHARED_EXPEDITED = (1 <<1 ), +MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED = (1 <<2 ), +MEMBARRIER_CMD_PRIVATE_EXPEDITED = (1 <<3 ), +MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED = (1 <<4 ), };#endif /* _UAPI_LINUX_MEMBARRIER_H */
kernel/sched/membarrier.c +109 -5
153 lines not on this page · raw message
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH v12 22/22 membarrier: selftest: Test shared expedited cmd
In reply to Mathieu DesnoyersTest the new MEMBARRIER_CMD_SHARED_EXPEDITED and MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED commands.
--- .../testing/selftests/membarrier/membarrier_test.c | 51 +++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-)
tools/testing/selftests/membarrier/membarrier_test.c +50 -1
@@ -132,6 +132,40 @@ static int test_membarrier_private_expedited_success(void)return 0 ; } +static int test_membarrier_register_shared_expedited_success (void ) +{ +int cmd =MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED , flags =0 ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_REGISTER_SHARED_EXPEDITED" ; + +if (sys_membarrier (cmd, flags) !=0 ) { +ksft_exit_fail_msg ( +"%s test: flags = %d, errno = %d\n" , + test_name, flags, errno); + } + +ksft_test_result_pass ( +"%s test: flags = %d\n" , + test_name, flags); +return 0 ; +} + +static int test_membarrier_shared_expedited_success (void ) +{ +int cmd =MEMBARRIER_CMD_SHARED_EXPEDITED , flags =0 ; +const char *test_name ="sys membarrier MEMBARRIER_CMD_SHARED_EXPEDITED" ; + +if (sys_membarrier (cmd, flags) !=0 ) { +ksft_exit_fail_msg ( +"%s test: flags = %d, errno = %d\n" , + test_name, flags, errno); + } + +ksft_test_result_pass ( +"%s test: flags = %d\n" , + test_name, flags); +return 0 ; +} +static int test_membarrier (void ) {int status;
@@ -154,6 +188,19 @@ static int test_membarrier(void) status =test_membarrier_private_expedited_success ();if (status)return status; +/* +* It is valid to send a shared membarrier from a non-registered +* process. +*/ + status =test_membarrier_shared_expedited_success (); +if (status) +return status; + status =test_membarrier_register_shared_expedited_success (); +if (status) +return status; + status =test_membarrier_shared_expedited_success (); +if (status) +return status;return 0 ; }
@@ -173,8 +220,10 @@ static int test_membarrier_query(void) }ksft_exit_fail_msg ("sys_membarrier() failed\n" ); } -if (!(ret &MEMBARRIER_CMD_SHARED )) +if (!(ret &MEMBARRIER_CMD_SHARED )) { +ksft_test_result_fail ("sys_membarrier() CMD_SHARED query failed\n" );ksft_exit_fail_msg ("sys_membarrier is not supported.\n" ); + }ksft_test_result_pass ("sys_membarrier available\n" );return 0 ;
Signature · 1 line
2.11.0
Quoted 6 lines · Mathieu Desnoyers
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
My suggestion would be that you also split out the opv system call. That seems to be main contention point currently, and the restartable sequences should be useful without it.
-Andi
Mathieu Desnoyers
maintainerQuoted 10 lines · ----- On Nov 21, 2017, at 12:21 PM, Andi Kleen [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
My suggestion would be that you also split out the opv system call. That seems to be main contention point currently, and the restartable sequences should be useful without it.
I consider rseq to be incomplete and a pain to use in various scenarios without cpu_opv.
About the contention point you refer to:
Using vDSO as an example of how things should be done is just wrong: the vDSO interaction with debugger instruction single-stepping is broken, as I detailed in my previous email.
Thomas' proposal of handling single-stepping with a user-space locking fallback, which is pretty much what I had in 2016, pushes a lot of complexity to user-space, requires an extra branch in the fast-path, as well as additional store-release/load-acquire semantics for consistency. I don't plan going down that route.
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
The only opposition against cpu_opv is that there *should* be an hypothetical simpler solution. The rseq idea is not new: it's been presented by Paul Turner in 2012 at LPC. And so far, cpu_opv is the overall simplest and most efficient way I encountered to handle single-stepping, and it gives extra benefits, as described in my changelog.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 13 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Nov 21, 2017, at 12:21 PM, Andi Kleen…
Quoted 6 lines · Mathieu Desnoyers
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
My suggestion would be that you also split out the opv system call. That seems to be main contention point currently, and the restartable sequences should be useful without it.
Quoted 8 lines
I consider rseq to be incomplete and a pain to use in various scenarios without
cpu_opv.About the contention point you refer to:
Using vDSO as an example of how things should be done is just wrong: the vDSO interaction with debugger instruction single-stepping is broken, as I detailed in my previous email.
Let me turn that around. You're lamenting about a conditional branch in your rseq thing for performance reasons and at the same time you want to force extra code into the VDSO? clock_gettime() is one of the hottest vsyscalls in certain scenarions. So why would we want to have extra code there? Just to make debuggers happy. You really can't be serious about that.
Quoted 8 lines
Thomas' proposal of handling single-stepping with a user-space locking fallback, which is pretty much what I had in 2016, pushes a lot of complexity to user-space, requires an extra branch in the fast-path, as well as additional store-release/load-acquire semantics for consistency. I don't plan going down that route.
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
You provided the details today. Up to that point all we had was handwaving and inconsistent information.
Quoted 5 lines
The only opposition against
cpu_opvis that there *should* be an hypothetical simpler solution. The rseq idea is not new: it's been presented by Paul Turner in 2012 at LPC. And so far,cpu_opvis the overall simplest and most efficient way I encountered to handle single-stepping, and it gives extra benefits, as described in my changelog.
That's how you define it and that does not make cpu_opv less complex and more debuggable. There is no way to debug that and still you claim that it removes compexity from user space. That ops stuff comes from user space and is not magically constructed by the kernel. In some of your use cases it even has different semantics than the rseq section code. So how is that removing any complexity from user space? All it buys you is an extra branch less in your rseq hotpath and that's your justification to shove that thing into the kernel.
The version I reviewed was just undigestable. I did not have time to look at the hastily cobbled together version of today. Aside of that the scheduler portion of it has not seen any review from scheduler folks either.
AFAICT there is not a single reviewed-by tag on the sys_rseq and the sys_opv patches either.
Are you seriously expecting that new syscalls of that kind are going to be merged without a deep and thorough review just based on your decision to declare them ready?
Thanks,
tglx
Mathieu Desnoyers
maintainerQuoted 23 lines · ----- On Nov 21, 2017, at 5:59 PM, Thomas Gleixner [email protected] wrote:
Quoted 13 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Nov 21, 2017, at 12:21 PM, Andi Kleen [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
My suggestion would be that you also split out the opv system call. That seems to be main contention point currently, and the restartable sequences should be useful without it.
Quoted 8 lines
I consider rseq to be incomplete and a pain to use in various scenarios without
cpu_opv.About the contention point you refer to:
Using vDSO as an example of how things should be done is just wrong: the vDSO interaction with debugger instruction single-stepping is broken, as I detailed in my previous email.
Quoted 6 lines
Let me turn that around. You're lamenting about a conditional branch in your rseq thing for performance reasons and at the same time you want to force extra code into the VDSO?
clock_gettime()is one of the hottest vsyscalls in certain scenarions. So why would we want to have extra code there? Just to make debuggers happy. You really can't be serious about that.
There is *already* an existing branch in the clock_gettime vsyscall:
it's a loop. It won't hurt the fast-path to use that branch and make it do something else instead. It could even help the vDSO fast-path for some non-x86 architectures where branch prediction assumes that backward branches are always taken (adding an unlikely() does not help in those cases).
Quoted 7 lines
Quoted 7 lines
Thomas' proposal of handling single-stepping with a user-space locking fallback, which is pretty much what I had in 2016, pushes a lot of complexity to user-space, requires an extra branch in the fast-path, as well as additional store-release/load-acquire semantics for consistency. I don't plan going down that route.
Quoted 5 lines
Quoted 2 lines
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
You provided the details today. Up to that point all we had was handwaving and inconsistent information.
I mistakenly presumed you took interest in the past 2 years discussions. It appears I was wrong, and that information needed to be summarized in my changelog. This was my mistake and I fixed it.
Quoted 10 lines
Quoted 6 lines
The only opposition against
cpu_opvis that there *should* be an hypothetical simpler solution. The rseq idea is not new: it's been presented by Paul Turner in 2012 at LPC. And so far,cpu_opvis the overall simplest and most efficient way I encountered to handle single-stepping, and it gives extra benefits, as described in my changelog.That's how you define it and that does not make
cpu_opvless complex and more debuggable. There is no way to debug that and still you claim that it removes compexity from user space.
So I should ask: what kind of observability within cpu_opv() do you want ? I can add a tracepoint for each operation, which would technically take care of your concern. You main counter-argument seems to be a tooling issue.
Quoted 6 lines
That ops stuff comes from user space and
is not magically constructed by the kernel. In some of your use cases it even has different semantics than the rseq section code. So how is that removing any complexity from user space? All it buys you is an extra branch less in your rseq hotpath and that's your justification to shove that thing into the kernel.
Actually, the cpu-op user-space library can hide this difference from the
user: I implemented the equivalent rseq algorithm using a compare-and-store:
int cpu_op_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
off_t voffp, intptr_t *load, int cpu)
{
intptr_t oldv = READ_ONCE(*v);
intptr_t *newp = (intptr_t *)(oldv + voffp);
int ret;
if (oldv == expectnot)
return 1;
ret = cpu_op_cmpeqv_storep_expect_fault(v, oldv, newp, cpu);
if (!ret) {
*load = oldv;
return 0;
}
if (ret > 0) {
errno = EAGAIN;
return -1;
}
return -1;
}So from a library user perspective, the fast-path and slow-path are exactly the same.
Quoted 2 lines
The version I reviewed was just undigestable.
Thanks for the thorough coding style review by the way.
Quoted 4 lines
I did not have time to look
at the hastily cobbled together version of today. Aside of that the scheduler portion of it has not seen any review from scheduler folks either.
True. It appears that it really takes a merge window to get some people's attention. That's OK, you guys are really busy on other stuff. It's just unfortunate that the feedback about the cpu_opv concept did not come sooner, e.g. during first rounds of patches where the cpu_opv design was presented, or even at KS.
Quoted 3 lines
AFAICT there is not a single reviewed-by tag on the
sys_rseqand thesys_opvpatches either.
Very good point! Anyone in CC who cares about getting this in can find time to do some official review ?
Quoted 4 lines
Are you seriously expecting that new syscalls of that kind are going to be merged without a deep and thorough review just based on your decision to declare them ready?
In my reply to Andi, I merely state that I'm not willing to push an half-baked user-space ABI into the kernel, and rseq without cpu_opv is only part of the solution.
Let's see if others find time to do an official review.
Thanks,
Mathieu
Quoted 4 lines
Thanks,
tglx
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 12 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Nov 21, 2017, at 5:59 PM, Thomas Gleixner…
Quoted 3 lines
Using vDSO as an example of how things should be done is just wrong: the vDSO interaction with debugger instruction single-stepping is broken, as I detailed in my previous email.
Let me turn that around. You're lamenting about a conditional branch in your rseq thing for performance reasons and at the same time you want to force extra code into the VDSO?
clock_gettime()is one of the hottest vsyscalls in certain scenarions. So why would we want to have extra code there? Just to make debuggers happy. You really can't be serious about that.
Quoted 6 lines
There is *already* an existing branch in the
clock_gettimevsyscall:
it's a loop. It won't hurt the fast-path to use that branch and make it do something else instead. It could even help the vDSO fast-path for some non-x86 architectures where branch prediction assumes that backward branches are always taken (adding anunlikely()does not help in those cases).
Yes, there is an existing branch, but forcing that thing into the real syscall when the seqcount does not match is silly.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Thomas' proposal of handling single-stepping with a user-space locking fallback, which is pretty much what I had in 2016, pushes a lot of complexity to user-space, requires an extra branch in the fast-path, as well as additional store-release/load-acquire semantics for consistency. I don't plan going down that route.
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
Quoted 6 lines
Quoted 2 lines
You provided the details today. Up to that point all we had was handwaving and inconsistent information.
I mistakenly presumed you took interest in the past 2 years discussions. It appears I was wrong, and that information needed to be summarized in my changelog. This was my mistake and I fixed it.
I took interest, but I did not follow all the details. And it's not about me. Anyone who looks at those patches for whatever reason should get enough information.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
The only opposition against
cpu_opvis that there *should* be an hypothetical simpler solution. The rseq idea is not new: it's been presented by Paul Turner in 2012 at LPC. And so far,cpu_opvis the overall simplest and most efficient way I encountered to handle single-stepping, and it gives extra benefits, as described in my changelog.
Quoted 7 lines
Quoted 3 lines
That's how you define it and that does not make
cpu_opvless complex and more debuggable. There is no way to debug that and still you claim that it removes compexity from user space.So I should ask: what kind of observability within
cpu_opv()do you want ? I can add a tracepoint for each operation, which would technically take care of your concern. You main counter-argument seems to be a tooling issue.
Yes, it's a tooling issue and this is issue is not going to be solved faster than gdb support for skipping rseq sections.
Quoted 9 lines
Quoted 6 lines
That ops stuff comes from user space and
is not magically constructed by the kernel. In some of your use cases it even has different semantics than the rseq section code. So how is that removing any complexity from user space? All it buys you is an extra branch less in your rseq hotpath and that's your justification to shove that thing into the kernel.Actually, the cpu-op user-space library can hide this difference from the
user: I implemented the equivalent rseq algorithm using a compare-and-store:
Yes, you implemented it and I don't want to know how long it took to get it right. But syscalls are not coupled to be used with a particular library.
Quoted 2 lines
So from a library user perspective, the fast-path and slow-path are exactly the same.
Aside of the fact that they are not the same in terms of implementation and semantics. I'm looking at the syscall and not reviewing your magic user space library.
Thanks,
tglx
Quoted 13 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Nov 21, 2017, at 12:21 PM, Andi Kleen [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
My suggestion would be that you also split out the opv system call. That seems to be main contention point currently, and the restartable sequences should be useful without it.
Quoted 9 lines
I consider rseq to be incomplete and a pain to use in various scenarios without
cpu_opv.About the contention point you refer to:
Using vDSO as an example of how things should be done is just wrong: the vDSO interaction with debugger instruction single-stepping is broken, as I detailed in my previous email.
If anyone ever reports that as a problem, I'll gladly fix it in the kernel. That's doable without an ABI change. If rseq-like things started breaking single-stepping, we can't just fix it in the kernel.
Also, there is one and only one vclock_gettime. Debuggers can easily special-case it. For all I know, they already do.
Mathieu Desnoyers
maintainerQuoted 20 lines · ----- On Nov 22, 2017, at 10:28 AM, Andy Lutomirski [email protected] wrote:
Quoted 18 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Nov 21, 2017, at 12:21 PM, Andi Kleen [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Following changes based on a thorough coding style and patch changelog review from Thomas Gleixner and Peter Zijlstra, I'm respinning this series for another RFC.
My suggestion would be that you also split out the opv system call. That seems to be main contention point currently, and the restartable sequences should be useful without it.
I consider rseq to be incomplete and a pain to use in various scenarios without
cpu_opv.About the contention point you refer to:
Quoted 8 lines
Quoted 4 lines
Using vDSO as an example of how things should be done is just wrong: the vDSO interaction with debugger instruction single-stepping is broken, as I detailed in my previous email.
If anyone ever reports that as a problem, I'll gladly fix it in the kernel. That's doable without an ABI change. If rseq-like things started breaking single-stepping, we can't just fix it in the kernel.
Very true. And rseq does break both line-level and instruction-level single-stepping.
Quoted 3 lines
Also, there is one and only one
vclock_gettime. Debuggers can easily special-case it. For all I know, they already do.
As my tests demonstrate, they don't. clock_gettime() vDSO currently breaks instruction-level single-stepping (istep) with gdb. I'll forward you the writeup I did on that a few days ago.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 3 lines
Quoted 3 lines
If anyone ever reports that as a problem, I'll gladly fix it in the kernel. That's doable without an ABI change. If rseq-like things started breaking single-stepping, we can't just fix it in the kernel.
AFAIK nobody ever complained about it since we have vsyscalls and vDSOs.
Quoted 3 lines
Very true. And rseq does break both line-level and instruction-level single-stepping.
They can just set a break point after it and continue.
In fact it could be even expressed to the debugger to do that automatically based on some dwarf extension.
I also disagree that opv somehow "solves" debugging: it's a completely different code path that has nothing to do with the original code path. That's not debugging, that's at best a workaround. I don't think it's any better than the break point method.
-Andi
Peter Zijlstra
maintainerQuoted 2 lines · Mathieu Desnoyers
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
That's not entirely true; amluto did have an alternative in Prague: do full machine level instruction emulation till the end of the rseq when it gets 'preempted too often'.
Yes, implementing that will be an absolute royal pain. But it does remove the whole duplicate/dual program asm/bytecode thing and avoids the syscall entirely.
And we don't need to do a full x86_64/arch-of-choice emulator for this either; just as cpu_opv is fairly limited too. We can do a subset that allows dealing with the known sequences and go from there -- it can always fall back to not emulating and reverting to the pure rseq with debug/fwd progress 'issues'.
So what exactly is the problem of leaving out the whole cpu_opv thing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.
Quoted 18 lines · Peter Zijlstra
Quoted 2 lines · Mathieu Desnoyers
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
That's not entirely true; amluto did have an alternative in Prague: do full machine level instruction emulation till the end of the rseq when it gets 'preempted too often'.
Yes, implementing that will be an absolute royal pain. But it does remove the whole duplicate/dual program asm/bytecode thing and avoids the syscall entirely.
And we don't need to do a full x86_64/arch-of-choice emulator for this either; just as
cpu_opvis fairly limited too. We can do a subset that allows dealing with the known sequences and go from there -- it can always fall back to not emulating and reverting to the pure rseq with debug/fwd progress 'issues'.
Quoted 3 lines
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.
Drive-by "ack" to that. I'd really like a working rseq implementation in mainline, but I don't much care for another interpreter.
Will
Mathieu Desnoyers
maintainerQuoted 19 lines · ----- On Nov 22, 2017, at 2:37 PM, Will Deacon [email protected] wrote:
Quoted 18 lines · Peter Zijlstra
Quoted 2 lines · Mathieu Desnoyers
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
That's not entirely true; amluto did have an alternative in Prague: do full machine level instruction emulation till the end of the rseq when it gets 'preempted too often'.
Yes, implementing that will be an absolute royal pain. But it does remove the whole duplicate/dual program asm/bytecode thing and avoids the syscall entirely.
And we don't need to do a full x86_64/arch-of-choice emulator for this either; just as
cpu_opvis fairly limited too. We can do a subset that allows dealing with the known sequences and go from there -- it can always fall back to not emulating and reverting to the pure rseq with debug/fwd progress 'issues'.
Quoted 6 lines
Quoted 3 lines
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.Drive-by "ack" to that. I'd really like a working rseq implementation in mainline, but I don't much care for another interpreter.
Considering the arm 64 use-case of reading PMU counters from user-space using rseq to prevent migration, I understand that you're lucky enough to already have a system call at your disposal that can perform the slow-path in case of single-stepping.
So yes, your particular case is already covered, but unfortunately that's not the same situation for other use-cases that have been expressed.
Thanks,
Mathieu
Quoted 2 lines
Will
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 10 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Nov 22, 2017, at 2:37 PM, Will Deacon…
Quoted 4 lines · Peter Zijlstra
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.Drive-by "ack" to that. I'd really like a working rseq implementation in mainline, but I don't much care for another interpreter.
Quoted 7 lines
Considering the arm 64 use-case of reading PMU counters from user-space using rseq to prevent migration, I understand that you're lucky enough to already have a system call at your disposal that can perform the slow-path in case of single-stepping.
So yes, your particular case is already covered, but unfortunately that's not the same situation for other use-cases that have been expressed.
If we have users of rseq which can do without the other muck, then what's the reason not to support it?
The sysops thing can be sorted out on top and the use cases which need both will have to test for both syscalls being available anyway.
Tanks,
tglx
Mathieu Desnoyers
maintainerQuoted 16 lines · ----- On Nov 23, 2017, at 5:51 PM, Thomas Gleixner [email protected] wrote:
Quoted 15 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Nov 22, 2017, at 2:37 PM, Will Deacon [email protected] wrote:
Quoted 4 lines · Peter Zijlstra
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.Drive-by "ack" to that. I'd really like a working rseq implementation in mainline, but I don't much care for another interpreter.
Considering the arm 64 use-case of reading PMU counters from user-space using rseq to prevent migration, I understand that you're lucky enough to already have a system call at your disposal that can perform the slow-path in case of single-stepping.
Quoted 8 lines
Quoted 2 lines
So yes, your particular case is already covered, but unfortunately that's not the same situation for other use-cases that have been expressed.
If we have users of rseq which can do without the other muck, then what's the reason not to support it?
The sysops thing can be sorted out on top and the use cases which need both will have to test for both syscalls being available anyway.
I'm currently making sure CONFIG_RSEQ selects both CONFIG_CPU_OPV and CONFIG_MEMBARRIER, so the user-space fast-paths don't end up with various ways of doing the fallback/single-stepping/memory barrier handling depending on whether the kernel support each of those individually. So first of all, it reduces complexity from a user-space perspective.
Moreover, with a single already needed cpu_id vs cpu_id_start field comparison in the rseq fast-path, user-space knows that it can rely on having rseq, cpu_opv, and membarrier. Without this guarantee, user-space would have to detect individually whether each of those system calls is available, and test flags on the fast-path, for additional overhead.
Those are my main concerns about pushing an incomplete solution at this stage.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 32 lines · Mathieu Desnoyers
Quoted 16 lines · ----- On Nov 23, 2017, at 5:51 PM, Thomas Gleixner…
Quoted 15 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Nov 22, 2017, at 2:37 PM, Will Deacon…
Quoted 4 lines · Peter Zijlstra
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.Drive-by "ack" to that. I'd really like a working rseq implementation in mainline, but I don't much care for another interpreter.
Considering the arm 64 use-case of reading PMU counters from user-space using rseq to prevent migration, I understand that you're lucky enough to already have a system call at your disposal that can perform the slow-path in case of single-stepping.
Quoted 8 lines
Quoted 2 lines
So yes, your particular case is already covered, but unfortunately that's not the same situation for other use-cases that have been expressed.
If we have users of rseq which can do without the other muck, then what's the reason not to support it?
The sysops thing can be sorted out on top and the use cases which need both will have to test for both syscalls being available anyway.
I'm currently making sure
CONFIG_RSEQselects bothCONFIG_CPU_OPVandCONFIG_MEMBARRIER, so the user-space fast-paths don't end up with various ways of doing the fallback/single-stepping/memory barrier handling depending on whether the kernel support each of those individually. So first of all, it reduces complexity from a user-space perspective.
Quoted 5 lines
Moreover, with a single already needed
cpu_idvscpu_id_startfield comparison in the rseq fast-path, user-space knows that it can rely on having rseq,cpu_opv, and membarrier. Without this guarantee, user-space would have to detect individually whether each of those system calls is available, and test flags on the fast-path, for additional overhead.
You have to test for sys_rseq somewhere in the init code. So you can test for the other two being fully functional as well.
If one of them is missing then you can avoid that rseq fastpath either completely or because you never registered that rseq muck your rseq will just contain stale init data which kicks you into some slowpath fallback code.
You need something like this anyway unless you plan to ship code which cannot run on systems w/o rseq support at all.
Either you designed your thing wrong or you try to create an artifical dependency for political reasons.
Thanks,
tglx
Mathieu Desnoyers
maintainerQuoted 39 lines · ----- On Nov 23, 2017, at 6:38 PM, Thomas Gleixner [email protected] wrote:
Quoted 32 lines · Mathieu Desnoyers
Quoted 16 lines · ----- On Nov 23, 2017, at 5:51 PM, Thomas Gleixner [email protected] wrote:
Quoted 15 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Nov 22, 2017, at 2:37 PM, Will Deacon [email protected] wrote:
Quoted 4 lines · Peter Zijlstra
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.Drive-by "ack" to that. I'd really like a working rseq implementation in mainline, but I don't much care for another interpreter.
Considering the arm 64 use-case of reading PMU counters from user-space using rseq to prevent migration, I understand that you're lucky enough to already have a system call at your disposal that can perform the slow-path in case of single-stepping.
Quoted 8 lines
Quoted 2 lines
So yes, your particular case is already covered, but unfortunately that's not the same situation for other use-cases that have been expressed.
If we have users of rseq which can do without the other muck, then what's the reason not to support it?
The sysops thing can be sorted out on top and the use cases which need both will have to test for both syscalls being available anyway.
I'm currently making sure
CONFIG_RSEQselects bothCONFIG_CPU_OPVandCONFIG_MEMBARRIER, so the user-space fast-paths don't end up with various ways of doing the fallback/single-stepping/memory barrier handling depending on whether the kernel support each of those individually. So first of all, it reduces complexity from a user-space perspective.Quoted 5 lines
Moreover, with a single already needed
cpu_idvscpu_id_startfield comparison in the rseq fast-path, user-space knows that it can rely on having rseq,cpu_opv, and membarrier. Without this guarantee, user-space would have to detect individually whether each of those system calls is available, and test flags on the fast-path, for additional overhead.
Quoted 7 lines
You have to test for
sys_rseqsomewhere in the init code. So you can test for the other two being fully functional as well.If one of them is missing then you can avoid that rseq fastpath either completely or because you never registered that rseq muck your rseq will just contain stale init data which kicks you into some slowpath fallback code.
That would work if we could have more than one rseq TLS entry per thread. If it would be the case, then e.g. lttng-ust could own its own rseq TLS and do just as you explain above.
It's not the case with the current proposal. This means multiple user libraries will have to share the same cpu_id and cpu_id_start fields, which breaks your proposed new-app/old-kernel backward compatibility check proposal.
For instance, if glibc librseq.so happily registers rseq (and does not care about testing for cpu_opv or membarrier availability), then lttng-ust cannot leave stale rseq init data which kicks in its slowpath fallback.
Quoted 3 lines
You need something like this anyway unless you plan to ship code which cannot run on systems w/o rseq support at all.
My plan is to ensure that testing for
(TLS::rseq->cpu_id_start == TLS::rseq->cpu_id)
should be enough for fast-paths to guarantee that:
- rseq is available and registered for the current thread,
- cpu_opv is available as fallback,
- membarrier private_expedited and shared_expedited are available.
Quoted 3 lines
Either you designed your thing wrong or you try to create an artifical dependency for political reasons.
Having the rseq TLS shared across multiple library/app users within a single process does limit our options there. :-/
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 18 lines · Mathieu Desnoyers
Quoted 7 lines · ----- On Nov 23, 2017, at 6:38 PM, Thomas Gleixner…
You have to test for
sys_rseqsomewhere in the init code. So you can test for the other two being fully functional as well.If one of them is missing then you can avoid that rseq fastpath either completely or because you never registered that rseq muck your rseq will just contain stale init data which kicks you into some slowpath fallback code.
That would work if we could have more than one rseq TLS entry per thread. If it would be the case, then e.g. lttng-ust could own its own rseq TLS and do just as you explain above.
It's not the case with the current proposal. This means multiple user libraries will have to share the same
cpu_idandcpu_id_startfields, which breaks your proposed new-app/old-kernel backward compatibility check proposal.
Quoted 4 lines
For instance, if glibc librseq.so happily registers rseq (and does not care about testing for
cpu_opvor membarrier availability), then lttng-ust cannot leave stale rseq init data which kicks in its slowpath fallback.
You have to make sure that _ALL_ prerequisites are there before you start using it whether you have a shared rseq or not. If a setup has rseq working and sysops are blocked by a stupid mistake in a security filter, then your assumption of testing rseq alone is broken already and stuff will explode in hard to debug ways.
You CANNOT make such assumptions ever. Robustness is the first thing to look at and after that you can optimize the hell out of it, without violating robustness while doing that.
Thanks,
tglx
Mathieu Desnoyers
maintainerQuoted 7 lines · ----- On Nov 22, 2017, at 2:32 PM, Peter Zijlstra [email protected] wrote:
Quoted 2 lines · Mathieu Desnoyers
Other than that, I have not received any concrete alternative proposal to properly handle single-stepping.
That's not entirely true; amluto did have an alternative in Prague: do full machine level instruction emulation till the end of the rseq when it gets 'preempted too often'.
Yes, that's right. Andy did propose that alternative at KS. Which is also interpreter-based.
Quoted 4 lines
Yes, implementing that will be an absolute royal pain. But it does remove the whole duplicate/dual program asm/bytecode thing and avoids the syscall entirely.
Agreed on this being a royal pain that we'd have to do for each architecture.
By the way, I figured an interesting library API that would remove the need for code duplication for end-users:
e.g.
static inline __attribute__((always_inline)) int percpu_addv(intptr_t *v, intptr_t count, int cpu)
{
rseq_addv(v, count, cpu);
if (rseq_unlikely(rseq_addv(v, count, cpu)))
return cpu_op_addv(v, count, cpu);
return 0;
}And the caller becomes:
cpu = rseq_cpu_start();
ret = percpu_addv(&data->c[cpu].count, 1, cpu);
if (unlikely(ret)) {
perror("cpu_opv");
abort();
}So the caller does not even have to bother retrying in case of rseq error, it's all handled by the "percpu_*()" static inlines.
Quoted 6 lines
And we don't need to do a full x86_64/arch-of-choice emulator for this either; just as
cpu_opvis fairly limited too. We can do a subset that allows dealing with the known sequences and go from there -- it can always fall back to not emulating and reverting to the pure rseq with debug/fwd progress 'issues'.
I think trying to make the kernel ABI "developer-friendly" is the wrong approach. This kind of ease-of-use sugar should be provided by a library, not by the kernel ABI. The futex system call is a good example of low-level syscall meant to be used by libraries rather than directly by end-users.
Quoted 3 lines
So what exactly is the problem of leaving out the whole
cpu_opvthing for now? Pure rseq is usable -- albeit a bit cumbersome without additional debugger support.
Then rseq will cover _some_ use-cases, but will miss many others.
One example is the reserve+commit lttng-ust ring buffer operations, where the commit _needs_ to run on the same CPU as the reserve. Just rseq does not allow a tracer library to do that, rseq+cpu_opv allow that just fine.
So if I introduce just rseq for now, then all those other use-cases will need to check whether the kernel supports cpu_opv or not as well, cache the result into a variable, and it will add a forest of branches into those fast paths. No thanks.
Also, turning both line-level and instruction-level single-stepping into infinite loops looks pretty much like a new kernel facility that breaks user-space. It's a no-go from my point of view.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 3 lines
Also, turning both line-level and instruction-level single-stepping into infinite loops looks pretty much like a new kernel facility that breaks user-space. It's a no-go from my point of view.
You could fix it at the debugger level with suitable annotation. Just turn the whole rseq into an extended line, and make sure it is handled for instruction stepping too.
-Andi
Mathieu Desnoyers
maintainerPATCH 1/3 selftests: lib.mk: Introduce OVERRIDE_TARGETS
In reply to Mathieu DesnoyersIntroduce OVERRIDE_TARGETS to allow tests to express dependencies on header files and .so, which require to override the selftests lib.mk targets.
--- tools/testing/selftests/lib.mk | 4 ++++ 1 file changed, 4 insertions(+)
tools/testing/selftests/lib.mk +4 -0
@@ -105,6 +105,9 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) endif +# Selftest makefiles can override those targets by setting +# OVERRIDE_TARGETS = 1. +ifeq ($(OVERRIDE_TARGETS),) $(OUTPUT)/%:%.c $(LINK.c) $^ $(LDLIBS) -o $@
@@ -113,5 +116,6 @@ $(OUTPUT)/%.o:%.S $(OUTPUT)/%:%.S $(LINK.S) $^ $(LDLIBS) -o $@ +endif .PHONY: run_tests all clean install emit_tests
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerPATCH 2/3 cpu_opv: selftests: Implement selftests (v4)
Implement cpu_opv selftests. It needs to express dependencies on header files and .so, which require to override the selftests lib.mk targets. Use OVERRIDE_TARGETS define for this.
---
Changes since v1:
- Expose similar library API as rseq: Expose library API closely matching the rseq APIs, following removal of the event counter from the rseq kernel API.
- Update makefile to fix make run_tests dependency on "all".
- Introduce a OVERRIDE_TARGETS.
Changes since v2:
- Test page faults.
Changes since v3:
- Move lib.mk OVERRIDE_TARGETS change to its own patch.
- Printout TAP output.
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpu-opv/.gitignore | 1 + tools/testing/selftests/cpu-opv/Makefile | 17 + .../testing/selftests/cpu-opv/basic_cpu_opv_test.c | 1167 ++++++++++++++++++++ tools/testing/selftests/cpu-opv/cpu-op.c | 348 ++++++ tools/testing/selftests/cpu-opv/cpu-op.h | 68 ++ 7 files changed, 1603 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/.gitignore create mode 100644 tools/testing/selftests/cpu-opv/Makefile create mode 100644 tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.h
MAINTAINERS +1 -0
@@ -3734,6 +3734,7 @@ L: [email protected] S: Supported F: kernel/cpu_opv.c F: include/uapi/linux/cpu_opv.h +F: tools/testing/selftests/cpu-opv/ CRAMFS FILESYSTEM M: Nicolas Pitre <[email protected]>
tools/testing/selftests/Makefile +1 -0
@@ -5,6 +5,7 @@ TARGETS += breakpoints TARGETS += capabilities TARGETS += cpufreq TARGETS += cpu-hotplug +TARGETS += cpu-opv TARGETS += efivarfs TARGETS += exec TARGETS += firmware
tools/testing/selftests/cpu-opv/.gitignore new file +1 -0
@@ -0,0 +1 @@ +basic_cpu_opv_test
tools/testing/selftests/cpu-opv/Makefile new file +17 -0
@@ -0,0 +1,17 @@ +CFLAGS += -O2 -Wall -g -I./ -I../../../../usr/include/ -L./ -Wl,-rpath=./ + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_cpu_opv_test + +TEST_GEN_PROGS_EXTENDED = libcpu-op.so + +include ../lib.mk + +$(OUTPUT)/libcpu-op.so: cpu-op.c cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) cpu-op.h + $(CC) $(CFLAGS) $< -lcpu-op -o $@
tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c new file +1167 -0
1,167 lines not on this page · raw message
tools/testing/selftests/cpu-opv/cpu-op.c new file +348 -0
348 lines not on this page · raw message
tools/testing/selftests/cpu-opv/cpu-op.h new file +68 -0
68 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [PATCH update for 4.15 2/3] cpu_opv: selftests: Implement selftests (v4)
Quoted 45 lines · Mathieu Desnoyers
Implement
cpu_opvselftests. It needs to express dependencies on header files and .so, which require to override the selftests lib.mk targets. UseOVERRIDE_TARGETSdefine for this.CC Russell King <linux-lFZ/[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <andi-Vw/[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <torvalds-de/[email protected]>CC Andrew Morton <akpm-de/[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>---
Changes since v1:- Expose similar library API as rseq: Expose library API closely matching the rseq APIs, following removal of the event counter from the rseq kernel API.
- Update makefile to fix makerun_testsdependency on "all".
- Introduce aOVERRIDE_TARGETS.Changes since v2:
- Test page faults.
Changes since v3:
- Move lib.mk
OVERRIDE_TARGETSchange to its own patch.
- Printout TAP output.
Quoted 15 lines
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpu-opv/.gitignore | 1 + tools/testing/selftests/cpu-opv/Makefile | 17 + .../testing/selftests/cpu-opv/basic_cpu_opv_test.c | 1167 ++++++++++++++++++++ tools/testing/selftests/cpu-opv/cpu-op.c | 348 ++++++ tools/testing/selftests/cpu-opv/cpu-op.h | 68 ++ 7 files changed, 1603 insertions(+) create mode 100644 tools/testing/selftests/cpu-opv/.gitignore create mode 100644 tools/testing/selftests/cpu-opv/Makefile create mode 100644 tools/testing/selftests/cpu-opv/basic_cpu_opv_test.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.c create mode 100644 tools/testing/selftests/cpu-opv/cpu-op.h
Looks good.
thanks,
-- Shuah
Mathieu Desnoyers
maintainerPATCH 3/3 rseq: selftests: Provide self-tests (v4)
In reply to Mathieu DesnoyersImplements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works.
"basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness.
"param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.
A run_param_test.sh script runs many variants of the parametrizable tests.
As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the cpu_opv system call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the __rseq_table section. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.
The rseq library expose APIs that present the fast-path operations. The new from userspace is, e.g. for a counter increment:
cpu = rseq_cpu_start();
ret = rseq_addv(&data->c[cpu].count, 1, cpu);
if (likely(!ret))
return 0; /* Success. */
do {
cpu = rseq_current_cpu();
ret = cpu_op_addv(&data->c[cpu].count, 1, cpu);
if (likely(!ret))
return 0; /* Success. */
} while (ret > 0 || errno == EAGAIN);
perror("cpu_op_addv");
return -1; /* Unexpected error. */PowerPC tests have been implemented by Boqun Feng.
---
Changes since v1:
- Provide abort-ip signature: The abort-ip signature is located just before the abort-ip target. It is currently hardcoded, but a user-space application could use the __rseq_table to iterate on all abort-ip targets and use a random value as signature if needed in the future.
- Add rseq_prepare_unload(): Libraries and JIT code using rseq critical sections need to issue rseq_prepare_unload() on each thread at least once before reclaim of struct rseq_cs.
- Use initial-exec TLS model, non-weak symbol: The initial-exec model is signal-safe, whereas the global-dynamic model is not. Remove the "weak" symbol attribute from the __rseq_abi in rseq.c. The rseq.so library will have ownership of that symbol, and there is not reason for an application or user library to try to define that symbol. The expected use is to link against libreq.so, which owns and provide that symbol.
- Set cpu_id to -2 on register error
- Add rseq_len syscall parameter, rseq_cs version
- Ensure disassember-friendly signature: x86 32/64 disassembler have a hard time decoding the instruction stream after a bad instruction. Use a nopl instruction to encode the signature. Suggested by Andy Lutomirski.
- Exercise parametrized tests variants in a shell scripts.
- Restartable sequences selftests: Remove use of event counter.
- Use cpu_id_start field: With the cpu_id_start field, the C preparation phase of the fast-path does not need to compare cpu_id < 0 anymore.
- Signal-safe registration and refcounting: Allow libraries using librseq.so to register it from signal handlers.
- Use OVERRIDE_TARGETS in makefile.
- Use "m" constraints for rseq_cs field.
Changes since v2:
- Update based on Thomas Gleixner's comments.
Changes since v3:
- Generate param_test_skip_fastpath and param_test_benchmark with -DSKIP_FASTPATH and -DBENCHMARK (respectively). Add param_test_fastpath to run_param_test.sh.
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 + tools/testing/selftests/rseq/Makefile | 33 + .../testing/selftests/rseq/basic_percpu_ops_test.c | 333 +++++ tools/testing/selftests/rseq/basic_test.c | 55 + tools/testing/selftests/rseq/param_test.c | 1285 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 535 ++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 567 +++++++++ tools/testing/selftests/rseq/rseq-x86.h | 898 ++++++++++++++ tools/testing/selftests/rseq/rseq.c | 116 ++ tools/testing/selftests/rseq/rseq.h | 154 +++ tools/testing/selftests/rseq/run_param_test.sh | 126 ++ 13 files changed, 4108 insertions(+) create mode 100644 tools/testing/selftests/rseq/.gitignore create mode 100644 tools/testing/selftests/rseq/Makefile create mode 100644 tools/testing/selftests/rseq/basic_percpu_ops_test.c create mode 100644 tools/testing/selftests/rseq/basic_test.c create mode 100644 tools/testing/selftests/rseq/param_test.c create mode 100644 tools/testing/selftests/rseq/rseq-arm.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc.h create mode 100644 tools/testing/selftests/rseq/rseq-x86.h create mode 100644 tools/testing/selftests/rseq/rseq.c create mode 100644 tools/testing/selftests/rseq/rseq.h create mode 100755 tools/testing/selftests/rseq/run_param_test.sh
MAINTAINERS +1 -0
@@ -11634,6 +11634,7 @@ S: Supported F: kernel/rseq.c F: include/uapi/linux/rseq.h F: include/trace/events/rseq.h +F: tools/testing/selftests/rseq/ RFKILL M: Johannes Berg <[email protected]>
tools/testing/selftests/Makefile +1 -0
@@ -26,6 +26,7 @@ TARGETS += nsfs TARGETS += powerpc TARGETS += pstore TARGETS += ptrace +TARGETS += rseq TARGETS += seccomp TARGETS += sigaltstack TARGETS += size
tools/testing/selftests/rseq/.gitignore new file +4 -0
@@ -0,0 +1,4 @@ +basic_percpu_ops_test +basic_test +basic_rseq_op_test +param_test
tools/testing/selftests/rseq/Makefile new file +33 -0
@@ -0,0 +1,33 @@ +CFLAGS += -O2 -Wall -g -I./ -I../cpu-opv/ -I../../../../usr/include/ -L./ -Wl,-rpath=./ +LDLIBS += -lpthread + +# Own dependencies because we only want to build against 1st prerequisite, but +# still track changes to header files and depend on shared object. +OVERRIDE_TARGETS = 1 + +TEST_GEN_PROGS = basic_test basic_percpu_ops_test \ + param_test param_test_skip_fastpath \ + param_test_benchmark + +TEST_GEN_PROGS_EXTENDED = librseq.so libcpu-op.so + +TEST_PROGS = run_param_test.sh + +include ../lib.mk + +$(OUTPUT)/librseq.so: rseq.c rseq.h rseq-*.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/libcpu-op.so: ../cpu-opv/cpu-op.c ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) -shared -fPIC $< $(LDLIBS) -o $@ + +$(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) rseq.h rseq-*.h ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) $< $(LDLIBS) -lrseq -lcpu-op -o $@ + +$(OUTPUT)/param_test_skip_fastpath: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ + rseq.h rseq-*.h ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) -DSKIP_FASTPATH $< $(LDLIBS) -lrseq -lcpu-op -o $@ + +$(OUTPUT)/param_test_benchmark: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ + rseq.h rseq-*.h ../cpu-opv/cpu-op.h + $(CC) $(CFLAGS) -DBENCHMARK $< $(LDLIBS) -lrseq -lcpu-op -o $@
tools/testing/selftests/rseq/basic_percpu_ops_test.c new file +333 -0
333 lines not on this page · raw message
tools/testing/selftests/rseq/basic_test.c new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/rseq/param_test.c new file +1285 -0
1,285 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-arm.h new file +535 -0
535 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-ppc.h new file +567 -0
567 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-x86.h new file +898 -0
174 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.c new file +116 -0
116 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.h new file +154 -0
154 lines not on this page · raw message
tools/testing/selftests/rseq/run_param_test.sh new file +126 -0
126 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.11.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH update for 4.15 3/3] rseq: selftests: Provide self-tests (v4)
Quoted 124 lines · Mathieu Desnoyers
Implements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "
basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works."
basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness."
param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.A
run_param_test.shscript runs many variants of the parametrizable tests.As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the
cpu_opvsystem call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the__rseq_tablesection. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.The rseq library expose APIs that present the fast-path operations. The new from userspace is, e.g. for a counter increment:
cpu = rseq_cpu_start(); ret = rseq_addv(&data->c[cpu].count, 1, cpu); if (likely(!ret)) return 0; /* Success. */ do { cpu = rseq_current_cpu(); ret = cpu_op_addv(&data->c[cpu].count, 1, cpu); if (likely(!ret)) return 0; /* Success. */ } while (ret > 0 || errno == EAGAIN); perror("cpu_op_addv"); return -1; /* Unexpected error. */PowerPC tests have been implemented by Boqun Feng.
CC Russell King <linux-lFZ/[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <andi-Vw/[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <torvalds-de/[email protected]>CC Andrew Morton <akpm-de/[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>---
Changes since v1:
- Provide abort-ip signature: The abort-ip signature is located just before the abort-ip target. It is currently hardcoded, but a user-space application could use the__rseq_tableto iterate on all abort-ip targets and use a random value as signature if needed in the future.
- Addrseq_prepare_unload(): Libraries and JIT code using rseq critical sections need to issuerseq_prepare_unload()on each thread at least once before reclaim of structrseq_cs.
- Use initial-exec TLS model, non-weak symbol: The initial-exec model is signal-safe, whereas the global-dynamic model is not. Remove the "weak" symbol attribute from the__rseq_abiinrseq.c. The rseq.so library will have ownership of that symbol, and there is not reason for an application or user library to try to define that symbol. The expected use is to link against libreq.so, which owns and provide that symbol.
- Setcpu_idto -2 on register error
- Addrseq_lensyscall parameter,rseq_csversion
- Ensure disassember-friendly signature: x86 32/64 disassembler have a hard time decoding the instruction stream after a bad instruction. Use a nopl instruction to encode the signature. Suggested by Andy Lutomirski.
- Exercise parametrized tests variants in a shell scripts.
- Restartable sequences selftests: Remove use of event counter.
- Usecpu_id_startfield: With thecpu_id_startfield, the C preparation phase of the fast-path does not need to comparecpu_id< 0 anymore.
- Signal-safe registration and refcounting: Allow libraries using librseq.so to register it from signal handlers.
- UseOVERRIDE_TARGETSin makefile.
- Use "m" constraints forrseq_csfield.Changes since v2:
- Update based on Thomas Gleixner's comments.Changes since v3:
- Generateparam_test_skip_fastpathandparam_test_benchmarkwith -DSKIP_FASTPATH and -DBENCHMARK (respectively). Addparam_test_fastpathtorun_param_test.sh.--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 + tools/testing/selftests/rseq/Makefile | 33 + .../testing/selftests/rseq/basic_percpu_ops_test.c | 333 +++++ tools/testing/selftests/rseq/basic_test.c | 55 + tools/testing/selftests/rseq/param_test.c | 1285 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 535 ++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 567 +++++++++ tools/testing/selftests/rseq/rseq-x86.h | 898 ++++++++++++++ tools/testing/selftests/rseq/rseq.c | 116 ++ tools/testing/selftests/rseq/rseq.h | 154 +++ tools/testing/selftests/rseq/run_param_test.sh | 126 ++ 13 files changed, 4108 insertions(+) create mode 100644 tools/testing/selftests/rseq/.gitignore create mode 100644 tools/testing/selftests/rseq/Makefile create mode 100644 tools/testing/selftests/rseq/basic_percpu_ops_test.c
Quoted 9 lines
create mode 100644 tools/testing/selftests/rseq/basic_test.c create mode 100644 tools/testing/selftests/rseq/param_test.c create mode 100644 tools/testing/selftests/rseq/rseq-arm.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc.h create mode 100644 tools/testing/selftests/rseq/rseq-x86.h create mode 100644 tools/testing/selftests/rseq/rseq.c create mode 100644 tools/testing/selftests/rseq/rseq.h create mode 100755 tools/testing/selftests/rseq/run_param_test.sh
Looks good.
thanks,
-- Shuah
Mathieu Desnoyers
maintainerRe: [PATCH update for 4.15 3/3] rseq: selftests: Provide self-tests (v4)
Quoted 135 lines · ----- On Nov 22, 2017, at 10:23 AM, shuah [email protected] wrote:
Quoted 124 lines · Mathieu Desnoyers
Implements two basic tests of RSEQ functionality, and one more exhaustive parameterizable test.
The first, "
basic_test" only asserts that RSEQ works moderately correctly. E.g. that the CPUID pointer works."
basic_percpu_ops_test" is a slightly more "realistic" variant, implementing a few simple per-cpu operations and testing their correctness."
param_test" is a parametrizable restartable sequences test. See the "--help" output for usage.A
run_param_test.shscript runs many variants of the parametrizable tests.As part of those tests, a helper library "rseq" implements a user-space API around restartable sequences. It uses the
cpu_opvsystem call as fallback when single-stepped by a debugger. It exposes the instruction pointer addresses where the rseq assembly blocks begin and end, as well as the associated abort instruction pointer, in the__rseq_tablesection. This section allows debuggers may know where to place breakpoints when single-stepping through assembly blocks which may be aborted at any point by the kernel.The rseq library expose APIs that present the fast-path operations. The new from userspace is, e.g. for a counter increment:
cpu = rseq_cpu_start(); ret = rseq_addv(&data->c[cpu].count, 1, cpu); if (likely(!ret)) return 0; /* Success. */ do { cpu = rseq_current_cpu(); ret = cpu_op_addv(&data->c[cpu].count, 1, cpu); if (likely(!ret)) return 0; /* Success. */ } while (ret > 0 || errno == EAGAIN); perror("cpu_op_addv"); return -1; /* Unexpected error. */PowerPC tests have been implemented by Boqun Feng.
Signed-off-by Mathieu Desnoyers <[email protected]>CC Russell King <[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <[email protected]>CC Andrew Morton <[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>---
Changes since v1:
- Provide abort-ip signature: The abort-ip signature is located just before the abort-ip target. It is currently hardcoded, but a user-space application could use the__rseq_tableto iterate on all abort-ip targets and use a random value as signature if needed in the future.
- Addrseq_prepare_unload(): Libraries and JIT code using rseq critical sections need to issuerseq_prepare_unload()on each thread at least once before reclaim of structrseq_cs.
- Use initial-exec TLS model, non-weak symbol: The initial-exec model is signal-safe, whereas the global-dynamic model is not. Remove the "weak" symbol attribute from the__rseq_abiinrseq.c. The rseq.so library will have ownership of that symbol, and there is not reason for an application or user library to try to define that symbol. The expected use is to link against libreq.so, which owns and provide that symbol.
- Setcpu_idto -2 on register error
- Addrseq_lensyscall parameter,rseq_csversion
- Ensure disassember-friendly signature: x86 32/64 disassembler have a hard time decoding the instruction stream after a bad instruction. Use a nopl instruction to encode the signature. Suggested by Andy Lutomirski.
- Exercise parametrized tests variants in a shell scripts.
- Restartable sequences selftests: Remove use of event counter.
- Usecpu_id_startfield: With thecpu_id_startfield, the C preparation phase of the fast-path does not need to comparecpu_id< 0 anymore.
- Signal-safe registration and refcounting: Allow libraries using librseq.so to register it from signal handlers.
- UseOVERRIDE_TARGETSin makefile.
- Use "m" constraints forrseq_csfield.Changes since v2:
- Update based on Thomas Gleixner's comments.Changes since v3:
- Generateparam_test_skip_fastpathandparam_test_benchmarkwith -DSKIP_FASTPATH and -DBENCHMARK (respectively). Addparam_test_fastpathtorun_param_test.sh.--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rseq/.gitignore | 4 + tools/testing/selftests/rseq/Makefile | 33 + .../testing/selftests/rseq/basic_percpu_ops_test.c | 333 +++++ tools/testing/selftests/rseq/basic_test.c | 55 + tools/testing/selftests/rseq/param_test.c | 1285 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 535 ++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 567 +++++++++ tools/testing/selftests/rseq/rseq-x86.h | 898 ++++++++++++++ tools/testing/selftests/rseq/rseq.c | 116 ++ tools/testing/selftests/rseq/rseq.h | 154 +++ tools/testing/selftests/rseq/run_param_test.sh | 126 ++ 13 files changed, 4108 insertions(+) create mode 100644 tools/testing/selftests/rseq/.gitignore create mode 100644 tools/testing/selftests/rseq/Makefile create mode 100644 tools/testing/selftests/rseq/basic_percpu_ops_test.cQuoted 9 lines
create mode 100644 tools/testing/selftests/rseq/basic_test.c create mode 100644 tools/testing/selftests/rseq/param_test.c create mode 100644 tools/testing/selftests/rseq/rseq-arm.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc.h create mode 100644 tools/testing/selftests/rseq/rseq-x86.h create mode 100644 tools/testing/selftests/rseq/rseq.c create mode 100644 tools/testing/selftests/rseq/rseq.h create mode 100755 tools/testing/selftests/rseq/run_param_test.sh
Quoted 3 lines
Looks good.
Acked-by Shuah Khan <[email protected]>
Thanks the the reviews!
Mathieu
Quoted 3 lines
thanks,
-- Shuah
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [PATCH update for 4.15 1/3] selftests: lib.mk: Introduce OVERRIDE_TARGETS
In reply to Mathieu DesnoyersQuoted 3 lines · ----- On Nov 21, 2017, at 5:19 PM, Mathieu Desnoyers…
Introduce
OVERRIDE_TARGETSto allow tests to express dependencies on header files and .so, which require to override the selftests lib.mk targets.
Those 3 seftests update patches are still RFC (even though the subject tag is missing).
Thanks,
Mathieu
Quoted 52 lines
Signed-off-by Mathieu Desnoyers <[email protected]>CC Russell King <[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <[email protected]>CC Andrew Morton <[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>---
tools/testing/selftests/lib.mk| 4 ++++1 file changed, 4 insertions(+)tools/testing/selftests/lib.mk +0
-0@@ -105,6 +105,9 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)endif
tools/testing/selftests/lib.mk +3
-0+# Selftest makefiles can override those targets by setting +# OVERRIDE_TARGETS = 1. +ifeq ($(OVERRIDE_TARGETS),)$(OUTPUT)/%:%.c
$(LINK.c) $^ $(LDLIBS) -o $@tools/testing/selftests/lib.mk +0
-0@@ -113,5 +116,6 @@ $(OUTPUT)/%.o:%.S$(OUTPUT)/%:%.S
$(LINK.S) $^ $(LDLIBS) -o $@+endif
.PHONY:
run_testsall clean installemit_tests
--
2.11.0
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [PATCH update for 4.15 1/3] selftests: lib.mk: Introduce OVERRIDE_TARGETS
In reply to Mathieu DesnoyersQuoted 48 lines · Mathieu Desnoyers
Introduce
OVERRIDE_TARGETSto allow tests to express dependencies on header files and .so, which require to override the selftests lib.mk targets.CC Russell King <linux-lFZ/[email protected]>CC Catalin Marinas <[email protected]>CC Will Deacon <[email protected]>CC Thomas Gleixner <[email protected]>CC Paul Turner <[email protected]>CC Andrew Hunter <[email protected]>CC Peter Zijlstra <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <andi-Vw/[email protected]>CC Dave Watson <[email protected]>CC Chris Lameter <[email protected]>CC Ingo Molnar <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Ben Maurer <[email protected]>CC Steven Rostedt <[email protected]>CC "Paul E. McKenney" <[email protected]>CC Josh Triplett <[email protected]>CC Linus Torvalds <torvalds-de/[email protected]>CC Andrew Morton <akpm-de/[email protected]>CC Boqun Feng <[email protected]>CC Shuah Khan <[email protected]>--- tools/testing/selftests/lib.mk | 4 ++++ 1 file changed, 4 insertions(+)tools/testing/selftests/lib.mk +3
-0@@ -105,6 +105,9 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) endif +# Selftest makefiles can override those targets by setting +# OVERRIDE_TARGETS = 1. +ifeq ($(OVERRIDE_TARGETS),) $(OUTPUT)/%:%.c $(LINK.c) $^ $(LDLIBS) -o $@
tools/testing/selftests/lib.mk $(OUTPUT)/%.o:%.S · line 121
Quoted 6 lines
tools/testing/selftests/lib.mk +1
-0@@ -113,5 +116,6 @@ $(OUTPUT)/%.o:%.S $(OUTPUT)/%:%.S $(LINK.S) $^ $(LDLIBS) -o $@ +endif .PHONY: run_tests all clean install emit_tests
Thanks for splitting this patch. It will make it easier since it is kselftest framework change.
thanks,
-- Shuah