rseq uapi and selftest updates
Subsystems (3)
Mathieu Desnoyers
maintainercover rseq uapi and selftest updates
Update the rseq selftests to adapt to the userspace ABI chosen by glibc (will be released in February 2022). Provide a fallback implementation to work with older glibc as well.
Remove broken 32-bit little/big endian accessor fields in rseq uapi.
Integrate various fixes gathered from librseq, including work-around for known gcc and clang compiler bugs with asm goto.
The uplift to glibc's userspace ABI means we can now use %fs/%gs segment selectors directly in the inline assembler for accesses to the rseq thread area on x86.
Feedback is welcome,
Thanks,
Mathieu
Mathieu Desnoyers (15):
selftests/rseq: introduce own copy of rseq uapi header
rseq: Remove broken uapi field layout on 32-bit little endian selftests/rseq: Remove useless assignment to cpu variable selftests/rseq: Remove volatile from __rseq_abi selftests/rseq: Introduce rseq_get_abi() helper selftests/rseq: Introduce thread pointer getters selftests/rseq: Uplift rseq selftests for compatibility with
glibc-2.35
selftests/rseq: Fix ppc32: wrong rseq_cs 32-bit field pointer on big
endian
selftests/rseq: Fix ppc32 missing instruction selection "u" and "x"
for load/store
selftests/rseq: Fix ppc32 offsets by using long rather than off_t selftests/rseq: Fix warnings about #if checks of undefined tokens selftests/rseq: Remove arm/mips asm goto compiler work-around selftests/rseq: Fix: work-around asm goto compiler bugs selftests/rseq: x86-64: use %fs segment selector for accessing rseq
thread area
selftests/rseq: x86-32: use %gs segment selector for accessing rseq
thread area
include/uapi/linux/rseq.h | 17 +- tools/testing/selftests/rseq/Makefile | 2 +- .../selftests/rseq/basic_percpu_ops_test.c | 2 +- tools/testing/selftests/rseq/compiler.h | 30 +++ tools/testing/selftests/rseq/param_test.c | 8 +- tools/testing/selftests/rseq/rseq-abi.h | 151 +++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 110 +++++----- tools/testing/selftests/rseq/rseq-arm64.h | 79 +++++-- .../rseq/rseq-generic-thread-pointer.h | 25 +++ tools/testing/selftests/rseq/rseq-mips.h | 71 ++----- .../selftests/rseq/rseq-ppc-thread-pointer.h | 30 +++ tools/testing/selftests/rseq/rseq-ppc.h | 128 +++++++---- tools/testing/selftests/rseq/rseq-s390.h | 55 +++-- tools/testing/selftests/rseq/rseq-skip.h | 2 +- .../selftests/rseq/rseq-thread-pointer.h | 19 ++ .../selftests/rseq/rseq-x86-thread-pointer.h | 40 ++++ tools/testing/selftests/rseq/rseq-x86.h | 200 ++++++++++++------ tools/testing/selftests/rseq/rseq.c | 163 +++++++------- tools/testing/selftests/rseq/rseq.h | 29 ++- 19 files changed, 793 insertions(+), 368 deletions(-) create mode 100644 tools/testing/selftests/rseq/compiler.h create mode 100644 tools/testing/selftests/rseq/rseq-abi.h create mode 100644 tools/testing/selftests/rseq/rseq-generic-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-x86-thread-pointer.h
Signature · 1 line
2.17.1
Mathieu Desnoyers
maintainerRFC PATCH 1/15 selftests/rseq: introduce own copy of rseq uapi header
The Linux kernel rseq uapi header has a broken layout for the rseq_cs.ptr field on 32-bit little endian architectures. The entire rseq_cs.ptr field is planned for removal, leaving only the 64-bit rseq_cs.ptr64 field available.
Both glibc and librseq use their own copy of the Linux kernel uapi header, where they introduce proper union fields to access to the 32-bit low order bits of the rseq_cs pointer on 32-bit architectures.
Introduce a copy of the Linux kernel uapi headers in the Linux kernel selftests.
--- tools/testing/selftests/rseq/rseq-abi.h | 151 ++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.c | 14 +-- tools/testing/selftests/rseq/rseq.h | 10 +- 3 files changed, 161 insertions(+), 14 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-abi.h
tools/testing/selftests/rseq/rseq-abi.h new file +151 -0
@@ -0,0 +1,151 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef _RSEQ_ABI_H +#define _RSEQ_ABI_H + +/* +* rseq-abi.h +* +* Restartable sequences system call API +* +* Copyright (c) 2015-2022 Mathieu Desnoyers <[email protected]> +*/ + +#include <linux/types.h> +#include <asm/byteorder.h> + +enum rseq_abi_cpu_id_state { +RSEQ_ABI_CPU_ID_UNINITIALIZED =-1 , +RSEQ_ABI_CPU_ID_REGISTRATION_FAILED =-2 , +}; + +enum rseq_abi_flags { +RSEQ_ABI_FLAG_UNREGISTER = (1 <<0 ), +}; + +enum rseq_abi_cs_flags_bit { +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT_BIT =0 , +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT =1 , +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT =2 , +}; + +enum rseq_abi_cs_flags { +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT = + (1U <<RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT_BIT ), +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL = + (1U <<RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT ), +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE = + (1U <<RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT ), +}; + +/* +* struct rseq_abi_cs is aligned on 4 * 8 bytes to ensure it is always +* contained within a single cache-line. It is usually declared as +* link-time constant data. +*/ +struct rseq_abi_cs { +/* Version of this structure. */ +__u32 version; +/* enum rseq_abi_cs_flags */ +__u32 flags; +__u64 start_ip; +/* Offset from start_ip. */ +__u64 post_commit_offset; +__u64 abort_ip; +} __attribute__((aligned (4 *sizeof (__u64)))); + +/* +* struct rseq_abi is aligned on 4 * 8 bytes to ensure it is always +* contained within a single cache-line. +* +* A single struct rseq_abi per thread is allowed. +*/ +struct rseq_abi { +/* +* Restartable sequences cpu_id_start field. Updated by the +* kernel. Read by user-space with single-copy atomicity +* semantics. This field should only be read by the thread which +* registered this data structure. Aligned on 32-bit. Always +* contains a value in the range of possible CPUs, although the +* value may not be the actual current CPU (e.g. if rseq is not +* initialized). This CPU number value should always be compared +* against the value of the cpu_id field before performing a rseq +* commit or returning a value read from a data structure indexed +* using the cpu_id_start value. +*/ +__u32 cpu_id_start; +/* +* Restartable sequences cpu_id field. Updated by the kernel. +* Read by user-space with single-copy atomicity semantics. This +* field should only be read by the thread which registered this +* data structure. Aligned on 32-bit. Values +* RSEQ_CPU_ID_UNINITIALIZED and RSEQ_CPU_ID_REGISTRATION_FAILED +* have a special semantic: the former means "rseq uninitialized", +* and latter means "rseq initialization failed". This value is +* meant to be read within rseq critical sections and compared +* with the cpu_id_start value previously read, before performing +* the commit instruction, or read and compared with the +* cpu_id_start value before returning a value loaded from a data +* structure indexed using the cpu_id_start value. +*/ +__u32 cpu_id; +/* +* Restartable sequences rseq_cs field. +* +* Contains NULL when no critical section is active for the current +* thread, or holds a pointer to the currently active struct rseq_cs. +* +* Updated by user-space, which sets the address of the currently +* active rseq_cs at the beginning of assembly instruction sequence +* block, and set to NULL by the kernel when it restarts an assembly +* instruction sequence block, as well as when the kernel detects that +* it is preempting or delivering a signal outside of the range +* targeted by the rseq_cs. Also needs to be set to NULL by user-space +* before reclaiming memory that contains the targeted struct rseq_cs. +* +* Read and set by the kernel. Set by user-space with single-copy +* atomicity semantics. This field should only be updated by the +* thread which registered this data structure. Aligned on 64-bit. +*/ +union { +__u64 ptr64; + +/* +* The "arch" field provides architecture accessor for +* the ptr field based on architecture pointer size and +* endianness. +*/ +struct { +#ifdef __LP64__ +__u64 ptr; +#elif defined(__BYTE_ORDER) ? (__BYTE_ORDER == __BIG_ENDIAN) : defined(__BIG_ENDIAN) +__u32 padding;/* Initialized to zero. */ +__u32 ptr; +#else +__u32 ptr; +__u32 padding;/* Initialized to zero. */ +#endif + } arch; + } rseq_cs; + +/* +* Restartable sequences flags field. +* +* This field should only be updated by the thread which +* registered this data structure. Read by the kernel. +* Mainly used for single-stepping through rseq critical sections +* with debuggers. +* +* - RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT +* Inhibit instruction sequence block restart on preemption +* for this thread. +* - RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL +* Inhibit instruction sequence block restart on signal +* delivery for this thread. +* - RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE +* Inhibit instruction sequence block restart on migration for +* this thread. +*/ +__u32 flags; +} __attribute__((aligned (4 *sizeof (__u64)))); + +#endif /* _RSEQ_ABI_H */
tools/testing/selftests/rseq/rseq.c +7 -7
@@ -30,8 +30,8 @@#include "../kselftest.h" #include "rseq.h" -__threadvolatile struct rseq __rseq_abi = { - .cpu_id =RSEQ_CPU_ID_UNINITIALIZED , +__threadvolatile struct rseq_abi __rseq_abi = { + .cpu_id =RSEQ_ABI_CPU_ID_UNINITIALIZED , };/*
@@ -66,7 +66,7 @@ static void signal_restore(sigset_t oldset)abort (); } -static int sys_rseq (volatile struct rseq *rseq_abi,uint32_t rseq_len, +static int sys_rseq (volatile struct rseq_abi *rseq_abi,uint32_t rseq_len,int flags,uint32_t sig) {return syscall (__NR_rseq, rseq_abi, rseq_len, flags, sig);
@@ -86,13 +86,13 @@ int rseq_register_current_thread(void) }if (__rseq_refcount++) goto end; - rc =sys_rseq (&__rseq_abi,sizeof (struct rseq ),0 ,RSEQ_SIG ); + rc =sys_rseq (&__rseq_abi,sizeof (struct rseq_abi ),0 ,RSEQ_SIG );if (!rc) {assert (rseq_current_cpu_raw () >=0 ); goto end; }if (errno !=EBUSY ) - __rseq_abi.cpu_id =RSEQ_CPU_ID_REGISTRATION_FAILED ; + __rseq_abi.cpu_id =RSEQ_ABI_CPU_ID_REGISTRATION_FAILED ; ret =-1 ; __rseq_refcount--; end:
@@ -114,8 +114,8 @@ int rseq_unregister_current_thread(void) }if (--__rseq_refcount) goto end; - rc =sys_rseq (&__rseq_abi,sizeof (struct rseq ), -RSEQ_FLAG_UNREGISTER ,RSEQ_SIG ); + rc =sys_rseq (&__rseq_abi,sizeof (struct rseq_abi ), +RSEQ_ABI_FLAG_UNREGISTER ,RSEQ_SIG );if (!rc) goto end; __rseq_refcount =1 ;
tools/testing/selftests/rseq/rseq.h +3 -7
@@ -16,7 +16,7 @@#include <errno.h> #include <stdio.h> #include <stdlib.h> -#include <linux/rseq.h> +#include "rseq-abi.h" /* * Empty code injection macros, override when testing.
@@ -43,7 +43,7 @@#define RSEQ_INJECT_FAILED #endif -extern __threadvolatile struct rseq __rseq_abi; +extern __threadvolatile struct rseq_abi __rseq_abi;extern int __rseq_handled;#define rseq_likely (x) __builtin_expect(!!(x), 1)
@@ -139,11 +139,7 @@ static inline uint32_t rseq_current_cpu(void)static inline void rseq_clear_rseq_cs (void ) { -#ifdef __LP64__ - __rseq_abi.rseq_cs.ptr =0 ; -#else - __rseq_abi.rseq_cs.ptr.ptr32 =0 ; -#endif + __rseq_abi.rseq_cs.arch.ptr =0 ; }/*
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: introduce own copy of rseq uapi header
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5c105d55a9dc9e01535116ccfc26e703168a574f Gitweb: https://git.kernel.org/tip/5c105d55a9dc9e01535116ccfc26e703168a574f Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:39 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:33 +01:00
selftests/rseq: introduce own copy of rseq uapi header
The Linux kernel rseq uapi header has a broken layout for the rseq_cs.ptr field on 32-bit little endian architectures. The entire rseq_cs.ptr field is planned for removal, leaving only the 64-bit rseq_cs.ptr64 field available.
Both glibc and librseq use their own copy of the Linux kernel uapi header, where they introduce proper union fields to access to the 32-bit low order bits of the rseq_cs pointer on 32-bit architectures.
Introduce a copy of the Linux kernel uapi headers in the Linux kernel selftests.
--- tools/testing/selftests/rseq/rseq-abi.h | 151 +++++++++++++++++++++++- tools/testing/selftests/rseq/rseq.c | 14 +- tools/testing/selftests/rseq/rseq.h | 10 +-- 3 files changed, 161 insertions(+), 14 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-abi.h
tools/testing/selftests/rseq/rseq-abi.h new file +151 -0
@@ -0,0 +1,151 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef _RSEQ_ABI_H +#define _RSEQ_ABI_H + +/* +* rseq-abi.h +* +* Restartable sequences system call API +* +* Copyright (c) 2015-2022 Mathieu Desnoyers <[email protected]> +*/ + +#include <linux/types.h> +#include <asm/byteorder.h> + +enum rseq_abi_cpu_id_state { +RSEQ_ABI_CPU_ID_UNINITIALIZED =-1 , +RSEQ_ABI_CPU_ID_REGISTRATION_FAILED =-2 , +}; + +enum rseq_abi_flags { +RSEQ_ABI_FLAG_UNREGISTER = (1 <<0 ), +}; + +enum rseq_abi_cs_flags_bit { +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT_BIT =0 , +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT =1 , +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT =2 , +}; + +enum rseq_abi_cs_flags { +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT = + (1U <<RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT_BIT ), +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL = + (1U <<RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT ), +RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE = + (1U <<RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT ), +}; + +/* +* struct rseq_abi_cs is aligned on 4 * 8 bytes to ensure it is always +* contained within a single cache-line. It is usually declared as +* link-time constant data. +*/ +struct rseq_abi_cs { +/* Version of this structure. */ +__u32 version; +/* enum rseq_abi_cs_flags */ +__u32 flags; +__u64 start_ip; +/* Offset from start_ip. */ +__u64 post_commit_offset; +__u64 abort_ip; +} __attribute__((aligned (4 *sizeof (__u64)))); + +/* +* struct rseq_abi is aligned on 4 * 8 bytes to ensure it is always +* contained within a single cache-line. +* +* A single struct rseq_abi per thread is allowed. +*/ +struct rseq_abi { +/* +* Restartable sequences cpu_id_start field. Updated by the +* kernel. Read by user-space with single-copy atomicity +* semantics. This field should only be read by the thread which +* registered this data structure. Aligned on 32-bit. Always +* contains a value in the range of possible CPUs, although the +* value may not be the actual current CPU (e.g. if rseq is not +* initialized). This CPU number value should always be compared +* against the value of the cpu_id field before performing a rseq +* commit or returning a value read from a data structure indexed +* using the cpu_id_start value. +*/ +__u32 cpu_id_start; +/* +* Restartable sequences cpu_id field. Updated by the kernel. +* Read by user-space with single-copy atomicity semantics. This +* field should only be read by the thread which registered this +* data structure. Aligned on 32-bit. Values +* RSEQ_CPU_ID_UNINITIALIZED and RSEQ_CPU_ID_REGISTRATION_FAILED +* have a special semantic: the former means "rseq uninitialized", +* and latter means "rseq initialization failed". This value is +* meant to be read within rseq critical sections and compared +* with the cpu_id_start value previously read, before performing +* the commit instruction, or read and compared with the +* cpu_id_start value before returning a value loaded from a data +* structure indexed using the cpu_id_start value. +*/ +__u32 cpu_id; +/* +* Restartable sequences rseq_cs field. +* +* Contains NULL when no critical section is active for the current +* thread, or holds a pointer to the currently active struct rseq_cs. +* +* Updated by user-space, which sets the address of the currently +* active rseq_cs at the beginning of assembly instruction sequence +* block, and set to NULL by the kernel when it restarts an assembly +* instruction sequence block, as well as when the kernel detects that +* it is preempting or delivering a signal outside of the range +* targeted by the rseq_cs. Also needs to be set to NULL by user-space +* before reclaiming memory that contains the targeted struct rseq_cs. +* +* Read and set by the kernel. Set by user-space with single-copy +* atomicity semantics. This field should only be updated by the +* thread which registered this data structure. Aligned on 64-bit. +*/ +union { +__u64 ptr64; + +/* +* The "arch" field provides architecture accessor for +* the ptr field based on architecture pointer size and +* endianness. +*/ +struct { +#ifdef __LP64__ +__u64 ptr; +#elif defined(__BYTE_ORDER) ? (__BYTE_ORDER == __BIG_ENDIAN) : defined(__BIG_ENDIAN) +__u32 padding;/* Initialized to zero. */ +__u32 ptr; +#else +__u32 ptr; +__u32 padding;/* Initialized to zero. */ +#endif + } arch; + } rseq_cs; + +/* +* Restartable sequences flags field. +* +* This field should only be updated by the thread which +* registered this data structure. Read by the kernel. +* Mainly used for single-stepping through rseq critical sections +* with debuggers. +* +* - RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT +* Inhibit instruction sequence block restart on preemption +* for this thread. +* - RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL +* Inhibit instruction sequence block restart on signal +* delivery for this thread. +* - RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE +* Inhibit instruction sequence block restart on migration for +* this thread. +*/ +__u32 flags; +} __attribute__((aligned (4 *sizeof (__u64)))); + +#endif /* _RSEQ_ABI_H */
tools/testing/selftests/rseq/rseq.c +7 -7
@@ -30,8 +30,8 @@#include "../kselftest.h" #include "rseq.h" -__threadvolatile struct rseq __rseq_abi = { - .cpu_id =RSEQ_CPU_ID_UNINITIALIZED , +__threadvolatile struct rseq_abi __rseq_abi = { + .cpu_id =RSEQ_ABI_CPU_ID_UNINITIALIZED , };/*
@@ -66,7 +66,7 @@ static void signal_restore(sigset_t oldset)abort (); } -static int sys_rseq (volatile struct rseq *rseq_abi,uint32_t rseq_len, +static int sys_rseq (volatile struct rseq_abi *rseq_abi,uint32_t rseq_len,int flags,uint32_t sig) {return syscall (__NR_rseq, rseq_abi, rseq_len, flags, sig);
@@ -86,13 +86,13 @@ int rseq_register_current_thread(void) }if (__rseq_refcount++) goto end; - rc =sys_rseq (&__rseq_abi,sizeof (struct rseq ),0 ,RSEQ_SIG ); + rc =sys_rseq (&__rseq_abi,sizeof (struct rseq_abi ),0 ,RSEQ_SIG );if (!rc) {assert (rseq_current_cpu_raw () >=0 ); goto end; }if (errno !=EBUSY ) - __rseq_abi.cpu_id =RSEQ_CPU_ID_REGISTRATION_FAILED ; + __rseq_abi.cpu_id =RSEQ_ABI_CPU_ID_REGISTRATION_FAILED ; ret =-1 ; __rseq_refcount--; end:
@@ -114,8 +114,8 @@ int rseq_unregister_current_thread(void) }if (--__rseq_refcount) goto end; - rc =sys_rseq (&__rseq_abi,sizeof (struct rseq ), -RSEQ_FLAG_UNREGISTER ,RSEQ_SIG ); + rc =sys_rseq (&__rseq_abi,sizeof (struct rseq_abi ), +RSEQ_ABI_FLAG_UNREGISTER ,RSEQ_SIG );if (!rc) goto end; __rseq_refcount =1 ;
tools/testing/selftests/rseq/rseq.h +3 -7
@@ -16,7 +16,7 @@#include <errno.h> #include <stdio.h> #include <stdlib.h> -#include <linux/rseq.h> +#include "rseq-abi.h" /* * Empty code injection macros, override when testing.
@@ -43,7 +43,7 @@#define RSEQ_INJECT_FAILED #endif -extern __threadvolatile struct rseq __rseq_abi; +extern __threadvolatile struct rseq_abi __rseq_abi;extern int __rseq_handled;#define rseq_likely (x) __builtin_expect(!!(x), 1)
@@ -139,11 +139,7 @@ static inline uint32_t rseq_current_cpu(void)static inline void rseq_clear_rseq_cs (void ) { -#ifdef __LP64__ - __rseq_abi.rseq_cs.ptr =0 ; -#else - __rseq_abi.rseq_cs.ptr.ptr32 =0 ; -#endif + __rseq_abi.rseq_cs.arch.ptr =0 ; }/*
Mathieu Desnoyers
maintainerRFC PATCH 2/15 rseq: Remove broken uapi field layout on 32-bit little endian
In reply to Mathieu DesnoyersThe rseq rseq_cs.ptr.{ptr32,padding} uapi endianness handling is entirely wrong on 32-bit little endian: a preprocessor logic mistake wrongly uses the big endian field layout on 32-bit little endian architectures.
Fortunately, those ptr32 accessors were never used within the kernel, and only meant as a convenience for user-space.
Remove those and only leave the "ptr64" union field, as this is the only thing really needed to express the ABI. Document how 32-bit architectures are meant to interact with this "ptr64" union field.
--- include/uapi/linux/rseq.h | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-)
include/uapi/linux/rseq.h +4 -13
@@ -105,22 +105,13 @@ struct rseq {* Read and set by the kernel. Set by user-space with single-copy * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. +* +* 32-bit architectures should update the low order bits of the +* rseq_cs.ptr64 field, leaving the high order bits initialized +* to 0. */ union {__u64 ptr64; -#ifdef __LP64__ -__u64 ptr; -#else -struct { -#if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || defined(__BIG_ENDIAN) -__u32 padding;/* Initialized to zero. */ -__u32 ptr32; -#else /* LITTLE */ -__u32 ptr32; -__u32 padding;/* Initialized to zero. */ -#endif /* ENDIAN */ - } ptr; -#endif } rseq_cs;/*
Signature · 1 line
2.17.1
Re: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
Quoted 45 lines · Mathieu Desnoyers
The rseq rseq_cs.ptr.{ptr32,padding} uapi endianness handling is entirely wrong on 32-bit little endian: a preprocessor logic mistake wrongly uses the big endian field layout on 32-bit little endian architectures.
Fortunately, those ptr32 accessors were never used within the kernel, and only meant as a convenience for user-space.
Remove those and only leave the "ptr64" union field, as this is the only thing really needed to express the ABI. Document how 32-bit architectures are meant to interact with this "ptr64" union field.
Fixes ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes")Signed-off-by Mathieu Desnoyers <[email protected]>Cc Florian Weimer <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Boqun Feng <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Watson <[email protected]>Cc Paul Turner <[email protected]>Cc Andrew Morton <[email protected]>Cc Russell King <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Andi Kleen <[email protected]>Cc Christian Brauner <[email protected]>Cc Ben Maurer <[email protected]>Cc Steven Rostedt <[email protected]>Cc Josh Triplett <[email protected]>Cc Linus Torvalds <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>Cc Michael Kerrisk <[email protected]>Cc Joel Fernandes <[email protected]>Cc Paul E. McKenney <[email protected]>--- include/uapi/linux/rseq.h | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-)include/uapi/linux/rseq.h +0
-0
include/uapi/linux/rseq.h struct rseq { · line 113
Quoted 9 lines
include/uapi/linux/rseq.h +4
-0@@ -105,22 +105,13 @@ struct rseq {* Read and set by the kernel. Set by user-space with single-copy * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. +* +* 32-bit architectures should update the low order bits of the +* rseq_cs.ptr64 field, leaving the high order bits initialized +* to 0. */ union {
A bit unfortunate we seem to have to keep the union around even though it's just one field now.
Mathieu Desnoyers
maintainerRe: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
Quoted 51 lines · ----- On Jan 25, 2022, at 7:21 AM, Christian Brauner [email protected] wrote:
Quoted 50 lines · Mathieu Desnoyers
The rseq rseq_cs.ptr.{ptr32,padding} uapi endianness handling is entirely wrong on 32-bit little endian: a preprocessor logic mistake wrongly uses the big endian field layout on 32-bit little endian architectures.
Fortunately, those ptr32 accessors were never used within the kernel, and only meant as a convenience for user-space.
Remove those and only leave the "ptr64" union field, as this is the only thing really needed to express the ABI. Document how 32-bit architectures are meant to interact with this "ptr64" union field.
Fixes ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, updateincludes")
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Florian Weimer <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Boqun Feng <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Watson <[email protected]>Cc Paul Turner <[email protected]>Cc Andrew Morton <[email protected]>Cc Russell King <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Andi Kleen <[email protected]>Cc Christian Brauner <[email protected]>Cc Ben Maurer <[email protected]>Cc Steven Rostedt <[email protected]>Cc Josh Triplett <[email protected]>Cc Linus Torvalds <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>Cc Michael Kerrisk <[email protected]>Cc Joel Fernandes <[email protected]>Cc Paul E. McKenney <[email protected]>--- include/uapi/linux/rseq.h | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-)include/uapi/linux/rseq.h +1
-0@@ -105,22 +105,13 @@ struct rseq {* Read and set by the kernel. Set by user-space with single-copy * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. +*
Quoted 8 lines
include/uapi/linux/rseq.h struct rseq { · line 113
Quoted 5 lines
include/uapi/linux/rseq.h +3
-0+* 32-bit architectures should update the low order bits of the +* rseq_cs.ptr64 field, leaving the high order bits initialized +* to 0. */ union {A bit unfortunate we seem to have to keep the union around even though it's just one field now.
Well, as far as the user-space projects that I know of which use rseq are concerned (glibc, librseq, tcmalloc), those end up with their own copy of the uapi header anyway to deal with the big/little endian field on 32-bit. So I'm very much open to remove the union if we accept that this uapi header is really just meant to express the ABI and is not expected to be used as an API by user-space.
That would mean we also bring a uapi header copy into the kernel rseq selftests as well to minimize the gap between librseq and the kernel sefltests (the kernel sefltests pretty much include a copy of librseq for convenience. librseq is maintained out of tree).
Thoughts ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
Quoted 1 line · ----- On Jan 25, 2022, at 9:41 AM, Mathieu Desnoyers…
----- On Jan 25, 2022, at 7:21 AM, Christian Brauner [email protected] wrote:
[...]
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/linux/rseq.h | 17 ++++-------------
[...]
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
union {
A bit unfortunate we seem to have to keep the union around even though it's just one field now.
Well, as far as the user-space projects that I know of which use rseq are concerned (glibc, librseq, tcmalloc), those end up with their own copy of the uapi header anyway to deal with the big/little endian field on 32-bit. So I'm very much open to remove the union if we accept that this uapi header is really just meant to express the ABI and is not expected to be used as an API by user-space.
Quoted 6 lines
That would mean we also bring a uapi header copy into the kernel rseq selftests as well to minimize the gap between librseq and the kernel sefltests (the kernel sefltests pretty much include a copy of librseq for convenience. librseq is maintained out of tree).
Thoughts ?
Actually, if we go ahead and remove the union, and replace:
struct rseq {
union {
__u64 ptr64; } rseq_cs;
[...]
} v;
by:
struct rseq {
__u64 rseq_cs;
} v;
expressions such as these are unchanged:
- sizeof(v.rseq_cs),
- &v.rseq_cs,
- __alignof__(v.rseq_cs),
- offsetof(struct rseq, rseq_cs).
So users of the uapi rseq.h (as an API) can still use rseq_abi->rseq_cs before and after the change.
Based on this, I am inclined to remove the union, and just make the rseq_cs field a __u64.
Any objections ?
Thanks,
Mathieu
Quoted 10 lines
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
Quoted 47 lines · Mathieu Desnoyers
Quoted 1 line · ----- On Jan 25, 2022, at 9:41 AM, Mathieu Desnoyers…
----- On Jan 25, 2022, at 7:21 AM, Christian Brauner [email protected] wrote:
[...]
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/linux/rseq.h | 17 ++++-------------[...]
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
union {
A bit unfortunate we seem to have to keep the union around even though it's just one field now.
Well, as far as the user-space projects that I know of which use rseq are concerned (glibc, librseq, tcmalloc), those end up with their own copy of the uapi header anyway to deal with the big/little endian field on 32-bit. So I'm very much open to remove the union if we accept that this uapi header is really just meant to express the ABI and is not expected to be used as an API by user-space.
Quoted 6 lines
That would mean we also bring a uapi header copy into the kernel rseq selftests as well to minimize the gap between librseq and the kernel sefltests (the kernel sefltests pretty much include a copy of librseq for convenience. librseq is maintained out of tree).
Thoughts ?
Actually, if we go ahead and remove the union, and replace:
struct rseq {
union { __u64 ptr64;}
rseq_cs;
[...]
} v;by:
struct rseq {
__u64rseq_cs;
} v;expressions such as these are unchanged:
-
sizeof(v.rseq_cs),
- &v.rseq_cs,
-__alignof__(v.rseq_cs),
-offsetof(struct rseq, rseq_cs).
Quoted 7 lines
So users of the uapi
rseq.h(as an API) can still userseq_abi->rseq_csbefore and after the change.Based on this, I am inclined to remove the union, and just make the
rseq_csfield a__u64.Any objections ?
I do like it fwiw. But since I haven't been heavily involved in the userspace usage of this I can't speak confidently to the regression potential of a change like this. But I would think that we should risk it instead of dragging a pointless union around forever.
Re: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
* Christian Brauner:
Quoted 7 lines · Mathieu Desnoyers
Quoted 6 lines
So users of the uapi
rseq.h(as an API) can still userseq_abi->rseq_csbefore and after the change.Based on this, I am inclined to remove the union, and just make the
rseq_csfield a__u64.
Quoted 6 lines
Quoted 1 line
Any objections ?
I do like it fwiw. But since I haven't been heavily involved in the userspace usage of this I can't speak confidently to the regression potential of a change like this. But I would think that we should risk it instead of dragging a pointless union around forever.
I don't think glibc needs changes for this, it will keep building just fine. We'll need to adjust the included kernel header fragment that could be used by applications in some corner cases, but that's it.
RE: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
In reply to Mathieu DesnoyersFrom: Mathieu Desnoyers
Quoted 44 lines
Sent: 25 January 2022 19:01
Quoted 1 line · ----- On Jan 25, 2022, at 9:41 AM, Mathieu Desnoyers…
----- On Jan 25, 2022, at 7:21 AM, Christian Brauner [email protected] wrote:
[...]
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/linux/rseq.h | 17 ++++-------------[...]
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
union {
A bit unfortunate we seem to have to keep the union around even though it's just one field now.
Well, as far as the user-space projects that I know of which use rseq are concerned (glibc, librseq, tcmalloc), those end up with their own copy of the uapi header anyway to deal with the big/little endian field on 32-bit. So I'm very much open to remove the union if we accept that this uapi header is really just meant to express the ABI and is not expected to be used as an API by user-space.
Quoted 6 lines
That would mean we also bring a uapi header copy into the kernel rseq selftests as well to minimize the gap between librseq and the kernel sefltests (the kernel sefltests pretty much include a copy of librseq for convenience. librseq is maintained out of tree).
Thoughts ?
Actually, if we go ahead and remove the union, and replace:
struct rseq {
union { __u64 ptr64;}
rseq_cs;
[...]
} v;by:
struct rseq {
__u64rseq_cs;
} v;expressions such as these are unchanged:
Quoted 7 lines
-
sizeof(v.rseq_cs),
- &v.rseq_cs,
-__alignof__(v.rseq_cs),
-offsetof(struct rseq, rseq_cs).So users of the uapi
rseq.h(as an API) can still userseq_abi->rseq_csbefore and after the change.
But:
v.rseq_cs.ptr_64 = (uintptr_t)&foo;
is broken.
Quoted 2 lines
Based on this, I am inclined to remove the union, and just make the
rseq_csfield a__u64.
It really is a shame that you can't do:
void *rseq_cs __attribute__((size(8)));
and have the compiler just DTRT on 32bit systems.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Mathieu Desnoyers
maintainerRe: [RFC PATCH 02/15] rseq: Remove broken uapi field layout on 32-bit little endian
Quoted 51 lines · ----- On Jan 26, 2022, at 12:16 PM, David Laight [email protected] wrote:
From: Mathieu Desnoyers
Quoted 50 lines
Sent: 25 January 2022 19:01
----- On Jan 25, 2022, at 9:41 AM, Mathieu Desnoyers
Quoted 1 line · [email protected] wrote:
----- On Jan 25, 2022, at 7:21 AM, Christian Brauner [email protected] wrote:
[...]
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/linux/rseq.h | 17 ++++-------------[...]
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
union {
A bit unfortunate we seem to have to keep the union around even though it's just one field now.
Well, as far as the user-space projects that I know of which use rseq are concerned (glibc, librseq, tcmalloc), those end up with their own copy of the uapi header anyway to deal with the big/little endian field on 32-bit. So I'm very much open to remove the union if we accept that this uapi header is really just meant to express the ABI and is not expected to be used as an API by user-space.
Quoted 6 lines
That would mean we also bring a uapi header copy into the kernel rseq selftests as well to minimize the gap between librseq and the kernel sefltests (the kernel sefltests pretty much include a copy of librseq for convenience. librseq is maintained out of tree).
Thoughts ?
Actually, if we go ahead and remove the union, and replace:
struct rseq {
union { __u64 ptr64;}
rseq_cs;
[...]
} v;by:
struct rseq {
__u64rseq_cs;
} v;expressions such as these are unchanged:
-
sizeof(v.rseq_cs),
- &v.rseq_cs,
-__alignof__(v.rseq_cs),
-offsetof(struct rseq, rseq_cs).
Quoted 6 lines
Quoted 2 lines
So users of the uapi
rseq.h(as an API) can still userseq_abi->rseq_csbefore and after the change.But:
v.rseq_cs.ptr_64 = (uintptr_t)&foo;is broken.
True. But v.rseq_cs.ptr (on 64-bit) and v.rseq_cs.ptr.ptr32 (on 32-bit) are also broken with the planned field removal. So how is the v.rseq_cs_ptr64 situation different ?
My thinking here is that it does not matter if we break compilation for some users of the uapi as an API as long as the ABI stays the same, especially considering that all known users implement their own copy of the header.
I suspect that as far as the API is concerned, it is nice that we have at least one way to access the field which works both before and after the change. Simply using "v.rseq_cs" works both before/after for all use-cases that seem to matter here.
Quoted 8 lines
Quoted 4 lines
Based on this, I am inclined to remove the union, and just make the
rseq_csfield
a__u64.It really is a shame that you can't do:
void *rseq_cs __attribute__((size(8)));and have the compiler just DTRT on 32bit systems.
Indeed, the "size" directive appears to be ignored by the compiler.
Thanks,
Mathieu
Quoted 7 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRFC PATCH v2 1/1 rseq: Remove broken uapi field layout on 32-bit little endian
The rseq rseq_cs.ptr.{ptr32,padding} uapi endianness handling is entirely wrong on 32-bit little endian: a preprocessor logic mistake wrongly uses the big endian field layout on 32-bit little endian architectures.
Fortunately, those ptr32 accessors were never used within the kernel, and only meant as a convenience for user-space.
Remove those and replace the whole rseq_cs union by a __u64 type, as this is the only thing really needed to express the ABI. Document how 32-bit architectures are meant to interact with this field.
--- include/uapi/linux/rseq.h | 20 ++++---------------- kernel/rseq.c | 8 ++++---- 2 files changed, 8 insertions(+), 20 deletions(-)
include/uapi/linux/rseq.h +4 -16
@@ -105,23 +105,11 @@ struct rseq {* Read and set by the kernel. Set by user-space with single-copy * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. +* +* 32-bit architectures should update the low order bits of the +* rseq_cs field, leaving the high order bits initialized to 0. */ -union { -__u64 ptr64; -#ifdef __LP64__ -__u64 ptr; -#else -struct { -#if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || defined(__BIG_ENDIAN) -__u32 padding;/* Initialized to zero. */ -__u32 ptr32; -#else /* LITTLE */ -__u32 ptr32; -__u32 padding;/* Initialized to zero. */ -#endif /* ENDIAN */ - } ptr; -#endif - } rseq_cs; +__u64 rseq_cs;/* * Restartable sequences flags field.
kernel/rseq.c +4 -4
@@ -128,10 +128,10 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)int ret;#ifdef CONFIG_64BIT -if (get_user (ptr, &t->rseq->rseq_cs.ptr64)) +if (get_user (ptr, &t->rseq->rseq_cs))return -EFAULT ;#else -if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr))) +if (copy_from_user (&ptr, &t->rseq->rseq_cs,sizeof (ptr)))return -EFAULT ;#endif if (!ptr) {
@@ -217,9 +217,9 @@ static int clear_rseq_cs(struct task_struct *t)* Set rseq_cs to NULL. */ #ifdef CONFIG_64BIT -return put_user (0UL , &t->rseq->rseq_cs.ptr64); +return put_user (0UL , &t->rseq->rseq_cs);#else -if (clear_user (&t->rseq->rseq_cs.ptr64,sizeof (t->rseq->rseq_cs.ptr64))) +if (clear_user (&t->rseq->rseq_cs,sizeof (t->rseq->rseq_cs)))return -EFAULT ;return 0 ;#endif
Signature · 1 line
2.17.1
Re: [RFC PATCH v2] rseq: Remove broken uapi field layout on 32-bit little endian
Quoted 36 lines · Mathieu Desnoyers
The rseq rseq_cs.ptr.{ptr32,padding} uapi endianness handling is entirely wrong on 32-bit little endian: a preprocessor logic mistake wrongly uses the big endian field layout on 32-bit little endian architectures.
Fortunately, those ptr32 accessors were never used within the kernel, and only meant as a convenience for user-space.
Remove those and replace the whole
rseq_csunion by a__u64type, as this is the only thing really needed to express the ABI. Document how 32-bit architectures are meant to interact with this field.Fixes ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes")Signed-off-by Mathieu Desnoyers <[email protected]>Cc Florian Weimer <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Boqun Feng <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Watson <[email protected]>Cc Paul Turner <[email protected]>Cc Andrew Morton <[email protected]>Cc Russell King <[email protected]>Cc "H . Peter Anvin" <[email protected]>Cc Andi Kleen <[email protected]>Cc Christian Brauner <[email protected]>Cc Ben Maurer <[email protected]>Cc Steven Rostedt <[email protected]>Cc Josh Triplett <[email protected]>Cc Linus Torvalds <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>Cc Michael Kerrisk <[email protected]>Cc Joel Fernandes <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 1 line
---
Looks way cleaner now! Fwiw,
Quoted 68 lines
include/uapi/linux/rseq.h | 20 ++++---------------- kernel/rseq.c | 8 ++++---- 2 files changed, 8 insertions(+), 20 deletions(-)include/uapi/linux/rseq.h +4
-16@@ -105,23 +105,11 @@ struct rseq {* Read and set by the kernel. Set by user-space with single-copy * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. +* +* 32-bit architectures should update the low order bits of the +* rseq_cs field, leaving the high order bits initialized to 0. */ -union { -__u64 ptr64; -#ifdef __LP64__ -__u64 ptr; -#else -struct { -#if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || defined(__BIG_ENDIAN) -__u32 padding;/* Initialized to zero. */ -__u32 ptr32; -#else /* LITTLE */ -__u32 ptr32; -__u32 padding;/* Initialized to zero. */ -#endif /* ENDIAN */ - } ptr; -#endif - } rseq_cs; +__u64 rseq_cs;/* * Restartable sequences flags field. kernel/rseq.c +4
-4@@ -128,10 +128,10 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)int ret;#ifdef CONFIG_64BIT -if (get_user (ptr, &t->rseq->rseq_cs.ptr64)) +if (get_user (ptr, &t->rseq->rseq_cs))return -EFAULT ;#else -if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr))) +if (copy_from_user (&ptr, &t->rseq->rseq_cs,sizeof (ptr)))return -EFAULT ;#endif if (!ptr) {@@ -217,9 +217,9 @@ static int clear_rseq_cs(struct task_struct *t)* Set rseq_cs to NULL. */ #ifdef CONFIG_64BIT -return put_user (0UL , &t->rseq->rseq_cs.ptr64); +return put_user (0UL , &t->rseq->rseq_cs);#else -if (clear_user (&t->rseq->rseq_cs.ptr64,sizeof (t->rseq->rseq_cs.ptr64))) +if (clear_user (&t->rseq->rseq_cs,sizeof (t->rseq->rseq_cs)))return -EFAULT ;return 0 ;#endif Signature · 1 line
2.17.1
[tip: sched/core] rseq: Remove broken uapi field layout on 32-bit little endian
In reply to Mathieu DesnoyersThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: bfdf4e6208051ed7165b2e92035b4bf11f43eb63 Gitweb: https://git.kernel.org/tip/bfdf4e6208051ed7165b2e92035b4bf11f43eb63 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 27 Jan 2022 10:27:20 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:34 +01:00
rseq: Remove broken uapi field layout on 32-bit little endian
The rseq rseq_cs.ptr.{ptr32,padding} uapi endianness handling is entirely wrong on 32-bit little endian: a preprocessor logic mistake wrongly uses the big endian field layout on 32-bit little endian architectures.
Fortunately, those ptr32 accessors were never used within the kernel, and only meant as a convenience for user-space.
Remove those and replace the whole rseq_cs union by a __u64 type, as this is the only thing really needed to express the ABI. Document how 32-bit architectures are meant to interact with this field.
--- include/uapi/linux/rseq.h | 20 ++++---------------- kernel/rseq.c | 8 ++++---- 2 files changed, 8 insertions(+), 20 deletions(-)
include/uapi/linux/rseq.h +4 -16
@@ -105,23 +105,11 @@ struct rseq {* Read and set by the kernel. Set by user-space with single-copy * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. +* +* 32-bit architectures should update the low order bits of the +* rseq_cs field, leaving the high order bits initialized to 0. */ -union { -__u64 ptr64; -#ifdef __LP64__ -__u64 ptr; -#else -struct { -#if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || defined(__BIG_ENDIAN) -__u32 padding;/* Initialized to zero. */ -__u32 ptr32; -#else /* LITTLE */ -__u32 ptr32; -__u32 padding;/* Initialized to zero. */ -#endif /* ENDIAN */ - } ptr; -#endif - } rseq_cs; +__u64 rseq_cs;/* * Restartable sequences flags field.
kernel/rseq.c +4 -4
@@ -128,10 +128,10 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)int ret;#ifdef CONFIG_64BIT -if (get_user (ptr, &t->rseq->rseq_cs.ptr64)) +if (get_user (ptr, &t->rseq->rseq_cs))return -EFAULT ;#else -if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr))) +if (copy_from_user (&ptr, &t->rseq->rseq_cs,sizeof (ptr)))return -EFAULT ;#endif if (!ptr) {
@@ -217,9 +217,9 @@ static int clear_rseq_cs(struct task_struct *t)* Set rseq_cs to NULL. */ #ifdef CONFIG_64BIT -return put_user (0UL , &t->rseq->rseq_cs.ptr64); +return put_user (0UL , &t->rseq->rseq_cs);#else -if (clear_user (&t->rseq->rseq_cs.ptr64,sizeof (t->rseq->rseq_cs.ptr64))) +if (clear_user (&t->rseq->rseq_cs,sizeof (t->rseq->rseq_cs)))return -EFAULT ;return 0 ;#endif
Mathieu Desnoyers
maintainerRFC PATCH 3/15 selftests/rseq: Remove useless assignment to cpu variable
In reply to Mathieu Desnoyers--- tools/testing/selftests/rseq/param_test.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
tools/testing/selftests/rseq/param_test.c +1 -3
@@ -368,9 +368,7 @@ void *test_percpu_spinlock_thread(void *arg)abort (); reps = thread_data->reps;for (i =0 ; i < reps; i++) { -int cpu =rseq_cpu_start (); - - cpu =rseq_this_cpu_lock (&data->lock); +int cpu =rseq_this_cpu_lock (&data->lock); data->c[cpu].count++;rseq_percpu_unlock (&data->lock, cpu);#ifndef BENCHMARK
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Remove useless assignment to cpu variable
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 930378d056eac2c96407b02aafe4938d0ac9cc37 Gitweb: https://git.kernel.org/tip/930378d056eac2c96407b02aafe4938d0ac9cc37 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:41 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:34 +01:00
selftests/rseq: Remove useless assignment to cpu variable
--- tools/testing/selftests/rseq/param_test.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
tools/testing/selftests/rseq/param_test.c +1 -3
@@ -368,9 +368,7 @@ void *test_percpu_spinlock_thread(void *arg)abort (); reps = thread_data->reps;for (i =0 ; i < reps; i++) { -int cpu =rseq_cpu_start (); - - cpu =rseq_this_cpu_lock (&data->lock); +int cpu =rseq_this_cpu_lock (&data->lock); data->c[cpu].count++;rseq_percpu_unlock (&data->lock, cpu);#ifndef BENCHMARK
Mathieu Desnoyers
maintainerRFC PATCH 4/15 selftests/rseq: Remove volatile from __rseq_abi
In reply to Mathieu DesnoyersThis is done in preparation for the selftest uplift to become compatible with glibc-2.35.
All accesses to the __rseq_abi fields are volatile, but remove the volatile from the TLS variable declaration, otherwise we are stuck with volatile for the upcoming rseq_get_abi() helper.
--- tools/testing/selftests/rseq/rseq.c | 4 ++-- tools/testing/selftests/rseq/rseq.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
tools/testing/selftests/rseq/rseq.c +2 -2
@@ -30,7 +30,7 @@#include "../kselftest.h" #include "rseq.h" -__threadvolatile struct rseq_abi __rseq_abi = { +__threadstruct rseq_abi __rseq_abi = { .cpu_id =RSEQ_ABI_CPU_ID_UNINITIALIZED , };
@@ -92,7 +92,7 @@ int rseq_register_current_thread(void) goto end; }if (errno !=EBUSY ) - __rseq_abi.cpu_id =RSEQ_ABI_CPU_ID_REGISTRATION_FAILED ; +RSEQ_WRITE_ONCE (__rseq_abi.cpu_id,RSEQ_ABI_CPU_ID_REGISTRATION_FAILED ); ret =-1 ; __rseq_refcount--; end:
tools/testing/selftests/rseq/rseq.h +2 -2
@@ -43,7 +43,7 @@#define RSEQ_INJECT_FAILED #endif -extern __threadvolatile struct rseq_abi __rseq_abi; +extern __threadstruct rseq_abi __rseq_abi;extern int __rseq_handled;#define rseq_likely (x) __builtin_expect(!!(x), 1)
@@ -139,7 +139,7 @@ static inline uint32_t rseq_current_cpu(void)static inline void rseq_clear_rseq_cs (void ) { - __rseq_abi.rseq_cs.arch.ptr =0 ; +RSEQ_WRITE_ONCE (__rseq_abi.rseq_cs.arch.ptr,0 ); }/*
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Remove volatile from __rseq_abi
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 94b80a19ebfe347a01301d750040a61c38200e2b Gitweb: https://git.kernel.org/tip/94b80a19ebfe347a01301d750040a61c38200e2b Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:42 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:34 +01:00
selftests/rseq: Remove volatile from __rseq_abi
This is done in preparation for the selftest uplift to become compatible with glibc-2.35.
All accesses to the __rseq_abi fields are volatile, but remove the volatile from the TLS variable declaration, otherwise we are stuck with volatile for the upcoming rseq_get_abi() helper.
--- tools/testing/selftests/rseq/rseq.c | 4 ++-- tools/testing/selftests/rseq/rseq.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
tools/testing/selftests/rseq/rseq.c +2 -2
@@ -30,7 +30,7 @@#include "../kselftest.h" #include "rseq.h" -__threadvolatile struct rseq_abi __rseq_abi = { +__threadstruct rseq_abi __rseq_abi = { .cpu_id =RSEQ_ABI_CPU_ID_UNINITIALIZED , };
@@ -92,7 +92,7 @@ int rseq_register_current_thread(void) goto end; }if (errno !=EBUSY ) - __rseq_abi.cpu_id =RSEQ_ABI_CPU_ID_REGISTRATION_FAILED ; +RSEQ_WRITE_ONCE (__rseq_abi.cpu_id,RSEQ_ABI_CPU_ID_REGISTRATION_FAILED ); ret =-1 ; __rseq_refcount--; end:
tools/testing/selftests/rseq/rseq.h +2 -2
@@ -43,7 +43,7 @@#define RSEQ_INJECT_FAILED #endif -extern __threadvolatile struct rseq_abi __rseq_abi; +extern __threadstruct rseq_abi __rseq_abi;extern int __rseq_handled;#define rseq_likely (x) __builtin_expect(!!(x), 1)
@@ -139,7 +139,7 @@ static inline uint32_t rseq_current_cpu(void)static inline void rseq_clear_rseq_cs (void ) { - __rseq_abi.rseq_cs.arch.ptr =0 ; +RSEQ_WRITE_ONCE (__rseq_abi.rseq_cs.arch.ptr,0 ); }/*
Mathieu Desnoyers
maintainerRFC PATCH 5/15 selftests/rseq: Introduce rseq_get_abi() helper
In reply to Mathieu DesnoyersThis is done in preparation for the selftest uplift to become compatible with glibc-2.35.
glibc-2.35 exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the kernel selftests initially expected.
Introduce a rseq_get_abi() helper, initially using the __rseq_abi TLS variable, in preparation for changing this userspace ABI for one which is compatible with glibc-2.35.
Note that the __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.
--- tools/testing/selftests/rseq/rseq-arm.h | 32 +++++++++++------------ tools/testing/selftests/rseq/rseq-arm64.h | 32 +++++++++++------------ tools/testing/selftests/rseq/rseq-mips.h | 32 +++++++++++------------ tools/testing/selftests/rseq/rseq-ppc.h | 32 +++++++++++------------ tools/testing/selftests/rseq/rseq-s390.h | 24 ++++++++--------- tools/testing/selftests/rseq/rseq-x86.h | 30 ++++++++++----------- tools/testing/selftests/rseq/rseq.h | 11 +++++--- 7 files changed, 99 insertions(+), 94 deletions(-)
tools/testing/selftests/rseq/rseq-arm.h +16 -16
@@ -185,8 +185,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -255,8 +255,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -316,8 +316,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -381,8 +381,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -457,8 +457,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -537,8 +537,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -657,8 +657,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -782,8 +782,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-arm64.h +16 -16
@@ -230,8 +230,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -287,8 +287,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [expectnot] "r" (expectnot), [load] "Qo" (*load),
@@ -337,8 +337,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [count] "r" (count)RSEQ_INJECT_INPUT
@@ -388,8 +388,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -447,8 +447,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -508,8 +508,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [expect] "r" (expect), [v2] "Qo" (*v2),
@@ -569,8 +569,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -629,8 +629,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
tools/testing/selftests/rseq/rseq-mips.h +16 -16
@@ -190,8 +190,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -258,8 +258,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -319,8 +319,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -382,8 +382,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -456,8 +456,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -532,8 +532,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -649,8 +649,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -771,8 +771,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-ppc.h +16 -16
@@ -235,8 +235,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -301,8 +301,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -359,8 +359,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -419,8 +419,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -489,8 +489,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -560,8 +560,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -635,8 +635,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -711,8 +711,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-s390.h +12 -12
@@ -165,8 +165,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -233,8 +233,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -288,8 +288,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -347,8 +347,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -426,8 +426,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -534,8 +534,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-x86.h +15 -15
@@ -141,7 +141,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -207,7 +207,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -258,7 +258,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [count] "er" (count)
@@ -314,7 +314,7 @@ int rseq_offset_deref_addv(intptr_t *ptr, off_t off, intptr_t inc, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [ptr] "m" (*ptr), [off] "er" (off),
@@ -372,7 +372,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -449,7 +449,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -555,7 +555,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -719,7 +719,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -785,7 +785,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -836,7 +836,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [count] "ir" (count)
@@ -894,7 +894,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* try store input */ [v2] "m" (*v2), [newv2] "m" (newv2),
@@ -962,7 +962,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -1032,7 +1032,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -1142,7 +1142,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
@@ -1255,7 +1255,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
tools/testing/selftests/rseq/rseq.h +8 -3
@@ -46,6 +46,11 @@extern __threadstruct rseq_abi __rseq_abi;extern int __rseq_handled; +static inline struct rseq_abi *rseq_get_abi (void ) +{ +return &__rseq_abi; +} +#define rseq_likely (x) __builtin_expect(!!(x), 1)#define rseq_unlikely (x) __builtin_expect(!!(x), 0)#define rseq_barrier () __asm__ __volatile__("" : : : "memory")
@@ -108,7 +113,7 @@ int32_t rseq_fallback_current_cpu(void);*/ static inline int32_t rseq_current_cpu_raw (void ) { -return RSEQ_ACCESS_ONCE (__rseq_abi.cpu_id); +return RSEQ_ACCESS_ONCE (rseq_get_abi ()->cpu_id); }/*
@@ -124,7 +129,7 @@ static inline int32_t rseq_current_cpu_raw(void)*/ static inline uint32_t rseq_cpu_start (void ) { -return RSEQ_ACCESS_ONCE (__rseq_abi.cpu_id_start); +return RSEQ_ACCESS_ONCE (rseq_get_abi ()->cpu_id_start); }static inline uint32_t rseq_current_cpu (void )
@@ -139,7 +144,7 @@ static inline uint32_t rseq_current_cpu(void)static inline void rseq_clear_rseq_cs (void ) { -RSEQ_WRITE_ONCE (__rseq_abi.rseq_cs.arch.ptr,0 ); +RSEQ_WRITE_ONCE (rseq_get_abi ()->rseq_cs.arch.ptr,0 ); }/*
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Introduce rseq_get_abi() helper
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e546cd48ccc456074ddb8920732aef4af65d7ca7 Gitweb: https://git.kernel.org/tip/e546cd48ccc456074ddb8920732aef4af65d7ca7 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:43 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:35 +01:00
selftests/rseq: Introduce rseq_get_abi() helper
This is done in preparation for the selftest uplift to become compatible with glibc-2.35.
glibc-2.35 exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the kernel selftests initially expected.
Introduce a rseq_get_abi() helper, initially using the __rseq_abi TLS variable, in preparation for changing this userspace ABI for one which is compatible with glibc-2.35.
Note that the __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.
--- tools/testing/selftests/rseq/rseq-arm.h | 32 +++++++++++----------- tools/testing/selftests/rseq/rseq-arm64.h | 32 +++++++++++----------- tools/testing/selftests/rseq/rseq-mips.h | 32 +++++++++++----------- tools/testing/selftests/rseq/rseq-ppc.h | 32 +++++++++++----------- tools/testing/selftests/rseq/rseq-s390.h | 24 ++++++++--------- tools/testing/selftests/rseq/rseq-x86.h | 30 ++++++++++----------- tools/testing/selftests/rseq/rseq.h | 11 +++++--- 7 files changed, 99 insertions(+), 94 deletions(-)
tools/testing/selftests/rseq/rseq-arm.h +16 -16
@@ -185,8 +185,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -255,8 +255,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -316,8 +316,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -381,8 +381,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -457,8 +457,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -537,8 +537,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -657,8 +657,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -782,8 +782,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-arm64.h +16 -16
@@ -230,8 +230,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -287,8 +287,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [expectnot] "r" (expectnot), [load] "Qo" (*load),
@@ -337,8 +337,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [count] "r" (count)RSEQ_INJECT_INPUT
@@ -388,8 +388,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -447,8 +447,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -508,8 +508,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "Qo" (*v), [expect] "r" (expect), [v2] "Qo" (*v2),
@@ -569,8 +569,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -629,8 +629,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
tools/testing/selftests/rseq/rseq-mips.h +16 -16
@@ -190,8 +190,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -258,8 +258,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -319,8 +319,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -382,8 +382,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -456,8 +456,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -532,8 +532,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"5:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -649,8 +649,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -771,8 +771,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,"8:\n\t" : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-ppc.h +16 -16
@@ -235,8 +235,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -301,8 +301,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -359,8 +359,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -419,8 +419,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -489,8 +489,8 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -560,8 +560,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -635,8 +635,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -711,8 +711,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-s390.h +12 -12
@@ -165,8 +165,8 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -233,8 +233,8 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -288,8 +288,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -347,8 +347,8 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -426,8 +426,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -534,8 +534,8 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-x86.h +15 -15
@@ -141,7 +141,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -207,7 +207,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -258,7 +258,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [count] "er" (count)
@@ -314,7 +314,7 @@ int rseq_offset_deref_addv(intptr_t *ptr, off_t off, intptr_t inc, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [ptr] "m" (*ptr), [off] "er" (off),
@@ -372,7 +372,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -449,7 +449,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -555,7 +555,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -719,7 +719,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -785,7 +785,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -836,7 +836,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [count] "ir" (count)
@@ -894,7 +894,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* try store input */ [v2] "m" (*v2), [newv2] "m" (newv2),
@@ -962,7 +962,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -1032,7 +1032,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -1142,7 +1142,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
@@ -1255,7 +1255,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (&__rseq_abi), + [rseq_abi] "r" (rseq_get_abi()), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
tools/testing/selftests/rseq/rseq.h +8 -3
@@ -46,6 +46,11 @@extern __threadstruct rseq_abi __rseq_abi;extern int __rseq_handled; +static inline struct rseq_abi *rseq_get_abi (void ) +{ +return &__rseq_abi; +} +#define rseq_likely (x) __builtin_expect(!!(x), 1)#define rseq_unlikely (x) __builtin_expect(!!(x), 0)#define rseq_barrier () __asm__ __volatile__("" : : : "memory")
@@ -108,7 +113,7 @@ int32_t rseq_fallback_current_cpu(void);*/ static inline int32_t rseq_current_cpu_raw (void ) { -return RSEQ_ACCESS_ONCE (__rseq_abi.cpu_id); +return RSEQ_ACCESS_ONCE (rseq_get_abi ()->cpu_id); }/*
@@ -124,7 +129,7 @@ static inline int32_t rseq_current_cpu_raw(void)*/ static inline uint32_t rseq_cpu_start (void ) { -return RSEQ_ACCESS_ONCE (__rseq_abi.cpu_id_start); +return RSEQ_ACCESS_ONCE (rseq_get_abi ()->cpu_id_start); }static inline uint32_t rseq_current_cpu (void )
@@ -139,7 +144,7 @@ static inline uint32_t rseq_current_cpu(void)static inline void rseq_clear_rseq_cs (void ) { -RSEQ_WRITE_ONCE (__rseq_abi.rseq_cs.arch.ptr,0 ); +RSEQ_WRITE_ONCE (rseq_get_abi ()->rseq_cs.arch.ptr,0 ); }/*
Mathieu Desnoyers
maintainerRFC PATCH 6/15 selftests/rseq: Introduce thread pointer getters
In reply to Mathieu DesnoyersThis is done in preparation for the selftest uplift to become compatible with glibc-2.35.
glibc-2.35 exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer.
The toolchains do not implement accessing the thread pointer on all architectures. Provide thread pointer getters for ppc and x86 which lack (or lacked until recently) toolchain support.
--- .../rseq/rseq-generic-thread-pointer.h | 25 ++++++++++++ .../selftests/rseq/rseq-ppc-thread-pointer.h | 30 ++++++++++++++ .../selftests/rseq/rseq-thread-pointer.h | 19 +++++++++ .../selftests/rseq/rseq-x86-thread-pointer.h | 40 +++++++++++++++++++ 4 files changed, 114 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-generic-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-x86-thread-pointer.h
tools/testing/selftests/rseq/rseq-generic-thread-pointer.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-generic-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_GENERIC_THREAD_POINTER +#define _RSEQ_GENERIC_THREAD_POINTER + +#ifdef __cplusplus +extern "C" { +#endif + +/* Use gcc builtin thread pointer. */ +static inline void *rseq_thread_pointer (void ) +{ +return __builtin_thread_pointer (); +} + +#ifdef __cplusplus +} +#endif + +#endif
tools/testing/selftests/rseq/rseq-ppc-thread-pointer.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-ppc-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_PPC_THREAD_POINTER +#define _RSEQ_PPC_THREAD_POINTER + +#ifdef __cplusplus +extern "C" { +#endif + +static inline void *rseq_thread_pointer (void ) +{ +#ifdef __powerpc64__ + registervoid *__result asm ("r13" ); +#else + registervoid *__result asm ("r2" ); +#endif + asm ("" :"=r" (__result)); +return __result; +} + +#ifdef __cplusplus +} +#endif + +#endif
tools/testing/selftests/rseq/rseq-thread-pointer.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_THREAD_POINTER +#define _RSEQ_THREAD_POINTER + +#if defined(__x86_64__) || defined(__i386__) +#include "rseq-x86-thread-pointer.h" +#elif defined(__PPC__) +#include "rseq-ppc-thread-pointer.h" +#else +#include "rseq-generic-thread-pointer.h" +#endif + +#endif
tools/testing/selftests/rseq/rseq-x86-thread-pointer.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-x86-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_X86_THREAD_POINTER +#define _RSEQ_X86_THREAD_POINTER + +#include <features.h> + +#ifdef __cplusplus +extern "C" { +#endif + +#if __GNUC_PREREQ (11 ,1 ) +static inline void *rseq_thread_pointer (void ) +{ +return __builtin_thread_pointer (); +} +#else +static inline void *rseq_thread_pointer (void ) +{ +void *__result; + +# ifdef __x86_64__ + __asm__ ("mov %%fs:0, %0" :"=r" (__result)); +# else + __asm__ ("mov %%gs:0, %0" :"=r" (__result)); +# endif +return __result; +} +#endif /* !GCC 11 */ + +#ifdef __cplusplus +} +#endif + +#endif
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Introduce thread pointer getters
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 886ddfba933f5ce9d76c278165d834d114ba4ffc Gitweb: https://git.kernel.org/tip/886ddfba933f5ce9d76c278165d834d114ba4ffc Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:44 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:35 +01:00
selftests/rseq: Introduce thread pointer getters
This is done in preparation for the selftest uplift to become compatible with glibc-2.35.
glibc-2.35 exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer.
The toolchains do not implement accessing the thread pointer on all architectures. Provide thread pointer getters for ppc and x86 which lack (or lacked until recently) toolchain support.
--- tools/testing/selftests/rseq/rseq-generic-thread-pointer.h | 25 ++++- tools/testing/selftests/rseq/rseq-ppc-thread-pointer.h | 30 +++++- tools/testing/selftests/rseq/rseq-thread-pointer.h | 19 +++- tools/testing/selftests/rseq/rseq-x86-thread-pointer.h | 40 +++++++- 4 files changed, 114 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-generic-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-ppc-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-thread-pointer.h create mode 100644 tools/testing/selftests/rseq/rseq-x86-thread-pointer.h
tools/testing/selftests/rseq/rseq-generic-thread-pointer.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-generic-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_GENERIC_THREAD_POINTER +#define _RSEQ_GENERIC_THREAD_POINTER + +#ifdef __cplusplus +extern "C" { +#endif + +/* Use gcc builtin thread pointer. */ +static inline void *rseq_thread_pointer (void ) +{ +return __builtin_thread_pointer (); +} + +#ifdef __cplusplus +} +#endif + +#endif
tools/testing/selftests/rseq/rseq-ppc-thread-pointer.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-ppc-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_PPC_THREAD_POINTER +#define _RSEQ_PPC_THREAD_POINTER + +#ifdef __cplusplus +extern "C" { +#endif + +static inline void *rseq_thread_pointer (void ) +{ +#ifdef __powerpc64__ + registervoid *__result asm ("r13" ); +#else + registervoid *__result asm ("r2" ); +#endif + asm ("" :"=r" (__result)); +return __result; +} + +#ifdef __cplusplus +} +#endif + +#endif
tools/testing/selftests/rseq/rseq-thread-pointer.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_THREAD_POINTER +#define _RSEQ_THREAD_POINTER + +#if defined(__x86_64__) || defined(__i386__) +#include "rseq-x86-thread-pointer.h" +#elif defined(__PPC__) +#include "rseq-ppc-thread-pointer.h" +#else +#include "rseq-generic-thread-pointer.h" +#endif + +#endif
tools/testing/selftests/rseq/rseq-x86-thread-pointer.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq-x86-thread-pointer.h +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef _RSEQ_X86_THREAD_POINTER +#define _RSEQ_X86_THREAD_POINTER + +#include <features.h> + +#ifdef __cplusplus +extern "C" { +#endif + +#if __GNUC_PREREQ (11 ,1 ) +static inline void *rseq_thread_pointer (void ) +{ +return __builtin_thread_pointer (); +} +#else +static inline void *rseq_thread_pointer (void ) +{ +void *__result; + +# ifdef __x86_64__ + __asm__ ("mov %%fs:0, %0" :"=r" (__result)); +# else + __asm__ ("mov %%gs:0, %0" :"=r" (__result)); +# endif +return __result; +} +#endif /* !GCC 11 */ + +#ifdef __cplusplus +} +#endif + +#endif
Mathieu Desnoyers
maintainerRFC PATCH 7/15 selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35
In reply to Mathieu Desnoyersglibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.
The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.
Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:
- librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.
- librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.
- Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.
--- tools/testing/selftests/rseq/Makefile | 2 +- tools/testing/selftests/rseq/rseq.c | 161 ++++++++++++-------------- tools/testing/selftests/rseq/rseq.h | 13 ++- 3 files changed, 88 insertions(+), 88 deletions(-)
tools/testing/selftests/rseq/Makefile +1 -1
@@ -6,7 +6,7 @@ endif CFLAGS += -O2 -Wall -g -I./ -I../../../../usr/include/ -L$(OUTPUT) -Wl,-rpath=./ \ $(CLANG_FLAGS) -LDLIBS += -lpthread +LDLIBS += -lpthread -ldl # Own dependencies because we only want to build against 1st prerequisite, but # still track changes to header files and depend on shared object.
tools/testing/selftests/rseq/rseq.c +77 -84
207 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.h +10 -3
@@ -43,12 +43,19 @@#define RSEQ_INJECT_FAILED #endif -extern __threadstruct rseq_abi __rseq_abi; -extern int __rseq_handled; +#include "rseq-thread-pointer.h" + +/* Offset from the thread pointer to the rseq area. */ +extern int rseq_offset; +/* Size of the registered rseq area. 0 if the registration was +unsuccessful. */ +extern unsigned int rseq_size; +/* Flags used during rseq registration. */ +extern unsigned int rseq_flags;static inline struct rseq_abi *rseq_get_abi (void ) { -return &__rseq_abi; +return (struct rseq_abi *) ((uintptr_t )rseq_thread_pointer () + rseq_offset); }#define rseq_likely (x) __builtin_expect(!!(x), 1)
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 233e667e1ae3e348686bd9dd0172e62a09d852e1 Gitweb: https://git.kernel.org/tip/233e667e1ae3e348686bd9dd0172e62a09d852e1 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:45 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:35 +01:00
selftests/rseq: Uplift rseq selftests for compatibility with glibc-2.35
glibc-2.35 (upcoming release date 2022-02-01) exposes the rseq per-thread data in the TCB, accessible at an offset from the thread pointer, rather than through an actual Thread-Local Storage (TLS) variable, as the Linux kernel selftests initially expected.
The __rseq_abi TLS and glibc-2.35's ABI for per-thread data cannot actively coexist in a process, because the kernel supports only a single rseq registration per thread.
Here is the scheme introduced to ensure selftests can work both with an older glibc and with glibc-2.35+:
- librseq exposes its own "rseq_offset, rseq_size, rseq_flags" ABI.
- librseq queries for glibc rseq ABI (__rseq_offset, __rseq_size, __rseq_flags) using dlsym() in a librseq library constructor. If those are found, copy their values into rseq_offset, rseq_size, and rseq_flags.
- Else, if those glibc symbols are not found, handle rseq registration from librseq and use its own IE-model TLS to implement the rseq ABI per-thread storage.
--- tools/testing/selftests/rseq/Makefile | 2 +- tools/testing/selftests/rseq/rseq.c | 161 +++++++++++-------------- tools/testing/selftests/rseq/rseq.h | 13 +- 3 files changed, 88 insertions(+), 88 deletions(-)
tools/testing/selftests/rseq/Makefile +1 -1
@@ -6,7 +6,7 @@ endif CFLAGS += -O2 -Wall -g -I./ -I../../../../usr/include/ -L$(OUTPUT) -Wl,-rpath=./ \ $(CLANG_FLAGS) -LDLIBS += -lpthread +LDLIBS += -lpthread -ldl # Own dependencies because we only want to build against 1st prerequisite, but # still track changes to header files and depend on shared object.
tools/testing/selftests/rseq/rseq.c +77 -84
207 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.h +10 -3
@@ -43,12 +43,19 @@#define RSEQ_INJECT_FAILED #endif -extern __threadstruct rseq_abi __rseq_abi; -extern int __rseq_handled; +#include "rseq-thread-pointer.h" + +/* Offset from the thread pointer to the rseq area. */ +extern int rseq_offset; +/* Size of the registered rseq area. 0 if the registration was +unsuccessful. */ +extern unsigned int rseq_size; +/* Flags used during rseq registration. */ +extern unsigned int rseq_flags;static inline struct rseq_abi *rseq_get_abi (void ) { -return &__rseq_abi; +return (struct rseq_abi *) ((uintptr_t )rseq_thread_pointer () + rseq_offset); }#define rseq_likely (x) __builtin_expect(!!(x), 1)
Mathieu Desnoyers
maintainerRFC PATCH 8/15 selftests/rseq: Fix ppc32: wrong rseq_cs 32-bit field pointer on big endian
In reply to Mathieu Desnoyersppc32 incorrectly uses padding as rseq_cs pointer field. Fix this by using the rseq_cs.arch.ptr field.
Use this field across all architectures.
--- tools/testing/selftests/rseq/rseq-arm.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-arm64.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-mips.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-ppc.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-s390.h | 12 ++++++------ 5 files changed, 38 insertions(+), 38 deletions(-)
tools/testing/selftests/rseq/rseq-arm.h +8 -8
@@ -186,7 +186,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -256,7 +256,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -317,7 +317,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -382,7 +382,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -458,7 +458,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -538,7 +538,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -658,7 +658,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -783,7 +783,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-arm64.h +8 -8
@@ -231,7 +231,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -288,7 +288,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [expectnot] "r" (expectnot), [load] "Qo" (*load),
@@ -338,7 +338,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [count] "r" (count)RSEQ_INJECT_INPUT
@@ -389,7 +389,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -448,7 +448,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -509,7 +509,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [expect] "r" (expect), [v2] "Qo" (*v2),
@@ -570,7 +570,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -630,7 +630,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
tools/testing/selftests/rseq/rseq-mips.h +8 -8
@@ -191,7 +191,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -259,7 +259,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -320,7 +320,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -383,7 +383,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -457,7 +457,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -533,7 +533,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -650,7 +650,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -772,7 +772,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-ppc.h +8 -8
@@ -236,7 +236,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -302,7 +302,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -360,7 +360,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -420,7 +420,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -490,7 +490,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -561,7 +561,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -636,7 +636,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -712,7 +712,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-s390.h +6 -6
@@ -166,7 +166,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -234,7 +234,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -289,7 +289,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -348,7 +348,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -427,7 +427,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -535,7 +535,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Fix ppc32: wrong rseq_cs 32-bit field pointer on big endian
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 24d1136a29da5953de5c0cbc6c83eb62a1e0bf14 Gitweb: https://git.kernel.org/tip/24d1136a29da5953de5c0cbc6c83eb62a1e0bf14 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:46 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:35 +01:00
selftests/rseq: Fix ppc32: wrong rseq_cs 32-bit field pointer on big endian
ppc32 incorrectly uses padding as rseq_cs pointer field. Fix this by using the rseq_cs.arch.ptr field.
Use this field across all architectures.
--- tools/testing/selftests/rseq/rseq-arm.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-arm64.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-mips.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-ppc.h | 16 ++++++++-------- tools/testing/selftests/rseq/rseq-s390.h | 12 ++++++------ 5 files changed, 38 insertions(+), 38 deletions(-)
tools/testing/selftests/rseq/rseq-arm.h +8 -8
@@ -186,7 +186,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -256,7 +256,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -317,7 +317,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -382,7 +382,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -458,7 +458,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -538,7 +538,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -658,7 +658,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -783,7 +783,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-arm64.h +8 -8
@@ -231,7 +231,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -288,7 +288,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [expectnot] "r" (expectnot), [load] "Qo" (*load),
@@ -338,7 +338,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [count] "r" (count)RSEQ_INJECT_INPUT
@@ -389,7 +389,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -448,7 +448,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -509,7 +509,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "Qo" (*v), [expect] "r" (expect), [v2] "Qo" (*v2),
@@ -570,7 +570,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
@@ -630,7 +630,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [expect] "r" (expect), [v] "Qo" (*v), [newv] "r" (newv),
tools/testing/selftests/rseq/rseq-mips.h +8 -8
@@ -191,7 +191,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -259,7 +259,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -320,7 +320,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [count] "Ir" (count)RSEQ_INJECT_INPUT
@@ -383,7 +383,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -457,7 +457,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -533,7 +533,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -650,7 +650,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -772,7 +772,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-ppc.h +8 -8
@@ -236,7 +236,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -302,7 +302,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -360,7 +360,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -420,7 +420,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -490,7 +490,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -561,7 +561,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -636,7 +636,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
@@ -712,7 +712,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
tools/testing/selftests/rseq/rseq-s390.h +6 -6
@@ -166,7 +166,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -234,7 +234,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -289,7 +289,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [count] "r" (count)
@@ -348,7 +348,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -427,7 +427,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -535,7 +535,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
Mathieu Desnoyers
maintainerRFC PATCH 9/15 selftests/rseq: Fix ppc32 missing instruction selection "u" and "x" for load/store
In reply to Mathieu DesnoyersBuilding the rseq basic test with
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)
Target: powerpc-linux-gnu
leads to these errors:
/tmp/ccieEWxU.s: Assembler messages:/tmp/ccieEWxU.s:118: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:118: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:121: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:121: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:626: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:626: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:629: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:629: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:735: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:735: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:738: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:738: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:741: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:741: Error: junk at end of line: `,8' Makefile:581: recipe for target 'basic_percpu_ops_test.o' failed
Based on discussion with Linux powerpc maintainers and review of the use of the "m" operand in powerpc kernel code, add the missing %Un%Xn (where n is operand number) to the lwz, stw, ld, and std instructions when used with "m" operands.
Using "WORD" to mean either a 32-bit or 64-bit type depending on the architecture is misleading. The term "WORD" really means a 32-bit type in both 32-bit and 64-bit powerpc assembler. The intent here is to wrap load/store to intptr_t into common macros for both 32-bit and 64-bit.
Rename the macros with a RSEQ_ prefix, and use the terms "INT" for always 32-bit type, and "LONG" for architecture bitness-sized type.
--- tools/testing/selftests/rseq/rseq-ppc.h | 55 +++++++++++++------------ 1 file changed, 28 insertions(+), 27 deletions(-)
tools/testing/selftests/rseq/rseq-ppc.h +28 -27
@@ -47,10 +47,13 @@ do { \#ifdef __PPC64__ -#define STORE_WORD "std " -#define LOAD_WORD "ld " -#define LOADX_WORD "ldx " -#define CMP_WORD "cmpd " +#define RSEQ_STORE_LONG (arg) "std%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* To memory ("m" constraint) */ +#define RSEQ_STORE_INT (arg) "stw%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* To memory ("m" constraint) */ +#define RSEQ_LOAD_LONG (arg) "ld%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* From memory ("m" constraint) */ +#define RSEQ_LOAD_INT (arg) "lwz%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* From memory ("m" constraint) */ +#define RSEQ_LOADX_LONG "ldx "/* From base register ("b" constraint) */ +#define RSEQ_CMP_LONG "cmpd " +#define RSEQ_CMP_LONG_INT "cmpdi "#define __RSEQ_ASM_DEFINE_TABLE (label, version, flags, \ start_ip, post_commit_offset, abort_ip) \
@@ -89,10 +92,13 @@ do { \#else /* #ifdef __PPC64__ */ -#define STORE_WORD "stw " -#define LOAD_WORD "lwz " -#define LOADX_WORD "lwzx " -#define CMP_WORD "cmpw " +#define RSEQ_STORE_LONG (arg) "stw%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* To memory ("m" constraint) */ +#define RSEQ_STORE_INT (arg) RSEQ_STORE_LONG(arg)/* To memory ("m" constraint) */ +#define RSEQ_LOAD_LONG (arg) "lwz%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* From memory ("m" constraint) */ +#define RSEQ_LOAD_INT (arg) RSEQ_LOAD_LONG(arg)/* From memory ("m" constraint) */ +#define RSEQ_LOADX_LONG "lwzx "/* From base register ("b" constraint) */ +#define RSEQ_CMP_LONG "cmpw " +#define RSEQ_CMP_LONG_INT "cmpwi "#define __RSEQ_ASM_DEFINE_TABLE (label, version, flags, \ start_ip, post_commit_offset, abort_ip) \
@@ -125,7 +131,7 @@ do { \ RSEQ_INJECT_ASM(1) \ "lis %%r17, (" __rseq_str(cs_label) ")@ha\n\t" \ "addi %%r17, %%r17, (" __rseq_str(cs_label) ")@l\n\t" \ - "stw %%r17, %[" __rseq_str(rseq_cs) "]\n\t" \ + RSEQ_STORE_INT(rseq_cs) "%%r17, %[" __rseq_str(rseq_cs) "]\n\t" \ __rseq_str(label) ":\n\t"#endif /* #ifdef __PPC64__ */
@@ -136,7 +142,7 @@ do { \#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ RSEQ_INJECT_ASM(2) \ - "lwz %%r17, %[" __rseq_str(current_cpu_id) "]\n\t" \ + RSEQ_LOAD_INT(current_cpu_id) "%%r17, %[" __rseq_str(current_cpu_id) "]\n\t" \ "cmpw cr7, %[" __rseq_str(cpu_id) "], %%r17\n\t" \ "bne- cr7, " __rseq_str(label) "\n\t"
@@ -153,25 +159,25 @@ do { \* RSEQ_ASM_OP_* (else): doesn't have hard-code registers(unless cr7) */ #define RSEQ_ASM_OP_CMPEQ (var, expect, label) \ - LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ - CMP_WORD "cr7, %%r17, %[" __rseq_str(expect) "]\n\t" \ + RSEQ_LOAD_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t" \ + RSEQ_CMP_LONG "cr7, %%r17, %[" __rseq_str(expect) "]\n\t" \ "bne- cr7, " __rseq_str(label) "\n\t"#define RSEQ_ASM_OP_CMPNE (var, expectnot, label) \ - LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ - CMP_WORD "cr7, %%r17, %[" __rseq_str(expectnot) "]\n\t" \ + RSEQ_LOAD_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t" \ + RSEQ_CMP_LONG "cr7, %%r17, %[" __rseq_str(expectnot) "]\n\t" \ "beq- cr7, " __rseq_str(label) "\n\t"#define RSEQ_ASM_OP_STORE (value, var) \ - STORE_WORD "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" + RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t"/* Load @var to r17 */ #define RSEQ_ASM_OP_R_LOAD (var) \ - LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" + RSEQ_LOAD_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t"/* Store r17 to @var */ #define RSEQ_ASM_OP_R_STORE (var) \ - STORE_WORD "%%r17, %[" __rseq_str(var) "]\n\t" + RSEQ_STORE_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t"/* Add @count to r17 */ #define RSEQ_ASM_OP_R_ADD (count) \
@@ -179,11 +185,11 @@ do { \/* Load (r17 + voffp) to r17 */ #define RSEQ_ASM_OP_R_LOADX (voffp) \ - LOADX_WORD "%%r17, %[" __rseq_str(voffp) "], %%r17\n\t" + RSEQ_LOADX_LONG "%%r17, %[" __rseq_str(voffp) "], %%r17\n\t"/* TODO: implement a faster memcpy. */ #define RSEQ_ASM_OP_R_MEMCPY () \ - "cmpdi %%r19, 0\n\t" \ + RSEQ_CMP_LONG_INT "%%r19, 0\n\t" \ "beq 333f\n\t" \ "addi %%r20, %%r20, -1\n\t" \ "addi %%r21, %%r21, -1\n\t" \
@@ -191,16 +197,16 @@ do { \"lbzu %%r18, 1(%%r20)\n\t" \"stbu %%r18, 1(%%r21)\n\t" \"addi %%r19, %%r19, -1\n\t" \ -"cmpdi %%r19, 0\n\t" \ +RSEQ_CMP_LONG_INT "%%r19, 0\n\t" \"bne 222b\n\t" \ "333:\n\t" \#define RSEQ_ASM_OP_R_FINAL_STORE (var, post_commit_label) \ - STORE_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ + RSEQ_STORE_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t" \ __rseq_str(post_commit_label) ":\n\t"#define RSEQ_ASM_OP_FINAL_STORE (value, var, post_commit_label) \ - STORE_WORD "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \ + RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \ __rseq_str(post_commit_label) ":\n\t" static inline __attribute__((always_inline))
@@ -743,9 +749,4 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, #endif } -#undef STORE_WORD -#undef LOAD_WORD -#undef LOADX_WORD -#undef CMP_WORD -#endif /* !RSEQ_SKIP_FASTPATH */
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Fix ppc32 missing instruction selection "u" and "x" for load/store
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: de6b52a21420a18dc8a36438d581efd1313d5fe3 Gitweb: https://git.kernel.org/tip/de6b52a21420a18dc8a36438d581efd1313d5fe3 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:47 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:36 +01:00
selftests/rseq: Fix ppc32 missing instruction selection "u" and "x" for load/store
Building the rseq basic test with
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)
Target: powerpc-linux-gnu
leads to these errors:
/tmp/ccieEWxU.s: Assembler messages:/tmp/ccieEWxU.s:118: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:118: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:121: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:121: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:626: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:626: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:629: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:629: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:735: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:735: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:738: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:738: Error: junk at end of line: ,8' /tmp/ccieEWxU.s:741: Error: syntax error; found ,', expected (' /tmp/ccieEWxU.s:741: Error: junk at end of line: `,8' Makefile:581: recipe for target 'basic_percpu_ops_test.o' failed
Based on discussion with Linux powerpc maintainers and review of the use of the "m" operand in powerpc kernel code, add the missing %Un%Xn (where n is operand number) to the lwz, stw, ld, and std instructions when used with "m" operands.
Using "WORD" to mean either a 32-bit or 64-bit type depending on the architecture is misleading. The term "WORD" really means a 32-bit type in both 32-bit and 64-bit powerpc assembler. The intent here is to wrap load/store to intptr_t into common macros for both 32-bit and 64-bit.
Rename the macros with a RSEQ_ prefix, and use the terms "INT" for always 32-bit type, and "LONG" for architecture bitness-sized type.
--- tools/testing/selftests/rseq/rseq-ppc.h | 55 ++++++++++++------------ 1 file changed, 28 insertions(+), 27 deletions(-)
tools/testing/selftests/rseq/rseq-ppc.h +28 -27
@@ -47,10 +47,13 @@ do { \#ifdef __PPC64__ -#define STORE_WORD "std " -#define LOAD_WORD "ld " -#define LOADX_WORD "ldx " -#define CMP_WORD "cmpd " +#define RSEQ_STORE_LONG (arg) "std%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* To memory ("m" constraint) */ +#define RSEQ_STORE_INT (arg) "stw%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* To memory ("m" constraint) */ +#define RSEQ_LOAD_LONG (arg) "ld%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* From memory ("m" constraint) */ +#define RSEQ_LOAD_INT (arg) "lwz%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* From memory ("m" constraint) */ +#define RSEQ_LOADX_LONG "ldx "/* From base register ("b" constraint) */ +#define RSEQ_CMP_LONG "cmpd " +#define RSEQ_CMP_LONG_INT "cmpdi "#define __RSEQ_ASM_DEFINE_TABLE (label, version, flags, \ start_ip, post_commit_offset, abort_ip) \
@@ -89,10 +92,13 @@ do { \#else /* #ifdef __PPC64__ */ -#define STORE_WORD "stw " -#define LOAD_WORD "lwz " -#define LOADX_WORD "lwzx " -#define CMP_WORD "cmpw " +#define RSEQ_STORE_LONG (arg) "stw%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* To memory ("m" constraint) */ +#define RSEQ_STORE_INT (arg) RSEQ_STORE_LONG(arg)/* To memory ("m" constraint) */ +#define RSEQ_LOAD_LONG (arg) "lwz%U[" __rseq_str(arg) "]%X[" __rseq_str(arg) "] "/* From memory ("m" constraint) */ +#define RSEQ_LOAD_INT (arg) RSEQ_LOAD_LONG(arg)/* From memory ("m" constraint) */ +#define RSEQ_LOADX_LONG "lwzx "/* From base register ("b" constraint) */ +#define RSEQ_CMP_LONG "cmpw " +#define RSEQ_CMP_LONG_INT "cmpwi "#define __RSEQ_ASM_DEFINE_TABLE (label, version, flags, \ start_ip, post_commit_offset, abort_ip) \
@@ -125,7 +131,7 @@ do { \ RSEQ_INJECT_ASM(1) \ "lis %%r17, (" __rseq_str(cs_label) ")@ha\n\t" \ "addi %%r17, %%r17, (" __rseq_str(cs_label) ")@l\n\t" \ - "stw %%r17, %[" __rseq_str(rseq_cs) "]\n\t" \ + RSEQ_STORE_INT(rseq_cs) "%%r17, %[" __rseq_str(rseq_cs) "]\n\t" \ __rseq_str(label) ":\n\t"#endif /* #ifdef __PPC64__ */
@@ -136,7 +142,7 @@ do { \#define RSEQ_ASM_CMP_CPU_ID (cpu_id, current_cpu_id, label) \ RSEQ_INJECT_ASM(2) \ - "lwz %%r17, %[" __rseq_str(current_cpu_id) "]\n\t" \ + RSEQ_LOAD_INT(current_cpu_id) "%%r17, %[" __rseq_str(current_cpu_id) "]\n\t" \ "cmpw cr7, %[" __rseq_str(cpu_id) "], %%r17\n\t" \ "bne- cr7, " __rseq_str(label) "\n\t"
@@ -153,25 +159,25 @@ do { \* RSEQ_ASM_OP_* (else): doesn't have hard-code registers(unless cr7) */ #define RSEQ_ASM_OP_CMPEQ (var, expect, label) \ - LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ - CMP_WORD "cr7, %%r17, %[" __rseq_str(expect) "]\n\t" \ + RSEQ_LOAD_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t" \ + RSEQ_CMP_LONG "cr7, %%r17, %[" __rseq_str(expect) "]\n\t" \ "bne- cr7, " __rseq_str(label) "\n\t"#define RSEQ_ASM_OP_CMPNE (var, expectnot, label) \ - LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ - CMP_WORD "cr7, %%r17, %[" __rseq_str(expectnot) "]\n\t" \ + RSEQ_LOAD_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t" \ + RSEQ_CMP_LONG "cr7, %%r17, %[" __rseq_str(expectnot) "]\n\t" \ "beq- cr7, " __rseq_str(label) "\n\t"#define RSEQ_ASM_OP_STORE (value, var) \ - STORE_WORD "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" + RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t"/* Load @var to r17 */ #define RSEQ_ASM_OP_R_LOAD (var) \ - LOAD_WORD "%%r17, %[" __rseq_str(var) "]\n\t" + RSEQ_LOAD_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t"/* Store r17 to @var */ #define RSEQ_ASM_OP_R_STORE (var) \ - STORE_WORD "%%r17, %[" __rseq_str(var) "]\n\t" + RSEQ_STORE_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t"/* Add @count to r17 */ #define RSEQ_ASM_OP_R_ADD (count) \
@@ -179,11 +185,11 @@ do { \/* Load (r17 + voffp) to r17 */ #define RSEQ_ASM_OP_R_LOADX (voffp) \ - LOADX_WORD "%%r17, %[" __rseq_str(voffp) "], %%r17\n\t" + RSEQ_LOADX_LONG "%%r17, %[" __rseq_str(voffp) "], %%r17\n\t"/* TODO: implement a faster memcpy. */ #define RSEQ_ASM_OP_R_MEMCPY () \ - "cmpdi %%r19, 0\n\t" \ + RSEQ_CMP_LONG_INT "%%r19, 0\n\t" \ "beq 333f\n\t" \ "addi %%r20, %%r20, -1\n\t" \ "addi %%r21, %%r21, -1\n\t" \
@@ -191,16 +197,16 @@ do { \"lbzu %%r18, 1(%%r20)\n\t" \"stbu %%r18, 1(%%r21)\n\t" \"addi %%r19, %%r19, -1\n\t" \ -"cmpdi %%r19, 0\n\t" \ +RSEQ_CMP_LONG_INT "%%r19, 0\n\t" \"bne 222b\n\t" \ "333:\n\t" \#define RSEQ_ASM_OP_R_FINAL_STORE (var, post_commit_label) \ - STORE_WORD "%%r17, %[" __rseq_str(var) "]\n\t" \ + RSEQ_STORE_LONG(var) "%%r17, %[" __rseq_str(var) "]\n\t" \ __rseq_str(post_commit_label) ":\n\t"#define RSEQ_ASM_OP_FINAL_STORE (value, var, post_commit_label) \ - STORE_WORD "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \ + RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n\t" \ __rseq_str(post_commit_label) ":\n\t" static inline __attribute__((always_inline))
@@ -743,9 +749,4 @@ error2: #endif } -#undef STORE_WORD -#undef LOAD_WORD -#undef LOADX_WORD -#undef CMP_WORD -#endif /* !RSEQ_SKIP_FASTPATH */
Mathieu Desnoyers
maintainerRFC PATCH 10/15 selftests/rseq: Fix ppc32 offsets by using long rather than off_t
In reply to Mathieu DesnoyersThe semantic of off_t is for file offsets. We mean to use it as an offset from a pointer. We really expect it to fit in a single register, and not use a 64-bit type on 32-bit architectures.
Fix runtime issues on ppc32 where the offset is always 0 due to inconsistency between the argument type (off_t -> 64-bit) and type expected by the inline assembler (32-bit).
--- tools/testing/selftests/rseq/basic_percpu_ops_test.c | 2 +- tools/testing/selftests/rseq/param_test.c | 2 +- tools/testing/selftests/rseq/rseq-arm.h | 2 +- tools/testing/selftests/rseq/rseq-arm64.h | 2 +- tools/testing/selftests/rseq/rseq-mips.h | 2 +- tools/testing/selftests/rseq/rseq-ppc.h | 2 +- tools/testing/selftests/rseq/rseq-s390.h | 2 +- tools/testing/selftests/rseq/rseq-skip.h | 2 +- tools/testing/selftests/rseq/rseq-x86.h | 6 +++--- 9 files changed, 11 insertions(+), 11 deletions(-)
tools/testing/selftests/rseq/basic_percpu_ops_test.c +1 -1
@@ -167,7 +167,7 @@ struct percpu_list_node *this_cpu_list_pop(struct percpu_list *list,for (;;) {struct percpu_list_node *head;intptr_t *targetptr, expectnot, *load; -off_t offset; +long offset;int ret, cpu; cpu =rseq_cpu_start ();
tools/testing/selftests/rseq/param_test.c +1 -1
@@ -549,7 +549,7 @@ struct percpu_list_node *this_cpu_list_pop(struct percpu_list *list,for (;;) {struct percpu_list_node *head;intptr_t *targetptr, expectnot, *load; -off_t offset; +long offset;int ret; cpu =rseq_cpu_start ();
tools/testing/selftests/rseq/rseq-arm.h +1 -1
@@ -217,7 +217,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-arm64.h +1 -1
@@ -259,7 +259,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-mips.h +1 -1
@@ -222,7 +222,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-ppc.h +1 -1
@@ -270,7 +270,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-s390.h +1 -1
@@ -198,7 +198,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)*/ static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-skip.h +1 -1
@@ -13,7 +13,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {return -1 ; }
tools/testing/selftests/rseq/rseq-x86.h +3 -3
@@ -172,7 +172,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)*/ static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
@@ -286,7 +286,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)* *pval += inc; */ static inline __attribute__((always_inline)) -int rseq_offset_deref_addv (intptr_t *ptr,off_t off,intptr_t inc,int cpu) +int rseq_offset_deref_addv (intptr_t *ptr,long off,intptr_t inc,int cpu) {RSEQ_INJECT_C (9 )
@@ -750,7 +750,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)*/ static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Fix ppc32 offsets by using long rather than off_t
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 26dc8a6d8e11552f3b797b5aafe01071ca32d692 Gitweb: https://git.kernel.org/tip/26dc8a6d8e11552f3b797b5aafe01071ca32d692 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:48 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:36 +01:00
selftests/rseq: Fix ppc32 offsets by using long rather than off_t
The semantic of off_t is for file offsets. We mean to use it as an offset from a pointer. We really expect it to fit in a single register, and not use a 64-bit type on 32-bit architectures.
Fix runtime issues on ppc32 where the offset is always 0 due to inconsistency between the argument type (off_t -> 64-bit) and type expected by the inline assembler (32-bit).
--- tools/testing/selftests/rseq/basic_percpu_ops_test.c | 2 +- tools/testing/selftests/rseq/param_test.c | 2 +- tools/testing/selftests/rseq/rseq-arm.h | 2 +- tools/testing/selftests/rseq/rseq-arm64.h | 2 +- tools/testing/selftests/rseq/rseq-mips.h | 2 +- tools/testing/selftests/rseq/rseq-ppc.h | 2 +- tools/testing/selftests/rseq/rseq-s390.h | 2 +- tools/testing/selftests/rseq/rseq-skip.h | 2 +- tools/testing/selftests/rseq/rseq-x86.h | 6 +++--- 9 files changed, 11 insertions(+), 11 deletions(-)
tools/testing/selftests/rseq/basic_percpu_ops_test.c +1 -1
@@ -167,7 +167,7 @@ struct percpu_list_node *this_cpu_list_pop(struct percpu_list *list,for (;;) {struct percpu_list_node *head;intptr_t *targetptr, expectnot, *load; -off_t offset; +long offset;int ret, cpu; cpu =rseq_cpu_start ();
tools/testing/selftests/rseq/param_test.c +1 -1
@@ -549,7 +549,7 @@ struct percpu_list_node *this_cpu_list_pop(struct percpu_list *list,for (;;) {struct percpu_list_node *head;intptr_t *targetptr, expectnot, *load; -off_t offset; +long offset;int ret; cpu =rseq_cpu_start ();
tools/testing/selftests/rseq/rseq-arm.h +1 -1
@@ -217,7 +217,7 @@ error2:static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-arm64.h +1 -1
@@ -259,7 +259,7 @@ error2:static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-mips.h +1 -1
@@ -222,7 +222,7 @@ error2:static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-ppc.h +1 -1
@@ -270,7 +270,7 @@ error2:static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-s390.h +1 -1
@@ -198,7 +198,7 @@ error2:*/ static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
tools/testing/selftests/rseq/rseq-skip.h +1 -1
@@ -13,7 +13,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {return -1 ; }
tools/testing/selftests/rseq/rseq-x86.h +3 -3
@@ -172,7 +172,7 @@ error2:*/ static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
@@ -286,7 +286,7 @@ error1:* *pval += inc; */ static inline __attribute__((always_inline)) -int rseq_offset_deref_addv (intptr_t *ptr,off_t off,intptr_t inc,int cpu) +int rseq_offset_deref_addv (intptr_t *ptr,long off,intptr_t inc,int cpu) {RSEQ_INJECT_C (9 )
@@ -750,7 +750,7 @@ error2:*/ static inline __attribute__((always_inline))int rseq_cmpnev_storeoffp_load (intptr_t *v,intptr_t expectnot, -off_t voffp,intptr_t *load,int cpu) +long voffp,intptr_t *load,int cpu) {RSEQ_INJECT_C (9 )
Mathieu Desnoyers
maintainerRFC PATCH 11/15 selftests/rseq: Fix warnings about #if checks of undefined tokens
In reply to Mathieu Desnoyers--- tools/testing/selftests/rseq/param_test.c | 2 +- tools/testing/selftests/rseq/rseq-x86.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/rseq/param_test.c +1 -1
@@ -161,7 +161,7 @@ unsigned int yield_mod_cnt, nr_abort;" cbnz " INJECT_ASM_REG ", 222b\n" \ "333:\n" -#elif __PPC__ +#elif defined(__PPC__)#define RSEQ_INJECT_INPUT \ , [loop_cnt_1]"m"(loop_cnt[1]) \
tools/testing/selftests/rseq/rseq-x86.h +1 -1
@@ -600,7 +600,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#endif /* !RSEQ_SKIP_FASTPATH */ -#elif __i386__ +#elif defined(__i386__)#define rseq_smp_mb () \ __asm__ __volatile__ ("lock; addl $0,-128(%%esp)" ::: "memory", "cc")
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Fix warnings about #if checks of undefined tokens
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: d7ed99ade3e62b755584eea07b4e499e79240527 Gitweb: https://git.kernel.org/tip/d7ed99ade3e62b755584eea07b4e499e79240527 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:49 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:36 +01:00
selftests/rseq: Fix warnings about #if checks of undefined tokens
--- tools/testing/selftests/rseq/param_test.c | 2 +- tools/testing/selftests/rseq/rseq-x86.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/rseq/param_test.c +1 -1
@@ -161,7 +161,7 @@ unsigned int yield_mod_cnt, nr_abort;" cbnz " INJECT_ASM_REG ", 222b\n" \ "333:\n" -#elif __PPC__ +#elif defined(__PPC__)#define RSEQ_INJECT_INPUT \ , [loop_cnt_1]"m"(loop_cnt[1]) \
tools/testing/selftests/rseq/rseq-x86.h +1 -1
@@ -600,7 +600,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#endif /* !RSEQ_SKIP_FASTPATH */ -#elif __i386__ +#elif defined(__i386__)#define rseq_smp_mb () \ __asm__ __volatile__ ("lock; addl $0,-128(%%esp)" ::: "memory", "cc")
Mathieu Desnoyers
maintainerRFC PATCH 12/15 selftests/rseq: Remove arm/mips asm goto compiler work-around
In reply to Mathieu DesnoyersThe arm and mips work-around for asm goto size guess issues are not properly documented, and lack reference to specific compiler versions, upstream compiler bug tracker entry, and reproducer.
I can only find a loosely documented patch in my original LKML rseq post refering to gcc < 7 on ARM, but it does not appear to be sufficient to track the exact issue. Also, I am not sure MIPS really has the same limitation.
Therefore, remove the work-around until we can properly document this.
--- tools/testing/selftests/rseq/rseq-arm.h | 37 ------------------------ tools/testing/selftests/rseq/rseq-mips.h | 37 ------------------------ 2 files changed, 74 deletions(-)
tools/testing/selftests/rseq/rseq-arm.h +0 -37
@@ -147,14 +147,11 @@ do { \ teardown \ "b %l[" __rseq_str(cmpfail_label) "]\n\t" -#define rseq_workaround_gcc_asm_size_guess () __asm__ __volatile__("") - static inline __attribute__((always_inline))int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv,int cpu) { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -198,14 +195,11 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -221,7 +215,6 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -270,14 +263,11 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -292,7 +282,6 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */#ifdef RSEQ_COMPARE_TWICE
@@ -328,10 +317,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE
@@ -347,7 +334,6 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -398,14 +384,11 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -422,7 +405,6 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -474,14 +456,11 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -498,7 +477,6 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -554,14 +532,11 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -582,7 +557,6 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -678,21 +652,16 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("cpu_id comparison failed" ); error2: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("expected value comparison failed" );#endif }
@@ -706,7 +675,6 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -803,21 +771,16 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("cpu_id comparison failed" ); error2: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-mips.h +0 -37
@@ -154,14 +154,11 @@ do { \ teardown \ "b %l[" __rseq_str(cmpfail_label) "]\n\t" -#define rseq_workaround_gcc_asm_size_guess () __asm__ __volatile__("") - static inline __attribute__((always_inline))int rseq_cmpeqv_storev (intptr_t *v,intptr_t expect,intptr_t newv,int cpu) { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -203,14 +200,11 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -226,7 +220,6 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -273,14 +266,11 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -295,7 +285,6 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */#ifdef RSEQ_COMPARE_TWICE
@@ -331,10 +320,8 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE
@@ -350,7 +337,6 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -399,14 +385,11 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -423,7 +406,6 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -473,14 +455,11 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -497,7 +476,6 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, { RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -549,14 +527,11 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1:
@@ -577,7 +552,6 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -670,21 +644,16 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("cpu_id comparison failed" ); error2: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("expected value comparison failed" );#endif }
@@ -698,7 +667,6 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, RSEQ_INJECT_C(9) -rseq_workaround_gcc_asm_size_guess (); __asm__ __volatile__ goto ( RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail])
@@ -792,21 +760,16 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); -rseq_workaround_gcc_asm_size_guess ();return 0 ; abort: -rseq_workaround_gcc_asm_size_guess ();RSEQ_INJECT_FAILED return -1; cmpfail: -rseq_workaround_gcc_asm_size_guess ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("cpu_id comparison failed" ); error2: -rseq_workaround_gcc_asm_size_guess ();rseq_bug ("expected value comparison failed" );#endif }
Signature · 1 line
2.17.1
Mathieu Desnoyers
maintainerRFC PATCH 13/15 selftests/rseq: Fix: work-around asm goto compiler bugs
In reply to Mathieu Desnoyersgcc and clang each have their own compiler bugs with respect to asm goto. Implement a work-around for compiler versions known to have those bugs.
gcc prior to 4.8.2 miscompiles asm goto.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58670
gcc prior to 8.1.0 miscompiles asm goto at O1.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103908
clang prior to version 13.0.1 miscompiles asm goto at O2.
https://github.com/llvm/llvm-project/issues/52735
Work around these issues by adding a volatile inline asm with memory clobber in the fallthrough after the asm goto and at each label target. Emit this for all compilers in case other similar issues are found in the future.
--- tools/testing/selftests/rseq/compiler.h | 30 ++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 39 +++++++++++++ tools/testing/selftests/rseq/rseq-arm64.h | 45 +++++++++++++-- tools/testing/selftests/rseq/rseq-ppc.h | 39 +++++++++++++ tools/testing/selftests/rseq/rseq-s390.h | 29 ++++++++++ tools/testing/selftests/rseq/rseq-x86.h | 68 +++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 1 + 7 files changed, 245 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/rseq/compiler.h
tools/testing/selftests/rseq/compiler.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq/compiler.h +* +* Work-around asm goto compiler bugs. +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef RSEQ_COMPILER_H +#define RSEQ_COMPILER_H + +/* +* gcc prior to 4.8.2 miscompiles asm goto. +* https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58670 +* +* gcc prior to 8.1.0 miscompiles asm goto at O1. +* https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103908 +* +* clang prior to version 13.0.1 miscompiles asm goto at O2. +* https://github.com/llvm/llvm-project/issues/52735 +* +* Work around these issues by adding a volatile inline asm with +* memory clobber in the fallthrough after the asm goto and at each +* label target. Emit this for all compilers in case other similar +* issues are found in the future. +*/ +#define rseq_after_asm_goto () asm volatile ("" : : : "memory") + +#endif /* RSEQ_COMPILER_H_ */
tools/testing/selftests/rseq/rseq-arm.h +39 -0
@@ -195,16 +195,21 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -263,16 +268,21 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -317,12 +327,15 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" );#endif }
@@ -384,16 +397,21 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -456,16 +474,21 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -532,18 +555,24 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("1st expected value comparison failed" ); error3: +rseq_after_asm_goto ();rseq_bug ("2nd expected value comparison failed" );#endif }
@@ -652,16 +681,21 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -771,16 +805,21 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-arm64.h +39 -6
171 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-ppc.h +39 -0
@@ -254,16 +254,21 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -322,16 +327,21 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -378,12 +388,15 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" );#endif }
@@ -442,16 +455,21 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -512,16 +530,21 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -583,18 +606,24 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("1st expected value comparison failed" ); error3: +rseq_after_asm_goto ();rseq_bug ("2nd expected value comparison failed" );#endif }
@@ -659,16 +688,21 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -735,16 +769,21 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-s390.h +29 -0
@@ -178,16 +178,21 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -248,16 +253,21 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -301,12 +311,15 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" );#endif }
@@ -364,16 +377,21 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -443,18 +461,24 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("1st expected value comparison failed" ); error3: +rseq_after_asm_goto ();rseq_bug ("2nd expected value comparison failed" );#endif }
@@ -555,16 +579,21 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-x86.h +68 -0
288 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.h +1 -0
@@ -17,6 +17,7 @@#include <stdio.h> #include <stdlib.h> #include "rseq-abi.h" +#include "compiler.h" /* * Empty code injection macros, override when testing.
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: Fix: work-around asm goto compiler bugs
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b53823fb2ef854222853be164f3b1e815f315144 Gitweb: https://git.kernel.org/tip/b53823fb2ef854222853be164f3b1e815f315144 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:51 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:37 +01:00
selftests/rseq: Fix: work-around asm goto compiler bugs
gcc and clang each have their own compiler bugs with respect to asm goto. Implement a work-around for compiler versions known to have those bugs.
gcc prior to 4.8.2 miscompiles asm goto.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58670
gcc prior to 8.1.0 miscompiles asm goto at O1.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103908
clang prior to version 13.0.1 miscompiles asm goto at O2.
https://github.com/llvm/llvm-project/issues/52735
Work around these issues by adding a volatile inline asm with memory clobber in the fallthrough after the asm goto and at each label target. Emit this for all compilers in case other similar issues are found in the future.
--- tools/testing/selftests/rseq/compiler.h | 30 ++++++++++- tools/testing/selftests/rseq/rseq-arm.h | 39 +++++++++++++- tools/testing/selftests/rseq/rseq-arm64.h | 45 +++++++++++++-- tools/testing/selftests/rseq/rseq-ppc.h | 39 +++++++++++++- tools/testing/selftests/rseq/rseq-s390.h | 29 +++++++++- tools/testing/selftests/rseq/rseq-x86.h | 68 ++++++++++++++++++++++- tools/testing/selftests/rseq/rseq.h | 1 +- 7 files changed, 245 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/rseq/compiler.h
tools/testing/selftests/rseq/compiler.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-only OR MIT */ +/* +* rseq/compiler.h +* +* Work-around asm goto compiler bugs. +* +* (C) Copyright 2021 - Mathieu Desnoyers <[email protected]> +*/ + +#ifndef RSEQ_COMPILER_H +#define RSEQ_COMPILER_H + +/* +* gcc prior to 4.8.2 miscompiles asm goto. +* https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58670 +* +* gcc prior to 8.1.0 miscompiles asm goto at O1. +* https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103908 +* +* clang prior to version 13.0.1 miscompiles asm goto at O2. +* https://github.com/llvm/llvm-project/issues/52735 +* +* Work around these issues by adding a volatile inline asm with +* memory clobber in the fallthrough after the asm goto and at each +* label target. Emit this for all compilers in case other similar +* issues are found in the future. +*/ +#define rseq_after_asm_goto () asm volatile ("" : : : "memory") + +#endif /* RSEQ_COMPILER_H_ */
tools/testing/selftests/rseq/rseq-arm.h +39 -0
@@ -195,16 +195,21 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -263,16 +268,21 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -317,12 +327,15 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" );#endif }
@@ -384,16 +397,21 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -456,16 +474,21 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -532,18 +555,24 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("1st expected value comparison failed" ); error3: +rseq_after_asm_goto ();rseq_bug ("2nd expected value comparison failed" );#endif }
@@ -652,16 +681,21 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -771,16 +805,21 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-arm64.h +39 -6
171 lines not on this page · raw message
tools/testing/selftests/rseq/rseq-ppc.h +39 -0
@@ -254,16 +254,21 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -322,16 +327,21 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -378,12 +388,15 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" );#endif }
@@ -442,16 +455,21 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -512,16 +530,21 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -583,18 +606,24 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("1st expected value comparison failed" ); error3: +rseq_after_asm_goto ();rseq_bug ("2nd expected value comparison failed" );#endif }
@@ -659,16 +688,21 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -735,16 +769,21 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-s390.h +29 -0
@@ -178,16 +178,21 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -248,16 +253,21 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -301,12 +311,15 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) , error1#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" );#endif }
@@ -364,16 +377,21 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
@@ -443,18 +461,24 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, , error1, error2, error3#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("1st expected value comparison failed" ); error3: +rseq_after_asm_goto ();rseq_bug ("2nd expected value comparison failed" );#endif }
@@ -555,16 +579,21 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, , error1, error2#endif ); +rseq_after_asm_goto ();return 0 ; abort: +rseq_after_asm_goto ();RSEQ_INJECT_FAILED return -1; cmpfail: +rseq_after_asm_goto ();return 1 ;#ifdef RSEQ_COMPARE_TWICE error1: +rseq_after_asm_goto ();rseq_bug ("cpu_id comparison failed" ); error2: +rseq_after_asm_goto ();rseq_bug ("expected value comparison failed" );#endif }
tools/testing/selftests/rseq/rseq-x86.h +68 -0
288 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.h +1 -0
@@ -17,6 +17,7 @@#include <stdio.h> #include <stdlib.h> #include "rseq-abi.h" +#include "compiler.h" /* * Empty code injection macros, override when testing.
Mathieu Desnoyers
maintainerRFC PATCH 14/15 selftests/rseq: x86-64: use %fs segment selector for accessing rseq thread area
In reply to Mathieu DesnoyersRather than use rseq_get_abi() and pass its result through a register to the inline assembler, directly access the per-thread rseq area through a memory reference combining the %fs segment selector, the constant offset of the field in struct rseq, and the rseq_offset value (in a register).
--- tools/testing/selftests/rseq/rseq-x86.h | 58 +++++++++++++------------ 1 file changed, 30 insertions(+), 28 deletions(-)
tools/testing/selftests/rseq/rseq-x86.h +30 -28
@@ -28,6 +28,8 @@#ifdef __x86_64__ +#define RSEQ_ASM_TP_SEGMENT %%fs +#define rseq_smp_mb () \ __asm__ __volatile__ ("lock; addl $0,-128(%%rsp)" ::: "memory", "cc")#define rseq_smp_rmb () rseq_barrier()
@@ -123,14 +125,14 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpq %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpq %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -141,7 +143,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -189,15 +191,15 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"movq %[v], %%rbx\n\t" "cmpq %%rbx, %[expectnot]\n\t" "je %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"movq %[v], %%rbx\n\t" "cmpq %%rbx, %[expectnot]\n\t" "je %l[error2]\n\t"
@@ -212,7 +214,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -255,11 +257,11 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) #ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) #endif/* final store */ "addq %[count], %[v]\n\t"
@@ -268,7 +270,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [v] "m" (*v), [count] "er" (count)
@@ -309,11 +311,11 @@ int rseq_offset_deref_addv(intptr_t *ptr, long off, intptr_t inc, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) #ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) #endif/* get p+v */ "movq %[ptr], %%rbx\n\t"
@@ -327,7 +329,7 @@ int rseq_offset_deref_addv(intptr_t *ptr, long off, intptr_t inc, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [ptr] "m" (*ptr), [off] "er" (off),
@@ -364,14 +366,14 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpq %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpq %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -385,7 +387,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -444,8 +446,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpq %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t"
@@ -454,7 +456,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(5)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"cmpq %[v], %[expect]\n\t" "jnz %l[error2]\n\t" "cmpq %[v2], %[expect2]\n\t"
@@ -467,7 +469,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -524,14 +526,14 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, "movq %[dst], %[rseq_scratch1]\n\t" "movq %[len], %[rseq_scratch2]\n\t" /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) "cmpq %[v], %[expect]\n\t" "jnz 5f\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 6f) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f) "cmpq %[v], %[expect]\n\t" "jnz 7f\n\t"#endif
@@ -579,7 +581,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: x86-64: use %fs segment selector for accessing rseq thread area
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4e15bb766b6c6e963a4d33629034d0ec3b7637df Gitweb: https://git.kernel.org/tip/4e15bb766b6c6e963a4d33629034d0ec3b7637df Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:52 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:37 +01:00
selftests/rseq: x86-64: use %fs segment selector for accessing rseq thread area
Rather than use rseq_get_abi() and pass its result through a register to the inline assembler, directly access the per-thread rseq area through a memory reference combining the %fs segment selector, the constant offset of the field in struct rseq, and the rseq_offset value (in a register).
--- tools/testing/selftests/rseq/rseq-x86.h | 58 ++++++++++++------------ 1 file changed, 30 insertions(+), 28 deletions(-)
tools/testing/selftests/rseq/rseq-x86.h +30 -28
@@ -28,6 +28,8 @@#ifdef __x86_64__ +#define RSEQ_ASM_TP_SEGMENT %%fs +#define rseq_smp_mb () \ __asm__ __volatile__ ("lock; addl $0,-128(%%rsp)" ::: "memory", "cc")#define rseq_smp_rmb () rseq_barrier()
@@ -123,14 +125,14 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpq %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpq %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -141,7 +143,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -189,15 +191,15 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"movq %[v], %%rbx\n\t" "cmpq %%rbx, %[expectnot]\n\t" "je %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"movq %[v], %%rbx\n\t" "cmpq %%rbx, %[expectnot]\n\t" "je %l[error2]\n\t"
@@ -212,7 +214,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -255,11 +257,11 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) #ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) #endif/* final store */ "addq %[count], %[v]\n\t"
@@ -268,7 +270,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [v] "m" (*v), [count] "er" (count)
@@ -309,11 +311,11 @@ int rseq_offset_deref_addv(intptr_t *ptr, long off, intptr_t inc, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) #ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) #endif/* get p+v */ "movq %[ptr], %%rbx\n\t"
@@ -327,7 +329,7 @@ int rseq_offset_deref_addv(intptr_t *ptr, long off, intptr_t inc, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [ptr] "m" (*ptr), [off] "er" (off),
@@ -364,14 +366,14 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpq %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpq %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -385,7 +387,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -444,8 +446,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpq %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t"
@@ -454,7 +456,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(5)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"cmpq %[v], %[expect]\n\t" "jnz %l[error2]\n\t" "cmpq %[v2], %[expect2]\n\t"
@@ -467,7 +469,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -524,14 +526,14 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, "movq %[dst], %[rseq_scratch1]\n\t" "movq %[len], %[rseq_scratch2]\n\t" /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) "cmpq %[v], %[expect]\n\t" "jnz 5f\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 6f) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f) "cmpq %[v], %[expect]\n\t" "jnz 7f\n\t"#endif
@@ -579,7 +581,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" ((long)rseq_offset), /* final store input */ [v] "m" (*v), [expect] "r" (expect),
Mathieu Desnoyers
maintainerRFC PATCH 15/15 selftests/rseq: x86-32: use %gs segment selector for accessing rseq thread area
In reply to Mathieu DesnoyersRather than use rseq_get_abi() and pass its result through a register to the inline assembler, directly access the per-thread rseq area through a memory reference combining the %gs segment selector, the constant offset of the field in struct rseq, and the rseq_offset value (in a register).
--- tools/testing/selftests/rseq/rseq-x86.h | 66 +++++++++++++------------ 1 file changed, 34 insertions(+), 32 deletions(-)
tools/testing/selftests/rseq/rseq-x86.h +34 -32
@@ -633,6 +633,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#elif defined(__i386__) +#define RSEQ_ASM_TP_SEGMENT %%gs +#define rseq_smp_mb () \ __asm__ __volatile__ ("lock; addl $0,-128(%%esp)" ::: "memory", "cc")#define rseq_smp_rmb () \
@@ -732,14 +734,14 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpl %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpl %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -750,7 +752,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -798,15 +800,15 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"movl %[v], %%ebx\n\t" "cmpl %%ebx, %[expectnot]\n\t" "je %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"movl %[v], %%ebx\n\t" "cmpl %%ebx, %[expectnot]\n\t" "je %l[error2]\n\t"
@@ -821,7 +823,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -864,11 +866,11 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) #ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) #endif/* final store */ "addl %[count], %[v]\n\t"
@@ -877,7 +879,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [count] "ir" (count)
@@ -916,14 +918,14 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpl %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpl %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -938,7 +940,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* try store input */ [v2] "m" (*v2), [newv2] "m" (newv2),
@@ -987,15 +989,15 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"movl %[expect], %%eax\n\t" "cmpl %[v], %%eax\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"movl %[expect], %%eax\n\t" "cmpl %[v], %%eax\n\t" "jnz %l[error2]\n\t"
@@ -1011,7 +1013,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -1062,8 +1064,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpl %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t"
@@ -1072,7 +1074,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(5)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"cmpl %[v], %[expect]\n\t" "jnz %l[error2]\n\t" "cmpl %[expect2], %[v2]\n\t"
@@ -1086,7 +1088,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -1144,15 +1146,15 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, "movl %[dst], %[rseq_scratch1]\n\t" "movl %[len], %[rseq_scratch2]\n\t" /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) "movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 5f\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 6f) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)"movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 7f\n\t"
@@ -1202,7 +1204,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
@@ -1261,15 +1263,15 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, "movl %[dst], %[rseq_scratch1]\n\t" "movl %[len], %[rseq_scratch2]\n\t" /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) "movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 5f\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 6f) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)"movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 7f\n\t"
@@ -1320,7 +1322,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
Signature · 1 line
2.17.1
[tip: sched/core] selftests/rseq: x86-32: use %gs segment selector for accessing rseq thread area
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 127b6429d235ab7c358223bbfd8a8b8d8cc799b6 Gitweb: https://git.kernel.org/tip/127b6429d235ab7c358223bbfd8a8b8d8cc799b6 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Mon, 24 Jan 2022 12:12:53 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 02 Feb 2022 13:11:37 +01:00
selftests/rseq: x86-32: use %gs segment selector for accessing rseq thread area
Rather than use rseq_get_abi() and pass its result through a register to the inline assembler, directly access the per-thread rseq area through a memory reference combining the %gs segment selector, the constant offset of the field in struct rseq, and the rseq_offset value (in a register).
--- tools/testing/selftests/rseq/rseq-x86.h | 66 ++++++++++++------------ 1 file changed, 34 insertions(+), 32 deletions(-)
tools/testing/selftests/rseq/rseq-x86.h +34 -32
@@ -633,6 +633,8 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#elif defined(__i386__) +#define RSEQ_ASM_TP_SEGMENT %%gs +#define rseq_smp_mb () \ __asm__ __volatile__ ("lock; addl $0,-128(%%esp)" ::: "memory", "cc")#define rseq_smp_rmb () \
@@ -732,14 +734,14 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpl %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpl %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -750,7 +752,7 @@ int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), [v] "m" (*v), [expect] "r" (expect), [newv] "r" (newv)
@@ -798,15 +800,15 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"movl %[v], %%ebx\n\t" "cmpl %%ebx, %[expectnot]\n\t" "je %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"movl %[v], %%ebx\n\t" "cmpl %%ebx, %[expectnot]\n\t" "je %l[error2]\n\t"
@@ -821,7 +823,7 @@ int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [expectnot] "r" (expectnot),
@@ -864,11 +866,11 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu) RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) #ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) #endif/* final store */ "addl %[count], %[v]\n\t"
@@ -877,7 +879,7 @@ int rseq_addv(intptr_t *v, intptr_t count, int cpu)RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [count] "ir" (count)
@@ -916,14 +918,14 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpl %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1]) "cmpl %[v], %[expect]\n\t" "jnz %l[error2]\n\t"#endif
@@ -938,7 +940,7 @@ int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* try store input */ [v2] "m" (*v2), [newv2] "m" (newv2),
@@ -987,15 +989,15 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"movl %[expect], %%eax\n\t" "cmpl %[v], %%eax\n\t" "jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"movl %[expect], %%eax\n\t" "cmpl %[v], %%eax\n\t" "jnz %l[error2]\n\t"
@@ -1011,7 +1013,7 @@ int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* try store input */ [v2] "m" (*v2), [newv2] "r" (newv2),
@@ -1062,8 +1064,8 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3])#endif /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3)"cmpl %[v], %[expect]\n\t" "jnz %l[cmpfail]\n\t"
@@ -1072,7 +1074,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,"jnz %l[cmpfail]\n\t" RSEQ_INJECT_ASM(5)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), %l[error1]) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), %l[error1])"cmpl %[v], %[expect]\n\t" "jnz %l[error2]\n\t" "cmpl %[expect2], %[v2]\n\t"
@@ -1086,7 +1088,7 @@ int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,RSEQ_ASM_DEFINE_ABORT (4 ,"" , abort) : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* cmp2 input */ [v2] "m" (*v2), [expect2] "r" (expect2),
@@ -1144,15 +1146,15 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, "movl %[dst], %[rseq_scratch1]\n\t" "movl %[len], %[rseq_scratch2]\n\t" /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) "movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 5f\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 6f) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)"movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 7f\n\t"
@@ -1202,7 +1204,7 @@ int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
@@ -1261,15 +1263,15 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, "movl %[dst], %[rseq_scratch1]\n\t" "movl %[len], %[rseq_scratch2]\n\t" /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_CS_OFFSET(%[rseq_abi])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 4f) + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 4f) RSEQ_INJECT_ASM(3) "movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 5f\n\t" RSEQ_INJECT_ASM(4)#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_CPU_ID_OFFSET(%[rseq_abi]), 6f) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rseq_offset]), 6f)"movl %[expect], %%eax\n\t" "cmpl %%eax, %[v]\n\t" "jnz 7f\n\t"
@@ -1320,7 +1322,7 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,#endif : /* gcc asm goto does not allow outputs */ : [cpu_id] "r" (cpu), - [rseq_abi] "r" (rseq_get_abi()), + [rseq_offset] "r" (rseq_offset), /* final store input */ [v] "m" (*v), [expect] "m" (expect),
Peter Zijlstra
maintainerQuoted 9 lines · Mathieu Desnoyers
Mathieu Desnoyers (15):
selftests/rseq: introduce own copy of rseq uapi header
rseq: Remove broken uapi field layout on 32-bit little endian selftests/rseq: Remove useless assignment to cpu variable selftests/rseq: Remove volatile from__rseq_abiselftests/rseq: Introducerseq_get_abi()helper selftests/rseq: Introduce thread pointer getters selftests/rseq: Uplift rseq selftests for compatibility withglibc-2.35
Quoted 12 lines
selftests/rseq: Fix ppc32: wrong
rseq_cs32-bit field pointer on big
endian
selftests/rseq: Fix ppc32 missing instruction selection "u" and "x"
for load/store
selftests/rseq: Fix ppc32 offsets by using long rather thanoff_tselftests/rseq: Fix warnings about #if checks of undefined tokens selftests/rseq: Remove arm/mips asm goto compiler work-around selftests/rseq: Fix: work-around asm goto compiler bugs selftests/rseq: x86-64: use %fs segment selector for accessing rseq
thread area
selftests/rseq: x86-32: use %gs segment selector for accessing rseq
thread area
Thanks, I'll go stick then in sched/core.