rseq: add support for RSEQ operations
Subsystems (2)
cover rseq: add support for RSEQ operations
From: Olivier Dion <[email protected]>
Hi,
This series introduces RSEQ operations: a per-thread list of operations the kernel applies, on behalf of a ask, when returning to user space.
The main motivation is to help TCMalloc migrate from RSEQ v1 to RSEQ v2 and use the RSEQ area registered by glibc. TCMalloc currently relies on RSEQ v1 resetting the cpu_id field to invalidate a per-CPU pointer cached in TLS. This requires applications to disable glibc's implicit RSEQ registration and prevents sharing the glibc RSEQ area.
RSEQ operations provide an opt-in replacement for that invalidation mechanism. User space registers operation nodes with prctl; the kernel keeps the nodes in a per-thread circular list and applies them on return to user space. Only tasks with registered operations incur the additional RSEQ exit-path work.
The first patch adds the UAPI. The next three patches add the task state, operation execution, and prctl registration support. The final patch adds selftests covering normal operation and malformed-list and invalid-memory cases.
See also <https://lore.kernel.org/lkml/[email protected]>.
Thanks,
old
Olivier Dion (5):
rseq: uapi: add rseq operation definitions
rseq: add per-task rseq operation state
rseq: apply operations on exit to user space
rseq: register and unregister operations via prctl selftests/rseq: add coverage for rseq operations
include/linux/rseq.h | 10 +- include/linux/rseq_entry.h | 157 ++++++++++++++++- include/linux/rseq_types.h | 9 + include/uapi/linux/prctl.h | 12 ++ include/uapi/linux/rseq.h | 104 ++++++++++- kernel/rseq.c | 220 ++++++++++++++++++++++-- kernel/sys.c | 5 + tools/testing/selftests/rseq/.gitignore | 4 +- tools/testing/selftests/rseq/Makefile | 4 +- tools/testing/selftests/rseq/rseq-abi.h | 146 +++++++++++++--- tools/testing/selftests/rseq/rseq.c | 23 ++- tools/testing/selftests/rseq/rseq.h | 75 ++++++++ 12 files changed, 713 insertions(+), 56 deletions(-)
Signature · 1 line
2.54.0
RFC PATCH 1/5 rseq: uapi: add rseq operation definitions
From: Olivier Dion <[email protected]>
Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.
The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.
Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.
Overall, this puts the users in a situation of choosing between a fast sched_getcpu() and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibc
RSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them.
Add enum rseq_op_type and the rseq_op_node / rseq_op_reset / rseq_op_reset_with_stride structures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, the RSEQ_OP_LIST_LIMIT walk bound, and the PR_RSEQ_OP prctl with its REGISTER/UNREGISTER sub-commands.
The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq.
[0] Documentation/userspace-api/rseq.rst (Optimized RSEQ v2)
--- include/uapi/linux/prctl.h | 12 +++++ include/uapi/linux/rseq.h | 104 ++++++++++++++++++++++++++++++++++--- 2 files changed, 108 insertions(+), 8 deletions(-)
include/uapi/linux/prctl.h +12 -0
@@ -396,6 +396,18 @@ struct prctl_mm_map {*/ # define PR_RSEQ_SLICE_EXT_ENABLE 0x01 +/* +* RSEQ operation registration. +* +* arg3 is the user address of a struct rseq_op_node embedded in one of the +* rseq operation structures (see uapi/linux/rseq.h). Registering the first +* operation enables rseq operation processing for the thread; unregistering +* the last one disables it. +*/ +#define PR_RSEQ_OP 82 +# define PR_RSEQ_OP_REGISTER 1 +# define PR_RSEQ_OP_UNREGISTER 2 +/* * Get or set the control flow integrity (CFI) configuration for the * current thread.
include/uapi/linux/rseq.h +96 -8
135 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions
Quoted 31 lines · On Fri, 28 Aug 2026 at 17:33, <[email protected]> wrote:
From: Olivier Dion <[email protected]>
Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.
The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.
Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the
cpu_idbits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.Overall, this puts the users in a situation of choosing between a fast
sched_getcpu()and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibcRSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them.
Add enum
rseq_op_typeand therseq_op_node/rseq_op_reset/rseq_op_reset_with_stridestructures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, theRSEQ_OP_LIST_LIMITwalk bound, and thePR_RSEQ_OPprctl with its REGISTER/UNREGISTER sub-commands.
Quoted 4 lines
The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq.
[0]
Documentation/userspace-api/rseq.rst(Optimized RSEQ v2)
Hi Olivier,
This is very cool, thanks for working on this.
Quoted 35 lines
Signed-off-by Olivier Dion <[email protected]>--- include/uapi/linux/prctl.h | 12 +++++ include/uapi/linux/rseq.h | 104 ++++++++++++++++++++++++++++++++++--- 2 files changed, 108 insertions(+), 8 deletions(-)include/uapi/linux/prctl.h +12
-0@@ -396,6 +396,18 @@ struct prctl_mm_map {*/ # define PR_RSEQ_SLICE_EXT_ENABLE 0x01 +/* +* RSEQ operation registration. +* +* arg3 is the user address of a struct rseq_op_node embedded in one of the +* rseq operation structures (see uapi/linux/rseq.h). Registering the first +* operation enables rseq operation processing for the thread; unregistering +* the last one disables it. +*/ +#define PR_RSEQ_OP 82 +# define PR_RSEQ_OP_REGISTER 1 +# define PR_RSEQ_OP_UNREGISTER 2 +/* * Get or set the control flow integrity (CFI) configuration for the * current thread. include/uapi/linux/rseq.h +0
-0
include/uapi/linux/rseq.h line 19
Quoted 7 lines
include/uapi/linux/rseq.h +4
-0@@ -13,6 +13,11 @@#include <linux/types.h> #include <asm/byteorder.h> +/* +* Maximum number of nodes walked in the rseq operation list. +*/ +#define RSEQ_OP_LIST_LIMIT 2048
Does this belong to the user header? Kernel and userspace are not necessary built with the same headers, and we don't necessary want to set this const in stone. I think it may be more flexible to make this impl detail and just return ENOSPC.
Quoted 24 lines
include/uapi/linux/rseq.h +6
-0enum rseq_cpu_id_state {RSEQ_CPU_ID_UNINITIALIZED =-1 ,RSEQ_CPU_ID_REGISTRATION_FAILED =-2 ,@@ -33,6 +38,8 @@ enum rseq_cs_flags_bit {/* User read only feature flags */ RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT =4 ,RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT =5 , +RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT =6 , + RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT = 7, }; enum rseq_cs_flags {@@ -47,6 +54,10 @@ enum rseq_cs_flags { (1U <<RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT ),RSEQ_CS_FLAG_SLICE_EXT_ENABLED = (1U <<RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT ), +RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE = + (1U <<RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT ), +RSEQ_CS_FLAG_RSEQ_OP_ENABLED = + (1U << RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT), };/*
include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 106
Quoted 10 lines
include/uapi/linux/rseq.h +7
-0@@ -86,6 +97,77 @@ struct rseq_slice_ctrl { }; }; +/* +* enum rseq_op_type - Type of an rseq operation +* @RSEQ_OP_RESET: Plain reset. Uses struct rseq_op_reset. +* @RSEQ_OP_RESET_WITH_STRIDE_CPUID: Reset indexed by the current CPU ID. +* Uses struct rseq_op_reset_with_stride. +* @RSEQ_OP_RESET_WITH_STRIDE_MMCID: Reset indexed by the current MM CID. +* Uses struct rseq_op_reset_with_stride.
Do you have use-cases for these STRIDE ops?
I can think of some, but they would require executing ops when a task is descheduled, rather than when it's scheduled in.
Quoted 11 lines
include/uapi/linux/rseq.h +11
-0+*/ +enum rseq_op_type { +RSEQ_OP_RESET , +RSEQ_OP_RESET_WITH_STRIDE_CPUID , +RSEQ_OP_RESET_WITH_STRIDE_MMCID , +RSEQ_OP_NR , +}; + +/* +* struct rseq_op_node - Common header linking an rseq operation into the list +* @next: Address of the next node. Owned by the kernel.
include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 127
Quoted 10 lines
include/uapi/linux/rseq.h +10
-0+* @prev: Address of the previous node. Owned by the kernel. +* @type: Operation type. See enum rseq_op_type. +* @reserved: Must be zero on registration. +* +* User space allocates the node, sets @type and zeroes @next, @prev and +* @reserved before passing it to prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, node). +* The kernel owns @next and @prev for the lifetime of the registration and +* links the node into a circular doubly-linked list anchored by an internal +* sentinel in struct rseq. User space must not touch @next or @prev while the +* node is registered.
Have you considered keeping copies of these structs in the kernel? That (1) is more consistent with most of the other kernel APIs, (2) will make executing ops faster since we don't need to copy_from_user all the time, (3) will make code much simpler since we won't need to re-verify all data, (4) will make the code more secure since user-space won't be able to mess with these structs, (5) will simplify the contract and documentation.
There are robust lists that keep lists in user-space, but these are supposed to be concurrently modified by userspace. But these rseq ops are generally not expected to be added/removed often (and the current contract does not allow that anyway).
Quoted 4 lines
include/uapi/linux/rseq.h +4
-0+*/ +struct rseq_op_node { +__u64 next; +__u64 prev;
include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 141
Quoted 10 lines
include/uapi/linux/rseq.h +10
-0+struct { +__u8 type;/* enum rseq_op_type */ +__u8 reserved[7 ]; + }; +}; + +/* +* struct rseq_op_reset - Reset one word to a value on return to user space +* @node: Operation list node. +* @src: Address of the source word, or 0 to reset @dst to zero.
Is "or 0 to reset" part implemented in the series?
It would be useful to see an op that contains a const that is written, rather than an address. Address invovles additional indirection + copy_from_user.
And the contract for concurrent changes of the value pointed by src from user-space are not documented (atomicity?), w/o that having the address is not very useful.
Quoted 41 lines
include/uapi/linux/rseq.h +29
-5+* @dst: Address of the destination word. +* @len: Word length in bytes. Must be 4 or 8 (8 is 64-bit only). +*/ +struct rseq_op_reset { +struct rseq_op_node node; +__u64 src; +__u64 dst; +__u32 len; +}; + +/* +* struct rseq_op_reset_with_stride - Reset one word in a strided array +* @node: Operation list node. +* @src: Address of the source word, or 0 to reset the slot to zero. +* @dst: Base address of the strided destination array. +* @dst_stride: Stride in bytes between consecutive array slots. +* @len: Word length in bytes. Must be 4 or 8 (8 is 64-bit only). +* +* The destination slot is @dst + @dst_stride * index, where index is the +* current CPU ID or MM CID depending on the operation type. +*/ +struct rseq_op_reset_with_stride { +struct rseq_op_node node; +__u64 src; +__u64 dst; +__u64 dst_stride; +__u32 len; +}; +/* * The original size and alignment of the allocation for struct rseq is * 32 bytes. @@ -191,15 +273,21 @@ struct rseq {struct rseq_slice_ctrl slice_ctrl;/* -* Before rseq became extensible, its original size was 32 bytes even -* though the active rseq area was only 20 bytes. -* Exposing a 32 bytes feature size would make life needlessly painful -* for userspace. Therefore, add a reserved byte after byte 32 -* to bump the rseq feature size from 32 to 33.
include/uapi/linux/rseq.h struct rseq { · line 283
Quoted 10 lines
include/uapi/linux/rseq.h +7
-2-* The next field to be added to the rseq area will be larger -* than one byte, and will replace this reserved byte. +* Sentinel of the circular doubly-linked list of rseq operations +* registered via prctl(PR_RSEQ_OP, ...). Fully owned and maintained by +* the kernel: it is initialized to point to itself on registration and +* user space must never read or write it directly. +* +* The kernel only use next and prev from rseq_op_list. The rest of the +* bytes are reserved for later usage and should be zeroed. */
Do we really need the union and the op_used/reserved? Isn't it easier to say that everything needs to be 0'ed? What would userspace do with op_used? It's not possible to skip initialization of these fields when using both memset, struct initialization syntax, and relying on 0 init of global data.
Quoted 14 lines
include/uapi/linux/rseq.h +7
-2-__u8 __reserved; +union { +struct rseq_op_node rseq_op_list; +struct { +__u64 op_used[2 ]; +__u64 reserved; + }; + };/* * Flexible array member at end of structure, after last feature field. -- 2.54.0
Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions
Quoted 201 lines · Dmitry Vyukov
Quoted 31 lines · On Fri, 28 Aug 2026 at 17:33, <[email protected]> wrote:
From: Olivier Dion <[email protected]>
Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.
The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.
Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the
cpu_idbits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.Overall, this puts the users in a situation of choosing between a fast
sched_getcpu()and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibcRSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them.
Add enum
rseq_op_typeand therseq_op_node/rseq_op_reset/rseq_op_reset_with_stridestructures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, theRSEQ_OP_LIST_LIMITwalk bound, and thePR_RSEQ_OPprctl with its REGISTER/UNREGISTER sub-commands.Quoted 4 lines
The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq.
[0]
Documentation/userspace-api/rseq.rst(Optimized RSEQ v2)Hi Olivier,
This is very cool, thanks for working on this.
Quoted 35 lines
Signed-off-by Olivier Dion <[email protected]>--- include/uapi/linux/prctl.h | 12 +++++ include/uapi/linux/rseq.h | 104 ++++++++++++++++++++++++++++++++++--- 2 files changed, 108 insertions(+), 8 deletions(-)include/uapi/linux/prctl.h +12
-0@@ -396,6 +396,18 @@ struct prctl_mm_map {*/ # define PR_RSEQ_SLICE_EXT_ENABLE 0x01 +/* +* RSEQ operation registration. +* +* arg3 is the user address of a struct rseq_op_node embedded in one of the +* rseq operation structures (see uapi/linux/rseq.h). Registering the first +* operation enables rseq operation processing for the thread; unregistering +* the last one disables it. +*/ +#define PR_RSEQ_OP 82 +# define PR_RSEQ_OP_REGISTER 1 +# define PR_RSEQ_OP_UNREGISTER 2 +/* * Get or set the control flow integrity (CFI) configuration for the * current thread. include/uapi/linux/rseq.h +0
-0include/uapi/linux/rseq.h line 19
Quoted 7 lines
include/uapi/linux/rseq.h +4
-0@@ -13,6 +13,11 @@#include <linux/types.h> #include <asm/byteorder.h> +/* +* Maximum number of nodes walked in the rseq operation list. +*/ +#define RSEQ_OP_LIST_LIMIT 2048Does this belong to the user header? Kernel and userspace are not necessary built with the same headers, and we don't necessary want to set this const in stone. I think it may be more flexible to make this impl detail and just return ENOSPC.
Quoted 24 lines
include/uapi/linux/rseq.h +6
-0enum rseq_cpu_id_state {RSEQ_CPU_ID_UNINITIALIZED =-1 ,RSEQ_CPU_ID_REGISTRATION_FAILED =-2 ,@@ -33,6 +38,8 @@ enum rseq_cs_flags_bit {/* User read only feature flags */ RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT =4 ,RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT =5 , +RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT =6 , + RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT = 7, }; enum rseq_cs_flags {@@ -47,6 +54,10 @@ enum rseq_cs_flags { (1U <<RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT ),RSEQ_CS_FLAG_SLICE_EXT_ENABLED = (1U <<RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT ), +RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE = + (1U <<RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT ), +RSEQ_CS_FLAG_RSEQ_OP_ENABLED = + (1U << RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT), };/* include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 106
Quoted 10 lines
include/uapi/linux/rseq.h +7
-0@@ -86,6 +97,77 @@ struct rseq_slice_ctrl { }; }; +/* +* enum rseq_op_type - Type of an rseq operation +* @RSEQ_OP_RESET: Plain reset. Uses struct rseq_op_reset. +* @RSEQ_OP_RESET_WITH_STRIDE_CPUID: Reset indexed by the current CPU ID. +* Uses struct rseq_op_reset_with_stride. +* @RSEQ_OP_RESET_WITH_STRIDE_MMCID: Reset indexed by the current MM CID. +* Uses struct rseq_op_reset_with_stride. Do you have use-cases for these STRIDE ops?
I can think of some, but they would require executing ops when a task is descheduled, rather than when it's scheduled in.Quoted 11 lines
include/uapi/linux/rseq.h +11
-0+*/ +enum rseq_op_type { +RSEQ_OP_RESET , +RSEQ_OP_RESET_WITH_STRIDE_CPUID , +RSEQ_OP_RESET_WITH_STRIDE_MMCID , +RSEQ_OP_NR , +}; + +/* +* struct rseq_op_node - Common header linking an rseq operation into the list +* @next: Address of the next node. Owned by the kernel. include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 127
Quoted 10 lines
include/uapi/linux/rseq.h +10
-0+* @prev: Address of the previous node. Owned by the kernel. +* @type: Operation type. See enum rseq_op_type. +* @reserved: Must be zero on registration. +* +* User space allocates the node, sets @type and zeroes @next, @prev and +* @reserved before passing it to prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, node). +* The kernel owns @next and @prev for the lifetime of the registration and +* links the node into a circular doubly-linked list anchored by an internal +* sentinel in struct rseq. User space must not touch @next or @prev while the +* node is registered. Have you considered keeping copies of these structs in the kernel? That (1) is more consistent with most of the other kernel APIs, (2) will make executing ops faster since we don't need to
copy_from_userall the time, (3) will make code much simpler since we won't need to re-verify all data, (4) will make the code more secure since user-space won't be able to mess with these structs, (5) will simplify the contract and documentation.
There are robust lists that keep lists in user-space, but these are supposed to be concurrently modified by userspace. But these rseq ops are generally not expected to be added/removed often (and the current contract does not allow that anyway).Quoted 4 lines
include/uapi/linux/rseq.h +4
-0+*/ +struct rseq_op_node { +__u64 next; +__u64 prev;include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 141
Quoted 10 lines
include/uapi/linux/rseq.h +10
-0+struct { +__u8 type;/* enum rseq_op_type */ +__u8 reserved[7 ]; + }; +}; + +/* +* struct rseq_op_reset - Reset one word to a value on return to user space +* @node: Operation list node. +* @src: Address of the source word, or 0 to reset @dst to zero. Is "or 0 to reset" part implemented in the series?
It would be useful to see an op that contains a const that is written, rather than an address. Address invovles additional indirection +
copy_from_user.And the contract for concurrent changes of the value pointed by src from user-space are not documented (atomicity?), w/o that having the address is not very useful.
Quoted 12 lines
include/uapi/linux/rseq.h +12
-0+* @dst: Address of the destination word. +* @len: Word length in bytes. Must be 4 or 8 (8 is 64-bit only). +*/ +struct rseq_op_reset { +struct rseq_op_node node; +__u64 src; +__u64 dst; +__u32 len; +}; + +/* +* struct rseq_op_reset_with_stride - Reset one word in a strided array
Quoted 8 lines
include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 161
Quoted 8 lines
include/uapi/linux/rseq.h +8
-0+* @node: Operation list node. +* @src: Address of the source word, or 0 to reset the slot to zero. +* @dst: Base address of the strided destination array. +* @dst_stride: Stride in bytes between consecutive array slots. +* @len: Word length in bytes. Must be 4 or 8 (8 is 64-bit only). +* +* The destination slot is @dst + @dst_stride * index, where index is the +* current CPU ID or MM CID depending on the operation type.
It would be useful to explicitly clarify when is the max value for index.
I understand it's "number of CPUs". But what about online/offline CPUs? Where/how user is expected to read the right value? If I use CIDs and affinity mask, can I expect the index is bounded by the affinity bits?
Quoted 52 lines
Quoted 21 lines
include/uapi/linux/rseq.h +9
-5+*/ +struct rseq_op_reset_with_stride { +struct rseq_op_node node; +__u64 src; +__u64 dst; +__u64 dst_stride; +__u32 len; +}; +/* * The original size and alignment of the allocation for struct rseq is * 32 bytes. @@ -191,15 +273,21 @@ struct rseq {struct rseq_slice_ctrl slice_ctrl;/* -* Before rseq became extensible, its original size was 32 bytes even -* though the active rseq area was only 20 bytes. -* Exposing a 32 bytes feature size would make life needlessly painful -* for userspace. Therefore, add a reserved byte after byte 32 -* to bump the rseq feature size from 32 to 33. include/uapi/linux/rseq.h struct rseq { · line 283
Quoted 10 lines
include/uapi/linux/rseq.h +7
-2-* The next field to be added to the rseq area will be larger -* than one byte, and will replace this reserved byte. +* Sentinel of the circular doubly-linked list of rseq operations +* registered via prctl(PR_RSEQ_OP, ...). Fully owned and maintained by +* the kernel: it is initialized to point to itself on registration and +* user space must never read or write it directly. +* +* The kernel only use next and prev from rseq_op_list. The rest of the +* bytes are reserved for later usage and should be zeroed. */ Do we really need the union and the op_used/reserved? Isn't it easier to say that everything needs to be 0'ed? What would userspace do with
op_used? It's not possible to skip initialization of these fields when using both memset, struct initialization syntax, and relying on 0 init of global data.Quoted 14 lines
include/uapi/linux/rseq.h +7
-2-__u8 __reserved; +union { +struct rseq_op_node rseq_op_list; +struct { +__u64 op_used[2 ]; +__u64 reserved; + }; + };/* * Flexible array member at end of structure, after last feature field. -- 2.54.0
Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions
In reply to [email protected]Quoted 11 lines
Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.
The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.
Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the
cpu_idbits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.
The commit message does not quite say how this addresses tcmalloc needs. I assume the idea is to reset some other data structure and not the rseq fields. Is my assumption correct?
Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.
My concern is that this would turn into another bytecode interpreter over time, basically reimplementing BPF.
Thanks,
Florian
Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions
On Mon, 31 Aug 2026, Florian Weimer <[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 5 lines
Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the
cpu_idbits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.The commit message does not quite say how this addresses tcmalloc needs. I assume the idea is to reset some other data structure and not the rseq fields. Is my assumption correct?
Indeed. That would allow TCMalloc to not have to overlap their pointer cache with their RSEQ area. In other words, they could be using the glibc area and be compatible with the rest of the ecosystem.
Quoted 3 lines
Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.
Well I had in mind another solution, albeit some might find it ugly but I think it does have the merit to be explored.
Basically, instead of doing all the operations in the kernel, we could setup something akin to a signal handler by pushing a frame on the userspace stack and resuming the execution of the thread into that "handler". Userspace is now responsible of everything, including returning to the original IP.
However, we need to be careful with things like the red-zone, the shadow-track and the extended states. Other than that, the only thing I am unsure is how to handle nesting. Perhaps someone with more knowledges could expand on that.
Quoted 2 lines
My concern is that this would turn into another bytecode interpreter over time, basically reimplementing BPF.
I did think about adding BPF type to run BPF bytecode. However, I don't think it fits well the goal here and would make the feature unusable on configurations where BPF is not available.
Overall, my idea with the RSEQ operations was make that could benifit the userspace as a whole instead of handling only the TCMalloc situation. Otherwise, I would have just name it RSEQ reset.
For example, there is a CPU-stride variant of the reset operation. I dabble with this operation to cache a TLS that is global-dynamic into a per-cpu cache. Thus, making a TLS global-dynamic access as fast as a TLS initial-exec.
Thanks,
Olivier
Signature · 3 lines
Olivier Dion EfficiOS Inc. https://www.efficios.com
Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions
* Olivier Dion:
Quoted 4 lines
Quoted 3 lines
Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.
Quoted 8 lines
Well I had in mind another solution, albeit some might find it ugly but I think it does have the merit to be explored.
Basically, instead of doing all the operations in the kernel, we could setup something akin to a signal handler by pushing a frame on the userspace stack and resuming the execution of the thread into that "handler". Userspace is now responsible of everything, including returning to the original IP.
On mere context switch (not signal delivery), I don't think the kernel assumes that there is a valid stack pointer. These scheme would change that. This would be fine for managed code, but this scheme has to work with arbitrary code that reassigns registers temporarily.
(We had a longjmp on POWER for a while that corrupted the stack pointer temporarily, and that was visible only upon signal delivery, not regular context switch.)
Quoted 4 lines
For example, there is a CPU-stride variant of the reset operation. I dabble with this operation to cache a TLS that is global-dynamic into a per-cpu cache. Thus, making a TLS global-dynamic access as fast as a TLS initial-exec.
I don't think this can work? The problem with global-dynamic TLS is that it can't be a single memory block (it may have to grow). Even if the access path changes, we can't relocate the memory because TLS objects must live in the linear address space.
We can do away with the current complicated scheme in glibc and use an array of arrays, where the inner arrays follow an exponential growth policy (for size and alignment). That would still have dependent loads, but would be quite a bit faster than our current global-dynamic scheme.
New TLS allocator for glibc
<https://conf.gnu-tools-cauldron.org/opo25/talk/LQTU3G/>
(Sadly still not implemented.)
Thanks,
Florian
Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions
Quoted 13 lines · Florian Weimer
Quoted 4 lines
Quoted 3 lines
Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.
Quoted 8 lines
Well I had in mind another solution, albeit some might find it ugly but I think it does have the merit to be explored.
Basically, instead of doing all the operations in the kernel, we could setup something akin to a signal handler by pushing a frame on the userspace stack and resuming the execution of the thread into that "handler". Userspace is now responsible of everything, including returning to the original IP.
Quoted 4 lines
On mere context switch (not signal delivery), I don't think the kernel assumes that there is a valid stack pointer. These scheme would change that. This would be fine for managed code, but this scheme has to work with arbitrary code that reassigns registers temporarily.
You are right that aside of signal handling there is no assumption, but as signal handling can happen at arbitrary times user space should not play with SP unless there is an alternative signal stack, which has its own nasties as we all know.
If there is truly code which needs to trainwreck SP for a sane reason and not just because and has a good reason to utilize such a library/functionality, then we can work around that by a mandating a separate "fixup" stack per thread to enable this functionality. Then the screwed up user space stack pointer becomes completely irrelevant and the only interesting register on the kernel entry stack is the return IP which got pushed by the hardware. And provide an option to work without this indirection of course. :)
There are probably other solutions, but I really want to move this to user space and not introduce yet another limited single purpose "interpreter" in the kernel, which will be "extended" and duct taped forever.
TBH. At some point we have to stop pretending that we can support all possible insanities and their arbitrary combination. That just limits ourself and puts the burden and the complexity into the wrong places just because.
Thanks,
tglx
RFC PATCH 2/5 rseq: add per-task rseq operation state
In reply to [email protected]From: Olivier Dion <[email protected]>
Add the task-side state for rseq operations: the rseq_event::rseq_op state indicating operation processing is enabled for the task, and the nr_ops counter in struct rseq_data. nr_ops is edge triggered: its 0<->1 transition enables/disables rseq_event::rseq_op and the user visible RSEQ_CS_FLAG_RSEQ_OP_ENABLED flag. This makes so that only thread that enable RSEQ operations through prctl takes a performance hit.
Include rseq_op in the sched-switch raise condition so pending operations force a return through the rseq exit path, and declare the rseq_op_prctl() entry point (with a CONFIG_RSEQ=n stub).
--- include/linux/rseq.h | 10 +++++++++- include/linux/rseq_types.h | 9 +++++++++ 2 files changed, 18 insertions(+), 1 deletion(-)
include/linux/rseq.h +9 -1
@@ -69,7 +69,9 @@ static __always_inline void rseq_sched_switch_event(struct task_struct *t)* was via interrupt from user space. ev->has_rseq does not have * to be evaluated here because rseq_v2() implies has_rseq. */ -bool raise = ev->user_irq | ev->ids_changed; +bool raise = (ev->user_irq | + ev->ids_changed | + ev->rseq_op);if (raise) { ev->sched_switch = true;
@@ -172,6 +174,8 @@ static inline unsigned int rseq_alloc_align(void) return 1U << get_count_order(offsetof(struct rseq, end)); } +int rseq_op_prctl (unsigned long arg2,unsigned long arg3); +#else /* CONFIG_RSEQ */ static inline bool rseq_v2 (struct task_struct *t) {return false; }static inline void rseq_handle_slowpath (struct pt_regs *regs) { }
@@ -182,6 +186,10 @@ static inline void rseq_force_update(void) { }static inline void rseq_virt_userspace_exit (void ) { }static inline void rseq_fork (struct task_struct *t,u64 clone_flags) { }static inline void rseq_execve (struct task_struct *t) { } +static inline int rseq_op_prctl (unsigned long arg2,unsigned long arg3) +{ +return -ENOTSUPP ; +}#endif /* !CONFIG_RSEQ */ #ifdef CONFIG_DEBUG_RSEQ
include/linux/rseq_types.h +9 -0
@@ -23,6 +23,7 @@ struct rseq;* exit to user * @ids_changed: Indicator that IDs need to be updated * @user_irq: True on interrupt entry from user mode +* @rseq_op: Rseq operation processing is enabled for the task * @has_rseq: Greater than 0 if the task has a rseq pointer installed. * Contains the RSEQ version number * @error: Compound error code for the slow path to analyze
@@ -44,6 +45,7 @@ struct rseq_event {u8 sched_switch;u8 ids_changed;u8 user_irq; +u8 rseq_op; }; };
@@ -115,6 +117,7 @@ struct rseq_slice {* @event: Storage for event management * @ids: Storage for cached CPU ID and MM CID * @slice: Storage for time slice extension data +* @nr_ops: Number of registered rseq operations */ struct rseq_data {struct rseq __user *usrptr;
@@ -125,6 +128,12 @@ struct rseq_data {#ifdef CONFIG_RSEQ_SLICE_EXTENSION struct rseq_slice slice;#endif +/* +* Number of rseq operations registered for the task. Edge triggered: +* the 0<->1 transition enables/disables rseq_event::rseq_op and the +* RSEQ_CS_FLAG_RSEQ_OP_ENABLED user flag. +*/ + u32 nr_ops; };#else /* CONFIG_RSEQ */
Signature · 1 line
2.54.0
Re: [RFC PATCH 2/5] rseq: add per-task rseq operation state
Quoted 84 lines · On Fri, 28 Aug 2026 at 17:34, <[email protected]> wrote:
From: Olivier Dion <[email protected]>
Add the task-side state for rseq operations: the
rseq_event::rseq_opstate indicating operation processing is enabled for the task, and thenr_opscounter in structrseq_data.nr_opsis edge triggered: its 0<->1 transition enables/disablesrseq_event::rseq_opand the user visibleRSEQ_CS_FLAG_RSEQ_OP_ENABLEDflag. This makes so that only thread that enable RSEQ operations through prctl takes a performance hit.Include
rseq_opin the sched-switch raise condition so pending operations force a return through the rseq exit path, and declare therseq_op_prctl()entry point (with aCONFIG_RSEQ=nstub).Signed-off-by Olivier Dion <[email protected]>--- include/linux/rseq.h | 10 +++++++++- include/linux/rseq_types.h | 9 +++++++++ 2 files changed, 18 insertions(+), 1 deletion(-)include/linux/rseq.h +9
-1@@ -69,7 +69,9 @@ static __always_inline void rseq_sched_switch_event(struct task_struct *t)* was via interrupt from user space. ev->has_rseq does not have * to be evaluated here because rseq_v2() implies has_rseq. */ -bool raise = ev->user_irq | ev->ids_changed; +bool raise = (ev->user_irq | + ev->ids_changed | + ev->rseq_op);if (raise) { ev->sched_switch = true;@@ -172,6 +174,8 @@ static inline unsigned int rseq_alloc_align(void) return 1U << get_count_order(offsetof(struct rseq, end)); } +int rseq_op_prctl (unsigned long arg2,unsigned long arg3); +#else /* CONFIG_RSEQ */ static inline bool rseq_v2 (struct task_struct *t) {return false; }static inline void rseq_handle_slowpath (struct pt_regs *regs) { }@@ -182,6 +186,10 @@ static inline void rseq_force_update(void) { }static inline void rseq_virt_userspace_exit (void ) { }static inline void rseq_fork (struct task_struct *t,u64 clone_flags) { }static inline void rseq_execve (struct task_struct *t) { } +static inline int rseq_op_prctl (unsigned long arg2,unsigned long arg3) +{ +return -ENOTSUPP ; +}#endif /* !CONFIG_RSEQ */ #ifdef CONFIG_DEBUG_RSEQ include/linux/rseq_types.h +3
-0@@ -23,6 +23,7 @@ struct rseq;* exit to user * @ids_changed: Indicator that IDs need to be updated * @user_irq: True on interrupt entry from user mode +* @rseq_op: Rseq operation processing is enabled for the task * @has_rseq: Greater than 0 if the task has a rseq pointer installed. * Contains the RSEQ version number * @error: Compound error code for the slow path to analyze @@ -44,6 +45,7 @@ struct rseq_event {u8 sched_switch;u8 ids_changed;u8 user_irq; +u8 rseq_op; }; };@@ -115,6 +117,7 @@ struct rseq_slice {* @event: Storage for event management * @ids: Storage for cached CPU ID and MM CID * @slice: Storage for time slice extension data +* @nr_ops: Number of registered rseq operations */ struct rseq_data {struct rseq __user *usrptr;
include/linux/rseq_types.h struct rseq_data { · line 136
Quoted 9 lines
include/linux/rseq_types.h +6
-0@@ -125,6 +128,12 @@ struct rseq_data {#ifdef CONFIG_RSEQ_SLICE_EXTENSION struct rseq_slice slice;#endif +/* +* Number of rseq operations registered for the task. Edge triggered: +* the 0<->1 transition enables/disables rseq_event::rseq_op and the +* RSEQ_CS_FLAG_RSEQ_OP_ENABLED user flag. +*/ +u32 nr_ops;
If we copy ops to kernel memory, it would be useful to have 1 op embeded here (with fallback array for more ops). For now the only known use-case for this is memory allocators, so I would assume in most cases it will be either 0 or 1 ops registered.
Quoted 6 lines
include/linux/rseq_types.h +0
-1};#else /* CONFIG_RSEQ */ --2.54.0
RFC PATCH 3/5 rseq: apply operations on exit to user space
In reply to [email protected]From: Olivier Dion <[email protected]>
Add rseq_apply_ops(), which walks the task's circular operation list on return to user space and applies each operation.
Wire the walk into rseq_exit_user_update() for both the ids-unchanged and ids-changed paths, gated on rseq_event::rseq_op. Since operation processing is a persistent per-task state rather than a one-shot event, introduce rseq_clear_one_shot_events() which preserves the rseq_op bit when clearing events on the way out.
len, src and dst are re-read from user memory and re-validated on every application, since user space may change them after registration. A broken ABI contract will fault the process.
--- include/linux/rseq_entry.h | 157 +++++++++++++++++++++++++++++++++++-- 1 file changed, 152 insertions(+), 5 deletions(-)
include/linux/rseq_entry.h +152 -5
@@ -247,6 +247,137 @@ static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un#define rseq_slice_clear_user (rseq, efault) do { } while (0)#endif /* !CONFIG_RSEQ_SLICE_EXTENSION */ +/* +* Validate and perform a single reset word operation: reset @dst to *@src, +* or to zero when @src is 0. @len, @src and @dst come from user memory and +* are re-validated on every application, since user space may have changed +* them since registration. A broken ABI contract returns false, which faults +* the process. +*/ +static __always_inline bool rseq_op_reset_word(u64 src, u64 dst, u32 len, +void __user *node) +{ +if (unlikely (!IS_ALIGNED (dst, len) || (src && !IS_ALIGNED (src, len)))) { +pr_info_ratelimited ("rseq: bad operation alignment len=%u src=%llx dst=%llx from %p\n" , + len, src, dst, node); +return false; + } + +switch (len) { +case 4 : { +u32 __user *udst = (u32 __user *)dst; +u32 v =0 ; + +if (src) { +u32 __user *usrc = (u32 __user *)src; + + scoped_user_read_access(usrc, efault) +unsafe_get_user (v, usrc, efault); + } + scoped_user_write_access(udst, efault) +unsafe_put_user (v, udst, efault); +return true; + } +case 8 : { +u64 __user *udst = (u64 __user *)dst; +u64 v =0 ; + +if (src) { +u64 __user *usrc = (u64 __user *)src; + + scoped_user_read_access(usrc, efault) +unsafe_get_user (v, usrc, efault); + } + scoped_user_write_access(udst, efault) +unsafe_put_user (v, udst, efault); +return true; + } +default : +pr_info ("rseq: bad operation length=%u from %p\n" , len, node); +return false; + } +efault: +pr_info ("rseq: fault while applying operation from %p\n" , node); +return false; +} + +static __always_inline bool rseq_apply_ops(struct task_struct *t) +{ +struct rseq __user *rseq = t->rseq.usrptr; +struct rseq_op_node __user *sentinel = &rseq->rseq_op_list; +struct rseq_op_node __user *node; +unsigned int limit =RSEQ_OP_LIST_LIMIT ; +u64 next; + +WARN_ONCE (!t->rseq.event.rseq_op,"rseq operations not enabled" ); + + scoped_user_read_access(sentinel, efault) +unsafe_get_user (next, &sentinel->next, efault); + + node = (struct rseq_op_node __user *)next; + +while (node != sentinel && limit--) { +u8 type; + + scoped_user_read_access(node, efault) { +unsafe_get_user (type, &node->type, efault); +unsafe_get_user (next, &node->next, efault); + } + +switch (type) { +case RSEQ_OP_RESET : { +struct rseq_op_reset __user *op = + (struct rseq_op_reset __user *)node; +u64 src, dst; +u32 len; + + scoped_user_read_access(op, efault) { +unsafe_get_user (src, &op->src, efault); +unsafe_get_user (dst, &op->dst, efault); +unsafe_get_user (len, &op->len, efault); + } + +if (!rseq_op_reset_word (src, dst, len, node)) +return false; +break ; + } +case RSEQ_OP_RESET_WITH_STRIDE_CPUID :/* fall through */ +case RSEQ_OP_RESET_WITH_STRIDE_MMCID : { +struct rseq_op_reset_with_stride __user *op = + (struct rseq_op_reset_with_stride __user *)node; +u64 src, dst, dst_stride; +u32 len, index; + + scoped_user_read_access(op, efault) { +unsafe_get_user (src, &op->src, efault); +unsafe_get_user (dst, &op->dst, efault); +unsafe_get_user (dst_stride, &op->dst_stride, efault); +unsafe_get_user (len, &op->len, efault); + } + index = (type ==RSEQ_OP_RESET_WITH_STRIDE_CPUID ) ? + t->rseq.ids.cpu_id : t->rseq.ids.mm_cid; + dst += dst_stride * index; + +if (!rseq_op_reset_word (src, dst, len, node)) +return false; + +break ; + } +default : +pr_info ("rseq: bad operation type=%u from %p\n" , + type, node); +return false; + } + + node = (struct rseq_op_node __user *)next; + } + +return node == sentinel; +efault: +pr_info ("rseq: fault while walking operation list\n" ); +return false; +} +bool rseq_debug_update_user_cs (struct task_struct *t,struct pt_regs *regs,unsigned long csaddr); static __always_inline void rseq_note_user_irq_entry(void)
@@ -632,6 +763,10 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct tif (unlikely (!rseq_update_user_cs (t, regs, csaddr)))return false; } + +if (t->rseq.event.rseq_op && !rseq_apply_ops (t)) +return false; +return true; }
@@ -642,11 +777,22 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct t .node_id = cpu_to_node(cpu), }; -return rseq_update_usr (t, regs, &ids); +if (!rseq_update_usr (t, regs, &ids)) +return false; + +if (t->rseq.event.rseq_op && !rseq_apply_ops (t)) +return false; + +return true; efault:return false; } +static __always_inline void rseq_clear_one_shot_events(struct rseq_event *ev) +{ + ev->events &= (struct rseq_event ){ .rseq_op = true }.events; +} + static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *regs) {struct task_struct *t = current;
@@ -674,8 +820,9 @@ static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *regif (unlikely (!rseq_exit_user_update (regs, t)))return true; } -/* Clear state so next entry starts from a clean slate */ - t->rseq.event.events =0 ; +/* Clear one-shot events so next entry starts from a clean slate */ +rseq_clear_one_shot_events (&t->rseq.event); +return false; }
@@ -730,7 +877,7 @@ static __always_inline void rseq_syscall_exit_to_user_mode(void)/* Needed to remove the store for the !lockdep case */ if (IS_ENABLED (CONFIG_LOCKDEP )) {WARN_ON_ONCE (ev->sched_switch); - ev->events =0 ; +rseq_clear_one_shot_events (ev); } }
@@ -747,7 +894,7 @@ static __always_inline void rseq_irqentry_exit_to_user_mode(void)* interrupt did not result in a schedule and therefore the * rseq processing could not clear it. */ - ev->events =0 ; +rseq_clear_one_shot_events (ev); }void __rseq_debug_syscall_return (struct pt_regs *regs);
Signature · 1 line
2.54.0
Re: [RFC PATCH 3/5] rseq: apply operations on exit to user space
Quoted 66 lines · On Fri, 28 Aug 2026 at 17:34, <[email protected]> wrote:
From: Olivier Dion <[email protected]>
Add
rseq_apply_ops(), which walks the task's circular operation list on return to user space and applies each operation.Wire the walk into
rseq_exit_user_update()for both the ids-unchanged and ids-changed paths, gated onrseq_event::rseq_op. Since operation processing is a persistent per-task state rather than a one-shot event, introducerseq_clear_one_shot_events()which preserves therseq_opbit when clearing events on the way out.len, src and dst are re-read from user memory and re-validated on every application, since user space may change them after registration. A broken ABI contract will fault the process.
Signed-off-by Olivier Dion <[email protected]>--- include/linux/rseq_entry.h | 157 +++++++++++++++++++++++++++++++++++-- 1 file changed, 152 insertions(+), 5 deletions(-)include/linux/rseq_entry.h +37
-0@@ -247,6 +247,137 @@ static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un#define rseq_slice_clear_user (rseq, efault) do { } while (0)#endif /* !CONFIG_RSEQ_SLICE_EXTENSION */ +/* +* Validate and perform a single reset word operation: reset @dst to *@src, +* or to zero when @src is 0. @len, @src and @dst come from user memory and +* are re-validated on every application, since user space may have changed +* them since registration. A broken ABI contract returns false, which faults +* the process. +*/ +static __always_inline bool rseq_op_reset_word(u64 src, u64 dst, u32 len, +void __user *node) +{ +if (unlikely (!IS_ALIGNED (dst, len) || (src && !IS_ALIGNED (src, len)))) { +pr_info_ratelimited ("rseq: bad operation alignment len=%u src=%llx dst=%llx from %p\n" , + len, src, dst, node); +return false; + } + +switch (len) { +case 4 : { +u32 __user *udst = (u32 __user *)dst; +u32 v =0 ; + +if (src) { +u32 __user *usrc = (u32 __user *)src; + + scoped_user_read_access(usrc, efault) +unsafe_get_user (v, usrc, efault); + } + scoped_user_write_access(udst, efault) +unsafe_put_user (v, udst, efault); +return true; + } +case 8 : { +u64 __user *udst = (u64 __user *)dst; +u64 v =0 ; + +if (src) { +u64 __user *usrc = (u64 __user *)src;
include/linux/rseq_entry.h static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un · line 296
Quoted 10 lines
include/linux/rseq_entry.h +10
-0+ + scoped_user_read_access(usrc, efault) +unsafe_get_user (v, usrc, efault); + } + scoped_user_write_access(udst, efault) +unsafe_put_user (v, udst, efault); +return true; + } +default : +pr_info ("rseq: bad operation length=%u from %p\n" , len, node);
pr_info_ratelimited here and below.
Kernel does not generally warn on the global console about misbehaving userspace programs (which generally do not have access to the output anyway, as it requires root nowadays).
If you decide to do that, it should be at least rate-limited to avoid DoS.
Quoted 56 lines
include/linux/rseq_entry.h +56
-0+return false; + } +efault: +pr_info ("rseq: fault while applying operation from %p\n" , node); +return false; +} + +static __always_inline bool rseq_apply_ops(struct task_struct *t) +{ +struct rseq __user *rseq = t->rseq.usrptr; +struct rseq_op_node __user *sentinel = &rseq->rseq_op_list; +struct rseq_op_node __user *node; +unsigned int limit =RSEQ_OP_LIST_LIMIT ; +u64 next; + +WARN_ONCE (!t->rseq.event.rseq_op,"rseq operations not enabled" ); + + scoped_user_read_access(sentinel, efault) +unsafe_get_user (next, &sentinel->next, efault); + + node = (struct rseq_op_node __user *)next; + +while (node != sentinel && limit--) { +u8 type; + + scoped_user_read_access(node, efault) { +unsafe_get_user (type, &node->type, efault); +unsafe_get_user (next, &node->next, efault); + } + +switch (type) { +case RSEQ_OP_RESET : { +struct rseq_op_reset __user *op = + (struct rseq_op_reset __user *)node; +u64 src, dst; +u32 len; + + scoped_user_read_access(op, efault) { +unsafe_get_user (src, &op->src, efault); +unsafe_get_user (dst, &op->dst, efault); +unsafe_get_user (len, &op->len, efault); + } + +if (!rseq_op_reset_word (src, dst, len, node)) +return false; +break ; + } +case RSEQ_OP_RESET_WITH_STRIDE_CPUID :/* fall through */ +case RSEQ_OP_RESET_WITH_STRIDE_MMCID : { +struct rseq_op_reset_with_stride __user *op = + (struct rseq_op_reset_with_stride __user *)node; +u64 src, dst, dst_stride; +u32 len, index; + + scoped_user_read_access(op, efault) { +unsafe_get_user (src, &op->src, efault);
include/linux/rseq_entry.h static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un · line 362
Quoted 10 lines
include/linux/rseq_entry.h +10
-0+unsafe_get_user (dst, &op->dst, efault); +unsafe_get_user (dst_stride, &op->dst_stride, efault); +unsafe_get_user (len, &op->len, efault); + } + index = (type ==RSEQ_OP_RESET_WITH_STRIDE_CPUID ) ? + t->rseq.ids.cpu_id : t->rseq.ids.mm_cid; + dst += dst_stride * index; + +if (!rseq_op_reset_word (src, dst, len, node)) +return false;
Declaration of src/dst/len and rseq_op_reset_word call potentially can be deduplicated across these 2 cases.
Quoted 89 lines
include/linux/rseq_entry.h +39
-6+ +break ; + } +default : +pr_info ("rseq: bad operation type=%u from %p\n" , + type, node); +return false; + } + + node = (struct rseq_op_node __user *)next; + } + +return node == sentinel; +efault: +pr_info ("rseq: fault while walking operation list\n" ); +return false; +} +bool rseq_debug_update_user_cs (struct task_struct *t,struct pt_regs *regs,unsigned long csaddr); static __always_inline void rseq_note_user_irq_entry(void)@@ -632,6 +763,10 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct tif (unlikely (!rseq_update_user_cs (t, regs, csaddr)))return false; } + +if (t->rseq.event.rseq_op && !rseq_apply_ops (t)) +return false; +return true; }@@ -642,11 +777,22 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct t .node_id = cpu_to_node(cpu), }; -return rseq_update_usr (t, regs, &ids); +if (!rseq_update_usr (t, regs, &ids)) +return false; + +if (t->rseq.event.rseq_op && !rseq_apply_ops (t)) +return false; + +return true; efault:return false; } +static __always_inline void rseq_clear_one_shot_events(struct rseq_event *ev) +{ + ev->events &= (struct rseq_event ){ .rseq_op = true }.events; +} + static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *regs) {struct task_struct *t = current;@@ -674,8 +820,9 @@ static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *regif (unlikely (!rseq_exit_user_update (regs, t)))return true; } -/* Clear state so next entry starts from a clean slate */ - t->rseq.event.events =0 ; +/* Clear one-shot events so next entry starts from a clean slate */ +rseq_clear_one_shot_events (&t->rseq.event); +return false; }@@ -730,7 +877,7 @@ static __always_inline void rseq_syscall_exit_to_user_mode(void)/* Needed to remove the store for the !lockdep case */ if (IS_ENABLED (CONFIG_LOCKDEP )) {WARN_ON_ONCE (ev->sched_switch); - ev->events =0 ; +rseq_clear_one_shot_events (ev); } }@@ -747,7 +894,7 @@ static __always_inline void rseq_irqentry_exit_to_user_mode(void)* interrupt did not result in a schedule and therefore the * rseq processing could not clear it. */ - ev->events =0 ; +rseq_clear_one_shot_events (ev); }void __rseq_debug_syscall_return (struct pt_regs *regs); --2.54.0
RFC PATCH 4/5 rseq: register and unregister operations via prctl
In reply to [email protected]From: Olivier Dion <[email protected]>
Implement the PR_RSEQ_OP prctl. Registration splices a user-provided pristine node into the head of the circular operation list, and unregistration unsplices it after validating it is properly linked between its neighbours; the kernel owns the node's next/prev links for the lifetime of the registration. The 0<->1 nr_ops transition reflects the enabled state into rseq_event::rseq_op and the user visible flag via rseq_op_update_enabled().
Initialize the operation list sentinel as an empty self-pointing circular list on rseq registration, and reset nr_ops / rseq_op state there.
Apply pending operations on the slow path and on signal delivery, and factor the common "clear error and force SIGSEGV" fixup into force_fault().
--- kernel/rseq.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++--- kernel/sys.c | 5 ++ 2 files changed, 212 insertions(+), 13 deletions(-)
kernel/rseq.c +207 -13
@@ -253,17 +253,18 @@ static bool rseq_handle_cs(struct task_struct *t, struct pt_regs *regs)static void rseq_slowpath_update_usr (struct pt_regs *regs) {/* -* Preserve has_rseq and user_irq state. The generic entry code clears -* user_irq on the way out, the non-generic entry architectures are not -* setting user_irq. +* Preserve has_rseq, rseq_op and user_irq state. The generic entry +* code clears user_irq on the way out, the non-generic entry +* architectures are not setting user_irq. */ const struct rseq_event evt_mask = { .has_rseq =RSEQ_HAS_RSEQ_VERSION_MASK , + .rseq_op = true, .user_irq = true, };struct task_struct *t = current;struct rseq_ids ids; -bool event; +bool event, should_fault = false;if (unlikely (t->flags &PF_EXITING ))return ;
@@ -300,7 +301,12 @@ static void rseq_slowpath_update_usr(struct pt_regs *regs) ids.node_id =cpu_to_node (ids.cpu_id); -if (unlikely (!rseq_update_usr (t, regs, &ids))) { +if (unlikely (!rseq_update_usr (t, regs, &ids))) + should_fault = true; +else if (t->rseq.event.rseq_op &&unlikely (!rseq_apply_ops (t))) + should_fault = true; + +if (should_fault) {/* * Clear the errors just in case this might survive magically, but * leave the rest intact.
@@ -329,8 +335,20 @@ void __rseq_handle_slowpath(struct pt_regs *regs)rseq_slowpath_update_usr (regs); } +static inline void force_fault (int sig) +{ +/* +* Clear the errors just in case this might survive magically, but leave +* the rest intact. +*/ + current->rseq.event.error =0 ; +force_sigsegv (sig); +} +void __rseq_signal_deliver (int sig,struct pt_regs *regs) { +bool should_fault = false; +rseq_stat_inc (rseq_stats.signal);/*
@@ -339,14 +357,13 @@ void __rseq_signal_deliver(int sig, struct pt_regs *regs)* the interrupted context as after this point the instruction * pointer in @regs points to the signal handler. */ -if (unlikely (!rseq_handle_cs (current, regs))) { -/* -* Clear the errors just in case this might survive -* magically, but leave the rest intact. -*/ - current->rseq.event.error =0 ; -force_sigsegv (sig); - } +if (unlikely (!rseq_handle_cs (current, regs))) + should_fault = true; +else if (current->rseq.event.rseq_op &&unlikely (!rseq_apply_ops (current))) + should_fault = true; + +if (should_fault) +force_fault (sig);/* * In legacy mode, force the update of IDs before returning to user
@@ -436,6 +453,8 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3 rseqfl |=RSEQ_CS_FLAG_SLICE_EXT_ENABLED ; } } +if (version >1 ) + rseqfl |=RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE ; scoped_user_write_access(rseq, efault) {/*
@@ -458,8 +477,16 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3* registrations. */ if (version >1 ) { +u64 sentinel = (u64)&rseq->rseq_op_list; +if (IS_ENABLED (CONFIG_RSEQ_SLICE_EXTENSION ))unsafe_put_user (0U , &rseq->slice_ctrl.all, efault); +/* +* Initialize the rseq operation list sentinel as an +* empty circular doubly-linked list pointing to itself. +*/ +unsafe_put_user (sentinel, &rseq->rseq_op_list.next, efault); +unsafe_put_user (sentinel, &rseq->rseq_op_list.prev, efault); } }
@@ -474,6 +501,12 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3#ifdef CONFIG_RSEQ_SLICE_EXTENSION current->rseq.slice.state.enabled = !!(rseqfl &RSEQ_CS_FLAG_SLICE_EXT_ENABLED );#endif +/* +* A fresh registration starts with no operations, so operation +* processing is disabled until the first one is registered. +*/ + current->rseq.event.rseq_op = false; + current->rseq.nr_ops =0 ;/* * Ensure the cpu_id_start and cpu_id fields are updated before
@@ -887,3 +920,164 @@ device_initcall(rseq_slice_init);#else static void rseq_slice_ext_init (struct dentry *root_dir) { }#endif /* CONFIG_RSEQ_SLICE_EXTENSION */ + +/* +* Reflect the operation enabled state into the user visible flags field. +* Called on the 0<->1 transition of nr_ops. The kill path is taken on fault +* because losing this update leaves user space and kernel state inconsistent. +*/ +static int rseq_op_update_enabled (struct task_struct *t,bool enable) +{ +struct rseq __user *rseq = t->rseq.usrptr; +u32 rflags; + +if (get_user (rflags, &rseq->flags)) +return -EFAULT ; + + rflags &= ~RSEQ_CS_FLAG_RSEQ_OP_ENABLED ; + rflags |=RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE ; +if (enable) + rflags |=RSEQ_CS_FLAG_RSEQ_OP_ENABLED ; + +if (put_user (rflags, &rseq->flags)) +return -EFAULT ; + + t->rseq.event.rseq_op = enable; +return 0 ; +} + +/* +* Register @node at the head of the circular doubly-linked operation list +* anchored by the kernel owned sentinel in struct rseq. +*/ +static int rseq_op_register (struct task_struct *t,struct rseq_op_node __user *node) +{ +struct rseq_op_node __user *sentinel = &t->rseq.usrptr->rseq_op_list; +struct rseq_op_node __user *first; +u64 next, prev, first_addr; +u8 type, i; + +if (t->rseq.nr_ops >=RSEQ_OP_LIST_LIMIT ) +return -ENOSPC ; + +if (!IS_ALIGNED ((unsigned long )node, __alignof__(struct rseq_op_node ))) +return -EINVAL ; +if (!access_ok (node,sizeof (*node))) +return -EFAULT ; + +/* +* The node links are owned by the kernel. User space must present a +* pristine node: next, prev and the reserved bytes all zeroed, and a +* known operation type. +*/ +if (get_user (next, &node->next) ||get_user (prev, &node->prev) || +get_user (type, &node->type)) +return -EFAULT ; +if (next || prev) +return -EINVAL ; +if (type >=RSEQ_OP_NR ) +return -EINVAL ; +for (i =1 ; i <sizeof (node->reserved); i++) { +u8 r; + +if (get_user (r, &node->reserved[i])) +return -EFAULT ; +if (r) +return -EINVAL ; + } + +/* Splice the node in right after the sentinel. */ +if (get_user (first_addr, &sentinel->next)) + goto die; + first = (struct rseq_op_node __user *)first_addr; + +if (put_user ((u64 )(unsigned long )first, &node->next) || +put_user ((u64 )(unsigned long )sentinel, &node->prev) || +put_user ((u64 )(unsigned long )node, &first->prev) || +put_user ((u64 )(unsigned long )node, &sentinel->next)) + goto die; + + t->rseq.nr_ops +=1 ; + +if (t->rseq.nr_ops ==1 ) +return rseq_op_update_enabled (t, true) ? -EFAULT :0 ; +return 0 ; +die: +force_sig (SIGSEGV ); +return -EFAULT ; +} + +/* +* Unregister @node from the operation list. The node links are validated +* against its neighbours to reject bogus or double unregistration. +*/ +static int rseq_op_unregister (struct task_struct *t,struct rseq_op_node __user *node) +{ +struct rseq_op_node __user *prev, *next; +u64 prev_addr, next_addr, tmp; + +if (!t->rseq.nr_ops) +return -ENOENT ; + +if (!IS_ALIGNED ((unsigned long )node, __alignof__(struct rseq_op_node ))) +return -EINVAL ; +if (!access_ok (node,sizeof (*node))) +return -EFAULT ; + +if (get_user (next_addr, &node->next) ||get_user (prev_addr, &node->prev)) +return -EFAULT ; + prev = (struct rseq_op_node __user *)prev_addr; + next = (struct rseq_op_node __user *)next_addr; + +/* A registered node always has both links set. */ +if (!prev || !next) +return -EINVAL ; + +/* Verify the node is properly linked between its neighbours. */ +if (get_user (tmp, &prev->next)) + goto die; +if (tmp != (u64 )(unsigned long )node) +return -EINVAL ; +if (get_user (tmp, &next->prev)) + goto die; +if (tmp != (u64 )(unsigned long )node) +return -EINVAL ; + +/* Unsplice and clear the node links so it can be reused. */ +if (put_user (next_addr, &prev->next) || +put_user (prev_addr, &next->prev) || +put_user (0ULL , &node->next) || +put_user (0ULL , &node->prev)) + goto die; + + t->rseq.nr_ops -=1 ; + +if (t->rseq.nr_ops ==0 ) +return rseq_op_update_enabled (t, false) ? -EFAULT :0 ; +return 0 ; +die: +force_sig (SIGSEGV ); +return -EFAULT ; +} + +int rseq_op_prctl (unsigned long arg2,unsigned long arg3) +{ +struct rseq_op_node __user *node = (struct rseq_op_node __user *)arg3; +struct task_struct *t = current; + +if (!t->rseq.usrptr) +return -ENXIO ; +if (!rseq_v2 (t)) +return -ENOTSUPP ; +if (!node) +return -EINVAL ; + +switch (arg2) { +case PR_RSEQ_OP_REGISTER : +return rseq_op_register (t, node); +case PR_RSEQ_OP_UNREGISTER : +return rseq_op_unregister (t, node); +default : +return -EINVAL ; + } +}
kernel/sys.c +5 -0
@@ -2889,6 +2889,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,return -EINVAL ; error =rseq_slice_extension_prctl (arg2, arg3);break ; +case PR_RSEQ_OP : +if (arg4 || arg5) +return -EINVAL ; + error =rseq_op_prctl (arg2, arg3); +break ;case PR_GET_CFI :if (arg2 !=PR_CFI_BRANCH_LANDING_PADS )return -EINVAL ;
Signature · 1 line
2.54.0
Re: [RFC PATCH 4/5] rseq: register and unregister operations via prctl
Quoted 91 lines · On Fri, 28 Aug 2026 at 17:34, <[email protected]> wrote:
From: Olivier Dion <[email protected]>
Implement the
PR_RSEQ_OPprctl. Registration splices a user-provided pristine node into the head of the circular operation list, and unregistration unsplices it after validating it is properly linked between its neighbours; the kernel owns the node's next/prev links for the lifetime of the registration. The 0<->1nr_opstransition reflects the enabled state intorseq_event::rseq_opand the user visible flag viarseq_op_update_enabled().Initialize the operation list sentinel as an empty self-pointing circular list on rseq registration, and reset
nr_ops/rseq_opstate there.Apply pending operations on the slow path and on signal delivery, and factor the common "clear error and force SIGSEGV" fixup into
force_fault().Signed-off-by Olivier Dion <[email protected]>--- kernel/rseq.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++--- kernel/sys.c | 5 ++ 2 files changed, 212 insertions(+), 13 deletions(-)kernel/rseq.c +23
-6@@ -253,17 +253,18 @@ static bool rseq_handle_cs(struct task_struct *t, struct pt_regs *regs)static void rseq_slowpath_update_usr (struct pt_regs *regs) {/* -* Preserve has_rseq and user_irq state. The generic entry code clears -* user_irq on the way out, the non-generic entry architectures are not -* setting user_irq. +* Preserve has_rseq, rseq_op and user_irq state. The generic entry +* code clears user_irq on the way out, the non-generic entry +* architectures are not setting user_irq. */ const struct rseq_event evt_mask = { .has_rseq =RSEQ_HAS_RSEQ_VERSION_MASK , + .rseq_op = true, .user_irq = true, };struct task_struct *t = current;struct rseq_ids ids; -bool event; +bool event, should_fault = false;if (unlikely (t->flags &PF_EXITING ))return ;@@ -300,7 +301,12 @@ static void rseq_slowpath_update_usr(struct pt_regs *regs) ids.node_id =cpu_to_node (ids.cpu_id); -if (unlikely (!rseq_update_usr (t, regs, &ids))) { +if (unlikely (!rseq_update_usr (t, regs, &ids))) + should_fault = true; +else if (t->rseq.event.rseq_op &&unlikely (!rseq_apply_ops (t))) + should_fault = true; + +if (should_fault) {/* * Clear the errors just in case this might survive magically, but * leave the rest intact. @@ -329,8 +335,20 @@ void __rseq_handle_slowpath(struct pt_regs *regs)rseq_slowpath_update_usr (regs); } +static inline void force_fault (int sig) +{ +/* +* Clear the errors just in case this might survive magically, but leave +* the rest intact. +*/ + current->rseq.event.error =0 ; +force_sigsegv (sig); +} +void __rseq_signal_deliver (int sig,struct pt_regs *regs) { +bool should_fault = false; +rseq_stat_inc (rseq_stats.signal);/* @@ -339,14 +357,13 @@ void __rseq_signal_deliver(int sig, struct pt_regs *regs)* the interrupted context as after this point the instruction * pointer in @regs points to the signal handler. */ -if (unlikely (!rseq_handle_cs (current, regs))) {
kernel/rseq.c void __rseq_signal_deliver(int sig, struct pt_regs *regs) · line 362
Quoted 10 lines
kernel/rseq.c +3
-7-/* -* Clear the errors just in case this might survive -* magically, but leave the rest intact. -*/ - current->rseq.event.error =0 ; -force_sigsegv (sig); - } +if (unlikely (!rseq_handle_cs (current, regs))) + should_fault = true; +else if (current->rseq.event.rseq_op &&unlikely (!rseq_apply_ops (current)))
Just to make sure: membarrier delivered to a task will also trigger rseq_apply_ops, right?
Quoted 20 lines
kernel/rseq.c +6
-0+ should_fault = true; + +if (should_fault) +force_fault (sig);/* * In legacy mode, force the update of IDs before returning to user @@ -436,6 +453,8 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3 rseqfl |=RSEQ_CS_FLAG_SLICE_EXT_ENABLED ; } } +if (version >1 ) + rseqfl |=RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE ; scoped_user_write_access(rseq, efault) {/* @@ -458,8 +477,16 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3* registrations. */ if (version >1 ) {
kernel/rseq.c static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3 · line 489
Quoted 10 lines
kernel/rseq.c +8
-0+u64 sentinel = (u64)&rseq->rseq_op_list; +if (IS_ENABLED (CONFIG_RSEQ_SLICE_EXTENSION ))unsafe_put_user (0U , &rseq->slice_ctrl.all, efault); +/* +* Initialize the rseq operation list sentinel as an +* empty circular doubly-linked list pointing to itself. +*/ +unsafe_put_user (sentinel, &rseq->rseq_op_list.next, efault); +unsafe_put_user (sentinel, &rseq->rseq_op_list.prev, efault);
Is there always enough space to write these? Can this corrupt program memory?
Quoted 200 lines
kernel/rseq.c +167
-0} }@@ -474,6 +501,12 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3#ifdef CONFIG_RSEQ_SLICE_EXTENSION current->rseq.slice.state.enabled = !!(rseqfl &RSEQ_CS_FLAG_SLICE_EXT_ENABLED );#endif +/* +* A fresh registration starts with no operations, so operation +* processing is disabled until the first one is registered. +*/ + current->rseq.event.rseq_op = false; + current->rseq.nr_ops =0 ;/* * Ensure the cpu_id_start and cpu_id fields are updated before @@ -887,3 +920,164 @@ device_initcall(rseq_slice_init);#else static void rseq_slice_ext_init (struct dentry *root_dir) { }#endif /* CONFIG_RSEQ_SLICE_EXTENSION */ + +/* +* Reflect the operation enabled state into the user visible flags field. +* Called on the 0<->1 transition of nr_ops. The kill path is taken on fault +* because losing this update leaves user space and kernel state inconsistent. +*/ +static int rseq_op_update_enabled (struct task_struct *t,bool enable) +{ +struct rseq __user *rseq = t->rseq.usrptr; +u32 rflags; + +if (get_user (rflags, &rseq->flags)) +return -EFAULT ; + + rflags &= ~RSEQ_CS_FLAG_RSEQ_OP_ENABLED ; + rflags |=RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE ; +if (enable) + rflags |=RSEQ_CS_FLAG_RSEQ_OP_ENABLED ; + +if (put_user (rflags, &rseq->flags)) +return -EFAULT ; + + t->rseq.event.rseq_op = enable; +return 0 ; +} + +/* +* Register @node at the head of the circular doubly-linked operation list +* anchored by the kernel owned sentinel in struct rseq. +*/ +static int rseq_op_register (struct task_struct *t,struct rseq_op_node __user *node) +{ +struct rseq_op_node __user *sentinel = &t->rseq.usrptr->rseq_op_list; +struct rseq_op_node __user *first; +u64 next, prev, first_addr; +u8 type, i; + +if (t->rseq.nr_ops >=RSEQ_OP_LIST_LIMIT ) +return -ENOSPC ; + +if (!IS_ALIGNED ((unsigned long )node, __alignof__(struct rseq_op_node ))) +return -EINVAL ; +if (!access_ok (node,sizeof (*node))) +return -EFAULT ; + +/* +* The node links are owned by the kernel. User space must present a +* pristine node: next, prev and the reserved bytes all zeroed, and a +* known operation type. +*/ +if (get_user (next, &node->next) ||get_user (prev, &node->prev) || +get_user (type, &node->type)) +return -EFAULT ; +if (next || prev) +return -EINVAL ; +if (type >=RSEQ_OP_NR ) +return -EINVAL ; +for (i =1 ; i <sizeof (node->reserved); i++) { +u8 r; + +if (get_user (r, &node->reserved[i])) +return -EFAULT ; +if (r) +return -EINVAL ; + } + +/* Splice the node in right after the sentinel. */ +if (get_user (first_addr, &sentinel->next)) + goto die; + first = (struct rseq_op_node __user *)first_addr; + +if (put_user ((u64 )(unsigned long )first, &node->next) || +put_user ((u64 )(unsigned long )sentinel, &node->prev) || +put_user ((u64 )(unsigned long )node, &first->prev) || +put_user ((u64 )(unsigned long )node, &sentinel->next)) + goto die; + + t->rseq.nr_ops +=1 ; + +if (t->rseq.nr_ops ==1 ) +return rseq_op_update_enabled (t, true) ? -EFAULT :0 ; +return 0 ; +die: +force_sig (SIGSEGV ); +return -EFAULT ; +} + +/* +* Unregister @node from the operation list. The node links are validated +* against its neighbours to reject bogus or double unregistration. +*/ +static int rseq_op_unregister (struct task_struct *t,struct rseq_op_node __user *node) +{ +struct rseq_op_node __user *prev, *next; +u64 prev_addr, next_addr, tmp; + +if (!t->rseq.nr_ops) +return -ENOENT ; + +if (!IS_ALIGNED ((unsigned long )node, __alignof__(struct rseq_op_node ))) +return -EINVAL ; +if (!access_ok (node,sizeof (*node))) +return -EFAULT ; + +if (get_user (next_addr, &node->next) ||get_user (prev_addr, &node->prev)) +return -EFAULT ; + prev = (struct rseq_op_node __user *)prev_addr; + next = (struct rseq_op_node __user *)next_addr; + +/* A registered node always has both links set. */ +if (!prev || !next) +return -EINVAL ; + +/* Verify the node is properly linked between its neighbours. */ +if (get_user (tmp, &prev->next)) + goto die; +if (tmp != (u64 )(unsigned long )node) +return -EINVAL ; +if (get_user (tmp, &next->prev)) + goto die; +if (tmp != (u64 )(unsigned long )node) +return -EINVAL ; + +/* Unsplice and clear the node links so it can be reused. */ +if (put_user (next_addr, &prev->next) || +put_user (prev_addr, &next->prev) || +put_user (0ULL , &node->next) || +put_user (0ULL , &node->prev)) + goto die; + + t->rseq.nr_ops -=1 ; + +if (t->rseq.nr_ops ==0 ) +return rseq_op_update_enabled (t, false) ? -EFAULT :0 ; +return 0 ; +die: +force_sig (SIGSEGV ); +return -EFAULT ; +} + +int rseq_op_prctl (unsigned long arg2,unsigned long arg3) +{ +struct rseq_op_node __user *node = (struct rseq_op_node __user *)arg3; +struct task_struct *t = current; + +if (!t->rseq.usrptr) +return -ENXIO ; +if (!rseq_v2 (t)) +return -ENOTSUPP ; +if (!node) +return -EINVAL ; + +switch (arg2) { +case PR_RSEQ_OP_REGISTER : +return rseq_op_register (t, node); +case PR_RSEQ_OP_UNREGISTER : +return rseq_op_unregister (t, node); +default : +return -EINVAL ; + } +}kernel/sys.c +5
-1@@ -2889,6 +2889,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,return -EINVAL ; error =rseq_slice_extension_prctl (arg2, arg3);break ; +case PR_RSEQ_OP : +if (arg4 || arg5) +return -EINVAL ; + error =rseq_op_prctl (arg2, arg3); +break ;case PR_GET_CFI :if (arg2 !=PR_CFI_BRANCH_LANDING_PADS )return -EINVAL ; --2.54.0
RFC PATCH 5/5 selftests/rseq: add coverage for rseq operations
In reply to [email protected]From: Olivier Dion <[email protected]>
Add selftest-side support for the rseq operation ABI and wire a new rseq_op_test into the rseq selftest suite.
The new test covers:
- reset before entering a signal handler,
- reset on return to userspace after poll(),
- reset after scheduler-driven preemption,
- no-operation cases where nothing should be reset,
- register-time rejection of non-pristine nodes and unknown types,
- fatal cases for corrupted list links (self and transitive cycles),
- invalid destination and next pointers,
- unsupported lengths,
- unaligned destinations, and
- NULL-source clearing.
Also extend the selftest ABI/helpers with the rseq operation node definitions and prctl-based register/unregister helpers, and add a dedicated runner that disables glibc's implicit rseq registration.
--- tools/testing/selftests/rseq/.gitignore | 4 +- tools/testing/selftests/rseq/Makefile | 4 +- tools/testing/selftests/rseq/rseq-abi.h | 146 ++++++++++++++++++++---- tools/testing/selftests/rseq/rseq.c | 23 +++- tools/testing/selftests/rseq/rseq.h | 75 ++++++++++++ 5 files changed, 223 insertions(+), 29 deletions(-)
tools/testing/selftests/rseq/.gitignore +3 -1
@@ -2,7 +2,6 @@ basic_percpu_ops_test basic_percpu_ops_mm_cid_test basic_test -basic_rseq_op_test param_test param_test_benchmark param_test_compare_twice
@@ -11,3 +10,6 @@ param_test_mm_cid_benchmark param_test_mm_cid_compare_twice syscall_errors_test slice_test +check_optimized +legacy_check +rseq_op_test
tools/testing/selftests/rseq/Makefile +3 -1
@@ -22,12 +22,14 @@ TEST_GEN_PROGS_EXTENDED = librseq.so \ param_test_compare_twice \ param_test_mm_cid \ param_test_mm_cid_compare_twice \ + rseq_op_test \ syscall_errors_test \ legacy_check \ slice_test \ check_optimized -TEST_PROGS = run_param_test.sh run_syscall_errors_test.sh run_legacy_check.sh run_timeslice_test.sh +TEST_PROGS = run_param_test.sh run_syscall_errors_test.sh run_legacy_check.sh run_timeslice_test.sh \ + run_rseq_op_test.sh TEST_FILES := settings
tools/testing/selftests/rseq/rseq-abi.h +124 -22
177 lines not on this page · raw message
tools/testing/selftests/rseq/rseq.c +18 -5
@@ -116,6 +116,17 @@ bool rseq_available(void) } } +/* The rseq areas need to be at least 32 bytes. */ +static +unsigned int get_rseq_min_alloc_size (void ) +{ +unsigned int alloc_size = rseq_size; + +if ((int ) alloc_size <ORIG_RSEQ_ALLOC_SIZE ) + alloc_size =ORIG_RSEQ_ALLOC_SIZE ; +return alloc_size; +} +/* * Return the feature size supported by the kernel. *
@@ -261,12 +272,14 @@ void rseq_init(void)/* rseq flags are deprecated, always set to 0. */ rseq_flags =0 ; + { +unsigned int rseq_kernel_feature_size =get_rseq_kernel_feature_size (); -/* -* Set the size to 0 until at least one thread registers to mimic the -* libc behavior. -*/ - rseq_size =0 ; +if (rseq_kernel_feature_size <=RSEQ_THREAD_AREA_ALLOC_SIZE ) + rseq_size = rseq_kernel_feature_size; +else + rseq_size =ORIG_RSEQ_ALLOC_SIZE ; + } } static __attribute__((destructor))
tools/testing/selftests/rseq/rseq.h +75 -0
85 lines not on this page · raw message
Signature · 1 line
2.54.0
Olivier!
Quoted 9 lines · On Fri, Aug 28 2026 at 11:33, [email protected] wrote:
This series introduces RSEQ operations: a per-thread list of operations the kernel applies, on behalf of a ask, when returning to user space.
The main motivation is to help TCMalloc migrate from RSEQ v1 to RSEQ v2 and use the RSEQ area registered by glibc. TCMalloc currently relies on RSEQ v1 resetting the
cpu_idfield to invalidate a per-CPU pointer cached in TLS. This requires applications to disable glibc's implicit RSEQ registration and prevents sharing the glibc RSEQ area.
Quoted 5 lines
RSEQ operations provide an opt-in replacement for that invalidation mechanism. User space registers operation nodes with prctl; the kernel keeps the nodes in a per-thread circular list and applies them on return to user space. Only tasks with registered operations incur the additional RSEQ exit-path work.
TBH, to me this sounds like a horrible idea. It's yet another "interpreter" for a very limited use case and a lot of complexity.
I think this can be done in user space with some help of the kernel of course. If user space has registered an entry point for this functionality then the kernel can emulate a call to that on return to user space:
if (tsk->rseq.needs_fixup) {
// modifies regs->sp
create_callframe_on_user stack(regs);
regs->ip = tsk->rseq.fixup_ip;
}
ret_to_user()The user space fixup does:
fixup_ip()
....
user_rseq.fixup_in_progress = false;
restore_and_return() // Returns to the original IPA reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.
Of course the above is way too simple to be true :) It works except when it nests. But that's solvable too:
if (tsk->rseq.needs_fixup) {
// modifies regs->sp
create_callframe_on_user stack(regs);
regs->ip = tsk->rseq.fixup_ip;
// Save the callframe SP
tsk->rseq.fixup_sp = regs->sp;
}When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:
regs->ip = tsk->rseq.fixup_abort_ip;
regs->sp = tsk->rseq.fixup_sp;
user_rseq.fixup_in_progress = false;
tsk->rseq.needs_fixup = true;Which rewinds the stack to the callframe and makes the interrupted fixup continue at the fixup_abort_ip which just restores registers from the callframe and returns to the original IP.
Then the flow becomes in case of a trivial interrupt which just returns:
fixup_ip()
-> interrupt
...
if (user_rseq.fixup_in_progress)
rewind_fixup(regs);
...
if (needs_fixup)
setup_fixup(regs);
return to user;
fixup_ip()
...
user_rseq.fixup_in_progress = false
restore_and_return() fixup_abort_ip:
restore_and_return() // Returns to orig_ip
In case of a signal delivery:
fixup_ip()
-> interrupt
...
if (user_rseq.fixup_in_progress)
rewind_fixup(regs);
deliver_signal()
setup_sigframe()
...
if (needs_fixup)
setup_fixup(regs);
return to user;
fixup_ip()
...
user_rseq.fixup_in_progress = false
restore_and_return()
signal_handler()
sys_sigreturn()
restore(regs)
return to user; fixup_abort_ip:
restore_and_return() // Returns to orig_ip
There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)
I've played around with something similar pre RSEQ, but all I could still find are the notes I took back then, which I duly transcribed into todays RSEQ world. The patches themself would be utterly useless anyway as all parts which need to be touched have been through the mincer at least once.
The fixup address and the fixup abort address should probably not be stored in user rseq memory. For an initial POC they just can be registered through sys_rseq() which also enables the fixup mechanics, i.e. the unconditional set of tsk->rseq.needs_fixup in schedule(), which should be completely independent of the rseq.sched_switch and rseq.ids_changed bits and not change the existing functionality and semantics like the current POC does. Keep it separate.
Once the POC dust has settled the registration interface should change for the final implementation because otherwise this would create the very same problem of who owns it again.
The final solution should provide a trivial dispatch mechanism in the VDSO and the RSEQ user area should be expanded to provide storage for registration. The syscall registration would change to registering a lib specific callback/data pair and the kernel would reorganize the storage. Whether that's a linked list or a size limited array in the RSEQ user area does not matter. As this is thread local there is no concurrency and the syscall can rearange that storage completely undisturbed. Libraries can manage their own thread local marker which is checked in the lib specific callback to decide whether they want to run or not.
Using a VDSO dispatcher and expanding the RSEQ storage for that should just work out of the box with any libc which supports RSEQ_V2 and would therefore not create any additional libc dependencies for other library developers.
The dispatcher entry needs to be trivial ASM for the restore/return
fixup_ip:
// setup_fixup() in the kernel stored the TLS user RSEQ address in
// RDI or whatever an architecture uses for the first argument
CALL fixup_cfixup_ip_abort:
ASM_RESTORE_REGS
RET
The C part should be trivial too:
fixup_c(struct rseq *rseq)
{
for_each_fixup(fn, data, rseq)
fn(data);
}
or something along these lines.
Thoughts?
Thanks,
tglx
Quoted 12 lines · Thomas Gleixner
Olivier!
Quoted 9 lines · On Fri, Aug 28 2026 at 11:33, [email protected] wrote:
This series introduces RSEQ operations: a per-thread list of operations the kernel applies, on behalf of a ask, when returning to user space.
The main motivation is to help TCMalloc migrate from RSEQ v1 to RSEQ v2 and use the RSEQ area registered by glibc. TCMalloc currently relies on RSEQ v1 resetting the
cpu_idfield to invalidate a per-CPU pointer cached in TLS. This requires applications to disable glibc's implicit RSEQ registration and prevents sharing the glibc RSEQ area.
Quoted 8 lines
Quoted 5 lines
RSEQ operations provide an opt-in replacement for that invalidation mechanism. User space registers operation nodes with prctl; the kernel keeps the nodes in a per-thread circular list and applies them on return to user space. Only tasks with registered operations incur the additional RSEQ exit-path work.
TBH, to me this sounds like a horrible idea. It's yet another "interpreter" for a very limited use case and a lot of complexity.
I post this RFC before going on vacation. I tend to very much agree with you reply overall, after revisiting the problem and potential issues this solution could introduce.
Quoted 12 lines
I think this can be done in user space with some help of the kernel of course. If user space has registered an entry point for this functionality then the kernel can emulate a call to that on return to user space:
if (tsk->rseq.needs_fixup) { // modifies regs->sp create_callframe_on_user stack(regs); regs->ip = tsk->rseq.fixup_ip; } ret_to_user()
Quoted 6 lines
The user space fixup does:
fixup_ip().... user_rseq.fixup_in_progress = false; restore_and_return() // Returns to the original IP
Yes. This is what I had in mind (see reply to Florian).
Quoted 3 lines
A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.
I suppose that no floating point also mean no xsave performance trashing, which is obviously something we want to avoid. Is that what you mean by sigframe disaster? Also, I am not sure how we can enforce this no syscall/floating point policies other than asking users to be good citizen.
Quoted 14 lines
Of course the above is way too simple to be true :) It works except when it nests. But that's solvable too:
if (tsk->rseq.needs_fixup) { // modifies regs->sp create_callframe_on_user stack(regs); regs->ip = tsk->rseq.fixup_ip; // Save the callframe SP tsk->rseq.fixup_sp = regs->sp; }When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:
Quoted 8 lines
regs->ip = tsk->rseq.fixup_abort_ip; regs->sp = tsk->rseq.fixup_sp; user_rseq.fixup_in_progress = false; tsk->rseq.needs_fixup = true;Which rewinds the stack to the callframe and makes the interrupted fixup continue at the
fixup_abort_ipwhich just restores registers from the callframe and returns to the original IP.
So the fixup handler would have an abort label, akin to RSEQ region. But then there will be no guarantee that the fixup operations succeed? Or did I completely misunderstood?
[...]
Quoted 3 lines
There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)
I could certainly make a POC out of this for x86. It could handle red-zone and shadow-stack to start with. I don't know if there are other architecture-specific quirks that need to be aware of, given I am not familiar enough with architectures outside of x86.
[...]
Quoted 3 lines
Once the POC dust has settled the registration interface should change for the final implementation because otherwise this would create the very same problem of who owns it again.
Right we don't want to play in that movie again :-)
Quoted 28 lines
The final solution should provide a trivial dispatch mechanism in the VDSO and the RSEQ user area should be expanded to provide storage for registration. The syscall registration would change to registering a lib specific callback/data pair and the kernel would reorganize the storage. Whether that's a linked list or a size limited array in the RSEQ user area does not matter. As this is thread local there is no concurrency and the syscall can rearange that storage completely undisturbed. Libraries can manage their own thread local marker which is checked in the lib specific callback to decide whether they want to run or not.
Using a VDSO dispatcher and expanding the RSEQ storage for that should just work out of the box with any libc which supports
RSEQ_V2and would therefore not create any additional libc dependencies for other library developers.The dispatcher entry needs to be trivial ASM for the restore/return
fixup_ip:// setup_fixup() in the kernel stored the TLS user RSEQ address in // RDI or whatever an architecture uses for the first argument CALL fixup_c
fixup_ip_abort:
ASM_RESTORE_REGS
RETThe C part should be trivial too:
Quoted 7 lines
fixup_c(struct rseq *rseq)
{
for_each_fixup(fn, data, rseq)
fn(data);
}or something along these lines.
Thanks for the insight. I think this will certainly help me in making the POC.
One aspect that I have not think of for now is allowing fast registration/unregistration. This RFC uses a syscall for registration. My original intent for this was to do a edge trigger detection of the first/last registration/unregistration in the syscall and set/unset a flag into the RSEQ state of the thread task. That way, we don't need to touch another cache line to know if the thread registered something. This is obviously incompatible with short-live registrations. However, I did not come with a case for it yet, so perhaps I am over-thinking this.
Thanks,
Olivier
Signature · 3 lines
Olivier Dion EfficiOS Inc. https://www.efficios.com
Quoted 12 lines · Olivier Dion
Quoted 4 lines · Thomas Gleixner
fixup_ip().... user_rseq.fixup_in_progress = false; restore_and_return() // Returns to the original IPYes. This is what I had in mind (see reply to Florian).
Quoted 3 lines
A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.
Quoted 5 lines
I suppose that no floating point also mean no xsave performance trashing, which is obviously something we want to avoid. Is that what you mean by sigframe disaster? Also, I am not sure how we can enforce this no syscall/floating point policies other than asking users to be good citizen.
The sigframe disaster is that sigaltstack has been insufficient for the ever growing XSTATE to save (its even uncompressed XSTATE). Also these fixups really should do a few selective stores and not huge computations or wipe out a GB of memory. So avoiding the XSAVE/XRSTOR overhead completely is certainly a benefit.
Quoted 8 lines
Quoted 8 lines
When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:
regs->ip = tsk->rseq.fixup_abort_ip; regs->sp = tsk->rseq.fixup_sp; user_rseq.fixup_in_progress = false; tsk->rseq.needs_fixup = true;
Quoted 7 lines
Quoted 3 lines
Which rewinds the stack to the callframe and makes the interrupted fixup continue at the
fixup_abort_ipwhich just restores registers from the callframe and returns to the original IP.So the fixup handler would have an abort label, akin to RSEQ region. But then there will be no guarantee that the fixup operations succeed? Or did I completely misunderstood?
They are only aborted when there is nesting, so the nesting context starts over and redoes them. Once the nesting context returns to the first fixup (abort IP) the state is correct and then the first fixup returns to the original return IP.
That abort on nesting avoids the following issue:
user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user
fixup_ip:
fixup_c(data)
...
interrupt(#2)
schedule()
MMCID changes
fixup requested
fixup_ip:
fixup_c(data)
data::mmcid = user_rseq::mmcid
restore_and_return()
data::mmcid = user_rseq::mmcid
restore_and_return()
If the interrupt #2 hits between the load of user_rseq::mmcid and the store to data::mmcid of the first fixup, then the return to the first fixup context would obviously write the wrong ID back.
And you can't skip the fixup on return from interrupt #2 and just return to the already running one in that case either.
That would be possible if the fixups are truly idempotent, e.g. can only zero out memory.
The trivial nest case would be:
user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user
fixup_ip:
fixup_c(data)
start zeroing
interrupt(#2)
schedule()
MMCID changes
fixup requested
observes fixup running
return to user
continue zeroing
restore_and_return()
But that does not work with signals because the signal delivery does not return to the interrupted IP. It creates the sigframe and returns to the signal handler, which then goes back with sys_rt_sigreturn(). Then the kernel restores the interrupted context. So in that case you'd need:
user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
// Must be a counter
user_rseq->fixup_running++;
return to user
fixup_ip:
fixup_c(data)
start zeroing
interrupt(#2)
signal_delivery()
setup_sigframe()
observes fixup running
setup_callframe(....)
user_rseq->fixup_running++;
return to user fixup_ip:
fixup_c(data)
zero everything
user_rseq->fixup_running--;
restore_and_return() signal_handler()
sys_rt_sigreturn()
restore_regs()
observes fixup running
return to user
continue zeroing
user_rseq->fixup_running--;
restore_and_return()
Idempotent fixup functions restrict obviously what can be done there. But if that restriction is fine, then this approach works too.
I have no strong opinion either way.
Quoted 6 lines
Quoted 3 lines
There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)
I could certainly make a POC out of this for x86. It could handle red-zone and shadow-stack to start with. I don't know if there are
I wouldn't even bother with shadow-stacks for a POC. The red-zone skip when setting up the callframe, i.e. SP - 128 is unavoidable, but that's it.
Quoted 2 lines
other architecture-specific quirks that need to be aware of, given I am not familiar enough with architectures outside of x86.
Well every architecture has some quirks but I'm not aware of one which would fundamentally stand in the way. The main difference is going to be how the callframe is set up and how the fixup function needs to look. That's always architecture specific and there are wizards for each architecture to help with that :)
Quoted 9 lines
One aspect that I have not think of for now is allowing fast registration/unregistration. This RFC uses a syscall for registration. My original intent for this was to do a edge trigger detection of the first/last registration/unregistration in the syscall and set/unset a flag into the RSEQ state of the thread task. That way, we don't need to touch another cache line to know if the thread registered something. This is obviously incompatible with short-live registrations. However, I did not come with a case for it yet, so perhaps I am over-thinking this.
TCMalloc wont have a short-lived registration, neither wont a tracer or something like that. Once a library is initialized it won't go away just because. So if the facility needs to pause the callback intermittently then this can simply be:
cb(data)
if (!data->run)
return;That won't be the end of the world if those pauses are not taking forever. If they do then the syscall is justified.
Let's get the basic principles working first on a KISS basis and then think about how to keep it as simple as possible.
So for the POC you neither need multi-lib support nor the VDSO bits. Just hack up the ASM fixup and a trivial C demonstrator in user space and register fixup and abort address with a hacked up sys_rseq().
Once that works, the extra bells and whistels are not hard to add. They are hard to get right, but without the prove of concept wasting time on them is pretty pointless. :)
Thanks,
tglx
---
Everything should be made as simple as possible, but not simpler.
- attributed to Albert Einstein
Quoted 207 lines · Thomas Gleixner
Quoted 12 lines · Olivier Dion
Quoted 4 lines · Thomas Gleixner
fixup_ip().... user_rseq.fixup_in_progress = false; restore_and_return() // Returns to the original IPYes. This is what I had in mind (see reply to Florian).
Quoted 3 lines
A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.
Quoted 5 lines
I suppose that no floating point also mean no xsave performance trashing, which is obviously something we want to avoid. Is that what you mean by sigframe disaster? Also, I am not sure how we can enforce this no syscall/floating point policies other than asking users to be good citizen.
The sigframe disaster is that sigaltstack has been insufficient for the ever growing XSTATE to save (its even uncompressed XSTATE). Also these fixups really should do a few selective stores and not huge computations or wipe out a GB of memory. So avoiding the XSAVE/XRSTOR overhead completely is certainly a benefit.
Quoted 8 lines
Quoted 8 lines
When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:
regs->ip = tsk->rseq.fixup_abort_ip; regs->sp = tsk->rseq.fixup_sp; user_rseq.fixup_in_progress = false; tsk->rseq.needs_fixup = true;Quoted 7 lines
Quoted 3 lines
Which rewinds the stack to the callframe and makes the interrupted fixup continue at the
fixup_abort_ipwhich just restores registers from the callframe and returns to the original IP.So the fixup handler would have an abort label, akin to RSEQ region. But then there will be no guarantee that the fixup operations succeed? Or did I completely misunderstood?
They are only aborted when there is nesting, so the nesting context starts over and redoes them. Once the nesting context returns to the first fixup (abort IP) the state is correct and then the first fixup returns to the original return IP.
That abort on nesting avoids the following issue:
user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user
fixup_ip:
fixup_c(data)
...
interrupt(#2)
schedule()
MMCID changes
fixup requested
fixup_ip:
fixup_c(data)
data::mmcid=user_rseq::mmcid
restore_and_return()
data::mmcid=user_rseq::mmcid
restore_and_return()If the interrupt #2 hits between the load of
user_rseq::mmcidand the store todata::mmcidof the first fixup, then the return to the first fixup context would obviously write the wrong ID back.And you can't skip the fixup on return from interrupt #2 and just return to the already running one in that case either.
That would be possible if the fixups are truly idempotent, e.g. can only zero out memory.
The trivial nest case would be:
user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user
fixup_ip:
fixup_c(data)
start zeroinginterrupt(#2)
schedule()
MMCID changes
fixup requestedobserves fixup running
return to usercontinue zeroing
restore_and_return()But that does not work with signals because the signal delivery does not return to the interrupted IP. It creates the sigframe and returns to the signal handler, which then goes back with
sys_rt_sigreturn(). Then the kernel restores the interrupted context. So in that case you'd need:
user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
// Must be a counter
user_rseq->fixup_running++;
return to user
fixup_ip:fixup_c(data) start zeroing interrupt(#2) signal_delivery() setup_sigframe() observes fixup running setup_callframe(....) user_rseq->fixup_running++; return to user
fixup_ip:fixup_c(data) zero everything user_rseq->fixup_running--; restore_and_return()
signal_handler()
sys_rt_sigreturn()
restore_regs()
observes fixup running
return to usercontinue zeroing
user_rseq->fixup_running--;
restore_and_return()Idempotent fixup functions restrict obviously what can be done there. But if that restriction is fine, then this approach works too.
I have no strong opinion either way.
Quoted 6 lines
Quoted 3 lines
There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)
I could certainly make a POC out of this for x86. It could handle red-zone and shadow-stack to start with. I don't know if there are
I wouldn't even bother with shadow-stacks for a POC. The red-zone skip when setting up the callframe, i.e. SP - 128 is unavoidable, but that's it.
Quoted 2 lines
other architecture-specific quirks that need to be aware of, given I am not familiar enough with architectures outside of x86.
Well every architecture has some quirks but I'm not aware of one which would fundamentally stand in the way. The main difference is going to be how the callframe is set up and how the fixup function needs to look. That's always architecture specific and there are wizards for each architecture to help with that :)
Quoted 9 lines
One aspect that I have not think of for now is allowing fast registration/unregistration. This RFC uses a syscall for registration. My original intent for this was to do a edge trigger detection of the first/last registration/unregistration in the syscall and set/unset a flag into the RSEQ state of the thread task. That way, we don't need to touch another cache line to know if the thread registered something. This is obviously incompatible with short-live registrations. However, I did not come with a case for it yet, so perhaps I am over-thinking this.
TCMalloc wont have a short-lived registration, neither wont a tracer or something like that. Once a library is initialized it won't go away just because. So if the facility needs to pause the callback intermittently then this can simply be:
cb(data)
if (!data->run) return;That won't be the end of the world if those pauses are not taking forever. If they do then the syscall is justified.
Let's get the basic principles working first on a KISS basis and then think about how to keep it as simple as possible.
Quoted 7 lines
So for the POC you neither need multi-lib support nor the VDSO bits. Just hack up the ASM fixup and a trivial C demonstrator in user space and register fixup and abort address with a hacked up
sys_rseq().Once that works, the extra bells and whistels are not hard to add. They are hard to get right, but without the prove of concept wasting time on them is pretty pointless. :)
I totally agree with the "yet another general-purpose interpreter" argument. I was thinking of some schemes where we would need to clear lots of words, and doing this with an in-kernel interpreter may be too slow; or clearing just 1 or 2 bytes; or even caching a pointer in the GS register, and resetting it with wrgsbase.
I think we even brainstormed with Mathieu something similar to resetting the PC to a custom user-space handler at Plumbers few years ago, but we got stuck on signal handlers and re-entrancy problem.