Skip to contentSkip to conversation

rseq: add support for RSEQ operations

RFC PATCH In review [email protected] 4 participants 19 messages Aug 28 – Sep 9

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover rseq: add support for RSEQ operations

From: Olivier Dion <[email protected]>

Hi,

This series introduces RSEQ operations: a per-thread list of operations the kernel applies, on behalf of a ask, when returning to user space.

The main motivation is to help TCMalloc migrate from RSEQ v1 to RSEQ v2 and use the RSEQ area registered by glibc. TCMalloc currently relies on RSEQ v1 resetting the cpu_id field to invalidate a per-CPU pointer cached in TLS. This requires applications to disable glibc's implicit RSEQ registration and prevents sharing the glibc RSEQ area.

RSEQ operations provide an opt-in replacement for that invalidation mechanism. User space registers operation nodes with prctl; the kernel keeps the nodes in a per-thread circular list and applies them on return to user space. Only tasks with registered operations incur the additional RSEQ exit-path work.

The first patch adds the UAPI. The next three patches add the task state, operation execution, and prctl registration support. The final patch adds selftests covering normal operation and malformed-list and invalid-memory cases.

See also <https://lore.kernel.org/lkml/[email protected]>.

Thanks,
old

Olivier Dion (5):
rseq: uapi: add rseq operation definitions
rseq: add per-task rseq operation state
rseq: apply operations on exit to user space
rseq: register and unregister operations via prctl selftests/rseq: add coverage for rseq operations

 include/linux/rseq.h                    |  10 +-
 include/linux/rseq_entry.h              | 157 ++++++++++++++++-
 include/linux/rseq_types.h              |   9 +
 include/uapi/linux/prctl.h              |  12 ++
 include/uapi/linux/rseq.h               | 104 ++++++++++-
 kernel/rseq.c                           | 220 ++++++++++++++++++++++--
 kernel/sys.c                            |   5 +
 tools/testing/selftests/rseq/.gitignore |   4 +-
 tools/testing/selftests/rseq/Makefile   |   4 +-
 tools/testing/selftests/rseq/rseq-abi.h | 146 +++++++++++++---
 tools/testing/selftests/rseq/rseq.c     |  23 ++-
 tools/testing/selftests/rseq/rseq.h     |  75 ++++++++
 12 files changed, 713 insertions(+), 56 deletions(-)
Signature · 1 line
2.54.0
#1 lore raw

RFC PATCH 1/5 rseq: uapi: add rseq operation definitions

From: Olivier Dion <[email protected]>

Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.

The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.

Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.

Overall, this puts the users in a situation of choosing between a fast sched_getcpu() and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibc

RSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them.

Add enum rseq_op_type and the rseq_op_node / rseq_op_reset / rseq_op_reset_with_stride structures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, the RSEQ_OP_LIST_LIMIT walk bound, and the PR_RSEQ_OP prctl with its REGISTER/UNREGISTER sub-commands.

The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq.

[0] Documentation/userspace-api/rseq.rst (Optimized RSEQ v2)

---
 include/uapi/linux/prctl.h |  12 +++++
 include/uapi/linux/rseq.h  | 104 ++++++++++++++++++++++++++++++++++---
 2 files changed, 108 insertions(+), 8 deletions(-)
include/uapi/linux/prctl.h +12 -0
index b6ec6f693719..4cb6356a271a 100644
@@ -396,6 +396,18 @@ struct prctl_mm_map {
  */
 # define PR_RSEQ_SLICE_EXT_ENABLE		0x01
 
+/*
+ * RSEQ operation registration.
+ *
+ * arg3 is the user address of a struct rseq_op_node embedded in one of the
+ * rseq operation structures (see uapi/linux/rseq.h). Registering the first
+ * operation enables rseq operation processing for the thread; unregistering
+ * the last one disables it.
+ */
+#define PR_RSEQ_OP				82
+# define PR_RSEQ_OP_REGISTER			1
+# define PR_RSEQ_OP_UNREGISTER			2
+
 /*
  * Get or set the control flow integrity (CFI) configuration for the
  * current thread.
include/uapi/linux/rseq.h +96 -8

135 lines not on this page · raw message

Signature · 1 line
2.54.0
#2 lore raw

Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions

Quoted 31 lines · On Fri, 28 Aug 2026 at 17:33, <[email protected]> wrote:

From: Olivier Dion <[email protected]>

Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.

The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.

Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.

Overall, this puts the users in a situation of choosing between a fast sched_getcpu() and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibc

RSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them.

Add enum rseq_op_type and the rseq_op_node / rseq_op_reset / rseq_op_reset_with_stride structures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, the RSEQ_OP_LIST_LIMIT walk bound, and the PR_RSEQ_OP prctl with its REGISTER/UNREGISTER sub-commands.

Quoted 4 lines

The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq.

[0] Documentation/userspace-api/rseq.rst (Optimized RSEQ v2)

Hi Olivier,

This is very cool, thanks for working on this.

Quoted 35 lines
---
 include/uapi/linux/prctl.h |  12 +++++
 include/uapi/linux/rseq.h  | 104 ++++++++++++++++++++++++++++++++++---
 2 files changed, 108 insertions(+), 8 deletions(-)
include/uapi/linux/prctl.h +12 -0
index b6ec6f693719..4cb6356a271a 100644
@@ -396,6 +396,18 @@ struct prctl_mm_map {
  */
 # define PR_RSEQ_SLICE_EXT_ENABLE              0x01
 
+/*
+ * RSEQ operation registration.
+ *
+ * arg3 is the user address of a struct rseq_op_node embedded in one of the
+ * rseq operation structures (see uapi/linux/rseq.h). Registering the first
+ * operation enables rseq operation processing for the thread; unregistering
+ * the last one disables it.
+ */
+#define PR_RSEQ_OP                             82
+# define PR_RSEQ_OP_REGISTER                   1
+# define PR_RSEQ_OP_UNREGISTER                 2
+
 /*
  * Get or set the control flow integrity (CFI) configuration for the
  * current thread.
include/uapi/linux/rseq.h +0 -0
index ca6fe1f9d05e..b664d1991c48 100644

include/uapi/linux/rseq.h line 19

Quoted 7 lines
include/uapi/linux/rseq.h +4 -0
@@ -13,6 +13,11 @@
 #include <linux/types.h>
 #include <asm/byteorder.h>
 
+/*
+ * Maximum number of nodes walked in the rseq operation list.
+ */
+#define RSEQ_OP_LIST_LIMIT     2048

Does this belong to the user header? Kernel and userspace are not necessary built with the same headers, and we don't necessary want to set this const in stone. I think it may be more flexible to make this impl detail and just return ENOSPC.

Quoted 24 lines
include/uapi/linux/rseq.h +6 -0
 enum rseq_cpu_id_state {
        RSEQ_CPU_ID_UNINITIALIZED               = -1,
        RSEQ_CPU_ID_REGISTRATION_FAILED         = -2,
@@ -33,6 +38,8 @@ enum rseq_cs_flags_bit {
        /* User read only feature flags */
        RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT    = 4,
        RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT      = 5,
+       RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT      = 6,
+       RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT        = 7,
 };
 
 enum rseq_cs_flags {
@@ -47,6 +54,10 @@ enum rseq_cs_flags {
                (1U << RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT),
        RSEQ_CS_FLAG_SLICE_EXT_ENABLED          =
                (1U << RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT),
+       RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE          =
+               (1U << RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT),
+       RSEQ_CS_FLAG_RSEQ_OP_ENABLED            =
+               (1U << RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT),
 };
 
 /*

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 106

Quoted 10 lines
include/uapi/linux/rseq.h +7 -0
@@ -86,6 +97,77 @@ struct rseq_slice_ctrl {
        };
 };
 
+/*
+ * enum rseq_op_type - Type of an rseq operation
+ * @RSEQ_OP_RESET:                     Plain reset. Uses struct rseq_op_reset.
+ * @RSEQ_OP_RESET_WITH_STRIDE_CPUID:   Reset indexed by the current CPU ID.
+ *                                     Uses struct rseq_op_reset_with_stride.
+ * @RSEQ_OP_RESET_WITH_STRIDE_MMCID:   Reset indexed by the current MM CID.
+ *                                     Uses struct rseq_op_reset_with_stride.

Do you have use-cases for these STRIDE ops?
I can think of some, but they would require executing ops when a task is descheduled, rather than when it's scheduled in.

Quoted 11 lines
include/uapi/linux/rseq.h +11 -0
+ */
+enum rseq_op_type {
+       RSEQ_OP_RESET,
+       RSEQ_OP_RESET_WITH_STRIDE_CPUID,
+       RSEQ_OP_RESET_WITH_STRIDE_MMCID,
+       RSEQ_OP_NR,
+};
+
+/*
+ * struct rseq_op_node - Common header linking an rseq operation into the list
+ * @next:      Address of the next node. Owned by the kernel.

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 127

Quoted 10 lines
include/uapi/linux/rseq.h +10 -0
+ * @prev:      Address of the previous node. Owned by the kernel.
+ * @type:      Operation type. See enum rseq_op_type.
+ * @reserved:  Must be zero on registration.
+ *
+ * User space allocates the node, sets @type and zeroes @next, @prev and
+ * @reserved before passing it to prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, node).
+ * The kernel owns @next and @prev for the lifetime of the registration and
+ * links the node into a circular doubly-linked list anchored by an internal
+ * sentinel in struct rseq. User space must not touch @next or @prev while the
+ * node is registered.

Have you considered keeping copies of these structs in the kernel? That (1) is more consistent with most of the other kernel APIs, (2) will make executing ops faster since we don't need to copy_from_user all the time, (3) will make code much simpler since we won't need to re-verify all data, (4) will make the code more secure since user-space won't be able to mess with these structs, (5) will simplify the contract and documentation.
There are robust lists that keep lists in user-space, but these are supposed to be concurrently modified by userspace. But these rseq ops are generally not expected to be added/removed often (and the current contract does not allow that anyway).

Quoted 4 lines
include/uapi/linux/rseq.h +4 -0
+ */
+struct rseq_op_node {
+       __u64 next;
+       __u64 prev;

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 141

Quoted 10 lines
include/uapi/linux/rseq.h +10 -0
+       struct {
+               __u8  type; /* enum rseq_op_type */
+               __u8  reserved[7];
+       };
+};
+
+/*
+ * struct rseq_op_reset - Reset one word to a value on return to user space
+ * @node:      Operation list node.
+ * @src:       Address of the source word, or 0 to reset @dst to zero.

Is "or 0 to reset" part implemented in the series?

It would be useful to see an op that contains a const that is written, rather than an address. Address invovles additional indirection + copy_from_user.

And the contract for concurrent changes of the value pointed by src from user-space are not documented (atomicity?), w/o that having the address is not very useful.

Quoted 41 lines
include/uapi/linux/rseq.h +29 -5
+ * @dst:       Address of the destination word.
+ * @len:       Word length in bytes. Must be 4 or 8 (8 is 64-bit only).
+ */
+struct rseq_op_reset {
+       struct rseq_op_node     node;
+       __u64                   src;
+       __u64                   dst;
+       __u32                   len;
+};
+
+/*
+ * struct rseq_op_reset_with_stride - Reset one word in a strided array
+ * @node:      Operation list node.
+ * @src:       Address of the source word, or 0 to reset the slot to zero.
+ * @dst:       Base address of the strided destination array.
+ * @dst_stride:        Stride in bytes between consecutive array slots.
+ * @len:       Word length in bytes. Must be 4 or 8 (8 is 64-bit only).
+ *
+ * The destination slot is @dst + @dst_stride * index, where index is the
+ * current CPU ID or MM CID depending on the operation type.
+ */
+struct rseq_op_reset_with_stride {
+       struct rseq_op_node     node;
+       __u64                   src;
+       __u64                   dst;
+       __u64                   dst_stride;
+       __u32                   len;
+};
+
 /*
  * The original size and alignment of the allocation for struct rseq is
  * 32 bytes.
@@ -191,15 +273,21 @@ struct rseq {
        struct rseq_slice_ctrl slice_ctrl;
 
        /*
-        * Before rseq became extensible, its original size was 32 bytes even
-        * though the active rseq area was only 20 bytes.
-        * Exposing a 32 bytes feature size would make life needlessly painful
-        * for userspace. Therefore, add a reserved byte after byte 32
-        * to bump the rseq feature size from 32 to 33.

include/uapi/linux/rseq.h struct rseq { · line 283

Quoted 10 lines
include/uapi/linux/rseq.h +7 -2
-        * The next field to be added to the rseq area will be larger
-        * than one byte, and will replace this reserved byte.
+        * Sentinel of the circular doubly-linked list of rseq operations
+        * registered via prctl(PR_RSEQ_OP, ...). Fully owned and maintained by
+        * the kernel: it is initialized to point to itself on registration and
+        * user space must never read or write it directly.
+        *
+        * The kernel only use next and prev from rseq_op_list.  The rest of the
+        * bytes are reserved for later usage and should be zeroed.
         */

Do we really need the union and the op_used/reserved? Isn't it easier to say that everything needs to be 0'ed? What would userspace do with op_used? It's not possible to skip initialization of these fields when using both memset, struct initialization syntax, and relying on 0 init of global data.

Quoted 14 lines
include/uapi/linux/rseq.h +7 -2
-       __u8 __reserved;
+       union {
+               struct rseq_op_node rseq_op_list;
+               struct {
+                       __u64   op_used[2];
+                       __u64   reserved;
+               };
+       };
 
        /*
         * Flexible array member at end of structure, after last feature field.
--

2.54.0

#3 lore raw

Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions

Quoted 201 lines · Dmitry Vyukov
Quoted 31 lines · On Fri, 28 Aug 2026 at 17:33, <[email protected]> wrote:

From: Olivier Dion <[email protected]>

Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.

The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.

Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.

Overall, this puts the users in a situation of choosing between a fast sched_getcpu() and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibc

RSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them.

Add enum rseq_op_type and the rseq_op_node / rseq_op_reset / rseq_op_reset_with_stride structures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, the RSEQ_OP_LIST_LIMIT walk bound, and the PR_RSEQ_OP prctl with its REGISTER/UNREGISTER sub-commands.

Quoted 4 lines

The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq.

[0] Documentation/userspace-api/rseq.rst (Optimized RSEQ v2)

Hi Olivier,

This is very cool, thanks for working on this.

Quoted 35 lines
---
 include/uapi/linux/prctl.h |  12 +++++
 include/uapi/linux/rseq.h  | 104 ++++++++++++++++++++++++++++++++++---
 2 files changed, 108 insertions(+), 8 deletions(-)
include/uapi/linux/prctl.h +12 -0
index b6ec6f693719..4cb6356a271a 100644
@@ -396,6 +396,18 @@ struct prctl_mm_map {
  */
 # define PR_RSEQ_SLICE_EXT_ENABLE              0x01
 
+/*
+ * RSEQ operation registration.
+ *
+ * arg3 is the user address of a struct rseq_op_node embedded in one of the
+ * rseq operation structures (see uapi/linux/rseq.h). Registering the first
+ * operation enables rseq operation processing for the thread; unregistering
+ * the last one disables it.
+ */
+#define PR_RSEQ_OP                             82
+# define PR_RSEQ_OP_REGISTER                   1
+# define PR_RSEQ_OP_UNREGISTER                 2
+
 /*
  * Get or set the control flow integrity (CFI) configuration for the
  * current thread.
include/uapi/linux/rseq.h +0 -0
index ca6fe1f9d05e..b664d1991c48 100644

include/uapi/linux/rseq.h line 19

Quoted 7 lines
include/uapi/linux/rseq.h +4 -0
@@ -13,6 +13,11 @@
 #include <linux/types.h>
 #include <asm/byteorder.h>
 
+/*
+ * Maximum number of nodes walked in the rseq operation list.
+ */
+#define RSEQ_OP_LIST_LIMIT     2048

Does this belong to the user header? Kernel and userspace are not necessary built with the same headers, and we don't necessary want to set this const in stone. I think it may be more flexible to make this impl detail and just return ENOSPC.

Quoted 24 lines
include/uapi/linux/rseq.h +6 -0
 enum rseq_cpu_id_state {
        RSEQ_CPU_ID_UNINITIALIZED               = -1,
        RSEQ_CPU_ID_REGISTRATION_FAILED         = -2,
@@ -33,6 +38,8 @@ enum rseq_cs_flags_bit {
        /* User read only feature flags */
        RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT    = 4,
        RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT      = 5,
+       RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT      = 6,
+       RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT        = 7,
 };
 
 enum rseq_cs_flags {
@@ -47,6 +54,10 @@ enum rseq_cs_flags {
                (1U << RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT),
        RSEQ_CS_FLAG_SLICE_EXT_ENABLED          =
                (1U << RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT),
+       RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE          =
+               (1U << RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT),
+       RSEQ_CS_FLAG_RSEQ_OP_ENABLED            =
+               (1U << RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT),
 };
 
 /*

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 106

Quoted 10 lines
include/uapi/linux/rseq.h +7 -0
@@ -86,6 +97,77 @@ struct rseq_slice_ctrl {
        };
 };
 
+/*
+ * enum rseq_op_type - Type of an rseq operation
+ * @RSEQ_OP_RESET:                     Plain reset. Uses struct rseq_op_reset.
+ * @RSEQ_OP_RESET_WITH_STRIDE_CPUID:   Reset indexed by the current CPU ID.
+ *                                     Uses struct rseq_op_reset_with_stride.
+ * @RSEQ_OP_RESET_WITH_STRIDE_MMCID:   Reset indexed by the current MM CID.
+ *                                     Uses struct rseq_op_reset_with_stride.

Do you have use-cases for these STRIDE ops?
I can think of some, but they would require executing ops when a task is descheduled, rather than when it's scheduled in.

Quoted 11 lines
include/uapi/linux/rseq.h +11 -0
+ */
+enum rseq_op_type {
+       RSEQ_OP_RESET,
+       RSEQ_OP_RESET_WITH_STRIDE_CPUID,
+       RSEQ_OP_RESET_WITH_STRIDE_MMCID,
+       RSEQ_OP_NR,
+};
+
+/*
+ * struct rseq_op_node - Common header linking an rseq operation into the list
+ * @next:      Address of the next node. Owned by the kernel.

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 127

Quoted 10 lines
include/uapi/linux/rseq.h +10 -0
+ * @prev:      Address of the previous node. Owned by the kernel.
+ * @type:      Operation type. See enum rseq_op_type.
+ * @reserved:  Must be zero on registration.
+ *
+ * User space allocates the node, sets @type and zeroes @next, @prev and
+ * @reserved before passing it to prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, node).
+ * The kernel owns @next and @prev for the lifetime of the registration and
+ * links the node into a circular doubly-linked list anchored by an internal
+ * sentinel in struct rseq. User space must not touch @next or @prev while the
+ * node is registered.

Have you considered keeping copies of these structs in the kernel? That (1) is more consistent with most of the other kernel APIs, (2) will make executing ops faster since we don't need to copy_from_user all the time, (3) will make code much simpler since we won't need to re-verify all data, (4) will make the code more secure since user-space won't be able to mess with these structs, (5) will simplify the contract and documentation.
There are robust lists that keep lists in user-space, but these are supposed to be concurrently modified by userspace. But these rseq ops are generally not expected to be added/removed often (and the current contract does not allow that anyway).

Quoted 4 lines
include/uapi/linux/rseq.h +4 -0
+ */
+struct rseq_op_node {
+       __u64 next;
+       __u64 prev;

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 141

Quoted 10 lines
include/uapi/linux/rseq.h +10 -0
+       struct {
+               __u8  type; /* enum rseq_op_type */
+               __u8  reserved[7];
+       };
+};
+
+/*
+ * struct rseq_op_reset - Reset one word to a value on return to user space
+ * @node:      Operation list node.
+ * @src:       Address of the source word, or 0 to reset @dst to zero.

Is "or 0 to reset" part implemented in the series?

It would be useful to see an op that contains a const that is written, rather than an address. Address invovles additional indirection + copy_from_user.

And the contract for concurrent changes of the value pointed by src from user-space are not documented (atomicity?), w/o that having the address is not very useful.

Quoted 12 lines
include/uapi/linux/rseq.h +12 -0
+ * @dst:       Address of the destination word.
+ * @len:       Word length in bytes. Must be 4 or 8 (8 is 64-bit only).
+ */
+struct rseq_op_reset {
+       struct rseq_op_node     node;
+       __u64                   src;
+       __u64                   dst;
+       __u32                   len;
+};
+
+/*
+ * struct rseq_op_reset_with_stride - Reset one word in a strided array
Quoted 8 lines

include/uapi/linux/rseq.h struct rseq_slice_ctrl { · line 161

Quoted 8 lines
include/uapi/linux/rseq.h +8 -0
+ * @node:      Operation list node.
+ * @src:       Address of the source word, or 0 to reset the slot to zero.
+ * @dst:       Base address of the strided destination array.
+ * @dst_stride:        Stride in bytes between consecutive array slots.
+ * @len:       Word length in bytes. Must be 4 or 8 (8 is 64-bit only).
+ *
+ * The destination slot is @dst + @dst_stride * index, where index is the
+ * current CPU ID or MM CID depending on the operation type.

It would be useful to explicitly clarify when is the max value for index.

I understand it's "number of CPUs". But what about online/offline CPUs? Where/how user is expected to read the right value? If I use CIDs and affinity mask, can I expect the index is bounded by the affinity bits?

Quoted 52 lines
Quoted 21 lines
include/uapi/linux/rseq.h +9 -5
+ */
+struct rseq_op_reset_with_stride {
+       struct rseq_op_node     node;
+       __u64                   src;
+       __u64                   dst;
+       __u64                   dst_stride;
+       __u32                   len;
+};
+
 /*
  * The original size and alignment of the allocation for struct rseq is
  * 32 bytes.
@@ -191,15 +273,21 @@ struct rseq {
        struct rseq_slice_ctrl slice_ctrl;
 
        /*
-        * Before rseq became extensible, its original size was 32 bytes even
-        * though the active rseq area was only 20 bytes.
-        * Exposing a 32 bytes feature size would make life needlessly painful
-        * for userspace. Therefore, add a reserved byte after byte 32
-        * to bump the rseq feature size from 32 to 33.

include/uapi/linux/rseq.h struct rseq { · line 283

Quoted 10 lines
include/uapi/linux/rseq.h +7 -2
-        * The next field to be added to the rseq area will be larger
-        * than one byte, and will replace this reserved byte.
+        * Sentinel of the circular doubly-linked list of rseq operations
+        * registered via prctl(PR_RSEQ_OP, ...). Fully owned and maintained by
+        * the kernel: it is initialized to point to itself on registration and
+        * user space must never read or write it directly.
+        *
+        * The kernel only use next and prev from rseq_op_list.  The rest of the
+        * bytes are reserved for later usage and should be zeroed.
         */

Do we really need the union and the op_used/reserved? Isn't it easier to say that everything needs to be 0'ed? What would userspace do with op_used? It's not possible to skip initialization of these fields when using both memset, struct initialization syntax, and relying on 0 init of global data.

Quoted 14 lines
include/uapi/linux/rseq.h +7 -2
-       __u8 __reserved;
+       union {
+               struct rseq_op_node rseq_op_list;
+               struct {
+                       __u64   op_used[2];
+                       __u64   reserved;
+               };
+       };
 
        /*
         * Flexible array member at end of structure, after last feature field.
--

2.54.0

#4 lore raw

Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions

In reply to [email protected]
Quoted 11 lines

Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space.

The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region.

Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.

The commit message does not quite say how this addresses tcmalloc needs. I assume the idea is to reset some other data structure and not the rseq fields. Is my assumption correct?

Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.

My concern is that this would turn into another bytecode interpreter over time, basically reimplementing BPF.

Thanks,
Florian

#5 lore raw

Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions

On Mon, 31 Aug 2026, Florian Weimer <[email protected]> wrote:
[...]

Quoted 9 lines
Quoted 5 lines

Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache.

The commit message does not quite say how this addresses tcmalloc needs. I assume the idea is to reset some other data structure and not the rseq fields. Is my assumption correct?

Indeed. That would allow TCMalloc to not have to overlap their pointer cache with their RSEQ area. In other words, they could be using the glibc area and be compatible with the rest of the ecosystem.

Quoted 3 lines

Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.

Well I had in mind another solution, albeit some might find it ugly but I think it does have the merit to be explored.

Basically, instead of doing all the operations in the kernel, we could setup something akin to a signal handler by pushing a frame on the userspace stack and resuming the execution of the thread into that "handler". Userspace is now responsible of everything, including returning to the original IP.

However, we need to be careful with things like the red-zone, the shadow-track and the extended states. Other than that, the only thing I am unsure is how to handle nesting. Perhaps someone with more knowledges could expand on that.

Quoted 2 lines

My concern is that this would turn into another bytecode interpreter over time, basically reimplementing BPF.

I did think about adding BPF type to run BPF bytecode. However, I don't think it fits well the goal here and would make the feature unusable on configurations where BPF is not available.

Overall, my idea with the RSEQ operations was make that could benifit the userspace as a whole instead of handling only the TCMalloc situation. Otherwise, I would have just name it RSEQ reset.

For example, there is a CPU-stride variant of the reset operation. I dabble with this operation to cache a TLS that is global-dynamic into a per-cpu cache. Thus, making a TLS global-dynamic access as fast as a TLS initial-exec.

Thanks,
Olivier

Signature · 3 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
#6 lore raw

Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions

* Olivier Dion:

Quoted 4 lines
Quoted 3 lines

Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.

Quoted 8 lines

Well I had in mind another solution, albeit some might find it ugly but I think it does have the merit to be explored.

Basically, instead of doing all the operations in the kernel, we could setup something akin to a signal handler by pushing a frame on the userspace stack and resuming the execution of the thread into that "handler". Userspace is now responsible of everything, including returning to the original IP.

On mere context switch (not signal delivery), I don't think the kernel assumes that there is a valid stack pointer. These scheme would change that. This would be fine for managed code, but this scheme has to work with arbitrary code that reassigns registers temporarily.

(We had a longjmp on POWER for a while that corrupted the stack pointer temporarily, and that was visible only upon signal delivery, not regular context switch.)

Quoted 4 lines

For example, there is a CPU-stride variant of the reset operation. I dabble with this operation to cache a TLS that is global-dynamic into a per-cpu cache. Thus, making a TLS global-dynamic access as fast as a TLS initial-exec.

I don't think this can work? The problem with global-dynamic TLS is that it can't be a single memory block (it may have to grow). Even if the access path changes, we can't relocate the memory because TLS objects must live in the linear address space.

We can do away with the current complicated scheme in glibc and use an array of arrays, where the inner arrays follow an exponential growth policy (for size and alignment). That would still have dependent loads, but would be quite a bit faster than our current global-dynamic scheme.

New TLS allocator for glibc
<https://conf.gnu-tools-cauldron.org/opo25/talk/LQTU3G/>

(Sadly still not implemented.)

Thanks,
Florian

#7 lore raw

Re: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions

Quoted 13 lines · Florian Weimer
Quoted 4 lines
Quoted 3 lines

Would it be possible to use actual CPU instructions running in userspace for this? I assume not because it's not possible to restore the register contents.

Quoted 8 lines

Well I had in mind another solution, albeit some might find it ugly but I think it does have the merit to be explored.

Basically, instead of doing all the operations in the kernel, we could setup something akin to a signal handler by pushing a frame on the userspace stack and resuming the execution of the thread into that "handler". Userspace is now responsible of everything, including returning to the original IP.

Quoted 4 lines

On mere context switch (not signal delivery), I don't think the kernel assumes that there is a valid stack pointer. These scheme would change that. This would be fine for managed code, but this scheme has to work with arbitrary code that reassigns registers temporarily.

You are right that aside of signal handling there is no assumption, but as signal handling can happen at arbitrary times user space should not play with SP unless there is an alternative signal stack, which has its own nasties as we all know.

If there is truly code which needs to trainwreck SP for a sane reason and not just because and has a good reason to utilize such a library/functionality, then we can work around that by a mandating a separate "fixup" stack per thread to enable this functionality. Then the screwed up user space stack pointer becomes completely irrelevant and the only interesting register on the kernel entry stack is the return IP which got pushed by the hardware. And provide an option to work without this indirection of course. :)

There are probably other solutions, but I really want to move this to user space and not introduce yet another limited single purpose "interpreter" in the kernel, which will be "extended" and duct taped forever.

TBH. At some point we have to stop pretending that we can support all possible insanities and their arbitrary combination. That just limits ourself and puts the burden and the complexity into the wrong places just because.

Thanks,

tglx

#8 lore raw

RFC PATCH 2/5 rseq: add per-task rseq operation state

In reply to [email protected]

From: Olivier Dion <[email protected]>

Add the task-side state for rseq operations: the rseq_event::rseq_op state indicating operation processing is enabled for the task, and the nr_ops counter in struct rseq_data. nr_ops is edge triggered: its 0<->1 transition enables/disables rseq_event::rseq_op and the user visible RSEQ_CS_FLAG_RSEQ_OP_ENABLED flag. This makes so that only thread that enable RSEQ operations through prctl takes a performance hit.

Include rseq_op in the sched-switch raise condition so pending operations force a return through the rseq exit path, and declare the rseq_op_prctl() entry point (with a CONFIG_RSEQ=n stub).

---
 include/linux/rseq.h       | 10 +++++++++-
 include/linux/rseq_types.h |  9 +++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
include/linux/rseq.h +9 -1
index 7ef79b25e714..d1e33906d863 100644
@@ -69,7 +69,9 @@ static __always_inline void rseq_sched_switch_event(struct task_struct *t)
 		 * was via interrupt from user space. ev->has_rseq does not have
 		 * to be evaluated here because rseq_v2() implies has_rseq.
 		 */
-		bool raise = ev->user_irq | ev->ids_changed;
+		bool raise = (ev->user_irq |
+			      ev->ids_changed |
+			      ev->rseq_op);
 
 		if (raise) {
 			ev->sched_switch = true;
@@ -172,6 +174,8 @@ static inline unsigned int rseq_alloc_align(void)
 	return 1U << get_count_order(offsetof(struct rseq, end));
 }
 
+int rseq_op_prctl(unsigned long arg2, unsigned long arg3);
+
 #else /* CONFIG_RSEQ */
 static inline bool rseq_v2(struct task_struct *t) { return false; }
 static inline void rseq_handle_slowpath(struct pt_regs *regs) { }
@@ -182,6 +186,10 @@ static inline void rseq_force_update(void) { }
 static inline void rseq_virt_userspace_exit(void) { }
 static inline void rseq_fork(struct task_struct *t, u64 clone_flags) { }
 static inline void rseq_execve(struct task_struct *t) { }
+static inline int rseq_op_prctl(unsigned long arg2, unsigned long arg3)
+{
+	return -ENOTSUPP;
+}
 #endif  /* !CONFIG_RSEQ */
 
 #ifdef CONFIG_DEBUG_RSEQ
include/linux/rseq_types.h +9 -0
index 85739a63e85e..059292695c34 100644
@@ -23,6 +23,7 @@ struct rseq;
  *			exit to user
  * @ids_changed:	Indicator that IDs need to be updated
  * @user_irq:		True on interrupt entry from user mode
+ * @rseq_op:		Rseq operation processing is enabled for the task
  * @has_rseq:		Greater than 0 if the task has a rseq pointer installed.
  *			Contains the RSEQ version number
  * @error:		Compound error code for the slow path to analyze
@@ -44,6 +45,7 @@ struct rseq_event {
 					u8	sched_switch;
 					u8	ids_changed;
 					u8	user_irq;
+					u8	rseq_op;
 				};
 			};
 
@@ -115,6 +117,7 @@ struct rseq_slice {
  * @event:	Storage for event management
  * @ids:	Storage for cached CPU ID and MM CID
  * @slice:	Storage for time slice extension data
+ * @nr_ops:	Number of registered rseq operations
  */
 struct rseq_data {
 	struct rseq __user		*usrptr;
@@ -125,6 +128,12 @@ struct rseq_data {
 #ifdef CONFIG_RSEQ_SLICE_EXTENSION
 	struct rseq_slice		slice;
 #endif
+	/*
+	 * Number of rseq operations registered for the task. Edge triggered:
+	 * the 0<->1 transition enables/disables rseq_event::rseq_op and the
+	 * RSEQ_CS_FLAG_RSEQ_OP_ENABLED user flag.
+	 */
+	u32				nr_ops;
 };
 
 #else /* CONFIG_RSEQ */
Signature · 1 line
2.54.0
#9 lore raw

Re: [RFC PATCH 2/5] rseq: add per-task rseq operation state

Quoted 84 lines · On Fri, 28 Aug 2026 at 17:34, <[email protected]> wrote:

From: Olivier Dion <[email protected]>

Add the task-side state for rseq operations: the rseq_event::rseq_op state indicating operation processing is enabled for the task, and the nr_ops counter in struct rseq_data. nr_ops is edge triggered: its 0<->1 transition enables/disables rseq_event::rseq_op and the user visible RSEQ_CS_FLAG_RSEQ_OP_ENABLED flag. This makes so that only thread that enable RSEQ operations through prctl takes a performance hit.

Include rseq_op in the sched-switch raise condition so pending operations force a return through the rseq exit path, and declare the rseq_op_prctl() entry point (with a CONFIG_RSEQ=n stub).

---
 include/linux/rseq.h       | 10 +++++++++-
 include/linux/rseq_types.h |  9 +++++++++
 2 files changed, 18 insertions(+), 1 deletion(-)
include/linux/rseq.h +9 -1
index 7ef79b25e714..d1e33906d863 100644
@@ -69,7 +69,9 @@ static __always_inline void rseq_sched_switch_event(struct task_struct *t)
                 * was via interrupt from user space. ev->has_rseq does not have
                 * to be evaluated here because rseq_v2() implies has_rseq.
                 */
-               bool raise = ev->user_irq | ev->ids_changed;
+               bool raise = (ev->user_irq |
+                             ev->ids_changed |
+                             ev->rseq_op);
 
                if (raise) {
                        ev->sched_switch = true;
@@ -172,6 +174,8 @@ static inline unsigned int rseq_alloc_align(void)
        return 1U << get_count_order(offsetof(struct rseq, end));
 }
 
+int rseq_op_prctl(unsigned long arg2, unsigned long arg3);
+
 #else /* CONFIG_RSEQ */
 static inline bool rseq_v2(struct task_struct *t) { return false; }
 static inline void rseq_handle_slowpath(struct pt_regs *regs) { }
@@ -182,6 +186,10 @@ static inline void rseq_force_update(void) { }
 static inline void rseq_virt_userspace_exit(void) { }
 static inline void rseq_fork(struct task_struct *t, u64 clone_flags) { }
 static inline void rseq_execve(struct task_struct *t) { }
+static inline int rseq_op_prctl(unsigned long arg2, unsigned long arg3)
+{
+       return -ENOTSUPP;
+}
 #endif  /* !CONFIG_RSEQ */
 
 #ifdef CONFIG_DEBUG_RSEQ
include/linux/rseq_types.h +3 -0
index 85739a63e85e..059292695c34 100644
@@ -23,6 +23,7 @@ struct rseq;
  *                     exit to user
  * @ids_changed:       Indicator that IDs need to be updated
  * @user_irq:          True on interrupt entry from user mode
+ * @rseq_op:           Rseq operation processing is enabled for the task
  * @has_rseq:          Greater than 0 if the task has a rseq pointer installed.
  *                     Contains the RSEQ version number
  * @error:             Compound error code for the slow path to analyze
@@ -44,6 +45,7 @@ struct rseq_event {
                                        u8      sched_switch;
                                        u8      ids_changed;
                                        u8      user_irq;
+                                       u8      rseq_op;
                                };
                        };
 
@@ -115,6 +117,7 @@ struct rseq_slice {
  * @event:     Storage for event management
  * @ids:       Storage for cached CPU ID and MM CID
  * @slice:     Storage for time slice extension data
+ * @nr_ops:    Number of registered rseq operations
  */
 struct rseq_data {
        struct rseq __user              *usrptr;

include/linux/rseq_types.h struct rseq_data { · line 136

Quoted 9 lines
include/linux/rseq_types.h +6 -0
@@ -125,6 +128,12 @@ struct rseq_data {
 #ifdef CONFIG_RSEQ_SLICE_EXTENSION
        struct rseq_slice               slice;
 #endif
+       /*
+        * Number of rseq operations registered for the task. Edge triggered:
+        * the 0<->1 transition enables/disables rseq_event::rseq_op and the
+        * RSEQ_CS_FLAG_RSEQ_OP_ENABLED user flag.
+        */
+       u32                             nr_ops;

If we copy ops to kernel memory, it would be useful to have 1 op embeded here (with fallback array for more ops). For now the only known use-case for this is memory allocators, so I would assume in most cases it will be either 0 or 1 ops registered.

Quoted 6 lines
include/linux/rseq_types.h +0 -1
 };
 
 #else /* CONFIG_RSEQ */
--

2.54.0

#10 lore raw

RFC PATCH 3/5 rseq: apply operations on exit to user space

In reply to [email protected]

From: Olivier Dion <[email protected]>

Add rseq_apply_ops(), which walks the task's circular operation list on return to user space and applies each operation.

Wire the walk into rseq_exit_user_update() for both the ids-unchanged and ids-changed paths, gated on rseq_event::rseq_op. Since operation processing is a persistent per-task state rather than a one-shot event, introduce rseq_clear_one_shot_events() which preserves the rseq_op bit when clearing events on the way out.

len, src and dst are re-read from user memory and re-validated on every application, since user space may change them after registration. A broken ABI contract will fault the process.

---
 include/linux/rseq_entry.h | 157 +++++++++++++++++++++++++++++++++++--
 1 file changed, 152 insertions(+), 5 deletions(-)
include/linux/rseq_entry.h +152 -5
index ed9da6e41a2a..df083b0ee14a 100644
@@ -247,6 +247,137 @@ static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un
 #define rseq_slice_clear_user(rseq, efault) do { } while (0)
 #endif /* !CONFIG_RSEQ_SLICE_EXTENSION */
 
+/*
+ * Validate and perform a single reset word operation: reset @dst to *@src,
+ * or to zero when @src is 0. @len, @src and @dst come from user memory and
+ * are re-validated on every application, since user space may have changed
+ * them since registration. A broken ABI contract returns false, which faults
+ * the process.
+ */
+static __always_inline bool rseq_op_reset_word(u64 src, u64 dst, u32 len,
+					       void __user *node)
+{
+	if (unlikely(!IS_ALIGNED(dst, len) || (src && !IS_ALIGNED(src, len)))) {
+		pr_info_ratelimited("rseq: bad operation alignment len=%u src=%llx dst=%llx from %p\n",
+				len, src, dst, node);
+		return false;
+	}
+
+	switch (len) {
+	case 4: {
+		u32 __user *udst = (u32 __user *)dst;
+		u32 v = 0;
+
+		if (src) {
+			u32 __user *usrc = (u32 __user *)src;
+
+			scoped_user_read_access(usrc, efault)
+				unsafe_get_user(v, usrc, efault);
+		}
+		scoped_user_write_access(udst, efault)
+			unsafe_put_user(v, udst, efault);
+		return true;
+	}
+	case 8: {
+		u64 __user *udst = (u64 __user *)dst;
+		u64 v = 0;
+
+		if (src) {
+			u64 __user *usrc = (u64 __user *)src;
+
+			scoped_user_read_access(usrc, efault)
+				unsafe_get_user(v, usrc, efault);
+		}
+		scoped_user_write_access(udst, efault)
+			unsafe_put_user(v, udst, efault);
+		return true;
+	}
+	default:
+		pr_info("rseq: bad operation length=%u from %p\n", len, node);
+		return false;
+	}
+efault:
+	pr_info("rseq: fault while applying operation from %p\n", node);
+	return false;
+}
+
+static __always_inline bool rseq_apply_ops(struct task_struct *t)
+{
+	struct rseq __user *rseq = t->rseq.usrptr;
+	struct rseq_op_node __user *sentinel = &rseq->rseq_op_list;
+	struct rseq_op_node __user *node;
+	unsigned int limit = RSEQ_OP_LIST_LIMIT;
+	u64 next;
+
+	WARN_ONCE(!t->rseq.event.rseq_op, "rseq operations not enabled");
+
+	scoped_user_read_access(sentinel, efault)
+		unsafe_get_user(next, &sentinel->next, efault);
+
+	node = (struct rseq_op_node __user *)next;
+
+	while (node != sentinel && limit--) {
+		u8 type;
+
+		scoped_user_read_access(node, efault) {
+			unsafe_get_user(type, &node->type, efault);
+			unsafe_get_user(next, &node->next, efault);
+		}
+
+		switch (type) {
+		case RSEQ_OP_RESET: {
+			struct rseq_op_reset __user *op =
+				(struct rseq_op_reset __user *)node;
+			u64 src, dst;
+			u32 len;
+
+			scoped_user_read_access(op, efault) {
+				unsafe_get_user(src, &op->src, efault);
+				unsafe_get_user(dst, &op->dst, efault);
+				unsafe_get_user(len, &op->len, efault);
+			}
+
+			if (!rseq_op_reset_word(src, dst, len, node))
+				return false;
+			break;
+		}
+		case RSEQ_OP_RESET_WITH_STRIDE_CPUID: /* fall through */
+		case RSEQ_OP_RESET_WITH_STRIDE_MMCID: {
+			struct rseq_op_reset_with_stride __user *op =
+				(struct rseq_op_reset_with_stride __user *)node;
+			u64 src, dst, dst_stride;
+			u32 len, index;
+
+			scoped_user_read_access(op, efault) {
+				unsafe_get_user(src, &op->src, efault);
+				unsafe_get_user(dst, &op->dst, efault);
+				unsafe_get_user(dst_stride, &op->dst_stride, efault);
+				unsafe_get_user(len, &op->len, efault);
+			}
+			index = (type == RSEQ_OP_RESET_WITH_STRIDE_CPUID) ?
+				t->rseq.ids.cpu_id : t->rseq.ids.mm_cid;
+			dst += dst_stride * index;
+
+			if (!rseq_op_reset_word(src, dst, len, node))
+				return false;
+
+			break;
+		}
+		default:
+			pr_info("rseq: bad operation type=%u from %p\n",
+				type, node);
+			return false;
+		}
+
+		node = (struct rseq_op_node __user *)next;
+	}
+
+	return node == sentinel;
+efault:
+	pr_info("rseq: fault while walking operation list\n");
+	return false;
+}
+
 bool rseq_debug_update_user_cs(struct task_struct *t, struct pt_regs *regs, unsigned long csaddr);
 
 static __always_inline void rseq_note_user_irq_entry(void)
@@ -632,6 +763,10 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct t
 			if (unlikely(!rseq_update_user_cs(t, regs, csaddr)))
 				return false;
 		}
+
+		if (t->rseq.event.rseq_op && !rseq_apply_ops(t))
+			return false;
+
 		return true;
 	}
 
@@ -642,11 +777,22 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct t
 		.node_id = cpu_to_node(cpu),
 	};
 
-	return rseq_update_usr(t, regs, &ids);
+	if (!rseq_update_usr(t, regs, &ids))
+		return false;
+
+	if (t->rseq.event.rseq_op && !rseq_apply_ops(t))
+		return false;
+
+	return true;
 efault:
 	return false;
 }
 
+static __always_inline void rseq_clear_one_shot_events(struct rseq_event *ev)
+{
+	ev->events &= (struct rseq_event){ .rseq_op = true }.events;
+}
+
 static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *regs)
 {
 	struct task_struct *t = current;
@@ -674,8 +820,9 @@ static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *reg
 		if (unlikely(!rseq_exit_user_update(regs, t)))
 			return true;
 	}
-	/* Clear state so next entry starts from a clean slate */
-	t->rseq.event.events = 0;
+	/* Clear one-shot events so next entry starts from a clean slate */
+	rseq_clear_one_shot_events(&t->rseq.event);
+
 	return false;
 }
 
@@ -730,7 +877,7 @@ static __always_inline void rseq_syscall_exit_to_user_mode(void)
 	/* Needed to remove the store for the !lockdep case */
 	if (IS_ENABLED(CONFIG_LOCKDEP)) {
 		WARN_ON_ONCE(ev->sched_switch);
-		ev->events = 0;
+		rseq_clear_one_shot_events(ev);
 	}
 }
 
@@ -747,7 +894,7 @@ static __always_inline void rseq_irqentry_exit_to_user_mode(void)
 	 * interrupt did not result in a schedule and therefore the
 	 * rseq processing could not clear it.
 	 */
-	ev->events = 0;
+	rseq_clear_one_shot_events(ev);
 }
 
 void __rseq_debug_syscall_return(struct pt_regs *regs);
Signature · 1 line
2.54.0
#11 lore raw

Re: [RFC PATCH 3/5] rseq: apply operations on exit to user space

Quoted 66 lines · On Fri, 28 Aug 2026 at 17:34, <[email protected]> wrote:

From: Olivier Dion <[email protected]>

Add rseq_apply_ops(), which walks the task's circular operation list on return to user space and applies each operation.

Wire the walk into rseq_exit_user_update() for both the ids-unchanged and ids-changed paths, gated on rseq_event::rseq_op. Since operation processing is a persistent per-task state rather than a one-shot event, introduce rseq_clear_one_shot_events() which preserves the rseq_op bit when clearing events on the way out.

len, src and dst are re-read from user memory and re-validated on every application, since user space may change them after registration. A broken ABI contract will fault the process.

---
 include/linux/rseq_entry.h | 157 +++++++++++++++++++++++++++++++++++--
 1 file changed, 152 insertions(+), 5 deletions(-)
include/linux/rseq_entry.h +37 -0
index ed9da6e41a2a..df083b0ee14a 100644
@@ -247,6 +247,137 @@ static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un
 #define rseq_slice_clear_user(rseq, efault) do { } while (0)
 #endif /* !CONFIG_RSEQ_SLICE_EXTENSION */
 
+/*
+ * Validate and perform a single reset word operation: reset @dst to *@src,
+ * or to zero when @src is 0. @len, @src and @dst come from user memory and
+ * are re-validated on every application, since user space may have changed
+ * them since registration. A broken ABI contract returns false, which faults
+ * the process.
+ */
+static __always_inline bool rseq_op_reset_word(u64 src, u64 dst, u32 len,
+                                              void __user *node)
+{
+       if (unlikely(!IS_ALIGNED(dst, len) || (src && !IS_ALIGNED(src, len)))) {
+               pr_info_ratelimited("rseq: bad operation alignment len=%u src=%llx dst=%llx from %p\n",
+                               len, src, dst, node);
+               return false;
+       }
+
+       switch (len) {
+       case 4: {
+               u32 __user *udst = (u32 __user *)dst;
+               u32 v = 0;
+
+               if (src) {
+                       u32 __user *usrc = (u32 __user *)src;
+
+                       scoped_user_read_access(usrc, efault)
+                               unsafe_get_user(v, usrc, efault);
+               }
+               scoped_user_write_access(udst, efault)
+                       unsafe_put_user(v, udst, efault);
+               return true;
+       }
+       case 8: {
+               u64 __user *udst = (u64 __user *)dst;
+               u64 v = 0;
+
+               if (src) {
+                       u64 __user *usrc = (u64 __user *)src;

include/linux/rseq_entry.h static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un · line 296

Quoted 10 lines
include/linux/rseq_entry.h +10 -0
+
+                       scoped_user_read_access(usrc, efault)
+                               unsafe_get_user(v, usrc, efault);
+               }
+               scoped_user_write_access(udst, efault)
+                       unsafe_put_user(v, udst, efault);
+               return true;
+       }
+       default:
+               pr_info("rseq: bad operation length=%u from %p\n", len, node);

pr_info_ratelimited here and below.
Kernel does not generally warn on the global console about misbehaving userspace programs (which generally do not have access to the output anyway, as it requires root nowadays).
If you decide to do that, it should be at least rate-limited to avoid DoS.

Quoted 56 lines
include/linux/rseq_entry.h +56 -0
+               return false;
+       }
+efault:
+       pr_info("rseq: fault while applying operation from %p\n", node);
+       return false;
+}
+
+static __always_inline bool rseq_apply_ops(struct task_struct *t)
+{
+       struct rseq __user *rseq = t->rseq.usrptr;
+       struct rseq_op_node __user *sentinel = &rseq->rseq_op_list;
+       struct rseq_op_node __user *node;
+       unsigned int limit = RSEQ_OP_LIST_LIMIT;
+       u64 next;
+
+       WARN_ONCE(!t->rseq.event.rseq_op, "rseq operations not enabled");
+
+       scoped_user_read_access(sentinel, efault)
+               unsafe_get_user(next, &sentinel->next, efault);
+
+       node = (struct rseq_op_node __user *)next;
+
+       while (node != sentinel && limit--) {
+               u8 type;
+
+               scoped_user_read_access(node, efault) {
+                       unsafe_get_user(type, &node->type, efault);
+                       unsafe_get_user(next, &node->next, efault);
+               }
+
+               switch (type) {
+               case RSEQ_OP_RESET: {
+                       struct rseq_op_reset __user *op =
+                               (struct rseq_op_reset __user *)node;
+                       u64 src, dst;
+                       u32 len;
+
+                       scoped_user_read_access(op, efault) {
+                               unsafe_get_user(src, &op->src, efault);
+                               unsafe_get_user(dst, &op->dst, efault);
+                               unsafe_get_user(len, &op->len, efault);
+                       }
+
+                       if (!rseq_op_reset_word(src, dst, len, node))
+                               return false;
+                       break;
+               }
+               case RSEQ_OP_RESET_WITH_STRIDE_CPUID: /* fall through */
+               case RSEQ_OP_RESET_WITH_STRIDE_MMCID: {
+                       struct rseq_op_reset_with_stride __user *op =
+                               (struct rseq_op_reset_with_stride __user *)node;
+                       u64 src, dst, dst_stride;
+                       u32 len, index;
+
+                       scoped_user_read_access(op, efault) {
+                               unsafe_get_user(src, &op->src, efault);

include/linux/rseq_entry.h static __always_inline bool rseq_grant_slice_extension(unsigned long ti_work, un · line 362

Quoted 10 lines
include/linux/rseq_entry.h +10 -0
+                               unsafe_get_user(dst, &op->dst, efault);
+                               unsafe_get_user(dst_stride, &op->dst_stride, efault);
+                               unsafe_get_user(len, &op->len, efault);
+                       }
+                       index = (type == RSEQ_OP_RESET_WITH_STRIDE_CPUID) ?
+                               t->rseq.ids.cpu_id : t->rseq.ids.mm_cid;
+                       dst += dst_stride * index;
+
+                       if (!rseq_op_reset_word(src, dst, len, node))
+                               return false;

Declaration of src/dst/len and rseq_op_reset_word call potentially can be deduplicated across these 2 cases.

Quoted 89 lines
include/linux/rseq_entry.h +39 -6
+
+                       break;
+               }
+               default:
+                       pr_info("rseq: bad operation type=%u from %p\n",
+                               type, node);
+                       return false;
+               }
+
+               node = (struct rseq_op_node __user *)next;
+       }
+
+       return node == sentinel;
+efault:
+       pr_info("rseq: fault while walking operation list\n");
+       return false;
+}
+
 bool rseq_debug_update_user_cs(struct task_struct *t, struct pt_regs *regs, unsigned long csaddr);
 
 static __always_inline void rseq_note_user_irq_entry(void)
@@ -632,6 +763,10 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct t
                        if (unlikely(!rseq_update_user_cs(t, regs, csaddr)))
                                return false;
                }
+
+               if (t->rseq.event.rseq_op && !rseq_apply_ops(t))
+                       return false;
+
                return true;
        }
 
@@ -642,11 +777,22 @@ static __always_inline bool rseq_exit_user_update(struct pt_regs *regs, struct t
                .node_id = cpu_to_node(cpu),
        };
 
-       return rseq_update_usr(t, regs, &ids);
+       if (!rseq_update_usr(t, regs, &ids))
+               return false;
+
+       if (t->rseq.event.rseq_op && !rseq_apply_ops(t))
+               return false;
+
+       return true;
 efault:
        return false;
 }
 
+static __always_inline void rseq_clear_one_shot_events(struct rseq_event *ev)
+{
+       ev->events &= (struct rseq_event){ .rseq_op = true }.events;
+}
+
 static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *regs)
 {
        struct task_struct *t = current;
@@ -674,8 +820,9 @@ static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_regs *reg
                if (unlikely(!rseq_exit_user_update(regs, t)))
                        return true;
        }
-       /* Clear state so next entry starts from a clean slate */
-       t->rseq.event.events = 0;
+       /* Clear one-shot events so next entry starts from a clean slate */
+       rseq_clear_one_shot_events(&t->rseq.event);
+
        return false;
 }
 
@@ -730,7 +877,7 @@ static __always_inline void rseq_syscall_exit_to_user_mode(void)
        /* Needed to remove the store for the !lockdep case */
        if (IS_ENABLED(CONFIG_LOCKDEP)) {
                WARN_ON_ONCE(ev->sched_switch);
-               ev->events = 0;
+               rseq_clear_one_shot_events(ev);
        }
 }
 
@@ -747,7 +894,7 @@ static __always_inline void rseq_irqentry_exit_to_user_mode(void)
         * interrupt did not result in a schedule and therefore the
         * rseq processing could not clear it.
         */
-       ev->events = 0;
+       rseq_clear_one_shot_events(ev);
 }
 
 void __rseq_debug_syscall_return(struct pt_regs *regs);
--

2.54.0

#12 lore raw

RFC PATCH 4/5 rseq: register and unregister operations via prctl

In reply to [email protected]

From: Olivier Dion <[email protected]>

Implement the PR_RSEQ_OP prctl. Registration splices a user-provided pristine node into the head of the circular operation list, and unregistration unsplices it after validating it is properly linked between its neighbours; the kernel owns the node's next/prev links for the lifetime of the registration. The 0<->1 nr_ops transition reflects the enabled state into rseq_event::rseq_op and the user visible flag via rseq_op_update_enabled().

Initialize the operation list sentinel as an empty self-pointing circular list on rseq registration, and reset nr_ops / rseq_op state there.

Apply pending operations on the slow path and on signal delivery, and factor the common "clear error and force SIGSEGV" fixup into force_fault().

---
 kernel/rseq.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++---
 kernel/sys.c  |   5 ++
 2 files changed, 212 insertions(+), 13 deletions(-)
kernel/rseq.c +207 -13
index e75e3a5e312c..a277c11dae99 100644
@@ -253,17 +253,18 @@ static bool rseq_handle_cs(struct task_struct *t, struct pt_regs *regs)
 static void rseq_slowpath_update_usr(struct pt_regs *regs)
 {
 	/*
-	 * Preserve has_rseq and user_irq state. The generic entry code clears
-	 * user_irq on the way out, the non-generic entry architectures are not
-	 * setting user_irq.
+	 * Preserve has_rseq, rseq_op and user_irq state. The generic entry
+	 * code clears user_irq on the way out, the non-generic entry
+	 * architectures are not setting user_irq.
 	 */
 	const struct rseq_event evt_mask = {
 		.has_rseq	= RSEQ_HAS_RSEQ_VERSION_MASK,
+		.rseq_op	= true,
 		.user_irq	= true,
 	};
 	struct task_struct *t = current;
 	struct rseq_ids ids;
-	bool event;
+	bool event, should_fault = false;
 
 	if (unlikely(t->flags & PF_EXITING))
 		return;
@@ -300,7 +301,12 @@ static void rseq_slowpath_update_usr(struct pt_regs *regs)
 
 	ids.node_id = cpu_to_node(ids.cpu_id);
 
-	if (unlikely(!rseq_update_usr(t, regs, &ids))) {
+	if (unlikely(!rseq_update_usr(t, regs, &ids)))
+		should_fault = true;
+	else if (t->rseq.event.rseq_op && unlikely(!rseq_apply_ops(t)))
+		should_fault = true;
+
+	if (should_fault) {
 		/*
 		 * Clear the errors just in case this might survive magically, but
 		 * leave the rest intact.
@@ -329,8 +335,20 @@ void __rseq_handle_slowpath(struct pt_regs *regs)
 	rseq_slowpath_update_usr(regs);
 }
 
+static inline void force_fault(int sig)
+{
+	/*
+	 * Clear the errors just in case this might survive magically, but leave
+	 * the rest intact.
+	 */
+	current->rseq.event.error = 0;
+	force_sigsegv(sig);
+}
+
 void __rseq_signal_deliver(int sig, struct pt_regs *regs)
 {
+	bool should_fault = false;
+
 	rseq_stat_inc(rseq_stats.signal);
 
 	/*
@@ -339,14 +357,13 @@ void __rseq_signal_deliver(int sig, struct pt_regs *regs)
 	 * the interrupted context as after this point the instruction
 	 * pointer in @regs points to the signal handler.
 	 */
-	if (unlikely(!rseq_handle_cs(current, regs))) {
-		/*
-		 * Clear the errors just in case this might survive
-		 * magically, but leave the rest intact.
-		 */
-		current->rseq.event.error = 0;
-		force_sigsegv(sig);
-	}
+	if (unlikely(!rseq_handle_cs(current, regs)))
+		should_fault = true;
+	else if (current->rseq.event.rseq_op && unlikely(!rseq_apply_ops(current)))
+		should_fault = true;
+
+	if (should_fault)
+		force_fault(sig);
 
 	/*
 	 * In legacy mode, force the update of IDs before returning to user
@@ -436,6 +453,8 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3
 				rseqfl |= RSEQ_CS_FLAG_SLICE_EXT_ENABLED;
 		}
 	}
+	if (version > 1)
+		rseqfl |= RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE;
 
 	scoped_user_write_access(rseq, efault) {
 		/*
@@ -458,8 +477,16 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3
 		 * registrations.
 		 */
 		if (version > 1) {
+			u64 sentinel = (u64)&rseq->rseq_op_list;
+
 			if (IS_ENABLED(CONFIG_RSEQ_SLICE_EXTENSION))
 				unsafe_put_user(0U, &rseq->slice_ctrl.all, efault);
+			/*
+			 * Initialize the rseq operation list sentinel as an
+			 * empty circular doubly-linked list pointing to itself.
+			 */
+			unsafe_put_user(sentinel, &rseq->rseq_op_list.next, efault);
+			unsafe_put_user(sentinel, &rseq->rseq_op_list.prev, efault);
 		}
 	}
 
@@ -474,6 +501,12 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3
 #ifdef CONFIG_RSEQ_SLICE_EXTENSION
 	current->rseq.slice.state.enabled = !!(rseqfl & RSEQ_CS_FLAG_SLICE_EXT_ENABLED);
 #endif
+	/*
+	 * A fresh registration starts with no operations, so operation
+	 * processing is disabled until the first one is registered.
+	 */
+	current->rseq.event.rseq_op = false;
+	current->rseq.nr_ops = 0;
 
 	/*
 	 * Ensure the cpu_id_start and cpu_id fields are updated before
@@ -887,3 +920,164 @@ device_initcall(rseq_slice_init);
 #else
 static void rseq_slice_ext_init(struct dentry *root_dir) { }
 #endif /* CONFIG_RSEQ_SLICE_EXTENSION */
+
+/*
+ * Reflect the operation enabled state into the user visible flags field.
+ * Called on the 0<->1 transition of nr_ops. The kill path is taken on fault
+ * because losing this update leaves user space and kernel state inconsistent.
+ */
+static int rseq_op_update_enabled(struct task_struct *t, bool enable)
+{
+	struct rseq __user *rseq = t->rseq.usrptr;
+	u32 rflags;
+
+	if (get_user(rflags, &rseq->flags))
+		return -EFAULT;
+
+	rflags &= ~RSEQ_CS_FLAG_RSEQ_OP_ENABLED;
+	rflags |= RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE;
+	if (enable)
+		rflags |= RSEQ_CS_FLAG_RSEQ_OP_ENABLED;
+
+	if (put_user(rflags, &rseq->flags))
+		return -EFAULT;
+
+	t->rseq.event.rseq_op = enable;
+	return 0;
+}
+
+/*
+ * Register @node at the head of the circular doubly-linked operation list
+ * anchored by the kernel owned sentinel in struct rseq.
+ */
+static int rseq_op_register(struct task_struct *t, struct rseq_op_node __user *node)
+{
+	struct rseq_op_node __user *sentinel = &t->rseq.usrptr->rseq_op_list;
+	struct rseq_op_node __user *first;
+	u64 next, prev, first_addr;
+	u8 type, i;
+
+	if (t->rseq.nr_ops >= RSEQ_OP_LIST_LIMIT)
+		return -ENOSPC;
+
+	if (!IS_ALIGNED((unsigned long)node, __alignof__(struct rseq_op_node)))
+		return -EINVAL;
+	if (!access_ok(node, sizeof(*node)))
+		return -EFAULT;
+
+	/*
+	 * The node links are owned by the kernel. User space must present a
+	 * pristine node: next, prev and the reserved bytes all zeroed, and a
+	 * known operation type.
+	 */
+	if (get_user(next, &node->next) || get_user(prev, &node->prev) ||
+	    get_user(type, &node->type))
+		return -EFAULT;
+	if (next || prev)
+		return -EINVAL;
+	if (type >= RSEQ_OP_NR)
+		return -EINVAL;
+	for (i = 1; i < sizeof(node->reserved); i++) {
+		u8 r;
+
+		if (get_user(r, &node->reserved[i]))
+			return -EFAULT;
+		if (r)
+			return -EINVAL;
+	}
+
+	/* Splice the node in right after the sentinel. */
+	if (get_user(first_addr, &sentinel->next))
+		goto die;
+	first = (struct rseq_op_node __user *)first_addr;
+
+	if (put_user((u64)(unsigned long)first, &node->next) ||
+	    put_user((u64)(unsigned long)sentinel, &node->prev) ||
+	    put_user((u64)(unsigned long)node, &first->prev) ||
+	    put_user((u64)(unsigned long)node, &sentinel->next))
+		goto die;
+
+	t->rseq.nr_ops += 1;
+
+	if (t->rseq.nr_ops == 1)
+		return rseq_op_update_enabled(t, true) ? -EFAULT : 0;
+	return 0;
+die:
+	force_sig(SIGSEGV);
+	return -EFAULT;
+}
+
+/*
+ * Unregister @node from the operation list. The node links are validated
+ * against its neighbours to reject bogus or double unregistration.
+ */
+static int rseq_op_unregister(struct task_struct *t, struct rseq_op_node __user *node)
+{
+	struct rseq_op_node __user *prev, *next;
+	u64 prev_addr, next_addr, tmp;
+
+	if (!t->rseq.nr_ops)
+		return -ENOENT;
+
+	if (!IS_ALIGNED((unsigned long)node, __alignof__(struct rseq_op_node)))
+		return -EINVAL;
+	if (!access_ok(node, sizeof(*node)))
+		return -EFAULT;
+
+	if (get_user(next_addr, &node->next) || get_user(prev_addr, &node->prev))
+		return -EFAULT;
+	prev = (struct rseq_op_node __user *)prev_addr;
+	next = (struct rseq_op_node __user *)next_addr;
+
+	/* A registered node always has both links set. */
+	if (!prev || !next)
+		return -EINVAL;
+
+	/* Verify the node is properly linked between its neighbours. */
+	if (get_user(tmp, &prev->next))
+		goto die;
+	if (tmp != (u64)(unsigned long)node)
+		return -EINVAL;
+	if (get_user(tmp, &next->prev))
+		goto die;
+	if (tmp != (u64)(unsigned long)node)
+		return -EINVAL;
+
+	/* Unsplice and clear the node links so it can be reused. */
+	if (put_user(next_addr, &prev->next) ||
+	    put_user(prev_addr, &next->prev) ||
+	    put_user(0ULL, &node->next) ||
+	    put_user(0ULL, &node->prev))
+		goto die;
+
+	t->rseq.nr_ops -= 1;
+
+	if (t->rseq.nr_ops == 0)
+		return rseq_op_update_enabled(t, false) ? -EFAULT : 0;
+	return 0;
+die:
+	force_sig(SIGSEGV);
+	return -EFAULT;
+}
+
+int rseq_op_prctl(unsigned long arg2, unsigned long arg3)
+{
+	struct rseq_op_node __user *node = (struct rseq_op_node __user *)arg3;
+	struct task_struct *t = current;
+
+	if (!t->rseq.usrptr)
+		return -ENXIO;
+	if (!rseq_v2(t))
+		return -ENOTSUPP;
+	if (!node)
+		return -EINVAL;
+
+	switch (arg2) {
+	case PR_RSEQ_OP_REGISTER:
+		return rseq_op_register(t, node);
+	case PR_RSEQ_OP_UNREGISTER:
+		return rseq_op_unregister(t, node);
+	default:
+		return -EINVAL;
+	}
+}
kernel/sys.c +5 -0
index df69bd71de03..494c91b03c3f 100644
@@ -2889,6 +2889,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 			return -EINVAL;
 		error = rseq_slice_extension_prctl(arg2, arg3);
 		break;
+	case PR_RSEQ_OP:
+		if (arg4 || arg5)
+			return -EINVAL;
+		error = rseq_op_prctl(arg2, arg3);
+		break;
 	case PR_GET_CFI:
 		if (arg2 != PR_CFI_BRANCH_LANDING_PADS)
 			return -EINVAL;
Signature · 1 line
2.54.0
#13 lore raw

Re: [RFC PATCH 4/5] rseq: register and unregister operations via prctl

Quoted 91 lines · On Fri, 28 Aug 2026 at 17:34, <[email protected]> wrote:

From: Olivier Dion <[email protected]>

Implement the PR_RSEQ_OP prctl. Registration splices a user-provided pristine node into the head of the circular operation list, and unregistration unsplices it after validating it is properly linked between its neighbours; the kernel owns the node's next/prev links for the lifetime of the registration. The 0<->1 nr_ops transition reflects the enabled state into rseq_event::rseq_op and the user visible flag via rseq_op_update_enabled().

Initialize the operation list sentinel as an empty self-pointing circular list on rseq registration, and reset nr_ops / rseq_op state there.

Apply pending operations on the slow path and on signal delivery, and factor the common "clear error and force SIGSEGV" fixup into force_fault().

---
 kernel/rseq.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++---
 kernel/sys.c  |   5 ++
 2 files changed, 212 insertions(+), 13 deletions(-)
kernel/rseq.c +23 -6
index e75e3a5e312c..a277c11dae99 100644
@@ -253,17 +253,18 @@ static bool rseq_handle_cs(struct task_struct *t, struct pt_regs *regs)
 static void rseq_slowpath_update_usr(struct pt_regs *regs)
 {
        /*
-        * Preserve has_rseq and user_irq state. The generic entry code clears
-        * user_irq on the way out, the non-generic entry architectures are not
-        * setting user_irq.
+        * Preserve has_rseq, rseq_op and user_irq state. The generic entry
+        * code clears user_irq on the way out, the non-generic entry
+        * architectures are not setting user_irq.
         */
        const struct rseq_event evt_mask = {
                .has_rseq       = RSEQ_HAS_RSEQ_VERSION_MASK,
+               .rseq_op        = true,
                .user_irq       = true,
        };
        struct task_struct *t = current;
        struct rseq_ids ids;
-       bool event;
+       bool event, should_fault = false;
 
        if (unlikely(t->flags & PF_EXITING))
                return;
@@ -300,7 +301,12 @@ static void rseq_slowpath_update_usr(struct pt_regs *regs)
 
        ids.node_id = cpu_to_node(ids.cpu_id);
 
-       if (unlikely(!rseq_update_usr(t, regs, &ids))) {
+       if (unlikely(!rseq_update_usr(t, regs, &ids)))
+               should_fault = true;
+       else if (t->rseq.event.rseq_op && unlikely(!rseq_apply_ops(t)))
+               should_fault = true;
+
+       if (should_fault) {
                /*
                 * Clear the errors just in case this might survive magically, but
                 * leave the rest intact.
@@ -329,8 +335,20 @@ void __rseq_handle_slowpath(struct pt_regs *regs)
        rseq_slowpath_update_usr(regs);
 }
 
+static inline void force_fault(int sig)
+{
+       /*
+        * Clear the errors just in case this might survive magically, but leave
+        * the rest intact.
+        */
+       current->rseq.event.error = 0;
+       force_sigsegv(sig);
+}
+
 void __rseq_signal_deliver(int sig, struct pt_regs *regs)
 {
+       bool should_fault = false;
+
        rseq_stat_inc(rseq_stats.signal);
 
        /*
@@ -339,14 +357,13 @@ void __rseq_signal_deliver(int sig, struct pt_regs *regs)
         * the interrupted context as after this point the instruction
         * pointer in @regs points to the signal handler.
         */
-       if (unlikely(!rseq_handle_cs(current, regs))) {

kernel/rseq.c void __rseq_signal_deliver(int sig, struct pt_regs *regs) · line 362

Quoted 10 lines
kernel/rseq.c +3 -7
-               /*
-                * Clear the errors just in case this might survive
-                * magically, but leave the rest intact.
-                */
-               current->rseq.event.error = 0;
-               force_sigsegv(sig);
-       }
+       if (unlikely(!rseq_handle_cs(current, regs)))
+               should_fault = true;
+       else if (current->rseq.event.rseq_op && unlikely(!rseq_apply_ops(current)))

Just to make sure: membarrier delivered to a task will also trigger rseq_apply_ops, right?

Quoted 20 lines
kernel/rseq.c +6 -0
+               should_fault = true;
+
+       if (should_fault)
+               force_fault(sig);
 
        /*
         * In legacy mode, force the update of IDs before returning to user
@@ -436,6 +453,8 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3
                                rseqfl |= RSEQ_CS_FLAG_SLICE_EXT_ENABLED;
                }
        }
+       if (version > 1)
+               rseqfl |= RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE;
 
        scoped_user_write_access(rseq, efault) {
                /*
@@ -458,8 +477,16 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3
                 * registrations.
                 */
                if (version > 1) {

kernel/rseq.c static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3 · line 489

Quoted 10 lines
kernel/rseq.c +8 -0
+                       u64 sentinel = (u64)&rseq->rseq_op_list;
+
                        if (IS_ENABLED(CONFIG_RSEQ_SLICE_EXTENSION))
                                unsafe_put_user(0U, &rseq->slice_ctrl.all, efault);
+                       /*
+                        * Initialize the rseq operation list sentinel as an
+                        * empty circular doubly-linked list pointing to itself.
+                        */
+                       unsafe_put_user(sentinel, &rseq->rseq_op_list.next, efault);
+                       unsafe_put_user(sentinel, &rseq->rseq_op_list.prev, efault);

Is there always enough space to write these? Can this corrupt program memory?

Quoted 200 lines
kernel/rseq.c +167 -0
                }
        }
 
@@ -474,6 +501,12 @@ static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u3
 #ifdef CONFIG_RSEQ_SLICE_EXTENSION
        current->rseq.slice.state.enabled = !!(rseqfl & RSEQ_CS_FLAG_SLICE_EXT_ENABLED);
 #endif
+       /*
+        * A fresh registration starts with no operations, so operation
+        * processing is disabled until the first one is registered.
+        */
+       current->rseq.event.rseq_op = false;
+       current->rseq.nr_ops = 0;
 
        /*
         * Ensure the cpu_id_start and cpu_id fields are updated before
@@ -887,3 +920,164 @@ device_initcall(rseq_slice_init);
 #else
 static void rseq_slice_ext_init(struct dentry *root_dir) { }
 #endif /* CONFIG_RSEQ_SLICE_EXTENSION */
+
+/*
+ * Reflect the operation enabled state into the user visible flags field.
+ * Called on the 0<->1 transition of nr_ops. The kill path is taken on fault
+ * because losing this update leaves user space and kernel state inconsistent.
+ */
+static int rseq_op_update_enabled(struct task_struct *t, bool enable)
+{
+       struct rseq __user *rseq = t->rseq.usrptr;
+       u32 rflags;
+
+       if (get_user(rflags, &rseq->flags))
+               return -EFAULT;
+
+       rflags &= ~RSEQ_CS_FLAG_RSEQ_OP_ENABLED;
+       rflags |= RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE;
+       if (enable)
+               rflags |= RSEQ_CS_FLAG_RSEQ_OP_ENABLED;
+
+       if (put_user(rflags, &rseq->flags))
+               return -EFAULT;
+
+       t->rseq.event.rseq_op = enable;
+       return 0;
+}
+
+/*
+ * Register @node at the head of the circular doubly-linked operation list
+ * anchored by the kernel owned sentinel in struct rseq.
+ */
+static int rseq_op_register(struct task_struct *t, struct rseq_op_node __user *node)
+{
+       struct rseq_op_node __user *sentinel = &t->rseq.usrptr->rseq_op_list;
+       struct rseq_op_node __user *first;
+       u64 next, prev, first_addr;
+       u8 type, i;
+
+       if (t->rseq.nr_ops >= RSEQ_OP_LIST_LIMIT)
+               return -ENOSPC;
+
+       if (!IS_ALIGNED((unsigned long)node, __alignof__(struct rseq_op_node)))
+               return -EINVAL;
+       if (!access_ok(node, sizeof(*node)))
+               return -EFAULT;
+
+       /*
+        * The node links are owned by the kernel. User space must present a
+        * pristine node: next, prev and the reserved bytes all zeroed, and a
+        * known operation type.
+        */
+       if (get_user(next, &node->next) || get_user(prev, &node->prev) ||
+           get_user(type, &node->type))
+               return -EFAULT;
+       if (next || prev)
+               return -EINVAL;
+       if (type >= RSEQ_OP_NR)
+               return -EINVAL;
+       for (i = 1; i < sizeof(node->reserved); i++) {
+               u8 r;
+
+               if (get_user(r, &node->reserved[i]))
+                       return -EFAULT;
+               if (r)
+                       return -EINVAL;
+       }
+
+       /* Splice the node in right after the sentinel. */
+       if (get_user(first_addr, &sentinel->next))
+               goto die;
+       first = (struct rseq_op_node __user *)first_addr;
+
+       if (put_user((u64)(unsigned long)first, &node->next) ||
+           put_user((u64)(unsigned long)sentinel, &node->prev) ||
+           put_user((u64)(unsigned long)node, &first->prev) ||
+           put_user((u64)(unsigned long)node, &sentinel->next))
+               goto die;
+
+       t->rseq.nr_ops += 1;
+
+       if (t->rseq.nr_ops == 1)
+               return rseq_op_update_enabled(t, true) ? -EFAULT : 0;
+       return 0;
+die:
+       force_sig(SIGSEGV);
+       return -EFAULT;
+}
+
+/*
+ * Unregister @node from the operation list. The node links are validated
+ * against its neighbours to reject bogus or double unregistration.
+ */
+static int rseq_op_unregister(struct task_struct *t, struct rseq_op_node __user *node)
+{
+       struct rseq_op_node __user *prev, *next;
+       u64 prev_addr, next_addr, tmp;
+
+       if (!t->rseq.nr_ops)
+               return -ENOENT;
+
+       if (!IS_ALIGNED((unsigned long)node, __alignof__(struct rseq_op_node)))
+               return -EINVAL;
+       if (!access_ok(node, sizeof(*node)))
+               return -EFAULT;
+
+       if (get_user(next_addr, &node->next) || get_user(prev_addr, &node->prev))
+               return -EFAULT;
+       prev = (struct rseq_op_node __user *)prev_addr;
+       next = (struct rseq_op_node __user *)next_addr;
+
+       /* A registered node always has both links set. */
+       if (!prev || !next)
+               return -EINVAL;
+
+       /* Verify the node is properly linked between its neighbours. */
+       if (get_user(tmp, &prev->next))
+               goto die;
+       if (tmp != (u64)(unsigned long)node)
+               return -EINVAL;
+       if (get_user(tmp, &next->prev))
+               goto die;
+       if (tmp != (u64)(unsigned long)node)
+               return -EINVAL;
+
+       /* Unsplice and clear the node links so it can be reused. */
+       if (put_user(next_addr, &prev->next) ||
+           put_user(prev_addr, &next->prev) ||
+           put_user(0ULL, &node->next) ||
+           put_user(0ULL, &node->prev))
+               goto die;
+
+       t->rseq.nr_ops -= 1;
+
+       if (t->rseq.nr_ops == 0)
+               return rseq_op_update_enabled(t, false) ? -EFAULT : 0;
+       return 0;
+die:
+       force_sig(SIGSEGV);
+       return -EFAULT;
+}
+
+int rseq_op_prctl(unsigned long arg2, unsigned long arg3)
+{
+       struct rseq_op_node __user *node = (struct rseq_op_node __user *)arg3;
+       struct task_struct *t = current;
+
+       if (!t->rseq.usrptr)
+               return -ENXIO;
+       if (!rseq_v2(t))
+               return -ENOTSUPP;
+       if (!node)
+               return -EINVAL;
+
+       switch (arg2) {
+       case PR_RSEQ_OP_REGISTER:
+               return rseq_op_register(t, node);
+       case PR_RSEQ_OP_UNREGISTER:
+               return rseq_op_unregister(t, node);
+       default:
+               return -EINVAL;
+       }
+}
kernel/sys.c +5 -1
index df69bd71de03..494c91b03c3f 100644
@@ -2889,6 +2889,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
                        return -EINVAL;
                error = rseq_slice_extension_prctl(arg2, arg3);
                break;
+       case PR_RSEQ_OP:
+               if (arg4 || arg5)
+                       return -EINVAL;
+               error = rseq_op_prctl(arg2, arg3);
+               break;
        case PR_GET_CFI:
                if (arg2 != PR_CFI_BRANCH_LANDING_PADS)
                        return -EINVAL;
--

2.54.0

#14 lore raw

RFC PATCH 5/5 selftests/rseq: add coverage for rseq operations

In reply to [email protected]

From: Olivier Dion <[email protected]>

Add selftest-side support for the rseq operation ABI and wire a new rseq_op_test into the rseq selftest suite.

The new test covers:
- reset before entering a signal handler,
- reset on return to userspace after poll(),
- reset after scheduler-driven preemption,
- no-operation cases where nothing should be reset,
- register-time rejection of non-pristine nodes and unknown types,
- fatal cases for corrupted list links (self and transitive cycles),
- invalid destination and next pointers,
- unsupported lengths,
- unaligned destinations, and
- NULL-source clearing.

Also extend the selftest ABI/helpers with the rseq operation node definitions and prctl-based register/unregister helpers, and add a dedicated runner that disables glibc's implicit rseq registration.

---
 tools/testing/selftests/rseq/.gitignore |   4 +-
 tools/testing/selftests/rseq/Makefile   |   4 +-
 tools/testing/selftests/rseq/rseq-abi.h | 146 ++++++++++++++++++++----
 tools/testing/selftests/rseq/rseq.c     |  23 +++-
 tools/testing/selftests/rseq/rseq.h     |  75 ++++++++++++
 5 files changed, 223 insertions(+), 29 deletions(-)
tools/testing/selftests/rseq/.gitignore +3 -1
index ec01d164c1f0..8d5f02d79b98 100644
@@ -2,7 +2,6 @@
 basic_percpu_ops_test
 basic_percpu_ops_mm_cid_test
 basic_test
-basic_rseq_op_test
 param_test
 param_test_benchmark
 param_test_compare_twice
@@ -11,3 +10,6 @@ param_test_mm_cid_benchmark
 param_test_mm_cid_compare_twice
 syscall_errors_test
 slice_test
+check_optimized
+legacy_check
+rseq_op_test
\ No newline at end of file
tools/testing/selftests/rseq/Makefile +3 -1
index 50d69e22ee7a..5e761df0e5aa 100644
@@ -22,12 +22,14 @@ TEST_GEN_PROGS_EXTENDED = librseq.so \
 	param_test_compare_twice \
 	param_test_mm_cid \
 	param_test_mm_cid_compare_twice \
+	rseq_op_test \
 	syscall_errors_test \
 	legacy_check \
 	slice_test \
 	check_optimized
 
-TEST_PROGS = run_param_test.sh run_syscall_errors_test.sh run_legacy_check.sh run_timeslice_test.sh
+TEST_PROGS = run_param_test.sh run_syscall_errors_test.sh run_legacy_check.sh run_timeslice_test.sh \
+	run_rseq_op_test.sh
 
 TEST_FILES := settings
 
tools/testing/selftests/rseq/rseq-abi.h +124 -22

177 lines not on this page · raw message

tools/testing/selftests/rseq/rseq.c +18 -5
index be0d0a97031e..6a5406f3353b 100644
@@ -116,6 +116,17 @@ bool rseq_available(void)
 	}
 }
 
+/* The rseq areas need to be at least 32 bytes. */
+static
+unsigned int get_rseq_min_alloc_size(void)
+{
+	unsigned int alloc_size = rseq_size;
+
+	if ((int) alloc_size < ORIG_RSEQ_ALLOC_SIZE)
+		alloc_size = ORIG_RSEQ_ALLOC_SIZE;
+	return alloc_size;
+}
+
 /*
  * Return the feature size supported by the kernel.
  *
@@ -261,12 +272,14 @@ void rseq_init(void)
 
 	/* rseq flags are deprecated, always set to 0. */
 	rseq_flags = 0;
+	{
+		unsigned int rseq_kernel_feature_size = get_rseq_kernel_feature_size();
 
-	/*
-	 * Set the size to 0 until at least one thread registers to mimic the
-	 * libc behavior.
-	 */
-	rseq_size = 0;
+		if (rseq_kernel_feature_size <= RSEQ_THREAD_AREA_ALLOC_SIZE)
+			rseq_size = rseq_kernel_feature_size;
+		else
+			rseq_size = ORIG_RSEQ_ALLOC_SIZE;
+	}
 }
 
 static __attribute__((destructor))
tools/testing/selftests/rseq/rseq.h +75 -0

85 lines not on this page · raw message

Signature · 1 line
2.54.0
#15 lore raw
In reply to [email protected]

Olivier!

Quoted 9 lines · On Fri, Aug 28 2026 at 11:33, [email protected] wrote:

This series introduces RSEQ operations: a per-thread list of operations the kernel applies, on behalf of a ask, when returning to user space.

The main motivation is to help TCMalloc migrate from RSEQ v1 to RSEQ v2 and use the RSEQ area registered by glibc. TCMalloc currently relies on RSEQ v1 resetting the cpu_id field to invalidate a per-CPU pointer cached in TLS. This requires applications to disable glibc's implicit RSEQ registration and prevents sharing the glibc RSEQ area.

Quoted 5 lines

RSEQ operations provide an opt-in replacement for that invalidation mechanism. User space registers operation nodes with prctl; the kernel keeps the nodes in a per-thread circular list and applies them on return to user space. Only tasks with registered operations incur the additional RSEQ exit-path work.

TBH, to me this sounds like a horrible idea. It's yet another "interpreter" for a very limited use case and a lot of complexity.

I think this can be done in user space with some help of the kernel of course. If user space has registered an entry point for this functionality then the kernel can emulate a call to that on return to user space:

      if (tsk->rseq.needs_fixup) {
                // modifies regs->sp
		create_callframe_on_user stack(regs);
                regs->ip = tsk->rseq.fixup_ip;
      }
      ret_to_user()

The user space fixup does:

fixup_ip()

      ....
      user_rseq.fixup_in_progress = false;
      restore_and_return() // Returns to the original IP

A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.

Of course the above is way too simple to be true :) It works except when it nests. But that's solvable too:

      if (tsk->rseq.needs_fixup) {
                // modifies regs->sp
		create_callframe_on_user stack(regs);
                regs->ip = tsk->rseq.fixup_ip;
                // Save the callframe SP
                tsk->rseq.fixup_sp = regs->sp;
      }

When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:

        regs->ip = tsk->rseq.fixup_abort_ip;
        regs->sp = tsk->rseq.fixup_sp;
        user_rseq.fixup_in_progress = false;
        tsk->rseq.needs_fixup = true;

Which rewinds the stack to the callframe and makes the interrupted fixup continue at the fixup_abort_ip which just restores registers from the callframe and returns to the original IP.

Then the flow becomes in case of a trivial interrupt which just returns:

    fixup_ip()
      -> interrupt
                                ...
                                if (user_rseq.fixup_in_progress)
                                   rewind_fixup(regs);
                                 ...
                                if (needs_fixup)
                                   setup_fixup(regs);
                                return to user;
       fixup_ip()
          ...
          user_rseq.fixup_in_progress = false
          restore_and_return()

fixup_abort_ip:
restore_and_return() // Returns to orig_ip

In case of a signal delivery:

    fixup_ip()
      -> interrupt
                                ...
                                if (user_rseq.fixup_in_progress)
                                   rewind_fixup(regs);

                                   deliver_signal()
                                     setup_sigframe()
                                  ...
                                if (needs_fixup)
                                   setup_fixup(regs);
                                return to user;
       fixup_ip()
          ...
          user_rseq.fixup_in_progress = false
          restore_and_return()
       signal_handler()
          sys_sigreturn()
                                restore(regs)
                                return to user;

fixup_abort_ip:
restore_and_return() // Returns to orig_ip

There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)

I've played around with something similar pre RSEQ, but all I could still find are the notes I took back then, which I duly transcribed into todays RSEQ world. The patches themself would be utterly useless anyway as all parts which need to be touched have been through the mincer at least once.

The fixup address and the fixup abort address should probably not be stored in user rseq memory. For an initial POC they just can be registered through sys_rseq() which also enables the fixup mechanics, i.e. the unconditional set of tsk->rseq.needs_fixup in schedule(), which should be completely independent of the rseq.sched_switch and rseq.ids_changed bits and not change the existing functionality and semantics like the current POC does. Keep it separate.

Once the POC dust has settled the registration interface should change for the final implementation because otherwise this would create the very same problem of who owns it again.

The final solution should provide a trivial dispatch mechanism in the VDSO and the RSEQ user area should be expanded to provide storage for registration. The syscall registration would change to registering a lib specific callback/data pair and the kernel would reorganize the storage. Whether that's a linked list or a size limited array in the RSEQ user area does not matter. As this is thread local there is no concurrency and the syscall can rearange that storage completely undisturbed. Libraries can manage their own thread local marker which is checked in the lib specific callback to decide whether they want to run or not.

Using a VDSO dispatcher and expanding the RSEQ storage for that should just work out of the box with any libc which supports RSEQ_V2 and would therefore not create any additional libc dependencies for other library developers.

The dispatcher entry needs to be trivial ASM for the restore/return

fixup_ip:

        // setup_fixup() in the kernel stored the TLS user RSEQ address in
        // RDI or whatever an architecture uses for the first argument
	CALL	fixup_c

fixup_ip_abort:
ASM_RESTORE_REGS
RET

The C part should be trivial too:

fixup_c(struct rseq *rseq)
{
for_each_fixup(fn, data, rseq)
fn(data);
}

or something along these lines.

Thoughts?

Thanks,

tglx

#16 lore raw
Quoted 12 lines · Thomas Gleixner

Olivier!

Quoted 9 lines · On Fri, Aug 28 2026 at 11:33, [email protected] wrote:

This series introduces RSEQ operations: a per-thread list of operations the kernel applies, on behalf of a ask, when returning to user space.

The main motivation is to help TCMalloc migrate from RSEQ v1 to RSEQ v2 and use the RSEQ area registered by glibc. TCMalloc currently relies on RSEQ v1 resetting the cpu_id field to invalidate a per-CPU pointer cached in TLS. This requires applications to disable glibc's implicit RSEQ registration and prevents sharing the glibc RSEQ area.

Quoted 8 lines
Quoted 5 lines

RSEQ operations provide an opt-in replacement for that invalidation mechanism. User space registers operation nodes with prctl; the kernel keeps the nodes in a per-thread circular list and applies them on return to user space. Only tasks with registered operations incur the additional RSEQ exit-path work.

TBH, to me this sounds like a horrible idea. It's yet another "interpreter" for a very limited use case and a lot of complexity.

I post this RFC before going on vacation. I tend to very much agree with you reply overall, after revisiting the problem and potential issues this solution could introduce.

Quoted 12 lines

I think this can be done in user space with some help of the kernel of course. If user space has registered an entry point for this functionality then the kernel can emulate a call to that on return to user space:

      if (tsk->rseq.needs_fixup) {
                // modifies regs->sp
		create_callframe_on_user stack(regs);
                regs->ip = tsk->rseq.fixup_ip;
      }
      ret_to_user()
Quoted 6 lines

The user space fixup does:

fixup_ip()

      ....
      user_rseq.fixup_in_progress = false;
      restore_and_return() // Returns to the original IP

Yes. This is what I had in mind (see reply to Florian).

Quoted 3 lines

A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.

I suppose that no floating point also mean no xsave performance trashing, which is obviously something we want to avoid. Is that what you mean by sigframe disaster? Also, I am not sure how we can enforce this no syscall/floating point policies other than asking users to be good citizen.

Quoted 14 lines

Of course the above is way too simple to be true :) It works except when it nests. But that's solvable too:

      if (tsk->rseq.needs_fixup) {
                // modifies regs->sp
		create_callframe_on_user stack(regs);
                regs->ip = tsk->rseq.fixup_ip;
                // Save the callframe SP
                tsk->rseq.fixup_sp = regs->sp;
      }

When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:

Quoted 8 lines
        regs->ip = tsk->rseq.fixup_abort_ip;
        regs->sp = tsk->rseq.fixup_sp;
        user_rseq.fixup_in_progress = false;
        tsk->rseq.needs_fixup = true;

Which rewinds the stack to the callframe and makes the interrupted fixup continue at the fixup_abort_ip which just restores registers from the callframe and returns to the original IP.

So the fixup handler would have an abort label, akin to RSEQ region. But then there will be no guarantee that the fixup operations succeed? Or did I completely misunderstood?

[...]

Quoted 3 lines

There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)

I could certainly make a POC out of this for x86. It could handle red-zone and shadow-stack to start with. I don't know if there are other architecture-specific quirks that need to be aware of, given I am not familiar enough with architectures outside of x86.

[...]

Quoted 3 lines

Once the POC dust has settled the registration interface should change for the final implementation because otherwise this would create the very same problem of who owns it again.

Right we don't want to play in that movie again :-)

Quoted 28 lines

The final solution should provide a trivial dispatch mechanism in the VDSO and the RSEQ user area should be expanded to provide storage for registration. The syscall registration would change to registering a lib specific callback/data pair and the kernel would reorganize the storage. Whether that's a linked list or a size limited array in the RSEQ user area does not matter. As this is thread local there is no concurrency and the syscall can rearange that storage completely undisturbed. Libraries can manage their own thread local marker which is checked in the lib specific callback to decide whether they want to run or not.

Using a VDSO dispatcher and expanding the RSEQ storage for that should just work out of the box with any libc which supports RSEQ_V2 and would therefore not create any additional libc dependencies for other library developers.

The dispatcher entry needs to be trivial ASM for the restore/return

fixup_ip:

        // setup_fixup() in the kernel stored the TLS user RSEQ address in
        // RDI or whatever an architecture uses for the first argument
	CALL	fixup_c

fixup_ip_abort:
ASM_RESTORE_REGS
RET

The C part should be trivial too:

Quoted 7 lines

fixup_c(struct rseq *rseq)
{
for_each_fixup(fn, data, rseq)
fn(data);
}

or something along these lines.

Thanks for the insight. I think this will certainly help me in making the POC.

One aspect that I have not think of for now is allowing fast registration/unregistration. This RFC uses a syscall for registration. My original intent for this was to do a edge trigger detection of the first/last registration/unregistration in the syscall and set/unset a flag into the RSEQ state of the thread task. That way, we don't need to touch another cache line to know if the thread registered something. This is obviously incompatible with short-live registrations. However, I did not come with a case for it yet, so perhaps I am over-thinking this.

Thanks,
Olivier

Signature · 3 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
#17 lore raw
Quoted 12 lines · Olivier Dion
Quoted 4 lines · Thomas Gleixner

fixup_ip()

      ....
      user_rseq.fixup_in_progress = false;
      restore_and_return() // Returns to the original IP

Yes. This is what I had in mind (see reply to Florian).

Quoted 3 lines

A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.

Quoted 5 lines

I suppose that no floating point also mean no xsave performance trashing, which is obviously something we want to avoid. Is that what you mean by sigframe disaster? Also, I am not sure how we can enforce this no syscall/floating point policies other than asking users to be good citizen.

The sigframe disaster is that sigaltstack has been insufficient for the ever growing XSTATE to save (its even uncompressed XSTATE). Also these fixups really should do a few selective stores and not huge computations or wipe out a GB of memory. So avoiding the XSAVE/XRSTOR overhead completely is certainly a benefit.

Quoted 8 lines
Quoted 8 lines

When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:

        regs->ip = tsk->rseq.fixup_abort_ip;
        regs->sp = tsk->rseq.fixup_sp;
        user_rseq.fixup_in_progress = false;
        tsk->rseq.needs_fixup = true;
Quoted 7 lines
Quoted 3 lines

Which rewinds the stack to the callframe and makes the interrupted fixup continue at the fixup_abort_ip which just restores registers from the callframe and returns to the original IP.

So the fixup handler would have an abort label, akin to RSEQ region. But then there will be no guarantee that the fixup operations succeed? Or did I completely misunderstood?

They are only aborted when there is nesting, so the nesting context starts over and redoes them. Once the nesting context returns to the first fixup (abort IP) the state is correct and then the first fixup returns to the original return IP.

That abort on nesting avoids the following issue:

user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user

fixup_ip:
fixup_c(data)
...
interrupt(#2)
schedule()
MMCID changes
fixup requested

fixup_ip:
fixup_c(data)
data::mmcid = user_rseq::mmcid
restore_and_return()

data::mmcid = user_rseq::mmcid
restore_and_return()

If the interrupt #2 hits between the load of user_rseq::mmcid and the store to data::mmcid of the first fixup, then the return to the first fixup context would obviously write the wrong ID back.

And you can't skip the fixup on return from interrupt #2 and just return to the already running one in that case either.

That would be possible if the fixups are truly idempotent, e.g. can only zero out memory.

The trivial nest case would be:

user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user

fixup_ip:
fixup_c(data)
start zeroing

interrupt(#2)
schedule()
MMCID changes
fixup requested

observes fixup running
return to user

continue zeroing
restore_and_return()

But that does not work with signals because the signal delivery does not return to the interrupted IP. It creates the sigframe and returns to the signal handler, which then goes back with sys_rt_sigreturn(). Then the kernel restores the interrupted context. So in that case you'd need:

user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
// Must be a counter
user_rseq->fixup_running++;
return to user

fixup_ip:

    fixup_c(data)
    start zeroing

    interrupt(#2)
      signal_delivery()
      setup_sigframe()
      observes fixup running
      setup_callframe(....)
      user_rseq->fixup_running++;
      return to user

fixup_ip:

    fixup_c(data)
    zero everything
    user_rseq->fixup_running--;
      restore_and_return()

signal_handler()
sys_rt_sigreturn()
restore_regs()
observes fixup running
return to user

continue zeroing
user_rseq->fixup_running--;
restore_and_return()

Idempotent fixup functions restrict obviously what can be done there. But if that restriction is fine, then this approach works too.

I have no strong opinion either way.

Quoted 6 lines
Quoted 3 lines

There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)

I could certainly make a POC out of this for x86. It could handle red-zone and shadow-stack to start with. I don't know if there are

I wouldn't even bother with shadow-stacks for a POC. The red-zone skip when setting up the callframe, i.e. SP - 128 is unavoidable, but that's it.

Quoted 2 lines

other architecture-specific quirks that need to be aware of, given I am not familiar enough with architectures outside of x86.

Well every architecture has some quirks but I'm not aware of one which would fundamentally stand in the way. The main difference is going to be how the callframe is set up and how the fixup function needs to look. That's always architecture specific and there are wizards for each architecture to help with that :)

Quoted 9 lines

One aspect that I have not think of for now is allowing fast registration/unregistration. This RFC uses a syscall for registration. My original intent for this was to do a edge trigger detection of the first/last registration/unregistration in the syscall and set/unset a flag into the RSEQ state of the thread task. That way, we don't need to touch another cache line to know if the thread registered something. This is obviously incompatible with short-live registrations. However, I did not come with a case for it yet, so perhaps I am over-thinking this.

TCMalloc wont have a short-lived registration, neither wont a tracer or something like that. Once a library is initialized it won't go away just because. So if the facility needs to pause the callback intermittently then this can simply be:

cb(data)

        if (!data->run)
        	return;

That won't be the end of the world if those pauses are not taking forever. If they do then the syscall is justified.

Let's get the basic principles working first on a KISS basis and then think about how to keep it as simple as possible.

So for the POC you neither need multi-lib support nor the VDSO bits. Just hack up the ASM fixup and a trivial C demonstrator in user space and register fixup and abort address with a hacked up sys_rseq().

Once that works, the extra bells and whistels are not hard to add. They are hard to get right, but without the prove of concept wasting time on them is pretty pointless. :)

Thanks,

tglx

---
Everything should be made as simple as possible, but not simpler.

- attributed to Albert Einstein

#18 lore raw
Quoted 207 lines · Thomas Gleixner
Quoted 12 lines · Olivier Dion
Quoted 4 lines · Thomas Gleixner

fixup_ip()

      ....
      user_rseq.fixup_in_progress = false;
      restore_and_return() // Returns to the original IP

Yes. This is what I had in mind (see reply to Florian).

Quoted 3 lines

A reasonable limitation for the fixup function should be a strict "no syscalls and no floating point within the fixup" rule. No floating point avoids the whole sigframe disaster.

Quoted 5 lines

I suppose that no floating point also mean no xsave performance trashing, which is obviously something we want to avoid. Is that what you mean by sigframe disaster? Also, I am not sure how we can enforce this no syscall/floating point policies other than asking users to be good citizen.

The sigframe disaster is that sigaltstack has been insufficient for the ever growing XSTATE to save (its even uncompressed XSTATE). Also these fixups really should do a few selective stores and not huge computations or wipe out a GB of memory. So avoiding the XSAVE/XRSTOR overhead completely is certainly a benefit.

Quoted 8 lines
Quoted 8 lines

When return to user observes user_rseq.fixup_in_progress then it can mangle regs before doing anything else:

        regs->ip = tsk->rseq.fixup_abort_ip;
        regs->sp = tsk->rseq.fixup_sp;
        user_rseq.fixup_in_progress = false;
        tsk->rseq.needs_fixup = true;
Quoted 7 lines
Quoted 3 lines

Which rewinds the stack to the callframe and makes the interrupted fixup continue at the fixup_abort_ip which just restores registers from the callframe and returns to the original IP.

So the fixup handler would have an abort label, akin to RSEQ region. But then there will be no guarantee that the fixup operations succeed? Or did I completely misunderstood?

They are only aborted when there is nesting, so the nesting context starts over and redoes them. Once the nesting context returns to the first fixup (abort IP) the state is correct and then the first fixup returns to the original return IP.

That abort on nesting avoids the following issue:

user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user

fixup_ip:
fixup_c(data)
...
interrupt(#2)
schedule()
MMCID changes
fixup requested

fixup_ip:
fixup_c(data)
data::mmcid = user_rseq::mmcid
restore_and_return()

data::mmcid = user_rseq::mmcid
restore_and_return()

If the interrupt #2 hits between the load of user_rseq::mmcid and the store to data::mmcid of the first fixup, then the return to the first fixup context would obviously write the wrong ID back.

And you can't skip the fixup on return from interrupt #2 and just return to the already running one in that case either.

That would be possible if the fixups are truly idempotent, e.g. can only zero out memory.

The trivial nest case would be:

user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
return to user

fixup_ip:
fixup_c(data)
start zeroing

interrupt(#2)
schedule()
MMCID changes
fixup requested

observes fixup running
return to user

continue zeroing
restore_and_return()

But that does not work with signals because the signal delivery does not return to the interrupted IP. It creates the sigframe and returns to the signal handler, which then goes back with sys_rt_sigreturn(). Then the kernel restores the interrupted context. So in that case you'd need:

user_function()
...
interrupt(#1)
schedule()
MMCID changes
fixup requested
...
setup_callframe(....)
// Must be a counter
user_rseq->fixup_running++;
return to user

fixup_ip:

    fixup_c(data)
    start zeroing

    interrupt(#2)
      signal_delivery()
      setup_sigframe()
      observes fixup running
      setup_callframe(....)
      user_rseq->fixup_running++;
      return to user

fixup_ip:

    fixup_c(data)
    zero everything
    user_rseq->fixup_running--;
      restore_and_return()

signal_handler()
sys_rt_sigreturn()
restore_regs()
observes fixup running
return to user

continue zeroing
user_rseq->fixup_running--;
restore_and_return()

Idempotent fixup functions restrict obviously what can be done there. But if that restriction is fine, then this approach works too.

I have no strong opinion either way.

Quoted 6 lines
Quoted 3 lines

There are obviously a ton of details to take care of (/me mumbles shadow stacks and RSEQ CS interaction), but the general principle should just work. Emphasis on should and I'm so NOT going to hack that up. :)

I could certainly make a POC out of this for x86. It could handle red-zone and shadow-stack to start with. I don't know if there are

I wouldn't even bother with shadow-stacks for a POC. The red-zone skip when setting up the callframe, i.e. SP - 128 is unavoidable, but that's it.

Quoted 2 lines

other architecture-specific quirks that need to be aware of, given I am not familiar enough with architectures outside of x86.

Well every architecture has some quirks but I'm not aware of one which would fundamentally stand in the way. The main difference is going to be how the callframe is set up and how the fixup function needs to look. That's always architecture specific and there are wizards for each architecture to help with that :)

Quoted 9 lines

One aspect that I have not think of for now is allowing fast registration/unregistration. This RFC uses a syscall for registration. My original intent for this was to do a edge trigger detection of the first/last registration/unregistration in the syscall and set/unset a flag into the RSEQ state of the thread task. That way, we don't need to touch another cache line to know if the thread registered something. This is obviously incompatible with short-live registrations. However, I did not come with a case for it yet, so perhaps I am over-thinking this.

TCMalloc wont have a short-lived registration, neither wont a tracer or something like that. Once a library is initialized it won't go away just because. So if the facility needs to pause the callback intermittently then this can simply be:

cb(data)

        if (!data->run)
                return;

That won't be the end of the world if those pauses are not taking forever. If they do then the syscall is justified.

Let's get the basic principles working first on a KISS basis and then think about how to keep it as simple as possible.

Quoted 7 lines

So for the POC you neither need multi-lib support nor the VDSO bits. Just hack up the ASM fixup and a trivial C demonstrator in user space and register fixup and abort address with a hacked up sys_rseq().

Once that works, the extra bells and whistels are not hard to add. They are hard to get right, but without the prove of concept wasting time on them is pretty pointless. :)

I totally agree with the "yet another general-purpose interpreter" argument. I was thinking of some schemes where we would need to clear lots of words, and doing this with an in-kernel interpreter may be too slow; or clearing just 1 or 2 bytes; or even caching a pointer in the GS register, and resetting it with wrgsbase.

I think we even brainstormed with Mathieu something similar to resetting the PC to a custom user-space handler at Plumbers few years ago, but we got stuck on signal handlers and re-entrancy problem.