Skip to contentSkip to conversation

optimize cost of inter-process communication

RFC In review Bo Li 8 participants 46 messages May 30, 2025 – Jun 3, 2025

Subsystems (11)
lore
Share & tools
lore

Bo Li

author
#0 lore raw

v2 cover optimize cost of inter-process communication

Changelog:

v2:
- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

link to v1:
https://lore.kernel.org/lkml/CAP2HCOmAkRVTci0ObtyW=3v6GFOrt9zCn2NwLUbZ+Di49xkBiw@mail.gmail.com/

--------------------------------------------------------------------------

# Introduction

We mainly apply RPAL to the service mesh architecture widely adopted in modern cloud-native data centers. Before the rise of the service mesh architecture, network functions were usually integrated into monolithic applications as libraries, and the main business programs invoked them through function calls. However, to facilitate the independent development and operation and maintenance of the main business programs and network functions, the service mesh removed the network functions from the main business programs and made them independent processes (called sidecars). Inter-process communication (IPC) is used for interaction between the main business program and the sidecar, and the introduced inter-process communication has led to a sharp increase in resource consumption in cloud-native data centers, and may even occupy more than 10% of the CPU of the entire microservice cluster.

To achieve the efficient function call mechanism of the monolithic architecture under the service mesh architecture, we introduced the RPAL (Running Process As Library) architecture, which implements the sharing of the virtual address space of processes and the switching threads in user mode. Through the analysis of the service mesh architecture, we found that the process memory isolation between the main business program and the sidecar is not particularly important because they are split from one application and were an integral part of the original monolithic application. It is more important for the two processes to be independent of each other because they need to be independently developed and maintained to ensure the architectural advantages of the service mesh. Therefore, RPAL breaks the isolation between processes while preserving the independence between them. We think that RPAL can also be applied to other scenarios featuring sidecar-like architectures, such as distributed file storage systems in LLM infra.

In RPAL architecture, multiple processes share a virtual address space, so this architecture can be regarded as an advanced version of the Linux shared memory mechanism:

1. Traditional shared memory requires two processes to negotiate to ensure the mapping of the same piece of memory. In RPAL architecture, two RPAL processes still need to reach a consensus before they can successfully invoke the relevant system calls of RPAL to share the virtual address space.
2. Traditional shared memory only shares part of the data. However, in RPAL architecture, processes that have established an RPAL communication relationship share a virtual address space, and all user memory (such as data segments and code segments) of each RPAL process is shared among these processes. However, a process cannot access the memory of other processes at any time. We use the MPK mechanism to ensure that the memory of other processes can only be accessed when special RPAL functions are called. Otherwise, a page fault will be triggered.
3. In RPAL architecture, to ensure the consistency of the execution context of the shared code (such as the stack and thread local storage), we further implement the thread context switching in user mode based on the ability to share the virtual address space of different processes, enabling the threads of different processes to directly perform fast switching in user mode without falling into kernel mode for slow switching.

# Background

In traditional inter-process communication (IPC) scenarios, Unix domain sockets are commonly used in conjunction with the epoll() family for event multiplexing. IPC operations involve system calls on both the data and control planes, thereby imposing a non-trivial overhead on the interacting processes. Even when shared memory is employed to optimize the data plane, two data copies still remain. Specifically, data is initially copied from a process's private memory space into the shared memory area, and then it is copied from the shared memory into the private memory of another process.

This poses a question: Is it possible to reduce the overhead of IPC with only minimal modifications at the application level? To address this, we observed that the functionality of IPC, which encompasses data transfer and invocation of the target thread, is similar to a function call, where arguments are passed and the callee function is invoked to process them. Inspired by this analogy, we introduce RPAL (Run Process As Library), a framework designed to enable one process to invoke another as if making a local function call, all without going through the kernel.

# Design

First, let’s formalize RPAL’s core objectives:

1. Data-plane efficiency: Reduce the number of data copies from two (in the shared memory solution) to one.
2. Control-plane optimization: Eliminate the overhead of system calls and kernel's thread switches.
3. Application compatibility: Minimize the modifications to existing applications that utilize Unix domain sockets and the epoll() family.

To attain the first objective, processes that use RPAL share the same virtual address space. So one process can access another's data directly via a data pointer. This means data can be transferred from one process to another with just one copy operation.

To meet the second goal, RPAL relies on the shared address space to do lightweight context switching in user space, which we call an "RPAL call". This allows one process to execute another process's code just like a local function call.

To achieve the third target, RPAL stays compatible with the epoll family of functions, like epoll_create(), epoll_wait(), and epoll_ctl(). If an application uses epoll for IPC, developers can switch to RPAL with just a few small changes. For instance, you can just replace epoll_wait() with rpal_epoll_wait(). The basic epoll procedure, where a process waits for another to write to a monitored descriptor using an epoll file descriptor, still works fine with RPAL.

## Address space sharing

For address space sharing, RPAL partitions the entire userspace virtual address space and allocates non-overlapping memory ranges to each process. On x86_64 architectures, RPAL uses a memory range size covered by a single PUD (Page Upper Directory) entry, which is 512GB. This restricts each process’s virtual address space to 512GB on x86_64, sufficient for most applications in our scenario. The rationale is straightforward:
address space sharing can be simply achieved by copying the PUD from one process’s page table to another’s. So one process can directly use the data pointer to access another's memory.

 |------------| <- 0
 |------------| <- 512 GB
 |  Process A |
 |------------| <- 2*512 GB
 |------------| <- n*512 GB
 |  Process B |
 |------------| <- (n+1)*512 GB
 |------------| <- STACK_TOP
 |  Kernel    |
 |------------|

## RPAL call

We refer to the lightweight userspace context switching mechanism as RPAL call. It enables the caller (or sender) thread of one process to directly switch to the callee (or receiver) thread of another process.

When Process A’s caller thread initiates an RPAL call to Process B’s callee thread, the CPU saves the caller’s context and loads the callee’s context. This enables direct userspace control flow transfer from the caller to the callee. After the callee finishes data processing, the CPU saves Process B’s callee context and switches back to Process A’s caller context, completing a full IPC cycle.

 |------------|                |---------------------|  
 |  Process A |                |  Process B          |
 | |-------|  |                | |-------|           |     
 | | caller| --- RPAL call --> | | callee|    handle |
 | | thread| <------------------ | thread| -> event  |
 | |-------|  |                | |-------|           |
 |------------|                |---------------------|

# Security and compatibility with kernel subsystems

## Memory protection between processes

Since processes using RPAL share the address space, unintended cross-process memory access may occur and corrupt the data of another process. To mitigate this, we leverage Memory Protection Keys (MPK) on x86 architectures.

MPK assigns 4 bits in each page table entry to a "protection key", which is paired with a userspace register (PKRU). The PKRU register defines access permissions for memory regions protected by specific keys (for detailed implementation, refer to the kernel documentation "Memory Protection Keys"). With MPK, even though the address space is shared among processes, cross-process access is restricted: a process can only access the memory protected by a key if its PKRU register is configured with the corresponding permission. This ensures that processes cannot access each other’s memory unless an explicit PKRU configuration is set.

## Page fault handling and TLB flushing

Due to the shared address space architecture, both page fault handling and TLB flushing require careful consideration. For instance, when Process A accesses Process B’s memory, a page fault may occur in Process A's context, but the faulting address belongs to Process B. In this case, we must pass Process B's mm_struct to the page fault handler.

TLB flushing is more complex. When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

## Lazy switch of kernel context

In RPAL, a mismatch may arise between the user context and the kernel context. The RPAL call is designed solely to switch the user context, leaving the kernel context unchanged. For instance, when a RPAL call takes place, transitioning from caller thread to callee thread, and subsequently a system call is initiated within callee thread, the kernel will incorrectly utilize the caller's kernel context (such as the kernel stack) to process the system call.

To resolve context mismatch issues, a kernel context switch is triggered at the kernel entry point when the callee initiates a syscall or an exception/interrupt occurs. This mechanism ensures context consistency before processing system calls, interrupts, or exceptions. We refer to this kernel context switch as a "lazy switch" because it defers the switching operation from the traditional thread switch point to the next kernel entry point.

Lazy switch should be minimized as much as possible, as it significantly degrades performance. We currently utilize RPAL in an RPC framework, in which the RPC sender thread relies on the RPAL call to invoke the RPC receiver thread entirely in user space. In most cases, the receiver thread is free of system calls and the code execution time is relatively short. This characteristic effectively reduces the probability of a lazy switch occurring.

## Time slice correction

After an RPAL call, the callee's user mode code executes. However, the kernel incorrectly attributes this CPU time to the caller due to the unchanged kernel context.

To resolve this, we use the Time Stamp Counter (TSC) register to measure CPU time consumed by the callee thread in user space. The kernel then uses this user-reported timing data to adjust the CPU accounting for both the caller and callee thread, similar to how CPU steal time is implemented.

## Process recovery

Since processes can access each other’s memory, there is a risk that the target process’s memory may become invalid at the access time (e.g., if the target process has exited unexpectedly). The kernel must handle such cases; otherwise, the accessing process could be terminated due to failures originating from another process.

To address this issue, each thread of the process should pre-establish a recovery point when accessing the memory of other processes. When such an invalid access occurs, the thread traps into the kernel. Inside the page fault handler, the kernel restores the user context of the thread to the recovery point. This mechanism ensures that processes maintain mutual independence, preventing cascading failures caused by cross-process memory issues.

# Performance

To quantify the performance improvements driven by RPAL, we measured latency both before and after its deployment. Experiments were conducted on a server equipped with two Intel(R) Xeon(R) Platinum 8336C CPUs (2.30 GHz) and 1 TB of memory. Latency was defined as the duration from when the client thread initiates a message to when the server thread is invoked and receives it.

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

We have applied RPAL to an RPC framework that is widely used in our data center. With RPAL, we have successfully achieved up to 15.5% reduction in the CPU utilization of processes in real-world microservice scenario. The gains primarily stem from minimizing control plane overhead through the utilization of userspace context switches. Additionally, by leveraging address space sharing, the number of memory copies is significantly reduced.

# Future Work

Currently, RPAL requires the MPK (Memory Protection Key) hardware feature, which is supported by a range of Intel CPUs. For AMD architectures, MPK is supported only on the latest processor, specifically, 3th Generation AMD EPYC™ Processors and subsequent generations. Patch sets that extend RPAL support to systems lacking MPK hardware will be provided later.

Accompanying test programs are also provided in the samples/rpal/ directory. And the user-mode RPAL library, which realizes user-space RPAL call, is in the samples/rpal/librpal directory.

We hope to get some community discussions and feedback on RPAL's optimization approaches and architecture.

Look forward to your comments.

Bo Li (35):
Kbuild: rpal support
RPAL: add struct rpal_service
RPAL: add service registration interface
RPAL: add member to task_struct and mm_struct
RPAL: enable virtual address space partitions
RPAL: add user interface
RPAL: enable shared page mmap
RPAL: enable sender/receiver registration
RPAL: enable address space sharing
RPAL: allow service enable/disable
RPAL: add service request/release
RPAL: enable service disable notification
RPAL: add tlb flushing support
RPAL: enable page fault handling
RPAL: add sender/receiver state
RPAL: add cpu lock interface
RPAL: add a mapping between fsbase and tasks
sched: pick a specified task
RPAL: add lazy switch main logic
RPAL: add rpal_ret_from_lazy_switch
RPAL: add kernel entry handling for lazy switch
RPAL: rebuild receiver state
RPAL: resume cpumask when fork
RPAL: critical section optimization
RPAL: add MPK initialization and interface
RPAL: enable MPK support
RPAL: add epoll support
RPAL: add rpal_uds_fdmap() support
RPAL: fix race condition in pkru update
RPAL: fix pkru setup when fork
RPAL: add receiver waker
RPAL: fix unknown nmi on AMD CPU
RPAL: enable time slice correction
RPAL: enable fast epoll wait
samples/rpal: add RPAL samples

 arch/x86/Kbuild                               |    2 +
 arch/x86/Kconfig                              |    2 +
 arch/x86/entry/entry_64.S                     |  160 ++
 arch/x86/events/amd/core.c                    |   14 +
 arch/x86/include/asm/pgtable.h                |   25 +
 arch/x86/include/asm/pgtable_types.h          |   11 +
 arch/x86/include/asm/tlbflush.h               |   10 +
 arch/x86/kernel/asm-offsets.c                 |    3 +
 arch/x86/kernel/cpu/common.c                  |    8 +-
 arch/x86/kernel/fpu/core.c                    |    8 +-
 arch/x86/kernel/nmi.c                         |   20 +
 arch/x86/kernel/process.c                     |   25 +-
 arch/x86/kernel/process_64.c                  |  118 +
 arch/x86/mm/fault.c                           |  271 ++
 arch/x86/mm/mmap.c                            |   10 +
 arch/x86/mm/tlb.c                             |  172 ++
 arch/x86/rpal/Kconfig                         |   21 +
 arch/x86/rpal/Makefile                        |    6 +
 arch/x86/rpal/core.c                          |  477 ++++
 arch/x86/rpal/internal.h                      |   69 +
 arch/x86/rpal/mm.c                            |  426 +++
 arch/x86/rpal/pku.c                           |  196 ++
 arch/x86/rpal/proc.c                          |  279 ++
 arch/x86/rpal/service.c                       |  776 ++++++
 arch/x86/rpal/thread.c                        |  313 +++
 fs/binfmt_elf.c                               |   98 +-
 fs/eventpoll.c                                |  320 +++
 fs/exec.c                                     |   11 +
 include/linux/mm_types.h                      |    3 +
 include/linux/rpal.h                          |  633 +++++
 include/linux/sched.h                         |   21 +
 init/init_task.c                              |    6 +
 kernel/exit.c                                 |    5 +
 kernel/fork.c                                 |   32 +
 kernel/sched/core.c                           |  676 +++++
 kernel/sched/fair.c                           |  109 +
 kernel/sched/sched.h                          |    8 +
 mm/mmap.c                                     |   16 +
 mm/mprotect.c                                 |  106 +
 mm/rmap.c                                     |    4 +
 mm/vma.c                                      |   18 +
 samples/rpal/Makefile                         |   17 +
 samples/rpal/asm_define.c                     |   14 +
 samples/rpal/client.c                         |  178 ++
 samples/rpal/librpal/asm_define.h             |    6 +
 samples/rpal/librpal/asm_x86_64_rpal_call.S   |   57 +
 samples/rpal/librpal/debug.h                  |   12 +
 samples/rpal/librpal/fiber.c                  |  119 +
 samples/rpal/librpal/fiber.h                  |   64 +
 .../rpal/librpal/jump_x86_64_sysv_elf_gas.S   |   81 +
 .../rpal/librpal/make_x86_64_sysv_elf_gas.S   |   82 +
 .../rpal/librpal/ontop_x86_64_sysv_elf_gas.S  |   84 +
 samples/rpal/librpal/private.h                |  341 +++
 samples/rpal/librpal/rpal.c                   | 2351 +++++++++++++++++
 samples/rpal/librpal/rpal.h                   |  149 ++
 samples/rpal/librpal/rpal_pkru.h              |   78 +
 samples/rpal/librpal/rpal_queue.c             |  239 ++
 samples/rpal/librpal/rpal_queue.h             |   55 +
 samples/rpal/librpal/rpal_x86_64_call_ret.S   |   45 +
 samples/rpal/offset.sh                        |    5 +
 samples/rpal/server.c                         |  249 ++
 61 files changed, 9710 insertions(+), 4 deletions(-)
 create mode 100644 arch/x86/rpal/Kconfig
 create mode 100644 arch/x86/rpal/Makefile
 create mode 100644 arch/x86/rpal/core.c
 create mode 100644 arch/x86/rpal/internal.h
 create mode 100644 arch/x86/rpal/mm.c
 create mode 100644 arch/x86/rpal/pku.c
 create mode 100644 arch/x86/rpal/proc.c
 create mode 100644 arch/x86/rpal/service.c
 create mode 100644 arch/x86/rpal/thread.c
 create mode 100644 include/linux/rpal.h
 create mode 100644 samples/rpal/Makefile
 create mode 100644 samples/rpal/asm_define.c
 create mode 100644 samples/rpal/client.c
 create mode 100644 samples/rpal/librpal/asm_define.h
 create mode 100644 samples/rpal/librpal/asm_x86_64_rpal_call.S
 create mode 100644 samples/rpal/librpal/debug.h
 create mode 100644 samples/rpal/librpal/fiber.c
 create mode 100644 samples/rpal/librpal/fiber.h
 create mode 100644 samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/make_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/private.h
 create mode 100644 samples/rpal/librpal/rpal.c
 create mode 100644 samples/rpal/librpal/rpal.h
 create mode 100644 samples/rpal/librpal/rpal_pkru.h
 create mode 100644 samples/rpal/librpal/rpal_queue.c
 create mode 100644 samples/rpal/librpal/rpal_queue.h
 create mode 100644 samples/rpal/librpal/rpal_x86_64_call_ret.S
 create mode 100755 samples/rpal/offset.sh
 create mode 100644 samples/rpal/server.c
Signature · 1 line
2.20.1
author
#1 lore raw

RFC v2 1/35 Kbuild: rpal support

Add kbuild support for RPAL, including new folder arch/x86/kernel/rpal and new config CONFIG_RPAL.

---
 arch/x86/Kbuild        |  2 ++
 arch/x86/Kconfig       |  2 ++
 arch/x86/rpal/Kconfig  | 11 +++++++++++
 arch/x86/rpal/Makefile |  0
 4 files changed, 15 insertions(+)
 create mode 100644 arch/x86/rpal/Kconfig
 create mode 100644 arch/x86/rpal/Makefile
arch/x86/Kbuild +2 -0
index f7fb3d88c57b..26c406442d79 100644
@@ -34,5 +34,7 @@ obj-$(CONFIG_KEXEC_FILE) += purgatory/
 
 obj-y += virt/
 
+obj-y += rpal/
+
 # for cleaning
 subdir- += boot tools
arch/x86/Kconfig +2 -0
index 121f9f03bd5c..3f53b6fc943f 100644
@@ -2359,6 +2359,8 @@ config X86_BUS_LOCK_DETECT
 	  Enable Split Lock Detect and Bus Lock Detect functionalities.
 	  See <file:Documentation/arch/x86/buslock.rst> for more information.
 
+source "arch/x86/rpal/Kconfig"
+
 endmenu
 
 config CC_HAS_NAMED_AS
arch/x86/rpal/Kconfig new file +11 -0
new file mode 100644
index 000000000000..e5e6996553ea
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# This Kconfig describes RPAL options
+#
+
+config RPAL
+	def_bool y
+	depends on X86_64
+	help
+		This option enables system support for Run Process As
+		library (RPAL).
\ No newline at end of file
arch/x86/rpal/Makefile new file +0 -0
new file mode 100644
index 000000000000..e69de29bb2d1
Signature · 1 line
2.20.1
author
#2 lore raw

RFC v2 2/35 RPAL: add struct rpal_service

In reply to Bo Li

Each process that uses RPAL features is called an RPAL service.

This patch adds the RPAL header file rpal.h and defines the rpal_service structure. The struct rpal_service uses a dedicated kmem_cache for allocation and deallocation, and atomic variables to maintain references to the struct rpal_service. Additionally, the patch introduces the rpal_get_service() and rpal_put_service() interfaces to manage reference counts.

---
 arch/x86/rpal/Makefile   |  5 ++++
 arch/x86/rpal/core.c     | 32 +++++++++++++++++++++++
 arch/x86/rpal/internal.h | 13 ++++++++++
 arch/x86/rpal/service.c  | 56 ++++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h     | 43 ++++++++++++++++++++++++++++++
 5 files changed, 149 insertions(+)
 create mode 100644 arch/x86/rpal/core.c
 create mode 100644 arch/x86/rpal/internal.h
 create mode 100644 arch/x86/rpal/service.c
 create mode 100644 include/linux/rpal.h
arch/x86/rpal/Makefile +5 -0
index e69de29bb2d1..ee3698b5a9b3 100644
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_RPAL)		+= rpal.o
+
+rpal-y := service.o core.o
arch/x86/rpal/core.c new file +32 -0
new file mode 100644
index 000000000000..495dbc1b1536
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * RPAL service level operations
+ * Copyright (c) 2025, ByteDance. All rights reserved.
+ *
+ *     Author: Jiadong Sun <[email protected]>
+ */
+
+#include <linux/rpal.h>
+
+#include "internal.h"
+
+int __init rpal_init(void);
+
+bool rpal_inited;
+
+int __init rpal_init(void)
+{
+	int ret = 0;
+
+	ret = rpal_service_init();
+	if (ret)
+		goto fail;
+
+	rpal_inited = true;
+	return 0;
+
+fail:
+	rpal_err("rpal init fail\n");
+	return -1;
+}
+subsys_initcall(rpal_init);
arch/x86/rpal/internal.h new file +13 -0
new file mode 100644
index 000000000000..e44e6fc79677
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * RPAL service level operations
+ * Copyright (c) 2025, ByteDance. All rights reserved.
+ *
+ *     Author: Jiadong Sun <[email protected]>
+ */
+
+extern bool rpal_inited;
+
+/* service.c */
+int __init rpal_service_init(void);
+void __init rpal_service_exit(void);
arch/x86/rpal/service.c new file +56 -0

56 lines not on this page · raw message

include/linux/rpal.h new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#3 lore raw

RFC v2 3/35 RPAL: add service registration interface

In reply to Bo Li

Every rpal service should be registered and managed. Each RPAL service has a 64-bit key as its unique identifier, the key should never repeat before kernel reboot. Each RPAL service has an ID to indicate which 512GB virtual address space it can use. Any alive RPAL service has its unique ID, which will never be reused until the service dead.

This patch adds a registration interface for RPAL services. Newly registered rpal_service instances are assigned a key that increments starting from 1. The 64-bit length of the key ensures that keys are almost impossible to exhaust before system reboot. Meanwhile, a bitmap is used to allocate IDs, ensuring no duplicate IDs are assigned. RPAL services are managed via a hash list, which facilitates quick lookup of the corresponding rpal_service by key.

---
 arch/x86/rpal/service.c | 130 ++++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h    |  31 ++++++++++
 2 files changed, 161 insertions(+)
arch/x86/rpal/service.c +130 -0
index c8e609798d4f..609c9550540d 100644
@@ -13,13 +13,56 @@
 
 #include "internal.h"
 
+static DECLARE_BITMAP(rpal_id_bitmap, RPAL_NR_ID);
+static atomic64_t service_key_counter;
+static DEFINE_HASHTABLE(service_hash_table, ilog2(RPAL_NR_ID));
+DEFINE_SPINLOCK(hash_table_lock);
 static struct kmem_cache *service_cache;
 
+static inline void rpal_free_service_id(int id)
+{
+	clear_bit(id, rpal_id_bitmap);
+}
+
 static void __rpal_put_service(struct rpal_service *rs)
 {
 	kmem_cache_free(service_cache, rs);
 }
 
+static int rpal_alloc_service_id(void)
+{
+	int id;
+
+	do {
+		id = find_first_zero_bit(rpal_id_bitmap, RPAL_NR_ID);
+		if (id == RPAL_NR_ID) {
+			id = RPAL_INVALID_ID;
+			break;
+		}
+	} while (test_and_set_bit(id, rpal_id_bitmap));
+
+	return id;
+}
+
+static bool is_valid_id(int id)
+{
+	return id >= 0 && id < RPAL_NR_ID;
+}
+
+static u64 rpal_alloc_service_key(void)
+{
+	u64 key;
+
+	/* confirm we do not run out keys */
+	if (unlikely(atomic64_read(&service_key_counter) == _AC(-1, UL))) {
+		rpal_err("key is exhausted\n");
+		return RPAL_INVALID_KEY;
+	}
+
+	key = atomic64_fetch_inc(&service_key_counter);
+	return key;
+}
+
 struct rpal_service *rpal_get_service(struct rpal_service *rs)
 {
 	if (!rs)
@@ -37,6 +80,90 @@ void rpal_put_service(struct rpal_service *rs)
 		__rpal_put_service(rs);
 }
 
+static u32 get_hash_key(u64 key)
+{
+	return key % RPAL_NR_ID;
+}
+
+struct rpal_service *rpal_get_service_by_key(u64 key)
+{
+	struct rpal_service *rs, *rsp;
+	u32 hash_key = get_hash_key(key);
+
+	rs = NULL;
+	hash_for_each_possible(service_hash_table, rsp, hlist, hash_key) {
+		if (rsp->key == key) {
+			rs = rsp;
+			break;
+		}
+	}
+	return rpal_get_service(rs);
+}
+
+static void insert_service(struct rpal_service *rs)
+{
+	unsigned long flags;
+	int hash_key;
+
+	hash_key = get_hash_key(rs->key);
+
+	spin_lock_irqsave(&hash_table_lock, flags);
+	hash_add(service_hash_table, &rs->hlist, hash_key);
+	spin_unlock_irqrestore(&hash_table_lock, flags);
+}
+
+static void delete_service(struct rpal_service *rs)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&hash_table_lock, flags);
+	hash_del(&rs->hlist);
+	spin_unlock_irqrestore(&hash_table_lock, flags);
+}
+
+struct rpal_service *rpal_register_service(void)
+{
+	struct rpal_service *rs;
+
+	if (!rpal_inited)
+		return NULL;
+
+	rs = kmem_cache_zalloc(service_cache, GFP_KERNEL);
+	if (!rs)
+		goto alloc_fail;
+
+	rs->id = rpal_alloc_service_id();
+	if (!is_valid_id(rs->id))
+		goto id_fail;
+
+	rs->key = rpal_alloc_service_key();
+	if (unlikely(rs->key == RPAL_INVALID_KEY))
+		goto key_fail;
+
+	atomic_set(&rs->refcnt, 1);
+
+	insert_service(rs);
+
+	return rs;
+
+key_fail:
+	kmem_cache_free(service_cache, rs);
+id_fail:
+	rpal_free_service_id(rs->id);
+alloc_fail:
+	return NULL;
+}
+
+void rpal_unregister_service(struct rpal_service *rs)
+{
+	if (!rs)
+		return;
+
+	delete_service(rs);
+
+	rpal_put_service(rs);
+}
+
 int __init rpal_service_init(void)
 {
 	service_cache = kmem_cache_create("rpal_service_cache",
@@ -47,6 +174,9 @@ int __init rpal_service_init(void)
 		return -1;
 	}
 
+	bitmap_zero(rpal_id_bitmap, RPAL_NR_ID);
+	atomic64_set(&service_key_counter, RPAL_FIRST_KEY);
+
 	return 0;
 }
 
include/linux/rpal.h +31 -0
index 73468884cc5d..75c5acf33844 100644
@@ -11,13 +11,40 @@
 
 #include <linux/sched.h>
 #include <linux/types.h>
+#include <linux/hashtable.h>
 #include <linux/atomic.h>
 
 #define RPAL_ERROR_MSG "rpal error: "
 #define rpal_err(x...) pr_err(RPAL_ERROR_MSG x)
 #define rpal_err_ratelimited(x...) pr_err_ratelimited(RPAL_ERROR_MSG x)
 
+/*
+ * The first 512GB is reserved due to mmap_min_addr.
+ * The last 512GB is dropped since stack will be initially
+ * allocated at TASK_SIZE_MAX.
+ */
+#define RPAL_NR_ID 254
+#define RPAL_INVALID_ID -1
+#define RPAL_FIRST_KEY _AC(1, UL)
+#define RPAL_INVALID_KEY _AC(0, UL)
+
+/*
+ * Each RPAL service has a 64-bit key as its unique identifier, and
+ * the 64-bit length ensures that the key will never repeat before
+ * the kernel reboot.
+ *
+ * Each RPAL service has an ID to indicate which 512GB virtual address
+ * space it can use. All alive RPAL processes have unique IDs, ensuring
+ * their address spaces do not overlap. When a process exits, its ID
+ * is released, allowing newly started RPAL services to reuse the ID.
+ */
 struct rpal_service {
+	/* Unique identifier for RPAL service */
+	u64 key;
+	/* virtual address space id */
+	int id;
+	/* Hashtable list for this struct */
+	struct hlist_node hlist;
 	/* reference count of this struct */
 	atomic_t refcnt;
 };
@@ -40,4 +67,8 @@ struct rpal_service *rpal_get_service(struct rpal_service *rs);
  * @param rs The struct rpal_service to put.
  */
 void rpal_put_service(struct rpal_service *rs);
+
+void rpal_unregister_service(struct rpal_service *rs);
+struct rpal_service *rpal_register_service(void);
+struct rpal_service *rpal_get_service_by_key(u64 key);
 #endif
Signature · 1 line
2.20.1
author
#4 lore raw

RFC v2 4/35 RPAL: add member to task_struct and mm_struct

In reply to Bo Li

In lazy switch and memory-related operations, there is a need to quickly locate the corresponding rpal_service structure. Therefore, rpal_service members are added to these two data structures.

This patch adds an rpal_service member to both task_struct and mm_struct, and introduces initialization operations. Meanwhile, rpal_service is also augmented with references to the task_struct and mm_struct of the group_leader. For threads created via fork, the kernel acquires a reference to rpal_service and assigns it to the new task_struct. References to rpal_service are released when threads exit.

Regarding the deallocation of rpal_struct, since rpal_put_service may be called in an atomic context (where mmdrop() cannot be invoked), this patch uses delayed work for deallocation. The work delay is set to 30 seconds, which ensures that IDs are not recycled and reused in the short term, preventing other processes from confusing the reallocated ID with the previous one due to race conditions.

---
 arch/x86/rpal/service.c  | 77 +++++++++++++++++++++++++++++++++++++---
 fs/exec.c                | 11 ++++++
 include/linux/mm_types.h |  3 ++
 include/linux/rpal.h     | 29 +++++++++++++++
 include/linux/sched.h    |  5 +++
 init/init_task.c         |  3 ++
 kernel/exit.c            |  5 +++
 kernel/fork.c            | 16 +++++++++
 8 files changed, 145 insertions(+), 4 deletions(-)
arch/x86/rpal/service.c +73 -4
index 609c9550540d..55ecb7e0ef8c 100644
@@ -26,9 +26,24 @@ static inline void rpal_free_service_id(int id)
 
 static void __rpal_put_service(struct rpal_service *rs)
 {
+	pr_debug("rpal: free service %d, tgid: %d\n", rs->id,
+		 rs->group_leader->pid);
+
+	rs->mm->rpal_rs = NULL;
+	mmdrop(rs->mm);
+	put_task_struct(rs->group_leader);
+	rpal_free_service_id(rs->id);
 	kmem_cache_free(service_cache, rs);
 }
 
+static void rpal_put_service_async_fn(struct work_struct *work)
+{
+	struct rpal_service *rs =
+		container_of(work, struct rpal_service, delayed_put_work.work);
+
+	__rpal_put_service(rs);
+}
+
 static int rpal_alloc_service_id(void)
 {
 	int id;
@@ -75,9 +90,16 @@ void rpal_put_service(struct rpal_service *rs)
 {
 	if (!rs)
 		return;
-
-	if (atomic_dec_and_test(&rs->refcnt))
-		__rpal_put_service(rs);
+    /*
+     * Since __rpal_put_service() calls mmdrop() (which
+     * cannot be invoked in atomic context), we use
+     * delayed work to release rpal_service.
+     */
+	if (atomic_dec_and_test(&rs->refcnt)) {
+		INIT_DELAYED_WORK(&rs->delayed_put_work,
+				  rpal_put_service_async_fn);
+		schedule_delayed_work(&rs->delayed_put_work, HZ * 30);
+	}
 }
 
 static u32 get_hash_key(u64 key)
@@ -128,6 +150,12 @@ struct rpal_service *rpal_register_service(void)
 	if (!rpal_inited)
 		return NULL;
 
+	if (!thread_group_leader(current)) {
+		rpal_err("task %d is not group leader %d\n", current->pid,
+			 current->tgid);
+		goto alloc_fail;
+	}
+
 	rs = kmem_cache_zalloc(service_cache, GFP_KERNEL);
 	if (!rs)
 		goto alloc_fail;
@@ -140,10 +168,27 @@ struct rpal_service *rpal_register_service(void)
 	if (unlikely(rs->key == RPAL_INVALID_KEY))
 		goto key_fail;
 
-	atomic_set(&rs->refcnt, 1);
+	current->rpal_rs = rs;
+
+	rs->group_leader = get_task_struct(current);
+	mmgrab(current->mm);
+	current->mm->rpal_rs = rs;
+	rs->mm = current->mm;
+
+	/*
+	 * The reference comes from:
+	 * 1. registered service always has one reference
+	 * 2. leader_thread also has one reference
+	 * 3. mm also hold one reference
+	 */
+	atomic_set(&rs->refcnt, 3);
 
 	insert_service(rs);
 
+	pr_debug(
+		"rpal: register service, key: %llx, id: %d, command: %s, tgid: %d\n",
+		rs->key, rs->id, current->comm, current->tgid);
+
 	return rs;
 
 key_fail:
@@ -161,7 +206,31 @@ void rpal_unregister_service(struct rpal_service *rs)
 
 	delete_service(rs);
 
+	pr_debug("rpal: unregister service, id: %d, tgid: %d\n", rs->id,
+		 rs->group_leader->tgid);
+
+	rpal_put_service(rs);
+}
+
+void copy_rpal(struct task_struct *p)
+{
+	struct rpal_service *cur = rpal_current_service();
+
+	p->rpal_rs = rpal_get_service(cur);
+}
+
+void exit_rpal(bool group_dead)
+{
+	struct rpal_service *rs = rpal_current_service();
+
+	if (!rs)
+		return;
+
+	current->rpal_rs = NULL;
 	rpal_put_service(rs);
+
+	if (group_dead)
+		rpal_unregister_service(rs);
 }
 
 int __init rpal_service_init(void)
fs/exec.c +11 -0
index cfbb2b9ee3c9..922728aebebe 100644
@@ -68,6 +68,7 @@
 #include <linux/user_events.h>
 #include <linux/rseq.h>
 #include <linux/ksm.h>
+#include <linux/rpal.h>
 
 #include <linux/uaccess.h>
 #include <asm/mmu_context.h>
@@ -1076,6 +1077,16 @@ static int de_thread(struct task_struct *tsk)
 	/* we have changed execution domain */
 	tsk->exit_signal = SIGCHLD;
 
+#if IS_ENABLED(CONFIG_RPAL)
+	/*
+	 * The rpal process is going to load another binary, we
+	 * need to unregister rpal since it is going to be another
+	 * process. Other threads have already exited by the time
+	 * we come here, we need to set group_dead as true.
+	 */
+	exit_rpal(true);
+#endif
+
 	BUG_ON(!thread_group_leader(tsk));
 	return 0;
 
include/linux/mm_types.h +3 -0
index 32ba5126e221..b29adef082c6 100644
@@ -1172,6 +1172,9 @@ struct mm_struct {
 #ifdef CONFIG_MM_ID
 		mm_id_t mm_id;
 #endif /* CONFIG_MM_ID */
+#ifdef CONFIG_RPAL
+		struct rpal_service *rpal_rs;
+#endif
 	} __randomize_layout;
 
 	/*
include/linux/rpal.h +29 -0
index 75c5acf33844..7b9d90b62b3f 100644
@@ -11,6 +11,8 @@
 
 #include <linux/sched.h>
 #include <linux/types.h>
+#include <linux/sched/mm.h>
+#include <linux/workqueue.h>
 #include <linux/hashtable.h>
 #include <linux/atomic.h>
 
@@ -29,6 +31,9 @@
 #define RPAL_INVALID_KEY _AC(0, UL)
 
 /*
+ * Each RPAL process (a.k.a RPAL service) should have a pointer to
+ * struct rpal_service in all its tasks' task_struct.
+ *
  * Each RPAL service has a 64-bit key as its unique identifier, and
  * the 64-bit length ensures that the key will never repeat before
  * the kernel reboot.
@@ -39,10 +44,23 @@
  * is released, allowing newly started RPAL services to reuse the ID.
  */
 struct rpal_service {
+	/* The task_struct of thread group leader. */
+	struct task_struct *group_leader;
+	/* mm_struct of thread group */
+	struct mm_struct *mm;
 	/* Unique identifier for RPAL service */
 	u64 key;
 	/* virtual address space id */
 	int id;
+
+    /*
+     * Fields above should never change after initialization.
+     * Fields below may change after initialization.
+     */
+
+	/* delayed service put work */
+	struct delayed_work delayed_put_work;
+
 	/* Hashtable list for this struct */
 	struct hlist_node hlist;
 	/* reference count of this struct */
@@ -68,7 +86,18 @@ struct rpal_service *rpal_get_service(struct rpal_service *rs);
  */
 void rpal_put_service(struct rpal_service *rs);
 
+#ifdef CONFIG_RPAL
+static inline struct rpal_service *rpal_current_service(void)
+{
+	return current->rpal_rs;
+}
+#else
+static inline struct rpal_service *rpal_current_service(void) { return NULL; }
+#endif
+
 void rpal_unregister_service(struct rpal_service *rs);
 struct rpal_service *rpal_register_service(void);
 struct rpal_service *rpal_get_service_by_key(u64 key);
+void copy_rpal(struct task_struct *p);
+void exit_rpal(bool group_dead);
 #endif
include/linux/sched.h +5 -0
index 45e5953b8f32..ad35b197543c 100644
@@ -72,6 +72,7 @@ struct rcu_node;
 struct reclaim_state;
 struct robust_list_head;
 struct root_domain;
+struct rpal_service;
 struct rq;
 struct sched_attr;
 struct sched_dl_entity;
@@ -1645,6 +1646,10 @@ struct task_struct {
 	struct user_event_mm		*user_event_mm;
 #endif
 
+#ifdef CONFIG_RPAL
+	struct rpal_service			*rpal_rs;
+#endif
+
 	/* CPU-specific state of this task: */
 	struct thread_struct		thread;
 
init/init_task.c +3 -0
index e557f622bd90..0c5b1927da41 100644
@@ -220,6 +220,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 #ifdef CONFIG_SECCOMP_FILTER
 	.seccomp	= { .filter_count = ATOMIC_INIT(0) },
 #endif
+#ifdef CONFIG_RPAL
+	.rpal_rs = NULL,
+#endif
 };
 EXPORT_SYMBOL(init_task);
 
kernel/exit.c +5 -0
index 38645039dd8f..0c8387da59da 100644
@@ -70,6 +70,7 @@
 #include <linux/user_events.h>
 #include <linux/uaccess.h>
 #include <linux/pidfs.h>
+#include <linux/rpal.h>
 
 #include <uapi/linux/wait.h>
 
@@ -944,6 +945,10 @@ void __noreturn do_exit(long code)
 	taskstats_exit(tsk, group_dead);
 	trace_sched_process_exit(tsk, group_dead);
 
+#if IS_ENABLED(CONFIG_RPAL)
+	exit_rpal(group_dead);
+#endif
+
 	exit_mm();
 
 	if (group_dead)
kernel/fork.c +16 -0
index 85afccfdf3b1..1d1c8484a8f2 100644
@@ -105,6 +105,7 @@
 #include <uapi/linux/pidfd.h>
 #include <linux/pidfs.h>
 #include <linux/tick.h>
+#include <linux/rpal.h>
 
 #include <asm/pgalloc.h>
 #include <linux/uaccess.h>
@@ -1216,6 +1217,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 	tsk->mm_cid_active = 0;
 	tsk->migrate_from_cpu = -1;
 #endif
+
+#ifdef CONFIG_RPAL
+	tsk->rpal_rs = NULL;
+#endif
 	return tsk;
 
 free_stack:
@@ -1312,6 +1317,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
 #endif
 	mm_init_uprobes_state(mm);
 	hugetlb_count_init(mm);
+#ifdef CONFIG_RPAL
+	mm->rpal_rs = NULL;
+#endif
 
 	if (current->mm) {
 		mm->flags = mmf_init_flags(current->mm->flags);
@@ -2651,6 +2659,14 @@ __latent_entropy struct task_struct *copy_process(
 			current->signal->nr_threads++;
 			current->signal->quick_threads++;
 			atomic_inc(&current->signal->live);
+#if IS_ENABLED(CONFIG_RPAL)
+			/*
+			 * For rpal process, the child thread needs to
+			 * inherit p->rpal_rs. Therefore, we can get the
+			 * struct rpal_service for any thread of rpal process.
+			 */
+			copy_rpal(p);
+#endif
 			refcount_inc(&current->signal->sigcnt);
 			task_join_group_stop(p);
 			list_add_tail_rcu(&p->thread_node,
Signature · 1 line
2.20.1
author
#5 lore raw

RFC v2 5/35 RPAL: enable virtual address space partitions

In reply to Bo Li

Each RPAL service occupies a contiguous 512GB virtual address space, with its base address determined by the id assigned during initialization. For userspace virtual address space beyond this 512GB range, we employ memory ballooning to occupy these regions, ensuring that processes do not utilize these virtual addresses.

Since the address space layout is determined when the process is loaded, RPAL sets the unused fields in the header of the ELF binary to the "RPAL" characters to alter the loading method of RPAL processes, enabling the process to be located within the correct 512GB address space upon loading.

---
 arch/x86/mm/mmap.c      | 10 +++++
 arch/x86/rpal/Makefile  |  2 +-
 arch/x86/rpal/mm.c      | 70 +++++++++++++++++++++++++++++
 arch/x86/rpal/service.c |  8 ++++
 fs/binfmt_elf.c         | 98 ++++++++++++++++++++++++++++++++++++++++-
 include/linux/rpal.h    | 65 +++++++++++++++++++++++++++
 6 files changed, 251 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/rpal/mm.c
arch/x86/mm/mmap.c +10 -0
index 5ed2109211da..504f2b9a0e8e 100644
@@ -19,6 +19,7 @@
 #include <linux/sched/mm.h>
 #include <linux/compat.h>
 #include <linux/elf-randomize.h>
+#include <linux/rpal.h>
 #include <asm/elf.h>
 #include <asm/io.h>
 
@@ -119,6 +120,15 @@ static void arch_pick_mmap_base(unsigned long *base, unsigned long *legacy_base,
 		*base = mmap_base(random_factor, task_size, rlim_stack);
 }
 
+#ifdef CONFIG_RPAL
+void rpal_pick_mmap_base(struct mm_struct *mm, struct rlimit *rlim_stack)
+{
+	arch_pick_mmap_base(&mm->mmap_base, &mm->mmap_legacy_base,
+			arch_rnd(RPAL_MAX_RAND_BITS), rpal_get_top(mm->rpal_rs),
+			rlim_stack);
+}
+#endif
+
 void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack)
 {
 	if (mmap_is_legacy())
arch/x86/rpal/Makefile +1 -1
index ee3698b5a9b3..2c858a8d7b9e 100644
@@ -2,4 +2,4 @@
 
 obj-$(CONFIG_RPAL)		+= rpal.o
 
-rpal-y := service.o core.o
+rpal-y := service.o core.o mm.o
arch/x86/rpal/mm.c new file +70 -0

70 lines not on this page · raw message

arch/x86/rpal/service.c +8 -0
index 55ecb7e0ef8c..caa4afa5a2c6 100644
@@ -143,6 +143,11 @@ static void delete_service(struct rpal_service *rs)
 	spin_unlock_irqrestore(&hash_table_lock, flags);
 }
 
+static inline unsigned long calculate_base_address(int id)
+{
+	return RPAL_ADDRESS_SPACE_LOW + RPAL_ADDR_SPACE_SIZE * id;
+}
+
 struct rpal_service *rpal_register_service(void)
 {
 	struct rpal_service *rs;
@@ -168,6 +173,9 @@ struct rpal_service *rpal_register_service(void)
 	if (unlikely(rs->key == RPAL_INVALID_KEY))
 		goto key_fail;
 
+	rs->bad_service = false;
+	rs->base = calculate_base_address(rs->id);
+
 	current->rpal_rs = rs;
 
 	rs->group_leader = get_task_struct(current);
fs/binfmt_elf.c +97 -1

144 lines not on this page · raw message

include/linux/rpal.h +65 -0

98 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#6 lore raw

RFC v2 6/35 RPAL: add user interface

In reply to Bo Li

Add the userspace interface of RPAL. The interface makes use of /proc files. Compared with adding syscalls, /proc files provide more interfaces, such as mmap, poll, etc. These interfaces can facilitate RPAL to implement more complex kernel-space/user-space interaction functions in the future.

This patch implements the ioctl interface, The interfaces initially implemented include obtaining the RPAL version, and retrieving the key and ID of the RPAL service.

---
 arch/x86/rpal/Makefile   |  2 +-
 arch/x86/rpal/core.c     |  3 ++
 arch/x86/rpal/internal.h |  3 ++
 arch/x86/rpal/proc.c     | 71 ++++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h     | 34 +++++++++++++++++++
 5 files changed, 112 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/rpal/proc.c
arch/x86/rpal/Makefile +1 -1
index 2c858a8d7b9e..a5926fc19334 100644
@@ -2,4 +2,4 @@
 
 obj-$(CONFIG_RPAL)		+= rpal.o
 
-rpal-y := service.o core.o mm.o
+rpal-y := service.o core.o mm.o proc.o
arch/x86/rpal/core.c +3 -0
index 495dbc1b1536..61f5d40b0157 100644
@@ -13,11 +13,14 @@
 int __init rpal_init(void);
 
 bool rpal_inited;
+unsigned long rpal_cap;
 
 int __init rpal_init(void)
 {
 	int ret = 0;
 
+	rpal_cap = 0;
+
 	ret = rpal_service_init();
 	if (ret)
 		goto fail;
arch/x86/rpal/internal.h +3 -0
index e44e6fc79677..c102a4c50515 100644
@@ -6,6 +6,9 @@
  *     Author: Jiadong Sun <[email protected]>
  */
 
+#define RPAL_COMPAT_VERSION 1
+#define RPAL_API_VERSION 1
+
 extern bool rpal_inited;
 
 /* service.c */
arch/x86/rpal/proc.c new file +71 -0

71 lines not on this page · raw message

include/linux/rpal.h +34 -0
index f7c0de747f55..3bc2a2a44265 100644
@@ -77,6 +77,8 @@
 #define RPAL_ADDRESS_SPACE_LOW  ((0UL) + RPAL_ADDR_SPACE_SIZE)
 #define RPAL_ADDRESS_SPACE_HIGH ((0UL) + RPAL_NR_ADDR_SPACE * RPAL_ADDR_SPACE_SIZE)
 
+extern unsigned long rpal_cap;
+
 /*
  * Each RPAL process (a.k.a RPAL service) should have a pointer to
  * struct rpal_service in all its tasks' task_struct.
@@ -118,6 +120,38 @@ struct rpal_service {
 	atomic_t refcnt;
 };
 
+/*
+ * Following structures should have the same memory layout with user.
+ * It seems nothing being different between kernel and user structure
+ * padding by different C compilers on x86_64, so we need to do nothing
+ * special here.
+ */
+/* Begin */
+struct rpal_version_info {
+	int compat_version;
+	int api_version;
+	unsigned long cap;
+};
+
+/* End */
+
+enum rpal_command_type {
+	RPAL_CMD_GET_API_VERSION_AND_CAP,
+	RPAL_CMD_GET_SERVICE_KEY,
+	RPAL_CMD_GET_SERVICE_ID,
+	RPAL_NR_CMD,
+};
+
+/* RPAL ioctl macro */
+#define RPAL_IOCTL_MAGIC 0x33
+#define RPAL_IOCTL_GET_API_VERSION_AND_CAP                        \
+	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_GET_API_VERSION_AND_CAP, \
+	      struct rpal_version_info *)
+#define RPAL_IOCTL_GET_SERVICE_KEY \
+	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_GET_SERVICE_KEY, u64 *)
+#define RPAL_IOCTL_GET_SERVICE_ID \
+	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_GET_SERVICE_ID, int *)
+
 /**
  * @brief get new reference to a rpal service, a corresponding
  *  rpal_put_service() should be called later by the caller.
Signature · 1 line
2.20.1
author
#7 lore raw

RFC v2 7/35 RPAL: enable shared page mmap

In reply to Bo Li

RPAL needs to create shared memory between the kernel and user space for the transfer of states and data.

This patch implements the rpal_mmap() interface. User processes can create shared memory by calling mmap() on /proc/rpal. To prevent users from creating excessive memory, rpal_mmap() limits the total size of the shared memory that can be created. The shared memory is maintained through reference counting, and rpal_munmap() is implemented for the release of the shared memory.

---
 arch/x86/rpal/internal.h |  20 ++++++
 arch/x86/rpal/mm.c       | 147 +++++++++++++++++++++++++++++++++++++++
 arch/x86/rpal/proc.c     |   1 +
 arch/x86/rpal/service.c  |   4 ++
 include/linux/rpal.h     |  15 ++++
 mm/mmap.c                |   4 ++
 6 files changed, 191 insertions(+)
arch/x86/rpal/internal.h +20 -0
index c102a4c50515..65fd14a26f0e 100644
@@ -9,8 +9,28 @@
 #define RPAL_COMPAT_VERSION 1
 #define RPAL_API_VERSION 1
 
+#include <linux/mm.h>
+#include <linux/file.h>
+
 extern bool rpal_inited;
 
 /* service.c */
 int __init rpal_service_init(void);
 void __init rpal_service_exit(void);
+
+/* mm.c */
+static inline struct rpal_shared_page *
+rpal_get_shared_page(struct rpal_shared_page *rsp)
+{
+	atomic_inc(&rsp->refcnt);
+	return rsp;
+}
+
+static inline void rpal_put_shared_page(struct rpal_shared_page *rsp)
+{
+	atomic_dec(&rsp->refcnt);
+}
+
+int rpal_mmap(struct file *filp, struct vm_area_struct *vma);
+struct rpal_shared_page *rpal_find_shared_page(struct rpal_service *rs,
+					       unsigned long addr);
arch/x86/rpal/mm.c +147 -0

156 lines not on this page · raw message

arch/x86/rpal/proc.c +1 -0
index 1ced30e25c15..86947dc233d0 100644
@@ -61,6 +61,7 @@ static long rpal_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 const struct proc_ops proc_rpal_operations = {
 	.proc_open = rpal_open,
 	.proc_ioctl = rpal_ioctl,
+	.proc_mmap = rpal_mmap,
 };
 
 static int __init proc_rpal_init(void)
arch/x86/rpal/service.c +4 -0
index caa4afa5a2c6..f29a046fc22f 100644
@@ -173,6 +173,10 @@ struct rpal_service *rpal_register_service(void)
 	if (unlikely(rs->key == RPAL_INVALID_KEY))
 		goto key_fail;
 
+	mutex_init(&rs->mutex);
+	rs->nr_shared_pages = 0;
+	INIT_LIST_HEAD(&rs->shared_pages);
+
 	rs->bad_service = false;
 	rs->base = calculate_base_address(rs->id);
 
include/linux/rpal.h +15 -0
index 3bc2a2a44265..986dfbd16fc9 100644
@@ -110,6 +110,12 @@ struct rpal_service {
      * Fields above should never change after initialization.
      * Fields below may change after initialization.
      */
+	/* Mutex for time consuming operations */
+	struct mutex mutex;
+
+	/* pinned pages */
+	int nr_shared_pages;
+	struct list_head shared_pages;
 
 	/* delayed service put work */
 	struct delayed_work delayed_put_work;
@@ -135,6 +141,14 @@ struct rpal_version_info {
 
 /* End */
 
+struct rpal_shared_page {
+	unsigned long user_start;
+	unsigned long kernel_start;
+	int npage;
+	atomic_t refcnt;
+	struct list_head list;
+};
+
 enum rpal_command_type {
 	RPAL_CMD_GET_API_VERSION_AND_CAP,
 	RPAL_CMD_GET_SERVICE_KEY,
@@ -196,6 +210,7 @@ struct rpal_service *rpal_get_service_by_key(u64 key);
 void copy_rpal(struct task_struct *p);
 void exit_rpal(bool group_dead);
 int rpal_balloon_init(unsigned long base);
+void rpal_exit_mmap(struct mm_struct *mm);
 
 extern void rpal_pick_mmap_base(struct mm_struct *mm,
 	struct rlimit *rlim_stack);
mm/mmap.c +4 -0
index bd210aaf7ebd..98bb33d2091e 100644
@@ -48,6 +48,7 @@
 #include <linux/sched/mm.h>
 #include <linux/ksm.h>
 #include <linux/memfd.h>
+#include <linux/rpal.h>
 
 #include <linux/uaccess.h>
 #include <asm/cacheflush.h>
@@ -1319,6 +1320,9 @@ void exit_mmap(struct mm_struct *mm)
 	__mt_destroy(&mm->mm_mt);
 	mmap_write_unlock(mm);
 	vm_unacct_memory(nr_accounted);
+#if IS_ENABLED(CONFIG_RPAL)
+	rpal_exit_mmap(mm);
+#endif
 }
 
 /* Insert vm structure into process list sorted by address
Signature · 1 line
2.20.1
author
#8 lore raw

RFC v2 8/35 RPAL: enable sender/receiver registration

In reply to Bo Li

In RPAL, there are two roles: the sender (caller) and the receiver ( callee). This patch provides an interface for threads to register as a sender or a receiver with the kernel. Each sender and receiver has its own data structure, along with a block of memory shared between the user space and the kernel space, which is allocated through rpal_mmap().

---
 arch/x86/rpal/Makefile   |   2 +-
 arch/x86/rpal/internal.h |   7 ++
 arch/x86/rpal/proc.c     |  12 +++
 arch/x86/rpal/service.c  |   6 ++
 arch/x86/rpal/thread.c   | 165 +++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h     |  79 +++++++++++++++++++
 include/linux/sched.h    |  15 ++++
 init/init_task.c         |   2 +
 kernel/fork.c            |   2 +
 9 files changed, 289 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/rpal/thread.c
arch/x86/rpal/Makefile +1 -1
index a5926fc19334..89f745382c51 100644
@@ -2,4 +2,4 @@
 
 obj-$(CONFIG_RPAL)		+= rpal.o
 
-rpal-y := service.o core.o mm.o proc.o
+rpal-y := service.o core.o mm.o proc.o thread.o
arch/x86/rpal/internal.h +7 -0
index 65fd14a26f0e..3559c9c6e868 100644
@@ -34,3 +34,10 @@ static inline void rpal_put_shared_page(struct rpal_shared_page *rsp)
 int rpal_mmap(struct file *filp, struct vm_area_struct *vma);
 struct rpal_shared_page *rpal_find_shared_page(struct rpal_service *rs,
 					       unsigned long addr);
+
+/* thread.c */
+int rpal_register_sender(unsigned long addr);
+int rpal_unregister_sender(void);
+int rpal_register_receiver(unsigned long addr);
+int rpal_unregister_receiver(void);
+void exit_rpal_thread(void);
arch/x86/rpal/proc.c +12 -0
index 86947dc233d0..8a1e4a8a2271 100644
@@ -51,6 +51,18 @@ static long rpal_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	case RPAL_IOCTL_GET_SERVICE_ID:
 		ret = put_user(cur->id, (int __user *)arg);
 		break;
+	case RPAL_IOCTL_REGISTER_SENDER:
+		ret = rpal_register_sender(arg);
+		break;
+	case RPAL_IOCTL_UNREGISTER_SENDER:
+		ret = rpal_unregister_sender();
+		break;
+	case RPAL_IOCTL_REGISTER_RECEIVER:
+		ret = rpal_register_receiver(arg);
+		break;
+	case RPAL_IOCTL_UNREGISTER_RECEIVER:
+		ret = rpal_unregister_receiver();
+		break;
 	default:
 		return -EINVAL;
 	}
arch/x86/rpal/service.c +6 -0
index f29a046fc22f..42fb719dbb2a 100644
@@ -176,6 +176,7 @@ struct rpal_service *rpal_register_service(void)
 	mutex_init(&rs->mutex);
 	rs->nr_shared_pages = 0;
 	INIT_LIST_HEAD(&rs->shared_pages);
+	atomic_set(&rs->thread_cnt, 0);
 
 	rs->bad_service = false;
 	rs->base = calculate_base_address(rs->id);
@@ -216,6 +217,9 @@ void rpal_unregister_service(struct rpal_service *rs)
 	if (!rs)
 		return;
 
+	while (atomic_read(&rs->thread_cnt) != 0)
+		schedule();
+
 	delete_service(rs);
 
 	pr_debug("rpal: unregister service, id: %d, tgid: %d\n", rs->id,
@@ -238,6 +242,8 @@ void exit_rpal(bool group_dead)
 	if (!rs)
 		return;
 
+	exit_rpal_thread();
+
 	current->rpal_rs = NULL;
 	rpal_put_service(rs);
 
arch/x86/rpal/thread.c new file +165 -0

165 lines not on this page · raw message

include/linux/rpal.h +79 -0

115 lines not on this page · raw message

include/linux/sched.h +15 -0
index ad35b197543c..5f25cc09fb71 100644
@@ -72,6 +72,9 @@ struct rcu_node;
 struct reclaim_state;
 struct robust_list_head;
 struct root_domain;
+struct rpal_common_data;
+struct rpal_receiver_data;
+struct rpal_sender_data;
 struct rpal_service;
 struct rq;
 struct sched_attr;
@@ -1648,6 +1651,18 @@ struct task_struct {
 
 #ifdef CONFIG_RPAL
 	struct rpal_service			*rpal_rs;
+	unsigned long				rpal_flag;
+	/*
+	 * The first member of both rpal_sd and rpal_rd has a type
+	 * of struct rpal_common_data. So if we do not care whether
+	 * it is a struct rpal_sender_data or a struct rpal_receiver_data,
+	 * use rpal_cd instead of rpal_sd or rpal_rd.
+	 */
+	union {
+		struct rpal_common_data *rpal_cd;
+		struct rpal_sender_data *rpal_sd;
+		struct rpal_receiver_data *rpal_rd;
+	};
 #endif
 
 	/* CPU-specific state of this task: */
init/init_task.c +2 -0
index 0c5b1927da41..2eb08b96e66b 100644
@@ -222,6 +222,8 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 #endif
 #ifdef CONFIG_RPAL
 	.rpal_rs = NULL,
+	.rpal_flag = 0,
+	.rpal_cd = NULL,
 #endif
 };
 EXPORT_SYMBOL(init_task);
kernel/fork.c +2 -0
index 1d1c8484a8f2..01cd48eadf68 100644
@@ -1220,6 +1220,8 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 
 #ifdef CONFIG_RPAL
 	tsk->rpal_rs = NULL;
+	tsk->rpal_flag = 0;
+	tsk->rpal_cd = NULL;
 #endif
 	return tsk;
 
Signature · 1 line
2.20.1
author
#9 lore raw

RFC v2 9/35 RPAL: enable address space sharing

In reply to Bo Li

RPAL's memory sharing is implemented by copying p4d entries, which requires implementing corresponding interfaces. Meanwhile, copying p4d entries can cause the process's page table to contain p4d entries that do not belong to it, and RPAL needs to resolve compatibility issues with other subsystems caused by this.

This patch implements the rpal_map_service() interface to complete the mutual copying of p4d entries between two RPAL services. For the copied p4d entries, RPAL adds a _PAGE_RPAL_IGN flag to them. This flag makes p4d_none() return true and p4d_present() return false, ensuring that these p4d entries are invisible to other kernel subsystems. The protection of p4d entries is guaranteed by the memory balloon, which ensures that the address space corresponding to the p4d entries is not used by the current service.

---
 arch/x86/include/asm/pgtable.h       |  25 ++++
 arch/x86/include/asm/pgtable_types.h |  11 ++
 arch/x86/rpal/internal.h             |   2 +
 arch/x86/rpal/mm.c                   | 175 +++++++++++++++++++++++++++
 4 files changed, 213 insertions(+)
arch/x86/include/asm/pgtable.h +25 -0
index 5ddba366d3b4..54351bfe4e47 100644
@@ -1137,12 +1137,37 @@ static inline int pud_bad(pud_t pud)
 #if CONFIG_PGTABLE_LEVELS > 3
 static inline int p4d_none(p4d_t p4d)
 {
+#if IS_ENABLED(CONFIG_RPAL)
+	p4dval_t p4dv = native_p4d_val(p4d);
+
+	/*
+	 * Since RPAL copy p4d entry to share address space,
+	 * it is important that other process will not manipulate
+	 * this copied p4d. Thus, make p4d_none() always return
+	 * 0 to bypass kernel page table logic on copied p4d.
+	 */
+	return (p4dv & _PAGE_RPAL_IGN) ||
+	       ((p4dv & ~(_PAGE_KNL_ERRATUM_MASK)) == 0);
+#else
 	return (native_p4d_val(p4d) & ~(_PAGE_KNL_ERRATUM_MASK)) == 0;
+#endif
 }
 
 static inline int p4d_present(p4d_t p4d)
 {
+#if IS_ENABLED(CONFIG_RPAL)
+	p4dval_t p4df = p4d_flags(p4d);
+
+	/*
+	 * Since RPAL copy p4d entry to share address space,
+	 * it is important that other process will not manipulate
+	 * this copied p4d. Thus, make p4d_present() always return
+	 * 0 to bypass kernel page table logic on copied p4d.
+	 */
+	return ((p4df & (_PAGE_PRESENT | _PAGE_RPAL_IGN)) == _PAGE_PRESENT);
+#else
 	return p4d_flags(p4d) & _PAGE_PRESENT;
+#endif
 }
 
 static inline pud_t *p4d_pgtable(p4d_t p4d)
arch/x86/include/asm/pgtable_types.h +11 -0
index b74ec5c3643b..781b0f5bc359 100644
@@ -35,6 +35,13 @@
 #define _PAGE_BIT_SOFT_DIRTY	_PAGE_BIT_SOFTW3 /* software dirty tracking */
 #define _PAGE_BIT_KERNEL_4K	_PAGE_BIT_SOFTW3 /* page must not be converted to large */
 #define _PAGE_BIT_DEVMAP	_PAGE_BIT_SOFTW4
+/*
+ * _PAGE_BIT_SOFTW1 is used by _PAGE_BIT_SPECIAL.
+ * but we are not conflicted with _PAGE_BIT_SPECIAL
+ * as we use it only on p4d/pud level and _PAGE_BIT_SPECIAL
+ * is only used on pte level.
+ */
+#define _PAGE_BIT_RPAL_IGN	_PAGE_BIT_SOFTW1
 
 #ifdef CONFIG_X86_64
 #define _PAGE_BIT_SAVED_DIRTY	_PAGE_BIT_SOFTW5 /* Saved Dirty bit (leaf) */
@@ -95,6 +102,10 @@
 #define _PAGE_SOFT_DIRTY	(_AT(pteval_t, 0))
 #endif
 
+#if IS_ENABLED(CONFIG_RPAL)
+#define _PAGE_RPAL_IGN	(_AT(pteval_t, 1) << _PAGE_BIT_RPAL_IGN)
+#endif
+
 /*
  * Tracking soft dirty bit when a page goes to a swap is tricky.
  * We need a bit which can be stored in pte _and_ not conflict
arch/x86/rpal/internal.h +2 -0
index 3559c9c6e868..65f2cf4baf8f 100644
@@ -34,6 +34,8 @@ static inline void rpal_put_shared_page(struct rpal_shared_page *rsp)
 int rpal_mmap(struct file *filp, struct vm_area_struct *vma);
 struct rpal_shared_page *rpal_find_shared_page(struct rpal_service *rs,
 					       unsigned long addr);
+int rpal_map_service(struct rpal_service *tgt);
+void rpal_unmap_service(struct rpal_service *tgt);
 
 /* thread.c */
 int rpal_register_sender(unsigned long addr);
arch/x86/rpal/mm.c +175 -0

178 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#10 lore raw

RFC v2 10/35 RPAL: allow service enable/disable

In reply to Bo Li

Since RPAL involves communication between services, and services require certain preparations (e.g., registering senders/receivers) before communication, the kernel needs to sense whether a service is ready to perform RPAL call-related operations.

This patch adds two interfaces: rpal_enable_service() and rpal_disable_service(). rpal_enable_service() passes necessary information to the kernel and marks the service as enabled. RPAL only permits communication between services in the enabled state. rpal_disable_service() clears the service's enabled state, thereby prohibiting communication between the service and others via RPAL.

---
 arch/x86/rpal/internal.h |  2 ++
 arch/x86/rpal/proc.c     |  6 +++++
 arch/x86/rpal/service.c  | 50 ++++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h     | 18 +++++++++++++++
 4 files changed, 76 insertions(+)
arch/x86/rpal/internal.h +2 -0
index 65f2cf4baf8f..769d3bbe5a6b 100644
@@ -17,6 +17,8 @@ extern bool rpal_inited;
 /* service.c */
 int __init rpal_service_init(void);
 void __init rpal_service_exit(void);
+int rpal_enable_service(unsigned long arg);
+int rpal_disable_service(void);
 
 /* mm.c */
 static inline struct rpal_shared_page *
arch/x86/rpal/proc.c +6 -0
index 8a1e4a8a2271..acd814f31649 100644
@@ -63,6 +63,12 @@ static long rpal_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	case RPAL_IOCTL_UNREGISTER_RECEIVER:
 		ret = rpal_unregister_receiver();
 		break;
+	case RPAL_IOCTL_ENABLE_SERVICE:
+		ret = rpal_enable_service(arg);
+		break;
+	case RPAL_IOCTL_DISABLE_SERVICE:
+		ret = rpal_disable_service();
+		break;
 	default:
 		return -EINVAL;
 	}
arch/x86/rpal/service.c +50 -0

68 lines not on this page · raw message

include/linux/rpal.h +18 -0
index c33425e896af..2e5010602177 100644
@@ -84,6 +84,14 @@ enum rpal_task_flag_bits {
 	RPAL_RECEIVER_BIT,
 };
 
+/*
+ * user_meta will be sent to other service when requested.
+ */
+struct rpal_service_metadata {
+	unsigned long version;
+	void __user *user_meta;
+};
+
 /*
  * Each RPAL process (a.k.a RPAL service) should have a pointer to
  * struct rpal_service in all its tasks' task_struct.
@@ -125,6 +133,10 @@ struct rpal_service {
 	/* sender/receiver thread count */
 	atomic_t thread_cnt;
 
+	/* service metadata */
+	bool enabled;
+	struct rpal_service_metadata rsm;
+
 	/* delayed service put work */
 	struct delayed_work delayed_put_work;
 
@@ -206,6 +218,8 @@ enum rpal_command_type {
 	RPAL_CMD_UNREGISTER_SENDER,
 	RPAL_CMD_REGISTER_RECEIVER,
 	RPAL_CMD_UNREGISTER_RECEIVER,
+	RPAL_CMD_ENABLE_SERVICE,
+	RPAL_CMD_DISABLE_SERVICE,
 	RPAL_NR_CMD,
 };
 
@@ -226,6 +240,10 @@ enum rpal_command_type {
 	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_REGISTER_RECEIVER, unsigned long)
 #define RPAL_IOCTL_UNREGISTER_RECEIVER \
 	_IO(RPAL_IOCTL_MAGIC, RPAL_CMD_UNREGISTER_RECEIVER)
+#define RPAL_IOCTL_ENABLE_SERVICE \
+	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_ENABLE_SERVICE, unsigned long)
+#define RPAL_IOCTL_DISABLE_SERVICE \
+	_IO(RPAL_IOCTL_MAGIC, RPAL_CMD_DISABLE_SERVICE)
 
 /**
  * @brief get new reference to a rpal service, a corresponding
Signature · 1 line
2.20.1
author
#11 lore raw

RFC v2 11/35 RPAL: add service request/release

In reply to Bo Li

Services communicating via RPAL require a series of operations to perform RPAL calls, such as mapping each other's memory and obtaining each other's metadata.

This patch adds the rpal_request_service() and rpal_release_service() interfaces. Before communication, services must first complete a handshake process by mutually requesting each other. Only after both parties have completed their requests will RPAL copy each other's p4d entries into the other party's page tables, thereby achieving address space sharing. The patch defines RPAL_REQUEST_MAP and RPAL_REVERSE_MAP to indicate whether a service has requested another service or has been requested by another service.

rpal_release_service() can release previously requested services, which triggers the removal of mutual p4d entries and terminates address space sharing. When a service exits the enabled state, the kernel will release all services it has ever requested, thereby terminating all address space sharing involving this service.

---
 arch/x86/rpal/internal.h |   5 +
 arch/x86/rpal/proc.c     |   6 +
 arch/x86/rpal/service.c  | 265 ++++++++++++++++++++++++++++++++++++++-
 include/linux/rpal.h     |  42 +++++++
 4 files changed, 316 insertions(+), 2 deletions(-)
arch/x86/rpal/internal.h +5 -0
index 769d3bbe5a6b..c504b6efff64 100644
@@ -12,6 +12,9 @@
 #include <linux/mm.h>
 #include <linux/file.h>
 
+#define RPAL_REQUEST_MAP 0x1
+#define RPAL_REVERSE_MAP 0x2
+
 extern bool rpal_inited;
 
 /* service.c */
@@ -19,6 +22,8 @@ int __init rpal_service_init(void);
 void __init rpal_service_exit(void);
 int rpal_enable_service(unsigned long arg);
 int rpal_disable_service(void);
+int rpal_request_service(unsigned long arg);
+int rpal_release_service(u64 key);
 
 /* mm.c */
 static inline struct rpal_shared_page *
arch/x86/rpal/proc.c +6 -0
index acd814f31649..f001afd40562 100644
@@ -69,6 +69,12 @@ static long rpal_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	case RPAL_IOCTL_DISABLE_SERVICE:
 		ret = rpal_disable_service();
 		break;
+	case RPAL_IOCTL_REQUEST_SERVICE:
+		ret = rpal_request_service(arg);
+		break;
+	case RPAL_IOCTL_RELEASE_SERVICE:
+		ret = rpal_release_service(arg);
+		break;
 	default:
 		return -EINVAL;
 	}
arch/x86/rpal/service.c +263 -2

292 lines not on this page · raw message

include/linux/rpal.h +42 -0

84 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#12 lore raw

RFC v2 12/35 RPAL: enable service disable notification

In reply to Bo Li

When a service is disabled, all services that request this service need to be notified.

This patch use poll functions of file to implement such notification. When a service is disabled, it will notify all services that request it by set bit in others services' dead_key_bitmap. And the poll function will then issue a poll epoll event, other services can aware the service has been disabled.

The key of disabled service can be read from the proc file.

---
 arch/x86/rpal/proc.c    | 61 +++++++++++++++++++++++++++++++++++++++++
 arch/x86/rpal/service.c | 37 +++++++++++++++++++++++--
 include/linux/rpal.h    | 10 +++++++
 3 files changed, 106 insertions(+), 2 deletions(-)
arch/x86/rpal/proc.c +61 -0
index f001afd40562..16ac9612bfc5 100644
@@ -8,6 +8,7 @@
 
 #include <linux/rpal.h>
 #include <linux/proc_fs.h>
+#include <linux/poll.h>
 
 #include "internal.h"
 
@@ -82,10 +83,70 @@ static long rpal_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	return ret;
 }
 
+static ssize_t rpal_read(struct file *file, char __user *buf, size_t count,
+		  loff_t *pos)
+{
+	struct rpal_service *cur = rpal_current_service();
+	struct rpal_poll_data *rpd;
+	u64 released_keys[MAX_REQUEST_SERVICE];
+	unsigned long flags;
+	int nr_key = 0;
+	int nr_byte = 0;
+	int idx;
+
+	if (!cur)
+		return -EINVAL;
+
+	rpd = &cur->rpd;
+
+	spin_lock_irqsave(&rpd->poll_lock, flags);
+	idx = find_first_bit(rpd->dead_key_bitmap, RPAL_NR_ID);
+	while (idx < RPAL_NR_ID) {
+		released_keys[nr_key++] = rpd->dead_keys[idx];
+		idx = find_next_bit(rpd->dead_key_bitmap, RPAL_NR_ID, idx + 1);
+	}
+	spin_unlock_irqrestore(&rpd->poll_lock, flags);
+	nr_byte = nr_key * sizeof(u64);
+
+	if (copy_to_user(buf, released_keys, nr_byte)) {
+		nr_byte = -EAGAIN;
+		goto out;
+	}
+out:
+	return nr_byte;
+}
+
+static __poll_t rpal_poll(struct file *filep, struct poll_table_struct *wait)
+{
+	struct rpal_service *cur = rpal_current_service();
+	struct rpal_poll_data *rpd;
+	unsigned long flags;
+	__poll_t mask = 0;
+
+	if (unlikely(!cur)) {
+		rpal_err("Not a rpal service\n");
+		goto out;
+	}
+
+	rpd = &cur->rpd;
+
+	poll_wait(filep, &rpd->rpal_waitqueue, wait);
+
+	spin_lock_irqsave(&rpd->poll_lock, flags);
+	if (find_first_bit(rpd->dead_key_bitmap, RPAL_NR_ID) < RPAL_NR_ID)
+		mask |= EPOLLIN | EPOLLRDNORM;
+	spin_unlock_irqrestore(&rpd->poll_lock, flags);
+
+out:
+	return mask;
+}
+
 const struct proc_ops proc_rpal_operations = {
 	.proc_open = rpal_open,
+	.proc_read = rpal_read,
 	.proc_ioctl = rpal_ioctl,
 	.proc_mmap = rpal_mmap,
+	.proc_poll = rpal_poll,
 };
 
 static int __init proc_rpal_init(void)
arch/x86/rpal/service.c +35 -2
index 16a2155873a1..f490ab07301d 100644
@@ -181,6 +181,9 @@ struct rpal_service *rpal_register_service(void)
 	atomic_set(&rs->req_avail_cnt, MAX_REQUEST_SERVICE);
 	bitmap_zero(rs->requested_service_bitmap, RPAL_NR_ID);
 	spin_lock_init(&rs->lock);
+	spin_lock_init(&rs->rpd.poll_lock);
+	bitmap_zero(rs->rpd.dead_key_bitmap, RPAL_NR_ID);
+	init_waitqueue_head(&rs->rpd.rpal_waitqueue);
 
 	rs->bad_service = false;
 	rs->base = calculate_base_address(rs->id);
@@ -296,6 +299,7 @@ static void remove_mapped_service(struct rpal_service *rs, int id, int type_bit)
 
 	clear_bit(type_bit, &node->type);
 	if (type_bit == RPAL_REQUEST_MAP) {
+		clear_bit(id, rs->rpd.dead_key_bitmap);
 		clear_requested_service_bitmap(rs, id);
 		atomic_inc(&rs->req_avail_cnt);
 	}
@@ -424,15 +428,30 @@ static int release_service(struct rpal_service *cur, struct rpal_service *tgt)
 	return 0;
 }
 
+static void rpal_notify_disable(struct rpal_poll_data *rpd, u64 key, int id)
+{
+	unsigned long flags;
+	bool need_wake = false;
+
+	spin_lock_irqsave(&rpd->poll_lock, flags);
+	if (!test_bit(id, rpd->dead_key_bitmap)) {
+		need_wake = true;
+		rpd->dead_keys[id] = key;
+		set_bit(id, rpd->dead_key_bitmap);
+	}
+	spin_unlock_irqrestore(&rpd->poll_lock, flags);
+	if (need_wake)
+		wake_up_interruptible(&rpd->rpal_waitqueue);
+}
+
 static void rpal_release_service_all(void)
 {
 	struct rpal_service *cur = rpal_current_service();
 	struct rpal_service *tgt;
+	struct rpal_mapped_service *node;
 	int ret, i;
 
 	rpal_for_each_requested_service(cur, i) {
-		struct rpal_mapped_service *node;
-
 		if (i == cur->id)
 			continue;
 		node = rpal_get_mapped_node(cur, i);
@@ -449,6 +468,20 @@ static void rpal_release_service_all(void)
 		}
 		rpal_put_service(tgt);
 	}
+
+	for (i = 0; i < RPAL_NR_ID; i++) {
+		if (i == cur->id)
+			continue;
+
+		node = rpal_get_mapped_node(cur, i);
+		tgt = rpal_get_service(node->rs);
+		if (!tgt)
+			continue;
+
+		if (test_bit(RPAL_REVERSE_MAP, &node->type))
+			rpal_notify_disable(&tgt->rpd, cur->key, cur->id);
+		rpal_put_service(tgt);
+	}
 }
 
 int rpal_release_service(u64 key)
include/linux/rpal.h +10 -0
index 1fe177523a36..b9622f0235bf 100644
@@ -107,6 +107,13 @@ struct rpal_mapped_service {
 	struct rpal_service *rs;
 };
 
+struct rpal_poll_data {
+	spinlock_t poll_lock;
+	u64 dead_keys[RPAL_NR_ID];
+	DECLARE_BITMAP(dead_key_bitmap, RPAL_NR_ID);
+	wait_queue_head_t rpal_waitqueue;
+};
+
 /*
  * Each RPAL process (a.k.a RPAL service) should have a pointer to
  * struct rpal_service in all its tasks' task_struct.
@@ -161,6 +168,9 @@ struct rpal_service {
 	struct rpal_mapped_service service_map[RPAL_NR_ID];
 	DECLARE_BITMAP(requested_service_bitmap, RPAL_NR_ID);
 
+	/* Notify service is released by others */
+	struct rpal_poll_data rpd;
+
 	/* delayed service put work */
 	struct delayed_work delayed_put_work;
 
Signature · 1 line
2.20.1
author
#13 lore raw

RFC v2 13/35 RPAL: add tlb flushing support

In reply to Bo Li

When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

This patch extend flush_tlb_info to store other process's mm_struct, and when a CPU in the union of the mm_cpumasks if invoked to handle tlb flushing, it will check whether cpu_tlbstate.loaded_mm matches any of mm_structs stored in flush_tlb_info. If match, the CPU will do local tlb flushing for that mm_struct.

---
 arch/x86/include/asm/tlbflush.h |  10 ++
 arch/x86/mm/tlb.c               | 172 ++++++++++++++++++++++++++++++++
 arch/x86/rpal/internal.h        |   3 -
 include/linux/rpal.h            |  12 +++
 mm/rmap.c                       |   4 +
 5 files changed, 198 insertions(+), 3 deletions(-)
arch/x86/include/asm/tlbflush.h +10 -0
index e9b81876ebe4..f57b745af75c 100644
@@ -227,6 +227,11 @@ struct flush_tlb_info {
 	u8			stride_shift;
 	u8			freed_tables;
 	u8			trim_cpumask;
+#ifdef CONFIG_RPAL
+	struct mm_struct **mm_list;
+	u64 *tlb_gen_list;
+	int nr_mm;
+#endif
 };
 
 void flush_tlb_local(void);
@@ -356,6 +361,11 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
 	mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
 }
 
+#ifdef CONFIG_RPAL
+void rpal_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
+	struct mm_struct *mm);
+#endif
+
 static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)
 {
 	flush_tlb_mm(mm);
arch/x86/mm/tlb.c +172 -0

196 lines not on this page · raw message

arch/x86/rpal/internal.h +0 -3
index c504b6efff64..cf6d608a994a 100644
@@ -12,9 +12,6 @@
 #include <linux/mm.h>
 #include <linux/file.h>
 
-#define RPAL_REQUEST_MAP 0x1
-#define RPAL_REVERSE_MAP 0x2
-
 extern bool rpal_inited;
 
 /* service.c */
include/linux/rpal.h +12 -0
index b9622f0235bf..36be1ab6a9f3 100644
@@ -80,6 +80,11 @@
 /* No more than 15 services can be requested due to limitation of MPK. */
 #define MAX_REQUEST_SERVICE 15
 
+enum {
+	RPAL_REQUEST_MAP,
+	RPAL_REVERSE_MAP,
+};
+
 extern unsigned long rpal_cap;
 
 enum rpal_task_flag_bits {
@@ -326,6 +331,13 @@ rpal_get_mapped_node(struct rpal_service *rs, int id)
 	return &rs->service_map[id];
 }
 
+static inline bool rpal_service_mapped(struct rpal_mapped_service *node)
+{
+	unsigned long type = (1 << RPAL_REQUEST_MAP) | (1 << RPAL_REVERSE_MAP);
+
+	return (node->type & type) == type;
+}
+
 #ifdef CONFIG_RPAL
 static inline struct rpal_service *rpal_current_service(void)
 {
mm/rmap.c +4 -0
index 67bb273dfb80..e68384f97ab9 100644
@@ -682,6 +682,10 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval,
 		return;
 
 	arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, start, end);
+#ifdef CONFIG_RPAL
+	if (mm->rpal_rs)
+		rpal_tlbbatch_add_pending(&tlb_ubc->arch, mm);
+#endif
 	tlb_ubc->flush_required = true;
 
 	/*
Signature · 1 line
2.20.1
author
#14 lore raw

RFC v2 14/35 RPAL: enable page fault handling

In reply to Bo Li

RPAL's address space sharing allows one process to access the memory of another process, which may trigger page faults. To ensure programs can run normally, RPAL needs to handle page faults occurring in the address space of other processes. Additionally, to prevent processes from generating coredumps due to invalid memory in other processes, RPAL must also restore the current thread state to a pre-saved state under specific circumstances.

For handling page faults, by passing the correct vm_area_struct to handle_page_fault(), RPAL locates the process corresponding to the address where the page fault occurred and uses its mm_struct to handle the page fault. Regarding thread state restoration, RPAL restores the thread's state to a predefined state in userspace when it cannot locate the mm_struct of the corresponding process (i.e., when the process has already exited).

---
 arch/x86/mm/fault.c     | 271 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/rpal/mm.c      |  34 +++++
 arch/x86/rpal/service.c |  24 ++++
 arch/x86/rpal/thread.c  |  23 ++++
 include/linux/rpal.h    |  81 ++++++++----
 5 files changed, 412 insertions(+), 21 deletions(-)
arch/x86/mm/fault.c +271 -0
index 998bd807fc7b..35f7c60a5e4f 100644
@@ -19,6 +19,7 @@
 #include <linux/mm_types.h>
 #include <linux/mm.h>			/* find_and_lock_vma() */
 #include <linux/vmalloc.h>
+#include <linux/rpal.h>
 
 #include <asm/cpufeature.h>		/* boot_cpu_has, ...		*/
 #include <asm/traps.h>			/* dotraplinkage, ...		*/
@@ -1460,6 +1461,268 @@ trace_page_fault_entries(struct pt_regs *regs, unsigned long error_code,
 		trace_page_fault_kernel(address, regs, error_code);
 }
 
+#if IS_ENABLED(CONFIG_RPAL)
+static void rpal_do_user_addr_fault(struct pt_regs *regs, unsigned long error_code,
+			     unsigned long address, struct mm_struct *real_mm)
+{
+	struct vm_area_struct *vma;
+	vm_fault_t fault;
+	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+
+	if (unlikely(error_code & X86_PF_RSVD))
+		pgtable_bad(regs, error_code, address);
+
+	if (unlikely(cpu_feature_enabled(X86_FEATURE_SMAP) &&
+		     !(error_code & X86_PF_USER) &&
+		     !(regs->flags & X86_EFLAGS_AC))) {
+		page_fault_oops(regs, error_code, address);
+		return;
+	}
+
+	if (unlikely(faulthandler_disabled())) {
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
+	}
+
+	if (WARN_ON_ONCE(!(regs->flags & X86_EFLAGS_IF))) {
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
+	}
+
+	local_irq_enable();
+
+	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
+
+	if (error_code & X86_PF_SHSTK)
+		flags |= FAULT_FLAG_WRITE;
+	if (error_code & X86_PF_WRITE)
+		flags |= FAULT_FLAG_WRITE;
+	if (error_code & X86_PF_INSTR)
+		flags |= FAULT_FLAG_INSTRUCTION;
+
+	if (user_mode(regs))
+		flags |= FAULT_FLAG_USER;
+
+#ifdef CONFIG_X86_64
+	if (is_vsyscall_vaddr(address)) {
+		if (emulate_vsyscall(error_code, regs, address))
+			return;
+	}
+#endif
+
+	if (!(flags & FAULT_FLAG_USER))
+		goto lock_mmap;
+
+	vma = lock_vma_under_rcu(real_mm, address);
+	if (!vma)
+		goto lock_mmap;
+
+	if (unlikely(access_error(error_code, vma))) {
+		bad_area_access_error(regs, error_code, address, NULL, vma);
+		count_vm_vma_lock_event(VMA_LOCK_SUCCESS);
+		return;
+	}
+
+	fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs);
+	if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED)))
+		vma_end_read(vma);
+
+	if (!(fault & VM_FAULT_RETRY)) {
+		count_vm_vma_lock_event(VMA_LOCK_SUCCESS);
+		goto done;
+	}
+	count_vm_vma_lock_event(VMA_LOCK_RETRY);
+	if (fault & VM_FAULT_MAJOR)
+		flags |= FAULT_FLAG_TRIED;
+
+	/* Quick path to respond to signals */
+	if (fault_signal_pending(fault, regs)) {
+		if (!user_mode(regs))
+			kernelmode_fixup_or_oops(regs, error_code, address,
+						 SIGBUS, BUS_ADRERR,
+						 ARCH_DEFAULT_PKEY);
+		return;
+	}
+lock_mmap:
+
+retry:
+	/*
+	 * Here we don't need to lock current->mm since no vma in
+	 * current->mm is used to handle this page fault. However,
+	 * we do need to lock real_mm, as the address belongs to
+	 * real_mm's vma.
+	 */
+	vma = lock_mm_and_find_vma(real_mm, address, regs);
+	if (unlikely(!vma)) {
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
+	}
+
+	if (unlikely(access_error(error_code, vma))) {
+		bad_area_access_error(regs, error_code, address, real_mm, vma);
+		return;
+	}
+
+	fault = handle_mm_fault(vma, address, flags, regs);
+
+	if (fault_signal_pending(fault, regs)) {
+		/*
+		 * Quick path to respond to signals.  The core mm code
+		 * has unlocked the mm for us if we get here.
+		 */
+		if (!user_mode(regs))
+			kernelmode_fixup_or_oops(regs, error_code, address,
+						 SIGBUS, BUS_ADRERR,
+						 ARCH_DEFAULT_PKEY);
+		return;
+	}
+
+	/* The fault is fully completed (including releasing mmap lock) */
+	if (fault & VM_FAULT_COMPLETED)
+		return;
+
+	if (unlikely(fault & VM_FAULT_RETRY)) {
+		flags |= FAULT_FLAG_TRIED;
+		goto retry;
+	}
+
+	mmap_read_unlock(real_mm);
+done:
+	if (likely(!(fault & VM_FAULT_ERROR)))
+		return;
+
+	if (fatal_signal_pending(current) && !user_mode(regs)) {
+		kernelmode_fixup_or_oops(regs, error_code, address, 0, 0,
+					 ARCH_DEFAULT_PKEY);
+		return;
+	}
+
+	if (fault & VM_FAULT_OOM) {
+		/* Kernel mode? Handle exceptions or die: */
+		if (!user_mode(regs)) {
+			kernelmode_fixup_or_oops(regs, error_code, address,
+						 SIGSEGV, SEGV_MAPERR,
+						 ARCH_DEFAULT_PKEY);
+			return;
+		}
+
+		pagefault_out_of_memory();
+	} else {
+		if (fault & (VM_FAULT_SIGBUS|VM_FAULT_HWPOISON|
+			     VM_FAULT_HWPOISON_LARGE))
+			do_sigbus(regs, error_code, address, fault);
+		else if (fault & VM_FAULT_SIGSEGV)
+			bad_area_nosemaphore(regs, error_code, address);
+		else
+			BUG();
+	}
+}
+NOKPROBE_SYMBOL(rpal_do_user_addr_fault);
+
+static inline void rpal_try_to_rebuild_context(struct pt_regs *regs,
+					       unsigned long address,
+					       int error_code)
+{
+	int handle_more = 0;
+
+	/*
+	 * We only rebuild sender's context, as other threads are not supposed
+	 * to access other process's memory, thus they will not trigger a page
+	 * fault.
+	 */
+	handle_more = rpal_rebuild_sender_context_on_fault(regs, address, -1);
+	/*
+	 * If we are not able to rebuild sender's context, just
+	 * send a signal to let it coredump.
+	 */
+	if (handle_more)
+		force_sig_fault(SIGSEGV, SEGV_MAPERR, (void __user *)address);
+}
+
+/*
+ * Most logic of this function is copied from do_user_addr_fault().
+ * RPAL logic is added to handle special cases, such as find another
+ * process's mm and rebuild sender's context if such page table is
+ * not able to be handled.
+ */
+static bool rpal_try_user_addr_fault(struct pt_regs *regs, unsigned long error_code,
+			      unsigned long address)
+{
+	struct mm_struct *real_mm;
+	int rebuild = 0;
+
+	/* fast path: avoid mmget and mmput */
+	if (unlikely((error_code & (X86_PF_USER | X86_PF_INSTR)) ==
+		     X86_PF_INSTR)) {
+		/*
+		 * Whoops, this is kernel mode code trying to execute from
+		 * user memory.  Unless this is AMD erratum #93, which
+		 * corrupts RIP such that it looks like a user address,
+		 * this is unrecoverable.  Don't even try to look up the
+		 * VMA or look for extable entries.
+		 */
+		if (is_errata93(regs, address))
+			return true;
+
+		page_fault_oops(regs, error_code, address);
+		return true;
+	}
+
+	/* kprobes don't want to hook the spurious faults: */
+	if (WARN_ON_ONCE(kprobe_page_fault(regs, X86_TRAP_PF)))
+		return true;
+
+	real_mm = rpal_pf_get_real_mm(address, &rebuild);
+
+	if (real_mm) {
+#ifdef CONFIG_MEMCG
+		struct mem_cgroup *memcg = NULL;
+
+		prefetchw(&real_mm->mmap_lock);
+		/* try to charge page alloc to real_mm's memcg */
+		if (!current->active_memcg) {
+			memcg = get_mem_cgroup_from_mm(real_mm);
+			if (memcg)
+				set_active_memcg(memcg);
+		}
+		rpal_do_user_addr_fault(regs, error_code, address, real_mm);
+		if (memcg) {
+			set_active_memcg(NULL);
+			mem_cgroup_put(memcg);
+		}
+#else
+		prefetchw(&real_mm->mmap_lock);
+		rpal_do_user_addr_fault(regs, error_code, address, real_mm);
+#endif
+		mmput_async(real_mm);
+		return true;
+	} else if (user_mode(regs) && rebuild) {
+		rpal_try_to_rebuild_context(regs, address, -1);
+		return true;
+	}
+
+	return false;
+}
+
+static bool rpal_handle_page_fault(struct pt_regs *regs, unsigned long error_code,
+			   unsigned long address)
+{
+	struct rpal_service *cur = rpal_current_service();
+
+	/*
+	 * For RPAL process, it may access another process's memory and
+	 * there may be page fault. We handle this case with our own routine.
+	 * If we cannot handle this page fault, just let it go and handle
+	 * it as a normal page fault.
+	 */
+	if (cur && !rpal_is_correct_address(cur, address)) {
+		if (rpal_try_user_addr_fault(regs, error_code, address))
+			return true;
+	}
+	return false;
+}
+#endif
+
 static __always_inline void
 handle_page_fault(struct pt_regs *regs, unsigned long error_code,
 			      unsigned long address)
@@ -1473,7 +1736,15 @@ handle_page_fault(struct pt_regs *regs, unsigned long error_code,
 	if (unlikely(fault_in_kernel_space(address))) {
 		do_kern_addr_fault(regs, error_code, address);
 	} else {
+#ifdef CONFIG_RPAL
+		if (rpal_handle_page_fault(regs, error_code, address)) {
+			local_irq_disable();
+			return;
+		}
+		do_user_addr_fault(regs, error_code, address);
+#else /* !CONFIG_RPAL */
 		do_user_addr_fault(regs, error_code, address);
+#endif
 		/*
 		 * User address page fault handling might have reenabled
 		 * interrupts. Fixing up all potential exit points of
arch/x86/rpal/mm.c +34 -0
index f1003baae001..be7714ede2bf 100644
@@ -390,3 +390,37 @@ void rpal_unmap_service(struct rpal_service *tgt)
 	}
 	mm_unlink_p4d(cur_mm, tgt->base);
 }
+
+static inline bool check_service_mapped(struct rpal_service *cur, int tgt_id)
+{
+	struct rpal_mapped_service *node;
+	bool is_mapped = true;
+	unsigned long type = (1 << RPAL_REVERSE_MAP) | (1 << RPAL_REQUEST_MAP);
+
+	node = rpal_get_mapped_node(cur, tgt_id);
+	if (unlikely((node->type & type) != type))
+		is_mapped = false;
+
+	return is_mapped;
+}
+
+struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild)
+{
+	struct rpal_service *cur, *tgt;
+	struct mm_struct *mm = NULL;
+
+	cur = rpal_current_service();
+
+	tgt = rpal_get_mapped_service_by_addr(cur, address);
+	if (tgt == NULL)
+		goto out;
+
+	mm = tgt->mm;
+	if (unlikely(!check_service_mapped(cur, tgt->id) ||
+		     !mmget_not_zero(mm)))
+		mm = NULL;
+	*rebuild = 1;
+	rpal_put_service(tgt);
+out:
+	return mm;
+}
arch/x86/rpal/service.c +24 -0
index f490ab07301d..49458321e7dc 100644
@@ -148,6 +148,30 @@ static inline unsigned long calculate_base_address(int id)
 	return RPAL_ADDRESS_SPACE_LOW + RPAL_ADDR_SPACE_SIZE * id;
 }
 
+struct rpal_service *rpal_get_mapped_service_by_id(struct rpal_service *rs,
+						   int id)
+{
+	struct rpal_service *ret;
+
+	if (!is_valid_id(id))
+		return NULL;
+
+	ret = rpal_get_service(rs->service_map[id].rs);
+
+	return ret;
+}
+
+/* This function must be called after rpal_is_correct_address () */
+struct rpal_service *rpal_get_mapped_service_by_addr(struct rpal_service *rs,
+						     unsigned long addr)
+{
+	int id;
+
+	id = (addr - RPAL_ADDRESS_SPACE_LOW) / RPAL_ADDR_SPACE_SIZE;
+
+	return rpal_get_mapped_service_by_id(rs, id);
+}
+
 struct rpal_service *rpal_register_service(void)
 {
 	struct rpal_service *rs;
arch/x86/rpal/thread.c +23 -0
index 7550ad94b63f..e50a4c865ff8 100644
@@ -155,6 +155,29 @@ int rpal_unregister_receiver(void)
 	return ret;
 }
 
+int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
+					 unsigned long addr, int error_code)
+{
+	if (rpal_test_current_thread_flag(RPAL_SENDER_BIT)) {
+		struct rpal_sender_call_context *scc = current->rpal_sd->scc;
+		unsigned long erip, ersp;
+		int magic;
+
+		erip = scc->ec.erip;
+		ersp = scc->ec.ersp;
+		magic = scc->ec.magic;
+		if (magic == RPAL_ERROR_MAGIC) {
+			regs->ax = error_code;
+			regs->ip = erip;
+			regs->sp = ersp;
+			/* avoid rebuild again */
+			scc->ec.magic = 0;
+			return 0;
+		}
+	}
+	return -EINVAL;
+}
+
 void exit_rpal_thread(void)
 {
 	if (rpal_test_current_thread_flag(RPAL_SENDER_BIT))
include/linux/rpal.h +60 -21

118 lines not on this page · raw message

Signature · 1 line
2.20.1
#15 lore raw

Re: [RFC v2 14/35] RPAL: enable page fault handling

Quoted 6 lines · Bo Li
 arch/x86/mm/fault.c     | 271 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/rpal/mm.c      |  34 +++++
 arch/x86/rpal/service.c |  24 ++++
 arch/x86/rpal/thread.c  |  23 ++++
 include/linux/rpal.h    |  81 ++++++++----
 5 files changed, 412 insertions(+), 21 deletions(-)

I'm actually impressed again that you've managed to get this ported over to a newer kernel _and_ broken it up.

But just taking a quick peek at _one_ patch, this is far, far below the standards by which we do kernel development. This appears to have simply copied chunks of existing code, hacked it to work with "RPAL" and then #ifdef'd.

This is, unfortunately, copying and pasting at its worst. It creates dual paths that inevitably bit rot in some way and are hard to maintain.

So, just to be clear: it's a full an unequivocal NAK from me on this series. This introduces massive change, massive security risk, can't possibly be backward compatible, has no users (well, maybe one) and the series is not put together in anything remotely resembling how we like to do kernel development.

I'd appreciate if you could not cc me on future versions if you choose to go forward with this. But I urge you to stop now.

author
#16 lore raw

RFC v2 15/35 RPAL: add sender/receiver state

In reply to Bo Li

The lazy switch defines six receiver states, and their state transitions are as follows:

|<--->READY<----> WAIT <----> CALL ----> LAZY_SWITCH ---> KERNEL_RET

   |                                             |               |
RUNNING <----------------------------------------|---------------|

The receiver thread initially starts in the RUNNING state and can transition to the WAIT state voluntarily. The READY state is a temporary state before entering WAIT state. For a receiver in the WAIT state, it must be in the TASK_INTERRUPTIBLE state. If the receiver thread is woken up, the WAIT state can transition to the RUNNING state.

Once the receiver is in the WAIT state, the sender thread can initiate an RPAL call, causing the receiver to enter the CALL state. A receiver thread in the CALL state cannot be awakened until a lazy switch occurs or its state changes. The call state carries additional service_id and sender_id information.

If the sender completes executing the receiver's code without entering the kernel after issuing the RPAL call, the receiver transitions back from the CALL state to the WAIT state. Conversely, if the sender enters the kernel during the RPAL call, the receiver's state changes to LAZY_SWITCH.

From the LAZY_SWITCH state, the receiver thread has two possible state
transitions: When the receiver thread finishes execution and switches back to the sender via a lazy switch, it first enters the KERNEL_RET state and then transitions to the RUNNING state. If the receiver thread runs for too long and the scheduler resumes the sender, the receiver directly transitions to the RUNNING state. Transitions to the RUNNING state can be done in userspace.

The lazy switch mechanism defines three states for the sender thread:

- RUNNING: The sender starts in this state. When the sender initiates an RPAL call to switch from user mode to the receiver, it transitions to the CALL state.

- CALL: The sender remains in this state while the receiver is executing the code triggered by the RPAL call. When the receiver switches back to the sender from user mode, the sender returns to the RUNNING state.

- KERNEL_RET: If the receiver takes an extended period to switch back to the sender after a lazy switch, the scheduler may preempt the sender to run other tasks. In this case, the sender enters the KERNEL_RET state while in the kernel. Once the sender resumes execution in user mode, it transitions back to the RUNNING state.

This patch implements the handling and transition of the receiver's state. When a receiver leaves the run queue in the READY state, its state transitions to the WAIT state; otherwise, it transitions to the RUNNING state. The patch also modifies try_to_wake_up() to handling different
states: for the READY and WAIT states, try_to_wake_up() causes the state to change to the RUNNING state. For the CALL state, try_to_wake_up() cannot wake up the task. The patch provides a special interface, rpal_try_to_wake_up(), to wake up tasks in the CALL state, which can be used for lazy switches.

---
 arch/x86/kernel/process_64.c |  43 ++++++++++++
 arch/x86/rpal/internal.h     |   7 ++
 include/linux/rpal.h         |  50 ++++++++++++++
 kernel/sched/core.c          | 130 +++++++++++++++++++++++++++++++++++
 4 files changed, 230 insertions(+)
arch/x86/kernel/process_64.c +43 -0
index f39ff02e498d..4830e9215de7 100644
@@ -40,6 +40,7 @@
 #include <linux/ftrace.h>
 #include <linux/syscalls.h>
 #include <linux/iommu.h>
+#include <linux/rpal.h>
 
 #include <asm/processor.h>
 #include <asm/pkru.h>
@@ -596,6 +597,36 @@ void compat_start_thread(struct pt_regs *regs, u32 new_ip, u32 new_sp, bool x32)
 }
 #endif
 
+#ifdef CONFIG_RPAL
+static void rpal_receiver_enter_wait(struct task_struct *prev_p)
+{
+	if (READ_ONCE(prev_p->__state) == TASK_INTERRUPTIBLE) {
+		atomic_cmpxchg(&prev_p->rpal_rd->rcc->receiver_state,
+			       RPAL_RECEIVER_STATE_READY,
+			       RPAL_RECEIVER_STATE_WAIT);
+	} else {
+		/*
+		 * Simply check RPAL_RECEIVER_STATE_READY is not enough. It is
+		 * possible task's state is TASK_RUNNING. Consider following case:
+		 *
+		 * CPU 0(prev_p)            CPU 1(waker)
+		 * set TASK_INTERRUPTIBLE
+		 * set RPAL_RECEIVER_STATE_READY
+		 *                          check TASK_INTERRUPTIBLE
+		 * clear RPAL_RECEIVER_STATE_READY
+		 * clear TASK_INTERRUPTIBLE
+		 * set TASK_INTERRUPTIBLE
+		 * set RPAL_RECEIVER_STATE_READY
+		 *                          ttwu_runnable()
+		 * schedule()
+		 */
+		atomic_cmpxchg(&prev_p->rpal_rd->rcc->receiver_state,
+			       RPAL_RECEIVER_STATE_READY,
+			       RPAL_RECEIVER_STATE_RUNNING);
+	}
+}
+#endif
+
 /*
  *	switch_to(x,y) should switch tasks from x to y.
  *
@@ -704,6 +735,18 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 			loadsegment(ss, __KERNEL_DS);
 	}
 
+#ifdef CONFIG_RPAL
+	/*
+	 * When we come to here, the stack switching is finished. Therefore,
+	 * the receiver thread is prepared for a lazy switch. We then change
+	 * the receiver_state from RPAL_RECEIVER_STATE_REAY to
+	 * RPAL_RECEIVER_STATE_WAIT and other thread is able to call it with
+	 * RPAL call.
+	 */
+	if (rpal_test_task_thread_flag(prev_p, RPAL_RECEIVER_BIT))
+		rpal_receiver_enter_wait(prev_p);
+#endif
+
 	/* Load the Intel cache allocation PQR MSR. */
 	resctrl_sched_in(next_p);
 
arch/x86/rpal/internal.h +7 -0
index cf6d608a994a..6256172bb79e 100644
@@ -47,3 +47,10 @@ int rpal_unregister_sender(void);
 int rpal_register_receiver(unsigned long addr);
 int rpal_unregister_receiver(void);
 void exit_rpal_thread(void);
+
+static inline unsigned long
+rpal_build_call_state(const struct rpal_sender_data *rsd)
+{
+	return ((rsd->rcd.service_id << RPAL_SID_SHIFT) |
+		(rsd->scc->sender_id << RPAL_ID_SHIFT) | RPAL_RECEIVER_STATE_CALL);
+}
include/linux/rpal.h +50 -0

83 lines not on this page · raw message

kernel/sched/core.c +130 -0

179 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#17 lore raw

RFC v2 16/35 RPAL: add cpu lock interface

In reply to Bo Li

Lazy switch enables the kernel to switch from one task to another to keep the kernel context and user context matched. For the scheduler, both tasks involved in the context switch must reside in the same run queue (rq). Therefore, before a lazy switch occurs, the kernel must first bind both tasks to the same CPU to facilitate the subsequent context switch.

This patch introduces the rpal_lock_cpu() interface, which binds two tasks to the same CPU while bypassing cpumask restrictions. The rpal_unlock_cpu() function serves as the inverse operation to release this binding. To ensure consistency, the kernel must prevent other threads from modifying the CPU affinity of tasks locked by rpal_lock_cpu(). Therefore, when using set_cpus_allowed_ptr() to change a task's CPU affinity, other threads must wait until the binding established by rpal_lock_cpu() is released before proceeding with modifications.

---
 arch/x86/rpal/core.c   |  18 +++++++
 arch/x86/rpal/thread.c |  14 ++++++
 include/linux/rpal.h   |   8 +++
 kernel/sched/core.c    | 109 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 149 insertions(+)
arch/x86/rpal/core.c +18 -0
index 61f5d40b0157..c185a453c1b2 100644
@@ -15,6 +15,24 @@ int __init rpal_init(void);
 bool rpal_inited;
 unsigned long rpal_cap;
 
+static inline void rpal_lock_cpu(struct task_struct *tsk)
+{
+	rpal_set_cpus_allowed_ptr(tsk, true);
+	if (unlikely(!irqs_disabled())) {
+		local_irq_disable();
+		rpal_err("%s: irq is enabled\n", __func__);
+	}
+}
+
+static inline void rpal_unlock_cpu(struct task_struct *tsk)
+{
+	rpal_set_cpus_allowed_ptr(tsk, false);
+	if (unlikely(!irqs_disabled())) {
+		local_irq_disable();
+		rpal_err("%s: irq is enabled\n", __func__);
+	}
+}
+
 int __init rpal_init(void)
 {
 	int ret = 0;
arch/x86/rpal/thread.c +14 -0
index e50a4c865ff8..bc203e9c6e5e 100644
@@ -47,6 +47,10 @@ int rpal_register_sender(unsigned long addr)
 	}
 
 	rpal_common_data_init(&rsd->rcd);
+	if (rpal_init_thread_pending(&rsd->rcd)) {
+		ret = -ENOMEM;
+		goto free_rsd;
+	}
 	rsd->rsp = rsp;
 	rsd->scc = (struct rpal_sender_call_context *)(addr - rsp->user_start +
 						       rsp->kernel_start);
@@ -58,6 +62,8 @@ int rpal_register_sender(unsigned long addr)
 
 	return 0;
 
+free_rsd:
+	kfree(rsd);
 put_shared_page:
 	rpal_put_shared_page(rsp);
 out:
@@ -77,6 +83,7 @@ int rpal_unregister_sender(void)
 
 	rpal_put_shared_page(rsd->rsp);
 	rpal_clear_current_thread_flag(RPAL_SENDER_BIT);
+	rpal_free_thread_pending(&rsd->rcd);
 	kfree(rsd);
 
 	atomic_dec(&cur->thread_cnt);
@@ -116,6 +123,10 @@ int rpal_register_receiver(unsigned long addr)
 	}
 
 	rpal_common_data_init(&rrd->rcd);
+	if (rpal_init_thread_pending(&rrd->rcd)) {
+		ret = -ENOMEM;
+		goto free_rrd;
+	}
 	rrd->rsp = rsp;
 	rrd->rcc =
 		(struct rpal_receiver_call_context *)(addr - rsp->user_start +
@@ -128,6 +139,8 @@ int rpal_register_receiver(unsigned long addr)
 
 	return 0;
 
+free_rrd:
+	kfree(rrd);
 put_shared_page:
 	rpal_put_shared_page(rsp);
 out:
@@ -147,6 +160,7 @@ int rpal_unregister_receiver(void)
 
 	rpal_put_shared_page(rrd->rsp);
 	rpal_clear_current_thread_flag(RPAL_RECEIVER_BIT);
+	rpal_free_thread_pending(&rrd->rcd);
 	kfree(rrd);
 
 	atomic_dec(&cur->thread_cnt);
include/linux/rpal.h +8 -0
index 4f4719bb7eae..5b115be14a55 100644
@@ -99,6 +99,7 @@ extern unsigned long rpal_cap;
 enum rpal_task_flag_bits {
 	RPAL_SENDER_BIT,
 	RPAL_RECEIVER_BIT,
+	RPAL_CPU_LOCKED_BIT,
 };
 
 enum rpal_receiver_state {
@@ -270,8 +271,12 @@ struct rpal_shared_page {
 struct rpal_common_data {
 	/* back pointer to task_struct */
 	struct task_struct *bp_task;
+	/* pending struct for cpu locking */
+	void *pending;
 	/* service id of rpal_service */
 	int service_id;
+	/* cpumask before locked */
+	cpumask_t old_mask;
 };
 
 struct rpal_receiver_data {
@@ -464,4 +469,7 @@ struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild);
 extern void rpal_pick_mmap_base(struct mm_struct *mm,
 	struct rlimit *rlim_stack);
 int rpal_try_to_wake_up(struct task_struct *p);
+int rpal_init_thread_pending(struct rpal_common_data *rcd);
+void rpal_free_thread_pending(struct rpal_common_data *rcd);
+int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock);
 #endif
kernel/sched/core.c +109 -0

122 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#18 lore raw

RFC v2 17/35 RPAL: add a mapping between fsbase and tasks

In reply to Bo Li

RPAL relies on the value of the fsbase register to determine whether a lazy switch is necessary. Therefore, a mapping between fsbase and tasks must be established.

This patch allows a thread to register its fsbase value when it is registered as a receiver. The rpal_find_next_task() interface is used to locate the receiver corresponding to a given fsbase value. Additionally, a new rpal_misidentify() interface has been added to check if the current fsbase value matches the current task. If they do not match, the task corresponding to the fsbase is identified, the RPAL_LAZY_SWITCHED_BIT flag is set, and the current task is recorded. The kernel can later use this flag and the recorded task to backtrack to the task before the lazy switch.

---
 arch/x86/rpal/core.c   | 85 ++++++++++++++++++++++++++++++++++++++++++
 arch/x86/rpal/thread.c | 57 +++++++++++++++++++++++++++-
 include/linux/rpal.h   | 15 ++++++++
 3 files changed, 156 insertions(+), 1 deletion(-)
arch/x86/rpal/core.c +85 -0
index c185a453c1b2..19c4ef38bca3 100644
@@ -7,6 +7,7 @@
  */
 
 #include <linux/rpal.h>
+#include <asm/fsgsbase.h>
 
 #include "internal.h"
 
@@ -33,12 +34,96 @@ static inline void rpal_unlock_cpu(struct task_struct *tsk)
 	}
 }
 
+
+static inline struct task_struct *rpal_get_sender_task(void)
+{
+	struct task_struct *next;
+
+	next = current->rpal_rd->sender;
+	current->rpal_rd->sender = NULL;
+
+	return next;
+}
+
+/*
+ * RPAL uses the value of fsbase (which libc uses as the base
+ * address for thread-local storage) to determine whether a
+ * lazy switch should be performed.
+ */
+static inline struct task_struct *rpal_misidentify(void)
+{
+	struct task_struct *next = NULL;
+	struct rpal_service *cur = rpal_current_service();
+	unsigned long fsbase;
+
+	fsbase = rdfsbase();
+	if (unlikely(!rpal_is_correct_address(cur, fsbase))) {
+		if (rpal_test_current_thread_flag(RPAL_LAZY_SWITCHED_BIT)) {
+			/* current is receiver, next is sender */
+			next = rpal_get_sender_task();
+			if (unlikely(next == NULL)) {
+				rpal_err("cannot find sender task\n");
+				goto out;
+			}
+		} else {
+			/* current is sender, next is receiver */
+			next = rpal_find_next_task(fsbase);
+			if (unlikely(next == NULL)) {
+				rpal_err(
+					"cannot find receiver task, fsbase: 0x%016lx\n",
+					fsbase);
+				goto out;
+			}
+			rpal_set_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT);
+			next->rpal_rd->sender = current;
+		}
+	}
+out:
+	return next;
+}
+
+struct task_struct *rpal_find_next_task(unsigned long fsbase)
+{
+	struct rpal_service *cur = rpal_current_service();
+	struct rpal_service *tgt;
+	struct task_struct *tsk = NULL;
+	int i;
+
+	tgt = rpal_get_mapped_service_by_addr(cur, fsbase);
+	if (unlikely(!tgt)) {
+		pr_debug("rpal debug: cannot find legal rs, fsbase: 0x%016lx\n",
+			 fsbase);
+		return NULL;
+	}
+	for (i = 0; i < RPAL_MAX_RECEIVER_NUM; ++i) {
+		if (tgt->fs_tsk_map[i].fsbase == fsbase) {
+			tsk = tgt->fs_tsk_map[i].tsk;
+			break;
+		}
+	}
+	rpal_put_service(tgt);
+
+	return tsk;
+}
+
+static bool check_hardware_features(void)
+{
+	if (!boot_cpu_has(X86_FEATURE_FSGSBASE)) {
+		rpal_err("no fsgsbase feature\n");
+		return false;
+	}
+	return true;
+}
+
 int __init rpal_init(void)
 {
 	int ret = 0;
 
 	rpal_cap = 0;
 
+	if (!check_hardware_features())
+		goto fail;
+
 	ret = rpal_service_init();
 	if (ret)
 		goto fail;
arch/x86/rpal/thread.c +56 -1

98 lines not on this page · raw message

include/linux/rpal.h +15 -0
index 5b115be14a55..45137770fac6 100644
@@ -80,6 +80,9 @@
 /* No more than 15 services can be requested due to limitation of MPK. */
 #define MAX_REQUEST_SERVICE 15
 
+/* We allow at most 16 receiver thread in one process */
+#define RPAL_MAX_RECEIVER_NUM  16
+
 enum {
 	RPAL_REQUEST_MAP,
 	RPAL_REVERSE_MAP,
@@ -100,6 +103,7 @@ enum rpal_task_flag_bits {
 	RPAL_SENDER_BIT,
 	RPAL_RECEIVER_BIT,
 	RPAL_CPU_LOCKED_BIT,
+	RPAL_LAZY_SWITCHED_BIT,
 };
 
 enum rpal_receiver_state {
@@ -145,6 +149,11 @@ struct rpal_poll_data {
 	wait_queue_head_t rpal_waitqueue;
 };
 
+struct rpal_fsbase_tsk_map {
+	unsigned long fsbase;
+	struct task_struct *tsk;
+};
+
 /*
  * Each RPAL process (a.k.a RPAL service) should have a pointer to
  * struct rpal_service in all its tasks' task_struct.
@@ -202,6 +211,9 @@ struct rpal_service {
 	/* Notify service is released by others */
 	struct rpal_poll_data rpd;
 
+	/* fsbase / pid map */
+	struct rpal_fsbase_tsk_map fs_tsk_map[RPAL_MAX_RECEIVER_NUM];
+
 	/* delayed service put work */
 	struct delayed_work delayed_put_work;
 
@@ -283,12 +295,14 @@ struct rpal_receiver_data {
 	struct rpal_common_data rcd;
 	struct rpal_shared_page *rsp;
 	struct rpal_receiver_call_context *rcc;
+	struct task_struct *sender;
 };
 
 struct rpal_sender_data {
 	struct rpal_common_data rcd;
 	struct rpal_shared_page *rsp;
 	struct rpal_sender_call_context *scc;
+	struct task_struct *receiver;
 };
 
 enum rpal_command_type {
@@ -465,6 +479,7 @@ struct rpal_service *rpal_get_mapped_service_by_id(struct rpal_service *rs,
 int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 					 unsigned long addr, int error_code);
 struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild);
+struct task_struct *rpal_find_next_task(unsigned long fsbase);
 
 extern void rpal_pick_mmap_base(struct mm_struct *mm,
 	struct rlimit *rlim_stack);
Signature · 1 line
2.20.1
author
#19 lore raw

RFC v2 18/35 sched: pick a specified task

In reply to Bo Li

When a lazy switch occurs, the kernel already gets the task_struct of the next task to switch to. However, the CFS does not provide an interface to explicitly specify the next task. Therefore, RPAL must implement its own mechanism to pick a specified task.

This patch introduces two interfaces, rpal_pick_next_task_fair() and rpal_pick_task_fair(), to achieve this functionality. These interfaces leverage the sched_entity of the target task to modify the CFS data structures directly. Additionally, the patch adapts to the SCHED_CORE feature by temporarily setting the highest weight for the specified task, ensuring that the core will select this task preferentially during scheduling decisions.

---
 kernel/sched/core.c  | 212 +++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/fair.c  | 109 ++++++++++++++++++++++
 kernel/sched/sched.h |   8 ++
 3 files changed, 329 insertions(+)
kernel/sched/core.c +212 -0
index a862bf4a0161..2e76376c5172 100644
@@ -11003,3 +11003,215 @@ void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx)
 		set_next_task(rq, ctx->p);
 }
 #endif	/* CONFIG_SCHED_CLASS_EXT */
+
+#ifdef CONFIG_RPAL
+#ifdef CONFIG_SCHED_CORE
+static inline struct task_struct *
+__rpal_pick_next_task(struct rq *rq, struct task_struct *prev,
+		      struct task_struct *next, struct rq_flags *rf)
+{
+	struct task_struct *p;
+
+	if (likely(prev->sched_class == &fair_sched_class &&
+		   next->sched_class == &fair_sched_class)) {
+		p = rpal_pick_next_task_fair(prev, next, rq, rf);
+		return p;
+	}
+
+	BUG();
+}
+
+static struct task_struct *rpal_pick_next_task(struct rq *rq,
+					       struct task_struct *prev,
+					       struct task_struct *next,
+					       struct rq_flags *rf)
+{
+	struct task_struct *p;
+	const struct cpumask *smt_mask;
+	bool fi_before = false;
+	bool core_clock_updated = (rq == rq->core);
+	unsigned long cookie;
+	int i, cpu, occ = 0;
+	struct rq *rq_i;
+	bool need_sync;
+
+	if (!sched_core_enabled(rq))
+		return __rpal_pick_next_task(rq, prev, next, rf);
+
+	cpu = cpu_of(rq);
+
+	/* Stopper task is switching into idle, no need core-wide selection. */
+	if (cpu_is_offline(cpu)) {
+		rq->core_pick = NULL;
+		return __rpal_pick_next_task(rq, prev, next, rf);
+	}
+
+	if (rq->core->core_pick_seq == rq->core->core_task_seq &&
+	    rq->core->core_pick_seq != rq->core_sched_seq &&
+		rq->core_pick) {
+		WRITE_ONCE(rq->core_sched_seq, rq->core->core_pick_seq);
+
+		/* ignore rq->core_pick, always pick next */
+		if (rq->core_pick == next) {
+			put_prev_task(rq, prev);
+			set_next_task(rq, next);
+
+			rq->core_pick = NULL;
+			goto out;
+		}
+	}
+
+	put_prev_task_balance(rq, prev, rf);
+
+	smt_mask = cpu_smt_mask(cpu);
+	need_sync = !!rq->core->core_cookie;
+
+	/* reset state */
+	rq->core->core_cookie = 0UL;
+	if (rq->core->core_forceidle_count) {
+		if (!core_clock_updated) {
+			update_rq_clock(rq->core);
+			core_clock_updated = true;
+		}
+		sched_core_account_forceidle(rq);
+		/* reset after accounting force idle */
+		rq->core->core_forceidle_start = 0;
+		rq->core->core_forceidle_count = 0;
+		rq->core->core_forceidle_occupation = 0;
+		need_sync = true;
+		fi_before = true;
+	}
+
+	rq->core->core_task_seq++;
+
+	if (!need_sync) {
+		next = rpal_pick_task_fair(rq, next);
+		if (!next->core_cookie) {
+			rq->core_pick = NULL;
+			/*
+			 * For robustness, update the min_vruntime_fi for
+			 * unconstrained picks as well.
+			 */
+			WARN_ON_ONCE(fi_before);
+			task_vruntime_update(rq, next, false);
+			goto out_set_next;
+		}
+	}
+
+	for_each_cpu_wrap(i, smt_mask, cpu) {
+		rq_i = cpu_rq(i);
+
+		if (i != cpu && (rq_i != rq->core || !core_clock_updated))
+			update_rq_clock(rq_i);
+
+		/* ignore prio, always pick next */
+		if (i == cpu)
+			rq_i->core_pick = rpal_pick_task_fair(rq, next);
+		else
+			rq_i->core_pick = pick_task(rq_i);
+	}
+
+	cookie = rq->core->core_cookie = next->core_cookie;
+
+	for_each_cpu(i, smt_mask) {
+		rq_i = cpu_rq(i);
+		p = rq_i->core_pick;
+
+		if (!cookie_equals(p, cookie)) {
+			p = NULL;
+			if (cookie)
+				p = sched_core_find(rq_i, cookie);
+			if (!p)
+				p = idle_sched_class.pick_task(rq_i);
+		}
+
+		rq_i->core_pick = p;
+
+		if (p == rq_i->idle) {
+			if (rq_i->nr_running) {
+				rq->core->core_forceidle_count++;
+				if (!fi_before)
+					rq->core->core_forceidle_seq++;
+			}
+		} else {
+			occ++;
+		}
+	}
+
+	if (schedstat_enabled() && rq->core->core_forceidle_count) {
+		rq->core->core_forceidle_start = rq_clock(rq->core);
+		rq->core->core_forceidle_occupation = occ;
+	}
+
+	rq->core->core_pick_seq = rq->core->core_task_seq;
+	WARN_ON_ONCE(next != rq->core_pick);
+	rq->core_sched_seq = rq->core->core_pick_seq;
+
+	for_each_cpu(i, smt_mask) {
+		rq_i = cpu_rq(i);
+
+		/*
+		 * An online sibling might have gone offline before a task
+		 * could be picked for it, or it might be offline but later
+		 * happen to come online, but its too late and nothing was
+		 * picked for it.  That's Ok - it will pick tasks for itself,
+		 * so ignore it.
+		 */
+		if (!rq_i->core_pick)
+			continue;
+
+		/*
+		 * Update for new !FI->FI transitions, or if continuing to be in !FI:
+		 * fi_before     fi      update?
+		 *  0            0       1
+		 *  0            1       1
+		 *  1            0       1
+		 *  1            1       0
+		 */
+		if (!(fi_before && rq->core->core_forceidle_count))
+			task_vruntime_update(rq_i, rq_i->core_pick,
+					     !!rq->core->core_forceidle_count);
+
+		rq_i->core_pick->core_occupation = occ;
+
+		if (i == cpu) {
+			rq_i->core_pick = NULL;
+			continue;
+		}
+
+		/* Did we break L1TF mitigation requirements? */
+		WARN_ON_ONCE(!cookie_match(next, rq_i->core_pick));
+
+		if (rq_i->curr == rq_i->core_pick) {
+			rq_i->core_pick = NULL;
+			continue;
+		}
+
+		resched_curr(rq_i);
+	}
+
+out_set_next:
+	set_next_task(rq, next);
+out:
+	if (rq->core->core_forceidle_count && next == rq->idle)
+		queue_core_balance(rq);
+
+	return next;
+}
+#else
+static inline struct task_struct *
+rpal_pick_next_task(struct rq *rq, struct task_struct *prev,
+		      struct task_struct *next, struct rq_flags *rf)
+{
+	struct task_struct *p;
+
+	if (likely(prev->sched_class == &fair_sched_class &&
+		   next->sched_class == &fair_sched_class)) {
+		p = rpal_pick_next_task_fair(prev, next, rq, rf);
+		return p;
+	}
+
+	BUG();
+}
+#endif
+#endif
kernel/sched/fair.c +109 -0

115 lines not on this page · raw message

kernel/sched/sched.h +8 -0
index c5a6a503eb6d..f8fd26b584c9 100644
@@ -2575,6 +2575,14 @@ static inline bool sched_fair_runnable(struct rq *rq)
 
 extern struct task_struct *pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf);
 extern struct task_struct *pick_task_idle(struct rq *rq);
+#ifdef CONFIG_RPAL
+extern struct task_struct *rpal_pick_task_fair(struct rq *rq,
+					       struct task_struct *next);
+extern struct task_struct *rpal_pick_next_task_fair(struct task_struct *prev,
+						    struct task_struct *next,
+						    struct rq *rq,
+						    struct rq_flags *rf);
+#endif
 
 #define SCA_CHECK		0x01
 #define SCA_MIGRATE_DISABLE	0x02
Signature · 1 line
2.20.1
author
#20 lore raw

RFC v2 19/35 RPAL: add lazy switch main logic

In reply to Bo Li

The implementation of lazy switch differs from a regular schedule() in three key aspects:

1. It occurs at the kernel entry with irq disabled.
2. The next task is explicitly pre-determined rather than selected by the scheduler.
3. User-space context (excluding general-purpose registers) remains unchanged across the switch.

This patch introduces the rpal_schedule() interface to address these requirements. Firstly, the rpal_schedule() skips irq enabling in finish_lock_switch(), preserving the irq-disabled state required during kernel entry. Secondly, the rpal_pick_next_task() interface is used to explicitly specify the target task, bypassing the default scheduler's decision-making process. Thirdly, non-general-purpose registers (e.g., FPU, vector units) are not restored during the switch, ensuring user space context remains intact. Handling of general-purpose registers will be addressed in a subsequent patch by RPAL before invoking rpal_schedule().

---
 arch/x86/kernel/process_64.c |  75 +++++++++++++++++++++
 include/linux/rpal.h         |   3 +
 kernel/sched/core.c          | 126 +++++++++++++++++++++++++++++++++++
 3 files changed, 204 insertions(+)
arch/x86/kernel/process_64.c +75 -0
index 4830e9215de7..efc3f238c486 100644
@@ -753,6 +753,81 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	return prev_p;
 }
 
+#ifdef CONFIG_RPAL
+__no_kmsan_checks
+__visible __notrace_funcgraph struct task_struct *
+__rpal_switch_to(struct task_struct *prev_p, struct task_struct *next_p)
+{
+	struct thread_struct *prev = &prev_p->thread;
+	struct thread_struct *next = &next_p->thread;
+	int cpu = smp_processor_id();
+
+	WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
+		     this_cpu_read(hardirq_stack_inuse));
+
+	/* no need to switch fpu */
+	/* __fpu_invalidate_fpregs_state() */
+	x86_task_fpu(prev_p)->last_cpu = -1;
+	/* fpregs_activate() */
+	__this_cpu_write(fpu_fpregs_owner_ctx, x86_task_fpu(next_p));
+	trace_x86_fpu_regs_activated(x86_task_fpu(next_p));
+	x86_task_fpu(next_p)->last_cpu = cpu;
+	set_tsk_thread_flag(prev_p, TIF_NEED_FPU_LOAD);
+	clear_tsk_thread_flag(next_p, TIF_NEED_FPU_LOAD);
+
+	/* no need to save fs */
+	savesegment(gs, prev_p->thread.gsindex);
+	if (static_cpu_has(X86_FEATURE_FSGSBASE))
+		prev_p->thread.gsbase = __rdgsbase_inactive();
+	else
+		save_base_legacy(prev_p, prev_p->thread.gsindex, GS);
+
+	load_TLS(next, cpu);
+
+	arch_end_context_switch(next_p);
+
+	savesegment(es, prev->es);
+	if (unlikely(next->es | prev->es))
+		loadsegment(es, next->es);
+
+	savesegment(ds, prev->ds);
+	if (unlikely(next->ds | prev->ds))
+		loadsegment(ds, next->ds);
+
+	/* no need to load fs */
+	if (static_cpu_has(X86_FEATURE_FSGSBASE)) {
+		if (unlikely(prev->gsindex || next->gsindex))
+			loadseg(GS, next->gsindex);
+
+		__wrgsbase_inactive(next->gsbase);
+	} else {
+		load_seg_legacy(prev->gsindex, prev->gsbase, next->gsindex,
+				next->gsbase, GS);
+	}
+
+	/* skip pkru load as we will use pkru in RPAL */
+
+	this_cpu_write(current_task, next_p);
+	this_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p));
+
+	/* no need to load fpu */
+
+	update_task_stack(next_p);
+	switch_to_extra(prev_p, next_p);
+
+	if (static_cpu_has_bug(X86_BUG_SYSRET_SS_ATTRS)) {
+		unsigned short ss_sel;
+
+		savesegment(ss, ss_sel);
+		if (ss_sel != __KERNEL_DS)
+			loadsegment(ss, __KERNEL_DS);
+	}
+	resctrl_sched_in(next_p);
+
+	return prev_p;
+}
+#endif
+
 void set_personality_64bit(void)
 {
 	/* inherit personality from parent */
include/linux/rpal.h +3 -0
index 45137770fac6..0813db4552c0 100644
@@ -487,4 +487,7 @@ int rpal_try_to_wake_up(struct task_struct *p);
 int rpal_init_thread_pending(struct rpal_common_data *rcd);
 void rpal_free_thread_pending(struct rpal_common_data *rcd);
 int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock);
+void rpal_schedule(struct task_struct *next);
+asmlinkage struct task_struct *
+__rpal_switch_to(struct task_struct *prev_p, struct task_struct *next_p);
 #endif
kernel/sched/core.c +126 -0

142 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#21 lore raw

RFC v2 20/35 RPAL: add rpal_ret_from_lazy_switch

In reply to Bo Li

After lazy switch the task before the lazy switch will lose its user mode context (which is passed to the task after the lazy switch). Therefore, RPAL needs to handle the issue of the previous task losing its user mode context.

After the lazy switch occurs, the sender can resume execution in two ways. One way is to be scheduled by the scheduler. In this case, RPAL handles this issue in a manner similar to ret_from_fork. the sender will enter rpal_ret_from_lazy_switch through the constructed stack frame by lazy switchto execute the return logic and finally return to the pre-defined user mode (referred to as "kernel return"). The other way is to be switched back to by the receiver through another lazy switch. In this case, the receiver will pass the user mode context to the sender, so there is no need to construct a user mode context for the sender. And the receiver can return to the user mode through the kernel return method.

rpal_ret_from_lazy_switch primarily handles scheduler cleanup work, similar to schedule_tail(), but does not perform set_child_tid-otherwise, it might cause set_child_tid to be executed repeatedly. It then calls rpal_kernel_ret(), which is primarily used to set the states of the sender and receiver and attempt to unlock the CPU. Finally, it performs syscall cleanup work and returns to user mode.

---
 arch/x86/entry/entry_64.S | 23 ++++++++++++++++++++
 arch/x86/rpal/core.c      | 45 +++++++++++++++++++++++++++++++++++++--
 include/linux/rpal.h      |  5 ++++-
 kernel/sched/core.c       | 25 +++++++++++++++++++++-
 4 files changed, 94 insertions(+), 4 deletions(-)
arch/x86/entry/entry_64.S +23 -0
index ed04a968cc7d..13b4d0684575 100644
@@ -169,6 +169,29 @@ SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL)
 	int3
 SYM_CODE_END(entry_SYSCALL_64)
 
+#ifdef CONFIG_RPAL
+SYM_CODE_START(rpal_ret_from_lazy_switch)
+	UNWIND_HINT_END_OF_STACK
+	ANNOTATE_NOENDBR
+	movq	%rax, %rdi
+	call	rpal_schedule_tail
+
+	movq	%rsp, %rdi
+	call	rpal_kernel_ret
+
+	movq	%rsp, %rdi
+	call	syscall_exit_to_user_mode	/* returns with IRQs disabled */
+
+	UNWIND_HINT_REGS
+#ifdef CONFIG_X86_FRED
+	ALTERNATIVE "jmp swapgs_restore_regs_and_return_to_usermode", \
+		    "jmp asm_fred_exit_user", X86_FEATURE_FRED
+#else
+	jmp	swapgs_restore_regs_and_return_to_usermode
+#endif
+SYM_CODE_END(rpal_ret_from_lazy_switch)
+#endif
+
 /*
  * %rdi: prev task
  * %rsi: next task
arch/x86/rpal/core.c +43 -2

63 lines not on this page · raw message

include/linux/rpal.h +4 -1
index 0813db4552c0..01b582fa821e 100644
@@ -480,14 +480,17 @@ int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 					 unsigned long addr, int error_code);
 struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild);
 struct task_struct *rpal_find_next_task(unsigned long fsbase);
+void rpal_kernel_ret(struct pt_regs *regs);
 
 extern void rpal_pick_mmap_base(struct mm_struct *mm,
 	struct rlimit *rlim_stack);
 int rpal_try_to_wake_up(struct task_struct *p);
 int rpal_init_thread_pending(struct rpal_common_data *rcd);
 void rpal_free_thread_pending(struct rpal_common_data *rcd);
-int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock);
+int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock,
+	bool is_kernel_ret);
 void rpal_schedule(struct task_struct *next);
 asmlinkage struct task_struct *
 __rpal_switch_to(struct task_struct *prev_p, struct task_struct *next_p);
+asmlinkage __visible void rpal_schedule_tail(struct task_struct *prev);
 #endif
kernel/sched/core.c +24 -1
index 760d88458b39..0f9343698198 100644
@@ -3181,7 +3181,8 @@ void rpal_free_thread_pending(struct rpal_common_data *rcd)
 /*
  * CPU lock is forced and all cpumask will be ignored by RPAL temporary.
  */
-int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock)
+int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock,
+							 bool is_kernel_ret)
 {
 	const struct cpumask *cpu_valid_mask = cpu_active_mask;
 	struct set_affinity_pending *pending = p->rpal_cd->pending;
@@ -3210,6 +3211,9 @@ int rpal_set_cpus_allowed_ptr(struct task_struct *p, bool is_lock)
 		rpal_clear_task_thread_flag(p, RPAL_CPU_LOCKED_BIT);
 	}
 
+	if (is_kernel_ret)
+		return __set_cpus_allowed_ptr_locked(p, &ac, rq, &rf);
+
 	update_rq_clock(rq);
 
 	if (cpumask_equal(&p->cpus_mask, ac.new_mask))
@@ -11011,6 +11015,25 @@ void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx)
 #endif	/* CONFIG_SCHED_CLASS_EXT */
 
 #ifdef CONFIG_RPAL
+asmlinkage __visible void rpal_schedule_tail(struct task_struct *prev)
+	__releases(rq->lock)
+{
+	/*
+	 * New tasks start with FORK_PREEMPT_COUNT, see there and
+	 * finish_task_switch() for details.
+	 *
+	 * finish_task_switch() will drop rq->lock() and lower preempt_count
+	 * and the preempt_enable() will end up enabling preemption (on
+	 * PREEMPT_COUNT kernels).
+	 */
+
+	finish_task_switch(prev);
+	trace_sched_exit_tp(true, CALLER_ADDR0);
+	preempt_enable();
+
+	calculate_sigpending();
+}
+
 static struct rq *rpal_finish_task_switch(struct task_struct *prev)
 	__releases(rq->lock)
 {
Signature · 1 line
2.20.1
author
#22 lore raw

RFC v2 21/35 RPAL: add kernel entry handling for lazy switch

In reply to Bo Li

At the kernel entry point, RPAL performs a lazy switch. Therefore, it is necessary to hook all kernel entry points to execute the logic related to the lazy switch. At the kernel entry, apart from some necessary operations related to the lazy switch (such as ensuring that the general-purpose registers remain unchanged before and after the lazy switch), the task before the lazy switch will lose its user mode context (which is passed to the task after the lazy switch). Therefore, the kernel entry also needs to handle the issue of the previous task losing its user mode context.

This patch hooks all locations where the transition from user mode to kernel mode occurs, including entry_SYSCALL_64, error_entry, and asm_exc_nmi. When the kernel detects a mismatch between the kernel-mode and user mode contexts, it executes the logic related to the lazy switch. Taking the switch from the sender to the receiver as an example, the receiver thread is first locked to the CPU where the sender is located. Then, the receiver thread in the CALL state is woken up through rpal_try_to_wake_up(). The general purpose register state (pt_regs) of the sender is copied to the receiver, and rpal_schedule() is executed to complete the lazy switch. Regarding the issue of the sender losing its context, the kernel loads the pre-saved user mode context of the sender into the sender's pt_regs and constructs the kernel stack frame of the sender in a manner similar to the fork operation.

The handling of the switch from the receiver to the sender is similar, except that the receiver will be unlocked from the current CPU, and the receiver can only return to the user mode through the kernel return method.

---
 arch/x86/entry/entry_64.S     | 137 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/asm-offsets.c |   3 +
 arch/x86/rpal/core.c          | 137 ++++++++++++++++++++++++++++++++++
 include/linux/rpal.h          |   6 ++
 4 files changed, 283 insertions(+)
arch/x86/entry/entry_64.S +137 -0
index 13b4d0684575..59c38627510d 100644
@@ -118,6 +118,20 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)
 	UNTRAIN_RET
 	CLEAR_BRANCH_HISTORY
 
+#ifdef CONFIG_RPAL
+	/*
+	 * We first check if it is a RPAL sender/receiver with
+	 * current->rpal_cd. For non-RPAL task, we just skip it.
+	 * For rpal task, We may need to check if it needs to do
+	 * lazy switch.
+	 */
+	movq	PER_CPU_VAR(current_task), %r13
+	movq	TASK_rpal_cd(%r13), %rax
+	testq	%rax, %rax
+	jz		_do_syscall
+	jmp 	do_rpal_syscall
+_do_syscall:
+#endif
 	call	do_syscall_64		/* returns with IRQs disabled */
 
 	/*
@@ -190,6 +204,101 @@ SYM_CODE_START(rpal_ret_from_lazy_switch)
 	jmp	swapgs_restore_regs_and_return_to_usermode
 #endif
 SYM_CODE_END(rpal_ret_from_lazy_switch)
+
+/* return address offset of stack frame */
+#define RPAL_FRAME_RET_ADDR_OFFSET -56
+
+SYM_CODE_START(do_rpal_syscall)
+	movq	%rsp, %r14
+	call	rpal_syscall_64_context_switch
+	testq   %rax, %rax
+	jz		1f
+
+	/*
+	 * When we come here, everything but stack switching is finished.
+	 * This makes current task use another task's kernel stack. Thus,
+	 * we need to do stack switching here.
+	 *
+	 * At the meanwhile, the previous task's stack content is corrupted,
+	 * we also need to rebuild its stack frames, so that it will jump to
+	 * rpal_ret_from_lazy_switch when it is scheduled in. This is inspired
+	 * by ret_from_fork.
+	 */
+	movq    TASK_threadsp(%rax), %rsp
+#ifdef CONFIG_STACKPROTECTOR
+	movq	TASK_stack_canary(%rax), %rbx
+	movq	%rbx, PER_CPU_VAR(__stack_chk_guard)
+#endif
+	/* rebuild src's frame */
+	movq	$rpal_ret_from_lazy_switch, -8(%r14)
+	leaq	RPAL_FRAME_RET_ADDR_OFFSET(%r14), %rbx
+	movq	%rbx, TASK_threadsp(%r13)
+
+	movq	%r13, %rdi
+	/*
+	 * Everything of task switch is done, but we still need to do
+	 * a little extra things for lazy switch.
+	 */
+	call	rpal_lazy_switch_tail
+
+1:
+	movq	ORIG_RAX(%rsp), %rsi
+	movq	%rsp, %rdi
+	jmp		_do_syscall
+SYM_CODE_END(do_rpal_syscall)
+
+SYM_CODE_START(do_rpal_error)
+	popq	%r12
+	movq	%rax, %rsp
+	movq	%rax, %r14
+	movq	%rax, %rdi
+	call	rpal_exception_context_switch
+	testq   %rax, %rax
+	jz		1f
+
+	movq	TASK_threadsp(%rax), %rsp
+	ENCODE_FRAME_POINTER
+#ifdef CONFIG_STACKPROTECTOR
+	movq	TASK_stack_canary(%rax), %rbx
+	movq	%rbx, PER_CPU_VAR(__stack_chk_guard)
+#endif
+	/* rebuild src's frame */
+	movq	$rpal_ret_from_lazy_switch, -8(%r14)
+	leaq	RPAL_FRAME_RET_ADDR_OFFSET(%r14), %rbx
+	movq	%rbx, TASK_threadsp(%r13)
+
+	movq	%r13, %rdi
+	call	rpal_lazy_switch_tail
+1:
+	movq	%rsp, %rax
+	pushq	%r12
+	jmp		_do_error
+SYM_CODE_END(do_rpal_error)
+
+SYM_CODE_START(do_rpal_nmi)
+	movq	%rsp, %r14
+	movq	%rsp, %rdi
+	call	rpal_nmi_context_switch
+	testq   %rax, %rax
+	jz		1f
+
+	movq    TASK_threadsp(%rax), %rsp
+	ENCODE_FRAME_POINTER
+#ifdef CONFIG_STACKPROTECTOR
+	movq	TASK_stack_canary(%rax), %rbx
+	movq	%rbx, PER_CPU_VAR(__stack_chk_guard)
+#endif
+	/* rebuild src's frame */
+	movq	$rpal_ret_from_lazy_switch, -8(%r14)
+	leaq	RPAL_FRAME_RET_ADDR_OFFSET(%r14), %rbx
+	movq	%rbx, TASK_threadsp(%r13)
+
+	movq	%r13, %rdi
+	call	rpal_lazy_switch_tail
+
+1:
+	jmp		_do_nmi
+SYM_CODE_END(do_rpal_nmi)
 #endif
 
 /*
@@ -1047,7 +1156,22 @@ SYM_CODE_START(error_entry)
 
 	leaq	8(%rsp), %rdi			/* arg0 = pt_regs pointer */
 	/* Put us onto the real thread stack. */
+#ifdef CONFIG_RPAL
+	call sync_regs
+	/*
+	 * Check whether we need to perform lazy switch after we
+	 * switch to the real thread stack.
+	 */
+	movq	PER_CPU_VAR(current_task), %r13
+	movq	TASK_rpal_cd(%r13), %rdi
+	testq	%rdi, %rdi
+	jz		_do_error
+	jmp 	do_rpal_error
+_do_error:
+	RET
+#else
 	jmp	sync_regs
+#endif
 
 	/*
 	 * There are two places in the kernel that can potentially fault with
@@ -1206,6 +1330,19 @@ SYM_CODE_START(asm_exc_nmi)
 	IBRS_ENTER
 	UNTRAIN_RET
 
+#ifdef CONFIG_RPAL
+	/*
+	 * Check whether we need to perform lazy switch only when
+	 * we come from userspace.
+	 */
+	movq	PER_CPU_VAR(current_task), %r13
+	movq	TASK_rpal_cd(%r13), %rax
+	testq	%rax, %rax
+	jz		_do_nmi
+	jmp 	do_rpal_nmi
+_do_nmi:
+#endif
+
 	/*
 	 * At this point we no longer need to worry about stack damage
 	 * due to nesting -- we're on the normal thread stack and we're
arch/x86/kernel/asm-offsets.c +3 -0
index 6259b474073b..010202c31b37 100644
@@ -46,6 +46,9 @@ static void __used common(void)
 #ifdef CONFIG_STACKPROTECTOR
 	OFFSET(TASK_stack_canary, task_struct, stack_canary);
 #endif
+#ifdef CONFIG_RPAL
+	OFFSET(TASK_rpal_cd, task_struct, rpal_cd);
+#endif
 
 	BLANK();
 	OFFSET(pbe_address, pbe, address);
arch/x86/rpal/core.c +137 -0

167 lines not on this page · raw message

include/linux/rpal.h +6 -0
index 01b582fa821e..b24176f3f245 100644
@@ -479,7 +479,13 @@ struct rpal_service *rpal_get_mapped_service_by_id(struct rpal_service *rs,
 int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 					 unsigned long addr, int error_code);
 struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild);
+__visible struct task_struct *
+rpal_syscall_64_context_switch(struct pt_regs *regs, unsigned long nr);
+__visible struct task_struct *
+rpal_exception_context_switch(struct pt_regs *regs);
+__visible struct task_struct *rpal_nmi_context_switch(struct pt_regs *regs);
 struct task_struct *rpal_find_next_task(unsigned long fsbase);
+void rpal_lazy_switch_tail(struct task_struct *tsk);
 void rpal_kernel_ret(struct pt_regs *regs);
 
 extern void rpal_pick_mmap_base(struct mm_struct *mm,
Signature · 1 line
2.20.1
author
#23 lore raw

RFC v2 22/35 RPAL: rebuild receiver state

In reply to Bo Li

When an RPAL call occurs, the sender modifies the receiver's state. If the sender exits abnormally after modifying the state or encounters an unhandled page fault and returns to a recovery point, the receiver's state will remain as modified by the sender (e.g., in the CALL state). Since the sender may have exited, the lazy switch will not occur, leaving the receiver unrecoverable (unable to be woken up via try_to_wake_up()). Therefore, the kernel must ensure the receiver's state remains valid in these cases.

This patch addresses this by rebuild receiver's state during unhandled page faults or sender exits. The kernel detect the fsbase value recorded by the sender and use the fsbase value to locate the corresponding receiver. Then kernel checking if the receiver is in the CALL state set by the sender (using sender_id and service_id carried in the CALL state). If true, transitioning the receiver from CALL to WAIT state and notifying the receiver via sender_state that the RPAL call has completed.

This ensures that even if the sender fails, the receiver can recover and resume normal operation by resetting its state and avoiding permanent blocking.

---
 arch/x86/rpal/thread.c | 44 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)
arch/x86/rpal/thread.c +43 -1
index db3b13ff82be..02c1a9c22dd7 100644
@@ -224,6 +224,45 @@ int rpal_unregister_receiver(void)
 	return ret;
 }
 
+/* sender may corrupt receiver's state if unexpectedly exited, rebuild it */
+static void rpal_rebuild_receiver_context_on_exit(void)
+{
+	struct task_struct *receiver = NULL;
+	struct rpal_sender_data *rsd = current->rpal_sd;
+	struct rpal_sender_call_context *scc = rsd->scc;
+	struct rpal_receiver_data *rrd;
+	struct rpal_receiver_call_context *rcc;
+	unsigned long fsbase;
+	int state = rpal_build_call_state(rsd);
+
+	if (scc->ec.magic != RPAL_ERROR_MAGIC)
+		goto out;
+
+	fsbase = scc->ec.fsbase;
+	if (rpal_is_correct_address(rpal_current_service(), fsbase))
+		goto out;
+
+	receiver = rpal_find_next_task(fsbase);
+	if (!receiver)
+		goto out;
+
+	rrd = receiver->rpal_rd;
+	if (!rrd)
+		goto out;
+
+	rcc = rrd->rcc;
+
+	if (atomic_read(&rcc->receiver_state) == state) {
+		atomic_cmpxchg(&rcc->sender_state, RPAL_SENDER_STATE_CALL,
+			       RPAL_SENDER_STATE_KERNEL_RET);
+		atomic_cmpxchg(&rcc->receiver_state, state,
+			       RPAL_RECEIVER_STATE_WAIT);
+	}
+
+out:
+	return;
+}
+
 int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 					 unsigned long addr, int error_code)
 {
@@ -232,6 +271,7 @@ int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 		unsigned long erip, ersp;
 		int magic;
 
+		rpal_rebuild_receiver_context_on_exit();
 		erip = scc->ec.erip;
 		ersp = scc->ec.ersp;
 		magic = scc->ec.magic;
@@ -249,8 +289,10 @@ int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 
 void exit_rpal_thread(void)
 {
-	if (rpal_test_current_thread_flag(RPAL_SENDER_BIT))
+	if (rpal_test_current_thread_flag(RPAL_SENDER_BIT)) {
+		rpal_rebuild_receiver_context_on_exit();
 		rpal_unregister_sender();
+	}
 
 	if (rpal_test_current_thread_flag(RPAL_RECEIVER_BIT))
 		rpal_unregister_receiver();
Signature · 1 line
2.20.1
author
#24 lore raw

RFC v2 23/35 RPAL: resume cpumask when fork

In reply to Bo Li

After a lazy switch occurs, RPAL locks the receiver to the current CPU by modifying its cpumask. If the receiver performs a fork operation at this point, the kernel will copy the modified cpumask to the new task, causing the new task to be permanently locked on the current CPU.

This patch addresses this issue by detecting whether the original task is locked to the current CPU by RPAL during fork. If locked, assigning the cpumask that existed before the lazy switch to the new task. This ensures the new task will not be locked to the current CPU.

---
 arch/x86/kernel/process.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/x86/kernel/process.c +18 -0
index c1d2dac72b9c..be8845e2ca4d 100644
@@ -29,6 +29,7 @@
 #include <trace/events/power.h>
 #include <linux/hw_breakpoint.h>
 #include <linux/entry-common.h>
+#include <linux/rpal.h>
 #include <asm/cpu.h>
 #include <asm/cpuid/api.h>
 #include <asm/apic.h>
@@ -88,6 +89,19 @@ EXPORT_PER_CPU_SYMBOL(cpu_tss_rw);
 DEFINE_PER_CPU(bool, __tss_limit_invalid);
 EXPORT_PER_CPU_SYMBOL_GPL(__tss_limit_invalid);
 
+#ifdef CONFIG_RPAL
+static void rpal_fix_task_dump(struct task_struct *dst,
+			      struct task_struct *src)
+{
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&src->pi_lock, flags);
+	if (rpal_test_task_thread_flag(src, RPAL_CPU_LOCKED_BIT))
+		cpumask_copy(&dst->cpus_mask, &src->rpal_cd->old_mask);
+	raw_spin_unlock_irqrestore(&src->pi_lock, flags);
+}
+#endif
+
 /*
  * this gets called so that we can store lazy state into memory and copy the
  * current task into the new thread.
@@ -100,6 +114,10 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 #ifdef CONFIG_VM86
 	dst->thread.vm86 = NULL;
 #endif
+#ifdef CONFIG_RPAL
+	if (src->rpal_rs)
+		rpal_fix_task_dump(dst, src);
+#endif
 
 	return 0;
 }
Signature · 1 line
2.20.1
#25 lore raw
In reply to Bo Li

Bo,

You have outstanding feedback on your v1 from me and Dave Hansen. I'm not quite sure why you're sending a v2 without responding to that.

This isn't how the upstream kernel works...

Thanks, Lorenzo

Quoted 432 lines · Bo Li

Changelog:

v2:
- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

link to v1:
https://lore.kernel.org/lkml/CAP2HCOmAkRVTci0ObtyW=3v6GFOrt9zCn2NwLUbZ+Di49xkBiw@mail.gmail.com/

--------------------------------------------------------------------------

# Introduction

We mainly apply RPAL to the service mesh architecture widely adopted in modern cloud-native data centers. Before the rise of the service mesh architecture, network functions were usually integrated into monolithic applications as libraries, and the main business programs invoked them through function calls. However, to facilitate the independent development and operation and maintenance of the main business programs and network functions, the service mesh removed the network functions from the main business programs and made them independent processes (called sidecars). Inter-process communication (IPC) is used for interaction between the main business program and the sidecar, and the introduced inter-process communication has led to a sharp increase in resource consumption in cloud-native data centers, and may even occupy more than 10% of the CPU of the entire microservice cluster.

To achieve the efficient function call mechanism of the monolithic architecture under the service mesh architecture, we introduced the RPAL (Running Process As Library) architecture, which implements the sharing of the virtual address space of processes and the switching threads in user mode. Through the analysis of the service mesh architecture, we found that the process memory isolation between the main business program and the sidecar is not particularly important because they are split from one application and were an integral part of the original monolithic application. It is more important for the two processes to be independent of each other because they need to be independently developed and maintained to ensure the architectural advantages of the service mesh. Therefore, RPAL breaks the isolation between processes while preserving the independence between them. We think that RPAL can also be applied to other scenarios featuring sidecar-like architectures, such as distributed file storage systems in LLM infra.

In RPAL architecture, multiple processes share a virtual address space, so this architecture can be regarded as an advanced version of the Linux shared memory mechanism:

1. Traditional shared memory requires two processes to negotiate to ensure the mapping of the same piece of memory. In RPAL architecture, two RPAL processes still need to reach a consensus before they can successfully invoke the relevant system calls of RPAL to share the virtual address space.
2. Traditional shared memory only shares part of the data. However, in RPAL architecture, processes that have established an RPAL communication relationship share a virtual address space, and all user memory (such as data segments and code segments) of each RPAL process is shared among these processes. However, a process cannot access the memory of other processes at any time. We use the MPK mechanism to ensure that the memory of other processes can only be accessed when special RPAL functions are called. Otherwise, a page fault will be triggered.
3. In RPAL architecture, to ensure the consistency of the execution context of the shared code (such as the stack and thread local storage), we further implement the thread context switching in user mode based on the ability to share the virtual address space of different processes, enabling the threads of different processes to directly perform fast switching in user mode without falling into kernel mode for slow switching.

# Background

In traditional inter-process communication (IPC) scenarios, Unix domain sockets are commonly used in conjunction with the epoll() family for event multiplexing. IPC operations involve system calls on both the data and control planes, thereby imposing a non-trivial overhead on the interacting processes. Even when shared memory is employed to optimize the data plane, two data copies still remain. Specifically, data is initially copied from a process's private memory space into the shared memory area, and then it is copied from the shared memory into the private memory of another process.

This poses a question: Is it possible to reduce the overhead of IPC with only minimal modifications at the application level? To address this, we observed that the functionality of IPC, which encompasses data transfer and invocation of the target thread, is similar to a function call, where arguments are passed and the callee function is invoked to process them. Inspired by this analogy, we introduce RPAL (Run Process As Library), a framework designed to enable one process to invoke another as if making a local function call, all without going through the kernel.

# Design

First, let’s formalize RPAL’s core objectives:

1. Data-plane efficiency: Reduce the number of data copies from two (in the shared memory solution) to one.
2. Control-plane optimization: Eliminate the overhead of system calls and kernel's thread switches.
3. Application compatibility: Minimize the modifications to existing applications that utilize Unix domain sockets and the epoll() family.

To attain the first objective, processes that use RPAL share the same virtual address space. So one process can access another's data directly via a data pointer. This means data can be transferred from one process to another with just one copy operation.

To meet the second goal, RPAL relies on the shared address space to do lightweight context switching in user space, which we call an "RPAL call". This allows one process to execute another process's code just like a local function call.

To achieve the third target, RPAL stays compatible with the epoll family of functions, like epoll_create(), epoll_wait(), and epoll_ctl(). If an application uses epoll for IPC, developers can switch to RPAL with just a few small changes. For instance, you can just replace epoll_wait() with rpal_epoll_wait(). The basic epoll procedure, where a process waits for another to write to a monitored descriptor using an epoll file descriptor, still works fine with RPAL.

## Address space sharing

For address space sharing, RPAL partitions the entire userspace virtual address space and allocates non-overlapping memory ranges to each process. On x86_64 architectures, RPAL uses a memory range size covered by a single PUD (Page Upper Directory) entry, which is 512GB. This restricts each process’s virtual address space to 512GB on x86_64, sufficient for most applications in our scenario. The rationale is straightforward:
address space sharing can be simply achieved by copying the PUD from one process’s page table to another’s. So one process can directly use the data pointer to access another's memory.

 |------------| <- 0
 |------------| <- 512 GB
 |  Process A |
 |------------| <- 2*512 GB
 |------------| <- n*512 GB
 |  Process B |
 |------------| <- (n+1)*512 GB
 |------------| <- STACK_TOP
 |  Kernel    |
 |------------|

## RPAL call

We refer to the lightweight userspace context switching mechanism as RPAL call. It enables the caller (or sender) thread of one process to directly switch to the callee (or receiver) thread of another process.

When Process A’s caller thread initiates an RPAL call to Process B’s callee thread, the CPU saves the caller’s context and loads the callee’s context. This enables direct userspace control flow transfer from the caller to the callee. After the callee finishes data processing, the CPU saves Process B’s callee context and switches back to Process A’s caller context, completing a full IPC cycle.

 |------------|                |---------------------|
 |  Process A |                |  Process B          |
 | |-------|  |                | |-------|           |
 | | caller| --- RPAL call --> | | callee|    handle |
 | | thread| <------------------ | thread| -> event  |
 | |-------|  |                | |-------|           |
 |------------|                |---------------------|

# Security and compatibility with kernel subsystems

## Memory protection between processes

Since processes using RPAL share the address space, unintended cross-process memory access may occur and corrupt the data of another process. To mitigate this, we leverage Memory Protection Keys (MPK) on x86 architectures.

MPK assigns 4 bits in each page table entry to a "protection key", which is paired with a userspace register (PKRU). The PKRU register defines access permissions for memory regions protected by specific keys (for detailed implementation, refer to the kernel documentation "Memory Protection Keys"). With MPK, even though the address space is shared among processes, cross-process access is restricted: a process can only access the memory protected by a key if its PKRU register is configured with the corresponding permission. This ensures that processes cannot access each other’s memory unless an explicit PKRU configuration is set.

## Page fault handling and TLB flushing

Due to the shared address space architecture, both page fault handling and TLB flushing require careful consideration. For instance, when Process A accesses Process B’s memory, a page fault may occur in Process A's context, but the faulting address belongs to Process B. In this case, we must pass Process B's mm_struct to the page fault handler.

TLB flushing is more complex. When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

## Lazy switch of kernel context

In RPAL, a mismatch may arise between the user context and the kernel context. The RPAL call is designed solely to switch the user context, leaving the kernel context unchanged. For instance, when a RPAL call takes place, transitioning from caller thread to callee thread, and subsequently a system call is initiated within callee thread, the kernel will incorrectly utilize the caller's kernel context (such as the kernel stack) to process the system call.

To resolve context mismatch issues, a kernel context switch is triggered at the kernel entry point when the callee initiates a syscall or an exception/interrupt occurs. This mechanism ensures context consistency before processing system calls, interrupts, or exceptions. We refer to this kernel context switch as a "lazy switch" because it defers the switching operation from the traditional thread switch point to the next kernel entry point.

Lazy switch should be minimized as much as possible, as it significantly degrades performance. We currently utilize RPAL in an RPC framework, in which the RPC sender thread relies on the RPAL call to invoke the RPC receiver thread entirely in user space. In most cases, the receiver thread is free of system calls and the code execution time is relatively short. This characteristic effectively reduces the probability of a lazy switch occurring.

## Time slice correction

After an RPAL call, the callee's user mode code executes. However, the kernel incorrectly attributes this CPU time to the caller due to the unchanged kernel context.

To resolve this, we use the Time Stamp Counter (TSC) register to measure CPU time consumed by the callee thread in user space. The kernel then uses this user-reported timing data to adjust the CPU accounting for both the caller and callee thread, similar to how CPU steal time is implemented.

## Process recovery

Since processes can access each other’s memory, there is a risk that the target process’s memory may become invalid at the access time (e.g., if the target process has exited unexpectedly). The kernel must handle such cases; otherwise, the accessing process could be terminated due to failures originating from another process.

To address this issue, each thread of the process should pre-establish a recovery point when accessing the memory of other processes. When such an invalid access occurs, the thread traps into the kernel. Inside the page fault handler, the kernel restores the user context of the thread to the recovery point. This mechanism ensures that processes maintain mutual independence, preventing cascading failures caused by cross-process memory issues.

# Performance

To quantify the performance improvements driven by RPAL, we measured latency both before and after its deployment. Experiments were conducted on a server equipped with two Intel(R) Xeon(R) Platinum 8336C CPUs (2.30 GHz) and 1 TB of memory. Latency was defined as the duration from when the client thread initiates a message to when the server thread is invoked and receives it.

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

We have applied RPAL to an RPC framework that is widely used in our data center. With RPAL, we have successfully achieved up to 15.5% reduction in the CPU utilization of processes in real-world microservice scenario. The gains primarily stem from minimizing control plane overhead through the utilization of userspace context switches. Additionally, by leveraging address space sharing, the number of memory copies is significantly reduced.

# Future Work

Currently, RPAL requires the MPK (Memory Protection Key) hardware feature, which is supported by a range of Intel CPUs. For AMD architectures, MPK is supported only on the latest processor, specifically, 3th Generation AMD EPYC™ Processors and subsequent generations. Patch sets that extend RPAL support to systems lacking MPK hardware will be provided later.

Accompanying test programs are also provided in the samples/rpal/ directory. And the user-mode RPAL library, which realizes user-space RPAL call, is in the samples/rpal/librpal directory.

We hope to get some community discussions and feedback on RPAL's optimization approaches and architecture.

Look forward to your comments.

Bo Li (35):
Kbuild: rpal support
RPAL: add struct rpal_service
RPAL: add service registration interface
RPAL: add member to task_struct and mm_struct
RPAL: enable virtual address space partitions
RPAL: add user interface
RPAL: enable shared page mmap
RPAL: enable sender/receiver registration
RPAL: enable address space sharing
RPAL: allow service enable/disable
RPAL: add service request/release
RPAL: enable service disable notification
RPAL: add tlb flushing support
RPAL: enable page fault handling
RPAL: add sender/receiver state
RPAL: add cpu lock interface
RPAL: add a mapping between fsbase and tasks
sched: pick a specified task
RPAL: add lazy switch main logic
RPAL: add rpal_ret_from_lazy_switch
RPAL: add kernel entry handling for lazy switch
RPAL: rebuild receiver state
RPAL: resume cpumask when fork
RPAL: critical section optimization
RPAL: add MPK initialization and interface
RPAL: enable MPK support
RPAL: add epoll support
RPAL: add rpal_uds_fdmap() support
RPAL: fix race condition in pkru update
RPAL: fix pkru setup when fork
RPAL: add receiver waker
RPAL: fix unknown nmi on AMD CPU
RPAL: enable time slice correction
RPAL: enable fast epoll wait
samples/rpal: add RPAL samples

 arch/x86/Kbuild                               |    2 +
 arch/x86/Kconfig                              |    2 +
 arch/x86/entry/entry_64.S                     |  160 ++
 arch/x86/events/amd/core.c                    |   14 +
 arch/x86/include/asm/pgtable.h                |   25 +
 arch/x86/include/asm/pgtable_types.h          |   11 +
 arch/x86/include/asm/tlbflush.h               |   10 +
 arch/x86/kernel/asm-offsets.c                 |    3 +
 arch/x86/kernel/cpu/common.c                  |    8 +-
 arch/x86/kernel/fpu/core.c                    |    8 +-
 arch/x86/kernel/nmi.c                         |   20 +
 arch/x86/kernel/process.c                     |   25 +-
 arch/x86/kernel/process_64.c                  |  118 +
 arch/x86/mm/fault.c                           |  271 ++
 arch/x86/mm/mmap.c                            |   10 +
 arch/x86/mm/tlb.c                             |  172 ++
 arch/x86/rpal/Kconfig                         |   21 +
 arch/x86/rpal/Makefile                        |    6 +
 arch/x86/rpal/core.c                          |  477 ++++
 arch/x86/rpal/internal.h                      |   69 +
 arch/x86/rpal/mm.c                            |  426 +++
 arch/x86/rpal/pku.c                           |  196 ++
 arch/x86/rpal/proc.c                          |  279 ++
 arch/x86/rpal/service.c                       |  776 ++++++
 arch/x86/rpal/thread.c                        |  313 +++
 fs/binfmt_elf.c                               |   98 +-
 fs/eventpoll.c                                |  320 +++
 fs/exec.c                                     |   11 +
 include/linux/mm_types.h                      |    3 +
 include/linux/rpal.h                          |  633 +++++
 include/linux/sched.h                         |   21 +
 init/init_task.c                              |    6 +
 kernel/exit.c                                 |    5 +
 kernel/fork.c                                 |   32 +
 kernel/sched/core.c                           |  676 +++++
 kernel/sched/fair.c                           |  109 +
 kernel/sched/sched.h                          |    8 +
 mm/mmap.c                                     |   16 +
 mm/mprotect.c                                 |  106 +
 mm/rmap.c                                     |    4 +
 mm/vma.c                                      |   18 +
 samples/rpal/Makefile                         |   17 +
 samples/rpal/asm_define.c                     |   14 +
 samples/rpal/client.c                         |  178 ++
 samples/rpal/librpal/asm_define.h             |    6 +
 samples/rpal/librpal/asm_x86_64_rpal_call.S   |   57 +
 samples/rpal/librpal/debug.h                  |   12 +
 samples/rpal/librpal/fiber.c                  |  119 +
 samples/rpal/librpal/fiber.h                  |   64 +
 .../rpal/librpal/jump_x86_64_sysv_elf_gas.S   |   81 +
 .../rpal/librpal/make_x86_64_sysv_elf_gas.S   |   82 +
 .../rpal/librpal/ontop_x86_64_sysv_elf_gas.S  |   84 +
 samples/rpal/librpal/private.h                |  341 +++
 samples/rpal/librpal/rpal.c                   | 2351 +++++++++++++++++
 samples/rpal/librpal/rpal.h                   |  149 ++
 samples/rpal/librpal/rpal_pkru.h              |   78 +
 samples/rpal/librpal/rpal_queue.c             |  239 ++
 samples/rpal/librpal/rpal_queue.h             |   55 +
 samples/rpal/librpal/rpal_x86_64_call_ret.S   |   45 +
 samples/rpal/offset.sh                        |    5 +
 samples/rpal/server.c                         |  249 ++
 61 files changed, 9710 insertions(+), 4 deletions(-)
 create mode 100644 arch/x86/rpal/Kconfig
 create mode 100644 arch/x86/rpal/Makefile
 create mode 100644 arch/x86/rpal/core.c
 create mode 100644 arch/x86/rpal/internal.h
 create mode 100644 arch/x86/rpal/mm.c
 create mode 100644 arch/x86/rpal/pku.c
 create mode 100644 arch/x86/rpal/proc.c
 create mode 100644 arch/x86/rpal/service.c
 create mode 100644 arch/x86/rpal/thread.c
 create mode 100644 include/linux/rpal.h
 create mode 100644 samples/rpal/Makefile
 create mode 100644 samples/rpal/asm_define.c
 create mode 100644 samples/rpal/client.c
 create mode 100644 samples/rpal/librpal/asm_define.h
 create mode 100644 samples/rpal/librpal/asm_x86_64_rpal_call.S
 create mode 100644 samples/rpal/librpal/debug.h
 create mode 100644 samples/rpal/librpal/fiber.c
 create mode 100644 samples/rpal/librpal/fiber.h
 create mode 100644 samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/make_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/private.h
 create mode 100644 samples/rpal/librpal/rpal.c
 create mode 100644 samples/rpal/librpal/rpal.h
 create mode 100644 samples/rpal/librpal/rpal_pkru.h
 create mode 100644 samples/rpal/librpal/rpal_queue.c
 create mode 100644 samples/rpal/librpal/rpal_queue.h
 create mode 100644 samples/rpal/librpal/rpal_x86_64_call_ret.S
 create mode 100755 samples/rpal/offset.sh
 create mode 100644 samples/rpal/server.c

--
2.20.1

author
#26 lore raw

Hi Lorenzo,

Quoted 442 lines · Lorenzo Stoakes

Bo,

You have outstanding feedback on your v1 from me and Dave Hansen. I'm not quite sure why you're sending a v2 without responding to that.

This isn't how the upstream kernel works...

Thanks, Lorenzo

Quoted 432 lines · Bo Li

Changelog:

v2:
- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

link to v1:
https://lore.kernel.org/lkml/CAP2HCOmAkRVTci0ObtyW=3v6GFOrt9zCn2NwLUbZ+Di49xkBiw@mail.gmail.com/

--------------------------------------------------------------------------

# Introduction

We mainly apply RPAL to the service mesh architecture widely adopted in modern cloud-native data centers. Before the rise of the service mesh architecture, network functions were usually integrated into monolithic applications as libraries, and the main business programs invoked them through function calls. However, to facilitate the independent development and operation and maintenance of the main business programs and network functions, the service mesh removed the network functions from the main business programs and made them independent processes (called sidecars). Inter-process communication (IPC) is used for interaction between the main business program and the sidecar, and the introduced inter-process communication has led to a sharp increase in resource consumption in cloud-native data centers, and may even occupy more than 10% of the CPU of the entire microservice cluster.

To achieve the efficient function call mechanism of the monolithic architecture under the service mesh architecture, we introduced the RPAL (Running Process As Library) architecture, which implements the sharing of the virtual address space of processes and the switching threads in user mode. Through the analysis of the service mesh architecture, we found that the process memory isolation between the main business program and the sidecar is not particularly important because they are split from one application and were an integral part of the original monolithic application. It is more important for the two processes to be independent of each other because they need to be independently developed and maintained to ensure the architectural advantages of the service mesh. Therefore, RPAL breaks the isolation between processes while preserving the independence between them. We think that RPAL can also be applied to other scenarios featuring sidecar-like architectures, such as distributed file storage systems in LLM infra.

In RPAL architecture, multiple processes share a virtual address space, so this architecture can be regarded as an advanced version of the Linux shared memory mechanism:

1. Traditional shared memory requires two processes to negotiate to ensure the mapping of the same piece of memory. In RPAL architecture, two RPAL processes still need to reach a consensus before they can successfully invoke the relevant system calls of RPAL to share the virtual address space.
2. Traditional shared memory only shares part of the data. However, in RPAL architecture, processes that have established an RPAL communication relationship share a virtual address space, and all user memory (such as data segments and code segments) of each RPAL process is shared among these processes. However, a process cannot access the memory of other processes at any time. We use the MPK mechanism to ensure that the memory of other processes can only be accessed when special RPAL functions are called. Otherwise, a page fault will be triggered.
3. In RPAL architecture, to ensure the consistency of the execution context of the shared code (such as the stack and thread local storage), we further implement the thread context switching in user mode based on the ability to share the virtual address space of different processes, enabling the threads of different processes to directly perform fast switching in user mode without falling into kernel mode for slow switching.

# Background

In traditional inter-process communication (IPC) scenarios, Unix domain sockets are commonly used in conjunction with the epoll() family for event multiplexing. IPC operations involve system calls on both the data and control planes, thereby imposing a non-trivial overhead on the interacting processes. Even when shared memory is employed to optimize the data plane, two data copies still remain. Specifically, data is initially copied from a process's private memory space into the shared memory area, and then it is copied from the shared memory into the private memory of another process.

This poses a question: Is it possible to reduce the overhead of IPC with only minimal modifications at the application level? To address this, we observed that the functionality of IPC, which encompasses data transfer and invocation of the target thread, is similar to a function call, where arguments are passed and the callee function is invoked to process them. Inspired by this analogy, we introduce RPAL (Run Process As Library), a framework designed to enable one process to invoke another as if making a local function call, all without going through the kernel.

# Design

First, let’s formalize RPAL’s core objectives:

1. Data-plane efficiency: Reduce the number of data copies from two (in the shared memory solution) to one.
2. Control-plane optimization: Eliminate the overhead of system calls and kernel's thread switches.
3. Application compatibility: Minimize the modifications to existing applications that utilize Unix domain sockets and the epoll() family.

To attain the first objective, processes that use RPAL share the same virtual address space. So one process can access another's data directly via a data pointer. This means data can be transferred from one process to another with just one copy operation.

To meet the second goal, RPAL relies on the shared address space to do lightweight context switching in user space, which we call an "RPAL call". This allows one process to execute another process's code just like a local function call.

To achieve the third target, RPAL stays compatible with the epoll family of functions, like epoll_create(), epoll_wait(), and epoll_ctl(). If an application uses epoll for IPC, developers can switch to RPAL with just a few small changes. For instance, you can just replace epoll_wait() with rpal_epoll_wait(). The basic epoll procedure, where a process waits for another to write to a monitored descriptor using an epoll file descriptor, still works fine with RPAL.

## Address space sharing

For address space sharing, RPAL partitions the entire userspace virtual address space and allocates non-overlapping memory ranges to each process. On x86_64 architectures, RPAL uses a memory range size covered by a single PUD (Page Upper Directory) entry, which is 512GB. This restricts each process’s virtual address space to 512GB on x86_64, sufficient for most applications in our scenario. The rationale is straightforward:
address space sharing can be simply achieved by copying the PUD from one process’s page table to another’s. So one process can directly use the data pointer to access another's memory.

  |------------| <- 0
  |------------| <- 512 GB
  |  Process A |
  |------------| <- 2*512 GB
  |------------| <- n*512 GB
  |  Process B |
  |------------| <- (n+1)*512 GB
  |------------| <- STACK_TOP
  |  Kernel    |
  |------------|

## RPAL call

We refer to the lightweight userspace context switching mechanism as RPAL call. It enables the caller (or sender) thread of one process to directly switch to the callee (or receiver) thread of another process.

When Process A’s caller thread initiates an RPAL call to Process B’s callee thread, the CPU saves the caller’s context and loads the callee’s context. This enables direct userspace control flow transfer from the caller to the callee. After the callee finishes data processing, the CPU saves Process B’s callee context and switches back to Process A’s caller context, completing a full IPC cycle.

  |------------|                |---------------------|
  |  Process A |                |  Process B          |
  | |-------|  |                | |-------|           |
  | | caller| --- RPAL call --> | | callee|    handle |
  | | thread| <------------------ | thread| -> event  |
  | |-------|  |                | |-------|           |
  |------------|                |---------------------|

# Security and compatibility with kernel subsystems

## Memory protection between processes

Since processes using RPAL share the address space, unintended cross-process memory access may occur and corrupt the data of another process. To mitigate this, we leverage Memory Protection Keys (MPK) on x86 architectures.

MPK assigns 4 bits in each page table entry to a "protection key", which is paired with a userspace register (PKRU). The PKRU register defines access permissions for memory regions protected by specific keys (for detailed implementation, refer to the kernel documentation "Memory Protection Keys"). With MPK, even though the address space is shared among processes, cross-process access is restricted: a process can only access the memory protected by a key if its PKRU register is configured with the corresponding permission. This ensures that processes cannot access each other’s memory unless an explicit PKRU configuration is set.

## Page fault handling and TLB flushing

Due to the shared address space architecture, both page fault handling and TLB flushing require careful consideration. For instance, when Process A accesses Process B’s memory, a page fault may occur in Process A's context, but the faulting address belongs to Process B. In this case, we must pass Process B's mm_struct to the page fault handler.

TLB flushing is more complex. When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

## Lazy switch of kernel context

In RPAL, a mismatch may arise between the user context and the kernel context. The RPAL call is designed solely to switch the user context, leaving the kernel context unchanged. For instance, when a RPAL call takes place, transitioning from caller thread to callee thread, and subsequently a system call is initiated within callee thread, the kernel will incorrectly utilize the caller's kernel context (such as the kernel stack) to process the system call.

To resolve context mismatch issues, a kernel context switch is triggered at the kernel entry point when the callee initiates a syscall or an exception/interrupt occurs. This mechanism ensures context consistency before processing system calls, interrupts, or exceptions. We refer to this kernel context switch as a "lazy switch" because it defers the switching operation from the traditional thread switch point to the next kernel entry point.

Lazy switch should be minimized as much as possible, as it significantly degrades performance. We currently utilize RPAL in an RPC framework, in which the RPC sender thread relies on the RPAL call to invoke the RPC receiver thread entirely in user space. In most cases, the receiver thread is free of system calls and the code execution time is relatively short. This characteristic effectively reduces the probability of a lazy switch occurring.

## Time slice correction

After an RPAL call, the callee's user mode code executes. However, the kernel incorrectly attributes this CPU time to the caller due to the unchanged kernel context.

To resolve this, we use the Time Stamp Counter (TSC) register to measure CPU time consumed by the callee thread in user space. The kernel then uses this user-reported timing data to adjust the CPU accounting for both the caller and callee thread, similar to how CPU steal time is implemented.

## Process recovery

Since processes can access each other’s memory, there is a risk that the target process’s memory may become invalid at the access time (e.g., if the target process has exited unexpectedly). The kernel must handle such cases; otherwise, the accessing process could be terminated due to failures originating from another process.

To address this issue, each thread of the process should pre-establish a recovery point when accessing the memory of other processes. When such an invalid access occurs, the thread traps into the kernel. Inside the page fault handler, the kernel restores the user context of the thread to the recovery point. This mechanism ensures that processes maintain mutual independence, preventing cascading failures caused by cross-process memory issues.

# Performance

To quantify the performance improvements driven by RPAL, we measured latency both before and after its deployment. Experiments were conducted on a server equipped with two Intel(R) Xeon(R) Platinum 8336C CPUs (2.30 GHz) and 1 TB of memory. Latency was defined as the duration from when the client thread initiates a message to when the server thread is invoked and receives it.

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

We have applied RPAL to an RPC framework that is widely used in our data center. With RPAL, we have successfully achieved up to 15.5% reduction in the CPU utilization of processes in real-world microservice scenario. The gains primarily stem from minimizing control plane overhead through the utilization of userspace context switches. Additionally, by leveraging address space sharing, the number of memory copies is significantly reduced.

# Future Work

Currently, RPAL requires the MPK (Memory Protection Key) hardware feature, which is supported by a range of Intel CPUs. For AMD architectures, MPK is supported only on the latest processor, specifically, 3th Generation AMD EPYC™ Processors and subsequent generations. Patch sets that extend RPAL support to systems lacking MPK hardware will be provided later.

Accompanying test programs are also provided in the samples/rpal/ directory. And the user-mode RPAL library, which realizes user-space RPAL call, is in the samples/rpal/librpal directory.

We hope to get some community discussions and feedback on RPAL's optimization approaches and architecture.

Look forward to your comments.

Bo Li (35):
Kbuild: rpal support
RPAL: add struct rpal_service
RPAL: add service registration interface
RPAL: add member to task_struct and mm_struct
RPAL: enable virtual address space partitions
RPAL: add user interface
RPAL: enable shared page mmap
RPAL: enable sender/receiver registration
RPAL: enable address space sharing
RPAL: allow service enable/disable
RPAL: add service request/release
RPAL: enable service disable notification
RPAL: add tlb flushing support
RPAL: enable page fault handling
RPAL: add sender/receiver state
RPAL: add cpu lock interface
RPAL: add a mapping between fsbase and tasks
sched: pick a specified task
RPAL: add lazy switch main logic
RPAL: add rpal_ret_from_lazy_switch
RPAL: add kernel entry handling for lazy switch
RPAL: rebuild receiver state
RPAL: resume cpumask when fork
RPAL: critical section optimization
RPAL: add MPK initialization and interface
RPAL: enable MPK support
RPAL: add epoll support
RPAL: add rpal_uds_fdmap() support
RPAL: fix race condition in pkru update
RPAL: fix pkru setup when fork
RPAL: add receiver waker
RPAL: fix unknown nmi on AMD CPU
RPAL: enable time slice correction
RPAL: enable fast epoll wait samples/rpal: add RPAL samples

  arch/x86/Kbuild                               |    2 +
  arch/x86/Kconfig                              |    2 +
  arch/x86/entry/entry_64.S                     |  160 ++
  arch/x86/events/amd/core.c                    |   14 +
  arch/x86/include/asm/pgtable.h                |   25 +
  arch/x86/include/asm/pgtable_types.h          |   11 +
  arch/x86/include/asm/tlbflush.h               |   10 +
  arch/x86/kernel/asm-offsets.c                 |    3 +
  arch/x86/kernel/cpu/common.c                  |    8 +-
  arch/x86/kernel/fpu/core.c                    |    8 +-
  arch/x86/kernel/nmi.c                         |   20 +
  arch/x86/kernel/process.c                     |   25 +-
  arch/x86/kernel/process_64.c                  |  118 +
  arch/x86/mm/fault.c                           |  271 ++
  arch/x86/mm/mmap.c                            |   10 +
  arch/x86/mm/tlb.c                             |  172 ++
  arch/x86/rpal/Kconfig                         |   21 +
  arch/x86/rpal/Makefile                        |    6 +
  arch/x86/rpal/core.c                          |  477 ++++
  arch/x86/rpal/internal.h                      |   69 +
  arch/x86/rpal/mm.c                            |  426 +++
  arch/x86/rpal/pku.c                           |  196 ++
  arch/x86/rpal/proc.c                          |  279 ++
  arch/x86/rpal/service.c                       |  776 ++++++
  arch/x86/rpal/thread.c                        |  313 +++
  fs/binfmt_elf.c                               |   98 +-
  fs/eventpoll.c                                |  320 +++
  fs/exec.c                                     |   11 +
  include/linux/mm_types.h                      |    3 +
  include/linux/rpal.h                          |  633 +++++
  include/linux/sched.h                         |   21 +
  init/init_task.c                              |    6 +
  kernel/exit.c                                 |    5 +
  kernel/fork.c                                 |   32 +
  kernel/sched/core.c                           |  676 +++++
  kernel/sched/fair.c                           |  109 +
  kernel/sched/sched.h                          |    8 +
  mm/mmap.c                                     |   16 +
  mm/mprotect.c                                 |  106 +
  mm/rmap.c                                     |    4 +
  mm/vma.c                                      |   18 +
  samples/rpal/Makefile                         |   17 +
  samples/rpal/asm_define.c                     |   14 +
  samples/rpal/client.c                         |  178 ++
  samples/rpal/librpal/asm_define.h             |    6 +
  samples/rpal/librpal/asm_x86_64_rpal_call.S   |   57 +
  samples/rpal/librpal/debug.h                  |   12 +
  samples/rpal/librpal/fiber.c                  |  119 +
  samples/rpal/librpal/fiber.h                  |   64 +
  .../rpal/librpal/jump_x86_64_sysv_elf_gas.S   |   81 +
  .../rpal/librpal/make_x86_64_sysv_elf_gas.S   |   82 +
  .../rpal/librpal/ontop_x86_64_sysv_elf_gas.S  |   84 +
  samples/rpal/librpal/private.h                |  341 +++
  samples/rpal/librpal/rpal.c                   | 2351 +++++++++++++++++
  samples/rpal/librpal/rpal.h                   |  149 ++
  samples/rpal/librpal/rpal_pkru.h              |   78 +
  samples/rpal/librpal/rpal_queue.c             |  239 ++
  samples/rpal/librpal/rpal_queue.h             |   55 +
  samples/rpal/librpal/rpal_x86_64_call_ret.S   |   45 +
  samples/rpal/offset.sh                        |    5 +
  samples/rpal/server.c                         |  249 ++
  61 files changed, 9710 insertions(+), 4 deletions(-)

create mode 100644 arch/x86/rpal/Kconfig
create mode 100644 arch/x86/rpal/Makefile
create mode 100644 arch/x86/rpal/core.c
create mode 100644 arch/x86/rpal/internal.h
create mode 100644 arch/x86/rpal/mm.c
create mode 100644 arch/x86/rpal/pku.c
create mode 100644 arch/x86/rpal/proc.c
create mode 100644 arch/x86/rpal/service.c
create mode 100644 arch/x86/rpal/thread.c
create mode 100644 include/linux/rpal.h
create mode 100644 samples/rpal/Makefile
create mode 100644 samples/rpal/asm_define.c
create mode 100644 samples/rpal/client.c
create mode 100644 samples/rpal/librpal/asm_define.h create mode 100644 samples/rpal/librpal/asm_x86_64_rpal_call.S create mode 100644 samples/rpal/librpal/debug.h
create mode 100644 samples/rpal/librpal/fiber.c
create mode 100644 samples/rpal/librpal/fiber.h
create mode 100644 samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S create mode 100644 samples/rpal/librpal/make_x86_64_sysv_elf_gas.S create mode 100644 samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S create mode 100644 samples/rpal/librpal/private.h
create mode 100644 samples/rpal/librpal/rpal.c
create mode 100644 samples/rpal/librpal/rpal.h
create mode 100644 samples/rpal/librpal/rpal_pkru.h create mode 100644 samples/rpal/librpal/rpal_queue.c create mode 100644 samples/rpal/librpal/rpal_queue.h create mode 100644 samples/rpal/librpal/rpal_x86_64_call_ret.S create mode 100755 samples/rpal/offset.sh
create mode 100644 samples/rpal/server.c

--
2.20.1

Thank you for your feedback! There might be some misunderstanding. According to the feedback in RPAL V1, we rebased the RPAL to the latest stable kernel and added an introduction section to explain our considerations regarding the process isolation of the RPAL architecture.

Thanks!

#27 lore raw
Quoted 13 lines · Bo Li

Hi Lorenzo,

Quoted 10 lines · Lorenzo Stoakes

Bo,

You have outstanding feedback on your v1 from me and Dave Hansen. I'm not quite sure why you're sending a v2 without responding to that.

This isn't how the upstream kernel works...

Thanks, Lorenzo

On Fri, May 30, 2025 at 05:27:28PM +0800, Bo Li wrote:

[snip]

Quoted 6 lines

Thank you for your feedback! There might be some misunderstanding. According to the feedback in RPAL V1, we rebased the RPAL to the latest stable kernel and added an introduction section to explain our considerations regarding the process isolation of the RPAL architecture.

Thanks!

Hi Bo,

You need to engage in _conversation_ with maintainers, not simply resend giant RFC's with changes made based on your interpretation of the feedback.

You've not addressed my comments, you've interpreted them to be 'ok do X, Y, Z', then done them without a word. This is, again, not how upstream works. You've seemingly ignored Dave altogether.

Others have highlighted it, but let me repeat what they have (in effect) said - this is just not mergeable upstream in any way shape or form, sorry.

It's a NAK and there's just no way it's not a NAK, you're doing too many crazy things here that are just not acceptable, not to mention the issues people have raised.

You should have engaged with upstream WAY earlier.

It's a pity you've put so much work into this without having done so, but I'm afraid you're going to have to maintain this out-of-tree indefinitely.

I hope you can at least can take some lessons from this on how best in future to engage with upstream (early and often! :)

Thanks, Lorenzo

author
#28 lore raw

RFC v2 24/35 RPAL: critical section optimization

In reply to Bo Li

The critical section is defined as the user mode code segment within the receiver that executes when control returns from the receiver to the sender. This code segment, located in the receiver, involves operations such as switching the fsbase register and changing the stack pointer. Handling the critical section can be categorized into two scenarios:

- First Scenario: If no lazy switch has occurred prior to the return and the fsbase switch is incomplete, a lazy switch is triggered to transition the kernel context from the sender to the receiver. After the fsbase is updated in user mode, another lazy switch occurs to revert the kernel context from the receiver back to the sender. This results in two unnecessary lazy switches.

- Second Scenario: If a lazy switch has already occurred during execution of the critical section, the lazy switch can be preemptively triggered. This avoids re-entering the kernel solely to initiate another lazy switch.

The implementation of the critical section involves modifying the fsbase register in kernel mode and setting the sender's user mode context to a predefined state. These steps minimize redundant user/kernel transitions and lazy switches.

---
 arch/x86/rpal/core.c    | 88 ++++++++++++++++++++++++++++++++++++++++-
 arch/x86/rpal/service.c | 12 ++++++
 include/linux/rpal.h    |  6 +++
 3 files changed, 104 insertions(+), 2 deletions(-)
arch/x86/rpal/core.c +86 -2
index c48df1ce4324..406d54788bac 100644
@@ -219,14 +219,98 @@ static inline struct task_struct *rpal_misidentify(void)
 	return next;
 }
 
+static bool in_ret_section(struct rpal_service *rs, unsigned long ip)
+{
+	return ip >= rs->rsm.rcs.ret_begin && ip < rs->rsm.rcs.ret_end;
+}
+
+/*
+ * rpal_update_fsbase - fastpath when RPAL call returns
+ * @regs: pt_regs saved in kernel entry
+ *
+ * If the user is executing rpal call return code and it does
+ * not update fsbase yet, force fsbase update to perform a
+ * lazy switch immediately.
+ */
+static inline void rpal_update_fsbase(struct pt_regs *regs)
+{
+	struct rpal_service *cur = rpal_current_service();
+	struct task_struct *sender = current->rpal_rd->sender;
+
+	if (in_ret_section(cur, regs->ip))
+		wrfsbase(sender->thread.fsbase);
+}
+
+/*
+ * rpal_skip_receiver_code - skip rpal call return code
+ * @next: the next task to be lazy switched to.
+ * @regs: pt_regs saved in kernel entry
+ *
+ * If the user is executing rpal call return code and we are about
+ * to perform a lazy switch, skip the remaining return code to
+ * release receiver's stack. This avoids stack conflict when there
+ * are more than one senders calls the receiver.
+ */
+static inline void rpal_skip_receiver_code(struct task_struct *next,
+					   struct pt_regs *regs)
+{
+	rebuild_sender_stack(next->rpal_sd, regs);
+}
+
+/*
+ * rpal_skip_receiver_code - skip lazy switch when rpal call return
+ * @next: the next task to be lazy switched to.
+ * @regs: pt_regs saved in kernel entry
+ *
+ * If the user is executing rpal call return code and we have not
+ * performed a lazy switch, there is no need to perform lazy switch
+ * now. Update fsbase and other states to avoid lazy switch.
+ */
+static inline struct task_struct *
+rpal_skip_lazy_switch(struct task_struct *next, struct pt_regs *regs)
+{
+	struct rpal_service *tgt;
+
+	tgt = next->rpal_rs;
+	if (in_ret_section(tgt, regs->ip)) {
+		wrfsbase(current->thread.fsbase);
+		rebuild_sender_stack(current->rpal_sd, regs);
+		rpal_clear_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT);
+		next->rpal_rd->sender = NULL;
+		next = NULL;
+	}
+	return next;
+}
+
+static struct task_struct *rpal_fix_critical_section(struct task_struct *next,
+						     struct pt_regs *regs)
+{
+	struct rpal_service *cur = rpal_current_service();
+
+	/* sender->receiver */
+	if (rpal_test_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT))
+		next = rpal_skip_lazy_switch(next, regs);
+	/* receiver->sender */
+	else if (rpal_is_correct_address(cur, regs->ip))
+		rpal_skip_receiver_code(next, regs);
+
+	return next;
+}
+
 static inline struct task_struct *
 rpal_kernel_context_switch(struct pt_regs *regs)
 {
 	struct task_struct *next = NULL;
 
+	if (rpal_test_current_thread_flag(RPAL_LAZY_SWITCHED_BIT))
+		rpal_update_fsbase(regs);
+
 	next = rpal_misidentify();
-	if (unlikely(next != NULL))
-		next = rpal_do_kernel_context_switch(next, regs);
+	if (unlikely(next != NULL)) {
+		next = rpal_fix_critical_section(next, regs);
+		if (next)
+			next = rpal_do_kernel_context_switch(next, regs);
+	}
 
 	return next;
 }
arch/x86/rpal/service.c +12 -0
index 49458321e7dc..16e94d710445 100644
@@ -545,6 +545,13 @@ int rpal_release_service(u64 key)
 	return ret;
 }
 
+static bool rpal_check_critical_section(struct rpal_service *rs,
+			     struct rpal_critical_section *rcs)
+{
+	return rpal_is_correct_address(rs, rcs->ret_begin) &&
+	       rpal_is_correct_address(rs, rcs->ret_end);
+}
+
 int rpal_enable_service(unsigned long arg)
 {
 	struct rpal_service *cur = rpal_current_service();
@@ -562,6 +569,11 @@ int rpal_enable_service(unsigned long arg)
 		goto out;
 	}
 
+	if (!rpal_check_critical_section(cur, &rsm.rcs)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
 	mutex_lock(&cur->mutex);
 	if (!cur->enabled) {
 		cur->rsm = rsm;
include/linux/rpal.h +6 -0
index b24176f3f245..4f1d92053818 100644
@@ -122,12 +122,18 @@ enum rpal_sender_state {
 	RPAL_SENDER_STATE_KERNEL_RET,
 };
 
+struct rpal_critical_section {
+	unsigned long ret_begin;
+	unsigned long ret_end;
+};
+
 /*
  * user_meta will be sent to other service when requested.
  */
 struct rpal_service_metadata {
 	unsigned long version;
 	void __user *user_meta;
+	struct rpal_critical_section rcs;
 };
 
 struct rpal_request_arg {
Signature · 1 line
2.20.1
author
#29 lore raw

RFC v2 25/35 RPAL: add MPK initialization and interface

In reply to Bo Li

RPAL uses MPK (Memory Protection Keys) to protect memory. Therefore, RPAL needs to perform MPK initialization, allocation, and other related tasks, while providing corresponding user-mode interfaces.

This patch executes MPK initialization operations, including feature detection, implementation of user mode interfaces for setting and retrieving pkeys, and development of utility functions. For pkey allocation, RPAL prioritizes using pkeys provided by user mode, with user mode responsible for preventing pkey collisions between different services. If user mode does not provide a valid pkey, RPAL generates a pkey via id % arch_max_pkey() to maximize the avoidance of pkey collisions. Additionally, RPAL does not permit services to manipulate pkeys independently; thus, all pkeys are marked as allocated, and services are prohibited from releasing pkeys.

---
 arch/x86/rpal/Kconfig    | 12 +++++++-
 arch/x86/rpal/Makefile   |  1 +
 arch/x86/rpal/core.c     | 13 ++++++++
 arch/x86/rpal/internal.h |  5 +++
 arch/x86/rpal/pku.c      | 47 ++++++++++++++++++++++++++++
 arch/x86/rpal/proc.c     |  5 +++
 arch/x86/rpal/service.c  | 24 +++++++++++++++
 include/linux/rpal.h     | 66 ++++++++++++++++++++++++++++++++++++++++
 mm/mprotect.c            |  9 ++++++
 9 files changed, 181 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/rpal/pku.c
arch/x86/rpal/Kconfig +11 -1
index e5e6996553ea..5434fdb2940d 100644
@@ -8,4 +8,14 @@ config RPAL
 	depends on X86_64
 	help
 		This option enables system support for Run Process As
-		library (RPAL).
\ No newline at end of file
+		library (RPAL).
+
+config RPAL_PKU
+	bool "mpk protection for RPAL"
+	default y
+	depends on RPAL
+	help
+		Memory protection key (MPK) can achieve intra-process
+		memory separation which is broken by RPAL, Always keep
+		it on when use RPAL. CPU feature will be detected at
+		boot time as some CPUs do not support it.
\ No newline at end of file
arch/x86/rpal/Makefile +1 -0
index 89f745382c51..42a42b0393be 100644
@@ -3,3 +3,4 @@
 obj-$(CONFIG_RPAL)		+= rpal.o
 
 rpal-y := service.o core.o mm.o proc.o thread.o
+rpal-$(CONFIG_RPAL_PKU) += pku.o
\ No newline at end of file
arch/x86/rpal/core.c +13 -0
index 406d54788bac..41111d693994 100644
@@ -8,6 +8,7 @@
 
 #include <linux/rpal.h>
 #include <linux/sched/task_stack.h>
+#include <linux/pkeys.h>
 #include <asm/fsgsbase.h>
 
 #include "internal.h"
@@ -374,6 +375,14 @@ static bool check_hardware_features(void)
 		rpal_err("no fsgsbase feature\n");
 		return false;
 	}
+
+#ifdef CONFIG_RPAL_PKU
+	if (!arch_pkeys_enabled()) {
+		rpal_err("MPK is not enabled\n");
+		return false;
+	}
+#endif
+
 	return true;
 }
 
@@ -390,6 +399,10 @@ int __init rpal_init(void)
 	if (ret)
 		goto fail;
 
+#ifdef CONFIG_RPAL_PKU
+	rpal_set_cap(RPAL_CAP_PKU);
+#endif
+
 	rpal_inited = true;
 	return 0;
 
arch/x86/rpal/internal.h +5 -0
index 6256172bb79e..71afa8225450 100644
@@ -54,3 +54,8 @@ rpal_build_call_state(const struct rpal_sender_data *rsd)
 	return ((rsd->rcd.service_id << RPAL_SID_SHIFT) |
 		(rsd->scc->sender_id << RPAL_ID_SHIFT) | RPAL_RECEIVER_STATE_CALL);
 }
+
+/* pkey.c */
+int rpal_alloc_pkey(struct rpal_service *rs, int pkey);
+int rpal_pkey_setup(struct rpal_service *rs, int pkey);
+void rpal_service_pku_init(void);
arch/x86/rpal/pku.c new file +47 -0

47 lines not on this page · raw message

arch/x86/rpal/proc.c +5 -0
index 16ac9612bfc5..2f9cceec4992 100644
@@ -76,6 +76,11 @@ static long rpal_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 	case RPAL_IOCTL_RELEASE_SERVICE:
 		ret = rpal_release_service(arg);
 		break;
+#ifdef CONFIG_RPAL_PKU
+	case RPAL_IOCTL_GET_SERVICE_PKEY:
+		ret = put_user(cur->pkey, (int __user *)arg);
+		break;
+#endif
 	default:
 		return -EINVAL;
 	}
arch/x86/rpal/service.c +24 -0
index 16e94d710445..ca795dacc90d 100644
@@ -208,6 +208,10 @@ struct rpal_service *rpal_register_service(void)
 	spin_lock_init(&rs->rpd.poll_lock);
 	bitmap_zero(rs->rpd.dead_key_bitmap, RPAL_NR_ID);
 	init_waitqueue_head(&rs->rpd.rpal_waitqueue);
+#ifdef CONFIG_RPAL_PKU
+	rs->pkey = -1;
+	rpal_service_pku_init();
+#endif
 
 	rs->bad_service = false;
 	rs->base = calculate_base_address(rs->id);
@@ -288,6 +292,9 @@ static int add_mapped_service(struct rpal_service *rs, struct rpal_service *tgt,
 	if (node->rs == NULL) {
 		node->rs = rpal_get_service(tgt);
 		set_bit(type_bit, &node->type);
+#ifdef CONFIG_RPAL_PKU
+		node->pkey = tgt->pkey;
+#endif
 	} else {
 		if (node->rs != tgt) {
 			ret = -EINVAL;
@@ -397,6 +404,19 @@ int rpal_request_service(unsigned long arg)
 		goto put_service;
 	}
 
+#ifdef CONFIG_RPAL_PKU
+	if (cur->pkey == tgt->pkey) {
+		ret = -EINVAL;
+		goto put_service;
+	}
+
+	ret = put_user(tgt->pkey, rra.pkey);
+	if (ret) {
+		ret = -EFAULT;
+		goto put_service;
+	}
+#endif
+
 	ret = put_user((unsigned long)(tgt->rsm.user_meta), rra.user_metap);
 	if (ret) {
 		ret = -EFAULT;
@@ -577,6 +597,10 @@ int rpal_enable_service(unsigned long arg)
 	mutex_lock(&cur->mutex);
 	if (!cur->enabled) {
 		cur->rsm = rsm;
+#ifdef CONFIG_RPAL_PKU
+		rsm.pkey = rpal_alloc_pkey(cur, rsm.pkey);
+		rpal_pkey_setup(cur, rsm.pkey);
+#endif
 		cur->enabled = true;
 	}
 	mutex_unlock(&cur->mutex);
include/linux/rpal.h +66 -0

119 lines not on this page · raw message

mm/mprotect.c +9 -0
index 62c1f7945741..982f911ffaba 100644
@@ -33,6 +33,7 @@
 #include <linux/userfaultfd_k.h>
 #include <linux/memory-tiers.h>
 #include <uapi/linux/mman.h>
+#include <linux/rpal.h>
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/tlbflush.h>
@@ -895,6 +896,14 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
 {
 	int ret;
 
+#ifdef CONFIG_RPAL_PKU
+	if (rpal_current_service()) {
+		rpal_err("try_to_free pkey: %d %s\n", current->pid,
+			 current->comm);
+		return -EINVAL;
+	}
+#endif
+
 	mmap_write_lock(current->mm);
 	ret = mm_pkey_free(current->mm, pkey);
 	mmap_write_unlock(current->mm);
Signature · 1 line
2.20.1
author
#30 lore raw

RFC v2 26/35 RPAL: enable MPK support

In reply to Bo Li

RPAL leverages Memory Protection Keys (MPK) to safeguard shared memory from illegal access and corruption by other processes. MPK-based memory protection involves two key mechanisms: First, for already allocated memory, when RPAL is enabled, the protection key fields in all page tables must be set to the process’s corresponding pkey value. Second, for newly allocated memory, when the kernel detects that the process is an RPAL service, it sets the corresponding pkey flag in the relevant memory data structures. Together, these measures ensure that all memory belonging to the current process is protected by its own pkey.

For MPK initialization, RPAL needs to set the pkeys of all allocated page table pages to the pkeys assigned by RPAL to the service. This is completed in three steps: First, enable permissions for all pkeys of the service, allowing it to access memory protected by any pkey. Then, update the pkeys in the page tables. Since permissions for all pkeys are already enabled at this stage, even if old and new pkeys coexist during the page table update, the service's memory access remains unaffected. Finally, after the page table update is complete, set the service's pkey permissions to the corresponding values, thereby achieving memory protection.

Additionally, RPAL must manage the values of the PKRU register during lazy switch operations and signal handling. This ensures the process avoids coredumps causing by MPK.

---
 arch/x86/kernel/cpu/common.c |   8 +-
 arch/x86/kernel/fpu/core.c   |   8 +-
 arch/x86/kernel/process.c    |   7 +-
 arch/x86/rpal/core.c         |  14 +++-
 arch/x86/rpal/internal.h     |   1 +
 arch/x86/rpal/pku.c          | 139 ++++++++++++++++++++++++++++++++++-
 arch/x86/rpal/service.c      |   1 +
 arch/x86/rpal/thread.c       |   5 ++
 include/linux/rpal.h         |   3 +
 kernel/sched/core.c          |   3 +
 mm/mmap.c                    |  12 +++
 mm/mprotect.c                |  96 ++++++++++++++++++++++++
 mm/vma.c                     |  18 +++++
 13 files changed, 310 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/common.c +7 -1
index 8feb8fd2957a..2678453cdf76 100644
@@ -26,6 +26,7 @@
 #include <linux/pgtable.h>
 #include <linux/stackprotector.h>
 #include <linux/utsname.h>
+#include <linux/rpal.h>
 
 #include <asm/alternative.h>
 #include <asm/cmdline.h>
@@ -532,7 +533,12 @@ static __always_inline void setup_pku(struct cpuinfo_x86 *c)
 
 	cr4_set_bits(X86_CR4_PKE);
 	/* Load the default PKRU value */
-	pkru_write_default();
+#ifdef CONFIG_RPAL_PKU
+	if (rpal_current_service() && rpal_current_service()->pku_on)
+		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	else
+#endif
+		pkru_write_default();
 }
 
 #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
arch/x86/kernel/fpu/core.c +7 -1
index ea138583dd92..251b1ddee726 100644
@@ -20,6 +20,7 @@
 #include <linux/hardirq.h>
 #include <linux/pkeys.h>
 #include <linux/vmalloc.h>
+#include <linux/rpal.h>
 
 #include "context.h"
 #include "internal.h"
@@ -746,7 +747,12 @@ static inline void restore_fpregs_from_init_fpstate(u64 features_mask)
 	else
 		frstor(&init_fpstate.regs.fsave);
 
-	pkru_write_default();
+#ifdef CONFIG_RPAL_PKU
+	if (rpal_current_service() && rpal_current_service()->pku_on)
+		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	else
+#endif
+		pkru_write_default();
 }
 
 /*
arch/x86/kernel/process.c +6 -1
index be8845e2ca4d..b74de35218f9 100644
@@ -285,7 +285,12 @@ static void pkru_flush_thread(void)
 	 * If PKRU is enabled the default PKRU value has to be loaded into
 	 * the hardware right here (similar to context switch).
 	 */
-	pkru_write_default();
+#ifdef CONFIG_RPAL_PKU
+	if (rpal_current_service() && rpal_current_service()->pku_on)
+		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	else
+#endif
+		pkru_write_default();
 }
 
 void flush_thread(void)
arch/x86/rpal/core.c +13 -1
index 41111d693994..47c9e551344e 100644
@@ -275,6 +275,13 @@ rpal_skip_lazy_switch(struct task_struct *next, struct pt_regs *regs)
 	tgt = next->rpal_rs;
 	if (in_ret_section(tgt, regs->ip)) {
 		wrfsbase(current->thread.fsbase);
+#ifdef CONFIG_RPAL_PKU
+		rpal_set_current_pkru(
+			rpal_pkru_union(
+				rpal_pkey_to_pkru(rpal_current_service()->pkey),
+				rpal_pkey_to_pkru(next->rpal_rs->pkey)),
+			RPAL_PKRU_SET);
+#endif
 		rebuild_sender_stack(current->rpal_sd, regs);
 		rpal_clear_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT);
 		next->rpal_rd->sender = NULL;
@@ -292,8 +299,13 @@ static struct task_struct *rpal_fix_critical_section(struct task_struct *next,
 	if (rpal_test_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT))
 		next = rpal_skip_lazy_switch(next, regs);
 	/* receiver->sender */
-	else if (rpal_is_correct_address(cur, regs->ip))
+	else if (rpal_is_correct_address(cur, regs->ip)) {
 		rpal_skip_receiver_code(next, regs);
+#ifdef CONFIG_RPAL_PKU
+		write_pkru(rpal_pkru_union(
+			rpal_pkey_to_pkru(next->rpal_rs->pkey), rdpkru()));
+#endif
+	}
 
 	return next;
 }
arch/x86/rpal/internal.h +1 -0
index 71afa8225450..e49febce8645 100644
@@ -58,4 +58,5 @@ rpal_build_call_state(const struct rpal_sender_data *rsd)
 /* pkey.c */
 int rpal_alloc_pkey(struct rpal_service *rs, int pkey);
 int rpal_pkey_setup(struct rpal_service *rs, int pkey);
+void rpal_set_current_pkru(u32 val, int mode);
 void rpal_service_pku_init(void);
arch/x86/rpal/pku.c +138 -1

150 lines not on this page · raw message

arch/x86/rpal/service.c +1 -0
index ca795dacc90d..7a83e85cf096 100644
@@ -210,6 +210,7 @@ struct rpal_service *rpal_register_service(void)
 	init_waitqueue_head(&rs->rpd.rpal_waitqueue);
 #ifdef CONFIG_RPAL_PKU
 	rs->pkey = -1;
+	rs->pku_on = false;
 	rpal_service_pku_init();
 #endif
 
arch/x86/rpal/thread.c +5 -0
index 02c1a9c22dd7..fcc592baaac0 100644
@@ -281,6 +281,11 @@ int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 			regs->sp = ersp;
 			/* avoid rebuild again */
 			scc->ec.magic = 0;
+#ifdef CONFIG_RPAL_PKU
+			rpal_set_current_pkru(
+				rpal_pkey_to_pkru(rpal_current_service()->pkey),
+				RPAL_PKRU_SET);
+#endif
 			return 0;
 		}
 	}
include/linux/rpal.h +3 -0
index 2f2982d281cc..f2474cb53abe 100644
@@ -239,6 +239,7 @@ struct rpal_service {
 
 #ifdef CONFIG_RPAL_PKU
 	/* pkey */
+	bool pku_on;
 	int pkey;
 #endif
 
@@ -571,4 +572,6 @@ void rpal_schedule(struct task_struct *next);
 asmlinkage struct task_struct *
 __rpal_switch_to(struct task_struct *prev_p, struct task_struct *next_p);
 asmlinkage __visible void rpal_schedule_tail(struct task_struct *prev);
+int do_rpal_mprotect_pkey(unsigned long start, size_t len, int pkey);
+void rpal_set_pku_schedule_tail(struct task_struct *prev);
 #endif
kernel/sched/core.c +3 -0
index 0f9343698198..eb5d5bd51597 100644
@@ -11029,6 +11029,9 @@ asmlinkage __visible void rpal_schedule_tail(struct task_struct *prev)
 
 	finish_task_switch(prev);
 	trace_sched_exit_tp(true, CALLER_ADDR0);
+#ifdef CONFIG_RPAL_PKU
+	rpal_set_pku_schedule_tail(prev);
+#endif
 	preempt_enable();
 
 	calculate_sigpending();
mm/mmap.c +12 -0
index 98bb33d2091e..d36ea4ea2bd0 100644
@@ -396,6 +396,18 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
 		if (pkey < 0)
 			pkey = 0;
 	}
+#ifdef CONFIG_RPAL_PKU
+	/*
+	 * For RPAL process, if pku is enabled, we always use
+	 * its service pkey for new vma.
+	 */
+	do {
+		struct rpal_service *cur = rpal_current_service();
+
+		if (cur && cur->pku_on)
+			pkey = cur->pkey;
+	} while (0);
+#endif
 
 	/* Do simple checking here so the lower-level routines won't have
 	 * to. we assume access permissions have been handled by the open
mm/mprotect.c +96 -0

108 lines not on this page · raw message

mm/vma.c +18 -0
index a468d4c29c0c..fa9d8f694e6e 100644
@@ -4,6 +4,8 @@
  * VMA-specific functions.
  */
 
+#include <linux/rpal.h>
+
 #include "vma_internal.h"
 #include "vma.h"
 
@@ -2622,6 +2624,22 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,
 {
 	struct mm_struct *mm = current->mm;
 
+#ifdef CONFIG_RPAL_PKU
+	/*
+	 * Any memory need to use RPAL service pkey
+	 * once service is enabled.
+	 */
+	struct rpal_service *cur = rpal_current_service();
+	unsigned long vma_pkey_mask;
+
+	if (cur && cur->pku_on) {
+		vma_pkey_mask = VM_PKEY_BIT0 | VM_PKEY_BIT1 | VM_PKEY_BIT2 |
+				VM_PKEY_BIT3;
+		flags &= ~vma_pkey_mask;
+		flags |= ((unsigned long)cur->pkey) << VM_PKEY_SHIFT;
+	}
+#endif
+
 	/*
 	 * Check against address space limits by the changed size
 	 * Note: This happens *after* clearing old mappings in some code paths.
Signature · 1 line
2.20.1
#31 lore raw

Re: [RFC v2 26/35] RPAL: enable MPK support

Quoted 2 lines · Bo Li

RPAL leverages Memory Protection Keys (MPK) to safeguard shared memory from illegal access and corruption by other processes.

... as long as nobody uses the completely unprivileged WRPKRU or XRSTOR instructions. Right?

... the instructions that are plainly available in super obscure libraries like glibc?

This seems like a rather major oversight.

Oh, speaking of major oversights, you stymied pkey_alloc() but forgot pkey_free(). There's nothing to stop folks from pkey_free()'ing a pkey that RPAL is using and then letting someone else allocate it.

Is this all for real and serious?

author
#32 lore raw

RFC v2 27/35 RPAL: add epoll support

In reply to Bo Li

To support the epoll family, RPAL needs to add new logic for RPAL services to the existing epoll logic, ensuring that user mode can execute RPAL service-related logic through identical interfaces.

When the receiver thread calls epoll_wait(), it can set RPAL_EP_POLL_MAGIC to notify the kernel to invoke RPAL-related logic. The kernel then sets the receiver's state to RPAL_RECEIVER_STATE_READY and transitions it to RPAL_RECEIVER_STATE_WAIT when the receiver is actually removed from the runqueue, allowing the sender to perform RPAL calls on the receiver thread.

---
 arch/x86/rpal/core.c |   4 +
 fs/eventpoll.c       | 200 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h |  21 +++++
 kernel/sched/core.c  |  17 ++++
 4 files changed, 242 insertions(+)
arch/x86/rpal/core.c +4 -0
index 47c9e551344e..6a22b9faa100 100644
@@ -9,6 +9,7 @@
 #include <linux/rpal.h>
 #include <linux/sched/task_stack.h>
 #include <linux/pkeys.h>
+#include <linux/file.h>
 #include <asm/fsgsbase.h>
 
 #include "internal.h"
@@ -63,6 +64,7 @@ void rpal_kernel_ret(struct pt_regs *regs)
 
 	if (rpal_test_current_thread_flag(RPAL_RECEIVER_BIT)) {
 		rcc = current->rpal_rd->rcc;
+		regs->ax = rpal_try_send_events(current->rpal_rd->ep, rcc);
 		atomic_xchg(&rcc->receiver_state, RPAL_RECEIVER_STATE_KERNEL_RET);
 	} else {
 		tsk = current->rpal_sd->receiver;
@@ -142,6 +144,7 @@ rpal_do_kernel_context_switch(struct task_struct *next, struct pt_regs *regs)
 	struct task_struct *prev = current;
 
 	if (rpal_test_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT)) {
+		rpal_resume_ep(next);
 		current->rpal_sd->receiver = next;
 		rpal_lock_cpu(current);
 		rpal_lock_cpu(next);
@@ -154,6 +157,7 @@ rpal_do_kernel_context_switch(struct task_struct *next, struct pt_regs *regs)
 		 */
 		rebuild_sender_stack(current->rpal_sd, regs);
 		rpal_schedule(next);
+		fdput(next->rpal_rd->f);
 	} else {
 		update_dst_stack(next, regs);
 		/*
fs/eventpoll.c +200 -0

219 lines not on this page · raw message

include/linux/rpal.h +21 -0
index f2474cb53abe..5912ffec6e28 100644
@@ -16,6 +16,8 @@
 #include <linux/hashtable.h>
 #include <linux/atomic.h>
 #include <linux/sizes.h>
+#include <linux/file.h>
+#include <linux/hrtimer.h>
 
 #define RPAL_ERROR_MSG "rpal error: "
 #define rpal_err(x...) pr_err(RPAL_ERROR_MSG x)
@@ -89,6 +91,7 @@ enum {
 };
 
 #define RPAL_ERROR_MAGIC 0x98CC98CC
+#define RPAL_EP_POLL_MAGIC 0xCC98CC98
 
 #define RPAL_SID_SHIFT 24
 #define RPAL_ID_SHIFT 8
@@ -103,6 +106,9 @@ enum {
 #define RPAL_PKRU_UNION 1
 #define RPAL_PKRU_INTERSECT 2
 
+#define RPAL_KERNEL_PENDING 0x1
+#define RPAL_USER_PENDING 0x2
+
 extern unsigned long rpal_cap;
 
 enum rpal_task_flag_bits {
@@ -282,6 +288,12 @@ struct rpal_receiver_call_context {
 	int receiver_id;
 	atomic_t receiver_state;
 	atomic_t sender_state;
+	atomic_t ep_pending;
+	int rpal_ep_poll_magic;
+	int epfd;
+	void __user *events;
+	int maxevents;
+	int timeout;
 };
 
 /* recovery point for sender */
@@ -325,6 +337,10 @@ struct rpal_receiver_data {
 	struct rpal_shared_page *rsp;
 	struct rpal_receiver_call_context *rcc;
 	struct task_struct *sender;
+	void *ep;
+	struct fd f;
+	struct hrtimer_sleeper ep_sleeper;
+	wait_queue_entry_t ep_wait;
 };
 
 struct rpal_sender_data {
@@ -574,4 +590,9 @@ __rpal_switch_to(struct task_struct *prev_p, struct task_struct *next_p);
 asmlinkage __visible void rpal_schedule_tail(struct task_struct *prev);
 int do_rpal_mprotect_pkey(unsigned long start, size_t len, int pkey);
 void rpal_set_pku_schedule_tail(struct task_struct *prev);
+int rpal_ep_autoremove_wake_function(wait_queue_entry_t *curr,
+	unsigned int mode, int wake_flags,
+	void *key);
+void rpal_resume_ep(struct task_struct *tsk);
+int rpal_try_send_events(void *ep, struct rpal_receiver_call_context *rcc);
 #endif
kernel/sched/core.c +17 -0
index eb5d5bd51597..486d59bdd3fc 100644
@@ -6794,6 +6794,23 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 #define SM_RTLOCK_WAIT		2
 
 #ifdef CONFIG_RPAL
+int rpal_ep_autoremove_wake_function(wait_queue_entry_t *curr,
+				     unsigned int mode, int wake_flags,
+				     void *key)
+{
+	struct rpal_receiver_data *rrd = curr->private;
+	struct task_struct *tsk = rrd->rcd.bp_task;
+	int ret;
+
+	ret = try_to_wake_up(tsk, mode, wake_flags);
+
+	list_del_init_careful(&curr->entry);
+	if (!ret)
+		atomic_or(RPAL_KERNEL_PENDING, &rrd->rcc->ep_pending);
+
+	return 1;
+}
+
 static inline void rpal_check_ready_state(struct task_struct *tsk, int state)
 {
 	if (rpal_test_task_thread_flag(tsk, RPAL_RECEIVER_BIT)) {
Signature · 1 line
2.20.1
author
#33 lore raw

RFC v2 28/35 RPAL: add rpal_uds_fdmap() support

In reply to Bo Li

For a UDS connection between a sender and a receiver, neither side knows which file descriptor (fd) the other uses to manage the connection. The sender cannot determine which user space fd's buffer in the receiver to write data to, necessitating a complex process for both sides to inform each other of fd mappings. This process incurs significant overhead when managing a large number of connections, which requires optimization.

This patch introduces the RPAL_IOCTL_UDS_FDMAP interface, which simplifies the establishment of fd mappings between sender and receiver processes for files monitored by epoll. This avoids the need for a complex setup process each time a new connection is created.

---
 arch/x86/rpal/internal.h |   3 +
 arch/x86/rpal/proc.c     | 117 +++++++++++++++++++++++++++++++++++++++
 fs/eventpoll.c           |  19 +++++++
 include/linux/rpal.h     |  11 ++++
 4 files changed, 150 insertions(+)
arch/x86/rpal/internal.h +3 -0
index e49febce8645..e03f8a90619d 100644
@@ -11,6 +11,7 @@
 
 #include <linux/mm.h>
 #include <linux/file.h>
+#include <net/af_unix.h>
 
 extern bool rpal_inited;
 
@@ -60,3 +61,5 @@ int rpal_alloc_pkey(struct rpal_service *rs, int pkey);
 int rpal_pkey_setup(struct rpal_service *rs, int pkey);
 void rpal_set_current_pkru(u32 val, int mode);
 void rpal_service_pku_init(void);
+
+extern struct sock *unix_peer_get(struct sock *sk);
arch/x86/rpal/proc.c +117 -0

135 lines not on this page · raw message

fs/eventpoll.c +19 -0
index 437cd5764c03..791321639561 100644
@@ -2143,6 +2143,25 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 }
 
 #ifdef CONFIG_RPAL
+void *rpal_get_epitemep(wait_queue_entry_t *wait)
+{
+	struct epitem *epi = ep_item_from_wait(wait);
+
+	if (!epi)
+		return NULL;
+
+	return epi->ep;
+}
+
+int rpal_get_epitemfd(wait_queue_entry_t *wait)
+{
+	struct epitem *epi = ep_item_from_wait(wait);
+
+	if (!epi)
+		return -1;
+
+	return epi->ffd.fd;
+}
 
 void rpal_resume_ep(struct task_struct *tsk)
 {
include/linux/rpal.h +11 -0
index 5912ffec6e28..7657e6c6393b 100644
@@ -350,6 +350,12 @@ struct rpal_sender_data {
 	struct task_struct *receiver;
 };
 
+struct rpal_uds_fdmap_arg {
+	int service_id;
+	int cfd;
+	unsigned long *res;
+};
+
 enum rpal_command_type {
 	RPAL_CMD_GET_API_VERSION_AND_CAP,
 	RPAL_CMD_GET_SERVICE_KEY,
@@ -363,6 +369,7 @@ enum rpal_command_type {
 	RPAL_CMD_REQUEST_SERVICE,
 	RPAL_CMD_RELEASE_SERVICE,
 	RPAL_CMD_GET_SERVICE_PKEY,
+	RPAL_CMD_UDS_FDMAP,
 	RPAL_NR_CMD,
 };
 
@@ -393,6 +400,8 @@ enum rpal_command_type {
 	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_RELEASE_SERVICE, unsigned long)
 #define RPAL_IOCTL_GET_SERVICE_PKEY \
 	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_GET_SERVICE_PKEY, int *)
+#define RPAL_IOCTL_UDS_FDMAP \
+	_IOWR(RPAL_IOCTL_MAGIC, RPAL_CMD_UDS_FDMAP, unsigned long)
 
 #define rpal_for_each_requested_service(rs, idx)                             \
 	for (idx = find_first_bit(rs->requested_service_bitmap, RPAL_NR_ID); \
@@ -594,5 +603,7 @@ int rpal_ep_autoremove_wake_function(wait_queue_entry_t *curr,
 	unsigned int mode, int wake_flags,
 	void *key);
 void rpal_resume_ep(struct task_struct *tsk);
+void *rpal_get_epitemep(wait_queue_entry_t *wait);
+int rpal_get_epitemfd(wait_queue_entry_t *wait);
 int rpal_try_send_events(void *ep, struct rpal_receiver_call_context *rcc);
 #endif
Signature · 1 line
2.20.1
author
#34 lore raw

RFC v2 29/35 RPAL: fix race condition in pkru update

In reply to Bo Li

When setting up MPK, RPAL uses IPIs to notify tasks running on each core in the thread group to modify their PKRU values and update the PKEY fields in all VMA page tables. A race condition exists here: when updating PKRU, the page table updates may not yet be complete. In such cases, writing PKRU permissions at locations that require calling pkru_write_default() (e.g., during signal handling) must not be restricted to a single PKEY, as this would cause PKRU permissions to fail to accommodate both old and new page table PKEY settings.

This patch introduces a pku_on state with values PKU_ON_FALSE, PKU_ON_INIT, and PKU_ON_FINISH, representing the states before, during, and after page table PKEY updates, respectively. For RPAL services, all calls to pkru_write_default() are replaced with rpal_pkru_write_default().

- Before page table setup (PKU_ON_FALSE), rpal_pkru_write_default() directly calls pkru_write_default().
- During page table setup (PKU_ON_INIT), rpal_pkru_write_default() enables permissions for all PKEYs, ensuring the task can access both old and new page tables simultaneously.
- After page table setup completes (PKU_ON_FINISH), rpal_pkru_write_default() tightens permissions to match the updated page tables.

For newly allocated page tables, the new PKEY is only used when pku_on is PKU_ON_FINISH. The mmap lock is used to ensure no race conditions occur during this process.

---
 arch/x86/kernel/cpu/common.c |  4 ++--
 arch/x86/kernel/fpu/core.c   |  4 ++--
 arch/x86/kernel/process.c    |  4 ++--
 arch/x86/rpal/pku.c          | 14 +++++++++++++-
 arch/x86/rpal/service.c      |  2 +-
 include/linux/rpal.h         |  9 ++++++++-
 mm/mmap.c                    |  2 +-
 mm/mprotect.c                |  1 +
 mm/vma.c                     |  2 +-
 9 files changed, 31 insertions(+), 11 deletions(-)
arch/x86/kernel/cpu/common.c +2 -2
index 2678453cdf76..d21f44873b86 100644
@@ -534,8 +534,8 @@ static __always_inline void setup_pku(struct cpuinfo_x86 *c)
 	cr4_set_bits(X86_CR4_PKE);
 	/* Load the default PKRU value */
 #ifdef CONFIG_RPAL_PKU
-	if (rpal_current_service() && rpal_current_service()->pku_on)
-		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	if (rpal_current_service())
+		rpal_pkru_write_default();
 	else
 #endif
 		pkru_write_default();
arch/x86/kernel/fpu/core.c +2 -2
index 251b1ddee726..4b413af0b179 100644
@@ -748,8 +748,8 @@ static inline void restore_fpregs_from_init_fpstate(u64 features_mask)
 		frstor(&init_fpstate.regs.fsave);
 
 #ifdef CONFIG_RPAL_PKU
-	if (rpal_current_service() && rpal_current_service()->pku_on)
-		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	if (rpal_current_service())
+		rpal_pkru_write_default();
 	else
 #endif
 		pkru_write_default();
arch/x86/kernel/process.c +2 -2
index b74de35218f9..898a9e0b23e7 100644
@@ -286,8 +286,8 @@ static void pkru_flush_thread(void)
 	 * the hardware right here (similar to context switch).
 	 */
 #ifdef CONFIG_RPAL_PKU
-	if (rpal_current_service() && rpal_current_service()->pku_on)
-		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	if (rpal_current_service())
+		rpal_pkru_write_default();
 	else
 #endif
 		pkru_write_default();
arch/x86/rpal/pku.c +13 -1
index 26cef324f41f..8e530931fb23 100644
@@ -161,7 +161,7 @@ int rpal_pkey_setup(struct rpal_service *rs, int pkey)
 	rs->pkey = pkey;
 	/* others must see rs->pkey before rs->pku_on */
 	barrier();
-	rs->pku_on = true;
+	rs->pku_on = PKU_ON_INIT;
 	mmap_write_unlock(current->mm);
 	rpal_set_group_pkru(val, RPAL_PKRU_UNION);
 	err = do_rpal_mprotect_pkey(rs->base, RPAL_ADDR_SPACE_SIZE, pkey);
@@ -182,3 +182,15 @@ int rpal_alloc_pkey(struct rpal_service *rs, int pkey)
 
 	return ret;
 }
+
+void rpal_pkru_write_default(void)
+{
+	struct rpal_service *cur = rpal_current_service();
+
+	if (cur->pku_on == PKU_ON_INIT)
+		write_pkru(0);
+	else if (cur->pku_on == PKU_ON_FINISH)
+		write_pkru(rpal_pkey_to_pkru(rpal_current_service()->pkey));
+	else
+		pkru_write_default();
+}
arch/x86/rpal/service.c +1 -1
index 7a83e85cf096..9fd568fa9a29 100644
@@ -210,7 +210,7 @@ struct rpal_service *rpal_register_service(void)
 	init_waitqueue_head(&rs->rpd.rpal_waitqueue);
 #ifdef CONFIG_RPAL_PKU
 	rs->pkey = -1;
-	rs->pku_on = false;
+	rs->pku_on = PKU_ON_FALSE;
 	rpal_service_pku_init();
 #endif
 
include/linux/rpal.h +8 -1
index 7657e6c6393b..16a3c80383f7 100644
@@ -138,6 +138,12 @@ enum rpal_capability {
 	RPAL_CAP_PKU
 };
 
+enum {
+	PKU_ON_FALSE,
+	PKU_ON_INIT,
+	PKU_ON_FINISH,
+};
+
 struct rpal_critical_section {
 	unsigned long ret_begin;
 	unsigned long ret_end;
@@ -245,7 +251,7 @@ struct rpal_service {
 
 #ifdef CONFIG_RPAL_PKU
 	/* pkey */
-	bool pku_on;
+	int pku_on;
 	int pkey;
 #endif
 
@@ -599,6 +605,7 @@ __rpal_switch_to(struct task_struct *prev_p, struct task_struct *next_p);
 asmlinkage __visible void rpal_schedule_tail(struct task_struct *prev);
 int do_rpal_mprotect_pkey(unsigned long start, size_t len, int pkey);
 void rpal_set_pku_schedule_tail(struct task_struct *prev);
+void rpal_pkru_write_default(void);
 int rpal_ep_autoremove_wake_function(wait_queue_entry_t *curr,
 	unsigned int mode, int wake_flags,
 	void *key);
mm/mmap.c +1 -1
index d36ea4ea2bd0..85a4a33491ab 100644
@@ -404,7 +404,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
 	do {
 		struct rpal_service *cur = rpal_current_service();
 
-		if (cur && cur->pku_on)
+		if (cur && cur->pku_on == PKU_ON_FINISH)
 			pkey = cur->pkey;
 	} while (0);
 #endif
mm/mprotect.c +1 -0
index e9ae828e377d..ac162180553e 100644
@@ -938,6 +938,7 @@ int do_rpal_mprotect_pkey(unsigned long start, size_t len, int pkey)
 	}
 	tlb_finish_mmu(&tlb);
 
+	rpal_current_service()->pku_on = PKU_ON_FINISH;
 out:
 	mmap_write_unlock(current->mm);
 	return error;
mm/vma.c +1 -1
index fa9d8f694e6e..57ec99a5969d 100644
@@ -2632,7 +2632,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	struct rpal_service *cur = rpal_current_service();
 	unsigned long vma_pkey_mask;
 
-	if (cur && cur->pku_on) {
+	if (cur && cur->pku_on == PKU_ON_FINISH) {
 		vma_pkey_mask = VM_PKEY_BIT0 | VM_PKEY_BIT1 | VM_PKEY_BIT2 |
 				VM_PKEY_BIT3;
 		flags &= ~vma_pkey_mask;
Signature · 1 line
2.20.1
author
#35 lore raw

RFC v2 30/35 RPAL: fix pkru setup when fork

In reply to Bo Li

When a task performs a fork operation, the PKRU value of the newly forked task is set to the value read from hardware. At this point, if the service is executing rpal_pkey_setup(), the newly forked task has not yet been added to the task list, so PKRU settings cannot be synchronized to the new task. This results in the new task's PKRU not being set to the correct value when it is woken up.

This patch addresses this issue by:

- After the newly forked task is added to the task list, further updating its PKRU value.
- Acquiring a mutex lock to ensure that the PKRU update occurs either before or after the invocation of rpal_pkey_setup(). This avoids race conditions with rpal_pkey_setup() and guarantees that the re-updated PKRU value is always correct.

---
 kernel/fork.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/fork.c +13 -0
index 01cd48eadf68..11cba74d07c8 100644
@@ -2683,6 +2683,19 @@ __latent_entropy struct task_struct *copy_process(
 	syscall_tracepoint_update(p);
 	write_unlock_irq(&tasklist_lock);
 
+#ifdef CONFIG_RPAL_PKU
+	do {
+		struct rpal_service *cur = rpal_current_service();
+
+		if (cur) {
+			/* ensure we are not in rpal_enable_service() */
+			mutex_lock(&cur->mutex);
+			p->thread.pkru = rdpkru();
+			mutex_unlock(&cur->mutex);
+		}
+	} while (0);
+#endif
+
 	if (pidfile)
 		fd_install(pidfd, pidfile);
 
Signature · 1 line
2.20.1
author
#36 lore raw

RFC v2 31/35 RPAL: add receiver waker

In reply to Bo Li

In an RPAL call, the receiver thread is in the TASK_INTERRUPTIBLE state and cannot be awakened, which may lead to missed wakeups. For example, if no kernel event occurs during the entire RPAL call, the receiver thread will remain in the TASK_INTERRUPTIBLE state after the RPAL call completes.

To address this issue, RPAL adds a flag to the receiver whenever it encounters an unawakened state and introduces a "waker" work. The waker work runs automatically on every tick to check for receiver threads that have missed wakeups. If any are found, it wakes them up. For epoll, the waker also checks for pending user mode events and wakes the receiver thread if such events exist.

---
 arch/x86/rpal/internal.h |  4 ++
 arch/x86/rpal/service.c  | 98 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/rpal/thread.c   |  3 ++
 include/linux/rpal.h     | 11 +++++
 kernel/sched/core.c      |  3 ++
 5 files changed, 119 insertions(+)
arch/x86/rpal/internal.h +4 -0
index e03f8a90619d..117357dabdec 100644
@@ -22,6 +22,10 @@ int rpal_enable_service(unsigned long arg);
 int rpal_disable_service(void);
 int rpal_request_service(unsigned long arg);
 int rpal_release_service(u64 key);
+void rpal_insert_wake_list(struct rpal_service *rs,
+			   struct rpal_receiver_data *rrd);
+void rpal_remove_wake_list(struct rpal_service *rs,
+			   struct rpal_receiver_data *rrd);
 
 /* mm.c */
 static inline struct rpal_shared_page *
arch/x86/rpal/service.c +98 -0

116 lines not on this page · raw message

arch/x86/rpal/thread.c +3 -0
index fcc592baaac0..51c9eec639cb 100644
@@ -186,6 +186,8 @@ int rpal_register_receiver(unsigned long addr)
 	current->rpal_rd = rrd;
 	rpal_set_current_thread_flag(RPAL_RECEIVER_BIT);
 
+	rpal_insert_wake_list(cur, rrd);
+
 	atomic_inc(&cur->thread_cnt);
 
 	return 0;
@@ -214,6 +216,7 @@ int rpal_unregister_receiver(void)
 	clear_fs_tsk_map();
 
 	rpal_put_shared_page(rrd->rsp);
+	rpal_remove_wake_list(cur, rrd);
 	rpal_clear_current_thread_flag(RPAL_RECEIVER_BIT);
 	rpal_free_thread_pending(&rrd->rcd);
 	kfree(rrd);
include/linux/rpal.h +11 -0
index 16a3c80383f7..1d8c1bdc90f2 100644
@@ -116,6 +116,7 @@ enum rpal_task_flag_bits {
 	RPAL_RECEIVER_BIT,
 	RPAL_CPU_LOCKED_BIT,
 	RPAL_LAZY_SWITCHED_BIT,
+	RPAL_WAKE_BIT,
 };
 
 enum rpal_receiver_state {
@@ -189,6 +190,12 @@ struct rpal_fsbase_tsk_map {
 	struct task_struct *tsk;
 };
 
+struct rpal_waker_struct {
+	spinlock_t lock;
+	struct list_head wake_head;
+	struct delayed_work waker_work;
+};
+
 /*
  * Each RPAL process (a.k.a RPAL service) should have a pointer to
  * struct rpal_service in all its tasks' task_struct.
@@ -255,6 +262,9 @@ struct rpal_service {
 	int pkey;
 #endif
 
+	/* receiver thread waker */
+	struct rpal_waker_struct waker;
+
 	/* delayed service put work */
 	struct delayed_work delayed_put_work;
 
@@ -347,6 +357,7 @@ struct rpal_receiver_data {
 	struct fd f;
 	struct hrtimer_sleeper ep_sleeper;
 	wait_queue_entry_t ep_wait;
+	struct list_head wake_list;
 };
 
 struct rpal_sender_data {
kernel/sched/core.c +3 -0
index 486d59bdd3fc..c219ada29d34 100644
@@ -3943,6 +3943,7 @@ static bool rpal_check_state(struct task_struct *p)
 		struct rpal_receiver_call_context *rcc = p->rpal_rd->rcc;
 		int state;
 
+		rpal_clear_task_thread_flag(p, RPAL_WAKE_BIT);
 retry:
 		state = atomic_read(&rcc->receiver_state) & RPAL_RECEIVER_STATE_MASK;
 		switch (state) {
@@ -3957,6 +3958,7 @@ static bool rpal_check_state(struct task_struct *p)
 		case RPAL_RECEIVER_STATE_RUNNING:
 			break;
 		case RPAL_RECEIVER_STATE_CALL:
+			rpal_set_task_thread_flag(p, RPAL_WAKE_BIT);
 			ret = false;
 			break;
 		default:
@@ -4522,6 +4524,7 @@ int rpal_try_to_wake_up(struct task_struct *p)
 
 	BUG_ON(READ_ONCE(p->__state) == TASK_RUNNING);
 
+	rpal_clear_task_thread_flag(p, RPAL_WAKE_BIT);
 	scoped_guard (raw_spinlock_irqsave, &p->pi_lock) {
 		smp_mb__after_spinlock();
 		if (!ttwu_state_match(p, TASK_NORMAL, &success))
Signature · 1 line
2.20.1
author
#37 lore raw

RFC v2 32/35 RPAL: fix unknown nmi on AMD CPU

In reply to Bo Li

In Lazy switch, the function event_sched_out() will be called. This function deletes the perf event of the task being scheduled out, causing the active_mask in cpu_hw_events to be cleared. In AMD's NMI handler, if the bit corresponding to active_mask is not set, the CPU will not handle the NMI event, ultimately triggering an unknown NMI error. Additionally, event_sched_out() may call amd_pmu_wait_on_overflow(), leading to a busy wait of up to 50us during lazy switch.

This patch adds two per_cpu variables. rpal_nmi_handle is set when an NMI occurs. When encountering an unknown NMI, this NMI is skipped. rpal_nmi is set before lazy switch and cleared after lazy switch, preventing the busy wait caused by amd_pmu_wait_on_overflow().

---
 arch/x86/events/amd/core.c | 14 ++++++++++++++
 arch/x86/kernel/nmi.c      | 20 ++++++++++++++++++++
 arch/x86/rpal/core.c       | 17 ++++++++++++++++-
 3 files changed, 50 insertions(+), 1 deletion(-)
arch/x86/events/amd/core.c +14 -0
index b20661b8621d..633a9ac4e77c 100644
@@ -719,6 +719,10 @@ static void amd_pmu_wait_on_overflow(int idx)
 	}
 }
 
+#ifdef CONFIG_RPAL
+DEFINE_PER_CPU(bool, rpal_nmi);
+#endif
+
 static void amd_pmu_check_overflow(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -732,6 +736,11 @@ static void amd_pmu_check_overflow(void)
 	if (in_nmi())
 		return;
 
+#ifdef CONFIG_RPAL
+	if (this_cpu_read(rpal_nmi))
+		return;
+#endif
+
 	/*
 	 * Check each counter for overflow and wait for it to be reset by the
 	 * NMI if it has overflowed. This relies on the fact that all active
@@ -807,6 +816,11 @@ static void amd_pmu_disable_event(struct perf_event *event)
 	if (in_nmi())
 		return;
 
+#ifdef CONFIG_RPAL
+	if (this_cpu_read(rpal_nmi))
+		return;
+#endif
+
 	amd_pmu_wait_on_overflow(event->hw.idx);
 }
 
arch/x86/kernel/nmi.c +20 -0
index be93ec7255bf..dd72b6d1c7f9 100644
@@ -351,12 +351,23 @@ NOKPROBE_SYMBOL(unknown_nmi_error);
 
 static DEFINE_PER_CPU(bool, swallow_nmi);
 static DEFINE_PER_CPU(unsigned long, last_nmi_rip);
+#ifdef CONFIG_RPAL
+DEFINE_PER_CPU(bool, rpal_nmi_handle);
+#endif
 
 static noinstr void default_do_nmi(struct pt_regs *regs)
 {
 	unsigned char reason = 0;
 	int handled;
 	bool b2b = false;
+#ifdef CONFIG_RPAL
+	bool rpal_handle = false;
+
+	if (__this_cpu_read(rpal_nmi_handle)) {
+		__this_cpu_write(rpal_nmi_handle, false);
+		rpal_handle = true;
+	}
+#endif
 
 	/*
 	 * Back-to-back NMIs are detected by comparing the RIP of the
@@ -471,6 +482,15 @@ static noinstr void default_do_nmi(struct pt_regs *regs)
 	 */
 	if (b2b && __this_cpu_read(swallow_nmi))
 		__this_cpu_add(nmi_stats.swallow, 1);
+#ifdef CONFIG_RPAL
+	/*
+	 * Lazy switch may clear the bit in active_mask, causing
+	 * nmi event not handled. This will lead to unknown nmi,
+	 * try to avoid this.
+	 */
+	else if (rpal_handle)
+		goto out;
+#endif
 	else
 		unknown_nmi_error(reason, regs);
 
arch/x86/rpal/core.c +16 -1
index 6a22b9faa100..92281b557a6c 100644
@@ -376,11 +376,26 @@ rpal_exception_context_switch(struct pt_regs *regs)
 	return next;
 }
 
+DECLARE_PER_CPU(bool, rpal_nmi_handle);
+DECLARE_PER_CPU(bool, rpal_nmi);
 __visible struct task_struct *rpal_nmi_context_switch(struct pt_regs *regs)
 {
 	struct task_struct *next;
 
-	next = rpal_kernel_context_switch(regs);
+	if (rpal_test_current_thread_flag(RPAL_LAZY_SWITCHED_BIT))
+		rpal_update_fsbase(regs);
+
+	next = rpal_misidentify();
+	if (unlikely(next != NULL)) {
+		next = rpal_fix_critical_section(next, regs);
+		if (next) {
+			__this_cpu_write(rpal_nmi_handle, true);
+			/* avoid wait in amd_pmu_check_overflow */
+			__this_cpu_write(rpal_nmi, true);
+			next = rpal_do_kernel_context_switch(next, regs);
+			__this_cpu_write(rpal_nmi, false);
+		}
+	}
 
 	return next;
 }
Signature · 1 line
2.20.1
author
#38 lore raw

RFC v2 33/35 RPAL: enable time slice correction

In reply to Bo Li

After an RPAL call, the receiver's user mode code executes. However, the kernel incorrectly attributes this CPU time to the sender due to the unchanged kernel context. This results in incorrect runtime statistics.

This patch adds a new member total_time to both rpal_sender_call_context and rpal_receiver_call_context. This member tracks how much runtime ( measured in CPU cycles via rdtsc()) has been incorrectly accounted for. The kernel measures total_time at the entry of __schedule() and corrects the delta in the update_rq_clock_task() function.

Additionally, since RPAL calls occur in user space, runtime statistics are typically calculated by user space. However, when a lazy switch happens, the kernel takes over. To address this, the patch introduces a start_time member to record when an RPAL call is initiated, enabling the kernel to accurately calculate the runtime that needs correction.

---
 arch/x86/rpal/core.c   |  8 ++++++++
 arch/x86/rpal/thread.c |  6 ++++++
 include/linux/rpal.h   |  3 +++
 include/linux/sched.h  |  1 +
 init/init_task.c       |  1 +
 kernel/fork.c          |  1 +
 kernel/sched/core.c    | 42 ++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 62 insertions(+)
arch/x86/rpal/core.c +8 -0
index 92281b557a6c..2ac5d932f69c 100644
@@ -144,6 +144,13 @@ rpal_do_kernel_context_switch(struct task_struct *next, struct pt_regs *regs)
 	struct task_struct *prev = current;
 
 	if (rpal_test_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT)) {
+		struct rpal_receiver_call_context *rcc = next->rpal_rd->rcc;
+		struct rpal_sender_call_context *scc = current->rpal_sd->scc;
+		u64 slice = rdtsc_ordered() - scc->start_time;
+
+		rcc->total_time += slice;
+		scc->total_time += slice;
+
 		rpal_resume_ep(next);
 		current->rpal_sd->receiver = next;
 		rpal_lock_cpu(current);
@@ -169,6 +176,7 @@ rpal_do_kernel_context_switch(struct task_struct *next, struct pt_regs *regs)
 		rpal_schedule(next);
 		rpal_clear_task_thread_flag(prev, RPAL_LAZY_SWITCHED_BIT);
 		prev->rpal_rd->sender = NULL;
+		next->rpal_sd->scc->start_time = rdtsc_ordered();
 	}
 	if (unlikely(!irqs_disabled())) {
 		local_irq_disable();
arch/x86/rpal/thread.c +6 -0
index 51c9eec639cb..5cd0be631521 100644
@@ -99,6 +99,8 @@ int rpal_register_sender(unsigned long addr)
 	rsd->scc = (struct rpal_sender_call_context *)(addr - rsp->user_start +
 						       rsp->kernel_start);
 	rsd->receiver = NULL;
+	rsd->scc->start_time = 0;
+	rsd->scc->total_time = 0;
 
 	current->rpal_sd = rsd;
 	rpal_set_current_thread_flag(RPAL_SENDER_BIT);
@@ -182,6 +184,7 @@ int rpal_register_receiver(unsigned long addr)
 		(struct rpal_receiver_call_context *)(addr - rsp->user_start +
 						      rsp->kernel_start);
 	rrd->sender = NULL;
+	rrd->rcc->total_time = 0;
 
 	current->rpal_rd = rrd;
 	rpal_set_current_thread_flag(RPAL_RECEIVER_BIT);
@@ -289,6 +292,9 @@ int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs,
 				rpal_pkey_to_pkru(rpal_current_service()->pkey),
 				RPAL_PKRU_SET);
 #endif
+			if (!rpal_is_correct_address(rpal_current_service(), regs->ip))
+				/* receiver has crashed */
+				scc->total_time += rdtsc_ordered() - scc->start_time;
 			return 0;
 		}
 	}
include/linux/rpal.h +3 -0
index 1d8c1bdc90f2..f5f4da63f28c 100644
@@ -310,6 +310,7 @@ struct rpal_receiver_call_context {
 	void __user *events;
 	int maxevents;
 	int timeout;
+	int64_t total_time;
 };
 
 /* recovery point for sender */
@@ -325,6 +326,8 @@ struct rpal_sender_call_context {
 	struct rpal_task_context rtc;
 	struct rpal_error_context ec;
 	int sender_id;
+	s64 start_time;
+	s64 total_time;
 };
 
 /* End */
include/linux/sched.h +1 -0
index 5f25cc09fb71..a03113fecdc5 100644
@@ -1663,6 +1663,7 @@ struct task_struct {
 		struct rpal_sender_data *rpal_sd;
 		struct rpal_receiver_data *rpal_rd;
 	};
+	s64 rpal_steal_time;
 #endif
 
 	/* CPU-specific state of this task: */
init/init_task.c +1 -0
index 2eb08b96e66b..3606cf701dfe 100644
@@ -224,6 +224,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
 	.rpal_rs = NULL,
 	.rpal_flag = 0,
 	.rpal_cd = NULL,
+	.rpal_steal_time = 0,
 #endif
 };
 EXPORT_SYMBOL(init_task);
kernel/fork.c +1 -0
index 11cba74d07c8..ff6331a28987 100644
@@ -1222,6 +1222,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 	tsk->rpal_rs = NULL;
 	tsk->rpal_flag = 0;
 	tsk->rpal_cd = NULL;
+	tsk->rpal_steal_time = 0;
 #endif
 	return tsk;
 
kernel/sched/core.c +42 -0

60 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#39 lore raw

RFC v2 34/35 RPAL: enable fast epoll wait

In reply to Bo Li

When a kernel event occurs during an RPAL call and triggers a lazy switch, the kernel context switches from the sender to the receiver. When the receiver later returns from user space to the sender, a second lazy switch is required to switch the kernel context back to the sender. In the current implementation, after the second lazy switch, the receiver returns to user space via rpal_kernel_ret() and then calls epoll_wait() from user space to re-enter the kernel. This causes the receiver to be unable to process epoll events for a long period, degrading performance.

This patch introduces a fast epoll wait feature. During the second lazy switch, the kernel configures epoll-related data structures so that the receiver can directly enter the epoll wait state without first returning to user space and then calling epoll_wait(). The patch adds a new state RPAL_RECEIVER_STATE_READY_LS, which is used to mark that the receiver can transition to RPAL_RECEIVER_STATE_WAIT during the second lazy switch. The kernel then performs this state transition in rpal_lazy_switch_tail().

---
 arch/x86/rpal/core.c |  29 ++++++++++++-
 fs/eventpoll.c       | 101 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/rpal.h |   3 ++
 kernel/sched/core.c  |  13 +++++-
 4 files changed, 143 insertions(+), 3 deletions(-)
arch/x86/rpal/core.c +27 -2
index 2ac5d932f69c..7b6efde23e48 100644
@@ -51,7 +51,25 @@ void rpal_lazy_switch_tail(struct task_struct *tsk)
 		atomic_cmpxchg(&rcc->receiver_state, rpal_build_call_state(tsk->rpal_sd),
 			       RPAL_RECEIVER_STATE_LAZY_SWITCH);
 	} else {
+		/* tsk is receiver */
+		int state;
+
+		rcc = tsk->rpal_rd->rcc;
+		state = atomic_read(&rcc->receiver_state);
+		/* receiver may be scheduled on another cpu after unlock. */
 		rpal_unlock_cpu(tsk);
+		/*
+		 * We must not use RPAL_RECEIVER_STATE_READY instead of
+		 * RPAL_RECEIVER_STATE_READY_LS. As receiver may at
+		 * TASK_RUNNING state and then call epoll_wait() again,
+		 * the state may become RPAL_RECEIVER_STATE_READY, we should
+		 * not changed its state to RPAL_RECEIVER_STATE_WAIT since
+		 * the state is set by another RPAL call.
+		 */
+		if (state == RPAL_RECEIVER_STATE_READY_LS)
+			atomic_cmpxchg(&rcc->receiver_state,
+				       RPAL_RECEIVER_STATE_READY_LS,
+				       RPAL_RECEIVER_STATE_WAIT);
 		rpal_unlock_cpu(current);
 	}
 }
@@ -63,8 +81,14 @@ void rpal_kernel_ret(struct pt_regs *regs)
 	int state;
 
 	if (rpal_test_current_thread_flag(RPAL_RECEIVER_BIT)) {
-		rcc = current->rpal_rd->rcc;
-		regs->ax = rpal_try_send_events(current->rpal_rd->ep, rcc);
+		struct rpal_receiver_data *rrd = current->rpal_rd;
+
+		rcc = rrd->rcc;
+		if (rcc->timeout > 0)
+			hrtimer_cancel(&rrd->ep_sleeper.timer);
+		rpal_remove_ep_wait_list(rrd);
+		regs->ax = rpal_try_send_events(rrd->ep, rcc);
+		fdput(rrd->f);
 		atomic_xchg(&rcc->receiver_state, RPAL_RECEIVER_STATE_KERNEL_RET);
 	} else {
 		tsk = current->rpal_sd->receiver;
@@ -173,6 +197,7 @@ rpal_do_kernel_context_switch(struct task_struct *next, struct pt_regs *regs)
 		 * Otherwise, sender's user context will be corrupted.
 		 */
 		rebuild_receiver_stack(current->rpal_rd, regs);
+		rpal_fast_ep_poll(current->rpal_rd, regs);
 		rpal_schedule(next);
 		rpal_clear_task_thread_flag(prev, RPAL_LAZY_SWITCHED_BIT);
 		prev->rpal_rd->sender = NULL;
fs/eventpoll.c +101 -0

107 lines not on this page · raw message

include/linux/rpal.h +3 -0
index f5f4da63f28c..676113f0ba1f 100644
@@ -126,6 +126,7 @@ enum rpal_receiver_state {
 	RPAL_RECEIVER_STATE_WAIT,
 	RPAL_RECEIVER_STATE_CALL,
 	RPAL_RECEIVER_STATE_LAZY_SWITCH,
+	RPAL_RECEIVER_STATE_READY_LS,
 	RPAL_RECEIVER_STATE_MAX,
 };
 
@@ -627,4 +628,6 @@ void rpal_resume_ep(struct task_struct *tsk);
 void *rpal_get_epitemep(wait_queue_entry_t *wait);
 int rpal_get_epitemfd(wait_queue_entry_t *wait);
 int rpal_try_send_events(void *ep, struct rpal_receiver_call_context *rcc);
+void rpal_remove_ep_wait_list(struct rpal_receiver_data *rrd);
+void rpal_fast_ep_poll(struct rpal_receiver_data *rrd, struct pt_regs *regs);
 #endif
kernel/sched/core.c +12 -1
index d6f8e0d76fc0..1728b04d1387 100644
@@ -3965,6 +3965,11 @@ static bool rpal_check_state(struct task_struct *p)
 		case RPAL_RECEIVER_STATE_LAZY_SWITCH:
 		case RPAL_RECEIVER_STATE_RUNNING:
 			break;
+		/*
+		 * Allow RPAL_RECEIVER_STATE_READY_LS to be woken will cause irq
+		 * being enabled in rpal_unlock_cpu.
+		 */
+		case RPAL_RECEIVER_STATE_READY_LS:
 		case RPAL_RECEIVER_STATE_CALL:
 			rpal_set_task_thread_flag(p, RPAL_WAKE_BIT);
 			ret = false;
@@ -11403,7 +11408,13 @@ void __sched notrace rpal_schedule(struct task_struct *next)
 
 	prev_state = READ_ONCE(prev->__state);
 	if (prev_state) {
-		try_to_block_task(rq, prev, &prev_state);
+		if (!try_to_block_task(rq, prev, &prev_state)) {
+			/*
+			 * As the task enter TASK_RUNNING state, we should clean up
+			 * RPAL_RECEIVER_STATE_READY_LS status.
+			 */
+			rpal_check_ready_state(prev, RPAL_RECEIVER_STATE_READY_LS);
+		}
 		switch_count = &prev->nvcsw;
 	}
 
Signature · 1 line
2.20.1
author
#40 lore raw

RFC v2 35/35 samples/rpal: add RPAL samples

In reply to Bo Li

Added test samples for RPAL (with librpal included). Compile via:

    cd samples/rpal && make

And run it using the following command:

    ./server & ./client

Example output:

    EPOLL: Message length: 32 bytes, Total TSC cycles: 16439927066,
    Message count: 1000000, Average latency: 16439 cycles
    RPAL: Message length: 32 bytes, Total TSC cycles: 2197479484,
    Message count: 1000000, Average latency: 2197 cycles
---
 samples/rpal/Makefile                         |   17 +
 samples/rpal/asm_define.c                     |   14 +
 samples/rpal/client.c                         |  178 ++
 samples/rpal/librpal/asm_define.h             |    6 +
 samples/rpal/librpal/asm_x86_64_rpal_call.S   |   57 +
 samples/rpal/librpal/debug.h                  |   12 +
 samples/rpal/librpal/fiber.c                  |  119 +
 samples/rpal/librpal/fiber.h                  |   64 +
 .../rpal/librpal/jump_x86_64_sysv_elf_gas.S   |   81 +
 .../rpal/librpal/make_x86_64_sysv_elf_gas.S   |   82 +
 .../rpal/librpal/ontop_x86_64_sysv_elf_gas.S  |   84 +
 samples/rpal/librpal/private.h                |  341 +++
 samples/rpal/librpal/rpal.c                   | 2351 +++++++++++++++++
 samples/rpal/librpal/rpal.h                   |  149 ++
 samples/rpal/librpal/rpal_pkru.h              |   78 +
 samples/rpal/librpal/rpal_queue.c             |  239 ++
 samples/rpal/librpal/rpal_queue.h             |   55 +
 samples/rpal/librpal/rpal_x86_64_call_ret.S   |   45 +
 samples/rpal/offset.sh                        |    5 +
 samples/rpal/server.c                         |  249 ++
 20 files changed, 4226 insertions(+)
 create mode 100644 samples/rpal/Makefile
 create mode 100644 samples/rpal/asm_define.c
 create mode 100644 samples/rpal/client.c
 create mode 100644 samples/rpal/librpal/asm_define.h
 create mode 100644 samples/rpal/librpal/asm_x86_64_rpal_call.S
 create mode 100644 samples/rpal/librpal/debug.h
 create mode 100644 samples/rpal/librpal/fiber.c
 create mode 100644 samples/rpal/librpal/fiber.h
 create mode 100644 samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/make_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/private.h
 create mode 100644 samples/rpal/librpal/rpal.c
 create mode 100644 samples/rpal/librpal/rpal.h
 create mode 100644 samples/rpal/librpal/rpal_pkru.h
 create mode 100644 samples/rpal/librpal/rpal_queue.c
 create mode 100644 samples/rpal/librpal/rpal_queue.h
 create mode 100644 samples/rpal/librpal/rpal_x86_64_call_ret.S
 create mode 100755 samples/rpal/offset.sh
 create mode 100644 samples/rpal/server.c
samples/rpal/Makefile new file +17 -0
new file mode 100644
index 000000000000..25627a970028
@@ -0,0 +1,17 @@
+.PHONY: rpal
+
+all: server client offset
+
+offset: asm_define.c
+	$(shell ./offset.sh)
+
+server: server.c librpal/*.c librpal/*.S
+	$(CC) $^ -lpthread -g -o $@
+	@printf "RPAL" | dd of=./server bs=1 count=4 conv=notrunc seek=12
+
+client: client.c librpal/*.c librpal/*.S
+	$(CC) $^ -lpthread -g -o $@
+	@printf "RPAL" | dd of=./client bs=1 count=4 conv=notrunc seek=12
+
+clean:
+	rm server client
samples/rpal/asm_define.c new file +14 -0
new file mode 100644
index 000000000000..6f7731ebc870
@@ -0,0 +1,14 @@
+#include <stddef.h>
+#include "librpal/private.h"
+
+#define DEFINE(sym, val) asm volatile("\n-> " #sym " %0 " #val "\n" : : "i" (val))
+
+static void common(void)
+{
+    DEFINE(RCI_SENDER_TLS_BASE, offsetof(rpal_call_info_t, sender_tls_base));
+    DEFINE(RCI_SENDER_FCTX, offsetof(rpal_call_info_t, sender_fctx));
+    DEFINE(RCI_PKRU, offsetof(rpal_call_info_t, pkru));
+    DEFINE(RC_SENDER_STATE, offsetof(receiver_context_t, sender_state));
+    DEFINE(RET_BEGIN, offsetof(critical_section_t, ret_begin));
+    DEFINE(RET_END, offsetof(critical_section_t, ret_end));
+}
samples/rpal/client.c new file +178 -0

178 lines not on this page · raw message

samples/rpal/librpal/asm_define.h new file +6 -0
new file mode 100644
index 000000000000..bc57586cda58
@@ -0,0 +1,6 @@
+#define RCI_SENDER_TLS_BASE 0
+#define RCI_SENDER_FCTX 16
+#define RCI_PKRU 8
+#define RC_SENDER_STATE 72
+#define RET_BEGIN 0
+#define RET_END 8
samples/rpal/librpal/asm_x86_64_rpal_call.S new file +57 -0

57 lines not on this page · raw message

samples/rpal/librpal/debug.h new file +12 -0
new file mode 100644
index 000000000000..10d2fef8d69a
@@ -0,0 +1,12 @@
+#ifndef RPAL_DEBUG_H
+#define RPAL_DEBUG_H
+
+typedef enum {
+	RPAL_DEBUG_MANAGEMENT = (1 << 0),
+	RPAL_DEBUG_SENDER = (1 << 1),
+	RPAL_DEBUG_RECVER = (1 << 2),
+	RPAL_DEBUG_FIBER = (1 << 3),
+
+	__RPAL_DEBUG_ALL = ~(0ULL),
+} rpal_debug_flag_t;
+#endif
samples/rpal/librpal/fiber.c new file +119 -0

119 lines not on this page · raw message

samples/rpal/librpal/fiber.h new file +64 -0

64 lines not on this page · raw message

samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S new file +81 -0

81 lines not on this page · raw message

samples/rpal/librpal/make_x86_64_sysv_elf_gas.S new file +82 -0

83 lines not on this page · raw message

samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S new file +84 -0

84 lines not on this page · raw message

samples/rpal/librpal/private.h new file +341 -0

341 lines not on this page · raw message

samples/rpal/librpal/rpal.c new file +2351 -0

1,944 lines not on this page · raw message

samples/rpal/librpal/rpal.h new file +149 -0

149 lines not on this page · raw message

samples/rpal/librpal/rpal_pkru.h new file +78 -0

78 lines not on this page · raw message

samples/rpal/librpal/rpal_queue.c new file +239 -0

239 lines not on this page · raw message

samples/rpal/librpal/rpal_queue.h new file +55 -0

55 lines not on this page · raw message

samples/rpal/librpal/rpal_x86_64_call_ret.S new file +45 -0

45 lines not on this page · raw message

samples/rpal/offset.sh new file +5 -0
new file mode 100755
index 000000000000..f5ae77b893e8

Lines of this file are not shown. View the raw message.

samples/rpal/server.c new file +249 -0

249 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.1

Part of the diff is not shown here. View the raw message.

reviewer
#41 lore raw
In reply to Bo Li
Quoted 290 lines · Bo Li

Changelog:

v2:
- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

link to v1:
https://lore.kernel.org/lkml/CAP2HCOmAkRVTci0ObtyW=3v6GFOrt9zCn2NwLUbZ+Di49xkBiw@mail.gmail.com/

--------------------------------------------------------------------------

# Introduction

We mainly apply RPAL to the service mesh architecture widely adopted in modern cloud-native data centers. Before the rise of the service mesh architecture, network functions were usually integrated into monolithic applications as libraries, and the main business programs invoked them through function calls. However, to facilitate the independent development and operation and maintenance of the main business programs and network functions, the service mesh removed the network functions from the main business programs and made them independent processes (called sidecars). Inter-process communication (IPC) is used for interaction between the main business program and the sidecar, and the introduced inter-process communication has led to a sharp increase in resource consumption in cloud-native data centers, and may even occupy more than 10% of the CPU of the entire microservice cluster.

To achieve the efficient function call mechanism of the monolithic architecture under the service mesh architecture, we introduced the RPAL (Running Process As Library) architecture, which implements the sharing of the virtual address space of processes and the switching threads in user mode. Through the analysis of the service mesh architecture, we found that the process memory isolation between the main business program and the sidecar is not particularly important because they are split from one application and were an integral part of the original monolithic application. It is more important for the two processes to be independent of each other because they need to be independently developed and maintained to ensure the architectural advantages of the service mesh. Therefore, RPAL breaks the isolation between processes while preserving the independence between them. We think that RPAL can also be applied to other scenarios featuring sidecar-like architectures, such as distributed file storage systems in LLM infra.

In RPAL architecture, multiple processes share a virtual address space, so this architecture can be regarded as an advanced version of the Linux shared memory mechanism:

1. Traditional shared memory requires two processes to negotiate to ensure the mapping of the same piece of memory. In RPAL architecture, two RPAL processes still need to reach a consensus before they can successfully invoke the relevant system calls of RPAL to share the virtual address space.
2. Traditional shared memory only shares part of the data. However, in RPAL architecture, processes that have established an RPAL communication relationship share a virtual address space, and all user memory (such as data segments and code segments) of each RPAL process is shared among these processes. However, a process cannot access the memory of other processes at any time. We use the MPK mechanism to ensure that the memory of other processes can only be accessed when special RPAL functions are called. Otherwise, a page fault will be triggered.
3. In RPAL architecture, to ensure the consistency of the execution context of the shared code (such as the stack and thread local storage), we further implement the thread context switching in user mode based on the ability to share the virtual address space of different processes, enabling the threads of different processes to directly perform fast switching in user mode without falling into kernel mode for slow switching.

# Background

In traditional inter-process communication (IPC) scenarios, Unix domain sockets are commonly used in conjunction with the epoll() family for event multiplexing. IPC operations involve system calls on both the data and control planes, thereby imposing a non-trivial overhead on the interacting processes. Even when shared memory is employed to optimize the data plane, two data copies still remain. Specifically, data is initially copied from a process's private memory space into the shared memory area, and then it is copied from the shared memory into the private memory of another process.

This poses a question: Is it possible to reduce the overhead of IPC with only minimal modifications at the application level? To address this, we observed that the functionality of IPC, which encompasses data transfer and invocation of the target thread, is similar to a function call, where arguments are passed and the callee function is invoked to process them. Inspired by this analogy, we introduce RPAL (Run Process As Library), a framework designed to enable one process to invoke another as if making a local function call, all without going through the kernel.

# Design

First, let’s formalize RPAL’s core objectives:

1. Data-plane efficiency: Reduce the number of data copies from two (in the shared memory solution) to one.
2. Control-plane optimization: Eliminate the overhead of system calls and kernel's thread switches.
3. Application compatibility: Minimize the modifications to existing applications that utilize Unix domain sockets and the epoll() family.

To attain the first objective, processes that use RPAL share the same virtual address space. So one process can access another's data directly via a data pointer. This means data can be transferred from one process to another with just one copy operation.

To meet the second goal, RPAL relies on the shared address space to do lightweight context switching in user space, which we call an "RPAL call". This allows one process to execute another process's code just like a local function call.

To achieve the third target, RPAL stays compatible with the epoll family of functions, like epoll_create(), epoll_wait(), and epoll_ctl(). If an application uses epoll for IPC, developers can switch to RPAL with just a few small changes. For instance, you can just replace epoll_wait() with rpal_epoll_wait(). The basic epoll procedure, where a process waits for another to write to a monitored descriptor using an epoll file descriptor, still works fine with RPAL.

## Address space sharing

For address space sharing, RPAL partitions the entire userspace virtual address space and allocates non-overlapping memory ranges to each process. On x86_64 architectures, RPAL uses a memory range size covered by a single PUD (Page Upper Directory) entry, which is 512GB. This restricts each process’s virtual address space to 512GB on x86_64, sufficient for most applications in our scenario. The rationale is straightforward:
address space sharing can be simply achieved by copying the PUD from one process’s page table to another’s. So one process can directly use the data pointer to access another's memory.

 |------------| <- 0
 |------------| <- 512 GB
 |  Process A |
 |------------| <- 2*512 GB
 |------------| <- n*512 GB
 |  Process B |
 |------------| <- (n+1)*512 GB
 |------------| <- STACK_TOP
 |  Kernel    |
 |------------|

## RPAL call

We refer to the lightweight userspace context switching mechanism as RPAL call. It enables the caller (or sender) thread of one process to directly switch to the callee (or receiver) thread of another process.

When Process A’s caller thread initiates an RPAL call to Process B’s callee thread, the CPU saves the caller’s context and loads the callee’s context. This enables direct userspace control flow transfer from the caller to the callee. After the callee finishes data processing, the CPU saves Process B’s callee context and switches back to Process A’s caller context, completing a full IPC cycle.

 |------------|                |---------------------|  
 |  Process A |                |  Process B          |
 | |-------|  |                | |-------|           |     
 | | caller| --- RPAL call --> | | callee|    handle |
 | | thread| <------------------ | thread| -> event  |
 | |-------|  |                | |-------|           |
 |------------|                |---------------------|

# Security and compatibility with kernel subsystems

## Memory protection between processes

Since processes using RPAL share the address space, unintended cross-process memory access may occur and corrupt the data of another process. To mitigate this, we leverage Memory Protection Keys (MPK) on x86 architectures.

MPK assigns 4 bits in each page table entry to a "protection key", which is paired with a userspace register (PKRU). The PKRU register defines access permissions for memory regions protected by specific keys (for detailed implementation, refer to the kernel documentation "Memory Protection Keys"). With MPK, even though the address space is shared among processes, cross-process access is restricted: a process can only access the memory protected by a key if its PKRU register is configured with the corresponding permission. This ensures that processes cannot access each other’s memory unless an explicit PKRU configuration is set.

## Page fault handling and TLB flushing

Due to the shared address space architecture, both page fault handling and TLB flushing require careful consideration. For instance, when Process A accesses Process B’s memory, a page fault may occur in Process A's context, but the faulting address belongs to Process B. In this case, we must pass Process B's mm_struct to the page fault handler.

TLB flushing is more complex. When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

## Lazy switch of kernel context

In RPAL, a mismatch may arise between the user context and the kernel context. The RPAL call is designed solely to switch the user context, leaving the kernel context unchanged. For instance, when a RPAL call takes place, transitioning from caller thread to callee thread, and subsequently a system call is initiated within callee thread, the kernel will incorrectly utilize the caller's kernel context (such as the kernel stack) to process the system call.

To resolve context mismatch issues, a kernel context switch is triggered at the kernel entry point when the callee initiates a syscall or an exception/interrupt occurs. This mechanism ensures context consistency before processing system calls, interrupts, or exceptions. We refer to this kernel context switch as a "lazy switch" because it defers the switching operation from the traditional thread switch point to the next kernel entry point.

Lazy switch should be minimized as much as possible, as it significantly degrades performance. We currently utilize RPAL in an RPC framework, in which the RPC sender thread relies on the RPAL call to invoke the RPC receiver thread entirely in user space. In most cases, the receiver thread is free of system calls and the code execution time is relatively short. This characteristic effectively reduces the probability of a lazy switch occurring.

## Time slice correction

After an RPAL call, the callee's user mode code executes. However, the kernel incorrectly attributes this CPU time to the caller due to the unchanged kernel context.

To resolve this, we use the Time Stamp Counter (TSC) register to measure CPU time consumed by the callee thread in user space. The kernel then uses this user-reported timing data to adjust the CPU accounting for both the caller and callee thread, similar to how CPU steal time is implemented.

## Process recovery

Since processes can access each other’s memory, there is a risk that the target process’s memory may become invalid at the access time (e.g., if the target process has exited unexpectedly). The kernel must handle such cases; otherwise, the accessing process could be terminated due to failures originating from another process.

To address this issue, each thread of the process should pre-establish a recovery point when accessing the memory of other processes. When such an invalid access occurs, the thread traps into the kernel. Inside the page fault handler, the kernel restores the user context of the thread to the recovery point. This mechanism ensures that processes maintain mutual independence, preventing cascading failures caused by cross-process memory issues.

# Performance

To quantify the performance improvements driven by RPAL, we measured latency both before and after its deployment. Experiments were conducted on a server equipped with two Intel(R) Xeon(R) Platinum 8336C CPUs (2.30 GHz) and 1 TB of memory. Latency was defined as the duration from when the client thread initiates a message to when the server thread is invoked and receives it.

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

We have applied RPAL to an RPC framework that is widely used in our data center. With RPAL, we have successfully achieved up to 15.5% reduction in the CPU utilization of processes in real-world microservice scenario. The gains primarily stem from minimizing control plane overhead through the utilization of userspace context switches. Additionally, by leveraging address space sharing, the number of memory copies is significantly reduced.

# Future Work

Currently, RPAL requires the MPK (Memory Protection Key) hardware feature, which is supported by a range of Intel CPUs. For AMD architectures, MPK is supported only on the latest processor, specifically, 3th Generation AMD EPYC™ Processors and subsequent generations. Patch sets that extend RPAL support to systems lacking MPK hardware will be provided later.

Quoted 8 lines

Accompanying test programs are also provided in the samples/rpal/ directory. And the user-mode RPAL library, which realizes user-space RPAL call, is in the samples/rpal/librpal directory.

We hope to get some community discussions and feedback on RPAL's optimization approaches and architecture.

Look forward to your comments.

The first time you posted, you got two NACKs (from Dave Hansen and Lorenzo). You didn't reply and now you post this flood of patches? Please don't?

From my end it's also a Big Ol' NACK.

Quoted 122 lines

Bo Li (35):
Kbuild: rpal support
RPAL: add struct rpal_service
RPAL: add service registration interface
RPAL: add member to task_struct and mm_struct
RPAL: enable virtual address space partitions
RPAL: add user interface
RPAL: enable shared page mmap
RPAL: enable sender/receiver registration
RPAL: enable address space sharing
RPAL: allow service enable/disable
RPAL: add service request/release
RPAL: enable service disable notification
RPAL: add tlb flushing support
RPAL: enable page fault handling
RPAL: add sender/receiver state
RPAL: add cpu lock interface
RPAL: add a mapping between fsbase and tasks
sched: pick a specified task
RPAL: add lazy switch main logic
RPAL: add rpal_ret_from_lazy_switch
RPAL: add kernel entry handling for lazy switch
RPAL: rebuild receiver state
RPAL: resume cpumask when fork
RPAL: critical section optimization
RPAL: add MPK initialization and interface
RPAL: enable MPK support
RPAL: add epoll support
RPAL: add rpal_uds_fdmap() support
RPAL: fix race condition in pkru update
RPAL: fix pkru setup when fork
RPAL: add receiver waker
RPAL: fix unknown nmi on AMD CPU
RPAL: enable time slice correction
RPAL: enable fast epoll wait
samples/rpal: add RPAL samples

 arch/x86/Kbuild                               |    2 +
 arch/x86/Kconfig                              |    2 +
 arch/x86/entry/entry_64.S                     |  160 ++
 arch/x86/events/amd/core.c                    |   14 +
 arch/x86/include/asm/pgtable.h                |   25 +
 arch/x86/include/asm/pgtable_types.h          |   11 +
 arch/x86/include/asm/tlbflush.h               |   10 +
 arch/x86/kernel/asm-offsets.c                 |    3 +
 arch/x86/kernel/cpu/common.c                  |    8 +-
 arch/x86/kernel/fpu/core.c                    |    8 +-
 arch/x86/kernel/nmi.c                         |   20 +
 arch/x86/kernel/process.c                     |   25 +-
 arch/x86/kernel/process_64.c                  |  118 +
 arch/x86/mm/fault.c                           |  271 ++
 arch/x86/mm/mmap.c                            |   10 +
 arch/x86/mm/tlb.c                             |  172 ++
 arch/x86/rpal/Kconfig                         |   21 +
 arch/x86/rpal/Makefile                        |    6 +
 arch/x86/rpal/core.c                          |  477 ++++
 arch/x86/rpal/internal.h                      |   69 +
 arch/x86/rpal/mm.c                            |  426 +++
 arch/x86/rpal/pku.c                           |  196 ++
 arch/x86/rpal/proc.c                          |  279 ++
 arch/x86/rpal/service.c                       |  776 ++++++
 arch/x86/rpal/thread.c                        |  313 +++
 fs/binfmt_elf.c                               |   98 +-
 fs/eventpoll.c                                |  320 +++
 fs/exec.c                                     |   11 +
 include/linux/mm_types.h                      |    3 +
 include/linux/rpal.h                          |  633 +++++
 include/linux/sched.h                         |   21 +
 init/init_task.c                              |    6 +
 kernel/exit.c                                 |    5 +
 kernel/fork.c                                 |   32 +
 kernel/sched/core.c                           |  676 +++++
 kernel/sched/fair.c                           |  109 +
 kernel/sched/sched.h                          |    8 +
 mm/mmap.c                                     |   16 +
 mm/mprotect.c                                 |  106 +
 mm/rmap.c                                     |    4 +
 mm/vma.c                                      |   18 +
 samples/rpal/Makefile                         |   17 +
 samples/rpal/asm_define.c                     |   14 +
 samples/rpal/client.c                         |  178 ++
 samples/rpal/librpal/asm_define.h             |    6 +
 samples/rpal/librpal/asm_x86_64_rpal_call.S   |   57 +
 samples/rpal/librpal/debug.h                  |   12 +
 samples/rpal/librpal/fiber.c                  |  119 +
 samples/rpal/librpal/fiber.h                  |   64 +
 .../rpal/librpal/jump_x86_64_sysv_elf_gas.S   |   81 +
 .../rpal/librpal/make_x86_64_sysv_elf_gas.S   |   82 +
 .../rpal/librpal/ontop_x86_64_sysv_elf_gas.S  |   84 +
 samples/rpal/librpal/private.h                |  341 +++
 samples/rpal/librpal/rpal.c                   | 2351 +++++++++++++++++
 samples/rpal/librpal/rpal.h                   |  149 ++
 samples/rpal/librpal/rpal_pkru.h              |   78 +
 samples/rpal/librpal/rpal_queue.c             |  239 ++
 samples/rpal/librpal/rpal_queue.h             |   55 +
 samples/rpal/librpal/rpal_x86_64_call_ret.S   |   45 +
 samples/rpal/offset.sh                        |    5 +
 samples/rpal/server.c                         |  249 ++
 61 files changed, 9710 insertions(+), 4 deletions(-)
 create mode 100644 arch/x86/rpal/Kconfig
 create mode 100644 arch/x86/rpal/Makefile
 create mode 100644 arch/x86/rpal/core.c
 create mode 100644 arch/x86/rpal/internal.h
 create mode 100644 arch/x86/rpal/mm.c
 create mode 100644 arch/x86/rpal/pku.c
 create mode 100644 arch/x86/rpal/proc.c
 create mode 100644 arch/x86/rpal/service.c
 create mode 100644 arch/x86/rpal/thread.c
 create mode 100644 include/linux/rpal.h
 create mode 100644 samples/rpal/Makefile
 create mode 100644 samples/rpal/asm_define.c
 create mode 100644 samples/rpal/client.c
 create mode 100644 samples/rpal/librpal/asm_define.h
 create mode 100644 samples/rpal/librpal/asm_x86_64_rpal_call.S
 create mode 100644 samples/rpal/librpal/debug.h
 create mode 100644 samples/rpal/librpal/fiber.c
 create mode 100644 samples/rpal/librpal/fiber.h
 create mode 100644 samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/make_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/private.h
Quoted 8 lines
 create mode 100644 samples/rpal/librpal/rpal.c
 create mode 100644 samples/rpal/librpal/rpal.h
 create mode 100644 samples/rpal/librpal/rpal_pkru.h
 create mode 100644 samples/rpal/librpal/rpal_queue.c
 create mode 100644 samples/rpal/librpal/rpal_queue.h
 create mode 100644 samples/rpal/librpal/rpal_x86_64_call_ret.S
 create mode 100755 samples/rpal/offset.sh
 create mode 100644 samples/rpal/server.c

Seriously, look at all the files you're touching. All the lines you're changing. All the maintainers you had to CC. All for a random new RPC method you developed. This is _not_ mergeable.

Signature · 1 line
Pedro
#42 lore raw
In reply to Bo Li
Quoted 15 lines

## Address space sharing

For address space sharing, RPAL partitions the entire userspace virtual address space and allocates non-overlapping memory ranges to each process. On x86_64 architectures, RPAL uses a memory range size covered by a single PUD (Page Upper Directory) entry, which is 512GB. This restricts each process’s virtual address space to 512GB on x86_64, sufficient for most applications in our scenario. The rationale is straightforward:
address space sharing can be simply achieved by copying the PUD from one process’s page table to another’s. So one process can directly use the data pointer to access another's memory.

Quoted 10 lines
  |------------| <- 0
  |------------| <- 512 GB
  |  Process A |
  |------------| <- 2*512 GB
  |------------| <- n*512 GB
  |  Process B |
  |------------| <- (n+1)*512 GB
  |------------| <- STACK_TOP
  |  Kernel    |
  |------------|

Oh my.

It reminds me a bit about mshare -- just that mshare tries to do it in a less hacky way..

Quoted 42 lines

## RPAL call

We refer to the lightweight userspace context switching mechanism as RPAL call. It enables the caller (or sender) thread of one process to directly switch to the callee (or receiver) thread of another process.

When Process A’s caller thread initiates an RPAL call to Process B’s callee thread, the CPU saves the caller’s context and loads the callee’s context. This enables direct userspace control flow transfer from the caller to the callee. After the callee finishes data processing, the CPU saves Process B’s callee context and switches back to Process A’s caller context, completing a full IPC cycle.

  |------------|                |---------------------|
  |  Process A |                |  Process B          |
  | |-------|  |                | |-------|           |
  | | caller| --- RPAL call --> | | callee|    handle |
  | | thread| <------------------ | thread| -> event  |
  | |-------|  |                | |-------|           |
  |------------|                |---------------------|

# Security and compatibility with kernel subsystems

## Memory protection between processes

Since processes using RPAL share the address space, unintended cross-process memory access may occur and corrupt the data of another process. To mitigate this, we leverage Memory Protection Keys (MPK) on x86 architectures.

MPK assigns 4 bits in each page table entry to a "protection key", which is paired with a userspace register (PKRU). The PKRU register defines access permissions for memory regions protected by specific keys (for detailed implementation, refer to the kernel documentation "Memory Protection Keys"). With MPK, even though the address space is shared among processes, cross-process access is restricted: a process can only access the memory protected by a key if its PKRU register is configured with the corresponding permission. This ensures that processes cannot access each other’s memory unless an explicit PKRU configuration is set.

Quoted 7 lines

## Page fault handling and TLB flushing

Due to the shared address space architecture, both page fault handling and TLB flushing require careful consideration. For instance, when Process A accesses Process B’s memory, a page fault may occur in Process A's context, but the faulting address belongs to Process B. In this case, we must pass Process B's mm_struct to the page fault handler.

In an mshare region, all faults would be rerouted to the mshare MM either way.

Quoted 7 lines

TLB flushing is more complex. When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

Oh my.

It all reminds me of mshare, just the context switch handling is different (and significantly ... more problematic).

Maybe something could be built on top of mshare, but I'm afraid the real magic is the address space sharing combined with the context switching ... which sounds like a big can of worms.

So in the current form, I understand all the NACKs.

Signature · 3 lines
Cheers,

David / dhildenb
maintainer
#43 lore raw
In reply to Bo Li
Quoted 10 lines · On Fri, 30 May 2025 17:27:28 +0800 Bo Li <[email protected]> wrote:

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

Quoted 3 lines

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

Noted ;)

Quick question:

Quoted 17 lines
 arch/x86/Kbuild                               |    2 +
 arch/x86/Kconfig                              |    2 +
 arch/x86/entry/entry_64.S                     |  160 ++
 arch/x86/events/amd/core.c                    |   14 +
 arch/x86/include/asm/pgtable.h                |   25 +
 arch/x86/include/asm/pgtable_types.h          |   11 +
 arch/x86/include/asm/tlbflush.h               |   10 +
 arch/x86/kernel/asm-offsets.c                 |    3 +
 arch/x86/kernel/cpu/common.c                  |    8 +-
 arch/x86/kernel/fpu/core.c                    |    8 +-
 arch/x86/kernel/nmi.c                         |   20 +
 arch/x86/kernel/process.c                     |   25 +-
 arch/x86/kernel/process_64.c                  |  118 +
 arch/x86/mm/fault.c                           |  271 ++
 arch/x86/mm/mmap.c                            |   10 +
 arch/x86/mm/tlb.c                             |  172 ++
 arch/x86/rpal/Kconfig                         |   21 +
Quoted 8 lines
 arch/x86/rpal/Makefile                        |    6 +
 arch/x86/rpal/core.c                          |  477 ++++
 arch/x86/rpal/internal.h                      |   69 +
 arch/x86/rpal/mm.c                            |  426 +++
 arch/x86/rpal/pku.c                           |  196 ++
 arch/x86/rpal/proc.c                          |  279 ++
 arch/x86/rpal/service.c                       |  776 ++++++
 arch/x86/rpal/thread.c                        |  313 +++

The changes are very x86-heavy. Is that a necessary thing? Would another architecture need to implement a similar amount to enable RPAL? IOW, how much of the above could be made arch-neutral?

#44 lore raw
In reply to Bo Li
Quoted 19 lines · Bo Li

# Performance

To quantify the performance improvements driven by RPAL, we measured latency both before and after its deployment. Experiments were conducted on a server equipped with two Intel(R) Xeon(R) Platinum 8336C CPUs (2.30 GHz) and 1 TB of memory. Latency was defined as the duration from when the client thread initiates a message to when the server thread is invoked and receives it.

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

Quoted 3 lines

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

No, these results do not necessarily confirm that.

19,616 cycles per message on a vanilla kernel on a 2.3 GHz CPU suggests a messaging performance of 117k messages/second or 8.5 usecs/message, which is *way* beyond typical kernel interprocess communication latencies on comparable CPUs:

  root@localhost:~# taskset 1 perf bench sched pipe
  # Running 'sched/pipe' benchmark:
  # Executed 1000000 pipe operations between two processes
       Total time: 2.790 [sec]

       2.790614 usecs/op
         358344 ops/sec

And my 2.8 usecs result was from a kernel running inside a KVM sandbox
...

( I used 'taskset' to bind the benchmark to a single CPU, to remove any
inter-CPU migration noise from the measurement. )

The scheduler parts of your series simply try to remove much of scheduler and context switching functionality to create a special fast-path with no FPU context switching and TLB flushing AFAICS, for the purposes of message latency benchmarking in essence, and you then compare it against the full scheduling and MM context switching costs of full-blown Linux processes.

I'm not convinced, at all, that this many changes are required to speed up the usecase you are trying to optimize:

Quoted 1 line
 61 files changed, 9710 insertions(+), 4 deletions(-)

Nor am I convinced that 9,700 lines of *new* code of a parallel facility are needed, crudely wrapped in 1970s technology (#ifdefs), instead of optimizing/improving facilities we already have...

So NAK for the scheduler bits, until proven otherwise (and presented in a clean fashion, which the current series is very far from).

I'll be the first one to acknowledge that our process and MM context switching overhead is too high and could be improved, and I have no objections against the general goal of improving Linux inter-process messaging performance either, I only NAK this particular implementation/approach.

Thanks,

Ingo

#45 lore raw
In reply to Bo Li
Quoted 10 lines · Bo Li

Changelog:

v2:
- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

link to v1:
https://lore.kernel.org/lkml/CAP2HCOmAkRVTci0ObtyW=3v6GFOrt9zCn2NwLUbZ+Di49xkBiw@mail.gmail.com/

Okay,

First of all, I agree with most of the other reviewers that this is insane. Second of all, calling this "optimize cost of inter-process communication" is *extremely* misleading, to the point that one could worry about it being malicious.

What you are doing is attempting to provide isolation between threads running in the same memory space. *By definition* those are not processes.

Secondly, doing function calls from one thread to another in the same memory space isn't really IPC at all, as the scheduler is not involved.

Third, this is something that should be possible to do entirely in user space (mostly in a modified libc). Most of the facilities that you seem to implement already have equivalents (/dev/shm, ET_DYN, ld.so, ...)

This isn't a new idea; this is where the microkernel people eventually ended up when they tried to get performant. It didn't work well for the same reason -- without involving the kernel (or dedicated hardware facilities; x86 segments and MPK are *not* designed for this), the isolation *can't* be enforced. You can, of course, have a kernel interface to switch the address space around -- and you have just (re)invented processes.

From what I can see, a saner version of this would probably be something like a sched_yield_to(X) system call, basically a request to the scheduler "if possible, give the rest of my time slice to process/thread <X>, as if I had been continuing to run." The rest of the communication can be done with shared memory.

The other option is that if you actually are OK with your workloads living in the same privilege domain to simply use threads.

If this somehow isn't what you're doing, and I (and others) have somehow misread the intentions entirely, we will need a whole lot of additional explanations.

-hpa

-hpa