Skip to content

RFC optimize cost of inter-process communication

Bo Li linux-fsdevel linux-mm linux-perf-users lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v2 RFC Latest35 patches 10 repliesIn review

Changes in v2 from the cover letter

- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/35Kbuild: rpal supportarch/x86/Kbuild arch/x86/Kconfig arch/x86/rpal/Kconfig +1 more+15 -00 replies
2/35RPAL: add struct rpal_servicearch/x86/rpal/Makefile arch/x86/rpal/core.c arch/x86/rpal/internal.h +2 more+149 -00 replies
3/35RPAL: add service registration interfacearch/x86/rpal/service.c include/linux/rpal.h+161 -00 replies
4/35RPAL: add member to task_struct and mm_structarch/x86/rpal/service.c fs/exec.c include/linux/mm_types.h +5 more+145 -40 replies
5/35RPAL: enable virtual address space partitionsarch/x86/mm/mmap.c arch/x86/rpal/Makefile arch/x86/rpal/mm.c +3 more+251 -20 replies
6/35RPAL: add user interfacearch/x86/rpal/Makefile arch/x86/rpal/core.c arch/x86/rpal/internal.h +2 more+112 -10 replies
7/35RPAL: enable shared page mmaparch/x86/rpal/internal.h arch/x86/rpal/mm.c arch/x86/rpal/proc.c +3 more+191 -00 replies
8/35RPAL: enable sender/receiver registrationarch/x86/rpal/Makefile arch/x86/rpal/internal.h arch/x86/rpal/proc.c +6 more+289 -10 replies
9/35RPAL: enable address space sharingarch/x86/include/asm/pgtable.h arch/x86/include/asm/pgtable_types.h arch/x86/rpal/internal.h +1 more+213 -00 replies
10/35RPAL: allow service enable/disablearch/x86/rpal/internal.h arch/x86/rpal/proc.c arch/x86/rpal/service.c +1 more+76 -00 replies
11/35RPAL: add service request/releasearch/x86/rpal/internal.h arch/x86/rpal/proc.c arch/x86/rpal/service.c +1 more+316 -20 replies
12/35RPAL: enable service disable notificationarch/x86/rpal/proc.c arch/x86/rpal/service.c include/linux/rpal.h+106 -20 replies
13/35RPAL: add tlb flushing supportarch/x86/include/asm/tlbflush.h arch/x86/mm/tlb.c arch/x86/rpal/internal.h +2 more+198 -30 replies
14/35RPAL: enable page fault handlingarch/x86/mm/fault.c arch/x86/rpal/mm.c arch/x86/rpal/service.c +2 more+412 -211 reply
15/35RPAL: add sender/receiver statearch/x86/kernel/process_64.c arch/x86/rpal/internal.h include/linux/rpal.h +1 more+230 -00 replies
16/35RPAL: add cpu lock interfacearch/x86/rpal/core.c arch/x86/rpal/thread.c include/linux/rpal.h +1 more+149 -00 replies
17/35RPAL: add a mapping between fsbase and tasksarch/x86/rpal/core.c arch/x86/rpal/thread.c include/linux/rpal.h+156 -10 replies
18/35sched: pick a specified taskkernel/sched/core.c kernel/sched/fair.c kernel/sched/sched.h+329 -00 replies
19/35RPAL: add lazy switch main logicarch/x86/kernel/process_64.c include/linux/rpal.h kernel/sched/core.c+204 -00 replies
20/35RPAL: add rpal_ret_from_lazy_switcharch/x86/entry/entry_64.S arch/x86/rpal/core.c include/linux/rpal.h +1 more+94 -40 replies
21/35RPAL: add kernel entry handling for lazy switcharch/x86/entry/entry_64.S arch/x86/kernel/asm-offsets.c arch/x86/rpal/core.c +1 more+283 -00 replies
22/35RPAL: rebuild receiver statearch/x86/rpal/thread.c+43 -10 replies
23/35RPAL: resume cpumask when forkarch/x86/kernel/process.c+18 -00 replies
24/35RPAL: critical section optimizationarch/x86/rpal/core.c arch/x86/rpal/service.c include/linux/rpal.h+104 -20 replies
25/35RPAL: add MPK initialization and interfacearch/x86/rpal/Kconfig arch/x86/rpal/Makefile arch/x86/rpal/core.c +6 more+181 -10 replies
26/35RPAL: enable MPK supportarch/x86/kernel/cpu/common.c arch/x86/kernel/fpu/core.c arch/x86/kernel/process.c +10 more+310 -51 reply
27/35RPAL: add epoll supportarch/x86/rpal/core.c fs/eventpoll.c include/linux/rpal.h +1 more+242 -00 replies
28/35RPAL: add rpal_uds_fdmap() supportarch/x86/rpal/internal.h arch/x86/rpal/proc.c fs/eventpoll.c +1 more+150 -00 replies
29/35RPAL: fix race condition in pkru updatearch/x86/kernel/cpu/common.c arch/x86/kernel/fpu/core.c arch/x86/kernel/process.c +6 more+31 -110 replies
30/35RPAL: fix pkru setup when forkkernel/fork.c+13 -00 replies
31/35RPAL: add receiver wakerarch/x86/rpal/internal.h arch/x86/rpal/service.c arch/x86/rpal/thread.c +2 more+119 -00 replies
32/35RPAL: fix unknown nmi on AMD CPUarch/x86/events/amd/core.c arch/x86/kernel/nmi.c arch/x86/rpal/core.c+50 -10 replies
33/35RPAL: enable time slice correctionarch/x86/rpal/core.c arch/x86/rpal/thread.c include/linux/rpal.h +4 more+62 -00 replies
34/35RPAL: enable fast epoll waitarch/x86/rpal/core.c fs/eventpoll.c include/linux/rpal.h +1 more+143 -30 replies
35/35samples/rpal: add RPAL samplessamples/rpal/Makefile samples/rpal/asm_define.c samples/rpal/client.c +17 more+4226 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [RFC v2 00/35] optimize cost of inter-process communication

Changelog:

v2:
- Port the RPAL functions to the latest v6.15 kernel.
- Add a supplementary introduction to the application scenarios and security considerations of RPAL.

link to v1:
https://lore.kernel.org/lkml/CAP2HCOmAkRVTci0ObtyW=3v6GFOrt9zCn2NwLUbZ+Di49xkBiw@mail.gmail.com/

--------------------------------------------------------------------------

# Introduction

We mainly apply RPAL to the service mesh architecture widely adopted in modern cloud-native data centers. Before the rise of the service mesh architecture, network functions were usually integrated into monolithic applications as libraries, and the main business programs invoked them through function calls. However, to facilitate the independent development and operation and maintenance of the main business programs and network functions, the service mesh removed the network functions from the main business programs and made them independent processes (called sidecars). Inter-process communication (IPC) is used for interaction between the main business program and the sidecar, and the introduced inter-process communication has led to a sharp increase in resource consumption in cloud-native data centers, and may even occupy more than 10% of the CPU of the entire microservice cluster.

To achieve the efficient function call mechanism of the monolithic architecture under the service mesh architecture, we introduced the RPAL (Running Process As Library) architecture, which implements the sharing of the virtual address space of processes and the switching threads in user mode. Through the analysis of the service mesh architecture, we found that the process memory isolation between the main business program and the sidecar is not particularly important because they are split from one application and were an integral part of the original monolithic application. It is more important for the two processes to be independent of each other because they need to be independently developed and maintained to ensure the architectural advantages of the service mesh. Therefore, RPAL breaks the isolation between processes while preserving the independence between them. We think that RPAL can also be applied to other scenarios featuring sidecar-like architectures, such as distributed file storage systems in LLM infra.

In RPAL architecture, multiple processes share a virtual address space, so this architecture can be regarded as an advanced version of the Linux shared memory mechanism:

1. Traditional shared memory requires two processes to negotiate to ensure the mapping of the same piece of memory. In RPAL architecture, two RPAL processes still need to reach a consensus before they can successfully invoke the relevant system calls of RPAL to share the virtual address space.
2. Traditional shared memory only shares part of the data. However, in RPAL architecture, processes that have established an RPAL communication relationship share a virtual address space, and all user memory (such as data segments and code segments) of each RPAL process is shared among these processes. However, a process cannot access the memory of other processes at any time. We use the MPK mechanism to ensure that the memory of other processes can only be accessed when special RPAL functions are called. Otherwise, a page fault will be triggered.
3. In RPAL architecture, to ensure the consistency of the execution context of the shared code (such as the stack and thread local storage), we further implement the thread context switching in user mode based on the ability to share the virtual address space of different processes, enabling the threads of different processes to directly perform fast switching in user mode without falling into kernel mode for slow switching.

# Background

In traditional inter-process communication (IPC) scenarios, Unix domain sockets are commonly used in conjunction with the epoll() family for event multiplexing. IPC operations involve system calls on both the data and control planes, thereby imposing a non-trivial overhead on the interacting processes. Even when shared memory is employed to optimize the data plane, two data copies still remain. Specifically, data is initially copied from a process's private memory space into the shared memory area, and then it is copied from the shared memory into the private memory of another process.

This poses a question: Is it possible to reduce the overhead of IPC with only minimal modifications at the application level? To address this, we observed that the functionality of IPC, which encompasses data transfer and invocation of the target thread, is similar to a function call, where arguments are passed and the callee function is invoked to process them. Inspired by this analogy, we introduce RPAL (Run Process As Library), a framework designed to enable one process to invoke another as if making a local function call, all without going through the kernel.

# Design

First, let’s formalize RPAL’s core objectives:

1. Data-plane efficiency: Reduce the number of data copies from two (in the shared memory solution) to one.
2. Control-plane optimization: Eliminate the overhead of system calls and kernel's thread switches.
3. Application compatibility: Minimize the modifications to existing applications that utilize Unix domain sockets and the epoll() family.

To attain the first objective, processes that use RPAL share the same virtual address space. So one process can access another's data directly via a data pointer. This means data can be transferred from one process to another with just one copy operation.

To meet the second goal, RPAL relies on the shared address space to do lightweight context switching in user space, which we call an "RPAL call". This allows one process to execute another process's code just like a local function call.

To achieve the third target, RPAL stays compatible with the epoll family of functions, like epoll_create(), epoll_wait(), and epoll_ctl(). If an application uses epoll for IPC, developers can switch to RPAL with just a few small changes. For instance, you can just replace epoll_wait() with rpal_epoll_wait(). The basic epoll procedure, where a process waits for another to write to a monitored descriptor using an epoll file descriptor, still works fine with RPAL.

## Address space sharing

For address space sharing, RPAL partitions the entire userspace virtual address space and allocates non-overlapping memory ranges to each process. On x86_64 architectures, RPAL uses a memory range size covered by a single PUD (Page Upper Directory) entry, which is 512GB. This restricts each process’s virtual address space to 512GB on x86_64, sufficient for most applications in our scenario. The rationale is straightforward:
address space sharing can be simply achieved by copying the PUD from one process’s page table to another’s. So one process can directly use the data pointer to access another's memory.

 |------------| <- 0
 |------------| <- 512 GB
 |  Process A |
 |------------| <- 2*512 GB
 |------------| <- n*512 GB
 |  Process B |
 |------------| <- (n+1)*512 GB
 |------------| <- STACK_TOP
 |  Kernel    |
 |------------|

## RPAL call

We refer to the lightweight userspace context switching mechanism as RPAL call. It enables the caller (or sender) thread of one process to directly switch to the callee (or receiver) thread of another process.

When Process A’s caller thread initiates an RPAL call to Process B’s callee thread, the CPU saves the caller’s context and loads the callee’s context. This enables direct userspace control flow transfer from the caller to the callee. After the callee finishes data processing, the CPU saves Process B’s callee context and switches back to Process A’s caller context, completing a full IPC cycle.

 |------------|                |---------------------|  
 |  Process A |                |  Process B          |
 | |-------|  |                | |-------|           |     
 | | caller| --- RPAL call --> | | callee|    handle |
 | | thread| <------------------ | thread| -> event  |
 | |-------|  |                | |-------|           |
 |------------|                |---------------------|

# Security and compatibility with kernel subsystems

## Memory protection between processes

Since processes using RPAL share the address space, unintended cross-process memory access may occur and corrupt the data of another process. To mitigate this, we leverage Memory Protection Keys (MPK) on x86 architectures.

MPK assigns 4 bits in each page table entry to a "protection key", which is paired with a userspace register (PKRU). The PKRU register defines access permissions for memory regions protected by specific keys (for detailed implementation, refer to the kernel documentation "Memory Protection Keys"). With MPK, even though the address space is shared among processes, cross-process access is restricted: a process can only access the memory protected by a key if its PKRU register is configured with the corresponding permission. This ensures that processes cannot access each other’s memory unless an explicit PKRU configuration is set.

## Page fault handling and TLB flushing

Due to the shared address space architecture, both page fault handling and TLB flushing require careful consideration. For instance, when Process A accesses Process B’s memory, a page fault may occur in Process A's context, but the faulting address belongs to Process B. In this case, we must pass Process B's mm_struct to the page fault handler.

TLB flushing is more complex. When a thread flushes the TLB, since the address space is shared, not only other threads in the current process but also other processes that share the address space may access the corresponding memory (related to the TLB flush). Therefore, the cpuset used for TLB flushing should be the union of the mm_cpumasks of all processes that share the address space.

## Lazy switch of kernel context

In RPAL, a mismatch may arise between the user context and the kernel context. The RPAL call is designed solely to switch the user context, leaving the kernel context unchanged. For instance, when a RPAL call takes place, transitioning from caller thread to callee thread, and subsequently a system call is initiated within callee thread, the kernel will incorrectly utilize the caller's kernel context (such as the kernel stack) to process the system call.

To resolve context mismatch issues, a kernel context switch is triggered at the kernel entry point when the callee initiates a syscall or an exception/interrupt occurs. This mechanism ensures context consistency before processing system calls, interrupts, or exceptions. We refer to this kernel context switch as a "lazy switch" because it defers the switching operation from the traditional thread switch point to the next kernel entry point.

Lazy switch should be minimized as much as possible, as it significantly degrades performance. We currently utilize RPAL in an RPC framework, in which the RPC sender thread relies on the RPAL call to invoke the RPC receiver thread entirely in user space. In most cases, the receiver thread is free of system calls and the code execution time is relatively short. This characteristic effectively reduces the probability of a lazy switch occurring.

## Time slice correction

After an RPAL call, the callee's user mode code executes. However, the kernel incorrectly attributes this CPU time to the caller due to the unchanged kernel context.

To resolve this, we use the Time Stamp Counter (TSC) register to measure CPU time consumed by the callee thread in user space. The kernel then uses this user-reported timing data to adjust the CPU accounting for both the caller and callee thread, similar to how CPU steal time is implemented.

## Process recovery

Since processes can access each other’s memory, there is a risk that the target process’s memory may become invalid at the access time (e.g., if the target process has exited unexpectedly). The kernel must handle such cases; otherwise, the accessing process could be terminated due to failures originating from another process.

To address this issue, each thread of the process should pre-establish a recovery point when accessing the memory of other processes. When such an invalid access occurs, the thread traps into the kernel. Inside the page fault handler, the kernel restores the user context of the thread to the recovery point. This mechanism ensures that processes maintain mutual independence, preventing cascading failures caused by cross-process memory issues.

# Performance

To quantify the performance improvements driven by RPAL, we measured latency both before and after its deployment. Experiments were conducted on a server equipped with two Intel(R) Xeon(R) Platinum 8336C CPUs (2.30 GHz) and 1 TB of memory. Latency was defined as the duration from when the client thread initiates a message to when the server thread is invoked and receives it.

During testing, the client transmitted 1 million 32-byte messages, and we computed the per-message average latency. The results are as follows:

*****************
Without RPAL: Message length: 32 bytes, Total TSC cycles: 19616222534,
Message count: 1000000, Average latency: 19616 cycles
With RPAL: Message length: 32 bytes, Total TSC cycles: 1703459326,
Message count: 1000000, Average latency: 1703 cycles
*****************

These results confirm that RPAL delivers substantial latency improvements over the current epoll implementation—achieving a 17,913-cycle reduction (an ~91.3% improvement) for 32-byte messages.

We have applied RPAL to an RPC framework that is widely used in our data center. With RPAL, we have successfully achieved up to 15.5% reduction in the CPU utilization of processes in real-world microservice scenario. The gains primarily stem from minimizing control plane overhead through the utilization of userspace context switches. Additionally, by leveraging address space sharing, the number of memory copies is significantly reduced.

# Future Work

Currently, RPAL requires the MPK (Memory Protection Key) hardware feature, which is supported by a range of Intel CPUs. For AMD architectures, MPK is supported only on the latest processor, specifically, 3th Generation AMD EPYC™ Processors and subsequent generations. Patch sets that extend RPAL support to systems lacking MPK hardware will be provided later.

Accompanying test programs are also provided in the samples/rpal/ directory. And the user-mode RPAL library, which realizes user-space RPAL call, is in the samples/rpal/librpal directory.

We hope to get some community discussions and feedback on RPAL's optimization approaches and architecture.

Look forward to your comments.

Bo Li (35):
Kbuild: rpal support
RPAL: add struct rpal_service
RPAL: add service registration interface
RPAL: add member to task_struct and mm_struct
RPAL: enable virtual address space partitions
RPAL: add user interface
RPAL: enable shared page mmap
RPAL: enable sender/receiver registration
RPAL: enable address space sharing
RPAL: allow service enable/disable
RPAL: add service request/release
RPAL: enable service disable notification
RPAL: add tlb flushing support
RPAL: enable page fault handling
RPAL: add sender/receiver state
RPAL: add cpu lock interface
RPAL: add a mapping between fsbase and tasks
sched: pick a specified task
RPAL: add lazy switch main logic
RPAL: add rpal_ret_from_lazy_switch
RPAL: add kernel entry handling for lazy switch
RPAL: rebuild receiver state
RPAL: resume cpumask when fork
RPAL: critical section optimization
RPAL: add MPK initialization and interface
RPAL: enable MPK support
RPAL: add epoll support
RPAL: add rpal_uds_fdmap() support
RPAL: fix race condition in pkru update
RPAL: fix pkru setup when fork
RPAL: add receiver waker
RPAL: fix unknown nmi on AMD CPU
RPAL: enable time slice correction
RPAL: enable fast epoll wait
samples/rpal: add RPAL samples

 arch/x86/Kbuild                               |    2 +
 arch/x86/Kconfig                              |    2 +
 arch/x86/entry/entry_64.S                     |  160 ++
 arch/x86/events/amd/core.c                    |   14 +
 arch/x86/include/asm/pgtable.h                |   25 +
 arch/x86/include/asm/pgtable_types.h          |   11 +
 arch/x86/include/asm/tlbflush.h               |   10 +
 arch/x86/kernel/asm-offsets.c                 |    3 +
 arch/x86/kernel/cpu/common.c                  |    8 +-
 arch/x86/kernel/fpu/core.c                    |    8 +-
 arch/x86/kernel/nmi.c                         |   20 +
 arch/x86/kernel/process.c                     |   25 +-
 arch/x86/kernel/process_64.c                  |  118 +
 arch/x86/mm/fault.c                           |  271 ++
 arch/x86/mm/mmap.c                            |   10 +
 arch/x86/mm/tlb.c                             |  172 ++
 arch/x86/rpal/Kconfig                         |   21 +
 arch/x86/rpal/Makefile                        |    6 +
 arch/x86/rpal/core.c                          |  477 ++++
 arch/x86/rpal/internal.h                      |   69 +
 arch/x86/rpal/mm.c                            |  426 +++
 arch/x86/rpal/pku.c                           |  196 ++
 arch/x86/rpal/proc.c                          |  279 ++
 arch/x86/rpal/service.c                       |  776 ++++++
 arch/x86/rpal/thread.c                        |  313 +++
 fs/binfmt_elf.c                               |   98 +-
 fs/eventpoll.c                                |  320 +++
 fs/exec.c                                     |   11 +
 include/linux/mm_types.h                      |    3 +
 include/linux/rpal.h                          |  633 +++++
 include/linux/sched.h                         |   21 +
 init/init_task.c                              |    6 +
 kernel/exit.c                                 |    5 +
 kernel/fork.c                                 |   32 +
 kernel/sched/core.c                           |  676 +++++
 kernel/sched/fair.c                           |  109 +
 kernel/sched/sched.h                          |    8 +
 mm/mmap.c                                     |   16 +
 mm/mprotect.c                                 |  106 +
 mm/rmap.c                                     |    4 +
 mm/vma.c                                      |   18 +
 samples/rpal/Makefile                         |   17 +
 samples/rpal/asm_define.c                     |   14 +
 samples/rpal/client.c                         |  178 ++
 samples/rpal/librpal/asm_define.h             |    6 +
 samples/rpal/librpal/asm_x86_64_rpal_call.S   |   57 +
 samples/rpal/librpal/debug.h                  |   12 +
 samples/rpal/librpal/fiber.c                  |  119 +
 samples/rpal/librpal/fiber.h                  |   64 +
 .../rpal/librpal/jump_x86_64_sysv_elf_gas.S   |   81 +
 .../rpal/librpal/make_x86_64_sysv_elf_gas.S   |   82 +
 .../rpal/librpal/ontop_x86_64_sysv_elf_gas.S  |   84 +
 samples/rpal/librpal/private.h                |  341 +++
 samples/rpal/librpal/rpal.c                   | 2351 +++++++++++++++++
 samples/rpal/librpal/rpal.h                   |  149 ++
 samples/rpal/librpal/rpal_pkru.h              |   78 +
 samples/rpal/librpal/rpal_queue.c             |  239 ++
 samples/rpal/librpal/rpal_queue.h             |   55 +
 samples/rpal/librpal/rpal_x86_64_call_ret.S   |   45 +
 samples/rpal/offset.sh                        |    5 +
 samples/rpal/server.c                         |  249 ++
 61 files changed, 9710 insertions(+), 4 deletions(-)
 create mode 100644 arch/x86/rpal/Kconfig
 create mode 100644 arch/x86/rpal/Makefile
 create mode 100644 arch/x86/rpal/core.c
 create mode 100644 arch/x86/rpal/internal.h
 create mode 100644 arch/x86/rpal/mm.c
 create mode 100644 arch/x86/rpal/pku.c
 create mode 100644 arch/x86/rpal/proc.c
 create mode 100644 arch/x86/rpal/service.c
 create mode 100644 arch/x86/rpal/thread.c
 create mode 100644 include/linux/rpal.h
 create mode 100644 samples/rpal/Makefile
 create mode 100644 samples/rpal/asm_define.c
 create mode 100644 samples/rpal/client.c
 create mode 100644 samples/rpal/librpal/asm_define.h
 create mode 100644 samples/rpal/librpal/asm_x86_64_rpal_call.S
 create mode 100644 samples/rpal/librpal/debug.h
 create mode 100644 samples/rpal/librpal/fiber.c
 create mode 100644 samples/rpal/librpal/fiber.h
 create mode 100644 samples/rpal/librpal/jump_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/make_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/ontop_x86_64_sysv_elf_gas.S
 create mode 100644 samples/rpal/librpal/private.h
 create mode 100644 samples/rpal/librpal/rpal.c
 create mode 100644 samples/rpal/librpal/rpal.h
 create mode 100644 samples/rpal/librpal/rpal_pkru.h
 create mode 100644 samples/rpal/librpal/rpal_queue.c
 create mode 100644 samples/rpal/librpal/rpal_queue.h
 create mode 100644 samples/rpal/librpal/rpal_x86_64_call_ret.S
 create mode 100755 samples/rpal/offset.sh
 create mode 100644 samples/rpal/server.c
Signature · 1 line
2.20.1