Skip to content

RFC io_uring: add IPC channel infrastructure

Daniel Hodges io-uring lkml llvm oe-kbuild-all 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest2 patches 8 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2io_uring: add high-performance IPC channel infrastructureinclude/linux/io_uring_types.h include/uapi/linux/io_uring.h io_uring/Kconfig +6 more+1309 -00 replies
2/2selftests/ipc: Add io_uring IPC selftestMAINTAINERS tools/testing/selftests/ipc/Makefile tools/testing/selftests/ipc/io_uring_ipc.c+1267 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/2] io_uring: add IPC channel infrastructure

io_uring currently lacks a dedicated mechanism for efficient inter-process communication. Applications needing low-latency IPC must use pipes, Unix domain sockets, or hand-roll shared memory with manual synchronization -- none of which integrate with the io_uring completion model or offer built-in fan-out semantics.

This series adds an IPC channel primitive to io_uring that provides:

- Shared memory ring buffer for zero-copy-style message passing
- Lock-free CAS-based producer (no mutex on the send hot path)
- RCU-based subscriber lookup on send/recv paths
- Three delivery modes:
* Unicast: single consumer, shared consumer.head via cmpxchg
* Broadcast: all subscribers receive every message via per-subscriber local_head tracking
* Multicast: round-robin distribution across receivers with cached recv_count for O(1) target selection
- Lazy broadcast consumer.head advancement (amortized O(N) scan every 16 messages instead of per-recv)
- Channel-based design supporting multiple subscribers across processes via anonymous file + mmap
- Permission model based on Unix file permissions (mode bits)
- Four registration commands: CREATE, ATTACH, DETACH, DESTROY
- Two new opcodes: IORING_OP_IPC_SEND and IORING_OP_IPC_RECV
- Targeted unicast send via sqe->file_index for subscriber ID

Benchmark results (VM, 32 vCPUs, 32 GB RAM):

Point-to-point latency (1 sender, 1 receiver, ns/msg):

  Msg Size   pipe    unix sock  shm+eventfd  io_uring unicast
  --------   ----    ---------  -----------  ---------------
  64 B        212      436        222           632
  256 B       240      845        216           597
  1 KB        424      613        550           640
  4 KB        673    1,326        350           848
  16 KB     1,982    1,477      2,169         1,893
  32 KB     4,777    3,667      2,443         3,185

For point-to-point, io_uring IPC is within 1.5-2.5x of pipe/shm for small messages due to the io_uring submission overhead. At 16-32 KB the copy cost dominates and all mechanisms converge.

Broadcast to 16 receivers (ns/msg, sender-side):

Msg Size pipe unix sock shm+eventfd io_uring bcast

  --------   ------   ---------  -----------  --------------
  64 B       28,550    32,504      5,970         5,674
  256 B      27,588    34,777      5,429         6,600
  1 KB       28,072    34,845      6,542         6,095
  4 KB       37,277    46,154     11,520         6,367
  16 KB      57,998    58,348     34,969         7,592
  32 KB      89,404    83,496     93,082         8,202

The shared-ring broadcast architecture is the key differentiator: at 16 receivers with 32 KB messages, io_uring broadcast is 10.9x faster than pipe and 11.3x faster than shm+eventfd because data is written once to the shared ring rather than copied N times.

Scaling from 1 to 16 receivers (64 B messages):

  Mechanism        1 recv   16 recv   Degradation
  ---------        ------   -------   -----------
  pipe               212    28,550       135x
  unix sock          436    32,504        75x
  shm+eventfd        222     5,970        27x
  io_uring bcast     651     5,674       8.7x
  io_uring mcast     569     1,406       2.5x

Multicast sender throughput (ns/msg):

  Msg Size   1 recv  2 recv  4 recv  8 recv  16 recv
  --------   ------  ------  ------  ------  -------
  64 B         569     557     726     916    1,406
  4 KB         825     763     829   1,395    1,630
  32 KB      3,067   3,107   3,218   3,576    4,415

Multicast scales nearly flat because the CAS producer only contends with the shared consumer.head, not with individual receivers.

Daniel Hodges (2):
io_uring: add high-performance IPC channel infrastructure selftests/ipc: Add io_uring IPC selftest

 MAINTAINERS                                |    1 +
 include/linux/io_uring_types.h             |    7 +
 include/uapi/linux/io_uring.h              |   74 ++
 io_uring/Kconfig                           |   14 +
 io_uring/Makefile                          |    1 +
 io_uring/io_uring.c                        |    6 +
 io_uring/ipc.c                             | 1002 ++++++++++++++++
 io_uring/ipc.h                             |  161 +++
 io_uring/opdef.c                           |   19 +
 io_uring/register.c                        |   25 +
 tools/testing/selftests/ipc/Makefile       |    2 +-
 tools/testing/selftests/ipc/io_uring_ipc.c | 1265 ++++++++++++++++++++
 12 files changed, 2576 insertions(+), 1 deletion(-)
 create mode 100644 io_uring/ipc.c
 create mode 100644 io_uring/ipc.h
 create mode 100644 tools/testing/selftests/ipc/io_uring_ipc.c

--
2.52.0