Skip to contentSkip to conversation

net: implement the QUIC protocol in linux kernel

PATCH In review Xin Long 5 participants 14 messages Sep 10, 2024 – Mar 8, 2025

Subsystems (7)
lore
Share & tools
lore

Xin Long

author
#0 lore raw

cover net: implement the QUIC protocol in linux kernel

Introduction
============

The QUIC protocol, as defined in RFC9000, offers a UDP-based, secure transport with flow-controlled streams for efficient communication, low-latency connection setup, and network path migration, ensuring confidentiality, integrity, and availability across various deployments.

This implementation introduces QUIC support in Linux Kernel, offering several key advantages:

- Seamless Integration for Kernel Subsystems: Kernel subsystems such as SMB and NFS can operate over QUIC seamlessly after the handshake, leveraging the net/handshake APIs.

- Standardized Socket APIs for QUIC: This implementation standardizes the socket APIs for QUIC, covering essential operations like listen, accept, connect, sendmsg, recvmsg, close, get/setsockopt, and getsock/peername().

- Efficient ALPN Routing: It incorporates ALPN routing within the kernel, efficiently directing incoming requests to the appropriate applications across different processes based on ALPN.

- Performance Enhancements: By minimizing data duplication through zero-copy techniques such as sendfile(), and paving the way for crypto offloading in NICs, this implementation enhances performance and prepares for future optimizations.

This implementation offers fundamental support for the following RFCs:

- RFC9000 - QUIC: A UDP-Based Multiplexed and Secure Transport
- RFC9001 - Using TLS to Secure QUIC
- RFC9002 - QUIC Loss Detection and Congestion Control
- RFC9221 - An Unreliable Datagram Extension to QUIC
- RFC9287 - Greasing the QUIC Bit
- RFC9368 - Compatible Version Negotiation for QUIC
- RFC9369 - QUIC Version 2

The socket APIs for QUIC follow the RFC draft [1]:

- The Sockets API Extensions for In-kernel QUIC Implementations

Implementation
==============

The core idea is to implement QUIC within the kernel, using a userspace handshake approach.

Only the processing and creation of raw TLS Handshake Messages are handled in userspace, facilitated by a TLS library like GnuTLS. These messages are exchanged between kernel and userspace via sendmsg() and recvmsg(), with cryptographic details conveyed through control messages (cmsg).

The entire QUIC protocol, aside from the TLS Handshake Messages processing and creation, is managed within the kernel. Rather than using a Upper Layer Protocol (ULP) layer, this implementation establishes a socket of type IPPROTO_QUIC (similar to IPPROTO_MPTCP), operating over UDP tunnels.

Kernel consumers can initiate a handshake request from the kernel to userspace using the existing net/handshake netlink. The userspace component, such as tlshd service [2], then manages the processing of the QUIC handshake request.

- Handshake Architecture:

  ┌──────┐  ┌──────┐
  │ APP1 │  │ APP2 │ ...
  └──────┘  └──────┘
  ┌──────────────────────────────────────────┐
  │     {quic_client/server_handshake()}     │<─────────────┐
  └──────────────────────────────────────────┘       ┌─────────────┐
   {send/recvmsg()}      {set/getsockopt()}          │    tlshd    │
   [CMSG handshake_info] [SOCKOPT_CRYPTO_SECRET]     └─────────────┘
                         [SOCKOPT_TRANSPORT_PARAM_EXT]    │   ^
                │ ^                  │ ^                  │   │
  Userspace     │ │                  │ │                  │   │
  ──────────────│─│──────────────────│─│──────────────────│───│────────
  Kernel        │ │                  │ │                  │   │
                v │                  v │                  v   │
  ┌──────────────────────────────────────────┐       ┌─────────────┐
  │ socket (IPPROTO_QUIC) |     protocol     │<──┐   │ handshake   │
  ├──────────────────────────────────────────┤   │   │netlink APIs │
  │ stream | connid | cong  | path  | timer  │   │   └─────────────┘
  ├──────────────────────────────────────────┤   │      │       │
  │  packet  |  frame  |  crypto  |  pnmap   │   │   ┌─────┐ ┌─────┐
  ├──────────────────────────────────────────┤   │   │     │ │     │
  │        input       |       output        │   │───│ SMB │ │ NFS │...
  ├──────────────────────────────────────────┤   │   │     │ │     │
  │                UDP tunnels               │   │   └─────┘ └─────┘
  └──────────────────────────────────────────┘   └──────┴───────┘

- User Data Architecture:

  ┌──────┐  ┌──────┐
  │ APP1 │  │ APP2 │ ...
  └──────┘  └──────┘
   {send/recvmsg()}      {set/getsockopt()}
   [CMSG stream_info]    [SOCKOPT_KEY_UPDATE]
                         [SOCKOPT_CONNECTION_MIGRATION]
                         [SOCKOPT_STREAM_OPEN/RESET/STOP_SENDING]
                │ ^                  │ ^
  Userspace     │ │                  │ │
  ──────────────│─│──────────────────│─│────────────────────────
  Kernel        │ │                  │ │
                v │                  v │
  ┌──────────────────────────────────────────┐
  │ socket (IPPROTO_QUIC) |     protocol     │<──┐{kernel_send/recvmsg()}
  ├──────────────────────────────────────────┤   │{kernel_set/getsockopt()}
  │ stream | connid | cong  | path  | timer  │   │
  ├──────────────────────────────────────────┤   │
  │  packet  |  frame  |  crypto  |  pnmap   │   │   ┌─────┐ ┌─────┐
  ├──────────────────────────────────────────┤   │   │     │ │     │
  │        input       |       output        │   │───│ SMB │ │ NFS │...
  ├──────────────────────────────────────────┤   │   │     │ │     │
  │                UDP tunnels               │   │   └─────┘ └─────┘
  └──────────────────────────────────────────┘   └──────┴───────┘

Usage
=====

This implementation supports a mapping of QUIC into sockets APIs. Similar to TCP and SCTP, a typical Server and Client use the following system call sequence to communicate:

Client Server

  ──────────────────────────────────────────────────────────────────────
  sockfd = socket(IPPROTO_QUIC)      listenfd = socket(IPPROTO_QUIC)
  bind(sockfd)                       bind(listenfd)

listen(listenfd)
connect(sockfd)
quic_client_handshake(sockfd)
sockfd = accecpt(listenfd) quic_server_handshake(sockfd, cert)

sendmsg(sockfd) recvmsg(sockfd)
close(sockfd) close(sockfd)
close(listenfd)

Please note that quic_client_handshake() and quic_server_handshake() functions are currently sourced from libquic [3]. These functions are responsible for receiving and processing the raw TLS handshake messages until the completion of the handshake process.

For utilization by kernel consumers, it is essential to have tlshd service [2] installed and running in userspace. This service receives and manages kernel handshake requests for kernel sockets. In the kernel, the APIs closely resemble those used in userspace:

Client Server

  ────────────────────────────────────────────────────────────────────────
  __sock_create(IPPROTO_QUIC, &sock)  __sock_create(IPPROTO_QUIC, &sock)
  kernel_bind(sock)                   kernel_bind(sock)
                                      kernel_listen(sock)
  kernel_connect(sock)
  tls_client_hello_x509(args:{sock})
                                      kernel_accept(sock, &newsock)
                                      tls_server_hello_x509(args:{newsock})

kernel_sendmsg(sock) kernel_recvmsg(newsock)
sock_release(sock) sock_release(newsock)
sock_release(sock)

Please be aware that tls_client_hello_x509() and tls_server_hello_x509() are APIs from net/handshake/. They are used to dispatch the handshake request to the userspace tlshd service and subsequently block until the handshake process is completed.

Tests
=====

1. Functional testing

The testing can be run by make check in libquic [3] and include:

- [Function Tests (PSK)]
- [Function Tests (Certificate)]
- [Performance Tests (IPv4)]
- [Performance Tests (IPv6, Disable 1RTT Encryption)]
- [Performance Tests (IPv6)]
- [Performance Tests (IPv4, 10% packet loss on both sides)]
- [Performance Tests (IPv6, 10% packet loss on both sides)]
- [InterOperability Tests (IPv4, msquic -> lkquic)]
- [InterOperability Tests (IPv6, lkquic -> msquic)]
- [Http/3 Tests (http3_test -> Public Websites)]
- [Http/3 Tests (http3_test client -> http3_test server)]
- [Session Resumption Tests]
- [Sample Tests]
- [ALPN and Preferred Address Tests]

2. Kernel Consumer Test via tlshd

The testing can be run using make check tests=tlshd in libquic after tlshd service [2] is installed and configured. The tests include:

- [Kernel Tests (kernel -> lkquic, Certificate, Sample)]
- [Kernel Tests (lkquic -> kernel, Certificate, Sample)]
- [Kernel Tests (kernel -> lkquic, PSK, Sample)]
- [Kernel Tests (lkquic -> kernel, PSK, Sample)]
- [Kernel Tests (kernel -> lkquic, Certificate, Session Resumption)]
- [Kernel Tests (lkquic -> kernel, Certificate, Session Resumption)]

3. HTTP/3 Interoperability testing via curl

Linux Kernel QUIC is being integrated for HTTP/3 in curl [4]. Below is an overview of connecting to various HTTP/3 servers using different QUIC implementations:

  # curl --http3-only --ipv4 https://cloudflare-quic.com/
  # curl --http3-only --ipv4 https://facebook.com/
  # curl --http3-only --ipv4 https://litespeedtech.com/
  # curl --http3-only --ipv4 https://nghttp2.org:4433/
  # curl --http3-only --ipv4 https://outlook.office.com/
  # curl --http3-only --ipv4 https://www.google.com/

4. Performance testing via iperf

The performance testing was conducted using iperf [5] over a 100G physical NIC, evaluating various packet sizes and MTUs:

- QUIC vs. kTLS:

    UNIT        size:1024      size:4096      size:16384     size:65536
    Gbits/sec   QUIC | kTLS    QUIC | kTLS    QUIC | kTLS    QUIC | kTLS
    ────────────────────────────────────────────────────────────────────
    mtu:1500    1.67 | 2.16    3.04 | 5.04    3.49 | 7.84    3.83 | 7.95
    ────────────────────────────────────────────────────────────────────
    mtu:9000    2.17 | 2.41    5.47 | 6.19    6.45 | 8.66    7.48 | 8.90

- QUIC(disable_1rtt_encryption) vs. TCP:

    UNIT        size:1024      size:4096      size:16384     size:65536
    Gbits/sec   QUIC | TCP     QUIC | TCP     QUIC | TCP     QUIC | TCP
    ────────────────────────────────────────────────────────────────────
    mtu:1500    2.17 | 2.49    3.59 | 8.36    6.09 | 15.1    6.92 | 16.2
    ────────────────────────────────────────────────────────────────────
    mtu:9000    2.47 | 2.54    7.66 | 7.97    14.7 | 20.3    19.1 | 31.3

The performance gap between QUIC and kTLS may be attributed to:

- The absence of Generic Segmentation Offload (GSO) for QUIC.
- An additional data copy on the transmission (TX) path.
- Extra encryption required for header protection in QUIC.
- A longer header length for the stream data in QUIC.

NOTE: The QUIC module is currently labeled as "EXPERIMENTAL".

[1] https://www.ietf.org/archive/id/draft-lxin-quic-socket-apis-00.html
[2] https://github.com/oracle/ktls-utils
[3] https://github.com/lxin/quic
[4] https://github.com/moritzbuhl/curl (-b linux_curl)
[5] https://github.com/lxin/iperf

Xin Long (5):
net: define IPPROTO_QUIC and SOL_QUIC constants for QUIC protocol
net: include quic.h in include/uapi/linux for QUIC protocol
net: implement QUIC protocol code in net/quic directory
net: integrate QUIC build configuration into Kconfig and Makefile
Documentation: introduce quic.rst to provide description of QUIC
protocol

 Documentation/networking/quic.rst |  178 +++
 include/linux/quic.h              |   19 +
 include/linux/socket.h            |    1 +
 include/uapi/linux/in.h           |    2 +
 include/uapi/linux/quic.h         |  192 +++
 net/Kconfig                       |    1 +
 net/Makefile                      |    1 +
 net/quic/Kconfig                  |   34 +
 net/quic/Makefile                 |   19 +
 net/quic/cong.c                   |  630 +++++++++
 net/quic/cong.h                   |  118 ++
 net/quic/connid.c                 |  188 +++
 net/quic/connid.h                 |  120 ++
 net/quic/crypto.c                 |  996 +++++++++++++
 net/quic/crypto.h                 |  153 ++
 net/quic/frame.c                  | 1903 +++++++++++++++++++++++++
 net/quic/frame.h                  |  198 +++
 net/quic/hashtable.h              |  145 ++
 net/quic/input.c                  |  602 ++++++++
 net/quic/input.h                  |  155 ++
 net/quic/number.h                 |  314 +++++
 net/quic/output.c                 |  748 ++++++++++
 net/quic/output.h                 |  199 +++
 net/quic/packet.c                 | 1523 ++++++++++++++++++++
 net/quic/packet.h                 |  125 ++
 net/quic/path.c                   |  422 ++++++
 net/quic/path.h                   |  143 ++
 net/quic/pnspace.c                |  184 +++
 net/quic/pnspace.h                |  209 +++
 net/quic/protocol.c               |  950 +++++++++++++
 net/quic/protocol.h               |   71 +
 net/quic/socket.c                 | 2183 +++++++++++++++++++++++++++++
 net/quic/socket.h                 |  267 ++++
 net/quic/stream.c                 |  252 ++++
 net/quic/stream.h                 |  150 ++
 net/quic/test/sample_test.c       |  615 ++++++++
 net/quic/test/unit_test.c         | 1190 ++++++++++++++++
 net/quic/timer.c                  |  302 ++++
 net/quic/timer.h                  |   43 +
 39 files changed, 15545 insertions(+)
 create mode 100644 Documentation/networking/quic.rst
 create mode 100644 include/linux/quic.h
 create mode 100644 include/uapi/linux/quic.h
 create mode 100644 net/quic/Kconfig
 create mode 100644 net/quic/Makefile
 create mode 100644 net/quic/cong.c
 create mode 100644 net/quic/cong.h
 create mode 100644 net/quic/connid.c
 create mode 100644 net/quic/connid.h
 create mode 100644 net/quic/crypto.c
 create mode 100644 net/quic/crypto.h
 create mode 100644 net/quic/frame.c
 create mode 100644 net/quic/frame.h
 create mode 100644 net/quic/hashtable.h
 create mode 100644 net/quic/input.c
 create mode 100644 net/quic/input.h
 create mode 100644 net/quic/number.h
 create mode 100644 net/quic/output.c
 create mode 100644 net/quic/output.h
 create mode 100644 net/quic/packet.c
 create mode 100644 net/quic/packet.h
 create mode 100644 net/quic/path.c
 create mode 100644 net/quic/path.h
 create mode 100644 net/quic/pnspace.c
 create mode 100644 net/quic/pnspace.h
 create mode 100644 net/quic/protocol.c
 create mode 100644 net/quic/protocol.h
 create mode 100644 net/quic/socket.c
 create mode 100644 net/quic/socket.h
 create mode 100644 net/quic/stream.c
 create mode 100644 net/quic/stream.h
 create mode 100644 net/quic/test/sample_test.c
 create mode 100644 net/quic/test/unit_test.c
 create mode 100644 net/quic/timer.c
 create mode 100644 net/quic/timer.h
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/5 net: define IPPROTO_QUIC and SOL_QUIC constants for QUIC protocol

This patch adds IPPROTO_QUIC and SOL_QUIC constants to the networking subsystem. These definitions are essential for applications to set socket options and protocol identifiers related to the QUIC protocol.

QUIC does not possess a protocol number allocated from IANA, and like IPPROTO_MPTCP, IPPROTO_QUIC is merely a value used when opening a QUIC socket with:

  socket(AF_INET, SOCK_STREAM, IPPROTO_QUIC);

Note we did not opt for UDP ULP for QUIC implementation due to several considerations:

- QUIC's connection Migration requires at least 2 UDP sockets for one QUIC connection at the same time, not to mention the multipath feature in one of its draft RFCs.

- In-Kernel QUIC, as a Transport Protocol, wants to provide users with the TCP or SCTP like Socket APIs, like connect()/listen()/accept()... Note that a single UDP socket might even be used for multiple QUIC connections.

The use of IPPROTO_QUIC type sockets over UDP tunnel will effectively address these challenges and provides a more flexible and scalable solution.

---
 include/linux/socket.h  | 1 +
 include/uapi/linux/in.h | 2 ++
 2 files changed, 3 insertions(+)
include/linux/socket.h +1 -0
index df9cdb8bbfb8..e9db28c55a37 100644
@@ -385,6 +385,7 @@ struct ucred {
 #define SOL_MCTP	285
 #define SOL_SMC		286
 #define SOL_VSOCK	287
+#define SOL_QUIC	288
 
 /* IPX options */
 #define IPX_TYPE	1
include/uapi/linux/in.h +2 -0
index 5d32d53508d9..63f900290b96 100644
@@ -83,6 +83,8 @@ enum {
 #define IPPROTO_RAW		IPPROTO_RAW
   IPPROTO_SMC = 256,		/* Shared Memory Communications		*/
 #define IPPROTO_SMC		IPPROTO_SMC
+  IPPROTO_QUIC = 261,		/* A UDP-Based Multiplexed and Secure Transport	*/
+#define IPPROTO_QUIC		IPPROTO_QUIC
   IPPROTO_MPTCP = 262,		/* Multipath TCP connection		*/
 #define IPPROTO_MPTCP		IPPROTO_MPTCP
   IPPROTO_MAX
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH 2/5 net: include quic.h in include/uapi/linux for QUIC protocol

In reply to Xin Long

This commit introduces quic.h to the include/uapi/linux directory, providing header file with necessary definitions for QUIC. Including quic.h enables both user space applications and kernel consumers to access QUIC-related Send/Receive Control Messages, Socket Options and Events/Notifications APIs.

---
 include/uapi/linux/quic.h | 192 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 192 insertions(+)
 create mode 100644 include/uapi/linux/quic.h
include/uapi/linux/quic.h new file +192 -0
new file mode 100644
index 000000000000..aab7b85838d6
@@ -0,0 +1,192 @@
+/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
+/* QUIC kernel implementation
+ * (C) Copyright Red Hat Corp. 2023
+ *
+ * This file is part of the QUIC kernel implementation
+ *
+ * Written or modified by:
+ *    Xin Long <[email protected]>
+ */
+
+#ifndef __uapi_quic_h__
+#define __uapi_quic_h__
+
+#include <linux/socket.h>
+#ifdef __KERNEL__
+#include <linux/types.h>
+#else
+#include <stdint.h>
+#endif
+
+/* Send or Receive Options APIs */
+enum quic_cmsg_type {
+	QUIC_STREAM_INFO,
+	QUIC_HANDSHAKE_INFO,
+};
+
+#define QUIC_STREAM_TYPE_SERVER_MASK	0x01
+#define QUIC_STREAM_TYPE_UNI_MASK	0x02
+#define QUIC_STREAM_TYPE_MASK		0x03
+
+enum quic_msg_flags {
+	/* flags for stream_flags */
+	MSG_STREAM_NEW		= 0x400,
+	MSG_STREAM_FIN		= 0x200,
+	MSG_STREAM_UNI		= 0x800,
+	MSG_STREAM_DONTWAIT	= 0x10000,
+
+	/* extented flags for msg_flags */
+	MSG_DATAGRAM		= 0x10,
+	MSG_NOTIFICATION	= 0x8000,
+};
+
+enum quic_crypto_level {
+	QUIC_CRYPTO_APP,
+	QUIC_CRYPTO_INITIAL,
+	QUIC_CRYPTO_HANDSHAKE,
+	QUIC_CRYPTO_EARLY,
+	QUIC_CRYPTO_MAX,
+};
+
+struct quic_handshake_info {
+	uint8_t	crypto_level;
+};
+
+struct quic_stream_info {
+	int64_t  stream_id;
+	uint32_t stream_flags;
+};
+
+/* Socket Options APIs */
+#define QUIC_SOCKOPT_EVENT				0
+#define QUIC_SOCKOPT_STREAM_OPEN			1
+#define QUIC_SOCKOPT_STREAM_RESET			2
+#define QUIC_SOCKOPT_STREAM_STOP_SENDING		3
+#define QUIC_SOCKOPT_CONNECTION_CLOSE			4
+#define QUIC_SOCKOPT_CONNECTION_MIGRATION		5
+#define QUIC_SOCKOPT_KEY_UPDATE				6
+#define QUIC_SOCKOPT_TRANSPORT_PARAM			7
+#define QUIC_SOCKOPT_CONFIG				8
+#define QUIC_SOCKOPT_TOKEN				9
+#define QUIC_SOCKOPT_ALPN				10
+#define QUIC_SOCKOPT_SESSION_TICKET			11
+#define QUIC_SOCKOPT_CRYPTO_SECRET			12
+#define QUIC_SOCKOPT_TRANSPORT_PARAM_EXT		13
+#define QUIC_SOCKOPT_RETIRE_CONNECTION_ID		14
+#define QUIC_SOCKOPT_ACTIVE_CONNECTION_ID		15
+
+#define QUIC_VERSION_V1			0x1
+#define QUIC_VERSION_V2			0x6b3343cf
+
+struct quic_transport_param {
+	uint8_t		remote;
+	uint8_t		disable_active_migration;
+	uint8_t		grease_quic_bit;
+	uint8_t		stateless_reset;
+	uint8_t		disable_1rtt_encryption;
+	uint8_t		disable_compatible_version;
+	uint64_t	max_udp_payload_size;
+	uint64_t	ack_delay_exponent;
+	uint64_t	max_ack_delay;
+	uint64_t	active_connection_id_limit;
+	uint64_t	max_idle_timeout;
+	uint64_t	max_datagram_frame_size;
+	uint64_t	max_data;
+	uint64_t	max_stream_data_bidi_local;
+	uint64_t	max_stream_data_bidi_remote;
+	uint64_t	max_stream_data_uni;
+	uint64_t	max_streams_bidi;
+	uint64_t	max_streams_uni;
+};
+
+struct quic_config {
+	uint32_t	version;
+	uint32_t	plpmtud_probe_interval;
+	uint64_t	initial_smoothed_rtt;
+	uint8_t		congestion_control_algo;
+	uint8_t		validate_peer_address;
+	uint32_t	payload_cipher_type;
+	uint8_t		receive_session_ticket;
+	uint8_t		certificate_request;
+};
+
+struct quic_crypto_secret {
+	uint8_t send;  /* send or recv */
+	uint8_t level; /* crypto level */
+	uint32_t type; /* TLS_CIPHER_* */
+	uint8_t secret[48];
+};
+
+enum {
+	QUIC_CONG_ALG_RENO,
+	QUIC_CONG_ALG_CUBIC,
+	QUIC_CONG_ALG_MAX,
+};
+
+struct quic_errinfo {
+	uint64_t stream_id;
+	uint32_t errcode;
+};
+
+struct quic_connection_id_info {
+	uint32_t source;
+	uint32_t dest;
+};
+
+struct quic_event_option {
+	uint8_t type;
+	uint8_t on;
+};
+
+/* Event APIs */
+enum quic_event_type {
+	QUIC_EVENT_NONE,
+	QUIC_EVENT_STREAM_UPDATE,
+	QUIC_EVENT_STREAM_MAX_STREAM,
+	QUIC_EVENT_CONNECTION_CLOSE,
+	QUIC_EVENT_CONNECTION_MIGRATION,
+	QUIC_EVENT_KEY_UPDATE,
+	QUIC_EVENT_NEW_TOKEN,
+	QUIC_EVENT_NEW_SESSION_TICKET,
+	QUIC_EVENT_END,
+	QUIC_EVENT_MAX = QUIC_EVENT_END - 1,
+};
+
+enum {
+	QUIC_STREAM_SEND_STATE_READY,
+	QUIC_STREAM_SEND_STATE_SEND,
+	QUIC_STREAM_SEND_STATE_SENT,
+	QUIC_STREAM_SEND_STATE_RECVD,
+	QUIC_STREAM_SEND_STATE_RESET_SENT,
+	QUIC_STREAM_SEND_STATE_RESET_RECVD,
+
+	QUIC_STREAM_RECV_STATE_RECV,
+	QUIC_STREAM_RECV_STATE_SIZE_KNOWN,
+	QUIC_STREAM_RECV_STATE_RECVD,
+	QUIC_STREAM_RECV_STATE_READ,
+	QUIC_STREAM_RECV_STATE_RESET_RECVD,
+	QUIC_STREAM_RECV_STATE_RESET_READ,
+};
+
+struct quic_stream_update {
+	uint64_t id;
+	uint32_t state;
+	uint32_t errcode;
+	uint64_t finalsz;
+};
+
+struct quic_connection_close {
+	uint32_t errcode;
+	uint8_t frame;
+	uint8_t phrase[];
+};
+
+union quic_event {
+	struct quic_stream_update update;
+	struct quic_connection_close close;
+	uint64_t max_stream;
+	uint8_t local_migration;
+	uint8_t key_update_phase;
+};
+
+#endif /* __uapi_quic_h__ */
Signature · 1 line
2.43.0
author
#3 lore raw

PATCH 3/5 net: implement QUIC protocol code in net/quic directory

In reply to Xin Long

This commit adds the initial implementation of the QUIC protocol code. The new net/quic directory contains the necessary source files to handle QUIC functionality within the networking subsystem:

- protocol.c: module init/exit and family_ops for inet and inet6.
- socket.c: definition of functions within the 'quic_prot' struct.
- connid.c: management of source and dest connection IDs.
- stream.c: bidi/unidirectional stream handling and management.
- cong.c: RTT measurement and congestion control mechanisms.
- timer.c: definition of essential timers including RTX/PROBE/IDLE/ACK.
- packet.c: creation and processing of various of short/long packets.
- frame.c: creation and processing of diverse types of frames.
- crypto.c: key derivation/update and header/payload de/encryption.
- pnspace.c: packet number namespaces and SACK range handling.
- input.c: socket lookup and stream/event frames enqueuing to userspace.
- output.c: frames enqueuing for send/resend as well as acknowledgment.
- path.c: src/dst path management including UDP tunnels and PLPMTUD.
- test/unit_test.c: tests for APIs defined in some of the above files.
- test/sample_test.c: a sample showcasing usage from the kernel space.

It introduces fundamental support for the following RFCs:

- RFC9000 - QUIC: A UDP-Based Multiplexed and Secure Transport
- RFC9001 - Using TLS to Secure QUIC
- RFC9002 - QUIC Loss Detection and Congestion Control
- RFC9221 - An Unreliable Datagram Extension to QUIC
- RFC9287 - Greasing the QUIC Bit
- RFC9368 - Compatible Version Negotiation for QUIC
- RFC9369 - QUIC Version 2

The QUIC module is currently labeled as "EXPERIMENTAL".

---
 include/linux/quic.h        |   19 +
 net/quic/Kconfig            |   34 +
 net/quic/Makefile           |   19 +
 net/quic/cong.c             |  630 ++++++++++
 net/quic/cong.h             |  118 ++
 net/quic/connid.c           |  188 +++
 net/quic/connid.h           |  120 ++
 net/quic/crypto.c           |  996 ++++++++++++++++
 net/quic/crypto.h           |  153 +++
 net/quic/frame.c            | 1903 ++++++++++++++++++++++++++++++
 net/quic/frame.h            |  198 ++++
 net/quic/hashtable.h        |  145 +++
 net/quic/input.c            |  602 ++++++++++
 net/quic/input.h            |  155 +++
 net/quic/number.h           |  314 +++++
 net/quic/output.c           |  748 ++++++++++++
 net/quic/output.h           |  199 ++++
 net/quic/packet.c           | 1523 ++++++++++++++++++++++++
 net/quic/packet.h           |  125 ++
 net/quic/path.c             |  422 +++++++
 net/quic/path.h             |  143 +++
 net/quic/pnspace.c          |  184 +++
 net/quic/pnspace.h          |  209 ++++
 net/quic/protocol.c         |  950 +++++++++++++++
 net/quic/protocol.h         |   71 ++
 net/quic/socket.c           | 2183 +++++++++++++++++++++++++++++++++++
 net/quic/socket.h           |  267 +++++
 net/quic/stream.c           |  252 ++++
 net/quic/stream.h           |  150 +++
 net/quic/test/sample_test.c |  615 ++++++++++
 net/quic/test/unit_test.c   | 1190 +++++++++++++++++++
 net/quic/timer.c            |  302 +++++
 net/quic/timer.h            |   43 +
 33 files changed, 15170 insertions(+)
 create mode 100644 include/linux/quic.h
 create mode 100644 net/quic/Kconfig
 create mode 100644 net/quic/Makefile
 create mode 100644 net/quic/cong.c
 create mode 100644 net/quic/cong.h
 create mode 100644 net/quic/connid.c
 create mode 100644 net/quic/connid.h
 create mode 100644 net/quic/crypto.c
 create mode 100644 net/quic/crypto.h
 create mode 100644 net/quic/frame.c
 create mode 100644 net/quic/frame.h
 create mode 100644 net/quic/hashtable.h
 create mode 100644 net/quic/input.c
 create mode 100644 net/quic/input.h
 create mode 100644 net/quic/number.h
 create mode 100644 net/quic/output.c
 create mode 100644 net/quic/output.h
 create mode 100644 net/quic/packet.c
 create mode 100644 net/quic/packet.h
 create mode 100644 net/quic/path.c
 create mode 100644 net/quic/path.h
 create mode 100644 net/quic/pnspace.c
 create mode 100644 net/quic/pnspace.h
 create mode 100644 net/quic/protocol.c
 create mode 100644 net/quic/protocol.h
 create mode 100644 net/quic/socket.c
 create mode 100644 net/quic/socket.h
 create mode 100644 net/quic/stream.c
 create mode 100644 net/quic/stream.h
 create mode 100644 net/quic/test/sample_test.c
 create mode 100644 net/quic/test/unit_test.c
 create mode 100644 net/quic/timer.c
 create mode 100644 net/quic/timer.h
include/linux/quic.h new file +19 -0
new file mode 100644
index 000000000000..b3fc365e9c22
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
+/* QUIC kernel implementation
+ * (C) Copyright Red Hat Corp. 2023
+ *
+ * This file is part of the QUIC kernel implementation
+ *
+ * Written or modified by:
+ *    Xin Long <[email protected]>
+ */
+
+#ifndef __linux_quic_h__
+#define __linux_quic_h__
+
+#include <uapi/linux/quic.h>
+
+int quic_sock_setopt(struct sock *sk, int optname, void *optval, unsigned int optlen);
+int quic_sock_getopt(struct sock *sk, int optname, void *optval, unsigned int *optlen);
+
+#endif
net/quic/Kconfig new file +34 -0
new file mode 100644
index 000000000000..076c6e4380df
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# QUIC configuration
+#
+
+menuconfig IP_QUIC
+	tristate "QUIC: A UDP-Based Multiplexed and Secure Transport (Experimental)"
+	depends on INET
+	depends on IPV6
+	select NET_UDP_TUNNEL
+	help
+	  QUIC: A UDP-Based Multiplexed and Secure Transport
+
+	  From rfc9000 <https://www.rfc-editor.org/rfc/rfc9000.html>.
+
+	  QUIC provides applications with flow-controlled streams for structured
+	  communication, low-latency connection establishment, and network path
+	  migration.  QUIC includes security measures that ensure
+	  confidentiality, integrity, and availability in a range of deployment
+	  circumstances.  Accompanying documents describe the integration of
+	  TLS for key negotiation, loss detection, and an exemplary congestion
+	  control algorithm.
+
+	  To compile this protocol support as a module, choose M here: the
+	  module will be called quic. Debug messages are handled by the
+	  kernel's dynamic debugging framework.
+
+	  If in doubt, say N.
+
+if IP_QUIC
+config IP_QUIC_TEST
+	depends on NET_HANDSHAKE || KUNIT
+	def_tristate m
+endif
net/quic/Makefile new file +19 -0
new file mode 100644
index 000000000000..ca6d8c745fac
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Makefile for QUIC support code.
+#
+
+obj-$(CONFIG_IP_QUIC) += quic.o
+
+quic-y := protocol.o socket.o connid.o stream.o path.o packet.o \
+	  frame.o input.o output.o crypto.o pnspace.o timer.o cong.o
+
+ifdef CONFIG_KUNIT
+	obj-$(CONFIG_IP_QUIC_TEST) += quic_unit_test.o
+	quic_unit_test-y := test/unit_test.o
+endif
+
+ifdef CONFIG_NET_HANDSHAKE
+	obj-$(CONFIG_IP_QUIC_TEST) += quic_sample_test.o
+	quic_sample_test-y := test/sample_test.o
+endif
net/quic/cong.c new file +630 -0

630 lines not on this page · raw message

net/quic/cong.h new file +118 -0

118 lines not on this page · raw message

net/quic/connid.c new file +188 -0

188 lines not on this page · raw message

net/quic/connid.h new file +120 -0

120 lines not on this page · raw message

net/quic/crypto.c new file +996 -0

996 lines not on this page · raw message

net/quic/crypto.h new file +153 -0

153 lines not on this page · raw message

net/quic/frame.c new file +1903 -0

723 lines not on this page · raw message

net/quic/frame.h new file +198 -0

198 lines not on this page · raw message

net/quic/hashtable.h new file +145 -0

145 lines not on this page · raw message

net/quic/input.c new file +602 -0

602 lines not on this page · raw message

net/quic/input.h new file +155 -0

155 lines not on this page · raw message

net/quic/number.h new file +314 -0

314 lines not on this page · raw message

net/quic/output.c new file +748 -0

748 lines not on this page · raw message

net/quic/output.h new file +199 -0

199 lines not on this page · raw message

net/quic/packet.c new file +1523 -0

1,523 lines not on this page · raw message

net/quic/packet.h new file +125 -0

125 lines not on this page · raw message

net/quic/path.c new file +422 -0

422 lines not on this page · raw message

net/quic/path.h new file +143 -0

143 lines not on this page · raw message

net/quic/pnspace.c new file +184 -0

184 lines not on this page · raw message

net/quic/pnspace.h new file +209 -0

209 lines not on this page · raw message

net/quic/protocol.c new file +950 -0

950 lines not on this page · raw message

net/quic/protocol.h new file +71 -0

71 lines not on this page · raw message

net/quic/socket.c new file +2183 -0

2,183 lines not on this page · raw message

net/quic/socket.h new file +267 -0

267 lines not on this page · raw message

net/quic/stream.c new file +252 -0

252 lines not on this page · raw message

net/quic/stream.h new file +150 -0

150 lines not on this page · raw message

net/quic/test/sample_test.c new file +615 -0

615 lines not on this page · raw message

net/quic/test/unit_test.c new file +1190 -0

1,190 lines not on this page · raw message

net/quic/timer.c new file +302 -0

302 lines not on this page · raw message

net/quic/timer.h new file +43 -0

43 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.43.0

Part of the diff is not shown here. View the raw message.

#4 lore raw

Re: [PATCH net-next 3/5] net: implement QUIC protocol code in net/quic directory

Quoted 4 lines · Xin Long

This commit adds the initial implementation of the QUIC protocol code. The new net/quic directory contains the necessary source files to handle QUIC functionality within the networking subsystem:

Quoted 16 lines

- protocol.c: module init/exit and family_ops for inet and inet6.
- socket.c: definition of functions within the 'quic_prot' struct.
- connid.c: management of source and dest connection IDs.
- stream.c: bidi/unidirectional stream handling and management.
- cong.c: RTT measurement and congestion control mechanisms.
- timer.c: definition of essential timers including RTX/PROBE/IDLE/ACK.
- packet.c: creation and processing of various of short/long packets.
- frame.c: creation and processing of diverse types of frames.
- crypto.c: key derivation/update and header/payload de/encryption.
- pnspace.c: packet number namespaces and SACK range handling.
- input.c: socket lookup and stream/event frames enqueuing to userspace.
- output.c: frames enqueuing for send/resend as well as acknowledgment.
- path.c: src/dst path management including UDP tunnels and PLPMTUD.
- test/unit_test.c: tests for APIs defined in some of the above files.
- test/sample_test.c: a sample showcasing usage from the kernel space.

Hi Xin,

I was intended to review your implementation, but I didn't know where to start. All your implementations are in one patch, making it quite difficult to review, so I gave up. 🙁

I think maybe you could consider adding interoperability tests with other variants of QUIC implementations. When we were working on xquic, this helped us discover many implementation issues, and it might be beneficial for you as well.

You can check it out from https://interop.seemann.io/.

author
#5 lore raw

Re: [PATCH net-next 3/5] net: implement QUIC protocol code in net/quic directory

Quoted 25 lines · D. Wythe
Quoted 4 lines · Xin Long

This commit adds the initial implementation of the QUIC protocol code. The new net/quic directory contains the necessary source files to handle QUIC functionality within the networking subsystem:

Quoted 16 lines

- protocol.c: module init/exit and family_ops for inet and inet6.
- socket.c: definition of functions within the 'quic_prot' struct.
- connid.c: management of source and dest connection IDs.
- stream.c: bidi/unidirectional stream handling and management.
- cong.c: RTT measurement and congestion control mechanisms.
- timer.c: definition of essential timers including RTX/PROBE/IDLE/ACK.
- packet.c: creation and processing of various of short/long packets.
- frame.c: creation and processing of diverse types of frames.
- crypto.c: key derivation/update and header/payload de/encryption.
- pnspace.c: packet number namespaces and SACK range handling.
- input.c: socket lookup and stream/event frames enqueuing to userspace.
- output.c: frames enqueuing for send/resend as well as acknowledgment.
- path.c: src/dst path management including UDP tunnels and PLPMTUD.
- test/unit_test.c: tests for APIs defined in some of the above files.
- test/sample_test.c: a sample showcasing usage from the kernel space.

Quoted 5 lines

Hi Xin,

I was intended to review your implementation, but I didn't know where to start. All your implementations
are in one patch, making it quite difficult to review, so I gave up. 🙁

Hi,

This is a protocol initial, and it's difficult to split it except that I already moved these APIs related files, building configuration and documentation. Honestly I couldn't think of a nice way to split more that could help much for the review.

But you can try https://github.com/lxin/quic/, where you will get more information including change history, configurations and all kinds of tests running etc, although its first commit for the protocol initial is kinda a big patch.

Quoted 8 lines

I think maybe you could consider adding interoperability tests with other variants of QUIC implementations.
When we were working on xquic, this helped us discover many implementation issues, and it might be
beneficial for you as well.

You can check it out from https://interop.seemann.io/.

As you may already see from the cover-letter, the current interoperability tests we've done are via curl http3 over linux QUIC to connect different http3 servers that use different QUIC implementations, which are mostly covering handshakes and some version negotiations.

For the link shared, it does offer more coverage and it's nice to include it in the future.

Thanks for your comment.

author
#6 lore raw

PATCH 4/5 net: integrate QUIC build configuration into Kconfig and Makefile

In reply to Xin Long

This commit introduces build configurations for QUIC within the networking subsystem. The Kconfig and Makefile files in the net directory are updated to include options and rules necessary for building QUIC protocol support.

To enable building the QUIC module, include the following configurations in your .config file:

CONFIG_IP_QUIC=m
CONFIG_IP_QUIC_TEST=m

---
 net/Kconfig  | 1 +
 net/Makefile | 1 +
 2 files changed, 2 insertions(+)
net/Kconfig +1 -0
index d27d0deac0bf..3bbea4138c58 100644
@@ -238,6 +238,7 @@ endif
 
 source "net/dccp/Kconfig"
 source "net/sctp/Kconfig"
+source "net/quic/Kconfig"
 source "net/rds/Kconfig"
 source "net/tipc/Kconfig"
 source "net/atm/Kconfig"
net/Makefile +1 -0
index 65bb8c72a35e..0562e72482b9 100644
@@ -44,6 +44,7 @@ obj-y				+= 8021q/
 endif
 obj-$(CONFIG_IP_DCCP)		+= dccp/
 obj-$(CONFIG_IP_SCTP)		+= sctp/
+obj-$(CONFIG_IP_QUIC)		+= quic/
 obj-$(CONFIG_RDS)		+= rds/
 obj-$(CONFIG_WIRELESS)		+= wireless/
 obj-$(CONFIG_MAC80211)		+= mac80211/
Signature · 1 line
2.43.0
maintainer
#7 lore raw

Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile

Quoted 3 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:

This commit introduces build configurations for QUIC within the networking subsystem. The Kconfig and Makefile files in the net directory are updated to include options and rules necessary for building QUIC protocol support.

Don't split out trivial config changes like this, what's the point. It just make build testing harder.

Speaking of which, it doesn't build on 32bit:

ERROR: modpost: "__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!

If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.

Signature · 1 line
pw-bot: cr
author
#8 lore raw

Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile

Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:

This commit introduces build configurations for QUIC within the networking subsystem. The Kconfig and Makefile files in the net directory are updated to include options and rules necessary for building QUIC protocol support.

Don't split out trivial config changes like this, what's the point. It just make build testing harder.

I will move this to the Patch 3/5.

Quoted 6 lines

Speaking of which, it doesn't build on 32bit:

ERROR: modpost: "__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!

The tests were done on x86_64, aarch64, s390x and ppc64le. Sorry for missing 32bit machines.

Quoted 3 lines

If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.

Copy that.

Thanks.

#9 lore raw

Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile

Am 12.09.24 um 16:57 schrieb Xin Long:

Quoted 17 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:
Quoted 3 lines

This commit introduces build configurations for QUIC within the networking subsystem. The Kconfig and Makefile files in the net directory are updated to include options and rules necessary for building QUIC protocol support.

Don't split out trivial config changes like this, what's the point. It just make build testing harder.

I will move this to the Patch 3/5.

Quoted 6 lines

Speaking of which, it doesn't build on 32bit:

ERROR: modpost: "__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!

Quoted 7 lines

The tests were done on x86_64, aarch64, s390x and ppc64le. Sorry for missing 32bit machines.

Quoted 3 lines

If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.

Copy that.

I'm seeing some activity in https://github.com/lxin/quic/commits/main/

What's the progress on upstreaming this?

Any chance to get this into 6.15?

Thanks!
metze

author
#10 lore raw

Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile

Quoted 25 lines · Stefan Metzmacher

Am 12.09.24 um 16:57 schrieb Xin Long:

Quoted 23 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:
Quoted 3 lines

This commit introduces build configurations for QUIC within the networking subsystem. The Kconfig and Makefile files in the net directory are updated to include options and rules necessary for building QUIC protocol support.

Don't split out trivial config changes like this, what's the point. It just make build testing harder.

I will move this to the Patch 3/5.

Quoted 6 lines

Speaking of which, it doesn't build on 32bit:

ERROR: modpost: "__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!

The tests were done on x86_64, aarch64, s390x and ppc64le. Sorry for missing 32bit machines.

Quoted 3 lines

If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.

Quoted 8 lines
Quoted 1 line

Copy that.

I'm seeing some activity in https://github.com/lxin/quic/commits/main/

What's the progress on upstreaming this?

Any chance to get this into 6.15?

Hi, Stefan, thank you for reaching out.

If 6.14 is released by the end of March, net-next will be closed very soon. I think it's very difficult to get it posted and reviewed on time. I expected to post v3 some time in April.

Thanks.

author
#11 lore raw

PATCH 5/5 Documentation: introduce quic.rst to provide description of QUIC protocol

In reply to Xin Long

This commit adds quic.rst to the documentation directory, providing comprehensive information about the QUIC protocol. This new file serves as a reference for users and developers seeking details on QUIC implementation and usage.

---
 Documentation/networking/quic.rst | 178 ++++++++++++++++++++++++++++++
 1 file changed, 178 insertions(+)
 create mode 100644 Documentation/networking/quic.rst
Documentation/networking/quic.rst new file +178 -0
new file mode 100644
index 000000000000..461eec9c504d
@@ -0,0 +1,178 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================
+Linux Kernel QUIC
+=================
+
+Introduction
+============
+
+The QUIC protocol, as defined in RFC9000, offers a UDP-based, secure
+transport with flow-controlled streams for efficient communication,
+low-latency connection setup, and network path migration, ensuring
+confidentiality, integrity, and availability across various deployments.
+
+This implementation introduces QUIC support in Linux Kernel, offering
+several key advantages:
+
+- Seamless Integration for Kernel Subsystems: Kernel subsystems such as
+  SMB and NFS can operate over QUIC seamlessly after the handshake,
+  leveraging the net/handshake APIs.
+
+- Standardized Socket APIs for QUIC: This implementation standardizes the
+  socket APIs for QUIC, covering essential operations like listen, accept,
+  connect, sendmsg, recvmsg, close, get/setsockopt, and getsock/peername().
+
+- Efficient ALPN Routing: It incorporates ALPN routing within the kernel,
+  efficiently directing incoming requests to the appropriate applications
+  across different processes based on ALPN.
+
+- Performance Enhancements: By minimizing data duplication through
+  zero-copy techniques such as sendfile(), and paving the way for crypto
+  offloading in NICs, this implementation enhances performance and prepares
+  for future optimizations.
+
+This implementation offers fundamental support for the following RFCs:
+
+- RFC9000 - QUIC: A UDP-Based Multiplexed and Secure Transport
+- RFC9001 - Using TLS to Secure QUIC
+- RFC9002 - QUIC Loss Detection and Congestion Control
+- RFC9221 - An Unreliable Datagram Extension to QUIC
+- RFC9287 - Greasing the QUIC Bit
+- RFC9368 - Compatible Version Negotiation for QUIC
+- RFC9369 - QUIC Version 2
+
+The socket APIs for QUIC follow the RFC draft [1]:
+
+- The Sockets API Extensions for In-kernel QUIC Implementations
+
+Implementation
+==============
+
+The core idea is to implement QUIC within the kernel, using a userspace
+handshake approach.
+
+Only the processing and creation of raw TLS Handshake Messages are handled
+in userspace, facilitated by a TLS library like GnuTLS. These messages are
+exchanged between kernel and userspace via sendmsg() and recvmsg(), with
+cryptographic details conveyed through control messages (cmsg).
+
+The entire QUIC protocol, aside from the TLS Handshake Messages processing
+and creation, is managed within the kernel. Rather than using a Upper Layer
+Protocol (ULP) layer, this implementation establishes a socket of type
+IPPROTO_QUIC (similar to IPPROTO_MPTCP), operating over UDP tunnels.
+
+Kernel consumers can initiate a handshake request from the kernel to
+userspace using the existing net/handshake netlink. The userspace
+component, such as tlshd service [2], then manages the processing
+of the QUIC handshake request.
+
+- Handshake Architecture:
+
+  ┌──────┐  ┌──────┐
+  │ APP1 │  │ APP2 │ ...
+  └──────┘  └──────┘
+  ┌──────────────────────────────────────────┐
+  │     {quic_client/server_handshake()}     │<─────────────┐
+  └──────────────────────────────────────────┘       ┌─────────────┐
+   {send/recvmsg()}      {set/getsockopt()}          │    tlshd    │
+   [CMSG handshake_info] [SOCKOPT_CRYPTO_SECRET]     └─────────────┘
+                         [SOCKOPT_TRANSPORT_PARAM_EXT]    │   ^
+                │ ^                  │ ^                  │   │
+  Userspace     │ │                  │ │                  │   │
+  ──────────────│─│──────────────────│─│──────────────────│───│────────
+  Kernel        │ │                  │ │                  │   │
+                v │                  v │                  v   │
+  ┌──────────────────────────────────────────┐       ┌─────────────┐
+  │ socket (IPPROTO_QUIC) |     protocol     │<──┐   │ handshake   │
+  ├──────────────────────────────────────────┤   │   │netlink APIs │
+  │ stream | connid | cong  | path  | timer  │   │   └─────────────┘
+  ├──────────────────────────────────────────┤   │      │       │
+  │  packet  |  frame  |  crypto  |  pnmap   │   │   ┌─────┐ ┌─────┐
+  ├──────────────────────────────────────────┤   │   │     │ │     │
+  │        input       |       output        │   │───│ SMB │ │ NFS │...
+  ├──────────────────────────────────────────┤   │   │     │ │     │
+  │                UDP tunnels               │   │   └─────┘ └─────┘
+  └──────────────────────────────────────────┘   └──────┴───────┘
+
+- User Data Architecture:
+
+  ┌──────┐  ┌──────┐
+  │ APP1 │  │ APP2 │ ...
+  └──────┘  └──────┘
+   {send/recvmsg()}      {set/getsockopt()}
+   [CMSG stream_info]    [SOCKOPT_KEY_UPDATE]
+                         [SOCKOPT_CONNECTION_MIGRATION]
+                         [SOCKOPT_STREAM_OPEN/RESET/STOP_SENDING]
+                │ ^                  │ ^
+  Userspace     │ │                  │ │
+  ──────────────│─│──────────────────│─│────────────────────────
+  Kernel        │ │                  │ │
+                v │                  v │
+  ┌──────────────────────────────────────────┐
+  │ socket (IPPROTO_QUIC) |     protocol     │<──┐{kernel_send/recvmsg()}
+  ├──────────────────────────────────────────┤   │{kernel_set/getsockopt()}
+  │ stream | connid | cong  | path  | timer  │   │
+  ├──────────────────────────────────────────┤   │
+  │  packet  |  frame  |  crypto  |  pnmap   │   │   ┌─────┐ ┌─────┐
+  ├──────────────────────────────────────────┤   │   │     │ │     │
+  │        input       |       output        │   │───│ SMB │ │ NFS │...
+  ├──────────────────────────────────────────┤   │   │     │ │     │
+  │                UDP tunnels               │   │   └─────┘ └─────┘
+  └──────────────────────────────────────────┘   └──────┴───────┘
+
+Usage
+=====
+
+This implementation supports a mapping of QUIC into sockets APIs. Similar
+to TCP and SCTP, a typical Server and Client use the following system call
+sequence to communicate:
+
+    Client                             Server
+  ──────────────────────────────────────────────────────────────────────
+  sockfd = socket(IPPROTO_QUIC)      listenfd = socket(IPPROTO_QUIC)
+  bind(sockfd)                       bind(listenfd)
+                                     listen(listenfd)
+  connect(sockfd)
+  quic_client_handshake(sockfd)
+                                     sockfd = accecpt(listenfd)
+                                     quic_server_handshake(sockfd, cert)
+
+  sendmsg(sockfd)                    recvmsg(sockfd)
+  close(sockfd)                      close(sockfd)
+                                     close(listenfd)
+
+Please note that quic_client_handshake() and quic_server_handshake()
+functions are currently sourced from libquic [3]. These functions are
+responsible for receiving and processing the raw TLS handshake messages
+until the completion of the handshake process.
+
+For utilization by kernel consumers, it is essential to have tlshd
+service [2] installed and running in userspace. This service receives
+and manages kernel handshake requests for kernel sockets. In the kernel,
+the APIs closely resemble those used in userspace:
+
+    Client                             Server
+  ────────────────────────────────────────────────────────────────────────
+  __sock_create(IPPROTO_QUIC, &sock)  __sock_create(IPPROTO_QUIC, &sock)
+  kernel_bind(sock)                   kernel_bind(sock)
+                                      kernel_listen(sock)
+  kernel_connect(sock)
+  tls_client_hello_x509(args:{sock})
+                                      kernel_accept(sock, &newsock)
+                                      tls_server_hello_x509(args:{newsock})
+
+  kernel_sendmsg(sock)                kernel_recvmsg(newsock)
+  sock_release(sock)                  sock_release(newsock)
+                                      sock_release(sock)
+
+Please be aware that tls_client_hello_x509() and tls_server_hello_x509()
+are APIs from net/handshake/. They are used to dispatch the handshake
+request to the userspace tlshd service and subsequently block until the
+handshake process is completed.
+
+The QUIC module is currently labeled as "EXPERIMENTAL".
+
+[1] https://datatracker.ietf.org/doc/draft-lxin-quic-socket-apis
+[2] https://github.com/oracle/ktls-utils
+[3] https://github.com/lxin/quic
Signature · 1 line
2.43.0
#12 lore raw
In reply to Xin Long
Quoted 24 lines · Xin Long

4. Performance testing via iperf

The performance testing was conducted using iperf [5] over a 100G physical NIC, evaluating various packet sizes and MTUs:

- QUIC vs. kTLS:

    UNIT        size:1024      size:4096      size:16384     size:65536
    Gbits/sec   QUIC | kTLS    QUIC | kTLS    QUIC | kTLS    QUIC | kTLS
    ────────────────────────────────────────────────────────────────────
    mtu:1500    1.67 | 2.16    3.04 | 5.04    3.49 | 7.84    3.83 | 7.95
    ────────────────────────────────────────────────────────────────────
    mtu:9000    2.17 | 2.41    5.47 | 6.19    6.45 | 8.66    7.48 | 8.90

- QUIC(disable_1rtt_encryption) vs. TCP:

    UNIT        size:1024      size:4096      size:16384     size:65536
    Gbits/sec   QUIC | TCP     QUIC | TCP     QUIC | TCP     QUIC | TCP
    ────────────────────────────────────────────────────────────────────
    mtu:1500    2.17 | 2.49    3.59 | 8.36    6.09 | 15.1    6.92 | 16.2
    ────────────────────────────────────────────────────────────────────
    mtu:9000    2.47 | 2.54    7.66 | 7.97    14.7 | 20.3    19.1 | 31.3
Quoted 7 lines

The performance gap between QUIC and kTLS may be attributed to:

- The absence of Generic Segmentation Offload (GSO) for QUIC.
- An additional data copy on the transmission (TX) path.
- Extra encryption required for header protection in QUIC.
- A longer header length for the stream data in QUIC.

This is not appealing.

However, I can offer you one more possible advantage of in-kernel QUIC. You can think about adding iouring support for QUIC socket, because that could possibly chain the socket fastpath operations together which opens the door for more optimization.

Just my two cents.

Thanks!

author
#13 lore raw
Quoted 34 lines · Cong Wang
Quoted 24 lines · Xin Long

4. Performance testing via iperf

The performance testing was conducted using iperf [5] over a 100G physical NIC, evaluating various packet sizes and MTUs:

- QUIC vs. kTLS:

    UNIT        size:1024      size:4096      size:16384     size:65536
    Gbits/sec   QUIC | kTLS    QUIC | kTLS    QUIC | kTLS    QUIC | kTLS
    ────────────────────────────────────────────────────────────────────
    mtu:1500    1.67 | 2.16    3.04 | 5.04    3.49 | 7.84    3.83 | 7.95
    ────────────────────────────────────────────────────────────────────
    mtu:9000    2.17 | 2.41    5.47 | 6.19    6.45 | 8.66    7.48 | 8.90

- QUIC(disable_1rtt_encryption) vs. TCP:

    UNIT        size:1024      size:4096      size:16384     size:65536
    Gbits/sec   QUIC | TCP     QUIC | TCP     QUIC | TCP     QUIC | TCP
    ────────────────────────────────────────────────────────────────────
    mtu:1500    2.17 | 2.49    3.59 | 8.36    6.09 | 15.1    6.92 | 16.2
    ────────────────────────────────────────────────────────────────────
    mtu:9000    2.47 | 2.54    7.66 | 7.97    14.7 | 20.3    19.1 | 31.3
Quoted 7 lines

The performance gap between QUIC and kTLS may be attributed to:

- The absence of Generic Segmentation Offload (GSO) for QUIC.
- An additional data copy on the transmission (TX) path.
- Extra encryption required for header protection in QUIC.
- A longer header length for the stream data in QUIC.

Quoted 7 lines

This is not appealing.

However, I can offer you one more possible advantage of in-kernel QUIC. You can think about adding iouring support for QUIC socket, because that could possibly chain the socket fastpath operations together which opens the door for more optimization.

I haven't had the chance to try io_uring. From what I understand, it doesn’t require any changes to the protocol stack (Linux QUIC doesn’t support MSG_ZEROCOPY at this time).

Thanks for offering this suggestion, it sounds worth exploring when using the QUIC socket in the future.

Thanks.