net: implement the QUIC protocol in linux kernel
Subsystems (7)
cover net: implement the QUIC protocol in linux kernel
Introduction
============
The QUIC protocol, as defined in RFC9000, offers a UDP-based, secure transport with flow-controlled streams for efficient communication, low-latency connection setup, and network path migration, ensuring confidentiality, integrity, and availability across various deployments.
This implementation introduces QUIC support in Linux Kernel, offering several key advantages:
- Seamless Integration for Kernel Subsystems: Kernel subsystems such as SMB and NFS can operate over QUIC seamlessly after the handshake, leveraging the net/handshake APIs.
- Standardized Socket APIs for QUIC: This implementation standardizes the socket APIs for QUIC, covering essential operations like listen, accept, connect, sendmsg, recvmsg, close, get/setsockopt, and getsock/peername().
- Efficient ALPN Routing: It incorporates ALPN routing within the kernel, efficiently directing incoming requests to the appropriate applications across different processes based on ALPN.
- Performance Enhancements: By minimizing data duplication through zero-copy techniques such as sendfile(), and paving the way for crypto offloading in NICs, this implementation enhances performance and prepares for future optimizations.
This implementation offers fundamental support for the following RFCs:
- RFC9000 - QUIC: A UDP-Based Multiplexed and Secure Transport
- RFC9001 - Using TLS to Secure QUIC
- RFC9002 - QUIC Loss Detection and Congestion Control
- RFC9221 - An Unreliable Datagram Extension to QUIC
- RFC9287 - Greasing the QUIC Bit
- RFC9368 - Compatible Version Negotiation for QUIC
- RFC9369 - QUIC Version 2
The socket APIs for QUIC follow the RFC draft [1]:
- The Sockets API Extensions for In-kernel QUIC Implementations
Implementation
==============
The core idea is to implement QUIC within the kernel, using a userspace handshake approach.
Only the processing and creation of raw TLS Handshake Messages are handled in userspace, facilitated by a TLS library like GnuTLS. These messages are exchanged between kernel and userspace via sendmsg() and recvmsg(), with cryptographic details conveyed through control messages (cmsg).
The entire QUIC protocol, aside from the TLS Handshake Messages processing and creation, is managed within the kernel. Rather than using a Upper Layer Protocol (ULP) layer, this implementation establishes a socket of type IPPROTO_QUIC (similar to IPPROTO_MPTCP), operating over UDP tunnels.
Kernel consumers can initiate a handshake request from the kernel to userspace using the existing net/handshake netlink. The userspace component, such as tlshd service [2], then manages the processing of the QUIC handshake request.
- Handshake Architecture:
┌──────┐ ┌──────┐
│ APP1 │ │ APP2 │ ...
└──────┘ └──────┘
┌──────────────────────────────────────────┐
│ {quic_client/server_handshake()} │<─────────────┐
└──────────────────────────────────────────┘ ┌─────────────┐
{send/recvmsg()} {set/getsockopt()} │ tlshd │
[CMSG handshake_info] [SOCKOPT_CRYPTO_SECRET] └─────────────┘
[SOCKOPT_TRANSPORT_PARAM_EXT] │ ^
│ ^ │ ^ │ │
Userspace │ │ │ │ │ │
──────────────│─│──────────────────│─│──────────────────│───│────────
Kernel │ │ │ │ │ │
v │ v │ v │
┌──────────────────────────────────────────┐ ┌─────────────┐
│ socket (IPPROTO_QUIC) | protocol │<──┐ │ handshake │
├──────────────────────────────────────────┤ │ │netlink APIs │
│ stream | connid | cong | path | timer │ │ └─────────────┘
├──────────────────────────────────────────┤ │ │ │
│ packet | frame | crypto | pnmap │ │ ┌─────┐ ┌─────┐
├──────────────────────────────────────────┤ │ │ │ │ │
│ input | output │ │───│ SMB │ │ NFS │...
├──────────────────────────────────────────┤ │ │ │ │ │
│ UDP tunnels │ │ └─────┘ └─────┘
└──────────────────────────────────────────┘ └──────┴───────┘- User Data Architecture:
┌──────┐ ┌──────┐
│ APP1 │ │ APP2 │ ...
└──────┘ └──────┘
{send/recvmsg()} {set/getsockopt()}
[CMSG stream_info] [SOCKOPT_KEY_UPDATE]
[SOCKOPT_CONNECTION_MIGRATION]
[SOCKOPT_STREAM_OPEN/RESET/STOP_SENDING]
│ ^ │ ^
Userspace │ │ │ │
──────────────│─│──────────────────│─│────────────────────────
Kernel │ │ │ │
v │ v │
┌──────────────────────────────────────────┐
│ socket (IPPROTO_QUIC) | protocol │<──┐{kernel_send/recvmsg()}
├──────────────────────────────────────────┤ │{kernel_set/getsockopt()}
│ stream | connid | cong | path | timer │ │
├──────────────────────────────────────────┤ │
│ packet | frame | crypto | pnmap │ │ ┌─────┐ ┌─────┐
├──────────────────────────────────────────┤ │ │ │ │ │
│ input | output │ │───│ SMB │ │ NFS │...
├──────────────────────────────────────────┤ │ │ │ │ │
│ UDP tunnels │ │ └─────┘ └─────┘
└──────────────────────────────────────────┘ └──────┴───────┘Usage
=====
This implementation supports a mapping of QUIC into sockets APIs. Similar to TCP and SCTP, a typical Server and Client use the following system call sequence to communicate:
Client Server
────────────────────────────────────────────────────────────────────── sockfd = socket(IPPROTO_QUIC) listenfd = socket(IPPROTO_QUIC) bind(sockfd) bind(listenfd)
listen(listenfd)
connect(sockfd)
quic_client_handshake(sockfd)
sockfd = accecpt(listenfd) quic_server_handshake(sockfd, cert)
sendmsg(sockfd) recvmsg(sockfd)
close(sockfd) close(sockfd)
close(listenfd)
Please note that quic_client_handshake() and quic_server_handshake() functions are currently sourced from libquic [3]. These functions are responsible for receiving and processing the raw TLS handshake messages until the completion of the handshake process.
For utilization by kernel consumers, it is essential to have tlshd service [2] installed and running in userspace. This service receives and manages kernel handshake requests for kernel sockets. In the kernel, the APIs closely resemble those used in userspace:
Client Server
────────────────────────────────────────────────────────────────────────
__sock_create(IPPROTO_QUIC, &sock) __sock_create(IPPROTO_QUIC, &sock)
kernel_bind(sock) kernel_bind(sock)
kernel_listen(sock)
kernel_connect(sock)
tls_client_hello_x509(args:{sock})
kernel_accept(sock, &newsock)
tls_server_hello_x509(args:{newsock}) kernel_sendmsg(sock) kernel_recvmsg(newsock)
sock_release(sock) sock_release(newsock)
sock_release(sock)
Please be aware that tls_client_hello_x509() and tls_server_hello_x509() are APIs from net/handshake/. They are used to dispatch the handshake request to the userspace tlshd service and subsequently block until the handshake process is completed.
Tests
=====
1. Functional testing
The testing can be run by make check in libquic [3] and include:
- [Function Tests (PSK)]
- [Function Tests (Certificate)]
- [Performance Tests (IPv4)]
- [Performance Tests (IPv6, Disable 1RTT Encryption)]
- [Performance Tests (IPv6)]
- [Performance Tests (IPv4, 10% packet loss on both sides)]
- [Performance Tests (IPv6, 10% packet loss on both sides)]
- [InterOperability Tests (IPv4, msquic -> lkquic)]
- [InterOperability Tests (IPv6, lkquic -> msquic)]
- [Http/3 Tests (http3_test -> Public Websites)]
- [Http/3 Tests (http3_test client -> http3_test server)]
- [Session Resumption Tests]
- [Sample Tests]
- [ALPN and Preferred Address Tests]
2. Kernel Consumer Test via tlshd
The testing can be run using make check tests=tlshd in libquic after tlshd service [2] is installed and configured. The tests include:
- [Kernel Tests (kernel -> lkquic, Certificate, Sample)]
- [Kernel Tests (lkquic -> kernel, Certificate, Sample)]
- [Kernel Tests (kernel -> lkquic, PSK, Sample)]
- [Kernel Tests (lkquic -> kernel, PSK, Sample)]
- [Kernel Tests (kernel -> lkquic, Certificate, Session Resumption)]
- [Kernel Tests (lkquic -> kernel, Certificate, Session Resumption)]
3. HTTP/3 Interoperability testing via curl
Linux Kernel QUIC is being integrated for HTTP/3 in curl [4]. Below is an overview of connecting to various HTTP/3 servers using different QUIC implementations:
# curl --http3-only --ipv4 https://cloudflare-quic.com/ # curl --http3-only --ipv4 https://facebook.com/ # curl --http3-only --ipv4 https://litespeedtech.com/ # curl --http3-only --ipv4 https://nghttp2.org:4433/ # curl --http3-only --ipv4 https://outlook.office.com/ # curl --http3-only --ipv4 https://www.google.com/
4. Performance testing via iperf
The performance testing was conducted using iperf [5] over a 100G physical NIC, evaluating various packet sizes and MTUs:
- QUIC vs. kTLS:
UNIT size:1024 size:4096 size:16384 size:65536
Gbits/sec QUIC | kTLS QUIC | kTLS QUIC | kTLS QUIC | kTLS
────────────────────────────────────────────────────────────────────
mtu:1500 1.67 | 2.16 3.04 | 5.04 3.49 | 7.84 3.83 | 7.95
────────────────────────────────────────────────────────────────────
mtu:9000 2.17 | 2.41 5.47 | 6.19 6.45 | 8.66 7.48 | 8.90 - QUIC(disable_1rtt_encryption) vs. TCP:
UNIT size:1024 size:4096 size:16384 size:65536
Gbits/sec QUIC | TCP QUIC | TCP QUIC | TCP QUIC | TCP
────────────────────────────────────────────────────────────────────
mtu:1500 2.17 | 2.49 3.59 | 8.36 6.09 | 15.1 6.92 | 16.2
────────────────────────────────────────────────────────────────────
mtu:9000 2.47 | 2.54 7.66 | 7.97 14.7 | 20.3 19.1 | 31.3The performance gap between QUIC and kTLS may be attributed to:
- The absence of Generic Segmentation Offload (GSO) for QUIC.
- An additional data copy on the transmission (TX) path.
- Extra encryption required for header protection in QUIC.
- A longer header length for the stream data in QUIC.
NOTE: The QUIC module is currently labeled as "EXPERIMENTAL".
[1] https://www.ietf.org/archive/id/draft-lxin-quic-socket-apis-00.html
[2] https://github.com/oracle/ktls-utils
[3] https://github.com/lxin/quic
[4] https://github.com/moritzbuhl/curl (-b linux_curl)
[5] https://github.com/lxin/iperf
Xin Long (5):
net: define IPPROTO_QUIC and SOL_QUIC constants for QUIC protocol
net: include quic.h in include/uapi/linux for QUIC protocol
net: implement QUIC protocol code in net/quic directory
net: integrate QUIC build configuration into Kconfig and Makefile
Documentation: introduce quic.rst to provide description of QUIC
protocol
Documentation/networking/quic.rst | 178 +++ include/linux/quic.h | 19 + include/linux/socket.h | 1 + include/uapi/linux/in.h | 2 + include/uapi/linux/quic.h | 192 +++ net/Kconfig | 1 + net/Makefile | 1 + net/quic/Kconfig | 34 + net/quic/Makefile | 19 + net/quic/cong.c | 630 +++++++++ net/quic/cong.h | 118 ++ net/quic/connid.c | 188 +++ net/quic/connid.h | 120 ++ net/quic/crypto.c | 996 +++++++++++++ net/quic/crypto.h | 153 ++ net/quic/frame.c | 1903 +++++++++++++++++++++++++ net/quic/frame.h | 198 +++ net/quic/hashtable.h | 145 ++ net/quic/input.c | 602 ++++++++ net/quic/input.h | 155 ++ net/quic/number.h | 314 +++++ net/quic/output.c | 748 ++++++++++ net/quic/output.h | 199 +++ net/quic/packet.c | 1523 ++++++++++++++++++++ net/quic/packet.h | 125 ++ net/quic/path.c | 422 ++++++ net/quic/path.h | 143 ++ net/quic/pnspace.c | 184 +++ net/quic/pnspace.h | 209 +++ net/quic/protocol.c | 950 +++++++++++++ net/quic/protocol.h | 71 + net/quic/socket.c | 2183 +++++++++++++++++++++++++++++ net/quic/socket.h | 267 ++++ net/quic/stream.c | 252 ++++ net/quic/stream.h | 150 ++ net/quic/test/sample_test.c | 615 ++++++++ net/quic/test/unit_test.c | 1190 ++++++++++++++++ net/quic/timer.c | 302 ++++ net/quic/timer.h | 43 + 39 files changed, 15545 insertions(+) create mode 100644 Documentation/networking/quic.rst create mode 100644 include/linux/quic.h create mode 100644 include/uapi/linux/quic.h create mode 100644 net/quic/Kconfig create mode 100644 net/quic/Makefile create mode 100644 net/quic/cong.c create mode 100644 net/quic/cong.h create mode 100644 net/quic/connid.c create mode 100644 net/quic/connid.h create mode 100644 net/quic/crypto.c create mode 100644 net/quic/crypto.h create mode 100644 net/quic/frame.c create mode 100644 net/quic/frame.h create mode 100644 net/quic/hashtable.h create mode 100644 net/quic/input.c create mode 100644 net/quic/input.h create mode 100644 net/quic/number.h create mode 100644 net/quic/output.c create mode 100644 net/quic/output.h create mode 100644 net/quic/packet.c create mode 100644 net/quic/packet.h create mode 100644 net/quic/path.c create mode 100644 net/quic/path.h create mode 100644 net/quic/pnspace.c create mode 100644 net/quic/pnspace.h create mode 100644 net/quic/protocol.c create mode 100644 net/quic/protocol.h create mode 100644 net/quic/socket.c create mode 100644 net/quic/socket.h create mode 100644 net/quic/stream.c create mode 100644 net/quic/stream.h create mode 100644 net/quic/test/sample_test.c create mode 100644 net/quic/test/unit_test.c create mode 100644 net/quic/timer.c create mode 100644 net/quic/timer.h
Signature · 1 line
2.43.0
PATCH 1/5 net: define IPPROTO_QUIC and SOL_QUIC constants for QUIC protocol
This patch adds IPPROTO_QUIC and SOL_QUIC constants to the networking subsystem. These definitions are essential for applications to set socket options and protocol identifiers related to the QUIC protocol.
QUIC does not possess a protocol number allocated from IANA, and like IPPROTO_MPTCP, IPPROTO_QUIC is merely a value used when opening a QUIC socket with:
socket(AF_INET, SOCK_STREAM, IPPROTO_QUIC);
Note we did not opt for UDP ULP for QUIC implementation due to several considerations:
- QUIC's connection Migration requires at least 2 UDP sockets for one QUIC connection at the same time, not to mention the multipath feature in one of its draft RFCs.
- In-Kernel QUIC, as a Transport Protocol, wants to provide users with the TCP or SCTP like Socket APIs, like connect()/listen()/accept()... Note that a single UDP socket might even be used for multiple QUIC connections.
The use of IPPROTO_QUIC type sockets over UDP tunnel will effectively address these challenges and provides a more flexible and scalable solution.
--- include/linux/socket.h | 1 + include/uapi/linux/in.h | 2 ++ 2 files changed, 3 insertions(+)
include/linux/socket.h +1 -0
@@ -385,6 +385,7 @@ struct ucred {#define SOL_MCTP 285#define SOL_SMC 286#define SOL_VSOCK 287 +#define SOL_QUIC 288/* IPX options */ #define IPX_TYPE 1
include/uapi/linux/in.h +2 -0
@@ -83,6 +83,8 @@ enum {#define IPPROTO_RAW IPPROTO_RAW IPPROTO_SMC = 256, /* Shared Memory Communications */#define IPPROTO_SMC IPPROTO_SMC + IPPROTO_QUIC = 261, /* A UDP-Based Multiplexed and Secure Transport */ +#define IPPROTO_QUIC IPPROTO_QUIC IPPROTO_MPTCP = 262, /* Multipath TCP connection */#define IPPROTO_MPTCP IPPROTO_MPTCPIPPROTO_MAX
Signature · 1 line
2.43.0
PATCH 2/5 net: include quic.h in include/uapi/linux for QUIC protocol
In reply to Xin LongThis commit introduces quic.h to the include/uapi/linux directory, providing header file with necessary definitions for QUIC. Including quic.h enables both user space applications and kernel consumers to access QUIC-related Send/Receive Control Messages, Socket Options and Events/Notifications APIs.
--- include/uapi/linux/quic.h | 192 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 192 insertions(+) create mode 100644 include/uapi/linux/quic.h
include/uapi/linux/quic.h new file +192 -0
@@ -0,0 +1,192 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +/* QUIC kernel implementation +* (C) Copyright Red Hat Corp. 2023 +* +* This file is part of the QUIC kernel implementation +* +* Written or modified by: +* Xin Long <[email protected]> +*/ + +#ifndef __uapi_quic_h__ +#define __uapi_quic_h__ + +#include <linux/socket.h> +#ifdef __KERNEL__ +#include <linux/types.h> +#else +#include <stdint.h> +#endif + +/* Send or Receive Options APIs */ +enum quic_cmsg_type { +QUIC_STREAM_INFO , +QUIC_HANDSHAKE_INFO , +}; + +#define QUIC_STREAM_TYPE_SERVER_MASK 0x01 +#define QUIC_STREAM_TYPE_UNI_MASK 0x02 +#define QUIC_STREAM_TYPE_MASK 0x03 + +enum quic_msg_flags { +/* flags for stream_flags */ +MSG_STREAM_NEW =0x400 , +MSG_STREAM_FIN =0x200 , +MSG_STREAM_UNI =0x800 , +MSG_STREAM_DONTWAIT =0x10000 , + +/* extented flags for msg_flags */ +MSG_DATAGRAM =0x10 , +MSG_NOTIFICATION =0x8000 , +}; + +enum quic_crypto_level { +QUIC_CRYPTO_APP , +QUIC_CRYPTO_INITIAL , +QUIC_CRYPTO_HANDSHAKE , +QUIC_CRYPTO_EARLY , +QUIC_CRYPTO_MAX , +}; + +struct quic_handshake_info { +uint8_t crypto_level; +}; + +struct quic_stream_info { +int64_t stream_id; +uint32_t stream_flags; +}; + +/* Socket Options APIs */ +#define QUIC_SOCKOPT_EVENT 0 +#define QUIC_SOCKOPT_STREAM_OPEN 1 +#define QUIC_SOCKOPT_STREAM_RESET 2 +#define QUIC_SOCKOPT_STREAM_STOP_SENDING 3 +#define QUIC_SOCKOPT_CONNECTION_CLOSE 4 +#define QUIC_SOCKOPT_CONNECTION_MIGRATION 5 +#define QUIC_SOCKOPT_KEY_UPDATE 6 +#define QUIC_SOCKOPT_TRANSPORT_PARAM 7 +#define QUIC_SOCKOPT_CONFIG 8 +#define QUIC_SOCKOPT_TOKEN 9 +#define QUIC_SOCKOPT_ALPN 10 +#define QUIC_SOCKOPT_SESSION_TICKET 11 +#define QUIC_SOCKOPT_CRYPTO_SECRET 12 +#define QUIC_SOCKOPT_TRANSPORT_PARAM_EXT 13 +#define QUIC_SOCKOPT_RETIRE_CONNECTION_ID 14 +#define QUIC_SOCKOPT_ACTIVE_CONNECTION_ID 15 + +#define QUIC_VERSION_V1 0x1 +#define QUIC_VERSION_V2 0x6b3343cf + +struct quic_transport_param { +uint8_t remote; +uint8_t disable_active_migration; +uint8_t grease_quic_bit; +uint8_t stateless_reset; +uint8_t disable_1rtt_encryption; +uint8_t disable_compatible_version; +uint64_t max_udp_payload_size; +uint64_t ack_delay_exponent; +uint64_t max_ack_delay; +uint64_t active_connection_id_limit; +uint64_t max_idle_timeout; +uint64_t max_datagram_frame_size; +uint64_t max_data; +uint64_t max_stream_data_bidi_local; +uint64_t max_stream_data_bidi_remote; +uint64_t max_stream_data_uni; +uint64_t max_streams_bidi; +uint64_t max_streams_uni; +}; + +struct quic_config { +uint32_t version; +uint32_t plpmtud_probe_interval; +uint64_t initial_smoothed_rtt; +uint8_t congestion_control_algo; +uint8_t validate_peer_address; +uint32_t payload_cipher_type; +uint8_t receive_session_ticket; +uint8_t certificate_request; +}; + +struct quic_crypto_secret { +uint8_t send;/* send or recv */ +uint8_t level;/* crypto level */ +uint32_t type;/* TLS_CIPHER_* */ +uint8_t secret[48 ]; +}; + +enum { +QUIC_CONG_ALG_RENO , +QUIC_CONG_ALG_CUBIC , +QUIC_CONG_ALG_MAX , +}; + +struct quic_errinfo { +uint64_t stream_id; +uint32_t errcode; +}; + +struct quic_connection_id_info { +uint32_t source; +uint32_t dest; +}; + +struct quic_event_option { +uint8_t type; +uint8_t on; +}; + +/* Event APIs */ +enum quic_event_type { +QUIC_EVENT_NONE , +QUIC_EVENT_STREAM_UPDATE , +QUIC_EVENT_STREAM_MAX_STREAM , +QUIC_EVENT_CONNECTION_CLOSE , +QUIC_EVENT_CONNECTION_MIGRATION , +QUIC_EVENT_KEY_UPDATE , +QUIC_EVENT_NEW_TOKEN , +QUIC_EVENT_NEW_SESSION_TICKET , +QUIC_EVENT_END , +QUIC_EVENT_MAX =QUIC_EVENT_END -1 , +}; + +enum { +QUIC_STREAM_SEND_STATE_READY , +QUIC_STREAM_SEND_STATE_SEND , +QUIC_STREAM_SEND_STATE_SENT , +QUIC_STREAM_SEND_STATE_RECVD , +QUIC_STREAM_SEND_STATE_RESET_SENT , +QUIC_STREAM_SEND_STATE_RESET_RECVD , + +QUIC_STREAM_RECV_STATE_RECV , +QUIC_STREAM_RECV_STATE_SIZE_KNOWN , +QUIC_STREAM_RECV_STATE_RECVD , +QUIC_STREAM_RECV_STATE_READ , +QUIC_STREAM_RECV_STATE_RESET_RECVD , +QUIC_STREAM_RECV_STATE_RESET_READ , +}; + +struct quic_stream_update { +uint64_t id; +uint32_t state; +uint32_t errcode; +uint64_t finalsz; +}; + +struct quic_connection_close { +uint32_t errcode; +uint8_t frame; +uint8_t phrase[]; +}; + +union quic_event { +struct quic_stream_update update; +struct quic_connection_close close; +uint64_t max_stream; +uint8_t local_migration; +uint8_t key_update_phase; +}; + +#endif /* __uapi_quic_h__ */
Signature · 1 line
2.43.0
PATCH 3/5 net: implement QUIC protocol code in net/quic directory
In reply to Xin LongThis commit adds the initial implementation of the QUIC protocol code. The new net/quic directory contains the necessary source files to handle QUIC functionality within the networking subsystem:
- protocol.c: module init/exit and family_ops for inet and inet6.
- socket.c: definition of functions within the 'quic_prot' struct.
- connid.c: management of source and dest connection IDs.
- stream.c: bidi/unidirectional stream handling and management.
- cong.c: RTT measurement and congestion control mechanisms.
- timer.c: definition of essential timers including RTX/PROBE/IDLE/ACK.
- packet.c: creation and processing of various of short/long packets.
- frame.c: creation and processing of diverse types of frames.
- crypto.c: key derivation/update and header/payload de/encryption.
- pnspace.c: packet number namespaces and SACK range handling.
- input.c: socket lookup and stream/event frames enqueuing to userspace.
- output.c: frames enqueuing for send/resend as well as acknowledgment.
- path.c: src/dst path management including UDP tunnels and PLPMTUD.
- test/unit_test.c: tests for APIs defined in some of the above files.
- test/sample_test.c: a sample showcasing usage from the kernel space.
It introduces fundamental support for the following RFCs:
- RFC9000 - QUIC: A UDP-Based Multiplexed and Secure Transport
- RFC9001 - Using TLS to Secure QUIC
- RFC9002 - QUIC Loss Detection and Congestion Control
- RFC9221 - An Unreliable Datagram Extension to QUIC
- RFC9287 - Greasing the QUIC Bit
- RFC9368 - Compatible Version Negotiation for QUIC
- RFC9369 - QUIC Version 2
The QUIC module is currently labeled as "EXPERIMENTAL".
--- include/linux/quic.h | 19 + net/quic/Kconfig | 34 + net/quic/Makefile | 19 + net/quic/cong.c | 630 ++++++++++ net/quic/cong.h | 118 ++ net/quic/connid.c | 188 +++ net/quic/connid.h | 120 ++ net/quic/crypto.c | 996 ++++++++++++++++ net/quic/crypto.h | 153 +++ net/quic/frame.c | 1903 ++++++++++++++++++++++++++++++ net/quic/frame.h | 198 ++++ net/quic/hashtable.h | 145 +++ net/quic/input.c | 602 ++++++++++ net/quic/input.h | 155 +++ net/quic/number.h | 314 +++++ net/quic/output.c | 748 ++++++++++++ net/quic/output.h | 199 ++++ net/quic/packet.c | 1523 ++++++++++++++++++++++++ net/quic/packet.h | 125 ++ net/quic/path.c | 422 +++++++ net/quic/path.h | 143 +++ net/quic/pnspace.c | 184 +++ net/quic/pnspace.h | 209 ++++ net/quic/protocol.c | 950 +++++++++++++++ net/quic/protocol.h | 71 ++ net/quic/socket.c | 2183 +++++++++++++++++++++++++++++++++++ net/quic/socket.h | 267 +++++ net/quic/stream.c | 252 ++++ net/quic/stream.h | 150 +++ net/quic/test/sample_test.c | 615 ++++++++++ net/quic/test/unit_test.c | 1190 +++++++++++++++++++ net/quic/timer.c | 302 +++++ net/quic/timer.h | 43 + 33 files changed, 15170 insertions(+) create mode 100644 include/linux/quic.h create mode 100644 net/quic/Kconfig create mode 100644 net/quic/Makefile create mode 100644 net/quic/cong.c create mode 100644 net/quic/cong.h create mode 100644 net/quic/connid.c create mode 100644 net/quic/connid.h create mode 100644 net/quic/crypto.c create mode 100644 net/quic/crypto.h create mode 100644 net/quic/frame.c create mode 100644 net/quic/frame.h create mode 100644 net/quic/hashtable.h create mode 100644 net/quic/input.c create mode 100644 net/quic/input.h create mode 100644 net/quic/number.h create mode 100644 net/quic/output.c create mode 100644 net/quic/output.h create mode 100644 net/quic/packet.c create mode 100644 net/quic/packet.h create mode 100644 net/quic/path.c create mode 100644 net/quic/path.h create mode 100644 net/quic/pnspace.c create mode 100644 net/quic/pnspace.h create mode 100644 net/quic/protocol.c create mode 100644 net/quic/protocol.h create mode 100644 net/quic/socket.c create mode 100644 net/quic/socket.h create mode 100644 net/quic/stream.c create mode 100644 net/quic/stream.h create mode 100644 net/quic/test/sample_test.c create mode 100644 net/quic/test/unit_test.c create mode 100644 net/quic/timer.c create mode 100644 net/quic/timer.h
include/linux/quic.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +/* QUIC kernel implementation +* (C) Copyright Red Hat Corp. 2023 +* +* This file is part of the QUIC kernel implementation +* +* Written or modified by: +* Xin Long <[email protected]> +*/ + +#ifndef __linux_quic_h__ +#define __linux_quic_h__ + +#include <uapi/linux/quic.h> + +int quic_sock_setopt (struct sock *sk,int optname,void *optval,unsigned int optlen); +int quic_sock_getopt (struct sock *sk,int optname,void *optval,unsigned int *optlen); + +#endif
net/quic/Kconfig new file +34 -0
@@ -0,0 +1,34 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# QUIC configuration +# + +menuconfig IP_QUIC + tristate "QUIC: A UDP-Based Multiplexed and Secure Transport (Experimental)" + depends on INET + depends on IPV6 + select NET_UDP_TUNNEL + help + QUIC: A UDP-Based Multiplexed and Secure Transport + + From rfc9000 <https://www.rfc-editor.org/rfc/rfc9000.html>. + + QUIC provides applications with flow-controlled streams for structured + communication, low-latency connection establishment, and network path + migration. QUIC includes security measures that ensure + confidentiality, integrity, and availability in a range of deployment + circumstances. Accompanying documents describe the integration of + TLS for key negotiation, loss detection, and an exemplary congestion + control algorithm. + + To compile this protocol support as a module, choose M here: the + module will be called quic. Debug messages are handled by the + kernel's dynamic debugging framework. + + If in doubt, say N. + +if IP_QUIC +config IP_QUIC_TEST + depends on NET_HANDSHAKE || KUNIT + def_tristate m +endif
net/quic/Makefile new file +19 -0
@@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Makefile for QUIC support code. +# + +obj-$(CONFIG_IP_QUIC) += quic.o + +quic-y := protocol.o socket.o connid.o stream.o path.o packet.o \ + frame.o input.o output.o crypto.o pnspace.o timer.o cong.o + +ifdef CONFIG_KUNIT + obj-$(CONFIG_IP_QUIC_TEST) += quic_unit_test.o + quic_unit_test-y := test/unit_test.o +endif + +ifdef CONFIG_NET_HANDSHAKE + obj-$(CONFIG_IP_QUIC_TEST) += quic_sample_test.o + quic_sample_test-y := test/sample_test.o +endif
net/quic/cong.c new file +630 -0
630 lines not on this page · raw message
net/quic/cong.h new file +118 -0
118 lines not on this page · raw message
net/quic/connid.c new file +188 -0
188 lines not on this page · raw message
net/quic/connid.h new file +120 -0
120 lines not on this page · raw message
net/quic/crypto.c new file +996 -0
996 lines not on this page · raw message
net/quic/crypto.h new file +153 -0
153 lines not on this page · raw message
net/quic/frame.c new file +1903 -0
723 lines not on this page · raw message
net/quic/frame.h new file +198 -0
198 lines not on this page · raw message
net/quic/hashtable.h new file +145 -0
145 lines not on this page · raw message
net/quic/input.c new file +602 -0
602 lines not on this page · raw message
net/quic/input.h new file +155 -0
155 lines not on this page · raw message
net/quic/number.h new file +314 -0
314 lines not on this page · raw message
net/quic/output.c new file +748 -0
748 lines not on this page · raw message
net/quic/output.h new file +199 -0
199 lines not on this page · raw message
net/quic/packet.c new file +1523 -0
1,523 lines not on this page · raw message
net/quic/packet.h new file +125 -0
125 lines not on this page · raw message
net/quic/path.c new file +422 -0
422 lines not on this page · raw message
net/quic/path.h new file +143 -0
143 lines not on this page · raw message
net/quic/pnspace.c new file +184 -0
184 lines not on this page · raw message
net/quic/pnspace.h new file +209 -0
209 lines not on this page · raw message
net/quic/protocol.c new file +950 -0
950 lines not on this page · raw message
net/quic/protocol.h new file +71 -0
71 lines not on this page · raw message
net/quic/socket.c new file +2183 -0
2,183 lines not on this page · raw message
net/quic/socket.h new file +267 -0
267 lines not on this page · raw message
net/quic/stream.c new file +252 -0
252 lines not on this page · raw message
net/quic/stream.h new file +150 -0
150 lines not on this page · raw message
net/quic/test/sample_test.c new file +615 -0
615 lines not on this page · raw message
net/quic/test/unit_test.c new file +1190 -0
1,190 lines not on this page · raw message
net/quic/timer.c new file +302 -0
302 lines not on this page · raw message
net/quic/timer.h new file +43 -0
43 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.43.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH net-next 3/5] net: implement QUIC protocol code in net/quic directory
Quoted 4 lines · Xin Long
This commit adds the initial implementation of the QUIC protocol code. The new net/quic directory contains the necessary source files to handle QUIC functionality within the networking subsystem:
Quoted 16 lines
-
protocol.c: module init/exit andfamily_opsfor inet and inet6.
-socket.c: definition of functions within the 'quic_prot' struct.
-connid.c: management of source and dest connection IDs.
-stream.c: bidi/unidirectional stream handling and management.
-cong.c: RTT measurement and congestion control mechanisms.
-timer.c: definition of essential timers including RTX/PROBE/IDLE/ACK.
-packet.c: creation and processing of various of short/long packets.
-frame.c: creation and processing of diverse types of frames.
-crypto.c: key derivation/update and header/payload de/encryption.
-pnspace.c: packet number namespaces and SACK range handling.
-input.c: socket lookup and stream/event frames enqueuing to userspace.
-output.c: frames enqueuing for send/resend as well as acknowledgment.
-path.c: src/dst path management including UDP tunnels and PLPMTUD.
-test/unit_test.c: tests for APIs defined in some of the above files.
-test/sample_test.c: a sample showcasing usage from the kernel space.
Hi Xin,
I was intended to review your implementation, but I didn't know where to start. All your implementations are in one patch, making it quite difficult to review, so I gave up. 🙁
I think maybe you could consider adding interoperability tests with other variants of QUIC implementations. When we were working on xquic, this helped us discover many implementation issues, and it might be beneficial for you as well.
You can check it out from https://interop.seemann.io/.
Re: [PATCH net-next 3/5] net: implement QUIC protocol code in net/quic directory
Quoted 25 lines · D. Wythe
Quoted 4 lines · Xin Long
This commit adds the initial implementation of the QUIC protocol code. The new net/quic directory contains the necessary source files to handle QUIC functionality within the networking subsystem:
Quoted 16 lines
-
protocol.c: module init/exit andfamily_opsfor inet and inet6.
-socket.c: definition of functions within the 'quic_prot' struct.
-connid.c: management of source and dest connection IDs.
-stream.c: bidi/unidirectional stream handling and management.
-cong.c: RTT measurement and congestion control mechanisms.
-timer.c: definition of essential timers including RTX/PROBE/IDLE/ACK.
-packet.c: creation and processing of various of short/long packets.
-frame.c: creation and processing of diverse types of frames.
-crypto.c: key derivation/update and header/payload de/encryption.
-pnspace.c: packet number namespaces and SACK range handling.
-input.c: socket lookup and stream/event frames enqueuing to userspace.
-output.c: frames enqueuing for send/resend as well as acknowledgment.
-path.c: src/dst path management including UDP tunnels and PLPMTUD.
-test/unit_test.c: tests for APIs defined in some of the above files.
-test/sample_test.c: a sample showcasing usage from the kernel space.
Quoted 5 lines
Hi Xin,
I was intended to review your implementation, but I didn't know where to start. All your implementations
are in one patch, making it quite difficult to review, so I gave up. 🙁
Hi,
This is a protocol initial, and it's difficult to split it except that I already moved these APIs related files, building configuration and documentation. Honestly I couldn't think of a nice way to split more that could help much for the review.
But you can try https://github.com/lxin/quic/, where you will get more information including change history, configurations and all kinds of tests running etc, although its first commit for the protocol initial is kinda a big patch.
Quoted 8 lines
I think maybe you could consider adding interoperability tests with other variants of QUIC implementations.
When we were working on xquic, this helped us discover many implementation issues, and it might be
beneficial for you as well.You can check it out from https://interop.seemann.io/.
As you may already see from the cover-letter, the current interoperability tests we've done are via curl http3 over linux QUIC to connect different http3 servers that use different QUIC implementations, which are mostly covering handshakes and some version negotiations.
For the link shared, it does offer more coverage and it's nice to include it in the future.
Thanks for your comment.
PATCH 4/5 net: integrate QUIC build configuration into Kconfig and Makefile
In reply to Xin LongThis commit introduces build configurations for QUIC within the networking subsystem. The Kconfig and Makefile files in the net directory are updated to include options and rules necessary for building QUIC protocol support.
To enable building the QUIC module, include the following configurations in your .config file:
CONFIG_IP_QUIC=m
CONFIG_IP_QUIC_TEST=m
--- net/Kconfig | 1 + net/Makefile | 1 + 2 files changed, 2 insertions(+)
net/Kconfig +1 -0
@@ -238,6 +238,7 @@ endif source "net/dccp/Kconfig" source "net/sctp/Kconfig" +source "net/quic/Kconfig" source "net/rds/Kconfig" source "net/tipc/Kconfig" source "net/atm/Kconfig"
net/Makefile +1 -0
@@ -44,6 +44,7 @@ obj-y += 8021q/ endif obj-$(CONFIG_IP_DCCP) += dccp/ obj-$(CONFIG_IP_SCTP) += sctp/ +obj-$(CONFIG_IP_QUIC) += quic/ obj-$(CONFIG_RDS) += rds/ obj-$(CONFIG_WIRELESS) += wireless/ obj-$(CONFIG_MAC80211) += mac80211/
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerRe: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile
Quoted 3 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:
This commit introduces build configurations for QUIC within the networking subsystem. The
KconfigandMakefilefiles in the net directory are updated to include options and rules necessary for building QUIC protocol support.
Don't split out trivial config changes like this, what's the point. It just make build testing harder.
Speaking of which, it doesn't build on 32bit:
ERROR: modpost: "__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!
If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.
Signature · 1 line
pw-bot: cr
Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile
Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:
This commit introduces build configurations for QUIC within the networking subsystem. The
KconfigandMakefilefiles in the net directory are updated to include options and rules necessary for building QUIC protocol support.Don't split out trivial config changes like this, what's the point. It just make build testing harder.
I will move this to the Patch 3/5.
Quoted 6 lines
Speaking of which, it doesn't build on 32bit:
ERROR: modpost: "
__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!
The tests were done on x86_64, aarch64, s390x and ppc64le. Sorry for missing 32bit machines.
Quoted 3 lines
If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.
Copy that.
Thanks.
Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile
Am 12.09.24 um 16:57 schrieb Xin Long:
Quoted 17 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:
Quoted 3 lines
This commit introduces build configurations for QUIC within the networking subsystem. The
KconfigandMakefilefiles in the net directory are updated to include options and rules necessary for building QUIC protocol support.Don't split out trivial config changes like this, what's the point. It just make build testing harder.
I will move this to the Patch 3/5.
Quoted 6 lines
Speaking of which, it doesn't build on 32bit:
ERROR: modpost: "
__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!
Quoted 7 lines
The tests were done on
x86_64, aarch64, s390x and ppc64le. Sorry for missing 32bit machines.Quoted 3 lines
If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.
Copy that.
I'm seeing some activity in https://github.com/lxin/quic/commits/main/
What's the progress on upstreaming this?
Any chance to get this into 6.15?
Thanks!
metze
Re: [PATCH net-next 4/5] net: integrate QUIC build configuration into Kconfig and Makefile
Quoted 25 lines · Stefan Metzmacher
Am 12.09.24 um 16:57 schrieb Xin Long:
Quoted 23 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 9 Sep 2024 22:30:19 -0400 Xin Long wrote:
Quoted 3 lines
This commit introduces build configurations for QUIC within the networking subsystem. The
KconfigandMakefilefiles in the net directory are updated to include options and rules necessary for building QUIC protocol support.Don't split out trivial config changes like this, what's the point. It just make build testing harder.
I will move this to the Patch 3/5.
Quoted 6 lines
Speaking of which, it doesn't build on 32bit:
ERROR: modpost: "
__udivmoddi4" [net/quic/quic.ko] undefined!
ERROR: modpost: "__umoddi3" [net/quic/quic.ko] undefined!
ERROR: modpost: "__udivdi3" [net/quic/quic.ko] undefined!The tests were done on
x86_64, aarch64, s390x and ppc64le. Sorry for missing 32bit machines.Quoted 3 lines
If you repost before 6.12-rc1 please post as RFC, due to LPC / netconf we won't have enough time to review for 6.12 even if Linus cuts -rc8.
Quoted 8 lines
Quoted 1 line
Copy that.
I'm seeing some activity in https://github.com/lxin/quic/commits/main/
What's the progress on upstreaming this?
Any chance to get this into 6.15?
Hi, Stefan, thank you for reaching out.
If 6.14 is released by the end of March, net-next will be closed very soon. I think it's very difficult to get it posted and reviewed on time. I expected to post v3 some time in April.
Thanks.
PATCH 5/5 Documentation: introduce quic.rst to provide description of QUIC protocol
In reply to Xin LongThis commit adds quic.rst to the documentation directory, providing comprehensive information about the QUIC protocol. This new file serves as a reference for users and developers seeking details on QUIC implementation and usage.
--- Documentation/networking/quic.rst | 178 ++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 Documentation/networking/quic.rst
Documentation/networking/quic.rst new file +178 -0
@@ -0,0 +1,178 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +Linux Kernel QUIC +================= + +Introduction +============ + +The QUIC protocol, as defined in RFC9000, offers a UDP-based, secure +transport with flow-controlled streams for efficient communication, +low-latency connection setup, and network path migration, ensuring +confidentiality, integrity, and availability across various deployments. + +This implementation introduces QUIC support in Linux Kernel, offering +several key advantages: + +- Seamless Integration for Kernel Subsystems: Kernel subsystems such as + SMB and NFS can operate over QUIC seamlessly after the handshake, + leveraging the net/handshake APIs. + +- Standardized Socket APIs for QUIC: This implementation standardizes the + socket APIs for QUIC, covering essential operations like listen, accept, + connect, sendmsg, recvmsg, close, get/setsockopt, and getsock/peername(). + +- Efficient ALPN Routing: It incorporates ALPN routing within the kernel, + efficiently directing incoming requests to the appropriate applications + across different processes based on ALPN. + +- Performance Enhancements: By minimizing data duplication through + zero-copy techniques such as sendfile(), and paving the way for crypto + offloading in NICs, this implementation enhances performance and prepares + for future optimizations. + +This implementation offers fundamental support for the following RFCs: + +- RFC9000 - QUIC: A UDP-Based Multiplexed and Secure Transport +- RFC9001 - Using TLS to Secure QUIC +- RFC9002 - QUIC Loss Detection and Congestion Control +- RFC9221 - An Unreliable Datagram Extension to QUIC +- RFC9287 - Greasing the QUIC Bit +- RFC9368 - Compatible Version Negotiation for QUIC +- RFC9369 - QUIC Version 2 + +The socket APIs for QUIC follow the RFC draft [1]: + +- The Sockets API Extensions for In-kernel QUIC Implementations + +Implementation +============== + +The core idea is to implement QUIC within the kernel, using a userspace +handshake approach. + +Only the processing and creation of raw TLS Handshake Messages are handled +in userspace, facilitated by a TLS library like GnuTLS. These messages are +exchanged between kernel and userspace via sendmsg() and recvmsg(), with +cryptographic details conveyed through control messages (cmsg). + +The entire QUIC protocol, aside from the TLS Handshake Messages processing +and creation, is managed within the kernel. Rather than using a Upper Layer +Protocol (ULP) layer, this implementation establishes a socket of type +IPPROTO_QUIC (similar to IPPROTO_MPTCP), operating over UDP tunnels. + +Kernel consumers can initiate a handshake request from the kernel to +userspace using the existing net/handshake netlink. The userspace +component, such as tlshd service [2], then manages the processing +of the QUIC handshake request. + +- Handshake Architecture: + + ┌──────┐ ┌──────┐ + │ APP1 │ │ APP2 │ ... + └──────┘ └──────┘ + ┌──────────────────────────────────────────┐ + │ {quic_client/server_handshake()} │<─────────────┐ + └──────────────────────────────────────────┘ ┌─────────────┐ + {send/recvmsg()} {set/getsockopt()} │ tlshd │ + [CMSG handshake_info] [SOCKOPT_CRYPTO_SECRET] └─────────────┘ + [SOCKOPT_TRANSPORT_PARAM_EXT] │ ^ + │ ^ │ ^ │ │ + Userspace │ │ │ │ │ │ + ──────────────│─│──────────────────│─│──────────────────│───│──────── + Kernel │ │ │ │ │ │ + v │ v │ v │ + ┌──────────────────────────────────────────┐ ┌─────────────┐ + │ socket (IPPROTO_QUIC) | protocol │<──┐ │ handshake │ + ├──────────────────────────────────────────┤ │ │netlink APIs │ + │ stream | connid | cong | path | timer │ │ └─────────────┘ + ├──────────────────────────────────────────┤ │ │ │ + │ packet | frame | crypto | pnmap │ │ ┌─────┐ ┌─────┐ + ├──────────────────────────────────────────┤ │ │ │ │ │ + │ input | output │ │───│ SMB │ │ NFS │... + ├──────────────────────────────────────────┤ │ │ │ │ │ + │ UDP tunnels │ │ └─────┘ └─────┘ + └──────────────────────────────────────────┘ └──────┴───────┘ + +- User Data Architecture: + + ┌──────┐ ┌──────┐ + │ APP1 │ │ APP2 │ ... + └──────┘ └──────┘ + {send/recvmsg()} {set/getsockopt()} + [CMSG stream_info] [SOCKOPT_KEY_UPDATE] + [SOCKOPT_CONNECTION_MIGRATION] + [SOCKOPT_STREAM_OPEN/RESET/STOP_SENDING] + │ ^ │ ^ + Userspace │ │ │ │ + ──────────────│─│──────────────────│─│──────────────────────── + Kernel │ │ │ │ + v │ v │ + ┌──────────────────────────────────────────┐ + │ socket (IPPROTO_QUIC) | protocol │<──┐{kernel_send/recvmsg()} + ├──────────────────────────────────────────┤ │{kernel_set/getsockopt()} + │ stream | connid | cong | path | timer │ │ + ├──────────────────────────────────────────┤ │ + │ packet | frame | crypto | pnmap │ │ ┌─────┐ ┌─────┐ + ├──────────────────────────────────────────┤ │ │ │ │ │ + │ input | output │ │───│ SMB │ │ NFS │... + ├──────────────────────────────────────────┤ │ │ │ │ │ + │ UDP tunnels │ │ └─────┘ └─────┘ + └──────────────────────────────────────────┘ └──────┴───────┘ + +Usage +===== + +This implementation supports a mapping of QUIC into sockets APIs. Similar +to TCP and SCTP, a typical Server and Client use the following system call +sequence to communicate: + + Client Server + ────────────────────────────────────────────────────────────────────── + sockfd = socket(IPPROTO_QUIC) listenfd = socket(IPPROTO_QUIC) + bind(sockfd) bind(listenfd) + listen(listenfd) + connect(sockfd) + quic_client_handshake(sockfd) + sockfd = accecpt(listenfd) + quic_server_handshake(sockfd, cert) + + sendmsg(sockfd) recvmsg(sockfd) + close(sockfd) close(sockfd) + close(listenfd) + +Please note that quic_client_handshake() and quic_server_handshake() +functions are currently sourced from libquic [3]. These functions are +responsible for receiving and processing the raw TLS handshake messages +until the completion of the handshake process. + +For utilization by kernel consumers, it is essential to have tlshd +service [2] installed and running in userspace. This service receives +and manages kernel handshake requests for kernel sockets. In the kernel, +the APIs closely resemble those used in userspace: + + Client Server + ──────────────────────────────────────────────────────────────────────── + __sock_create(IPPROTO_QUIC, &sock) __sock_create(IPPROTO_QUIC, &sock) + kernel_bind(sock) kernel_bind(sock) + kernel_listen(sock) + kernel_connect(sock) + tls_client_hello_x509(args:{sock}) + kernel_accept(sock, &newsock) + tls_server_hello_x509(args:{newsock}) + + kernel_sendmsg(sock) kernel_recvmsg(newsock) + sock_release(sock) sock_release(newsock) + sock_release(sock) + +Please be aware that tls_client_hello_x509() and tls_server_hello_x509() +are APIs from net/handshake/. They are used to dispatch the handshake +request to the userspace tlshd service and subsequently block until the +handshake process is completed. + +The QUIC module is currently labeled as "EXPERIMENTAL". + +[1] https://datatracker.ietf.org/doc/draft-lxin-quic-socket-apis +[2] https://github.com/oracle/ktls-utils +[3] https://github.com/lxin/quic
Signature · 1 line
2.43.0
Quoted 24 lines · Xin Long
4. Performance testing via iperf
The performance testing was conducted using iperf [5] over a 100G physical NIC, evaluating various packet sizes and MTUs:
- QUIC vs. kTLS:
UNIT size:1024 size:4096 size:16384 size:65536 Gbits/sec QUIC | kTLS QUIC | kTLS QUIC | kTLS QUIC | kTLS ──────────────────────────────────────────────────────────────────── mtu:1500 1.67 | 2.16 3.04 | 5.04 3.49 | 7.84 3.83 | 7.95 ──────────────────────────────────────────────────────────────────── mtu:9000 2.17 | 2.41 5.47 | 6.19 6.45 | 8.66 7.48 | 8.90-
QUIC(disable_1rtt_encryption)vs. TCP:UNIT size:1024 size:4096 size:16384 size:65536 Gbits/sec QUIC | TCP QUIC | TCP QUIC | TCP QUIC | TCP ──────────────────────────────────────────────────────────────────── mtu:1500 2.17 | 2.49 3.59 | 8.36 6.09 | 15.1 6.92 | 16.2 ──────────────────────────────────────────────────────────────────── mtu:9000 2.47 | 2.54 7.66 | 7.97 14.7 | 20.3 19.1 | 31.3
Quoted 7 lines
The performance gap between QUIC and kTLS may be attributed to:
- The absence of Generic Segmentation Offload (GSO) for QUIC.
- An additional data copy on the transmission (TX) path.
- Extra encryption required for header protection in QUIC.
- A longer header length for the stream data in QUIC.
This is not appealing.
However, I can offer you one more possible advantage of in-kernel QUIC. You can think about adding iouring support for QUIC socket, because that could possibly chain the socket fastpath operations together which opens the door for more optimization.
Just my two cents.
Thanks!
Quoted 34 lines · Cong Wang
Quoted 24 lines · Xin Long
4. Performance testing via iperf
The performance testing was conducted using iperf [5] over a 100G physical NIC, evaluating various packet sizes and MTUs:
- QUIC vs. kTLS:
UNIT size:1024 size:4096 size:16384 size:65536 Gbits/sec QUIC | kTLS QUIC | kTLS QUIC | kTLS QUIC | kTLS ──────────────────────────────────────────────────────────────────── mtu:1500 1.67 | 2.16 3.04 | 5.04 3.49 | 7.84 3.83 | 7.95 ──────────────────────────────────────────────────────────────────── mtu:9000 2.17 | 2.41 5.47 | 6.19 6.45 | 8.66 7.48 | 8.90-
QUIC(disable_1rtt_encryption)vs. TCP:UNIT size:1024 size:4096 size:16384 size:65536 Gbits/sec QUIC | TCP QUIC | TCP QUIC | TCP QUIC | TCP ──────────────────────────────────────────────────────────────────── mtu:1500 2.17 | 2.49 3.59 | 8.36 6.09 | 15.1 6.92 | 16.2 ──────────────────────────────────────────────────────────────────── mtu:9000 2.47 | 2.54 7.66 | 7.97 14.7 | 20.3 19.1 | 31.3Quoted 7 lines
The performance gap between QUIC and kTLS may be attributed to:
- The absence of Generic Segmentation Offload (GSO) for QUIC.
- An additional data copy on the transmission (TX) path.
- Extra encryption required for header protection in QUIC.
- A longer header length for the stream data in QUIC.
Quoted 7 lines
This is not appealing.
However, I can offer you one more possible advantage of in-kernel QUIC. You can think about adding iouring support for QUIC socket, because that could possibly chain the socket fastpath operations together which opens the door for more optimization.
I haven't had the chance to try io_uring. From what I understand, it doesn’t require any changes to the protocol stack (Linux QUIC doesn’t support MSG_ZEROCOPY at this time).
Thanks for offering this suggestion, it sounds worth exploring when using the QUIC socket in the future.
Thanks.