Implement a TLS handshake upcall
Subsystems (14)
cover Implement a TLS handshake upcall
There are a few upper-layer (storage) protocols that want to have a full TLS implementation available in the Linux kernel.
The Linux kernel currently has an implementation of the TLS record protocol, known as kTLS. However it does not have a complete TLS implementation because it has no implementation of the TLS handshake protocol. In-kernel storage protocols need both to use TLS properly.
In the long run, our preference is to have a TLS handshake implementation in the kernel. However, it appears that would take a long time and there is some desire to avoid adding to the Linux kernel's "attack surface" without good reasons. So in the meantime we've created a prototype handshake implementation that calls out to user space where the actual handshake can be done by an existing library implementation of TLS.
The prototype serves several purposes, including:
- Proof of concept: can a handshake upcall actually be implemented?
- Scaffold to enable prototyping upper-layer protocol support for TLS:
Is there any demand for in-kernel TLS?
- Performance impact of always-on encryption with both software and hardware kTLS
- Understanding what features, if any, an upcall handshake cannot provide
The prototype currently supports client-side PSK and anonymous x.509 ClientHello. We would like some feedback on the approach before proceeding with ServerHello and mutual x.509 authentication.
User agent: https://github.com/oracle/ktls-utils
Who will use this implementation?
--------------------------------
This series implements only the upcall. I plan to post a second series that shows how it can be used to implement the RPC-with-TlS
standard: https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/
https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git topic-rpc-with-tls
Dr. Hannes Reinecke has a series that implements NVMe-TLS here:
https://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git tls-upcall.v4
We are also working with a few developers in the CIFS community who are interested in SMB-over-QUIC. QUICv1 (RFC 9000) uses the TLSv1.3 handshake protocol, and we hope they can leverage this prototype capability when QUIC comes to the Linux kernel.
---
Chuck Lever (4):
net: Add distinct sk_psock field
net/tls: Add an AF_TLSH address family
net/tls: Add support for PF_TLSH (a TLS handshake listener) net/tls: Add observability for AF_TLSH sockets
Hannes Reinecke (1):
tls: build proto after context has been initialized
.../networking/tls-in-kernel-handshake.rst | 103 ++ include/linux/skmsg.h | 2 +- include/linux/socket.h | 5 +- include/net/sock.h | 7 +- include/net/tls.h | 15 + include/net/tlsh.h | 22 + include/uapi/linux/tls.h | 16 + net/core/skmsg.c | 6 +- net/core/sock.c | 4 +- net/socket.c | 1 + net/tls/Makefile | 2 +- net/tls/af_tlsh.c | 1078 +++++++++++++++++ net/tls/tls_main.c | 13 +- net/tls/trace.c | 3 + net/tls/trace.h | 355 ++++++ security/selinux/hooks.c | 4 +- security/selinux/include/classmap.h | 4 +- .../perf/trace/beauty/include/linux/socket.h | 4 +- 18 files changed, 1631 insertions(+), 13 deletions(-) create mode 100644 Documentation/networking/tls-in-kernel-handshake.rst create mode 100644 include/net/tlsh.h create mode 100644 net/tls/af_tlsh.c
--
Chuck Lever
RFC PATCH 1/5 net: Add distinct sk_psock field
The sk_psock facility populates the sk_user_data field with the address of an extra bit of metadata. User space sockets never populate the sk_user_data field, so this has worked out fine.
However, kernel consumers such as the RPC client and server do populate the sk_user_data field. The sk_psock() function cannot tell that the content of sk_user_data does not point to psock metadata, so it will happily return a pointer to something else, cast to a struct sk_psock.
Thus kernel consumers and psock currently cannot co-exist.
We could educate sk_psock() to return NULL if sk_user_data does not point to a struct sk_psock. However, a more general solution that enables full co-existence psock and other uses of sk_user_data might be more interesting.
Move the struct sk_psock address to its own pointer field so that the contents of the sk_user_data field is preserved.
--- include/linux/skmsg.h | 2 +- include/net/sock.h | 4 +++- net/core/skmsg.c | 6 +++--- 3 files changed, 7 insertions(+), 5 deletions(-)
include/linux/skmsg.h +1 -1
@@ -277,7 +277,7 @@ static inline void sk_msg_sg_copy_clear(struct sk_msg *msg, u32 start)static inline struct sk_psock *sk_psock (const struct sock *sk) { -return rcu_dereference_sk_user_data (sk); +return rcu_dereference (sk->sk_psock); } static inline void sk_psock_set_state(struct sk_psock *psock,
include/net/sock.h +3 -1
@@ -327,7 +327,8 @@ struct sk_filter;* @sk_tskey: counter to disambiguate concurrent tstamp requests * @sk_zckey: counter to order MSG_ZEROCOPY notifications * @sk_socket: Identd and reporting IO signals -* @sk_user_data: RPC layer private data +* @sk_user_data: Upper layer private data +* @sk_psock: socket policy data (bpf) * @sk_frag: cached page frag * @sk_peek_off: current peek_offset value * @sk_send_head: front of stuff to transmit
@@ -519,6 +520,7 @@ struct sock {struct socket *sk_socket;void *sk_user_data; +struct sk_psock __rcu *sk_psock;#ifdef CONFIG_SECURITY void *sk_security;#endif
net/core/skmsg.c +3 -3
@@ -695,7 +695,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node)write_lock_bh (&sk->sk_callback_lock); -if (sk->sk_user_data) { +if (sk->sk_psock) { psock =ERR_PTR (-EBUSY ); goto out; }
@@ -726,7 +726,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node)sk_psock_set_state (psock,SK_PSOCK_TX_ENABLED );refcount_set (&psock->refcnt,1 ); -rcu_assign_sk_user_data_nocopy (sk, psock); +rcu_assign_pointer (sk->sk_psock, psock);sock_hold (sk); out:
@@ -825,7 +825,7 @@ void sk_psock_drop(struct sock *sk, struct sk_psock *psock) {write_lock_bh (&sk->sk_callback_lock);sk_psock_restore_proto (sk, psock); -rcu_assign_sk_user_data (sk,NULL ); +rcu_assign_pointer (sk->sk_psock,NULL );if (psock->progs.stream_parser)sk_psock_stop_strp (sk, psock);else if (psock->progs.stream_verdict || psock->progs.skb_verdict)
Re: [PATCH RFC 1/5] net: Add distinct sk_psock field
Quoted 20 lines · Chuck Lever
The
sk_psockfacility populates thesk_user_datafield with the address of an extra bit of metadata. User space sockets never populate thesk_user_datafield, so this has worked out fine.However, kernel consumers such as the RPC client and server do populate the
sk_user_datafield. Thesk_psock()function cannot tell that the content ofsk_user_datadoes not point to psock metadata, so it will happily return a pointer to something else, cast to a structsk_psock.Thus kernel consumers and psock currently cannot co-exist.
We could educate
sk_psock()to return NULL ifsk_user_datadoes not point to a structsk_psock. However, a more general solution that enables full co-existence psock and other uses ofsk_user_datamight be more interesting.Move the struct
sk_psockaddress to its own pointer field so that the contents of thesk_user_datafield is preserved.
Quoted 7 lines
Signed-off-by Chuck Lever <[email protected]>--- include/linux/skmsg.h | 2 +- include/net/sock.h | 4 +++- net/core/skmsg.c | 6 +++--- 3 files changed, 7 insertions(+), 5 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
RFC PATCH 1/5 net: Add distinct sk_psock field
Quoted 28 lines · Chuck Lever
Quoted 20 lines
The
sk_psockfacility populates thesk_user_datafield with the address of an extra bit of metadata. User space sockets never populate thesk_user_datafield, so this has worked out fine.However, kernel consumers such as the RPC client and server do populate the
sk_user_datafield. Thesk_psock()function cannot tell that the content ofsk_user_datadoes not point to psock metadata, so it will happily return a pointer to something else, cast to a structsk_psock.Thus kernel consumers and psock currently cannot co-exist.
We could educate
sk_psock()to return NULL ifsk_user_datadoes not point to a structsk_psock. However, a more general solution that enables full co-existence psock and other uses ofsk_user_datamight be more interesting.Move the struct
sk_psockaddress to its own pointer field so that the contents of thesk_user_datafield is preserved.Quoted 7 lines
Signed-off-by Chuck Lever <[email protected]>--- include/linux/skmsg.h | 2 +- include/net/sock.h | 4 +++- net/core/skmsg.c | 6 +++--- 3 files changed, 7 insertions(+), 5 deletions(-)
Quoted 5 lines
Reviewed-by Hannes Reinecke <[email protected]>Cheers,
Hannes
In Patchwork website, this patch fails the checks on netdev/cc_maintainers.
So maybe you need CC folks pointed out by scripts/get_maintainer.pl script, which is suggested by Jakub Kicinski <[email protected]>.
What's more, Syskaller reports
refcount bug in sk_psock_get (2).
In this bug, the problem is that smc and psock, both use sk_user_data field to save their private data. So they will treat field in their own way.
Quoted 5 lines
in
smc_switch_to_fallback(), and setsmc->clcsock->sk_user_datato origin smc insmc_fback_replace_callbacks().Later,
sk_psock_get()will treat thesmc->clcsock->sk_user_dataassk_psocktype, which triggers the refcnt warning.
I have tested this patch and the reproducer did not trigger any issue. For more details, you can check the email
[PATCH] smc: fix refcount bug in sk_psock_get (2)
_______________________________________________
Linux-kernel-mentees mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
RFC PATCH 2/5 tls: build proto after context has been initialized
In reply to Chuck LeverFrom: Hannes Reinecke <[email protected]>
We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.
--- net/tls/tls_main.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
net/tls/tls_main.c +1 -2
@@ -836,8 +836,6 @@ static int tls_init(struct sock *sk)struct tls_context *ctx;int rc =0 ; -tls_build_proto (sk); -#ifdef CONFIG_TLS_TOE if (tls_toe_bypass (sk))return 0 ;
@@ -862,6 +860,7 @@ static int tls_init(struct sock *sk) ctx->tx_conf =TLS_BASE ; ctx->rx_conf =TLS_BASE ; +tls_build_proto (sk);update_sk_prot (sk, ctx); out:write_unlock_bh (&sk->sk_callback_lock);
Jakub Kicinski
maintainerRe: [PATCH RFC 2/5] tls: build proto after context has been initialized
Quoted 5 lines · On Mon, 18 Apr 2022 12:49:36 -0400 Chuck Lever wrote:
From: Hannes Reinecke <[email protected]>
We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.
Can you say more about the crash you see? protos are not used until at least one socket calls update_sk_prot().
Re: [PATCH RFC 2/5] tls: build proto after context has been initialized
Quoted 13 lines · Jakub Kicinski
Quoted 5 lines · On Mon, 18 Apr 2022 12:49:36 -0400 Chuck Lever wrote:
From: Hannes Reinecke <[email protected]>
We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.
Can you say more about the crash you see? protos are not used until at least one socket calls
update_sk_prot().
Hannes reported the crash and supplied the fix. I didn't have a problem with this patch series without this patch. Hannes?
--
Chuck Lever
Re: [PATCH RFC 2/5] tls: build proto after context has been initialized
In reply to Chuck LeverQuoted 36 lines · Chuck Lever
From: Hannes Reinecke <[email protected]>
We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.
Signed-off-by Hannes Reinecke <[email protected]>Signed-off-by Chuck Lever <[email protected]>---
net/tls/tls_main.c| 3 +--1 file changed, 1 insertion(+), 2 deletions(-)net/tls/tls_main.c +1
-2@@ -836,8 +836,6 @@ static int tls_init(struct sock *sk)struct tls_context *ctx;int rc =0 ; -tls_build_proto (sk); -#ifdef CONFIG_TLS_TOE if (tls_toe_bypass (sk))return 0 ;@@ -862,6 +860,7 @@ static int tls_init(struct sock *sk) ctx->tx_conf =TLS_BASE ; ctx->rx_conf =TLS_BASE ; +tls_build_proto (sk);update_sk_prot (sk, ctx);out:
Quoted 3 lines
write_unlock_bh(&sk->sk_callback_lock);
While we're working on a better upcall mechanism, I'm dropping this one from the series. tls_build_proto() can grab a mutex, so tls_init() can't call it while holding sk_callback_lock.
--
Chuck Lever
RFC PATCH 3/5 net/tls: Add an AF_TLSH address family
In reply to Chuck LeverAdd definitions for an AF_TLSH address family. The next patch explains its purpose and operation.
--- include/linux/socket.h | 4 +++- net/core/sock.c | 2 +- net/socket.c | 1 + security/selinux/hooks.c | 4 +++- security/selinux/include/classmap.h | 4 +++- tools/perf/trace/beauty/include/linux/socket.h | 4 +++- 6 files changed, 14 insertions(+), 5 deletions(-)
include/linux/socket.h +3 -1
@@ -226,8 +226,9 @@ struct ucred {#define AF_MCTP 45/* Management component * transport protocol */ +#define AF_TLSH 46/* TLS handshake request */ -#define AF_MAX 46/* For now.. */ +#define AF_MAX 47/* For now.. */ /* Protocol families, same as address families. */ #define PF_UNSPEC AF_UNSPEC
@@ -278,6 +279,7 @@ struct ucred {#define PF_SMC AF_SMC#define PF_XDP AF_XDP#define PF_MCTP AF_MCTP +#define PF_TLSH AF_TLSH#define PF_MAX AF_MAX/* Maximum queue length specifiable by listen. */
net/core/sock.c +1 -1
@@ -224,7 +224,7 @@ static struct lock_class_key af_family_kern_slock_keys[AF_MAX]; x"AF_IEEE802154" , x"AF_CAIF" , x"AF_ALG" , \ x"AF_NFC" , x"AF_VSOCK" , x"AF_KCM" , \ x"AF_QIPCRTR" , x"AF_SMC" , x"AF_XDP" , \ - x"AF_MCTP" , \ + x"AF_MCTP" , x"AF_TLSH" , \ x"AF_MAX" static const char *const af_family_key_strings[AF_MAX+1] = {
net/socket.c +1 -0
@@ -214,6 +214,7 @@ static const char * const pf_family_names[] = { [PF_SMC] = "PF_SMC", [PF_XDP] = "PF_XDP", [PF_MCTP] = "PF_MCTP", + [PF_TLSH] = "PF_TLSH", };/*
security/selinux/hooks.c +3 -1
@@ -1259,7 +1259,9 @@ static inline u16 socket_type_to_security_class(int family, int type, int protocreturn SECCLASS_XDP_SOCKET ;case PF_MCTP :return SECCLASS_MCTP_SOCKET ; -#if PF_MAX >46 +case PF_TLSH : +return SECCLASS_TLSH_SOCKET ; +#if PF_MAX >47 #error New address family defined, please update this function.#endif }
security/selinux/include/classmap.h +3 -1
@@ -237,6 +237,8 @@ struct security_class_mapping secclass_map[] = { { COMMON_SOCK_PERMS, NULL } }, {"smc_socket" , { COMMON_SOCK_PERMS, NULL } }, + {"tlsh_socket" , + { COMMON_SOCK_PERMS, NULL } }, { "infiniband_pkey", { "access", NULL } }, { "infiniband_endport",
@@ -257,6 +259,6 @@ struct security_class_mapping secclass_map[] = { { NULL } }; -#if PF_MAX >46 +#if PF_MAX >47 #error New address family defined, please update secclass_map.#endif
tools/perf/trace/beauty/include/linux/socket.h +3 -1
@@ -226,8 +226,9 @@ struct ucred {#define AF_MCTP 45/* Management component * transport protocol */ +#define AF_TLSH 46/* TLS handshake request */ -#define AF_MAX 46/* For now.. */ +#define AF_MAX 47/* For now.. */ /* Protocol families, same as address families. */ #define PF_UNSPEC AF_UNSPEC
@@ -278,6 +279,7 @@ struct ucred {#define PF_SMC AF_SMC#define PF_XDP AF_XDP#define PF_MCTP AF_MCTP +#define PF_TLSH AF_TLSH#define PF_MAX AF_MAX/* Maximum queue length specifiable by listen. */
Re: [PATCH RFC 3/5] net/tls: Add an AF_TLSH address family
Quoted 4 lines · Chuck Lever
Add definitions for an
AF_TLSHaddress family. The next patch explains its purpose and operation.Signed-off-by Chuck Lever <[email protected]>
Quoted 9 lines
--- include/linux/socket.h | 4 +++- net/core/sock.c | 2 +- net/socket.c | 1 + security/selinux/hooks.c | 4 +++- security/selinux/include/classmap.h | 4 +++- tools/perf/trace/beauty/include/linux/socket.h | 4 +++- 6 files changed, 14 insertions(+), 5 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
RFC PATCH 4/5 net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Chuck LeverIn-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
--- .../networking/tls-in-kernel-handshake.rst | 103 ++ include/linux/socket.h | 1 include/net/sock.h | 3 include/net/tls.h | 15 include/net/tlsh.h | 22 include/uapi/linux/tls.h | 16 net/core/sock.c | 2 net/tls/Makefile | 2 net/tls/af_tlsh.c | 1040 ++++++++++++++++++++ net/tls/tls_main.c | 10 10 files changed, 1213 insertions(+), 1 deletion(-) create mode 100644 Documentation/networking/tls-in-kernel-handshake.rst create mode 100644 include/net/tlsh.h create mode 100644 net/tls/af_tlsh.c
Documentation/networking/tls-in-kernel-handshake.rst new file +103 -0
@@ -0,0 +1,103 @@ +.. _kernel_tls: + +======================= +In-Kernel TLS Handshake +======================= + +Overview +======== + +Transport Layer Security (TLS) is a Upper Layer Protocol (ULP) that runs over +TCP. TLS provides end-to-end data integrity and confidentiality. + +kTLS handles the TLS record subprotocol, but does not handle the TLS handshake +subprotocol, used to establish a TLS session. In user space, a TLS library +performs the handshake on a socket which is converted to kTLS operation. In +the kernel it is much the same. The TLS handshake is done in user space by a +library TLS implementation. + + +User agent +========== + +With the current implementation, a user agent is started in each network +namespace where a kernel consumer might require a TLS handshake. This agent +listens on an AF_TLSH socket for requests from the kernel to perform a +handshake on an open and connected TCP socket. + +The open socket is passed to user space via accept(), which creates a file +descriptor. If the handshake completes successfully, the user agent promotes +the socket to use the TLS ULP and sets the session information using the +SOL_TLS socket options. The user agent returns the socket to the kernel by +closing the accepted file descriptor. + + +Kernel Handshake API +==================== + +A kernel consumer initiates a client-side TLS handshake on an open +socket by invoking one of the tls_client_hello() functions. For +example: + +.. code-block:: c + + ret = tls_client_hello_x509(sock, done_func, cookie, priorities, + peerid, cert); + +The function returns zero when the handshake request is under way. A +zero return guarantees the callback function @done_func will be invoked +for this socket. + +The function returns a negative errno if the handshake could not be +started. A negative errno guarantees the callback function @done_func +will not be invoked on this socket. + +The @sock argument is an open and connected IPPROTO_TCP socket. The +caller must hold a reference on the socket to prevent it from being +destroyed while the handshake is in progress. + +@done_func and @cookie are a callback function that is invoked when the +handshake has completed (either successfully or not). The success status +of the handshake is returned via the @status parameter of the callback +function. A good practice is to close and destroy the socket immediately +if the handshake has failed. + +@priorities is a GnuTLS priorities string that controls the handshake. +The special value TLSH_DEFAULT_PRIORITIES causes the handshake to +operate using user space configured default TLS priorities. However, +the caller can use the string to (for example) adjust the handshake to +use a restricted set of ciphers (say, if the kernel is in FIPS mode or +the kernel consumer wants to mandate only a limited set of ciphers). + +@peerid is the serial number of a key on the XXXYYYZZZ keyring that +contains a private key. + +@cert is the serial number of a key on the XXXYYYYZZZ keyring that +contains a {PEM,DER} format x.509 certificate that the user agent +presents to the server as the local peer's identity. + +To initiate a client-side TLS handshake with a pre-shared key, use: + +.. code-block:: c + + ret = tls_client_hello_psk(sock, done_func, cookie, priorities, + peerid); + +@peerid is the serial number of a key on the XXXYYYZZZ keyring that +contains the pre-shared key. + +The other parameters are as above. + + +Other considerations +-------------------- + +While the handshake is under way, the kernel consumer must alter the +socket's sk_data_ready callback function to ignore incoming data. +Once the callback function has been invoked, normal receive operation +can be resumed. + +See tls.rst for details on how a kTLS consumer recognizes incoming +(decrypted) application data, alerts, and handshake packets once the +socket has been promoted to use the TLS ULP. +
include/linux/socket.h +1 -0
@@ -369,6 +369,7 @@ struct ucred {#define SOL_MPTCP 284#define SOL_MCTP 285#define SOL_SMC 286 +#define SOL_TLSH 287/* IPX options */ #define IPX_TYPE 1
include/net/sock.h +3 -0
@@ -353,6 +353,7 @@ struct sk_filter;* @sk_txtime_report_errors: set report errors mode for SO_TXTIME * @sk_txtime_unused: unused txtime flags * @ns_tracker: tracker for netns reference +* @sk_tlsh_priv: private data for TLS handshake upcall */ struct sock {/*
@@ -544,6 +545,8 @@ struct sock {#endif struct rcu_head sk_rcu;netns_tracker ns_tracker; + +void *sk_tlsh_priv; };enum sk_pacing {
include/net/tls.h +15 -0
@@ -51,6 +51,18 @@#include <uapi/linux/tls.h> +struct tlsh_sock { +/* struct sock must remain the first field */ +struct sock th_sk; + +int th_bind_family; +}; + +static inline struct tlsh_sock *tlsh_sk (struct sock *sk) +{ +return (struct tlsh_sock *)sk; +} +/* Maximum data size carried in a TLS record */ #define TLS_MAX_PAYLOAD_SIZE ((size_t)1 << 14)
@@ -356,6 +368,9 @@ struct tls_context *tls_ctx_create(struct sock *sk);void tls_ctx_free (struct sock *sk,struct tls_context *ctx);void update_sk_prot (struct sock *sk,struct tls_context *ctx); +int tlsh_pf_create (struct net *net,struct socket *sock,int protocol, +int kern); +int wait_on_pending_writer (struct sock *sk,long *timeo);int tls_sk_query (struct sock *sk,int optname,char __user *optval,int __user *optlen);
include/net/tlsh.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* PF_TLSH protocol family socket handler. +* +* Author: Chuck Lever <[email protected]> +* +* Copyright (c) 2021, Oracle and/or its affiliates. +*/ + +#ifndef _TLS_HANDSHAKE_H +#define _TLS_HANDSHAKE_H + +extern int tls_client_hello_psk (struct socket *sock, +void (*done)(void *data,int status), +void *data,const char *priorities, +key_serial_t peerid); +extern int tls_client_hello_x509 (struct socket *sock, +void (*done)(void *data,int status), +void *data,const char *priorities, +key_serial_t peerid,key_serial_t cert); + +#endif /* _TLS_HANDSHAKE_H */
include/uapi/linux/tls.h +16 -0
@@ -40,6 +40,22 @@#define TLS_TX 1/* Set transmit parameters */ #define TLS_RX 2/* Set receive parameters */ +/* TLSH socket options */ +#define TLSH_PRIORITIES 1/* Retrieve TLS priorities string */ +#define TLSH_PEERID 2/* Retrieve peer identity */ +#define TLSH_HANDSHAKE_TYPE 3/* Retrieve handshake type */ +#define TLSH_X509_CERTIFICATE 4/* Retrieve x.509 certificate */ + +#define TLSH_DEFAULT_PRIORITIES (NULL) +#define TLSH_NO_PEERID (0) +#define TLSH_NO_CERT (0) + +/* TLSH handshake types */ +enum tlsh_hs_type { +TLSH_TYPE_CLIENTHELLO_X509 , +TLSH_TYPE_CLIENTHELLO_PSK , +}; +/* Supported versions */ #define TLS_VERSION_MINOR (ver) ((ver) & 0xFF)#define TLS_VERSION_MAJOR (ver) (((ver) >> 8) & 0xFF)
net/core/sock.c +2 -0
@@ -3295,6 +3295,8 @@ void sock_init_data(struct socket *sock, struct sock *sk) sk->sk_incoming_cpu =-1 ; sk->sk_txrehash =SOCK_TXREHASH_DEFAULT ; + sk->sk_tlsh_priv =NULL ; +sk_rx_queue_clear (sk);/* * Before updating sk_refcnt, we must commit prior changes to memory
net/tls/Makefile +1 -1
@@ -7,7 +7,7 @@ CFLAGS_trace.o := -I$(src) obj-$(CONFIG_TLS) += tls.o -tls-y := tls_main.o tls_sw.o tls_proc.o trace.o +tls-y := af_tlsh.o tls_main.o tls_sw.o tls_proc.o trace.o tls-$(CONFIG_TLS_TOE) += tls_toe.o tls-$(CONFIG_TLS_DEVICE) += tls_device.o tls_device_fallback.o
net/tls/af_tlsh.c new file +1040 -0
1,040 lines not on this page · raw message
net/tls/tls_main.c +10 -0
@@ -49,6 +49,7 @@ MODULE_AUTHOR("Mellanox Technologies");MODULE_DESCRIPTION ("Transport Layer Security Support" );MODULE_LICENSE ("Dual BSD/GPL" );MODULE_ALIAS_TCP_ULP ("tls" ); +MODULE_ALIAS_NETPROTO (PF_TLSH );enum {TLSV4 ,
@@ -982,6 +983,12 @@ static struct tcp_ulp_ops tcp_tls_ulp_ops __read_mostly = { .get_info_size = tls_get_info_size, }; +static const struct net_proto_family tlsh_pf_ops = { + .family =PF_TLSH , + .create = tlsh_pf_create, + .owner =THIS_MODULE , +}; +static int __init tls_register (void ) {int err;
@@ -993,11 +1000,14 @@ static int __init tls_register(void)tls_device_init ();tcp_register_ulp (&tcp_tls_ulp_ops); +sock_register (&tlsh_pf_ops); +return 0 ; } static void __exit tls_unregister(void) { +sock_unregister (PF_TLSH );tcp_unregister_ulp (&tcp_tls_ulp_ops);tls_device_cleanup ();unregister_pernet_subsys (&tls_proc_ops);
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 41 lines · Chuck Lever
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Signed-off-by Chuck Lever <[email protected]>--- .../networking/tls-in-kernel-handshake.rst | 103 ++ include/linux/socket.h | 1 include/net/sock.h | 3 include/net/tls.h | 15 include/net/tlsh.h | 22 include/uapi/linux/tls.h | 16 net/core/sock.c | 2 net/tls/Makefile | 2 net/tls/af_tlsh.c | 1040 ++++++++++++++++++++ net/tls/tls_main.c | 10 10 files changed, 1213 insertions(+), 1 deletion(-)
Quoted 4 lines
create mode 100644
Documentation/networking/tls-in-kernel-handshake.rstcreate mode 100644include/net/tlsh.h
create mode 100644net/tls/af_tlsh.c
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Chuck LeverQuoted 19 lines · On Mon, 18 Apr 2022 12:49:50 -0400 Chuck Lever wrote:
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~
Quoted 8 lines
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 25 lines · Jakub Kicinski
Quoted 24 lines · On Mon, 18 Apr 2022 12:49:50 -0400 Chuck Lever wrote:
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Quoted 8 lines
Quoted 3 lines
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible. Not to mention having to rework the nvme driver to accept sockets from userspace instead of creating them internally.
With this approach we can keep existing infrastructure, and can get a common implementation for either transport.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
Sagi Grimberg
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 11 lines
Quoted 9 lines
Quoted 4 lines
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.
And we will still need the upcall anyways when we reconnect (re-establish the socket)
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 11 lines · On Tue, 26 Apr 2022 17:29:03 +0300 Sagi Grimberg wrote:
Quoted 8 lines
Quoted 4 lines
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.And we will still need the upcall anyways when we reconnect (re-establish the socket)
That totally flew over my head, I have zero familiarity with in-kernel storage network users :S
In all honesty the tls code in the kernel is a bit of a dumping ground. People come, dump a bunch of code and disappear. Nobody seems to care that the result is still (years in) not ready for production use :/ Until a month ago it'd break connections even under moderate memory pressure. This set does not even have selftests.
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 26 Apr 2022 17:29:03 +0300 Sagi Grimberg wrote:
Quoted 8 lines
Quoted 4 lines
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.
Quoted 6 lines
Quoted 2 lines
And we will still need the upcall anyways when we reconnect (re-establish the socket)
That totally flew over my head, I have zero familiarity with in-kernel storage network users :S
Count yourself lucky.
Quoted 6 lines
In all honesty the tls code in the kernel is a bit of a dumping ground. People come, dump a bunch of code and disappear. Nobody seems to care that the result is still (years in) not ready for production use :/ Until a month ago it'd break connections even under moderate memory pressure. This set does not even have selftests.
Well, I'd been surprised that it worked, too.
And even more so that Boris Piskenny @ Nvidia is actively working on it. (Thanks, Sagi!)
Quoted 3 lines
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
Which is precisely what we hope to discuss at LSF.
(Yes, I know, probably not the best venue to discuss network stuff ...)
Each approach has its drawbacks:
- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established.
(And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here)
(Especially call_usermodehelper() is out of the game)
- Having ULP helpers (as with this design) mitigates that problem somewhat in the sense that you can mlock() that daemon and having it polling on an intermediate socket; that solves the notification problem. But you have to have ULP special sauce here to make it work.
- Moving everything in kernel is ... possible. But then you have yet another security-relevant piece of code in the kernel which needs to be audited, CVEd etc. Not to mention the usual policy discussion whether it really belongs into the kernel.
So I don't really see any obvious way to go; best we can do is to pick the least ugly :-(
Cheers,
Hannes
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 6 lines · On Tue, 26 Apr 2022 17:58:39 +0200 Hannes Reinecke wrote:
Quoted 3 lines
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
Which is precisely what we hope to discuss at LSF.
(Yes, I know, probably not the best venue to discuss network stuff ...)
Indeed.
Quoted 8 lines
Each approach has its drawbacks:
- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established. (And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here) (Especially
call_usermodehelper()is out of the game)
Indeed, we may need _some_ form of a notification mechanism and that's okay. Can be a (more generic) socket, can be something based on existing network storage APIs (IDK what you have there).
My thinking was that establishing the session in user space would be easiest. We wouldn't need all the special getsockopt()s which AFAIU work around part of the handshake being done in the kernel, and which, I hope we can agree, are not beautiful.
Quoted 4 lines
- Having ULP helpers (as with this design) mitigates that problem somewhat in the sense that you can
mlock()that daemon and having it polling on an intermediate socket; that solves the notification problem. But you have to have ULP special sauce here to make it work.
TBH I don't see how this is much different to option 1 in terms of constraints & requirements on the user space agent. We can implement option 1 over a socket-like interface, too, and that'll carry notifications all the same.
Quoted 4 lines
- Moving everything in kernel is ... possible. But then you have yet another security-relevant piece of code in the kernel which needs to be audited, CVEd etc. Not to mention the usual policy discussion whether it really belongs into the kernel.
Yeah, if that gets posted it'd be great if it includes removing me from the TLS maintainers 'cause I want to sleep at night ;)
Quoted 2 lines
So I don't really see any obvious way to go; best we can do is to pick the least ugly :-(
True, I'm sure we can find some middle ground between 1 and 2. Preferably implemented in a way where the mechanism is separated from the fact it's carrying TLS handshake requests, so that it can carry something else tomorrow.
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 12 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 26 Apr 2022 17:58:39 +0200 Hannes Reinecke wrote:
- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established. (And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here) (Especially
call_usermodehelper()is out of the game)
Quoted 8 lines
Indeed, we may need _some_ form of a notification mechanism and that's okay. Can be a (more generic) socket, can be something based on existing network storage APIs (IDK what you have there).
My thinking was that establishing the session in user space would be easiest. We wouldn't need all the special getsockopt()s which AFAIU work around part of the handshake being done in the kernel, and which, I hope we can agree, are not beautiful.
In the prototype, the new socket options on AF_TLSH sockets include:
#define TLSH_PRIORITIES 1 /* Retrieve TLS priorities string */ #define TLSH_PEERID 2 /* Retrieve peer identity */ #define TLSH_HANDSHAKE_TYPE 3 /* Retrieve handshake type */ #define TLSH_X509_CERTIFICATE 4 /* Retrieve x.509 certificate */
PRIORITIES is the TLS priorities string that the GnuTLS library uses to parametrize the handshake (which TLS versions, ciphers, and so on).
PEERID is a keyring serial number for the key that contains the a Pre-Shared Key (for PSK handshakes) or the private key (for x.509 handshakes).
HANDSHAKE_TYPE is an integer that represents the type of handshake being requested: ClientHello, ServerHello, Rekey, and so on. This option enables the repertoire of handshake types to be expanded.
X509_CERTIFICATE is a keyring serial number for the key that contains an x.509 certificate.
When each handshake is complete, the handshake agent instantiates the IV into the passed-in socket using existing kTLS socket options before it returns the endpoint to the kernel.
There is nothing in these options that indicates to the handshake agent what upper layer protocol is going to be used inside the TLS session.
----
The new AF_TLSH socket options are not there because the handshake is split between the kernel and user space. They are there because the initial requester is (eg, in the case of NFS) mount.nfs, another user space program. mount.nfs has to package up an x.509 cert or pre-shared key and place it on a keyring to make it available to the handshake agent.
The basic issue is that the administrative interfaces that parametrize the handshakes are quite distant from the in-kernel consumers that make handshake requests.
----
Further, in order to support server side TLS handshakes in the kernel, we really do have to pass a kernel-created socket up to user space. NFSD (and maybe the NVMe target) use in-kernel listeners to accept incoming connections. Each new endpoint is created in the kernel.
So if you truly seek generality in this facility, the user space componentry must work with passed-in sockets rather than creating them in user space.
--
Chuck Lever
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Jakub KicinskiQuoted 19 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 26 Apr 2022 17:58:39 +0200 Hannes Reinecke wrote:
Quoted 3 lines
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
Which is precisely what we hope to discuss at LSF.
(Yes, I know, probably not the best venue to discuss network stuff ...)Indeed.
Quoted 8 lines
Each approach has its drawbacks:
- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established. (And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here) (Especially
call_usermodehelper()is out of the game)
Quoted 4 lines
Indeed, we may need _some_ form of a notification mechanism and that's okay. Can be a (more generic) socket, can be something based on existing network storage APIs (IDK what you have there).
Which is one of the issues; we don't have any.
Network storage is accessed from userspace via read()/write(), and for control everyone rolls its own thing. So speaking of a 'network storage API' is quite optimistic.
Quoted 5 lines
My thinking was that establishing the session in user space would be easiest. We wouldn't need all the special getsockopt()s which AFAIU work around part of the handshake being done in the kernel, and which, I hope we can agree, are not beautiful.
Well; that is open to debate.
During open-iscsi development (which followed your model of having a control- and dataplane split between userspace and kernel) we found that the model is great for keeping the in-kernel code simple. But we also found that it's not so great once you come down to the gritty details; you have to duplicate quite some protocol handling in userspace, and you have to worry about session re-establishment. Up to the point where we start wondering if moving things into userspace was a good idea at all...
As for this particular interface: the problem we're facing is that TLS has to be started in-band. For some reason NVMexpress decided not to follow the traditional method of establishing the socket, start TLS, and then start protocol processing, but it rather has specified a hybrid model: establishing the socket, doing authentication, _then_ start TLS, and only then continue with protocol processing on the TLS socket.
Moving _that_ into userspace would require us to move the most of the protocol logic into userspace, too. Which really is something we want to avoid, as this would be quite a code duplication. Not to mention the maintenance burden, as issues would need to be fixed in two locations.
(Talk to me about iscsiuio.)
And that's before we even get to things like TLS session tickets, which opens yet another can of worms.
Quoted 18 lines
Quoted 4 lines
- Having ULP helpers (as with this design) mitigates that problem somewhat in the sense that you can
mlock()that daemon and having it polling on an intermediate socket; that solves the notification problem. But you have to have ULP special sauce here to make it work.TBH I don't see how this is much different to option 1 in terms of constraints & requirements on the user space agent. We can implement option 1 over a socket-like interface, too, and that'll carry notifications all the same.
Quoted 4 lines
- Moving everything in kernel is ... possible. But then you have yet another security-relevant piece of code in the kernel which needs to be audited, CVEd etc. Not to mention the usual policy discussion whether it really belongs into the kernel.
Yeah, if that gets posted it'd be great if it includes removing me from the TLS maintainers 'cause I want to sleep at night ;)
Quoted 8 lines
Quoted 2 lines
So I don't really see any obvious way to go; best we can do is to pick the least ugly :-(
True, I'm sure we can find some middle ground between 1 and 2. Preferably implemented in a way where the mechanism is separated from the fact it's carrying TLS handshake requests, so that it can carry something else tomorrow.
Which was actually our goal.
The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa. While there is fd passing between processes via AF_UNIX, there is no such mechanism between kernel and userspace.
So accept() was an easy way out as the implementation was quite simple.
And it was moved into net/tls as this was the primary use-case.
But it's by no means meant to be exclusive for TLS; we could expand it to other things once there is a need. But we really didn't want to over-engineer things here.
However, if you have another idea, by all means, do tell.
It's just that I don't think creating sockets in userspace is a great fit for us.
Cheers,
Hannes
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 4 lines · On Thu, 28 Apr 2022 09:26:41 +0200 Hannes Reinecke wrote:
The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa.
While there is fd passing between processes viaAF_UNIX, there is no such mechanism between kernel and userspace.
Noob question - the kernel <> user space FD sharing is just not implemented yet, or somehow fundamentally hard because kernel fds are "special"?
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 9 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 28 Apr 2022 09:26:41 +0200 Hannes Reinecke wrote:
The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa.
While there is fd passing between processes viaAF_UNIX, there is no such mechanism between kernel and userspace.Noob question - the kernel <> user space FD sharing is just not implemented yet, or somehow fundamentally hard because kernel fds are "special"?
Noob reply: wish I knew.
(I somewhat hoped _you_ would've been able to tell me.)
Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel.
So you probably have to move a similar logic over to AF_NETLINK. And design a new interface on how fds should be passed over AF_NETLINK.
But then you have to face the issue that AF_NELINK is essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...
And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the 'accept()' mechanism Chuck implemented.
I'm open to suggestions, though.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 27 lines · Hannes Reinecke
Quoted 9 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 28 Apr 2022 09:26:41 +0200 Hannes Reinecke wrote:
The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa.
While there is fd passing between processes viaAF_UNIX, there is no such mechanism between kernel and userspace.Noob question - the kernel <> user space FD sharing is just not implemented yet, or somehow fundamentally hard because kernel fds are "special"?
Noob reply: wish I knew. (I somewhat hoped _you_ would've been able to tell me.)
Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's
AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel. So you probably have to move a similar logic over toAF_NETLINK. And design a new interface on how fds should be passed overAF_NETLINK.But then you have to face the issue that
AF_NELINKis essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...
Quoted 5 lines
And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the '
accept()' mechanism Chuck implemented.I'm open to suggestions, though.
EXPORT_SYMBOL(receive_fd) would allow interesting implementations.
The kernel keyring facilities have a good API for creating various key_types which are able to perform work such as this from userspace contexts.
I have a working prototype for a keyring key instantiation which allows a userspace process to install a kernel fd on its file table. The problem here is how to match/route such fd passing to appropriate processes in appropriate namespaces. I think this problem is shared by all kernel-to-userspace upcalls, which I hope we can discuss at LSF/MM.
I don't think kernel fds are very special as compared to userspace fds.
Ben
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 10 lines · On Thu, 28 Apr 2022 10:09:17 -0400 Benjamin Coddington wrote:
Quoted 10 lines
Noob reply: wish I knew. (I somewhat hoped _you_ would've been able to tell me.)
Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's
AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel. So you probably have to move a similar logic over toAF_NETLINK. And design a new interface on how fds should be passed overAF_NETLINK.
Quoted 5 lines
Quoted 5 lines
But then you have to face the issue that
AF_NELINKis essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...
Sort of. I double checked the netlink upcall implementations we have, they work by user space entity "registering" their netlink address (portid) at startup. Kernel then directs the upcalls to that address. But AFAICT there's currently no way for the netlink "server" to see when a "client" goes away, which makes me slightly uneasy about using such schemes for security related stuff. The user agent may crash and something else could grab the same address, I think.
Let me CC OvS who uses it the most, perhaps I'm missing a trick.
My thinking was to use the netlink attribute format (just to reuse the helpers and parsing, but we can invent a new TLV format if needed) but create a new socket type specifically for upcalls.
Quoted 8 lines
Quoted 5 lines
And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the '
accept()' mechanism Chuck implemented.I'm open to suggestions, though.
EXPORT_SYMBOL(receive_fd)would allow interesting implementations.
Quoted 8 lines
The kernel keyring facilities have a good API for creating various
key_typeswhich are able to perform work such as this from userspace contexts.I have a working prototype for a keyring key instantiation which allows a userspace process to install a kernel fd on its file table. The problem here is how to match/route such fd passing to appropriate processes in appropriate namespaces. I think this problem is shared by all kernel-to-userspace upcalls, which I hope we can discuss at LSF/MM.
Almost made me wish I was coming to LFS/MM :)
Quoted 1 line
I don't think kernel fds are very special as compared to userspace fds.
Re: [ovs-dev] [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 25 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 28 Apr 2022 10:09:17 -0400 Benjamin Coddington wrote:
Quoted 10 lines
Noob reply: wish I knew. (I somewhat hoped _you_ would've been able to tell me.)
Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's
AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel. So you probably have to move a similar logic over toAF_NETLINK. And design a new interface on how fds should be passed overAF_NETLINK.Quoted 5 lines
Quoted 5 lines
But then you have to face the issue that
AF_NELINKis essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...Sort of. I double checked the netlink upcall implementations we have, they work by user space entity "registering" their netlink address (portid) at startup. Kernel then directs the upcalls to that address. But AFAICT there's currently no way for the netlink "server" to see when a "client" goes away, which makes me slightly uneasy about using such schemes for security related stuff. The user agent may crash and something else could grab the same address, I think.
Quoted 1 line
Let me CC OvS who uses it the most, perhaps I'm missing a trick.
I don't think there are any tricks. From what I see OVS creates several netlink sockets, connects them to the kernel (nl_pid = 0) and obtains their nl_pid's from the kernel.
These pids are either just a task_tgid_vnr() or a random negative value from the [S32_MIN, -4096] range. After that OVS "registers" those pids in the openvswitch kernel module. That just means sending an array of integers to the kernel. Kernel will later use these integer pids to find the socket and send data to the userspace.
openvswitch module inside the kernel has no way to detect that socket with a certain pid no longer exists. So, it will continue to try to find the socket and send, even if the user-space process is dead.
So, if you can find a way to reliably create a process with the same task_tgid or trick the randomizer inside the netlink_autobind(), you can start receiving upcalls from the kernel in a new process, IIUC. Also, netlink_bind() allows to just specify the nl_pid for listening sockets. That might be another way.
Quoted 30 lines
My thinking was to use the netlink attribute format (just to reuse the helpers and parsing, but we can invent a new TLV format if needed) but create a new socket type specifically for upcalls.
Quoted 8 lines
Quoted 5 lines
And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the '
accept()' mechanism Chuck implemented.I'm open to suggestions, though.
EXPORT_SYMBOL(receive_fd)would allow interesting implementations.Quoted 8 lines
The kernel keyring facilities have a good API for creating various
key_typeswhich are able to perform work such as this from userspace contexts.I have a working prototype for a keyring key instantiation which allows a userspace process to install a kernel fd on its file table. The problem here is how to match/route such fd passing to appropriate processes in appropriate namespaces. I think this problem is shared by all kernel-to-userspace upcalls, which I hope we can discuss at LSF/MM.
Almost made me wish I was coming to LFS/MM :)
Quoted 1 line
I don't think kernel fds are very special as compared to userspace fds.
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Hannes ReineckeQuoted 11 lines · On Tue, 26 Apr 2022 11:43:37 +0200 Hannes Reinecke wrote:
Quoted 4 lines
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.
Not to mention having to rework the nvme driver to accept sockets from userspace instead of creating them internally.
Quoted 2 lines
With this approach we can keep existing infrastructure, and can get a common implementation for either transport.
You add 1.5kLoC and require running a user space agent, surely you're adding new interfaces and are not backward-compatible already.
I don't understand your argument, maybe you could rephrase / dumb it down for me?
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Jakub KicinskiHi Jakub-
Quoted 27 lines · Jakub Kicinski
Quoted 24 lines · On Mon, 18 Apr 2022 12:49:50 -0400 Chuck Lever wrote:
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Quoted 7 lines
Quoted 3 lines
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.
--
Chuck Lever
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 8 lines · On Tue, 26 Apr 2022 13:48:20 +0000 Chuck Lever III wrote:
Quoted 3 lines
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.
Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).
Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 14 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 26 Apr 2022 13:48:20 +0000 Chuck Lever III wrote:
Quoted 3 lines
Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.
The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.
Quoted 5 lines
Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).
Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?
RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.
You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?
Also, why has no-one mentioned the NBD on TLS implementation to us before? I will try to review that code soon.
Quoted 1 line
This set does not even have selftests.
I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.
Quoted 3 lines
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?
I'd prefer an in-kernel handshake implementation over a user space one (even one that is sharable amongst transports and ULPs as my proposal is intended to be). However, so far we've been told that an in-kernel handshake implementation is a non-starter.
But in the abstract, we agree that having a single TLS handshake mechanism for kernel consumers is preferable.
--
Chuck Lever
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 18 lines · On Tue, 26 Apr 2022 15:58:29 +0000 Chuck Lever III wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 4 lines
The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.
Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).
Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?
RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.
Quoted 4 lines
You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?
I meant scale with the number of possible crypto protocols, I mentioned three there.
Quoted 2 lines
Also, why has no-one mentioned the NBD on TLS implementation to us before? I will try to review that code soon.
Oops, maybe that thing had never seen the light of a public mailing list then :S Dave Watson was working on it at Facebook, but he since moved to greener pastures.
Quoted 6 lines
Quoted 1 line
This set does not even have selftests.
I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.
Whatever level of testing makes you as an engineer comfortable with saying "this test suite is sufficient"? ;)
For TLS we have tools/testing/selftests/net/tls.c - it's hardly an example of excellence but, you know, it catches bugs here and there.
Quoted 8 lines
Quoted 3 lines
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?
Hm, do you remember who you discussed it with? Would be good to loop those folks in. I wasn't involved at the beginning of the TLS work, I know second hand that HW offload and nbd were involved and that the design went thru some serious re-architecting along the way. In the beginning there was a separate socket for control records, and that was nacked.
But also (and perhaps most importantly) I'm not really objecting to creating the socket in the kernel. I'm primarily objecting to a second type of a special TLS socket which has TLS semantics.
Quoted 7 lines
I'd prefer an in-kernel handshake implementation over a user space one (even one that is sharable amongst transports and ULPs as my proposal is intended to be). However, so far we've been told that an in-kernel handshake implementation is a non-starter.
But in the abstract, we agree that having a single TLS handshake mechanism for kernel consumers is preferable.
For some definition of "we" which doesn't not include me?
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 23 lines · Jakub Kicinski
Quoted 18 lines · On Tue, 26 Apr 2022 15:58:29 +0000 Chuck Lever III wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 4 lines
The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.
Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).
Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?
RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.
Quoted 7 lines
Quoted 4 lines
You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?
I meant scale with the number of possible crypto protocols, I mentioned three there.
I'm looking at previous emails. The "three crypto protocols" don't stand out to me. Which ones?
The prototype has a "handshake type" option that enables the kernel to request handshakes for different transport layer security protocols. Is that the kind of scalability you mean?
For TLS, we expect to have at least:
- ClientHello
- X509
- PSK
- ServerHello
- Re-key
It should be straightforward to add the ability to service other handshake types.
Quoted 14 lines
Quoted 2 lines
Also, why has no-one mentioned the NBD on TLS implementation to us before? I will try to review that code soon.
Oops, maybe that thing had never seen the light of a public mailing list then :S Dave Watson was working on it at Facebook, but he since moved to greener pastures.
Quoted 6 lines
Quoted 1 line
This set does not even have selftests.
I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.
Quoted 6 lines
Whatever level of testing makes you as an engineer comfortable with saying "this test suite is sufficient"? ;)
For TLS we have
tools/testing/selftests/net/tls.c- it's hardly an example of excellence but, you know, it catches bugs here and there.
My question wasn't clear, sorry. I meant, what framework is appropriate to use for unit tests in this area?
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.
When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?
Hm, do you remember who you discussed it with? Would be good to loop those folks in.
Yes, I remember. Trond Myklebust discussed this with Dave Miller during a hallway conversation at a conference (probably Plumbers) in 2018 or 2019.
Trond is Cc'd on this thread via linux-nfs@ and Dave is Cc'd via netdev@.
I also traded email with Boris Pismenny about this a year ago, and if memory serves he also recommended passing an existing socket up to user space. He is Cc'd on this directly.
Quoted 9 lines
I wasn't involved at the beginning of the
TLS work, I know second hand that HW offload and nbd were involved and that the design went thru some serious re-architecting along the way. In the beginning there was a separate socket for control records, and that was nacked.But also (and perhaps most importantly) I'm not really objecting to creating the socket in the kernel. I'm primarily objecting to a second type of a special TLS socket which has TLS semantics.
I don't understand your objection. Can you clarify?
AF_TLSH is a listen-only socket. It's just a rendezvous point for passing a kernel socket up to user space. It doesn't have any particular "TLS semantics". It's the user space agent listening on that endpoint that implements particular handshake behaviors.
In fact, if the name AF_TLSH gives you hives, that can be made more generic. However, that makes it harder for the kernel to figure out which listening endpoint handles handshake requests.
Quoted 9 lines
Quoted 7 lines
I'd prefer an in-kernel handshake implementation over a user space one (even one that is sharable amongst transports and ULPs as my proposal is intended to be). However, so far we've been told that an in-kernel handshake implementation is a non-starter.
But in the abstract, we agree that having a single TLS handshake mechanism for kernel consumers is preferable.
For some definition of "we" which doesn't not include me?
The double negative made me blink a couple of times.
I'm working with folks from the Linux NFS community, the Linux block community, and the Linux SMB community. We would be happy to include you in our effort, if you would like to be more involved.
--
Chuck Lever
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 12 lines · On Wed, 27 Apr 2022 14:42:53 +0000 Chuck Lever III wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines
RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.
You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?
Quoted 5 lines
Quoted 2 lines
I meant scale with the number of possible crypto protocols, I mentioned three there.
I'm looking at previous emails. The "three crypto protocols" don't stand out to me. Which ones?
TLS, QUIC and PSP maybe that was in a different email that what you quoted, sorry:
https://lore.kernel.org/all/[email protected]/
PSP:
https://raw.githubusercontent.com/google/psp/main/doc/PSP_Arch_Spec.pdf
Quoted 23 lines
The prototype has a "handshake type" option that enables the kernel to request handshakes for different transport layer security protocols. Is that the kind of scalability you mean?
For TLS, we expect to have at least:
- ClientHello
- X509
- PSK
- ServerHello
- Re-keyIt should be straightforward to add the ability to service other handshake types.
Quoted 8 lines
Quoted 4 lines
I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.
Whatever level of testing makes you as an engineer comfortable with saying "this test suite is sufficient"? ;)
Quoted 6 lines
Quoted 3 lines
For TLS we have
tools/testing/selftests/net/tls.c- it's hardly an example of excellence but, you know, it catches bugs here and there.My question wasn't clear, sorry. I meant, what framework is appropriate to use for unit tests in this area?
Nothing area specific, tools/testing/selftests/kselftest_harness.h is what the tls test uses.
Quoted 12 lines
Quoted 7 lines
Quoted 4 lines
When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?
Hm, do you remember who you discussed it with? Would be good to loop those folks in.
Yes, I remember. Trond Myklebust discussed this with Dave Miller during a hallway conversation at a conference (probably Plumbers) in 2018 or 2019.
Quoted 6 lines
Trond is Cc'd on this thread via linux-nfs@ and Dave is Cc'd via netdev@.
I also traded email with Boris Pismenny about this a year ago, and if memory serves he also recommended passing an existing socket up to user space. He is Cc'd on this directly.
I see.
Quoted 12 lines
Quoted 9 lines
I wasn't involved at the beginning of the
TLS work, I know second hand that HW offload and nbd were involved and that the design went thru some serious re-architecting along the way. In the beginning there was a separate socket for control records, and that was nacked.But also (and perhaps most importantly) I'm not really objecting to creating the socket in the kernel. I'm primarily objecting to a second type of a special TLS socket which has TLS semantics.
I don't understand your objection. Can you clarify?
Quoted 8 lines
AF_TLSHis a listen-only socket. It's just a rendezvous point for passing a kernel socket up to user space. It doesn't have any particular "TLS semantics". It's the user space agent listening on that endpoint that implements particular handshake behaviors.In fact, if the name
AF_TLSHgives you hives, that can be made more generic.
Yes, a more generic "user space please bake my socket" interface is what I'm leaning towards.
Quoted 2 lines
However, that makes it harder for the kernel to figure out which listening endpoint handles handshake requests.
Right, the listening endpoint...
Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the getsockopt() now)?
The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via recv()?
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 23 lines · Jakub Kicinski
Quoted 12 lines · On Wed, 27 Apr 2022 14:42:53 +0000 Chuck Lever III wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines
RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.
You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?
Quoted 5 lines
Quoted 2 lines
I meant scale with the number of possible crypto protocols, I mentioned three there.
I'm looking at previous emails. The "three crypto protocols" don't stand out to me. Which ones?
Quoted 6 lines
TLS, QUIC and PSP maybe that was in a different email that what you quoted, sorry:
https://lore.kernel.org/all/[email protected]/PSP:
https://raw.githubusercontent.com/google/psp/main/doc/PSP_Arch_Spec.pdf
During the design process, we discussed both TLS and QUIC handshake requirements, which are nearly the same. QUIC will want a TLSv1.3 handshake on a UDP socket, effectively. We can support DTLS in a similar fashion.
We hope that the proposed design can be used for all of those, and barring anything unforeseen in the description of PSP you provided, PSP can be supported as well.
The handshake agent is really only a shim around a TLS library. There isn't much to it.
Quoted 7 lines
Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the
getsockopt()now)?The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via
recv()?
We considered this kind of design. A reasonable place to start there would be to fabricate new NETLINK messages to do this. I don't see much benefit over what is done now, it's just a different isomer of syntactic sugar, but it could be considered.
The issue is how the connected socket is materialized in user space. accept(2) is the historical way to instantiate an already connected socket in a process's file table, and seems like a natural fit. When the handshake agent is done with the handshake, it closes the socket. This invokes the tlsh_release() function which can check whether the IV implantation was successful.
So instead of an AF_TLSH listener we could use a named pipe or a netlink socket and a blocking recv(), as long as there is a reasonable solution to how a connected socket fd is attached to the handshake agent process.
I'm flexible about the mechanism for passing handshake parameters. Attaching them to the connected socket seems convenient, but perhaps not aesthetic.
--
Chuck Lever
Jakub Kicinski
maintainerRe: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 13 lines · On Thu, 28 Apr 2022 01:29:10 +0000 Chuck Lever III wrote:
Quoted 7 lines
Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the
getsockopt()now)?The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via
recv()?We considered this kind of design. A reasonable place to start there would be to fabricate new NETLINK messages to do this. I don't see much benefit over what is done now, it's just a different isomer of syntactic sugar, but it could be considered.
Quoted 5 lines
The issue is how the connected socket is materialized in user space. accept(2) is the historical way to instantiate an already connected socket in a process's file table, and seems like a natural fit. When the handshake agent is done with the handshake, it closes the socket. This invokes the
tlsh_release()function which can check
Actually - is that strictly necessary? It seems reasonable for NFS to check that it got TLS, since that's what it explicitly asks for per standard. But it may not always be the goal. In large data center networks there can be a policy the user space agent consults to choose what security to install. It may end up doing the auth but not enable crypto if confidentiality is deemed unnecessary.
Obviously you may not have those requirements but if we can cover them without extra complexity it'd be great.
Quoted 1 line
whether the IV implantation was successful.
I'm used to IV meaning Initialization Vector in context of crypto, what does "IV implementation" stand for?
Quoted 8 lines
So instead of an
AF_TLSHlistener we could use a named pipe or a netlink socket and a blockingrecv(), as long as there is a reasonable solution to how a connected socket fd is attached to the handshake agent process.I'm flexible about the mechanism for passing handshake parameters. Attaching them to the connected socket seems convenient, but perhaps not aesthetic.
recv()-based version would certainly make me happy.
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 31 lines · Jakub Kicinski
Quoted 13 lines · On Thu, 28 Apr 2022 01:29:10 +0000 Chuck Lever III wrote:
Quoted 7 lines
Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the
getsockopt()now)?The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via
recv()?We considered this kind of design. A reasonable place to start there would be to fabricate new NETLINK messages to do this. I don't see much benefit over what is done now, it's just a different isomer of syntactic sugar, but it could be considered.
Quoted 5 lines
The issue is how the connected socket is materialized in user space. accept(2) is the historical way to instantiate an already connected socket in a process's file table, and seems like a natural fit. When the handshake agent is done with the handshake, it closes the socket. This invokes the
tlsh_release()function which can checkActually - is that strictly necessary? It seems reasonable for NFS to check that it got TLS, since that's what it explicitly asks for per standard. But it may not always be the goal. In large data center networks there can be a policy the user space agent consults to choose what security to install. It may end up doing the auth but not enable crypto if confidentiality is deemed unnecessary.
Quoted 2 lines
Obviously you may not have those requirements but if we can cover them without extra complexity it'd be great.
We can be flexible about how/where handshake success is checked.
However, using a simple close(2) to signal that the handshake has completed does not communicate whether the handshake was indeed successful. We will need a (richer) return/error code from the handshake agent for that use case.
Quoted 4 lines
Quoted 1 line
whether the IV implantation was successful.
I'm used to IV meaning Initialization Vector in context of crypto, what does "IV implementation" stand for?
Implantation, not implementation. The handshake agent implants the initialization vector in the socket before it closes it.
--
Chuck Lever
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Chuck LeverQuoted 19 lines · Chuck Lever
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~
Quoted 9 lines
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.
Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.
A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 45 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~Quoted 9 lines
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.
Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS
close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.
Quoted 7 lines
A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?
JFYI:
For in kernel NFSD hadnshakes we also use the gssproxy unix socket in the kernel, which allows GSSAPI handshakes to be relayed from the kernel to a user space listening daemon.
The infrastructure is technically already available and could be reasonably simply extended to do TLS negotiations as well.
Not saying it is the best interface, but it is already available, and already used by NFS code.
Simo.
Signature · 3 lines
Simo Sorce RHEL Crypto Team Red Hat, Inc
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 58 lines · Simo Sorce
Quoted 45 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~Quoted 9 lines
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.
Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS
close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.Quoted 7 lines
A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?
Quoted 7 lines
JFYI:
For in kernel NFSD hadnshakes we also use the gssproxy unix socket in the kernel, which allows GSSAPI handshakes to be relayed from the kernel to a user space listening daemon.The infrastructure is technically already available and could be reasonably simply extended to do TLS negotiations as well.
To fill in a little about our design thinking:
We chose not to use either gssproxy or gssd for the TLS handshake prototype so that we don't add a dependency on RPC infrastructure for other TLS consumers such as NVMe. Non-RPC consumers view that kind of dependency as quite undesirable.
Also, neither of those existing mechanisms helped us address the issue of passing a connected socket endpoint. listen/poll/accept/close addresses that issue quite directly.
--
Chuck Lever
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
In reply to Boris PismennyQuoted 40 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~Quoted 9 lines
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.
Quoted 8 lines
Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS
close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.
I agree that control messages need to be handled as well. For the moment, the prototype simply breaks the connection when a control message is encountered, and a new session is negotiated. That of course is not the desired long-term solution.
If we believe that control messages are going to be distinct for each transport security layer, then perhaps we cannot make the handshake mechanism generic -- it will have to be specific to each security layer. Just a thought.
Quoted 7 lines
A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?
Certainly piping the handshake messages up to user space instead of handing off a socket is possible. The TLS libraries would need to tolerate this, and GnuTLS (at least) appears OK with performing a handshake on an AF_TLSH socket.
However, I don't see a need to outright avoid passing a connected endpoint to user space. The only difficulty with it seems to be that it hasn't been done before in quite this way.
--
Chuck Lever
Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)
Quoted 68 lines · Chuck Lever III
Quoted 40 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever
In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.
I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.
This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.
Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~Quoted 9 lines
This prototype is net-namespace aware.
The kernel has no mechanism to attest that the listening user space agent is trustworthy.
Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.
Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.
Quoted 8 lines
Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS
close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.I agree that control messages need to be handled as well. For the moment, the prototype simply breaks the connection when a control message is encountered, and a new session is negotiated. That of course is not the desired long-term solution.
If we believe that control messages are going to be distinct for each transport security layer, then perhaps we cannot make the handshake mechanism generic -- it will have to be specific to each security layer. Just a thought.
Quoted 7 lines
A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?
Quoted 5 lines
Certainly piping the handshake messages up to user space instead of handing off a socket is possible. The TLS libraries would need to tolerate this, and GnuTLS (at least) appears OK with performing a handshake on an
AF_TLSHsocket.
Yeah, and I guess that'll be the hard part.
We would need to design an entirely data path for gnutls when going down that path.
The beauty of the fd-passing idea is that gnutls (and openssl for that matter) will 'just work' (tm), without us have to do larger surgery there.
Just for reference, I've raised an issue with gnutls to accept long identifiers in TLS 1.3 (issue #1323), which is required for NVMe-over-TLS support. That one is lingering for over two months now. And that's a relatively simple change; I don't want to imagine how long it'd take to try to push in a larger redesign...
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
RFC PATCH 5/5 net/tls: Add observability for AF_TLSH sockets
In reply to Chuck Lever--- net/tls/af_tlsh.c | 50 +++++++ net/tls/trace.c | 3 net/tls/trace.h | 355 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 402 insertions(+), 6 deletions(-)
net/tls/af_tlsh.c +44 -6
@@ -239,8 +239,10 @@ static bool tlsh_handshake_done(struct sock *sk)tlsh_sock_restore_locked (sk);if (tlsh_crypto_info_initialized (sk)) { +trace_tlsh_handshake_ok (sk);done (data,0 ); }else { +trace_tlsh_handshake_failed (sk);done (data, -EACCES ); } }
@@ -282,6 +284,8 @@ static int tlsh_release(struct socket *sock)if (!sk)return 0 ; +trace_tlsh_release (sock); +switch (sk->sk_family) {case AF_INET :if (!tlsh_handshake_done (sk))
@@ -364,6 +368,7 @@ static int tlsh_bind(struct socket *sock, struct sockaddr *uaddr, int addrlen) } tsk->th_bind_family = uaddr->sa_family; +trace_tlsh_bind (sock);return 0 ; }
@@ -386,6 +391,8 @@ static int tlsh_accept(struct socket *listener, struct socket *newsock, int flaglong timeo;int rc; +trace_tlsh_accept (listener); + rc = -EPERM ;if (!capable (CAP_NET_BIND_SERVICE )) goto out;
@@ -429,6 +436,7 @@ static int tlsh_accept(struct socket *listener, struct socket *newsock, int flag }sock_graft (newsk, newsock); +trace_tlsh_newsock (newsock, newsk); out_release:release_sock (sk);
@@ -450,6 +458,8 @@ static int tlsh_getname(struct socket *sock, struct sockaddr *uaddr, int peer) {struct sock *sk = sock->sk; +trace_tlsh_getname (sock); +switch (sk->sk_family) {case AF_INET :return inet_getname (sock, uaddr, peer);
@@ -486,6 +496,7 @@ static __poll_t tlsh_poll(struct file *file, struct socket *sock, mask |=EPOLLIN |EPOLLRDNORM ;if (sk_is_readable (sk)) mask |=EPOLLIN |EPOLLRDNORM ; +trace_tlsh_poll_listener (sock, mask);return mask; }
@@ -504,6 +515,7 @@ static __poll_t tlsh_poll(struct file *file, struct socket *sock,if (sk->sk_err || !skb_queue_empty_lockless (&sk->sk_error_queue)) mask |=EPOLLERR ; +trace_tlsh_poll (sock, mask);return mask; }
@@ -539,6 +551,7 @@ static int tlsh_listen(struct socket *sock, int backlog) sk->sk_state =TCP_LISTEN ;tlsh_register_listener (sk); +trace_tlsh_listen (sock); rc =0 ; out:
@@ -559,6 +572,8 @@ static int tlsh_shutdown(struct socket *sock, int how) {struct sock *sk = sock->sk; +trace_tlsh_shutdown (sock); +switch (sk->sk_family) {case AF_INET :break ;
@@ -590,6 +605,8 @@ static int tlsh_setsockopt(struct socket *sock, int level, int optname, {struct sock *sk = sock->sk; +trace_tlsh_setsockopt (sock); +switch (sk->sk_family) {case AF_INET :break ;
@@ -756,6 +773,8 @@ static int tlsh_getsockopt(struct socket *sock, int level, int optname,struct sock *sk = sock->sk;int ret; +trace_tlsh_getsockopt (sock); +switch (sk->sk_family) {case AF_INET :break ;
@@ -803,6 +822,9 @@ static int tlsh_getsockopt(struct socket *sock, int level, int optname,static int tlsh_sendmsg (struct socket *sock,struct msghdr *msg,size_t size) {struct sock *sk = sock->sk; +int ret; + +trace_tlsh_sendmsg_start (sock, size);switch (sk->sk_family) {case AF_INET :
@@ -812,12 +834,19 @@ static int tlsh_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)break ;#endif default : -return -EOPNOTSUPP ; + ret = -EOPNOTSUPP ; + goto out; + } + +if (unlikely (inet_send_prepare (sk))) { + ret = -EAGAIN ; + goto out; } + ret = sk->sk_prot->sendmsg (sk, msg, size); -if (unlikely (inet_send_prepare (sk))) -return -EAGAIN ; -return sk->sk_prot->sendmsg (sk, msg, size); +out: +trace_tlsh_sendmsg_result (sock, ret); +return ret; }/**
@@ -835,6 +864,9 @@ static int tlsh_recvmsg(struct socket *sock, struct msghdr *msg, size_t size, int flags) {struct sock *sk = sock->sk; +int ret; + +trace_tlsh_recvmsg_start (sock, size);switch (sk->sk_family) {case AF_INET :
@@ -844,12 +876,17 @@ static int tlsh_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,break ;#endif default : -return -EOPNOTSUPP ; + ret = -EOPNOTSUPP ; + goto out; }if (likely (!(flags &MSG_ERRQUEUE )))sock_rps_record_flow (sk); -return sock_common_recvmsg (sock, msg, size, flags); + ret =sock_common_recvmsg (sock, msg, size, flags); + +out: +trace_tlsh_recvmsg_result (sock, ret); +return ret; }static const struct proto_ops tlsh_proto_ops = {
@@ -920,6 +957,7 @@ int tlsh_pf_create(struct net *net, struct socket *sock, int protocol, int kern) }tlsh_sk (sk)->th_bind_family =AF_UNSPEC ; +trace_tlsh_pf_create (sock);return 0 ; err_sk_put:
net/tls/trace.c +3 -0
@@ -2,6 +2,9 @@/* Copyright (C) 2019 Netronome Systems, Inc. */ #include <linux/module.h> +#include <linux/net.h> +#include <net/sock.h> +#include <net/tls.h> #ifndef __CHECKER__#define CREATE_TRACE_POINTS
net/tls/trace.h +355 -0
367 lines not on this page · raw message
Re: [PATCH RFC 5/5] net/tls: Add observability for AF_TLSH sockets
Quoted 7 lines · Chuck Lever
Signed-off-by Chuck Lever <[email protected]>--- net/tls/af_tlsh.c | 50 +++++++ net/tls/trace.c | 3 net/tls/trace.h | 355 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 402 insertions(+), 6 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer