Skip to contentSkip to conversation

Implement a TLS handshake upcall

RFC PATCH In review Chuck Lever 9 participants 42 messages Apr 18, 2022 – Jul 13, 2022

Subsystems (14)
lore
Share & tools
lore
#0 lore raw

cover Implement a TLS handshake upcall

There are a few upper-layer (storage) protocols that want to have a full TLS implementation available in the Linux kernel.

The Linux kernel currently has an implementation of the TLS record protocol, known as kTLS. However it does not have a complete TLS implementation because it has no implementation of the TLS handshake protocol. In-kernel storage protocols need both to use TLS properly.

In the long run, our preference is to have a TLS handshake implementation in the kernel. However, it appears that would take a long time and there is some desire to avoid adding to the Linux kernel's "attack surface" without good reasons. So in the meantime we've created a prototype handshake implementation that calls out to user space where the actual handshake can be done by an existing library implementation of TLS.

The prototype serves several purposes, including:

- Proof of concept: can a handshake upcall actually be implemented?

- Scaffold to enable prototyping upper-layer protocol support for TLS:
Is there any demand for in-kernel TLS?

- Performance impact of always-on encryption with both software and hardware kTLS

- Understanding what features, if any, an upcall handshake cannot provide

The prototype currently supports client-side PSK and anonymous x.509 ClientHello. We would like some feedback on the approach before proceeding with ServerHello and mutual x.509 authentication.

User agent: https://github.com/oracle/ktls-utils

Who will use this implementation?
--------------------------------

This series implements only the upcall. I plan to post a second series that shows how it can be used to implement the RPC-with-TlS
standard: https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git topic-rpc-with-tls

Dr. Hannes Reinecke has a series that implements NVMe-TLS here:

https://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel.git tls-upcall.v4

We are also working with a few developers in the CIFS community who are interested in SMB-over-QUIC. QUICv1 (RFC 9000) uses the TLSv1.3 handshake protocol, and we hope they can leverage this prototype capability when QUIC comes to the Linux kernel.

---

Chuck Lever (4):
net: Add distinct sk_psock field
net/tls: Add an AF_TLSH address family
net/tls: Add support for PF_TLSH (a TLS handshake listener) net/tls: Add observability for AF_TLSH sockets

Hannes Reinecke (1):
tls: build proto after context has been initialized

 .../networking/tls-in-kernel-handshake.rst    |  103 ++
 include/linux/skmsg.h                         |    2 +-
 include/linux/socket.h                        |    5 +-
 include/net/sock.h                            |    7 +-
 include/net/tls.h                             |   15 +
 include/net/tlsh.h                            |   22 +
 include/uapi/linux/tls.h                      |   16 +
 net/core/skmsg.c                              |    6 +-
 net/core/sock.c                               |    4 +-
 net/socket.c                                  |    1 +
 net/tls/Makefile                              |    2 +-
 net/tls/af_tlsh.c                             | 1078 +++++++++++++++++
 net/tls/tls_main.c                            |   13 +-
 net/tls/trace.c                               |    3 +
 net/tls/trace.h                               |  355 ++++++
 security/selinux/hooks.c                      |    4 +-
 security/selinux/include/classmap.h           |    4 +-
 .../perf/trace/beauty/include/linux/socket.h  |    4 +-
 18 files changed, 1631 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/networking/tls-in-kernel-handshake.rst
 create mode 100644 include/net/tlsh.h
 create mode 100644 net/tls/af_tlsh.c

--
Chuck Lever

#1 lore raw

RFC PATCH 1/5 net: Add distinct sk_psock field

The sk_psock facility populates the sk_user_data field with the address of an extra bit of metadata. User space sockets never populate the sk_user_data field, so this has worked out fine.

However, kernel consumers such as the RPC client and server do populate the sk_user_data field. The sk_psock() function cannot tell that the content of sk_user_data does not point to psock metadata, so it will happily return a pointer to something else, cast to a struct sk_psock.

Thus kernel consumers and psock currently cannot co-exist.

We could educate sk_psock() to return NULL if sk_user_data does not point to a struct sk_psock. However, a more general solution that enables full co-existence psock and other uses of sk_user_data might be more interesting.

Move the struct sk_psock address to its own pointer field so that the contents of the sk_user_data field is preserved.

---
 include/linux/skmsg.h |    2 +-
 include/net/sock.h    |    4 +++-
 net/core/skmsg.c      |    6 +++---
 3 files changed, 7 insertions(+), 5 deletions(-)
include/linux/skmsg.h +1 -1
index c5a2d6f50f25..5ef3a07c5b6c 100644
@@ -277,7 +277,7 @@ static inline void sk_msg_sg_copy_clear(struct sk_msg *msg, u32 start)
 
 static inline struct sk_psock *sk_psock(const struct sock *sk)
 {
-	return rcu_dereference_sk_user_data(sk);
+	return rcu_dereference(sk->sk_psock);
 }
 
 static inline void sk_psock_set_state(struct sk_psock *psock,
include/net/sock.h +3 -1
index c4b91fc19b9c..d2a513169527 100644
@@ -327,7 +327,8 @@ struct sk_filter;
   *	@sk_tskey: counter to disambiguate concurrent tstamp requests
   *	@sk_zckey: counter to order MSG_ZEROCOPY notifications
   *	@sk_socket: Identd and reporting IO signals
-  *	@sk_user_data: RPC layer private data
+  *	@sk_user_data: Upper layer private data
+  *	@sk_psock: socket policy data (bpf)
   *	@sk_frag: cached page frag
   *	@sk_peek_off: current peek_offset value
   *	@sk_send_head: front of stuff to transmit
@@ -519,6 +520,7 @@ struct sock {
 
 	struct socket		*sk_socket;
 	void			*sk_user_data;
+	struct sk_psock	__rcu	*sk_psock;
 #ifdef CONFIG_SECURITY
 	void			*sk_security;
 #endif
net/core/skmsg.c +3 -3
index cc381165ea08..2b3d01d92790 100644
@@ -695,7 +695,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node)
 
 	write_lock_bh(&sk->sk_callback_lock);
 
-	if (sk->sk_user_data) {
+	if (sk->sk_psock) {
 		psock = ERR_PTR(-EBUSY);
 		goto out;
 	}
@@ -726,7 +726,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node)
 	sk_psock_set_state(psock, SK_PSOCK_TX_ENABLED);
 	refcount_set(&psock->refcnt, 1);
 
-	rcu_assign_sk_user_data_nocopy(sk, psock);
+	rcu_assign_pointer(sk->sk_psock, psock);
 	sock_hold(sk);
 
 out:
@@ -825,7 +825,7 @@ void sk_psock_drop(struct sock *sk, struct sk_psock *psock)
 {
 	write_lock_bh(&sk->sk_callback_lock);
 	sk_psock_restore_proto(sk, psock);
-	rcu_assign_sk_user_data(sk, NULL);
+	rcu_assign_pointer(sk->sk_psock, NULL);
 	if (psock->progs.stream_parser)
 		sk_psock_stop_strp(sk, psock);
 	else if (psock->progs.stream_verdict || psock->progs.skb_verdict)
Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH RFC 1/5] net: Add distinct sk_psock field

Quoted 20 lines · Chuck Lever

The sk_psock facility populates the sk_user_data field with the address of an extra bit of metadata. User space sockets never populate the sk_user_data field, so this has worked out fine.

However, kernel consumers such as the RPC client and server do populate the sk_user_data field. The sk_psock() function cannot tell that the content of sk_user_data does not point to psock metadata, so it will happily return a pointer to something else, cast to a struct sk_psock.

Thus kernel consumers and psock currently cannot co-exist.

We could educate sk_psock() to return NULL if sk_user_data does not point to a struct sk_psock. However, a more general solution that enables full co-existence psock and other uses of sk_user_data might be more interesting.

Move the struct sk_psock address to its own pointer field so that the contents of the sk_user_data field is preserved.

Quoted 7 lines
---
  include/linux/skmsg.h |    2 +-
  include/net/sock.h    |    4 +++-
  net/core/skmsg.c      |    6 +++---
  3 files changed, 7 insertions(+), 5 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
#3 lore raw

RFC PATCH 1/5 net: Add distinct sk_psock field

Quoted 28 lines · Chuck Lever
Quoted 20 lines

The sk_psock facility populates the sk_user_data field with the address of an extra bit of metadata. User space sockets never populate the sk_user_data field, so this has worked out fine.

However, kernel consumers such as the RPC client and server do populate the sk_user_data field. The sk_psock() function cannot tell that the content of sk_user_data does not point to psock metadata, so it will happily return a pointer to something else, cast to a struct sk_psock.

Thus kernel consumers and psock currently cannot co-exist.

We could educate sk_psock() to return NULL if sk_user_data does not point to a struct sk_psock. However, a more general solution that enables full co-existence psock and other uses of sk_user_data might be more interesting.

Move the struct sk_psock address to its own pointer field so that the contents of the sk_user_data field is preserved.

Quoted 7 lines
---
  include/linux/skmsg.h |    2 +-
  include/net/sock.h    |    4 +++-
  net/core/skmsg.c      |    6 +++---
  3 files changed, 7 insertions(+), 5 deletions(-)
Quoted 5 lines

Cheers,

Hannes

In Patchwork website, this patch fails the checks on netdev/cc_maintainers.

So maybe you need CC folks pointed out by scripts/get_maintainer.pl script, which is suggested by Jakub Kicinski <[email protected]>.

What's more, Syskaller reports
refcount bug in sk_psock_get (2).

In this bug, the problem is that smc and psock, both use sk_user_data field to save their private data. So they will treat field in their own way.

Quoted 5 lines

in smc_switch_to_fallback(), and set smc->clcsock->sk_user_data to origin smc in smc_fback_replace_callbacks().

Later, sk_psock_get() will treat the smc->clcsock->sk_user_data as sk_psock type, which triggers the refcnt warning.

I have tested this patch and the reproducer did not trigger any issue. For more details, you can check the email
[PATCH] smc: fix refcount bug in sk_psock_get (2)
_______________________________________________
Linux-kernel-mentees mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees

#4 lore raw

RFC PATCH 2/5 tls: build proto after context has been initialized

In reply to Chuck Lever

From: Hannes Reinecke <[email protected]>

We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.

---
 net/tls/tls_main.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
net/tls/tls_main.c +1 -2
index 7b2b0e7ffee4..7eca4d9a83c4 100644
@@ -836,8 +836,6 @@ static int tls_init(struct sock *sk)
 	struct tls_context *ctx;
 	int rc = 0;
 
-	tls_build_proto(sk);
-
 #ifdef CONFIG_TLS_TOE
 	if (tls_toe_bypass(sk))
 		return 0;
@@ -862,6 +860,7 @@ static int tls_init(struct sock *sk)
 
 	ctx->tx_conf = TLS_BASE;
 	ctx->rx_conf = TLS_BASE;
+	tls_build_proto(sk);
 	update_sk_prot(sk, ctx);
 out:
 	write_unlock_bh(&sk->sk_callback_lock);
maintainer
#5 lore raw

Re: [PATCH RFC 2/5] tls: build proto after context has been initialized

Quoted 5 lines · On Mon, 18 Apr 2022 12:49:36 -0400 Chuck Lever wrote:

From: Hannes Reinecke <[email protected]>

We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.

Can you say more about the crash you see? protos are not used until at least one socket calls update_sk_prot().

#6 lore raw

Re: [PATCH RFC 2/5] tls: build proto after context has been initialized

Quoted 13 lines · Jakub Kicinski
Quoted 5 lines · On Mon, 18 Apr 2022 12:49:36 -0400 Chuck Lever wrote:

From: Hannes Reinecke <[email protected]>

We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.

Can you say more about the crash you see? protos are not used until at least one socket calls update_sk_prot().

Hannes reported the crash and supplied the fix. I didn't have a problem with this patch series without this patch. Hannes?

--
Chuck Lever

#7 lore raw

Re: [PATCH RFC 2/5] tls: build proto after context has been initialized

In reply to Chuck Lever
Quoted 36 lines · Chuck Lever

From: Hannes Reinecke <[email protected]>

We have to build the proto ops only after the context has been initialized, as otherwise we might crash when I/O is ongoing during initialisation.

---
net/tls/tls_main.c | 3 +--

1 file changed, 1 insertion(+), 2 deletions(-)
net/tls/tls_main.c +1 -2
index 7b2b0e7ffee4..7eca4d9a83c4 100644
@@ -836,8 +836,6 @@ static int tls_init(struct sock *sk)
 	struct tls_context *ctx;
 	int rc = 0;
 
-	tls_build_proto(sk);
-
 #ifdef CONFIG_TLS_TOE
 	if (tls_toe_bypass(sk))
 		return 0;
@@ -862,6 +860,7 @@ static int tls_init(struct sock *sk)
 
 	ctx->tx_conf = TLS_BASE;
 	ctx->rx_conf = TLS_BASE;
+	tls_build_proto(sk);
 	update_sk_prot(sk, ctx);

out:

Quoted 3 lines
	write_unlock_bh(&sk->sk_callback_lock);

While we're working on a better upcall mechanism, I'm dropping this one from the series. tls_build_proto() can grab a mutex, so tls_init() can't call it while holding sk_callback_lock.

--
Chuck Lever

#8 lore raw

RFC PATCH 3/5 net/tls: Add an AF_TLSH address family

In reply to Chuck Lever

Add definitions for an AF_TLSH address family. The next patch explains its purpose and operation.

---
 include/linux/socket.h                         |    4 +++-
 net/core/sock.c                                |    2 +-
 net/socket.c                                   |    1 +
 security/selinux/hooks.c                       |    4 +++-
 security/selinux/include/classmap.h            |    4 +++-
 tools/perf/trace/beauty/include/linux/socket.h |    4 +++-
 6 files changed, 14 insertions(+), 5 deletions(-)
include/linux/socket.h +3 -1
index 6f85f5d957ef..fc28c68e6b5f 100644
@@ -226,8 +226,9 @@ struct ucred {
 #define AF_MCTP		45	/* Management component
 				 * transport protocol
 				 */
+#define AF_TLSH		46	/* TLS handshake request */
 
-#define AF_MAX		46	/* For now.. */
+#define AF_MAX		47	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -278,6 +279,7 @@ struct ucred {
 #define PF_SMC		AF_SMC
 #define PF_XDP		AF_XDP
 #define PF_MCTP		AF_MCTP
+#define PF_TLSH		AF_TLSH
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */
net/core/sock.c +1 -1
index 1180a0cb0110..81bc14b67468 100644
@@ -224,7 +224,7 @@ static struct lock_class_key af_family_kern_slock_keys[AF_MAX];
   x "AF_IEEE802154",	x "AF_CAIF"	,	x "AF_ALG"      , \
   x "AF_NFC"   ,	x "AF_VSOCK"    ,	x "AF_KCM"      , \
   x "AF_QIPCRTR",	x "AF_SMC"	,	x "AF_XDP"	, \
-  x "AF_MCTP"  , \
+  x "AF_MCTP"  ,	x "AF_TLSH"	, \
   x "AF_MAX"
 
 static const char *const af_family_key_strings[AF_MAX+1] = {
net/socket.c +1 -0
index 6887840682bb..fc43ebd5ad66 100644
@@ -214,6 +214,7 @@ static const char * const pf_family_names[] = {
 	[PF_SMC]	= "PF_SMC",
 	[PF_XDP]	= "PF_XDP",
 	[PF_MCTP]	= "PF_MCTP",
+	[PF_TLSH]	= "PF_TLSH",
 };
 
 /*
security/selinux/hooks.c +3 -1
index e9e959343de9..734e284eb06a 100644
@@ -1259,7 +1259,9 @@ static inline u16 socket_type_to_security_class(int family, int type, int protoc
 			return SECCLASS_XDP_SOCKET;
 		case PF_MCTP:
 			return SECCLASS_MCTP_SOCKET;
-#if PF_MAX > 46
+		case PF_TLSH:
+			return SECCLASS_TLSH_SOCKET;
+#if PF_MAX > 47
 #error New address family defined, please update this function.
 #endif
 		}
security/selinux/include/classmap.h +3 -1
index 35aac62a662e..673b51f8281d 100644
@@ -237,6 +237,8 @@ struct security_class_mapping secclass_map[] = {
 	  { COMMON_SOCK_PERMS, NULL } },
 	{ "smc_socket",
 	  { COMMON_SOCK_PERMS, NULL } },
+	{ "tlsh_socket",
+	  { COMMON_SOCK_PERMS, NULL } },
 	{ "infiniband_pkey",
 	  { "access", NULL } },
 	{ "infiniband_endport",
@@ -257,6 +259,6 @@ struct security_class_mapping secclass_map[] = {
 	{ NULL }
   };
 
-#if PF_MAX > 46
+#if PF_MAX > 47
 #error New address family defined, please update secclass_map.
 #endif
tools/perf/trace/beauty/include/linux/socket.h +3 -1
index 6f85f5d957ef..fc28c68e6b5f 100644
@@ -226,8 +226,9 @@ struct ucred {
 #define AF_MCTP		45	/* Management component
 				 * transport protocol
 				 */
+#define AF_TLSH		46	/* TLS handshake request */
 
-#define AF_MAX		46	/* For now.. */
+#define AF_MAX		47	/* For now.. */
 
 /* Protocol families, same as address families. */
 #define PF_UNSPEC	AF_UNSPEC
@@ -278,6 +279,7 @@ struct ucred {
 #define PF_SMC		AF_SMC
 #define PF_XDP		AF_XDP
 #define PF_MCTP		AF_MCTP
+#define PF_TLSH		AF_TLSH
 #define PF_MAX		AF_MAX
 
 /* Maximum queue length specifiable by listen.  */
Reviewed-by on v1 3/5
#9 lore raw

Re: [PATCH RFC 3/5] net/tls: Add an AF_TLSH address family

Quoted 4 lines · Chuck Lever

Add definitions for an AF_TLSH address family. The next patch explains its purpose and operation.

Quoted 9 lines
---
  include/linux/socket.h                         |    4 +++-
  net/core/sock.c                                |    2 +-
  net/socket.c                                   |    1 +
  security/selinux/hooks.c                       |    4 +++-
  security/selinux/include/classmap.h            |    4 +++-
  tools/perf/trace/beauty/include/linux/socket.h |    4 +++-
  6 files changed, 14 insertions(+), 5 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
#10 lore raw

RFC PATCH 4/5 net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

---
 .../networking/tls-in-kernel-handshake.rst         |  103 ++
 include/linux/socket.h                             |    1 
 include/net/sock.h                                 |    3 
 include/net/tls.h                                  |   15 
 include/net/tlsh.h                                 |   22 
 include/uapi/linux/tls.h                           |   16 
 net/core/sock.c                                    |    2 
 net/tls/Makefile                                   |    2 
 net/tls/af_tlsh.c                                  | 1040 ++++++++++++++++++++
 net/tls/tls_main.c                                 |   10 
 10 files changed, 1213 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/networking/tls-in-kernel-handshake.rst
 create mode 100644 include/net/tlsh.h
 create mode 100644 net/tls/af_tlsh.c
Documentation/networking/tls-in-kernel-handshake.rst new file +103 -0
new file mode 100644
index 000000000000..73ed6928f4b2
@@ -0,0 +1,103 @@
+.. _kernel_tls:
+
+=======================
+In-Kernel TLS Handshake
+=======================
+
+Overview
+========
+
+Transport Layer Security (TLS) is a Upper Layer Protocol (ULP) that runs over
+TCP. TLS provides end-to-end data integrity and confidentiality.
+
+kTLS handles the TLS record subprotocol, but does not handle the TLS handshake
+subprotocol, used to establish a TLS session. In user space, a TLS library
+performs the handshake on a socket which is converted to kTLS operation. In
+the kernel it is much the same. The TLS handshake is done in user space by a
+library TLS implementation.
+
+
+User agent
+==========
+
+With the current implementation, a user agent is started in each network
+namespace where a kernel consumer might require a TLS handshake. This agent
+listens on an AF_TLSH socket for requests from the kernel to perform a
+handshake on an open and connected TCP socket.
+
+The open socket is passed to user space via accept(), which creates a file
+descriptor. If the handshake completes successfully, the user agent promotes
+the socket to use the TLS ULP and sets the session information using the
+SOL_TLS socket options. The user agent returns the socket to the kernel by
+closing the accepted file descriptor.
+
+
+Kernel Handshake API
+====================
+
+A kernel consumer initiates a client-side TLS handshake on an open
+socket by invoking one of the tls_client_hello() functions. For
+example:
+
+.. code-block:: c
+
+  ret = tls_client_hello_x509(sock, done_func, cookie, priorities,
+                              peerid, cert);
+
+The function returns zero when the handshake request is under way. A
+zero return guarantees the callback function @done_func will be invoked
+for this socket.
+
+The function returns a negative errno if the handshake could not be
+started. A negative errno guarantees the callback function @done_func
+will not be invoked on this socket.
+
+The @sock argument is an open and connected IPPROTO_TCP socket. The
+caller must hold a reference on the socket to prevent it from being
+destroyed while the handshake is in progress.
+
+@done_func and @cookie are a callback function that is invoked when the
+handshake has completed (either successfully or not). The success status
+of the handshake is returned via the @status parameter of the callback
+function. A good practice is to close and destroy the socket immediately
+if the handshake has failed.
+
+@priorities is a GnuTLS priorities string that controls the handshake.
+The special value TLSH_DEFAULT_PRIORITIES causes the handshake to
+operate using user space configured default TLS priorities. However,
+the caller can use the string to (for example) adjust the handshake to
+use a restricted set of ciphers (say, if the kernel is in FIPS mode or
+the kernel consumer wants to mandate only a limited set of ciphers).
+
+@peerid is the serial number of a key on the XXXYYYZZZ keyring that
+contains a private key.
+
+@cert is the serial number of a key on the XXXYYYYZZZ keyring that
+contains a {PEM,DER} format x.509 certificate that the user agent
+presents to the server as the local peer's identity.
+
+To initiate a client-side TLS handshake with a pre-shared key, use:
+
+.. code-block:: c
+
+  ret = tls_client_hello_psk(sock, done_func, cookie, priorities,
+                             peerid);
+
+@peerid is the serial number of a key on the XXXYYYZZZ keyring that
+contains the pre-shared key.
+
+The other parameters are as above.
+
+
+Other considerations
+--------------------
+
+While the handshake is under way, the kernel consumer must alter the
+socket's sk_data_ready callback function to ignore incoming data.
+Once the callback function has been invoked, normal receive operation
+can be resumed.
+
+See tls.rst for details on how a kTLS consumer recognizes incoming
+(decrypted) application data, alerts, and handshake packets once the
+socket has been promoted to use the TLS ULP.
+
include/linux/socket.h +1 -0
index fc28c68e6b5f..69acb5668d34 100644
@@ -369,6 +369,7 @@ struct ucred {
 #define SOL_MPTCP	284
 #define SOL_MCTP	285
 #define SOL_SMC		286
+#define SOL_TLSH	287
 
 /* IPX options */
 #define IPX_TYPE	1
include/net/sock.h +3 -0
index d2a513169527..d5a5d5fd6682 100644
@@ -353,6 +353,7 @@ struct sk_filter;
   *	@sk_txtime_report_errors: set report errors mode for SO_TXTIME
   *	@sk_txtime_unused: unused txtime flags
   *	@ns_tracker: tracker for netns reference
+  *	@sk_tlsh_priv: private data for TLS handshake upcall
   */
 struct sock {
 	/*
@@ -544,6 +545,8 @@ struct sock {
 #endif
 	struct rcu_head		sk_rcu;
 	netns_tracker		ns_tracker;
+
+	void			*sk_tlsh_priv;
 };
 
 enum sk_pacing {
include/net/tls.h +15 -0
index b6968a5b5538..6b1bf46daa34 100644
@@ -51,6 +51,18 @@
 #include <uapi/linux/tls.h>
 
 
+struct tlsh_sock {
+	/* struct sock must remain the first field */
+	struct sock	th_sk;
+
+	int		th_bind_family;
+};
+
+static inline struct tlsh_sock *tlsh_sk(struct sock *sk)
+{
+	return (struct tlsh_sock *)sk;
+}
+
 /* Maximum data size carried in a TLS record */
 #define TLS_MAX_PAYLOAD_SIZE		((size_t)1 << 14)
 
@@ -356,6 +368,9 @@ struct tls_context *tls_ctx_create(struct sock *sk);
 void tls_ctx_free(struct sock *sk, struct tls_context *ctx);
 void update_sk_prot(struct sock *sk, struct tls_context *ctx);
 
+int tlsh_pf_create(struct net *net, struct socket *sock, int protocol,
+		   int kern);
+
 int wait_on_pending_writer(struct sock *sk, long *timeo);
 int tls_sk_query(struct sock *sk, int optname, char __user *optval,
 		int __user *optlen);
include/net/tlsh.h new file +22 -0
new file mode 100644
index 000000000000..8725fd83df60
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * PF_TLSH protocol family socket handler.
+ *
+ * Author: Chuck Lever <[email protected]>
+ *
+ * Copyright (c) 2021, Oracle and/or its affiliates.
+ */
+
+#ifndef _TLS_HANDSHAKE_H
+#define _TLS_HANDSHAKE_H
+
+extern int tls_client_hello_psk(struct socket *sock,
+				void (*done)(void *data, int status),
+				void *data, const char *priorities,
+				key_serial_t peerid);
+extern int tls_client_hello_x509(struct socket *sock,
+				 void (*done)(void *data, int status),
+				 void *data, const char *priorities,
+				 key_serial_t peerid, key_serial_t cert);
+
+#endif /* _TLS_HANDSHAKE_H */
include/uapi/linux/tls.h +16 -0
index 5f38be0ec0f3..d0ffbb6ea0e4 100644
@@ -40,6 +40,22 @@
 #define TLS_TX			1	/* Set transmit parameters */
 #define TLS_RX			2	/* Set receive parameters */
 
+/* TLSH socket options */
+#define TLSH_PRIORITIES		1	/* Retrieve TLS priorities string */
+#define TLSH_PEERID		2	/* Retrieve peer identity */
+#define TLSH_HANDSHAKE_TYPE	3	/* Retrieve handshake type */
+#define TLSH_X509_CERTIFICATE	4	/* Retrieve x.509 certificate */
+
+#define TLSH_DEFAULT_PRIORITIES		(NULL)
+#define TLSH_NO_PEERID			(0)
+#define TLSH_NO_CERT			(0)
+
+/* TLSH handshake types */
+enum tlsh_hs_type {
+	TLSH_TYPE_CLIENTHELLO_X509,
+	TLSH_TYPE_CLIENTHELLO_PSK,
+};
+
 /* Supported versions */
 #define TLS_VERSION_MINOR(ver)	((ver) & 0xFF)
 #define TLS_VERSION_MAJOR(ver)	(((ver) >> 8) & 0xFF)
net/core/sock.c +2 -0
index 81bc14b67468..d9f700e5ea1a 100644
@@ -3295,6 +3295,8 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 	sk->sk_incoming_cpu = -1;
 	sk->sk_txrehash = SOCK_TXREHASH_DEFAULT;
 
+	sk->sk_tlsh_priv = NULL;
+
 	sk_rx_queue_clear(sk);
 	/*
 	 * Before updating sk_refcnt, we must commit prior changes to memory
net/tls/Makefile +1 -1
index f1ffbfe8968d..d159a03b94f3 100644
@@ -7,7 +7,7 @@ CFLAGS_trace.o := -I$(src)
 
 obj-$(CONFIG_TLS) += tls.o
 
-tls-y := tls_main.o tls_sw.o tls_proc.o trace.o
+tls-y := af_tlsh.o tls_main.o tls_sw.o tls_proc.o trace.o
 
 tls-$(CONFIG_TLS_TOE) += tls_toe.o
 tls-$(CONFIG_TLS_DEVICE) += tls_device.o tls_device_fallback.o
net/tls/af_tlsh.c new file +1040 -0

1,040 lines not on this page · raw message

net/tls/tls_main.c +10 -0
index 7eca4d9a83c4..c5e0a7b3aa2e 100644
@@ -49,6 +49,7 @@ MODULE_AUTHOR("Mellanox Technologies");
 MODULE_DESCRIPTION("Transport Layer Security Support");
 MODULE_LICENSE("Dual BSD/GPL");
 MODULE_ALIAS_TCP_ULP("tls");
+MODULE_ALIAS_NETPROTO(PF_TLSH);
 
 enum {
 	TLSV4,
@@ -982,6 +983,12 @@ static struct tcp_ulp_ops tcp_tls_ulp_ops __read_mostly = {
 	.get_info_size		= tls_get_info_size,
 };
 
+static const struct net_proto_family tlsh_pf_ops = {
+	.family = PF_TLSH,
+	.create = tlsh_pf_create,
+	.owner	= THIS_MODULE,
+};
+
 static int __init tls_register(void)
 {
 	int err;
@@ -993,11 +1000,14 @@ static int __init tls_register(void)
 	tls_device_init();
 	tcp_register_ulp(&tcp_tls_ulp_ops);
 
+	sock_register(&tlsh_pf_ops);
+
 	return 0;
 }
 
 static void __exit tls_unregister(void)
 {
+	sock_unregister(PF_TLSH);
 	tcp_unregister_ulp(&tcp_tls_ulp_ops);
 	tls_device_cleanup();
 	unregister_pernet_subsys(&tls_proc_ops);
Reviewed-by on v1 4/5
#11 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 41 lines · Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

---
  .../networking/tls-in-kernel-handshake.rst         |  103 ++
  include/linux/socket.h                             |    1
  include/net/sock.h                                 |    3
  include/net/tls.h                                  |   15
  include/net/tlsh.h                                 |   22
  include/uapi/linux/tls.h                           |   16
  net/core/sock.c                                    |    2
  net/tls/Makefile                                   |    2
  net/tls/af_tlsh.c                                  | 1040 ++++++++++++++++++++
  net/tls/tls_main.c                                 |   10
  10 files changed, 1213 insertions(+), 1 deletion(-)
Quoted 4 lines

create mode 100644 Documentation/networking/tls-in-kernel-handshake.rst create mode 100644 include/net/tlsh.h
create mode 100644 net/tls/af_tlsh.c

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
maintainer
#12 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Chuck Lever
Quoted 19 lines · On Mon, 18 Apr 2022 12:49:50 -0400 Chuck Lever wrote:

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

Quoted 8 lines

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.

#13 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 25 lines · Jakub Kicinski
Quoted 24 lines · On Mon, 18 Apr 2022 12:49:50 -0400 Chuck Lever wrote:

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Quoted 8 lines
Quoted 3 lines

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.

But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible. Not to mention having to rework the nvme driver to accept sockets from userspace instead of creating them internally.

With this approach we can keep existing infrastructure, and can get a common implementation for either transport.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
maintainer
#14 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 11 lines
Quoted 9 lines
Quoted 4 lines

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel.  This is how NBD + TLS works.  Scales better and requires much less kernel code.

But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.

And we will still need the upcall anyways when we reconnect (re-establish the socket)

maintainer
#15 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 11 lines · On Tue, 26 Apr 2022 17:29:03 +0300 Sagi Grimberg wrote:
Quoted 8 lines
Quoted 4 lines

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel.  This is how NBD + TLS works.  Scales better and requires much less kernel code.

But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.

And we will still need the upcall anyways when we reconnect (re-establish the socket)

That totally flew over my head, I have zero familiarity with in-kernel storage network users :S

In all honesty the tls code in the kernel is a bit of a dumping ground. People come, dump a bunch of code and disappear. Nobody seems to care that the result is still (years in) not ready for production use :/ Until a month ago it'd break connections even under moderate memory pressure. This set does not even have selftests.

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

#16 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 26 Apr 2022 17:29:03 +0300 Sagi Grimberg wrote:
Quoted 8 lines
Quoted 4 lines

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel.  This is how NBD + TLS works.  Scales better and requires much less kernel code.

But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.

Quoted 6 lines
Quoted 2 lines

And we will still need the upcall anyways when we reconnect (re-establish the socket)

That totally flew over my head, I have zero familiarity with in-kernel storage network users :S

Count yourself lucky.

Quoted 6 lines

In all honesty the tls code in the kernel is a bit of a dumping ground. People come, dump a bunch of code and disappear. Nobody seems to care that the result is still (years in) not ready for production use :/ Until a month ago it'd break connections even under moderate memory pressure. This set does not even have selftests.

Well, I'd been surprised that it worked, too.
And even more so that Boris Piskenny @ Nvidia is actively working on it. (Thanks, Sagi!)

Quoted 3 lines

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

Which is precisely what we hope to discuss at LSF.
(Yes, I know, probably not the best venue to discuss network stuff ...)

Each approach has its drawbacks:

- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established.
(And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here)
(Especially call_usermodehelper() is out of the game)
- Having ULP helpers (as with this design) mitigates that problem somewhat in the sense that you can mlock() that daemon and having it polling on an intermediate socket; that solves the notification problem. But you have to have ULP special sauce here to make it work.
- Moving everything in kernel is ... possible. But then you have yet another security-relevant piece of code in the kernel which needs to be audited, CVEd etc. Not to mention the usual policy discussion whether it really belongs into the kernel.

So I don't really see any obvious way to go; best we can do is to pick the least ugly :-(

Cheers,

Hannes

maintainer
#17 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 6 lines · On Tue, 26 Apr 2022 17:58:39 +0200 Hannes Reinecke wrote:
Quoted 3 lines

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

Which is precisely what we hope to discuss at LSF.
(Yes, I know, probably not the best venue to discuss network stuff ...)

Indeed.

Quoted 8 lines

Each approach has its drawbacks:

- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established. (And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here) (Especially call_usermodehelper() is out of the game)

Indeed, we may need _some_ form of a notification mechanism and that's okay. Can be a (more generic) socket, can be something based on existing network storage APIs (IDK what you have there).

My thinking was that establishing the session in user space would be easiest. We wouldn't need all the special getsockopt()s which AFAIU work around part of the handshake being done in the kernel, and which, I hope we can agree, are not beautiful.

Quoted 4 lines

- Having ULP helpers (as with this design) mitigates that problem somewhat in the sense that you can mlock() that daemon and having it polling on an intermediate socket; that solves the notification problem. But you have to have ULP special sauce here to make it work.

TBH I don't see how this is much different to option 1 in terms of constraints & requirements on the user space agent. We can implement option 1 over a socket-like interface, too, and that'll carry notifications all the same.

Quoted 4 lines

- Moving everything in kernel is ... possible. But then you have yet another security-relevant piece of code in the kernel which needs to be audited, CVEd etc. Not to mention the usual policy discussion whether it really belongs into the kernel.

Yeah, if that gets posted it'd be great if it includes removing me from the TLS maintainers 'cause I want to sleep at night ;)

Quoted 2 lines

So I don't really see any obvious way to go; best we can do is to pick the least ugly :-(

True, I'm sure we can find some middle ground between 1 and 2. Preferably implemented in a way where the mechanism is separated from the fact it's carrying TLS handshake requests, so that it can carry something else tomorrow.

#18 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 12 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 26 Apr 2022 17:58:39 +0200 Hannes Reinecke wrote:

- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established. (And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here) (Especially call_usermodehelper() is out of the game)

Quoted 8 lines

Indeed, we may need _some_ form of a notification mechanism and that's okay. Can be a (more generic) socket, can be something based on existing network storage APIs (IDK what you have there).

My thinking was that establishing the session in user space would be easiest. We wouldn't need all the special getsockopt()s which AFAIU work around part of the handshake being done in the kernel, and which, I hope we can agree, are not beautiful.

In the prototype, the new socket options on AF_TLSH sockets include:

#define TLSH_PRIORITIES        1       /* Retrieve TLS priorities string */
#define TLSH_PEERID            2       /* Retrieve peer identity */
#define TLSH_HANDSHAKE_TYPE    3       /* Retrieve handshake type */
#define TLSH_X509_CERTIFICATE  4       /* Retrieve x.509 certificate */

PRIORITIES is the TLS priorities string that the GnuTLS library uses to parametrize the handshake (which TLS versions, ciphers, and so on).

PEERID is a keyring serial number for the key that contains the a Pre-Shared Key (for PSK handshakes) or the private key (for x.509 handshakes).

HANDSHAKE_TYPE is an integer that represents the type of handshake being requested: ClientHello, ServerHello, Rekey, and so on. This option enables the repertoire of handshake types to be expanded.

X509_CERTIFICATE is a keyring serial number for the key that contains an x.509 certificate.

When each handshake is complete, the handshake agent instantiates the IV into the passed-in socket using existing kTLS socket options before it returns the endpoint to the kernel.

There is nothing in these options that indicates to the handshake agent what upper layer protocol is going to be used inside the TLS session.

----

The new AF_TLSH socket options are not there because the handshake is split between the kernel and user space. They are there because the initial requester is (eg, in the case of NFS) mount.nfs, another user space program. mount.nfs has to package up an x.509 cert or pre-shared key and place it on a keyring to make it available to the handshake agent.

The basic issue is that the administrative interfaces that parametrize the handshakes are quite distant from the in-kernel consumers that make handshake requests.

----

Further, in order to support server side TLS handshakes in the kernel, we really do have to pass a kernel-created socket up to user space. NFSD (and maybe the NVMe target) use in-kernel listeners to accept incoming connections. Each new endpoint is created in the kernel.

So if you truly seek generality in this facility, the user space componentry must work with passed-in sockets rather than creating them in user space.

--
Chuck Lever

#19 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Jakub Kicinski
Quoted 19 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 26 Apr 2022 17:58:39 +0200 Hannes Reinecke wrote:
Quoted 3 lines

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

Which is precisely what we hope to discuss at LSF.
(Yes, I know, probably not the best venue to discuss network stuff ...)

Indeed.

Quoted 8 lines

Each approach has its drawbacks:

- Establishing sockets from userspace will cause issues during reconnection, as then someone (aka the kernel) will have to inform userspace that a new connection will need to be established. (And that has to happen while the root filesystem is potentially inaccessible, so you can't just call arbitrary commands here) (Especially call_usermodehelper() is out of the game)

Quoted 4 lines

Indeed, we may need _some_ form of a notification mechanism and that's okay. Can be a (more generic) socket, can be something based on existing network storage APIs (IDK what you have there).

Which is one of the issues; we don't have any.
Network storage is accessed from userspace via read()/write(), and for control everyone rolls its own thing. So speaking of a 'network storage API' is quite optimistic.

Quoted 5 lines

My thinking was that establishing the session in user space would be easiest. We wouldn't need all the special getsockopt()s which AFAIU work around part of the handshake being done in the kernel, and which, I hope we can agree, are not beautiful.

Well; that is open to debate.
During open-iscsi development (which followed your model of having a control- and dataplane split between userspace and kernel) we found that the model is great for keeping the in-kernel code simple. But we also found that it's not so great once you come down to the gritty details; you have to duplicate quite some protocol handling in userspace, and you have to worry about session re-establishment. Up to the point where we start wondering if moving things into userspace was a good idea at all...

As for this particular interface: the problem we're facing is that TLS has to be started in-band. For some reason NVMexpress decided not to follow the traditional method of establishing the socket, start TLS, and then start protocol processing, but it rather has specified a hybrid model: establishing the socket, doing authentication, _then_ start TLS, and only then continue with protocol processing on the TLS socket.

Moving _that_ into userspace would require us to move the most of the protocol logic into userspace, too. Which really is something we want to avoid, as this would be quite a code duplication. Not to mention the maintenance burden, as issues would need to be fixed in two locations.
(Talk to me about iscsiuio.)

And that's before we even get to things like TLS session tickets, which opens yet another can of worms.

Quoted 18 lines
Quoted 4 lines

- Having ULP helpers (as with this design) mitigates that problem somewhat in the sense that you can mlock() that daemon and having it polling on an intermediate socket; that solves the notification problem. But you have to have ULP special sauce here to make it work.

TBH I don't see how this is much different to option 1 in terms of constraints & requirements on the user space agent. We can implement option 1 over a socket-like interface, too, and that'll carry notifications all the same.

Quoted 4 lines

- Moving everything in kernel is ... possible. But then you have yet another security-relevant piece of code in the kernel which needs to be audited, CVEd etc. Not to mention the usual policy discussion whether it really belongs into the kernel.

Yeah, if that gets posted it'd be great if it includes removing me from the TLS maintainers 'cause I want to sleep at night ;)

Quoted 8 lines
Quoted 2 lines

So I don't really see any obvious way to go; best we can do is to pick the least ugly :-(

True, I'm sure we can find some middle ground between 1 and 2. Preferably implemented in a way where the mechanism is separated from the fact it's carrying TLS handshake requests, so that it can carry something else tomorrow.

Which was actually our goal.
The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa. While there is fd passing between processes via AF_UNIX, there is no such mechanism between kernel and userspace.

So accept() was an easy way out as the implementation was quite simple.
And it was moved into net/tls as this was the primary use-case.
But it's by no means meant to be exclusive for TLS; we could expand it to other things once there is a need. But we really didn't want to over-engineer things here.

However, if you have another idea, by all means, do tell.
It's just that I don't think creating sockets in userspace is a great fit for us.

Cheers,

Hannes

maintainer
#20 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 4 lines · On Thu, 28 Apr 2022 09:26:41 +0200 Hannes Reinecke wrote:

The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa.
While there is fd passing between processes via AF_UNIX, there is no such mechanism between kernel and userspace.

Noob question - the kernel <> user space FD sharing is just not implemented yet, or somehow fundamentally hard because kernel fds are "special"?

#21 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 9 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 28 Apr 2022 09:26:41 +0200 Hannes Reinecke wrote:

The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa.
While there is fd passing between processes via AF_UNIX, there is no such mechanism between kernel and userspace.

Noob question - the kernel <> user space FD sharing is just not implemented yet, or somehow fundamentally hard because kernel fds are "special"?

Noob reply: wish I knew.
(I somewhat hoped _you_ would've been able to tell me.)

Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel.
So you probably have to move a similar logic over to AF_NETLINK. And design a new interface on how fds should be passed over AF_NETLINK.

But then you have to face the issue that AF_NELINK is essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...

And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the 'accept()' mechanism Chuck implemented.

I'm open to suggestions, though.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
#22 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 27 lines · Hannes Reinecke
Quoted 9 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 28 Apr 2022 09:26:41 +0200 Hannes Reinecke wrote:

The whole thing started off with the problem on _how_ sockets could be passed between kernel and userspace and vice versa.
While there is fd passing between processes via AF_UNIX, there is no such mechanism between kernel and userspace.

Noob question - the kernel <> user space FD sharing is just not implemented yet, or somehow fundamentally hard because kernel fds are "special"?

Noob reply: wish I knew. (I somewhat hoped _you_ would've been able to tell me.)

Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel. So you probably have to move a similar logic over to AF_NETLINK. And design a new interface on how fds should be passed over AF_NETLINK.

But then you have to face the issue that AF_NELINK is essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...

Quoted 5 lines

And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the 'accept()' mechanism Chuck implemented.

I'm open to suggestions, though.

EXPORT_SYMBOL(receive_fd) would allow interesting implementations.

The kernel keyring facilities have a good API for creating various key_types which are able to perform work such as this from userspace contexts.

I have a working prototype for a keyring key instantiation which allows a userspace process to install a kernel fd on its file table. The problem here is how to match/route such fd passing to appropriate processes in appropriate namespaces. I think this problem is shared by all kernel-to-userspace upcalls, which I hope we can discuss at LSF/MM.

I don't think kernel fds are very special as compared to userspace fds.

Ben

maintainer
#23 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 10 lines · On Thu, 28 Apr 2022 10:09:17 -0400 Benjamin Coddington wrote:
Quoted 10 lines

Noob reply: wish I knew. (I somewhat hoped _you_ would've been able to tell me.)

Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel. So you probably have to move a similar logic over to AF_NETLINK. And design a new interface on how fds should be passed over AF_NETLINK.

Quoted 5 lines
Quoted 5 lines

But then you have to face the issue that AF_NELINK is essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...

Sort of. I double checked the netlink upcall implementations we have, they work by user space entity "registering" their netlink address (portid) at startup. Kernel then directs the upcalls to that address. But AFAICT there's currently no way for the netlink "server" to see when a "client" goes away, which makes me slightly uneasy about using such schemes for security related stuff. The user agent may crash and something else could grab the same address, I think.

Let me CC OvS who uses it the most, perhaps I'm missing a trick.

My thinking was to use the netlink attribute format (just to reuse the helpers and parsing, but we can invent a new TLV format if needed) but create a new socket type specifically for upcalls.

Quoted 8 lines
Quoted 5 lines

And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the 'accept()' mechanism Chuck implemented.

I'm open to suggestions, though.

EXPORT_SYMBOL(receive_fd) would allow interesting implementations.

Quoted 8 lines

The kernel keyring facilities have a good API for creating various key_types which are able to perform work such as this from userspace contexts.

I have a working prototype for a keyring key instantiation which allows a userspace process to install a kernel fd on its file table. The problem here is how to match/route such fd passing to appropriate processes in appropriate namespaces. I think this problem is shared by all kernel-to-userspace upcalls, which I hope we can discuss at LSF/MM.

Almost made me wish I was coming to LFS/MM :)

Quoted 1 line

I don't think kernel fds are very special as compared to userspace fds.

#24 lore raw

Re: [ovs-dev] [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 25 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 28 Apr 2022 10:09:17 -0400 Benjamin Coddington wrote:
Quoted 10 lines

Noob reply: wish I knew. (I somewhat hoped _you_ would've been able to tell me.)

Thing is, the only method I could think of for fd passing is the POSIX fd passing via unix_attach_fds()/unix_detach_fds(). But that's AF_UNIX, which really is designed for process-to-process communication, not process-to-kernel. So you probably have to move a similar logic over to AF_NETLINK. And design a new interface on how fds should be passed over AF_NETLINK.

Quoted 5 lines
Quoted 5 lines

But then you have to face the issue that AF_NELINK is essentially UDP, and you have _no_ idea if and how many processes do listen on the other end. Thing is, you (as the sender) have to copy the fd over to the receiving process, so you'd better _hope_ there is a receiving process. Not to mention that there might be several processes listening in...

Sort of. I double checked the netlink upcall implementations we have, they work by user space entity "registering" their netlink address (portid) at startup. Kernel then directs the upcalls to that address. But AFAICT there's currently no way for the netlink "server" to see when a "client" goes away, which makes me slightly uneasy about using such schemes for security related stuff. The user agent may crash and something else could grab the same address, I think.

Quoted 1 line

Let me CC OvS who uses it the most, perhaps I'm missing a trick.

I don't think there are any tricks. From what I see OVS creates several netlink sockets, connects them to the kernel (nl_pid = 0) and obtains their nl_pid's from the kernel.
These pids are either just a task_tgid_vnr() or a random negative value from the [S32_MIN, -4096] range. After that OVS "registers" those pids in the openvswitch kernel module. That just means sending an array of integers to the kernel. Kernel will later use these integer pids to find the socket and send data to the userspace.

openvswitch module inside the kernel has no way to detect that socket with a certain pid no longer exists. So, it will continue to try to find the socket and send, even if the user-space process is dead.

So, if you can find a way to reliably create a process with the same task_tgid or trick the randomizer inside the netlink_autobind(), you can start receiving upcalls from the kernel in a new process, IIUC. Also, netlink_bind() allows to just specify the nl_pid for listening sockets. That might be another way.

Quoted 30 lines

My thinking was to use the netlink attribute format (just to reuse the helpers and parsing, but we can invent a new TLV format if needed) but create a new socket type specifically for upcalls.

Quoted 8 lines
Quoted 5 lines

And that's something I _definitely_ don't feel comfortable with without guidance from the networking folks, so I didn't pursue it further and we went with the 'accept()' mechanism Chuck implemented.

I'm open to suggestions, though.

EXPORT_SYMBOL(receive_fd) would allow interesting implementations.

Quoted 8 lines

The kernel keyring facilities have a good API for creating various key_types which are able to perform work such as this from userspace contexts.

I have a working prototype for a keyring key instantiation which allows a userspace process to install a kernel fd on its file table. The problem here is how to match/route such fd passing to appropriate processes in appropriate namespaces. I think this problem is shared by all kernel-to-userspace upcalls, which I hope we can discuss at LSF/MM.

Almost made me wish I was coming to LFS/MM :)

Quoted 1 line

I don't think kernel fds are very special as compared to userspace fds.

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

maintainer
#25 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Hannes Reinecke
Quoted 11 lines · On Tue, 26 Apr 2022 11:43:37 +0200 Hannes Reinecke wrote:
Quoted 4 lines

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.

But we can't, as the existing mechanisms (at least for NVMe) creates the socket in-kernel.
Having to create the socket in userspace would require a completely new interface for nvme and will not be backwards compatible.
Not to mention having to rework the nvme driver to accept sockets from userspace instead of creating them internally.

Quoted 2 lines

With this approach we can keep existing infrastructure, and can get a common implementation for either transport.

You add 1.5kLoC and require running a user space agent, surely you're adding new interfaces and are not backward-compatible already.

I don't understand your argument, maybe you could rephrase / dumb it down for me?

#26 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Jakub Kicinski

Hi Jakub-

Quoted 27 lines · Jakub Kicinski
Quoted 24 lines · On Mon, 18 Apr 2022 12:49:50 -0400 Chuck Lever wrote:

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Quoted 7 lines
Quoted 3 lines

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.

The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.

--
Chuck Lever

maintainer
#27 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 8 lines · On Tue, 26 Apr 2022 13:48:20 +0000 Chuck Lever III wrote:
Quoted 3 lines

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.

The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.

Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).

Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?

#28 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 14 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 26 Apr 2022 13:48:20 +0000 Chuck Lever III wrote:
Quoted 3 lines

Create the socket in user space, do all the handshakes you need there and then pass it to the kernel. This is how NBD + TLS works. Scales better and requires much less kernel code.

The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.

Quoted 5 lines

Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).

Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?

RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.

You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?

Also, why has no-one mentioned the NBD on TLS implementation to us before? I will try to review that code soon.

Quoted 1 line

This set does not even have selftests.

I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.

Quoted 3 lines

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?

I'd prefer an in-kernel handshake implementation over a user space one (even one that is sharable amongst transports and ULPs as my proposal is intended to be). However, so far we've been told that an in-kernel handshake implementation is a non-starter.

But in the abstract, we agree that having a single TLS handshake mechanism for kernel consumers is preferable.

--
Chuck Lever

maintainer
#29 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 18 lines · On Tue, 26 Apr 2022 15:58:29 +0000 Chuck Lever III wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 4 lines

The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.

Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).

Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?

RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.

Quoted 4 lines

You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?

I meant scale with the number of possible crypto protocols, I mentioned three there.

Quoted 2 lines

Also, why has no-one mentioned the NBD on TLS implementation to us before? I will try to review that code soon.

Oops, maybe that thing had never seen the light of a public mailing list then :S Dave Watson was working on it at Facebook, but he since moved to greener pastures.

Quoted 6 lines
Quoted 1 line

This set does not even have selftests.

I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.

Whatever level of testing makes you as an engineer comfortable with saying "this test suite is sufficient"? ;)

For TLS we have tools/testing/selftests/net/tls.c - it's hardly an example of excellence but, you know, it catches bugs here and there.

Quoted 8 lines
Quoted 3 lines

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?

Hm, do you remember who you discussed it with? Would be good to loop those folks in. I wasn't involved at the beginning of the TLS work, I know second hand that HW offload and nbd were involved and that the design went thru some serious re-architecting along the way. In the beginning there was a separate socket for control records, and that was nacked.

But also (and perhaps most importantly) I'm not really objecting to creating the socket in the kernel. I'm primarily objecting to a second type of a special TLS socket which has TLS semantics.

Quoted 7 lines

I'd prefer an in-kernel handshake implementation over a user space one (even one that is sharable amongst transports and ULPs as my proposal is intended to be). However, so far we've been told that an in-kernel handshake implementation is a non-starter.

But in the abstract, we agree that having a single TLS handshake mechanism for kernel consumers is preferable.

For some definition of "we" which doesn't not include me?

#30 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 23 lines · Jakub Kicinski
Quoted 18 lines · On Tue, 26 Apr 2022 15:58:29 +0000 Chuck Lever III wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 4 lines

The RPC-with-TLS standard allows unencrypted RPC traffic on the connection before sending ClientHello. I think we'd like to stick with creating the socket in the kernel, for this reason and for the reasons Hannes mentions in his reply.

Umpf, I presume that's reviewed by security people in IETF so I guess it's done right this time (tm).

Your wording seems careful not to imply that you actually need that, tho. Am I over-interpreting?

RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.

Quoted 7 lines
Quoted 4 lines

You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?

I meant scale with the number of possible crypto protocols, I mentioned three there.

I'm looking at previous emails. The "three crypto protocols" don't stand out to me. Which ones?

The prototype has a "handshake type" option that enables the kernel to request handshakes for different transport layer security protocols. Is that the kind of scalability you mean?

For TLS, we expect to have at least:

- ClientHello
- X509
- PSK
- ServerHello
- Re-key

It should be straightforward to add the ability to service other handshake types.

Quoted 14 lines
Quoted 2 lines

Also, why has no-one mentioned the NBD on TLS implementation to us before? I will try to review that code soon.

Oops, maybe that thing had never seen the light of a public mailing list then :S Dave Watson was working on it at Facebook, but he since moved to greener pastures.

Quoted 6 lines
Quoted 1 line

This set does not even have selftests.

I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.

Quoted 6 lines

Whatever level of testing makes you as an engineer comfortable with saying "this test suite is sufficient"? ;)

For TLS we have tools/testing/selftests/net/tls.c - it's hardly an example of excellence but, you know, it catches bugs here and there.

My question wasn't clear, sorry. I meant, what framework is appropriate to use for unit tests in this area?

Quoted 11 lines
Quoted 8 lines
Quoted 3 lines

Plus there are more protocols being actively worked on (QUIC, PSP etc.) Having per ULP special sauce to invoke a user space helper is not the paradigm we chose, and the time as inopportune as ever to change that.

When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?

Hm, do you remember who you discussed it with? Would be good to loop those folks in.

Yes, I remember. Trond Myklebust discussed this with Dave Miller during a hallway conversation at a conference (probably Plumbers) in 2018 or 2019.

Trond is Cc'd on this thread via linux-nfs@ and Dave is Cc'd via netdev@.

I also traded email with Boris Pismenny about this a year ago, and if memory serves he also recommended passing an existing socket up to user space. He is Cc'd on this directly.

Quoted 9 lines

I wasn't involved at the beginning of the
TLS work, I know second hand that HW offload and nbd were involved and that the design went thru some serious re-architecting along the way. In the beginning there was a separate socket for control records, and that was nacked.

But also (and perhaps most importantly) I'm not really objecting to creating the socket in the kernel. I'm primarily objecting to a second type of a special TLS socket which has TLS semantics.

I don't understand your objection. Can you clarify?

AF_TLSH is a listen-only socket. It's just a rendezvous point for passing a kernel socket up to user space. It doesn't have any particular "TLS semantics". It's the user space agent listening on that endpoint that implements particular handshake behaviors.

In fact, if the name AF_TLSH gives you hives, that can be made more generic. However, that makes it harder for the kernel to figure out which listening endpoint handles handshake requests.

Quoted 9 lines
Quoted 7 lines

I'd prefer an in-kernel handshake implementation over a user space one (even one that is sharable amongst transports and ULPs as my proposal is intended to be). However, so far we've been told that an in-kernel handshake implementation is a non-starter.

But in the abstract, we agree that having a single TLS handshake mechanism for kernel consumers is preferable.

For some definition of "we" which doesn't not include me?

The double negative made me blink a couple of times.

I'm working with folks from the Linux NFS community, the Linux block community, and the Linux SMB community. We would be happy to include you in our effort, if you would like to be more involved.

--
Chuck Lever

maintainer
#31 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 12 lines · On Wed, 27 Apr 2022 14:42:53 +0000 Chuck Lever III wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines

RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.

You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?

Quoted 5 lines
Quoted 2 lines

I meant scale with the number of possible crypto protocols, I mentioned three there.

I'm looking at previous emails. The "three crypto protocols" don't stand out to me. Which ones?

TLS, QUIC and PSP maybe that was in a different email that what you quoted, sorry:
https://lore.kernel.org/all/[email protected]/

PSP:
https://raw.githubusercontent.com/google/psp/main/doc/PSP_Arch_Spec.pdf

Quoted 23 lines

The prototype has a "handshake type" option that enables the kernel to request handshakes for different transport layer security protocols. Is that the kind of scalability you mean?

For TLS, we expect to have at least:

- ClientHello
- X509
- PSK
- ServerHello
- Re-key

It should be straightforward to add the ability to service other handshake types.

Quoted 8 lines
Quoted 4 lines

I can include unit tests with the prototype. Someone needs to educate me on what is the preferred unit test paradigm for this type of subsystem. Examples in the current kernel code base would help too.

Whatever level of testing makes you as an engineer comfortable with saying "this test suite is sufficient"? ;)

Quoted 6 lines
Quoted 3 lines

For TLS we have tools/testing/selftests/net/tls.c - it's hardly an example of excellence but, you know, it catches bugs here and there.

My question wasn't clear, sorry. I meant, what framework is appropriate to use for unit tests in this area?

Nothing area specific, tools/testing/selftests/kselftest_harness.h is what the tls test uses.

Quoted 12 lines
Quoted 7 lines
Quoted 4 lines

When we started discussing TLS handshake requirements with some community members several years ago, creating the socket in kernel and passing it up to a user agent was the suggested design. Has that recommendation changed since then?

Hm, do you remember who you discussed it with? Would be good to loop those folks in.

Yes, I remember. Trond Myklebust discussed this with Dave Miller during a hallway conversation at a conference (probably Plumbers) in 2018 or 2019.

Quoted 6 lines

Trond is Cc'd on this thread via linux-nfs@ and Dave is Cc'd via netdev@.

I also traded email with Boris Pismenny about this a year ago, and if memory serves he also recommended passing an existing socket up to user space. He is Cc'd on this directly.

I see.

Quoted 12 lines
Quoted 9 lines

I wasn't involved at the beginning of the
TLS work, I know second hand that HW offload and nbd were involved and that the design went thru some serious re-architecting along the way. In the beginning there was a separate socket for control records, and that was nacked.

But also (and perhaps most importantly) I'm not really objecting to creating the socket in the kernel. I'm primarily objecting to a second type of a special TLS socket which has TLS semantics.

I don't understand your objection. Can you clarify?

Quoted 8 lines

AF_TLSH is a listen-only socket. It's just a rendezvous point for passing a kernel socket up to user space. It doesn't have any particular "TLS semantics". It's the user space agent listening on that endpoint that implements particular handshake behaviors.

In fact, if the name AF_TLSH gives you hives, that can be made more generic.

Yes, a more generic "user space please bake my socket" interface is what I'm leaning towards.

Quoted 2 lines

However, that makes it harder for the kernel to figure out which listening endpoint handles handshake requests.

Right, the listening endpoint...

Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the getsockopt() now)?

The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via recv()?

#32 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 23 lines · Jakub Kicinski
Quoted 12 lines · On Wed, 27 Apr 2022 14:42:53 +0000 Chuck Lever III wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines

RPC-with-TLS requires one RPC as a "starttls" token. That could be done in user space as part of the handshake, but it is currently done in the kernel to enable the user agent to be shared with other kernel consumers of TLS. Keep in mind that we already have two real consumers: NVMe and RPC-with-TLS; and possibly QUIC.

You asserted earlier that creating sockets in user space "scales better" but did not provide any data. Can we see some? How well does it need to scale for storage protocols that use long-lived connections?

Quoted 5 lines
Quoted 2 lines

I meant scale with the number of possible crypto protocols, I mentioned three there.

I'm looking at previous emails. The "three crypto protocols" don't stand out to me. Which ones?

Quoted 6 lines

TLS, QUIC and PSP maybe that was in a different email that what you quoted, sorry:
https://lore.kernel.org/all/[email protected]/

PSP:
https://raw.githubusercontent.com/google/psp/main/doc/PSP_Arch_Spec.pdf

During the design process, we discussed both TLS and QUIC handshake requirements, which are nearly the same. QUIC will want a TLSv1.3 handshake on a UDP socket, effectively. We can support DTLS in a similar fashion.

We hope that the proposed design can be used for all of those, and barring anything unforeseen in the description of PSP you provided, PSP can be supported as well.

The handshake agent is really only a shim around a TLS library. There isn't much to it.

Quoted 7 lines

Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the getsockopt() now)?

The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via recv()?

We considered this kind of design. A reasonable place to start there would be to fabricate new NETLINK messages to do this. I don't see much benefit over what is done now, it's just a different isomer of syntactic sugar, but it could be considered.

The issue is how the connected socket is materialized in user space. accept(2) is the historical way to instantiate an already connected socket in a process's file table, and seems like a natural fit. When the handshake agent is done with the handshake, it closes the socket. This invokes the tlsh_release() function which can check whether the IV implantation was successful.

So instead of an AF_TLSH listener we could use a named pipe or a netlink socket and a blocking recv(), as long as there is a reasonable solution to how a connected socket fd is attached to the handshake agent process.

I'm flexible about the mechanism for passing handshake parameters. Attaching them to the connected socket seems convenient, but perhaps not aesthetic.

--
Chuck Lever

maintainer
#33 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 13 lines · On Thu, 28 Apr 2022 01:29:10 +0000 Chuck Lever III wrote:
Quoted 7 lines

Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the getsockopt() now)?

The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via recv()?

We considered this kind of design. A reasonable place to start there would be to fabricate new NETLINK messages to do this. I don't see much benefit over what is done now, it's just a different isomer of syntactic sugar, but it could be considered.

Quoted 5 lines

The issue is how the connected socket is materialized in user space. accept(2) is the historical way to instantiate an already connected socket in a process's file table, and seems like a natural fit. When the handshake agent is done with the handshake, it closes the socket. This invokes the tlsh_release() function which can check

Actually - is that strictly necessary? It seems reasonable for NFS to check that it got TLS, since that's what it explicitly asks for per standard. But it may not always be the goal. In large data center networks there can be a policy the user space agent consults to choose what security to install. It may end up doing the auth but not enable crypto if confidentiality is deemed unnecessary.

Obviously you may not have those requirements but if we can cover them without extra complexity it'd be great.

Quoted 1 line

whether the IV implantation was successful.

I'm used to IV meaning Initialization Vector in context of crypto, what does "IV implementation" stand for?

Quoted 8 lines

So instead of an AF_TLSH listener we could use a named pipe or a netlink socket and a blocking recv(), as long as there is a reasonable solution to how a connected socket fd is attached to the handshake agent process.

I'm flexible about the mechanism for passing handshake parameters. Attaching them to the connected socket seems convenient, but perhaps not aesthetic.

recv()-based version would certainly make me happy.

#34 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 31 lines · Jakub Kicinski
Quoted 13 lines · On Thu, 28 Apr 2022 01:29:10 +0000 Chuck Lever III wrote:
Quoted 7 lines

Is it possible to instead create a fd-passing-like structured message which could carry the fd and all the relevant context (what goes via the getsockopt() now)?

The user space agent can open such upcall socket, then bind to whatever entity it wants to talk to on the kernel side and read the notifications via recv()?

We considered this kind of design. A reasonable place to start there would be to fabricate new NETLINK messages to do this. I don't see much benefit over what is done now, it's just a different isomer of syntactic sugar, but it could be considered.

Quoted 5 lines

The issue is how the connected socket is materialized in user space. accept(2) is the historical way to instantiate an already connected socket in a process's file table, and seems like a natural fit. When the handshake agent is done with the handshake, it closes the socket. This invokes the tlsh_release() function which can check

Actually - is that strictly necessary? It seems reasonable for NFS to check that it got TLS, since that's what it explicitly asks for per standard. But it may not always be the goal. In large data center networks there can be a policy the user space agent consults to choose what security to install. It may end up doing the auth but not enable crypto if confidentiality is deemed unnecessary.

Quoted 2 lines

Obviously you may not have those requirements but if we can cover them without extra complexity it'd be great.

We can be flexible about how/where handshake success is checked.

However, using a simple close(2) to signal that the handshake has completed does not communicate whether the handshake was indeed successful. We will need a (richer) return/error code from the handshake agent for that use case.

Quoted 4 lines
Quoted 1 line

whether the IV implantation was successful.

I'm used to IV meaning Initialization Vector in context of crypto, what does "IV implementation" stand for?

Implantation, not implementation. The handshake agent implants the initialization vector in the socket before it closes it.

--
Chuck Lever

#35 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Chuck Lever
Quoted 19 lines · Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

Quoted 9 lines

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.

Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.

A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?

#36 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 45 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

Quoted 9 lines

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.

Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.

Quoted 7 lines

A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?

JFYI:
For in kernel NFSD hadnshakes we also use the gssproxy unix socket in the kernel, which allows GSSAPI handshakes to be relayed from the kernel to a user space listening daemon.

The infrastructure is technically already available and could be reasonably simply extended to do TLS negotiations as well.

Not saying it is the best interface, but it is already available, and already used by NFS code.

Simo.

Signature · 3 lines
Simo Sorce
RHEL Crypto Team
Red Hat, Inc
#37 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 58 lines · Simo Sorce
Quoted 45 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

Quoted 9 lines

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.

Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.

Quoted 7 lines

A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?

Quoted 7 lines

JFYI:
For in kernel NFSD hadnshakes we also use the gssproxy unix socket in the kernel, which allows GSSAPI handshakes to be relayed from the kernel to a user space listening daemon.

The infrastructure is technically already available and could be reasonably simply extended to do TLS negotiations as well.

To fill in a little about our design thinking:

We chose not to use either gssproxy or gssd for the TLS handshake prototype so that we don't add a dependency on RPC infrastructure for other TLS consumers such as NVMe. Non-RPC consumers view that kind of dependency as quite undesirable.

Also, neither of those existing mechanisms helped us address the issue of passing a connected socket endpoint. listen/poll/accept/close addresses that issue quite directly.

--
Chuck Lever

#38 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

In reply to Boris Pismenny
Quoted 40 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

Quoted 9 lines

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.

Quoted 8 lines

Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.

I agree that control messages need to be handled as well. For the moment, the prototype simply breaks the connection when a control message is encountered, and a new session is negotiated. That of course is not the desired long-term solution.

If we believe that control messages are going to be distinct for each transport security layer, then perhaps we cannot make the handshake mechanism generic -- it will have to be specific to each security layer. Just a thought.

Quoted 7 lines

A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?

Certainly piping the handshake messages up to user space instead of handing off a socket is possible. The TLS libraries would need to tolerate this, and GnuTLS (at least) appears OK with performing a handshake on an AF_TLSH socket.

However, I don't see a need to outright avoid passing a connected endpoint to user space. The only difficulty with it seems to be that it hasn't been done before in quite this way.

--
Chuck Lever

#39 lore raw

Re: [PATCH RFC 4/5] net/tls: Add support for PF_TLSH (a TLS handshake listener)

Quoted 68 lines · Chuck Lever III
Quoted 40 lines · Boris Pismenny
Quoted 19 lines · Chuck Lever

In-kernel TLS consumers need a way to perform a TLS handshake. In the absence of a handshake implementation in the kernel itself, a mechanism to perform the handshake in user space, using an existing TLS handshake library, is necessary.

I've designed a way to pass a connected kernel socket endpoint to user space using the traditional listen/accept mechanism. accept(2) gives us a well-understood way to materialize a socket endpoint as a normal file descriptor in a specific user space process. Like any open socket descriptor, the accepted FD can then be passed to a library such as openSSL to perform a TLS handshake.

This prototype currently handles only initiating client-side TLS handshakes. Server-side handshakes and key renegotiation are left to do.

Security Considerations
~~~~~~~~ ~~~~~~~~~~~~~~

Quoted 9 lines

This prototype is net-namespace aware.

The kernel has no mechanism to attest that the listening user space agent is trustworthy.

Currently the prototype does not handle multiple listeners that overlap -- multiple listeners in the same net namespace that have overlapping bind addresses.

Thanks for posting this. As we discussed offline, I think this approach is more manageable compared to a full in-kernel TLS handshake. A while ago, I've hacked around TLS to implement the data-path for NVMe-TLS and the data-path is indeed very simple provided an infrastructure such as this one.

Quoted 8 lines

Making this more generic is desirable, and this obviously requires supporting multiple listeners for multiple protocols (TLS, DTLS, QUIC, PSP, etc.), which suggests that it will reside somewhere outside of net/tls. Moreover, there is a need to support (TLS) control messages here too. These will occasionally require going back to the userspace daemon during kernel packet processing. A few examples are handling: TLS rekey, TLS close_notify, and TLS keepalives. I'm not saying that we need to support everything from day-1, but there needs to be a way to support these.

I agree that control messages need to be handled as well. For the moment, the prototype simply breaks the connection when a control message is encountered, and a new session is negotiated. That of course is not the desired long-term solution.

If we believe that control messages are going to be distinct for each transport security layer, then perhaps we cannot make the handshake mechanism generic -- it will have to be specific to each security layer. Just a thought.

Quoted 7 lines

A related kernel interface is the XFRM netlink where the kernel asks a userspace daemon to perform an IKE handshake for establishing IPsec SAs. This works well when the handshake runs on a different socket, perhaps that interface can be extended to do handshakes on a given socket that lives in the kernel without actually passing the fd to userespace. If we avoid instantiating a full socket fd in userspace, then the need for an accept(2) interface is reduced, right?

Quoted 5 lines

Certainly piping the handshake messages up to user space instead of handing off a socket is possible. The TLS libraries would need to tolerate this, and GnuTLS (at least) appears OK with performing a handshake on an AF_TLSH socket.

Yeah, and I guess that'll be the hard part.
We would need to design an entirely data path for gnutls when going down that path.
The beauty of the fd-passing idea is that gnutls (and openssl for that matter) will 'just work' (tm), without us have to do larger surgery there.
Just for reference, I've raised an issue with gnutls to accept long identifiers in TLS 1.3 (issue #1323), which is required for NVMe-over-TLS support. That one is lingering for over two months now. And that's a relatively simple change; I don't want to imagine how long it'd take to try to push in a larger redesign...

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
#40 lore raw

RFC PATCH 5/5 net/tls: Add observability for AF_TLSH sockets

In reply to Chuck Lever
---
 net/tls/af_tlsh.c |   50 +++++++
 net/tls/trace.c   |    3 
 net/tls/trace.h   |  355 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 402 insertions(+), 6 deletions(-)
net/tls/af_tlsh.c +44 -6
index 4d1c1de3a474..31eae3e0d54c 100644
@@ -239,8 +239,10 @@ static bool tlsh_handshake_done(struct sock *sk)
 		tlsh_sock_restore_locked(sk);
 
 		if (tlsh_crypto_info_initialized(sk)) {
+			trace_tlsh_handshake_ok(sk);
 			done(data, 0);
 		} else {
+			trace_tlsh_handshake_failed(sk);
 			done(data, -EACCES);
 		}
 	}
@@ -282,6 +284,8 @@ static int tlsh_release(struct socket *sock)
 	if (!sk)
 		return 0;
 
+	trace_tlsh_release(sock);
+
 	switch (sk->sk_family) {
 	case AF_INET:
 		if (!tlsh_handshake_done(sk))
@@ -364,6 +368,7 @@ static int tlsh_bind(struct socket *sock, struct sockaddr *uaddr, int addrlen)
 	}
 
 	tsk->th_bind_family = uaddr->sa_family;
+	trace_tlsh_bind(sock);
 	return 0;
 }
 
@@ -386,6 +391,8 @@ static int tlsh_accept(struct socket *listener, struct socket *newsock, int flag
 	long timeo;
 	int rc;
 
+	trace_tlsh_accept(listener);
+
 	rc = -EPERM;
 	if (!capable(CAP_NET_BIND_SERVICE))
 		goto out;
@@ -429,6 +436,7 @@ static int tlsh_accept(struct socket *listener, struct socket *newsock, int flag
 	}
 
 	sock_graft(newsk, newsock);
+	trace_tlsh_newsock(newsock, newsk);
 
 out_release:
 	release_sock(sk);
@@ -450,6 +458,8 @@ static int tlsh_getname(struct socket *sock, struct sockaddr *uaddr, int peer)
 {
 	struct sock *sk = sock->sk;
 
+	trace_tlsh_getname(sock);
+
 	switch (sk->sk_family) {
 	case AF_INET:
 		return inet_getname(sock, uaddr, peer);
@@ -486,6 +496,7 @@ static __poll_t tlsh_poll(struct file *file, struct socket *sock,
 			mask |= EPOLLIN | EPOLLRDNORM;
 		if (sk_is_readable(sk))
 			mask |= EPOLLIN | EPOLLRDNORM;
+		trace_tlsh_poll_listener(sock, mask);
 		return mask;
 	}
 
@@ -504,6 +515,7 @@ static __poll_t tlsh_poll(struct file *file, struct socket *sock,
 	if (sk->sk_err || !skb_queue_empty_lockless(&sk->sk_error_queue))
 		mask |= EPOLLERR;
 
+	trace_tlsh_poll(sock, mask);
 	return mask;
 }
 
@@ -539,6 +551,7 @@ static int tlsh_listen(struct socket *sock, int backlog)
 	sk->sk_state = TCP_LISTEN;
 	tlsh_register_listener(sk);
 
+	trace_tlsh_listen(sock);
 	rc = 0;
 
 out:
@@ -559,6 +572,8 @@ static int tlsh_shutdown(struct socket *sock, int how)
 {
 	struct sock *sk = sock->sk;
 
+	trace_tlsh_shutdown(sock);
+
 	switch (sk->sk_family) {
 	case AF_INET:
 		break;
@@ -590,6 +605,8 @@ static int tlsh_setsockopt(struct socket *sock, int level, int optname,
 {
 	struct sock *sk = sock->sk;
 
+	trace_tlsh_setsockopt(sock);
+
 	switch (sk->sk_family) {
 	case AF_INET:
 		break;
@@ -756,6 +773,8 @@ static int tlsh_getsockopt(struct socket *sock, int level, int optname,
 	struct sock *sk = sock->sk;
 	int ret;
 
+	trace_tlsh_getsockopt(sock);
+
 	switch (sk->sk_family) {
 	case AF_INET:
 		break;
@@ -803,6 +822,9 @@ static int tlsh_getsockopt(struct socket *sock, int level, int optname,
 static int tlsh_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
 {
 	struct sock *sk = sock->sk;
+	int ret;
+
+	trace_tlsh_sendmsg_start(sock, size);
 
 	switch (sk->sk_family) {
 	case AF_INET:
@@ -812,12 +834,19 @@ static int tlsh_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
 		break;
 #endif
 	default:
-		return -EOPNOTSUPP;
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (unlikely(inet_send_prepare(sk))) {
+		ret = -EAGAIN;
+		goto out;
 	}
+	ret = sk->sk_prot->sendmsg(sk, msg, size);
 
-	if (unlikely(inet_send_prepare(sk)))
-		return -EAGAIN;
-	return sk->sk_prot->sendmsg(sk, msg, size);
+out:
+	trace_tlsh_sendmsg_result(sock, ret);
+	return ret;
 }
 
 /**
@@ -835,6 +864,9 @@ static int tlsh_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,
 			int flags)
 {
 	struct sock *sk = sock->sk;
+	int ret;
+
+	trace_tlsh_recvmsg_start(sock, size);
 
 	switch (sk->sk_family) {
 	case AF_INET:
@@ -844,12 +876,17 @@ static int tlsh_recvmsg(struct socket *sock, struct msghdr *msg, size_t size,
 		break;
 #endif
 	default:
-		return -EOPNOTSUPP;
+		ret = -EOPNOTSUPP;
+		goto out;
 	}
 
 	if (likely(!(flags & MSG_ERRQUEUE)))
 		sock_rps_record_flow(sk);
-	return sock_common_recvmsg(sock, msg, size, flags);
+	ret = sock_common_recvmsg(sock, msg, size, flags);
+
+out:
+	trace_tlsh_recvmsg_result(sock, ret);
+	return ret;
 }
 
 static const struct proto_ops tlsh_proto_ops = {
@@ -920,6 +957,7 @@ int tlsh_pf_create(struct net *net, struct socket *sock, int protocol, int kern)
 	}
 
 	tlsh_sk(sk)->th_bind_family = AF_UNSPEC;
+	trace_tlsh_pf_create(sock);
 	return 0;
 
 err_sk_put:
net/tls/trace.c +3 -0
index e374913cf9c9..3747ca2ede67 100644
@@ -2,6 +2,9 @@
 /* Copyright (C) 2019 Netronome Systems, Inc. */
 
 #include <linux/module.h>
+#include <linux/net.h>
+#include <net/sock.h>
+#include <net/tls.h>
 
 #ifndef __CHECKER__
 #define CREATE_TRACE_POINTS
net/tls/trace.h +355 -0

367 lines not on this page · raw message

Reviewed-by on v1 5/5
#41 lore raw

Re: [PATCH RFC 5/5] net/tls: Add observability for AF_TLSH sockets

Quoted 7 lines · Chuck Lever
---
  net/tls/af_tlsh.c |   50 +++++++
  net/tls/trace.c   |    3
  net/tls/trace.h   |  355 +++++++++++++++++++++++++++++++++++++++++++++++++++++
  3 files changed, 402 insertions(+), 6 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer