tls: rx: avoid skb_cow_data()
Subsystems (3)
Jakub Kicinski
maintainerv2 cover tls: rx: avoid skb_cow_data()
TLS calls skb_cow_data() on the skb it received from strparser whenever it needs to hold onto the skb with the decrypted data. (The alternative being decrypting directly to a user space buffer in whic case the input skb doesn't get modified or used after.) TLS needs the decrypted skb:
- almost always with TLS 1.3 (unless the new NoPad is enabled);
- when user space buffer is too small to fit the record;
- when BPF sockmap is enabled.
Most of the time the skb we get out of strparser is a clone of a 64kB data unit coalsced by GRO. To make things worse skb_cow_data() tries to output a linear skb and allocates it with GFP_ATOMIC. This occasionally fails even under moderate memory pressure.
This patch set rejigs the TLS Rx so that we don't expect decryption in place. The decryption handlers return an skb which may or may not be the skb from strparser. For TLS 1.3 this results in a 20-30% performance improvement without NoPad enabled.
v2: rebase after 3d8c51b25a23 ("net/tls: Check for errors in tls_device_init")
Jakub Kicinski (11):
tls: rx: allow only one reader at a time
tls: rx: don't try to keep the skbs always on the list
tls: rx: don't keep decrypted skbs on ctx->recv_pkt
tls: rx: remove the message decrypted tracking
tls: rx: factor out device darg update
tls: rx: read the input skb from ctx->recv_pkt
tls: rx: return the decrypted skb via darg
tls: rx: async: adjust record geometry immediately
tls: rx: async: hold onto the input skb
tls: rx: async: don't put async zc on the list
tls: rx: decrypt into a fresh skb
include/net/strparser.h | 1 - include/net/tls.h | 4 + net/tls/Makefile | 2 +- net/tls/tls.h | 20 +- net/tls/tls_device.c | 25 ++- net/tls/tls_strp.c | 17 ++ net/tls/tls_sw.c | 458 ++++++++++++++++++++++++---------------- 7 files changed, 333 insertions(+), 194 deletions(-) create mode 100644 net/tls/tls_strp.c
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 1/11 tls: rx: allow only one reader at a time
recvmsg() in TLS gets data from the skb list (rx_list) or fresh skbs we read from TCP via strparser. The former holds skbs which were already decrypted for peek or decrypted and partially consumed.
tls_wait_data() only notices appearance of fresh skbs coming out of TCP (or psock). It is possible, if there is a concurrent call to peek() and recv() that the peek() will move the data from input to rx_list without recv() noticing. recv() will then read data out of order or never wake up.
This is not a practical use case/concern, but it makes the self tests less reliable. This patch solves the problem by allowing only one reader in.
Because having multiple processes calling read()/peek() is not normal avoid adding a lock and try to fast-path the single reader case.
--- include/net/tls.h | 3 +++ net/tls/tls_sw.c | 61 +++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 57 insertions(+), 7 deletions(-)
include/net/tls.h +3 -0
@@ -116,11 +116,14 @@ struct tls_sw_context_rx {void (*saved_data_ready)(struct sock *sk);struct sk_buff *recv_pkt; +u8 reader_present; u8 async_capable:1; u8 zc_capable:1; + u8 reader_contended:1;atomic_t decrypt_pending;/* protect crypto_wait with decrypt_pending*/ spinlock_t decrypt_compl_lock; +struct wait_queue_head wq; };struct tls_record_info {
net/tls/tls_sw.c +54 -7
103 lines not on this page · raw message
Signature · 1 line
2.36.1
Re: [PATCH net-next v2 01/11] tls: rx: allow only one reader at a time
Quoted 63 lines · Jakub Kicinski
recvmsg()in TLS gets data from the skb list (rx_list) or fresh skbs we read from TCP via strparser. The former holds skbs which were already decrypted for peek or decrypted and partially consumed.
tls_wait_data()only notices appearance of fresh skbs coming out of TCP (or psock). It is possible, if there is a concurrent call topeek()andrecv()that thepeek()will move the data from input torx_listwithoutrecv()noticing.recv()will then read data out of order or never wake up.This is not a practical use case/concern, but it makes the self tests less reliable. This patch solves the problem by allowing only one reader in.
Because having multiple processes calling read()/peek() is not normal avoid adding a lock and try to fast-path the single reader case.
Signed-off-by Jakub Kicinski <[email protected]>--- include/net/tls.h | 3 +++ net/tls/tls_sw.c | 61 +++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 57 insertions(+), 7 deletions(-)include/net/tls.h +3
-0@@ -116,11 +116,14 @@ struct tls_sw_context_rx {void (*saved_data_ready)(struct sock *sk);struct sk_buff *recv_pkt; +u8 reader_present; u8 async_capable:1; u8 zc_capable:1; + u8 reader_contended:1;atomic_t decrypt_pending;/* protect crypto_wait with decrypt_pending*/ spinlock_t decrypt_compl_lock; +struct wait_queue_head wq; };struct tls_record_info {net/tls/tls_sw.c +11
-0@@ -1753,6 +1753,51 @@ tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,sk_flush_backlog (sk); } +static long tls_rx_reader_lock (struct sock *sk,struct tls_sw_context_rx *ctx, +bool nonblock) +{ +long timeo; + +lock_sock (sk); + + timeo =sock_rcvtimeo (sk, nonblock); + +while (unlikely (ctx->reader_present)) { +DEFINE_WAIT_FUNC (wait, woken_wake_function);
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1776
Quoted 10 lines
net/tls/tls_sw.c +10
-0+ + ctx->reader_contended =1 ; + +add_wait_queue (&ctx->wq, &wait); +sk_wait_event (sk, &timeo, + !READ_ONCE (ctx->reader_present), &wait); +remove_wait_queue (&ctx->wq, &wait); + +if (!timeo) +return -EAGAIN ;
We return with socket lock held, and callers seem to not release the lock.
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1778
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (signal_pending (current)) +return sock_intr_errno (timeo);
same here.
Let's wait for syzbot to catch up :)
Quoted 86 lines
net/tls/tls_sw.c +31
-8+ } + +WRITE_ONCE (ctx->reader_present,1 ); + +return timeo; +} + +static void tls_rx_reader_unlock (struct sock *sk,struct tls_sw_context_rx *ctx) +{ +if (unlikely (ctx->reader_contended)) { +if (wq_has_sleeper (&ctx->wq)) +wake_up (&ctx->wq); +else + ctx->reader_contended =0 ; + +WARN_ON_ONCE (!ctx->reader_present); + } + +WRITE_ONCE (ctx->reader_present,0 ); +release_sock (sk); +} + int tls_sw_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,@@ -1782,7 +1827,9 @@ int tls_sw_recvmsg(struct sock *sk,return sock_recv_errqueue (sk, msg, len,SOL_IP ,IP_RECVERR ); psock =sk_psock_get (sk); -lock_sock (sk); + timeo =tls_rx_reader_lock (sk, ctx, flags &MSG_DONTWAIT ); +if (timeo <0 ) +return timeo; bpf_strp_enabled =sk_psock_strp_enabled (psock);/* If crypto failed the connection is broken */ @@ -1801,7 +1848,6 @@ int tls_sw_recvmsg(struct sock *sk, target =sock_rcvlowat (sk, flags &MSG_WAITALL , len); len = len - copied; - timeo =sock_rcvtimeo (sk, flags &MSG_DONTWAIT ); zc_capable = !bpf_strp_enabled && !is_kvec && !is_peek && ctx->zc_capable;@@ -1956,7 +2002,7 @@ int tls_sw_recvmsg(struct sock *sk, copied += decrypted; end: -release_sock (sk); +tls_rx_reader_unlock (sk, ctx);if (psock)sk_psock_put (sk, psock);return copied ? : err;@@ -1978,9 +2024,9 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,long timeo;int chunk; -lock_sock (sk); - - timeo =sock_rcvtimeo (sk, flags &SPLICE_F_NONBLOCK ); + timeo =tls_rx_reader_lock (sk, ctx, flags &SPLICE_F_NONBLOCK ); +if (timeo <0 ) +return timeo; from_queue = !skb_queue_empty (&ctx->rx_list);if (from_queue) {@@ -2029,7 +2075,7 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos, } splice_read_end: -release_sock (sk); +tls_rx_reader_unlock (sk, ctx);return copied ? : err; }@@ -2371,6 +2417,7 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) }else {crypto_init_wait (&sw_ctx_rx->async_wait);spin_lock_init (&sw_ctx_rx->decrypt_compl_lock); +init_waitqueue_head (&sw_ctx_rx->wq); crypto_info = &ctx->crypto_recv.info; cctx = &ctx->rx;skb_queue_head_init (&sw_ctx_rx->rx_list); --2.36.1
Jakub Kicinski
maintainerRe: [PATCH net-next v2 01/11] tls: rx: allow only one reader at a time
Quoted 9 lines · On Wed, 20 Jul 2022 10:37:02 +0200 Eric Dumazet wrote:
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1776
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (!timeo) +return -EAGAIN ;We return with socket lock held, and callers seem to not release the lock.
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1778
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (signal_pending (current)) +return sock_intr_errno (timeo);same here.
Thanks a lot for catching these.
Quoted 1 line
Let's wait for syzbot to catch up :)
I'll send the fixes later today. This is just a passing comment, right? There isn't a report you know is coming? Otherwise I can wait to give syzbot credit, too.
I have two additional questions while I have you :)
Is the timeo supposed to be for the entire operation? Right now TLS seems to use a fresh timeo every time it goes to wait so the cumulative wait can be much longer, as long as some data keeps coming in :/
Last one - I posted a bit of a disemboweling patch for TCP, LMK if it's no bueno:
Re: [PATCH net-next v2 01/11] tls: rx: allow only one reader at a time
Quoted 12 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 20 Jul 2022 10:37:02 +0200 Eric Dumazet wrote:
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1776
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (!timeo) +return -EAGAIN ;We return with socket lock held, and callers seem to not release the lock.
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1778
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (signal_pending (current)) +return sock_intr_errno (timeo);same here.
Quoted 7 lines
Thanks a lot for catching these.
Quoted 1 line
Let's wait for syzbot to catch up :)
I'll send the fixes later today. This is just a passing comment, right? There isn't a report you know is coming? Otherwise I can wait to give syzbot credit, too.
I now have a full syzbot report, with a repro and bisection, I am releasing it now.
Quoted 6 lines
I have two additional questions while I have you :)
Is the timeo supposed to be for the entire operation? Right now TLS seems to use a fresh timeo every time it goes to wait so the cumulative wait can be much longer, as long as some data keeps coming in :/
Good question. I am not sure how this timeout is used in applications, but I would think it serves as a way to make sure a stall is detected. So restarting the timeout every time there is progress would make sense.
Application needing a different behavior can still use regular timer, independent of networking, ( alarm() being the most simple one)
Quoted 5 lines
Last one - I posted a bit of a disemboweling patch for TCP, LMK if it's no bueno:
Re: [PATCH net-next v2 01/11] tls: rx: allow only one reader at a time
Quoted 16 lines · Eric Dumazet
Quoted 14 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 20 Jul 2022 10:37:02 +0200 Eric Dumazet wrote:
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1776
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (!timeo) +return -EAGAIN ;We return with socket lock held, and callers seem to not release the lock.
net/tls/tls_sw.c tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot, · line 1778
Quoted 2 lines
net/tls/tls_sw.c +2
-0+if (signal_pending (current)) +return sock_intr_errno (timeo);same here.
Thanks a lot for catching these.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Let's wait for syzbot to catch up :)
I'll send the fixes later today. This is just a passing comment, right? There isn't a report you know is coming? Otherwise I can wait to give syzbot credit, too.
I now have a full syzbot report, with a repro and bisection, I am releasing it now.
( [syzbot] WARNING: still has locks held in tls_rx_reader_lock )
Quoted 8 lines
Quoted 7 lines
I have two additional questions while I have you :)
Is the timeo supposed to be for the entire operation? Right now TLS seems to use a fresh timeo every time it goes to wait so the cumulative wait can be much longer, as long as some data keeps coming in :/
Quoted 12 lines
Good question. I am not sure how this timeout is used in applications, but I would think it serves as a way to make sure a stall is detected. So restarting the timeout every time there is progress would make sense.
Application needing a different behavior can still use regular timer, independent of networking, (
alarm()being the most simple one)Quoted 5 lines
Last one - I posted a bit of a disemboweling patch for TCP, LMK if it's no bueno:
Ah sorry, I have no comments, I guess we might try to factorize all these similar functions at some point.
Jakub Kicinski
maintainerPATCH v2 2/11 tls: rx: don't try to keep the skbs always on the list
In reply to Jakub KicinskiI thought that having the skb either always on the ctx->rx_list or ctx->recv_pkt will simplify the handling, as we would not have to remember to flip it from one to the other on exit paths.
This became a little harder to justify with the fix for BPF sockmaps. Subsequent changes will make the situation even worse. Queue the skbs only when really needed.
--- net/tls/tls_sw.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
net/tls/tls_sw.c +12 -11
@@ -1861,8 +1861,11 @@ int tls_sw_recvmsg(struct sock *sk,if (psock) { chunk =sk_msg_recvmsg (sk, psock, msg, len, flags); -if (chunk >0 ) - goto leave_on_list; +if (chunk >0 ) { + decrypted += chunk; + len -= chunk; +continue ; + } } goto recv_end; }
@@ -1908,14 +1911,14 @@ int tls_sw_recvmsg(struct sock *sk, ctx->recv_pkt =NULL ;__strp_unpause (&ctx->strp); -__skb_queue_tail (&ctx->rx_list, skb);if (async) {/* TLS 1.2-only, to_decrypt must be text length */ chunk =min_t (int, to_decrypt, len); -leave_on_list: +put_on_rx_list: decrypted += chunk; len -= chunk; +__skb_queue_tail (&ctx->rx_list, skb);continue ; }/* TLS 1.3 may have updated the length by more than overhead */
@@ -1925,8 +1928,6 @@ int tls_sw_recvmsg(struct sock *sk,bool partially_consumed = chunk > len;if (bpf_strp_enabled) { -/* BPF may try to queue the skb */ -__skb_unlink (skb, &ctx->rx_list); err =sk_psock_tls_strp_read (psock, skb);if (err != __SK_PASS) { rxm->offset = rxm->offset + rxm->full_len;
@@ -1935,7 +1936,6 @@ int tls_sw_recvmsg(struct sock *sk,consume_skb (skb);continue ; } -__skb_queue_tail (&ctx->rx_list, skb); }if (partially_consumed)
@@ -1943,23 +1943,24 @@ int tls_sw_recvmsg(struct sock *sk, err =skb_copy_datagram_msg (skb, rxm->offset, msg, chunk); -if (err <0 ) +if (err <0 ) { +__skb_queue_tail (&ctx->rx_list, skb); goto recv_end; + }if (is_peek) - goto leave_on_list; + goto put_on_rx_list;if (partially_consumed) { rxm->offset += chunk; rxm->full_len -= chunk; - goto leave_on_list; + goto put_on_rx_list; } } decrypted += chunk; len -= chunk; -__skb_unlink (skb, &ctx->rx_list);consume_skb (skb);/* Return full control message to userspace before trying
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 3/11 tls: rx: don't keep decrypted skbs on ctx->recv_pkt
In reply to Jakub KicinskiDetach the skb from ctx->recv_pkt after decryption is done, even if we can't consume it.
--- net/tls/tls_sw.c | 49 +++++++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 21 deletions(-)
net/tls/tls_sw.c +28 -21
@@ -1648,6 +1648,12 @@ static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,return 1 ; } +static void tls_rx_rec_done (struct tls_sw_context_rx *ctx) +{ + ctx->recv_pkt =NULL ; +__strp_unpause (&ctx->strp); +} +/* This function traverses the rx_list in tls receive context to copies the * decrypted records into the buffer provided by caller zero copy is not * true. Further, the records are removed from the rx_list if it is not a peek
@@ -1902,15 +1908,20 @@ int tls_sw_recvmsg(struct sock *sk,* For tls1.3, we disable async. */ err =tls_record_content_type (msg, tlm, &control); -if (err <=0 ) +if (err <=0 ) { +tls_rx_rec_done (ctx); +put_on_rx_list_err: +__skb_queue_tail (&ctx->rx_list, skb); goto recv_end; + }/* periodically flush backlog, and feed strparser */ tls_read_flush_backlog (sk, prot, len, to_decrypt, decrypted + copied, &flushed_at); - ctx->recv_pkt =NULL ; -__strp_unpause (&ctx->strp); +/* TLS 1.3 may have updated the length by more than overhead */ + chunk = rxm->full_len; +tls_rx_rec_done (ctx);if (async) {/* TLS 1.2-only, to_decrypt must be text length */
@@ -1921,8 +1932,6 @@ int tls_sw_recvmsg(struct sock *sk,__skb_queue_tail (&ctx->rx_list, skb);continue ; } -/* TLS 1.3 may have updated the length by more than overhead */ - chunk = rxm->full_len;if (!darg.zc) {bool partially_consumed = chunk > len;
@@ -1943,10 +1952,8 @@ int tls_sw_recvmsg(struct sock *sk, err =skb_copy_datagram_msg (skb, rxm->offset, msg, chunk); -if (err <0 ) { -__skb_queue_tail (&ctx->rx_list, skb); - goto recv_end; - } +if (err <0 ) + goto put_on_rx_list_err;if (is_peek) goto put_on_rx_list;
@@ -2020,7 +2027,6 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,struct tls_msg *tlm;struct sk_buff *skb;ssize_t copied =0 ; -bool from_queue;int err =0 ;long timeo;int chunk;
@@ -2029,8 +2035,7 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,if (timeo <0 )return timeo; - from_queue = !skb_queue_empty (&ctx->rx_list); -if (from_queue) { +if (!skb_queue_empty (&ctx->rx_list)) { skb =__skb_dequeue (&ctx->rx_list); }else {struct tls_decrypt_arg darg = {};
@@ -2047,6 +2052,8 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,tls_err_abort (sk, -EBADMSG ); goto splice_read_end; } + +tls_rx_rec_done (ctx); } rxm =strp_msg (skb);
@@ -2055,29 +2062,29 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,/* splice does not support reading control messages */ if (tlm->control !=TLS_RECORD_TYPE_DATA ) { err = -EINVAL ; - goto splice_read_end; + goto splice_requeue; } chunk = min_t(unsigned int, rxm->full_len, len); copied =skb_splice_bits (skb, sk, rxm->offset, pipe, chunk, flags);if (copied <0 ) - goto splice_read_end; + goto splice_requeue; -if (!from_queue) { - ctx->recv_pkt =NULL ; -__strp_unpause (&ctx->strp); - }if (chunk < rxm->full_len) { -__skb_queue_head (&ctx->rx_list, skb); rxm->offset += len; rxm->full_len -= len; - }else { -consume_skb (skb); + goto splice_requeue; } +consume_skb (skb); + splice_read_end:tls_rx_reader_unlock (sk, ctx);return copied ? : err; + +splice_requeue: +__skb_queue_head (&ctx->rx_list, skb); + goto splice_read_end; }bool tls_sw_sock_is_readable (struct sock *sk)
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 4/11 tls: rx: remove the message decrypted tracking
In reply to Jakub KicinskiWe no longer allow a decrypted skb to remain linked to ctx->recv_pkt. Anything on the list is decrypted, anything on ctx->recv_pkt needs to be decrypted.
--- include/net/strparser.h | 1 - net/tls/tls_sw.c | 10 ---------- 2 files changed, 11 deletions(-)
include/net/strparser.h +0 -1
@@ -72,7 +72,6 @@ struct sk_skb_cb {/* strp users' data follows */ struct tls_msg {u8 control; -u8 decrypted; } tls;/* temp_reg is a temporary register used for bpf_convert_data_end_access * when dst_reg == src_reg.
net/tls/tls_sw.c +0 -10
@@ -1563,21 +1563,13 @@ static int decrypt_skb_update(struct sock *sk, struct sk_buff *skb,struct tls_context *tls_ctx =tls_get_ctx (sk);struct tls_prot_info *prot = &tls_ctx->prot_info;struct strp_msg *rxm =strp_msg (skb); -struct tls_msg *tlm =tls_msg (skb);int pad, err; -if (tlm->decrypted) { - darg->zc = false; - darg->async = false; -return 0 ; - } -if (tls_ctx->rx_conf ==TLS_HW ) { err =tls_device_decrypted (sk, tls_ctx, skb, rxm);if (err <0 )return err;if (err >0 ) { - tlm->decrypted =1 ; darg->zc = false; darg->async = false; goto decrypt_done;
@@ -1610,7 +1602,6 @@ static int decrypt_skb_update(struct sock *sk, struct sk_buff *skb, rxm->full_len -= pad; rxm->offset += prot->prepend_size; rxm->full_len -= prot->overhead_size; - tlm->decrypted =1 ; decrypt_next:tls_advance_record_sn (sk, prot, &tls_ctx->rx);
@@ -2130,7 +2121,6 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)if (ret <0 ) goto read_failure; - tlm->decrypted =0 ; tlm->control = header[0 ]; data_len = ((header[4 ] &0xFF ) | (header[3 ] <<8 ));
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 5/11 tls: rx: factor out device darg update
In reply to Jakub KicinskiI already forgot to transform darg from input to output semantics once on the NIC inline crypto fastpath. To avoid this happening again create a device equivalent of decrypt_internal(). A function responsible for decryption and transforming darg.
While at it rename decrypt_internal() to a hopefully slightly more meaningful name.
--- net/tls/tls_sw.c | 60 +++++++++++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 19 deletions(-)
net/tls/tls_sw.c +41 -19
@@ -1404,18 +1404,27 @@ static int tls_setup_from_iter(struct iov_iter *from,return rc; } +/* Decrypt handlers +* +* tls_decrypt_sg() and tls_decrypt_device() are decrypt handlers. +* They must transform the darg in/out argument are as follows: +* | Input | Output +* ------------------------------------------------------------------- +* zc | Zero-copy decrypt allowed | Zero-copy performed +* async | Async decrypt allowed | Async crypto used / in progress +*/ +/* This function decrypts the input skb into either out_iov or in out_sg -* or in skb buffers itself. The input parameter 'zc' indicates if +* or in skb buffers itself. The input parameter 'darg->zc' indicates if * zero-copy mode needs to be tried or not. With zero-copy mode, either * out_iov or out_sg must be non-NULL. In case both out_iov and out_sg are * NULL, then the decryption happens inside skb buffers itself, i.e. -* zero-copy gets disabled and 'zc' is updated. +* zero-copy gets disabled and 'darg->zc' is updated. */ - -static int decrypt_internal (struct sock *sk,struct sk_buff *skb, -struct iov_iter *out_iov, -struct scatterlist *out_sg, -struct tls_decrypt_arg *darg) +static int tls_decrypt_sg (struct sock *sk,struct sk_buff *skb, +struct iov_iter *out_iov, +struct scatterlist *out_sg, +struct tls_decrypt_arg *darg) {struct tls_context *tls_ctx =tls_get_ctx (sk);struct tls_sw_context_rx *ctx =tls_sw_ctx_rx (tls_ctx);
@@ -1556,6 +1565,24 @@ static int decrypt_internal(struct sock *sk, struct sk_buff *skb,return err; } +static int +tls_decrypt_device (struct sock *sk,struct tls_context *tls_ctx, +struct sk_buff *skb,struct tls_decrypt_arg *darg) +{ +int err; + +if (tls_ctx->rx_conf !=TLS_HW ) +return 0 ; + + err =tls_device_decrypted (sk, tls_ctx, skb,strp_msg (skb)); +if (err <=0 ) +return err; + + darg->zc = false; + darg->async = false; +return 1 ; +} +static int decrypt_skb_update (struct sock *sk,struct sk_buff *skb,struct iov_iter *dest,struct tls_decrypt_arg *darg)
@@ -1565,18 +1592,13 @@ static int decrypt_skb_update(struct sock *sk, struct sk_buff *skb,struct strp_msg *rxm =strp_msg (skb);int pad, err; -if (tls_ctx->rx_conf ==TLS_HW ) { - err =tls_device_decrypted (sk, tls_ctx, skb, rxm); -if (err <0 ) -return err; -if (err >0 ) { - darg->zc = false; - darg->async = false; - goto decrypt_done; - } - } + err =tls_decrypt_device (sk, tls_ctx, skb, darg); +if (err <0 ) +return err; +if (err) + goto decrypt_done; - err =decrypt_internal (sk, skb, dest,NULL , darg); + err =tls_decrypt_sg (sk, skb, dest,NULL , darg);if (err <0 ) {if (err == -EBADMSG )TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSDECRYPTERROR );
@@ -1613,7 +1635,7 @@ int decrypt_skb(struct sock *sk, struct sk_buff *skb, {struct tls_decrypt_arg darg = { .zc = true, }; -return decrypt_internal (sk, skb,NULL , sgout, &darg); +return tls_decrypt_sg (sk, skb,NULL , sgout, &darg); } static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 6/11 tls: rx: read the input skb from ctx->recv_pkt
In reply to Jakub KicinskiCallers always pass ctx->recv_pkt into decrypt_skb_update(), and it propagates it to its callees. This may give someone the false impression that those functions can accept any valid skb containing a TLS record. That's not the case, the record sequence number is read from the context, and they can only take the next record coming out of the strp.
Let the functions get the skb from the context instead of passing it in. This will also make it cleaner to return a different skb than ctx->recv_pkt as the decrypted one later on.
Since we're touching the definition of decrypt_skb_update() use this as an opportunity to rename it.
--- net/tls/tls.h | 14 ++++++++------ net/tls/tls_device.c | 25 ++++++++++++++++--------- net/tls/tls_sw.c | 37 ++++++++++++++++++------------------- 3 files changed, 42 insertions(+), 34 deletions(-)
net/tls/tls.h +8 -6
@@ -118,8 +118,7 @@ void tls_device_write_space(struct sock *sk, struct tls_context *ctx);int tls_process_cmsg (struct sock *sk,struct msghdr *msg,unsigned char *record_type); -int decrypt_skb (struct sock *sk,struct sk_buff *skb, -struct scatterlist *sgout); +int decrypt_skb (struct sock *sk,struct scatterlist *sgout); int tls_sw_fallback_init(struct sock *sk, struct tls_offload_context_tx *offload_ctx,
@@ -132,6 +131,11 @@ static inline struct tls_msg *tls_msg(struct sk_buff *skb)return &scb->tls; } +static inline struct sk_buff *tls_strp_msg (struct tls_sw_context_rx *ctx) +{ +return ctx->recv_pkt; +} +#ifdef CONFIG_TLS_DEVICE int tls_device_init (void );void tls_device_cleanup (void );
@@ -140,8 +144,7 @@ void tls_device_free_resources_tx(struct sock *sk);int tls_set_device_offload_rx (struct sock *sk,struct tls_context *ctx);void tls_device_offload_cleanup_rx (struct sock *sk);void tls_device_rx_resync_new_rec (struct sock *sk,u32 rcd_len,u32 seq); -int tls_device_decrypted (struct sock *sk,struct tls_context *tls_ctx, -struct sk_buff *skb,struct strp_msg *rxm); +int tls_device_decrypted (struct sock *sk,struct tls_context *tls_ctx);#else static inline int tls_device_init (void ) {return 0 ; }static inline void tls_device_cleanup (void ) {}
@@ -165,8 +168,7 @@ static inline void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq) {} static inline int -tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx, - struct sk_buff *skb, struct strp_msg *rxm) +tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx) {return 0 ; }
net/tls/tls_device.c +16 -9
@@ -889,14 +889,19 @@ static void tls_device_core_ctrl_rx_resync(struct tls_context *tls_ctx, } } -static int tls_device_reencrypt (struct sock *sk,struct sk_buff *skb) +static int +tls_device_reencrypt (struct sock *sk,struct tls_sw_context_rx *sw_ctx) { -struct strp_msg *rxm =strp_msg (skb); -int err =0 , offset = rxm->offset, copy, nsg, data_len, pos; -struct sk_buff *skb_iter, *unused; +int err =0 , offset, copy, nsg, data_len, pos; +struct sk_buff *skb, *skb_iter, *unused;struct scatterlist sg[1 ]; +struct strp_msg *rxm;char *orig_buf, *buf; + skb =tls_strp_msg (sw_ctx); + rxm =strp_msg (skb); + offset = rxm->offset; + orig_buf =kmalloc (rxm->full_len +TLS_HEADER_SIZE +TLS_CIPHER_AES_GCM_128_IV_SIZE , sk->sk_allocation);if (!orig_buf)
@@ -919,7 +924,7 @@ static int tls_device_reencrypt(struct sock *sk, struct sk_buff *skb) goto free_buf;/* We are interested only in the decrypted data not the auth */ - err =decrypt_skb (sk, skb, sg); + err =decrypt_skb (sk, sg);if (err != -EBADMSG ) goto free_buf;else
@@ -974,10 +979,12 @@ static int tls_device_reencrypt(struct sock *sk, struct sk_buff *skb)return err; } -int tls_device_decrypted (struct sock *sk,struct tls_context *tls_ctx, -struct sk_buff *skb,struct strp_msg *rxm) +int tls_device_decrypted (struct sock *sk,struct tls_context *tls_ctx) {struct tls_offload_context_rx *ctx =tls_offload_ctx_rx (tls_ctx); +struct tls_sw_context_rx *sw_ctx =tls_sw_ctx_rx (tls_ctx); +struct sk_buff *skb =tls_strp_msg (sw_ctx); +struct strp_msg *rxm =strp_msg (skb);int is_decrypted = skb->decrypted;int is_encrypted = !is_decrypted;struct sk_buff *skb_iter;
@@ -1000,7 +1007,7 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx,* likely have initial fragments decrypted, and final ones not * decrypted. We need to reencrypt that single SKB. */ -return tls_device_reencrypt (sk, skb); +return tls_device_reencrypt (sk, sw_ctx); }/* Return immediately if the record is either entirely plaintext or
@@ -1017,7 +1024,7 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx, } ctx->resync_nh_reset =1 ; -return tls_device_reencrypt (sk, skb); +return tls_device_reencrypt (sk, sw_ctx); } static void tls_device_attach(struct tls_context *ctx, struct sock *sk,
net/tls/tls_sw.c +18 -19
@@ -1421,8 +1421,7 @@ static int tls_setup_from_iter(struct iov_iter *from,* NULL, then the decryption happens inside skb buffers itself, i.e. * zero-copy gets disabled and 'darg->zc' is updated. */ -static int tls_decrypt_sg (struct sock *sk,struct sk_buff *skb, -struct iov_iter *out_iov, +static int tls_decrypt_sg (struct sock *sk,struct iov_iter *out_iov,struct scatterlist *out_sg,struct tls_decrypt_arg *darg) {
@@ -1430,6 +1429,7 @@ static int tls_decrypt_sg(struct sock *sk, struct sk_buff *skb,struct tls_sw_context_rx *ctx =tls_sw_ctx_rx (tls_ctx);struct tls_prot_info *prot = &tls_ctx->prot_info;int n_sgin, n_sgout, aead_size, err, pages =0 ; +struct sk_buff *skb =tls_strp_msg (ctx);struct strp_msg *rxm =strp_msg (skb);struct tls_msg *tlm =tls_msg (skb);struct aead_request *aead_req;
@@ -1567,14 +1567,14 @@ static int tls_decrypt_sg(struct sock *sk, struct sk_buff *skb,static int tls_decrypt_device (struct sock *sk,struct tls_context *tls_ctx, -struct sk_buff *skb,struct tls_decrypt_arg *darg) +struct tls_decrypt_arg *darg) {int err;if (tls_ctx->rx_conf !=TLS_HW )return 0 ; - err =tls_device_decrypted (sk, tls_ctx, skb,strp_msg (skb)); + err =tls_device_decrypted (sk, tls_ctx);if (err <=0 )return err;
@@ -1583,22 +1583,22 @@ tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx,return 1 ; } -static int decrypt_skb_update (struct sock *sk,struct sk_buff *skb, -struct iov_iter *dest, -struct tls_decrypt_arg *darg) +static int tls_rx_one_record (struct sock *sk,struct iov_iter *dest, +struct tls_decrypt_arg *darg) {struct tls_context *tls_ctx =tls_get_ctx (sk); +struct tls_sw_context_rx *ctx =tls_sw_ctx_rx (tls_ctx);struct tls_prot_info *prot = &tls_ctx->prot_info; -struct strp_msg *rxm =strp_msg (skb); +struct strp_msg *rxm;int pad, err; - err =tls_decrypt_device (sk, tls_ctx, skb, darg); + err =tls_decrypt_device (sk, tls_ctx, darg);if (err <0 )return err;if (err) goto decrypt_done; - err =tls_decrypt_sg (sk, skb, dest,NULL , darg); + err =tls_decrypt_sg (sk, dest,NULL , darg);if (err <0 ) {if (err == -EBADMSG )TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSDECRYPTERROR );
@@ -1613,14 +1613,15 @@ static int decrypt_skb_update(struct sock *sk, struct sk_buff *skb,if (!darg->tail)TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSRXNOPADVIOL );TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSDECRYPTRETRY ); -return decrypt_skb_update (sk, skb, dest, darg); +return tls_rx_one_record (sk, dest, darg); } decrypt_done: - pad =tls_padding_length (prot, skb, darg); + pad =tls_padding_length (prot, ctx->recv_pkt, darg);if (pad <0 )return pad; + rxm =strp_msg (ctx->recv_pkt); rxm->full_len -= pad; rxm->offset += prot->prepend_size; rxm->full_len -= prot->overhead_size;
@@ -1630,12 +1631,11 @@ static int decrypt_skb_update(struct sock *sk, struct sk_buff *skb,return 0 ; } -int decrypt_skb (struct sock *sk,struct sk_buff *skb, -struct scatterlist *sgout) +int decrypt_skb (struct sock *sk,struct scatterlist *sgout) {struct tls_decrypt_arg darg = { .zc = true, }; -return tls_decrypt_sg (sk, skb,NULL , sgout, &darg); +return tls_decrypt_sg (sk,NULL , sgout, &darg); } static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,
@@ -1905,7 +1905,7 @@ int tls_sw_recvmsg(struct sock *sk,else darg.async = false; - err =decrypt_skb_update (sk, skb, &msg->msg_iter, &darg); + err =tls_rx_one_record (sk, &msg->msg_iter, &darg);if (err <0 ) {tls_err_abort (sk, -EBADMSG ); goto recv_end;
@@ -2058,14 +2058,13 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,if (err <=0 ) goto splice_read_end; - skb = ctx->recv_pkt; - - err =decrypt_skb_update (sk, skb,NULL , &darg); + err =tls_rx_one_record (sk,NULL , &darg);if (err <0 ) {tls_err_abort (sk, -EBADMSG ); goto splice_read_end; } + skb = ctx->recv_pkt;tls_rx_rec_done (ctx); }
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 7/11 tls: rx: return the decrypted skb via darg
In reply to Jakub KicinskiInstead of using ctx->recv_pkt after decryption read the skb from darg.skb. This moves the decision of what the "output skb" is to the decrypt handlers. For now after decrypt handler returns successfully ctx->recv_pkt is simply moved to darg.skb, but it will change soon.
Note that tls_decrypt_sg() cannot clear the ctx->recv_pkt because it gets called to re-encrypt (i.e. by the device offload). So we need an awkward temporary if() in tls_rx_one_record().
--- net/tls/tls_sw.c | 49 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 10 deletions(-)
net/tls/tls_sw.c +39 -10
@@ -47,9 +47,13 @@#include "tls.h" struct tls_decrypt_arg { + struct_group(inargs, bool zc;bool async;u8 tail; + ); + +struct sk_buff *skb; };struct tls_decrypt_ctx {
@@ -1412,6 +1416,7 @@ static int tls_setup_from_iter(struct iov_iter *from,* ------------------------------------------------------------------- * zc | Zero-copy decrypt allowed | Zero-copy performed * async | Async decrypt allowed | Async crypto used / in progress +* skb | * | Output skb */ /* This function decrypts the input skb into either out_iov or in out_sg
@@ -1551,12 +1556,17 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,/* Prepare and submit AEAD request */ err =tls_do_decryption (sk, skb, sgin, sgout, dctx->iv, data_len + prot->tail_size, aead_req, darg); +if (err) + goto exit_free_pages; + + darg->skb =tls_strp_msg (ctx);if (darg->async)return 0 ;if (prot->tail_size) darg->tail = dctx->tail; +exit_free_pages:/* Release the pages in case iov was mapped to pages */ for (; pages >0 ; pages--)put_page (sg_page (&sgout[pages]));
@@ -1569,6 +1579,7 @@ static int tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx, struct tls_decrypt_arg *darg) { + struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);int err;if (tls_ctx->rx_conf !=TLS_HW )
@@ -1580,6 +1591,8 @@ tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx, darg->zc = false; darg->async = false; + darg->skb =tls_strp_msg (ctx); + ctx->recv_pkt =NULL ;return 1 ; }
@@ -1604,8 +1617,11 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSDECRYPTERROR );return err; } -if (darg->async) +if (darg->async) { +if (darg->skb == ctx->recv_pkt) + ctx->recv_pkt =NULL ; goto decrypt_next; + }/* If opportunistic TLS 1.3 ZC failed retry without ZC */ if (unlikely (darg->zc && prot->version ==TLS_1_3_VERSION && darg->tail !=TLS_RECORD_TYPE_DATA )) {
@@ -1616,12 +1632,17 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,return tls_rx_one_record (sk, dest, darg); } +if (darg->skb == ctx->recv_pkt) + ctx->recv_pkt =NULL ; + decrypt_done: - pad =tls_padding_length (prot, ctx->recv_pkt, darg); -if (pad <0 ) + pad =tls_padding_length (prot, darg->skb, darg); +if (pad <0 ) { +consume_skb (darg->skb);return pad; + } - rxm =strp_msg (ctx->recv_pkt); + rxm =strp_msg (darg->skb); rxm->full_len -= pad; rxm->offset += prot->prepend_size; rxm->full_len -= prot->overhead_size;
@@ -1663,6 +1684,7 @@ static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,static void tls_rx_rec_done (struct tls_sw_context_rx *ctx) { +consume_skb (ctx->recv_pkt); ctx->recv_pkt =NULL ;__strp_unpause (&ctx->strp); }
@@ -1872,7 +1894,7 @@ int tls_sw_recvmsg(struct sock *sk, ctx->zc_capable; decrypted =0 ;while (len && (decrypted + copied < target || ctx->recv_pkt)) { -struct tls_decrypt_arg darg = {}; +struct tls_decrypt_arg darg;int to_decrypt, chunk; err =tls_rx_rec_wait (sk, psock, flags &MSG_DONTWAIT , timeo);
@@ -1889,9 +1911,10 @@ int tls_sw_recvmsg(struct sock *sk, goto recv_end; } - skb = ctx->recv_pkt; - rxm =strp_msg (skb); - tlm =tls_msg (skb); +memset (&darg.inargs,0 ,sizeof (darg.inargs)); + + rxm =strp_msg (ctx->recv_pkt); + tlm =tls_msg (ctx->recv_pkt); to_decrypt = rxm->full_len - prot->overhead_size;
@@ -1911,6 +1934,10 @@ int tls_sw_recvmsg(struct sock *sk, goto recv_end; } + skb = darg.skb; + rxm =strp_msg (skb); + tlm =tls_msg (skb); + async |= darg.async;/* If the type of records being processed is not known yet,
@@ -2051,21 +2078,23 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,if (!skb_queue_empty (&ctx->rx_list)) { skb =__skb_dequeue (&ctx->rx_list); }else { -struct tls_decrypt_arg darg = {}; +struct tls_decrypt_arg darg; err =tls_rx_rec_wait (sk,NULL , flags &SPLICE_F_NONBLOCK , timeo);if (err <=0 ) goto splice_read_end; +memset (&darg.inargs,0 ,sizeof (darg.inargs)); + err =tls_rx_one_record (sk,NULL , &darg);if (err <0 ) {tls_err_abort (sk, -EBADMSG ); goto splice_read_end; } - skb = ctx->recv_pkt;tls_rx_rec_done (ctx); + skb = darg.skb; } rxm =strp_msg (skb);
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 8/11 tls: rx: async: adjust record geometry immediately
In reply to Jakub KicinskiAsync crypto TLS Rx currently waits for crypto to be done in order to strip the TLS header and tailer. Simplify the code by moving the pointers immediately, since only TLS 1.2 is supported here there is no message padding.
This simplifies the decryption into a new skb in the next patch as we don't have to worry about input vs output skb in the decrypt_done() handler any more.
--- net/tls/tls_sw.c | 49 ++++++++++-------------------------------------- 1 file changed, 10 insertions(+), 39 deletions(-)
net/tls/tls_sw.c +10 -39
@@ -184,39 +184,22 @@ static void tls_decrypt_done(struct crypto_async_request *req, int err)struct scatterlist *sgin = aead_req->src;struct tls_sw_context_rx *ctx;struct tls_context *tls_ctx; -struct tls_prot_info *prot;struct scatterlist *sg; -struct sk_buff *skb;unsigned int pages; +struct sock *sk; - skb = (struct sk_buff *)req->data; - tls_ctx =tls_get_ctx (skb->sk); + sk = (struct sock *)req->data; + tls_ctx =tls_get_ctx (sk); ctx =tls_sw_ctx_rx (tls_ctx); - prot = &tls_ctx->prot_info;/* Propagate if there was an err */ if (err) {if (err == -EBADMSG ) -TLS_INC_STATS (sock_net (skb->sk), -LINUX_MIB_TLSDECRYPTERROR ); +TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSDECRYPTERROR ); ctx->async_wait.err = err; -tls_err_abort (skb->sk, err); - }else { -struct strp_msg *rxm =strp_msg (skb); - -/* No TLS 1.3 support with async crypto */ -WARN_ON (prot->tail_size); - - rxm->offset += prot->prepend_size; - rxm->full_len -= prot->overhead_size; +tls_err_abort (sk, err); } -/* After using skb->sk to propagate sk through crypto async callback -* we need to NULL it again. -*/ - skb->sk =NULL ; - -/* Free the destination pages if skb was not decrypted inplace */ if (sgout != sgin) {/* Skip the first S/G entry as it points to AAD */
@@ -236,7 +219,6 @@ static void tls_decrypt_done(struct crypto_async_request *req, int err) } static int tls_do_decryption(struct sock *sk, - struct sk_buff *skb, struct scatterlist *sgin, struct scatterlist *sgout, char *iv_recv,
@@ -256,16 +238,9 @@ static int tls_do_decryption(struct sock *sk, (u8 *)iv_recv);if (darg->async) { -/* Using skb->sk to push sk through to crypto async callback -* handler. This allows propagating errors up to the socket -* if needed. It _must_ be cleared in the async handler -* before consume_skb is called. We _know_ skb->sk is NULL -* because it is a clone from strparser. -*/ - skb->sk = sk;aead_request_set_callback (aead_req,CRYPTO_TFM_REQ_MAY_BACKLOG , - tls_decrypt_done, skb); + tls_decrypt_done, sk);atomic_inc (&ctx->decrypt_pending); }else { aead_request_set_callback(aead_req,
@@ -1554,7 +1529,7 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, }/* Prepare and submit AEAD request */ - err =tls_do_decryption (sk, skb, sgin, sgout, dctx->iv, + err =tls_do_decryption (sk, sgin, sgout, dctx->iv, data_len + prot->tail_size, aead_req, darg);if (err) goto exit_free_pages;
@@ -1617,11 +1592,8 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,TLS_INC_STATS (sock_net (sk),LINUX_MIB_TLSDECRYPTERROR );return err; } -if (darg->async) { -if (darg->skb == ctx->recv_pkt) - ctx->recv_pkt =NULL ; - goto decrypt_next; - } +if (darg->async) + goto decrypt_done;/* If opportunistic TLS 1.3 ZC failed retry without ZC */ if (unlikely (darg->zc && prot->version ==TLS_1_3_VERSION && darg->tail !=TLS_RECORD_TYPE_DATA )) {
@@ -1632,10 +1604,10 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,return tls_rx_one_record (sk, dest, darg); } +decrypt_done:if (darg->skb == ctx->recv_pkt) ctx->recv_pkt =NULL ; -decrypt_done: pad =tls_padding_length (prot, darg->skb, darg);if (pad <0 ) {consume_skb (darg->skb);
@@ -1646,7 +1618,6 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest, rxm->full_len -= pad; rxm->offset += prot->prepend_size; rxm->full_len -= prot->overhead_size; -decrypt_next:tls_advance_record_sn (sk, prot, &tls_ctx->rx);return 0 ;
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 9/11 tls: rx: async: hold onto the input skb
In reply to Jakub KicinskiAsync crypto currently benefits from the fact that we decrypt in place. When we allow input and output to be different skbs we will have to hang onto the input while we move to the next record. Clone the inputs and keep them on a list.
--- include/net/tls.h | 1 + net/tls/Makefile | 2 +- net/tls/tls.h | 3 +++ net/tls/tls_strp.c | 17 +++++++++++++++++ net/tls/tls_sw.c | 26 +++++++++++++++++--------- 5 files changed, 39 insertions(+), 10 deletions(-) create mode 100644 net/tls/tls_strp.c
include/net/tls.h +1 -0
@@ -123,6 +123,7 @@ struct tls_sw_context_rx {atomic_t decrypt_pending;/* protect crypto_wait with decrypt_pending*/ spinlock_t decrypt_compl_lock; +struct sk_buff_head async_hold;struct wait_queue_head wq; };
net/tls/Makefile +1 -1
@@ -7,7 +7,7 @@ CFLAGS_trace.o := -I$(src) obj-$(CONFIG_TLS) += tls.o -tls-y := tls_main.o tls_sw.o tls_proc.o trace.o +tls-y := tls_main.o tls_sw.o tls_proc.o trace.o tls_strp.o tls-$(CONFIG_TLS_TOE) += tls_toe.o tls-$(CONFIG_TLS_DEVICE) += tls_device.o tls_device_fallback.o
net/tls/tls.h +3 -0
@@ -124,6 +124,9 @@ int tls_sw_fallback_init(struct sock *sk,struct tls_offload_context_tx *offload_ctx, struct tls_crypto_info *crypto_info); +int tls_strp_msg_hold (struct sock *sk,struct sk_buff *skb, +struct sk_buff_head *dst); +static inline struct tls_msg *tls_msg (struct sk_buff *skb) {struct sk_skb_cb *scb = (struct sk_skb_cb *)skb->cb;
net/tls/tls_strp.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include <linux/skbuff.h> + +#include "tls.h" + +int tls_strp_msg_hold (struct sock *sk,struct sk_buff *skb, +struct sk_buff_head *dst) +{ +struct sk_buff *clone; + + clone =skb_clone (skb, sk->sk_allocation); +if (!clone) +return -ENOMEM ; +__skb_queue_tail (dst, clone); +return 0 ; +}
net/tls/tls_sw.c +17 -9
@@ -1535,8 +1535,13 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov, goto exit_free_pages; darg->skb =tls_strp_msg (ctx); -if (darg->async) -return 0 ; + +if (unlikely (darg->async)) { + err =tls_strp_msg_hold (sk, skb, &ctx->async_hold); +if (err) +__skb_queue_tail (&ctx->async_hold, darg->skb); +return err; + }if (prot->tail_size) darg->tail = dctx->tail;
@@ -1998,14 +2003,16 @@ int tls_sw_recvmsg(struct sock *sk,reinit_completion (&ctx->async_wait.completion); pending =atomic_read (&ctx->decrypt_pending);spin_unlock_bh (&ctx->decrypt_compl_lock); -if (pending) { + ret =0 ; +if (pending) ret =crypto_wait_req (-EINPROGRESS , &ctx->async_wait); -if (ret) { -if (err >=0 || err == -EINPROGRESS ) - err = ret; - decrypted =0 ; - goto end; - } +__skb_queue_purge (&ctx->async_hold); + +if (ret) { +if (err >=0 || err == -EINPROGRESS ) + err = ret; + decrypted =0 ; + goto end; }/* Drain records from the rx_list & copy if required */
@@ -2440,6 +2447,7 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) crypto_info = &ctx->crypto_recv.info; cctx = &ctx->rx;skb_queue_head_init (&sw_ctx_rx->rx_list); +skb_queue_head_init (&sw_ctx_rx->async_hold); aead = &sw_ctx_rx->aead_recv; }
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 10/11 tls: rx: async: don't put async zc on the list
In reply to Jakub KicinskiThe "zero-copy" path in SW TLS will engage either for no skbs or for all but last. If the recvmsg parameters are right and the socket can do ZC we'll ZC until the iterator can't fit a full record at which point we'll decrypt one more record and copy over the necessary bits to fill up the request.
The only reason we hold onto the ZC skbs which went thru the async path until the end of recvmsg() is to count bytes. We need an accurate count of zc'ed bytes so that we can calculate how much of the non-zc'd data to copy. To allow freeing input skbs on the ZC path count only how much of the list we'll need to consume.
--- net/tls/tls_sw.c | 40 +++++++++++++++++++--------------------- 1 file changed, 19 insertions(+), 21 deletions(-)
net/tls/tls_sw.c +19 -21
@@ -1675,7 +1675,6 @@ static int process_rx_list(struct tls_sw_context_rx *ctx,u8 *control, size_t skip, size_t len, - bool zc, bool is_peek) { struct sk_buff *skb = skb_peek(&ctx->rx_list);
@@ -1709,12 +1708,10 @@ static int process_rx_list(struct tls_sw_context_rx *ctx,if (err <=0 ) goto out; -if (!zc || (rxm->full_len - skip) > len) { - err =skb_copy_datagram_msg (skb, rxm->offset + skip, - msg, chunk); -if (err <0 ) - goto out; - } + err =skb_copy_datagram_msg (skb, rxm->offset + skip, + msg, chunk); +if (err <0 ) + goto out; len = len - chunk; copied = copied + chunk;
@@ -1824,9 +1821,9 @@ int tls_sw_recvmsg(struct sock *sk,struct tls_context *tls_ctx =tls_get_ctx (sk);struct tls_sw_context_rx *ctx =tls_sw_ctx_rx (tls_ctx);struct tls_prot_info *prot = &tls_ctx->prot_info; +ssize_t decrypted =0 , async_copy_bytes =0 ;struct sk_psock *psock;unsigned char control =0 ; -ssize_t decrypted =0 ;size_t flushed_at =0 ;struct strp_msg *rxm;struct tls_msg *tlm;
@@ -1855,7 +1852,7 @@ int tls_sw_recvmsg(struct sock *sk, goto end;/* Process pending decrypted records. It must be non-zero-copy */ - err =process_rx_list (ctx, msg, &control,0 , len, false, is_peek); + err =process_rx_list (ctx, msg, &control,0 , len, is_peek);if (err <0 ) goto end;
@@ -1939,19 +1936,20 @@ int tls_sw_recvmsg(struct sock *sk, chunk = rxm->full_len;tls_rx_rec_done (ctx); -if (async) { -/* TLS 1.2-only, to_decrypt must be text length */ - chunk =min_t (int, to_decrypt, len); -put_on_rx_list: - decrypted += chunk; - len -= chunk; -__skb_queue_tail (&ctx->rx_list, skb); -continue ; - } -if (!darg.zc) {bool partially_consumed = chunk > len; +if (async) { +/* TLS 1.2-only, to_decrypt must be text len */ + chunk =min_t (int, to_decrypt, len); + async_copy_bytes += chunk; +put_on_rx_list: + decrypted += chunk; + len -= chunk; +__skb_queue_tail (&ctx->rx_list, skb); +continue ; + } +if (bpf_strp_enabled) { err =sk_psock_tls_strp_read (psock, skb);if (err != __SK_PASS) {
@@ -2018,10 +2016,10 @@ int tls_sw_recvmsg(struct sock *sk,/* Drain records from the rx_list & copy if required */ if (is_peek || is_kvec) err =process_rx_list (ctx, msg, &control, copied, - decrypted, false, is_peek); + decrypted, is_peek);else err =process_rx_list (ctx, msg, &control,0 , - decrypted, true, is_peek); + async_copy_bytes, is_peek); decrypted =max (err,0 ); }
Signature · 1 line
2.36.1
Jakub Kicinski
maintainerPATCH v2 11/11 tls: rx: decrypt into a fresh skb
In reply to Jakub KicinskiWe currently CoW Rx skbs whenever we can't decrypt to a user space buffer. The skbs can be enormous (64kB) and CoW does a linear alloc which has a strong chance of failing under memory pressure. Or even without, skb_cow_data() assumes GFP_ATOMIC.
Allocate a new frag'd skb and decrypt into it. We finally take advantage of the decrypted skb getting returned via darg.
--- net/tls/tls.h | 3 ++ net/tls/tls_sw.c | 106 ++++++++++++++++++++++++++++++----------------- 2 files changed, 72 insertions(+), 37 deletions(-)
net/tls/tls.h +3 -0
@@ -39,6 +39,9 @@#include <linux/skmsg.h> #include <net/tls.h> +#define TLS_PAGE_ORDER (min_t(unsigned int, PAGE_ALLOC_COSTLY_ORDER, \ + TLS_MAX_PAYLOAD_SIZE >> PAGE_SHIFT)) +#define __TLS_INC_STATS (net, field) \ __SNMP_INC_STATS((net)->mib.tls_statistics, field)#define TLS_INC_STATS (net, field) \
net/tls/tls_sw.c +69 -37
163 lines not on this page · raw message
Signature · 1 line
2.36.1
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Quoted 10 lines · On Thu, 14 Jul 2022 22:22:24 -0700 you wrote:
TLS calls
skb_cow_data()on the skb it received from strparser whenever it needs to hold onto the skb with the decrypted data. (The alternative being decrypting directly to a user space buffer in whic case the input skb doesn't get modified or used after.) TLS needs the decrypted skb:
- almost always with TLS 1.3 (unless the new NoPad is enabled);
- when user space buffer is too small to fit the record;
- when BPF sockmap is enabled.[...]
Here is the summary with links:
- [net-next,v2,01/11] tls: rx: allow only one reader at a time
https://git.kernel.org/netdev/net-next/c/4cbc325ed6b4 - [net-next,v2,02/11] tls: rx: don't try to keep the skbs always on the list https://git.kernel.org/netdev/net-next/c/008141de8557 - [net-next,v2,03/11] tls: rx: don't keep decrypted skbs on ctx->recv_pkt https://git.kernel.org/netdev/net-next/c/abb47dc95dc6 - [net-next,v2,04/11] tls: rx: remove the message decrypted tracking https://git.kernel.org/netdev/net-next/c/53d57999fe02 - [net-next,v2,05/11] tls: rx: factor out device darg update https://git.kernel.org/netdev/net-next/c/8a958732818b - [net-next,v2,06/11] tls: rx: read the input skb from ctx->recv_pkt https://git.kernel.org/netdev/net-next/c/541cc48be3b1 - [net-next,v2,07/11] tls: rx: return the decrypted skb via darg https://git.kernel.org/netdev/net-next/c/6bd116c8c654 - [net-next,v2,08/11] tls: rx: async: adjust record geometry immediately https://git.kernel.org/netdev/net-next/c/6ececdc51369 - [net-next,v2,09/11] tls: rx: async: hold onto the input skb https://git.kernel.org/netdev/net-next/c/c618db2afe7c - [net-next,v2,10/11] tls: rx: async: don't put async zc on the list https://git.kernel.org/netdev/net-next/c/cbbdee9918a2 - [net-next,v2,11/11] tls: rx: decrypt into a fresh skb https://git.kernel.org/netdev/net-next/c/fd31f3996af2
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html