Skip to contentSkip to conversation

TCP Extra-option framework for TCP-MD5 and SMC

RFC In review Christoph Paasch 2 participants 46 messages Dec 11, 2017 – Dec 15, 2017 Page 1 of 2

Subsystems (6)
lore
Share & tools
lore

Christoph Paasch

author
#0 lore raw

v3 cover TCP Extra-option framework for TCP-MD5 and SMC

3rd version of the set.

Now, fully tested and bugs fixed.

There are a two rather big changes (most of it is commented on in the notes):

* Made the tcp_option_list and RCU-protected hlist. We need RCU-protection because listener sockets are lockless nowadays.
* Previously, in tcp_v4_send_reset, we would lookup the MD5-signature and if there was one, go look for the listening-socket (if sk == NULL) to see if there is an MD5-key.
Nowadays, we can't do this anymore because extra-options are only parsed when we actually have a socket. However, this shouldn't have an impact as far as I can see. Because, if there is a listening-socket, __inet_lookup_skb will match on it and when we then send a RST when coming from tcp_rcv_state_process we have the pointer to the listening socket.
The only scenario wherer tcp_v4_send_reset() gets called with a NULL sk is when coming straight from tcp_v4_rcv.

If there are no major comments, I would like to go to netdev as a next step.

Christoph Paasch (14):
tcp md5sig: Use skb's saddr when replying to an incoming segment
tcp: Write options after the header has been fully done
tcp: Pass sock and skb to tcp_options_write
tcp: Allow tcp_fast_parse_options to drop segments
tcp: Make smc_parse_options return 1 on success
tcp_smc: Make SMC use TCP extra-option framework
tcp_md5: Don't pass along md5-key
tcp_md5: Detect key inside tcp_v4_send_ack instead of passing it as an
argument
tcp_md5: Detect key inside tcp_v6_send_response instead of passing it
as an argument
tcp_md5: Check for TCP_MD5 after TCP Timestamps in
tcp_established_options
tcp_md5: Move TCP-MD5 code out of TCP itself
tcp_md5: Use tcp_extra_options in output path
tcp_md5: Cleanup TCP-code
tcp_md5: Use TCP extra-options on the input path

Mat Martineau (1):
tcp: Register handlers for extra TCP options

 drivers/infiniband/hw/cxgb4/cm.c |    2 +-
 include/linux/inet_diag.h        |    1 +
 include/linux/tcp.h              |   43 +-
 include/linux/tcp_md5.h          |   39 ++
 include/net/inet_sock.h          |    3 +-
 include/net/tcp.h                |  209 +++---
 net/ipv4/Makefile                |    1 +
 net/ipv4/syncookies.c            |    6 +-
 net/ipv4/tcp.c                   |  399 +++++++++---
 net/ipv4/tcp_diag.c              |   81 +--
 net/ipv4/tcp_input.c             |  140 ++--
 net/ipv4/tcp_ipv4.c              |  545 ++--------------
 net/ipv4/tcp_md5.c               | 1340 ++++++++++++++++++++++++++++++++++++++
 net/ipv4/tcp_minisocks.c         |   75 +--
 net/ipv4/tcp_output.c            |  179 +----
 net/ipv6/syncookies.c            |    6 +-
 net/ipv6/tcp_ipv6.c              |  382 +----------
 net/smc/af_smc.c                 |  175 ++++-
 18 files changed, 2181 insertions(+), 1445 deletions(-)
 create mode 100644 include/linux/tcp_md5.h
 create mode 100644 net/ipv4/tcp_md5.c
Signature · 1 line
2.15.0
author
#1 lore raw

RFC v3 1/15 tcp md5sig: Use skb's saddr when replying to an incoming segment

The MD5-key that belongs to a connection is identified by the peer's IP-address. When we are in tcp_v4(6)_reqsk_send_ack(), we are replying to an incoming segment from tcp_check_req() that failed the seq-number checks.

Thus, to find the correct key, we need to use the skb's saddr and not the daddr.

This bug seems to have been there since quite a while, but probably got unnoticed because the consequences are not catastrophic. We will call tcp_v4_reqsk_send_ack only to send a challenge-ACK back to the peer, thus the connection doesn't really fail.

Fixes 9501f9722922 ("tcp md5sig: Let the caller pass appropriate key for tcp_v{4,6}_do_calc_md5_hash().")
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>
---
 net/ipv4/tcp_ipv4.c | 2 +-
 net/ipv6/tcp_ipv6.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
net/ipv4/tcp_ipv4.c +1 -1
index 77ea45da0fe9..94e28350f420 100644
@@ -848,7 +848,7 @@ static void tcp_v4_reqsk_send_ack(const struct sock *sk, struct sk_buff *skb,
 			tcp_time_stamp_raw() + tcp_rsk(req)->ts_off,
 			req->ts_recent,
 			0,
-			tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&ip_hdr(skb)->daddr,
+			tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&ip_hdr(skb)->saddr,
 					  AF_INET),
 			inet_rsk(req)->no_srccheck ? IP_REPLY_ARG_NOSRCCHECK : 0,
 			ip_hdr(skb)->tos);
net/ipv6/tcp_ipv6.c +1 -1
index 1f04ec0e4a7a..7178476b3d2f 100644
@@ -994,7 +994,7 @@ static void tcp_v6_reqsk_send_ack(const struct sock *sk, struct sk_buff *skb,
 			req->rsk_rcv_wnd >> inet_rsk(req)->rcv_wscale,
 			tcp_time_stamp_raw() + tcp_rsk(req)->ts_off,
 			req->ts_recent, sk->sk_bound_dev_if,
-			tcp_v6_md5_do_lookup(sk, &ipv6_hdr(skb)->daddr),
+			tcp_v6_md5_do_lookup(sk, &ipv6_hdr(skb)->saddr),
 			0, 0);
 }
 
Signature · 1 line
2.15.0
#2 lore raw

Re: [MPTCP] [RFC v3 01/15] tcp md5sig: Use skb's saddr when replying to an incoming segment

Quoted 8 lines · Christoph Paasch

The MD5-key that belongs to a connection is identified by the peer's IP-address. When we are in tcp_v4(6)_reqsk_send_ack(), we are replying to an incoming segment from tcp_check_req() that failed the seq-number checks.

Thus, to find the correct key, we need to use the skb's saddr and not the daddr.

Quoted 7 lines

This bug seems to have been there since quite a while, but probably got unnoticed because the consequences are not catastrophic. We will call tcp_v4_reqsk_send_ack only to send a challenge-ACK back to the peer, thus the connection doesn't really fail.

Fixes 9501f9722922 ("tcp md5sig: Let the caller pass appropriate key for tcp_v{4,6}_do_calc_md5_hash().")
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

I saw that this was merged on netdev already. Thumbs up!

--
Mat Martineau
Intel OTC

author
#3 lore raw

Re: [MPTCP] [RFC v3 01/15] tcp md5sig: Use skb's saddr when replying to an incoming segment

Quoted 16 lines · Mat Martineau
Quoted 13 lines · Christoph Paasch

The MD5-key that belongs to a connection is identified by the peer's IP-address. When we are in tcp_v4(6)_reqsk_send_ack(), we are replying to an incoming segment from tcp_check_req() that failed the seq-number checks.

Thus, to find the correct key, we need to use the skb's saddr and not the daddr.

This bug seems to have been there since quite a while, but probably got unnoticed because the consequences are not catastrophic. We will call tcp_v4_reqsk_send_ack only to send a challenge-ACK back to the peer, thus the connection doesn't really fail.

Quoted 4 lines
Quoted 2 lines
Fixes 9501f9722922 ("tcp md5sig: Let the caller pass appropriate key for tcp_v{4,6}_do_calc_md5_hash().")
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

I saw that this was merged on netdev already. Thumbs up!

Yeah, quite a nasty bug :)

Christoph

author
#4 lore raw

RFC v3 2/15 tcp: Write options after the header has been fully done

In reply to Christoph Paasch

This is needed later on for TCP_MD5, as we need the complete TCP-header to compute the TCP_MD5-hash.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>
---
 net/ipv4/tcp_output.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/ipv4/tcp_output.c +2 -2
index a4d214c7b506..e0335be600d3 100644
@@ -1126,7 +1126,6 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
 		}
 	}
 
-	tcp_options_write((__be32 *)(th + 1), tp, &opts);
 	skb_shinfo(skb)->gso_type = sk->sk_gso_type;
 	if (likely(!(tcb->tcp_flags & TCPHDR_SYN))) {
 		th->window      = htons(tcp_select_window(sk));
@@ -1137,6 +1136,7 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
 		 */
 		th->window	= htons(min(tp->rcv_wnd, 65535U));
 	}
+	tcp_options_write((__be32 *)(th + 1), tp, &opts);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Calculate the MD5 hash, as we have all we need now */
 	if (md5) {
@@ -3245,8 +3245,8 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
 
 	/* RFC1323: The window in SYN & SYN/ACK segments is never scaled. */
 	th->window = htons(min(req->rsk_rcv_wnd, 65535U));
-	tcp_options_write((__be32 *)(th + 1), NULL, &opts);
 	th->doff = (tcp_header_size >> 2);
+	tcp_options_write((__be32 *)(th + 1), NULL, &opts);
 	__TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTSEGS);
 
 #ifdef CONFIG_TCP_MD5SIG
Signature · 1 line
2.15.0
author
#5 lore raw

RFC v3 3/15 tcp: Pass sock and skb to tcp_options_write

In reply to Christoph Paasch

Because, to write the extra-options, we need the request-socket to parse the list of extra-options to be written to the TCP-header.
The skb is also needed so that extra-options like TCP_MD5SIG can compute the hash properly.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged with next commit to make it to reduce the number of commits.

 net/ipv4/tcp_output.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
net/ipv4/tcp_output.c +7 -3
index e0335be600d3..751396f22bb3 100644
@@ -444,10 +444,14 @@ struct tcp_out_options {
  * At least SACK_PERM as the first option is known to lead to a disaster
  * (but it may well be that other scenarios fail similarly).
  */
-static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
+static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 			      struct tcp_out_options *opts)
 {
 	u16 options = opts->options;	/* mungable copy */
+	struct tcp_sock *tp = NULL;
+
+	if (sk_fullsock(sk))
+		tp = tcp_sk(sk);
 
 	if (unlikely(OPTION_MD5 & options)) {
 		*ptr++ = htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16) |
@@ -1136,7 +1140,7 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
 		 */
 		th->window	= htons(min(tp->rcv_wnd, 65535U));
 	}
-	tcp_options_write((__be32 *)(th + 1), tp, &opts);
+	tcp_options_write((__be32 *)(th + 1), skb, sk, &opts);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Calculate the MD5 hash, as we have all we need now */
 	if (md5) {
@@ -3246,7 +3250,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
 	/* RFC1323: The window in SYN & SYN/ACK segments is never scaled. */
 	th->window = htons(min(req->rsk_rcv_wnd, 65535U));
 	th->doff = (tcp_header_size >> 2);
-	tcp_options_write((__be32 *)(th + 1), NULL, &opts);
+	tcp_options_write((__be32 *)(th + 1), skb, req_to_sk(req), &opts);
 	__TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTSEGS);
 
 #ifdef CONFIG_TCP_MD5SIG
Signature · 1 line
2.15.0
#6 lore raw

Re: [MPTCP] [RFC v3 03/15] tcp: Pass sock and skb to tcp_options_write

Quoted 4 lines · Christoph Paasch

Because, to write the extra-options, we need the request-socket to parse the list of extra-options to be written to the TCP-header.
The skb is also needed so that extra-options like TCP_MD5SIG can compute the hash properly.

Without reading ahead in the patch set, people won't know what "extra-options" are. A little more context would help:

"""

An upcoming patch adds a configurable, per-socket list of TCP options to populate in the TCP header. This requires tcp_options_write() to know the socket (to use the options list) and the skb (to provide visibility to the packet data for options like TCP_MD5SIG).

"""

Mat

Quoted 52 lines
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged with next commit to make it to reduce the number of commits.

net/ipv4/tcp_output.c | 10 +++++++---

1 file changed, 7 insertions(+), 3 deletions(-)
net/ipv4/tcp_output.c +7 -3
index e0335be600d3..751396f22bb3 100644
@@ -444,10 +444,14 @@ struct tcp_out_options {
 * At least SACK_PERM as the first option is known to lead to a disaster
 * (but it may well be that other scenarios fail similarly).
 */
-static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
+static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 			      struct tcp_out_options *opts)
 {
 	u16 options = opts->options;	/* mungable copy */
+	struct tcp_sock *tp = NULL;
+
+	if (sk_fullsock(sk))
+		tp = tcp_sk(sk);
 
 	if (unlikely(OPTION_MD5 & options)) {
 		*ptr++ = htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16) |
@@ -1136,7 +1140,7 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
 		 */
 		th->window	= htons(min(tp->rcv_wnd, 65535U));
 	}
-	tcp_options_write((__be32 *)(th + 1), tp, &opts);
+	tcp_options_write((__be32 *)(th + 1), skb, sk, &opts);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Calculate the MD5 hash, as we have all we need now */
 	if (md5) {
@@ -3246,7 +3250,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
 	/* RFC1323: The window in SYN & SYN/ACK segments is never scaled. */
 	th->window = htons(min(req->rsk_rcv_wnd, 65535U));
 	th->doff = (tcp_header_size >> 2);
-	tcp_options_write((__be32 *)(th + 1), NULL, &opts);
+	tcp_options_write((__be32 *)(th + 1), skb, req_to_sk(req), &opts);
 	__TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTSEGS);
 
 #ifdef CONFIG_TCP_MD5SIG
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#7 lore raw

Re: [MPTCP] [RFC v3 03/15] tcp: Pass sock and skb to tcp_options_write

Quoted 11 lines · Mat Martineau
Quoted 4 lines · Christoph Paasch

Because, to write the extra-options, we need the request-socket to parse the list of extra-options to be written to the TCP-header.
The skb is also needed so that extra-options like TCP_MD5SIG can compute the hash properly.

Without reading ahead in the patch set, people won't know what "extra-options" are. A little more context would help:

Quoted 8 lines

"""

An upcoming patch adds a configurable, per-socket list of TCP options to populate in the TCP header. This requires tcp_options_write() to know the socket (to use the options list) and the skb (to provide visibility to the packet data for options like TCP_MD5SIG).

"""

Thanks, updated the commit-message.

Christoph

author
#8 lore raw

RFC v3 4/15 tcp: Allow tcp_fast_parse_options to drop segments

In reply to Christoph Paasch

The TCP extra-options parser might decide to drop the segment based on the present option. Especially, for TCP_MD5, we want to do this as soon as possible (thus, right after the parsing).

This patch allows to do this by splitting out tcp_fast_parse_options() in tcp_validate_incoming().

Originally, tcp_fast_parse_options() returned false, when there is no timestamp option (except in the case of the slow-path processing through tcp_parse_options() where it always returns true.

So, the return-value of tcp_fast_parse_options() was kind of inconsistent. With this patch, we make it return true when the segment should get dropped based on the parsed options, and false otherwise.

In tcp_validate_incoming, we will then just check for tp->rx_opt.saw_tstamp to see if we should verify PAWS.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Improved commit-message

 net/ipv4/tcp_input.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
net/ipv4/tcp_input.c +11 -5
index 9550cc42de2d..cba194a0222f 100644
@@ -3832,6 +3832,8 @@ static bool tcp_parse_aligned_timestamp(struct tcp_sock *tp, const struct tcphdr
 
 /* Fast parse options. This hopes to only see timestamps.
  * If it is wrong it falls back on tcp_parse_options().
+ *
+ * Returns true if we should drop this packet based on present TCP-options.
  */
 static bool tcp_fast_parse_options(const struct net *net,
 				   const struct sk_buff *skb,
@@ -3842,18 +3844,20 @@ static bool tcp_fast_parse_options(const struct net *net,
 	 */
 	if (th->doff == (sizeof(*th) / 4)) {
 		tp->rx_opt.saw_tstamp = 0;
-		return false;
+		goto extra_opt_check;
 	} else if (tp->rx_opt.tstamp_ok &&
 		   th->doff == ((sizeof(*th) + TCPOLEN_TSTAMP_ALIGNED) / 4)) {
 		if (tcp_parse_aligned_timestamp(tp, th))
-			return true;
+			goto extra_opt_check;
 	}
 
 	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
-	return true;
+extra_opt_check:
+	/* ToDo - will be added here */
+	return false;
 }
 
 #ifdef CONFIG_TCP_MD5SIG
@@ -5173,9 +5177,11 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
 	struct tcp_sock *tp = tcp_sk(sk);
 	bool rst_seq_match = false;
 
+	if (tcp_fast_parse_options(sock_net(sk), skb, th, tp))
+		goto discard;
+
 	/* RFC1323: H1. Apply PAWS check first. */
-	if (tcp_fast_parse_options(sock_net(sk), skb, th, tp) &&
-	    tp->rx_opt.saw_tstamp &&
+	if (tp->rx_opt.saw_tstamp &&
 	    tcp_paws_discard(sk, skb)) {
 		if (!th->rst) {
 			NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSESTABREJECTED);
Signature · 1 line
2.15.0
#9 lore raw

Re: [MPTCP] [RFC v3 04/15] tcp: Allow tcp_fast_parse_options to drop segments

Quoted 10 lines · Christoph Paasch

The TCP extra-options parser might decide to drop the segment based on the present option. Especially, for TCP_MD5, we want to do this as soon as possible (thus, right after the parsing).

This patch allows to do this by splitting out tcp_fast_parse_options() in tcp_validate_incoming().

Originally, tcp_fast_parse_options() returned false, when there is no timestamp option (except in the case of the slow-path processing through tcp_parse_options() where it always returns true.

Minor nitpick: unmatched paren above.

Quoted 41 lines

So, the return-value of tcp_fast_parse_options() was kind of inconsistent. With this patch, we make it return true when the segment should get dropped based on the parsed options, and false otherwise.

In tcp_validate_incoming, we will then just check for tp->rx_opt.saw_tstamp to see if we should verify PAWS.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Improved commit-message

net/ipv4/tcp_input.c | 16 +++++++++++-----

1 file changed, 11 insertions(+), 5 deletions(-)
net/ipv4/tcp_input.c +4 -2
index 9550cc42de2d..cba194a0222f 100644
@@ -3832,6 +3832,8 @@ static bool tcp_parse_aligned_timestamp(struct tcp_sock *tp, const struct tcphdr
 
 /* Fast parse options. This hopes to only see timestamps.
 * If it is wrong it falls back on tcp_parse_options().
+ *
+ * Returns true if we should drop this packet based on present TCP-options.
 */
 static bool tcp_fast_parse_options(const struct net *net,
 				   const struct sk_buff *skb,
@@ -3842,18 +3844,20 @@ static bool tcp_fast_parse_options(const struct net *net,
 	 */
 	if (th->doff == (sizeof(*th) / 4)) {
 		tp->rx_opt.saw_tstamp = 0;
-		return false;
+		goto extra_opt_check;
 	} else if (tp->rx_opt.tstamp_ok &&
 		   th->doff == ((sizeof(*th) + TCPOLEN_TSTAMP_ALIGNED) / 4)) {
 		if (tcp_parse_aligned_timestamp(tp, th))
-			return true;
+			goto extra_opt_check;

net/ipv4/tcp_input.c static bool tcp_fast_parse_options(const struct net *net, · line 3860

Quoted 10 lines
net/ipv4/tcp_input.c +3 -1
 	}
 
 	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
-	return true;
+extra_opt_check:
+	/* ToDo - will be added here */
+	return false;

I'm not sure the "ToDo" will be liked - it would be more of a standalone change to wait until the option registration patch to add the gotos. The main consideration is that git bisect works fine either way, so it's not a big deal to leave this as-is and see if anyone objects.

Mat

Quoted 21 lines
net/ipv4/tcp_input.c +4 -2
 }
 
 #ifdef CONFIG_TCP_MD5SIG
@@ -5173,9 +5177,11 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
 	struct tcp_sock *tp = tcp_sk(sk);
 	bool rst_seq_match = false;
 
+	if (tcp_fast_parse_options(sock_net(sk), skb, th, tp))
+		goto discard;
+
 	/* RFC1323: H1. Apply PAWS check first. */
-	if (tcp_fast_parse_options(sock_net(sk), skb, th, tp) &&
-	    tp->rx_opt.saw_tstamp &&
+	if (tp->rx_opt.saw_tstamp &&
 	    tcp_paws_discard(sk, skb)) {
 		if (!th->rst) {
 			NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSESTABREJECTED);
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#10 lore raw

Re: [MPTCP] [RFC v3 04/15] tcp: Allow tcp_fast_parse_options to drop segments

Quoted 7 lines · Mat Martineau
Quoted 4 lines · Christoph Paasch

The TCP extra-options parser might decide to drop the segment based on the present option. Especially, for TCP_MD5, we want to do this as soon as possible (thus, right after the parsing).

Quoted 8 lines
Quoted 6 lines

This patch allows to do this by splitting out tcp_fast_parse_options() in tcp_validate_incoming().

Originally, tcp_fast_parse_options() returned false, when there is no timestamp option (except in the case of the slow-path processing through tcp_parse_options() where it always returns true.

Minor nitpick: unmatched paren above.

Thanks, I fixed this.

Quoted 48 lines
Quoted 48 lines

So, the return-value of tcp_fast_parse_options() was kind of inconsistent. With this patch, we make it return true when the segment should get dropped based on the parsed options, and false otherwise.

In tcp_validate_incoming, we will then just check for tp->rx_opt.saw_tstamp to see if we should verify PAWS.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Improved commit-message

net/ipv4/tcp_input.c | 16 +++++++++++-----

1 file changed, 11 insertions(+), 5 deletions(-)
net/ipv4/tcp_input.c +4 -2
index 9550cc42de2d..cba194a0222f 100644
@@ -3832,6 +3832,8 @@ static bool tcp_parse_aligned_timestamp(struct tcp_sock *tp, const struct tcphdr
 
 /* Fast parse options. This hopes to only see timestamps.
 * If it is wrong it falls back on tcp_parse_options().
+ *
+ * Returns true if we should drop this packet based on present TCP-options.
 */
 static bool tcp_fast_parse_options(const struct net *net,
 				   const struct sk_buff *skb,
@@ -3842,18 +3844,20 @@ static bool tcp_fast_parse_options(const struct net *net,
 	 */
 	if (th->doff == (sizeof(*th) / 4)) {
 		tp->rx_opt.saw_tstamp = 0;
-		return false;
+		goto extra_opt_check;
 	} else if (tp->rx_opt.tstamp_ok &&
 		   th->doff == ((sizeof(*th) + TCPOLEN_TSTAMP_ALIGNED) / 4)) {
 		if (tcp_parse_aligned_timestamp(tp, th))
-			return true;
+			goto extra_opt_check;
 	}
 
 	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
Quoted 9 lines

net/ipv4/tcp_input.c static bool tcp_fast_parse_options(const struct net *net, · line 3860

Quoted 4 lines
net/ipv4/tcp_input.c +3 -1
-	return true;
+extra_opt_check:
+	/* ToDo - will be added here */
+	return false;

I'm not sure the "ToDo" will be liked - it would be more of a standalone change to wait until the option registration patch to add the gotos. The main consideration is that git bisect works fine either way, so it's not a big deal to leave this as-is and see if anyone objects.

I agree that the ToDo won't be liked. I will remove that.

Christoph

Quoted 29 lines

Mat

Quoted 21 lines
net/ipv4/tcp_input.c +4 -2
 }
 
 #ifdef CONFIG_TCP_MD5SIG
@@ -5173,9 +5177,11 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
 	struct tcp_sock *tp = tcp_sk(sk);
 	bool rst_seq_match = false;
 
+	if (tcp_fast_parse_options(sock_net(sk), skb, th, tp))
+		goto discard;
+
 	/* RFC1323: H1. Apply PAWS check first. */
-	if (tcp_fast_parse_options(sock_net(sk), skb, th, tp) &&
-	    tp->rx_opt.saw_tstamp &&
+	if (tp->rx_opt.saw_tstamp &&
 	    tcp_paws_discard(sk, skb)) {
 		if (!th->rst) {
 			NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSESTABREJECTED);
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#11 lore raw

Re: [MPTCP] [RFC v3 04/15] tcp: Allow tcp_fast_parse_options to drop segments

Quoted 11 lines · Christoph Paasch
Quoted 10 lines · Mat Martineau
Quoted 7 lines · Christoph Paasch

The TCP extra-options parser might decide to drop the segment based on the present option. Especially, for TCP_MD5, we want to do this as soon as possible (thus, right after the parsing).

This patch allows to do this by splitting out tcp_fast_parse_options() in tcp_validate_incoming().

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Originally, tcp_fast_parse_options() returned false, when there is no timestamp option (except in the case of the slow-path processing through tcp_parse_options() where it always returns true.

Minor nitpick: unmatched paren above.

Thanks, I fixed this.

Also, changed the commit-message to the below text to avoid the mention of extra-options explicitly:

=====
tcp: Allow tcp_fast_parse_options to drop segments

After parsing the TCP-options, some option-kinds might trigger a drop of the segment (e.g., as is the case for TCP_MD5). As we are moving to consolidate the TCP_MD5-code in follow-up patches, we need to add the capability to drop a segment right after parsing the options in tcp_fast_parse_options().

Originally, tcp_fast_parse_options() returned false, when there is no timestamp option, except in the case of the slow-path processing through tcp_parse_options() where it always returns true.

So, the return-value of tcp_fast_parse_options() was kind of inconsistent. With this patch, we make it return true when the segment should get dropped based on the parsed options, and false otherwise.

In tcp_validate_incoming, we will then just check for tp->rx_opt.saw_tstamp to see if we should verify PAWS.

The goto will be used in a follow-up patch to check whether one of the options triggers a drop of the segment.

author
#12 lore raw

RFC v3 5/15 tcp: Make smc_parse_options return 1 on success

In reply to Christoph Paasch

Will be needed to allow extra-options to parse experimental TCP options. That way, if the SMC-options don't match, we check the extra-options.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>
---
 net/ipv4/tcp_input.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
net/ipv4/tcp_input.c +11 -8
index cba194a0222f..eedb355a0533 100644
@@ -3676,19 +3676,22 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
 	foc->exp = exp_opt;
 }
 
-static void smc_parse_options(const struct tcphdr *th,
-			      struct tcp_options_received *opt_rx,
-			      const unsigned char *ptr,
-			      int opsize)
+static int smc_parse_options(const struct tcphdr *th,
+			     struct tcp_options_received *opt_rx,
+			     const unsigned char *ptr,
+			     int opsize)
 {
 #if IS_ENABLED(CONFIG_SMC)
 	if (static_branch_unlikely(&tcp_have_smc)) {
 		if (th->syn && !(opsize & 1) &&
 		    opsize >= TCPOLEN_EXP_SMC_BASE &&
-		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
+		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
 			opt_rx->smc_ok = 1;
+			return 1;
+		}
 	}
 #endif
+	return 0;
 }
 
 /* Look for tcp options. Normally only called on SYN and SYNACK packets.
@@ -3798,9 +3801,9 @@ void tcp_parse_options(const struct net *net,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
-				else
-					smc_parse_options(th, opt_rx, ptr,
-							  opsize);
+				else if (smc_parse_options(th, opt_rx, ptr,
+							   opsize))
+					break;
 				break;
 
 			}
Signature · 1 line
2.15.0
#13 lore raw

Re: [MPTCP] [RFC v3 05/15] tcp: Make smc_parse_options return 1 on success

Tag as "tcp_smc: ..." ?

Quoted 2 lines · Christoph Paasch

Will be needed to allow extra-options to parse experimental TCP options. That way, if the SMC-options don't match, we check the extra-options.

Similar to the earlier patch, the "extra-options" name is still unfamiliar to most on netdev.

Quoted 39 lines
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---
net/ipv4/tcp_input.c | 19 +++++++++++--------

1 file changed, 11 insertions(+), 8 deletions(-)
net/ipv4/tcp_input.c +8 -5
index cba194a0222f..eedb355a0533 100644
@@ -3676,19 +3676,22 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
 	foc->exp = exp_opt;
 }
 
-static void smc_parse_options(const struct tcphdr *th,
-			      struct tcp_options_received *opt_rx,
-			      const unsigned char *ptr,
-			      int opsize)
+static int smc_parse_options(const struct tcphdr *th,
+			     struct tcp_options_received *opt_rx,
+			     const unsigned char *ptr,
+			     int opsize)
 {
 #if IS_ENABLED(CONFIG_SMC)
 	if (static_branch_unlikely(&tcp_have_smc)) {
 		if (th->syn && !(opsize & 1) &&
 		    opsize >= TCPOLEN_EXP_SMC_BASE &&
-		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
+		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
 			opt_rx->smc_ok = 1;
+			return 1;
+		}
 	}
 #endif
+	return 0;
 }
 
 /* Look for tcp options. Normally only called on SYN and SYNACK packets.

net/ipv4/tcp_input.c void tcp_parse_options(const struct net *net, · line 3807

Quoted 10 lines
net/ipv4/tcp_input.c +3 -3
@@ -3798,9 +3801,9 @@ void tcp_parse_options(const struct net *net,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
-				else
-					smc_parse_options(th, opt_rx, ptr,
-							  opsize);
+				else if (smc_parse_options(th, opt_rx, ptr,
+							   opsize))
+					break;
 				break;

From a "standalone patch" perspective, I can see someone wondering why there would be an if statement when the break happens no matter what smc_parse_options returns. I'm not sure if it's best to set the stage for the next patch, as this does, or just leave this chunk of code alone until the next patch.

Quoted 6 lines
net/ipv4/tcp_input.c +0 -0
 			}
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#14 lore raw

Re: [MPTCP] [RFC v3 05/15] tcp: Make smc_parse_options return 1 on success

Quoted 2 lines · Mat Martineau

Tag as "tcp_smc: ..." ?

Yes, done.

Quoted 8 lines · Christoph Paasch
Quoted 3 lines

Will be needed to allow extra-options to parse experimental TCP options. That way, if the SMC-options don't match, we check the extra-options.

Similar to the earlier patch, the "extra-options" name is still unfamiliar to most on netdev.

Yes, reworded it to:

tcp: Make smc_parse_options return 1 on success

As we allow a generic TCP-option parser that also parses experimental TCP options, we need to add a return-value to smc_parse_options() that indicates whether the option actually matched or not.

Quoted 48 lines
Quoted 48 lines
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---
net/ipv4/tcp_input.c | 19 +++++++++++--------

1 file changed, 11 insertions(+), 8 deletions(-)
net/ipv4/tcp_input.c +10 -8
index cba194a0222f..eedb355a0533 100644
@@ -3676,19 +3676,22 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
 	foc->exp = exp_opt;
 }
 
-static void smc_parse_options(const struct tcphdr *th,
-			      struct tcp_options_received *opt_rx,
-			      const unsigned char *ptr,
-			      int opsize)
+static int smc_parse_options(const struct tcphdr *th,
+			     struct tcp_options_received *opt_rx,
+			     const unsigned char *ptr,
+			     int opsize)
 {
 #if IS_ENABLED(CONFIG_SMC)
 	if (static_branch_unlikely(&tcp_have_smc)) {
 		if (th->syn && !(opsize & 1) &&
 		    opsize >= TCPOLEN_EXP_SMC_BASE &&
-		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC)
+		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
 			opt_rx->smc_ok = 1;
+			return 1;
+		}
 	}
 #endif
+	return 0;
 }
 
 /* Look for tcp options. Normally only called on SYN and SYNACK packets.
@@ -3798,9 +3801,9 @@ void tcp_parse_options(const struct net *net,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
-				else
-					smc_parse_options(th, opt_rx, ptr,
-							  opsize);
+				else if (smc_parse_options(th, opt_rx, ptr,
+							   opsize))
Quoted 8 lines

net/ipv4/tcp_input.c void tcp_parse_options(const struct net *net, · line 3807

Quoted 2 lines
net/ipv4/tcp_input.c +1 -0
+					break;
 				break;

From a "standalone patch" perspective, I can see someone wondering why there would be an if statement when the break happens no matter what smc_parse_options returns. I'm not sure if it's best to set the stage for the next patch, as this does, or just leave this chunk of code alone until the next patch.

Agreed - I will remove this last chunk of the change.

Christoph

author
#15 lore raw

RFC v3 6/15 tcp: Register handlers for extra TCP options

In reply to Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

 drivers/infiniband/hw/cxgb4/cm.c |   2 +-
 include/linux/tcp.h              |  28 ++++
 include/net/tcp.h                | 106 ++++++++++++-
 net/ipv4/syncookies.c            |   6 +-
 net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
 net/ipv4/tcp_input.c             |  47 +++++-
 net/ipv4/tcp_ipv4.c              |  90 ++++++++---
 net/ipv4/tcp_minisocks.c         |  30 +++-
 net/ipv4/tcp_output.c            |  37 ++---
 net/ipv6/syncookies.c            |   6 +-
 net/ipv6/tcp_ipv6.c              |  28 ++++
 11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
  * to increase this, although since:
  *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +105 -1

122 lines not on this page · raw message

net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +334 -1

359 lines not on this page · raw message

net/ipv4/tcp_input.c +41 -6

119 lines not on this page · raw message

net/ipv4/tcp_ipv4.c +72 -18

166 lines not on this page · raw message

net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
  *
  * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -0
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
Signature · 1 line
2.15.0
#16 lore raw

Re: [MPTCP] [RFC v3 06/15] tcp: Register handlers for extra TCP options

Quoted 19 lines · Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

Quoted 5 lines

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

I'm still on the fence about using a static key at all, especially without rate limiting. It made a lot more sense when used in conjunction with the global option list, and it's not widely used in net/. There are ~60 occurences of static_branch_unlikely in the entire mainline tree today, and this patch adds ~30!

After I get through the rest of the patch series, I'll take another look at where static_branch_unlikely is used. If the sensive code paths have a known socket type, that would allow use of "if (unlikely(!hlist_empty(&some_sk->tcp_option_list)) && ..." without having to involve a function call to tcp_extopt_get_list().

Quoted 194 lines

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

drivers/infiniband/hw/cxgb4/cm.c |   2 +-
include/linux/tcp.h              |  28 ++++
include/net/tcp.h                | 106 ++++++++++++-
net/ipv4/syncookies.c            |   6 +-
net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
net/ipv4/tcp_input.c             |  47 +++++-
net/ipv4/tcp_ipv4.c              |  90 ++++++++---
net/ipv4/tcp_minisocks.c         |  30 +++-
net/ipv4/tcp_output.c            |  37 ++---
net/ipv6/syncookies.c            |   6 +-
net/ipv6/tcp_ipv6.c              |  28 ++++
11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
 * to increase this, although since:
 *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +48 -1
index 3c3744e52cd1..86133aa24fca 100644
@@ -202,6 +202,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
 #define TCPOLEN_EXP_SMC_BASE   6
+#define TCPOLEN_EXP_BASE       6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -403,7 +404,8 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 		int flags, int *addr_len);
 void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
-		       int estab, struct tcp_fastopen_cookie *foc);
+		       int estab, struct tcp_fastopen_cookie *foc,
+		       struct sock *sk);
 const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
@@ -2064,4 +2066,106 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 #if IS_ENABLED(CONFIG_SMC)
 extern struct static_key_false tcp_have_smc;
 #endif
+
+extern struct static_key_false tcp_extopt_enabled;
+struct tcp_extopt_store;
+
+struct tcp_extopt_ops {
+	u32			option_kind;
+	unsigned char		priority;
+	void (*parse)(int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct sock *sk,
+		      struct tcp_extopt_store *store);
+	bool (*check)(struct sock *sk,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct tcp_extopt_store *store);
+	void (*post_process)(struct sock *sk,
+			     struct tcp_options_received *opt_rx,
+			     struct tcp_extopt_store *store);
+	/* Return the number of bytes consumed */
+	unsigned int (*prepare)(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*write)(__be32 *ptr, struct sk_buff *skb,
+			 struct tcp_out_options *opts, struct sock *sk,
+			 struct tcp_extopt_store *store);
+	int (*response_prepare)(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*response_write)(__be32 *ptr, struct sk_buff *orig,
+				  struct tcphdr *th,
+				  struct tcp_out_options *opts,
+				  const struct sock *sk,
+				  struct tcp_extopt_store *store);
+	int (*add_header_len)(const struct sock *listener,
+			      const struct sock *sk,
+			      struct tcp_extopt_store *store);
+	struct tcp_extopt_store *(*copy)(struct sock *listener,
+					 struct request_sock *req,
+					 struct tcp_options_received *opt,
+					 struct tcp_extopt_store *from);

include/net/tcp.h static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk) · line 2123

Quoted 10 lines
include/net/tcp.h +10 -0
+	struct tcp_extopt_store *(*move)(struct sock *from, struct sock *to,
+					 struct tcp_extopt_store *store);
+	void (*destroy)(struct tcp_extopt_store *store);
+	struct module		*owner;
+};
+
+/* Every extra-option in a TCP-socket gets stored in a _store structure which
+ * has to have tcp_extopt_store as the first element.
+ *
+ * That way, an extra option registers passes this to tcp_register_extopt,

Maybe "That way, code registering extra options can pass this to tcp_register_extopt" ...

Quoted 125 lines
include/net/tcp.h +47 -0
+ * and the surrounding structure can contain per-option state.
+ */
+struct tcp_extopt_store {
+	struct hlist_node		list;
+	const struct tcp_extopt_ops	*ops;
+};
+
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk);
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx);
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk);
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt);
+
+void tcp_extopt_move(struct sock *from, struct sock *to);
+
+void tcp_extopt_destroy(struct sock *sk);
+
 #endif	/* _TCP_H */
net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +31 -0
index c470fec9062f..6676fc3c0059 100644
@@ -305,6 +305,10 @@ EXPORT_SYMBOL(tcp_have_smc);
 struct percpu_counter tcp_sockets_allocated;
 EXPORT_SYMBOL(tcp_sockets_allocated);
 
+/* Optional TCP option handlers */
+DEFINE_STATIC_KEY_FALSE(tcp_extopt_enabled);
+EXPORT_SYMBOL_GPL(tcp_extopt_enabled);
+
 /*
 * TCP splice context
 */
@@ -416,6 +420,7 @@ void tcp_init_sock(struct sock *sk)
 	tcp_init_xmit_timers(sk);
 	INIT_LIST_HEAD(&tp->tsq_node);
 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
+	INIT_HLIST_HEAD(&tp->tcp_option_list);
 
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
@@ -3473,6 +3478,335 @@ EXPORT_SYMBOL(tcp_md5_hash_key);
 
 #endif
 
+static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
+{
+	if (sk_fullsock(sk))
+		return &tcp_sk(sk)->tcp_option_list;
+	else if (sk->sk_state == TCP_NEW_SYN_RECV)
+		return &tcp_rsk(inet_reqsk(sk))->tcp_option_list;
+	else if (sk->sk_state == TCP_TIME_WAIT)
+		return &tcp_twsk(sk)->tcp_option_list;
+
+	return NULL;
+}
+
+/* Caller must ensure that rcu is locked */
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == kind)
+			return entry;
+	}
+
+	return NULL;

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3516

Quoted 10 lines
net/ipv4/tcp.c +10 -0
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_find_kind);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+
+	rcu_read_lock_bh();

I'm curious, why the _bh variant in these functions? It would require use of hlist_for_each_entry_rcu_bh to traverse the list, in tcp_extopt_find_kind or in the functions below. It's ok for rcu_read_lock() to be used in interrupt context.

Quoted 193 lines
net/ipv4/tcp.c +193 -0
+	entry = tcp_extopt_find_kind(opcode, sk);
+
+	if (entry && entry->ops->parse)
+		entry->ops->parse(opsize, opptr, skb, opt_rx, sk, entry);
+	rcu_read_unlock_bh();
+}
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	bool drop = false;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		bool ret = false;
+
+		if (entry->ops->check)
+			ret = entry->ops->check(sk, skb, opt_rx, entry);
+
+		if (ret)
+			drop = true;
+	}
+	rcu_read_unlock_bh();
+
+	return drop;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_check);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->post_process)
+			entry->ops->post_process(sk, opt_rx, entry);
+	}
+	rcu_read_unlock_bh();
+}
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->prepare))
+			continue;
+
+		used += entry->ops->prepare(skb, flags, remaining - used, opts,
+					    sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->write))
+			continue;
+
+		ptr = entry->ops->write(ptr, skb, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_write);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		int ret;
+
+		if (unlikely(!entry->ops->response_prepare))
+			continue;
+
+		ret = entry->ops->response_prepare(orig, flags,
+						   remaining - used, opts,
+						   sk, entry);
+
+		used += ret;
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_prepare);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->response_write))
+			continue;
+
+		ptr = entry->ops->response_write(ptr, orig, th, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_write);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	int tcp_header_len = 0;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, &tcp_sk(sk)->tcp_option_list, list) {
+		if (unlikely(!entry->ops->add_header_len))
+			continue;
+
+		tcp_header_len += entry->ops->add_header_len(listener, sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return tcp_header_len;
+}
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk)
+{
+	struct hlist_node *add_before = NULL;
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	int ret = 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	if (!store->ops->option_kind)
+		return -EINVAL;
+
+	if (!try_module_get(store->ops->owner))
+		return -ENOENT;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == store->ops->option_kind) {
+			rcu_read_unlock_bh();
+			pr_notice("Option kind %u already registered\n",
+				  store->ops->option_kind);
+			module_put(store->ops->owner);
+			return -EEXIST;

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3719

Quoted 10 lines
net/ipv4/tcp.c +10 -0
+		}
+
+		if (entry->ops->priority <= store->ops->priority)
+			add_before = &entry->list;
+	}
+
+	if (add_before)
+		hlist_add_behind_rcu(&store->list, add_before);
+	else
+		hlist_add_head_rcu(&store->list, lhead);

I'm assuming the socket lock is being used to synchonize modifications to the options list and must be held before this function is called. It would help to add comments indicating which functions require the socket lock to be held (here or in include/net/tcp.h)

What's more, rcu_read_lock/unlock aren't needed when the write lock (the socket lock in this case) is held - so no need to call those here or in other functions where the lock is required before calling.

Quoted 75 lines
net/ipv4/tcp.c +75 -0
+
+	rcu_read_unlock_bh();
+	pr_debug("Option kind %u registered\n", store->ops->option_kind);
+
+	static_branch_inc(&tcp_extopt_enabled);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tcp_register_extopt);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *from, *to;
+
+	from = tcp_extopt_get_list(listener);
+	to = tcp_extopt_get_list(req_to_sk(req));
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, from, list) {
+		struct tcp_extopt_store *new;
+
+		if (!try_module_get(entry->ops->owner)) {
+			pr_err("%s Module get failed while copying\n", __func__);
+			continue;
+		}
+
+		new = entry->ops->copy(listener, req, opt, entry);
+		if (!new) {
+			module_put(entry->ops->owner);
+			continue;
+		}
+
+		hlist_add_tail_rcu(&new->list, to);
+
+		static_branch_inc(&tcp_extopt_enabled);
+	}
+	rcu_read_unlock_bh();
+}
+
+void tcp_extopt_move(struct sock *from, struct sock *to)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lfrom, *lto;
+	struct hlist_node *tmp;
+
+	lfrom = tcp_extopt_get_list(from);
+	lto = tcp_extopt_get_list(to);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lfrom, list) {
+		hlist_del_rcu(&entry->list);
+
+		if (entry->ops->move) {
+			entry = entry->ops->move(from, to, entry);
+			if (!entry)
+				continue;
+		}
+
+		hlist_add_tail_rcu(&entry->list, lto);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_move);
+
+void tcp_extopt_destroy(struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	struct hlist_node *tmp;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3804

Quoted 10 lines
net/ipv4/tcp.c +10 -0
+	hlist_for_each_entry_safe(entry, tmp, lhead, list) {
+		struct module *owner = entry->ops->owner;
+
+		hlist_del_rcu(&entry->list);
+
+		entry->ops->destroy(entry);
+
+		static_branch_dec(&tcp_extopt_enabled);
+
+		module_put(owner);

Is it ok to allow the module to unload before synchronize_rcu() or synchronize_net()? Maybe the static_branch_dec and module_put needs to be invoked via call_rcu?

Mat

Quoted 612 lines
net/ipv4/tcp.c +5 -1
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_destroy);
+
 void tcp_done(struct sock *sk)
 {
 	struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
@@ -3622,7 +3956,6 @@ void __init tcp_init(void)
 		INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
 	}
 
-
 	cnt = tcp_hashinfo.ehash_mask + 1;
 	sysctl_tcp_max_orphans = cnt / 2;
 
net/ipv4/tcp_input.c +13 -2
index eedb355a0533..d4d3be8a496c 100644
@@ -3701,7 +3701,7 @@ static int smc_parse_options(const struct tcphdr *th,
 void tcp_parse_options(const struct net *net,
 		       const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx, int estab,
-		       struct tcp_fastopen_cookie *foc)
+		       struct tcp_fastopen_cookie *foc, struct sock *sk)
 {
 	const unsigned char *ptr;
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -3804,6 +3804,17 @@ void tcp_parse_options(const struct net *net,
 				else if (smc_parse_options(th, opt_rx, ptr,
 							   opsize))
 					break;
+				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
+					 opsize >= TCPOLEN_EXP_BASE)
+					tcp_extopt_parse(get_unaligned_be32(ptr),
+							 opsize, ptr, skb,
+							 opt_rx, sk);
+				break;
+
+			default:
+				if (static_branch_unlikely(&tcp_extopt_enabled))
+					tcp_extopt_parse(opcode, opsize, ptr,
+							 skb, opt_rx, sk);
 				break;
 
 			}
@@ -3854,12 +3865,13 @@ static bool tcp_fast_parse_options(const struct net *net,
 			goto extra_opt_check;
 	}
 
-	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
+	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL, tcp_to_sk(tp));
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 

extra_opt_check:

net/ipv4/tcp_input.c +2 -1
-	/* ToDo - will be added here */
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		return tcp_extopt_check(tcp_to_sk(tp), skb, &tp->rx_opt);
	return false;
}
net/ipv4/tcp_input.c +26 -3
@@ -5333,6 +5345,10 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 	tp->rx_opt.saw_tstamp = 0;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    !hlist_empty(&tp->tcp_option_list))
+		goto slow_path;
+
 	/*	pred_flags is 0xS?10 << 16 + snd_wnd
 	 *	if header_prediction is to be made
 	 *	'S' will always be tp->tcp_header_len >> 2
@@ -5520,7 +5536,7 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 		/* Get original SYNACK MSS value if user MSS sets mss_clamp */
 		tcp_clear_options(&opt);
 		opt.user_mss = opt.mss_clamp = 0;
-		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL, sk);
 		mss = opt.mss_clamp;
 	}
 
@@ -5583,10 +5599,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 	int saved_clamp = tp->rx_opt.mss_clamp;
 	bool fastopen_fail;
 
-	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
+	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc, sk);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tp->rx_opt))
+		goto discard;
+
 	if (th->ack) {
 		/* rfc793:
 		 * "If the state is SYN-SENT then
@@ -5669,6 +5689,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 			tp->tcp_header_len = sizeof(struct tcphdr);
 		}
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
@@ -5762,6 +5785,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 
 		tcp_ecn_rcv_syn(tp, th);
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_mtup_init(sk);
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
@@ -6245,12 +6271,17 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
 	tmp_opt.user_mss  = tp->rx_opt.user_mss;
 	tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
-			  want_cookie ? NULL : &foc);
+			  want_cookie ? NULL : &foc, sk);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tmp_opt))
+		goto drop_and_free;
 
 	if (want_cookie && !tmp_opt.saw_tstamp)
 		tcp_clear_options(&tmp_opt);
@@ -6311,6 +6342,10 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 		tcp_reqsk_record_syn(sk, req, skb);
 		fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
 	}
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_copy(sk, req, &tmp_opt);
+
 	if (fastopen_sk) {
 		af_ops->send_synack(fastopen_sk, dst, &fl, req,
 				    &foc, TCP_SYNACK_FASTOPEN);
net/ipv4/tcp_ipv4.c +72 -18
index 94e28350f420..ad87dd2db092 100644
@@ -600,9 +600,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-#ifdef CONFIG_TCP_MD5SIG
-		__be32 opt[(TCPOLEN_MD5SIG_ALIGNED >> 2)];
-#endif
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct ip_reply_arg arg;
 #ifdef CONFIG_TCP_MD5SIG
@@ -613,6 +611,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	struct sock *sk1 = NULL;
 #endif
 	struct net *net;
+	int offset = 0;
 
 	/* Never send a reset in response to a reset. */
 	if (th->rst)
@@ -678,19 +677,44 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 			goto out;
 
 	}
+#endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.opt);
+#ifdef CONFIG_TCP_MD5SIG
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
 
+		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
+
+		tcp_extopt_response_write(&rep.opt[0], skb, &rep.th, &opts, sk);
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
 	if (key) {
-		rep.opt[0] = htonl((TCPOPT_NOP << 24) |
-				   (TCPOPT_NOP << 16) |
-				   (TCPOPT_MD5SIG << 8) |
-				   TCPOLEN_MD5SIG);
+		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
+					  (TCPOPT_NOP << 16) |
+					  (TCPOPT_MD5SIG << 8) |
+					  TCPOLEN_MD5SIG);
 		/* Update length and the length the header thinks exists */
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
 
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
-				     key, ip_hdr(skb)->saddr,
-				     ip_hdr(skb)->daddr, &rep.th);
+		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
+				    key, ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
@@ -742,14 +766,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-		__be32 opt[(TCPOLEN_TSTAMP_ALIGNED >> 2)
-#ifdef CONFIG_TCP_MD5SIG
-			   + (TCPOLEN_MD5SIG_ALIGNED >> 2)
-#endif
-			];
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
+	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
@@ -763,6 +784,7 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		rep.opt[1] = htonl(tsval);
 		rep.opt[2] = htonl(tsecr);
 		arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
+		offset += 3;
 	}
 
 	/* Swap the send and the receive. */
@@ -774,22 +796,45 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
+
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		int offset = (tsecr) ? 3 : 0;
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
+		used = tcp_extopt_response_prepare(skb, TCPHDR_ACK, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
+		tcp_extopt_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
+
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
+	if (key) {
 		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
 					  (TCPOPT_NOP << 16) |
 					  (TCPOPT_MD5SIG << 8) |
 					  TCPOLEN_MD5SIG);
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len/4;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
 		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
 				    key, ip_hdr(skb)->saddr,
 				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
+
 	arg.flags = reply_flags;
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -893,6 +938,9 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
 */
 static void tcp_v4_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
@@ -1387,6 +1435,10 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 
 	tcp_initialize_rcv_mss(newsk);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
@@ -1907,6 +1959,8 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
 	skb_rbtree_purge(&tp->out_of_order_queue);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Clean up the MD5 key list, if any */
 	if (tp->md5sig_info) {
net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -0
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#17 lore raw

Re: [MPTCP] [RFC v3 06/15] tcp: Register handlers for extra TCP options

Hi Mat,

Quoted 34 lines · Mat Martineau
Quoted 19 lines · Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

Quoted 5 lines

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

I'm still on the fence about using a static key at all, especially without rate limiting. It made a lot more sense when used in conjunction with the global option list, and it's not widely used in net/. There are ~60 occurences of static_branch_unlikely in the entire mainline tree today, and this patch adds ~30!

Quoted 5 lines

After I get through the rest of the patch series, I'll take another look at where static_branch_unlikely is used. If the sensive code paths have a known socket type, that would allow use of "if (unlikely(!hlist_empty(&some_sk->tcp_option_list)) && ..." without having to involve a function call to tcp_extopt_get_list().

I am pretty sure that this would trigger the response from Eric that such a test would add an additional access to a potentially cold cache-line in the hot data-path.

For the static-key question and rate-limiting, I would suggest that we see what netdev's feedback will be. We can add this as one of the discussion-points in the cover-letter mail.

Quoted 198 lines
Quoted 198 lines

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

drivers/infiniband/hw/cxgb4/cm.c |   2 +-
include/linux/tcp.h              |  28 ++++
include/net/tcp.h                | 106 ++++++++++++-
net/ipv4/syncookies.c            |   6 +-
net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
net/ipv4/tcp_input.c             |  47 +++++-
net/ipv4/tcp_ipv4.c              |  90 ++++++++---
net/ipv4/tcp_minisocks.c         |  30 +++-
net/ipv4/tcp_output.c            |  37 ++---
net/ipv6/syncookies.c            |   6 +-
net/ipv6/tcp_ipv6.c              |  28 ++++
11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
 * to increase this, although since:
 *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +53 -1
index 3c3744e52cd1..86133aa24fca 100644
@@ -202,6 +202,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
 #define TCPOLEN_EXP_SMC_BASE   6
+#define TCPOLEN_EXP_BASE       6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -403,7 +404,8 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 		int flags, int *addr_len);
 void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
-		       int estab, struct tcp_fastopen_cookie *foc);
+		       int estab, struct tcp_fastopen_cookie *foc,
+		       struct sock *sk);
 const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
@@ -2064,4 +2066,106 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 #if IS_ENABLED(CONFIG_SMC)
 extern struct static_key_false tcp_have_smc;
 #endif
+
+extern struct static_key_false tcp_extopt_enabled;
+struct tcp_extopt_store;
+
+struct tcp_extopt_ops {
+	u32			option_kind;
+	unsigned char		priority;
+	void (*parse)(int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct sock *sk,
+		      struct tcp_extopt_store *store);
+	bool (*check)(struct sock *sk,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct tcp_extopt_store *store);
+	void (*post_process)(struct sock *sk,
+			     struct tcp_options_received *opt_rx,
+			     struct tcp_extopt_store *store);
+	/* Return the number of bytes consumed */
+	unsigned int (*prepare)(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*write)(__be32 *ptr, struct sk_buff *skb,
+			 struct tcp_out_options *opts, struct sock *sk,
+			 struct tcp_extopt_store *store);
+	int (*response_prepare)(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*response_write)(__be32 *ptr, struct sk_buff *orig,
+				  struct tcphdr *th,
+				  struct tcp_out_options *opts,
+				  const struct sock *sk,
+				  struct tcp_extopt_store *store);
+	int (*add_header_len)(const struct sock *listener,
+			      const struct sock *sk,
+			      struct tcp_extopt_store *store);
+	struct tcp_extopt_store *(*copy)(struct sock *listener,
+					 struct request_sock *req,
+					 struct tcp_options_received *opt,
+					 struct tcp_extopt_store *from);
+	struct tcp_extopt_store *(*move)(struct sock *from, struct sock *to,
+					 struct tcp_extopt_store *store);
+	void (*destroy)(struct tcp_extopt_store *store);
+	struct module		*owner;
+};
Quoted 8 lines

include/net/tcp.h static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk) · line 2123

Quoted 5 lines
include/net/tcp.h +5 -0
+
+/* Every extra-option in a TCP-socket gets stored in a _store structure which
+ * has to have tcp_extopt_store as the first element.
+ *
+ * That way, an extra option registers passes this to tcp_register_extopt,

Maybe "That way, code registering extra options can pass this to tcp_register_extopt" ...

Sure, changed it.

Quoted 133 lines
Quoted 133 lines
include/net/tcp.h +47 -0
+ * and the surrounding structure can contain per-option state.
+ */
+struct tcp_extopt_store {
+	struct hlist_node		list;
+	const struct tcp_extopt_ops	*ops;
+};
+
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk);
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx);
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk);
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt);
+
+void tcp_extopt_move(struct sock *from, struct sock *to);
+
+void tcp_extopt_destroy(struct sock *sk);
+
 #endif	/* _TCP_H */
net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +38 -0
index c470fec9062f..6676fc3c0059 100644
@@ -305,6 +305,10 @@ EXPORT_SYMBOL(tcp_have_smc);
 struct percpu_counter tcp_sockets_allocated;
 EXPORT_SYMBOL(tcp_sockets_allocated);
 
+/* Optional TCP option handlers */
+DEFINE_STATIC_KEY_FALSE(tcp_extopt_enabled);
+EXPORT_SYMBOL_GPL(tcp_extopt_enabled);
+
 /*
 * TCP splice context
 */
@@ -416,6 +420,7 @@ void tcp_init_sock(struct sock *sk)
 	tcp_init_xmit_timers(sk);
 	INIT_LIST_HEAD(&tp->tsq_node);
 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
+	INIT_HLIST_HEAD(&tp->tcp_option_list);
 
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
@@ -3473,6 +3478,335 @@ EXPORT_SYMBOL(tcp_md5_hash_key);
 
 #endif
 
+static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
+{
+	if (sk_fullsock(sk))
+		return &tcp_sk(sk)->tcp_option_list;
+	else if (sk->sk_state == TCP_NEW_SYN_RECV)
+		return &tcp_rsk(inet_reqsk(sk))->tcp_option_list;
+	else if (sk->sk_state == TCP_TIME_WAIT)
+		return &tcp_twsk(sk)->tcp_option_list;
+
+	return NULL;
+}
+
+/* Caller must ensure that rcu is locked */
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == kind)
+			return entry;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_find_kind);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk)
+{
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3516

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+	struct tcp_extopt_store *entry;
+
+	rcu_read_lock_bh();

I'm curious, why the _bh variant in these functions? It would require use of hlist_for_each_entry_rcu_bh to traverse the list, in tcp_extopt_find_kind or in the functions below. It's ok for rcu_read_lock() to be used in interrupt context.

Yes, I think you are right after researching more. I guess I misunderstood something.

I thought that I need to call the _bh variant to disable interrupts. Because, the TCP-input code path can now be called from both contexts (softirq and with interrupts enabled).

I will change all of this to rcu_read_lock() and rcu_read_unlock().

Quoted 201 lines
Quoted 201 lines
net/ipv4/tcp.c +200 -0
+	entry = tcp_extopt_find_kind(opcode, sk);
+
+	if (entry && entry->ops->parse)
+		entry->ops->parse(opsize, opptr, skb, opt_rx, sk, entry);
+	rcu_read_unlock_bh();
+}
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	bool drop = false;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		bool ret = false;
+
+		if (entry->ops->check)
+			ret = entry->ops->check(sk, skb, opt_rx, entry);
+
+		if (ret)
+			drop = true;
+	}
+	rcu_read_unlock_bh();
+
+	return drop;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_check);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->post_process)
+			entry->ops->post_process(sk, opt_rx, entry);
+	}
+	rcu_read_unlock_bh();
+}
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->prepare))
+			continue;
+
+		used += entry->ops->prepare(skb, flags, remaining - used, opts,
+					    sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->write))
+			continue;
+
+		ptr = entry->ops->write(ptr, skb, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_write);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		int ret;
+
+		if (unlikely(!entry->ops->response_prepare))
+			continue;
+
+		ret = entry->ops->response_prepare(orig, flags,
+						   remaining - used, opts,
+						   sk, entry);
+
+		used += ret;
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_prepare);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->response_write))
+			continue;
+
+		ptr = entry->ops->response_write(ptr, orig, th, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_write);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	int tcp_header_len = 0;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, &tcp_sk(sk)->tcp_option_list, list) {
+		if (unlikely(!entry->ops->add_header_len))
+			continue;
+
+		tcp_header_len += entry->ops->add_header_len(listener, sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return tcp_header_len;
+}
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk)
+{
+	struct hlist_node *add_before = NULL;
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	int ret = 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	if (!store->ops->option_kind)
+		return -EINVAL;
+
+	if (!try_module_get(store->ops->owner))
+		return -ENOENT;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == store->ops->option_kind) {
+			rcu_read_unlock_bh();
+			pr_notice("Option kind %u already registered\n",
+				  store->ops->option_kind);
+			module_put(store->ops->owner);
+			return -EEXIST;
+		}
+
+		if (entry->ops->priority <= store->ops->priority)
+			add_before = &entry->list;
+	}
+
+	if (add_before)
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3719

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+		hlist_add_behind_rcu(&store->list, add_before);
+	else
+		hlist_add_head_rcu(&store->list, lhead);

I'm assuming the socket lock is being used to synchonize modifications to the options list and must be held before this function is called. It would help to add comments indicating which functions require the socket lock to be held (here or in include/net/tcp.h)

I will add a comment in tcp.h and here.

Quoted 3 lines

What's more, rcu_read_lock/unlock aren't needed when the write lock (the socket lock in this case) is held - so no need to call those here or in other functions where the lock is required before calling.

Yep - will remove the rcu_read_lock/unlock.

Quoted 80 lines
Quoted 80 lines
net/ipv4/tcp.c +80 -0
+	rcu_read_unlock_bh();
+	pr_debug("Option kind %u registered\n", store->ops->option_kind);
+
+	static_branch_inc(&tcp_extopt_enabled);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tcp_register_extopt);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *from, *to;
+
+	from = tcp_extopt_get_list(listener);
+	to = tcp_extopt_get_list(req_to_sk(req));
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, from, list) {
+		struct tcp_extopt_store *new;
+
+		if (!try_module_get(entry->ops->owner)) {
+			pr_err("%s Module get failed while copying\n", __func__);
+			continue;
+		}
+
+		new = entry->ops->copy(listener, req, opt, entry);
+		if (!new) {
+			module_put(entry->ops->owner);
+			continue;
+		}
+
+		hlist_add_tail_rcu(&new->list, to);
+
+		static_branch_inc(&tcp_extopt_enabled);
+	}
+	rcu_read_unlock_bh();
+}
+
+void tcp_extopt_move(struct sock *from, struct sock *to)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lfrom, *lto;
+	struct hlist_node *tmp;
+
+	lfrom = tcp_extopt_get_list(from);
+	lto = tcp_extopt_get_list(to);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lfrom, list) {
+		hlist_del_rcu(&entry->list);
+
+		if (entry->ops->move) {
+			entry = entry->ops->move(from, to, entry);
+			if (!entry)
+				continue;
+		}
+
+		hlist_add_tail_rcu(&entry->list, lto);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_move);
+
+void tcp_extopt_destroy(struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	struct hlist_node *tmp;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lhead, list) {
+		struct module *owner = entry->ops->owner;
+
+		hlist_del_rcu(&entry->list);
+
+		entry->ops->destroy(entry);
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3804

Quoted 4 lines
net/ipv4/tcp.c +4 -0
+
+		static_branch_dec(&tcp_extopt_enabled);
+
+		module_put(owner);

Is it ok to allow the module to unload before synchronize_rcu() or synchronize_net()? Maybe the static_branch_dec and module_put needs to be invoked via call_rcu?

I think it's fine. From what I see, before really unloading the module (coming from the delete_module() system-call), it does a synchronize_sched().

Christoph

Quoted 621 lines

Mat

Quoted 612 lines
net/ipv4/tcp.c +5 -1
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_destroy);
+
 void tcp_done(struct sock *sk)
 {
 	struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
@@ -3622,7 +3956,6 @@ void __init tcp_init(void)
 		INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
 	}
 
-
 	cnt = tcp_hashinfo.ehash_mask + 1;
 	sysctl_tcp_max_orphans = cnt / 2;
 
net/ipv4/tcp_input.c +13 -2
index eedb355a0533..d4d3be8a496c 100644
@@ -3701,7 +3701,7 @@ static int smc_parse_options(const struct tcphdr *th,
 void tcp_parse_options(const struct net *net,
 		       const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx, int estab,
-		       struct tcp_fastopen_cookie *foc)
+		       struct tcp_fastopen_cookie *foc, struct sock *sk)
 {
 	const unsigned char *ptr;
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -3804,6 +3804,17 @@ void tcp_parse_options(const struct net *net,
 				else if (smc_parse_options(th, opt_rx, ptr,
 							   opsize))
 					break;
+				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
+					 opsize >= TCPOLEN_EXP_BASE)
+					tcp_extopt_parse(get_unaligned_be32(ptr),
+							 opsize, ptr, skb,
+							 opt_rx, sk);
+				break;
+
+			default:
+				if (static_branch_unlikely(&tcp_extopt_enabled))
+					tcp_extopt_parse(opcode, opsize, ptr,
+							 skb, opt_rx, sk);
 				break;
 
 			}
@@ -3854,12 +3865,13 @@ static bool tcp_fast_parse_options(const struct net *net,
 			goto extra_opt_check;
 	}
 
-	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
+	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL, tcp_to_sk(tp));
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 

extra_opt_check:

net/ipv4/tcp_input.c +2 -1
-	/* ToDo - will be added here */
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		return tcp_extopt_check(tcp_to_sk(tp), skb, &tp->rx_opt);
	return false;
}
net/ipv4/tcp_input.c +26 -3
@@ -5333,6 +5345,10 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 	tp->rx_opt.saw_tstamp = 0;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    !hlist_empty(&tp->tcp_option_list))
+		goto slow_path;
+
 	/*	pred_flags is 0xS?10 << 16 + snd_wnd
 	 *	if header_prediction is to be made
 	 *	'S' will always be tp->tcp_header_len >> 2
@@ -5520,7 +5536,7 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 		/* Get original SYNACK MSS value if user MSS sets mss_clamp */
 		tcp_clear_options(&opt);
 		opt.user_mss = opt.mss_clamp = 0;
-		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL, sk);
 		mss = opt.mss_clamp;
 	}
 
@@ -5583,10 +5599,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 	int saved_clamp = tp->rx_opt.mss_clamp;
 	bool fastopen_fail;
 
-	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
+	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc, sk);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tp->rx_opt))
+		goto discard;
+
 	if (th->ack) {
 		/* rfc793:
 		 * "If the state is SYN-SENT then
@@ -5669,6 +5689,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 			tp->tcp_header_len = sizeof(struct tcphdr);
 		}
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
@@ -5762,6 +5785,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 
 		tcp_ecn_rcv_syn(tp, th);
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_mtup_init(sk);
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
@@ -6245,12 +6271,17 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
 	tmp_opt.user_mss  = tp->rx_opt.user_mss;
 	tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
-			  want_cookie ? NULL : &foc);
+			  want_cookie ? NULL : &foc, sk);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tmp_opt))
+		goto drop_and_free;
 
 	if (want_cookie && !tmp_opt.saw_tstamp)
 		tcp_clear_options(&tmp_opt);
@@ -6311,6 +6342,10 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 		tcp_reqsk_record_syn(sk, req, skb);
 		fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
 	}
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_copy(sk, req, &tmp_opt);
+
 	if (fastopen_sk) {
 		af_ops->send_synack(fastopen_sk, dst, &fl, req,
 				    &foc, TCP_SYNACK_FASTOPEN);
net/ipv4/tcp_ipv4.c +72 -18
index 94e28350f420..ad87dd2db092 100644
@@ -600,9 +600,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-#ifdef CONFIG_TCP_MD5SIG
-		__be32 opt[(TCPOLEN_MD5SIG_ALIGNED >> 2)];
-#endif
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct ip_reply_arg arg;
 #ifdef CONFIG_TCP_MD5SIG
@@ -613,6 +611,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	struct sock *sk1 = NULL;
 #endif
 	struct net *net;
+	int offset = 0;
 
 	/* Never send a reset in response to a reset. */
 	if (th->rst)
@@ -678,19 +677,44 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 			goto out;
 
 	}
+#endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.opt);
+#ifdef CONFIG_TCP_MD5SIG
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
 
+		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
+
+		tcp_extopt_response_write(&rep.opt[0], skb, &rep.th, &opts, sk);
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
 	if (key) {
-		rep.opt[0] = htonl((TCPOPT_NOP << 24) |
-				   (TCPOPT_NOP << 16) |
-				   (TCPOPT_MD5SIG << 8) |
-				   TCPOLEN_MD5SIG);
+		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
+					  (TCPOPT_NOP << 16) |
+					  (TCPOPT_MD5SIG << 8) |
+					  TCPOLEN_MD5SIG);
 		/* Update length and the length the header thinks exists */
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
 
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
-				     key, ip_hdr(skb)->saddr,
-				     ip_hdr(skb)->daddr, &rep.th);
+		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
+				    key, ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
@@ -742,14 +766,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-		__be32 opt[(TCPOLEN_TSTAMP_ALIGNED >> 2)
-#ifdef CONFIG_TCP_MD5SIG
-			   + (TCPOLEN_MD5SIG_ALIGNED >> 2)
-#endif
-			];
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
+	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
@@ -763,6 +784,7 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		rep.opt[1] = htonl(tsval);
 		rep.opt[2] = htonl(tsecr);
 		arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
+		offset += 3;
 	}
 
 	/* Swap the send and the receive. */
@@ -774,22 +796,45 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
+
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		int offset = (tsecr) ? 3 : 0;
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
+		used = tcp_extopt_response_prepare(skb, TCPHDR_ACK, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
+		tcp_extopt_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
+
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
+	if (key) {
 		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
 					  (TCPOPT_NOP << 16) |
 					  (TCPOPT_MD5SIG << 8) |
 					  TCPOLEN_MD5SIG);
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len/4;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
 		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
 				    key, ip_hdr(skb)->saddr,
 				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
+
 	arg.flags = reply_flags;
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -893,6 +938,9 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
 */
 static void tcp_v4_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
@@ -1387,6 +1435,10 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 
 	tcp_initialize_rcv_mss(newsk);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
@@ -1907,6 +1959,8 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
 	skb_rbtree_purge(&tp->out_of_order_queue);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Clean up the MD5 key list, if any */
 	if (tp->md5sig_info) {
net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -0
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

#18 lore raw

Re: [MPTCP] [RFC v3 06/15] tcp: Register handlers for extra TCP options

Hi Christoph,

Quoted 43 lines · Christoph Paasch

Hi Mat,

Quoted 34 lines · Mat Martineau
Quoted 19 lines · Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

Quoted 5 lines

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

I'm still on the fence about using a static key at all, especially without rate limiting. It made a lot more sense when used in conjunction with the global option list, and it's not widely used in net/. There are ~60 occurences of static_branch_unlikely in the entire mainline tree today, and this patch adds ~30!

Quoted 5 lines

After I get through the rest of the patch series, I'll take another look at where static_branch_unlikely is used. If the sensive code paths have a known socket type, that would allow use of "if (unlikely(!hlist_empty(&some_sk->tcp_option_list)) && ..." without having to involve a function call to tcp_extopt_get_list().

Quoted 3 lines

I am pretty sure that this would trigger the response from Eric that such a test would add an additional access to a potentially cold cache-line in the hot data-path.

I'd expect him to be thinking about that too, but also that static branches only solve that problem if no socket on the entire system is using extended options.

I think cache hits are only a concern on the output path, since the input path only tries to look up extended optoins if there's an unrecognized option number. And on the output path, we could try to be on a hot cache line. There are a number of tcp_sock members accessed in the hot path.

Quoted 4 lines

For the static-key question and rate-limiting, I would suggest that we see what netdev's feedback will be. We can add this as one of the discussion-points in the cover-letter mail.

It is a good discussion point to bring up, but we don't need static branching in there to do so. I can work up a patch to take out the static branch stuff, which can be squashed in before posting. In case discussion leads us back to static branches, we'll have the patch around to put the code back.

Mat

Quoted 668 lines
Quoted 199 lines
Quoted 199 lines

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

drivers/infiniband/hw/cxgb4/cm.c |   2 +-
include/linux/tcp.h              |  28 ++++
include/net/tcp.h                | 106 ++++++++++++-
net/ipv4/syncookies.c            |   6 +-
net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
net/ipv4/tcp_input.c             |  47 +++++-
net/ipv4/tcp_ipv4.c              |  90 ++++++++---
net/ipv4/tcp_minisocks.c         |  30 +++-
net/ipv4/tcp_output.c            |  37 ++---
net/ipv6/syncookies.c            |   6 +-
net/ipv6/tcp_ipv6.c              |  28 ++++
11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
 * to increase this, although since:
 *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +53 -1
index 3c3744e52cd1..86133aa24fca 100644
@@ -202,6 +202,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
 #define TCPOLEN_EXP_SMC_BASE   6
+#define TCPOLEN_EXP_BASE       6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -403,7 +404,8 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 		int flags, int *addr_len);
 void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
-		       int estab, struct tcp_fastopen_cookie *foc);
+		       int estab, struct tcp_fastopen_cookie *foc,
+		       struct sock *sk);
 const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
@@ -2064,4 +2066,106 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 #if IS_ENABLED(CONFIG_SMC)
 extern struct static_key_false tcp_have_smc;
 #endif
+
+extern struct static_key_false tcp_extopt_enabled;
+struct tcp_extopt_store;
+
+struct tcp_extopt_ops {
+	u32			option_kind;
+	unsigned char		priority;
+	void (*parse)(int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct sock *sk,
+		      struct tcp_extopt_store *store);
+	bool (*check)(struct sock *sk,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct tcp_extopt_store *store);
+	void (*post_process)(struct sock *sk,
+			     struct tcp_options_received *opt_rx,
+			     struct tcp_extopt_store *store);
+	/* Return the number of bytes consumed */
+	unsigned int (*prepare)(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*write)(__be32 *ptr, struct sk_buff *skb,
+			 struct tcp_out_options *opts, struct sock *sk,
+			 struct tcp_extopt_store *store);
+	int (*response_prepare)(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*response_write)(__be32 *ptr, struct sk_buff *orig,
+				  struct tcphdr *th,
+				  struct tcp_out_options *opts,
+				  const struct sock *sk,
+				  struct tcp_extopt_store *store);
+	int (*add_header_len)(const struct sock *listener,
+			      const struct sock *sk,
+			      struct tcp_extopt_store *store);
+	struct tcp_extopt_store *(*copy)(struct sock *listener,
+					 struct request_sock *req,
+					 struct tcp_options_received *opt,
+					 struct tcp_extopt_store *from);
+	struct tcp_extopt_store *(*move)(struct sock *from, struct sock *to,
+					 struct tcp_extopt_store *store);
+	void (*destroy)(struct tcp_extopt_store *store);
+	struct module		*owner;
+};
Quoted 8 lines

include/net/tcp.h static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk) · line 2123

Quoted 5 lines
include/net/tcp.h +5 -0
+
+/* Every extra-option in a TCP-socket gets stored in a _store structure which
+ * has to have tcp_extopt_store as the first element.
+ *
+ * That way, an extra option registers passes this to tcp_register_extopt,

Maybe "That way, code registering extra options can pass this to tcp_register_extopt" ...

Sure, changed it.

Quoted 133 lines
Quoted 133 lines
include/net/tcp.h +47 -0
+ * and the surrounding structure can contain per-option state.
+ */
+struct tcp_extopt_store {
+	struct hlist_node		list;
+	const struct tcp_extopt_ops	*ops;
+};
+
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk);
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx);
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk);
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt);
+
+void tcp_extopt_move(struct sock *from, struct sock *to);
+
+void tcp_extopt_destroy(struct sock *sk);
+
 #endif	/* _TCP_H */
net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +38 -0
index c470fec9062f..6676fc3c0059 100644
@@ -305,6 +305,10 @@ EXPORT_SYMBOL(tcp_have_smc);
 struct percpu_counter tcp_sockets_allocated;
 EXPORT_SYMBOL(tcp_sockets_allocated);
 
+/* Optional TCP option handlers */
+DEFINE_STATIC_KEY_FALSE(tcp_extopt_enabled);
+EXPORT_SYMBOL_GPL(tcp_extopt_enabled);
+
 /*
 * TCP splice context
 */
@@ -416,6 +420,7 @@ void tcp_init_sock(struct sock *sk)
 	tcp_init_xmit_timers(sk);
 	INIT_LIST_HEAD(&tp->tsq_node);
 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
+	INIT_HLIST_HEAD(&tp->tcp_option_list);
 
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
@@ -3473,6 +3478,335 @@ EXPORT_SYMBOL(tcp_md5_hash_key);
 
 #endif
 
+static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
+{
+	if (sk_fullsock(sk))
+		return &tcp_sk(sk)->tcp_option_list;
+	else if (sk->sk_state == TCP_NEW_SYN_RECV)
+		return &tcp_rsk(inet_reqsk(sk))->tcp_option_list;
+	else if (sk->sk_state == TCP_TIME_WAIT)
+		return &tcp_twsk(sk)->tcp_option_list;
+
+	return NULL;
+}
+
+/* Caller must ensure that rcu is locked */
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == kind)
+			return entry;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_find_kind);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk)
+{
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3516

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+	struct tcp_extopt_store *entry;
+
+	rcu_read_lock_bh();

I'm curious, why the _bh variant in these functions? It would require use of hlist_for_each_entry_rcu_bh to traverse the list, in tcp_extopt_find_kind or in the functions below. It's ok for rcu_read_lock() to be used in interrupt context.

Yes, I think you are right after researching more. I guess I misunderstood something.

I thought that I need to call the _bh variant to disable interrupts. Because, the TCP-input code path can now be called from both contexts (softirq and with interrupts enabled).

I will change all of this to rcu_read_lock() and rcu_read_unlock().

Quoted 201 lines
Quoted 201 lines
net/ipv4/tcp.c +200 -0
+	entry = tcp_extopt_find_kind(opcode, sk);
+
+	if (entry && entry->ops->parse)
+		entry->ops->parse(opsize, opptr, skb, opt_rx, sk, entry);
+	rcu_read_unlock_bh();
+}
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	bool drop = false;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		bool ret = false;
+
+		if (entry->ops->check)
+			ret = entry->ops->check(sk, skb, opt_rx, entry);
+
+		if (ret)
+			drop = true;
+	}
+	rcu_read_unlock_bh();
+
+	return drop;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_check);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->post_process)
+			entry->ops->post_process(sk, opt_rx, entry);
+	}
+	rcu_read_unlock_bh();
+}
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->prepare))
+			continue;
+
+		used += entry->ops->prepare(skb, flags, remaining - used, opts,
+					    sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->write))
+			continue;
+
+		ptr = entry->ops->write(ptr, skb, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_write);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		int ret;
+
+		if (unlikely(!entry->ops->response_prepare))
+			continue;
+
+		ret = entry->ops->response_prepare(orig, flags,
+						   remaining - used, opts,
+						   sk, entry);
+
+		used += ret;
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_prepare);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->response_write))
+			continue;
+
+		ptr = entry->ops->response_write(ptr, orig, th, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_write);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	int tcp_header_len = 0;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, &tcp_sk(sk)->tcp_option_list, list) {
+		if (unlikely(!entry->ops->add_header_len))
+			continue;
+
+		tcp_header_len += entry->ops->add_header_len(listener, sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return tcp_header_len;
+}
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk)
+{
+	struct hlist_node *add_before = NULL;
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	int ret = 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	if (!store->ops->option_kind)
+		return -EINVAL;
+
+	if (!try_module_get(store->ops->owner))
+		return -ENOENT;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == store->ops->option_kind) {
+			rcu_read_unlock_bh();
+			pr_notice("Option kind %u already registered\n",
+				  store->ops->option_kind);
+			module_put(store->ops->owner);
+			return -EEXIST;
+		}
+
+		if (entry->ops->priority <= store->ops->priority)
+			add_before = &entry->list;
+	}
+
+	if (add_before)
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3719

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+		hlist_add_behind_rcu(&store->list, add_before);
+	else
+		hlist_add_head_rcu(&store->list, lhead);

I'm assuming the socket lock is being used to synchonize modifications to the options list and must be held before this function is called. It would help to add comments indicating which functions require the socket lock to be held (here or in include/net/tcp.h)

I will add a comment in tcp.h and here.

Quoted 3 lines

What's more, rcu_read_lock/unlock aren't needed when the write lock (the socket lock in this case) is held - so no need to call those here or in other functions where the lock is required before calling.

Yep - will remove the rcu_read_lock/unlock.

Quoted 80 lines
Quoted 80 lines
net/ipv4/tcp.c +80 -0
+	rcu_read_unlock_bh();
+	pr_debug("Option kind %u registered\n", store->ops->option_kind);
+
+	static_branch_inc(&tcp_extopt_enabled);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tcp_register_extopt);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *from, *to;
+
+	from = tcp_extopt_get_list(listener);
+	to = tcp_extopt_get_list(req_to_sk(req));
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, from, list) {
+		struct tcp_extopt_store *new;
+
+		if (!try_module_get(entry->ops->owner)) {
+			pr_err("%s Module get failed while copying\n", __func__);
+			continue;
+		}
+
+		new = entry->ops->copy(listener, req, opt, entry);
+		if (!new) {
+			module_put(entry->ops->owner);
+			continue;
+		}
+
+		hlist_add_tail_rcu(&new->list, to);
+
+		static_branch_inc(&tcp_extopt_enabled);
+	}
+	rcu_read_unlock_bh();
+}
+
+void tcp_extopt_move(struct sock *from, struct sock *to)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lfrom, *lto;
+	struct hlist_node *tmp;
+
+	lfrom = tcp_extopt_get_list(from);
+	lto = tcp_extopt_get_list(to);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lfrom, list) {
+		hlist_del_rcu(&entry->list);
+
+		if (entry->ops->move) {
+			entry = entry->ops->move(from, to, entry);
+			if (!entry)
+				continue;
+		}
+
+		hlist_add_tail_rcu(&entry->list, lto);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_move);
+
+void tcp_extopt_destroy(struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	struct hlist_node *tmp;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lhead, list) {
+		struct module *owner = entry->ops->owner;
+
+		hlist_del_rcu(&entry->list);
+
+		entry->ops->destroy(entry);
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3804

Quoted 4 lines
net/ipv4/tcp.c +4 -0
+
+		static_branch_dec(&tcp_extopt_enabled);
+
+		module_put(owner);

Is it ok to allow the module to unload before synchronize_rcu() or synchronize_net()? Maybe the static_branch_dec and module_put needs to be invoked via call_rcu?

Quoted 629 lines

I think it's fine. From what I see, before really unloading the module (coming from the delete_module() system-call), it does a synchronize_sched().

Christoph

Quoted 621 lines

Mat

Quoted 609 lines
net/ipv4/tcp.c +5 -1
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_destroy);
+
 void tcp_done(struct sock *sk)
 {
 	struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
@@ -3622,7 +3956,6 @@ void __init tcp_init(void)
 		INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
 	}
 
-
 	cnt = tcp_hashinfo.ehash_mask + 1;
 	sysctl_tcp_max_orphans = cnt / 2;
 
net/ipv4/tcp_input.c +13 -2
index eedb355a0533..d4d3be8a496c 100644
@@ -3701,7 +3701,7 @@ static int smc_parse_options(const struct tcphdr *th,
 void tcp_parse_options(const struct net *net,
 		       const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx, int estab,
-		       struct tcp_fastopen_cookie *foc)
+		       struct tcp_fastopen_cookie *foc, struct sock *sk)
 {
 	const unsigned char *ptr;
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -3804,6 +3804,17 @@ void tcp_parse_options(const struct net *net,
 				else if (smc_parse_options(th, opt_rx, ptr,
 							   opsize))
 					break;
+				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
+					 opsize >= TCPOLEN_EXP_BASE)
+					tcp_extopt_parse(get_unaligned_be32(ptr),
+							 opsize, ptr, skb,
+							 opt_rx, sk);
+				break;
+
+			default:
+				if (static_branch_unlikely(&tcp_extopt_enabled))
+					tcp_extopt_parse(opcode, opsize, ptr,
+							 skb, opt_rx, sk);
 				break;
 
 			}
@@ -3854,12 +3865,13 @@ static bool tcp_fast_parse_options(const struct net *net,
 			goto extra_opt_check;
 	}
 
-	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
+	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL, tcp_to_sk(tp));
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 

extra_opt_check:

net/ipv4/tcp_input.c +2 -1
-	/* ToDo - will be added here */
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		return tcp_extopt_check(tcp_to_sk(tp), skb, &tp->rx_opt);
	return false;
}
net/ipv4/tcp_input.c +26 -3
@@ -5333,6 +5345,10 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 	tp->rx_opt.saw_tstamp = 0;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    !hlist_empty(&tp->tcp_option_list))
+		goto slow_path;
+
 	/*	pred_flags is 0xS?10 << 16 + snd_wnd
 	 *	if header_prediction is to be made
 	 *	'S' will always be tp->tcp_header_len >> 2
@@ -5520,7 +5536,7 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 		/* Get original SYNACK MSS value if user MSS sets mss_clamp */
 		tcp_clear_options(&opt);
 		opt.user_mss = opt.mss_clamp = 0;
-		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL, sk);
 		mss = opt.mss_clamp;
 	}
 
@@ -5583,10 +5599,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 	int saved_clamp = tp->rx_opt.mss_clamp;
 	bool fastopen_fail;
 
-	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
+	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc, sk);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tp->rx_opt))
+		goto discard;
+
 	if (th->ack) {
 		/* rfc793:
 		 * "If the state is SYN-SENT then
@@ -5669,6 +5689,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 			tp->tcp_header_len = sizeof(struct tcphdr);
 		}
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
@@ -5762,6 +5785,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 
 		tcp_ecn_rcv_syn(tp, th);
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_mtup_init(sk);
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
@@ -6245,12 +6271,17 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
 	tmp_opt.user_mss  = tp->rx_opt.user_mss;
 	tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
-			  want_cookie ? NULL : &foc);
+			  want_cookie ? NULL : &foc, sk);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tmp_opt))
+		goto drop_and_free;
 
 	if (want_cookie && !tmp_opt.saw_tstamp)
 		tcp_clear_options(&tmp_opt);
@@ -6311,6 +6342,10 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 		tcp_reqsk_record_syn(sk, req, skb);
 		fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
 	}
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_copy(sk, req, &tmp_opt);
+
 	if (fastopen_sk) {
 		af_ops->send_synack(fastopen_sk, dst, &fl, req,
 				    &foc, TCP_SYNACK_FASTOPEN);
net/ipv4/tcp_ipv4.c +72 -18
index 94e28350f420..ad87dd2db092 100644
@@ -600,9 +600,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-#ifdef CONFIG_TCP_MD5SIG
-		__be32 opt[(TCPOLEN_MD5SIG_ALIGNED >> 2)];
-#endif
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct ip_reply_arg arg;
 #ifdef CONFIG_TCP_MD5SIG
@@ -613,6 +611,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	struct sock *sk1 = NULL;
 #endif
 	struct net *net;
+	int offset = 0;
 
 	/* Never send a reset in response to a reset. */
 	if (th->rst)
@@ -678,19 +677,44 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 			goto out;
 
 	}
+#endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.opt);
+#ifdef CONFIG_TCP_MD5SIG
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
 
+		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
+
+		tcp_extopt_response_write(&rep.opt[0], skb, &rep.th, &opts, sk);
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
 	if (key) {
-		rep.opt[0] = htonl((TCPOPT_NOP << 24) |
-				   (TCPOPT_NOP << 16) |
-				   (TCPOPT_MD5SIG << 8) |
-				   TCPOLEN_MD5SIG);
+		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
+					  (TCPOPT_NOP << 16) |
+					  (TCPOPT_MD5SIG << 8) |
+					  TCPOLEN_MD5SIG);
 		/* Update length and the length the header thinks exists */
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
 
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
-				     key, ip_hdr(skb)->saddr,
-				     ip_hdr(skb)->daddr, &rep.th);
+		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
+				    key, ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
@@ -742,14 +766,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-		__be32 opt[(TCPOLEN_TSTAMP_ALIGNED >> 2)
-#ifdef CONFIG_TCP_MD5SIG
-			   + (TCPOLEN_MD5SIG_ALIGNED >> 2)
-#endif
-			];
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
+	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
@@ -763,6 +784,7 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		rep.opt[1] = htonl(tsval);
 		rep.opt[2] = htonl(tsecr);
 		arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
+		offset += 3;
 	}
 
 	/* Swap the send and the receive. */
@@ -774,22 +796,45 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
+
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		int offset = (tsecr) ? 3 : 0;
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
+		used = tcp_extopt_response_prepare(skb, TCPHDR_ACK, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
+		tcp_extopt_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
+
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
+	if (key) {
 		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
 					  (TCPOPT_NOP << 16) |
 					  (TCPOPT_MD5SIG << 8) |
 					  TCPOLEN_MD5SIG);
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len/4;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
 		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
 				    key, ip_hdr(skb)->saddr,
 				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
+
 	arg.flags = reply_flags;
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -893,6 +938,9 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
 */
 static void tcp_v4_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
@@ -1387,6 +1435,10 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 
 	tcp_initialize_rcv_mss(newsk);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
@@ -1907,6 +1959,8 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
 	skb_rbtree_purge(&tp->out_of_order_queue);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Clean up the MD5 key list, if any */
 	if (tp->md5sig_info) {
net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -1
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
--
Quoted 3 lines

2.15.0

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

author
#19 lore raw

Re: [MPTCP] [RFC v3 06/15] tcp: Register handlers for extra TCP options

Quoted 52 lines · Mat Martineau

Hi Christoph,

Quoted 43 lines · Christoph Paasch

Hi Mat,

Quoted 34 lines · Mat Martineau
Quoted 19 lines · Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

Quoted 5 lines

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

I'm still on the fence about using a static key at all, especially without rate limiting. It made a lot more sense when used in conjunction with the global option list, and it's not widely used in net/. There are ~60 occurences of static_branch_unlikely in the entire mainline tree today, and this patch adds ~30!

Quoted 5 lines

After I get through the rest of the patch series, I'll take another look at where static_branch_unlikely is used. If the sensive code paths have a known socket type, that would allow use of "if (unlikely(!hlist_empty(&some_sk->tcp_option_list)) && ..." without having to involve a function call to tcp_extopt_get_list().

Quoted 3 lines

I am pretty sure that this would trigger the response from Eric that such a test would add an additional access to a potentially cold cache-line in the hot data-path.

Quoted 8 lines

I'd expect him to be thinking about that too, but also that static branches only solve that problem if no socket on the entire system is using extended options.

I think cache hits are only a concern on the output path, since the input path only tries to look up extended optoins if there's an unrecognized option number. And on the output path, we could try to be on a hot cache line. There are a number of tcp_sock members accessed in the hot path.

Branching is also a concern. With static keys the branching is avoided as well.

Quoted 7 lines
Quoted 6 lines

For the static-key question and rate-limiting, I would suggest that we see what netdev's feedback will be. We can add this as one of the discussion-points in the cover-letter mail.

Quoted 5 lines

It is a good discussion point to bring up, but we don't need static branching in there to do so. I can work up a patch to take out the static branch stuff, which can be squashed in before posting. In case discussion leads us back to static branches, we'll have the patch around to put the code back.

Ok, let's go with this. I will quickly change that and post a last final version here on the list.

Do you want to add your "Reviewed-by"-tag to the patches?

Christoph

Quoted 1,305 lines

Mat

Quoted 668 lines
Quoted 199 lines
Quoted 199 lines

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

drivers/infiniband/hw/cxgb4/cm.c |   2 +-
include/linux/tcp.h              |  28 ++++
include/net/tcp.h                | 106 ++++++++++++-
net/ipv4/syncookies.c            |   6 +-
net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
net/ipv4/tcp_input.c             |  47 +++++-
net/ipv4/tcp_ipv4.c              |  90 ++++++++---
net/ipv4/tcp_minisocks.c         |  30 +++-
net/ipv4/tcp_output.c            |  37 ++---
net/ipv6/syncookies.c            |   6 +-
net/ipv6/tcp_ipv6.c              |  28 ++++
11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
 * to increase this, although since:
 *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +53 -1
index 3c3744e52cd1..86133aa24fca 100644
@@ -202,6 +202,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
 #define TCPOLEN_EXP_SMC_BASE   6
+#define TCPOLEN_EXP_BASE       6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -403,7 +404,8 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 		int flags, int *addr_len);
 void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
-		       int estab, struct tcp_fastopen_cookie *foc);
+		       int estab, struct tcp_fastopen_cookie *foc,
+		       struct sock *sk);
 const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
@@ -2064,4 +2066,106 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 #if IS_ENABLED(CONFIG_SMC)
 extern struct static_key_false tcp_have_smc;
 #endif
+
+extern struct static_key_false tcp_extopt_enabled;
+struct tcp_extopt_store;
+
+struct tcp_extopt_ops {
+	u32			option_kind;
+	unsigned char		priority;
+	void (*parse)(int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct sock *sk,
+		      struct tcp_extopt_store *store);
+	bool (*check)(struct sock *sk,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct tcp_extopt_store *store);
+	void (*post_process)(struct sock *sk,
+			     struct tcp_options_received *opt_rx,
+			     struct tcp_extopt_store *store);
+	/* Return the number of bytes consumed */
+	unsigned int (*prepare)(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*write)(__be32 *ptr, struct sk_buff *skb,
+			 struct tcp_out_options *opts, struct sock *sk,
+			 struct tcp_extopt_store *store);
+	int (*response_prepare)(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*response_write)(__be32 *ptr, struct sk_buff *orig,
+				  struct tcphdr *th,
+				  struct tcp_out_options *opts,
+				  const struct sock *sk,
+				  struct tcp_extopt_store *store);
+	int (*add_header_len)(const struct sock *listener,
+			      const struct sock *sk,
+			      struct tcp_extopt_store *store);
+	struct tcp_extopt_store *(*copy)(struct sock *listener,
+					 struct request_sock *req,
+					 struct tcp_options_received *opt,
+					 struct tcp_extopt_store *from);
+	struct tcp_extopt_store *(*move)(struct sock *from, struct sock *to,
+					 struct tcp_extopt_store *store);
+	void (*destroy)(struct tcp_extopt_store *store);
+	struct module		*owner;
+};
Quoted 8 lines

include/net/tcp.h static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk) · line 2123

Quoted 5 lines
include/net/tcp.h +5 -0
+
+/* Every extra-option in a TCP-socket gets stored in a _store structure which
+ * has to have tcp_extopt_store as the first element.
+ *
+ * That way, an extra option registers passes this to tcp_register_extopt,

Maybe "That way, code registering extra options can pass this to tcp_register_extopt" ...

Sure, changed it.

Quoted 133 lines
Quoted 133 lines
include/net/tcp.h +47 -0
+ * and the surrounding structure can contain per-option state.
+ */
+struct tcp_extopt_store {
+	struct hlist_node		list;
+	const struct tcp_extopt_ops	*ops;
+};
+
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk);
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx);
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk);
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt);
+
+void tcp_extopt_move(struct sock *from, struct sock *to);
+
+void tcp_extopt_destroy(struct sock *sk);
+
 #endif	/* _TCP_H */
net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +38 -0
index c470fec9062f..6676fc3c0059 100644
@@ -305,6 +305,10 @@ EXPORT_SYMBOL(tcp_have_smc);
 struct percpu_counter tcp_sockets_allocated;
 EXPORT_SYMBOL(tcp_sockets_allocated);
 
+/* Optional TCP option handlers */
+DEFINE_STATIC_KEY_FALSE(tcp_extopt_enabled);
+EXPORT_SYMBOL_GPL(tcp_extopt_enabled);
+
 /*
 * TCP splice context
 */
@@ -416,6 +420,7 @@ void tcp_init_sock(struct sock *sk)
 	tcp_init_xmit_timers(sk);
 	INIT_LIST_HEAD(&tp->tsq_node);
 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
+	INIT_HLIST_HEAD(&tp->tcp_option_list);
 
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
@@ -3473,6 +3478,335 @@ EXPORT_SYMBOL(tcp_md5_hash_key);
 
 #endif
 
+static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
+{
+	if (sk_fullsock(sk))
+		return &tcp_sk(sk)->tcp_option_list;
+	else if (sk->sk_state == TCP_NEW_SYN_RECV)
+		return &tcp_rsk(inet_reqsk(sk))->tcp_option_list;
+	else if (sk->sk_state == TCP_TIME_WAIT)
+		return &tcp_twsk(sk)->tcp_option_list;
+
+	return NULL;
+}
+
+/* Caller must ensure that rcu is locked */
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == kind)
+			return entry;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_find_kind);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk)
+{
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3516

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+	struct tcp_extopt_store *entry;
+
+	rcu_read_lock_bh();

I'm curious, why the _bh variant in these functions? It would require use of hlist_for_each_entry_rcu_bh to traverse the list, in tcp_extopt_find_kind or in the functions below. It's ok for rcu_read_lock() to be used in interrupt context.

Yes, I think you are right after researching more. I guess I misunderstood something.

I thought that I need to call the _bh variant to disable interrupts. Because, the TCP-input code path can now be called from both contexts (softirq and with interrupts enabled).

I will change all of this to rcu_read_lock() and rcu_read_unlock().

Quoted 201 lines
Quoted 201 lines
net/ipv4/tcp.c +200 -0
+	entry = tcp_extopt_find_kind(opcode, sk);
+
+	if (entry && entry->ops->parse)
+		entry->ops->parse(opsize, opptr, skb, opt_rx, sk, entry);
+	rcu_read_unlock_bh();
+}
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	bool drop = false;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		bool ret = false;
+
+		if (entry->ops->check)
+			ret = entry->ops->check(sk, skb, opt_rx, entry);
+
+		if (ret)
+			drop = true;
+	}
+	rcu_read_unlock_bh();
+
+	return drop;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_check);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->post_process)
+			entry->ops->post_process(sk, opt_rx, entry);
+	}
+	rcu_read_unlock_bh();
+}
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->prepare))
+			continue;
+
+		used += entry->ops->prepare(skb, flags, remaining - used, opts,
+					    sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->write))
+			continue;
+
+		ptr = entry->ops->write(ptr, skb, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_write);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		int ret;
+
+		if (unlikely(!entry->ops->response_prepare))
+			continue;
+
+		ret = entry->ops->response_prepare(orig, flags,
+						   remaining - used, opts,
+						   sk, entry);
+
+		used += ret;
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_prepare);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->response_write))
+			continue;
+
+		ptr = entry->ops->response_write(ptr, orig, th, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_write);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	int tcp_header_len = 0;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, &tcp_sk(sk)->tcp_option_list, list) {
+		if (unlikely(!entry->ops->add_header_len))
+			continue;
+
+		tcp_header_len += entry->ops->add_header_len(listener, sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return tcp_header_len;
+}
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk)
+{
+	struct hlist_node *add_before = NULL;
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	int ret = 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	if (!store->ops->option_kind)
+		return -EINVAL;
+
+	if (!try_module_get(store->ops->owner))
+		return -ENOENT;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == store->ops->option_kind) {
+			rcu_read_unlock_bh();
+			pr_notice("Option kind %u already registered\n",
+				  store->ops->option_kind);
+			module_put(store->ops->owner);
+			return -EEXIST;
+		}
+
+		if (entry->ops->priority <= store->ops->priority)
+			add_before = &entry->list;
+	}
+
+	if (add_before)
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3719

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+		hlist_add_behind_rcu(&store->list, add_before);
+	else
+		hlist_add_head_rcu(&store->list, lhead);

I'm assuming the socket lock is being used to synchonize modifications to the options list and must be held before this function is called. It would help to add comments indicating which functions require the socket lock to be held (here or in include/net/tcp.h)

I will add a comment in tcp.h and here.

Quoted 3 lines

What's more, rcu_read_lock/unlock aren't needed when the write lock (the socket lock in this case) is held - so no need to call those here or in other functions where the lock is required before calling.

Yep - will remove the rcu_read_lock/unlock.

Quoted 80 lines
Quoted 80 lines
net/ipv4/tcp.c +80 -0
+	rcu_read_unlock_bh();
+	pr_debug("Option kind %u registered\n", store->ops->option_kind);
+
+	static_branch_inc(&tcp_extopt_enabled);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tcp_register_extopt);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *from, *to;
+
+	from = tcp_extopt_get_list(listener);
+	to = tcp_extopt_get_list(req_to_sk(req));
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, from, list) {
+		struct tcp_extopt_store *new;
+
+		if (!try_module_get(entry->ops->owner)) {
+			pr_err("%s Module get failed while copying\n", __func__);
+			continue;
+		}
+
+		new = entry->ops->copy(listener, req, opt, entry);
+		if (!new) {
+			module_put(entry->ops->owner);
+			continue;
+		}
+
+		hlist_add_tail_rcu(&new->list, to);
+
+		static_branch_inc(&tcp_extopt_enabled);
+	}
+	rcu_read_unlock_bh();
+}
+
+void tcp_extopt_move(struct sock *from, struct sock *to)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lfrom, *lto;
+	struct hlist_node *tmp;
+
+	lfrom = tcp_extopt_get_list(from);
+	lto = tcp_extopt_get_list(to);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lfrom, list) {
+		hlist_del_rcu(&entry->list);
+
+		if (entry->ops->move) {
+			entry = entry->ops->move(from, to, entry);
+			if (!entry)
+				continue;
+		}
+
+		hlist_add_tail_rcu(&entry->list, lto);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_move);
+
+void tcp_extopt_destroy(struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	struct hlist_node *tmp;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lhead, list) {
+		struct module *owner = entry->ops->owner;
+
+		hlist_del_rcu(&entry->list);
+
+		entry->ops->destroy(entry);
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3804

Quoted 4 lines
net/ipv4/tcp.c +4 -0
+
+		static_branch_dec(&tcp_extopt_enabled);
+
+		module_put(owner);

Is it ok to allow the module to unload before synchronize_rcu() or synchronize_net()? Maybe the static_branch_dec and module_put needs to be invoked via call_rcu?

Quoted 629 lines

I think it's fine. From what I see, before really unloading the module (coming from the delete_module() system-call), it does a synchronize_sched().

Christoph

Quoted 621 lines

Mat

Quoted 609 lines
net/ipv4/tcp.c +5 -1
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_destroy);
+
 void tcp_done(struct sock *sk)
 {
 	struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
@@ -3622,7 +3956,6 @@ void __init tcp_init(void)
 		INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
 	}
 
-
 	cnt = tcp_hashinfo.ehash_mask + 1;
 	sysctl_tcp_max_orphans = cnt / 2;
 
net/ipv4/tcp_input.c +13 -2
index eedb355a0533..d4d3be8a496c 100644
@@ -3701,7 +3701,7 @@ static int smc_parse_options(const struct tcphdr *th,
 void tcp_parse_options(const struct net *net,
 		       const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx, int estab,
-		       struct tcp_fastopen_cookie *foc)
+		       struct tcp_fastopen_cookie *foc, struct sock *sk)
 {
 	const unsigned char *ptr;
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -3804,6 +3804,17 @@ void tcp_parse_options(const struct net *net,
 				else if (smc_parse_options(th, opt_rx, ptr,
 							   opsize))
 					break;
+				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
+					 opsize >= TCPOLEN_EXP_BASE)
+					tcp_extopt_parse(get_unaligned_be32(ptr),
+							 opsize, ptr, skb,
+							 opt_rx, sk);
+				break;
+
+			default:
+				if (static_branch_unlikely(&tcp_extopt_enabled))
+					tcp_extopt_parse(opcode, opsize, ptr,
+							 skb, opt_rx, sk);
 				break;
 
 			}
@@ -3854,12 +3865,13 @@ static bool tcp_fast_parse_options(const struct net *net,
 			goto extra_opt_check;
 	}
 
-	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
+	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL, tcp_to_sk(tp));
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 

extra_opt_check:

net/ipv4/tcp_input.c +2 -1
-	/* ToDo - will be added here */
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		return tcp_extopt_check(tcp_to_sk(tp), skb, &tp->rx_opt);
	return false;
}
net/ipv4/tcp_input.c +26 -3
@@ -5333,6 +5345,10 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 	tp->rx_opt.saw_tstamp = 0;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    !hlist_empty(&tp->tcp_option_list))
+		goto slow_path;
+
 	/*	pred_flags is 0xS?10 << 16 + snd_wnd
 	 *	if header_prediction is to be made
 	 *	'S' will always be tp->tcp_header_len >> 2
@@ -5520,7 +5536,7 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 		/* Get original SYNACK MSS value if user MSS sets mss_clamp */
 		tcp_clear_options(&opt);
 		opt.user_mss = opt.mss_clamp = 0;
-		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL, sk);
 		mss = opt.mss_clamp;
 	}
 
@@ -5583,10 +5599,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 	int saved_clamp = tp->rx_opt.mss_clamp;
 	bool fastopen_fail;
 
-	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
+	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc, sk);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tp->rx_opt))
+		goto discard;
+
 	if (th->ack) {
 		/* rfc793:
 		 * "If the state is SYN-SENT then
@@ -5669,6 +5689,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 			tp->tcp_header_len = sizeof(struct tcphdr);
 		}
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
@@ -5762,6 +5785,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 
 		tcp_ecn_rcv_syn(tp, th);
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_mtup_init(sk);
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
@@ -6245,12 +6271,17 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
 	tmp_opt.user_mss  = tp->rx_opt.user_mss;
 	tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
-			  want_cookie ? NULL : &foc);
+			  want_cookie ? NULL : &foc, sk);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tmp_opt))
+		goto drop_and_free;
 
 	if (want_cookie && !tmp_opt.saw_tstamp)
 		tcp_clear_options(&tmp_opt);
@@ -6311,6 +6342,10 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 		tcp_reqsk_record_syn(sk, req, skb);
 		fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
 	}
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_copy(sk, req, &tmp_opt);
+
 	if (fastopen_sk) {
 		af_ops->send_synack(fastopen_sk, dst, &fl, req,
 				    &foc, TCP_SYNACK_FASTOPEN);
net/ipv4/tcp_ipv4.c +72 -18
index 94e28350f420..ad87dd2db092 100644
@@ -600,9 +600,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-#ifdef CONFIG_TCP_MD5SIG
-		__be32 opt[(TCPOLEN_MD5SIG_ALIGNED >> 2)];
-#endif
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct ip_reply_arg arg;
 #ifdef CONFIG_TCP_MD5SIG
@@ -613,6 +611,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	struct sock *sk1 = NULL;
 #endif
 	struct net *net;
+	int offset = 0;
 
 	/* Never send a reset in response to a reset. */
 	if (th->rst)
@@ -678,19 +677,44 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 			goto out;
 
 	}
+#endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.opt);
+#ifdef CONFIG_TCP_MD5SIG
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
 
+		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
+
+		tcp_extopt_response_write(&rep.opt[0], skb, &rep.th, &opts, sk);
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
 	if (key) {
-		rep.opt[0] = htonl((TCPOPT_NOP << 24) |
-				   (TCPOPT_NOP << 16) |
-				   (TCPOPT_MD5SIG << 8) |
-				   TCPOLEN_MD5SIG);
+		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
+					  (TCPOPT_NOP << 16) |
+					  (TCPOPT_MD5SIG << 8) |
+					  TCPOLEN_MD5SIG);
 		/* Update length and the length the header thinks exists */
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
 
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
-				     key, ip_hdr(skb)->saddr,
-				     ip_hdr(skb)->daddr, &rep.th);
+		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
+				    key, ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
@@ -742,14 +766,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-		__be32 opt[(TCPOLEN_TSTAMP_ALIGNED >> 2)
-#ifdef CONFIG_TCP_MD5SIG
-			   + (TCPOLEN_MD5SIG_ALIGNED >> 2)
-#endif
-			];
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
+	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
@@ -763,6 +784,7 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		rep.opt[1] = htonl(tsval);
 		rep.opt[2] = htonl(tsecr);
 		arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
+		offset += 3;
 	}
 
 	/* Swap the send and the receive. */
@@ -774,22 +796,45 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
+
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		int offset = (tsecr) ? 3 : 0;
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
+		used = tcp_extopt_response_prepare(skb, TCPHDR_ACK, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
+		tcp_extopt_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
+
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
+	if (key) {
 		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
 					  (TCPOPT_NOP << 16) |
 					  (TCPOPT_MD5SIG << 8) |
 					  TCPOLEN_MD5SIG);
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len/4;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
 		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
 				    key, ip_hdr(skb)->saddr,
 				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
+
 	arg.flags = reply_flags;
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -893,6 +938,9 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
 */
 static void tcp_v4_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
@@ -1387,6 +1435,10 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 
 	tcp_initialize_rcv_mss(newsk);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
@@ -1907,6 +1959,8 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
 	skb_rbtree_purge(&tp->out_of_order_queue);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Clean up the MD5 key list, if any */
 	if (tp->md5sig_info) {
net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -1
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
--
Quoted 3 lines

2.15.0

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

#20 lore raw

Re: [MPTCP] [RFC v3 06/15] tcp: Register handlers for extra TCP options

Quoted 78 lines · Christoph Paasch
Quoted 52 lines · Mat Martineau

Hi Christoph,

Quoted 43 lines · Christoph Paasch

Hi Mat,

Quoted 34 lines · Mat Martineau
Quoted 19 lines · Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

Quoted 5 lines

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

I'm still on the fence about using a static key at all, especially without rate limiting. It made a lot more sense when used in conjunction with the global option list, and it's not widely used in net/. There are ~60 occurences of static_branch_unlikely in the entire mainline tree today, and this patch adds ~30!

Quoted 5 lines

After I get through the rest of the patch series, I'll take another look at where static_branch_unlikely is used. If the sensive code paths have a known socket type, that would allow use of "if (unlikely(!hlist_empty(&some_sk->tcp_option_list)) && ..." without having to involve a function call to tcp_extopt_get_list().

Quoted 3 lines

I am pretty sure that this would trigger the response from Eric that such a test would add an additional access to a potentially cold cache-line in the hot data-path.

Quoted 8 lines

I'd expect him to be thinking about that too, but also that static branches only solve that problem if no socket on the entire system is using extended options.

I think cache hits are only a concern on the output path, since the input path only tries to look up extended optoins if there's an unrecognized option number. And on the output path, we could try to be on a hot cache line. There are a number of tcp_sock members accessed in the hot path.

Branching is also a concern. With static keys the branching is avoided as well.

Quoted 7 lines
Quoted 6 lines

For the static-key question and rate-limiting, I would suggest that we see what netdev's feedback will be. We can add this as one of the discussion-points in the cover-letter mail.

Quoted 5 lines

It is a good discussion point to bring up, but we don't need static branching in there to do so. I can work up a patch to take out the static branch stuff, which can be squashed in before posting. In case discussion leads us back to static branches, we'll have the patch around to put the code back.

Quoted 4 lines

Ok, let's go with this. I will quickly change that and post a last final version here on the list.

Do you want to add your "Reviewed-by"-tag to the patches?

Yes (other than the one with my signoff). I'm finishing up the last two patches right now.

Mat

Quoted 1,308 lines

Christoph

Quoted 1,305 lines

Mat

Quoted 668 lines
Quoted 199 lines
Quoted 199 lines

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

drivers/infiniband/hw/cxgb4/cm.c |   2 +-
include/linux/tcp.h              |  28 ++++
include/net/tcp.h                | 106 ++++++++++++-
net/ipv4/syncookies.c            |   6 +-
net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
net/ipv4/tcp_input.c             |  47 +++++-
net/ipv4/tcp_ipv4.c              |  90 ++++++++---
net/ipv4/tcp_minisocks.c         |  30 +++-
net/ipv4/tcp_output.c            |  37 ++---
net/ipv6/syncookies.c            |   6 +-
net/ipv6/tcp_ipv6.c              |  28 ++++
11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
 * to increase this, although since:
 *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +53 -1
index 3c3744e52cd1..86133aa24fca 100644
@@ -202,6 +202,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
 #define TCPOLEN_EXP_SMC_BASE   6
+#define TCPOLEN_EXP_BASE       6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -403,7 +404,8 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 		int flags, int *addr_len);
 void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
-		       int estab, struct tcp_fastopen_cookie *foc);
+		       int estab, struct tcp_fastopen_cookie *foc,
+		       struct sock *sk);
 const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
@@ -2064,4 +2066,106 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 #if IS_ENABLED(CONFIG_SMC)
 extern struct static_key_false tcp_have_smc;
 #endif
+
+extern struct static_key_false tcp_extopt_enabled;
+struct tcp_extopt_store;
+
+struct tcp_extopt_ops {
+	u32			option_kind;
+	unsigned char		priority;
+	void (*parse)(int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct sock *sk,
+		      struct tcp_extopt_store *store);
+	bool (*check)(struct sock *sk,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct tcp_extopt_store *store);
+	void (*post_process)(struct sock *sk,
+			     struct tcp_options_received *opt_rx,
+			     struct tcp_extopt_store *store);
+	/* Return the number of bytes consumed */
+	unsigned int (*prepare)(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*write)(__be32 *ptr, struct sk_buff *skb,
+			 struct tcp_out_options *opts, struct sock *sk,
+			 struct tcp_extopt_store *store);
+	int (*response_prepare)(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*response_write)(__be32 *ptr, struct sk_buff *orig,
+				  struct tcphdr *th,
+				  struct tcp_out_options *opts,
+				  const struct sock *sk,
+				  struct tcp_extopt_store *store);
+	int (*add_header_len)(const struct sock *listener,
+			      const struct sock *sk,
+			      struct tcp_extopt_store *store);
+	struct tcp_extopt_store *(*copy)(struct sock *listener,
+					 struct request_sock *req,
+					 struct tcp_options_received *opt,
+					 struct tcp_extopt_store *from);
+	struct tcp_extopt_store *(*move)(struct sock *from, struct sock *to,
+					 struct tcp_extopt_store *store);
+	void (*destroy)(struct tcp_extopt_store *store);
+	struct module		*owner;
+};
Quoted 8 lines

include/net/tcp.h static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk) · line 2123

Quoted 5 lines
include/net/tcp.h +5 -0
+
+/* Every extra-option in a TCP-socket gets stored in a _store structure which
+ * has to have tcp_extopt_store as the first element.
+ *
+ * That way, an extra option registers passes this to tcp_register_extopt,

Maybe "That way, code registering extra options can pass this to tcp_register_extopt" ...

Sure, changed it.

Quoted 133 lines
Quoted 133 lines
include/net/tcp.h +47 -0
+ * and the surrounding structure can contain per-option state.
+ */
+struct tcp_extopt_store {
+	struct hlist_node		list;
+	const struct tcp_extopt_ops	*ops;
+};
+
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk);
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx);
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk);
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt);
+
+void tcp_extopt_move(struct sock *from, struct sock *to);
+
+void tcp_extopt_destroy(struct sock *sk);
+
 #endif	/* _TCP_H */
net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +38 -0
index c470fec9062f..6676fc3c0059 100644
@@ -305,6 +305,10 @@ EXPORT_SYMBOL(tcp_have_smc);
 struct percpu_counter tcp_sockets_allocated;
 EXPORT_SYMBOL(tcp_sockets_allocated);
 
+/* Optional TCP option handlers */
+DEFINE_STATIC_KEY_FALSE(tcp_extopt_enabled);
+EXPORT_SYMBOL_GPL(tcp_extopt_enabled);
+
 /*
 * TCP splice context
 */
@@ -416,6 +420,7 @@ void tcp_init_sock(struct sock *sk)
 	tcp_init_xmit_timers(sk);
 	INIT_LIST_HEAD(&tp->tsq_node);
 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
+	INIT_HLIST_HEAD(&tp->tcp_option_list);
 
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
@@ -3473,6 +3478,335 @@ EXPORT_SYMBOL(tcp_md5_hash_key);
 
 #endif
 
+static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
+{
+	if (sk_fullsock(sk))
+		return &tcp_sk(sk)->tcp_option_list;
+	else if (sk->sk_state == TCP_NEW_SYN_RECV)
+		return &tcp_rsk(inet_reqsk(sk))->tcp_option_list;
+	else if (sk->sk_state == TCP_TIME_WAIT)
+		return &tcp_twsk(sk)->tcp_option_list;
+
+	return NULL;
+}
+
+/* Caller must ensure that rcu is locked */
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == kind)
+			return entry;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_find_kind);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk)
+{
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3516

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+	struct tcp_extopt_store *entry;
+
+	rcu_read_lock_bh();

I'm curious, why the _bh variant in these functions? It would require use of hlist_for_each_entry_rcu_bh to traverse the list, in tcp_extopt_find_kind or in the functions below. It's ok for rcu_read_lock() to be used in interrupt context.

Yes, I think you are right after researching more. I guess I misunderstood something.

I thought that I need to call the _bh variant to disable interrupts. Because, the TCP-input code path can now be called from both contexts (softirq and with interrupts enabled).

I will change all of this to rcu_read_lock() and rcu_read_unlock().

Quoted 201 lines
Quoted 201 lines
net/ipv4/tcp.c +200 -0
+	entry = tcp_extopt_find_kind(opcode, sk);
+
+	if (entry && entry->ops->parse)
+		entry->ops->parse(opsize, opptr, skb, opt_rx, sk, entry);
+	rcu_read_unlock_bh();
+}
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	bool drop = false;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		bool ret = false;
+
+		if (entry->ops->check)
+			ret = entry->ops->check(sk, skb, opt_rx, entry);
+
+		if (ret)
+			drop = true;
+	}
+	rcu_read_unlock_bh();
+
+	return drop;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_check);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->post_process)
+			entry->ops->post_process(sk, opt_rx, entry);
+	}
+	rcu_read_unlock_bh();
+}
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->prepare))
+			continue;
+
+		used += entry->ops->prepare(skb, flags, remaining - used, opts,
+					    sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->write))
+			continue;
+
+		ptr = entry->ops->write(ptr, skb, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_write);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		int ret;
+
+		if (unlikely(!entry->ops->response_prepare))
+			continue;
+
+		ret = entry->ops->response_prepare(orig, flags,
+						   remaining - used, opts,
+						   sk, entry);
+
+		used += ret;
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_prepare);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->response_write))
+			continue;
+
+		ptr = entry->ops->response_write(ptr, orig, th, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_write);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	int tcp_header_len = 0;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, &tcp_sk(sk)->tcp_option_list, list) {
+		if (unlikely(!entry->ops->add_header_len))
+			continue;
+
+		tcp_header_len += entry->ops->add_header_len(listener, sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return tcp_header_len;
+}
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk)
+{
+	struct hlist_node *add_before = NULL;
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	int ret = 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	if (!store->ops->option_kind)
+		return -EINVAL;
+
+	if (!try_module_get(store->ops->owner))
+		return -ENOENT;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == store->ops->option_kind) {
+			rcu_read_unlock_bh();
+			pr_notice("Option kind %u already registered\n",
+				  store->ops->option_kind);
+			module_put(store->ops->owner);
+			return -EEXIST;
+		}
+
+		if (entry->ops->priority <= store->ops->priority)
+			add_before = &entry->list;
+	}
+
+	if (add_before)
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3719

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+		hlist_add_behind_rcu(&store->list, add_before);
+	else
+		hlist_add_head_rcu(&store->list, lhead);

I'm assuming the socket lock is being used to synchonize modifications to the options list and must be held before this function is called. It would help to add comments indicating which functions require the socket lock to be held (here or in include/net/tcp.h)

I will add a comment in tcp.h and here.

Quoted 3 lines

What's more, rcu_read_lock/unlock aren't needed when the write lock (the socket lock in this case) is held - so no need to call those here or in other functions where the lock is required before calling.

Yep - will remove the rcu_read_lock/unlock.

Quoted 80 lines
Quoted 80 lines
net/ipv4/tcp.c +80 -0
+	rcu_read_unlock_bh();
+	pr_debug("Option kind %u registered\n", store->ops->option_kind);
+
+	static_branch_inc(&tcp_extopt_enabled);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tcp_register_extopt);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *from, *to;
+
+	from = tcp_extopt_get_list(listener);
+	to = tcp_extopt_get_list(req_to_sk(req));
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, from, list) {
+		struct tcp_extopt_store *new;
+
+		if (!try_module_get(entry->ops->owner)) {
+			pr_err("%s Module get failed while copying\n", __func__);
+			continue;
+		}
+
+		new = entry->ops->copy(listener, req, opt, entry);
+		if (!new) {
+			module_put(entry->ops->owner);
+			continue;
+		}
+
+		hlist_add_tail_rcu(&new->list, to);
+
+		static_branch_inc(&tcp_extopt_enabled);
+	}
+	rcu_read_unlock_bh();
+}
+
+void tcp_extopt_move(struct sock *from, struct sock *to)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lfrom, *lto;
+	struct hlist_node *tmp;
+
+	lfrom = tcp_extopt_get_list(from);
+	lto = tcp_extopt_get_list(to);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lfrom, list) {
+		hlist_del_rcu(&entry->list);
+
+		if (entry->ops->move) {
+			entry = entry->ops->move(from, to, entry);
+			if (!entry)
+				continue;
+		}
+
+		hlist_add_tail_rcu(&entry->list, lto);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_move);
+
+void tcp_extopt_destroy(struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	struct hlist_node *tmp;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lhead, list) {
+		struct module *owner = entry->ops->owner;
+
+		hlist_del_rcu(&entry->list);
+
+		entry->ops->destroy(entry);
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3804

Quoted 4 lines
net/ipv4/tcp.c +4 -0
+
+		static_branch_dec(&tcp_extopt_enabled);
+
+		module_put(owner);

Is it ok to allow the module to unload before synchronize_rcu() or synchronize_net()? Maybe the static_branch_dec and module_put needs to be invoked via call_rcu?

Quoted 629 lines

I think it's fine. From what I see, before really unloading the module (coming from the delete_module() system-call), it does a synchronize_sched().

Christoph

Quoted 621 lines

Mat

Quoted 609 lines
net/ipv4/tcp.c +5 -1
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_destroy);
+
 void tcp_done(struct sock *sk)
 {
 	struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
@@ -3622,7 +3956,6 @@ void __init tcp_init(void)
 		INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
 	}
 
-
 	cnt = tcp_hashinfo.ehash_mask + 1;
 	sysctl_tcp_max_orphans = cnt / 2;
 
net/ipv4/tcp_input.c +13 -2
index eedb355a0533..d4d3be8a496c 100644
@@ -3701,7 +3701,7 @@ static int smc_parse_options(const struct tcphdr *th,
 void tcp_parse_options(const struct net *net,
 		       const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx, int estab,
-		       struct tcp_fastopen_cookie *foc)
+		       struct tcp_fastopen_cookie *foc, struct sock *sk)
 {
 	const unsigned char *ptr;
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -3804,6 +3804,17 @@ void tcp_parse_options(const struct net *net,
 				else if (smc_parse_options(th, opt_rx, ptr,
 							   opsize))
 					break;
+				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
+					 opsize >= TCPOLEN_EXP_BASE)
+					tcp_extopt_parse(get_unaligned_be32(ptr),
+							 opsize, ptr, skb,
+							 opt_rx, sk);
+				break;
+
+			default:
+				if (static_branch_unlikely(&tcp_extopt_enabled))
+					tcp_extopt_parse(opcode, opsize, ptr,
+							 skb, opt_rx, sk);
 				break;
 
 			}
@@ -3854,12 +3865,13 @@ static bool tcp_fast_parse_options(const struct net *net,
 			goto extra_opt_check;
 	}
 
-	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
+	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL, tcp_to_sk(tp));
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 

extra_opt_check:

net/ipv4/tcp_input.c +2 -1
-	/* ToDo - will be added here */
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		return tcp_extopt_check(tcp_to_sk(tp), skb, &tp->rx_opt);
	return false;
}
net/ipv4/tcp_input.c +26 -3
@@ -5333,6 +5345,10 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 	tp->rx_opt.saw_tstamp = 0;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    !hlist_empty(&tp->tcp_option_list))
+		goto slow_path;
+
 	/*	pred_flags is 0xS?10 << 16 + snd_wnd
 	 *	if header_prediction is to be made
 	 *	'S' will always be tp->tcp_header_len >> 2
@@ -5520,7 +5536,7 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 		/* Get original SYNACK MSS value if user MSS sets mss_clamp */
 		tcp_clear_options(&opt);
 		opt.user_mss = opt.mss_clamp = 0;
-		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL, sk);
 		mss = opt.mss_clamp;
 	}
 
@@ -5583,10 +5599,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 	int saved_clamp = tp->rx_opt.mss_clamp;
 	bool fastopen_fail;
 
-	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
+	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc, sk);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tp->rx_opt))
+		goto discard;
+
 	if (th->ack) {
 		/* rfc793:
 		 * "If the state is SYN-SENT then
@@ -5669,6 +5689,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 			tp->tcp_header_len = sizeof(struct tcphdr);
 		}
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
@@ -5762,6 +5785,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 
 		tcp_ecn_rcv_syn(tp, th);
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_mtup_init(sk);
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
@@ -6245,12 +6271,17 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
 	tmp_opt.user_mss  = tp->rx_opt.user_mss;
 	tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
-			  want_cookie ? NULL : &foc);
+			  want_cookie ? NULL : &foc, sk);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tmp_opt))
+		goto drop_and_free;
 
 	if (want_cookie && !tmp_opt.saw_tstamp)
 		tcp_clear_options(&tmp_opt);
@@ -6311,6 +6342,10 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 		tcp_reqsk_record_syn(sk, req, skb);
 		fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
 	}
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_copy(sk, req, &tmp_opt);
+
 	if (fastopen_sk) {
 		af_ops->send_synack(fastopen_sk, dst, &fl, req,
 				    &foc, TCP_SYNACK_FASTOPEN);
net/ipv4/tcp_ipv4.c +72 -18
index 94e28350f420..ad87dd2db092 100644
@@ -600,9 +600,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-#ifdef CONFIG_TCP_MD5SIG
-		__be32 opt[(TCPOLEN_MD5SIG_ALIGNED >> 2)];
-#endif
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct ip_reply_arg arg;
 #ifdef CONFIG_TCP_MD5SIG
@@ -613,6 +611,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	struct sock *sk1 = NULL;
 #endif
 	struct net *net;
+	int offset = 0;
 
 	/* Never send a reset in response to a reset. */
 	if (th->rst)
@@ -678,19 +677,44 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 			goto out;
 
 	}
+#endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.opt);
+#ifdef CONFIG_TCP_MD5SIG
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
 
+		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
+
+		tcp_extopt_response_write(&rep.opt[0], skb, &rep.th, &opts, sk);
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
 	if (key) {
-		rep.opt[0] = htonl((TCPOPT_NOP << 24) |
-				   (TCPOPT_NOP << 16) |
-				   (TCPOPT_MD5SIG << 8) |
-				   TCPOLEN_MD5SIG);
+		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
+					  (TCPOPT_NOP << 16) |
+					  (TCPOPT_MD5SIG << 8) |
+					  TCPOLEN_MD5SIG);
 		/* Update length and the length the header thinks exists */
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
 
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
-				     key, ip_hdr(skb)->saddr,
-				     ip_hdr(skb)->daddr, &rep.th);
+		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
+				    key, ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
@@ -742,14 +766,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-		__be32 opt[(TCPOLEN_TSTAMP_ALIGNED >> 2)
-#ifdef CONFIG_TCP_MD5SIG
-			   + (TCPOLEN_MD5SIG_ALIGNED >> 2)
-#endif
-			];
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
+	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
@@ -763,6 +784,7 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		rep.opt[1] = htonl(tsval);
 		rep.opt[2] = htonl(tsecr);
 		arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
+		offset += 3;
 	}
 
 	/* Swap the send and the receive. */
@@ -774,22 +796,45 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
+
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		int offset = (tsecr) ? 3 : 0;
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
+		used = tcp_extopt_response_prepare(skb, TCPHDR_ACK, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
+		tcp_extopt_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
+
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
+	if (key) {
 		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
 					  (TCPOPT_NOP << 16) |
 					  (TCPOPT_MD5SIG << 8) |
 					  TCPOLEN_MD5SIG);
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len/4;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
 		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
 				    key, ip_hdr(skb)->saddr,
 				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
+
 	arg.flags = reply_flags;
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -893,6 +938,9 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
 */
 static void tcp_v4_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
@@ -1387,6 +1435,10 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 
 	tcp_initialize_rcv_mss(newsk);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
@@ -1907,6 +1959,8 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
 	skb_rbtree_purge(&tp->out_of_order_queue);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Clean up the MD5 key list, if any */
 	if (tp->md5sig_info) {
net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -1
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
--
Quoted 3 lines

2.15.0

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

author
#21 lore raw

Re: [MPTCP] [RFC v3 06/15] tcp: Register handlers for extra TCP options

Quoted 81 lines · Mat Martineau
Quoted 78 lines · Christoph Paasch
Quoted 52 lines · Mat Martineau

Hi Christoph,

Quoted 43 lines · Christoph Paasch

Hi Mat,

Quoted 34 lines · Mat Martineau
Quoted 19 lines · Christoph Paasch

From: Mat Martineau <mathew.j.martineau(a)linux.intel.com>

Allow additional TCP options to be handled by registered hook functions.

Registered options have a priority that determines the order in which options are prepared and written. Lower priority numbers are handled first.

Option parsing will call the provided 'parse' function when a TCP option number is not recognized by the normal option parsing code.

After parsing, there are two places where we post-process the options. First, a 'check' callback that allows to drop the packet based on the parsed options (e.g., usefull for TCP MD5SIG). Then, a 'post_process' function that gets called after other validity checks (aka, in-window, PAWS,...). This post_process function can then update other state for this particular extra-option.

Quoted 5 lines

In the output-path, the 'prepare' function determines the required space for registered options and store associated data. 'write' adds the option to the TCP header.

A static key is used to minimize the performance impact.

I'm still on the fence about using a static key at all, especially without rate limiting. It made a lot more sense when used in conjunction with the global option list, and it's not widely used in net/. There are ~60 occurences of static_branch_unlikely in the entire mainline tree today, and this patch adds ~30!

Quoted 5 lines

After I get through the rest of the patch series, I'll take another look at where static_branch_unlikely is used. If the sensive code paths have a known socket type, that would allow use of "if (unlikely(!hlist_empty(&some_sk->tcp_option_list)) && ..." without having to involve a function call to tcp_extopt_get_list().

Quoted 3 lines

I am pretty sure that this would trigger the response from Eric that such a test would add an additional access to a potentially cold cache-line in the hot data-path.

Quoted 8 lines

I'd expect him to be thinking about that too, but also that static branches only solve that problem if no socket on the entire system is using extended options.

I think cache hits are only a concern on the output path, since the input path only tries to look up extended optoins if there's an unrecognized option number. And on the output path, we could try to be on a hot cache line. There are a number of tcp_sock members accessed in the hot path.

Branching is also a concern. With static keys the branching is avoided as well.

Quoted 12 lines
Quoted 6 lines

For the static-key question and rate-limiting, I would suggest that we see what netdev's feedback will be. We can add this as one of the discussion-points in the cover-letter mail.

It is a good discussion point to bring up, but we don't need static branching in there to do so. I can work up a patch to take out the static branch stuff, which can be squashed in before posting. In case discussion leads us back to static branches, we'll have the patch around to put the code back.

Quoted 6 lines
Quoted 4 lines

Ok, let's go with this. I will quickly change that and post a last final version here on the list.

Do you want to add your "Reviewed-by"-tag to the patches?

Yes (other than the one with my signoff).

Ok, I will add it in my v4 final here on the list.

Quoted 2 lines

I'm finishing up the last two
patches right now.

Cool!

Thanks,
Christoph

Quoted 1,316 lines

Mat

Quoted 1,308 lines

Christoph

Quoted 1,305 lines

Mat

Quoted 668 lines
Quoted 199 lines
Quoted 199 lines

These additional TCP-options are stored in hlists of the TCP-socket. To pass the state and options around during the 3-way handshake and in time-wait state, the hlists are also on the tcp_request_sock and tcp_timewait_sock.
The list is copied from the listener to the request-socket (calling into the 'copy' callback). Then, moved from the request-socket to the TCP-socket and finally to the time-wait socket.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Merged my previous commits from v2 into this one
* Removed a dangling rcu_read_lock() from tcp_transmit_skb()
* Renamed all to tcp_extopt_*
* Initialize the request socket's tcp_option_list earlier, because we might access it before tcp_openreq_init().
* Fix the setting of rep.th.doff in tcp_v4_send_ack,reset,... We need it correctly set before we call the _write-function because MD5 needs it to compute the hash.
* Added a return-value to the *write callbacks as we need to progress the ptr
* For tcp_extopt_move, we need to get the return-value of the callback
* Transitioned to RCU-protected hlists for the tcp_option_list.
We need the RCU-protection because listeners work lockless. Further, an hlist allows to have just one pointer - thus saves memory :)
v2: * Merged my previous commits from v1 into this one
* Moved rep.th.doff = arg.iov[0].iov_len / 4; to after the MD5-code in tcp_v4_send_ack

drivers/infiniband/hw/cxgb4/cm.c |   2 +-
include/linux/tcp.h              |  28 ++++
include/net/tcp.h                | 106 ++++++++++++-
net/ipv4/syncookies.c            |   6 +-
net/ipv4/tcp.c                   | 335 ++++++++++++++++++++++++++++++++++++++-
net/ipv4/tcp_input.c             |  47 +++++-
net/ipv4/tcp_ipv4.c              |  90 ++++++++---
net/ipv4/tcp_minisocks.c         |  30 +++-
net/ipv4/tcp_output.c            |  37 ++---
net/ipv6/syncookies.c            |   6 +-
net/ipv6/tcp_ipv6.c              |  28 ++++
11 files changed, 664 insertions(+), 51 deletions(-)
drivers/infiniband/hw/cxgb4/cm.c +1 -1
index 21db3b48a617..a1ea5583f07b 100644
@@ -3746,7 +3746,7 @@ static void build_cpl_pass_accept_req(struct sk_buff *skb, int stid , u8 tos)
 	 */
 	memset(&tmp_opt, 0, sizeof(tmp_opt));
 	tcp_clear_options(&tmp_opt);
-	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL);
+	tcp_parse_options(&init_net, skb, &tmp_opt, 0, NULL, NULL);
 
 	req = __skb_push(skb, sizeof(*req));
 	memset(req, 0, sizeof(*req));
include/linux/tcp.h +28 -0
index ca4a6361389b..b14426228a01 100644
@@ -115,6 +115,24 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #endif
 }
 
+#define OPTION_SACK_ADVERTISE	(1 << 0)
+#define OPTION_TS		(1 << 1)
+#define OPTION_MD5		(1 << 2)
+#define OPTION_WSCALE		(1 << 3)
+#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
+#define OPTION_SMC		(1 << 9)
+
+struct tcp_out_options {
+	u16 options;		/* bit field of OPTION_* */
+	u16 mss;		/* 0 to disable */
+	u8 ws;			/* window scale, 0 to disable */
+	u8 num_sack_blocks;	/* number of SACK blocks to include */
+	u8 hash_size;		/* bytes in hash_location */
+	__u8 *hash_location;	/* temporary pointer, overloaded */
+	__u32 tsval, tsecr;	/* need to include OPTION_TS */
+	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+};
+
 /* This is the max number of SACKS that we'll generate and process. It's safe
 * to increase this, although since:
 *   size = TCPOLEN_SACK_BASE_ALIGNED (4) + n * TCPOLEN_SACK_PERBLOCK (8)
@@ -137,6 +155,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	struct hlist_head		tcp_option_list;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
@@ -373,6 +392,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	struct hlist_head tcp_option_list;
 };
 
 enum tsq_enum {
@@ -400,6 +421,11 @@ static inline struct tcp_sock *tcp_sk(const struct sock *sk)
 	return (struct tcp_sock *)sk;
 }
 
+static inline struct sock *tcp_to_sk(const struct tcp_sock *tp)
+{
+	return (struct sock *)tp;
+}
+
 struct tcp_timewait_sock {
 	struct inet_timewait_sock tw_sk;
 #define tw_rcv_nxt tw_sk.__tw_common.skc_tw_rcv_nxt
@@ -412,6 +438,8 @@ struct tcp_timewait_sock {
 	u32			  tw_last_oow_ack_time;
 
 	long			  tw_ts_recent_stamp;
+
+	struct hlist_head	  tcp_option_list;
 #ifdef CONFIG_TCP_MD5SIG
 	struct tcp_md5sig_key	  *tw_md5_key;
 #endif
include/net/tcp.h +53 -1
index 3c3744e52cd1..86133aa24fca 100644
@@ -202,6 +202,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_FASTOPEN_BASE  2
 #define TCPOLEN_EXP_FASTOPEN_BASE  4
 #define TCPOLEN_EXP_SMC_BASE   6
+#define TCPOLEN_EXP_BASE       6
 
 /* But this is what stacks really send out. */
 #define TCPOLEN_TSTAMP_ALIGNED		12
@@ -403,7 +404,8 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
 		int flags, int *addr_len);
 void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
-		       int estab, struct tcp_fastopen_cookie *foc);
+		       int estab, struct tcp_fastopen_cookie *foc,
+		       struct sock *sk);
 const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
@@ -2064,4 +2066,106 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 #if IS_ENABLED(CONFIG_SMC)
 extern struct static_key_false tcp_have_smc;
 #endif
+
+extern struct static_key_false tcp_extopt_enabled;
+struct tcp_extopt_store;
+
+struct tcp_extopt_ops {
+	u32			option_kind;
+	unsigned char		priority;
+	void (*parse)(int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct sock *sk,
+		      struct tcp_extopt_store *store);
+	bool (*check)(struct sock *sk,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx,
+		      struct tcp_extopt_store *store);
+	void (*post_process)(struct sock *sk,
+			     struct tcp_options_received *opt_rx,
+			     struct tcp_extopt_store *store);
+	/* Return the number of bytes consumed */
+	unsigned int (*prepare)(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*write)(__be32 *ptr, struct sk_buff *skb,
+			 struct tcp_out_options *opts, struct sock *sk,
+			 struct tcp_extopt_store *store);
+	int (*response_prepare)(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk,
+				struct tcp_extopt_store *store);
+	__be32 *(*response_write)(__be32 *ptr, struct sk_buff *orig,
+				  struct tcphdr *th,
+				  struct tcp_out_options *opts,
+				  const struct sock *sk,
+				  struct tcp_extopt_store *store);
+	int (*add_header_len)(const struct sock *listener,
+			      const struct sock *sk,
+			      struct tcp_extopt_store *store);
+	struct tcp_extopt_store *(*copy)(struct sock *listener,
+					 struct request_sock *req,
+					 struct tcp_options_received *opt,
+					 struct tcp_extopt_store *from);
+	struct tcp_extopt_store *(*move)(struct sock *from, struct sock *to,
+					 struct tcp_extopt_store *store);
+	void (*destroy)(struct tcp_extopt_store *store);
+	struct module		*owner;
+};
Quoted 8 lines

include/net/tcp.h static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk) · line 2123

Quoted 5 lines
include/net/tcp.h +5 -0
+
+/* Every extra-option in a TCP-socket gets stored in a _store structure which
+ * has to have tcp_extopt_store as the first element.
+ *
+ * That way, an extra option registers passes this to tcp_register_extopt,

Maybe "That way, code registering extra options can pass this to tcp_register_extopt" ...

Sure, changed it.

Quoted 133 lines
Quoted 133 lines
include/net/tcp.h +47 -0
+ * and the surrounding structure can contain per-option state.
+ */
+struct tcp_extopt_store {
+	struct hlist_node		list;
+	const struct tcp_extopt_ops	*ops;
+};
+
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk);
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx);
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk);
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt);
+
+void tcp_extopt_move(struct sock *from, struct sock *to);
+
+void tcp_extopt_destroy(struct sock *sk);
+
 #endif	/* _TCP_H */
net/ipv4/syncookies.c +5 -1
index fda37f2862c9..172fd4cf9c85 100644
@@ -313,7 +313,7 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcp_ts_off(sock_net(sk),
@@ -325,6 +325,10 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp_request_sock_ops, sk, false); /* for safety */
 	if (!req)
net/ipv4/tcp.c +38 -0
index c470fec9062f..6676fc3c0059 100644
@@ -305,6 +305,10 @@ EXPORT_SYMBOL(tcp_have_smc);
 struct percpu_counter tcp_sockets_allocated;
 EXPORT_SYMBOL(tcp_sockets_allocated);
 
+/* Optional TCP option handlers */
+DEFINE_STATIC_KEY_FALSE(tcp_extopt_enabled);
+EXPORT_SYMBOL_GPL(tcp_extopt_enabled);
+
 /*
 * TCP splice context
 */
@@ -416,6 +420,7 @@ void tcp_init_sock(struct sock *sk)
 	tcp_init_xmit_timers(sk);
 	INIT_LIST_HEAD(&tp->tsq_node);
 	INIT_LIST_HEAD(&tp->tsorted_sent_queue);
+	INIT_HLIST_HEAD(&tp->tcp_option_list);
 
 	icsk->icsk_rto = TCP_TIMEOUT_INIT;
 	tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT);
@@ -3473,6 +3478,335 @@ EXPORT_SYMBOL(tcp_md5_hash_key);
 
 #endif
 
+static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
+{
+	if (sk_fullsock(sk))
+		return &tcp_sk(sk)->tcp_option_list;
+	else if (sk->sk_state == TCP_NEW_SYN_RECV)
+		return &tcp_rsk(inet_reqsk(sk))->tcp_option_list;
+	else if (sk->sk_state == TCP_TIME_WAIT)
+		return &tcp_twsk(sk)->tcp_option_list;
+
+	return NULL;
+}
+
+/* Caller must ensure that rcu is locked */
+struct tcp_extopt_store *tcp_extopt_find_kind(u32 kind, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == kind)
+			return entry;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_find_kind);
+
+void tcp_extopt_parse(u32 opcode, int opsize, const unsigned char *opptr,
+		      const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx, struct sock *sk)
+{
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3516

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+	struct tcp_extopt_store *entry;
+
+	rcu_read_lock_bh();

I'm curious, why the _bh variant in these functions? It would require use of hlist_for_each_entry_rcu_bh to traverse the list, in tcp_extopt_find_kind or in the functions below. It's ok for rcu_read_lock() to be used in interrupt context.

Yes, I think you are right after researching more. I guess I misunderstood something.

I thought that I need to call the _bh variant to disable interrupts. Because, the TCP-input code path can now be called from both contexts (softirq and with interrupts enabled).

I will change all of this to rcu_read_lock() and rcu_read_unlock().

Quoted 201 lines
Quoted 201 lines
net/ipv4/tcp.c +200 -0
+	entry = tcp_extopt_find_kind(opcode, sk);
+
+	if (entry && entry->ops->parse)
+		entry->ops->parse(opsize, opptr, skb, opt_rx, sk, entry);
+	rcu_read_unlock_bh();
+}
+
+bool tcp_extopt_check(struct sock *sk, const struct sk_buff *skb,
+		      struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	bool drop = false;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		bool ret = false;
+
+		if (entry->ops->check)
+			ret = entry->ops->check(sk, skb, opt_rx, entry);
+
+		if (ret)
+			drop = true;
+	}
+	rcu_read_unlock_bh();
+
+	return drop;
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_check);
+
+void tcp_extopt_post_process(struct sock *sk,
+			     struct tcp_options_received *opt_rx)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->post_process)
+			entry->ops->post_process(sk, opt_rx, entry);
+	}
+	rcu_read_unlock_bh();
+}
+
+unsigned int tcp_extopt_prepare(struct sk_buff *skb, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->prepare))
+			continue;
+
+		used += entry->ops->prepare(skb, flags, remaining - used, opts,
+					    sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+
+void tcp_extopt_write(__be32 *ptr, struct sk_buff *skb,
+		      struct tcp_out_options *opts, struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->write))
+			continue;
+
+		ptr = entry->ops->write(ptr, skb, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_write);
+
+int tcp_extopt_response_prepare(struct sk_buff *orig, u8 flags,
+				unsigned int remaining,
+				struct tcp_out_options *opts,
+				const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	unsigned int used = 0;
+
+	if (!sk)
+		return 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		int ret;
+
+		if (unlikely(!entry->ops->response_prepare))
+			continue;
+
+		ret = entry->ops->response_prepare(orig, flags,
+						   remaining - used, opts,
+						   sk, entry);
+
+		used += ret;
+	}
+	rcu_read_unlock_bh();
+
+	return roundup(used, 4);
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_prepare);
+
+void tcp_extopt_response_write(__be32 *ptr, struct sk_buff *orig,
+			       struct tcphdr *th, struct tcp_out_options *opts,
+			       const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+
+	if (!sk)
+		return;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (unlikely(!entry->ops->response_write))
+			continue;
+
+		ptr = entry->ops->response_write(ptr, orig, th, opts, sk, entry);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_response_write);
+
+int tcp_extopt_add_header(const struct sock *listener, const struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	int tcp_header_len = 0;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, &tcp_sk(sk)->tcp_option_list, list) {
+		if (unlikely(!entry->ops->add_header_len))
+			continue;
+
+		tcp_header_len += entry->ops->add_header_len(listener, sk, entry);
+	}
+	rcu_read_unlock_bh();
+
+	return tcp_header_len;
+}
+
+int tcp_register_extopt(struct tcp_extopt_store *store, struct sock *sk)
+{
+	struct hlist_node *add_before = NULL;
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	int ret = 0;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	if (!store->ops->option_kind)
+		return -EINVAL;
+
+	if (!try_module_get(store->ops->owner))
+		return -ENOENT;
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, lhead, list) {
+		if (entry->ops->option_kind == store->ops->option_kind) {
+			rcu_read_unlock_bh();
+			pr_notice("Option kind %u already registered\n",
+				  store->ops->option_kind);
+			module_put(store->ops->owner);
+			return -EEXIST;
+		}
+
+		if (entry->ops->priority <= store->ops->priority)
+			add_before = &entry->list;
+	}
+
+	if (add_before)
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3719

Quoted 3 lines
net/ipv4/tcp.c +3 -0
+		hlist_add_behind_rcu(&store->list, add_before);
+	else
+		hlist_add_head_rcu(&store->list, lhead);

I'm assuming the socket lock is being used to synchonize modifications to the options list and must be held before this function is called. It would help to add comments indicating which functions require the socket lock to be held (here or in include/net/tcp.h)

I will add a comment in tcp.h and here.

Quoted 3 lines

What's more, rcu_read_lock/unlock aren't needed when the write lock (the socket lock in this case) is held - so no need to call those here or in other functions where the lock is required before calling.

Yep - will remove the rcu_read_lock/unlock.

Quoted 80 lines
Quoted 80 lines
net/ipv4/tcp.c +80 -0
+	rcu_read_unlock_bh();
+	pr_debug("Option kind %u registered\n", store->ops->option_kind);
+
+	static_branch_inc(&tcp_extopt_enabled);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tcp_register_extopt);
+
+void tcp_extopt_copy(struct sock *listener, struct request_sock *req,
+		     struct tcp_options_received *opt)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *from, *to;
+
+	from = tcp_extopt_get_list(listener);
+	to = tcp_extopt_get_list(req_to_sk(req));
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_rcu(entry, from, list) {
+		struct tcp_extopt_store *new;
+
+		if (!try_module_get(entry->ops->owner)) {
+			pr_err("%s Module get failed while copying\n", __func__);
+			continue;
+		}
+
+		new = entry->ops->copy(listener, req, opt, entry);
+		if (!new) {
+			module_put(entry->ops->owner);
+			continue;
+		}
+
+		hlist_add_tail_rcu(&new->list, to);
+
+		static_branch_inc(&tcp_extopt_enabled);
+	}
+	rcu_read_unlock_bh();
+}
+
+void tcp_extopt_move(struct sock *from, struct sock *to)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lfrom, *lto;
+	struct hlist_node *tmp;
+
+	lfrom = tcp_extopt_get_list(from);
+	lto = tcp_extopt_get_list(to);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lfrom, list) {
+		hlist_del_rcu(&entry->list);
+
+		if (entry->ops->move) {
+			entry = entry->ops->move(from, to, entry);
+			if (!entry)
+				continue;
+		}
+
+		hlist_add_tail_rcu(&entry->list, lto);
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_move);
+
+void tcp_extopt_destroy(struct sock *sk)
+{
+	struct tcp_extopt_store *entry;
+	struct hlist_head *lhead;
+	struct hlist_node *tmp;
+
+	lhead = tcp_extopt_get_list(sk);
+
+	rcu_read_lock_bh();
+	hlist_for_each_entry_safe(entry, tmp, lhead, list) {
+		struct module *owner = entry->ops->owner;
+
+		hlist_del_rcu(&entry->list);
+
+		entry->ops->destroy(entry);
Quoted 8 lines

net/ipv4/tcp.c EXPORT_SYMBOL(tcp_md5_hash_key); · line 3804

Quoted 4 lines
net/ipv4/tcp.c +4 -0
+
+		static_branch_dec(&tcp_extopt_enabled);
+
+		module_put(owner);

Is it ok to allow the module to unload before synchronize_rcu() or synchronize_net()? Maybe the static_branch_dec and module_put needs to be invoked via call_rcu?

Quoted 629 lines

I think it's fine. From what I see, before really unloading the module (coming from the delete_module() system-call), it does a synchronize_sched().

Christoph

Quoted 621 lines

Mat

Quoted 609 lines
net/ipv4/tcp.c +5 -1
+	}
+	rcu_read_unlock_bh();
+}
+EXPORT_SYMBOL_GPL(tcp_extopt_destroy);
+
 void tcp_done(struct sock *sk)
 {
 	struct request_sock *req = tcp_sk(sk)->fastopen_rsk;
@@ -3622,7 +3956,6 @@ void __init tcp_init(void)
 		INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain);
 	}
 
-
 	cnt = tcp_hashinfo.ehash_mask + 1;
 	sysctl_tcp_max_orphans = cnt / 2;
 
net/ipv4/tcp_input.c +13 -2
index eedb355a0533..d4d3be8a496c 100644
@@ -3701,7 +3701,7 @@ static int smc_parse_options(const struct tcphdr *th,
 void tcp_parse_options(const struct net *net,
 		       const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx, int estab,
-		       struct tcp_fastopen_cookie *foc)
+		       struct tcp_fastopen_cookie *foc, struct sock *sk)
 {
 	const unsigned char *ptr;
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -3804,6 +3804,17 @@ void tcp_parse_options(const struct net *net,
 				else if (smc_parse_options(th, opt_rx, ptr,
 							   opsize))
 					break;
+				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
+					 opsize >= TCPOLEN_EXP_BASE)
+					tcp_extopt_parse(get_unaligned_be32(ptr),
+							 opsize, ptr, skb,
+							 opt_rx, sk);
+				break;
+
+			default:
+				if (static_branch_unlikely(&tcp_extopt_enabled))
+					tcp_extopt_parse(opcode, opsize, ptr,
+							 skb, opt_rx, sk);
 				break;
 
 			}
@@ -3854,12 +3865,13 @@ static bool tcp_fast_parse_options(const struct net *net,
 			goto extra_opt_check;
 	}
 
-	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL);
+	tcp_parse_options(net, skb, &tp->rx_opt, 1, NULL, tcp_to_sk(tp));
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 

extra_opt_check:

net/ipv4/tcp_input.c +2 -1
-	/* ToDo - will be added here */
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		return tcp_extopt_check(tcp_to_sk(tp), skb, &tp->rx_opt);
	return false;
}
net/ipv4/tcp_input.c +26 -3
@@ -5333,6 +5345,10 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
 
 	tp->rx_opt.saw_tstamp = 0;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    !hlist_empty(&tp->tcp_option_list))
+		goto slow_path;
+
 	/*	pred_flags is 0xS?10 << 16 + snd_wnd
 	 *	if header_prediction is to be made
 	 *	'S' will always be tp->tcp_header_len >> 2
@@ -5520,7 +5536,7 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 		/* Get original SYNACK MSS value if user MSS sets mss_clamp */
 		tcp_clear_options(&opt);
 		opt.user_mss = opt.mss_clamp = 0;
-		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), synack, &opt, 0, NULL, sk);
 		mss = opt.mss_clamp;
 	}
 
@@ -5583,10 +5599,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 	int saved_clamp = tp->rx_opt.mss_clamp;
 	bool fastopen_fail;
 
-	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc);
+	tcp_parse_options(sock_net(sk), skb, &tp->rx_opt, 0, &foc, sk);
 	if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
 		tp->rx_opt.rcv_tsecr -= tp->tsoffset;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tp->rx_opt))
+		goto discard;
+
 	if (th->ack) {
 		/* rfc793:
 		 * "If the state is SYN-SENT then
@@ -5669,6 +5689,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 			tp->tcp_header_len = sizeof(struct tcphdr);
 		}
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
@@ -5762,6 +5785,9 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 
 		tcp_ecn_rcv_syn(tp, th);
 
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			tcp_extopt_post_process(sk, &tp->rx_opt);
+
 		tcp_mtup_init(sk);
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
@@ -6245,12 +6271,17 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
 	tmp_opt.user_mss  = tp->rx_opt.user_mss;
 	tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0,
-			  want_cookie ? NULL : &foc);
+			  want_cookie ? NULL : &foc, sk);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tmp_opt))
+		goto drop_and_free;
 
 	if (want_cookie && !tmp_opt.saw_tstamp)
 		tcp_clear_options(&tmp_opt);
@@ -6311,6 +6342,10 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 		tcp_reqsk_record_syn(sk, req, skb);
 		fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
 	}
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_copy(sk, req, &tmp_opt);
+
 	if (fastopen_sk) {
 		af_ops->send_synack(fastopen_sk, dst, &fl, req,
 				    &foc, TCP_SYNACK_FASTOPEN);
net/ipv4/tcp_ipv4.c +72 -18
index 94e28350f420..ad87dd2db092 100644
@@ -600,9 +600,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-#ifdef CONFIG_TCP_MD5SIG
-		__be32 opt[(TCPOLEN_MD5SIG_ALIGNED >> 2)];
-#endif
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct ip_reply_arg arg;
 #ifdef CONFIG_TCP_MD5SIG
@@ -613,6 +611,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	struct sock *sk1 = NULL;
 #endif
 	struct net *net;
+	int offset = 0;
 
 	/* Never send a reset in response to a reset. */
 	if (th->rst)
@@ -678,19 +677,44 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 			goto out;
 
 	}
+#endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.opt);
+#ifdef CONFIG_TCP_MD5SIG
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
 
+		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
+
+		tcp_extopt_response_write(&rep.opt[0], skb, &rep.th, &opts, sk);
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
 	if (key) {
-		rep.opt[0] = htonl((TCPOPT_NOP << 24) |
-				   (TCPOPT_NOP << 16) |
-				   (TCPOPT_MD5SIG << 8) |
-				   TCPOLEN_MD5SIG);
+		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
+					  (TCPOPT_NOP << 16) |
+					  (TCPOPT_MD5SIG << 8) |
+					  TCPOLEN_MD5SIG);
 		/* Update length and the length the header thinks exists */
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
 
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
-				     key, ip_hdr(skb)->saddr,
-				     ip_hdr(skb)->daddr, &rep.th);
+		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
+				    key, ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
@@ -742,14 +766,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct {
 		struct tcphdr th;
-		__be32 opt[(TCPOLEN_TSTAMP_ALIGNED >> 2)
-#ifdef CONFIG_TCP_MD5SIG
-			   + (TCPOLEN_MD5SIG_ALIGNED >> 2)
-#endif
-			];
+		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
+	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
@@ -763,6 +784,7 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		rep.opt[1] = htonl(tsval);
 		rep.opt[2] = htonl(tsecr);
 		arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
+		offset += 3;
 	}
 
 	/* Swap the send and the receive. */
@@ -774,22 +796,45 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining;
+		struct tcp_out_options opts;
+		int used;
+
+		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
+
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		int offset = (tsecr) ? 3 : 0;
+		if (key)
+			remaining -= TCPOLEN_MD5SIG_ALIGNED;
+#endif
+
+		memset(&opts, 0, sizeof(opts));
+		used = tcp_extopt_response_prepare(skb, TCPHDR_ACK, remaining,
+						   &opts, sk);
+
+		arg.iov[0].iov_len += used;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
+		tcp_extopt_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
+
+		offset += used / 4;
+	}
+
+#ifdef CONFIG_TCP_MD5SIG
+	if (key) {
 		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
 					  (TCPOPT_NOP << 16) |
 					  (TCPOPT_MD5SIG << 8) |
 					  TCPOLEN_MD5SIG);
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len/4;
+		rep.th.doff = arg.iov[0].iov_len / 4;
 
 		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
 				    key, ip_hdr(skb)->saddr,
 				    ip_hdr(skb)->daddr, &rep.th);
 	}
 #endif
+
 	arg.flags = reply_flags;
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -893,6 +938,9 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
 */
 static void tcp_v4_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
@@ -1387,6 +1435,10 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 
 	tcp_initialize_rcv_mss(newsk);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
@@ -1907,6 +1959,8 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	/* Cleans up our, hopefully empty, out_of_order_queue. */
 	skb_rbtree_purge(&tp->out_of_order_queue);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* Clean up the MD5 key list, if any */
 	if (tp->md5sig_info) {
net/ipv4/tcp_minisocks.c +26 -4
index b079b619b60c..c38d12bc7dde 100644
@@ -95,9 +95,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	bool paws_reject = false;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
-		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(twsk_net(tw), skb, &tmp_opt, 0, NULL,
+				  (struct sock *)tw);
 
 		if (tmp_opt.saw_tstamp) {
 			if (tmp_opt.rcv_tsecr)
@@ -108,6 +109,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check((struct sock *)tw, skb, &tmp_opt))
+		return TCP_TW_SUCCESS;
+
 	if (tw->tw_substate == TCP_FIN_WAIT2) {
 		/* Just repeat all the checks of tcp_rcv_state_process() */
 
@@ -271,6 +276,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		tcptw->tw_ts_recent_stamp = tp->rx_opt.ts_recent_stamp;
 		tcptw->tw_ts_offset	= tp->tsoffset;
 		tcptw->tw_last_oow_ack_time = 0;
+		INIT_HLIST_HEAD(&tcptw->tcp_option_list);
 
 #if IS_ENABLED(CONFIG_IPV6)
 		if (tw->tw_family == PF_INET6) {
@@ -284,6 +290,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		}
 #endif
 
+		if (static_branch_unlikely(&tcp_extopt_enabled)) {
+			tcp_extopt_move(sk, (struct sock *)tw);
+			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
+		}
 #ifdef CONFIG_TCP_MD5SIG
 		/*
 		 * The timewait bucket does not have the key DB from the
@@ -340,6 +350,9 @@ void tcp_twsk_destructor(struct sock *sk)
 	if (twsk->tw_md5_key)
 		kfree_rcu(twsk->tw_md5_key, rcu);
 #endif
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(sk);
 }
 EXPORT_SYMBOL_GPL(tcp_twsk_destructor);
 
@@ -469,6 +482,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 
 		INIT_LIST_HEAD(&newtp->tsq_node);
 		INIT_LIST_HEAD(&newtp->tsorted_sent_queue);
+		INIT_HLIST_HEAD(&newtp->tcp_option_list);
 
 		tcp_init_wl(newtp, treq->rcv_isn);
 
@@ -544,6 +558,9 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		if (newtp->af_specific->md5_lookup(sk, newsk))
 			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
+		if (static_branch_unlikely(&tcp_extopt_enabled))
+			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
+
 		if (skb->len >= TCP_MSS_DEFAULT + newtp->tcp_header_len)
 			newicsk->icsk_ack.last_seg_size = skb->len - newtp->tcp_header_len;
 		newtp->rx_opt.mss_clamp = req->mss;
@@ -586,9 +603,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 	bool paws_reject = false;
 	bool own_req;
 
-	tmp_opt.saw_tstamp = 0;
+	tcp_clear_options(&tmp_opt);
 	if (th->doff > (sizeof(struct tcphdr)>>2)) {
-		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL);
+		tcp_parse_options(sock_net(sk), skb, &tmp_opt, 0, NULL,
+				  req_to_sk(req));
 
 		if (tmp_opt.saw_tstamp) {
 			tmp_opt.ts_recent = req->ts_recent;
@@ -603,6 +621,10 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(req_to_sk(req), skb, &tmp_opt))
+		return NULL;
+
 	/* Check for pure retransmitted SYN. */
 	if (TCP_SKB_CB(skb)->seq == tcp_rsk(req)->rcv_isn &&
 	    flg == TCP_FLAG_SYN &&
net/ipv4/tcp_output.c +19 -18
index 751396f22bb3..e7ea9596cec7 100644
@@ -398,13 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-#define OPTION_SACK_ADVERTISE	(1 << 0)
-#define OPTION_TS		(1 << 1)
-#define OPTION_MD5		(1 << 2)
-#define OPTION_WSCALE		(1 << 3)
-#define OPTION_FAST_OPEN_COOKIE	(1 << 8)
-#define OPTION_SMC		(1 << 9)
-
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
 #if IS_ENABLED(CONFIG_SMC)
@@ -420,17 +413,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-	u16 options;		/* bit field of OPTION_* */
-	u16 mss;		/* 0 to disable */
-	u8 ws;			/* window scale, 0 to disable */
-	u8 num_sack_blocks;	/* number of SACK blocks to include */
-	u8 hash_size;		/* bytes in hash_location */
-	__u8 *hash_location;	/* temporary pointer, overloaded */
-	__u32 tsval, tsecr;	/* need to include OPTION_TS */
-	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -543,6 +525,9 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 	}
 
 	smc_options_write(ptr, &options);
+
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,
@@ -645,6 +630,10 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
+						opts, tcp_to_sk(tp));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -708,6 +697,11 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
+						remaining, opts,
+						req_to_sk(req));
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -741,6 +735,10 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 		size += TCPOLEN_TSTAMP_ALIGNED;
 	}
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
+					   opts, tcp_to_sk(tp));
+
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
 	if (unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
@@ -3306,6 +3304,9 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tp->tcp_header_len += tcp_extopt_add_header(sk, sk);
+
 	/* If user gave his TCP_MAXSEG, record it to clamp */
 	if (tp->rx_opt.user_mss)
 		tp->rx_opt.mss_clamp = tp->rx_opt.user_mss;
net/ipv6/syncookies.c +5 -1
index e7a3a6b6cf56..65a1fb35c0cb 100644
@@ -162,7 +162,7 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 
 	/* check for timestamp cookie support */
 	memset(&tcp_opt, 0, sizeof(tcp_opt));
-	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
+	tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL, sk);
 
 	if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
 		tsoff = secure_tcpv6_ts_off(sock_net(sk),
@@ -174,6 +174,10 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
 	if (!cookie_timestamp_decode(sock_net(sk), &tcp_opt))
 		goto out;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled) &&
+	    tcp_extopt_check(sk, skb, &tcp_opt))
+		goto out;
+
 	ret = NULL;
 	req = inet_reqsk_alloc(&tcp6_request_sock_ops, sk, false);
 	if (!req)
net/ipv6/tcp_ipv6.c +28 -1
index 7178476b3d2f..05e43465aff7 100644
@@ -500,6 +500,9 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst,
 
 static void tcp_v6_reqsk_destructor(struct request_sock *req)
 {
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_destroy(req_to_sk(req));
+
 	kfree(inet_rsk(req)->ipv6_opt);
 	kfree_skb(inet_rsk(req)->pktopts);
 }
@@ -789,6 +792,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	unsigned int tot_len = sizeof(struct tcphdr);
 	struct dst_entry *dst;
 	__be32 *topt;
+	struct tcp_out_options extraopts;
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
@@ -797,6 +801,22 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		unsigned int remaining = MAX_TCP_OPTION_SPACE - tot_len;
+		u8 extraflags = rst ? TCPHDR_RST : 0;
+		int used;
+
+		if (!rst || !th->ack)
+			extraflags |= TCPHDR_ACK;
+
+		memset(&extraopts, 0, sizeof(extraopts));
+
+		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
+						   &extraopts, sk);
+
+		tot_len += used;
+	}
+
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
@@ -837,6 +857,9 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 #endif
 
+	if (static_branch_unlikely(&tcp_extopt_enabled))
+		tcp_extopt_response_write(topt, skb, t1, &extraopts, sk);
+
 	memset(&fl6, 0, sizeof(fl6));
 	fl6.daddr = ipv6_hdr(skb)->saddr;
 	fl6.saddr = ipv6_hdr(skb)->daddr;
@@ -1196,6 +1219,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	newinet->inet_daddr = newinet->inet_saddr = LOOPBACK4_IPV6;
 	newinet->inet_rcv_saddr = LOOPBACK4_IPV6;
 
+	if (static_branch_unlikely(&tcp_extopt_enabled)) {
+		tcp_extopt_move(req_to_sk(req), newsk);
+		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	/* Copy over the MD5 key from the original socket */
 	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
--
Quoted 3 lines

2.15.0

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

author
#22 lore raw

RFC v3 7/15 tcp_smc: Make SMC use TCP extra-option framework

In reply to Christoph Paasch

Adopt the extra-option framework for SMC.
It allows us to entirely remove SMC-code out of the TCP-stack.

The static key is gone, as this is now covered by the static key of the extra-option framework.

We allocate state (struct tcp_smc_opt) that indicates whether SMC was successfully negotiated or not and check this state in the relevant functions.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>
---
 include/linux/tcp.h      |   3 +-
 include/net/inet_sock.h  |   3 +-
 include/net/tcp.h        |   4 --
 net/ipv4/tcp.c           |   5 --
 net/ipv4/tcp_input.c     |  36 ----------
 net/ipv4/tcp_minisocks.c |  18 -----
 net/ipv4/tcp_output.c    |  54 ---------------
 net/smc/af_smc.c         | 175 +++++++++++++++++++++++++++++++++++++++++++++--
 8 files changed, 171 insertions(+), 127 deletions(-)
include/linux/tcp.h +1 -2
index b14426228a01..5dc4067100af 100644
@@ -257,8 +257,7 @@ struct tcp_sock {
 		syn_fastopen_ch:1, /* Active TFO re-enabling probe */
 		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
 		save_syn:1,	/* Save headers of SYN packet */
-		is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
-		syn_smc:1;	/* SYN includes SMC */
+		is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
 	u32	tlp_high_seq;	/* snd_nxt at the time of TLP retransmit. */
 
 /* RTT measurement */
include/net/inet_sock.h +1 -2
index 39efb968b7a4..8e51b4a69088 100644
@@ -90,8 +90,7 @@ struct inet_request_sock {
 				wscale_ok  : 1,
 				ecn_ok	   : 1,
 				acked	   : 1,
-				no_srccheck: 1,
-				smc_ok	   : 1;
+				no_srccheck: 1;
 	u32                     ir_mark;
 	union {
 		struct ip_options_rcu __rcu	*ireq_opt;
include/net/tcp.h +0 -4
index 86133aa24fca..27e71cb5e4cb 100644
@@ -2063,10 +2063,6 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 	return (tcp_call_bpf(sk, BPF_SOCK_OPS_NEEDS_ECN) == 1);
 }
 
-#if IS_ENABLED(CONFIG_SMC)
-extern struct static_key_false tcp_have_smc;
-#endif
-
 extern struct static_key_false tcp_extopt_enabled;
 struct tcp_extopt_store;
 
net/ipv4/tcp.c +0 -5
index 6676fc3c0059..c5d9484064c9 100644
@@ -294,11 +294,6 @@ EXPORT_SYMBOL(sysctl_tcp_mem);
 atomic_long_t tcp_memory_allocated;	/* Current allocated memory. */
 EXPORT_SYMBOL(tcp_memory_allocated);
 
-#if IS_ENABLED(CONFIG_SMC)
-DEFINE_STATIC_KEY_FALSE(tcp_have_smc);
-EXPORT_SYMBOL(tcp_have_smc);
-#endif
-
 /*
  * Current number of TCP sockets.
  */
net/ipv4/tcp_input.c +0 -36
index d4d3be8a496c..5f4a2f2e3b87 100644
@@ -3676,24 +3676,6 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
 	foc->exp = exp_opt;
 }
 
-static int smc_parse_options(const struct tcphdr *th,
-			     struct tcp_options_received *opt_rx,
-			     const unsigned char *ptr,
-			     int opsize)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (th->syn && !(opsize & 1) &&
-		    opsize >= TCPOLEN_EXP_SMC_BASE &&
-		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
-			opt_rx->smc_ok = 1;
-			return 1;
-		}
-	}
-#endif
-	return 0;
-}
-
 /* Look for tcp options. Normally only called on SYN and SYNACK packets.
  * But, this can also be called on packets in the established flow when
  * the fast version below fails.
@@ -3801,9 +3783,6 @@ void tcp_parse_options(const struct net *net,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
-				else if (smc_parse_options(th, opt_rx, ptr,
-							   opsize))
-					break;
 				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
 					 opsize >= TCPOLEN_EXP_BASE)
 					tcp_extopt_parse(get_unaligned_be32(ptr),
@@ -5580,16 +5559,6 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 	return false;
 }
 
-static void smc_check_reset_syn(struct tcp_sock *tp)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc && !tp->rx_opt.smc_ok)
-			tp->syn_smc = 0;
-	}
-#endif
-}
-
 static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 					 const struct tcphdr *th)
 {
@@ -5700,8 +5669,6 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		 * is initialized. */
 		tp->copied_seq = tp->rcv_nxt;
 
-		smc_check_reset_syn(tp);
-
 		smp_mb();
 
 		tcp_finish_connect(sk, skb);
@@ -6158,9 +6125,6 @@ static void tcp_openreq_init(struct request_sock *req,
 	ireq->ir_rmt_port = tcp_hdr(skb)->source;
 	ireq->ir_num = ntohs(tcp_hdr(skb)->dest);
 	ireq->ir_mark = inet_request_mark(sk, skb);
-#if IS_ENABLED(CONFIG_SMC)
-	ireq->smc_ok = rx_opt->smc_ok;
-#endif
 }
 
 struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
net/ipv4/tcp_minisocks.c +0 -18
index c38d12bc7dde..ce8d1f4548a2 100644
@@ -434,21 +434,6 @@ void tcp_ca_openreq_child(struct sock *sk, const struct dst_entry *dst)
 }
 EXPORT_SYMBOL_GPL(tcp_ca_openreq_child);
 
-static void smc_check_reset_syn_req(struct tcp_sock *oldtp,
-				    struct request_sock *req,
-				    struct tcp_sock *newtp)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	struct inet_request_sock *ireq;
-
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		ireq = inet_rsk(req);
-		if (oldtp->syn_smc && !ireq->smc_ok)
-			newtp->syn_smc = 0;
-	}
-#endif
-}
-
 /* This is not only more efficient than what we used to do, it eliminates
  * a lot of code duplication between IPv4/IPv6 SYN recv processing. -DaveM
  *
@@ -466,9 +451,6 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		struct tcp_request_sock *treq = tcp_rsk(req);
 		struct inet_connection_sock *newicsk = inet_csk(newsk);
 		struct tcp_sock *newtp = tcp_sk(newsk);
-		struct tcp_sock *oldtp = tcp_sk(sk);
-
-		smc_check_reset_syn_req(oldtp, req, newtp);
 
 		/* Now setup tcp_sock */
 		newtp->pred_flags = 0;
net/ipv4/tcp_output.c +0 -54

82 lines not on this page · raw message

net/smc/af_smc.c +169 -6

265 lines not on this page · raw message

Signature · 1 line
2.15.0
#23 lore raw

Re: [MPTCP] [RFC v3 07/15] tcp_smc: Make SMC use TCP extra-option framework

Quoted 11 lines · Christoph Paasch

Adopt the extra-option framework for SMC.
It allows us to entirely remove SMC-code out of the TCP-stack.

The static key is gone, as this is now covered by the static key of the extra-option framework.

We allocate state (struct tcp_smc_opt) that indicates whether SMC was successfully negotiated or not and check this state in the relevant functions.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

Unless you heard otherwise from the smc author, I think it would be good to restore her cc: when this is posted to netdev.

Is it possible to test these changes without specific hardware?

A number of SMC patches were merged last week, but didn't cause any merge conflicts when I applied this patch set to net-next (as of Tuesday).

I came across https://lwn.net/Articles/723123/ when I was trying to better understand the SMC subsystem - interesting that the person who wrote that would have preferred using AF_INET/AF_INET6 rather than adding AF_SMC. There are some parallels with MPTCP API choices there, confirming the approach of using IPPROTO_MPTCP with AF_INET/AF_INET6.

Quoted 356 lines

---

include/linux/tcp.h      |   3 +-
include/net/inet_sock.h  |   3 +-
include/net/tcp.h        |   4 --
net/ipv4/tcp.c           |   5 --
net/ipv4/tcp_input.c     |  36 ----------
net/ipv4/tcp_minisocks.c |  18 -----
net/ipv4/tcp_output.c    |  54 ---------------
net/smc/af_smc.c         | 175 +++++++++++++++++++++++++++++++++++++++++++++--
8 files changed, 171 insertions(+), 127 deletions(-)
include/linux/tcp.h +1 -2
index b14426228a01..5dc4067100af 100644
@@ -257,8 +257,7 @@ struct tcp_sock {
 		syn_fastopen_ch:1, /* Active TFO re-enabling probe */
 		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
 		save_syn:1,	/* Save headers of SYN packet */
-		is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
-		syn_smc:1;	/* SYN includes SMC */
+		is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
 	u32	tlp_high_seq;	/* snd_nxt at the time of TLP retransmit. */
 
 /* RTT measurement */
include/net/inet_sock.h +1 -2
index 39efb968b7a4..8e51b4a69088 100644
@@ -90,8 +90,7 @@ struct inet_request_sock {
 				wscale_ok  : 1,
 				ecn_ok	   : 1,
 				acked	   : 1,
-				no_srccheck: 1,
-				smc_ok	   : 1;
+				no_srccheck: 1;
 	u32                     ir_mark;
 	union {
 		struct ip_options_rcu __rcu	*ireq_opt;
include/net/tcp.h +0 -4
index 86133aa24fca..27e71cb5e4cb 100644
@@ -2063,10 +2063,6 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 	return (tcp_call_bpf(sk, BPF_SOCK_OPS_NEEDS_ECN) == 1);
 }
 
-#if IS_ENABLED(CONFIG_SMC)
-extern struct static_key_false tcp_have_smc;
-#endif
-
 extern struct static_key_false tcp_extopt_enabled;
 struct tcp_extopt_store;
 
net/ipv4/tcp.c +0 -0
index 6676fc3c0059..c5d9484064c9 100644
@@ -294,11 +294,6 @@ EXPORT_SYMBOL(sysctl_tcp_mem);

atomic_long_t tcp_memory_allocated; /* Current allocated memory. */
EXPORT_SYMBOL(tcp_memory_allocated);

net/ipv4/tcp.c +0 -5
-#if IS_ENABLED(CONFIG_SMC)
-DEFINE_STATIC_KEY_FALSE(tcp_have_smc);
-EXPORT_SYMBOL(tcp_have_smc);
-#endif
-

/*
* Current number of TCP sockets.
*/

net/ipv4/tcp_input.c +0 -36
index d4d3be8a496c..5f4a2f2e3b87 100644
@@ -3676,24 +3676,6 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
 	foc->exp = exp_opt;
 }
 
-static int smc_parse_options(const struct tcphdr *th,
-			     struct tcp_options_received *opt_rx,
-			     const unsigned char *ptr,
-			     int opsize)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (th->syn && !(opsize & 1) &&
-		    opsize >= TCPOLEN_EXP_SMC_BASE &&
-		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
-			opt_rx->smc_ok = 1;
-			return 1;
-		}
-	}
-#endif
-	return 0;
-}
-
 /* Look for tcp options. Normally only called on SYN and SYNACK packets.
 * But, this can also be called on packets in the established flow when
 * the fast version below fails.
@@ -3801,9 +3783,6 @@ void tcp_parse_options(const struct net *net,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
-				else if (smc_parse_options(th, opt_rx, ptr,
-							   opsize))
-					break;
 				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
 					 opsize >= TCPOLEN_EXP_BASE)
 					tcp_extopt_parse(get_unaligned_be32(ptr),
@@ -5580,16 +5559,6 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 	return false;
 }
 
-static void smc_check_reset_syn(struct tcp_sock *tp)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc && !tp->rx_opt.smc_ok)
-			tp->syn_smc = 0;
-	}
-#endif
-}
-
 static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 					 const struct tcphdr *th)
 {
@@ -5700,8 +5669,6 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		 * is initialized. */
 		tp->copied_seq = tp->rcv_nxt;
 
-		smc_check_reset_syn(tp);
-
 		smp_mb();
 
 		tcp_finish_connect(sk, skb);
@@ -6158,9 +6125,6 @@ static void tcp_openreq_init(struct request_sock *req,
 	ireq->ir_rmt_port = tcp_hdr(skb)->source;
 	ireq->ir_num = ntohs(tcp_hdr(skb)->dest);
 	ireq->ir_mark = inet_request_mark(sk, skb);
-#if IS_ENABLED(CONFIG_SMC)
-	ireq->smc_ok = rx_opt->smc_ok;
-#endif
 }
 
 struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
net/ipv4/tcp_minisocks.c +0 -18
index c38d12bc7dde..ce8d1f4548a2 100644
@@ -434,21 +434,6 @@ void tcp_ca_openreq_child(struct sock *sk, const struct dst_entry *dst)
 }
 EXPORT_SYMBOL_GPL(tcp_ca_openreq_child);
 
-static void smc_check_reset_syn_req(struct tcp_sock *oldtp,
-				    struct request_sock *req,
-				    struct tcp_sock *newtp)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	struct inet_request_sock *ireq;
-
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		ireq = inet_rsk(req);
-		if (oldtp->syn_smc && !ireq->smc_ok)
-			newtp->syn_smc = 0;
-	}
-#endif
-}
-
 /* This is not only more efficient than what we used to do, it eliminates
 * a lot of code duplication between IPv4/IPv6 SYN recv processing. -DaveM
 *
@@ -466,9 +451,6 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		struct tcp_request_sock *treq = tcp_rsk(req);
 		struct inet_connection_sock *newicsk = inet_csk(newsk);
 		struct tcp_sock *newtp = tcp_sk(newsk);
-		struct tcp_sock *oldtp = tcp_sk(sk);
-
-		smc_check_reset_syn_req(oldtp, req, newtp);
 
 		/* Now setup tcp_sock */
 		newtp->pred_flags = 0;
net/ipv4/tcp_output.c +0 -54
index e7ea9596cec7..0dc736849fc0 100644
@@ -398,21 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-static void smc_options_write(__be32 *ptr, u16 *options)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (unlikely(OPTION_SMC & *options)) {
-			*ptr++ = htonl((TCPOPT_NOP  << 24) |
-				       (TCPOPT_NOP  << 16) |
-				       (TCPOPT_EXP <<  8) |
-				       (TCPOLEN_EXP_SMC_BASE));
-			*ptr++ = htonl(TCPOPT_SMC_MAGIC);
-		}
-	}
-#endif
-}
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -524,45 +509,10 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 		ptr += (len + 3) >> 2;
 	}
 
-	smc_options_write(ptr, &options);
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
-static void smc_set_option(const struct tcp_sock *tp,
-			   struct tcp_out_options *opts,
-			   unsigned int *remaining)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc) {
-			if (*remaining >= TCPOLEN_EXP_SMC_BASE_ALIGNED) {
-				opts->options |= OPTION_SMC;
-				*remaining -= TCPOLEN_EXP_SMC_BASE_ALIGNED;
-			}
-		}
-	}
-#endif
-}
-
-static void smc_set_option_cond(const struct tcp_sock *tp,
-				const struct inet_request_sock *ireq,
-				struct tcp_out_options *opts,
-				unsigned int *remaining)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc && ireq->smc_ok) {
-			if (*remaining >= TCPOLEN_EXP_SMC_BASE_ALIGNED) {
-				opts->options |= OPTION_SMC;
-				*remaining -= TCPOLEN_EXP_SMC_BASE_ALIGNED;
-			}
-		}
-	}
-#endif
-}
-
 /* Compute TCP options for SYN packets. This is not the final
 * network wire format yet.
 */
@@ -628,8 +578,6 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
-	smc_set_option(tp, opts, &remaining);
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
 						opts, tcp_to_sk(tp));
@@ -695,8 +643,6 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 		}
 	}
 
-	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
 						remaining, opts,
net/smc/af_smc.c +77 -0
index daf8075f5a4c..3ef9588a4f87 100644
@@ -44,6 +44,132 @@
 #include "smc_rx.h"
 #include "smc_close.h"
 
+static unsigned int tcp_smc_opt_prepare(struct sk_buff *skb, u8 flags,
+					unsigned int remaining,
+					struct tcp_out_options *opts,
+					const struct sock *sk,
+					struct tcp_extopt_store *store);
+static __be32 *tcp_smc_opt_write(__be32 *ptr, struct sk_buff *skb,
+				 struct tcp_out_options *opts,
+				 struct sock *sk,
+				 struct tcp_extopt_store *store);
+static void tcp_smc_opt_parse(int opsize, const unsigned char *opptr,
+			      const struct sk_buff *skb,
+			      struct tcp_options_received *opt_rx,
+			      struct sock *sk,
+			      struct tcp_extopt_store *store);
+static void tcp_smc_opt_post_process(struct sock *sk,
+				     struct tcp_options_received *opt,
+				     struct tcp_extopt_store *store);
+static struct tcp_extopt_store *tcp_smc_opt_copy(struct sock *listener,
+						 struct request_sock *req,
+						 struct tcp_options_received *opt,
+						 struct tcp_extopt_store *store);
+static void tcp_smc_opt_destroy(struct tcp_extopt_store *store);
+
+struct tcp_smc_opt {
+	struct tcp_extopt_store	store;
+	int			smc_ok:1; /* SMC supported on this connection */
+	struct rcu_head		rcu;
+};
+
+static const struct tcp_extopt_ops tcp_smc_extra_ops = {
+	.option_kind	= TCPOPT_SMC_MAGIC,
+	.parse		= tcp_smc_opt_parse,
+	.post_process	= tcp_smc_opt_post_process,
+	.prepare	= tcp_smc_opt_prepare,
+	.write		= tcp_smc_opt_write,
+	.copy		= tcp_smc_opt_copy,
+	.destroy	= tcp_smc_opt_destroy,
+	.owner		= THIS_MODULE,
+};
+
+static unsigned int tcp_smc_opt_prepare(struct sk_buff *skb, u8 flags,
+					unsigned int remaining,
+					struct tcp_out_options *opts,
+					const struct sock *sk,
+					struct tcp_extopt_store *store)
+{
+	if (remaining >= TCPOLEN_EXP_SMC_BASE_ALIGNED) {
+		opts->options |= OPTION_SMC;
+		return TCPOLEN_EXP_SMC_BASE_ALIGNED;
+	}
+
+	return 0;
+}
+
+static __be32 *tcp_smc_opt_write(__be32 *ptr, struct sk_buff *skb,
+				 struct tcp_out_options *opts,
+				 struct sock *sk,
+				 struct tcp_extopt_store *store)
+{
+	if (unlikely(OPTION_SMC & opts->options)) {
+		*ptr++ = htonl((TCPOPT_NOP  << 24) |
+			       (TCPOPT_NOP  << 16) |
+			       (TCPOPT_EXP <<  8) |
+			       (TCPOLEN_EXP_SMC_BASE));
+		*ptr++ = htonl(TCPOPT_SMC_MAGIC);
+	}
+
+	return ptr;
+}
+
+static void tcp_smc_opt_parse(int opsize, const unsigned char *opptr,
+			      const struct sk_buff *skb,
+			      struct tcp_options_received *opt_rx,
+			      struct sock *sk,
+			      struct tcp_extopt_store *store)
+{
+	struct tcphdr *th = tcp_hdr(skb);

net/smc/af_smc.c line 133

Quoted 10 lines
net/smc/af_smc.c +10 -0
+
+	if (th->syn && !(opsize & 1) && opsize >= TCPOLEN_EXP_SMC_BASE)
+		opt_rx->smc_ok = 1;
+}
+
+static void tcp_smc_opt_post_process(struct sock *sk,
+				     struct tcp_options_received *opt,
+				     struct tcp_extopt_store *store)
+{
+	struct tcp_smc_opt *smc_opt = (struct tcp_smc_opt *)store;

The container_of() macro would set a type-safe example for other tcp_extopt users. If that's the standard, it would get rid of the requirement to have the tcp_extopt_store struct at the beginning of the container struct.

Mat

Quoted 190 lines
net/smc/af_smc.c +39 -0
+
+	if (sk->sk_state != TCP_SYN_SENT)
+		return;
+
+	if (opt->smc_ok)
+		smc_opt->smc_ok = 1;
+	else
+		smc_opt->smc_ok = 0;
+}
+
+static struct tcp_extopt_store *tcp_smc_opt_copy(struct sock *listener,
+						 struct request_sock *req,
+						 struct tcp_options_received *opt,
+						 struct tcp_extopt_store *store)
+{
+	struct tcp_smc_opt *smc_opt;
+
+	/* First, check if the peer sent us the smc-opt */
+	if (!opt->smc_ok)
+		return NULL;
+
+	smc_opt = kzalloc(sizeof(*smc_opt), GFP_KERNEL);
+	if (!smc_opt)
+		return NULL;
+
+	smc_opt->store.ops = &tcp_smc_extra_ops;
+
+	smc_opt->smc_ok = 1;
+
+	return (struct tcp_extopt_store *)smc_opt;
+}
+
+static void tcp_smc_opt_destroy(struct tcp_extopt_store *store)
+{
+	struct tcp_smc_opt *smc_opt = (struct tcp_smc_opt *)store;
+
+	kfree_rcu(smc_opt, rcu);
+}
+

static DEFINE_MUTEX(smc_create_lgr_pending); /* serialize link group

						 * creation
						 */
net/smc/af_smc.c +42 -4
@@ -384,13 +510,16 @@ static int smc_connect_rdma(struct smc_sock *smc)
 	struct smc_clc_msg_accept_confirm aclc;
 	int local_contact = SMC_FIRST_CONTACT;
 	struct smc_ib_device *smcibdev;
+	struct tcp_smc_opt *smc_opt;
 	struct smc_link *link;
 	u8 srv_first_contact;
 	int reason_code = 0;
 	int rc = 0;
 	u8 ibport;
 
-	if (!tcp_sk(smc->clcsock->sk)->syn_smc) {
+	smc_opt = (struct tcp_smc_opt *)tcp_extopt_find_kind(TCPOPT_SMC_MAGIC,
+							     smc->clcsock->sk);
+	if (!smc_opt || !smc_opt->smc_ok) {
 		/* peer has not signalled SMC-capability */
 		smc->use_fallback = true;
 		goto out_connected;
@@ -535,6 +664,7 @@ static int smc_connect_rdma(struct smc_sock *smc)
 static int smc_connect(struct socket *sock, struct sockaddr *addr,
 		       int alen, int flags)
 {
+	struct tcp_smc_opt *smc_opt;
 	struct sock *sk = sock->sk;
 	struct smc_sock *smc;
 	int rc = -EINVAL;
@@ -548,9 +678,17 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
 		goto out_err;
 	smc->addr = addr;	/* needed for nonblocking connect */
 
+	smc_opt = kzalloc(sizeof(*smc_opt), GFP_KERNEL);
+	if (!smc_opt) {
+		rc = -ENOMEM;
+		goto out_err;
+	}
+	smc_opt->store.ops = &tcp_smc_extra_ops;
+
 	lock_sock(sk);
 	switch (sk->sk_state) {
 	default:
+		rc = -EINVAL;
 		goto out;
 	case SMC_ACTIVE:
 		rc = -EISCONN;
@@ -560,8 +698,15 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
 		break;
 	}
 
+	/* We are the only owner of smc->clcsock->sk, so we can be lockless */
+	rc = tcp_register_extopt(&smc_opt->store, smc->clcsock->sk);
+	if (rc) {
+		release_sock(smc->clcsock->sk);
+		kfree(smc_opt);
+		goto out_err;
+	}
+
 	smc_copy_sock_settings_to_clc(smc);
-	tcp_sk(smc->clcsock->sk)->syn_smc = 1;
 	rc = kernel_connect(smc->clcsock, addr, alen, flags);
 	if (rc)
 		goto out;
@@ -760,6 +905,7 @@ static void smc_listen_work(struct work_struct *work)
 	struct smc_clc_msg_proposal *pclc;
 	struct smc_ib_device *smcibdev;
 	struct sockaddr_in peeraddr;
+	struct tcp_smc_opt *smc_opt;
 	u8 buf[SMC_CLC_MAX_LEN];
 	struct smc_link *link;
 	int reason_code = 0;
@@ -769,7 +915,9 @@ static void smc_listen_work(struct work_struct *work)
 	u8 ibport;
 
 	/* check if peer is smc capable */
-	if (!tcp_sk(newclcsock->sk)->syn_smc) {
+	smc_opt = (struct tcp_smc_opt *)tcp_extopt_find_kind(TCPOPT_SMC_MAGIC,
+							     newclcsock->sk);
+	if (!smc_opt || !smc_opt->smc_ok) {
 		new_smc->use_fallback = true;
 		goto out_connected;
 	}
@@ -962,10 +1110,18 @@ static void smc_tcp_listen_work(struct work_struct *work)
 
 static int smc_listen(struct socket *sock, int backlog)
 {
+	struct tcp_smc_opt *smc_opt;
 	struct sock *sk = sock->sk;
 	struct smc_sock *smc;
 	int rc;
 
+	smc_opt = kzalloc(sizeof(*smc_opt), GFP_KERNEL);
+	if (!smc_opt) {
+		rc = -ENOMEM;
+		goto out_err;
+	}
+	smc_opt->store.ops = &tcp_smc_extra_ops;
+
 	smc = smc_sk(sk);
 	lock_sock(sk);
 
@@ -978,11 +1134,19 @@ static int smc_listen(struct socket *sock, int backlog)
 		sk->sk_max_ack_backlog = backlog;
 		goto out;
 	}
+
+	/* We are the only owner of smc->clcsock->sk, so we can be lockless */
+	rc = tcp_register_extopt(&smc_opt->store, smc->clcsock->sk);
+	if (rc) {
+		release_sock(smc->clcsock->sk);
+		kfree(smc_opt);
+		goto out_err;
+	}
+
 	/* some socket options are handled in core, so we could not apply
 	 * them to the clc socket -- copy smc socket options to clc socket
 	 */
 	smc_copy_sock_settings_to_clc(smc);
-	tcp_sk(smc->clcsock->sk)->syn_smc = 1;
 
 	rc = kernel_listen(smc->clcsock, backlog);
 	if (rc)
@@ -995,6 +1159,7 @@ static int smc_listen(struct socket *sock, int backlog)
 

out:

	release_sock(sk);
+out_err:
	return rc;
}
net/smc/af_smc.c +0 -1
@@ -1425,7 +1590,6 @@ static int __init smc_init(void)
 		goto out_sock;
 	}
 
-	static_branch_enable(&tcp_have_smc);
 	return 0;
 

out_sock:

net/smc/af_smc.c +0 -1
@@ -1450,7 +1614,6 @@ static void __exit smc_exit(void)
 		list_del_init(&lgr->list);
 		smc_lgr_free(lgr); /* free link group */
 	}
-	static_branch_disable(&tcp_have_smc);
 	smc_ib_unregister_client();
 	sock_unregister(PF_SMC);
 	proto_unregister(&smc_proto);
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#24 lore raw

Re: [MPTCP] [RFC v3 07/15] tcp_smc: Make SMC use TCP extra-option framework

Quoted 9 lines · Mat Martineau
Quoted 6 lines · Christoph Paasch

Adopt the extra-option framework for SMC.
It allows us to entirely remove SMC-code out of the TCP-stack.

The static key is gone, as this is now covered by the static key of the extra-option framework.

Quoted 8 lines
Quoted 5 lines

We allocate state (struct tcp_smc_opt) that indicates whether SMC was successfully negotiated or not and check this state in the relevant functions.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

Unless you heard otherwise from the smc author, I think it would be good to restore her cc: when this is posted to netdev.

Yes, I will Cc the author.

Quoted 1 line

Is it possible to test these changes without specific hardware?

Actually, it seems it is possible :)

Tested it, and fixed two bugs:

* GFP_KERNEL should be GFP_ATOMIC
* The _prepare callback needs this:
+ if (!(flags & TCPHDR_SYN))
+ return 0;

Quoted 5 lines

A number of SMC patches were merged last week, but didn't cause any merge conflicts when I applied this patch set to net-next (as of Tuesday).

I came across https://lwn.net/Articles/723123/ when I was trying to better

Wow, interesting thread :)

Quoted 4 lines

understand the SMC subsystem - interesting that the person who wrote that would have preferred using AF_INET/AF_INET6 rather than adding AF_SMC. There are some parallels with MPTCP API choices there, confirming the approach of using IPPROTO_MPTCP with AF_INET/AF_INET6.

Agreed!

Quoted 363 lines
Quoted 363 lines

---

include/linux/tcp.h      |   3 +-
include/net/inet_sock.h  |   3 +-
include/net/tcp.h        |   4 --
net/ipv4/tcp.c           |   5 --
net/ipv4/tcp_input.c     |  36 ----------
net/ipv4/tcp_minisocks.c |  18 -----
net/ipv4/tcp_output.c    |  54 ---------------
net/smc/af_smc.c         | 175 +++++++++++++++++++++++++++++++++++++++++++++--
8 files changed, 171 insertions(+), 127 deletions(-)
include/linux/tcp.h +1 -2
index b14426228a01..5dc4067100af 100644
@@ -257,8 +257,7 @@ struct tcp_sock {
 		syn_fastopen_ch:1, /* Active TFO re-enabling probe */
 		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
 		save_syn:1,	/* Save headers of SYN packet */
-		is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
-		syn_smc:1;	/* SYN includes SMC */
+		is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
 	u32	tlp_high_seq;	/* snd_nxt at the time of TLP retransmit. */
 
 /* RTT measurement */
include/net/inet_sock.h +1 -2
index 39efb968b7a4..8e51b4a69088 100644
@@ -90,8 +90,7 @@ struct inet_request_sock {
 				wscale_ok  : 1,
 				ecn_ok	   : 1,
 				acked	   : 1,
-				no_srccheck: 1,
-				smc_ok	   : 1;
+				no_srccheck: 1;
 	u32                     ir_mark;
 	union {
 		struct ip_options_rcu __rcu	*ireq_opt;
include/net/tcp.h +0 -4
index 86133aa24fca..27e71cb5e4cb 100644
@@ -2063,10 +2063,6 @@ static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
 	return (tcp_call_bpf(sk, BPF_SOCK_OPS_NEEDS_ECN) == 1);
 }
 
-#if IS_ENABLED(CONFIG_SMC)
-extern struct static_key_false tcp_have_smc;
-#endif
-
 extern struct static_key_false tcp_extopt_enabled;
 struct tcp_extopt_store;
 
net/ipv4/tcp.c +0 -0
index 6676fc3c0059..c5d9484064c9 100644
@@ -294,11 +294,6 @@ EXPORT_SYMBOL(sysctl_tcp_mem);

atomic_long_t tcp_memory_allocated; /* Current allocated memory. */
EXPORT_SYMBOL(tcp_memory_allocated);

net/ipv4/tcp.c +0 -5
-#if IS_ENABLED(CONFIG_SMC)
-DEFINE_STATIC_KEY_FALSE(tcp_have_smc);
-EXPORT_SYMBOL(tcp_have_smc);
-#endif
-

/*
* Current number of TCP sockets.
*/

net/ipv4/tcp_input.c +0 -36
index d4d3be8a496c..5f4a2f2e3b87 100644
@@ -3676,24 +3676,6 @@ static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
 	foc->exp = exp_opt;
 }
 
-static int smc_parse_options(const struct tcphdr *th,
-			     struct tcp_options_received *opt_rx,
-			     const unsigned char *ptr,
-			     int opsize)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (th->syn && !(opsize & 1) &&
-		    opsize >= TCPOLEN_EXP_SMC_BASE &&
-		    get_unaligned_be32(ptr) == TCPOPT_SMC_MAGIC) {
-			opt_rx->smc_ok = 1;
-			return 1;
-		}
-	}
-#endif
-	return 0;
-}
-
 /* Look for tcp options. Normally only called on SYN and SYNACK packets.
 * But, this can also be called on packets in the established flow when
 * the fast version below fails.
@@ -3801,9 +3783,6 @@ void tcp_parse_options(const struct net *net,
 					tcp_parse_fastopen_option(opsize -
 						TCPOLEN_EXP_FASTOPEN_BASE,
 						ptr + 2, th->syn, foc, true);
-				else if (smc_parse_options(th, opt_rx, ptr,
-							   opsize))
-					break;
 				else if (static_branch_unlikely(&tcp_extopt_enabled) &&
 					 opsize >= TCPOLEN_EXP_BASE)
 					tcp_extopt_parse(get_unaligned_be32(ptr),
@@ -5580,16 +5559,6 @@ static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
 	return false;
 }
 
-static void smc_check_reset_syn(struct tcp_sock *tp)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc && !tp->rx_opt.smc_ok)
-			tp->syn_smc = 0;
-	}
-#endif
-}
-
 static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 					 const struct tcphdr *th)
 {
@@ -5700,8 +5669,6 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		 * is initialized. */
 		tp->copied_seq = tp->rcv_nxt;
 
-		smc_check_reset_syn(tp);
-
 		smp_mb();
 
 		tcp_finish_connect(sk, skb);
@@ -6158,9 +6125,6 @@ static void tcp_openreq_init(struct request_sock *req,
 	ireq->ir_rmt_port = tcp_hdr(skb)->source;
 	ireq->ir_num = ntohs(tcp_hdr(skb)->dest);
 	ireq->ir_mark = inet_request_mark(sk, skb);
-#if IS_ENABLED(CONFIG_SMC)
-	ireq->smc_ok = rx_opt->smc_ok;
-#endif
 }
 
 struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
net/ipv4/tcp_minisocks.c +0 -18
index c38d12bc7dde..ce8d1f4548a2 100644
@@ -434,21 +434,6 @@ void tcp_ca_openreq_child(struct sock *sk, const struct dst_entry *dst)
 }
 EXPORT_SYMBOL_GPL(tcp_ca_openreq_child);
 
-static void smc_check_reset_syn_req(struct tcp_sock *oldtp,
-				    struct request_sock *req,
-				    struct tcp_sock *newtp)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	struct inet_request_sock *ireq;
-
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		ireq = inet_rsk(req);
-		if (oldtp->syn_smc && !ireq->smc_ok)
-			newtp->syn_smc = 0;
-	}
-#endif
-}
-
 /* This is not only more efficient than what we used to do, it eliminates
 * a lot of code duplication between IPv4/IPv6 SYN recv processing. -DaveM
 *
@@ -466,9 +451,6 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		struct tcp_request_sock *treq = tcp_rsk(req);
 		struct inet_connection_sock *newicsk = inet_csk(newsk);
 		struct tcp_sock *newtp = tcp_sk(newsk);
-		struct tcp_sock *oldtp = tcp_sk(sk);
-
-		smc_check_reset_syn_req(oldtp, req, newtp);
 
 		/* Now setup tcp_sock */
 		newtp->pred_flags = 0;
net/ipv4/tcp_output.c +0 -54
index e7ea9596cec7..0dc736849fc0 100644
@@ -398,21 +398,6 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 	return tp->snd_una != tp->snd_up;
 }
 
-static void smc_options_write(__be32 *ptr, u16 *options)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (unlikely(OPTION_SMC & *options)) {
-			*ptr++ = htonl((TCPOPT_NOP  << 24) |
-				       (TCPOPT_NOP  << 16) |
-				       (TCPOPT_EXP <<  8) |
-				       (TCPOLEN_EXP_SMC_BASE));
-			*ptr++ = htonl(TCPOPT_SMC_MAGIC);
-		}
-	}
-#endif
-}
-
 /* Write previously computed TCP options to the packet.
 *
 * Beware: Something in the Internet is very sensitive to the ordering of
@@ -524,45 +509,10 @@ static void tcp_options_write(__be32 *ptr, struct sk_buff *skb, struct sock *sk,
 		ptr += (len + 3) >> 2;
 	}
 
-	smc_options_write(ptr, &options);
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		tcp_extopt_write(ptr, skb, opts, sk);
 }
 
-static void smc_set_option(const struct tcp_sock *tp,
-			   struct tcp_out_options *opts,
-			   unsigned int *remaining)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc) {
-			if (*remaining >= TCPOLEN_EXP_SMC_BASE_ALIGNED) {
-				opts->options |= OPTION_SMC;
-				*remaining -= TCPOLEN_EXP_SMC_BASE_ALIGNED;
-			}
-		}
-	}
-#endif
-}
-
-static void smc_set_option_cond(const struct tcp_sock *tp,
-				const struct inet_request_sock *ireq,
-				struct tcp_out_options *opts,
-				unsigned int *remaining)
-{
-#if IS_ENABLED(CONFIG_SMC)
-	if (static_branch_unlikely(&tcp_have_smc)) {
-		if (tp->syn_smc && ireq->smc_ok) {
-			if (*remaining >= TCPOLEN_EXP_SMC_BASE_ALIGNED) {
-				opts->options |= OPTION_SMC;
-				*remaining -= TCPOLEN_EXP_SMC_BASE_ALIGNED;
-			}
-		}
-	}
-#endif
-}
-
 /* Compute TCP options for SYN packets. This is not the final
 * network wire format yet.
 */
@@ -628,8 +578,6 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 		}
 	}
 
-	smc_set_option(tp, opts, &remaining);
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN, remaining,
 						opts, tcp_to_sk(tp));
@@ -695,8 +643,6 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 		}
 	}
 
-	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		remaining -= tcp_extopt_prepare(skb, TCPHDR_SYN | TCPHDR_ACK,
 						remaining, opts,
net/smc/af_smc.c +83 -0
index daf8075f5a4c..3ef9588a4f87 100644
@@ -44,6 +44,132 @@
 #include "smc_rx.h"
 #include "smc_close.h"
 
+static unsigned int tcp_smc_opt_prepare(struct sk_buff *skb, u8 flags,
+					unsigned int remaining,
+					struct tcp_out_options *opts,
+					const struct sock *sk,
+					struct tcp_extopt_store *store);
+static __be32 *tcp_smc_opt_write(__be32 *ptr, struct sk_buff *skb,
+				 struct tcp_out_options *opts,
+				 struct sock *sk,
+				 struct tcp_extopt_store *store);
+static void tcp_smc_opt_parse(int opsize, const unsigned char *opptr,
+			      const struct sk_buff *skb,
+			      struct tcp_options_received *opt_rx,
+			      struct sock *sk,
+			      struct tcp_extopt_store *store);
+static void tcp_smc_opt_post_process(struct sock *sk,
+				     struct tcp_options_received *opt,
+				     struct tcp_extopt_store *store);
+static struct tcp_extopt_store *tcp_smc_opt_copy(struct sock *listener,
+						 struct request_sock *req,
+						 struct tcp_options_received *opt,
+						 struct tcp_extopt_store *store);
+static void tcp_smc_opt_destroy(struct tcp_extopt_store *store);
+
+struct tcp_smc_opt {
+	struct tcp_extopt_store	store;
+	int			smc_ok:1; /* SMC supported on this connection */
+	struct rcu_head		rcu;
+};
+
+static const struct tcp_extopt_ops tcp_smc_extra_ops = {
+	.option_kind	= TCPOPT_SMC_MAGIC,
+	.parse		= tcp_smc_opt_parse,
+	.post_process	= tcp_smc_opt_post_process,
+	.prepare	= tcp_smc_opt_prepare,
+	.write		= tcp_smc_opt_write,
+	.copy		= tcp_smc_opt_copy,
+	.destroy	= tcp_smc_opt_destroy,
+	.owner		= THIS_MODULE,
+};
+
+static unsigned int tcp_smc_opt_prepare(struct sk_buff *skb, u8 flags,
+					unsigned int remaining,
+					struct tcp_out_options *opts,
+					const struct sock *sk,
+					struct tcp_extopt_store *store)
+{
+	if (remaining >= TCPOLEN_EXP_SMC_BASE_ALIGNED) {
+		opts->options |= OPTION_SMC;
+		return TCPOLEN_EXP_SMC_BASE_ALIGNED;
+	}
+
+	return 0;
+}
+
+static __be32 *tcp_smc_opt_write(__be32 *ptr, struct sk_buff *skb,
+				 struct tcp_out_options *opts,
+				 struct sock *sk,
+				 struct tcp_extopt_store *store)
+{
+	if (unlikely(OPTION_SMC & opts->options)) {
+		*ptr++ = htonl((TCPOPT_NOP  << 24) |
+			       (TCPOPT_NOP  << 16) |
+			       (TCPOPT_EXP <<  8) |
+			       (TCPOLEN_EXP_SMC_BASE));
+		*ptr++ = htonl(TCPOPT_SMC_MAGIC);
+	}
+
+	return ptr;
+}
+
+static void tcp_smc_opt_parse(int opsize, const unsigned char *opptr,
+			      const struct sk_buff *skb,
+			      struct tcp_options_received *opt_rx,
+			      struct sock *sk,
+			      struct tcp_extopt_store *store)
+{
+	struct tcphdr *th = tcp_hdr(skb);
+
+	if (th->syn && !(opsize & 1) && opsize >= TCPOLEN_EXP_SMC_BASE)
+		opt_rx->smc_ok = 1;
+}
+
+static void tcp_smc_opt_post_process(struct sock *sk,
Quoted 8 lines

net/smc/af_smc.c line 133

Quoted 4 lines
net/smc/af_smc.c +4 -0
+				     struct tcp_options_received *opt,
+				     struct tcp_extopt_store *store)
+{
+	struct tcp_smc_opt *smc_opt = (struct tcp_smc_opt *)store;

The container_of() macro would set a type-safe example for other tcp_extopt users. If that's the standard, it would get rid of the requirement to have the tcp_extopt_store struct at the beginning of the container struct.

Good point - I will change that.

Christoph

Quoted 199 lines

Mat

Quoted 190 lines
net/smc/af_smc.c +39 -0
+
+	if (sk->sk_state != TCP_SYN_SENT)
+		return;
+
+	if (opt->smc_ok)
+		smc_opt->smc_ok = 1;
+	else
+		smc_opt->smc_ok = 0;
+}
+
+static struct tcp_extopt_store *tcp_smc_opt_copy(struct sock *listener,
+						 struct request_sock *req,
+						 struct tcp_options_received *opt,
+						 struct tcp_extopt_store *store)
+{
+	struct tcp_smc_opt *smc_opt;
+
+	/* First, check if the peer sent us the smc-opt */
+	if (!opt->smc_ok)
+		return NULL;
+
+	smc_opt = kzalloc(sizeof(*smc_opt), GFP_KERNEL);
+	if (!smc_opt)
+		return NULL;
+
+	smc_opt->store.ops = &tcp_smc_extra_ops;
+
+	smc_opt->smc_ok = 1;
+
+	return (struct tcp_extopt_store *)smc_opt;
+}
+
+static void tcp_smc_opt_destroy(struct tcp_extopt_store *store)
+{
+	struct tcp_smc_opt *smc_opt = (struct tcp_smc_opt *)store;
+
+	kfree_rcu(smc_opt, rcu);
+}
+

static DEFINE_MUTEX(smc_create_lgr_pending); /* serialize link group

						 * creation
						 */
net/smc/af_smc.c +42 -4
@@ -384,13 +510,16 @@ static int smc_connect_rdma(struct smc_sock *smc)
 	struct smc_clc_msg_accept_confirm aclc;
 	int local_contact = SMC_FIRST_CONTACT;
 	struct smc_ib_device *smcibdev;
+	struct tcp_smc_opt *smc_opt;
 	struct smc_link *link;
 	u8 srv_first_contact;
 	int reason_code = 0;
 	int rc = 0;
 	u8 ibport;
 
-	if (!tcp_sk(smc->clcsock->sk)->syn_smc) {
+	smc_opt = (struct tcp_smc_opt *)tcp_extopt_find_kind(TCPOPT_SMC_MAGIC,
+							     smc->clcsock->sk);
+	if (!smc_opt || !smc_opt->smc_ok) {
 		/* peer has not signalled SMC-capability */
 		smc->use_fallback = true;
 		goto out_connected;
@@ -535,6 +664,7 @@ static int smc_connect_rdma(struct smc_sock *smc)
 static int smc_connect(struct socket *sock, struct sockaddr *addr,
 		       int alen, int flags)
 {
+	struct tcp_smc_opt *smc_opt;
 	struct sock *sk = sock->sk;
 	struct smc_sock *smc;
 	int rc = -EINVAL;
@@ -548,9 +678,17 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
 		goto out_err;
 	smc->addr = addr;	/* needed for nonblocking connect */
 
+	smc_opt = kzalloc(sizeof(*smc_opt), GFP_KERNEL);
+	if (!smc_opt) {
+		rc = -ENOMEM;
+		goto out_err;
+	}
+	smc_opt->store.ops = &tcp_smc_extra_ops;
+
 	lock_sock(sk);
 	switch (sk->sk_state) {
 	default:
+		rc = -EINVAL;
 		goto out;
 	case SMC_ACTIVE:
 		rc = -EISCONN;
@@ -560,8 +698,15 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
 		break;
 	}
 
+	/* We are the only owner of smc->clcsock->sk, so we can be lockless */
+	rc = tcp_register_extopt(&smc_opt->store, smc->clcsock->sk);
+	if (rc) {
+		release_sock(smc->clcsock->sk);
+		kfree(smc_opt);
+		goto out_err;
+	}
+
 	smc_copy_sock_settings_to_clc(smc);
-	tcp_sk(smc->clcsock->sk)->syn_smc = 1;
 	rc = kernel_connect(smc->clcsock, addr, alen, flags);
 	if (rc)
 		goto out;
@@ -760,6 +905,7 @@ static void smc_listen_work(struct work_struct *work)
 	struct smc_clc_msg_proposal *pclc;
 	struct smc_ib_device *smcibdev;
 	struct sockaddr_in peeraddr;
+	struct tcp_smc_opt *smc_opt;
 	u8 buf[SMC_CLC_MAX_LEN];
 	struct smc_link *link;
 	int reason_code = 0;
@@ -769,7 +915,9 @@ static void smc_listen_work(struct work_struct *work)
 	u8 ibport;
 
 	/* check if peer is smc capable */
-	if (!tcp_sk(newclcsock->sk)->syn_smc) {
+	smc_opt = (struct tcp_smc_opt *)tcp_extopt_find_kind(TCPOPT_SMC_MAGIC,
+							     newclcsock->sk);
+	if (!smc_opt || !smc_opt->smc_ok) {
 		new_smc->use_fallback = true;
 		goto out_connected;
 	}
@@ -962,10 +1110,18 @@ static void smc_tcp_listen_work(struct work_struct *work)
 
 static int smc_listen(struct socket *sock, int backlog)
 {
+	struct tcp_smc_opt *smc_opt;
 	struct sock *sk = sock->sk;
 	struct smc_sock *smc;
 	int rc;
 
+	smc_opt = kzalloc(sizeof(*smc_opt), GFP_KERNEL);
+	if (!smc_opt) {
+		rc = -ENOMEM;
+		goto out_err;
+	}
+	smc_opt->store.ops = &tcp_smc_extra_ops;
+
 	smc = smc_sk(sk);
 	lock_sock(sk);
 
@@ -978,11 +1134,19 @@ static int smc_listen(struct socket *sock, int backlog)
 		sk->sk_max_ack_backlog = backlog;
 		goto out;
 	}
+
+	/* We are the only owner of smc->clcsock->sk, so we can be lockless */
+	rc = tcp_register_extopt(&smc_opt->store, smc->clcsock->sk);
+	if (rc) {
+		release_sock(smc->clcsock->sk);
+		kfree(smc_opt);
+		goto out_err;
+	}
+
 	/* some socket options are handled in core, so we could not apply
 	 * them to the clc socket -- copy smc socket options to clc socket
 	 */
 	smc_copy_sock_settings_to_clc(smc);
-	tcp_sk(smc->clcsock->sk)->syn_smc = 1;
 
 	rc = kernel_listen(smc->clcsock, backlog);
 	if (rc)
@@ -995,6 +1159,7 @@ static int smc_listen(struct socket *sock, int backlog)
 

out:

	release_sock(sk);
+out_err:
	return rc;
}
net/smc/af_smc.c +0 -1
@@ -1425,7 +1590,6 @@ static int __init smc_init(void)
 		goto out_sock;
 	}
 
-	static_branch_enable(&tcp_have_smc);
 	return 0;
 

out_sock:

net/smc/af_smc.c +0 -1
@@ -1450,7 +1614,6 @@ static void __exit smc_exit(void)
 		list_del_init(&lgr->list);
 		smc_lgr_free(lgr); /* free link group */
 	}
-	static_branch_disable(&tcp_have_smc);
 	smc_ib_unregister_client();
 	sock_unregister(PF_SMC);
 	proto_unregister(&smc_proto);
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC

author
#25 lore raw

RFC v3 8/15 tcp_md5: Don't pass along md5-key

In reply to Christoph Paasch

It is much cleaner to store the key-pointer in tcp_out_options.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:

    v3: * Don't init to NULL opts->md5 in tcp_syn_options
 include/linux/tcp.h   |  1 +
 net/ipv4/tcp_output.c | 46 +++++++++++++++++++---------------------------
 2 files changed, 20 insertions(+), 27 deletions(-)
include/linux/tcp.h +1 -0
index 5dc4067100af..a1633d467af5 100644
@@ -131,6 +131,7 @@ struct tcp_out_options {
 	__u8 *hash_location;	/* temporary pointer, overloaded */
 	__u32 tsval, tsecr;	/* need to include OPTION_TS */
 	struct tcp_fastopen_cookie *fastopen_cookie;	/* Fast open cookie */
+	struct tcp_md5sig_key *md5; /* TCP_MD5 signature key */
 };
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
net/ipv4/tcp_output.c +19 -27

149 lines not on this page · raw message

Signature · 1 line
2.15.0
author
#26 lore raw

RFC v3 9/15 tcp_md5: Detect key inside tcp_v4_send_ack instead of passing it as an argument

In reply to Christoph Paasch

This will simplify to consolidate the TCP_MD5-code into a single place.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Use tcp_twsk() directly

 net/ipv4/tcp_ipv4.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
net/ipv4/tcp_ipv4.c +14 -4
index ad87dd2db092..74667de1e049 100644
@@ -760,7 +760,6 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 static void tcp_v4_send_ack(const struct sock *sk,
 			    struct sk_buff *skb, u32 seq, u32 ack,
 			    u32 win, u32 tsval, u32 tsecr, int oif,
-			    struct tcp_md5sig_key *key,
 			    int reply_flags, u8 tos)
 {
 	const struct tcphdr *th = tcp_hdr(skb);
@@ -768,6 +767,9 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		struct tcphdr th;
 		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
+#ifdef CONFIG_TCP_MD5SIG
+	struct tcp_md5sig_key *key;
+#endif
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
 	int offset = 0;
@@ -796,6 +798,17 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.ack     = 1;
 	rep.th.window  = htons(win);
 
+#ifdef CONFIG_TCP_MD5SIG
+	if (sk->sk_state == TCP_TIME_WAIT) {
+		key = tcp_twsk_md5_key(tcp_twsk(sk));
+	} else if (sk->sk_state == TCP_NEW_SYN_RECV) {
+		key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&ip_hdr(skb)->saddr,
+					AF_INET);
+	} else {
+		key = NULL;	/* Should not happen */
+	}
+#endif
+
 	if (static_branch_unlikely(&tcp_extopt_enabled)) {
 		unsigned int remaining;
 		struct tcp_out_options opts;
@@ -865,7 +878,6 @@ static void tcp_v4_timewait_ack(struct sock *sk, struct sk_buff *skb)
 			tcp_time_stamp_raw() + tcptw->tw_ts_offset,
 			tcptw->tw_ts_recent,
 			tw->tw_bound_dev_if,
-			tcp_twsk_md5_key(tcptw),
 			tw->tw_transparent ? IP_REPLY_ARG_NOSRCCHECK : 0,
 			tw->tw_tos
 			);
@@ -893,8 +905,6 @@ static void tcp_v4_reqsk_send_ack(const struct sock *sk, struct sk_buff *skb,
 			tcp_time_stamp_raw() + tcp_rsk(req)->ts_off,
 			req->ts_recent,
 			0,
-			tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&ip_hdr(skb)->saddr,
-					  AF_INET),
 			inet_rsk(req)->no_srccheck ? IP_REPLY_ARG_NOSRCCHECK : 0,
 			ip_hdr(skb)->tos);
 }
Signature · 1 line
2.15.0
author
#27 lore raw

RFC v3 10/15 tcp_md5: Detect key inside tcp_v6_send_response instead of passing it as an argument

In reply to Christoph Paasch

We want to move all the TCP-MD5 code to a single place which enables us to factor the TCP-MD5 code out of the TCP-stack into the extra-option framework.

Detection of whether or not to drop the segment (as done in tcp_v6_send_reset()) has now been moved to tcp_v6_send_response(). So we needed to adapt the latter so that it can handle the case where we want to exit without sending anything.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Fix a compiler error because of #if CONFIG_TCP_MD5SIG

 net/ipv6/tcp_ipv6.c | 119 +++++++++++++++++++++++++---------------------------
 1 file changed, 57 insertions(+), 62 deletions(-)
net/ipv6/tcp_ipv6.c +57 -62
index 05e43465aff7..fccfce82f102 100644
@@ -82,12 +82,6 @@ static const struct inet_connection_sock_af_ops ipv6_specific;
 #ifdef CONFIG_TCP_MD5SIG
 static const struct tcp_sock_af_ops tcp_sock_ipv6_specific;
 static const struct tcp_sock_af_ops tcp_sock_ipv6_mapped_specific;
-#else
-static struct tcp_md5sig_key *tcp_v6_md5_do_lookup(const struct sock *sk,
-						   const struct in6_addr *addr)
-{
-	return NULL;
-}
 #endif
 
 static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
@@ -780,12 +774,11 @@ static const struct tcp_request_sock_ops tcp_request_sock_ipv6_ops = {
 
 static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32 seq,
 				 u32 ack, u32 win, u32 tsval, u32 tsecr,
-				 int oif, struct tcp_md5sig_key *key, int rst,
-				 u8 tclass, __be32 label)
+				 int oif, int rst, u8 tclass, __be32 label)
 {
 	const struct tcphdr *th = tcp_hdr(skb);
 	struct tcphdr *t1;
-	struct sk_buff *buff;
+	struct sk_buff *buff = NULL;
 	struct flowi6 fl6;
 	struct net *net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev);
 	struct sock *ctl_sk = net->ipv6.tcp_sk;
@@ -793,10 +786,54 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	struct dst_entry *dst;
 	__be32 *topt;
 	struct tcp_out_options extraopts;
+#ifdef CONFIG_TCP_MD5SIG
+	struct tcp_md5sig_key *key = NULL;
+	const __u8 *hash_location = NULL;
+	struct ipv6hdr *ipv6h = ipv6_hdr(skb);
+#endif
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
 #ifdef CONFIG_TCP_MD5SIG
+	rcu_read_lock();
+	hash_location = tcp_parse_md5sig_option(th);
+	if (sk && sk_fullsock(sk)) {
+		key = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
+	} else if (sk && sk->sk_state == TCP_TIME_WAIT) {
+		struct tcp_timewait_sock *tcptw = tcp_twsk(sk);
+
+		key = tcp_twsk_md5_key(tcptw);
+	} else if (sk && sk->sk_state == TCP_NEW_SYN_RECV) {
+		key = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
+	} else if (hash_location) {
+		unsigned char newhash[16];
+		struct sock *sk1 = NULL;
+		int genhash;
+
+		/* active side is lost. Try to find listening socket through
+		 * source port, and then find md5 key through listening socket.
+		 * we are not loose security here:
+		 * Incoming packet is checked with md5 hash with finding key,
+		 * no RST generated if md5 hash doesn't match.
+		 */
+		sk1 = inet6_lookup_listener(dev_net(skb_dst(skb)->dev),
+					    &tcp_hashinfo, NULL, 0,
+					    &ipv6h->saddr,
+					    th->source, &ipv6h->daddr,
+					    ntohs(th->source), tcp_v6_iif(skb),
+					    tcp_v6_sdif(skb));
+		if (!sk1)
+			goto out;
+
+		key = tcp_v6_md5_do_lookup(sk1, &ipv6h->saddr);
+		if (!key)
+			goto out;
+
+		genhash = tcp_v6_md5_hash_skb(newhash, key, NULL, skb);
+		if (genhash || memcmp(hash_location, newhash, 16) != 0)
+			goto out;
+	}
+
 	if (key)
 		tot_len += TCPOLEN_MD5SIG_ALIGNED;
 #endif
@@ -820,7 +857,7 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + tot_len,
 			 GFP_ATOMIC);
 	if (!buff)
-		return;
+		goto out;
 
 	skb_reserve(buff, MAX_HEADER + sizeof(struct ipv6hdr) + tot_len);
 
@@ -897,24 +934,21 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		TCP_INC_STATS(net, TCP_MIB_OUTSEGS);
 		if (rst)
 			TCP_INC_STATS(net, TCP_MIB_OUTRSTS);
-		return;
+		buff = NULL;
 	}
 
+out:
 	kfree_skb(buff);
+
+#ifdef CONFIG_TCP_MD5SIG
+	rcu_read_unlock();
+#endif
 }
 
 static void tcp_v6_send_reset(const struct sock *sk, struct sk_buff *skb)
 {
 	const struct tcphdr *th = tcp_hdr(skb);
 	u32 seq = 0, ack_seq = 0;
-	struct tcp_md5sig_key *key = NULL;
-#ifdef CONFIG_TCP_MD5SIG
-	const __u8 *hash_location = NULL;
-	struct ipv6hdr *ipv6h = ipv6_hdr(skb);
-	unsigned char newhash[16];
-	int genhash;
-	struct sock *sk1 = NULL;
-#endif
 	int oif = 0;
 
 	if (th->rst)
@@ -926,38 +960,6 @@ static void tcp_v6_send_reset(const struct sock *sk, struct sk_buff *skb)
 	if (!sk && !ipv6_unicast_destination(skb))
 		return;
 
-#ifdef CONFIG_TCP_MD5SIG
-	rcu_read_lock();
-	hash_location = tcp_parse_md5sig_option(th);
-	if (sk && sk_fullsock(sk)) {
-		key = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
-	} else if (hash_location) {
-		/*
-		 * active side is lost. Try to find listening socket through
-		 * source port, and then find md5 key through listening socket.
-		 * we are not loose security here:
-		 * Incoming packet is checked with md5 hash with finding key,
-		 * no RST generated if md5 hash doesn't match.
-		 */
-		sk1 = inet6_lookup_listener(dev_net(skb_dst(skb)->dev),
-					   &tcp_hashinfo, NULL, 0,
-					   &ipv6h->saddr,
-					   th->source, &ipv6h->daddr,
-					   ntohs(th->source), tcp_v6_iif(skb),
-					   tcp_v6_sdif(skb));
-		if (!sk1)
-			goto out;
-
-		key = tcp_v6_md5_do_lookup(sk1, &ipv6h->saddr);
-		if (!key)
-			goto out;
-
-		genhash = tcp_v6_md5_hash_skb(newhash, key, NULL, skb);
-		if (genhash || memcmp(hash_location, newhash, 16) != 0)
-			goto out;
-	}
-#endif
-
 	if (th->ack)
 		seq = ntohl(th->ack_seq);
 	else
@@ -969,20 +971,14 @@ static void tcp_v6_send_reset(const struct sock *sk, struct sk_buff *skb)
 		trace_tcp_send_reset(sk, skb);
 	}
 
-	tcp_v6_send_response(sk, skb, seq, ack_seq, 0, 0, 0, oif, key, 1, 0, 0);
-
-#ifdef CONFIG_TCP_MD5SIG
-out:
-	rcu_read_unlock();
-#endif
+	tcp_v6_send_response(sk, skb, seq, ack_seq, 0, 0, 0, oif, 1, 0, 0);
 }
 
 static void tcp_v6_send_ack(const struct sock *sk, struct sk_buff *skb, u32 seq,
 			    u32 ack, u32 win, u32 tsval, u32 tsecr, int oif,
-			    struct tcp_md5sig_key *key, u8 tclass,
-			    __be32 label)
+			    u8 tclass, __be32 label)
 {
-	tcp_v6_send_response(sk, skb, seq, ack, win, tsval, tsecr, oif, key, 0,
+	tcp_v6_send_response(sk, skb, seq, ack, win, tsval, tsecr, oif, 0,
 			     tclass, label);
 }
 
@@ -994,7 +990,7 @@ static void tcp_v6_timewait_ack(struct sock *sk, struct sk_buff *skb)
 	tcp_v6_send_ack(sk, skb, tcptw->tw_snd_nxt, tcptw->tw_rcv_nxt,
 			tcptw->tw_rcv_wnd >> tw->tw_rcv_wscale,
 			tcp_time_stamp_raw() + tcptw->tw_ts_offset,
-			tcptw->tw_ts_recent, tw->tw_bound_dev_if, tcp_twsk_md5_key(tcptw),
+			tcptw->tw_ts_recent, tw->tw_bound_dev_if,
 			tw->tw_tclass, cpu_to_be32(tw->tw_flowlabel));
 
 	inet_twsk_put(tw);
@@ -1017,7 +1013,6 @@ static void tcp_v6_reqsk_send_ack(const struct sock *sk, struct sk_buff *skb,
 			req->rsk_rcv_wnd >> inet_rsk(req)->rcv_wscale,
 			tcp_time_stamp_raw() + tcp_rsk(req)->ts_off,
 			req->ts_recent, sk->sk_bound_dev_if,
-			tcp_v6_md5_do_lookup(sk, &ipv6_hdr(skb)->saddr),
 			0, 0);
 }
 
Signature · 1 line
2.15.0
author
#28 lore raw

RFC v3 11/15 tcp_md5: Check for TCP_MD5 after TCP Timestamps in tcp_established_options

In reply to Christoph Paasch

It really does not matter, because we never use TCP timestamps when TCP_MD5 is enabled (see tcp_syn_options).

Moving TCP_MD5 a bit lower allows for easier adoption of the tcp_extra_option framework.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>
---
 net/ipv4/tcp_output.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
net/ipv4/tcp_output.c +7 -7
index 2ca662499c28..b0297f19984f 100644
@@ -659,6 +659,13 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 
 	opts->options = 0;
 
+	if (likely(tp->rx_opt.tstamp_ok)) {
+		opts->options |= OPTION_TS;
+		opts->tsval = skb ? tcp_skb_timestamp(skb) + tp->tsoffset : 0;
+		opts->tsecr = tp->rx_opt.ts_recent;
+		size += TCPOLEN_TSTAMP_ALIGNED;
+	}
+
 #ifdef CONFIG_TCP_MD5SIG
 	opts->md5 = tp->af_specific->md5_lookup(sk, sk);
 	if (unlikely(opts->md5)) {
@@ -669,13 +676,6 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 	opts->md5 = NULL;
 #endif
 
-	if (likely(tp->rx_opt.tstamp_ok)) {
-		opts->options |= OPTION_TS;
-		opts->tsval = skb ? tcp_skb_timestamp(skb) + tp->tsoffset : 0;
-		opts->tsecr = tp->rx_opt.ts_recent;
-		size += TCPOLEN_TSTAMP_ALIGNED;
-	}
-
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		size += tcp_extopt_prepare(skb, 0, MAX_TCP_OPTION_SPACE - size,
 					   opts, tcp_to_sk(tp));
Signature · 1 line
2.15.0
author
#29 lore raw

RFC v3 12/15 tcp_md5: Move TCP-MD5 code out of TCP itself

In reply to Christoph Paasch

This is all just copy-pasting the TCP_MD5-code into functions that are placed in net/ipv4/tcp_md5.c.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Add missing EXPORT_SYMBOL_GPL

 include/linux/inet_diag.h |    1 +
 include/linux/tcp_md5.h   |  138 ++++++
 include/net/tcp.h         |   77 ----
 net/ipv4/Makefile         |    1 +
 net/ipv4/tcp.c            |  133 +-----
 net/ipv4/tcp_diag.c       |   81 +---
 net/ipv4/tcp_input.c      |   38 --
 net/ipv4/tcp_ipv4.c       |  520 ++-------------------
 net/ipv4/tcp_md5.c        | 1103 +++++++++++++++++++++++++++++++++++++++++++++
 net/ipv4/tcp_minisocks.c  |   25 +-
 net/ipv4/tcp_output.c     |    4 +-
 net/ipv6/tcp_ipv6.c       |  318 +------------
 12 files changed, 1306 insertions(+), 1133 deletions(-)
 create mode 100644 include/linux/tcp_md5.h
 create mode 100644 net/ipv4/tcp_md5.c
include/linux/inet_diag.h +1 -0
index 39faaaf843e1..1ef6727e41c9 100644
@@ -2,6 +2,7 @@
 #ifndef _INET_DIAG_H_
 #define _INET_DIAG_H_ 1
 
+#include <linux/user_namespace.h>
 #include <uapi/linux/inet_diag.h>
 
 struct net;
include/linux/tcp_md5.h new file +138 -0

138 lines not on this page · raw message

include/net/tcp.h +0 -77

95 lines not on this page · raw message

net/ipv4/Makefile +1 -0
index c6c8ad1d4b6d..9262d9a01035 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_TCP_CONG_LP) += tcp_lp.o
 obj-$(CONFIG_TCP_CONG_YEAH) += tcp_yeah.o
 obj-$(CONFIG_TCP_CONG_ILLINOIS) += tcp_illinois.o
 obj-$(CONFIG_NETLABEL) += cipso_ipv4.o
+obj-$(CONFIG_TCP_MD5SIG) += tcp_md5.o
 
 obj-$(CONFIG_XFRM) += xfrm4_policy.o xfrm4_state.o xfrm4_input.o \
 		      xfrm4_output.o xfrm4_protocol.o
net/ipv4/tcp.c +1 -132

145 lines not on this page · raw message

net/ipv4/tcp_diag.c +6 -75

103 lines not on this page · raw message

net/ipv4/tcp_input.c +0 -38
index 5f4a2f2e3b87..e89e22920c2b 100644
@@ -3854,44 +3854,6 @@ static bool tcp_fast_parse_options(const struct net *net,
 	return false;
 }
 
-#ifdef CONFIG_TCP_MD5SIG
-/*
- * Parse MD5 Signature option
- */
-const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
-{
-	int length = (th->doff << 2) - sizeof(*th);
-	const u8 *ptr = (const u8 *)(th + 1);
-
-	/* If the TCP option is too short, we can short cut */
-	if (length < TCPOLEN_MD5SIG)
-		return NULL;
-
-	while (length > 0) {
-		int opcode = *ptr++;
-		int opsize;
-
-		switch (opcode) {
-		case TCPOPT_EOL:
-			return NULL;
-		case TCPOPT_NOP:
-			length--;
-			continue;
-		default:
-			opsize = *ptr++;
-			if (opsize < 2 || opsize > length)
-				return NULL;
-			if (opcode == TCPOPT_MD5SIG)
-				return opsize == TCPOLEN_MD5SIG ? ptr : NULL;
-		}
-		ptr += opsize - 2;
-		length -= opsize;
-	}
-	return NULL;
-}
-EXPORT_SYMBOL(tcp_parse_md5sig_option);
-#endif
-
 /* Sorry, PAWS as specified is broken wrt. pure-ACKs -DaveM
  *
  * It is not fatal. If this ACK does _not_ change critical state (seqs, window)
net/ipv4/tcp_ipv4.c +35 -485

644 lines not on this page · raw message

net/ipv4/tcp_md5.c new file +1103 -0

1,103 lines not on this page · raw message

net/ipv4/tcp_minisocks.c +4 -21
index ce8d1f4548a2..aeb68687a75c 100644
@@ -22,6 +22,7 @@
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/sysctl.h>
+#include <linux/tcp_md5.h>
 #include <linux/workqueue.h>
 #include <linux/static_key.h>
 #include <net/tcp.h>
@@ -295,21 +296,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
 		}
 #ifdef CONFIG_TCP_MD5SIG
-		/*
-		 * The timewait bucket does not have the key DB from the
-		 * sock structure. We just make a quick copy of the
-		 * md5 key being used (if indeed we are using one)
-		 * so the timewait ack generating code has the key.
-		 */
-		do {
-			struct tcp_md5sig_key *key;
-			tcptw->tw_md5_key = NULL;
-			key = tp->af_specific->md5_lookup(sk, sk);
-			if (key) {
-				tcptw->tw_md5_key = kmemdup(key, sizeof(*key), GFP_ATOMIC);
-				BUG_ON(tcptw->tw_md5_key && !tcp_alloc_md5sig_pool());
-			}
-		} while (0);
+		tcp_md5_time_wait(sk, tw);
 #endif
 
 		/* Get the TIME_WAIT timeout firing. */
@@ -345,10 +332,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 void tcp_twsk_destructor(struct sock *sk)
 {
 #ifdef CONFIG_TCP_MD5SIG
-	struct tcp_timewait_sock *twsk = tcp_twsk(sk);
-
-	if (twsk->tw_md5_key)
-		kfree_rcu(twsk->tw_md5_key, rcu);
+	tcp_md5_twsk_destructor(sk);
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled))
@@ -537,8 +521,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		newtp->tsoffset = treq->ts_off;
 #ifdef CONFIG_TCP_MD5SIG
 		newtp->md5sig_info = NULL;	/*XXX*/
-		if (newtp->af_specific->md5_lookup(sk, newsk))
-			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
+		tcp_md5_add_header_len(sk, newsk);
 #endif
 		if (static_branch_unlikely(&tcp_extopt_enabled))
 			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
net/ipv4/tcp_output.c +2 -2
index b0297f19984f..e15006277275 100644
@@ -42,6 +42,7 @@
 #include <linux/gfp.h>
 #include <linux/module.h>
 #include <linux/static_key.h>
+#include <linux/tcp_md5.h>
 
 #include <trace/events/tcp.h>
 
@@ -3238,8 +3239,7 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_TSTAMP_ALIGNED;
 
 #ifdef CONFIG_TCP_MD5SIG
-	if (tp->af_specific->md5_lookup(sk, sk))
-		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
+	tcp_md5_add_header_len(sk, sk);
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled))
net/ipv6/tcp_ipv6.c +15 -303

391 lines not on this page · raw message

Signature · 1 line
2.15.0
#30 lore raw

Re: [MPTCP] [RFC v3 12/15] tcp_md5: Move TCP-MD5 code out of TCP itself

Quoted 676 lines · Christoph Paasch

This is all just copy-pasting the TCP_MD5-code into functions that are placed in net/ipv4/tcp_md5.c.

Signed-off-by Christoph Paasch <cpaasch(a)apple.com>

---

Notes:
v3: * Add missing EXPORT_SYMBOL_GPL

include/linux/inet_diag.h |    1 +
include/linux/tcp_md5.h   |  138 ++++++
include/net/tcp.h         |   77 ----
net/ipv4/Makefile         |    1 +
net/ipv4/tcp.c            |  133 +-----
net/ipv4/tcp_diag.c       |   81 +---
net/ipv4/tcp_input.c      |   38 --
net/ipv4/tcp_ipv4.c       |  520 ++-------------------
net/ipv4/tcp_md5.c        | 1103 +++++++++++++++++++++++++++++++++++++++++++++
net/ipv4/tcp_minisocks.c  |   25 +-
net/ipv4/tcp_output.c     |    4 +-
net/ipv6/tcp_ipv6.c       |  318 +------------
12 files changed, 1306 insertions(+), 1133 deletions(-)

create mode 100644 include/linux/tcp_md5.h
create mode 100644 net/ipv4/tcp_md5.c

include/linux/inet_diag.h +1 -0
index 39faaaf843e1..1ef6727e41c9 100644
@@ -2,6 +2,7 @@
 #ifndef _INET_DIAG_H_
 #define _INET_DIAG_H_ 1
 
+#include <linux/user_namespace.h>
 #include <uapi/linux/inet_diag.h>
 
 struct net;
include/linux/tcp_md5.h new file +138 -0
new file mode 100644
index 000000000000..f6a681cdded4
@@ -0,0 +1,138 @@
+#ifndef _LINUX_TCP_MD5_H
+#define _LINUX_TCP_MD5_H
+
+#include <linux/skbuff.h>
+
+#ifdef CONFIG_TCP_MD5SIG
+#include <linux/types.h>
+
+#include <net/tcp.h>
+
+union tcp_md5_addr {
+	struct in_addr  a4;
+#if IS_ENABLED(CONFIG_IPV6)
+	struct in6_addr	a6;
+#endif
+};
+
+/* - key database */
+struct tcp_md5sig_key {
+	struct hlist_node	node;
+	u8			keylen;
+	u8			family; /* AF_INET or AF_INET6 */
+	union tcp_md5_addr	addr;
+	u8			prefixlen;
+	u8			key[TCP_MD5SIG_MAXKEYLEN];
+	struct rcu_head		rcu;
+};
+
+/* - sock block */
+struct tcp_md5sig_info {
+	struct hlist_head	head;
+	struct rcu_head		rcu;
+};
+
+union tcp_md5sum_block {
+	struct tcp4_pseudohdr ip4;
+#if IS_ENABLED(CONFIG_IPV6)
+	struct tcp6_pseudohdr ip6;
+#endif
+};
+
+/* - pool: digest algorithm, hash description and scratch buffer */
+struct tcp_md5sig_pool {
+	struct ahash_request	*md5_req;
+	void			*scratch;
+};
+
+extern const struct tcp_sock_af_ops tcp_sock_ipv4_specific;
+extern const struct tcp_sock_af_ops tcp_sock_ipv6_specific;
+extern const struct tcp_sock_af_ops tcp_sock_ipv6_mapped_specific;
+
+/* - functions */
+int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
+			const struct sock *sk, const struct sk_buff *skb);
+
+struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
+					 const struct sock *addr_sk);
+
+void tcp_v4_md5_destroy_sock(struct sock *sk);
+
+int tcp_v4_md5_send_response_prepare(struct sk_buff *skb, u8 flags,
+				     unsigned int remaining,
+				     struct tcp_out_options *opts,
+				     const struct sock *sk);
+
+void tcp_v4_md5_send_response_write(__be32 *topt, struct sk_buff *skb,
+				    struct tcphdr *t1,
+				    struct tcp_out_options *opts,
+				    const struct sock *sk);
+
+int tcp_v6_md5_send_response_prepare(struct sk_buff *skb, u8 flags,
+				     unsigned int remaining,
+				     struct tcp_out_options *opts,
+				     const struct sock *sk);
+
+void tcp_v6_md5_send_response_write(__be32 *topt, struct sk_buff *skb,
+				    struct tcphdr *t1,
+				    struct tcp_out_options *opts,
+				    const struct sock *sk);
+
+bool tcp_v4_inbound_md5_hash(const struct sock *sk,
+			     const struct sk_buff *skb);
+
+void tcp_v4_md5_syn_recv_sock(const struct sock *listener, struct sock *sk);
+
+void tcp_v6_md5_syn_recv_sock(const struct sock *listener, struct sock *sk);
+
+void tcp_md5_time_wait(struct sock *sk, struct inet_timewait_sock *tw);
+
+struct tcp_md5sig_key *tcp_v6_md5_lookup(const struct sock *sk,
+					 const struct sock *addr_sk);
+
+int tcp_v6_md5_hash_skb(char *md5_hash,
+			const struct tcp_md5sig_key *key,
+			const struct sock *sk,
+			const struct sk_buff *skb);
+
+bool tcp_v6_inbound_md5_hash(const struct sock *sk,
+			     const struct sk_buff *skb);
+
+static inline void tcp_md5_twsk_destructor(struct sock *sk)
+{
+	struct tcp_timewait_sock *twsk = tcp_twsk(sk);
+
+	if (twsk->tw_md5_key)
+		kfree_rcu(twsk->tw_md5_key, rcu);
+}
+
+static inline void tcp_md5_add_header_len(const struct sock *listener,
+					  struct sock *sk)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+
+	if (tp->af_specific->md5_lookup(listener, sk))
+		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
+}
+
+int tcp_md5_diag_get_aux(struct sock *sk, bool net_admin, struct sk_buff *skb);
+
+int tcp_md5_diag_get_aux_size(struct sock *sk, bool net_admin);
+
+#else
+
+static inline bool tcp_v4_inbound_md5_hash(const struct sock *sk,
+					   const struct sk_buff *skb)
+{
+	return false;
+}
+
+static inline bool tcp_v6_inbound_md5_hash(const struct sock *sk,
+					   const struct sk_buff *skb)
+{
+	return false;
+}
+
+#endif
+
+#endif /* _LINUX_TCP_MD5_H */
include/net/tcp.h +0 -77
index 27e71cb5e4cb..d7a407ab11a3 100644
@@ -406,7 +406,6 @@ void tcp_parse_options(const struct net *net, const struct sk_buff *skb,
 		       struct tcp_options_received *opt_rx,
 		       int estab, struct tcp_fastopen_cookie *foc,
 		       struct sock *sk);
-const u8 *tcp_parse_md5sig_option(const struct tcphdr *th);
 
 /*
 *	TCP v4 functions exported for the inet6 API
@@ -1415,30 +1414,6 @@ static inline void tcp_clear_all_retrans_hints(struct tcp_sock *tp)
 	tp->retransmit_skb_hint = NULL;
 }
 
-union tcp_md5_addr {
-	struct in_addr  a4;
-#if IS_ENABLED(CONFIG_IPV6)
-	struct in6_addr	a6;
-#endif
-};
-
-/* - key database */
-struct tcp_md5sig_key {
-	struct hlist_node	node;
-	u8			keylen;
-	u8			family; /* AF_INET or AF_INET6 */
-	union tcp_md5_addr	addr;
-	u8			prefixlen;
-	u8			key[TCP_MD5SIG_MAXKEYLEN];
-	struct rcu_head		rcu;
-};
-
-/* - sock block */
-struct tcp_md5sig_info {
-	struct hlist_head	head;
-	struct rcu_head		rcu;
-};
-
 /* - pseudo header */
 struct tcp4_pseudohdr {
 	__be32		saddr;
@@ -1455,58 +1430,6 @@ struct tcp6_pseudohdr {
 	__be32		protocol;	/* including padding */
 };
 
-union tcp_md5sum_block {
-	struct tcp4_pseudohdr ip4;
-#if IS_ENABLED(CONFIG_IPV6)
-	struct tcp6_pseudohdr ip6;
-#endif
-};
-
-/* - pool: digest algorithm, hash description and scratch buffer */
-struct tcp_md5sig_pool {
-	struct ahash_request	*md5_req;
-	void			*scratch;
-};
-
-/* - functions */
-int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
-			const struct sock *sk, const struct sk_buff *skb);
-int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
-		   int family, u8 prefixlen, const u8 *newkey, u8 newkeylen,
-		   gfp_t gfp);
-int tcp_md5_do_del(struct sock *sk, const union tcp_md5_addr *addr,
-		   int family, u8 prefixlen);
-struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
-					 const struct sock *addr_sk);
-
-#ifdef CONFIG_TCP_MD5SIG
-struct tcp_md5sig_key *tcp_md5_do_lookup(const struct sock *sk,
-					 const union tcp_md5_addr *addr,
-					 int family);
-#define tcp_twsk_md5_key(twsk)	((twsk)->tw_md5_key)
-#else
-static inline struct tcp_md5sig_key *tcp_md5_do_lookup(const struct sock *sk,
-					 const union tcp_md5_addr *addr,
-					 int family)
-{
-	return NULL;
-}
-#define tcp_twsk_md5_key(twsk)	NULL
-#endif
-
-bool tcp_alloc_md5sig_pool(void);
-
-struct tcp_md5sig_pool *tcp_get_md5sig_pool(void);
-static inline void tcp_put_md5sig_pool(void)
-{
-	local_bh_enable();
-}
-
-int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *, const struct sk_buff *,
-			  unsigned int header_len);
-int tcp_md5_hash_key(struct tcp_md5sig_pool *hp,
-		     const struct tcp_md5sig_key *key);
-
 /* From tcp_fastopen.c */
 void tcp_fastopen_cache_get(struct sock *sk, u16 *mss,
 			    struct tcp_fastopen_cookie *cookie, int *syn_loss,
net/ipv4/Makefile +0 -0
index c6c8ad1d4b6d..9262d9a01035 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_TCP_CONG_LP) += tcp_lp.o

obj-$(CONFIG_TCP_CONG_YEAH) += tcp_yeah.o
obj-$(CONFIG_TCP_CONG_ILLINOIS) += tcp_illinois.o
obj-$(CONFIG_NETLABEL) += cipso_ipv4.o
+obj-$(CONFIG_TCP_MD5SIG) += tcp_md5.o

obj-$(CONFIG_XFRM) += xfrm4_policy.o xfrm4_state.o xfrm4_input.o \
xfrm4_output.o xfrm4_protocol.o

net/ipv4/tcp.c +1 -132
index c5d9484064c9..a1a7f8e92677 100644
@@ -271,6 +271,7 @@
 #include <linux/slab.h>
 #include <linux/errqueue.h>
 #include <linux/static_key.h>
+#include <linux/tcp_md5.h>
 
 #include <net/icmp.h>
 #include <net/inet_common.h>
@@ -3341,138 +3342,6 @@ int compat_tcp_getsockopt(struct sock *sk, int level, int optname,
 EXPORT_SYMBOL(compat_tcp_getsockopt);
 #endif
 
-#ifdef CONFIG_TCP_MD5SIG
-static DEFINE_PER_CPU(struct tcp_md5sig_pool, tcp_md5sig_pool);
-static DEFINE_MUTEX(tcp_md5sig_mutex);
-static bool tcp_md5sig_pool_populated = false;
-
-static void __tcp_alloc_md5sig_pool(void)
-{
-	struct crypto_ahash *hash;
-	int cpu;
-
-	hash = crypto_alloc_ahash("md5", 0, CRYPTO_ALG_ASYNC);
-	if (IS_ERR(hash))
-		return;
-
-	for_each_possible_cpu(cpu) {
-		void *scratch = per_cpu(tcp_md5sig_pool, cpu).scratch;
-		struct ahash_request *req;
-
-		if (!scratch) {
-			scratch = kmalloc_node(sizeof(union tcp_md5sum_block) +
-					       sizeof(struct tcphdr),
-					       GFP_KERNEL,
-					       cpu_to_node(cpu));
-			if (!scratch)
-				return;
-			per_cpu(tcp_md5sig_pool, cpu).scratch = scratch;
-		}
-		if (per_cpu(tcp_md5sig_pool, cpu).md5_req)
-			continue;
-
-		req = ahash_request_alloc(hash, GFP_KERNEL);
-		if (!req)
-			return;
-
-		ahash_request_set_callback(req, 0, NULL, NULL);
-
-		per_cpu(tcp_md5sig_pool, cpu).md5_req = req;
-	}
-	/* before setting tcp_md5sig_pool_populated, we must commit all writes
-	 * to memory. See smp_rmb() in tcp_get_md5sig_pool()
-	 */
-	smp_wmb();
-	tcp_md5sig_pool_populated = true;
-}
-
-bool tcp_alloc_md5sig_pool(void)
-{
-	if (unlikely(!tcp_md5sig_pool_populated)) {
-		mutex_lock(&tcp_md5sig_mutex);
-
-		if (!tcp_md5sig_pool_populated)
-			__tcp_alloc_md5sig_pool();
-
-		mutex_unlock(&tcp_md5sig_mutex);
-	}
-	return tcp_md5sig_pool_populated;
-}
-EXPORT_SYMBOL(tcp_alloc_md5sig_pool);
-
-
-/**
- *	tcp_get_md5sig_pool - get md5sig_pool for this user
- *
- *	We use percpu structure, so if we succeed, we exit with preemption
- *	and BH disabled, to make sure another thread or softirq handling
- *	wont try to get same context.
- */
-struct tcp_md5sig_pool *tcp_get_md5sig_pool(void)
-{
-	local_bh_disable();
-
-	if (tcp_md5sig_pool_populated) {
-		/* coupled with smp_wmb() in __tcp_alloc_md5sig_pool() */
-		smp_rmb();
-		return this_cpu_ptr(&tcp_md5sig_pool);
-	}
-	local_bh_enable();
-	return NULL;
-}
-EXPORT_SYMBOL(tcp_get_md5sig_pool);
-
-int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp,
-			  const struct sk_buff *skb, unsigned int header_len)
-{
-	struct scatterlist sg;
-	const struct tcphdr *tp = tcp_hdr(skb);
-	struct ahash_request *req = hp->md5_req;
-	unsigned int i;
-	const unsigned int head_data_len = skb_headlen(skb) > header_len ?
-					   skb_headlen(skb) - header_len : 0;
-	const struct skb_shared_info *shi = skb_shinfo(skb);
-	struct sk_buff *frag_iter;
-
-	sg_init_table(&sg, 1);
-
-	sg_set_buf(&sg, ((u8 *) tp) + header_len, head_data_len);
-	ahash_request_set_crypt(req, &sg, NULL, head_data_len);
-	if (crypto_ahash_update(req))
-		return 1;
-
-	for (i = 0; i < shi->nr_frags; ++i) {
-		const struct skb_frag_struct *f = &shi->frags[i];
-		unsigned int offset = f->page_offset;
-		struct page *page = skb_frag_page(f) + (offset >> PAGE_SHIFT);
-
-		sg_set_page(&sg, page, skb_frag_size(f),
-			    offset_in_page(offset));
-		ahash_request_set_crypt(req, &sg, NULL, skb_frag_size(f));
-		if (crypto_ahash_update(req))
-			return 1;
-	}
-
-	skb_walk_frags(skb, frag_iter)
-		if (tcp_md5_hash_skb_data(hp, frag_iter, 0))
-			return 1;
-
-	return 0;
-}
-EXPORT_SYMBOL(tcp_md5_hash_skb_data);
-
-int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, const struct tcp_md5sig_key *key)
-{
-	struct scatterlist sg;
-
-	sg_init_one(&sg, key->key, key->keylen);
-	ahash_request_set_crypt(hp->md5_req, &sg, NULL, key->keylen);
-	return crypto_ahash_update(hp->md5_req);
-}
-EXPORT_SYMBOL(tcp_md5_hash_key);
-
-#endif
-
 static struct hlist_head *tcp_extopt_get_list(const struct sock *sk)
 {
 	if (sk_fullsock(sk))
net/ipv4/tcp_diag.c +6 -75
index abbf0edcf6c2..5cfe5dc8f8dd 100644
@@ -15,6 +15,7 @@
 #include <linux/inet_diag.h>
 
 #include <linux/tcp.h>
+#include <linux/tcp_md5.h>
 
 #include <net/netlink.h>
 #include <net/tcp.h>
@@ -37,70 +38,14 @@ static void tcp_diag_get_info(struct sock *sk, struct inet_diag_msg *r,
 		tcp_get_info(sk, info);
 }
 
-#ifdef CONFIG_TCP_MD5SIG
-static void tcp_diag_md5sig_fill(struct tcp_diag_md5sig *info,
-				 const struct tcp_md5sig_key *key)
-{
-	info->tcpm_family = key->family;
-	info->tcpm_prefixlen = key->prefixlen;
-	info->tcpm_keylen = key->keylen;
-	memcpy(info->tcpm_key, key->key, key->keylen);
-
-	if (key->family == AF_INET)
-		info->tcpm_addr[0] = key->addr.a4.s_addr;
-	#if IS_ENABLED(CONFIG_IPV6)
-	else if (key->family == AF_INET6)
-		memcpy(&info->tcpm_addr, &key->addr.a6,
-		       sizeof(info->tcpm_addr));
-	#endif
-}
-
-static int tcp_diag_put_md5sig(struct sk_buff *skb,
-			       const struct tcp_md5sig_info *md5sig)
-{
-	const struct tcp_md5sig_key *key;
-	struct tcp_diag_md5sig *info;
-	struct nlattr *attr;
-	int md5sig_count = 0;
-
-	hlist_for_each_entry_rcu(key, &md5sig->head, node)
-		md5sig_count++;
-	if (md5sig_count == 0)
-		return 0;
-
-	attr = nla_reserve(skb, INET_DIAG_MD5SIG,
-			   md5sig_count * sizeof(struct tcp_diag_md5sig));
-	if (!attr)
-		return -EMSGSIZE;
-
-	info = nla_data(attr);
-	memset(info, 0, md5sig_count * sizeof(struct tcp_diag_md5sig));
-	hlist_for_each_entry_rcu(key, &md5sig->head, node) {
-		tcp_diag_md5sig_fill(info++, key);
-		if (--md5sig_count == 0)
-			break;
-	}
-
-	return 0;
-}
-#endif
-
 static int tcp_diag_get_aux(struct sock *sk, bool net_admin,
 			    struct sk_buff *skb)
 {
 #ifdef CONFIG_TCP_MD5SIG
-	if (net_admin) {
-		struct tcp_md5sig_info *md5sig;
-		int err = 0;
-
-		rcu_read_lock();
-		md5sig = rcu_dereference(tcp_sk(sk)->md5sig_info);
-		if (md5sig)
-			err = tcp_diag_put_md5sig(skb, md5sig);
-		rcu_read_unlock();
-		if (err < 0)
-			return err;
-	}
+	int err = tcp_md5_diag_get_aux(sk, net_admin, skb);
+
+	if (err < 0)
+		return err;
 #endif
 
 	return 0;
@@ -111,21 +56,7 @@ static size_t tcp_diag_get_aux_size(struct sock *sk, bool net_admin)
 	size_t size = 0;
 
 #ifdef CONFIG_TCP_MD5SIG
-	if (net_admin && sk_fullsock(sk)) {
-		const struct tcp_md5sig_info *md5sig;
-		const struct tcp_md5sig_key *key;
-		size_t md5sig_count = 0;
-
-		rcu_read_lock();
-		md5sig = rcu_dereference(tcp_sk(sk)->md5sig_info);
-		if (md5sig) {
-			hlist_for_each_entry_rcu(key, &md5sig->head, node)
-				md5sig_count++;
-		}
-		rcu_read_unlock();
-		size += nla_total_size(md5sig_count *
-				       sizeof(struct tcp_diag_md5sig));
-	}
+	size += tcp_md5_diag_get_aux_size(sk, net_admin);
 #endif
 
 	return size;
net/ipv4/tcp_input.c +0 -38
index 5f4a2f2e3b87..e89e22920c2b 100644
@@ -3854,44 +3854,6 @@ static bool tcp_fast_parse_options(const struct net *net,
 	return false;
 }
 
-#ifdef CONFIG_TCP_MD5SIG
-/*
- * Parse MD5 Signature option
- */
-const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
-{
-	int length = (th->doff << 2) - sizeof(*th);
-	const u8 *ptr = (const u8 *)(th + 1);
-
-	/* If the TCP option is too short, we can short cut */
-	if (length < TCPOLEN_MD5SIG)
-		return NULL;
-
-	while (length > 0) {
-		int opcode = *ptr++;
-		int opsize;
-
-		switch (opcode) {
-		case TCPOPT_EOL:
-			return NULL;
-		case TCPOPT_NOP:
-			length--;
-			continue;
-		default:
-			opsize = *ptr++;
-			if (opsize < 2 || opsize > length)
-				return NULL;
-			if (opcode == TCPOPT_MD5SIG)
-				return opsize == TCPOLEN_MD5SIG ? ptr : NULL;
-		}
-		ptr += opsize - 2;
-		length -= opsize;
-	}
-	return NULL;
-}
-EXPORT_SYMBOL(tcp_parse_md5sig_option);
-#endif
-
 /* Sorry, PAWS as specified is broken wrt. pure-ACKs -DaveM
 *
 * It is not fatal. If this ACK does _not_ change critical state (seqs, window)
net/ipv4/tcp_ipv4.c +4 -30
index 74667de1e049..c00d1e8828e8 100644
@@ -62,6 +62,7 @@
 #include <linux/init.h>
 #include <linux/times.h>
 #include <linux/slab.h>
+#include <linux/tcp_md5.h>
 
 #include <net/net_namespace.h>
 #include <net/icmp.h>
@@ -87,11 +88,6 @@
 
 #include <trace/events/tcp.h>
 
-#ifdef CONFIG_TCP_MD5SIG
-static int tcp_v4_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
-			       __be32 daddr, __be32 saddr, const struct tcphdr *th);
-#endif
-
 struct inet_hashinfo tcp_hashinfo;
 EXPORT_SYMBOL(tcp_hashinfo);
 
@@ -602,14 +598,8 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 		struct tcphdr th;
 		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
+	struct tcp_out_options opts;
 	struct ip_reply_arg arg;
-#ifdef CONFIG_TCP_MD5SIG
-	struct tcp_md5sig_key *key = NULL;
-	const __u8 *hash_location = NULL;
-	unsigned char newhash[16];
-	int genhash;
-	struct sock *sk1 = NULL;
-#endif
 	struct net *net;
 	int offset = 0;
 
@@ -623,6 +613,8 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	if (!sk && skb_rtable(skb)->rt_type != RTN_LOCAL)
 		return;
 
+	memset(&opts, 0, sizeof(opts));
+
 	/* Swap the send and the receive. */
 	memset(&rep, 0, sizeof(rep));
 	rep.th.dest   = th->source;
@@ -643,55 +635,32 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	arg.iov[0].iov_len  = sizeof(rep.th);
 
 	net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev);
-#ifdef CONFIG_TCP_MD5SIG
-	rcu_read_lock();
-	hash_location = tcp_parse_md5sig_option(th);
-	if (sk && sk_fullsock(sk)) {
-		key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)
-					&ip_hdr(skb)->saddr, AF_INET);
-	} else if (hash_location) {
-		/*
-		 * active side is lost. Try to find listening socket through
-		 * source port, and then find md5 key through listening socket.
-		 * we are not loose security here:
-		 * Incoming packet is checked with md5 hash with finding key,
-		 * no RST generated if md5 hash doesn't match.
-		 */
-		sk1 = __inet_lookup_listener(net, &tcp_hashinfo, NULL, 0,
-					     ip_hdr(skb)->saddr,
-					     th->source, ip_hdr(skb)->daddr,
-					     ntohs(th->source), inet_iif(skb),

net/ipv4/tcp_ipv4.c static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb) · line 639

Quoted 10 lines
net/ipv4/tcp_ipv4.c +1 -8
-					     tcp_v4_sdif(skb));
-		/* don't send rst if it can't find key */
-		if (!sk1)
-			goto out;
 
-		key = tcp_md5_do_lookup(sk1, (union tcp_md5_addr *)
-					&ip_hdr(skb)->saddr, AF_INET);
-		if (!key)
-			goto out;
+#ifdef CONFIG_TCP_MD5DSIG

Typo - extra 'D' in CONFIG_TCP_MD5SIG

net/ipv4/tcp_ipv4.c static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb) · line 641

Quoted 2 lines
net/ipv4/tcp_ipv4.c +2 -0
+{
+	int ret;

"if (IS_ENABLED(CONFIG_TCP_MD5SIG))" (C 'if', not preprocessor '#if') is another way introduce a smaller scope for 'ret' - the bare {} block is not typical kernel style. That said, it would be most stylistically consistent to declare ret at the beginning of the function.

I don't know that it's a huge deal, given this is an intermediate refactoring step.

Quoted 88 lines
net/ipv4/tcp_ipv4.c +11 -29
+	ret = tcp_v4_md5_send_response_prepare(skb, 0,
+					       MAX_TCP_OPTION_SPACE - arg.iov[0].iov_len,
+					       &opts, sk);
 
-		genhash = tcp_v4_md5_hash_skb(newhash, key, NULL, skb);
-		if (genhash || memcmp(hash_location, newhash, 16) != 0)
-			goto out;
+	if (ret == -1)
+		return;
 
-	}
+	arg.iov[0].iov_len += ret;
+}
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled)) {
 		unsigned int remaining;
-		struct tcp_out_options opts;
 		int used;
 
 		remaining = sizeof(rep.opt);
 #ifdef CONFIG_TCP_MD5SIG
-		if (key)
+		if (opts.md5)
 			remaining -= TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
-		memset(&opts, 0, sizeof(opts));
-
 		used = tcp_extopt_response_prepare(skb, TCPHDR_RST, remaining,
 						   &opts, sk);
 
@@ -703,19 +672,7 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	}
 
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
-					  (TCPOPT_NOP << 16) |
-					  (TCPOPT_MD5SIG << 8) |
-					  TCPOLEN_MD5SIG);
-		/* Update length and the length the header thinks exists */
-		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
-		rep.th.doff = arg.iov[0].iov_len / 4;
-
-		tcp_v4_md5_hash_hdr((__u8 *)&rep.opt[offset],
-				    key, ip_hdr(skb)->saddr,
-				    ip_hdr(skb)->daddr, &rep.th);
-	}
+	tcp_v4_md5_send_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
 #endif
 	arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
 				      ip_hdr(skb)->saddr, /* XXX */
@@ -746,11 +703,6 @@ static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb)
 	__TCP_INC_STATS(net, TCP_MIB_OUTSEGS);
 	__TCP_INC_STATS(net, TCP_MIB_OUTRSTS);
 	local_bh_enable();
-
-#ifdef CONFIG_TCP_MD5SIG
-out:
-	rcu_read_unlock();
-#endif
 }
 
 /* The code following below sending ACKs in SYN-RECV and TIME-WAIT states
@@ -767,15 +719,14 @@ static void tcp_v4_send_ack(const struct sock *sk,
 		struct tcphdr th;
 		__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
 	} rep;
-#ifdef CONFIG_TCP_MD5SIG
-	struct tcp_md5sig_key *key;
-#endif
+	struct tcp_out_options opts;
 	struct net *net = sock_net(sk);
 	struct ip_reply_arg arg;
 	int offset = 0;
 
 	memset(&rep.th, 0, sizeof(struct tcphdr));
 	memset(&arg, 0, sizeof(arg));
+	memset(&opts, 0, sizeof(opts));
 
 	arg.iov[0].iov_base = (unsigned char *)&rep;
 	arg.iov[0].iov_len  = sizeof(rep.th);
@@ -799,25 +750,28 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	rep.th.window  = htons(win);
 
 #ifdef CONFIG_TCP_MD5SIG

net/ipv4/tcp_ipv4.c static void tcp_v4_send_ack(const struct sock *sk, · line 754

Quoted 10 lines
net/ipv4/tcp_ipv4.c +2 -8
-	if (sk->sk_state == TCP_TIME_WAIT) {
-		key = tcp_twsk_md5_key(tcp_twsk(sk));
-	} else if (sk->sk_state == TCP_NEW_SYN_RECV) {
-		key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&ip_hdr(skb)->saddr,
-					AF_INET);
-	} else {
-		key = NULL;	/* Should not happen */
-	}
+{
+	int ret;

Same as above - bare {} block is unusual.

Mat

Quoted 2,080 lines
net/ipv4/tcp_ipv4.c +15 -410
+
+	ret = tcp_v4_md5_send_response_prepare(skb, 0,
+					       MAX_TCP_OPTION_SPACE - arg.iov[0].iov_len,
+					       &opts, sk);
+
+	if (ret == -1)
+		return;
+
+	arg.iov[0].iov_len += ret;
+}
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled)) {
 		unsigned int remaining;
-		struct tcp_out_options opts;
 		int used;
 
 		remaining = sizeof(rep.th) + sizeof(rep.opt) - arg.iov[0].iov_len;
 
 #ifdef CONFIG_TCP_MD5SIG
-		if (key)
+		if (opts.md5)
 			remaining -= TCPOLEN_MD5SIG_ALIGNED;
 #endif
 
@@ -834,18 +788,11 @@ static void tcp_v4_send_ack(const struct sock *sk,
 	}
 
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
-					  (TCPOPT_NOP << 16) |
-					  (TCPOPT_MD5SIG << 8) |
-					  TCPOLEN_MD5SIG);
+	if (opts.md5) {
 		arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
 		rep.th.doff = arg.iov[0].iov_len / 4;
-
-		tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
-				    key, ip_hdr(skb)->saddr,
-				    ip_hdr(skb)->daddr, &rep.th);
 	}
+	tcp_v4_md5_send_response_write(&rep.opt[offset], skb, &rep.th, &opts, sk);
 #endif
 
 	arg.flags = reply_flags;
@@ -954,374 +901,6 @@ static void tcp_v4_reqsk_destructor(struct request_sock *req)
 	kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
 }
 
-#ifdef CONFIG_TCP_MD5SIG
-/*
- * RFC2385 MD5 checksumming requires a mapping of
- * IP address->MD5 Key.
- * We need to maintain these in the sk structure.
- */
-
-/* Find the Key structure for an address.  */
-struct tcp_md5sig_key *tcp_md5_do_lookup(const struct sock *sk,
-					 const union tcp_md5_addr *addr,
-					 int family)
-{
-	const struct tcp_sock *tp = tcp_sk(sk);
-	struct tcp_md5sig_key *key;
-	const struct tcp_md5sig_info *md5sig;
-	__be32 mask;
-	struct tcp_md5sig_key *best_match = NULL;
-	bool match;
-
-	/* caller either holds rcu_read_lock() or socket lock */
-	md5sig = rcu_dereference_check(tp->md5sig_info,
-				       lockdep_sock_is_held(sk));
-	if (!md5sig)
-		return NULL;
-
-	hlist_for_each_entry_rcu(key, &md5sig->head, node) {
-		if (key->family != family)
-			continue;
-
-		if (family == AF_INET) {
-			mask = inet_make_mask(key->prefixlen);
-			match = (key->addr.a4.s_addr & mask) ==
-				(addr->a4.s_addr & mask);
-#if IS_ENABLED(CONFIG_IPV6)
-		} else if (family == AF_INET6) {
-			match = ipv6_prefix_equal(&key->addr.a6, &addr->a6,
-						  key->prefixlen);
-#endif
-		} else {
-			match = false;
-		}
-
-		if (match && (!best_match ||
-			      key->prefixlen > best_match->prefixlen))
-			best_match = key;
-	}
-	return best_match;
-}
-EXPORT_SYMBOL(tcp_md5_do_lookup);
-
-static struct tcp_md5sig_key *tcp_md5_do_lookup_exact(const struct sock *sk,
-						      const union tcp_md5_addr *addr,
-						      int family, u8 prefixlen)
-{
-	const struct tcp_sock *tp = tcp_sk(sk);
-	struct tcp_md5sig_key *key;
-	unsigned int size = sizeof(struct in_addr);
-	const struct tcp_md5sig_info *md5sig;
-
-	/* caller either holds rcu_read_lock() or socket lock */
-	md5sig = rcu_dereference_check(tp->md5sig_info,
-				       lockdep_sock_is_held(sk));
-	if (!md5sig)
-		return NULL;
-#if IS_ENABLED(CONFIG_IPV6)
-	if (family == AF_INET6)
-		size = sizeof(struct in6_addr);
-#endif
-	hlist_for_each_entry_rcu(key, &md5sig->head, node) {
-		if (key->family != family)
-			continue;
-		if (!memcmp(&key->addr, addr, size) &&
-		    key->prefixlen == prefixlen)
-			return key;
-	}
-	return NULL;
-}
-
-struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
-					 const struct sock *addr_sk)
-{
-	const union tcp_md5_addr *addr;
-
-	addr = (const union tcp_md5_addr *)&addr_sk->sk_daddr;
-	return tcp_md5_do_lookup(sk, addr, AF_INET);
-}
-EXPORT_SYMBOL(tcp_v4_md5_lookup);
-
-/* This can be called on a newly created socket, from other files */
-int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
-		   int family, u8 prefixlen, const u8 *newkey, u8 newkeylen,
-		   gfp_t gfp)
-{
-	/* Add Key to the list */
-	struct tcp_md5sig_key *key;
-	struct tcp_sock *tp = tcp_sk(sk);
-	struct tcp_md5sig_info *md5sig;
-
-	key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen);
-	if (key) {
-		/* Pre-existing entry - just update that one. */
-		memcpy(key->key, newkey, newkeylen);
-		key->keylen = newkeylen;
-		return 0;
-	}
-
-	md5sig = rcu_dereference_protected(tp->md5sig_info,
-					   lockdep_sock_is_held(sk));
-	if (!md5sig) {
-		md5sig = kmalloc(sizeof(*md5sig), gfp);
-		if (!md5sig)
-			return -ENOMEM;
-
-		sk_nocaps_add(sk, NETIF_F_GSO_MASK);
-		INIT_HLIST_HEAD(&md5sig->head);
-		rcu_assign_pointer(tp->md5sig_info, md5sig);
-	}
-
-	key = sock_kmalloc(sk, sizeof(*key), gfp);
-	if (!key)
-		return -ENOMEM;
-	if (!tcp_alloc_md5sig_pool()) {
-		sock_kfree_s(sk, key, sizeof(*key));
-		return -ENOMEM;
-	}
-
-	memcpy(key->key, newkey, newkeylen);
-	key->keylen = newkeylen;
-	key->family = family;
-	key->prefixlen = prefixlen;
-	memcpy(&key->addr, addr,
-	       (family == AF_INET6) ? sizeof(struct in6_addr) :
-				      sizeof(struct in_addr));
-	hlist_add_head_rcu(&key->node, &md5sig->head);
-	return 0;
-}
-EXPORT_SYMBOL(tcp_md5_do_add);
-
-int tcp_md5_do_del(struct sock *sk, const union tcp_md5_addr *addr, int family,
-		   u8 prefixlen)
-{
-	struct tcp_md5sig_key *key;
-
-	key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen);
-	if (!key)
-		return -ENOENT;
-	hlist_del_rcu(&key->node);
-	atomic_sub(sizeof(*key), &sk->sk_omem_alloc);
-	kfree_rcu(key, rcu);
-	return 0;
-}
-EXPORT_SYMBOL(tcp_md5_do_del);
-
-static void tcp_clear_md5_list(struct sock *sk)
-{
-	struct tcp_sock *tp = tcp_sk(sk);
-	struct tcp_md5sig_key *key;
-	struct hlist_node *n;
-	struct tcp_md5sig_info *md5sig;
-
-	md5sig = rcu_dereference_protected(tp->md5sig_info, 1);
-
-	hlist_for_each_entry_safe(key, n, &md5sig->head, node) {
-		hlist_del_rcu(&key->node);
-		atomic_sub(sizeof(*key), &sk->sk_omem_alloc);
-		kfree_rcu(key, rcu);
-	}
-}
-
-static int tcp_v4_parse_md5_keys(struct sock *sk, int optname,
-				 char __user *optval, int optlen)
-{
-	struct tcp_md5sig cmd;
-	struct sockaddr_in *sin = (struct sockaddr_in *)&cmd.tcpm_addr;
-	u8 prefixlen = 32;
-
-	if (optlen < sizeof(cmd))
-		return -EINVAL;
-
-	if (copy_from_user(&cmd, optval, sizeof(cmd)))
-		return -EFAULT;
-
-	if (sin->sin_family != AF_INET)
-		return -EINVAL;
-
-	if (optname == TCP_MD5SIG_EXT &&
-	    cmd.tcpm_flags & TCP_MD5SIG_FLAG_PREFIX) {
-		prefixlen = cmd.tcpm_prefixlen;
-		if (prefixlen > 32)
-			return -EINVAL;
-	}
-
-	if (!cmd.tcpm_keylen)
-		return tcp_md5_do_del(sk, (union tcp_md5_addr *)&sin->sin_addr.s_addr,
-				      AF_INET, prefixlen);
-
-	if (cmd.tcpm_keylen > TCP_MD5SIG_MAXKEYLEN)
-		return -EINVAL;
-
-	return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin->sin_addr.s_addr,
-			      AF_INET, prefixlen, cmd.tcpm_key, cmd.tcpm_keylen,
-			      GFP_KERNEL);
-}
-
-static int tcp_v4_md5_hash_headers(struct tcp_md5sig_pool *hp,
-				   __be32 daddr, __be32 saddr,
-				   const struct tcphdr *th, int nbytes)
-{
-	struct tcp4_pseudohdr *bp;
-	struct scatterlist sg;
-	struct tcphdr *_th;
-
-	bp = hp->scratch;
-	bp->saddr = saddr;
-	bp->daddr = daddr;
-	bp->pad = 0;
-	bp->protocol = IPPROTO_TCP;
-	bp->len = cpu_to_be16(nbytes);
-
-	_th = (struct tcphdr *)(bp + 1);
-	memcpy(_th, th, sizeof(*th));
-	_th->check = 0;
-
-	sg_init_one(&sg, bp, sizeof(*bp) + sizeof(*th));
-	ahash_request_set_crypt(hp->md5_req, &sg, NULL,
-				sizeof(*bp) + sizeof(*th));
-	return crypto_ahash_update(hp->md5_req);
-}
-
-static int tcp_v4_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
-			       __be32 daddr, __be32 saddr, const struct tcphdr *th)
-{
-	struct tcp_md5sig_pool *hp;
-	struct ahash_request *req;
-
-	hp = tcp_get_md5sig_pool();
-	if (!hp)
-		goto clear_hash_noput;
-	req = hp->md5_req;
-
-	if (crypto_ahash_init(req))
-		goto clear_hash;
-	if (tcp_v4_md5_hash_headers(hp, daddr, saddr, th, th->doff << 2))
-		goto clear_hash;
-	if (tcp_md5_hash_key(hp, key))
-		goto clear_hash;
-	ahash_request_set_crypt(req, NULL, md5_hash, 0);
-	if (crypto_ahash_final(req))
-		goto clear_hash;
-
-	tcp_put_md5sig_pool();
-	return 0;
-
-clear_hash:
-	tcp_put_md5sig_pool();
-clear_hash_noput:
-	memset(md5_hash, 0, 16);
-	return 1;
-}
-
-int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
-			const struct sock *sk,
-			const struct sk_buff *skb)
-{
-	struct tcp_md5sig_pool *hp;
-	struct ahash_request *req;
-	const struct tcphdr *th = tcp_hdr(skb);
-	__be32 saddr, daddr;
-
-	if (sk) { /* valid for establish/request sockets */
-		saddr = sk->sk_rcv_saddr;
-		daddr = sk->sk_daddr;
-	} else {
-		const struct iphdr *iph = ip_hdr(skb);
-		saddr = iph->saddr;
-		daddr = iph->daddr;
-	}
-
-	hp = tcp_get_md5sig_pool();
-	if (!hp)
-		goto clear_hash_noput;
-	req = hp->md5_req;
-
-	if (crypto_ahash_init(req))
-		goto clear_hash;
-
-	if (tcp_v4_md5_hash_headers(hp, daddr, saddr, th, skb->len))
-		goto clear_hash;
-	if (tcp_md5_hash_skb_data(hp, skb, th->doff << 2))
-		goto clear_hash;
-	if (tcp_md5_hash_key(hp, key))
-		goto clear_hash;
-	ahash_request_set_crypt(req, NULL, md5_hash, 0);
-	if (crypto_ahash_final(req))
-		goto clear_hash;
-
-	tcp_put_md5sig_pool();
-	return 0;
-
-clear_hash:
-	tcp_put_md5sig_pool();
-clear_hash_noput:
-	memset(md5_hash, 0, 16);
-	return 1;
-}
-EXPORT_SYMBOL(tcp_v4_md5_hash_skb);
-
-#endif
-
-/* Called with rcu_read_lock() */
-static bool tcp_v4_inbound_md5_hash(const struct sock *sk,
-				    const struct sk_buff *skb)
-{
-#ifdef CONFIG_TCP_MD5SIG
-	/*
-	 * This gets called for each TCP segment that arrives
-	 * so we want to be efficient.
-	 * We have 3 drop cases:
-	 * o No MD5 hash and one expected.
-	 * o MD5 hash and we're not expecting one.
-	 * o MD5 hash and its wrong.
-	 */
-	const __u8 *hash_location = NULL;
-	struct tcp_md5sig_key *hash_expected;
-	const struct iphdr *iph = ip_hdr(skb);
-	const struct tcphdr *th = tcp_hdr(skb);
-	int genhash;
-	unsigned char newhash[16];
-
-	hash_expected = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&iph->saddr,
-					  AF_INET);
-	hash_location = tcp_parse_md5sig_option(th);
-
-	/* We've parsed the options - do we have a hash? */
-	if (!hash_expected && !hash_location)
-		return false;
-
-	if (hash_expected && !hash_location) {
-		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5NOTFOUND);
-		return true;
-	}
-
-	if (!hash_expected && hash_location) {
-		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5UNEXPECTED);
-		return true;
-	}
-
-	/* Okay, so this is hash_expected and hash_location -
-	 * so we need to calculate the checksum.
-	 */
-	genhash = tcp_v4_md5_hash_skb(newhash,
-				      hash_expected,
-				      NULL, skb);
-
-	if (genhash || memcmp(hash_location, newhash, 16) != 0) {
-		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5FAILURE);
-		net_info_ratelimited("MD5 Hash failed for (%pI4, %d)->(%pI4, %d)%s\n",
-				     &iph->saddr, ntohs(th->source),
-				     &iph->daddr, ntohs(th->dest),
-				     genhash ? " tcp_v4_calc_md5_hash failed"
-				     : "");
-		return true;
-	}
-	return false;
-#endif
-	return false;
-}
-
 static void tcp_v4_init_req(struct request_sock *req,
 			    const struct sock *sk_listener,
 			    struct sk_buff *skb)
@@ -1397,9 +976,6 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 	struct inet_sock *newinet;
 	struct tcp_sock *newtp;
 	struct sock *newsk;
-#ifdef CONFIG_TCP_MD5SIG
-	struct tcp_md5sig_key *key;
-#endif
 	struct ip_options_rcu *inet_opt;
 
 	if (sk_acceptq_is_full(sk))
@@ -1450,20 +1026,7 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 		INIT_HLIST_HEAD(&tcp_rsk(req)->tcp_option_list);
 	}
 #ifdef CONFIG_TCP_MD5SIG
-	/* Copy over the MD5 key from the original socket */
-	key = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&newinet->inet_daddr,
-				AF_INET);
-	if (key) {
-		/*
-		 * We're using one, so create a matching key
-		 * on the newsk structure. If we fail to get
-		 * memory, then we end up not copying the key
-		 * across. Shucks.
-		 */
-		tcp_md5_do_add(newsk, (union tcp_md5_addr *)&newinet->inet_daddr,
-			       AF_INET, 32, key->key, key->keylen, GFP_ATOMIC);
-		sk_nocaps_add(newsk, NETIF_F_GSO_MASK);
-	}
+	tcp_v4_md5_syn_recv_sock(sk, newsk);
 #endif
 
 	if (__inet_inherit_port(sk, newsk) < 0)
@@ -1922,14 +1485,6 @@ const struct inet_connection_sock_af_ops ipv4_specific = {
 };
 EXPORT_SYMBOL(ipv4_specific);
 
-#ifdef CONFIG_TCP_MD5SIG
-static const struct tcp_sock_af_ops tcp_sock_ipv4_specific = {
-	.md5_lookup		= tcp_v4_md5_lookup,
-	.calc_md5_hash		= tcp_v4_md5_hash_skb,
-	.md5_parse		= tcp_v4_parse_md5_keys,
-};
-#endif
-
 /* NOTE: A lot of things set to zero explicitly by call to
 *       sk_alloc() so need not be done here.
 */
@@ -1972,12 +1527,7 @@ void tcp_v4_destroy_sock(struct sock *sk)
 	if (static_branch_unlikely(&tcp_extopt_enabled))
 		tcp_extopt_destroy(sk);
 #ifdef CONFIG_TCP_MD5SIG
-	/* Clean up the MD5 key list, if any */
-	if (tp->md5sig_info) {
-		tcp_clear_md5_list(sk);
-		kfree_rcu(tp->md5sig_info, rcu);
-		tp->md5sig_info = NULL;
-	}
+	tcp_v4_md5_destroy_sock(sk);
 #endif
 
 	/* Clean up a referenced TCP bind bucket. */
net/ipv4/tcp_md5.c new file +1103 -0
new file mode 100644
index 000000000000..ab2059a015d3
@@ -0,0 +1,1103 @@
+#include <linux/inet_diag.h>
+#include <linux/inetdevice.h>
+#include <linux/tcp.h>
+#include <linux/tcp_md5.h>
+
+#include <crypto/hash.h>
+
+#include <net/inet6_hashtables.h>
+
+static DEFINE_PER_CPU(struct tcp_md5sig_pool, tcp_md5sig_pool);
+static DEFINE_MUTEX(tcp_md5sig_mutex);
+static bool tcp_md5sig_pool_populated;
+
+#define tcp_twsk_md5_key(twsk)	((twsk)->tw_md5_key)
+
+static void __tcp_alloc_md5sig_pool(void)
+{
+	struct crypto_ahash *hash;
+	int cpu;
+
+	hash = crypto_alloc_ahash("md5", 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(hash))
+		return;
+
+	for_each_possible_cpu(cpu) {
+		void *scratch = per_cpu(tcp_md5sig_pool, cpu).scratch;
+		struct ahash_request *req;
+
+		if (!scratch) {
+			scratch = kmalloc_node(sizeof(union tcp_md5sum_block) +
+					       sizeof(struct tcphdr),
+					       GFP_KERNEL,
+					       cpu_to_node(cpu));
+			if (!scratch)
+				return;
+			per_cpu(tcp_md5sig_pool, cpu).scratch = scratch;
+		}
+		if (per_cpu(tcp_md5sig_pool, cpu).md5_req)
+			continue;
+
+		req = ahash_request_alloc(hash, GFP_KERNEL);
+		if (!req)
+			return;
+
+		ahash_request_set_callback(req, 0, NULL, NULL);
+
+		per_cpu(tcp_md5sig_pool, cpu).md5_req = req;
+	}
+	/* before setting tcp_md5sig_pool_populated, we must commit all writes
+	 * to memory. See smp_rmb() in tcp_get_md5sig_pool()
+	 */
+	smp_wmb();
+	tcp_md5sig_pool_populated = true;
+}
+
+static bool tcp_alloc_md5sig_pool(void)
+{
+	if (unlikely(!tcp_md5sig_pool_populated)) {
+		mutex_lock(&tcp_md5sig_mutex);
+
+		if (!tcp_md5sig_pool_populated)
+			__tcp_alloc_md5sig_pool();
+
+		mutex_unlock(&tcp_md5sig_mutex);
+	}
+	return tcp_md5sig_pool_populated;
+}
+
+static void tcp_put_md5sig_pool(void)
+{
+	local_bh_enable();
+}
+
+/**
+ *	tcp_get_md5sig_pool - get md5sig_pool for this user
+ *
+ *	We use percpu structure, so if we succeed, we exit with preemption
+ *	and BH disabled, to make sure another thread or softirq handling
+ *	wont try to get same context.
+ */
+static struct tcp_md5sig_pool *tcp_get_md5sig_pool(void)
+{
+	local_bh_disable();
+
+	if (tcp_md5sig_pool_populated) {
+		/* coupled with smp_wmb() in __tcp_alloc_md5sig_pool() */
+		smp_rmb();
+		return this_cpu_ptr(&tcp_md5sig_pool);
+	}
+	local_bh_enable();
+	return NULL;
+}
+
+static struct tcp_md5sig_key *tcp_md5_do_lookup_exact(const struct sock *sk,
+						      const union tcp_md5_addr *addr,
+						      int family, u8 prefixlen)
+{
+	const struct tcp_sock *tp = tcp_sk(sk);
+	struct tcp_md5sig_key *key;
+	unsigned int size = sizeof(struct in_addr);
+	const struct tcp_md5sig_info *md5sig;
+
+	/* caller either holds rcu_read_lock() or socket lock */
+	md5sig = rcu_dereference_check(tp->md5sig_info,
+				       lockdep_sock_is_held(sk));
+	if (!md5sig)
+		return NULL;
+#if IS_ENABLED(CONFIG_IPV6)
+	if (family == AF_INET6)
+		size = sizeof(struct in6_addr);
+#endif
+	hlist_for_each_entry_rcu(key, &md5sig->head, node) {
+		if (key->family != family)
+			continue;
+		if (!memcmp(&key->addr, addr, size) &&
+		    key->prefixlen == prefixlen)
+			return key;
+	}
+	return NULL;
+}
+
+/* This can be called on a newly created socket, from other files */
+static int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
+			  int family, u8 prefixlen, const u8 *newkey,
+			  u8 newkeylen, gfp_t gfp)
+{
+	/* Add Key to the list */
+	struct tcp_md5sig_key *key;
+	struct tcp_sock *tp = tcp_sk(sk);
+	struct tcp_md5sig_info *md5sig;
+
+	key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen);
+	if (key) {
+		/* Pre-existing entry - just update that one. */
+		memcpy(key->key, newkey, newkeylen);
+		key->keylen = newkeylen;
+		return 0;
+	}
+
+	md5sig = rcu_dereference_protected(tp->md5sig_info,
+					   lockdep_sock_is_held(sk));
+	if (!md5sig) {
+		md5sig = kmalloc(sizeof(*md5sig), gfp);
+		if (!md5sig)
+			return -ENOMEM;
+
+		sk_nocaps_add(sk, NETIF_F_GSO_MASK);
+		INIT_HLIST_HEAD(&md5sig->head);
+		rcu_assign_pointer(tp->md5sig_info, md5sig);
+	}
+
+	key = sock_kmalloc(sk, sizeof(*key), gfp);
+	if (!key)
+		return -ENOMEM;
+	if (!tcp_alloc_md5sig_pool()) {
+		sock_kfree_s(sk, key, sizeof(*key));
+		return -ENOMEM;
+	}
+
+	memcpy(key->key, newkey, newkeylen);
+	key->keylen = newkeylen;
+	key->family = family;
+	key->prefixlen = prefixlen;
+	memcpy(&key->addr, addr,
+	       (family == AF_INET6) ? sizeof(struct in6_addr) :
+				      sizeof(struct in_addr));
+	hlist_add_head_rcu(&key->node, &md5sig->head);
+	return 0;
+}
+
+static void tcp_clear_md5_list(struct sock *sk)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+	struct tcp_md5sig_key *key;
+	struct hlist_node *n;
+	struct tcp_md5sig_info *md5sig;
+
+	md5sig = rcu_dereference_protected(tp->md5sig_info, 1);
+
+	hlist_for_each_entry_safe(key, n, &md5sig->head, node) {
+		hlist_del_rcu(&key->node);
+		atomic_sub(sizeof(*key), &sk->sk_omem_alloc);
+		kfree_rcu(key, rcu);
+	}
+}
+
+static int tcp_md5_do_del(struct sock *sk, const union tcp_md5_addr *addr,
+			  int family, u8 prefixlen)
+{
+	struct tcp_md5sig_key *key;
+
+	key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen);
+	if (!key)
+		return -ENOENT;
+	hlist_del_rcu(&key->node);
+	atomic_sub(sizeof(*key), &sk->sk_omem_alloc);
+	kfree_rcu(key, rcu);
+	return 0;
+}
+
+static int tcp_md5_hash_key(struct tcp_md5sig_pool *hp,
+			    const struct tcp_md5sig_key *key)
+{
+	struct scatterlist sg;
+
+	sg_init_one(&sg, key->key, key->keylen);
+	ahash_request_set_crypt(hp->md5_req, &sg, NULL, key->keylen);
+	return crypto_ahash_update(hp->md5_req);
+}
+
+static int tcp_v4_parse_md5_keys(struct sock *sk, int optname,
+				 char __user *optval, int optlen)
+{
+	struct tcp_md5sig cmd;
+	struct sockaddr_in *sin = (struct sockaddr_in *)&cmd.tcpm_addr;
+	u8 prefixlen = 32;
+
+	if (optlen < sizeof(cmd))
+		return -EINVAL;
+
+	if (copy_from_user(&cmd, optval, sizeof(cmd)))
+		return -EFAULT;
+
+	if (sin->sin_family != AF_INET)
+		return -EINVAL;
+
+	if (optname == TCP_MD5SIG_EXT &&
+	    cmd.tcpm_flags & TCP_MD5SIG_FLAG_PREFIX) {
+		prefixlen = cmd.tcpm_prefixlen;
+		if (prefixlen > 32)
+			return -EINVAL;
+	}
+
+	if (!cmd.tcpm_keylen)
+		return tcp_md5_do_del(sk, (union tcp_md5_addr *)&sin->sin_addr.s_addr,
+				      AF_INET, prefixlen);
+
+	if (cmd.tcpm_keylen > TCP_MD5SIG_MAXKEYLEN)
+		return -EINVAL;
+
+	return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin->sin_addr.s_addr,
+			      AF_INET, prefixlen, cmd.tcpm_key, cmd.tcpm_keylen,
+			      GFP_KERNEL);
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static int tcp_v6_parse_md5_keys(struct sock *sk, int optname,
+				 char __user *optval, int optlen)
+{
+	struct tcp_md5sig cmd;
+	struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)&cmd.tcpm_addr;
+	u8 prefixlen;
+
+	if (optlen < sizeof(cmd))
+		return -EINVAL;
+
+	if (copy_from_user(&cmd, optval, sizeof(cmd)))
+		return -EFAULT;
+
+	if (sin6->sin6_family != AF_INET6)
+		return -EINVAL;
+
+	if (optname == TCP_MD5SIG_EXT &&
+	    cmd.tcpm_flags & TCP_MD5SIG_FLAG_PREFIX) {
+		prefixlen = cmd.tcpm_prefixlen;
+		if (prefixlen > 128 || (ipv6_addr_v4mapped(&sin6->sin6_addr) &&
+					prefixlen > 32))
+			return -EINVAL;
+	} else {
+		prefixlen = ipv6_addr_v4mapped(&sin6->sin6_addr) ? 32 : 128;
+	}
+
+	if (!cmd.tcpm_keylen) {
+		if (ipv6_addr_v4mapped(&sin6->sin6_addr))
+			return tcp_md5_do_del(sk, (union tcp_md5_addr *)&sin6->sin6_addr.s6_addr32[3],
+					      AF_INET, prefixlen);
+		return tcp_md5_do_del(sk, (union tcp_md5_addr *)&sin6->sin6_addr,
+				      AF_INET6, prefixlen);
+	}
+
+	if (cmd.tcpm_keylen > TCP_MD5SIG_MAXKEYLEN)
+		return -EINVAL;
+
+	if (ipv6_addr_v4mapped(&sin6->sin6_addr))
+		return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin6->sin6_addr.s6_addr32[3],
+				      AF_INET, prefixlen, cmd.tcpm_key,
+				      cmd.tcpm_keylen, GFP_KERNEL);
+
+	return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin6->sin6_addr,
+			      AF_INET6, prefixlen, cmd.tcpm_key,
+			      cmd.tcpm_keylen, GFP_KERNEL);
+}
+#endif
+
+static int tcp_v4_md5_hash_headers(struct tcp_md5sig_pool *hp,
+				   __be32 daddr, __be32 saddr,
+				   const struct tcphdr *th, int nbytes)
+{
+	struct tcp4_pseudohdr *bp;
+	struct scatterlist sg;
+	struct tcphdr *_th;
+
+	bp = hp->scratch;
+	bp->saddr = saddr;
+	bp->daddr = daddr;
+	bp->pad = 0;
+	bp->protocol = IPPROTO_TCP;
+	bp->len = cpu_to_be16(nbytes);
+
+	_th = (struct tcphdr *)(bp + 1);
+	memcpy(_th, th, sizeof(*th));
+	_th->check = 0;
+
+	sg_init_one(&sg, bp, sizeof(*bp) + sizeof(*th));
+	ahash_request_set_crypt(hp->md5_req, &sg, NULL,
+				sizeof(*bp) + sizeof(*th));
+	return crypto_ahash_update(hp->md5_req);
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static int tcp_v6_md5_hash_headers(struct tcp_md5sig_pool *hp,
+				   const struct in6_addr *daddr,
+				   const struct in6_addr *saddr,
+				   const struct tcphdr *th, int nbytes)
+{
+	struct tcp6_pseudohdr *bp;
+	struct scatterlist sg;
+	struct tcphdr *_th;
+
+	bp = hp->scratch;
+	/* 1. TCP pseudo-header (RFC2460) */
+	bp->saddr = *saddr;
+	bp->daddr = *daddr;
+	bp->protocol = cpu_to_be32(IPPROTO_TCP);
+	bp->len = cpu_to_be32(nbytes);
+
+	_th = (struct tcphdr *)(bp + 1);
+	memcpy(_th, th, sizeof(*th));
+	_th->check = 0;
+
+	sg_init_one(&sg, bp, sizeof(*bp) + sizeof(*th));
+	ahash_request_set_crypt(hp->md5_req, &sg, NULL,
+				sizeof(*bp) + sizeof(*th));
+	return crypto_ahash_update(hp->md5_req);
+}
+#endif
+
+static int tcp_v4_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
+			       __be32 daddr, __be32 saddr,
+			       const struct tcphdr *th)
+{
+	struct tcp_md5sig_pool *hp;
+	struct ahash_request *req;
+
+	hp = tcp_get_md5sig_pool();
+	if (!hp)
+		goto clear_hash_noput;
+	req = hp->md5_req;
+
+	if (crypto_ahash_init(req))
+		goto clear_hash;
+	if (tcp_v4_md5_hash_headers(hp, daddr, saddr, th, th->doff << 2))
+		goto clear_hash;
+	if (tcp_md5_hash_key(hp, key))
+		goto clear_hash;
+	ahash_request_set_crypt(req, NULL, md5_hash, 0);
+	if (crypto_ahash_final(req))
+		goto clear_hash;
+
+	tcp_put_md5sig_pool();
+	return 0;
+
+clear_hash:
+	tcp_put_md5sig_pool();
+clear_hash_noput:
+	memset(md5_hash, 0, 16);
+	return 1;
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static int tcp_v6_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
+			       const struct in6_addr *daddr,
+			       struct in6_addr *saddr, const struct tcphdr *th)
+{
+	struct tcp_md5sig_pool *hp;
+	struct ahash_request *req;
+
+	hp = tcp_get_md5sig_pool();
+	if (!hp)
+		goto clear_hash_noput;
+	req = hp->md5_req;
+
+	if (crypto_ahash_init(req))
+		goto clear_hash;
+	if (tcp_v6_md5_hash_headers(hp, daddr, saddr, th, th->doff << 2))
+		goto clear_hash;
+	if (tcp_md5_hash_key(hp, key))
+		goto clear_hash;
+	ahash_request_set_crypt(req, NULL, md5_hash, 0);
+	if (crypto_ahash_final(req))
+		goto clear_hash;
+
+	tcp_put_md5sig_pool();
+	return 0;
+
+clear_hash:
+	tcp_put_md5sig_pool();
+clear_hash_noput:
+	memset(md5_hash, 0, 16);
+	return 1;
+}
+#endif
+
+/* RFC2385 MD5 checksumming requires a mapping of
+ * IP address->MD5 Key.
+ * We need to maintain these in the sk structure.
+ */
+
+/* Find the Key structure for an address.  */
+static struct tcp_md5sig_key *tcp_md5_do_lookup(const struct sock *sk,
+						const union tcp_md5_addr *addr,
+						int family)
+{
+	const struct tcp_sock *tp = tcp_sk(sk);
+	struct tcp_md5sig_key *key;
+	const struct tcp_md5sig_info *md5sig;
+	__be32 mask;
+	struct tcp_md5sig_key *best_match = NULL;
+	bool match;
+
+	/* caller either holds rcu_read_lock() or socket lock */
+	md5sig = rcu_dereference_check(tp->md5sig_info,
+				       lockdep_sock_is_held(sk));
+	if (!md5sig)
+		return NULL;
+
+	hlist_for_each_entry_rcu(key, &md5sig->head, node) {
+		if (key->family != family)
+			continue;
+
+		if (family == AF_INET) {
+			mask = inet_make_mask(key->prefixlen);
+			match = (key->addr.a4.s_addr & mask) ==
+				(addr->a4.s_addr & mask);
+#if IS_ENABLED(CONFIG_IPV6)
+		} else if (family == AF_INET6) {
+			match = ipv6_prefix_equal(&key->addr.a6, &addr->a6,
+						  key->prefixlen);
+#endif
+		} else {
+			match = false;
+		}
+
+		if (match && (!best_match ||
+			      key->prefixlen > best_match->prefixlen))
+			best_match = key;
+	}
+	return best_match;
+}
+
+/* Parse MD5 Signature option */
+static const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
+{
+	int length = (th->doff << 2) - sizeof(*th);
+	const u8 *ptr = (const u8 *)(th + 1);
+
+	/* If the TCP option is too short, we can short cut */
+	if (length < TCPOLEN_MD5SIG)
+		return NULL;
+
+	while (length > 0) {
+		int opcode = *ptr++;
+		int opsize;
+
+		switch (opcode) {
+		case TCPOPT_EOL:
+			return NULL;
+		case TCPOPT_NOP:
+			length--;
+			continue;
+		default:
+			opsize = *ptr++;
+			if (opsize < 2 || opsize > length)
+				return NULL;
+			if (opcode == TCPOPT_MD5SIG)
+				return opsize == TCPOLEN_MD5SIG ? ptr : NULL;
+		}
+		ptr += opsize - 2;
+		length -= opsize;
+	}
+	return NULL;
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static struct tcp_md5sig_key *tcp_v6_md5_do_lookup(const struct sock *sk,
+						   const struct in6_addr *addr)
+{
+	return tcp_md5_do_lookup(sk, (union tcp_md5_addr *)addr, AF_INET6);
+}
+#endif
+
+static int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp,
+				 const struct sk_buff *skb,
+				 unsigned int header_len)
+{
+	struct scatterlist sg;
+	const struct tcphdr *tp = tcp_hdr(skb);
+	struct ahash_request *req = hp->md5_req;
+	unsigned int i;
+	const unsigned int head_data_len = skb_headlen(skb) > header_len ?
+					   skb_headlen(skb) - header_len : 0;
+	const struct skb_shared_info *shi = skb_shinfo(skb);
+	struct sk_buff *frag_iter;
+
+	sg_init_table(&sg, 1);
+
+	sg_set_buf(&sg, ((u8 *)tp) + header_len, head_data_len);
+	ahash_request_set_crypt(req, &sg, NULL, head_data_len);
+	if (crypto_ahash_update(req))
+		return 1;
+
+	for (i = 0; i < shi->nr_frags; ++i) {
+		const struct skb_frag_struct *f = &shi->frags[i];
+		unsigned int offset = f->page_offset;
+		struct page *page = skb_frag_page(f) + (offset >> PAGE_SHIFT);
+
+		sg_set_page(&sg, page, skb_frag_size(f),
+			    offset_in_page(offset));
+		ahash_request_set_crypt(req, &sg, NULL, skb_frag_size(f));
+		if (crypto_ahash_update(req))
+			return 1;
+	}
+
+	skb_walk_frags(skb, frag_iter)
+		if (tcp_md5_hash_skb_data(hp, frag_iter, 0))
+			return 1;
+
+	return 0;
+}
+
+int tcp_v4_md5_send_response_prepare(struct sk_buff *skb, u8 flags,
+				     unsigned int remaining,
+				     struct tcp_out_options *opts,
+				     const struct sock *sk)
+{
+	const struct tcphdr *th = tcp_hdr(skb);
+	const struct iphdr *iph = ip_hdr(skb);
+	const __u8 *hash_location = NULL;
+
+	rcu_read_lock();
+	hash_location = tcp_parse_md5sig_option(th);
+	if (sk && sk_fullsock(sk)) {
+		opts->md5 = tcp_md5_do_lookup(sk,
+					      (union tcp_md5_addr *)&iph->saddr,
+					      AF_INET);
+	} else if (sk && sk->sk_state == TCP_TIME_WAIT) {
+		struct tcp_timewait_sock *tcptw = tcp_twsk(sk);
+
+		opts->md5 = tcp_twsk_md5_key(tcptw);
+	} else if (sk && sk->sk_state == TCP_NEW_SYN_RECV) {
+		opts->md5 = tcp_md5_do_lookup(sk,
+					      (union tcp_md5_addr *)&iph->saddr,
+					      AF_INET);
+	} else if (hash_location) {
+		unsigned char newhash[16];
+		struct sock *sk1;
+		int genhash;
+
+		/* active side is lost. Try to find listening socket through
+		 * source port, and then find md5 key through listening socket.
+		 * we are not loose security here:
+		 * Incoming packet is checked with md5 hash with finding key,
+		 * no RST generated if md5 hash doesn't match.
+		 */
+		sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev),
+					     &tcp_hashinfo, NULL, 0,
+					     iph->saddr,
+					     th->source, iph->daddr,
+					     ntohs(th->source), inet_iif(skb),
+					     tcp_v4_sdif(skb));
+		/* don't send rst if it can't find key */
+		if (!sk1)
+			goto out_err;
+
+		opts->md5 = tcp_md5_do_lookup(sk1, (union tcp_md5_addr *)
+					      &iph->saddr, AF_INET);
+		if (!opts->md5)
+			goto out_err;
+
+		genhash = tcp_v4_md5_hash_skb(newhash, opts->md5, NULL, skb);
+		if (genhash || memcmp(hash_location, newhash, 16) != 0)
+			goto out_err;
+	}
+
+	if (opts->md5)
+		return TCPOLEN_MD5SIG_ALIGNED;
+
+	rcu_read_unlock();
+	return 0;
+
+out_err:
+	rcu_read_unlock();
+	return -1;
+}
+
+void tcp_v4_md5_send_response_write(__be32 *topt, struct sk_buff *skb,
+				    struct tcphdr *t1,
+				    struct tcp_out_options *opts,
+				    const struct sock *sk)
+{
+	if (opts->md5) {
+		*topt++ = htonl((TCPOPT_NOP << 24) |
+				(TCPOPT_NOP << 16) |
+				(TCPOPT_MD5SIG << 8) |
+				TCPOLEN_MD5SIG);
+
+		tcp_v4_md5_hash_hdr((__u8 *)topt, opts->md5,
+				    ip_hdr(skb)->saddr,
+				    ip_hdr(skb)->daddr, t1);
+		rcu_read_unlock();
+	}
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+int tcp_v6_md5_send_response_prepare(struct sk_buff *skb, u8 flags,
+				     unsigned int remaining,
+				     struct tcp_out_options *opts,
+				     const struct sock *sk)
+{
+	const struct tcphdr *th = tcp_hdr(skb);
+	struct ipv6hdr *ipv6h = ipv6_hdr(skb);
+	const __u8 *hash_location = NULL;
+
+	rcu_read_lock();
+	hash_location = tcp_parse_md5sig_option(th);
+	if (sk && sk_fullsock(sk)) {
+		opts->md5 = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
+	} else if (sk && sk->sk_state == TCP_TIME_WAIT) {
+		struct tcp_timewait_sock *tcptw = tcp_twsk(sk);
+
+		opts->md5 = tcp_twsk_md5_key(tcptw);
+	} else if (sk && sk->sk_state == TCP_NEW_SYN_RECV) {
+		opts->md5 = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
+	} else if (hash_location) {
+		unsigned char newhash[16];
+		struct sock *sk1;
+		int genhash;
+
+		/* active side is lost. Try to find listening socket through
+		 * source port, and then find md5 key through listening socket.
+		 * we are not loose security here:
+		 * Incoming packet is checked with md5 hash with finding key,
+		 * no RST generated if md5 hash doesn't match.
+		 */
+		sk1 = inet6_lookup_listener(dev_net(skb_dst(skb)->dev),
+					    &tcp_hashinfo, NULL, 0,
+					    &ipv6h->saddr,
+					    th->source, &ipv6h->daddr,
+					    ntohs(th->source), tcp_v6_iif(skb),
+					    tcp_v6_sdif(skb));
+		if (!sk1)
+			goto out_err;
+
+		opts->md5 = tcp_v6_md5_do_lookup(sk1, &ipv6h->saddr);
+		if (!opts->md5)
+			goto out_err;
+
+		genhash = tcp_v6_md5_hash_skb(newhash, opts->md5, NULL, skb);
+		if (genhash || memcmp(hash_location, newhash, 16) != 0)
+			goto out_err;
+	}
+
+	if (opts->md5)
+		return TCPOLEN_MD5SIG_ALIGNED;
+
+	rcu_read_unlock();
+	return 0;
+
+out_err:
+	rcu_read_unlock();
+	return -1;
+}
+EXPORT_SYMBOL_GPL(tcp_v6_md5_send_response_prepare);
+
+void tcp_v6_md5_send_response_write(__be32 *topt, struct sk_buff *skb,
+				    struct tcphdr *t1,
+				    struct tcp_out_options *opts,
+				    const struct sock *sk)
+{
+	if (opts->md5) {
+		*topt++ = htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16) |
+				(TCPOPT_MD5SIG << 8) | TCPOLEN_MD5SIG);
+		tcp_v6_md5_hash_hdr((__u8 *)topt, opts->md5,
+				    &ipv6_hdr(skb)->saddr,
+				    &ipv6_hdr(skb)->daddr, t1);
+
+		rcu_read_unlock();
+	}
+}
+EXPORT_SYMBOL_GPL(tcp_v6_md5_send_response_write);
+#endif
+
+struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
+					 const struct sock *addr_sk)
+{
+	const union tcp_md5_addr *addr;
+
+	addr = (const union tcp_md5_addr *)&addr_sk->sk_daddr;
+	return tcp_md5_do_lookup(sk, addr, AF_INET);
+}
+EXPORT_SYMBOL(tcp_v4_md5_lookup);
+
+int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
+			const struct sock *sk,
+			const struct sk_buff *skb)
+{
+	struct tcp_md5sig_pool *hp;
+	struct ahash_request *req;
+	const struct tcphdr *th = tcp_hdr(skb);
+	__be32 saddr, daddr;
+
+	if (sk) { /* valid for establish/request sockets */
+		saddr = sk->sk_rcv_saddr;
+		daddr = sk->sk_daddr;
+	} else {
+		const struct iphdr *iph = ip_hdr(skb);
+
+		saddr = iph->saddr;
+		daddr = iph->daddr;
+	}
+
+	hp = tcp_get_md5sig_pool();
+	if (!hp)
+		goto clear_hash_noput;
+	req = hp->md5_req;
+
+	if (crypto_ahash_init(req))
+		goto clear_hash;
+
+	if (tcp_v4_md5_hash_headers(hp, daddr, saddr, th, skb->len))
+		goto clear_hash;
+	if (tcp_md5_hash_skb_data(hp, skb, th->doff << 2))
+		goto clear_hash;
+	if (tcp_md5_hash_key(hp, key))
+		goto clear_hash;
+	ahash_request_set_crypt(req, NULL, md5_hash, 0);
+	if (crypto_ahash_final(req))
+		goto clear_hash;
+
+	tcp_put_md5sig_pool();
+	return 0;
+
+clear_hash:
+	tcp_put_md5sig_pool();
+clear_hash_noput:
+	memset(md5_hash, 0, 16);
+	return 1;
+}
+EXPORT_SYMBOL(tcp_v4_md5_hash_skb);
+
+#if IS_ENABLED(CONFIG_IPV6)
+int tcp_v6_md5_hash_skb(char *md5_hash,
+			const struct tcp_md5sig_key *key,
+			const struct sock *sk,
+			const struct sk_buff *skb)
+{
+	const struct in6_addr *saddr, *daddr;
+	struct tcp_md5sig_pool *hp;
+	struct ahash_request *req;
+	const struct tcphdr *th = tcp_hdr(skb);
+
+	if (sk) { /* valid for establish/request sockets */
+		saddr = &sk->sk_v6_rcv_saddr;
+		daddr = &sk->sk_v6_daddr;
+	} else {
+		const struct ipv6hdr *ip6h = ipv6_hdr(skb);
+
+		saddr = &ip6h->saddr;
+		daddr = &ip6h->daddr;
+	}
+
+	hp = tcp_get_md5sig_pool();
+	if (!hp)
+		goto clear_hash_noput;
+	req = hp->md5_req;
+
+	if (crypto_ahash_init(req))
+		goto clear_hash;
+
+	if (tcp_v6_md5_hash_headers(hp, daddr, saddr, th, skb->len))
+		goto clear_hash;
+	if (tcp_md5_hash_skb_data(hp, skb, th->doff << 2))
+		goto clear_hash;
+	if (tcp_md5_hash_key(hp, key))
+		goto clear_hash;
+	ahash_request_set_crypt(req, NULL, md5_hash, 0);
+	if (crypto_ahash_final(req))
+		goto clear_hash;
+
+	tcp_put_md5sig_pool();
+	return 0;
+
+clear_hash:
+	tcp_put_md5sig_pool();
+clear_hash_noput:
+	memset(md5_hash, 0, 16);
+	return 1;
+}
+EXPORT_SYMBOL_GPL(tcp_v6_md5_hash_skb);
+#endif
+
+/* Called with rcu_read_lock() */
+bool tcp_v4_inbound_md5_hash(const struct sock *sk,
+			     const struct sk_buff *skb)
+{
+	/* This gets called for each TCP segment that arrives
+	 * so we want to be efficient.
+	 * We have 3 drop cases:
+	 * o No MD5 hash and one expected.
+	 * o MD5 hash and we're not expecting one.
+	 * o MD5 hash and its wrong.
+	 */
+	const __u8 *hash_location = NULL;
+	struct tcp_md5sig_key *hash_expected;
+	const struct iphdr *iph = ip_hdr(skb);
+	const struct tcphdr *th = tcp_hdr(skb);
+	int genhash;
+	unsigned char newhash[16];
+
+	hash_expected = tcp_md5_do_lookup(sk, (union tcp_md5_addr *)&iph->saddr,
+					  AF_INET);
+	hash_location = tcp_parse_md5sig_option(th);
+
+	/* We've parsed the options - do we have a hash? */
+	if (!hash_expected && !hash_location)
+		return false;
+
+	if (hash_expected && !hash_location) {
+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5NOTFOUND);
+		return true;
+	}
+
+	if (!hash_expected && hash_location) {
+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5UNEXPECTED);
+		return true;
+	}
+
+	/* Okay, so this is hash_expected and hash_location -
+	 * so we need to calculate the checksum.
+	 */
+	genhash = tcp_v4_md5_hash_skb(newhash,
+				      hash_expected,
+				      NULL, skb);
+
+	if (genhash || memcmp(hash_location, newhash, 16) != 0) {
+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5FAILURE);
+		net_info_ratelimited("MD5 Hash failed for (%pI4, %d)->(%pI4, %d)%s\n",
+				     &iph->saddr, ntohs(th->source),
+				     &iph->daddr, ntohs(th->dest),
+				     genhash ? " tcp_v4_calc_md5_hash failed"
+				     : "");
+		return true;
+	}
+	return false;
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+bool tcp_v6_inbound_md5_hash(const struct sock *sk,
+			     const struct sk_buff *skb)
+{
+	const __u8 *hash_location = NULL;
+	struct tcp_md5sig_key *hash_expected;
+	const struct ipv6hdr *ip6h = ipv6_hdr(skb);
+	const struct tcphdr *th = tcp_hdr(skb);
+	int genhash;
+	u8 newhash[16];
+
+	hash_expected = tcp_v6_md5_do_lookup(sk, &ip6h->saddr);
+	hash_location = tcp_parse_md5sig_option(th);
+
+	/* We've parsed the options - do we have a hash? */
+	if (!hash_expected && !hash_location)
+		return false;
+
+	if (hash_expected && !hash_location) {
+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5NOTFOUND);
+		return true;
+	}
+
+	if (!hash_expected && hash_location) {
+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5UNEXPECTED);
+		return true;
+	}
+
+	/* check the signature */
+	genhash = tcp_v6_md5_hash_skb(newhash,
+				      hash_expected,
+				      NULL, skb);
+
+	if (genhash || memcmp(hash_location, newhash, 16) != 0) {
+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5FAILURE);
+		net_info_ratelimited("MD5 Hash %s for [%pI6c]:%u->[%pI6c]:%u\n",
+				     genhash ? "failed" : "mismatch",
+				     &ip6h->saddr, ntohs(th->source),
+				     &ip6h->daddr, ntohs(th->dest));
+		return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(tcp_v6_inbound_md5_hash);
+#endif
+
+void tcp_v4_md5_destroy_sock(struct sock *sk)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+
+	/* Clean up the MD5 key list, if any */
+	if (tp->md5sig_info) {
+		tcp_clear_md5_list(sk);
+		kfree_rcu(tp->md5sig_info, rcu);
+		tp->md5sig_info = NULL;
+	}
+}
+
+void tcp_v4_md5_syn_recv_sock(const struct sock *listener, struct sock *sk)
+{
+	struct inet_sock *inet = inet_sk(sk);
+	struct tcp_md5sig_key *key;
+
+	/* Copy over the MD5 key from the original socket */
+	key = tcp_md5_do_lookup(listener, (union tcp_md5_addr *)&inet->inet_daddr,
+				AF_INET);
+	if (key) {
+		/* We're using one, so create a matching key
+		 * on the sk structure. If we fail to get
+		 * memory, then we end up not copying the key
+		 * across. Shucks.
+		 */
+		tcp_md5_do_add(sk, (union tcp_md5_addr *)&inet->inet_daddr,
+			       AF_INET, 32, key->key, key->keylen, GFP_ATOMIC);
+		sk_nocaps_add(sk, NETIF_F_GSO_MASK);
+	}
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+void tcp_v6_md5_syn_recv_sock(const struct sock *listener, struct sock *sk)
+{
+	struct tcp_md5sig_key *key;
+
+	/* Copy over the MD5 key from the original socket */
+	key = tcp_v6_md5_do_lookup(listener, &sk->sk_v6_daddr);
+	if (key) {
+		/* We're using one, so create a matching key
+		 * on the newsk structure. If we fail to get
+		 * memory, then we end up not copying the key
+		 * across. Shucks.
+		 */
+		tcp_md5_do_add(sk, (union tcp_md5_addr *)&sk->sk_v6_daddr,
+			       AF_INET6, 128, key->key, key->keylen,
+			       sk_gfp_mask(sk, GFP_ATOMIC));
+	}
+}
+EXPORT_SYMBOL_GPL(tcp_v6_md5_syn_recv_sock);
+
+struct tcp_md5sig_key *tcp_v6_md5_lookup(const struct sock *sk,
+					 const struct sock *addr_sk)
+{
+	return tcp_v6_md5_do_lookup(sk, &addr_sk->sk_v6_daddr);
+}
+EXPORT_SYMBOL_GPL(tcp_v6_md5_lookup);
+#endif
+
+void tcp_md5_time_wait(struct sock *sk, struct inet_timewait_sock *tw)
+{
+	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
+	struct tcp_sock *tp = tcp_sk(sk);
+	struct tcp_md5sig_key *key;
+
+	/* The timewait bucket does not have the key DB from the
+	 * sock structure. We just make a quick copy of the
+	 * md5 key being used (if indeed we are using one)
+	 * so the timewait ack generating code has the key.
+	 */
+	tcptw->tw_md5_key = NULL;
+	key = tp->af_specific->md5_lookup(sk, sk);
+	if (key) {
+		tcptw->tw_md5_key = kmemdup(key, sizeof(*key), GFP_ATOMIC);
+		BUG_ON(tcptw->tw_md5_key && !tcp_alloc_md5sig_pool());
+	}
+}
+
+static void tcp_diag_md5sig_fill(struct tcp_diag_md5sig *info,
+				 const struct tcp_md5sig_key *key)
+{
+	info->tcpm_family = key->family;
+	info->tcpm_prefixlen = key->prefixlen;
+	info->tcpm_keylen = key->keylen;
+	memcpy(info->tcpm_key, key->key, key->keylen);
+
+	if (key->family == AF_INET)
+		info->tcpm_addr[0] = key->addr.a4.s_addr;
+	#if IS_ENABLED(CONFIG_IPV6)
+	else if (key->family == AF_INET6)
+		memcpy(&info->tcpm_addr, &key->addr.a6,
+		       sizeof(info->tcpm_addr));
+	#endif
+}
+
+static int tcp_diag_put_md5sig(struct sk_buff *skb,
+			       const struct tcp_md5sig_info *md5sig)
+{
+	const struct tcp_md5sig_key *key;
+	struct tcp_diag_md5sig *info;
+	struct nlattr *attr;
+	int md5sig_count = 0;
+
+	hlist_for_each_entry_rcu(key, &md5sig->head, node)
+		md5sig_count++;
+	if (md5sig_count == 0)
+		return 0;
+
+	attr = nla_reserve(skb, INET_DIAG_MD5SIG,
+			   md5sig_count * sizeof(struct tcp_diag_md5sig));
+	if (!attr)
+		return -EMSGSIZE;
+
+	info = nla_data(attr);
+	memset(info, 0, md5sig_count * sizeof(struct tcp_diag_md5sig));
+	hlist_for_each_entry_rcu(key, &md5sig->head, node) {
+		tcp_diag_md5sig_fill(info++, key);
+		if (--md5sig_count == 0)
+			break;
+	}
+
+	return 0;
+}
+
+int tcp_md5_diag_get_aux(struct sock *sk, bool net_admin, struct sk_buff *skb)
+{
+	if (net_admin) {
+		struct tcp_md5sig_info *md5sig;
+		int err = 0;
+
+		rcu_read_lock();
+		md5sig = rcu_dereference(tcp_sk(sk)->md5sig_info);
+		if (md5sig)
+			err = tcp_diag_put_md5sig(skb, md5sig);
+		rcu_read_unlock();
+		if (err < 0)
+			return err;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(tcp_md5_diag_get_aux);
+
+int tcp_md5_diag_get_aux_size(struct sock *sk, bool net_admin)
+{
+	int size = 0;
+
+	if (net_admin && sk_fullsock(sk)) {
+		const struct tcp_md5sig_info *md5sig;
+		const struct tcp_md5sig_key *key;
+		size_t md5sig_count = 0;
+
+		rcu_read_lock();
+		md5sig = rcu_dereference(tcp_sk(sk)->md5sig_info);
+		if (md5sig) {
+			hlist_for_each_entry_rcu(key, &md5sig->head, node)
+				md5sig_count++;
+		}
+		rcu_read_unlock();
+		size += nla_total_size(md5sig_count *
+				       sizeof(struct tcp_diag_md5sig));
+	}
+
+	return size;
+}
+EXPORT_SYMBOL_GPL(tcp_md5_diag_get_aux_size);
+
+const struct tcp_sock_af_ops tcp_sock_ipv4_specific = {
+	.md5_lookup	= tcp_v4_md5_lookup,
+	.calc_md5_hash	= tcp_v4_md5_hash_skb,
+	.md5_parse	= tcp_v4_parse_md5_keys,
+};
+
+#if IS_ENABLED(CONFIG_IPV6)
+const struct tcp_sock_af_ops tcp_sock_ipv6_specific = {
+	.md5_lookup	=	tcp_v6_md5_lookup,
+	.calc_md5_hash	=	tcp_v6_md5_hash_skb,
+	.md5_parse	=	tcp_v6_parse_md5_keys,
+};
+EXPORT_SYMBOL_GPL(tcp_sock_ipv6_specific);
+
+const struct tcp_sock_af_ops tcp_sock_ipv6_mapped_specific = {
+	.md5_lookup	=	tcp_v4_md5_lookup,
+	.calc_md5_hash	=	tcp_v4_md5_hash_skb,
+	.md5_parse	=	tcp_v6_parse_md5_keys,
+};
+EXPORT_SYMBOL_GPL(tcp_sock_ipv6_mapped_specific);
+#endif
+
net/ipv4/tcp_minisocks.c +4 -21
index ce8d1f4548a2..aeb68687a75c 100644
@@ -22,6 +22,7 @@
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/sysctl.h>
+#include <linux/tcp_md5.h>
 #include <linux/workqueue.h>
 #include <linux/static_key.h>
 #include <net/tcp.h>
@@ -295,21 +296,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 			INIT_HLIST_HEAD(&tcp_sk(sk)->tcp_option_list);
 		}
 #ifdef CONFIG_TCP_MD5SIG
-		/*
-		 * The timewait bucket does not have the key DB from the
-		 * sock structure. We just make a quick copy of the
-		 * md5 key being used (if indeed we are using one)
-		 * so the timewait ack generating code has the key.
-		 */
-		do {
-			struct tcp_md5sig_key *key;
-			tcptw->tw_md5_key = NULL;
-			key = tp->af_specific->md5_lookup(sk, sk);
-			if (key) {
-				tcptw->tw_md5_key = kmemdup(key, sizeof(*key), GFP_ATOMIC);
-				BUG_ON(tcptw->tw_md5_key && !tcp_alloc_md5sig_pool());
-			}
-		} while (0);
+		tcp_md5_time_wait(sk, tw);
 #endif
 
 		/* Get the TIME_WAIT timeout firing. */
@@ -345,10 +332,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 void tcp_twsk_destructor(struct sock *sk)
 {
 #ifdef CONFIG_TCP_MD5SIG
-	struct tcp_timewait_sock *twsk = tcp_twsk(sk);
-
-	if (twsk->tw_md5_key)
-		kfree_rcu(twsk->tw_md5_key, rcu);
+	tcp_md5_twsk_destructor(sk);
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled))
@@ -537,8 +521,7 @@ struct sock *tcp_create_openreq_child(const struct sock *sk,
 		newtp->tsoffset = treq->ts_off;
 #ifdef CONFIG_TCP_MD5SIG
 		newtp->md5sig_info = NULL;	/*XXX*/
-		if (newtp->af_specific->md5_lookup(sk, newsk))
-			newtp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
+		tcp_md5_add_header_len(sk, newsk);
 #endif
 		if (static_branch_unlikely(&tcp_extopt_enabled))
 			newtp->tcp_header_len += tcp_extopt_add_header(sk, newsk);
net/ipv4/tcp_output.c +2 -2
index b0297f19984f..e15006277275 100644
@@ -42,6 +42,7 @@
 #include <linux/gfp.h>
 #include <linux/module.h>
 #include <linux/static_key.h>
+#include <linux/tcp_md5.h>
 
 #include <trace/events/tcp.h>
 
@@ -3238,8 +3239,7 @@ static void tcp_connect_init(struct sock *sk)
 		tp->tcp_header_len += TCPOLEN_TSTAMP_ALIGNED;
 
 #ifdef CONFIG_TCP_MD5SIG
-	if (tp->af_specific->md5_lookup(sk, sk))
-		tp->tcp_header_len += TCPOLEN_MD5SIG_ALIGNED;
+	tcp_md5_add_header_len(sk, sk);
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled))
net/ipv6/tcp_ipv6.c +14 -268
index fccfce82f102..69ceebebb123 100644
@@ -43,6 +43,7 @@
 #include <linux/ipv6.h>
 #include <linux/icmpv6.h>
 #include <linux/random.h>
+#include <linux/tcp_md5.h>
 
 #include <net/tcp.h>
 #include <net/ndisc.h>
@@ -79,10 +80,6 @@ static int	tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb);
 
 static const struct inet_connection_sock_af_ops ipv6_mapped;
 static const struct inet_connection_sock_af_ops ipv6_specific;
-#ifdef CONFIG_TCP_MD5SIG
-static const struct tcp_sock_af_ops tcp_sock_ipv6_specific;
-static const struct tcp_sock_af_ops tcp_sock_ipv6_mapped_specific;
-#endif
 
 static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
 {
@@ -501,218 +498,6 @@ static void tcp_v6_reqsk_destructor(struct request_sock *req)
 	kfree_skb(inet_rsk(req)->pktopts);
 }
 
-#ifdef CONFIG_TCP_MD5SIG
-static struct tcp_md5sig_key *tcp_v6_md5_do_lookup(const struct sock *sk,
-						   const struct in6_addr *addr)
-{
-	return tcp_md5_do_lookup(sk, (union tcp_md5_addr *)addr, AF_INET6);
-}
-
-static struct tcp_md5sig_key *tcp_v6_md5_lookup(const struct sock *sk,
-						const struct sock *addr_sk)
-{
-	return tcp_v6_md5_do_lookup(sk, &addr_sk->sk_v6_daddr);
-}
-
-static int tcp_v6_parse_md5_keys(struct sock *sk, int optname,
-				 char __user *optval, int optlen)
-{
-	struct tcp_md5sig cmd;
-	struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)&cmd.tcpm_addr;
-	u8 prefixlen;
-
-	if (optlen < sizeof(cmd))
-		return -EINVAL;
-
-	if (copy_from_user(&cmd, optval, sizeof(cmd)))
-		return -EFAULT;
-
-	if (sin6->sin6_family != AF_INET6)
-		return -EINVAL;
-
-	if (optname == TCP_MD5SIG_EXT &&
-	    cmd.tcpm_flags & TCP_MD5SIG_FLAG_PREFIX) {
-		prefixlen = cmd.tcpm_prefixlen;
-		if (prefixlen > 128 || (ipv6_addr_v4mapped(&sin6->sin6_addr) &&
-					prefixlen > 32))
-			return -EINVAL;
-	} else {
-		prefixlen = ipv6_addr_v4mapped(&sin6->sin6_addr) ? 32 : 128;
-	}
-
-	if (!cmd.tcpm_keylen) {
-		if (ipv6_addr_v4mapped(&sin6->sin6_addr))
-			return tcp_md5_do_del(sk, (union tcp_md5_addr *)&sin6->sin6_addr.s6_addr32[3],
-					      AF_INET, prefixlen);
-		return tcp_md5_do_del(sk, (union tcp_md5_addr *)&sin6->sin6_addr,
-				      AF_INET6, prefixlen);
-	}
-
-	if (cmd.tcpm_keylen > TCP_MD5SIG_MAXKEYLEN)
-		return -EINVAL;
-
-	if (ipv6_addr_v4mapped(&sin6->sin6_addr))
-		return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin6->sin6_addr.s6_addr32[3],
-				      AF_INET, prefixlen, cmd.tcpm_key,
-				      cmd.tcpm_keylen, GFP_KERNEL);
-
-	return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin6->sin6_addr,
-			      AF_INET6, prefixlen, cmd.tcpm_key,
-			      cmd.tcpm_keylen, GFP_KERNEL);
-}
-
-static int tcp_v6_md5_hash_headers(struct tcp_md5sig_pool *hp,
-				   const struct in6_addr *daddr,
-				   const struct in6_addr *saddr,
-				   const struct tcphdr *th, int nbytes)
-{
-	struct tcp6_pseudohdr *bp;
-	struct scatterlist sg;
-	struct tcphdr *_th;
-
-	bp = hp->scratch;
-	/* 1. TCP pseudo-header (RFC2460) */
-	bp->saddr = *saddr;
-	bp->daddr = *daddr;
-	bp->protocol = cpu_to_be32(IPPROTO_TCP);
-	bp->len = cpu_to_be32(nbytes);
-
-	_th = (struct tcphdr *)(bp + 1);
-	memcpy(_th, th, sizeof(*th));
-	_th->check = 0;
-
-	sg_init_one(&sg, bp, sizeof(*bp) + sizeof(*th));
-	ahash_request_set_crypt(hp->md5_req, &sg, NULL,
-				sizeof(*bp) + sizeof(*th));
-	return crypto_ahash_update(hp->md5_req);
-}
-
-static int tcp_v6_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
-			       const struct in6_addr *daddr, struct in6_addr *saddr,
-			       const struct tcphdr *th)
-{
-	struct tcp_md5sig_pool *hp;
-	struct ahash_request *req;
-
-	hp = tcp_get_md5sig_pool();
-	if (!hp)
-		goto clear_hash_noput;
-	req = hp->md5_req;
-
-	if (crypto_ahash_init(req))
-		goto clear_hash;
-	if (tcp_v6_md5_hash_headers(hp, daddr, saddr, th, th->doff << 2))
-		goto clear_hash;
-	if (tcp_md5_hash_key(hp, key))
-		goto clear_hash;
-	ahash_request_set_crypt(req, NULL, md5_hash, 0);
-	if (crypto_ahash_final(req))
-		goto clear_hash;
-
-	tcp_put_md5sig_pool();
-	return 0;
-
-clear_hash:
-	tcp_put_md5sig_pool();
-clear_hash_noput:
-	memset(md5_hash, 0, 16);
-	return 1;
-}
-
-static int tcp_v6_md5_hash_skb(char *md5_hash,
-			       const struct tcp_md5sig_key *key,
-			       const struct sock *sk,
-			       const struct sk_buff *skb)
-{
-	const struct in6_addr *saddr, *daddr;
-	struct tcp_md5sig_pool *hp;
-	struct ahash_request *req;
-	const struct tcphdr *th = tcp_hdr(skb);
-
-	if (sk) { /* valid for establish/request sockets */
-		saddr = &sk->sk_v6_rcv_saddr;
-		daddr = &sk->sk_v6_daddr;
-	} else {
-		const struct ipv6hdr *ip6h = ipv6_hdr(skb);
-		saddr = &ip6h->saddr;
-		daddr = &ip6h->daddr;
-	}
-
-	hp = tcp_get_md5sig_pool();
-	if (!hp)
-		goto clear_hash_noput;
-	req = hp->md5_req;
-
-	if (crypto_ahash_init(req))
-		goto clear_hash;
-
-	if (tcp_v6_md5_hash_headers(hp, daddr, saddr, th, skb->len))
-		goto clear_hash;
-	if (tcp_md5_hash_skb_data(hp, skb, th->doff << 2))
-		goto clear_hash;
-	if (tcp_md5_hash_key(hp, key))
-		goto clear_hash;
-	ahash_request_set_crypt(req, NULL, md5_hash, 0);
-	if (crypto_ahash_final(req))
-		goto clear_hash;
-
-	tcp_put_md5sig_pool();
-	return 0;
-
-clear_hash:
-	tcp_put_md5sig_pool();
-clear_hash_noput:
-	memset(md5_hash, 0, 16);
-	return 1;
-}
-
-#endif
-
-static bool tcp_v6_inbound_md5_hash(const struct sock *sk,
-				    const struct sk_buff *skb)
-{
-#ifdef CONFIG_TCP_MD5SIG
-	const __u8 *hash_location = NULL;
-	struct tcp_md5sig_key *hash_expected;
-	const struct ipv6hdr *ip6h = ipv6_hdr(skb);
-	const struct tcphdr *th = tcp_hdr(skb);
-	int genhash;
-	u8 newhash[16];
-
-	hash_expected = tcp_v6_md5_do_lookup(sk, &ip6h->saddr);
-	hash_location = tcp_parse_md5sig_option(th);
-
-	/* We've parsed the options - do we have a hash? */
-	if (!hash_expected && !hash_location)
-		return false;
-
-	if (hash_expected && !hash_location) {
-		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5NOTFOUND);
-		return true;
-	}
-
-	if (!hash_expected && hash_location) {
-		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5UNEXPECTED);
-		return true;
-	}
-
-	/* check the signature */
-	genhash = tcp_v6_md5_hash_skb(newhash,
-				      hash_expected,
-				      NULL, skb);
-
-	if (genhash || memcmp(hash_location, newhash, 16) != 0) {
-		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMD5FAILURE);
-		net_info_ratelimited("MD5 Hash %s for [%pI6c]:%u->[%pI6c]:%u\n",
-				     genhash ? "failed" : "mismatch",
-				     &ip6h->saddr, ntohs(th->source),
-				     &ip6h->daddr, ntohs(th->dest));
-		return true;
-	}
-#endif
-	return false;
-}
-
 static void tcp_v6_init_req(struct request_sock *req,
 			    const struct sock *sk_listener,
 			    struct sk_buff *skb)
@@ -786,56 +571,24 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	struct dst_entry *dst;
 	__be32 *topt;
 	struct tcp_out_options extraopts;
-#ifdef CONFIG_TCP_MD5SIG
-	struct tcp_md5sig_key *key = NULL;
-	const __u8 *hash_location = NULL;
-	struct ipv6hdr *ipv6h = ipv6_hdr(skb);
-#endif
+
+	memset(&extraopts, 0, sizeof(extraopts));
 
 	if (tsecr)
 		tot_len += TCPOLEN_TSTAMP_ALIGNED;
 #ifdef CONFIG_TCP_MD5SIG
-	rcu_read_lock();
-	hash_location = tcp_parse_md5sig_option(th);
-	if (sk && sk_fullsock(sk)) {
-		key = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
-	} else if (sk && sk->sk_state == TCP_TIME_WAIT) {
-		struct tcp_timewait_sock *tcptw = tcp_twsk(sk);
-
-		key = tcp_twsk_md5_key(tcptw);
-	} else if (sk && sk->sk_state == TCP_NEW_SYN_RECV) {
-		key = tcp_v6_md5_do_lookup(sk, &ipv6h->saddr);
-	} else if (hash_location) {
-		unsigned char newhash[16];
-		struct sock *sk1 = NULL;
-		int genhash;
-
-		/* active side is lost. Try to find listening socket through
-		 * source port, and then find md5 key through listening socket.
-		 * we are not loose security here:
-		 * Incoming packet is checked with md5 hash with finding key,
-		 * no RST generated if md5 hash doesn't match.
-		 */
-		sk1 = inet6_lookup_listener(dev_net(skb_dst(skb)->dev),
-					    &tcp_hashinfo, NULL, 0,
-					    &ipv6h->saddr,
-					    th->source, &ipv6h->daddr,
-					    ntohs(th->source), tcp_v6_iif(skb),
-					    tcp_v6_sdif(skb));
-		if (!sk1)
-			goto out;
+{
+	int ret;
 
-		key = tcp_v6_md5_do_lookup(sk1, &ipv6h->saddr);
-		if (!key)
-			goto out;
+	ret = tcp_v6_md5_send_response_prepare(skb, 0,
+					       MAX_TCP_OPTION_SPACE - tot_len,
+					       &extraopts, sk);
 
-		genhash = tcp_v6_md5_hash_skb(newhash, key, NULL, skb);
-		if (genhash || memcmp(hash_location, newhash, 16) != 0)
-			goto out;
-	}
+	if (ret == -1)
+		goto out;
 
-	if (key)
-		tot_len += TCPOLEN_MD5SIG_ALIGNED;
+	tot_len += ret;
+}
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled)) {
@@ -846,8 +599,6 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 		if (!rst || !th->ack)
 			extraflags |= TCPHDR_ACK;
 
-		memset(&extraopts, 0, sizeof(extraopts));
-
 		used = tcp_extopt_response_prepare(skb, extraflags, remaining,
 						   &extraopts, sk);
 
@@ -885,13 +636,8 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 	}
 
 #ifdef CONFIG_TCP_MD5SIG
-	if (key) {
-		*topt++ = htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16) |
-				(TCPOPT_MD5SIG << 8) | TCPOLEN_MD5SIG);
-		tcp_v6_md5_hash_hdr((__u8 *)topt, key,
-				    &ipv6_hdr(skb)->saddr,
-				    &ipv6_hdr(skb)->daddr, t1);
-	}
+	if (extraopts.md5)
+		tcp_v6_md5_send_response_write(topt, skb, t1, &extraopts, sk);
 #endif
 
 	if (static_branch_unlikely(&tcp_extopt_enabled))
@@ -939,10 +685,6 @@ static void tcp_v6_send_response(const struct sock *sk, struct sk_buff *skb, u32
 

out:

	kfree_skb(buff);
net/ipv6/tcp_ipv6.c +0 -4
-
-#ifdef CONFIG_TCP_MD5SIG
-	rcu_read_unlock();
-#endif

}

static void tcp_v6_send_reset(const struct sock *sk, struct sk_buff *skb)

net/ipv6/tcp_ipv6.c +1 -31
@@ -1068,9 +810,6 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	struct inet_sock *newinet;
 	struct tcp_sock *newtp;
 	struct sock *newsk;
-#ifdef CONFIG_TCP_MD5SIG
-	struct tcp_md5sig_key *key;
-#endif
 	struct flowi6 fl6;
 
 	if (skb->protocol == htons(ETH_P_IP)) {
@@ -1220,18 +959,7 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 	}
 
 #ifdef CONFIG_TCP_MD5SIG
-	/* Copy over the MD5 key from the original socket */
-	key = tcp_v6_md5_do_lookup(sk, &newsk->sk_v6_daddr);
-	if (key) {
-		/* We're using one, so create a matching key
-		 * on the newsk structure. If we fail to get
-		 * memory, then we end up not copying the key
-		 * across. Shucks.
-		 */
-		tcp_md5_do_add(newsk, (union tcp_md5_addr *)&newsk->sk_v6_daddr,
-			       AF_INET6, 128, key->key, key->keylen,
-			       sk_gfp_mask(sk, GFP_ATOMIC));
-	}
+	tcp_v6_md5_syn_recv_sock(sk, newsk);
 #endif
 
 	if (__inet_inherit_port(sk, newsk) < 0) {
@@ -1688,14 +1416,6 @@ static const struct inet_connection_sock_af_ops ipv6_specific = {
 	.mtu_reduced	   = tcp_v6_mtu_reduced,
 };
 
-#ifdef CONFIG_TCP_MD5SIG
-static const struct tcp_sock_af_ops tcp_sock_ipv6_specific = {
-	.md5_lookup	=	tcp_v6_md5_lookup,
-	.calc_md5_hash	=	tcp_v6_md5_hash_skb,
-	.md5_parse	=	tcp_v6_parse_md5_keys,
-};
-#endif
-
 /*
 *	TCP over IPv4 via INET6 API
 */
@@ -1718,14 +1438,6 @@ static const struct inet_connection_sock_af_ops ipv6_mapped = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
-#ifdef CONFIG_TCP_MD5SIG
-static const struct tcp_sock_af_ops tcp_sock_ipv6_mapped_specific = {
-	.md5_lookup	=	tcp_v4_md5_lookup,
-	.calc_md5_hash	=	tcp_v4_md5_hash_skb,
-	.md5_parse	=	tcp_v6_parse_md5_keys,
-};
-#endif
-
 /* NOTE: A lot of things set to zero explicitly by call to
 *       sk_alloc() so need not be done here.
 */
Signature · 1 line
2.15.0

--
Mat Martineau
Intel OTC