mptcp: token container refactor
Subsystems (2)
Paolo Abeni
v2 cover mptcp: token container refactor
This series supersede the RTC patch for token refactor. The relevant patch is unchanged, but this additional bring in some related minor cleanup (patch 1/4) and KUNIT (!!!) self-tests (4/4). To make things cleaner it also moves the existing in-kernel tests (crypto) to KUNIT.
The most relevant changes from v1 are in patch 2/4.
This addresses most of the issues raised by Christoph, Davide and Mat, with the notable exception of shirinking the hash table bucket size - I'm unable to do that with the current fixes for the list curruption noted by Christoph.
I hope the trade-off beween the self-contained fix and the avoided memory optimization is acceptable.
Paolo Abeni (4):
mptcp: add __init annotation on setup functions
mptcp: refactor token container.
mptcp: move crypto test to KUNIT
mptcp: introduce token KUNIT self-tests
net/mptcp/Kconfig | 19 ++- net/mptcp/Makefile | 4 + net/mptcp/crypto.c | 65 +--------- net/mptcp/crypto_test.c | 72 +++++++++++ net/mptcp/pm.c | 2 +- net/mptcp/pm_netlink.c | 2 +- net/mptcp/protocol.c | 39 +++--- net/mptcp/protocol.h | 15 ++- net/mptcp/subflow.c | 12 +- net/mptcp/token.c | 258 ++++++++++++++++++++++++++++------------ net/mptcp/token_test.c | 138 +++++++++++++++++++++ 11 files changed, 453 insertions(+), 173 deletions(-) create mode 100644 net/mptcp/crypto_test.c create mode 100644 net/mptcp/token_test.c
Signature · 1 line
2.21.3
Paolo Abeni
PATCH v2 1/4 mptcp: add __init annotation on setup functions
Add the missing annotation in some setup-only
functions.
Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
v1 -> v2
- move mptcp_token_init() into the next patch
- added missing annotation for pm_netlink
--- net/mptcp/pm.c | 2 +- net/mptcp/pm_netlink.c | 2 +- net/mptcp/protocol.c | 4 ++-- net/mptcp/protocol.h | 10 +++++----- net/mptcp/subflow.c | 2 +- 5 files changed, 10 insertions(+), 10 deletions(-)
net/mptcp/pm.c +1 -1
@@ -234,7 +234,7 @@ void mptcp_pm_close(struct mptcp_sock *msk)sock_put ((struct sock *)msk); } -void mptcp_pm_init (void ) +void __init mptcp_pm_init (void ) { pm_wq =alloc_workqueue ("pm_wq" ,WQ_UNBOUND |WQ_MEM_RECLAIM ,8 );if (!pm_wq)
net/mptcp/pm_netlink.c +1 -1
@@ -851,7 +851,7 @@ static struct pernet_operations mptcp_pm_pernet_ops = { .size = sizeof(struct pm_nl_pernet), }; -void mptcp_pm_nl_init (void ) +void __init mptcp_pm_nl_init (void ) {if (register_pernet_subsys (&mptcp_pm_pernet_ops) <0 )panic ("Failed to register MPTCP PM pernet subsystem.\n" );
net/mptcp/protocol.c +2 -2
@@ -2028,7 +2028,7 @@ static struct inet_protosw mptcp_protosw = { .flags = INET_PROTOSW_ICSK, }; -void mptcp_proto_init (void ) +void __init mptcp_proto_init (void ) { mptcp_prot.h.hashinfo = tcp_prot.h.hashinfo;
@@ -2090,7 +2090,7 @@ static struct inet_protosw mptcp_v6_protosw = { .flags = INET_PROTOSW_ICSK, }; -int mptcp_proto_v6_init (void ) +int __init mptcp_proto_v6_init (void ) {int err;
net/mptcp/protocol.h +5 -5
@@ -342,7 +342,7 @@ mptcp_subflow_get_mapped_dsn(const struct mptcp_subflow_context *subflow)int mptcp_is_enabled (struct net *net);bool mptcp_subflow_data_available (struct sock *sk); -void mptcp_subflow_init (void ); +void __init mptcp_subflow_init (void );/* called with sk socket lock held */ int __mptcp_subflow_connect(struct sock *sk, int ifindex,
@@ -365,9 +365,9 @@ extern const struct inet_connection_sock_af_ops ipv4_specific;extern const struct inet_connection_sock_af_ops ipv6_specific;#endif -void mptcp_proto_init (void ); +void __init mptcp_proto_init (void );#if IS_ENABLED (CONFIG_MPTCP_IPV6 ) -int mptcp_proto_v6_init (void ); +int __init mptcp_proto_v6_init (void );#endif struct sock *mptcp_sk_clone(const struct sock *sk,
@@ -404,7 +404,7 @@ static inline void mptcp_crypto_key_gen_sha(u64 *key, u32 *token, u64 *idsn)void mptcp_crypto_hmac_sha (u64 key1,u64 key2,u8 *msg,int len,void *hmac); -void mptcp_pm_init (void ); +void __init mptcp_pm_init (void );void mptcp_pm_data_init (struct mptcp_sock *msk);void mptcp_pm_close (struct mptcp_sock *msk);void mptcp_pm_new_connection (struct mptcp_sock *msk,int server_side);
@@ -438,7 +438,7 @@ bool mptcp_pm_addr_signal(struct mptcp_sock *msk, unsigned int remaining, struct mptcp_addr_info *saddr);int mptcp_pm_get_local_id (struct mptcp_sock *msk,struct sock_common *skc); -void mptcp_pm_nl_init (void ); +void __init mptcp_pm_nl_init (void );void mptcp_pm_nl_data_init (struct mptcp_sock *msk);void mptcp_pm_nl_fully_established (struct mptcp_sock *msk);void mptcp_pm_nl_subflow_established (struct mptcp_sock *msk);
net/mptcp/subflow.c +1 -1
@@ -1238,7 +1238,7 @@ static int subflow_ops_init(struct request_sock_ops *subflow_ops)return 0 ; } -void mptcp_subflow_init (void ) +void __init mptcp_subflow_init (void ) { subflow_request_sock_ops = tcp_request_sock_ops;if (subflow_ops_init (&subflow_request_sock_ops) !=0 )
Signature · 1 line
2.21.3
Paolo Abeni
PATCH v2 2/4 mptcp: refactor token container.
In reply to Paolo AbeniReplace the radix tree with an hash table allocated at boot time. The radix tree has some short coming:
a single lock is contented by all the mptcp operation, the lookup currently use such lock, and traversing all the items would require lock, too.
With hash table instead we trade a little memory to address all the above - a per bucket lock is used.
To hash the MPTCP sockets, we re-use the msk' sk_node
entry: the MPTCP sockets are never hashed by the stack. Replace the existing hash proto callbacks with dummy implementation, annotating the above constraint.
Additionally refactor the token creation to code to:
- limit the number of consecutive attempts to a fixed maximum. Hitting an hash bucket with long chain is considered a failed attempt
- accept() no longer can fail to to token management.
- if token creation fails at connect() time, we do fallback to TCP (before the connection was closed)
---
v1 -> v2:
- __token_bucket_busy is now static (Mat)
- drop refcnt on lookup loop (CP)
- use sk_node to hash the msk.
--- net/mptcp/protocol.c | 35 +++--- net/mptcp/protocol.h | 5 +- net/mptcp/subflow.c | 10 +- net/mptcp/token.c | 249 ++++++++++++++++++++++++++++++------------- 4 files changed, 204 insertions(+), 95 deletions(-)
net/mptcp/protocol.c +19 -16
@@ -1424,19 +1424,7 @@ struct sock *mptcp_sk_clone(const struct sock *sk, msk->token = subflow_req->token; msk->subflow =NULL ; -if (unlikely (mptcp_token_new_accept (subflow_req->token, nsk))) { - nsk->sk_state =TCP_CLOSE ; -bh_unlock_sock (nsk); - -/* we can't call into mptcp_close() here - possible BH context -* free the sock directly. -* sk_clone_lock() sets nsk refcnt to two, hence call sk_free() -* too. -*/ -sk_common_release (nsk); -sk_free (nsk); -return NULL ; - } +mptcp_token_accept (subflow_req, msk); msk->write_seq = subflow_req->idsn +1 ;atomic64_set (&msk->snd_una, msk->write_seq);
@@ -1611,6 +1599,20 @@ static void mptcp_release_cb(struct sock *sk) } } +static int mptcp_hash (struct sock *sk) +{ +/* should never be called, +* we hash the TCP subflows not the master socket +*/ +WARN_ON_ONCE (1 ); +return 0 ; +} + +static void mptcp_unhash (struct sock *sk) +{ +/* called from sk_common_release(), but nothing to do here */ +} +static int mptcp_get_port (struct sock *sk,unsigned short snum) {struct mptcp_sock *msk =mptcp_sk (sk);
@@ -1654,7 +1656,6 @@ void mptcp_finish_connect(struct sock *ssk)*/ WRITE_ONCE (msk->remote_key, subflow->remote_key);WRITE_ONCE (msk->local_key, subflow->local_key); -WRITE_ONCE (msk->token, subflow->token);WRITE_ONCE (msk->write_seq, subflow->idsn +1 );WRITE_ONCE (msk->ack_seq, ack_seq);WRITE_ONCE (msk->can_ack,1 );
@@ -1728,8 +1729,8 @@ static struct proto mptcp_prot = { .sendmsg = mptcp_sendmsg, .recvmsg = mptcp_recvmsg, .release_cb = mptcp_release_cb, - .hash = inet_hash, - .unhash = inet_unhash, + .hash = mptcp_hash, + .unhash = mptcp_unhash, .get_port = mptcp_get_port, .sockets_allocated = &mptcp_sockets_allocated, .memory_allocated = &tcp_memory_allocated,
@@ -1738,6 +1739,7 @@ static struct proto mptcp_prot = { .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_mem = sysctl_tcp_mem, .obj_size = sizeof(struct mptcp_sock), + .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, };
@@ -2037,6 +2039,7 @@ void __init mptcp_proto_init(void)mptcp_subflow_init ();mptcp_pm_init (); +mptcp_token_init ();if (proto_register (&mptcp_prot,1 ) !=0 )panic ("Failed to register MPTCP proto.\n" );
net/mptcp/protocol.h +4 -1
@@ -255,6 +255,7 @@ struct mptcp_subflow_request_sock {u64 thmac;u32 local_nonce;u32 remote_nonce; +struct hlist_nulls_node token_node; }; static inline struct mptcp_subflow_request_sock *
@@ -382,10 +383,12 @@ bool mptcp_finish_join(struct sock *sk);void mptcp_data_acked (struct sock *sk);void mptcp_subflow_eof (struct sock *sk); +void __init mptcp_token_init (void );int mptcp_token_new_request (struct request_sock *req);void mptcp_token_destroy_request (u32 token);int mptcp_token_new_connect (struct sock *sk); -int mptcp_token_new_accept (u32 token,struct sock *conn); +void mptcp_token_accept (struct mptcp_subflow_request_sock *r, +struct mptcp_sock *msk);struct mptcp_sock *mptcp_token_get_sock (u32 token);void mptcp_token_destroy (u32 token);
net/mptcp/subflow.c +5 -5
@@ -31,11 +31,14 @@ static void SUBFLOW_REQ_INC_STATS(struct request_sock *req,static int subflow_rebuild_header (struct sock *sk) {struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); -int local_id, err =0 ; +int local_id;if (subflow->request_mptcp && !subflow->token) {pr_debug ("subflow=%p" , sk); - err =mptcp_token_new_connect (sk); +if (mptcp_token_new_connect (sk)) { + subflow->mp_capable =0 ; + goto out; + } }else if (subflow->request_join && !subflow->local_nonce) {struct mptcp_sock *msk = (struct mptcp_sock *)subflow->conn;
@@ -56,9 +59,6 @@ static int subflow_rebuild_header(struct sock *sk) } out: -if (err) -return err; -return subflow->icsk_af_ops->rebuild_header (sk); }
net/mptcp/token.c +176 -73
307 lines not on this page · raw message
Signature · 1 line
2.21.3
Christoph Paasch
[MPTCP] Re: [PATCH v2 2/4] mptcp: refactor token container.
Hello,
(sorry for the late reply - I kind of missed this email)
Quoted 226 lines · Paolo Abeni
Replace the radix tree with an hash table allocated at boot time. The radix tree has some short coming:
a single lock is contented by all the mptcp operation, the lookup currently use such lock, and traversing all the items would require lock, too.With hash table instead we trade a little memory to address all the above - a per bucket lock is used.
To hash the MPTCP sockets, we re-use the msk'
sk_node
entry: the MPTCP sockets are never hashed by the stack. Replace the existing hash proto callbacks with dummy implementation, annotating the above constraint.Additionally refactor the token creation to code to:
- limit the number of consecutive attempts to a fixed maximum. Hitting an hash bucket with long chain is considered a failed attempt
-
accept()no longer can fail to to token management.- if token creation fails at
connect()time, we do fallback to TCP (before the connection was closed)Signed-off-by Paolo Abeni <pabeni(a)redhat.com>---
v1 -> v2:
-__token_bucket_busyis now static (Mat)
- drop refcnt on lookup loop (CP)
- usesk_nodeto hash the msk.--- net/mptcp/protocol.c | 35 +++--- net/mptcp/protocol.h | 5 +- net/mptcp/subflow.c | 10 +- net/mptcp/token.c | 249 ++++++++++++++++++++++++++++++------------- 4 files changed, 204 insertions(+), 95 deletions(-)net/mptcp/protocol.c +19
-16@@ -1424,19 +1424,7 @@ struct sock *mptcp_sk_clone(const struct sock *sk, msk->token = subflow_req->token; msk->subflow =NULL ; -if (unlikely (mptcp_token_new_accept (subflow_req->token, nsk))) { - nsk->sk_state =TCP_CLOSE ; -bh_unlock_sock (nsk); - -/* we can't call into mptcp_close() here - possible BH context -* free the sock directly. -* sk_clone_lock() sets nsk refcnt to two, hence call sk_free() -* too. -*/ -sk_common_release (nsk); -sk_free (nsk); -return NULL ; - } +mptcp_token_accept (subflow_req, msk); msk->write_seq = subflow_req->idsn +1 ;atomic64_set (&msk->snd_una, msk->write_seq);@@ -1611,6 +1599,20 @@ static void mptcp_release_cb(struct sock *sk) } } +static int mptcp_hash (struct sock *sk) +{ +/* should never be called, +* we hash the TCP subflows not the master socket +*/ +WARN_ON_ONCE (1 ); +return 0 ; +} + +static void mptcp_unhash (struct sock *sk) +{ +/* called from sk_common_release(), but nothing to do here */ +} +static int mptcp_get_port (struct sock *sk,unsigned short snum) {struct mptcp_sock *msk =mptcp_sk (sk);@@ -1654,7 +1656,6 @@ void mptcp_finish_connect(struct sock *ssk)*/ WRITE_ONCE (msk->remote_key, subflow->remote_key);WRITE_ONCE (msk->local_key, subflow->local_key); -WRITE_ONCE (msk->token, subflow->token);WRITE_ONCE (msk->write_seq, subflow->idsn +1 );WRITE_ONCE (msk->ack_seq, ack_seq);WRITE_ONCE (msk->can_ack,1 );@@ -1728,8 +1729,8 @@ static struct proto mptcp_prot = { .sendmsg = mptcp_sendmsg, .recvmsg = mptcp_recvmsg, .release_cb = mptcp_release_cb, - .hash = inet_hash, - .unhash = inet_unhash, + .hash = mptcp_hash, + .unhash = mptcp_unhash, .get_port = mptcp_get_port, .sockets_allocated = &mptcp_sockets_allocated, .memory_allocated = &tcp_memory_allocated,@@ -1738,6 +1739,7 @@ static struct proto mptcp_prot = { .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_mem = sysctl_tcp_mem, .obj_size = sizeof(struct mptcp_sock), + .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, };@@ -2037,6 +2039,7 @@ void __init mptcp_proto_init(void)mptcp_subflow_init ();mptcp_pm_init (); +mptcp_token_init ();if (proto_register (&mptcp_prot,1 ) !=0 )panic ("Failed to register MPTCP proto.\n" );net/mptcp/protocol.h +4
-1@@ -255,6 +255,7 @@ struct mptcp_subflow_request_sock {u64 thmac;u32 local_nonce;u32 remote_nonce; +struct hlist_nulls_node token_node; }; static inline struct mptcp_subflow_request_sock *@@ -382,10 +383,12 @@ bool mptcp_finish_join(struct sock *sk);void mptcp_data_acked (struct sock *sk);void mptcp_subflow_eof (struct sock *sk); +void __init mptcp_token_init (void );int mptcp_token_new_request (struct request_sock *req);void mptcp_token_destroy_request (u32 token);int mptcp_token_new_connect (struct sock *sk); -int mptcp_token_new_accept (u32 token,struct sock *conn); +void mptcp_token_accept (struct mptcp_subflow_request_sock *r, +struct mptcp_sock *msk);struct mptcp_sock *mptcp_token_get_sock (u32 token);void mptcp_token_destroy (u32 token);net/mptcp/subflow.c +5
-5@@ -31,11 +31,14 @@ static void SUBFLOW_REQ_INC_STATS(struct request_sock *req,static int subflow_rebuild_header (struct sock *sk) {struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); -int local_id, err =0 ; +int local_id;if (subflow->request_mptcp && !subflow->token) {pr_debug ("subflow=%p" , sk); - err =mptcp_token_new_connect (sk); +if (mptcp_token_new_connect (sk)) { + subflow->mp_capable =0 ; + goto out; + } }else if (subflow->request_join && !subflow->local_nonce) {struct mptcp_sock *msk = (struct mptcp_sock *)subflow->conn;@@ -56,9 +59,6 @@ static int subflow_rebuild_header(struct sock *sk) } out: -if (err) -return err; -return subflow->icsk_af_ops->rebuild_header (sk); }net/mptcp/token.c +30
-5@@ -24,7 +24,7 @@#include <linux/kernel.h> #include <linux/module.h> -#include <linux/radix-tree.h> +#include <linux/memblock.h> #include <linux/ip.h> #include <linux/tcp.h> #include <net/sock.h> @@ -33,10 +33,53 @@#include <net/mptcp.h> #include "protocol.h" -static RADIX_TREE(token_tree, GFP_ATOMIC); -static RADIX_TREE(token_req_tree, GFP_ATOMIC); -static DEFINE_SPINLOCK (token_tree_lock); -static int token_used__read_mostly ; +#define TOKEN_MAX_RETRIES 4 +#define TOKEN_MAX_CHAIN_LEN 4 + +struct token_bucket { +spinlock_t lock; +int chain_len; +struct hlist_nulls_head req_chain; +struct hlist_nulls_head msk_chain; +}; + +static struct token_bucket *token_hash__read_mostly ; +static unsigned int token_mask__read_mostly ; + +static struct token_bucket *token_bucket (u32 token) +{ +return &token_hash[token & token_mask]; +} + +static struct mptcp_subflow_request_sock * +__token_lookup_req (struct token_bucket *t,u32 token) +{ +struct mptcp_subflow_request_sock *req; +struct hlist_nulls_node *pos; + +hlist_nulls_for_each_entry_rcu (req, pos, &t->req_chain, token_node) +if (req->token == token) +return req; +return NULL ; +}
net/mptcp/token.c line 74
Quoted 10 lines
net/mptcp/token.c +10
-0+ +static struct mptcp_sock * +__token_lookup_msk (struct token_bucket *t,u32 token) +{ +struct hlist_nulls_node *pos; +struct sock *sk; + +sk_nulls_for_each_rcu (sk, pos, &t->msk_chain) +if (mptcp_sk (sk)->token == token) +return mptcp_sk (sk);
won't you need a "goto again" here if you end up on the wrong nulls? Same way as in mptcp_token_get_sock.
Christoph
Paolo Abeni
[MPTCP] Re: [PATCH v2 2/4] mptcp: refactor token container.
Quoted 236 lines · Christoph Paasch
Hello,
(sorry for the late reply - I kind of missed this email)
Quoted 231 lines · Paolo Abeni
Replace the radix tree with an hash table allocated at boot time. The radix tree has some short coming:
a single lock is contented by all the mptcp operation, the lookup currently use such lock, and traversing all the items would require lock, too.With hash table instead we trade a little memory to address all the above - a per bucket lock is used.
To hash the MPTCP sockets, we re-use the msk'
sk_node
entry: the MPTCP sockets are never hashed by the stack. Replace the existing hash proto callbacks with dummy implementation, annotating the above constraint.Additionally refactor the token creation to code to:
- limit the number of consecutive attempts to a fixed maximum. Hitting an hash bucket with long chain is considered a failed attempt
-
accept()no longer can fail to to token management.- if token creation fails at
connect()time, we do fallback to TCP (before the connection was closed)Signed-off-by Paolo Abeni <pabeni(a)redhat.com>---
v1 -> v2:
-__token_bucket_busyis now static (Mat)
- drop refcnt on lookup loop (CP)
- usesk_nodeto hash the msk.--- net/mptcp/protocol.c | 35 +++--- net/mptcp/protocol.h | 5 +- net/mptcp/subflow.c | 10 +- net/mptcp/token.c | 249 ++++++++++++++++++++++++++++++------------- 4 files changed, 204 insertions(+), 95 deletions(-)net/mptcp/protocol.c +19
-16@@ -1424,19 +1424,7 @@ struct sock *mptcp_sk_clone(const struct sock *sk, msk->token = subflow_req->token; msk->subflow =NULL ; -if (unlikely (mptcp_token_new_accept (subflow_req->token, nsk))) { - nsk->sk_state =TCP_CLOSE ; -bh_unlock_sock (nsk); - -/* we can't call into mptcp_close() here - possible BH context -* free the sock directly. -* sk_clone_lock() sets nsk refcnt to two, hence call sk_free() -* too. -*/ -sk_common_release (nsk); -sk_free (nsk); -return NULL ; - } +mptcp_token_accept (subflow_req, msk); msk->write_seq = subflow_req->idsn +1 ;atomic64_set (&msk->snd_una, msk->write_seq);@@ -1611,6 +1599,20 @@ static void mptcp_release_cb(struct sock *sk) } } +static int mptcp_hash (struct sock *sk) +{ +/* should never be called, +* we hash the TCP subflows not the master socket +*/ +WARN_ON_ONCE (1 ); +return 0 ; +} + +static void mptcp_unhash (struct sock *sk) +{ +/* called from sk_common_release(), but nothing to do here */ +} +static int mptcp_get_port (struct sock *sk,unsigned short snum) {struct mptcp_sock *msk =mptcp_sk (sk);@@ -1654,7 +1656,6 @@ void mptcp_finish_connect(struct sock *ssk)*/ WRITE_ONCE (msk->remote_key, subflow->remote_key);WRITE_ONCE (msk->local_key, subflow->local_key); -WRITE_ONCE (msk->token, subflow->token);WRITE_ONCE (msk->write_seq, subflow->idsn +1 );WRITE_ONCE (msk->ack_seq, ack_seq);WRITE_ONCE (msk->can_ack,1 );@@ -1728,8 +1729,8 @@ static struct proto mptcp_prot = { .sendmsg = mptcp_sendmsg, .recvmsg = mptcp_recvmsg, .release_cb = mptcp_release_cb, - .hash = inet_hash, - .unhash = inet_unhash, + .hash = mptcp_hash, + .unhash = mptcp_unhash, .get_port = mptcp_get_port, .sockets_allocated = &mptcp_sockets_allocated, .memory_allocated = &tcp_memory_allocated,@@ -1738,6 +1739,7 @@ static struct proto mptcp_prot = { .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_mem = sysctl_tcp_mem, .obj_size = sizeof(struct mptcp_sock), + .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, };@@ -2037,6 +2039,7 @@ void __init mptcp_proto_init(void)mptcp_subflow_init ();mptcp_pm_init (); +mptcp_token_init ();if (proto_register (&mptcp_prot,1 ) !=0 )panic ("Failed to register MPTCP proto.\n" );net/mptcp/protocol.h +4
-1@@ -255,6 +255,7 @@ struct mptcp_subflow_request_sock {u64 thmac;u32 local_nonce;u32 remote_nonce; +struct hlist_nulls_node token_node; }; static inline struct mptcp_subflow_request_sock *@@ -382,10 +383,12 @@ bool mptcp_finish_join(struct sock *sk);void mptcp_data_acked (struct sock *sk);void mptcp_subflow_eof (struct sock *sk); +void __init mptcp_token_init (void );int mptcp_token_new_request (struct request_sock *req);void mptcp_token_destroy_request (u32 token);int mptcp_token_new_connect (struct sock *sk); -int mptcp_token_new_accept (u32 token,struct sock *conn); +void mptcp_token_accept (struct mptcp_subflow_request_sock *r, +struct mptcp_sock *msk);struct mptcp_sock *mptcp_token_get_sock (u32 token);void mptcp_token_destroy (u32 token);net/mptcp/subflow.c +5
-5@@ -31,11 +31,14 @@ static void SUBFLOW_REQ_INC_STATS(struct request_sock *req,static int subflow_rebuild_header (struct sock *sk) {struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); -int local_id, err =0 ; +int local_id;if (subflow->request_mptcp && !subflow->token) {pr_debug ("subflow=%p" , sk); - err =mptcp_token_new_connect (sk); +if (mptcp_token_new_connect (sk)) { + subflow->mp_capable =0 ; + goto out; + } }else if (subflow->request_join && !subflow->local_nonce) {struct mptcp_sock *msk = (struct mptcp_sock *)subflow->conn;@@ -56,9 +59,6 @@ static int subflow_rebuild_header(struct sock *sk) } out: -if (err) -return err; -return subflow->icsk_af_ops->rebuild_header (sk); }net/mptcp/token.c +35
-5@@ -24,7 +24,7 @@#include <linux/kernel.h> #include <linux/module.h> -#include <linux/radix-tree.h> +#include <linux/memblock.h> #include <linux/ip.h> #include <linux/tcp.h> #include <net/sock.h> @@ -33,10 +33,53 @@#include <net/mptcp.h> #include "protocol.h" -static RADIX_TREE(token_tree, GFP_ATOMIC); -static RADIX_TREE(token_req_tree, GFP_ATOMIC); -static DEFINE_SPINLOCK (token_tree_lock); -static int token_used__read_mostly ; +#define TOKEN_MAX_RETRIES 4 +#define TOKEN_MAX_CHAIN_LEN 4 + +struct token_bucket { +spinlock_t lock; +int chain_len; +struct hlist_nulls_head req_chain; +struct hlist_nulls_head msk_chain; +}; + +static struct token_bucket *token_hash__read_mostly ; +static unsigned int token_mask__read_mostly ; + +static struct token_bucket *token_bucket (u32 token) +{ +return &token_hash[token & token_mask]; +} + +static struct mptcp_subflow_request_sock * +__token_lookup_req (struct token_bucket *t,u32 token) +{ +struct mptcp_subflow_request_sock *req; +struct hlist_nulls_node *pos; + +hlist_nulls_for_each_entry_rcu (req, pos, &t->req_chain, token_node) +if (req->token == token) +return req; +return NULL ; +} + +static struct mptcp_sock * +__token_lookup_msk (struct token_bucket *t,u32 token) +{ +struct hlist_nulls_node *pos;
Quoted 8 lines
net/mptcp/token.c line 74
Quoted 5 lines
net/mptcp/token.c +5
-0+struct sock *sk; + +sk_nulls_for_each_rcu (sk, pos, &t->msk_chain) +if (mptcp_sk (sk)->token == token) +return mptcp_sk (sk);won't you need a "goto again" here if you end up on the wrong nulls? Same way as in
mptcp_token_get_sock.
I think it's not needed here. This functions are are always called with bucket lock held, so the items can't move under the hood. I can add some comment to clarify the above, if needed.
Thanks!
Paolo
Christoph Paasch
[MPTCP] Re: [PATCH v2 2/4] mptcp: refactor token container.
Quoted 250 lines · Paolo Abeni
Quoted 236 lines · Christoph Paasch
Hello,
(sorry for the late reply - I kind of missed this email)
Quoted 232 lines · Paolo Abeni
Replace the radix tree with an hash table allocated at boot time. The radix tree has some short coming:
a single lock is contented by all the mptcp operation, the lookup currently use such lock, and traversing all the items would require lock, too.With hash table instead we trade a little memory to address all the above - a per bucket lock is used.
To hash the MPTCP sockets, we re-use the msk'
sk_node
entry: the MPTCP sockets are never hashed by the stack. Replace the existing hash proto callbacks with dummy implementation, annotating the above constraint.Additionally refactor the token creation to code to:
- limit the number of consecutive attempts to a fixed maximum. Hitting an hash bucket with long chain is considered a failed attempt
-
accept()no longer can fail to to token management.- if token creation fails at
connect()time, we do fallback to TCP (before the connection was closed)Signed-off-by Paolo Abeni <pabeni(a)redhat.com>---
v1 -> v2:
-__token_bucket_busyis now static (Mat)
- drop refcnt on lookup loop (CP)
- usesk_nodeto hash the msk.
---net/mptcp/protocol.c | 35 +++--- net/mptcp/protocol.h | 5 +- net/mptcp/subflow.c | 10 +- net/mptcp/token.c | 249 ++++++++++++++++++++++++++++++------------- 4 files changed, 204 insertions(+), 95 deletions(-)net/mptcp/protocol.c +19
-16@@ -1424,19 +1424,7 @@ struct sock *mptcp_sk_clone(const struct sock *sk, msk->token = subflow_req->token; msk->subflow =NULL ; -if (unlikely (mptcp_token_new_accept (subflow_req->token, nsk))) { - nsk->sk_state =TCP_CLOSE ; -bh_unlock_sock (nsk); - -/* we can't call into mptcp_close() here - possible BH context -* free the sock directly. -* sk_clone_lock() sets nsk refcnt to two, hence call sk_free() -* too. -*/ -sk_common_release (nsk); -sk_free (nsk); -return NULL ; - } +mptcp_token_accept (subflow_req, msk); msk->write_seq = subflow_req->idsn +1 ;atomic64_set (&msk->snd_una, msk->write_seq);@@ -1611,6 +1599,20 @@ static void mptcp_release_cb(struct sock *sk) } } +static int mptcp_hash (struct sock *sk) +{ +/* should never be called, +* we hash the TCP subflows not the master socket +*/ +WARN_ON_ONCE (1 ); +return 0 ; +} + +static void mptcp_unhash (struct sock *sk) +{ +/* called from sk_common_release(), but nothing to do here */ +} +static int mptcp_get_port (struct sock *sk,unsigned short snum) {struct mptcp_sock *msk =mptcp_sk (sk);@@ -1654,7 +1656,6 @@ void mptcp_finish_connect(struct sock *ssk)*/ WRITE_ONCE (msk->remote_key, subflow->remote_key);WRITE_ONCE (msk->local_key, subflow->local_key); -WRITE_ONCE (msk->token, subflow->token);WRITE_ONCE (msk->write_seq, subflow->idsn +1 );WRITE_ONCE (msk->ack_seq, ack_seq);WRITE_ONCE (msk->can_ack,1 );@@ -1728,8 +1729,8 @@ static struct proto mptcp_prot = { .sendmsg = mptcp_sendmsg, .recvmsg = mptcp_recvmsg, .release_cb = mptcp_release_cb, - .hash = inet_hash, - .unhash = inet_unhash, + .hash = mptcp_hash, + .unhash = mptcp_unhash, .get_port = mptcp_get_port, .sockets_allocated = &mptcp_sockets_allocated, .memory_allocated = &tcp_memory_allocated,@@ -1738,6 +1739,7 @@ static struct proto mptcp_prot = { .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_mem = sysctl_tcp_mem, .obj_size = sizeof(struct mptcp_sock), + .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, };@@ -2037,6 +2039,7 @@ void __init mptcp_proto_init(void)mptcp_subflow_init ();mptcp_pm_init (); +mptcp_token_init ();if (proto_register (&mptcp_prot,1 ) !=0 )panic ("Failed to register MPTCP proto.\n" );net/mptcp/protocol.h +1
-0@@ -255,6 +255,7 @@ struct mptcp_subflow_request_sock {u64 thmac;u32 local_nonce;u32 remote_nonce; +struct hlist_nulls_node token_node; };static inline struct
mptcp_subflow_request_sock*net/mptcp/protocol.h +3
-1@@ -382,10 +383,12 @@ bool mptcp_finish_join(struct sock *sk);void mptcp_data_acked (struct sock *sk);void mptcp_subflow_eof (struct sock *sk); +void __init mptcp_token_init (void );int mptcp_token_new_request (struct request_sock *req);void mptcp_token_destroy_request (u32 token);int mptcp_token_new_connect (struct sock *sk); -int mptcp_token_new_accept (u32 token,struct sock *conn); +void mptcp_token_accept (struct mptcp_subflow_request_sock *r, +struct mptcp_sock *msk);struct mptcp_sock *mptcp_token_get_sock (u32 token);void mptcp_token_destroy (u32 token);net/mptcp/subflow.c +5
-2@@ -31,11 +31,14 @@ static void SUBFLOW_REQ_INC_STATS(struct request_sock *req,static int subflow_rebuild_header (struct sock *sk) {struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); -int local_id, err =0 ; +int local_id;if (subflow->request_mptcp && !subflow->token) {pr_debug ("subflow=%p" , sk); - err =mptcp_token_new_connect (sk); +if (mptcp_token_new_connect (sk)) { + subflow->mp_capable =0 ; + goto out; + } }else if (subflow->request_join && !subflow->local_nonce) {struct mptcp_sock *msk = (struct mptcp_sock *)subflow->conn;@@ -56,9 +59,6 @@ static int subflow_rebuild_header(struct sock *sk) }out:
net/mptcp/subflow.c +0
-3-if (err) -return err; -return subflow->icsk_af_ops->rebuild_header (sk);}
net/mptcp/token.c +35
-5@@ -24,7 +24,7 @@#include <linux/kernel.h> #include <linux/module.h> -#include <linux/radix-tree.h> +#include <linux/memblock.h> #include <linux/ip.h> #include <linux/tcp.h> #include <net/sock.h> @@ -33,10 +33,53 @@#include <net/mptcp.h> #include "protocol.h" -static RADIX_TREE(token_tree, GFP_ATOMIC); -static RADIX_TREE(token_req_tree, GFP_ATOMIC); -static DEFINE_SPINLOCK (token_tree_lock); -static int token_used__read_mostly ; +#define TOKEN_MAX_RETRIES 4 +#define TOKEN_MAX_CHAIN_LEN 4 + +struct token_bucket { +spinlock_t lock; +int chain_len; +struct hlist_nulls_head req_chain; +struct hlist_nulls_head msk_chain; +}; + +static struct token_bucket *token_hash__read_mostly ; +static unsigned int token_mask__read_mostly ; + +static struct token_bucket *token_bucket (u32 token) +{ +return &token_hash[token & token_mask]; +} + +static struct mptcp_subflow_request_sock * +__token_lookup_req (struct token_bucket *t,u32 token) +{ +struct mptcp_subflow_request_sock *req; +struct hlist_nulls_node *pos; + +hlist_nulls_for_each_entry_rcu (req, pos, &t->req_chain, token_node) +if (req->token == token) +return req; +return NULL ; +} + +static struct mptcp_sock * +__token_lookup_msk (struct token_bucket *t,u32 token) +{ +struct hlist_nulls_node *pos;Quoted 8 lines
net/mptcp/token.c line 74
Quoted 5 lines
net/mptcp/token.c +5
-0+struct sock *sk; + +sk_nulls_for_each_rcu (sk, pos, &t->msk_chain) +if (mptcp_sk (sk)->token == token) +return mptcp_sk (sk);won't you need a "goto again" here if you end up on the wrong nulls? Same way as in
mptcp_token_get_sock.
Quoted 2 lines
I think it's not needed here. This functions are are always called with bucket lock held, so the items can't move under the hood.
Oh, I see!
Quoted 2 lines
I can add
some comment to clarify the above, if needed.
Yeah, why not! Series looks good to me. You can add my reviewed-by.
Christoph
Quoted 5 lines
Thanks!
Paolo
Paolo Abeni
PATCH v2 3/4 mptcp: move crypto test to KUNIT
In reply to Paolo Abenicurrently MPTCP uses a custom hook to executed unit test at boot time. Let's use the KUNIT framework instead. Additionally move the relevant code to a separate file and export the function needed by the test when self-tests are build as a module.
---
v1 -> v2:
- add SPDX line (Davide)
--- net/mptcp/Kconfig | 19 +++++++---- net/mptcp/Makefile | 3 ++ net/mptcp/crypto.c | 65 ++----------------------------------- net/mptcp/crypto_test.c | 72 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 91 insertions(+), 68 deletions(-) create mode 100644 net/mptcp/crypto_test.c
net/mptcp/Kconfig +13 -6
@@ -19,12 +19,19 @@ config MPTCP_IPV6 select IPV6 default y -config MPTCP_HMAC_TEST - bool "Tests for MPTCP HMAC implementation" +endif + +config MPTCP_KUNIT_TESTS + tristate "This builds the MPTCP KUnit tests" + select MPTCP + depends on KUNIT help - This option enable boot time self-test for the HMAC implementation - used by the MPTCP code + Currently covers the MPTCP crypto helpers. + Only useful for kernel devs running KUnit test harness and are not + for inclusion into a production build. - Say N if you are unsure. + For more information on KUnit and unit tests in general please refer + to the KUnit documentation in Documentation/dev-tools/kunit/. + + If unsure, say N. -endif
net/mptcp/Makefile +3 -0
@@ -3,3 +3,6 @@ obj-$(CONFIG_MPTCP) += mptcp.o mptcp-y := protocol.o subflow.o options.o token.o crypto.o ctrl.o pm.o diag.o \ mib.o pm_netlink.o + +mptcp_crypto_test-objs := crypto_test.o +obj-$(CONFIG_MPTCP_KUNIT_TESTS) += mptcp_crypto_test.o
net/mptcp/crypto.c +3 -62
69 lines not on this page · raw message
net/mptcp/crypto_test.c new file +72 -0
72 lines not on this page · raw message
Signature · 1 line
2.21.3
Paolo Abeni
PATCH v2 4/4 mptcp: introduce token KUNIT self-tests
In reply to Paolo AbeniUnit tests for the internal MPTCP token APIs, using KUNIT
---
v1 -> v2:
- add SPDX line (Davide)
--- net/mptcp/Makefile | 3 +- net/mptcp/token.c | 9 +++ net/mptcp/token_test.c | 138 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 net/mptcp/token_test.c
net/mptcp/Makefile +2 -1
@@ -5,4 +5,5 @@ mptcp-y := protocol.o subflow.o options.o token.o crypto.o ctrl.o pm.o diag.o \ mib.o pm_netlink.o mptcp_crypto_test-objs := crypto_test.o -obj-$(CONFIG_MPTCP_KUNIT_TESTS) += mptcp_crypto_test.o +mptcp_token_test-objs := token_test.o +obj-$(CONFIG_MPTCP_KUNIT_TESTS) += mptcp_crypto_test.o mptcp_token_test.o
net/mptcp/token.c +9 -0
@@ -297,3 +297,12 @@ void __init mptcp_token_init(void)spin_lock_init (&token_hash[i].lock); } } + +#if IS_MODULE (CONFIG_MPTCP_KUNIT_TESTS ) +EXPORT_SYMBOL_GPL (mptcp_token_new_request); +EXPORT_SYMBOL_GPL (mptcp_token_new_connect); +EXPORT_SYMBOL_GPL (mptcp_token_accept); +EXPORT_SYMBOL_GPL (mptcp_token_get_sock); +EXPORT_SYMBOL_GPL (mptcp_token_destroy_request); +EXPORT_SYMBOL_GPL (mptcp_token_destroy); +#endif
net/mptcp/token_test.c new file +138 -0
139 lines not on this page · raw message
Signature · 1 line
2.21.3
Mat Martineau
Hi Paolo,
Quoted 11 lines · Paolo Abeni
This series supersede the RTC patch for token refactor. The relevant patch is unchanged, but this additional bring in some related minor cleanup (patch 1/4) and KUNIT (!!!) self-tests (4/4). To make things cleaner it also moves the existing in-kernel tests (crypto) to KUNIT.
The most relevant changes from v1 are in patch 2/4.
This addresses most of the issues raised by Christoph, Davide and Mat, with the notable exception of shirinking the hash table bucket size - I'm unable to do that with the current fixes for the list curruption noted by Christoph.
Quoted 9 lines
I hope the trade-off beween the self-contained fix and the avoided memory optimization is acceptable.
Paolo Abeni (4):
mptcp: add__initannotation on setup functions
mptcp: refactor token container.
mptcp: move crypto test to KUNIT
mptcp: introduce token KUNIT self-tests
The series looks good to me. Is this something we should let bake in mptcp-next for a little bit or do you want to submit for net-next while it's still open?
I'm curious to see if there's any netdv pushback on the boot-time memory allocation.
--
Mat Martineau
Intel
Paolo Abeni
Quoted 25 lines · Mat Martineau
Hi Paolo,
Quoted 11 lines · Paolo Abeni
This series supersede the RTC patch for token refactor. The relevant patch is unchanged, but this additional bring in some related minor cleanup (patch 1/4) and KUNIT (!!!) self-tests (4/4). To make things cleaner it also moves the existing in-kernel tests (crypto) to KUNIT.
The most relevant changes from v1 are in patch 2/4.
This addresses most of the issues raised by Christoph, Davide and Mat, with the notable exception of shirinking the hash table bucket size - I'm unable to do that with the current fixes for the list curruption noted by Christoph.
Quoted 9 lines
I hope the trade-off beween the self-contained fix and the avoided memory optimization is acceptable.
Paolo Abeni (4):
mptcp: add__initannotation on setup functions
mptcp: refactor token container.
mptcp: move crypto test to KUNIT
mptcp: introduce token KUNIT self-tests
Quoted 6 lines
The series looks good to me. Is this something we should let bake in mptcp-next for a little bit or do you want to submit for net-next while it's still open?
I'm curious to see if there's any netdv pushback on the boot-time memory allocation.
I hope should not be problematic, but only the submission could prove it ;)
My idea would be staging this thing a bit in the export branch to let syzkaller and internal usage find the most gross mistakes. I'm open to other options ;)
Thanks!
Paolo