Skip to contentSkip to conversation

net/crypto: Introduce crypto_pool

PATCH In review Dmitry Safonov 6 participants 16 messages Jul 26, 2022 – Aug 23, 2022

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover net/crypto: Introduce crypto_pool

Add crypto_pool - an API for allocating per-CPU array of crypto requests on slow-path (in sleep'able context) and to use them on a fast-path, which is RX/TX for net/ users (or in any other bh-disabled users). The design is based on the current implementations of md5sig_pool.

Previously, I've suggested to add such API on TCP-AO patch submission [1], where Herbert kindly suggested to help with introducing new crypto API.

New API will allow:
- to reuse per-CPU ahash_request(s) for different users
- to allocate only one per-CPU scratch buffer rather than a new one for each user
- to have a common API for net/ users that need ahash on RX/TX fast path

In this version I've wired up TCP-MD5 and IPv6-SR-HMAC as users. Potentially, xfrm_ipcomp and xfrm_ah can be converted as well. The initial reason for patches would be to have TCP-AO as a user, which would let it share per-CPU crypto_request for any supported hashing algorithm.

While at it, I've also made TCP-MD5 static key dynamically switchable. This means that after TCP-MD5 was used and the last key got destroyed, the static branch is disabled and any potential penalty for checking tcp_md5sig_info is gone, and the system's tcp performance should be as if it never had TCP-MD5 key defined.

[1]: http://lkml.kernel.org/r/[email protected]

Cc Andy Lutomirski <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc David Ahern <[email protected]>
Cc "David S. Miller" <[email protected]>
Cc Dmitry Safonov <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Eric Dumazet <[email protected]>
Cc Francesco Ruggeri <[email protected]>
Cc Herbert Xu <[email protected]>
Cc Hideaki YOSHIFUJI <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Leonard Crestez <[email protected]>
Cc Paolo Abeni <[email protected]>
Cc Salam Noureddine <[email protected]>

Dmitry Safonov (6):
crypto: Introduce crypto_pool
crypto_pool: Add crypto_pool_reserve_scratch() net/tcp: Separate tcp_md5sig_info allocation into

    tcp_md5sig_info_add()

net/tcp: Disable TCP-MD5 static key on tcp_md5sig_info destruction net/tcp: Use crypto_pool for TCP-MD5
net/ipv6: sr: Switch to using crypto_pool

 crypto/Kconfig           |  12 ++
 crypto/Makefile          |   1 +
 crypto/crypto_pool.c     | 316 +++++++++++++++++++++++++++++++++++++++
 include/crypto/pool.h    |  33 ++++
 include/net/seg6_hmac.h  |   7 -
 include/net/tcp.h        |  32 ++--
 net/ipv4/Kconfig         |   2 +-
 net/ipv4/tcp.c           | 102 ++++---------
 net/ipv4/tcp_ipv4.c      | 153 ++++++++++++-------
 net/ipv4/tcp_minisocks.c |  23 ++-
 net/ipv4/tcp_output.c    |   4 +-
 net/ipv6/Kconfig         |   2 +-
 net/ipv6/seg6.c          |   3 -
 net/ipv6/seg6_hmac.c     | 204 ++++++++++---------------
 net/ipv6/tcp_ipv6.c      |  63 ++++----
 15 files changed, 634 insertions(+), 323 deletions(-)
 create mode 100644 crypto/crypto_pool.c
 create mode 100644 include/crypto/pool.h

base-commit: 058affafc65a74cf54499fb578b66ad0b18f939b

Signature · 1 line
2.36.1
#1 lore raw

PATCH 1/6 crypto: Introduce crypto_pool

Introduce a per-CPU pool of async crypto requests that can be used in bh-disabled contexts (designed with net RX/TX softirqs as users in mind). Allocation can sleep and is a slow-path.
Initial implementation has only ahash as a backend and a fix-sized array of possible algorithms used in parallel.

---
 crypto/Kconfig        |   6 +
 crypto/Makefile       |   1 +
 crypto/crypto_pool.c  | 287 ++++++++++++++++++++++++++++++++++++++++++
 include/crypto/pool.h |  34 +++++
 4 files changed, 328 insertions(+)
 create mode 100644 crypto/crypto_pool.c
 create mode 100644 include/crypto/pool.h
crypto/Kconfig +6 -0
index bb427a835e44..aeddaa3dcc77 100644
@@ -2128,6 +2128,12 @@ config CRYPTO_STATS
 config CRYPTO_HASH_INFO
 	bool
 
+config CRYPTO_POOL
+	tristate "Per-CPU crypto pool"
+	default n
+	help
+	  Per-CPU pool of crypto requests ready for usage in atomic contexts.
+
 source "drivers/crypto/Kconfig"
 source "crypto/asymmetric_keys/Kconfig"
 source "certs/Kconfig"
crypto/Makefile +1 -0
index 167c004dbf4f..6d1d9801b76b 100644
@@ -63,6 +63,7 @@ obj-$(CONFIG_CRYPTO_ACOMP2) += crypto_acompress.o
 cryptomgr-y := algboss.o testmgr.o
 
 obj-$(CONFIG_CRYPTO_MANAGER2) += cryptomgr.o
+obj-$(CONFIG_CRYPTO_POOL) += crypto_pool.o
 obj-$(CONFIG_CRYPTO_USER) += crypto_user.o
 crypto_user-y := crypto_user_base.o
 crypto_user-$(CONFIG_CRYPTO_STATS) += crypto_user_stat.o
crypto/crypto_pool.c new file +287 -0

287 lines not on this page · raw message

include/crypto/pool.h new file +34 -0
new file mode 100644
index 000000000000..2c61aa45faff
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef _CRYPTO_POOL_H
+#define _CRYPTO_POOL_H
+
+#include <crypto/hash.h>
+
+#define DEFAULT_CRYPTO_POOL_SCRATCH_SZ	128
+
+struct crypto_pool {
+	void *scratch;
+};
+
+/*
+ * struct crypto_pool_ahash - per-CPU pool of ahash_requests
+ * @base: common members that can be used by any async crypto ops
+ * @req: pre-allocated ahash request
+ */
+struct crypto_pool_ahash {
+	struct crypto_pool base;
+	struct ahash_request *req;
+};
+
+int crypto_pool_alloc_ahash(const char *alg);
+void crypto_pool_add(unsigned int id);
+void crypto_pool_release(unsigned int id);
+
+int crypto_pool_get(unsigned int id, struct crypto_pool *c);
+static inline void crypto_pool_put(void)
+{
+	local_bh_enable();
+}
+size_t crypto_pool_algo(unsigned int id, char *buf, size_t buf_len);
+
+#endif /* _CRYPTO_POOL_H */
Signature · 1 line
2.36.1
#2 lore raw

PATCH 2/6 crypto_pool: Add crypto_pool_reserve_scratch()

In reply to Dmitry Safonov

Instead of having build-time hardcoded constant, reallocate scratch area, if needed by user. Different algos, different users may need different size of temp per-CPU buffer. Only up-sizing supported for simplicity.

---
 crypto/Kconfig        |  6 ++++
 crypto/crypto_pool.c  | 65 +++++++++++++++++++++++++++++++------------
 include/crypto/pool.h |  3 +-
 3 files changed, 54 insertions(+), 20 deletions(-)
crypto/Kconfig +6 -0
index aeddaa3dcc77..e5865be483be 100644
@@ -2134,6 +2134,12 @@ config CRYPTO_POOL
 	help
 	  Per-CPU pool of crypto requests ready for usage in atomic contexts.
 
+config CRYPTO_POOL_DEFAULT_SCRATCH_SIZE
+	hex "Per-CPU default scratch area size"
+	depends on CRYPTO_POOL
+	default 0x100
+	range 0x100 0x10000
+
 source "drivers/crypto/Kconfig"
 source "crypto/asymmetric_keys/Kconfig"
 source "certs/Kconfig"
crypto/crypto_pool.c +47 -18

98 lines not on this page · raw message

include/crypto/pool.h +1 -2
index 2c61aa45faff..c7d817860cc3 100644
@@ -4,8 +4,6 @@
 
 #include <crypto/hash.h>
 
-#define DEFAULT_CRYPTO_POOL_SCRATCH_SZ	128
-
 struct crypto_pool {
 	void *scratch;
 };
@@ -20,6 +18,7 @@ struct crypto_pool_ahash {
 	struct ahash_request *req;
 };
 
+int crypto_pool_reserve_scratch(unsigned long size);
 int crypto_pool_alloc_ahash(const char *alg);
 void crypto_pool_add(unsigned int id);
 void crypto_pool_release(unsigned int id);
Signature · 1 line
2.36.1
#3 lore raw

PATCH 3/6 net/tcp: Separate tcp_md5sig_info allocation into tcp_md5sig_info_add()

In reply to Dmitry Safonov

Add a helper to allocate tcp_md5sig_info, that will help later to do/allocate things when info allocated, once per socket.

---
 net/ipv4/tcp_ipv4.c | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)
net/ipv4/tcp_ipv4.c +21 -9
index 0c83780dc9bf..55e4092209a5 100644
@@ -1152,6 +1152,24 @@ struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
 }
 EXPORT_SYMBOL(tcp_v4_md5_lookup);
 
+static int tcp_md5sig_info_add(struct sock *sk, gfp_t gfp)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+	struct tcp_md5sig_info *md5sig;
+
+	if (rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk)))
+		return 0;
+
+	md5sig = kmalloc(sizeof(*md5sig), gfp);
+	if (!md5sig)
+		return -ENOMEM;
+
+	sk_gso_disable(sk);
+	INIT_HLIST_HEAD(&md5sig->head);
+	rcu_assign_pointer(tp->md5sig_info, md5sig);
+	return 0;
+}
+
 /* This can be called on a newly created socket, from other files */
 int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
 		   int family, u8 prefixlen, int l3index, u8 flags,
@@ -1182,17 +1200,11 @@ int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
 		return 0;
 	}
 
+	if (tcp_md5sig_info_add(sk, gfp))
+		return -ENOMEM;
+
 	md5sig = rcu_dereference_protected(tp->md5sig_info,
 					   lockdep_sock_is_held(sk));
-	if (!md5sig) {
-		md5sig = kmalloc(sizeof(*md5sig), gfp);
-		if (!md5sig)
-			return -ENOMEM;
-
-		sk_gso_disable(sk);
-		INIT_HLIST_HEAD(&md5sig->head);
-		rcu_assign_pointer(tp->md5sig_info, md5sig);
-	}
 
 	key = sock_kmalloc(sk, sizeof(*key), gfp | __GFP_ZERO);
 	if (!key)
Signature · 1 line
2.36.1
#4 lore raw

PATCH 4/6 net/tcp: Disable TCP-MD5 static key on tcp_md5sig_info destruction

In reply to Dmitry Safonov

To do that, separate two scenarios:
- where it's the first MD5 key on the system, which means that enabling of the static key may need to sleep;
- copying of an existing key from a listening socket to the request socket upon receiving a signed TCP segment, where static key was already enabled (when the key was added to the listening socket).

Now the life-time of the static branch for TCP-MD5 is until:
- last tcp_md5sig_info is destroyed
- last socket in time-wait state with MD5 key is closed.

Which means that after all sockets with TCP-MD5 keys are gone, the system gets back the performance of disabled md5-key static branch.

---
 include/net/tcp.h        | 10 ++++++---
 net/ipv4/tcp.c           |  5 +----
 net/ipv4/tcp_ipv4.c      | 45 +++++++++++++++++++++++++++++++---------
 net/ipv4/tcp_minisocks.c |  9 +++++---
 net/ipv4/tcp_output.c    |  4 ++--
 net/ipv6/tcp_ipv6.c      | 10 ++++-----
 6 files changed, 55 insertions(+), 28 deletions(-)
include/net/tcp.h +7 -3
index f366626cbbba..aa735f963723 100644
@@ -1663,7 +1663,11 @@ int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
 			const struct sock *sk, const struct sk_buff *skb);
 int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
 		   int family, u8 prefixlen, int l3index, u8 flags,
-		   const u8 *newkey, u8 newkeylen, gfp_t gfp);
+		   const u8 *newkey, u8 newkeylen);
+int tcp_md5_key_copy(struct sock *sk, const union tcp_md5_addr *addr,
+		     int family, u8 prefixlen, int l3index,
+		     struct tcp_md5sig_key *key);
+
 int tcp_md5_do_del(struct sock *sk, const union tcp_md5_addr *addr,
 		   int family, u8 prefixlen, int l3index, u8 flags);
 struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
@@ -1671,7 +1675,7 @@ struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
 
 #ifdef CONFIG_TCP_MD5SIG
 #include <linux/jump_label.h>
-extern struct static_key_false tcp_md5_needed;
+extern struct static_key_false_deferred tcp_md5_needed;
 struct tcp_md5sig_key *__tcp_md5_do_lookup(const struct sock *sk, int l3index,
 					   const union tcp_md5_addr *addr,
 					   int family);
@@ -1679,7 +1683,7 @@ static inline struct tcp_md5sig_key *
 tcp_md5_do_lookup(const struct sock *sk, int l3index,
 		  const union tcp_md5_addr *addr, int family)
 {
-	if (!static_branch_unlikely(&tcp_md5_needed))
+	if (!static_branch_unlikely(&tcp_md5_needed.key))
 		return NULL;
 	return __tcp_md5_do_lookup(sk, l3index, addr, family);
 }
net/ipv4/tcp.c +1 -4
index 635c6782cdbb..300056008e90 100644
@@ -4404,11 +4404,8 @@ bool tcp_alloc_md5sig_pool(void)
 	if (unlikely(!tcp_md5sig_pool_populated)) {
 		mutex_lock(&tcp_md5sig_mutex);
 
-		if (!tcp_md5sig_pool_populated) {
+		if (!tcp_md5sig_pool_populated)
 			__tcp_alloc_md5sig_pool();
-			if (tcp_md5sig_pool_populated)
-				static_branch_inc(&tcp_md5_needed);
-		}
 
 		mutex_unlock(&tcp_md5sig_mutex);
 	}
net/ipv4/tcp_ipv4.c +35 -10

89 lines not on this page · raw message

net/ipv4/tcp_minisocks.c +6 -3
index cb95d88497ae..5d475a45a478 100644
@@ -291,13 +291,14 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 		 */
 		do {
 			tcptw->tw_md5_key = NULL;
-			if (static_branch_unlikely(&tcp_md5_needed)) {
+			if (static_branch_unlikely(&tcp_md5_needed.key)) {
 				struct tcp_md5sig_key *key;
 
 				key = tp->af_specific->md5_lookup(sk, sk);
 				if (key) {
 					tcptw->tw_md5_key = kmemdup(key, sizeof(*key), GFP_ATOMIC);
 					BUG_ON(tcptw->tw_md5_key && !tcp_alloc_md5sig_pool());
+					atomic_inc(&tcp_md5_needed.key.key.enabled);
 				}
 			}
 		} while (0);
@@ -337,11 +338,13 @@ EXPORT_SYMBOL(tcp_time_wait);
 void tcp_twsk_destructor(struct sock *sk)
 {
 #ifdef CONFIG_TCP_MD5SIG
-	if (static_branch_unlikely(&tcp_md5_needed)) {
+	if (static_branch_unlikely(&tcp_md5_needed.key)) {
 		struct tcp_timewait_sock *twsk = tcp_twsk(sk);
 
-		if (twsk->tw_md5_key)
+		if (twsk->tw_md5_key) {
 			kfree_rcu(twsk->tw_md5_key, rcu);
+			static_branch_slow_dec_deferred(&tcp_md5_needed);
+		}
 	}
 #endif
 }
net/ipv4/tcp_output.c +2 -2
index 78b654ff421b..9e12845a8758 100644
@@ -766,7 +766,7 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	*md5 = NULL;
 #ifdef CONFIG_TCP_MD5SIG
-	if (static_branch_unlikely(&tcp_md5_needed) &&
+	if (static_branch_unlikely(&tcp_md5_needed.key) &&
 	    rcu_access_pointer(tp->md5sig_info)) {
 		*md5 = tp->af_specific->md5_lookup(sk, sk);
 		if (*md5) {
@@ -922,7 +922,7 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 
 	*md5 = NULL;
 #ifdef CONFIG_TCP_MD5SIG
-	if (static_branch_unlikely(&tcp_md5_needed) &&
+	if (static_branch_unlikely(&tcp_md5_needed.key) &&
 	    rcu_access_pointer(tp->md5sig_info)) {
 		*md5 = tp->af_specific->md5_lookup(sk, sk);
 		if (*md5) {
net/ipv6/tcp_ipv6.c +4 -6
index e54eee80ce5f..cb891a71db0d 100644
@@ -658,12 +658,11 @@ static int tcp_v6_parse_md5_keys(struct sock *sk, int optname,
 	if (ipv6_addr_v4mapped(&sin6->sin6_addr))
 		return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin6->sin6_addr.s6_addr32[3],
 				      AF_INET, prefixlen, l3index, flags,
-				      cmd.tcpm_key, cmd.tcpm_keylen,
-				      GFP_KERNEL);
+				      cmd.tcpm_key, cmd.tcpm_keylen);
 
 	return tcp_md5_do_add(sk, (union tcp_md5_addr *)&sin6->sin6_addr,
 			      AF_INET6, prefixlen, l3index, flags,
-			      cmd.tcpm_key, cmd.tcpm_keylen, GFP_KERNEL);
+			      cmd.tcpm_key, cmd.tcpm_keylen);
 }
 
 static int tcp_v6_md5_hash_headers(struct tcp_md5sig_pool *hp,
@@ -1359,9 +1358,8 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
 		 * memory, then we end up not copying the key
 		 * across. Shucks.
 		 */
-		tcp_md5_do_add(newsk, (union tcp_md5_addr *)&newsk->sk_v6_daddr,
-			       AF_INET6, 128, l3index, key->flags, key->key, key->keylen,
-			       sk_gfp_mask(sk, GFP_ATOMIC));
+		tcp_md5_key_copy(newsk, (union tcp_md5_addr *)&newsk->sk_v6_daddr,
+			       AF_INET6, 128, l3index, key);
 	}
 #endif
 
Signature · 1 line
2.36.1
#5 lore raw

Re: [PATCH 4/6] net/tcp: Disable TCP-MD5 static key on tcp_md5sig_info destruction

Thank you for the patch! Perhaps something to improve:

Hi Dmitry,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on 058affafc65a74cf54499fb578b66ad0b18f939b]

url:    https://github.com/intel-lab-lkp/linux/commits/Dmitry-Safonov/net-crypto-Introduce-crypto_pool/20220727-041830
base:   058affafc65a74cf54499fb578b66ad0b18f939b
config: x86_64-defconfig (https://download.01.org/0day-ci/archive/20220814/[email protected]/config)
compiler: gcc-11 (Debian 11.3.0-3) 11.3.0
reproduce (this is a W=1 build):
        # https://github.com/intel-lab-lkp/linux/commit/a4ee3ecdaada036ed6747ed86eaf7270d3f27bab
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Dmitry-Safonov/net-crypto-Introduce-crypto_pool/20220727-041830
        git checkout a4ee3ecdaada036ed6747ed86eaf7270d3f27bab
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        make W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash net/ipv4/

If you fix the issue, kindly add following tag where applicable

All warnings (new ones prefixed by >>):

>> net/ipv4/tcp_ipv4.c:1174:5: warning: no previous prototype for '__tcp_md5_do_add' [-Wmissing-prototypes]

1174 | int __tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
| ^~~~~~~~~~~~~~~~

vim +/__tcp_md5_do_add +1174 net/ipv4/tcp_ipv4.c

1172
1173 /* This can be called on a newly created socket, from other files */

> 1174	int __tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,

1175 int family, u8 prefixlen, int l3index, u8 flags,
1176 const u8 *newkey, u8 newkeylen, gfp_t gfp)
1177 {
1178 /* Add Key to the list */
1179 struct tcp_md5sig_key *key;
1180 struct tcp_sock *tp = tcp_sk(sk);
1181 struct tcp_md5sig_info *md5sig;
1182
1183 key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen, l3index, flags);
1184 if (key) {
1185 /* Pre-existing entry - just update that one.
1186 * Note that the key might be used concurrently.
1187 * data_race() is telling kcsan that we do not care of
1188 * key mismatches, since changing MD5 key on live flows
1189 * can lead to packet drops.
1190 */
1191 data_race(memcpy(key->key, newkey, newkeylen));
1192
1193 /* Pairs with READ_ONCE() in tcp_md5_hash_key().
1194 * Also note that a reader could catch new key->keylen value
1195 * but old key->key[], this is the reason we use __GFP_ZERO
1196 * at sock_kmalloc() time below these lines.
1197 */
1198 WRITE_ONCE(key->keylen, newkeylen);
1199
1200 return 0;
1201 }
1202
1203 md5sig = rcu_dereference_protected(tp->md5sig_info,
1204 lockdep_sock_is_held(sk));
1205
1206 key = sock_kmalloc(sk, sizeof(*key), gfp | __GFP_ZERO);
1207 if (!key)
1208 return -ENOMEM;
1209 if (!tcp_alloc_md5sig_pool()) {
1210 sock_kfree_s(sk, key, sizeof(*key));
1211 return -ENOMEM;
1212 }
1213
1214 memcpy(key->key, newkey, newkeylen);
1215 key->keylen = newkeylen;
1216 key->family = family;
1217 key->prefixlen = prefixlen;
1218 key->l3index = l3index;
1219 key->flags = flags;
1220 memcpy(&key->addr, addr,
1221 (IS_ENABLED(CONFIG_IPV6) && family == AF_INET6) ? sizeof(struct in6_addr) :
1222 sizeof(struct in_addr));
1223 hlist_add_head_rcu(&key->node, &md5sig->head);
1224 return 0;
1225 }
1226

Signature · 2 lines
0-DAY CI Kernel Test Service
https://01.org/lkp
#6 lore raw

Re: [PATCH 4/6] net/tcp: Disable TCP-MD5 static key on tcp_md5sig_info destruction

In reply to Dmitry Safonov
Thank you for the patch! Perhaps something to improve:

Hi Dmitry,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on 058affafc65a74cf54499fb578b66ad0b18f939b]

url:    https://github.com/intel-lab-lkp/linux/commits/Dmitry-Safonov/net-crypto-Introduce-crypto_pool/20220727-041830
base:   058affafc65a74cf54499fb578b66ad0b18f939b
config: i386-randconfig-a005 (https://download.01.org/0day-ci/archive/20220814/[email protected]/config)
compiler: gcc-11 (Debian 11.3.0-3) 11.3.0
reproduce (this is a W=1 build):
        # https://github.com/intel-lab-lkp/linux/commit/a4ee3ecdaada036ed6747ed86eaf7270d3f27bab
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Dmitry-Safonov/net-crypto-Introduce-crypto_pool/20220727-041830
        git checkout a4ee3ecdaada036ed6747ed86eaf7270d3f27bab
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash net/ipv4/

If you fix the issue, kindly add following tag where applicable

All warnings (new ones prefixed by >>):

>> net/ipv4/tcp_ipv4.c:1174:5: warning: no previous prototype for '__tcp_md5_do_add' [-Wmissing-prototypes]

1174 | int __tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
| ^~~~~~~~~~~~~~~~

vim +/__tcp_md5_do_add +1174 net/ipv4/tcp_ipv4.c

1172
1173 /* This can be called on a newly created socket, from other files */

> 1174	int __tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,

1175 int family, u8 prefixlen, int l3index, u8 flags,
1176 const u8 *newkey, u8 newkeylen, gfp_t gfp)
1177 {
1178 /* Add Key to the list */
1179 struct tcp_md5sig_key *key;
1180 struct tcp_sock *tp = tcp_sk(sk);
1181 struct tcp_md5sig_info *md5sig;
1182
1183 key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen, l3index, flags);
1184 if (key) {
1185 /* Pre-existing entry - just update that one.
1186 * Note that the key might be used concurrently.
1187 * data_race() is telling kcsan that we do not care of
1188 * key mismatches, since changing MD5 key on live flows
1189 * can lead to packet drops.
1190 */
1191 data_race(memcpy(key->key, newkey, newkeylen));
1192
1193 /* Pairs with READ_ONCE() in tcp_md5_hash_key().
1194 * Also note that a reader could catch new key->keylen value
1195 * but old key->key[], this is the reason we use __GFP_ZERO
1196 * at sock_kmalloc() time below these lines.
1197 */
1198 WRITE_ONCE(key->keylen, newkeylen);
1199
1200 return 0;
1201 }
1202
1203 md5sig = rcu_dereference_protected(tp->md5sig_info,
1204 lockdep_sock_is_held(sk));
1205
1206 key = sock_kmalloc(sk, sizeof(*key), gfp | __GFP_ZERO);
1207 if (!key)
1208 return -ENOMEM;
1209 if (!tcp_alloc_md5sig_pool()) {
1210 sock_kfree_s(sk, key, sizeof(*key));
1211 return -ENOMEM;
1212 }
1213
1214 memcpy(key->key, newkey, newkeylen);
1215 key->keylen = newkeylen;
1216 key->family = family;
1217 key->prefixlen = prefixlen;
1218 key->l3index = l3index;
1219 key->flags = flags;
1220 memcpy(&key->addr, addr,
1221 (IS_ENABLED(CONFIG_IPV6) && family == AF_INET6) ? sizeof(struct in6_addr) :
1222 sizeof(struct in_addr));
1223 hlist_add_head_rcu(&key->node, &md5sig->head);
1224 return 0;
1225 }
1226

Signature · 2 lines
0-DAY CI Kernel Test Service
https://01.org/lkp
#7 lore raw

PATCH 5/6 net/tcp: Use crypto_pool for TCP-MD5

In reply to Dmitry Safonov

Use crypto_pool API that was designed with tcp_md5sig_pool in mind. The conversion to use crypto_pool will allow:
- to reuse ahash_request(s) for different users
- to allocate only one per-CPU scratch buffer rather than a new one for each user
- to have a common API for net/ users that need ahash on RX/TX fast path

---
 include/net/tcp.h        | 22 +++------
 net/ipv4/Kconfig         |  2 +-
 net/ipv4/tcp.c           | 99 +++++++++++-----------------------------
 net/ipv4/tcp_ipv4.c      | 90 +++++++++++++++++++++---------------
 net/ipv4/tcp_minisocks.c | 22 +++++++--
 net/ipv6/tcp_ipv6.c      | 53 ++++++++++-----------
 6 files changed, 127 insertions(+), 161 deletions(-)
include/net/tcp.h +7 -15
index aa735f963723..7060dd1cf6cd 100644
@@ -1652,12 +1652,6 @@ union tcp_md5sum_block {
 #endif
 };
 
-/* - pool: digest algorithm, hash description and scratch buffer */
-struct tcp_md5sig_pool {
-	struct ahash_request	*md5_req;
-	void			*scratch;
-};
-
 /* - functions */
 int tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
 			const struct sock *sk, const struct sk_buff *skb);
@@ -1713,17 +1707,15 @@ tcp_inbound_md5_hash(const struct sock *sk, const struct sk_buff *skb,
 #define tcp_twsk_md5_key(twsk)	NULL
 #endif
 
-bool tcp_alloc_md5sig_pool(void);
-
-struct tcp_md5sig_pool *tcp_get_md5sig_pool(void);
-static inline void tcp_put_md5sig_pool(void)
-{
-	local_bh_enable();
-}
+struct crypto_pool_ahash;
+int tcp_md5_alloc_crypto_pool(void);
+void tcp_md5_release_crypto_pool(void);
+void tcp_md5_add_crypto_pool(void);
+extern int tcp_md5_crypto_pool_id;
 
-int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *, const struct sk_buff *,
+int tcp_md5_hash_skb_data(struct crypto_pool_ahash *, const struct sk_buff *,
 			  unsigned int header_len);
-int tcp_md5_hash_key(struct tcp_md5sig_pool *hp,
+int tcp_md5_hash_key(struct crypto_pool_ahash *hp,
 		     const struct tcp_md5sig_key *key);
 
 /* From tcp_fastopen.c */
net/ipv4/Kconfig +1 -1
index e983bb0c5012..c341864e4398 100644
@@ -733,7 +733,7 @@ config DEFAULT_TCP_CONG
 
 config TCP_MD5SIG
 	bool "TCP: MD5 Signature Option support (RFC2385)"
-	select CRYPTO
+	select CRYPTO_POOL
 	select CRYPTO_MD5
 	help
 	  RFC2385 specifies a method of giving MD5 protection to TCP sessions.
net/ipv4/tcp.c +26 -73

140 lines not on this page · raw message

net/ipv4/tcp_ipv4.c +52 -38

188 lines not on this page · raw message

net/ipv4/tcp_minisocks.c +17 -5
index 5d475a45a478..d1d30337ffec 100644
@@ -297,8 +297,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 				key = tp->af_specific->md5_lookup(sk, sk);
 				if (key) {
 					tcptw->tw_md5_key = kmemdup(key, sizeof(*key), GFP_ATOMIC);
-					BUG_ON(tcptw->tw_md5_key && !tcp_alloc_md5sig_pool());
+					if (WARN_ON(!tcptw->tw_md5_key))
+						break;
 					atomic_inc(&tcp_md5_needed.key.key.enabled);
+					tcp_md5_add_crypto_pool();
 				}
 			}
 		} while (0);
@@ -335,16 +337,26 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)
 }
 EXPORT_SYMBOL(tcp_time_wait);
 
+#ifdef CONFIG_TCP_MD5SIG
+static void tcp_md5_twsk_free_rcu(struct rcu_head *head)
+{
+	struct tcp_md5sig_key *key;
+
+	key = container_of(head, struct tcp_md5sig_key, rcu);
+	kfree(key);
+	static_branch_slow_dec_deferred(&tcp_md5_needed);
+	tcp_md5_release_crypto_pool();
+}
+#endif
+
 void tcp_twsk_destructor(struct sock *sk)
 {
 #ifdef CONFIG_TCP_MD5SIG
 	if (static_branch_unlikely(&tcp_md5_needed.key)) {
 		struct tcp_timewait_sock *twsk = tcp_twsk(sk);
 
-		if (twsk->tw_md5_key) {
-			kfree_rcu(twsk->tw_md5_key, rcu);
-			static_branch_slow_dec_deferred(&tcp_md5_needed);
-		}
+		if (twsk->tw_md5_key)
+			call_rcu(&twsk->tw_md5_key->rcu, tcp_md5_twsk_free_rcu);
 	}
 #endif
 }
net/ipv6/tcp_ipv6.c +24 -29

119 lines not on this page · raw message

Signature · 1 line
2.36.1
#8 lore raw

PATCH 6/6 net/ipv6: sr: Switch to using crypto_pool

In reply to Dmitry Safonov

The conversion to use crypto_pool has the following upsides:
- now SR uses asynchronous API which may potentially free CPU cycles and improve performance for of CPU crypto algorithm providers;
- hash descriptors now don't have to be allocated on boot, but only at the moment SR starts using HMAC and until the last HMAC secret is deleted;
- potentially reuse ahash_request(s) for different users
- allocate only one per-CPU scratch buffer rather than a new one for each user
- have a common API for net/ users that need ahash on RX/TX fast path

---
 include/net/seg6_hmac.h |   7 --
 net/ipv6/Kconfig        |   2 +-
 net/ipv6/seg6.c         |   3 -
 net/ipv6/seg6_hmac.c    | 204 ++++++++++++++++------------------------
 4 files changed, 80 insertions(+), 136 deletions(-)
include/net/seg6_hmac.h +0 -7
index 2b5d2ee5613e..d6b7820ecda2 100644
@@ -32,13 +32,6 @@ struct seg6_hmac_info {
 	u8 alg_id;
 };
 
-struct seg6_hmac_algo {
-	u8 alg_id;
-	char name[64];
-	struct crypto_shash * __percpu *tfms;
-	struct shash_desc * __percpu *shashs;
-};
-
 extern int seg6_hmac_compute(struct seg6_hmac_info *hinfo,
 			     struct ipv6_sr_hdr *hdr, struct in6_addr *saddr,
 			     u8 *output);
net/ipv6/Kconfig +1 -1
index 658bfed1df8b..5be1dab0f178 100644
@@ -304,7 +304,7 @@ config IPV6_SEG6_LWTUNNEL
 config IPV6_SEG6_HMAC
 	bool "IPv6: Segment Routing HMAC support"
 	depends on IPV6
-	select CRYPTO
+	select CRYPTO_POOL
 	select CRYPTO_HMAC
 	select CRYPTO_SHA1
 	select CRYPTO_SHA256
net/ipv6/seg6.c +0 -3
index 73aaabf0e966..96b80e1d04c1 100644
@@ -552,9 +552,6 @@ int __init seg6_init(void)
 
 void seg6_exit(void)
 {
-#ifdef CONFIG_IPV6_SEG6_HMAC
-	seg6_hmac_exit();
-#endif
 #ifdef CONFIG_IPV6_SEG6_LWTUNNEL
 	seg6_iptunnel_exit();
 #endif
net/ipv6/seg6_hmac.c +79 -125

290 lines not on this page · raw message

Signature · 1 line
2.36.1
maintainer
#9 lore raw

Re: [PATCH 6/6] net/ipv6: sr: Switch to using crypto_pool

Quoted 10 lines · On Tue, 26 Jul 2022 21:16:00 +0100 Dmitry Safonov wrote:

The conversion to use crypto_pool has the following upsides:
- now SR uses asynchronous API which may potentially free CPU cycles and improve performance for of CPU crypto algorithm providers;
- hash descriptors now don't have to be allocated on boot, but only at the moment SR starts using HMAC and until the last HMAC secret is deleted;
- potentially reuse ahash_request(s) for different users
- allocate only one per-CPU scratch buffer rather than a new one for each user
- have a common API for net/ users that need ahash on RX/TX fast path

Build problems on allmodconfig:

ERROR: modpost: "crypto_pool_reserve_scratch" [net/ipv6/ipv6.ko] undefined!

#10 lore raw

Re: [PATCH 6/6] net/ipv6: sr: Switch to using crypto_pool

In reply to Dmitry Safonov

On 2022/7/27 4:16, Dmitry Safonov Wrote:

Quoted 3 lines

The conversion to use crypto_pool has the following upsides:
- now SR uses asynchronous API which may potentially free CPU cycles and improve performance for of CPU crypto algorithm providers;

Is there any specific test data for this performance improvement? How much has the memory increased?
How much has the performance improved?
Thanks
Longfang.

Quoted 367 lines

- hash descriptors now don't have to be allocated on boot, but only at the moment SR starts using HMAC and until the last HMAC secret is deleted;
- potentially reuse ahash_request(s) for different users
- allocate only one per-CPU scratch buffer rather than a new one for each user
- have a common API for net/ users that need ahash on RX/TX fast path

---
 include/net/seg6_hmac.h |   7 --
 net/ipv6/Kconfig        |   2 +-
 net/ipv6/seg6.c         |   3 -
 net/ipv6/seg6_hmac.c    | 204 ++++++++++++++++------------------------
 4 files changed, 80 insertions(+), 136 deletions(-)
include/net/seg6_hmac.h +0 -7
index 2b5d2ee5613e..d6b7820ecda2 100644
@@ -32,13 +32,6 @@ struct seg6_hmac_info {
 	u8 alg_id;
 };
 
-struct seg6_hmac_algo {
-	u8 alg_id;
-	char name[64];
-	struct crypto_shash * __percpu *tfms;
-	struct shash_desc * __percpu *shashs;
-};
-
 extern int seg6_hmac_compute(struct seg6_hmac_info *hinfo,
 			     struct ipv6_sr_hdr *hdr, struct in6_addr *saddr,
 			     u8 *output);
net/ipv6/Kconfig +1 -1
index 658bfed1df8b..5be1dab0f178 100644
@@ -304,7 +304,7 @@ config IPV6_SEG6_LWTUNNEL
 config IPV6_SEG6_HMAC
 	bool "IPv6: Segment Routing HMAC support"
 	depends on IPV6
-	select CRYPTO
+	select CRYPTO_POOL
 	select CRYPTO_HMAC
 	select CRYPTO_SHA1
 	select CRYPTO_SHA256
net/ipv6/seg6.c +0 -3
index 73aaabf0e966..96b80e1d04c1 100644
@@ -552,9 +552,6 @@ int __init seg6_init(void)
 
 void seg6_exit(void)
 {
-#ifdef CONFIG_IPV6_SEG6_HMAC
-	seg6_hmac_exit();
-#endif
 #ifdef CONFIG_IPV6_SEG6_LWTUNNEL
 	seg6_iptunnel_exit();
 #endif
net/ipv6/seg6_hmac.c +79 -125
index d43c50a7310d..3732dd993925 100644
@@ -35,6 +35,7 @@
 #include <net/xfrm.h>
 
 #include <crypto/hash.h>
+#include <crypto/pool.h>
 #include <net/seg6.h>
 #include <net/genetlink.h>
 #include <net/seg6_hmac.h>
@@ -70,6 +71,12 @@ static const struct rhashtable_params rht_params = {
 	.obj_cmpfn		= seg6_hmac_cmpfn,
 };
 
+struct seg6_hmac_algo {
+	u8 alg_id;
+	char name[64];
+	int crypto_pool_id;
+};
+
 static struct seg6_hmac_algo hmac_algos[] = {
 	{
 		.alg_id = SEG6_HMAC_ALGO_SHA1,
@@ -115,55 +122,17 @@ static struct seg6_hmac_algo *__hmac_get_algo(u8 alg_id)
 	return NULL;
 }
 
-static int __do_hmac(struct seg6_hmac_info *hinfo, const char *text, u8 psize,
-		     u8 *output, int outlen)
-{
-	struct seg6_hmac_algo *algo;
-	struct crypto_shash *tfm;
-	struct shash_desc *shash;
-	int ret, dgsize;
-
-	algo = __hmac_get_algo(hinfo->alg_id);
-	if (!algo)
-		return -ENOENT;
-
-	tfm = *this_cpu_ptr(algo->tfms);
-
-	dgsize = crypto_shash_digestsize(tfm);
-	if (dgsize > outlen) {
-		pr_debug("sr-ipv6: __do_hmac: digest size too big (%d / %d)\n",
-			 dgsize, outlen);
-		return -ENOMEM;
-	}
-
-	ret = crypto_shash_setkey(tfm, hinfo->secret, hinfo->slen);
-	if (ret < 0) {
-		pr_debug("sr-ipv6: crypto_shash_setkey failed: err %d\n", ret);
-		goto failed;
-	}
-
-	shash = *this_cpu_ptr(algo->shashs);
-	shash->tfm = tfm;
-
-	ret = crypto_shash_digest(shash, text, psize, output);
-	if (ret < 0) {
-		pr_debug("sr-ipv6: crypto_shash_digest failed: err %d\n", ret);
-		goto failed;
-	}
-
-	return dgsize;
-
-failed:
-	return ret;
-}
-
 int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr,
 		      struct in6_addr *saddr, u8 *output)
 {
 	__be32 hmackeyid = cpu_to_be32(hinfo->hmackeyid);
-	u8 tmp_out[SEG6_HMAC_MAX_DIGESTSIZE];
+	struct crypto_pool_ahash hp;
+	struct seg6_hmac_algo *algo;
 	int plen, i, dgsize, wrsize;
+	struct crypto_ahash *tfm;
+	struct scatterlist sg;
 	char *ring, *off;
+	int err;
 
 	/* a 160-byte buffer for digest output allows to store highest known
 	 * hash function (RadioGatun) with up to 1216 bits
@@ -176,6 +145,10 @@ int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr,
 	if (plen >= SEG6_HMAC_RING_SIZE)
 		return -EMSGSIZE;
 
+	algo = __hmac_get_algo(hinfo->alg_id);
+	if (!algo)
+		return -ENOENT;
+
 	/* Let's build the HMAC text on the ring buffer. The text is composed
 	 * as follows, in order:
 	 *
@@ -186,8 +159,36 @@ int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr,
 	 * 5. All segments in the segments list (n * 128 bits)
 	 */
 
-	local_bh_disable();
+	err = crypto_pool_get(algo->crypto_pool_id, (struct crypto_pool *)&hp);
+	if (err)
+		return err;
+
 	ring = this_cpu_ptr(hmac_ring);
+
+	sg_init_one(&sg, ring, plen);
+
+	tfm = crypto_ahash_reqtfm(hp.req);
+	dgsize = crypto_ahash_digestsize(tfm);
+	if (dgsize > SEG6_HMAC_MAX_DIGESTSIZE) {
+		pr_debug("digest size too big (%d / %d)\n",
+			 dgsize, SEG6_HMAC_MAX_DIGESTSIZE);
+		err = -ENOMEM;
+		goto err_put_pool;
+	}
+
+	err = crypto_ahash_setkey(tfm, hinfo->secret, hinfo->slen);
+	if (err) {
+		pr_debug("crypto_ahash_setkey failed: err %d\n", err);
+		goto err_put_pool;
+	}
+
+	err = crypto_ahash_init(hp.req);
+	if (err)
+		goto err_put_pool;
+
+	ahash_request_set_crypt(hp.req, &sg,
+				hp.base.scratch, SEG6_HMAC_MAX_DIGESTSIZE);
+
 	off = ring;
 
 	/* source address */
@@ -210,21 +211,25 @@ int seg6_hmac_compute(struct seg6_hmac_info *hinfo, struct ipv6_sr_hdr *hdr,
 		off += 16;
 	}
 
-	dgsize = __do_hmac(hinfo, ring, plen, tmp_out,
-			   SEG6_HMAC_MAX_DIGESTSIZE);
-	local_bh_enable();
+	err = crypto_ahash_update(hp.req);
+	if (err)
+		goto err_put_pool;
 
-	if (dgsize < 0)
-		return dgsize;
+	err = crypto_ahash_final(hp.req);
+	if (err)
+		goto err_put_pool;
 
 	wrsize = SEG6_HMAC_FIELD_LEN;
 	if (wrsize > dgsize)
 		wrsize = dgsize;
 
 	memset(output, 0, SEG6_HMAC_FIELD_LEN);
-	memcpy(output, tmp_out, wrsize);
+	memcpy(output, hp.base.scratch, wrsize);
 
-	return 0;
+err_put_pool:
+	crypto_pool_put();
+
+	return err;
 }
 EXPORT_SYMBOL(seg6_hmac_compute);
 
@@ -291,12 +296,24 @@ EXPORT_SYMBOL(seg6_hmac_info_lookup);
 int seg6_hmac_info_add(struct net *net, u32 key, struct seg6_hmac_info *hinfo)
 {
 	struct seg6_pernet_data *sdata = seg6_pernet(net);
-	int err;
+	struct seg6_hmac_algo *algo;
+	int ret;
 
-	err = rhashtable_lookup_insert_fast(&sdata->hmac_infos, &hinfo->node,
+	algo = __hmac_get_algo(hinfo->alg_id);
+	if (!algo)
+		return -ENOENT;
+
+	ret = crypto_pool_alloc_ahash(algo->name);
+	if (ret < 0)
+		return ret;
+	algo->crypto_pool_id = ret;
+
+	ret = rhashtable_lookup_insert_fast(&sdata->hmac_infos, &hinfo->node,
 					    rht_params);
+	if (ret)
+		crypto_pool_release(algo->crypto_pool_id);
 
-	return err;
+	return ret;
 }
 EXPORT_SYMBOL(seg6_hmac_info_add);
 
@@ -304,6 +321,7 @@ int seg6_hmac_info_del(struct net *net, u32 key)
 {
 	struct seg6_pernet_data *sdata = seg6_pernet(net);
 	struct seg6_hmac_info *hinfo;
+	struct seg6_hmac_algo *algo;
 	int err = -ENOENT;
 
 	hinfo = rhashtable_lookup_fast(&sdata->hmac_infos, &key, rht_params);
@@ -315,6 +333,12 @@ int seg6_hmac_info_del(struct net *net, u32 key)
 	if (err)
 		goto out;
 
+	algo = __hmac_get_algo(hinfo->alg_id);
+	if (algo)
+		crypto_pool_release(algo->crypto_pool_id);
+	else
+		WARN_ON_ONCE(1);
+
 	seg6_hinfo_release(hinfo);
 
 out:
@@ -348,56 +372,9 @@ int seg6_push_hmac(struct net *net, struct in6_addr *saddr,
 }
 EXPORT_SYMBOL(seg6_push_hmac);
 
-static int seg6_hmac_init_algo(void)
-{
-	struct seg6_hmac_algo *algo;
-	struct crypto_shash *tfm;
-	struct shash_desc *shash;
-	int i, alg_count, cpu;
-
-	alg_count = ARRAY_SIZE(hmac_algos);
-
-	for (i = 0; i < alg_count; i++) {
-		struct crypto_shash **p_tfm;
-		int shsize;
-
-		algo = &hmac_algos[i];
-		algo->tfms = alloc_percpu(struct crypto_shash *);
-		if (!algo->tfms)
-			return -ENOMEM;
-
-		for_each_possible_cpu(cpu) {
-			tfm = crypto_alloc_shash(algo->name, 0, 0);
-			if (IS_ERR(tfm))
-				return PTR_ERR(tfm);
-			p_tfm = per_cpu_ptr(algo->tfms, cpu);
-			*p_tfm = tfm;
-		}
-
-		p_tfm = raw_cpu_ptr(algo->tfms);
-		tfm = *p_tfm;
-
-		shsize = sizeof(*shash) + crypto_shash_descsize(tfm);
-
-		algo->shashs = alloc_percpu(struct shash_desc *);
-		if (!algo->shashs)
-			return -ENOMEM;
-
-		for_each_possible_cpu(cpu) {
-			shash = kzalloc_node(shsize, GFP_KERNEL,
-					     cpu_to_node(cpu));
-			if (!shash)
-				return -ENOMEM;
-			*per_cpu_ptr(algo->shashs, cpu) = shash;
-		}
-	}
-
-	return 0;
-}
-
 int __init seg6_hmac_init(void)
 {
-	return seg6_hmac_init_algo();
+	return crypto_pool_reserve_scratch(SEG6_HMAC_MAX_DIGESTSIZE);
 }
 
 int __net_init seg6_hmac_net_init(struct net *net)
@@ -407,29 +384,6 @@ int __net_init seg6_hmac_net_init(struct net *net)
 	return rhashtable_init(&sdata->hmac_infos, &rht_params);
 }
 
-void seg6_hmac_exit(void)
-{
-	struct seg6_hmac_algo *algo = NULL;
-	int i, alg_count, cpu;
-
-	alg_count = ARRAY_SIZE(hmac_algos);
-	for (i = 0; i < alg_count; i++) {
-		algo = &hmac_algos[i];
-		for_each_possible_cpu(cpu) {
-			struct crypto_shash *tfm;
-			struct shash_desc *shash;
-
-			shash = *per_cpu_ptr(algo->shashs, cpu);
-			kfree(shash);
-			tfm = *per_cpu_ptr(algo->tfms, cpu);
-			crypto_free_shash(tfm);
-		}
-		free_percpu(algo->tfms);
-		free_percpu(algo->shashs);
-	}
-}
-EXPORT_SYMBOL(seg6_hmac_exit);
-
 void __net_exit seg6_hmac_net_exit(struct net *net)
 {
 	struct seg6_pernet_data *sdata = seg6_pernet(net);
#11 lore raw

Re: [PATCH 6/6] net/ipv6: sr: Switch to using crypto_pool

Quoted 8 lines · liulongfang

On 2022/7/27 4:16, Dmitry Safonov Wrote:

Quoted 3 lines

The conversion to use crypto_pool has the following upsides:
- now SR uses asynchronous API which may potentially free CPU cycles and improve performance for of CPU crypto algorithm providers;

Is there any specific test data for this performance improvement? How much has the memory increased?
How much has the performance improved?

I have no test data for this.
The idea behind this improvement is the same as behind crypto ahash: if your CPU can benefit by offloading work to hw hash accelerator and there is a driver that provides crypto backend, you may certainly win some performance. The number depends on platform/driver in use.

Quoted 7 lines
Quoted 7 lines

- hash descriptors now don't have to be allocated on boot, but only at the moment SR starts using HMAC and until the last HMAC secret is deleted;
- potentially reuse ahash_request(s) for different users
- allocate only one per-CPU scratch buffer rather than a new one for each user
- have a common API for net/ users that need ahash on RX/TX fast path

[..]

Thanks,
Dmitry

maintainer
#12 lore raw
In reply to Dmitry Safonov
Quoted 7 lines · Dmitry Safonov

Add crypto_pool - an API for allocating per-CPU array of crypto requests on slow-path (in sleep'able context) and to use them on a fast-path, which is RX/TX for net/ users (or in any other bh-disabled users). The design is based on the current implementations of md5sig_pool.

Previously, I've suggested to add such API on TCP-AO patch submission [1], where Herbert kindly suggested to help with introducing new crypto API.

What I was suggesting is modifying the actual ahash interface so that the tfm can be shared between different key users by moving the key into the request object.

Thanks,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#13 lore raw
Quoted 6 lines · Herbert Xu
Quoted 5 lines · Dmitry Safonov

Add crypto_pool - an API for allocating per-CPU array of crypto requests on slow-path (in sleep'able context) and to use them on a fast-path, which is RX/TX for net/ users (or in any other bh-disabled users). The design is based on the current implementations of md5sig_pool.

Quoted 6 lines
Quoted 2 lines

Previously, I've suggested to add such API on TCP-AO patch submission [1], where Herbert kindly suggested to help with introducing new crypto API.

What I was suggesting is modifying the actual ahash interface so that the tfm can be shared between different key users by moving the key into the request object.

The fact that setkey is implemented at the crypto_ahash instead of the ahash_request level is baked into all algorithm implementations (including many hardware-specific ones). Changing this seems extremely difficult.

Supporting setkey at the tfm level could be achieved by making it an optional capability on a per-algorithm basis, then something like crypto_pool could detect this scenario and avoid allocating a per-cpu tfm. This would also require a crypto_pool_setkey wrapper.

As it stands right now multiple crypto-api users needs to duplicate logic for allocating a percpu array of transforms so adding this "pool" API is an useful step forward.

As far as I remember the requirement for a per-cpu scratch buffer is based on weird architectures having limitations on what kind of memory can be passed to crypto api so this will have to remain.

--
Regards,
Leonard

maintainer
#14 lore raw
Quoted 4 lines · Leonard Crestez

The fact that setkey is implemented at the crypto_ahash instead of the ahash_request level is baked into all algorithm implementations (including many hardware-specific ones). Changing this seems extremely difficult.

What I had in mind is simply making the tfm setkey optional. That way you could then have an additional setkey at the request level.

If the key is provided in either place you're allowed to perform the hash.

This should have minimal impact on existing code.

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#15 lore raw
In reply to Herbert Xu

Hi Herbert,

Quoted 6 lines · Herbert Xu
Quoted 5 lines · Dmitry Safonov

Add crypto_pool - an API for allocating per-CPU array of crypto requests on slow-path (in sleep'able context) and to use them on a fast-path, which is RX/TX for net/ users (or in any other bh-disabled users). The design is based on the current implementations of md5sig_pool.

Quoted 6 lines
Quoted 2 lines

Previously, I've suggested to add such API on TCP-AO patch submission [1], where Herbert kindly suggested to help with introducing new crypto API.

What I was suggesting is modifying the actual ahash interface so that the tfm can be shared between different key users by moving the key into the request object.

My impression here is that we're looking at different issues.
1. The necessity of allocating per-CPU ahash_requests.
2. Managing the lifetime and sharing of ahash_request between different kernel users.

Removing (1) will allow saving (num_possible_cpus() - 1)*(sizeof(struct ahash_request) + crypto_ahash_reqsize(tfm)) bytes. Which would be very nice for the new fancy CPUs with hundreds of threads.

For (2) many kernel users try manage it themselves, resulting in different implementations, as well as some users trying to avoid using any complication like ref counting and allocating the request only once, without freeing it until the module is unloaded. Here for example, introducing TCP-AO would result in copy'n'paste of tcp_md5sig_pool code. As well as RFC5925 for TCP-AO let user to have any supported hashing algorithms, with the requirement from RFC5926 of hmac(sha1) & aes(cmac). If a user wants more algorithms that implementation would need to be patched.

I see quite a few net/ users that could use some common API for this besides TCP-MD5 and TCP-AO. That have the same pattern of allocating crypto algorithm on a slow-path (adding a key or module initialization) and using it of a fast-path, which is RX/TX.
Besides of sharing and lifetime managing, those users need a temporary buffer (usually the name is `scratch'), IIUC, it is needed for async algorithms that could use some hardware accelerator instead of CPU and need to write the result anywhere, but on vmapped stack.

So, here I'm trying to address (2) in order to avoid copy'n'pasting of tcp_md5sig_pool code for introduction of TCP-AO support. I've also patched tcp-md5 code to dynamically disable the static branch, which is not crypto change.

There's also a chance I've misunderstood what is your proposal :-)

Thanks,
Dmitry