Multibuffer hashing take two
Subsystems (2)
Herbert Xu
maintainercover Multibuffer hashing take two
Multibuffer hashing was a constant sore while it was part of the kernel. It was very buggy and unnecessarily complex. Finally it was removed when it had been broken for a while without anyone noticing.
Peace reigned in its absence, until Eric Biggers made a proposal for its comeback :)
The issue is that the SHA algorithm (and possibly others) is inherently not parallelisable. Therefore the only way to exploit parallelism on modern CPUs is to hash multiple indendent streams of data.
Eric's proposal is a simple interface bolted onto shash that takes two streams of data of identical length. I thought the limitation of two was too small, and Eric addressed that in his latest version:
However, I still disliked the addition of this to shash as it meant that users would have to spend extra effort in order to accumulate and maintain multiple streams of data.
My preference is to use ahash as the basis of multibuffer, because its request object interface is perfectly suited to chaining.
The ahash interface is almost universally hated because of its use of the SG list. So to sweeten the deal I have added virtual address support to ahash, thus rendering the shash interface redundant.
Note that ahash can already be used synchronously by asking for sync-only algorithms. Thus there is no need to handle callbacks and such *if* you don't want to.
This patch-set introduces two additions to the ahash interface. First of all request chaining is added so that an arbitrary number of requests can be submitted in one go. Incidentally this also reduces the cost of indirect calls by amortisation.
It then adds virtual address support to ahash. This allows the user to supply a virtual address as the input instead of an SG list.
This is assumed to be not DMA-capable so it is always copied before it's passed to an existing ahash driver. New drivers can elect to take virtual addresses directly. Of course existing shash algorithms are able to take virtual addresses without any copying.
The final patch resurrects the old SHA2 AVX2 muiltibuffer code as a proof of concept that this API works. The result shows that with a full complement of 8 requests, this API is able to achieve parity with the more modern but single-threaded SHA-NI code.
Herbert Xu (6):
crypto: ahash - Only save callback and data in ahash_save_req
crypto: hash - Add request chaining API
crypto: tcrypt - Restore multibuffer ahash tests
crypto: ahash - Add virtual address support
crypto: ahash - Set default reqsize from ahash_alg
crypto: x86/sha2 - Restore multibuffer AVX2 support
arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/sha256_mb_mgr_datastruct.S | 304 +++++++++++ arch/x86/crypto/sha256_ssse3_glue.c | 523 ++++++++++++++++-- arch/x86/crypto/sha256_x8_avx2.S | 596 +++++++++++++++++++++ crypto/ahash.c | 566 ++++++++++++++++--- crypto/tcrypt.c | 227 ++++++++ include/crypto/algapi.h | 10 + include/crypto/hash.h | 68 ++- include/crypto/internal/hash.h | 17 +- include/linux/crypto.h | 26 + 10 files changed, 2209 insertions(+), 130 deletions(-) create mode 100644 arch/x86/crypto/sha256_mb_mgr_datastruct.S create mode 100644 arch/x86/crypto/sha256_x8_avx2.S
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH 1/6 crypto: ahash - Only save callback and data in ahash_save_req
As unaligned operations are supported by the underlying algorithm, ahash_save_req and ahash_restore_req can be greatly simplified to only preserve the callback and data.
--- crypto/ahash.c | 97 ++++++++++++++++--------------------------- include/crypto/hash.h | 3 -- 2 files changed, 35 insertions(+), 65 deletions(-)
crypto/ahash.c +35 -62
@@ -27,6 +27,12 @@#define CRYPTO_ALG_TYPE_AHASH_MASK 0x0000000e +struct ahash_save_req_state { +struct ahash_request *req; +crypto_completion_t compl; +void *data; +}; +/* * For an ahash tfm that is using an shash algorithm (instead of an ahash * algorithm), this returns the underlying shash tfm.
@@ -262,67 +268,34 @@ int crypto_ahash_init(struct ahash_request *req) }EXPORT_SYMBOL_GPL (crypto_ahash_init); -static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt, -bool has_state) +static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -unsigned int ds =crypto_ahash_digestsize (tfm); -struct ahash_request *subreq; -unsigned int subreq_size; -unsigned int reqsize; -u8 *result; +struct ahash_save_req_state *state;gfp_t gfp;u32 flags; - subreq_size =sizeof (*subreq); - reqsize =crypto_ahash_reqsize (tfm); - reqsize =ALIGN (reqsize,crypto_tfm_ctx_alignment ()); - subreq_size += reqsize; - subreq_size += ds; - flags =ahash_request_flags (req); gfp = (flags &CRYPTO_TFM_REQ_MAY_SLEEP ) ?GFP_KERNEL :GFP_ATOMIC ; - subreq =kmalloc (subreq_size, gfp); -if (!subreq) + state =kmalloc (sizeof (*state), gfp); +if (!state)return -ENOMEM ; -ahash_request_set_tfm (subreq, tfm); -ahash_request_set_callback (subreq, flags, cplt, req); - - result = (u8 *)(subreq +1 ) + reqsize; - -ahash_request_set_crypt (subreq, req->src, result, req->nbytes); - -if (has_state) { -void *state; - - state =kmalloc (crypto_ahash_statesize (tfm), gfp); -if (!state) { -kfree (subreq); -return -ENOMEM ; - } - -crypto_ahash_export (req, state); -crypto_ahash_import (subreq, state); -kfree_sensitive (state); - } - - req->priv = subreq; + state->compl = req->base.complete; + state->data = req->base.data; + req->base.complete = cplt; + req->base.data = state; + state->req = req;return 0 ; } -static void ahash_restore_req (struct ahash_request *req,int err) +static void ahash_restore_req (struct ahash_request *req) { -struct ahash_request *subreq = req->priv; +struct ahash_save_req_state *state = req->base.data; -if (!err) -memcpy (req->result, subreq->result, -crypto_ahash_digestsize (crypto_ahash_reqtfm (req))); - - req->priv =NULL ; - -kfree_sensitive (subreq); + req->base.complete = state->compl; + req->base.data = state->data; +kfree (state); }int crypto_ahash_update (struct ahash_request *req)
@@ -374,51 +347,51 @@ EXPORT_SYMBOL_GPL(crypto_ahash_digest);static void ahash_def_finup_done2 (void *data,int err) { -struct ahash_request *areq = data; +struct ahash_save_req_state *state = data; +struct ahash_request *areq = state->req;if (err == -EINPROGRESS )return ; -ahash_restore_req (areq, err); - +ahash_restore_req (areq);ahash_request_complete (areq, err); }static int ahash_def_finup_finish1 (struct ahash_request *req,int err) { -struct ahash_request *subreq = req->priv; -if (err) goto out; - subreq->base.complete = ahash_def_finup_done2; + req->base.complete = ahash_def_finup_done2; - err =crypto_ahash_alg (crypto_ahash_reqtfm (req))->final (subreq); + err =crypto_ahash_alg (crypto_ahash_reqtfm (req))->final (req);if (err == -EINPROGRESS || err == -EBUSY )return err; out: -ahash_restore_req (req, err); +ahash_restore_req (req);return err; }static void ahash_def_finup_done1 (void *data,int err) { -struct ahash_request *areq = data; -struct ahash_request *subreq; +struct ahash_save_req_state *state0 = data; +struct ahash_save_req_state state; +struct ahash_request *areq; + state = *state0; + areq = state.req;if (err == -EINPROGRESS ) goto out; - subreq = areq->priv; - subreq->base.flags &=CRYPTO_TFM_REQ_MAY_BACKLOG ; + areq->base.flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP ; err =ahash_def_finup_finish1 (areq, err);if (err == -EINPROGRESS || err == -EBUSY )return ; out: -ahash_request_complete (areq, err); + state.compl (state.data, err); }static int ahash_def_finup (struct ahash_request *req)
@@ -426,11 +399,11 @@ static int ahash_def_finup(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);int err; - err =ahash_save_req (req, ahash_def_finup_done1, true); + err =ahash_save_req (req, ahash_def_finup_done1);if (err)return err; - err =crypto_ahash_alg (tfm)->update (req->priv); + err =crypto_ahash_alg (tfm)->update (req);if (err == -EINPROGRESS || err == -EBUSY )return err;
include/crypto/hash.h +0 -3
@@ -55,9 +55,6 @@ struct ahash_request {struct scatterlist *src;u8 *result; -/* This field may only be used by the ahash API code. */ -void *priv; - void *__ctx[] CRYPTO_MINALIGN_ATTR; };
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH 2/6 crypto: hash - Add request chaining API
In reply to Herbert XuThis adds request chaining to the ahash interface. Request chaining allows multiple requests to be submitted in one shot. An algorithm can elect to receive chained requests by setting the flag CRYPTO_ALG_REQ_CHAIN. If this bit is not set, the API will break up chained requests and submit them one-by-one.
A new err field is added to struct crypto_async_request to record the return value for each individual request.
--- crypto/ahash.c | 259 +++++++++++++++++++++++++++++---- include/crypto/algapi.h | 10 ++ include/crypto/hash.h | 25 ++++ include/crypto/internal/hash.h | 10 ++ include/linux/crypto.h | 26 ++++ 5 files changed, 304 insertions(+), 26 deletions(-)
crypto/ahash.c +233 -26
@@ -28,11 +28,19 @@#define CRYPTO_ALG_TYPE_AHASH_MASK 0x0000000estruct ahash_save_req_state { -struct ahash_request *req; +struct list_head head; +struct ahash_request *req0; +struct ahash_request *cur; +int (*op)(struct ahash_request *req);crypto_completion_t compl;void *data; }; +static void ahash_reqchain_done (void *data,int err); +static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt); +static void ahash_restore_req (struct ahash_request *req); +static int ahash_def_finup (struct ahash_request *req); +/* * For an ahash tfm that is using an shash algorithm (instead of an ahash * algorithm), this returns the underlying shash tfm.
@@ -256,24 +264,150 @@ int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key, }EXPORT_SYMBOL_GPL (crypto_ahash_setkey); +static int ahash_reqchain_finish (struct ahash_save_req_state *state, +int err,u32 mask) +{ +struct ahash_request *req0 = state->req0; +struct ahash_request *req = state->cur; +struct ahash_request *n; + + req->base.err = err; + +if (req == req0) +INIT_LIST_HEAD (&req->base.list); +else +list_add_tail (&req->base.list, &req0->base.list); + +list_for_each_entry_safe (req, n, &state->head, base.list) { +list_del_init (&req->base.list); + + req->base.flags &= mask; + req->base.complete = ahash_reqchain_done; + req->base.data = state; + state->cur = req; + err = state->op (req); + +if (err == -EINPROGRESS ) { +if (!list_empty (&state->head)) + err = -EBUSY ; + goto out; + } + +if (err == -EBUSY ) + goto out; + + req->base.err = err; +list_add_tail (&req->base.list, &req0->base.list); + } + +ahash_restore_req (req0); + +out: +return err; +} + +static void ahash_reqchain_done (void *data,int err) +{ +struct ahash_save_req_state *state = data; +crypto_completion_t compl = state->compl; + + data = state->data; + +if (err == -EINPROGRESS ) { +if (!list_empty (&state->head)) +return ; + goto notify; + } + + err =ahash_reqchain_finish (state, err,CRYPTO_TFM_REQ_MAY_BACKLOG ); +if (err == -EBUSY ) +return ; + +notify: +compl (data, err); +} + +static int ahash_do_req_chain (struct ahash_request *req, +int (*op)(struct ahash_request *req)) +{ +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ahash_save_req_state *state; +struct ahash_save_req_state state0; +int err; + +if (!ahash_request_chained (req) ||list_empty (&req->base.list) || +crypto_ahash_req_chain (tfm)) +return op (req); + + state = &state0; + +if (ahash_is_async (tfm)) { + err =ahash_save_req (req, ahash_reqchain_done); +if (err) { +struct ahash_request *r2; + + req->base.err = err; +list_for_each_entry (r2, &req->base.list, base.list) + r2->base.err = err; + +return err; + } + + state = req->base.data; + } + + state->op = op; + state->cur = req; +INIT_LIST_HEAD (&state->head); +list_splice (&req->base.list, &state->head); + + err =op (req); +if (err == -EBUSY || err == -EINPROGRESS ) +return -EBUSY ; + +return ahash_reqchain_finish (state, err, ~0 ); +} +int crypto_ahash_init (struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return crypto_shash_init (prepare_shash_desc (req, tfm)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; + + err =crypto_shash_init (prepare_shash_desc (req, tfm)); + req->base.err = err; +if (!ahash_request_chained (req)) +return err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =prepare_shash_desc (r2, tfm); + r2->base.err =crypto_shash_init (desc); + } + +return 0 ; + } +if (crypto_ahash_get_flags (tfm) &CRYPTO_TFM_NEED_KEY )return -ENOKEY ; -return crypto_ahash_alg (tfm)->init (req); + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->init); }EXPORT_SYMBOL_GPL (crypto_ahash_init);static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt) { +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct ahash_save_req_state *state;gfp_t gfp;u32 flags; +if (!ahash_is_async (tfm)) +return 0 ; + flags =ahash_request_flags (req); gfp = (flags &CRYPTO_TFM_REQ_MAY_SLEEP ) ?GFP_KERNEL :GFP_ATOMIC ; state =kmalloc (sizeof (*state), gfp);
@@ -284,14 +418,20 @@ static int ahash_save_req(struct ahash_request *req, crypto_completion_t cplt) state->data = req->base.data; req->base.complete = cplt; req->base.data = state; - state->req = req; + state->req0 = req;return 0 ; }static void ahash_restore_req (struct ahash_request *req) { -struct ahash_save_req_state *state = req->base.data; +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ahash_save_req_state *state; + +if (!ahash_is_async (tfm)) +return ; + + state = req->base.data; req->base.complete = state->compl; req->base.data = state->data;
@@ -302,10 +442,26 @@ int crypto_ahash_update(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return shash_ahash_update (req,ahash_request_ctx (req)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; -return crypto_ahash_alg (tfm)->update (req); + err =shash_ahash_update (req,ahash_request_ctx (req)); + req->base.err = err; +if (!ahash_request_chained (req)) +return err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =ahash_request_ctx (r2); + r2->base.err =shash_ahash_update (r2, desc); + } + +return 0 ; + } + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->update); }EXPORT_SYMBOL_GPL (crypto_ahash_update);
@@ -313,10 +469,26 @@ int crypto_ahash_final(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return crypto_shash_final (ahash_request_ctx (req), req->result); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; -return crypto_ahash_alg (tfm)->final (req); + err =crypto_shash_final (ahash_request_ctx (req), req->result); + req->base.err = err; +if (!ahash_request_chained (req)) +return err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =ahash_request_ctx (r2); + r2->base.err =crypto_shash_final (desc, r2->result); + } + +return 0 ; + } + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->final); }EXPORT_SYMBOL_GPL (crypto_ahash_final);
@@ -324,10 +496,29 @@ int crypto_ahash_finup(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return shash_ahash_finup (req,ahash_request_ctx (req)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; -return crypto_ahash_alg (tfm)->finup (req); + err =shash_ahash_finup (req,ahash_request_ctx (req)); + req->base.err = err; +if (!ahash_request_chained (req)) +return err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =ahash_request_ctx (r2); + r2->base.err =shash_ahash_finup (r2, desc); + } + +return 0 ; + } + +if (!crypto_ahash_alg (tfm)->finup) +return ahash_def_finup (req); + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->finup); }EXPORT_SYMBOL_GPL (crypto_ahash_finup);
@@ -335,20 +526,36 @@ int crypto_ahash_digest(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return shash_ahash_digest (req,prepare_shash_desc (req, tfm)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; + + err =shash_ahash_digest (req,prepare_shash_desc (req, tfm)); + req->base.err = err; +if (!ahash_request_chained (req)) +return err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =prepare_shash_desc (r2, tfm); + r2->base.err =shash_ahash_digest (r2, desc); + } + +return 0 ; + }if (crypto_ahash_get_flags (tfm) &CRYPTO_TFM_NEED_KEY )return -ENOKEY ; -return crypto_ahash_alg (tfm)->digest (req); +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->digest); }EXPORT_SYMBOL_GPL (crypto_ahash_digest);static void ahash_def_finup_done2 (void *data,int err) {struct ahash_save_req_state *state = data; -struct ahash_request *areq = state->req; +struct ahash_request *areq = state->req0;if (err == -EINPROGRESS )return ;
@@ -359,12 +566,15 @@ static void ahash_def_finup_done2(void *data, int err)static int ahash_def_finup_finish1 (struct ahash_request *req,int err) { +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +if (err) goto out; - req->base.complete = ahash_def_finup_done2; +if (ahash_is_async (tfm)) + req->base.complete = ahash_def_finup_done2; - err =crypto_ahash_alg (crypto_ahash_reqtfm (req))->final (req); + err =crypto_ahash_final (req);if (err == -EINPROGRESS || err == -EBUSY )return err;
@@ -380,7 +590,7 @@ static void ahash_def_finup_done1(void *data, int err)struct ahash_request *areq; state = *state0; - areq = state.req; + areq = state.req0;if (err == -EINPROGRESS ) goto out;
@@ -396,14 +606,13 @@ static void ahash_def_finup_done1(void *data, int err)static int ahash_def_finup (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);int err; err =ahash_save_req (req, ahash_def_finup_done1);if (err)return err; - err =crypto_ahash_alg (tfm)->update (req); + err =crypto_ahash_update (req);if (err == -EINPROGRESS || err == -EBUSY )return err;
@@ -618,8 +827,6 @@ static int ahash_prepare_alg(struct ahash_alg *alg) base->cra_type = &crypto_ahash_type; base->cra_flags |=CRYPTO_ALG_TYPE_AHASH ; -if (!alg->finup) - alg->finup = ahash_def_finup;if (!alg->setkey) alg->setkey = ahash_nosetkey;
include/crypto/algapi.h +10 -0
@@ -271,4 +271,14 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm) return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK; } +static inline bool crypto_request_chained (struct crypto_async_request *req) +{ +return req->flags &CRYPTO_TFM_REQ_CHAIN ; +} + +static inline bool crypto_tfm_req_chain (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags &CRYPTO_ALG_REQ_CHAIN ; +} +#endif /* _CRYPTO_ALGAPI_H */
include/crypto/hash.h +25 -0
@@ -621,6 +621,7 @@ static inline void ahash_request_set_callback(struct ahash_request *req, { req->base.complete = compl; req->base.data = data; + flags &= ~CRYPTO_TFM_REQ_CHAIN ; req->base.flags = flags; }
@@ -646,6 +647,20 @@ static inline void ahash_request_set_crypt(struct ahash_request *req, req->result = result; } +static inline void ahash_reqchain_init (struct ahash_request *req, +u32 flags,crypto_completion_t compl, +void *data) +{ +ahash_request_set_callback (req, flags, compl, data); +crypto_reqchain_init (&req->base); +} + +static inline void ahash_request_chain (struct ahash_request *req, +struct ahash_request *head) +{ +crypto_request_chain (&req->base, &head->base); +} +/** * DOC: Synchronous Message Digest API *
@@ -947,4 +962,14 @@ static inline void shash_desc_zero(struct shash_desc *desc) sizeof(*desc) + crypto_shash_descsize(desc->tfm)); } +static inline int ahash_request_err (struct ahash_request *req) +{ +return req->base.err; +} + +static inline bool ahash_is_async (struct crypto_ahash *tfm) +{ +return crypto_tfm_is_async (&tfm->base); +} +#endif /* _CRYPTO_HASH_H */
include/crypto/internal/hash.h +10 -0
@@ -270,5 +270,15 @@ static inline struct crypto_shash *__crypto_shash_cast(struct crypto_tfm *tfm) return container_of(tfm, struct crypto_shash, base); } +static inline bool ahash_request_chained (struct ahash_request *req) +{ +return crypto_request_chained (&req->base); +} + +static inline bool crypto_ahash_req_chain (struct crypto_ahash *tfm) +{ +return crypto_tfm_req_chain (&tfm->base); +} +#endif /* _CRYPTO_INTERNAL_HASH_H */
include/linux/crypto.h +26 -0
@@ -13,6 +13,8 @@#define _LINUX_CRYPTO_H#include <linux/completion.h> +#include <linux/errno.h> +#include <linux/list.h> #include <linux/refcount.h> #include <linux/slab.h> #include <linux/types.h>
@@ -124,6 +126,9 @@*/ #define CRYPTO_ALG_FIPS_INTERNAL 0x00020000 +/* Set if the algorithm supports request chains. */ +#define CRYPTO_ALG_REQ_CHAIN 0x00040000 +/* * Transform masks and values (for crt_flags). */
@@ -133,6 +138,7 @@#define CRYPTO_TFM_REQ_FORBID_WEAK_KEYS 0x00000100#define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200#define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400 +#define CRYPTO_TFM_REQ_CHAIN 0x00000800/* * Miscellaneous stuff.
@@ -174,6 +180,7 @@ struct crypto_async_request {struct crypto_tfm *tfm;u32 flags; +int err; };/**
@@ -540,5 +547,24 @@ int crypto_comp_decompress(struct crypto_comp *tfm, const u8 *src, unsigned int slen, u8 *dst, unsigned int *dlen); +static inline void crypto_reqchain_init (struct crypto_async_request *req) +{ + req->err = -EINPROGRESS ; + req->flags |=CRYPTO_TFM_REQ_CHAIN ; +INIT_LIST_HEAD (&req->list); +} + +static inline void crypto_request_chain (struct crypto_async_request *req, +struct crypto_async_request *head) +{ + req->err = -EINPROGRESS ; +list_add_tail (&req->list, &head->list); +} + +static inline bool crypto_tfm_is_async (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags &CRYPTO_ALG_ASYNC ; +} +#endif /* _LINUX_CRYPTO_H */
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH 3/6 crypto: tcrypt - Restore multibuffer ahash tests
In reply to Herbert XuThis patch is a revert of commit 388ac25efc8ce3bf9768ce7bf24268d6fac285d5.
As multibuffer ahash is coming back in the form of request chaining, restore the multibuffer ahash tests using the new interface.
--- crypto/tcrypt.c | 227 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+)
crypto/tcrypt.c +227 -0
@@ -716,6 +716,203 @@ static inline int do_one_ahash_op(struct ahash_request *req, int ret)return crypto_wait_req (ret, wait); } +struct test_mb_ahash_data { +struct scatterlist sg[XBUFSIZE ]; +char result[64 ]; +struct ahash_request *req; +struct crypto_wait wait; +char *xbuf[XBUFSIZE ]; +}; + +static inline int do_mult_ahash_op (struct test_mb_ahash_data *data,u32 num_mb, +int *rc) +{ +int i, err; + +/* Fire up a bunch of concurrent requests */ + err =crypto_ahash_digest (data[0 ].req); + +/* Wait for all requests to finish */ + err =crypto_wait_req (err, &data[0 ].wait); + +for (i =0 ; i < num_mb; i++) { + rc[i] =ahash_request_err (data[i].req); +if (rc[i]) { +pr_info ("concurrent request %d error %d\n" , i, rc[i]); + err = rc[i]; + } + } + +return err; +} + +static int test_mb_ahash_jiffies (struct test_mb_ahash_data *data,int blen, +int secs,u32 num_mb) +{ +unsigned long start, end; +int bcount; +int ret =0 ; +int *rc; + + rc =kcalloc (num_mb,sizeof (*rc),GFP_KERNEL ); +if (!rc) +return -ENOMEM ; + +for (start = jiffies, end = start + secs *HZ , bcount =0 ; +time_before (jiffies, end); bcount++) { + ret =do_mult_ahash_op (data, num_mb, rc); +if (ret) + goto out; + } + +pr_cont ("%d operations in %d seconds (%llu bytes)\n" , + bcount * num_mb, secs, (u64 )bcount * blen * num_mb); + +out: +kfree (rc); +return ret; +} + +static int test_mb_ahash_cycles (struct test_mb_ahash_data *data,int blen, +u32 num_mb) +{ +unsigned long cycles =0 ; +int ret =0 ; +int i; +int *rc; + + rc =kcalloc (num_mb,sizeof (*rc),GFP_KERNEL ); +if (!rc) +return -ENOMEM ; + +/* Warm-up run. */ +for (i =0 ; i <4 ; i++) { + ret =do_mult_ahash_op (data, num_mb, rc); +if (ret) + goto out; + } + +/* The real thing. */ +for (i =0 ; i <8 ; i++) { +cycles_t start, end; + + start =get_cycles (); + ret =do_mult_ahash_op (data, num_mb, rc); + end =get_cycles (); + +if (ret) + goto out; + + cycles += end - start; + } + +pr_cont ("1 operation in %lu cycles (%d bytes)\n" , + (cycles +4 ) / (8 * num_mb), blen); + +out: +kfree (rc); +return ret; +} + +static void test_mb_ahash_speed (const char *algo,unsigned int secs, +struct hash_speed *speed,u32 num_mb) +{ +struct test_mb_ahash_data *data; +struct crypto_ahash *tfm; +unsigned int i, j, k; +int ret; + + data =kcalloc (num_mb,sizeof (*data),GFP_KERNEL ); +if (!data) +return ; + + tfm =crypto_alloc_ahash (algo,0 ,0 ); +if (IS_ERR (tfm)) { +pr_err ("failed to load transform for %s: %ld\n" , + algo,PTR_ERR (tfm)); + goto free_data; + } + +for (i =0 ; i < num_mb; ++i) { +if (testmgr_alloc_buf (data[i].xbuf)) + goto out; + +crypto_init_wait (&data[i].wait); + + data[i].req =ahash_request_alloc (tfm,GFP_KERNEL ); +if (!data[i].req) { +pr_err ("alg: hash: Failed to allocate request for %s\n" , + algo); + goto out; + } + +if (i) +ahash_request_chain (data[i].req, data[0 ].req); +else +ahash_reqchain_init (data[i].req,0 , crypto_req_done, + &data[i].wait); + +sg_init_table (data[i].sg,XBUFSIZE ); +for (j =0 ; j <XBUFSIZE ; j++) { +sg_set_buf (data[i].sg + j, data[i].xbuf[j],PAGE_SIZE ); +memset (data[i].xbuf[j],0xff ,PAGE_SIZE ); + } + } + +pr_info ("\ntesting speed of multibuffer %s (%s)\n" , algo, +get_driver_name (crypto_ahash, tfm)); + +for (i =0 ; speed[i].blen !=0 ; i++) { +/* For some reason this only tests digests. */ +if (speed[i].blen != speed[i].plen) +continue ; + +if (speed[i].blen >XBUFSIZE *PAGE_SIZE ) { +pr_err ("template (%u) too big for tvmem (%lu)\n" , + speed[i].blen,XBUFSIZE *PAGE_SIZE ); + goto out; + } + +if (klen) +crypto_ahash_setkey (tfm, tvmem[0 ], klen); + +for (k =0 ; k < num_mb; k++) +ahash_request_set_crypt (data[k].req, data[k].sg, + data[k].result, speed[i].blen); + +pr_info ("test%3u " +"(%5u byte blocks,%5u bytes per update,%4u updates): " , + i, speed[i].blen, speed[i].plen, + speed[i].blen / speed[i].plen); + +if (secs) { + ret =test_mb_ahash_jiffies (data, speed[i].blen, secs, + num_mb); +cond_resched (); + }else { + ret =test_mb_ahash_cycles (data, speed[i].blen, num_mb); + } + + +if (ret) { +pr_err ("At least one hashing failed ret=%d\n" , ret); +break ; + } + } + +out: +for (k =0 ; k < num_mb; ++k) +ahash_request_free (data[k].req); + +for (k =0 ; k < num_mb; ++k) +testmgr_free_buf (data[k].xbuf); + +crypto_free_ahash (tfm); + +free_data: +kfree (data); +} +static int test_ahash_jiffies_digest (struct ahash_request *req,int blen,char *out,int secs) {
@@ -2395,6 +2592,36 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)test_ahash_speed ("sm3" , sec, generic_hash_speed_template);if (mode >400 && mode <500 )break ; fallthrough; +case 450 : +test_mb_ahash_speed ("sha1" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 451 : +test_mb_ahash_speed ("sha256" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 452 : +test_mb_ahash_speed ("sha512" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 453 : +test_mb_ahash_speed ("sm3" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 454 : +test_mb_ahash_speed ("streebog256" , sec, + generic_hash_speed_template, num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 455 : +test_mb_ahash_speed ("streebog512" , sec, + generic_hash_speed_template, num_mb); +if (mode >400 && mode <500 )break ; + fallthrough;case 499 :break ;
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH 4/6 crypto: ahash - Add virtual address support
In reply to Herbert XuThis patch adds virtual address support to ahash. Virtual addresses were previously only supported through shash. The user may choose to use virtual addresses with ahash by calling ahash_request_set_virt instead of ahash_request_set_crypt.
The API will take care of translating this to an SG list if necessary, unless the algorithm declares that it supports chaining. Therefore in order for an ahash algorithm to support chaining, it must also support virtual addresses directly.
--- crypto/ahash.c | 282 +++++++++++++++++++++++++++++---- include/crypto/hash.h | 39 ++++- include/crypto/internal/hash.h | 7 +- include/linux/crypto.h | 2 +- 4 files changed, 294 insertions(+), 36 deletions(-)
crypto/ahash.c +251 -31
@@ -34,11 +34,17 @@ struct ahash_save_req_state {int (*op)(struct ahash_request *req);crypto_completion_t compl;void *data; +struct scatterlist sg; +const u8 *src; +u8 *page; +unsigned int offset; +unsigned int nbytes; };static void ahash_reqchain_done (void *data,int err);static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt); -static void ahash_restore_req (struct ahash_request *req); +static void ahash_restore_req (struct ahash_save_req_state *state); +static void ahash_def_finup_done1 (void *data,int err);static int ahash_def_finup (struct ahash_request *req);/*
@@ -100,6 +106,10 @@ int shash_ahash_digest(struct ahash_request *req, struct shash_desc *desc)unsigned int offset;int err; +if (ahash_request_isvirt (req)) +return crypto_shash_digest (desc, req->svirt, nbytes, + req->result); +if (nbytes && (sg = req->src, offset = sg->offset, nbytes <=min (sg->length, ((unsigned int )(PAGE_SIZE )) - offset))) {
@@ -182,6 +192,9 @@ static int hash_walk_new_entry(struct crypto_hash_walk *walk)int crypto_hash_walk_done (struct crypto_hash_walk *walk,int err) { +if ((walk->flags &CRYPTO_AHASH_REQ_VIRT )) +return err; + walk->data -= walk->offset;kunmap_local (walk->data);
@@ -209,14 +222,20 @@ int crypto_hash_walk_first(struct ahash_request *req, struct crypto_hash_walk *walk) { walk->total = req->nbytes; + walk->entrylen =0 ; -if (!walk->total) { - walk->entrylen =0 ; +if (!walk->total)return 0 ; + + walk->flags = req->base.flags; + +if (ahash_request_isvirt (req)) { + walk->data = req->svirt; + walk->total =0 ; +return req->nbytes; } walk->sg = req->src; - walk->flags = req->base.flags;return hash_walk_new_entry (walk); }
@@ -264,20 +283,85 @@ int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key, }EXPORT_SYMBOL_GPL (crypto_ahash_setkey); +static bool ahash_request_hasvirt (struct ahash_request *req) +{ +struct ahash_request *r2; + +if (ahash_request_isvirt (req)) +return true; + +if (!ahash_request_chained (req)) +return false; + +list_for_each_entry (r2, &req->base.list, base.list) +if (ahash_request_isvirt (r2)) +return true; + +return false; +} + +static int ahash_reqchain_virt (struct ahash_save_req_state *state, +int err,u32 mask) +{ +struct ahash_request *req = state->cur; + +for (;;) { +unsigned len = state->nbytes; + + req->base.err = err; + +if (!state->offset) +break ; + +if (state->offset == len || err) { +u8 *result = req->result; + +ahash_request_set_virt (req, state->src, result, len); + state->offset =0 ; +break ; + } + + len -= state->offset; + + len =min (PAGE_SIZE , len); +memcpy (state->page, state->src + state->offset, len); + state->offset += len; + req->nbytes = len; + + err = state->op (req); +if (err == -EINPROGRESS ) { +if (!list_empty (&state->head) || + state->offset < state->nbytes) + err = -EBUSY ; +break ; + } + +if (err == -EBUSY ) +break ; + } + +return err; +} +static int ahash_reqchain_finish (struct ahash_save_req_state *state,int err,u32 mask) {struct ahash_request *req0 = state->req0;struct ahash_request *req = state->cur; +struct crypto_ahash *tfm;struct ahash_request *n; +bool update; - req->base.err = err; + err =ahash_reqchain_virt (state, err, mask);if (req == req0)INIT_LIST_HEAD (&req->base.list);else list_add_tail (&req->base.list, &req0->base.list); + tfm =crypto_ahash_reqtfm (req); + update = state->op ==crypto_ahash_alg (tfm)->update; +list_for_each_entry_safe (req, n, &state->head, base.list) {list_del_init (&req->base.list);
@@ -285,10 +369,27 @@ static int ahash_reqchain_finish(struct ahash_save_req_state *state, req->base.complete = ahash_reqchain_done; req->base.data = state; state->cur = req; + +if (update &&ahash_request_isvirt (req) && req->nbytes) { +unsigned len = req->nbytes; +u8 *result = req->result; + + state->src = req->svirt; + state->nbytes = len; + + len =min (PAGE_SIZE , len); + +memcpy (state->page, req->svirt, len); + state->offset = len; + +ahash_request_set_crypt (req, &state->sg, result, len); + } + err = state->op (req);if (err == -EINPROGRESS ) { -if (!list_empty (&state->head)) +if (!list_empty (&state->head) || + state->offset < state->nbytes) err = -EBUSY ; goto out; }
@@ -296,11 +397,14 @@ static int ahash_reqchain_finish(struct ahash_save_req_state *state,if (err == -EBUSY ) goto out; - req->base.err = err; + err =ahash_reqchain_virt (state, err, mask); +if (err == -EINPROGRESS || err == -EBUSY ) + goto out; +list_add_tail (&req->base.list, &req0->base.list); } -ahash_restore_req (req0); +ahash_restore_req (state); out:return err;
@@ -314,7 +418,7 @@ static void ahash_reqchain_done(void *data, int err) data = state->data;if (err == -EINPROGRESS ) { -if (!list_empty (&state->head)) +if (!list_empty (&state->head) || state->offset < state->nbytes)return ; goto notify; }
@@ -331,41 +435,84 @@ static int ahash_do_req_chain(struct ahash_request *req, int (*op)(struct ahash_request *req)) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +bool update = op ==crypto_ahash_alg (tfm)->update;struct ahash_save_req_state *state;struct ahash_save_req_state state0; +struct ahash_request *r2; +u8 *page =NULL ;int err; -if (!ahash_request_chained (req) ||list_empty (&req->base.list) || -crypto_ahash_req_chain (tfm)) +if (crypto_ahash_req_chain (tfm) || + ((!ahash_request_chained (req) ||list_empty (&req->base.list)) && + (!update || !ahash_request_isvirt (req))))return op (req); - state = &state0; +if (update &&ahash_request_hasvirt (req)) { +gfp_t gfp; +u32 flags; + flags =ahash_request_flags (req); + gfp = (flags &CRYPTO_TFM_REQ_MAY_SLEEP ) ? +GFP_KERNEL :GFP_ATOMIC ; + page = (void *)__get_free_page (gfp); + err = -ENOMEM ; +if (!page) + goto out_set_chain; + } + + state = &state0;if (ahash_is_async (tfm)) { err =ahash_save_req (req, ahash_reqchain_done); -if (err) { -struct ahash_request *r2; - - req->base.err = err; -list_for_each_entry (r2, &req->base.list, base.list) - r2->base.err = err; - -return err; - } +if (err) + goto out_free_page; state = req->base.data; } state->op = op; state->cur = req; + state->page = page; + state->offset =0 ; + state->nbytes =0 ;INIT_LIST_HEAD (&state->head);list_splice (&req->base.list, &state->head); +if (page) +sg_init_one (&state->sg, page,PAGE_SIZE ); + +if (update &&ahash_request_isvirt (req) && req->nbytes) { +unsigned len = req->nbytes; +u8 *result = req->result; + + state->src = req->svirt; + state->nbytes = len; + + len =min (PAGE_SIZE , len); + +memcpy (page, req->svirt, len); + state->offset = len; + +ahash_request_set_crypt (req, &state->sg, result, len); + } + err =op (req);if (err == -EBUSY || err == -EINPROGRESS )return -EBUSY ;return ahash_reqchain_finish (state, err, ~0 ); + +out_free_page: +if (page) { +memset (page,0 ,PAGE_SIZE ); +free_page ((unsigned long )page); + } + +out_set_chain: + req->base.err = err; +list_for_each_entry (r2, &req->base.list, base.list) + r2->base.err = err; + +return err; }int crypto_ahash_init (struct ahash_request *req)
@@ -419,15 +566,19 @@ static int ahash_save_req(struct ahash_request *req, crypto_completion_t cplt) req->base.complete = cplt; req->base.data = state; state->req0 = req; + state->page =NULL ;return 0 ; } -static void ahash_restore_req (struct ahash_request *req) +static void ahash_restore_req (struct ahash_save_req_state *state) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -struct ahash_save_req_state *state; +struct ahash_request *req = state->req0; +struct crypto_ahash *tfm; +free_page ((unsigned long )state->page); + + tfm =crypto_ahash_reqtfm (req);if (!ahash_is_async (tfm))return ;
@@ -515,13 +666,74 @@ int crypto_ahash_finup(struct ahash_request *req)return 0 ; } -if (!crypto_ahash_alg (tfm)->finup) +if (!crypto_ahash_alg (tfm)->finup || + (!crypto_ahash_req_chain (tfm) &&ahash_request_hasvirt (req)))return ahash_def_finup (req);return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->finup); }EXPORT_SYMBOL_GPL (crypto_ahash_finup); +static int ahash_def_digest_finish (struct ahash_save_req_state *state,int err) +{ +struct ahash_request *req = state->req0; +struct crypto_ahash *tfm; + +if (err) + goto out; + + tfm =crypto_ahash_reqtfm (req); +if (ahash_is_async (tfm)) + req->base.complete = ahash_def_finup_done1; + + err =crypto_ahash_update (req); +if (err == -EINPROGRESS || err == -EBUSY ) +return err; + +out: +ahash_restore_req (state); +return err; +} + +static void ahash_def_digest_done (void *data,int err) +{ +struct ahash_save_req_state *state0 = data; +struct ahash_save_req_state state; +struct ahash_request *areq; + + state = *state0; + areq = state.req0; +if (err == -EINPROGRESS ) + goto out; + + areq->base.flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP ; + + err =ahash_def_digest_finish (state0, err); +if (err == -EINPROGRESS || err == -EBUSY ) +return ; + +out: + state.compl (state.data, err); +} + +static int ahash_def_digest (struct ahash_request *req) +{ +struct ahash_save_req_state *state; +int err; + + err =ahash_save_req (req, ahash_def_digest_done); +if (err) +return err; + + state = req->base.data; + + err =crypto_ahash_init (req); +if (err == -EINPROGRESS || err == -EBUSY ) +return err; + +return ahash_def_digest_finish (state, err); +} +int crypto_ahash_digest (struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);
@@ -545,6 +757,9 @@ int crypto_ahash_digest(struct ahash_request *req)return 0 ; } +if (!crypto_ahash_req_chain (tfm) &&ahash_request_hasvirt (req)) +return ahash_def_digest (req); +if (crypto_ahash_get_flags (tfm) &CRYPTO_TFM_NEED_KEY )return -ENOKEY ;
@@ -560,17 +775,19 @@ static void ahash_def_finup_done2(void *data, int err)if (err == -EINPROGRESS )return ; -ahash_restore_req (areq); +ahash_restore_req (state);ahash_request_complete (areq, err); } -static int ahash_def_finup_finish1 (struct ahash_request *req,int err) +static int ahash_def_finup_finish1 (struct ahash_save_req_state *state,int err) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ahash_request *req = state->req0; +struct crypto_ahash *tfm;if (err) goto out; + tfm =crypto_ahash_reqtfm (req);if (ahash_is_async (tfm)) req->base.complete = ahash_def_finup_done2;
@@ -579,7 +796,7 @@ static int ahash_def_finup_finish1(struct ahash_request *req, int err)return err; out: -ahash_restore_req (req); +ahash_restore_req (state);return err; }
@@ -596,7 +813,7 @@ static void ahash_def_finup_done1(void *data, int err) areq->base.flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP ; - err =ahash_def_finup_finish1 (areq, err); + err =ahash_def_finup_finish1 (state0, err);if (err == -EINPROGRESS || err == -EBUSY )return ;
@@ -606,17 +823,20 @@ static void ahash_def_finup_done1(void *data, int err)static int ahash_def_finup (struct ahash_request *req) { +struct ahash_save_req_state *state;int err; err =ahash_save_req (req, ahash_def_finup_done1);if (err)return err; + state = req->base.data; + err =crypto_ahash_update (req);if (err == -EINPROGRESS || err == -EBUSY )return err; -return ahash_def_finup_finish1 (req, err); +return ahash_def_finup_finish1 (state, err); }int crypto_ahash_export (struct ahash_request *req,void *out)
include/crypto/hash.h +36 -3
@@ -12,6 +12,9 @@#include <linux/crypto.h> #include <linux/string.h> +/* Set this bit for virtual address instead of SG list. */ +#define CRYPTO_AHASH_REQ_VIRT 0x00000001 +struct crypto_ahash ;/**
@@ -52,7 +55,10 @@ struct ahash_request {struct crypto_async_request base;unsigned int nbytes; -struct scatterlist *src; +union { +struct scatterlist *src; +const u8 *svirt; + };u8 *result; void *__ctx[] CRYPTO_MINALIGN_ATTR;
@@ -619,10 +625,13 @@ static inline void ahash_request_set_callback(struct ahash_request *req,crypto_completion_t compl, void *data) { + u32 keep = CRYPTO_TFM_REQ_CHAIN | CRYPTO_AHASH_REQ_VIRT; + req->base.complete = compl; req->base.data = data; - flags &= ~CRYPTO_TFM_REQ_CHAIN ; - req->base.flags = flags; + flags &= ~keep; + req->base.flags &= ~keep; + req->base.flags |= flags; }/**
@@ -645,6 +654,30 @@ static inline void ahash_request_set_crypt(struct ahash_request *req, req->src = src; req->nbytes = nbytes; req->result = result; + req->base.flags &= ~CRYPTO_AHASH_REQ_VIRT ; +} + +/** +* ahash_request_set_virt() - set virtual address data buffers +* @req: ahash_request handle to be updated +* @src: source virtual address +* @result: buffer that is filled with the message digest -- the caller must +* ensure that the buffer has sufficient space by, for example, calling +* crypto_ahash_digestsize() +* @nbytes: number of bytes to process from the source virtual address +* +* By using this call, the caller references the source virtual address. +* The source virtual address points to the data the message digest is to +* be calculated for. +*/ +static inline void ahash_request_set_virt (struct ahash_request *req, +const u8 *src,u8 *result, +unsigned int nbytes) +{ + req->svirt = src; + req->nbytes = nbytes; + req->result = result; + req->base.flags |=CRYPTO_AHASH_REQ_VIRT ; } static inline void ahash_reqchain_init(struct ahash_request *req,
include/crypto/internal/hash.h +6 -1
@@ -15,7 +15,7 @@ struct ahash_request;struct scatterlist ;struct crypto_hash_walk { -char *data; +const char *data;unsigned int offset;unsigned int flags;
@@ -275,6 +275,11 @@ static inline bool ahash_request_chained(struct ahash_request *req)return crypto_request_chained (&req->base); } +static inline bool ahash_request_isvirt (struct ahash_request *req) +{ +return req->base.flags &CRYPTO_AHASH_REQ_VIRT ; +} +static inline bool crypto_ahash_req_chain (struct crypto_ahash *tfm) {return crypto_tfm_req_chain (&tfm->base);
include/linux/crypto.h +1 -1
@@ -126,7 +126,7 @@*/ #define CRYPTO_ALG_FIPS_INTERNAL 0x00020000 -/* Set if the algorithm supports request chains. */ +/* Set if the algorithm supports request chains and virtual addresses. */ #define CRYPTO_ALG_REQ_CHAIN 0x00040000/*
Signature · 1 line
2.39.5
kernel test robot
build botRe: [PATCH 4/6] crypto: ahash - Add virtual address support
kernel test robot noticed "BUG:sleeping_function_called_from_invalid_context_at_crypto/internal.h" on:
Hello,
kernel test robot noticed "BUG:sleeping_function_called_from_invalid_context_at_crypto/internal.h" on:
commit: 1e1d7cc33bd78b992ab90e28b02d7a2feef96538 ("[PATCH 4/6] crypto: ahash - Add virtual address support")
url: https://github.com/intel-lab-lkp/linux/commits/Herbert-Xu/crypto-ahash-Only-save-callback-and-data-in-ahash_save_req/20241027-174811
base: https://git.kernel.org/cgit/linux/kernel/git/herbert/cryptodev-2.6.git master
patch link: https://lore.kernel.org/all/bffef4bab1bf250bd64a3d02de53eb1fd047a96e.1730021644.git.herbert@gondor.apana.org.au/
patch subject: [PATCH 4/6] crypto: ahash - Add virtual address support
in testcase: kernel-selftests-bpf
version:
with following parameters:
group: net/netfilter
test: nft_flowtable.sh
config: x86_64-rhel-8.3-bpf
compiler: gcc-12
test machine: 36 threads 1 sockets Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz (Cascade Lake) with 32G memory
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
kern :err : [ 43.366785] BUG: sleeping function called from invalid context at crypto/internal.h:189 kern :err : [ 43.367576] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2600, name: socat kern :err : [ 43.368344] preempt_count: 101, expected: 0 kern :err : [ 43.368804] RCU nest depth: 1, expected: 0 kern :warn : [ 43.369258] CPU: 6 UID: 0 PID: 2600 Comm: socat Tainted: G S 6.12.0-rc1-00092-g1e1d7cc33bd7 #1 kern :warn : [ 43.370202] Tainted: [S]=CPU_OUT_OF_SPEC kern :warn : [ 43.370639] Hardware name: Gigabyte Technology Co., Ltd. X299 UD4 Pro/X299 UD4 Pro-CF, BIOS F8a 04/27/2021 kern :warn : [ 43.371552] Call Trace: kern :warn : [ 43.371871] <IRQ> kern :warn : [ 43.372155] dump_stack_lvl (lib/dump_stack.c:123) kern :warn : [ 43.372574] __might_resched (kernel/sched/core.c:8632) kern :warn : [ 43.373014] crypto_hash_walk_done (include/linux/sched.h:2031 crypto/internal.h:189 crypto/ahash.c:201) kern :warn : [ 43.373483] shash_ahash_finup (crypto/ahash.c:96) kern :warn : [ 43.373929] crypto_ahash_digest (crypto/ahash.c:747) kern :warn : [ 43.374390] crypto_authenc_genicv (crypto/authenc.c:151) authenc kern :warn : [ 43.374929] esp_output_tail (net/ipv4/esp4.c:627) esp4 kern :warn : [ 43.375417] esp_output (net/ipv4/esp4.c:701) esp4 kern :warn : [ 43.375869] xfrm_output_one (net/xfrm/xfrm_output.c:553) kern :warn : [ 43.376310] xfrm_output_resume (net/xfrm/xfrm_output.c:588) kern :warn : [ 43.376763] ? __pfx_csum_partial_ext (include/net/checksum.h:120) kern :warn : [ 43.377252] ? __pfx_csum_block_add_ext (net/core/skbuff.c:103) kern :warn : [ 43.377754] ? skb_checksum_help (net/core/dev.c:3346) kern :warn : [ 43.378215] __netif_receive_skb_one_core (net/core/dev.c:5662 (discriminator 4)) kern :warn : [ 43.378732] process_backlog (include/linux/rcupdate.h:882 net/core/dev.c:6108) kern :warn : [ 43.379163] __napi_poll+0x28/0x1c0 kern :warn : [ 43.379652] net_rx_action (net/core/dev.c:6842 net/core/dev.c:6962) kern :warn : [ 43.380078] handle_softirqs (kernel/softirq.c:554) kern :warn : [ 43.380518] do_softirq (kernel/softirq.c:455 kernel/softirq.c:442) kern :warn : [ 43.380907] </IRQ> kern :warn : [ 43.381196] <TASK> kern :warn : [ 43.381485] __local_bh_enable_ip (kernel/softirq.c:382) kern :warn : [ 43.381945] tcp_sendmsg (net/ipv4/tcp.c:1361) kern :warn : [ 43.382342] sock_write_iter (net/socket.c:729 net/socket.c:744 net/socket.c:1165) kern :warn : [ 43.382781] ? sock_recvmsg (net/socket.c:1051 net/socket.c:1073) kern :warn : [ 43.383199] vfs_write (fs/read_write.c:590 fs/read_write.c:683) kern :warn : [ 43.383598] ksys_write (include/linux/file.h:83 fs/read_write.c:739) kern :warn : [ 43.383987] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83) kern :warn : [ 43.384406] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
kern :warn : [ 43.384937] RIP: 0033:0x7f39ddcef240
kern :warn : [ 43.385348] Code: 40 00 48 8b 15 c1 9b 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d a1 23 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89 All code
========
0: 40 00 48 8b rex add %cl,-0x75(%rax) 4: 15 c1 9b 0d 00 adc $0xd9bc1,%eax 9: f7 d8 neg %eax b: 64 89 02 mov %eax,%fs:(%rdx) e: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax 15: eb b7 jmp 0xffffffffffffffce 17: 0f 1f 00 nopl (%rax) 1a: 80 3d a1 23 0e 00 00 cmpb $0x0,0xe23a1(%rip) # 0xe23c2 21: 74 17 je 0x3a 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 58 ja 0x8a 32: c3 ret 33: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) 3a: 48 83 ec 28 sub $0x28,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 58 ja 0x60 8: c3 ret 9: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) 10: 48 83 ec 28 sub $0x28,%rsp 14: 48 rex.W 15: 89 .byte 0x89 kern :warn : [ 43.387012] RSP: 002b:00007fff1b6b3288 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 kern :warn : [ 43.387756] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f39ddcef240 kern :warn : [ 43.388465] RDX: 0000000000002000 RSI: 000055ba398a7000 RDI: 0000000000000007 kern :warn : [ 43.389173] RBP: 000055ba398a7000 R08: 0000000000002000 R09: 0000000000000000 kern :warn : [ 43.389881] R10: 00007f39ddc104f0 R11: 0000000000000202 R12: 0000000000002000 kern :warn : [ 43.390590] R13: 0000000000000007 R14: 0000000000002000 R15: 000055ba398a7000 kern :warn : [ 43.391300] </TASK>
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20241105/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Herbert Xu
maintainerPATCH 5/6 crypto: ahash - Set default reqsize from ahash_alg
In reply to Herbert XuAdd a reqsize field to struct ahash_alg and use it to set the default reqsize so that algorithms with a static reqsize are not forced to create an init_tfm function.
--- crypto/ahash.c | 4 ++++ include/crypto/hash.h | 3 +++ 2 files changed, 7 insertions(+)
crypto/ahash.c +4 -0
@@ -875,6 +875,7 @@ static int crypto_ahash_init_tfm(struct crypto_tfm *tfm)struct ahash_alg *alg =crypto_ahash_alg (hash);crypto_ahash_set_statesize (hash, alg->halg.statesize); +crypto_ahash_set_reqsize (hash, alg->reqsize);if (tfm->__crt_alg->cra_type == &crypto_shash_type)return crypto_init_ahash_using_shash (tfm);
@@ -1040,6 +1041,9 @@ static int ahash_prepare_alg(struct ahash_alg *alg)if (alg->halg.statesize ==0 )return -EINVAL ; +if (alg->reqsize && alg->reqsize < alg->halg.statesize) +return -EINVAL ; + err =hash_prepare_alg (&alg->halg);if (err)return err;
include/crypto/hash.h +3 -0
@@ -135,6 +135,7 @@ struct ahash_request {* This is a counterpart to @init_tfm, used to remove * various changes set in @init_tfm. * @clone_tfm: Copy transform into new object, may allocate memory. +* @reqsize: Size of the request context. * @halg: see struct hash_alg_common */ struct ahash_alg {
@@ -151,6 +152,8 @@ struct ahash_alg {void (*exit_tfm)(struct crypto_ahash *tfm);int (*clone_tfm)(struct crypto_ahash *dst,struct crypto_ahash *src); +unsigned int reqsize; +struct hash_alg_common halg; };
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH 6/6 crypto: x86/sha2 - Restore multibuffer AVX2 support
In reply to Herbert XuResurrect the old multibuffer AVX2 code removed by commit ab8085c130ed ("crypto: x86 - remove SHA multibuffer routines and mcryptd") using the new request chaining interface.
This is purely a proof of concept and only meant to illustrate the utility of the new API rather than a serious attempt at improving the performance.
However, it is interesting to note that with x8 multibuffer the performance of AVX2 is on par with SHA-NI.
testing speed of multibuffer sha256 (sha256-avx2)
tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 1 operation in 184 cycles (16 bytes)
tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 1 operation in 165 cycles (64 bytes)
tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 1 operation in 444 cycles (256 bytes)
tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 1 operation in 1549 cycles (1024 bytes)
tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 1 operation in 3060 cycles (2048 bytes)
tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 1 operation in 5983 cycles (4096 bytes)
tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 1 operation in 11980 cycles (8192 bytes)
tcrypt: testing speed of async sha256 (sha256-avx2)
tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 475 cycles/operation, 29 cycles/byte
tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 780 cycles/operation, 12 cycles/byte
tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 1872 cycles/operation, 7 cycles/byte
tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 5416 cycles/operation, 5 cycles/byte
tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 10339 cycles/operation, 5 cycles/byte
tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 20214 cycles/operation, 4 cycles/byte
tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 40042 cycles/operation, 4 cycles/byte
tcrypt: testing speed of async sha256-ni (sha256-ni)
tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 207 cycles/operation, 12 cycles/byte
tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 299 cycles/operation, 4 cycles/byte
tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 543 cycles/operation, 2 cycles/byte
tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 1523 cycles/operation, 1 cycles/byte
tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 2835 cycles/operation, 1 cycles/byte
tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 5459 cycles/operation, 1 cycles/byte
tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 10724 cycles/operation, 1 cycles/byte
--- arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/sha256_mb_mgr_datastruct.S | 304 +++++++++++ arch/x86/crypto/sha256_ssse3_glue.c | 523 ++++++++++++++++-- arch/x86/crypto/sha256_x8_avx2.S | 596 +++++++++++++++++++++ 4 files changed, 1382 insertions(+), 43 deletions(-) create mode 100644 arch/x86/crypto/sha256_mb_mgr_datastruct.S create mode 100644 arch/x86/crypto/sha256_x8_avx2.S
arch/x86/crypto/Makefile +1 -1
@@ -60,7 +60,7 @@ sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o -sha256-ssse3-y := sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256_ssse3_glue.o +sha256-ssse3-y := sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256_ssse3_glue.o sha256_x8_avx2.o sha256-ssse3-$(CONFIG_AS_SHA256_NI) += sha256_ni_asm.o obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o
arch/x86/crypto/sha256_mb_mgr_datastruct.S new file +304 -0
304 lines not on this page · raw message
arch/x86/crypto/sha256_ssse3_glue.c +481 -42
603 lines not on this page · raw message
arch/x86/crypto/sha256_x8_avx2.S new file +596 -0
596 lines not on this page · raw message
Signature · 1 line
2.39.5
Hi Herbert,
Quoted 19 lines · Herbert Xu
Multibuffer hashing was a constant sore while it was part of the kernel. It was very buggy and unnecessarily complex. Finally it was removed when it had been broken for a while without anyone noticing.
Peace reigned in its absence, until Eric Biggers made a proposal for its comeback :)
The issue is that the SHA algorithm (and possibly others) is inherently not parallelisable. Therefore the only way to exploit parallelism on modern CPUs is to hash multiple indendent streams of data.
Eric's proposal is a simple interface bolted onto shash that takes two streams of data of identical length. I thought the limitation of two was too small, and Eric addressed that in his latest version:
Quoted 8 lines
However, I still disliked the addition of this to shash as it meant that users would have to spend extra effort in order to accumulate and maintain multiple streams of data.
My preference is to use ahash as the basis of multibuffer, because its request object interface is perfectly suited to chaining.
As I've explained before, I think your proposed approach is much too complex, inefficient, and broken compared to my much simpler patchset
https://lore.kernel.org/linux-crypto/[email protected]/.
If this is really going to be the API for multibuffer hashing, then I'm not very interested in using or contributing to it.
The much larger positive diffstat in your patchset alone should speak for itself, especially when it doesn't include essential pieces that were included in my smaller patchset, such as self-tests, dm-verity and fs-verity support, SHA-NI and ARMv8 CE support. Note that due to the complexity of your API, it would require far more updates to the tests in order to cover all the new edge cases. This patchset also removes the shash support for sha256-avx2, which is not realistic, as there are still ~100 users of shash in the kernel.
You say that your API is needed so that users don't need to "spend extra effort in order to accumulate and maintain multiple streams of data." That's incorrect, though. The users, e.g. {dm,fs}-verity, will need to do that anyway even with your API. I think this would have been clear if you had tried to update them to use your API.
With this patchset I am also seeing random crashes in the x86 sha256 glue code, and all multibuffer SHA256 hashes come back as all-zeroes. Bugs like this were predictable, of course. There's a high amount of complexity inherent in the ahash request chaining approach, both at the API layer and in the per-algorithm glue code. It will be hard to get everything right. And I am just submitting 8 equal-length requests, so I haven't even tried any of the edge cases that your proposed API allows, like submitting requests that aren't synced up properly.
I don't think it's worth the time for me to try to debug and fix your code and add the missing pieces, when we could just choose a much simpler design that would result in far fewer bugs. Especially for cryptographic code, choosing sound designs that minimize the number of bugs should be the highest priority.
I understand that you're trying to contribute something useful, and perhaps solve a wider set of problems than I set out to solve. The reality, though, is that this patchset is creating more problems than it's solving. Compared to my patchset, it makes things harder, more error-prone, and less efficient for the users who actually want the multibuffer hashing, and likewise for wiring it up to the low-level algorithms. It also doesn't bring us meaningfully closer to applying multibuffer crypto in other applications like IPsec where it will be very difficult to apply, is irrelevant for the most common algorithm used, and would at best provide less benefit than other easier-to-implement optimizations.
The virtual address to support in ahash is a somewhat nice addition, but it could go in independently, and ultimately it seems not that useful, given that users could just use shash or lib/crypto/ instead. shash will still have less overhead, and lib/crypto/ even less, despite ahash getting slightly better. Also, users who actually care about old-school style crypto accelerators need pages + async processing anyway for optimal performance, especially given this patchset's proposed approach of handling virtual addresses using a bounce page.
If you're really interested in the AVX2 multibuffer SHA256 for some reason, I'd be willing to clean up that assembly code and wire it up to the much simpler API that I proposed. Despite the favorable microbenchmark result, this would be of limited use, for various reasons that I've explained before. But it could be done if desired, and it would be much simpler than what you have.
- Eric
Herbert Xu
maintainerQuoted 6 lines · Eric Biggers
You say that your API is needed so that users don't need to "spend extra effort in order to accumulate and maintain multiple streams of data." That's incorrect, though. The users, e.g. {dm,fs}-verity, will need to do that anyway even with your API. I think this would have been clear if you had tried to update them to use your API.
It's a lot easier once you switch them back to dynamically allocated requests instead of having them on the stack. Storing the hash state on the stack of course limits your ability to aggregate the hash operations since each one is hundreds-of-bytes long.
We could also introduce SYNC_AHASH_REQ_ON_STACK like we do for skcipher but I think we should move away from that for the cases where aggregation makes sense.
Note that when I say switching back to ahash, I'm not talking about switching back to an asynchronous API. If you're happy with the synchronous offerings then you're totally free to use ahash with synchronous-only implementations, just like skcipher.
Quoted 2 lines
With this patchset I am also seeing random crashes in the x86 sha256 glue code, and all multibuffer SHA256 hashes come back as all-zeroes. Bugs like this were
Guilty as charged. I haven't tested this at all apart from timing the speed.
However, adding a proper test is trivial. We already have the ahash_requests ready to go so they just have to be chained together and submitted en-masse.
Quoted 5 lines
If you're really interested in the AVX2 multibuffer SHA256 for some reason, I'd be willing to clean up that assembly code and wire it up to the much simpler API that I proposed. Despite the favorable microbenchmark result, this would be of limited use, for various reasons that I've explained before. But it could be done if desired, and it would be much simpler than what you have.
No I have zero interest in AVX2. I simply picked that because it was already in the kernel git history and I wasn't certain whether my CPU is recent enough to see much of a benefit from your SHA-NI code.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt