Multibuffer hashing take two
Herbert Xu
maintainerv2 cover Multibuffer hashing take two
This patch-set introduces two additions to the ahash interface. First of all request chaining is added so that an arbitrary number of requests can be submitted in one go. Incidentally this also reduces the cost of indirect calls by amortisation.
It then adds virtual address support to ahash. This allows the user to supply a virtual address as the input instead of an SG list.
This is assumed to be not DMA-capable so it is always copied before it's passed to an existing ahash driver. New drivers can elect to take virtual addresses directly. Of course existing shash algorithms are able to take virtual addresses without any copying.
The next patch resurrects the old SHA2 AVX2 muiltibuffer code as a proof of concept that this API works. The result shows that with a full complement of 8 requests, this API is able to achieve parity with the more modern but single-threaded SHA-NI code. This passes the multibuffer fuzz tests.
Finally introduce a sync hash interface that is similar to the sync skcipher interface. This will replace the shash interface for users. Use it in fsverity and enable multibuffer hashing.
Eric Biggers (1):
fsverity: improve performance by using multibuffer hashing
Herbert Xu (10):
crypto: ahash - Only save callback and data in ahash_save_req
crypto: x86/ghash - Use proper helpers to clone request
crypto: hash - Add request chaining API
crypto: tcrypt - Restore multibuffer ahash tests
crypto: ahash - Add virtual address support
crypto: ahash - Set default reqsize from ahash_alg
crypto: testmgr - Add multibuffer hash testing
crypto: x86/sha2 - Restore multibuffer AVX2 support
crypto: hash - Add sync hash interface
fsverity: Use sync hash instead of shash
arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/ghash-clmulni-intel_glue.c | 23 +- arch/x86/crypto/sha256_mb_mgr_datastruct.S | 304 +++++++++++ arch/x86/crypto/sha256_ssse3_glue.c | 540 ++++++++++++++++-- arch/x86/crypto/sha256_x8_avx2.S | 598 ++++++++++++++++++++ crypto/ahash.c | 605 ++++++++++++++++++--- crypto/algapi.c | 2 +- crypto/tcrypt.c | 231 ++++++++ crypto/testmgr.c | 132 ++++- fs/verity/fsverity_private.h | 4 +- fs/verity/hash_algs.c | 41 +- fs/verity/verify.c | 179 +++++- include/crypto/algapi.h | 11 + include/crypto/hash.h | 172 +++++- include/crypto/internal/hash.h | 17 +- include/linux/crypto.h | 24 + 16 files changed, 2659 insertions(+), 226 deletions(-) create mode 100644 arch/x86/crypto/sha256_mb_mgr_datastruct.S create mode 100644 arch/x86/crypto/sha256_x8_avx2.S
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 1/11 crypto: ahash - Only save callback and data in ahash_save_req
As unaligned operations are supported by the underlying algorithm, ahash_save_req and ahash_restore_req can be greatly simplified to only preserve the callback and data.
--- crypto/ahash.c | 97 ++++++++++++++++--------------------------- include/crypto/hash.h | 3 -- 2 files changed, 35 insertions(+), 65 deletions(-)
crypto/ahash.c +35 -62
@@ -27,6 +27,12 @@#define CRYPTO_ALG_TYPE_AHASH_MASK 0x0000000e +struct ahash_save_req_state { +struct ahash_request *req; +crypto_completion_t compl; +void *data; +}; +/* * For an ahash tfm that is using an shash algorithm (instead of an ahash * algorithm), this returns the underlying shash tfm.
@@ -262,67 +268,34 @@ int crypto_ahash_init(struct ahash_request *req) }EXPORT_SYMBOL_GPL (crypto_ahash_init); -static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt, -bool has_state) +static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -unsigned int ds =crypto_ahash_digestsize (tfm); -struct ahash_request *subreq; -unsigned int subreq_size; -unsigned int reqsize; -u8 *result; +struct ahash_save_req_state *state;gfp_t gfp;u32 flags; - subreq_size =sizeof (*subreq); - reqsize =crypto_ahash_reqsize (tfm); - reqsize =ALIGN (reqsize,crypto_tfm_ctx_alignment ()); - subreq_size += reqsize; - subreq_size += ds; - flags =ahash_request_flags (req); gfp = (flags &CRYPTO_TFM_REQ_MAY_SLEEP ) ?GFP_KERNEL :GFP_ATOMIC ; - subreq =kmalloc (subreq_size, gfp); -if (!subreq) + state =kmalloc (sizeof (*state), gfp); +if (!state)return -ENOMEM ; -ahash_request_set_tfm (subreq, tfm); -ahash_request_set_callback (subreq, flags, cplt, req); - - result = (u8 *)(subreq +1 ) + reqsize; - -ahash_request_set_crypt (subreq, req->src, result, req->nbytes); - -if (has_state) { -void *state; - - state =kmalloc (crypto_ahash_statesize (tfm), gfp); -if (!state) { -kfree (subreq); -return -ENOMEM ; - } - -crypto_ahash_export (req, state); -crypto_ahash_import (subreq, state); -kfree_sensitive (state); - } - - req->priv = subreq; + state->compl = req->base.complete; + state->data = req->base.data; + req->base.complete = cplt; + req->base.data = state; + state->req = req;return 0 ; } -static void ahash_restore_req (struct ahash_request *req,int err) +static void ahash_restore_req (struct ahash_request *req) { -struct ahash_request *subreq = req->priv; +struct ahash_save_req_state *state = req->base.data; -if (!err) -memcpy (req->result, subreq->result, -crypto_ahash_digestsize (crypto_ahash_reqtfm (req))); - - req->priv =NULL ; - -kfree_sensitive (subreq); + req->base.complete = state->compl; + req->base.data = state->data; +kfree (state); }int crypto_ahash_update (struct ahash_request *req)
@@ -374,51 +347,51 @@ EXPORT_SYMBOL_GPL(crypto_ahash_digest);static void ahash_def_finup_done2 (void *data,int err) { -struct ahash_request *areq = data; +struct ahash_save_req_state *state = data; +struct ahash_request *areq = state->req;if (err == -EINPROGRESS )return ; -ahash_restore_req (areq, err); - +ahash_restore_req (areq);ahash_request_complete (areq, err); }static int ahash_def_finup_finish1 (struct ahash_request *req,int err) { -struct ahash_request *subreq = req->priv; -if (err) goto out; - subreq->base.complete = ahash_def_finup_done2; + req->base.complete = ahash_def_finup_done2; - err =crypto_ahash_alg (crypto_ahash_reqtfm (req))->final (subreq); + err =crypto_ahash_alg (crypto_ahash_reqtfm (req))->final (req);if (err == -EINPROGRESS || err == -EBUSY )return err; out: -ahash_restore_req (req, err); +ahash_restore_req (req);return err; }static void ahash_def_finup_done1 (void *data,int err) { -struct ahash_request *areq = data; -struct ahash_request *subreq; +struct ahash_save_req_state *state0 = data; +struct ahash_save_req_state state; +struct ahash_request *areq; + state = *state0; + areq = state.req;if (err == -EINPROGRESS ) goto out; - subreq = areq->priv; - subreq->base.flags &=CRYPTO_TFM_REQ_MAY_BACKLOG ; + areq->base.flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP ; err =ahash_def_finup_finish1 (areq, err);if (err == -EINPROGRESS || err == -EBUSY )return ; out: -ahash_request_complete (areq, err); + state.compl (state.data, err); }static int ahash_def_finup (struct ahash_request *req)
@@ -426,11 +399,11 @@ static int ahash_def_finup(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);int err; - err =ahash_save_req (req, ahash_def_finup_done1, true); + err =ahash_save_req (req, ahash_def_finup_done1);if (err)return err; - err =crypto_ahash_alg (tfm)->update (req->priv); + err =crypto_ahash_alg (tfm)->update (req);if (err == -EINPROGRESS || err == -EBUSY )return err;
include/crypto/hash.h +0 -3
@@ -55,9 +55,6 @@ struct ahash_request {struct scatterlist *src;u8 *result; -/* This field may only be used by the ahash API code. */ -void *priv; - void *__ctx[] CRYPTO_MINALIGN_ATTR; };
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 2/11 crypto: x86/ghash - Use proper helpers to clone request
In reply to Herbert XuRather than copying a request by hand with memcpy, use the correct API helpers to setup the new request. This will matter once the API helpers start setting up chained requests as a simple memcpy will break chaining.
--- arch/x86/crypto/ghash-clmulni-intel_glue.c | 23 ++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
arch/x86/crypto/ghash-clmulni-intel_glue.c +17 -6
@@ -189,6 +189,20 @@ static int ghash_async_init(struct ahash_request *req)return crypto_shash_init (desc); } +static void ghash_init_cryptd_req (struct ahash_request *req) +{ +struct ahash_request *cryptd_req =ahash_request_ctx (req); +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ghash_async_ctx *ctx =crypto_ahash_ctx (tfm); +struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; + +ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base); +ahash_request_set_callback (cryptd_req, req->base.flags, + req->base.complete, req->base.data); +ahash_request_set_crypt (cryptd_req, req->src, req->result, + req->nbytes); +} +static int ghash_async_update (struct ahash_request *req) {struct ahash_request *cryptd_req =ahash_request_ctx (req);
@@ -198,8 +212,7 @@ static int ghash_async_update(struct ahash_request *req)if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) { -memcpy (cryptd_req, req,sizeof (*req)); -ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base); +ghash_init_cryptd_req (req);return crypto_ahash_update (cryptd_req); }else {struct shash_desc *desc =cryptd_shash_desc (cryptd_req);
@@ -216,8 +229,7 @@ static int ghash_async_final(struct ahash_request *req)if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) { -memcpy (cryptd_req, req,sizeof (*req)); -ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base); +ghash_init_cryptd_req (req);return crypto_ahash_final (cryptd_req); }else {struct shash_desc *desc =cryptd_shash_desc (cryptd_req);
@@ -257,8 +269,7 @@ static int ghash_async_digest(struct ahash_request *req)if (!crypto_simd_usable () || (in_atomic () &&cryptd_ahash_queued (cryptd_tfm))) { -memcpy (cryptd_req, req,sizeof (*req)); -ahash_request_set_tfm (cryptd_req, &cryptd_tfm->base); +ghash_init_cryptd_req (req);return crypto_ahash_digest (cryptd_req); }else {struct shash_desc *desc =cryptd_shash_desc (cryptd_req);
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 3/11 crypto: hash - Add request chaining API
In reply to Herbert XuThis adds request chaining to the ahash interface. Request chaining allows multiple requests to be submitted in one shot. An algorithm can elect to receive chained requests by setting the flag CRYPTO_ALG_REQ_CHAIN. If this bit is not set, the API will break up chained requests and submit them one-by-one.
A new err field is added to struct crypto_async_request to record the return value for each individual request.
--- crypto/ahash.c | 261 +++++++++++++++++++++++++++++---- crypto/algapi.c | 2 +- include/crypto/algapi.h | 11 ++ include/crypto/hash.h | 28 ++-- include/crypto/internal/hash.h | 10 ++ include/linux/crypto.h | 24 +++ 6 files changed, 299 insertions(+), 37 deletions(-)
crypto/ahash.c +235 -26
@@ -28,11 +28,19 @@#define CRYPTO_ALG_TYPE_AHASH_MASK 0x0000000estruct ahash_save_req_state { -struct ahash_request *req; +struct list_head head; +struct ahash_request *req0; +struct ahash_request *cur; +int (*op)(struct ahash_request *req);crypto_completion_t compl;void *data; }; +static void ahash_reqchain_done (void *data,int err); +static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt); +static void ahash_restore_req (struct ahash_request *req); +static int ahash_def_finup (struct ahash_request *req); +/* * For an ahash tfm that is using an shash algorithm (instead of an ahash * algorithm), this returns the underlying shash tfm.
@@ -256,24 +264,145 @@ int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key, }EXPORT_SYMBOL_GPL (crypto_ahash_setkey); +static int ahash_reqchain_finish (struct ahash_save_req_state *state, +int err,u32 mask) +{ +struct ahash_request *req0 = state->req0; +struct ahash_request *req = state->cur; +struct ahash_request *n; + + req->base.err = err; + +if (req != req0) +list_add_tail (&req->base.list, &req0->base.list); + +list_for_each_entry_safe (req, n, &state->head, base.list) { +list_del_init (&req->base.list); + + req->base.flags &= mask; + req->base.complete = ahash_reqchain_done; + req->base.data = state; + state->cur = req; + err = state->op (req); + +if (err == -EINPROGRESS ) { +if (!list_empty (&state->head)) + err = -EBUSY ; + goto out; + } + +if (err == -EBUSY ) + goto out; + + req->base.err = err; +list_add_tail (&req->base.list, &req0->base.list); + } + +ahash_restore_req (req0); + +out: +return err; +} + +static void ahash_reqchain_done (void *data,int err) +{ +struct ahash_save_req_state *state = data; +crypto_completion_t compl = state->compl; + + data = state->data; + +if (err == -EINPROGRESS ) { +if (!list_empty (&state->head)) +return ; + goto notify; + } + + err =ahash_reqchain_finish (state, err,CRYPTO_TFM_REQ_MAY_BACKLOG ); +if (err == -EBUSY ) +return ; + +notify: +compl (data, err); +} + +static int ahash_do_req_chain (struct ahash_request *req, +int (*op)(struct ahash_request *req)) +{ +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ahash_save_req_state *state; +struct ahash_save_req_state state0; +int err; + +if (!ahash_request_chained (req) ||crypto_ahash_req_chain (tfm)) +return op (req); + + state = &state0; + +if (ahash_is_async (tfm)) { + err =ahash_save_req (req, ahash_reqchain_done); +if (err) { +struct ahash_request *r2; + + req->base.err = err; +list_for_each_entry (r2, &req->base.list, base.list) + r2->base.err = err; + +return err; + } + + state = req->base.data; + } + + state->op = op; + state->cur = req; +INIT_LIST_HEAD (&state->head); +list_splice_init (&req->base.list, &state->head); + + err =op (req); +if (err == -EBUSY || err == -EINPROGRESS ) +return -EBUSY ; + +return ahash_reqchain_finish (state, err, ~0 ); +} +int crypto_ahash_init (struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return crypto_shash_init (prepare_shash_desc (req, tfm)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; + + err =crypto_shash_init (prepare_shash_desc (req, tfm)); + req->base.err = err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =prepare_shash_desc (r2, tfm); + r2->base.err =crypto_shash_init (desc); + } + +return err; + } +if (crypto_ahash_get_flags (tfm) &CRYPTO_TFM_NEED_KEY )return -ENOKEY ; -return crypto_ahash_alg (tfm)->init (req); + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->init); }EXPORT_SYMBOL_GPL (crypto_ahash_init);static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt) { +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct ahash_save_req_state *state;gfp_t gfp;u32 flags; +if (!ahash_is_async (tfm)) +return 0 ; + flags =ahash_request_flags (req); gfp = (flags &CRYPTO_TFM_REQ_MAY_SLEEP ) ?GFP_KERNEL :GFP_ATOMIC ; state =kmalloc (sizeof (*state), gfp);
@@ -284,14 +413,20 @@ static int ahash_save_req(struct ahash_request *req, crypto_completion_t cplt) state->data = req->base.data; req->base.complete = cplt; req->base.data = state; - state->req = req; + state->req0 = req;return 0 ; }static void ahash_restore_req (struct ahash_request *req) { -struct ahash_save_req_state *state = req->base.data; +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ahash_save_req_state *state; + +if (!ahash_is_async (tfm)) +return ; + + state = req->base.data; req->base.complete = state->compl; req->base.data = state->data;
@@ -302,10 +437,24 @@ int crypto_ahash_update(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return shash_ahash_update (req,ahash_request_ctx (req)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; -return crypto_ahash_alg (tfm)->update (req); + err =shash_ahash_update (req,ahash_request_ctx (req)); + req->base.err = err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =ahash_request_ctx (r2); + r2->base.err =shash_ahash_update (r2, desc); + } + +return err; + } + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->update); }EXPORT_SYMBOL_GPL (crypto_ahash_update);
@@ -313,10 +462,24 @@ int crypto_ahash_final(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return crypto_shash_final (ahash_request_ctx (req), req->result); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; -return crypto_ahash_alg (tfm)->final (req); + err =crypto_shash_final (ahash_request_ctx (req), req->result); + req->base.err = err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =ahash_request_ctx (r2); + r2->base.err =crypto_shash_final (desc, r2->result); + } + +return err; + } + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->final); }EXPORT_SYMBOL_GPL (crypto_ahash_final);
@@ -324,10 +487,27 @@ int crypto_ahash_finup(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return shash_ahash_finup (req,ahash_request_ctx (req)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; -return crypto_ahash_alg (tfm)->finup (req); + err =shash_ahash_finup (req,ahash_request_ctx (req)); + req->base.err = err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =ahash_request_ctx (r2); + r2->base.err =shash_ahash_finup (r2, desc); + } + +return err; + } + +if (!crypto_ahash_alg (tfm)->finup) +return ahash_def_finup (req); + +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->finup); }EXPORT_SYMBOL_GPL (crypto_ahash_finup);
@@ -335,20 +515,34 @@ int crypto_ahash_digest(struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -if (likely (tfm->using_shash)) -return shash_ahash_digest (req,prepare_shash_desc (req, tfm)); +if (likely (tfm->using_shash)) { +struct ahash_request *r2; +int err; + + err =shash_ahash_digest (req,prepare_shash_desc (req, tfm)); + req->base.err = err; + +list_for_each_entry (r2, &req->base.list, base.list) { +struct shash_desc *desc; + + desc =prepare_shash_desc (r2, tfm); + r2->base.err =shash_ahash_digest (r2, desc); + } + +return err; + }if (crypto_ahash_get_flags (tfm) &CRYPTO_TFM_NEED_KEY )return -ENOKEY ; -return crypto_ahash_alg (tfm)->digest (req); +return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->digest); }EXPORT_SYMBOL_GPL (crypto_ahash_digest);static void ahash_def_finup_done2 (void *data,int err) {struct ahash_save_req_state *state = data; -struct ahash_request *areq = state->req; +struct ahash_request *areq = state->req0;if (err == -EINPROGRESS )return ;
@@ -359,12 +553,15 @@ static void ahash_def_finup_done2(void *data, int err)static int ahash_def_finup_finish1 (struct ahash_request *req,int err) { +struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +if (err) goto out; - req->base.complete = ahash_def_finup_done2; +if (ahash_is_async (tfm)) + req->base.complete = ahash_def_finup_done2; - err =crypto_ahash_alg (crypto_ahash_reqtfm (req))->final (req); + err =crypto_ahash_final (req);if (err == -EINPROGRESS || err == -EBUSY )return err;
@@ -380,7 +577,7 @@ static void ahash_def_finup_done1(void *data, int err)struct ahash_request *areq; state = *state0; - areq = state.req; + areq = state.req0;if (err == -EINPROGRESS ) goto out;
@@ -396,14 +593,13 @@ static void ahash_def_finup_done1(void *data, int err)static int ahash_def_finup (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);int err; err =ahash_save_req (req, ahash_def_finup_done1);if (err)return err; - err =crypto_ahash_alg (tfm)->update (req); + err =crypto_ahash_update (req);if (err == -EINPROGRESS || err == -EBUSY )return err;
@@ -618,8 +814,6 @@ static int ahash_prepare_alg(struct ahash_alg *alg) base->cra_type = &crypto_ahash_type; base->cra_flags |=CRYPTO_ALG_TYPE_AHASH ; -if (!alg->finup) - alg->finup = ahash_def_finup;if (!alg->setkey) alg->setkey = ahash_nosetkey;
@@ -690,5 +884,20 @@ int ahash_register_instance(struct crypto_template *tmpl, }EXPORT_SYMBOL_GPL (ahash_register_instance); +void ahash_request_free (struct ahash_request *req) +{ +struct ahash_request *tmp; +struct ahash_request *r2; + +if (unlikely (!req)) +return ; + +list_for_each_entry_safe (r2, tmp, &req->base.list, base.list) +kfree_sensitive (r2); + +kfree_sensitive (req); +} +EXPORT_SYMBOL_GPL (ahash_request_free); +MODULE_LICENSE ("GPL" );MODULE_DESCRIPTION ("Asynchronous cryptographic hash type" );
crypto/algapi.c +1 -1
@@ -955,7 +955,7 @@ struct crypto_async_request *crypto_dequeue_request(struct crypto_queue *queue) queue->backlog = queue->backlog->next; request = queue->list.next; -list_del (request); +list_del_init (request); return list_entry(request, struct crypto_async_request, list); }
include/crypto/algapi.h +11 -0
@@ -11,6 +11,7 @@#include <linux/align.h> #include <linux/cache.h> #include <linux/crypto.h> +#include <linux/list.h> #include <linux/types.h> #include <linux/workqueue.h>
@@ -271,4 +272,14 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm) return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK; } +static inline bool crypto_request_chained (struct crypto_async_request *req) +{ +return !list_empty (&req->list); +} + +static inline bool crypto_tfm_req_chain (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags &CRYPTO_ALG_REQ_CHAIN ; +} +#endif /* _CRYPTO_ALGAPI_H */
include/crypto/hash.h +18 -10
@@ -572,16 +572,7 @@ static inline struct ahash_request *ahash_request_alloc_noprof(* ahash_request_free() - zeroize and free the request data structure * @req: request data structure cipher handle to be freed */ -static inline void ahash_request_free (struct ahash_request *req) -{ -kfree_sensitive (req); -} - -static inline void ahash_request_zero (struct ahash_request *req) -{ -memzero_explicit (req,sizeof (*req) + -crypto_ahash_reqsize (crypto_ahash_reqtfm (req))); -} +void ahash_request_free (struct ahash_request *req);static inline struct ahash_request *ahash_request_cast (struct crypto_async_request *req)
@@ -622,6 +613,7 @@ static inline void ahash_request_set_callback(struct ahash_request *req, req->base.complete = compl; req->base.data = data; req->base.flags = flags; +crypto_reqchain_init (&req->base); }/**
@@ -646,6 +638,12 @@ static inline void ahash_request_set_crypt(struct ahash_request *req, req->result = result; } +static inline void ahash_request_chain (struct ahash_request *req, +struct ahash_request *head) +{ +crypto_request_chain (&req->base, &head->base); +} +/** * DOC: Synchronous Message Digest API *
@@ -947,4 +945,14 @@ static inline void shash_desc_zero(struct shash_desc *desc) sizeof(*desc) + crypto_shash_descsize(desc->tfm)); } +static inline int ahash_request_err (struct ahash_request *req) +{ +return req->base.err; +} + +static inline bool ahash_is_async (struct crypto_ahash *tfm) +{ +return crypto_tfm_is_async (&tfm->base); +} +#endif /* _CRYPTO_HASH_H */
include/crypto/internal/hash.h +10 -0
@@ -270,5 +270,15 @@ static inline struct crypto_shash *__crypto_shash_cast(struct crypto_tfm *tfm) return container_of(tfm, struct crypto_shash, base); } +static inline bool ahash_request_chained (struct ahash_request *req) +{ +return crypto_request_chained (&req->base); +} + +static inline bool crypto_ahash_req_chain (struct crypto_ahash *tfm) +{ +return crypto_tfm_req_chain (&tfm->base); +} +#endif /* _CRYPTO_INTERNAL_HASH_H */
include/linux/crypto.h +24 -0
@@ -13,6 +13,8 @@#define _LINUX_CRYPTO_H#include <linux/completion.h> +#include <linux/errno.h> +#include <linux/list.h> #include <linux/refcount.h> #include <linux/slab.h> #include <linux/types.h>
@@ -124,6 +126,9 @@*/ #define CRYPTO_ALG_FIPS_INTERNAL 0x00020000 +/* Set if the algorithm supports request chains. */ +#define CRYPTO_ALG_REQ_CHAIN 0x00040000 +/* * Transform masks and values (for crt_flags). */
@@ -174,6 +179,7 @@ struct crypto_async_request {struct crypto_tfm *tfm;u32 flags; +int err; };/**
@@ -540,5 +546,23 @@ int crypto_comp_decompress(struct crypto_comp *tfm, const u8 *src, unsigned int slen, u8 *dst, unsigned int *dlen); +static inline void crypto_reqchain_init (struct crypto_async_request *req) +{ + req->err = -EINPROGRESS ; +INIT_LIST_HEAD (&req->list); +} + +static inline void crypto_request_chain (struct crypto_async_request *req, +struct crypto_async_request *head) +{ + req->err = -EINPROGRESS ; +list_add_tail (&req->list, &head->list); +} + +static inline bool crypto_tfm_is_async (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags &CRYPTO_ALG_ASYNC ; +} +#endif /* _LINUX_CRYPTO_H */
Signature · 1 line
2.39.5
Re: [v2 PATCH 03/11] crypto: hash - Add request chaining API
Hi Herbert,
Quoted 10 lines · Herbert Xu
This adds request chaining to the ahash interface. Request chaining allows multiple requests to be submitted in one shot. An algorithm can elect to receive chained requests by setting the flag
CRYPTO_ALG_REQ_CHAIN. If this bit is not set, the API will break up chained requests and submit them one-by-one.A new err field is added to struct
crypto_async_requestto record the return value for each individual request.Signed-off-by Herbert Xu <[email protected]>
Quoted 8 lines
--- crypto/ahash.c | 261 +++++++++++++++++++++++++++++---- crypto/algapi.c | 2 +- include/crypto/algapi.h | 11 ++ include/crypto/hash.h | 28 ++-- include/crypto/internal/hash.h | 10 ++ include/linux/crypto.h | 24 +++ 6 files changed, 299 insertions(+), 37 deletions(-)
The following patch seems to be breaking selftests in SA2UL driver.
The failure signature:
root@j721e-evm:~# modprobe sa2ul [ 32.254126] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c root@j721e-evm:~# [ 32.374996] Unable to handle kernel paging request at virtual address fefefefefefeff46 [ 32.401815] Unable to handle kernel paging request at virtual address fefefefefefeff46 [ 32.449576] Unable to handle kernel paging request at virtual address fefefefefefeff46 [ 32.459025] Mem abort info: [ 32.461812] ESR = 0x0000000096000044 [ 32.480762] Mem abort info: [ 32.503389] ESR = 0x0000000096000044 [ 32.512483] EC = 0x25: DABT (current EL), IL = 32 bits [ 32.519478] Mem abort info: [ 32.534472] EC = 0x25: DABT (current EL), IL = 32 bits [ 32.542380] ESR = 0x0000000096000044 [ 32.546123] EC = 0x25: DABT (current EL), IL = 32 bits [ 32.554977] SET = 0, FnV = 0 [ 32.572112] SET = 0, FnV = 0 [ 32.579134] EA = 0, S1PTW = 0 [ 32.597889] EA = 0, S1PTW = 0 [ 32.603045] FSC = 0x04: level 0 translation fault [ 32.615500] SET = 0, FnV = 0 [ 32.628186] FSC = 0x04: level 0 translation fault [ 32.645274] EA = 0, S1PTW = 0 [ 32.651268] Data abort info: [ 32.654145] ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000 [ 32.675265] Data abort info: [ 32.678614] FSC = 0x04: level 0 translation fault [ 32.701391] ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000 [ 32.721251] Data abort info: [ 32.725864] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 [ 32.742907] ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000 [ 32.751647] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 [ 32.770854] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 32.790381] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 [ 32.795591] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 32.807232] [fefefefefefeff46] address between user and kernel address ranges [ 32.826504] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 32.832211] [fefefefefefeff46] address between user and kernel address ranges [ 32.854007] Internal error: Oops: 0000000096000044 [#1] SMP [ 32.859661] Modules linked in: des_generic libdes cbc sa2ul authenc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 pinctrl_tps6594 tps6594_esm tps6594_regulator gpio_regmap tps6594_pfsm ti_am335x_adc kfifo_buf pru_rproc cdns3 irq_pruss_intc cdns_pltfrm cdns_usb_common snd_soc_j721e_evm display_connector phy_can_transceiver phy_j721e_wiz omap_mailbox ti_k3_r5_remoteproc tidss tps6594_i2c drm_client_lib cdns_mhdp8546 drm_dma_helper tps6594_core at24 drm_display_helper k3_j72xx_bandgap drm_kms_helper m_can_platform m_can ti_am335x_tscadc pruss snd_soc_davinci_mcasp snd_soc_ti_udma ti_j721e_ufs can_dev snd_soc_ti_edma ti_k3_dsp_remoteproc snd_soc_ti_sdma cdns3_ti snd_soc_pcm3168a_i2c snd_soc_pcm3168a rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [ 32.929820] CPU: 0 UID: 0 PID: 1253 Comm: cryptomgr_test Not tainted 6.14.0-rc7-next-20250324-build-configs-dirty #2 PREEMPT [ 32.941098] Hardware name: Texas Instruments J721e EVM (DT) [ 32.946653] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 32.953598] pc : crypto_ahash_init+0x6c/0xf0 [ 32.957866] lr : crypto_ahash_init+0x50/0xf0 [ 32.962124] sp : ffff8000851cb590 [ 32.965425] x29: ffff8000851cb590 x28: 0000000000000000 x27: ffff8000851cb788 [ 32.972546] x26: ffff000802dace00 x25: 0000000000000000 x24: ffff000804294010 [ 32.979667] x23: ffff80007be3fb30 x22: 0000000000000000 x21: ffff0008095c1250 [ 32.986787] x20: ffff000802dace90 x19: fefefefefefefefe x18: 00000000ffffffff [ 32.993908] x17: 0000000000373931 x16: 2d3732322d34322d x15: ffff8000851cb740 [ 33.001028] x14: ffff8001051cbaa7 x13: 0000000000000000 x12: 0000000000000000 [ 33.008148] x11: 0000000000000100 x10: 0000000000000001 x9 : ffff80007be3fadc [ 33.015268] x8 : ffff8000851cb668 x7 : 0000000000000000 x6 : 0010000000000000 [ 33.022388] x5 : efcdab8967452301 x4 : 1032547698badcfe x3 : 00000000c3d2e1f0 [ 33.029509] x2 : 0000000000000000 x1 : ffff0008095c1280 x0 : 0000000000000000 [ 33.036629] Call trace: [ 33.039065] crypto_ahash_init+0x6c/0xf0 (P) [ 33.043325] sa_sha_init+0x4c/0xa0 [sa2ul] [ 33.047419] ahash_do_req_chain+0x144/0x280 [ 33.051591] crypto_ahash_init+0xc8/0xf0 [ 33.055502] do_ahash_op+0x34/0xb8 [ 33.058895] test_ahash_vec_cfg+0x3e4/0x800 [ 33.063068] test_hash_vec+0xbc/0x230 [ 33.066719] __alg_test_hash+0x288/0x3d8 [ 33.070629] alg_test_hash+0x108/0x1a0 [ 33.074367] alg_test+0x148/0x658 [ 33.077672] cryptomgr_test+0x2c/0x50 [ 33.081322] kthread+0x134/0x218 [ 33.084542] ret_from_fork+0x10/0x20 [ 33.088109] Code: eb13029f 540001e0 d503201f f94012a1 (f9002661) [ 33.094185] ---[ end trace 0000000000000000 ]--- [ 33.100497] [fefefefefefeff46] address between user and kernel address ranges [ 33.114241] Internal error: Oops: 0000000096000044 [#2] SMP [ 33.119887] Modules linked in: des_generic libdes cbc sa2ul authenc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 pinctrl_tps6594 tps6594_esm tps6594_regulator gpio_regmap tps6594_pfsm ti_am335x_adc kfifo_buf pru_rproc cdns3 irq_pruss_intc cdns_pltfrm cdns_usb_common snd_soc_j721e_evm display_connector phy_can_transceiver phy_j721e_wiz omap_mailbox ti_k3_r5_remoteproc tidss tps6594_i2c drm_client_lib cdns_mhdp8546 drm_dma_helper tps6594_core at24 drm_display_helper k3_j72xx_bandgap drm_kms_helper m_can_platform m_can ti_am335x_tscadc pruss snd_soc_davinci_mcasp snd_soc_ti_udma ti_j721e_ufs can_dev snd_soc_ti_edma ti_k3_dsp_remoteproc snd_soc_ti_sdma cdns3_ti snd_soc_pcm3168a_i2c snd_soc_pcm3168a rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [ 33.190029] CPU: 0 UID: 0 PID: 1256 Comm: cryptomgr_test Tainted: G D 6.14.0-rc7-next-20250324-build-configs-dirty #2 PREEMPT [ 33.202868] Tainted: [D]=DIE [ 33.205737] Hardware name: Texas Instruments J721e EVM (DT) [ 33.211293] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 33.218236] pc : crypto_ahash_init+0x6c/0xf0 [ 33.222498] lr : crypto_ahash_init+0x50/0xf0 [ 33.226756] sp : ffff8000852ab590 [ 33.230057] x29: ffff8000852ab590 x28: 0000000000000000 x27: ffff8000852ab788 [ 33.237178] x26: ffff000807d69a00 x25: 0000000000000000 x24: ffff000804204810 [ 33.244298] x23: ffff80007be3fb30 x22: 0000000000000000 x21: ffff0008095c1890 [ 33.251419] x20: ffff000807d69a90 x19: fefefefefefefefe x18: 00000000ffffffff [ 33.258539] x17: 0000000000000000 x16: 0000000000000000 x15: ffff8000852ab740 [ 33.265659] x14: ffff8001052abaa7 x13: 0000000000000000 x12: 0000000000000000 [ 33.272779] x11: 0000000000000000 x10: ffff00087f7c8308 x9 : ffff80007be3fadc [ 33.279899] x8 : bb67ae8584caa73b x7 : 3c6ef372fe94f82b x6 : a54ff53a5f1d36f1 [ 33.287019] x5 : 510e527fade682d1 x4 : 9b05688c2b3e6c1f x3 : 1f83d9abfb41bd6b [ 33.294139] x2 : 5be0cd19137e2179 x1 : ffff0008095c1100 x0 : 0000000000000000 [ 33.301259] Call trace: [ 33.303695] crypto_ahash_init+0x6c/0xf0 (P) [ 33.307954] sa_sha_init+0x4c/0xa0 [sa2ul] [ 33.312045] ahash_do_req_chain+0x144/0x280 [ 33.316217] crypto_ahash_init+0xc8/0xf0 [ 33.320129] do_ahash_op+0x34/0xb8 [ 33.323520] test_ahash_vec_cfg+0x3e4/0x800 [ 33.327691] test_hash_vec+0xbc/0x230 [ 33.331341] __alg_test_hash+0x288/0x3d8 [ 33.335252] alg_test_hash+0x108/0x1a0 [ 33.338990] alg_test+0x148/0x658 [ 33.342294] cryptomgr_test+0x2c/0x50 [ 33.345944] kthread+0x134/0x218 [ 33.349161] ret_from_fork+0x10/0x20 [ 33.352727] Code: eb13029f 540001e0 d503201f f94012a1 (f9002661) [ 33.358803] ---[ end trace 0000000000000000 ]--- [ 33.363437] Internal error: Oops: 0000000096000044 [#3] SMP [ 33.369081] Modules linked in: des_generic libdes cbc sa2ul authenc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 pinctrl_tps6594 tps6594_esm tps6594_regulator gpio_regmap tps6594_pfsm ti_am335x_adc kfifo_buf pru_rproc cdns3 irq_pruss_intc cdns_pltfrm cdns_usb_common snd_soc_j721e_evm display_connector phy_can_transceiver phy_j721e_wiz omap_mailbox ti_k3_r5_remoteproc tidss tps6594_i2c drm_client_lib cdns_mhdp8546 drm_dma_helper tps6594_core at24 drm_display_helper k3_j72xx_bandgap drm_kms_helper m_can_platform m_can ti_am335x_tscadc pruss snd_soc_davinci_mcasp snd_soc_ti_udma ti_j721e_ufs can_dev snd_soc_ti_edma ti_k3_dsp_remoteproc snd_soc_ti_sdma cdns3_ti snd_soc_pcm3168a_i2c snd_soc_pcm3168a rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [ 33.439214] CPU: 0 UID: 0 PID: 1255 Comm: cryptomgr_test Tainted: G D 6.14.0-rc7-next-20250324-build-configs-dirty #2 PREEMPT [ 33.452052] Tainted: [D]=DIE [ 33.454921] Hardware name: Texas Instruments J721e EVM (DT) [ 33.460476] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 33.467419] pc : crypto_ahash_init+0x6c/0xf0 [ 33.471678] lr : crypto_ahash_init+0x50/0xf0 [ 33.475935] sp : ffff8000851fb590 [ 33.479236] x29: ffff8000851fb590 x28: 0000000000000000 x27: ffff8000851fb788 [ 33.486357] x26: ffff000808b0da00 x25: 0000000000000000 x24: ffff000801270c10 [ 33.493477] x23: ffff80007be3fb30 x22: 0000000000000000 x21: ffff0008095c1490 [ 33.500596] x20: ffff000808b0da90 x19: fefefefefefefefe x18: 00000000ffffffff [ 33.507717] x17: 0000000000000000 x16: 0000000000000000 x15: ffff8000851fb740 [ 33.514837] x14: ffff8001051fbaa7 x13: 0000000000000000 x12: 0000000000000000 [ 33.521956] x11: 0000000000000000 x10: ffff00087f7c8308 x9 : ffff80007be3fadc [ 33.529077] x8 : ffff8000851fb668 x7 : 0000000000000000 x6 : 0010000000000000 [ 33.536197] x5 : bb67ae856a09e667 x4 : a54ff53a3c6ef372 x3 : 9b05688c510e527f [ 33.543316] x2 : 5be0cd191f83d9ab x1 : ffff0008095c1680 x0 : 0000000000000000 [ 33.550437] Call trace: [ 33.552871] crypto_ahash_init+0x6c/0xf0 (P) [ 33.557130] sa_sha_init+0x4c/0xa0 [sa2ul] [ 33.561217] ahash_do_req_chain+0x144/0x280 [ 33.565388] crypto_ahash_init+0xc8/0xf0 [ 33.569299] do_ahash_op+0x34/0xb8 [ 33.572691] test_ahash_vec_cfg+0x3e4/0x800 [ 33.576861] test_hash_vec+0xbc/0x230 [ 33.580512] __alg_test_hash+0x288/0x3d8 [ 33.584423] alg_test_hash+0x108/0x1a0 [ 33.588161] alg_test+0x148/0x658 [ 33.591465] cryptomgr_test+0x2c/0x50 [ 33.595115] kthread+0x134/0x218 [ 33.598334] ret_from_fork+0x10/0x20 [ 33.601898] Code: eb13029f 540001e0 d503201f f94012a1 (f9002661) [ 33.607973] ---[ end trace 0000000000000000 ]--- [ 47.186729] kauditd_printk_skb: 5 callbacks suppressed [ 47.186738] audit: type=1334 audit(1742976993.333:24): prog-id=20 op=UNLOAD [ 47.200991] audit: type=1334 audit(1742976993.345:25): prog-id=19 op=UNLOAD [ 47.208310] audit: type=1334 audit(1742976993.345:26): prog-id=18 op=UNLOAD
Don't have sufficient knowledge to understand what is going wrong with this change, could you help out?
Regards,
Manorit
Herbert Xu
maintainerPATCH 1/1 crypto: sa2ul - Use proper helpers to setup request
Quoted 2 lines · Manorit Chawdhry
The following patch seems to be breaking selftests in SA2UL driver.
Thanks for the report.
This patch should fix the problem:
---8<---
Rather than setting up a request by hand, use the correct API helpers to setup the new request. This is because the API helpers will setup chaining.
Also change the fallback allocation to explicitly request for a sync algorithm as this driver will crash if given an async one.
drivers/crypto/sa2ul.c +16 -50
@@ -1415,22 +1415,13 @@ static int sa_sha_run(struct ahash_request *req) (auth_len >= SA_UNSAFE_DATA_SZ_MIN && auth_len <= SA_UNSAFE_DATA_SZ_MAX)) {struct ahash_request *subreq = &rctx->fallback_req; -int ret =0 ; +int ret;ahash_request_set_tfm (subreq, ctx->fallback.ahash); - subreq->base.flags = req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (subreq, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (subreq, req->src, req->result, auth_len); -crypto_ahash_init (subreq); - - subreq->nbytes = auth_len; - subreq->src = req->src; - subreq->result = req->result; - - ret |=crypto_ahash_update (subreq); - - subreq->nbytes =0 ; - - ret |=crypto_ahash_final (subreq); + ret =crypto_ahash_digest (subreq);return ret; }
@@ -1502,8 +1493,7 @@ static int sa_sha_cra_init_alg(struct crypto_tfm *tfm, const char *alg_base)return ret;if (alg_base) { - ctx->shash =crypto_alloc_shash (alg_base,0 , -CRYPTO_ALG_NEED_FALLBACK ); + ctx->shash =crypto_alloc_shash (alg_base,0 ,0 );if (IS_ERR (ctx->shash)) {dev_err (sa_k3_dev,"base driver %s couldn't be loaded\n" , alg_base);
@@ -1511,8 +1501,7 @@ static int sa_sha_cra_init_alg(struct crypto_tfm *tfm, const char *alg_base) }/* for fallback */ ctx->fallback.ahash = -crypto_alloc_ahash (alg_base,0 , -CRYPTO_ALG_NEED_FALLBACK ); +crypto_alloc_ahash (alg_base,0 ,CRYPTO_ALG_ASYNC );if (IS_ERR (ctx->fallback.ahash)) {dev_err (ctx->dev_data->dev,"Could not load fallback driver\n" );
@@ -1546,54 +1535,38 @@ static int sa_sha_init(struct ahash_request *req) crypto_ahash_digestsize(tfm), rctx);ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req,NULL ,NULL ,0 );return crypto_ahash_init (&rctx->fallback_req); }static int sa_sha_update (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; - rctx->fallback_req.nbytes = req->nbytes; - rctx->fallback_req.src = req->src; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req, req->src,NULL , req->nbytes);return crypto_ahash_update (&rctx->fallback_req); }static int sa_sha_final (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; - rctx->fallback_req.result = req->result; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req,NULL , req->result,0 );return crypto_ahash_final (&rctx->fallback_req); }static int sa_sha_finup (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; - - rctx->fallback_req.nbytes = req->nbytes; - rctx->fallback_req.src = req->src; - rctx->fallback_req.result = req->result; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req, req->src, req->result, req->nbytes);return crypto_ahash_finup (&rctx->fallback_req); }
@@ -1601,12 +1574,8 @@ static int sa_sha_finup(struct ahash_request *req)static int sa_sha_import (struct ahash_request *req,const void *in) {struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = req->base.flags & -CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL );return crypto_ahash_import (&rctx->fallback_req, in); }
@@ -1614,12 +1583,9 @@ static int sa_sha_import(struct ahash_request *req, const void *in)static int sa_sha_export (struct ahash_request *req,void *out) {struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm);struct ahash_request *subreq = &rctx->fallback_req; -ahash_request_set_tfm (subreq, ctx->fallback.ahash); - subreq->base.flags = req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (subreq, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL );return crypto_ahash_export (subreq, out); }
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [PATCH] crypto: sa2ul - Use proper helpers to setup request
Hi Herbert,
Quoted 13 lines · Herbert Xu
Quoted 2 lines · Manorit Chawdhry
The following patch seems to be breaking selftests in SA2UL driver.
Thanks for the report.
This patch should fix the problem:
---8<---
Rather than setting up a request by hand, use the correct API helpers to setup the new request. This is because the API helpers will setup chaining.
Quoted 5 lines
Also change the fallback allocation to explicitly request for a sync algorithm as this driver will crash if given an async one.
Signed-off-by Herbert Xu <[email protected]>
Thanks for the quick fix, though now I see error in import rather than init which was there previously.
root@j721e-evm:~# modprobe sa2ul [ 155.283088] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c root@j721e-evm:~# [ 155.401918] Unable to handle kernel paging request at virtual address fefefefefefefeee [ 155.430127] Unable to handle kernel paging request at virtual address fefefefefefefeee [ 155.463959] Unable to handle kernel paging request at virtual address fefefefefefefeee [ 155.480086] Mem abort info: [ 155.503068] Mem abort info: [ 155.506689] ESR = 0x0000000096000004 [ 155.527264] ESR = 0x0000000096000004 [ 155.531009] EC = 0x25: DABT (current EL), IL = 32 bits [ 155.538758] Mem abort info: [ 155.543371] EC = 0x25: DABT (current EL), IL = 32 bits [ 155.559119] ESR = 0x0000000096000004 [ 155.580125] SET = 0, FnV = 0 [ 155.583176] EA = 0, S1PTW = 0 [ 155.589283] EC = 0x25: DABT (current EL), IL = 32 bits [ 155.607886] SET = 0, FnV = 0 [ 155.610938] EA = 0, S1PTW = 0 [ 155.633650] SET = 0, FnV = 0 [ 155.638300] FSC = 0x04: level 0 translation fault [ 155.667607] FSC = 0x04: level 0 translation fault [ 155.673165] EA = 0, S1PTW = 0 [ 155.686121] Data abort info: [ 155.697270] FSC = 0x04: level 0 translation fault [ 155.709990] Data abort info: [ 155.714312] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 [ 155.736268] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 [ 155.745374] Data abort info: [ 155.763508] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 [ 155.770677] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 [ 155.791894] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 [ 155.801343] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 155.815914] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 [ 155.829811] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 155.847633] [fefefefefefefeee] address between user and kernel address ranges [ 155.859395] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 155.872011] [fefefefefefefeee] address between user and kernel address ranges [ 155.893911] [fefefefefefefeee] address between user and kernel address ranges [ 155.901649] Internal error: Oops: 0000000096000004 [#1] SMP [ 155.907297] Modules linked in: des_generic cbc libdes sa2ul authenc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 tps6594_esm pinctrl_tps6594 tps6594_pfsm gpio_regmap tps6594_regulator ti_am335x_adc pru_rproc kfifo_buf irq_pruss_intc cdns3 cdns_pltfrm cdns_usb_common snd_soc_j721e_evm display_connector phy_can_transceiver omap_mailbox phy_j721e_wiz ti_k3_r5_remoteproc tps6594_i2c tps6594_core at24 tidss k3_j72xx_bandgap drm_client_lib drm_dma_helper cdns_mhdp8546 ti_am335x_tscadc m_can_platform snd_soc_davinci_mcasp drm_display_helper pruss snd_soc_ti_udma m_can snd_soc_ti_edma drm_kms_helper ti_j721e_ufs snd_soc_ti_sdma ti_k3_dsp_remoteproc snd_soc_pcm3168a_i2c cdns3_ti can_dev snd_soc_pcm3168a rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [ 155.977452] CPU: 0 UID: 0 PID: 1252 Comm: cryptomgr_test Not tainted 6.14.0-rc7-next-20250324-build-configs-00001-g82a16a3a2a73-dirty #3 PREEMPT [ 155.990466] Hardware name: Texas Instruments J721e EVM (DT) [ 155.996022] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 156.002966] pc : crypto_ahash_import+0x18/0x68 [ 156.007409] lr : sa_sha_import+0x48/0x60 [sa2ul] [ 156.012024] sp : ffff8000853336f0 [ 156.015326] x29: ffff8000853336f0 x28: 0000000000000000 x27: 0000000000000000 [ 156.022447] x26: ffff000807ed6c00 x25: ffff8000853337e8 x24: 0000000000000000 [ 156.029568] x23: ffff000801307440 x22: ffff800085333c08 x21: ffff000801307400 [ 156.036688] x20: ffff800081510c10 x19: ffff8000814e13e8 x18: 00000000ffffffff [ 156.043809] x17: 0000000000000000 x16: 0010000000000000 x15: ffff800085333740 [ 156.050929] x14: ffff800105333aa7 x13: 0000000000000000 x12: 0000000000000000 [ 156.058049] x11: 0000000000000000 x10: ffff00087f7c8308 x9 : ffff80007be3aa38 [ 156.065170] x8 : ffff000807ed6d50 x7 : fefefefefefefefe x6 : 0101010101010101 [ 156.072291] x5 : 00000000ffffff8d x4 : 0000000000000000 x3 : fefefefefefefefe [ 156.079411] x2 : ffff000807ed6c00 x1 : ffff000803fc9c00 x0 : ffff000807ed6c90 [ 156.086531] Call trace: [ 156.088966] crypto_ahash_import+0x18/0x68 (P) [ 156.093399] sa_sha_import+0x48/0x60 [sa2ul] [ 156.097658] crypto_ahash_import+0x54/0x68 [ 156.101744] test_ahash_vec_cfg+0x638/0x800 [ 156.105916] test_hash_vec+0xbc/0x230 [ 156.109568] __alg_test_hash+0x288/0x3d8 [ 156.113478] alg_test_hash+0x108/0x1a0 [ 156.117217] alg_test+0x148/0x658 [ 156.120520] cryptomgr_test+0x2c/0x50 [ 156.124171] kthread+0x134/0x218 [ 156.127390] ret_from_fork+0x10/0x20 [ 156.130958] Code: d503233f a9bf7bfd 910003fd f9401003 (385f0064) [ 156.137034] ---[ end trace 0000000000000000 ]--- [ 156.147224] Internal error: Oops: 0000000096000004 [#2] SMP [ 156.152873] Modules linked in: des_generic cbc libdes sa2ul authenc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 tps6594_esm pinctrl_tps6594 tps6594_pfsm gpio_regmap tps6594_regulator ti_am335x_adc pru_rproc kfifo_buf irq_pruss_intc cdns3 cdns_pltfrm cdns_usb_common snd_soc_j721e_evm display_connector phy_can_transceiver omap_mailbox phy_j721e_wiz ti_k3_r5_remoteproc tps6594_i2c tps6594_core at24 tidss k3_j72xx_bandgap drm_client_lib drm_dma_helper cdns_mhdp8546 ti_am335x_tscadc m_can_platform snd_soc_davinci_mcasp drm_display_helper pruss snd_soc_ti_udma m_can snd_soc_ti_edma drm_kms_helper ti_j721e_ufs snd_soc_ti_sdma ti_k3_dsp_remoteproc snd_soc_pcm3168a_i2c cdns3_ti can_dev snd_soc_pcm3168a rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [ 156.223015] CPU: 0 UID: 0 PID: 1253 Comm: cryptomgr_test Tainted: G D 6.14.0-rc7-next-20250324-build-configs-00001-g82a16a3a2a73-dirty #3 PREEMPT [ 156.237588] Tainted: [D]=DIE [ 156.240456] Hardware name: Texas Instruments J721e EVM (DT) [ 156.246010] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 156.252953] pc : crypto_ahash_import+0x18/0x68 [ 156.257389] lr : sa_sha_import+0x48/0x60 [sa2ul] [ 156.261998] sp : ffff8000853cb6f0 [ 156.265299] x29: ffff8000853cb6f0 x28: 0000000000000000 x27: 0000000000000000 [ 156.272420] x26: ffff0008033aba00 x25: ffff8000853cb7e8 x24: 0000000000000000 [ 156.279540] x23: ffff000808b8a040 x22: ffff8000853cbc08 x21: ffff000808b8a000 [ 156.286660] x20: ffff800081510970 x19: ffff8000814e13e8 x18: ffff0008033abb28 [ 156.293781] x17: 0000000000000050 x16: 000000000000005e x15: fefefefefefefefe [ 156.300900] x14: fefefefefefefefe x13: fefefefefefefefe x12: fefefefefefefefe [ 156.308021] x11: fefefefefefefefe x10: fefefefefefefefe x9 : ffff80007be3aa38 [ 156.315141] x8 : ffff0008033abbb0 x7 : fefefefefefefefe x6 : 0101010101010101 [ 156.322261] x5 : 00000000ffffff8d x4 : 0000000000000000 x3 : fefefefefefefefe [ 156.329381] x2 : ffff0008033aba00 x1 : ffff00080337da00 x0 : ffff0008033aba90 [ 156.336502] Call trace: [ 156.338937] crypto_ahash_import+0x18/0x68 (P) [ 156.343371] sa_sha_import+0x48/0x60 [sa2ul] [ 156.347630] crypto_ahash_import+0x54/0x68 [ 156.351716] test_ahash_vec_cfg+0x638/0x800 [ 156.355888] test_hash_vec+0xbc/0x230 [ 156.359538] __alg_test_hash+0x288/0x3d8 [ 156.363449] alg_test_hash+0x108/0x1a0 [ 156.367187] alg_test+0x148/0x658 [ 156.370491] cryptomgr_test+0x2c/0x50 [ 156.374141] kthread+0x134/0x218 [ 156.377359] ret_from_fork+0x10/0x20 [ 156.380924] Code: d503233f a9bf7bfd 910003fd f9401003 (385f0064) [ 156.386999] ---[ end trace 0000000000000000 ]--- [ 156.391626] Internal error: Oops: 0000000096000004 [#3] SMP [ 156.397270] Modules linked in: des_generic cbc libdes sa2ul authenc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 tps6594_esm pinctrl_tps6594 tps6594_pfsm gpio_regmap tps6594_regulator ti_am335x_adc pru_rproc kfifo_buf irq_pruss_intc cdns3 cdns_pltfrm cdns_usb_common snd_soc_j721e_evm display_connector phy_can_transceiver omap_mailbox phy_j721e_wiz ti_k3_r5_remoteproc tps6594_i2c tps6594_core at24 tidss k3_j72xx_bandgap drm_client_lib drm_dma_helper cdns_mhdp8546 ti_am335x_tscadc m_can_platform snd_soc_davinci_mcasp drm_display_helper pruss snd_soc_ti_udma m_can snd_soc_ti_edma drm_kms_helper ti_j721e_ufs snd_soc_ti_sdma ti_k3_dsp_remoteproc snd_soc_pcm3168a_i2c cdns3_ti can_dev snd_soc_pcm3168a rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [ 156.467403] CPU: 0 UID: 0 PID: 1250 Comm: cryptomgr_test Tainted: G D 6.14.0-rc7-next-20250324-build-configs-00001-g82a16a3a2a73-dirty #3 PREEMPT [ 156.481974] Tainted: [D]=DIE [ 156.484842] Hardware name: Texas Instruments J721e EVM (DT) [ 156.490396] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 156.497340] pc : crypto_ahash_import+0x18/0x68 [ 156.501773] lr : sa_sha_import+0x48/0x60 [sa2ul] [ 156.506379] sp : ffff8000851d36f0 [ 156.509680] x29: ffff8000851d36f0 x28: 0000000000000000 x27: 0000000000000000 [ 156.516800] x26: ffff000803c75400 x25: ffff8000851d37e8 x24: 0000000000000000 [ 156.523920] x23: ffff000808b8b040 x22: ffff8000851d3c08 x21: ffff000808b8b000 [ 156.531040] x20: ffff800081510e98 x19: ffff8000814e13e8 x18: 00000000ffffffff [ 156.538160] x17: 0000000000000000 x16: 0000000000000000 x15: ffff8000851d3740 [ 156.545280] x14: ffff8001051d3aa7 x13: 0000000000000000 x12: 0000000000000000 [ 156.552400] x11: 0000000000000000 x10: ffff00087f7c8308 x9 : ffff80007be3aa38 [ 156.559520] x8 : ffff000803c75548 x7 : fefefefefefefefe x6 : 0101010101010101 [ 156.566641] x5 : 00000000ffffff8d x4 : 0000000000000000 x3 : fefefefefefefefe [ 156.573761] x2 : ffff000803c75400 x1 : ffff000812686a00 x0 : ffff000803c75490 [ 156.580881] Call trace: [ 156.583316] crypto_ahash_import+0x18/0x68 (P) [ 156.587747] sa_sha_import+0x48/0x60 [sa2ul] [ 156.592006] crypto_ahash_import+0x54/0x68 [ 156.596090] test_ahash_vec_cfg+0x638/0x800 [ 156.600261] test_hash_vec+0xbc/0x230 [ 156.603912] __alg_test_hash+0x288/0x3d8 [ 156.607822] alg_test_hash+0x108/0x1a0 [ 156.611560] alg_test+0x148/0x658 [ 156.614864] cryptomgr_test+0x2c/0x50 [ 156.618514] kthread+0x134/0x218 [ 156.621731] ret_from_fork+0x10/0x20 [ 156.625298] Code: d503233f a9bf7bfd 910003fd f9401003 (385f0064) [ 156.631372] ---[ end trace 0000000000000000 ]---
Regards,
Manorit
Herbert Xu
maintainerPATCH v2 1/1 crypto: sa2ul - Use proper helpers to setup request
Quoted 3 lines · Manorit Chawdhry
Thanks for the quick fix, though now I see error in import rather than init which was there previously.
Oops, I removed one line too many from the import function. It should set the tfm just like init:
---8<---
Rather than setting up a request by hand, use the correct API helpers to setup the new request. This is because the API helpers will setup chaining.
Also change the fallback allocation to explicitly request for a sync algorithm as this driver will crash if given an async one.
drivers/crypto/sa2ul.c +16 -47
@@ -1415,22 +1415,13 @@ static int sa_sha_run(struct ahash_request *req) (auth_len >= SA_UNSAFE_DATA_SZ_MIN && auth_len <= SA_UNSAFE_DATA_SZ_MAX)) {struct ahash_request *subreq = &rctx->fallback_req; -int ret =0 ; +int ret;ahash_request_set_tfm (subreq, ctx->fallback.ahash); - subreq->base.flags = req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (subreq, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (subreq, req->src, req->result, auth_len); -crypto_ahash_init (subreq); - - subreq->nbytes = auth_len; - subreq->src = req->src; - subreq->result = req->result; - - ret |=crypto_ahash_update (subreq); - - subreq->nbytes =0 ; - - ret |=crypto_ahash_final (subreq); + ret =crypto_ahash_digest (subreq);return ret; }
@@ -1502,8 +1493,7 @@ static int sa_sha_cra_init_alg(struct crypto_tfm *tfm, const char *alg_base)return ret;if (alg_base) { - ctx->shash =crypto_alloc_shash (alg_base,0 , -CRYPTO_ALG_NEED_FALLBACK ); + ctx->shash =crypto_alloc_shash (alg_base,0 ,0 );if (IS_ERR (ctx->shash)) {dev_err (sa_k3_dev,"base driver %s couldn't be loaded\n" , alg_base);
@@ -1511,8 +1501,7 @@ static int sa_sha_cra_init_alg(struct crypto_tfm *tfm, const char *alg_base) }/* for fallback */ ctx->fallback.ahash = -crypto_alloc_ahash (alg_base,0 , -CRYPTO_ALG_NEED_FALLBACK ); +crypto_alloc_ahash (alg_base,0 ,CRYPTO_ALG_ASYNC );if (IS_ERR (ctx->fallback.ahash)) {dev_err (ctx->dev_data->dev,"Could not load fallback driver\n" );
@@ -1546,54 +1535,38 @@ static int sa_sha_init(struct ahash_request *req) crypto_ahash_digestsize(tfm), rctx);ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req,NULL ,NULL ,0 );return crypto_ahash_init (&rctx->fallback_req); }static int sa_sha_update (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; - rctx->fallback_req.nbytes = req->nbytes; - rctx->fallback_req.src = req->src; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req, req->src,NULL , req->nbytes);return crypto_ahash_update (&rctx->fallback_req); }static int sa_sha_final (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; - rctx->fallback_req.result = req->result; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req,NULL , req->result,0 );return crypto_ahash_final (&rctx->fallback_req); }static int sa_sha_finup (struct ahash_request *req) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm); -ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = - req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; - - rctx->fallback_req.nbytes = req->nbytes; - rctx->fallback_req.src = req->src; - rctx->fallback_req.result = req->result; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL ); +ahash_request_set_crypt (&rctx->fallback_req, req->src, req->result, req->nbytes);return crypto_ahash_finup (&rctx->fallback_req); }
@@ -1605,8 +1578,7 @@ static int sa_sha_import(struct ahash_request *req, const void *in)struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm);ahash_request_set_tfm (&rctx->fallback_req, ctx->fallback.ahash); - rctx->fallback_req.base.flags = req->base.flags & -CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (&rctx->fallback_req, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL );return crypto_ahash_import (&rctx->fallback_req, in); }
@@ -1614,12 +1586,9 @@ static int sa_sha_import(struct ahash_request *req, const void *in)static int sa_sha_export (struct ahash_request *req,void *out) {struct sa_sha_req_ctx *rctx =ahash_request_ctx (req); -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -struct sa_tfm_ctx *ctx =crypto_ahash_ctx (tfm);struct ahash_request *subreq = &rctx->fallback_req; -ahash_request_set_tfm (subreq, ctx->fallback.ahash); - subreq->base.flags = req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ; +ahash_request_set_callback (subreq, req->base.flags &CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL );return crypto_ahash_export (subreq, out); }
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Hi Herbert,
Quoted 13 lines · Herbert Xu
Quoted 3 lines · Manorit Chawdhry
Thanks for the quick fix, though now I see error in import rather than init which was there previously.
Oops, I removed one line too many from the import function. It should set the tfm just like init:
---8<---
Rather than setting up a request by hand, use the correct API helpers to setup the new request. This is because the API helpers will setup chaining.
Quoted 4 lines
Also change the fallback allocation to explicitly request for a sync algorithm as this driver will crash if given an async one.
Signed-off-by Herbert Xu <[email protected]>
Thanks for the fix! Although, it still fails probably due to the introduction of multibuffer hash testing in "crypto: testmgr - Add multibuffer hash testing" but that we will have to fix for our driver I assume.
[ 32.408283] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(9/13/uneven) src_divs=[100.0%@+860] key_offset=17" [...] [ 32.885927] alg: ahash: sha512-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: use_digest multibuffer(6/9/uneven) nosimd src_divs=[93.34%@+3634, 6.66%@+16] iv_offset=9 key_offset=70" [...] [ 33.135286] alg: ahash: sha256-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(15/16/uneven) src_divs=[100.0%@alignmask+26] key_offset=1"
Regards,
Manorit
Herbert Xu
maintainerRe: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Quoted 11 lines · Manorit Chawdhry
Thanks for the fix! Although, it still fails probably due to the introduction of multibuffer hash testing in "crypto: testmgr - Add multibuffer hash testing" but that we will have to fix for our driver I assume.
[ 32.408283] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(9/13/uneven) src_divs=[100.0%@+860] key_offset=17" [...] [ 32.885927] alg: ahash: sha512-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: use_digest multibuffer(6/9/uneven) nosimd src_divs=[93.34%@+3634, 6.66%@+16] iv_offset=9 key_offset=70" [...] [ 33.135286] alg: ahash: sha256-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(15/16/uneven) src_divs=[100.0%@alignmask+26] key_offset=1"
There are no other messages?
This means that one of the filler test requests triggered an EINVAL from your driver. A filler request in an uneven test can range from 0 to 2 * PAGE_SIZE bytes long.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu
maintainerRe: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Quoted 4 lines · Herbert Xu
This means that one of the filler test requests triggered an EINVAL from your driver. A filler request in an uneven test can range from 0 to 2 *
PAGE_SIZEbytes long.
Make that 0 to 16 * PAGE_SIZE bytes.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
In reply to Herbert XuHi Herbert,
Quoted 7 lines · Herbert Xu
Quoted 6 lines · Manorit Chawdhry
Thanks for the fix! Although, it still fails probably due to the introduction of multibuffer hash testing in "crypto: testmgr - Add multibuffer hash testing" but that we will have to fix for our driver I assume.
Quoted 7 lines
Quoted 5 lines
[ 32.408283] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(9/13/uneven) src_divs=[100.0%@+860] key_offset=17" [...] [ 32.885927] alg: ahash: sha512-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: use_digest multibuffer(6/9/uneven) nosimd src_divs=[93.34%@+3634, 6.66%@+16] iv_offset=9 key_offset=70" [...] [ 33.135286] alg: ahash: sha256-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(15/16/uneven) src_divs=[100.0%@alignmask+26] key_offset=1"There are no other messages?
This is the full failure log:
root@j721e-evm:~# modprobe sa2ul [59910.170612] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c root@j721e-evm:~# [59910.331792] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: may_sleep use_digest multibuffer(0/10/uneven) src_divs=[53.50%@+816, 14.50%@+2101, 32.0%@+1281] key_offset=114" [59910.354517] alg: ahash: sha512-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: may_sleep use_digest multibuffer(0/6/uneven) src_divs=[3.96%@+26, 88.54%@+3968, 7.50%@+20] dst_divs=[100.0%@alignmask+2] key_offset=33" [59910.454646] alg: ahash: sha256-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: use_digest multibuffer(4/14/uneven) nosimd src_divs=[50.0%@+29, 25.0%@+28, 25.0%@+4] key_offset=65" [59910.494415] alg: self-tests for sha1 using sha1-sa2ul failed (rc=-22) [59910.494424] ------------[ cut here ]------------ [59910.505522] alg: self-tests for sha1 using sha1-sa2ul failed (rc=-22) [59910.512463] alg: self-tests for sha512 using sha512-sa2ul failed (rc=-22) [59910.548673] ------------[ cut here ]------------ [59910.560115] alg: self-tests for sha512 using sha512-sa2ul failed (rc=-22) [59910.577041] WARNING: CPU: 0 PID: 1959 at crypto/testmgr.c:5997 alg_test+0x5d0/0x658 [59910.591470] Modules linked in: sa2ul authenc des_generic libdes cbc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 tps6594_pfsm tps6594_esm pinctrl_tps6594 tps6594_regulator gpio_regmap ti_am335x_adc kfifo_buf pru_rproc irq_pruss_intc cdns3 cdns_usb_common cdns_pltfrm snd_soc_j721e_evm display_connector phy_j721e_wiz phy_can_transceiver omap_mailbox ti_k3_r5_remoteproc at24 tps6594_i2c tps6594_core tidss drm_client_lib k3_j72xx_bandgap drm_dma_helper cdns_mhdp8546 m_can_platform drm_display_helper m_can ti_am335x_tscadc pruss drm_kms_helper snd_soc_pcm3168a_i2c snd_soc_davinci_mcasp can_dev snd_soc_pcm3168a snd_soc_ti_udma snd_soc_ti_edma ti_j721e_ufs ti_k3_dsp_remoteproc cdns3_ti snd_soc_ti_sdma rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [last unloaded: authenc] [59910.663794] CPU: 0 UID: 0 PID: 1959 Comm: cryptomgr_test Tainted: G W 6.14.0-rc1-build-configs-00186-g8b54e6a8f415-dirty #1 [59910.676286] Tainted: [W]=WARN [59910.679241] Hardware name: Texas Instruments J721e EVM (DT) [59910.684797] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [59910.691740] pc : alg_test+0x5d0/0x658 [59910.695392] lr : alg_test+0x5d0/0x658 [59910.699042] sp : ffff80008515bd40 [59910.702343] x29: ffff80008515bde0 x28: 0000000000000000 x27: 0000000000000000 [59910.709464] x26: 00000000ffffffea x25: 00000000ffffffff x24: 000000000000017b [59910.716585] x23: ffff80008384be88 x22: 000000000000118f x21: ffff0008032c5a80 [59910.723705] x20: ffff0008032c5a00 x19: ffff8000814bf320 x18: 0000000002004c00 [59910.730825] x17: 0000000002004400 x16: 00000000000000ee x15: cb299d3b567fbd0e [59910.737945] x14: 6cc9dff4249846de x13: 0000000000000000 x12: 0000000000020005 [59910.745066] x11: 000000e200000016 x10: 0000000000000af0 x9 : ffff8000800f8ba0 [59910.752186] x8 : ffff000809b10b50 x7 : 00000000005285f6 x6 : 000000000000001e [59910.759305] x5 : 0000000000000000 x4 : 0000000000000002 x3 : 0000000000000208 [59910.766425] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000809b10000 [59910.773545] Call trace: [59910.775981] alg_test+0x5d0/0x658 (P) [59910.779634] cryptomgr_test+0x2c/0x50 [59910.783286] kthread+0x134/0x218 [59910.786504] ret_from_fork+0x10/0x20 [59910.790070] ---[ end trace 0000000000000000 ]--- [59910.799050] alg: self-tests for sha256 using sha256-sa2ul failed (rc=-22) [59910.799057] ------------[ cut here ]------------ [59910.810440] alg: self-tests for sha256 using sha256-sa2ul failed (rc=-22) [59910.810468] WARNING: CPU: 0 PID: 1962 at crypto/testmgr.c:5997 alg_test+0x5d0/0x658 [59910.824882] Modules linked in: sa2ul authenc des_generic libdes cbc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 tps6594_pfsm tps6594_esm pinctrl_tps6594 tps6594_regulator gpio_regmap ti_am335x_adc kfifo_buf pru_rproc irq_pruss_intc cdns3 cdns_usb_common cdns_pltfrm snd_soc_j721e_evm display_connector phy_j721e_wiz phy_can_transceiver omap_mailbox ti_k3_r5_remoteproc at24 tps6594_i2c tps6594_core tidss drm_client_lib k3_j72xx_bandgap drm_dma_helper cdns_mhdp8546 m_can_platform drm_display_helper m_can ti_am335x_tscadc pruss drm_kms_helper snd_soc_pcm3168a_i2c snd_soc_davinci_mcasp can_dev snd_soc_pcm3168a snd_soc_ti_udma snd_soc_ti_edma ti_j721e_ufs ti_k3_dsp_remoteproc cdns3_ti snd_soc_ti_sdma rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [last unloaded: authenc] [59910.897196] CPU: 0 UID: 0 PID: 1962 Comm: cryptomgr_test Tainted: G W 6.14.0-rc1-build-configs-00186-g8b54e6a8f415-dirty #1 [59910.909688] Tainted: [W]=WARN [59910.912642] Hardware name: Texas Instruments J721e EVM (DT) [59910.918198] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [59910.925141] pc : alg_test+0x5d0/0x658 [59910.928792] lr : alg_test+0x5d0/0x658 [59910.932443] sp : ffff800085313d40 [59910.935744] x29: ffff800085313de0 x28: 0000000000000000 x27: 0000000000000000 [59910.942865] x26: 00000000ffffffea x25: 00000000ffffffff x24: 000000000000018d [59910.949986] x23: ffff80008384be88 x22: 000000000000118f x21: ffff000808b03e80 [59910.957106] x20: ffff000808b03e00 x19: ffff8000814bf320 x18: 00000000fffffffe [59910.964226] x17: ffff8007fd27e000 x16: ffff800080000000 x15: ffff8000852bb8e0 [59910.971346] x14: 0000000000000000 x13: ffff800083814452 x12: 0000000000000000 [59910.978465] x11: ffff00087f7a4d80 x10: 0000000000000af0 x9 : ffff8000800f8ba0 [59910.985586] x8 : ffff0008052e8b50 x7 : 0000000000019aff x6 : 000000000000000d [59910.992705] x5 : 0000000000000000 x4 : 0000000000000002 x3 : 0000000000000208 [59910.999825] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0008052e8000 [59911.006946] Call trace: [59911.009382] alg_test+0x5d0/0x658 (P) [59911.013034] cryptomgr_test+0x2c/0x50 [59911.016685] kthread+0x134/0x218 [59911.019905] ret_from_fork+0x10/0x20 [59911.023469] ---[ end trace 0000000000000000 ]--- [59911.028107] WARNING: CPU: 0 PID: 1961 at crypto/testmgr.c:5997 alg_test+0x5d0/0x658 [59911.035749] Modules linked in: sa2ul authenc des_generic libdes cbc onboard_usb_dev rpmsg_ctrl rpmsg_char phy_cadence_torrent phy_cadence_sierra rtc_tps6594 tps6594_pfsm tps6594_esm pinctrl_tps6594 tps6594_regulator gpio_regmap ti_am335x_adc kfifo_buf pru_rproc irq_pruss_intc cdns3 cdns_usb_common cdns_pltfrm snd_soc_j721e_evm display_connector phy_j721e_wiz phy_can_transceiver omap_mailbox ti_k3_r5_remoteproc at24 tps6594_i2c tps6594_core tidss drm_client_lib k3_j72xx_bandgap drm_dma_helper cdns_mhdp8546 m_can_platform drm_display_helper m_can ti_am335x_tscadc pruss drm_kms_helper snd_soc_pcm3168a_i2c snd_soc_davinci_mcasp can_dev snd_soc_pcm3168a snd_soc_ti_udma snd_soc_ti_edma ti_j721e_ufs ti_k3_dsp_remoteproc cdns3_ti snd_soc_ti_sdma rti_wdt overlay cfg80211 rfkill fuse drm backlight ipv6 [last unloaded: authenc] [59911.108050] CPU: 0 UID: 0 PID: 1961 Comm: cryptomgr_test Tainted: G W 6.14.0-rc1-build-configs-00186-g8b54e6a8f415-dirty #1 [59911.120541] Tainted: [W]=WARN [59911.123495] Hardware name: Texas Instruments J721e EVM (DT) [59911.129049] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [59911.135992] pc : alg_test+0x5d0/0x658 [59911.139643] lr : alg_test+0x5d0/0x658 [59911.143293] sp : ffff8000852bbd40 [59911.146594] x29: ffff8000852bbde0 x28: 0000000000000000 x27: 0000000000000000 [59911.153714] x26: 00000000ffffffea x25: 00000000ffffffff x24: 000000000000017f [59911.160835] x23: ffff80008384be88 x22: 000000000000118f x21: ffff000803f5aa80 [59911.167954] x20: ffff000803f5aa00 x19: ffff8000814bf320 x18: 00000000fffffffe [59911.175074] x17: ffff8007fd27e000 x16: ffff800080000000 x15: 0000000000000000 [59911.182194] x14: 00003d0971c5fa00 x13: ffffffff919fcffd x12: 0000000000000000 [59911.189313] x11: ffff00087f7a4d80 x10: 0000000000000af0 x9 : ffff8000800f8ba0 [59911.196433] x8 : ffff000809a37450 x7 : 0000000000026a7c x6 : 000000000000000f [59911.203553] x5 : 0000000000000000 x4 : 0000000000000002 x3 : 0000000000000208 [59911.210672] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000809a36900 [59911.217793] Call trace: [59911.220227] alg_test+0x5d0/0x658 (P) [59911.223879] cryptomgr_test+0x2c/0x50 [59911.227529] kthread+0x134/0x218 [59911.230746] ret_from_fork+0x10/0x20 [59911.234310] ---[ end trace 0000000000000000 ]---
Quoted 5 lines
This means that one of the filler test requests triggered an EINVAL from your driver. A filler request in an uneven test can range from 0 to 2 *
PAGE_SIZEbytes long.
I tracked it down and see [0] returning -EINVAL. Do you have any insights as to what changed that it's not working anymore...
Regards,
Manorit
Quoted 5 lines
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Hi Herbert,
Quoted 15 lines · Manorit Chawdhry
Hi Herbert,
Quoted 12 lines · Herbert Xu
Quoted 11 lines · Manorit Chawdhry
Thanks for the fix! Although, it still fails probably due to the introduction of multibuffer hash testing in "crypto: testmgr - Add multibuffer hash testing" but that we will have to fix for our driver I assume.
[ 32.408283] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(9/13/uneven) src_divs=[100.0%@+860] key_offset=17" [...] [ 32.885927] alg: ahash: sha512-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: use_digest multibuffer(6/9/uneven) nosimd src_divs=[93.34%@+3634, 6.66%@+16] iv_offset=9 key_offset=70" [...] [ 33.135286] alg: ahash: sha256-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(15/16/uneven) src_divs=[100.0%@alignmask+26] key_offset=1"
[..]
Quoted 10 lines
Quoted 5 lines
This means that one of the filler test requests triggered an EINVAL from your driver. A filler request in an uneven test can range from 0 to 2 *
PAGE_SIZEbytes long.I tracked it down and see [0] returning
-EINVAL. Do you have any insights as to what changed that it's not working anymore...
Added some more prints.
drivers/crypto/sa2ul.c +6 -0
@@ -1176,7 +1176,13 @@ static int sa_run(struct sa_req *req) mapped_sg->sgt.orig_nents = sg_nents; ret =dma_map_sgtable (ddev, &mapped_sg->sgt, dir_src,0 );if (ret) { +for (struct scatterlist *temp = src; temp; temp =sg_next (temp)) { +pr_info ("%s: %d: temp->length: %d, temp->offset: %d\n" , __func__, __LINE__, temp->length, temp->offset); + } +pr_info ("%s: %d: req->size: %d, src: %p\n" , __func__, __LINE__, req->size, req->src); +pr_info ("%s: %d: sgl: %p, orig_nents: %d\n" , __func__, __LINE__, mapped_sg->sgt.sgl, mapped_sg->sgt.orig_nents);kfree (rxd); +pr_info ("%s: %d: ret=%d\n" , __func__, __LINE__, ret);return ret; }
root@j721e-evm:~# modprobe sa2ul [ 32.890801] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c root@j721e-evm:~# [ 32.981093] sa_run: 1180: temp->length: 8192, temp->offset: 0 [ 32.996268] sa_run: 1180: temp->length: 8192, temp->offset: 0 [ 33.002029] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.007512] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.012986] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.018458] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.023930] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.029402] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.034874] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.040345] sa_run: 1182: req->size: 40187, src: 00000000f1859ae0 [ 33.046426] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22 [ 33.052419] sa_run: 1185: ret=-22 [ 33.055852] sa_run: 1180: temp->length: 8192, temp->offset: 0 [ 33.061589] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.067061] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.072532] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.078004] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.083475] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.088947] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.094419] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.099890] sa_run: 1182: req->size: 32768, src: 00000000f1859ae0 [ 33.105969] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22 [ 33.111962] sa_run: 1185: ret=-22 [ 33.115268] sa_run: 1180: temp->length: 8192, temp->offset: 0 [ 33.121001] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.126472] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.131944] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.137416] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.142888] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.148360] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.153832] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.159304] sa_run: 1182: req->size: 34117, src: 00000000f1859ae0 [ 33.165382] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22 [ 33.171375] sa_run: 1185: ret=-22 [ 33.174725] sa_run: 1180: temp->length: 8192, temp->offset: 0 [ 33.180459] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.185936] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.191408] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.196879] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.202351] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.207822] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.213294] sa_run: 1180: temp->length: 0, temp->offset: 0 [ 33.218765] sa_run: 1182: req->size: 31875, src: 00000000f1859ae0 [ 33.224845] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22 [ 33.230838] sa_run: 1185: ret=-22 [ 33.234204] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: may_sleep use_digest multibuffer(8/14/uneven) src_divs=[100.0%@alignmask+3110] dst_divs=[100.0%@+2132] key_offset=63" [
Regards,
Manorit
Quoted 9 lines
Regards,
ManoritQuoted 5 lines
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu
maintainerPATCH 1/1 crypto: testmgr - Initialise full_sgl properly
Quoted 3 lines · Manorit Chawdhry
[ 33.040345] sa_run: 1182: req->size: 40187, src: 00000000f1859ae0 [ 33.046426] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22
Thanks for the info! The filler SG initialisation was broken:
---8<---
Initialise the whole full_sgl array rather than the first entry.
crypto/testmgr.c +1 -1
@@ -689,7 +689,7 @@ static int build_test_sglist(struct test_sglist *tsgl,sg_init_table (tsgl->full_sgl,XBUFSIZE );for (i =0 ; i <XBUFSIZE ; i++) -sg_set_buf (tsgl->full_sgl, tsgl->bufs[i],PAGE_SIZE *2 ); +sg_set_buf (&tsgl->full_sgl[i], tsgl->bufs[i],PAGE_SIZE *2 );return 0 ; }
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [PATCH] crypto: testmgr - Initialise full_sgl properly
Hi Herbert,
Quoted 5 lines · Herbert Xu
Quoted 3 lines · Manorit Chawdhry
[ 33.040345] sa_run: 1182: req->size: 40187, src: 00000000f1859ae0 [ 33.046426] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22
Quoted 9 lines
Thanks for the info! The filler SG initialisation was broken:
---8<---
Initialise the wholefull_sglarray rather than the first entry.Fixes 8b54e6a8f415 ("crypto: testmgr - Add multibuffer hash testing")Reported-by Manorit Chawdhry <[email protected]>Signed-off-by Herbert Xu <[email protected]>
Thanks, this fixes it.
root@j721e-evm:~# modprobe sa2ul [ 35.293140] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c
Regards,
Manorit
Re: [PATCH] crypto: testmgr - Initialise full_sgl properly
Hi Herbert,
Quoted 13 lines · Manorit Chawdhry
Hi Herbert,
Quoted 10 lines · Herbert Xu
Quoted 3 lines · Manorit Chawdhry
[ 33.040345] sa_run: 1182: req->size: 40187, src: 00000000f1859ae0 [ 33.046426] sa_run: 1183: sgl: 00000000f1859ae0, orig_nents: -22Thanks for the info! The filler SG initialisation was broken:
---8<---
Initialise the wholefull_sglarray rather than the first entry.
Quoted 6 lines
Quoted 4 lines
Fixes 8b54e6a8f415 ("crypto: testmgr - Add multibuffer hash testing")Reported-by Manorit Chawdhry <[email protected]>Signed-off-by Herbert Xu <[email protected]>Thanks, this fixes it.
Though it really makes me wonder.. I was actually thinking that it was our driver problem and not something core as when I fell back to the software fallbacks everything was fine... How is it possible, do you have any insights on that? Is something missing?
[ Without this patch + the patch mentioned below ]
drivers/crypto/sa2ul.c +2 -2
@@ -1417,9 +1417,9 @@ static int sa_sha_run(struct ahash_request *req)if (!auth_len)return zero_message_process (req); -if (auth_len >SA_MAX_DATA_SZ || +if (1 || (auth_len >SA_MAX_DATA_SZ || (auth_len >=SA_UNSAFE_DATA_SZ_MIN && - auth_len <=SA_UNSAFE_DATA_SZ_MAX )) { + auth_len <=SA_UNSAFE_DATA_SZ_MAX ))) {struct ahash_request *subreq = &rctx->fallback_req;int ret;
This passes the tests.
root@j721e-evm:~# modprobe sa2ul [ 53.380168] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c
Regards,
Manorit
Quoted 8 lines
root@j721e-evm:~# modprobe sa2ul [ 35.293140] omap_rng 4e10000.rng: Random Number Generator ver. 241b34cTested-by Manorit Chawdhry <[email protected]>Regards,
Manorit
Herbert Xu
maintainerRe: [PATCH] crypto: testmgr - Initialise full_sgl properly
Quoted 5 lines · Manorit Chawdhry
Though it really makes me wonder.. I was actually thinking that it was our driver problem and not something core as when I fell back to the software fallbacks everything was fine... How is it possible, do you have any insights on that? Is something missing?
I think the software SG walker simply exits if it detects a shorter than expected SG list. So if you ask it to hash 128KB of data, but only supply an 8KB SG list, it will hash 8KB and then declare that the job is done.
That is arguably suboptimal.
I'm in the process of rewriting the software walker to add multibuffer support so I might fix this in the process.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
In reply to Manorit ChawdhryHi Herbert,
Quoted 19 lines · Manorit Chawdhry
Hi Herbert,
Quoted 16 lines · Herbert Xu
Quoted 3 lines · Manorit Chawdhry
Thanks for the quick fix, though now I see error in import rather than init which was there previously.
Oops, I removed one line too many from the import function. It should set the tfm just like init:
---8<---
Rather than setting up a request by hand, use the correct API helpers to setup the new request. This is because the API helpers will setup chaining.Also change the fallback allocation to explicitly request for a sync algorithm as this driver will crash if given an async one.
Quoted 6 lines
Quoted 1 line
Signed-off-by Herbert Xu <[email protected]>Thanks for the fix! Although, it still fails probably due to the introduction of multibuffer hash testing in "crypto: testmgr - Add multibuffer hash testing" but that we will have to fix for our driver I assume.
I see multibuffer hashing is reverted but with chaining changes we would require the following patch.. I see the chaining changes in 6.15-rc1 but I don't see the following patch in 6.15-rc1, could you queue it for next RC?
Regards,
Manorit
Quoted 12 lines
[ 32.408283] alg: ahash: sha1-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(9/13/uneven) src_divs=[100.0%@+860] key_offset=17" [...] [ 32.885927] alg: ahash: sha512-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: use_digest multibuffer(6/9/uneven) nosimd src_divs=[93.34%@+3634, 6.66%@+16] iv_offset=9 key_offset=70" [...] [ 33.135286] alg: ahash: sha256-sa2ul digest() failed on test vector 0; expected_error=0, actual_error=-22, cfg="random: inplace_two_sglists may_sleep use_digest multibuffer(15/16/uneven) src_divs=[100.0%@alignmask+26] key_offset=1"Tested-by Manorit Chawdhry <[email protected]>Regards,
Manorit
Herbert Xu
maintainerRe: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Quoted 5 lines · Manorit Chawdhry
I see multibuffer hashing is reverted but with chaining changes we would require the following patch.. I see the chaining changes in 6.15-rc1 but I don't see the following patch in 6.15-rc1, could you queue it for next RC?
This patch is in cryptodev. There won't be any chaining in 6.15 so it's not needed there.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Hi Herbert,
Quoted 9 lines · Herbert Xu
Quoted 5 lines · Manorit Chawdhry
I see multibuffer hashing is reverted but with chaining changes we would require the following patch.. I see the chaining changes in 6.15-rc1 but I don't see the following patch in 6.15-rc1, could you queue it for next RC?
This patch is in cryptodev. There won't be any chaining in 6.15 so it's not needed there.
I see the chaining patches in 6.15-rc1.. [0] Are you planning to revert them as well?
[0]: https://github.com/torvalds/linux/commits/v6.15-rc1/crypto/ahash.c
Regards,
Manorit
Quoted 6 lines
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu
maintainerRe: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Quoted 5 lines · Manorit Chawdhry
I see the chaining patches in 6.15-rc1.. [0] Are you planning to revert them as well?
[0]: https://github.com/torvalds/linux/commits/v6.15-rc1/crypto/ahash.c
With the multibuffer tests removed there is no way to call into the chaining code in 6.15.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [v2 PATCH] crypto: sa2ul - Use proper helpers to setup request
Hi Herbert,
Quoted 10 lines · Herbert Xu
Quoted 5 lines · Manorit Chawdhry
I see the chaining patches in 6.15-rc1.. [0] Are you planning to revert them as well?
[0]: https://github.com/torvalds/linux/commits/v6.15-rc1/crypto/ahash.c
With the multibuffer tests removed there is no way to call into the chaining code in 6.15.
root@j721e-evm:~# uname -a Linux j721e-evm 6.15.0-rc1 #3 SMP PREEMPT Fri Apr 11 11:37:56 IST 2025 aarch64 GNU/Linux root@j721e-evm:~# modprobe sa2ul [ 42.395465] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c root@j721e-evm:~# [ 42.515589] Unable to handle kernel paging request at virtual address fefefefefefeff46 [ 42.549426] Unable to handle kernel paging request at virtual address fefefefefefeff46 [ 42.558286] Unable to handle kernel paging request at virtual address fefefefefefeff46 [ 42.592270] Mem abort info: [ 42.623088] ESR = 0x0000000096000044 [ 42.628660] Mem abort info: [ 42.631496] ESR = 0x0000000096000044 [ 42.635478] Mem abort info: [ 42.655279] ESR = 0x0000000096000044 [ 42.661660] EC = 0x25: DABT (current EL), IL = 32 bits [ 42.681101] EC = 0x25: DABT (current EL), IL = 32 bits [ 42.687850] EC = 0x25: DABT (current EL), IL = 32 bits [ 42.701449] SET = 0, FnV = 0 [ 42.704524] EA = 0, S1PTW = 0 [ 42.727712] SET = 0, FnV = 0 [ 42.731640] SET = 0, FnV = 0 [ 42.755404] FSC = 0x04: level 0 translation fault [ 42.760711] EA = 0, S1PTW = 0 [ 42.763923] EA = 0, S1PTW = 0 [ 42.793049] FSC = 0x04: level 0 translation fault [ 42.798305] FSC = 0x04: level 0 translation fault [ 42.809586] Data abort info: [ 42.812463] ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000 [ 42.830629] Data abort info: [ 42.833982] Data abort info: [ 42.854520] ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000 [ 42.860776] ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000 [ 42.880775] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 [ 42.897045] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 [ 42.902752] CM = 0, WnR = 1, TnD = 0, TagAccess = 0 [ 42.913445] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 42.931845] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 42.937545] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 42.948838] [fefefefefefeff46] address between user and kernel address ranges [ 42.968662] [fefefefefefeff46] address between user and kernel address ranges [ 42.976773] [fefefefefefeff46] address between user and kernel address ranges [..]
Maybe it's not the chaining but the way chaining was implemented that requires us to start using these correct API helpers otherwise we crash so I think we would require the following patch.
Regards,
Manorit
Quoted 5 lines
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert Xu
maintainerPATCH 1/1 crypto: ahash - Disable request chaining
Quoted 4 lines · Manorit Chawdhry
Maybe it's not the chaining but the way chaining was implemented that requires us to start using these correct API helpers otherwise we crash so I think we would require the following patch.
You're right. This needs to be disabled more thoroughly for 6.15. Please try this patch:
---8<---
Disable hash request chaining in case a driver that copies an ahash_request object by hand accidentally triggers chaining.
crypto/ahash.c +1 -75
@@ -315,16 +315,7 @@ EXPORT_SYMBOL_GPL(crypto_ahash_setkey);static bool ahash_request_hasvirt (struct ahash_request *req) { -struct ahash_request *r2; - -if (ahash_request_isvirt (req)) -return true; - -list_for_each_entry (r2, &req->base.list, base.list) -if (ahash_request_isvirt (r2)) -return true; - -return false; +return ahash_request_isvirt (req); } static int ahash_reqchain_virt(struct ahash_save_req_state *state,
@@ -472,7 +463,6 @@ static int ahash_do_req_chain(struct ahash_request *req,bool update = op ==crypto_ahash_alg (tfm)->update;struct ahash_save_req_state *state;struct ahash_save_req_state state0; -struct ahash_request *r2;u8 *page =NULL ;int err;
@@ -509,7 +499,6 @@ static int ahash_do_req_chain(struct ahash_request *req, state->offset =0 ; state->nbytes =0 ;INIT_LIST_HEAD (&state->head); -list_splice_init (&req->base.list, &state->head);if (page)sg_init_one (&state->sg, page,PAGE_SIZE );
@@ -540,9 +529,6 @@ static int ahash_do_req_chain(struct ahash_request *req, out_set_chain: req->base.err = err; -list_for_each_entry (r2, &req->base.list, base.list) - r2->base.err = err; -return err; }
@@ -551,19 +537,10 @@ int crypto_ahash_init(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);if (likely (tfm->using_shash)) { -struct ahash_request *r2;int err; err =crypto_shash_init (prepare_shash_desc (req, tfm)); req->base.err = err; - -list_for_each_entry (r2, &req->base.list, base.list) { -struct shash_desc *desc; - - desc =prepare_shash_desc (r2, tfm); - r2->base.err =crypto_shash_init (desc); - } -return err; }
@@ -620,19 +597,10 @@ int crypto_ahash_update(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);if (likely (tfm->using_shash)) { -struct ahash_request *r2;int err; err =shash_ahash_update (req,ahash_request_ctx (req)); req->base.err = err; - -list_for_each_entry (r2, &req->base.list, base.list) { -struct shash_desc *desc; - - desc =ahash_request_ctx (r2); - r2->base.err =shash_ahash_update (r2, desc); - } -return err; }
@@ -645,19 +613,10 @@ int crypto_ahash_final(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);if (likely (tfm->using_shash)) { -struct ahash_request *r2;int err; err =crypto_shash_final (ahash_request_ctx (req), req->result); req->base.err = err; - -list_for_each_entry (r2, &req->base.list, base.list) { -struct shash_desc *desc; - - desc =ahash_request_ctx (r2); - r2->base.err =crypto_shash_final (desc, r2->result); - } -return err; }
@@ -670,19 +629,10 @@ int crypto_ahash_finup(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);if (likely (tfm->using_shash)) { -struct ahash_request *r2;int err; err =shash_ahash_finup (req,ahash_request_ctx (req)); req->base.err = err; - -list_for_each_entry (r2, &req->base.list, base.list) { -struct shash_desc *desc; - - desc =ahash_request_ctx (r2); - r2->base.err =shash_ahash_finup (r2, desc); - } -return err; }
@@ -757,19 +707,10 @@ int crypto_ahash_digest(struct ahash_request *req)struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);if (likely (tfm->using_shash)) { -struct ahash_request *r2;int err; err =shash_ahash_digest (req,prepare_shash_desc (req, tfm)); req->base.err = err; - -list_for_each_entry (r2, &req->base.list, base.list) { -struct shash_desc *desc; - - desc =prepare_shash_desc (r2, tfm); - r2->base.err =shash_ahash_digest (r2, desc); - } -return err; }
@@ -1133,20 +1074,5 @@ int ahash_register_instance(struct crypto_template *tmpl, }EXPORT_SYMBOL_GPL (ahash_register_instance); -void ahash_request_free (struct ahash_request *req) -{ -struct ahash_request *tmp; -struct ahash_request *r2; - -if (unlikely (!req)) -return ; - -list_for_each_entry_safe (r2, tmp, &req->base.list, base.list) -kfree_sensitive (r2); - -kfree_sensitive (req); -} -EXPORT_SYMBOL_GPL (ahash_request_free); -MODULE_LICENSE ("GPL" );MODULE_DESCRIPTION ("Asynchronous cryptographic hash type" );
include/crypto/hash.h +5 -1
@@ -10,6 +10,7 @@#include <linux/atomic.h> #include <linux/crypto.h> +#include <linux/slab.h> #include <linux/string.h> /* Set this bit for virtual address instead of SG list. */
@@ -581,7 +582,10 @@ static inline struct ahash_request *ahash_request_alloc_noprof(* ahash_request_free() - zeroize and free the request data structure * @req: request data structure cipher handle to be freed */ -void ahash_request_free (struct ahash_request *req); +static inline void ahash_request_free (struct ahash_request *req) +{ +kfree_sensitive (req); +}static inline struct ahash_request *ahash_request_cast (struct crypto_async_request *req)
include/crypto/internal/hash.h +1 -1
@@ -249,7 +249,7 @@ static inline struct crypto_shash *__crypto_shash_cast(struct crypto_tfm *tfm)static inline bool ahash_request_chained (struct ahash_request *req) { -return crypto_request_chained (&req->base); +return false; }static inline bool ahash_request_isvirt (struct ahash_request *req)
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [PATCH] crypto: ahash - Disable request chaining
Hi Herbert,
Quoted 9 lines · Herbert Xu
Quoted 4 lines · Manorit Chawdhry
Maybe it's not the chaining but the way chaining was implemented that requires us to start using these correct API helpers otherwise we crash so I think we would require the following patch.
You're right. This needs to be disabled more thoroughly for 6.15. Please try this patch:
Quoted 8 lines
---8<---
Disable hash request chaining in case a driver that copies anahash_requestobject by hand accidentally triggers chaining.Reported-by Manorit Chawdhry <[email protected]>Fixes f2ffe5a9183d ("crypto: hash - Add request chaining API")Signed-off-by Herbert Xu <[email protected]>
This fixes it, thanks!
root@j721e-evm:~# uname -a Linux j721e-evm 6.15.0-rc1-00001-gdcd7f62f8e5e-dirty #4 SMP PREEMPT Fri Apr 11 13:18:03 IST 2025 aarch64 GNU/Linux root@j721e-evm:~# modprobe sa2ul [ 414.110972] omap_rng 4e10000.rng: Random Number Generator ver. 241b34c
Regards,
Manorit
Herbert Xu
maintainerPATCH v2 4/11 crypto: tcrypt - Restore multibuffer ahash tests
In reply to Herbert XuThis patch is a revert of commit 388ac25efc8ce3bf9768ce7bf24268d6fac285d5.
As multibuffer ahash is coming back in the form of request chaining, restore the multibuffer ahash tests using the new interface.
--- crypto/tcrypt.c | 231 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+)
crypto/tcrypt.c +231 -0
@@ -716,6 +716,207 @@ static inline int do_one_ahash_op(struct ahash_request *req, int ret)return crypto_wait_req (ret, wait); } +struct test_mb_ahash_data { +struct scatterlist sg[XBUFSIZE ]; +char result[64 ]; +struct ahash_request *req; +struct crypto_wait wait; +char *xbuf[XBUFSIZE ]; +}; + +static inline int do_mult_ahash_op (struct test_mb_ahash_data *data,u32 num_mb, +int *rc) +{ +int i, err; + +/* Fire up a bunch of concurrent requests */ + err =crypto_ahash_digest (data[0 ].req); + +/* Wait for all requests to finish */ + err =crypto_wait_req (err, &data[0 ].wait); +if (num_mb <2 ) +return err; + +for (i =0 ; i < num_mb; i++) { + rc[i] =ahash_request_err (data[i].req); +if (rc[i]) { +pr_info ("concurrent request %d error %d\n" , i, rc[i]); + err = rc[i]; + } + } + +return err; +} + +static int test_mb_ahash_jiffies (struct test_mb_ahash_data *data,int blen, +int secs,u32 num_mb) +{ +unsigned long start, end; +int bcount; +int ret =0 ; +int *rc; + + rc =kcalloc (num_mb,sizeof (*rc),GFP_KERNEL ); +if (!rc) +return -ENOMEM ; + +for (start = jiffies, end = start + secs *HZ , bcount =0 ; +time_before (jiffies, end); bcount++) { + ret =do_mult_ahash_op (data, num_mb, rc); +if (ret) + goto out; + } + +pr_cont ("%d operations in %d seconds (%llu bytes)\n" , + bcount * num_mb, secs, (u64 )bcount * blen * num_mb); + +out: +kfree (rc); +return ret; +} + +static int test_mb_ahash_cycles (struct test_mb_ahash_data *data,int blen, +u32 num_mb) +{ +unsigned long cycles =0 ; +int ret =0 ; +int i; +int *rc; + + rc =kcalloc (num_mb,sizeof (*rc),GFP_KERNEL ); +if (!rc) +return -ENOMEM ; + +/* Warm-up run. */ +for (i =0 ; i <4 ; i++) { + ret =do_mult_ahash_op (data, num_mb, rc); +if (ret) + goto out; + } + +/* The real thing. */ +for (i =0 ; i <8 ; i++) { +cycles_t start, end; + + start =get_cycles (); + ret =do_mult_ahash_op (data, num_mb, rc); + end =get_cycles (); + +if (ret) + goto out; + + cycles += end - start; + } + +pr_cont ("1 operation in %lu cycles (%d bytes)\n" , + (cycles +4 ) / (8 * num_mb), blen); + +out: +kfree (rc); +return ret; +} + +static void test_mb_ahash_speed (const char *algo,unsigned int secs, +struct hash_speed *speed,u32 num_mb) +{ +struct test_mb_ahash_data *data; +struct crypto_ahash *tfm; +unsigned int i, j, k; +int ret; + + data =kcalloc (num_mb,sizeof (*data),GFP_KERNEL ); +if (!data) +return ; + + tfm =crypto_alloc_ahash (algo,0 ,0 ); +if (IS_ERR (tfm)) { +pr_err ("failed to load transform for %s: %ld\n" , + algo,PTR_ERR (tfm)); + goto free_data; + } + +for (i =0 ; i < num_mb; ++i) { +if (testmgr_alloc_buf (data[i].xbuf)) + goto out; + +crypto_init_wait (&data[i].wait); + + data[i].req =ahash_request_alloc (tfm,GFP_KERNEL ); +if (!data[i].req) { +pr_err ("alg: hash: Failed to allocate request for %s\n" , + algo); + goto out; + } + + +if (i) { +ahash_request_set_callback (data[i].req,0 ,NULL ,NULL ); +ahash_request_chain (data[i].req, data[0 ].req); + }else +ahash_request_set_callback (data[0 ].req,0 , + crypto_req_done, + &data[0 ].wait); + +sg_init_table (data[i].sg,XBUFSIZE ); +for (j =0 ; j <XBUFSIZE ; j++) { +sg_set_buf (data[i].sg + j, data[i].xbuf[j],PAGE_SIZE ); +memset (data[i].xbuf[j],0xff ,PAGE_SIZE ); + } + } + +pr_info ("\ntesting speed of multibuffer %s (%s)\n" , algo, +get_driver_name (crypto_ahash, tfm)); + +for (i =0 ; speed[i].blen !=0 ; i++) { +/* For some reason this only tests digests. */ +if (speed[i].blen != speed[i].plen) +continue ; + +if (speed[i].blen >XBUFSIZE *PAGE_SIZE ) { +pr_err ("template (%u) too big for tvmem (%lu)\n" , + speed[i].blen,XBUFSIZE *PAGE_SIZE ); + goto out; + } + +if (klen) +crypto_ahash_setkey (tfm, tvmem[0 ], klen); + +for (k =0 ; k < num_mb; k++) +ahash_request_set_crypt (data[k].req, data[k].sg, + data[k].result, speed[i].blen); + +pr_info ("test%3u " +"(%5u byte blocks,%5u bytes per update,%4u updates): " , + i, speed[i].blen, speed[i].plen, + speed[i].blen / speed[i].plen); + +if (secs) { + ret =test_mb_ahash_jiffies (data, speed[i].blen, secs, + num_mb); +cond_resched (); + }else { + ret =test_mb_ahash_cycles (data, speed[i].blen, num_mb); + } + + +if (ret) { +pr_err ("At least one hashing failed ret=%d\n" , ret); +break ; + } + } + +out: +ahash_request_free (data[0 ].req); + +for (k =0 ; k < num_mb; ++k) +testmgr_free_buf (data[k].xbuf); + +crypto_free_ahash (tfm); + +free_data: +kfree (data); +} +static int test_ahash_jiffies_digest (struct ahash_request *req,int blen,char *out,int secs) {
@@ -2391,6 +2592,36 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)test_ahash_speed ("sm3" , sec, generic_hash_speed_template);if (mode >400 && mode <500 )break ; fallthrough; +case 450 : +test_mb_ahash_speed ("sha1" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 451 : +test_mb_ahash_speed ("sha256" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 452 : +test_mb_ahash_speed ("sha512" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 453 : +test_mb_ahash_speed ("sm3" , sec, generic_hash_speed_template, + num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 454 : +test_mb_ahash_speed ("streebog256" , sec, + generic_hash_speed_template, num_mb); +if (mode >400 && mode <500 )break ; + fallthrough; +case 455 : +test_mb_ahash_speed ("streebog512" , sec, + generic_hash_speed_template, num_mb); +if (mode >400 && mode <500 )break ; + fallthrough;case 499 :break ;
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 5/11 crypto: ahash - Add virtual address support
In reply to Herbert XuThis patch adds virtual address support to ahash. Virtual addresses were previously only supported through shash. The user may choose to use virtual addresses with ahash by calling ahash_request_set_virt instead of ahash_request_set_crypt.
The API will take care of translating this to an SG list if necessary, unless the algorithm declares that it supports chaining. Therefore in order for an ahash algorithm to support chaining, it must also support virtual addresses directly.
--- crypto/ahash.c | 280 +++++++++++++++++++++++++++++---- include/crypto/hash.h | 38 ++++- include/crypto/internal/hash.h | 7 +- include/linux/crypto.h | 2 +- 4 files changed, 293 insertions(+), 34 deletions(-)
crypto/ahash.c +250 -30
@@ -34,11 +34,17 @@ struct ahash_save_req_state {int (*op)(struct ahash_request *req);crypto_completion_t compl;void *data; +struct scatterlist sg; +const u8 *src; +u8 *page; +unsigned int offset; +unsigned int nbytes; };static void ahash_reqchain_done (void *data,int err);static int ahash_save_req (struct ahash_request *req,crypto_completion_t cplt); -static void ahash_restore_req (struct ahash_request *req); +static void ahash_restore_req (struct ahash_save_req_state *state); +static void ahash_def_finup_done1 (void *data,int err);static int ahash_def_finup (struct ahash_request *req);/*
@@ -100,6 +106,10 @@ int shash_ahash_digest(struct ahash_request *req, struct shash_desc *desc)unsigned int offset;int err; +if (ahash_request_isvirt (req)) +return crypto_shash_digest (desc, req->svirt, nbytes, + req->result); +if (nbytes && (sg = req->src, offset = sg->offset, nbytes <=min (sg->length, ((unsigned int )(PAGE_SIZE )) - offset))) {
@@ -182,6 +192,9 @@ static int hash_walk_new_entry(struct crypto_hash_walk *walk)int crypto_hash_walk_done (struct crypto_hash_walk *walk,int err) { +if ((walk->flags &CRYPTO_AHASH_REQ_VIRT )) +return err; + walk->data -= walk->offset;kunmap_local (walk->data);
@@ -209,14 +222,20 @@ int crypto_hash_walk_first(struct ahash_request *req, struct crypto_hash_walk *walk) { walk->total = req->nbytes; + walk->entrylen =0 ; -if (!walk->total) { - walk->entrylen =0 ; +if (!walk->total)return 0 ; + + walk->flags = req->base.flags; + +if (ahash_request_isvirt (req)) { + walk->data = req->svirt; + walk->total =0 ; +return req->nbytes; } walk->sg = req->src; - walk->flags = req->base.flags;return hash_walk_new_entry (walk); }
@@ -264,18 +283,82 @@ int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key, }EXPORT_SYMBOL_GPL (crypto_ahash_setkey); +static bool ahash_request_hasvirt (struct ahash_request *req) +{ +struct ahash_request *r2; + +if (ahash_request_isvirt (req)) +return true; + +list_for_each_entry (r2, &req->base.list, base.list) +if (ahash_request_isvirt (r2)) +return true; + +return false; +} + +static int ahash_reqchain_virt (struct ahash_save_req_state *state, +int err,u32 mask) +{ +struct ahash_request *req = state->cur; + +for (;;) { +unsigned len = state->nbytes; + + req->base.err = err; + +if (!state->offset) +break ; + +if (state->offset == len || err) { +u8 *result = req->result; + +ahash_request_set_virt (req, state->src, result, len); + state->offset =0 ; +break ; + } + + len -= state->offset; + + len =min (PAGE_SIZE , len); +memcpy (state->page, state->src + state->offset, len); + state->offset += len; + req->nbytes = len; + + err = state->op (req); +if (err == -EINPROGRESS ) { +if (!list_empty (&state->head) || + state->offset < state->nbytes) + err = -EBUSY ; +break ; + } + +if (err == -EBUSY ) +break ; + } + +return err; +} +static int ahash_reqchain_finish (struct ahash_save_req_state *state,int err,u32 mask) {struct ahash_request *req0 = state->req0;struct ahash_request *req = state->cur; +struct crypto_ahash *tfm;struct ahash_request *n; +bool update; - req->base.err = err; + err =ahash_reqchain_virt (state, err, mask); +if (err == -EINPROGRESS || err == -EBUSY ) + goto out;if (req != req0)list_add_tail (&req->base.list, &req0->base.list); + tfm =crypto_ahash_reqtfm (req); + update = state->op ==crypto_ahash_alg (tfm)->update; +list_for_each_entry_safe (req, n, &state->head, base.list) {list_del_init (&req->base.list);
@@ -283,10 +366,27 @@ static int ahash_reqchain_finish(struct ahash_save_req_state *state, req->base.complete = ahash_reqchain_done; req->base.data = state; state->cur = req; + +if (update &&ahash_request_isvirt (req) && req->nbytes) { +unsigned len = req->nbytes; +u8 *result = req->result; + + state->src = req->svirt; + state->nbytes = len; + + len =min (PAGE_SIZE , len); + +memcpy (state->page, req->svirt, len); + state->offset = len; + +ahash_request_set_crypt (req, &state->sg, result, len); + } + err = state->op (req);if (err == -EINPROGRESS ) { -if (!list_empty (&state->head)) +if (!list_empty (&state->head) || + state->offset < state->nbytes) err = -EBUSY ; goto out; }
@@ -294,11 +394,14 @@ static int ahash_reqchain_finish(struct ahash_save_req_state *state,if (err == -EBUSY ) goto out; - req->base.err = err; + err =ahash_reqchain_virt (state, err, mask); +if (err == -EINPROGRESS || err == -EBUSY ) + goto out; +list_add_tail (&req->base.list, &req0->base.list); } -ahash_restore_req (req0); +ahash_restore_req (state); out:return err;
@@ -312,7 +415,7 @@ static void ahash_reqchain_done(void *data, int err) data = state->data;if (err == -EINPROGRESS ) { -if (!list_empty (&state->head)) +if (!list_empty (&state->head) || state->offset < state->nbytes)return ; goto notify; }
@@ -329,40 +432,84 @@ static int ahash_do_req_chain(struct ahash_request *req, int (*op)(struct ahash_request *req)) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +bool update = op ==crypto_ahash_alg (tfm)->update;struct ahash_save_req_state *state;struct ahash_save_req_state state0; +struct ahash_request *r2; +u8 *page =NULL ;int err; -if (!ahash_request_chained (req) ||crypto_ahash_req_chain (tfm)) +if (crypto_ahash_req_chain (tfm) || + (!ahash_request_chained (req) && + (!update || !ahash_request_isvirt (req))))return op (req); - state = &state0; +if (update &&ahash_request_hasvirt (req)) { +gfp_t gfp; +u32 flags; + flags =ahash_request_flags (req); + gfp = (flags &CRYPTO_TFM_REQ_MAY_SLEEP ) ? +GFP_KERNEL :GFP_ATOMIC ; + page = (void *)__get_free_page (gfp); + err = -ENOMEM ; +if (!page) + goto out_set_chain; + } + + state = &state0;if (ahash_is_async (tfm)) { err =ahash_save_req (req, ahash_reqchain_done); -if (err) { -struct ahash_request *r2; - - req->base.err = err; -list_for_each_entry (r2, &req->base.list, base.list) - r2->base.err = err; - -return err; - } +if (err) + goto out_free_page; state = req->base.data; } state->op = op; state->cur = req; + state->page = page; + state->offset =0 ; + state->nbytes =0 ;INIT_LIST_HEAD (&state->head);list_splice_init (&req->base.list, &state->head); +if (page) +sg_init_one (&state->sg, page,PAGE_SIZE ); + +if (update &&ahash_request_isvirt (req) && req->nbytes) { +unsigned len = req->nbytes; +u8 *result = req->result; + + state->src = req->svirt; + state->nbytes = len; + + len =min (PAGE_SIZE , len); + +memcpy (page, req->svirt, len); + state->offset = len; + +ahash_request_set_crypt (req, &state->sg, result, len); + } + err =op (req);if (err == -EBUSY || err == -EINPROGRESS )return -EBUSY ;return ahash_reqchain_finish (state, err, ~0 ); + +out_free_page: +if (page) { +memset (page,0 ,PAGE_SIZE ); +free_page ((unsigned long )page); + } + +out_set_chain: + req->base.err = err; +list_for_each_entry (r2, &req->base.list, base.list) + r2->base.err = err; + +return err; }int crypto_ahash_init (struct ahash_request *req)
@@ -414,15 +561,19 @@ static int ahash_save_req(struct ahash_request *req, crypto_completion_t cplt) req->base.complete = cplt; req->base.data = state; state->req0 = req; + state->page =NULL ;return 0 ; } -static void ahash_restore_req (struct ahash_request *req) +static void ahash_restore_req (struct ahash_save_req_state *state) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); -struct ahash_save_req_state *state; +struct ahash_request *req = state->req0; +struct crypto_ahash *tfm; +free_page ((unsigned long )state->page); + + tfm =crypto_ahash_reqtfm (req);if (!ahash_is_async (tfm))return ;
@@ -504,13 +655,74 @@ int crypto_ahash_finup(struct ahash_request *req)return err; } -if (!crypto_ahash_alg (tfm)->finup) +if (!crypto_ahash_alg (tfm)->finup || + (!crypto_ahash_req_chain (tfm) &&ahash_request_hasvirt (req)))return ahash_def_finup (req);return ahash_do_req_chain (req,crypto_ahash_alg (tfm)->finup); }EXPORT_SYMBOL_GPL (crypto_ahash_finup); +static int ahash_def_digest_finish (struct ahash_save_req_state *state,int err) +{ +struct ahash_request *req = state->req0; +struct crypto_ahash *tfm; + +if (err) + goto out; + + tfm =crypto_ahash_reqtfm (req); +if (ahash_is_async (tfm)) + req->base.complete = ahash_def_finup_done1; + + err =crypto_ahash_update (req); +if (err == -EINPROGRESS || err == -EBUSY ) +return err; + +out: +ahash_restore_req (state); +return err; +} + +static void ahash_def_digest_done (void *data,int err) +{ +struct ahash_save_req_state *state0 = data; +struct ahash_save_req_state state; +struct ahash_request *areq; + + state = *state0; + areq = state.req0; +if (err == -EINPROGRESS ) + goto out; + + areq->base.flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP ; + + err =ahash_def_digest_finish (state0, err); +if (err == -EINPROGRESS || err == -EBUSY ) +return ; + +out: + state.compl (state.data, err); +} + +static int ahash_def_digest (struct ahash_request *req) +{ +struct ahash_save_req_state *state; +int err; + + err =ahash_save_req (req, ahash_def_digest_done); +if (err) +return err; + + state = req->base.data; + + err =crypto_ahash_init (req); +if (err == -EINPROGRESS || err == -EBUSY ) +return err; + +return ahash_def_digest_finish (state, err); +} +int crypto_ahash_digest (struct ahash_request *req) {struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);
@@ -532,6 +744,9 @@ int crypto_ahash_digest(struct ahash_request *req)return err; } +if (!crypto_ahash_req_chain (tfm) &&ahash_request_hasvirt (req)) +return ahash_def_digest (req); +if (crypto_ahash_get_flags (tfm) &CRYPTO_TFM_NEED_KEY )return -ENOKEY ;
@@ -547,17 +762,19 @@ static void ahash_def_finup_done2(void *data, int err)if (err == -EINPROGRESS )return ; -ahash_restore_req (areq); +ahash_restore_req (state);ahash_request_complete (areq, err); } -static int ahash_def_finup_finish1 (struct ahash_request *req,int err) +static int ahash_def_finup_finish1 (struct ahash_save_req_state *state,int err) { -struct crypto_ahash *tfm =crypto_ahash_reqtfm (req); +struct ahash_request *req = state->req0; +struct crypto_ahash *tfm;if (err) goto out; + tfm =crypto_ahash_reqtfm (req);if (ahash_is_async (tfm)) req->base.complete = ahash_def_finup_done2;
@@ -566,7 +783,7 @@ static int ahash_def_finup_finish1(struct ahash_request *req, int err)return err; out: -ahash_restore_req (req); +ahash_restore_req (state);return err; }
@@ -583,7 +800,7 @@ static void ahash_def_finup_done1(void *data, int err) areq->base.flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP ; - err =ahash_def_finup_finish1 (areq, err); + err =ahash_def_finup_finish1 (state0, err);if (err == -EINPROGRESS || err == -EBUSY )return ;
@@ -593,17 +810,20 @@ static void ahash_def_finup_done1(void *data, int err)static int ahash_def_finup (struct ahash_request *req) { +struct ahash_save_req_state *state;int err; err =ahash_save_req (req, ahash_def_finup_done1);if (err)return err; + state = req->base.data; + err =crypto_ahash_update (req);if (err == -EINPROGRESS || err == -EBUSY )return err; -return ahash_def_finup_finish1 (req, err); +return ahash_def_finup_finish1 (state, err); }int crypto_ahash_export (struct ahash_request *req,void *out)
include/crypto/hash.h +36 -2
@@ -12,6 +12,9 @@#include <linux/crypto.h> #include <linux/string.h> +/* Set this bit for virtual address instead of SG list. */ +#define CRYPTO_AHASH_REQ_VIRT 0x00000001 +struct crypto_ahash ;/**
@@ -52,7 +55,10 @@ struct ahash_request {struct crypto_async_request base;unsigned int nbytes; -struct scatterlist *src; +union { +struct scatterlist *src; +const u8 *svirt; + };u8 *result; void *__ctx[] CRYPTO_MINALIGN_ATTR;
@@ -610,9 +616,13 @@ static inline void ahash_request_set_callback(struct ahash_request *req,crypto_completion_t compl, void *data) { + u32 keep = CRYPTO_AHASH_REQ_VIRT; + req->base.complete = compl; req->base.data = data; - req->base.flags = flags; + flags &= ~keep; + req->base.flags &= keep; + req->base.flags |= flags;crypto_reqchain_init (&req->base); }
@@ -636,6 +646,30 @@ static inline void ahash_request_set_crypt(struct ahash_request *req, req->src = src; req->nbytes = nbytes; req->result = result; + req->base.flags &= ~CRYPTO_AHASH_REQ_VIRT ; +} + +/** +* ahash_request_set_virt() - set virtual address data buffers +* @req: ahash_request handle to be updated +* @src: source virtual address +* @result: buffer that is filled with the message digest -- the caller must +* ensure that the buffer has sufficient space by, for example, calling +* crypto_ahash_digestsize() +* @nbytes: number of bytes to process from the source virtual address +* +* By using this call, the caller references the source virtual address. +* The source virtual address points to the data the message digest is to +* be calculated for. +*/ +static inline void ahash_request_set_virt (struct ahash_request *req, +const u8 *src,u8 *result, +unsigned int nbytes) +{ + req->svirt = src; + req->nbytes = nbytes; + req->result = result; + req->base.flags |=CRYPTO_AHASH_REQ_VIRT ; } static inline void ahash_request_chain(struct ahash_request *req,
include/crypto/internal/hash.h +6 -1
@@ -15,7 +15,7 @@ struct ahash_request;struct scatterlist ;struct crypto_hash_walk { -char *data; +const char *data;unsigned int offset;unsigned int flags;
@@ -275,6 +275,11 @@ static inline bool ahash_request_chained(struct ahash_request *req)return crypto_request_chained (&req->base); } +static inline bool ahash_request_isvirt (struct ahash_request *req) +{ +return req->base.flags &CRYPTO_AHASH_REQ_VIRT ; +} +static inline bool crypto_ahash_req_chain (struct crypto_ahash *tfm) {return crypto_tfm_req_chain (&tfm->base);
include/linux/crypto.h +1 -1
@@ -126,7 +126,7 @@*/ #define CRYPTO_ALG_FIPS_INTERNAL 0x00020000 -/* Set if the algorithm supports request chains. */ +/* Set if the algorithm supports request chains and virtual addresses. */ #define CRYPTO_ALG_REQ_CHAIN 0x00040000/*
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 6/11 crypto: ahash - Set default reqsize from ahash_alg
In reply to Herbert XuAdd a reqsize field to struct ahash_alg and use it to set the default reqsize so that algorithms with a static reqsize are not forced to create an init_tfm function.
--- crypto/ahash.c | 4 ++++ include/crypto/hash.h | 3 +++ 2 files changed, 7 insertions(+)
crypto/ahash.c +4 -0
@@ -862,6 +862,7 @@ static int crypto_ahash_init_tfm(struct crypto_tfm *tfm)struct ahash_alg *alg =crypto_ahash_alg (hash);crypto_ahash_set_statesize (hash, alg->halg.statesize); +crypto_ahash_set_reqsize (hash, alg->reqsize);if (tfm->__crt_alg->cra_type == &crypto_shash_type)return crypto_init_ahash_using_shash (tfm);
@@ -1027,6 +1028,9 @@ static int ahash_prepare_alg(struct ahash_alg *alg)if (alg->halg.statesize ==0 )return -EINVAL ; +if (alg->reqsize && alg->reqsize < alg->halg.statesize) +return -EINVAL ; + err =hash_prepare_alg (&alg->halg);if (err)return err;
include/crypto/hash.h +3 -0
@@ -135,6 +135,7 @@ struct ahash_request {* This is a counterpart to @init_tfm, used to remove * various changes set in @init_tfm. * @clone_tfm: Copy transform into new object, may allocate memory. +* @reqsize: Size of the request context. * @halg: see struct hash_alg_common */ struct ahash_alg {
@@ -151,6 +152,8 @@ struct ahash_alg {void (*exit_tfm)(struct crypto_ahash *tfm);int (*clone_tfm)(struct crypto_ahash *dst,struct crypto_ahash *src); +unsigned int reqsize; +struct hash_alg_common halg; };
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 7/11 crypto: testmgr - Add multibuffer hash testing
In reply to Herbert XuThis is based on a patch by Eric Biggers <[email protected]>.
Add limited self-test for multibuffer hash code path. This tests only a single request in chain of a random length. The other requests are all of the same length as the one being tested.
Potential extension include testing all requests rather than just the single one, and varying the length of each request.
--- crypto/testmgr.c | 132 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 103 insertions(+), 29 deletions(-)
crypto/testmgr.c +103 -29
@@ -58,6 +58,9 @@ module_param(fuzz_iterations, uint, 0644);MODULE_PARM_DESC (fuzz_iterations,"number of fuzz test iterations" );#endif +/* Multibuffer hashing is unlimited. Set arbitrary limit for testing. */ +#define HASH_TEST_MAX_MB_MSGS 16 +#ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS /* a perfect nop */
@@ -299,6 +302,11 @@ struct test_sg_division {* @key_offset_relative_to_alignmask: if true, add the algorithm's alignmask to * the @key_offset * @finalization_type: what finalization function to use for hashes +* @multibuffer: test with multibuffer +* @multibuffer_index: random number used to generate the message index to use +* for multibuffer. +* @multibuffer_count: random number used to generate the num_msgs parameter +* for multibuffer * @nosimd: execute with SIMD disabled? Requires !CRYPTO_TFM_REQ_MAY_SLEEP. * This applies to the parts of the operation that aren't controlled * individually by @nosimd_setkey or @src_divs[].nosimd.
@@ -318,6 +326,9 @@ struct testvec_config {enum finalization_type finalization_type;bool nosimd;bool nosimd_setkey; +bool multibuffer; +unsigned int multibuffer_index; +unsigned int multibuffer_count; };#define TESTVEC_CONFIG_NAMELEN 192
@@ -1146,6 +1157,13 @@ static void generate_random_testvec_config(struct rnd_state *rng,break ; } +if (prandom_bool (rng)) { + cfg->multibuffer = true; + cfg->multibuffer_index =prandom_u32_state (rng); + cfg->multibuffer_count =prandom_u32_state (rng); + p +=scnprintf (p, end - p," multibuffer" ); + } +if (!(cfg->req_flags &CRYPTO_TFM_REQ_MAY_SLEEP )) {if (prandom_bool (rng)) { cfg->nosimd = true;
@@ -1446,16 +1464,61 @@ static int test_shash_vec_cfg(const struct hash_testvec *vec, driver, cfg); } -static int do_ahash_op (int (*op)(struct ahash_request *req), -struct ahash_request *req, -struct crypto_wait *wait,bool nosimd) +static int do_ahash_op_multibuffer ( +int (*op)(struct ahash_request *req), +struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS ], +struct crypto_wait *wait, +const struct testvec_config *cfg) { +struct ahash_request *req = reqs[0 ]; +u8 trash[HASH_MAX_DIGESTSIZE ]; +unsigned int num_msgs; +unsigned int msg_idx; +int err; +int i; + + num_msgs =1 + (cfg->multibuffer_count %HASH_TEST_MAX_MB_MSGS ); +if (num_msgs ==1 ) +return op (req); + + msg_idx = cfg->multibuffer_index % num_msgs; +for (i =1 ; i < num_msgs; i++) { +struct ahash_request *r2 = reqs[i]; + +ahash_request_set_callback (r2, req->base.flags,NULL ,NULL ); +ahash_request_set_crypt (r2, req->src, trash, req->nbytes); +ahash_request_chain (r2, req); + } + +if (msg_idx) { + reqs[msg_idx]->result = req->result; + req->result = trash; + } + + err =op (req); + +if (msg_idx) + req->result = reqs[msg_idx]->result; + +return err; +} + +static int do_ahash_op (int (*op)(struct ahash_request *req), +struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS ], +struct crypto_wait *wait, +const struct testvec_config *cfg, +bool nosimd) +{ +struct ahash_request *req = reqs[0 ];int err;if (nosimd)crypto_disable_simd_for_test (); - err =op (req); +if (cfg->multibuffer) + err =do_ahash_op_multibuffer (op, reqs, wait, cfg); +else + err =op (req);if (nosimd)crypto_reenable_simd_for_test ();
@@ -1485,10 +1548,11 @@ static int check_nonfinal_ahash_op(const char *op, int err,static int test_ahash_vec_cfg (const struct hash_testvec *vec,const char *vec_name,const struct testvec_config *cfg, -struct ahash_request *req, +struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS ],struct test_sglist *tsgl,u8 *hashstate) { +struct ahash_request *req = reqs[0 ];struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);const unsigned int digestsize =crypto_ahash_digestsize (tfm);const unsigned int statesize =crypto_ahash_statesize (tfm);
@@ -1540,7 +1604,7 @@ static int test_ahash_vec_cfg(const struct hash_testvec *vec,ahash_request_set_callback (req, req_flags, crypto_req_done, &wait);ahash_request_set_crypt (req, tsgl->sgl, result, vec->psize); - err =do_ahash_op (crypto_ahash_digest, req, &wait, cfg->nosimd); + err =do_ahash_op (crypto_ahash_digest, reqs, &wait, cfg, cfg->nosimd);if (err) {if (err == vec->digest_error)return 0 ;
@@ -1561,7 +1625,7 @@ static int test_ahash_vec_cfg(const struct hash_testvec *vec,ahash_request_set_callback (req, req_flags, crypto_req_done, &wait);ahash_request_set_crypt (req,NULL , result,0 ); - err =do_ahash_op (crypto_ahash_init, req, &wait, cfg->nosimd); + err =do_ahash_op (crypto_ahash_init, reqs, &wait, cfg, cfg->nosimd); err =check_nonfinal_ahash_op ("init" , err, result, digestsize, driver, vec_name, cfg);if (err)
@@ -1577,8 +1641,8 @@ static int test_ahash_vec_cfg(const struct hash_testvec *vec, crypto_req_done, &wait);ahash_request_set_crypt (req, pending_sgl, result, pending_len); - err =do_ahash_op (crypto_ahash_update, req, &wait, - divs[i]->nosimd); + err =do_ahash_op (crypto_ahash_update, reqs, &wait, + cfg, divs[i]->nosimd); err =check_nonfinal_ahash_op ("update" , err, result, digestsize, driver, vec_name, cfg);
@@ -1621,12 +1685,13 @@ static int test_ahash_vec_cfg(const struct hash_testvec *vec,ahash_request_set_crypt (req, pending_sgl, result, pending_len);if (cfg->finalization_type ==FINALIZATION_TYPE_FINAL ) {/* finish with update() and final() */ - err =do_ahash_op (crypto_ahash_update, req, &wait, cfg->nosimd); + err =do_ahash_op (crypto_ahash_update, reqs, &wait, cfg, cfg->nosimd); err =check_nonfinal_ahash_op ("update" , err, result, digestsize, driver, vec_name, cfg);if (err)return err; - err =do_ahash_op (crypto_ahash_final, req, &wait, cfg->nosimd); +ahash_request_set_callback (req, req_flags, crypto_req_done, &wait); + err =do_ahash_op (crypto_ahash_final, reqs, &wait, cfg, cfg->nosimd);if (err) {pr_err ("alg: ahash: %s final() failed with err %d on test vector %s, cfg=\"%s\"\n" , driver, err, vec_name, cfg->name);
@@ -1634,7 +1699,7 @@ static int test_ahash_vec_cfg(const struct hash_testvec *vec, } }else {/* finish with finup() */ - err =do_ahash_op (crypto_ahash_finup, req, &wait, cfg->nosimd); + err =do_ahash_op (crypto_ahash_finup, reqs, &wait, cfg, cfg->nosimd);if (err) {pr_err ("alg: ahash: %s finup() failed with err %d on test vector %s, cfg=\"%s\"\n" , driver, err, vec_name, cfg->name);
@@ -1650,7 +1715,7 @@ static int test_ahash_vec_cfg(const struct hash_testvec *vec,static int test_hash_vec_cfg (const struct hash_testvec *vec,const char *vec_name,const struct testvec_config *cfg, -struct ahash_request *req, +struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS ],struct shash_desc *desc,struct test_sglist *tsgl,u8 *hashstate)
@@ -1670,11 +1735,12 @@ static int test_hash_vec_cfg(const struct hash_testvec *vec,return err; } -return test_ahash_vec_cfg (vec, vec_name, cfg, req, tsgl, hashstate); +return test_ahash_vec_cfg (vec, vec_name, cfg, reqs, tsgl, hashstate); }static int test_hash_vec (const struct hash_testvec *vec,unsigned int vec_num, -struct ahash_request *req,struct shash_desc *desc, +struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS ], +struct shash_desc *desc,struct test_sglist *tsgl,u8 *hashstate) {char vec_name[16 ];
@@ -1686,7 +1752,7 @@ static int test_hash_vec(const struct hash_testvec *vec, unsigned int vec_num,for (i =0 ; i <ARRAY_SIZE (default_hash_testvec_configs); i++) { err =test_hash_vec_cfg (vec, vec_name, &default_hash_testvec_configs[i], - req, desc, tsgl, hashstate); + reqs, desc, tsgl, hashstate);if (err)return err; }
@@ -1703,7 +1769,7 @@ static int test_hash_vec(const struct hash_testvec *vec, unsigned int vec_num,generate_random_testvec_config (&rng, &cfg, cfgname,sizeof (cfgname)); err =test_hash_vec_cfg (vec, vec_name, &cfg, - req, desc, tsgl, hashstate); + reqs, desc, tsgl, hashstate);if (err)return err;cond_resched ();
@@ -1762,11 +1828,12 @@ static void generate_random_hash_testvec(struct rnd_state *rng,*/ static int test_hash_vs_generic_impl (const char *generic_driver,unsigned int maxkeysize, -struct ahash_request *req, +struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS ],struct shash_desc *desc,struct test_sglist *tsgl,u8 *hashstate) { +struct ahash_request *req = reqs[0 ];struct crypto_ahash *tfm =crypto_ahash_reqtfm (req);const unsigned int digestsize =crypto_ahash_digestsize (tfm);const unsigned int blocksize =crypto_ahash_blocksize (tfm);
@@ -1864,7 +1931,7 @@ static int test_hash_vs_generic_impl(const char *generic_driver, sizeof(cfgname)); err =test_hash_vec_cfg (&vec, vec_name, cfg, - req, desc, tsgl, hashstate); + reqs, desc, tsgl, hashstate);if (err) goto out;cond_resched ();
@@ -1929,8 +1996,8 @@ static int __alg_test_hash(const struct hash_testvec *vecs, u32 type, u32 mask, const char *generic_driver, unsigned int maxkeysize) { + struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS] = {};struct crypto_ahash *atfm =NULL ; -struct ahash_request *req =NULL ;struct crypto_shash *stfm =NULL ;struct shash_desc *desc =NULL ;struct test_sglist *tsgl =NULL ;
@@ -1954,12 +2021,14 @@ static int __alg_test_hash(const struct hash_testvec *vecs, } driver =crypto_ahash_driver_name (atfm); - req =ahash_request_alloc (atfm,GFP_KERNEL ); -if (!req) { -pr_err ("alg: hash: failed to allocate request for %s\n" , - driver); - err = -ENOMEM ; - goto out; +for (i =0 ; i <HASH_TEST_MAX_MB_MSGS ; i++) { + reqs[i] =ahash_request_alloc (atfm,GFP_KERNEL ); +if (!reqs[i]) { +pr_err ("alg: hash: failed to allocate request for %s\n" , + driver); + err = -ENOMEM ; + goto out; + } }/*
@@ -1995,12 +2064,12 @@ static int __alg_test_hash(const struct hash_testvec *vecs,if (fips_enabled && vecs[i].fips_skip)continue ; - err =test_hash_vec (&vecs[i], i, req, desc, tsgl, hashstate); + err =test_hash_vec (&vecs[i], i, reqs, desc, tsgl, hashstate);if (err) goto out;cond_resched (); } - err =test_hash_vs_generic_impl (generic_driver, maxkeysize, req, + err =test_hash_vs_generic_impl (generic_driver, maxkeysize, reqs, desc, tsgl, hashstate); out:kfree (hashstate);
@@ -2010,7 +2079,12 @@ static int __alg_test_hash(const struct hash_testvec *vecs, }kfree (desc);crypto_free_shash (stfm); -ahash_request_free (req); +if (reqs[0 ]) { +ahash_request_set_callback (reqs[0 ],0 ,NULL ,NULL ); +for (i =1 ; i <HASH_TEST_MAX_MB_MSGS && reqs[i]; i++) +ahash_request_chain (reqs[i], reqs[0 ]); +ahash_request_free (reqs[0 ]); + }crypto_free_ahash (atfm);return err; }
Signature · 1 line
2.39.5
kernel test robot
build botRe: [v2 PATCH 07/11] crypto: testmgr - Add multibuffer hash testing
kernel test robot noticed the following build errors:
Hi Herbert,
kernel test robot noticed the following build errors:
[auto build test ERROR on herbert-cryptodev-2.6/master]
[also build test ERROR on next-20250214]
[cannot apply to herbert-crypto-2.6/master brauner-vfs/vfs.all linus/master v6.14-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Herbert-Xu/crypto-ahash-Only-save-callback-and-data-in-ahash_save_req/20250216-150941 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master patch link: https://lore.kernel.org/r/bb74b3f54e24308bef2def8d25ed917d13590921.1739674648.git.herbert%40gondor.apana.org.au patch subject: [v2 PATCH 07/11] crypto: testmgr - Add multibuffer hash testing config: arc-randconfig-001-20250216 (https://download.01.org/0day-ci/archive/20250216/[email protected]/config) compiler: arceb-elf-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250216/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
crypto/testmgr.c: In function '__alg_test_hash':
>> crypto/testmgr.c:2072:69: error: passing argument 3 of 'test_hash_vs_generic_impl' from incompatible pointer type [-Werror=incompatible-pointer-types]
2072 | err = test_hash_vs_generic_impl(generic_driver, maxkeysize, reqs,
| ^~~~
| |
| struct ahash_request **
crypto/testmgr.c:1952:60: note: expected 'struct ahash_request *' but argument is of type 'struct ahash_request **'
1952 | struct ahash_request *req,
| ~~~~~~~~~~~~~~~~~~~~~~^~~cc1: some warnings being treated as errors
vim +/test_hash_vs_generic_impl +2072 crypto/testmgr.c
1993
1994 static int __alg_test_hash(const struct hash_testvec *vecs,
1995 unsigned int num_vecs, const char *driver,
1996 u32 type, u32 mask,
1997 const char *generic_driver, unsigned int maxkeysize)
1998 {
1999 struct ahash_request *reqs[HASH_TEST_MAX_MB_MSGS] = {};
2000 struct crypto_ahash *atfm = NULL;
2001 struct crypto_shash *stfm = NULL;
2002 struct shash_desc *desc = NULL;
2003 struct test_sglist *tsgl = NULL;
2004 u8 *hashstate = NULL;
2005 unsigned int statesize;
2006 unsigned int i;
2007 int err;
2008
2009 /*
2010 * Always test the ahash API. This works regardless of whether the
2011 * algorithm is implemented as ahash or shash.
2012 */
2013
2014 atfm = crypto_alloc_ahash(driver, type, mask);
2015 if (IS_ERR(atfm)) {
2016 if (PTR_ERR(atfm) == -ENOENT)
2017 return 0;
2018 pr_err("alg: hash: failed to allocate transform for %s: %ld\n",
2019 driver, PTR_ERR(atfm));
2020 return PTR_ERR(atfm);
2021 }
2022 driver = crypto_ahash_driver_name(atfm);
2023
2024 for (i = 0; i < HASH_TEST_MAX_MB_MSGS; i++) {
2025 reqs[i] = ahash_request_alloc(atfm, GFP_KERNEL);
2026 if (!reqs[i]) {
2027 pr_err("alg: hash: failed to allocate request for %s\n",
2028 driver);
2029 err = -ENOMEM;
2030 goto out;
2031 }
2032 }
2033
2034 /*
2035 * If available also test the shash API, to cover corner cases that may
2036 * be missed by testing the ahash API only.
2037 */
2038 err = alloc_shash(driver, type, mask, &stfm, &desc);
2039 if (err)
2040 goto out;
2041
2042 tsgl = kmalloc(sizeof(*tsgl), GFP_KERNEL);
2043 if (!tsgl || init_test_sglist(tsgl) != 0) {
2044 pr_err("alg: hash: failed to allocate test buffers for %s\n",
2045 driver);
2046 kfree(tsgl);
2047 tsgl = NULL;
2048 err = -ENOMEM;
2049 goto out;
2050 }
2051
2052 statesize = crypto_ahash_statesize(atfm);
2053 if (stfm)
2054 statesize = max(statesize, crypto_shash_statesize(stfm));
2055 hashstate = kmalloc(statesize + TESTMGR_POISON_LEN, GFP_KERNEL);
2056 if (!hashstate) {
2057 pr_err("alg: hash: failed to allocate hash state buffer for %s\n",
2058 driver);
2059 err = -ENOMEM;
2060 goto out;
2061 }
2062
2063 for (i = 0; i < num_vecs; i++) {
2064 if (fips_enabled && vecs[i].fips_skip)
2065 continue;
2066
2067 err = test_hash_vec(&vecs[i], i, reqs, desc, tsgl, hashstate);
2068 if (err)
2069 goto out;
2070 cond_resched();
2071 }
> 2072 err = test_hash_vs_generic_impl(generic_driver, maxkeysize, reqs, 2073 desc, tsgl, hashstate);
2074 out:
2075 kfree(hashstate);
2076 if (tsgl) {
2077 destroy_test_sglist(tsgl);
2078 kfree(tsgl);
2079 }
2080 kfree(desc);
2081 crypto_free_shash(stfm);
2082 if (reqs[0]) {
2083 ahash_request_set_callback(reqs[0], 0, NULL, NULL);
2084 for (i = 1; i < HASH_TEST_MAX_MB_MSGS && reqs[i]; i++)
2085 ahash_request_chain(reqs[i], reqs[0]);
2086 ahash_request_free(reqs[0]);
2087 }
2088 crypto_free_ahash(atfm);
2089 return err;
2090 }
2091
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Herbert Xu
maintainerPATCH v2 8/11 crypto: x86/sha2 - Restore multibuffer AVX2 support
In reply to Herbert XuResurrect the old multibuffer AVX2 code removed by commit ab8085c130ed ("crypto: x86 - remove SHA multibuffer routines and mcryptd") using the new request chaining interface.
This is purely a proof of concept and only meant to illustrate the utility of the new API rather than a serious attempt at improving the performance.
However, it is interesting to note that with x8 multibuffer the performance of AVX2 is on par with SHA-NI.
testing speed of multibuffer sha256 (sha256-avx2)
tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 1 operation in 184 cycles (16 bytes)
tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 1 operation in 165 cycles (64 bytes)
tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 1 operation in 444 cycles (256 bytes)
tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 1 operation in 1549 cycles (1024 bytes)
tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 1 operation in 3060 cycles (2048 bytes)
tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 1 operation in 5983 cycles (4096 bytes)
tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 1 operation in 11980 cycles (8192 bytes)
tcrypt: testing speed of async sha256 (sha256-avx2)
tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 475 cycles/operation, 29 cycles/byte
tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 780 cycles/operation, 12 cycles/byte
tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 1872 cycles/operation, 7 cycles/byte
tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 5416 cycles/operation, 5 cycles/byte
tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 10339 cycles/operation, 5 cycles/byte
tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 20214 cycles/operation, 4 cycles/byte
tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 40042 cycles/operation, 4 cycles/byte
tcrypt: testing speed of async sha256-ni (sha256-ni)
tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 207 cycles/operation, 12 cycles/byte
tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 299 cycles/operation, 4 cycles/byte
tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 543 cycles/operation, 2 cycles/byte
tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 1523 cycles/operation, 1 cycles/byte
tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 2835 cycles/operation, 1 cycles/byte
tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 5459 cycles/operation, 1 cycles/byte
tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 10724 cycles/operation, 1 cycles/byte
--- arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/sha256_mb_mgr_datastruct.S | 304 +++++++++++ arch/x86/crypto/sha256_ssse3_glue.c | 540 +++++++++++++++++-- arch/x86/crypto/sha256_x8_avx2.S | 598 +++++++++++++++++++++ 4 files changed, 1401 insertions(+), 43 deletions(-) create mode 100644 arch/x86/crypto/sha256_mb_mgr_datastruct.S create mode 100644 arch/x86/crypto/sha256_x8_avx2.S
arch/x86/crypto/Makefile +1 -1
@@ -60,7 +60,7 @@ sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o -sha256-ssse3-y := sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256_ssse3_glue.o +sha256-ssse3-y := sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256_ssse3_glue.o sha256_x8_avx2.o sha256-ssse3-$(CONFIG_AS_SHA256_NI) += sha256_ni_asm.o obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o
arch/x86/crypto/sha256_mb_mgr_datastruct.S new file +304 -0
304 lines not on this page · raw message
arch/x86/crypto/sha256_ssse3_glue.c +498 -42
620 lines not on this page · raw message
arch/x86/crypto/sha256_x8_avx2.S new file +598 -0
598 lines not on this page · raw message
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 9/11 crypto: hash - Add sync hash interface
In reply to Herbert XuIntroduce a new sync hash interface based on ahash, similar to sync skcipher.
It will replace shash for existing users.
--- crypto/ahash.c | 37 ++++++++++++++++ include/crypto/hash.h | 100 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+)
crypto/ahash.c +37 -0
@@ -949,6 +949,27 @@ struct crypto_ahash *crypto_alloc_ahash(const char *alg_name, u32 type, }EXPORT_SYMBOL_GPL (crypto_alloc_ahash); +struct crypto_sync_hash *crypto_alloc_sync_hash (const char *alg_name, +u32 type,u32 mask) +{ +struct crypto_ahash *tfm; + +/* Only sync algorithms allowed. */ + mask |=CRYPTO_ALG_ASYNC ; + type &= ~CRYPTO_ALG_ASYNC ; + + tfm =crypto_alloc_ahash (alg_name, type, mask); + +if (!IS_ERR (tfm) &&WARN_ON (crypto_ahash_reqsize (tfm) > +MAX_SYNC_HASH_REQSIZE )) { +crypto_free_ahash (tfm); +return ERR_PTR (-EINVAL ); + } + + return container_of(tfm, struct crypto_sync_hash, base); +} +EXPORT_SYMBOL_GPL (crypto_alloc_sync_hash); +int crypto_has_ahash (const char *alg_name,u32 type,u32 mask) {return crypto_type_has_alg (alg_name, &crypto_ahash_type, type, mask);
@@ -1123,5 +1144,21 @@ void ahash_request_free(struct ahash_request *req) }EXPORT_SYMBOL_GPL (ahash_request_free); +int crypto_sync_hash_digest (struct crypto_sync_hash *tfm,const u8 *data, +unsigned int len,u8 *out) +{ +SYNC_HASH_REQUEST_ON_STACK (req, tfm); +int err; + +ahash_request_set_callback (req,0 ,NULL ,NULL ); +ahash_request_set_virt (req, data, out, len); + err =crypto_ahash_digest (req); + +ahash_request_zero (req); + +return err; +} +EXPORT_SYMBOL_GPL (crypto_shash_tfm_digest); +MODULE_LICENSE ("GPL" );MODULE_DESCRIPTION ("Asynchronous cryptographic hash type" );
include/crypto/hash.h +100 -0
188 lines not on this page · raw message
Signature · 1 line
2.39.5
kernel test robot
build botRe: [v2 PATCH 09/11] crypto: hash - Add sync hash interface
kernel test robot noticed the following build warnings:
Hi Herbert,
kernel test robot noticed the following build warnings:
[auto build test WARNING on herbert-cryptodev-2.6/master]
[also build test WARNING on next-20250214]
[cannot apply to herbert-crypto-2.6/master brauner-vfs/vfs.all linus/master v6.14-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Herbert-Xu/crypto-ahash-Only-save-callback-and-data-in-ahash_save_req/20250216-150941 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master patch link: https://lore.kernel.org/r/d6e10dbf172f0b7c791f5406d55e8f1c74492d57.1739674648.git.herbert%40gondor.apana.org.au patch subject: [v2 PATCH 09/11] crypto: hash - Add sync hash interface config: um-randconfig-002-20250216 (https://download.01.org/0day-ci/archive/20250216/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250216/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from include/crypto/internal/hash.h:12,
from crypto/hash.h:10,
from crypto/ahash.c:26:
In function 'ahash_request_set_callback',
inlined from 'crypto_sync_hash_digest' at crypto/ahash.c:1153:2:
>> include/crypto/hash.h:690:18: warning: '*(struct ahash_request *)(&__req_req[0]).base.flags' is used uninitialized [-Wuninitialized]
690 | req->base.flags &= keep;
| ~~~~~~~~~^~~~~~
crypto/ahash.c: In function 'crypto_sync_hash_digest':
include/crypto/hash.h:183:14: note: '__req_req' declared here
183 | char __##name##_req[sizeof(struct ahash_request) + \
| ^~
crypto/ahash.c:1150:9: note: in expansion of macro 'SYNC_HASH_REQUEST_ON_STACK'
1150 | SYNC_HASH_REQUEST_ON_STACK(req, tfm);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~vim +690 include/crypto/hash.h
18e33e6d5cc049 Herbert Xu 2008-07-10 654
90240ffb127729 Stephan Mueller 2014-11-12 655 /**
90240ffb127729 Stephan Mueller 2014-11-12 656 * ahash_request_set_callback() - set asynchronous callback function
90240ffb127729 Stephan Mueller 2014-11-12 657 * @req: request handle
90240ffb127729 Stephan Mueller 2014-11-12 658 * @flags: specify zero or an ORing of the flags
90240ffb127729 Stephan Mueller 2014-11-12 659 * CRYPTO_TFM_REQ_MAY_BACKLOG the request queue may back log and
90240ffb127729 Stephan Mueller 2014-11-12 660 * increase the wait queue beyond the initial maximum size;
90240ffb127729 Stephan Mueller 2014-11-12 661 * CRYPTO_TFM_REQ_MAY_SLEEP the request processing may sleep
90240ffb127729 Stephan Mueller 2014-11-12 662 * @compl: callback function pointer to be registered with the request handle
90240ffb127729 Stephan Mueller 2014-11-12 663 * @data: The data pointer refers to memory that is not used by the kernel
90240ffb127729 Stephan Mueller 2014-11-12 664 * crypto API, but provided to the callback function for it to use. Here,
90240ffb127729 Stephan Mueller 2014-11-12 665 * the caller can provide a reference to memory the callback function can
90240ffb127729 Stephan Mueller 2014-11-12 666 * operate on. As the callback function is invoked asynchronously to the
90240ffb127729 Stephan Mueller 2014-11-12 667 * related functionality, it may need to access data structures of the
90240ffb127729 Stephan Mueller 2014-11-12 668 * related functionality which can be referenced using this pointer. The
90240ffb127729 Stephan Mueller 2014-11-12 669 * callback function can access the memory via the "data" field in the
90240ffb127729 Stephan Mueller 2014-11-12 670 * &crypto_async_request data structure provided to the callback function.
90240ffb127729 Stephan Mueller 2014-11-12 671 *
90240ffb127729 Stephan Mueller 2014-11-12 672 * This function allows setting the callback function that is triggered once
90240ffb127729 Stephan Mueller 2014-11-12 673 * the cipher operation completes.
90240ffb127729 Stephan Mueller 2014-11-12 674 *
90240ffb127729 Stephan Mueller 2014-11-12 675 * The callback function is registered with the &ahash_request handle and
0184cfe72d2f13 Stephan Mueller 2016-10-21 676 * must comply with the following template::
90240ffb127729 Stephan Mueller 2014-11-12 677 *
90240ffb127729 Stephan Mueller 2014-11-12 678 * void callback_function(struct crypto_async_request *req, int error)
90240ffb127729 Stephan Mueller 2014-11-12 679 */
18e33e6d5cc049 Herbert Xu 2008-07-10 680 static inline void ahash_request_set_callback(struct ahash_request *req,
18e33e6d5cc049 Herbert Xu 2008-07-10 681 u32 flags,
3e3dc25fe7d5e3 Mark Rustad 2014-07-25 682 crypto_completion_t compl,
18e33e6d5cc049 Herbert Xu 2008-07-10 683 void *data)
18e33e6d5cc049 Herbert Xu 2008-07-10 684 {
07b1948dc8bac5 Herbert Xu 2025-02-16 685 u32 keep = CRYPTO_AHASH_REQ_VIRT;
07b1948dc8bac5 Herbert Xu 2025-02-16 686
3e3dc25fe7d5e3 Mark Rustad 2014-07-25 687 req->base.complete = compl;
18e33e6d5cc049 Herbert Xu 2008-07-10 688 req->base.data = data;
07b1948dc8bac5 Herbert Xu 2025-02-16 689 flags &= ~keep;
07b1948dc8bac5 Herbert Xu 2025-02-16 @690 req->base.flags &= keep;
07b1948dc8bac5 Herbert Xu 2025-02-16 691 req->base.flags |= flags;
c0cd3e787da854 Herbert Xu 2025-02-16 692 crypto_reqchain_init(&req->base);
18e33e6d5cc049 Herbert Xu 2008-07-10 693 }
18e33e6d5cc049 Herbert Xu 2008-07-10 694 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [v2 PATCH 09/11] crypto: hash - Add sync hash interface
In reply to Herbert Xukernel test robot noticed the following build warnings:
Hi Herbert,
kernel test robot noticed the following build warnings:
[auto build test WARNING on herbert-cryptodev-2.6/master]
[cannot apply to herbert-crypto-2.6/master brauner-vfs/vfs.all linus/master v6.14-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Herbert-Xu/crypto-ahash-Only-save-callback-and-data-in-ahash_save_req/20250216-150941 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master patch link: https://lore.kernel.org/r/d6e10dbf172f0b7c791f5406d55e8f1c74492d57.1739674648.git.herbert%40gondor.apana.org.au patch subject: [v2 PATCH 09/11] crypto: hash - Add sync hash interface config: s390-randconfig-001-20250216 (https://download.01.org/0day-ci/archive/20250216/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250216/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>, old ones prefixed by <<):
ERROR: modpost: vmlinux: 'crypto_shash_tfm_digest' exported twice. Previous export was in vmlinux
WARNING: modpost: missing MODULE_DESCRIPTION() in lib/slub_kunit.o
>> WARNING: modpost: EXPORT symbol "crypto_shash_tfm_digest" [vmlinux] version generation failed, symbol will not be versioned.
Is "crypto_shash_tfm_digest" prototyped in <asm/asm-prototypes.h>?
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Herbert Xu
maintainerPATCH v2 10/11 fsverity: Use sync hash instead of shash
In reply to Herbert XuUse the sync hash interface instead of shash.
--- fs/verity/fsverity_private.h | 2 +- fs/verity/hash_algs.c | 41 +++++++++++++++++++----------------- 2 files changed, 23 insertions(+), 20 deletions(-)
fs/verity/fsverity_private.h +1 -1
@@ -20,7 +20,7 @@/* A hash algorithm supported by fs-verity */ struct fsverity_hash_alg { -struct crypto_shash *tfm;/* hash tfm, allocated on demand */ +struct crypto_sync_hash *tfm;/* hash tfm, allocated on demand */ const char *name;/* crypto API name, e.g. sha256 */ unsigned int digest_size;/* digest size in bytes, e.g. 32 for SHA-256 */ unsigned int block_size;/* block size in bytes, e.g. 64 for SHA-256 */
fs/verity/hash_algs.c +22 -19
116 lines not on this page · raw message
Signature · 1 line
2.39.5
PATCH v2 11/11 fsverity: improve performance by using multibuffer hashing
In reply to Herbert XuWhen supported by the hash algorithm, use crypto_shash_finup_mb() to interleave the hashing of pairs of data blocks. On some CPUs this nearly doubles hashing performance. The increase in overall throughput of cold-cache fsverity reads that I'm seeing on arm64 and x86_64 is roughly 35% (though this metric is hard to measure as it jumps around a lot).
For now this is only done on the verification path, and only for data blocks, not Merkle tree blocks. We could use finup_mb on Merkle tree blocks too, but that is less important as there aren't as many Merkle tree blocks as data blocks, and that would require some additional code restructuring. We could also use finup_mb to accelerate building the Merkle tree, but verification performance is more important.
--- fs/verity/fsverity_private.h | 2 + fs/verity/verify.c | 179 +++++++++++++++++++++++++++++------ 2 files changed, 151 insertions(+), 30 deletions(-)
fs/verity/fsverity_private.h +2 -0
@@ -152,6 +152,8 @@ static inline void fsverity_init_signature(void)/* verify.c */ +#define FS_VERITY_MAX_PENDING_DATA_BLOCKS 2 +void __init fsverity_init_workqueue (void );#endif /* _FSVERITY_PRIVATE_H */
fs/verity/verify.c +149 -30
255 lines not on this page · raw message
Signature · 1 line
2.39.5
Herbert Xu
maintainerPATCH v2 11/11 fsverity: improve performance by using multibuffer hashing
In reply to Herbert XuFrom: Eric Biggers <[email protected]>
When supported by the hash algorithm, use crypto_shash_finup_mb() to interleave the hashing of pairs of data blocks. On some CPUs this nearly doubles hashing performance. The increase in overall throughput of cold-cache fsverity reads that I'm seeing on arm64 and x86_64 is roughly 35% (though this metric is hard to measure as it jumps around a lot).
For now this is only done on the verification path, and only for data blocks, not Merkle tree blocks. We could use finup_mb on Merkle tree blocks too, but that is less important as there aren't as many Merkle tree blocks as data blocks, and that would require some additional code restructuring. We could also use finup_mb to accelerate building the Merkle tree, but verification performance is more important.
--- fs/verity/fsverity_private.h | 2 + fs/verity/verify.c | 179 +++++++++++++++++++++++++++++------ 2 files changed, 151 insertions(+), 30 deletions(-)
fs/verity/fsverity_private.h +2 -0
@@ -152,6 +152,8 @@ static inline void fsverity_init_signature(void)/* verify.c */ +#define FS_VERITY_MAX_PENDING_DATA_BLOCKS 2 +void __init fsverity_init_workqueue (void );#endif /* _FSVERITY_PRIVATE_H */
fs/verity/verify.c +149 -30
255 lines not on this page · raw message
Signature · 1 line
2.39.5
kernel test robot
build botRe: [v2 PATCH 11/11] fsverity: improve performance by using multibuffer hashing
kernel test robot noticed the following build warnings:
Hi Herbert,
kernel test robot noticed the following build warnings:
[auto build test WARNING on herbert-cryptodev-2.6/master]
[also build test WARNING on next-20250214]
[cannot apply to herbert-crypto-2.6/master brauner-vfs/vfs.all linus/master v6.14-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Herbert-Xu/crypto-ahash-Only-save-callback-and-data-in-ahash_save_req/20250216-150941 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master patch link: https://lore.kernel.org/r/dd11ea3e7231fb46f68d902ba38a5b688e378bfe.1739674648.1.git.herbert%40gondor.apana.org.au patch subject: [v2 PATCH 11/11] fsverity: improve performance by using multibuffer hashing config: hexagon-allyesconfig (https://download.01.org/0day-ci/archive/20250217/[email protected]/config) compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250217/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> fs/verity/verify.c:295:1: warning: stack frame size (1312) exceeds limit (1024) in 'fsverity_verify_pending_blocks' [-Wframe-larger-than]
295 | fsverity_verify_pending_blocks(struct fsverity_verification_context *ctx)
| ^
1 warning generated.
vim +/fsverity_verify_pending_blocks +295 fs/verity/verify.c
293
294 static bool
> 295 fsverity_verify_pending_blocks(struct fsverity_verification_context *ctx)
296 {
297 struct inode *inode = ctx->inode;
298 struct fsverity_info *vi = ctx->vi;
299 const struct merkle_tree_params *params = &vi->tree_params;
300 SYNC_HASH_REQUESTS_ON_STACK(reqs, FS_VERITY_MAX_PENDING_DATA_BLOCKS, params->hash_alg->tfm);
301 struct ahash_request *req;
302 int i;
303 int err;
304
305 if (ctx->num_pending == 0)
306 return true;
307
308 req = sync_hash_requests(reqs, 0);
309 for (i = 0; i < ctx->num_pending; i++) {
310 struct ahash_request *reqi = sync_hash_requests(reqs, i);
311
312 ahash_request_set_callback(reqi, CRYPTO_TFM_REQ_MAY_SLEEP,
313 NULL, NULL);
314 ahash_request_set_virt(reqi, ctx->pending_blocks[i].data,
315 ctx->pending_blocks[i].real_hash,
316 params->block_size);
317 if (i)
318 ahash_request_chain(reqi, req);
319 if (!params->hashstate)
320 continue;
321
322 err = crypto_ahash_import(reqi, params->hashstate);
323 if (err) {
324 fsverity_err(inode, "Error %d importing hash state", err);
325 return false;
326 }
327 }
328
329 if (params->hashstate)
330 err = crypto_ahash_finup(req);
331 else
332 err = crypto_ahash_digest(req);
333 if (err) {
334 fsverity_err(inode, "Error %d computing block hashes", err);
335 return false;
336 }
337
338 for (i = 0; i < ctx->num_pending; i++) {
339 if (!verify_data_block(inode, vi, &ctx->pending_blocks[i],
340 ctx->max_ra_pages))
341 return false;
342 }
343
344 fsverity_clear_pending_blocks(ctx);
345 return true;
346 }
347
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [v2 PATCH 11/11] fsverity: improve performance by using multibuffer hashing
In reply to Herbert Xukernel test robot noticed the following build warnings:
Hi Herbert,
kernel test robot noticed the following build warnings:
[auto build test WARNING on herbert-cryptodev-2.6/master]
[also build test WARNING on next-20250214]
[cannot apply to herbert-crypto-2.6/master brauner-vfs/vfs.all linus/master v6.14-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Herbert-Xu/crypto-ahash-Only-save-callback-and-data-in-ahash_save_req/20250216-150941 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master patch link: https://lore.kernel.org/r/dd11ea3e7231fb46f68d902ba38a5b688e378bfe.1739674648.1.git.herbert%40gondor.apana.org.au patch subject: [v2 PATCH 11/11] fsverity: improve performance by using multibuffer hashing config: arc-allyesconfig (https://download.01.org/0day-ci/archive/20250217/[email protected]/config) compiler: arceb-elf-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250217/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
fs/verity/verify.c: In function 'fsverity_verify_pending_blocks':
>> fs/verity/verify.c:346:1: warning: the frame size of 1148 bytes is larger than 1024 bytes [-Wframe-larger-than=]
346 | }
| ^vim +346 fs/verity/verify.c
293
294 static bool
295 fsverity_verify_pending_blocks(struct fsverity_verification_context *ctx)
296 {
297 struct inode *inode = ctx->inode;
298 struct fsverity_info *vi = ctx->vi;
299 const struct merkle_tree_params *params = &vi->tree_params;
300 SYNC_HASH_REQUESTS_ON_STACK(reqs, FS_VERITY_MAX_PENDING_DATA_BLOCKS, params->hash_alg->tfm);
301 struct ahash_request *req;
302 int i;
303 int err;
304
305 if (ctx->num_pending == 0)
306 return true;
307
308 req = sync_hash_requests(reqs, 0);
309 for (i = 0; i < ctx->num_pending; i++) {
310 struct ahash_request *reqi = sync_hash_requests(reqs, i);
311
312 ahash_request_set_callback(reqi, CRYPTO_TFM_REQ_MAY_SLEEP,
313 NULL, NULL);
314 ahash_request_set_virt(reqi, ctx->pending_blocks[i].data,
315 ctx->pending_blocks[i].real_hash,
316 params->block_size);
317 if (i)
318 ahash_request_chain(reqi, req);
319 if (!params->hashstate)
320 continue;
321
322 err = crypto_ahash_import(reqi, params->hashstate);
323 if (err) {
324 fsverity_err(inode, "Error %d importing hash state", err);
325 return false;
326 }
327 }
328
329 if (params->hashstate)
330 err = crypto_ahash_finup(req);
331 else
332 err = crypto_ahash_digest(req);
333 if (err) {
334 fsverity_err(inode, "Error %d computing block hashes", err);
335 return false;
336 }
337
338 for (i = 0; i < ctx->num_pending; i++) {
339 if (!verify_data_block(inode, vi, &ctx->pending_blocks[i],
340 ctx->max_ra_pages))
341 return false;
342 }
343
344 fsverity_clear_pending_blocks(ctx);
345 return true;
> 346 }
347
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 50 lines · Herbert Xu
This patch-set introduces two additions to the ahash interface. First of all request chaining is added so that an arbitrary number of requests can be submitted in one go. Incidentally this also reduces the cost of indirect calls by amortisation.
It then adds virtual address support to ahash. This allows the user to supply a virtual address as the input instead of an SG list.
This is assumed to be not DMA-capable so it is always copied before it's passed to an existing ahash driver. New drivers can elect to take virtual addresses directly. Of course existing shash algorithms are able to take virtual addresses without any copying.
The next patch resurrects the old SHA2 AVX2 muiltibuffer code as a proof of concept that this API works. The result shows that with a full complement of 8 requests, this API is able to achieve parity with the more modern but single-threaded SHA-NI code. This passes the multibuffer fuzz tests.
Finally introduce a sync hash interface that is similar to the sync skcipher interface. This will replace the shash interface for users. Use it in fsverity and enable multibuffer hashing.
Eric Biggers (1):
fsverity: improve performance by using multibuffer hashingHerbert Xu (10):
crypto: ahash - Only save callback and data inahash_save_req
crypto: x86/ghash - Use proper helpers to clone request
crypto: hash - Add request chaining API
crypto: tcrypt - Restore multibuffer ahash tests
crypto: ahash - Add virtual address support
crypto: ahash - Set default reqsize fromahash_alg
crypto: testmgr - Add multibuffer hash testing
crypto: x86/sha2 - Restore multibuffer AVX2 support
crypto: hash - Add sync hash interface
fsverity: Use sync hash instead of shasharch/x86/crypto/Makefile | 2 +- arch/x86/crypto/ghash-clmulni-intel_glue.c | 23 +- arch/x86/crypto/sha256_mb_mgr_datastruct.S | 304 +++++++++++ arch/x86/crypto/sha256_ssse3_glue.c | 540 ++++++++++++++++-- arch/x86/crypto/sha256_x8_avx2.S | 598 ++++++++++++++++++++ crypto/ahash.c | 605 ++++++++++++++++++--- crypto/algapi.c | 2 +- crypto/tcrypt.c | 231 ++++++++ crypto/testmgr.c | 132 ++++- fs/verity/fsverity_private.h | 4 +- fs/verity/hash_algs.c | 41 +-
Quoted 8 lines
fs/verity/verify.c | 179 +++++- include/crypto/algapi.h | 11 + include/crypto/hash.h | 172 +++++- include/crypto/internal/hash.h | 17 +- include/linux/crypto.h | 24 + 16 files changed, 2659 insertions(+), 226 deletions(-) create mode 100644 arch/x86/crypto/sha256_mb_mgr_datastruct.S create mode 100644 arch/x86/crypto/sha256_x8_avx2.S
This new version hasn't fundamentally changed anything. It's still a much worse, unnecessarily complex and still incomplete implementation compared to my patchset which has been ready to go for nearly a year already. Please refer to all the previous feedback that I've given.
- Eric
Herbert Xu
maintainerQuoted 5 lines · Eric Biggers
This new version hasn't fundamentally changed anything. It's still a much worse, unnecessarily complex and still incomplete implementation compared to my patchset which has been ready to go for nearly a year already. Please refer to all the previous feedback that I've given.
FWIW, my interface is a lot simpler than yours to implement, since it doesn't deal with the partial buffer non-sense in assembly. In fact that was a big mistake with the original API, the partial data handling should've been moved to the API layer a long time ago.
Here is the result for sha256-ni-mb with my code:
testing speed of multibuffer sha256 (sha256-ni-mb)
[ 73.212300] tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 1 operation in 284 cycles (16 bytes)
[ 73.212805] tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 1 operation in 311 cycles (64 bytes)
[ 73.213256] tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 1 operation in 481 cycles (256 bytes)
[ 73.213715] tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 1 operation in 1209 cycles (1024 bytes)
[ 73.214181] tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 1 operation in 36107 cycles (2048 bytes)
[ 73.214904] tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 1 operation in 4097 cycles (4096 bytes)
[ 73.215416] tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 1 operation in 7991 cycles (8192 bytes)
[ 86.522453] tcrypt:
testing speed of multibuffer sha256 (sha256-ni)
[ 86.522737] tcrypt: test 0 ( 16 byte blocks, 16 bytes per update, 1 updates): 1 operation in 224 cycles (16 bytes)
[ 86.523164] tcrypt: test 2 ( 64 byte blocks, 64 bytes per update, 1 updates): 1 operation in 296 cycles (64 bytes)
[ 86.523586] tcrypt: test 5 ( 256 byte blocks, 256 bytes per update, 1 updates): 1 operation in 531 cycles (256 bytes)
[ 86.524012] tcrypt: test 8 ( 1024 byte blocks, 1024 bytes per update, 1 updates): 1 operation in 1466 cycles (1024 bytes)
[ 86.524602] tcrypt: test 12 ( 2048 byte blocks, 2048 bytes per update, 1 updates): 1 operation in 2911 cycles (2048 bytes)
[ 86.525199] tcrypt: test 16 ( 4096 byte blocks, 4096 bytes per update, 1 updates): 1 operation in 5566 cycles (4096 bytes)
[ 86.525806] tcrypt: test 21 ( 8192 byte blocks, 8192 bytes per update, 1 updates): 1 operation in 10901 cycles (8192 bytes)So about a 36% jump in throughput at 4K on my aging Intel CPU.
Cheers,
arch/x86/crypto/sha256_ni_asm.S +210 -0
@@ -174,6 +174,216 @@ SYM_TYPED_FUNC_START(sha256_ni_transform) RET SYM_FUNC_END(sha256_ni_transform) +#undef DIGEST_PTR +#undef DATA_PTR +#undef NUM_BLKS +#undef SHA256CONSTANTS +#undef MSG +#undef STATE0 +#undef STATE1 +#undef MSG0 +#undef MSG1 +#undef MSG2 +#undef MSG3 +#undef TMP +#undef SHUF_MASK +#undef ABEF_SAVE +#undef CDGH_SAVE + +// parameters for sha256_ni_x2() +#define MBCTX %rdi +#define BLOCKS %rsi +#define DATA1 %rdx +#define DATA2 %rcx + +// other scalar variables +#define SHA256CONSTANTS %rax +#define COUNT %r10 +#define COUNT32 %r10d + +// rbx is used as a temporary. + +#define MSG %xmm0 // sha256rnds2 implicit operand +#define STATE0_A %xmm1 +#define STATE1_A %xmm2 +#define STATE0_B %xmm3 +#define STATE1_B %xmm4 +#define TMP_A %xmm5 +#define TMP_B %xmm6 +#define MSG0_A %xmm7 +#define MSG1_A %xmm8 +#define MSG2_A %xmm9 +#define MSG3_A %xmm10 +#define MSG0_B %xmm11 +#define MSG1_B %xmm12 +#define MSG2_B %xmm13 +#define MSG3_B %xmm14 +#define SHUF_MASK %xmm15 + +#define OFFSETOF_STATEA 0 // offsetof(struct sha256_x2_mbctx, state[0]) +#define OFFSETOF_STATEB 32 // offsetof(struct sha256_x2_mbctx, state[1]) +#define OFFSETOF_INPUT0 64 // offsetof(struct sha256_x2_mbctx, input[0]) +#define OFFSETOF_INPUT1 72 // offsetof(struct sha256_x2_mbctx, input[1]) + +// Do 4 rounds of SHA-256 for each of two messages (interleaved). m0_a and m0_b +// contain the current 4 message schedule words for the first and second message +// respectively. +// +// If not all the message schedule words have been computed yet, then this also +// computes 4 more message schedule words for each message. m1_a-m3_a contain +// the next 3 groups of 4 message schedule words for the first message, and +// likewise m1_b-m3_b for the second. After consuming the current value of +// m0_a, this macro computes the group after m3_a and writes it to m0_a, and +// likewise for *_b. This means that the next (m0_a, m1_a, m2_a, m3_a) is the +// current (m1_a, m2_a, m3_a, m0_a), and likewise for *_b, so the caller must +// cycle through the registers accordingly. +.macro do_4rounds_2x i, m0_a, m1_a, m2_a, m3_a, m0_b, m1_b, m2_b, m3_b + movdqa (\i-32)*4(SHA256CONSTANTS), TMP_A + movdqa TMP_A, TMP_B + paddd \m0_a, TMP_A + paddd \m0_b, TMP_B +.if \i < 48 + sha256msg1 \m1_a, \m0_a + sha256msg1 \m1_b, \m0_b +.endif + movdqa TMP_A, MSG + sha256rnds2 STATE0_A, STATE1_A + movdqa TMP_B, MSG + sha256rnds2 STATE0_B, STATE1_B + pshufd $0x0E, TMP_A, MSG + sha256rnds2 STATE1_A, STATE0_A + pshufd $0x0E, TMP_B, MSG + sha256rnds2 STATE1_B, STATE0_B +.if \i < 48 + movdqa \m3_a, TMP_A + movdqa \m3_b, TMP_B + palignr $4, \m2_a, TMP_A + palignr $4, \m2_b, TMP_B + paddd TMP_A, \m0_a + paddd TMP_B, \m0_b + sha256msg2 \m3_a, \m0_a + sha256msg2 \m3_b, \m0_b +.endif +.endm + +// +// void sha256_ni_x2(struct sha256_x2_mbctx *mbctx, int blocks) +// +// This function computes the SHA-256 digests of two messages that are +// both |blocks| blocks long, starting from the individual initial states +// in |mbctx|. +// +// The instructions for the two SHA-256 operations are interleaved. On many +// CPUs, this is almost twice as fast as hashing each message individually due +// to taking better advantage of the CPU's SHA-256 and SIMD throughput. +// +SYM_FUNC_START(sha256_ni_x2) + // Allocate 64 bytes of stack space, 16-byte aligned. + push %rbx + push %rbp + mov %rsp, %rbp + sub $64, %rsp + and $~15, %rsp + + // Load the shuffle mask for swapping the endianness of 32-bit words. + movdqa PSHUFFLE_BYTE_FLIP_MASK(%rip), SHUF_MASK + + // Set up pointer to the round constants. + lea K256+32*4(%rip), SHA256CONSTANTS + + // Load the initial state from sctx->state. + movdqu OFFSETOF_STATEA+0*16(MBCTX), STATE0_A // DCBA + movdqu OFFSETOF_STATEA+1*16(MBCTX), STATE1_A // HGFE + movdqu OFFSETOF_STATEB+0*16(MBCTX), STATE0_B // DCBA + movdqu OFFSETOF_STATEB+1*16(MBCTX), STATE1_B // HGFE + + movdqa STATE0_A, TMP_A + movdqa STATE0_B, TMP_B + punpcklqdq STATE1_A, STATE0_A // FEBA + punpcklqdq STATE1_B, STATE0_B // FEBA + punpckhqdq TMP_A, STATE1_A // DCHG + punpckhqdq TMP_B, STATE1_B // DCHG + pshufd $0x1B, STATE0_A, STATE0_A // ABEF + pshufd $0x1B, STATE0_B, STATE0_B // ABEF + pshufd $0xB1, STATE1_A, STATE1_A // CDGH + pshufd $0xB1, STATE1_B, STATE1_B // CDGH + + mov OFFSETOF_INPUT0+0(MBCTX),DATA1 + mov OFFSETOF_INPUT1+0(MBCTX),DATA2 + +.Lfinup2x_loop: + // Load the next two data blocks. + movdqu 0*16(DATA1), MSG0_A + movdqu 0*16(DATA2), MSG0_B + movdqu 1*16(DATA1), MSG1_A + movdqu 1*16(DATA2), MSG1_B + movdqu 2*16(DATA1), MSG2_A + movdqu 2*16(DATA2), MSG2_B + movdqu 3*16(DATA1), MSG3_A + movdqu 3*16(DATA2), MSG3_B + add $64, DATA1 + add $64, DATA2 + + // Convert the words of the data blocks from big endian. + pshufb SHUF_MASK, MSG0_A + pshufb SHUF_MASK, MSG0_B + pshufb SHUF_MASK, MSG1_A + pshufb SHUF_MASK, MSG1_B + pshufb SHUF_MASK, MSG2_A + pshufb SHUF_MASK, MSG2_B + pshufb SHUF_MASK, MSG3_A + pshufb SHUF_MASK, MSG3_B + + // Save the original state for each block. + movdqa STATE0_A, 0*16(%rsp) + movdqa STATE0_B, 1*16(%rsp) + movdqa STATE1_A, 2*16(%rsp) + movdqa STATE1_B, 3*16(%rsp) + + // Do the SHA-256 rounds on each block. +.irp i, 0, 16, 32, 48 + do_4rounds_2x (\i + 0), MSG0_A, MSG1_A, MSG2_A, MSG3_A, \ + MSG0_B, MSG1_B, MSG2_B, MSG3_B + do_4rounds_2x (\i + 4), MSG1_A, MSG2_A, MSG3_A, MSG0_A, \ + MSG1_B, MSG2_B, MSG3_B, MSG0_B + do_4rounds_2x (\i + 8), MSG2_A, MSG3_A, MSG0_A, MSG1_A, \ + MSG2_B, MSG3_B, MSG0_B, MSG1_B + do_4rounds_2x (\i + 12), MSG3_A, MSG0_A, MSG1_A, MSG2_A, \ + MSG3_B, MSG0_B, MSG1_B, MSG2_B +.endr + + // Add the original state for each block. + paddd 0*16(%rsp), STATE0_A + paddd 1*16(%rsp), STATE0_B + paddd 2*16(%rsp), STATE1_A + paddd 3*16(%rsp), STATE1_B + + // Update BLOCKS and loop back if more blocks remain. + sub $1, BLOCKS + jne .Lfinup2x_loop + + // Write the two digests with all bytes in the correct order. + movdqa STATE0_A, TMP_A + movdqa STATE0_B, TMP_B + punpcklqdq STATE1_A, STATE0_A // GHEF + punpcklqdq STATE1_B, STATE0_B + punpckhqdq TMP_A, STATE1_A // ABCD + punpckhqdq TMP_B, STATE1_B + pshufd $0xB1, STATE0_A, STATE0_A // HGFE + pshufd $0xB1, STATE0_B, STATE0_B + pshufd $0x1B, STATE1_A, STATE1_A // DCBA + pshufd $0x1B, STATE1_B, STATE1_B + movdqu STATE0_A, OFFSETOF_STATEA+1*16(MBCTX) + movdqu STATE0_B, OFFSETOF_STATEB+1*16(MBCTX) + movdqu STATE1_A, OFFSETOF_STATEA+0*16(MBCTX) + movdqu STATE1_B, OFFSETOF_STATEB+0*16(MBCTX) + + mov %rbp, %rsp + pop %rbp + pop %rbx + RET +SYM_FUNC_END(sha256_ni_x2) + .section .rodata.cst256.K256, "aM", @progbits, 256 .align 64 K256:
arch/x86/crypto/sha256_ssse3_glue.c +310 -88
459 lines not on this page · raw message
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Eric Biggers
maintainerQuoted 11 lines · Herbert Xu
Quoted 5 lines · Eric Biggers
This new version hasn't fundamentally changed anything. It's still a much worse, unnecessarily complex and still incomplete implementation compared to my patchset which has been ready to go for nearly a year already. Please refer to all the previous feedback that I've given.
FWIW, my interface is a lot simpler than yours to implement, since it doesn't deal with the partial buffer non-sense in assembly. In fact that was a big mistake with the original API, the partial data handling should've been moved to the API layer a long time ago.
We've already discussed this. It is part of the multibuffer optimization, as instruction interleaving is applicable to partial block handling and finalization too. It also makes those parts able to use SIMD. Both of those improve performance. But more importantly it eliminates the need for a separate descriptor for each message. That results in massive simplifications up the stack. The per-algorithm C glue code is much simpler, the API is much simpler and has fewer edge cases, and with only one descriptor being needed it becomes feasible to allocate it on the stack. Overall it's just much more streamlined.
For all these reasons my version ends up with a much smaller diffstat, despite the assembly code being longer and more optimized.
I see that you didn't even bother to include any tests for all the edge cases in your API where descriptors and/or scatterlists aren't synced up. As I've explained before, these cases would be a huge pain to get right.
But of course, there is no need to go there in the first place. Cryptographic APIs should be simple and not include unnecessary edge cases. It seems you still have a misconception that your more complex API would make my work useful for IPsec, but again that is still incorrect, as I've explained many times. The latest bogus claims that you've been making, like that GHASH is not parallelizable, don't exactly inspire confidence either.
- Eric
Herbert Xu
maintainerQuoted 7 lines · Eric Biggers
But of course, there is no need to go there in the first place. Cryptographic APIs should be simple and not include unnecessary edge cases. It seems you still have a misconception that your more complex API would make my work useful for IPsec, but again that is still incorrect, as I've explained many times. The latest bogus claims that you've been making, like that GHASH is not parallelizable, don't exactly inspire confidence either.
Sure, everyone hates complexity. But you're not removing it. You're simply pushing the complexity into the algorithm implementation and more importantly, the user. With your interface the user has to jump through unnecessary hoops to get multiple requests going, which is probably why you limited it to just 2.
If anything we should be pushing the complexity into the API code itself and away from the algorithm implementation. Why? Because it's shared and therefore the test coverage works much better.
Look over the years at how many buggy edge cases such as block left-overs we have had in arch crypto code. Now if those edge cases were moved into shared API code it would be much better. Sure it could still be buggy, but it would affect everyone equally and that means it's much easier to catch.
It's much easier for the fuzz tests to catch a bug in shared API code than individual assembly code.
Here is an example the new hash interface looks like when used in fsverity. It allows unlimited chaining, without holding all those unnecessary kmap's:
commit 0a0be692829c3e69a14b7b10ed412250da458825
Author: Herbert Xu <[email protected]> Date: Tue Feb 18 17:46:20 2025 +0800
fsverity: restore ahash support and remove chaining limit
Use the hash interface instead of shash. This allows the chaining limit to be removed as the request no longer has to be allocated on the stack.
Memory allocations can always fail, but they *rarely* do. Resolve the OOM case by using a stack request as a fallback.
fs/verity/fsverity_private.h +1 -1
@@ -20,7 +20,7 @@/* A hash algorithm supported by fs-verity */ struct fsverity_hash_alg { -struct crypto_sync_hash *tfm;/* hash tfm, allocated on demand */ +struct crypto_hash *tfm;/* hash tfm, allocated on demand */ const char *name;/* crypto API name, e.g. sha256 */ unsigned int digest_size;/* digest size in bytes, e.g. 32 for SHA-256 */ unsigned int block_size;/* block size in bytes, e.g. 64 for SHA-256 */
fs/verity/hash_algs.c +10 -10
@@ -43,7 +43,7 @@ const struct fsverity_hash_alg *fsverity_get_hash_alg(const struct inode *inode, unsigned int num) {struct fsverity_hash_alg *alg; -struct crypto_sync_hash *tfm; +struct crypto_hash *tfm;int err; if (num >= ARRAY_SIZE(fsverity_hash_algs) ||
@@ -62,7 +62,7 @@ const struct fsverity_hash_alg *fsverity_get_hash_alg(const struct inode *inode,if (alg->tfm !=NULL ) goto out_unlock; - tfm =crypto_alloc_sync_hash (alg->name,0 ,0 ); + tfm =crypto_alloc_hash (alg->name,0 ,0 );if (IS_ERR (tfm)) {if (PTR_ERR (tfm) == -ENOENT ) { fsverity_warn(inode,
@@ -79,20 +79,20 @@ const struct fsverity_hash_alg *fsverity_get_hash_alg(const struct inode *inode, } err = -EINVAL ; -if (WARN_ON_ONCE (alg->digest_size !=crypto_sync_hash_digestsize (tfm))) +if (WARN_ON_ONCE (alg->digest_size !=crypto_hash_digestsize (tfm))) goto err_free_tfm; -if (WARN_ON_ONCE (alg->block_size !=crypto_sync_hash_blocksize (tfm))) +if (WARN_ON_ONCE (alg->block_size !=crypto_hash_blocksize (tfm))) goto err_free_tfm;pr_info ("%s using implementation \"%s\"\n" , - alg->name,crypto_sync_hash_driver_name (tfm)); + alg->name,crypto_hash_driver_name (tfm));/* pairs with smp_load_acquire() above */ smp_store_release (&alg->tfm, tfm); goto out_unlock; err_free_tfm: -crypto_free_sync_hash (tfm); +crypto_free_hash (tfm); alg =ERR_PTR (err); out_unlock:mutex_unlock (&fsverity_hash_alg_init_mutex);
@@ -112,7 +112,7 @@ const u8 *fsverity_prepare_hash_state(const struct fsverity_hash_alg *alg, const u8 *salt, size_t salt_size) {u8 *hashstate =NULL ; -SYNC_HASH_REQUEST_ON_STACK (req, alg->tfm); +HASH_REQUEST_ON_STACK (req, alg->tfm);u8 *padded_salt =NULL ;size_t padded_salt_size;int err;
@@ -120,7 +120,7 @@ const u8 *fsverity_prepare_hash_state(const struct fsverity_hash_alg *alg,if (salt_size ==0 )return NULL ; - hashstate =kmalloc (crypto_sync_hash_statesize (alg->tfm),GFP_KERNEL ); + hashstate =kmalloc (crypto_hash_statesize (alg->tfm),GFP_KERNEL );if (!hashstate)return ERR_PTR (-ENOMEM );
@@ -178,7 +178,7 @@ const u8 *fsverity_prepare_hash_state(const struct fsverity_hash_alg *alg,int fsverity_hash_block (const struct merkle_tree_params *params,const struct inode *inode,const void *data,u8 *out) { -SYNC_HASH_REQUEST_ON_STACK (req, params->hash_alg->tfm); +HASH_REQUEST_ON_STACK (req, params->hash_alg->tfm);int err;ahash_request_set_callback (req,CRYPTO_TFM_REQ_MAY_SLEEP ,NULL ,NULL );
@@ -212,7 +212,7 @@ int fsverity_hash_block(const struct merkle_tree_params *params,int fsverity_hash_buffer (const struct fsverity_hash_alg *alg,const void *data,size_t size,u8 *out) { -return crypto_sync_hash_digest (alg->tfm, data, size, out); +return crypto_hash_digest (alg->tfm, data, size, out); }void __init fsverity_check_hash_algs (void )
fs/verity/verify.c +85 -64
228 lines not on this page · raw message
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Eric Biggers
maintainerQuoted 10 lines · Herbert Xu
Quoted 7 lines · Eric Biggers
But of course, there is no need to go there in the first place. Cryptographic APIs should be simple and not include unnecessary edge cases. It seems you still have a misconception that your more complex API would make my work useful for IPsec, but again that is still incorrect, as I've explained many times. The latest bogus claims that you've been making, like that GHASH is not parallelizable, don't exactly inspire confidence either.
Sure, everyone hates complexity. But you're not removing it.
I'm avoiding adding it in the first place.
Quoted 9 lines
You're simply pushing the complexity into the algorithm implementation and more importantly, the user. With your interface the user has to jump through unnecessary hoops to get multiple requests going, which is probably why you limited it to just 2.
If anything we should be pushing the complexity into the API code itself and away from the algorithm implementation. Why? Because it's shared and therefore the test coverage works much better.
Quoted 5 lines
Look over the years at how many buggy edge cases such as block left-overs we have had in arch crypto code. Now if those edge cases were moved into shared API code it would be much better. Sure it could still be buggy, but it would affect everyone equally and that means it's much easier to catch.
You cannot ignore complexity in the API, as that is the worst kind.
In addition, your (slower) solution has a large amount of complexity in the per-algorithm glue code, making it still more lines of code *per algorithm* than my (faster) solution, which you're ignoring.
Also, users still have to queue up multiple requests anyway. There are no "unnecessary hoops" with my patches -- just a faster, simpler, easier to use and less error-prone API.
Quoted 2 lines
Memory allocations can always fail, but they *rarely* do. Resolve the OOM case by using a stack request as a fallback.
Rarely executed fallbacks that are only executed in extremely rare OOM situations that won't be covered by xfstests? No thank you. Why would you even think that would be reasonable?
Anyway, I am getting tired of responding to all your weird arguments that don't bring anything new to the table. Please continue to treat your patches as nacked and don't treat silence as agreement. I am just tired of this.
- Eric
Herbert Xu
maintainerQuoted 4 lines · Eric Biggers
In addition, your (slower) solution has a large amount of complexity in the per-algorithm glue code, making it still more lines of code *per algorithm* than my (faster) solution, which you're ignoring.
My patches make the per-alg glue code look bad, but the intention is to share them not just between sha256 implementatinos, but across all ahash implementations as a whole. They will become the new hash walking interface.
Quoted 3 lines
Anyway, I am getting tired of responding to all your weird arguments that don't bring anything new to the table. Please continue to treat your patches as nacked and don't treat silence as agreement. I am just tired of this.
Talk about weird arguments, here's something even weirder to ponder over:
Rather than passing a single block, let's pass the whole bio all at once. Then depending on how much memory is available to store the hash result, we either return the whole thing, or hash as much as we can fit into a page and then iterate (just a single hash in the worst case (OOM)).
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt