crypto: qce driver fixes for gcm
Subsystems (2)
cover crypto: qce driver fixes for gcm
I've finally managed to get gcm(aes) working with the qce crypto engine.
These first patch fixes a bug where the gcm authentication tag was being overwritten during gcm decryption, because it was passed in the same sgl buffer as the crypto payload. The qce driver appends some private state buffer to the request destination sgl, but it was not checking the length of the sgl being passed.
The second patch works around a problem, which I frankly can't pinpoint what exactly is the cause, but after some help from Ard Biesheuvel, I think it is related to DMA. When gcm sends a request in crypto_gcm_setkey, it stores the hash (the crypto payload) and the iv in the same data struct. When the drivers updates the IV, then the payload gets overwritten with the unencrypted data, or all zeroes, it may be a coincidence.
However, it works if I pass the request down to the fallback driver--it is used by the driver to accept 192-bit-key requests. All I had to do was setup the fallback regardless of key size, and then check the payload length along with the keysize to pass the request to the fallback. This turns out to enhance performance, because of the avoided latency that comes with using the hardware.
I've started with checking for a single 16-byte AES block, and that is enough to make gcm work. Next thing I've done was to tune the request size for performance. What got me started into looking at the qce driver was reports of it being detrimental to VPN speed, by the way. I've tested this win an Asus RT-AC58U, but the slow VPN reports[1] have more devices affected. Access to the device was kindly provided by @simsasss.
I've used the openssl speed util to measure the speed, with an AF_ALG engine I've written to make use of the kernel driver from userspace[2], running on 4.19.78--I can't run this on a newer kernel yet.
TLDR: In the worst (where the hardware is slowest) case, hardware and software speed match at aroung 768 bytes, but I lowered the threshold to 512 to benefit the CPU offload.
Here's the script I've used:
#!/bin/sh
for len in 256 512 768 1024; do
echo Block-size: ${len} bytes
for key in 128 256; do
for mode in cbc ctr ecb; do
rmmod qcrypto
openssl speed -elapsed -evp aes-${key}-${mode} -engine afalg \
-bytes ${len} 2>&1 \
| grep ^aes \
| sed "s/aes-${key}-${mode} /aes-${key}-${mode} soft/"
insmod /tmp/qcrypto.ko
openssl speed -elapsed -evp aes-${key}-${mode} -engine afalg \
-bytes ${len} 2>&1 \
| grep ^aes \
| sed "s/aes-${key}-${mode} /aes-${key}-${mode} qce /"
done done
done
Here's a sample run--numbers vary from run to run, sometimes greatly:
./test_speed.sh
Block-size: 256 bytes
aes-128-cbc soft 6808.92k aes-128-cbc qce 2704.10k aes-128-ctr soft 6785.63k aes-128-ctr qce 2675.07k aes-128-ecb soft 7596.86k aes-128-ecb qce 2772.16k aes-256-cbc soft 5970.02k aes-256-cbc qce 2678.84k aes-256-ctr soft 6164.46k aes-256-ctr qce 2634.15k aes-256-ecb soft 6529.03k aes-256-ecb qce 2720.88k Block-size: 512 bytes aes-128-cbc soft 9402.31k aes-128-cbc qce 5345.69k aes-128-ctr soft 9766.23k aes-128-ctr qce 5179.25k aes-128-ecb soft 10638.85k aes-128-ecb qce 5437.13k aes-256-cbc soft 7742.98k aes-256-cbc qce 5230.08k aes-256-ctr soft 8174.93k aes-256-ctr qce 5115.89k aes-256-ecb soft 8772.61k aes-256-ecb qce 7282.35k
Block-size: 768 bytes
aes-128-cbc soft 10466.38k
aes-128-cbc qce 7814.59k
aes-128-ctr soft 11161.69k
aes-128-ctr qce 7639.93k
aes-128-ecb soft 12122.37k
aes-128-ecb qce 10764.84k
aes-256-cbc soft 8725.50k aes-256-cbc qce 9184.41k aes-256-ctr soft 9233.15k aes-256-ctr qce 7392.32k
aes-256-ecb soft 10039.30k
aes-256-ecb qce 9148.45k
Block-size: 1024 bytes
aes-128-cbc soft 11418.80k
aes-128-cbc qce 12314.37k
aes-128-ctr soft 11940.86k
aes-128-ctr qce 11982.51k
aes-128-ecb soft 13350.23k
aes-128-ecb qce 10375.28k
aes-256-cbc soft 9003.32k
aes-256-cbc qce 12017.66k
aes-256-ctr soft 9898.89k
aes-256-ctr qce 9672.18k
aes-256-ecb soft 10679.74k
aes-256-ecb qce 12314.37k
I imagine that if I were to run the benchmark within the kernel, the resulting threshould would be eve higher, since there's a pretty much fixed latency from the context switches. Nonetheless, I think it's better to let the engine run more, to offload the CPU.
Cheers,
Eneas
[1] https://forum.openwrt.org/t/ipsec-performance-issue/39690
[2] https://github.com/cotequeiroz/afalg_engine
Eneas U de Queiroz (2):
crypto: qce - use cryptlen when adding extra sgl
crypto: qce - use AES fallback when len <= 512
drivers/crypto/qce/dma.c | 11 ++++++----- drivers/crypto/qce/dma.h | 2 +- drivers/crypto/qce/skcipher.c | 17 +++++++---------- 3 files changed, 14 insertions(+), 16 deletions(-)
PATCH 1/2 crypto: qce - use cryptlen when adding extra sgl
The qce crypto driver appends an extra entry to the dst sgl, to maintain private state information.
When the gcm driver sends requests to the ctr skcipher, it passes the authentication tag after the actual crypto payload, but it must not be touched.
Commit 1336c2221bee ("crypto: qce - save a sg table slot for result buf") limited the destination sgl to avoid overwriting the authentication tag but it assumed the tag would be in a separate sgl entry.
This is not always the case, so it is better to limit the length of the destination buffer to req->cryptlen before appending the result buf.
drivers/crypto/qce/dma.c +6 -5
@@ -48,9 +48,10 @@ void qce_dma_release(struct qce_dma_data *dma)struct scatterlist *qce_sgtable_add (struct sg_table *sgt,struct scatterlist *new_sgl, -int max_ents) +unsigned int max_len) {struct scatterlist *sg = sgt->sgl, *sg_last =NULL ; +unsigned int new_len;while (sg) {if (!sg_page (sg))
@@ -61,13 +62,13 @@ qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,if (!sg)return ERR_PTR (-EINVAL ); -while (new_sgl && sg && max_ents) { -sg_set_page (sg,sg_page (new_sgl), new_sgl->length, - new_sgl->offset); +while (new_sgl && sg && max_len) { + new_len = new_sgl->length > max_len ? max_len : new_sgl->length; +sg_set_page (sg,sg_page (new_sgl), new_len, new_sgl->offset); sg_last = sg; sg =sg_next (sg); new_sgl =sg_next (new_sgl); - max_ents--; + max_len -= new_len; }return sg_last;
drivers/crypto/qce/dma.h +1 -1
@@ -43,6 +43,6 @@ void qce_dma_issue_pending(struct qce_dma_data *dma);int qce_dma_terminate_all (struct qce_dma_data *dma);struct scatterlist *qce_sgtable_add (struct sg_table *sgt,struct scatterlist *sg_add, -int max_ents); +unsigned int max_len);#endif /* _DMA_H_ */
drivers/crypto/qce/skcipher.c +3 -2
@@ -97,13 +97,14 @@ qce_skcipher_async_req_handle(struct crypto_async_request *async_req)sg_init_one (&rctx->result_sg, qce->dma.result_buf,QCE_RESULT_BUF_SZ ); - sg =qce_sgtable_add (&rctx->dst_tbl, req->dst, rctx->dst_nents -1 ); + sg =qce_sgtable_add (&rctx->dst_tbl, req->dst, req->cryptlen);if (IS_ERR (sg)) { ret =PTR_ERR (sg); goto error_free; } - sg =qce_sgtable_add (&rctx->dst_tbl, &rctx->result_sg,1 ); + sg =qce_sgtable_add (&rctx->dst_tbl, &rctx->result_sg, +QCE_RESULT_BUF_SZ );if (IS_ERR (sg)) { ret =PTR_ERR (sg); goto error_free;
PATCH 2/2 crypto: qce - use AES fallback when len <= 512
In reply to Eneas U de QueirozProcess small blocks using the fallback cipher, as a workaround for an observed failure (DMA-related, apparently) when computing the GCM ghash key. This brings a speed gain as well, since it avoids the latency of using the hardware engine to process small blocks.
Using software for all 16-byte requests would be enough to make GCM work, but to increase performance, a larger threshold would be better. Measuring the performance of supported ciphers with openssl speed, software matches hardware at around 768-1024 bytes.
Considering the 256-bit ciphers, software is 2-3 times faster than qce at 256-bytes, 30% faster at 512, and about even at 768-bytes. With 128-bit keys, the break-even point would be around 1024-bytes.
The threshold is being set a little lower, to 512 bytes, to balance the cost in CPU usage.
drivers/crypto/qce/skcipher.c +4 -8
@@ -166,15 +166,10 @@ static int qce_skcipher_setkey(struct crypto_skcipher *ablk, const u8 *key,switch (IS_XTS (flags) ? keylen >>1 : keylen) {case AES_KEYSIZE_128 :case AES_KEYSIZE_256 : +memcpy (ctx->enc_key, key, keylen);break ; -default : - goto fallback; } - ctx->enc_keylen = keylen; -memcpy (ctx->enc_key, key, keylen); -return 0 ; -fallback: ret =crypto_sync_skcipher_setkey (ctx->fallback, key, keylen);if (!ret) ctx->enc_keylen = keylen;
@@ -224,8 +219,9 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt) rctx->flags |= encrypt ?QCE_ENCRYPT :QCE_DECRYPT ; keylen =IS_XTS (rctx->flags) ? ctx->enc_keylen >>1 : ctx->enc_keylen; -if (IS_AES (rctx->flags) && keylen !=AES_KEYSIZE_128 && - keylen !=AES_KEYSIZE_256 ) { +if (IS_AES (rctx->flags) && + ((keylen !=AES_KEYSIZE_128 && keylen !=AES_KEYSIZE_256 ) + || req->cryptlen <=512 )) {SYNC_SKCIPHER_REQUEST_ON_STACK (subreq, ctx->fallback);skcipher_request_set_sync_tfm (subreq, ctx->fallback);
v2 cover crypto: qce driver fixes for gcm
In reply to Eneas U de QueirozI finally managed to get the tcrypt module working to make some measurements straight from the kernel. The reason the module was not loading was that AES-XTS was hanging, so the tests never finished, and I couldn't get any messages on /proc/kmsg.
By trial and error, I concluded that xts-aes-qce does not take requests that are greater than 512-bytes, and not a multiple of 512. So, when I tried to run the tests with 768 bytes, it would just hang.
As a workaround we can use the fallback to fullfill those requests.
As part of the v2, I'm using a module paramenter to set the software threshold, instead of fixing it at 512.
The results of the tcrypt tests confirmed my previous estimates, so I'm leaving the default at 512 bytes.
Here's a sample run of my tests. Just like with openssl, numbers vary from run to run, more than I would expect.
testing speed of async cbc(aes) (cbc-aes-qce) encryptionaes_sw_max_len 32.768 512 0
------------------ ---------- ---------- ----------
128 bit 16 bytes 8.081.136 5.614.448 430.416
128 bit 64 bytes 13.152.768 13.205.952 1.745.088 128 bit 256 bytes 16.094.464 16.101.120 6.969.600 128 bit 512 bytes 16.701.440 16.705.024 12.866.048 128 bit 768 bytes 16.883.712 13.192.704 15.186.432 128 bit 1024 bytes 17.036.288 17.149.952 19.716.096 128 bit 2048 bytes 17.108.992 30.842.880 32.868.352 128 bit 4096 bytes 17.203.200 44.929.024 49.655.808 128 bit 8192 bytes 17.219.584 58.966.016 74.186.752 256 bit 16 bytes 6.962.432 1.943.616 419.088 256 bit 64 bytes 10.485.568 10.421.952 1.681.536 256 bit 256 bytes 12.211.712 12.160.000 6.701.312 256 bit 512 bytes 12.499.456 12.584.448 9.882.112 256 bit 768 bytes 12.622.080 12.550.656 14.701.824 256 bit 1024 bytes 12.750.848 16.079.872 19.585.024 256 bit 2048 bytes 12.812.288 28.293.120 27.693.056 256 bit 4096 bytes 12.939.264 34.234.368 44.142.592 256 bit 8192 bytes 12.845.056 50.274.304 63.520.768
Eneas U de Queiroz (3):
crypto: qce - use cryptlen when adding extra sgl
crypto: qce - use AES fallback for small requests
crypto: qce - handle AES-XTS cases that qce fails
drivers/crypto/Kconfig | 23 +++++++++++++++++++++++ drivers/crypto/qce/common.c | 2 -- drivers/crypto/qce/common.h | 3 +++ drivers/crypto/qce/dma.c | 11 ++++++----- drivers/crypto/qce/dma.h | 2 +- drivers/crypto/qce/skcipher.c | 28 ++++++++++++++++++---------- 6 files changed, 51 insertions(+), 18 deletions(-)
PATCH v2 1/3 crypto: qce - use cryptlen when adding extra sgl
The qce crypto driver appends an extra entry to the dst sgl, to maintain private state information.
When the gcm driver sends requests to the ctr skcipher, it passes the authentication tag after the actual crypto payload, but it must not be touched.
Commit 1336c2221bee ("crypto: qce - save a sg table slot for result buf") limited the destination sgl to avoid overwriting the authentication tag but it assumed the tag would be in a separate sgl entry.
This is not always the case, so it is better to limit the length of the destination buffer to req->cryptlen before appending the result buf.
drivers/crypto/qce/dma.c +6 -5
@@ -48,9 +48,10 @@ void qce_dma_release(struct qce_dma_data *dma)struct scatterlist *qce_sgtable_add (struct sg_table *sgt,struct scatterlist *new_sgl, -int max_ents) +unsigned int max_len) {struct scatterlist *sg = sgt->sgl, *sg_last =NULL ; +unsigned int new_len;while (sg) {if (!sg_page (sg))
@@ -61,13 +62,13 @@ qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,if (!sg)return ERR_PTR (-EINVAL ); -while (new_sgl && sg && max_ents) { -sg_set_page (sg,sg_page (new_sgl), new_sgl->length, - new_sgl->offset); +while (new_sgl && sg && max_len) { + new_len = new_sgl->length > max_len ? max_len : new_sgl->length; +sg_set_page (sg,sg_page (new_sgl), new_len, new_sgl->offset); sg_last = sg; sg =sg_next (sg); new_sgl =sg_next (new_sgl); - max_ents--; + max_len -= new_len; }return sg_last;
drivers/crypto/qce/dma.h +1 -1
@@ -43,6 +43,6 @@ void qce_dma_issue_pending(struct qce_dma_data *dma);int qce_dma_terminate_all (struct qce_dma_data *dma);struct scatterlist *qce_sgtable_add (struct sg_table *sgt,struct scatterlist *sg_add, -int max_ents); +unsigned int max_len);#endif /* _DMA_H_ */
drivers/crypto/qce/skcipher.c +3 -2
@@ -97,13 +97,14 @@ qce_skcipher_async_req_handle(struct crypto_async_request *async_req)sg_init_one (&rctx->result_sg, qce->dma.result_buf,QCE_RESULT_BUF_SZ ); - sg =qce_sgtable_add (&rctx->dst_tbl, req->dst, rctx->dst_nents -1 ); + sg =qce_sgtable_add (&rctx->dst_tbl, req->dst, req->cryptlen);if (IS_ERR (sg)) { ret =PTR_ERR (sg); goto error_free; } - sg =qce_sgtable_add (&rctx->dst_tbl, &rctx->result_sg,1 ); + sg =qce_sgtable_add (&rctx->dst_tbl, &rctx->result_sg, +QCE_RESULT_BUF_SZ );if (IS_ERR (sg)) { ret =PTR_ERR (sg); goto error_free;
PATCH v2 2/3 crypto: qce - use AES fallback for small requests
In reply to Eneas U de QueirozProcess small blocks using the fallback cipher, as a workaround for an observed failure (DMA-related, apparently) when computing the GCM ghash key. This brings a speed gain as well, since it avoids the latency of using the hardware engine to process small blocks.
Using software for all 16-byte requests would be enough to make GCM work, but to increase performance, a larger threshold would be better. Measuring the performance of supported ciphers with openssl speed, software matches hardware at around 768-1024 bytes.
Considering the 256-bit ciphers, software is 2-3 times faster than qce at 256-bytes, 30% faster at 512, and about even at 768-bytes. With 128-bit keys, the break-even point would be around 1024-bytes.
This adds the 'aes_sw_max_len' parameter, to set the largest request length processed by the software fallback. Its default is being set to 512 bytes, a little lower than the break-even point, to balance the cost in CPU usage.
crypto: qce - add aes_sw_max_len parameter
This adds the AES fallback threshold as a parameter, so it can be changed by the user.
drivers/crypto/Kconfig +23 -0
@@ -685,6 +685,29 @@ choice endchoice +config CRYPTO_DEV_QCE_SW_MAX_LEN + int "Default maximum request size to use software for AES" + depends on CRYPTO_DEV_QCE && CRYPTO_DEV_QCE_SKCIPHER + default 512 + help + This sets the default maximum request size to perform AES requests + using software instead of the crypto engine. It can be changed by + setting the aes_sw_max_len parameter. + + Small blocks are processed faster in software than hardware. + Considering the 256-bit ciphers, software is 2-3 times faster than + qce at 256-bytes, 30% faster at 512, and about even at 768-bytes. + With 128-bit keys, the break-even point would be around 1024-bytes. + + The default is set a little lower, to 512 bytes, to balance the + cost in CPU usage. The minimum recommended setting is 16-bytes + (1 AES block), since AES-GCM will fail if you set it lower. + Setting this to zero will send all requests to the hardware. + + Note that 192-bit keys are not supported by the hardware and are + always processed by the software fallback, and all DES requests + are done by the hardware. + config CRYPTO_DEV_QCOM_RNG tristate "Qualcomm Random Number Generator Driver" depends on ARCH_QCOM || COMPILE_TEST
drivers/crypto/qce/skcipher.c +12 -8
@@ -5,6 +5,7 @@#include <linux/device.h> #include <linux/interrupt.h> +#include <linux/moduleparam.h> #include <linux/types.h> #include <crypto/aes.h> #include <crypto/internal/des.h>
@@ -12,6 +13,13 @@#include "cipher.h" +unsigned int aes_sw_max_len =CONFIG_CRYPTO_DEV_QCE_SW_MAX_LEN ; +module_param (aes_sw_max_len, uint,0644 ); +MODULE_PARM_DESC (aes_sw_max_len, +"Only use hardware for AES requests larger than this " +"[0=always use hardware; anything <16 breaks AES-GCM; default=" + __stringify(CONFIG_CRYPTO_DEV_QCE_SOFT_THRESHOLD)"]"); +static LIST_HEAD (skcipher_algs);static void qce_skcipher_done (void *data)
@@ -166,15 +174,10 @@ static int qce_skcipher_setkey(struct crypto_skcipher *ablk, const u8 *key,switch (IS_XTS (flags) ? keylen >>1 : keylen) {case AES_KEYSIZE_128 :case AES_KEYSIZE_256 : +memcpy (ctx->enc_key, key, keylen);break ; -default : - goto fallback; } - ctx->enc_keylen = keylen; -memcpy (ctx->enc_key, key, keylen); -return 0 ; -fallback: ret =crypto_sync_skcipher_setkey (ctx->fallback, key, keylen);if (!ret) ctx->enc_keylen = keylen;
@@ -224,8 +227,9 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt) rctx->flags |= encrypt ?QCE_ENCRYPT :QCE_DECRYPT ; keylen =IS_XTS (rctx->flags) ? ctx->enc_keylen >>1 : ctx->enc_keylen; -if (IS_AES (rctx->flags) && keylen !=AES_KEYSIZE_128 && - keylen !=AES_KEYSIZE_256 ) { +if (IS_AES (rctx->flags) && + ((keylen !=AES_KEYSIZE_128 && keylen !=AES_KEYSIZE_256 ) + || req->cryptlen <= aes_sw_max_len)) {SYNC_SKCIPHER_REQUEST_ON_STACK (subreq, ctx->fallback);skcipher_request_set_sync_tfm (subreq, ctx->fallback);
PATCH v2 3/3 crypto: qce - handle AES-XTS cases that qce fails
In reply to Eneas U de QueirozQCE hangs when presented with an AES-XTS request whose length is larger than QCE_SECTOR_SIZE (512-bytes), and is not a multiple of it. Let the fallback cipher handle them.
drivers/crypto/qce/common.c +0 -2
@@ -15,8 +15,6 @@#include "regs-v5.h" #include "sha.h" -#define QCE_SECTOR_SIZE 512 -static inline u32 qce_read (struct qce_device *qce,u32 offset) {return readl (qce->base + offset);
drivers/crypto/qce/common.h +3 -0
@@ -12,6 +12,9 @@#include <crypto/hash.h> #include <crypto/internal/skcipher.h> +/* xts du size */ +#define QCE_SECTOR_SIZE 512 +/* key size in bytes */ #define QCE_SHA_HMAC_KEY_SIZE 64#define QCE_MAX_CIPHER_KEY_SIZE AES_KEYSIZE_256
drivers/crypto/qce/skcipher.c +5 -1
@@ -227,9 +227,13 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt) rctx->flags |= encrypt ?QCE_ENCRYPT :QCE_DECRYPT ; keylen =IS_XTS (rctx->flags) ? ctx->enc_keylen >>1 : ctx->enc_keylen; +/* qce is hanging when AES-XTS request len > QCE_SECTOR_SIZE and +* is not a multiple of it; pass such requests to the fallback */ if (IS_AES (rctx->flags) && ((keylen !=AES_KEYSIZE_128 && keylen !=AES_KEYSIZE_256 ) - || req->cryptlen <= aes_sw_max_len)) { + || req->cryptlen <= aes_sw_max_len) + || (IS_XTS (rctx->flags) && req->cryptlen >QCE_SECTOR_SIZE && + req->cryptlen %QCE_SECTOR_SIZE )) {SYNC_SKCIPHER_REQUEST_ON_STACK (subreq, ctx->fallback);skcipher_request_set_sync_tfm (subreq, ctx->fallback);
kbuild test robot
build botRe: [PATCH v2 3/3] crypto: qce - handle AES-XTS cases that qce fails
Thank you for the patch! Perhaps something to improve:
Hi Eneas,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on cryptodev/master]
[also build test WARNING on crypto/master next-20200206]
[cannot apply to sparc-next/master v5.5]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Eneas-U-de-Queiroz/crypto-qce-driver-fixes-for-gcm/20200207-051805 base: https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git master config: m68k-allmodconfig (attached as .config) compiler: m68k-linux-gcc (GCC) 7.5.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.5.0 make.cross ARCH=m68k
If you fix the issue, kindly add following tag
All warnings (new ones prefixed by >>):
drivers/crypto/qce/skcipher.c: In function 'qce_skcipher_crypt':
>> drivers/crypto/qce/skcipher.c:232:26: warning: suggest parentheses around '&&' within '||' [-Wparentheses]
if (IS_AES(rctx->flags) &&vim +232 drivers/crypto/qce/skcipher.c
5feaaae1b549f3 drivers/crypto/qce/ablkcipher.c Herbert Xu 2019-04-11 216
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 217 static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt)
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 218 {
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 219 struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 220 struct qce_cipher_ctx *ctx = crypto_skcipher_ctx(tfm);
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 221 struct qce_cipher_reqctx *rctx = skcipher_request_ctx(req);
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 222 struct qce_alg_template *tmpl = to_cipher_tmpl(tfm);
7de4c2bd196f11 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2019-12-20 223 int keylen;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 224 int ret;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 225
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 226 rctx->flags = tmpl->alg_flags;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 227 rctx->flags |= encrypt ? QCE_ENCRYPT : QCE_DECRYPT;
7de4c2bd196f11 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2019-12-20 228 keylen = IS_XTS(rctx->flags) ? ctx->enc_keylen >> 1 : ctx->enc_keylen;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 229
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 230 /* qce is hanging when AES-XTS request len > QCE_SECTOR_SIZE and
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 231 * is not a multiple of it; pass such requests to the fallback */
c23a1c2b41c486 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 @232 if (IS_AES(rctx->flags) &&
c23a1c2b41c486 drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 233 ((keylen != AES_KEYSIZE_128 && keylen != AES_KEYSIZE_256)
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 234 || req->cryptlen <= aes_sw_max_len)
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 235 || (IS_XTS(rctx->flags) && req->cryptlen > QCE_SECTOR_SIZE &&
f8b4400d05347c drivers/crypto/qce/skcipher.c Eneas U de Queiroz 2020-02-05 236 req->cryptlen % QCE_SECTOR_SIZE)) {
d1e4ba83b0286b drivers/crypto/qce/ablkcipher.c Kees Cook 2018-09-18 237 SYNC_SKCIPHER_REQUEST_ON_STACK(subreq, ctx->fallback);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 238
d1e4ba83b0286b drivers/crypto/qce/ablkcipher.c Kees Cook 2018-09-18 239 skcipher_request_set_sync_tfm(subreq, ctx->fallback);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 240 skcipher_request_set_callback(subreq, req->base.flags,
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 241 NULL, NULL);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 242 skcipher_request_set_crypt(subreq, req->src, req->dst,
8bf0871539faa0 drivers/crypto/qce/skcipher.c Ard Biesheuvel 2019-11-09 243 req->cryptlen, req->iv);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 244 ret = encrypt ? crypto_skcipher_encrypt(subreq) :
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 245 crypto_skcipher_decrypt(subreq);
2d20ce070d3b78 drivers/crypto/qce/ablkcipher.c Herbert Xu 2016-06-29 246 skcipher_request_zero(subreq);
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 247 return ret;
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 248 }
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 249
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 250 return tmpl->qce->async_req_enqueue(tmpl->qce, &req->base);
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 251 }
ec8f5d8f6f76b9 drivers/crypto/qce/ablkcipher.c Stanimir Varbanov 2014-06-25 252 :::::: The code at line 232 was first introduced by commit
:::::: c23a1c2b41c486bff4ad5cf8b0968e3f55907eba crypto: qce - use AES fallback for small requests
:::::: TO: Eneas U de Queiroz <[email protected]> :::::: CC: 0day robot <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation
PATCH v3 1/3 crypto: qce - use cryptlen when adding extra sgl
In reply to Eneas U de QueirozThe qce crypto driver appends an extra entry to the dst sgl, to maintain private state information.
When the gcm driver sends requests to the ctr skcipher, it passes the authentication tag after the actual crypto payload, but it must not be touched.
Commit 1336c2221bee ("crypto: qce - save a sg table slot for result buf") limited the destination sgl to avoid overwriting the authentication tag but it assumed the tag would be in a separate sgl entry.
This is not always the case, so it is better to limit the length of the destination buffer to req->cryptlen before appending the result buf.
drivers/crypto/qce/dma.c +6 -5
@@ -48,9 +48,10 @@ void qce_dma_release(struct qce_dma_data *dma)struct scatterlist *qce_sgtable_add (struct sg_table *sgt,struct scatterlist *new_sgl, -int max_ents) +unsigned int max_len) {struct scatterlist *sg = sgt->sgl, *sg_last =NULL ; +unsigned int new_len;while (sg) {if (!sg_page (sg))
@@ -61,13 +62,13 @@ qce_sgtable_add(struct sg_table *sgt, struct scatterlist *new_sgl,if (!sg)return ERR_PTR (-EINVAL ); -while (new_sgl && sg && max_ents) { -sg_set_page (sg,sg_page (new_sgl), new_sgl->length, - new_sgl->offset); +while (new_sgl && sg && max_len) { + new_len = new_sgl->length > max_len ? max_len : new_sgl->length; +sg_set_page (sg,sg_page (new_sgl), new_len, new_sgl->offset); sg_last = sg; sg =sg_next (sg); new_sgl =sg_next (new_sgl); - max_ents--; + max_len -= new_len; }return sg_last;
drivers/crypto/qce/dma.h +1 -1
@@ -43,6 +43,6 @@ void qce_dma_issue_pending(struct qce_dma_data *dma);int qce_dma_terminate_all (struct qce_dma_data *dma);struct scatterlist *qce_sgtable_add (struct sg_table *sgt,struct scatterlist *sg_add, -int max_ents); +unsigned int max_len);#endif /* _DMA_H_ */
drivers/crypto/qce/skcipher.c +3 -2
@@ -97,13 +97,14 @@ qce_skcipher_async_req_handle(struct crypto_async_request *async_req)sg_init_one (&rctx->result_sg, qce->dma.result_buf,QCE_RESULT_BUF_SZ ); - sg =qce_sgtable_add (&rctx->dst_tbl, req->dst, rctx->dst_nents -1 ); + sg =qce_sgtable_add (&rctx->dst_tbl, req->dst, req->cryptlen);if (IS_ERR (sg)) { ret =PTR_ERR (sg); goto error_free; } - sg =qce_sgtable_add (&rctx->dst_tbl, &rctx->result_sg,1 ); + sg =qce_sgtable_add (&rctx->dst_tbl, &rctx->result_sg, +QCE_RESULT_BUF_SZ );if (IS_ERR (sg)) { ret =PTR_ERR (sg); goto error_free;
PATCH v3 2/3 crypto: qce - use AES fallback for small requests
In reply to Eneas U de QueirozProcess small blocks using the fallback cipher, as a workaround for an observed failure (DMA-related, apparently) when computing the GCM ghash key. This brings a speed gain as well, since it avoids the latency of using the hardware engine to process small blocks.
Using software for all 16-byte requests would be enough to make GCM work, but to increase performance, a larger threshold would be better. Measuring the performance of supported ciphers with openssl speed, software matches hardware at around 768-1024 bytes.
Considering the 256-bit ciphers, software is 2-3 times faster than qce at 256-bytes, 30% faster at 512, and about even at 768-bytes. With 128-bit keys, the break-even point would be around 1024-bytes.
This adds the 'aes_sw_max_len' parameter, to set the largest request length processed by the software fallback. Its default is being set to 512 bytes, a little lower than the break-even point, to balance the cost in CPU usage.
drivers/crypto/Kconfig +23 -0
@@ -685,6 +685,29 @@ choice endchoice +config CRYPTO_DEV_QCE_SW_MAX_LEN + int "Default maximum request size to use software for AES" + depends on CRYPTO_DEV_QCE && CRYPTO_DEV_QCE_SKCIPHER + default 512 + help + This sets the default maximum request size to perform AES requests + using software instead of the crypto engine. It can be changed by + setting the aes_sw_max_len parameter. + + Small blocks are processed faster in software than hardware. + Considering the 256-bit ciphers, software is 2-3 times faster than + qce at 256-bytes, 30% faster at 512, and about even at 768-bytes. + With 128-bit keys, the break-even point would be around 1024-bytes. + + The default is set a little lower, to 512 bytes, to balance the + cost in CPU usage. The minimum recommended setting is 16-bytes + (1 AES block), since AES-GCM will fail if you set it lower. + Setting this to zero will send all requests to the hardware. + + Note that 192-bit keys are not supported by the hardware and are + always processed by the software fallback, and all DES requests + are done by the hardware. + config CRYPTO_DEV_QCOM_RNG tristate "Qualcomm Random Number Generator Driver" depends on ARCH_QCOM || COMPILE_TEST
drivers/crypto/qce/skcipher.c +12 -8
@@ -5,6 +5,7 @@#include <linux/device.h> #include <linux/interrupt.h> +#include <linux/moduleparam.h> #include <linux/types.h> #include <crypto/aes.h> #include <crypto/internal/des.h>
@@ -12,6 +13,13 @@#include "cipher.h" +unsigned int aes_sw_max_len =CONFIG_CRYPTO_DEV_QCE_SW_MAX_LEN ; +module_param (aes_sw_max_len, uint,0644 ); +MODULE_PARM_DESC (aes_sw_max_len, +"Only use hardware for AES requests larger than this " +"[0=always use hardware; anything <16 breaks AES-GCM; default=" + __stringify(CONFIG_CRYPTO_DEV_QCE_SOFT_THRESHOLD) "]"); +static LIST_HEAD (skcipher_algs);static void qce_skcipher_done (void *data)
@@ -166,15 +174,10 @@ static int qce_skcipher_setkey(struct crypto_skcipher *ablk, const u8 *key,switch (IS_XTS (flags) ? keylen >>1 : keylen) {case AES_KEYSIZE_128 :case AES_KEYSIZE_256 : +memcpy (ctx->enc_key, key, keylen);break ; -default : - goto fallback; } - ctx->enc_keylen = keylen; -memcpy (ctx->enc_key, key, keylen); -return 0 ; -fallback: ret =crypto_sync_skcipher_setkey (ctx->fallback, key, keylen);if (!ret) ctx->enc_keylen = keylen;
@@ -224,8 +227,9 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt) rctx->flags |= encrypt ?QCE_ENCRYPT :QCE_DECRYPT ; keylen =IS_XTS (rctx->flags) ? ctx->enc_keylen >>1 : ctx->enc_keylen; -if (IS_AES (rctx->flags) && keylen !=AES_KEYSIZE_128 && - keylen !=AES_KEYSIZE_256 ) { +if (IS_AES (rctx->flags) && + ((keylen !=AES_KEYSIZE_128 && keylen !=AES_KEYSIZE_256 ) + || req->cryptlen <= aes_sw_max_len)) {SYNC_SKCIPHER_REQUEST_ON_STACK (subreq, ctx->fallback);skcipher_request_set_sync_tfm (subreq, ctx->fallback);
PATCH v3 3/3 crypto: qce - handle AES-XTS cases that qce fails
In reply to Eneas U de QueirozQCE hangs when presented with an AES-XTS request whose length is larger than QCE_SECTOR_SIZE (512-bytes), and is not a multiple of it. Let the fallback cipher handle them.
drivers/crypto/qce/common.c +0 -2
@@ -15,8 +15,6 @@#include "regs-v5.h" #include "sha.h" -#define QCE_SECTOR_SIZE 512 -static inline u32 qce_read (struct qce_device *qce,u32 offset) {return readl (qce->base + offset);
drivers/crypto/qce/common.h +3 -0
@@ -12,6 +12,9 @@#include <crypto/hash.h> #include <crypto/internal/skcipher.h> +/* xts du size */ +#define QCE_SECTOR_SIZE 512 +/* key size in bytes */ #define QCE_SHA_HMAC_KEY_SIZE 64#define QCE_MAX_CIPHER_KEY_SIZE AES_KEYSIZE_256
drivers/crypto/qce/skcipher.c +6 -1
@@ -227,9 +227,14 @@ static int qce_skcipher_crypt(struct skcipher_request *req, int encrypt) rctx->flags |= encrypt ?QCE_ENCRYPT :QCE_DECRYPT ; keylen =IS_XTS (rctx->flags) ? ctx->enc_keylen >>1 : ctx->enc_keylen; +/* qce is hanging when AES-XTS request len > QCE_SECTOR_SIZE and +* is not a multiple of it; pass such requests to the fallback +*/ if (IS_AES (rctx->flags) && ((keylen !=AES_KEYSIZE_128 && keylen !=AES_KEYSIZE_256 ) - || req->cryptlen <= aes_sw_max_len)) { + || req->cryptlen <= aes_sw_max_len) + || (IS_XTS (rctx->flags) && req->cryptlen >QCE_SECTOR_SIZE && + req->cryptlen %QCE_SECTOR_SIZE )) {SYNC_SKCIPHER_REQUEST_ON_STACK (subreq, ctx->fallback);skcipher_request_set_sync_tfm (subreq, ctx->fallback);