AF_ALG AIO and IV
Subsystems (2)
Hi,
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in the AF_ALG context that is unique to one particular AF_ALG socket. Note the
analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the AF_ALG socket context is registered with the new request and the cipher operation is invoked.
You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
This needs to be solved somehow. I see the following options which I would like to have vetted by the community.
1. Require that the cipher implementations serialize any AIO requests that have dependencies. I.e. for CBC, requests need to be serialized by the driver. For, say, ECB or XTS no serialization is necessary.
2. Change AF_ALG to require a per-request IV. This could be implemented by moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg code path would obtain the IV.
I would tend to favor option 2 as this requires code change at only location. If option 2 is considered, I would recommend to still allow setting the IV via sendmsg CMSG (to keep the interface stable). If, however, the caller provides an IV via recvmsg, this takes precedence.
If there are other options, please allow us to learn about them.
Ciao
Stephan
PATCH 1/1 crypto: AF_ALG - inline IV support
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in the AF_ALG context that is unique to one particular AF_ALG socket. Note the analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the AF_ALG socket context is registered with the new request and the cipher operation is invoked.
You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the AF_ALG interface treats the ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.
If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that ctx->used is updated. Thus, the ctx->used access in _aead_recvmsg must be moved below the IV gathering.
The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
--- crypto/af_alg.c | 79 ++++++++++++++++++++++++++++++++++++++++++++- crypto/algif_aead.c | 54 ++++++++++++++----------------- crypto/algif_skcipher.c | 14 +++++--- include/crypto/if_alg.h | 17 ++++++++++ include/uapi/linux/if_alg.h | 6 ++-- 5 files changed, 133 insertions(+), 37 deletions(-)
crypto/af_alg.c +78 -1
@@ -14,6 +14,7 @@#include <linux/atomic.h> #include <crypto/if_alg.h> +#include <crypto/scatterwalk.h> #include <linux/crypto.h> #include <linux/init.h> #include <linux/kernel.h>
@@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,struct af_alg_control con = {};long copied =0 ;bool enc =0 ; +bool iiv =0 ;bool init =0 ;int err =0 ;
@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return err; init =1 ; -switch (con.op) { +switch (con.op &ALG_OP_CIPHER_MASK ) {case ALG_OP_ENCRYPT : enc =1 ;break ;
@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return -EINVAL ; } +if (con.op &ALG_OP_INLINE_IV ) + iiv =1 ; +if (con.iv && con.iv->ivlen != ivsize)return -EINVAL ; }
@@ -866,6 +871,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,if (init) { ctx->enc = enc; + ctx->iiv = iiv;if (con.iv)memcpy (ctx->iv, con.iv->iv, ivsize);
@@ -1031,6 +1037,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)struct sock *sk = areq->sk;af_alg_free_areq_sgls (areq); +if (areq->ivlen) +sock_kfree_s (sk, areq->iv, areq->ivlen);sock_kfree_s (sk, areq, areq->areqlen); }EXPORT_SYMBOL_GPL (af_alg_free_resources);
@@ -1175,6 +1183,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, }EXPORT_SYMBOL_GPL (af_alg_get_rsgl); +/** +* af_alg_get_tsgl_sg - get first SG entry with data from TX SGL +* @ctx [in] AF_ALG context with TX sgl +* @return pointer to SG with data or NULL if none found +*/ +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx) +{ +struct af_alg_tsgl *sgl, *tmp; +struct scatterlist *sg =NULL ; +unsigned int i; + +list_for_each_entry_safe (sgl, tmp, &ctx->tsgl_list, list) { +for (i =0 ; i < sgl->cur; i++) { +struct scatterlist *process_sg = sgl->sg + i; + +if (!(process_sg->length) || !sg_page (process_sg)) +continue ; + sg = process_sg; +break ; + } +if (sg) +break ; + } + +return sg; +} +EXPORT_SYMBOL_GPL (af_alg_get_tsgl_sg); + +/** +* af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX +* +* @sk [in] AF_ALG socket +* @areq [in/out] request buffer that will receive the IV +* to be used for the cipher +* @return 0 on success, < 0 on error +*/ +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; +struct scatterlist *sg; + +/* No inline IV or cipher has no IV, use ctx IV buffer. */ +if (!ctx->iiv || !ctx->ivlen) { + areq->iv = ctx->iv; + areq->ivlen =0 ;// areq->iv will not be freed +return 0 ; + } + +/* There must be the IV data present. */ +if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list)) +return -EINVAL ; + + areq->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL ); +if (unlikely (!areq->iv)) +return -ENOMEM ; + areq->ivlen = ctx->ivlen;// areq->iv will be freed + +/* Get ivlen data from TX SGL and copy it into areq->iv. */ + sg =af_alg_get_tsgl_sg (ctx); +if (!sg) +return -EFAULT ; +scatterwalk_map_and_copy (areq->iv, sg,0 , ctx->ivlen,0 ); +af_alg_pull_tsgl (sk, ctx->ivlen,NULL ,0 ); + +return 0 ; +} +EXPORT_SYMBOL_GPL (af_alg_get_iv); +static int __init af_alg_init (void ) {int err =proto_register (&alg_proto,0 );
crypto/algif_aead.c +24 -30
113 lines not on this page · raw message
crypto/algif_skcipher.c +10 -4
@@ -77,6 +77,11 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); +/* Set areq->iv */ + err =af_alg_get_iv (sk, areq); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq,-1 , &len);if (err)
@@ -112,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ skcipher_request_set_tfm (&areq->cra_u.skcipher_req, tfm);skcipher_request_set_crypt (&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sg, len, areq->iv);if (msg->msg_iocb && !is_sync_kiocb (msg->msg_iocb)) {/* AIO operation */
@@ -345,14 +350,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)if (!ctx)return -ENOMEM ; - ctx->iv =sock_kmalloc (sk,crypto_skcipher_ivsize (tfm), -GFP_KERNEL ); + ctx->ivlen =crypto_skcipher_ivsize (tfm); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +memset (ctx->iv,0 , ctx->ivlen);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;
@@ -361,6 +366,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =0 ;crypto_init_wait (&ctx->wait); ask->private = ctx;
include/crypto/if_alg.h +17 -0
@@ -95,6 +95,14 @@ struct af_alg_rsgl {* @tsgl_entries: Number of entries in priv. TX SGL * @outlen: Number of output bytes generated by crypto op * @areqlen: Length of this data structure +* @iv: Buffer holding the IV for the cipher operation (this +* is either ctx->iv where ivlen is set to 0 or a newly +* allocated buffer where ivlen represents the size of +* this buffer). +* @ivlen: IV size -- if the IV buffer is to be freed during +* releasing of this data structure, ivlen is non-zero. +* If ivlen is zero, but iv is non-NULL, the iv buffer is +* not freed. * @cra_u: Cipher request */ struct af_alg_async_req {
@@ -111,6 +119,9 @@ struct af_alg_async_req {unsigned int outlen;unsigned int areqlen; +void *iv; +unsigned int ivlen; +union {struct aead_request aead_req;struct skcipher_request skcipher_req;
@@ -127,6 +138,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlen: IV size * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation * @used: TX bytes sent to kernel. This variable is used to
@@ -140,12 +152,14 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +unsigned int ivlen;size_t aead_assoclen;struct crypto_wait wait;
@@ -156,6 +170,7 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +bool iiv;unsigned int len; };
@@ -252,5 +267,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq); +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /* _CRYPTO_IF_ALG_H */
include/uapi/linux/if_alg.h +4 -2
@@ -37,7 +37,9 @@ struct af_alg_iv {#define ALG_SET_AEAD_AUTHSIZE 5/* Operations */ -#define ALG_OP_DECRYPT 0 -#define ALG_OP_ENCRYPT 1 +#define ALG_OP_DECRYPT 0x0 +#define ALG_OP_ENCRYPT 0x1 +#define ALG_OP_CIPHER_MASK 0xf +#define ALG_OP_INLINE_IV 0x10#endif /* _LINUX_IF_ALG_H */
Signature · 1 line
2.14.3
Re: [PATCH] crypto: AF_ALG - inline IV support
Hi Herbert,
I tried to summarize the use cases of the AIO support at [1].
The use case covering the inline IV support is documented in section [2]. It naturally would depend on this patch to be accepted. What is your take on this use case?
What is your take on the issue outlined at [3]? Should the affected drivers be updated?
Thanks
Stephan
[1] https://github.com/smuellerDD/libkcapi/commit/29132075b8f045f3f92367c0190add81ccf5da11
[2] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11#diff- dce7a00cbc610df94f0dc27eb769d01dR644
[3] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11#diff- dce7a00cbc610df94f0dc27eb769d01dR575
Re: [PATCH] crypto: AF_ALG - inline IV support
Quoted 10 lines · Stephan Müller
Hi Herbert,
I tried to summarize the use cases of the AIO support at [1].
The use case covering the inline IV support is documented in section [2]. It naturally would depend on this patch to be accepted. What is your take on this use case?
What is your take on the issue outlined at [3]? Should the affected drivers be updated?
Till now driver is using content of req->info as IV for cipher operation and It treats each request independent of previous request.
Quoted 1 line
From description given at [1]. Following is the new requirement for Driver.
"The cipher implementations ensure that although the IOVECs cause parallel invocation of the recvmsg handler, they are serialized such that the one IV sent to the kernel is updated by the first cipher operation and used as input to the second cipher operation, and so on. This implies, for example, that the CBC block chaining operation is applied for all IOVECs."
To make this work how driver will know that it has to delay the processing of current request because previous request is not complete(or IV is not available) .
Quoted 15 lines
Thanks
Stephan[1] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11[2] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11#diff- dce7a00cbc610df94f0dc27eb769d01dR644[3] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11#diff- dce7a00cbc610df94f0dc27eb769d01dR575
PATCH 1/1 crypto: AF_ALG AIO - lock context IV
Hi Harsh,
may I as you to try the attached patch on your Chelsio driver? Please invoke the following command from libkcapi:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
The result should now be a fully block-chained result.
Note, the patch goes on top of the inline IV patch.
Thanks
---8<---
In case of AIO where multiple IOCBs are sent to the kernel and inline IV is not used, the ctx->iv is used for each IOCB. The ctx->iv is read for the crypto operation and written back after the crypto operation. Thus, processing of multiple IOCBs must be serialized.
As the AF_ALG interface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
--- crypto/af_alg.c | 22 ++++++++++++++++++++++ crypto/algif_aead.c | 2 ++ crypto/algif_skcipher.c | 2 ++ include/crypto/if_alg.h | 3 +++ 4 files changed, 29 insertions(+)
crypto/af_alg.c +0 -0
@@ -1059,6 +1059,8 @@ void af_alg_async_cb(struct crypto_async_request *_req,
int err)
struct kiocb *iocb = areq->iocb;
unsigned int resultlen;
+ af_alg_put_iv(sk);
+
/* Buffer size written by crypto operation. */ resultlen = areq->outlen;
@@ -1227,6 +1229,11 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
/* No inline IV or cipher has no IV, use ctx IV buffer. */ if (!ctx->iiv || !ctx->ivlen) {
+ int ret = mutex_lock_interruptible(&ctx->ivlock);
+
+ if (ret)
+ return ret;
+
areq->iv = ctx->iv;
areq->ivlen = 0; // areq->iv will not be freed
return 0;
@@ -1252,6 +1259,21 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
}
EXPORT_SYMBOL_GPL(af_alg_get_iv);
+/**
+ * af_alg_put_iv - release lock on IV in case CTX IV is used
+ *
+ * @sk [in] AF_ALG socket
+ */
+void af_alg_put_iv(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+
+ if (!ctx->iiv || !ctx->ivlen)
+ mutex_unlock(&ctx->ivlock);
+}
+EXPORT_SYMBOL_GPL(af_alg_put_iv);
+
static int __init af_alg_init(void)
{
int err = proto_register(&alg_proto, 0);
crypto/algif_aead.c +0 -0
@@ -309,6 +309,7 @@ static int _aead_recvmsg(struct socket *sock, struct
msghdr *msg,
&ctx->wait);
}
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -555,6 +556,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
return -ENOMEM;
}
memset(ctx->iv, 0, ctx->ivlen);
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
crypto/algif_skcipher.c +0 -0
@@ -151,6 +151,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct
msghdr *msg,
&ctx->wait);
}
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -358,6 +359,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
}
memset(ctx->iv, 0, ctx->ivlen);
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
include/crypto/if_alg.h +2 -0
@@ -19,6 +19,7 @@#include <linux/scatterlist.h> #include <linux/types.h> #include <linux/atomic.h> +#include <linux/mutex.h> #include <net/sock.h> #include <crypto/aead.h>
@@ -160,6 +161,7 @@ struct af_alg_ctx {void *iv;unsigned int ivlen; +struct mutex ivlock;size_t aead_assoclen;struct crypto_wait wait;
@@ -268,6 +270,7 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg,
int flags,
struct af_alg_async_req *areq, size_t maxsize, size_t *outlen);
int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
+void af_alg_put_iv(struct sock *sk);
struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
#endif /* _CRYPTO_IF_ALG_H */
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Am Dienstag, 30. Januar 2018, 09:27:04 CET schrieb Stephan Müller:
Hi,
Quoted 4 lines
(quoted diff) +4
-0+/** + * af_alg_put_iv - release lock on IV in case CTX IV is used + * + * @sk [in] AF_ALG socket
Review
Quoted 10 lines
(quoted diff) +10
-0+ */ +void af_alg_put_iv(struct sock *sk) +{ + struct alg_sock *ask = alg_sk(sk); + struct af_alg_ctx *ctx = ask->private; + + if (!ctx->iiv || !ctx->ivlen) + mutex_unlock(&ctx->ivlock); +} +EXPORT_SYMBOL_GPL(af_alg_put_iv);
Having this function implies that ctx->iiv must be set once at the beginning and cannot be toggled for a CTX thereafter as otherwise the release of the mutex is racy. This implies that the inline IV patch needs a slight revision to prevent toggling the ctx->iiv value.
I will send a new revision of the inline IV and the lock context IV patch covering this issue.
Ciao
Stephan
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
In reply to Stephan MüllerQuoted 26 lines · On Tue, 30 Jan 2018 09:27:04 +0100 Stephan Müller <[email protected]> wrote:
Hi Harsh,
may I as you to try the attached patch on your Chelsio driver? Please invoke the following command from libkcapi:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910The result should now be a fully block-chained result.
Note, the patch goes on top of the inline IV patch.
Thanks
---8<---
In case of AIO where multiple IOCBs are sent to the kernel and inline IV is not used, the
ctx->ivis used for each IOCB. Thectx->ivis read for the crypto operation and written back after the crypto operation. Thus, processing of multiple IOCBs must be serialized.As the
AF_ALGinterface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.
Quoted 5 lines
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
Signed-off-by Stephan Mueller <[email protected]>
As a reference point, holding outside the kernel (in at least the case of our hardware with 8K CBC) drops us down to around 30% of performance with separate IVs. Putting a queue in kernel (and hence allowing most setup of descriptors / DMA etc) gets us up to 50% of raw non chained performance.
So whilst this will work in principle I suspect anyone who cares about the cost will be looking at adding their own software queuing and dependency handling in driver. How much that matters will depend on just how quick the hardware is vs overheads.
Anyhow, just posted the driver as an RFC. There are a few corners that need resolving and the outcome of this thread effects whether we need to play games with IVs or not.
https://marc.info/?l=linux-crypto-vger&m=151732626428206&w=2
grep for softqueue.
We have a number of processing units that will grab requests from the HW queues in parallel. So 'lots' of requests from a given HW queue may be in flight at the same time (the problem case).
Logic is fairly simple.
1. Only create a softqueue if the encryption mode requires chaining. Currently in this driver that is CBC and CTR modes only. Otherwise put everything straight in the hardware monitored queues.
2. If we have an IV embedded within the request it will have a different address to the one used previously (guaranteed if that one is still in flight and it doesn't matter if it isn't). If that occurs we can safely add it to the hardware queue. To avoid DoS issues against and earlier set of messages using the a chained IV we actually make sure nothing is in the software queue (not needed for correctness)
3. If IV matches previous IV and there are existing elements in SW or HW queues we need to hold it in the SW queue.
4. On receiving a response from the hardware and if the hardware queue is empty, we can release an element from the software queue into the hardware queue.
This maintains chaining when needed and gets a good chunk of the lost performance back. I believe (though yet to verify) the remaining lost performance is mostly down to the fact we can't coalesce interrupts if chaining.
Note the driver is deliberately a touch 'simplistic' so there may be other optimization opportunities to get some of the lost performance back or it may be fundamental due to the fact the raw processing can't be in parallel.
Quoted 128 lines
--- crypto/af_alg.c | 22 ++++++++++++++++++++++ crypto/algif_aead.c | 2 ++ crypto/algif_skcipher.c | 2 ++ include/crypto/if_alg.h | 3 +++ 4 files changed, 29 insertions(+)crypto/af_alg.c +7
-0@@ -1059,6 +1059,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)struct kiocb *iocb = areq->iocb;unsigned int resultlen; +af_alg_put_iv (sk); +/* Buffer size written by crypto operation. */ resultlen = areq->outlen;@@ -1227,6 +1229,11 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)/* No inline IV or cipher has no IV, use ctx IV buffer. */ if (!ctx->iiv || !ctx->ivlen) { +int ret =mutex_lock_interruptible (&ctx->ivlock); + +if (ret) +return ret; + areq->iv = ctx->iv; areq->ivlen =0 ;// areq->iv will not be freed crypto/af_alg.c +15
-0return 0 ;@@ -1252,6 +1259,21 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq) }EXPORT_SYMBOL_GPL (af_alg_get_iv); +/** +* af_alg_put_iv - release lock on IV in case CTX IV is used +* +* @sk [in] AF_ALG socket +*/ +void af_alg_put_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +if (!ctx->iiv || !ctx->ivlen) +mutex_unlock (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_put_iv); +static int __init af_alg_init (void ) {crypto/af_alg.c +0
-0int err =proto_register (&alg_proto,0 );crypto/algif_aead.c +1
-0@@ -309,6 +309,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } +af_alg_put_iv (sk); free:crypto/algif_aead.c +1
-0af_alg_free_resources (areq);@@ -555,6 +556,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)return -ENOMEM ; }memset (ctx->iv,0 , ctx->ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list);crypto/algif_aead.c +0
-0ctx->len = len;crypto/algif_skcipher.c +2
-0@@ -151,6 +151,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } +af_alg_put_iv (sk); free:af_alg_free_resources (areq);@@ -358,6 +359,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) }memset (ctx->iv,0 , ctx->ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;include/crypto/if_alg.h +3
-0@@ -19,6 +19,7 @@#include <linux/scatterlist.h> #include <linux/types.h> #include <linux/atomic.h> +#include <linux/mutex.h> #include <net/sock.h> #include <crypto/aead.h> @@ -160,6 +161,7 @@ struct af_alg_ctx {void *iv;unsigned int ivlen; +struct mutex ivlock;size_t aead_assoclen;struct crypto_wait wait;@@ -268,6 +270,7 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg,int flags, struct af_alg_async_req *areq, size_t maxsize, size_t *outlen);int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq); +void af_alg_put_iv (struct sock *sk);struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /*
_CRYPTO_IF_ALG_H*/
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Quoted 2 lines · On Tue, 30 Jan 2018 15:51:40 +0000 Jonathan Cameron…
On Tue, 30 Jan 2018 09:27:04 +0100
Stephan Müller <[email protected]> wrote:
A few clarifications from me after discussions with Stephan this morning.
Quoted 27 lines
Quoted 27 lines
Hi Harsh,
may I as you to try the attached patch on your Chelsio driver? Please invoke the following command from libkcapi:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910The result should now be a fully block-chained result.
Note, the patch goes on top of the inline IV patch.
Thanks
---8<---
In case of AIO where multiple IOCBs are sent to the kernel and inline IV is not used, the
ctx->ivis used for each IOCB. Thectx->ivis read for the crypto operation and written back after the crypto operation. Thus, processing of multiple IOCBs must be serialized.As the
AF_ALGinterface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.
Quoted 5 lines
Quoted 5 lines
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
Signed-off-by Stephan Mueller <[email protected]>
Firstly, as far as I can tell (not tested it yet) the patch does the job and is about the best we can easily do in the AF_ALG code.
I'd suggest that this (or v2 anyway) is stable material as well (which, as Stephan observed, will require reordering of the two patches).
Quoted 25 lines
As a reference point, holding outside the kernel (in at least the case of our hardware with 8K CBC) drops us down to around 30% of performance with separate IVs. Putting a queue in kernel (and hence allowing most setup of descriptors / DMA etc) gets us up to 50% of raw non chained performance.
So whilst this will work in principle I suspect anyone who cares about the cost will be looking at adding their own software queuing and dependency handling in driver. How much that matters will depend on just how quick the hardware is vs overheads.
Anyhow, just posted the driver as an RFC. There are a few corners that need resolving and the outcome of this thread effects whether we need to play games with IVs or not.
https://marc.info/?l=linux-crypto-vger&m=151732626428206&w=2
grep for softqueue.We have a number of processing units that will grab requests from the HW queues in parallel. So 'lots' of requests from a given HW queue may be in flight at the same time (the problem case).
Quoted 15 lines
Logic is fairly simple.
1. Only create a softqueue if the encryption mode requires chaining. Currently in this driver that is CBC and CTR modes only. Otherwise put everything straight in the hardware monitored queues.
2. If we have an IV embedded within the request it will have a different address to the one used previously (guaranteed if that one is still in flight and it doesn't matter if it isn't). If that occurs we can safely add it to the hardware queue. To avoid DoS issues against and earlier set of messages using the a chained IV we actually make sure nothing is in the software queue (not needed for correctness)
3. If IV matches previous IV and there are existing elements in SW or HW queues we need to hold it in the SW queue.
4. On receiving a response from the hardware and if the hardware queue is empty, we can release an element from the software queue into the hardware queue.
The differences are better shown with pictures...
To compare the two approaches. If we break up the data flow the alternatives are
1) Mutex causes queuing in AF ALG code
The key thing is the [Build / Map HW Descs] for small packets, up to perhaps 32K, is a significant task we can't avoid. At 8k it looks like it takes perhaps 20-30% of the time (though I only have end to end performance numbers so far)
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
NOTHING BLOCKING (lock mut) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (lock mut)
| [BUILD / MAP HW DESCS] * AFTER SERIALIZATION *
Don't care beyond here. |
[Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here. 2) The queuing approach I did in the driver moves that serialization to after the BUILD / MAP HW DESCs stage.
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
[BUILD / MAP HW DESCS] |
| [BUILD / MAP HW DESCS] * BEFORE SERIALIZATION *
NOTHING BLOCKING |
| IV in flight (enqueue sw queue)
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| [Place in HW Queue]
Don't care beyond here. |
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here. Quoted 11 lines
This maintains chaining when needed and gets a good chunk of the lost performance back. I believe (though yet to verify) the remaining lost performance is mostly down to the fact we can't coalesce interrupts if chaining.
Note the driver is deliberately a touch 'simplistic' so there may be other optimization opportunities to get some of the lost performance back or it may be fundamental due to the fact the raw processing can't be in parallel.
Quoting a private email from Stephan (at Stephan's suggestion)
Quoted 3 lines
What I however could fathom is that we introduce a flag where a driver implements its own queuing mechanism. In this case, the mutex handling would be disregarded.
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.
Jonathan
Quoted 129 lines
Quoted 128 lines
--- crypto/af_alg.c | 22 ++++++++++++++++++++++ crypto/algif_aead.c | 2 ++ crypto/algif_skcipher.c | 2 ++ include/crypto/if_alg.h | 3 +++ 4 files changed, 29 insertions(+)crypto/af_alg.c +7
-0@@ -1059,6 +1059,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)struct kiocb *iocb = areq->iocb;unsigned int resultlen; +af_alg_put_iv (sk); +/* Buffer size written by crypto operation. */ resultlen = areq->outlen;@@ -1227,6 +1229,11 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)/* No inline IV or cipher has no IV, use ctx IV buffer. */ if (!ctx->iiv || !ctx->ivlen) { +int ret =mutex_lock_interruptible (&ctx->ivlock); + +if (ret) +return ret; + areq->iv = ctx->iv; areq->ivlen =0 ;// areq->iv will not be freed crypto/af_alg.c +15
-0return 0 ;@@ -1252,6 +1259,21 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq) }EXPORT_SYMBOL_GPL (af_alg_get_iv); +/** +* af_alg_put_iv - release lock on IV in case CTX IV is used +* +* @sk [in] AF_ALG socket +*/ +void af_alg_put_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +if (!ctx->iiv || !ctx->ivlen) +mutex_unlock (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_put_iv); +static int __init af_alg_init (void ) {crypto/af_alg.c +0
-0int err =proto_register (&alg_proto,0 );crypto/algif_aead.c +1
-0@@ -309,6 +309,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } +af_alg_put_iv (sk); free:crypto/algif_aead.c +1
-0af_alg_free_resources (areq);@@ -555,6 +556,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)return -ENOMEM ; }memset (ctx->iv,0 , ctx->ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list);crypto/algif_aead.c +0
-0ctx->len = len;crypto/algif_skcipher.c +2
-0@@ -151,6 +151,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } +af_alg_put_iv (sk); free:af_alg_free_resources (areq);@@ -358,6 +359,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) }memset (ctx->iv,0 , ctx->ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;include/crypto/if_alg.h +3
-0@@ -19,6 +19,7 @@#include <linux/scatterlist.h> #include <linux/types.h> #include <linux/atomic.h> +#include <linux/mutex.h> #include <net/sock.h> #include <crypto/aead.h> @@ -160,6 +161,7 @@ struct af_alg_ctx {void *iv;unsigned int ivlen; +struct mutex ivlock;size_t aead_assoclen;struct crypto_wait wait;@@ -268,6 +270,7 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg,int flags, struct af_alg_async_req *areq, size_t maxsize, size_t *outlen);int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq); +void af_alg_put_iv (struct sock *sk);struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /*
_CRYPTO_IF_ALG_H*/
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Hi,
Quoted 60 lines · Jonathan Cameron
Quoted 2 lines · On Tue, 30 Jan 2018 15:51:40 +0000 Jonathan Cameron…
On Tue, 30 Jan 2018 09:27:04 +0100
Stephan Müller <[email protected]> wrote:A few clarifications from me after discussions with Stephan this morning.
Quoted 27 lines
Quoted 27 lines
Hi Harsh,
may I as you to try the attached patch on your Chelsio driver? Please invoke the following command from libkcapi:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910The result should now be a fully block-chained result.
Note, the patch goes on top of the inline IV patch.
Thanks
---8<---
In case of AIO where multiple IOCBs are sent to the kernel and inline IV is not used, the
ctx->ivis used for each IOCB. Thectx->ivis read for the crypto operation and written back after the crypto operation. Thus, processing of multiple IOCBs must be serialized.As the
AF_ALGinterface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.Quoted 5 lines
Quoted 5 lines
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
Signed-off-by Stephan Mueller <[email protected]>Firstly, as far as I can tell (not tested it yet) the patch does the job and is about the best we can easily do in the
AF_ALGcode.I'd suggest that this (or v2 anyway) is stable material as well (which, as Stephan observed, will require reordering of the two patches).
Quoted 13 lines
As a reference point, holding outside the kernel (in at least the case of our hardware with 8K CBC) drops us down to around 30% of performance with separate IVs. Putting a queue in kernel (and hence allowing most setup of descriptors / DMA etc) gets us up to 50% of raw non chained performance.
So whilst this will work in principle I suspect anyone who cares about the cost will be looking at adding their own software queuing and dependency handling in driver. How much that matters will depend on just how quick the hardware is vs overheads.
<snip>
Quoted 93 lines
The differences are better shown with pictures...
To compare the two approaches. If we break up the data flow the alternatives are1) Mutex causes queuing in AF ALG code
The key thing is the [Build / Map HW Descs] for small packets, up to perhaps 32K, is a significant task we can't avoid. At 8k it looks like it takes perhaps 20-30% of the time (though I only have end to end performance numbers so far)
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
NOTHING BLOCKING (lock mut) |
| Queued on Mutex[BUILD / MAP HW DESCS] | | | [Place in HW Queue] | | | [Process] | | | [Interrupt] | | | [Respond and update IV] Nothing Blocking (lock mut) | [BUILD / MAP HW DESCS] * AFTER SERIALIZATION * Don't care beyond here. | [Place in HW Queue] | [Process] | [Interrupt] | [Respond and update IV] | Don't care beyond here.2) The queuing approach I did in the driver moves that serialization to after the BUILD / MAP HW DESCs stage.
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
[BUILD / MAP HW DESCS] |
| [BUILD / MAP HW DESCS] * BEFORE SERIALIZATION *
NOTHING BLOCKING |
| IV in flight (enqueue sw queue)[Place in HW Queue] | | | [Process] | | | [Interrupt] | | | [Respond and update IV] Nothing Blocking (dequeue sw queue) | [Place in HW Queue] Don't care beyond here. | [Process] | [Interrupt] | [Respond and update IV] | Don't care beyond here.Quoted 11 lines
This maintains chaining when needed and gets a good chunk of the lost performance back. I believe (though yet to verify) the remaining lost performance is mostly down to the fact we can't coalesce interrupts if chaining.
Note the driver is deliberately a touch 'simplistic' so there may be other optimization opportunities to get some of the lost performance back or it may be fundamental due to the fact the raw processing can't be in parallel.
Quoting a private email from Stephan (at Stephan's suggestion)
Quoted 3 lines
What I however could fathom is that we introduce a flag where a driver implements its own queuing mechanism. In this case, the mutex handling would be disregarded.
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
Quoted 8 lines
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.
The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
Consider that the issue we are dealing with is in no way unique to the AF_ALG use case, it just happens to expose it.
The grander issue is, I believe, that we are missing an API for chained crypto operations. One that allows submitting multiple concurrent but dependent requests
to tfm providers.
Trying to second guess whether or not there is a dependency between calls from the address of the IV is not a clean solution. Why don't we make it explicit?
For example, a flag that can be set on a tfm that states that the subsequent series of requests are interdependent. If you need a different stream, simply allocated anotehr
tfm object.
This will let each driver do it's best, be it a simple mutex, software queuing or hardware
dependency tracking as the case m may be.
Than of course, AF_ALG code (or any other user for that matter) will not need to handle
interdependence, just set the right flag.
Do you think this makes sense?
Thanks,
Gilad
Signature · 6 lines
Gilad Ben-Yossef Chief Coffee Drinker "If you take a class in large-scale robotics, can you end up in a situation where the homework eats your dog?" -- Jean-Baptiste Queru
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 199 lines
Hi,
On Wed, Jan 31, 2018 at 2:29 PM, Jonathan Cameron
Quoted 59 lines · <[email protected]> wrote:
On Tue, 30 Jan 2018 15:51:40 +0000
Quoted 3 lines · Jonathan Cameron
On Tue, 30 Jan 2018 09:27:04 +0100
Stephan Müller <[email protected]> wrote:
A few clarifications from me after discussions with Stephan this morning.
Quoted 27 lines
Quoted 27 lines
Hi Harsh,
may I as you to try the attached patch on your Chelsio driver? Please invoke the following command from libkcapi:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910The result should now be a fully block-chained result.
Note, the patch goes on top of the inline IV patch.
Thanks
---8<---
In case of AIO where multiple IOCBs are sent to the kernel and inline IV
is not used, thectx->ivis used for each IOCB. Thectx->ivis read for the crypto operation and written back after the crypto operation. Thus, processing of multiple IOCBs must be serialized.As the
AF_ALGinterface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.Quoted 6 lines
Quoted 6 lines
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.Signed-off-by Stephan Mueller <[email protected]>Firstly, as far as I can tell (not tested it yet) the patch does the job and is about the best we can easily do in the
AF_ALGcode.I'd suggest that this (or v2 anyway) is stable material as well (which, as Stephan observed, will require reordering of the two patches).
Quoted 11 lines
As a reference point, holding outside the kernel (in at least the case of our hardware with 8K CBC) drops us down to around 30% of performance with separate IVs. Putting a queue in kernel (and hence allowing most setup of descriptors / DMA etc) gets us up to 50% of raw non chained performance.
So whilst this will work in principle I suspect anyone who cares about the cost will be looking at adding their own software queuing and dependency handling in driver. How much that matters will depend on just how quick the hardware is vs overheads.
<snip>
Quoted 112 lines
The differences are better shown with pictures...
To compare the two approaches. If we break up the data flow the alternatives are1) Mutex causes queuing in AF ALG code
The key thing is the [Build / Map HW Descs] for small packets, up to perhaps 32K, is a significant task we can't avoid. At 8k it looks like it takes perhaps 20-30% of the time (though I only have end to end performance numbers so far)
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
NOTHING BLOCKING (lock mut) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
[Place in HW Queue] |
[Process] |
[Interrupt] |
[Respond and update IV] Nothing Blocking (lock mut)
| [BUILD / MAP HW DESCS] * AFTER | SERIALIZATION *Don't care beyond here. |
[Place in HW Queue]
[Process]
[Interrupt]
[Respond and update IV]
Don't care beyond here.
2) The queuing approach I did in the driver moves that serialization to after the BUILD / MAP HW DESCs stage.
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
[BUILD / MAP HW DESCS] |
| [BUILD / MAP HW DESCS] * BEFORE | SERIALIZATION *NOTHING BLOCKING |
| IV in flight (enqueue sw queue)
[Place in HW Queue] |
[Process] |
[Interrupt] |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| [Place in HW Queue]
Don't care beyond here. |
[Process]
[Interrupt]
[Respond and update IV]
Don't care beyond here.
Quoted 11 lines
This maintains chaining when needed and gets a good chunk of the lost performance back. I believe (though yet to verify) the remaining lost performance is mostly down to the fact we can't coalesce interrupts if chaining.
Note the driver is deliberately a touch 'simplistic' so there may be other
optimization opportunities to get some of the lost performance back or it may be fundamental due to the fact the raw processing can't be in parallel.Quoting a private email from Stephan (at Stephan's suggestion)
Quoted 3 lines
What I however could fathom is that we introduce a flag where a driver implements its own queuing mechanism. In this case, the mutex handling would be disregarded.
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
Quoted 8 lines
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
Consider that the issue we are dealing with is in no way unique to the
AF_ALGuse case, it just happens to expose it.The grander issue is, I believe, that we are missing an API for chained crypto operations. One that allows submitting multiple concurrent but dependent requests
to tfm providers.
Quoted 8 lines
Trying to second guess whether or not there is a dependency between calls from the address of the IV is not a clean solution. Why don't we make it explicit?
For example, a flag that can be set on a tfm that states that the subsequent series of requests are interdependent. If you need a different stream, simply allocated anotehr
tfm object.
But this is already implemented, is it not considering the discussed patches? Again, here are the use cases I see for AIO:
- multiple IOCBs which are totally separate and have no interdependencies: the inline IV approach
- multiple IOCBs which are interdependent: the current API with the patches that we are discussing here (note, a new patch set is coming after the merge window)
- if you have several independent invocations of multiple interdependent
IOCBs: call accept() again on the TFM-FD to get another operations FD. This is followed either option one or two above. Note, this is the idea behind the kcapi_handle_reinit() call just added to libkcapi. So, you have multiple operation-FDs on which you do either interdependent operation or inline IV handling.
Quoted 4 lines
This will let each driver do it's best, be it a simple mutex, software queuing or hardware
dependency tracking as the case m may be.
Again, I am wondering that with the discussed patch set we have this functionality already.
Quoted 4 lines
Than of course,
AF_ALGcode (or any other user for that matter) will not need to handle
interdependence, just set the right flag.
The issue is that some drivers simply do not handle this interdependency at all -- see the bug report from Harsh.
Thus, the current patch set (again which will be updated after the merge window) contains:
1. adding a mutex to AF_ALG to ensure serialization of interdependent calls (option 2 from above) irrespective whether the driver implements support for it or not.
2. add the inline IV handling (to serve option 1)
3. add a flag that can be set by the crypto implementation. If this flag is set, then the mutex of AF_ALG is *not* set assuming that the crypto driver handles the serialization.
Note, option 3 from above is implemented in the proper usage of the AF_ALG interface.
Quoted 5 lines
Do you think this makes sense?
Thanks,
Gilad
Ciao
Stephan
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Quoted 6 lines · Stephan Mueller
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 2 lines
Hi,
<snip>
Quoted 81 lines
Quoted 33 lines
Quoted 13 lines
Quoting a private email from Stephan (at Stephan's suggestion)
Quoted 3 lines
What I however could fathom is that we introduce a flag where a driver implements its own queuing mechanism. In this case, the mutex handling would be disregarded.
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
Quoted 8 lines
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
Consider that the issue we are dealing with is in no way unique to the
AF_ALGuse case, it just happens to expose it.The grander issue is, I believe, that we are missing an API for chained crypto operations. One that allows submitting multiple concurrent but dependent requests
to tfm providers.Quoted 8 lines
Trying to second guess whether or not there is a dependency between calls from the address of the IV is not a clean solution. Why don't we make it explicit?
For example, a flag that can be set on a tfm that states that the subsequent series of requests are interdependent. If you need a different stream, simply allocated anotehr
tfm object.But this is already implemented, is it not considering the discussed patches?
Again, here are the use cases I see for AIO:
- multiple IOCBs which are totally separate and have no interdependencies: the inline IV approach
- multiple IOCBs which are interdependent: the current API with the patches that we are discussing here (note, a new patch set is coming after the merge window)
- if you have several independent invocations of multiple interdependent
IOCBs: callaccept()again on the TFM-FD to get another operations FD. This is followed either option one or two above. Note, this is the idea behind thekcapi_handle_reinit()call just added to libkcapi. So, you have multiple operation-FDs on which you do either interdependent operation or inline IV handling.Quoted 4 lines
This will let each driver do it's best, be it a simple mutex, software queuing or hardware
dependency tracking as the case m may be.Again, I am wondering that with the discussed patch set we have this functionality already.
Quoted 4 lines
Than of course,
AF_ALGcode (or any other user for that matter) will not need to handle
interdependence, just set the right flag.The issue is that some drivers simply do not handle this interdependency at all -- see the bug report from Harsh.
Thus, the current patch set (again which will be updated after the merge window) contains:
1. adding a mutex to
AF_ALGto ensure serialization of interdependent calls (option 2 from above) irrespective whether the driver implements support for it or not.
Quoted 8 lines
2. add the inline IV handling (to serve option 1)
3. add a flag that can be set by the crypto implementation. If this flag is set, then the mutex of
AF_ALGis *not* set assuming that the crypto driver handles the serialization.Note, option 3 from above is implemented in the proper usage of the
AF_ALGinterface.
Sorry for not communicating clearly enough. I was not objecting at all. Let me try again.
What I was trying to say is, more succinctly hopefully this time:
1. Advocating for that 3rd option. I did not understand the plan include adding it,
2. Pointing out that the problem solved (and rightfully so) by mutex in AF_ALG AIO implementation must exists elsewhere as well - for example IPsec, and is probably solved there too, so if we add the flag as suggested, it can be used there as well to gain similar benefits to what Jonathan is suggesting.
Thanks,
Gilad
Signature · 6 lines
Gilad Ben-Yossef Chief Coffee Drinker "If you take a class in large-scale robotics, can you end up in a situation where the homework eats your dog?" -- Jean-Baptiste Queru
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Am Donnerstag, 1. Februar 2018, 11:06:30 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 5 lines
2. Pointing out that the problem solved (and rightfully so) by mutex in
AF_ALGAIO implementation must exists elsewhere as well - for example IPsec, and is probably solved there too, so if we add the flag as suggested, it can be used there as well to gain similar benefits to what Jonathan is suggesting.
You are quite right. this patch could speed up things for IPSec and the disk encryption logic too. So I think we should CC also the IPSec folks and the disk crypto folks to review the patch.
Ciao
Stephan
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
In reply to Gilad Ben-YossefAm Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 10 lines
Quoted 10 lines
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
Quoted 8 lines
Quoted 5 lines
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
What you made me think of is the following: shouldn't we relay the inline IV flag on to the crypto drivers?
The reason is to allow a clean implementation of the enabling or disabling of the dependency handling in the driver. Jonathan's driver, for example, decides based on the pointer value of the IV buffer whether it is the same buffer and thus dependency handling is to be applied. This is fragile.
As AF_ALG knows whether the inline IV with separate IVs per request or the serialization with one IV buffer for all requests is requested, it should relay this state on to the drivers. Thus, for example, Jonathan's driver can be changed to rely on this flag instead on the buffer pointer value to decide whether to enable its dependency handling.
Ciao
Stephan
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Quoted 31 lines · Stephan Mueller
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 10 lines
Quoted 10 lines
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
Quoted 8 lines
Quoted 5 lines
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
What you made me think of is the following: shouldn't we relay the inline IV flag on to the crypto drivers?
The reason is to allow a clean implementation of the enabling or disabling of the dependency handling in the driver. Jonathan's driver, for example, decides based on the pointer value of the IV buffer whether it is the same buffer and thus dependency handling is to be applied. This is fragile.
Quoted 5 lines
As
AF_ALGknows whether the inline IV with separate IVs per request or the serialization with one IV buffer for all requests is requested, it should relay this state on to the drivers. Thus, for example, Jonathan's driver can be changed to rely on this flag instead on the buffer pointer value to decide whether to enable its dependency handling.
Yes, that is exactly what I was trying to point out :-)
Thanks,
Gilad
Signature · 6 lines
Gilad Ben-Yossef Chief Coffee Drinker "If you take a class in large-scale robotics, can you end up in a situation where the homework eats your dog?" -- Jean-Baptiste Queru
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Quoted 24 lines · On Thu, 1 Feb 2018 12:07:21 +0200 Gilad Ben-Yossef <[email protected]> wrote:
Quoted 23 lines · Stephan Mueller
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 18 lines
Quoted 7 lines
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
Quoted 8 lines
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
Quoted 7 lines
Quoted 7 lines
What you made me think of is the following: shouldn't we relay the inline IV flag on to the crypto drivers?
The reason is to allow a clean implementation of the enabling or disabling of the dependency handling in the driver. Jonathan's driver, for example, decides based on the pointer value of the IV buffer whether it is the same buffer and thus dependency handling is to be applied. This is fragile.
I agree it's inelegant and a flag would be better than pointer tricks (though they are safe currently - we never know what might change in future) It was really a minimal example rather than a suggestion of the ultimate solution ;) I was planning on suggesting a flag myself once the basic discussion concluded the approach was worthwhile.
Quoted 8 lines
Quoted 6 lines
As
AF_ALGknows whether the inline IV with separate IVs per request or the serialization with one IV buffer for all requests is requested, it should relay this state on to the drivers. Thus, for example, Jonathan's driver can be changed to rely on this flag instead on the buffer pointer value to decide whether to enable its dependency handling.Yes, that is exactly what I was trying to point out :-)
Agreed - make things explicit rather than basically relying on knowing how the above layers are working.
Thanks,
Jonathan
Re: [PATCH] crypto: AF_ALG AIO - lock context IV
Quoted 37 lines · Jonathan Cameron
Quoted 21 lines · On Thu, 1 Feb 2018 12:07:21 +0200 Gilad Ben-Yossef <[email protected]> wrote:
Quoted 20 lines · Stephan Mueller
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
Quoted 16 lines
Quoted 6 lines
Which works well for the sort of optimization I did and for hardware that can do iv dependency tracking itself. If hardware dependency tracking was avilable, you would be able to queue up requests with a chained IV without taking any special precautions at all. The hardware would handle the IV update dependencies.
Quoted 8 lines
So in conclusion, Stephan's approach should work and give us a nice small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag that Stephan is proposing adding to bypass the mutex approach.The patches from Stephan looks good to me, but I think we can do better for the long term approach you are discussing.
Quoted 7 lines
Quoted 7 lines
What you made me think of is the following: shouldn't we relay the inline IV flag on to the crypto drivers?
The reason is to allow a clean implementation of the enabling or disabling of the dependency handling in the driver. Jonathan's driver, for example, decides based on the pointer value of the IV buffer whether it is the same buffer and thus dependency handling is to be applied. This is fragile.
I agree it's inelegant and a flag would be better than pointer tricks (though they are safe currently - we never know what might change in future) It was really a minimal example rather than a suggestion of the ultimate solution ;) I was planning on suggesting a flag myself once the basic discussion concluded the approach was worthwhile.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
As
AF_ALGknows whether the inline IV with separate IVs per request or the serialization with one IV buffer for all requests is requested, it should relay this state on to the drivers. Thus, for example, Jonathan's driver can be changed to rely on this flag instead on the buffer pointer value to decide whether to enable its dependency handling.Yes, that is exactly what I was trying to point out :-)
Agreed - make things explicit rather than basically relying on knowing how the above layers are working.
IPSec layer may not get benefit from this because they send complete sg list in single request only. They don't need partial mode support.
Quoted 5 lines
Thanks,
Jonathan
v2 cover crypto: AF_ALG AIO improvements
In reply to Jonathan CameronHi Herbert,
Herbert, the patch 1 is meant for stable. However, this patch as is only applies to the new AF_ALG interface implementation. Though, the issue goes back to the first implementation of AIO support. Shall I try prepare a patch for the old AF_ALG implementation as well?
Changes from v1:
* integrate the inline IV and locking patch into one patch set
* reverse the order of lock context IV patch and inline IV patch -- the reason is to allow the first patch to be back-ported to stable
* mark the first patch (locking of the context IV) as applicable to stable as there is an existing inconsistency which was demonstrated by Harsh with the Chelsio driver vs the AES-NI driver
* modify the inline IV patch to have proper unlocking of the mutex in case of errors
* prevent locking if no IV is defined by cipher
* add a patch to allow crypto drivers to report whether they support serialization -- in this case the locking in AF_ALG shall be disabled
* add a patch to inform the crypto drivers that their serialization support should actually be enabled and used because AF_ALG does not serialize the interdependent parallel AIO requests
* streamline the code in patch 1 and 2 slightly
I would like to ask the folks with real AIO hardware (Harsh, Jonathan) to test the patches. Especially, is the locking patch should be tested by Harsh as you have seen the issue with your hardware.
Thanks.
Stephan Mueller (4):
crypto: AF_ALG AIO - lock context IV
crypto: AF_ALG - inline IV support
crypto: AF_ALG - allow driver to serialize IV access
crypto: add CRYPTO_TFM_REQ_PARALLEL flag
crypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++- crypto/algif_aead.c | 86 +++++++++++++++++--------------- crypto/algif_skcipher.c | 38 ++++++++++---- include/crypto/if_alg.h | 37 ++++++++++++++ include/linux/crypto.h | 16 ++++++ include/uapi/linux/if_alg.h | 6 ++- 6 files changed, 249 insertions(+), 53 deletions(-)
Signature · 1 line
2.14.3
PATCH v2 1/4 crypto: AF_ALG AIO - lock context IV
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in the AF_ALG context that is unique to one particular AF_ALG socket. Note the analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the AF_ALG socket context is registered with the new request and the cipher operation is invoked.
You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
As the AF_ALG interface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
--- crypto/af_alg.c | 31 +++++++++++++++++++++++++++++++ crypto/algif_aead.c | 20 +++++++++++++------- crypto/algif_skcipher.c | 12 +++++++++--- include/crypto/if_alg.h | 5 +++++ 4 files changed, 58 insertions(+), 10 deletions(-)
crypto/af_alg.c +31 -0
@@ -1051,6 +1051,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)struct kiocb *iocb = areq->iocb;unsigned int resultlen; +af_alg_put_iv (sk); +/* Buffer size written by crypto operation. */ resultlen = areq->outlen;
@@ -1175,6 +1177,35 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, }EXPORT_SYMBOL_GPL (af_alg_get_rsgl); +/** +* af_alg_get_iv +* +* @sk [in] AF_ALG socket +* @return 0 on success, < 0 on error +*/ +int af_alg_get_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +return mutex_lock_interruptible (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_get_iv); + +/** +* af_alg_put_iv - release lock on IV in case CTX IV is used +* +* @sk [in] AF_ALG socket +*/ +void af_alg_put_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +mutex_unlock (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_put_iv); +static int __init af_alg_init (void ) {int err =proto_register (&alg_proto,0 );
crypto/algif_aead.c +13 -7
@@ -159,10 +159,14 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); + err =af_alg_get_iv (sk); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq, outlen, &usedpages);if (err) - goto free; + goto unlock;/* * Ensure output buffer is sufficiently large. If the caller provides
@@ -176,7 +180,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (used < less) { err = -EINVAL ; - goto free; + goto unlock; } used -= less; outlen -= less;
@@ -197,7 +201,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, }if (processed && !tsgl_src) { err = -EFAULT ; - goto free; + goto unlock; }/*
@@ -230,7 +234,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, err =crypto_aead_copy_sgl (null_tfm, tsgl_src, areq->first_rsgl.sgl.sg, processed);if (err) - goto free; + goto unlock;af_alg_pull_tsgl (sk, processed,NULL ,0 ); }else {/*
@@ -248,7 +252,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, err =crypto_aead_copy_sgl (null_tfm, tsgl_src, areq->first_rsgl.sgl.sg, outlen);if (err) - goto free; + goto unlock;/* Create TX SGL for tag and chain it to RX SGL. */ areq->tsgl_entries = af_alg_count_tsgl(sk, processed,
@@ -260,7 +264,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, GFP_KERNEL);if (!areq->tsgl) { err = -ENOMEM ; - goto free; + goto unlock; }sg_init_table (areq->tsgl, areq->tsgl_entries);
@@ -316,7 +320,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } - +unlock: +af_alg_put_iv (sk); free:af_alg_free_resources (areq);
@@ -562,6 +567,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)return -ENOMEM ; }memset (ctx->iv,0 , ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;
crypto/algif_skcipher.c +9 -3
@@ -77,10 +77,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); + err =af_alg_get_iv (sk); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq,-1 , &len);if (err) - goto free; + goto unlock;/* Process only as much RX buffers for which we have TX data */ if (len > ctx->used)
@@ -104,7 +108,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, GFP_KERNEL);if (!areq->tsgl) { err = -ENOMEM ; - goto free; + goto unlock; }sg_init_table (areq->tsgl, areq->tsgl_entries);af_alg_pull_tsgl (sk, len, areq->tsgl,0 );
@@ -146,7 +150,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } - +unlock: +af_alg_put_iv (sk); free:af_alg_free_resources (areq);
@@ -353,6 +358,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) }memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;
include/crypto/if_alg.h +5 -0
@@ -19,6 +19,7 @@#include <linux/scatterlist.h> #include <linux/types.h> #include <linux/atomic.h> +#include <linux/mutex.h> #include <net/sock.h> #include <crypto/aead.h>
@@ -127,6 +128,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlock: Mutex to serialize access to the IV * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation * @used: TX bytes sent to kernel. This variable is used to
@@ -146,6 +148,7 @@ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +struct mutex ivlock;size_t aead_assoclen;struct crypto_wait wait;
@@ -252,5 +255,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); +int af_alg_get_iv (struct sock *sk); +void af_alg_put_iv (struct sock *sk);#endif /* _CRYPTO_IF_ALG_H */
Signature · 1 line
2.14.3
PATCH v2 2/4 crypto: AF_ALG - inline IV support
In reply to Stephan MüllerThe kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in the AF_ALG context that is unique to one particular AF_ALG socket. Note the analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.
As of now, the individual IOCBs are serialized with respect to the IV handling. This implies that the kernel does not perform a truly parallel invocation of the cipher implementations. However, if the user wants to perform cryptographic operations on multiple IOCBs where each IOCB is truly independent from the other, parallel invocations are possible. This would require that each IOCB provides its own IV to ensure true separation of the IOCBs.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the AF_ALG interface treats the ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.
If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The inline IV handling support is only allowed to be enabled during the first sendmsg call for a context. Any subsequent sendmsg calls are not allowed to change the setting of the inline IV handling (either enable or disable it) as this would open up a race condition with the locking and unlocking of the ctx->ivlock mutex.
The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that ctx->used is updated. Thus, the ctx->used access in _aead_recvmsg must be moved below the IV gathering.
The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
--- crypto/af_alg.c | 93 ++++++++++++++++++++++++++++++++++++++++++--- crypto/algif_aead.c | 58 ++++++++++++---------------- crypto/algif_skcipher.c | 12 +++--- include/crypto/if_alg.h | 21 +++++++++- include/uapi/linux/if_alg.h | 6 ++- 5 files changed, 143 insertions(+), 47 deletions(-)
crypto/af_alg.c +88 -5
@@ -14,6 +14,7 @@#include <linux/atomic.h> #include <crypto/if_alg.h> +#include <crypto/scatterwalk.h> #include <linux/crypto.h> #include <linux/init.h> #include <linux/kernel.h>
@@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,struct af_alg_control con = {};long copied =0 ;bool enc =0 ; +int iiv =ALG_IIV_DISABLE ;bool init =0 ;int err =0 ;
@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return err; init =1 ; -switch (con.op) { +switch (con.op &ALG_OP_CIPHER_MASK ) {case ALG_OP_ENCRYPT : enc =1 ;break ;
@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return -EINVAL ; } +if (con.op &ALG_OP_INLINE_IV ) + iiv =ALG_IIV_USE ; +if (con.iv && con.iv->ivlen != ivsize)return -EINVAL ; }
@@ -866,6 +871,19 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,if (init) { ctx->enc = enc; + +/* +* IIV can only be enabled once with the first sendmsg call. +* This prevents a race in locking and unlocking the +* ctx->ivlock mutex. +*/ +if (ctx->iiv ==ALG_IIV_UNSET ) { + ctx->iiv = iiv; + }else if (iiv ==ALG_IIV_USE ) { + err = -EINVAL ; + goto unlock; + } +if (con.iv)memcpy (ctx->iv, con.iv->iv, ivsize);
@@ -1031,6 +1049,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)struct sock *sk = areq->sk;af_alg_free_areq_sgls (areq); +if (areq->ivlen) +sock_kfree_s (sk, areq->iv, areq->ivlen);sock_kfree_s (sk, areq, areq->areqlen); }EXPORT_SYMBOL_GPL (af_alg_free_resources);
@@ -1178,17 +1198,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,EXPORT_SYMBOL_GPL (af_alg_get_rsgl);/** -* af_alg_get_iv +* af_alg_get_tsgl_sg - get first SG entry with data from TX SGL +* @ctx [in] AF_ALG context with TX sgl +* @return pointer to SG with data or NULL if none found +*/ +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx) +{ +struct af_alg_tsgl *sgl, *tmp; +struct scatterlist *sg =NULL ; +unsigned int i; + +list_for_each_entry_safe (sgl, tmp, &ctx->tsgl_list, list) { +for (i =0 ; i < sgl->cur; i++) { +struct scatterlist *process_sg = sgl->sg + i; + +if (!(process_sg->length) || !sg_page (process_sg)) +continue ; + sg = process_sg; +break ; + } +if (sg) +break ; + } + +return sg; +} +EXPORT_SYMBOL_GPL (af_alg_get_tsgl_sg); + +/** +* af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX * * @sk [in] AF_ALG socket +* @areq [in/out] request buffer that will receive the IV +* to be used for the cipher * @return 0 on success, < 0 on error */ -int af_alg_get_iv (struct sock *sk) +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq) {struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; +struct scatterlist *sg; + + areq->iv = ctx->iv; + areq->ivlen =0 ;// areq->iv will not be freed + +/* If cipher has no IV, no need to lock it or do IIV processing */ +if (!ctx->ivlen) +return 0 ; -return mutex_lock_interruptible (&ctx->ivlock); +/* No inline IV, use ctx IV buffer and lock it */ +if (ctx->iiv ==ALG_IIV_DISABLE ) +return mutex_lock_interruptible (&ctx->ivlock); + +/* Inline IV handling: There must be the IV data present. */ +if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list)) +return -EINVAL ; + + areq->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL ); +if (unlikely (!areq->iv)) +return -ENOMEM ; + areq->ivlen = ctx->ivlen;// areq->iv will be freed + +/* Get ivlen data from TX SGL and copy it into areq->iv. */ + sg =af_alg_get_tsgl_sg (ctx); +if (!sg) +return -EFAULT ; +scatterwalk_map_and_copy (areq->iv, sg,0 , ctx->ivlen,0 ); +af_alg_pull_tsgl (sk, ctx->ivlen,NULL ,0 ); + +return 0 ; }EXPORT_SYMBOL_GPL (af_alg_get_iv);
@@ -1202,7 +1280,12 @@ void af_alg_put_iv(struct sock *sk)struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; -mutex_unlock (&ctx->ivlock); +/* To resemble af_alg_get_iv, do not merge the two branches. */ +if (!ctx->ivlen) +return ; + +if (ctx->iiv ==ALG_IIV_DISABLE ) +mutex_unlock (&ctx->ivlock); }EXPORT_SYMBOL_GPL (af_alg_put_iv);
crypto/algif_aead.c +24 -34
117 lines not on this page · raw message
crypto/algif_skcipher.c +7 -5
@@ -77,7 +77,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); - err =af_alg_get_iv (sk); +/* Set areq->iv */ + err =af_alg_get_iv (sk, areq);if (err) goto free;
@@ -116,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ skcipher_request_set_tfm (&areq->cra_u.skcipher_req, tfm);skcipher_request_set_crypt (&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sg, len, areq->iv);if (msg->msg_iocb && !is_sync_kiocb (msg->msg_iocb)) {/* AIO operation */
@@ -350,14 +351,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)if (!ctx)return -ENOMEM ; - ctx->iv =sock_kmalloc (sk,crypto_skcipher_ivsize (tfm), -GFP_KERNEL ); + ctx->ivlen =crypto_skcipher_ivsize (tfm); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +memset (ctx->iv,0 , ctx->ivlen);mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list);
@@ -367,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =ALG_IIV_UNSET ;crypto_init_wait (&ctx->wait); ask->private = ctx;
include/crypto/if_alg.h +20 -1
@@ -96,6 +96,14 @@ struct af_alg_rsgl {* @tsgl_entries: Number of entries in priv. TX SGL * @outlen: Number of output bytes generated by crypto op * @areqlen: Length of this data structure +* @iv: Buffer holding the IV for the cipher operation (this +* is either ctx->iv where ivlen is set to 0 or a newly +* allocated buffer where ivlen represents the size of +* this buffer). +* @ivlen: IV size -- if the IV buffer is to be freed during +* releasing of this data structure, ivlen is non-zero. +* If ivlen is zero, but iv is non-NULL, the iv buffer is +* not freed. * @cra_u: Cipher request */ struct af_alg_async_req {
@@ -112,6 +120,9 @@ struct af_alg_async_req {unsigned int outlen;unsigned int areqlen; +void *iv; +unsigned int ivlen; +union {struct aead_request aead_req;struct skcipher_request skcipher_req;
@@ -128,6 +139,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlen: IV size * @ivlock: Mutex to serialize access to the IV * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation
@@ -142,12 +154,14 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +unsigned int ivlen;struct mutex ivlock;size_t aead_assoclen;
@@ -159,9 +173,13 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +int iiv;unsigned int len; }; +#define ALG_IIV_UNSET (-1) +#define ALG_IIV_DISABLE (0) +#define ALG_IIV_USE (1)int af_alg_register_type (const struct af_alg_type *type);int af_alg_unregister_type (const struct af_alg_type *type);
@@ -255,7 +273,8 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); -int af_alg_get_iv (struct sock *sk); +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq);void af_alg_put_iv (struct sock *sk); +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /* _CRYPTO_IF_ALG_H */
include/uapi/linux/if_alg.h +4 -2
@@ -37,7 +37,9 @@ struct af_alg_iv {#define ALG_SET_AEAD_AUTHSIZE 5/* Operations */ -#define ALG_OP_DECRYPT 0 -#define ALG_OP_ENCRYPT 1 +#define ALG_OP_DECRYPT 0x0 +#define ALG_OP_ENCRYPT 0x1 +#define ALG_OP_CIPHER_MASK 0xf +#define ALG_OP_INLINE_IV 0x10#endif /* _LINUX_IF_ALG_H */
Signature · 1 line
2.14.3
Re: [PATCH v2 2/4] crypto: AF_ALG - inline IV support
Quoted 51 lines · On Wed, 7 Feb 2018 08:43:32 +0100 Stephan Müller <[email protected]> wrote:
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.As of now, the individual IOCBs are serialized with respect to the IV handling. This implies that the kernel does not perform a truly parallel invocation of the cipher implementations. However, if the user wants to perform cryptographic operations on multiple IOCBs where each IOCB is truly independent from the other, parallel invocations are possible. This would require that each IOCB provides its own IV to ensure true separation of the IOCBs.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the
AF_ALGinterface treats theALG_SET_OPflag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The inline IV handling support is only allowed to be enabled during the first sendmsg call for a context. Any subsequent sendmsg calls are not allowed to change the setting of the inline IV handling (either enable or disable it) as this would open up a race condition with the locking and unlocking of the
ctx->ivlockmutex.The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that
ctx->usedis updated. Thus, thectx->usedaccess in_aead_recvmsgmust be moved below the IV gathering.The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
Quoted 6 lines
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
Signed-off-by Stephan Mueller <[email protected]>
It's not a 'bug' as such, but this does currently break the crypto-perf test for aio and iiv in the libkcapi branch. We can perhaps make it more forgiving...
I suggest we let noop cases where we set IIV on when it is already on to not result in an error but to just be ignored.
Jonathan
Quoted 50 lines
--- crypto/af_alg.c | 93 ++++++++++++++++++++++++++++++++++++++++++--- crypto/algif_aead.c | 58 ++++++++++++---------------- crypto/algif_skcipher.c | 12 +++--- include/crypto/if_alg.h | 21 +++++++++- include/uapi/linux/if_alg.h | 6 ++- 5 files changed, 143 insertions(+), 47 deletions(-)crypto/af_alg.c +6
-1@@ -14,6 +14,7 @@#include <linux/atomic.h> #include <crypto/if_alg.h> +#include <crypto/scatterwalk.h> #include <linux/crypto.h> #include <linux/init.h> #include <linux/kernel.h> @@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,struct af_alg_control con = {};long copied =0 ;bool enc =0 ; +int iiv =ALG_IIV_DISABLE ;bool init =0 ;int err =0 ;@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return err; init =1 ; -switch (con.op) { +switch (con.op &ALG_OP_CIPHER_MASK ) {case ALG_OP_ENCRYPT : enc =1 ;break ;@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return -EINVAL ; } +if (con.op &ALG_OP_INLINE_IV ) + iiv =ALG_IIV_USE ; +if (con.iv && con.iv->ivlen != ivsize)return -EINVAL ; }@@ -866,6 +871,19 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,if (init) {
crypto/af_alg.c int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, · line 882
Quoted 10 lines
crypto/af_alg.c +9
-0ctx->enc = enc; + +/* +* IIV can only be enabled once with the first sendmsg call. +* This prevents a race in locking and unlocking the +* ctx->ivlock mutex. +*/ +if (ctx->iiv ==ALG_IIV_UNSET ) { + ctx->iiv = iiv; + }else if (iiv ==ALG_IIV_USE ) {
So the issue in kcapi is that we are running a tight loop around a test case that does a certain number of aio messages. At the start of each run of that performance loop we reinitialize.
There is an optmisation in there though which avoids an accept call (which would have given a new context) if we have one available already.
The result is that we are reusing the context which already has this set.
So, can we be a little less restrictive and let things through if and only iff iiv = ctx->iiv - i.e. the set is a noop?
That will make life nicer for performance testing and other code which just sets this without thinking on repeat calls.
Quoted 380 lines
crypto/af_alg.c +73
-4+ err = -EINVAL ; + goto unlock; + } +if (con.iv)memcpy (ctx->iv, con.iv->iv, ivsize);@@ -1031,6 +1049,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)struct sock *sk = areq->sk;af_alg_free_areq_sgls (areq); +if (areq->ivlen) +sock_kfree_s (sk, areq->iv, areq->ivlen);sock_kfree_s (sk, areq, areq->areqlen); }EXPORT_SYMBOL_GPL (af_alg_free_resources);@@ -1178,17 +1198,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,EXPORT_SYMBOL_GPL (af_alg_get_rsgl);/** -* af_alg_get_iv +* af_alg_get_tsgl_sg - get first SG entry with data from TX SGL +* @ctx [in] AF_ALG context with TX sgl +* @return pointer to SG with data or NULL if none found +*/ +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx) +{ +struct af_alg_tsgl *sgl, *tmp; +struct scatterlist *sg =NULL ; +unsigned int i; + +list_for_each_entry_safe (sgl, tmp, &ctx->tsgl_list, list) { +for (i =0 ; i < sgl->cur; i++) { +struct scatterlist *process_sg = sgl->sg + i; + +if (!(process_sg->length) || !sg_page (process_sg)) +continue ; + sg = process_sg; +break ; + } +if (sg) +break ; + } + +return sg; +} +EXPORT_SYMBOL_GPL (af_alg_get_tsgl_sg); + +/** +* af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX * * @sk [in] AF_ALG socket +* @areq [in/out] request buffer that will receive the IV +* to be used for the cipher * @return 0 on success, < 0 on error */ -int af_alg_get_iv (struct sock *sk) +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq) {struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; +struct scatterlist *sg; + + areq->iv = ctx->iv; + areq->ivlen =0 ;// areq->iv will not be freed + +/* If cipher has no IV, no need to lock it or do IIV processing */ +if (!ctx->ivlen) +return 0 ; -return mutex_lock_interruptible (&ctx->ivlock); +/* No inline IV, use ctx IV buffer and lock it */ +if (ctx->iiv ==ALG_IIV_DISABLE ) +return mutex_lock_interruptible (&ctx->ivlock); + +/* Inline IV handling: There must be the IV data present. */ +if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list)) +return -EINVAL ; + + areq->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL ); +if (unlikely (!areq->iv)) +return -ENOMEM ; + areq->ivlen = ctx->ivlen;// areq->iv will be freed + +/* Get ivlen data from TX SGL and copy it into areq->iv. */ + sg =af_alg_get_tsgl_sg (ctx); +if (!sg) +return -EFAULT ; +scatterwalk_map_and_copy (areq->iv, sg,0 , ctx->ivlen,0 ); +af_alg_pull_tsgl (sk, ctx->ivlen,NULL ,0 ); + +return 0 ; }EXPORT_SYMBOL_GPL (af_alg_get_iv);@@ -1202,7 +1280,12 @@ void af_alg_put_iv(struct sock *sk)struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; -mutex_unlock (&ctx->ivlock); +/* To resemble af_alg_get_iv, do not merge the two branches. */ +if (!ctx->ivlen) +return ; + +if (ctx->iiv ==ALG_IIV_DISABLE ) +mutex_unlock (&ctx->ivlock); }EXPORT_SYMBOL_GPL (af_alg_put_iv);crypto/algif_aead.c +24
-34@@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,struct aead_tfm *aeadc = pask->private;struct crypto_aead *tfm = aeadc->aead;struct crypto_skcipher *null_tfm = aeadc->null_tfm; -unsigned int i, as =crypto_aead_authsize (tfm); +unsigned int as =crypto_aead_authsize (tfm);struct af_alg_async_req *areq; -struct af_alg_tsgl *tsgl, *tmp;struct scatterlist *rsgl_src, *tsgl_src =NULL ;int err =0 ;size_t used =0 ;/* [in] TX bufs to be en/decrypted */ @@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,return err; } -/* -* Data length provided by caller via sendmsg/sendpage that has not -* yet been processed. -*/ - used = ctx->used; -/* * Make sure sufficient data is present -- note, the same check is * is also present in sendmsg/sendpage. The checks in sendpage/sendmsg @@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (!aead_sufficient_data (sk))return -EINVAL ; +/* Allocate cipher request for current operation. */ + areq =af_alg_alloc_areq (sk,sizeof (struct af_alg_async_req ) + +crypto_aead_reqsize (tfm)); +if (IS_ERR (areq)) +return PTR_ERR (areq); + +/* Set areq->iv. */ + err =af_alg_get_iv (sk, areq); +if (err) + goto free; + +/* +* Data length provided by caller via sendmsg/sendpage that has not +* yet been processed. +*/ + used = ctx->used; +/* * Calculate the minimum output buffer size holding the result of the * cipher operation. When encrypting data, the receiving buffer is @@ -153,16 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,*/ used -= ctx->aead_assoclen; -/* Allocate cipher request for current operation. */ - areq =af_alg_alloc_areq (sk,sizeof (struct af_alg_async_req ) + -crypto_aead_reqsize (tfm)); -if (IS_ERR (areq)) -return PTR_ERR (areq); - - err =af_alg_get_iv (sk); -if (err) - goto free; -/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq, outlen, &usedpages);if (err)@@ -187,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, } processed = used + ctx->aead_assoclen; -list_for_each_entry_safe (tsgl, tmp, &ctx->tsgl_list, list) { -for (i =0 ; i < tsgl->cur; i++) { -struct scatterlist *process_sg = tsgl->sg + i; - -if (!(process_sg->length) || !sg_page (process_sg)) -continue ; - tsgl_src = process_sg; -break ; - } -if (tsgl_src) -break ; - } + tsgl_src =af_alg_get_tsgl_sg (ctx);if (processed && !tsgl_src) { err = -EFAULT ; goto unlock;@@ -286,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ aead_request_set_crypt (&areq->cra_u.aead_req, rsgl_src, - areq->first_rsgl.sgl.sg, used, ctx->iv); + areq->first_rsgl.sgl.sg, used, areq->iv);aead_request_set_ad (&areq->cra_u.aead_req, ctx->aead_assoclen);aead_request_set_tfm (&areq->cra_u.aead_req, tfm);@@ -554,19 +543,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)struct aead_tfm *tfm = private;struct crypto_aead *aead = tfm->aead;unsigned int len =sizeof (*ctx); -unsigned int ivlen =crypto_aead_ivsize (aead); ctx =sock_kmalloc (sk, len,GFP_KERNEL );if (!ctx)return -ENOMEM ;memset (ctx,0 , len); - ctx->iv =sock_kmalloc (sk, ivlen,GFP_KERNEL ); + ctx->ivlen =crypto_aead_ivsize (aead); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 , ivlen); +memset (ctx->iv,0 , ctx->ivlen);mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list);@@ -576,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =ALG_IIV_UNSET ; ctx->aead_assoclen =0 ;crypto_init_wait (&ctx->wait);crypto/algif_skcipher.c +7
-5@@ -77,7 +77,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); - err =af_alg_get_iv (sk); +/* Set areq->iv */ + err =af_alg_get_iv (sk, areq);if (err) goto free;@@ -116,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ skcipher_request_set_tfm (&areq->cra_u.skcipher_req, tfm);skcipher_request_set_crypt (&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sg, len, areq->iv);if (msg->msg_iocb && !is_sync_kiocb (msg->msg_iocb)) {/* AIO operation */ @@ -350,14 +351,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)if (!ctx)return -ENOMEM ; - ctx->iv =sock_kmalloc (sk,crypto_skcipher_ivsize (tfm), -GFP_KERNEL ); + ctx->ivlen =crypto_skcipher_ivsize (tfm); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +memset (ctx->iv,0 , ctx->ivlen);mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list);@@ -367,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =ALG_IIV_UNSET ;crypto_init_wait (&ctx->wait); ask->private = ctx;include/crypto/if_alg.h +20
-1@@ -96,6 +96,14 @@ struct af_alg_rsgl {* @tsgl_entries: Number of entries in priv. TX SGL * @outlen: Number of output bytes generated by crypto op * @areqlen: Length of this data structure +* @iv: Buffer holding the IV for the cipher operation (this +* is either ctx->iv where ivlen is set to 0 or a newly +* allocated buffer where ivlen represents the size of +* this buffer). +* @ivlen: IV size -- if the IV buffer is to be freed during +* releasing of this data structure, ivlen is non-zero. +* If ivlen is zero, but iv is non-NULL, the iv buffer is +* not freed. * @cra_u: Cipher request */ struct af_alg_async_req {@@ -112,6 +120,9 @@ struct af_alg_async_req {unsigned int outlen;unsigned int areqlen; +void *iv; +unsigned int ivlen; +union {struct aead_request aead_req;struct skcipher_request skcipher_req;@@ -128,6 +139,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlen: IV size * @ivlock: Mutex to serialize access to the IV * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation @@ -142,12 +154,14 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +unsigned int ivlen;struct mutex ivlock;size_t aead_assoclen;@@ -159,9 +173,13 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +int iiv;unsigned int len; }; +#define ALG_IIV_UNSET (-1) +#define ALG_IIV_DISABLE (0) +#define ALG_IIV_USE (1)int af_alg_register_type (const struct af_alg_type *type);int af_alg_unregister_type (const struct af_alg_type *type);@@ -255,7 +273,8 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); -int af_alg_get_iv (struct sock *sk); +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq);void af_alg_put_iv (struct sock *sk); +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /* _CRYPTO_IF_ALG_H */ include/uapi/linux/if_alg.h +4
-2@@ -37,7 +37,9 @@ struct af_alg_iv {#define ALG_SET_AEAD_AUTHSIZE 5/* Operations */ -#define ALG_OP_DECRYPT 0 -#define ALG_OP_ENCRYPT 1 +#define ALG_OP_DECRYPT 0x0 +#define ALG_OP_ENCRYPT 0x1 +#define ALG_OP_CIPHER_MASK 0xf +#define ALG_OP_INLINE_IV 0x10#endif /* _LINUX_IF_ALG_H */
Re: [PATCH v2 2/4] crypto: AF_ALG - inline IV support
Am Mittwoch, 7. Februar 2018, 14:54:08 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 4 lines
It's not a 'bug' as such, but this does currently break the crypto-perf test for aio and iiv in the libkcapi branch. We can perhaps make it more forgiving...
I just pushed an update for the IIV branch of libkcapi that handles it appropriately.
Quoted 3 lines
I suggest we let noop cases where we set IIV on when it is already on to not result in an error but to just be ignored.
I do not like the kernel to behave differently on one and the same flag. Though, the check could be more forgiving indeed: if the flag from user space is set again and identical to what we already have, we would not report an error. Only if it is different from what was already set, we report the error.
Quoted 15 lines
crypto/af_alg.c int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size, · line 882
Quoted 10 lines
crypto/af_alg.c +9
-0+ +/* +* IIV can only be enabled once with the first sendmsg call. +* This prevents a race in locking and unlocking the +* ctx->ivlock mutex. +*/ +if (ctx->iiv ==ALG_IIV_UNSET ) { + ctx->iiv = iiv; + }else if (iiv ==ALG_IIV_USE ) {So the issue in kcapi is that we are running a tight loop around a test case that does a certain number of aio messages. At the start of each run of that performance loop we reinitialize.
Quoted 8 lines
There is an optmisation in there though which avoids an accept call (which would have given a new context) if we have one available already.
The result is that we are reusing the context which already has this set.
So, can we be a little less restrictive and let things through if and only iff iiv =
ctx->iiv- i.e. the set is a noop?
Precisely, will do in the next version of the patch set.
Ciao
Stephan
PATCH v2 3/4 crypto: AF_ALG - allow driver to serialize IV access
In reply to Stephan MüllerThe mutex in AF_ALG to serialize access to the IV ensures full serialization of requests sent to the crypto driver.
However, the hardware may implement serialization to the IV such that preparation work without touching the IV can already happen while the IV is processed by another operation. This may speed up the AIO processing.
The following ASCII art demonstrates this.
AF_ALG mutex handling implements the following logic:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
NOTHING BLOCKING (lock mutex) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] |
| |
[unlock mutex] Nothing Blocking (lock mutex)
| |
Don't care beyond here. [BUILD / MAP HW DESCS]
|
[Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.A crypto driver may implement the following serialization:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
[BUILD / MAP HW DESCS] [BUILD / MAP HW DESCS]
| |
NOTHING BLOCKING IV in flight (enqueue sw queue)
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| |
Don't care beyond here. [Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.If the driver implements its own serialization (i.e. AF_ALG does not serialize the access to the IV), the crypto implementation must set the flag CRYPTO_ALG_SERIALIZES_IV_ACCESS.
--- crypto/af_alg.c | 13 ++++++++----- crypto/algif_aead.c | 1 + crypto/algif_skcipher.c | 1 + include/crypto/if_alg.h | 13 +++++++++++++ include/linux/crypto.h | 15 +++++++++++++++ 5 files changed, 38 insertions(+), 5 deletions(-)
crypto/af_alg.c +8 -5
@@ -1247,8 +1247,10 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)return 0 ;/* No inline IV, use ctx IV buffer and lock it */ -if (ctx->iiv ==ALG_IIV_DISABLE ) -return mutex_lock_interruptible (&ctx->ivlock); +if (ctx->iiv ==ALG_IIV_DISABLE ) { +return (ctx->lock_iv) ? +mutex_lock_interruptible (&ctx->ivlock) :0 ; + }/* Inline IV handling: There must be the IV data present. */ if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list))
@@ -1280,12 +1282,13 @@ void af_alg_put_iv(struct sock *sk)struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; -/* To resemble af_alg_get_iv, do not merge the two branches. */ if (!ctx->ivlen)return ; -if (ctx->iiv ==ALG_IIV_DISABLE ) -mutex_unlock (&ctx->ivlock); +if (ctx->iiv ==ALG_IIV_DISABLE ) { +if (ctx->lock_iv) +mutex_unlock (&ctx->ivlock); + } }EXPORT_SYMBOL_GPL (af_alg_put_iv);
crypto/algif_aead.c +1 -0
@@ -565,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->lock_iv =af_alg_lock_iv (crypto_aead_tfm (aead)); ctx->iiv =ALG_IIV_UNSET ; ctx->aead_assoclen =0 ;crypto_init_wait (&ctx->wait);
crypto/algif_skcipher.c +1 -0
@@ -368,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->lock_iv =af_alg_lock_iv (crypto_skcipher_tfm (tfm)); ctx->iiv =ALG_IIV_UNSET ;crypto_init_wait (&ctx->wait);
include/crypto/if_alg.h +13 -0
@@ -154,6 +154,7 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @lock_iv: Shall IV be locked? * @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */
@@ -173,6 +174,7 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +bool lock_iv;int iiv;unsigned int len;
@@ -251,6 +253,17 @@ static inline bool af_alg_readable(struct sock *sk)return PAGE_SIZE <=af_alg_rcvbuf (sk); } +/** +* Shall access to the ctx->iv be serialized using a mutex? +* +* @tfm TFM of to be used for cipher operation +* @return true => lock, false => do not lock +*/ +static inline bool af_alg_lock_iv (struct crypto_tfm *tfm) +{ +return !(crypto_tfm_alg_flags (tfm) &CRYPTO_ALG_SERIALIZES_IV_ACCESS ); +} +int af_alg_alloc_tsgl (struct sock *sk);unsigned int af_alg_count_tsgl (struct sock *sk,size_t bytes,size_t offset); void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst,
include/linux/crypto.h +15 -0
@@ -112,6 +112,16 @@*/ #define CRYPTO_ALG_OPTIONAL_KEY 0x00004000 +/* +* Set if the algorithm implementation provides its own serialization +* of IV accesses. This prevents the AF_ALG interface to serialize +* cipher requests. This flag allows the implementation to implement +* a more streamlined IV serialization logic, such as only serializing +* the submission of a request to hardware whereas the preparation steps +* for sending data to hardware are not serialized. +*/ +#define CRYPTO_ALG_SERIALIZES_IV_ACCESS 0x00008000 +/* * Transform masks and values (for crt_flags). */
@@ -674,6 +684,11 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm)return tfm->__crt_alg->cra_flags &CRYPTO_ALG_TYPE_MASK ; } +static inline u32 crypto_tfm_alg_flags (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags; +} +static inline unsigned int crypto_tfm_alg_blocksize (struct crypto_tfm *tfm) {return tfm->__crt_alg->cra_blocksize;
Signature · 1 line
2.14.3
PATCH v2 4/4 crypto: add CRYPTO_TFM_REQ_PARALLEL flag
In reply to Stephan MüllerCrypto drivers may implement a streamlined serialization support for AIO requests that is reported by the CRYPTO_TFM_REQ_PARALLEL flag to the crypto user. When the user decides that he wants to send multiple AIO requests concurrently and wants the crypto driver to handle the serialization, the caller has to set CRYPTO_TFM_REQ_PARALLEL to notify the crypto driver.
Only when this flag is enabled, the crypto driver shall apply its serialization logic for handling IV updates between requests. If this flag is not provided, the serialization logic shall not be applied by the driver as the caller decides that it does not need it (because no parallel AIO requests are sent) or that it performs its own serialization.
--- crypto/algif_aead.c | 15 ++++++++++++--- crypto/algif_skcipher.c | 15 ++++++++++++--- include/linux/crypto.h | 1 + 3 files changed, 25 insertions(+), 6 deletions(-)
crypto/algif_aead.c +12 -3
@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm); +struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE ) + tfm->crt_flags |=CRYPTO_TFM_REQ_PARALLEL ; + +return ret; } static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
crypto/algif_skcipher.c +12 -3
@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg, {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk); -struct crypto_skcipher *tfm = pask->private; -unsigned ivsize =crypto_skcipher_ivsize (tfm); +struct crypto_skcipher *skc = pask->private; +struct crypto_tfm *tfm =crypto_skcipher_tfm (skc); +unsigned int ivsize =crypto_skcipher_ivsize (skc); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE ) + tfm->crt_flags |=CRYPTO_TFM_REQ_PARALLEL ; + +return ret; } static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
include/linux/crypto.h +1 -0
@@ -133,6 +133,7 @@#define CRYPTO_TFM_REQ_WEAK_KEY 0x00000100#define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200#define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400 +#define CRYPTO_TFM_REQ_PARALLEL 0x00000800#define CRYPTO_TFM_RES_WEAK_KEY 0x00100000#define CRYPTO_TFM_RES_BAD_KEY_LEN 0x00200000#define CRYPTO_TFM_RES_BAD_KEY_SCHED 0x00400000
Signature · 1 line
2.14.3
Re: [PATCH v2 4/4] crypto: add CRYPTO_TFM_REQ_PARALLEL flag
Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan Müller:
Hi,
Quoted 14 lines
crypto/algif_aead.c +1
-2@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm);
crypto/algif_aead.c static int aead_sendmsg(struct socket *sock, struct · line 81
Quoted 10 lines
crypto/algif_aead.c +8
-1+struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE )
This should be ALG_IIV_DISABLE of course.
Quoted 19 lines
crypto/algif_aead.c +3
-0+ tfm->crt_flags |=CRYPTO_TFM_REQ_PARALLEL ; + +return ret; } static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,crypto/algif_skcipher.c +1
-2@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg, {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk); -struct crypto_skcipher *tfm = pask->private; -unsigned ivsize =crypto_skcipher_ivsize (tfm);
crypto/algif_skcipher.c static int skcipher_sendmsg(struct socket *sock, struct · line 57
Quoted 10 lines
crypto/algif_skcipher.c +8
-1+struct crypto_skcipher *skc = pask->private; +struct crypto_tfm *tfm =crypto_skcipher_tfm (skc); +unsigned int ivsize =crypto_skcipher_ivsize (skc); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE )
Dto.
I will wait for some more comments before sending a new patch.
Ciao
Stephan
Re: [PATCH v2 4/4] crypto: add CRYPTO_TFM_REQ_PARALLEL flag
Quoted 27 lines · On Wed, 7 Feb 2018 13:48:32 +0100 Stephan Mueller <[email protected]> wrote:
Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan Müller:
Hi,
Quoted 22 lines
crypto/algif_aead.c +8
-2@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm); +struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret;
Quoted 4 lines
crypto/algif_aead.c static int aead_sendmsg(struct socket *sock, struct · line 81
Quoted 2 lines
crypto/algif_aead.c +1
-1-return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE )This should be
ALG_IIV_DISABLEof course.
You say that, but my initial reading was that the core was requesting that the driver do things in parallel irrespective of whether the driver thought it was safe. So I would think this was correct.
Definitely needs some documentation or a clearer name.
Quoted 39 lines
Quoted 20 lines
crypto/algif_aead.c +3
-0+ tfm->crt_flags |=CRYPTO_TFM_REQ_PARALLEL ; + +return ret; } static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,crypto/algif_skcipher.c +1
-2@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg, {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk); -struct crypto_skcipher *tfm = pask->private; -unsigned ivsize =crypto_skcipher_ivsize (tfm);crypto/algif_skcipher.c static int skcipher_sendmsg(struct socket *sock, struct · line 57
Quoted 10 lines
crypto/algif_skcipher.c +8
-1+struct crypto_skcipher *skc = pask->private; +struct crypto_tfm *tfm =crypto_skcipher_tfm (skc); +unsigned int ivsize =crypto_skcipher_ivsize (skc); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE )Dto.
I will wait for some more comments before sending a new patch.
Ciao
Stephan
Re: [PATCH v2 4/4] crypto: add CRYPTO_TFM_REQ_PARALLEL flag
Am Mittwoch, 7. Februar 2018, 16:39:11 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 38 lines
On Wed, 7 Feb 2018 13:48:32 +0100
Quoted 30 lines · Stephan Mueller
Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan Müller:
Hi,
Quoted 26 lines
crypto/algif_aead.c +8
-2@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm); +struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret;Quoted 4 lines
crypto/algif_aead.c static int aead_sendmsg(struct socket *sock, struct
Quoted 2 lines
crypto/algif_aead.c +1
-1-return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE )This should be
ALG_IIV_DISABLEof course.
Quoted 6 lines
You say that, but my initial reading was that the core was requesting that the driver do things in parallel irrespective of whether the driver thought it was safe. So I would think this was correct.
Definitely needs some documentation or a clearer name.
How about:
ALG_IV_SERIAL_PROCESSING (was ALG_IIV_DISABLE)ALG_IV_PARALLEL_PROCESSING (was ALG_IIV_USE)
Ciao
Stephan
Re: [PATCH v2 4/4] crypto: add CRYPTO_TFM_REQ_PARALLEL flag
Quoted 47 lines · On Wed, 7 Feb 2018 16:43:10 +0100 Stephan Mueller <[email protected]> wrote:
Am Mittwoch, 7. Februar 2018, 16:39:11 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 43 lines
On Wed, 7 Feb 2018 13:48:32 +0100
Quoted 30 lines · Stephan Mueller
Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan Müller:
Hi,
Quoted 26 lines
crypto/algif_aead.c +8
-2@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm); +struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret;Quoted 4 lines
crypto/algif_aead.c static int aead_sendmsg(struct socket *sock, struct
Quoted 2 lines
crypto/algif_aead.c +1
-1-return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IIV_USE )This should be
ALG_IIV_DISABLEof course.You say that, but my initial reading was that the core was requesting that the driver do things in parallel irrespective of whether the driver thought it was safe. So I would think this was correct.
Quoted 7 lines
Quoted 1 line
Definitely needs some documentation or a clearer name.
How about:
ALG_IV_SERIAL_PROCESSING(wasALG_IIV_DISABLE)ALG_IV_PARALLEL_PROCESSING(wasALG_IIV_USE)
Actually those were fine on the basis that inline iv is obvious enough, it was CRYPTO_TFM_REQ_PARALLEL that was causing me confusion.
Sorry, wasn't terribly clear on that!
Jonathan
Quoted 4 lines
Ciao
Stephan
Re: [PATCH v2 4/4] crypto: add CRYPTO_TFM_REQ_PARALLEL flag
Am Mittwoch, 7. Februar 2018, 17:14:12 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 5 lines
Actually those were fine on the basis that inline iv is obvious enough, it was
CRYPTO_TFM_REQ_PARALLELthat was causing me confusion.Sorry, wasn't terribly clear on that!
Ok, I will add some comments around that new REQ flag.
Ciao
Stephan
Re: [PATCH v2 0/4] crypto: AF_ALG AIO improvements
In reply to Stephan MüllerQuoted 32 lines · Stephan Müller
Hi Herbert,
Herbert, the patch 1 is meant for stable. However, this patch as is only applies to the new
AF_ALGinterface implementation. Though, the issue goes back to the first implementation of AIO support. Shall I try prepare a patch for the oldAF_ALGimplementation as well?Changes from v1:
* integrate the inline IV and locking patch into one patch set
* reverse the order of lock context IV patch and inline IV patch -- the reason is to allow the first patch to be back-ported to stable
* mark the first patch (locking of the context IV) as applicable to stable as there is an existing inconsistency which was demonstrated by Harsh with the Chelsio driver vs the AES-NI driver
* modify the inline IV patch to have proper unlocking of the mutex in case of errors
* prevent locking if no IV is defined by cipher
* add a patch to allow crypto drivers to report whether they support serialization -- in this case the locking in
AF_ALGshall be disabled* add a patch to inform the crypto drivers that their serialization support should actually be enabled and used because
AF_ALGdoes not serialize the interdependent parallel AIO requests
Quoted 5 lines
* streamline the code in patch 1 and 2 slightly
I would like to ask the folks with real AIO hardware (Harsh, Jonathan) to test the patches. Especially, is the locking patch should be tested by Harsh as you have seen the issue with your hardware.
Sure I will test the patch and let you know.
Quoted 17 lines
Thanks.
Stephan Mueller (4):
crypto:AF_ALGAIO - lock context IV
crypto:AF_ALG- inline IV support
crypto:AF_ALG- allow driver to serialize IV access
crypto: addCRYPTO_TFM_REQ_PARALLELflagcrypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++- crypto/algif_aead.c | 86 +++++++++++++++++--------------- crypto/algif_skcipher.c | 38 ++++++++++---- include/crypto/if_alg.h | 37 ++++++++++++++ include/linux/crypto.h | 16 ++++++ include/uapi/linux/if_alg.h | 6 ++- 6 files changed, 249 insertions(+), 53 deletions(-)
Re: [PATCH v2 0/4] crypto: AF_ALG AIO improvements
In reply to Stephan MüllerQuoted 32 lines · On Wed, 7 Feb 2018 08:42:59 +0100 Stephan Müller <[email protected]> wrote:
Hi Herbert,
Herbert, the patch 1 is meant for stable. However, this patch as is only applies to the new
AF_ALGinterface implementation. Though, the issue goes back to the first implementation of AIO support. Shall I try prepare a patch for the oldAF_ALGimplementation as well?Changes from v1:
* integrate the inline IV and locking patch into one patch set
* reverse the order of lock context IV patch and inline IV patch -- the reason is to allow the first patch to be back-ported to stable
* mark the first patch (locking of the context IV) as applicable to stable as there is an existing inconsistency which was demonstrated by Harsh with the Chelsio driver vs the AES-NI driver
* modify the inline IV patch to have proper unlocking of the mutex in case of errors
* prevent locking if no IV is defined by cipher
* add a patch to allow crypto drivers to report whether they support serialization -- in this case the locking in
AF_ALGshall be disabled* add a patch to inform the crypto drivers that their serialization support should actually be enabled and used because
AF_ALGdoes not serialize the interdependent parallel AIO requests
Quoted 6 lines
* streamline the code in patch 1 and 2 slightly
I would like to ask the folks with real AIO hardware (Harsh, Jonathan) to test the patches. Especially, is the locking patch should be tested by Harsh as you have seen the issue with your hardware.
I've updated my driver to take advantage of this and it is working pretty much the same as it was with the dirty hacks I was running before.
So Tested-by: Jonathan Cameron <[email protected]> with that minor change to sensitivity to having it set to IIV multiple times.
Hmm. Naming of the parallel request probably needs some thought though. Will reply to that patch.
Quoted 16 lines
Thanks.
Stephan Mueller (4):
crypto:AF_ALGAIO - lock context IV
crypto:AF_ALG- inline IV support
crypto:AF_ALG- allow driver to serialize IV access
crypto: addCRYPTO_TFM_REQ_PARALLELflagcrypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++- crypto/algif_aead.c | 86 +++++++++++++++++--------------- crypto/algif_skcipher.c | 38 ++++++++++---- include/crypto/if_alg.h | 37 ++++++++++++++ include/linux/crypto.h | 16 ++++++ include/uapi/linux/if_alg.h | 6 ++- 6 files changed, 249 insertions(+), 53 deletions(-)
v3 cover crypto: AF_ALG AIO improvements
In reply to Stephan MüllerHi,
Herbert, the patch 1 is meant for stable. However, this patch as is only applies to the new AF_ALG interface implementation. Though, the issue goes back to the first implementation of AIO support. Shall I try prepare a patch for the old AF_ALG implementation as well?
Changes from v2:
* rename ALG_IIV flags into ALG_IV_...
* rename CRYPTO_TFM_REQ_PARALLEL into CRYPTO_TFM_REQ_IV_SERIALIZE
* fix branch in patch 4 to add CRYPTO_TFM_REQ_IV_SERIALIZE flag when ctx->iiv == ALG_IV_SERIAL_PROCESSING
* fix patch description of patch 4
Stephan Mueller (4):
crypto: AF_ALG AIO - lock context IV
crypto: AF_ALG - inline IV support
crypto: AF_ALG - allow driver to serialize IV access
crypto: add CRYPTO_TFM_REQ_IV_SERIALIZE flag
crypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++- crypto/algif_aead.c | 86 +++++++++++++++++--------------- crypto/algif_skcipher.c | 38 ++++++++++---- include/crypto/if_alg.h | 37 ++++++++++++++ include/linux/crypto.h | 16 ++++++ include/uapi/linux/if_alg.h | 6 ++- 6 files changed, 249 insertions(+), 53 deletions(-)
Signature · 1 line
2.14.3
PATCH v3 1/4 crypto: AF_ALG AIO - lock context IV
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in the AF_ALG context that is unique to one particular AF_ALG socket. Note the analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the AF_ALG socket context is registered with the new request and the cipher operation is invoked.
You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
As the AF_ALG interface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
--- crypto/af_alg.c | 31 +++++++++++++++++++++++++++++++ crypto/algif_aead.c | 20 +++++++++++++------- crypto/algif_skcipher.c | 12 +++++++++--- include/crypto/if_alg.h | 5 +++++ 4 files changed, 58 insertions(+), 10 deletions(-)
crypto/af_alg.c +31 -0
@@ -1051,6 +1051,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)struct kiocb *iocb = areq->iocb;unsigned int resultlen; +af_alg_put_iv (sk); +/* Buffer size written by crypto operation. */ resultlen = areq->outlen;
@@ -1175,6 +1177,35 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, }EXPORT_SYMBOL_GPL (af_alg_get_rsgl); +/** +* af_alg_get_iv +* +* @sk [in] AF_ALG socket +* @return 0 on success, < 0 on error +*/ +int af_alg_get_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +return mutex_lock_interruptible (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_get_iv); + +/** +* af_alg_put_iv - release lock on IV in case CTX IV is used +* +* @sk [in] AF_ALG socket +*/ +void af_alg_put_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +mutex_unlock (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_put_iv); +static int __init af_alg_init (void ) {int err =proto_register (&alg_proto,0 );
crypto/algif_aead.c +13 -7
@@ -159,10 +159,14 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); + err =af_alg_get_iv (sk); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq, outlen, &usedpages);if (err) - goto free; + goto unlock;/* * Ensure output buffer is sufficiently large. If the caller provides
@@ -176,7 +180,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (used < less) { err = -EINVAL ; - goto free; + goto unlock; } used -= less; outlen -= less;
@@ -197,7 +201,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, }if (processed && !tsgl_src) { err = -EFAULT ; - goto free; + goto unlock; }/*
@@ -230,7 +234,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, err =crypto_aead_copy_sgl (null_tfm, tsgl_src, areq->first_rsgl.sgl.sg, processed);if (err) - goto free; + goto unlock;af_alg_pull_tsgl (sk, processed,NULL ,0 ); }else {/*
@@ -248,7 +252,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, err =crypto_aead_copy_sgl (null_tfm, tsgl_src, areq->first_rsgl.sgl.sg, outlen);if (err) - goto free; + goto unlock;/* Create TX SGL for tag and chain it to RX SGL. */ areq->tsgl_entries = af_alg_count_tsgl(sk, processed,
@@ -260,7 +264,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, GFP_KERNEL);if (!areq->tsgl) { err = -ENOMEM ; - goto free; + goto unlock; }sg_init_table (areq->tsgl, areq->tsgl_entries);
@@ -316,7 +320,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } - +unlock: +af_alg_put_iv (sk); free:af_alg_free_resources (areq);
@@ -562,6 +567,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)return -ENOMEM ; }memset (ctx->iv,0 , ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;
crypto/algif_skcipher.c +9 -3
@@ -77,10 +77,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); + err =af_alg_get_iv (sk); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq,-1 , &len);if (err) - goto free; + goto unlock;/* Process only as much RX buffers for which we have TX data */ if (len > ctx->used)
@@ -104,7 +108,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, GFP_KERNEL);if (!areq->tsgl) { err = -ENOMEM ; - goto free; + goto unlock; }sg_init_table (areq->tsgl, areq->tsgl_entries);af_alg_pull_tsgl (sk, len, areq->tsgl,0 );
@@ -146,7 +150,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } - +unlock: +af_alg_put_iv (sk); free:af_alg_free_resources (areq);
@@ -353,6 +358,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) }memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;
include/crypto/if_alg.h +5 -0
@@ -19,6 +19,7 @@#include <linux/scatterlist.h> #include <linux/types.h> #include <linux/atomic.h> +#include <linux/mutex.h> #include <net/sock.h> #include <crypto/aead.h>
@@ -127,6 +128,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlock: Mutex to serialize access to the IV * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation * @used: TX bytes sent to kernel. This variable is used to
@@ -146,6 +148,7 @@ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +struct mutex ivlock;size_t aead_assoclen;struct crypto_wait wait;
@@ -252,5 +255,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); +int af_alg_get_iv (struct sock *sk); +void af_alg_put_iv (struct sock *sk);#endif /* _CRYPTO_IF_ALG_H */
Signature · 1 line
2.14.3
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Quoted 54 lines · Stephan Müller
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
Quoted 7 lines
As the
AF_ALGinterface is used by user space, a mutex provides the serialization which puts the caller to sleep in case a previous IOCB processing is not yet finished.If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation of processing arriving requests ensures that the blocked IOCBs are unblocked in the right order.
Hi Stephen,
Patch set is working fine with chelsio Driver.
Do we really need IV locking mechanism for AEAD algo because AEAD algo's don't support Partial mode operation and Driver are not updating(atleast Chelsio) IV's on AEAD request completions.
Quoted 227 lines
CC <[email protected]> #4.14Fixes e870456d8e7c8 ("crypto: algif_skcipher - overhaul memory management")Fixes d887c52d6ae43 ("crypto: algif_aead - overhaul memory management")Signed-off-by Stephan Mueller <[email protected]>--- crypto/af_alg.c | 31 +++++++++++++++++++++++++++++++ crypto/algif_aead.c | 20 +++++++++++++------- crypto/algif_skcipher.c | 12 +++++++++--- include/crypto/if_alg.h | 5 +++++ 4 files changed, 58 insertions(+), 10 deletions(-)crypto/af_alg.c +31
-0@@ -1051,6 +1051,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)struct kiocb *iocb = areq->iocb;unsigned int resultlen; +af_alg_put_iv (sk); +/* Buffer size written by crypto operation. */ resultlen = areq->outlen;@@ -1175,6 +1177,35 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, }EXPORT_SYMBOL_GPL (af_alg_get_rsgl); +/** +* af_alg_get_iv +* +* @sk [in] AF_ALG socket +* @return 0 on success, < 0 on error +*/ +int af_alg_get_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +return mutex_lock_interruptible (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_get_iv); + +/** +* af_alg_put_iv - release lock on IV in case CTX IV is used +* +* @sk [in] AF_ALG socket +*/ +void af_alg_put_iv (struct sock *sk) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; + +mutex_unlock (&ctx->ivlock); +} +EXPORT_SYMBOL_GPL (af_alg_put_iv); +static int __init af_alg_init (void ) {int err =proto_register (&alg_proto,0 );crypto/algif_aead.c +13
-7@@ -159,10 +159,14 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); + err =af_alg_get_iv (sk); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq, outlen, &usedpages);if (err) - goto free; + goto unlock;/* * Ensure output buffer is sufficiently large. If the caller provides @@ -176,7 +180,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (used < less) { err = -EINVAL ; - goto free; + goto unlock; } used -= less; outlen -= less;@@ -197,7 +201,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, }if (processed && !tsgl_src) { err = -EFAULT ; - goto free; + goto unlock; }/* @@ -230,7 +234,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, err =crypto_aead_copy_sgl (null_tfm, tsgl_src, areq->first_rsgl.sgl.sg, processed);if (err) - goto free; + goto unlock;af_alg_pull_tsgl (sk, processed,NULL ,0 ); }else {/* @@ -248,7 +252,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, err =crypto_aead_copy_sgl (null_tfm, tsgl_src, areq->first_rsgl.sgl.sg, outlen);if (err) - goto free; + goto unlock;/* Create TX SGL for tag and chain it to RX SGL. */ areq->tsgl_entries = af_alg_count_tsgl(sk, processed,@@ -260,7 +264,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, GFP_KERNEL);if (!areq->tsgl) { err = -ENOMEM ; - goto free; + goto unlock; }sg_init_table (areq->tsgl, areq->tsgl_entries);@@ -316,7 +320,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } - +unlock: +af_alg_put_iv (sk); free:af_alg_free_resources (areq);@@ -562,6 +567,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)return -ENOMEM ; }memset (ctx->iv,0 , ivlen); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;crypto/algif_skcipher.c +9
-3@@ -77,10 +77,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); + err =af_alg_get_iv (sk); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq,-1 , &len);if (err) - goto free; + goto unlock;/* Process only as much RX buffers for which we have TX data */ if (len > ctx->used)@@ -104,7 +108,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, GFP_KERNEL);if (!areq->tsgl) { err = -ENOMEM ; - goto free; + goto unlock; }sg_init_table (areq->tsgl, areq->tsgl_entries);af_alg_pull_tsgl (sk, len, areq->tsgl,0 );@@ -146,7 +150,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, &ctx->wait); } - +unlock: +af_alg_put_iv (sk); free:af_alg_free_resources (areq);@@ -353,6 +358,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) }memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;include/crypto/if_alg.h +5
-0@@ -19,6 +19,7 @@#include <linux/scatterlist.h> #include <linux/types.h> #include <linux/atomic.h> +#include <linux/mutex.h> #include <net/sock.h> #include <crypto/aead.h> @@ -127,6 +128,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlock: Mutex to serialize access to the IV * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation * @used: TX bytes sent to kernel. This variable is used to @@ -146,6 +148,7 @@ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +struct mutex ivlock;size_t aead_assoclen;struct crypto_wait wait;@@ -252,5 +255,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); +int af_alg_get_iv (struct sock *sk); +void af_alg_put_iv (struct sock *sk);#endif /* _CRYPTO_IF_ALG_H */
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 2 lines
Patch set is working fine with chelsio Driver.
Thank you.
Quoted 3 lines
Do we really need IV locking mechanism for AEAD algo because AEAD algo's don't support Partial mode operation and Driver are not updating(atleast Chelsio) IV's on AEAD request completions.
Yes, I think we would need it. It is technically possible to have multiple IOCBs for AEAD ciphers. Even though your implementation may not write the IV back, others may do that. At least I do not see a guarantee that the IV is *not* written back by a driver.
In case your driver does not write the IV back and thus does not need to serialize, the driver can report CRYPTO_ALG_SERIALIZES_IV_ACCESS. In this case, the higher level functions would not serialize as the driver serializes the requests (or the driver deems it appropriate that no serialization is needed as is the case with your driver).
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Quoted 7 lines · Stephan Mueller
Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 1 line
Patch set is working fine with chelsio Driver.
Thank you.
Quoted 7 lines
Quoted 3 lines
Do we really need IV locking mechanism for AEAD algo because AEAD algo's don't support Partial mode operation and Driver are not updating(atleast Chelsio) IV's on AEAD request completions.
Yes, I think we would need it. It is technically possible to have multiple IOCBs for AEAD ciphers. Even though your implementation may not write the IV back, others may do that. At least I do not see a guarantee that the IV is *not* written back by a driver.
There is no use of writing IV back in AEAD algo till Framework starts supporting Partial mode. Even if Driver starts updating IV for AEAD, Multiple IOCB's in both cases will yield wrong results only.
Case 1 : If we have AEAD IV serialization applied, Encryption will be wrong if same IV gets used.
Case 2: If we do not have IV serialization for AEAD. Encryption will be fine but user will have multiple Authentication tag (that too with final block processed). Its like 2nd Block encryption is based on IV received from 1st block and Authentication Tag value is based on 2nd block content only.
Quoted 11 lines
In case your driver does not write the IV back and thus does not need to serialize, the driver can report
CRYPTO_ALG_SERIALIZES_IV_ACCESS. In this case, the higher level functions would not serialize as the driver serializes the requests (or the driver deems it appropriate that no serialization is needed as is the case with your driver).Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Am Donnerstag, 15. Februar 2018, 06:30:36 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 13 lines · Stephan Mueller
Quoted 12 lines
Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 1 line
Patch set is working fine with chelsio Driver.
Thank you.
Quoted 3 lines
Do we really need IV locking mechanism for AEAD algo because AEAD algo's don't support Partial mode operation and Driver are not updating(atleast Chelsio) IV's on AEAD request completions.
Quoted 7 lines
Quoted 4 lines
Yes, I think we would need it. It is technically possible to have multiple IOCBs for AEAD ciphers. Even though your implementation may not write the IV back, others may do that. At least I do not see a guarantee that the IV is *not* written back by a driver.
There is no use of writing IV back in AEAD algo till Framework starts supporting Partial mode.
I agree.
Quoted 2 lines
Even if Driver starts updating IV for AEAD, Multiple IOCB's in both cases will yield wrong results only.
This would only be the case if the driver would not implicitly or explicitly serialize the requests.
Quoted 3 lines
Case 1 : If we have AEAD IV serialization applied, Encryption will be wrong if same IV gets used.
Agreed.
Quoted 5 lines
Case 2: If we do not have IV serialization for
AEAD. Encryption will be fine but user will have multiple Authentication tag (that too with final block processed). Its like 2nd Block encryption is based on IV received from 1st block and Authentication Tag value is based on 2nd block content only.
Agreed.
But are we sure that all drivers behave correctly? Before you notified us of the issue, I was not even aware of the fact that this serialization may not be done in the driver. And we only have seen that issue with AF_ALG where we test for multiple concurrent AIO operations.
Besides, when we do not have the locking for AEAD, what would we gain: one less lock to take vs. guarantee that the AEAD operation is always properly serialized.
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Quoted 36 lines · Stephan Mueller
Am Donnerstag, 15. Februar 2018, 06:30:36 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 11 lines · Stephan Mueller
Quoted 10 lines
Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 1 line
Patch set is working fine with chelsio Driver.
Thank you.
Quoted 3 lines
Do we really need IV locking mechanism for AEAD algo because AEAD algo's don't support Partial mode operation and Driver are not updating(atleast Chelsio) IV's on AEAD request completions.
Quoted 6 lines
Quoted 4 lines
Yes, I think we would need it. It is technically possible to have multiple IOCBs for AEAD ciphers. Even though your implementation may not write the IV back, others may do that. At least I do not see a guarantee that the IV is *not* written back by a driver.
There is no use of writing IV back in AEAD algo till Framework starts supporting Partial mode.
I agree.
Quoted 2 lines
Even if Driver starts updating IV for AEAD, Multiple IOCB's in both cases will yield wrong results only.
This would only be the case if the driver would not implicitly or explicitly serialize the requests.
Quoted 2 lines
Case 1 : If we have AEAD IV serialization applied, Encryption will be wrong if same IV gets used.
Agreed.
Quoted 5 lines
Case 2: If we do not have IV serialization for
AEAD. Encryption will be fine but user will have multiple Authentication tag (that too with final block processed). Its like 2nd Block encryption is based on IV received from 1st block and Authentication Tag value is based on 2nd block content only.
Quoted 6 lines
Agreed.
But are we sure that all drivers behave correctly? Before you notified us of the issue, I was not even aware of the fact that this serialization may not be done in the driver. And we only have seen that issue with
AF_ALGwhere we test for multiple concurrent AIO operations.
I am sure other H/W will have similar problem, It's just that we tested it first.
Quoted 4 lines
Besides, when we do not have the locking for AEAD, what would we gain: one less lock to take vs. guarantee that the AEAD operation is always properly serialized.
Even after guarantee of serialization, In the end we will get wrong result as mentioned above. which destination side cannot decrypt it. What I feel is scenario of sending 2 of more IOCB in case of AEAD itself is wrong. We should not allow this type of requests for AEAD. Can you think of any use case it is going to solve?
Can receiver decrypt(with 2 IOCB) the same request successfully without knowing sender has done the operation in 2 request with size "x" each?
Quoted 5 lines
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 3 lines
Even after guarantee of serialization, In the end we will get wrong result as mentioned above. which destination side cannot decrypt it. What I feel is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
Without the inline IV handling, I would concur.
Quoted 2 lines
We
should not allow this type of requests for AEAD.
"Not allow" as in "technically block"? As a user would only shoot itself when he does that not knowing the consequences, I am not in favor of such an artificial block.
Quoted 2 lines
Can you think of any use
case it is going to solve?
Well, I could fathom a use case of this. In FIPS 140-2 (yes, a term not well received by some here), NIST insists for GCM that the IV is handled by the cryptographic implementation.
So, when using GCM for TLS, for example, the GCM implementation would know a bit about how the IV is updated as a session ID. I.e. after the end of one AEAD operation, the IV is written back but modified such to comply with the rules of some higher level proto. Thus, if such a scenarios is implemented by a driver here, multiple IOCBs could be used with such "TLSified" GCM, for example.
And such "TLSification" could be as simple as implementing an IV generator that can be used with every (AEAD) cipher implementation.
Quoted 4 lines
Can receiver decrypt(with 2 IOCB) the same request successfully without knowing sender has done the operation in 2 request with size "x" each?
Quoted 2 lines
Ciao
Stephan
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Quoted 8 lines · Stephan Mueller
Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 3 lines
Even after guarantee of serialization, In the end we will get wrong result as mentioned above. which destination side cannot decrypt it. What I feel is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
Without the inline IV handling, I would concur.
Even with Inline IV, We will have 2 Auth Tag. can we authenticate the data with 2 Auth Tags?
Quoted 6 lines
Quoted 3 lines
We
should not allow this type of requests for AEAD."Not allow" as in "technically block"? As a user would only shoot itself when he does that not knowing the consequences, I am not in favor of such an artificial block.
Agreed, It may not be right thing to do, but if we allowed it, What he will do with Auth( each processed with final Block) tags received in each request.
I personally feels AEAD IV serialization logic is incomplete without partial tag support.
Quoted 26 lines
Quoted 2 lines
Can you think of any use
case it is going to solve?Well, I could fathom a use case of this. In FIPS 140-2 (yes, a term not well received by some here), NIST insists for GCM that the IV is handled by the cryptographic implementation.
So, when using GCM for TLS, for example, the GCM implementation would know a bit about how the IV is updated as a session ID. I.e. after the end of one AEAD operation, the IV is written back but modified such to comply with the rules of some higher level proto. Thus, if such a scenarios is implemented by a driver here, multiple IOCBs could be used with such "TLSified" GCM, for example.
And such "TLSification" could be as simple as implementing an IV generator that can be used with every (AEAD) cipher implementation.
Quoted 4 lines
Can receiver decrypt(with 2 IOCB) the same request successfully without knowing sender has done the operation in 2 request with size "x" each?
Quoted 2 lines
Ciao
StephanCiao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Am Donnerstag, 15. Februar 2018, 12:38:12 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 10 lines · Stephan Mueller
Quoted 9 lines
Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 4 lines
Even after guarantee of serialization, In the end we will get wrong result
as mentioned above. which destination side cannot decrypt it. What I feel is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
Quoted 4 lines
Quoted 1 line
Without the inline IV handling, I would concur.
Even with Inline IV, We will have 2 Auth Tag. can we authenticate the data with 2 Auth Tags?
The AAD and the tag are both sent to the kernel like in the inline IV case as part of the data via sendmsg.
Thus, when you use inline IV support, an entire self-contained AEAD cipher operation can be defined with one IOCB. Thus, if you have multiple IOCBs, inline IV allow fully parallel execution of different AEAD requests.
See the following kernel comment:
/*
* Encryption operation - The in-place cipher operation is
* achieved by the following operation:
*
* TX SGL: AAD || PT
* | |
* | copy |
* v v
* RX SGL: AAD || PT || Tag
/*
* Decryption operation - To achieve an in-place cipher
* operation, the following SGL structure is used:
*
* TX SGL: AAD || CT || Tag
* | | ^
* | copy | | Create SGL link.
* v v |
* RX SGL: AAD || CT ----+
*/
Note, the TX SGL is what the caller submitted via sendmsg and the RX SGL is the data the caller obtains via recvmsg.
Hence, in inline IV support, the caller sends the following:
encryption: IV || AAD || PT
decryption: IV || AAD || CT || Tag
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
We
should not allow this type of requests for AEAD."Not allow" as in "technically block"? As a user would only shoot itself when he does that not knowing the consequences, I am not in favor of such an artificial block.
Agreed, It may not be right thing to do, but if we allowed it, What he will do with Auth( each processed with final Block) tags received in each request.
Each tag is returned as part of the recvmsg data. Thus, AEAD cipher operations can commence fully parallel if inline IV handling is used.
Quoted 3 lines
I personally feels AEAD IV serialization logic is incomplete without partial tag support.
Could you please elaborate what you mean with "partial tag" support?
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Quoted 60 lines · Stephan Mueller
Am Donnerstag, 15. Februar 2018, 12:38:12 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 5 lines · Stephan Mueller
Quoted 4 lines
Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Even after guarantee of serialization, In the end we will get wrong result
as mentioned above. which destination side cannot decrypt it. What I feel is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.Without the inline IV handling, I would concur.
Even with Inline IV, We will have 2 Auth Tag. can we authenticate the data with 2 Auth Tags?
The AAD and the tag are both sent to the kernel like in the inline IV case as part of the data via sendmsg.
Thus, when you use inline IV support, an entire self-contained AEAD cipher operation can be defined with one IOCB. Thus, if you have multiple IOCBs, inline IV allow fully parallel execution of different AEAD requests.
See the following kernel comment:
/*
* Encryption operation - The in-place cipher operation is
* achieved by the following operation:
*
* TX SGL: AAD || PT
* | |
* | copy |
* v v
* RX SGL: AAD || PT || Tag/*
* Decryption operation - To achieve an in-place cipher
* operation, the following SGL structure is used:
*
* TX SGL: AAD || CT || Tag
* | | ^
* | copy | | Create SGL link.
* v v |
* RX SGL: AAD || CT ----+
*/Note, the TX SGL is what the caller submitted via sendmsg and the RX SGL is the data the caller obtains via recvmsg.
Hence, in inline IV support, the caller sends the following:
encryption: IV || AAD || PT
decryption: IV || AAD || CT || Tag
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
We
should not allow this type of requests for AEAD."Not allow" as in "technically block"? As a user would only shoot itself when he does that not knowing the consequences, I am not in favor of such an artificial block.
Quoted 8 lines
Quoted 3 lines
Agreed, It may not be right thing to do, but if we allowed it, What he will do with Auth( each processed with final Block) tags received in each request.
Each tag is returned as part of the recvmsg data. Thus, AEAD cipher operations can commence fully parallel if inline IV handling is used.
Quoted 2 lines
I personally feels AEAD IV serialization logic is incomplete without partial tag support.
Could you please elaborate what you mean with "partial tag" support?
Here is the catch, Calculation of tag depends on total payload length atleast for shaX, gcm,ccm mode on which I have worked.
If we take an example of shaX. It appends 1 special block at the end of user data which includes total input length in bit. Refer "sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2 IOCB of 16 bytes each.
Expected result : 32 encrypted bytes + sha auth tag considering length 32 bytes.
What we will get : 16 bytes + sha auth tag considering length 16 bytes + 16 encrypted bytes + another sha tag considering 16 bytes.
Quoted 5 lines
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Am Donnerstag, 15. Februar 2018, 13:45:53 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 5 lines
Quoted 1 line
Could you please elaborate what you mean with "partial tag" support?
Here is the catch, Calculation of tag depends on total payload length atleast for shaX, gcm,ccm mode on which I have worked.
Quoted 7 lines
If we take an example of shaX. It appends 1 special block at the end of user data which includes total input length in bit. Refer "
sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2 IOCB of 16 bytes each. Expected result : 32 encrypted bytes + sha auth tag considering length 32 bytes. What we will get : 16 bytes + sha auth tag considering length 16 bytes + 16 encrypted bytes + another sha tag considering 16 bytes.
As AF_ALG for AEAD is implemented, there is no stream support where the hash is calculated at the end. This is even not supported in the current AEAD API of the kernel crypto API as far as I see. The only "stream-like" support is that you can invoke multiple separate sendmsg calls to provide the input data for the AEAD. But once you call recvmsg, the ciphertext and the tag is calculated and thus the recvmsg is akin to a hash_final operation.
Complete parallel execution of multiple independent AEAD cipher operations with AIO is possible using the inline IV as suggested in the patch set.
The locking of the ctx->iv is only there to support a driver not to whack the IV buffer. However, if others also agree that the ctx->ivlock should not be taken in the AEAD code path because a potential whacking the IV buffer by a driver does not need to be prevented, I would change the patch.
Ciao
Stephan
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
Quoted 15 lines · Stephan Mueller
Am Donnerstag, 15. Februar 2018, 13:45:53 CET schrieb Harsh Jain:
Quoted 5 lines
Quoted 1 line
Could you please elaborate what you mean with "partial tag" support?
Here is the catch, Calculation of tag depends on total payload length atleast for shaX, gcm,ccm mode on which I have worked.
Quoted 7 lines
If we take an example of shaX. It appends 1 special block at the end of user data which includes total input length in bit. Refer "
sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2 IOCB of 16 bytes each. Expected result : 32 encrypted bytes + sha auth tag considering length 32 bytes. What we will get : 16 bytes + sha auth tag considering length 16 bytes + 16 encrypted bytes + another sha tag considering 16 bytes.
Quoted 6 lines
As
AF_ALGfor AEAD is implemented, there is no stream support where the hash is calculated at the end. This is even not supported in the current AEAD API of the kernel crypto API as far as I see. The only "stream-like" support is that you can invoke multiple separate sendmsg calls to provide the input data for the AEAD. But once you call recvmsg, the ciphertext and the tag is calculated and thus the recvmsg is akin to ahash_finaloperation.
If you follow Bernstein's protocol design philosophy, then messages should be no larger than about 4K in size. From
https://nacl.cr.yp.to/valid.html:
This is one of several reasons [1] that callers should (1) split all data into packets sent through the network; (2) put a small global limit on packet length; and (3) separately encrypt and authenticate each packet.
With the [1] link being
https://groups.google.com/forum/#!original/boring-crypto/BpUmNMXKMYQ/EEwAIeQdjacJ
Jeff
Re: [PATCH v3 1/4] crypto: AF_ALG AIO - lock context IV
In reply to Stephan MuellerAm Donnerstag, 15. Februar 2018, 14:04:53 CET schrieb Stephan Mueller:
Hi Stephan,
Quoted 26 lines
Am Donnerstag, 15. Februar 2018, 13:45:53 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 5 lines
Quoted 1 line
Could you please elaborate what you mean with "partial tag" support?
Here is the catch, Calculation of tag depends on total payload length atleast for shaX, gcm,ccm mode on which I have worked.
Quoted 8 lines
If we take an example of shaX. It appends 1 special block at the end of user data which includes total input length in bit. Refer "
sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2 IOCB
of 16 bytes each. Expected result : 32 encrypted bytes + sha auth tag considering length 32 bytes. What we will get : 16 bytes + sha auth tag considering length 16 bytes + 16 encrypted bytes + another sha tag considering 16 bytes.As
AF_ALGfor AEAD is implemented, there is no stream support where the hash is calculated at the end. This is even not supported in the current AEAD API of the kernel crypto API as far as I see. The only "stream-like" support is that you can invoke multiple separate sendmsg calls to provide the input data for the AEAD. But once you call recvmsg, the ciphertext and the tag is calculated and thus the recvmsg is akin to ahash_finaloperation.
Quoted 7 lines
Complete parallel execution of multiple independent AEAD cipher operations with AIO is possible using the inline IV as suggested in the patch set.
The locking of the
ctx->ivis only there to support a driver not to whack the IV buffer. However, if others also agree that thectx->ivlockshould not be taken in the AEAD code path because a potential whacking the IV buffer by a driver does not need to be prevented, I would change the patch.
I was about to change the implementation. However, I already found an implementation where the driver changes the IV of the request buffer:crypto_ccm_init_crypt modifies the original buffer of the IV. I expect that there are more.
Therefore, I would think that keeping the patch set as it is right now is the right thing to do. Thus, we should cover the AEAD ciphers with the lock and thus serialize the AEAD request. This guarantees that the IV buffer is at least not mangled while the cipher operation is ongoing.
Again, if a particular AEAD driver guarantees the IV is not mangled, it can very well set CRYPTO_ALG_SERIALIZES_IV_ACCESS in its flag set. This would imply the lock is not set.
Ciao
Stephan
PATCH v3 2/4 crypto: AF_ALG - inline IV support
In reply to Stephan MüllerThe kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in the AF_ALG context that is unique to one particular AF_ALG socket. Note the analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.
As of now, the individual IOCBs are serialized with respect to the IV handling. This implies that the kernel does not perform a truly parallel invocation of the cipher implementations. However, if the user wants to perform cryptographic operations on multiple IOCBs where each IOCB is truly independent from the other, parallel invocations are possible. This would require that each IOCB provides its own IV to ensure true separation of the IOCBs.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the AF_ALG interface treats the ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.
If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The inline IV handling support is only allowed to be enabled during the first sendmsg call for a context. Any subsequent sendmsg calls are not allowed to change the setting of the inline IV handling (either enable or disable it) as this would open up a race condition with the locking and unlocking of the ctx->ivlock mutex.
The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that ctx->used is updated. Thus, the ctx->used access in _aead_recvmsg must be moved below the IV gathering.
The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
--- crypto/af_alg.c | 93 ++++++++++++++++++++++++++++++++++++++++++--- crypto/algif_aead.c | 58 ++++++++++++---------------- crypto/algif_skcipher.c | 12 +++--- include/crypto/if_alg.h | 21 +++++++++- include/uapi/linux/if_alg.h | 6 ++- 5 files changed, 143 insertions(+), 47 deletions(-)
crypto/af_alg.c +88 -5
@@ -14,6 +14,7 @@#include <linux/atomic.h> #include <crypto/if_alg.h> +#include <crypto/scatterwalk.h> #include <linux/crypto.h> #include <linux/init.h> #include <linux/kernel.h>
@@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,struct af_alg_control con = {};long copied =0 ;bool enc =0 ; +int iiv =ALG_IV_SERIAL_PROCESSING ;bool init =0 ;int err =0 ;
@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return err; init =1 ; -switch (con.op) { +switch (con.op &ALG_OP_CIPHER_MASK ) {case ALG_OP_ENCRYPT : enc =1 ;break ;
@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return -EINVAL ; } +if (con.op &ALG_OP_INLINE_IV ) + iiv =ALG_IV_PARALLEL_PROCESSING ; +if (con.iv && con.iv->ivlen != ivsize)return -EINVAL ; }
@@ -866,6 +871,19 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,if (init) { ctx->enc = enc; + +/* +* IIV can only be enabled once with the first sendmsg call. +* This prevents a race in locking and unlocking the +* ctx->ivlock mutex. +*/ +if (ctx->iiv ==ALG_IV_PROCESSING_UNDEF ) { + ctx->iiv = iiv; + }else if (ctx->iiv != iiv) { + err = -EINVAL ; + goto unlock; + } +if (con.iv)memcpy (ctx->iv, con.iv->iv, ivsize);
@@ -1031,6 +1049,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)struct sock *sk = areq->sk;af_alg_free_areq_sgls (areq); +if (areq->ivlen) +sock_kfree_s (sk, areq->iv, areq->ivlen);sock_kfree_s (sk, areq, areq->areqlen); }EXPORT_SYMBOL_GPL (af_alg_free_resources);
@@ -1178,17 +1198,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,EXPORT_SYMBOL_GPL (af_alg_get_rsgl);/** -* af_alg_get_iv +* af_alg_get_tsgl_sg - get first SG entry with data from TX SGL +* @ctx [in] AF_ALG context with TX sgl +* @return pointer to SG with data or NULL if none found +*/ +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx) +{ +struct af_alg_tsgl *sgl, *tmp; +struct scatterlist *sg =NULL ; +unsigned int i; + +list_for_each_entry_safe (sgl, tmp, &ctx->tsgl_list, list) { +for (i =0 ; i < sgl->cur; i++) { +struct scatterlist *process_sg = sgl->sg + i; + +if (!(process_sg->length) || !sg_page (process_sg)) +continue ; + sg = process_sg; +break ; + } +if (sg) +break ; + } + +return sg; +} +EXPORT_SYMBOL_GPL (af_alg_get_tsgl_sg); + +/** +* af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX * * @sk [in] AF_ALG socket +* @areq [in/out] request buffer that will receive the IV +* to be used for the cipher * @return 0 on success, < 0 on error */ -int af_alg_get_iv (struct sock *sk) +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq) {struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; +struct scatterlist *sg; + + areq->iv = ctx->iv; + areq->ivlen =0 ;// areq->iv will not be freed + +/* If cipher has no IV, no need to lock it or do IIV processing */ +if (!ctx->ivlen) +return 0 ; -return mutex_lock_interruptible (&ctx->ivlock); +/* No inline IV, use ctx IV buffer and lock it */ +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) +return mutex_lock_interruptible (&ctx->ivlock); + +/* Inline IV handling: There must be the IV data present. */ +if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list)) +return -EINVAL ; + + areq->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL ); +if (unlikely (!areq->iv)) +return -ENOMEM ; + areq->ivlen = ctx->ivlen;// areq->iv will be freed + +/* Get ivlen data from TX SGL and copy it into areq->iv. */ + sg =af_alg_get_tsgl_sg (ctx); +if (!sg) +return -EFAULT ; +scatterwalk_map_and_copy (areq->iv, sg,0 , ctx->ivlen,0 ); +af_alg_pull_tsgl (sk, ctx->ivlen,NULL ,0 ); + +return 0 ; }EXPORT_SYMBOL_GPL (af_alg_get_iv);
@@ -1202,7 +1280,12 @@ void af_alg_put_iv(struct sock *sk)struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; -mutex_unlock (&ctx->ivlock); +/* To resemble af_alg_get_iv, do not merge the two branches. */ +if (!ctx->ivlen) +return ; + +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) +mutex_unlock (&ctx->ivlock); }EXPORT_SYMBOL_GPL (af_alg_put_iv);
crypto/algif_aead.c +24 -34
117 lines not on this page · raw message
crypto/algif_skcipher.c +7 -5
@@ -77,7 +77,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); - err =af_alg_get_iv (sk); +/* Set areq->iv */ + err =af_alg_get_iv (sk, areq);if (err) goto free;
@@ -116,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ skcipher_request_set_tfm (&areq->cra_u.skcipher_req, tfm);skcipher_request_set_crypt (&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sg, len, areq->iv);if (msg->msg_iocb && !is_sync_kiocb (msg->msg_iocb)) {/* AIO operation */
@@ -350,14 +351,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)if (!ctx)return -ENOMEM ; - ctx->iv =sock_kmalloc (sk,crypto_skcipher_ivsize (tfm), -GFP_KERNEL ); + ctx->ivlen =crypto_skcipher_ivsize (tfm); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +memset (ctx->iv,0 , ctx->ivlen);mutex_init (&ctx->ivlock);INIT_LIST_HEAD (&ctx->tsgl_list);
@@ -367,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =ALG_IV_PROCESSING_UNDEF ;crypto_init_wait (&ctx->wait); ask->private = ctx;
include/crypto/if_alg.h +20 -1
@@ -96,6 +96,14 @@ struct af_alg_rsgl {* @tsgl_entries: Number of entries in priv. TX SGL * @outlen: Number of output bytes generated by crypto op * @areqlen: Length of this data structure +* @iv: Buffer holding the IV for the cipher operation (this +* is either ctx->iv where ivlen is set to 0 or a newly +* allocated buffer where ivlen represents the size of +* this buffer). +* @ivlen: IV size -- if the IV buffer is to be freed during +* releasing of this data structure, ivlen is non-zero. +* If ivlen is zero, but iv is non-NULL, the iv buffer is +* not freed. * @cra_u: Cipher request */ struct af_alg_async_req {
@@ -112,6 +120,9 @@ struct af_alg_async_req {unsigned int outlen;unsigned int areqlen; +void *iv; +unsigned int ivlen; +union {struct aead_request aead_req;struct skcipher_request skcipher_req;
@@ -128,6 +139,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlen: IV size * @ivlock: Mutex to serialize access to the IV * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation
@@ -142,12 +154,14 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +unsigned int ivlen;struct mutex ivlock;size_t aead_assoclen;
@@ -159,9 +173,13 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +int iiv;unsigned int len; }; +#define ALG_IV_PROCESSING_UNDEF (-1) +#define ALG_IV_SERIAL_PROCESSING (0) +#define ALG_IV_PARALLEL_PROCESSING (1)int af_alg_register_type (const struct af_alg_type *type);int af_alg_unregister_type (const struct af_alg_type *type);
@@ -255,7 +273,8 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); -int af_alg_get_iv (struct sock *sk); +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq);void af_alg_put_iv (struct sock *sk); +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /* _CRYPTO_IF_ALG_H */
include/uapi/linux/if_alg.h +4 -2
@@ -37,7 +37,9 @@ struct af_alg_iv {#define ALG_SET_AEAD_AUTHSIZE 5/* Operations */ -#define ALG_OP_DECRYPT 0 -#define ALG_OP_ENCRYPT 1 +#define ALG_OP_DECRYPT 0x0 +#define ALG_OP_ENCRYPT 0x1 +#define ALG_OP_CIPHER_MASK 0xf +#define ALG_OP_INLINE_IV 0x10#endif /* _LINUX_IF_ALG_H */
Signature · 1 line
2.14.3
PATCH v3 3/4 crypto: AF_ALG - allow driver to serialize IV access
In reply to Stephan MüllerThe mutex in AF_ALG to serialize access to the IV ensures full serialization of requests sent to the crypto driver.
However, the hardware may implement serialization to the IV such that preparation work without touching the IV can already happen while the IV is processed by another operation. This may speed up the AIO processing.
The following ASCII art demonstrates this.
AF_ALG mutex handling implements the following logic:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
NOTHING BLOCKING (lock mutex) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] |
| |
[unlock mutex] Nothing Blocking (lock mutex)
| |
Don't care beyond here. [BUILD / MAP HW DESCS]
|
[Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.A crypto driver may implement the following serialization:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
[BUILD / MAP HW DESCS] [BUILD / MAP HW DESCS]
| |
NOTHING BLOCKING IV in flight (enqueue sw queue)
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| |
Don't care beyond here. [Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.If the driver implements its own serialization (i.e. AF_ALG does not serialize the access to the IV), the crypto implementation must set the flag CRYPTO_ALG_SERIALIZES_IV_ACCESS.
--- crypto/af_alg.c | 13 ++++++++----- crypto/algif_aead.c | 1 + crypto/algif_skcipher.c | 1 + include/crypto/if_alg.h | 13 +++++++++++++ include/linux/crypto.h | 15 +++++++++++++++ 5 files changed, 38 insertions(+), 5 deletions(-)
crypto/af_alg.c +8 -5
@@ -1247,8 +1247,10 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)return 0 ;/* No inline IV, use ctx IV buffer and lock it */ -if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) -return mutex_lock_interruptible (&ctx->ivlock); +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) { +return (ctx->lock_iv) ? +mutex_lock_interruptible (&ctx->ivlock) :0 ; + }/* Inline IV handling: There must be the IV data present. */ if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list))
@@ -1280,12 +1282,13 @@ void af_alg_put_iv(struct sock *sk)struct alg_sock *ask =alg_sk (sk);struct af_alg_ctx *ctx = ask->private; -/* To resemble af_alg_get_iv, do not merge the two branches. */ if (!ctx->ivlen)return ; -if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) -mutex_unlock (&ctx->ivlock); +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) { +if (ctx->lock_iv) +mutex_unlock (&ctx->ivlock); + } }EXPORT_SYMBOL_GPL (af_alg_put_iv);
crypto/algif_aead.c +1 -0
@@ -565,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->lock_iv =af_alg_lock_iv (crypto_aead_tfm (aead)); ctx->iiv =ALG_IV_PROCESSING_UNDEF ; ctx->aead_assoclen =0 ;crypto_init_wait (&ctx->wait);
crypto/algif_skcipher.c +1 -0
@@ -368,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->lock_iv =af_alg_lock_iv (crypto_skcipher_tfm (tfm)); ctx->iiv =ALG_IV_PROCESSING_UNDEF ;crypto_init_wait (&ctx->wait);
include/crypto/if_alg.h +13 -0
@@ -154,6 +154,7 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @lock_iv: Shall IV be locked? * @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */
@@ -173,6 +174,7 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +bool lock_iv;int iiv;unsigned int len;
@@ -251,6 +253,17 @@ static inline bool af_alg_readable(struct sock *sk)return PAGE_SIZE <=af_alg_rcvbuf (sk); } +/** +* Shall access to the ctx->iv be serialized using a mutex? +* +* @tfm TFM of to be used for cipher operation +* @return true => lock, false => do not lock +*/ +static inline bool af_alg_lock_iv (struct crypto_tfm *tfm) +{ +return !(crypto_tfm_alg_flags (tfm) &CRYPTO_ALG_SERIALIZES_IV_ACCESS ); +} +int af_alg_alloc_tsgl (struct sock *sk);unsigned int af_alg_count_tsgl (struct sock *sk,size_t bytes,size_t offset); void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst,
include/linux/crypto.h +15 -0
@@ -112,6 +112,16 @@*/ #define CRYPTO_ALG_OPTIONAL_KEY 0x00004000 +/* +* Set if the algorithm implementation provides its own serialization +* of IV accesses. This prevents the AF_ALG interface to serialize +* cipher requests. This flag allows the implementation to implement +* a more streamlined IV serialization logic, such as only serializing +* the submission of a request to hardware whereas the preparation steps +* for sending data to hardware are not serialized. +*/ +#define CRYPTO_ALG_SERIALIZES_IV_ACCESS 0x00008000 +/* * Transform masks and values (for crt_flags). */
@@ -674,6 +684,11 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm)return tfm->__crt_alg->cra_flags &CRYPTO_ALG_TYPE_MASK ; } +static inline u32 crypto_tfm_alg_flags (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags; +} +static inline unsigned int crypto_tfm_alg_blocksize (struct crypto_tfm *tfm) {return tfm->__crt_alg->cra_blocksize;
Signature · 1 line
2.14.3
PATCH v3 4/4 crypto: add CRYPTO_TFM_REQ_IV_SERIALIZE flag
In reply to Stephan MüllerCrypto drivers may implement a streamlined serialization support for AIO requests that is reported by the CRYPTO_ALG_SERIALIZES_IV_ACCESS flag to the crypto user. When the user decides that he wants to send multiple AIO requests concurrently and wants the crypto driver to handle the serialization, the caller has to set CRYPTO_TFM_REQ_IV_SERIALIZE to notify the crypto driver.
Only when this flag is enabled, the crypto driver shall apply its serialization logic for handling IV updates between requests. If this flag is not provided, the serialization logic shall not be applied by the driver as the caller decides that it does not need it (because no parallel AIO requests are sent) or that it performs its own serialization.
--- crypto/algif_aead.c | 15 ++++++++++++--- crypto/algif_skcipher.c | 15 ++++++++++++--- include/linux/crypto.h | 1 + 3 files changed, 25 insertions(+), 6 deletions(-)
crypto/algif_aead.c +12 -3
@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm); +struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) + tfm->crt_flags |=CRYPTO_TFM_REQ_IV_SERIALIZE ; + +return ret; } static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
crypto/algif_skcipher.c +12 -3
@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg, {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk); -struct crypto_skcipher *tfm = pask->private; -unsigned ivsize =crypto_skcipher_ivsize (tfm); +struct crypto_skcipher *skc = pask->private; +struct crypto_tfm *tfm =crypto_skcipher_tfm (skc); +unsigned int ivsize =crypto_skcipher_ivsize (skc); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) + tfm->crt_flags |=CRYPTO_TFM_REQ_IV_SERIALIZE ; + +return ret; } static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
include/linux/crypto.h +1 -0
@@ -133,6 +133,7 @@#define CRYPTO_TFM_REQ_WEAK_KEY 0x00000100#define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200#define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400 +#define CRYPTO_TFM_REQ_IV_SERIALIZE 0x00000800#define CRYPTO_TFM_RES_WEAK_KEY 0x00100000#define CRYPTO_TFM_RES_BAD_KEY_LEN 0x00200000#define CRYPTO_TFM_RES_BAD_KEY_SCHED 0x00400000
Signature · 1 line
2.14.3
Re: [PATCH v3 4/4] crypto: add CRYPTO_TFM_REQ_IV_SERIALIZE flag
Quoted 6 lines · Stephan Müller
Crypto drivers may implement a streamlined serialization support for AIO requests that is reported by the
CRYPTO_ALG_SERIALIZES_IV_ACCESSflag to the crypto user. When the user decides that he wants to send multiple AIO requests concurrently and wants the crypto driver to handle the serialization, the caller has to setCRYPTO_TFM_REQ_IV_SERIALIZEto notify the crypto driver.
Will crypto_alloc_* API takes cares of this flag?. For Kernel Crypto user IV Synchronization logic depends on weather tfm allocated supports IV Serialisation or not.
Quoted 86 lines
Only when this flag is enabled, the crypto driver shall apply its serialization logic for handling IV updates between requests. If this flag is not provided, the serialization logic shall not be applied by the driver as the caller decides that it does not need it (because no parallel AIO requests are sent) or that it performs its own serialization.
Signed-off-by Stephan Mueller <[email protected]>--- crypto/algif_aead.c | 15 ++++++++++++--- crypto/algif_skcipher.c | 15 ++++++++++++--- include/linux/crypto.h | 1 + 3 files changed, 25 insertions(+), 6 deletions(-)crypto/algif_aead.c +12
-3@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk);struct aead_tfm *aeadc = pask->private; -struct crypto_aead *tfm = aeadc->aead; -unsigned int ivsize =crypto_aead_ivsize (tfm); +struct crypto_aead *aead = aeadc->aead; +struct crypto_tfm *tfm =crypto_aead_tfm (aead); +unsigned int ivsize =crypto_aead_ivsize (aead); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) + tfm->crt_flags |=CRYPTO_TFM_REQ_IV_SERIALIZE ; + +return ret; } static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,crypto/algif_skcipher.c +12
-3@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg, {struct sock *sk = sock->sk;struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private;struct sock *psk = ask->parent;struct alg_sock *pask =alg_sk (psk); -struct crypto_skcipher *tfm = pask->private; -unsigned ivsize =crypto_skcipher_ivsize (tfm); +struct crypto_skcipher *skc = pask->private; +struct crypto_tfm *tfm =crypto_skcipher_tfm (skc); +unsigned int ivsize =crypto_skcipher_ivsize (skc); +int ret =af_alg_sendmsg (sock, msg, size, ivsize); + +if (ret <0 ) +return ret; -return af_alg_sendmsg (sock, msg, size, ivsize); +if (ctx->iiv ==ALG_IV_SERIAL_PROCESSING ) + tfm->crt_flags |=CRYPTO_TFM_REQ_IV_SERIALIZE ; + +return ret; } static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,include/linux/crypto.h +1
-0@@ -133,6 +133,7 @@#define CRYPTO_TFM_REQ_WEAK_KEY 0x00000100#define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200#define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400 +#define CRYPTO_TFM_REQ_IV_SERIALIZE 0x00000800#define CRYPTO_TFM_RES_WEAK_KEY 0x00100000#define CRYPTO_TFM_RES_BAD_KEY_LEN 0x00200000#define CRYPTO_TFM_RES_BAD_KEY_SCHED 0x00400000
Re: [PATCH v3 4/4] crypto: add CRYPTO_TFM_REQ_IV_SERIALIZE flag
Am Mittwoch, 14. Februar 2018, 06:50:38 CET schrieb Harsh Jain:
Hi Harsh,
Quoted 11 lines · Stephan Müller
Quoted 6 lines
Crypto drivers may implement a streamlined serialization support for AIO requests that is reported by the
CRYPTO_ALG_SERIALIZES_IV_ACCESSflag to the crypto user. When the user decides that he wants to send multiple AIO requests concurrently and wants the crypto driver to handle the serialization, the caller has to setCRYPTO_TFM_REQ_IV_SERIALIZEto notify the crypto driver.Will crypto_alloc_* API takes cares of this flag?. For Kernel Crypto user IV Synchronization logic depends on weather tfm allocated supports IV Serialisation or not.
The alloc API calls are not related to this flag. This flag is set in the request sent to the driver. If the driver sees this flag, it shall perform its serialization.
The idea is the following: the driver reports CRYPTO_ALG_SERIALIZES_IV_ACCESS if it can serialize requests. In this case, the higher level functions (like AF_ALG) would not serialize the request. Now, the higher level functions must inform the driver that its serialization function shall be performed which implemented with CRYPTO_TFM_REQ_IV_SERIALIZE.
Note, the higher level functions may decide that no serialization is necessary (e.g. in the case the inline IV handling is followed by AF_ALG). This implies that the CRYPTO_TFM_REQ_IV_SERIALIZE flag would not be set even though the driver is capable of serializing (and thus would report CRYPTO_ALG_SERIALIZES_IV_ACCESS).
Ciao
Stephan
Re: [PATCH] crypto: AF_ALG - inline IV support
In reply to Stephan MuellerQuoted 72 lines · On Mon, 15 Jan 2018 10:35:34 +0100 Stephan Mueller <[email protected]> wrote:
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the
AF_ALGinterface treats theALG_SET_OPflag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that
ctx->usedis updated. Thus, thectx->usedaccess in_aead_recvmsgmust be moved below the IV gathering.The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
Quoted 6 lines
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
Signed-off-by Stephan Mueller <[email protected]>
Firstly it works
Now to be really useful for my particular hardware (which doesn't do dependency tracking in its out of order queues) what I need to know is whether I need to let previous queued up entries finish before I can run this one or not.
Now there is an obvious 'hack' I can use, which is that the above only matters is if the IV is still 'in flight'. Given it is still inflight the memory is in use. Thus until it is finished it's address is clearly only being used by 'this IV'.
Hence I can just compare against the address of the IV for the previous packet. If it changed we know there is no need to wait for previous packets to finish.
I'll still have to be a little careful to avoid DoS issues if we flip from one mode to the other but I can do that by ensuring my software queue is empty before pushing to the hardware queue.
Anyhow using the address does seem a little fragile, any suggestions for a better way?
On the plus side, with 8k iovs I'm getting around a 50% performance boost by not having to serialize the entry into the hardware queues. From a few back of the envelope calculations the key thing is that I can coalesce interrupts.
For very small iovs the overhead of setting them us is enough to mean we rarely end up with many in the queue at a time so the benefits aren't seen.
On that note I have a hacked up libkcapi speed test that adds a parameter to control this. I'll send you a pull request later so you can take a look. I have only done the skcipher case though and I am sure it could be done in a neater fashion!
Thanks,
Jonathan
Quoted 400 lines
--- crypto/af_alg.c | 79 ++++++++++++++++++++++++++++++++++++++++++++- crypto/algif_aead.c | 54 ++++++++++++++----------------- crypto/algif_skcipher.c | 14 +++++--- include/crypto/if_alg.h | 17 ++++++++++ include/uapi/linux/if_alg.h | 6 ++-- 5 files changed, 133 insertions(+), 37 deletions(-)crypto/af_alg.c +78
-1@@ -14,6 +14,7 @@#include <linux/atomic.h> #include <crypto/if_alg.h> +#include <crypto/scatterwalk.h> #include <linux/crypto.h> #include <linux/init.h> #include <linux/kernel.h> @@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,struct af_alg_control con = {};long copied =0 ;bool enc =0 ; +bool iiv =0 ;bool init =0 ;int err =0 ;@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return err; init =1 ; -switch (con.op) { +switch (con.op &ALG_OP_CIPHER_MASK ) {case ALG_OP_ENCRYPT : enc =1 ;break ;@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,return -EINVAL ; } +if (con.op &ALG_OP_INLINE_IV ) + iiv =1 ; +if (con.iv && con.iv->ivlen != ivsize)return -EINVAL ; }@@ -866,6 +871,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,if (init) { ctx->enc = enc; + ctx->iiv = iiv;if (con.iv)memcpy (ctx->iv, con.iv->iv, ivsize);@@ -1031,6 +1037,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)struct sock *sk = areq->sk;af_alg_free_areq_sgls (areq); +if (areq->ivlen) +sock_kfree_s (sk, areq->iv, areq->ivlen);sock_kfree_s (sk, areq, areq->areqlen); }EXPORT_SYMBOL_GPL (af_alg_free_resources);@@ -1175,6 +1183,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, }EXPORT_SYMBOL_GPL (af_alg_get_rsgl); +/** +* af_alg_get_tsgl_sg - get first SG entry with data from TX SGL +* @ctx [in] AF_ALG context with TX sgl +* @return pointer to SG with data or NULL if none found +*/ +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx) +{ +struct af_alg_tsgl *sgl, *tmp; +struct scatterlist *sg =NULL ; +unsigned int i; + +list_for_each_entry_safe (sgl, tmp, &ctx->tsgl_list, list) { +for (i =0 ; i < sgl->cur; i++) { +struct scatterlist *process_sg = sgl->sg + i; + +if (!(process_sg->length) || !sg_page (process_sg)) +continue ; + sg = process_sg; +break ; + } +if (sg) +break ; + } + +return sg; +} +EXPORT_SYMBOL_GPL (af_alg_get_tsgl_sg); + +/** +* af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX +* +* @sk [in] AF_ALG socket +* @areq [in/out] request buffer that will receive the IV +* to be used for the cipher +* @return 0 on success, < 0 on error +*/ +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq) +{ +struct alg_sock *ask =alg_sk (sk); +struct af_alg_ctx *ctx = ask->private; +struct scatterlist *sg; + +/* No inline IV or cipher has no IV, use ctx IV buffer. */ +if (!ctx->iiv || !ctx->ivlen) { + areq->iv = ctx->iv; + areq->ivlen =0 ;// areq->iv will not be freed +return 0 ; + } + +/* There must be the IV data present. */ +if (ctx->used < ctx->ivlen ||list_empty (&ctx->tsgl_list)) +return -EINVAL ; + + areq->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL ); +if (unlikely (!areq->iv)) +return -ENOMEM ; + areq->ivlen = ctx->ivlen;// areq->iv will be freed + +/* Get ivlen data from TX SGL and copy it into areq->iv. */ + sg =af_alg_get_tsgl_sg (ctx); +if (!sg) +return -EFAULT ; +scatterwalk_map_and_copy (areq->iv, sg,0 , ctx->ivlen,0 ); +af_alg_pull_tsgl (sk, ctx->ivlen,NULL ,0 ); + +return 0 ; +} +EXPORT_SYMBOL_GPL (af_alg_get_iv); +static int __init af_alg_init (void ) {int err =proto_register (&alg_proto,0 );crypto/algif_aead.c +24
-30@@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,struct aead_tfm *aeadc = pask->private;struct crypto_aead *tfm = aeadc->aead;struct crypto_skcipher *null_tfm = aeadc->null_tfm; -unsigned int i, as =crypto_aead_authsize (tfm); +unsigned int as =crypto_aead_authsize (tfm);struct af_alg_async_req *areq; -struct af_alg_tsgl *tsgl, *tmp;struct scatterlist *rsgl_src, *tsgl_src =NULL ;int err =0 ;size_t used =0 ;/* [in] TX bufs to be en/decrypted */ @@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,return err; } -/* -* Data length provided by caller via sendmsg/sendpage that has not -* yet been processed. -*/ - used = ctx->used; -/* * Make sure sufficient data is present -- note, the same check is * is also present in sendmsg/sendpage. The checks in sendpage/sendmsg @@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,if (!aead_sufficient_data (sk))return -EINVAL ; +/* Allocate cipher request for current operation. */ + areq =af_alg_alloc_areq (sk,sizeof (struct af_alg_async_req ) + +crypto_aead_reqsize (tfm)); +if (IS_ERR (areq)) +return PTR_ERR (areq); + +/* Set areq->iv. */ + err =af_alg_get_iv (sk, areq); +if (err) + goto free; + +/* +* Data length provided by caller via sendmsg/sendpage that has not +* yet been processed. +*/ + used = ctx->used; +/* * Calculate the minimum output buffer size holding the result of the * cipher operation. When encrypting data, the receiving buffer is @@ -153,12 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,*/ used -= ctx->aead_assoclen; -/* Allocate cipher request for current operation. */ - areq =af_alg_alloc_areq (sk,sizeof (struct af_alg_async_req ) + -crypto_aead_reqsize (tfm)); -if (IS_ERR (areq)) -return PTR_ERR (areq); -/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq, outlen, &usedpages);if (err)@@ -183,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, } processed = used + ctx->aead_assoclen; -list_for_each_entry_safe (tsgl, tmp, &ctx->tsgl_list, list) { -for (i =0 ; i < tsgl->cur; i++) { -struct scatterlist *process_sg = tsgl->sg + i; - -if (!(process_sg->length) || !sg_page (process_sg)) -continue ; - tsgl_src = process_sg; -break ; - } -if (tsgl_src) -break ; - } + tsgl_src =af_alg_get_tsgl_sg (ctx);if (processed && !tsgl_src) { err = -EFAULT ; goto free;@@ -282,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ aead_request_set_crypt (&areq->cra_u.aead_req, rsgl_src, - areq->first_rsgl.sgl.sg, used, ctx->iv); + areq->first_rsgl.sgl.sg, used, areq->iv);aead_request_set_ad (&areq->cra_u.aead_req, ctx->aead_assoclen);aead_request_set_tfm (&areq->cra_u.aead_req, tfm);@@ -549,19 +542,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)struct aead_tfm *tfm = private;struct crypto_aead *aead = tfm->aead;unsigned int len =sizeof (*ctx); -unsigned int ivlen =crypto_aead_ivsize (aead); ctx =sock_kmalloc (sk, len,GFP_KERNEL );if (!ctx)return -ENOMEM ;memset (ctx,0 , len); - ctx->iv =sock_kmalloc (sk, ivlen,GFP_KERNEL ); + ctx->ivlen =crypto_aead_ivsize (aead); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 , ivlen); +memset (ctx->iv,0 , ctx->ivlen);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;@@ -570,6 +563,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =0 ; ctx->aead_assoclen =0 ;crypto_init_wait (&ctx->wait);crypto/algif_skcipher.c +10
-4@@ -77,6 +77,11 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,if (IS_ERR (areq))return PTR_ERR (areq); +/* Set areq->iv */ + err =af_alg_get_iv (sk, areq); +if (err) + goto free; +/* convert iovecs of output buffers into RX SGL */ err =af_alg_get_rsgl (sk, msg, flags, areq,-1 , &len);if (err)@@ -112,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,/* Initialize the crypto operation */ skcipher_request_set_tfm (&areq->cra_u.skcipher_req, tfm);skcipher_request_set_crypt (&areq->cra_u.skcipher_req, areq->tsgl, - areq->first_rsgl.sgl.sg, len, ctx->iv); + areq->first_rsgl.sgl.sg, len, areq->iv);if (msg->msg_iocb && !is_sync_kiocb (msg->msg_iocb)) {/* AIO operation */ @@ -345,14 +350,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)if (!ctx)return -ENOMEM ; - ctx->iv =sock_kmalloc (sk,crypto_skcipher_ivsize (tfm), -GFP_KERNEL ); + ctx->ivlen =crypto_skcipher_ivsize (tfm); + ctx->iv =sock_kmalloc (sk, ctx->ivlen,GFP_KERNEL );if (!ctx->iv) {sock_kfree_s (sk, ctx, len);return -ENOMEM ; } -memset (ctx->iv,0 ,crypto_skcipher_ivsize (tfm)); +memset (ctx->iv,0 , ctx->ivlen);INIT_LIST_HEAD (&ctx->tsgl_list); ctx->len = len;@@ -361,6 +366,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk) ctx->more =0 ; ctx->merge =0 ; ctx->enc =0 ; + ctx->iiv =0 ;crypto_init_wait (&ctx->wait); ask->private = ctx;include/crypto/if_alg.h +17
-0@@ -95,6 +95,14 @@ struct af_alg_rsgl {* @tsgl_entries: Number of entries in priv. TX SGL * @outlen: Number of output bytes generated by crypto op * @areqlen: Length of this data structure +* @iv: Buffer holding the IV for the cipher operation (this +* is either ctx->iv where ivlen is set to 0 or a newly +* allocated buffer where ivlen represents the size of +* this buffer). +* @ivlen: IV size -- if the IV buffer is to be freed during +* releasing of this data structure, ivlen is non-zero. +* If ivlen is zero, but iv is non-NULL, the iv buffer is +* not freed. * @cra_u: Cipher request */ struct af_alg_async_req {@@ -111,6 +119,9 @@ struct af_alg_async_req {unsigned int outlen;unsigned int areqlen; +void *iv; +unsigned int ivlen; +union {struct aead_request aead_req;struct skcipher_request skcipher_req;@@ -127,6 +138,7 @@ struct af_alg_async_req {* * @tsgl_list: Link to TX SGL * @iv: IV for cipher operation +* @ivlen: IV size * @aead_assoclen: Length of AAD for AEAD cipher operations * @completion: Work queue for synchronous operation * @used: TX bytes sent to kernel. This variable is used to @@ -140,12 +152,14 @@ struct af_alg_async_req {* SG? * @enc: Cryptographic operation to be performed when * recvmsg is invoked. +* @iiv: Use inline IV: first part of TX data is IV * @len: Length of memory allocated for this data structure. */ struct af_alg_ctx {struct list_head tsgl_list;void *iv; +unsigned int ivlen;size_t aead_assoclen;struct crypto_wait wait;@@ -156,6 +170,7 @@ struct af_alg_ctx {bool more;bool merge;bool enc; +bool iiv;unsigned int len; };@@ -252,5 +267,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,int af_alg_get_rsgl (struct sock *sk,struct msghdr *msg,int flags,struct af_alg_async_req *areq,size_t maxsize,size_t *outlen); +int af_alg_get_iv (struct sock *sk,struct af_alg_async_req *areq); +struct scatterlist *af_alg_get_tsgl_sg (struct af_alg_ctx *ctx);#endif /* _CRYPTO_IF_ALG_H */ include/uapi/linux/if_alg.h +4
-2@@ -37,7 +37,9 @@ struct af_alg_iv {#define ALG_SET_AEAD_AUTHSIZE 5/* Operations */ -#define ALG_OP_DECRYPT 0 -#define ALG_OP_ENCRYPT 1 +#define ALG_OP_DECRYPT 0x0 +#define ALG_OP_ENCRYPT 0x1 +#define ALG_OP_CIPHER_MASK 0xf +#define ALG_OP_INLINE_IV 0x10#endif /* _LINUX_IF_ALG_H */
Re: [PATCH] crypto: AF_ALG - inline IV support
Am Montag, 22. Januar 2018, 15:11:53 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 80 lines
On Mon, 15 Jan 2018 10:35:34 +0100
Quoted 77 lines · Stephan Mueller
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the
AF_ALGinterface treats theALG_SET_OPflag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that
ctx->usedis updated. Thus, thectx->usedaccess in_aead_recvmsgmust be moved below the IV gathering.The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
Quoted 4 lines
Quoted 1 line
Signed-off-by Stephan Mueller <[email protected]>Firstly it works
Tested-by Jonathan Cameron <[email protected]>
Thank you.
Quoted 6 lines
Now to be really useful for my particular hardware (which doesn't do dependency tracking in its out of order queues) what I need to know is whether I need to let previous queued up entries finish before I can run this one or not.
Quoted 8 lines
Now there is an obvious 'hack' I can use, which is that the above only matters is if the IV is still 'in flight'. Given it is still inflight the memory is in use. Thus until it is finished it's address is clearly only being used by 'this IV'.
Hence I can just compare against the address of the IV for the previous packet. If it changed we know there is no need to wait for previous packets to finish.
That is dangerous, because the IV is in a malloced location. It is possible that a new malloc will return the same address.
What about simply applying the cipher operation as "one shot". I.e. one invocation of a cipher operation is isolated from the next. Thus, queue one request defined with one request data structure and handle each request independent from the next.
Quoted 13 lines
I'll still have to be a little careful to avoid DoS issues if we flip from one mode to the other but I can do that by ensuring my software queue is empty before pushing to the hardware queue.
Anyhow using the address does seem a little fragile, any suggestions for a better way?
On the plus side, with 8k iovs I'm getting around a 50% performance boost by not having to serialize the entry into the hardware queues. From a few back of the envelope calculations the key thing is that I can coalesce interrupts.
Quoted 8 lines
For very small iovs the overhead of setting them us is enough to mean we rarely end up with many in the queue at a time so the benefits aren't seen.
On that note I have a hacked up libkcapi speed test that adds a parameter to control this. I'll send you a pull request later so you can take a look. I have only done the skcipher case though and I am sure it could be done in a neater fashion!
Thank you.
Ciao
Stephan
Re: [PATCH] crypto: AF_ALG - inline IV support
Quoted 3 lines · On Mon, 22 Jan 2018 15:30:39 +0100 Stephan Mueller <[email protected]> wrote:
Am Montag, 22. Januar 2018, 15:11:53 CET schrieb Jonathan Cameron:
Hi Jonathan,
Hi Stephan,
Quoted 98 lines
Quoted 81 lines
On Mon, 15 Jan 2018 10:35:34 +0100
Quoted 77 lines · Stephan Mueller
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
The solution is to allow providing the IV data supplied as part of the plaintext/ciphertext. To do so, the
AF_ALGinterface treats theALG_SET_OPflag usable with sendmsg as a bit-array allowing to set the cipher operation together with the flag whether the operation should enable support for inline IV handling.If inline IV handling is enabled, the IV is expected to be the first part of the input plaintext/ciphertext. This IV is only used for one cipher operation and will not retained in the kernel for subsequent cipher operations.
The AEAD support required a slight re-arragning of the code, because obtaining the IV implies that
ctx->usedis updated. Thus, thectx->usedaccess in_aead_recvmsgmust be moved below the IV gathering.The AEAD code to find the first SG with data in the TX SGL is moved to a common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is allowed to provide an IV via sendmsg. It only extends the interface by giving the user the choice to provide the IV either via sendmsg (the current approach) or as part of the data (the additional approach).
Quoted 4 lines
Quoted 1 line
Signed-off-by Stephan Mueller <[email protected]>Firstly it works
Tested-by Jonathan Cameron <[email protected]>Thank you.
Quoted 11 lines
Now to be really useful for my particular hardware (which doesn't do dependency tracking in its out of order queues) what I need to know is whether I need to let previous queued up entries finish before I can run this one or not.
Now there is an obvious 'hack' I can use, which is that the above only matters is if the IV is still 'in flight'. Given it is still inflight the memory is in use. Thus until it is finished it's address is clearly only being used by 'this IV'.
Quoted 6 lines
Quoted 3 lines
Hence I can just compare against the address of the IV for the previous packet. If it changed we know there is no need to wait for previous packets to finish.
That is dangerous, because the IV is in a malloced location. It is possible that a new malloc will return the same address.
True, but this can only happen once a free has occurred which means the IV is done with from the previous point of view and we no longer need to track the dependency.
So it doesn't tell us if the two messages use the same IV or not, but in the case where it is wrong (and indeed I typically get the same address a lot of times when running little else on the machine) it doesn't matter because the element is already finished and so there is nothing to queue behind.
Quoted 5 lines
What about simply applying the cipher operation as "one shot". I.e. one invocation of a cipher operation is isolated from the next. Thus, queue one request defined with one request data structure and handle each request independent from the next.
I do queue one request from one structure, but if chaining is going on (the first one should save the chained IV for the second one) I have to wait for the update to happen before I can put the second one on the queue.
This is done by having a front end software queue when in chaining modes. The conditions for entry into the hardware queue are:
(hardware queue empty AND software feeder queue empty) OR (no dependency AND software feeder queue empty)
The completion hander will then move a message from software to hardware queues if there is one waiting.
It is this dependency tracking that indirectly takes the time as I can only have one invocation in the hardware queues at a time - hence each interrupt is for only one invocation. Given I have a number of hardware engines behind the queue I really want to get as many in flight as possible. To do this I need to know they are independent at ingress.
I need to get the driver through our internal processes, but hopefully can post an RFC sometime later this week.
Jonathan
Quoted 28 lines
Quoted 13 lines
I'll still have to be a little careful to avoid DoS issues if we flip from one mode to the other but I can do that by ensuring my software queue is empty before pushing to the hardware queue.
Anyhow using the address does seem a little fragile, any suggestions for a better way?
On the plus side, with 8k iovs I'm getting around a 50% performance boost by not having to serialize the entry into the hardware queues. From a few back of the envelope calculations the key thing is that I can coalesce interrupts.
Quoted 8 lines
For very small iovs the overhead of setting them us is enough to mean we rarely end up with many in the queue at a time so the benefits aren't seen.
On that note I have a hacked up libkcapi speed test that adds a parameter to control this. I'll send you a pull request later so you can take a look. I have only done the skcipher case though and I am sure it could be done in a neater fashion!
Thank you.
Ciao
Stephan
Am Freitag, 12. Januar 2018, 14:21:15 CET schrieb Stephan Mueller:
Hi,
Quoted 8 lines
1. Require that the cipher implementations serialize any AIO requests that have dependencies. I.e. for CBC, requests need to be serialized by the driver. For, say, ECB or XTS no serialization is necessary.
2. Change
AF_ALGto require a per-request IV. This could be implemented by moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg code path would obtain the IV.
With the released patch, I found a third way that seems to be much less
intrusive: adding inline IV handling where the IV is sent as part of the data to be processed.
In the libkcapi source code tree, I added a branch [1] that contains support for the inline IV handling. It also has received numerous tests [2] verifying that the inline IV kernel patch works.
[1] https://github.com/smuellerDD/libkcapi/tree/iiv
[2] https://github.com/smuellerDD/libkcapi/commit/f56991ff2f975caf1aa6cb75f2b6fc104cc72f9c
Ciao
Stephan
Quoted 7 lines · On Fri, 12 Jan 2018 14:21:15 +0100 Stephan Mueller <[email protected]> wrote:
Hi,
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
Silly question, are we obliged to always write it back? In CBC it is obviously the same as the last n bytes of the encrypted message. I guess for ease of handling it makes sense to do so though.
Quoted 40 lines
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the
analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
Quoted 9 lines
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
This needs to be solved somehow. I see the following options which I would like to have vetted by the community.
Taking some 'entirely hypothetical' hardware with the following structure for all my responses - it's about as flexible as I think we'll see in the near future - though I'm sure someone has something more complex out there :)
N hardware queues feeding M processing engines in a scheduler driven fashion. Actually we might have P sets of these, but load balancing and tracking and transferring contexts between these is a complexity I think we can ignore. If you want to use more than one of these P you'll just have to handle it yourself in userspace. Note messages may be shorter than IOCBs which raises another question I've been meaning to ask. Are all crypto algorithms obliged to run unlimited length IOCBs?
If there are M messages in a particular queue and none elsewhere it is capable of processing them all at once (and perhaps returning out of order but we can fudge them back in order in the driver to avoid that additional complexity from an interface point of view).
So I'm going to look at this from the hardware point of view - you have well addressed software management above.
Three ways context management can be handled (in CBC this is basically just the IV).
1. Each 'work item' queued on a hardware queue has it's IV embedded with the data. This requires external synchronization if we are chaining across multiple 'work items' - note the hardware may have restrictions that mean it has to split large pieces of data up to encrypt them. Not all hardware may support per 'work item' IVs (I haven't done a survey to find out if everyone does...)
2. Each queue has a context assigned. We get a new queue whenever we want to have a different context. Runs out eventually but our hypothetical hardware may support a lot of queues. Note this version could be 'faked' by putting a cryptoengine queue on the front of the hardware queues.
3. The hardware supports IV dependency tracking in it's queues. That is, it can check if the address pointing to the IV is in use by one of the processing units which has not yet updated the IV ready for chaining with the next message. Note it might use a magic token rather than the IV pointer. For modes with out chaining (including counter modes) the IV pointer will inherently always be different.
The hardware then simply schedules something else until it can safely run that particular processing unit.
Quoted 3 lines
1. Require that the cipher implementations serialize any AIO requests that have dependencies. I.e. for CBC, requests need to be serialized by the driver. For, say, ECB or XTS no serialization is necessary.
There is a certain requirement to do this anyway as we may have a streaming type situation and we don't want to have to do the chaining in userspace.
So we send first X MB block to HW but before it has come back we have more data arrive that needs decrypting so we queue that behind it. The IV then needs to be updated automatically (or the code needs to do it on the first work item coming back). If you don't have option 3 above, you have to do this. This is what I was planning to implement for our existing hardware before you raised this question and I don't think we get around it being necessary for performance in any case. Setting up IOMMUs etc is costly so we want to be doing everything we can before the IV update is ready.
Quoted 9 lines
2. Change
AF_ALGto require a per-request IV. This could be implemented by moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg code path would obtain the IV.I would tend to favor option 2 as this requires code change at only location. If option 2 is considered, I would recommend to still allow setting the IV via sendmsg CMSG (to keep the interface stable). If, however, the caller provides an IV via recvmsg, this takes precedence.
We definitely want to keep option 1 (which runs on the existing interface and does the magic in driver) for those who want it.
So the only one left is the case 3 above where the hardware is capable of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of resources.
1) Whenever we want to allocate a new context we spin up a new socket and effectively associate a single IV with that (and it's chained updates) much like we do in the existing interface.
2) We allow a token based tracking of IVs. So userspace code maintains a counter and tags ever message and the initial IV setup with that counter.
As the socket typically belongs to a userspace process tag creation can be in userspace and it can ensure it doesn't overlap tags (or it'll get the wrong answer).
Kernel driver can then handle making sure any internal token / addresses are correct. I haven't looked at in depth but would imagine this one would be rather more invasive to support.
Quoted 3 lines
If there are other options, please allow us to learn about them.
Glad we are addressing these usecases and that we have AIO support in general. Makes for a better discussion around whether in kernel support for these interfaces is actually as effective as moving to userspace drivers...
Jonathan
Quoted 4 lines
Ciao
Stephan
Am Montag, 15. Januar 2018, 12:05:03 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 5 lines
On Fri, 12 Jan 2018 14:21:15 +0100
Quoted 2 lines · Stephan Mueller
Hi,
Quoted 7 lines
Quoted 6 lines
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
Silly question, are we obliged to always write it back?
Well, in general, yes. The AF_ALG interface should allow a "stream" mode of operation:
socket
accept
setsockopt(setkey)sendmsg(IV, data)
recvmsg(data)
sendmsg(data)
recvmsg(data)
..
For such synchronous operation, I guess it is clear that the IV needs to be written back.
If you want to play with it, use the "stream" API of libkcapi and the associated test cases.
Quoted 59 lines
In CBC it is
obviously the same as the last n bytes of the encrypted message. I guess for ease of handling it makes sense to do so though.Quoted 41 lines
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
Quoted 8 lines
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
This needs to be solved somehow. I see the following options which I would like to have vetted by the community.
Taking some 'entirely hypothetical' hardware with the following structure for all my responses - it's about as flexible as I think we'll see in the near future - though I'm sure someone has something more complex out there :)
Quoted 7 lines
N hardware queues feeding M processing engines in a scheduler driven fashion. Actually we might have P sets of these, but load balancing and tracking and transferring contexts between these is a complexity I think we can ignore. If you want to use more than one of these P you'll just have to handle it yourself in userspace. Note messages may be shorter than IOCBs which raises another question I've been meaning to ask. Are all crypto algorithms obliged to run unlimited length IOCBs?
There are instances where hardware may reject large data chunks. IIRC I have seen some limits around 32k. But in this case, the driver must chunk up the scatter-gather lists (SGLs) with the data and feed it to the hardware in the chunk size necessary.
Quoted 1 line
From the kernel crypto API point of view, the driver must support unlimited
sized IOCBs / SGLs.
Quoted 24 lines
If there are M messages in a particular queue and none elsewhere it is capable of processing them all at once (and perhaps returning out of order but we can fudge them back in order in the driver to avoid that additional complexity from an interface point of view).
So I'm going to look at this from the hardware point of view - you have well addressed software management above.
Three ways context management can be handled (in CBC this is basically just the IV).
1. Each 'work item' queued on a hardware queue has it's IV embedded with the data. This requires external synchronization if we are chaining across multiple 'work items' - note the hardware may have restrictions that mean it has to split large pieces of data up to encrypt them. Not all hardware may support per 'work item' IVs (I haven't done a survey to find out if everyone does...)
2. Each queue has a context assigned. We get a new queue whenever we want to have a different context. Runs out eventually but our hypothetical hardware may support a lot of queues. Note this version could be 'faked' by putting a cryptoengine queue on the front of the hardware queues.
Quoted 8 lines
3. The hardware supports IV dependency tracking in it's queues. That is, it can check if the address pointing to the IV is in use by one of the processing units which has not yet updated the IV ready for chaining with the next message. Note it might use a magic token rather than the IV pointer. For modes with out chaining (including counter modes) the IV pointer will inherently always be different.
The hardware then simply schedules something else until it can safely run that particular processing unit.
The kernel crypto API has the following concept:
- a TFM holds the data that is stable for an entire cipher operation, such as the key -- one cipher operation may consist of individual calls
- a request structure holds the volatile data, i.e. the data that is valid for one particular call, such as the input plaintext or the IV
Thus, your hardware queue should expect one request and it must be capable of handling that one request with the given data. If you want to split up the request because you have sufficient hardware resources as you mentioned above, your driver/hardware must process the request accordingly.
Coming back to the AF_ALG interface: in order to support the aforementioned "stream" mode, the requests for each cipher call invoked by one recvmsg syscall points to the same IV buffer.
In case of AIO with multiple IOCBs, user space conceptually calls:
sendmsg
recvmsg
recvmsg
..
where all recvmsg calls execute in parallel. As each recvmsg call has one request associated with it, the question is what happens to a buffer that is pointed to by multiple request structures in such parallel execution.
If your hardware is capable of serializing the recvmsg calls or tracking the dependency, the current AF_ALG interface is fully sufficient.
But there may be hardware that cannot/will not track such dependencies. Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For such a case, AF_ALG currently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.
Quoted 6 lines
Quoted 3 lines
1. Require that the cipher implementations serialize any AIO requests that have dependencies. I.e. for CBC, requests need to be serialized by the driver. For, say, ECB or XTS no serialization is necessary.
There is a certain requirement to do this anyway as we may have a streaming type situation and we don't want to have to do the chaining in userspace.
Absolutely. If you have proper hardware/driver support, that would be beneficial. This would be supported with the current AF_ALG interface.
But I guess there are also folks out there who simply want to offer multiple hardware queues to allow independent cipher operations to be invoked concurrently without any dependency handling. This is currently not supported with AF_ALG.
Quoted 16 lines
So we send first X MB block to HW but before it has come back we have more data arrive that needs decrypting so we queue that behind it. The IV then needs to be updated automatically (or the code needs to do it on the first work item coming back). If you don't have option 3 above, you have to do this. This is what I was planning to implement for our existing hardware before you raised this question and I don't think we get around it being necessary for performance in any case. Setting up IOMMUs etc is costly so we want to be doing everything we can before the IV update is ready.
Quoted 5 lines
2. Change
AF_ALGto require a per-request IV. This could be implemented by moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg
code path would obtain the IV.
Quoted 7 lines
Quoted 4 lines
I would tend to favor option 2 as this requires code change at only location. If option 2 is considered, I would recommend to still allow setting the IV via sendmsg CMSG (to keep the interface stable). If, however, the caller provides an IV via recvmsg, this takes precedence.
We definitely want to keep option 1 (which runs on the existing interface and does the magic in driver) for those who want it.
Agreed.
Quoted 10 lines
So the only one left is the case 3 above where the hardware is capable of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of resources.
1) Whenever we want to allocate a new context we spin up a new socket and effectively associate a single IV with that (and it's chained updates) much like we do in the existing interface.
I would not like that because it is too heavyweight. Moreover, considering the kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
Quoted 3 lines
2) We allow a token based tracking of IVs. So userspace code maintains a counter and tags ever message and the initial IV setup with that counter.
I think the option I offer with the patch, we have an even more lightweight approach.
Quoted 9 lines
As the socket typically belongs to a userspace process tag creation can be in userspace and it can ensure it doesn't overlap tags (or it'll get the wrong answer).
Kernel driver can then handle making sure any internal token / addresses are correct. I haven't looked at in depth but would imagine this one would be rather more invasive to support.
Quoted 6 lines
Quoted 1 line
If there are other options, please allow us to learn about them.
Glad we are addressing these usecases and that we have AIO support in general. Makes for a better discussion around whether in kernel support for these interfaces is actually as effective as moving to userspace drivers...
:-)
I would like to have more code in user space than in kernel space...
Ciao
Stephan
Quoted 29 lines · On Mon, 15 Jan 2018 13:07:16 +0100 Stephan Mueller <[email protected]> wrote:
Am Montag, 15. Januar 2018, 12:05:03 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 5 lines
On Fri, 12 Jan 2018 14:21:15 +0100
Quoted 2 lines · Stephan Mueller
Hi,
Quoted 7 lines
Quoted 6 lines
The kernel crypto API requires the caller to set an IV in the request data structure. That request data structure shall define one particular cipher operation. During the cipher operation, the IV is read by the cipher implementation and eventually the potentially updated IV (e.g. in case of CBC) is written back to the memory location the request data structure points to.
Silly question, are we obliged to always write it back?
Well, in general, yes. The
AF_ALGinterface should allow a "stream" mode of operation:socket
accept
setsockopt(setkey)sendmsg(IV, data)
recvmsg(data)
sendmsg(data)
recvmsg(data)
..
Quoted 5 lines
For such synchronous operation, I guess it is clear that the IV needs to be written back.
If you want to play with it, use the "stream" API of libkcapi and the associated test cases.
Thanks for the pointer - will do.
Quoted 68 lines
Quoted 60 lines
In CBC it is
obviously the same as the last n bytes of the encrypted message. I guess for ease of handling it makes sense to do so though.Quoted 41 lines
AF_ALGallows setting the IV with a sendmsg request, where the IV is stored in theAF_ALGcontext that is unique to one particularAF_ALGsocket. Note the analogy: anAF_ALGsocket is like a TFM where one recvmsg operation uses one request with the TFM from the socket.
AF_ALGthese days supports AIO operations with multiple IOCBs. I.e. with one recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived from one IOVEC) implies that one request data structure is created with the data to be processed by the cipher implementation. The IV that was set with the sendmsg call is registered with the request data structure before the cipher operation.In case of an AIO operation, the cipher operation invocation returns immediately, queuing the request to the hardware. While the AIO request is processed by the hardware, recvmsg processes the next IOVEC for which another request is created. Again, the IV buffer from the
AF_ALGsocket context is registered with the new request and the cipher operation is invoked.You may now see that there is a potential race condition regarding the IV handling, because there is *no* separate IV buffer for the different requests. This is nicely demonstrated with libkcapi using the following command which creates an AIO request with two IOCBs each encrypting one AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423-i7fbc02ebf5b93322329df9bfccb635af-p48981da18e4bb9ef7e2e3162d16b1910When the first AIO request finishes before the 2nd AIO request is processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32I.e. two blocks where the IV output from the first request is the IV input to the 2nd block.
In case the first AIO request is not completed before the 2nd request commences, the result is two identical AES blocks (i.e. both use the same IV):
Quoted 8 lines
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71This inconsistent result may even lead to the conclusion that there can be a memory corruption in the IV buffer if both AIO requests write to the IV buffer at the same time.
This needs to be solved somehow. I see the following options which I would like to have vetted by the community.
Taking some 'entirely hypothetical' hardware with the following structure for all my responses - it's about as flexible as I think we'll see in the near future - though I'm sure someone has something more complex out there :)
Quoted 7 lines
N hardware queues feeding M processing engines in a scheduler driven fashion. Actually we might have P sets of these, but load balancing and tracking and transferring contexts between these is a complexity I think we can ignore. If you want to use more than one of these P you'll just have to handle it yourself in userspace. Note messages may be shorter than IOCBs which raises another question I've been meaning to ask. Are all crypto algorithms obliged to run unlimited length IOCBs?
Quoted 7 lines
There are instances where hardware may reject large data chunks. IIRC I have seen some limits around 32k. But in this case, the driver must chunk up the scatter-gather lists (SGLs) with the data and feed it to the hardware in the chunk size necessary.
From the kernel crypto API point of view, the driver must support unlimited sized IOCBs / SGLs.
Hmm. This can be somewhat of a pain to do but fair enough (though the limit in question for us is a lot more than 32k and covers the vast majority of real use cases)
Quoted 61 lines
Quoted 24 lines
If there are M messages in a particular queue and none elsewhere it is capable of processing them all at once (and perhaps returning out of order but we can fudge them back in order in the driver to avoid that additional complexity from an interface point of view).
So I'm going to look at this from the hardware point of view - you have well addressed software management above.
Three ways context management can be handled (in CBC this is basically just the IV).
1. Each 'work item' queued on a hardware queue has it's IV embedded with the data. This requires external synchronization if we are chaining across multiple 'work items' - note the hardware may have restrictions that mean it has to split large pieces of data up to encrypt them. Not all hardware may support per 'work item' IVs (I haven't done a survey to find out if everyone does...)
2. Each queue has a context assigned. We get a new queue whenever we want to have a different context. Runs out eventually but our hypothetical hardware may support a lot of queues. Note this version could be 'faked' by putting a cryptoengine queue on the front of the hardware queues.
Quoted 8 lines
3. The hardware supports IV dependency tracking in it's queues. That is, it can check if the address pointing to the IV is in use by one of the processing units which has not yet updated the IV ready for chaining with the next message. Note it might use a magic token rather than the IV pointer. For modes with out chaining (including counter modes) the IV pointer will inherently always be different.
The hardware then simply schedules something else until it can safely run that particular processing unit.The kernel crypto API has the following concept:
- a TFM holds the data that is stable for an entire cipher operation, such as the key -- one cipher operation may consist of individual calls
- a request structure holds the volatile data, i.e. the data that is valid for one particular call, such as the input plaintext or the IV
Thus, your hardware queue should expect one request and it must be capable of handling that one request with the given data. If you want to split up the request because you have sufficient hardware resources as you mentioned above, your driver/hardware must process the request accordingly.
Coming back to the
AF_ALGinterface: in order to support the aforementioned "stream" mode, the requests for each cipher call invoked by one recvmsg syscall points to the same IV buffer.In case of AIO with multiple IOCBs, user space conceptually calls:
sendmsg
recvmsg
recvmsg
..where all recvmsg calls execute in parallel. As each recvmsg call has one request associated with it, the question is what happens to a buffer that is pointed to by multiple request structures in such parallel execution.
Quoted 8 lines
If your hardware is capable of serializing the recvmsg calls or tracking the dependency, the current
AF_ALGinterface is fully sufficient.But there may be hardware that cannot/will not track such dependencies. Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For such a case,
AF_ALGcurrently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.
Agreed, something like your new support is needed - I just suspect we need a level between one socket one iv chain and every IOCB with own IV and right now the only way to hit that balance is to have a separate socket for each IV chain. Not exactly efficient use of resources though it will work.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
1. Require that the cipher implementations serialize any AIO requests that have dependencies. I.e. for CBC, requests need to be serialized by the driver. For, say, ECB or XTS no serialization is necessary.
There is a certain requirement to do this anyway as we may have a streaming type situation and we don't want to have to do the chaining in userspace.
Quoted 7 lines
Absolutely. If you have proper hardware/driver support, that would be beneficial. This would be supported with the current
AF_ALGinterface.But I guess there are also folks out there who simply want to offer multiple hardware queues to allow independent cipher operations to be invoked concurrently without any dependency handling. This is currently not supported with
AF_ALG.
I'd argue the multiple hardware queues is supported fine in that it is easy to have a driver supporting them via separate AF_ALG sockets. This is what we are currently doing. It means that if two processes come along, they can have their own queues that don't directly influence each other - one isn't blocked behind another (though they are sharing the processing resources so will run slower). One process can of course also open multiple sockets though with your new patch that becomes less important.
The tricky case is the hardware queues where you want to run multiple contexts through a single AF_ALG socket with iv chaining not effectively done in userspace.
Quoted 32 lines
Quoted 16 lines
So we send first X MB block to HW but before it has come back we have more data arrive that needs decrypting so we queue that behind it. The IV then needs to be updated automatically (or the code needs to do it on the first work item coming back). If you don't have option 3 above, you have to do this. This is what I was planning to implement for our existing hardware before you raised this question and I don't think we get around it being necessary for performance in any case. Setting up IOMMUs etc is costly so we want to be doing everything we can before the IV update is ready.
Quoted 5 lines
2. Change
AF_ALGto require a per-request IV. This could be implemented by moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg
code path would obtain the IV.Quoted 7 lines
Quoted 4 lines
I would tend to favor option 2 as this requires code change at only location. If option 2 is considered, I would recommend to still allow setting the IV via sendmsg CMSG (to keep the interface stable). If, however, the caller provides an IV via recvmsg, this takes precedence.
We definitely want to keep option 1 (which runs on the existing interface and does the magic in driver) for those who want it.
Agreed.
Quoted 7 lines
So the only one left is the case 3 above where the hardware is capable of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of resources.
Quoted 7 lines
Quoted 3 lines
1) Whenever we want to allocate a new context we spin up a new socket and effectively associate a single IV with that (and it's chained updates) much like we do in the existing interface.
I would not like that because it is too heavyweight. Moreover, considering the kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
Agreed, though as I mention above if you have multiple processes you probably want to give them their own resources anyway (own socket and probably hardware queue if you can spare one) so as to avoid denial of service from one to another.
Quoted 6 lines
Quoted 3 lines
2) We allow a token based tracking of IVs. So userspace code maintains a counter and tags ever message and the initial IV setup with that counter.
I think the option I offer with the patch, we have an even more lightweight approach.
Except that I think you have to go all the way back to userspace - unless I am missing the point - you can't have multiple elements of a stream queued up. Performance will stink if you have a small number of contexts and can't keep the processing engines busy. At the moment option 1 here is the only way to implement this.
Quoted 11 lines
Quoted 11 lines
As the socket typically belongs to a userspace process tag creation can be in userspace and it can ensure it doesn't overlap tags (or it'll get the wrong answer).
Kernel driver can then handle making sure any internal token / addresses are correct. I haven't looked at in depth but would imagine this one would be rather more invasive to support.
Quoted 1 line
If there are other options, please allow us to learn about them.
Quoted 8 lines
Quoted 4 lines
Glad we are addressing these usecases and that we have AIO support in general. Makes for a better discussion around whether in kernel support for these interfaces is actually as effective as moving to userspace drivers...
:-)
I would like to have more code in user space than in kernel space...
Probably not all of it though, which is basically the alternative here.
Jonathan
Quoted 6 lines
Ciao
Stephan
Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 7 lines
Quoted 6 lines
But there may be hardware that cannot/will not track such dependencies. Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For such a case,
AF_ALGcurrently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.
Quoted 5 lines
Agreed, something like your new support is needed - I just suspect we need a level between one socket one iv chain and every IOCB with own IV and right now the only way to hit that balance is to have a separate socket for each IV chain. Not exactly efficient use of resources though it will work.
How would you propose such support via AF_ALG? Wouldn't it be possible to arrange the IOVEC array in user space appropriately before calling the AF_ALG interface? In this case, I would still see that the current AF_ALG (plus the patch) would support all use cases I am aware of.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
So the only one left is the case 3 above where the hardware is capable of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of
resources.1) Whenever we want to allocate a new context we spin up a new socket and
effectively associate a single IV with that (and it's chained updates) much
like we do in the existing interface.
Quoted 8 lines
Quoted 3 lines
I would not like that because it is too heavyweight. Moreover, considering the kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
Agreed, though as I mention above if you have multiple processes you probably want to give them their own resources anyway (own socket and probably hardware queue if you can spare one) so as to avoid denial of service from one to another.
That is for sure, different processes shall never share a socket as otherwise one can obtain data belonging to the other which would violate process isolation.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
2) We allow a token based tracking of IVs. So userspace code maintains a counter and tags ever message and the initial IV setup with that counter.
I think the option I offer with the patch, we have an even more lightweight
approach.
Quoted 5 lines
Except that I think you have to go all the way back to userspace - unless I am missing the point - you can't have multiple elements of a stream queued up. Performance will stink if you have a small number of contexts and can't keep the processing engines busy. At the moment option 1 here is the only way to implement this.
What AF_ALG should do is to enable different vendors like yourself to use the most appropriate solution. AF_ALG shall not limit users in any way.
Thus, AF_ALG allows multiple sockets, if desired. It allows a stream usage with one setiv call applicable to multiple cipher operations. And with the offered patch it would allow multiple concurrent and yet independent cipher operations. Whichever use case is right for you, AF_ALG should not block you from applying it. Yet, what is good for you may not be good for others. Thus, these others may implement a different usage strategy for AF_ALG. The good thing is that this strategy is defined by user space.
In case you see a use case that is prevented by AF_ALG, it would be great to hear about it to see whether we can support it.
Ciao
Stephan
Quoted 16 lines · On Mon, 15 Jan 2018 14:15:42 +0100 Stephan Mueller <[email protected]> wrote:
Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 7 lines
Quoted 6 lines
But there may be hardware that cannot/will not track such dependencies. Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For such a case,
AF_ALGcurrently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.Quoted 5 lines
Agreed, something like your new support is needed - I just suspect we need a level between one socket one iv chain and every IOCB with own IV and right now the only way to hit that balance is to have a separate socket for each IV chain. Not exactly efficient use of resources though it will work.
Quoted 5 lines
How would you propose such support via
AF_ALG?
Wouldn't it be possible to
arrange the IOVEC array in user space appropriately before calling theAF_ALGinterface? In this case, I would still see that the currentAF_ALG(plus the patch) would support all use cases I am aware of.
I'm not sure how that would work, but maybe I'm missing something - are you suggesting we could contrive the situation where the kernel side can tell it is getting the same IV multiple times and hence know that it should chain it? We are talking streaming here - we don't have the data for the later elements when the first ones are queued up.
One approach to handling token based IV - where the token refers to an IV without being it's value would be to add another flag similar to the one you used for inline IV.
You would then set the IV as you have done, but also provide a magic value by which to track the chain. Later IOCBs using the same IV chain would just provide the magic token.
You'd also need some way of retiring the IV eventually once you were done with it or ultimately you would run out of resources.
Quoted 35 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
So the only one left is the case 3 above where the hardware is capable of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of
resources.1) Whenever we want to allocate a new context we spin up a new socket and
effectively associate a single IV with that (and it's chained updates) much
like we do in the existing interface.Quoted 8 lines
Quoted 3 lines
I would not like that because it is too heavyweight. Moreover, considering the kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
Agreed, though as I mention above if you have multiple processes you probably want to give them their own resources anyway (own socket and probably hardware queue if you can spare one) so as to avoid denial of service from one to another.
That is for sure, different processes shall never share a socket as otherwise one can obtain data belonging to the other which would violate process isolation.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
2) We allow a token based tracking of IVs. So userspace code maintains a counter and tags ever message and the initial IV setup with that counter.
I think the option I offer with the patch, we have an even more lightweight
approach.
Quoted 8 lines
Quoted 5 lines
Except that I think you have to go all the way back to userspace - unless I am missing the point - you can't have multiple elements of a stream queued up. Performance will stink if you have a small number of contexts and can't keep the processing engines busy. At the moment option 1 here is the only way to implement this.
What
AF_ALGshould do is to enable different vendors like yourself to use the most appropriate solution.AF_ALGshall not limit users in any way.
Agreed, but we also need to have some consistency for userspace to have some awareness of what it should be using. Last thing we want is lots of higher level software having to have knowledge of the encryption hardware underneath. Hence I think we should keep the options to the minimum possible or put the burden on drivers that must play well with all options (be it not as efficiently for the ones that work badly for them).
Quoted 9 lines
Thus,
AF_ALGallows multiple sockets, if desired. It allows a stream usage with one setiv call applicable to multiple cipher operations. And with the offered patch it would allow multiple concurrent and yet independent cipher operations. Whichever use case is right for you,AF_ALGshould not block you from applying it. Yet, what is good for you may not be good for others. Thus, these others may implement a different usage strategy forAF_ALG. The good thing is that this strategy is defined by user space.
Quoted 3 lines
In case you see a use case that is prevented by
AF_ALG, it would be great to hear about it to see whether we can support it.
The usecase isn't blocked, but if you have hardware that is doing the IV management then it is not efficiently handled. Either
1) You move the chaining up to userspace - throughput on a given chain will be awful - basically all the advantages of AIO are gone - fine if you know you only care about bandwidth with lots of separate IV chains.
2) You open a socket per IV chain and eat resources.
Jonathan
Quoted 4 lines
Ciao
Stephan
Am Montag, 15. Januar 2018, 15:25:38 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 35 lines
On Mon, 15 Jan 2018 14:15:42 +0100
Quoted 20 lines · Stephan Mueller
Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 8 lines
Quoted 7 lines
But there may be hardware that cannot/will not track such dependencies.
Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For
such a case,AF_ALGcurrently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.Quoted 7 lines
Agreed, something like your new support is needed - I just suspect we need
a level between one socket one iv chain and every IOCB with own IV and right now the only way to hit that balance is to have a separate socket for each IV chain. Not exactly efficient use of resources though it will
work.Quoted 5 lines
How would you propose such support via
AF_ALG?
Wouldn't it be possible to
arrange the IOVEC array in user space appropriately before calling theAF_ALGinterface? In this case, I would still see that the currentAF_ALG(plus the patch) would support all use cases I am aware of.I'm not sure how that would work, but maybe I'm missing something - are you suggesting we could contrive the situation where the kernel side can tell it is getting the same IV multiple times and hence know that it should chain it? We are talking streaming here - we don't have the data for the later elements when the first ones are queued up.
Quoted 3 lines
One approach to handling token based IV - where the token refers to an IV without being it's value would be to add another flag similar to the one you used for inline IV.
What about:
sendmsg(IV, data)
sendmsg(data)
..
AIO recvmsg with multiple IOCBs
AIO recvmsg with multiple IOCBs
..sendmsg(IV, data)
..
This implies, however, that before the sendmsg with the second IV is sent, all AIO operations from the first invocation would need to be finished.
Quoted 7 lines
You would then set the IV as you have done, but also provide a magic value by which to track the chain. Later IOCBs using the same IV chain would just provide the magic token.
You'd also need some way of retiring the IV eventually once you were done with it or ultimately you would run out of resources.
Let me think about that approach a bit.
Quoted 19 lines
Quoted 3 lines
What
AF_ALGshould do is to enable different vendors like yourself to use the most appropriate solution.AF_ALGshall not limit users in any way.Agreed, but we also need to have some consistency for userspace to have some awareness of what it should be using. Last thing we want is lots of higher level software having to have knowledge of the encryption hardware underneath. Hence I think we should keep the options to the minimum possible or put the burden on drivers that must play well with all options (be it not as efficiently for the ones that work badly for them).
Quoted 9 lines
Thus,
AF_ALGallows multiple sockets, if desired. It allows a stream usage with one setiv call applicable to multiple cipher operations. And with the offered patch it would allow multiple concurrent and yet independent cipher
operations. Whichever use case is right for you,AF_ALGshould not block you from applying it. Yet, what is good for you may not be good for others. Thus, these others may implement a different usage strategy forAF_ALG. The good thing is that this strategy is defined by user space.
Quoted 8 lines
Quoted 2 lines
In case you see a use case that is prevented by
AF_ALG, it would be great to hear about it to see whether we can support it.The usecase isn't blocked, but if you have hardware that is doing the IV management then it is not efficiently handled. Either
1) You move the chaining up to userspace - throughput on a given chain will be awful - basically all the advantages of AIO are gone - fine if you
know you only care about bandwidth with lots of separate IV chains.
This sounds not like the right path.
Quoted 2 lines
2) You open a socket per IV chain and eat resources.
Ok, AF_ALG allows this.
Quoted 5 lines
Jonathan
Quoted 2 lines
Ciao
Stephan
Ciao
Stephan
Quoted 54 lines · On Mon, 15 Jan 2018 15:31:42 +0100 Stephan Mueller <[email protected]> wrote:
Am Montag, 15. Januar 2018, 15:25:38 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 35 lines
On Mon, 15 Jan 2018 14:15:42 +0100
Quoted 20 lines · Stephan Mueller
Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 8 lines
Quoted 7 lines
But there may be hardware that cannot/will not track such dependencies.
Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For
such a case,AF_ALGcurrently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.Quoted 7 lines
Agreed, something like your new support is needed - I just suspect we need
a level between one socket one iv chain and every IOCB with own IV and right now the only way to hit that balance is to have a separate socket for each IV chain. Not exactly efficient use of resources though it will
work.Quoted 5 lines
How would you propose such support via
AF_ALG?
Wouldn't it be possible to
arrange the IOVEC array in user space appropriately before calling theAF_ALGinterface? In this case, I would still see that the currentAF_ALG(plus the patch) would support all use cases I am aware of.I'm not sure how that would work, but maybe I'm missing something - are you suggesting we could contrive the situation where the kernel side can tell it is getting the same IV multiple times and hence know that it should chain it? We are talking streaming here - we don't have the data for the later elements when the first ones are queued up.
Quoted 3 lines
One approach to handling token based IV - where the token refers to an IV without being it's value would be to add another flag similar to the one you used for inline IV.
What about:
sendmsg(IV, data)
sendmsg(data)
..
AIO recvmsg with multiple IOCBs
AIO recvmsg with multiple IOCBs
..sendmsg(IV, data)
..
Quoted 2 lines
This implies, however, that before the sendmsg with the second IV is sent, all AIO operations from the first invocation would need to be finished.
Yes that works fine, but rather restricts the flow - you would end up waiting until you could concatenate a bunch of data in userspace so as to trade off against the slow down whenever you need to synchronize back up to userspace.
Quoted 38 lines
Quoted 7 lines
You would then set the IV as you have done, but also provide a magic value by which to track the chain. Later IOCBs using the same IV chain would just provide the magic token.
You'd also need some way of retiring the IV eventually once you were done with it or ultimately you would run out of resources.
Let me think about that approach a bit.
Quoted 19 lines
Quoted 3 lines
What
AF_ALGshould do is to enable different vendors like yourself to use the most appropriate solution.AF_ALGshall not limit users in any way.Agreed, but we also need to have some consistency for userspace to have some awareness of what it should be using. Last thing we want is lots of higher level software having to have knowledge of the encryption hardware underneath. Hence I think we should keep the options to the minimum possible or put the burden on drivers that must play well with all options (be it not as efficiently for the ones that work badly for them).
Quoted 9 lines
Thus,
AF_ALGallows multiple sockets, if desired. It allows a stream usage with one setiv call applicable to multiple cipher operations. And with the offered patch it would allow multiple concurrent and yet independent cipher
operations. Whichever use case is right for you,AF_ALGshould not block you from applying it. Yet, what is good for you may not be good for others. Thus, these others may implement a different usage strategy forAF_ALG. The good thing is that this strategy is defined by user space.Quoted 8 lines
Quoted 2 lines
In case you see a use case that is prevented by
AF_ALG, it would be great to hear about it to see whether we can support it.The usecase isn't blocked, but if you have hardware that is doing the IV management then it is not efficiently handled. Either
1) You move the chaining up to userspace - throughput on a given chain will be awful - basically all the advantages of AIO are gone - fine if you
know you only care about bandwidth with lots of separate IV chains.
Quoted 5 lines
This sounds not like the right path.
Quoted 2 lines
2) You open a socket per IV chain and eat resources.
Ok,
AF_ALGallows this.
That was my plan before this discussion started. Ugly but works without any AF_ALG changes.
We can probably play some internal games to make this not as bad as it initially seems, but still not nice.
Jonathan
Quoted 12 lines
Quoted 5 lines
Jonathan
Quoted 2 lines
Ciao
StephanCiao
Stephan
Am Montag, 15. Januar 2018, 15:42:58 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 11 lines
Quoted 11 lines
What about:
sendmsg(IV, data)
sendmsg(data)
..
AIO recvmsg with multiple IOCBs
AIO recvmsg with multiple IOCBs
..sendmsg(IV, data)
..
Quoted 6 lines
Quoted 2 lines
This implies, however, that before the sendmsg with the second IV is sent, all AIO operations from the first invocation would need to be finished.
Yes that works fine, but rather restricts the flow - you would end up waiting until you could concatenate a bunch of data in userspace so as to trade off against the slow down whenever you need to synchronize back up to userspace.
I think the solution is already present and even libkcapi's architecture is set up to handle this scenario:
We have 2 types of FDs: one obtained from socket() and one from accept(). The socket-FD is akin to the TFM. The accept FD is exactly what you want:
tfmfd = socket()
setkey(tfmfd)
opfd = accept()
opfd2 = accept()sendmsg(opfd, IV, data)recvmsg(opfd, data)
sendmsg(opfd2, IV, data)recvmsg(opfd2, data)
sendmsg(opfd, data)
..
There can be multipe FDs from accept and these are the "identifiers" for your cipher operation stream that belongs together.
libkcapi has already the architecture for this type of work, but it is not exposed to the API yet. The internal calls for sendmsg/recvmsg all take an (op)FD parameter. E.g. _kcapi_common_send_meta_fd has the fdptr variable. internal.h currently wraps this call into _kcapi_common_send_meta where the handle->opfd variable is used.
The idea why I implemented that is because the caller could maintain an array of opfds. If we would expose these internal calls with the FD argument, you can maintain multiple opfds implementing your use case.
The only change would be to expose the internal libkcapi calls.
Ciao
Stephan
Quoted 50 lines · On Tue, 16 Jan 2018 07:28:06 +0100 Stephan Mueller <[email protected]> wrote:
Am Montag, 15. Januar 2018, 15:42:58 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 11 lines
Quoted 11 lines
What about:
sendmsg(IV, data)
sendmsg(data)
..
AIO recvmsg with multiple IOCBs
AIO recvmsg with multiple IOCBs
..sendmsg(IV, data)
..Quoted 6 lines
Quoted 2 lines
This implies, however, that before the sendmsg with the second IV is sent, all AIO operations from the first invocation would need to be finished.
Yes that works fine, but rather restricts the flow - you would end up waiting until you could concatenate a bunch of data in userspace so as to trade off against the slow down whenever you need to synchronize back up to userspace.
I think the solution is already present and even libkcapi's architecture is set up to handle this scenario:
We have 2 types of FDs: one obtained from
socket()and one fromaccept(). The socket-FD is akin to the TFM. The accept FD is exactly what you want:tfmfd =
socket()
setkey(tfmfd)
opfd =accept()
opfd2 =accept()sendmsg(opfd, IV, data)recvmsg(opfd, data)
sendmsg(opfd2, IV, data)recvmsg(opfd2, data)
sendmsg(opfd, data)
..There can be multipe FDs from accept and these are the "identifiers" for your cipher operation stream that belongs together.
libkcapi has already the architecture for this type of work, but it is not exposed to the API yet. The internal calls for sendmsg/recvmsg all take an (op)FD parameter. E.g.
_kcapi_common_send_meta_fdhas the fdptr variable.internal.hcurrently wraps this call into_kcapi_common_send_metawhere thehandle->opfdvariable is used.
Quoted 8 lines
The idea why I implemented that is because the caller could maintain an array of opfds. If we would expose these internal calls with the FD argument, you can maintain multiple opfds implementing your use case.
The only change would be to expose the internal libkcapi calls.
Ciao
Stephan
Thanks, I'll take a look at this soonish. Having a busy week.
Jonathan
Quoted 2 lines
Quoted 106 lines · On Mon, 15 Jan 2018 14:25:38 +0000 Jonathan Cameron…
Quoted 16 lines · On Mon, 15 Jan 2018 14:15:42 +0100 Stephan Mueller <[email protected]> wrote:
Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
Hi Jonathan,
Quoted 7 lines
Quoted 6 lines
But there may be hardware that cannot/will not track such dependencies. Yet, it has multiple hardware queues. Such hardware can still handle parallel requests when they are totally independent from each other. For such a case,
AF_ALGcurrently has no support, because it lacks the support for setting multiple IVs for the multiple concurrent calls.Quoted 5 lines
Agreed, something like your new support is needed - I just suspect we need a level between one socket one iv chain and every IOCB with own IV and right now the only way to hit that balance is to have a separate socket for each IV chain. Not exactly efficient use of resources though it will work.
Quoted 5 lines
How would you propose such support via
AF_ALG?
Wouldn't it be possible to
arrange the IOVEC array in user space appropriately before calling theAF_ALGinterface? In this case, I would still see that the currentAF_ALG(plus the patch) would support all use cases I am aware of.I'm not sure how that would work, but maybe I'm missing something - are you suggesting we could contrive the situation where the kernel side can tell it is getting the same IV multiple times and hence know that it should chain it? We are talking streaming here - we don't have the data for the later elements when the first ones are queued up.
One approach to handling token based IV - where the token refers to an IV without being it's value would be to add another flag similar to the one you used for inline IV.
You would then set the IV as you have done, but also provide a magic value by which to track the chain. Later IOCBs using the same IV chain would just provide the magic token.
You'd also need some way of retiring the IV eventually once you were done with it or ultimately you would run out of resources.
Quoted 35 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
So the only one left is the case 3 above where the hardware is capable of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of
resources.1) Whenever we want to allocate a new context we spin up a new socket and
effectively associate a single IV with that (and it's chained updates) much
like we do in the existing interface.Quoted 8 lines
Quoted 3 lines
I would not like that because it is too heavyweight. Moreover, considering the kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
Agreed, though as I mention above if you have multiple processes you probably want to give them their own resources anyway (own socket and probably hardware queue if you can spare one) so as to avoid denial of service from one to another.
That is for sure, different processes shall never share a socket as otherwise one can obtain data belonging to the other which would violate process isolation.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
2) We allow a token based tracking of IVs. So userspace code maintains a counter and tags ever message and the initial IV setup with that counter.
I think the option I offer with the patch, we have an even more lightweight
approach.Quoted 8 lines
Quoted 5 lines
Except that I think you have to go all the way back to userspace - unless I am missing the point - you can't have multiple elements of a stream queued up. Performance will stink if you have a small number of contexts and can't keep the processing engines busy. At the moment option 1 here is the only way to implement this.
What
AF_ALGshould do is to enable different vendors like yourself to use the most appropriate solution.AF_ALGshall not limit users in any way.Agreed, but we also need to have some consistency for userspace to have some awareness of what it should be using. Last thing we want is lots of higher level software having to have knowledge of the encryption hardware underneath. Hence I think we should keep the options to the minimum possible or put the burden on drivers that must play well with all options (be it not as efficiently for the ones that work badly for them).
Quoted 9 lines
Thus,
AF_ALGallows multiple sockets, if desired. It allows a stream usage with one setiv call applicable to multiple cipher operations. And with the offered patch it would allow multiple concurrent and yet independent cipher operations. Whichever use case is right for you,AF_ALGshould not block you from applying it. Yet, what is good for you may not be good for others. Thus, these others may implement a different usage strategy forAF_ALG. The good thing is that this strategy is defined by user space.Quoted 3 lines
In case you see a use case that is prevented by
AF_ALG, it would be great to hear about it to see whether we can support it.
Quoted 7 lines
The usecase isn't blocked, but if you have hardware that is doing the IV management then it is not efficiently handled. Either
1) You move the chaining up to userspace - throughput on a given chain will be awful - basically all the advantages of AIO are gone - fine if you know you only care about bandwidth with lots of separate IV chains.2) You open a socket per IV chain and eat resources.
Actually thinking more on this its worse, a given driver may need to know in advance which of these two options it is working with. Is it expected to do dependency tracking and serialization or not?
In case 1 you would just have a software queue and be able to assume that whenever you got something back from the hardware you could stick in the next IOCB and the IV would be right.
In case 2 you would want to push the whole lot to the hardware queue asap.
Given for 2 we are explicitly providing the IV I guess we could use that info to conclude any chaining was up in userspace but that would only work if it was very much and either or case which isn't nice.
Jonathan
Quoted 12 lines
Jonathan
Quoted 4 lines
Ciao
Stephan_______________________________________________
Linuxarm mailing list
[email protected]
http://hulk.huawei.com/mailman/listinfo/linuxarm