Skip to contentSkip to conversation

crypto: atmel-authenc: add support to authenc(hmac(shaX),Y(aes)) modes

PATCH In review Cyrille Pitchen 2 participants 14 messages Jan 26, 2017 – Feb 3, 2017

Subsystems (2)
Crypto API Arm/Microchip (AT91) SoC support
lore
Share & tools
lore

Cyrille Pitchen

author
#0 lore raw

v3 cover crypto: atmel-authenc: add support to authenc(hmac(shaX),Y(aes)) modes

Hi all,

this series of patches has been based and tested on next-20170125 with CRYPTO_MANAGER_DISABLED_TESTS not set.

The series adds support to the hmac(shaX) algorithms first, then combines both the Atmel SHA and AES hardware accelerators to implement authenc(hmac(shaX),Y(aes)) algorithms as used by IPSEC/SSL connections.

It has also been tested with strongswan + xl2tpd to create an IPSEC+L2TP (transport mode) VPN and strongswan only (tunnel mode) for an IPSEC VPN.

Then iperf was used to measure the bandwidth improvement in tunnel mode:

drivers AES SHA SPLIP iperf half-duplex
Mbit/s

authenc(hmac(sha1-generic),cbc(aes))        SW   SW  N/A 27.7
authenc(hmac(sha1-generic),atmel-cbc-aes)   HW   SW  N/A 30.2 (mainline)
authenc(atmel-hmac-sha1,atmel-cbc-aes)      HW   HW   no 29.1
atmel-authenc-hmac-sha1-cbc-aes             HW   HW  yes 38.8

SPLIP: Secure Protocol Layers Improved Performances (AES+SHA combined).

Some patches of this series are purely transitional: I've split the modifications into many patches to ease the review.

Best regards,

Cyrille

ChangeLog:

v2 -> v3:
- add calls to memzero_explicit(&keys, ...) before exiting from atmel_aes_authenc_setkey().
- add missing comparison test between req->cryptlen and authsize values in atmel_aes_authenc_crypt().
- remove the atmel_aes_authenc_copy_assoc() function: I have tested with a strongswan IPSec ESP Tunnel Mode connection and it still works even if this driver no longer copies the AAD. FWI, I have also tested with the crypto/authenc.c driver after having removed the crypto_authenc_copy_assoc() function and it worked as well. However I didn't check whether I was in the special case req->src == req->dst.

v1 -> v2:
- add missing drivers/crypto/atmel-authenc.h file in patch 11.

Cyrille Pitchen (12):
crypto: atmel-sha: create function to get an Atmel SHA device
crypto: atmel-sha: update request queue management to make it more
generic
crypto: atmel-sha: make atmel_sha_done_task more generic
crypto: atmel-sha: redefine SHA_FLAGS_SHA* flags to match
SHA_MR_ALGO_SHA*
crypto: atmel-sha: add atmel_sha_wait_for_data_ready()
crypto: atmel-sha: add SHA_MR_MODE_IDATAR0
crypto: atmel-sha: add atmel_sha_cpu_start()
crypto: atmel-sha: add simple DMA transfers
crypto: atmel-sha: add support to hmac(shaX)
crypto: atmel-aes: fix atmel_aes_handle_queue()
crypto: atmel-authenc: add support to authenc(hmac(shaX),Y(aes)) modes
crypto: atmel-sha: add verbose debug facilities to print hw register
names

 drivers/crypto/Kconfig          |   12 +
 drivers/crypto/atmel-aes-regs.h |   16 +
 drivers/crypto/atmel-aes.c      |  455 ++++++++++++-
 drivers/crypto/atmel-authenc.h  |   64 ++
 drivers/crypto/atmel-sha-regs.h |   20 +
 drivers/crypto/atmel-sha.c      | 1438 +++++++++++++++++++++++++++++++++++++--
 6 files changed, 1937 insertions(+), 68 deletions(-)
 create mode 100644 drivers/crypto/atmel-authenc.h
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v3 1/12 crypto: atmel-sha: create function to get an Atmel SHA device

This is a transitional patch: it creates the atmel_sha_find_dev() function, which will be used in further patches to share the source code responsible for finding a Atmel SHA device.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
drivers/crypto/atmel-sha.c +11 -4
index 97e34799e077..33a36e667547 100644
@@ -260,11 +260,8 @@ static void atmel_sha_fill_padding(struct atmel_sha_reqctx *ctx, int length)
 	}
 }
 
-static int atmel_sha_init(struct ahash_request *req)
+static struct atmel_sha_dev *atmel_sha_find_dev(struct atmel_sha_ctx *tctx)
 {
-	struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
-	struct atmel_sha_ctx *tctx = crypto_ahash_ctx(tfm);
-	struct atmel_sha_reqctx *ctx = ahash_request_ctx(req);
 	struct atmel_sha_dev *dd = NULL;
 	struct atmel_sha_dev *tmp;
 

@@ -281,6 +278,16 @@ static int atmel_sha_init(struct ahash_request *req)
 
 	spin_unlock_bh(&atmel_sha.lock);
 
+	return dd;
+}
+
+static int atmel_sha_init(struct ahash_request *req)
+{
+	struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+	struct atmel_sha_ctx *tctx = crypto_ahash_ctx(tfm);
+	struct atmel_sha_reqctx *ctx = ahash_request_ctx(req);
+	struct atmel_sha_dev *dd = atmel_sha_find_dev(tctx);
+
 	ctx->dd = dd;
 
 	ctx->flags = 0;
Signature · 1 line
2.7.4
author
#2 lore raw

PATCH v3 2/12 crypto: atmel-sha: update request queue management to make it more generic

In reply to Cyrille Pitchen

This patch is a transitional patch. It splits the atmel_sha_handle_queue() function. Now atmel_sha_handle_queue() only manages the request queue and calls a new .start() hook from the atmel_sha_ctx structure. This hook allows to implement different kind of requests still handled by a single queue.

Also when the req parameter of atmel_sha_handle_queue() refers to the very same request as the one returned by crypto_dequeue_request(), the queue management now gives a chance to this crypto request to be handled synchronously, hence reducing latencies. The .start() hook returns 0 if the crypto request was handled synchronously and -EINPROGRESS if the crypto request still need to be handled asynchronously.

Besides, the new .is_async member of the atmel_sha_dev structure helps tagging this asynchronous state. Indeed, the req->base.complete() callback should not be called if the crypto request is handled synchronously.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 74 +++++++++++++++++++++++++++++++++-------------
 1 file changed, 54 insertions(+), 20 deletions(-)
drivers/crypto/atmel-sha.c +54 -20
index 33a36e667547..2dbed8bb8d26 100644
@@ -105,8 +105,11 @@ struct atmel_sha_reqctx {
 	u8 buffer[SHA_BUFFER_LEN + SHA512_BLOCK_SIZE] __aligned(sizeof(u32));
 };
 
+typedef int (*atmel_sha_fn_t)(struct atmel_sha_dev *);
+
 struct atmel_sha_ctx {
 	struct atmel_sha_dev	*dd;
+	atmel_sha_fn_t		start;
 
 	unsigned long		flags;
 };

@@ -134,6 +137,7 @@ struct atmel_sha_dev {
 	unsigned long		flags;
 	struct crypto_queue	queue;
 	struct ahash_request	*req;
+	bool			is_async;
 
 	struct atmel_sha_dma	dma_lch_in;
 

@@ -163,6 +167,24 @@ static inline void atmel_sha_write(struct atmel_sha_dev *dd,
 	writel_relaxed(value, dd->io_base + offset);
 }
 
+static inline int atmel_sha_complete(struct atmel_sha_dev *dd, int err)
+{
+	struct ahash_request *req = dd->req;
+
+	dd->flags &= ~(SHA_FLAGS_BUSY | SHA_FLAGS_FINAL | SHA_FLAGS_CPU |
+		       SHA_FLAGS_DMA_READY | SHA_FLAGS_OUTPUT_READY);
+
+	clk_disable(dd->iclk);
+
+	if (dd->is_async && req->base.complete)
+		req->base.complete(&req->base, err);
+
+	/* handle new request */
+	tasklet_schedule(&dd->queue_task);
+
+	return err;
+}
+
 static size_t atmel_sha_append_sg(struct atmel_sha_reqctx *ctx)
 {
 	size_t count;

@@ -474,6 +496,8 @@ static void atmel_sha_dma_callback(void *data)
 {
 	struct atmel_sha_dev *dd = data;
 
+	dd->is_async = true;
+
 	/* dma_lch_in - completed - wait DATRDY */
 	atmel_sha_write(dd, SHA_IER, SHA_INT_DATARDY);
 }

@@ -509,7 +533,7 @@ static int atmel_sha_xmit_dma(struct atmel_sha_dev *dd, dma_addr_t dma_addr1,
 			DMA_MEM_TO_DEV, DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
 	}
 	if (!in_desc)
-		return -EINVAL;
+		atmel_sha_complete(dd, -EINVAL);
 
 	in_desc->callback = atmel_sha_dma_callback;
 	in_desc->callback_param = dd;

@@ -566,7 +590,7 @@ static int atmel_sha_xmit_dma_map(struct atmel_sha_dev *dd,
 	if (dma_mapping_error(dd->dev, ctx->dma_addr)) {
 		dev_err(dd->dev, "dma %u bytes error\n", ctx->buflen +
 				ctx->block_size);
-		return -EINVAL;
+		atmel_sha_complete(dd, -EINVAL);
 	}
 
 	ctx->flags &= ~SHA_FLAGS_SG;

@@ -657,7 +681,7 @@ static int atmel_sha_update_dma_start(struct atmel_sha_dev *dd)
 		if (dma_mapping_error(dd->dev, ctx->dma_addr)) {
 			dev_err(dd->dev, "dma %u bytes error\n",
 				ctx->buflen + ctx->block_size);
-			return -EINVAL;
+			atmel_sha_complete(dd, -EINVAL);
 		}
 
 		if (length == 0) {

@@ -671,7 +695,7 @@ static int atmel_sha_update_dma_start(struct atmel_sha_dev *dd)
 			if (!dma_map_sg(dd->dev, ctx->sg, 1,
 				DMA_TO_DEVICE)) {
 					dev_err(dd->dev, "dma_map_sg  error\n");
-					return -EINVAL;
+					atmel_sha_complete(dd, -EINVAL);
 			}
 
 			ctx->flags |= SHA_FLAGS_SG;

@@ -685,7 +709,7 @@ static int atmel_sha_update_dma_start(struct atmel_sha_dev *dd)
 
 	if (!dma_map_sg(dd->dev, ctx->sg, 1, DMA_TO_DEVICE)) {
 		dev_err(dd->dev, "dma_map_sg  error\n");
-		return -EINVAL;
+		atmel_sha_complete(dd, -EINVAL);
 	}
 
 	ctx->flags |= SHA_FLAGS_SG;

@@ -843,16 +867,7 @@ static void atmel_sha_finish_req(struct ahash_request *req, int err)
 	}
 
 	/* atomic operation is not needed here */
-	dd->flags &= ~(SHA_FLAGS_BUSY | SHA_FLAGS_FINAL | SHA_FLAGS_CPU |
-			SHA_FLAGS_DMA_READY | SHA_FLAGS_OUTPUT_READY);
-
-	clk_disable(dd->iclk);
-
-	if (req->base.complete)
-		req->base.complete(&req->base, err);
-
-	/* handle new request */
-	tasklet_schedule(&dd->queue_task);
+	(void)atmel_sha_complete(dd, err);
 }
 
 static int atmel_sha_hw_init(struct atmel_sha_dev *dd)

@@ -893,8 +908,9 @@ static int atmel_sha_handle_queue(struct atmel_sha_dev *dd,
 				  struct ahash_request *req)
 {
 	struct crypto_async_request *async_req, *backlog;
-	struct atmel_sha_reqctx *ctx;
+	struct atmel_sha_ctx *ctx;
 	unsigned long flags;
+	bool start_async;
 	int err = 0, ret = 0;
 
 	spin_lock_irqsave(&dd->lock, flags);

@@ -919,9 +935,22 @@ static int atmel_sha_handle_queue(struct atmel_sha_dev *dd,
 	if (backlog)
 		backlog->complete(backlog, -EINPROGRESS);
 
-	req = ahash_request_cast(async_req);
-	dd->req = req;
-	ctx = ahash_request_ctx(req);
+	ctx = crypto_tfm_ctx(async_req->tfm);
+
+	dd->req = ahash_request_cast(async_req);
+	start_async = (dd->req != req);
+	dd->is_async = start_async;
+
+	/* WARNING: ctx->start() MAY change dd->is_async. */
+	err = ctx->start(dd);
+	return (start_async) ? ret : err;
+}
+
+static int atmel_sha_start(struct atmel_sha_dev *dd)
+{
+	struct ahash_request *req = dd->req;
+	struct atmel_sha_reqctx *ctx = ahash_request_ctx(req);
+	int err;
 
 	dev_dbg(dd->dev, "handling new req, op: %lu, nbytes: %d\n",
 						ctx->op, req->nbytes);

@@ -947,7 +976,7 @@ static int atmel_sha_handle_queue(struct atmel_sha_dev *dd,
 
 	dev_dbg(dd->dev, "exit, err: %d\n", err);
 
-	return ret;
+	return err;
 }
 
 static int atmel_sha_enqueue(struct ahash_request *req, unsigned int op)

@@ -1043,8 +1072,11 @@ static int atmel_sha_import(struct ahash_request *req, const void *in)
 
 static int atmel_sha_cra_init(struct crypto_tfm *tfm)
 {
+	struct atmel_sha_ctx *ctx = crypto_tfm_ctx(tfm);
+
 	crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
 				 sizeof(struct atmel_sha_reqctx));
+	ctx->start = atmel_sha_start;
 
 	return 0;
 }

@@ -1188,6 +1220,8 @@ static void atmel_sha_done_task(unsigned long data)
 	struct atmel_sha_dev *dd = (struct atmel_sha_dev *)data;
 	int err = 0;
 
+	dd->is_async = true;
+
 	if (SHA_FLAGS_CPU & dd->flags) {
 		if (SHA_FLAGS_OUTPUT_READY & dd->flags) {
 			dd->flags &= ~SHA_FLAGS_OUTPUT_READY;
Signature · 1 line
2.7.4
author
#3 lore raw

PATCH v3 3/12 crypto: atmel-sha: make atmel_sha_done_task more generic

In reply to Cyrille Pitchen

This patch is a transitional patch. It updates atmel_sha_done_task() to make it more generic. Indeed, it adds a new .resume() member in the atmel_sha_dev structure. This hook is called from atmel_sha_done_task() to resume processing an asynchronous request.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
drivers/crypto/atmel-sha.c +16 -5
index 2dbed8bb8d26..643d79a05dda 100644
@@ -138,6 +138,7 @@ struct atmel_sha_dev {
 	struct crypto_queue	queue;
 	struct ahash_request	*req;
 	bool			is_async;
+	atmel_sha_fn_t		resume;
 
 	struct atmel_sha_dma	dma_lch_in;
 

@@ -946,6 +947,8 @@ static int atmel_sha_handle_queue(struct atmel_sha_dev *dd,
 	return (start_async) ? ret : err;
 }
 
+static int atmel_sha_done(struct atmel_sha_dev *dd);
+
 static int atmel_sha_start(struct atmel_sha_dev *dd)
 {
 	struct ahash_request *req = dd->req;

@@ -960,6 +963,7 @@ static int atmel_sha_start(struct atmel_sha_dev *dd)
 	if (err)
 		goto err1;
 
+	dd->resume = atmel_sha_done;
 	if (ctx->op == SHA_OP_UPDATE) {
 		err = atmel_sha_update_req(dd);
 		if (err != -EINPROGRESS && (ctx->flags & SHA_FLAGS_FINUP))

@@ -1215,13 +1219,10 @@ static void atmel_sha_queue_task(unsigned long data)
 	atmel_sha_handle_queue(dd, NULL);
 }
 
-static void atmel_sha_done_task(unsigned long data)
+static int atmel_sha_done(struct atmel_sha_dev *dd)
 {
-	struct atmel_sha_dev *dd = (struct atmel_sha_dev *)data;
 	int err = 0;
 
-	dd->is_async = true;
-
 	if (SHA_FLAGS_CPU & dd->flags) {
 		if (SHA_FLAGS_OUTPUT_READY & dd->flags) {
 			dd->flags &= ~SHA_FLAGS_OUTPUT_READY;

@@ -1245,11 +1246,21 @@ static void atmel_sha_done_task(unsigned long data)
 				goto finish;
 		}
 	}
-	return;
+	return err;
 
 finish:
 	/* finish curent request */
 	atmel_sha_finish_req(dd->req, err);
+
+	return err;
+}
+
+static void atmel_sha_done_task(unsigned long data)
+{
+	struct atmel_sha_dev *dd = (struct atmel_sha_dev *)data;
+
+	dd->is_async = true;
+	(void)dd->resume(dd);
 }
 
 static irqreturn_t atmel_sha_irq(int irq, void *dev_id)
Signature · 1 line
2.7.4
author
#4 lore raw

PATCH v3 4/12 crypto: atmel-sha: redefine SHA_FLAGS_SHA* flags to match SHA_MR_ALGO_SHA*

In reply to Cyrille Pitchen

This patch modifies the SHA_FLAGS_SHA* flags: those algo flags are now organized as values of a single bitfield instead of individual bits. This allows to reduce the number of bits needed to encode all possible values. Also the new values match the SHA_MR_ALGO_SHA* values hence the algorithm bitfield of the SHA_MR register could simply be set with:

mr = (mr & ~SHA_FLAGS_ALGO_MASK) | (ctx->flags & SHA_FLAGS_ALGO_MASK)

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha-regs.h |  1 +
 drivers/crypto/atmel-sha.c      | 45 +++++++++++++++++++++++++++++------------
 2 files changed, 33 insertions(+), 13 deletions(-)
drivers/crypto/atmel-sha-regs.h +1 -0
index e08897109cab..deb0b0b15096 100644
@@ -19,6 +19,7 @@
 #define SHA_MR_PROCDLY			(1 << 4)
 #define SHA_MR_UIHV			(1 << 5)
 #define SHA_MR_UIEHV			(1 << 6)
+#define SHA_MR_ALGO_MASK		GENMASK(10, 8)
 #define SHA_MR_ALGO_SHA1		(0 << 8)
 #define SHA_MR_ALGO_SHA256		(1 << 8)
 #define SHA_MR_ALGO_SHA384		(2 << 8)
drivers/crypto/atmel-sha.c +32 -13

82 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#5 lore raw

PATCH v3 5/12 crypto: atmel-sha: add atmel_sha_wait_for_data_ready()

In reply to Cyrille Pitchen

This patch simply defines a helper function to test the 'Data Ready' flag of the Status Register. It also gives a chance for the crypto request to be processed synchronously if this 'Data Ready' flag is already set when polling the Status Register. Indeed, running synchronously avoid the latency of the 'Data Ready' interrupt.

When the 'Data Ready' flag has not been set yet, we enable the associated interrupt and resume processing the crypto request asynchronously from the 'done' task just as before.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/crypto/atmel-sha.c +13 -0
index b29a4e5bc404..be0d72cf4352 100644
@@ -434,6 +434,19 @@ static void atmel_sha_write_ctrl(struct atmel_sha_dev *dd, int dma)
 	atmel_sha_write(dd, SHA_MR, valmr);
 }
 
+static inline int atmel_sha_wait_for_data_ready(struct atmel_sha_dev *dd,
+						atmel_sha_fn_t resume)
+{
+	u32 isr = atmel_sha_read(dd, SHA_ISR);
+
+	if (unlikely(isr & SHA_INT_DATARDY))
+		return resume(dd);
+
+	dd->resume = resume;
+	atmel_sha_write(dd, SHA_IER, SHA_INT_DATARDY);
+	return -EINPROGRESS;
+}
+
 static int atmel_sha_xmit_cpu(struct atmel_sha_dev *dd, const u8 *buf,
 			      size_t length, int final)
 {
Signature · 1 line
2.7.4
author
#6 lore raw

PATCH v3 6/12 crypto: atmel-sha: add SHA_MR_MODE_IDATAR0

In reply to Cyrille Pitchen

This patch defines an alias macro to SHA_MR_MODE_PDC, which is not suited for DMA usage.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha-regs.h | 1 +
 1 file changed, 1 insertion(+)
drivers/crypto/atmel-sha-regs.h +1 -0
index deb0b0b15096..8d62d31eda08 100644
@@ -16,6 +16,7 @@
 #define SHA_MR_MODE_MANUAL		0x0
 #define SHA_MR_MODE_AUTO		0x1
 #define SHA_MR_MODE_PDC			0x2
+#define SHA_MR_MODE_IDATAR0		0x2
 #define SHA_MR_PROCDLY			(1 << 4)
 #define SHA_MR_UIHV			(1 << 5)
 #define SHA_MR_UIEHV			(1 << 6)
Signature · 1 line
2.7.4
author
#7 lore raw

PATCH v3 7/12 crypto: atmel-sha: add atmel_sha_cpu_start()

In reply to Cyrille Pitchen

This patch adds a simple function to perform data transfer with PIO, hence handled by the CPU.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)
drivers/crypto/atmel-sha.c +90 -0
index be0d72cf4352..58d9ca8ac0f2 100644
@@ -64,6 +64,8 @@
 #define SHA_FLAGS_ERROR		BIT(23)
 #define SHA_FLAGS_PAD		BIT(24)
 #define SHA_FLAGS_RESTORE	BIT(25)
+#define SHA_FLAGS_IDATAR0	BIT(26)
+#define SHA_FLAGS_WAIT_DATARDY	BIT(27)
 
 #define SHA_OP_UPDATE	1
 #define SHA_OP_FINAL	2

@@ -141,6 +143,7 @@ struct atmel_sha_dev {
 	struct ahash_request	*req;
 	bool			is_async;
 	atmel_sha_fn_t		resume;
+	atmel_sha_fn_t		cpu_transfer_complete;
 
 	struct atmel_sha_dma	dma_lch_in;
 

@@ -1317,6 +1320,93 @@ static irqreturn_t atmel_sha_irq(int irq, void *dev_id)
 	return IRQ_NONE;
 }
 
+
+/* CPU transfer functions */
+
+static int atmel_sha_cpu_transfer(struct atmel_sha_dev *dd)
+{
+	struct ahash_request *req = dd->req;
+	struct atmel_sha_reqctx *ctx = ahash_request_ctx(req);
+	const u32 *words = (const u32 *)ctx->buffer;
+	size_t i, num_words;
+	u32 isr, din, din_inc;
+
+	din_inc = (ctx->flags & SHA_FLAGS_IDATAR0) ? 0 : 1;
+	for (;;) {
+		/* Write data into the Input Data Registers. */
+		num_words = DIV_ROUND_UP(ctx->bufcnt, sizeof(u32));
+		for (i = 0, din = 0; i < num_words; ++i, din += din_inc)
+			atmel_sha_write(dd, SHA_REG_DIN(din), words[i]);
+
+		ctx->offset += ctx->bufcnt;
+		ctx->total -= ctx->bufcnt;
+
+		if (!ctx->total)
+			break;
+
+		/*
+		 * Prepare next block:
+		 * Fill ctx->buffer now with the next data to be written into
+		 * IDATARx: it gives time for the SHA hardware to process
+		 * the current data so the SHA_INT_DATARDY flag might be set
+		 * in SHA_ISR when polling this register at the beginning of
+		 * the next loop.
+		 */
+		ctx->bufcnt = min_t(size_t, ctx->block_size, ctx->total);
+		scatterwalk_map_and_copy(ctx->buffer, ctx->sg,
+					 ctx->offset, ctx->bufcnt, 0);
+
+		/* Wait for hardware to be ready again. */
+		isr = atmel_sha_read(dd, SHA_ISR);
+		if (!(isr & SHA_INT_DATARDY)) {
+			/* Not ready yet. */
+			dd->resume = atmel_sha_cpu_transfer;
+			atmel_sha_write(dd, SHA_IER, SHA_INT_DATARDY);
+			return -EINPROGRESS;
+		}
+	}
+
+	if (unlikely(!(ctx->flags & SHA_FLAGS_WAIT_DATARDY)))
+		return dd->cpu_transfer_complete(dd);
+
+	return atmel_sha_wait_for_data_ready(dd, dd->cpu_transfer_complete);
+}
+
+static int atmel_sha_cpu_start(struct atmel_sha_dev *dd,
+			       struct scatterlist *sg,
+			       unsigned int len,
+			       bool idatar0_only,
+			       bool wait_data_ready,
+			       atmel_sha_fn_t resume)
+{
+	struct ahash_request *req = dd->req;
+	struct atmel_sha_reqctx *ctx = ahash_request_ctx(req);
+
+	if (!len)
+		return resume(dd);
+
+	ctx->flags &= ~(SHA_FLAGS_IDATAR0 | SHA_FLAGS_WAIT_DATARDY);
+
+	if (idatar0_only)
+		ctx->flags |= SHA_FLAGS_IDATAR0;
+
+	if (wait_data_ready)
+		ctx->flags |= SHA_FLAGS_WAIT_DATARDY;
+
+	ctx->sg = sg;
+	ctx->total = len;
+	ctx->offset = 0;
+
+	/* Prepare the first block to be written. */
+	ctx->bufcnt = min_t(size_t, ctx->block_size, ctx->total);
+	scatterwalk_map_and_copy(ctx->buffer, ctx->sg,
+				 ctx->offset, ctx->bufcnt, 0);
+
+	dd->cpu_transfer_complete = resume;
+	return atmel_sha_cpu_transfer(dd);
+}
+
+
 static void atmel_sha_unregister_algs(struct atmel_sha_dev *dd)
 {
 	int i;
Signature · 1 line
2.7.4
author
#8 lore raw

PATCH v3 8/12 crypto: atmel-sha: add simple DMA transfers

In reply to Cyrille Pitchen

This patch adds a simple function to perform data transfer with the DMA controller.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 116 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)
drivers/crypto/atmel-sha.c +116 -0
index 58d9ca8ac0f2..a4fc60b67099 100644
@@ -123,6 +123,9 @@ struct atmel_sha_ctx {
 struct atmel_sha_dma {
 	struct dma_chan			*chan;
 	struct dma_slave_config dma_conf;
+	struct scatterlist	*sg;
+	int			nents;
+	unsigned int		last_sg_length;
 };
 
 struct atmel_sha_dev {

@@ -1321,6 +1324,119 @@ static irqreturn_t atmel_sha_irq(int irq, void *dev_id)
 }
 
 
+/* DMA transfer functions */
+
+static bool atmel_sha_dma_check_aligned(struct atmel_sha_dev *dd,
+					struct scatterlist *sg,
+					size_t len)
+{
+	struct atmel_sha_dma *dma = &dd->dma_lch_in;
+	struct ahash_request *req = dd->req;
+	struct atmel_sha_reqctx *ctx = ahash_request_ctx(req);
+	size_t bs = ctx->block_size;
+	int nents;
+
+	for (nents = 0; sg; sg = sg_next(sg), ++nents) {
+		if (!IS_ALIGNED(sg->offset, sizeof(u32)))
+			return false;
+
+		/*
+		 * This is the last sg, the only one that is allowed to
+		 * have an unaligned length.
+		 */
+		if (len <= sg->length) {
+			dma->nents = nents + 1;
+			dma->last_sg_length = sg->length;
+			sg->length = ALIGN(len, sizeof(u32));
+			return true;
+		}
+
+		/* All other sg lengths MUST be aligned to the block size. */
+		if (!IS_ALIGNED(sg->length, bs))
+			return false;
+
+		len -= sg->length;
+	}
+
+	return false;
+}
+
+static void atmel_sha_dma_callback2(void *data)
+{
+	struct atmel_sha_dev *dd = data;
+	struct atmel_sha_dma *dma = &dd->dma_lch_in;
+	struct scatterlist *sg;
+	int nents;
+
+	dmaengine_terminate_all(dma->chan);
+	dma_unmap_sg(dd->dev, dma->sg, dma->nents, DMA_TO_DEVICE);
+
+	sg = dma->sg;
+	for (nents = 0; nents < dma->nents - 1; ++nents)
+		sg = sg_next(sg);
+	sg->length = dma->last_sg_length;
+
+	dd->is_async = true;
+	(void)atmel_sha_wait_for_data_ready(dd, dd->resume);
+}
+
+static int atmel_sha_dma_start(struct atmel_sha_dev *dd,
+			       struct scatterlist *src,
+			       size_t len,
+			       atmel_sha_fn_t resume)
+{
+	struct atmel_sha_dma *dma = &dd->dma_lch_in;
+	struct dma_slave_config *config = &dma->dma_conf;
+	struct dma_chan *chan = dma->chan;
+	struct dma_async_tx_descriptor *desc;
+	dma_cookie_t cookie;
+	unsigned int sg_len;
+	int err;
+
+	dd->resume = resume;
+
+	/*
+	 * dma->nents has already been initialized by
+	 * atmel_sha_dma_check_aligned().
+	 */
+	dma->sg = src;
+	sg_len = dma_map_sg(dd->dev, dma->sg, dma->nents, DMA_TO_DEVICE);
+	if (!sg_len) {
+		err = -ENOMEM;
+		goto exit;
+	}
+
+	config->src_maxburst = 16;
+	config->dst_maxburst = 16;
+	err = dmaengine_slave_config(chan, config);
+	if (err)
+		goto unmap_sg;
+
+	desc = dmaengine_prep_slave_sg(chan, dma->sg, sg_len, DMA_MEM_TO_DEV,
+				       DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+	if (!desc) {
+		err = -ENOMEM;
+		goto unmap_sg;
+	}
+
+	desc->callback = atmel_sha_dma_callback2;
+	desc->callback_param = dd;
+	cookie = dmaengine_submit(desc);
+	err = dma_submit_error(cookie);
+	if (err)
+		goto unmap_sg;
+
+	dma_async_issue_pending(chan);
+
+	return -EINPROGRESS;
+
+unmap_sg:
+	dma_unmap_sg(dd->dev, dma->sg, dma->nents, DMA_TO_DEVICE);
+exit:
+	return atmel_sha_complete(dd, err);
+}
+
+
 /* CPU transfer functions */
 
 static int atmel_sha_cpu_transfer(struct atmel_sha_dev *dd)
Signature · 1 line
2.7.4
author
#9 lore raw

PATCH v3 9/12 crypto: atmel-sha: add support to hmac(shaX)

In reply to Cyrille Pitchen

This patch adds support to the hmac(shaX) algorithms.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha-regs.h |   4 +
 drivers/crypto/atmel-sha.c      | 598 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 601 insertions(+), 1 deletion(-)
drivers/crypto/atmel-sha-regs.h +4 -0
index 8d62d31eda08..1b9f3d33079e 100644
@@ -26,6 +26,7 @@
 #define SHA_MR_ALGO_SHA384		(2 << 8)
 #define SHA_MR_ALGO_SHA512		(3 << 8)
 #define SHA_MR_ALGO_SHA224		(4 << 8)
+#define SHA_MR_HMAC			(1 << 11)
 #define	SHA_MR_DUALBUFF			(1 << 16)
 
 #define SHA_IER				0x10

@@ -42,6 +43,9 @@
 #define SHA_ISR_URAT_MR			(0x2 << 12)
 #define SHA_ISR_URAT_WO			(0x5 << 12)
 
+#define SHA_MSR				0x20
+#define SHA_BCR				0x30
+
 #define	SHA_HW_VERSION		0xFC
 
 #define SHA_TPR				0x108
drivers/crypto/atmel-sha.c +597 -1

667 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#10 lore raw

PATCH v3 10/12 crypto: atmel-aes: fix atmel_aes_handle_queue()

In reply to Cyrille Pitchen

This patch fixes the value returned by atmel_aes_handle_queue(), which could have been wrong previously when the crypto request was started synchronously but became asynchronous during the ctx->start() call.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-aes.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/atmel-aes.c +5 -2
index 0e3d0d655b96..9fd2f63b8bc0 100644
@@ -879,6 +879,7 @@ static int atmel_aes_handle_queue(struct atmel_aes_dev *dd,
 	struct crypto_async_request *areq, *backlog;
 	struct atmel_aes_base_ctx *ctx;
 	unsigned long flags;
+	bool start_async;
 	int err, ret = 0;
 
 	spin_lock_irqsave(&dd->lock, flags);

@@ -904,10 +905,12 @@ static int atmel_aes_handle_queue(struct atmel_aes_dev *dd,
 
 	dd->areq = areq;
 	dd->ctx = ctx;
-	dd->is_async = (areq != new_areq);
+	start_async = (areq != new_areq);
+	dd->is_async = start_async;
 
+	/* WARNING: ctx->start() MAY change dd->is_async. */
 	err = ctx->start(dd);
-	return (dd->is_async) ? ret : err;
+	return (start_async) ? ret : err;
 }
Signature · 1 line
2.7.4
author
#11 lore raw

PATCH v3 11/12 crypto: atmel-authenc: add support to authenc(hmac(shaX),Y(aes)) modes

In reply to Cyrille Pitchen

This patchs allows to combine the AES and SHA hardware accelerators on some Atmel SoCs. Doing so, AES blocks are only written to/read from the AES hardware. Those blocks are also transferred from the AES to the SHA accelerator internally, without additionnal accesses to the system busses.

Hence, the AES and SHA accelerators work in parallel to process all the data blocks, instead of serializing the process by (de)crypting those blocks first then authenticating them after like the generic crypto/authenc.c driver does.

Of course, both the AES and SHA hardware accelerators need to be available before we can start to process the data blocks. Hence we use their crypto request queue to synchronize both drivers.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/Kconfig          |  12 ++
 drivers/crypto/atmel-aes-regs.h |  16 ++
 drivers/crypto/atmel-aes.c      | 448 +++++++++++++++++++++++++++++++++++++++-
 drivers/crypto/atmel-authenc.h  |  64 ++++++
 drivers/crypto/atmel-sha-regs.h |  14 ++
 drivers/crypto/atmel-sha.c      | 344 ++++++++++++++++++++++++++++--
 6 files changed, 883 insertions(+), 15 deletions(-)
 create mode 100644 drivers/crypto/atmel-authenc.h
drivers/crypto/Kconfig +12 -0
index bf7da55cffe6..74824612d3e9 100644
@@ -415,6 +415,18 @@ config CRYPTO_DEV_BFIN_CRC
 	  Newer Blackfin processors have CRC hardware. Select this if you
 	  want to use the Blackfin CRC module.
 
+config CRYPTO_DEV_ATMEL_AUTHENC
+	tristate "Support for Atmel IPSEC/SSL hw accelerator"
+	depends on (ARCH_AT91 && HAS_DMA) || COMPILE_TEST
+	select CRYPTO_AUTHENC
+	select CRYPTO_DEV_ATMEL_AES
+	select CRYPTO_DEV_ATMEL_SHA
+	help
+	  Some Atmel processors can combine the AES and SHA hw accelerators
+	  to enhance support of IPSEC/SSL.
+	  Select this if you want to use the Atmel modules for
+	  authenc(hmac(shaX),Y(cbc)) algorithms.
+
 config CRYPTO_DEV_ATMEL_AES
 	tristate "Support for Atmel AES hw accelerator"
 	depends on HAS_DMA
drivers/crypto/atmel-aes-regs.h +16 -0
index 0ec04407b533..7694679802b3 100644
@@ -68,6 +68,22 @@
 #define AES_CTRR	0x98
 #define AES_GCMHR(x)	(0x9c + ((x) * 0x04))
 
+#define AES_EMR		0xb0
+#define AES_EMR_APEN		BIT(0)	/* Auto Padding Enable */
+#define AES_EMR_APM		BIT(1)	/* Auto Padding Mode */
+#define AES_EMR_APM_IPSEC	0x0
+#define AES_EMR_APM_SSL		BIT(1)
+#define AES_EMR_PLIPEN		BIT(4)	/* PLIP Enable */
+#define AES_EMR_PLIPD		BIT(5)	/* PLIP Decipher */
+#define AES_EMR_PADLEN_MASK	(0xFu << 8)
+#define AES_EMR_PADLEN_OFFSET	8
+#define AES_EMR_PADLEN(padlen)	(((padlen) << AES_EMR_PADLEN_OFFSET) &\
+				 AES_EMR_PADLEN_MASK)
+#define AES_EMR_NHEAD_MASK	(0xFu << 16)
+#define AES_EMR_NHEAD_OFFSET	16
+#define AES_EMR_NHEAD(nhead)	(((nhead) << AES_EMR_NHEAD_OFFSET) &\
+				 AES_EMR_NHEAD_MASK)
+
 #define AES_TWR(x)	(0xc0 + ((x) * 0x04))
 #define AES_ALPHAR(x)	(0xd0 + ((x) * 0x04))
drivers/crypto/atmel-aes.c +447 -1

534 lines not on this page · raw message

drivers/crypto/atmel-authenc.h new file +64 -0

64 lines not on this page · raw message

drivers/crypto/atmel-sha-regs.h +14 -0
index 1b9f3d33079e..1b0eba4a2706 100644
@@ -29,6 +29,20 @@
 #define SHA_MR_HMAC			(1 << 11)
 #define	SHA_MR_DUALBUFF			(1 << 16)
 
+#define SHA_FLAGS_ALGO_MASK	SHA_MR_ALGO_MASK
+#define SHA_FLAGS_SHA1		SHA_MR_ALGO_SHA1
+#define SHA_FLAGS_SHA256	SHA_MR_ALGO_SHA256
+#define SHA_FLAGS_SHA384	SHA_MR_ALGO_SHA384
+#define SHA_FLAGS_SHA512	SHA_MR_ALGO_SHA512
+#define SHA_FLAGS_SHA224	SHA_MR_ALGO_SHA224
+#define SHA_FLAGS_HMAC		SHA_MR_HMAC
+#define SHA_FLAGS_HMAC_SHA1	(SHA_FLAGS_HMAC | SHA_FLAGS_SHA1)
+#define SHA_FLAGS_HMAC_SHA256	(SHA_FLAGS_HMAC | SHA_FLAGS_SHA256)
+#define SHA_FLAGS_HMAC_SHA384	(SHA_FLAGS_HMAC | SHA_FLAGS_SHA384)
+#define SHA_FLAGS_HMAC_SHA512	(SHA_FLAGS_HMAC | SHA_FLAGS_SHA512)
+#define SHA_FLAGS_HMAC_SHA224	(SHA_FLAGS_HMAC | SHA_FLAGS_SHA224)
+#define SHA_FLAGS_MODE_MASK	(SHA_FLAGS_HMAC | SHA_FLAGS_ALGO_MASK)
+
 #define SHA_IER				0x10
 #define SHA_IDR				0x14
 #define SHA_IMR				0x18
drivers/crypto/atmel-sha.c +330 -14

380 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#12 lore raw

PATCH v3 12/12 crypto: atmel-sha: add verbose debug facilities to print hw register names

In reply to Cyrille Pitchen

When VERBOSE_DEBUG is defined and SHA_FLAGS_DUMP_REG flag is set in dd->flags, this patch prints the register names and values when performing IO accesses.

Signed-off-by Cyrille Pitchen <[email protected]>
---
 drivers/crypto/atmel-sha.c | 110 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 108 insertions(+), 2 deletions(-)
drivers/crypto/atmel-sha.c +108 -2
index cc5294dbead4..22d0c0c118da 100644
@@ -51,6 +51,7 @@
 #define SHA_FLAGS_INIT			BIT(4)
 #define SHA_FLAGS_CPU			BIT(5)
 #define SHA_FLAGS_DMA_READY		BIT(6)
+#define SHA_FLAGS_DUMP_REG	BIT(7)
 
 /* bits[11:8] are reserved. */
 

@@ -167,14 +168,118 @@ static struct atmel_sha_drv atmel_sha = {
 	.lock = __SPIN_LOCK_UNLOCKED(atmel_sha.lock),
 };
 
+#ifdef VERBOSE_DEBUG
+static const char *atmel_sha_reg_name(u32 offset, char *tmp, size_t sz, bool wr)
+{
+	switch (offset) {
+	case SHA_CR:
+		return "CR";
+
+	case SHA_MR:
+		return "MR";
+
+	case SHA_IER:
+		return "IER";
+
+	case SHA_IDR:
+		return "IDR";
+
+	case SHA_IMR:
+		return "IMR";
+
+	case SHA_ISR:
+		return "ISR";
+
+	case SHA_MSR:
+		return "MSR";
+
+	case SHA_BCR:
+		return "BCR";
+
+	case SHA_REG_DIN(0):
+	case SHA_REG_DIN(1):
+	case SHA_REG_DIN(2):
+	case SHA_REG_DIN(3):
+	case SHA_REG_DIN(4):
+	case SHA_REG_DIN(5):
+	case SHA_REG_DIN(6):
+	case SHA_REG_DIN(7):
+	case SHA_REG_DIN(8):
+	case SHA_REG_DIN(9):
+	case SHA_REG_DIN(10):
+	case SHA_REG_DIN(11):
+	case SHA_REG_DIN(12):
+	case SHA_REG_DIN(13):
+	case SHA_REG_DIN(14):
+	case SHA_REG_DIN(15):
+		snprintf(tmp, sz, "IDATAR[%u]", (offset - SHA_REG_DIN(0)) >> 2);
+		break;
+
+	case SHA_REG_DIGEST(0):
+	case SHA_REG_DIGEST(1):
+	case SHA_REG_DIGEST(2):
+	case SHA_REG_DIGEST(3):
+	case SHA_REG_DIGEST(4):
+	case SHA_REG_DIGEST(5):
+	case SHA_REG_DIGEST(6):
+	case SHA_REG_DIGEST(7):
+	case SHA_REG_DIGEST(8):
+	case SHA_REG_DIGEST(9):
+	case SHA_REG_DIGEST(10):
+	case SHA_REG_DIGEST(11):
+	case SHA_REG_DIGEST(12):
+	case SHA_REG_DIGEST(13):
+	case SHA_REG_DIGEST(14):
+	case SHA_REG_DIGEST(15):
+		if (wr)
+			snprintf(tmp, sz, "IDATAR[%u]",
+				 16u + ((offset - SHA_REG_DIGEST(0)) >> 2));
+		else
+			snprintf(tmp, sz, "ODATAR[%u]",
+				 (offset - SHA_REG_DIGEST(0)) >> 2);
+		break;
+
+	case SHA_HW_VERSION:
+		return "HWVER";
+
+	default:
+		snprintf(tmp, sz, "0x%02x", offset);
+		break;
+	}
+
+	return tmp;
+}
+
+#endif /* VERBOSE_DEBUG */
+
 static inline u32 atmel_sha_read(struct atmel_sha_dev *dd, u32 offset)
 {
-	return readl_relaxed(dd->io_base + offset);
+	u32 value = readl_relaxed(dd->io_base + offset);
+
+#ifdef VERBOSE_DEBUG
+	if (dd->flags & SHA_FLAGS_DUMP_REG) {
+		char tmp[16];
+
+		dev_vdbg(dd->dev, "read 0x%08x from %s\n", value,
+			 atmel_sha_reg_name(offset, tmp, sizeof(tmp), false));
+	}
+#endif /* VERBOSE_DEBUG */
+
+	return value;
 }
 
 static inline void atmel_sha_write(struct atmel_sha_dev *dd,
 					u32 offset, u32 value)
 {
+#ifdef VERBOSE_DEBUG
+	if (dd->flags & SHA_FLAGS_DUMP_REG) {
+		char tmp[16];
+
+		dev_vdbg(dd->dev, "write 0x%08x into %s\n", value,
+			 atmel_sha_reg_name(offset, tmp, sizeof(tmp), true));
+	}
+#endif /* VERBOSE_DEBUG */
+
 	writel_relaxed(value, dd->io_base + offset);
 }
 

@@ -183,7 +288,8 @@ static inline int atmel_sha_complete(struct atmel_sha_dev *dd, int err)
 	struct ahash_request *req = dd->req;
 
 	dd->flags &= ~(SHA_FLAGS_BUSY | SHA_FLAGS_FINAL | SHA_FLAGS_CPU |
-		       SHA_FLAGS_DMA_READY | SHA_FLAGS_OUTPUT_READY);
+		       SHA_FLAGS_DMA_READY | SHA_FLAGS_OUTPUT_READY |
+		       SHA_FLAGS_DUMP_REG);
 
 	clk_disable(dd->iclk);
Signature · 1 line
2.7.4
maintainer
#13 lore raw
In reply to Cyrille Pitchen
Quoted 21 lines · Cyrille Pitchen

Hi all,

this series of patches has been based and tested on next-20170125 with CRYPTO_MANAGER_DISABLED_TESTS not set.

The series adds support to the hmac(shaX) algorithms first, then combines both the Atmel SHA and AES hardware accelerators to implement authenc(hmac(shaX),Y(aes)) algorithms as used by IPSEC/SSL connections.

It has also been tested with strongswan + xl2tpd to create an IPSEC+L2TP (transport mode) VPN and strongswan only (tunnel mode) for an IPSEC VPN.

Then iperf was used to measure the bandwidth improvement in tunnel mode:

drivers AES SHA SPLIP iperf half-duplex
Mbit/s

authenc(hmac(sha1-generic),cbc(aes))        SW   SW  N/A 27.7
authenc(hmac(sha1-generic),atmel-cbc-aes)   HW   SW  N/A 30.2 (mainline)
authenc(atmel-hmac-sha1,atmel-cbc-aes)      HW   HW   no 29.1
atmel-authenc-hmac-sha1-cbc-aes             HW   HW  yes 38.8
Quoted 4 lines

SPLIP: Secure Protocol Layers Improved Performances (AES+SHA combined).

Some patches of this series are purely transitional: I've split the modifications into many patches to ease the review.

All applied. Thanks.

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt