Skip to contentSkip to conversation

crypto: ARM/arm64 CRC-T10DIF/CRC32/CRC32C roundup

PATCH New Ard Biesheuvel 3 participants 11 messages Dec 5, 2016 – Dec 7, 2016

Subsystems (4)
Arm (32-bit) Crypto API arm64 Arm STM32
lore
Share & tools
lore

Ard Biesheuvel

author
#0 lore raw

v3 cover crypto: ARM/arm64 CRC-T10DIF/CRC32/CRC32C roundup

This v3 combines the CRC-T10DIF and CRC32 implementations for both ARM and arm64 that I sent out a couple of weeks ago, and adds support to the latter for CRC32C.

Changes since v2:
- fix a couple of big-endian bugs in CRC32/CRC32C
- add back handling to the CRC-T10DIF routines of buffers that are not a multiple of 16 bytes (but they still must be 16 byte aligned)

Ard Biesheuvel (6):
crypto: testmgr - avoid overlap in chunked tests
crypto: testmgr - add/enhance test cases for CRC-T10DIF
crypto: arm64/crct10dif - port x86 SSE implementation to arm64
crypto: arm/crct10dif - port x86 SSE implementation to ARM
crypto: arm64/crc32 - accelerated support based on x86 SSE
implementation
crypto: arm/crc32 - accelerated support based on x86 SSE
implementation

 arch/arm/crypto/Kconfig               |  10 +
 arch/arm/crypto/Makefile              |   4 +
 arch/arm/crypto/crc32-ce-core.S       | 306 ++++++++++++++
 arch/arm/crypto/crc32-ce-glue.c       | 242 +++++++++++
 arch/arm/crypto/crct10dif-ce-core.S   | 427 ++++++++++++++++++++
 arch/arm/crypto/crct10dif-ce-glue.c   | 101 +++++
 arch/arm64/crypto/Kconfig             |  11 +
 arch/arm64/crypto/Makefile            |   6 +
 arch/arm64/crypto/crc32-ce-core.S     | 266 ++++++++++++
 arch/arm64/crypto/crc32-ce-glue.c     | 212 ++++++++++
 arch/arm64/crypto/crct10dif-ce-core.S | 392 ++++++++++++++++++
 arch/arm64/crypto/crct10dif-ce-glue.c |  95 +++++
 crypto/testmgr.c                      |   2 +-
 crypto/testmgr.h                      |  70 ++--
 14 files changed, 2115 insertions(+), 29 deletions(-)
 create mode 100644 arch/arm/crypto/crc32-ce-core.S
 create mode 100644 arch/arm/crypto/crc32-ce-glue.c
 create mode 100644 arch/arm/crypto/crct10dif-ce-core.S
 create mode 100644 arch/arm/crypto/crct10dif-ce-glue.c
 create mode 100644 arch/arm64/crypto/crc32-ce-core.S
 create mode 100644 arch/arm64/crypto/crc32-ce-glue.c
 create mode 100644 arch/arm64/crypto/crct10dif-ce-core.S
 create mode 100644 arch/arm64/crypto/crct10dif-ce-glue.c
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v3 1/6 crypto: testmgr - avoid overlap in chunked tests

The IDXn offsets are chosen such that tap values (which may go up to
255) end up overlapping in the xbuf allocation. In particular, IDX1 and IDX3 are too close together, so update IDX3 to avoid this issue.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 crypto/testmgr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/testmgr.c +1 -1
index ded50b67c757..670893bcf361 100644
@@ -63,7 +63,7 @@ int alg_test(const char *driver, const char *alg, u32 type, u32 mask)
  */
 #define IDX1		32
 #define IDX2		32400
-#define IDX3		1
+#define IDX3		511
 #define IDX4		8193
 #define IDX5		22222
 #define IDX6		17101
Signature · 1 line
2.7.4
#2 lore raw

PATCH v3 1/6 crypto: testmgr - avoid overlap in chunked tests

Quoted 4 lines · Ard Biesheuvel

The IDXn offsets are chosen such that tap values (which may go up to
255) end up overlapping in the xbuf allocation. In particular, IDX1 and IDX3 are too close together, so update IDX3 to avoid this issue.

Hi Ard,

This patch is causing the self-tests for "xts(ecb(aes-asm))" to fail. This is on x86. Any idea why? Here's what I see in the log:

	alg: skcipher: Chunk test 1 failed on encryption at page 0 for xts(ecb(aes-asm))
	00000000: 1c 3b 3a 10 2f 77 03 86 e4 83 6c 99 e3 70 cf 9b
	00000010: ea 00 80 3f 5e 48 23 57 a4 ae 12 d4 14 a3 e6 3b
	00000020: 5d 31 e2 76 f8 fe 4a 8d 66 b3 17 f9 ac 68 3f 44
	00000030: 68 0a 86 ac 35 ad fc 33 45 be fe cb 4b b1 88 fd
	00000040: 57 76 92 6c 49 a3 09 5e b1 08 fd 10 98 ba ec 70
	00000050: aa a6 69 99 a7 2a 82 f2 7d 84 8b 21 d4 a7 41 b0
	00000060: c5 cd 4d 5f ff 9d ac 89 ae ba 12 29 61 d0 3a 75
	00000070: 71 23 e9 87 0f 8a cf 10 00 02 08 87 89 14 29 ca
	00000080: 2a 3e 7a 7d 7d f7 b1 03 55 16 5c 8b 9a 6d 0a 7d
	00000090: e8 b0 62 c4 50 0d c4 cd 12 0c 0f 74 18 da e3 d0
	000000a0: b5 78 1c 34 80 3f a7 54 21 c7 90 df e1 de 18 34
	000000b0: f2 80 d7 66 7b 32 7f 6c 8c d7 55 7e 12 ac 3a 0f
	000000c0: 93 ec 05 c5 2e 04 93 ef 31 a1 2d 3d 92 60 f7 9a
	000000d0: 28 9d 6a 37 9b c7 0c 50 84 14 73 d1 a8 cc 81 ec
	000000e0: 58 3e 96 45 e0 7b 8d 96 70 65 5b a5 bb cf ec c6
	000000f0: dc 39 66 38 0a d8 fe cb 17 b6 ba 02 46 9a 02 0a
	00000100: 84 e1 8e 8f 84 25 20 70 c1 3e 9f 1f 28 9b e5 4f
	00000110: bc 48 14 57 77 8f 61 60 15 e1 32 7a 02 b1 40 f1
	00000120: 50 5e b3 09 32 6d 68 37 8f 83 74 59 5c 84 9d 84
	00000130: f4 c3 33 ec 44 23 88 51 43 cb 47 bd 71 c5 ed ae
	00000140: 9b e6 9a 2f fe ce b1 be c9 de 24 4f be 15 99 2b
	00000150: 11 b7 7c 04 0f 12 bd 8f 6a 97 5a 44 a0 f9 0c 29
	00000160: a9 ab c3 d4 d8 93 92 72 84 c5 87 54 cc e2 94 52
	00000170: 9f 86 14 dc d2 ab a9 91 92 5f ed c4 ae 74 ff ac
	00000180: 6e 33 3b 93 eb 4a ff 04 79 da 9a 41 0e 44 50 e0
	00000190: dd 7a e4 c6 e2 91 09 00 57 5d a4 01 fc 07 05 9f
	000001a0: 64 5e 8b 7e 9b fd ef 33 94 30 54 ff 84 01 14 93
	000001b0: c2 7b 34 29 ea ed b4 ed 53 76 44 1a 77 ed 43 85
	000001c0: 1a d7 7f 16 f5 41 df d2 69 d5 0d 6a 5f 14 fb 0a
	000001d0: 1e 2a 8f 42 61 9e 5e c2 59 bd 96 d0 e5 cc 23 1f
	000001e0: fb 84 ed 15 a8 eb 66 07 31 6b f6 ef

Eric

author
#3 lore raw

PATCH v3 1/6 crypto: testmgr - avoid overlap in chunked tests

Quoted 10 lines · Eric Biggers
Quoted 4 lines · Ard Biesheuvel

The IDXn offsets are chosen such that tap values (which may go up to
255) end up overlapping in the xbuf allocation. In particular, IDX1 and IDX3 are too close together, so update IDX3 to avoid this issue.

Hi Ard,

This patch is causing the self-tests for "xts(ecb(aes-asm))" to fail. This is on x86. Any idea why?

Does this help at all?

crypto/testmgr.c +0 -0
index 670893bcf361..59e67f5b544b 100644
@@ -63,7 +63,7 @@ int alg_test(const char *driver, const char *alg,

u32 type, u32 mask)
*/

 #define IDX1           32
 #define IDX2           32400
-#define IDX3           511
+#define IDX3           1511
 #define IDX4           8193
 #define IDX5           22222
 #define IDX6           17101
Quoted 36 lines

Here's what I see in the log:

        alg: skcipher: Chunk test 1 failed on encryption at page 0 for xts(ecb(aes-asm))
        00000000: 1c 3b 3a 10 2f 77 03 86 e4 83 6c 99 e3 70 cf 9b
        00000010: ea 00 80 3f 5e 48 23 57 a4 ae 12 d4 14 a3 e6 3b
        00000020: 5d 31 e2 76 f8 fe 4a 8d 66 b3 17 f9 ac 68 3f 44
        00000030: 68 0a 86 ac 35 ad fc 33 45 be fe cb 4b b1 88 fd
        00000040: 57 76 92 6c 49 a3 09 5e b1 08 fd 10 98 ba ec 70
        00000050: aa a6 69 99 a7 2a 82 f2 7d 84 8b 21 d4 a7 41 b0
        00000060: c5 cd 4d 5f ff 9d ac 89 ae ba 12 29 61 d0 3a 75
        00000070: 71 23 e9 87 0f 8a cf 10 00 02 08 87 89 14 29 ca
        00000080: 2a 3e 7a 7d 7d f7 b1 03 55 16 5c 8b 9a 6d 0a 7d
        00000090: e8 b0 62 c4 50 0d c4 cd 12 0c 0f 74 18 da e3 d0
        000000a0: b5 78 1c 34 80 3f a7 54 21 c7 90 df e1 de 18 34
        000000b0: f2 80 d7 66 7b 32 7f 6c 8c d7 55 7e 12 ac 3a 0f
        000000c0: 93 ec 05 c5 2e 04 93 ef 31 a1 2d 3d 92 60 f7 9a
        000000d0: 28 9d 6a 37 9b c7 0c 50 84 14 73 d1 a8 cc 81 ec
        000000e0: 58 3e 96 45 e0 7b 8d 96 70 65 5b a5 bb cf ec c6
        000000f0: dc 39 66 38 0a d8 fe cb 17 b6 ba 02 46 9a 02 0a
        00000100: 84 e1 8e 8f 84 25 20 70 c1 3e 9f 1f 28 9b e5 4f
        00000110: bc 48 14 57 77 8f 61 60 15 e1 32 7a 02 b1 40 f1
        00000120: 50 5e b3 09 32 6d 68 37 8f 83 74 59 5c 84 9d 84
        00000130: f4 c3 33 ec 44 23 88 51 43 cb 47 bd 71 c5 ed ae
        00000140: 9b e6 9a 2f fe ce b1 be c9 de 24 4f be 15 99 2b
        00000150: 11 b7 7c 04 0f 12 bd 8f 6a 97 5a 44 a0 f9 0c 29
        00000160: a9 ab c3 d4 d8 93 92 72 84 c5 87 54 cc e2 94 52
        00000170: 9f 86 14 dc d2 ab a9 91 92 5f ed c4 ae 74 ff ac
        00000180: 6e 33 3b 93 eb 4a ff 04 79 da 9a 41 0e 44 50 e0
        00000190: dd 7a e4 c6 e2 91 09 00 57 5d a4 01 fc 07 05 9f
        000001a0: 64 5e 8b 7e 9b fd ef 33 94 30 54 ff 84 01 14 93
        000001b0: c2 7b 34 29 ea ed b4 ed 53 76 44 1a 77 ed 43 85
        000001c0: 1a d7 7f 16 f5 41 df d2 69 d5 0d 6a 5f 14 fb 0a
        000001d0: 1e 2a 8f 42 61 9e 5e c2 59 bd 96 d0 e5 cc 23 1f
        000001e0: fb 84 ed 15 a8 eb 66 07 31 6b f6 ef

Eric

#4 lore raw

PATCH v3 1/6 crypto: testmgr - avoid overlap in chunked tests

Quoted 15 lines · Ard Biesheuvel

Does this help at all?

crypto/testmgr.c +1 -1
index 670893bcf361..59e67f5b544b 100644
@@ -63,7 +63,7 @@ int alg_test(const char *driver, const char *alg,
 u32 type, u32 mask)
  */
 #define IDX1           32
 #define IDX2           32400
-#define IDX3           511
+#define IDX3           1511
 #define IDX4           8193
 #define IDX5           22222
Quoted 2 lines

#define IDX6 17101

Yes, with that change made the self-tests pass.

Eric

author
#5 lore raw

PATCH v3 2/6 crypto: testmgr - add/enhance test cases for CRC-T10DIF

In reply to Ard Biesheuvel

The existing test cases only exercise a small slice of the various possible code paths through the x86 SSE/PCLMULQDQ implementation, and the upcoming ports of it for arm64. So add one that exceeds 256 bytes in size, and convert another to a chunked test.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 crypto/testmgr.h | 70 ++++++++++++--------
 1 file changed, 42 insertions(+), 28 deletions(-)
crypto/testmgr.h +42 -28
index e64a4ef9d8ca..9b656be7f52f 100644
@@ -1334,36 +1334,50 @@ static struct hash_testvec rmd320_tv_template[] = {
 	}
 };
 
-#define CRCT10DIF_TEST_VECTORS	3
+#define CRCT10DIF_TEST_VECTORS	ARRAY_SIZE(crct10dif_tv_template)
 static struct hash_testvec crct10dif_tv_template[] = {
 	{
-		.plaintext = "abc",
-		.psize  = 3,
-#ifdef __LITTLE_ENDIAN
-		.digest = "\x3b\x44",
-#else
-		.digest = "\x44\x3b",
-#endif
-	}, {
-		.plaintext = "1234567890123456789012345678901234567890"
-			     "123456789012345678901234567890123456789",
-		.psize	= 79,
-#ifdef __LITTLE_ENDIAN
-		.digest	= "\x70\x4b",
-#else
-		.digest	= "\x4b\x70",
-#endif
-	}, {
-		.plaintext =
-		"abcddddddddddddddddddddddddddddddddddddddddddddddddddddd",
-		.psize  = 56,
-#ifdef __LITTLE_ENDIAN
-		.digest = "\xe3\x9c",
-#else
-		.digest = "\x9c\xe3",
-#endif
-		.np     = 2,
-		.tap    = { 28, 28 }
+		.plaintext	= "abc",
+		.psize		= 3,
+		.digest		= (u8 *)(u16 []){ 0x443b },
+	}, {
+		.plaintext 	= "1234567890123456789012345678901234567890"
+				  "123456789012345678901234567890123456789",
+		.psize		= 79,
+		.digest 	= (u8 *)(u16 []){ 0x4b70 },
+		.np		= 2,
+		.tap		= { 63, 16 },
+	}, {
+		.plaintext	= "abcdddddddddddddddddddddddddddddddddddddddd"
+				  "ddddddddddddd",
+		.psize		= 56,
+		.digest		= (u8 *)(u16 []){ 0x9ce3 },
+		.np		= 8,
+		.tap		= { 1, 2, 28, 7, 6, 5, 4, 3 },
+	}, {
+		.plaintext 	= "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "123456789012345678901234567890123456789",
+		.psize		= 319,
+		.digest		= (u8 *)(u16 []){ 0x44c6 },
+	}, {
+		.plaintext 	= "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "1234567890123456789012345678901234567890"
+				  "123456789012345678901234567890123456789",
+		.psize		= 319,
+		.digest		= (u8 *)(u16 []){ 0x44c6 },
+		.np		= 4,
+		.tap		= { 1, 255, 57, 6 },
 	}
 };
Signature · 1 line
2.7.4
author
#6 lore raw

PATCH v3 3/6 crypto: arm64/crct10dif - port x86 SSE implementation to arm64

In reply to Ard Biesheuvel

This is a transliteration of the Intel algorithm implemented using SSE and PCLMULQDQ instructions that resides in the file arch/x86/crypto/crct10dif-pcl-asm_64.S, but simplified to only operate on buffers that are 16 byte aligned (but of any size)

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/crypto/Kconfig             |   5 +
 arch/arm64/crypto/Makefile            |   3 +
 arch/arm64/crypto/crct10dif-ce-core.S | 392 ++++++++++++++++++++
 arch/arm64/crypto/crct10dif-ce-glue.c |  95 +++++
 4 files changed, 495 insertions(+)
arch/arm64/crypto/Kconfig +5 -0
index 2cf32e9887e1..d773c0659202 100644
@@ -23,6 +23,11 @@ config CRYPTO_GHASH_ARM64_CE
 	depends on ARM64 && KERNEL_MODE_NEON
 	select CRYPTO_HASH
 
+config CRYPTO_CRCT10DIF_ARM64_CE
+	tristate "CRCT10DIF digest algorithm using PMULL instructions"
+	depends on KERNEL_MODE_NEON && CRC_T10DIF
+	select CRYPTO_HASH
+
 config CRYPTO_AES_ARM64_CE
 	tristate "AES core cipher using ARMv8 Crypto Extensions"
 	depends on ARM64 && KERNEL_MODE_NEON
arch/arm64/crypto/Makefile +3 -0
index abb79b3cfcfe..36fd3eb4201b 100644
@@ -17,6 +17,9 @@ sha2-ce-y := sha2-ce-glue.o sha2-ce-core.o
 obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o
 ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o
 
+obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM64_CE) += crct10dif-ce.o
+crct10dif-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o
+
 obj-$(CONFIG_CRYPTO_AES_ARM64_CE) += aes-ce-cipher.o
 CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto
arch/arm64/crypto/crct10dif-ce-core.S new file +392 -0

392 lines not on this page · raw message

arch/arm64/crypto/crct10dif-ce-glue.c new file +95 -0

95 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#7 lore raw

PATCH v3 4/6 crypto: arm/crct10dif - port x86 SSE implementation to ARM

In reply to Ard Biesheuvel

This is a transliteration of the Intel algorithm implemented using SSE and PCLMULQDQ instructions that resides in the file arch/x86/crypto/crct10dif-pcl-asm_64.S, but simplified to only operate on buffers that are 16 byte aligned (but of any size)

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm/crypto/Kconfig             |   5 +
 arch/arm/crypto/Makefile            |   2 +
 arch/arm/crypto/crct10dif-ce-core.S | 427 ++++++++++++++++++++
 arch/arm/crypto/crct10dif-ce-glue.c | 101 +++++
 4 files changed, 535 insertions(+)
arch/arm/crypto/Kconfig +5 -0
index 27ed1b1cd1d7..fce801fa52a1 100644
@@ -120,4 +120,9 @@ config CRYPTO_GHASH_ARM_CE
 	  that uses the 64x64 to 128 bit polynomial multiplication (vmull.p64)
 	  that is part of the ARMv8 Crypto Extensions
 
+config CRYPTO_CRCT10DIF_ARM_CE
+	tristate "CRCT10DIF digest algorithm using PMULL instructions"
+	depends on KERNEL_MODE_NEON && CRC_T10DIF
+	select CRYPTO_HASH
+
 endif
arch/arm/crypto/Makefile +2 -0
index fc5150702b64..fc77265014b7 100644
@@ -13,6 +13,7 @@ ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
 ce-obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
 ce-obj-$(CONFIG_CRYPTO_SHA2_ARM_CE) += sha2-arm-ce.o
 ce-obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) += ghash-arm-ce.o
+ce-obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM_CE) += crct10dif-arm-ce.o
 
 ifneq ($(ce-obj-y)$(ce-obj-m),)
 ifeq ($(call as-instr,.fpu crypto-neon-fp-armv8,y,n),y)

@@ -36,6 +37,7 @@ sha1-arm-ce-y	:= sha1-ce-core.o sha1-ce-glue.o
 sha2-arm-ce-y	:= sha2-ce-core.o sha2-ce-glue.o
 aes-arm-ce-y	:= aes-ce-core.o aes-ce-glue.o
 ghash-arm-ce-y	:= ghash-ce-core.o ghash-ce-glue.o
+crct10dif-arm-ce-y	:= crct10dif-ce-core.o crct10dif-ce-glue.o
 
 quiet_cmd_perl = PERL    $@
       cmd_perl = $(PERL) $(<) > $(@)
arch/arm/crypto/crct10dif-ce-core.S new file +427 -0

427 lines not on this page · raw message

arch/arm/crypto/crct10dif-ce-glue.c new file +101 -0

101 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#8 lore raw

PATCH v3 5/6 crypto: arm64/crc32 - accelerated support based on x86 SSE implementation

In reply to Ard Biesheuvel

This is a combination of the the Intel algorithm implemented using SSE and PCLMULQDQ instructions from arch/x86/crypto/crc32-pclmul_asm.S, and the new CRC32 extensions introduced for both 32-bit and 64-bit ARM in version 8 of the architecture. Two versions of the above combo are provided, one for CRC32 and one for CRC32C.

The PMULL/NEON algorithm is faster, but operates on blocks of at least 64 bytes, and on multiples of 16 bytes only. For the remaining input, or for all input on systems that lack the PMULL 64x64->128 instructions, the CRC32 instructions will be used.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm64/crypto/Kconfig         |   6 +
 arch/arm64/crypto/Makefile        |   3 +
 arch/arm64/crypto/crc32-ce-core.S | 266 ++++++++++++++++++++
 arch/arm64/crypto/crc32-ce-glue.c | 212 ++++++++++++++++
 4 files changed, 487 insertions(+)
arch/arm64/crypto/Kconfig +6 -0
index d773c0659202..21835deb1ab9 100644
@@ -28,6 +28,11 @@ config CRYPTO_CRCT10DIF_ARM64_CE
 	depends on KERNEL_MODE_NEON && CRC_T10DIF
 	select CRYPTO_HASH
 
+config CRYPTO_CRC32_ARM64_CE
+	tristate "CRC32 and CRC32C digest algorithms using PMULL instructions"
+	depends on KERNEL_MODE_NEON && CRC32
+	select CRYPTO_HASH
+
 config CRYPTO_AES_ARM64_CE
 	tristate "AES core cipher using ARMv8 Crypto Extensions"
 	depends on ARM64 && KERNEL_MODE_NEON

@@ -58,4 +63,5 @@ config CRYPTO_CRC32_ARM64
 	tristate "CRC32 and CRC32C using optional ARMv8 instructions"
 	depends on ARM64
 	select CRYPTO_HASH
+
 endif
arch/arm64/crypto/Makefile +3 -0
index 36fd3eb4201b..144387805a46 100644
@@ -20,6 +20,9 @@ ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o
 obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM64_CE) += crct10dif-ce.o
 crct10dif-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o
 
+obj-$(CONFIG_CRYPTO_CRC32_ARM64_CE) += crc32-ce.o
+crc32-ce-y:= crc32-ce-core.o crc32-ce-glue.o
+
 obj-$(CONFIG_CRYPTO_AES_ARM64_CE) += aes-ce-cipher.o
 CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto
arch/arm64/crypto/crc32-ce-core.S new file +266 -0

266 lines not on this page · raw message

arch/arm64/crypto/crc32-ce-glue.c new file +212 -0

212 lines not on this page · raw message

Signature · 1 line
2.7.4
author
#9 lore raw

PATCH v3 6/6 crypto: arm/crc32 - accelerated support based on x86 SSE implementation

In reply to Ard Biesheuvel

This is a combination of the the Intel algorithm implemented using SSE and PCLMULQDQ instructions from arch/x86/crypto/crc32-pclmul_asm.S, and the new CRC32 extensions introduced for both 32-bit and 64-bit ARM in version 8 of the architecture. Two versions of the above combo are provided, one for CRC32 and one for CRC32C.

The PMULL/NEON algorithm is faster, but operates on blocks of at least 64 bytes, and on multiples of 16 bytes only. For the remaining input, or for all input on systems that lack the PMULL 64x64->128 instructions, the CRC32 instructions will be used.

Signed-off-by Ard Biesheuvel <[email protected]>
---
 arch/arm/crypto/Kconfig         |   5 +
 arch/arm/crypto/Makefile        |   2 +
 arch/arm/crypto/crc32-ce-core.S | 306 ++++++++++++++++++++
 arch/arm/crypto/crc32-ce-glue.c | 242 ++++++++++++++++
 4 files changed, 555 insertions(+)
arch/arm/crypto/Kconfig +5 -0
index fce801fa52a1..de7bb20815bf 100644
@@ -125,4 +125,9 @@ config CRYPTO_CRCT10DIF_ARM_CE
 	depends on KERNEL_MODE_NEON && CRC_T10DIF
 	select CRYPTO_HASH
 
+config CRYPTO_CRC32_ARM_CE
+	tristate "CRC32(C) digest algorithm using CRC and/or PMULL instructions"
+	depends on KERNEL_MODE_NEON && CRC32
+	select CRYPTO_HASH
+
 endif
arch/arm/crypto/Makefile +2 -0
index fc77265014b7..b578a1820ab1 100644
@@ -14,6 +14,7 @@ ce-obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
 ce-obj-$(CONFIG_CRYPTO_SHA2_ARM_CE) += sha2-arm-ce.o
 ce-obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) += ghash-arm-ce.o
 ce-obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM_CE) += crct10dif-arm-ce.o
+ce-obj-$(CONFIG_CRYPTO_CRC32_ARM_CE) += crc32-arm-ce.o
 
 ifneq ($(ce-obj-y)$(ce-obj-m),)
 ifeq ($(call as-instr,.fpu crypto-neon-fp-armv8,y,n),y)

@@ -38,6 +39,7 @@ sha2-arm-ce-y	:= sha2-ce-core.o sha2-ce-glue.o
 aes-arm-ce-y	:= aes-ce-core.o aes-ce-glue.o
 ghash-arm-ce-y	:= ghash-ce-core.o ghash-ce-glue.o
 crct10dif-arm-ce-y	:= crct10dif-ce-core.o crct10dif-ce-glue.o
+crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o
 
 quiet_cmd_perl = PERL    $@
       cmd_perl = $(PERL) $(<) > $(@)
arch/arm/crypto/crc32-ce-core.S new file +306 -0

306 lines not on this page · raw message

arch/arm/crypto/crc32-ce-glue.c new file +242 -0

242 lines not on this page · raw message

Signature · 1 line
2.7.4
maintainer
#10 lore raw

v3 cover crypto: ARM/arm64 CRC-T10DIF/CRC32/CRC32C roundup

In reply to Ard Biesheuvel
Quoted 3 lines · Ard Biesheuvel

This v3 combines the CRC-T10DIF and CRC32 implementations for both ARM and arm64 that I sent out a couple of weeks ago, and adds support to the latter for CRC32C.

All applied. Thanks.

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt