Optimize CRC32C calculation using PCLMULQDQ in crc32c-intel module

PATCH In review Tim Chen 3 participants 10 messages Sep 27, 2012 – Feb 27, 2013 2 messages not in the archive

Subsystems (2)

x86 Crypto API

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Tim Chen

author

2012-09-27 22:44 UTC #1 lore raw

v2 cover Optimize CRC32C calculation using PCLMULQDQ in crc32c-intel module

Version 2
This version of the patch series fixes compilation errors for 32 bit x86 targets.

Version 1
This patch series optimized CRC32C calculations with PCLMULQDQ instruction for crc32c-intel module. It speeds up the original implementation by 1.6x for 1K buffer and by 3x for buffer 4k or more. The tcrypt module was enhanced for doing speed test on crc32c calculations.

Tim

Signed-off-by Tim Chen <[email protected]>

---

Tim Chen (3):
Rename crc32c-intel.c to crc32c-intel_glue.c Optimize CRC32C calculation with PCLMULQDQ instruction Added speed test in tcrypt for crc32c

 arch/x86/crypto/Makefile                           |    2 +
 .../crypto/{crc32c-intel.c => crc32c-intel_glue.c} |   81 ++++
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S          |  460 ++++++++++++++++++++
 crypto/Kconfig                                     |   10 +
 crypto/tcrypt.c                                    |    4 +
 5 files changed, 557 insertions(+), 0 deletions(-)
 rename arch/x86/crypto/{crc32c-intel.c => crc32c-intel_glue.c} (69%)
 create mode 100644 arch/x86/crypto/crc32c-pcl-intel-asm_64.S

Signature · 1 line

1.7.7.6

H. Peter Anvin

reviewer

2012-09-28 02:54 UTC #2 lore raw

Quoted 11 lines · Tim Chen

Version 2
This version of the patch series fixes compilation errors for 32 bit x86 targets.
Version 1
This patch series optimized CRC32C calculations with PCLMULQDQ instruction for crc32c-intel module. It speeds up the original implementation by 1.6x for 1K buffer and by 3x for buffer 4k or more. The tcrypt module was enhanced for doing speed test on crc32c calculations.

Herbert - you are handling this one, right?

-hpa

Signature · 2 lines

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

A message that is not in the archive (sent off-list or not yet received): CAA-+O6EkeJQtO=GFhDCZxFoX9L42CAJW71oT2miivOKL9MzDyA@mail.gmail.com

Tim Chen

author

2012-10-05 16:47 UTC #4 lore raw

Quoted 16 lines · Herbert Xu

Quoted 15 lines · 在 2012-9-28 上午10:54，"H. Peter Anvin" <[email protected]>写道：
Quoted 12 lines · Tim Chen
Version 2
This version of the patch series fixes compilation errors for 32 bit x86 targets.
Version 1
This patch series optimized CRC32C calculations with PCLMULQDQ instruction for crc32c-intel module. It speeds up the original implementation by 1.6x for 1K buffer and by 3x for buffer 4k or more. The tcrypt module was enhanced for doing speed test on crc32c calculations.

Quoted 5 lines

Quoted 1 line
Herbert - you are handling this one, right?
Yes thanks

Hi Herbert, are there any other changes I should do on my end? Is the patchset ready to be picked up in crypto-dev?

Thanks.

Tim

Herbert Xu

2012-10-15 14:32 UTC #5 lore raw

In reply to Tim Chen

Quoted 11 lines · Tim Chen

Version 2
This version of the patch series fixes compilation errors for 32 bit x86 targets.
Version 1
This patch series optimized CRC32C calculations with PCLMULQDQ instruction for crc32c-intel module. It speeds up the original implementation by 1.6x for 1K buffer and by 3x for buffer 4k or more. The tcrypt module was enhanced for doing speed test on crc32c calculations.

Quoted 3 lines

Tim
Signed-off-by Tim Chen <[email protected]>

All applied. Thanks!

Signature · 3 lines

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Tim Chen

author

2012-09-27 22:44 UTC #6 lore raw

PATCH v2 1/3 Rename crc32c-intel.c to crc32c-intel_glue.c

In reply to a message not in the archive

This patch renames the crc32c-intel.c file to crc32c-intel_glue.c file in preparation for linking with the new crc32c-pcl-intel-asm.S file, which contains optimized crc32c calculation based on PCLMULQDQ instruction.

Signed-off-by Tim Chen <[email protected]>

---
 arch/x86/crypto/Makefile                           |    1 +
 .../crypto/{crc32c-intel.c => crc32c-intel_glue.c} |    0
 2 files changed, 1 insertions(+), 0 deletions(-)
 rename arch/x86/crypto/{crc32c-intel.c => crc32c-intel_glue.c} (100%)

arch/x86/crypto/Makefile +1 -0

index e908e5d..edd2268 100644
@@ -43,3 +43,4 @@ serpent-avx-x86_64-y := serpent-avx-x86_64-asm_64.o serpent_avx_glue.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
+crc32c-intel-y := crc32c-intel_glue.o

arch/x86/crypto/crc32c-intel_glue.c renamed from arch/x86/crypto/crc32c-intel.c +0 -0

similarity index 100%
rename from arch/x86/crypto/crc32c-intel.c
rename to arch/x86/crypto/crc32c-intel_glue.c

Signature · 1 line

1.7.7.6

Tim Chen

author

2012-09-27 22:44 UTC #7 lore raw

PATCH v2 2/3 Optimize CRC32C calculation with PCLMULQDQ instruction

In reply to a message not in the archive

This patch adds the crc_pcl function that calculates CRC32C checksum using the PCLMULQDQ instruction on processors that support this feature. This will provide speedup over using CRC32 instruction only.
The usage of PCLMULQDQ necessitate the invocation of kernel_fpu_begin and kernel_fpu_end and incur some overhead. So the new crc_pcl function is only invoked for buffer size of 512 bytes or more. Larger sized
buffers will expect to see greater speedup. This feature is best used coupled with eager_fpu which reduces the kernel_fpu_begin/end overhead. For buffer size of 1K the speedup is around 1.6x and for buffer size greater than 4K, the speedup is around 3x compared to original implementation in crc32c-intel module. Test was performed on Sandy Bridge based platform with constant frequency set for cpu.

A white paper detailing the algorithm can be found here:
http://download.intel.com/design/intarch/papers/323405.pdf

Signed-off-by Tim Chen <[email protected]>

---
 arch/x86/crypto/Makefile                  |    1 +
 arch/x86/crypto/crc32c-intel_glue.c       |   81 +++++
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S |  460 +++++++++++++++++++++++++++++
 crypto/Kconfig                            |   10 +
 4 files changed, 552 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/crypto/crc32c-pcl-intel-asm_64.S

arch/x86/crypto/Makefile +1 -0

index edd2268..7b6db64 100644
@@ -44,3 +44,4 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
 crc32c-intel-y := crc32c-intel_glue.o
+crc32c-intel-$(CONFIG_CRYPTO_CRC32C_X86_64) += crc32c-pcl-intel-asm_64.o

arch/x86/crypto/crc32c-intel_glue.c +81 -0

105 lines not on this page · raw message

arch/x86/crypto/crc32c-pcl-intel-asm_64.S new file +460 -0

460 lines not on this page · raw message

crypto/Kconfig +10 -0

index a323805..c0aa7ef 100644
@@ -324,9 +324,19 @@ config CRYPTO_CRC32C
 	  by iSCSI for header and data digests and by others.
 	  See Castagnoli93.  Module will be crc32c.
 
+config CRYPTO_CRC32C_X86_64
+	bool
+	depends on X86 && 64BIT
+	select CRYPTO_HASH
+	help
+	  In Intel processor with SSE4.2 supported, the processor will
+	  support CRC32C calculation using hardware accelerated CRC32
+	  instruction optimized with PCLMULQDQ instruction when available.
+
 config CRYPTO_CRC32C_INTEL
 	tristate "CRC32c INTEL hardware acceleration"
 	depends on X86
+	select CRYPTO_CRC32C_X86_64 if 64BIT
 	select CRYPTO_HASH
 	help
 	  In Intel processor with SSE4.2 supported, the processor will

Signature · 1 line

1.7.7.6

Herbert Xu

2013-02-26 09:54 UTC #8 lore raw

Re: [PATCH 2/3 v2] Optimize CRC32C calculation with PCLMULQDQ instruction

Quoted 30 lines · Tim Chen

This patch adds the crc_pcl function that calculates CRC32C checksum using the PCLMULQDQ instruction on processors that support this feature. This will provide speedup over using CRC32 instruction only.
The usage of PCLMULQDQ necessitate the invocation of kernel_fpu_begin and kernel_fpu_end and incur some overhead. So the new crc_pcl function is only invoked for buffer size of 512 bytes or more. Larger sized
buffers will expect to see greater speedup. This feature is best used coupled with eager_fpu which reduces the kernel_fpu_begin/end overhead. For buffer size of 1K the speedup is around 1.6x and for buffer size greater than 4K, the speedup is around 3x compared to original implementation in crc32c-intel module. Test was performed on Sandy Bridge based platform with constant frequency set for cpu.
A white paper detailing the algorithm can be found here:
http://download.intel.com/design/intarch/papers/323405.pdf
Signed-off-by Tim Chen <[email protected]>
---
 arch/x86/crypto/Makefile                  |    1 +
 arch/x86/crypto/crc32c-intel_glue.c       |   81 +++++
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S |  460 +++++++++++++++++++++++++++++
 crypto/Kconfig                            |   10 +
 4 files changed, 552 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/crypto/crc32c-pcl-intel-asm_64.S
arch/x86/crypto/Makefile +0 -0
index edd2268..7b6db64 100644

arch/x86/crypto/Makefile aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o · line 47

Quoted 4 lines

arch/x86/crypto/Makefile +1 -0

@@ -44,3 +44,4 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
 crc32c-intel-y := crc32c-intel_glue.o
+crc32c-intel-$(CONFIG_CRYPTO_CRC32C_X86_64) += crc32c-pcl-intel-asm_64.o

It seems that this option is redundant. So I'd like to kill it with the following patch. If this breaks anything like i386, please let me know.

Thanks!

commit ca81a1a1b8d79dd6706c9463a81e9491e940ca2b

Author: Herbert Xu <[email protected]>
Date:   Tue Feb 26 17:52:15 2013 +0800

crypto: crc32c - Kill pointless CRYPTO_CRC32C_X86_64 option

This bool option can never be set to anything other than y. So let's just kill it.

Signed-off-by Herbert Xu <[email protected]>

arch/x86/crypto/Makefile +1 -1

index 63947a8..ca96761 100644
@@ -52,5 +52,5 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
 crc32c-intel-y := crc32c-intel_glue.o
-crc32c-intel-$(CONFIG_CRYPTO_CRC32C_X86_64) += crc32c-pcl-intel-asm_64.o
+crc32c-intel-$(CONFIG_64BIT) += crc32c-pcl-intel-asm_64.o
 crc32-pclmul-y := crc32-pclmul_asm.o crc32-pclmul_glue.o

crypto/Kconfig +0 ~~-10~~

index 05c0ce5..aed52b2 100644
@@ -322,19 +322,9 @@ config CRYPTO_CRC32C
 	  by iSCSI for header and data digests and by others.
 	  See Castagnoli93.  Module will be crc32c.
 
-config CRYPTO_CRC32C_X86_64
-	bool
-	depends on X86 && 64BIT
-	select CRYPTO_HASH
-	help
-	  In Intel processor with SSE4.2 supported, the processor will
-	  support CRC32C calculation using hardware accelerated CRC32
-	  instruction optimized with PCLMULQDQ instruction when available.
-
 config CRYPTO_CRC32C_INTEL
 	tristate "CRC32c INTEL hardware acceleration"
 	depends on X86
-	select CRYPTO_CRC32C_X86_64 if 64BIT
 	select CRYPTO_HASH
 	help
 	  In Intel processor with SSE4.2 supported, the processor will

Signature · 3 lines

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Tim Chen

author

2013-02-26 16:57 UTC #9 lore raw

Re: [PATCH 2/3 v2] Optimize CRC32C calculation with PCLMULQDQ instruction

Quoted 14 lines · Herbert Xu

Quoted 13 lines · Tim Chen
This patch adds the crc_pcl function that calculates CRC32C checksum using the PCLMULQDQ instruction on processors that support this feature. This will provide speedup over using CRC32 instruction only.
The usage of PCLMULQDQ necessitate the invocation of kernel_fpu_begin and kernel_fpu_end and incur some overhead. So the new crc_pcl function is only invoked for buffer size of 512 bytes or more. Larger sized
buffers will expect to see greater speedup. This feature is best used coupled with eager_fpu which reduces the kernel_fpu_begin/end overhead. For buffer size of 1K the speedup is around 1.6x and for buffer size greater than 4K, the speedup is around 3x compared to original implementation in crc32c-intel module. Test was performed on Sandy Bridge based platform with constant frequency set for cpu.

Quoted 4 lines

Quoted 4 lines
A white paper detailing the algorithm can be found here:
http://download.intel.com/design/intarch/papers/323405.pdf
Signed-off-by Tim Chen <[email protected]>

Acked.

BTW, Herbert, I've also sent you a patch a few days ago to update the link to the whitepaper on the CRC32C algorithm in the code. Wonder if you have received it. Thanks.

Tim

Herbert Xu

2013-02-27 02:02 UTC #10 lore raw

Re: [PATCH 2/3 v2] Optimize CRC32C calculation with PCLMULQDQ instruction

Quoted 4 lines · Tim Chen

BTW, Herbert, I've also sent you a patch a few days ago to update the link to the whitepaper on the CRC32C algorithm in the code. Wonder if you have received it. Thanks.

Tim it's in my queue.

Cheers,

Signature · 3 lines

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Tim Chen

author

2012-09-27 22:44 UTC #11 lore raw

PATCH v2 3/3 Added speed test in tcrypt for crc32c

In reply to a message not in the archive

This patch adds a test case in tcrypt to perform speed test for crc32c checksum calculation.

Signed-off-by Tim Chen <[email protected]>

---
 crypto/tcrypt.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

crypto/tcrypt.c +4 -0

index 581081d..6deb77f 100644
@@ -1437,6 +1437,10 @@ static int do_test(int m)
 		test_hash_speed("ghash-generic", sec, hash_speed_template_16);
 		if (mode > 300 && mode < 400) break;
 
+	case 319:
+		test_hash_speed("crc32c", sec, generic_hash_speed_template);
+		if (mode > 300 && mode < 400) break;
+
 	case 399:
 		break;

Signature · 1 line

1.7.7.6