Optimize CRC32C calculation using PCLMULQDQ in crc32c-intel module

This patch series optimized CRC32C calculations with PCLMULQDQ instruction for crc32c-intel module. It speeds up the original implementation by 1.6x for 1K buffer and by 3x for buffer 4k or more. The tcrypt module was enhanced for doing speed test on crc32c calculations.

Tim

---
Tim Chen (3):
Rename crc32c-intel.c to crc32c-intel_glue.c Optimize CRC32C calculation with PCLMULQDQ instruction Added speed test in tcrypt for crc32c

 arch/x86/crypto/Makefile                           |    1 +
 .../crypto/{crc32c-intel.c => crc32c-intel_glue.c} |   75 ++++
 arch/x86/crypto/crc32c-pcl-intel-asm.S             |  460 ++++++++++++++++++++
 crypto/tcrypt.c                                    |    4 +
 4 files changed, 540 insertions(+), 0 deletions(-)
 rename arch/x86/crypto/{crc32c-intel.c => crc32c-intel_glue.c} (70%)
 create mode 100644 arch/x86/crypto/crc32c-pcl-intel-asm.S

1.7.7.6

From: Tim Chen <[email protected]>
Date: Tue, 25 Sep 2012 14:50:08 -0700

Great work.

I intend to do something nearly identical on sparc64 since we have similar instructions in the form of "xmulx" and "xmulxhi" which return the lower and upper 64-bits (respectively) of a XOR multiply.

All applied. Thanks Tim!

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

This patch rename the crc32c-intel.c file to crc32c-intel_glue.c file in preparation for linking with the new crc32c-pcl-intel-asm.S file, which contains optimized crc32c calculation based on PCLMULQDQ instruction.

Tim

---
 arch/x86/crypto/Makefile                           |    1 +
 .../crypto/{crc32c-intel.c => crc32c-intel_glue.c} |    0
 2 files changed, 1 insertions(+), 0 deletions(-)
 rename arch/x86/crypto/{crc32c-intel.c => crc32c-intel_glue.c} (100%)

index e908e5d..edd2268 100644
@@ -43,3 +43,4 @@ serpent-avx-x86_64-y := serpent-avx-x86_64-asm_64.o serpent_avx_glue.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
+crc32c-intel-y := crc32c-intel_glue.o

similarity index 100%
rename from arch/x86/crypto/crc32c-intel.c
rename to arch/x86/crypto/crc32c-intel_glue.c

1.7.7.6

This patch adds the crc_pcl function that calculates CRC32C checksum using the PCLMULQDQ instruction on processors that support this feature. This will provide speedup over using CRC32 instruction only.
The usage of PCLMULQDQ necessitate the invocation of kernel_fpu_begin and kernel_fpu_end and incur some overhead. So the new crc_pcl function is only invoked for buffer size of 512 bytes or more. Larger sized
buffers will expect to see greater speedup. This feature is best used coupled with eager_fpu which reduces the kernel_fpu_begin/end overhead. For buffer size of 1K the speedup is around 1.6x and for buffer size greater than 4K, the speedup is around 3x compared to original implementation in crc32c-intel module. Test was performed on Sandy Bridge based platform with constant frequency set for cpu.

A white paper detailing the algorithm can be found here:
http://download.intel.com/design/intarch/papers/323405.pdf

Tim

---
 arch/x86/crypto/Makefile               |    2 +-
 arch/x86/crypto/crc32c-intel_glue.c    |   75 +++++
 arch/x86/crypto/crc32c-pcl-intel-asm.S |  460 ++++++++++++++++++++++++++++++++
 3 files changed, 536 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/crypto/crc32c-pcl-intel-asm.S

index edd2268..0babdcb 100644
@@ -43,4 +43,4 @@ serpent-avx-x86_64-y := serpent-avx-x86_64-asm_64.o serpent_avx_glue.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
-crc32c-intel-y := crc32c-intel_glue.o
+crc32c-intel-y := crc32c-pcl-intel-asm.o crc32c-intel_glue.o

1.7.7.6

This patch adds a test case in tcrypt to perform speed test for crc32c checksum calculation.

Tim

---
 crypto/tcrypt.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

index 581081d..6deb77f 100644
@@ -1437,6 +1437,10 @@ static int do_test(int m)
 		test_hash_speed("ghash-generic", sec, hash_speed_template_16);
 		if (mode > 300 && mode < 400) break;
 
+	case 319:
+		test_hash_speed("crc32c", sec, generic_hash_speed_template);
+		if (mode > 300 && mode < 400) break;
+
 	case 399:
 		break;
 

1.7.7.6