lib/crypto: x86/sha: Add PHE Extensions support

lib/crypto/Makefile +2 -1

index d2845b214..069069377 100644
@@ -205,7 +205,8 @@ endif
 libsha1-$(CONFIG_SPARC) += sparc/sha1_asm.o
 libsha1-$(CONFIG_X86) += x86/sha1-ssse3-and-avx.o \
 			 x86/sha1-avx2-asm.o \
-			 x86/sha1-ni-asm.o
+			 x86/sha1-ni-asm.o \
+			 x86/sha1-phe-asm.o
 endif # CONFIG_CRYPTO_LIB_SHA1_ARCH
 
 ################################################################################

lib/crypto/x86/sha1-phe-asm.S new file +61 -0

new file mode 100644
index 000000000..eff086104
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * PHE Extensions optimized implementation of a SHA-1 update function
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 	* Redistributions of source code must retain the above copyright
+ * 	  notice, this list of conditions and the following disclaimer.
+ * 	* Redistributions in binary form must reproduce the above copyright
+ * 	  notice, this list of conditions and the following disclaimer in
+ * 	  the documentation and/or other materials provided with the
+ * 	  distribution.
+ * 	* Neither the name of Intel Corporation nor the names of its
+ * 	  contributors may be used to endorse or promote products derived
+ * 	  from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include <linux/linkage.h>
+
+/*
+ * PHE Extensions optimized implementation of a SHA-1 block function
+ *
+ * This function takes a pointer to the current SHA-1 state, a pointer to the
+ * input data, and the number of 64-byte blocks to process.  The number of
+ * blocks to process is assumed to be nonzero.  Once all blocks have been
+ * processed, the state is updated with the new state.  This function only
+ * processes complete blocks.  State initialization, buffering of partial
+ * blocks, and digest finalization are expected to be handled elsewhere.
+ *
+ * void sha1_transform_phe(u8 *state, const u8 *data, size_t nblocks)

[+Cc [email protected]]

Quoted 86 lines · AlanSong-oc

lib/crypto/Makefile +2 -1

index d2845b214..069069377 100644
@@ -205,7 +205,8 @@ endif
 libsha1-$(CONFIG_SPARC) += sparc/sha1_asm.o
 libsha1-$(CONFIG_X86) += x86/sha1-ssse3-and-avx.o \
                       x86/sha1-avx2-asm.o \
-                      x86/sha1-ni-asm.o
+                      x86/sha1-ni-asm.o \
+                      x86/sha1-phe-asm.o
 endif # CONFIG_CRYPTO_LIB_SHA1_ARCH
 
 ################################################################################

lib/crypto/x86/sha1-phe-asm.S new file +66 -0

new file mode 100644
index 000000000..eff086104
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * PHE Extensions optimized implementation of a SHA-1 update function
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *   * Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in
+ *     the documentation and/or other materials provided with the
+ *     distribution.
+ *   * Neither the name of Intel Corporation nor the names of its
+ *     contributors may be used to endorse or promote products derived
+ *     from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include <linux/linkage.h>
+
+/*
+ * PHE Extensions optimized implementation of a SHA-1 block function
+ *
+ * This function takes a pointer to the current SHA-1 state, a pointer to the
+ * input data, and the number of 64-byte blocks to process.  The number of
+ * blocks to process is assumed to be nonzero.  Once all blocks have been
+ * processed, the state is updated with the new state.  This function only
+ * processes complete blocks.  State initialization, buffering of partial
+ * blocks, and digest finalization are expected to be handled elsewhere.
+ *
+ * void sha1_transform_phe(u8 *state, const u8 *data, size_t nblocks)
+ */
+.text
+SYM_FUNC_START(sha1_transform_phe)
+     mov             $-1, %rax
+     mov             %rdx, %rcx

lib/crypto/x86/sha1-phe-asm.S line 71

Quoted 5 lines

lib/crypto/x86/sha1-phe-asm.S new file +5 -0

+
+     .byte   0xf3,0x0f,0xa6,0xc8
+
+     RET
+SYM_FUNC_END(sha1_transform_phe)

Please make this an inline asm statement instead of using a .S file. It's just one instruction.

lib/crypto/x86/sha1-phe-asm.S

Quoted 10 lines

lib/crypto/x86/sha1-phe-asm.S new file +9 -0

+#define PHE_ALIGNMENT 16
+asmlinkage void sha1_transform_phe(u8 *state, const u8 *data, size_t nblocks);
+static void sha1_blocks_phe(struct sha1_block_state *state,
+                          const u8 *data, size_t nblocks)
+{
+     /*
+      * XSHA1 requires %edi to point to a 32-byte, 16-byte-aligned
+      * buffer on Zhaoxin processors.
+      */

What is the largest 'nblocks' that the instruction supports?

What happens if the instruction is interrupted partway through? Does the CPU correctly resume it in all cases?

Is it supported in both 32-bit and 64-bit modes? Your patch doesn't check for CONFIG_64BIT. Should it? New optimized assembly code generally should be 64-bit only.

Where is this instruction specified? Please add a comment that links to the specification.

lib/crypto/x86/sha1-phe-asm.S

Quoted 8 lines

lib/crypto/x86/sha1-phe-asm.S new file +7 -0

+     u8 buf[32 + PHE_ALIGNMENT - 1];
+     u8 *dst = PTR_ALIGN(&buf[0], PHE_ALIGNMENT);
+
+     memcpy(dst, (u8 *)(state), SHA1_DIGEST_SIZE);
+     sha1_transform_phe(dst, data, nblocks);
+     memcpy((u8 *)(state), dst, SHA1_DIGEST_SIZE);
+}

The casts to 'u8 *' are unnecessary.

lib/crypto/x86/sha1-phe-asm.S static void sha1_mod_init_arch(void) · line 81

Quoted 11 lines

lib/crypto/x86/sha1-phe-asm.S new file +4 -0

+
 static void sha1_blocks(struct sha1_block_state *state,
                      const u8 *data, size_t nblocks)
 {

@@ -59,6 +76,9 @@ static void sha1_mod_init_arch(void)
 {
      if (boot_cpu_has(X86_FEATURE_SHA_NI)) {
              static_call_update(sha1_blocks_x86, sha1_blocks_ni);
+     } else if (boot_cpu_has(X86_FEATURE_PHE) && boot_cpu_has(X86_FEATURE_PHE_EN)) {
+             if (cpu_data(0).x86 >= 0x07)
+                     static_call_update(sha1_blocks_x86, sha1_blocks_phe);

Check IS_ENABLED(CONFIG_CPU_SUP_ZHAOXIN) first, so that the code gets compiled out when support for Zhaoxin CPUs isn't included in the kernel.

There are hardly any mentions of 'cpu_data(0).x86' in the kernel. I think you mean 'boot_cpu_data.x86', which is used much more frequently.

What is the difference between X86_FEATURE_PHE and X86_FEATURE_PHE_EN, and why are both needed?

All these comments apply to the SHA-256 patch too.

Quoted 3 lines

Is it supported in both 32-bit and 64-bit modes? Your patch doesn't check for CONFIG_64BIT. Should it? New optimized assembly code generally should be 64-bit only.

The XSHA1 and XSHA256 are supported in both 32-bit and 64-bit modes. Since newly optimized assembly code is typically 64-bit only, and XSHA1 and XSHA256 fully support 64-bit mode, an explicit CONFIG_64BIT check should not required.

Quoted 2 lines

What is the difference between X86_FEATURE_PHE and X86_FEATURE_PHE_EN, and why are both needed?

The X86_FEATURE_PHE indicates the presence of the XSHA1 and XSHA256 instructions, whereas the X86_FEATURE_PHE_EN indicates that these instructions are enabled for normal use.

Quoted 8 lines · AlanSong-oc

Quoted 3 lines

Is it supported in both 32-bit and 64-bit modes? Your patch doesn't check for CONFIG_64BIT. Should it? New optimized assembly code generally should be 64-bit only.

The XSHA1 and XSHA256 are supported in both 32-bit and 64-bit modes. Since newly optimized assembly code is typically 64-bit only, and XSHA1 and XSHA256 fully support 64-bit mode, an explicit CONFIG_64BIT check should not required.

Right, all the x86-optimized SHA-1 and SHA-256 code is already 64-bit specific, due to CONFIG_CRYPTO_LIB_SHA1_ARCH and CONFIG_CRYPTO_LIB_SHA256_ARCH being enabled only when CONFIG_x86_64=y. So there's no need to check for 64-bit again.

Quoted 6 lines

Quoted 2 lines

What is the difference between X86_FEATURE_PHE and X86_FEATURE_PHE_EN, and why are both needed?

The X86_FEATURE_PHE indicates the presence of the XSHA1 and XSHA256 instructions, whereas the X86_FEATURE_PHE_EN indicates that these instructions are enabled for normal use.

I still don't understand the difference.

If you look at the other CPU feature flags, like X86_FEATURE_SHA_NI for example, there's just a single flag for the feature. We don't have X86_FEATURE_SHA_NI and X86_FEATURE_SHA_NI_EN. If the CPU supports the feature but the kernel decides it can't or shouldn't be used for whatever reason, the kernel just doesn't set the flag. There's no separate flag that tracks the CPU support independently.

Why can't the PHE flag work the same way?

For Zhaoxin processors, the XSHA1 instruction requires the total memory allocated at %rdi register must be 32 bytes, while the XSHA1 and XSHA256 instruction doesn't perform any operation when %ecx is zero.

Due to these requirements, the current padlock-sha driver does not work correctly with Zhaoxin processors. It cannot pass the self-tests and therefore does not activate the driver on Zhaoxin processors. This issue has been reported in Debian [1]. The self-tests fail with the following messages [2]:

alg: shash: sha1-padlock-nano test failed (wrong result) on test vector 0, cfg="init+update+final aligned buffer"
alg: self-tests for sha1 using sha1-padlock-nano failed (rc=-22)

------------[ cut here ]------------

alg: shash: sha256-padlock-nano test failed (wrong result) on test vector 0, cfg="init+update+final aligned buffer"
alg: self-tests for sha256 using sha256-padlock-nano failed (rc=-22)

------------[ cut here ]------------

To enable XSHA1 and XSHA256 instruction support on Zhaoxin processors, this series adds PHE Extensions support to lib/crypto for SHA-1 and SHA-256, following the suggestion in [3].

v1 link is below:
https://lore.kernel.org/linux-crypto/[email protected]/

Quoted 14 lines · AlanSong-oc

For Zhaoxin processors, the XSHA1 instruction requires the total memory allocated at %rdi register must be 32 bytes, while the XSHA1 and XSHA256 instruction doesn't perform any operation when %ecx is zero.

Due to these requirements, the current padlock-sha driver does not work correctly with Zhaoxin processors. It cannot pass the self-tests and therefore does not activate the driver on Zhaoxin processors. This issue has been reported in Debian [1]. The self-tests fail with the following messages [2]:

alg: shash: sha1-padlock-nano test failed (wrong result) on test vector 0, cfg="init+update+final aligned buffer"
alg: self-tests for sha1 using sha1-padlock-nano failed (rc=-22)

------------[ cut here ]------------

Quoted 3 lines

alg: shash: sha256-padlock-nano test failed (wrong result) on test vector 0, cfg="init+update+final aligned buffer"
alg: self-tests for sha256 using sha256-padlock-nano failed (rc=-22)

------------[ cut here ]------------

This cover letter is misleading, as those self-test failures will still exist regardless of this patch series.

Quoted 4 lines

To enable XSHA1 and XSHA256 instruction support on Zhaoxin processors, this series adds PHE Extensions support to lib/crypto for SHA-1 and SHA-256, following the suggestion in [3].

Quoted 2 lines

v1 link is below:
https://lore.kernel.org/linux-crypto/[email protected]/

Please run the sha1 and sha256 KUnit test suites (CRYPTO_LIB_SHA1_KUNIT_TEST and CRYPTO_LIB_SHA256_KUNIT_TEST) before and after this series, with the benchmark enabled (CRYPTO_LIB_BENCHMARK), and show the results. For this series to be considered, the tests need to pass and there needs to be a significant performance improvement.