arm64: Implement optimised IP checksum helpers
PATCH v2 1/1 arm64: Implement optimised IP checksum helpers
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of ip_fast_checksum() to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation for csum_fold() by defining it a slightly different way round from the generic version, so throw that into the mix too.
---
Minor changes: include types.h for correctness, add Luke's ack.
arch/arm64/include/asm/checksum.h | 51 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 arch/arm64/include/asm/checksum.h
arch/arm64/include/asm/checksum.h new file +51 -0
@@ -0,0 +1,51 @@ +/* +* Copyright (C) 2016 ARM Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ASM_CHECKSUM_H +#define __ASM_CHECKSUM_H + +#include <linux/types.h> + +static inline __sum16 csum_fold (__wsum csum) +{ +u32 sum = (__force u32 )csum; + sum += (sum >>16 ) | (sum <<16 ); +return ~(__force __sum16)(sum >>16 ); +} +#define csum_fold csum_fold + +static inline __sum16 ip_fast_csum (const void *iph,unsigned int ihl) +{ +__uint128_t tmp; +u64 sum; + + tmp = *(const __uint128_t *)iph; + iph +=16 ; + ihl -=4 ; + tmp += ((tmp >>64 ) | (tmp <<64 )); + sum = tmp >>64 ; +do { + sum += *(const u32 *)iph; + iph +=4 ; + }while (--ihl); + + sum += ((sum >>32 ) | (sum <<32 )); +return csum_fold (sum >>32 ); +} +#define ip_fast_csum ip_fast_csum + +#include <asm-generic/checksum.h> + +#endif /* __ASM_CHECKSUM_H */
Signature · 1 line
2.8.1.dirty
PATCH v2 1/1 arm64: Implement optimised IP checksum helpers
Hi Robin,
Quoted 9 lines · Robin Murphy
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of
ip_fast_checksum()to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation forcsum_fold()by defining it a slightly different way round from the generic version, so throw that into the mix too.
Quoted 8 lines
Suggested-by Luke Starrett <[email protected]>Acked-by Luke Starrett <[email protected]>Signed-off-by Robin Murphy <[email protected]>---
Minor changes: include
types.hfor correctness, add Luke's ack.arch/arm64/include/asm/checksum.h | 51 +++++++++++++++++++++++++++++++++++++++
Maybe a nit, don't you need to remove the 'generic-y += checksum.h' line from arch/arm64/include/asm/Kbuild to avoid the generated version being created too? [0]
The compiler on my box picks your header in preference to the generated one, but
[1] suggests it isn't to be trusted!
Thanks,
James
[0] d8ecc5cd8e22 ("kbuild: asm-generic support")
[1] https://lkml.org/lkml/2016/5/23/78
PATCH v2 1/1 arm64: Implement optimised IP checksum helpers
Hi James,
Quoted 17 lines · James Morse
Hi Robin,
Quoted 14 lines · Robin Murphy
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of
ip_fast_checksum()to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation forcsum_fold()by defining it a slightly different way round from the generic version, so throw that into the mix too.Suggested-by Luke Starrett <[email protected]>Acked-by Luke Starrett <[email protected]>Signed-off-by Robin Murphy <[email protected]>---
Quoted 6 lines
Quoted 3 lines
Minor changes: include
types.hfor correctness, add Luke's ack.arch/arm64/include/asm/checksum.h | 51 +++++++++++++++++++++++++++++++++++++++Maybe a nit, don't you need to remove the 'generic-y +=
checksum.h' line fromarch/arm64/include/asm/Kbuildto avoid the generated version being created too? [0]
I guess you skimmed the very end of the patch ;)
+#include <asm-generic/checksum.h>
Quoted 2 lines
The compiler on my box picks your header in preference to the generated one, but
[1] suggests it isn't to be trusted!
We only cherry-pick a couple of functions to optimise, so not building the generic file would necessitate copy-pasting the rest of it. I'd hope this pattern is sufficiently robust, since it's already in use on a number of other architectures! AFAICS [1] only applies when the generic file shouldn't be built at all, so cross fingers we're probably OK.
Robin.
Quoted 12 lines
Thanks,
James
[0]
d8ecc5cd8e22("kbuild: asm-generic support")
[1] https://lkml.org/lkml/2016/5/23/78_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/1 arm64: Implement optimised IP checksum helpers
Quoted 25 lines · Robin Murphy
Hi James,
Quoted 22 lines · James Morse
Hi Robin,
Quoted 18 lines · Robin Murphy
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of
ip_fast_checksum()to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation forcsum_fold()by defining it a slightly different way round from the generic version, so throw that into the mix too.Suggested-by Luke Starrett <[email protected]>Acked-by Luke Starrett <[email protected]>Signed-off-by Robin Murphy <[email protected]>---
Minor changes: include
types.hfor correctness, add Luke's ack.arch/arm64/include/asm/checksum.h | 51+++++++++++++++++++++++++++++++++++++++
Quoted 8 lines
Quoted 4 lines
Maybe a nit, don't you need to remove the 'generic-y +=
checksum.h' line fromarch/arm64/include/asm/Kbuildto avoid the generated version being created too? [0]I guess you skimmed the very end of the patch ;)
+#include <
asm-generic/checksum.h>
Oh wait, I'm being a massive idiot - include/asm-generic is not the same thing as arch/arm/include/generated/asm, is it? I see what you mean now, and yes it probably does want fixing...
Sorry!
Robin.
Quoted 30 lines
Quoted 3 lines
The compiler on my box picks your header in preference to the generated one, but
[1] suggests it isn't to be trusted!We only cherry-pick a couple of functions to optimise, so not building the generic file would necessitate copy-pasting the rest of it. I'd hope this pattern is sufficiently robust, since it's already in use on a number of other architectures! AFAICS [1] only applies when the generic file shouldn't be built at all, so cross fingers we're probably OK.
Robin.
Quoted 4 lines
Thanks,
James
Quoted 8 lines
[0]
d8ecc5cd8e22("kbuild: asm-generic support")
[1] https://lkml.org/lkml/2016/5/23/78_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v3 1/1 arm64: Implement optimised IP checksum helpers
In reply to Robin MurphyAArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of ip_fast_checksum() to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation for csum_fold() by defining it a slightly different way round from the generic version, so throw that into the mix too.
---
v3: Don't generate generic header [James]
v2: Include types.h, add Luke's ack
arch/arm64/include/asm/Kbuild | 1 - arch/arm64/include/asm/checksum.h | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/checksum.h
arch/arm64/include/asm/Kbuild +0 -1
@@ -1,6 +1,5 @@ generic-y += bug.h generic-y += bugs.h -generic-y += checksum.h generic-y += clkdev.h generic-y += cputime.h generic-y += current.h
arch/arm64/include/asm/checksum.h new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
2.8.1.dirty
PATCH v3 1/1 arm64: Implement optimised IP checksum helpers
Thanks for the arm64 checksum file. I saw a 4 fold speedup when calculating checksum for a 20 byte buffer on my test platform.
Thanks,
Sameer
Quoted 91 lines · Robin Murphy
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of
ip_fast_checksum()to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation forcsum_fold()by defining it a slightly different way round from the generic version, so throw that into the mix too.Suggested-by Luke Starrett <[email protected]>Acked-by Luke Starrett <[email protected]>Signed-off-by Robin Murphy <[email protected]>---
v3: Don't generate generic header [James]
v2: Includetypes.h, add Luke's ackarch/arm64/include/asm/Kbuild | 1 - arch/arm64/include/asm/checksum.h | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/checksum.harch/arm64/include/asm/Kbuild +0
-1@@ -1,6 +1,5 @@ generic-y += bug.h generic-y += bugs.h -generic-y += checksum.h generic-y += clkdev.h generic-y += cputime.h generic-y += current.harch/arm64/include/asm/checksum.h new file +51
-0@@ -0,0 +1,51 @@ +/* +* Copyright (C) 2016 ARM Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ASM_CHECKSUM_H +#define __ASM_CHECKSUM_H + +#include <linux/types.h> + +static inline __sum16 csum_fold (__wsum csum) +{ +u32 sum = (__force u32 )csum; + sum += (sum >>16 ) | (sum <<16 ); +return ~(__force __sum16)(sum >>16 ); +} +#define csum_fold csum_fold + +static inline __sum16 ip_fast_csum (const void *iph,unsigned int ihl) +{ +__uint128_t tmp; +u64 sum; + + tmp = *(const __uint128_t *)iph; + iph +=16 ; + ihl -=4 ; + tmp += ((tmp >>64 ) | (tmp <<64 )); + sum = tmp >>64 ; +do { + sum += *(const u32 *)iph; + iph +=4 ; + }while (--ihl); + + sum += ((sum >>32 ) | (sum <<32 )); +return csum_fold (sum >>32 ); +} +#define ip_fast_csum ip_fast_csum + +#include <asm-generic/checksum.h> + +#endif /* __ASM_CHECKSUM_H */
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
Catalin Marinas
maintainerPATCH v3 1/1 arm64: Implement optimised IP checksum helpers
In reply to Robin MurphyQuoted 9 lines · Robin Murphy
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of
ip_fast_checksum()to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation forcsum_fold()by defining it a slightly different way round from the generic version, so throw that into the mix too.
Quoted 3 lines
Suggested-by Luke Starrett <[email protected]>Acked-by Luke Starrett <[email protected]>Signed-off-by Robin Murphy <[email protected]>
I now applied the correct version. Thanks for pointing out.
Signature · 1 line
Catalin