Skip to content

riscv: Add fine-tuned checksum functions

Charlie Jenkins linux-arch linux-riscv lkml 15 versions – Applied

Status from this message: This series was applied to riscv/linux.git (fixes)

Get series with b4
b4 am [email protected] change-id 20230804-optimize_checksum-db145288ac21

Versions newest first

  1. v15 Latest5 patches 8 repliesApplied 5/5
  2. v145 patches 2 repliesSuperseded
  3. v135 patches 4 repliesSuperseded
Earlier versions · 12
VersionPostedPatchesRepliesStatus
v125 patches4 repliesSuperseded
v115 patches2 repliesSuperseded
v105 patches3 repliesSuperseded
v95 patches7 repliesSuperseded
v85 patches7 repliesSuperseded
v74 patches9 repliesSuperseded
v64 patches5 repliesSuperseded
v54 patches4 repliesSuperseded
v45 patches18 repliesSuperseded
v35 patches0 repliesSuperseded
v25 patches10 repliesSuperseded
v15 patches10 repliesSuperseded

Changes in v15 from the cover letter

- Create modify_unaligned_access_branches to consolidate duplicate code
(Evan)
- Link to v14: https://lore.kernel.org/r/[email protected]

Patches in v15 All 5 applied

#Subject and filesSizeRepliesApplied
1/5asm-generic: Improve csum_foldinclude/asm-generic/checksum.h+3 -30 repliesApplied
2/5riscv: Add static key for misaligned accessesarch/riscv/include/asm/cpufeature.h arch/riscv/kernel/cpufeature.c+89 -30 repliesApplied
3/5riscv: Add checksum headerarch/riscv/include/asm/checksum.h+82 -00 repliesApplied
4/5riscv: Add checksum libraryarch/riscv/include/asm/checksum.h arch/riscv/lib/Makefile arch/riscv/lib/csum.c+338 -00 repliesApplied
5/5kunit: Add tests for csum_ipv6_magic and ip_fast_csumlib/checksum_kunit.c+283 -18 repliesApplied

Reviews trailers per patch

4 of 5 patches have a Reviewed-by or Acked-by.

PatchDavid LaightEvan GreenConor DooleyXiao Wang
1/5R-b
2/5R-b
3/5A-b R-b
4/5A-b R-b
5/5

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v15 thread

Cover letter [PATCH v15 0/5] riscv: Add fine-tuned checksum functions

Each architecture generally implements fine-tuned checksum functions to leverage the instruction set. This patch adds the main checksum functions that are used in networking. Tested on QEMU, this series allows the CHECKSUM_KUNIT tests to complete an average of 50.9% faster.

This patch takes heavy use of the Zbb extension using alternatives patching.

To test this patch, enable the configs for KUNIT, then CHECKSUM_KUNIT.

I have attempted to make these functions as optimal as possible, but I have not ran anything on actual riscv hardware. My performance testing has been limited to inspecting the assembly, running the algorithms on x86 hardware, and running in QEMU.

ip_fast_csum is a relatively small function so even though it is possible to read 64 bits at a time on compatible hardware, the bottleneck becomes the clean up and setup code so loading 32 bits at a time is actually faster.

Relies on https://lore.kernel.org/lkml/[email protected]/

---

The algorithm proposed to replace the default csum_fold can be seen to compute the same result by running all 2^32 possible inputs.

static inline unsigned int ror32(unsigned int word, unsigned int shift)

{
	return (word >> (shift & 31)) | (word << ((-shift) & 31));
}

unsigned short csum_fold(unsigned int csum)

{
	unsigned int sum = csum;
	sum = (sum & 0xffff) + (sum >> 16);
	sum = (sum & 0xffff) + (sum >> 16);
	return ~sum;
}

unsigned short csum_fold_arc(unsigned int csum)

{
	return ((~csum - ror32(csum, 16)) >> 16);
}

int main()

{
	unsigned int start = 0x0;
	do {
		if (csum_fold(start) != csum_fold_arc(start)) {
			printf("Not the same %u\n", start);
			return -1;
		}
		start += 1;
	} while(start != 0x0);
	printf("The same\n");
	return 0;
}

Cc: Paul Walmsley <[email protected]>
Cc: Albert Ou <[email protected]>
Cc: Arnd Bergmann <[email protected]>
To: Charlie Jenkins <[email protected]>
To: Palmer Dabbelt <[email protected]>
To: Conor Dooley <[email protected]>
To: Samuel Holland <[email protected]>
To: David Laight <[email protected]>
To: Xiao Wang <[email protected]>
To: Evan Green <[email protected]>
To: Guo Ren <[email protected]>
To: [email protected]
To: [email protected]
To: [email protected]

---
Changes in v15:
- Create modify_unaligned_access_branches to consolidate duplicate code
(Evan)
- Link to v14: https://lore.kernel.org/r/[email protected]

Changes in v14:
- Update misaligned static branch when CPUs are hotplugged (Guo)
- Leave off Evan's reviewed-by on patch 2 since it was completely
re-written
- Link to v13: https://lore.kernel.org/r/[email protected]

Changes in v13:
- Move cast from patch 4 to patch 3
- Link to v12: https://lore.kernel.org/r/[email protected]

Changes in v12:
- Rebase onto 6.7-rc5
- Add performance stats in the cover letter
- Link to v11: https://lore.kernel.org/r/[email protected]

Changes in v11:
- Extensive modifications to comply to sparse
- Organize include statements (Xiao)
- Add csum_ipv6_magic to commit message (Xiao)
- Remove extraneous len statement (Xiao)
- Add kasan_check_read call (Xiao)
- Improve comment field checksum.h (Xiao)
- Consolidate "buff" and "len" into one parameter "end" (Xiao)
- Link to v10: https://lore.kernel.org/r/[email protected]

Changes in v10:
- Move tests that were riscv-specific to be arch agnostic (Arnd)
- Link to v9: https://lore.kernel.org/r/[email protected]

Changes in v9:
- Use ror64 (Xiao)
- Move do_csum and csum_ipv6_magic headers to patch 4 (Xiao)
- Remove word "IP" from checksum headers (Xiao)
- Swap to using ifndef CONFIG_32BIT instead of ifdef CONFIG_64BIT (Xiao)
- Run no alignment code when buff is aligned (Xiao)
- Consolidate two do_csum implementations overlap into do_csum_common
- Link to v8: https://lore.kernel.org/r/[email protected]

Changes in v8:
- Speedups of 12% without Zbb and 21% with Zbb when cpu supports fast
misaligned accesses for do_csum
- Various formatting updates
- Patch now relies on https://lore.kernel.org/lkml/[email protected]/
- Link to v7: https://lore.kernel.org/r/[email protected]

Changes in v7:
- Included linux/bitops.h in asm-generic/checksum.h to use ror (Conor)
- Optimized loop in do_csum (David)
- Used ror instead of shifting (David)
- Unfortunately had to reintroduce ifdefs because gcc is not smart
enough to not throw warnings on code that will never execute
- Use ifdef instead of IS_ENABLED on __LITTLE_ENDIAN because IS_ENABLED
does not work on that
- Only optimize for zbb when alternatives is enabled in do_csum
- Link to v6: https://lore.kernel.org/r/[email protected]

Changes in v6:
- Fix accuracy of commit message for csum_fold
- Fix indentation
- Link to v5: https://lore.kernel.org/r/[email protected]

Changes in v5:
- Drop vector patches
- Check ZBB enabled before doing any ZBB code (Conor)
- Check endianness in IS_ENABLED
- Revert to the simpler non-tree based version of ipv6_csum_magic since
David pointed out that the tree based version is not better.
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- Suggestion by David Laight to use an improved checksum used in
arch/arc.
- Eliminates zero-extension on rv32, but not on rv64.
- Reduces data dependency which should improve execution speed on
rv32 and rv64
- Still passes CHECKSUM_KUNIT and RISCV_CHECKSUM_KUNIT on rv32 and
rv64 with and without zbb.
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- Use riscv_has_extension_likely and has_vector where possible (Conor)
- Reduce ifdefs by using IS_ENABLED where possible (Conor)
- Use kernel_vector_begin in the vector code (Samuel)
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- After more benchmarking, rework functions to improve performance.
- Remove tests that overlapped with the already existing checksum
tests and make tests more extensive.
- Use alternatives to activate code with Zbb and vector extensions
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Charlie Jenkins (5):
asm-generic: Improve csum_fold
riscv: Add static key for misaligned accesses
riscv: Add checksum header
riscv: Add checksum library
kunit: Add tests for csum_ipv6_magic and ip_fast_csum

 arch/riscv/include/asm/checksum.h   |  93 ++++++++++
 arch/riscv/include/asm/cpufeature.h |   2 +
 arch/riscv/kernel/cpufeature.c      |  90 +++++++++-
 arch/riscv/lib/Makefile             |   1 +
 arch/riscv/lib/csum.c               | 326 ++++++++++++++++++++++++++++++++++++
 include/asm-generic/checksum.h      |   6 +-
 lib/checksum_kunit.c                | 284 ++++++++++++++++++++++++++++++-
 7 files changed, 795 insertions(+), 7 deletions(-)

---
base-commit: a39b6ac3781d46ba18193c9dbb2110f31e9bffe9
change-id: 20230804-optimize_checksum-db145288ac21

Signature · 1 line
- Charlie