net: optimize raw checksum computation
Subsystems (2)
v12 cover net: optimize raw checksum computation
From: Scott Mitchell <[email protected]>
This series optimizes __rte_raw_cksum() by replacing memcpy-based access with unaligned_uint16_t pointer access, enabling vectorization in both GCC and Clang. The series is split into three patches to clearly separate the core optimization from compiler-specific workarounds.
Performance improvement from cksum_perf_autotest on Intel Xeon (Cascade Lake, AVX-512) with Clang 18.1 (TSC cycles/byte):
Block size Before After Improvement
100 0.40 0.24 ~40%
1500 0.50 0.06 ~8x
9000 0.49 0.06 ~8xChanges in v12:
- Split into 3-patch series per reviewer feedback
- Patch 1/3: Core optimization and test additions
- Patch 2/3: UBSAN alignment workaround (separate from GCC bug)
- Patch 3/3: GCC optimization bug workaround
- Reverted len & 1 to len % 2 and restored unlikely() per feedback
- Renamed RTE_SUPPRESS_UNINITIALIZED_WARNING to RTE_FORCE_INIT_BARRIER
- Applied minimal changes (no refactoring) to existing code
- Deferred hinic driver refactoring to future series
Note: Patch 1/3 will trigger compiler warnings/failures on GCC versions with the optimization bug (GCC 11.5.0 and others seen on DPDK CI). These are resolved by patches 2/3 and 3/3.
Scott Mitchell (3):
net: optimize __rte_raw_cksum and add tests
eal: add workaround for UBSAN alignment false positive eal/net: add workaround for GCC optimization bug
app/test/meson.build | 1 + app/test/test_cksum_fuzz.c | 240 +++++++++++++++++++++++++++++++ app/test/test_cksum_perf.c | 2 +- drivers/net/hinic/hinic_pmd_tx.c | 2 + drivers/net/mlx5/mlx5_flow_dv.c | 2 + lib/eal/include/rte_common.h | 23 +++ lib/net/rte_cksum.h | 15 +- lib/net/rte_ip4.h | 1 + lib/net/rte_ip6.h | 1 + 9 files changed, 277 insertions(+), 10 deletions(-) create mode 100644 app/test/test_cksum_fuzz.c
--
2.39.5 (Apple Git-154)
PATCH v12 1/3 net: optimize __rte_raw_cksum and add tests
From: Scott <[email protected]>
__rte_raw_cksum uses a loop with memcpy on each iteration. GCC 15+ is able to vectorize the loop but Clang 18.1 is not. Replacing the memcpy with unaligned_uint16_t pointer access enables both GCC and Clang to vectorize with SSE/AVX/AVX-512.
This patch adds comprehensive fuzz testing and updates the performance test to measure the optimization impact.
Performance results from cksum_perf_autotest on Intel Xeon (Cascade Lake, AVX-512) built with Clang 18.1 (TSC cycles/byte):
Block size Before After Improvement
100 0.40 0.24 ~40%
1500 0.50 0.06 ~8x
9000 0.49 0.06 ~8x--- app/test/meson.build | 1 + app/test/test_cksum_fuzz.c | 240 +++++++++++++++++++++++++++++++++++++ app/test/test_cksum_perf.c | 2 +- lib/net/rte_cksum.h | 14 +-- 4 files changed, 247 insertions(+), 10 deletions(-) create mode 100644 app/test/test_cksum_fuzz.c
app/test/meson.build +1 -0
@@ -38,6 +38,7 @@ source_file_deps = { 'test_byteorder.c': [], 'test_cfgfile.c': ['cfgfile'], 'test_cksum.c': ['net'], + 'test_cksum_fuzz.c': ['net'], 'test_cksum_perf.c': ['net'], 'test_cmdline.c': [], 'test_cmdline_cirbuf.c': [],
app/test/test_cksum_fuzz.c new file +240 -0
240 lines not on this page · raw message
app/test/test_cksum_perf.c +1 -1
@@ -15,7 +15,7 @@#define NUM_BLOCKS 10#define ITERATIONS 1000000 -static const size_t data_sizes[] = {20 ,21 ,100 ,101 ,1500 ,1501 }; +static const size_t data_sizes[] = {20 ,21 ,100 ,101 ,1500 ,1501 ,9000 ,9001 ,65536 ,65537 }; static __rte_noinline uint16_tdo_rte_raw_cksum (const void *buf,size_t len)
lib/net/rte_cksum.h +5 -9
@@ -42,15 +42,11 @@ extern "C" {static inline uint32_t __rte_raw_cksum (const void *buf,size_t len,uint32_t sum) { -const void *end; - -for (end =RTE_PTR_ADD (buf,RTE_ALIGN_FLOOR (len,sizeof (uint16_t ))); - buf != end; buf =RTE_PTR_ADD (buf,sizeof (uint16_t ))) { -uint16_t v; - -memcpy (&v, buf,sizeof (uint16_t )); - sum += v; - } +/* Process uint16 chunks to preserve overflow/carry math. GCC/Clang vectorize the loop. */ +const unaligned_uint16_t *buf16 = (const unaligned_uint16_t *)buf; +const unaligned_uint16_t *end = buf16 + (len /sizeof (uint16_t )); +for (; buf16 != end; buf16++) + sum += *buf16;/* if length is odd, keeping it byte order independent */ if (unlikely (len %2 )) {
Signature · 1 line
2.39.5 (Apple Git-154)
Re: [PATCH v12 1/3] net: optimize __rte_raw_cksum and add tests
I will fix the ERROR:NO_AUTHOR_SIGN_OFF issue in a followup submission, but will wait in case there is other feedback to address.
RE: [PATCH v12 1/3] net: optimize __rte_raw_cksum and add tests
In reply to [email protected]Quoted 13 lines
From: Scott <[email protected]>
__rte_raw_cksumuses a loop with memcpy on each iteration. GCC 15+ is able to vectorize the loop but Clang 18.1 is not. Replacing the memcpy withunaligned_uint16_tpointer access enables both GCC and Clang to vectorize with SSE/AVX/AVX-512.This patch adds comprehensive fuzz testing and updates the performance test to measure the optimization impact.
Performance results from
cksum_perf_autoteston Intel Xeon (Cascade Lake, AVX-512) built with Clang 18.1 (TSC cycles/byte):
Quoted 7 lines
Block size Before After Improvement
100 0.40 0.24 ~40% 1500 0.50 0.06 ~8x 9000 0.49 0.06 ~8xSigned-off-by Scott Mitchell <[email protected]>---
Probably makes no practical difference, but consider marking the __rte_raw_cksum() function __rte_pure:
https://elixir.bootlin.com/dpdk/v25.11/source/lib/eal/include/rte_common.h#L228
With or without __rte_pure marking,
PATCH v12 2/3 eal: add workaround for UBSAN alignment false positive
In reply to [email protected]From: Scott <[email protected]>
The optimized __rte_raw_cksum() uses unaligned_uint16_t pointer access which triggers UBSAN alignment warnings even though the access is safe due to the unaligned type definition.
Add __rte_no_ubsan_alignment attribute to suppress these false positive warnings while preserving other UBSAN checks.
--- lib/eal/include/rte_common.h | 9 +++++++++ lib/net/rte_cksum.h | 1 + 2 files changed, 10 insertions(+)
lib/eal/include/rte_common.h +9 -0
@@ -546,6 +546,15 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)#define __rte_no_asan#endif +/** +* Disable UndefinedBehaviorSanitizer alignment check on some code +*/ +#if defined(RTE_CC_CLANG ) || defined(RTE_CC_GCC ) +#define __rte_no_ubsan_alignment __attribute__((no_sanitize("alignment"))) +#else +#define __rte_no_ubsan_alignment +#endif +/*********** Macros for pointer arithmetic ********/ /**
lib/net/rte_cksum.h +1 -0
@@ -39,6 +39,7 @@ extern "C" {* @return * sum += Sum of all words in the buffer. */ +__rte_no_ubsan_alignment static inline uint32_t__rte_raw_cksum (const void *buf,size_t len,uint32_t sum) {
Signature · 1 line
2.39.5 (Apple Git-154)
RE: [PATCH v12 2/3] eal: add workaround for UBSAN alignment false positive
Quoted 8 lines
From: Scott <[email protected]>
The optimized
__rte_raw_cksum()usesunaligned_uint16_tpointer access which triggers UBSAN alignment warnings even though the access is safe due to the unaligned type definition.Add
__rte_no_ubsan_alignmentattribute to suppress these false positive warnings while preserving other UBSAN checks.
I think root cause of the false positive warnings is a faulty type definition of unaligned_uint16_t in rte_common.h [1].
For architectures with strict alignment requirements, the type is correctly defined as:
typedef uint16_t unaligned_uint16_t __rte_aligned(1);
But for architectures with relaxed alignment requirements, the type definition is incomplete:
typedef uint16_t unaligned_uint16_t;
When UBSAN runs with the latter definition (which doesn't remove the type's alignment requirement), UBSAN is correctly warning about the misalignment.
Either UBSAN should run with the first definition (i.e. with __rte_aligned(1)), or the type definitions with __rte_aligned(1) should be used for all architectures.
[1]: https://elixir.bootlin.com/dpdk/v25.11/source/lib/eal/include/rte_common.h#L124
Stephen, WDYT?
Quoted 44 lines
Signed-off-by Scott Mitchell <[email protected]>--- lib/eal/include/rte_common.h | 9 +++++++++ lib/net/rte_cksum.h | 1 + 2 files changed, 10 insertions(+)lib/eal/include/rte_common.h +9
-0@@ -546,6 +546,15 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)#define __rte_no_asan#endif +/** +* Disable UndefinedBehaviorSanitizer alignment check on some code +*/ +#if defined(RTE_CC_CLANG ) || defined(RTE_CC_GCC ) +#define __rte_no_ubsan_alignment __attribute__((no_sanitize("alignment"))) +#else +#define __rte_no_ubsan_alignment +#endif +/*********** Macros for pointer arithmetic ********/ /**
lib/net/rte_cksum.h +1
-1@@ -39,6 +39,7 @@ extern "C" {* @return * sum += Sum of all words in the buffer. */ +__rte_no_ubsan_alignment static inline uint32_t__rte_raw_cksum (const void *buf,size_t len,uint32_t sum) { --2.39.5 (Apple Git-154)
PATCH v12 3/3 eal/net: add workaround for GCC optimization bug
In reply to [email protected]From: Scott <[email protected]>
GCC has a bug where it incorrectly elides struct initialization in inline functions when strict aliasing is enabled (-O2/-O3/-Os), causing reads from uninitialized memory. This affects both designated initializers and manual field assignment.
Add RTE_FORCE_INIT_BARRIER macro that uses an asm volatile memory barrier to prevent the compiler from incorrectly optimizing away struct initialization. Apply the workaround to pseudo-header checksum functions in rte_ip4.h, rte_ip6.h, hinic driver, and mlx5 driver.
--- drivers/net/hinic/hinic_pmd_tx.c | 2 ++ drivers/net/mlx5/mlx5_flow_dv.c | 2 ++ lib/eal/include/rte_common.h | 14 ++++++++++++++ lib/net/rte_ip4.h | 1 + lib/net/rte_ip6.h | 1 + 5 files changed, 20 insertions(+)
drivers/net/hinic/hinic_pmd_tx.c +2 -0
@@ -725,6 +725,7 @@ hinic_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags)rte_cpu_to_be_16 (rte_be_to_cpu_16 (ipv4_hdr->total_length) -rte_ipv4_hdr_len (ipv4_hdr)); } +RTE_FORCE_INIT_BARRIER (psd_hdr);return rte_raw_cksum (&psd_hdr,sizeof (psd_hdr)); }
@@ -743,6 +744,7 @@ hinic_ipv6_phdr_cksum(const struct rte_ipv6_hdr *ipv6_hdr, uint64_t ol_flags)else psd_hdr.len = ipv6_hdr->payload_len; +RTE_FORCE_INIT_BARRIER (psd_hdr); sum =__rte_raw_cksum (&ipv6_hdr->src_addr,sizeof (ipv6_hdr->src_addr) +sizeof (ipv6_hdr->dst_addr),0 ); sum =__rte_raw_cksum (&psd_hdr,sizeof (psd_hdr), sum);
drivers/net/mlx5/mlx5_flow_dv.c +2 -0
@@ -4445,6 +4445,8 @@ __flow_encap_decap_resource_register(struct rte_eth_dev *dev, .reserve = 0, } }; +RTE_FORCE_INIT_BARRIER (encap_decap_key); +struct mlx5_flow_cb_ctx ctx = { .error = error, .data = resource,
lib/eal/include/rte_common.h +14 -0
@@ -555,6 +555,20 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)#define __rte_no_ubsan_alignment#endif +/** +* Force struct initialization to prevent GCC optimization bug. +* GCC has a bug where it incorrectly elides struct initialization in +* inline functions when strict aliasing is enabled, causing reads from +* uninitialized memory. This memory barrier prevents the misoptimization. +*/ +#ifdef RTE_CC_GCC +#define RTE_FORCE_INIT_BARRIER (var) do { \ + asm volatile("" : "+m" (var)); \ +} while (0) +#else +#define RTE_FORCE_INIT_BARRIER (var) +#endif +/*********** Macros for pointer arithmetic ********/ /**
lib/net/rte_ip4.h +1 -0
@@ -238,6 +238,7 @@ rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags) psd_hdr.len =rte_cpu_to_be_16 ((uint16_t )(l3_len -rte_ipv4_hdr_len (ipv4_hdr))); } +RTE_FORCE_INIT_BARRIER (psd_hdr);return rte_raw_cksum (&psd_hdr,sizeof (psd_hdr)); }
lib/net/rte_ip6.h +1 -0
@@ -572,6 +572,7 @@ rte_ipv6_phdr_cksum(const struct rte_ipv6_hdr *ipv6_hdr, uint64_t ol_flags)else psd_hdr.len = ipv6_hdr->payload_len; +RTE_FORCE_INIT_BARRIER (psd_hdr); sum =__rte_raw_cksum (&ipv6_hdr->src_addr,sizeof (ipv6_hdr->src_addr) +sizeof (ipv6_hdr->dst_addr),0 );
Signature · 1 line
2.39.5 (Apple Git-154)
RE: [PATCH v12 3/3] eal/net: add workaround for GCC optimization bug
Quoted 6 lines
From: Scott <[email protected]>
GCC has a bug where it incorrectly elides struct initialization in inline functions when strict aliasing is enabled (-O2/-O3/-Os), causing reads from uninitialized memory. This affects both designated initializers and manual field assignment.
Does this still happen after you replaced the RTE_PTR_ADD() with native pointer arithmetic in the checksum function? In other words: Is this workaround still necessary?
This is a showstopper:
If the workaround is necessary, applications with similar use cases also need to apply the workaround. If we cannot somehow enforce that, the series is likely to break some applications, which is unacceptable.
Quoted 113 lines
Add
RTE_FORCE_INIT_BARRIERmacro that uses an asm volatile memory barrier
to prevent the compiler from incorrectly optimizing away struct initialization. Apply the workaround to pseudo-header checksum functions
inrte_ip4.h,rte_ip6.h, hinic driver, and mlx5 driver.Signed-off-by Scott Mitchell <[email protected]>--- drivers/net/hinic/hinic_pmd_tx.c | 2 ++ drivers/net/mlx5/mlx5_flow_dv.c | 2 ++ lib/eal/include/rte_common.h | 14 ++++++++++++++ lib/net/rte_ip4.h | 1 + lib/net/rte_ip6.h | 1 + 5 files changed, 20 insertions(+)drivers/net/hinic/hinic_pmd_tx.c +2
-0@@ -725,6 +725,7 @@ hinic_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags)rte_cpu_to_be_16 (rte_be_to_cpu_16 (ipv4_hdr->total_length) -rte_ipv4_hdr_len (ipv4_hdr)); } +RTE_FORCE_INIT_BARRIER (psd_hdr);return rte_raw_cksum (&psd_hdr,sizeof (psd_hdr)); }@@ -743,6 +744,7 @@ hinic_ipv6_phdr_cksum(const struct rte_ipv6_hdr *ipv6_hdr, uint64_t ol_flags) else psd_hdr.len = ipv6_hdr->payload_len; +RTE_FORCE_INIT_BARRIER (psd_hdr); sum = __rte_raw_cksum(&ipv6_hdr->src_addr, sizeof(ipv6_hdr->src_addr) + sizeof(ipv6_hdr->dst_addr),0);
drivers/net/hinic/hinic_pmd_tx.c +0
-0sum =__rte_raw_cksum (&psd_hdr,sizeof (psd_hdr), sum);drivers/net/mlx5/mlx5_flow_dv.c +2
-0@@ -4445,6 +4445,8 @@ __flow_encap_decap_resource_register(structrte_eth_dev *dev, .reserve = 0, } }; +RTE_FORCE_INIT_BARRIER (encap_decap_key ); +struct mlx5_flow_cb_ctx ctx = { .error = error,drivers/net/mlx5/mlx5_flow_dv.c +0
-0.data = resource,lib/eal/include/rte_common.h +4
-0@@ -555,6 +555,20 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)#define __rte_no_ubsan_alignment#endif +/** +* Force struct initialization to prevent GCC optimization bug. +* GCC has a bug where it incorrectly elides struct initialization in +* inline functions when strict aliasing is enabled, causing reads from
+ * uninitialized memory. This memory barrier prevents the misoptimization.
+ */lib/eal/include/rte_common.h +8
-0+#ifdef RTE_CC_GCC +#define RTE_FORCE_INIT_BARRIER (var) do { \ + asm volatile("" : "+m" (var)); \ +} while (0) +#else +#define RTE_FORCE_INIT_BARRIER (var) +#endif +/*********** Macros for pointer arithmetic ********/ /** lib/net/rte_ip4.h +1
-0@@ -238,6 +238,7 @@ rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags) psd_hdr.len =rte_cpu_to_be_16 ((uint16_t )(l3_len -rte_ipv4_hdr_len (ipv4_hdr))); } +RTE_FORCE_INIT_BARRIER (psd_hdr);return rte_raw_cksum (&psd_hdr,sizeof (psd_hdr)); }lib/net/rte_ip6.h +1
-0@@ -572,6 +572,7 @@ rte_ipv6_phdr_cksum(const struct rte_ipv6_hdr *ipv6_hdr, uint64_t ol_flags) else psd_hdr.len = ipv6_hdr->payload_len; +RTE_FORCE_INIT_BARRIER (psd_hdr); sum = __rte_raw_cksum(&ipv6_hdr->src_addr, sizeof(ipv6_hdr->src_addr) + sizeof(ipv6_hdr->dst_addr),0);
--
2.39.5 (Apple Git-154)
Re: [PATCH v12 3/3] eal/net: add workaround for GCC optimization bug
Quoted 2 lines
Does this still happen after you replaced the
RTE_PTR_ADD()with native pointer arithmetic in the checksum function? In other words: Is this workaround still necessary?
Yes unfortunately it is necessary with the pointer access. I updated the reproducer which shows this case:
https://gist.github.com/Scottmitch/bf23748b4588e68c9bdb8d124f92f1bd
Quoted 3 lines
This is a showstopper:
If the workaround is necessary, applications with similar use cases also need to apply the workaround. If we cannot somehow enforce that, the series is likely to break some applications, which is unacceptable.
That is a great point. This API isn't internal-only and this would effectively be
an API breaking change which doesn't seem justified.
Given what I've learned through this process (thank you & stephen for valuable feedback)
we have a few paths to achieve my goal (clang optimizes __rte_raw_cksum). I've verified
if the RTE_PTR_ADD macros are changed to use char* clang optimizes (and gcc still does too) [1]. To achieve this we have some options:
A. Modify RTE_PTR_[ADD|SUB] to use pointers
pros:
- [if API can be preserved] provides benefits to all use cases w/out usage changes
- no additional API surface to expose
cons:
- more complex macro implementation to preserve API compatibility.
B. Add RTE_CONST_PTR_[ADD|SUB] with const [void*|char*] & use it in __rte_raw_cksum
pros:
- no risk of impacting existing RTE_PTR_[ADD|SUB] APIs
- simple implementation using pointers from the start
cons:
- API may not support all use cases as RTE_PTR_[ADD|SUB] (e.g. ptr arg as raw integer)
- requires manual opt-in to new API to get any benefit
I have a draft of A I will submit as a patch and we can discuss if it makes sense
or fallback to B (or other approaches).
RE: [PATCH v12 3/3] eal/net: add workaround for GCC optimization bug
Quoted 10 lines
From: Scott Mitchell [mailto:[email protected]]
Sent: Sunday, 11 January 2026 07.22Quoted 1 line
Does this still happen after you replaced the
RTE_PTR_ADD()withnative pointer arithmetic in the checksum function?
Quoted 1 line
In other words: Is this workaround still necessary?
Yes unfortunately it is necessary with the pointer access. I updated the reproducer which shows this case:
https://gist.github.com/Scottmitch/bf23748b4588e68c9bdb8d124f92f1bd
I wonder if some of the magic types in rte_memcpy() [1] could be used to fix the aliasing/eliding problem in the checksum function:
struct __rte_packed_begin rte_unaligned_uint16_alias {
unaligned_uint16_t val;
} __rte_packed_end __rte_may_alias;[1]: https://elixir.bootlin.com/dpdk/v25.11/source/lib/eal/x86/include/rte_memcpy.h#L66
Re: [PATCH v12 3/3] eal/net: add workaround for GCC optimization bug
Quoted 8 lines
I wonder if some of the magic types in
rte_memcpy()[1] could be used to fix the aliasing/eliding problem in the checksum function:struct __rte_packed_begin rte_unaligned_uint16_alias { unaligned_uint16_t val; } __rte_packed_end __rte_may_alias;[1]: https://elixir.bootlin.com/dpdk/v25.11/source/lib/eal/x86/include/rte_memcpy.h#L66
Nice tip! clang/gcc optimize this (https://godbolt.org/z/WxjEbj6s4) and my reproducer doesn't trigger the gcc bug. v13 coming soon.
Quoted 46 lines · On Fri, 9 Jan 2026 20:56:48 -0500 [email protected] wrote:
From: Scott Mitchell <[email protected]>
This series optimizes
__rte_raw_cksum()by replacing memcpy-based access withunaligned_uint16_tpointer access, enabling vectorization in both GCC and Clang. The series is split into three patches to clearly separate the core optimization from compiler-specific workarounds.Performance improvement from
cksum_perf_autoteston Intel Xeon (Cascade Lake, AVX-512) with Clang 18.1 (TSC cycles/byte):Block size Before After Improvement
100 0.40 0.24 ~40% 1500 0.50 0.06 ~8x 9000 0.49 0.06 ~8xChanges in v12:
- Split into 3-patch series per reviewer feedback
- Patch 1/3: Core optimization and test additions
- Patch 2/3: UBSAN alignment workaround (separate from GCC bug)
- Patch 3/3: GCC optimization bug workaround
- Reverted len & 1 to len % 2 and restoredunlikely()per feedback
- RenamedRTE_SUPPRESS_UNINITIALIZED_WARNINGtoRTE_FORCE_INIT_BARRIER
- Applied minimal changes (no refactoring) to existing code
- Deferred hinic driver refactoring to future seriesNote: Patch 1/3 will trigger compiler warnings/failures on GCC versions with the optimization bug (GCC 11.5.0 and others seen on DPDK CI). These are resolved by patches 2/3 and 3/3.
Scott Mitchell (3):
net: optimize__rte_raw_cksumand add tests
eal: add workaround for UBSAN alignment false positive eal/net: add workaround for GCC optimization bugapp/test/meson.build | 1 + app/test/test_cksum_fuzz.c | 240 +++++++++++++++++++++++++++++++ app/test/test_cksum_perf.c | 2 +- drivers/net/hinic/hinic_pmd_tx.c | 2 + drivers/net/mlx5/mlx5_flow_dv.c | 2 + lib/eal/include/rte_common.h | 23 +++ lib/net/rte_cksum.h | 15 +- lib/net/rte_ip4.h | 1 + lib/net/rte_ip6.h | 1 + 9 files changed, 277 insertions(+), 10 deletions(-) create mode 100644 app/test/test_cksum_fuzz.c
Quoted 3 lines
--
2.39.5 (Apple Git-154)
Do not like to see so much compiler hacking, and not sure if it is worth having to do so much special case code for a few cycles.
Feeling lazy, asked AI to compare the FreeBSD and Linux checksum and its answer was enlightening.
The optimium solution would in my opinion would be:
- specific IP and IPv6 only routines using assembly on a platform basis. Add with carry will be faster than all this compiler flags.
- non-inline function for the general case with data.
This should eliminate the need for compiler flags.
Compare the algorithm used to compute standard IP checksum in FreeBSD and Linux
Great question! Both implement the same RFC 1071 one's complement checksum, but their optimization strategies differ in interesting ways.
## Core Algorithm
Both compute a 16-bit one's complement sum: accumulate 16-bit words, fold carries back in, and complement the result. The differences are in the implementation details.
## Linux Approach
Linux uses a layered architecture with generic C fallbacks and arch-specific optimizations:
**Generic path** (lib/checksum.c, include/net/checksum.h):
- do_csum() uses a 64-bit accumulator, processing 32 bytes per loop iteration (heavily unrolled)
- Defers folding until the end to minimize carry handling
- csum_fold() does the final 32→16 bit fold with a clever trick:
```c sum = (sum & 0xffff) + (sum >> 16); sum = (sum & 0xffff) + (sum >> 16); return ~sum; ```
**IP header fast path** (ip_fast_csum()):
- Specialized for the common 20-byte header case
- On x86-64, uses ADC (add-with-carry) chains to exploit hardware carry propagation
- ARM64 uses similar ADCS instruction chains
## FreeBSD Approach
FreeBSD's in_cksum() (sys/netinet/in_cksum.c) takes a different approach:
**mbuf-oriented design**:
- Walks mbuf chains directly, handling the common case of headers split across mbufs
- Tracks byte-swapping state when data crosses 16-bit boundaries between mbufs
**Accumulation strategy**:
- Uses int (32-bit) accumulator in the generic version
- Processes in 16-byte chunks with explicit unrolling
- Folds more frequently (after each mbuf) rather than deferring everything
**Architecture-specific versions**:
- x86 versions historically used inline assembly with ADC chains
- More recent versions lean on compiler intrinsics
## Key Differences
| Aspect | Linux | FreeBSD | |--------|-------|---------| | Accumulator | 64-bit, fold once at end | 32-bit, fold per mbuf | | Loop unroll | 32 bytes/iteration | 16 bytes/iteration | | Data structure | Works on flat buffers via `skb_checksum()` | Direct mbuf chain walking | | Alignment | Handles via initial byte adjustment | Similar, but tracks "odd byte" state across mbufs | | IP header | Dedicated `ip_fast_csum()` | Often uses general `in_cksum()` |
## Performance Implications
Linux's approach tends to be faster for large contiguous buffers due to larger unrolling and deferred folding. FreeBSD's mbuf-aware design is arguably cleaner for scattered data but may do more fold operations.
On modern CPUs with good branch prediction and out-of-order execution, the differences are often marginal—the ADC chain optimizations matter more than the high-level strategy. Both have moved toward letting the compiler handle more optimization rather than hand-tuned assembly, though Linux still maintains more arch-specific variants.
Given your DPDK work, you've probably noticed DPDK's rte_raw_cksum() takes yet another approach—often vectorized with SIMD when available, which can significantly outperform both kernel implementations for large payloads.