net: optimize raw checksum computation
Subsystems (2)
PATCH v5 1/1 net: optimize raw checksum computation
From: Scott Mitchell <[email protected]>
__rte_raw_cksum uses a loop with memcpy on each iteration. GCC 15+ is able to vectorize the loop but Clang 18.1 is not. Replacing the memcpy with unaligned_uint16_t pointer access enables both GCC and Clang to vectorize with SSE/AVX/AVX-512.
Performance results from cksum_perf_autotest on Intel Xeon (Cascade Lake, AVX-512) built with Clang 18.1 (TSC cycles/byte):
Block size Before After Improvement
100 0.40 0.24 ~40%
1500 0.50 0.06 ~8x
9000 0.49 0.06 ~8x---
Changes in v5:
- Simplified __rte_raw_cksum pointer arithmetic using type-safe uint16_t* instead of RTE_PTR_ADD byte arithmetic
- Fixed GCC -Wmaybe-uninitialized false-positive warnings in rte_ipv4_phdr_cksum and rte_ipv6_phdr_cksum using compiler barriers (no assembly impact)
- Refactored hinic driver's local phdr_cksum implementations to call common functions, eliminating duplication
- Simplified phdr_cksum struct initialization with designated initializers
Changes in v4:
- Replaced manual 64-byte loop unrolling with simple pointer iteration
- Removed __rte_no_ubsan_alignment macro (no longer needed)
- Removed rte_ip6.h false-positive warning fix (unable to repro locally)
- Updated performance numbers: AVX-512 vectorization on Clang achieves ~8x improvement for large packets vs scalar implementation
Changes in v3:
- Added __rte_no_ubsan_alignment macro to suppress false-positive UBSAN alignment warnings when using unaligned_uint16_t
- Fixed false-positive GCC maybe-uninitialized warning in rte_ip6.h exposed by optimization (can be split to separate patch once verified on CI)
Changes in v2:
- Fixed UndefinedBehaviorSanitizer errors by adding uint32_t casts to prevent signed integer overflow in addition chains
- Restored uint32_t sum accumulator instead of uint64_t
- Added 64k length to test_cksum_perf.c
app/test/meson.build | 1 + app/test/test_cksum_fuzz.c | 241 +++++++++++++++++++++++++++++++ app/test/test_cksum_perf.c | 2 +- drivers/net/hinic/hinic_pmd_tx.c | 38 +---- lib/net/rte_cksum.h | 19 +-- lib/net/rte_ip4.h | 28 ++-- lib/net/rte_ip6.h | 19 +-- 7 files changed, 276 insertions(+), 72 deletions(-) create mode 100644 app/test/test_cksum_fuzz.c
app/test/meson.build +1 -0
@@ -38,6 +38,7 @@ source_file_deps = { 'test_byteorder.c': [], 'test_cfgfile.c': ['cfgfile'], 'test_cksum.c': ['net'], + 'test_cksum_fuzz.c': ['net'], 'test_cksum_perf.c': ['net'], 'test_cmdline.c': [], 'test_cmdline_cirbuf.c': [],
app/test/test_cksum_fuzz.c new file +241 -0
241 lines not on this page · raw message
app/test/test_cksum_perf.c +1 -1
@@ -15,7 +15,7 @@#define NUM_BLOCKS 10#define ITERATIONS 1000000 -static const size_t data_sizes[] = {20 ,21 ,100 ,101 ,1500 ,1501 }; +static const size_t data_sizes[] = {20 ,21 ,100 ,101 ,1500 ,1501 ,9000 ,9001 ,65536 ,65537 }; static __rte_noinline uint16_tdo_rte_raw_cksum (const void *buf,size_t len)
drivers/net/hinic/hinic_pmd_tx.c +2 -36
@@ -706,47 +706,13 @@ hinic_get_sq_wqe(struct hinic_txq *txq, int wqebb_cnt,static inline uint16_t hinic_ipv4_phdr_cksum (const struct rte_ipv4_hdr *ipv4_hdr,uint64_t ol_flags) { -struct ipv4_psd_header { -uint32_t src_addr;/* IP address of source host. */ -uint32_t dst_addr;/* IP address of destination host. */ -uint8_t zero;/* zero. */ -uint8_t proto;/* L4 protocol type. */ -uint16_t len;/* L4 length. */ - } psd_hdr; - - psd_hdr.src_addr = ipv4_hdr->src_addr; - psd_hdr.dst_addr = ipv4_hdr->dst_addr; - psd_hdr.zero =0 ; - psd_hdr.proto = ipv4_hdr->next_proto_id; -if (ol_flags &RTE_MBUF_F_TX_TCP_SEG ) { - psd_hdr.len =0 ; - }else { - psd_hdr.len = -rte_cpu_to_be_16 (rte_be_to_cpu_16 (ipv4_hdr->total_length) - -rte_ipv4_hdr_len (ipv4_hdr)); - } -return rte_raw_cksum (&psd_hdr,sizeof (psd_hdr)); +return rte_ipv4_phdr_cksum (ipv4_hdr, ol_flags &RTE_MBUF_F_TX_TCP_SEG ); }static inline uint16_t hinic_ipv6_phdr_cksum (const struct rte_ipv6_hdr *ipv6_hdr,uint64_t ol_flags) { -uint32_t sum; -struct { -uint32_t len;/* L4 length. */ -uint32_t proto;/* L4 protocol - top 3 bytes must be zero */ - } psd_hdr; - - psd_hdr.proto = (ipv6_hdr->proto <<24 ); -if (ol_flags &RTE_MBUF_F_TX_TCP_SEG ) - psd_hdr.len =0 ; -else - psd_hdr.len = ipv6_hdr->payload_len; - - sum =__rte_raw_cksum (&ipv6_hdr->src_addr, -sizeof (ipv6_hdr->src_addr) +sizeof (ipv6_hdr->dst_addr),0 ); - sum =__rte_raw_cksum (&psd_hdr,sizeof (psd_hdr), sum); -return __rte_raw_cksum_reduce (sum); +return rte_ipv6_phdr_cksum (ipv6_hdr, ol_flags &RTE_MBUF_F_TX_TCP_SEG ); } static inline void hinic_get_outer_cs_pld_offset(struct rte_mbuf *m,
lib/net/rte_cksum.h +8 -11
@@ -42,20 +42,17 @@ extern "C" {static inline uint32_t __rte_raw_cksum (const void *buf,size_t len,uint32_t sum) { -const void *end; - -for (end =RTE_PTR_ADD (buf,RTE_ALIGN_FLOOR (len,sizeof (uint16_t ))); - buf != end; buf =RTE_PTR_ADD (buf,sizeof (uint16_t ))) { -uint16_t v; - -memcpy (&v, buf,sizeof (uint16_t )); - sum += v; - } +/* Process as uint16 chunks to preserve overflow/carry math. */ +const unaligned_uint16_t *buf16 = (const unaligned_uint16_t *)buf; +/* Divide byte length by 2 to get number of uint16_t elements. */ +const unaligned_uint16_t *end = buf16 + (len >>1 ); +/* GCC and Clang vectorize the loop. */ +for (; buf16 != end; buf16++) + sum += *buf16;/* if length is odd, keeping it byte order independent */ -if (unlikely (len %2 )) { +if (len &1 ) {uint16_t left =0 ; -memcpy (&left, end,1 ); sum += left; }
lib/net/rte_ip4.h +13 -15
@@ -223,21 +223,19 @@ rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags)uint8_t zero;/* zero. */ uint8_t proto;/* L4 protocol type. */ uint16_t len;/* L4 length. */ - } psd_hdr; - -uint32_t l3_len; - - psd_hdr.src_addr = ipv4_hdr->src_addr; - psd_hdr.dst_addr = ipv4_hdr->dst_addr; - psd_hdr.zero =0 ; - psd_hdr.proto = ipv4_hdr->next_proto_id; -if (ol_flags & (RTE_MBUF_F_TX_TCP_SEG |RTE_MBUF_F_TX_UDP_SEG )) { - psd_hdr.len =0 ; - }else { - l3_len =rte_be_to_cpu_16 (ipv4_hdr->total_length); - psd_hdr.len =rte_cpu_to_be_16 ((uint16_t )(l3_len - -rte_ipv4_hdr_len (ipv4_hdr))); - } + } psd_hdr = { + .src_addr = ipv4_hdr->src_addr, + .dst_addr = ipv4_hdr->dst_addr, + .proto = ipv4_hdr->next_proto_id, + .len = (ol_flags & (RTE_MBUF_F_TX_TCP_SEG |RTE_MBUF_F_TX_UDP_SEG )) ?0 : +rte_cpu_to_be_16 ((uint16_t )(rte_be_to_cpu_16 (ipv4_hdr->total_length) - +rte_ipv4_hdr_len (ipv4_hdr))) + }; +#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endif +return rte_raw_cksum (&psd_hdr,sizeof (psd_hdr)); }
lib/net/rte_ip6.h +10 -9
@@ -560,19 +560,20 @@ rte_ipv6_phdr_cksum(const struct rte_ipv6_hdr *ipv6_hdr, uint64_t ol_flags)static inline uint16_t rte_ipv6_phdr_cksum (const struct rte_ipv6_hdr *ipv6_hdr,uint64_t ol_flags) { -uint32_t sum;struct {rte_be32_t len;/* L4 length. */ rte_be32_t proto;/* L4 protocol - top 3 bytes must be zero */ - } psd_hdr; - - psd_hdr.proto = (uint32_t )(ipv6_hdr->proto <<24 ); -if (ol_flags & (RTE_MBUF_F_TX_TCP_SEG |RTE_MBUF_F_TX_UDP_SEG )) - psd_hdr.len =0 ; -else - psd_hdr.len = ipv6_hdr->payload_len; + } psd_hdr = { + .proto = (uint32_t )(ipv6_hdr->proto <<24 ), + .len = (ol_flags & (RTE_MBUF_F_TX_TCP_SEG |RTE_MBUF_F_TX_UDP_SEG )) ?0 : + ipv6_hdr->payload_len + }; +#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endif - sum =__rte_raw_cksum (&ipv6_hdr->src_addr, +uint32_t sum =__rte_raw_cksum (&ipv6_hdr->src_addr,sizeof (ipv6_hdr->src_addr) +sizeof (ipv6_hdr->dst_addr),0 ); sum =__rte_raw_cksum (&psd_hdr,sizeof (psd_hdr), sum);
Signature · 1 line
2.39.5 (Apple Git-154)
lib/net/rte_ip4.h rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags)
Quoted 4 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
lib/net/rte_ip4.h +4
-0+#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endi
This seems odd and like a compiler bug, there must be a better solution.
lib/net/rte_ip4.h rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags) · line 237
Quoted 5 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
lib/net/rte_ip4.h +4
-0+#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endif
Maybe rte_compiler_barrier() will do same thing?
Quoted 11 lines · Stephen Hemminger
lib/net/rte_ip4.h rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags) · line 237
Quoted 5 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
lib/net/rte_ip4.h +4
-0+#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endif Maybe
rte_compiler_barrier()will do same thing?
Agreed it feels like a compiler bug but looking for advice if I'm missing something :)
My initial concern with rte_compiler_barrier is its a general barrier which may have broader impacts on
optimizations and compiled code. Will that be an issue in this case? I wasn't sure and the approach
in the patch is targeted at a specific variable and assembly from clang/gcc was the same. I will
introduce a macro to make it cleaner and I can replace it with rte_compiler_barrier if preferred.
Quoted 17 lines · On Thu, 8 Jan 2026 16:19:37 -0500 Scott Mitchell <[email protected]>…
Quoted 11 lines · Stephen Hemminger
lib/net/rte_ip4.h rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags) · line 237
Quoted 5 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
lib/net/rte_ip4.h +4
-0+#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endif Maybe
rte_compiler_barrier()will do same thing?Agreed it feels like a compiler bug but looking for advice if I'm missing something :)
Quoted 8 lines
My initial concern with
rte_compiler_barrieris its a general barrier which may have broader impacts on
optimizations and compiled code. Will that be an issue in this case? I wasn't sure and the approach
in the patch is targeted at a specific variable and assembly from clang/gcc was the same. I will
introduce a macro to make it cleaner and I can replace it withrte_compiler_barrierif preferred.
Maybe try with -fanalyzer and it might tell you more. I suspect some of the aliasing setting are causing issues. Some drivers are turning on no-strict-aliasing
Quoted 30 lines · Stephen Hemminger
Quoted 17 lines · On Thu, 8 Jan 2026 16:19:37 -0500 Scott Mitchell <[email protected]>…
Quoted 11 lines · Stephen Hemminger
lib/net/rte_ip4.h rte_ipv4_phdr_cksum(const struct rte_ipv4_hdr *ipv4_hdr, uint64_t ol_flags) · line 237
Quoted 5 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
lib/net/rte_ip4.h +4
-0+#ifdef RTE_CC_GCC +/* Suppress GCC -Wmaybe-uninitialized false positive. No assembly/runtime impacts. */ + asmvolatile ("" :"+m" (psd_hdr)); +#endif Maybe
rte_compiler_barrier()will do same thing?Agreed it feels like a compiler bug but looking for advice if I'm missing something :)
Quoted 8 lines
My initial concern with
rte_compiler_barrieris its a general barrier which may have broader impacts on
optimizations and compiled code. Will that be an issue in this case? I wasn't sure and the approach
in the patch is targeted at a specific variable and assembly from clang/gcc was the same. I will
introduce a macro to make it cleaner and I can replace it withrte_compiler_barrierif preferred.
Quoted 3 lines
Maybe try with -fanalyzer and it might tell you more. I suspect some of the aliasing setting are causing issues. Some drivers are turning on no-strict-aliasing
I have more evidence this is a GCC optimizer bug.
The RTE_SUPPRESS_UNINITIALIZED_WARNING approach serves
as a workaround to avoid the bug. I created a more minimal reproducer:
https://gist.github.com/Scottmitch/bf23748b4588e68c9bdb8d124f92f1bd
Your suspicion was correct, -fno-strict-aliasing avoids the bug but I don't think it is desirable to enable this broadly for DPDK when we have a more targeted workaround.
I will reach out to RH to confirm but in the interim I suggest we keep RTE_SUPPRESS_UNINITIALIZED_WARNING (or similar alternative).
+Aaron, please read up on this discussion, and step in if you can help.
(Aaron is the DPDK Project testing leader, and works at Red Hat.)
Quoted 47 lines
From: Scott Mitchell [mailto:[email protected]]
Sent: Friday, 9 January 2026 05.58Quoted 4 lines · Stephen Hemminger
On Thu, 8 Jan 2026 16:19:37 -0500
Scott Mitchell <[email protected]> wrote:Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Stephen Hemminger
Review
Quoted 2 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
(quoted diff) +2
-0+#ifdef RTE_CC_GCC + /* Suppress GCC -Wmaybe-uninitialized false positive. Noassembly/runtime impacts. */
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ asm volatile("" : "+m" (psd_hdr)); +#endifMaybe
rte_compiler_barrier()will do same thing?Agreed it feels like a compiler bug but looking for advice if I'm missing something :)
My initial concern with
rte_compiler_barrieris its a generalbarrier
Quoted 2 lines
Quoted 2 lines
which may have broader impacts on
optimizations and compiled code. Will that be an issue in thiscase? I
Quoted 9 lines
Quoted 5 lines
wasn't sure and the approach
in the patch is targeted at a specific variable and assembly from clang/gcc was the same. I will
introduce a macro to make it cleaner and I can replace it withrte_compiler_barrierif preferred.Maybe try with -fanalyzer and it might tell you more. I suspect some of the aliasing setting are causing issues. Some drivers are turning on no-strict-aliasing
I have more evidence this is a GCC optimizer bug.
TheRTE_SUPPRESS_UNINITIALIZED_WARNINGapproach serves
as a workaround to avoid the bug. I created a more minimal reproducer:
https://gist.github.com/Scottmitch/bf23748b4588e68c9bdb8d124f92f1bd
Quoted 7 lines
Your suspicion was correct, -fno-strict-aliasing avoids the bug but I don't
think it is desirable to enable this broadly for DPDK when we have a more targeted workaround.I will reach out to RH to confirm but in the interim I suggest we keep
RTE_SUPPRESS_UNINITIALIZED_WARNING(or similar alternative).
If this is a GCC compiler bug limited to the GCC version offered by RHEL 11, I prefer splitting the patch into a series with the following steps:
Patch 1/2: Add the optimization and new test cases in their minimal form, designed to work on normal compilers. Disregard bugs/warnings from the weird RHEL 11 compiler. I.e. don't modify lib/eal/include/rte_common.h, lib/net/rte_ip6.h, lib/net/rte_ip4.h, drivers/net/hinic/hinic_pmd_tx.c, drivers/net/mlx5/mlx5_flow_dv.c. Patch 2/2: Add the workarounds required by the RHEL 11 compiler.
Also, the change to drivers/net/hinic/hinic_pmd_tx.c should be moved to a patch independent of this series.
It's not directly related to this series, so let's not add more to the discussion than we need to. ;-)
And the implementation in the driver only considers RTE_MBUF_F_TX_TCP_SEG, whereas the DPDK function also considers RTE_MBUF_F_TX_UDP_SEG, so it warrants a separate discussion; it possibly fixes a bug.
Maybe even move the RHEL 11 related patches (my suggested patch 2/2) into a separate series, for the same conceptual reasons as moving the HINIC driver patch into a separate series. You can use the Depends-On tag (https://doc.dpdk.org/guides/contributing/patches.html#patch-dependencies) for the follow-on changes to __rte_raw_cksum().
Quoted 65 lines · Morten Brørup
+Aaron, please read up on this discussion, and step in if you can help.
(Aaron is the DPDK Project testing leader, and works at Red Hat.)
Quoted 47 lines
From: Scott Mitchell [mailto:[email protected]]
Sent: Friday, 9 January 2026 05.58Quoted 4 lines · Stephen Hemminger
On Thu, 8 Jan 2026 16:19:37 -0500
Scott Mitchell <[email protected]> wrote:Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Stephen Hemminger
Review
Quoted 2 lines · On Thu, 8 Jan 2026 01:13:38 -0500 [email protected] wrote:
(quoted diff) +2
-0+#ifdef RTE_CC_GCC + /* Suppress GCC -Wmaybe-uninitialized false positive. Noassembly/runtime impacts. */
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ asm volatile("" : "+m" (psd_hdr)); +#endifMaybe
rte_compiler_barrier()will do same thing?Agreed it feels like a compiler bug but looking for advice if I'm missing something :)
My initial concern with
rte_compiler_barrieris its a generalbarrier
Quoted 2 lines
Quoted 2 lines
which may have broader impacts on
optimizations and compiled code. Will that be an issue in thiscase? I
Quoted 9 lines
Quoted 5 lines
wasn't sure and the approach
in the patch is targeted at a specific variable and assembly from clang/gcc was the same. I will
introduce a macro to make it cleaner and I can replace it withrte_compiler_barrierif preferred.Maybe try with -fanalyzer and it might tell you more. I suspect some of the aliasing setting are causing issues. Some drivers are turning on no-strict-aliasing
I have more evidence this is a GCC optimizer bug.
TheRTE_SUPPRESS_UNINITIALIZED_WARNINGapproach serves
as a workaround to avoid the bug. I created a more minimal reproducer:
https://gist.github.com/Scottmitch/bf23748b4588e68c9bdb8d124f92f1bdQuoted 7 lines
Your suspicion was correct, -fno-strict-aliasing avoids the bug but I don't
think it is desirable to enable this broadly for DPDK when we have a more targeted workaround.I will reach out to RH to confirm but in the interim I suggest we keep
RTE_SUPPRESS_UNINITIALIZED_WARNING(or similar alternative).If this is a GCC compiler bug limited to the GCC version offered by RHEL 11, I prefer splitting the patch into a series with the following steps:
Patch 1/2: Add the optimization and new test cases in their minimal form, designed to work on normal compilers. Disregard bugs/warnings from the weird RHEL 11 compiler. I.e. don't modifylib/eal/include/rte_common.h,lib/net/rte_ip6.h,lib/net/rte_ip4.h,drivers/net/hinic/hinic_pmd_tx.c,drivers/net/mlx5/mlx5_flow_dv.c. Patch 2/2: Add the workarounds required by the RHEL 11 compiler.
Quoted 7 lines
Also, the change to
drivers/net/hinic/hinic_pmd_tx.cshould be moved to a patch independent of this series.
It's not directly related to this series, so let's not add more to the discussion than we need to. ;-)
And the implementation in the driver only considersRTE_MBUF_F_TX_TCP_SEG, whereas the DPDK function also considersRTE_MBUF_F_TX_UDP_SEG, so it warrants a separate discussion; it possibly fixes a bug.Maybe even move the RHEL 11 related patches (my suggested patch 2/2) into a separate series, for the same conceptual reasons as moving the HINIC driver patch into a separate series. You can use the Depends-On tag (https://doc.dpdk.org/guides/contributing/patches.html#patch-dependencies) for the follow-on changes to
__rte_raw_cksum().
General strategy makes sense!
The GCC bug showed up on dpdk CI on non-RH platforms too
https://github.com/ovsrobot/dpdk/actions/runs/20744990187/job/59560072349.
There is also a ubsan warning flagged while running the new fuzz test
(https://github.com/ovsrobot/dpdk/actions/runs/20821548318/job/59811205423).
Seems like a false positive and I added __rte_no_ubsan_alignment to get a clean build.
I'll restructure the changes following this approach:
Series: "net: optimize raw checksum computation" (3 patches) 1/3: net: optimize __rte_raw_cksum and add tests 2/3: add workaround for UBSAN alignment false positive 3/3: add workaround for GCC optimization bug
I'll defer the hinic refactoring (switching to common checksum functions) to a separate follow-up series as suggested, since it's independent of the core