ci: display UBSan stack trace
PATCH 1/1 ci: display UBSan stack trace
When UBSan raises an error, we get really few context.
Example, on a recently submitted patch:
RTE>>cksum_fuzz_autotest../lib/net/rte_cksum.h:49:10: runtime error: load of misaligned address
0x0001816c2e81 for type 'const unaligned_uint16_t' (aka 'const
unsigned short'), which requires 2 byte alignment
0x0001816c2e81: note: pointer points here
00 00 00 00 70 f2 00 00 00 00 00 00 00 00 00 00 00 00 00
^
00 00 00 00 00 00 00 00 00 00 00 00 00
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
../lib/net/rte_cksum.h:49:10 inAsk for the full stack.
--- .ci/linux-build.sh | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
.ci/linux-build.sh +14 -8
@@ -7,7 +7,11 @@ if [ -z "${DEF_LIB:-}" ]; then fi # Builds are run as root in containers, no need for sudo -[ "$(id -u)" != '0' ] || alias sudo= +if [ "$(id -u)" = '0' ]; then + run_as_root="" +else + run_as_root="sudo -E" +fi install_libabigail() { version=$1
@@ -28,15 +32,15 @@ configure_coredump() { # No point in configuring coredump without gdb which gdb >/dev/null || return 0 ulimit -c unlimited - sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + $run_as_root sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p } catch_coredump() { ls /tmp/dpdk-core.*.* 2>/dev/null || return 0 for core in /tmp/dpdk-core.*.*; do - binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) + binary=$($run_as_root readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) [ -x $binary ] || binary= - sudo gdb $binary -c $core \ + $run_as_root gdb $binary -c $core \ -ex 'info threads' \ -ex 'thread apply all bt full' \ -ex 'quit'
@@ -53,9 +57,9 @@ catch_ubsan() { check_traces() { which babeltrace >/dev/null || return 0 - for file in $(sudo find $HOME -name metadata); do - ! sudo babeltrace $(dirname $file) >/dev/null 2>&1 || continue - sudo babeltrace $(dirname $file) + for file in $($run_as_root find $HOME -name metadata); do + ! $run_as_root babeltrace $(dirname $file) >/dev/null 2>&1 || continue + $run_as_root babeltrace $(dirname $file) done }
@@ -136,6 +140,7 @@ fi if [ "$UBSAN" = "true" ]; then sanitizer=${sanitizer:+$sanitizer,}undefined + export UBSAN_OPTIONS=print_stacktrace=1 if [ "$RUN_TESTS" = "true" ]; then # UBSan takes too much memory with -O2 buildtype=plain
@@ -218,7 +223,8 @@ fi if [ "$RUN_TESTS" = "true" ]; then failed= configure_coredump - sudo meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs || failed="true" + $run_as_root meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs || + failed="true" catch_coredump catch_ubsan DPDK:fast-tests build/meson-logs/testlog.txt check_traces
Signature · 1 line
2.52.0
Quoted 34 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Wednesday 28 January 2026 11:05
To: [email protected]
Cc: [email protected]; Aaron Conole <[email protected]>
Subject: [PATCH] ci: display UBSan stack traceWhen UBSan raises an error, we get really few context.
Example, on a recently submitted patch:
RTE>>cksum_fuzz_autotest
../lib/net/rte_cksum.h:49:10: runtime error: load of misaligned address0x0001816c2e81 for type 'const unaligned_uint16_t' (aka 'const unsigned short'), which requires 2 byte alignment 0x0001816c2e81: note: pointer points here 00 00 00 00 70 f2 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ 00 00 00 00 00 00 00 00 00 00 00 00 00 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../lib/net/rte_cksum.h:49:10 inAsk for the full stack.
Signed-off-by David Marchand <[email protected]>--- .ci/linux-build.sh | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-).ci/linux-build.sh +0
-0
.ci/linux-build.sh if [ -z "${DEF_LIB:-}" ]; then · line 13
Quoted 8 lines
.ci/linux-build.sh +4
-1@@ -7,7 +7,11 @@ if [ -z "${DEF_LIB:-}" ]; then fi # Builds are run as root in containers, no need for sudo -[ "$(id -u)" != '0' ] || alias sudo= +if [ "$(id -u)" = '0' ]; then + run_as_root="" +else + run_as_root="sudo -E"
Just out of general principles, can we have --preserve-env=UBSAN_OPTIONS (or necessary list) instead?
And/or, do we need to preserve environment for all commands below?
Quoted 37 lines
.ci/linux-build.sh +7
-6+fi install_libabigail() { version=$1@@ -28,15 +32,15 @@ configure_coredump() { # No point in configuring coredump without gdb which gdb >/dev/null || return 0 ulimit -c unlimited - sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + $run_as_root sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p } catch_coredump() { ls /tmp/dpdk-core.*.* 2>/dev/null || return 0 for core in /tmp/dpdk-core.*.*; do - binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) + binary=$($run_as_root readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) [ -x $binary ] || binary= - sudo gdb $binary -c $core \ + $run_as_root gdb $binary -c $core \ -ex 'info threads' \ -ex 'thread apply all bt full' \ -ex 'quit'@@ -53,9 +57,9 @@ catch_ubsan() { check_traces() { which babeltrace >/dev/null || return 0 - for file in $(sudo find $HOME -name metadata); do - ! sudo babeltrace $(dirname $file) >/dev/null 2>&1 || continue - sudo babeltrace $(dirname $file) + for file in $($run_as_root find $HOME -name metadata); do + ! $run_as_root babeltrace $(dirname $file) >/dev/null 2>&1 || continue + $run_as_root babeltrace $(dirname $file) done }
.ci/linux-build.sh fi · line 143
Quoted 4 lines
.ci/linux-build.sh +1
-0@@ -136,6 +140,7 @@ fi if [ "$UBSAN" = "true" ]; then sanitizer=${sanitizer:+$sanitizer,}undefined + export UBSAN_OPTIONS=print_stacktrace=1
Won't it replace options meson sets by default? I actually see print_stacktrace=1 among them, although maybe depends on version and environment.
Also, do we care only about UBSAN_OPTIONS, or also ASAN_OPTIONS and LSAN_OPTIONS here?
I will share ones we are using in case you find some of them interesting:
ASAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:check_initialization_order=true:detect_stack_use_after_return=true:heap_profile=false:print_scariness=true:strict_init_order=true:verify_asan_link_order=false LSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true UBSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:print_stacktrace=true:report_error_type=true
(Admittedly some of them might be unnecessary or nonsensical, they depend on compiler and it's not easy to find documentation.)
Quoted 14 lines
.ci/linux-build.sh +2
-2if [ "$RUN_TESTS" = "true" ]; then # UBSan takes too much memory with -O2 buildtype=plain@@ -218,7 +223,8 @@ fi if [ "$RUN_TESTS" = "true" ]; then failed= configure_coredump - sudo meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs || failed="true" + $run_as_root meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs || + failed="true" catch_coredump catch_ubsan DPDK:fast-tests build/meson-logs/testlog.txt check_traces --
Quoted 1 line
2.52.0
With or without comments above addressed,
Hello,
Quoted 41 lines · Marat Khalili
Quoted 29 lines
Subject: [PATCH] ci: display UBSan stack trace
When UBSan raises an error, we get really few context.
Example, on a recently submitted patch:
RTE>>cksum_fuzz_autotest
../lib/net/rte_cksum.h:49:10: runtime error: load of misaligned address
0x0001816c2e81for type 'constunaligned_uint16_t' (aka 'const unsigned short'), which requires 2 byte alignment0x0001816c2e81: note: pointer points here
00 00 00 00 70 f2 00 00 00 00 00 00 00 00 00 00 00 00 00^ 00 00 00 00 00 00 00 00 00 00 00 00 00 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../lib/net/rte_cksum.h:49:10 inAsk for the full stack.
Signed-off-by David Marchand <[email protected]>--- .ci/linux-build.sh | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-).ci/linux-build.sh +0
-0.ci/linux-build.sh if [ -z "${DEF_LIB:-}" ]; then · line 13
Quoted 8 lines
.ci/linux-build.sh +4
-1@@ -7,7 +7,11 @@ if [ -z "${DEF_LIB:-}" ]; then fi # Builds are run as root in containers, no need for sudo -[ "$(id -u)" != '0' ] || alias sudo= +if [ "$(id -u)" = '0' ]; then + run_as_root="" +else + run_as_root="sudo -E"Just out of general principles, can we have --preserve-env=UBSAN_OPTIONS (or necessary list) instead?
Quoted 1 line
And/or, do we need to preserve environment for all commands below?
We could, but I wanted to keep the change simple.
In this patch, a single point in this script is used to trigger use of sudo or not.
Otherwise, I would have to keep two separate variable/alias for calling simple sudo, and running the unit tests.
And like this, there is no question about which variable must be preserved.
Quoted 39 lines
Quoted 39 lines
.ci/linux-build.sh +7
-6+fi install_libabigail() { version=$1@@ -28,15 +32,15 @@ configure_coredump() { # No point in configuring coredump without gdb which gdb >/dev/null || return 0 ulimit -c unlimited - sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + $run_as_root sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p } catch_coredump() { ls /tmp/dpdk-core.*.* 2>/dev/null || return 0 for core in /tmp/dpdk-core.*.*; do - binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) + binary=$($run_as_root readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) [ -x $binary ] || binary= - sudo gdb $binary -c $core \ + $run_as_root gdb $binary -c $core \ -ex 'info threads' \ -ex 'thread apply all bt full' \ -ex 'quit'@@ -53,9 +57,9 @@ catch_ubsan() { check_traces() { which babeltrace >/dev/null || return 0 - for file in $(sudo find $HOME -name metadata); do - ! sudo babeltrace $(dirname $file) >/dev/null 2>&1 || continue - sudo babeltrace $(dirname $file) + for file in $($run_as_root find $HOME -name metadata); do + ! $run_as_root babeltrace $(dirname $file) >/dev/null 2>&1 || continue + $run_as_root babeltrace $(dirname $file) done }@@ -136,6 +140,7 @@ fi
Quoted 7 lines
.ci/linux-build.sh fi · line 143
Quoted 3 lines
.ci/linux-build.sh +1
-0if [ "$UBSAN" = "true" ]; then sanitizer=${sanitizer:+$sanitizer,}undefined + export UBSAN_OPTIONS=print_stacktrace=1Won't it replace options meson sets by default? I actually see print_stacktrace=1 among them, although maybe depends on version and environment.
This change may overwrite anything that meson sets, but on the other hand, we need it with the current meson version used in the CI.
Quoted 4 lines
Also, do we care only about
UBSAN_OPTIONS, or alsoASAN_OPTIONSandLSAN_OPTIONShere?
Quoted 8 lines
I will share ones we are using in case you find some of them interesting:
ASAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:check_initialization_order=true:detect_stack_use_after_return=true:heap_profile=false:print_scariness=true:strict_init_order=true:verify_asan_link_order=false LSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true UBSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:print_stacktrace=true:report_error_type=true
(Admittedly some of them might be unnecessary or nonsensical, they depend on compiler and it's not easy to find documentation.)
Did those options help with understanding failures in unit tests? I see "abort" or "halt" options in this list, does it stop execution of all the tests?
Overall, the current script does everything, from detecting it needs to run parts with sudo, to configuring ubsan stuff (for example here). Maybe it could be interesting to better split the scripts, isolate what needs special permissions, but that's a different topic.
Signature · 1 line
David Marchand
Quoted 10 lines
Quoted 3 lines
I will share ones we are using in case you find some of them interesting:
ASAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_check s=true:check_initialization_order=true:detect_stack_use_after_return=true:heap_profile=false:print_sca riness=true:strict_init_order=true:verify_asan_link_order=false
Quoted 1 line
LSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_check s=true
Quoted 1 line
Quoted 7 lines
UBSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_chec ks=true:print_stacktrace=true:report_error_type=true
Quoted 3 lines
(Admittedly some of them might be unnecessary or nonsensical, they depend on compiler and it's not easy to find documentation.)
Did those options help with understanding failures in unit tests?
print_stacktrace is probably the most important for understanding failures. Some of the rest detect additional problems, others are mostly nice-to-have cosmetics (e.g. disabling ANSI sequences).
heap_profile was found to be useful for debugging but not for everyday tests since it may cause process to hang on exit (may depend on other options).
verify_asan_link_order=false is generally not needed, it disables a self-check.
report_error_type is very useful for adding suppressions.
Quoted 2 lines
I see "abort" or "halt" options in this list, does it stop execution of all the tests?
No, they only affects dpdk-test process, and meson runs each test in a separate process here, so continues running other tests. Both of them get set by my version of meson itself BTW. If set to false sanitizer complains to terminal but the test may actually pass (depending on specific problem).