ci: catch coredumps
RFC PATCH 1/1 ci: catch coredumps
Parts of the unit tests code rely on forked/secondary processes (expectedly) failing.
A crash in those situations could be missed so add a check on coredumps presence after unit tests have run.
In some situations (like explicitly call rte_panic), coredump generation must be disabled to avoid false positives.
---
Sending as a RFC, as this is a "nice to have" patch I had in store for some time, but I did not see the actual need so far.
We could attach the coredumps in GHA result, but it would be hardly usable without attaching all generated binaries...
Opinions?
--- .ci/linux-build.sh | 8 ++++++++ app/test/test_debug.c | 11 +++++++++-- app/test/test_mbuf.c | 9 ++++++++- 3 files changed, 25 insertions(+), 3 deletions(-)
.ci/linux-build.sh +8 -0
@@ -57,7 +57,11 @@ meson build --werror $OPTS ninja -C build if [ "$AARCH64" != "true" ]; then + ulimit -c unlimited + sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + devtools/test-null.sh + ! ls /tmp/dpdk-core.*.* 2>/dev/null fi if [ "$ABI_CHECKS" = "true" ]; then
@@ -102,5 +106,9 @@ if [ "$ABI_CHECKS" = "true" ]; then fi if [ "$RUN_TESTS" = "true" ]; then + ulimit -c unlimited + sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + sudo meson test -C build --suite fast-tests -t 3 + ! ls /tmp/dpdk-core.*.* 2>/dev/null fi
app/test/test_debug.c +9 -2
@@ -4,6 +4,8 @@#include <stdio.h> #include <stdint.h> +#include <sys/resource.h> +#include <sys/time.h> #include <sys/wait.h> #include <unistd.h>
@@ -28,9 +30,14 @@ test_panic(void) pid =fork (); -if (pid ==0 ) +if (pid ==0 ) { +struct rlimit rl; + +/* No need to generate a coredump when panicking. */ + rl.rlim_cur = rl.rlim_max =0 ; +setrlimit (RLIMIT_CORE , &rl);rte_panic ("Test Debug\n" ); -else if (pid <0 ){ + }else if (pid <0 ) {printf ("Fork Failed\n" );return -1 ; }
app/test/test_mbuf.c +8 -1
@@ -1174,6 +1174,8 @@ test_refcnt_mbuf(void) }#include <unistd.h> +#include <sys/resource.h> +#include <sys/time.h> #include <sys/wait.h> /* use fork() to test mbuf errors panic */
@@ -1186,9 +1188,14 @@ verify_mbuf_check_panics(struct rte_mbuf *buf) pid =fork ();if (pid ==0 ) { +struct rlimit rl; + +/* No need to generate a coredump when panicking. */ + rl.rlim_cur = rl.rlim_max =0 ; +setrlimit (RLIMIT_CORE , &rl);rte_mbuf_sanity_check (buf,1 );/* should panic */ exit (0 );/* return normally if it doesn't panic */ - }else if (pid <0 ){ + }else if (pid <0 ) {printf ("Fork Failed\n" );return -1 ; }
Signature · 1 line
2.23.0
Quoted 8 lines · David Marchand
Parts of the unit tests code rely on forked/secondary processes (expectedly) failing.
A crash in those situations could be missed so add a check on coredumps presence after unit tests have run.In some situations (like explicitly call
rte_panic), coredump generation must be disabled to avoid false positives.
Quoted 8 lines
Signed-off-by David Marchand <[email protected]>---
Sending as a RFC, as this is a "nice to have" patch I had in store for some time, but I did not see the actual need so far.We could attach the coredumps in GHA result, but it would be hardly usable without attaching all generated binaries...
Opinions?
I think it's a good start. We may not need to attach the binaries at all - if we have access to gdb, we can run a script to just run simple commands like 'thread apply all bt' and maybe some info commands. That can all be stuffed into the logs. Usually a source level backtrace is enough to work through what happened.
Quoted 91 lines
--- .ci/linux-build.sh | 8 ++++++++ app/test/test_debug.c | 11 +++++++++-- app/test/test_mbuf.c | 9 ++++++++- 3 files changed, 25 insertions(+), 3 deletions(-).ci/linux-build.sh +8
-0@@ -57,7 +57,11 @@ meson build --werror $OPTS ninja -C build if [ "$AARCH64" != "true" ]; then + ulimit -c unlimited + sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + devtools/test-null.sh + ! ls /tmp/dpdk-core.*.* 2>/dev/null fi if [ "$ABI_CHECKS" = "true" ]; then@@ -102,5 +106,9 @@ if [ "$ABI_CHECKS" = "true" ]; then fi if [ "$RUN_TESTS" = "true" ]; then + ulimit -c unlimited + sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p + sudo meson test -C build --suite fast-tests -t 3 + ! ls /tmp/dpdk-core.*.* 2>/dev/null fiapp/test/test_debug.c +9
-2@@ -4,6 +4,8 @@#include <stdio.h> #include <stdint.h> +#include <sys/resource.h> +#include <sys/time.h> #include <sys/wait.h> #include <unistd.h> @@ -28,9 +30,14 @@ test_panic(void) pid =fork (); -if (pid ==0 ) +if (pid ==0 ) { +struct rlimit rl; + +/* No need to generate a coredump when panicking. */ + rl.rlim_cur = rl.rlim_max =0 ; +setrlimit (RLIMIT_CORE , &rl);rte_panic ("Test Debug\n" ); -else if (pid <0 ){ + }else if (pid <0 ) {printf ("Fork Failed\n" );return -1 ; }app/test/test_mbuf.c +8
-1@@ -1174,6 +1174,8 @@ test_refcnt_mbuf(void) }#include <unistd.h> +#include <sys/resource.h> +#include <sys/time.h> #include <sys/wait.h> /* use fork() to test mbuf errors panic */ @@ -1186,9 +1188,14 @@ verify_mbuf_check_panics(struct rte_mbuf *buf) pid =fork ();if (pid ==0 ) { +struct rlimit rl; + +/* No need to generate a coredump when panicking. */ + rl.rlim_cur = rl.rlim_max =0 ; +setrlimit (RLIMIT_CORE , &rl);rte_mbuf_sanity_check (buf,1 );/* should panic */ exit (0 );/* return normally if it doesn't panic */ - }else if (pid <0 ){ + }else if (pid <0 ) {printf ("Fork Failed\n" );return -1 ; }
PATCH v2 1/1 ci: catch coredumps
In reply to David MarchandParts of the unit tests code rely on forked/secondary processes (expectedly) failing.
A crash in those situations could be missed so add a check on coredumps presence after unit tests have run.
When unit tests fail, it can also help checking for coredumps as it could give more insights on what happened.
In some situations (like explicit call to rte_panic), coredump generation must be disabled to avoid false positives.
---
Changelog since RFC v1
- removed RFC,
- pushed coredumps into gdb for in-situ analysis,
- gdb presence is used to enable the check. Travis config is left untouched for now,
--- .ci/linux-build.sh | 37 ++++++++++++++++++++++++++++++++++--- .github/workflows/build.yml | 4 ++++ app/test/test_debug.c | 11 +++++++++-- app/test/test_mbuf.c | 9 ++++++++- 4 files changed, 55 insertions(+), 6 deletions(-)
.ci/linux-build.sh +34 -3
@@ -4,7 +4,10 @@ on_error() { if [ $? = 0 ]; then exit fi - FILES_TO_PRINT="build/meson-logs/testlog.txt build/.ninja_log build/meson-logs/meson-log.txt" + FILES_TO_PRINT="build/meson-logs/testlog.txt" + FILES_TO_PRINT="$FILES_TO_PRINT build/.ninja_log" + FILES_TO_PRINT="$FILES_TO_PRINT build/meson-logs/meson-log.txt" + FILES_TO_PRINT="$FILES_TO_PRINT build/gdb.log" for pr_file in $FILES_TO_PRINT; do if [ -e "$pr_file" ]; then
@@ -30,6 +33,26 @@ install_libabigail() { rm ${version}.tar.gz } +configure_coredump() { + # No point in configuring coredump without gdb + which gdb >/dev/null || return 0 + ulimit -c unlimited + sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p +} + +catch_coredump() { + ls /tmp/dpdk-core.*.* 2>/dev/null || return 0 + for core in /tmp/dpdk-core.*.*; do + binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1) + [ -x $binary ] || binary= + sudo gdb $binary -c $core \ + -ex 'info threads' \ + -ex 'thread apply all bt full' \ + -ex 'quit' + done |tee -a build/gdb.log + return 1 +} + if [ "$AARCH64" = "true" ]; then # convert the arch specifier OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
@@ -57,7 +80,11 @@ meson build --werror $OPTS ninja -C build if [ "$AARCH64" != "true" ]; then - devtools/test-null.sh + failed= + configure_coredump + devtools/test-null.sh || failed="true" + catch_coredump + [ "$failed" != "true" ] fi if [ "$ABI_CHECKS" = "true" ]; then
@@ -102,5 +129,9 @@ if [ "$ABI_CHECKS" = "true" ]; then fi if [ "$RUN_TESTS" = "true" ]; then - sudo meson test -C build --suite fast-tests -t 3 + failed= + configure_coredump + sudo meson test -C build --suite fast-tests -t 3 || failed="true" + catch_coredump + [ "$failed" != "true" ] fi
.github/workflows/build.yml +4 -0
@@ -104,6 +104,9 @@ jobs: if: env.AARCH64 == 'true' run: sudo apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross pkg-config-aarch64-linux-gnu + - name: Install test tools packages + if: env.AARCH64 != 'true' || env.RUN_TESTS == 'true' + run: sudo apt install -y gdb - name: Install doc generation packages if: env.BUILD_DOCS == 'true' run: sudo apt install -y doxygen graphviz python3-sphinx
@@ -124,3 +127,4 @@ jobs: build/meson-logs/testlog.txt build/.ninja_log build/meson-logs/meson-log.txt + build/gdb.log
app/test/test_debug.c +9 -2
@@ -4,6 +4,8 @@#include <stdio.h> #include <stdint.h> +#include <sys/resource.h> +#include <sys/time.h> #include <sys/wait.h> #include <unistd.h>
@@ -28,9 +30,14 @@ test_panic(void) pid =fork (); -if (pid ==0 ) +if (pid ==0 ) { +struct rlimit rl; + +/* No need to generate a coredump when panicking. */ + rl.rlim_cur = rl.rlim_max =0 ; +setrlimit (RLIMIT_CORE , &rl);rte_panic ("Test Debug\n" ); -else if (pid <0 ){ + }else if (pid <0 ) {printf ("Fork Failed\n" );return -1 ; }
app/test/test_mbuf.c +8 -1
@@ -1174,6 +1174,8 @@ test_refcnt_mbuf(void) }#include <unistd.h> +#include <sys/resource.h> +#include <sys/time.h> #include <sys/wait.h> /* use fork() to test mbuf errors panic */
@@ -1186,9 +1188,14 @@ verify_mbuf_check_panics(struct rte_mbuf *buf) pid =fork ();if (pid ==0 ) { +struct rlimit rl; + +/* No need to generate a coredump when panicking. */ + rl.rlim_cur = rl.rlim_max =0 ; +setrlimit (RLIMIT_CORE , &rl);rte_mbuf_sanity_check (buf,1 );/* should panic */ exit (0 );/* return normally if it doesn't panic */ - }else if (pid <0 ){ + }else if (pid <0 ) {printf ("Fork Failed\n" );return -1 ; }
Signature · 1 line
2.23.0
Quoted 18 lines · David Marchand
Parts of the unit tests code rely on forked/secondary processes (expectedly) failing.
A crash in those situations could be missed so add a check on coredumps presence after unit tests have run.
When unit tests fail, it can also help checking for coredumps as it could give more insights on what happened.In some situations (like explicit call to
rte_panic), coredump generation must be disabled to avoid false positives.Signed-off-by David Marchand <[email protected]>---
Changelog since RFC v1
- removed RFC,
- pushed coredumps into gdb for in-situ analysis,
- gdb presence is used to enable the check. Travis config is left untouched for now,
Quoted 6 lines
--- .ci/linux-build.sh | 37 ++++++++++++++++++++++++++++++++++--- .github/workflows/build.yml | 4 ++++ app/test/test_debug.c | 11 +++++++++-- app/test/test_mbuf.c | 9 ++++++++- 4 files changed, 55 insertions(+), 6 deletions(-)
This fixes an issue for me, with running the debug_autotest under a reproducible build CI environment.
Signature · 2 lines
Kind regards, Luca Boccassi
Quoted 7 lines · David Marchand
Parts of the unit tests code rely on forked/secondary processes (expectedly) failing.
A crash in those situations could be missed so add a check on coredumps presence after unit tests have run.
When unit tests fail, it can also help checking for coredumps as it could give more insights on what happened.
Quoted 5 lines
In some situations (like explicit call to
rte_panic), coredump generation must be disabled to avoid false positives.Signed-off-by David Marchand <[email protected]>---
I had this in my backlog and accidentally marked it 'visited' - sorry about that.
Quoted 11 lines · David Marchand
Parts of the unit tests code rely on forked/secondary processes (expectedly) failing.
A crash in those situations could be missed so add a check on coredumps presence after unit tests have run.
When unit tests fail, it can also help checking for coredumps as it could give more insights on what happened.In some situations (like explicit call to
rte_panic), coredump generation must be disabled to avoid false positives.
As suggested by Luca, marking for backport.
Cc: [email protected]
Quoted 1 line
Signed-off-by David Marchand <[email protected]>
Signature · 1 line
David Marchand