test/common: stop relying on bash $SECONDS in check_dmesg()
As requested, this is the broken-down, first part of
https://lore.kernel.org/linux-cxl/[email protected]/
test: fail on unexpected kernel error & warning, not just "Call Trace"
I will resubmit the rest after the review of this first part is completed. The different parts are logically separate (different "features") but they are interleaved in the same function and I don't really have the time to fix one git conflict per review comment.
While it is a requirement to catch warnings and errors later, the better accuracy provided by this first part is useful alone. If you are interested in what's next, just look at the bigger patch above. You don't have to!
PATCH 1/3 test/common: add missing quotes
From: Marc Herbert <[email protected]>
This makes shellcheck much happier and its output readable and usable.
--- test/common | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
test/common +6 -6
@@ -4,7 +4,7 @@ # Global variables # NDCTL -if [ -z $NDCTL ]; then +if [ -z "$NDCTL" ]; then if [ -f "../ndctl/ndctl" ] && [ -x "../ndctl/ndctl" ]; then export NDCTL=../ndctl/ndctl elif [ -f "./ndctl/ndctl" ] && [ -x "./ndctl/ndctl" ]; then
@@ -16,7 +16,7 @@ if [ -z $NDCTL ]; then fi # DAXCTL -if [ -z $DAXCTL ]; then +if [ -z "$DAXCTL" ]; then if [ -f "../daxctl/daxctl" ] && [ -x "../daxctl/daxctl" ]; then export DAXCTL=../daxctl/daxctl elif [ -f "./daxctl/daxctl" ] && [ -x "./daxctl/daxctl" ]; then
@@ -28,7 +28,7 @@ if [ -z $DAXCTL ]; then fi # CXL -if [ -z $CXL ]; then +if [ -z "$CXL" ]; then if [ -f "../cxl/cxl" ] && [ -x "../cxl/cxl" ]; then export CXL=../cxl/cxl elif [ -f "./cxl/cxl" ] && [ -x "./cxl/cxl" ]; then
@@ -39,7 +39,7 @@ if [ -z $CXL ]; then fi fi -if [ -z $TEST_PATH ]; then +if [ -z "$TEST_PATH" ]; then export TEST_PATH=. fi
@@ -103,7 +103,7 @@ check_min_kver() # do_skip() { - echo kernel $(uname -r): $1 + echo kernel "$(uname -r)": "$1" exit 77 }
@@ -147,7 +147,7 @@ check_dmesg() # validate no WARN or lockdep report during the run sleep 1 log=$(journalctl -r -k --since "-$((SECONDS+1))s") - grep -q "Call Trace" <<< $log && err $1 + grep -q "Call Trace" <<< "$log" && err "$1" true }
Signature · 1 line
2.50.1
Re: [ndctl PATCH 1/3] test/common: add missing quotes
Quoted 3 lines · On 7/24/25 3:00 PM, [email protected] wrote:
From: Marc Herbert <[email protected]>
This makes shellcheck much happier and its output readable and usable.
Probably should mention what exactly you are changing.
Quoted 2 lines
Signed-off-by Marc Herbert <[email protected]>
Quoted 62 lines
--- test/common | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)test/common +6
-6@@ -4,7 +4,7 @@ # Global variables # NDCTL -if [ -z $NDCTL ]; then +if [ -z "$NDCTL" ]; then if [ -f "../ndctl/ndctl" ] && [ -x "../ndctl/ndctl" ]; then export NDCTL=../ndctl/ndctl elif [ -f "./ndctl/ndctl" ] && [ -x "./ndctl/ndctl" ]; then@@ -16,7 +16,7 @@ if [ -z $NDCTL ]; then fi # DAXCTL -if [ -z $DAXCTL ]; then +if [ -z "$DAXCTL" ]; then if [ -f "../daxctl/daxctl" ] && [ -x "../daxctl/daxctl" ]; then export DAXCTL=../daxctl/daxctl elif [ -f "./daxctl/daxctl" ] && [ -x "./daxctl/daxctl" ]; then@@ -28,7 +28,7 @@ if [ -z $DAXCTL ]; then fi # CXL -if [ -z $CXL ]; then +if [ -z "$CXL" ]; then if [ -f "../cxl/cxl" ] && [ -x "../cxl/cxl" ]; then export CXL=../cxl/cxl elif [ -f "./cxl/cxl" ] && [ -x "./cxl/cxl" ]; then@@ -39,7 +39,7 @@ if [ -z $CXL ]; then fi fi -if [ -z $TEST_PATH ]; then +if [ -z "$TEST_PATH" ]; then export TEST_PATH=. fi@@ -103,7 +103,7 @@ check_min_kver() # do_skip() { - echo kernel $(uname -r): $1 + echo kernel "$(uname -r)": "$1" exit 77 }@@ -147,7 +147,7 @@ check_dmesg() # validate no WARN or lockdep report during the run sleep 1 log=$(journalctl -r -k --since "-$((SECONDS+1))s") - grep -q "Call Trace" <<< $log && err $1 + grep -q "Call Trace" <<< "$log" && err "$1" true }
Alison Schofield
maintainerRe: [ndctl PATCH 1/3] test/common: add missing quotes
In reply to [email protected]Quoted 5 lines · On Thu, Jul 24, 2025 at 10:00:44PM +0000, [email protected] wrote:
From: Marc Herbert <[email protected]>
This makes shellcheck much happier and its output readable and usable.
Signed-off-by Marc Herbert <[email protected]>
Thanks!
Applied to https://github.com/pmem/ndctl/commits/pending/
with [alison: edit commit message and log]
Quoted 65 lines
--- test/common | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)test/common +6
-6@@ -4,7 +4,7 @@ # Global variables # NDCTL -if [ -z $NDCTL ]; then +if [ -z "$NDCTL" ]; then if [ -f "../ndctl/ndctl" ] && [ -x "../ndctl/ndctl" ]; then export NDCTL=../ndctl/ndctl elif [ -f "./ndctl/ndctl" ] && [ -x "./ndctl/ndctl" ]; then@@ -16,7 +16,7 @@ if [ -z $NDCTL ]; then fi # DAXCTL -if [ -z $DAXCTL ]; then +if [ -z "$DAXCTL" ]; then if [ -f "../daxctl/daxctl" ] && [ -x "../daxctl/daxctl" ]; then export DAXCTL=../daxctl/daxctl elif [ -f "./daxctl/daxctl" ] && [ -x "./daxctl/daxctl" ]; then@@ -28,7 +28,7 @@ if [ -z $DAXCTL ]; then fi # CXL -if [ -z $CXL ]; then +if [ -z "$CXL" ]; then if [ -f "../cxl/cxl" ] && [ -x "../cxl/cxl" ]; then export CXL=../cxl/cxl elif [ -f "./cxl/cxl" ] && [ -x "./cxl/cxl" ]; then@@ -39,7 +39,7 @@ if [ -z $CXL ]; then fi fi -if [ -z $TEST_PATH ]; then +if [ -z "$TEST_PATH" ]; then export TEST_PATH=. fi@@ -103,7 +103,7 @@ check_min_kver() # do_skip() { - echo kernel $(uname -r): $1 + echo kernel "$(uname -r)": "$1" exit 77 }@@ -147,7 +147,7 @@ check_dmesg() # validate no WARN or lockdep report during the run sleep 1 log=$(journalctl -r -k --since "-$((SECONDS+1))s") - grep -q "Call Trace" <<< $log && err $1 + grep -q "Call Trace" <<< "$log" && err "$1" true }Signature · 1 line
2.50.1
PATCH 2/3 test/common: move err() function at the top
In reply to [email protected]From: Marc Herbert <[email protected]>
move err() function at the top so we can fail early. err() does not have any dependency so it can be first.
--- test/common | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
test/common +14 -11
@@ -1,6 +1,20 @@ # SPDX-License-Identifier: GPL-2.0 # Copyright (C) 2018, FUJITSU LIMITED. All rights reserved. +# err +# $1: line number which error detected +# $2: cleanup function (optional) +# + +test_basename=$(basename "$0") + +err() +{ + echo test/"$test_basename": failed at line "$1" + [ -n "$2" ] && "$2" + exit "$rc" +} + # Global variables # NDCTL
@@ -53,17 +67,6 @@ E820_BUS="e820" # Functions -# err -# $1: line number which error detected -# $2: cleanup function (optional) -# -err() -{ - echo test/$(basename $0): failed at line $1 - [ -n "$2" ] && "$2" - exit $rc -} - reset() { $NDCTL disable-region -b $NFIT_TEST_BUS0 all
Signature · 1 line
2.50.1
Re: [ndctl PATCH 2/3] test/common: move err() function at the top
Quoted 6 lines · On 7/24/25 3:00 PM, [email protected] wrote:
From: Marc Herbert <[email protected]>
move
err()function at the top so we can fail early.err()does not have any dependency so it can be first.Signed-off-by Marc Herbert <[email protected]>
Quoted 47 lines
--- test/common | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)test/common +14
-11@@ -1,6 +1,20 @@ # SPDX-License-Identifier: GPL-2.0 # Copyright (C) 2018, FUJITSU LIMITED. All rights reserved. +# err +# $1: line number which error detected +# $2: cleanup function (optional) +# + +test_basename=$(basename "$0") + +err() +{ + echo test/"$test_basename": failed at line "$1" + [ -n "$2" ] && "$2" + exit "$rc" +} + # Global variables # NDCTL@@ -53,17 +67,6 @@ E820_BUS="e820" # Functions -# err -# $1: line number which error detected -# $2: cleanup function (optional) -# -err() -{ - echo test/$(basename $0): failed at line $1 - [ -n "$2" ] && "$2" - exit $rc -} - reset() { $NDCTL disable-region -b $NFIT_TEST_BUS0 all
Alison Schofield
maintainerRe: [ndctl PATCH 2/3] test/common: move err() function at the top
In reply to [email protected]Quoted 6 lines · On Thu, Jul 24, 2025 at 10:00:45PM +0000, [email protected] wrote:
From: Marc Herbert <[email protected]>
move
err()function at the top so we can fail early.err()does not have any dependency so it can be first.Signed-off-by Marc Herbert <[email protected]>
Thanks!
Applied to https://github.com/pmem/ndctl/commits/pending/
with [alison: edit commit message and log]
Quoted 50 lines
--- test/common | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)test/common +14
-11@@ -1,6 +1,20 @@ # SPDX-License-Identifier: GPL-2.0 # Copyright (C) 2018, FUJITSU LIMITED. All rights reserved. +# err +# $1: line number which error detected +# $2: cleanup function (optional) +# + +test_basename=$(basename "$0") + +err() +{ + echo test/"$test_basename": failed at line "$1" + [ -n "$2" ] && "$2" + exit "$rc" +} + # Global variables # NDCTL@@ -53,17 +67,6 @@ E820_BUS="e820" # Functions -# err -# $1: line number which error detected -# $2: cleanup function (optional) -# -err() -{ - echo test/$(basename $0): failed at line $1 - [ -n "$2" ] && "$2" - exit $rc -} - reset() { $NDCTL disable-region -b $NFIT_TEST_BUS0 allSignature · 1 line
2.50.1
PATCH 3/3 test/common: stop relying on bash $SECONDS in check_dmesg()
In reply to [email protected]From: Marc Herbert <[email protected]>
Stop relying on the imprecise bash $SECONDS variable as a test start timestamp when scanning the logs for issues.
$SECONDS was convenient because it came "for free" and did not require any time_init(). But it was not fine-grained enough and its rounding process is not even documented. Keep using $SECONDS in log messages where it is easy to use and more user-friendly than bare timestamps, but switch the critical journalctl scan to a new, absolute NDTEST_START timestamp initialized when test/common is sourced. Use a SECONDS-based, rough sanity check in time_init() to make sure test/common is always sourced early.
--- test/common | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-)
test/common +50 -2
@@ -15,6 +15,25 @@ err() exit "$rc" } +# Initialize the NDTEST_START timestamp used to scan the logs. +# Insert an anchor/bookmark in the logs to quickly locate the start of any test. +time_init() +{ + # Refuse to run if anything lasted for too long before this point + # because that would make NDTEST_START incorrect. + test "$SECONDS" -le 1 || err 'test/common must be included first!' + + NDTEST_START=$(LC_TIME=C date '+%F %T.%3N') + + # Log anchor, especially useful when running tests back to back + printf "<5>%s@%ds: sourcing test/common: NDTEST_START=%s\n" \ + "$test_basename" "$SECONDS" "$NDTEST_START" > /dev/kmsg + + # Default value, can be overridden by the environment + : "${NDTEST_LOG_DBG:=false}" +} +time_init + # Global variables # NDCTL
@@ -147,11 +166,40 @@ json2var() # $1: line number where this is called check_dmesg() { + if "$NDTEST_LOG_DBG"; then + # Keep a record of which log lines we scanned + journalctl -q -b --since "$NDTEST_START" \ + -o short-precise > journal-"$(basename "$0")".log + fi + # After enabling with `NDTEST_LOG_DBG=true meson test`, inspect with: + # head -n 7 $(ls -1t build/journal-*.log | tac) + # journalctl --since='- 5 min' -o short-precise -g 'test/common' + # validate no WARN or lockdep report during the run - sleep 1 - log=$(journalctl -r -k --since "-$((SECONDS+1))s") + log=$(journalctl -r -k --since "$NDTEST_START") grep -q "Call Trace" <<< "$log" && err "$1" true + + # Log anchor, especially useful when running tests back to back + printf "<5>%s@%ds: test/common: check_dmesg() OK\n" "$test_basename" "$SECONDS" > /dev/kmsg + + if "$NDTEST_LOG_DBG"; then + log_stress from_check_dmesg + fi +} + +# While they should, many tests don't use check_dmesg(). So double down here. Also, this +# runs later which is better. +# Before enabling NDTEST_LOG_DBG=true, make sure no test started defining its own +# EXIT trap. +if "$NDTEST_LOG_DBG"; then + trap 'log_stress from_trap' EXIT +fi + +log_stress() +{ + printf '<3>%s@%ds: NDTEST_LOG_DBG Call Trace; trying to break the next check_dmesg() %s\n' \ + "$test_basename" "$SECONDS" "$1" > /dev/kmsg } # CXL COMMON
Signature · 1 line
2.50.1
Quoted 14 lines · On 7/24/25 3:00 PM, [email protected] wrote:
From: Marc Herbert <[email protected]>
Stop relying on the imprecise bash $SECONDS variable as a test start timestamp when scanning the logs for issues.
$SECONDS was convenient because it came "for free" and did not require any
time_init(). But it was not fine-grained enough and its rounding process is not even documented. Keep using $SECONDS in log messages where it is easy to use and more user-friendly than bare timestamps, but switch the critical journalctl scan to a new, absoluteNDTEST_STARTtimestamp initialized when test/common is sourced. Use a SECONDS-based, rough sanity check intime_init()to make sure test/common is always sourced early.
Quoted 1 line
Signed-off-by Marc Herbert <[email protected]>
Looks ok to me, but I'm not great with bash scripting or journalctl.
Quoted 77 lines
--- test/common | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-)test/common +50
-2@@ -15,6 +15,25 @@ err() exit "$rc" } +# Initialize the NDTEST_START timestamp used to scan the logs. +# Insert an anchor/bookmark in the logs to quickly locate the start of any test. +time_init() +{ + # Refuse to run if anything lasted for too long before this point + # because that would make NDTEST_START incorrect. + test "$SECONDS" -le 1 || err 'test/common must be included first!' + + NDTEST_START=$(LC_TIME=C date '+%F %T.%3N') + + # Log anchor, especially useful when running tests back to back + printf "<5>%s@%ds: sourcing test/common: NDTEST_START=%s\n" \ + "$test_basename" "$SECONDS" "$NDTEST_START" > /dev/kmsg + + # Default value, can be overridden by the environment + : "${NDTEST_LOG_DBG:=false}" +} +time_init + # Global variables # NDCTL@@ -147,11 +166,40 @@ json2var() # $1: line number where this is called check_dmesg() { + if "$NDTEST_LOG_DBG"; then + # Keep a record of which log lines we scanned + journalctl -q -b --since "$NDTEST_START" \ + -o short-precise > journal-"$(basename "$0")".log + fi + # After enabling with `NDTEST_LOG_DBG=true meson test`, inspect with: + # head -n 7 $(ls -1t build/journal-*.log | tac) + # journalctl --since='- 5 min' -o short-precise -g 'test/common' + # validate no WARN or lockdep report during the run - sleep 1 - log=$(journalctl -r -k --since "-$((SECONDS+1))s") + log=$(journalctl -r -k --since "$NDTEST_START") grep -q "Call Trace" <<< "$log" && err "$1" true + + # Log anchor, especially useful when running tests back to back + printf "<5>%s@%ds: test/common: check_dmesg() OK\n" "$test_basename" "$SECONDS" > /dev/kmsg + + if "$NDTEST_LOG_DBG"; then + log_stress from_check_dmesg + fi +} + +# While they should, many tests don't use check_dmesg(). So double down here. Also, this +# runs later which is better. +# Before enabling NDTEST_LOG_DBG=true, make sure no test started defining its own +# EXIT trap. +if "$NDTEST_LOG_DBG"; then + trap 'log_stress from_trap' EXIT +fi + +log_stress() +{ + printf '<3>%s@%ds: NDTEST_LOG_DBG Call Trace; trying to break the next check_dmesg() %s\n' \ + "$test_basename" "$SECONDS" "$1" > /dev/kmsg } # CXL COMMON
Alison Schofield
maintainerQuoted 7 lines · On 7/24/25 3:00 PM, [email protected] wrote:
Quoted 5 lines
From: Marc Herbert <[email protected]>
Stop relying on the imprecise bash $SECONDS variable as a test start timestamp when scanning the logs for issues.
Quoted 8 lines
Quoted 8 lines
$SECONDS was convenient because it came "for free" and did not require any
time_init(). But it was not fine-grained enough and its rounding process is not even documented. Keep using $SECONDS in log messages where it is easy to use and more user-friendly than bare timestamps, but switch the critical journalctl scan to a new, absoluteNDTEST_STARTtimestamp initialized when test/common is sourced. Use a SECONDS-based, rough sanity check intime_init()to make sure test/common is always sourced early.
This patch has one fix, plus 4 while-we-are-at-it's, with only one of the latter mentioned in the commit message.
Limit this patch to fixing the known problem - replacing $SECONDS with an initialized NDTEST_START for accurate log scanning. You could reasonably include the log anchor additions since those help folks who view the dmesg log in one go and tie directly to the timing fix. Then, if there is still justification for the other debug functionality after we've tightened up the timing, add them in a separate patch with proper explanation of what problem they solve.
The commit message is explaining the drawbacks of seconds, but not mentioning why that is a problem for ndctl. This previous text, captured the issue:
Quoted 4 lines
Quoted 4 lines
Using the bash variable 'SECONDS' plus 1 for searching the dmesg log sometimes led to one test picking up error messages from the previous test when run as a suite. SECONDS alone may miss some logs, but SECONDS + 1 is just as often too great.
Then state what the patch does to resolve it.
Initialize and use a timestamp with ms granularity with journalctl.
a few comments on the implementation...
Quoted 14 lines
Quoted 14 lines
Signed-off-by Marc Herbert <[email protected]>--- test/common | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-)test/common +0
-0@@ -15,6 +15,25 @@ err() exit "$rc" }
Quoted 7 lines
test/common err() · line 24
Quoted 7 lines
test/common +7
-0+# Initialize the NDTEST_START timestamp used to scan the logs. +# Insert an anchor/bookmark in the logs to quickly locate the start of any test. +time_init() +{ + # Refuse to run if anything lasted for too long before this point + # because that would make NDTEST_START incorrect. + test "$SECONDS" -le 1 || err 'test/common must be included first!'
Agree w previous reviewer comments, this is not needed.
Quoted 36 lines
Quoted 36 lines
test/common +24
-2+ + NDTEST_START=$(LC_TIME=C date '+%F %T.%3N') + + # Log anchor, especially useful when running tests back to back + printf "<5>%s@%ds: sourcing test/common: NDTEST_START=%s\n" \ + "$test_basename" "$SECONDS" "$NDTEST_START" > /dev/kmsg + + # Default value, can be overridden by the environment + : "${NDTEST_LOG_DBG:=false}" +} +time_init + # Global variables # NDCTL@@ -147,11 +166,40 @@ json2var() # $1: line number where this is called check_dmesg() { + if "$NDTEST_LOG_DBG"; then + # Keep a record of which log lines we scanned + journalctl -q -b --since "$NDTEST_START" \ + -o short-precise > journal-"$(basename "$0")".log + fi + # After enabling with `NDTEST_LOG_DBG=true meson test`, inspect with: + # head -n 7 $(ls -1t build/journal-*.log | tac) + # journalctl --since='- 5 min' -o short-precise -g 'test/common' + # validate no WARN or lockdep report during the run - sleep 1 - log=$(journalctl -r -k --since "-$((SECONDS+1))s") + log=$(journalctl -r -k --since "$NDTEST_START") grep -q "Call Trace" <<< "$log" && err "$1" true + + # Log anchor, especially useful when running tests back to back
Quoted 8 lines
test/common json2var()
Quoted 8 lines
test/common +8
-0+ printf "<5>%s@%ds: test/common: check_dmesg() OK\n" "$test_basename" "$SECONDS" > /dev/kmsg + + if "$NDTEST_LOG_DBG"; then + log_stress from_check_dmesg + fi +} + +# While they should, many tests don't use check_dmesg().
I suspect we get more value by directly going after the outliers like scripts not checking dmesg, or unit tests not wrapped in scripts. (wrapping unit tests in scripts would be a new requirement)
Although not clear what the below code has to do with tests that don't check_dmesg().
Quoted 16 lines
Quoted 15 lines
So double down here. Also, this
test/common +11
-0+# runs later which is better. +# Before enabling NDTEST_LOG_DBG=true, make sure no test started defining its own +# EXIT trap. +if "$NDTEST_LOG_DBG"; then + trap 'log_stress from_trap' EXIT +fi + +log_stress() +{ + printf '<3>%s@%ds: NDTEST_LOG_DBG Call Trace; trying to break the next check_dmesg() %s\n' \ + "$test_basename" "$SECONDS" "$1" > /dev/kmsg } # CXL COMMON
Alison Schofield
maintainerMarc - There is something off with the patchset format. The cover letter is usually Patch 0, so this would look like like:
[ndctl PATCH 0/3] test/common: stop relying on bash $SECONDS in check_dmesg() [ndctl PATCH 1/3] test/common: add missing quotes
[ndctl PATCH 2/3] test/common: move err() function at the top [ndctl PATCH 3/3] test/common: stop relying on bash $SECONDS in check_dmesg()
The cover letter commit message should be general, not same as one of it's child patches. This ship has sailed, but for next time take a closer look at what your patch sending tool is generating.
Quoted 14 lines · On Thu, Jul 24, 2025 at 10:00:43PM +0000, [email protected] wrote:
As requested, this is the broken-down, first part of
https://lore.kernel.org/linux-cxl/[email protected]/
test: fail on unexpected kernel error & warning, not just "Call Trace"I will resubmit the rest after the review of this first part is completed. The different parts are logically separate (different "features") but they are interleaved in the same function and I don't really have the time to fix one git conflict per review comment.
While it is a requirement to catch warnings and errors later, the better accuracy provided by this first part is useful alone. If you are interested in what's next, just look at the bigger patch above. You don't have to!