Skip to contentSkip to conversation

ktest: Updates to ktest for 7.1

PATCH New Steven Rostedt 1 participant 10 messages Mar 16 Ktest

lore
Share & tools
lore
#0 lore raw

cover ktest: Updates to ktest for 7.1

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-ktest.git
for-next

Head SHA1: 229517449879b3ca8ca5588593cbea6a67ba0ad2

Ricardo B. Marlière (9):
ktest: Avoid undef warning when WARNINGS_FILE is unset
ktest: Resolve LOG_FILE in test option context
ktest: Treat undefined self-reference as empty
ktest: Honor empty per-test option overrides
ktest: Run commands through list-form shell open
ktest: Stop dropping console output during power-cycle reboot
ktest: Add PRE_KTEST_DIE for PRE_KTEST failures
ktest: Run POST_KTEST hooks on failure and cancellation
ktest: Add a --dry-run mode

----

 tools/testing/ktest/ktest.pl    | 158 +++++++++++++++++++++++++++++-----------
 tools/testing/ktest/sample.conf |   6 ++
 2 files changed, 122 insertions(+), 42 deletions(-)
#1 lore raw

PATCH 1/9 ktest: Avoid undef warning when WARNINGS_FILE is unset

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

check_buildlog() probes $warnings_file with -f even when WARNINGS_FILE is not configured. Perl warns about the uninitialized value and adds noise to the test log, which can hide the output we actually care about.

Check that WARNINGS_FILE is defined before testing whether the file exists.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
Fixes 4283b169abfb ("ktest: Add make_warnings_file and process full warnings")
---
 tools/testing/ktest/ktest.pl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/ktest/ktest.pl +1 -1
index 001c4df9f7df..f48ee64c69da 100755
@@ -2508,7 +2508,7 @@ sub check_buildlog {
     my $save_no_reboot = $no_reboot;
     $no_reboot = 1;
 
-    if (-f $warnings_file) {
+    if (defined($warnings_file) && -f $warnings_file) {
 	open(IN, $warnings_file) or
 	    dodie "Error opening $warnings_file";
 
Signature · 1 line
2.51.0
#2 lore raw

PATCH 2/9 ktest: Resolve LOG_FILE in test option context

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

LOG_FILE is expanded immediately after the config file is parsed with eval_option(..., -1). That uses the default context, not the same option resolution path used for tests. If LOG_FILE depends on options that are finalized per test, it can be resolved from stale values before the first test starts.

Resolve LOG_FILE through set_test_option("LOG_FILE", 1) instead so it uses the same expansion rules as the rest of the test options.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
---
 tools/testing/ktest/ktest.pl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/ktest/ktest.pl +1 -1
index f48ee64c69da..b7a1c8c617e0 100755
@@ -4391,7 +4391,7 @@ EOF
 read_config $ktest_config;
 
 if (defined($opt{"LOG_FILE"})) {
-    $opt{"LOG_FILE"} = eval_option("LOG_FILE", $opt{"LOG_FILE"}, -1);
+    $opt{"LOG_FILE"} = set_test_option("LOG_FILE", 1);
 }
 
 # Append any configs entered in manually to the config file.
Signature · 1 line
2.51.0
#3 lore raw

PATCH 3/9 ktest: Treat undefined self-reference as empty

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

Config variables are expanded when they are assigned. A first-time append such as:

  VAR := ${VAR} foo

leaves the literal ${VAR} in the stored value because VAR has not been defined yet. Later expansions then carry the self-reference forward instead of behaving like an empty prefix.

Drop an unescaped self-reference when the variable has no current value, and trim the outer whitespace left behind. Keep escaped \${VAR} references unchanged so literal text still works.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
---
 tools/testing/ktest/ktest.pl | 8 ++++++++
 1 file changed, 8 insertions(+)
tools/testing/ktest/ktest.pl +8 -0
index b7a1c8c617e0..b8fcdabffffe 100755
@@ -910,6 +910,14 @@ sub set_variable {
     if (defined($command_tmp_vars{$lvalue})) {
 	return;
     }
+
+    # If a variable is undefined, treat an unescaped self-reference as empty.
+    if (!defined($variable{$lvalue})) {
+	$rvalue =~ s/(?<!\\)\$\{\Q$lvalue\E\}//g;
+	$rvalue =~ s/^\s+//;
+	$rvalue =~ s/\s+$//;
+    }
+
     if ($rvalue =~ /^\s*$/) {
 	delete $variable{$lvalue};
     } else {
Signature · 1 line
2.51.0
#4 lore raw

PATCH 4/9 ktest: Honor empty per-test option overrides

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

A per-test override can clear an inherited default option by assigning an empty value, but __set_test_option() still used option_defined() to decide whether a per-test key existed. That turned an empty per-test assignment back into "fall back to the default", so tests still could not clear inherited settings.

For example:

DEFAULTS
(...)
LOG_FILE = /tmp/ktest-empty-override.log
CLEAR_LOG = 1
ADD_CONFIG = /tmp/.config

TEST_START
TEST_TYPE = build
BUILD_TYPE = nobuild
ADD_CONFIG =

This would run the test with ADD_CONFIG[1] = /tmp/.config

Fix by checking whether the per-test key exists before falling back. If it does exist but is empty, treat it as unset for that test and stop the fallback chain there.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
Fixes 22c37a9ac49d ("ktest: Allow tests to undefine default options")
---
 tools/testing/ktest/ktest.pl | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/testing/ktest/ktest.pl +4 -2
index b8fcdabffffe..42bc505e14cb 100755
@@ -4191,7 +4191,8 @@ sub __set_test_option {
 
     my $option = "$name\[$i\]";
 
-    if (option_defined($option)) {
+    if (exists($opt{$option})) {
+	return undef if (!option_defined($option));
 	return $opt{$option};
     }
 
@@ -4199,7 +4200,8 @@ sub __set_test_option {
 	if ($i >= $test &&
 	    $i < $test + $repeat_tests{$test}) {
 	    $option = "$name\[$test\]";
-	    if (option_defined($option)) {
+	    if (exists($opt{$option})) {
+		return undef if (!option_defined($option));
 		return $opt{$option};
 	    }
 	}
Signature · 1 line
2.51.0
#5 lore raw

PATCH 5/9 ktest: Run commands through list-form shell open

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

run_command() currently uses string-form open():

open(CMD, "$command 2>&1 |")

That delegates parsing to the shell but also mixes the stderr redirection into the command string. Switch to list-form open() with an explicit sh -c wrapper so shell syntax errors are captured in the same output stream as command output. Otherwise, important errors can not be retrieved from the ktest LOG_FILE.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
---
 tools/testing/ktest/ktest.pl | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tools/testing/ktest/ktest.pl +4 -1
index 42bc505e14cb..9d6f50045dbd 100755
@@ -1921,7 +1921,10 @@ sub run_command {
     doprint("$command ... ");
     $start_time = time;
 
-    $pid = open(CMD, "$command 2>&1 |") or
+    $pid = open(CMD, "-|",
+		"sh", "-c",
+		'command=$1; shift; exec 2>&1; eval "$command"',
+		"sh", $command) or
 	(fail "unable to exec $command" and return 0);
 
     if (defined($opt{"LOG_FILE"})) {
Signature · 1 line
2.51.0
#6 lore raw

PATCH 6/9 ktest: Stop dropping console output during power-cycle reboot

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

The POWER_CYCLE fallback added to reboot() flushes monitor output at the wrong time. In the untimed reboot path, flushing immediately after start_monitor() can consume the first output from the new boot before monitor() begins reading it. In the timed path, flushing after POWER_CYCLE can eat the "Linux version" banner or REBOOT_SUCCESS_LINE from the new kernel.

That makes ktest miss the boot it is waiting for and can trigger an unnecessary second power cycle.

Start the monitor before POWER_CYCLE so the reference counting stays balanced, but only flush when reboot() was asked to wait for a timed reboot. Perform that flush before issuing POWER_CYCLE so it drains stale output from the old kernel instead of consuming the next boot.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
---
 tools/testing/ktest/ktest.pl | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
tools/testing/ktest/ktest.pl +6 -5
index 9d6f50045dbd..bd2e2311884c 100755
@@ -1499,12 +1499,13 @@ sub reboot {
     }
 
     if ($powercycle) {
-	run_command "$power_cycle";
-
 	start_monitor;
-	# flush out current monitor
-	# May contain the reboot success line
-	wait_for_monitor 1;
+	if (defined($time)) {
+		# Flush stale console output from the old kernel before power-cycling.
+		wait_for_monitor 1;
+	}
+
+	run_command "$power_cycle";
 
     } else {
 	# Make sure everything has been written to disk
Signature · 1 line
2.51.0
#7 lore raw

PATCH 7/9 ktest: Add PRE_KTEST_DIE for PRE_KTEST failures

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

PRE_KTEST runs before the first test, but its return status is currently ignored. A failing setup hook can leave the rest of the run executing in a partially initialized environment.

Add PRE_KTEST_DIE so PRE_KTEST can fail the run in the same way PRE_BUILD_DIE and PRE_TEST_DIE already can. Keep the default behavior unchanged when the new option is not set.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
---
 tools/testing/ktest/ktest.pl    | 8 +++++++-
 tools/testing/ktest/sample.conf | 6 ++++++
 2 files changed, 13 insertions(+), 1 deletion(-)
tools/testing/ktest/ktest.pl +7 -1
index bd2e2311884c..b018b937e028 100755
@@ -101,6 +101,7 @@ my $build_type;
 my $build_options;
 my $final_post_ktest;
 my $pre_ktest;
+my $pre_ktest_die;
 my $post_ktest;
 my $pre_test;
 my $pre_test_die;
@@ -283,6 +284,7 @@ my %option_map = (
     "BUILD_DIR"			=> \$builddir,
     "TEST_TYPE"			=> \$test_type,
     "PRE_KTEST"			=> \$pre_ktest,
+    "PRE_KTEST_DIE"		=> \$pre_ktest_die,
     "POST_KTEST"		=> \$post_ktest,
     "PRE_TEST"			=> \$pre_test,
     "PRE_TEST_DIE"		=> \$pre_test_die,
@@ -4506,7 +4508,11 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     if ($i == 1) {
 	if (defined($pre_ktest)) {
 	    doprint "\n";
-	    run_command $pre_ktest;
+	    my $ret = run_command $pre_ktest;
+	    if (!$ret && defined($pre_ktest_die) &&
+		$pre_ktest_die) {
+		dodie "failed to pre_ktest\n";
+	    }
 	}
 	if ($email_when_started) {
 	    my $name = get_test_name;
tools/testing/ktest/sample.conf +6 -0
index 9c4c449a8f3e..b6e439ef511b 100644
@@ -494,6 +494,12 @@
 #
 # default (undefined)
 #PRE_KTEST = ${SSH} ~/set_up_test
+#
+# To specify if the test should fail if PRE_KTEST fails,
+# PRE_KTEST_DIE needs to be set to 1. Otherwise the PRE_KTEST
+# result is ignored.
+# (default 0)
+#PRE_KTEST_DIE = 1
 
 # If you want to execute some command after all the tests have
 # completed, you can set this option. Note, it can be set as a
Signature · 1 line
2.51.0
#8 lore raw

PATCH 8/9 ktest: Run POST_KTEST hooks on failure and cancellation

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

PRE_KTEST can be useful for setting up the environment and POST_KTEST to tear it down, however POST_KTEST only runs on the normal end-of-run path. It is skipped when ktest exits through dodie() or cancel_test(). Final cleanup hooks are skipped.

Factor the final hook execution into run_post_ktest(), call it from the normal exit path and from the early exit paths, and guard it so the hook runs at most once.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
Fixes 921ed4c7208e ("ktest: Add PRE/POST_KTEST and TEST options")
---
 tools/testing/ktest/ktest.pl | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)
tools/testing/ktest/ktest.pl +22 -5
index b018b937e028..8962857ce4a6 100755
@@ -100,6 +100,7 @@ my $test_type;
 my $build_type;
 my $build_options;
 my $final_post_ktest;
+my $post_ktest_done = 0;
 my $pre_ktest;
 my $pre_ktest_die;
 my $post_ktest;
@@ -1586,6 +1587,24 @@ sub get_test_name() {
     return $name;
 }
 
+sub run_post_ktest {
+    my $cmd;
+
+    return if ($post_ktest_done);
+
+    if (defined($final_post_ktest)) {
+	$cmd = $final_post_ktest;
+    } elsif (defined($post_ktest)) {
+	$cmd = $post_ktest;
+    } else {
+	return;
+    }
+
+    my $cp_post_ktest = eval_kernel_version($cmd);
+    run_command $cp_post_ktest;
+    $post_ktest_done = 1;
+}
+
 sub dodie {
     # avoid recursion
     return if ($in_die);
@@ -1645,6 +1664,7 @@ sub dodie {
     if (defined($post_test)) {
 	run_command $post_test;
     }
+    run_post_ktest;
 
     die @_, "\n";
 }
@@ -4314,6 +4334,7 @@ sub cancel_test {
 	send_email("KTEST: Your [$name] test was cancelled",
 	    "Your test started at $script_start_time was cancelled: sig int");
     }
+    run_post_ktest;
     die "\nCaught Sig Int, test interrupted: $!\n"
 }
 
@@ -4679,11 +4700,7 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) {
     success $i;
 }
 
-if (defined($final_post_ktest)) {
-
-    my $cp_final_post_ktest = eval_kernel_version $final_post_ktest;
-    run_command $cp_final_post_ktest;
-}
+run_post_ktest;
 
 if ($opt{"POWEROFF_ON_SUCCESS"}) {
     halt;
Signature · 1 line
2.51.0
#9 lore raw

PATCH 9/9 ktest: Add a --dry-run mode

In reply to Steven Rostedt

From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= <[email protected]>

When working on a ktest configuration, it is often useful to inspect the final option values after includes, defaults, per-test overrides, and variable expansion have been applied, without actually starting a test run.

Add a --dry-run option that reads the configuration, prints the test preamble using resolved option values, and exits before opening LOG_FILE or executing any test logic.

This is useful for debugging ktest configurations and for scripts that need to validate the final resolved settings without triggering side effects.

Cc John Hawley <[email protected]>
Cc Andrea Righi <[email protected]>
Cc Marcos Paulo de Souza <[email protected]>
Cc Matthieu Baerts <[email protected]>
Cc Fernando Fernandez Mancera <[email protected]>
Cc Pedro Falcato <[email protected]>
---
 tools/testing/ktest/ktest.pl | 89 +++++++++++++++++++++++++-----------
 1 file changed, 63 insertions(+), 26 deletions(-)
tools/testing/ktest/ktest.pl +63 -26
index 8962857ce4a6..de99b82d16ad 100755
@@ -85,6 +85,7 @@ my %default = (
 );
 
 my $test_log_start = 0;
+my $dry_run = 0;
 
 my $ktest_config = "ktest.conf";
 my $version;
@@ -587,7 +588,7 @@ sub end_monitor;
 sub wait_for_monitor;
 
 sub _logit {
-    if (defined($opt{"LOG_FILE"})) {
+    if (defined($opt{"LOG_FILE"}) && defined(fileno(LOG))) {
 	print LOG @_;
     }
 }
@@ -1365,6 +1366,9 @@ sub read_config {
 	    print "$option\n";
 	}
 	print "Set IGNORE_UNUSED = 1 to have ktest ignore unused variables\n";
+	if ($dry_run) {
+	    return;
+	}
 	if (!read_yn "Do you want to continue?") {
 	    exit -1;
 	}
@@ -4249,6 +4253,53 @@ sub set_test_option {
     return eval_option($name, $option, $i);
 }
 
+sub print_test_preamble {
+    my ($resolved) = @_;
+
+    doprint "\n\nSTARTING AUTOMATED TESTS\n\n";
+
+    for (my $i = 0, my $repeat = 1; $i <= $opt{"NUM_TESTS"}; $i += $repeat) {
+
+	if (!$i) {
+	    doprint "DEFAULT OPTIONS:\n";
+	} else {
+	    doprint "\nTEST $i OPTIONS";
+	    if (defined($repeat_tests{$i})) {
+		$repeat = $repeat_tests{$i};
+		doprint " ITERATE $repeat";
+	    }
+	    doprint "\n";
+	}
+
+	foreach my $option (sort keys %opt) {
+	    my $value;
+
+	    if ($option =~ /\[(\d+)\]$/) {
+		next if ($i != $1);
+
+		if ($resolved) {
+		    my $name = $option;
+		    $name =~ s/\[\d+\]$//;
+		    $value = set_test_option($name, $i);
+		} else {
+		    $value = $opt{$option};
+		}
+	    } else {
+		next if ($i);
+
+		if ($resolved) {
+		    $value = set_test_option($option, 0);
+		} else {
+		    $value = $opt{$option};
+		}
+	    }
+
+	    $value = "" if (!defined($value));
+	    doprint "$option = $value\n";
+	}
+    }
+}
+
 sub find_mailer {
     my ($mailer) = @_;
 
@@ -4348,6 +4399,8 @@ ktest.pl version: $VERSION
                     Sets global BUILD_NOCLEAN to 1
                 -D TEST_TYPE[2]=build
                     Sets TEST_TYPE of test 2 to "build"
+       --dry-run
+                Print resolved test options and exit without running tests.
 
 	        It can also override all temp variables.
                  -D USE_TEMP_DIR:=1
@@ -4379,6 +4432,9 @@ while ( $#ARGV >= 0 ) {
 	} else {
 	    $command_vars[$#command_vars + 1] = $val;
 	}
+    } elsif ( $ARGV[0] eq "--dry-run" ) {
+	$dry_run = 1;
+	shift;
     } elsif ( $ARGV[0] eq "-h" ) {
 	die_usage;
     } else {
@@ -4427,6 +4483,11 @@ EOF
 }
 read_config $ktest_config;
 
+if ($dry_run) {
+    print_test_preamble 1;
+    exit 0;
+}
+
 if (defined($opt{"LOG_FILE"})) {
     $opt{"LOG_FILE"} = set_test_option("LOG_FILE", 1);
 }
@@ -4458,31 +4519,7 @@ if (defined($opt{"LOG_FILE"})) {
     LOG->autoflush(1);
 }
 
-doprint "\n\nSTARTING AUTOMATED TESTS\n\n";
-
-for (my $i = 0, my $repeat = 1; $i <= $opt{"NUM_TESTS"}; $i += $repeat) {
-
-    if (!$i) {
-	doprint "DEFAULT OPTIONS:\n";
-    } else {
-	doprint "\nTEST $i OPTIONS";
-	if (defined($repeat_tests{$i})) {
-	    $repeat = $repeat_tests{$i};
-	    doprint " ITERATE $repeat";
-	}
-	doprint "\n";
-    }
-
-    foreach my $option (sort keys %opt) {
-	if ($option =~ /\[(\d+)\]$/) {
-	    next if ($i != $1);
-	} else {
-	    next if ($i);
-	}
-
-	doprint "$option = $opt{$option}\n";
-    }
-}
+print_test_preamble 0;
 
 $SIG{INT} = qw(cancel_test);
 
Signature · 1 line
2.51.0