ktest.pl: Fix ssh hanging and reseting of console
Subsystems (2)
Steven Rostedt
maintainercover ktest.pl: Fix ssh hanging and reseting of console
I've noticed that my ssh sessions would hang during test runs, which is really frustrating when you kick off a 13 hour test before going to bed, and the second test (1 hour into it) hangs, and you need to kick it off again in the morning (wasting all that time over night).
I finally figured out the cause. There is a disconnect between the run_command that executes the test, and the "wait_for_input" that monitors the test. The wait_for_input has a default of 2 minute timeout if it doesn't see any output it returns. The run_command takes the empty string from wait_for_input as the test is finished, and then stops monitoring it, and calls waitpid() waiting for the test to exit.
The problem is that if the test has a lot of output, it will continue writing into the pipe that was suppose to go to the monitor, which has now exited the loop. When the pipe fills up, it will not finish. When the test is over, it just hangs waiting for the pipe to flush (which never happens).
To fix this, change the run_command to by default have an infinite run (which can be overridden by the new RUN_TIMEOUT option), and make the wait_for_input also wait indefinitely in this case. It now Now the tests will have its content continuously read and will exit normally.
While debugging this, I also found out why you can lose stdout on the terminal sometimes. Especially if you hit Ctrl^C while the monitor is running. It was due to missing "end_monitor" which gives back the tty to the terminal. The first two patches fix that.
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-ktest.git
devel
Head SHA1: aa9aba9382884554fe6a303744884866d137422d
Steven Rostedt (3):
ktest.pl: Fix missing "end_monitor" when machine check fails ktest.pl: Give back console on Ctrt^C on monitor ktest.pl: Add RUN_TIMEOUT option with default unlimited
----
tools/testing/ktest/ktest.pl | 26 +++++++++++++++++++++----- tools/testing/ktest/sample.conf | 5 +++++ 2 files changed, 26 insertions(+), 5 deletions(-)
Steven Rostedt
maintainerPATCH 1/3 ktest.pl: Fix missing "end_monitor" when machine check fails
From: Steven Rostedt <[email protected]>
In the "reboot" command, it does a check of the machine to see if it is still alive with a simple "ssh echo" command. If it fails, it will assume that a normal "ssh reboot" is not possible and force a power cycle.
In this case, the "start_monitor" is executed, but the "end_monitor" is not, and this causes the screen will not be given back to the console. That is, after the test, a "reset" command needs to be performed, as "echo" is turned off.
--- tools/testing/ktest/ktest.pl | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/ktest/ktest.pl +2 -1
@@ -1499,7 +1499,8 @@ sub reboot { # Still need to wait for the reboot to finish wait_for_monitor($time, $reboot_success_line); - + } + if ($powercycle || $time) { end_monitor; } }
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 2/3 ktest.pl: Give back console on Ctrt^C on monitor
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
When monitoring the console output, the stdout is being redirected to do so. If Ctrl^C is hit during this mode, the stdout is not back to the console, the user does not see anything they type (no echo).
Add "end_monitor" to the SIGINT interrupt handler to give back the console on Ctrl^C.
--- tools/testing/ktest/ktest.pl | 3 +++ 1 file changed, 3 insertions(+)
tools/testing/ktest/ktest.pl +3 -0
@@ -4205,6 +4205,9 @@ sub send_email { } sub cancel_test { + if ($monitor_cnt) { + end_monitor; + } if ($email_when_canceled) { my $name = get_test_name; send_email("KTEST: Your [$name] test was cancelled",
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 3/3 ktest.pl: Add RUN_TIMEOUT option with default unlimited
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
There is a disconnect between the run_command function and the wait_for_input. The wait_for_input has a default timeout of 2 minutes. But if that happens, the run_command loop will exit out to the waitpid() of the executing command. This fails in that it no longer monitors the command, and also, the ssh to the test box can hang when its finished, as it's waiting for the pipe it's writing to to flush, but the loop that reads that pipe has already exited, leaving the command stuck, and the test hangs.
Instead, make the default "wait_for_input" of the run_command infinite, and allow the user to override it if they want with a default timeout option "RUN_TIMEOUT".
But this fixes the hang that happens when the pipe is full and the ssh session never exits.
--- tools/testing/ktest/ktest.pl | 20 ++++++++++++++++---- tools/testing/ktest/sample.conf | 5 +++++ 2 files changed, 21 insertions(+), 4 deletions(-)
tools/testing/ktest/ktest.pl +16 -4
@@ -178,6 +178,7 @@ my $store_failures; my $store_successes; my $test_name; my $timeout; +my $run_timeout; my $connect_timeout; my $config_bisect_exec; my $booted_timeout;
@@ -340,6 +341,7 @@ my %option_map = ( "STORE_SUCCESSES" => \$store_successes, "TEST_NAME" => \$test_name, "TIMEOUT" => \$timeout, + "RUN_TIMEOUT" => \$run_timeout, "CONNECT_TIMEOUT" => \$connect_timeout, "CONFIG_BISECT_EXEC" => \$config_bisect_exec, "BOOTED_TIMEOUT" => \$booted_timeout,
@@ -1862,6 +1864,14 @@ sub run_command { $command =~ s/\$SSH_USER/$ssh_user/g; $command =~ s/\$MACHINE/$machine/g; + if (!defined($timeout)) { + $timeout = $run_timeout; + } + + if (!defined($timeout)) { + $timeout = -1; # tell wait_for_input to wait indefinitely + } + doprint("$command ... "); $start_time = time;
@@ -1888,13 +1898,10 @@ sub run_command { while (1) { my $fp = \*CMD; - if (defined($timeout)) { - doprint "timeout = $timeout\n"; - } my $line = wait_for_input($fp, $timeout); if (!defined($line)) { my $now = time; - if (defined($timeout) && (($now - $start_time) >= $timeout)) { + if ($timeout >= 0 && (($now - $start_time) >= $timeout)) { doprint "Hit timeout of $timeout, killing process\n"; $hit_timeout = 1; kill 9, $pid;
@@ -2066,6 +2073,11 @@ sub wait_for_input { $time = $timeout; } + if ($time < 0) { + # Negative number means wait indefinitely + undef $time; + } + $rin = ''; vec($rin, fileno($fp), 1) = 1; vec($rin, fileno(\*STDIN), 1) = 1;
tools/testing/ktest/sample.conf +5 -0
@@ -817,6 +817,11 @@ # is issued instead of a reboot. # CONNECT_TIMEOUT = 25 +# The timeout in seconds for how long to wait for any running command +# to timeout. If not defined, it will let it go indefinitely. +# (default undefined) +#RUN_TIMEOUT = 600 + # In between tests, a reboot of the box may occur, and this # is the time to wait for the console after it stops producing # output. Some machines may not produce a large lag on reboot
Signature · 1 line
2.39.0
Quoted 25 lines · On Wed, 18 Jan 2023 16:54:35 -0500 Steven Rostedt <[email protected]> wrote:
I've noticed that my ssh sessions would hang during test runs, which is really frustrating when you kick off a 13 hour test before going to bed, and the second test (1 hour into it) hangs, and you need to kick it off again in the morning (wasting all that time over night).
I finally figured out the cause. There is a disconnect between the
run_commandthat executes the test, and the "wait_for_input" that monitors the test. Thewait_for_inputhas a default of 2 minute timeout if it doesn't see any output it returns. Therun_commandtakes the empty string fromwait_for_inputas the test is finished, and then stops monitoring it, and callswaitpid()waiting for the test to exit.The problem is that if the test has a lot of output, it will continue writing into the pipe that was suppose to go to the monitor, which has now exited the loop. When the pipe fills up, it will not finish. When the test is over, it just hangs waiting for the pipe to flush (which never happens).
To fix this, change the
run_commandto by default have an infinite run (which can be overridden by the newRUN_TIMEOUToption), and make thewait_for_inputalso wait indefinitely in this case. It now Now the tests will have its content continuously read and will exit normally.
Quoted 6 lines
While debugging this, I also found out why you can lose stdout on the terminal sometimes. Especially if you hit Ctrl^C while the monitor is running. It was due to missing "
end_monitor" which gives back the tty to the terminal. The first two patches fix that.
Thanks for updating. I ran the test and confirmed that the terminal setting is recovered :)
BTW, I found another issue that if I didn't set up sendmail, it doesn't recover stty. Let me send a fix.
Quoted 3 lines
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-ktest.git
devel
Also, this devel branch seems not pushed yet.
Thank you,
Quoted 13 lines
Head SHA1:
aa9aba9382884554fe6a303744884866d137422dSteven Rostedt (3):
ktest.pl: Fix missing "end_monitor" when machine check fails ktest.pl: Give back console on Ctrt^C on monitor ktest.pl: AddRUN_TIMEOUToption with default unlimited----
tools/testing/ktest/ktest.pl | 26 +++++++++++++++++++++----- tools/testing/ktest/sample.conf | 5 +++++ 2 files changed, 26 insertions(+), 5 deletions(-)
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>