ktest.pl always returns 0?
Hi Steven,
I'm trying to use ktest to do build tests of the stable patch series to verify I didn't mess anything up, but I'm finding that ktest always returns 0 when finished, no matter if the build test was successful or failed.
Is this right? Is there some other way to determine if ktest fails other than greping the output log?
Oh, and any hints on kicking off a ktest process on a remote machine in a "simple" way? I'm just using ssh to copy over a script that runs there, wrapping ktest.pl up with other stuff, I didn't miss the fact that ktest itself can run remotely already, did I?
thanks,
greg k-h
Quoted 1 line · Greg KH
Hi Steven,
Note, emailing my RH account is hit or miss. If I'm traveling I don't read it, and I wont return messages until I'm back. It's best to email my [email protected] account, as I have better access to that account. I author my patches by the email of the people that pay me to write them. This isn't for your "who wrote the kernel" scripts. This is for anyone that happens to do a git log.
Quoted 5 lines
I'm trying to use ktest to do build tests of the stable patch series to verify I didn't mess anything up, but I'm finding that ktest always returns 0 when finished, no matter if the build test was successful or failed.
Hmm, I should fix that. Yeah, I agree, if it fails a test it should return something other than zero. But I think that only happens if you have DIE_ON_FAILURE = 0. As IIRC, the perl "die" command should exit the application with an error code.
But yeah, I agree, if one of the tests fail, the error code should not be zero. I'll write up a patch to fix that. Or at least add an option to make that happen.
Quoted 3 lines
Is this right? Is there some other way to determine if ktest fails other than greping the output log?
If you have DIE_ON_FAILURE = 1 (default) it should exit with non zero.
Quoted 5 lines
Oh, and any hints on kicking off a ktest process on a remote machine in a "simple" way? I'm just using ssh to copy over a script that runs there, wrapping ktest.pl up with other stuff, I didn't miss the fact that ktest itself can run remotely already, did I?
I'm a little confused by this question. Do you want a server ktest? That is, have a ktest daemon that listens for clients that sends it config files and then runs them? That would actually be a fun project ;-)
You're not running ktest on the target machine are you? The way I use it is the following:
I have a server that I ssh to and run ktest from. It does all the builds there on the server and this server has a means to monitor some target. I use ttywatch that connects to the serial of the target, in which ktest uses to read from.
Sometimes this "server" is the machine I'm logged in to. And I just run ktest directly.
Can you explain more of what you are looking for?
Thanks!
-- Steve
Quoted 9 lines · Steven Rostedt
Quoted 1 line · Greg KH
Hi Steven,
Note, emailing my RH account is hit or miss. If I'm traveling I don't read it, and I wont return messages until I'm back. It's best to email my [email protected] account, as I have better access to that account. I author my patches by the email of the people that pay me to write them. This isn't for your "who wrote the kernel" scripts. This is for anyone that happens to do a git log.
Hey, it's not my fault your employer has a crummy email system that can't handle remote access well, I just went off of the Author: line in your ktest.pl kernel commits :)
Quoted 5 lines
Quoted 4 lines
I'm trying to use ktest to do build tests of the stable patch series to verify I didn't mess anything up, but I'm finding that ktest always returns 0 when finished, no matter if the build test was successful or failed.
Quoted 8 lines
Hmm, I should fix that. Yeah, I agree, if it fails a test it should return something other than zero. But I think that only happens if you have
DIE_ON_FAILURE= 0. As IIRC, the perl "die" command should exit the application with an error code.But yeah, I agree, if one of the tests fail, the error code should not be zero. I'll write up a patch to fix that. Or at least add an option to make that happen.
That would be great.
Quoted 4 lines
Quoted 2 lines
Is this right? Is there some other way to determine if ktest fails other than greping the output log?
If you have
DIE_ON_FAILURE= 1 (default) it should exit with non zero.
It doesn't do that, test it and see (this is with what is in Linus's 3.6-rc7 tree, I didn't test linux-next if that is newer, my apologies.)
Quoted 17 lines
Quoted 4 lines
Oh, and any hints on kicking off a ktest process on a remote machine in a "simple" way? I'm just using ssh to copy over a script that runs there, wrapping ktest.pl up with other stuff, I didn't miss the fact that ktest itself can run remotely already, did I?
I'm a little confused by this question. Do you want a server ktest? That is, have a ktest daemon that listens for clients that sends it config files and then runs them? That would actually be a fun project ;-)
You're not running ktest on the target machine are you? The way I use it is the following:
I have a server that I ssh to and run ktest from. It does all the builds there on the server and this server has a means to monitor some target. I use ttywatch that connects to the serial of the target, in which ktest uses to read from.
Quoted 4 lines
Sometimes this "server" is the machine I'm logged in to. And I just run ktest directly.
Can you explain more of what you are looking for?
I want to be able to say:
- take this set of stable patches and go run a 'make allmodconfig' build on a remote machine and email me back the answer because I might not be able to keep an internet connection open for the next 5-15 minutes it might take to complete that task.
I don't do boot tests with these kernel build tests, although sometime in the future it would be nice to do that. Right now I do that testing manually, as it's pretty infrequent (once per release usually.)
So yes, a 'ktest' server would be nice. I've attached the (horrible) script below that I'm using for this so far. It seems to work well, and I can do builds on a "cloud" server as well as my local build server just fine, only thing needed to do is change the user and machine name in the script.
I know ktest doesn't handle quilt patches yet, which is why I apply them "by hand" now to a given git tree branch, if you ever do add that option, I'll gladly test it out and change my script to use whatever format it needs.
thanks,
greg k-h
#!/bin/bash
#
# Testing script to take a stable kernel patch set, build it on a remote
# machine using ktest, and email back the results. # # Copyright 2012 Greg Kroah-Hartman <[email protected]> # # Released under the GPLv2 only. # # # Some variables you might want to mess with are: # # EMAIL: who to send the email to # REMOTE_STABLE_GIT: on the remote machine, where the linux-stable git tree is located # REMOTE_WORK: on the remote machine, what temporary location we can use to create a subdirectory and do our work in # REMOTE_SERVER: the remote machine name # REMOTE_USER: the username to run the script on the remote machine # LOCAL_WORK: temporay location on the local machine to create some files before we copy them to the remote machine # LOCAL_KTEST: local location of a version of ktest.pl that you want to run remotely (usually better than the version in the stable tree under testing due to age issues.) # QUEUE_DIR: local location of the stable-queue git tree we are wanting to test
EMAIL="[email protected]"
REMOTE_STABLE_GIT="/home/gregkh/linux/stable/linux-stable/" REMOTE_WORK="/home/gregkh/tmp/" REMOTE_SERVER="build"
REMOTE_USER="gregkh"
LOCAL_WORK="/tmp/" LOCAL_KTEST="/home/gregkh/linux/gregkh/tools/testing/ktest/ktest.pl" QUEUE_DIR="/home/gregkh/linux/stable/stable-queue/"
#######################################################
# I doubt you need to touch anything below this line,
# unless you want to fix my bugs, or rewrite the scripts # to be saner (hint, feel free to do so...) #######################################################
# grab the kernel version from the command line KERNEL_VERSION="$1"
if [ "${KERNEL_VERSION}" == "" ] ; then
echo "$0 KERNEL_VERSION" exit;
fi
# create the local temporary directory to do the work in.
SCRIPT_HOME=`mktemp -d ${LOCAL_WORK}/stable_test_XXXXX`
echo "Using ${SCRIPT_HOME} for our local files"# create the remote directory
REMOTE_DIR=`ssh ${REMOTE_USER}@${REMOTE_SERVER} mktemp -d ${REMOTE_WORK}/stable_test_XXXXX`
echo "Using ${REMOTE_DIR} for the remote directory"# tar up stable patch queue for just this kernel version
cd ${QUEUE_DIR}/queue-${KERNEL_VERSION}/
tar -c . | gzip > ${SCRIPT_HOME}/stable_queue.tar.gz# create the script to run remotely
cat << __EOF__ > ${SCRIPT_HOME}/run_test.sh #!/bin/bash
# Test script for the ${KERNEL_VERSION}-stable kernel # autogenerated, do not edit by hand
#
STABLE_GIT="${REMOTE_STABLE_GIT}" KERNEL_VERSION="${KERNEL_VERSION}" PATCHES="stable_queue.tar.gz"
cd ${REMOTE_DIR}
# create the linux clone
git clone -s ${REMOTE_STABLE_GIT} linux
cd linux# checkout the branch we need
git checkout -t -b linux-${KERNEL_VERSION}.y origin/linux-${KERNEL_VERSION}.y# remove the .git directory as we don't need that space hanging around anymore #rm -rf .git/
# create a patches/ directory for the stable patches to apply
mkdir patches
cd patches
tar -zxvf ${REMOTE_DIR}/\${PATCHES}
cd ..# Apply the patch queue
QUILT_PATCHES=patches QUILT_SERIES=patches/series quilt push -aq --quiltrc
# build stuff perl ../ktest.pl ../ktest.conf
KTEST_RUN=$?
echo "KTEST_RUN = \${KTEST_RUN}"
cd ..
if [ "\${KTEST_RUN}" = "0" ]; then
# test succeeded
SUBJECT="Build of ${KERNEL_VERSION} was good"
else
SUBJECT="Build of ${KERNEL_VERSION} FAILED!"
fimutt -s "\${SUBJECT}" -a linux/log -- ${EMAIL} < output_log
# now that we are done, clean up after ourselves
rm -rf ${REMOTE_DIR}__EOF__
chmod 755 ${SCRIPT_HOME}/run_test.sh
# create the ktest.conf file
cat <<__EOF__ > ${SCRIPT_HOME}/ktest.conf
TEST_START
TEST_TYPE = buildDEFAULTSBUILD_TYPE = allmodconfigOUTPUT_DIR = \${PWD}/outputLOG_FILE = \${PWD}/log
LOCALVERSION = -testBUILD_OPTIONS = -j16
MACHINE = awsBUILD_DIR = \${PWD}__EOF__
# copy a version of ktest that we want to run
cp ${LOCAL_KTEST} ${SCRIPT_HOME}# copy the files we created to the remote machine
scp -q ${SCRIPT_HOME}/* ${REMOTE_USER}@${REMOTE_SERVER}:${REMOTE_DIR}# remove the local files as we don't need them anymore
rm -rf ${SCRIPT_HOME}# execute the script on the remote machine
#ssh ${REMOTE_USER}@${REMOTE_SERVER} "cd ${REMOTE_DIR} && nohup \"cd ${REMOTE_DIR} && ./run_test.sh & > ${REMOTE_DIR} output_log \" " #ssh ${REMOTE_USER}@${REMOTE_SERVER} "cd ${REMOTE_DIR} && nohup \"${REMOTE_DIR}/run_test.sh & > ${REMOTE_DIR}/output_log\"" #ssh ${REMOTE_USER}@${REMOTE_SERVER} "nohup ${REMOTE_DIR}/run_test.sh > ${REMOTE_DIR}/output_log &"
#ssh ${REMOTE_USER}@${REMOTE_SERVER} "${REMOTE_DIR}/run_test.sh > ${REMOTE_DIR}/output_log"
ssh -n -f ${REMOTE_USER}@${REMOTE_SERVER} "sh -c \"cd ${REMOTE_DIR}; nohup ./run_test.sh > ${REMOTE_DIR}/output_log 2>&1 &\" "
exit
Quoted 3 lines · Greg KH
Hey, it's not my fault your employer has a crummy email system that can't handle remote access well, I just went off of the Author: line in your ktest.pl kernel commits :)
Yeah, I'm not upset by it. I just want to warn people that there's times I may spend long periods of not answering that email.
Quoted 17 lines
Quoted 6 lines
Quoted 5 lines
I'm trying to use ktest to do build tests of the stable patch series to verify I didn't mess anything up, but I'm finding that ktest always returns 0 when finished, no matter if the build test was successful or failed.
Quoted 8 lines
Hmm, I should fix that. Yeah, I agree, if it fails a test it should return something other than zero. But I think that only happens if you have
DIE_ON_FAILURE= 0. As IIRC, the perl "die" command should exit the application with an error code.But yeah, I agree, if one of the tests fail, the error code should not be zero. I'll write up a patch to fix that. Or at least add an option to make that happen.
That would be great.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Is this right? Is there some other way to determine if ktest fails other than greping the output log?
If you have
DIE_ON_FAILURE= 1 (default) it should exit with non zero.It doesn't do that, test it and see (this is with what is in Linus's 3.6-rc7 tree, I didn't test linux-next if that is newer, my apologies.)
This should have been something from day one. I'll go ahead and try it out. According to the perl-doc man pages the "die" command has:
If an uncaught exception results in interpreter exit, the exit code is determined from the values of $! and $? with this pseudocode:
exit $! if $!; # errno
exit $? >> 8 if $? >> 8; # child exit status
exit 255; # last resort
I'll investigate this further.
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Oh, and any hints on kicking off a ktest process on a remote machine in a "simple" way? I'm just using ssh to copy over a script that runs there, wrapping ktest.pl up with other stuff, I didn't miss the fact that ktest itself can run remotely already, did I?
I'm a little confused by this question. Do you want a server ktest? That is, have a ktest daemon that listens for clients that sends it config files and then runs them? That would actually be a fun project ;-)
You're not running ktest on the target machine are you? The way I use it is the following:
I have a server that I ssh to and run ktest from. It does all the builds there on the server and this server has a means to monitor some target. I use ttywatch that connects to the serial of the target, in which ktest uses to read from.
Sometimes this "server" is the machine I'm logged in to. And I just run ktest directly.
Quoted 8 lines
Quoted 1 line
Can you explain more of what you are looking for?
I want to be able to say:
- take this set of stable patches and go run a 'make allmodconfig' build on a remote machine and email me back the answer because I might not be able to keep an internet connection open for the next 5-15 minutes it might take to complete that task.
I cheat and run all my ktests in screen sessions ;-)
Quoted 10 lines
I don't do boot tests with these kernel build tests, although sometime in the future it would be nice to do that. Right now I do that testing manually, as it's pretty infrequent (once per release usually.)
So yes, a 'ktest' server would be nice. I've attached the (horrible) script below that I'm using for this so far. It seems to work well, and I can do builds on a "cloud" server as well as my local build server just fine, only thing needed to do is change the user and machine name in the script.
This looks like my next "when I have time" project ;-).
Quoted 6 lines
I know ktest doesn't handle quilt patches yet, which is why I apply them "by hand" now to a given git tree branch, if you ever do add that option, I'll gladly test it out and change my script to use whatever format it needs.
Yeah, I need to make ktest work with quilt, as I'm still a fan.
But currently the ones that pay me actually are giving me things to do. Something about satisfying customers or some other crap. Thus, my "down time" is limited at the moment :-( But when things on the customer side slows down again, I'll definitely work on these changes.
Thanks for the ideas! I'm actually looking forward to working on this. But in the mean time, I will test the next time ktest fails on me to see what the result of $? is.
-- Steve
Quoted 102 lines · Steven Rostedt
Quoted 3 lines · Greg KH
Hey, it's not my fault your employer has a crummy email system that can't handle remote access well, I just went off of the Author: line in your ktest.pl kernel commits :)
Yeah, I'm not upset by it. I just want to warn people that there's times I may spend long periods of not answering that email.
Quoted 17 lines
Quoted 6 lines
Quoted 5 lines
I'm trying to use ktest to do build tests of the stable patch series to verify I didn't mess anything up, but I'm finding that ktest always returns 0 when finished, no matter if the build test was successful or failed.
Quoted 8 lines
Hmm, I should fix that. Yeah, I agree, if it fails a test it should return something other than zero. But I think that only happens if you have
DIE_ON_FAILURE= 0. As IIRC, the perl "die" command should exit the application with an error code.But yeah, I agree, if one of the tests fail, the error code should not be zero. I'll write up a patch to fix that. Or at least add an option to make that happen.
That would be great.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Is this right? Is there some other way to determine if ktest fails other than greping the output log?
If you have
DIE_ON_FAILURE= 1 (default) it should exit with non zero.It doesn't do that, test it and see (this is with what is in Linus's 3.6-rc7 tree, I didn't test linux-next if that is newer, my apologies.)
This should have been something from day one. I'll go ahead and try it out. According to the perl-doc man pages the "die" command has:
If an uncaught exception results in interpreter exit, the exit code is determined from the values of $! and $? with this pseudocode:
exit $! if $!; # errno
exit $? >> 8 if $? >> 8; # child exit status
exit 255; # last resortI'll investigate this further.
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Oh, and any hints on kicking off a ktest process on a remote machine in a "simple" way? I'm just using ssh to copy over a script that runs there, wrapping ktest.pl up with other stuff, I didn't miss the fact that ktest itself can run remotely already, did I?
I'm a little confused by this question. Do you want a server ktest? That is, have a ktest daemon that listens for clients that sends it config files and then runs them? That would actually be a fun project ;-)
You're not running ktest on the target machine are you? The way I use it is the following:
I have a server that I ssh to and run ktest from. It does all the builds there on the server and this server has a means to monitor some target. I use ttywatch that connects to the serial of the target, in which ktest uses to read from.
Sometimes this "server" is the machine I'm logged in to. And I just run ktest directly.
Quoted 8 lines
Quoted 1 line
Can you explain more of what you are looking for?
I want to be able to say:
- take this set of stable patches and go run a 'make allmodconfig' build on a remote machine and email me back the answer because I might not be able to keep an internet connection open for the next 5-15 minutes it might take to complete that task.I cheat and run all my ktests in screen sessions ;-)
Quoted 10 lines
I don't do boot tests with these kernel build tests, although sometime in the future it would be nice to do that. Right now I do that testing manually, as it's pretty infrequent (once per release usually.)
So yes, a 'ktest' server would be nice. I've attached the (horrible) script below that I'm using for this so far. It seems to work well, and I can do builds on a "cloud" server as well as my local build server just fine, only thing needed to do is change the user and machine name in the script.
This looks like my next "when I have time" project ;-).
Quoted 6 lines
I know ktest doesn't handle quilt patches yet, which is why I apply them "by hand" now to a given git tree branch, if you ever do add that option, I'll gladly test it out and change my script to use whatever format it needs.
Yeah, I need to make ktest work with quilt, as I'm still a fan.
Quoted 9 lines
But currently the ones that pay me actually are giving me things to do. Something about satisfying customers or some other crap. Thus, my "down time" is limited at the moment :-( But when things on the customer side slows down again, I'll definitely work on these changes.
Thanks for the ideas! I'm actually looking forward to working on this. But in the mean time, I will test the next time ktest fails on me to see what the result of $? is.
I just forced a build failure to see what ktest would show. This is my result:
cp /home/rostedt/work/git/configs/ixf/config-use /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS touch /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
Applying minimum configurations into /home/rostedt/work/git/nobackup/ixf/trace/.config.new
mv /home/rostedt/work/git/nobackup/ixf/trace/.config.new /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace oldnoconfig ... SUCCESS GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace -j40 ... FAILED!
CRITICAL FAILURE... failed build
See /home/rostedt/work/git/nobackup/ixf/ixf.log for more info.
failed build
[rostedt@fedora git]$ echo $?
25
So it seems that if DIE_ON_FAILURE is set, it returns non-zero error. But if it doesn't then it wont. But for that, I have this patch:
ktest.pl renamed from ktest.pl.orig +2 -1
@@ -3709,4 +3709,5 @@ doprint "\n $successes of $opt{NUM_TESTS} tests were successful\n\n"; -exit 0; +# Return number of failures (Yeah if we had 256 tests fail this breaks) +exit $opt{NUM_TESTS} - $successes
Quoted 16 lines · Steven Rostedt
I just forced a build failure to see what ktest would show. This is my result:
cp
/home/rostedt/work/git/configs/ixf/config-use/home/rostedt/work/git/nobackup/ixf/trace/.config... SUCCESS touch/home/rostedt/work/git/nobackup/ixf/trace/.config... SUCCESS
Applying minimum configurations into/home/rostedt/work/git/nobackup/ixf/trace/.config.new
mv/home/rostedt/work/git/nobackup/ixf/trace/.config.new/home/rostedt/work/git/nobackup/ixf/trace/.config... SUCCESS GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace oldnoconfig ... SUCCESS GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace -j40 ... FAILED!
CRITICAL FAILURE... failed build
See/home/rostedt/work/git/nobackup/ixf/ixf.logfor more info.
failed build
[rostedt@fedora git]$ echo $?
25
Quoted 1 line
So it seems that if
DIE_ON_FAILUREis set, it returns non-zero error.
Ugh, you are right, my mistake, it was a bug in my script that I was not checking the correct return value (for those wondering, my bash script was writing out a bash script that then got run, I forgot to escape the "$?" properly, so it was being intrepreted when written out and not when run.
So sorry for the noise, all is now working just fine.
greg "I should stick to C" k-h