Skip to contentSkip to conversation

ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

PATCH In review Steven Rostedt 3 participants 6 messages Nov 8, 2010 – Nov 18, 2010

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 ktest.pl: Easy and flexible testing script for Linux Kernel Developers (now with config bisect)

I previously posted the announcement about ktest (and shamelessly promoted it at both Kernel Summit and Linux Plumbers), but I just updated it with a new feature.

TEST_TYPE = config_bisect

(See http://marc.info/?l=linux-kernel&m=128829496215347 for original post)

With the new config_bisect, the config is tested in halves. To handle strange dependencies, the following is done.

First the "bad-config" needs to be prepared for the current working git tree. As with all other tests, a "min-config" may be specified and that will be override any config settings in the "bad-config". Since this may produce new settings to be set (or removed) some preparation needs to be done. This is done automatically with ktest:

1) overrides all configs with what is in the MIN_CONFIG or ADD_CONFIG settings.

2) Runs the result through a make oldconfig to see if what may have been added or removed (it reports these).

3) creates what is about to be tested and again runs the make oldconfig to see if anything else might have been removed by dependencies.

All of the above changes are reported by the tool.

Finally, it gets into the bisect. It starts with a list of configs to test. It cuts it in half, and then creates a .config file, and runs that through oldconfig. If none of the configs in the list are enabled (because they all depend on configs in the other half) then the second half of the list is tried. If there's still no configs set, there must be a circular dependency and the test fails here (but this should never happen).

It compiles, and performs the type of test specified by the CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all the configs that were enabled in the .config are removed from the configs to test and added to the good configs to always enable. This allows for configs in the other half to be enabled if they depended on these configs.

If the test fails, then we have a new subset of configs that is known to cause the failure. Any config that was not set in the .config for this run is removed from the test and will not be enabled again.

This iterates until we have one config left and that config will be reported.

Because of dependencies and selects, the config reported may not be the only config to cause the error. If you find that you can still produce an error after disabling the config it reports, then simply disable it and run the bisect again. Most likely it will turn up the problem. I may automate this part in the future.

Anyway, you can download the latest ktest from:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git

I need to write up a man page and perhaps more documentation. The sample.conf that comes in that repo has loads of information on how to do the tests.

Have fun!

-- Steve

#1 lore raw
Quoted 6 lines · Steven Rostedt

It compiles, and performs the type of test specified by the CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all the configs that were enabled in the .config are removed from the configs to test and added to the good configs to always enable. This allows for configs in the other half to be enabled if they depended on these configs.

Haven't had time to play, but I was thinking about this when bisecting an i915 bug last week. In that case none of the kernels failed to boot, they just had busted modelines. Do you have support in this tool for testing some condition on the system after it boots? I would guess I probably would have needed something more heavyweight (that could run a script to look for busted modes on the target after booting).

Jon.

#2 lore raw
Quoted 9 lines · Jon Masters
Quoted 6 lines · Steven Rostedt

It compiles, and performs the type of test specified by the CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all the configs that were enabled in the .config are removed from the configs to test and added to the good configs to always enable. This allows for configs in the other half to be enabled if they depended on these configs.

Quoted 6 lines

Haven't had time to play, but I was thinking about this when bisecting an i915 bug last week. In that case none of the kernels failed to boot, they just had busted modelines. Do you have support in this tool for testing some condition on the system after it boots? I would guess I probably would have needed something more heavyweight (that could run a script to look for busted modes on the target after booting).

If you set:

BISECT_TYPE = test
TEST = ssh user@host test_script

Where the test_script resides on the test target, and will exit with 0 on success and non-0 otherwise, it should work.

I could also change the code to give you access to the bootlog (actually you already have it, but it's in the temp directory).

-- Steve

#3 lore raw
In reply to Steven Rostedt

Question:

Has anyone tried this out? Does anyone feel that I should get this ready to be included in 2.6.38 tools directory?

Just want to know if I should go ahead and prepare it, or if it will just be a waste of my time.

Thanks,

-- Steve

Quoted 71 lines · Steven Rostedt

I previously posted the announcement about ktest (and shamelessly promoted it at both Kernel Summit and Linux Plumbers), but I just updated it with a new feature.

TEST_TYPE = config_bisect

(See http://marc.info/?l=linux-kernel&m=128829496215347 for original post)

With the new config_bisect, the config is tested in halves. To handle strange dependencies, the following is done.

First the "bad-config" needs to be prepared for the current working git tree. As with all other tests, a "min-config" may be specified and that will be override any config settings in the "bad-config". Since this may produce new settings to be set (or removed) some preparation needs to be done. This is done automatically with ktest:

1) overrides all configs with what is in the MIN_CONFIG or ADD_CONFIG settings.

2) Runs the result through a make oldconfig to see if what may have been added or removed (it reports these).

3) creates what is about to be tested and again runs the make oldconfig to see if anything else might have been removed by dependencies.

All of the above changes are reported by the tool.

Finally, it gets into the bisect. It starts with a list of configs to test. It cuts it in half, and then creates a .config file, and runs that through oldconfig. If none of the configs in the list are enabled (because they all depend on configs in the other half) then the second half of the list is tried. If there's still no configs set, there must be a circular dependency and the test fails here (but this should never happen).

It compiles, and performs the type of test specified by the CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all the configs that were enabled in the .config are removed from the configs to test and added to the good configs to always enable. This allows for configs in the other half to be enabled if they depended on these configs.

If the test fails, then we have a new subset of configs that is known to cause the failure. Any config that was not set in the .config for this run is removed from the test and will not be enabled again.

This iterates until we have one config left and that config will be reported.

Because of dependencies and selects, the config reported may not be the only config to cause the error. If you find that you can still produce an error after disabling the config it reports, then simply disable it and run the bisect again. Most likely it will turn up the problem. I may automate this part in the future.

Anyway, you can download the latest ktest from:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git

I need to write up a man page and perhaps more documentation. The sample.conf that comes in that repo has loads of information on how to do the tests.

Have fun!

-- Steve

#4 lore raw
Quoted 7 lines · Steven Rostedt

Question:

Has anyone tried this out? Does anyone feel that I should get this ready to be included in 2.6.38 tools directory?

Just want to know if I should go ahead and prepare it, or if it will just be a waste of my time.

I had done some testing only on initial internal version but did not get a chance to do more testing later.

But I believe it is worth to put effort to keep this script going. I am especially interested in part where I can just automate compilation and testing of a patch on a remote machine.

Thanks
Vivek

Quoted 79 lines

Thanks,

-- Steve

Quoted 71 lines · Steven Rostedt

I previously posted the announcement about ktest (and shamelessly promoted it at both Kernel Summit and Linux Plumbers), but I just updated it with a new feature.

TEST_TYPE = config_bisect

(See http://marc.info/?l=linux-kernel&m=128829496215347 for original post)

With the new config_bisect, the config is tested in halves. To handle strange dependencies, the following is done.

First the "bad-config" needs to be prepared for the current working git tree. As with all other tests, a "min-config" may be specified and that will be override any config settings in the "bad-config". Since this may produce new settings to be set (or removed) some preparation needs to be done. This is done automatically with ktest:

1) overrides all configs with what is in the MIN_CONFIG or ADD_CONFIG settings.

2) Runs the result through a make oldconfig to see if what may have been added or removed (it reports these).

3) creates what is about to be tested and again runs the make oldconfig to see if anything else might have been removed by dependencies.

All of the above changes are reported by the tool.

Finally, it gets into the bisect. It starts with a list of configs to test. It cuts it in half, and then creates a .config file, and runs that through oldconfig. If none of the configs in the list are enabled (because they all depend on configs in the other half) then the second half of the list is tried. If there's still no configs set, there must be a circular dependency and the test fails here (but this should never happen).

It compiles, and performs the type of test specified by the CONFIG_BISECT_TYPE option (build, boot or test). If it passes, then all the configs that were enabled in the .config are removed from the configs to test and added to the good configs to always enable. This allows for configs in the other half to be enabled if they depended on these configs.

If the test fails, then we have a new subset of configs that is known to cause the failure. Any config that was not set in the .config for this run is removed from the test and will not be enabled again.

This iterates until we have one config left and that config will be reported.

Because of dependencies and selects, the config reported may not be the only config to cause the error. If you find that you can still produce an error after disabling the config it reports, then simply disable it and run the bisect again. Most likely it will turn up the problem. I may automate this part in the future.

Anyway, you can download the latest ktest from:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git

I need to write up a man page and perhaps more documentation. The sample.conf that comes in that repo has loads of information on how to do the tests.

Have fun!

-- Steve

#5 lore raw

[ Finally remembered to remove that annoying [email protected]
from the Cc list ]

Quoted 9 lines · Vivek Goyal
Quoted 7 lines · Steven Rostedt

Question:

Has anyone tried this out? Does anyone feel that I should get this ready to be included in 2.6.38 tools directory?

Just want to know if I should go ahead and prepare it, or if it will just be a waste of my time.

Quoted 6 lines

I had done some testing only on initial internal version but did not get a chance to do more testing later.

But I believe it is worth to put effort to keep this script going. I am especially interested in part where I can just automate compilation and testing of a patch on a remote machine.

Thanks for the reply! I'll start setting up a git tree that places all the changes into the tools directory.

-- Steve