Testing
One thing we typically cover at Kernel Summit is some of the activity that's going on around testing upstream. I think it'd be useful to have some more of those discussions, both in terms of making people aware of what's available and in terms of helping the people doing testing figure out what would be useful. A lot of this is probably well suited to a workshop session between the interested people but I do think some element in the core day beyond just a readout will be useful.
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at. Should we be more generally slimming defconfigs and moving things into fragments?
and there's always the the perennial ones about what people would like to see testing for.
Quoted 4 lines · Mark Brown
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at.
This will gravitate everyone to running the same config which is the opposite of what you want.
Quoted 9 lines · Alexey Dobriyan
Quoted 4 lines · Mark Brown
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at.
This will gravitate everyone to running the same config which is the opposite of what you want.
Perhaps, perhaps not - it's not an unequivocal thing either way. The more barriers there are to enabling things the more likely it is that people just won't bother in the first place (or that they'll run into somme problem and give up before they get things working) and it's not clear that having to figure these things out is always a good use of people's time.
Quoted 11 lines · On Tue, 7 Jul 2015 14:14:11 +0100 Mark Brown <[email protected]> wrote:
Quoted 9 lines · Alexey Dobriyan
Quoted 4 lines · Mark Brown
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at.
This will gravitate everyone to running the same config which is the opposite of what you want.
Quoted 6 lines
Perhaps, perhaps not - it's not an unequivocal thing either way. The more barriers there are to enabling things the more likely it is that people just won't bother in the first place (or that they'll run into somme problem and give up before they get things working) and it's not clear that having to figure these things out is always a good use of people's time.
The testing/selftests tests should have three results: PASS, FAIL, UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel configuration doesn't have the needed features configured. For example, if you run the ftrace selftests without function tracing enabled, all the tests that test the function tracer return UNSUPPORTED.
Perhaps we should have a central location that each test needs to add the required configuration for it to be properly tested. Then if users want to test various subsystems, they would look in this location for the proper configs (be it a directory that has files of the tests they represent, and contain the configs needed). Then there should be no real barrier for people to run these tests.
Of course if the test requires certain hardware, or a file system, then that should be properly documented.
-- Steve
Quoted 34 lines · Steven Rostedt
Quoted 11 lines · On Tue, 7 Jul 2015 14:14:11 +0100 Mark Brown <[email protected]> wrote:
Quoted 9 lines · Alexey Dobriyan
Quoted 4 lines · Mark Brown
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at.
This will gravitate everyone to running the same config which is the opposite of what you want.
Quoted 6 lines
Perhaps, perhaps not - it's not an unequivocal thing either way. The more barriers there are to enabling things the more likely it is that people just won't bother in the first place (or that they'll run into somme problem and give up before they get things working) and it's not clear that having to figure these things out is always a good use of people's time.
The testing/selftests tests should have three results: PASS, FAIL, UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel configuration doesn't have the needed features configured. For example, if you run the ftrace selftests without function tracing enabled, all the tests that test the function tracer return UNSUPPORTED.
Perhaps we should have a central location that each test needs to add the required configuration for it to be properly tested. Then if users want to test various subsystems, they would look in this location for the proper configs (be it a directory that has files of the tests they represent, and contain the configs needed). Then there should be no real barrier for people to run these tests.
Quoted 2 lines
Of course if the test requires certain hardware, or a file system, then that should be properly documented.
There's also sysctl settings, and we also want privilege level as part of that configuration. (i.e. some tests should run as root, some show not, etc). It'd be nice to have the tests report these in some machine-readable form so that a test harness could check, prepare the environment, and then run the tests.
-Kees
Signature · 2 lines
Kees Cook Chrome OS Security
Quoted 6 lines · Kees Cook
There's also sysctl settings, and we also want privilege level as part of that configuration. (i.e. some tests should run as root, some show not, etc). It'd be nice to have the tests report these in some machine-readable form so that a test harness could check, prepare the environment, and then run the tests.
There was a fair amount of discussion about machine-readable output at the last KS. I don't know whether it ever got implemented.
--Andy
Quoted 2 lines · Andy Lutomirski
There was a fair amount of discussion about machine-readable output at the last KS. I don't know whether it ever got implemented.
Yes, but that's very much at the most basic stage of reporting pass and fail - there's probably room for looking at if we're ready to extend that.
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · Mark Brown
Perhaps, perhaps not - it's not an unequivocal thing either way. The more barriers there are to enabling things the more likely it is that people just won't bother in the first place (or that they'll run into somme problem and give up before they get things working) and it's not clear that having to figure these things out is always a good use of people's time.
Quoted 5 lines
The testing/selftests tests should have three results: PASS, FAIL, UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel configuration doesn't have the needed features configured. For example, if you run the ftrace selftests without function tracing enabled, all the tests that test the function tracer return UNSUPPORTED.
That's roughly what they're supposed to be doing now (I'd need to go check exactly what happens in the unsupported case).
Quoted 6 lines
Perhaps we should have a central location that each test needs to add the required configuration for it to be properly tested. Then if users want to test various subsystems, they would look in this location for the proper configs (be it a directory that has files of the tests they represent, and contain the configs needed). Then there should be no real barrier for people to run these tests.
Right, this is what I'm suggesting roughly - make the configurations required to run tests easier to pick up.
Quoted 21 lines · Steven Rostedt
Quoted 11 lines · On Tue, 7 Jul 2015 14:14:11 +0100 Mark Brown <[email protected]> wrote:
Quoted 9 lines · Alexey Dobriyan
Quoted 4 lines · Mark Brown
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at.
This will gravitate everyone to running the same config which is the opposite of what you want.
Quoted 6 lines
Perhaps, perhaps not - it's not an unequivocal thing either way. The more barriers there are to enabling things the more likely it is that people just won't bother in the first place (or that they'll run into somme problem and give up before they get things working) and it's not clear that having to figure these things out is always a good use of people's time.
Quoted 5 lines
The testing/selftests tests should have three results: PASS, FAIL, UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel configuration doesn't have the needed features configured. For example, if you run the ftrace selftests without function tracing enabled, all the tests that test the function tracer return UNSUPPORTED.
This may be an off-topic, but I'd like to ask the selftest for tools. Currently tools/testing/selftests tests the kernel itself, but there are many tools under tools/, like perf too.
Those are not configured by the kconfig, but selftests are also needed for tools. I have a runtests script which is just a bit modified ftracetest for perf-probe. I'd like to integrate it to selftests but I'm not sure that is a scope of kselftests.
Quoted 6 lines
Perhaps we should have a central location that each test needs to add the required configuration for it to be properly tested. Then if users want to test various subsystems, they would look in this location for the proper configs (be it a directory that has files of the tests they represent, and contain the configs needed). Then there should be no real barrier for people to run these tests.
/proc/kconfig[.gz]? I think we can add a list of required kconfigs for each testcase and indicate it. Moreover, we can include it as a part of kconfig and introduce CONFIG_KSELFTEST to enable those configs :)
Thank you,
Quoted 10 lines
Of course if the test requires certain hardware, or a file system, then that should be properly documented.
-- Steve
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Signature · 5 lines
Masami HIRAMATSU Linux Technology Research Center, System Productivity Research Dept. Center for Technology Innovation - Systems Engineering Hitachi, Ltd., Research & Development Group E-mail: [email protected]
Quoted 6 lines · On Thu, 09 Jul 2015 19:24:55 +0900 Masami Hiramatsu…
/proc/kconfig[.gz]? I think we can add a list of required kconfigs for each testcase and indicate it. Moreover, we can include it as a part of kconfig and introduce
CONFIG_KSELFTESTto enable those configs :)
I like the idea of having each test a separate config. We could have a config called "ALL TESTS" and selects all of them, for those that want to test all tests. But for each test, it would select the needed config options to test everything for that specific test.
-- Steve
Quoted 8 lines · Masami Hiramatsu
This may be an off-topic, but I'd like to ask the selftest for tools. Currently
tools/testing/selfteststests the kernel itself, but there are many tools undertools/, like perf too.Those are not configured by the kconfig, but selftests are also needed for tools. I have a runtests script which is just a bit modified ftracetest for perf-probe. I'd like to integrate it to selftests but I'm not sure that is a scope of kselftests.
This confusion is partially created by peculiar place where people who wrote testsuite put it.
Gentlemen,
testsuite should be first class citizen in toplevel test/ directory, command to run it should be "make test" not "make kselftest". Only placing it in very visible place and using names which are intuitive and familiar from userspace (git's t/ directory, glibc "make test") will give hope that other developers will notice it and start using and improving it. Excuse me, but tools/testing/selftests is hopeless.
Quoted 11 lines
Quoted 6 lines
Perhaps we should have a central location that each test needs to add the required configuration for it to be properly tested. Then if users want to test various subsystems, they would look in this location for the proper configs (be it a directory that has files of the tests they represent, and contain the configs needed). Then there should be no real barrier for people to run these tests.
/proc/kconfig[.gz]? I think we can add a list of required kconfigs for each testcase and indicate it. Moreover, we can include it as a part of kconfig and introduce
CONFIG_KSELFTESTto enable those configs :)
I think primary use case is this:
* user builds and reboots into kernel with his custom config,
* user runs "make test" from fresh build directory,
* test harness runs everything runnable and maybe reports necessary config options to run more
/proc/kconfig.gz should be kept strictly for runtime config.
Quoted 15 lines · Alexey Dobriyan
Quoted 8 lines · Masami Hiramatsu
This may be an off-topic, but I'd like to ask the selftest for tools. Currently
tools/testing/selfteststests the kernel itself, but there are many tools undertools/, like perf too.Those are not configured by the kconfig, but selftests are also needed for tools. I have a runtests script which is just a bit modified ftracetest for perf-probe. I'd like to integrate it to selftests but I'm not sure that is a scope of kselftests.
This confusion is partially created by peculiar place where people who wrote testsuite put it.
Quoted 7 lines
Gentlemen,
testsuite should be first class citizen in toplevel test/ directory, command to run it should be "make test" not "make kselftest". Only placing it in very visible place and using names which are intuitive and familiar from userspace (git's t/ directory, glibc "make test") will give hope that other developers will notice it and start using and improving it. Excuse me, buttools/testing/selftestsis hopeless.
selftests are intended for kernel developers primarily. If developers and users don't want to figure out what are the ways to test, then it doesn't matter what the option is named. I would like to hear some concrete data on why naming test would make it lot more usable.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Perhaps we should have a central location that each test needs to add the required configuration for it to be properly tested. Then if users want to test various subsystems, they would look in this location for the proper configs (be it a directory that has files of the tests they represent, and contain the configs needed). Then there should be no real barrier for people to run these tests.
Quoted 4 lines
Quoted 4 lines
/proc/kconfig[.gz]? I think we can add a list of required kconfigs for each testcase and indicate it. Moreover, we can include it as a part of kconfig and introduce
CONFIG_KSELFTESTto enable those configs :)
I don't believe adding yet another kernel config option especially for KSELFTEST is a good idea.
Quoted 9 lines
I think primary use case is this:
* user builds and reboots into kernel with his custom config,
* user runs "make test" from fresh build directory,
* test harness runs everything runnable and maybe reports necessary config options to run more
/proc/kconfig.gzshould be kept strictly for runtime config.
We do have ktest for that. Maybe ktest could include kselftest run in its default boot test.
thanks,
-- Shuah
Signature · 5 lines
Shuah Khan Sr. Linux Kernel Developer Open Source Innovation Group Samsung Research America (Silicon Valley) [email protected] | (970) 217-8978
Quoted 24 lines · Shuah Khan
Quoted 15 lines · Alexey Dobriyan
Quoted 8 lines · Masami Hiramatsu
This may be an off-topic, but I'd like to ask the selftest for tools. Currently
tools/testing/selfteststests the kernel itself, but there are many tools undertools/, like perf too.Those are not configured by the kconfig, but selftests are also needed for tools. I have a runtests script which is just a bit modified ftracetest for perf-probe. I'd like to integrate it to selftests but I'm not sure that is a scope of kselftests.
This confusion is partially created by peculiar place where people who wrote testsuite put it.
Quoted 7 lines
Gentlemen,
testsuite should be first class citizen in toplevel test/ directory, command to run it should be "make test" not "make kselftest". Only placing it in very visible place and using names which are intuitive and familiar from userspace (git's t/ directory, glibc "make test") will give hope that other developers will notice it and start using and improving it. Excuse me, buttools/testing/selftestsis hopeless.
Quoted 4 lines
selftests are intended for kernel developers primarily. If developers and users don't want to figure out what are the ways to test, then it doesn't matter what the option is named. I would like to hear some concrete data on why naming test would make it lot more usable.
Not usable, but visible. Top level implies "important", 3 directories deep implies some obscure driver you didn't even hear about.
And it's easier to type.
git gets it right:
test: all
$(MAKE) -C t/ all
AlexeyQuoted 7 lines · On Fri, 10 Jul 2015 08:02:14 -0600 Shuah Khan <[email protected]> wrote:
Quoted 7 lines
I think primary use case is this:
* user builds and reboots into kernel with his custom config,
* user runs "make test" from fresh build directory,
* test harness runs everything runnable and maybe reports necessary config options to run more
Quoted 5 lines
Quoted 2 lines
/proc/kconfig.gzshould be kept strictly for runtime config.We do have ktest for that. Maybe ktest could include kselftest run in its default boot test.
I could have an option to have ktest run the kselftests, but the user would still need to be able to update configs to tell ktest where the tests are. Ktest doesn't assume to be run from the repo (I never do that).
-- Steve
Quoted 13 lines · Steven Rostedt
Quoted 10 lines · On Fri, 10 Jul 2015 08:02:14 -0600 Shuah Khan <[email protected]> wrote:
Quoted 9 lines
I think primary use case is this:
* user builds and reboots into kernel with his custom config,
* user runs "make test" from fresh build directory,
* test harness runs everything runnable and maybe reports necessary config options to run more
/proc/kconfig.gzshould be kept strictly for runtime config.
Quoted 8 lines
Quoted 2 lines
We do have ktest for that. Maybe ktest could include kselftest run in its default boot test.
I could have an option to have ktest run the kselftests, but the user would still need to be able to update configs to tell ktest where the tests are. Ktest doesn't assume to be run from the repo (I never do that).
Right. Thanks for clarifying it. We have a small number of tests that depend on a specific kernel config option. Those tests are supposed to exit gracefully and let the rest of the tests run when the config option they depend on, isn't enabled. User needs to make a conscious choice on which tests they want to run. In other words, it is users responsibility to ensure that the kernel is built correctly to run a desired test. Just like ktest, the responsibility to pick the right config falls outside the kselftest framework. Kselftest run will test the parts of the kernel it can test based on the current configuration.
thanks,
-- Shuah
Signature · 5 lines
Shuah Khan Sr. Linux Kernel Developer Open Source Innovation Group Samsung Research America (Silicon Valley) [email protected] | (970) 217-8978
Quoted 8 lines · Mark Brown
One thing we typically cover at Kernel Summit is some of the activity that's going on around testing upstream. I think it'd be useful to have some more of those discussions, both in terms of making people aware of what's available and in terms of helping the people doing testing figure out what would be useful. A lot of this is probably well suited to a workshop session between the interested people but I do think some element in the core day beyond just a readout will be useful.
Quoted 4 lines
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Maybe list all known ones as a start ?
Quoted 7 lines
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it. Even the regular "Build regression" e-mails sent out by Geert seem to be widely ignored.
What I really found to help is to bisect new problems and send an e-mail to the responsible maintainer and to the submitter of the patch which introduced it. I'd like to automate that with my test system, but unfortunately I just don't have the time to do it.
Quoted 9 lines
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at. Should we be more generally slimming defconfigs and moving things into fragments?
and there's always the the perennial ones about what people would like to see testing for.
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant. Also, I would like to run more qemu configurations, but it is really hard to find working ones.
Guenter
Quoted 7 lines · Guenter Roeck
Quoted 3 lines · Mark Brown
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
Quoted 12 lines
Quoted 6 lines
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it. Even the regular "Build regression" e-mails sent out by Geert seem to be widely ignored.
Quoted 4 lines
What I really found to help is to bisect new problems and send an e-mail to the responsible maintainer and to the submitter of the patch which introduced it. I'd like to automate that with my test system, but unfortunately I just don't have the time to do it.
Yes, that's the "and interpreting" bit in the above - this only really works with people actively pushing. You do start to get people checking themselves once things are perceived as something people care about but it does take active work to establish and maintain that.
It also really helps if things are delivered promptly, and against trees people are actively developing for. But even with clear reports and sometimes patches not everyone shows an interest. As we get more and more actual testing running that's going to start to become more serious, breaking the build or boot will also mean that automated tests don't get to run.
This is one of the things 0day gets really right, when it kicks in it'll e-mail people directly and promptly.
Quoted 6 lines
Quoted 5 lines
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at. Should we be more generally slimming defconfigs and moving things into fragments?
Quoted 8 lines
Quoted 2 lines
and there's always the the perennial ones about what people would like to see testing for.
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant. Also, I would like to run more qemu configurations, but it is really hard to find working ones.
Grant (just CCed) was working intermittently on the qemu bit. I think the last plan was to enhance the scripts Kevin has for driving his build farm.
Quoted 7 lines · Mark Brown
Quoted 5 lines · Guenter Roeck
Quoted 3 lines · Mark Brown
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Quoted 6 lines
Quoted 1 line
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
The 0day service runs Coccinelle.
Coccinelle does not need the build to succeed.
julia
Quoted 51 lines
Quoted 12 lines
Quoted 6 lines
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it. Even the regular "Build regression" e-mails sent out by Geert seem to be widely ignored.
Quoted 4 lines
What I really found to help is to bisect new problems and send an e-mail to the responsible maintainer and to the submitter of the patch which introduced it. I'd like to automate that with my test system, but unfortunately I just don't have the time to do it.
Yes, that's the "and interpreting" bit in the above - this only really works with people actively pushing. You do start to get people checking themselves once things are perceived as something people care about but it does take active work to establish and maintain that.
It also really helps if things are delivered promptly, and against trees people are actively developing for. But even with clear reports and sometimes patches not everyone shows an interest. As we get more and more actual testing running that's going to start to become more serious, breaking the build or boot will also mean that automated tests don't get to run.
This is one of the things 0day gets really right, when it kicks in it'll e-mail people directly and promptly.
Quoted 6 lines
Quoted 5 lines
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at. Should we be more generally slimming defconfigs and moving things into fragments?
Quoted 8 lines
Quoted 2 lines
and there's always the the perennial ones about what people would like to see testing for.
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant. Also, I would like to run more qemu configurations, but it is really hard to find working ones.
Grant (just CCed) was working intermittently on the qemu bit. I think the last plan was to enhance the scripts Kevin has for driving his build farm.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 67 lines · Mark Brown
Quoted 8 lines · Guenter Roeck
Quoted 4 lines · Mark Brown
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
Quoted 14 lines
Quoted 7 lines
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it. Even the regular "Build regression" e-mails sent out by Geert seem to be widely ignored.
Quoted 4 lines
What I really found to help is to bisect new problems and send an e-mail to the responsible maintainer and to the submitter of the patch which introduced it. I'd like to automate that with my test system, but unfortunately I just don't have the time to do it.
Yes, that's the "and interpreting" bit in the above - this only really works with people actively pushing. You do start to get people checking themselves once things are perceived as something people care about but it does take active work to establish and maintain that.
It also really helps if things are delivered promptly, and against trees people are actively developing for. But even with clear reports and sometimes patches not everyone shows an interest. As we get more and more actual testing running that's going to start to become more serious, breaking the build or boot will also mean that automated tests don't get to run.
This is one of the things 0day gets really right, when it kicks in it'll e-mail people directly and promptly.
Quoted 10 lines
Quoted 9 lines
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at. Should we be more generally slimming defconfigs and moving things into fragments?
and there's always the the perennial ones about what people would like to see testing for.
Quoted 6 lines
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant. Also, I would like to run more qemu configurations, but it is really hard to find working ones.
Quoted 4 lines
Grant (just CCed) was working intermittently on the qemu bit. I think the last plan was to enhance the scripts Kevin has for driving his build farm.
Thanks for starting this discussion. Now that Kselftest install is in place and several cross-compile problems are fixed, I would like to gauge interest in being able to include kselftest in qemu boot tests. I added quicktest option in 4.2 to meet qemu environment time constraints.
thanks,
- -- Shuah
- --
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBAgAGBQJVnAtBAAoJEAsCRMQNDUMca5wP/jrsBO++J4IsOnaJYwChWRp1 XHNEtrVwnvuR9zGo295lQUZL0cihug8FiTkfHX3P6M+9kEJQaUKlMyJymh9y4dzU cmDLhh8kXwdtPq1qM8jzaTqBb6sqXvzXxRoMi3X7AYPd2yA4uLMluAgj+dg8n1x/ W55fIi4ghypNQvtLcC92zyp5MKtBd5zr75HUZPxsqPwo7AHtqy0KoV+1fx+qQ/Lt GnZxIlWQMmA0yubiAGZHl+nECfiwcx8iS302y30d3+8/YxAztUy9l/7VfgBBgyOp 9Ouw6xC8Xmyb8GfFUUBy4MIAvS0QrBcYQp/z9Qlmj5MioOBydPaS3+Y4AIKLEaeI Vu+/OVqZO9At+HML6TeDBvM44PWcjfC0plm8n8PLvFtruVnLEnxKeeaXCUrj9AG7 8GUTscjkOA5p8r3qqtKny3OftKvUINXZUL+zbEnV7kUlhlDuOBSN5SDZk6VkpTSP lTJOTew+At0JH0x8YEA5k6L94JNT4zfksS/4N7z6bGcIdBIITrn3MGYkoi1fCqh2 GLHqKV8fLMv5aEK4tt9phDgmJSxITsMbfwuwoPYK0KRY4hUfri73uuBB1Igg9noI d3H7yVYpL7Td3b98g7VuzIfOEdMbv9jFGgPJb8A0MiGYeu87iu56jJEXaj2JIAEo SxNLh6SDkNkuH+6l2FAJ
=jD4c
-----END PGP SIGNATURE-----
Quoted 1 line · Shuah Khan
-----BEGIN PGP SIGNED MESSAGE-----
[ ... ]
Quoted 7 lines
Thanks for starting this discussion. Now that Kselftest install is in place and several cross-compile problems are fixed, I would like to gauge interest in being able to include kselftest in qemu boot tests. I added quicktest option in 4.2 to meet qemu environment time constraints.
Me too, and very much so. Only reason for not doing it yet is lack of time.
Guenter
Quoted 7 lines · Mark Brown
Quoted 5 lines · Guenter Roeck
Quoted 3 lines · Mark Brown
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Quoted 7 lines
Quoted 1 line
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
Plus mine, of course. Only part missing is automated bisect and e-mail if something starts failing.
Which reminds me - do you use buildbot ? I think you are sending automated e-mail on failures. It would help me a lot if someone had automated bisect and the ability to e-mail results using buildbot to get me started.
Quoted 22 lines
Quoted 12 lines
Quoted 6 lines
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it. Even the regular "Build regression" e-mails sent out by Geert seem to be widely ignored.
Quoted 4 lines
What I really found to help is to bisect new problems and send an e-mail to the responsible maintainer and to the submitter of the patch which introduced it. I'd like to automate that with my test system, but unfortunately I just don't have the time to do it.
Yes, that's the "and interpreting" bit in the above - this only really works with people actively pushing. You do start to get people checking themselves once things are perceived as something people care about but it does take active work to establish and maintain that.
Quoted 7 lines
It also really helps if things are delivered promptly, and against trees people are actively developing for. But even with clear reports and sometimes patches not everyone shows an interest. As we get more and more actual testing running that's going to start to become more serious, breaking the build or boot will also mean that automated tests don't get to run.
Yes, I have seen that too. Especially 4.1 was pretty bad in this regard. 4.2 seems to be a bit better, though, so I hope that 4.1 was an exception.
Not really sure what to do about it. What turned out to help in the last two companies I worked for was automatic revert of broken patches. That sounds radical and I dislike it myself, but it helped.
Quoted 3 lines
This is one of the things 0day gets really right, when it kicks in it'll e-mail people directly and promptly.
Agreed.
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
- Should we start carrying config fragments upstream designed to support testing, things like the distro config fragments that keep getting discussed are one example here but there's other things like collections of debug options we could be looking at. Should we be more generally slimming defconfigs and moving things into fragments?
Quoted 8 lines
Quoted 2 lines
and there's always the the perennial ones about what people would like to see testing for.
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant. Also, I would like to run more qemu configurations, but it is really hard to find working ones.
Quoted 4 lines
Grant (just CCed) was working intermittently on the qemu bit. I think the last plan was to enhance the scripts Kevin has for driving his build farm.
Also of interest here (at least for me) would be to explore means to get more hardware (both architectures and platforms) supported in qemu, but I guess that may be a bit off topic.
Thanks,
Guenter
Quoted 7 lines · Guenter Roeck
Quoted 4 lines · Mark Brown
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
Quoted 6 lines
Plus mine, of course. Only part missing is automated bisect and e-mail if something starts failing.
Which reminds me - do you use buildbot ? I think you are sending automated e-mail on failures. It would help me a lot if someone had automated bisect and the ability to e-mail results using buildbot to get me started.
No, not me - all my failure reports are lovingly hand crafted using traditional artisan techniques. Kevin, Tyler and Fengguang have things but apart from 0day I think everything is still manually triggered.
Quoted 3 lines
Not really sure what to do about it. What turned out to help in the last two companies I worked for was automatic revert of broken patches. That sounds radical and I dislike it myself, but it helped.
Perhaps that's something we should be discussing? It may be something that we just evolve a solution for as we proceed though - right now it's largely theoretical. For -next Stephen will often carry extra patches that make sense.
Hi,
I definitely think discussing the next steps in terms of automated regression testing is important - so thanks for raising this issue!!
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
For the TPM subsystem I have several things in place:
- applying patches automatically triggers a build on travis-ci.org -- it triggers some basic style checkers
-- the build runs a qemu (with the tpm simulator) running another qemu (which now sees a real /dev/tpm0) with the new kernel.
--- the qemu-qemu-linux runs my tpm driver testsuite.
The reason behind this is that I can hook up besides the TPM1.2 simulator also the binary only windows based TPM2.0 simulator (with wine) without modifying qemu source -- but still the qemu-qemu-kernel sees 'real' hardware.
- some scripts for deploying new kernels to real hw machines and running tests with real hw tpms
- a mocked i2c tpm which I can hook up via the I2C_Slave interface, which also passes my driver testsuite -- so I can test the drivers within UML :)
- and of course wolfram's ninja-check scripts for style checkers
Quoted 2 lines
Plus mine, of course. Only part missing is automated bisect and e-mail if something starts failing.
Yeah - getting the reports reasonably fast is probably the most important stuff -- the 0day testing really does a great job here.
Thanks,
Peter
Quoted 7 lines · Mark Brown
Quoted 5 lines · Guenter Roeck
Quoted 3 lines · Mark Brown
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Quoted 7 lines
Quoted 1 line
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
There is also Marvin which has existed in some shape or form since November 2013. It checks once a month for new releases and executes a battery of tests on them across a range of machines. The primary purpose of it is performance verification and the tests are typically more complex than what I'd expect to see in kselftest. There are small amounts of overlap with 0-day but generally it's expected that Marvin runs tests that are more long-lived. Unlike 0-day, it also does not automatically notify people about regressions as some verification work is often required and I did not want it generating noise in any inbox. Technically, it does support automatic bisection but it's something I trigger manually when I confirm a problem is a performance regression and cannot quickly identify the root cause.
It actually has been publishing reports for several months now but I never mentioned it on the lists. I wrote up some details after reading this thread and posted it at
http://www.csn.ul.ie/~mel/blog/index.php?/archives/23-Continual-testing-of-mainline-kernels.html
If there was a workshop on testing then I'd be interested in attending and discussing what Marvin does if there was interest. Right now, performance tends to my area of interest so I'd be interesting is discussing if there are areas we are continually getting worse at that are slipping through the cracks. Chris proposed a topic in this general area that I think would be useful. I've only started looking at mainline kernel performance again recently and right now, I'm not aware of a single area where we are getting consistently worse. More commonly I see cases where we create problems and then later cover them up by fixing something else in the general area. Any time I find problems, it's a simple matter of programming and time to fix them but it'd be useful to hear what other peoples recent experiences have been.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 33 lines · Mel Gorman
Quoted 7 lines · Mark Brown
Quoted 5 lines · Guenter Roeck
Quoted 3 lines · Mark Brown
The main things I'm aware of that are happening at the minute are kselftest development, the 0day tester, plus kernelci.org and the other build and boot/test bots that are running against various trees.
Quoted 7 lines
Quoted 1 line
Maybe list all known ones as a start ?
Off the top of my head the automated ones I'm aware of are Olof's build & boot test, Dan running smatch and I think some other static analysis stuff, someone (not sure who?) running some coccinelle stuff, Coverity and I've got a builder too.
There is also Marvin which has existed in some shape or form since November 2013. It checks once a month for new releases and executes a battery of tests on them across a range of machines. The primary purpose of it is performance verification and the tests are typically more complex than what I'd expect to see in kselftest. There are small amounts of overlap with 0-day but generally it's expected that Marvin runs tests that are more long-lived. Unlike 0-day, it also does not automatically notify people about regressions as some verification work is often required and I did not want it generating noise in any inbox. Technically, it does support automatic bisection but it's something I trigger manually when I confirm a problem is a performance regression and cannot quickly identify the root cause.
It actually has been publishing reports for several months now but I never mentioned it on the lists. I wrote up some details after reading this thread and posted it at
http://www.csn.ul.ie/~mel/blog/index.php?/archives/23-Continual-testing-of-mainline-kernels.html
Quoted 13 lines
If there was a workshop on testing then I'd be interested in attending and discussing what Marvin does if there was interest. Right now, performance tends to my area of interest so I'd be interesting is discussing if there are areas we are continually getting worse at that are slipping through the cracks. Chris proposed a topic in this general area that I think would be useful. I've only started looking at mainline kernel performance again recently and right now, I'm not aware of a single area where we are getting consistently worse. More commonly I see cases where we create problems and then later cover them up by fixing something else in the general area. Any time I find problems, it's a simple matter of programming and time to fix them but it'd be useful to hear what other peoples recent experiences have been.
If there is enough interest, I can pull together a session on testing at the kernel summit. We can discuss kselftest future plans as well as adding performance tests into the kselftest, and discuss kselftest use in auotomated test environments.
thanks,
-- Shuahs
Signature · 5 lines
Shuah Khan Sr. Linux Kernel Developer Open Source Innovation Group Samsung Research America (Silicon Valley) [email protected] | (970) 217-8978
Quoted 11 lines · Guenter Roeck
Quoted 7 lines
In terms of discussion topics some of the issues I'm seeing are:
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it. Even the regular "Build regression" e-mails sent out by Geert seem to be widely ignored.
There's still a manual step involved, so I only download the logs and generate the regression emails for every rc release.
My original plan was to automate it more, so I could run it fully automated on the linux-next builds, too.
Quoted 4 lines
What I really found to help is to bisect new problems and send an e-mail to the responsible maintainer and to the submitter of the patch which introduced it. I'd like to automate that with my test system, but unfortunately I just don't have the time to do it.
IIRC, kisskb used to bisect failures, but stopped doing that when it started consuming too much time. Perhaps it can be restarted, given the system got an upgrade a few months ago and became much much faster? Stiil, there's the "analyzing" part.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Doesn't the 0day system obsolete the "Build regression" emails? It should be sending emails to everyone introducing new build warnings. Are people ignoring them?
regards,
dan carpenter
Hi Dan,
Quoted 1 line · Dan Carpenter
Doesn't the 0day system obsolete the "Build regression" emails? It
I think kisskb builds more/different configs.
Is there are list of configs and trees covered by the 0day system available?
Quoted 2 lines
should be sending emails to everyone introducing new build warnings. Are people ignoring them?
Of course people ignore them ;-)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Quoted 4 lines · Geert Uytterhoeven
Quoted 2 lines
should be sending emails to everyone introducing new build warnings. Are people ignoring them?
Of course people ignore them ;-)
I think the biggest problems with these are:
- they are all squashed together into one report, totally unrelated things together at one place. Noone is ever going to be actively looking into it to see whether something he's responsible for hasn't popped up
- they are not addressed to anybody explicitly. Sending them just to LKML is a direct ticket to the "be ignored" land
Thanks,
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 7 lines · Jiri Kosina
Quoted 4 lines · Geert Uytterhoeven
Quoted 2 lines
should be sending emails to everyone introducing new build warnings. Are people ignoring them?
Of course people ignore them ;-)
Quoted 8 lines
I think the biggest problems with these are:
- they are all squashed together into one report, totally unrelated things together at one place. Noone is ever going to be actively looking into it to see whether something he's responsible for hasn't popped up
- they are not addressed to anybody explicitly. Sending them just to LKML is a direct ticket to the "be ignored" land
Yep. The same goes for bug reports. LKML is a great archiving tool for digging through history, but it is useless as a form of general communication.
josh
Quoted 1 line · Jiri Kosina
On Wed, 8 Jul 2015, Geert Uytterhoeven wrote:
[0day]
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
should be sending emails to everyone introducing new build warnings. Are people ignoring them?
Of course people ignore them ;-)
Quoted 8 lines
I think the biggest problems with these are:
- they are all squashed together into one report, totally unrelated things together at one place. Noone is ever going to be actively looking into it to see whether something he's responsible for hasn't popped up
- they are not addressed to anybody explicitly. Sending them just to LKML is a direct ticket to the "be ignored" land
That's not the case for the 0day warning reports, they're sent directly to the people mentioned in the commit on a per-commit basis but as a one shot.
Hi Geert,
Quoted 8 lines · Geert Uytterhoeven
Hi Dan,
Quoted 1 line · Dan Carpenter
Doesn't the 0day system obsolete the "Build regression" emails? It
I think kisskb builds more/different configs.
Is there are list of configs and trees covered by the 0day system available?
Apart from randconfig and all(yes|no|def|mod) configs, the comprehensive list of configs can be found in arch/*/configs
$ find arch/*/configs -type f | wc -l
4610-day tests them all. Since there are so many of them, only 10% are run immediately after git pushes aiming for 1-hour respond time. The others will be tested in machine idle time and see longer delay.
The code coverage is now 543 git trees. Here is the comprehensive list:
https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/repo/linux
$ cat repo/linux/ext4 url: git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git
I'm always seeking to add more trees. Patches to add new URLs and tests are highly appreciated!
Thanks,
Fengguang
Quoted 2 lines · Dan Carpenter
Doesn't the 0day system obsolete the "Build regression" emails? It should be sending emails to everyone introducing new build warnings.
0day only covers some configurations and doesn't always manage to kick in (I know I've had stuff come in via other channels sometimes, it seems to depend on loading).
Quoted 1 line
Are people ignoring them?
They're not reliably followed through on, no, and one of the things with 0day is that it just generates a one time report so if things don't get followed up on then that's that. A regular "these are all the issues" mail helps chase down those issues.
Quoted 6 lines · Mark Brown
Quoted 2 lines · Dan Carpenter
Doesn't the 0day system obsolete the "Build regression" emails? It should be sending emails to everyone introducing new build warnings.
0day only covers some configurations
Last year it has extended kconfig coverage to all arch/*/configs/*. Adding defconfigs there will auto add them to 0day. :)
Quoted 3 lines
and doesn't always manage to kick
in (I know I've had stuff come in via other channels sometimes, it seems to depend on loading).
If so, it should be considered a bug. Feel free to report such cases to me.
If 0day failed to catch the error, it might be due to
- regressions in 0day itself -- it's still in active development, in the past year I did 1k patches to 0day build/boot scripts and 2k patches to the follow up LKP (Linux Kernel Performance) tests.
- the "report once" logic, it's tricky and can possibly hide issues
- failed bisect, rare but possible
- machine hang, network/disk fails, etc. maintenance incidents
"Loading" may add latency, however it's not the cause to miss errors.
Quoted 6 lines
Quoted 1 line
Are people ignoring them?
They're not reliably followed through on, no, and one of the things with 0day is that it just generates a one time report so if things don't get followed up on then that's that. A regular "these are all the issues" mail helps chase down those issues.
0day has such report type. It will be sent after each git push (unless you push too quickly) and it looks like this. Just drop me a note and list the git trees/branches you wish to receive such notice emails.
Subject: [amirv:for-upstream] b132dcd12e0ab2a49ae5b02b5549cb65408a96ef BUILD DONE
git://flatbed.openfabrics.org/~amirv/linux.git for-upstream b132dcd12e0ab2a49ae5b02b5549cb65408a96ef IB/mlx4: Use correct device for XRC [ib-next:issue_511091]
drivers/infiniband/core/cache.c:558:9: sparse: too many arguments for function __builtin_expect drivers/infiniband/core/device.c:176:9: sparse: too many arguments for function __builtin_expect drivers/infiniband/core/device.c:767:4-21: code aligned with following code on line 768
Error ids grouped by kconfigs:
recent_errors
├── i386-allmodconfig
│ └── drivers-infiniband-core-device.c:code-aligned-with-following-code-on-line
└── x86_64-allmodconfig
├── drivers-infiniband-core-cache.c:sparse:too-many-arguments-for-function-__builtin_expect
└── drivers-infiniband-core-device.c:sparse:too-many-arguments-for-function-__builtin_expectelapsed time: 53m
configs tested: 83
powerpc tqm8548_defconfig
powerpc tqm8555_defconfig
um alldefconfig
i386 randconfig-a0-201527x86_64 allnoconfig
sh titan_defconfig
sh rsk7269_defconfig
sh sh7785lcr_32bit_defconfig
sh allnoconfig
i386 allyesconfig
arm sa1100
arm at91_dt_defconfig
arm allnoconfig
arm samsung
...
Thanks,
Fengguang
Quoted 9 lines · Fengguang Wu
If 0day failed to catch the error, it might be due to
- regressions in 0day itself -- it's still in active development, in the past year I did 1k patches to 0day build/boot scripts and 2k patches to the follow up LKP (Linux Kernel Performance) tests.
- the "report once" logic, it's tricky and can possibly hide issues
- failed bisect, rare but possible
I think some of the issues have been due to bisection getting confused by issues appearing in merge commits but ICBW.
Quoted 3 lines
- machine hang, network/disk fails, etc. maintenance incidents
"Loading" may add latency, however it's not the cause to miss errors.
Latency seems like it might be an issue here, when I say I'm not seeing things that's issues coming up in the -next build before they're reported by 0day so they might well get fixed before 0day catches up.
Quoted 10 lines
Quoted 6 lines
Quoted 1 line
Are people ignoring them?
They're not reliably followed through on, no, and one of the things with 0day is that it just generates a one time report so if things don't get followed up on then that's that. A regular "these are all the issues" mail helps chase down those issues.
0day has such report type. It will be sent after each git push (unless you push too quickly) and it looks like this. Just drop me a note and list the git trees/branches you wish to receive such notice emails.
For me personally it'd be more interesting to be able to get them on demand (eg, from a web page) than e-mailed, or e-mailed by a human (possibly with fixes!). The kernelci.org reporting does a lot of this but doesn't cover anything except raw compiler warnings.
Quoted 15 lines · Mark Brown
Quoted 9 lines · Fengguang Wu
If 0day failed to catch the error, it might be due to
- regressions in 0day itself -- it's still in active development, in the past year I did 1k patches to 0day build/boot scripts and 2k patches to the follow up LKP (Linux Kernel Performance) tests.
- the "report once" logic, it's tricky and can possibly hide issues
- failed bisect, rare but possible
I think some of the issues have been due to bisection getting confused by issues appearing in merge commits but ICBW.
Quoted 7 lines
Quoted 3 lines
- machine hang, network/disk fails, etc. maintenance incidents
"Loading" may add latency, however it's not the cause to miss errors.
Latency seems like it might be an issue here, when I say I'm not seeing things that's issues coming up in the -next build before they're reported by 0day so they might well get fixed before 0day catches up.
Ah linux-next happen to have lower priority and can take many hours to finish. I'll increase its priority due to its importance.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Are people ignoring them?
They're not reliably followed through on, no, and one of the things with 0day is that it just generates a one time report so if things don't get
The real policy is in fact a bit more smart than "report-once".
Build errors will be auto re-reported if it's still not fixed after 10 days when the branch that introduced the error is updated.
Warnings won't be auto re-reported.
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
followed up on then that's that. A regular "these are all the issues" mail helps chase down those issues.
0day has such report type. It will be sent after each git push (unless you push too quickly) and it looks like this. Just drop me a note and list the git trees/branches you wish to receive such notice emails.
For me personally it'd be more interesting to be able to get them on demand (eg, from a web page) than e-mailed, or e-mailed by a human (possibly with fixes!). The kernelci.org reporting does a lot of this but doesn't cover anything except raw compiler warnings.
It should be mostly equivalent if you direct such emails to a local mbox and check it on demand. :)
For example, setup .procmailrc rule like this:
:0:
* ^Subject: \[.*\] ........................................ BUILD build-complete-notification
Thanks,
Fengguang
Quoted 8 lines · Fengguang Wu
Quoted 3 lines · Mark Brown
Latency seems like it might be an issue here, when I say I'm not seeing things that's issues coming up in the -next build before they're reported by 0day so they might well get fixed before 0day catches up.
Ah linux-next happen to have lower priority and can take many hours to finish. I'll increase its priority due to its importance.
This was actually things in my trees that get merged into -next (or which Stephen tries to merge into -next and finds problems with) prior to being reported by 0day.
Quoted 7 lines
Quoted 4 lines
For me personally it'd be more interesting to be able to get them on demand (eg, from a web page) than e-mailed, or e-mailed by a human (possibly with fixes!). The kernelci.org reporting does a lot of this but doesn't cover anything except raw compiler warnings.
It should be mostly equivalent if you direct such emails to a local mbox and check it on demand. :)
Well, they don't go away automatically when fixed then...
Quoted 7 lines · Mark Brown
Quoted 5 lines · Fengguang Wu
Quoted 3 lines · Mark Brown
Latency seems like it might be an issue here, when I say I'm not seeing things that's issues coming up in the -next build before they're reported by 0day so they might well get fixed before 0day catches up.
Quoted 6 lines
Quoted 2 lines
Ah linux-next happen to have lower priority and can take many hours to finish. I'll increase its priority due to its importance.
This was actually things in my trees that get merged into -next (or which Stephen tries to merge into -next and finds problems with) prior to being reported by 0day.
I feel sorry about that and would like to improve. When it happen in future, feel free to forward me of the missed/delayed error. Then I can try to root cause case-by-case.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
For me personally it'd be more interesting to be able to get them on demand (eg, from a web page) than e-mailed, or e-mailed by a human (possibly with fixes!). The kernelci.org reporting does a lot of this but doesn't cover anything except raw compiler warnings.
It should be mostly equivalent if you direct such emails to a local mbox and check it on demand. :)
Well, they don't go away automatically when fixed then...
You may just check the latest build compete notification email. It'll contain the latest information.
Thanks,
Fengguang
Quoted 11 lines · Guenter Roeck
Quoted 10 lines · Mark Brown
One thing we typically cover at Kernel Summit is some of the activity that's going on around testing upstream. I think it'd be useful to have some more of those discussions, both in terms of making people aware of what's available and in terms of helping the people doing testing figure out what would be useful. A lot of this is probably well suited to a workshop session between the interested people but I do think some element in the core day beyond just a readout will be useful.
In terms of discussion topics some of the issues I'm seeing are:
Quoted 8 lines
Quoted 4 lines
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it.
My concern with kisskb sending emails was always that I didn't want it to become a spam bot. So it can send emails, but it's opt-in.
The 0-day bot takes the opposite approach, ie. mails everyone without asking, and in hindsight that is clearly the better option in terms of getting people to act on the results.
Quoted 4 lines
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant.
Agreed. Your buildbot is epic. I'd love to see the config for that. My local buildbot is running only ~40 builders, which I thought was a lot until I saw yours :)
The kernelci.org stuff is also really interesting, that's the closest thing anyone has at the moment to a "proper" kernel CI setup AFAIK.
cheers
Hi Michael,
Quoted 29 lines · Michael Ellerman
Quoted 11 lines · Guenter Roeck
Quoted 10 lines · Mark Brown
One thing we typically cover at Kernel Summit is some of the activity that's going on around testing upstream. I think it'd be useful to have some more of those discussions, both in terms of making people aware of what's available and in terms of helping the people doing testing figure out what would be useful. A lot of this is probably well suited to a workshop session between the interested people but I do think some element in the core day beyond just a readout will be useful.
In terms of discussion topics some of the issues I'm seeing are:
Quoted 8 lines
Quoted 4 lines
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it.
My concern with kisskb sending emails was always that I didn't want it to become a spam bot. So it can send emails, but it's opt-in.
The 0-day bot takes the opposite approach, ie. mails everyone without asking, and in hindsight that is clearly the better option in terms of getting people to act on the results.
Quoted 9 lines
Quoted 4 lines
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant.
Agreed. Your buildbot is epic. I'd love to see the config for that. My local buildbot is running only ~40 builders, which I thought was a lot until I saw yours :)
It is on github: https://github.com/groeck/linux-build-test.
If I count correctly, it runs more than 900 builders. Major hiccup is with caching - it collects around 10GB of logging data per month, and under some circumstances keeps it all in memory, so after about two months it consumes the entire 32GB of RAM on the server it is running on, and I have to do some manual cleanup. Other than that, it runs surprisingly well.
Quoted 3 lines
The kernelci.org stuff is also really interesting, that's the closest thing anyone has at the moment to a "proper" kernel CI setup AFAIK.
Agreed.
Guenter
Quoted 41 lines · Guenter Roeck
Hi Michael,
Quoted 29 lines · Michael Ellerman
Quoted 11 lines · Guenter Roeck
Quoted 10 lines · Mark Brown
One thing we typically cover at Kernel Summit is some of the activity that's going on around testing upstream. I think it'd be useful to have some more of those discussions, both in terms of making people aware of what's available and in terms of helping the people doing testing figure out what would be useful. A lot of this is probably well suited to a workshop session between the interested people but I do think some element in the core day beyond just a readout will be useful.
In terms of discussion topics some of the issues I'm seeing are:
Quoted 8 lines
Quoted 4 lines
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it.
My concern with kisskb sending emails was always that I didn't want it to become a spam bot. So it can send emails, but it's opt-in.
The 0-day bot takes the opposite approach, ie. mails everyone without asking, and in hindsight that is clearly the better option in terms of getting people to act on the results.
Quoted 9 lines
Quoted 4 lines
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant.
Agreed. Your buildbot is epic. I'd love to see the config for that. My local buildbot is running only ~40 builders, which I thought was a lot until I saw yours :)
Quoted 7 lines
It is on github: https://github.com/groeck/linux-build-test.
If I count correctly, it runs more than 900 builders. Major hiccup is with caching - it collects around 10GB of logging data per month, and under some circumstances keeps it all in memory, so after about two months it consumes the entire 32GB of RAM on the server it is running on, and I have to do some manual cleanup. Other than that, it runs surprisingly well.
How soon after a branch is pushed are the build results available?
Are the build artfiacts ([bz]Image, System.map, modules, etc.) made available someplace?
Quoted 4 lines
Quoted 2 lines
The kernelci.org stuff is also really interesting, that's the closest thing anyone has at the moment to a "proper" kernel CI setup AFAIK.
Agreed.
Glad you find it useful.
Speaking for kernelci.org... with limited time/resources, we'd like to focus less on building (others are doing this faster/better) and more on boot testing across a wide variety of hardware (e.g for the latest -next build, we did ~430 boots on 88 unique boards covering 23 different SoCs[1]). We're also in the process of automating kselftest runs on our boards.
If we could consume the output of other builders, we'd happily do that instead of doing our own builds. Ideally, the builders should produce some sort of machine readable data with the build artifacts.. Currently our builders produce a JSON file[2] which can be submitted to kernelci.org using a RESTful API[3].
AFAICT, 0day doesn't have publically available build artfiacts, and I haven't had a chance to look closely at Gunter's stuff.
Kevin
[1] http://kernelci.org/boot/all/job/next/kernel/next-20150708/
[2] example for an ARM multi_v7_defconfib build:
http://storage.kernelci.org/next/next-20150708/arm-multi_v7_defconfig/build.json
Quoted 45 lines · Kevin Hilman
Quoted 43 lines · Guenter Roeck
Hi Michael,
Quoted 29 lines · Michael Ellerman
Quoted 11 lines · Guenter Roeck
Quoted 10 lines · Mark Brown
One thing we typically cover at Kernel Summit is some of the activity that's going on around testing upstream. I think it'd be useful to have some more of those discussions, both in terms of making people aware of what's available and in terms of helping the people doing testing figure out what would be useful. A lot of this is probably well suited to a workshop session between the interested people but I do think some element in the core day beyond just a readout will be useful.
In terms of discussion topics some of the issues I'm seeing are:
Quoted 8 lines
Quoted 4 lines
- Can we pool resources to share the workload of running things and interpreting results, ideally also providing some central way for people to discover what results are out there for them to look at for a given kernel in the different systems?
That might be quite useful. However, I have seen that it doesn't really help to just provide the test results. kissb test results have been available for ages, and people just don't look at it.
My concern with kisskb sending emails was always that I didn't want it to become a spam bot. So it can send emails, but it's opt-in.
The 0-day bot takes the opposite approach, ie. mails everyone without asking, and in hindsight that is clearly the better option in terms of getting people to act on the results.
Quoted 9 lines
Quoted 4 lines
Sharing as many test bot configuration scripts and relevant configurations as possible would be quite helpful. For example, I am building various configurations for all architectures, but I don't really know if they are relevant.
Agreed. Your buildbot is epic. I'd love to see the config for that. My local buildbot is running only ~40 builders, which I thought was a lot until I saw yours :)
It is on github: https://github.com/groeck/linux-build-test.
Quoted 8 lines
Quoted 5 lines
If I count correctly, it runs more than 900 builders. Major hiccup is with caching - it collects around 10GB of logging data per month, and under some circumstances keeps it all in memory, so after about two months it consumes the entire 32GB of RAM on the server it is running on, and I have to do some manual cleanup. Other than that, it runs surprisingly well.
How soon after a branch is pushed are the build results available?
Depends on the branch and the load on the system. Building a single branch takes maybe - I am guessing here - 90 minutes. If several branches are changed at the same time (like when Greg pushes changes into the -stable branches he manages), the overall build time is multiplied with the number of branches to build.
In addition to that, each branch has a different set of rules. I only pull mainline a couple of times per week, because it changes too much and by itself would keep the system busy all the time. -next is pulled once per day, as is -mmotm. Plus, I have a number of different methods on how to handle changes. For example, Greg tends to push many changes into -stable queues within a short period of time, resulting in a lot of build churn if I would start builds immediately after a change is detected. So if a change on a -stable branch maintained by Greg is detected, the system waits for a couple of hours before actually starting a build to reduce churn. Other branches see less churn; -mmotm and -next are built almost immediately after a change is detected, for example.
In a nutshell, it can take anytime between about two hours and about three days for results to be available, depending on the branch and the load on the system.
Quoted 3 lines
Are the build artfiacts ([bz]Image, System.map, modules, etc.) made available someplace?
Not currently. I could make it available if there is interest, though I'd probably have to find some place out in the web to keep the data (we are talking 16 branches times ~150 builds per branch times the number of builds kept in the log). If I assume that five builds are kept per branch, and each build takes only 10MB of space, this already adds up to more than 100 GB of space needed, plus of course all the associated net bandwidth. There are a couple of sites pulling the complete build history from my servers on a daily basis (happily ignoring robots.txt), so that might end up consuming a lot of bandwidth.
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines
The kernelci.org stuff is also really interesting, that's the closest thing anyone has at the moment to a "proper" kernel CI setup AFAIK.
Agreed.
Glad you find it useful.
Speaking for kernelci.org... with limited time/resources, we'd like to focus less on building (others are doing this faster/better) and more on boot testing across a wide variety of hardware (e.g for the latest -next build, we did ~430 boots on 88 unique boards covering 23 different SoCs[1]). We're also in the process of automating kselftest runs on our boards.
Quoted 6 lines
If we could consume the output of other builders, we'd happily do that instead of doing our own builds. Ideally, the builders should produce some sort of machine readable data with the build artifacts.. Currently our builders produce a JSON file[2] which can be submitted to kernelci.org using a RESTful API[3].
Unfortunately, I had to disable JSON on my buildbot installation. Reason is again that there are sites pulling the entire build history on a regular basis, and my server ended up getting stuck doing nothing but serving JSON requests (those take a lot of CPU).
Guenter
Quoted 8 lines · Guenter Roeck
Quoted 3 lines
Are the build artfiacts ([bz]Image, System.map, modules, etc.) made available someplace?
Not currently. I could make it available if there is interest, though I'd probably have to find some place out in the web to keep the data (we are talking 16 branches times ~150 builds per branch times the number of builds kept in the log). If I assume that five builds are kept per branch, and each build takes only 10MB of space, this already adds up to more than 100 GB of space needed,
Understood. storage.kernelci.org is currently using ~400G for 45 days worth of builds from the 55+ branches x ~150 builds/branch we do, and includes boot logs/results too.
Kevin
Quoted 11 lines · Kevin Hilman
Speaking for kernelci.org... with limited time/resources, we'd like to focus less on building (others are doing this faster/better) and more on boot testing across a wide variety of hardware (e.g for the latest -next build, we did ~430 boots on 88 unique boards covering 23 different SoCs[1]). We're also in the process of automating kselftest runs on our boards.
If we could consume the output of other builders, we'd happily do that instead of doing our own builds. Ideally, the builders should produce some sort of machine readable data with the build artifacts..
Unfortunately providing build artifacts means you're distributing binaries, and so life just got complicated. At least for folks like me who work at a company with too many lawyers.
cheers
Quoted 9 lines · Michael Ellerman
Quoted 8 lines · Kevin Hilman
Speaking for kernelci.org... with limited time/resources, we'd like to focus less on building (others are doing this faster/better) and more on boot testing across a wide variety of hardware (e.g for the latest -next build, we did ~430 boots on 88 unique boards covering 23 different SoCs[1]). We're also in the process of automating kselftest runs on our boards.
Quoted 8 lines
Quoted 3 lines
If we could consume the output of other builders, we'd happily do that instead of doing our own builds. Ideally, the builders should produce some sort of machine readable data with the build artifacts..
Unfortunately providing build artifacts means you're distributing binaries, and so life just got complicated. At least for folks like me who work at a company with too many lawyers.
Never thought about that aspect, but you are right, there would be some hoops to go through if the test infrastructure runs in a corporate environment. Should not be too difficult, though, at least if the lawyers are reasonable. And if not, maybe they just need a different set of guildelines.
Guenter