Skip to contentSkip to conversation

kselftest - What's in 3.17 and plans for 3.18 and beyond

Ksummit-discuss Shuah Khan 16 participants 40 messages Aug 7, 2014 – Aug 18, 2014

lore
Share & tools
lore

Shuah Khan

author
#0 lore raw

As a first step towards a larger goal to enable developer friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

What's planned for 3.18 and beyond:

- get feedback on the new kselftest target from the community
- add more tests to be run under kselftest umbrella
- identify existing tests under /lib and other areas that make a good candidate to be included under kselftest
- Some of these could be run as a tool and/or a independent test with a few changes and some probably aren't like the /lib/locking tests.
- As a goal, try to leverage existing tests and modify them as needed to run them as a black-box test (e.g: look into ways to make it run as a tool)
- Greg KH sparked the kernel selftest idea, has been in the loop for the work done so far, and reviewed the plan for 3.18.

thanks,
-- Shuah

Shuah Khan
Senior Linux Kernel Developer - Open Source Group Samsung Research America(Silicon Valley) [email protected] | (970) 672-0658

#1 lore raw
Quoted 20 lines · Shuah Khan

As a first step towards a larger goal to enable developer friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

What's planned for 3.18 and beyond:

Quoted 12 lines

- get feedback on the new kselftest target from the community
- add more tests to be run under kselftest umbrella
- identify existing tests under /lib and other areas that make a good candidate to be included under kselftest
- Some of these could be run as a tool and/or a independent test with a few changes and some probably aren't like the /lib/locking tests.
- As a goal, try to leverage existing tests and modify them as needed to run them as a black-box test (e.g: look into ways to make it run as a tool)
- Greg KH sparked the kernel selftest idea, has been in the loop for the work done so far, and reviewed the plan for 3.18.

I'm quite interested in this (as is the CEWG), and I have lots of questions. Will this be discussed/presented at KS or LinuxCon? Should I ask my questions on this list or wait until Chicago? Are there any LKML threads with info I can look at in the meantime?

Thanks. This sounds like great work.
-- Tim

author
#2 lore raw
Quoted 7 lines · Tim

I'm quite interested in this (as is the CEWG), and I have lots of questions. Will this be discussed/presented at KS or LinuxCon? Should I ask my questions on this list or wait until Chicago? Are there any LKML threads with info I can look at in the meantime?

Thanks. This sounds like great work.

The plan to discuss this at the KS. Please continue this thread with questions you have. I can point you to links to the patches for the work done so far in 3.17 if you are interested.

-- Shuah

Signature · 4 lines
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
[email protected] | (970) 672-0658
#3 lore raw
In reply to Shuah Khan

Hello,

I'm also interested in the selftests, especially its framework (top-level script).

http://article.gmane.org/gmane.linux.kernel/1763766

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...
Also, as I said in above mail, I'd like to suggest to add at least log management and statistics to the test. Those are good to automate regression tests. :)

Quoted 15 lines

What's planned for 3.18 and beyond:

- get feedback on the new kselftest target from the community
- add more tests to be run under kselftest umbrella
- identify existing tests under /lib and other areas that make a good candidate to be included under kselftest
- Some of these could be run as a tool and/or a independent test with a few changes and some probably aren't like the /lib/locking tests.
- As a goal, try to leverage existing tests and modify them as needed to run them as a black-box test (e.g: look into ways to make it run as a tool)
- Greg KH sparked the kernel selftest idea, has been in the loop for the work done so far, and reviewed the plan for 3.18.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
author
#4 lore raw
Quoted 6 lines · Masami Hiramatsu

Hello,

I'm also interested in the selftests, especially its framework (top-level script).

http://article.gmane.org/gmane.linux.kernel/1763766

Great.

Quoted 18 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

One of the goals is to leverage developer tests that we already have.
When a developer makes a kernel change and wants to see if that change
lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

Existing selfests are a collection of tools that exercise several testcases that are specific each area they target. This is a good start and we can definitely start enhancing the tools and build testcases.

Quoted 4 lines

Also, as I said in above mail, I'd like to suggest to add at least log management and statistics to the test. Those are good to automate regression tests. :)

Right. Some existing selftests do that, but not all. It would be helpful to add that to all.

-- Shuah

Signature · 4 lines
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
[email protected] | (970) 672-0658
#5 lore raw
Quoted 21 lines · Shuah Khan
Quoted 17 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

Quoted 4 lines

Existing selfests are a collection of tools that exercise several testcases that are specific each area they target. This is a good start and we can definitely start enhancing the tools and build testcases.

Agreed.

Quoted 8 lines
Quoted 5 lines

Also, as I said in above mail, I'd like to suggest to add at least log management and statistics to the test. Those are good to automate regression tests. :)

Right. Some existing selftests do that, but not all. It would be helpful to add that to all.

Yeah, and I hope the selftest top-level script to provide such functions to each test, because those should be common, and easy to manage.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#6 lore raw
Quoted 22 lines · Masami Hiramatsu
Quoted 21 lines · Shuah Khan
Quoted 20 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 8 lines
Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

g.

#7 lore raw
Quoted 29 lines · Grant Likely
Quoted 28 lines · Masami Hiramatsu
Quoted 26 lines · Shuah Khan
Quoted 17 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

Quoted 9 lines
Quoted 2 lines

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Do you have that public yet ? I might want to use that for my -stable sanity tests.

Thanks,
Guenter

#8 lore raw
Quoted 33 lines · Guenter Roeck
Quoted 32 lines · Grant Likely
Quoted 30 lines · Masami Hiramatsu
Quoted 21 lines · Shuah Khan
Quoted 17 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

Quoted 8 lines
Quoted 6 lines

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Do you have that public yet ? I might want to use that for my -stable sanity tests.

IMHO, We'd better share those testing tools/environments as much as possible so that each developer can ensure no regressions in their patches before sending it.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#9 lore raw
Quoted 5 lines · Masami Hiramatsu
Quoted 5 lines
Quoted 5 lines

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Like Aboriginal Linux (landley.net/aboriginal/)?

Quoted 5 lines
Quoted 1 line

Do you have that public yet ? I might want to use that for my -stable sanity tests.

IMHO, We'd better share those testing tools/environments as much as possible so that each developer can ensure no regressions in their patches before sending it.

Yes!

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#11 lore raw
In reply to Guenter Roeck
Quoted 50 lines · Guenter Roeck
Quoted 42 lines · Grant Likely
Quoted 35 lines · Masami Hiramatsu
Quoted 26 lines · Shuah Khan
Quoted 21 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 7 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

Quoted 2 lines

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

Quoted 6 lines

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 2 lines

Do you have that public yet ? I might want to use that for my -stable sanity tests.

Nothing yet. It's currently hacked into my build environment script. I'm working on getting it into a form usable by others. Ideally I think it should go into the kernel tools/testing directory.

g.

#12 lore raw
Quoted 8 lines · Grant Likely
Quoted 2 lines · Guenter Roeck

Do you have that public yet ? I might want to use that for my -stable sanity tests.

Nothing yet. It's currently hacked into my build environment script. I'm working on getting it into a form usable by others. Ideally I think it should go into the kernel tools/testing directory.

While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports.

#13 lore raw
Quoted 10 lines · Mark Brown
Quoted 8 lines · Grant Likely
Quoted 2 lines · Guenter Roeck

Do you have that public yet ? I might want to use that for my -stable sanity tests.

Nothing yet. It's currently hacked into my build environment script. I'm working on getting it into a form usable by others. Ideally I think it should go into the kernel tools/testing directory.

Quoted 5 lines

While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports.

Where is the current copy of those scripts?

g.

#14 lore raw
Quoted 9 lines · Grant Likely
Quoted 5 lines · Mark Brown

While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports.

Where is the current copy of those scripts?

Kevin's copy is here:

git://git.linaro.org/people/khilman/build-scripts.git

#15 lore raw
Quoted 9 lines · Mark Brown
Quoted 7 lines · Grant Likely
Quoted 5 lines · Mark Brown

While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports.

Quoted 5 lines
Quoted 1 line

Where is the current copy of those scripts?

Kevin's copy is here:

git://git.linaro.org/people/khilman/build-scripts.git

404

Kevin's name got expanded:

git://git.linaro.org/people/kevin.hilman/build-scripts.git

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#16 lore raw
Quoted 11 lines · Geert Uytterhoeven
Quoted 3 lines · Mark Brown

Kevin's copy is here:

git://git.linaro.org/people/khilman/build-scripts.git

404

Kevin's name got expanded:

git://git.linaro.org/people/kevin.hilman/build-scripts.git

Interesting... how are you cloning? Both work for me:

$ git clone git://git.linaro.org/people/khilman/build-scripts.git
Cloning into 'build-scripts'...
remote: Counting objects: 489, done.
remote: Compressing objects: 100% (307/307), done.
remote: Total 489 (delta 302), reused 295 (delta 181)
Receiving objects: 100% (489/489), 73.90 KiB, done.
Resolving deltas: 100% (302/302), done.
#17 lore raw

Hi Mark,

Quoted 15 lines · Mark Brown
Quoted 11 lines · Geert Uytterhoeven
Quoted 3 lines · Mark Brown

Kevin's copy is here:

git://git.linaro.org/people/khilman/build-scripts.git

404

Kevin's name got expanded:

git://git.linaro.org/people/kevin.hilman/build-scripts.git

Interesting... how are you cloning? Both work for me:

Quoted 7 lines
$ git clone git://git.linaro.org/people/khilman/build-scripts.git
Cloning into 'build-scripts'...
remote: Counting objects: 489, done.
remote: Compressing objects: 100% (307/307), done.
remote: Total 489 (delta 302), reused 295 (delta 181)
Receiving objects: 100% (489/489), 73.90 KiB, done.
Resolving deltas: 100% (302/302), done.

Bummer, I didn't clone, but opened
http://git.linaro.org/people/khilman/build-scripts.git in my web browser by clicking on the link.
So Linaro gitweb supports the full names only, while real git can access both (cloning indeed works for both).

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#18 lore raw
In reply to Mark Brown
Quoted 9 lines · Mark Brown
Quoted 7 lines · Grant Likely
Quoted 5 lines · Mark Brown

While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports.

Quoted 5 lines
Quoted 1 line

Where is the current copy of those scripts?

Kevin's copy is here:

git://git.linaro.org/people/khilman/build-scripts.git

FWIW, that repo is just a hodge-podge collection of various build/boot scripts I'm using. I just cleaned it up a little and removed a bunch of them that I no longer use.

They get the job done for me, but are certainly not pretty.

Also for boot testing, the main tool I'm using is pyboot, which is essentially a tool to automate u-boot/fastboot over the serial console. It relies heavily on conmux to abstract the console and power cycling.

https://git.linaro.org/people/kevin.hilman/pyboot.git

In any case, the combination of those build-scripts and pyboot are what I'm using to generated the automated build/boot reports I'm sending to the kernel-build-reports list (e.g.
http://lists.linaro.org/pipermail/kernel-build-reports/2014-August/004862.html)

Kevin

#19 lore raw
Quoted 27 lines · Kevin Hilman
Quoted 9 lines · Mark Brown
Quoted 7 lines · Grant Likely
Quoted 5 lines · Mark Brown

While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports.

Quoted 5 lines
Quoted 1 line

Where is the current copy of those scripts?

Kevin's copy is here:

git://git.linaro.org/people/khilman/build-scripts.git

FWIW, that repo is just a hodge-podge collection of various build/boot scripts I'm using. I just cleaned it up a little and removed a bunch of them that I no longer use.

They get the job done for me, but are certainly not pretty.

Also for boot testing, the main tool I'm using is pyboot, which is essentially a tool to automate u-boot/fastboot over the serial console. It relies heavily on conmux to abstract the console and power cycling.

Quoted 6 lines

https://git.linaro.org/people/kevin.hilman/pyboot.git

In any case, the combination of those build-scripts and pyboot are what I'm using to generated the automated build/boot reports I'm sending to the kernel-build-reports list (e.g.
http://lists.linaro.org/pipermail/kernel-build-reports/2014-August/004862.html)

Yeah, I'm in the same boat (but I don't have a public git repo with my scripts). They're all just big hacks to get things going, and I'm not so sure it makes sense to collaborate on them too much. pyboot has been useful to share between us, but even there it's been more work to share a code base than having separate copies and borrowing ideas.

If things get turned too generic then they just become more cumbersome to modify and improve (or even use).

-Olof

#20 lore raw
Quoted 8 lines · Olof Johansson

Yeah, I'm in the same boat (but I don't have a public git repo with my scripts). They're all just big hacks to get things going, and I'm not so sure it makes sense to collaborate on them too much. pyboot has been useful to share between us, but even there it's been more work to share a code base than having separate copies and borrowing ideas.

If things get turned too generic then they just become more cumbersome to modify and improve (or even use).

I think the reporting side is a lot more tractable than the execution side here - there's enough going on with crunching a set of build and boot logs into a useful report to be useful and it's a pretty common need. The execution side is different, that's a lot more likely to be tied into the environment especially once you start running actual runtime tests, but there's still common elements.

#21 lore raw
In reply to Grant Likely
Quoted 51 lines · Grant Likely
Quoted 50 lines · Guenter Roeck
Quoted 48 lines · Grant Likely
Quoted 35 lines · Masami Hiramatsu
Quoted 26 lines · Shuah Khan
Quoted 21 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 7 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

Quoted 2 lines

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 6 lines
Quoted 2 lines

Do you have that public yet ? I might want to use that for my -stable sanity tests.

Nothing yet. It's currently hacked into my build environment script. I'm working on getting it into a form usable by others. Ideally I think it should go into the kernel tools/testing directory.

I've been doing the same thing, focused on using Aboriginal. I have
some scripts that I've used for years for abstracting the build/boot/test cycle for cross-platform hardware, that I can share. But my qemu foo is not so good.

I thought my scripts might overlap with the ktest stuff Steve Rostedt is doing, and wanted to chat with him at the summit about what parts might make sense in the kernel tree, and how I might integrate my higher-level scripts to work with ktest.pl.

As for tests, one of the ones I'd really like to see in mainline is a basic size test. I have one based on my scripts I can dust off. It generates the "size" for all config options by doing on vs. off build/boot/size comparisons for each config option.

-- Tim

#22 lore raw
Quoted 9 lines · On Tue, 12 Aug 2014 10:46:40 -0700 Tim Bird <[email protected]> wrote:

I've been doing the same thing, focused on using Aboriginal. I have some scripts that I've used for years for abstracting the build/boot/test cycle for cross-platform hardware, that I can share. But my qemu foo is not so good.

I thought my scripts might overlap with the ktest stuff Steve Rostedt is doing, and wanted to chat with him at the summit about what parts might make sense in the kernel tree, and how I might integrate my higher-level scripts to work with ktest.pl.

Unfortunately, I wont be at KS this year. I have a family commitment that conflicts for the date.

I will be, though, at ELCEU. I'm suspecting that you will be there too. Can it wait till October?

-- Steve

Quoted 7 lines

As for tests, one of the ones I'd really like to see in mainline is a basic size test. I have one based on my scripts I can dust off. It generates the "size" for all config options by doing on vs. off build/boot/size comparisons for each config option.

-- Tim

#23 lore raw
Quoted 14 lines · Steven Rostedt
Quoted 9 lines · On Tue, 12 Aug 2014 10:46:40 -0700 Tim Bird <[email protected]> wrote:

I've been doing the same thing, focused on using Aboriginal. I have some scripts that I've used for years for abstracting the build/boot/test cycle for cross-platform hardware, that I can share. But my qemu foo is not so good.

I thought my scripts might overlap with the ktest stuff Steve Rostedt is doing, and wanted to chat with him at the summit about what parts might make sense in the kernel tree, and how I might integrate my higher-level scripts to work with ktest.pl.

Quoted 5 lines

Unfortunately, I wont be at KS this year. I have a family commitment that conflicts for the date.

I will be, though, at ELCEU. I'm suspecting that you will be there too. Can it wait till October?

If I feel like pushing stuff sooner, we can always chat on LKML over patches. :-) But basicallly, yeah, we can chat at ELCE.
-- Tim

#24 lore raw
In reply to Steven Rostedt
Quoted 14 lines · Steven Rostedt
Quoted 9 lines · On Tue, 12 Aug 2014 10:46:40 -0700 Tim Bird <[email protected]> wrote:

I've been doing the same thing, focused on using Aboriginal. I have some scripts that I've used for years for abstracting the build/boot/test cycle for cross-platform hardware, that I can share. But my qemu foo is not so good.

I thought my scripts might overlap with the ktest stuff Steve Rostedt is doing, and wanted to chat with him at the summit about what parts might make sense in the kernel tree, and how I might integrate my higher-level scripts to work with ktest.pl.

Quoted 5 lines

Unfortunately, I wont be at KS this year. I have a family commitment that conflicts for the date.

I will be, though, at ELCEU. I'm suspecting that you will be there too. Can it wait till October?

I'm sure we'll talk about it more than once. :-)

g.

#25 lore raw
In reply to Guenter Roeck
Quoted 42 lines · Guenter Roeck
Quoted 40 lines · Grant Likely
Quoted 36 lines · Masami Hiramatsu
Quoted 25 lines · Shuah Khan
Quoted 20 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 7 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

Quoted 8 lines
Quoted 6 lines

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Do you have that public yet ? I might want to use that for my -stable sanity tests.

I've been working on a somewhat orthogonal approach: I have a script that boots a kernel from a bunch of files, including, optionally, the host system. So far, it supports x86 and arm, but adding architectures is very simple -- it just needs some hints for how to invoke QEMU.

https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git

One of virtme's explicit goals is to make it easy to boot a kernel, run a script, and shut down. The syntax for that is a bit awkward right now, but it works.

It still needs a rootfs for non-native boots, so your tool could help with that.

--Andy

author
#26 lore raw
In reply to Grant Likely
Quoted 23 lines · Grant Likely
Quoted 22 lines · Masami Hiramatsu
Quoted 21 lines · Shuah Khan
Quoted 20 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I haven't given this too much thought yet. It is at the back on my mind. A couple of issues that will need to be worked to get kselftest working in cross-build env.:

- tests (Makefiles especially) probably need work to pick the right compiler and so on. This would be first step.
- some tests probably aren't suited for all cross env. The goal should be though to keep this set to minimum.

Quoted 7 lines

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Great. Why not? The first step would be get kselftest compiling in cross-builds. I haven't tried it yet, however in a private response to this thread, one developer mentioned that some tests don't pick the correct compiler result of hard-coded gcc instead of using $(CC) and assumptions on libraries. So as we go forward we have to get these things fixed along the way.

-- Shuah

Signature · 4 lines
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
[email protected] | (970) 672-0658
#27 lore raw
In reply to Grant Likely
Quoted 10 lines · Grant Likely
Quoted 2 lines · Masami Hiramatsu

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

If such cross builds are triggered from git pushes akin to how Fengguang's 0-day is already working we are in a positive flow I think.

Yours,
Linus Walleij

#28 lore raw
Quoted 12 lines · Linus Walleij
Quoted 10 lines · Grant Likely
Quoted 2 lines · Masami Hiramatsu

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Guenter

#29 lore raw
Quoted 35 lines · Guenter Roeck
Quoted 19 lines · Linus Walleij
Quoted 9 lines · Grant Likely
Quoted 2 lines · Masami Hiramatsu

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

Quoted 5 lines

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

Quoted 6 lines

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements.

--Andy

#30 lore raw
Quoted 34 lines · On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
Quoted 25 lines · Guenter Roeck
Quoted 9 lines · Linus Walleij
Quoted 5 lines · Grant Likely

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

Quoted 6 lines

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Quoted 4 lines

virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements.

I'm wondering if the kernel test tree might be a good place to keep such virtual machine/QEMU configurations. They should be only about 1 line (or a few lines) per machine, and they would be useful for automated testing. I also think they won't change every kernel release, so it shouldn't lead to the churn problem we had with defconfigs.

Would there be objections to hosting this information in the kernel source tree?
-- Tim

#31 lore raw
Quoted 40 lines · Tim
Quoted 34 lines · On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
Quoted 25 lines · Guenter Roeck
Quoted 9 lines · Linus Walleij
Quoted 5 lines · Grant Likely

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

Quoted 6 lines

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Quoted 4 lines

virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements.

Quoted 8 lines

I'm wondering if the kernel test tree might be a good place to keep such virtual machine/QEMU configurations. They should be only about 1 line (or a few lines) per machine, and they would be useful for automated testing. I also think they won't change every kernel release, so it shouldn't lead to the churn problem we had with defconfigs.

Would there be objections to hosting this information in the kernel source tree?

This is exactly what I'm trying to put together.

g.

#32 lore raw
Quoted 47 lines · Grant Likely
Quoted 46 lines · Tim
Quoted 34 lines · On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
Quoted 25 lines · Guenter Roeck
Quoted 9 lines · Linus Walleij
Quoted 5 lines · Grant Likely

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

Quoted 6 lines

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Quoted 4 lines

virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements.

I'm wondering if the kernel test tree might be a good place to keep such virtual machine/QEMU configurations. They should be only about 1 line (or a few lines) per machine, and they would be useful for automated testing. I also think they won't change every kernel release, so it shouldn't lead to the churn problem we had with defconfigs.

Quoted 4 lines
Quoted 2 lines

Would there be objections to hosting this information in the kernel source tree?

This is exactly what I'm trying to put together.

If this ends up in machine-readable form, it'll be extremely useful. Thanks!

--Andy

#33 lore raw
In reply to Bird, Tim
Quoted 40 lines · Tim
Quoted 34 lines · On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
Quoted 25 lines · Guenter Roeck
Quoted 9 lines · Linus Walleij
Quoted 5 lines · Grant Likely

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

Quoted 6 lines

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Quoted 4 lines

virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements.

Quoted 6 lines

I'm wondering if the kernel test tree might be a good place to keep such virtual machine/QEMU configurations. They should be only about 1 line (or a few lines) per machine, and they would be useful for automated testing. I also think they won't change every kernel release, so it shouldn't lead to the churn problem we had with defconfigs.

You would also need working configurations. There may be a few exceptions, where one of the defconfigs can be used as is, but in most cases at least some minor tweaks are needed to create a kernel that is working with qemu. Or at least that is my experience.

Guenter

#34 lore raw
In reply to Andy Lutomirski
Quoted 8 lines · Andy Lutomirski
Quoted 6 lines · Guenter Roeck

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Quoted 4 lines

virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements.

My aboriginal linux system is designed around the assumption that nobody will ever actually bother to use it. (That's why I ship prebuilt binary tarballs.)

I also write extensive documentation nobody will ever read:

http://landley.net/aboriginal/about.html

Each system-image tarball has a "run-emulator.sh" script that's just the qemu invocation. If you want to know how to launch qemu for the target:
there you go.

The kernel config is using the "miniconfig" technique:

http://landley.net/aboriginal/FAQ.html#dev_miniconfig

You assemble the miniconfig from the target-independent parts:

http://landley.net/hg/aboriginal/file/tip/sources/baseconfig-linux

And then concatenate the target-specific bits from the relevant sources/targets file. ala:

http://landley.net/hg/aboriginal/file/tip/sources/targets/i686#l19

(All the target-specific information is in a single file under sources/targets. There are various patches in sources/patches some of which are target specific. All of them should go upstream, but kernel development is so insular it's just not worth the effort.)

If you don't want to use miniconfig, the expanded configs are in the root-filesystem tarballs (in the src directory).

Quoted 3 lines

--Andy
.

#35 lore raw
In reply to Guenter Roeck

The most interesting email conversations occur when the machine with all my email filters is in the shop...

Quoted 15 lines · Guenter Roeck
Quoted 14 lines · Linus Walleij
Quoted 11 lines · Grant Likely
Quoted 2 lines · Masami Hiramatsu

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 6 lines
Quoted 4 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Yes, and to some degree buildroot.

Aboriginal is intentionally building the simplest system capable of rebuilding itself under itself, with an emphasis on native compiling from there.

(It's sort of in pieces on the floor at the moment as I switch from uClibc to musl, which breaks rather a lot of subtle assumptions and requires everything to be debugged again. Oh well.)

I'd happily make it work with other people's toolchains, but I've never managed to beat a working _native_ toolchain out of buildroot, it cross compiles everything and then the resulting system is not expected to be a development environment. Building on an emulated target system is apparently not most people's idea of fun.

Quoted 2 lines

Rob's attempts to support multiple architectures also shows its limitations.

Some of this is the limitations of the toolchain I'm using, which is the last GPLv2 release of gcc and binutils. I don't mess with GPLv3 code unless it's for work, it is _never_ fun.

I keep hoping http://ellcc.org or similar will approach reasonably functional, but so far that one isn't even using http://lld.llvm.org
instead of binutils, so...

(Also, the llvm build is checking for gcc 4.7 at compile time and refuses to build on anything older. Yes, llvm has incestuous dependencies on gcc.)

Quoted 2 lines

For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.

Laurent Vivier is working on a qemu fork to add support for that, but neither he nor I have had time to put into it in forever.

It's something like git://gitorious.org/qemu-m68k/qemu-m68k branch refs/heads/q800

(My dayjob has nothing to do with anything interesting, I'm paid to work on a a company's fork of a vendor's bsp fork of a really old kernel. For the new board, we just upgraded _to_ a version that's only 4 years old. When they finally ship there will be a nominal compliance tarball put on a website somewhere that nobody will ever look at or care about. I keep meaning to put up a patreon to see if people would be willing to sponsor me to spend all my time on actually _interesting_ stuff like this, but the chances are low enough I haven't bothered.)

Quoted 4 lines

I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

QEMU does not support m68k, qemu supports coldfire which is a nommu subset of m68k. But people have used the aranym emulator to fake an atari machine that _has_ run the root filesystems I've been building. (With a different kernel config, and a largeish patch for aranym devices that since went upstream. But it showed the basics were right.)

The problem with using aranym is my setup expects a certain pile of devices. If /dev/console goes to stdin/stdout than it's easy to script the sucker with tcl/expect and log the output with "tee". If you have a virtual network card you can move the heavy lifting of compilation outside the emulator with distcc hooked up to the cross compiler (_without_ reintroducing the horrible complexity of cross compiling because configure and make and linking and header preprocessing and such all run natively inside the emulator where there's only a single build context.) You need at least 256 megs of ram for 64 bit gcc to function. It's really convenient to have three emulated hard drives (one for the root filesystem, one for writeable space on /home, and one for the source control and build scripts on /mnt. I described that setup in
http://landley.net/aboriginal/control-images and there are example build control images there.)

Now that my initmpfs patches are in, I'm moving the simple-root-filesystem to initmpfs so I can use network mounts instead of filesystem images for all that... but the emulated network card becomes a bit of a bottleneck the. The emulated hard drives tend to fling data around faster. (Also I wanted to use v9fs but the v9fs developers went ABSOLUTELY INSANE and accepted an IBM patch that broke things so badly Red Hat yanked the 9p source code out of their kernel.)

http://sourceforge.net/p/v9fs/mailman/message/31629549/

http://scientificlinuxforum.org/index.php?showtopic=2858

Quoted 7 lines

For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem
of qemu command line parameters, where each target and architecture requires
its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination.

Tell me about it. I collect them, and my system images document them. I was in the process of trying to get Alpha to work when musl got close enough to 1.0 that switching over became a higher priority, and now there's the chicken and egg problem of adding musl support and getting an emulated development enviornment working...

Mostly it's just "dayjob eats my time/energy, I do what I can in the time I have left".

Quoted 1 line

Guenter

Rob

#36 lore raw

Hi Rob,

Quoted 14 lines · Rob Landley

(My dayjob has nothing to do with anything interesting, I'm paid to work on a a company's fork of a vendor's bsp fork of a really old kernel. For the new board, we just upgraded _to_ a version that's only 4 years old. When they finally ship there will be a nominal compliance tarball put on a website somewhere that nobody will ever look at or care about. I keep meaning to put up a patreon to see if people would be willing to sponsor me to spend all my time on actually _interesting_ stuff like this, but the chances are low enough I haven't bothered.)

Quoted 4 lines

I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work.

Quoted 5 lines

QEMU does not support m68k, qemu supports coldfire which is a nommu subset of m68k. But people have used the aranym emulator to fake an atari machine that _has_ run the root filesystems I've been building. (With a different kernel config, and a largeish patch for aranym devices that since went upstream. But it showed the basics were right.)

These days atari_defconfig in upstream should run fine on ARAnyM.

Quoted 3 lines

The problem with using aranym is my setup expects a certain pile of devices. If /dev/console goes to stdin/stdout than it's easy to script the sucker with tcl/expect and log the output with "tee". If you have a

Ah, ARAnyM's nfcon only does stdout, not stdin.

Quoted 2 lines

virtual network card you can move the heavy lifting of compilation outside the emulator with distcc hooked up to the cross compiler

But nfeth works fine.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#37 lore raw
In reply to Linus Walleij
Quoted 12 lines · Linus Walleij
Quoted 10 lines · Grant Likely
Quoted 2 lines · Masami Hiramatsu

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 3 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Hmm, I also consider that the Aboriginal Linux and other busybox-based minimal distro may not have "make" on their rootfs. In that case, we need to make self- executable binaries and install it with testing "ash" scripts so that it can run on busybox. It may be just a technical issue.

Quoted 3 lines

If such cross builds are triggered from git pushes akin to how Fengguang's 0-day is already working we are in a positive flow I think.

Agreed.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#38 lore raw
Quoted 13 lines · Masami Hiramatsu
Quoted 12 lines · Linus Walleij
Quoted 10 lines · Grant Likely
Quoted 2 lines · Masami Hiramatsu

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

Quoted 8 lines
Quoted 3 lines

Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

Hmm, I also consider that the Aboriginal Linux and other busybox-based minimal distro may not have "make" on their rootfs. In that case, we need to make self- executable binaries and install it with testing "ash" scripts so that it can run on busybox. It may be just a technical issue.

Aboriginal Linux has make in the native root filesystem. (Again, last gplv2 version so it's a touch stale now. Back when I did busybox stuff I planned to write a make for that, and now I plan to write one for toybox, but it's a post 1.0 todo item.)

There's also a native-compiler tarball that's just the stuff it adds on top of the simple-root-filesystem to make the development environment. (This includes make and bash 2.05b.)

Now that my initmpfs patches are upstream, I'm refactoring things to put the simple-root-filesystem in initramfs and splice the native-compiler stuff from hda into the $PATH at runtime, in the init script. But I'm two kernel releases behind, and should probably ship a 3.15 version before doing much more development...

Rob

(P.S. Making a stripped down linux from scratch build based on busybox and uclibc is why I got into busybox development in the first place. That's why I wound up doing so much work on it I got the maintainership handed to me for a bit. Now I'm doing it all again in toybox because Android won't use busybox because it's under the GPL.)

#39 lore raw
In reply to Grant Likely
Quoted 29 lines · Grant Likely
Quoted 28 lines · Masami Hiramatsu
Quoted 26 lines · Shuah Khan
Quoted 17 lines

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer

Quoted 6 lines

friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

Quoted 9 lines

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests to start with.

Quoted 3 lines

Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea...

One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target.

Quoted 9 lines
Quoted 2 lines

I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also.

I am interested in this as well, I will reach Chicago only on 18th afternoon (due to flight availability issues). I have been using
https://github.com/autotest/autotest-client-tests for running sanity tests on different kernel versions. It can be driven with sandboxed setup using https://github.com/autotest/virt-test

-aneesh