Skip to contentSkip to conversation

automated linux kernel testing results

ANNOUNCE Martin J. Bligh 4 participants 8 messages Jun 2, 2005 – Jun 7, 2005

lore
Share & tools
lore
#0 lore raw

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system. The config files it uses are linked by the machine names in the column headers.

Andrew, you should be able to see build failures in -mm more easily now ;-) I'll work on some automatic email triggers to you if you want them.

Thanks to all the other IBM people who've worked on the ABAT test system that this stuff relies on - too many to list, but especially Andy, Adam, and Enrique, who have fixed endless bugs, and put up with my incessant bitching about it all not working as it should ;-)

It will do various other bits and pieces ... more profiling, more tests, it'll do patches, etc as well. I don't want to push much volume up to kernel.org (I just copy a small subset of the generated files right now), but there are more runs queued with some fixup patches on top of -mm tree, for example. It did also do -bk and -git nightly builds, they're not in this matrix, but I'll add them back soon.

Andrew, if you want some other test added to the mix, please let me know ... though theoretically we can do multi-machine tests (eg networked), I want to stick with single-machine ones for now. There's a huge pile of tests we intend to add ... but I thought I'd throw the general results mechanism out there for people to look at.

Clicking on the failure ones error codes should take you to somewhere vaguely helpful to diagnose it. Clicking on the job number just below that takes you to the info I'm publishing right now, which should include perf results and profiles, etc. I'll add graphs, etc later, comparing performance across kernels (I have them ... just not automated).

M.

#1 lore raw
Quoted 10 lines · Martin J. Bligh

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system. The config files it uses are linked by the machine names in the column headers.

Nice, very nice, congrats to all involved.

Now, any chance you can do this on the nightly -git snapshots too? :)

And I don't see the -stable releases in there...

thanks,

greg k-h

#2 lore raw

--Greg KH <[email protected]> wrote (on Thursday, June 02, 2005 22:51:57 -0700):

Quoted 12 lines · Martin J. Bligh
Quoted 10 lines

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system. The config files it uses are linked by the machine names in the column headers.

Quoted 5 lines

Nice, very nice, congrats to all involved.

Now, any chance you can do this on the nightly -git snapshots too? :)

And I don't see the -stable releases in there...

It does do both. I just didn't pull in all the historical data, I just repopulated the external set with a brief snapshot (I have way more internally, but it has some crap unpublishable benchmarks in it). If you look at the latest rev, about 3 up it has 2.6.12-rc5-git7, and I think I've fixed it to monitor for new ones automatically now. I guess we'll see if it worked in the morning ;-)

M.

#3 lore raw
Quoted 20 lines · Martin J. Bligh

--Greg KH <[email protected]> wrote (on Thursday, June 02, 2005 22:51:57 -0700):

Quoted 16 lines · Martin J. Bligh
Quoted 10 lines

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system. The config files it uses are linked by the machine names in the column headers.

Nice, very nice, congrats to all involved.

Now, any chance you can do this on the nightly -git snapshots too? :)

Quoted 7 lines
Quoted 1 line

And I don't see the -stable releases in there...

It does do both. I just didn't pull in all the historical data, I just repopulated the external set with a brief snapshot (I have way more internally, but it has some crap unpublishable benchmarks in it). If you look at the latest rev, about 3 up it has 2.6.12-rc5-git7, and I think I've fixed it to monitor for new ones automatically now.

Ah, sorry about that, you are correct, I missed it while seeing all of the -mm releases in there :)

thanks,

greg k-h

#4 lore raw
In reply to Martin J. Bligh
Quoted 10 lines · Martin J. Bligh

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system. The config files it uses are linked by the machine names in the column headers.

Wow. 8]

Some (but not all) green links like [GOOD 4550] lead to 404 land. Is this intended? --
vda

#5 lore raw

--Denis Vlasenko <[email protected]> wrote (on Friday, June 03, 2005 16:12:30 +0300):

Quoted 12 lines · Martin J. Bligh
Quoted 10 lines

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system. The config files it uses are linked by the machine names in the column headers.

Quoted 3 lines

Wow. 8]

Some (but not all) green links like [GOOD 4550] lead to 404 land. Is this intended?

Nope, it's buggered. Some of the new ones are not replicating out properly. Fixing it ...

M.

#6 lore raw
In reply to Martin J. Bligh
Quoted 3 lines

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

...

Quoted 5 lines

Clicking on the failure ones error codes should take you to somewhere vaguely helpful to diagnose it. Clicking on the job number just below that takes you to the info I'm publishing right now, which should include perf results and profiles, etc. I'll add graphs, etc later, comparing performance across kernels (I have them ... just not automated).

OK, there's performance graphs out there for tbench, dbench, kernbench, and reaim now.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/perf_matrix.html

-mm seems to be rather sucky on kernbench recently, I'll have a drill down into why, and try to send out some more data later on. The vertical bars on the kernbench graph are std deviation between sets of runs (a rough error margin).

Suggestions for improvements welcome ...

M.