Nightly regression runs against current bk tree

How about doing a LTP run too with some difference file for new FAILs/BROKs ? That's not strictly a benchmark, but would help catching regressions quickly.

I notice your benchmark mix is very IO heavy, it would be nice to test other aspects of the system too. Perhaps lmbench and reaim compute workload?

It would be nice if we had a new linux-testresults list where such updates could be posted regularly. I don't think it belong on l-k because it would be too noisy. Perhaps such a list could be added to vger. David, what do you think?

-Andi

From: Andi Kleen <[email protected]>
Date: 29 May 2003 23:11:17 +0200

David, what do you think?

Would it have a single poster?

OSDL, Mark's IBM team and possible LTP ?

I assume there will be more once the list exists; automated regression tests seem to be currently in fashion.

-Andi

On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <[email protected]> wrote:

| On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
| >    From: Andi Kleen <[email protected]>
| >    Date: 29 May 2003 23:11:17 +0200
| >    
| >    David, what do you think?
| >    
| > Would it have a single poster?
| 
| OSDL, Mark's IBM team and possible LTP ?
| 
| I assume there will be more once the list exists; automated regression 
| tests seem to be currently in fashion.

If DaveM doesn't want to do it, I think that we can do it. (I say without checking.... :)

--
~Randy

From: "Randy.Dunlap" <[email protected]>
Date: Thu, 29 May 2003 14:38:20 -0700

On Thu, 29 May 2003 23:29:29 +0200 Andi Kleen <[email protected]> wrote:

   | On Thu, May 29, 2003 at 02:25:15PM -0700, David S. Miller wrote:
   | > Would it have a single poster?
   | 
   | OSDL, Mark's IBM team and possible LTP ?
   | 
   | I assume there will be more once the list exists; automated regression 
   | tests seem to be currently in fashion.

If DaveM doesn't want to do it, I think that we can do it. (I say without checking.... :)

Please do :-)

The issue is that I'm easier about adding a new list if I can restrict the poster list.

OSDL has a mail list that is used to discuss the stability of the linux kernel. This would be a perfect list to use for posting test results. The list name is [email protected]. It is a public list administered by OSDL. To subscribe: http://www.osdl.org/mailman/listinfo/linstab

Craig Thomas
[email protected]

That's fairly obscure (Nobody knew of it before). Perhaps a well publicized list on vger would be better.

-Andi

Perhaps - though we can publicize any new list.
We're content to leave the decision to DaveM and the list team - if they don't want the extra work, we're always glad to help. cliffw
OSDL

From: Andi Kleen <[email protected]>
Date: Fri, 30 May 2003 00:05:40 +0200

That's fairly obscure (Nobody knew of it before). Perhaps a well publicized list on vger would be better.

I don't see why the OSDL list cannot be widely publicized and the vger variant could. Don't be rediculious Andi :-)

[...] Would you be willing to change your scripts to report any new results to this new list?

-Andi

I would be very interested in this if it leads to more people *looking* at the tests results.

Automating this stuff is the easy part - getting intelligence out of the results is harder.
The more eyeballs we can get to look, the easier this gets.

If a new list, or better use of an old list will help, i'll change whatever is necessary.
If people like the Web, but don't like our paper layout, i'll change that too.

cliffw

As

The linux stabilization web page uses results from the STP runs I mentioned (40+ per day). The STP emails results summaries after test runs so we could easily redirect the results to this new list.

-Nathan Dabney

We have not automated the posting process ... yet, and will be happy to post whereever is acceptable.

Mark

Our intention was to have one "main" poster and another person for backup.

Mark

The OSDL has a serious amount of automated testing we could point the results of to a separate list if it is created.

Right now we avoid pointing that sort of thing to l-k because it would drive people nuts. On average we complete 40+ tests a day.

With all the testing efforts going on, a central list to post and analyze results would be good. People interested in helping out could easily work with testers to look for trends and help with root cause analysis.

When results are found to contain significant data, we can always notify l-k.

-Nathan Dabney

Easy of viewing should be considered. We have tried to show a high level summary that allows the users to quickly, looking in one place, determine if any significant data is found. When the users seems something of interest, they only need follow the links to see the details. Its shouldn't be necessary for users to sift through one email for each test. If finding signficant data was easier, and I think it can be made easier, users would look at it themselves and there wouldn't be the need to have to notify l-k.

I'm not trying to be competetive here. I just think results and comparisons can be made that covers a large amount of tests in a single page or note. One note per day does not IMHO seem like too much. That note can always be the "tip of the iceberg" pointing to many other things. Thus those not interested can simply skip that note.

Mark

I'm under the impression that LTP and other test efforts seemed to focus more on functional evaluation, which is fine. We are trying to focus purely on the performance differences seen from day to day.

Your correct. We're just getting started with this effort and we used this mix to get things going. Once ppl are happy with the presentation of data, we planned to add more tests to provide a more balanced mix. But since you asked, we have added lmbench to our -bk3 regression run. :)

Mark

Mark, it would be nice to get a graph of the combined results for each test. Something like:

tiobench sequential write rate

  |                                  +++++++++++++++++      + = -mm-ext3
M |        ++++++++++++++++++++++++++*****************      * = linus-ext3
B | +++++++*****************   ######                       # = -ac-ext3
/ |                                                         . = -mm-XFS
s |                                                         = = -ac-XFS
  |                         *********                       etc
  |

+----------------------------------------------------
date

This allows at-a-glance trends for each group of tests and (as in the example above you could see easily when a performance bug was added and fixed in -ac before in the linus kernel, for example). Probably having all of the comparable results on the same page, or even in the same graph results is a win.

Bonus points if you can click on a spot in the graph and get the results page for that date/test ;-).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/