Skip to contentSkip to conversation

Giving developers clue how many testers verified certain kernel version

DISCUSSION Blaisorblade 10 participants 16 messages Jul 23, 2005

lore
Share & tools
lore
#0 lore raw
Quoted 10 lines · Adrian Bunk <bunk <at> stusta.de> writes:
Quoted 6 lines · Alejandro Bonilla

How do we know that something is OK or wrong? just by the fact that
it works or not, it doesn't mean like is OK.

There has to be a process for any user to be able to verify and study a problem. We don't have that yet.

If the user doesn't notice the difference then there's no problem for him.

Some performance regressions aren't easily noticeable without benchmarks... and we've had people claiming unnoticed regressions since 2.6.2
(http://kerneltrap.org/node/4940)

Quoted 10 lines

If there's a problem the user notices, then the process is to send an email to linux-kernel and/or open a bug in the kernel Bugzilla and follow the "please send the output of foo" and "please test patch bar" instructions.

What comes nearest to what you are talking about is that you run LTP and/or various benchmarks against every -git and every -mm kernel and report regressions. But this is sinply a task someone could do (and I don't know how much of it is already done e.g. at OSDL), and not something every user could contribute to.

Forgot drivers testing? That is where most of the bugs are hidden, and where wide user testing is definitely needed because of the various hardware bugs and different configurations existing in real world.

IMHO, I think that publishing statistics about kernel patches downloads would be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was even talked about at Kernel Summit (or at least I thought of it there), but I've not understood if this is going to happen.

Signature · 3 lines
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade
#1 lore raw
Quoted 4 lines · Blaisorblade

IMHO, I think that publishing statistics about kernel patches downloads would be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was even talked about at Kernel Summit (or at least I thought of it there), but I've not understood if this is going to happen.

remember that most downloads will be from mirrors, and they don't get stats from them.

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#2 lore raw
Quoted 8 lines · David Lang
Quoted 4 lines · Blaisorblade

IMHO, I think that publishing statistics about kernel patches downloads would be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was even talked about at Kernel Summit (or at least I thought of it there), but I've not understood if this is going to happen.

Quoted 4 lines

remember that most downloads will be from mirrors, and they don't get stats from them.

David Lang

That, plus there is http+ftp+rsync (not to mention git downloads, etc.) and the noise caused by other sites mirroring *us*.

-hpa

#3 lore raw
In reply to Blaisorblade
Quoted 15 lines · Blaisorblade
Quoted 9 lines · Adrian Bunk <bunk <at> stusta.de> writes:
Quoted 6 lines · Alejandro Bonilla

How do we know that something is OK or wrong? just by the fact that
it works or not, it doesn't mean like is OK.

There has to be a process for any user to be able to verify and study a problem. We don't have that yet.

Quoted 6 lines
Quoted 3 lines

If the user doesn't notice the difference then there's no problem for him.

Some performance regressions aren't easily noticeable without benchmarks... and we've had people claiming unnoticed regressions since 2.6.2 (http://kerneltrap.org/node/4940)

I will get flames for this, but my laptop boots faster and sometimes responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact for me. IBM T42.

Quoted 3 lines
Quoted 3 lines

If there's a problem the user notices, then the process is to send an email to linux-kernel and/or open a bug in the kernel Bugzilla and follow the "please send the output of foo" and "please test patch bar" instructions.

The thing is, I might not be able to know there *are* issues. I most just notice that something is strange. And then wait for a new kernel version because i might think it is something silly.

Quoted 10 lines
Quoted 6 lines

What comes nearest to what you are talking about is that you run LTP and/or various benchmarks against every -git and every -mm kernel and report regressions. But this is sinply a task someone could do (and I don't know how much of it is already done e.g. at OSDL), and not something every user could contribute to.

Forgot drivers testing? That is where most of the bugs are hidden, and where wide user testing is definitely needed because of the various hardware bugs and different configurations existing in real world.

This is my opinion too. If someone could do a simple script or benchmarking file, then users would be able to report most common important differences from previous kernel versions on their systems.

i.e. i would run the script that checks the write speed, CPU, latencys, and I don't know how many more tests and then compare it with the results that were with the previous git or full kernel release. Sometimes the users don't even know the commands to benchmark this parts of the systems. I don't know them.

Quoted 3 lines

IMHO, I think that publishing statistics about kernel patches downloads would be a very Good Thing(tm) to do. Peter, what's your opinion? I think that was even talked about at Kernel Summit (or at least I thought of it there), but I've not understood if this is going to happen.

What can we do here? Can we probably create a project like the janitors so that we can report this kind of thing? Should we report here? How can we make a script to really benchmark the system and then say, since this guy sent a patch for the Pentium M CPU's, things are running slower? Or my SCSI drive is running slower since rc2, but not with rc1.

At least if the user notices this kind of things, then one will be able to google for patches for your controller for the last weeks and see if someone screwed up with a change they sent to the kernel.

In other words, kernel testing is not really easy for normal users, it can only really be benchmarked by the one that knows... Which are many, but not everyone.

And I really want to give my 2 cent on this.

.Alejandro

#4 lore raw
Quoted 3 lines · Alejandro Bonilla

I will get flames for this, but my laptop boots faster and sometimes responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact for me. IBM T42.

Sorry dude, but there's just no way that any automated process can catch these.

You will have to provide a detailed bug report (with numbers) like everyone else so we can fix it. "Waiting for it to fix itself" is the WORST thing you can do.

If you find a regression vs. an earlier kernel, please assume that you're the ONLY one to notice it and respond accordingly.

Lee

#5 lore raw
Quoted 11 lines · Lee Revell
Quoted 3 lines · Alejandro Bonilla

I will get flames for this, but my laptop boots faster and sometimes responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact for me. IBM T42.

Sorry dude, but there's just no way that any automated process can catch these.

I'm not looking for an automated process for this. But for all in general, when moving from 2.6.11 to 2.6.12 or from any version to another. (At least in the same kernel branch)

Quoted 5 lines

You will have to provide a detailed bug report (with numbers) like everyone else so we can fix it. "Waiting for it to fix itself" is the WORST thing you can do.

I never do this, believe me, but I could if I don't really see a problem. But there could really be one behind.

Quoted 4 lines

If you find a regression vs. an earlier kernel, please assume that you're the ONLY one to notice it and respond accordingly.

OK, I will, but I first of all need to learn how to tell if benchmarks are better or worse.

Quoted 4 lines

Lee

.Alejandro

#6 lore raw
Quoted 2 lines · Alejandro Bonilla

OK, I will, but I first of all need to learn how to tell if benchmarks are better or worse.

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions. It certainly has confirmed what we already knew re: SCHED_FIFO performance, if we extend that to SCHED_OTHER which is a more interesting problem then there's serious potential for improvement. AFAIK no one has posted any 2.4 vs 2.6 interbench results yet...

I suspect a lot of the boot time issue is due to userspace. But, it should be trivial to benchmark this one, just use the TSC or whatever to measure the time from first kernel entry to execing init().

Lee

#7 lore raw
Quoted 3 lines · Lee Revell

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions.

I doubt that _any_ of the regressions that are user-visible are scheduler-related. They all tend to be disk IO issues (bad scheduling or just plain bad drivers), and then sometimes just VM misbehaviour.

People are looking at all these RT patches, when the thing is that most nobody will ever be able to tell the difference between 10us and 1ms latencies unless it causes a skip in audio.

Linus

#8 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Lee Revell

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions.

Quoted 7 lines

I doubt that _any_ of the regressions that are user-visible are scheduler-related. They all tend to be disk IO issues (bad scheduling or just plain bad drivers), and then sometimes just VM misbehaviour.

People are looking at all these RT patches, when the thing is that most nobody will ever be able to tell the difference between 10us and 1ms latencies unless it causes a skip in audio.

I agree re: the RT patches, but what makes Con's benchmark useful is that it also tests interactivity (measuring in msecs vs. usecs) with everything running SCHED_NORMAL, which is a much better approximation of a desktop load. And the numbers do go well up into the range where people would notice, tens and hundreds of ms.

Lee

#9 lore raw
Quoted 13 lines · Lee Revell
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Lee Revell

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions.

I doubt that _any_ of the regressions that are user-visible are scheduler-related. They all tend to be disk IO issues (bad scheduling or just plain bad drivers), and then sometimes just VM misbehaviour.

People are looking at all these RT patches, when the thing is that most nobody will ever be able to tell the difference between 10us and 1ms latencies unless it causes a skip in audio.

Quoted 5 lines

I agree re: the RT patches, but what makes Con's benchmark useful is that it also tests interactivity (measuring in msecs vs. usecs) with everything running SCHED_NORMAL, which is a much better approximation of a desktop load. And the numbers do go well up into the range where people would notice, tens and hundreds of ms.

Indeed, and the purpose of the benchmark is to quantify something rather than leave it to subjective feeling. Fortunately if I was to quantify the current kernel's situation I would say everything is fine.

Cheers,
Con

#10 lore raw
Quoted 3 lines · Con Kolivas

Indeed, and the purpose of the benchmark is to quantify something rather than leave it to subjective feeling. Fortunately if I was to quantify the current kernel's situation I would say everything is fine.

Agreed. Unfortunately everything in userspace is not fine. I think a lot of these interactivity problems are due to X and broken/bloated apps.

Lee

#11 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Lee Revell

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions.

Quoted 9 lines

I doubt that _any_ of the regressions that are user-visible are scheduler-related. They all tend to be disk IO issues (bad scheduling or just plain bad drivers), and then sometimes just VM misbehaviour.

People are looking at all these RT patches, when the thing is that most nobody will ever be able to tell the difference between 10us and 1ms latencies unless it causes a skip in audio.

True, and I just couldn't agree more with Lee that lots of the delays that one looks at is because of user space. Still, I have some doubt on how faster 2.6 is sometimes, where 2.4 is faster in other things.

i.e. As my newbie view, I can see 2.6 running faster in X, Compiling and stuff, but I see 2.4 working much faster when running commands, response and interaction in the console. But then again, this could be only me...

Quoted 4 lines

Linus

.Alejandro

#12 lore raw
In reply to Lee Revell
Quoted 7 lines · Lee Revell
Quoted 3 lines · Alejandro Bonilla

OK, I will, but I first of all need to learn how to tell if benchmarks are better or worse.

Quoted 8 lines

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions. It certainly has confirmed what we already knew re: SCHED_FIFO performance, if we extend that to SCHED_OTHER which is a more interesting problem then there's serious potential for improvement. AFAIK no one has posted any 2.4 vs 2.6 interbench results yet...

I will give it a try.

Quoted 5 lines

I suspect a lot of the boot time issue is due to userspace. But, it should be trivial to benchmark this one, just use the TSC or whatever to measure the time from first kernel entry to execing init().

You got it! As a laptop user, I think it just takes too much more. I think it is maybe hotplugs fault with the kernel? I don't know how much is done by the kernel or userspace but it definitely takes longer.

I could do some sort of benchmarks, but believe me, I hate to say this, but I use 2.6 because of much more power managements features in it. Else I like 2.4 a lot more. Is like, the feels is sharper. Sometimes when I got into a tty1, it takes some time after I put my username in to prompt me for a password. This does not occur when I boot with 2.4.27. Strange huh?

I don't want to be an ass and say that 2.4 is better, instead I want to help and let determine why is it that I feel 2.6 slower.

.Alejandro

Quoted 5 lines

Lee

#13 lore raw

RE: Giving developers clue how many testers verifiedcertain kernel version

In reply to Lee Revell

Lee Revell wrote: {

Quoted 2 lines · Alejandro Bonilla

OK, I will, but I first of all need to learn how to tell if benchmarks are better or worse.

Con's interactivity benchmark looks quite promising for finding scheduler related interactivity regressions.
}

Scheduler performance does not imply net system performance.

In fact, a well tuned scheduler hides absolute performance-related issues!

--Al

#14 lore raw
In reply to Alejandro Bonilla
Quoted 8 lines · Alejandro Bonilla
Quoted 6 lines · Lee Revell
Quoted 3 lines · Alejandro Bonilla

I will get flames for this, but my laptop boots faster and sometimes responds faster in 2.4.27 than in 2.6.12. Sorry, but this is the fact for me. IBM T42.

Quoted 7 lines
Quoted 3 lines

Sorry dude, but there's just no way that any automated process can catch these.

I'm not looking for an automated process for this. But for all in general, when moving from 2.6.11 to 2.6.12 or from any version to another. (At least in the same kernel branch)
...

You send:
- a problem description X
- tell that the last working kernel was Y
- tell that it is broken in kernel Z

The probability of any kernel developer being interested in your problem increases:
- the better the description X is
- the nearer versions Y and Z are together
- the more recent version Y is

Ideally, you are able to say that patch A in the latest -mm kernel broke it.

It's perfectly OK to send a description X that says:
- with version Y and the following workload B, everything is working perfectly
- with version Z and the same workload B, XMMS is stuttering

If any kernel developer is interested in your bug report, he will tell you which data might be interesting for debugging the problem.

The problem is that debugging a problem often requires knowledge about possible causes and changes between versions Y and Z in this area. Even a kernel developer who perfectly knows one part of the kernel might not be able to debug a problem in a completely different area of the kernel.

Quoted 1 line

.Alejandro

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#15 lore raw
In reply to Blaisorblade

I gmane.linux.kernel, skrev Blaisorblade:

Quoted 3 lines

Forgot drivers testing? That is where most of the bugs are hidden, and where wide user testing is definitely needed because of the various hardware bugs and different configurations existing in real world.

A way that could raise the testing upon a particular kernel, would be to provide; (debian example follows):
... example ..
An apt-repository with the newest tagged kernel build modular for the architecture.

Just drop all tagged kernels in a common repository that the users can follow, then I'd be happy to test a new kernel on every reboot on my system. I'd probably still would respond if anything was broken in the new kernel..

Then it wouldn't be: "try this patch and see if that solves anything" but do:

apt-get install kernel-image-386-torvalds-linux-2.6-v2.6.13-rc3

(automatically build from the "torvalds/linux-2.6"-branch with tag "v2.6.13-rc3" using a modular kernel-configuration similar to the one used in the stock debian kernels.

Then I find and report something and "Pavel Machek" releases a "try-fix", by tagging a branch ind a tree and tells me to try kernel-image-386-pavel-good-2.6-v2.6.13-rc3
instead.

(and variations.. acip/no-acip smp, etc. etc. )

... example end ..

It would be quite a lot central kernel-building, but as far as I can see, it can be fully automated.

It would defininately lower the barrier for being able to paticipate in testing, but I am not the one to decide if that would be a desirable goal? Or for that matter, worth the work.

Jesper

Signature · 1 line
./Jesper Krogh, [email protected], Jabber ID: [email protected]