Linux 2.6.21

...
So it's been over two and a half months, and while it's certainly not the longest release cycle ever, it still dragged out a bit longer than I'd have hoped for and it should have. As usual, I'd like to thank Adrian (and the people who jumped on the entries Adrian had) for keeping everybody on their toes with the regression list - there's a few entries there still, but it got to the point where we didn't even know if they were real regressions, and delaying things further just wasn't going to help.
...

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

Quoted 2 lines · Adrian Bunk

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

I sure hope you don't do this.

Tracking these is tough, and I think you are doing a great job with it.

No release will have no regressions, there's just too many different combinations of hardware and sometimes people don't have the time to test to see if their original report is even fixed or not.

And some of them will get fixed with patches coming in the next kernel release, which will then be tracked down and added to the -stable releases.

So if you can, please keep it up, if you think it's a thankless job, here's my hearty thanks for doing this work. It's really needed and I really appreciate it.

Quoted 9 lines · Linus Torvalds

...
So it's been over two and a half months, and while it's certainly not the longest release cycle ever, it still dragged out a bit longer than I'd have hoped for and it should have. As usual, I'd like to thank Adrian (and the people who jumped on the entries Adrian had) for keeping everybody on their toes with the regression list - there's a few entries there still, but it got to the point where we didn't even know if they were real regressions, and delaying things further just wasn't going to help.
...

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release:
14

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release that were first reported in March or earlier:
8

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release with patches available at the time of the 2.6.21 release [1]:
3

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

We have an astonishing amount of -rc testers, but obviously not the developer manpower for handling them.

If we would take "no regressions" seriously, it might take 4 or 5 months between releases due to the lack of developer manpower for handling regressions. But that should be considered OK if avoiding regressions was considered more important than getting as quick as possible to the next two week regression-merge window.

But releasing with so many known regressions is insulting for the many people who spent their time testing -rc kernels.

Quoted 1 line · Linus Torvalds

...

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

We have an astonishing amount of -rc testers, but obviously not the developer manpower for handling them.

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release:
14

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release with patches available at the time of the 2.6.21 release [1]:
3

Quoted 4 lines

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release:
14

I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found to be inapplicable.

Quoted 4 lines

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release with patches available at the time of the 2.6.21 release [1]:
3

The -stable team can presumably take care of these in 2.6.21.1, right? That leaves 10 that need developer attention.

 John Stultz seems to be taking care of 3 of them.

 Oliver Neukum has 1.

Quoted 3 lines · Adrian Bunk

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release:
14

I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found to be inapplicable.

Quoted 4 lines

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release with patches available at the time of the 2.6.21 release [1]:
3

The -stable team can presumably take care of these in 2.6.21.1, right?

That leaves 10 that need developer attention.

John Stultz seems to be taking care of 3 of them.

Oliver Neukum has 1.

2 are particular drivers (ali_pata and rtl8139, according to the reports).

2 seem to be ACPI-related; at least one has a candidate patch now.

1 seems to be an ALSA problem.

1 is STD and being debugged.

It looks like all of the known regressions are being worked on, and getting fixes in for them is -stable material at this point. Furthermore, it doesn't look to me like anyone who is needed for dealing with these regressions is trying to get stuff into the 2.6.22 merge window.
...

-Daniel

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

Quoted 9 lines · Linus Torvalds

...
So it's been over two and a half months, and while it's certainly not the longest release cycle ever, it still dragged out a bit longer than I'd have hoped for and it should have. As usual, I'd like to thank Adrian (and the people who jumped on the entries Adrian had) for keeping everybody on their toes with the regression list - there's a few entries there still, but it got to the point where we didn't even know if they were real regressions, and delaying things further just wasn't going to help.
...

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release:
14

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release that were first reported in March or earlier:
8

Number of different known regressions compared to 2.6.20 at the time of the 2.6.21 release with patches available at the time of the 2.6.21 release [1]:
3

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.

We have an astonishing amount of -rc testers, but obviously not the developer manpower for handling them.

If we would take "no regressions" seriously, it might take 4 or 5 months between releases due to the lack of developer manpower for handling regressions. But that should be considered OK if avoiding regressions was considered more important than getting as quick as possible to the next two week regression-merge window.

But releasing with so many known regressions is insulting for the many people who spent their time testing -rc kernels.

There is a conflict between Linus trying to release kernels every 2 months and releasing with few regressions.

Trying to avoid regressions might in the worst case result in an -rc12 and 4 months between releases. If the focus is on avoiding regressions this has to be accepted.

And a serious delay of the next regression-merge window due to unfixed regressions might even have the positive side effect of more developers becoming interested in fixing the current regressions for getting their shiny new regressions^Wfeatures faster into Linus' tree.

0 regressions is never realistic (especially since many regressions might not be reported during -rc), but IMHO we could do much better than what happened in 2.6.20 and 2.6.21.

I'm not satisfied with the result, and the world won't stop turning when I'm not tracking 2.6.22-rc regressions.

Quoted 2 lines · Adrian Bunk

There is a conflict between Linus trying to release kernels every 2 months and releasing with few regressions.

No.

Regressions _increase_ with longer release cycles. They don't get fewer.

The fact is, we have a -stable series for a reason.

Problem is, not enough developers pay attention to the -stable series. Adrian, maybe you could shift your attention there and stop trying to track the bleeding edge?

From my humble POV, it's a problem that all this discussion was generated

From my humble POV, it's a problem that all this discussion was generated on what Adrian does or stop doing. Apparently, unless Adrian posts his list of know regressions, most of the people doesn't look at the bugzilla at all. Maybe it'd be useful to create a per-release bug tracker in the bugzilla or collect them into one of the a kernel.org's wiki, to make easier to follow the current state of all the "important" regressions.

I bet that's true even of a lot of people who are more "web oriented" than I am. They may look at webpages, but getting notified by email is still the wakeup call. There's a difference between "active and directed pushing

Bugzilla sucks quite a lot at email, but you can answer emails and they get into the bugzilla database; and there're two mailing lists (listed in Documentation/HOWTO) that send notifications about every new bug added/modified- I know it's not the perfect email interface every hacker wants, but it's better than nothing.

I suggested some time ago that it'd be useful to send every new bug notification from bugme-new to the LKML (and/or other lists).

I can understand Adrian's resign. Bugzilla is crap, but there're users reporting bugs there and willing to cooperate to fix them, and they're not getting listened.

There're even a few description of patches (ie: "line
6 in foo.c is wrong and it breaks our testing, it should read like this:") that have been sitting there for *years* and not getting merged.

So I, or anyone else, could try to do Adrian's job. But if Adrian (a guy that sends patches to make global functions static 8) got tired of doing that job, I suspect that I, or anyone else would also got tired of it even sooner.

There're other big projects with probably more bug reports than linux, they don't work this way, and they look more succesful in their bug handling.

I think it could be more interesting if part of the job was doing the tools. Tools *are* important. Most of my actual _development_ for the last couple of years has been on "git", not the kernel, but I think I was more productive that way, so I don't think that's wasted time at all.

So yes, automation would be a good idea, but I don't think bugzilla is it.

So I really don't think you can compare to "other projects". They simply don't have these issues.

The thing is, these reports MUST NOT go to "everybody". If they do, that is actually *worse* than nothing, because people will just ignore them entirely, since they aren't "directed".

The emails need to be directed to the appropriate parties, not go to everybody. There is nobody who is interested in seeing all regressions, except perhaps me and Andrew. Most *real* developers (as opposed to people like me, who are integrators, not "real developers") want to be notified about problems in *their* area, and if it's just automation that sends out everything, it just dilutes the value of the thing, to the point where people will ignore it even for the cases when they happen to be related to what they do.

Let me put it another way: I would never use a source control system that forces me to look at my 22,000 files one at a time. I think such a system is fundamentally broken, because it makes it impossible to get the big picture ("what changed in the last week" kind of thing). The same is true of bugzilla: if you *know* which bug you're looking at, it's good. For anythign else, it's almost worse than useless, exactly because there is no way to get an overview

(I've said this before, but I'll say it again: one thing that would already make bugzilla better is to just always drop any bug reports that are more than a week old and haven't been touched. It wouldn't need *much* touching, but if a reporter cannot be bothered to say "still true with current snapshot" once a week, then it shouldn't be seen as being somehow up to those scare resources we call "developers" to have to go through it).

So there are probably things that bugzilla could do to become more useful, but I don't see that happening. We'd need either a smarter/better bugzilla, or somebody who actually turns noise into real information. Adrian did that (although in fairness to others, other people definitely do it too. Dave Jones, for example. Very useful).

Yes, human involvement from someone with half a brain would be better. Andrew does a lot of that. Not a particularly good use of talent really. but still.

As Andrew has pointed out before though - even though he forwards the bugs, nobody does anything with it. The sad truth seems to be that people have very little interest in fixing bugs when they are reported - it's not sexy, I guess.

I'm still unconvinced the users or the tool are the problem, but if it makes you happier, we can do that.

On Saturday April 28, [email protected] wrote:
...

Quoted 4 lines

As Andrew has pointed out before though - even though he forwards the bugs, nobody does anything with it. The sad truth seems to be that people have very little interest in fixing bugs when they are reported - it's not sexy, I guess.

Not sexy, and also not at all easy. A lot of the interesting bugs
seem to be subtle interactions between separate parts of the kernel - one part making an assumption or exhibiting a behaviour that the other part didn't expect. And we all know that writing bug^Wcode is easier than removing bugs. I can spend hour and hours reading through code trying to get the big picture, and end up finding a one-line change that then needs documenting, testing and external review. It's not easy.

Quoted 2 lines

I'm still unconvinced the users or the tool are the problem, but if it makes you happier, we can do that.

No, they aren't the problem. Bugs are the problem. But they might be a more effective part of the solution.

My perception of the kernel bugzilla is that visibility is very low.

I think there is value in weekly reminders, and I wouldn't mind seeing a weekly Email on linux-kernel with something like a list of open bugs that have not seen any activity in between 1 and 2 weeks. It might get someone out-of-area interested, and might be noticed by someone who thinks they are in-area and get them wondering why they didn't find out when the bug was first reported.

NeilBrown

Quoted 7 lines · Neil Brown

I think there is value in weekly reminders, and I wouldn't mind seeing a weekly Email on linux-kernel with something like a list of open bugs that have not seen any activity in between 1 and 2 weeks. It might get someone out-of-area interested, and might be noticed by someone who thinks they are in-area and get them wondering why they didn't find out when the bug was first reported.

The 100 kB email limit has to be lifted for this...

More seriously, there are > 1000 open bugs in the kernel Bugzilla without any activity during the last 2 weeks.

Quoted 22 lines · Neil Brown

On Saturday April 28, [email protected] wrote:
...

Quoted 4 lines

As Andrew has pointed out before though - even though he forwards the bugs, nobody does anything with it. The sad truth seems to be that people have very little interest in fixing bugs when they are reported - it's not sexy, I guess.

Not sexy, and also not at all easy. A lot of the interesting bugs
seem to be subtle interactions between separate parts of the kernel - one part making an assumption or exhibiting a behaviour that the other part didn't expect. And we all know that writing bug^Wcode is easier than removing bugs. I can spend hour and hours reading through code trying to get the big picture, and end up finding a one-line change that then needs documenting, testing and external review. It's not easy.

Quoted 2 lines

I'm still unconvinced the users or the tool are the problem, but if it makes you happier, we can do that.

No, they aren't the problem. Bugs are the problem. But they might be a more effective part of the solution.

Quoted 8 lines

My perception of the kernel bugzilla is that visibility is very low.

I think there is value in weekly reminders, and I wouldn't mind seeing a weekly Email on linux-kernel with something like a list of open bugs that have not seen any activity in between 1 and 2 weeks. It might get someone out-of-area interested, and might be noticed by someone who thinks they are in-area and get them wondering why they didn't find out when the bug was first reported.

The 100 kB email limit has to be lifted for this...

More seriously, there are > 1000 open bugs in the kernel Bugzilla without any activity during the last 2 weeks.

The problem is usually either "Not sexy, and also not at all easy." or "no maintainer".

I think more than half a brain is needed to do this well. You need a reasonable understanding of how all the bits of the kernel work together so that you have a good chance of sending the bug in the right direction.

Quoted 4 lines · On Saturday April 28, [email protected] wrote:

Yes, human involvement from someone with half a brain would be better. Andrew does a lot of that. Not a particularly good use of talent really. but still.

I think more than half a brain is needed to do this well. You need a reasonable understanding of how all the bits of the kernel work together so that you have a good chance of sending the bug in the right direction. You need a good understanding of the kernel community and various sub communities so that you know who might be both able and willing to deal with the bug. And it wouldn't hurt to have a good over-view of the current 'hot' areas of the kernel so you know if it is really worth suggesting "try with the latest -mm" or not. And you need good people skills.

So I think you really need a lot of up-to-date knowledge to do this well. Because of Andrew's position as a funnel, he has a lot of that knowledge.

It would be really nice if he had some help though.

And I
really think that would mean finding someone in the community who would rather be coding (and currently are) and convincing them that there is a higher calling for them. Finding someone out side or on the edge of the community is less likely to be effective.

Go to http://bugzilla.kernel.org. Hit query. Find the box that says "Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.

74 bugs found.

Not hard to do.

Quoted 7 lines · Martin J. Bligh

Go to http://bugzilla.kernel.org. Hit query. Find the box that says "Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.

74 bugs found.

Not hard to do.

And what part of the "directed" did you miss?

Do you really expect me to go there every day to look at all bugs? That's nbot a bug tracker. That's just a noise-maker.

It needs to be email, not some "mouse around for 30 seconds and type thing", and it needs to be *directed*. Preferably with somebody who actually did some manual scanning over it and spent a few minutes just looking at whether it looks like a worthy bug.

In other words: we shouldn't have all developers wasting time doing this.

It would be much better to have _one_ person (or a group of people) doing it,

...
(I've said this before, but I'll say it again: one thing that would already make bugzilla better is to just always drop any bug reports that are more than a week old and haven't been touched. It wouldn't need *much* touching, but if a reporter cannot be bothered to say "still true with current snapshot" once a week, then it shouldn't be seen as being somehow up to those scare resources we call "developers" to have to go through it).
...

Limis

We are already quite good at ignoring bug reports that come through linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more than 1600 open bugs because this tells how bad we are at handling bugs. How many thousand bug reports have been ignored during the same time on linux-kernel?

Quoted 5 lines · Adrian Bunk

We are already quite good at ignoring bug reports that come through linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more than 1600 open bugs because this tells how bad we are at handling bugs. How many thousand bug reports have been ignored during the same time on linux-kernel?

However, look at this bug:

http://bugme.osdl.org/show_bug.cgi?id=7760

It's outside my knowledge to be able to fix for various reasons:

Quoted 5 lines · Adrian Bunk

We are already quite good at ignoring bug reports that come through linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more than 1600 open bugs because this tells how bad we are at handling bugs. How many thousand bug reports have been ignored during the same time on linux-kernel?

However, look at this bug:

http://bugme.osdl.org/show_bug.cgi?id=7760

It's outside my knowledge to be able to fix for various reasons:
...
I'm personally very tempted to close it as "won't fix" (I wish there was a "can't fix" category.)
...

I'm no longer serial maintainer. Bug IDs after about 7000 reflect bugs submitted since I've resigned my serial maintainership, and therefore I've ignored them.
...

It's far easier to ignore bug reports in bugzilla
than it is to get categories reassigned (to whom? - dunno) or even deleted (if no one steps up presumably that's what needs to happen?)
...

Russell King

That's exactly where Linus' "drop any bug reports that are more than a week old" suggestion is completely flawed - no matter what the submitter does, how often he tests latest kernels, noone will help him.

Quoted 4 lines · Adrian Bunk

That's exactly where Linus' "drop any bug reports that are more than a week old" suggestion is completely flawed - no matter what the submitter does, how often he tests latest kernels, noone will help him.

You talk, but what do you actually *suggest*?

Talk is cheap. You use to do the walk too, but you've already said that you're not interested in that any more. So excuse me if I'm not impressed.

The thing is, bugzilla is totally broken because it's designed to help track bugs, but it's *not* designed to actually handle the much harder problem, which is to actually get the *right* developers to be aware of the *right* bugs!

Quoted 5 lines · Linus Torvalds

The thing is, bugzilla is totally broken because it's designed to help track bugs, but it's *not* designed to actually handle the much harder problem, which is to actually get the *right* developers to be aware of the *right* bugs!

This means we need people who figure out who to assign bugs too. Aka bugmasters.

In theory it could be nearly automated. Figure out what files related to the bug and assign to the last 5 people who submitted patches for them and/or signed off.

BTW one big problem in our current bugzilla is that a lot of people cannot reassign bugs they don't own. I sometimes see bugs that I don't own bug I know who is responsible, but bugzilla doesn't allow me to do it.

So I think what would help:

- Ask more people to just categorize and reassign bugs (anybody interested?)
- Give more people in bugzilla the power to reassign arbitary bugs (bugzilla maintainers would need to do that)

Quoted 3 lines · Andi Kleen

- Ask more people to just categorize and reassign bugs (anybody interested?)
- Give more people in bugzilla the power to reassign arbitary bugs (bugzilla maintainers would need to do that)

I think both of these are good ideas, but I don't think it's enough. There must be some better form of bug tracking than bugzilla. Some relly distributed way of letting people work together, *without* having to congregate on some central web-site kind of thing. A "git for bugs", where you can track bugs locally and without a web interface.

And I think it would be something much closer to "distributed email mbox with status tracking facilities", _not_ a web clicky interface.

Quoted 9 lines · Andi Kleen

Quoted 5 lines · Linus Torvalds

The thing is, bugzilla is totally broken because it's designed to help track bugs, but it's *not* designed to actually handle the much harder problem, which is to actually get the *right* developers to be aware of the *right* bugs!

This means we need people who figure out who to assign bugs too. Aka bugmasters.

Yes. But not using bugzilla.

I don't know if you've noticed already, but I'm not the only one that doesn't have a very high opinon of bugzilla ;)

What works is somebody who is a bugmaster, and it doesn't really matter *what* bug tracker he points to (bugzilla being one of the possibilities, although not necessarily the best, and absolutely NOT the only choice), and turn them into emails. Once they are emails, bugzilla can track them.

Andrew does this, and yes, Adrian did it.

Quoted 3 lines

In theory it could be nearly automated. Figure out what files related to the bug and assign to the last 5 people who submitted patches for them and/or signed off.

I do think you're pretty optimistic. I think it's true for trivial driver bugs, but even for trivial driver bugs the initial report is often not enough to pinpoint the driver.

Let's take the sis900 bug as an example (not because I want to rag on that being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's a good case of something really really easy - and if that easy case isn't handled trivially and obviously, then the bug-reporting doesn't work).

In that case, the initial report was (condensed version, but fairly accurate): "system hangs on boot at random points. 2.6.21-rc7 worked well".

Now, realistically, if that entry had been in bugzilla, what would you do?

Equally realistically, let's ignore bugzilla for a moment, and ask what the best method for handling something like this would be? Have an open mind, no rules on "have to use bug tracker XYZ".

You know what? The report went to me and the kernel mailing list as email. And that was the *right* thing in that case. There was no good sign of who it should go to, and while there wasn't a whole lot of information there, there *was* a very tight timeframe (ie it could pinpoint to within about a week when it started). But the only thing I could really ask for was for the person to bisect it.

Would bugzilla have helped? HELL NO. It would have been a disaster. It would have wasted reporter time, it would have wasted developer time, and it would likely have been ignored because the bug report wasn't specific enough to really trigger any good queries or trigger a maintainer.

So bugzilla wouldn't have helped even for a _trivial_ bug.

I personally suspect that bugzilla helps more if a bug actually has a real history, and people want to actually save that history because they are stumped. But I think it should come in *then*, not immediately. Because it is actually too intrustive for the simple stuff - and most bugs actually are simple, it's just that they also get resolved so simply that people don't even think of them as bugs (ie we have a lot of "duh" kind of fixes too).

My personal experience with bugzilla is that it's very unfriendly to reporters. IMHO it's suitable for tracking unresolved problems along with debug patches, system information etc., but not for _reporting_ new ones.

Besides the primary point of bug tracking is not to be friendly to someone, but to (a) fix the bugs and (b) know how many bugs there for a given release. Any replacement would need to solve this problem too.

Email does not solve it as far as I can see.

Quoted 7 lines

Besides the primary point of bug tracking is not to be friendly to someone, but to (a) fix the bugs and (b) know how many bugs there for a given release. Any replacement would need to solve this problem too.

Email does not solve it as far as I can see.

Email fixes a _lot_ more bugs than bugzilla does.

End of story. I don't think anybody who cannot accept that UNDENIABLE FACT should even participate in this discussion. Wake up and look at all the bugs we fix - most of them have never been in bugzilla.

That's a FACT.

Don't go around ignoring reality.

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

Why you didn't proposed (used) Debian's BTS as alternative to bugzilla,

[...]

Quoted 1 line

Why you didn't proposed (used) Debian's BTS as alternative to bugzilla,

[...]

BTS has been mentioned in that thread in a few posts; mostly positively as I recall.

I thought somebody, who familiar with that, might propose to setup/tune it, but not doing yet another NIH thing,

On Thu, Jun 14, 2007 at 05:33:40PM +0200, Stefan Richter wrote:
[]

Quoted 6 lines

[...]

Quoted 1 line

Why you didn't proposed (used) Debian's BTS as alternative to bugzilla,

[...]

BTS has been mentioned in that thread in a few posts; mostly positively as I recall.

I know, that most developers here are not working/familiar with what Debian has as its bug shooting weapon ``The system is mainly controlled by e-mail, but the bug reports can be viewed using the WWW.''[0].

I thought somebody, who familiar with that, might propose to setup/tune it, but not doing yet another NIH thing, especially from e-mail integration POV. I doubt mozilla guys can think about it without javascript and/or java servlets :)
...

On Thu, Jun 14, 2007 at 06:39:23PM +0200, Oleg Verych wrote:

Quoted 9 lines

I know, that most developers here are not working/familiar with what Debian has as its bug shooting weapon ``The system is mainly controlled by e-mail, but the bug reports can be viewed using the WWW.''[0].

I thought somebody, who familiar with that, might propose to setup/tune it, but not doing yet another NIH thing, especially from e-mail integration POV. I doubt mozilla guys can think about it without javascript and/or java servlets :)
...

The problem isn't Bugzilla, and the Debian BTS wouldn't solve any problem.

What is missing?

We need people who know one or more subsystems and who are willing to regularly handle bug reports in their area.

And we need a release process that makes debugging, and if possible fixing, all regressions prior to the release mandatory. You might never come down to zero regressions and might not be able to handle all last-minute reported regressions, but the 2.6.21 situation with 3 week old known regressions not ever being debugged by a kernel developer before the release has much room for improvements.

Changing the BTS would make sense if some core developers would state that they would start using the BTS after this change. But otherwise it doesn't matter which BTS to use.

...
Also, after i saw Linus' message about doing mostly tools last couple of years, i wonder why you, Adrian, didn't think about your tools first, before you've started regression tracking? You are not running in front of a train, unlike you know who does, plus bugzilla issues are known for years. Luckily Fedora kernel guys also upstream developers, thus lkml and other MLs under their view.

After having read all that, i've asked you, my question, as the person who supposedly used BTS as a maintainer.

Yes, in current form it might not be in suitable configuration, i.e. kernel sub-systems instead of packages etc, anyway main thing is the way BTS is handled. While i was looking and replying for bug reports in the Debian kernel, that i saw in lkml, i've noticed, just how guys work with it there. Now they even came up with tracking upstream bugzilla, it seems [0]. I left that activity due to RL some months ago, but now trying to catch up things again.
...

Quoted 10 lines

And we need a release process that makes debugging, and if possible fixing, all regressions prior to the release mandatory. You might never come down to zero regressions and might not be able to handle all last-minute reported regressions, but the 2.6.21 situation with 3 week old known regressions not ever being debugged by a kernel developer before the release has much room for improvements.

Changing the BTS would make sense if some core developers would state that they would start using the BTS after this change. But otherwise it doesn't matter which BTS to use.

So, as i've wrote before: one must give them pretty-shiny tool, kindly barking in their inboxes, instead of for example

"Guilty: **** ***** <????@****.com>",

as it was on the very beginning.

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

Quoted 3 lines · Oleg Verych

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

IOW, I really don't think we _resolved_ anything, although the work that Adrian started is continuing through the wiki and other people trying to track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the long run. I think everybody wants a better bug-tracking system, but whether something that makes people satisfied can even be built is open. It sure doesn't seem to exist right now ;)

Linus

Quoted 9 lines · Linus Torvalds

Quoted 3 lines · Oleg Verych

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

Quoted 8 lines

IOW, I really don't think we _resolved_ anything, although the work that Adrian started is continuing through the wiki and other people trying to track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the long run. I think everybody wants a better bug-tracking system, but whether something that makes people satisfied can even be built is open. It sure doesn't seem to exist right now ;)

The problem is not the bug tracking system, be it manual tracking in a text file or a Wiki or be it in Bugzilla or any other bug tracking system.

One problem is the lack of experienced developers willing to debug bug reports.

But what really annoyed me was the missing integration of regression tracking into the release process, IOW how _you_ handled the regression lists.

If we want to offer something less of a disaster than 2.6.21, and if we want to encourage people to start and continue testing -rc kernels, we must try to fix as many reported regressions as reasonably possible.

This means going through every single point in the regression list asking "Have we tried everything possible to solve this regression?". There are very mysterious regressions and there are regressions that might simply be reported too late. But if at the time of the final release 3 week old regressions hadn't been debugged at all there's definitely room for improvement. And mere mortals like me reminding people is often not enough, sometimes an email by Linus Torvalds himself asking a patch author or maintainer to look after a regression might be required.

And a low hanging fruit to improve the release would be if you could release one last -rc, wait for 48 hours, and then release either this -rc unchanged as -final or another -rc (and wait another 48 hours). There were at least two different regressions people ran into in 2.6.21 who successfully tested -rc7.

Quoted 23 lines · Adrian Bunk

Quoted 9 lines · Linus Torvalds

Quoted 3 lines · Oleg Verych

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

Quoted 8 lines

IOW, I really don't think we _resolved_ anything, although the work that Adrian started is continuing through the wiki and other people trying to track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the long run. I think everybody wants a better bug-tracking system, but whether something that makes people satisfied can even be built is open. It sure doesn't seem to exist right now ;)

The problem is not the bug tracking system, be it manual tracking in a text file or a Wiki or be it in Bugzilla or any other bug tracking system.

Quoted 2 lines

One problem is the lack of experienced developers willing to debug bug reports.

*debug*

I hope you saw what subject i've chosen to bring this discussion back. Yes, "tracking", as the first brick for big wall.

Your arguments about developers and users, you've said already, but i've asked different questions, have i?

Lets look on regular automatic report, like this one:

Message-ID: <[email protected]>
Xref: news.gmane.org gmane.linux.debian.devel.general:116248
Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.general/116248

And what we see? Basic packages, like ``dpkg'', ``grub'', ``mc'' are in the list, requesting help. And as you can see for quite some time. And it's *OK*, because distribution is working, development is going on. Sometimes slowly, sometimes with delays.

Quoted 3 lines

But what really annoyed me was the missing integration of regression tracking into the release process, IOW how _you_ handled the regression lists.

So, _tracking_ or _debugging_?

Quoted 3 lines

If we want to offer something less of a disaster than 2.6.21, and if we want to encourage people to start and continue testing -rc kernels, we must try to fix as many reported regressions as reasonably possible.

*tracking*

...

Quoted 9 lines

This means going through every single point in the regression list asking "Have we tried everything possible to solve this regression?". There are very mysterious regressions and there are regressions that might simply be reported too late. But if at the time of the final release 3 week old regressions hadn't been debugged at all there's definitely room for improvement. And mere mortals like me reminding people is often not enough, sometimes an email by Linus Torvalds himself asking a patch author or maintainer to look after a regression might be required.

*social* (first approximation)

That's a social problem, just like Debian loosing good kernel team members.

For example you feel, that you've wasted time. But after all, if you've came up with some kind of tool, everybody else could take it. No problems, useful ideas must and will evolve. But _ideally_ this must not be from ground zero every time. _Ideally_ from technical, not personal point of view ;).

That's why people in Debian have started *team* maintenance with alioth.

Unfortunately problems with individuals in big machine with bad people, got randomly elected, can't be solved (IMHO). Even LKML's rule "patches are welcome", that is very technical, thus good, doesn't work there.

Finally, read the end of 2.6.21 release message ;)

Quoted 5 lines

And a low hanging fruit to improve the release would be if you could release one last -rc, wait for 48 hours, and then release either this -rc unchanged as -final or another -rc (and wait another 48 hours). There were at least two different regressions people ran into in 2.6.21 who successfully tested -rc7.

What about Linus' tree is a development tree, Andrew's one is a "crazy development one" (quoting Linus)?

What about open (web page still exists) position on bug manager in Google?

What about *volunteers* working from both developer's and user's sides? What about "release is a reward" for everybody?

Balanced eco-system will oscillate. Be it .19(---++), .20(-++++), .21(----+) *relese*. That's natural, unless pushed to extremes.

I think, i can trust Linus in it, and you are welcome too.

*tools*

That's why i'm talking about tools, and started to discuss them.

My last try: reportbug (there's "-ng" one also), Debian BTS.

Adrian, what can/must be done to adopt them? If not, your experience may provide information about "why?" (re-consider my first mail about background, please).

Quoted 27 lines · Oleg Verych

Quoted 25 lines · Adrian Bunk

Quoted 9 lines · Linus Torvalds

Quoted 3 lines · Oleg Verych

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

Quoted 8 lines

IOW, I really don't think we _resolved_ anything, although the work that Adrian started is continuing through the wiki and other people trying to track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the long run. I think everybody wants a better bug-tracking system, but whether something that makes people satisfied can even be built is open. It sure doesn't seem to exist right now ;)

The problem is not the bug tracking system, be it manual tracking in a text file or a Wiki or be it in Bugzilla or any other bug tracking system.

One problem is the lack of experienced developers willing to debug bug reports.

Quoted 4 lines

*debug*

I hope you saw what subject i've chosen to bring this discussion back. Yes, "tracking", as the first brick for big wall.

Tracking regressions is not a real problem. Especially since it doesn't require much technical knowledge.

Quoted 5 lines

Your arguments about developers and users, you've said already, but i've asked different questions, have i?

Lets look on regular automatic report, like this one:

Quoted 8 lines

Message-ID: <[email protected]>
Xref: news.gmane.org gmane.linux.debian.devel.general:116248
Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.general/116248

And what we see? Basic packages, like ``dpkg'', ``grub'', ``mc'' are in the list, requesting help. And as you can see for quite some time. And it's *OK*, because distribution is working, development is going on. Sometimes slowly, sometimes with delays.

I sent weekly regression reports.
Not automatically generated but manually - but that doesn't matter.

The problem is that sending reports itself does not fix anything.

Quoted 5 lines

Quoted 3 lines

But what really annoyed me was the missing integration of regression tracking into the release process, IOW how _you_ handled the regression lists.

So, _tracking_ or _debugging_?

_debugging_ (can be indirectly by poking other people to do debugging)

Quoted 4 lines

Quoted 3 lines

If we want to offer something less of a disaster than 2.6.21, and if we want to encourage people to start and continue testing -rc kernels, we must try to fix as many reported regressions as reasonably possible.

Quoted 1 line

*tracking*

no, *debugging*

I tracked regressions for the 2.6.21 disaster, and the not debugged regressions that I had tracked are exactly where we should be better.

Quoted 11 lines

...

Quoted 9 lines

This means going through every single point in the regression list asking "Have we tried everything possible to solve this regression?". There are very mysterious regressions and there are regressions that might simply be reported too late. But if at the time of the final release 3 week old regressions hadn't been debugged at all there's definitely room for improvement. And mere mortals like me reminding people is often not enough, sometimes an email by Linus Torvalds himself asking a patch author or maintainer to look after a regression might be required.

Quoted 1 line

*social* (first approximation)

Yes.

Quoted 1 line

That's a social problem, just like Debian loosing good kernel team members.

Different social problem.

Quoted 5 lines

For example you feel, that you've wasted time. But after all, if you've came up with some kind of tool, everybody else could take it. No problems, useful ideas must and will evolve. But _ideally_ this must not be from ground zero every time. _Ideally_ from technical, not personal point of view ;).

As I expected, someone else has picked up regression tracking. And other from what you claim, this did not require any kind of tool.

Quoted 1 line

That's why people in Debian have started *team* maintenance with alioth.

The problem for the Linux kernel is that for a better bug handling you'd need people willing to learn other people's code and to do the hard work of debugging bug reports.

E.g. writing a new filesystem is simply much more fun than learning and debugging other people's code in some old filesystem.

Talking about "team maintenance" sounds nice, but the problem in the kernel starts with code that has zero maintainers.

And if there's already a maintainer, it's unlikely that he'll not accept patches from some new person debugging bug reports. But how to find people who will become good maintainers?

Quoted 3 lines

Unfortunately problems with individuals in big machine with bad people, got randomly elected, can't be solved (IMHO). Even LKML's rule "patches are welcome", that is very technical, thus good, doesn't work there.

Debian has it's own problems, Linux kernel development at least has a working structure for getting decisions and regular releases.

Quoted 11 lines

Finally, read the end of 2.6.21 release message ;)

Quoted 5 lines

And a low hanging fruit to improve the release would be if you could release one last -rc, wait for 48 hours, and then release either this -rc unchanged as -final or another -rc (and wait another 48 hours). There were at least two different regressions people ran into in 2.6.21 who successfully tested -rc7.

What about Linus' tree is a development tree, Andrew's one is a "crazy development one" (quoting Linus)?

Quoted 5 lines

What about open (web page still exists) position on bug manager in Google?

What about *volunteers* working from both developer's and user's sides? What about "release is a reward" for everybody?

People aren't that dumb that some empty words like "release is a reward" would change anything.

Quoted 7 lines

Balanced eco-system will oscillate. Be it .19(---++), .20(-++++), .21(----+) *relese*. That's natural, unless pushed to extremes.

I think, i can trust Linus in it, and you are welcome too.

*tools*

Quoted 7 lines

That's why i'm talking about tools, and started to discuss them.

My last try: reportbug (there's "-ng" one also), Debian BTS.

Adrian, what can/must be done to adopt them? If not, your experience may provide information about "why?" (re-consider my first mail about background, please).

Bug tracking for the kernel is more or less working. The main problem is getting people to debug bug reports.

We need the main problem fixed, not a different tool in an area that is not the main problem.

...

Quoted 2 lines · Adrian Bunk

Quoted 1 line · Oleg Verych

On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:

...

Quoted 8 lines

Quoted 5 lines

For example you feel, that you've wasted time. But after all, if you've came up with some kind of tool, everybody else could take it. No problems, useful ideas must and will evolve. But _ideally_ this must not be from ground zero every time. _Ideally_ from technical, not personal point of view ;).

As I expected, someone else has picked up regression tracking. And other from what you claim, this did not require any kind of tool.

So you expected good, doing bad. ``Bad'' means bringing pointless flame about what everybody should do, without constructive approach like: "OK, i can't do it due to my POV{0}, useless manual work. Everybody willing to bring another way of dealing with it is welcome."

Your first reply:

"And it's not that Linus started developing the Linux kernel with writing git, the first 10 years of Linux development were without any SCM." {3}

to my note about, that you are not hurry anywhere, that after all that years of Open Source and Free Software development you are not trying to deal with such important thing like regression/bug tracking in ***organized way***, is rather pointless.

Quoted 5 lines

Quoted 1 line

That's why people in Debian have started *team* maintenance with alioth.

The problem for the Linux kernel is that for a better bug handling you'd need people willing to learn other people's code and to do the hard work of debugging bug reports.

... {0}++

Do you know, for example, why i'm not making my "hacker's career" doing that?

1. because i ended up with lynx, slrn, mutt, emacs-nox. Including "zarro bogs found" kind of thing and other "userspace suck" problems.
2. i have no way to know if something *really* broken, unless it right on my hardware

This all unlike Debian BTS using reportbug, where you have basic information, mbox format messages for easy "mutt -f", and other funny things, real maintainers aware of (i'm trying to know, learn about).

Thus organized, non brain-damaged way of reporting and tracking is the key issue.

...

Quoted 2 lines

Talking about "team maintenance" sounds nice, but the problem in the kernel starts with code that has zero maintainers.

Floppy went pretty fine, before it was started to be maintained, and you know that. But you also told that unmaintained and not-working are different things.

...
I see (this repetition). Maybe i've managed to express my POV, that it can be seeing more cleanly.

On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:

Quoted 2 lines

This means going through every single point in the regression list asking "Have we tried everything possible to solve this regression?".

Quoted 5 lines

And a low hanging fruit to improve the release would be if you could release one last -rc, wait for 48 hours, and then release either this -rc unchanged as -final or another -rc (and wait another 48 hours). There were at least two different regressions people ran into in 2.6.21 who successfully tested -rc7.

What about Linus' tree is a development tree, Andrew's one is a "crazy development one" (quoting Linus)?

Well, if _other_ subsystems would get regressions in Linus' tree fixed quicker, there might perhaps be more people who would consider to run -rc kernels and would catch and report "my" regressions.

[Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too few FireWire driver users run -rc kernels".]

...
[Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too few FireWire driver users run -rc kernels".]

Stefan Richter

Quoted 3 lines

...
[Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too few FireWire driver users run -rc kernels".]

Getting more people testing -rc kernels might be possible, and I don't think it would be too hard. And not only FireWire would benefit from this, remember e.g. that at least 2 out of the last 5 kernels Linus released contained filesystem corruption regressions.

The problem is that we aren't able to handle the many regression reports we get today, so asking for more testing and regression reports today would attack it at the wrong part of the chain.

Additionally, every reported and unhandled regression will frustrate the reporter - never forget that we have _many_ unhandled bug reports (including but not limited to regression reports) where the submitter spent much time and energy in writing a good bug report.

If we somehow gain the missing manpower for debugging regressions we can actively ask for more testing. Missing manpower (of people knowing some part of the kernel well) for debugging bug reports is IMHO the one big source of quality problems in the Linux kernel. If we get this solved, things like getting more testers for -rc kernels will become low hanging fruits.

Review

Quoted 2 lines · On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski…

(quoted diff) +2 -0

+If the patch introduces a new regression and this regression was not fixed
+in seven days, then the patch will be reverted.

Those regressions where we know which patch caused them are the easy ones. Often we don't know which patch (or even which subsystem merge) is at fault.

I think. How many of the present 2.6.22-rc regressions which you're presently tracking have such a well-identified cause?

Quoted 11 lines · Andrew Morton

Quoted 1 line · On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski…

+If the patch introduces a new regression and this regression was not

fixed

Quoted 1 line

+in seven days, then the patch will be reverted.

Those regressions where we know which patch caused them are the easy ones. Often we don't know which patch (or even which subsystem merge) is at fault.

Quoted 3 lines

I think. How many of the present 2.6.22-rc regressions which you're presently tracking have such a well-identified cause?

Here lays the problem.

git-bisect is a killer app, people should start using it.

Quoted 16 lines · Michal Piotrowski

Quoted 14 lines · Andrew Morton

Quoted 1 line · On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski…

+If the patch introduces a new regression and this regression was not

fixed

Quoted 1 line

+in seven days, then the patch will be reverted.

Those regressions where we know which patch caused them are the easy ones. Often we don't know which patch (or even which subsystem merge) is at fault.

I think. How many of the present 2.6.22-rc regressions which you're presently tracking have such a well-identified cause?

Quoted 3 lines

Here lays the problem.

git-bisect is a killer app, people should start using it.

It's OK _only_ in case of unknown, hard to find *hardware* bugs.

If you think it's "a good thing" for bad, untested by developer code, then something is completely wrong.

On Sunday, 17 June 2007 13:47, Oleg Verych wrote:

Quoted 4 lines

It's OK _only_ in case of unknown, hard to find *hardware* bugs.

If you think it's "a good thing" for bad, untested by developer code, then something is completely wrong.

Oh, I've just fixed two purely software bugs pointed out by binary searching in the code that I'm sure has been tested, not only by its developers, but the bugs only showed up in my configuration (on one out of four test boxes).

There are so many different kernel configurations possible that there's no way a developer can test them all.

Quoted 1 line · Rafael J. Wysocki

On Sunday, 17 June 2007 13:47, Oleg Verych wrote:

[]

Quoted 9 lines

Quoted 4 lines

It's OK _only_ in case of unknown, hard to find *hardware* bugs.

If you think it's "a good thing" for bad, untested by developer code, then something is completely wrong.

Oh, I've just fixed two purely software bugs pointed out by binary searching in the code that I'm sure has been tested, not only by its developers, but the bugs only showed up in my configuration (on one out of four test boxes).

Quoted 2 lines

There are so many different kernel configurations possible that there's no way a developer can test them all.

With current state of affairs it's not only hard for developers, but and for users: <[email protected]>,

              <[email protected]>

I'm trying to re-do some kbuild stuff, but i'm getting rather offensive answers :( <1182020654.8176.398.camel@chaos>
...

Quoted 12 lines · Michal Piotrowski

Quoted 11 lines · Andrew Morton

Quoted 1 line · On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski…

+If the patch introduces a new regression and this regression was not

fixed

Quoted 1 line

+in seven days, then the patch will be reverted.

Those regressions where we know which patch caused them are the easy ones. Often we don't know which patch (or even which subsystem merge) is at fault.

Quoted 7 lines

Quoted 3 lines

I think. How many of the present 2.6.22-rc regressions which you're presently tracking have such a well-identified cause?

Here lays the problem.

git-bisect is a killer app, people should start using it.

It's OK _only_ in case of unknown, hard to find *hardware* bugs.

If you think it's "a good thing" for bad, untested by developer code, then something is completely wrong.

And if there's no debugger in the mainline kernel, which is developer's tool, then why do you think testers must stick with git-bisect, as their debugger-like tool (bandwidth in most and time consuming in some cases)?

That's wrong if developers are tending to reply only one thing -- git-bisect.

If things are going to be that bad, then better to start dealing with the cause, not consequences. In this situation requesting test-cases is a better way, as it's going to influence developer as cause of potential problems. If tests will show *hardware* side of problem, then, well some parts may be not obvious, thus bisecting is a way to continue.

On Sun, 17 Jun 2007, Oleg Verych wrote:

Quoted 8 lines

That's wrong if developers are tending to reply only one thing -- git-bisect.

If things are going to be that bad, then better to start dealing with the cause, not consequences. In this situation requesting test-cases is a better way, as it's going to influence developer as cause of potential problems. If tests will show *hardware* side of problem, then, well some parts may be not obvious, thus bisecting is a way to continue.

most people who report bugs don't know enough about what's actually going wrong to be able to write a test case (those that do can probably just write a patch to fix it). Along similar lines a debugger wouldn't be of much use either.

the fact that git-bisect doesn't require any knowledge other then knowledge the reporter has demonstrated that they already have (the ability to compile and install their own kernel) puts it within the reach of testers.

unfortunantly, as good as it is it can take a lot of effort, especially if the bug takes time to show up. it's not perfect, but it's a huge help.

and developers aren't always responding with 'do a bisect', sometimes they respond with 'yes, we know about that'

or 'that sounds like X',

so it's still worthwhile for people to report the problem first before going to the ffort of doing a bisect.

David Lang

Quoted 6 lines · Andrew Morton

Review

Quoted 2 lines · On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski…

(quoted diff) +2 -0

+If the patch introduces a new regression and this regression was not fixed
+in seven days, then the patch will be reverted.

Those regressions where we know which patch caused them are the easy ones.

Quoted 6 lines

Often we don't know which patch (or even which subsystem merge) is at fault.

I think. How many of the present 2.6.22-rc regressions which you're presently tracking have such a well-identified cause?

Here lays the problem.

git-bisect is a killer app, people should start using it.

Hi all,

Adrian Bunk pisze:

Quoted 13 lines · Stefan Richter

Quoted 3 lines

...
[Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too few FireWire driver users run -rc kernels".]

Getting more people testing -rc kernels might be possible, and I don't think it would be too hard. And not only FireWire would benefit from this, remember e.g. that at least 2 out of the last 5 kernels Linus released contained filesystem corruption regressions.
The problem is that we aren't able to handle the many regression reports we get today, so asking for more testing and regression reports today would attack it at the wrong part of the chain.
Additionally, every reported and unhandled regression will frustrate the reporter - never forget that we have _many_ unhandled bug reports

Quoted 8 lines

(including but not limited to regression reports) where the submitter spent much time and energy in writing a good bug report.
If we somehow gain the missing manpower for debugging regressions we can actively ask for more testing. Missing manpower (of people knowing some part of the kernel well) for debugging bug reports is IMHO the one big source of quality problems in the Linux kernel. If we get this solved, things like getting more testers for -rc kernels will become low hanging fruits.

Adrian, I agree with _all_ your points.

I bet that developers will hate me for this.

Please consider for 2.6.23

Regards,
Michal
...

Quoted 29 lines · Michal Piotrowski

Hi all,

Adrian Bunk pisze:

Quoted 13 lines · Stefan Richter

Quoted 3 lines

...
[Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too few FireWire driver users run -rc kernels".]

Getting more people testing -rc kernels might be possible, and I don't think it would be too hard. And not only FireWire would benefit from this, remember e.g. that at least 2 out of the last 5 kernels Linus released contained filesystem corruption regressions.
The problem is that we aren't able to handle the many regression reports we get today, so asking for more testing and regression reports today would attack it at the wrong part of the chain.
Additionally, every reported and unhandled regression will frustrate the reporter - never forget that we have _many_ unhandled bug reports

Quoted 8 lines

(including but not limited to regression reports) where the submitter spent much time and energy in writing a good bug report.
If we somehow gain the missing manpower for debugging regressions we can actively ask for more testing. Missing manpower (of people knowing some part of the kernel well) for debugging bug reports is IMHO the one big source of quality problems in the Linux kernel. If we get this solved, things like getting more testers for -rc kernels will become low hanging fruits.

Adrian, I agree with _all_ your points.

I bet that developers will hate me for this.

Quoted 1 line

Please consider for 2.6.23

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

Quoted 3 lines

Regards,
Michal
...

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

"choose minor evil to prevent a greater one"

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 6 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

There's now much money in the Linux market, and the kernel quality problems might result in real costs in the support of companies like IBM, SGI, Redhat or Novell (plus it harms the Linux image which might result in lower revenues).

If [1] this is true, it might even pay pay off for them to each assign X man hours per month of experienced kernel developers to upstream kernel bug handling?

This is just a wild thought and it might be nonsense - better suggestions for solving our quality problems would be highly welcome...

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

Quoted 8 lines · Adrian Bunk

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

...

Perhaps make lists of

- bug reports which never lead to any debug activity
(no responsible person/team was found, or a seemingly person/team did not start to debug the report)

- known regressions on release,

- regressions that became known after release,

- subsystems with notable backlogs of old bugs,

- other categories?

Select typical cases from each categories, analyze what went wrong in these cases, and try to identify practicable countermeasures.

Another approach: Figure out areas where quality is exemplary and try to draw conclusions for areas where quality is lacking.
--
Stefan Richter
-=====-=-=== -==- =---=
http://arcgraph.de/sr/

Quoted 8 lines · Adrian Bunk

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

...

Perhaps make lists of

- bug reports which never lead to any debug activity
(no responsible person/team was found, or a seemingly person/team did not start to debug the report)

- known regressions on release,

- regressions that became known after release,

- subsystems with notable backlogs of old bugs,

- other categories?

Select typical cases from each categories, analyze what went wrong in these cases, and try to identify practicable countermeasures.

Another approach: Figure out areas where quality is exemplary and try to draw conclusions for areas where quality is lacking.

Stefan Richter

Quoted 2 lines · Stefan Richter

Another approach: Figure out areas where quality is exemplary and try to draw conclusions for areas where quality is lacking.

ieee1394 has a maintainer who is looking after all bug reports he gets.

Quoted 19 lines · Stefan Richter

Quoted 8 lines · Adrian Bunk

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

...

Perhaps make lists of

- bug reports which never lead to any debug activity
(no responsible person/team was found, or a seemingly person/team did not start to debug the report)

- known regressions on release,

Quoted 8 lines

- regressions that became known after release,

- subsystems with notable backlogs of old bugs,

- other categories?

Select typical cases from each categories, analyze what went wrong in these cases, and try to identify practicable countermeasures.

No maintainer or no maintainer who is debugging bug reports is the major problem in all parts of your list.

Quoted 2 lines

Another approach: Figure out areas where quality is exemplary and try to draw conclusions for areas where quality is lacking.

ieee1394 has a maintainer who is looking after all bug reports he gets.

Conclusion: We need such maintainers for all parts of the kernel.

Quoted 1 line

Stefan Richter

cu
Adrian

Quoted 3 lines · Adrian Bunk

Quoted 2 lines · Stefan Richter

Quoted 1 line · Adrian Bunk

Quoted 1 line

Quoted 1 line

And we should be aware that reverting is only a workaround for the

real

Quoted 29 lines

Quoted 17 lines

Quoted 7 lines

Quoted 2 lines

Quoted 1 line

problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

...

Perhaps make lists of

- bug reports which never lead to any debug activity
(no responsible person/team was found, or a seemingly person/team did not start to debug the report)

- known regressions on release,

Quoted 8 lines

- regressions that became known after release,

- subsystems with notable backlogs of old bugs,

- other categories?

Select typical cases from each categories, analyze what went wrong in these cases, and try to identify practicable countermeasures.

No maintainer or no maintainer who is debugging bug reports is the major problem in all parts of your list.

Quoted 7 lines

Quoted 2 lines

Another approach: Figure out areas where quality is exemplary and try to draw conclusions for areas where quality is lacking.

ieee1394 has a maintainer who is looking after all bug reports he gets.

Conclusion: We need such maintainers for all parts of the kernel.

I noticed some areas are well maintained because there is an awesome maintainer, or good and well coordinated team - and this is mostly in the "fun" areas ;) But there are "boring" areas that are about to be deprecated or no new development expected etc. It will be hard to get a dedicated person to take care of such. How about having people on rotation, or jury duty so to speak - for a period of time (completely voluntary!) Nice stats on the report about contributions in non-native areas for a developer would be great accomplishment and also good chance to look into other things! Besides, this way "old parts" will get attention to be be revised and re-implemented sooner. And we can post "Temp maintainer needed" list...

just reading through it and cleaning up some code isn't so difficult and can be done by many people -

if they're allowed/wanted to do so.

Quoted 2 lines · Markus Rechberger

just reading through it and cleaning up some code isn't so difficult and can be done by many people -

Doing cleanups is a good way to get into the matter, to become able to eventually fix bugs of the difficult type.

Quoted 1 line

if they're allowed/wanted to do so.

Everybody is allowed to submit. But there is a certain degree of both persistence and adaptability required to get one's first submissions upstream. However, these qualities are also required to fix difficult bugs.

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 3 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

Quoted 25 lines · Adrian Bunk

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 3 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 8 lines

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

I think that we can handle bug reports like we handle modifications of code.

Namely, for each subsystem there can be a person (or a team) responsible for handling bugs, by which I don't mean fixing them, but directing bug reports at the right developers or subsystem maintainers, following the history of each bug report etc. [Of course, these people can choose to use the bugzilla or any other bug tracking system they want, as long as it works for them.]

The email addresses of these people should be known (and even documented), so that everyone can notify them if need be and so that it's clear who should handle given bug reports.

Just an idea. :-)

Quoted 43 lines · Rafael J. Wysocki

Quoted 25 lines · Adrian Bunk

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 3 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 8 lines

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

I think that we can handle bug reports like we handle modifications of code.

Namely, for each subsystem there can be a person (or a team) responsible for handling bugs, by which I don't mean fixing them, but directing bug reports at the right developers or subsystem maintainers, following the history of each bug report etc. [Of course, these people can choose to use the bugzilla or any other bug tracking system they want, as long as it works for them.]

Quoted 6 lines

The email addresses of these people should be known (and even documented), so that everyone can notify them if need be and so that it's clear who should handle given bug reports.

Just an idea. :-)

Those are very good ideas indeed. The whole development process came to the point when all realize that something needs to be done for the team to balance out new development and old and recent unresolved issues that are piling up...

I've looked through a number of bugzillas recently and here is my scoop on shortcomings and some ideas. I am not sure how realistic they are, probably might fall into "wishful thinking" category.

The way bugs get tracked and resolved is definitely a "no guarantee", and main reasons are:
not enough time for a maintainer to attend to them all nobody else (except at best very few busy people) knows about
majority of the problems. Andrew and Adrian and Michal post the most pressing ones. But there are many many smaller ones that are not assessed and not being taken care of.
many problems are not easily reproducible and not easy to verify
because there is no identical system, motherboard, application, etc. in case if reporter doesn't stick around until the end of the bug's life.

Maybe along with bugzilla there should be another tracking tool - for resources and systems that are available to individual developers.

Someone might have same or similar system to verify fixes in case if the reporter disappears or "the system is gone now". Requests for specific hardware can be automatically generated by the bugzilla say. Those can be posted once in a while for everyone to see and chip in and acknowledge if they happen to have such hardware and able to run a quick test to at least verify the patch. Statistically, such need doesn't happen often for each type of hardware, so it shouldn't be a big burden for owners.

Besides, the database and resources can be useful for developers who want to test their new patches on variety of hardware. This might prevent future regressions which often caused by lack of testing as we all know.

Quoted 25 lines · Adrian Bunk

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 3 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 8 lines

Quoted 3 lines

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

I think that we can handle bug reports like we handle modifications of code.

Namely, for each subsystem there can be a person (or a team) responsible for handling bugs, by which I don't mean fixing them, but directing bug reports at the right developers or subsystem maintainers, following the history of each bug report etc. [Of course, these people can choose to use the bugzilla or any other bug tracking system they want, as long as it works for them.]

The email addresses of these people should be known (and even documented), so that everyone can notify them if need be and so that it's clear who should handle given bug reports.

Just an idea. :-)

Greetings,
Rafael

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 6 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

There's now much money in the Linux market, and the kernel quality problems might result in real costs in the support of companies like IBM, SGI, Redhat or Novell (plus it harms the Linux image which might result in lower revenues).

If [1] this is true, it might even pay pay off for them to each assign X man hours per month of experienced kernel developers to upstream kernel bug handling?

This is just a wild thought and it might be nonsense - better suggestions for solving our quality problems would be highly welcome...

From reviewer's POV: I have invested my time into review, discovered real

From my side I think I'll start adding less formal "Reviewed-by" to IDE

IMO we should concentrate more on preventing regressions than on fixing them. In the long-term preventing bugs is cheaper than fixing them afterwards.

First let me tell you all a little story...

Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs in it (in other words I potentially prevented three regressions). I also asked for more thorough verification of the patch as I suspected that it may have more problems. The author fixed the issues and replied that he hasn't done the full verification yet but he doesn't suspect any problems...

Fast forward...

Year later I discover that the final version of the patch hit the mainline. I don't remember ever seeing the final version in my mailbox (there are no
cc: lines in the patch description) and I saw that I'm not credited in the patch description. However the worse part is that it seems that the full
verification has never been done. The result? Regression in the release
kernel (exactly the issue that I was worried about) which required three patches and over a month to be fixed completely. It seems that a year was not enough to get this ~70k _cleanup_ patch fully verified and tested (it hit -mm soon before being merged)...

Quoted 1 line

From reviewer's POV: I have invested my time into review, discovered real

issues and as a reward I got no credit et all and extra frustration from the fact that part of my review was forgotten/ignored (the part which resulted in real regression in the release kernel)... Oh and in the past the said developer has already been asked (politely in private message) to pay more attention to his changes (after I silently fixed some other regression caused by his other patch).

But wait there is more, I happend to be the maintainer of the subsystem which got directly hit by the issue and I was getting bugreports from the users about the problem... :-)

It wasn't my first/last bad experience as a reviewer... finally I just gave up on reviewing other people patches unless they are stricly for IDE subsystem.

The moral of the story is that currently it just doesn't pay off to do code reviews.

From personal POV it pays much more to wait until buggy patch
hits the mainline and then fix the issues yourself (at least you will get some credit). To change this we should put more ephasize on the importance of code reviews by "rewarding" people investing their time into reviews and "rewarding" developers/maintainers taking reviews seriously.

We should credit reviewers more, sometimes it takes more time/knowledge to review the patch than to make it so getting near to zero credit for review doesn't sound too attractive. Hmm, wait it can be worse - your review may be ignored... ;-)

Quoted 1 line

From my side I think I'll start adding less formal "Reviewed-by" to IDE

patches even if the review resulted in no issues being found (in additon to explicit "Acked-by" tags and crediting people for finding real issues - which I currently always do as a way for showing my appreciation for their work).

I also encourage other maintainers/developers to pay more attention to adding "Acked-by"/"Reviewed-by" tags and crediting reviewers. I hope that maintainers will promote changes that have been reviewed by others by giving them priority over other ones (if the changes are on more-or-less the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high regressions/patches ratio?

I think that we should have info about regressions integrated into SCM, i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion. This feature combined with
"Author:" info after some time should give us some very interesting statistics (Top Ten "Regressors"). It wouldn't be ideal (ie. we need some patches threshold to filter out people with 1 patch and >= 1 regression(s), we need to remember that some code areas are more difficult than the others and that patches are not equal per se etc.) however I believe than making it into Top Ten "Regressors" should give the winners some motivation to improve their work ethic. Well, in the worst case we would just get some extra trivial/documentation patches. ;-)

Quoted 15 lines · On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz…

IMO we should concentrate more on preventing regressions than on fixing them. In the long-term preventing bugs is cheaper than fixing them afterwards.

First let me tell you all a little story...

Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs in it (in other words I potentially prevented three regressions). I also asked for more thorough verification of the patch as I suspected that it may have more problems. The author fixed the issues and replied that he hasn't done the full verification yet but he doesn't suspect any problems...

Fast forward...

Quoted 9 lines

Year later I discover that the final version of the patch hit the mainline. I don't remember ever seeing the final version in my mailbox (there are no
cc: lines in the patch description) and I saw that I'm not credited in the patch description. However the worse part is that it seems that the full
verification has never been done. The result? Regression in the release
kernel (exactly the issue that I was worried about) which required three patches and over a month to be fixed completely. It seems that a year was not enough to get this ~70k _cleanup_ patch fully verified and tested (it hit -mm soon before being merged)...

crap. Commit ID, please ;)

Quoted 12 lines

Quoted 1 line

From reviewer's POV: I have invested my time into review, discovered real

issues and as a reward I got no credit et all and extra frustration from the fact that part of my review was forgotten/ignored (the part which resulted in real regression in the release kernel)... Oh and in the past the said developer has already been asked (politely in private message) to pay more attention to his changes (after I silently fixed some other regression caused by his other patch).

But wait there is more, I happend to be the maintainer of the subsystem which got directly hit by the issue and I was getting bugreports from the users about the problem... :-)

Quoted 5 lines

It wasn't my first/last bad experience as a reviewer... finally I just gave up on reviewing other people patches unless they are stricly for IDE subsystem.

The moral of the story is that currently it just doesn't pay off to do code reviews.

I dunno. I suspect (hope) that this was an exceptional case, hence one should not draw general conclusions from it. It certainly sounds very bad.

Quoted 11 lines

From personal POV it pays much more to wait until buggy patch
hits the mainline and then fix the issues yourself (at least you will get some credit). To change this we should put more ephasize on the importance of code reviews by "rewarding" people investing their time into reviews and "rewarding" developers/maintainers taking reviews seriously.

We should credit reviewers more, sometimes it takes more time/knowledge to review the patch than to make it so getting near to zero credit for review doesn't sound too attractive. Hmm, wait it can be worse - your review may be ignored... ;-)

Quoted 4 lines

Quoted 1 line

From my side I think I'll start adding less formal "Reviewed-by" to IDE

patches even if the review resulted in no issues being found (in additon to explicit "Acked-by" tags and crediting people for finding real issues - which I currently always do as a way for showing my appreciation for their work).

yup, Reviewed-by: is good and I do think we should start adopting it, although I haven't thought through exactly how.

On my darker days I consider treating a Reviewed-by: as a prerequisite for merging. I suspect that would really get the feathers flying.

Quoted 8 lines

I also encourage other maintainers/developers to pay more attention to adding "Acked-by"/"Reviewed-by" tags and crediting reviewers. I hope that maintainers will promote changes that have been reviewed by others by giving them priority over other ones (if the changes are on more-or-less the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high regressions/patches ratio?

Ignoring a review would be a wildly wrong thing to do. It's so unusual that I'd be suspecting a lost email or an i-sent-the-wrong-patch.

As for high regressions/patches ratio: that'll be hard to calculate and tends to be dependent upon the code which is being altered rather than who is doing the altering: some stuff is just fragile, for various reasons.

One ratio which we might want to have a think about is the patches-sent versus reviews-done ratio ;)

Quoted 11 lines

I think that we should have info about regressions integrated into SCM, i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion. This feature combined with
"Author:" info after some time should give us some very interesting statistics (Top Ten "Regressors"). It wouldn't be ideal (ie. we need some patches threshold to filter out people with 1 patch and >= 1 regression(s), we need to remember that some code areas are more difficult than the others and that patches are not equal per se etc.) however I believe than making it into Top Ten "Regressors" should give the winners some motivation to improve their work ethic. Well, in the worst case we would just get some extra trivial/documentation patches. ;-)

We of course do want to minimise the amount of overhead for each developer. I'm a strong believer in specialisation: rather than requiring that *every* developer/maintainer integrate new steps in their processes it would be better to allow them to proceed in a close-to-usual fashion and to provide for a specialist person (or team) to do the sorts of things which you're thinking about.

where n is a number that needs to be determined (I think that n could be 3). Well, "negative comments" should also be defined more precisely. ;-)

Quoted 2 lines · Rafael J. Wysocki

where n is a number that needs to be determined (I think that n could be 3). Well, "negative comments" should also be defined more precisely. ;-)

I think that n should be a function of the number of accepted patches that this person sent in before, and the number of regressions he caused in the past.

Quoted 4 lines · Carlo Wood

Quoted 2 lines · Rafael J. Wysocki

where n is a number that needs to be determined (I think that n could be 3). Well, "negative comments" should also be defined more precisely. ;-)

Quoted 3 lines

I think that n should be a function of the number of accepted patches that this person sent in before, and the number of regressions he caused in the past.

The character of the patch (potential impacts, size...) and availability of reviewers and testers influence the required review time so much that other factors, like reputation of the submitter, hardly matter.

Quoted 15 lines · On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz…

IMO we should concentrate more on preventing regressions than on fixing them. In the long-term preventing bugs is cheaper than fixing them afterwards.

First let me tell you all a little story...

Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs in it (in other words I potentially prevented three regressions). I also asked for more thorough verification of the patch as I suspected that it may have more problems. The author fixed the issues and replied that he hasn't done the full verification yet but he doesn't suspect any problems...

Fast forward...

Quoted 9 lines

Year later I discover that the final version of the patch hit the mainline. I don't remember ever seeing the final version in my mailbox (there are no
cc: lines in the patch description) and I saw that I'm not credited in the patch description. However the worse part is that it seems that the full
verification has never been done. The result? Regression in the release
kernel (exactly the issue that I was worried about) which required three patches and over a month to be fixed completely. It seems that a year was not enough to get this ~70k _cleanup_ patch fully verified and tested (it hit -mm soon before being merged)...

crap. Commit ID, please ;)

Quoted 12 lines

Quoted 1 line

From reviewer's POV: I have invested my time into review, discovered real

issues and as a reward I got no credit et all and extra frustration from the fact that part of my review was forgotten/ignored (the part which resulted in real regression in the release kernel)... Oh and in the past the said developer has already been asked (politely in private message) to pay more attention to his changes (after I silently fixed some other regression caused by his other patch).

But wait there is more, I happend to be the maintainer of the subsystem which got directly hit by the issue and I was getting bugreports from the users about the problem... :-)

Quoted 5 lines

It wasn't my first/last bad experience as a reviewer... finally I just gave up on reviewing other people patches unless they are stricly for IDE subsystem.

The moral of the story is that currently it just doesn't pay off to do code reviews.

I dunno. I suspect (hope) that this was an exceptional case, hence one should not draw general conclusions from it. It certainly sounds very bad.

Quoted 11 lines

From personal POV it pays much more to wait until buggy patch
hits the mainline and then fix the issues yourself (at least you will get some credit). To change this we should put more ephasize on the importance of code reviews by "rewarding" people investing their time into reviews and "rewarding" developers/maintainers taking reviews seriously.

We should credit reviewers more, sometimes it takes more time/knowledge to review the patch than to make it so getting near to zero credit for review doesn't sound too attractive. Hmm, wait it can be worse - your review may be ignored... ;-)

Quoted 4 lines

Quoted 1 line

From my side I think I'll start adding less formal "Reviewed-by" to IDE

patches even if the review resulted in no issues being found (in additon to explicit "Acked-by" tags and crediting people for finding real issues - which I currently always do as a way for showing my appreciation for their work).

yup, Reviewed-by: is good and I do think we should start adopting it, although I haven't thought through exactly how.

On my darker days I consider treating a Reviewed-by: as a prerequisite for merging. I suspect that would really get the feathers flying.

Quoted 8 lines

I also encourage other maintainers/developers to pay more attention to adding "Acked-by"/"Reviewed-by" tags and crediting reviewers. I hope that maintainers will promote changes that have been reviewed by others by giving them priority over other ones (if the changes are on more-or-less the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high regressions/patches ratio?

Ignoring a review would be a wildly wrong thing to do. It's so unusual that I'd be suspecting a lost email or an i-sent-the-wrong-patch.

As for high regressions/patches ratio: that'll be hard to calculate and tends to be dependent upon the code which is being altered rather than who is doing the altering: some stuff is just fragile, for various reasons.

One ratio which we might want to have a think about is the patches-sent versus reviews-done ratio ;)

Quoted 11 lines

I think that we should have info about regressions integrated into SCM, i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion. This feature combined with
"Author:" info after some time should give us some very interesting statistics (Top Ten "Regressors"). It wouldn't be ideal (ie. we need some patches threshold to filter out people with 1 patch and >= 1 regression(s), we need to remember that some code areas are more difficult than the others and that patches are not equal per se etc.) however I believe than making it into Top Ten "Regressors" should give the winners some motivation to improve their work ethic. Well, in the worst case we would just get some extra trivial/documentation patches. ;-)

We of course do want to minimise the amount of overhead for each developer. I'm a strong believer in specialisation: rather than requiring that *every* developer/maintainer integrate new steps in their processes it would be better to allow them to proceed in a close-to-usual fashion and to provide for a specialist person (or team) to do the sorts of things which you're thinking about.

Quoted 1 line · Andrew Morton

On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <[email protected]> wrote:

Quoted 3 lines

yup, Reviewed-by: is good and I do think we should start adopting it, although I haven't thought through exactly how.

Adding Reviewed-by for reviews which highlighted real issues is obvious (with more detailed credits for noticed problems in the patch description).

Also when somebody reviewed your patch but the discussions it turned out that the patch is valid - the review itself was still valuable so it would be appropriate to credit the reviewer by adding Reviewed-by:.

Quoted 2 lines

On my darker days I consider treating a Reviewed-by: as a prerequisite for merging. I suspect that would really get the feathers flying.

Easy to workaround by a friendly mine "Reviewed-by:" for yours "Reviewed-by:" deals (without any _proper_ review being done in reality)... ;)

Quoted 11 lines

Quoted 8 lines

I also encourage other maintainers/developers to pay more attention to adding "Acked-by"/"Reviewed-by" tags and crediting reviewers. I hope that maintainers will promote changes that have been reviewed by others by giving them priority over other ones (if the changes are on more-or-less the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high regressions/patches ratio?

Ignoring a review would be a wildly wrong thing to do. It's so unusual that I'd be suspecting a lost email or an i-sent-the-wrong-patch.

It is not unusual et all. I mean patches which affect code in such way that it is difficult to prove it's (in)correctness without doing time consuming audit.

ie. lets imagine doing a small patch affecting many drivers - you've tested it quickly on your driver/hardware, then you skip the part of verifying correctness of new code in other drivers and just push the patch

As a patch author you can either assume "works for me" and push the patch or do the audit (requires good understanding of the changed code and could be time consuming). It is usually quite easy to find out which approach the author has choosen - the very sparse patch description combined with the changes in code behavior not mentioned in the patch description should raise the red flag. :)

As a reviewer having enough knowledge in the area of code affected by patch you can see the potential problems but you can't prove them without doing the time consuming part. You may try to NACK the patch if you have enough power but you will end up being bypassed by not proving incorrectness of the patch (not to mention that developer will feel bad about you NACKing his patch).

Now the funny thing is that despite the fact that audit takes more time/knowledge then making the patch you will end up with zero credit if patch turns out to be (luckily) correct. Even if you find out issues and report them you are still on mercy of author for being credited so from personal POV you are much better to wait and fix issues after they hit mainline kernel. You have to choose between being a good citizen and preventing kernel regressions or being bastard and getting the credit. ;)

If you happen to be maintainer of the affected code the choice is similar with more pros for letting the patch in especially if you can't afford the time to do audit (and by being maintainer you are guaranteed to be heavily time constrained).

I hope this makes people see the importance of proper review and proper recognition of reviewers in preventing kernel regressions.

Quoted 6 lines

As for high regressions/patches ratio: that'll be hard to calculate and tends to be dependent upon the code which is being altered rather than who is doing the altering: some stuff is just fragile, for various reasons.

One ratio which we might want to have a think about is the patches-sent versus reviews-done ratio ;)

Sounds like a good idea.

Quoted 12 lines

Quoted 11 lines

I think that we should have info about regressions integrated into SCM, i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion. This feature combined with
"Author:" info after some time should give us some very interesting statistics (Top Ten "Regressors"). It wouldn't be ideal (ie. we need some patches threshold to filter out people with 1 patch and >= 1 regression(s), we need to remember that some code areas are more difficult than the others and that patches are not equal per se etc.) however I believe than making it into Top Ten "Regressors" should give the winners some motivation to improve their work ethic. Well, in the worst case we would just get some extra trivial/documentation patches. ;-)

Quoted 6 lines

We of course do want to minimise the amount of overhead for each developer. I'm a strong believer in specialisation: rather than requiring that *every* developer/maintainer integrate new steps in their processes it would be better to allow them to proceed in a close-to-usual fashion and to provide for a specialist person (or team) to do the sorts of things which you're thinking about.

Makes sense... however we need to educate each and every developer about importance of the code review and proper recognition of reviewers.

Quoted 2 lines · Bartlomiej Zolnierkiewicz

Quoted 1 line · Andrew Morton

On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <[email protected]> wrote:

[--snip--]

Quoted 4 lines

Quoted 3 lines

yup, Reviewed-by: is good and I do think we should start adopting it, although I haven't thought through exactly how.

Quoted 2 lines

Adding Reviewed-by for reviews which highlighted real issues is obvious (with more detailed credits for noticed problems in the patch description).

Suppose you have modified the patch as a result of a review and you post the modified version. Is that still right to put "Reviewed-by" into it? Personally, I don't think so, because that suggests that this particular version of the patch has been reviewed and not the previous one.

Quoted 3 lines

Also when somebody reviewed your patch but the discussions it turned out that the patch is valid - the review itself was still valuable so it would be appropriate to credit the reviewer by adding Reviewed-by:.

Yes, IMO in such a case it would be appropriate to do that.

Also, the review need not lead to any negative comments from the reviewer, but in that case it's also appropriate to add a "Reviewed-by" to the patch.

Generally, if someone comments my patches, I add his/her address to the next version's CC list, which sort of documents that the reviewer was involved. Then, if the reviewer ACKs the patch, that will be recorded.

I think that for "Reviewed-by" to work correctly, we ought to have a two-stage process of accepting patches, where in the first stage the patch is reviewed and if there are no objections, the "Reviewed-by" (or "Acked-by") records are added to it in the next stage (the patch itself remains unmodified).

Quoted 26 lines

Quoted 2 lines

On my darker days I consider treating a Reviewed-by: as a prerequisite for merging. I suspect that would really get the feathers flying.

Easy to workaround by a friendly mine "Reviewed-by:" for yours "Reviewed-by:" deals (without any _proper_ review being done in reality)... ;)

Quoted 11 lines

Quoted 8 lines

I also encourage other maintainers/developers to pay more attention to adding "Acked-by"/"Reviewed-by" tags and crediting reviewers. I hope that maintainers will promote changes that have been reviewed by others by giving them priority over other ones (if the changes are on more-or-less the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high regressions/patches ratio?

Ignoring a review would be a wildly wrong thing to do. It's so unusual that I'd be suspecting a lost email or an i-sent-the-wrong-patch.

It is not unusual et all. I mean patches which affect code in such way that it is difficult to prove it's (in)correctness without doing time consuming audit.

ie. lets imagine doing a small patch affecting many drivers - you've tested it quickly on your driver/hardware, then you skip the part of verifying correctness of new code in other drivers and just push the patch

Quoted 6 lines

As a patch author you can either assume "works for me" and push the patch or do the audit (requires good understanding of the changed code and could be time consuming). It is usually quite easy to find out which approach the author has choosen - the very sparse patch description combined with the changes in code behavior not mentioned in the patch description should raise the red flag. :)

First of all, the author should have a good understanding of what he's doing and why. If there are any doubts with respect to that, the patch is likely to introduce bugs.

This also depends on who will be handling the bug reports related to the patch. If that will be the patch author, then so be it. ;-)

Quoted 6 lines

As a reviewer having enough knowledge in the area of code affected by patch you can see the potential problems but you can't prove them without doing the time consuming part. You may try to NACK the patch if you have enough power but you will end up being bypassed by not proving incorrectness of the patch (not to mention that developer will feel bad about you NACKing his patch).

Well, IMHO, the author of the patch should convince _you_ that the patch is correct, not the other way around. If you have doubts and make him think twice of the code and he still can't prove his point, this means that he doesn't understand what he's doing well enough.

Quoted 7 lines

Now the funny thing is that despite the fact that audit takes more time/knowledge then making the patch you will end up with zero credit if patch turns out to be (luckily) correct. Even if you find out issues and report them you are still on mercy of author for being credited so from personal POV you are much better to wait and fix issues after they hit mainline kernel. You have to choose between being a good citizen and preventing kernel regressions or being bastard and getting the credit. ;)

Unless you are the poor soul having to handle bug reports related to the problem.

Quoted 36 lines

If you happen to be maintainer of the affected code the choice is similar with more pros for letting the patch in especially if you can't afford the time to do audit (and by being maintainer you are guaranteed to be heavily time constrained).

I hope this makes people see the importance of proper review and proper recognition of reviewers in preventing kernel regressions.

Quoted 6 lines

As for high regressions/patches ratio: that'll be hard to calculate and tends to be dependent upon the code which is being altered rather than who is doing the altering: some stuff is just fragile, for various reasons.

One ratio which we might want to have a think about is the patches-sent versus reviews-done ratio ;)

Sounds like a good idea.

Quoted 18 lines

Quoted 11 lines

I think that we should have info about regressions integrated into SCM, i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion. This feature combined with
"Author:" info after some time should give us some very interesting statistics (Top Ten "Regressors"). It wouldn't be ideal (ie. we need some patches threshold to filter out people with 1 patch and >= 1 regression(s), we need to remember that some code areas are more difficult than the others and that patches are not equal per se etc.) however I believe than making it into Top Ten "Regressors" should give the winners some motivation to improve their work ethic. Well, in the worst case we would just get some extra trivial/documentation patches. ;-)

We of course do want to minimise the amount of overhead for each developer. I'm a strong believer in specialisation: rather than requiring that *every* developer/maintainer integrate new steps in their processes it would be better to allow them to proceed in a close-to-usual fashion and to provide for a specialist person (or team) to do the sorts of things which you're thinking about.

Quoted 2 lines

Makes sense... however we need to educate each and every developer about importance of the code review and proper recognition of reviewers.

I don't think that the education alone will be enough. IMO we need to have a system that promotes the reviewing of code.

despite the fact that audit takes
more time/knowledge then making the patch you will end up with zero credit if patch turns out to be (luckily) correct. Even if you find out issues and report them you are still on mercy of author for being credited

so from personal POV you are much better to wait and fix issues after they hit mainline kernel. You have to choose between being a good citizen and preventing kernel regressions or being bastard and getting the credit. ;)

If you happen to be maintainer of the affected code the choice is similar with more pros for letting the patch in especially if you can't afford the time to do audit (and by being maintainer you are guaranteed to be heavily time constrained).

Quoted 4 lines · Bartlomiej Zolnierkiewicz

despite the fact that audit takes
more time/knowledge then making the patch you will end up with zero credit if patch turns out to be (luckily) correct. Even if you find out issues and report them you are still on mercy of author for being credited

If we introduce a "Reviewed-by" with reasonably clear semantics (different from Signed-off-by; e.g. the reviewer is not a middle-man in patch forwarding; the reviewer might have had remaining reservations... very similar to but not entirely the same as "Acked-by" as currently defined in -mm) --- and also make the already somewhat established "Tested-by" more official, --- then the maintainers could start to make it a habit to add Reviewed-by and Tested-by.

Plus, reviewers and testers could formally reply with Reviewed-by and Tested-by lines to patch postings and even could explicitly ask the maintainer to add these lines.

Quoted 8 lines

so from personal POV you are much better to wait and fix issues after they hit mainline kernel. You have to choose between being a good citizen and preventing kernel regressions or being bastard and getting the credit. ;)

If you happen to be maintainer of the affected code the choice is similar with more pros for letting the patch in especially if you can't afford the time to do audit (and by being maintainer you are guaranteed to be heavily time constrained).

I don't think that a maintainer (who signs off on patches after all) can easily afford to take the "bastard approach". I may be naive.

In case of being maintainer "bastard approach" is more about not discouraging developers by holding patches for too long than about getting credit.

Tested-by

-----Message d'origine-----
De : [email protected] [mailto:[email protected]] De la part de Andrew Morton

Quoted 1 line · On Mon, 18 Jun 2007 01:15:15 +0200 Stefan Richter <[email protected]>…

Tested-by

Tested-by would be good too. Because over time, we will generate a list of people who own the relevant hardware and who are prepared to test changes.

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

Quoted 5 lines

-----Message d'origine-----
De : [email protected] [mailto:[email protected]] De la part de Andrew Morton

Quoted 8 lines

Quoted 1 line · On Mon, 18 Jun 2007 01:15:15 +0200 Stefan Richter <[email protected]>…

Tested-by

Tested-by would be good too. Because over time, we will generate a list of people who own the relevant hardware and who are prepared to test changes.

Why not include a user-space tool that, when invoked, if you agree to send personnal info, sends your hardware vs driver info to a web database + your email address (maybie even you .config, etc..) ... In case of help for testing new patches/finding a bug/etc.. your email could be used by maintainers to ask for help...

Quoted 3 lines

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

You would'nt even need to search in GIT. Maybie even when ever a patchset is being proposed a mail could be sent to appropriate hardware/or feature pseudo-auto-generated mailing-list?

On lkml I mostly try to follow patches/bugs associated with hardware I use. Why not try to automate the process and get more testers in?

- vin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 8 lines

Quoted 3 lines

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

You would'nt even need to search in GIT. Maybie even when ever a patchset is being proposed a mail could be sent to appropriate hardware/or feature pseudo-auto-generated mailing-list?

Quoted 3 lines

On lkml I mostly try to follow patches/bugs associated with hardware I use. Why not try to automate the process and get more testers in?

I think this is an excellent point. One data point could be a field in bugzilla to input the hardware information. Simple query can select common hardware and platform. So far it's not working when hardware is just mentioned in the text part.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

You would'nt even need to search in GIT. Maybie even when ever a patchset is being proposed a mail could be sent to appropriate hardware/or feature pseudo-auto-generated mailing-list?

Quoted 8 lines

Quoted 3 lines

On lkml I mostly try to follow patches/bugs associated with hardware I use. Why not try to automate the process and get more testers in?

I think this is an excellent point. One data point could be a field in bugzilla to input the hardware information. Simple query can select common hardware and platform. So far it's not working when hardware is just mentioned in the text part.

if it's free text it'll be useless for search ... I suppose we could do drop-downs for architecture at least? Not sure much beyond that would work ... *possibly* the common drivers, but I don't think we'd get enough coverage for it to be of use.

Quoted 17 lines · Martin Bligh

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

You would'nt even need to search in GIT. Maybie even when ever a patchset is being proposed a mail could be sent to appropriate hardware/or feature pseudo-auto-generated mailing-list?

Quoted 8 lines

Quoted 3 lines

On lkml I mostly try to follow patches/bugs associated with hardware I use. Why not try to automate the process and get more testers in?

I think this is an excellent point. One data point could be a field in bugzilla to input the hardware information. Simple query can select common hardware and platform. So far it's not working when hardware is just mentioned in the text part.

Quoted 4 lines

if it's free text it'll be useless for search ... I suppose we could do drop-downs for architecture at least? Not sure much beyond that would work ... *possibly* the common drivers, but I don't think we'd get enough coverage for it to be of use.

How about several buckets for model/BIOS version/chipset etc., at least optional, and some will be relevant some not for particular cases. But at least people will make an attempt to collect such data from their system, boards, etc.

Sorry to be a wet blanket, but I've seen those sort of things before, and they just don't seem to work, especially in the environment we're in with such a massive diversity of hardware.

Quoted 4 lines · Martin Bligh

Sorry to be a wet blanket, but I've seen those sort of things before, and they just don't seem to work, especially in the environment we're in with such a massive diversity of hardware.

I do agree. It _sounds_ like a great idea to try to control the flow of patches better, but in the end, it needs to also be easy and painfree to the people involved, and also make sure that any added workflow doesn't require even *more* load and expertise on the already often overworked maintainers..

In many cases, I think it tends to *sound* great to try to avoid regressions in the first place - but it also sounds like one of those "I wish the world didn't work the way it did" kind of things. A worthy goal, but not necessarily one that is compatible with reality.

Linus

Sure, simplicity is a key - but most of reporters on bugs are pretty professional folks (or very well rounded amateurs :) We can try still why not? the worst that can happen will be empty fields.

Maybe searching free text fields can then be implemented. Then every message exchange in bugzilla can be used for extracting such info - questions about HW specifics are asked a lot, almost in every one. It's a shame we cant' use this information. I was once searching for "VIA" and got "zero bugs found", but in reality there are hundreds! Probably something that makes sense to bring up with bugzilla project?

However, I've been working with other bugzillas (have to admit they were mostly company/corporate), where this was a required field that didn't seem to cause difficulties. I am planning to do some more research and get some more ideas from other bugzillas. I suppose we can have them discussed and revised sometime.

Quoted 3 lines

Sure, simplicity is a key - but most of reporters on bugs are pretty professional folks (or very well rounded amateurs :) We can try still why not? the worst that can happen will be empty fields.

mmm. added complexity and interface clutter for little or no benefit is what I'm trying to avoid - they did that in the IBM bugzilla and it turned into a big ugly unusable monster. You can call me either "experienced" or "bitter" depending what mood you're in ;-)

Not sure I'd agree that most of the bug submitters are all that professional, it's a very mixed bag.

Quoted 6 lines

Maybe searching free text fields can then be implemented. Then every message exchange in bugzilla can be used for extracting such info - questions about HW specifics are asked a lot, almost in every one. It's a shame we cant' use this information. I was once searching for "VIA" and got "zero bugs found", but in reality there are hundreds! Probably something that makes sense to bring up with bugzilla project?

That should work now ... seems to for me.

http://bugzilla.kernel.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=VIA&kernel_version_type=allwordssubstr&kernel_version=&bug_status=NEW&bug_status=REOPENED&bug_status=ASSIGNED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=REJECTED&bug_status=DEFERRED&bug_status=NEEDINFO&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&regression=both&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0=

Produces a metric-buttload of results. Go to the advanced search option and do "A Comment" contains the string "VIA". By default "Status" is only set to do open bugs, which you might want to change to all types.

Quoted 6 lines

However, I've been working with other bugzillas (have to admit they were mostly company/corporate), where this was a required field that didn't seem to cause difficulties. I am planning to do some more research and get some more ideas from other bugzillas. I suppose we can have them discussed and revised sometime.

Would be good, thanks. I tend to favour keeping things as simple as possible though, we have very little control over our users, and they're a very broad base. Making the barrier to entry for use as low as possible is the design we've been pursuing.

Quoted 12 lines

Quoted 6 lines

Maybe searching free text fields can then be implemented. Then every message exchange in bugzilla can be used for extracting such info - questions about HW specifics are asked a lot, almost in every one. It's a shame we cant' use this information. I was once searching for "VIA" and got "zero bugs found", but in reality there are hundreds! Probably something that makes sense to bring up with bugzilla project?

That should work now ... seems to for me.

http://bugzilla.kernel.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=VIA&kernel_version_type=allwordssubstr&kernel_version=&bug_status=NEW&bug_status=REOPENED&bug_status=ASSIGNED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=REJECTED&bug_status=DEFERRED&bug_status=NEEDINFO&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&regression=both&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0=

Quoted 3 lines

Produces a metric-buttload of results. Go to the advanced search option and do "A Comment" contains the string "VIA". By default "Status" is only set to do open bugs, which you might want to change to all types.

Yes it works great! Thanks... I'd say this should be really a default search, because first search screen is misleading - it promises find all for any "words" :)

Quoted 4 lines

Sorry to be a wet blanket, but I've seen those sort of things before, and they just don't seem to work, especially in the environment we're in with such a massive diversity of hardware.

I do agree. It _sounds_ like a great idea to try to control the flow of patches better,

but in the end, it needs to also be easy and painfree to the people involved, and also make sure that any added workflow doesn't require even *more* load and expertise on the already often overworked maintainers..

In many cases, I think it tends to *sound* great to try to avoid regressions in the first place - but it also sounds like one of those "I wish the world didn't work the way it did" kind of things. A worthy goal, but not necessarily one that is compatible with reality.

[Dear Debbug developers, i wish your ideas will be useful.]

* From: Linus Torvalds
* Newsgroups: gmane.linux.kernel
* Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)

Quoted 9 lines · Martin Bligh

Quoted 4 lines

Sorry to be a wet blanket, but I've seen those sort of things before, and they just don't seem to work, especially in the environment we're in with such a massive diversity of hardware.

I do agree. It _sounds_ like a great idea to try to control the flow of patches better,

There were some ideas, i will try to summarize:

* New Patches (or sets) need tracking, review, testing

Zero (tracking) done by sending (To, or bcc) [RFC] patch to something like [email protected] (like BTS now). Notifications will be sent to intrested maintainers (if meta-information is ok) or testers (see below)

First is mostly done by maintainers or interested individuals. Result is "Acked-by" and "Cc" entries in the next patch sent. Due to lack of tracking this things are done manually, are generally in trusted manner. But bad like <[email protected]> sometimes happens.

When patch in sent to this PTS, your lovely checkpatch/check-whatever-crap-has-being-sent tools can be set up as gatekeepers, thus making impossible stupid errors with MIME coding, line wrapping, whatever style you've came up with now in checking sent crap.

...
|-*- Feature Needed -*-
Addition, needed is hardware user tested have/had/used. Currently ``reportbug'' tool includes packed specific and system specific additions automaticly gathered and inserted to e-mail sent to BTS. (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)

...

Quoted 4 lines

but in the end, it needs to also be easy and painfree to the people involved, and also make sure that any added workflow doesn't require even *more* load and expertise on the already often overworked maintainers..

Experienced BTS users and developers. Please, correct me if i'm wrong. At least e-mail part of Debian's BTS and whole idea of it is *exactly* what is needed. Bugzilla fans, you can still use you useless pet, because Debian developers have done things, to track and e-mail states of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>
...

Quoted 24 lines · Oleg Verych

[Dear Debbug developers, i wish your ideas will be useful.]

* From: Linus Torvalds
* Newsgroups: gmane.linux.kernel
* Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)

Quoted 9 lines · Martin Bligh

Quoted 4 lines

Sorry to be a wet blanket, but I've seen those sort of things before, and they just don't seem to work, especially in the environment we're in with such a massive diversity of hardware.

I do agree. It _sounds_ like a great idea to try to control the flow of patches better,

There were some ideas, i will try to summarize:

* New Patches (or sets) need tracking, review, testing

Zero (tracking) done by sending (To, or bcc) [RFC] patch to something like [email protected] (like BTS now). Notifications will be sent to intrested maintainers (if meta-information is ok) or testers (see below)

Quoted 5 lines

First is mostly done by maintainers or interested individuals. Result is "Acked-by" and "Cc" entries in the next patch sent. Due to lack of tracking this things are done manually, are generally in trusted manner. But bad like <[email protected]> sometimes happens.

The goal is to get all patches for a maintained subsystem submitted to Linus by the maintainer.

Quoted 5 lines

When patch in sent to this PTS, your lovely checkpatch/check-whatever-crap-has-being-sent tools can be set up as gatekeepers, thus making impossible stupid errors with MIME coding, line wrapping, whatever style you've came up with now in checking sent crap.

The -mm kernel already implements what your proposed PTS would do.

Plus it gives testers more or less all patches currently pending inclusion into Linus' tree in one kernel they can test.

The problem are more social problems like patches Andrew has never heard of before getting into Linus' tree during the merge window.

Quoted 6 lines

...
|-*- Feature Needed -*-
Addition, needed is hardware user tested have/had/used. Currently ``reportbug'' tool includes packed specific and system specific additions automaticly gathered and inserted to e-mail sent to BTS. (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)

The problem is that most problems don't occur on one well-defined kind of hardware - patches often break in exactly the areas the patch author expected no problems in.

And in many cases a patch for a device driver was written due to a bug report - in such cases a tester with the hardware in question is already available.

Quoted 7 lines

...

Quoted 4 lines

but in the end, it needs to also be easy and painfree to the people involved, and also make sure that any added workflow doesn't require even *more* load and expertise on the already often overworked maintainers..

Quoted 6 lines

Experienced BTS users and developers. Please, correct me if i'm wrong. At least e-mail part of Debian's BTS and whole idea of it is *exactly* what is needed. Bugzilla fans, you can still use you useless pet, because Debian developers have done things, to track and e-mail states of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>
...

"useless pet"?
Be serious.
How many open source projects use Bugzilla and how many use the Debian BTS? And then start thinking about why the "useless pet" has so many more user...

The Debian BTS requires you to either write emails with control messages or generating control messages with external tools.

In Bugzilla the same works through a web interface.

I know both the Debian BTS and Bugzilla and although they are quite different they both are reasonable tools for their purpose.

cu
Adrian

Quoted 2 lines · Adrian Bunk

The Debian BTS requires you to either write emails with control messages or generating control messages with external tools.

Quoted 1 line

In Bugzilla the same works through a web interface.

Basic concept of Debian BTS is what i've discovered after many useless hours i spent in Bugzilla. And this is mainly because of one basic important thing, that nobody acknowledged (for newbies, like me):

* E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail (or exim) is the main communication toolset.

Can't you see that from Linux's patch sending policy?

Quoted 3 lines · Oleg Verych

Quoted 2 lines · Adrian Bunk

The Debian BTS requires you to either write emails with control messages or generating control messages with external tools.

...

Quoted 1 line

Quoted 1 line

In Bugzilla the same works through a web interface.

...

Quoted 7 lines

Basic concept of Debian BTS is what i've discovered after many useless hours i spent in Bugzilla. And this is mainly because of one basic important thing, that nobody acknowledged (for newbies, like me):

* E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail (or exim) is the main communication toolset.

Quoted 1 line

Can't you see that from Linux's patch sending policy?

That's for developers, not for users.

There are different people involved in
- patch handling,
- bug handling (bugs are reported by end-users),
therefore don't forget that PTS and BTS have different requirements.

Quoted 4 lines · Stefan Richter

There are different people involved in
- patch handling,
- bug handling (bugs are reported by end-users),
therefore don't forget that PTS and BTS have different requirements.

Sure. But if tracking was done, possible bugs where killed, user's bug report seems to depend on that patch (bisecting), why not to have a linkage here?

Current identification of problems and patch association have completely zero level of tracking or automation, while Bugzilla is believed by somebody to have positive efficiency in bug tracking.

[...]

Quoted 3 lines

Current identification of problems and patch association have completely zero level of tracking or automation, while Bugzilla is believed by somebody to have positive efficiency in bug tracking.

I, as maintainer of a small subsystem, can personally track bug--patch relationships with bugzilla just fine, on its near-zero level of automation and integration.

Nevertheless, would a more integrated bug/patch tracking system help me improve quality of my output? ---
a) Would it save me more time than it costs me to fit into the system (time that can be invested in actual debugging)?
This can only be answered after trying it.

b) Would it help me to spot mistakes in patches before I submit? No.

c) Would I get quicker feedback from testers?
That depends on whether such a system attracts testers and helps testers to work efficiently. This is also something that can only be speculated about without trying it.

The potential testers that I deal with are mostly either very non-technical persons, or persons which are experienced in their hardware/application area but *not* in kernel internals and kernel development procedures.

Quoted 3 lines

a) Would it save me more time than it costs me to fit into the system (time that can be invested in actual debugging)?
This can only be answered after trying it.

I'm not a wizard, if i will answer now: "No." [1:]

[1:] Your User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.4) Gecko/20070509 SeaMonkey/1.1.2

...

Quoted 1 line · Adrian Bunk

On Tue, Jun 19, 2007 at 06:06:47AM +0200, Oleg Verych wrote:

...

Quoted 7 lines

Quoted 5 lines

When patch in sent to this PTS, your lovely checkpatch/check-whatever-crap-has-being-sent tools can be set up as gatekeepers, thus making impossible stupid errors with MIME coding, line wrapping, whatever style you've came up with now in checking sent crap.

The -mm kernel already implements what your proposed PTS would do.

Having all-in-one patchset, like -mm is easy thing to provide interested parties with "you know what you have -- crazy development"

However [P]TS is tracking, recording, organizing tool. {1} Andrew's cron daemon easily can run script to check status of particular patch (cc, tested-by, acked-by). If patch have no TS ID, Andrew's watchdog is barking back to patch originator (with polite asking to send patch as:

* TS as "To:" target
* patch author as "Cc:" target, this is useful to require:
. author can check that copy himself with text-only pager program
(to see any MIME coding crap)
. preventing SPAM
* maybe somebody else in Cc or Bcc.)

Quoted 2 lines

Plus it gives testers more or less all patches currently pending inclusion into Linus' tree in one kernel they can test.

Crazy development{0}. Somebody knows, that comprehensively testing hibernation is their thing. I don't care about it, i care about foo, bar. Thus i can apply for example lguest patches and implement and test new asm-offset replacement, *easily*. Somebody, as you know, likes new fancy file system, and no-way other. Let them be happy testing that thing *easily*. Because another fancy NO_MHz will hang their testing bench right after best ever speed results were recorded.

Quoted 2 lines

The problem are more social problems like patches Andrew has never heard of before getting into Linus' tree during the merge window.

Linus' watchdog, as well, asking for valid patch id, or just doesn't care (in similar manner Linus does :).

So far no human is involved in social things. Do you agree?

Human power is worth and needed in particular patch discussion and testing under the participation (by Cc, acking, test-ok *e-mails*) of tracking system.

Quoted 10 lines

Quoted 6 lines

...
|-*- Feature Needed -*-
Addition, needed is hardware user tested have/had/used. Currently ``reportbug'' tool includes packed specific and system specific additions automaticly gathered and inserted to e-mail sent to BTS. (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)

The problem is that most problems don't occur on one well-defined kind of hardware - patches often break in exactly the areas the patch author expected no problems in.

I tried to test that new fancy FS, and couldn't boot because of yet-another ACPI crap. See theme{0}?

Overall testing, like Andrew does, is doubtless brave thing, but he have more time after {1}, isn't it?

Quoted 3 lines

And in many cases a patch for a device driver was written due to a bug report - in such cases a tester with the hardware in question is already available.

Tracking all possible testers (for next driver update, for example) is in question.

Quoted 13 lines

Quoted 12 lines

...

Quoted 4 lines

but in the end, it needs to also be easy and painfree to the people involved, and also make sure that any added workflow doesn't require even *more* load and expertise on the already often overworked maintainers..

Experienced BTS users and developers. Please, correct me if i'm wrong. At least e-mail part of Debian's BTS and whole idea of it is *exactly* what is needed. Bugzilla fans, you can still use you useless pet, because Debian developers have done things, to track and e-mail states of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>
...

Quoted 5 lines

"useless pet"?
Be serious.
How many open source projects use Bugzilla and how many use the Debian BTS? And then start thinking about why the "useless pet" has so many more user...

I might be stupid, but i faced it. On my amd64 512M laptop i *cannot* run mozillka any more, for example!

And i don't care, because Linus said his
opinion and i fully agree with him.

Quoted 2 lines

The Debian BTS requires you to either write emails with control messages or generating control messages with external tools.

Awesome!!! Are you wrote this reply to me by

Quoted 1 line

In Bugzilla the same works through a web interface.

web interface? If you did .........</dev/random dd bs=1 count=13..... Actually you didn't ;)

Quoted 2 lines

I know both the Debian BTS and Bugzilla and although they are quite different they both are reasonable tools for their purpose.

As you just might have seen here, i was talking about organizing, tracking, hopefully saving and redirecting useful main power. And i don't bother search e-mails i saw about Bugzilla's BD from many other prominent developers. I just know that, not from my dream or physical vacuum.

Basic concept of Debian BTS is what i've discovered after many useless hours i spent in Bugzilla. And this is mainly because of one basic important thing, that nobody acknowledged (for newbies, like me):

* E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail (or exim) is the main communication toolset.
...

Quoted 1 line · Adrian Bunk

The -mm kernel already implements what your proposed PTS would do.

Quoted 2 lines

Plus it gives testers more or less all patches currently pending inclusion into Linus' tree in one kernel they can test.

Crazy development{0}. Somebody knows, that comprehensively testing hibernation is their thing. I don't care about it, i care about foo, bar. Thus i can apply for example lguest patches and implement and test new asm-offset replacement, *easily*.

Quoted 5 lines

Crazy development{0}. Somebody knows, that comprehensively testing hibernation is their thing. I don't care about it, i care about foo, bar. Thus i can apply for example lguest patches and implement and test new asm-offset replacement, *easily*.

That's right. But the production of subsystem test patchkits is volunteer work which will be hard to unify.

I'm not saying it's impossible to reach some degree of organized production of test patchkits; after all we already have some standardization regarding patch submission which is volunteer work too.

The goal is to get all patches for a maintained subsystem submitted to Linus by the maintainer.

The -mm kernel already implements what your proposed PTS would do.

Plus it gives testers more or less all patches currently pending inclusion into Linus' tree in one kernel they can test.

The problem are more social problems like patches Andrew has never heard of before getting into Linus' tree during the merge window.

The problem is that most problems don't occur on one well-defined kind of hardware - patches often break in exactly the areas the patch author expected no problems in.

Quoted 3 lines · Adrian Bunk

The goal is to get all patches for a maintained subsystem submitted to Linus by the maintainer.

But quite frankly, anybody who aims for "perfect" without taking reality into account is just not realistic. And if that's part of the goal of some "new process", then I'm not even interested in listening to people discuss it.

If this plan cannot take reality into account, please stop Cc'ing me. I'm simply not interested.

I'm proposing kind of smart tracking, summarized before. I'm not an idealist, doing manual work. Making tools -- is what i've picked up from one of your mails. Thus hope of having more opinions on that.

Quoted 4 lines · Oleg Verych

I'm proposing kind of smart tracking, summarized before. I'm not an idealist, doing manual work. Making tools -- is what i've picked up from one of your mails. Thus hope of having more opinions on that.

Don't get me wrong, I wasn't actually responing to you personally, I was actually responding mostly to the tone of this thread.

So I was responding to things like the example from Bartlomiej about missed opportunity for taking developer review into account (and btw, I think a little public shaming might not be a bad idea - I believe more in *social* rules than in *technical* rules), and I'm responding to some of the commentary by Adrian and others about "no regressions *ever*".

These are things we can *wish* for. But the fact that we migth wish for them doesn't actually mean that they are really good ideas to aim for in practice.

Let me put it another way: a few weeks ago there was this big news story in the New York Times about how "forgetting" is a very essential part about remembering, and people passed this around as if it was a big revelation. People think that people with good memories have a "good thing".

And personally, I was like "Duh".

Good memory is not about remembering everything. Good memory is about forgetting the irrelevant, so that the important stuff stands out and you *can* remember it. But the big deal is that yes, you have to forget stuff, and that means that you *will* miss details - but you'll hopefully miss the stuff you don't care for. The keyword being "hopefully". It works most of the time, but we all know we've sometimes been able to forget a detail that turned out to be crucial after all.

So the *stupid* response to that is "we should remember everything". It misses the point. Yes, we sometimes forget even important details, but it's *so* important to forget details, that the fact that our brains occasionally forget things we later ended up needing is still *much* preferable to trying to remember everything.

The same tends to be true of bug hunting, and regression tracking.

There's a lot of "noise" there. We'll never get perfect, and I'll argue that if we don't have a system that tries to actively *remove* noise, we'll just be overwhelmed. But that _inevitably_ means that sometimes there was actually a signal in the noise that we ended up removing, because nobody saw it as anything but noise.

So I think people should concentrate on turning "noise" into "clear signal", but at the same time remember that that inevitably is a "lossy" transformation, and just accept the fact that it will mean that we occasionally make "mistakes".

This is why I've been advocating bugzilla "forget" stuff, for example. I tend to see bugzilla as a place where noise accumulates, rather than a place where noise is made into a signal.

Which gets my to the real issue I have: the notion of having a process for _tracking_ all the information is actually totally counter-productive, if a big part of the process isn't also about throwing noise away.

We don't want to "save" all the crud. I don't want "smart tracking" to keep track of everything. I want "smart forgetting", so that we are only left with the major signal - the stuff that matters.

Linus

Quoted 4 lines

I'm proposing kind of smart tracking, summarized before. I'm not an idealist, doing manual work. Making tools -- is what i've picked up from one of your mails. Thus hope of having more opinions on that.

Don't get me wrong, I wasn't actually responing to you personally, I was actually responding mostly to the tone of this thread.

...
This is why I've been advocating bugzilla "forget" stuff, for example. I tend to see bugzilla as a place where noise accumulates, rather than a place where noise is made into a signal.

Which gets my to the real issue I have: the notion of having a process for _tracking_ all the information is actually totally counter-productive, if a big part of the process isn't also about throwing noise away.

We don't want to "save" all the crud. I don't want "smart tracking" to keep track of everything. I want "smart forgetting", so that we are only left with the major signal - the stuff that matters.

Linus

Quoted 9 lines · Linus Torvalds

...
This is why I've been advocating bugzilla "forget" stuff, for example. I tend to see bugzilla as a place where noise accumulates, rather than a place where noise is made into a signal.

Which gets my to the real issue I have: the notion of having a process for _tracking_ all the information is actually totally counter-productive, if a big part of the process isn't also about throwing noise away.

Quoted 3 lines

We don't want to "save" all the crud. I don't want "smart tracking" to keep track of everything. I want "smart forgetting", so that we are only left with the major signal - the stuff that matters.

Even generating the perfect signal is a complete waste of time if there's no recipient for the signal...

Quoted 10 lines · Adrian Bunk

Quoted 9 lines · Linus Torvalds

...
This is why I've been advocating bugzilla "forget" stuff, for example. I tend to see bugzilla as a place where noise accumulates, rather than a place where noise is made into a signal.

Which gets my to the real issue I have: the notion of having a process for _tracking_ all the information is actually totally counter-productive, if a big part of the process isn't also about throwing noise away.

Quoted 6 lines

Quoted 3 lines

We don't want to "save" all the crud. I don't want "smart tracking" to keep track of everything. I want "smart forgetting", so that we are only left with the major signal - the stuff that matters.

Even generating the perfect signal is a complete waste of time if there's no recipient for the signal...

My argument is that *if* we had "more signal, less noise", we'd probably get more people looking at it.

In fact, I guarantee that's the case. You may not be 100% happy with the regression list, but every single maintainer/developer I've talked to has said they appreciated it and it made it easier (and thus more likely) for them to actually look at what the outstanding issues were.

Linus

On Tue, 19 Jun 2007, Adrian Bunk wrote:
...

Quoted 7 lines

The -mm kernel already implements what your proposed PTS would do.

Plus it gives testers more or less all patches currently pending inclusion into Linus' tree in one kernel they can test.

The problem are more social problems like patches Andrew has never heard of before getting into Linus' tree during the merge window.

Not really. The "problem" boils down to this:

	[torvalds@woody linux]$ git-rev-list --all --since=100.days.ago | wc -l
	7147
	[torvalds@woody linux]$ git-rev-list --no-merges --all --since=100.days.ago | wc -l
	6768

ie over the last hundred days, we have averaged over 70 changes per day, and even ignoring merges and only looking at "pure patches" we have more than an average of 65 patches per day. Every day. Day in and day out.

That translates to five hundred commits a week, two _thousand_ commits per month, and 25 thousand commits per year. As a fairly constant stream.

Will mistakes happen? Hell *yes*.

And I'd argue that any flow that tries to "guarantee" that mistakes don't happen is broken. It's a sure-fire way to just frustrate people, simply because it assumes a level of perfection in maintainers and developers that isn't possible.

The accepted industry standard for bug counts is basically one bug per a thousand lines of code. And that's for released, *debugged* code.

Yes, we should aim higher. Obviously. Let's say that we aim for 0.1 bugs per KLOC, and that we actually aim for that not just in _released_ code, but in patches.

What does that mean?

Do the math:

	git log -M -p --all --since=100.days.ago | grep '^+' | wc -l

That basically takes the last one hundred days of development, shows it all as patches, and just counts the "new" lines. It takes about ten seconds to run, and returns 517252 for me right now.

That's *over*half*a*million* lines added or changed!

And even with the expectation that we do ten times better than what is often quoted as an industry average, and even with the expectation that this is already fully debugged code, that's at least 50 bugs in the last one hundred days.

Yeah, we can be even more stringent, and actually subtract the number of lines _removed_ (274930), and assume that only *new* code contains bugs, and that's still just under a quarter million purely *added* lines, and maybe we'd expect just new 24 bugs in the last 100 days.

[ Argument: some of the old code also contained bugs, so the lines added
to replace it balance out. Counter-argument: new code is less well tested by *definition* than old code, so.. Counter-counter-argument: the new code was often added to _fix_ a bug, so the code removed had an even _higher_ bug rate than normal code..

End result? We don't know. This is all just food for thought. ]

So here's the deal: even by the most *stringent* reasonable rules, we add a new bug every four days. That's just something that people need to accept. The people who say "we must never introduce a regression" aren't living on planet earth, they are living in some wonderful world of Blarney, where mistakes don't happen, developers are perfect, hardware is perfect, and maintainers always catch things.
...

Linus

* From: Linus Torvalds
* Newsgroups: gmane.linux.kernel
* Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)

Quoted 2 lines

I do agree. It _sounds_ like a great idea to try to control the flow of patches better,

There were some ideas, i will try to summarize:

* New Patches (or sets) need tracking, review, testing

Zero (tracking) done by sending (To, or bcc) [RFC] patch to something like [email protected] (like BTS now). Notifications will be sent to intrested maintainers (if meta-information is ok) or testers (see below)

-----Message d'origine-----
De : Natalie Protasevich [mailto:[email protected]] Envoyé : 18 juin 2007 18:56

Quoted 17 lines · Martin Bligh

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

You would'nt even need to search in GIT. Maybie even when ever a patchset is being proposed a mail could be sent to appropriate hardware/or feature pseudo-auto-generated mailing-list?

Quoted 8 lines

Quoted 3 lines

On lkml I mostly try to follow patches/bugs associated with hardware I use. Why not try to automate the process and get more testers in?

I think this is an excellent point. One data point could be a field in bugzilla to input the hardware information. Simple query can select common hardware and platform. So far it's not working when hardware is just mentioned in the text part.

Quoted 5 lines

if it's free text it'll be useless for search ... I suppose we could do drop-downs for architecture at least? Not sure much beyond that would work ... *possibly* the common drivers, but I don't think we'd get enough coverage for it to be of use.

How about several buckets for model/BIOS version/chipset etc., at least optional, and some will be relevant some not for particular cases. But at least people will make an attempt to collect such data from their system, boards, etc.

Quoted 27 lines

-----Message d'origine-----
De : Natalie Protasevich [mailto:[email protected]] Envoyé : 18 juin 2007 18:56

Quoted 17 lines · Martin Bligh

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

So if you make changes to random-driver.c you can do `git-log random-driver.c|grep Tested-by" to find people who can test your changes for you.

You would'nt even need to search in GIT. Maybie even when ever a patchset is being proposed a mail could be sent to appropriate hardware/or feature pseudo-auto-generated mailing-list?

Quoted 8 lines

Quoted 3 lines

On lkml I mostly try to follow patches/bugs associated with hardware I use. Why not try to automate the process and get more testers in?

I think this is an excellent point. One data point could be a field in bugzilla to input the hardware information. Simple query can select common hardware and platform. So far it's not working when hardware is just mentioned in the text part.

Quoted 5 lines

if it's free text it'll be useless for search ... I suppose we could do drop-downs for architecture at least? Not sure much beyond that would work ... *possibly* the common drivers, but I don't think we'd get enough coverage for it to be of use.

Quoted 4 lines

How about several buckets for model/BIOS version/chipset etc., at least optional, and some will be relevant some not for particular cases. But at least people will make an attempt to collect such data from their system, boards, etc.

How about an easy way to send multiple hardware profiles to your bugzilla user account simultaniously linked to an online pciutils database and/or an hardware list database similar to overclocking web sites and why not even with a link to the git repository when possible?

A some sort of really usefull "send your profile" of RHN that would link the driver with the discovered hardware and add you to appropriate mailing lists to test patches/help reproducing & solving problems/etc.

In the end plenty of statistics and hardware compatibility list could be made. For example, that would make my life easier knowing what level of compatibility Linux can offer for old HP9000 K-boxes that we still have running at the office and presumably get people to contact to get help?

On 6/18/07, Fortier,Vincent [Montreal] <[email protected]> wrote:

Quoted 5 lines

In the end plenty of statistics and hardware compatibility list could be made. For example, that would make my life easier knowing what level of compatibility Linux can offer for old HP9000 K-boxes that we still have running at the office and presumably get people to contact to get help?

This is definitely something that can be done (and should) - well, especially having ability search by certain criteria - then all sorts of statistics and databases can be created.

Everything that helps to find a way to work on a patch and to test easier should be done to make bug fixing easier and even possible. Often times the most knowledgeable people are not able to make quick fix just because there is no way to reproduce the case or get access to HW.

Hi,

Quoted 39 lines · Adrian Bunk

Quoted 9 lines · Michal Piotrowski

On 17/06/07, Adrian Bunk <[email protected]> wrote:
...

Quoted 6 lines

Fine with me, but:

There are not so simple cases like big infrastructure patches with 20 other patches in the tree depending on it causing a regression, or even worse, a big infrastructure patch exposing a latent old bug in some completely different area of the kernel.

Quoted 5 lines

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all regressions" (and for a user it doesn't matter whether a new regression is introduced or an old regression exposed).

It could be we simply agree on this one. ;-)

Quoted 6 lines

Removal of 20 patches will be painful, but sometimes you need to "choose minor evil to prevent a greater one" [1].

Quoted 2 lines

And we should be aware that reverting is only a workaround for the real problem which lies in our bug handling.

...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the symptoms?

There's now much money in the Linux market, and the kernel quality problems might result in real costs in the support of companies like IBM, SGI, Redhat or Novell (plus it harms the Linux image which might result in lower revenues).

Quoted 6 lines

If [1] this is true, it might even pay pay off for them to each assign X man hours per month of experienced kernel developers to upstream kernel bug handling?

This is just a wild thought and it might be nonsense - better suggestions for solving our quality problems would be highly welcome...

IMO we should concentrate more on preventing regressions than on fixing them. In the long-term preventing bugs is cheaper than fixing them afterwards.

First let me tell you all a little story...

Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs in it (in other words I potentially prevented three regressions). I also asked for more thorough verification of the patch as I suspected that it may have more problems. The author fixed the issues and replied that he hasn't done the full verification yet but he doesn't suspect any problems...

Fast forward...

Year later I discover that the final version of the patch hit the mainline. I don't remember ever seeing the final version in my mailbox (there are no
cc: lines in the patch description) and I saw that I'm not credited in the patch description. However the worse part is that it seems that the full
verification has never been done. The result? Regression in the release
kernel (exactly the issue that I was worried about) which required three patches and over a month to be fixed completely. It seems that a year was not enough to get this ~70k _cleanup_ patch fully verified and tested (it hit -mm soon before being merged)...

From reviewer's POV: I have invested my time into review, discovered real issues and as a reward I got no credit et all and extra frustration from the fact that part of my review was forgotten/ignored (the part which resulted in real regression in the release kernel)... Oh and in the past the said developer has already been asked (politely in private message) to pay more attention to his changes (after I silently fixed some other regression caused by his other patch).

But wait there is more, I happend to be the maintainer of the subsystem which got directly hit by the issue and I was getting bugreports from the users about the problem... :-)

It wasn't my first/last bad experience as a reviewer... finally I just gave up on reviewing other people patches unless they are stricly for IDE subsystem.

The moral of the story is that currently it just doesn't pay off to do code reviews. From personal POV it pays much more to wait until buggy patch hits the mainline and then fix the issues yourself (at least you will get some credit). To change this we should put more ephasize on the importance of code reviews by "rewarding" people investing their time into reviews and "rewarding" developers/maintainers taking reviews seriously.

We should credit reviewers more, sometimes it takes more time/knowledge to review the patch than to make it so getting near to zero credit for review doesn't sound too attractive. Hmm, wait it can be worse - your review may be ignored... ;-)

From my side I think I'll start adding less formal "Reviewed-by" to IDE patches even if the review resulted in no issues being found (in additon to explicit "Acked-by" tags and crediting people for finding real issues - which I currently always do as a way for showing my appreciation for their work).

I also encourage other maintainers/developers to pay more attention to adding "Acked-by"/"Reviewed-by" tags and crediting reviewers. I hope that maintainers will promote changes that have been reviewed by others by giving them priority over other ones (if the changes are on more-or-less the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high regressions/patches ratio?

I think that we should have info about regressions integrated into SCM, i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion. This feature combined with
"Author:" info after some time should give us some very interesting statistics (Top Ten "Regressors"). It wouldn't be ideal (ie. we need some patches threshold to filter out people with 1 patch and >= 1 regression(s), we need to remember that some code areas are more difficult than the others and that patches are not equal per se etc.) however I believe than making it into Top Ten "Regressors" should give the winners some motivation to improve their work ethic. Well, in the worst case we would just get some extra trivial/documentation patches. ;-)

Sorry for a bit chaotic mail but I hope that message is clear.

Thanks,
Bart

Quoted 2 lines · Oleg Verych

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

IOW, I really don't think we _resolved_ anything, although the work that Adrian started is continuing through the wiki and other people trying to track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the long run. I think everybody wants a better bug-tracking system, but whether something that makes people satisfied can even be built is open. It sure doesn't seem to exist right now ;)

Quoted 7 lines · Linus Torvalds

Quoted 2 lines · Oleg Verych

I'm seeing this long (198) thread and just have no idea how it has ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

Quoted 8 lines

IOW, I really don't think we _resolved_ anything, although the work that Adrian started is continuing through the wiki and other people trying to track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the long run. I think everybody wants a better bug-tracking system, but whether something that makes people satisfied can even be built is open. It sure doesn't seem to exist right now ;)

I know you hate bugzilla ... but at least I can try to make that bit of the process work better.

The new version just rolled out does have a simple "regression" checkbox (and you can search on it), which will hopefully help people keep track of the ones already in bugzilla more easily.

Thanks to Jon T, Dave J et al. for helping to figure out methods and implement them.

Yes, good work, thanks a lot for it! The new interface is much better and more useful.

Greetings,
Rafael

PS
BTW, would that be possible to create the "Hibernation/Suspend" subcategory of "Power Management" that I asked for some time ago, please? :-)

Quoted 3 lines

My personal experience with bugzilla is that it's very unfriendly to reporters. IMHO it's suitable for tracking unresolved problems along with debug patches, system information etc., but not for _reporting_ new ones.

What did you find unfriendly?

While I also cannot say I love it I don't find it any less unfriendly than an email.

Besides the primary point of bug tracking is not to be friendly to someone, but to (a) fix the bugs and (b) know how many bugs there for a given release. Any replacement would need to solve this problem too.

Email does not solve it as far as I can see.

- You are required to select a category and 'component' for your report, which often is difficult (especially if you're not a kernel expert)
- You need to have a bugzilla account (or to create one, if you don't)

Quoted 5 lines · Andi Kleen

Quoted 3 lines

My personal experience with bugzilla is that it's very unfriendly to reporters. IMHO it's suitable for tracking unresolved problems along with debug patches, system information etc., but not for _reporting_ new ones.

What did you find unfriendly?

- You are required to select a category and 'component' for your report, which often is difficult (especially if you're not a kernel expert)

- You need to have a bugzilla account (or to create one, if you don't)
- If you want to add an address to the CC list, it must be known to bugzilla and there's no (obvious) way to check which addresses are known (bugzilla rejects the report if there's a 'wrong' email address in the list) [IMO this is really really broken.]

- You are asked to provide many details that need not be relevant and casual reporters don't know that they can skip this part
- Attaching files is tedious and referring to attachments unintuitive

And I think they are two _totally_ conceptually different things. You report a bug to let somebody know that there's a problem and this doesn't necessarily

For this reason there should be a simple means of filing initial bug reports with someone to look at them and forward them to appropriate people who will decide if the problem needs to be tracked. If they do, it's time to use bugzilla. Not earlier.

You are right, email is not suitable for tracking bugs. Still, it works quite well as a means of sending initial reports.

Quoted 9 lines · Rafael J. Wysocki

Quoted 5 lines · Andi Kleen

Quoted 3 lines

My personal experience with bugzilla is that it's very unfriendly to reporters. IMHO it's suitable for tracking unresolved problems along with debug patches, system information etc., but not for _reporting_ new ones.

What did you find unfriendly?

- You are required to select a category and 'component' for your report, which often is difficult (especially if you're not a kernel expert)

Usually there is other and then someone else figures it out.

Quoted 5 lines

- You need to have a bugzilla account (or to create one, if you don't)
- If you want to add an address to the CC list, it must be known to bugzilla and there's no (obvious) way to check which addresses are known (bugzilla rejects the report if there's a 'wrong' email address in the list) [IMO this is really really broken.]

The Novell bugzilla actually has that fixed. You have a search email button to look up addresses. Perhaps that feature will be ported someday into the kernel.org one (I would like to have it too)

Quoted 3 lines

- You are asked to provide many details that need not be relevant and casual reporters don't know that they can skip this part
- Attaching files is tedious and referring to attachments unintuitive

Anyways that are mostly all detail (except the registration requirement) that could be probably all easily fixed.

Quoted 2 lines

And I think they are two _totally_ conceptually different things. You report a bug to let somebody know that there's a problem and this doesn't necessarily

The problem is we need a way to route those reports to the right people. Routing it to a single person or broadcasting it just doesn't scale. And the best way I know of is to use some database that keeps track of the state.

That is what bugzilla is essentially.

Quoted 4 lines

For this reason there should be a simple means of filing initial bug reports with someone to look at them and forward them to appropriate people who will decide if the problem needs to be tracked. If they do, it's time to use bugzilla. Not earlier.

The only sane way to do that would be to save them somewhere and keep a list and then let a group of people process them.

From lkml.org, for example. Why don't we use that? The only missing piece

Hmm, wait... sounds like bugzilla, doesn't it?

Quoted 1 line

You are right, email is not suitable for tracking bugs.

Quoted 1 line

Still, it works quite well as a means of sending initial reports.

I disagree. It works small scale but does not really scale well.

...
But emailed reports _are_ saved anyway and we _know_ how to get a copy. From lkml.org, for example. Why don't we use that? The only missing piece is the 'keep a list' thing, but that's not a rocket science, IMHO.

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

_Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
keep track of each thread separately, so you can browse any of them at any time. In particular, you can see the _history_ of each bug report sent to LKML if you have a link to any message in its thread.

Really, if we ask reporters to put '[BUG]' in the subjects of their messages, you'll even be able to use the lkml.org archives plus wget and a couple of shell scripts to cherry pick the links to all bug reports sent to the list within a given time interval.

All of this functionality is out there already.
...

Greetings,
Rafael

Quoted 19 lines · Rafael J. Wysocki

...
But emailed reports _are_ saved anyway and we _know_ how to get a copy. From lkml.org, for example. Why don't we use that? The only missing piece is the 'keep a list' thing, but that's not a rocket science, IMHO.

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

_Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
keep track of each thread separately, so you can browse any of them at any time. In particular, you can see the _history_ of each bug report sent to LKML if you have a link to any message in its thread.

Really, if we ask reporters to put '[BUG]' in the subjects of their messages, you'll even be able to use the lkml.org archives plus wget and a couple of shell scripts to cherry pick the links to all bug reports sent to the list within a given time interval.

Quoted 2 lines

All of this functionality is out there already.
...

How can I get the functionality "show me all unfixed SATA bugs"?

That's one of the important functionalities of every bug tracking system.

Quoted 20 lines · Adrian Bunk

Quoted 19 lines · Rafael J. Wysocki

...
But emailed reports _are_ saved anyway and we _know_ how to get a copy. From lkml.org, for example. Why don't we use that? The only missing piece is the 'keep a list' thing, but that's not a rocket science, IMHO.

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

_Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
keep track of each thread separately, so you can browse any of them at any time. In particular, you can see the _history_ of each bug report sent to LKML if you have a link to any message in its thread.

Really, if we ask reporters to put '[BUG]' in the subjects of their messages, you'll even be able to use the lkml.org archives plus wget and a couple of shell scripts to cherry pick the links to all bug reports sent to the list within a given time interval.

Quoted 7 lines

Quoted 2 lines

All of this functionality is out there already.
...

How can I get the functionality "show me all unfixed SATA bugs"?

That's one of the important functionalities of every bug tracking system.

That's the missing piece, obviously.

BTW, I didn't want to say that one could entirely replace a bug-tracking system with tracking the LKML archives. What I wanted to say was that the email messages sent to the LKML were easily trackable and could be hooked up into a bug-tracking system, for example with the help of URLs.

In such a setup people could send initial reports to the LKML and the links to these messages might be put into a bug-tracking system as soon as it turned out that the bugs were worthy of tracking.

Greetings,
Rafael

Quoted 36 lines · Rafael J. Wysocki

Quoted 20 lines · Adrian Bunk

Quoted 19 lines · Rafael J. Wysocki

...
But emailed reports _are_ saved anyway and we _know_ how to get a copy. From lkml.org, for example. Why don't we use that? The only missing piece is the 'keep a list' thing, but that's not a rocket science, IMHO.

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

_Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
keep track of each thread separately, so you can browse any of them at any time. In particular, you can see the _history_ of each bug report sent to LKML if you have a link to any message in its thread.

Really, if we ask reporters to put '[BUG]' in the subjects of their messages, you'll even be able to use the lkml.org archives plus wget and a couple of shell scripts to cherry pick the links to all bug reports sent to the list within a given time interval.

Quoted 7 lines

Quoted 2 lines

All of this functionality is out there already.
...

How can I get the functionality "show me all unfixed SATA bugs"?

That's one of the important functionalities of every bug tracking system.

That's the missing piece, obviously.

BTW, I didn't want to say that one could entirely replace a bug-tracking system with tracking the LKML archives. What I wanted to say was that the email messages sent to the LKML were easily trackable and could be hooked up into a bug-tracking system, for example with the help of URLs.

Quoted 3 lines

In such a setup people could send initial reports to the LKML and the links to these messages might be put into a bug-tracking system as soon as it turned out that the bugs were worthy of tracking.

Who is doing this "might be put", and why don't you start with asking the submitter to submit bugs in a bug tracking system and forward the bug report from the bug tracking system (manually or automatically) to the developers and linux-kernel?

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

Quoted 3 lines · Rafael J. Wysocki

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

Last time I did this, bugzilla at osdl.org won't let me add original reporter to goddamn CC list. It would be el neat, because not everyone followed instructions and forwarding emails between reporter and bugzilla sucks.

Quoted 5 lines · Alexey Dobriyan

Quoted 3 lines · Rafael J. Wysocki

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

Quoted 4 lines

Last time I did this, bugzilla at osdl.org won't let me add original reporter to goddamn CC list. It would be el neat, because not everyone followed instructions and forwarding emails between reporter and bugzilla sucks.

That's related to what I said before. The requirement that the addresses on the CC list must be 'known' to bugzilla is deadly wrong in every case I can imagine.

Quoted 6 lines · On Mon, 30 Apr 2007 00:09:06 +0200 "Rafael J. Wysocki" <[email protected]> wrote:

Quoted 5 lines · Alexey Dobriyan

Quoted 3 lines · Rafael J. Wysocki

[For example, you can create a bugzilla entry with a link to the lkml.org copy of the relevant message, so why to require the reporter to file the report with the bugzilla himself?]

Quoted 8 lines

Quoted 4 lines

Last time I did this, bugzilla at osdl.org won't let me add original reporter to goddamn CC list. It would be el neat, because not everyone followed instructions and forwarding emails between reporter and bugzilla sucks.

That's related to what I said before. The requirement that the addresses on the CC list must be 'known' to bugzilla is deadly wrong in every case I can imagine.

But unknown-to-bugzilla email addresses are accepted when they're sent to [email protected]. This is why I'll very often switch a bug report to email, copying individuals and mailing lists and bugme-daemon. Then bugzilla just sits silently in the background recording everything.

But once a bug has switched to email, it needs to stay there - it would be bad if someone were to update the bug via the web UI because none of the emailed participants would know of the update. So i'll often explicitly ask "please follow up via emailed reply-to-all".

It's not great, but there's certainly enough material here for people to get in and work on the bug, should they be so inclined.

My overall approach with this stuff is: short-term bugs are handled via email and long-term ones are tracked in buzilla.

Hence someone (Hi, Mom) needs to track all the emailed-only bug reports and get them filed in bugzilla once they go stale.

- Ask more people to just categorize and reassign bugs (anybody interested?)

- Give more people in bugzilla the power to reassign arbitary bugs (bugzilla maintainers would need to do that)

Quoted 2 lines

- You are required to select a category and 'component' for your report, which often is difficult (especially if you're not a kernel expert)

Usually there is other and then someone else figures it out.