Skip to contentSkip to conversation

Tracking and crediting bug reporters

DISCUSSION Jonathan Corbet 15 participants 23 messages May 12, 2008 – May 21, 2008

lore
Share & tools
lore
#0 lore raw

Several members of the Linux Foundation's Technical Advisory Board recently got together with Andrew Morton to talk about kernel quality issues. One of the things which came out of that meeting was a desire to improve incentives for people who report bugs. Clearly, actually fixing those bugs would qualify; nobody has lost sight of that. But it was suggested that the creation and publication of statistics on bug reporting would also help.

One way to do this might be for Andrew (being the only one who actually reads every message posted on the list) to keep a spreadsheet along with everything else he does. That idea did not go over very well.

So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.

If these tags go into the commit messages in any sort of consistent way, it should be possible generate the usual sort of statistics from them. I'll then happily publicize them next to the traditional lists of people who are adding new bugs. The result will certainly be fame, fortune, and job offers for the people at the top of the list. Or something like that.

If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.

Thoughts?

jon

#1 lore raw

[Jonathan Corbet - Mon, May 12, 2008 at 11:27:52AM -0600]

| Several members of the Linux Foundation's Technical Advisory Board recently
| got together with Andrew Morton to talk about kernel quality issues.  One
| of the things which came out of that meeting was a desire to improve
| incentives for people who report bugs.  Clearly, actually fixing those bugs
| would qualify; nobody has lost sight of that.  But it was suggested that
| the creation and publication of statistics on bug reporting would also
| help.
| 
| One way to do this might be for Andrew (being the only one who actually
| reads every message posted on the list) to keep a spreadsheet along with
| everything else he does.  That idea did not go over very well.
| 
| So here's what we would like to try instead.  Whenever somebody sends up a
| patch fixing a reported bug, the name of the person who reported the bug
| would be immortalized with this tag:
| 
| 	Reported-by: A. Bug Reporter <[email protected]>
| 
| In particular, reporters who work with the developers toward the resolution
| of the bug should be thanked in this way.  If we wanted to take things
| further, perhaps we could add a Bisected-by: tag for really hard-core
| helpers. 
| 
| If these tags go into the commit messages in any sort of consistent way, it
| should be possible generate the usual sort of statistics from them.  I'll
| then happily publicize them next to the traditional lists of people who are
| adding new bugs.  The result will certainly be fame, fortune, and job
| offers for the people at the top of the list.  Or something like that.
| 
| If the rest of the community is agreeable, it would be nice to make an
| immediate start on this; it's not yet too late to get reasonable data for
| the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
| 
| Thoughts?
| 
| jon
| 

If my opinion does worth somehow - I'm absolutely agree!

(btw, it seems we forget the main tag ever was - Bug-made-by: ;)

- Cyrill -

#2 lore raw
In reply to Jonathan Corbet
Quoted 5 lines · On Mon, 12 May 2008 11:27:52 -0600 Jonathan Corbet <[email protected]> wrote:

So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

I like this idea. If people help with kernel development, they deserve kudos.

Signature · 1 line
All Rights Reversed
#3 lore raw
In reply to Jonathan Corbet
Quoted 5 lines

If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.

Thoughts?

I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws. So we need to document that it is added with permission of the reporter only.

Alan

#4 lore raw

From: Alan Cox <[email protected]>
Date: Mon, 12 May 2008 19:20:19 +0100

Quoted 10 lines
Quoted 5 lines

If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.

Thoughts?

I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws. So we need to document that it is added with permission of the reporter only.

This is why I personally do not accept private bug reports or kernel questions, I always direct them to post to the appropriate public mailing list and CC: me if appropriate.

That solves the whole "privacy" issue quite neatly.

#5 lore raw
Quoted 5 lines

This is why I personally do not accept private bug reports or kernel questions, I always direct them to post to the appropriate public mailing list and CC: me if appropriate.

That solves the whole "privacy" issue quite neatly.

Not really. About half the bugs I deal with and fix are from customers of Red Hat who have differing expectations.

Alan

#6 lore raw
Quoted 8 lines · Alan Cox
Quoted 5 lines

This is why I personally do not accept private bug reports or kernel questions, I always direct them to post to the appropriate public mailing list and CC: me if appropriate.

That solves the whole "privacy" issue quite neatly.

Not really. About half the bugs I deal with and fix are from customers of Red Hat who have differing expectations.

Those would just default to not being credited. Where is the problem?

-Andi

#7 lore raw
Quoted 1 line

Those would just default to not being credited. Where is the problem?

We need to be sure that people understand that when submitting patches they must not put Discovered-by: lines in without asking. This is essentially a documentation matter which is why I directed the original reply to Jonathan.

Alan

#8 lore raw
Quoted 6 lines · Alan Cox
Quoted 1 line

Those would just default to not being credited. Where is the problem?

We need to be sure that people understand that when submitting patches they must not put Discovered-by: lines in without asking. This is essentially a documentation matter which is why I directed the original reply to Jonathan.

I tend to put the credit in the patch description text, /without/ an email address by default, since I don't feel as comfortable stamping users' email addresses into the kernel changelog without their permission, and (I'm lazy) asking permission is another email round-trip. So my format is

blah blah blah patch description blah blah blah blah blah blah patch description blah blah blah blah blah blah patch description blah blah blah

Bug found by Lucifer Smith.

Regards,

Jeff

#9 lore raw
In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 5 lines

If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.

Thoughts?

I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws.

IANAL. Unless it's a public information, no? (As reports are.)

Otherwise we would be in a big trouble, since we do this as far as can git history tell at least.

#10 lore raw
Quoted 5 lines
Quoted 3 lines

I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws.

IANAL. Unless it's a public information, no? (As reports are.)

If they made it public fine - although building a database of that requires care. That was my point. You need specific permission of the bug reporter. Given bug reporters may have personal reasons (including 'works at Microsoft') and business ones (from 'company policy' to 'three letter agency') for not wanting the information public.

Quoted 2 lines

Otherwise we would be in a big trouble, since we do this as far as can git history tell at least.

No: The Signed-off-by in GIT references the 1.1 contributor agreement which specifically deals with that case.

#11 lore raw
Quoted 11 lines · Alan Cox
Quoted 5 lines
Quoted 3 lines

I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws.

IANAL. Unless it's a public information, no? (As reports are.)

If they made it public fine - although building a database of that requires care. That was my point. You need specific permission of the bug reporter. Given bug reporters may have personal reasons (including 'works at Microsoft') and business ones (from 'company policy' to 'three letter agency') for not wanting the information public.

Well, don't the mailing list archives build such data bases already?

By posting a report to a mailing list the archives of which are publicly available, and the LKML is one of those, you make your name and email address information publicly available and recorded forever anyway, in the context of the report.

Thanks,
Rafael

#12 lore raw
In reply to Jonathan Corbet
Quoted 29 lines · Jonathan Corbet

Several members of the Linux Foundation's Technical Advisory Board recently got together with Andrew Morton to talk about kernel quality issues. One of the things which came out of that meeting was a desire to improve incentives for people who report bugs. Clearly, actually fixing those bugs would qualify; nobody has lost sight of that. But it was suggested that the creation and publication of statistics on bug reporting would also help.

One way to do this might be for Andrew (being the only one who actually reads every message posted on the list) to keep a spreadsheet along with everything else he does. That idea did not go over very well.

So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.

If these tags go into the commit messages in any sort of consistent way, it should be possible generate the usual sort of statistics from them. I'll then happily publicize them next to the traditional lists of people who are adding new bugs. The result will certainly be fame, fortune, and job offers for the people at the top of the list. Or something like that.

Quoted 5 lines

If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.

Thoughts?

I like the idea.

Thanks,
Rafael

#13 lore raw
In reply to Jonathan Corbet
Quoted 11 lines · Jonathan Corbet

...
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.

I assume it will be valid for one person to have two or all three tags in one commit (not uncommon for some of the stuff I'm doing)?

Quoted 10 lines

If these tags go into the commit messages in any sort of consistent way, it should be possible generate the usual sort of statistics from them. I'll then happily publicize them next to the traditional lists of people who are adding new bugs. The result will certainly be fame, fortune, and job offers for the people at the top of the list. Or something like that.

If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
...

In which format do you want data for commits already in 2.6.26?

Quoted 1 line

jon

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#14 lore raw

[Operating in delayed response mode, sorry...]

Quoted 1 line · Adrian Bunk

In which format do you want data for commits already in 2.6.26?

I hadn't thought about trying to backfill data for existing commits; that sounds like a fair amount of work for coverage which could still be somewhat spotty. If you have the data at hand, though, then a simple file of the form

commit-id full-name <email>

could be dealt with. But is it really worth the effort?

Thanks,

jon

#15 lore raw
Quoted 4 lines · Jonathan Corbet

[Operating in delayed response mode, sorry...]

Adrian Bunk <[email protected]> wrote:

Quoted 8 lines
Quoted 1 line

In which format do you want data for commits already in 2.6.26?

I hadn't thought about trying to backfill data for existing commits; that sounds like a fair amount of work for coverage which could still be somewhat spotty. If you have the data at hand, though, then a simple file of the form

commit-id full-name <email>

How to indicate whether it's a Reported-by or a Bisected-by?

Quoted 1 line

could be dealt with. But is it really worth the effort?

Not sure whether you don't consider it important that the data you publish is correct, but I will contribute the data I know about.

Quoted 3 lines

Thanks,

jon

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#16 lore raw
Quoted 6 lines · Adrian Bunk
Quoted 5 lines · Jonathan Corbet

[Operating in delayed response mode, sorry...]

Quoted 1 line · Adrian Bunk

In which format do you want data for commits already in 2.6.26?

Quoted 8 lines
Quoted 6 lines

I hadn't thought about trying to backfill data for existing commits; that sounds like a fair amount of work for coverage which could still be somewhat spotty. If you have the data at hand, though, then a simple file of the form

commit-id full-name <email>

How to indicate whether it's a Reported-by or a Bisected-by?

Literal Bisected-by might not make for a very good tag. Bisection is a way to pin down a bug; just one of them. So say person A reports and person B says "Hey, I know what that is. Try reverting foo." B then doesn't fit the tag while the only difference is that B was clever enough to not need a bisection. So if this is about giving credit; Tracked-down-by maybe? Something else?

Rene.

#17 lore raw
In reply to Adrian Bunk
Quoted 1 line · Adrian Bunk

How to indicate whether it's a Reported-by or a Bisected-by?

Bisected-by was meant as a silly joke; I think that would be taking things a bit too far.

Thanks,

jon

#18 lore raw
In reply to Jonathan Corbet
Quoted 10 lines · Jonathan Corbet

So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.

Ingo Molnar recently already used the latter:

http://marc.info/?l=linux-kernel&m=121032355203154&w=4

and a reply of mine consisted of now also needing a Reported-By:

http://marc.info/?l=linux-kernel&m=121037403515110&w=4

I must say the Bisected-by thing did feel a bit childish ("ah, I'm not in it to get the bug fixed but to get my name mentioned!") but hey, I find things to be offended over easily. From the example it does seem you probably want both if you have one; if reporter, nailer-downer and fixer are three seperate people.

Rene.

#19 lore raw
Quoted 5 lines · Rene Herman

I must say the Bisected-by thing did feel a bit childish ("ah, I'm not in it to get the bug fixed but to get my name mentioned!") but hey, I find things to be offended over easily. From the example it does seem you probably want both if you have one; if reporter, nailer-downer and fixer are three seperate people.

Which tag do I use for the postman who delivered the device which I needed for debugging?

Signature · 3 lines
Stefan Richter
-=====-==--- -=-= -===-
http://arcgraph.de/sr/
#20 lore raw
In reply to Jonathan Corbet

Hi!

Quoted 8 lines

So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things

I believe we have enough tags already. Plus, if the reporter really works with the developer till the end, there's already accepter
Tested-by: flag, right?

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#21 lore raw

Hi,

Quoted 6 lines · Pavel Machek

Hi!

Quoted 4 lines

So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:

Quoted 8 lines
Quoted 4 lines

In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things

I believe we have enough tags already. Plus, if the reporter really works with the developer till the end, there's already accepter
Tested-by: flag, right?

I would more argue to remove Tested-by completely because it does not tell much. Code is not bugfree just because someone compiled and ran it. And if it breaks on some other systems later on, what does it help if you know someone tested it? It still breaks. And you can not even blame the tester because of his luck of a working configuration.

While a Reported-by in this case credits a person reporting a bug. Just that. And perhaps that the report was good enough to make a fix.

Hannes

#22 lore raw
Quoted 5 lines · Johannes Weiner

I would more argue to remove Tested-by completely because it does not tell much. Code is not bugfree just because someone compiled and ran it. And if it breaks on some other systems later on, what does it help if you know someone tested it? It still breaks. And you can not even blame the tester because of his luck of a working configuration.

Tested-by is really helpful for identifying potential testers when you change some part of the kernel. I use git history to identify people who are able/willing to test NUMA changes to the slab allocators that require a big ass NUMA box to trigger a bug, for example. Also, in some cases, testing a bug fix is not trivial and might require a lot of work that we should give credit for.