Tracking and crediting bug reporters
Several members of the Linux Foundation's Technical Advisory Board recently got together with Andrew Morton to talk about kernel quality issues. One of the things which came out of that meeting was a desire to improve incentives for people who report bugs. Clearly, actually fixing those bugs would qualify; nobody has lost sight of that. But it was suggested that the creation and publication of statistics on bug reporting would also help.
One way to do this might be for Andrew (being the only one who actually reads every message posted on the list) to keep a spreadsheet along with everything else he does. That idea did not go over very well.
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:
In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.
If these tags go into the commit messages in any sort of consistent way, it should be possible generate the usual sort of statistics from them. I'll then happily publicize them next to the traditional lists of people who are adding new bugs. The result will certainly be fame, fortune, and job offers for the people at the top of the list. Or something like that.
If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
Thoughts?
jon
[Jonathan Corbet - Mon, May 12, 2008 at 11:27:52AM -0600]
| Several members of the Linux Foundation's Technical Advisory Board recently | got together with Andrew Morton to talk about kernel quality issues. One | of the things which came out of that meeting was a desire to improve | incentives for people who report bugs. Clearly, actually fixing those bugs | would qualify; nobody has lost sight of that. But it was suggested that | the creation and publication of statistics on bug reporting would also | help. | | One way to do this might be for Andrew (being the only one who actually | reads every message posted on the list) to keep a spreadsheet along with | everything else he does. That idea did not go over very well. | | So here's what we would like to try instead. Whenever somebody sends up a | patch fixing a reported bug, the name of the person who reported the bug | would be immortalized with this tag: | | Reported-by: A. Bug Reporter <[email protected]> | | In particular, reporters who work with the developers toward the resolution | of the bug should be thanked in this way. If we wanted to take things | further, perhaps we could add a Bisected-by: tag for really hard-core | helpers. | | If these tags go into the commit messages in any sort of consistent way, it | should be possible generate the usual sort of statistics from them. I'll | then happily publicize them next to the traditional lists of people who are | adding new bugs. The result will certainly be fame, fortune, and job | offers for the people at the top of the list. Or something like that. | | If the rest of the community is agreeable, it would be nice to make an | immediate start on this; it's not yet too late to get reasonable data for | the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27. | | Thoughts? | | jon |
If my opinion does worth somehow - I'm absolutely agree!
(btw, it seems we forget the main tag ever was - Bug-made-by: ;)
- Cyrill -
Quoted 5 lines · On Mon, 12 May 2008 11:27:52 -0600 Jonathan Corbet <[email protected]> wrote:
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:
Reported-by A. Bug Reporter <[email protected]>
I like this idea. If people help with kernel development, they deserve kudos.
Signature · 1 line
All Rights Reversed
Quoted 5 lines
If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
Thoughts?
I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws. So we need to document that it is added with permission of the reporter only.
Alan
From: Alan Cox <[email protected]>
Date: Mon, 12 May 2008 19:20:19 +0100
Quoted 10 lines
Quoted 5 lines
If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
Thoughts?
I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws. So we need to document that it is added with permission of the reporter only.
This is why I personally do not accept private bug reports or kernel questions, I always direct them to post to the appropriate public mailing list and CC: me if appropriate.
That solves the whole "privacy" issue quite neatly.
Quoted 5 lines
This is why I personally do not accept private bug reports or kernel questions, I always direct them to post to the appropriate public mailing list and CC: me if appropriate.
That solves the whole "privacy" issue quite neatly.
Not really. About half the bugs I deal with and fix are from customers of Red Hat who have differing expectations.
Alan
Quoted 8 lines · Alan Cox
Quoted 5 lines
This is why I personally do not accept private bug reports or kernel questions, I always direct them to post to the appropriate public mailing list and CC: me if appropriate.
That solves the whole "privacy" issue quite neatly.
Not really. About half the bugs I deal with and fix are from customers of Red Hat who have differing expectations.
Those would just default to not being credited. Where is the problem?
-Andi
Quoted 1 line
Those would just default to not being credited. Where is the problem?
We need to be sure that people understand that when submitting patches they must not put Discovered-by: lines in without asking. This is essentially a documentation matter which is why I directed the original reply to Jonathan.
Alan
Quoted 6 lines · Alan Cox
Quoted 1 line
Those would just default to not being credited. Where is the problem?
We need to be sure that people understand that when submitting patches they must not put Discovered-by: lines in without asking. This is essentially a documentation matter which is why I directed the original reply to Jonathan.
I tend to put the credit in the patch description text, /without/ an email address by default, since I don't feel as comfortable stamping users' email addresses into the kernel changelog without their permission, and (I'm lazy) asking permission is another email round-trip. So my format is
blah blah blah patch description blah blah blah blah blah blah patch description blah blah blah blah blah blah patch description blah blah blah
Bug found by Lucifer Smith.
Regards,
Jeff
Quoted 9 lines · Alan Cox
Quoted 5 lines
If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
Thoughts?
I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws.
IANAL. Unless it's a public information, no? (As reports are.)
Otherwise we would be in a big trouble, since we do this as far as can git history tell at least.
Quoted 5 lines
Quoted 3 lines
I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws.
IANAL. Unless it's a public information, no? (As reports are.)
If they made it public fine - although building a database of that requires care. That was my point. You need specific permission of the bug reporter. Given bug reporters may have personal reasons (including 'works at Microsoft') and business ones (from 'company policy' to 'three letter agency') for not wanting the information public.
Quoted 2 lines
Otherwise we would be in a big trouble, since we do this as far as can git history tell at least.
No: The Signed-off-by in GIT references the 1.1 contributor agreement which specifically deals with that case.
Quoted 11 lines · Alan Cox
Quoted 5 lines
Quoted 3 lines
I cannot add a tag with a third party personal information upon it without their permission nor can anyone else in a part of the world with any vaguely resembling privacy laws.
IANAL. Unless it's a public information, no? (As reports are.)
If they made it public fine - although building a database of that requires care. That was my point. You need specific permission of the bug reporter. Given bug reporters may have personal reasons (including 'works at Microsoft') and business ones (from 'company policy' to 'three letter agency') for not wanting the information public.
Well, don't the mailing list archives build such data bases already?
By posting a report to a mailing list the archives of which are publicly available, and the LKML is one of those, you make your name and email address information publicly available and recorded forever anyway, in the context of the report.
Thanks,
Rafael
Quoted 29 lines · Jonathan Corbet
Several members of the Linux Foundation's Technical Advisory Board recently got together with Andrew Morton to talk about kernel quality issues. One of the things which came out of that meeting was a desire to improve incentives for people who report bugs. Clearly, actually fixing those bugs would qualify; nobody has lost sight of that. But it was suggested that the creation and publication of statistics on bug reporting would also help.
One way to do this might be for Andrew (being the only one who actually reads every message posted on the list) to keep a spreadsheet along with everything else he does. That idea did not go over very well.
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:
Reported-by A. Bug Reporter <[email protected]>In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.
If these tags go into the commit messages in any sort of consistent way, it should be possible generate the usual sort of statistics from them. I'll then happily publicize them next to the traditional lists of people who are adding new bugs. The result will certainly be fame, fortune, and job offers for the people at the top of the list. Or something like that.
Quoted 5 lines
If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
Thoughts?
I like the idea.
Thanks,
Rafael
Quoted 11 lines · Jonathan Corbet
...
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:Reported-by A. Bug Reporter <[email protected]>In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.
I assume it will be valid for one person to have two or all three tags in one commit (not uncommon for some of the stuff I'm doing)?
Quoted 10 lines
If these tags go into the commit messages in any sort of consistent way, it should be possible generate the usual sort of statistics from them. I'll then happily publicize them next to the traditional lists of people who are adding new bugs. The result will certainly be fame, fortune, and job offers for the people at the top of the list. Or something like that.
If the rest of the community is agreeable, it would be nice to make an immediate start on this; it's not yet too late to get reasonable data for the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
...
In which format do you want data for commits already in 2.6.26?
Quoted 1 line
jon
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed[Operating in delayed response mode, sorry...]
Quoted 1 line · Adrian Bunk
In which format do you want data for commits already in 2.6.26?
I hadn't thought about trying to backfill data for existing commits; that sounds like a fair amount of work for coverage which could still be somewhat spotty. If you have the data at hand, though, then a simple file of the form
commit-id full-name <email>
could be dealt with. But is it really worth the effort?
Thanks,
jon
Quoted 4 lines · Jonathan Corbet
[Operating in delayed response mode, sorry...]
Adrian Bunk <[email protected]> wrote:
Quoted 8 lines
Quoted 1 line
In which format do you want data for commits already in 2.6.26?
I hadn't thought about trying to backfill data for existing commits; that sounds like a fair amount of work for coverage which could still be somewhat spotty. If you have the data at hand, though, then a simple file of the form
commit-id full-name <email>
How to indicate whether it's a Reported-by or a Bisected-by?
Quoted 1 line
could be dealt with. But is it really worth the effort?
Not sure whether you don't consider it important that the data you publish is correct, but I will contribute the data I know about.
Quoted 3 lines
Thanks,
jon
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 6 lines · Adrian Bunk
Quoted 5 lines · Jonathan Corbet
[Operating in delayed response mode, sorry...]
Quoted 1 line · Adrian Bunk
In which format do you want data for commits already in 2.6.26?
Quoted 8 lines
Quoted 6 lines
I hadn't thought about trying to backfill data for existing commits; that sounds like a fair amount of work for coverage which could still be somewhat spotty. If you have the data at hand, though, then a simple file of the form
commit-id full-name <email>
How to indicate whether it's a Reported-by or a Bisected-by?
Literal Bisected-by might not make for a very good tag. Bisection is a way to pin down a bug; just one of them. So say person A reports and person B says "Hey, I know what that is. Try reverting foo." B then doesn't fit the tag while the only difference is that B was clever enough to not need a bisection. So if this is about giving credit; Tracked-down-by maybe? Something else?
Rene.
Quoted 1 line · Adrian Bunk
How to indicate whether it's a Reported-by or a Bisected-by?
Bisected-by was meant as a silly joke; I think that would be taking things a bit too far.
Thanks,
jon
Quoted 10 lines · Jonathan Corbet
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:
Reported-by A. Bug Reporter <[email protected]>In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things further, perhaps we could add a Bisected-by: tag for really hard-core helpers.
Ingo Molnar recently already used the latter:
http://marc.info/?l=linux-kernel&m=121032355203154&w=4
and a reply of mine consisted of now also needing a Reported-By:
http://marc.info/?l=linux-kernel&m=121037403515110&w=4
I must say the Bisected-by thing did feel a bit childish ("ah, I'm not in it to get the bug fixed but to get my name mentioned!") but hey, I find things to be offended over easily. From the example it does seem you probably want both if you have one; if reporter, nailer-downer and fixer are three seperate people.
Rene.
Quoted 5 lines · Rene Herman
I must say the Bisected-by thing did feel a bit childish ("ah, I'm not in it to get the bug fixed but to get my name mentioned!") but hey, I find things to be offended over easily. From the example it does seem you probably want both if you have one; if reporter, nailer-downer and fixer are three seperate people.
Which tag do I use for the postman who delivered the device which I needed for debugging?
Signature · 3 lines
Stefan Richter -=====-==--- -=-= -===- http://arcgraph.de/sr/
Hi!
Quoted 8 lines
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:
Reported-by A. Bug Reporter <[email protected]>In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things
I believe we have enough tags already. Plus, if the reporter really works with the developer till the end, there's already accepter
Tested-by: flag, right?
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hi,
Quoted 6 lines · Pavel Machek
Hi!
Quoted 4 lines
So here's what we would like to try instead. Whenever somebody sends up a patch fixing a reported bug, the name of the person who reported the bug would be immortalized with this tag:
Quoted 8 lines
Quoted 4 lines
Reported-by A. Bug Reporter <[email protected]>In particular, reporters who work with the developers toward the resolution of the bug should be thanked in this way. If we wanted to take things
I believe we have enough tags already. Plus, if the reporter really works with the developer till the end, there's already accepter
Tested-by: flag, right?
I would more argue to remove Tested-by completely because it does not tell much. Code is not bugfree just because someone compiled and ran it. And if it breaks on some other systems later on, what does it help if you know someone tested it? It still breaks. And you can not even blame the tester because of his luck of a working configuration.
While a Reported-by in this case credits a person reporting a bug. Just that. And perhaps that the report was good enough to make a fix.
Hannes
Quoted 5 lines · Johannes Weiner
I would more argue to remove Tested-by completely because it does not tell much. Code is not bugfree just because someone compiled and ran it. And if it breaks on some other systems later on, what does it help if you know someone tested it? It still breaks. And you can not even blame the tester because of his luck of a working configuration.
Tested-by is really helpful for identifying potential testers when you change some part of the kernel. I use git history to identify people who are able/willing to test NUMA changes to the slab allocators that require a big ass NUMA box to trigger a bug, for example. Also, in some cases, testing a bug fix is not trivial and might require a lot of work that we should give credit for.