bug database braindump from the kernel summit

Folks, since bug tracking is the next thing we are attacking here at BitMover, I have a great deal of interest in the bug tracking discussion which happened last night at the summit. We already have a prototyped bug tracking system which we are integrating into BitKeeper, but as usual, it isn't good enough for the kernel team who have an interesting set of requirements.

We want to make sure that the BK bug tracking system _could_ be used for the kernel effort. Whether it is or not will be decided after years of licensing flamewars, etc., etc. We're not going to go there so please don't try.

Where I want to go is a discussion of requirements, then we converge on a strawman proposal, and then all of the people who care enough about this can go implement an answer and Linus and/or Alan and/or whoever can choose one.

Let's stay focussed on arriving at a good set of agreed on requirements.

I've done a brain dump below. I'll track this mail thread and update this doc as the discussion unfolds. I'll post updates as it is needed. If this discussion takes on a life of its own, I may try and grab all the mail and archive it so that people can browse; we'll see, it may be that everyone is so busy that this will be the first and last message on the topic.

--lm

Brain dump on the bug tracking problem from the Kernel Summit discussions

		[SCCS/s.BUGS vers 1.2 2001/04/01 10:46:55]

Outline
Problems
Problem details
Past experiences
Requirements

Problems
- getting quality bug reports
- not losing any bugs
- sorting low signal vs high signal into a smaller high signal pile
- simplified, preferably NNTP, access to the bug database (Linus would use this; he's unlikely to use anything else)

Problem details
Bug report quality
There was lots of discussion on this. The main agreement was that we wanted the bug reporting system to dig out as much info as possible and prefill that. There was a lot of discussion about possible tools that would dig out the /proc/pci info; there was discussion about Andre's tools which can tell you if you can write your disk; someone else had something similar.

But the main thing was to extract all the info we could
automatically. One thing was the machine config (hardware and
at least kernel version). The other thing was extract any oops messages and get a stack traceback.

The other main thing was to define some sort of structure to the bug report and try and get the use to categorize if they could. In an ideal world, we would use the maintainers file and the stack traceback to cc the bug to the maintainer. I think we want
to explore this a bit. I'm not sure that the maintainer file is
the way to go, what if we divided it up into much broader chunks like "fs", "vm", "network drivers", and had a mail forwarder
for each area. That could fan out to the maintainers.

Not losing bugs
While there was much discussion about how to get rid of bad, incorrect, and/or duplicate bug reports, several people - Alan in particular - made the point that having a complete collection of all bug reports was important. You can do data mining across all/part of them and look for patterns. The point was that there is some useful signal amongst all the noise so we do not want to lose that signal.

Signal/noise
We had a lot of discussion about how to deal with signal/noise. The bugzilla proponents thought we could do this with some additional hacking to bugzilla. I, given the BitKeeper background, thought that we could do this by having two databases, one with all the crud in it and another with just the screened bugs in it. No matter how it is done, there needs to be some way to both keep a full list, which will likely be used only for data mining, and another, much smaller list of screened bugs. Jens wants there to be a queue of new bugs and a mechanism where people can come in the morning, pull a pile of bugs off of the queue, sort them, sending some to the real database. This idea has a lot of merit, it needs some pondering as DaveM would say, to get to the point that we have a workable mechanism which works in a distributed fashion.

The other key point seemed to be that if nobody picked up a bug and nobody said that this bug should be picked up, then the bug expires out of the pending queue. It gets stashed in the bug archive for mining purposes and it can be resurrected if it later becomes a real bug, but the key point seems to be that it _automatically_ disappears out of the pending queue. I personally am very supportive of this model. We need some way to just let junk stay junk. If junk has to be pruned out of the system by humans, the system sucks. The system, not humans, needs to autoprune.

Simplified access: browsing and updating
Linus made the point that mailing lists suck. He isn't on any and refuses to join any. He reads lists with a news reader. I think people should sit up and listen to that - it's a key point. If your mailing list isn't gatewayed to a newsgroup, he isn't reading it and a lot of other people aren't either.

There was a fair bit of discussion about how to get the bug database connected to news. There doesn't seem to be any reason that the bug system couldn't be a news server/gateway. You should be able to browse
bitbucket.kernel.bugs - all the unscreened crud screened.kernel.bugs - all bugs which have been screened fs.kernel.bugs - screened bugs in the "fs" category ext2.kernel.bugs - screened bugs in the "ext2" category eepro.kernel.bugs - screened bugs in the "eepro" category etc.

Furthermore, the bugs should be structured once they are screened, i.e., they have a set of fields like (this is a strawman):

Synopsis - one line man-page like summary of the bug Severity - how critical is this bug?
Priority - how soon does it need to be fixed?
Category - subsystem in which the bug occurs Description - details on the bug, oops, stack trace, etc. Hardware - hardware info
Software - kernel version, glibc version, etc. Suggested fix - any suggestion on how to fix it Interest list - set of email addresses and/or newsgroups for updates

It ought to work that if someone posts a followup to the bug then if the followup changes any of the fields that gets propagated to the underlying bug database. If this is done properly the news reader will be the only interface that most people use.

Past experiences
This is a catch all for sound bytes that we don't want to forget...

- Sorting bugs by hand is a pain in the ass (Ted burned out on it and Alan refuses to say that it is the joy of his life to do it)
- bug systems tend to "get in the way". Unless they are really trivial to submit, search, update then people get tired of using them and go back to the old way
- one key observation: let bugs "expire" much like news expires. If nobody has been whining enough that it gets into the high signal bug db then it probably isn't real. We really want a way where no activity means let it expire.
- Alan pointed out that having all of the bugs someplace is useful, you can search through the 200 similar bugs and notice that SMP is the common feature.

Requirements
This section is mostly empty, it's here as a catch all for people's bullet items.

- it would be very nice to be able to cross reference bugs to bug fixes in the source management system, as well as the other way around.

I'm really sick of being buried in useless information. The signal gets lost in the noise. It is easy to discard automatically generated bug reports, and way too annoying to wade through the crud.

When network connections hang, the console-tools package version isn't likely to be of any use. When ramfs leaks memory, nobody needs the content of /proc/pci.

Sometimes the bit of crud are HUGE. Imagine the hardware info for a 64-way SGI or Sun box with plenty of devices attached.

Disk space is 'free'. The information should be stored in a database where you can retrieve the information you need at will, while the back-end can statistically analyze the whole of the information looking for anomalies you would never have expected (like that network hang actually being caused by console-tools :) ).

Disk space isn't the issue. Just a few days ago I tried to help somebody who posted one of the bloated fill-in-the-form bug reports. I gave him a useless answer, because I didn't see amid all the junk that he had no problems with a 2.2.xx kernel. The good information had been buried in fluff.

On Sun, 1 Apr 2001, Larry McVoy wrote:

when generating the auto bug reports make sure that the system tells the user exactly what data is being sent.

sending a large chunk of unknown data off the machine is a big concern to many people.

David Lang

oops, I was going to quote a section from Larry's mail then decided not to and forgot to delete the header.

David Lang

Yeah. This is a good point, although I can't think of info about a system's hardware and software configuration that would be particularly sensitive, other than files that contain network topology or encrypted passwords. I'm sure others can come up with such a list. One candidate might be the smbfs configuration file, since network passwords can live there, right? tcpdump output might also be sensitive, but that type of info would need to get requested by network driver developers after the initial bug report anyhow.

Miles

Miles, if the system is just sending the config info it may not be a problem, but take the microsoft example, how do you know the bug report is just sending info that is relevent to the system and not trying to discover everything that you have installed in your system (in our case we can't be looking for pirated software, but assuming that the bug report is being sent as root how do you know that it's not sending out your password file if it just send 'stuff' out)

David Lang

[...]

Basically look through REPORTING-BUGS for a laundry list of helpful info.

(slight tangent)
One thing people have talked about in the past is updating MAINTAINERS to actually reflect real life... some of the active maintainers don't have maintainer entries. That may be intentional, maybe not...

For drivers at least, bugs -should- be mailed to the driver maintainer where possible, which is not always the same as the subsystem maintainer.

[...]

Jes?

Back when I was hacking GNOME, various janitors (to use the now-popular term) would go through and clean the bug database, sorting through a lot of the noise.

Newbies are finding the Kernel Janitors project as an excellent way to begin to contribute to the kernel. There are definitely interested, smart people willing to help, that at present don't know a lot about the kernel. Such volunteers, like the GNOMEs mentioned above, could be a vast benefit to this particular step of the bug tracking process. You
http://sourceforge.net/projects/kernel-janitor/ </plug>

Agreed, with a caveat: some bugs should stay around but not expire. There is a class of bugs that shouldn't go away from the "less noise" bug database when there is no activity. ie. hard problems we want to remember, but solve in a later version.

1) An e-mail interface.

For sorting through massive amounts of bugs, NNTP is far more useful. Maybe as a query interface as well, NNTP is useful. But for getting into the nitty-gritty details of handling a bug, e-mail is generally the primary medium of communication between developer and user.

debian-bugs assigns each bug a unique e-mail address, and all e-mail that arrives in that mailbox is appended to the bug's comments.

2) Support for binary attachments from users

This depends on the bug.

If someone finds a generic kernel problem while logging in on a serial terminal connected through a specialix-SX card, there is no reason to CC me as the maintainer of that driver. If however, the bug is in the driver, then I should be CC-ed.

Asking the submitter is I think the only way to do this:
You have a XYZ scsi Controller.
Do you think this bug relates to this driver? yes/no
Should the maintainer of the xyz controller be CCed? yes/no

You have a Specialix SX card.
Do you think this bug relates to this driver? yes/no
Should the maintainer of the specialix SX card be CCed? yes/no

Blindly CCing people from the Maintainers list is probably not a good idea. Better would be to allow the maintainers to specify HOW they want to be CC-ed. I for example, would want for every driver that I maintain, to have a bug submitted in my jitterbug database (instead of mailed to my private email address).

All this sorting and tagging should be done in ONE database. So, random users can add "count me in on this one", meaning that they too see this problem. And that someone trying to fix this can CC them too.

Also, say Jens can tag a problem as "a fluke", while Alan should still be able to tag it as: "Hmm. Odd problem. Heard of it before, Need more reports to figure out the common cause".

Queries to the database can then be "All problems tagged as real by at least one kernel hotshot".

Jitterbug already has this. However, every problem in Jitterbug is only in one queue/classification. My suggestion for the big thingy is that everybody could have a different view.

OK. But those "interested" in the bug (the Email of the submitter) should be notified: "Your reported bug is going to expire soon. Could you indicate that this is still bothering you"?.

"Enhancement requests" can stay around for a long, long time if nobody takes the time to implement the request.

Speaking of "Enhancement requests", there once was a professor in the field of "operating systems" who wanted to give out real Linux-ehancement projects as lab-projects for his course. Some people might tag enhancement requests with "80-hour project", and then allow students to do them for credit!

This sounds good, except that when you have different views towards the database, problems arise. We could just dump the database into a news-server every hour to make Linus happy, but that would probably defeat some of the advantages of the "news" interface (mark problems as "read").

This needs to be "forced". A submitter should get an EMail when a bug is about to be expired, and should be able to claim an extension. The reply should also ask for the latest kernel that the submitter tried, and was verfied to have the bug.

So the bug item in the DB will get tagged for example as: reported to exist on 2.4.1, 2.4.2-ac12 and 2.4.3 by REW.

- Dynamic keyword section.

Dynamic: You CAN enter keywords that are not already in the database. Those will be available to others from then on.

Someone may want to tag "disc-problem" as equivalent to "disk-problem".

The "limited list of possibilites" means that searches for...

... similar problems become easier/possible.

- Automatic inclusion of ".config" files. Nowadays, the .config is so large that submitting it with a bugreport is not efficient. However, if you would be submitting a bugreport into a database, the info could be very useful in reliably scanning for common configuration options in a set of reports.

- "me too" reports should be able to append a new ".config" to an existing bugreport.

Roger.

** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots.