oops.kernel.org prospect
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Anton Arapov
Quoted 2 lines
Why not just do that through email? You'll reach a much wider group of people than the tiny 80 developers at the conference.
Ouch! Someone to take it as replacement of email - the least I wanted. It will go email-way in either case.
Quoted 8 lines
Quoted 3 lines
These tiny 80 may give the most valuable feedback on the topic. And often it is the most difficult to get attention of them, especially via email. In case it fits the conference, it could dilute the heavy topics.
Usyually the best thing to do is to start the discussion on the mailing list (and we can do that on ksummit-2013-discuss, but this is always why it's sometimes useful to cc lkml on topic proposals, so we can jump start the discussion), and see if it's controversial or not.
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
// CC'd: lkml (please don't complain on styles yet, focus on functionality)
Quoted 5 lines
I suspect the biggest issue is one of "who will bell the cat". As in, if no one creates a new oops.kernel.org, and if we can't get the community distributions to be willing to set up their systems to automatically submit oops reports to the server, it's going to be somewhat pointless to have the discussion at the kernel summit.
It is created and distributions are willing to submit oopses automatically. Debian, Fedora, Ubuntu are already sending reports.
The above is exactly what I would be able to share on conference with more details.
Quoted 11 lines
Note that main value of oops.kernel.org is the fact that we find out which bugs are most likely hitting "real users", as opposed to the very valuable testing done by developers on the linux-next and -rc kernels. It's main drawback is that for privacy reasons, we don't get much more information than the stack trace. So if it's just linux-next testers and developers using oops.kernel.org, who are the people who are more likely able to submit a detailed bug report, the oops.kernel.org server probably won't add much value. In fact, if the testers depend on the sending of an anonymous stack trace, and don't send us more detailed reproduction information, the existence of oops.kernel.org could actually make things worse.
And the above here is one of the feedbacks I am willing to see. As well as ideas how to overcome it.
Quoted 3 lines
This means, realistically, it needs someone from Fedora or Open SuSE or Debian or Ubuntu being willing to sponsor the uploading of information to oops.kernel.org by default.
...up and running, now...
hth,
--
Anton
Quoted 17 lines · Anton Arapov
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Anton Arapov
Quoted 2 lines
Why not just do that through email? You'll reach a much wider group of people than the tiny 80 developers at the conference.
Ouch! Someone to take it as replacement of email - the least I wanted. It will go email-way in either case.
Quoted 8 lines
Quoted 3 lines
These tiny 80 may give the most valuable feedback on the topic. And often it is the most difficult to get attention of them, especially via email. In case it fits the conference, it could dilute the heavy topics.
Usyually the best thing to do is to start the discussion on the mailing list (and we can do that on ksummit-2013-discuss, but this is always why it's sometimes useful to cc lkml on topic proposals, so we can jump start the discussion), and see if it's controversial or not.
Quoted 4 lines
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
// CC'd: lkml (please don't complain on styles yet, focus on functionality)
I stumbled across this a week or so ago, and had some thoughts back then, but didn't mail them anywhere because I wasn't sure who ran it, and couldn't tell how far along it was.
Quick brain dump
* Visiting it with chromium gets an annoying warning about the https server identifying as a different server. (does it even need https?)
* There's a lot of tainted kernel traces in there. 99% of kernel developers will never care about these in my experience. You can adjust this on a per-query basis it seems, but better would be to turn them off globally, and have them available just for people who want to search for 'all' (tainted or untainted) oopses.
- That the tainted oopses are counted as 'regular' oopses is skewing the 'top bugs' on the front page.
- As well as proprietary, take care of 'out of tree' tainted modules in the same way.
* I clicked through some of the debian oopses, and saw these:
https://oops.kernel.org/browse-reports/oops-detail/?id=30497
https://oops.kernel.org/browse-reports/oops-detail/?id=30499
It would be useful to know if this was the same user. (It seems likely, but there's no way to know for sure). You don't need identifying info other than "These came from the same system" side-stepping any privacy concerns.
* In the Linked modules section, if there's an out-of-tree/proprietary module, we annotate those in oopses with (O), or (P). This seems to be lost in your UI. (Bonus points for making them stand out)
* The traces by default lack a lot of information, forcing clicking of the 'show raw oops' in every case. Missing useful info (at least): EIP/RIP, other registers.
* 'Show raw oops' doesn't. (At least on chromium)
* This bug last seen: 2013-08-17
Also useful here would be something like:
Seen on: 3.2-rc2, 3.10-rc10 (You can probably just list earliest/latest rather than
every single kernel it's been seen on, unless you want a 'show all' button)
* Instead of summaries like "general protection fault: 4000 [#1] SMP" Decode the EIP/RIP, and call it "general protection fault in i915_gem_do_execbuffer". Not only does it make reading summaries easier, it should allow you to detect dupes better. (Sidenote, abrt needs this too, when it files bugzillas)
* Looking over the summaries at https://oops.kernel.org/browse-reports/?distro=Fedora&search=submit
The first thing that comes to mind is "There's a lot of soft lockup bugs here" Some means of grouping similar looking bugs would be useful.
(In bugzilla, clicking 'sort by summary' kinda gives this, but it still sucks).
* When Arjan ran kerneloops, he would periodically mail out a "top 10 oopses" report on the latest tree. That seems like something that would be worth doing again, but only after filtering out the tainted stuff as mentioned above.
* Some kind of "find similar bugs in other bug trackers" feature would be really awesome.
* There's a bunch of bugs in there that have been tainted 'W'. These are almost never useful, because we're already deep in "bad shit happened" land at that point.
It'll also mean you could get flooded with oopses from a single crash if something keeps on spewing traces. Just give up after filing the first oops.
* Take for example: https://oops.kernel.org/browse-reports/oops-detail/?id=30410
This is a 2.6.27.5 kernel bug, that was filed *last week*.
I'd bet dollars to donuts no-one is going to give a crap about that bug. I'm not sure if it's better here to never file 'ancient' bugs, or to periodically archive/delete ones that have been in the db more than a few years.
* Looking at https://oops.kernel.org/browse-reports/?function=ironlake_crtc_disable&search=submit
It seems the hashing algorithm for detecting dupes could use some work.
Many of these traces are probably exactly the same problem.
Are you hashing symbols in the trace beginning with '? ' ? If so, you probably shouldn't be.
Dave
Quoted 31 lines · Dave Jones
Quoted 17 lines · Anton Arapov
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Anton Arapov
Quoted 2 lines
Why not just do that through email? You'll reach a much wider group of people than the tiny 80 developers at the conference.
Ouch! Someone to take it as replacement of email - the least I wanted. It will go email-way in either case.
Quoted 8 lines
Quoted 3 lines
These tiny 80 may give the most valuable feedback on the topic. And often it is the most difficult to get attention of them, especially via email. In case it fits the conference, it could dilute the heavy topics.
Usyually the best thing to do is to start the discussion on the mailing list (and we can do that on ksummit-2013-discuss, but this is always why it's sometimes useful to cc lkml on topic proposals, so we can jump start the discussion), and see if it's controversial or not.
Quoted 4 lines
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
// CC'd: lkml (please don't complain on styles yet, focus on functionality)
I stumbled across this a week or so ago, and had some thoughts back then, but didn't mail them anywhere because I wasn't sure who ran it, and couldn't tell how far along it was.
Quick brain dump
* Visiting it with chromium gets an annoying warning about the https server
...
[snip]
Quoted 2 lines
...
Dave
Thanks, Dave! Will be fixed and improved.
Anton.
Quoted 7 lines · Anton Arapov
Quoted 2 lines
* Visiting it with chromium gets an annoying warning about the https server
...[snip]
Quoted 2 lines
...
DaveThanks, Dave! Will be fixed and improved.
Yeah, collecting oopses is a good idea, so +1.
However, we probably want to think about what exactly we're going to do with that information. For example, if I want to address an issue, I probably want to know how I can reproduce the oops - maybe something like allowing the reporter to add free text note to the oops.
And yes, as tytso already said, we are very often going to need more info about a system causing the oops (dmesg, lspci, dmidecode, etc, etc). I'm not sure how we're going to collect that without sacrificing some privacy. Or maybe, we could be able to ask people to open a bug on bugzilla.kernel.org where further debugging can take place...
Which reminds me: maybe connecting bug reports on bugzilla.kernel.org with your stats could also be a way to connect bug reports with reporters...
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 7 lines · Borislav Petkov
Quoted 6 lines · Anton Arapov
Quoted 2 lines
* Visiting it with chromium gets an annoying warning about the https server
...[snip]
Quoted 2 lines
...
Dave
Quoted 8 lines
Quoted 1 line
Thanks, Dave! Will be fixed and improved.
Yeah, collecting oopses is a good idea, so +1.
However, we probably want to think about what exactly we're going to do with that information. For example, if I want to address an issue, I probably want to know how I can reproduce the oops - maybe something like allowing the reporter to add free text note to the oops.
abrt used to have a free-form entry like this.
What happened is users have no idea what to type in there, so you end up with bugs containing things like "don't know" or worse, some crazy moon language you can't even read.
Quoted 5 lines
And yes, as tytso already said, we are very often going to need more info about a system causing the oops (dmesg, lspci, dmidecode, etc, etc). I'm not sure how we're going to collect that without sacrificing some privacy. Or maybe, we could be able to ask people to open a bug on bugzilla.kernel.org where further debugging can take place...
Two things worth noting here, are 1) the original kerneloops also didn't collect anything like this, and was still very useful, and 2) for the more common issues (which let's face it, are going to be the only things people really look at) chances are pretty high that there's going to be someone also reporting it on lkml, or in a distro bug tracker.
What might be useful however, is collecting things like dmi/lspci/lsusb etc and _asking_ the user if they're ok with including them at time of filing. We might scare off some of the more paranoid OMGMYSECRETDATAS users, but chances are high most people won't care. This requires the client to have a UI though, which aiui, it currently doesn't. Anton?
We might also ask if they want to provide an email address for feedback, but that leads to a bunch of questions about how we expose that to developers without exposing it to spambots.
Dave
Quoted 12 lines · Dave Jones
Quoted 11 lines · Borislav Petkov
Quoted 7 lines · Anton Arapov
Quoted 2 lines
* Visiting it with chromium gets an annoying warning about the https server
...[snip]
Quoted 2 lines
...
DaveThanks, Dave! Will be fixed and improved.
Yeah, collecting oopses is a good idea, so +1.
Quoted 8 lines
Quoted 4 lines
However, we probably want to think about what exactly we're going to do with that information. For example, if I want to address an issue, I probably want to know how I can reproduce the oops - maybe something like allowing the reporter to add free text note to the oops.
abrt used to have a free-form entry like this.
What happened is users have no idea what to type in there, so you end up with bugs containing things like "don't know" or worse, some crazy moon language you can't even read.
Agree.
Quoted 11 lines
Quoted 5 lines
And yes, as tytso already said, we are very often going to need more info about a system causing the oops (dmesg, lspci, dmidecode, etc, etc). I'm not sure how we're going to collect that without sacrificing some privacy. Or maybe, we could be able to ask people to open a bug on bugzilla.kernel.org where further debugging can take place...
Two things worth noting here, are 1) the original kerneloops also didn't collect anything like this, and was still very useful, and 2) for the more common issues (which let's face it, are going to be the only things people really look at) chances are pretty high that there's going to be someone also reporting it on lkml, or in a distro bug tracker.
Quoted 5 lines
What might be useful however, is collecting things like dmi/lspci/lsusb etc and _asking_ the user if they're ok with including them at time of filing. We might scare off some of the more paranoid OMGMYSECRETDATAS users, but chances are high most people won't care. This requires the client to have a UI though, which aiui, it currently doesn't. Anton?
The above is possible with abrt/libreport-kerneloops it does have UI and a possibility to include the dmi/lspci/lsusb into the message to oops.kernel.org.
Some distros still using the old reporting tool written by Arjan that doesn't have UI.
I am going to research what and how distros are using nowadays, get in touch with people/distro_maintainers in order to align the process as well as gather their views and concerns on sharing anything other than just a stacktrace and doing unconditionally(w/o user intervention). oops.kernel.org can sanitize the 'private' data is it already does for oopses.
Will be keeping lkml posted on my progress.
Quoted 3 lines
We might also ask if they want to provide an email address for feedback, but that leads to a bunch of questions about how we expose that to developers without exposing it to spambots.
I'd not want to ask user about anything. In Fedora, Abrt end up this way -- abrt asks user to review the report and whether one is willing to send it to Bugzilla and oops.kernel.org. User also can check a "don't ask me in the future for this kind of issues - just send reports" checkbox. This is what I was able to get from Abrt folks so far.
Anton
Quoted 4 lines · Dave Jones
abrt used to have a free-form entry like this. What happened is users have no idea what to type in there, so you end up with bugs containing things like "don't know" or worse, some crazy moon language you can't even read.
Prepend the entry with an informative question maybe:
"Enter bug reproduction information here:"
Quoted 6 lines
Two things worth noting here, are 1) the original kerneloops also didn't collect anything like this, and was still very useful, and
2) for the more common issues (which let's face it, are going to be the only things people really look at) chances are pretty high that there's going to be someone also reporting it on lkml, or in a distro bug tracker.
Ok.
Quoted 6 lines
What might be useful however, is collecting things like dmi/lspci/lsusb etc and _asking_ the user if they're ok with including them at time of filing. We might scare off some of the more paranoid OMGMYSECRETDATAS users, but chances are high most people won't care. This requires the client to have a UI though, which aiui, it currently doesn't. Anton?
Definitely a step in the right direction.
Quoted 3 lines
We might also ask if they want to provide an email address for feedback, but that leads to a bunch of questions about how we expose that to developers without exposing it to spambots.
Right.
Quoted 4 lines · Dave Jones
* This bug last seen: 2013-08-17
Also useful here would be something like:
Seen on: 3.2-rc2, 3.10-rc10 (You can probably just list earliest/latest rather than
every single kernel it's been seen on, unless you want a 'show all' button)
Once you have the "seen on" stuff sorted out, it would also be really nice to be able to easily select bugs only seen on "versions after 3.8", just so we can filter out some of the older stuff. The kernel version in the filter is useful, but would be much more so if we had ranges, even if it was just "newer than $foo".
When you go to "Show Raw Oops", it usually doesn't show up for me in Chrome. There's a javascript error:
Quoted 1 line
The page at https://oops.kernel.org/browse-reports/oops-detail/?id=30565# displayed insecure content from http://oops.kernel.org/get-raw.php?id=30565&token=a94733d146ae15f8cec871d4f238956da80d7c5322f6253f9b49d4c5cc8fe8a1.
If I go over and load the page as plain old http, it works fine. Also, when those oopses show up, the font is variable-width. It would probably be a wee bit easier to read if it were displayed in a fixed-width font.
This is a much more minor nit, but the source code links (like clicking on bad_page from here:
https://oops.kernel.org/browse-reports/oops-detail/?id=30565#) from the traces for the distribution kernels link over to mainline kernel source.
This means that the line numbers don't _quite_ line up. It would be
really cool if it was able to dump you right over to a copy of the Debian-specific source in that bug's case. But, this is a generic problem that folks have who work across lots of distros: you don't always have the right source in front of you for any given kernel.
Anyway... cool stuff. I always forget that oops.kernel.org is there, but it's always fun to poke around when I run across it. :)
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Dave Jones
* This bug last seen: 2013-08-17
Also useful here would be something like:
Seen on: 3.2-rc2, 3.10-rc10 (You can probably just list earliest/latest rather than
every single kernel it's been seen on, unless you want a 'show all' button)
Quoted 6 lines
Once you have the "seen on" stuff sorted out, it would also be really nice to be able to easily select bugs only seen on "versions after 3.8", just so we can filter out some of the older stuff. The kernel version in the filter is useful, but would be much more so if we had ranges, even if it was just "newer than $foo".
A per subsystem filter would be nice to have too.
Guenter
Quoted 11 lines · Dave Hansen
Quoted 4 lines · Dave Jones
* This bug last seen: 2013-08-17
Also useful here would be something like:
Seen on: 3.2-rc2, 3.10-rc10 (You can probably just list earliest/latest rather than
every single kernel it's been seen on, unless you want a 'show all' button)Once you have the "seen on" stuff sorted out, it would also be really nice to be able to easily select bugs only seen on "versions after 3.8", just so we can filter out some of the older stuff. The kernel version in the filter is useful, but would be much more so if we had ranges, even if it was just "newer than $foo".
This is good idea.
Quoted 7 lines
When you go to "Show Raw Oops", it usually doesn't show up for me in Chrome. There's a javascript error:
Quoted 1 line
The page at https://oops.kernel.org/browse-reports/oops-detail/?id=30565# displayed insecure content from http://oops.kernel.org/get-raw.php?id=30565&token=a94733d146ae15f8cec871d4f238956da80d7c5322f6253f9b49d4c5cc8fe8a1.
If I go over and load the page as plain old http, it works fine. Also, when those oopses show up, the font is variable-width. It would probably be a wee bit easier to read if it were displayed in a fixed-width font.
The latest version of Firefox should have the same issue, it is prohibited now to show/get insecure content under the secure connection. I will most probably redirect any https request to http automatically to avoid this issue.
Quoted 9 lines
This is a much more minor nit, but the source code links (like clicking on
bad_pagefrom here:
https://oops.kernel.org/browse-reports/oops-detail/?id=30565#) from the traces for the distribution kernels link over to mainline kernel source.
This means that the line numbers don't _quite_ line up. It would be
really cool if it was able to dump you right over to a copy of the Debian-specific source in that bug's case. But, this is a generic problem that folks have who work across lots of distros: you don't always have the right source in front of you for any given kernel.
Will put into todo list, might be someday...
Quoted 2 lines
Anyway... cool stuff. I always forget that oops.kernel.org is there, but it's always fun to poke around when I run across it. :)
Thanks,
Anton.
Quoted 1 line
requests accumulated:
http://trello.com/b/ZvLKCkJX/oops-kernel-org-support-and-development
On Tue, Aug 20, 2013 at 08:20:04AM -0700, Dave Hansen wrote:
[...]
Quoted 9 lines
This is a much more minor nit, but the source code links (like clicking on
bad_pagefrom here:
https://oops.kernel.org/browse-reports/oops-detail/?id=30565#) from the traces for the distribution kernels link over to mainline kernel source.
This means that the line numbers don't _quite_ line up. It would be
really cool if it was able to dump you right over to a copy of the Debian-specific source in that bug's case. But, this is a generic problem that folks have who work across lots of distros: you don't always have the right source in front of you for any given kernel.
[...]
For Debian kernels this should be quite easy. The sources are browseable at:
http://sources.debian.net/src/linux/$PACKAGE_VERSION/$FILE#L$LINE
The package version is not the same as the kernel release string, but appears at the end of the same line in oops messages, e.g. for
<http://oops.kernel.org/browse-reports/oops-detail/?id=30218> the package version is 3.10.1-1.
This doesn't work for versions older than 3.2, or those with the RT patchset, as sources.debian.net can't show the patched source for these.
Ben.
Signature · 3 lines
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
- Albert CamusQuoted 18 lines · Ben Hutchings
On Tue, Aug 20, 2013 at 08:20:04AM -0700, Dave Hansen wrote:
[...]Quoted 9 lines
This is a much more minor nit, but the source code links (like clicking on
bad_pagefrom here:
https://oops.kernel.org/browse-reports/oops-detail/?id=30565#) from the traces for the distribution kernels link over to mainline kernel source.
This means that the line numbers don't _quite_ line up. It would be
really cool if it was able to dump you right over to a copy of the Debian-specific source in that bug's case. But, this is a generic problem that folks have who work across lots of distros: you don't always have the right source in front of you for any given kernel.[...]
For Debian kernels this should be quite easy. The sources are browseable at:
http://sources.debian.net/src/linux/$PACKAGE_VERSION/$FILE#L$LINE
Quoted 8 lines
The package version is not the same as the kernel release string, but appears at the end of the same line in oops messages, e.g. for
<http://oops.kernel.org/browse-reports/oops-detail/?id=30218> the package version is 3.10.1-1.This doesn't work for versions older than 3.2, or those with the RT patchset, as sources.debian.net can't show the patched source for these.
Thanks, Ben!
Anton
Quoted 90 lines · Dave Jones
Quoted 17 lines · Anton Arapov
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Anton Arapov
Quoted 2 lines
Why not just do that through email? You'll reach a much wider group of people than the tiny 80 developers at the conference.
Ouch! Someone to take it as replacement of email - the least I wanted. It will go email-way in either case.
Quoted 8 lines
Quoted 3 lines
These tiny 80 may give the most valuable feedback on the topic. And often it is the most difficult to get attention of them, especially via email. In case it fits the conference, it could dilute the heavy topics.
Usyually the best thing to do is to start the discussion on the mailing list (and we can do that on ksummit-2013-discuss, but this is always why it's sometimes useful to cc lkml on topic proposals, so we can jump start the discussion), and see if it's controversial or not.
Quoted 4 lines
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
// CC'd: lkml (please don't complain on styles yet, focus on functionality)
I stumbled across this a week or so ago, and had some thoughts back then, but didn't mail them anywhere because I wasn't sure who ran it, and couldn't tell how far along it was.
Quick brain dump
* Visiting it with chromium gets an annoying warning about the https server identifying as a different server. (does it even need https?)
* There's a lot of tainted kernel traces in there. 99% of kernel developers will never care about these in my experience. You can adjust this on a per-query basis it seems, but better would be to turn them off globally, and have them available just for people who want to search for 'all' (tainted or untainted) oopses.
- That the tainted oopses are counted as 'regular' oopses is skewing the 'top bugs' on the front page.
- As well as proprietary, take care of 'out of tree' tainted modules in the same way.
* I clicked through some of the debian oopses, and saw these:
https://oops.kernel.org/browse-reports/oops-detail/?id=30497
https://oops.kernel.org/browse-reports/oops-detail/?id=30499
It would be useful to know if this was the same user. (It seems likely, but there's no way to know for sure). You don't need identifying info other than "These came from the same system" side-stepping any privacy concerns.* In the Linked modules section, if there's an out-of-tree/proprietary module, we annotate those in oopses with (O), or (P). This seems to be lost in your UI. (Bonus points for making them stand out)
* The traces by default lack a lot of information, forcing clicking of the 'show raw oops' in every case. Missing useful info (at least): EIP/RIP, other registers.
* 'Show raw oops' doesn't. (At least on chromium)
* This bug last seen: 2013-08-17
Also useful here would be something like:
Seen on: 3.2-rc2, 3.10-rc10 (You can probably just list earliest/latest rather than
every single kernel it's been seen on, unless you want a 'show all' button)* Instead of summaries like "general protection fault: 4000 [#1] SMP" Decode the EIP/RIP, and call it "general protection fault in
i915_gem_do_execbuffer". Not only does it make reading summaries easier, it should allow you to detect dupes better. (Sidenote, abrt needs this too, when it files bugzillas)* Looking over the summaries at https://oops.kernel.org/browse-reports/?distro=Fedora&search=submit
The first thing that comes to mind is "There's a lot of soft lockup bugs here" Some means of grouping similar looking bugs would be useful.
(In bugzilla, clicking 'sort by summary' kinda gives this, but it still sucks).* When Arjan ran kerneloops, he would periodically mail out a "top 10 oopses" report on the latest tree. That seems like something that would be worth doing again, but only after filtering out the tainted stuff as mentioned above.
* Some kind of "find similar bugs in other bug trackers" feature would be really awesome.
* There's a bunch of bugs in there that have been tainted 'W'. These are almost never useful, because we're already deep in "bad shit happened" land at that point.
It'll also mean you could get flooded with oopses from a single crash if something keeps on spewing traces. Just give up after filing the first oops.* Take for example: https://oops.kernel.org/browse-reports/oops-detail/?id=30410
This is a 2.6.27.5 kernel bug, that was filed *last week*.
I'd bet dollars to donuts no-one is going to give a crap about that bug. I'm not sure if it's better here to never file 'ancient' bugs, or to periodically archive/delete ones that have been in the db more than a few years.
Quoted 4 lines
* Looking at https://oops.kernel.org/browse-reports/?function=ironlake_crtc_disable&search=submit
It seems the hashing algorithm for detecting dupes could use some work.
Many of these traces are probably exactly the same problem.
Are you hashing symbols in the trace beginning with '? ' ? If so, you probably shouldn't be.
Dave,
FYI,
I've put all the above, hopefully nothing missed, to the list that available here:
http://trello.com/b/ZvLKCkJX/oops-kernel-org-support-and-development
Will keep lkml posted on progress though.
Anton.
Quoted 23 lines · Dave Jones
Quoted 17 lines · Anton Arapov
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Anton Arapov
Quoted 2 lines
Why not just do that through email? You'll reach a much wider group of people than the tiny 80 developers at the conference.
Ouch! Someone to take it as replacement of email - the least I wanted. It will go email-way in either case.
Quoted 8 lines
Quoted 3 lines
These tiny 80 may give the most valuable feedback on the topic. And often it is the most difficult to get attention of them, especially via email. In case it fits the conference, it could dilute the heavy topics.
Usyually the best thing to do is to start the discussion on the mailing list (and we can do that on ksummit-2013-discuss, but this is always why it's sometimes useful to cc lkml on topic proposals, so we can jump start the discussion), and see if it's controversial or not.
Quoted 4 lines
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
// CC'd: lkml (please don't complain on styles yet, focus on functionality)
Quoted 8 lines
I stumbled across this a week or so ago, and had some thoughts back then, but didn't mail them anywhere because I wasn't sure who ran it, and couldn't tell how far along it was.
Quick brain dump
* Visiting it with chromium gets an annoying warning about the https server identifying as a different server. (does it even need https?)
It was an openshift+chromium issue, it should be resolved as per
https://bugzilla.redhat.com/show_bug.cgi?id=908417
Quoted 9 lines
* There's a lot of tainted kernel traces in there. 99% of kernel developers will never care about these in my experience. You can adjust this on a per-query basis it seems, but better would be to turn them off globally, and have them available just for people who want to search for 'all' (tainted or untainted) oopses.
- That the tainted oopses are counted as 'regular' oopses is skewing the 'top bugs' on the front page.
- As well as proprietary, take care of 'out of tree' tainted modules in the same way.
It is possible to filter out tainted reports now.
Quoted 6 lines
* I clicked through some of the debian oopses, and saw these:
https://oops.kernel.org/browse-reports/oops-detail/?id=30497
https://oops.kernel.org/browse-reports/oops-detail/?id=30499
It would be useful to know if this was the same user. (It seems likely, but there's no way to know for sure). You don't need identifying info other than "These came from the same system" side-stepping any privacy concerns.
Watching oopses from one source is still in to do.
But now you can see "Total count: 14 (from 7 unique sources) " per oops, for example:
http://oops.kernel.org/oops/warning-at-net-ipv4-tcp_input-c2776-tcp_fastretrans_alert0xc21-0xc60-6/
Quoted 3 lines
* In the Linked modules section, if there's an out-of-tree/proprietary module, we annotate those in oopses with (O), or (P). This seems to be lost in your UI. (Bonus points for making them stand out)
implemented.
Quoted 2 lines
* The traces by default lack a lot of information, forcing clicking of the 'show raw oops' in every case. Missing useful info (at least): EIP/RIP, other registers.
should be improved now.
Quoted 6 lines
* 'Show raw oops' doesn't. (At least on chromium)
* This bug last seen: 2013-08-17
Also useful here would be something like:
Seen on: 3.2-rc2, 3.10-rc10 (You can probably just list earliest/latest rather than
every single kernel it's been seen on, unless you want a 'show all' button)
implemented.
Quoted 4 lines
* Instead of summaries like "general protection fault: 4000 [#1] SMP" Decode the EIP/RIP, and call it "general protection fault in
i915_gem_do_execbuffer". Not only does it make reading summaries easier, it should allow you to detect dupes better. (Sidenote, abrt needs this too, when it files bugzillas)
fixed.
Quoted 4 lines
* Looking over the summaries at https://oops.kernel.org/browse-reports/?distro=Fedora&search=submit
The first thing that comes to mind is "There's a lot of soft lockup bugs here" Some means of grouping similar looking bugs would be useful.
(In bugzilla, clicking 'sort by summary' kinda gives this, but it still sucks).
improved && fixed
Quoted 3 lines
* When Arjan ran kerneloops, he would periodically mail out a "top 10 oopses" report on the latest tree. That seems like something that would be worth doing again, but only after filtering out the tainted stuff as mentioned above.
I will start to do it.
Quoted 1 line
* Some kind of "find similar bugs in other bug trackers" feature would be really awesome.
still in todo.
Quoted 11 lines
* There's a bunch of bugs in there that have been tainted 'W'. These are almost never useful, because we're already deep in "bad shit happened" land at that point.
It'll also mean you could get flooded with oopses from a single crash if something keeps on spewing traces. Just give up after filing the first oops.* Take for example: https://oops.kernel.org/browse-reports/oops-detail/?id=30410
This is a 2.6.27.5 kernel bug, that was filed *last week*.
I'd bet dollars to donuts no-one is going to give a crap about that bug. I'm not sure if it's better here to never file 'ancient' bugs, or to periodically archive/delete ones that have been in the db more than a few years.
Quoted 4 lines
* Looking at https://oops.kernel.org/browse-reports/?function=ironlake_crtc_disable&search=submit
It seems the hashing algorithm for detecting dupes could use some work.
Many of these traces are probably exactly the same problem.
Are you hashing symbols in the trace beginning with '? ' ? If so, you probably shouldn't be.
hash function improved.
Thanks for this feedback. There are still a number of improvements planned, mostly cosmetic ones. I will keep you posted.
Anton.
Anton Arapov <[email protected]> :
[...]
Quoted 2 lines
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
I have sorted the r8169 oopses by kernel revision to start with the most recent kernels. I don't get why the r8169 driver appears in the "Caused by:" field when
- the bug is about "scheduling while atomic: Xorg/3042/0×00000001"
- the kernel is PDWO with fglrx
- r8169 appears in the module list, nowhere else (not even the oops)
I tried :
(0 answer if Stack, Registers or Disassembled code is added)
("tainted=true" while "Untainted only" was asked for, huh ?)
The answers contains:
http://oops.kernel.org/browse-reports/oops-detail/?id=29778
and
http://oops.kernel.org/browse-reports/oops-detail/?id=29856
I can't even find the "r8169" word in those.
Is it the currently expected behavior ?
Signature · 1 line
Ueimor
Quoted 18 lines · Francois Romieu
Anton Arapov <[email protected]> :
[...]Quoted 2 lines
Oh well,... I didn't have a time for this right now, nor project is not exactly in the state I'm willing to show (mostly webui)
I have sorted the r8169 oopses by kernel revision to start with the most recent kernels. I don't get why the r8169 driver appears in the "Caused by:" field when
- the bug is about "scheduling while atomic: Xorg/3042/0×00000001"
- the kernel is PDWO with fglrx
- r8169 appears in the module list, nowhere else (not even the oops)(0 answer if Stack, Registers or Disassembled code is added) ("tainted=true" while "Untainted only" was asked for, huh ?)
Quoted 7 lines
The answers contains:
http://oops.kernel.org/browse-reports/oops-detail/?id=29778
and
http://oops.kernel.org/browse-reports/oops-detail/?id=29856I can't even find the "r8169" word in those.
Is it the currently expected behavior ?
Thanks for the report. I will check it.
Anton.
A message that is not in the archive (sent off-list or not yet received): 20130821223121.336c71d0@TOMWIJ-GENTOO
oops.kernel.org, reporting. [Was: [ATTEND] oops.kernel.org prospect]
Hello,
Thank you for contacting me. I will cross-post this to LKML, as there is an information that might be interested for other folks.
Quoted 4 lines · Tom Wijsman
We, the Gentoo Kernel Project Team at Gentoo Linux, might be interested to contribute oopses to oops.kernel.org; but from a first look over what can be found on the ML and the internet about that site, there seems to be no information available about submitting to oops.kernel.org.
This kind of information will be there soon.
Quoted 2 lines
So, could you please highlight the details on how submission happens? Does this (have to) happen automatically? Can we do manual submissions?
oops.kernel.org uses the old legacy kerneloops.org's api for data collecting. Both, http://submit.kerneloops.org/submitoops.php (old) and
http://oops.kernel.org/submitoops.php (new, stick to this one) works today.
Reporting doesn't happen automatically. Two tools available for sending reports: abrt/libreport (1) and former kerneloops daemon (2). Either of these can send reports automatically or manually.
(1) https://fedorahosted.org/abrt/ - iirc, is already packaged in gentoo.
(2) There are no link to kerneloops, this is what I was able to google:
http://ftp.debian.org/debian/pool/main/k/kerneloops/kerneloops_0.12+git20090217.orig.tar.gz
Abrt has a better scanner/reporter, which produce less false-positive reports.
I don't know what way Gentoo will choose, but the original kerneloops daemon requires some love to make it run under systemd and improve the oops scanner (can be ported from abrt).
The source of _scanner_ abrt/dumpoops_plugin:
https://git.fedorahosted.org/cgit/abrt.git/tree/src/plugins/abrt-dump-oops.c
The source of _reporter_ abrt/libreport/kerneloops_plugin:
https://git.fedorahosted.org/cgit/libreport.git/tree/src/plugins/reporter-kerneloops.c
libreport allows to report manually using CLI:
$ reporter-kerneloops Usage: reporter-kerneloops [-v] [-c CONFFILE]... -d DIR
Quoted 2 lines
I think that something that "calls home" might become a concern; maybe we could still provide that as opt-in, but manual submits would be neat.
Few years of experience with Abrt/Bugzilla reports, people fill the additional information in less than 2% of all reports. And, the Abrt team is working towards the least possible interaction with the user. This is mostly to address privacy concerns, as well as leaving the possibility to provide additional data.
Quoted 1 line
Thank you in advance for your explanation.
You are welcome, let me know if you'll need any help further.
Anton.
Re: oops.kernel.org, reporting. [Was: [ATTEND] oops.kernel.org prospect]
Quoted 10 lines · Anton Arapov
Hello,
Thank you for contacting me. I will cross-post this to LKML, as there is an information that might be interested for other folks.
Quoted 4 lines · Tom Wijsman
We, the Gentoo Kernel Project Team at Gentoo Linux, might be interested to contribute oopses to oops.kernel.org; but from a first look over what can be found on the ML and the internet about that site, there seems to be no information available about submitting to oops.kernel.org.
Quoted 8 lines
This kind of information will be there soon.
Quoted 2 lines
So, could you please highlight the details on how submission happens? Does this (have to) happen automatically? Can we do manual submissions?
oops.kernel.org uses the old legacy kerneloops.org's api for data collecting. Both, http://submit.kerneloops.org/submitoops.php (old) and
http://oops.kernel.org/submitoops.php (new, stick to this one) works today.
The Ubuntu distro is still configured to submit oops to
http://submit.kerneloops.org/submitoops.php using the kerneloops package developed by Arjan. However, it has been disabled since the original kerneloops.org site went down. We can re-enable oops submissions. Will the old submit URL still work, or are you migrating to the new
http://oops.kernel.org/submitoops.php URL?
Quoted 45 lines
Reporting doesn't happen automatically. Two tools available for sending reports: abrt/libreport (1) and former kerneloops daemon (2). Either of these can send reports automatically or manually.
(1) https://fedorahosted.org/abrt/ - iirc, is already packaged in gentoo.
(2) There are no link to kerneloops, this is what I was able to google:
http://ftp.debian.org/debian/pool/main/k/kerneloops/kerneloops_0.12+git20090217.orig.tar.gz
Abrt has a better scanner/reporter, which produce less false-positive reports.I don't know what way Gentoo will choose, but the original kerneloops daemon requires some love to make it run under systemd and improve the oops scanner (can be ported from abrt).
The source of _scanner_ abrt/dumpoops_plugin:
https://git.fedorahosted.org/cgit/abrt.git/tree/src/plugins/abrt-dump-oops.c
The source of _reporter_ abrt/libreport/kerneloops_plugin:
https://git.fedorahosted.org/cgit/libreport.git/tree/src/plugins/reporter-kerneloops.clibreport allows to report manually using CLI:
$ reporter-kerneloops Usage: reporter-kerneloops [-v] [-c CONFFILE]... -d DIRQuoted 2 lines
I think that something that "calls home" might become a concern; maybe we could still provide that as opt-in, but manual submits would be neat.
Few years of experience with Abrt/Bugzilla reports, people fill the additional information in less than 2% of all reports. And, the Abrt team is working towards the least possible interaction with the user. This is mostly to address privacy concerns, as well as leaving the possibility to provide additional data.
Quoted 1 line
Thank you in advance for your explanation.
You are welcome, let me know if you'll need any help further.
Anton.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: oops.kernel.org, reporting. [Was: [ATTEND] oops.kernel.org prospect]
Quoted 20 lines · Joseph Salisbury
Quoted 10 lines · Anton Arapov
Hello,
Thank you for contacting me. I will cross-post this to LKML, as there is an information that might be interested for other folks.
Quoted 4 lines · Tom Wijsman
We, the Gentoo Kernel Project Team at Gentoo Linux, might be interested to contribute oopses to oops.kernel.org; but from a first look over what can be found on the ML and the internet about that site, there seems to be no information available about submitting to oops.kernel.org.
Quoted 8 lines
This kind of information will be there soon.
Quoted 2 lines
So, could you please highlight the details on how submission happens? Does this (have to) happen automatically? Can we do manual submissions?
oops.kernel.org uses the old legacy kerneloops.org's api for data collecting. Both, http://submit.kerneloops.org/submitoops.php (old) and
http://oops.kernel.org/submitoops.php (new, stick to this one) works today.
Quoted 6 lines
The Ubuntu distro is still configured to submit oops to
http://submit.kerneloops.org/submitoops.php using the kerneloops package developed by Arjan. However, it has been disabled since the original kerneloops.org site went down. We can re-enable oops submissions. Will the old submit URL still work, or are you migrating to the new
http://oops.kernel.org/submitoops.php URL?
Old URL still works, so you can use the kerneloops package as is and it should work.
I will update the package itself later, with new url and few improvements for oopses detection.
Anton.
Quoted 47 lines
Quoted 46 lines
Reporting doesn't happen automatically. Two tools available for sending reports: abrt/libreport (1) and former kerneloops daemon (2). Either of these can send reports automatically or manually.
(1) https://fedorahosted.org/abrt/ - iirc, is already packaged in gentoo.
(2) There are no link to kerneloops, this is what I was able to google:
http://ftp.debian.org/debian/pool/main/k/kerneloops/kerneloops_0.12+git20090217.orig.tar.gz
Abrt has a better scanner/reporter, which produce less false-positive reports.I don't know what way Gentoo will choose, but the original kerneloops daemon requires some love to make it run under systemd and improve the oops scanner (can be ported from abrt).
The source of _scanner_ abrt/dumpoops_plugin:
https://git.fedorahosted.org/cgit/abrt.git/tree/src/plugins/abrt-dump-oops.c
The source of _reporter_ abrt/libreport/kerneloops_plugin:
https://git.fedorahosted.org/cgit/libreport.git/tree/src/plugins/reporter-kerneloops.clibreport allows to report manually using CLI:
$ reporter-kerneloops Usage: reporter-kerneloops [-v] [-c CONFFILE]... -d DIRQuoted 2 lines
I think that something that "calls home" might become a concern; maybe we could still provide that as opt-in, but manual submits would be neat.
Few years of experience with Abrt/Bugzilla reports, people fill the additional information in less than 2% of all reports. And, the Abrt team is working towards the least possible interaction with the user. This is mostly to address privacy concerns, as well as leaving the possibility to provide additional data.
Quoted 1 line
Thank you in advance for your explanation.
You are welcome, let me know if you'll need any help further.
Anton.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/