Oops/Warning report of the week of July 4th 2008
Subsystems (2)
This week, a total of 3541 oopses and warnings have been reported, compared to 3794 reports in the previous week.
The stats look very similar to last week; Fedora released a 2.6.25.9 based
kernel upgrade, which led to a new sighting (at rank 12): the rt25xx wireless
driver is calling flush_workqueue() with a NULL parameter in some cases.
There has been a lot of thrash about the last report with regard to inclusion of wireless.git into the Fedora kernel rpms. As an observer I can say that it's both a blessing and a bog. It's a blessing in that this allows bugs to show up early before wireless.git hits mainline (as an example: this is the third or fourth fedora rpm upgrade in a row that showed new and exciting oopses/warnings due to rt25xx... as a result of very active development). It's a bog in that it may expose users to not-quite-ready code. So far it seems the Fedora kernel maintainers are happy enough with the overall balance that they continue the practice.
Per file statistics
548 external/madwifi/wrapper (P)
264 drivers/parport/procfs.c
258 external/madwifi/binary (P)
249 fs/sysfs/dir.c
241 drivers/net/wireless/b43/main.c
229 fs/jbd/journal.c
154 external/fireglx/binary (P)
91 kernel/time/tick-broadcast.c
64 drivers/ata/libata-core.c
58 fs/ext3/super.c
43 kernel/workqueue.c
41 net/mac80211/tx.c
Reports seen on non-tainted kernels
===================================
Rank 2: parport_device_proc_register (warning)
Reported 264 times (2002 total reports)
Duplicate /proc registration in the parport driver
This warning was last seen in version 2.6.26-rc8-git2, and first seen in 2.6.24-rc5. More info: http://www.kerneloops.org/searchweek.php?search=parport_device_proc_register
Rank 4: b43_generate_noise_sample (warning)
Reported 239 times (1124 total reports)
[fixed] too strict WARN_ON in b43 driver
This got fixed recently, and Fedora will have a backport of the fix soon as well. This warning was last seen in version 2.6.26-rc4-git5, and first seen in 2.6.25.3. More info: http://www.kerneloops.org/searchweek.php?search=b43_generate_noise_sample
Rank 5: sysfs_add_one (warning)
Reported 238 times (1835 total reports)
Duplicate sysfs registration warning; USB hubs and USB audio are the most implicated parts This warning was last seen in version 2.6.26-rc3, and first seen in 2.6.24-rc6. More info: http://www.kerneloops.org/searchweek.php?search=sysfs_add_one
Rank 6: journal_update_superblock (warning)
Reported 226 times (1482 total reports)
Caused by the user removing a USB stick while mounted
(This isn't a corner case as much as it used to; todays desktop systems auto-mount USB sticks) This warning was last seen in version 2.6.26, and first seen in 2.6.24-rc6-git1. More info: http://www.kerneloops.org/searchweek.php?search=journal_update_superblock
Rank 8: tick_broadcast_oneshot_control (softlockup)
Reported 91 times (634 total reports)
Some interaction between tickless and systems with an AMD CPU and an ATI chipset (eg only seen on systems that both have an AMD cpu and an ATI chipset) Current suspicion is some kind of time-warp problem that causes the softlockup code to trigger incorrectly
This softlockup was last seen in version 2.6.25.9, and first seen in 2.6.24-rc4. More info: http://www.kerneloops.org/searchweek.php?search=tick_broadcast_oneshot_control
Rank 10: ata_hsm_move (warning)
Reported 63 times (233 total reports)
Linus just merged patches to get better diagnostics for this case This warning was last seen in version 2.6.25.9, and first seen in 2.6.25-rc9-git1. More info: http://www.kerneloops.org/searchweek.php?search=ata_hsm_move
Rank 11: ext3_commit_super (warning)
Reported 53 times (356 total reports)
Likely caused by the user removing a USB stick while mounted
This warning was last seen in version 2.6.26-rc8-git2, and first seen in 2.6.24. More info: http://www.kerneloops.org/searchweek.php?search=ext3_commit_super
Rank 12: flush_workqueue (oops)
Reported 39 times
Bug in the rt25xx driver passing NULL to the flush_workqueue() function This oops was last seen in version 2.6.25.9, and first seen in 2.6.25.9. More info: http://www.kerneloops.org/searchweek.php?search=flush_workqueue
Rank 13: ieee80211_master_start_xmit (warning)
Reported 38 times (183 total reports)
This used to be specific to the iwlwifi drivers, but now shows up with various other drivers as well
This warning was last seen in version 2.6.25.9, and first seen in 2.6.25.4. More info: http://www.kerneloops.org/searchweek.php?search=ieee80211_master_start_xmit
Only seen on tainted kernels
============================
Rank 1: ath_dynamic_sysctl_register (warning)
Reported 462 times (4375 total reports)
[external] Bug in the proprietary madwifi driver
warning only shows up in tainted kernels
This warning was last seen in version 2.6.25.9, and first seen in 2.6.24. More info: http://www.kerneloops.org/searchweek.php?search=ath_dynamic_sysctl_register
Rank 3: init_ath_hal (warning)
Reported 258 times (2658 total reports)
[external] Bug in the proprietary madwifi driver
warning only shows up in tainted kernels
This warning was last seen in version 2.6.25.9, and first seen in 2.6.24. More info: http://www.kerneloops.org/searchweek.php?search=init_ath_hal
Rank 7: firegl_ioctl (warning)
Reported 124 times (997 total reports)
[external] Bug in the proprietary fireglx driver
warning only shows up in tainted kernels
This warning was last seen in version 2.6.25.10, and first seen in 2.6.25. More info: http://www.kerneloops.org/searchweek.php?search=firegl_ioctl
Rank 9: ath_sysctl_register (warning)
Reported 86 times (1109 total reports)
[external] Bug in the proprietary madwifi driver
warning only shows up in tainted kernels
This warning was last seen in version 2.6.25.9, and first seen in 2.6.24-rc4-git4. More info: http://www.kerneloops.org/searchweek.php?search=ath_sysctl_register
Quoted 10 lines · Arjan van de Ven
The stats look very similar to last week; Fedora released a 2.6.25.9 based
kernel upgrade, which led to a new sighting (at rank 12): the rt25xx wireless
driver is callingflush_workqueue()with a NULL parameter in some cases.
There has been a lot of thrash about the last report with regard to inclusion of wireless.git into the Fedora kernel rpms. As an observer I can say that it's both a blessing and a bog. It's a blessing in that this allows bugs to show up early before wireless.git hits mainline (as an example: this is the third or fourth fedora rpm upgrade in a row that showed new and exciting oopses/warnings due to rt25xx... as a result of very active development). It's a bog in that it may expose users to not-quite-ready code. So far it seems the Fedora kernel maintainers are happy enough with the overall balance that they continue the practice.
I actually think we need to scale things back a notch wrt pushing wireless.git bits to users of released distros. The recent disaster in wireless caused a shitstorm in bugzilla that we never even saw in rawhide. A clear sign that we're pushing things too fast to users.
It's great that we're getting this stuff tested, but at the same time, it doesn't give a great impression, and makes users reluctant to always apply the latest updates if the last time around they have to deal with this kind of fallout.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 13 lines · Dave Jones
Quoted 10 lines · Arjan van de Ven
The stats look very similar to last week; Fedora released a 2.6.25.9 based
kernel upgrade, which led to a new sighting (at rank 12): the rt25xx wireless
driver is callingflush_workqueue()with a NULL parameter in some cases.
There has been a lot of thrash about the last report with regard to inclusion of wireless.git into the Fedora kernel rpms. As an observer I can say that it's both a blessing and a bog. It's a blessing in that this allows bugs to show up early before wireless.git hits mainline (as an example: this is the third or fourth fedora rpm upgrade in a row that showed new and exciting oopses/warnings due to rt25xx... as a result of very active development). It's a bog in that it may expose users to not-quite-ready code. So far it seems the Fedora kernel maintainers are happy enough with the overall balance that they continue the practice.
Quoted 4 lines
I actually think we need to scale things back a notch wrt pushing wireless.git bits to users of released distros. The recent disaster in wireless caused a shitstorm in bugzilla that we never even saw in rawhide. A clear sign that we're pushing things too fast to users.
Well I agree that we are pushing things too quickly to users. However I believe that improper use of Bodhi is at least as much to blame as I am.
FWIW, I'd like to point out that a user (Stefan Becker) helped to sort-out my "human error" screw-up (i.e. not some buggy patch from upstream) that caused the latest flurry regarding busted TKIP for mac80211. IMHO this is in the best traditions of open source, and a credit to our community.
Quoted 4 lines
It's great that we're getting this stuff tested, but at the same time, it doesn't give a great impression, and makes users reluctant to always apply the latest updates if the last time around they have to deal with this kind of fallout.
When patches went-in more slowly, we still hit nearly as many problems. Only then it took longer to solve them because we (i.e. Fedora) were so far behind what the upstream developers were doing that we couldn't get their attention. As it stands, Fedora gets quick attention even from developers who use other distros.
Still, maybe we are approaching a point where it is prudent to slow some things down for Fedora. I sent a proposal along those lines to fedora-kernel-list. I just don't think we should be too quick to drop the goodness we are getting by staying close to upstream on wireless. And in either event, we may need to find another monkey, because I've been finding it hard enough to keep-up with dumping patches to Fedora as-is.
John
Signature · 2 lines
John W. Linville [email protected]
Hi,
Quoted 5 lines
Rank 12:
flush_workqueue(oops)
Reported 39 times
Bug in the rt25xx driver passing NULL to theflush_workqueue()function This oops was last seen in version 2.6.25.9, and first seen in 2.6.25.9. More info: http://www.kerneloops.org/searchweek.php?search=flush_workqueue
Hmm, interesting I hadn't received such reports yet..
I am afraid this was caused by a patch which was send to 2.6.26 to fix a lockdep problem. :(
John, earlier today I released the patch:
rt2x00: Use ieee80211_hw->workqueue again http://article.gmane.org/gmane.linux.kernel.wireless.general/17305
actually intended for 2.6.27, but if this bug means 2.6.26 might be crashing, it might be wiser to push it to 2.6.26.
Ivo
Quoted 9 lines · Ivo van Doorn
Hi,
Quoted 5 lines
Rank 12:
flush_workqueue(oops)
Reported 39 times
Bug in the rt25xx driver passing NULL to theflush_workqueue()function This oops was last seen in version 2.6.25.9, and first seen in 2.6.25.9. More info: http://www.kerneloops.org/searchweek.php?search=flush_workqueueHmm, interesting I hadn't received such reports yet..
it only very recently showed up on kerneloops.org (the .9 release from Fedora is only a 2 or 3 days old). In addition, while I monitor what's going on, oops patterns need to cross roughly a 25/week pattern for me to notice them as standing out...
If I see other patterns come up I'll send out email again ;-)
Quoted 9 lines · Arjan van de Ven
Rank 8:
tick_broadcast_oneshot_control(softlockup)
Reported 91 times (634 total reports)Some interaction between tickless and systems with an AMD CPU and an ATI chipset (eg only seen on systems that both have an AMD cpu and an ATI chipset) Current suspicion is some kind of time-warp problem that causes the softlockup code to trigger incorrectly
Quoted 4 lines
This softlockup was last seen in version 2.6.25.9,
and first seen in 2.6.24-rc4. More info:http://www.kerneloops.org/searchweek.php?search=tick_broadcast_oneshot_control
ok, so it's a false positive due to bad timer readout, not a real lockup. That probably also explains how it was able to show up in such numbers - the system kept functioning just fine so every system that produced this warning was able to report it to kerneloops.org.
i'll think about extending the softlockup code with time warp detection and reporting - that would be a useful facility in itself as right now there's nothing that warns about time warps in monotonic system time.
Ingo