oops/warning report for the week of November 26, 2008
Subsystems (3)
In collecting this report, oopses and warnings with versions prior to 2.6.27 are ignored. This week, a total of 5450 oopses and warnings have been reported of version 2.6.27+, compared to 2198 reports in the previous week.
This report is a bit different than the previous weeks; all 2.6.26 and earlier issues are no longer used, which means the top 12 has shuffled quite a bit, with some new star appearances.
Also I've reworked the "are these two backtraces the same" algorithm; the website should now be presenting a more compact/concise view due to having the backtraces consolidated in a much more logical (for the human) way.
Per file statistics
936 external/virtualbox/module
602 drivers/pci/slot.c
455 drivers/net/wireless/iwlwifi/iwl-tx.c
364 kernel/power/main.c
274 drivers/net/r8169.c
231 drivers/net/wireless/iwlwifi/iwl-3945-rs.c
231 fs/jbd/journal.c
227 arch/x86/include/asm/mtrr.h
147 drivers/ata/libata-sff.c
137 drivers/net/sis900.c
71 net/ipv4/tcp.c
62 drivers/gpu/drm/radeon/radeon_cp.c
Rank 1: VBoxDrvLinuxIOCtl (warning)
Reported 934 times (1635 total reports)
[external] bug in the VirtualBox drivers
This warning was last seen in version 2.6.28-rc3, and first seen in 2.6.25.11. More info: http://www.kerneloops.org/searchweek.php?search=VBoxDrvLinuxIOCtl
Rank 2: pci_create_slot (warning)
Reported 603 times (639 total reports)
BIOS provided duplicated slot names, the PCI layer blindly passes to sysfs This warning was last seen in version 2.6.27.5, and first seen in 2.6.27-rc7-git1. More info: http://www.kerneloops.org/searchweek.php?search=pci_create_slot
Rank 3: iwl_tx_cmd_complete (warning)
Reported 455 times (693 total reports)
Bug in the IWL wireless driver; partial fix available
This warning was last seen in version 2.6.28-rc4, and first seen in 2.6.27-rc9. More info: http://www.kerneloops.org/searchweek.php?search=iwl_tx_cmd_complete
Rank 4: suspend_test_finish (warning)
Reported 362 times (1202 total reports)
Fedora is shipping with the suspend test on.. and it's failing everywhere. The patch to report what fails is in 2.6.28-rc6 and later
This warning was last seen in version 2.6.28-rc1, and first seen in 2.6.27-rc0-git14. More info: http://www.kerneloops.org/searchweek.php?search=suspend_test_finish
Rank 5: dev_watchdog(r8169) (oops)
Reported 274 times (1414 total reports)
Network driver not handling timeouts itself.
This oops was last seen in version 2.6.28-rc4, and first seen in 2.6.26.6. More info: http://www.kerneloops.org/searchweek.php?search=dev_watchdog(r8169)
Rank 6: rs_get_rate (oops)
Reported 232 times (1152 total reports)
Bug in the Intel IWL wireless drivers
This oops was last seen in version 2.6.27.5, and first seen in 2.6.25-rc2-git5. More info: http://www.kerneloops.org/searchweek.php?search=rs_get_rate
Rank 7: journal_update_superblock (warning)
Reported 231 times (6506 total reports)
Likely caused by the user removing a USB stick while mounted
This warning was last seen in version 2.6.27.7, and first seen in 2.6.24-rc6-git1. More info: http://www.kerneloops.org/searchweek.php?search=journal_update_superblock
Rank 8: mtrr_trim_uncached_memory (warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.
This warning was last seen in version 2.6.27.5, and first seen in 2.6.24. More info: http://www.kerneloops.org/searchweek.php?search=mtrr_trim_uncached_memory
Rank 9: __atapi_pio_bytes (warning)
Reported 146 times (224 total reports)
Alan said this was due to some other layer giving the libata drivers a weird scatter gather list. It just happens a lot, and somehow it mostly happens in virtualized environments
This warning was last seen in version 2.6.27.5, and first seen in 2.6.27.4. More info: http://www.kerneloops.org/searchweek.php?search=__atapi_pio_bytes
Rank 10: dev_watchdog(sis900) (oops)
Reported 137 times (1538 total reports)
This oops was last seen in version 2.6.27.6, and first seen in 2.6.26-rc4-git2. More info: http://www.kerneloops.org/searchweek.php?search=dev_watchdog(sis900)
Rank 11: tcp_recvmsg (warning)
Reported 71 times (167 total reports)
This warning was last seen in version 2.6.27.5, and first seen in 2.6.25. More info: http://www.kerneloops.org/searchweek.php?search=tcp_recvmsg
Rank 12: dev_watchdog(atl1) (oops)
Reported 56 times (109 total reports)
This oops was last seen in version 2.6.27.5, and first seen in 2.6.26.6. More info: http://www.kerneloops.org/searchweek.php?search=dev_watchdog(atl1)
Rank 13: nv_set_page_attrib_cached (warning)
Reported 56 times (65 total reports)
[external] bug in the binary nvidia driver
warning only shows up in tainted kernels
This warning was last seen in version 2.6.27.5, and first seen in 2.6.27.5. More info: http://www.kerneloops.org/searchweek.php?search=nv_set_page_attrib_cached
Quoted 6 lines · Arjan van de Ven
Rank 2:
pci_create_slot(warning)
Reported 603 times (639 total reports)
BIOS provided duplicated slot names, the PCI layer blindly passes to sysfs This warning was last seen in version 2.6.27.5, and first seen in
2.6.27-rc7-git1. More info:
http://www.kerneloops.org/searchweek.php?search=pci_create_slot
IIRC we fixed this one post-2.6.27. I didn't send the patches back to -stable because they were a bit big, but if someone were sufficiently motiviated I'm sure the backport wouldn't be that hard...
Jesse
Quoted 8 lines · Jesse Barnes
Quoted 6 lines · Arjan van de Ven
Rank 2:
pci_create_slot(warning)
Reported 603 times (639 total reports)
BIOS provided duplicated slot names, the PCI layer blindly passes to sysfs This warning was last seen in version 2.6.27.5, and first seen in
2.6.27-rc7-git1. More info:
http://www.kerneloops.org/searchweek.php?search=pci_create_slot
Quoted 4 lines
IIRC we fixed this one post-2.6.27. I didn't send the patches back to -stable because they were a bit big, but if someone were sufficiently motiviated I'm sure the backport wouldn't be that hard...
having the commit IDs mentioned here would be nice, should anyone feel motivated.
Ingo
* Jesse Barnes <[email protected]>:
Quoted 11 lines · Arjan van de Ven
Quoted 6 lines
Rank 2:
pci_create_slot(warning)
Reported 603 times (639 total reports)
BIOS provided duplicated slot names, the PCI layer blindly passes to sysfs This warning was last seen in version 2.6.27.5, and first seen in
2.6.27-rc7-git1. More info:
http://www.kerneloops.org/searchweek.php?search=pci_create_slotIIRC we fixed this one post-2.6.27. I didn't send the patches back to -stable because they were a bit big, but if someone were sufficiently motiviated I'm sure the backport wouldn't be that hard...
I can do this backport. A few questions though...
We're seeing a proliferation of this one presumably because Fedora10 uses 2.6.27.5 as a starting point? If I just backport the fixes against Greg's latest tree, do I have to do anything special to make sure they get into the Fedora kernel?
Also, does kerneloops capture any of the machine information, like DMI output, etc. or does it just get the oops? It would be nice to see which machines out there have the broken BIOS that causes this oops.
Thanks.
/ac
Quoted 14 lines · On Thu, 27 Nov 2008 12:42:10 -0700 Alex Chiang <[email protected]> wrote:
* Jesse Barnes <[email protected]>:
Quoted 8 lines · Arjan van de Ven
Quoted 6 lines
Rank 2:
pci_create_slot(warning)
Reported 603 times (639 total reports)
BIOS provided duplicated slot names, the PCI layer
blindly passes to sysfs This warning was last seen in version 2.6.27.5, and first seen in 2.6.27-rc7-git1. More info:
http://www.kerneloops.org/searchweek.php?search=pci_create_slotQuoted 4 lines
IIRC we fixed this one post-2.6.27. I didn't send the patches back to -stable because they were a bit big, but if someone were sufficiently motiviated I'm sure the backport wouldn't be that hard...
Quoted 6 lines
I can do this backport. A few questions though...
We're seeing a proliferation of this one presumably because Fedora10 uses 2.6.27.5 as a starting point? If I just backport the fixes against Greg's latest tree, do I have to do anything special to make sure they get into the Fedora kernel?
Fedora tends to follow -stable quite closely so that ought to be enough
Quoted 5 lines
Also, does kerneloops capture any of the machine information, like DMI output, etc. or does it just get the oops? It would be nice to see which machines out there have the broken BIOS that causes this oops.
right now we do this for oopses, but not for warnings ;( I'll make a patch to add this; it's generally useful.
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 5 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.
the warning here means: "the BIOS messed up but we fixed it up for you just fine".
Should we print a DMI descriptor so that it can be tracked back to the bad BIOSen in question? Or should we (partially) silence the warning itself? Those BIOS bugs need fixing really: older kernels will boot up with bad MTRR settings - resulting in a super-slow system or other weirdnesses. We can tone down the message so that it doesnt show up in kerneloops.org. It's up to you.
Ingo
Quoted 10 lines · Ingo Molnar
Quoted 5 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.the warning here means: "the BIOS messed up but we fixed it up for you just fine".
Quoted 6 lines
Should we print a DMI descriptor so that it can be tracked back to the bad BIOSen in question? Or should we (partially) silence the warning itself? Those BIOS bugs need fixing really: older kernels will boot up with bad MTRR settings - resulting in a super-slow system or other weirdnesses. We can tone down the message so that it doesnt show up in kerneloops.org. It's up to you.
I actually think we're doing something wrong here, since so many platforms have this behavior. It's likely that there's an undocumented, additional check needed to determine whether a slot is hot pluggable. Matthew Garrett recently posted a patch to check for ACPI _RMV methods, which should be an improvement. I'll be putting that into linux-next soon for testing.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 9 lines · Ingo Molnar
Quoted 5 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.the warning here means: "the BIOS messed up but we fixed it up for you just fine".
I don't believe that right now.
we see so many of these, including many "there's no MTRRs at all", that I am seriously suspecting that our code is just incorrect somehow and triggering too much.
Quoted 8 lines · Arjan van de Ven
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I
suspect we are too picky in accepting the MTRR configuration.
Quoted 6 lines
Quoted 2 lines
the warning here means: "the BIOS messed up but we fixed it up for you just fine".
I don't believe that right now. we see so many of these, including many "there's no MTRRs at all", that I am seriously suspecting that our code is just incorrect somehow and triggering too much.
well we looked at existing reports and Linux was right to fix them up. Show us one that is incorrect, then we can fix it up.
the "no MTRR's" are vmware/(also qemu?) guests not implementing a full CPU emulation.
Ingo
Quoted 19 lines · On Thu, 27 Nov 2008 21:18:36 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 9 lines · Arjan van de Ven
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.Quoted 6 lines
Quoted 2 lines
the warning here means: "the BIOS messed up but we fixed it up for you just fine".
I don't believe that right now. we see so many of these, including many "there's no MTRRs at all", that I am seriously suspecting that our code is just incorrect somehow and triggering too much.
Quoted 5 lines
well we looked at existing reports and Linux was right to fix them up. Show us one that is incorrect, then we can fix it up.
the "no MTRR's" are vmware/(also qemu?) guests not implementing a full CPU emulation.
... and it's still our fault in part, since we don't even check to see if a cpu claims to support MTRR before complaining about it...
easy to fix though:
Quoted 1 line
From
7e987ae541c41ce908b414fee9d8e2fd2099a083Mon Sep 17 00:00:00 2001
From: Arjan van de Ven <[email protected]>
Date: Thu, 27 Nov 2008 12:25:47 -0800
Subject: [PATCH] x86: make sure the CPU advertizes MTRR support before complaining about the lack thereoff...
We complain loudly if a CPU does not have MTRR support... but we don't check if the CPU exposes MTRR support in the CPUID flags first. While this might not fix all of the broken virtualization systems out there, it will at least fix those that properly don't advertize things they don't support.
--- arch/x86/kernel/cpu/mtrr/main.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/mtrr/main.c +2 -0
@@ -1567,6 +1567,8 @@ int __init mtrr_trim_uncached_memory(unsigned long end_pfn)* Make sure we only trim uncachable memory on machines that * support the Intel MTRR architecture: */ +if (!cpu_has_mtrr) +return 0 ;if (!is_cpu (INTEL ) || disable_mtrr_trim)return 0 ;rdmsr (MTRRdefType_MSR, def, dummy);
Signature · 8 lines
1.6.0.4 -- Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 25 lines · Arjan van de Ven
Quoted 22 lines · On Thu, 27 Nov 2008 21:18:36 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 9 lines · Arjan van de Ven
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.Quoted 6 lines
Quoted 2 lines
the warning here means: "the BIOS messed up but we fixed it up for you just fine".
I don't believe that right now. we see so many of these, including many "there's no MTRRs at all", that I am seriously suspecting that our code is just incorrect somehow and triggering too much.
well we looked at existing reports and Linux was right to fix them up. Show us one that is incorrect, then we can fix it up.
Quoted 7 lines
Quoted 2 lines
the "no MTRR's" are vmware/(also qemu?) guests not implementing a full CPU emulation.
... and it's still our fault in part, since we don't even check to see if a cpu claims to support MTRR before complaining about it...
easy to fix though:
IIRC the problem is that vmware _does_ claim that it supports MTRRs.
Ingo
Quoted 1 line · On Thu, 27 Nov 2008 21:47:14 +0100 Ingo Molnar <[email protected]> wrote:
IIRC the problem is that vmware _does_ claim that it supports MTRRs.
it might.
but even if they would fix that, we would still WARN ( at least we should do our side correctly...
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 8 lines · Arjan van de Ven
Quoted 1 line · On Thu, 27 Nov 2008 21:47:14 +0100 Ingo Molnar <[email protected]> wrote:
IIRC the problem is that vmware _does_ claim that it supports MTRRs.
it might.
but even if they would fix that, we would still WARN ( at least we should do our side correctly...
As pointed out in other parts of the thread, that is not the case.
Anyway, as i said it in the onset, if you think we should remove the warning altogether, or tweak it, we can do that - it is important to have relevant warnings show up in kerneloops.org.
To sum it up: the only remaining MTRR warnings we know of are either:
1) apparently genuine BIOS bugs that do cause problems if the (new) kernel does not fix them up.
The MTRR warning is relevant and correct in those cases.
or:
2) sucky virtualization solutions that cheat the guest OS by faking "MTRR support" in the CPUID info, but not actually showing any MTRRs. These virtualization solutions do not even properly identify themselves to the kernel.
The MTRR warning is unnecessary in this case.
So what we did in the x86 tree was remove the warning in the second case - is to properly identify vmware (and in general, virtualization) guests.
It was not a simple oneliner:
earth4:~/tip> gll linus..x86/detect-hyper
4e42ebd: x86: hypervisor - fix sparse warnings
c450d78: x86: vmware - fix sparse warnings
fd8cd7e: x86: vmware: look for DMI string in the product serial key 6bdbfe9: x86: VMware: Fix vmware_get_tsc code
395628e: x86: Skip verification by the watchdog for TSC clocksource.
eca0cd0: x86: Add a synthetic TSC_RELIABLE feature bit. 88b094f: x86: Hypervisor detection and get tsc_freq from hypervisor 49ab56a: x86: add X86_FEATURE_HYPERVISOR feature bit
b2bcc7b: x86: add a synthetic TSC_RELIABLE feature bit
and it will benefit vmware guests in many more areas than just a sharper MTRR warning message. That code is queued up for v2.6.29.
Ingo
H. Peter Anvin
reviewerarch/x86/kernel/cpu/mtrr/main.c int __init mtrr_trim_uncached_memory(unsigned long end_pfn) · line 1574
Quoted 5 lines · Arjan van de Ven
arch/x86/kernel/cpu/mtrr/main.c +2
-0+if (!cpu_has_mtrr) +return 0 ;if (!is_cpu (INTEL ) || disable_mtrr_trim)return 0 ;rdmsr (MTRRdefType_MSR, def, dummy);
cpu_has_mtrr there should presumably replace is_cpu(INTEL). I'm not sure if this can be replaced by use_intel(); in particular use_intel() relies on mtrr_if having been initialized.
Looking...
-hpa (out of town for Thanksgiving)
Quoted 49 lines · Arjan van de Ven
Quoted 12 lines · On Thu, 27 Nov 2008 21:18:36 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 10 lines · Arjan van de Ven
Quoted 9 lines · Ingo Molnar
Quoted 5 lines · Arjan van de Ven
Rank 8:
mtrr_trim_uncached_memory(warning)
Reported 227 times (619 total reports)
There is a high number of machines where our MTRR checks trigger. I suspect we are too picky in accepting the MTRR configuration.the warning here means: "the BIOS messed up but we fixed it up for you just fine".
Quoted 8 lines
Quoted 3 lines
I don't believe that right now. we see so many of these, including many "there's no MTRRs at all", that I am seriously suspecting that our code is just incorrect somehow and triggering too much.
well we looked at existing reports and Linux was right to fix them up. Show us one that is incorrect, then we can fix it up.
the "no MTRR's" are vmware/(also qemu?) guests not implementing a full CPU emulation.
... and it's still our fault in part, since we don't even check to see if a cpu claims to support MTRR before complaining about it...
easy to fix though:
From
7e987ae541c41ce908b414fee9d8e2fd2099a083Mon Sep 17 00:00:00 2001
From: Arjan van de Ven <[email protected]>
Date: Thu, 27 Nov 2008 12:25:47 -0800
Subject: [PATCH] x86: make sure the CPU advertizes MTRR support before complaining about the lack thereoff...We complain loudly if a CPU does not have MTRR support... but we don't check if the CPU exposes MTRR support in the CPUID flags first. While this might not fix all of the broken virtualization systems out there, it will at least fix those that properly don't advertize things they don't support.
Signed-off-by Arjan van de Ven <[email protected]>--- arch/x86/kernel/cpu/mtrr/main.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)arch/x86/kernel/cpu/mtrr/main.c +0
-0
arch/x86/kernel/cpu/mtrr/main.c int __init mtrr_trim_uncached_memory(unsigned long end_pfn) · line 1571
Quoted 5 lines
arch/x86/kernel/cpu/mtrr/main.c +2
-0@@ -1567,6 +1567,8 @@ int __init mtrr_trim_uncached_memory(unsigned long end_pfn)* Make sure we only trim uncachable memory on machines that * support the Intel MTRR architecture: */ +if (!cpu_has_mtrr) +return 0 ;
that is not needed, we already check that in mtrr_bp_init before this function is called, and it will assign mtrr_if
and
#define is_cpu(vnd) (mtrr_if && mtrr_if->vendor == X86_VENDOR_##vnd)
will make it sure mtrr is there.
ps: here INTEL mean any cpu has same interface like intel cpu's
YH
Quoted 3 lines
arch/x86/kernel/cpu/mtrr/main.c +0
-0if (!is_cpu (INTEL ) || disable_mtrr_trim)return 0 ;rdmsr (MTRRdefType_MSR, def, dummy);
H. Peter Anvin
reviewerQuoted 6 lines · Arjan van de Ven
arch/x86/kernel/cpu/mtrr/main.c +0
-0
arch/x86/kernel/cpu/mtrr/main.c int __init mtrr_trim_uncached_memory(unsigned long end_pfn) · line 1574
Quoted 8 lines
arch/x86/kernel/cpu/mtrr/main.c +2
-0@@ -1567,6 +1567,8 @@ int __init mtrr_trim_uncached_memory(unsigned long end_pfn)* Make sure we only trim uncachable memory on machines that * support the Intel MTRR architecture: */ +if (!cpu_has_mtrr) +return 0 ;if (!is_cpu (INTEL ) || disable_mtrr_trim)return 0 ;rdmsr (MTRRdefType_MSR, def, dummy);
Okay... is_cpu() here is defined as:
#define is_cpu(vnd) (mtrr_if && mtrr_if->vendor == X86_VENDOR_##vnd)
... so an MTRR interface has been identified. Therefore testing cpu_has_mtrr is redundant.
As far as use_intel() versus is_cpu(INTEL), it looks to me as though the two are identical in the current code -- mtrr_if->vendor is never set in the generic code, and so defaults to 0 - meaning X86_VENDOR_INTEL.
All in all, it looks like the vendor ID stuff is a bad case of "works by accident" in the MTRR code, however, *given the current code* I conclude that is_cpu(INTEL) == use_intel() and that neither can be true without MTRRs enabled.
-hpa
[trimmed the cc list down to netdev only]
Quoted 5 lines · On Wed, 26 Nov 2008 15:11:14 -0800 Arjan van de Ven <[email protected]>…
Rank 12:
dev_watchdog(atl1)(oops)
Reported 56 times (109 total reports)
This oops was last seen in version 2.6.27.5, and first seen
in 2.6.26.6. More info:
http://www.kerneloops.org/searchweek.php?search=dev_watchdog(atl1)
I can't reproduce this, so I've launched a request at fedoraforum.org hoping I can snag a Fedora user who's encountering the bug and willing to test.
The tx timeout reports at kerneloops.org appear to be happening on a startling variety of network drivers (startling to me, anyway): r8169, atl1, atl2, sis900, cdc_ether, orinoco_cs, tg3, ne2k-pci, via-rhine, 8139too, ath_pci, e1000, gl620a, sky2, hso, fealnx, forcedeth; probably others, but I quit looking.
Is it correct to assume all these drivers are showing symptoms of the poor timeout handling you mentioned in your r8169 comment, or is the occasional tx timeout to be expected, and the leaders in this category (r8169, sis900, atl1) are the only ones suffering from deficient timeout handling?
Jay
Quoted 8 lines · On Fri, 28 Nov 2008 11:18:27 -0600 Jay Cliburn <[email protected]> wrote:
[trimmed the cc list down to netdev only]
Quoted 5 lines
Rank 12:
dev_watchdog(atl1)(oops)
Reported 56 times (109 total reports)
This oops was last seen in version 2.6.27.5, and first seen
in 2.6.26.6. More info:
http://www.kerneloops.org/searchweek.php?search=dev_watchdog(atl1)
Quoted 5 lines
The tx timeout reports at kerneloops.org appear to be happening on a startling variety of network drivers (startling to me, anyway): r8169, atl1, atl2, sis900,
cdc_ether,orinoco_cs, tg3, ne2k-pci, via-rhine, 8139too,ath_pci, e1000, gl620a, sky2, hso, fealnx, forcedeth; probably others, but I quit looking.
to be specific in counts, the data I have so far is:
count | guilty
-------+----------------------------
1599 | dev_watchdog(sis900) 1501 | dev_watchdog(r8169) 280 | dev_watchdog(via-rhine) 264 | dev_watchdog(cdc_ether) 213 | dev_watchdog(usbnet) 192 | dev_watchdog(8139too) 164 | dev_watchdog(8390) 158 | dev_watchdog(via_rhine) 129 | dev_watchdog(ne2k-pci) 122 | dev_watchdog(atl1) 102 | dev_watchdog(atl2) 101 | dev_watchdog(orinoco)
and then a long tail of sub-100, omitted to keep this mail not too long; if anyone wants data on his/her driver not in the list, let me know.
(please don't read too much in the word "guilty"; it's just the name of the column in the kerneloops.org database used for identifing which function was the prime suspect of a backtrace)
Quoted 6 lines
Is it correct to assume all these drivers are showing symptoms of the poor timeout handling you mentioned in your r8169 comment, or is the occasional tx timeout to be expected, and the leaders in this category (r8169, sis900, atl1) are the only ones suffering from deficient timeout handling?
For me, sis900 and r8169 stand out; if you look at the data in the table above, both of these are an order of magnitude more frequent than the rest of the pack. ATL1 isn't doing all that bad in this regard, although your driver is still a little higher than other popular cards like tg3, e1000, e1000e etc. (those are all sub-50).
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 17 lines · On Fri, 28 Nov 2008 09:32:17 -0800 Arjan van de Ven <[email protected]>…
to be specific in counts, the data I have so far is:
count | guilty
-------+----------------------------1599 | dev_watchdog(sis900) 1501 | dev_watchdog(r8169) 280 | dev_watchdog(via-rhine) 264 | dev_watchdog(cdc_ether) 213 | dev_watchdog(usbnet) 192 | dev_watchdog(8139too) 164 | dev_watchdog(8390) 158 | dev_watchdog(via_rhine) 129 | dev_watchdog(ne2k-pci) 122 | dev_watchdog(atl1) 102 | dev_watchdog(atl2) 101 | dev_watchdog(orinoco)
Quoted 3 lines
ATL1 isn't doing all that bad in this
regard, although your driver is still a little higher than other popular cards like tg3, e1000, e1000e etc.
...And that's what troubles me: the L1 chip isn't what I'd characterize as "popular" -- it's LOM only, and it's found in only about 25 mainboards that I know of (from voluntary user reports) -- yet its prevalence in the tx timeout list seems to be quickly rising.
Can you produce a list from your database for me that includes the kernel version for each of the 122 reported atl1 dev_watchdog warnings? I'd like to see if I can correlate an increase in the warnings with a particular change we made.
Thanks.
Thanks.
Quoted 10 lines
Quoted 4 lines
ATL1 isn't doing all that bad in this
regard, although your driver is still a little higher than other popular cards like tg3, e1000, e1000e etc....And that's what troubles me: the L1 chip isn't what I'd characterize as "popular" -- it's LOM only, and it's found in only about 25 mainboards that I know of (from voluntary user reports) -- yet its prevalence in the tx timeout list seems to be quickly rising.
Quoted 4 lines
Can you produce a list from your database for me that includes the kernel version for each of the 122 reported atl1
dev_watchdogwarnings? I'd like to see if I can correlate an increase in the warnings with a particular change we made.
=> select count(version), version from oopses where guilty='dev_watchdog(atl1)' group by version order by version desc;
count | version
-------+-----------------
93 | 2.6.27.5
6 | 2.6.27.4
1 | 2.6.27.3
1 | 2.6.27.2
1 | 2.6.27-rc9
1 | 2.6.27-rc7-git1
1 | 2.6.27-rc7
1 | 2.6.27-rc6
6 | 2.6.27-rc3
7 | 2.6.27
4 | 2.6.26.6(11 rows)
or in more detail:
=> select count(full_version), full_version from oopses where \
guilty='dev_watchdog(atl1)' group by full_version order by \full_version desc;
count | full_version
-------+-----------------------------------
2 | 2.6.27.5-94.fc10.x86_64
26 | 2.6.27.5-41.fc9.x86_64
12 | 2.6.27.5-41.fc9.i686
15 | 2.6.27.5-37.fc9.x86_64
19 | 2.6.27.5-37.fc9.i686
11 | 2.6.27.5-117.fc10.x86_64
4 | 2.6.27.5-117.fc10.i686
1 | 2.6.27.5-109.fc10.x86_64
2 | 2.6.27.5-109.fc10.i686.PAE
1 | 2.6.27.5-109.fc10.i686
1 | 2.6.27.4-79.fc10.i686
1 | 2.6.27.4-68.fc10.x86_64
3 | 2.6.27.4-68.fc10.i686
1 | 2.6.27.4-26.fc9.x86_64
1 | 2.6.27.3-34.rc1.fc10.i686.PAE
1 | 2.6.27.2-23.rc1.fc10.x86_64
4 | 2.6.27-wl
1 | 2.6.27-rc7
1 | 2.6.27-rc6-wl-AUS32
6 | 2.6.27-rc3-wl-8KS-UVC
1 | 2.6.27-7-generic
1 | 2.6.27-0.398.rc9.fc10.x86_64
1 | 2.6.27-0.352.rc7.git1.fc10.x86_64
2 | 2.6.27
3 | 2.6.26.6-79.fc9.x86_64
1 | 2.6.26.6-79.fc9.i686(26 rows)
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
atl1 transmit timeout Was: Re: oops/warning report for the week of November 26, 2008
Quoted 14 lines · On Fri, 28 Nov 2008 10:50:44 -0800 Arjan van de Ven <[email protected]>…
=> select count(version), version from oopses where guilty='dev_watchdog(atl1)' group by version order by version desc;
count | version -------+-----------------93 | 2.6.27.5 6 | 2.6.27.4 1 | 2.6.27.3 1 | 2.6.27.2 1 | 2.6.27-rc9 1 | 2.6.27-rc7-git1 1 | 2.6.27-rc7 1 | 2.6.27-rc6 6 | 2.6.27-rc3 7 | 2.6.27 4 | 2.6.26.6
Quoted 1 line
(11 rows)
Wow. 4 hits in 2.6.26, then 118 in 2.6.27.
A history of changes between 2.6.26 and 2.6.27.5 shows a mere six changes to the driver.
commit event ====== ===== 788a5f3f 2.6.27.5 8dc186c1 atl1: fix vlan tag regression 056c7145 2.6.27.4 322df44b 2.6.27.3 6bcd6d77 2.6.27.2 bc5b8bb6 2.6.27.1 3fa8749e 2.6.27 4330ed8e 2.6.27-rc9 94aca1da 2.6.27-rc8 72d31053 2.6.27-rc7 adee14b2 2.6.27-rc6 24342c34 2.6.27-rc5 82c26a9d atl1: disable TSO by default 6a55617e 2.6.27-rc4 30a2f3c6 2.6.27-rc3 c2ac3ef3 atl1: deal with hardware rx checksum bug 0967d61e 2.6.27-rc2 6e86841d 2.6.27-rc1 39d48157 atl1: Do not wake queue before queue has been started. b102df14 atl1: use netdev_alloc_skb d63ddcec misc drivers/net endianness noise bce7f793 2.6.26
The only one that jumps out at me is 39d48157, which contains, in part:
commit 39d48157ac1a0ff3ec81212e5451bfd1bf5f50db
Author: David S. Miller <[email protected]> Date: Mon Jul 21 08:28:37 2008 -0700
atl1: Do not wake queue before queue has been started.
Based upon a bug report by Alexey Dobriyan, the patch is also tested by him and confirmed to fix the problem.
Packet flow during link state events should not be done by waking and stopping the TX queue anyways, that is handled transparently by netif_carrier_{on,off}().
So, remove the netif_{wake,stop}_queue() calls in the link check code, and add the necessary netif_start_queue() call to atl1_up().
[...]
@@ -2627,6 +2625,7 @@ static s32 atl1_up(struct atl1_adapter *adapter)
mod_timer(&adapter->watchdog_timer, jiffies);
atlx_irq_enable(adapter);
atl1_check_link(adapter);
+ netif_start_queue(netdev);
return 0;Would it be reasonable to increase the above mod_timer() expiry to jiffies + (5 * HZ)?
Re: atl1 transmit timeout Was: Re: oops/warning report for the week of November 26, 2008
Quoted 19 lines · Jay Cliburn
Quoted 14 lines · On Fri, 28 Nov 2008 10:50:44 -0800 Arjan van de Ven <[email protected]>…
=> select count(version), version from oopses where guilty='dev_watchdog(atl1)' group by version order by version desc;
count | version -------+-----------------93 | 2.6.27.5 6 | 2.6.27.4 1 | 2.6.27.3 1 | 2.6.27.2 1 | 2.6.27-rc9 1 | 2.6.27-rc7-git1 1 | 2.6.27-rc7 1 | 2.6.27-rc6 6 | 2.6.27-rc3 7 | 2.6.27 4 | 2.6.26.6Quoted 1 line
(11 rows)
Quoted 4 lines
Wow. 4 hits in 2.6.26, then 118 in 2.6.27.
A history of changes between 2.6.26 and 2.6.27.5 shows a mere six changes to the driver.
one thing to note is that for the .26 kernel, there was not very good data collection of this issue yet. (Although.. more than 4% I would say; Fedora had the patches to report the driver info backported for quite some time)
Arjan van de Ven <[email protected]> :
[...]
Quoted 3 lines
For me, sis900 and r8169 stand out; if you look at the data in the table above, both of these are an order of magnitude more frequent than the rest of the pack.
via-rhine + via_rhine = 438: it does not look too good either.
Is there an (ideally automated) way to retrieve more information ?
The r8169 driver handles three different chipsets and a plethora of phys. The "XID" line printed by the driver could hint at some specific PHY for instance.
Signature · 1 line
Ueimor
Quoted 9 lines · On Fri, 28 Nov 2008 20:50:18 +0100 Francois Romieu <[email protected]> wrote:
Arjan van de Ven <[email protected]> :
[...]Quoted 3 lines
For me, sis900 and r8169 stand out; if you look at the data in the table above, both of these are an order of magnitude more frequent than the rest of the pack.
via-rhine +
via_rhine= 438: it does not look too good either.Is there an (ideally automated) way to retrieve more information ?
this will need help from the driver and a bit of the core infrastructure.
the code that generates the warning is in net/sched/sch_generic.c:
char drivername[64];
WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit timed out\n", dev->name, netdev_drivername(dev, drivername, 64)); dev->tx_timeout(dev);
Quoted 3 lines
The r8169 driver handles three different chipsets and a plethora of phys. The "XID" line printed by the driver could hint at some specific PHY for instance.
anything you add to that WARN_ONCE will end up on kerneloops.org... it could be as simple storing some information in the net dev... or having a function pointer that can print some useful diagnostics information.
In addition, I'm trying to get a patch into .29 that prints, on x86, some basic DMI information in every WARN_ON class message; but this won't give you the details about the actual NIC, at most which motherboard is in use.
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 7 lines · Francois Romieu
Arjan van de Ven <[email protected]> :
[...]Quoted 3 lines
For me, sis900 and r8169 stand out; if you look at the data in the table above, both of these are an order of magnitude more frequent than the rest of the pack.
via-rhine +
via_rhine= 438: it does not look too good either.
Agreed. I was kinda hoping I'd get some clues for free when other drivers get fixed :-).
Quoted 5 lines
Is there an (ideally automated) way to retrieve more information ?
The r8169 driver handles three different chipsets and a plethora of phys. The "XID" line printed by the driver could hint at some specific PHY for instance.
For the Rhine, knowing the PCI rev would help identify problems tied to specific models.
Roger