Got bus error when loading kernel module on SB1250 Rev B2 board with 64 bit kernel

Hi experts,

I'm running 2.6.34-rc7 mainline kernel on SB1250 (Rev B2) board. And, I use the default sb1250 kernel config (sb1250-swarm_defconfig). So, 64 bit kernel is used. During kernel loading module got bus error, see below log:

root@bcm1250a-1:/root> insmod firmware_class.ko
DBE physical address: 0080e87020
Data bus error, epc == ffffffff801053e0, ra == ffffffff801725d0 Oops[#1]:
Cpu 0

$ 0 : 0000000000000000 0000000014001fe0 ffffffff8051d308 0000000000000000
$ 4 : a800000080e97000 0000000000000000 0000000000000000 0000000000000002
$ 8 : ffffffff80515868 0000000000000000 0000000000000001 a80000000fb0c030
$12 : 0000000000000010 ffffffff804ab728 a80000000f81ba00 0000000000000001
$16 : 0000000000000001 ffffffffc0001950 0000000000000000 c000000000005000
$20 : 0000000000000000 a80000000faabd90 c000000000008d18 c000000000007e98
$24 : 0000000000000000 0000000010010598
$28 : a80000000faa8000 a80000000faabd50 ffffffffc0001968 ffffffff801725d0
Hi : ffffffffffffffff
Lo : fffffffffffffff0
epc : ffffffff801053e0 memcpy+0x0/0x4
Not tainted
ra : ffffffff801725d0 load_module+0x1240/0x1c30
Status: 14001fe3 KX SX UX KERNEL EXL IE
Cause : 0080801c
PrId : 01040102 (SiByte SB1)
Modules linked in:
Process insmod (pid: 1799, threadinfo=a80000000faa8000,
task=a80000000f898800, tls=00000000100a7480)
Stack : 0000000000000000 ffffffff80484190 ffffffffc0001738 ffffffff00000003
0000000000000000 0000000000000000 0000000000000000 0000000200000000
0000017f00000180 c000000000008d58 ffffffff803ec728 0000000000001cf8
ffffffff80515868 a80000000f8d14a0 0000000000001b60 0000000000000000
0000000000000029 000000000000002d 000000000000003b 000000000000003a
c00000000000c008 a80000000f92aa20 c000000000007bba c000000000008a98
ffffffffc0004d88 ffffffff803ec718 ffffffff803ec718 ffffffff801df1f8
00000000000076ce 00000000100a0ff8 00000000100a1008 ffffffff804b0000
000000007f9d0f11 0000000000000003 0000000000000000 00000000100d0bc0
0000000000000000 ffffffff80173048 00000000100a0ff8 00000000000076ce
...
Call Trace:
[<ffffffff801053e0>] memcpy+0x0/0x4
[<ffffffff801725d0>] load_module+0x1240/0x1c30
[<ffffffff80173048>] SyS_init_module+0x88/0x230
[<ffffffff80102ec4>] handle_sysn32+0x44/0x9c

Code: 03e00008 00000000 00000000 <0080102d> cca00000 cc810000 2cca0008

30890007 cca00020
Disabling lock debugging due to kernel taint
Cache error exception on CPU 0:
Cache error exception on CPU 0:
c0_errorepc == 800000f0
c0_errctl == 40000000 dcache
c0_cerr_d == 88801000 fill/wb, multi-err data-DBE
c0_cerr_dpa == 0080e87000
Dcache index 0x1000 [Bank 2 Set 0x00] LRU > 1 2 0 3 > MRU
0 [PA 00004ad000] [state COH-SHD (0f)] raw tags: 1E724000-00000000004AC400 00-0000000000000000 9E-FFFFFFFFFFFFFED4 9E-FFFFFFFFFFFFFED4 4E-0000000000020002
1 [PA 000050f000] [state COH-SHD (0f)] raw tags: 1E724000-000000000050E400 EC-3E343C0A30207570 A7-203A202020302024 00-3030303030303030 00-3030303030303030
2 [PA 000f93d000] [state COH-SHD (0f)] raw tags: 1E724000-000000000F93C000 00-0000000000000000 00-FFFFFFFFFFFFFFFF 00-FFFFFFFFFFFFFFFF 00-FFFFFFFFFFFFFFFF
3 [PA 000f8db000] [state COH-E-C (16)] raw tags: 2C724000-000000000F8DA000 84-FFFFFFFF80504000 84-FFFFFFFF80504000 84-FFFFFFFF80504000 84-FFFFFFFF80504000
...didn't see indicated dcache problem
Bus watcher error counters: 00000000 00030000

Last recorded signature:
Request 00 from 0, answered by 7 with Dcode 4

Kernel panic - not syncing: unhandled cache error

Rebooting in 5 seconds..

It seems bus error occurs when copying module since the memcpy is called by percpu_modcopy. I recalled that SB1250 prior C0 version has a TLB M3 workaround which caused kernel module loading fail, however, it seems Ralf committed a patch to solve this issue. See git log:

commit 3d45285dd1ff4d4a1361b95e2d6508579a4402b5
Author: Ralf Baechle <[email protected]>
Date: Tue Mar 23 17:56:38 2010 +0100

MIPS: Sibyte: Fix M3 TLB exception handler workaround.

The M3 workaround needs to cmpare the region and VPN2 fields only.

It seems this patch can't solve the issue, or this is another different bug?

Thanks,
Yang

Whops. Fixes which were supposed to handle exactly this problem went upstream for 2.6.34-rc3 and were tested successfully by others on their systems.

I wonder if in arch/mips/sibyte/sb1250/setup.c you can instrument the function sb1250_m3_workaround_needed() and print the values of soc_type, soc_pass and the retun value of that function. Then let's take it from there.

Ralf

See below log:

Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)

And, soc_typs is 0x0 and soc_pass is 0x11, sb1250_m3_workaround_needed should return 1. So, tlb refill handler should go the m3 workaround code path.

Thanks,
Yang

It seems CPU_PREFETCH caused this issue. See commit:

commit 6b4caed2ebff4ee232f227d62eb3180d0b558a31

Author: Ralf Baechle <[email protected]>
Date:   Wed Jan 28 17:48:40 2009 +0000

MIPS: IP27: Switch from DMA_IP27 to DMA_COHERENT

commit 0d356eaa6316cbb3e89b4607de20b2f2d0ceda25 from linux-mips

The special IP27 DMA code selected by DMA_IP27 has been removed a while ago turning DMA_IP27 into almost a nop. Also fixup the broken logic of its last users memcpy.S and memcpy-inatomic.s.

If undef CPU_PREFETCH for SB1250, module can be loaded correctly.

Thanks,
Yang

This patch did below fix:

-#if !defined(CONFIG_DMA_COHERENT) || !defined(CONFIG_DMA_IP27) +#ifdef CONFIG_DMA_NONCOHERENT
#undef CONFIG_CPU_HAS_PREFETCH

Before the fix, CONFIG_DMA_IP27 is undefined for all of boards except IP27, so CONFIG_CPU_HAS_PREFETCH is undefined always.

Thanks,
Yang

Is this a commit ID of a Wind River tree? The commit ID for the kernel.org kernel tree is 634286f127bef8799cd04799d3e1d5471e8fd91c, for the lmo tree e3bf818d95cb372bfe78696957586d9afff0b405.

[...]

That's an old issue striking here then. Memcpy will prefetch beyond the end of the source or destination area, if the use of prefetch is enabled.

On non-coherent systems this is a problem because these prefetches might bring back data that was just flushed from the cache back into the cache resulting in corrupted DMA transfers.

Some systems that do very tight checking of address but do not differenciate between actual loads / stores and prefetch accesses may also also throw address error exceptions when prefetching from non-decoded address ranges. That would typically be just beyond the end of a RAM range. For those systems the solution is to either disable prefetching or to not use the last page of every physically contiguous area of memory.

I'm still looking at why we this wasn't triggered earlier. Something else must have changed to suddenly trigger this issue - and your system seems to be the only system affected.

Ralf

Yes, that commit is cherry-picked from mainline commit 634286f127bef8799cd04799d3e1d5471e8fd91c. If I revert this commit, module loading works well.

As I said above, before 634286f127bef8799cd04799d3e1d5471e8fd91c checking in, actually prefetch should be *undefined* for all of boards except IP27, so maybe no one noticed this issue.

Seems yes, just Sibyte is affected :(

Thanks,
Yang