Problem with TLB mcheck!

Hello linux community!
Please CC to me personaly, because I'am not subsribed yet.

I am relatively new Linux kernel (just work with some network adapter drivers) and my english is poor, so excuse me if something wrong. I work with Infineon ADM5120 chip (MIPS32 processor, little endian, using tlb), setting embedded Linux on it.
Almost everything is work now (I port some drivers). But there is big problem ( sg16lan is my network driver module):

[4294714.623000] sg16lan.c: shdsl_interrupt wake up dsl0
[4294714.627000] sg16lan.c: shdsl_interrupt wake up dsl0
[4294714.628000] Got mcheck at c0096604
[4294714.628000] Cpu 0
[4294714.628000] $ 0   : 00000000 10008400 00000000 00000000
[4294714.628000] $ 4   : c0098384 c00aa26c 00000001 00000001
[4294714.628000] $ 8   : 81261fe0 00008400 00000000 80318000
[4294714.628000] $12   : 0000000c 00 00000078 00000000
[4294714.628000] $16   : 000000d1 0000aea8 8038a220 8038a000
[4294714.628000] $20   : 80323420 80030000 00000002 8038a220
[4294714.628000] $24   : 00000000 802c0000
[4294714.628000] $28   : 81260000 81261e60 00003fde c0096604
[4294714.628000] Hi    : 00000280
[4294714.628000] Lo    : 00000230
[4294714.628000] epc   : c0096604 store_download+0x424/0x4fc [sg16lan]     Not tainted
[4294714.628000] ra    : c0096604 store_download+0x424/0x4fc [sg16lan]
[4294714.628000] Status: 10208403    KERNEL EXL IE
[4294714.628000] Cause : 00800060
[4294714.628000] PrId  : 0001800b
[4294714.628000]
[4294714.628000] Index:  4 pgmask=4kb va=c0094000 asid=b2
[4294714.628000]                        [pa=003d4000 c=3 d=1 v=1 g=1]
[4294714.628000]                        [pa=003d5000 c=3 d=1 v=1 g=1]
[4294714.628000]
[4294714.628000] Kernel panic - not syncing: Caught Machine Check exception - caused by multiple matching entries in the TLB.
[4294714.628000]

Panic occures when I work with big memory arrays (about 100Kb), when load firmware in the insmod-ed driver. BUT! if driver is staticaly compiled in kernel all is ok.
I have try loading with two variants:
1. request_firmware call (it allocates memory with valloc)
2. ioctl call (it allocate memory with kalloc).
The same thing (if driver insmoded - panic, if staticaly compiled -
all work correctly).

IMHO this is Memory Management subsystem problem, because ported driver is work fine on x86 processor and can not be reason of the problem. But I can be wrong!
Where can I look and whom ask the questions??

I woud be thankful for any help.
Thanks anyway, Artem

The "linux-mips" mailing list (as cc-ed in this response) is a better
place to ask such questions.

You haven't included some important information, such as the version
number of your Linux kernel and where you got your sources from. Without that bit all I can say is there is something wrong with handling of the TLB.

Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo*
registers in show_regs() if the cause is a machine check? I think it would be useful and these registers shouldn't have been corrupted since the triggering tlbw* instruction. A call to show_code() could be useful too, to determine which kind of TLB exception has been taken originally.

Maciej

On Wed, May 24, 2006 at 01:45:26PM +0100, Maciej W. Rozycki wrote:

j The "linux-mips" mailing list (as cc-ed in this response) is a better

Depends on when exactly a CPU will raise the machine check. On some cores the information in registers is totally useless if not even missloading. But generally a good idea, patch below.

Ralf

index 35cb08d..44a30e6 100644
@@ -819,8 +819,19 @@ asmlinkage void do_watch(struct pt_regs 
 
 asmlinkage void do_mcheck(struct pt_regs *regs)
 {
+	const int field = 2 * sizeof(unsigned long);
+
 	show_regs(regs);
+
+	printk("Hi    : %0*lx\n", field, regs->hi);
+	printk("Pagemask: %0*x\n", read_c0_pagemask());
+	printk("EntryHi : %0*lx\n", field, read_c0_entryhi());
+	printk("EntryLo0: %0*lx\n", field, read_c0_entrylo0());
+	printk("EntryLo1: %0*lx\n", field, read_c0_entrylo1());
+	printk("\n");
 	dump_tlb_all();
+	show_code((unsigned int *) regs->cp0_epc);
+
 	/*
 	 * Some chips may have other causes of machine check (e.g. SB1
 	 * graduation timer)

We have got PRId to filter out these. Though rev. 2 of the architecture
limits conditions when to raise the exception so it may eventually be a non-issue.

Except Index would be a bit more useful than HI. ;-)

Maciej

Doesn't really help, the exception is asynchronous by definition, so the CPU can be far away by the time it's struck be the lightning bolt. Machine check is just a _bad_ place to be.

Index may not matter at all in case of a TLBWR. But yes, will include index in the patch.

Ralf

It does help -- while it is asynchronous indeed, TLB writes are far rarer
than reads and happen in well defined places and a machine check will happen within limited time after such a write attempt, at the very worst. With the 4Kc the machine check looks synchronous.

Maciej

Hello Maciej, thanks for answer! when it is at first time it is wery important :)!

Wednesday, May 24, 2006, 7:45:26 PM, you wrote:

MWR> The "linux-mips" mailing list (as cc-ed in this response) is a better MWR> place to ask such questions.

MWR> You haven't included some important information, such as the version MWR> number of your Linux kernel and where you got your sources from. Without MWR> that bit all I can say is there is something wrong with handling of the MWR> TLB.

MWR> Ralf, BTW -- shouldn't we report the Index, EntryHi and possibly EntryLo* MWR> registers in show_regs() if the cause is a machine check? I think it MWR> would be useful and these registers shouldn't have been corrupted since MWR> the triggering tlbw* instruction. A call to show_code() could be useful MWR> too, to determine which kind of TLB exception has been taken originally.

MWR> Maciej

I know about linux-mips, but I have not get answer from that for two days! Maybe I write not complete message.
I will try Ralf Baechles diff and send the result a bit later. Information about kernel sources (how can I forgot this!!):
Version: linux-2.6.12-rc1-mipscvs-20050403 (this is tar-s full name). So I think this kernel is from linux-mips cvs repositary. It was downloaded from http://www.student.tue.nl/Q/t.f.a.wilms/adm5120/

Here is mcheck with respect to your patch:

[4294701.476000] Got mcheck at 8011c818
[4294701.476000] Hi    : 00000000
[4294701.476000] Pagemask: ffffffff
[4294701.476000] EntryHi : c008a013
[4294701.476000] EntryLo0: 0000f25f
[4294701.476000] EntryLo1: 0000ff9f
[4294701.476000]
[4294701.476000] Cpu 0
[4294701.476000] $ 0   : 00000000 10008400 00000000 802b17ac
[4294701.476000] $ 4   : 802b13ac 00000400 c008ac00 80353d6c
[4294701.476000] $ 8   : 10008400 1000001f 80393000 fffffff8
[4294701.476000] $12   : 802b0000 802b0000 00000001 ffffffff
[4294701.476000] $16   : 00000400 802b17ab 80353d6c 00000020
[4294701.476000] $20   : 80353e38 81144800 00000400 802b13ac
[4294701.476000] $24   : 00000006 0048ce30
[4294701.476000] $28   : 80352000 80353c70 7faf5d98 8011ce94
[4294701.476000] Hi    : 00000000
[4294701.476000] Lo    : 000c3500
[4294701.476000] epc   : 8011c818 vsnprintf+0x54/0x6bc     Not tainted
[4294701.476000] ra    : 8011ce94 vscnprintf+0x14/0x30
[4294701.476000] Status: 10208402    KERNEL EXL
[4294701.476000] Cause : 00800060
[4294701.476000] PrId  : 0001800b
[4294701.476000]
[4294701.476000] Index:  0 pgmask=4kb va=0050a000 asid=13
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]                        [pa=00379000 c=3 d=1 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  1 pgmask=4kb va=7faf4000 asid=13
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]                        [pa=0031b000 c=3 d=1 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  2 pgmask=4kb va=0048a000 asid=13
[4294701.476000]                        [pa=011bc000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011bd000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  5 pgmask=4kb va=00492000 asid=13
[4294701.476000]                        [pa=011c4000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011c5000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  6 pgmask=4kb va=004ee000 asid=13
[4294701.476000]                        [pa=0028a000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011f7000 c=3 d=1 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  7 pgmask=4kb va=0048c000 asid=13
[4294701.476000]                        [pa=011be000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011bf000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index:  8 pgmask=4kb va=004f0000 asid=13
[4294701.476000]                        [pa=00322000 c=3 d=1 v=1 g=0]
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]
[4294701.476000] Index:  9 pgmask=4kb va=00480000 asid=13
[4294701.476000]                        [pa=011b0000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011b1000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 10 pgmask=4kb va=00488000 asid=13
[4294701.476000]                        [pa=011b8000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=011b9000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 11 pgmask=4kb va=7faf6000 asid=13
[4294701.476000]                        [pa=00357000 c=3 d=1 v=1 g=0]
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]
[4294701.476000] Index: 12 pgmask=4kb va=00478000 asid=13
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]                        [pa=011a9000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 13 pgmask=4kb va=c008c000 asid=13
[4294701.476000]                        [pa=003ff000 c=3 d=1 v=1 g=1]
[4294701.476000]                        [pa=01200000 c=3 d=1 v=1 g=1]
[4294701.476000]
[4294701.476000] Index: 14 pgmask=4kb va=004aa000 asid=13
[4294701.476000]                        [pa=00286000 c=3 d=0 v=1 g=0]
[4294701.476000]                        [pa=00287000 c=3 d=0 v=1 g=0]
[4294701.476000]
[4294701.476000] Index: 15 pgmask=4kb va=00508000 asid=13
[4294701.476000]                        [pa=0031c000 c=3 d=1 v=1 g=0]
[4294701.476000]                        [pa=00000000 c=0 d=0 v=0 g=0]
[4294701.476000]
[4294701.476000]
[4294701.476000] Code: 0222102b  14400034  00000000 <80c60000> 14c00014  02e08021  0230102b  10400021  00001821
[4294701.476000] Caught Machine Check exception - caused by multiple matching entries in the TLB

Best regards,
 art                            mailto:[email protected]