Malta crashes on the latest 2.4 kernel

See the crash scene. Anybody knows the cause? It is strange to see the reserved exception.

Jun

FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize pcnet32.c:v1.27a 10.02.2002 [email protected]
pcnet32: PCnet/FAST III 79C973 at 0x1200, 00 d0 a0 00 01 e7 assigned IRQ 10.
eth0: registered as PCnet/FAST III 79C973
pcnet32: 1 cards_found.
SCSI subsystem driver Revision: 1.00
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 4096 bind 4096)
Sending BOOTP requests . OK
IP-Config: Got BOOTP answer from 10.0.0.75, my address is 10.0.18.6
IP-Config: Complete:

      device=eth0, addr=10.0.18.6, mask=255.255.0.0, gw=255.255.255.255,
     host=10.0.18.6, domain=, nis-domain=(none),
     bootserver=10.0.0.75, rootserver=10.0.0.75, rootpath=/opt/mvl-installs/mvl2

.1/hardhat/devkit/mips/fp_le/target
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
Looking up port of RPC 100003/2 on 10.0.0.75
Looking up port of RPC 100005/1 on 10.0.0.75
VFS: Mounted root (nfs filesystem).
Freeing prom memory: 956kb freed
Freeing unused kernel memory: 84k freed
Algorithmics/MIPS FPU Emulator v1.5
INIT: version 2.78 booting
$0 : 00000000 80000000 80002000 00000006 00000006 0040c000 0040c000 00000001 $8 : 80000000 00000034 8004bde8 8004bbf0 8004bde8 00000080 8004baf8 00000001 $16: 00800000 800071c0 1000fc01 8004c008 0000b000 8004c008 00800000 0040b000
$24: 00000000 00000080 8004a000 8004ba78 0000000b 8011db6c
Hi : ffffe4f4
Lo : 00000904
epc : 8010d528 Not tainted
Status: 1020fc02
Cause : 00800060

Kernel panic: Caught reserved exception - should not happen.

==========================================================ffffffff8010d490:

    92240080        lbu     $a0,128($s1)
ffffffff8010d494:       3c088000        lui     $t0,0x8000
ffffffff8010d498:       00a41025        or      $v0,$a1,$a0
ffffffff8010d49c:       40825000        mtc0    $v0,$10
ffffffff8010d4a0:       24a52000        addiu   $a1,$a1,8192
        ...
ffffffff8010d4bc:       42000008        tlbp
        ...
ffffffff8010d4d8:       40070000        mfc0    $a3,$0
ffffffff8010d4dc:       00000000        nop
ffffffff8010d4e0:       00e01021        move    $v0,$a3
ffffffff8010d4e4:       40801000        mtc0    $zero,$2
ffffffff8010d4e8:       00000000        nop
ffffffff8010d4ec:       40801800        mtc0    $zero,$3
ffffffff8010d4f0:       00000000        nop
ffffffff8010d4f4:       40885000        mtc0    $t0,$10
ffffffff8010d4f8:       04400013        bltz    $v0,ffffffff8010d548 <local_flus
h_tlb_range+0x150>
ffffffff8010d4fc:       00a6102b        sltu    $v0,$a1,$a2
        ...
ffffffff8010d518:       00071340        sll     $v0,$a3,0xd
ffffffff8010d51c:       3c018000        lui     $at,0x8000
ffffffff8010d520:       00221021        addu    $v0,$at,$v0
ffffffff8010d524:       40825000        mtc0    $v0,$10
ffffffff8010d528:       42000002        tlbwi
        ...
ffffffff8010d544:       00a6102b        sltu    $v0,$a1,$a2
ffffffff8010d548:       1440ffd4        bnez    $v0,ffffffff8010d49c <local_flus
h_tlb_range+0xa4>
ffffffff8010d54c:       00a41025        or      $v0,$a1,$a0
ffffffff8010d550:       40835000        mtc0    $v1,$10
ffffffff8010d554:       08043569        j       ffffffff8010d5a4 <local_flush_tl
b_range+0x1ac>
ffffffff8010d558:       00000000        nop
ffffffff8010d55c:       3c108025        lui     $s0,0x8025
ffffffff8010d560:       8e105910        lw      $s0,22800($s0)
ffffffff8010d564:       26100001        addiu   $s0,$s0,1
ffffffff8010d568:       320200ff        andi    $v0,$s0,0xff
ffffffff8010d56c:       14400005        bnez    $v0,ffffffff8010d584 <local_flus

h_tlb_range+0x18c>
ffffffff8010d570: 00000000 nop

The 2.4 kernel checked out around Jul 4 09:28 PDT works fine on Malta.

H.J.

Ralf,

Apparently this is a well-known problem. So *please* apply this patch or give
some constructive advice if the patch cannot be applied as it is.

Jun

diff -Nru arch/mips/mm/tlb-r4k.c.orig arch/mips/mm/tlb-r4k.c
@@ -125,6 +125,7 @@
 				BARRIER;
 				/* Make sure all entries differ. */
 				set_entryhi(KSEG0+idx*0x2000);
+				BARRIER;
 				tlb_write_indexed();
 				BARRIER;
 			}

Jun's patch certainly is correct for some MIPS32 CPUs; others may get away without this one. Previous experience with pipeline hazards on MIPS processors has shown that at times hazards may change even between minor revisions of a CPU; documentation isn't always trustworthy about such minor details of the pipeline.

Ralf

Excuse me, but I've seen this statement used by others in the past as an excuse for doing something silly or not doing something reasonable, and it generally hasn't washed. In what specific cases have the CP0 pipeline hazards changed between minor revisions of any production MIPS CPU? The *documentation* may have been corrected, but these hazards are fairly fundamental artifacts of the pipeline microarchitecture of a given processor.

The CP0 hazard between a write of EntryHi
and a subsequent TLBWI instruction is flagged
in the MIPS32 spec and noted as being "typically" 2 cycles. I'm not going to spend the time going through my full set of users manuals, but a representative sampling shows this hazard as being specified for every R4xxx and R5xxx CPU I checked. The fact that a given CPU *may* get away with it is no excuse for not protecting common code.

I note that Ralf has, in fact, applied the fix to the OSS CVS repository. I also note that "BARRIER" is still defined to be a string of 6 nops. I would argue (again) that those really, really ought to be ssnops, and that if they *were* ssnops, one could probably have fewer of them.

Regards,

Kevin K.

Sorry for being ignorant, but what's the difference between nop and ssnop?

I see that SSNOP is defined to be `sll zero,zero,1' in <asm/asm.h>, but that doesn't give me any clue.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

SSNOPs are "super-scalar NOPs", which were first invented (but not documented at the time) for the R8000, which was the first superscalar MIPS implementation. They wanted to be able to absorb the standard "overhead" NOPS associated with unfilled branch delay slots, etc, in the dual-issue mechanism, but still have some means to handle
CP0 hazards such that it could be assured to force a 1 cycle stall per instruction. While it wasn't officially a part of the architecture until the late 1990's, the convention was carried forward by the R5xxx
and R1xxxx families. There have been rumours of superscalar MIPS processors that do not enforce single-issue on SSNOPs, but I don't know of any offhand, and the MIPS32/MIPS64 specs formalize
the definition.

Regards,

Kevin K.

Ssnop is a superscalar nop. It's instruction encoding is the same as of sll, zero, zero, 1. Unlike a normal nop a ssnop is guaranteed to single issue even on superscalar implementations.

Ralf

Ancient TLB exception handler code was assuming out of order execution of the instruction stream in cp0 based on the documentation in appendix H of the R4400 manual, version 2. I wrote that code for a R4400 version 5.0 and it was running fine on R4000 3.0 but somebody found it to break on R4000 version 2.2. At least that are the details as I remember them. I don't blame MIPS (well, probably SGI at that time ...) for not documenting these details perfectly right for each and every R4[04]00 implementation. The code broken was written extremly aggressivly and eventually had to be changed anyway for the sake of other processors.

No argument about this one. We definately were lucky.

I've applied it because I think the whole update_mmu_cache implementation is ready for a reimplementation anyway. On the performance this isn't going to have measurable impact anyway as update_mmu_cache is only being called once per page fault.

BARRIER is defined as 6 nops since it was written somewhen during the summer '96. By that time Linux didn't yet support any processor that was featuring ssnop, so 6 nops certainly were too paranoid. These days you're certainly right, ssnops are the way to go, especially because they don't have any negative impact on pre-ssnop implementations.

Ralf

I actually discovered the hazard problem on a 4Kc, based on some older RTL, but the hazard some how disappear on a newer revision.
So you are absolutely right.

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com

Sounds like a problem I so a couple of months ago, I thought the fix was already in the CVS.

Here is my previous mail:

There seems to be a hazard problem in the local_flush_tlb_range function in tlb-r4k.c, which the patch below will fix.
It could hit anyone, but it probably only a problem on CPUs, which doesn't allow matching entries in the TLB.

/Carsten

Index: arch/mips/mm/tlb-r4k.c
===================================================================
RCS file: /cvs/linux/arch/mips/mm/tlb-r4k.c,v
retrieving revision 1.6.2.3

diff -u -r1.6.2.3 tlb-r4k.c
@@ -119,12 +119,11 @@
                                idx = get_index();
                                set_entrylo0(0);
                                set_entrylo1(0);
-                               set_entryhi(KSEG0);
                                if (idx < 0)
                                        continue;
-                               BARRIER;
                                /* Make sure all entries differ. */
                                set_entryhi(KSEG0+idx*0x2000);
+                               BARRIER;
                                tlb_write_indexed();
                                BARRIER;
                        }

--

_    _ ____  ___   Carsten Langgaard   Mailto:[email protected]
|\  /|||___)(___   MIPS Denmark        Direct: +45 4486 5527
| \/ |||    ____)  Lautrupvang 4B      Switch: +45 4486 5555
  TECHNOLOGIES     2750 Ballerup       Fax...: +45 4486 5556
                   Denmark             http://www.mips.com