Toshiba JMR 3927 working setup?

I have been working on a JMR 3927 based system for a number of years. For all of that time, we have been running:
binutils 2.11.90.0.1
gcc 2.95.3
glibc 2.2.1
linux 2.4.12

We want to update to a 2.6 kernel, recent build tools, and saner system libraries. Although, it seems that the JMR 3927 is still technically supported, I have not found any info on whether anybody is still running Linux on it and what combination of software they are using. Any idea? Is there a combination of software versions that are known to work on this hardware?

I have used crosstool 0.43 to build:
binutils 2.15
gcc 3.4.5
glibc 2.3.6

I cannot get these kernels to build:
linux-2.6.13
linux-2.6.15
linux-2.6.16.57
linux-2.6.17.14
linux-2.6.9

My colleague and I have built these:
linux-2.6.21.7
linux-2.6.23.9
linux-2.6.23.12

However, they all yield a TLBL exception similar to the following:

Exception! EPC=80056eb4 CAUSE=30000008(TLBL)
80056eb4 8ce4000c lw a0,12(a3) # 0xc

Each build has different exception values. The values are the same each attempt with the same build.

Is this a problem in the kernel code or the build tools? Any ideas on how to make it run?

Using the recently built tools, I am currently trying to build the 2.4.12 kernel that is known to work, which is proving difficult. If I can get it to build, I am hoping to see whether the tools are able to build a functioning kernel.

Thanks

http://www.linux-mips.org/wiki/Toolchains recommends binutils 2.16.1 and gcc 3.4.4, but I believe your choice is also ok for 32-bit systems.

Hard to tell from so little information, it would help to see the whole boot log.

Thiemo

I will try that combination also.

I wish that there were more of a boot log of which to speak. The following is a complete boot log:

Serial Number = WAC6200032
MAC Address 1= 00:0D:5D:00:eb:6f
Kernel Image Length = 0x695000, CRC = 0xd2ee6a23

Loading Linux ......
Downloading from ethernet, ^C to abort and restart pmonitor sendRRQ vmlinux.bin
load linux length 0x34408a
Checking CRC on downloaded RAM image
/
CRC Check passed
Image Started At Address 0x80020000.
Image Length = 3424394 (0x34408a).
Exception! EPC=80056eb4 CAUSE=30000008(TLBL)
80056eb4 8ce4000c lw a0,12(a3) # 0xc

All messages before the exception are from PMON. I am booting via TFTP. I also tried writing the kernel to flash and booting from that, but that fails identically.
I checked the other serial port on the device, but it is not showing any kernel messages either. The normal boot console is ttyS1. I tried setting the kernel parameters and the default parameters (from the kernel build config) to "console=ttyS1", but that made no difference. Regardless, I would think that the kernel knows where to send messages because I am seeing the exception dump on ttyS1. Further, the exception gets printed immediately after "Image Length = 3424394 (0x34408a)." The exception happens so soon that I doubt that the kernel does very much beforehand.

Thanks

The exception comes from PMON too - Did you look up the EPC in your System.map file ?

The kernel seems to be initializing and steps upon an virtual address and triggers an TLB exception which as the kernel is not yet ready to handle it itself, thus PMONs exception handler gets triggered.

My first guess is that you are catching a NULL Pointer exception somewhere.

FLo

Florian Lohoff                  [email protected]             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin

Linux 2.6.23.9:
80056eb4 T kernel_execve

My coworker built Linux 2.6.23.12 and got:
EPC=80056ee0 which does not correspond to anything in his System.map.

With a different set of tools, I also built 2.6.21.7 and got:
EPC=80056eb8 which also does not correspond to anything in the corresponding System.map.

If this does not ring any bells, do you have any suggestions for how I could debug this issue?

Thanks

Hm, your start address is 0x80020000, but the load address of the jmr3927 (in the www.linux-mips.org tree) is 0x80050000. So maybe the address is wrong, comparing with the old 2.4 kernel should tell.

True, it fails early, before it can take over the exception handlers.

Thiemo

I compared that against the working 2.4.12 kernel, which does indeed have 80020000 in arch/mips/Makefile. I changed load-$(CONFIG...TX3927) to use 80020000 instead of 80050000, but it still fails the same way. There was also something that I had changed in our 2.4.12 that I had forgotten, but my coworker reminded me that, in jmr3927.h, I changed JMR3927_ROMCE0 from 0x1fc000000 to 0x1f000000 to have the proper address to access some raw memory for internal purposes. I do not recall whether that address is specific to our hardware or a typo, but I do not think that it impacts the normal kernel anyway.

Our 2.4.12 kernel uses -mcpu=r3000 -mips1 to build the kernel. I tried switching the arch to r3000 from r3900 in 2.6.23.9, but that did not help. Perhaps -mips1 or an equivalent could help? I will try on Monday.

I am grasping at straws here.

Thanks to Thiemo and Florian for trying to help.

Have a good weekend.

I think both mcpu=r3000 and r3900 should work. But I believe at least one kernel patch is required for all MIPS I platforms including JMR3927.

http://www.linux-mips.org/archives/linux-mips/2007-02/msg00320.html

---
Atsushi Nemoto

Well, is mmiowb() needed at all on JMR3927 - or R3000 in general? Mmiowb() is meant to deal with weak ordering which only matters to a few SMP configurations.

Ralf

Well, serial_txx9 driver (mis)use mmiowb() to flush write buffer.

But I was wrong. TX39H2 core _has_ SYNC instruction. So the patch I mentioned is not required for JMR3927 platform anyway. Sorry for confusion.

---
Atsushi Nemoto

Are you loading an ELF binary or a raw binary image? If your loader does not handle ELF headers, you should do some trick to start running your kernel at correct address.

If you were using 2.6.23, CONFIG_BOOT_RAW might help you.

But it seems CONFIG_BOOT_RAW is broken on current git again. It will be an another story... :-<

---
Atsushi Nemoto

I tried setting CONFIG_BOOT_RAW to y in .config, but that did not help. Are there any other tricks for loading an ELF image?
Is there any way to verify that ELF is the issue?
I will do some research on ELF and loaders.

It took me many trials and some research to build the following tools for MIPS:
binutils-2.18
gcc-4.2.2
glibc-2.7

I include "ARCH=mips CROSS_COMPILE=mipsel-linux-gnu-" with each call of make when working on the kernel. I did mrproper after unarchiving and I used vmlinux followed by vmlinux.bin as build targets.

I built linux-2.6.23.9 with the above, but the results are still the same and the EPC is not in System.map. My problem does not appear to be a matter of tool versions.

Exception! EPC=8005625c CAUSE=30000008(TLBL)
8005625c 8e020098 lw v0,152(s0) # 0x98

I presume that 8e020098 is the full instruction, so I have tried searching for it in vmlinux.bin. The first occurrence is around 0x869b, which is more than 32k into the file. There is also nearly 1k worth of zero padding at the start of vmlinux.bin.

Does anybody have any other suggestions for getting down to the bottom of this problem?

Thank you

Are you searching the exact EPC value in System.map?
Usually you should find a function symbol which contains the EPC value in it.

Or you can do "mipsel-linux-objdump -d vmlinux" and search the EPC value.

The 1k zero padding is normal. Please refer arch/mips/kernel/head.S.

---
Atsushi Nemoto

The current error is:
Exception! EPC=80026290 CAUSE=00000020(Sys)
80026290 0000000c syscall

80026290 is not in System.map, however, the objdump is much more informative and does contain that value. That particular syscall is in:

8002628c <kernel_execve>:

8002628c:       24020fab        li      v0,4011
80026290:       0000000c        syscall
80026294:       00401821        move    v1,v0
80026298:       14e00003        bnez    a3,800262a8 <kernel_execve+0x1c>
8002629c:       00000000        nop
800262a0:       03e00008        jr      ra
800262a4:       00601021        move    v0,v1
800262a8:       03e00008        jr      ra
800262ac:       00031023        negu    v0,v1

Does that provide any clues?
I have to go now. I will take another look when I return, but I have looked at kernel_execve before.

Thanks

The kernel failed to set up the general exception handler correctly. It should have done that before attempting to start the first kernel thread.

Thiemo

From where in the kernel image should execution begin?

Presuming that the output of "objdump -d" reflects the disassembled binary from the beginning in order, it looks like my 2.6 kernel is running straight into run_init_process as the first real code executed. From what I have seen in the kernel code, run_init_process should be jumped to far later in the boot process. If what I am thinking is correct, then it also explains why the failure happens in kernel_execve.

I have also included the start of my working kernel, which has _ftext with non-zero data as its first entry. Is the _ftext the ELF header or some other info for the boot loader?

Thanks

linux-2.6.23.9/vmlinux: file format elf32-tradlittlemips

Disassembly of section .text:

80020000 <run_init_process-0x400>:

       ...

80020400 <run_init_process>:

80020400:       3c028033        lui     v0,0x8033
80020404:       3c068033        lui     a2,0x8033
80020408:       244594dc        addiu   a1,v0,-27428
8002040c:       24c69450        addiu   a2,a2,-27568
80020410:       080098a3        j       8002628c <kernel_execve> // Should this be happening already?
80020414:       ac4494dc        sw      a0,-27428(v0)

80020418 <init_post>:

run_init_process is from linux-2.6.23.9/init/main.c

Our working kernel starts differently:

linux-2.4.12/vmlinux: file format elf32-tradlittlemips

Disassembly of section .text:

80020000 <_ftext>:

80020000:       26 80 01 3c 14 42 3d ac 02 80 08 3c 80 04 08 25     &..<.B=....<...%
80020010:       08 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00     ................
       ...

80020298 <except_vec2_generic>:

80020298:       401a8000        mfc0    k0,$16
8002029c:       241bfff8        li      k1,-8
800202a0:       035bd024        and     k0,k0,k1
800202a4:       375a0002        ori     k0,k0,0x2
800202a8:       409a8000        mtc0    k0,$16
       ...
800202b8:       0800b248        j       8002c920 <cache_parity_error>
800202bc:       00000000        nop

800202c0 <except_vec4>:

Gregor Waltz wrote:
[snip]

Normally from kernel_entry, but your boot loader appears to start from the begin of the code segment it loaded.

This is likely code which jumps to kernel_entry (but the disassembler doesn't know since it sees no function symbol, so it defaults to data).

Enabling CONFIG_BOOT_RAW, as Atsushi already suggested, would have added a jump to kernel_entry in this place.

Thiemo

Yes, I tried that, but it made no difference at the time and Atsushi also said that it was broken in git as of 01/05/2008 (I have 2.6.23.9). The kernel config operations (oldconfig, menuconfig, et al) obliterate CONFIG_BOOT_RAW from the .config.

I double checked today and found that even the vmlinux make target removes that option from .config. Is there another way to set that option?

I saw the option used only in arch/mips/kernel/head.S, so I commented out the __INIT. Now, I see kernel_entry at the start of the kernel and the kernel does not cause an exception, however, it reboots instead saying "Rebooting..."

Thanks

The CONFIG_BOOT_RAW is not user-selectable. You must add "select BOOT_RAW" to TOSHIBA_JMR3927 block in arch/mips/Kconfig.

Hmm. The puts() in arch/mips/jmr3927/common/puts.c looks usable even on kernel entry. You can verify if it can really be used on start_kernel(), then start tracking down the problem.

---
Atsushi Nemoto

puts() has helped, but I wish that I had something to dump the stack. Is kgdb easy to set up?

In main.c, init_IRQ() eventually uses kmalloc:
arch_init_irq() -> txx9_irq_init() -> ioremap() -> __get_vm_area_node() -> kmalloc_node()...
malloc_sizes is not yet initialized, though, which means that cs_cachep is zero for all entries. My system reboots when it reaches cpu_cache_get() in mm/slab.c where cachep is zero.

It seems to me that kmem_cache_init() ought to be run before any kmallocs. kmem_cache_init() seems to require mem_init(). After I moved mem_init and kmem_cache_init before init_IRQ(), it gets down to pidhash_init() before rebooting, which I am looking into now.

What ought to be done to fix the init_IRQ()/kmalloc problem?

Thanks

Oops, that was my mistake. The txx9_irq_init() assumes its baseaddr can be remapped without TLB. This is true but plat_ioremap for jmr3927 was wrong.

Could you try this patch? (can be used for 2.6.23 and current git)

Subject: [MIPS] Fix plat_ioremap for JMR3927

TX39XX's "reserved" segment in CKSEG3 area is 0xff000000-0xfffeffff.

---

index aa131ad..ac3be35 100644
@@ -25,7 +25,7 @@ static inline void __iomem *plat_ioremap(phys_t offset, unsigned long size,
 {
 #define TXX9_DIRECTMAP_BASE	0xff000000ul
 	if (offset >= TXX9_DIRECTMAP_BASE &&
-	    offset < TXX9_DIRECTMAP_BASE + 0xf0000)
+	    offset < TXX9_DIRECTMAP_BASE + 0xff000)
 		return (void __iomem *)offset;
 	return NULL;
 }

It's years since I last had a report of the JMR3927. Since that time the code is maintained without the possibility of testing.

These themselves are rather old.

The build since we tried to fix the JMR3927 as good as possible without having access to hardware. Which of course means almost certainly the one or other buglet is left in the code ...

... quod erat demonstrandum.

Well possible a bit of both ...

You may want to switch to a recent binutils like 2.18 and gcc 4.2.2.

There was a change related to linker scripts and I think that change requires a recent binutils version.

2.4.12 had various alergies against modern toolchains. So you may want to retain a copy of your old toolchain for use with 2.4.12. later 2.4 versions have been fixed to build with recent toolchains.

Ralf

Ralf, are you or anybody else using that combination for mips?

With which versions of glibc and glibc-linuxthreads?

I have been having a joyous time trying to build more recent tools.
I keep getting errors like the following when building glibc of any version (tried from 2.3.6 through 2.7):
sysdeps/unix/sysv/linux/waitid.c:51: error: memory input 6 is not directly addressable

I found a patch for that (http://sourceware.org/ml/crossgcc/2005-04/msg00208.html), but, after passing that file, it fails similarly on pread.c. It seems to be an issue with INLINE_SYSCALL, rather than the call sites, that appears to be related to later versions of gcc.

Later versions of glibc (2.7, 2.6...) fail at configure time with the error: "The mipsel is not supported."

So, what are the lastest versions of the build tools that people are using on mips?

Thanks

This suggests you are missing the "ports" add-on.

Debian uses binutils 2.18+, gcc 4.2 and glibc 2.7 in the "unstable" development branch. This environment works fine for me.

Thiemo

In message: <[email protected]>
Gregor Waltz <[email protected]> writes:
: Ralf Baechle wrote:
: > You may want to switch to a recent binutils like 2.18 and gcc 4.2.2. :
: Ralf, are you or anybody else using that combination for mips?

Speaking of toolchains, are there known problems on mips with gcc 4.2.1?

Warner