Ingenic JZ4730 - illegal instruction

Hello!
I am rather playing than really working on a Ingenic JZ4730 based device. The JZ4730 is a MIPS32 SOC included in many types of devices, like media players and thelike but also in small power efficient subnotebooks (this is the device I am trying to support based on the Ingebic Linux kernel patch).

The current kernel patch from Ingenic

http://www.ingenic.cn/eng/productServ/App/JZ4730/pfCustomPage.aspx
or ftp://ftp.ingenic.cn/3sw/01linux/02kernel/linux-2.6.24/linux-2.6.24.3-jz-20090218.patch.gz

for the patch (I used an even older patch to start my board support but they basically only added newer CPU types in later patches).

The support for my board is almost in place but I see from time to time failing applications with "illegal instruction" faults. Most shell applications work pretty fine, especially more complex GUI applications seem to fail, like a webbrowser or such.
I also tested this with different GCC and glibc version which makes me pretty sure that I am seeing a kernel problem here rather than a userspace problem.

I am pretty clueless how to debug this. Apropos debig as another hint:
Some application work if I start them in GDB but fail outside.

Any hint how to start debugging this would be greatly appreciated! And a fix would be like a dream ;)

Many thanks!

Cheers
nils faerber

Signature · 4 lines

kernel concepts GbR        Tel: +49-271-771091-12
Sieghuetter Hauptweg 48    Fax: +49-271-771091-19
D-57072 Siegen             Mob: +49-176-21024535
http://www.kernelconcepts.de

Well, the Xburst-arch seems to be pretty fucked up beyond all repair.

Hello!
I am rather playing than really working on a Ingenic JZ4730 based device. The JZ4730 is a MIPS32 SOC included in many types of devices, like media players and thelike but also in small power efficient subnotebooks (this is the device I am trying to support based on the Ingebic Linux kernel patch).

The current kernel patch from Ingenic

http://www.ingenic.cn/eng/productServ/App/JZ4730/pfCustomPage.aspx
or ftp://ftp.ingenic.cn/3sw/01linux/02kernel/linux-2.6.24/linux-2.6.24.3-jz-20090218.patch.gz

for the patch (I used an even older patch to start my board support but they basically only added newer CPU types in later patches).

The support for my board is almost in place but I see from time to time failing applications with "illegal instruction" faults. Most shell applications work pretty fine, especially more complex GUI applications seem to fail, like a webbrowser or such.
I also tested this with different GCC and glibc version which makes me pretty sure that I am seeing a kernel problem here rather than a userspace problem.

I am pretty clueless how to debug this. Apropos debig as another hint:
Some application work if I start them in GDB but fail outside.

Any hint how to start debugging this would be greatly appreciated! And a fix would be like a dream ;)

Many thanks!

Cheers
nils faerber

The only thing that you've mentioned below that really makes me think that you're looking at a kernel bug is the comment about things not failing under GDB. But if *any* of the programs that are failing fail under gdb, I'd want to know just what instruction is at the place where

they're taking a SIGILL. If gdb heisenbergs things too much, then the basic brute force thing to do would be to instrument the kernel itself to report on what happened, and what it sees at the "bad instruction" address, using printk. If the memory value actually looks like a legit instruction, it would confirm the hypothesis that you've got an icache maintenance problem. I note that the Ingenic patch has a "flushcaches" routine that has hardwired assumptions about the cache organization. Could those be incorrect on the chip you're using?

Regards, and happy hunting,

Kevin K.

Hi Kevin!

Quoted 4 lines · Kevin D. Kissell

The only thing that you've mentioned below that really makes me think that you're looking at a kernel bug is the comment about things not failing under GDB. But if *any* of the programs that are failing fail under gdb, I'd want to know just what instruction is at the place where

Quoted 9 lines

they're taking a SIGILL. If gdb heisenbergs things too much, then the basic brute force thing to do would be to instrument the kernel itself to report on what happened, and what it sees at the "bad instruction" address, using printk. If the memory value actually looks like a legit instruction, it would confirm the hypothesis that you've got an icache maintenance problem. I note that the Ingenic patch has a "flushcaches" routine that has hardwired assumptions about the cache organization. Could those be incorrect on the chip you're using?

Thanks for having a thought about the issue!

By now I pitily have to admit that my GDB assumption was not all that correct :( After *a*lot* more tries I found an application that actually also fails inside GDB. But with some more tries I can now confirm that applications fail at random points - it is not a single instruction that causes the fault but rather random points.
So I think your memory/cache issue theory sounds pretty interesting... I just had a look at the JZ4730 code (in arch/mips/jz4730/) and the only
mention of a cache flush is in pm.c which will only be executed in case
of going to sleep (i.e. CPU deep sleep aka s2ram). arch/mips/mm/c-r4k.c also contains a JZ_RISC section for setting up cache options and arch/mips/mm/tlbex.c a TLB case special for the JZ.

Those look promising!
I could very well think of cases where a wrong cache flush could cause such or similar problems.

Quoted 2 lines

Regards, and happy hunting,

Happy? When I found it maybe. The annoying thing about this is that Ingenic is not very helpful. I emailed them several times already asking for the full datasheet of the CPU with no replay at all yet. The datasheet they hae on their webpage is just the brief with about 60 pages and not very helpful when you ar elooking for details like cache handling etc.

So I will have to resort to experiments - trial an error.

Thank you very much for your thoughts and idea!

Quoted 2 lines

Kevin K.

Cheers
nils faerber

I don't have time to go chasing this stuff any further on your behalf,

but it *does* smell to me like an icache management problem. Remember, MIPS processors almost universally have split I/D caches and no coherence support between them, so if you either (a) forget to do an explicit D-cache write-back operation after copying to a page mapped write-back that's going to be used as instructions/text, or (b) forget to do an explicit I-cache invalidate when you re-use a page for instructions that has been previously used for a different instruction page, you will have problems, even without going into DMA I/O coherence issues. If your problem were (b), though, you'd be seeing bad answers, segmentation violations, bus errors, etc., at least as often as you'd be seeing illegal instruction exceptions. So my money would be on (a).

The need for cache management is so fundamental to Linux for MIPS that all the necessary general hooks have been there for years. If I were you, I'd focus on the definitions of the primitives that you spotted in c-r4k.c. Does the stuff in the JZ_RISC section correspond to the

assembly language flush sequence done in the Ingenic patch to head.S? Are you sure that the JZ_RISC section is in fact the version of those functions that's being built into your kernel?

Regards,
Kevin K.

Quoted 3 lines · Kevin D. Kissell

Are you sure that the JZ_RISC section is in fact the version of those functions that's being built into your kernel?

Well, there is CONFIG_JZRISC=y in the kernel .config and a switch(current_cpu_type) { case CPU_JZRISC: ...} so I would assume it is being used. But I will verify that the CONFIG_JZRISC=y is correctly translated into a current_cpu_type.

Oh, one last question, in order to rule out the cache as bug-spot would the kernel option "run uncached" "solve" the issue (and be darn slow)?

Quoted 7 lines · Nils Faerber

Quoted 2 lines · Kevin D. Kissell

Are you sure that the JZ_RISC section is in fact the version of those functions that's being built into your kernel?

Well, there is CONFIG_JZRISC=y in the kernel .config and a switch(current_cpu_type) { case CPU_JZRISC: ...} so I would assume it is being used. But I will verify that the CONFIG_JZRISC=y is correctly translated into a current_cpu_type.

Your assumption is reasonable. But given that things aren't working, yes, it's good to verify.

Quoted 2 lines

Oh, one last question, in order to rule out the cache as bug-spot would the kernel option "run uncached" "solve" the issue (and be darn slow)?

It would certainly solve the issue, and would *probably* result in a system that would be fully functional but slow. Very high end and very

low end systems can be rendered unusable by forcing uncached operation, but it's certainly worth a try. Also, if your cache control logic

supports both write-back and write-through operation, if you set the default cache "attribute" for kernel and page tables (which is essentially what you're doing under-the-hood when you configure for uncached operation) to write-through, that should cure the problems with copying text pages, but *not* those with re-using them, with less performance impact. I'd be a little surprised if the Ingenic part offered both modes, though.

Regards,
Kevin K.

Hello,

Quoted 1 line · Markus Gothe

Well, the Xburst-arch seems to be pretty fucked up beyond all repair.

Hmmm, that doesn't sound promising. But do you have references for that?
Maybe google has some more info than since I first started searching.
Anyway: for now it happily runs debian, and for what I can see, the kernel patches have no real changes except for extra drivers and extra board and powermanagement drivers.
So if you have hints in which way the xburst deviates from "standard" mips, it could help us a lot.
In the mean time I am going to subscribe to the ingenic forum ( http://www.ingenic.cn/eng/forum/vvFrmDefault.aspx ) that contains more characters that I can't read than characters that I can read :-(.

Regards,
Ard

Signature · 1 line

.signature not found