Kernel boot problem on IXP422 Rev. A

Hello,

we currently try to boot a 2.6.21 kernel on a custom IXP422 based board. The boot loader (U-Boot) works fine so far, memory and flash test run successfully, the NPE is also able to ping a host. So we assume the board itself is more or less working at least. When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow. We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.
I would be glad for any hint to be able to solve the problem.

Thank you a lot in advance.
Best regards
Marcus
_____________________________________________________________________
Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
http://smartsurfer.web.de/?mc=100071&distributionid=000000000066

time to upgrade

you'll have better luck asking on the arm kernel mailing lists. they process this stuff every day.
-mike

Wrong answer!!!

Many embedded devices can't upgrade kernels easily because of
customer requirements and certifications. For example, I have worked on Linux based applications in the financial industry. A kernel upgrade here is viewed as equivalent to switching from Windows XP to Vista, and requires significant effort in certification testing from the customer's perspective. This doesn't make economic sense for either party.

Often developers not working in the embedded space don't understand
this reality.

Helpful responses on what does/does not work would be more
appreciated.

... Glenn

Glenn Henshaw                     Logical Outcome Ltd.
e: [email protected]     w: www.logicaloutcome.ca

not really

and there are plenty of *wrong* reasons that people are using old kernels.

if i'm not classified as an embedded developer, then i dont know what one is :P

i gave a realistic one and i sent him to the right place -- the arm kernel mailing lists
-mike

Glenn,
I totally agree with your certification statement statement as a legit reason to not upgrade. I once worked for a aircraft lighting supplier, their FAA certification testing process took about 4 years. Going through the cert process makes one more resistant to change. Craig

Dr. Craig Hollabaugh, [email protected], 970 240 0509
Author of Embedded Linux: Hardware, Software and Interfacing
www.embeddedlinuxinterfacing.com

While the answer may be not valid any time you see somebody complaining about an relatively old kernel not working, it looks perfectly valid here.

They are experimenting with a new board, (well, not all that new, probably), and a year-old kernel does not work. The obvous thing it to try the latest kernel and see it it's working. Should at least give a clue about what's wrong.

If this certification was granted despite Marcus' admission that the kernel doesn't even boot -- it dies between gunzip and start_kernel -- then I suspect it wasn't the kind of certification which takes 4 years to achieve.

_Most_ people who are having trouble with old kernels have extremely _bad_ reasons for not updating.

dwmw2

Fairly common answer actually. It translates to:

There's a decent chance whatever bug you're encountering is one we've already fixed. As volunteers, we're not really interested in reinventing the wheel. We're trying to come up with a better kernel, and debugging old version may help you but it doesn't help us.

If you try the current version and the problem persists, we're happy to help you debug it, because this makes the kernel better for all of us. If you have your own reason to want to stick with an old version, there are plenty of vendors happy to take your money to make this work for you, but you've sacrificed the support of the volunteer community in doing so.

I'm aware of this. That's what vendors are for.

Making economic sense and making sense to open source volunteers are two different sets of criteria. Part of "making economic sense" is realizing that you must replace volunteer effort with paid effort to do it that way.

The procedure is:

1) Try to reproduce the bug under a current kernel. (Set up a _test_ system.)

If you can't reproduce it:
Git bisect to find the fix, then backport it to your kernel if possible.
If you can reproduce it:
Demonstrate the bug to the community in the current kernel, get help
debugging the problem, then backport the fix to your kernel if possible.

Often developers who _are_ working in the embedded space don't understand how and why volunteers collaborate through the internet in this thing called "open source development".

He did give a helpful response. You didn't understand it. I elaborated.

Rob

"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Rob,

This is an excellent and concise description of the open source perspective on the problem. I'll add just one note below.

This sounds easy, but can be quite difficult.

Very often, product developers are several versions behind, with no easy way to use the current kernel version. For example, a common scenario is starting with a kernel that comes with a board (with source mind you), where the kernel came from the semi-conductor vendor, who paid a Linux vendor to do a port, and it was released in a time-frame relative to the Linux vendor's product schedule.

This is how you end up having people STARTING projects today using a 2.6.11 kernel. (I know of many).

The real difficulty, when a developer finds themselves in this position, is how to forward-port the BSP code necessary to reproduce the bug in the current kernel. Often, the code is not isolated well enough (this is a vendor problem that really needs attention. If you have the BSP in patches, it is usually not too bad to forward port even across several kernel versions. But many vendors don't ship stuff this way.)

The fact is, that by a series of small steps and delays by the linux vendor, chip vendor, board vendor,
and product developer the code is out-of step.

It's easy to say "don't get in this position", but this even happens when everyone is playing nice and actively trying to mainline stuff. BSP support in arch trees often lag mainline by a version or two.

The number of parties involved here is why, IMHO, it has taken so long to make improvements in this area.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

It's not a question of difficult or easy: it's the procedure that works.

You don't get support from a commercial vendor unless you pay them money, and you don't get support from open source developers unless you help us make the next release just a little bit better. (We never said our help was free, we just said it didn't cost _money_. Ok, the FSF did but they don't speak for all of us...)

I'm aware of that. But if you can't set up a test system to reproduce the bug on a current system, the rest of us haven't got a _chance_.

Then poke your vendor to fix the problem.

If you've decided to use a divergent fork from a vendor rather than the mainstream version, then the vendor has to support that fork for you because we're not going to be familiar with it. (You can _hire_ one of us to support it for you, but we're not going to do so on a volunteer basis.)

We're happy to debug _our_ code. But "our code" is the current vanilla release tarball. If you can't reproduce the problem in the current vanilla tarball, then it's not our bug. If you can only reproduce it in an older
version: congratulations, we must have fixed it since. If you can only reproduce it in some other fork, obviously their changes introduced the bug. If it's "your code plus this patch", we need to see the patch.

If _you_ can't reproduce it in our code, how do you expect _us_ to?

Oldest I've seen a new project launch with this year is 2.6.15, but I agree with your point.

Whoever decided backporting bug fixes to a 2.6.16 kernel forever was a good idea seems to have muddied the waters a bit. Ironically I don't know anybody actually _using_ that version, but I've seen several people point to it to show that "the community" supports arbitrarily older versions forever, and thus they don't have to upgrade to get support, and 2.6.18 is actually _newer_ than that...

Yup. Sucks, doesn't it? This is not a problem that improves with the passage of time.

Might be a good idea to make it clear up front that even if your changes never get mainlined, failure to break up and break out your patches is still likely to cause maintenance problems down the road.

Hence the importance of breaking out and breaking up the changes.

Getting out of sync is inevitable. Happens to full-time kernel developers, that's why they have their own trees. That's a separate issue from asking for patches and getting a source tarball that compiles instead. "Here's a haystack, find the needle."

Mainlining changes and breaking them up into clean patches on top of some vanilla version (_any_ vanilla version) are two separate things. You have to win one battle before you can even start the other.

The lack of a clear consistent message from us to the vendors hasn't helped.

Rob

"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

What bootloader are you using? It may be that the memory controller
is initialized improperly.

For other resources, try the NSLU2 groups/pages. It runs the same
processor and kernel (actually many 2.6 variants).

    ... Glenn

Glenn Henshaw                     Logical Outcome Ltd.
e: [email protected]     w: www.logicaloutcome.ca

Hi Marcus,

The most common problem like this I have seen are that the ARM machine type set by the boot loader does not match those compiled into the kernel. Also make sure that the RAM size being passed to the kernel is correct too.

Enable the kernels CONFIG_DEBUG_LL and it may give you some early debug output (like an id not supported message). That assumes a serial console...

Regards
Greg

------------------------------------------------------------------------

Greg Ungerer  --  Chief Software Dude       EMAIL:     [email protected]
Secure Computing Corporation                PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com

Hi Greg,

first, thanks for the hints.

This seems to be set correctly.

Will try that,
Today I downloaded the snapgear distribution and tried whether i can generate a kernel image that works with it...and it boots (2.4 and 2.6) :-) Then I downloaded a fresh 2.6.25 kernel and compiled it with toolchain from the snapgear homepage...and it does not boot. Are there any special patches in the 2.6.19-uc1 from the snapgear distribution that might have influence on a IXP422 based board? Additionally, it seems that we may have a little problem in the bootloader (uboot) when passing kernel parameters (e.g."setenv bootargs console=ttyS0,115200"). Setting such a paramter causes the same problem with a dying kernel. Maybe there is a relation.

Regards
Marcus

Hi Marcus,

I don't recall any specific fixes as such. There is support for a few extra IXPxxx based boards.

Do a quick diff against a stock 2.6.19. There may be a number of changes related to non-mmu arm in there, they won't have any impact on IXP4xx support though.

Kernel dying, not just no console?
Is your serial console on ttyS0 or ttyS1?

Regards
Greg

------------------------------------------------------------------------

Greg Ungerer  --  Chief Software Dude       EMAIL:     [email protected]
SnapGear -- a Secure Computing Company      PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com

Hello Greg,

To be honest I don't know ;-) Maybe it's only an issue with the console. It should be ttyS0 as this works with 2.6.19-uc1. The snapgear kernel works when I use the settings for the MonteJade board. Today I took a look at the sources of the snapgear kernel where the CONFIG_MARCH_MONTEJADE is used, and there are some modifications within the 8250 serial driver for this board. I will take a closer look at it tomorrow.

Best regards
Marcus

Hi Marcus,

Over the years there have been some changes to the serial setup on those boards. Some setups leave out the first serial port definition, and so what is otherwise considered ttyS1 will be ttyS0. Something to look out for anyway.

Regards
Greg

------------------------------------------------------------------------

Greg Ungerer  --  Chief Software Dude       EMAIL:     [email protected]
Secure Computing Corporation                PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...

HTH!

--
Regards,
George