another regression on Efika

Hi,

I spotted another regression, caused by the following commits:

2007-02-21 Hollis Blanchard <[email protected]>

* kern/powerpc/ieee1275/init.c (HEAP_SIZE): Removed. (HEAP_LIMIT): New macro. (grub_claim_heap): Claim memory up to `heaplimit'.

2007-02-21 Hollis Blanchard <[email protected]>

* conf/powerpc-ieee1275.rmk (kernel_elf_LDFLAGS): Link at 64KB.
* kern/powerpc/ieee1275/init.c (_end): Add declaration. (_start): Likewise.
(grub_arch_modules_addr): Return address after `_end'.
* util/powerpc/ieee1275/grub-mkimage.c: Include grub/misc.h. (load_modules): Use new parameter as p_paddr' and p_vaddr'. (add_segments): Calculate `_end' from phdr size and location. (ALIGN_UP): Moved to ...
* include/grub/misc.h: here.
* include/grub/powerpc/ieee1275/kernel.h (GRUB_IEEE1275_MOD_ALIGN):
New macro.
(GRUB_IEEE1275_MODULE_BASE): Removed.

Hi,

I spotted another regression, caused by the following commits:

2007-02-21 Hollis Blanchard <[email protected]>

* kern/powerpc/ieee1275/init.c (HEAP_SIZE): Removed. (HEAP_LIMIT): New macro. (grub_claim_heap): Claim memory up to `heaplimit'.

2007-02-21 Hollis Blanchard <[email protected]>

* conf/powerpc-ieee1275.rmk (kernel_elf_LDFLAGS): Link at 64KB.
* kern/powerpc/ieee1275/init.c (_end): Add declaration. (_start): Likewise.
(grub_arch_modules_addr): Return address after `_end'.
* util/powerpc/ieee1275/grub-mkimage.c: Include grub/misc.h. (load_modules): Use new parameter as p_paddr' and p_vaddr'. (add_segments): Calculate `_end' from phdr size and location. (ALIGN_UP): Moved to ...
* include/grub/misc.h: here.
* include/grub/powerpc/ieee1275/kernel.h (GRUB_IEEE1275_MOD_ALIGN):
New macro.
(GRUB_IEEE1275_MODULE_BASE): Removed.

They move GRUB from being able to initialise and fire up the menu (although still unable to access disks), to abort with:

  kern/mm.c:  grub_error (GRUB_ERR_OUT_OF_MEMORY, "out of memory");

Unfortunately my limited expertise doesn't let me figure out what these changes are doing. I'm attaching a reverse diff of the commit, such that when applied to HEAD makes GRUB work again (verified) [1]. Perhaps someone can make sense of it.

This problem could be caused by firmware presenting an incorrect /memory/available property. I would instrument grub_available_iterate() and heap_init() to confirm.

That property lists unclaimed areas of memory, which are free for us to use as heap. The memory used for firmware itself (including exception handlers at 0), and the memory used by the loaded ELF file (GRUB), are missing from /memory/available, so we know not to try to use those regions for the heap.

If that is in fact the problem (which wouldn't surprise me given this particular firmware's track record), then we'll have to add some hardcoded hack particularly for Efika in grub_machine_init(), bypassing the more general code we have now.

Hey,

Quoted 10 lines · Hollis

This problem could be caused by firmware presenting an incorrect /memory/available property. I would instrument grub_available_iterate() and heap_init() to confirm.

That property lists unclaimed areas of memory, which are free for us to use as heap. The memory used for firmware itself (including exception handlers at 0), and the memory used by the loaded ELF file (GRUB), are missing from /memory/available, so we know not to try to use those regions for the heap.

Quoted 4 lines

If that is in fact the problem (which wouldn't surprise me given this particular firmware's track record), then we'll have to add some hardcoded hack particularly for Efika in grub_machine_init(), bypassing the more general code we have now.

I'm afraid there must be more to this.

[...]

When the package was later on updated to a CVS snapshot of 20070505, GRUB started dropping me to OF at boot time. On Saturday, Robert helped me track this regression down to this same commit. After reverting and regenerating grub, I'm happy to report that for the first time my Powerbook G4 laptop was able to boot with me pressing no keys at all, using GRUB.

In short, whatever is causing these crashes is not only an Efika issue. I suspect Apple, or Apple powerbooks at least, are also affected. I got Sjoerd to try this on his powerbook, and he confirms this fixes for him. Paul also tried, with the same results. Not all the powerbooks are the very same model, there are different model revisions involved.

Quoted 20 lines · Jordi Mallach

Hey,

Quoted 10 lines · Hollis

This problem could be caused by firmware presenting an incorrect /memory/available property. I would instrument grub_available_iterate() and heap_init() to confirm.

That property lists unclaimed areas of memory, which are free for us to use as heap. The memory used for firmware itself (including exception handlers at 0), and the memory used by the loaded ELF file (GRUB), are missing from /memory/available, so we know not to try to use those regions for the heap.

Quoted 4 lines

If that is in fact the problem (which wouldn't surprise me given this particular firmware's track record), then we'll have to add some hardcoded hack particularly for Efika in grub_machine_init(), bypassing the more general code we have now.

I'm afraid there must be more to this.

Quoted 8 lines

[...]

When the package was later on updated to a CVS snapshot of 20070505, GRUB started dropping me to OF at boot time. On Saturday, Robert helped me track this regression down to this same commit. After reverting and regenerating grub, I'm happy to report that for the first time my Powerbook G4 laptop was able to boot with me pressing no keys at all, using GRUB.

Quoted 5 lines

In short, whatever is causing these crashes is not only an Efika issue. I suspect Apple, or Apple powerbooks at least, are also affected. I got Sjoerd to try this on his powerbook, and he confirms this fixes for him. Paul also tried, with the same results. Not all the powerbooks are the very same model, there are different model revisions involved.

The error Jordi was seeing before reverting your commits was very similar to what I get on my Efika (something like "claim failed").

Hollis, do you still think this is a firmware bug?

Quoted 9 lines

Quoted 5 lines

In short, whatever is causing these crashes is not only an Efika issue. I suspect Apple, or Apple powerbooks at least, are also affected. I got Sjoerd to try this on his powerbook, and he confirms this fixes for him. Paul also tried, with the same results. Not all the powerbooks are the very same model, there are different model revisions involved.

The error Jordi was seeing before reverting your commits was very similar to what I get on my Efika (something like "claim failed").

Quoted 1 line

Hollis, do you still think this is a firmware bug?

Could somebody please debug it and report back if it is really a GRUB issue? It could be as simple as adjusting the link address, or it could mean instrumenting grub_available_iterate() and checking that output with the contents of /memory/available.

Quoted 9 lines · Hollis Blanchard

Quoted 9 lines

Quoted 5 lines

In short, whatever is causing these crashes is not only an Efika issue. I suspect Apple, or Apple powerbooks at least, are also affected. I got Sjoerd to try this on his powerbook, and he confirms this fixes for him. Paul also tried, with the same results. Not all the powerbooks are the very same model, there are different model revisions involved.

The error Jordi was seeing before reverting your commits was very similar to what I get on my Efika (something like "claim failed").

Quoted 6 lines

Quoted 1 line

Hollis, do you still think this is a firmware bug?

Could somebody please debug it and report back if it is really a GRUB issue? It could be as simple as adjusting the link address, or it could mean instrumenting grub_available_iterate() and checking that output with the contents of /memory/available.

It turns out that grub_claim_heap() was asked to claim a region that starts after the heap limit. The start address (0x19111e4) is correctly obtained from /memory/available.

Should heap limit be increased?

Quoted 17 lines · Robert Millan

Quoted 9 lines · Hollis Blanchard

Quoted 9 lines

Quoted 5 lines

In short, whatever is causing these crashes is not only an Efika issue. I suspect Apple, or Apple powerbooks at least, are also affected. I got Sjoerd to try this on his powerbook, and he confirms this fixes for him. Paul also tried, with the same results. Not all the powerbooks are the very same model, there are different model revisions involved.

The error Jordi was seeing before reverting your commits was very similar to what I get on my Efika (something like "claim failed").

Quoted 6 lines

Quoted 1 line

Hollis, do you still think this is a firmware bug?

Could somebody please debug it and report back if it is really a GRUB issue? It could be as simple as adjusting the link address, or it could mean instrumenting grub_available_iterate() and checking that output with the contents of /memory/available.

Quoted 5 lines

It turns out that grub_claim_heap() was asked to claim a region that starts after the heap limit. The start address (0x19111e4) is correctly obtained from /memory/available.

Should heap limit be increased?

The attached patch fixes the problem for Efika, although I'm not 1:1 sure of its correctness.

It seems to me that heaplimit is meant to be used as a relative offset, not as an absolute memory address. This makes the current code fail when the start address in /memory/available is higher than heaplimit, and I don't think there's anything that can grant us this won't happen. From this POV, it's clear that moving from relative to absolute address is wrong.

OTOH, your commit indicates (as per variable names, etc) that this change was done intentionally; so I have to assume there would be a reason for that. Please can you clarify?

Btw, the attached patch doesn't fix the problem for Jordi (Mac). Maybe we're talking about two separate bugs after all.

I was indeed trying to make it absolute. GRUB is linked at 0x10000, or 64KB. Add in all the modules and you still are only using a couple hundred KB.

I don't remember *why* I was trying to make it absolute; I'm sure it was to fix a problem (rather than I was bored). If we could get away from this anachronistic Changelog crap, the commit message would have told us exactly what we need to know.

I *think* it was because we have a relocation problem with OSes that are linked at 4MB, which is the case with Xen (and I believe yaboot). If GRUB is using that memory, we simply can't load anything there.

I think we can add some somewhat-special-case logic that says if there is nothing available below HEAP_LIMIT, claim something beginning as low as we can. That reduces functionality on such platforms (as described above), but there's little we can do about that unless firmware is fixed.

My big question is this: why is Efika firmware saying that 0x19111e4 (around 25MB) is the first available address?

BTW, does Jordi have any details on this Mac problem?

Quoted 4 lines · Hollis Blanchard

I was indeed trying to make it absolute. GRUB is linked at 0x10000, or 64KB. Add in all the modules and you still are only using a couple hundred KB.

But if GRUB is linked at 0x10000, how come the firmware loads it since that's below the 0x19111e4 limit? (or am I missing some physical vs logical issue here?)

Quoted 4 lines

I don't remember *why* I was trying to make it absolute; I'm sure it was to fix a problem (rather than I was bored). If we could get away from this anachronistic Changelog crap, the commit message would have told us exactly what we need to know.

I pasted the ChangeLog entries in an earlier mail. Do you mean the CVS commit message might have more information?

Quoted 4 lines

I *think* it was because we have a relocation problem with OSes that are linked at 4MB, which is the case with Xen (and I believe yaboot). If GRUB is using that memory, we simply can't load anything there.

Quoted 5 lines

I think we can add some somewhat-special-case logic that says if there is nothing available below HEAP_LIMIT, claim something beginning as low as we can. That reduces functionality on such platforms (as described above), but there's little we can do about that unless firmware is fixed.

So the purpose of HEAP_LIMIT is to support code that is linked at fixed addresses?

Quoted 2 lines

My big question is this: why is Efika firmware saying that 0x19111e4 (around 25MB) is the first available address?

What is wrong with that? Is it general practice that all firmwares load at low addresses so that non-relocatable code can be linked at 4MB or such?

Quoted 10 lines · Robert Millan

Quoted 4 lines · Hollis Blanchard

I was indeed trying to make it absolute. GRUB is linked at 0x10000, or 64KB. Add in all the modules and you still are only using a couple hundred KB.

But if GRUB is linked at 0x10000, how come the firmware loads it since that's below the 0x19111e4 limit? (or am I missing some physical vs logical
issue here?)

No, you are correct; obviously firmware is doing something wrong. The question is can we accommodate it, and if so, how?

[...]

25MB; we know this because we managed to load grub there. This undoubtedly an Efika firmware bug, but as I described earlier I think we can devise another workaround for it without hurting the common case.

The efika has 128MB of builtin memory, and we know that everything above 24MB except a little heap at the top is free, so this should be enough for everyone, no ?

The 0 to 16MB area is also free normally, and the pre-powerpc kernel loaded themselves at 4 or 8 MB, and relocate the kernel at 0.

Using the 16MB to 24MB area is a big no-no though, since it would kill the firmware.

The efika/pegasos firmware is not able to relocate itself dynamically, and as it reserves the first 256MB (1 BAT area), but it could be used with a 128 MB (like the efika) or smaller memory DIMM, so it cannot reside on the top area of ram where it belongs.

There is a small stack going down from min(top-of-memory,256MB) of a couple MB though.

The efika/pegasos firmware furthermore runs in physical memory, and cannot switch to virtual memory.

Hope this helps, altough obviously i can make no prediction about future efika firmwares.

Friendly,

Sven Luther

17:19 <svenl> nyu: this is another not really bug but ugly feature of the efika firmware, not present in the pegasos branch.

17:19 <svenl> the problem is that the graphic memory is mapped at the start, and the firmware is located from 16MB to 24MB or so, maybe a bit more with stack. 17:20 <svenl> normally; the area between 0 and 16MB, except for the interrupt vectors, should be available.

Hi Sven,

I have a pair of questions about the memory issue..

[...]

Quoted 6 lines · Robert Millan

Hi Sven,

I have a pair of questions about the memory issue..

[...]

Ok, after some discussion on IRC, and invaluable help by Sven in documenting smartfirmware behaviour, I came up with this patch.

Please let me know what you think. If there are no objections I'll check it in in a few days.

2007-07-30 Robert Millan <[email protected]>

* include/grub/ieee1275/ieee1275.h (grub_ieee1275_flag): Add GRUB_IEEE1275_FLAG_EFIKA_SECRET_AVAILABLE_REGION flag.
* kern/powerpc/ieee1275/cmain.c (grub_ieee1275_find_options): Set GRUB_IEEE1275_FLAG_EFIKA_SECRET_AVAILABLE_REGION flag when running on an Efika (SmartFirmware).
* kern/powerpc/ieee1275/init.c (grub_claim_heap): Fail gracefuly when addr > heaplimit.
Use grub_printf instead of grub_error for error reporting, since grub_error doesn't work at this point of execution.
When GRUB_IEEE1275_FLAG_EFIKA_SECRET_AVAILABLE_REGION was set, release hardcoded 0x4000:0xffc000 region.

Quoted 2 lines

When GRUB_IEEE1275_FLAG_EFIKA_SECRET_AVAILABLE_REGION was set, release hardcoded 0x4000:0xffc000 region.

Hmm, does this actually work? Since GRUB itself falls within that regions, the grub_ieee1275_release() call will mark all the memory that the GRUB kernel and modules occupy as free, which means they will be clobbered by heap usage.

Quoted 7 lines · Hollis Blanchard

Quoted 2 lines

When GRUB_IEEE1275_FLAG_EFIKA_SECRET_AVAILABLE_REGION was set, release hardcoded 0x4000:0xffc000 region.

Hmm, does this actually work? Since GRUB itself falls within that regions, the grub_ieee1275_release() call will mark all the memory that the GRUB kernel and modules occupy as free, which means they will be clobbered by heap usage.

Didn't think of this.. (by chance it worked here, though). How about using max(_end,0x4000) instead of 0x4000 ?

(and adjusting region length not to overflow 0x1000000)

Quoted 9 lines · Robert Millan

Quoted 7 lines · Hollis Blanchard

Quoted 2 lines

When GRUB_IEEE1275_FLAG_EFIKA_SECRET_AVAILABLE_REGION was set, release hardcoded 0x4000:0xffc000 region.

Hmm, does this actually work? Since GRUB itself falls within that regions, the grub_ieee1275_release() call will mark all the memory that the GRUB kernel and modules occupy as free, which means they will be clobbered by heap usage.

Quoted 4 lines

Didn't think of this.. (by chance it worked here, though). How about using max(_end,0x4000) instead of 0x4000 ?

(and adjusting region length not to overflow 0x1000000)

That won't quite work because the modules are loaded in the area immediately following the kernel. However, because they are added in a post-processing step, we don't know their end address at link time.

I think the simplest solution will be to allocate Efika's heap high, as we talked about earlier. If the first "available" region is above 4MB, just reserve 4MB and don't worry about where it is. In general we should try to keep 4MB free, but at the end of the day it's Efika's fault and probably won't be a big deal for them anyways.

Quoted 5 lines · Hollis Blanchard

I think the simplest solution will be to allocate Efika's heap high, as we talked about earlier. If the first "available" region is above 4MB, just reserve 4MB and don't worry about where it is. In general we should try to keep 4MB free, but at the end of the day it's Efika's fault and probably won't be a big deal for them anyways.

Ok, let's do that. But what do we do if the first available region is just a few kBs below 4MB? Or if it's just too small? I think we need to determine what is the minimum heap size we want to accept, such that the ability to keep 4MB free still compensates having a small heap.

Any suggestion?

Yup, it's a pain. Here's one idea: set a "GET_HEAP_FROM_WHEREVER" flag, then later do

We could try doing that in all situations, except:
- this method isn't often used, so firmwares can have bugs here

- if MIN_HEAP_SIZE were small enough (e.g. 2MB), it would mean we probably would still have 4MB+ free. However, that depends on the size of the modules. I would hate to document "oh, if you can't load your OS, try using fewer GRUB modules." On the other hand, we already have this problem to some extent: if our modules are too large, we will fail to allocate any heap. So maybe this isn't a practical concern.

Finally, I think I got this right now. My efika boots and all the other setups should be happy as well. Please, review/test this and report.

[...]

Hell[o]

Quoted 1 line

[...]

I just checked cvs grub2 with this patch on older version of pegasos firmware (1.2 - 20040810) and it seems to fail with out of memory message.

What happens if you increase HEAP_MIN_SIZE ?

Hell[o]

Quoted 1 line

What happens if you increase HEAP_MIN_SIZE ?

Still fails I am affraid. I wonder is there any reason to use only first 4MB for heap ? I guess higer memory areas can be still accessed excluding 4-16 MB ?

I attached my proposed patch to solve this problem. It tires to allocate 4MB heap at fixed position at 28MB (To avoid problems with relocation later on) and if it fails use the grub_available_iterate() as before. If should be used with ealier patch from this thread. I tested it on Pegasos 1 (Only one machine), Pegasos 2 (With few OF revisions starting from 2004 to 2006) and efika and it works quite well.

The biggest issue now is that we all had to fix our generated configurations, as grub-probe and update-grub won't cope with a device.map that looks like this:

(hd) /dev/hda

This is a minor issue that I can fix by hand, luckily.

What naming/numbering scheme does this follow when you add more disks?

Quoted 1 line · Robert Millan

What naming/numbering scheme does this follow when you add more disks?

hd, hd1, hd2. Luckily Mozillion has Firewire disks.

Quoted 4 lines · Jordi Mallach

Quoted 1 line · Robert Millan

What naming/numbering scheme does this follow when you add more disks?

hd, hd1, hd2. Luckily Mozillion has Firewire disks.

My Efika counts: hd, hd0 ...

IEEE 1275 does not seem to cover this. What scheme should we follow when generating device.map ?

Quoted 6 lines · Robert Millan

Quoted 4 lines · Jordi Mallach

Quoted 1 line · Robert Millan

What naming/numbering scheme does this follow when you add more disks?

hd, hd1, hd2. Luckily Mozillion has Firewire disks.

Quoted 4 lines

My Efika counts: hd, hd0 ...

IEEE 1275 does not seem to cover this. What scheme should we follow when generating device.map ?

Since "hd, hd1, hd2 ..." seems to be much more common, I propose we use that and let user override it via device.map in case of disparity (which may only happen in systems with more than 1 disk).

Quoted 12 lines · Robert Millan

Quoted 10 lines · Robert Millan

Quoted 4 lines · Jordi Mallach

Quoted 1 line · Robert Millan

What naming/numbering scheme does this follow when you add more disks?

hd, hd1, hd2. Luckily Mozillion has Firewire disks.

My Efika counts: hd, hd0 ...

IEEE 1275 does not seem to cover this. What scheme should we follow when generating device.map ?

Quoted 3 lines

Since "hd, hd1, hd2 ..." seems to be much more common, I propose we use that and let user override it via device.map in case of disparity (which may only happen in systems with more than 1 disk).

Can we actually read via /proc or so how this mapping in reality is realised?

AFAIK we can only read the full OF path from /proc (with helper tools like ofpath / ofpathname), but not the short alias which grub uses. These can only be obtained when grub is running.

If the proposed solution is not good, maybe we could instead remap all disks to "standard" grub device names (hd0, hd1..), which grub-mkdevicemap already knows how to guess reasonably well.

[...] I'm attaching a reverse diff of the commit, such that when applied to HEAD makes GRUB work again (verified) [1].