ARM: relocation out of range (when loading a module)
Hello,
I've build a kernel 2.6.37 for an armv5-device which includes an initramfs of about 16MB (uncompressed, result is a kernel of about 6MB) and I now getting errors when trying to load a module (which is included in that initramfs):
----------------------------
# modprobe ipv6 modprobe: 'kernel/net/ipv6/ipv6.ko': invalid module format # dmesg | tail -n 1 [ 107.327672] ipv6: relocation out of range, section 2 reloc 0 sym 'snmp_mib_free' ----------------------------
To compile the kernel I've used gcc 4.5.2 and binutils 2.21.
If I apply the patch found at
http://lists.arm.linux.org.uk/lurker/message/20100729.090437.85f82ebd.en.html
the output from dmesg will be
----------------------------
[ 58.211776] ipv6: section 2 reloc 0 sym 'snmp_mib_free': relocation 28 out of range (0xbf0000a4 -> 0xc11b4858)
----------------------------
I don't know much about how the memory is organized and the relocation is done for arm, but reading that output it seems the 32MB limit is reached. Is that because of the initramfs (just speculating) and is there a workaround to still include such an initramfs in the kernel?
An excerpt from my config:
----------------------------CONFIG_ARM_THUMB=y
...
# CONFIG_HIGHMEM is not set
# CONFIG_ARCH_SPARSEMEM_DEFAULT is not set
# CONFIG_ARCH_SELECT_MEMORY_MODEL is not set
# CONFIG_HIGHMEM is not setCONFIG_SELECT_MEMORY_MODEL=yCONFIG_FLATMEM_MANUAL=yCONFIG_FLATMEM=yCONFIG_FLAT_NODE_MEM_MAP=yCONFIG_HAVE_MEMBLOCK=yCONFIG_PAGEFLAGS_EXTENDED=y
----------------------------
Regards,
Alexander
Quoted 5 lines · Alexander Holler
Hello,
I've build a kernel 2.6.37 for an armv5-device which includes an initramfs of about 16MB (uncompressed, result is a kernel of about 6MB) and I now getting errors when trying to load a module (which is included in that initramfs):
Do you really need to have such a bit initramfs?
What you can do to work around this issue is to load your initramfs separately as a initrd ramdisk image. , and not combine it with the kernel image.
Nicolas
Am 10.01.2011 23:26, schrieb Nicolas Pitre:
Quoted 7 lines · Alexander Holler
Quoted 4 lines
I've build a kernel 2.6.37 for an armv5-device which includes an initramfs of about 16MB (uncompressed, result is a kernel of about 6MB) and I now getting errors when trying to load a module (which is included in that initramfs):
Do you really need to have such a bit initramfs?
Not really, but it is handier to use than a separate file.
Quoted 3 lines
What you can do to work around this issue is to load your initramfs separately as a initrd ramdisk image. , and not combine it with the kernel image.
Point is that I would like to understand why including an initramfs has such consequences.
I have to search for some pictures which are explaining the memory layout (never had any interested in that before).
I assume some memory near the kernel is reserved for the modules and an initramfs will come inbetween the kernel and the reserved memory for modules.
Regards,
Alexander
Am 11.01.2011 07:34, schrieb Alexander Holler:
Quoted 21 lines
Am 10.01.2011 23:26, schrieb Nicolas Pitre:
Quoted 10 lines · Alexander Holler
Quoted 7 lines
I've build a kernel 2.6.37 for an armv5-device which includes an initramfs of
about 16MB (uncompressed, result is a kernel of about 6MB) and I now getting
errors when trying to load a module (which is included in that initramfs):Do you really need to have such a bit initramfs?
Not really, but it is handier to use than a separate file.
Quoted 3 lines
What you can do to work around this issue is to load your initramfs separately as a initrd ramdisk image. , and not combine it with the kernel image.
Point is that I would like to understand why including an initramfs has such consequences.
Quoted 6 lines
I have to search for some pictures which are explaining the memory layout (never had any interested in that before).
I assume some memory near the kernel is reserved for the modules and an initramfs will come inbetween the kernel and the reserved memory for modules.
Thinking a bit more about that (for me unexcpected) "relocation out of range" error when loading a module, I'm curious if there could be other conditions which would lead to that error. Especially conditions which could be generated by the userspace or remote from some network.
Regards,
Alexander
Quoted 4 lines · Alexander Holler
Thinking a bit more about that (for me unexcpected) "relocation out of range" error when loading a module, I'm curious if there could be other conditions which would lead to that error. Especially conditions which could be generated by the userspace or remote from some network.
It's purely about the relative position of the kernel text to the module text. Nothing outside of the compilation of the kernel will affect it.
It's all to do with the branch instructions in modules having a limited range.
We generally place the kernel such that it starts 32K into the kernel's system RAM mapping. We place modules immediately below this mapping. The initramfs image sits in the kernel body in the startup data, which sits between the startup text at the beginning of the kernel, and the main kernel text which follows.
We _could_ rearrange the placement of these sections within the kernel image to eliminate the problem, but it's not a simple matter to just rearrange them - there's various assumptions made about the current arrangement. There's also considerations over whether that would cause additional memory to be wasted due to the alignment requirements of the various sections, and whether it would degrade the ability to get DMA-able memory on certain platforms.
Quoted 4 lines · Alexander Holler
Thinking a bit more about that (for me unexcpected) "relocation out of range" error when loading a module, I'm curious if there could be other conditions which would lead to that error. Especially conditions which could be generated by the userspace or remote from some network.
No.
Nicolas
Quoted 9 lines · Alexander Holler
Point is that I would like to understand why including an initramfs has such consequences.
I have to search for some pictures which are explaining the memory layout (never had any interested in that before).
I assume some memory near the kernel is reserved for the modules and an initramfs will come inbetween the kernel and the reserved memory for modules.
Modules get placed 16MiB below PAGE_OFFSET, and when you add another ~16MiB initramfs between them and the kernel symbols, you exceed the offset range of the branch instruction.
It's possible to hack around this by placing the initramfs at the end of the kernel image rather than at the beginning with the rest of the init data. Something like the below should work, although you should also probably take care of alignment and also have this section freed when the rest of the init data is freed.
arch/arm/kernel/vmlinux.lds.S +5 -1
@@ -46,7 +46,6 @@ SECTIONS INIT_CALLS CON_INITCALL SECURITY_INITCALL - INIT_RAM_FS #ifndef CONFIG_XIP_KERNEL __init_begin = _stext;
@@ -170,6 +169,11 @@ SECTIONS _edata = .; } + + .initramfs : { + INIT_RAM_FS + } + _edata_loc = __data_loc + SIZEOF(.data); #ifdef CONFIG_HAVE_TCM
Quoted 5 lines · Rabin Vincent
It's possible to hack around this by placing the initramfs at the end of the kernel image rather than at the beginning with the rest of the init data. Something like the below should work, although you should also probably take care of alignment and also have this section freed when the rest of the init data is freed.
You're then running into problems as _sdata.._edata is copied to RAM on XIP kernels, and you really don't want to waste time copying the initramfs to RAM.
Hello,
Am 11.01.2011 16:59, schrieb Russell King - ARM Linux:
Quoted 10 lines · Rabin Vincent
Quoted 5 lines
It's possible to hack around this by placing the initramfs at the end of the kernel image rather than at the beginning with the rest of the init data. Something like the below should work, although you should also probably take care of alignment and also have this section freed when the rest of the init data is freed.
You're then running into problems as _sdata.._edata is copied to RAM on XIP kernels, and you really don't want to waste time copying the initramfs to RAM.
Thanks to all for the provided informations. I'm now seeing some light. ;)
However, looking at the vmlinux.lds.S I think I should better not try to fix that, I would likely do something wrong because of missing knowledge about all those sections.
I can offer a patch wich adds a TODO to vmlinux.lds.S, but if someone else feels the need to fix that, feel free to do so. ;)
Regards,
Alexander
-------------------------------------------------------------------------
From 2ce2934b6a55ac34734781d1a49569d79d0fdcda Mon Sep 17 00:00:00 2001
From: Alexander Holler <[email protected]>
Date: Wed, 12 Jan 2011 02:49:52 +0100
Subject: [PATCH] ARM: Add TODO to move INIT_RAM_FS to another point.
--- arch/arm/kernel/vmlinux.lds.S | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
arch/arm/kernel/vmlinux.lds.S +6 -0
@@ -62,6 +62,12 @@ SECTIONS INIT_CALLS CON_INITCALL SECURITY_INITCALL + /* + * TODO: The size of INIT_RAM_FS could easily reach a + * point (~16MB) when loading modules will fail because + * relocations will be out of range. So this place here + * isn't the best one. + */ INIT_RAM_FS #ifndef CONFIG_XIP_KERNEL
Signature · 2 lines
1.7.3.4 -------------------------------------------------------------------------
Quoted 1 line · Alexander Holler
Hello,
[...]
Quoted 10 lines
? ? ? ? ? ? ? ?INIT_CALLS
? ? ? ? ? ? ? ?CON_INITCALL
? ? ? ? ? ? ? ?SECURITY_INITCALL
+ ? ? ? ? ? ? ? /*
+ ? ? ? ? ? ? ? ?* TODO: The size ofINIT_RAM_FScould easily reach a
+ ? ? ? ? ? ? ? ?* point (~16MB) when loading modules will fail because
+ ? ? ? ? ? ? ? ?* relocations will be out of range. So this place here
+ ? ? ? ? ? ? ? ?* isn't the best one.
+ ? ? ? ? ? ? ? ?*/
? ? ? ? ? ? ? ?INIT_RAM_FS
In general, do we expect always to be able to avoid the situation where branches in the kernel may need to cover too large a range ... and is there any strategy for working aroung it?
If we have problems branching from the modules area into vmlinux, we could possibly build modules with -fPIC : this would remove the restriction on branch range, though there would also be some performance impact for the modules...
Cheers
---Dave
Quoted 3 lines · Dave Martin
In general, do we expect always to be able to avoid the situation where branches in the kernel may need to cover too large a range ... and is there any strategy for working aroung it?
It's not that big a problem - 99.999999% of setups never run into the problem. It's only those who use large initramfs's built into their kernel image at present, and that is ultimately solvable.
I don't think we need to litter code with TODO comments.
Quoted 4 lines
If we have problems branching from the modules area into vmlinux, we could possibly build modules with -fPIC : this would remove the restriction on branch range, though there would also be some performance impact for the modules...
That also brings in issues with GOT tables and the like, and also requires different build options for modules and the main kernel. I don't think kbuild is setup to do that - and I'd argue that it's unnecessary if we fix the layout of the kernel image.
As I said a few emails ago, shuffling sections around in the image is not as trivial as it looks on the face of it as we make assumptions about what is in _stext.._etext, _sdata.._edata, whether there's anything between _etext.._sdata, and other symbolic ranges.
At the moment, _stext.._etext + _sdata.._edata covers the entire kernel image - with the init sections at the start of _stext. Putting the init sections between _etext and _sdata makes a hole in the middle of that, which may be suboptimal for page allocation. It also means that the range no longer covers all kernel stuff. Also whether _stext = _text, _sdata = _data, etc.
Another technicality is that some of these ranges are used for stuff like DMA API debugging:
if (overlap(addr, len, _text, _etext) ||
overlap(addr, len, __start_rodata, __end_rodata))
err_printk(dev, NULL, "DMA-API: device driver maps memory from kernel text or rodata [addr=%p] [len=%lu]\n", addr, len);The last pieces of the puzzle is whether we have anything that implicitly relies on the init section being low down and freeing its pages (eg, as a way of stopping that memory being used for non-DMA stuff.)
I do have a patch which shuffles some of this stuff around, but I'm not entirely happy with it yet.
Quoted 3 lines · Dave Martin
In general, do we expect always to be able to avoid the situation where branches in the kernel may need to cover too large a range ... and is there any strategy for working aroung it?
Normally we try to keep the code close together. And at least for the .text section, the linker is able to insert indirect branch veneers when the range is too large. But in the case of modules, it's the in-kernel code that perform the final symbol relocation.
Quoted 4 lines
If we have problems branching from the modules area into vmlinux, we could possibly build modules with -fPIC : this would remove the restriction on branch range, though there would also be some performance impact for the modules...
If we really needed to do such thing, that would be even better to simply have the kernel create those indirect veneers dynamically. And in fact, Russell had that working and he posted the corresponding patch many years ago, but the module placement was made so that the indirect branches were unnecessary.
Even now, it's probably a better idea to rework the section layout and preserve the ability to perform direct branches into the kernel from modules as much as possible.
Nicolas
Quoted 5 lines · Nicolas Pitre
If we really needed to do such thing, that would be even better to simply have the kernel create those indirect veneers dynamically. And in fact, Russell had that working and he posted the corresponding patch many years ago, but the module placement was made so that the indirect branches were unnecessary.
Actually, it's something we used to do in 2.2 days when modules were prepared and linked in userspace before being uploaded into kernel space. This allowed the module to be inteligently linked - so the indirect branches were created only when they were necessary.
When the new kernel-based module linker happened, this presented a chicken and egg problem with that approach, which give us a choice:
either place the module within range of the kernel text and guarantee that the kernel text is reachable, or _always_ indirect every module branch through a jump table even if it was reachable from where the module was placed.
The decision was made to go with the former, so the latter never got implemented.
Then came along the embedded initrd/initramfs idea which rather buggered the scheme when large initramfs are embedded into the image.
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
Quoted 6 lines · Russell King - ARM Linux
Then came along the embedded initrd/initramfs idea which rather buggered the scheme when large initramfs are embedded into the image.
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
It is... but we have more than 32MB of RAM total now, and people are running standard distributions on ARM these days, such as Fedora or Ubuntu, including their corresponding initrd that may contain lots of modules, splashscreen data, etc. So it might be a good idea to think about fixing this limitation before it comes back again.
Nicolas
Hello,
Quoted 9 lines · Nicolas Pitre
Quoted 6 lines · Russell King - ARM Linux
Then came along the embedded initrd/initramfs idea which rather buggered the scheme when large initramfs are embedded into the image.
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
Quoted 6 lines
It is... but we have more than 32MB of RAM total now, and people are running standard distributions on ARM these days, such as Fedora or Ubuntu, including their corresponding initrd that may contain lots of modules, splashscreen data, etc. ?So it might be a good idea to think about fixing this limitation before it comes back again.
in fact it is not just the initrd/initramfs size, i am facing issues with loading modules on MMU-less envt(Cortex M3).
I have 128MB of Physical RAM and i am using a 4MB initrd to boot my system. when i try to load kernel modules i get relocation errors. As MMU-less systems are allowing modules to be loaded anywhere from RAM and not from any predefined limits of MODULES_VADDR and MODULES_END.
Do you have any pointers on how to tackle this situation for MMU less environments?
Regards,
Sachin.
Quoted 26 lines · Sachin Verma
Quoted 9 lines · Nicolas Pitre
Quoted 6 lines · Russell King - ARM Linux
Then came along the embedded initrd/initramfs idea which rather buggered the scheme when large initramfs are embedded into the image.
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
Quoted 6 lines
It is... but we have more than 32MB of RAM total now, and people are running standard distributions on ARM these days, such as Fedora or Ubuntu, including their corresponding initrd that may contain lots of modules, splashscreen data, etc. ?So it might be a good idea to think about fixing this limitation before it comes back again.
in fact it is not just the initrd/initramfs size, i am facing issues with loading modules on MMU-less envt(Cortex M3).
I have 128MB of Physical RAM and i am using a 4MB initrd to boot my system. when i try to load kernel modules i get relocation errors. As MMU-less systems are allowing modules to be loaded anywhere from RAM and not from any predefined limits ofMODULES_VADDRandMODULES_END.
Quoted 2 lines
Do you have any pointers on how to tackle this situation for MMU less environments?
I don't think there is a simple solution. As a hack, you could define DMA and Normal Zones in your system and force module allocations from the DMA zone (which would be close to the kernel).
I'm not sure whether the kernel could be compile with a larger memory model, but then you have to improve the kernel to handle new relocation types.
Signature · 1 line
Catalin
Quoted 25 lines · Sachin Verma
Hello,
Quoted 6 lines · Nicolas Pitre
Quoted 3 lines · Russell King - ARM Linux
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
Quoted 6 lines
It is... but we have more than 32MB of RAM total now, and people are running standard distributions on ARM these days, such as Fedora or Ubuntu, including their corresponding initrd that may contain lots of modules, splashscreen data, etc. ?So it might be a good idea to think about fixing this limitation before it comes back again.
in fact it is not just the initrd/initramfs size, i am facing issues with loading modules on MMU-less envt(Cortex M3).
I have 128MB of Physical RAM and i am using a 4MB initrd to boot my system. when i try to load kernel modules i get relocation errors. As MMU-less systems are allowing modules to be loaded anywhere from RAM and not from any predefined limits ofMODULES_VADDRandMODULES_END.
Quoted 2 lines
Do you have any pointers on how to tackle this situation for MMU less environments?
The MMU-less kernel should still favor allocations close to the kernel text for modules, and anything else away from the kernel going downwards.
Otherwise a veneer should be created by the module symbol resolver such that if the branch distance to reach, say, printk is too large, then the following code would have to be dynamically generated right next to the module:
ldr pc, [pc, #-4] .word <far_away_symbol>
Then, in your module, you patch the branch relocation for printk so that it branches to the code above instead, and then store the address of printk at the location represented by the .word directive.
Nicolas
Quoted 9 lines · Nicolas Pitre
The MMU-less kernel should still favor allocations close to the kernel text for modules, and anything else away from the kernel going downwards.
Otherwise a veneer should be created by the module symbol resolver such that if the branch distance to reach, say, printk is too large, then the following code would have to be dynamically generated right next to the module:
Quoted 6 lines
ldr pc, [pc, #-4] .word <far_away_symbol>Then, in your module, you patch the branch relocation for printk so that it branches to the code above instead, and then store the address of printk at the location represented by the .word directive.
What you're suggesting is what we used to do with the old user-space module tools, which would've been nice to carry forwards to the new module code. I never found a way to do it.
The problems:
1. Where do you create those veneers?
2. How many veneers do you allocate space for?
3. How do you determine that you need a veneer?
While you can say "next to the module" for (1), you can only do that at the point in time when the space for the module is allocated, and you need to know at that point how much space you require.
For (2), you could always allocate space for one veneer per symbol present in the module, but that's very wasteful.
(3) is almost impossible to know ahead of time as you don't have the relocations, realistically you have to allocate one veneer per symbol, and as you don't know whether it's a data or code symbol, you'll have to allocate one veneer for every symbol in a module.
I really don't like it, and I don't see that this is sanely solvable without giving architectures much more control over module loading, which I don't think will ever happen. It's probably simpler to build modules with whatever that magic option is to tell GCC to always generate 'far call' veneers for everything rather than trying to 'fix' the kernel module loader.
Quoted 21 lines · Russell King - ARM Linux
Quoted 9 lines · Nicolas Pitre
The MMU-less kernel should still favor allocations close to the kernel text for modules, and anything else away from the kernel going downwards.
Otherwise a veneer should be created by the module symbol resolver such that if the branch distance to reach, say, printk is too large, then the following code would have to be dynamically generated right next to the module:
Quoted 6 lines
ldr pc, [pc, #-4] .word <far_away_symbol>Then, in your module, you patch the branch relocation for printk so that it branches to the code above instead, and then store the address of printk at the location represented by the .word directive.
What you're suggesting is what we used to do with the old user-space module tools, which would've been nice to carry forwards to the new module code. I never found a way to do it.
Quoted 8 lines
The problems:
1. Where do you create those veneers?
2. How many veneers do you allocate space for?
3. How do you determine that you need a veneer?While you can say "next to the module" for (1), you can only do that at the point in time when the space for the module is allocated, and you need to know at that point how much space you require.
You would have to guess of course. Having a guess of 1/2 the module size should be pretty safe. So allocating 3/2 the space in module_alloc(), and then suffice to free the unused portion in module_finalize().
Quoted 7 lines
For (2), you could always allocate space for one veneer per symbol present in the module, but that's very wasteful.
(3) is almost impossible to know ahead of time as you don't have the relocations, realistically you have to allocate one veneer per symbol, and as you don't know whether it's a data or code symbol, you'll have to allocate one veneer for every symbol in a module.
I don't think you may know the number of symbols in advance either anyway.
Quoted 6 lines
I really don't like it, and I don't see that this is sanely solvable without giving architectures much more control over module loading, which I don't think will ever happen. It's probably simpler to build modules with whatever that magic option is to tell GCC to always generate 'far call' veneers for everything rather than trying to 'fix' the kernel module loader.
Well, this is certainly possible indeed to just use -mlong-calls to build modules which would solve the issue right away, and with a smaller cost than if each function calls were going through a veneer, but with a higher cost than a relocatable direct branch. So this could be decided with a Kconfig option.
Nicolas
Quoted 40 lines · Nicolas Pitre
Quoted 21 lines · Russell King - ARM Linux
Quoted 9 lines · Nicolas Pitre
The MMU-less kernel should still favor allocations close to the kernel text for modules, and anything else away from the kernel going downwards.
Otherwise a veneer should be created by the module symbol resolver such that if the branch distance to reach, say, printk is too large, then the following code would have to be dynamically generated right next to the module:
Quoted 6 lines
? ? ldr ? ? pc, [pc, #-4]
? ? .word ? <far_away_symbol>Then, in your module, you patch the branch relocation for printk so that it branches to the code above instead, and then store the address of printk at the location represented by the .word directive.
What you're suggesting is what we used to do with the old user-space module tools, which would've been nice to carry forwards to the new module code. ?I never found a way to do it.
Quoted 8 lines
The problems:
1. Where do you create those veneers?
2. How many veneers do you allocate space for?
3. How do you determine that you need a veneer?While you can say "next to the module" for (1), you can only do that at the point in time when the space for the module is allocated, and you need to know at that point how much space you require.
You would have to guess of course. ?Having a guess of 1/2 the module size should be pretty safe. ?So allocating 3/2 the space in
module_alloc(), and then suffice to free the unused portion inmodule_finalize().Quoted 3 lines
For (2), you could always allocate space for one veneer per symbol present in the module, but that's very wasteful.
Quoted 7 lines
Quoted 4 lines
(3) is almost impossible to know ahead of time as you don't have the relocations, realistically you have to allocate one veneer per symbol, and as you don't know whether it's a data or code symbol, you'll have to allocate one veneer for every symbol in a module.
I don't think you may know the number of symbols in advance either anyway.
You could probably cook up a good upper bound based on the size of the kernel and the number of symbols in the module: i.e., assume that every undefined symbol in the module needs to be fixed up to point at the most distant symbol in the kernel.
For people with normal-sized kernels, this bound will probably work out as zero most of the time (i.e., the current situation). For people with big kernels, or when many modules are already loaded, it may work out at 100% -- but that's the price to pay for guaranteed preallocation of the space required for the veneers. And anyway, you may really need a substantial chunk of those veneers in such cases.
---Dave
Quoted 11 lines · Dave Martin
You could probably cook up a good upper bound based on the size of the kernel and the number of symbols in the module: i.e., assume that every undefined symbol in the module needs to be fixed up to point at the most distant symbol in the kernel.
For people with normal-sized kernels, this bound will probably work out as zero most of the time (i.e., the current situation). For people with big kernels, or when many modules are already loaded, it may work out at 100% -- but that's the price to pay for guaranteed preallocation of the space required for the veneers. And anyway, you may really need a substantial chunk of those veneers in such cases.
I think it's going to be easier just to re-order the kernel image link order to solve that problem. That just leaves uclinux...
Quoted 13 lines · Russell King - ARM Linux
Quoted 11 lines · Dave Martin
You could probably cook up a good upper bound based on the size of the kernel and the number of symbols in the module: i.e., assume that every undefined symbol in the module needs to be fixed up to point at the most distant symbol in the kernel.
For people with normal-sized kernels, this bound will probably work out as zero most of the time (i.e., the current situation). ?For people with big kernels, or when many modules are already loaded, it may work out at 100% -- but that's the price to pay for guaranteed preallocation of the space required for the veneers. ?And anyway, you may really need a substantial chunk of those veneers in such cases.
Quoted 3 lines
I think it's going to be easier just to re-order the kernel image link order to solve that problem. ?That just leaves uclinux...
But if the kernel is big, or there are many modules, changing the order can't solve the problem surely? Or is the problem purely caused by having the initramfs in the way, and we think the amount of actual code will never be big enough to cause a problem?
Cheers
---Dave
Quoted 15 lines · Dave Martin
Quoted 13 lines · Russell King - ARM Linux
Quoted 11 lines · Dave Martin
You could probably cook up a good upper bound based on the size of the kernel and the number of symbols in the module: i.e., assume that every undefined symbol in the module needs to be fixed up to point at the most distant symbol in the kernel.
For people with normal-sized kernels, this bound will probably work out as zero most of the time (i.e., the current situation). ?For people with big kernels, or when many modules are already loaded, it may work out at 100% -- but that's the price to pay for guaranteed preallocation of the space required for the veneers. ?And anyway, you may really need a substantial chunk of those veneers in such cases.
Quoted 8 lines
Quoted 3 lines
I think it's going to be easier just to re-order the kernel image link order to solve that problem. ?That just leaves uclinux...
But if the kernel is big, or there are many modules, changing the order can't solve the problem surely? Or is the problem purely caused by having the initramfs in the way, and we think the amount of actual code will never be big enough to cause a problem?
Right now the biggest problem comes from the size of the initramfs. The solution for that is simply to move it elsewhere towards the end of the kernel image instead of at the beginning. And I think we're good for a couple years before the actual kernel code size becomes an issue.
Nicolas
Quoted 46 lines · Dave Martin
Quoted 45 lines · Nicolas Pitre
Quoted 21 lines · Russell King - ARM Linux
Quoted 9 lines · Nicolas Pitre
The MMU-less kernel should still favor allocations close to the kernel text for modules, and anything else away from the kernel going downwards.
Otherwise a veneer should be created by the module symbol resolver such that if the branch distance to reach, say, printk is too large, then the following code would have to be dynamically generated right next to the module:
Quoted 6 lines
? ? ldr ? ? pc, [pc, #-4]
? ? .word ? <far_away_symbol>Then, in your module, you patch the branch relocation for printk so that it branches to the code above instead, and then store the address of printk at the location represented by the .word directive.
What you're suggesting is what we used to do with the old user-space module tools, which would've been nice to carry forwards to the new module code. ?I never found a way to do it.
Quoted 8 lines
The problems:
1. Where do you create those veneers?
2. How many veneers do you allocate space for?
3. How do you determine that you need a veneer?While you can say "next to the module" for (1), you can only do that at the point in time when the space for the module is allocated, and you need to know at that point how much space you require.
You would have to guess of course. ?Having a guess of 1/2 the module size should be pretty safe. ?So allocating 3/2 the space in
module_alloc(), and then suffice to free the unused portion inmodule_finalize().Quoted 7 lines
For (2), you could always allocate space for one veneer per symbol present in the module, but that's very wasteful.
(3) is almost impossible to know ahead of time as you don't have the relocations, realistically you have to allocate one veneer per symbol, and as you don't know whether it's a data or code symbol, you'll have to allocate one veneer for every symbol in a module.
Quoted 7 lines
Quoted 2 lines
I don't think you may know the number of symbols in advance either anyway.
You could probably cook up a good upper bound based on the size of the kernel and the number of symbols in the module: i.e., assume that every undefined symbol in the module needs to be fixed up to point at the most distant symbol in the kernel.
Sure... It is just that the memory allocation is currently done before the number of symbols in the module is known. Changing that would require non trivial changes in the generic module loading code which potentially would affect all architectures, and therefore I don't think we want to go there.
The other solution would be to determine the number of objects in need of a veneer in apply_relocate(), allocate a replacement area for the module, copy everything over, and then create the veneers close to the module. But 1) the second allocation may fail, and 2) this will change the distance from the kernel potentially requiring more veneers than initially determined, and 3) the generic module code might still have pointer references into the old allocation area (didn't check but that can be expected). That's just too messy.
Quoted 6 lines
For people with normal-sized kernels, this bound will probably work out as zero most of the time (i.e., the current situation). For people with big kernels, or when many modules are already loaded, it may work out at 100% -- but that's the price to pay for guaranteed preallocation of the space required for the veneers. And anyway, you may really need a substantial chunk of those veneers in such cases.
I still think that compiling modules with -mlong-calls, and making that conditional on a kernel config option so only those who need it will have it, is the simplest solution. Nothing in the kernel code would need to be changed.
Nicolas
Quoted 78 lines · Nicolas Pitre
Quoted 46 lines · Dave Martin
Quoted 45 lines · Nicolas Pitre
Quoted 21 lines · Russell King - ARM Linux
Quoted 9 lines · Nicolas Pitre
The MMU-less kernel should still favor allocations close to the kernel text for modules, and anything else away from the kernel going downwards.
Otherwise a veneer should be created by the module symbol resolver such that if the branch distance to reach, say, printk is too large, then the following code would have to be dynamically generated right next to the module:
Quoted 6 lines
? ? ldr ? ? pc, [pc, #-4]
? ? .word ? <far_away_symbol>Then, in your module, you patch the branch relocation for printk so that it branches to the code above instead, and then store the address of printk at the location represented by the .word directive.
What you're suggesting is what we used to do with the old user-space module tools, which would've been nice to carry forwards to the new module code. ?I never found a way to do it.
Quoted 8 lines
The problems:
1. Where do you create those veneers?
2. How many veneers do you allocate space for?
3. How do you determine that you need a veneer?While you can say "next to the module" for (1), you can only do that at the point in time when the space for the module is allocated, and you need to know at that point how much space you require.
You would have to guess of course. ?Having a guess of 1/2 the module size should be pretty safe. ?So allocating 3/2 the space in
module_alloc(), and then suffice to free the unused portion inmodule_finalize().Quoted 7 lines
For (2), you could always allocate space for one veneer per symbol present in the module, but that's very wasteful.
(3) is almost impossible to know ahead of time as you don't have the relocations, realistically you have to allocate one veneer per symbol, and as you don't know whether it's a data or code symbol, you'll have to allocate one veneer for every symbol in a module.
Quoted 7 lines
Quoted 2 lines
I don't think you may know the number of symbols in advance either anyway.
You could probably cook up a good upper bound based on the size of the kernel and the number of symbols in the module: i.e., assume that every undefined symbol in the module needs to be fixed up to point at the most distant symbol in the kernel.
Sure... It is just that the memory allocation is currently done before the number of symbols in the module is known. ?Changing that would require non trivial changes in the generic module loading code which potentially would affect all architectures, and therefore I don't think we want to go there.
The other solution would be to determine the number of objects in need of a veneer in
apply_relocate(), allocate a replacement area for the module, copy everything over, and then create the veneers close to the module. ?But 1) the second allocation may fail, and 2) this will change the distance from the kernel potentially requiring more veneers than initially determined, and 3) the generic module code might still have pointer references into the old allocation area (didn't check but that can be expected). ?That's just too messy.Quoted 6 lines
For people with normal-sized kernels, this bound will probably work out as zero most of the time (i.e., the current situation). ?For people with big kernels, or when many modules are already loaded, it may work out at 100% -- but that's the price to pay for guaranteed preallocation of the space required for the veneers. ?And anyway, you may really need a substantial chunk of those veneers in such cases.
Quoted 4 lines
I still think that compiling modules with -mlong-calls, and making that conditional on a kernel config option so only those who need it will have it, is the simplest solution. ?Nothing in the kernel code would need to be changed.
Yes, this sounds like a reasonable solution for now ... since most people don't need this turned on anyway, and those that do need it know who they are.
if we can also move the initramfs somewhere less obstructive in the
future, then so much the better.
Cheers
---Dave
Quoted 2 lines · Dave Martin
if we can also move the initramfs somewhere less obstructive in the
future, then so much the better.
This may do it, and it might even work. Not particularly well tested yet. It'd be loads easier if we didn't have to worry about XIP.
arch/arm/kernel/etm.c | 2 +- arch/arm/kernel/smp.c | 4 +- arch/arm/kernel/vmlinux.lds.S | 133 +++++++++++++++++++++------------------- arch/arm/mm/init.c | 2 +- 4 files changed, 74 insertions(+), 67 deletions(-)
arch/arm/kernel/etm.c +1 -1
@@ -114,7 +114,7 @@ static int trace_start(struct tracectx *t)return -EFAULT ; } -etm_setup_address_range (t,1 , (unsigned long )_stext, +etm_setup_address_range (t,1 , (unsigned long )_text, (unsigned long )_etext,0 ,0 );etm_writel (t,0 ,ETMR_TRACEENCTRL2 );etm_writel (t,0 ,ETMR_TRACESSCTRL );
arch/arm/kernel/smp.c +2 -2
@@ -95,7 +95,7 @@ int __cpuinit __cpu_up(unsigned int cpu)#ifndef CONFIG_HOTPLUG_CPU identity_mapping_add (pgd,__pa (__init_begin),__pa (__init_end));#endif -identity_mapping_add (pgd,__pa (_stext),__pa (_etext)); +identity_mapping_add (pgd,__pa (_text),__pa (_etext));identity_mapping_add (pgd,__pa (_sdata),__pa (_edata)); }
@@ -143,7 +143,7 @@ int __cpuinit __cpu_up(unsigned int cpu)#ifndef CONFIG_HOTPLUG_CPU identity_mapping_del (pgd,__pa (__init_begin),__pa (__init_end));#endif -identity_mapping_del (pgd,__pa (_stext),__pa (_etext)); +identity_mapping_del (pgd,__pa (_text),__pa (_etext));identity_mapping_del (pgd,__pa (_sdata),__pa (_edata)); }
arch/arm/kernel/vmlinux.lds.S +70 -63
193 lines not on this page · raw message
arch/arm/mm/init.c +1 -1
@@ -294,7 +294,7 @@ void __init arm_memblock_init(struct meminfo *mi, struct machine_desc *mdesc)#ifdef CONFIG_XIP_KERNEL memblock_reserve (__pa (_sdata), _end - _sdata);#else -memblock_reserve (__pa (_stext), _end - _stext); +memblock_reserve (__pa (_text), _end - _text);#endif #ifdef CONFIG_BLK_DEV_INITRD if (phys_initrd_size &&
Nicolas Pitre
Acked-byQuoted 6 lines · Russell King - ARM Linux
Quoted 2 lines · Dave Martin
if we can also move the initramfs somewhere less obstructive in the
future, then so much the better.This may do it, and it might even work. Not particularly well tested yet. It'd be loads easier if we didn't have to worry about XIP.
Is anyone actually still using XIP these days? Now that NOR flash is not that much used anymore, if maintaining XIP support is more trouble than it is worth we could simply deprecate it. The linker script has become quite more hairy than it used to be, and from a quick look I'm not even sure if all the stuff in there is properly located for XIP.
Otherwise, for the __init stuff move:
Nicolas
Hello,
Am 12.01.2011 19:42, schrieb Russell King - ARM Linux:
Quoted 3 lines
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
Hmm, sorry, I have to disagree. ;)
E.g. 12MB out of the 16MB in the initramfs I've used are modules. I've also created some larger initramfs wich included X11 and such, and I find it very convenient to use the initramfs to include a rootfs in the the kernel and so only have to handle one file for all. E.g. just load the kernel with the included rootfs via tftp and you are done.
Sure this can all be done using an external initrd, but that is much less comfortable to use.
At least I like that feature since it is available and won't miss it now. ;)
Regards,
Alexander
Quoted 8 lines · Alexander Holler
Hello,
Am 12.01.2011 19:42, schrieb Russell King - ARM Linux:
Quoted 3 lines
As the overall feeling at the time was "don't use large initrds" it's something I've never really cared about - and I'm still of the opinion that 16MB of compressed initrd/initramfs is rather silly.
Quoted 6 lines
Hmm, sorry, I have to disagree. ;)
E.g. 12MB out of the 16MB in the initramfs I've used are modules. I've also created some larger initramfs wich included X11 and such, and I find it very convenient to use the initramfs to include a rootfs in the the kernel and so only have to handle one file for all. E.g. just load the kernel with the included rootfs via tftp and you are done.
Oddly, that's why I wrote the bootp wrapper, which predates the currently broken initrd-in-kernel idea.
Quoted 4 lines
Sure this can all be done using an external initrd, but that is much less comfortable to use.
At least I like that feature since it is available and won't miss it now. ;)
The feature isn't available if it doesn't work. It's not going to get fixed for 2.6.38 - it'll be 2.6.39 due to the timing and complexity of the problem.
Am 13.01.2011 11:04, schrieb Russell King - ARM Linux:
Quoted 5 lines
Quoted 1 line
At least I like that feature since it is available and won't miss it now. ;)
The feature isn't available if it doesn't work. It's not going to get fixed for 2.6.38 - it'll be 2.6.39 due to the timing and complexity of the problem.
I just wanted to express that I like that feature a lot and I didn't wanted to ask for a quick fix nor did I wanted to make pressure.
I've tried a the patch from Rabin Vincent (along with an alignment) and it works. It might not be the best solution and I certainly waste some memory (no free), but I don't care.
Regards,
Alexander
Russell King - ARM Linux a ?crit :
Quoted 11 lines · Rabin Vincent
Quoted 5 lines
It's possible to hack around this by placing the initramfs at the end of the kernel image rather than at the beginning with the rest of the init data. Something like the below should work, although you should also probably take care of alignment and also have this section freed when the rest of the init data is freed.
You're then running into problems as _sdata.._edata is copied to RAM on XIP kernels, and you really don't want to waste time copying the initramfs to RAM.
But in this case initramfs is after edata and before bss. So where is the problem ?
Matthieu
Quoted 8 lines · Matthieu CASTET
Russell King - ARM Linux a ?crit :
Quoted 7 lines · Rabin Vincent
Quoted 5 lines
It's possible to hack around this by placing the initramfs at the end of the kernel image rather than at the beginning with the rest of the init data. Something like the below should work, although you should also probably take care of alignment and also have this section freed when the rest of the init data is freed.
Quoted 5 lines
Quoted 4 lines
You're then running into problems as _sdata.._edata is copied to RAM on XIP kernels, and you really don't want to waste time copying the initramfs to RAM.
But in this case initramfs is after edata and before bss.
Right, so that isn't a concern.
Quoted 1 line
So where is the problem ?
There is a problem with it, as noted by Rabin. The initramfs will never be freed. Let's try to find a proper solution to this rather than everyone sticking their half-baked ideas at it. Let's see if we can rearrange the _entire_ layout in a way that sorts this out once and for all without making things even more complicated than they already are today.
We've already got enough variables to think about here without having yet more freeable sections scattered in the middle of the existing layout.
Quoted 11 lines · Alexander Holler
Am 10.01.2011 23:26, schrieb Nicolas Pitre:
Quoted 9 lines · Alexander Holler
Quoted 6 lines
I've build a kernel 2.6.37 for an armv5-device which includes an initramfs of
about 16MB (uncompressed, result is a kernel of about 6MB) and I now getting
errors when trying to load a module (which is included in that initramfs):Do you really need to have such a bit initramfs?
Quoted 8 lines
Not really, but it is handier to use than a separate file.
Quoted 3 lines
What you can do to work around this issue is to load your initramfs separately as a initrd ramdisk image. , and not combine it with the kernel image.
Point is that I would like to understand why including an initramfs has such consequences.
The layout of the kernel virtual memory goes like this:
0xbf000000 to 0xbfffffff: kernel module area0xc0008000 and above: kernel image
The module area is located just before the kernel memory because this allows function calls from modules to be linked directly without any indirect branches. For that to work, the branch destination has to be within 32MB of its origin location.
Now the kernel image has its __init section located at the beginning. It is located there because of some alignment restriction, and the initramfs data is located in the __init section. So if your initramfs is about 16MB, plus the size of the module area which is also 16MB you get a distance from a loaded module to the actual kernel core bigger than the possible branch fixup.
Nicolas