Single MIPS kernel
This question comes up every once in a while and I've also been approached during ELCE in Düsseldorf why there is no single MIPS kernel for all platforms, so I thought I should post a writeup on the topic.
The primary reason is that MIPS kernels are using non-PIC kernels. This means code is linked to a particular absolute address. The link address depends on the memory range available on a particular system's available memory range - there is no one size that fits all systems, not even a large fraction of supported systems.
What does it take to make kernels relocatable? A current kernel is not relocatable. One might do something along the lines of userland where the dynamic linker is in a similar situation and needs to first relocate itself before it can perform its actual job.
Two approaches. First keeping the non-PIC code. That requires keeping the entire relocation. A lasat_defconfig vmlinux is 5733098 bytes but built with --emit-relocs to keep the reloc information in the final binary the vmlinux file grows to 7217342 bytes! A quick look at the reloc sections:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 2] .rel.text REL 00000000 461538 0eedf8 08 34 1 4 [ 4] .rel__ex_table REL 00000000 550330 0040e0 08 34 3 4 [ 8] .rel.rodata REL 00000000 554410 0310e0 08 34 7 4 [10] .rel.pci_fixup REL 00000000 5854f0 000998 08 34 9 4 [12] .rel__ksymtab REL 00000000 585e88 00b3b0 08 34 11 4 [14] .rel__ksymtab_gpl REL 00000000 591238 007180 08 34 13 4 [17] .rel__param REL 00000000 5983b8 000858 08 34 16 4 [19] .rel__modver REL 00000000 598c10 000038 08 34 18 4 [21] .rel.data REL 00000000 598c48 00a130 08 34 20 4 [23] .rel.init.text REL 00000000 5a2d78 00f008 08 34 22 4 [25] .rel.init.data REL 00000000 5b1d80 001d08 08 34 24 4 [27] .rel.exit.text REL 00000000 5b3a88 000b78 08 34 26 4
The approach could probably be optimized but as a first order approximation this demonstrates there would be plenty of bloat to the binary. Positive side of this approach: no runtime penalty.
Alternatively: make the kernel PIC code. Over the thumb that'd going to inflate the kernel by 10 or 15%. Less than above approach but there'd also be significant runtime overhead. Probably nothing for a world where benchmarks like network performance on 64 byte packets decide on the fate of a product on the market.
Obviously there is the difference between 32 and 64 bit kernels. 64 bit uses additional instructions that are not available on 32 bit processors and using just 32 bit instructions won't fly on 64 bit kernels.
Hardware detection. That's all easy in a device tree world but in all reality many of the existing systems don't support device tree yet so a generic kernel would have to figure out what platform it's running on which would end up in something like an ISA style device probe.
Ralf
Hi Ralf,
Quoted 5 lines · Ralf Baechle
This question comes up every once in a while and I've also been approached during ELCE in Düsseldorf why there is no single MIPS kernel for all platforms, so I thought I should post a writeup on the topic.
for the SoCs supported by OpenWrt this is a no-go. we are already having a hard time fighting bloat. to get the images to fit we need to already build device specific images that only hold the DT, drivers, ... that a specific board needs. having a kernel that can boot on X devices wont even fit into flash and if it does there is not space left for the userland.
I think this feature is only interesting for the older platforms and the upcoming mobile SoC based in MIPS. i.e. the users are debian and android type device.
John
Quoted 9 lines · John Crispin
Hi Ralf,
Quoted 5 lines · Ralf Baechle
This question comes up every once in a while and I've also been approached during ELCE in Düsseldorf why there is no single MIPS kernel for all platforms, so I thought I should post a writeup on the topic.
Quoted 6 lines
for the SoCs supported by OpenWrt this is a no-go. we are already having a hard time fighting bloat. to get the images to fit we need to already build device specific images that only hold the DT, drivers, ... that a specific board needs. having a kernel that can boot on X devices wont even fit into flash and if it does there is not space left for the userland.
A multi-platform kernel should allow to compile in/out specific platforms (like what ARM does), such that we can still achieve a small kernel goal in OpenWrt.
Quoted 4 lines
I think this feature is only interesting for the older platforms and the upcoming mobile SoC based in MIPS. i.e. the users are debian and android type device.
You mean, older systems such as Sibyte, the SGI's IPxx and similar? Those may have memory architecture requirements (spaces.h) that make it difficult if possible to support.
--
Florian
Quoted 10 lines · Ralf Baechle
This question comes up every once in a while and I've also been approached during ELCE in Düsseldorf why there is no single MIPS kernel for all platforms, so I thought I should post a writeup on the topic.
The primary reason is that MIPS kernels are using non-PIC kernels. This means code is linked to a particular absolute address. The link address depends on the memory range available on a particular system's available memory range - there is no one size that fits all systems, not even a large fraction of supported systems.
Another reason is that the protocol between the bootloader and the kernel varies by platform. So you would have to have several different entry points, one for each booting protocol.
I am not sure how the bootloaders would know which entry point to use.
David Daney
Quoted 5 lines · David Daney
Another reason is that the protocol between the bootloader and the kernel varies by platform. So you would have to have several different entry points, one for each booting protocol.
I am not sure how the bootloaders would know which entry point to use.
That's where I foresaw the needs for the ISA style platform probe right at the kernel entry point before fanning out to a platform-specific entry point.
Since we already support compressed kernels I'm wondering if relocation might also be performed by the compression wrapper along with the hardware probe. That would leave the vmlinux itself untouched and the wrapper could be installed on the target.
It's just that all this for the sake of a unified kernel images seems way to painful!
Ralf
Quoted 6 lines · Ralf Baechle
Quoted 5 lines
Another reason is that the protocol between the bootloader and the kernel varies by platform. So you would have to have several different entry points, one for each booting protocol.
I am not sure how the bootloaders would know which entry point to use.
Quoted 8 lines
That's where I foresaw the needs for the ISA style platform probe right at the kernel entry point before fanning out to a platform-specific entry point.
Since we already support compressed kernels I'm wondering if relocation might also be performed by the compression wrapper along with the hardware probe. That would leave the vmlinux itself untouched and the wrapper could be installed on the target.
Wouldn't it make sense to make a unified kernel virtually mapped? That
would avoid the issue with RAM being present at different locations across systems and also if big pages were used, that I believe are available almost universally across the MIPS family, any performance hit would be minimal. There would be hardly any increase in the binary image size too. Run-time mappings such as kmalloc' or ioremap' could continue using unmapped segments.
Thoughts?
Maciej
Quoted 15 lines · Maciej W. Rozycki
Quoted 6 lines
Quoted 5 lines
Another reason is that the protocol between the bootloader and the kernel varies by platform. So you would have to have several different entry points, one for each booting protocol.
I am not sure how the bootloaders would know which entry point to use.
Quoted 8 lines
That's where I foresaw the needs for the ISA style platform probe right at the kernel entry point before fanning out to a platform-specific entry point.
Since we already support compressed kernels I'm wondering if relocation might also be performed by the compression wrapper along with the hardware probe. That would leave the vmlinux itself untouched and the wrapper could be installed on the target.
Quoted 7 lines
Wouldn't it make sense to make a unified kernel virtually mapped? That
would avoid the issue with RAM being present at different locations across systems and also if big pages were used, that I believe are available almost universally across the MIPS family, any performance hit would be minimal. There would be hardly any increase in the binary image size too. Run-time mappings such askmalloc' orioremap' could continue using unmapped segments.
I think some MIPS III CPUs were restricted to just 4MB max. page size. NEC VR4xxx I think. Still a pair would map 8MB which on the affected small memory systems should suffice. 16MB, 64MB are more typical sizes.
R3000 is a different kettle. To 4k or not to 4k is not a question ;-)
Now mapping the kernel alone wouldn't solve the security issue mentioned by David. The image would still lie around in KSEG0 / XKPHYS for whatever wants to run over so that should ideally also be a flexible address.
Otoh the mapped kernel certainly would have the lowest size overhead. I have faint memories of restrictions for TLB instructions or was it TLB exception handlers into mapped space, would have to do some rtfming on that topic.
Years ago I did test the impact of one less available TLB entry with lmbench; the loss was around 2%. That was on a CPU with 64 entries.
Ralf
Quoted 35 lines · Ralf Baechle
Quoted 15 lines · Maciej W. Rozycki
Quoted 6 lines
Quoted 5 lines
Another reason is that the protocol between the bootloader and the kernel varies by platform. So you would have to have several different entry points, one for each booting protocol.
I am not sure how the bootloaders would know which entry point to use.
Quoted 8 lines
That's where I foresaw the needs for the ISA style platform probe right at the kernel entry point before fanning out to a platform-specific entry point.
Since we already support compressed kernels I'm wondering if relocation might also be performed by the compression wrapper along with the hardware probe. That would leave the vmlinux itself untouched and the wrapper could be installed on the target.
Quoted 7 lines
Wouldn't it make sense to make a unified kernel virtually mapped? That
would avoid the issue with RAM being present at different locations across systems and also if big pages were used, that I believe are available almost universally across the MIPS family, any performance hit would be minimal. There would be hardly any increase in the binary image size too. Run-time mappings such askmalloc' orioremap' could continue using unmapped segments.I think some MIPS III CPUs were restricted to just 4MB max. page size. NEC VR4xxx I think. Still a pair would map 8MB which on the affected small memory systems should suffice. 16MB, 64MB are more typical sizes.
R3000 is a different kettle. To 4k or not to 4k is not a question ;-)
Now mapping the kernel alone wouldn't solve the security issue mentioned by David. The image would still lie around in KSEG0 / XKPHYS for whatever wants to run over so that should ideally also be a flexible address.
Quoted 8 lines
Otoh the mapped kernel certainly would have the lowest size overhead. I have faint memories of restrictions for TLB instructions or was it TLB exception handlers into mapped space, would have to do some rtfming on that topic.
Years ago I did test the impact of one less available TLB entry with lmbench; the loss was around 2%. That was on a CPU with 64 entries.
We have a private patch that does exactly this, the main motivation was to place the kernel in the same virtual address 256MB region as the modules, so that a direct calling sequence can be used in modules.
The resulting module code is much faster, so depending on the work load it may be a performance win. We see things like IPv6 forwarding improving something like 6% when IPv6 is built as a module.
Also we have many more TLB entries (128, or 256) so losing one is not a big deal.
David Daney
Hi,
Quoted 9 lines · Ralf Baechle
Quoted 7 lines · Maciej W. Rozycki
Wouldn't it make sense to make a unified kernel virtually mapped? That
would avoid the issue with RAM being present at different locations across systems and also if big pages were used, that I believe are available almost universally across the MIPS family, any performance hit would be minimal. There would be hardly any increase in the binary image size too. Run-time mappings such askmalloc' orioremap' could continue using unmapped segments.
Quoted 4 lines
Otoh the mapped kernel certainly would have the lowest size overhead. I have faint memories of restrictions for TLB instructions or was it TLB exception handlers into mapped space, would have to do some rtfming on that topic.
Yeh, KVM puts all tlb handling in arch/mips/kvm/tlb.c, which is built statically rather than being included in the kvm kernel module, exactly for this reason, so that it resides in unmapped memory space.
You'd have to guarantee not to get a TLB exception while the TLB registers contain important values, since they'll get clobbered by the taking of the exception itself (e.g. EntryHi gets set to failing address, EntryLo* undefined), or the TLB entry pointed to by CP0_Index may be replaced.
There's always CP0_Wired - its use in the kernel is a bit of a mess atm IIRC.
Cheers
James
Quoted 21 lines · James Hogan
Hi,
Quoted 9 lines · Ralf Baechle
Quoted 7 lines · Maciej W. Rozycki
Wouldn't it make sense to make a unified kernel virtually mapped? That
would avoid the issue with RAM being present at different locations across systems and also if big pages were used, that I believe are available almost universally across the MIPS family, any performance hit would be minimal. There would be hardly any increase in the binary image size too. Run-time mappings such askmalloc' orioremap' could continue using unmapped segments.Quoted 4 lines
Otoh the mapped kernel certainly would have the lowest size overhead. I have faint memories of restrictions for TLB instructions or was it TLB exception handlers into mapped space, would have to do some rtfming on that topic.
Yeh, KVM puts all tlb handling in
arch/mips/kvm/tlb.c, which is built statically rather than being included in the kvm kernel module, exactly for this reason, so that it resides in unmapped memory space.
Quoted 9 lines
You'd have to guarantee not to get a TLB exception while the TLB registers contain important values, since they'll get clobbered by the taking of the exception itself (e.g. EntryHi gets set to failing address, EntryLo* undefined), or the TLB entry pointed to by
CP0_Indexmay be replaced.There's always
CP0_Wired- its use in the kernel is a bit of a mess atm IIRC.
The current kernel.org kernel respects CP0_Wired. We use a single TLB entry (index 0) to map the entire kernel, and set CP0_Wired accordingly.
Everything works.
EBase still points to an unmapped address, so exception handlers still work as before, except they may have to use more code to directly call into the kernel due to the 256MB jump range thing. The general exception handler looks up the addresses in a table to that is not effected, only if you have a dedicated interrupt vector, do you have to use an indirect jump to reach the kernel.
David Daney
Quoted 9 lines · Ralf Baechle
This question comes up every once in a while and I've also been approached during ELCE in Düsseldorf why there is no single MIPS kernel for all platforms, so I thought I should post a writeup on the topic.
The primary reason is that MIPS kernels are using non-PIC kernels. This means code is linked to a particular absolute address. The link address depends on the memory range available on a particular system's available memory range - there is no one size that fits all systems, not even a large fraction of supported systems.
There is another reason to have a relocatable kernel: The security people are starting to demand it so that they can randomize the load address.
Quoted 27 lines
What does it take to make kernels relocatable? A current kernel is not relocatable. One might do something along the lines of userland where the dynamic linker is in a similar situation and needs to first relocate itself before it can perform its actual job.
Two approaches. First keeping the non-PIC code. That requires keeping the entire relocation. A
lasat_defconfigvmlinux is 5733098 bytes but built with --emit-relocs to keep the reloc information in the final binary the vmlinux file grows to 7217342 bytes! A quick look at the reloc sections:Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 2] .rel.text REL 00000000 461538 0eedf8 08 34 1 4 [ 4] .rel__ex_table REL 00000000 550330 0040e0 08 34 3 4 [ 8] .rel.rodata REL 00000000 554410 0310e0 08 34 7 4 [10] .rel.pci_fixup REL 00000000 5854f0 000998 08 34 9 4 [12] .rel__ksymtab REL 00000000 585e88 00b3b0 08 34 11 4 [14] .rel__ksymtab_gpl REL 00000000 591238 007180 08 34 13 4 [17] .rel__param REL 00000000 5983b8 000858 08 34 16 4 [19] .rel__modver REL 00000000 598c10 000038 08 34 18 4 [21] .rel.data REL 00000000 598c48 00a130 08 34 20 4 [23] .rel.init.text REL 00000000 5a2d78 00f008 08 34 22 4 [25] .rel.init.data REL 00000000 5b1d80 001d08 08 34 24 4 [27] .rel.exit.text REL 00000000 5b3a88 000b78 08 34 26 4
Quoted 4 lines
The approach could probably be optimized but as a first order approximation this demonstrates there would be plenty of bloat to the binary. Positive side of this approach: no runtime penalty.
This is the approach I was thinking of taking. There would be a small PIC wrapper that applied the relocations, and then passed control to the real entry point.
We would have to be careful of the ex_table, as that is now sorted at build time. For that, we could go to the scheme used by x86, and have that addresses in the ex_table be relative, build time sorting is already working for x86 relocatable kernels.
David Daney.
Quoted 2 lines · David Daney
There is another reason to have a relocatable kernel: The security people are starting to demand it so that they can randomize the load address.
That may work for some platforms - but in the MIPS world we still have to deal with very claustrophobic systems which barely leave any space to move a kernel around.
Quoted 8 lines
This is the approach I was thinking of taking. There would be a small PIC wrapper that applied the relocations, and then passed control to the real entry point.
We would have to be careful of the
ex_table, as that is now sorted at build time. For that, we could go to the scheme used by x86, and have that addresses in theex_tablebe relative, build time sorting is already working for x86 relocatable kernels.
That's probably more of an implementation detail. I'm more concerned about the overall bloat. I think many embedded users are so addivted to benchmark results that this going to make or break the whole scheme.
Ralf
Quoted 13 lines · Ralf Baechle
Quoted 2 lines · David Daney
There is another reason to have a relocatable kernel: The security people are starting to demand it so that they can randomize the load address.
That may work for some platforms - but in the MIPS world we still have to deal with very claustrophobic systems which barely leave any space to move a kernel around.
Quoted 4 lines
This is the approach I was thinking of taking. There would be a small PIC wrapper that applied the relocations, and then passed control to the real entry point.
Quoted 8 lines
Quoted 4 lines
We would have to be careful of the
ex_table, as that is now sorted at build time. For that, we could go to the scheme used by x86, and have that addresses in theex_tablebe relative, build time sorting is already working for x86 relocatable kernels.That's probably more of an implementation detail. I'm more concerned about the overall bloat. I think many embedded users are so addivted to benchmark results that this going to make or break the whole scheme.
If you can make relocation a configuration option (as on x86), it would allow distributions to build multiplatform kernels without preventing embedded users from building a kernel optimised for their specific system. But I know very little about MIPS or how intrusive the changes for relocation would have to be. Perhaps it would be too much of a maintenance burden to make this an option.
Ben.
Signature · 2 lines
Ben Hutchings For every action, there is an equal and opposite criticism. - Harrison
Quoted 11 lines · Ben Hutchings
Quoted 4 lines
That's probably more of an implementation detail. I'm more concerned about the overall bloat. I think many embedded users are so addivted to benchmark results that this going to make or break the whole scheme.
If you can make relocation a configuration option (as on x86), it would allow distributions to build multiplatform kernels without preventing embedded users from building a kernel optimised for their specific system. But I know very little about MIPS or how intrusive the changes for relocation would have to be. Perhaps it would be too much of a maintenance burden to make this an option.
The scope of the changes is relativly limited - we're much more concerned about the impact on binary size, memory size or performance of the various approaches under discussion.
I wonder kernels for which platforms would Debian want to unify?
Ralf
Quoted 14 lines · Ralf Baechle
Quoted 11 lines · Ben Hutchings
Quoted 4 lines
That's probably more of an implementation detail. I'm more concerned about the overall bloat. I think many embedded users are so addivted to benchmark results that this going to make or break the whole scheme.
If you can make relocation a configuration option (as on x86), it would allow distributions to build multiplatform kernels without preventing embedded users from building a kernel optimised for their specific system. But I know very little about MIPS or how intrusive the changes for relocation would have to be. Perhaps it would be too much of a maintenance burden to make this an option.
Quoted 5 lines
The scope of the changes is relativly limited - we're much more concerned about the impact on binary size, memory size or performance of the various approaches under discussion.
I wonder kernels for which platforms would Debian want to unify?
I don't have high expectations for being able to unify those we currently support. Realistically, I expect that most development effort will go into new platforms. (What we saw with ARM was that multi-platform was implemented for most ARMv7 platforms (for which we now need only 2 configurations) but only slowly for older chips (4 configurations, and that's after dropping 2 platforms).)
Anyway, we have one 32-bit configuration for each byte order (4kc-malta), and the following 64-bit configurations:
[big-endian]
r4k-ip22: CONFIG_SGI_IP22, CONFIG_CPU_R4X00 r5k-ip32: CONFIG_SGI_IP32, CONFIG_CPU_R5000 sb1-bcm91250a: CONFIG_SIBYTE_SWARM, CONFIG_CPU_SB1
5kc-malta: CONFIG_MIPS_MALTA, CONFIG_CPU_MIPS64_R1
octeon: CONFIG_CAVIUM_OCTEON_SOC
[little-endian]
sb1-bcm91250a: CONFIG_SIBYTE_SWARM, CONFIG_CPU_SB1
5kc-malta: CONFIG_MIPS_MALTA, CONFIG_CPU_MIPS64_R1 loongson-2e: CONFIG_MACH_LOONGSON, CONFIG_LEMOTE_FULOONG2E loongson-2f: CONFIG_MACH_LOONGSON, CONFIG_LEMOTE_MACH2F loongson-3: CONFIG_MACH_LOONGSON, CONFIG_LOONGSON_MACH3X
In general, I want our kernel packages to support any hardware that is or has been generally available to buy, that can feasibly run a general purpose distribution. I'm somewhat hopeful that Prpl members will be introducing new platforms that fit this description in the near future.
But I also want the packages to build natively in a few hours on each architecture. Currently, it takes about 17 hours on little-endian (Loongson 3A, quad-core) and longer on big-endian (Octeon v0.3, 6 cores used). So I can't accept a further increase in the number of configurations as new MIPS platforms appear. Without multi-platform support, we will have to drop one platform for each one we add, so we'll have to be quite picky about adding them.
Ben.
Signature · 3 lines
Ben Hutchings Q. Which is the greater problem in the world today, ignorance or apathy? A. I don't know and I couldn't care less.
Quoted 28 lines · Ben Hutchings
Quoted 14 lines · Ralf Baechle
Quoted 11 lines · Ben Hutchings
Quoted 4 lines
That's probably more of an implementation detail. I'm more concerned about the overall bloat. I think many embedded users are so addivted to benchmark results that this going to make or break the whole scheme.
If you can make relocation a configuration option (as on x86), it would allow distributions to build multiplatform kernels without preventing embedded users from building a kernel optimised for their specific system. But I know very little about MIPS or how intrusive the changes for relocation would have to be. Perhaps it would be too much of a maintenance burden to make this an option.
Quoted 5 lines
The scope of the changes is relativly limited - we're much more concerned about the impact on binary size, memory size or performance of the various approaches under discussion.
I wonder kernels for which platforms would Debian want to unify?
I don't have high expectations for being able to unify those we currently support. Realistically, I expect that most development effort will go into new platforms. (What we saw with ARM was that multi-platform was implemented for most ARMv7 platforms (for which we now need only 2 configurations) but only slowly for older chips (4 configurations, and that's after dropping 2 platforms).)
Quoted 6 lines
Anyway, we have one 32-bit configuration for each byte order (4kc-malta), and the following 64-bit configurations:
[big-endian]
r4k-ip22: CONFIG_SGI_IP22, CONFIG_CPU_R4X00 r5k-ip32: CONFIG_SGI_IP32, CONFIG_CPU_R5000
As far as I know, IRIX includes kernels specific to each SGI system (IPxx), but it seems they're CPU agnostic. They are relocatable, though. Been awhile since I watched sash boot followed by an IRIX kernel, but it does 3-4 relocations before finally booting. So a relocatable MIPS kernel on the SGI platforms seems possible. Probably requires arcane knowledge of ARCS, though.
Bootloader-wise, Stan's 'arcload' can handle booting multiple kernels across various SGI platforms. We used it on the Gentoo SGI LiveCD back in 2006 to create a single CD that could boot on IP22, IP27, IP30, & IP32, using different kernels for each system and CPU (I think there was one volume header slot left at the end for arcload itself).
Signature · 9 lines
Joshua Kinard Gentoo/MIPS [email protected] 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic