Merge strategy for klibc
Okay, as of this point, I think klibc is in quite good shape; my testing so far is showing that it can be used as a drop-in replacement for the kernel root-mounting code.
That being said, there is guaranteed to be breakage, for two reasons:
a. There are several architectures which don't have klibc ports yet.
Since I don't have access to them, I can't really do them, either. It's usually a matter of an afternoon or less to port klibc to a new architecture, though, if you have a working development environment for it.
b. There are a gajillion options in the way Linux handles its root
filesystem. I have tried to implement them all, but I haven't been able to test some of the more exotic conditions, mostly due to the fact that every boot environment I set up takes a *lot* of time and maintenance. This has in fact been the by far the bulk of the time I've spent on klibc, not writing code.
Thus, it's not clear to me what particular approach makes most sense for pushing upstream.
1. Mechanism: the easiest for me would of course be git pull, but I'm certainly willing to break it up into patches or any other useful way.
A diff of by git tree from 2.6.16 gives:
766 files changed, 84024 insertions(+), 3059 deletions(-)
2. State: the current git tree cleans up the arch-specific code for i386 and x86-64; other architectures are going to need some minor cleanup in setup.c or the equvalent.
The current git tree includes a number of utilities, like dash (sh), which aren't used by the default kinit configuration. Additionally, right now kinit is built monolitically, in other words there isn't a CONFIG_ option to turn off nfsmount, for example.
Again, I'm more than willing to put the tree in any particular state that makes sense.
3. Path: it probably would make sense to push this into -mm first?
It's taken this long, I'd like to make this actually happen...
-hpa
Quoted 5 lines · H. Peter Anvin
a. There are several architectures which don't have klibc ports yet.
Since I don't have access to them, I can't really do them, either. It's usually a matter of an afternoon or less to port klibc to a new architecture, though, if you have a working development environment for it.
I haven't yet looked at your code, but what actually needs to be done? Defining syscall macros?
I'd probably give it a try for VAX.
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O für einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
Quoted 6 lines · Jan-Benedict Glaw
I haven't yet looked at your code, but what actually needs to be done? Defining syscall macros?
I'd probably give it a try for VAX.
Each architecture needs an assembly stub generator for most syscalls, a crt0.S, plus adjustments for which syscalls are "special" (as in special ed) on that architecture. There are also a couple of header files which need per-architecture configuration information. All in all it is usually a few hours worth of work, no more.
-hpa
Quoted 11 lines · H. Peter Anvin
Okay, as of this point, I think klibc is in quite good shape; my testing so far is showing that it can be used as a drop-in replacement for the kernel root-mounting code.
That being said, there is guaranteed to be breakage, for two reasons:
a. There are several architectures which don't have klibc ports yet.
Since I don't have access to them, I can't really do them, either. It's usually a matter of an afternoon or less to port klibc to a new architecture, though, if you have a working development environment for it.
Which ones?
Quoted 8 lines · Al Viro
Quoted 6 lines · H. Peter Anvin
Okay, as of this point, I think klibc is in quite good shape; my testing so far is showing that it can be used as a drop-in replacement for the kernel root-mounting code.
That being said, there is guaranteed to be breakage, for two reasons:
Quoted 8 lines
Quoted 5 lines
a. There are several architectures which don't have klibc ports yet.
Since I don't have access to them, I can't really do them, either. It's usually a matter of an afternoon or less to port klibc to a new architecture, though, if you have a working development environment for it.Which ones?
This is the status of architectures in klibc, at least as far as I know.
Note that 64-bit architectures which have 32-bit fallback modes (e.g. MIPS) can use the 32-bit klibc if applicable.
alpha: Working arm-thumb: Untested arm: Working arm26: Not yet ported cris: Working h8300: Not yet ported i386: Working ia64: Working m32r: Untested m68k: Not yet ported mips: Working mips64: Not yet ported parisc: Working parisc64: Not yet ported ppc: Working ppc64: Working s390: Working static, shared untested s390x: Working sh: Untested sh64: Not yet ported sparc: Working sparc64: Untested v850: Not yet ported x86-64: Working xtensa: Not yet ported
-hpa
Hi,
first thanks for your good work on that!
Quoted 2 lines · H. Peter Anvin
The current git tree includes a number of utilities, like dash (sh), which aren't used by the default kinit configuration.
But its quite useful to be dropped into a shell, if anything goes wrong. e.g. ubuntu assumes an modular kernel, which I don't like to have and I got dropped into the shell quite often until all scripts worked as expected.
Quoted 3 lines
Additionally,
right now kinit is built monolitically, in other words there isn't aCONFIG_option to turn off nfsmount, for example.
Yes, since after development of your init setup, you want to shrink it as much as possible :-)
But all of this stuff is janitorial and can be done after merging the basic ideas and completing the setup code move to user space.
Another idea is to still allow the current archive setups for initramfs, because a static /dev might be best for embedded stuff where you basically pre-compose a system image and download that to flash.
Quoted 1 line
3. Path: it probably would make sense to push this into -mm first?
The usr/ part should go in ASAP, but please rip out the setup stuff only after a grace period announced in Documentation/feature-removal.txt or sth. like that.
Regards
Ingo Oeser
Hi,
On Mon, 20 Mar 2006, H. Peter Anvin wrote:
You forgot to provide any information (at least a summary) about what this is and how this will work. Please don't assume everyone is familiar with it.
There is one major question: how will this interface to distributions?
How can distributions add their own initializations and configurations or are they going to put an initrd on top of the kernel initrd? If this will have a kernel and a distribution part, it poses the question whether klibc has to be distributed with the kernel at all (a libc has a standard API after all) and the kernel just provides the kernel specific parts to whatever the distribution provides.
bye, Roman
Followup to: <[email protected]> By author: Roman Zippel <[email protected]> In newsgroup: linux.dev.kernel
Quoted 5 lines
You forgot to provide any information (at least a summary) about what this is and how this will work. Please don't assume everyone is familiar with it.
Quoted 9 lines
There is one major question: how will this interface to distributions?
How can distributions add their own initializations and configurations or are they going to put an initrd on top of the kernel initrd? If this will have a kernel and a distribution part, it poses the question whether klibc has to be distributed with the kernel at all (a libc has a standard API after all) and the kernel just provides the kernel specific parts to whatever the distribution provides.
Okay... quick summary (again)...
klibc is a small libc, small enough that it provides negible (or even negative) overhead to bundle it inside the kernel binary.
The kernel tree part is there so that we can rip out in-kernel code without breaking compatibility, or requiring a distribution-provided initramfs. In the future, we could consider retaining certain binaries in the rootfs and have "on-demand userspace" by the kernel, e.g. to do partition enumeration in userspace in a backwards-compatible manner.
The default build provides a single binary called kinit, which is (modulo any bugs) equivalent to the in-kernel root-mounting code, with all its variants (initrd, nfsroot, load ramdisk from floppy, yadda yadda.) The existence of kinit allows the in-kernel code to be removed without actually removing a feature. Hence, the reason to put this in the kernel tree is to make sure there is zero impact on distributions.
If the distribution uses initramfs directly, kinit goes unused. The klibc code is also available as a standalone distribution, which at least Ubuntu is currently using to build a custom initramfs. Because the kinit code is still userspace, it can share considerable amounts of code with the standalone klibc utilities collection; in fact most of the kinit pieces are available as standalone binaries which can be weaved together by scripts or other C code.
The advantages of moving this code to userspace, thus is:
- Simpler programming model (harder to screw up)
- Easier to share code with distribution-customized setups
- Code can be tested as standalone userspace binaries at runtime
A lot of the benefit is lost if, like now, there is a piece of code which has to be written for kernel-mode programming, separate from anything else and not testable except through a tedious kernel boot cycle.
-hpa
Quoted 52 lines · H. Peter Anvin
Followup to: <[email protected]> By author: Roman Zippel <[email protected]> In newsgroup: linux.dev.kernelQuoted 5 lines
You forgot to provide any information (at least a summary) about what this is and how this will work. Please don't assume everyone is familiar with it.
Quoted 9 lines
There is one major question: how will this interface to distributions?
How can distributions add their own initializations and configurations or are they going to put an initrd on top of the kernel initrd? If this will have a kernel and a distribution part, it poses the question whether klibc has to be distributed with the kernel at all (a libc has a standard API after all) and the kernel just provides the kernel specific parts to whatever the distribution provides.
Okay... quick summary (again)...
klibc is a small libc, small enough that it provides negible (or even negative) overhead to bundle it inside the kernel binary.
The kernel tree part is there so that we can rip out in-kernel code without breaking compatibility, or requiring a distribution-provided initramfs. In the future, we could consider retaining certain binaries in the rootfs and have "on-demand userspace" by the kernel, e.g. to do partition enumeration in userspace in a backwards-compatible manner.
The default build provides a single binary called kinit, which is (modulo any bugs) equivalent to the in-kernel root-mounting code, with all its variants (initrd, nfsroot, load ramdisk from floppy, yadda yadda.) The existence of kinit allows the in-kernel code to be removed without actually removing a feature. Hence, the reason to put this in the kernel tree is to make sure there is zero impact on distributions.
If the distribution uses initramfs directly, kinit goes unused. The klibc code is also available as a standalone distribution, which at least Ubuntu is currently using to build a custom initramfs. Because the kinit code is still userspace, it can share considerable amounts of code with the standalone klibc utilities collection; in fact most of the kinit pieces are available as standalone binaries which can be weaved together by scripts or other C code.
The advantages of moving this code to userspace, thus is:
- Simpler programming model (harder to screw up)
- Easier to share code with distribution-customized setups
- Code can be tested as standalone userspace binaries at runtime
Quoted 6 lines
A lot of the benefit is lost if, like now, there is a piece of code which has to be written for kernel-mode programming, separate from anything else and not testable except through a tedious kernel boot cycle.
-hpa
ISTM that we are (finally? ;) moving piece by piece to a mixed monolothic+microkenel, or rather that many of the things that were first implemented in-kernel (on linux or other unixes) are being slowly jettisoned to a kernel-provided userspace. All in all this is a good step forward :)
Regarding helping test/develop this, is there any small distro already using klibc for such purposes? Maybe you, hpa, could share you klibc testing rig? This looks ripe for a qemu-based testing at the moment, not being performance critical like many other current developments...
--
Greetz, Antonio Vargas aka winden of network
http://wind.codepixel.com/
[email protected]
[email protected]
Every day, every year
you have to work
you have to study
you have to scene.
Quoted 6 lines · Antonio Vargas
Regarding helping test/develop this, is there any small distro already using klibc for such purposes? Maybe you, hpa, could share you klibc testing rig? This looks ripe for a qemu-based testing at the moment, not being performance critical like many other current developments...
The klibc-integrated git tree is at:
git://git.kernel.org/pub/scm/linux/kernel/git/hpa/linux-2.6-klibc.git
At this point, the big thing is to drop this into as many weirdly-booting Linux setups as possible, and watch them hopefully have no effect whatsoever :)
For porting klibc, I have a set of tests which are also included. Unfortunately, they're not currently self-validating, which would be a significant enhancement.
-hpa
Quoted 3 lines · H. Peter Anvin
Okay, as of this point, I think klibc is in quite good shape; my testing so far is showing that it can be used as a drop-in replacement for the kernel root-mounting code.
[]
Quoted 2 lines
Thus, it's not clear to me what particular approach makes most sense for pushing upstream.
Why this needs to be "pushed" upstream in the first place? Isn't it simpler/easier/whatever to just require klibc to be present on the build system for kernel? If klibc is "sufficiently" independent of the kernel (is it? I see no reason it shouldn't), why it should go with kernel? Just point your CONFIG_INITRAMFS_SOURCE to some klibc directory tree and be done with it, no need to distribute/build klibc with kernel..
Thanks.
/mjt