klibc requirements, round 2

Hi,

Ok, now that we have a general idea that people are going to want to stick lots of different types of programs into the initramfs, does this change any of the initial requirement list for klibc that I sent out?

To summarize, here's a partial list of the programs people want to run:
- mount
- hotplug
- busybox
- dhcpcd
- image viewer
- mkreiserfs
- partition discovery (currently in the kernel)
- lots of other, existing in kernel code.

So on to the requirements:
1) portable
This is still true.

dietlibc and uClibc currently do not cover the
range of platforms that this code must run on. For either of them to be used, they must be worked on.

2) tiny
Still true.

For some people, glibc will be acceptable. For the rest of us, we want something a bit smaller :) Both uClibc and dietlibc fit this requirement.

3) dynamic version available
Probably not true anymore.
In talking with lots of people, and playing with the dynamic linking capabilities of different libraries, I don't think this is worth having as a requirement for this kind of library.

4) not suck
Still true :)

So, what's the available options to try to meet these requirements?

Here's some proposed solutions:
- dietlibc / uClibc
Nice options. Both of these libraries are already fairly complete. One drawback is they are not portable to all platforms. With some work, this can probably be solved.

- klibc
Portability can be achieved through using the kernel unistd.h file for the syscall logic, and having a very small _start function written. For an example of this kind of code, see the initramfs patches from Al Viro on his ftp site:
ftp://ftp.math.psu.edu/pub/viro/
This would involve writing/porting a lot of the basic library functions. They could be copied from the existing libc implementations, but this would be a separate project, requiring maintenance over time, and people willing to do the work.

Is this all a good summary? Any other comments from people?

How about responses from the dietlibc and uClibc people on the odds of them being able to port to the remaining platforms?

In the meantime, I'll go off and play with klibc, and see if I can get some portability based off of Al's example code. If anyone is interested, the code can be found at:

	http://linuxusb.bkbits.net:8088/klibc

thanks,

greg k-h

FWIW we (at Axis) run both uClibc and glibc for embedded Linux products, and both work fine but obviously with different target compromises wrgds to memory footprints, speed of porting etc. uClibc works fine with a majority of programs relevant to us, not just a small list of useful tools.

We have an arch/cris port of uClibc, which could be merged with the official branch at some point.

uClibc seem to work fine shared, that's how I've ran it for a while now. When doing a "complete" linux system in 2 MB flash to put in a product, every shared byte possible is worth to put some time on :) Granted, if the goal is just to compile 10 small tools it might not be very useful but since it's not that difficult to support it, and you're making yet another libc, why not keep it flexible.

I'm sorry I must have missed why yet another libc was needed in the first place :)

/BW

Please see these threads:

	http://marc.theaimsgroup.com/?t=101045445500001
	http://marc.theaimsgroup.com/?t=101051901100018

and other initramfs threads recently on this mailing list.

thanks,

greg k-h

[snip]

I think these are examples of misunderstanding what initramfs _can do_ with what we (might) need a klibc to do. What we need a klibc for is the lots of other, existing in kernel code, and related things which we can't do right now, but could do (IP PNP over pcmcia wasn't possible when I tried it ages ago, but should be when 2.5.x is over hopefully). These programs _might_ compile with a klibc, but I wouldn't worry about it. uClibc is what should be used for many of these custom applications of initramfs. And any of the kernel code pulled out into userland should end up being able to compile on any libc, since we'll need it in the final root, so it would probably be in the real roots /sbin too.

Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Thus spake Greg KH ([email protected]):

Portability should be achieved by using unified syscalls, which both the diet libc and uClibc now use. That saves space over the unistd.h approach.

I think I can speak for both Erik and myself when I say that we don't hate architectures and because of that don't support them. If we get a chance (and maybe a little help from someone who knows those platform), we will port our libc to that platform.

Sadly, I don't have the deep pockets to buy myself a hardware lab with a VAX to port my libc to it. So I (and Erik, too, obviously) would need at least an account on one of those boxes, with gcc, binutils, strace and gdb installed.

In my eyes that is a waste of time, really.
But it's your time, so don't let that stand in your way ;)

Felix

I would personally rather see dietlibc / uClibc get ported to the remaining platforms and have them become fairly standard options for staticly linking small programs rather then spend the effort to create a klibc and then make it portable as well.

I would rather have the people looking at making a small, optimized libc work on one codebase instead of scattering their efforts amoung several :-)

while the dietlibc / uClibc may not be quite as small and optimized as a klibc would be the fact that people are currently able to survive with glibc indicates that the space issue probably isn't _that_ high a priority (if klibc produces a 100K binary while dietlibc produces a 120K binary and glibc produces a 2M binary the difference in space between klibc and dietlibc really doesn't matter much :-)

David Lang

Fully agreed. Porting libc (diet or uClibc) is an issue of hardware access, access to the instruction set docs for the arch, access to a gnu toolchain, and (the biggest issue) an issue of time and motivation.

I agree here. dietlibc is GPL. uClibc is LGPL. I think they both address the problem space pretty well. Felix and I are both willing to accept patches.

Lets look at it the other way... Suppose you start making a separate klibc. You skip/eliminate a ton of stuff and next week someone complains that it's missing, say, the pivot_root syscall. So you add it. Then the week after, someone complains that you are missing varargs. So you add that too. Pretty soon, someone will complain about how printf feature foo is missing, and they just _need_ SuS2 wordexp compatibility, etc, etc. Trust me when I say you are on a very slippery slope, with a awful lot of redundant work ahead of you. If you want to pursue it, thats fine, and I'll even help you a bit. But I think that soon enough, you will converge on what dietlibc and uClibc already have.

-Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

There are several VAX emulators with varying degrees of b0rkenness. At least one of them manages to boot NetBSD...

... at which point you tell them to bugger off. If they refuse - man procmailrc. Problem solved.

Thus spake Alexander Viro ([email protected]):

I guess the point here is that you don't save anything over the diet libc. I went to great lengths to make sure not using printf would not incur any overhead. So if klibc is diet libc without printf, then using klibc won't produce smaller binaries than using diet libc (unless you use printf, of course, in which case klibc won't work at all).

By the way: one more architecture is now supported by the diet libc:
hp pa-risc. Someone surprised me by sending a patch.

Felix

Hi

greg> To summarize, here's a partial list of the programs people want to run:
greg> - mount
greg> - hotplug
greg> - busybox
greg> - dhcpcd
greg> - image viewer
greg> - mkreiserfs
greg> - partition discovery (currently in the kernel)
greg> - lots of other, existing in kernel code.

I still think that fsck at this point will be great. You will minimize the need to have the kernel special case for fsck the root fs with respect to the rest of fs.

Later, Juan.

In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

I _really_ doubt it. /init calling mount(2) - sure, but mount(8)?
Not really.

Assuming that it moves to userland, which is non-obvious. Going other way (i.e. clean kernel API for parsing _and_ changing partitioning) might be better. Current mechanisms for notifying kernel about partition table changes are really gross.

nfsroot mounting (socket/sendto/recvfrom/open/close/read/write/malloc/free/ snprintf/memcpy/strlen/mount).

_Kernel_ has no special-casing of that kind.

Note: said patches are against relatively old 2.4 trees. They demonstrate that quite a few things can be moved to userland, however it's mostly proof-of-concept stuff. In particular
* in the final variant we let the loader to leave archive(s)
in core, rather than linking them into the kernel image.
* nfsroot.c is *crap*
* quite a few things are currently done better (modulo moving
init/do_mounts.c code in userland process). IOW, reduction to syscalls could be (and already is) done better than in these patches.

I'll put current version of that stuff for public testing, but that will happen after the current mess with kdev_t/struct block_device * will be resolved...

The development sources for e2fsprogs will already work with diet libc. Unfortunately, diet libc doesn't do shared libraries, so resulting binaries are sufficiently big that I doubt they would be interesting for initrd and rescue floppy applications (which is why I tried the experiment in the first place).

In any case, given that e2fsprogs is already portable to NetBSD and Solaris (the latter so I can run purify to catch memory errors), it shouldn't be particularly difficult to get e2fsprogs to run on some other alternative libc.

- Ted

Let's separate out the rescue disk and normal boot scenarios here. Obviously the ideal for a rescue disk is to include as much useful stuff as physically possible and that's not something we should be concerning ourselves with for normal boot.

For normal boot (which is what we are talking about, as all booting is moving to initramfs), we want enough code to get to the point where we mount root read-only (no fsck!) and that means enough to load modules and possibly get NFS up. And whatever else that's already in the kernel simply for boot that can be moved to userspace. Nothing else.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Actually, the whole point of Juan's suggestion was that you _don't_ want to fsck a filesystem that is currently mounted. There is always a potential problem that fsck will change the on-disk data of the filesystem in a way that is not coherent with what the kernel has in-memory, which should force a system reboot before continuing (which most initscripts don't do). For ext2/ext3 this may be relatively safe (data/metadata don't move around much), but reiserfsck cannot (or will not) fsck a mounted filesystem at all.

It would be interesting to see if e2fsck could be linked into busybox or another multi-call binary to avoid the overhead of the static libc linking. I don't think there are any technical obstacles, just a matter of someone sitting down and doing it. It would still be pretty big (150kB or so).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Interesting point. Modulo any existing LVM brokenness, we can do this with a read-only snapshot and pivot_root afterwards. Alternately, a read-only /bootsupport or something of the sort which contains *fsck. What we don't want is initramfs to get big.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Err, you think putting the necessary LVM tools in initramfs (vgscan, vgchange, lvcreate, liblvm) will be _smaller_ than e2fsck??? Your "modulo" is also a very big one - I'd rather trust e2fsck than LVM in my boot environment any day.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

No, I forgot about that dependency entirely. Doh.

Fair enough. The deeper point is that the purpose of initramfs is to move stuff out of the kernel in to userland. Ergo, this all becomes a non-kernel issue. We do not want to be in the business here of packaging things into the ramfs archives, we rather want to give external tools and distros all the info they need to make intelligent choices about how to make the kernel bootable.

Let's just try to focus on what we're taking out of the kernel in this process and not on all the nifty stuff that can now be added to the initial boot process.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

The deeper issue is what portion of initramfs changes with each minor kernel version, and what portion stays static through a major release.

The volatile stuff (like the module tree) we will have to supply. The static stuff (like the /init script, shell to run it if it's not c code, dhcp client, root filesystem determination, nfs mount, etc) can live in its own directory as long as there is a clean well-defined API that it and the kernel can talk to each other through. It doesn't even have to get rebuilt, just scooped up into the archive.

When initramfs is required to boot, we're going to have to supply at least a skeleton version of it in the kernel tarball. But if the distributions supply their own uncompressed initramfs tree in a well-known location (something like /usr/src/initramfs, perhaps) then all the kernel build has to do is cpio that up, add on the volatile stuff (module directory, etc), and hand that archive and the bzImage off to whatever the install script is that copies them to /boot and updates the bootloader if necessary.

If /usr/src/initramfs isn't there, then the build process should use the minimalist skeleton version in the kernel tarball instead. (The skeleton's purpose would mostly as example code with the absolute minimum required to boot a fairly straightforward system, but it should work and the only way to really know that is to build it and try it.) It skeleton would largely serve as a baseline for distributors to add to, containing the common code they can't really avoid having.

So when we're breaking stuff out of the kernel and putting it into initramfs, we move the code it into the skeleton initramfs directory, and define a clean API that it and the kernel can talk to each other through. Then later, stuff can be dropped from the skeleton directory if we decide it doesn't need to live in the kernel tarball (things like klibc which have more than one implementation, or a micro dhcp client that not everybody really needs in order to boot).

So the first question is, what parts of initramfs are going to change with each new kernel version (I.E. the module tree), and what stuff is going to stay pretty stable between major releases (the /init script?) Second question, what are the APIs that the stable stuff will be calling so they don't necessarily have to be recompiled for each new minor release?

Let's focus on how to separate the volatile parts of initramfs that have to stay in the kernel tree, and the non-volatile parts that can talk to the rest of the kernel through cleanly defined APIs the kernel must support.

If a section of code can't talk to the kernel through a stable API, it can never be moved out of the kernel tree, and there's not too much point in moving it out of the kernel proper, really...

Rob

The whole fscking point is to _avoid_ special API. System calls. Normal, usual system calls and nothing else.

And frankly, moving out of the kernel tree is the least of my worries - if it's a normal userland code it's fine with me.

I don't see any problems with having the first component of image generated when you build the kernel - as the matter of fact you want that, since there is no reason to have nfsroot code there if you don't support NFS, etc.

This is an argument for the script called by 'make install' to peek at the config, not for the kernel to build any of the user space boot tools.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Followup to:  <[email protected]>
By author:    Theodore Tso <[email protected]>
In newsgroup: linux.dev.kernel

That would seem to be major lose, no?

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>

It works just fine with uClibc.. Both shared and static,

-Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--