initramfs merge, part 1 of N

Linus,

The attached below is the first of several changes for initramfs / early userspace.

This change is intentionally very simple, not really proving its worth until next week when patches 2 and 3 in this series arrive in your inbox. A description of "the future" follows description of this specific cset.

1) Introduce init/initramfs.c itself, which is a module that uncompresses a .cpio.gz archive, and uses it to populate rootfs with files early very in the bootup process (between signals_init and proc_root_init in init/main.c). People will see a small listing in dmesg of unpacked files. We need to keep this for now (and for now it's small), but we may want to remove this output or turn the knob down to KERN_DEBUG before 2.6.x release:
-> file1
-> file2
-> etc...

(architecture maintainers note!)
2) Introduce ARCHBLOBLFLAGS in arch/$arch/Makefile, for turning an arbitrary binary object into a .o file using objcopy.

3) Link the initramfs cpio archive in vmlinux image via arch/$arch/vmlinux.lds.S, in the init section.

4) Introduce the new linux/usr directory. Currently it is not very interesting, only containing a small host-built proggie that generates the initial cpio archive, gen_init_cpio. This program will go away when early userspace is further along. It currently exists to show initramfs is working, by allowing us to remove three simple lines from init/do_mounts.c.

The Future.

Early userspace is going to be merged in a series of evolutionary changes, following what I call "The Al Viro model." NO KERNEL BEHAVIOR SHOULD CHANGE. [that's for the lkml listeners, not you <g>] "make" will continue to simply Do The Right Thing(tm) on all platforms, while the kernel image continues to get progressively smaller. Here is the initial plan for early userspace, i.e. the patches you are going to be seeing next week:

#2 - merge klibc.

As I said earlier, I am not sure if we will wind up removing klibc just before 2.6.x release or not. Comments welcome. But for now, klibc will be merged into the kernel tarball, because otherwise version drift during the evolution of early userspace will be a huge PITA, and slow things down. It is a tiny libc written specifically for the kernel.

This patch will add klibc to the build system, and create a tiny, statically-linked binary "kinit". kinit is the beginnings of early userspace. Some tiny, token amount of do_mounts.c code will be moving into kinit in patch #2, only enough to prove the system is working.

#3 - move initrd to userspace

Unfortunately we don't start seeing tangible benefits to early userspace until this patch, but that's how evolution works :) Here, initrd unpacking code is moved to userspace, as much as possible. Some initrd code will inevitably stay in the kernel, because it is arch-specific how to grab the initrd image from bootmem [or whereever], but the vast majority of initrd code goes poof (yay!). No initrd behavior will change at all, from current kernels. It is simply getting moved to early userspace. Users will not need to do anything on their end to make sure their existing setups continue to work -- any such actions are a bug on my part.

This patch will also turn "kinit" into a shared binary, and introduce the gzip binary into early userspace. [see "Items For Discussion" below, too, WRT this.]

#4 - move mounting root to userspace

People probably breathed a sigh of relief at patch #3, they will heave a bigger sigh for this patch :) This moves mounting of the root filesystem to early userspace, including getting rid of NFSroot/bootp/dhcp code in the kernel.

#N - to infinity... and beyond!

I, and hopefully others, will continue in the series of evolutionary patches, moving more and more stuff to early userspace. There are a lot of possibilities, and I will be looking for input from others on useful things to move, as well as continuing my own work of finding items that can be moved.

Items For Discussion

#1 - shared kinit

"kinit" is _the_ early userspace binary -- but not necessarily the only one. Peter Anvin and Russell King have several binaries in the klibc tarball, gzip, ash, and several smaller utilities. Peter also put work into making klibc a shared object -- that doesn't need an shlib loader. It's pretty nifty how he does it, IMO: klibc.so becomes an ELF interpreter for any klibc-linked binaries. klibc-linked binaries are, to the ELF system, static binaries, but they wind up sharing klibc.so anyway due to this trick.

Anyway, there is a certain elegance in adding coding to kinit instead of an explosion of binaries and shell scripts. The other side of that coin is that with elegance you sacrifice some ease of making changes. I am 60% certain we want a shared klibc and multiple binaries, but am willing to be convinced in either direction. If you think about it, there _are_ several benefits to leaving kinit as the lone binary in the stock kernel early userspace build, so the decision is not as cut-n-dry as it may immediately seem.

#2 - klibc in the kernel tarball

It's going in, for now. That's not open to discussion. However, the future is... I know the old maxim of "once's it in, it never goes away." Maybe that's the case, and if so, no big deal. I know at least a couple people who would like to see it leave the kernel tarball before 2.6.0 is released. I solicit comments on this item, though I think we will won't be in a position to answer this question until 2.6.0 release is near.

That's it for now. Questions, comments, and flames welcome. If I missed some early userspace benefits, let me know. If you know of good things to move to early userspace, let me know. If there are upcoming bumps in the road I did not mention, let me know.

Credits: Al Viro for the initramfs work. hpa, rmk, Greg KH, Alan, and innumerable others have contributed ideas if not actual code towards this effort. And thanks to our Emporer Penguin for giving me a break, when I blew the Halloween deadline spending 24 hours debugging an _incredibly_ stupid bug on my part. I deserve not one but two brown paper bags for that one.

Linus, please do a

bk pull http://gkernel.bkbits.net/minitramfs-2.5

This will update the following files:

 Makefile                |    2 
 arch/i386/Makefile      |    1 
 arch/i386/vmlinux.lds.S |    4 
 init/Makefile           |    2 
 init/do_mounts.c        |    4 
 init/initramfs.c        |  466 +++++++++++++++++++++++++++++++++++++++++++++++-
 init/main.c             |    2 
 usr/Makefile            |   18 +
 usr/gen_init_cpio.c     |  137 ++++++++++++++
 9 files changed, 629 insertions(+), 7 deletions(-)

through these ChangeSets:

<[email protected]> (02/11/01 1.862.2.2)
Kill stupid bug in initramfs that prevented it from working. (thanks to Al Viro for his patience, I owe him one)

<[email protected]> (02/11/01 1.858.2.1)
Minimal initramfs support (based on Al Viro's work).

diff -Nru a/Makefile b/Makefile
@@ -209,7 +209,7 @@
 drivers-y	:= drivers/ sound/
 net-y		:= net/
 libs-y		:= lib/
-core-y		:=
+core-y		:= usr/
 SUBDIRS		:=
 
 ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),)

diff -Nru a/arch/i386/Makefile b/arch/i386/Makefile
@@ -18,6 +18,7 @@
 
 LDFLAGS		:= -m elf_i386
 OBJCOPYFLAGS	:= -O binary -R .note -R .comment -S
+ARCHBLOBLFLAGS	:= -I binary -O elf32-i386 -B i386
 LDFLAGS_vmlinux := -e stext
 
 CFLAGS += -pipe

diff -Nru a/arch/i386/vmlinux.lds.S b/arch/i386/vmlinux.lds.S
@@ -77,6 +77,10 @@
 	*(.initcall7.init)
   }
   __initcall_end = .;
+  . = ALIGN(4096);
+  __initramfs_start = .;
+  .init.ramfs : { *(.init.initramfs) }
+  __initramfs_end = .;
   . = ALIGN(32);
   __per_cpu_start = .;
   .data.percpu  : { *(.data.percpu) }

diff -Nru a/init/Makefile b/init/Makefile
@@ -2,7 +2,7 @@
 # Makefile for the linux kernel.
 #
 
-obj-y    := main.o version.o do_mounts.o
+obj-y    := main.o version.o do_mounts.o initramfs.o
 
 # files to be removed upon make clean
 clean-files := ../include/linux/compile.h

diff -Nru a/init/do_mounts.c b/init/do_mounts.c
@@ -748,9 +748,7 @@
 		mount_initrd = 0;
 	real_root_dev = ROOT_DEV;
 #endif
-	sys_mkdir("/dev", 0700);
-	sys_mkdir("/root", 0700);
-	sys_mknod("/dev/console", S_IFCHR|0600, MKDEV(TTYAUX_MAJOR, 1));
+
 #ifdef CONFIG_DEVFS_FS
 	sys_mount("devfs", "/dev", "devfs", 0, NULL);
 	do_devfs = 1;

diff -Nru a/init/main.c b/init/main.c
@@ -72,6 +72,7 @@
 extern void pte_chain_init(void);
 extern void radix_tree_init(void);
 extern void free_initmem(void);
+extern void populate_rootfs(void);
 
 #ifdef CONFIG_TC
 extern void tc_init(void);

@@ -433,6 +434,7 @@
 	vfs_caches_init(num_physpages);
 	radix_tree_init();
 	signals_init();
+	populate_rootfs();
 #ifdef CONFIG_PROC_FS
 	proc_root_init();
 #endif

diff -Nru a/usr/Makefile b/usr/Makefile
@@ -0,0 +1,18 @@
+
+include arch/$(ARCH)/Makefile
+
+obj-y := initramfs_data.o
+
+host-progs := gen_init_cpio
+
+clean-files := initramfs_data.cpio.gz
+
+$(obj)/initramfs_data.o: $(obj)/initramfs_data.cpio.gz
+	$(OBJCOPY) $(ARCHBLOBLFLAGS) \
+		--rename-section .data=.init.initramfs \
+		$(obj)/initramfs_data.cpio.gz $(obj)/initramfs_data.o
+	$(STRIP) -s $(obj)/initramfs_data.o
+
+$(obj)/initramfs_data.cpio.gz: $(obj)/gen_init_cpio
+	( cd $(obj) ; ./gen_init_cpio | gzip -9c > initramfs_data.cpio.gz )
+

Oh yeah... quick addition.

At some point in the evolution, I will add the ability to load initramfs in all the ways that initrd is currently loaded now (from the bootloader, etc.). Substituting a custom initramfs cpio archive in the kernel link will also be added at a later time.

Won't the initial userspace be linked into the kernel? If so, why will the kernel image get smaller?

Yes and no ;-)

Ignoring for a moment initramfses loaded from your bootloader (a la initrd)... The amount of code that runs in kernel space shrinks, which is the main point of early userspace. If you are talking in terms of overall kernel image size, yes, but the initramfs cpio archive is ditching along with the rest of __init code, so you're really only talking about wasting a couple of additional pages in vmlinux -- a slight increase in disk space usage, and that's it.

So runtime memory usage certainly does not increase...

Jeff

By the way, the final initramfs should typically be a union of whatever sources there are; with the ones linked into the kernel image unpacked first (so they can be overwritten if so specified to the bootloader.)

-hpa

Note that the reason I personally really want initramfs is not to make the kernel boot image smaller, or the kernel sources smaller. That won't happen for a long time, since I suspect that we'll be carrying the initramfs user space with us for quite a while (eventually it will probably split up into a project of its own, but certainly for the forseeable future it would be very closely tied to the kernel).

The real advantage to me is two-fold:

- make it easier for people to customize their initial system without having to muck with kernel code or even use a different boot sequence. One example of this is the difference between vendor install kernels (using initrd) and a normal install kernel (which doesn't).

So I'd much rather see us _always_ using initrd, and the difference between an install kernel and a regular kernel is really just the size of the initrd thing.

- Many things are much more easily done in user space, because user space has protections, "infinite stack", and in general a lot better infrastructure (ie easier to debug etc). At the same time, many things need to be done _before_ the kernel is fully ready to hand over control to a normal user space: do ACPI parsing so that we can initialize the devices so that we can use the "real" user space that is installed on disk etc.

Sometimes there is overlap between these two things (ie the "easier to do in user space" and "needs to be done before normal user space can be loaded"). ACPI is one potential example. Mounting the root filesystem over NFS after having done DHCP or other auto-discovery is another.

So "shrinking the kernel" is not on my list here. It's really a matter of "some initialization is better done in user space", and not primarily "we want to make the kernel smaller". I'm not a big believer in microkernels and trying to get everything out of the kernel itself, but I _do_ believe that sometimes it's easier to just let the user do his own choices (while still giving him all the protection implied by running in user space).

Linus

I agree 100% with this. I don't think <kernel>+<early userspace> will ever be smaller than the current kernel, but I have invested quite a bit of effort into it for exactly the reasons done above.

klibc binaries might not be what one usually tends to run, but during klibc development I could still use standard gdb, strace, and just plain "run it off the command line" debugging techniques from a full-blown environment. When that doesn't work (like testing dynamic klibc), chroot will usually do the trick. The compile-test-debug cycle is so much faster than for a kernel boot that it's just plain amazing.

-hpa

[snip]

Let me add the third one: userland is more limited. And no, that's not a typo - and it's a good thing. Userland has to use normal, regular syscalls instead of poking its fingers into hell knows what parts of kernel data structures.

Which means that it's more robust and that it doesn't stand in the way of work on kernel. 90% of PITA with super.c used to be of that kind - mounting root filesystem had been done with very ugly kludges and what's more, these kludges got filtered down in the normal codepath. Getting rid of that took a _lot_ of very careful manipulations with the guts of the thing. And guess what? There was no reason why all that black magic would be necessary - current code uses normal, garden-variety system calls.

In effect, we used to have special cases of mount(2), etc., with very kludgy semantics. They were not exposed to userland, but that didn't make them less nasty or less painful to work with. They still cluttered the code, they still stood in the way of work on the thing and they still were butt-ugly.

And that's what moving code to userland should prevent - it's much easier to catch somebody bringing a patch with magical extension of system call than to catch an attempt to sneak special-case code used only by kernel.

BTW, that's a thing we need to watch for - there obviously will be a lot of patches moving stuff to userland and there will be a strong temptation to add magic interfaces just for that. _That_ should be prevented - it's better to leave ugly crap as is than export the same crap to userland. The point is to get the things cleaned up and make sure that they stay clean, not to cement them in place by adding a magic ioctl/syscall/flag/whatnot. We may very well end up extending existing interfaces, but we'd damn better make sure that such additions make sense for generic use.

We have a lot of ugly crap that would be unnecessary if we had early access to writable fs. Basically, we got magic methods, magic codepaths, etc. simply because the normal access to the functionality in question required opened file descriptors. Now we _do_ have a writable filesystem mounted very early, so that cruft can be killed off. And moving code to userland acts as a filter - there we don't have access to magic, so all such magic immediately shows up. It could be done in the kernel (and quite a few things had been done already), but move to userland acts as a safeguard against reintroduction of magic crap.

Again for your consideration:

Initrd Dynamic (Dynamic Initial Ramdisk)

Initrd Dynamic allows extracting tar and tar.gz archives to the rootfs. It additonally cleans do_mounts, and rewrites the legacy initrd system.

It provides the same functionality of initramfs but in a more mature and robust system. It does not depend on legacy initrd operation. It will prepare 'early userspace' with klibc, et al.

With your acceptance an additonal patch will be forthcoming making the legacy initrd system a compile time option, and moving the call to initrd_mount() from do_mounts, to main. (IE compile time initrd.o)

Further patches will purge specific legacy initrd operations from the general code base and move them to initrd.c where appropreate.

A patch against 2.5.45 is here and attached:
http://ftp.psychosis.com/linux/initrd-dyn/kernelpatches/2.5.45/initrd_dynamic-2.5.45.diff.gz

You can view the primary files involved here, already 'post-patched' 2.5.45:
http://ftp.psychosis.com/linux/initrd-dyn/kernelpatches/2.5.45/do_mounts.c
http://ftp.psychosis.com/linux/initrd-dyn/kernelpatches/2.5.45/initrd.c
http://ftp.psychosis.com/linux/initrd-dyn/kernelpatches/2.5.45/untar.c

diff -uNr linux-2.5.45-virgin/drivers/block/Kconfig linux-2.5.45-initrd_dyn/drivers/block/Kconfig
@@ -331,6 +331,18 @@
 	  "real" root file system, etc. See <file:Documentation/initrd.txt>
 	  for details.
 
+config BLK_DEV_INITRD_UNTAR
+	bool "Initial RAM disk untar support (requires TMPFS)"
+	depends on BLK_DEV_INITRD && TMPFS
+	help
+	  Untar support to a tmpfs root. Say Y.
+
+config BLK_DEV_INITRD_GUNZIP
+	bool "Initial RAM disk gunzip support"
+	depends on BLK_DEV_INITRD
+	help
+	  Use gzip compressed images/archives with initrd. Say Y.
+
 config LBD
 	bool "Support for Large Block Devices"
 	depends on X86

diff -uNr linux-2.5.45-virgin/include/linux/root_dev.h linux-2.5.45-initrd_dyn/include/linux/root_dev.h
@@ -3,6 +3,7 @@
 
 enum {
 	Root_NFS = MKDEV(UNNAMED_MAJOR, 255),
+	Root_TMPFS = MKDEV(UNNAMED_MAJOR, 12),
 	Root_RAM0 = MKDEV(RAMDISK_MAJOR, 0),
 	Root_RAM1 = MKDEV(RAMDISK_MAJOR, 1),
 	Root_FD0 = MKDEV(FLOPPY_MAJOR, 0),

diff -uNr linux-2.5.45-virgin/lib/inflate.c linux-2.5.45-initrd_dyn/lib/inflate.c
@@ -161,6 +161,10 @@
 #define wp outcnt
 #define flush_output(w) (wp=(w),flush_window())
 
+#ifndef EXIT_CODE
+#define EXIT_CODE 0
+#endif
+
 /* Tables for deflate from PKZIP's appnote.txt. */
 static const unsigned border[] = {    /* Order of the bit length code lengths */
         16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15};

@@ -980,7 +984,7 @@
     gzip_release(&ptr);
     if (hufts > h)
       h = hufts;
-  } while (!e);
+  } while (!e && !EXIT_CODE);
 
   /* Undo too much lookahead. The next read will be byte aligned so we
    * can discard unused bits in the last meaningful byte.

@@ -1146,7 +1150,10 @@
 	    }
 	    return -1;
     }
-	    
+    
+    if (EXIT_CODE)
+        return 0;
+	
     /* Get the crc and original length */
     /* crc32  (see algorithm.doc)
      * uncompressed input size modulo 2^32

One idea I experimented some time ago with (and can revive after some sleep) is, rather than interpreting cpio in the kernel, objcopy a binary into a init and copy that into pagecache in a ramfs/libfs
file system. The population was all initfunctions, trying to make
it disappear at runtime. /dev/initrd was left for userspace to expand the rest of the loaders. With libfs, the write code reinstated so standard directories, device nodes, console and initrd nodes can be created and opened in userspace, further shrinking the static linked-in code.

This argues that this initial code is unshared and uncompressed (or rather, compressed like the rest of the kernel); for shared we would have to copy a couple of pieces this way. It traded off a table of offset,length,mode,name with cpio headers and parsing.

I had this running on 2.4.19-pre10 (around the time of the kernel summit, just before the fixed directory link counts went in) with busybox. (I seperated the 2.4 compat vs 2.5 stuff at that time).

Comments?

milton

For those of us who only develop on nfsroot-based systems, does this step include adding userspace network interface configuration and bootp/dhcp client functionality to kinit? I want to assume that "getting rid of NFSroot/bootp/dhcp" means moving that particular functionality as part of this step. Just wondering what the short-term impact will be on the poor embedded guys. :)

Regards,

Matt Porter
[email protected]
This is Linux Country. On a quiet night, you can hear Windows reboot.

Probably not to kinit, but to early userspace, yes. There is no real reason to put everything into kinit, and a lot of these things we have already written up as part of the klibc bundle.

-hpa

disclaimer: if this is the wrong ng to be posting this to, its only due to ignorance.. I dont know the first thing about where to post this question..

----------------------------------------------------------------------

Hello,
Im ready to be beaten up for asking this question ( I am not sure
which group to post to -- all this is new to me) but, I was wondering how one could figure out if the kernel was in idle mode (or idling).

I *have* tried to look for the answer and here is waht I have come up with so far :

Process 0 is the idle process.. but, I dont understand how you can tell if this means that the kernel is in idle mode. Do we just probe the state field of all process entries and check to see if everyone is sleeping and conclude that the kernel is idling??

for_each_process(p)

 {
    if(process->state == S)
     {
        countup;
     }

}

if countup == number of processes, then the kernel was idling?

-anu

********************************************************************************

Think, Train, Be

*******************************************************************************

Hi,

Well.. this mailing list is not that bad for questions like this. You got the idea somewhat right though it is implemented quite different. The big word here is scheduling. A scheduler is a piece of code that determines what thread is to be executed next. How this is done is something entire books are written about, and a topic that will be discussed on the lkml often.

With linux, the idle thread is entered when the scheduler finds no threads ready for executing. (Not only sleeping, but also waiting for Disk etc) With some BSD clones, there is something of an idle queue, a list of threads that is only to be executed when the system is actually idle. When the scheduler access this queue, you know it has nothing important to do anymore. Linux uses priority queueing (see the nice manual for info on that). But, for both solutions holds: as soon as the scheduler reaches the end of the queue(Linux) / queues (some BSDs) without finding a thread that can be executed, the scheduler enters the real idle thread.

In short: you don't really have to count. You only have to check if you reach the end of your thread list. Checking if a thread is able to run is what your scheduler already does.

See the scheduler code for more info.

Jos

Hello,
I am looking at some way of "automatically" figuring out when a
kernel might be idle -- more at the level of the kernel code itself. After a day's reading i have the following pieces of information:

.a. There is something called the run_queue which has a list of process
that can be run.
.b. When nothing is running, we have the swapper process running (process
0 ) that is the ancestor of all processes.
.c. the scheduler goes in and checks this readyqueue every so often, so,
we can figure out if there are no processes running at any given time..
.d. I am now trying to modify the kernel to do something interesting when
there is only the idle process running.. No idea what though. Linux
2.4.9 (which is the version i am looking at ) has a bunch of gotos and I think i have identified that the section under still_running_back: is the place to identify when the run_queue is empty.. I am thinking of putting in some printfs() to make the kernel put out a message that says something like "i am idling.." everytime the kernel is idling.. and execute a ps uax simultaneously to show that the kernel is indeed idling..

are there any obvious disasters that u chaps see? (im not an OS person..)

-a

********************************************************************************

Think, Train, Be

*******************************************************************************

There's more to is than just processes: if your kernel has runnable tasklets or pending interrupts, it is not truly idle, even though there may be no runnable processes.

In umlsim, I have some heuristics that seem to catch most cases, but may be a bit too paranoid. Look at timer.c:wait_kernel (called from idle) in http://www.almesberger.net/umlsim/umlsim-4.tar.gz

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

s/probably/definitely/

There is a lot of reasons for _not_ putting everything into one binary - if nothing else, it allows to deal with situations like

	/* do a lot of things that are OK for userland */
	/* do ugly magic */
	/* do a lot of things that are OK for userland */

without exporting ugly crap.

It's much better to have several userland helpers called from init sequence to do sane stuff in userland and leave remaining crap where it is, than to add user-visible interfaces that don't make sense.

Ok, sounds good. I only mentioned kinit since Jeff's roadmap seemed to be hazy on whether there was consensus on the single binary approach versus several binaries. For maintenance sake, it seems that optional separate binaries is the only way to go. Glad to hear that this is the plan.

Regards,

Matt Porter
[email protected]
This is Linux Country. On a quiet night, you can hear Windows reboot.