Skip to contentSkip to conversation

Booting a modular kernel through a multiple streams file

DISCUSSION Otto Wyss 10 participants 15 messages Dec 16, 2001 – Dec 27, 2001

lore
Share & tools
lore

Otto Wyss

author
#0 lore raw

Well I don't think a kernel without anything compiled in is currently functional but I'd rather like to have as less as possible. And I _don't_ like the solution with initrd. So I've thought about other solutions to solve the hen-egg dilemma when booting a modular kernel.

Well a simple solution would be if Linux supports the multiple streams file format. Assume the kernel and all necessary modules for booting (if not all modules) are combined into a single file. The boot loader (i.e. lilo) simply loads this file and starts the first stream (the kernel). It doesn't need to know the full multiple stream format (maybe nothing at all). The kernel of course needs this functionality to load the rest of the modules for a minimal working system.

I assume it's no problem to integrate the building of this boot file into either the Linux compilation or better into a separate setup phase (possibly together with good hardware/module detection).

Advantages:
- A simple boot loader can handle it without much tweaking
- The multiple streams file format is a standard concept usable anywhere
- No ramdisk is necessary
- This concept needs possibly less kernel functionality than initrd
- No change in the current compilation process except for the additional setup phase

Disadvantages:
- Someone else has to do it, I'm not a kernel/driver developer

Why did nobody else have this simple idea? I don't know, maybe the multiple streams file format isn't widely known in the Linux community. Who else does use it? The MacOS runs nicely with it, supporting data and resource streams. Maybe it is already ported from the MacOS X to Darwin. WindowsNT supports it but doesn't use it.

O. Wyss

#1 lore raw
Quoted 8 lines · Otto Wyss

Well a simple solution would be if Linux supports the multiple streams file format. Assume the kernel and all necessary modules for booting (if not all modules) are combined into a single file. The boot loader (i.e. lilo) simply loads this file and starts the first stream (the kernel). It doesn't need to know the full multiple stream format (maybe nothing at all). The kernel of course needs this functionality to load the rest of the modules for a minimal working system.

Quoted 7 lines

I assume it's no problem to integrate the building of this boot file into either the Linux compilation or better into a separate setup phase (possibly together with good hardware/module detection).

Advantages:
- A simple boot loader can handle it without much tweaking
- The multiple streams file format is a standard concept usable anywhere

ITYM useless.

Quoted 9 lines

- No ramdisk is necessary
- This concept needs possibly less kernel functionality than initrd
- No change in the current compilation process except for the additional setup phase

Disadvantages:
- Someone else has to do it, I'm not a kernel/driver developer

Why did nobody else have this simple idea? I don't know, maybe the multiple streams file format isn't widely known in the Linux community.

Ewww....

"Forked files" crap _is_ known. And not welcome.

There is a bog-standard way to combine several files in one - cpio. Or tar. No need to bring Apple Shit-For-Design(tm)(r) when standard tools are quite enough.

author
#2 lore raw
Quoted 9 lines
Quoted 2 lines

Well a simple solution would be if Linux supports the multiple streams file format.

"Forked files" crap _is_ known. And not welcome.

There is a bog-standard way to combine several files in one - cpio. Or tar. No need to bring Apple Shit-For-Design(tm)(r) when standard tools are quite enough.

I don't want to bring in any ...-Design if standard tools are enough! If cpio/tar do suffice, then much better. I don't know if they are simple and fitting enough to be handled by any boot loader and the kernel. Maybe you could elaborate more on this.

You have to admit that a multiple streams file format (regardless which kind) would be a good solution to the booting of a modular kernel. Anyway this format has to be supported by the kernel itself and in some extend by any boot loader. So anybody has to write a kernel module for the cpio/tar format and help with implementing it into boot loaders. Maybe you could give some help.

O. Wyss

#3 lore raw
In reply to Alexander Viro
Quoted 3 lines · Otto Wyss
Quoted 3 lines
Quoted 2 lines · Somebody not attributed

Well a simple solution would be if Linux supports the multiple streams file format.

Well "multiple streams file format" is purely a FILESYSTEM issue. When it is outside of the filesystem, it is an ARCHIVE (e.g. tar, cpio, zip, SHA, etc). So this whole concept is bogus.

Quoted 11 lines

I don't want to bring in any ...-Design if standard tools are enough! If cpio/tar do suffice, then much better. I don't know if they are simple and fitting enough to be handled by any boot loader and the kernel. Maybe you could elaborate more on this.

You have to admit that a multiple streams file format (regardless which kind) would be a good solution to the booting of a modular kernel. Anyway this format has to be supported by the kernel itself and in some extend by any boot loader.
So anybody has to write a kernel module for the cpio/tar format and help with implementing it into boot loaders. Maybe you could give some help.

Well, the good news is that this whole discussion is moot. Al Viro has already written a kernel patch which does all of this, and I'm sure he is just waiting to get it into the 2.5 kernel. It creates an "initramfs" which is populated from a cpio (or tar, can't remeber) archive attached to the kernel image.

This does things like BOOTP/DHCP discovery, mount the rootfs, load modules, etc, all before the root filesystem is mounted. This allows removing all sorts of junk from the kernel which was there because it needed to be done before the root filesystem was mounted.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

author
#4 lore raw
In reply to Alexander Viro
Quoted 4 lines

Well "multiple streams file format" is purely a FILESYSTEM issue. When it is outside of the filesystem, it is an ARCHIVE (e.g. tar, cpio, zip, SHA, etc). So this whole concept is bogus.

It is a FILESYSTEM issue and an ARCHIVE is simply an implementation of the "multiple streams file format" (MSFF), with archiving in mind. But it is not at all bogus as long as it _isn't_ supported directly by the kernel. The MSFF kernel support has more uses (i.e. transparently accessing anything inside as if it were a directory). Since it is a kind of fs within an fs it can be easy accessed from a boot loader regardless of the surrounding fs. I.e. Lilo can access it regardless if the MSFF is on SCSI raid system, a FAT floppy or a Mac Zip drive (on i386).

Quoted 9 lines
Quoted 2 lines

So anybody has to write a kernel module for the cpio/tar format and help with implementing it into boot loaders. Maybe you could give some help.

Well, the good news is that this whole discussion is moot. Al Viro has already written a kernel patch which does all of this, and I'm sure he is just waiting to get it into the 2.5 kernel. It creates an "initramfs" which is populated from a cpio (or tar, can't remeber) archive attached to the kernel image.

When I first read about initramfs I considered it just a slight improvement of initrd and now after more reading about it, I still think it's not the best solution. It still has to be loaded somewhere. Well initramfs might be usable but it is just another special case for the booting process.

With MSFF the kernel just sees it as an ordinary directory and accesses modules as if they where ordinary files (maybe within sub directories). The bootloader first just sees a file and tries to start the stream in it. Depending on how smart the MSFF is the boot loader does not even need to know the MSFF itself as long as it is able to start the stream. Otherwise if the MSFF is i.e. a tar archive even Lilo doesn't need much intelligence to start the kernel within.

O. Wyss

#5 lore raw
In reply to Otto Wyss
Followup to:  <[email protected]>
By author:    Otto Wyss <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 4 lines

Disadvantages:
- Someone else has to do it, I'm not a kernel/driver developer

By this you have pretty much shown yourself utterly unqualified to be a kernel *designer*. I won't even go into the various bogus assumption you're making.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#6 lore raw
In reply to Otto Wyss
Quoted 10 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    Otto Wyss <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 4 lines

Disadvantages:
- Someone else has to do it, I'm not a kernel/driver developer

By this you have pretty much shown yourself utterly unqualified to be a kernel *designer*. I won't even go into the various bogus

Agreed, but it should be also noted that to be able to do kernel coding is insufficient to do kernel design.
The same for user space, to be able to write correct C (or .*) will not stop people to write code with unsane design, organization and data structures.

You can see it on freshmeat and in many popular kernels every day. I see it with my code every time I read my last-month source code.

Signature · 4 lines
Salvatore Sanfilippo <[email protected]>
http://www.kyuzz.org/antirez
finger [email protected] for PGP key
28 52 F5 4A 49 65 34 29 - 1D 1B F6 DA 24 C7 12 BF
#7 lore raw
In reply to Otto Wyss
Quoted 8 lines · antirez
Quoted 3 lines
Quoted 1 line

By this you have pretty much shown yourself utterly unqualified to be a kernel *designer*. I won't even go into the various bogus

Agreed, but it should be also noted that to be able to do kernel coding is insufficient to do kernel design.

Agreed 100%.

-hpa

#8 lore raw
In reply to Otto Wyss
Quoted 1 line · Otto Wyss

modules) are combined into a single file. The boot loader (i.e. lilo)

LILO follows an outdated, broken concept and should once and for all be layed to rest, preferably with a stake through it's heart.

Quoted 1 line

simply loads this file and starts the first stream (the kernel).

The point to all these 'streams' escapes me. The proper way to implement this has all ready been done. It's called the Multi Boot Standard as implemented in GRUB bootloader. http://www.gnu.org/software/grub/

GRUB is similar to syslinux in that is can read directly from the the FS, but unlike syslinux supports just about all of them instead of just FAT.

Basically what Grub does is loads the kernel modules from disk into memory, and 'tells' the kernel the memory location to load them from, very similar to how an initrd file is loaded. The problem is Linux, is not MBS compilant and doesn't know to look for and load the modules.

HOWEVER I recieved patches from
"ChristianK."@t-online.de (Christian Koenig)
last week that actually implments the module loading in Linux! I'm trying to get Rusty interested in them, but thus far have not heard back from him. (Hey Paul! Hit that reply button!)

Anyone interested mail me off list to go over this in more detail.

Dave

Signature · 1 line
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/
#9 lore raw
Quoted 5 lines

Basically what Grub does is loads the kernel modules from disk into memory, and 'tells' the kernel the memory location to load them from, very similar to how an initrd file is loaded. The problem is Linux, is not MBS compilant and doesn't know to look for and load the modules.

And vendors who've shipped GRUB still have to ship Lilo because Grub plain doesn't work on some machines. Lilo has the virtue that its extremely simple in what it does and how it does it. It works in a suprisingly large number of cases and can handle interesting setups that GRUB really struggles with.

#10 lore raw
Quoted 3 lines · Alan Cox

And vendors who've shipped GRUB still have to ship Lilo because Grub plain doesn't work on some machines.

This is an issue with how GRUB deals with a FUBAR BIOS, and not necessarily a core GRUB design flaw.

Syslinux, seems to win hands down (IMO) when it comes to dealing with flacky BIOS's, but it's not because it's a dynamic loading design; it's because HPA knew enough and spent the time refining that 1% of the program that deals with bad BIOSes.

Should HPA and Werner take a day of their time to review and correct GRUB, I'm sure this issue would vanish overnight.

Quoted 4 lines

Lilo has the virtue that its extremely
simple in what it does and how it does it. It works in a suprisingly large number of cases and can handle interesting setups that GRUB really struggles with.

Where GRUB struggles LILO throws up brickwalls.

Coolest thing is the world you can do with grub:
Load a *partition* as your initrd
With this in mind it makes using images/archives for initrd for non-embedded systems seem rather clunky. It's much cleaner to just have an 8/16/32MB partition you dedicate to initrd. Users of GRUB have been able to do this for YEARS.

And something to say here: Yes I hate lilo, but not because I think Werner coded it lousy. A boot loader that uses a static block offset to find it's target is a bad idea, that should have died with M$-DOS.

That said, it's still a wonder to me why GRUB, or another loader similar in approach, was not adopted as the 'prefered' linux loader years ago.

Dave

Signature · 1 line
The time is now 22:54 (Totalitarian)  -  http://www.ccops.org/
#11 lore raw
In reply to Alan Cox
Quoted 10 lines · In article <[email protected]> you wrote:
Quoted 5 lines

Basically what Grub does is loads the kernel modules from disk into memory, and 'tells' the kernel the memory location to load them from, very similar to how an initrd file is loaded. The problem is Linux, is not MBS compilant and doesn't know to look for and load the modules.

And vendors who've shipped GRUB still have to ship Lilo because Grub plain doesn't work on some machines. Lilo has the virtue that its extremely simple in what it does and how it does it. It works in a suprisingly large number of cases and can handle interesting setups that GRUB really struggles with.

Apart that it moves the initrd somewhere unsafe on high memory machines and some other odds ends we have fixed, I know of exactly 1 problem with a hw raid controller, which we did not come around to debug yet.

All other machines, obscure as they may are, boot just fine and without problems with GRUB.

Ciao, Marcus

#12 lore raw
Quoted 6 lines

Apart that it moves the initrd somewhere unsafe on high memory machines and some other odds ends we have fixed, I know of exactly 1 problem with a hw raid controller, which we did not come around to debug yet.

All other machines, obscure as they may are, boot just fine and without problems with GRUB.

I can introduce you to machines that don't work with grub in my test collection. Things like the keyboard not working with a USB keyboard on one box.

Alan

#13 lore raw
In reply to Marcus Meissner
Quoted 15 lines · Marcus Meissner
Quoted 7 lines · In article <[email protected]> you wrote:
Quoted 6 lines

Basically what Grub does is loads the kernel modules from disk into memory, and 'tells' the kernel the memory location to load them from, very similar to how an initrd file is loaded. The problem is Linux, is not MBS compilant and doesn't know to look for and load the modules.

Quoted 5 lines

And vendors who've shipped GRUB still have to ship Lilo because Grub plain doesn't work on some machines. Lilo has the virtue that its extremely simple in what it does and how it does it. It works in a suprisingly large number of cases and can handle interesting setups that GRUB really struggles with.

Quoted 3 lines

Apart that it moves the initrd somewhere unsafe on high memory machines and some other odds ends we have fixed, I know of exactly 1 problem with a hw raid controller, which we did not come around to debug yet.

I'm just wondering if you mean the (almost) RAID HPT370. You can fix this by avoiding the "embed" command, which overwrites the raid setup information. Just "dd" the file system module to a higher offset (e.g.
16) and run grub's "install" command manually.

Wilfried

Signature · 2 lines
Terorists crashed an airplane into the server room, have to remove
/bin/laden. (rm -rf /bin/laden)
#14 lore raw
In reply to Otto Wyss
Quoted 6 lines · Dave Cinege
Quoted 1 line · Otto Wyss

modules) are combined into a single file. The boot loader (i.e. lilo)

LILO follows an outdated, broken concept and should once and for all be layed to rest, preferably with a stake through it's heart.

The way LILO does things is perfectly valid. Not flexible, not dynamic but valid.

Quoted 5 lines
Quoted 1 line

simply loads this file and starts the first stream (the kernel).

The point to all these 'streams' escapes me. The proper way to implement this has all ready been done. It's called the Multi Boot Standard as implemented in GRUB bootloader. http://www.gnu.org/software/grub/

GRUB is a good proof of concept bootloader but I wouldn't call it a good way to implement things. As for the Multi Boot Specification it is hardly a standard, and it got a lot of details wrong. A big one is who is supposed to do the BIOS calls.

It should be o.k. for a bootloader to be static, but still useful on multiple machines for years. What comes out of GRUB does not encourage a static bootloader. But the development of GRUB is slow enough that it is practically static...

Quoted 8 lines

GRUB is similar to syslinux in that is can read directly from the the FS, but unlike syslinux supports just about all of them instead of just FAT.

Basically what Grub does is loads the kernel modules from disk into memory, and 'tells' the kernel the memory location to load them from, very similar to how an initrd file is loaded. The problem is Linux, is not MBS compilant and doesn't know to look for and load the modules.

So tell me how you make an MBS compliant alpha kernel again?

Eric