PART1: Proposed init & module changes for 2.5

Highlights:
o Unified boot/module parameters, with type checking. Using PARAM(xxx, type, perms) and family.

o Simpler in-kernel module loading insmod can now be written in ~20 lines.

o Nicer internal module interface request_module("%s", foo) request_module_start/stop wrappers module_get()/module_put(), symbol_get()/symbol_put()

o Warm fuzzy bleeding edge feel

Todo:
(1) Incorporate non-ppc architectures (x86, sparc, sparc64, ppc64 already written, not in patch).

(2) Convert parts of the kernel *I* don't use.

(3) Use kbuild 2.5, so in-kernel params don't all get called "unknown.xxx".

(4) Update documentation.

(5) CONFIG_MODVERSIONS replacement and check for kernel version.

(6) Patch in proc/sys, so param entries can appear there (this is what the final arg to PARAM is for).

(7) Write more sophisticated modprobe:
o /etc/modules.conf
o Fallback to old modutils if old system.
o --force to override MODVERSIONS and kernel version.
o Save parameters from /proc on unloading.

(8) rmmod should not remove if busy (unless rmmod -f).

(9) Convert everyone to new safe module insert usage.

Get it from:

	http://netfilter.samba.org/diary/module-init-patch-001.bz2
	http://netfilter.filewatcher.org/diary/module-init-patch-001.bz2
	http://netfilter.gnumonks.org/diary/module-init-patch-001.bz2

(46885 bytes)

And v. v. simple insmod/rmmod/modprobe from:

	http://netfilter.samba.org/diary/module-init-tools-001.bz2
	http://netfilter.filewatcher.org/diary/module-init-tools-001.bz2
	http://netfilter.gnumonks.org/diary/module-init-tools-001.bz2

(33181 bytes)

HACKERS QUICKREF:

OLD: int parse_foo(char *s) { ... return 1; }

	__setup("foo=", parse_foo);
NEW:	int parse_foo(char *s, struct kernel_param *kp) { ... return 0; }
	PARAM_CALL(foo, parse_foo, NULL);
COMMENTS:
	This will match "foo=" only.  kb->name == "foo", kp->arg ==
	third arg of PARAM_CALL.

OLD: MODULE_PARM("foo", &foo, "i")
NEW: PARAM(foo, int, S_IRUGO);
COMMENTS:
foo must be in scope (and in this case, of type int) or the compiler will catch it.
S_IRUGO means that it will be accessible through proc (use 000 if this is not useful, or maybe S_IWUSR if root can change it at any time).
Use PARAM_NAMED to give a different name, PARAM_ARRAY for array.

OLD: inter_module_*
NEW: EXPORT_SYMBOL, symbol_get(foo) & symbol_put(foo)
COMMENTS:
These are now typesafe, so foo declaration must be in scope.

OLD: if (foo->owner) __MOD_INC_USE_COUNT(foo->owner);
NEW: module_get(foo->owner);

OLD: module_init(initfn)
NEW: init_and_startcall(initfn, startfn);
COMMENTS:
Please use a semicolon at the end. Modulable code should transition to a two-stage init (initfn sets everything up and may fail, and startfn which exposes the module to the rest of the kernel and can't fail). Some modules only need initfn or startfn, in which case use initcall() or startcall().

OLD: if (try_inc_mod_count(foo->owner))
NEW: module_get(foo->owner)

OLD: int initfn(void) { ...; return 0; }
module_init(initfn)
NEW: void initfn(void) { ...; }

	bootcall(initfn);

COMMENTS:
This is for code which can never be a module: we ignore the return value anyway, so why have one (panic() if you must).

OLD: void exitfn(void) { ...; }
module_exit(void)
NEW: int stopfn(void) { ...; return 0; }

	void exitfn(void) { ...; }
	stopcall(stopfn);
	exitcall(exitfn);

COMMENTS:
If there are neither, then module is not unloadable. This is perfectly OK. If stopfn returns 0 (otherwise it should be -errno) it must have deregistered itself from the rest of the kernel (ie. module count can never increase again), but still be usable to anyone using it currently. Once exitfn is called, it is guaranteed to be unused.

Enjoy!
Rusty.
--
Premature optmztion is rt of all evl. --DK

On Sunday 23 September 2001 2:37, Rusty Russell wrote:

Russ,

How about implementing MBS? (Bootloader module loading. IE as implmented in GRUB) so we can finally have completely modular kernels?

Dave

Hi Dave,

Why duplicate the module code in the boot loader? Surely a tiny initrd
or equiv is a better option here?

Rusty.
--
New mailer, beware.

Hi,

I like the PARAM() part, but still have some question. Hope you can clarify.

Why separating them?

Same question here: Why you changed this?

What is meant with "usable to anyone using it currently"? Does it mean the variables of it can still be read/written? Can the startcall or the initcall still be called after stopcall?

(Important if we rely on zero initialization, which we do in many modules).

Can IO to ports still be issued and "transactions" still be finished after stopcall has been issued?

Thanks and Regards

Ingo Oeser

In der Wunschphantasie vieler Mann-Typen [ist die Frau] unsigned und
operatorvertraeglich. --- Dietz Proepper in dasr

In message <[email protected]> you write:

Hi Ingo,

Glad you like the PARAM stuff. To answer your question,
consider the following standard case in module initialization:

	int init(void)
	{
		int ret;

		ret = register_something();
		if (ret != 0) return ret;

		ret = register_something_else();
		if (ret != 0) {
			unregister_something();
			return ret;
		}
		return 0;
	}

Imagine register_something() succeeds, but register_something_else() fails. Someone could be accessing the module (through register_something()): what do we do? We can't free the module and return the error to insmod...

Mind you, many modules currently ignore errors anyway...

Someone could still have a reference to the module when stopfn is called. If you register an interrupt handler, say, there could be someone running it on another CPU right now.

Yes: after stopfn, no NEW things can access the module, but it must be still usable for anyone who got access to the module before.

No: at this stage the module will have to be reloaded, exactly because of assumptions like zero-initialization.

Yes... The module is still "live", it's just you must unregister anything which might allow new accesses (eg. proc entries, filesystems, IRQ handlers, etc). Sometime after the module count falls to zero, exitfn will be called and the module actually freed.

Hope that clarifies,
Rusty.
--
Premature optmztion is rt of all evl. --DK

When we discussed this at linux.conf.au in Sydney, we agreed that we could call startfn after stopfn to handle the quiesce unload algorithm. That handles the rmmod race without exporting mod use count to everything, i.e.

rmmod
if use count == 0, call stopfn()

    synchronize_kernel()
    if use count == 0, exitfn()
    if use count != 0, startfn()

That catches the race where a second cpu has entered the module but not done MOD_INC_USECOUNT yet. The module is now in use but stopfn has been called, the best thing to do is accept that rmmod lost the race and reinstate the module.

In message <[email protected]> you write:

Hi Keith!

Yes, this was my original intention, but the more I thought
about it, the more I wondered if it was worth it. In fact, my patch adds CONFIG_MODULE_UNLOAD, because most people don't actually need it, and I don't really care if most modules are not unloadable. Loading modules makes sense. Unloading them is pretty much an optimization.

The reason I eventually decided to split initialization in
this version anyway, was that so many drivers get it wrong! Things like registering interrupt handlers before they have set up their internal state, etc. Splitting it into two functions is merely a mechanism to get them to think about things a little harder 8)

And honestly, I got distracted by the PARAM stuff 8)

Cheers!
Rusty.
--
Premature optmztion is rt of all evl. --DK

I absolutely agree that the split between probe and register needs to be done. The difference between calling startfn after stopfn and forbidding that sequence is the difference between

rmmod
loses race
return OK
module is still in use, user is confused

and

rmmod
loses race
calls startfn
returns failure
module is still in use, user is happy

Allowing startfn after stopfn gives a more consistent view of the operation.

Ah, I see... no, it's currently:

rmmod
loses race
waits for module count to hit zero

This can be desirable behavior (rmmod -f). It's not good if you wanted auto-module-unloading, but then, we've never wanted to take the hit to have a pagable kernel, and I don't think we should do so.

Cheers,
Rusty.

Russ,

No. It's much more restrictive. IMO the JOB of the boot loader is to provide minimal lowest level (arch dependant) device access to load and initialise the kernel, with enough user control to manipulate how the kernel is initialised. GRUB does a fantastic job of this on x86. It's the first bootloader I've seen 'get it right'.

I don't suggest implementing the complete module loading in the boot loader, (I'm still not even sure I really like it in the kernel. ; > ) only loading the modules themselves into memory for the kernel access during it's exec. (Like an initrd image is loaded) If a boot loader wants to understand and use modules.dep or do on the fly dependency checking, or have kernel userland utils to update it's conifg intelligently, that's it's business. I only want to see a way modules can be preloaded before kernel exec.

This has already been designed and is called the Multi Boot Standard. You can read the spec in the GRUB source archive.
http://www.gnu.org/software/grub/
It been around a few years. It's sad only HURD(?) is using it,

Why an initrd solution doesn't even touch on the problem:
You have a kernel.
It's scsi layer and scsi device drivers are all modules.

You have a system.
It contains a single scsi drive.
The kernel and modules reside on that drive.

Now the impossible exercise.
Get that machine the boot.

All the parts are there but you can't. Your only option is to use an initrd that has the required modules *merged* into it. This is just disgusting and silly. It's also impossible to make that initrd on that machine without a suplementary system. (Yes there are other ways around this by why not just KISS???)

We ALREADY have the boot loader and spec designed to allow reloading modules into memory this way. (GRUB) It should not be very hard to implement this from the pre-existing Linux intird code to read the modules out. It's also easy for other boot loaders to use their initrd code to implement the functionality. (Remember when you think about it's usefulness put it in the perspective of bootloaders that can read from a filesystem. IE GRUB and Syslinux.)

Please let's do this already...
I look forward to one day reaching the point it's *reasonable* to have a completely modular kernel and that any device drivers (even boot devices) can be upgraded without having to update the entire kernel.

If you've never played with GRUB yet, you can grab a nice raw floppy image from my ftp site. ftp://ftp.psychosis.com/linux/grub

Dave

In message <[email protected]> you write:

I was hoping someone would say this. (Note that my userspace tools are not as complete as the current ones, so take a pinch of salt):

Old kernel:

$ wc -l kernel/module.c
   1250 kernel/module.c
$ ls -l /sbin/insmod
-rwxr-xr-x    1 root     root        99824 Aug 30 09:35 /sbin/insmod
$ find modutils-2.4.1/ -name '*.[ch]' | xargs cat | wc -l
  20612

New kernel:

$ wc -l working-pmac-module/kernel/module.c         
    922 working-pmac-module/kernel/module.c
$ wc -l working-pmac-module/arch/ppc/kernel/module.c
    253 working-pmac-module/arch/ppc/kernel/module.c
$ ls -l modult-init-tools/insmod
-rwxrwxr-x    1 rusty    rusty        5240 Sep 24 15:23 insmod
$ find module-init-tools/ -name '*.[ch]' | xargs cat | wc -l
    661

Convinced yet?

Why not just put them in the initrd image? The kernel is not going to use the entire module as it is, so it will have to copy it anyway...

Um, how do you get the modules into grub then? If it can read ext2 at boot, then it could create an initrd for me. Otherwise, how is it different from creating an initrd?

It'd be easy to implement now. But it won't gain us anything, since we'll all be using initrd-tng in 2.5 anyway.

Cheers,
Rusty.
PS. Am big fan of LRP: it rocks.
--
Premature optmztion is rt of all evl. --DK

"Rusty Russell wrote:"

[...]

Somebody wants to modularize all binary formats, including binfmt_elf ... ?

Andrzej

=======================================================================
  Andrzej M. Krzysztofowicz               [email protected]
  phone (48)(58) 347 14 61
Faculty of Applied Phys. & Math.,   Technical University of Gdansk

If it's cleaner in operation then the current system, yes. What we've got now just seems so ugly.

Because images are not dynamic. : P

Yep you're still using LILO...(that still does that absolute block offset shit.) GRUB is like syslinux in that it can read FS. But not just ext2. It also knows reiserfs, fat, bsd, and some more. It also has a complete interactive commandline. Is more of a 'boot shell'. I gave you a like. Go try it. : >

Only the GRUB loader itself sits on an sbsolute sector.
So via the ominopotent symlink, you would need to do...nothing after you install a new kernel to boot. With initrd, you would have to make another initrd. But actually you would need one for every kernel. (I've kept up to 6 active on my box at one time...) This sucks...and it's why we all compile everything we really need static.

BTW if your initrd images gets corrupted or lost, you are paddling up fecal creek with your hands. Let's not forget how much fun it is if you get a library mismatch on your initrd boot utils. Hope you compiled them all static.

Loading system critcal kernel modules should be as straight forward as the feature I'm requesting. The boot loader puts it in memory. The kernel loads it from memory. It's all good.

Oh jesus...what have I missed? (I've been off the list a long time)

I hope so...you're still imortalized in the POSIXness script. ; > LRP is unfortunatly dated to the point it doesn't really rock right now. : < It's simply cool. : > If I ever get to work on LRP v5.0... (Hell I never got to release V4.0... : P) it will rock like nothing else ever has.

But since you bring up LRP, let's get down into the minimal space issues. It's a living hell tring to maintain kernels in an OS distibution and a primary reason is that you have to play roulette deciding what to make static. Then when you get it good, you have to fuck it all up to squeeze it onto a boot disk. Adding what I want fixes this quite well.

With the work I've done with LRP and embedded systems, I doubt many people out there have played with the initrd more then me. If you haven't noticed I'm back on the list because I just posted a new version of initrd dynamic with tmpfs. It's a lifesaver for initrd, since now the FS 'creatation' is 100% dynamic the the 'image' can be multiple tar.gz archives.

However as clean as it is, it's still not something I would want to load critical kernel parts for boot.

Saying one should use only initrd for this is like saying the kernel should not handle starting MD devices at all.

As a programming ethic it sounds good but I think few programmer have actually used some of these ideals in 'real world' system adminstration. Try finding yourself staring at a mission critical server that you now can not get to boot at 03:00 and you can't get
out to the net. That sort of experience changes your perspective. Been there. Many times. No fun.

So just code this feature for me. Please. Make Dave happy. ; >

Dave

The time is now 22:19 (Totalitarian)  -  http://www.ccops.org/clock.html