Kernel source tree splitting

Eh, Linus won't be happy making a bunch of tarballs. I've made it less work if you read the message here...

The message mirrored at:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105173077417526&w=2

Shows my pre-thought on this subject. I thought a bit more, and began to come up with a simple sketch to lead the way in case anyone becomes interested.

First off, the kernel tarballs would be built by a script that splits the source tree apart appropriately and tar's it up. How this is done is explained.

Second off, there's always a script to download that runs wget and gets the source tree from which it was downloaded. The whole thing. As in, every tarball is downloaded and untar'd for the user, assembling the full kernel source tree (as it would be if you untar'd it now).

Now, I explained LOD's in that message above in small detail. But, for clarity, LOD's are files which explain which pieces of source in the kernel tree belong to the LOD; what gets added to the config; where their makefiles are; what config options depend on other linux options; and what groups these LOD's are in.

A command such as make disttree should read the LOD's, split apart each linux option, tar 'em together, and then compress the tar's. Then Linus could just scp the new directory of tar's and a script up.

As for download, the script that goes up can be downloaded (duh), and then run (... why do I bother?). Now this script would run in "dumb mode" (unless the user tells it not to maybe?) and rip down the whole tree, untar it, and rebuild the original source tree. I think. I'm not sure, I really haven't tried yet. I'll tell you how it works after it's implimented, if ever that happens. This would likely require wget.

Of course there's always the ftp method. Go download the pieces you want, untar 'em, copy 'em to the same directory, and the build system adjusts. but newbies and developers, for completely opposite reasons, will want to use the script in dumb mode.

For experienced users, this will make configuration somewhat easier, as the user can avoid being prompted for irrelavent drivers. This is just a concept idea, not a fully thought-out idea. What do you think?

--Bluefox Icy

Hi,

I'm probably misreading this...but,

Have you tried this yet? Does it modify/customize all Kconfig and Makefiles for the selected tree splits?

A few days ago, in one tree, I rm-ed arch/{all that I don't need} and drivers/{all that I don't need}.
After that I couldn't run "make *config" because it wants all of those files, even if I don't want them.

So there are many edits that needed to be done in lots of Kconfig and Makefiles if one selectively pulls or omits certain sub-directories.

On Wed, 30 Apr 2003 19:46:13 -0400 rmoser <[email protected]> wrote:

| Eh, Linus won't be happy making a bunch of tarballs.
| I've made it less work if you read the message here...
| 
| The message mirrored at:
| 
| http://marc.theaimsgroup.com/?l=linux-kernel&m=105173077417526&w=2
| 
| Shows my pre-thought on this subject.  I thought a bit more,
| and began to come up with a simple sketch to lead the
| way in case anyone becomes interested.
| 
| First off, the kernel tarballs would be built by a script
| that splits the source tree apart appropriately and tar's it
| up.  How this is done is explained.
| 
| Second off, there's always a script to download that runs
| wget and gets the source tree from which it was downloaded.
| The whole thing.  As in, every tarball is downloaded and
| untar'd for the user, assembling the full kernel source
| tree (as it would be if you untar'd it now).
| 
| Now, I explained LOD's in that message above in small
| detail.  But, for clarity, LOD's are files which explain
| which pieces of source in the kernel tree belong to the
| LOD; what gets added to the config; where their makefiles
| are; what config options depend on other linux options;
| and what groups these LOD's are in.
| 
| A command such as `make disttree` should read the LOD's,
| split apart each linux option, tar 'em together, and
| then compress the tar's.  Then Linus could just scp the
| new directory of tar's and a script up.
| 
| As for download, the script that goes up can be
| downloaded (duh), and then run (... why do I bother?).
| Now this script would run in "dumb mode" (unless the
| user tells it not to maybe?) and rip down the whole
| tree, untar it, and rebuild the original source tree.
| I think.  I'm not sure, I really haven't tried yet.
| I'll tell you how it works after it's implimented, if
| ever that happens.  This would likely require wget.
| 
| Of course there's always the ftp method.  Go download
| the pieces you want, untar 'em, copy 'em to the same
| directory, and the build system adjusts.  but newbies
| and developers, for completely opposite reasons, will
| want to use the script in dumb mode.
| 
| For experienced users, this will make configuration
| somewhat easier, as the user can avoid being prompted
| for irrelavent drivers.  This is just a concept idea,
| not a fully thought-out idea.  What do you think?
| 
| --Bluefox Icy

--
~Randy

*********** REPLY SEPARATOR ***********

I didn't try it. It would require knowledge of all interdependencies between modules (i.e. ide-scsi is part of ide. ide-scsi depends on scsi), also all source files that belong to each config option would likely need to be understood by the persons working to do this, and also the entire build system semantics would need to be designed to work in pieces. Assuming this is ever done.

It goes like this:

make *config reads kernel-tree/options/foo.lod
make *config gets to configuration baz in bar.lod
make *config sees baz needs foo
make *config lists baz.
make *config sees biz needs data
make *config refuses to show biz
make missing-depends shows a list of unselectable options and -----------selecting one tells which kernel option is needed. make bzImage reads through kernel-tree/options/ and finds which -----------makefiles to call (current makes have these embedded) make bzImage builds a kernel.
make modules reads through kernel-tree/options/ and finds which -----------makefiles to call.
make modules builds a kernel.

That WILL break something.

The main Makefile will have to be redone. So would the kconfig things (make config/menuconfig/xconfig).

The extra plus to this is that other people can steal the build system for other projects lol.

or "That does break something" (when I tried it).

I seem to try for simple solutions when possible and feasible.

In this case, if I were doing it, I would try changing (e.g.) in arch/i386/kernel/Kconfig, this line:
source "drivers/eisa/Kconfig"
to
optsource "drivers/eisa/Kconfig"
where optsource means that the file is optional -- if not found, ignore it. And then see what happens, how far it can go, what the next problem is....

If this could be made to work, then entire subdirectories/subsystems could be optional.

~Randy

*********** REPLY SEPARATOR ***********

Yeah but hat about things that came out of nowhere, i.e. a piece of hardware compes with a linux driver source?

It would be *really* cool if the Makefile dependencies actually worked. It's a very little known fact but if you are in an RCS or SCCS (and BK looks like SCCS to make) source tree and the files are not checked out, you can just say

make

and make will look for a makefile, if there isn't one but there is a SCCS/s.[Mm]akefile it will check it out, look at the dependencies and start checking those out and keep doing it to satisfy the target.

It's a really pleasant way to work, the "make clobber" target "cleans" all the source so it isn't checked out, the directory is nice and empty. This makes it easy to see stuff you still need to check in or think about. It's definitely an old timer way of working, I'm pretty sure that the original Unix was done this way but just because it is old doesn't mean it is bad. Opinions differ on that :)

Here's a make in a cleaned BK source tree:

$ lf
Notes/  gnu/  ide/   port/  tmp/       utils/  win32/
SCCS/   gui/  mdbm/  t/     tomcrypt/  web/    zlib/
$ make p
get   SCCS/s.Makefile
Makefile 1.326: 684 lines
make CFLAGS="-g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes " all
get   SCCS/s.abort.c
abort.c 1.11: 157 lines
get   SCCS/s.unix.h
unix.h 1.48: 79 lines
get   SCCS/s.bkd.h
bkd.h 1.68: 157 lines
get   SCCS/s.lib_tcp.h
lib_tcp.h 1.12: 21 lines
get   SCCS/s.logging.h
logging.h 1.10: 80 lines
get   mdbm/SCCS/s.common.h
mdbm/common.h 1.6: 111 lines
get   mdbm/SCCS/s.mdbm.h
mdbm/mdbm.h 1.11: 426 lines
get   mdbm/SCCS/s.tune.h
mdbm/tune.h 1.3: 66 lines
get   SCCS/s.mmap.h
mmap.h 1.5: 30 lines
get   SCCS/s.purify.h
purify.h 1.31: 114 lines
get   SCCS/s.range.h
range.h 1.18: 42 lines
get   SCCS/s.rcs.h
rcs.h 1.6: 60 lines
get   SCCS/s.liblines.h
liblines.h 1.5: 81 lines
get   SCCS/s.resolve.h
resolve.h 1.36: 178 lines
get   SCCS/s.sccs.h
sccs.h 1.497: 1171 lines
get   SCCS/s.system.h
system.h 1.55: 157 lines
get   tomcrypt/SCCS/s.mpi-config.h
tomcrypt/mpi-config.h 1.1: 87 lines
get   tomcrypt/SCCS/s.mpi.h
tomcrypt/mpi.h 1.1: 225 lines
get   tomcrypt/SCCS/s.mycrypt_cfg.h
tomcrypt/mycrypt_cfg.h 1.4: 174 lines
get   tomcrypt/SCCS/s.mpi-types.h
tomcrypt/mpi-types.h 1.1: 19 lines
get   tomcrypt/SCCS/s.mycrypt.h
tomcrypt/mycrypt.h 1.2: 913 lines
get   SCCS/s.zgets.h
zgets.h 1.5: 23 lines
get   zlib/SCCS/s.deflate.h
zlib/deflate.h 1.4: 318 lines
get   zlib/SCCS/s.infblock.h
zlib/infblock.h 1.4: 39 lines
get   zlib/SCCS/s.infcodes.h
zlib/infcodes.h 1.4: 27 lines
get   zlib/SCCS/s.inffast.h
zlib/inffast.h 1.4: 17 lines
get   zlib/SCCS/s.inffixed.h
zlib/inffixed.h 1.2: 151 lines
get   zlib/SCCS/s.inftrees.h
zlib/inftrees.h 1.4: 58 lines
get   zlib/SCCS/s.infutil.h
zlib/infutil.h 1.4: 98 lines
get   zlib/SCCS/s.trees.h
zlib/trees.h 1.2: 128 lines
get   zlib/SCCS/s.zconf.h
zlib/zconf.h 1.4: 279 lines
get   zlib/SCCS/s.zlib.h
zlib/zlib.h 1.4: 893 lines
get   zlib/SCCS/s.zutil.h
zlib/zutil.h 1.3: 220 lines
cc -g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes  -Izlib -Imdbm -c abort.c -o abort.o
get   SCCS/s.adler32.c
adler32.c 1.15: 144 lines
cc -g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes  -Izlib -Imdbm -c adler32.c -o adler32.o
get   SCCS/s.admin.c
admin.c 1.122: 453 lines
cc -g -O2 -Wall -Wno-parentheses -Wno-char-subscripts -Wno-format-y2k -Wstrict-prototypes  -Izlib -Imdbm -c admin.c -o admin.o

---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

*********** REPLY SEPARATOR ***********

[cut]

....

Larry... what the heck are you talking about? No really you lost me :)

Are you talking about hitting a button and flushing all the code but what's new, or about actually making different tarballs cause the source tree to auto-adapt to what's in it?

What is RCS/SCCS

--Bluefox Icy

Larry> It's a very little known fact but if you are in an RCS Larry> or SCCS (and BK looks like SCCS to make) source tree and the Larry> files are not checked out, you can just say

Larry> make

Larry> and make will look for a makefile, if there isn't one but there Larry> is a SCCS/s.[Mm]akefile it will check it out, look at the Larry> dependencies and start checking those out and keep doing it to Larry> satisfy the target.

That has never worked for me with Linus' or Marcello's trees. It always falls down at some point or other.

Usually it fails looking for something under include, 2.5 also dies in make oldconfig.

-JimC

Indeed, I ran across the same thing a while back. Would be *really* nice to fix, if only so some poor sod over a modem can download a smaller tarball, or save some diskspace.

M.

Tried doing this a while ago with my own cvs server. I'd download the tarball untar it into the cvs dump. I made a modules file tha that described all kinds of possible modules and then modules that utilized those and so on until you could just do co -z9 linux-i386 to get a all the files required to make any i386 linux ...it was mostly to just separate the archs, not unneeded drivers. Of course i quickly ran into the make not working without all the other files from the other archs and i came to the conclusion you couldn't do this without editing certain build scripts and files (and since I didn't want the files to be altered in case people would eventually want to call security into question) i killed the whole idea.

This idea comes and goes like many others. As far as modular programming goes the kernel isn't written to be very modular, either that or it is and the build system just looks over that fact. It would be interesting to see how hard it would be to have the current config to not only generate its .config but also a list of commands that an autoconf created ./configure can read and use to generate the makefiles we then use to build the kernel. it could also help in enforcing minimum versions instead of hoping the user does it before complaining to people who could be doing better things with their time. Anyways, i'm sure this idea has been coined and dropped before as well. So it's back to the way things are done like normal.

This might also help highlight files that are not really in a natural place in the tree. If people building for x86 platforms start finding they need a file that is in the sparc part of the tree then that suggests that said file is probably at least partly non-arch specific. This would also apply to, say, the audio and video parts of the drivers section of the tree.

While these things do get reported on here occasionally, and fixed, making it possible to wholesale remove parts of the tree that people naturally assume you *shouldn't* need for a particular config ought to show these up more easily.

Of course it adds another variable into the mix when people report build problems....

Peter

On Wed, 30 Apr 2003 17:21:02 -0700 "Randy.Dunlap" <[email protected]> wrote:

| Hi,
| 
| I'm probably misreading this...but,
| 
| Have you tried this yet?  Does it modify/customize all Kconfig
| and Makefiles for the selected tree splits?
| 
| A few days ago, in one tree, I rm-ed arch/{all that I don't need}
| and drivers/{all that I don't need}.
| After that I couldn't run "make *config" because it wants all of
| those files, even if I don't want them.
| 
| So there are many edits that needed to be done in lots of
| Kconfig and Makefiles if one selectively pulls or omits certain
| sub-directories.

and on 2003-04-30 rddunlap wrote:

| I seem to try for simple solutions when possible and feasible.
| 
| In this case, if I were doing it, I would try changing (e.g.) in
| arch/i386/kernel/Kconfig, this line:
| source "drivers/eisa/Kconfig"
| to
| optsource "drivers/eisa/Kconfig"
| where optsource means that the file is optional -- if not found,
| ignore it.  And then see what happens, how far it can go,
| what the next problem is....
| 
| If this could be made to work, then entire subdirectories/subsystems
| could be optional.

So I did a proof-of-concept version of this, without modifying any source code. I rm-ed arch/<many>, drivers/<many>, and fs/<many> and then wrote a shell script that looks for missing dirs, Kconfigs, and Makefile.lib files and puts empty ones back in their places. The shell script only works for what I rm-ed, but it could be made smarter if anyone wants to pursue this. (attached)

After doing that I was able to build and boot that kernel, so it (concept) did work.

For a kernel source tree that hadn't been built/compiled in, the size was reduced from roughly 200 MB down to roughly 133 MB.

~Randy

| On Wed, 30 Apr 2003 19:46:13 -0400 rmoser <[email protected]> wrote:
| 
| | Eh, Linus won't be happy making a bunch of tarballs.
| | I've made it less work if you read the message here...
| | 
| | The message mirrored at:
| | 
| | http://marc.theaimsgroup.com/?l=linux-kernel&m=105173077417526&w=2
| | 
| | Shows my pre-thought on this subject.  I thought a bit more,
| | and began to come up with a simple sketch to lead the
| | way in case anyone becomes interested.
| | 
| | First off, the kernel tarballs would be built by a script
| | that splits the source tree apart appropriately and tar's it
| | up.  How this is done is explained.
| | 
| | Second off, there's always a script to download that runs
| | wget and gets the source tree from which it was downloaded.
| | The whole thing.  As in, every tarball is downloaded and
| | untar'd for the user, assembling the full kernel source
| | tree (as it would be if you untar'd it now).
| | 
| | Now, I explained LOD's in that message above in small
| | detail.  But, for clarity, LOD's are files which explain
| | which pieces of source in the kernel tree belong to the
| | LOD; what gets added to the config; where their makefiles
| | are; what config options depend on other linux options;
| | and what groups these LOD's are in.
| | 
| | A command such as `make disttree` should read the LOD's,
| | split apart each linux option, tar 'em together, and
| | then compress the tar's.  Then Linus could just scp the
| | new directory of tar's and a script up.
| | 
| | As for download, the script that goes up can be
| | downloaded (duh), and then run (... why do I bother?).
| | Now this script would run in "dumb mode" (unless the
| | user tells it not to maybe?) and rip down the whole
| | tree, untar it, and rebuild the original source tree.
| | I think.  I'm not sure, I really haven't tried yet.
| | I'll tell you how it works after it's implimented, if
| | ever that happens.  This would likely require wget.
| | 
| | Of course there's always the ftp method.  Go download
| | the pieces you want, untar 'em, copy 'em to the same
| | directory, and the build system adjusts.  but newbies
| | and developers, for completely opposite reasons, will
| | want to use the script in dumb mode.
| | 
| | For experienced users, this will make configuration
| | somewhat easier, as the user can avoid being prompted
| | for irrelavent drivers.  This is just a concept idea,
| | not a fully thought-out idea.  What do you think?
| | 
| | --Bluefox Icy

*********** REPLY SEPARATOR ***********

Yes. Well the build system (kernel configuration) would be modified to instead of having a list of Kconfigs and dir's and Makefile.libs and such, be able to scan a directory of files which tell it about those things. At least, in my design it would.

Well that's good.

............... You know. Maybe it's not enough to split into categories.
Maybe it should be categories and categorical breakdowns. I can see 3.0 or something (maybe even 2.8) reaching 1 gig.

--Bluefox Icy