Announce: Kernel Build for 2.5, Release 1.1 is available.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Content-Type: text/plain; charset=us-ascii

Release 1.1 of kernel build for kernel 2.5 (kbuild 2.5) is available.
http://sourceforge.net/projects/kbuild/, Package kbuild-2.5, download release 1.1.

http://marc.theaimsgroup.com/?l=linux-kernel&m=99725412902968&w=2
contains information about the base release.

Changes from Release 1.

Upgrade to kernel 2.4.8. Nice to see how simple the DRM Makefile is now.

Correct a race when parallel building the global makefile, not all objects were being recognised as targets and were not recognised as candidates for recompile.

Replace hand coded rules with side_effect().

Document kbuild targets and C to assembler conversions. As always, Documentation/kbuild/kbuild-2.5.txt is your friend.

Remove the assembler() command. kbuild now works out if the source is .c or .S, no need for human intervention.

If you explicitly make foo.i or foo.s then kbuild automatically generates the required rules with the same flags as the corresponding .o file. Useful for debugging pre-processor or assembler problems, especially when gcc -save-temps does not work with multiple directories.

Standard generation of .s from .c files, where a .s file is required according to make. This includes tracking the dependencies of the .c file.

That last change lets me solve a long standing problem with kbuild 2.4. Every architecture has Assembler that requires offsets of fields within C structures or the mapping of C names to numbers. Assembler cannot include the C definitions so we need a mapping from C constructs to Assembler numbers. Every architecture has handled this problem in a different way, none of the methods are 100% accurate nor dependable.

i386 hard codes the offsets into the assembler code and hopes that the structure definitions never change.

Alpha uses a C program that generates the text for the assembler. This does not work in a cross compile environment because it assumes that HOSTCC == CC.

Cris generates a chunk of assembler from C then uses .include instead of #include, with some fancy conditional selection.

Mips, parisc, ppc, sparc generate assembler then extract and reformat lines from the assembler. This works in both local and cross compile mode and is getting close to the correct way of doing it. But it still has problems, see below.

IA64 in 2.4 is particularly loathsome. It uses different methods in native and cross compile modes, when the cross compile version would do for both. It ships a copy of the generated asm/offsets.h which is totally unreliable because the real offsets.h depends on the user's .config. To add insult to injury, offsets.h is included in processor.h and ptrace.h on ia64 which means that it pollutes almost every C file.

None of the above methods handle dependency checking at all. PPC makes an attempt but it is manually defined and is incomplete, no other arch even makes an attempt. All architectures assume that the user always runs make dep after any config changes that affect the assembler offsets. If the user forgets to run make dep and the assembler and C values do not match - oops.

kbuild 2.5 has a solution which works in all modes, is standard across all architectures and automatically tracks dependency changes. No more room for human error.

* Create arch/$(ARCH)/offsets.c containing code like this, from arch/i386/offsets.c. This should be the standard format on all architectures, the only difference should be the list of fields to generate.

/*
* Generate definitions needed by assembly language modules.
* This code generates raw asm output which is post-processed to extract
* and format the required data.
*/

  #include <linux/types.h>
  #include <linux/stddef.h>
  #include <linux/sched.h>

/* Use marker if you need to separate the values later */

  #define DEFINE(sym, val, marker) \
    asm volatile("\n-> " #sym " %c0 " #val " " #marker : : "i" (val))

  int
  main(void)
  {
    DEFINE(state,        offsetof(struct task_struct, state),);
    DEFINE(flags,        offsetof(struct task_struct, flags),);
    DEFINE(sigpending,   offsetof(struct task_struct, sigpending),);
    DEFINE(addr_limit,   offsetof(struct task_struct, addr_limit),);
    DEFINE(exec_domain,  offsetof(struct task_struct, exec_domain),);
    DEFINE(need_resched, offsetof(struct task_struct, need_resched),);
    DEFINE(tsk_ptrace,   offsetof(struct task_struct, ptrace),);
    DEFINE(processor,    offsetof(struct task_struct, processor),);
    DEFINE(ENOSYS,       ENOSYS,);
    return 0;
  }

* When that code is compiled from .c to .s using CC for the target system, the generated Assembler contains lines like this

-> state 0 offsetof(struct task_struct, state)
-> flags 4 offsetof(struct task_struct, flags)
-> sigpending 8 offsetof(struct task_struct, sigpending) -> addr_limit 12 offsetof(struct task_struct, addr_limit) -> exec_domain 16 offsetof(struct task_struct, exec_domain) -> need_resched 20 offsetof(struct task_struct, need_resched) -> tsk_ptrace 24 offsetof(struct task_struct, ptrace) -> processor 52 offsetof(struct task_struct, processor) -> ENOSYS 38 ENOSYS

interspersed with Assembler declarations and blank lines.

* In arch/$(ARCH)/Makefile.in, define user commands to extract the '->' lines from offsets.s and reformat as required for your architecture. arch/i386/Makefile.in contains

# Convert raw asm offsets into something that can be included as

  # assembler definitions.  It converts
  #   -> symbol value source
  # into
  #   symbol  = value /* 0xvalue source */

user_command(offsets.h

	  ($(objfile offsets.s))
	  (set -e;
	    (echo "#ifndef __ASM_OFFSETS_H__";
	     echo "#define __ASM_OFFSETS_H__";
	     awk "/^->/{
	       sym = \$$2;
	       val = \$$3;
	       \$$1 = \"\";
	       \$$2 = \"\";
	       \$$3 = \"\";
	       printf(\"%-20s = %3d\t\t\t/* 0x%x\t%s */\n\",
		 sym, val, val, \$$0)
	     }";
	     echo "#endif";
	    ) < $< > $(@D).tmp_$(@F);
	    cmp -s $(@D).tmp_$(@F) $@ || mv $(@D).tmp_$(@F) $@
	  )
	  ()
	  )

The awk code is a little complicated because it has to cope with both shell (" -> \", $ -> \$) and make ($ -> $$) quoting rules but it works, and only has to be coded once. The output from awk is written to a .tmp_ file first, the result is compared with the previous version (if any) and only if they are different is offsets.h updated. The compare and update avoids spurious Assembler recompiles, most config changes do not affect offsets.h.

  The final output is in arch/$(ARCH)/offsets.h.  On i386, offsets.h
  contains

  #ifndef __ASM_OFFSETS_H__
  #define __ASM_OFFSETS_H__
  state        =   0 /* 0x0  offsetof(struct task_struct, state) */
  flags        =   4 /* 0x4  offsetof(struct task_struct, flags) */
  sigpending   =   8 /* 0x8  offsetof(struct task_struct, sigpending) */
  addr_limit   =  12 /* 0xc  offsetof(struct task_struct, addr_limit) */
  exec_domain  =  16 /* 0x10 offsetof(struct task_struct, exec_domain) */
  need_resched =  20 /* 0x14 offsetof(struct task_struct, need_resched) */
  tsk_ptrace   =  24 /* 0x18 offsetof(struct task_struct, ptrace) */
  processor    =  52 /* 0x34 offsetof(struct task_struct, processor) */
  ENOSYS       =  38 /* 0x26 ENOSYS */
  #endif

My aim is to standardize the format of the output from offsets.s and do any arch specific processing in the makefile. In the long run this will be easier than having eight different ways of generating assembler output. I have included a marker parameter in the DEFINE macro, marker appears at the end of the '->' line, just in case an architecture needs some indicator in order to split the offsets into multiple files.

* Do not, under any circumstances, include offsets.h in any files that are used by C source, offsets.h must only be used in Assembler code. IA64 made the mistake of including offsets.h in processor.h and ptrace.h so they have to ship the generated offsets.h to avoid C compile errors on parallel builds. But the version that is shipped is incorrect, it does not match the user's config, a potential source of human error.

* Do not ship arch/$(arch)/offsets.h nor include it in any patches. Add offsets.h to your don't diff list.

* Specify arch/$(ARCH)/offsets.h as a pre-requisite of only the objects that need it in order to assemble. arch/i386/kernel/Makefile.in has

  $(objfile entry.o): $(objfile /arch/$(ARCH)/offsets.h)

You only need to specify this dependency for the Assembler files that use the generated offset values.

When kbuild 2.5 wants to compile entry.o, it checks if offsets.h is up to date. user_command() says that offsets.h depends on offsets.s. offsets.s implicitly depends on offsets.c, there is no need for an explicit rule. Not only does offsets.s directly depend on offsets.c, it indirectly depends on all the files included by offsets.c, either directly or indirectly, and on all the CONFIG_ settings in offsets.c and the included files.

If anything that affects config.s has changed it is rebuilt, the '->' lines are extracted and, if offsets.h has changed, it is replaced. That will force a recompile of the affected assembler objects. The result is a standard, reliable and, above all, an automatic method of converting C values to Assembler lines.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.3 (GNU/Linux)
Comment: Exmh version 2.1.1 10/15/1999

iD8DBQE7dUkji4UHNye0ZOoRAiS6AJsEyMYRmEReH09ZFThu5bf7rVPDtQCgjhAY w+JAoVEaOASjcl+kHL34fiE=
=fuW1
-----END PGP SIGNATURE-----

I'm sorry, the ARM version of GCC does not support %c0 in a working state. The way we generate the offsets on ARM is here to stay for the next few years until GCC 3 has stabilised well enough for use with the kernel, and the ARM architecture specifically.

Please don't rely on %c0 working.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

If you have to use %0 instead of %c0, that is all right. Just remove the extra '$' in the Makefile as arch/arm/tools/Makefile already does. I would still like to see arm in 2.5 using the same style as i386, including printing the offset in decimal and hex together with the comment. But that is just style, the important thing is to generate assembler offsets so that kbuild can correctly track the dependencies.

I should think it can be made to work in 2.95.4. Did you try the patch I sent you a few months ago?

p.

Amazing response to the problem here. ;)

No - I've not had a need to rebuild gcc yet, and the patch is low priority since the kernel has to build with the compilers that people already have, not the bleeding edge.

Sorry, I don't do gcc, as I've explained before.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

Okay, I think I found some wierd problem.

$ make -f Makefile-2.5 ARCH=i386
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar'
Generating global Makefile
phase 1 (find all inputs)
phase 2 (evaluate selections)
pp_makefile2: drivers/char/defkeymap.o is selected but is not part of vmlinux, missing link_subdirs?
pp_makefile2: drivers/char/pc_keyb.o is selected but is not part of vmlinux, missing link_subdirs?
make: *** [/home/trini/work/kernel/kbuild/linux-2.4.8/.tmp_targets] Error 2

Prior to this i did make -f Makefile-2.5 ARCH=i386 oldconfig and said 'n' to all new questions.

Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Hi,

I'm wondering that you don't mention m68k, because we generate dependencies...
(Has nothing to do with kbuild, I'm just curious. :) )

bye, Roman

Because I did a quick sweep through the arch makefiles looking for the word 'offsets'. That is part of the problem, the offsets file has different names in some architectures. Most call it offsets, PPC uses mk_def and ppc_defs, arm uses getconstants and constants, m68k uses m68k_defs. It does not help when the code that generates the asm constants is in different directories on some architectures.

Now you know why I want a standard method, with a standard name and standard directory location.

[snip]

Okay, I've played with this a bit on PPC, and got it working to boot :) Now here's what I see as the slight problems with it. At least on PPC what we do is generate the offset bits, and then have another file, arch/ppc/kernel/ppc_asm.h include that file and have some other useful macros for .S files. So any of the .S files which include ppc_asm.h would need an additional
extra_aflags(foo.o $(src_includelist /arch/$(ARCH)))
or extra_aflags_all($(src_includelist /arch/$(ARCH))
or my ugly workaround for the moment of #include <../arch/ppc/offsets.h> which is wrong, but works.

But aside from that, it's nice that the file isn't generated every make like it was w/ the user_command workaround :)

Tom Rini (TR1265)
http://gate.crashing.org/~trini/

That will be required for all asm code that includes offsets.h, on all architectures, I doubt there will be more than 10 on any arch.

The alternative of having code in some arch directory updating include/asm-$(ARCH)/offsets.h is worse. It is a terrible design to have code in one makefile updating files in another directory. It is a layer violation which is always a bad idea.

With sensible (i.e. non-recursive) makefiles, surely this is far more acceptable?

--
dwmw2

No. The aim is for a user to look at the makefile in a directory and know everything that is created in that directory. If you allow creation of a file in one directory but storing it in another then you have to search all makefiles to find out what is created in any directory. Horrible!

I was very careful to code the select() and objlink() and related commands so they can only create files in the current directory, to enforce a clean design. You can read a file from another directory but you cannot write a file to another directory.

Fine. But what's stopping us from putting a makefile in the asm-$(ARCH)/include directory? There are already automatically generated files in there. What have you already done for stuff like asm-arm/mach-types.h?

--
dwmw2

Can we have a makefile in include/asm-$(ARCH) then please?

I think stuff like:

#include "../../../../arch/arm/constants.h"

or

#include "../../../../arch/arm/tools/mach-types.h"

is even more disgusting.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

Absolutely agree, which is why there are so many FIXME comments in the Makefile.in files, to remove all the relative includes. If a source needs a non-standard include ioutside the current directory then it should #include <foo.h> and the makefile should point to the directory containing foo.h. Knowledge about non-standard include files should be in the makefile, not embedded in lots of source code.

That still leaves the question of where the generated files should be written to. This is getting into philosophy rather than technical, one man's preference is another's pet hate. My preference is not to write to any directory under include. At the moment kbuild 2.5 has a clean separation between shipped files under /include and generated files under arch/$(ARCH). There are only 5 generated include files and they are only present for historical reasons.

include/linux/modversions.h
include/linux/version.h
include/linux/uts_release.h
include/linux/autoconf.h
include/linux/compile.h

Writing to include/asm is a problem for kbuild 2.5. When using separate source and object trees, the asm-$(ARCH) symlink cannot be created in the source tree, users may want to generate multiple architectures from a single source tree. .tmp_include/src_000/asm is created in the object tree, pointing at the relevant source directory and gcc scans .tmp_include/src_000, not the source tree.

If you write to asm/foo.h then you are updating the source tree. I can work around this by creating two sets of symlinks for include paths, one pointing to the object directory, one to the source and telling gcc to scan the object first. But that will make the generated commands even longer and will slow gcc down.

My preference is to generate files in the arch subtree. The source code does #include <foo.h> and the makefiles have lines like this

  extra_aflags(entry.o -I arch/$(ARCH))

Yes, you have to add extra_aflags on every object that uses the generated include file. But you already have to specify a dependency from the object to the generated file to ensure that parallel make waits until the file is generated. Doing two lines instead of one for the affected objects is no big deal.

  $(objfile entry.o): $(objfile /arch/$(ARCH)/offsets.h)
  extra_aflags(entry.o -I arch/$(ARCH))

BTW, mach-types.h is going to be a problem wherever it is stored. Several .h files include the generated file so all objects that include these .h files are dependent on mach-types.h being generated first.

include/asm-arm/arch-ebsa285/time.h:#include <asm/mach-types.h> include/asm-arm/arch-ebsa285/uncompress.h:#include <asm/mach-types.h> include/asm-arm/arch-ebsa285/irq.h:#include <asm/mach-types.h> include/asm-arm/arch-ebsa285/system.h:#include <asm/mach-types.h> include/asm-arm/arch-ebsa285/irqs.h:#include <asm/mach-types.h> include/asm-arm/arch-arc/ide.h:#include <asm/mach-types.h> include/asm-arm/arch-sa1100/uncompress.h:#include <asm/mach-types.h> include/asm-arm/arch-sa1100/ide.h:#include <asm/mach-types.h> include/asm-arm/arch-sa1100/irq.h:#include <asm/mach-types.h> include/asm-arm/arch-sa1100/hardware.h:#include <asm/mach-types.h>

I do not want to reinvent make dep because of the fundamental design problems with that approach. If any generated file depends on .config then it must be recreated after any config changes or patches but 99.9% of users do not do that. We must not rely on human intervention to update kernel files, it must be automatic. This requires that make be told the truth about the dependency graph. mach-types.h is going to require some more work, no matter where it is stored.

Hopefully not much more than 10, I hope.

Right. I figured it'd be worth pointing out, if nothing else.

Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Bug fixes against 2.4.8 + kbuild-2.5-2.4.8-1. I will put up a -2 later today.

Index: 8.7/drivers/sound/emu10k1/Makefile.in

@@ -3,8 +3,9 @@
 #
 # 12 Apr 2000 Rui Sousa
 
-objlink(emu10k1.o efxmgr.o emuadxmg.o hwaccess.o irqmgr.o joystick.o main.o
-	midi.o mixer.o passthrough.o recmgr.o timer.o voicemgr.o)
+objlink(emu10k1.o audio.o cardmi.o cardmo.o cardwi.o cardwo.o ecard.o efxmgr.o
+        emuadxmg.o hwaccess.o irqmgr.o joystick.o main.o midi.o mixer.o
+        passthrough.o recmgr.o timer.o voicemgr.o)
 
 select(CONFIG_SOUND_EMU10K1 emu10k1.o)
 

Index: 8.7/scripts/pp_makefile2.c

@@ -754,7 +754,10 @@ int special_oais(PP_DIRENT *dirent, PP_D
   }
   if (dirent->value.db) {
     DB *db = db_list[dirent->value.db];
-    if (!dirent->istarget) {
+    /* FIXME: when the go faster stripes are added, make sure that
+     * dirent->istarget and db->istarget are synced earlier.
+     */
+    if (!db->istarget && !dirent->istarget) {
       return(0);
     }
     switch (type) {

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Content-Type: text/plain; charset=us-ascii

I have replaced kbuild-2.5-2.4.8-1.bz2 with 2.5-2.4.8-2.bz2 under
http://sourceforge.net/projects/kbuild/, Package kbuild-2.5, download release 1.1. The patch is against a clean 2.4.8 kernel.

Changes from -1.

Update emu10k1/Makefile.in to the full 2.4.8 list of objects.

Correct a bug that dropped some objects from the build list.

Rework the generation of assembler offsets from C. It uses %0 instead of %c0, RMK says that %c0 does not work reliably on arm. The sample awk code handles %0 and has been cleaned up. The output to a .tmp file followed by compare has been removed, it caused problems for make install. More documentation, including settings for include files.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.3 (GNU/Linux)
Comment: Exmh version 2.1.1 10/15/1999

iD8DBQE7dgZOi4UHNye0ZOoRAj/rAJ48whc56FMuUpf9SIu4n/lwx8XUGACfcT6S Q34ZBrZfXQKJPjzbyTAQ+g0=
=rAh+
-----END PGP SIGNATURE-----

This is bad -- 'c' requests an address constant (CONSTANT_ADDRESS_P if you're playing from home), and as defined this constant must be valid in an instruction.

Such things are by nature target specific. On ia64, for instance, there are *no* valid address constants, since all valid memory addresses are simple registers.

You can do what you want here with just "%0".

r~