Skip to contentSkip to conversation

kbuild: possible regression?

DISCUSSION Jan Altenberg 2 participants 7 messages Oct 31, 2007 – Dec 11, 2007 Kbuild

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm not quite sure if this might be a regression, but I recognized a change to kbuild's behaviour, which causes some of my build scripts to fail.

The build scripts do:

return system(('make -C %s O=%s ARCH=%s CROSS_COMPILE=%s '+

                      'oldconfig %s %s < /dev/null || exit %i') % 
                      (srcdir, 
                       builddir, 
                       arch,
                       crosscompile, 
                       target,
                       modules_target,
                       FAILED_RETCODE))

which results in something like:

make -C /here/workdir/linux-2.6/common/src O=/here/workdir/linux-2.6/common/build/build \
ARCH=i386 CROSS_COMPILE=/opt/i686-linux/bin/i686-linux- \
oldconfig bzImage < /dev/null || exit 1

In the past, oldconfig was the first target, which has been handled.

Now, bzImage seems to get handled at first. That causes my build scripts to fail, because the bzImage target does a silentoldconfig, which doesn't allow console redirection.

[...]

Kernel log buffer size (16 => 64KB, 17 => 128KB) (LOG_BUF_SHIFT) [14] 14 Control Group support (CGROUPS) [N/y/?] (NEW) aborted!

Console input/output is redirected. Run 'make oldconfig' to update configuration.

make[5]: *** [silentoldconfig] Error 1
make[4]: *** [silentoldconfig] Error 2
make[3]: *** [include/config/auto.conf] Error 2
make[2]: *** [sub-make] Error 2
make[1]: *** [/here/workdir/linux-2.6/common/src/scripts/Kbuild.include] Error 2
make: *** [sub-make] Error 2
make: Leaving directory `/here/workdir/linux-2.6/common/src'

[...]

After silentoldconfig has failed, oldconfig seems to get executed and after that the bzImage build is starting. So far so good, BUT: My script checks the return value and fails after the execution of silentoldconfig. That's why I recognized the different behaviour.

I did a git bisect to identify the commit, which caused the change to kbuild's behaviour. The offending commit is:

commit 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2

Author: Milton Miller <[email protected]>
Date:   Fri Sep 21 18:09:02 2007 -0500

kbuild: call make once for all targets when O=.. is used

Change the invocations of make in the output directory Makefile and the main Makefile for separate object trees to pass all goals to one $(MAKE) via a new phony target "sub-make" and the existing target _all.

When compiling with separate object directories, a separate make is called in the context of another directory (from the output directory the main Makefile is called, the Makefile is then restarted with current directory set to the object tree). Before this patch, when multiple make command goals are specified, each target results in a separate make invocation. With make -j, these invocations may run in parallel, resulting in multiple commands running in the same directory clobbering each others results.

I did not try to address make -j for mixed dot-config and no-dot-config targets. Because the order does matter, a solution was not obvious. Perhaps a simple check for MAKEFLAGS having -j and refusing to run would be appropriate.

So, am I facing a kbuild regression?

Cheers,
Jan

#1 lore raw
Quoted 78 lines · Jan Altenberg

Hi all,

I'm not quite sure if this might be a regression, but I recognized a change to kbuild's behaviour, which causes some of my build scripts to fail.

The build scripts do:

return system(('make -C %s O=%s ARCH=%s CROSS_COMPILE=%s '+

                      'oldconfig %s %s < /dev/null || exit %i') % 
                      (srcdir, 
                       builddir, 
                       arch,
                       crosscompile, 
                       target,
                       modules_target,
                       FAILED_RETCODE))

which results in something like:

make -C /here/workdir/linux-2.6/common/src O=/here/workdir/linux-2.6/common/build/build \
ARCH=i386 CROSS_COMPILE=/opt/i686-linux/bin/i686-linux- \
oldconfig bzImage < /dev/null || exit 1

In the past, oldconfig was the first target, which has been handled.

Now, bzImage seems to get handled at first. That causes my build scripts to fail, because the bzImage target does a silentoldconfig, which doesn't allow console redirection.

[...]

Kernel log buffer size (16 => 64KB, 17 => 128KB) (LOG_BUF_SHIFT) [14] 14 Control Group support (CGROUPS) [N/y/?] (NEW) aborted!

Console input/output is redirected. Run 'make oldconfig' to update configuration.

make[5]: *** [silentoldconfig] Error 1
make[4]: *** [silentoldconfig] Error 2
make[3]: *** [include/config/auto.conf] Error 2
make[2]: *** [sub-make] Error 2
make[1]: *** [/here/workdir/linux-2.6/common/src/scripts/Kbuild.include] Error 2
make: *** [sub-make] Error 2
make: Leaving directory `/here/workdir/linux-2.6/common/src'

[...]

After silentoldconfig has failed, oldconfig seems to get executed and after that the bzImage build is starting. So far so good, BUT: My script checks the return value and fails after the execution of silentoldconfig. That's why I recognized the different behaviour.

I did a git bisect to identify the commit, which caused the change to kbuild's behaviour. The offending commit is:

commit 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2

Author: Milton Miller <[email protected]>
Date:   Fri Sep 21 18:09:02 2007 -0500

kbuild: call make once for all targets when O=.. is used

Change the invocations of make in the output directory Makefile and the main Makefile for separate object trees to pass all goals to one $(MAKE) via a new phony target "sub-make" and the existing target _all.

When compiling with separate object directories, a separate make is called in the context of another directory (from the output directory the main Makefile is called, the Makefile is then restarted with current directory set to the object tree). Before this patch, when multiple make command goals are specified, each target results in a separate make invocation. With make -j, these invocations may run in parallel, resulting in multiple commands running in the same directory clobbering each others results.

I did not try to address make -j for mixed dot-config and no-dot-config targets. Because the order does matter, a solution was not obvious. Perhaps a simple check for MAKEFLAGS having -j and refusing to run would be appropriate.

Quoted 4 lines

So, am I facing a kbuild regression?

Yes - I will try to fix it during the weekend (if Milton does not beat me). Thanks for reporting and bisecting!

Sam

#2 lore raw

Hi Sam,

Quoted 23 lines
Quoted 23 lines

commit 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2

Author: Milton Miller <[email protected]>
Date:   Fri Sep 21 18:09:02 2007 -0500

kbuild: call make once for all targets when O=.. is used

Change the invocations of make in the output directory Makefile and the main Makefile for separate object trees to pass all goals to one $(MAKE) via a new phony target "sub-make" and the existing target _all.

When compiling with separate object directories, a separate make is called in the context of another directory (from the output directory the main Makefile is called, the Makefile is then restarted with current directory set to the object tree). Before this patch, when multiple make command goals are specified, each target results in a separate make invocation. With make -j, these invocations may run in parallel, resulting in multiple commands running in the same directory clobbering each others results.

I did not try to address make -j for mixed dot-config and no-dot-config targets. Because the order does matter, a solution was not obvious. Perhaps a simple check for MAKEFLAGS having -j and refusing to run would be appropriate.

Quoted 7 lines
Quoted 4 lines

So, am I facing a kbuild regression?

Yes - I will try to fix it during the weekend (if Milton does not beat me). Thanks for reporting and bisecting!

Have you made any progress on this? Let me know, if I can assist with testing.

Cheers,
Jan

#3 lore raw
Quoted 28 lines · Jan Altenberg

Hi Sam,

Quoted 26 lines
Quoted 26 lines

commit 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2

Author: Milton Miller <[email protected]>
Date:   Fri Sep 21 18:09:02 2007 -0500

kbuild: call make once for all targets when O=.. is used

Change the invocations of make in the output directory Makefile and the main Makefile for separate object trees to pass all goals to one $(MAKE) via a new phony target "sub-make" and the existing target _all.

When compiling with separate object directories, a separate make is called in the context of another directory (from the output directory the main Makefile is called, the Makefile is then restarted with current directory set to the object tree). Before this patch, when multiple make command goals are specified, each target results in a separate make invocation. With make -j, these invocations may run in parallel, resulting in multiple commands running in the same directory clobbering each others results.

I did not try to address make -j for mixed dot-config and no-dot-config targets. Because the order does matter, a solution was not obvious. Perhaps a simple check for MAKEFLAGS having -j and refusing to run would be appropriate.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

So, am I facing a kbuild regression?

Yes - I will try to fix it during the weekend (if Milton does not beat me). Thanks for reporting and bisecting!

Have you made any progress on this? Let me know, if I can assist with testing.

Hi Jan.

Not at all. My limited linux time goes into some x86 unification that I have given higher priority.
But your report is saved and I will return to it.

Sam

#4 lore raw
In reply to Jan Altenberg
Quoted 28 lines · Jan Altenberg

Hi Sam,

Quoted 26 lines
Quoted 26 lines

commit 0b35786d77ba4037f181982cc8ca20a7a3bf0fd2

Author: Milton Miller <[email protected]>
Date:   Fri Sep 21 18:09:02 2007 -0500

kbuild: call make once for all targets when O=.. is used

Change the invocations of make in the output directory Makefile and the main Makefile for separate object trees to pass all goals to one $(MAKE) via a new phony target "sub-make" and the existing target _all.

When compiling with separate object directories, a separate make is called in the context of another directory (from the output directory the main Makefile is called, the Makefile is then restarted with current directory set to the object tree). Before this patch, when multiple make command goals are specified, each target results in a separate make invocation. With make -j, these invocations may run in parallel, resulting in multiple commands running in the same directory clobbering each others results.

I did not try to address make -j for mixed dot-config and no-dot-config targets. Because the order does matter, a solution was not obvious. Perhaps a simple check for MAKEFLAGS having -j and refusing to run would be appropriate.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

So, am I facing a kbuild regression?

Yes - I will try to fix it during the weekend (if Milton does not beat me). Thanks for reporting and bisecting!

Have you made any progress on this? Let me know, if I can assist with testing.

Following seems to fix it with my limited testing.
I did an allnoconfig and then deleted CONFIG_LOGSHIFT from .config. It failed before and succeeded after.

Please test and report back.

Sam

Makefile +6 -5
index 9c9c4bf..7bfac5b 100644
@@ -108,6 +108,9 @@ endif
 PHONY := _all
 _all:
 
+# Cancel implicit rules on top Makefile.
+$(CURDIR)/Makefile Makefile: ;
+
 ifneq ($(KBUILD_OUTPUT),)
 # Invoke a second make in the output directory, passing relevant variables
 # check that the output directory actually exists
@@ -121,7 +124,7 @@ $(if $(filter-out $(KBUILD_OUTPUT),$(shell /bin/pwd)),, \
 
 PHONY += $(MAKECMDGOALS) sub-make
 
-$(filter-out _all sub-make,$(MAKECMDGOALS)) _all: sub-make
+$(filter-out _all sub-make $(CURDIR)/Makefile, $(MAKECMDGOALS)) _all: sub-make
 	$(Q)@:
 
 sub-make: FORCE
@@ -293,7 +296,8 @@ export quiet Q KBUILD_VERBOSE
 # Look for make include files relative to root of kernel src
 MAKEFLAGS += --include-dir=$(srctree)
 
-# We need some generic definitions.
+# We need some generic definitions (do not try to remake the file).
+$(srctree)/scripts/Kbuild.include: ;
 include $(srctree)/scripts/Kbuild.include
 
 # Make variables (CC, etc...)
@@ -1567,9 +1571,6 @@ endif	# skip-makefile
 PHONY += FORCE
 FORCE:
 
-# Cancel implicit rules on top Makefile, `-rR' will apply to sub-makes.
-Makefile: ;
-
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
 .PHONY: $(PHONY)
#5 lore raw

Hi Sam,

sorry for the delayed feedback. Still suffering a bad stomach flu...

Quoted 1 line

Please test and report back.

Thanks - Your Patch seems to fix things for me!

Cheers,
Jan

#6 lore raw
Quoted 7 lines · Jan Altenberg

Hi Sam,

sorry for the delayed feedback. Still suffering a bad stomach flu...

Quoted 1 line

Please test and report back.

Thanks - Your Patch seems to fix things for me!

Thanks Jan, hope you recover soon.

Sam