Skip to contentSkip to conversation

kbuild: building with a mostly-clean /usr/src/linux and O=

PATCH In review Andreas Gruenbacher 3 participants 5 messages Sep 6, 2005 – Sep 10, 2005 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kbuild: building with a mostly-clean /usr/src/linux and O=

When compiling a kernel with a separate output directory,

/var/tmp/kernel-obj> cp $CONFIG_FILE .config /var/tmp/kernel-obj> make -C /usr/src/linux O=pwd

the main kernel Makefile first tries to make sure that /usr/src/linux is a clean source tree by checking if /usr/src/linux contains .config or the include/asm symlink. This test is more restrictive than necessary, and can be relaxed with a few additional changes so that /usr/src/linux may contain a few additional critical files, and may still be used as the kernel compilation source.

The additional files I have in mind are include/asm, include/linux/version.h, include/linux/autoconf.h, and perhaps also include/asm/offset.h. With these files present, it's possible to use kernel headers from user-space. I know that's evil and frowned upon, but it's still done once in a while, and sometimes it's the least painful path compared to the alternatives. The kbuild changes required to allow this IMHO are harmless; please consider for inclusion.

Thanks,
Andreas.

Makefile +9 -8
Index: linux-2.6.13-kbuild/Makefile
===================================================================
@@ -465,7 +465,7 @@ ifeq ($(KBUILD_EXTMOD),)
 scripts: scripts_basic include/config/MARKER
 	$(Q)$(MAKE) $(build)=$(@)
 
-scripts_basic: include/linux/autoconf.h
+scripts_basic: $(objtree)/include/linux/autoconf.h
 
 # Objects we will link into vmlinux / subdirs we need to visit
 init-y		:= init/
@@ -491,11 +491,11 @@ include .config
 
 # If .config is newer than include/linux/autoconf.h, someone tinkered
 # with it and forgot to run make oldconfig
-include/linux/autoconf.h: .config
+$(objtree)/include/linux/autoconf.h: .config
 	$(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig
 else
 # Dummy target needed, because used as prerequisite
-include/linux/autoconf.h: ;
+$(objtree)/include/linux/autoconf.h: ;
 endif
 
 # The all: target is the default when no target is given on the
@@ -757,7 +757,7 @@ $(vmlinux-dirs): prepare-all scripts
 prepare2:
 ifneq ($(KBUILD_SRC),)
 	@echo '  Using $(srctree) as source for kernel'
-	$(Q)if [ -h $(srctree)/include/asm -o -f $(srctree)/.config ]; then \
+	$(Q)if [ -f $(srctree)/.config ]; then \
 		echo "  $(srctree) is not clean, please run 'make mrproper'";\
 		echo "  in the '$(srctree)' directory.";\
 		/bin/false; \
@@ -769,7 +769,7 @@ endif
 # prepare1 creates a makefile if using a separate output directory
 prepare1: prepare2 outputmakefile
 
-prepare0: prepare1 include/linux/version.h include/asm include/config/MARKER
+prepare0: prepare1 $(objtree)/include/linux/version.h $(objtree)/include/asm include/config/MARKER
 ifneq ($(KBUILD_MODULES),)
 	$(Q)rm -rf $(MODVERDIR)
 	$(Q)mkdir -p $(MODVERDIR)
@@ -808,14 +808,15 @@ export CPPFLAGS_vmlinux.lds += -P -C -U$
 #	hard to detect, but I suppose "make mrproper" is a good idea
 #	before switching between archs anyway.
 
-include/asm:
+include/asm: $(objtree)/include/asm
+$(objtree)/include/asm:
 	@echo '  SYMLINK $@ -> include/asm-$(ARCH)'
 	$(Q)if [ ! -d include ]; then mkdir -p include; fi;
 	@ln -fsn asm-$(ARCH) $@
 
 # 	Split autoconf.h into include/linux/config/*
 
-include/config/MARKER: include/linux/autoconf.h
+include/config/MARKER: $(objtree)/include/linux/autoconf.h
 	@echo '  SPLIT   include/linux/autoconf.h -> include/config/*'
 	@scripts/basic/split-include include/linux/autoconf.h include/config
 	@touch $@
@@ -839,7 +840,7 @@ define filechk_version.h
 	)
 endef
 
-include/linux/version.h: $(srctree)/Makefile FORCE
+$(objtree)/include/linux/version.h: $(srctree)/Makefile FORCE
 	$(call filechk,version.h)
 
 # ---------------------------------------------------------------------------
scripts/Makefile.lib +1 -1
Index: linux-2.6.13-kbuild/scripts/Makefile.lib
===================================================================
@@ -112,7 +112,7 @@ __cpp_flags     = $(_cpp_flags)
 else
 
 # Prefix -I with $(srctree) if it is not an absolute path
-addtree = $(if $(filter-out -I/%,$(1)),$(patsubst -I%,-I$(srctree)/%,$(1))) $(1)
+addtree = $(1) $(if $(filter-out -I/%,$(1)),$(patsubst -I%,-I$(srctree)/%,$(1)))
 # Find all -I options and call addtree
 flags = $(foreach o,$($(1)),$(if $(filter -I%,$(o)),$(call addtree,$(o)),$(o)))
#1 lore raw
Quoted 12 lines · Andreas Gruenbacher

When compiling a kernel with a separate output directory,

/var/tmp/kernel-obj> cp $CONFIG_FILE .config /var/tmp/kernel-obj> make -C /usr/src/linux O=pwd

the main kernel Makefile first tries to make sure that /usr/src/linux is a clean source tree by checking if /usr/src/linux contains .config or the include/asm symlink. This test is more restrictive than necessary, and can be relaxed with a few additional changes so that /usr/src/linux may contain a few additional critical files, and may still be used as the kernel compilation source.

Quoted 7 lines

The additional files I have in mind are include/asm, include/linux/version.h, include/linux/autoconf.h, and perhaps also include/asm/offset.h. With these files present, it's possible to use kernel headers from user-space. I know that's evil and frowned upon, but it's still done once in a while, and sometimes it's the least painful path compared to the alternatives. The kbuild changes required to allow this IMHO are harmless; please consider for inclusion.

I've already included below patch from you.
It was included in -linus last night.

Do we really need more?

Sam

diff-tree d80e22460968ec7986c82fd7d207ebe3de59e03d (from c5f75eca120de6587e67a1951ce3e6912e2c6879)

Author: Sam Ravnborg <sam@mars.(none)>
Date:   Thu Jul 14 20:22:39 2005 +0000

kbuild: Don't fail if include/asm symlink exists

From: Andreas Gruenbacher <[email protected]>

We're having the following situation: There are user-space applications that include kernel headers directly. With a completely unconfigured /usr/src/linux tree, including most headers fails because essential files are not there:

include/asm
include/linux/autoconf.h
include/linux/version.h

So we create these files. On the other hand, we want to use /usr/src/linux as read-only source for building kernels or additional modules. Now when building a kernel with a separate output directory (O=), there is a check in the main makefile for the include/asm symlink. There is no real need for this check: if we ensure that $(objdir)/include/asm is always created as the patch does, $(srctree)/include/asm becomes irrelevant.

Makefile +4 -3
@@ -765,11 +765,11 @@ $(vmlinux-dirs): prepare-all scripts
 # 2) Create the include2 directory, used for the second asm symlink
 
 prepare2:
 ifneq ($(KBUILD_SRC),)
 	@echo '  Using $(srctree) as source for kernel'
-	$(Q)if [ -h $(srctree)/include/asm -o -f $(srctree)/.config ]; then \
+	$(Q)if [ -f $(srctree)/.config ]; then \
 		echo "  $(srctree) is not clean, please run 'make mrproper'";\
 		echo "  in the '$(srctree)' directory.";\
 		/bin/false; \
 	fi;
 	$(Q)if [ ! -d include2 ]; then mkdir -p include2; fi;
@@ -777,11 +777,12 @@ ifneq ($(KBUILD_SRC),)
 endif
 
 # prepare1 creates a makefile if using a separate output directory
 prepare1: prepare2 outputmakefile
 
-prepare0: prepare1 include/linux/version.h include/asm include/config/MARKER
+prepare0: prepare1 include/linux/version.h $(objtree)/include/asm \
+                   include/config/MARKER
 ifneq ($(KBUILD_MODULES),)
 	$(Q)rm -rf $(MODVERDIR)
 	$(Q)mkdir -p $(MODVERDIR)
 endif
 
@@ -816,11 +817,11 @@ export CPPFLAGS_vmlinux.lds += -P -C -U$
 
 # 	FIXME: The asm symlink changes when $(ARCH) changes. That's
 #	hard to detect, but I suppose "make mrproper" is a good idea
 #	before switching between archs anyway.
 
-include/asm:
+$(objtree)/include/asm:
 	@echo '  SYMLINK $@ -> include/asm-$(ARCH)'
 	$(Q)if [ ! -d include ]; then mkdir -p include; fi;
 	@ln -fsn asm-$(ARCH) $@
 
 # 	Split autoconf.h into include/linux/config/*
#2 lore raw
Quoted 2 lines · Sam Ravnborg

I've already included below patch from you.
It was included in -linus last night.

That was close.

Quoted 1 line

Do we really need more?

So it seems I'm afraid: With the version of this patch that just went in, Jan Beulich <[email protected]> found a bug when building vmlinux.lds on i386. He triggered it by by putting poisoned version.h and autoconf.h files in /usr/src/linux.

With the additional changes (rediff against linux-2.6.13-git6 below), things work as expected. There may be a slightly more minimalistic solution that also works --- I didn't want to risk strange failures.

(To not confuse people, let me add that this all doesn't matter if the source tree is distclean.)

Cheers,
Andreas.

Additional fixes for building with O= against a non-distclean tree

Makefile +8 -7
Index: linux-2.6.13-git6/Makefile
===================================================================
@@ -462,7 +462,7 @@ ifeq ($(KBUILD_EXTMOD),)
 scripts: scripts_basic include/config/MARKER
 	$(Q)$(MAKE) $(build)=$(@)
 
-scripts_basic: include/linux/autoconf.h
+scripts_basic: $(objtree)/include/linux/autoconf.h
 
 # Objects we will link into vmlinux / subdirs we need to visit
 init-y		:= init/
@@ -488,11 +488,11 @@ include .config
 
 # If .config is newer than include/linux/autoconf.h, someone tinkered
 # with it and forgot to run make oldconfig
-include/linux/autoconf.h: .config
+$(objtree)/include/linux/autoconf.h: .config
 	$(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig
 else
 # Dummy target needed, because used as prerequisite
-include/linux/autoconf.h: ;
+$(objtree)/include/linux/autoconf.h: ;
 endif
 
 # The all: target is the default when no target is given on the
@@ -798,7 +798,7 @@ endif
 # prepare1 creates a makefile if using a separate output directory
 prepare1: prepare2 outputmakefile
 
-prepare0: prepare1 include/linux/version.h include/asm \
+prepare0: prepare1 $(objtree)/include/linux/version.h $(objtree)/include/asm \
                    include/config/MARKER
 ifneq ($(KBUILD_MODULES),)
 	$(Q)rm -rf $(MODVERDIR)
@@ -838,14 +838,15 @@ export CPPFLAGS_vmlinux.lds += -P -C -U$
 #	hard to detect, but I suppose "make mrproper" is a good idea
 #	before switching between archs anyway.
 
-include/asm:
+include/asm: $(objtree)/include/asm
+$(objtree)/include/asm:
 	@echo '  SYMLINK $@ -> include/asm-$(ARCH)'
 	$(Q)if [ ! -d include ]; then mkdir -p include; fi;
 	@ln -fsn asm-$(ARCH) $@
 
 # 	Split autoconf.h into include/linux/config/*
 
-include/config/MARKER: include/linux/autoconf.h
+include/config/MARKER: $(objtree)/include/linux/autoconf.h
 	@echo '  SPLIT   include/linux/autoconf.h -> include/config/*'
 	@scripts/basic/split-include include/linux/autoconf.h include/config
 	@touch $@
@@ -869,7 +870,7 @@ define filechk_version.h
 	)
 endef
 
-include/linux/version.h: $(srctree)/Makefile FORCE
+$(objtree)/include/linux/version.h: $(srctree)/Makefile FORCE
 	$(call filechk,version.h)
 
 # ---------------------------------------------------------------------------
scripts/Makefile.lib +1 -1
Index: linux-2.6.13-git6/scripts/Makefile.lib
===================================================================
@@ -98,7 +98,7 @@ __cpp_flags     = $(_cpp_flags)
 else
 
 # Prefix -I with $(srctree) if it is not an absolute path
-addtree = $(if $(filter-out -I/%,$(1)),$(patsubst -I%,-I$(srctree)/%,$(1))) $(1)
+addtree = $(1) $(if $(filter-out -I/%,$(1)),$(patsubst -I%,-I$(srctree)/%,$(1)))
 # Find all -I options and call addtree
 flags = $(foreach o,$($(1)),$(if $(filter -I%,$(o)),$(call addtree,$(o)),$(o)))
#3 lore raw

Unfortunately this isn't sufficient, yet. In the architecture-specific makefiles asm-offsets.s (or however the specific architectures call this) now need their dependencies on include/linux/version.h changed (I wonder whether it wouldn't be more efficient to centralize these dependencies into an architecture-independent file, perhaps at once applying consistent naming across all architectures), and arch/um/Makefile additionally has a dangling dependency on include/linux/autoconf.h now.

greping all Make* across the tree easily shows all locations needing additional changes.

Jan

Quoted 1 line
Quoted 1 line
Quoted 1 line

Andreas Gruenbacher <[email protected]> 07.09.05 03:03:44 >>>

Quoted 2 lines · Sam Ravnborg

I've already included below patch from you.
It was included in -linus last night.

That was close.

Quoted 1 line

Do we really need more?

So it seems I'm afraid: With the version of this patch that just went in, Jan Beulich <[email protected]> found a bug when building vmlinux.lds on i386. He triggered it by by putting poisoned version.h and
autoconf.h files in /usr/src/linux.

With the additional changes (rediff against linux-2.6.13-git6 below), things work as expected. There may be a slightly more minimalistic solution that also works --- I didn't want to risk strange failures.

(To not confuse people, let me add that this all doesn't matter if the source tree is distclean.)

Cheers,
Andreas.

Additional fixes for building with O= against a non-distclean tree

Makefile +4 -4
Index: linux-2.6.13-git6/Makefile
===================================================================
@@ -462,7 +462,7 @@ ifeq ($(KBUILD_EXTMOD),)
 scripts: scripts_basic include/config/MARKER
 	$(Q)$(MAKE) $(build)=$(@)
 
-scripts_basic: include/linux/autoconf.h
+scripts_basic: $(objtree)/include/linux/autoconf.h
 
 # Objects we will link into vmlinux / subdirs we need to visit
 init-y		:= init/
@@ -488,11 +488,11 @@ include .config
 
 # If .config is newer than include/linux/autoconf.h, someone tinkered
 # with it and forgot to run make oldconfig
-include/linux/autoconf.h: .config
+$(objtree)/include/linux/autoconf.h: .config
 	$(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig
 else
 # Dummy target needed, because used as prerequisite
-include/linux/autoconf.h: ;
+$(objtree)/include/linux/autoconf.h: ;
 endif
 
 # The all: target is the default when no target is given on the
@@ -798,7 +798,7 @@ endif
 # prepare1 creates a makefile if using a separate output directory
 prepare1: prepare2 outputmakefile
 
-prepare0: prepare1 include/linux/version.h include/asm \
+prepare0: prepare1 $(objtree)/include/linux/version.h

$(objtree)/include/asm \
include/config/MARKER
ifneq ($(KBUILD_MODULES),)
$(Q)rm -rf $(MODVERDIR)
@@ -838,14 +838,15 @@ export CPPFLAGS_vmlinux.lds += -P -C -U$
# hard to detect, but I suppose "make mrproper" is a good idea
# before switching between archs anyway.

-include/asm:
+include/asm: $(objtree)/include/asm +$(objtree)/include/asm:
@echo ' SYMLINK $@ -> include/asm-$(ARCH)' $(Q)if [ ! -d include ]; then mkdir -p include; fi; @ln -fsn asm-$(ARCH) $@

# Split autoconf.h into include/linux/config/*

-include/config/MARKER: include/linux/autoconf.h +include/config/MARKER: $(objtree)/include/linux/autoconf.h
@echo ' SPLIT include/linux/autoconf.h -> include/config/*'
@scripts/basic/split-include include/linux/autoconf.h
include/config
@touch $@
@@ -869,7 +870,7 @@ define filechk_version.h
)
endef

-include/linux/version.h: $(srctree)/Makefile FORCE +$(objtree)/include/linux/version.h: $(srctree)/Makefile FORCE
$(call filechk,version.h)

#
---------------------------------------------------------------------------

scripts/Makefile.lib +1 -3
Index: linux-2.6.13-git6/scripts/Makefile.lib
===================================================================
@@ -98,7 +98,7 @@ __cpp_flags     = $(_cpp_flags)
 else
 
 # Prefix -I with $(srctree) if it is not an absolute path
-addtree = $(if $(filter-out -I/%,$(1)),$(patsubst
-I%,-I$(srctree)/%,$(1))) $(1)
+addtree = $(1) $(if $(filter-out -I/%,$(1)),$(patsubst
-I%,-I$(srctree)/%,$(1)))
 # Find all -I options and call addtree

flags = $(foreach o,$($(1)),$(if $(filter -I%,$(o)),$(call
addtree,$(o)),$(o)))

#4 lore raw
Quoted 6 lines · Jan Beulich

Unfortunately this isn't sufficient, yet. In the architecture-specific makefiles asm-offsets.s (or however the specific architectures call this) now need their dependencies on include/linux/version.h changed (I wonder whether it wouldn't be more efficient to centralize these dependencies into an architecture-independent file, perhaps at once applying consistent naming across all architectures)

This topic has been on my mind to implment over a long time. So I took a stamp on it and it hit -linus this night.

Dependencies on arch/$(ARCH)/kernel/asm-offsets.s is now tracked like any other dependency - so we now at last have full dependency checks for asm-offsets.h.
Take a look at the Kbuild file in the top-level directory.

With respect to the patch posted below I recall I had to zap two $(objtree) to fix alpha build.
Patch by Jan Dittmer - see 946dc121d7d1c606f6bbeb8ae778963a1e2ff59c

But that may have changed now after introducing generic asm-offsets.h support.

Sam