RFC: Further kernel build process changes?

PATCH Richard Purdie 5 participants 8 messages Jan 7, 2015 – Jan 13, 2015

Share & tools

Richard Purdie

author

2015-01-07 12:26 UTC #0 lore raw

I'm hearing (somewhat justified) complaints that the recent kernel changes have destablised builds. Part of the question is whether the recent changes are as clear to users as they could be, we're also running into some problems due to mixing kernel source and build artefacts in some places and not in others.

At this point I think it may be worth looking at making some more invasive but decisive changes, specifically that:

STAGING_KERNEL_DIR moves
from sysroots/MACHINE/usr/src/kernel
to work-shared/MACHINE/kernel-source

This is to make it clearer that the source here is not under the control of sstate. The reasons why we don't want it under the control of sstate are in other emails.

The second change would be to split the kernel source into two:

work-shared/MACHINE/kernel-source
work-shared/MACHINE/kernel-build

where kernel-build is the kernel build output and kernel-source is kept "pristine".

This means the defconfig, the kernel-abiversion, System.map files and output from "make scripts" would be in kernel-build.

External module builds do work in this configuration *if* you pass in the correct options e.g.:

make -C work-shared/MACHINE/kernel-source O=work-shared/MACHINE/kernel-build M=${S}

I've put together a quick proof of concept of this below.

There are two other things up for discussion. There is of confusion on how the kernel source gets into STAGING_KERNEL_DIR in the first place. We've added in "munging" code which does this in kernel.bbclass, linux-yocto also has its share of "crazy" mess in this regard.

We could wipe the slate clean and require that people put a parameter against the element in SRC_URI that represents the kernel indicating it should be directly extracted to STAGING_KERNEL_DIR. There is a bitbake issue with being unable to override the extraction directory at present but that can be fixed. The upside is that it would be clean, relatively clear and allow fragile code to be dropped. The downside is it means tweaking all kernel recipes.

The second issue is that of the dependency to populate STAGING_KERNEL_DIR which is now a "depends flag on do_install". The intent is to have kernelsrc.bbclass do this, looking at the things there (such as setting S=), I suspect it may not be fit for purpose. We could adjust the class to check for a variable and set up the dependency if its set.

Anyhow, this does need thought and discussion but I'm putting it here as a start to that, and to let people like Bruce see and experiment with the code below.

Cheers,

Richard

meta/classes/kernel-module-split.bbclass +1 -1

index 9a95b72..2d43b51 100644
@@ -70,7 +70,7 @@ python split_kernel_module_packages () {
         m = kerverrexp.match(kernelver)
         if m:
             kernelver_stripped = m.group(1)
-        staging_kernel_dir = d.getVar("STAGING_KERNEL_DIR", True)
+        staging_kernel_dir = d.getVar("STAGING_KERNEL_BUILDDIR", True)
         system_map_file = "%s/boot/System.map-%s" % (dvar, kernelver)
         if not os.path.exists(system_map_file):
             system_map_file = "%s/System.map-%s" % (staging_kernel_dir, kernelver)

meta/classes/kernel.bbclass +43 -0

55 lines not on this page · raw message

meta/classes/module-base.bbclass +4 -2

index 9537ba9..c34eee7 100644
@@ -3,16 +3,18 @@ inherit kernel-arch
 export OS = "${TARGET_OS}"
 export CROSS_COMPILE = "${TARGET_PREFIX}"
 
-export KERNEL_VERSION = "${@base_read_file('${STAGING_KERNEL_DIR}/kernel-abiversion')}"
+export KERNEL_VERSION = "${@base_read_file('${STAGING_KERNEL_BUILDDIR}/kernel-abiversion')}"
 KERNEL_OBJECT_SUFFIX = ".ko"
 
 # kernel modules are generally machine specific
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 
+do_configure[depends] += "virtual/kernel:do_install"
+
 # Function to ensure the kernel scripts are created. Expected to
 # be called before do_compile. See module.bbclass for an exmaple.
 do_make_scripts() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS 
 	make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
-	           -C ${STAGING_KERNEL_DIR} scripts
+	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts
 }

meta/classes/module.bbclass +2 -0

index ad6f7af..4f466d7 100644
@@ -13,6 +13,7 @@ module_do_compile() {
 		   KERNEL_VERSION=${KERNEL_VERSION}    \
 		   CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
 		   AR="${KERNEL_AR}" \
+	           O=${STAGING_KERNEL_BUILDDIR} \
 		   ${MAKE_TARGETS}
 }

@@ -21,6 +22,7 @@ module_do_install() {
 	oe_runmake DEPMOD=echo INSTALL_MOD_PATH="${D}" \
 	           KERNEL_SRC=${STAGING_KERNEL_DIR} \
 	           CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
+	           O=${STAGING_KERNEL_BUILDDIR} \
 	           modules_install
 }

meta/conf/bitbake.conf +2 -1

index 362d6b6..b1135c2 100644
@@ -394,7 +394,8 @@ SDKPATHNATIVE = "${SDKPATH}/sysroots/${SDK_SYS}"
 ##################################################################
 
 OLDEST_KERNEL = "2.6.32"
-STAGING_KERNEL_DIR = "${STAGING_DIR_HOST}/usr/src/kernel"
+STAGING_KERNEL_DIR = "${TMPDIR}/work-shared/${MACHINE}/kernel-source"
+STAGING_KERNEL_BUILDDIR = "${TMPDIR}/work-shared/${MACHINE}/kernel-build"
 
 ##################################################################
 # Specific image creation and rootfs population info.

meta/recipes-kernel/lttng/lttng-modules/sepbuild.patch new file +17 -0

new file mode 100644
index 0000000..1b46558
@@ -0,0 +1,17 @@
+Index: git/Makefile
+===================================================================
+--- git.orig/Makefile
++++ git/Makefile
+@@ -67,10 +67,10 @@ else # KERNELRELEASE
+ 	CFLAGS = $(EXTCFLAGS)
+ 
+ default:
+-	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) modules
++	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) O=$(O) modules
+ 
+ modules_install:
+-	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) modules_install
++	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) O=$(O) modules_install
+ 
+ clean:
+ 	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) clean

meta/recipes-kernel/lttng/lttng-modules_2.5.2.bb +1 -0

index 55df07f..58a4196 100644
@@ -19,6 +19,7 @@ SRC_URI = "git://git.lttng.org/lttng-modules.git;branch=stable-2.5 \
            file://lttng-modules-replace-KERNELDIR-with-KERNEL_SRC.patch \
            file://Fix-noargs-probes-should-calculate-alignment-and-eve.patch \
            file://Update-kvm-instrumentation-compile-on-3.17-rc1.patch \
+           file://sepbuild.patch \
            "
 
 export INSTALL_MOD_DIR="kernel/lttng-modules"

Bruce Ashfield

2015-01-07 13:22 UTC #1 lore raw

Quoted 5 lines · Richard Purdie

I'm hearing (somewhat justified) complaints that the recent kernel changes have destablised builds. Part of the question is whether the recent changes are as clear to users as they could be, we're also running into some problems due to mixing kernel source and build artefacts in some places and not in others.

And we've been bitten by the sheer diversity in the various additions and tweaks to the kernel build process .. which when wading in to try and make some improvements was always the risk :(

Quoted 11 lines

At this point I think it may be worth looking at making some more invasive but decisive changes, specifically that:
STAGING_KERNEL_DIR moves
from sysroots/MACHINE/usr/src/kernel
to work-shared/MACHINE/kernel-source
This is to make it clearer that the source here is not under the control of sstate. The reasons why we don't want it under the control of sstate are in other emails.

I'm in agreement here.

Quoted 11 lines

The second change would be to split the kernel source into two:
work-shared/MACHINE/kernel-source
work-shared/MACHINE/kernel-build
where kernel-build is the kernel build output and kernel-source is kept "pristine".
This means the defconfig, the kernel-abiversion, System.map files and output from "make scripts" would be in kernel-build.

Exactly. When setting up the minimal external module build environment, to keep the impact in the shared directories small, I went with the current approach.

Since we are breaking workflows with it .. this would be a good balance between the old and new efforts. I started mocking this up over the holidays but lost a week due to illness. I'll continue running with this now.

Quoted 4 lines

External module builds do work in this configuration *if* you pass in the correct options e.g.:

Quoted 8 lines

make -C work-shared/MACHINE/kernel-source O=work-shared/MACHINE/kernel-build M=${S}
I've put together a quick proof of concept of this below.
There are two other things up for discussion. There is of confusion on how the kernel source gets into STAGING_KERNEL_DIR in the first place. We've added in "munging" code which does this in kernel.bbclass, linux-yocto also has its share of "crazy" mess in this regard.

Quoted 16 lines

We could wipe the slate clean and require that people put a parameter against the element in SRC_URI that represents the kernel indicating it should be directly extracted to STAGING_KERNEL_DIR. There is a bitbake issue with being unable to override the extraction directory at present but that can be fixed. The upside is that it would be clean, relatively clear and allow fragile code to be dropped. The downside is it means tweaking all kernel recipes.
The second issue is that of the dependency to populate STAGING_KERNEL_DIR which is now a "depends flag on do_install". The intent is to have kernelsrc.bbclass do this, looking at the things there (such as setting S=), I suspect it may not be fit for purpose. We could adjust the class to check for a variable and set up the dependency if its set.

Quoted 3 lines

Anyhow, this does need thought and discussion but I'm putting it here as a start to that, and to let people like Bruce see and experiment with the code below.

I'll blend your RFC with what I have on the cooker today and see if I can get a patch out ASAP.

I still think it is worth working through these issues and pushing forward, we risk slipping deeper into the release if we drop everything and start over.

As we all know, the code is complex and we have many workflows to support, and I know I'm churning as fast as I can on fixes.

Bruce

Quoted 174 lines

Cheers,

Richard

meta/classes/kernel-module-split.bbclass +1 -1

index 9a95b72..2d43b51 100644
@@ -70,7 +70,7 @@ python split_kernel_module_packages () {
          m = kerverrexp.match(kernelver)
          if m:
              kernelver_stripped = m.group(1)
-        staging_kernel_dir = d.getVar("STAGING_KERNEL_DIR", True)
+        staging_kernel_dir = d.getVar("STAGING_KERNEL_BUILDDIR", True)
          system_map_file = "%s/boot/System.map-%s" % (dvar, kernelver)
          if not os.path.exists(system_map_file):
              system_map_file = "%s/System.map-%s" % (staging_kernel_dir, kernelver)

meta/classes/kernel.bbclass +43 -0

index 5cabc2c..b1a1ccf 100644
@@ -249,6 +249,10 @@ kernel_do_install() {
  	cp .config $kerneldir/
  	mkdir -p $kerneldir/include/config
  	cp include/config/kernel.release $kerneldir/include/config/kernel.release
+	if [ -e include/linux/version.h ]; then
+		mkdir -p $kerneldir/include/linux
+		cp include/linux/version.h $kerneldir/include/linux/version.h
+	fi
 
  	# As of Linux kernel version 3.0.1, the clean target removes
  	# arch/powerpc/lib/crtsavres.o which is present in

@@ -268,6 +272,45 @@ kernel_do_install() {
  }
  do_install[prefuncs] += "package_get_auto_pr"
 
+addtask do_shared_workdir after do_compile before do_install
+
+do_shared_workdir () {
+	cd ${B}
+
+	kerneldir=${STAGING_KERNEL_BUILDDIR}
+	install -d $kerneldir
+
+	#
+	# Store the kernel version in sysroots for module-base.bbclass
+	#
+
+	echo "${KERNEL_VERSION}" > $kerneldir/kernel-abiversion
+	
+	# Copy files required for module builds
+	cp System.map $kerneldir/System.map-${KERNEL_VERSION}
+	cp Module.symvers $kerneldir/
+	cp .config $kerneldir/
+	mkdir -p $kerneldir/include/config
+	cp include/config/kernel.release $kerneldir/include/config/kernel.release
+
+	# As of Linux kernel version 3.0.1, the clean target removes
+	# arch/powerpc/lib/crtsavres.o which is present in
+	# KBUILD_LDFLAGS_MODULE, making it required to build external modules.
+	if [ ${ARCH} = "powerpc" ]; then
+		mkdir -p $kerneldir/arch/powerpc/lib/
+		cp arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o
+	fi
+
+	mkdir -p $kerneldir/include/generated/
+	cp -fR include/generated/* $kerneldir/include/generated/
+
+	if [ -d arch/${ARCH}/include/generated ]; then
+		mkdir -p $kerneldir/arch/${ARCH}/include/generated/
+		cp -fR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/
+	fi
+}
+
+
  python sysroot_stage_all () {
      oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}"))
  }

meta/classes/module-base.bbclass +4 -2

index 9537ba9..c34eee7 100644
@@ -3,16 +3,18 @@ inherit kernel-arch
  export OS = "${TARGET_OS}"
  export CROSS_COMPILE = "${TARGET_PREFIX}"
 
-export KERNEL_VERSION = "${@base_read_file('${STAGING_KERNEL_DIR}/kernel-abiversion')}"
+export KERNEL_VERSION = "${@base_read_file('${STAGING_KERNEL_BUILDDIR}/kernel-abiversion')}"
  KERNEL_OBJECT_SUFFIX = ".ko"
 
  # kernel modules are generally machine specific
  PACKAGE_ARCH = "${MACHINE_ARCH}"
 
+do_configure[depends] += "virtual/kernel:do_install"
+
  # Function to ensure the kernel scripts are created. Expected to
  # be called before do_compile. See module.bbclass for an exmaple.
  do_make_scripts() {
  	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
  	make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
-	           -C ${STAGING_KERNEL_DIR} scripts
+	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts
  }

meta/classes/module.bbclass +2 -0

index ad6f7af..4f466d7 100644
@@ -13,6 +13,7 @@ module_do_compile() {
  		   KERNEL_VERSION=${KERNEL_VERSION}    \
  		   CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
  		   AR="${KERNEL_AR}" \
+	           O=${STAGING_KERNEL_BUILDDIR} \
  		   ${MAKE_TARGETS}
  }

@@ -21,6 +22,7 @@ module_do_install() {
  	oe_runmake DEPMOD=echo INSTALL_MOD_PATH="${D}" \
  	           KERNEL_SRC=${STAGING_KERNEL_DIR} \
  	           CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
+	           O=${STAGING_KERNEL_BUILDDIR} \
  	           modules_install
  }

meta/conf/bitbake.conf +2 -1

index 362d6b6..b1135c2 100644
@@ -394,7 +394,8 @@ SDKPATHNATIVE = "${SDKPATH}/sysroots/${SDK_SYS}"
  ##################################################################
 
  OLDEST_KERNEL = "2.6.32"
-STAGING_KERNEL_DIR = "${STAGING_DIR_HOST}/usr/src/kernel"
+STAGING_KERNEL_DIR = "${TMPDIR}/work-shared/${MACHINE}/kernel-source"
+STAGING_KERNEL_BUILDDIR = "${TMPDIR}/work-shared/${MACHINE}/kernel-build"
 
  ##################################################################
  # Specific image creation and rootfs population info.

meta/recipes-kernel/lttng/lttng-modules/sepbuild.patch new file +17 -0

new file mode 100644
index 0000000..1b46558
@@ -0,0 +1,17 @@
+Index: git/Makefile
+===================================================================
+--- git.orig/Makefile
++++ git/Makefile
+@@ -67,10 +67,10 @@ else # KERNELRELEASE
+ 	CFLAGS = $(EXTCFLAGS)
+
+ default:
+-	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) modules
++	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) O=$(O) modules
+
+ modules_install:
+-	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) modules_install
++	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) O=$(O) modules_install
+
+ clean:
+ 	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) clean

meta/recipes-kernel/lttng/lttng-modules_2.5.2.bb +1 -0

index 55df07f..58a4196 100644
@@ -19,6 +19,7 @@ SRC_URI = "git://git.lttng.org/lttng-modules.git;branch=stable-2.5 \
             file://lttng-modules-replace-KERNELDIR-with-KERNEL_SRC.patch \
             file://Fix-noargs-probes-should-calculate-alignment-and-eve.patch \
             file://Update-kvm-instrumentation-compile-on-3.17-rc1.patch \
+           file://sepbuild.patch \
             "
 
  export INSTALL_MOD_DIR="kernel/lttng-modules"

Darren Hart

2015-01-07 18:33 UTC #2 lore raw

Quoted 19 lines · Bruce Ashfield

Quoted 5 lines · Richard Purdie
I'm hearing (somewhat justified) complaints that the recent kernel changes have destablised builds. Part of the question is whether the recent changes are as clear to users as they could be, we're also running into some problems due to mixing kernel source and build artefacts in some places and not in others.
And we've been bitten by the sheer diversity in the various additions and tweaks to the kernel build process .. which when wading in to try and make some improvements was always the risk :(
Quoted 8 lines
At this point I think it may be worth looking at making some more invasive but decisive changes, specifically that:
STAGING_KERNEL_DIR moves
from sysroots/MACHINE/usr/src/kernel
to work-shared/MACHINE/kernel-source

Quoted 5 lines

Quoted 3 lines
This is to make it clearer that the source here is not under the control of sstate. The reasons why we don't want it under the control of sstate are in other emails.
I'm in agreement here.

I would prefer this approach as well.

Quoted 11 lines

Quoted 10 lines
The second change would be to split the kernel source into two:
work-shared/MACHINE/kernel-source
work-shared/MACHINE/kernel-build
where kernel-build is the kernel build output and kernel-source is kept "pristine".
This means the defconfig, the kernel-abiversion, System.map files and output from "make scripts" would be in kernel-build.

Quoted 8 lines

Exactly. When setting up the minimal external module build environment, to keep the impact in the shared directories small, I went with the current approach.
Since we are breaking workflows with it .. this would be a good balance between the old and new efforts. I started mocking this up over the holidays
but lost a week due to illness. I'll continue running with this now.

Also in agreement here, keeping the sourcedir pristine should reduce confusion and complexity elsewhere.

Quoted 8 lines

Quoted 8 lines
External module builds do work in this configuration *if* you pass in the correct options e.g.:
make -C work-shared/MACHINE/kernel-source
O=work-shared/MACHINE/kernel-build M=${S}
I've put together a quick proof of concept of this below.

I was concerned about how this would impact hello-mod and recipes modeled after it, however, in reviewing the patch below, it looks to have this covered. I¹ll verify this just as soon as my workstation is available (it¹s ³busy² updatingŠ sigh).

Quoted 16 lines

Quoted 5 lines
There are two other things up for discussion. There is of confusion on how the kernel source gets into STAGING_KERNEL_DIR in the first place. We've added in "munging" code which does this in kernel.bbclass, linux-yocto also has its share of "crazy" mess in this regard.
:)
Quoted 8 lines
We could wipe the slate clean and require that people put a parameter against the element in SRC_URI that represents the kernel indicating it should be directly extracted to STAGING_KERNEL_DIR. There is a bitbake issue with being unable to override the extraction directory at present but that can be fixed. The upside is that it would be clean, relatively clear and allow fragile code to be dropped. The downside is it means tweaking all kernel recipes.

I¹m concerned about adding yet more complexity to the SRC_URIŠ I don¹t have a better proposal though. Part of this fix will need to include fixes to the kernel-dev manual, I can take that on once we hash this out.

Quoted 15 lines

Quoted 11 lines
The second issue is that of the dependency to populate STAGING_KERNEL_DIR which is now a "depends flag on do_install". The intent is to have kernelsrc.bbclass do this, looking at the things there (such as setting S=), I suspect it may not be fit for purpose. We could adjust the class to check for a variable and set up the dependency if its set.
Anyhow, this does need thought and discussion but I'm putting it here as a start to that, and to let people like Bruce see and experiment with the code below.
I'll blend your RFC with what I have on the cooker today and see if I can get a patch out ASAP.

Quoted 8 lines

I still think it is worth working through these issues and pushing forward, we risk slipping deeper into the release if we drop everything and start over.
As we all know, the code is complex and we have many workflows to support, and I know I'm churning as fast as I can on fixes.
Bruce

A couple of first pass questions belowŠ prior to applying and testing myself...

Quoted 26 lines

Cheers,

Richard

meta/classes/kernel-module-split.bbclass +1 -1

index 9a95b72..2d43b51 100644
@@ -70,7 +70,7 @@ python split_kernel_module_packages () {
          m = kerverrexp.match(kernelver)
          if m:
              kernelver_stripped = m.group(1)
-        staging_kernel_dir = d.getVar("STAGING_KERNEL_DIR", True)
+        staging_kernel_dir = d.getVar("STAGING_KERNEL_BUILDDIR", True)
          system_map_file = "%s/boot/System.map-%s" % (dvar, kernelver)
          if not os.path.exists(system_map_file):
              system_map_file = "%s/System.map-%s" %

(staging_kernel_dir, kernelver)

meta/classes/kernel.bbclass +0 -0

index 5cabc2c..b1a1ccf 100644
@@ -249,6 +249,10 @@ kernel_do_install() {

Quoted 8 lines

meta/classes/kernel.bbclass kernel_do_install() { · line 255
Quoted 8 lines
meta/classes/kernel.bbclass +0 -0
  	cp .config $kerneldir/
  	mkdir -p $kerneldir/include/config
  	cp include/config/kernel.release
$kerneldir/include/config/kernel.release
meta/classes/kernel.bbclass +4 -0
+	if [ -e include/linux/version.h ]; then
+		mkdir -p $kerneldir/include/linux
+		cp include/linux/version.h $kerneldir/include/linux/version.h
+	fi

I know this bit someone with the recent changes, let¹s make sure we add a comment as to why this is required so we don¹t have to dig it out of the archives in a year when we¹re doing another round of fixes in this area.

Quoted 7 lines

Quoted 7 lines
# As of Linux kernel version 3.0.1, the clean target removes
  	# arch/powerpc/lib/crtsavres.o which is present in
meta/classes/kernel.bbclass +0 -0
@@ -268,6 +272,45 @@ kernel_do_install() {
  }
  do_install[prefuncs] += "package_get_auto_pr"
 

Quoted 12 lines

meta/classes/kernel.bbclass kernel_do_install() { · line 286

Quoted 12 lines

meta/classes/kernel.bbclass +12 -0

+addtask do_shared_workdir after do_compile before do_install
+
+do_shared_workdir () {
+	cd ${B}
+
+	kerneldir=${STAGING_KERNEL_BUILDDIR}
+	install -d $kerneldir
+
+	#
+	# Store the kernel version in sysroots for module-base.bbclass
+	#
+

OK, how does this impact the dependency on do_installŠ As we¹re doing stuff for building modules here, can that now depend on do_shared_workdir?

Quoted 19 lines

meta/classes/kernel.bbclass +8 -0

+	echo "${KERNEL_VERSION}" > $kerneldir/kernel-abiversion
+
+	# Copy files required for module builds
+	cp System.map $kerneldir/System.map-${KERNEL_VERSION}
+	cp Module.symvers $kerneldir/
+	cp .config $kerneldir/
+	mkdir -p $kerneldir/include/config
+	cp include/config/kernel.release

$kerneldir/include/config/kernel.release

meta/classes/kernel.bbclass +7 -0

+
+	# As of Linux kernel version 3.0.1, the clean target removes
+	# arch/powerpc/lib/crtsavres.o which is present in
+	# KBUILD_LDFLAGS_MODULE, making it required to build external modules.
+	if [ ${ARCH} = "powerpc" ]; then
+		mkdir -p $kerneldir/arch/powerpc/lib/
+		cp arch/powerpc/lib/crtsavres.o

$kerneldir/arch/powerpc/lib/crtsavres.o

meta/classes/kernel.bbclass +2 -0

+	fi
+

Quoted 8 lines

meta/classes/kernel.bbclass
Quoted 8 lines
meta/classes/kernel.bbclass +6 -0
+	mkdir -p $kerneldir/include/generated/
+	cp -fR include/generated/* $kerneldir/include/generated/
+
+	if [ -d arch/${ARCH}/include/generated ]; then
+		mkdir -p $kerneldir/arch/${ARCH}/include/generated/
+		cp -fR arch/${ARCH}/include/generated/*
$kerneldir/arch/${ARCH}/include/generated/
meta/classes/kernel.bbclass +1 -0
+	fi

Some of the above is also present in do_install, do we need it in both places? If so, we should abstract it if possible to avoid duplication and risk of only one changing in the future when both should.

Quoted 22 lines

Quoted 22 lines
meta/classes/kernel.bbclass +3 -0
+}
+
+
  python sysroot_stage_all () {
      oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"),
d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}"))
}
meta/classes/module-base.bbclass +0 -1
index 9537ba9..c34eee7 100644
@@ -3,16 +3,18 @@ inherit kernel-arch
  export OS = "${TARGET_OS}"
  export CROSS_COMPILE = "${TARGET_PREFIX}"
 
-export KERNEL_VERSION =
"${@base_read_file('${STAGING_KERNEL_DIR}/kernel-abiversion')}" +export KERNEL_VERSION = "${@base_read_file('${STAGING_KERNEL_BUILDDIR}/kernel-abiversion')}"
KERNEL_OBJECT_SUFFIX = ".ko"

Quoted 4 lines

Quoted 4 lines
  # kernel modules are generally machine specific
  PACKAGE_ARCH = "${MACHINE_ARCH}"
+do_configure[depends] += "virtual/kernel:do_install"

I¹m not clear on the separation of do_shared_workdir and do_install now...

Quoted 88 lines

Quoted 87 lines

meta/classes/module-base.bbclass +2 -1

+
  # Function to ensure the kernel scripts are created. Expected to
  # be called before do_compile. See module.bbclass for an exmaple.
  do_make_scripts() {
  	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
  	make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
-	           -C ${STAGING_KERNEL_DIR} scripts
+	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR}

scripts
}

meta/classes/module.bbclass +2 -0

index ad6f7af..4f466d7 100644
@@ -13,6 +13,7 @@ module_do_compile() {
  		   KERNEL_VERSION=${KERNEL_VERSION}    \
  		   CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
  		   AR="${KERNEL_AR}" \
+	           O=${STAGING_KERNEL_BUILDDIR} \
  		   ${MAKE_TARGETS}
  }

@@ -21,6 +22,7 @@ module_do_install() {
  	oe_runmake DEPMOD=echo INSTALL_MOD_PATH="${D}" \
  	           KERNEL_SRC=${STAGING_KERNEL_DIR} \
  	           CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
+	           O=${STAGING_KERNEL_BUILDDIR} \
  	           modules_install
  }

meta/conf/bitbake.conf +2 -1

index 362d6b6..b1135c2 100644
@@ -394,7 +394,8 @@ SDKPATHNATIVE = "${SDKPATH}/sysroots/${SDK_SYS}"
  ##################################################################
 
  OLDEST_KERNEL = "2.6.32"
-STAGING_KERNEL_DIR = "${STAGING_DIR_HOST}/usr/src/kernel"
+STAGING_KERNEL_DIR = "${TMPDIR}/work-shared/${MACHINE}/kernel-source"
+STAGING_KERNEL_BUILDDIR =

"${TMPDIR}/work-shared/${MACHINE}/kernel-build"

##################################################################
# Specific image creation and rootfs population info.

meta/recipes-kernel/lttng/lttng-modules/sepbuild.patch new file +17 -0

new file mode 100644
index 0000000..1b46558
@@ -0,0 +1,17 @@
+Index: git/Makefile
+===================================================================
+--- git.orig/Makefile
++++ git/Makefile
+@@ -67,10 +67,10 @@ else # KERNELRELEASE
+ 	CFLAGS = $(EXTCFLAGS)
+
+ default:
+-	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) modules
++	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) O=$(O) modules
+
+ modules_install:
+-	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) modules_install
++	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) O=$(O) modules_install
+
+ clean:
+ 	$(MAKE) -C $(KERNEL_SRC) M=$(PWD) clean

meta/recipes-kernel/lttng/lttng-modules_2.5.2.bb +0 -0

index 55df07f..58a4196 100644
@@ -19,6 +19,7 @@ SRC_URI =
 "git://git.lttng.org/lttng-modules.git;branch=stable-2.5 \
 
 file://lttng-modules-replace-KERNELDIR-with-KERNEL_SRC.patch \
 
 file://Fix-noargs-probes-should-calculate-alignment-and-eve.patch \

file://Update-kvm-instrumentation-compile-on-3.17-rc1.patch \

+           file://sepbuild.patch \
             "

export INSTALL_MOD_DIR="kernel/lttng-modules"

Richard Purdie

author

2015-01-07 21:16 UTC #3 lore raw

Quoted 9 lines · Darren Hart

Quoted 8 lines · Bruce Ashfield
Quoted 7 lines · Richard Purdie
External module builds do work in this configuration *if* you pass in the correct options e.g.:
make -C work-shared/MACHINE/kernel-source
O=work-shared/MACHINE/kernel-build M=${S}

Quoted 7 lines

Quoted 1 line
Quoted 1 line
I've put together a quick proof of concept of this below.
I was concerned about how this would impact hello-mod and recipes modeled after it, however, in reviewing the patch below, it looks to have this covered. I¹ll verify this just as soon as my workstation is available (it¹s ³busy² updatingŠ sigh).

I have not tested that yet, however it is something we need to do, if it has issues, I think they should be solvable.

Quoted 28 lines

A couple of first pass questions belowŠ prior to applying and testing myself...

Quoted 25 lines

meta/classes/kernel-module-split.bbclass +1 -1

index 9a95b72..2d43b51 100644
@@ -70,7 +70,7 @@ python split_kernel_module_packages () {
          m = kerverrexp.match(kernelver)
          if m:
              kernelver_stripped = m.group(1)
-        staging_kernel_dir = d.getVar("STAGING_KERNEL_DIR", True)
+        staging_kernel_dir = d.getVar("STAGING_KERNEL_BUILDDIR", True)
          system_map_file = "%s/boot/System.map-%s" % (dvar, kernelver)
          if not os.path.exists(system_map_file):
              system_map_file = "%s/System.map-%s" %

(staging_kernel_dir, kernelver)

meta/classes/kernel.bbclass +0 -0

index 5cabc2c..b1a1ccf 100644
@@ -249,6 +249,10 @@ kernel_do_install() {
  	cp .config $kerneldir/
  	mkdir -p $kerneldir/include/config
  	cp include/config/kernel.release

$kerneldir/include/config/kernel.release

meta/classes/kernel.bbclass +1 -0

+	if [ -e include/linux/version.h ]; then

Quoted 8 lines

Quoted 3 lines
meta/classes/kernel.bbclass
Quoted 3 lines
meta/classes/kernel.bbclass +3 -0
+		mkdir -p $kerneldir/include/linux
+		cp include/linux/version.h $kerneldir/include/linux/version.h
+	fi
I know this bit someone with the recent changes, let¹s make sure we add a comment as to why this is required so we don¹t have to dig it out of the archives in a year when we¹re doing another round of fixes in this area.

I know Bruce is not convinced about this part yet and talked with Otavio a little. We could do something like this:

meta/classes/module-base.bbclass +3 -1

index c34eee7..58ec3f4 100644
@@ -11,10 +11,12 @@ PACKAGE_ARCH = "${MACHINE_ARCH}"
 
 do_configure[depends] += "virtual/kernel:do_install"
 
+EXTRA_KERNELSRC_TARGETS = ""
+
 # Function to ensure the kernel scripts are created. Expected to
 # be called before do_compile. See module.bbclass for an exmaple.
 do_make_scripts() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS 
 	make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
-	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts
+	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts ${EXTRA_KERNELSRC_TARGETS}
 }

and then Otavio could add EXTRA_KERNELSRC_TARGETS += "linux/kernel/version.h" to the two problematic recipes.

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
# As of Linux kernel version 3.0.1, the clean target removes
  	# arch/powerpc/lib/crtsavres.o which is present in
meta/classes/kernel.bbclass +0 -0
@@ -268,6 +272,45 @@ kernel_do_install() {
  }
  do_install[prefuncs] += "package_get_auto_pr"
 

Quoted 15 lines

Quoted 4 lines
Quoted 4 lines
meta/classes/kernel.bbclass +4 -0
+addtask do_shared_workdir after do_compile before do_install
+
+do_shared_workdir () {
+	cd ${B}
Quoted 8 lines
meta/classes/kernel.bbclass kernel_do_install() { · line 286
Quoted 8 lines
meta/classes/kernel.bbclass +8 -0
+
+	kerneldir=${STAGING_KERNEL_BUILDDIR}
+	install -d $kerneldir
+
+	#
+	# Store the kernel version in sysroots for module-base.bbclass
+	#
+
OK, how does this impact the dependency on do_installŠ As we¹re doing stuff for building modules here, can that now depend on do_shared_workdir?

Yes, it can and the contents of do_install can be further pruned.

Quoted 23 lines

meta/classes/kernel.bbclass +8 -0

+	echo "${KERNEL_VERSION}" > $kerneldir/kernel-abiversion
+	
+	# Copy files required for module builds
+	cp System.map $kerneldir/System.map-${KERNEL_VERSION}
+	cp Module.symvers $kerneldir/
+	cp .config $kerneldir/
+	mkdir -p $kerneldir/include/config
+	cp include/config/kernel.release

$kerneldir/include/config/kernel.release

meta/classes/kernel.bbclass +7 -0

+
+	# As of Linux kernel version 3.0.1, the clean target removes
+	# arch/powerpc/lib/crtsavres.o which is present in
+	# KBUILD_LDFLAGS_MODULE, making it required to build external modules.
+	if [ ${ARCH} = "powerpc" ]; then
+		mkdir -p $kerneldir/arch/powerpc/lib/
+		cp arch/powerpc/lib/crtsavres.o

$kerneldir/arch/powerpc/lib/crtsavres.o

meta/classes/kernel.bbclass +6 -0

+	fi
+
+	mkdir -p $kerneldir/include/generated/
+	cp -fR include/generated/* $kerneldir/include/generated/
+
+	if [ -d arch/${ARCH}/include/generated ]; then

Quoted 8 lines

Quoted 4 lines
meta/classes/kernel.bbclass
Quoted 4 lines
meta/classes/kernel.bbclass +2 -0
+		mkdir -p $kerneldir/arch/${ARCH}/include/generated/
+		cp -fR arch/${ARCH}/include/generated/*
$kerneldir/arch/${ARCH}/include/generated/
meta/classes/kernel.bbclass +1 -0
+	fi
Some of the above is also present in do_install, do we need it in both places? If so, we should abstract it if possible to avoid duplication and risk of only one changing in the future when both should.

do_shared_workdir would be specifically to populate STAGING_KERNEL_BUILDDIR.

do_install would become "for target use only" in the form of packages. We may well defer all of that to the kernel-devsrc recipe. The patch was a proof of concept, not a complete implementation. If there is any commonality left, we need to share a function, agreed.

Quoted 22 lines

Quoted 22 lines
Quoted 22 lines
meta/classes/kernel.bbclass +3 -0
+}
+
+
  python sysroot_stage_all () {
      oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"),
d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}"))
}
meta/classes/module-base.bbclass +0 -1
index 9537ba9..c34eee7 100644
@@ -3,16 +3,18 @@ inherit kernel-arch
  export OS = "${TARGET_OS}"
  export CROSS_COMPILE = "${TARGET_PREFIX}"
 
-export KERNEL_VERSION =
"${@base_read_file('${STAGING_KERNEL_DIR}/kernel-abiversion')}" +export KERNEL_VERSION = "${@base_read_file('${STAGING_KERNEL_BUILDDIR}/kernel-abiversion')}"
KERNEL_OBJECT_SUFFIX = ".ko"

Quoted 7 lines

Quoted 4 lines
Quoted 4 lines
  # kernel modules are generally machine specific
  PACKAGE_ARCH = "${MACHINE_ARCH}"
+do_configure[depends] += "virtual/kernel:do_install"
I¹m not clear on the separation of do_shared_workdir and do_install now...

See above, this probably needs to change.

The patch was really a test to see how badly a build would explode with separate source and builddir and whether the kernel build system can cope with three separate locations (source, build objects, module). From that perspective it passed in that I could build oprofile, perf and some kernel modules. From here we need a step back and to go through and do it "properly".

Cheers,

Richard

Bruce Ashfield

2015-01-07 21:20 UTC #4 lore raw

Quoted 11 lines · Richard Purdie

Quoted 10 lines · Darren Hart
Quoted 9 lines · Bruce Ashfield
Quoted 8 lines · Richard Purdie
External module builds do work in this configuration *if* you pass in the correct options e.g.:
make -C work-shared/MACHINE/kernel-source
O=work-shared/MACHINE/kernel-build M=${S}
I've put together a quick proof of concept of this below.

<snip>

Quoted 10 lines

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
KERNEL_OBJECT_SUFFIX = ".ko"
   # kernel modules are generally machine specific
   PACKAGE_ARCH = "${MACHINE_ARCH}"
+do_configure[depends] += "virtual/kernel:do_install"
I¹m not clear on the separation of do_shared_workdir and do_install now...

Quoted 8 lines

See above, this probably needs to change.
The patch was really a test to see how badly a build would explode with separate source and builddir and whether the kernel build system can cope with three separate locations (source, build objects, module). From that perspective it passed in that I could build oprofile, perf and some kernel modules. From here we need a step back and to go through and do it "properly".

Agreed. I've queued all the known patches, and have proven that my builds work with this applied with minor adaptations.

I've got a first pass through with the consolidation and some other scribbles notes. I'll generate a topic branch and send it out when it looks reasonably sane (but not complete .. since we don't want to sit on large private changes).

Cheers,

Bruce

Quoted 5 lines

Cheers,
Richard

Robert Yang

2015-01-13 01:19 UTC #5 lore raw

In reply to Richard Purdie

Quoted 5 lines · Richard Purdie

I'm hearing (somewhat justified) complaints that the recent kernel changes have destablised builds. Part of the question is whether the recent changes are as clear to users as they could be, we're also running into some problems due to mixing kernel source and build artefacts in some places and not in others.

Currently, the kernel_do_install directly installs the files to sysroot, then we will get the errors like the following when rebuild:

| Using /buildarea/lyang1/testup3/tmp/sysroots/qemux86-64/usr/src/kernel as source for kernel

|   /buildarea/lyang1/testup3/tmp/sysroots/qemux86-64/usr/src/kernel is not clean, please run 'make mrproper'
|   in the '/buildarea/lyang1/testup3/tmp/sysroots/qemux86-64/usr/src/kernel' directory.

| make[3]: Nothing to be done for `relocs'.

How about we add a kerne-src.rpm which includes the usr/src/kernel, and use populate_sysroot to install the files to sysroots/qemux86-64/usr/src/kernel ?

// Robert

Quoted 227 lines