Skip to contentSkip to conversation

kernel.bbclass: copy some more needed files in do_shared_workdir

PATCH In review Dan Callaghan 2 participants 6 messages Apr 21, 2020 – Apr 24, 2020 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kernel.bbclass: copy some more needed files in do_shared_workdir

We need to copy include/config/auto.conf into ${STAGING_KERNEL_BUILDDIR} to avoid module compilation errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |
    |   ERROR: Kernel configuration is invalid.
    |          include/generated/autoconf.h or include/config/auto.conf are missing.
    |          Run 'make oldconfig && make prepare' on kernel src to fix it.
    |
    | make[2]: *** [build/tmp-glibc/work-shared/machine/kernel-source/Makefile:641: include/config/auto.conf] Error 1

We also need scripts/basic/fixdep to avoid errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |   CC [M]  build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o
    | /bin/sh: scripts/basic/fixdep: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.build:303: build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o] Error 127

And scripts/mod/modpost to avoid errors like this:

    |   Building modules, stage 2.
    |   MODPOST 1 modules
    | /bin/sh: scripts/mod/modpost: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.modpost:92: __modpost] Error 127

Most of the time, the absence of these files is masked by the make-mod-scripts recipe's do_configure task, which causes a whole bunch of extra files to be built into ${STAGING_KERNEL_BUILDDIR} (including these two). Typically make-mod-scripts will be built before any kernel modules, and so it didn't matter that do_shared_workdir was missing these files.

However the problem will be evident if you do certain sequences of tasks, for example a complete build first (so that make-mod-scripts is already built), then:

bitbake linux-yocto my-kernel-module -c clean
bitbake my-kernel-module -c install

---
 meta/classes/kernel.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes/kernel.bbclass +5 -0
index a724645466..5a49d01aa5 100644
@@ -452,6 +452,7 @@ do_shared_workdir () {
 	cp .config $kerneldir/
 	mkdir -p $kerneldir/include/config
 	cp include/config/kernel.release $kerneldir/include/config/kernel.release
+	cp include/config/auto.conf $kerneldir/include/config/
 	if [ -e certs/signing_key.x509 ]; then
 		# The signing_key.* files are stored in the certs/ dir in
 		# newer Linux kernels
@@ -460,6 +461,10 @@ do_shared_workdir () {
 	elif [ -e signing_key.priv ]; then
 		cp signing_key.* $kerneldir/
 	fi
+	mkdir -p $kerneldir/scripts/basic
+	cp scripts/basic/fixdep $kerneldir/scripts/basic/
+	mkdir -p $kerneldir/scripts/mod
+	cp scripts/mod/modpost $kerneldir/scripts/mod/
 
 	# We can also copy over all the generated files and avoid special cases
 	# like version.h, but we've opted to keep this small until file creep starts
Signature · 1 line
2.21.1
#1 lore raw
Quoted 28 lines · Dan Callaghan

We need to copy include/config/auto.conf into ${STAGING_KERNEL_BUILDDIR} to avoid module compilation errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |
    |   ERROR: Kernel configuration is invalid.
    |          include/generated/autoconf.h or include/config/auto.conf are missing.
    |          Run 'make oldconfig && make prepare' on kernel src to fix it.
    |
    | make[2]: *** [build/tmp-glibc/work-shared/machine/kernel-source/Makefile:641: include/config/auto.conf] Error 1

We also need scripts/basic/fixdep to avoid errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |   CC [M]  build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o
    | /bin/sh: scripts/basic/fixdep: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.build:303: build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o] Error 127

And scripts/mod/modpost to avoid errors like this:

    |   Building modules, stage 2.
    |   MODPOST 1 modules
    | /bin/sh: scripts/mod/modpost: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.modpost:92: __modpost] Error 127
Quoted 7 lines

Most of the time, the absence of these files is masked by the make-mod-scripts recipe's do_configure task, which causes a whole bunch of extra files to be built into ${STAGING_KERNEL_BUILDDIR} (including these two). Typically make-mod-scripts will be built before any kernel modules, and so it didn't matter that do_shared_workdir was missing these files.

The design is that, make-mod-scripts is what provides those files, we don't want to copy them, since it is an ever incomplete list (more so in the past, but the concept still holds).

Quoted 6 lines

However the problem will be evident if you do certain sequences of tasks, for example a complete build first (so that make-mod-scripts is already built), then:

bitbake linux-yocto my-kernel-module -c clean
bitbake my-kernel-module -c install

Why can't we fix the dependency ? We chased a few of these around when the mod-scripts recipe was introduced, and this just looks like a variant we missed.

Bruce

Quoted 33 lines
---
 meta/classes/kernel.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes/kernel.bbclass +5 -1
index a724645466..5a49d01aa5 100644
@@ -452,6 +452,7 @@ do_shared_workdir () {
        cp .config $kerneldir/
        mkdir -p $kerneldir/include/config
        cp include/config/kernel.release $kerneldir/include/config/kernel.release
+       cp include/config/auto.conf $kerneldir/include/config/
        if [ -e certs/signing_key.x509 ]; then
                # The signing_key.* files are stored in the certs/ dir in
                # newer Linux kernels
@@ -460,6 +461,10 @@ do_shared_workdir () {
        elif [ -e signing_key.priv ]; then
                cp signing_key.* $kerneldir/
        fi
+       mkdir -p $kerneldir/scripts/basic
+       cp scripts/basic/fixdep $kerneldir/scripts/basic/
+       mkdir -p $kerneldir/scripts/mod
+       cp scripts/mod/modpost $kerneldir/scripts/mod/
 
        # We can also copy over all the generated files and avoid special cases
        # like version.h, but we've opted to keep this small until file creep starts
--

2.21.1

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#2 lore raw

Excerpts from Bruce Ashfield's message of 2020-04-21 08:33:34 -04:00:

Quoted 38 lines
Quoted 28 lines · Dan Callaghan

We need to copy include/config/auto.conf into ${STAGING_KERNEL_BUILDDIR} to avoid module compilation errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |
    |   ERROR: Kernel configuration is invalid.
    |          include/generated/autoconf.h or include/config/auto.conf are missing.
    |          Run 'make oldconfig && make prepare' on kernel src to fix it.
    |
    | make[2]: *** [build/tmp-glibc/work-shared/machine/kernel-source/Makefile:641: include/config/auto.conf] Error 1

We also need scripts/basic/fixdep to avoid errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |   CC [M]  build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o
    | /bin/sh: scripts/basic/fixdep: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.build:303: build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o] Error 127

And scripts/mod/modpost to avoid errors like this:

    |   Building modules, stage 2.
    |   MODPOST 1 modules
    | /bin/sh: scripts/mod/modpost: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.modpost:92: __modpost] Error 127
Quoted 7 lines

Most of the time, the absence of these files is masked by the make-mod-scripts recipe's do_configure task, which causes a whole bunch of extra files to be built into ${STAGING_KERNEL_BUILDDIR} (including these two). Typically make-mod-scripts will be built before any kernel modules, and so it didn't matter that do_shared_workdir was missing these files.

Quoted 3 lines

The design is that, make-mod-scripts is what provides those files, we don't want to copy them, since it is an ever incomplete list (more so in the past, but the concept still holds).

Okay, I think you're saying that what do_shared_workdir claims to do (populate ${STAGING_KERNEL_BUILDDIR} with whatever files are needed to build kernel modules) should actually be handled by make-mod-scripts instead.

Does that mean, do_shared_workdir has no purpose? Or is it doing something else I don't realise?

Quoted 10 lines
Quoted 6 lines

However the problem will be evident if you do certain sequences of tasks, for example a complete build first (so that make-mod-scripts is already built), then:

bitbake linux-yocto my-kernel-module -c clean
bitbake my-kernel-module -c install

Why can't we fix the dependency ? We chased a few of these around when the mod-scripts recipe was introduced, and this just looks like a variant we missed.

Right, so there *is* already a correct dependency on make-mod-scripts:do_compile. The problem is, in the scenario I gave above, that task is already done and Bitbake is not going to run it again. But in fact, the stuff it left behind in ${STAGING_KERNEL_BUILDDIR} was cleaned out when I cleaned the linux-yocto recipe.

I guess the real issue is that we have this "shared" work directory which a bunch of different recipes contribute files to, but can be cleaned out by running the "clean" task on any one of several different recipes.

Signature · 3 lines
Dan Callaghan <[email protected]>
Software Engineer
Opengear <https://opengear.com/>
#3 lore raw
Quoted 44 lines · Dan Callaghan

Excerpts from Bruce Ashfield's message of 2020-04-21 08:33:34 -04:00:

Quoted 38 lines
Quoted 28 lines · Dan Callaghan

We need to copy include/config/auto.conf into ${STAGING_KERNEL_BUILDDIR} to avoid module compilation errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |
    |   ERROR: Kernel configuration is invalid.
    |          include/generated/autoconf.h or include/config/auto.conf are missing.
    |          Run 'make oldconfig && make prepare' on kernel src to fix it.
    |
    | make[2]: *** [build/tmp-glibc/work-shared/machine/kernel-source/Makefile:641: include/config/auto.conf] Error 1

We also need scripts/basic/fixdep to avoid errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |   CC [M]  build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o
    | /bin/sh: scripts/basic/fixdep: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.build:303: build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o] Error 127

And scripts/mod/modpost to avoid errors like this:

    |   Building modules, stage 2.
    |   MODPOST 1 modules
    | /bin/sh: scripts/mod/modpost: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.modpost:92: __modpost] Error 127
Quoted 7 lines

Most of the time, the absence of these files is masked by the make-mod-scripts recipe's do_configure task, which causes a whole bunch of extra files to be built into ${STAGING_KERNEL_BUILDDIR} (including these two). Typically make-mod-scripts will be built before any kernel modules, and so it didn't matter that do_shared_workdir was missing these files.

Quoted 3 lines

The design is that, make-mod-scripts is what provides those files, we don't want to copy them, since it is an ever incomplete list (more so in the past, but the concept still holds).

Quoted 7 lines

Okay, I think you're saying that what do_shared_workdir claims to do (populate ${STAGING_KERNEL_BUILDDIR} with whatever files are needed to build kernel modules) should actually be handled by make-mod-scripts instead.

Does that mean, do_shared_workdir has no purpose? Or is it doing something else I don't realise?

How would what populates it change the purpose of what it does ? Just like our SDK and on target module support, those files are generated by the kernel build itself, not copied.

shared-workdir was created to stop slinging around the sheer number of files in a kernel build, it was causing us severe build performance issues.

Having the smallest possible list of explicitly copied files is the goal. The lists are fragile, and inevitably need constant tweaking. Witness the messing around with kernel-devsrc that I have to do, and the only reason that is tolerated is to make the minimal possible development components.

Quoted 12 lines
Quoted 11 lines
Quoted 7 lines

However the problem will be evident if you do certain sequences of tasks, for example a complete build first (so that make-mod-scripts is already built), then:

bitbake linux-yocto my-kernel-module -c clean
bitbake my-kernel-module -c install

Why can't we fix the dependency ? We chased a few of these around when the mod-scripts recipe was introduced, and this just looks like a variant we missed.

Quoted 6 lines

Right, so there *is* already a correct dependency on make-mod-scripts:do_compile. The problem is, in the scenario I gave above, that task is already done and Bitbake is not going to run it again. But in fact, the stuff it left behind in ${STAGING_KERNEL_BUILDDIR} was cleaned out when I cleaned the linux-yocto recipe.

Yes, and that clean is supposed to invalidate things, and trigger the re-run.

Quoted 5 lines

I guess the real issue is that we have this "shared" work directory which a bunch of different recipes contribute files to, but can be cleaned out by running the "clean" task on any one of several different recipes.

We've solved that issue in the past, I'll have a look on Wednesday to see what may have changed and (re)broken it. There may be a bugzilla floating around for that, I'll see if I can track it down as well.

Bruce

Quoted 5 lines

--
Dan Callaghan <[email protected]>
Software Engineer
Opengear <https://opengear.com/>

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 87 lines · Bruce Ashfield via lists.openembedded.org
Quoted 44 lines · Dan Callaghan

Excerpts from Bruce Ashfield's message of 2020-04-21 08:33:34 -04:00:

Quoted 38 lines
Quoted 28 lines · Dan Callaghan

We need to copy include/config/auto.conf into ${STAGING_KERNEL_BUILDDIR} to avoid module compilation errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |
    |   ERROR: Kernel configuration is invalid.
    |          include/generated/autoconf.h or include/config/auto.conf are missing.
    |          Run 'make oldconfig && make prepare' on kernel src to fix it.
    |
    | make[2]: *** [build/tmp-glibc/work-shared/machine/kernel-source/Makefile:641: include/config/auto.conf] Error 1

We also need scripts/basic/fixdep to avoid errors like this:

    | make[1]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-source'
    | make[2]: Entering directory 'build/tmp-glibc/work-shared/machine/kernel-build-artifacts'
    |   CC [M]  build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o
    | /bin/sh: scripts/basic/fixdep: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.build:303: build/tmp-glibc/work/vendor-linux/tehuti-module/0.3.6-r0/git/TLK10232_phy_Linux.o] Error 127

And scripts/mod/modpost to avoid errors like this:

    |   Building modules, stage 2.
    |   MODPOST 1 modules
    | /bin/sh: scripts/mod/modpost: No such file or directory
    | make[3]: *** [build/tmp-glibc/work-shared/machine/kernel-source/scripts/Makefile.modpost:92: __modpost] Error 127
Quoted 7 lines

Most of the time, the absence of these files is masked by the make-mod-scripts recipe's do_configure task, which causes a whole bunch of extra files to be built into ${STAGING_KERNEL_BUILDDIR} (including these two). Typically make-mod-scripts will be built before any kernel modules, and so it didn't matter that do_shared_workdir was missing these files.

Quoted 3 lines

The design is that, make-mod-scripts is what provides those files, we don't want to copy them, since it is an ever incomplete list (more so in the past, but the concept still holds).

Quoted 7 lines

Okay, I think you're saying that what do_shared_workdir claims to do (populate ${STAGING_KERNEL_BUILDDIR} with whatever files are needed to build kernel modules) should actually be handled by make-mod-scripts instead.

Does that mean, do_shared_workdir has no purpose? Or is it doing something else I don't realise?

How would what populates it change the purpose of what it does ? Just like our SDK and on target module support, those files are generated by the kernel build itself, not copied.

shared-workdir was created to stop slinging around the sheer number of files in a kernel build, it was causing us severe build performance issues.

Having the smallest possible list of explicitly copied files is the goal. The lists are fragile, and inevitably need constant tweaking. Witness the messing around with kernel-devsrc that I have to do, and the only reason that is tolerated is to make the minimal possible development components.

Quoted 12 lines
Quoted 11 lines
Quoted 7 lines

However the problem will be evident if you do certain sequences of tasks, for example a complete build first (so that make-mod-scripts is already built), then:

bitbake linux-yocto my-kernel-module -c clean
bitbake my-kernel-module -c install

Why can't we fix the dependency ? We chased a few of these around when the mod-scripts recipe was introduced, and this just looks like a variant we missed.

Quoted 6 lines

Right, so there *is* already a correct dependency on make-mod-scripts:do_compile. The problem is, in the scenario I gave above, that task is already done and Bitbake is not going to run it again. But in fact, the stuff it left behind in ${STAGING_KERNEL_BUILDDIR} was cleaned out when I cleaned the linux-yocto recipe.

Quoted 2 lines

Yes, and that clean is supposed to invalidate things, and trigger the re-run.

Hi Dan,

Are you seeing this on master, or some other branch ?

I can run the following on master, as often as I like:

% bitbake linux-yocto hello-mod -c clean; bitbake hello-mod -c install

And I see the required scripts coming and going from the build artifacts as expected:

build [/home/bruc...-artifacts]> find . -name 'fixdep' ./tools/objtool/fixdep
./scripts/basic/fixdep

# after the clean:
build [/home/bruc...-artifacts]> find . -name 'fixdep'
# after the install
build [/home/bruc...-artifacts]> cd ..
build [/home/bruc...qemux86-64]> cd -
/home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-build-artifacts
build [/home/bruc...-artifacts]> find . -name 'fixdep'
./tools/objtool/fixdep
./scripts/basic/fixdep

Is there something different with your module recipe than the one in meta-skeleton ? Can you share it, so I can debug further ?

Bruce

Quoted 26 lines
Quoted 6 lines

I guess the real issue is that we have this "shared" work directory which a bunch of different recipes contribute files to, but can be cleaned out by running the "clean" task on any one of several different recipes.

We've solved that issue in the past, I'll have a look on Wednesday to see what may have changed and (re)broken it. There may be a bugzilla floating around for that, I'll see if I can track it down as well.

Bruce

Quoted 5 lines

--
Dan Callaghan <[email protected]>
Software Engineer
Opengear <https://opengear.com/>

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#6 lore raw

Excerpts from Bruce Ashfield's message of 2020-04-22 13:40:33 -04:00:

Quoted 1 line

Are you seeing this on master, or some other branch ?

Unfortunately we are still on Warrior, and I can't easily test any newer branches right now.

Quoted 14 lines

I can run the following on master, as often as I like:

% bitbake linux-yocto hello-mod -c clean; bitbake hello-mod -c install

And I see the required scripts coming and going from the build artifacts as expected:

build [/home/bruc...-artifacts]> find . -name 'fixdep' ./tools/objtool/fixdep
./scripts/basic/fixdep

# after the clean:
build [/home/bruc...-artifacts]> find . -name 'fixdep'
Quoted 7 lines
# after the install
build [/home/bruc...-artifacts]> cd ..
build [/home/bruc...qemux86-64]> cd -
/home/bruce/poky/build/tmp/work-shared/qemux86-64/kernel-build-artifacts
build [/home/bruc...-artifacts]> find . -name 'fixdep'
./tools/objtool/fixdep
./scripts/basic/fixdep

Okay, that's interesting. Thanks for testing it.

When you do install on the module, does Bitbake somehow know it must also re-run make-mod-scripts:do_compile, or is fixdep getting put into the kernel-build-artifacts directory by something else?

Quoted 2 lines

Is there something different with your module recipe than the one in meta-skeleton ? Can you share it, so I can debug further ?

We see this with all our out-of-tree kernel module recipes and they are all pretty ordinary, just using module.bbclass. Maybe there is something strange we have done with our kernel recipe which is causing it.

I can't share the entire layer but here is an example of one of the module recipes:

https://gist.github.com/danc86/09143b04f43e47f99ae91bbdee53bf2f

I'll just paste the entire thing here too since it is so short:

SUMMARY = "nct5104d watchdog driver"
DESCRIPTION = "Linux driver for NCT5104D watchdog function"
HOMEPAGE = "https://github.com/opengear/nct5104d-watchdog-driver" LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://LICENSE;md5=b234ee4d69f5fce4486a80fdaf4a4263" SRC_URI = "git://github.com/opengear/nct5104d-watchdog-driver.git;protocol=https;" SRCREV = "5495f4eaa15bb0b8804778df046f5276d66413ce"
PR = "r1"
S = "${WORKDIR}/git"

inherit module

Signature · 3 lines
Dan Callaghan <[email protected]>
Software Engineer
Opengear <https://opengear.com/>