kernel: consolidated pull request

Integrating the following fixes for yaffs2:

9f9649fffac9 yaffs2: Adjust the timer callback and init functions 0ef24a829eee yaffs2: include rawnand.h instead of nand.h

Signed-off-by Kevin Hao <[email protected]>

Signed-off-by Bruce Ashfield <[email protected]>

---
 meta/recipes-kernel/linux/linux-yocto-rt_4.14.bb   |  4 ++--
 meta/recipes-kernel/linux/linux-yocto-rt_4.15.bb   |  4 ++--
 meta/recipes-kernel/linux/linux-yocto-tiny_4.14.bb |  4 ++--
 meta/recipes-kernel/linux/linux-yocto-tiny_4.15.bb |  4 ++--
 meta/recipes-kernel/linux/linux-yocto_4.14.bb      | 18 +++++++++---------
 meta/recipes-kernel/linux/linux-yocto_4.15.bb      | 18 +++++++++---------
 6 files changed, 26 insertions(+), 26 deletions(-)

Hi Bruce,

Quoted 5 lines · Bruce Ashfield

Integrating the following fixes for yaffs2:

9f9649fffac9 yaffs2: Adjust the timer callback and init functions 0ef24a829eee yaffs2: include rawnand.h instead of nand.h

Quoted 8 lines

Signed-off-by Kevin Hao <[email protected]>

Signed-off-by Bruce Ashfield <[email protected]>

---
 meta/recipes-kernel/linux/linux-yocto-rt_4.14.bb   |  4 ++--
 meta/recipes-kernel/linux/linux-yocto-rt_4.15.bb   |  4 ++--
 meta/recipes-kernel/linux/linux-yocto-tiny_4.14.bb |  4 ++--
 meta/recipes-kernel/linux/linux-yocto-tiny_4.15.bb |  4 ++--
 meta/recipes-kernel/linux/linux-yocto_4.14.bb      | 18

+++++++++---------

Quoted 1 line

 meta/recipes-kernel/linux/linux-yocto_4.15.bb      | 18

+++++++++---------

Quoted 1 line

 6 files changed, 26 insertions(+), 26 deletions(-)

<snip>

For MIPS, KernelModuleTest fails in autobuilder ( probably due to this change - haven't bisected it)

https://autobuilder.yocto.io/builders/nightly-mips/builds/
806/steps/Running%20Sanity%20Tests/logs/stdio

RESULTS - kernelmodule.KernelModuleTest.test_kernel_module - Testcase 1541: FAILED

NOTE: ============================================================
==========
| NOTE: FAIL: test_kernel_module (kernelmodule.KernelModuleTest) | NOTE: ------------------------------------------------------------
----------
| NOTE: Traceback (most recent call last):
| File "/home/pokybuild/yocto-autobuilder/yocto-worker/
nightly-mips/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
| return func(*args, **kwargs)
| File "/home/pokybuild/yocto-autobuilder/yocto-worker/
nightly-mips/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
| return func(*args, **kwargs)
| File "/home/pokybuild/yocto-autobuilder/yocto-worker/
nightly-mips/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
| return func(*args, **kwargs)
| File "/home/pokybuild/yocto-autobuilder/yocto-worker/
nightly-mips/build/meta/lib/oeqa/runtime/cases/kernelmodule.py", line 40, in test_kernel_module

|     self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
| AssertionError: 2 != 0 : cd /usr/src/kernel && make scripts
| arch/mips/Makefile:240: arch/mips/Kbuild.platforms: No such file or

directory
| make: *** No rule to make target 'arch/mips/Kbuild.platforms'. Stop.

Best Regards,
Maxin
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

The existing kernel-devsrc package starts with a full copy of the kernel source and then starts to strip out elements that are not required.

This results in extra time (I/O) and extra space being taken up in the final package. The main purpose of the kernel-devsrc package has been to build modules against the running kernel, not to include a full copy of the source code for re-building the kernel. The end result was a 600M kernel-devsrc package.

This restructuring of the package uses an approach similar to other distros, where the kernel-devsrc package is for building against the running kernel and uses a curated set of copied infrastructure, versus a mass copy of the entire kernel.

The differences in this approach versus other is largely due to the architecture support and the split build/source directory of the kernel.

The result is a kernel-devsrc package of about 10M, which is capable of running "make scripts" and compiling kernel modules against the running kernel.

Along with the changes to the copying of the infrascture, we also have the following changes:

- a better/more explicit listing of dependencies for on-target builds of "make scripts" or "make modules_prepare"

- The kernel source is installed into /lib/modules/<version>/build and a symlink created from /usr/src/kernel to the new location. This aligns with the standard location for module support code

- There is also a symlink from /lib/modules/<version>/source -> build to reserve a spot for a new package that is simply the kernel source. That package is not part of this update.

[YOCTO #12535]

Signed-off-by Bruce Ashfield <[email protected]>

---
 meta/recipes-kernel/linux/kernel-devsrc.bb | 178
++++++++++++++++++++++-------
 1 file changed, 138 insertions(+), 40 deletions(-)

meta/recipes-kernel/linux/kernel-devsrc.bb +2 -7

index c1b5b7786d7c..f55b115e0936 100644
@@ -30,50 +30,148 @@ do_populate_sysroot[noexec] = "1"
 S = "${STAGING_KERNEL_DIR}"
 B = "${STAGING_KERNEL_BUILDDIR}"
 
-KERNEL_VERSION = "${@get_kernelversion_headers('${S}')}"
-
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 
+KERNEL_BUILD_ROOT="/lib/modules/"
+
 do_install() {
-        kerneldir=${D}${KERNEL_SRC_PATH}
-        install -d $kerneldir
-
-        #
-        # Copy the staging dir source (and module build support) into the

devsrc structure.
- # We can keep this copy simple and take everything, since a we'll
clean up any build

meta/recipes-kernel/linux/kernel-devsrc.bb +0 -7

-        # artifacts afterwards, and the extra i/o is not significant
-        #
-        cd ${B}
-        find . -type d -name '.git*' -prune -o -path '.debug' -prune -o
-type f -print0 | cpio --null -pdlu $kerneldir
-        cd ${S}
-       find . -type d -name '.git*' -prune -o -type d -name

'.kernel-meta' -prune -o -type f -print0 | cpio --null -pdlu $kerneldir

meta/recipes-kernel/linux/kernel-devsrc.bb +0 -2

-
-        # Explicitly set KBUILD_OUTPUT to ensure that the image directory

is cleaned and not
- # The main build artifacts. We clean the directory to avoid QA
errors on mismatched

meta/recipes-kernel/linux/kernel-devsrc.bb +0 -3

-        # architecture (since scripts and helpers are native format).
-        KBUILD_OUTPUT="$kerneldir"
-        oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}"

clean _mrproper_scripts

meta/recipes-kernel/linux/kernel-devsrc.bb +0 -11

-        # make clean generates an absolute path symlink called "source"
-        # in $kerneldir points to $kerneldir, which doesn't make any
-        # sense, so remove it.
-        if [ -L $kerneldir/source ]; then
-            bbnote "Removing $kerneldir/source symlink"
-            rm -f $kerneldir/source
-        fi
-
-        # As of Linux kernel version 3.0.1, the clean target removes
-        # arch/powerpc/lib/crtsavres.o which is present in
-        # KBUILD_LDFLAGS_MODULE, making it required to build external

modules.

meta/recipes-kernel/linux/kernel-devsrc.bb +0 -3

-        if [ ${ARCH} = "powerpc" ]; then
-                mkdir -p $kerneldir/arch/powerpc/lib/
-                cp ${B}/arch/powerpc/lib/crtsavres.o

$kerneldir/arch/powerpc/lib/crtsavres.o

meta/recipes-kernel/linux/kernel-devsrc.bb +26 -3

-        fi
-
-        chown -R root:root ${D}
+    kerneldir=${D}${KERNEL_BUILD_ROOT}${KERNEL_VERSION}
+    install -d $kerneldir
+
+    # create the directory structure
+    rm -f $kerneldir/build
+    rm -f $kerneldir/source
+    mkdir -p $kerneldir/build
+
+    # for compatibility with some older variants of this package, we
+    # create  a /usr/src/kernel symlink to /lib/modules/<version>/source
+    mkdir -p ${D}/usr/src
+    (
+       cd ${D}/usr/src
+       ln -s ${KERNEL_BUILD_ROOT}${KERNEL_VERSION}/source kernel
+    )
+
+    # for on target purposes, we unify build and source
+    (
+       cd $kerneldir
+       ln -s build source
+    )
+
+    # first copy everything
+    (
+       cd ${S}
+       cp --parents $(find  -type f -name "Makefile*" -o -name

"Kconfig*") $kerneldir/build

meta/recipes-kernel/linux/kernel-devsrc.bb +21 -0

+    )
+
+    # then drop all but the needed Makefiles/Kconfig files
+    rm -rf $kerneldir/build/Documentation
+    rm -rf $kerneldir/build/scripts
+    rm -rf $kerneldir/build/include
+
+    # now copy in parts from the build that we'll need later
+    (
+       cd ${B}
+
+       cp Module.symvers $kerneldir/build
+       cp System.map* $kerneldir/build
+       if [ -s Module.markers ]; then
+           cp Module.markers $kerneldir/build
+       fi
+
+       cp .config $kerneldir/build
+
+       # TODO: this scripts copy will likely blow up QA, so for now, we
+       #       require a more complex 'make scripts' to restore these,

versus

meta/recipes-kernel/linux/kernel-devsrc.bb +19 -0

+       #       copying them here. Left as a reference to indicate that we
+       #       know the scripts must be dealt with.
+       # cp -a scripts $kerneldir/build
+
+       if [ -f tools/objtool/objtool ]; then
+           cp -a tools/objtool/objtool $kerneldir/build/tools/objtool/
+       fi
+       if [ -d arch/${ARCH}/scripts ]; then
+           cp -a arch/${ARCH}/scripts $kerneldir/build/arch/${ARCH}
+       fi
+       if [ -f arch/${ARCH}/*lds ]; then
+           cp -a arch/${ARCH}/*lds $kerneldir/build/arch/${ARCH}
+       fi
+
+       rm -f $kerneldir/build/scripts/*.o
+       rm -f $kerneldir/build/scripts/*/*.o
+
+       if [ "${ARCH}" = "powerpc" ]; then
+           cp -a --parents arch/powerpc/lib/crtsavres.[So]

$kerneldir/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +15 -0

+       fi
+
+       cp -a include $kerneldir/build/include
+    )
+
+    # now grab the chunks from the source tree that we need
+    (
+       cd ${S}
+
+       cp -a scripts $kerneldir/build
+       if [ "${ARCH}" = "arm64" ]; then
+           # arch/arm64/include/asm/xen references arch/arm
+           cp -a --parents arch/arm/include/asm/xen $kerneldir/build/
+           # arch/arm64/include/asm/opcodes.h references arch/arm
+           cp -a --parents arch/arm/include/asm/opcodes.h

$kerneldir/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +3 -0

+       fi
+
+       # include the machine specific headers for ARM variants, if

available.

meta/recipes-kernel/linux/kernel-devsrc.bb +5 -0

+       if [ "${ARCH}" = "arm" ]; then
+           cp -a --parents arch/${ARCH}/mach-*/include $kerneldir/build/
+
+           # include a few files for 'make prepare'
+           cp -a --parents arch/arm/tools/gen-mach-types

$RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
+ cp -a --parents arch/arm/tools/mach-types
$RPM_BUILD_ROOT/lib/modules/$KernelVer/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +11 -0

+       fi
+
+       if [ -d arch/${ARCH}/include ]; then
+           cp -a --parents arch/${ARCH}/include $kerneldir/build/
+       fi
+
+       cp -a include $kerneldir/build
+
+       if [ "${ARCH}" = "x86" ]; then
+           # files for 'make prepare' to succeed with kernel-devel
+           cp -a --parents arch/x86/entry/syscalls/syscall_32.tbl

$kerneldir/build/
+ cp -a --parents arch/x86/entry/syscalls/syscalltbl.sh
$kerneldir/build/
+ cp -a --parents arch/x86/entry/syscalls/syscallhdr.sh
$kerneldir/build/
+ cp -a --parents arch/x86/entry/syscalls/syscall_64.tbl
$kerneldir/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +4 -0

+           cp -a --parents arch/x86/tools/relocs_32.c $kerneldir/build/
+           cp -a --parents arch/x86/tools/relocs_64.c $kerneldir/build/
+           cp -a --parents arch/x86/tools/relocs.c $kerneldir/build/
+           cp -a --parents arch/x86/tools/relocs_common.c

$kerneldir/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +2 -0

+           cp -a --parents arch/x86/tools/relocs.h $kerneldir/build/
+           cp -a --parents tools/include/tools/le_byteshift.h

$kerneldir/build/
+ cp -a --parents tools/include/tools/be_byteshift.h
$kerneldir/build/
+ cp -a --parents arch/x86/purgatory/purgatory.c
$kerneldir/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +5 -0

+           cp -a --parents arch/x86/purgatory/sha256.h $kerneldir/build/
+           cp -a --parents arch/x86/purgatory/sha256.c $kerneldir/build/
+           cp -a --parents arch/x86/purgatory/stack.S $kerneldir/build/
+           cp -a --parents arch/x86/purgatory/string.c $kerneldir/build/
+           cp -a --parents arch/x86/purgatory/setup-x86_64.S

$kerneldir/build/

meta/recipes-kernel/linux/kernel-devsrc.bb +8 -0

+           cp -a --parents arch/x86/purgatory/entry64.S $kerneldir/build/
+           cp -a --parents arch/x86/boot/string.h $kerneldir/build/
+           cp -a --parents arch/x86/boot/string.c $kerneldir/build/
+           cp -a --parents arch/x86/boot/ctype.h $kerneldir/build/
+       fi
+    )
+
+    # Make sure the Makefile and version.h have a matching timestamp so

that

meta/recipes-kernel/linux/kernel-devsrc.bb +2 -0

+    # external modules can be built
+    touch -r $kerneldir/build/Makefile $kerneldir/build/include/

generated/uapi/linux/version.h

meta/recipes-kernel/linux/kernel-devsrc.bb +2 -0

+
+    # Copy .config to include/config/auto.conf so "make prepare" is

unnecessary.

meta/recipes-kernel/linux/kernel-devsrc.bb +7 -4

+    cp $kerneldir/build/.config $kerneldir/build/include/config/auto.conf
+
+    chown -R root:root ${D}
 }
+
 # Ensure we don't race against "make scripts" during cpio
 do_install[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
 
-PACKAGES = "kernel-devsrc"
-FILES_${PN} = "${KERNEL_SRC_PATH}"
-RDEPENDS_${PN} = "bc"
+FILES_${PN} = "${KERNEL_BUILD_ROOT} ${KERNEL_SRC_PATH}"
+FILES_${PN}-dbg += "${KERNEL_BUILD_ROOT}*/build/scripts/*/.debug/*"
+RDEPENDS_${PN} = "bc glibc-utils"
--

2.5.0

Sorry, but this breaks on ARM:

ERROR: kernel-devsrc-1.0-r0 do_install: Function failed: do_install (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.25822)
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.25822
Log data follows:

| DEBUG: Executing shell function do_install
| cp: with --parents, the destination must be a directory
| Try 'cp --help' for more information.
| WARNING: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/run.do_install.25822:1 exit 1 from 'rm -rf $kerneldir/build/include'
| ERROR: Function failed: do_install (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.25822)

Ross

On 28 February 2018 at 02:06, Bruce Ashfield <[email protected] <mailto:[email protected]>> wrote:

    The existing kernel-devsrc package starts with a full copy of the kernel
    source and then starts to strip out elements that are not required.

    This results in extra time (I/O) and extra space being taken up in the
    final package. The main purpose of the kernel-devsrc package has been to
    build modules against the running kernel, not to include a full copy of
    the source code for re-building the kernel. The end result was a
    600M kernel-devsrc package.

    This restructuring of the package uses an approach similar to other
    distros, where the kernel-devsrc package is for building against the
    running kernel and uses a curated set of copied infrastructure, versus
    a mass copy of the entire kernel.

    The differences in this approach versus other is largely due to the
    architecture support and the split build/source directory of the
    kernel.

    The result is a kernel-devsrc package of about 10M, which is capable
    of running "make scripts" and compiling kernel modules against the
    running kernel.

    Along with the changes to the copying of the infrascture, we also
    have the following changes:

      - a better/more explicit listing of dependencies for on-target
        builds of "make scripts" or "make modules_prepare"

      - The kernel source is installed into /lib/modules/<version>/build
        and a symlink created from /usr/src/kernel to the new location.
        This aligns with the standard location for module support
        code

      - There is also a symlink from /lib/modules/<version>/source -> build
        to reserve a spot for a new package that is simply the kernel
        source. That package is not part of this update.

    [YOCTO #12535]

    Signed-off-by: Bruce Ashfield <[email protected]
    <mailto:[email protected]>>
    ---
      meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb> | 178 ++++++++++++++++++++++-------
      1 file changed, 138 insertions(+), 40 deletions(-)

    diff --git a/meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb>
    b/meta/recipes-kernel/linux/kernel-devsrc.bb <http://kernel-devsrc.bb>
    index c1b5b7786d7c..f55b115e0936 100644
    --- a/meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb>
    +++ b/meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb>
    @@ -30,50 +30,148 @@ do_populate_sysroot[noexec] = "1"
      S = "${STAGING_KERNEL_DIR}"
      B = "${STAGING_KERNEL_BUILDDIR}"

    -KERNEL_VERSION = "${@get_kernelversion_headers('${S}')}"
    -
      PACKAGE_ARCH = "${MACHINE_ARCH}"

    +KERNEL_BUILD_ROOT="/lib/modules/"
    +
      do_install() {
    -        kerneldir=${D}${KERNEL_SRC_PATH}
    -        install -d $kerneldir
    -
    -        #
    -        # Copy the staging dir source (and module build support)
    into the devsrc structure.
    -        # We can keep this copy simple and take everything, since a
    we'll clean up any build
    -        # artifacts afterwards, and the extra i/o is not significant
    -        #
    -        cd ${B}
    -        find . -type d -name '.git*' -prune -o -path '.debug'
    -prune -o -type f -print0 | cpio --null -pdlu $kerneldir
    -        cd ${S}
    -       find . -type d -name '.git*' -prune -o -type d -name
    '.kernel-meta' -prune -o -type f -print0 | cpio --null -pdlu $kerneldir
    -
    -        # Explicitly set KBUILD_OUTPUT to ensure that the image
    directory is cleaned and not
    -        # The main build artifacts. We clean the directory to avoid
    QA errors on mismatched
    -        # architecture (since scripts and helpers are native format).
    -        KBUILD_OUTPUT="$kerneldir"
    -        oe_runmake -C $kerneldir CC="${KERNEL_CC}"
    LD="${KERNEL_LD}" clean _mrproper_scripts
    -        # make clean generates an absolute path symlink called "source"
    -        # in $kerneldir points to $kerneldir, which doesn't make any
    -        # sense, so remove it.
    -        if [ -L $kerneldir/source ]; then
    -            bbnote "Removing $kerneldir/source symlink"
    -            rm -f $kerneldir/source
    -        fi
    -
    -        # As of Linux kernel version 3.0.1, the clean target removes
    -        # arch/powerpc/lib/crtsavres.o which is present in
    -        # KBUILD_LDFLAGS_MODULE, making it required to build
    external modules.
    -        if [ ${ARCH} = "powerpc" ]; then
    -                mkdir -p $kerneldir/arch/powerpc/lib/
    -                cp ${B}/arch/powerpc/lib/crtsavres.o
    $kerneldir/arch/powerpc/lib/crtsavres.o
    -        fi
    -
    -        chown -R root:root ${D}
    +    kerneldir=${D}${KERNEL_BUILD_ROOT}${KERNEL_VERSION}
    +    install -d $kerneldir
    +
    +    # create the directory structure
    +    rm -f $kerneldir/build
    +    rm -f $kerneldir/source
    +    mkdir -p $kerneldir/build
    +
    +    # for compatibility with some older variants of this package, we
    +    # create  a /usr/src/kernel symlink to
    /lib/modules/<version>/source
    +    mkdir -p ${D}/usr/src
    +    (
    +       cd ${D}/usr/src
    +       ln -s ${KERNEL_BUILD_ROOT}${KERNEL_VERSION}/source kernel
    +    )
    +
    +    # for on target purposes, we unify build and source
    +    (
    +       cd $kerneldir
    +       ln -s build source
    +    )
    +
    +    # first copy everything
    +    (
    +       cd ${S}
    +       cp --parents $(find  -type f -name "Makefile*" -o -name
    "Kconfig*") $kerneldir/build
    +    )
    +
    +    # then drop all but the needed Makefiles/Kconfig files
    +    rm -rf $kerneldir/build/Documentation
    +    rm -rf $kerneldir/build/scripts
    +    rm -rf $kerneldir/build/include
    +
    +    # now copy in parts from the build that we'll need later
    +    (
    +       cd ${B}
    +
    +       cp Module.symvers $kerneldir/build
    +       cp System.map* $kerneldir/build
    +       if [ -s Module.markers ]; then
    +           cp Module.markers $kerneldir/build
    +       fi
    +
    +       cp .config $kerneldir/build
    +
    +       # TODO: this scripts copy will likely blow up QA, so for now, we
    +       #       require a more complex 'make scripts' to restore
    these, versus
    +       #       copying them here. Left as a reference to indicate
    that we
    +       #       know the scripts must be dealt with.
    +       # cp -a scripts $kerneldir/build
    +
    +       if [ -f tools/objtool/objtool ]; then
    +           cp -a tools/objtool/objtool $kerneldir/build/tools/objtool/
    +       fi
    +       if [ -d arch/${ARCH}/scripts ]; then
    +           cp -a arch/${ARCH}/scripts $kerneldir/build/arch/${ARCH}
    +       fi
    +       if [ -f arch/${ARCH}/*lds ]; then
    +           cp -a arch/${ARCH}/*lds $kerneldir/build/arch/${ARCH}
    +       fi
    +
    +       rm -f $kerneldir/build/scripts/*.o
    +       rm -f $kerneldir/build/scripts/*/*.o
    +
    +       if [ "${ARCH}" = "powerpc" ]; then
    +           cp -a --parents arch/powerpc/lib/crtsavres.[So]
    $kerneldir/build/
    +       fi
    +
    +       cp -a include $kerneldir/build/include
    +    )
    +
    +    # now grab the chunks from the source tree that we need
    +    (
    +       cd ${S}
    +
    +       cp -a scripts $kerneldir/build
    +       if [ "${ARCH}" = "arm64" ]; then
    +           # arch/arm64/include/asm/xen references arch/arm
    +           cp -a --parents arch/arm/include/asm/xen $kerneldir/build/
    +           # arch/arm64/include/asm/opcodes.h references arch/arm
    +           cp -a --parents arch/arm/include/asm/opcodes.h
    $kerneldir/build/
    +       fi
    +
    +       # include the machine specific headers for ARM variants, if
    available.
    +       if [ "${ARCH}" = "arm" ]; then
    +           cp -a --parents arch/${ARCH}/mach-*/include
    $kerneldir/build/
    +
    +           # include a few files for 'make prepare'
    +           cp -a --parents arch/arm/tools/gen-mach-types
    $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
    +           cp -a --parents arch/arm/tools/mach-types
    $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
    +       fi
    +
    +       if [ -d arch/${ARCH}/include ]; then
    +           cp -a --parents arch/${ARCH}/include $kerneldir/build/
    +       fi
    +
    +       cp -a include $kerneldir/build
    +
    +       if [ "${ARCH}" = "x86" ]; then
    +           # files for 'make prepare' to succeed with kernel-devel
    +           cp -a --parents arch/x86/entry/syscalls/syscall_32.tbl
    $kerneldir/build/
    +           cp -a --parents arch/x86/entry/syscalls/syscalltbl.sh
    $kerneldir/build/
    +           cp -a --parents arch/x86/entry/syscalls/syscallhdr.sh
    $kerneldir/build/
    +           cp -a --parents arch/x86/entry/syscalls/syscall_64.tbl
    $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs_32.c $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs_64.c $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs.c $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs_common.c
    $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs.h $kerneldir/build/
    +           cp -a --parents tools/include/tools/le_byteshift.h
    $kerneldir/build/
    +           cp -a --parents tools/include/tools/be_byteshift.h
    $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/purgatory.c
    $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/sha256.h
    $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/sha256.c
    $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/stack.S $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/string.c
    $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/setup-x86_64.S
    $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/entry64.S
    $kerneldir/build/
    +           cp -a --parents arch/x86/boot/string.h $kerneldir/build/
    +           cp -a --parents arch/x86/boot/string.c $kerneldir/build/
    +           cp -a --parents arch/x86/boot/ctype.h $kerneldir/build/
    +       fi
    +    )
    +
    +    # Make sure the Makefile and version.h have a matching
    timestamp so that
    +    # external modules can be built
    +    touch -r $kerneldir/build/Makefile
    $kerneldir/build/include/generated/uapi/linux/version.h
    +
    +    # Copy .config to include/config/auto.conf so "make prepare" is
    unnecessary.
    +    cp $kerneldir/build/.config
    $kerneldir/build/include/config/auto.conf
    +
    +    chown -R root:root ${D}
      }
    +
      # Ensure we don't race against "make scripts" during cpio
      do_install[lockfiles] = "${TMPDIR}/kernel-scripts.lock"

    -PACKAGES = "kernel-devsrc"
    -FILES_${PN} = "${KERNEL_SRC_PATH}"
    -RDEPENDS_${PN} = "bc"
    +FILES_${PN} = "${KERNEL_BUILD_ROOT} ${KERNEL_SRC_PATH}"
    +FILES_${PN}-dbg += "${KERNEL_BUILD_ROOT}*/build/scripts/*/.debug/*"
    +RDEPENDS_${PN} = "bc glibc-utils"
    --
    2.5.0

Unfortunately, I can't reproduce that here.

This has been built and tested on all arches by myself and others, but I can try again .. just no promises when I'll get back to it.

Bruce

Quoted 301 lines · Ross

Sorry, but this breaks on ARM:

ERROR: kernel-devsrc-1.0-r0 do_install: Function failed: do_install (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.25822)

ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.25822

Log data follows:

| DEBUG: Executing shell function do_install
| cp: with --parents, the destination must be a directory
| Try 'cp --help' for more information.
| WARNING: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/run.do_install.25822:1 exit 1 from 'rm -rf $kerneldir/build/include'
| ERROR: Function failed: do_install (log file is located at /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/log.do_install.25822) 

Ross

On 28 February 2018 at 02:06, Bruce Ashfield <[email protected] <mailto:[email protected]>> wrote:

    The existing kernel-devsrc package starts with a full copy of the kernel source and then starts to strip out elements that are not required.

    This results in extra time (I/O) and extra space being taken up in the final package. The main purpose of the kernel-devsrc package has been to build modules against the running kernel, not to include a full copy of the source code for re-building the kernel. The end result was a 600M kernel-devsrc package.

    This restructuring of the package uses an approach similar to other distros, where the kernel-devsrc package is for building against the running kernel and uses a curated set of copied infrastructure, versus a mass copy of the entire kernel.

    The differences in this approach versus other is largely due to the architecture support and the split build/source directory of the kernel.

    The result is a kernel-devsrc package of about 10M, which is capable of running "make scripts" and compiling kernel modules against the running kernel.

    Along with the changes to the copying of the infrascture, we also have the following changes:

      - a better/more explicit listing of dependencies for on-target builds of "make scripts" or "make modules_prepare"

      - The kernel source is installed into /lib/modules/<version>/build and a symlink created from /usr/src/kernel to the new location. This aligns with the standard location for module support code

      - There is also a symlink from /lib/modules/<version>/source -> build to reserve a spot for a new package that is simply the kernel source. That package is not part of this update.

    [YOCTO #12535]

Signed-off-by Bruce Ashfield <[email protected]

    <mailto:[email protected]>>
    ---
      meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb> | 178 ++++++++++++++++++++++-------

      1 file changed, 138 insertions(+), 40 deletions(-)

    diff --git a/meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb>
    b/meta/recipes-kernel/linux/kernel-devsrc.bb <http://kernel-devsrc.bb>
    index c1b5b7786d7c..f55b115e0936 100644
    --- a/meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb>
    +++ b/meta/recipes-kernel/linux/kernel-devsrc.bb
    <http://kernel-devsrc.bb>
    @@ -30,50 +30,148 @@ do_populate_sysroot[noexec] = "1"
      S = "${STAGING_KERNEL_DIR}"
      B = "${STAGING_KERNEL_BUILDDIR}"

    -KERNEL_VERSION = "${@get_kernelversion_headers('${S}')}" -
      PACKAGE_ARCH = "${MACHINE_ARCH}"

    +KERNEL_BUILD_ROOT="/lib/modules/"
    +
      do_install() {
    -        kerneldir=${D}${KERNEL_SRC_PATH}
    -        install -d $kerneldir
    -
    -        #
    -        # Copy the staging dir source (and module build support) into the devsrc structure.
    -        # We can keep this copy simple and take everything, since a we'll clean up any build
    -        # artifacts afterwards, and the extra i/o is not significant -        #
    -        cd ${B}
    -        find . -type d -name '.git*' -prune -o -path '.debug' -prune -o -type f -print0 | cpio --null -pdlu $kerneldir
    -        cd ${S}
    -       find . -type d -name '.git*' -prune -o -type d -name '.kernel-meta' -prune -o -type f -print0 | cpio --null -pdlu $kerneldir -
    -        # Explicitly set KBUILD_OUTPUT to ensure that the image directory is cleaned and not
    -        # The main build artifacts. We clean the directory to avoid QA errors on mismatched
    -        # architecture (since scripts and helpers are native format). -        KBUILD_OUTPUT="$kerneldir"
    -        oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
    -        # make clean generates an absolute path symlink called "source" -        # in $kerneldir points to $kerneldir, which doesn't make any -        # sense, so remove it.
    -        if [ -L $kerneldir/source ]; then
    -            bbnote "Removing $kerneldir/source symlink"
    -            rm -f $kerneldir/source
    -        fi
    -
    -        # As of Linux kernel version 3.0.1, the clean target removes -        # arch/powerpc/lib/crtsavres.o which is present in -        # KBUILD_LDFLAGS_MODULE, making it required to build external modules.
    -        if [ ${ARCH} = "powerpc" ]; then
    -                mkdir -p $kerneldir/arch/powerpc/lib/
    -                cp ${B}/arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o
    -        fi
    -
    -        chown -R root:root ${D}
    +    kerneldir=${D}${KERNEL_BUILD_ROOT}${KERNEL_VERSION}
    +    install -d $kerneldir
    +
    +    # create the directory structure
    +    rm -f $kerneldir/build
    +    rm -f $kerneldir/source
    +    mkdir -p $kerneldir/build
    +
    +    # for compatibility with some older variants of this package, we +    # create  a /usr/src/kernel symlink to /lib/modules/<version>/source
    +    mkdir -p ${D}/usr/src
    +    (
    +       cd ${D}/usr/src
    +       ln -s ${KERNEL_BUILD_ROOT}${KERNEL_VERSION}/source kernel +    )
    +
    +    # for on target purposes, we unify build and source
    +    (
    +       cd $kerneldir
    +       ln -s build source
    +    )
    +
    +    # first copy everything
    +    (
    +       cd ${S}
    +       cp --parents $(find  -type f -name "Makefile*" -o -name "Kconfig*") $kerneldir/build
    +    )
    +
    +    # then drop all but the needed Makefiles/Kconfig files +    rm -rf $kerneldir/build/Documentation
    +    rm -rf $kerneldir/build/scripts
    +    rm -rf $kerneldir/build/include
    +
    +    # now copy in parts from the build that we'll need later +    (
    +       cd ${B}
    +
    +       cp Module.symvers $kerneldir/build
    +       cp System.map* $kerneldir/build
    +       if [ -s Module.markers ]; then
    +           cp Module.markers $kerneldir/build
    +       fi
    +
    +       cp .config $kerneldir/build
    +
    +       # TODO: this scripts copy will likely blow up QA, so for now, we +       #       require a more complex 'make scripts' to restore these, versus
    +       #       copying them here. Left as a reference to indicate that we
    +       #       know the scripts must be dealt with.
    +       # cp -a scripts $kerneldir/build
    +
    +       if [ -f tools/objtool/objtool ]; then
    +           cp -a tools/objtool/objtool $kerneldir/build/tools/objtool/ +       fi
    +       if [ -d arch/${ARCH}/scripts ]; then
    +           cp -a arch/${ARCH}/scripts $kerneldir/build/arch/${ARCH} +       fi
    +       if [ -f arch/${ARCH}/*lds ]; then
    +           cp -a arch/${ARCH}/*lds $kerneldir/build/arch/${ARCH} +       fi
    +
    +       rm -f $kerneldir/build/scripts/*.o
    +       rm -f $kerneldir/build/scripts/*/*.o
    +
    +       if [ "${ARCH}" = "powerpc" ]; then
    +           cp -a --parents arch/powerpc/lib/crtsavres.[So] $kerneldir/build/
    +       fi
    +
    +       cp -a include $kerneldir/build/include
    +    )
    +
    +    # now grab the chunks from the source tree that we need +    (
    +       cd ${S}
    +
    +       cp -a scripts $kerneldir/build
    +       if [ "${ARCH}" = "arm64" ]; then
    +           # arch/arm64/include/asm/xen references arch/arm +           cp -a --parents arch/arm/include/asm/xen $kerneldir/build/ +           # arch/arm64/include/asm/opcodes.h references arch/arm +           cp -a --parents arch/arm/include/asm/opcodes.h $kerneldir/build/
    +       fi
    +
    +       # include the machine specific headers for ARM variants, if available.
    +       if [ "${ARCH}" = "arm" ]; then
    +           cp -a --parents arch/${ARCH}/mach-*/include $kerneldir/build/
    +
    +           # include a few files for 'make prepare'
    +           cp -a --parents arch/arm/tools/gen-mach-types $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
    +           cp -a --parents arch/arm/tools/mach-types $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
    +       fi
    +
    +       if [ -d arch/${ARCH}/include ]; then
    +           cp -a --parents arch/${ARCH}/include $kerneldir/build/ +       fi
    +
    +       cp -a include $kerneldir/build
    +
    +       if [ "${ARCH}" = "x86" ]; then
    +           # files for 'make prepare' to succeed with kernel-devel +           cp -a --parents arch/x86/entry/syscalls/syscall_32.tbl $kerneldir/build/
    +           cp -a --parents arch/x86/entry/syscalls/syscalltbl.sh $kerneldir/build/
    +           cp -a --parents arch/x86/entry/syscalls/syscallhdr.sh $kerneldir/build/
    +           cp -a --parents arch/x86/entry/syscalls/syscall_64.tbl $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs_32.c $kerneldir/build/ +           cp -a --parents arch/x86/tools/relocs_64.c $kerneldir/build/ +           cp -a --parents arch/x86/tools/relocs.c $kerneldir/build/ +           cp -a --parents arch/x86/tools/relocs_common.c $kerneldir/build/
    +           cp -a --parents arch/x86/tools/relocs.h $kerneldir/build/ +           cp -a --parents tools/include/tools/le_byteshift.h $kerneldir/build/
    +           cp -a --parents tools/include/tools/be_byteshift.h $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/purgatory.c $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/sha256.h $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/sha256.c $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/stack.S $kerneldir/build/ +           cp -a --parents arch/x86/purgatory/string.c $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/setup-x86_64.S $kerneldir/build/
    +           cp -a --parents arch/x86/purgatory/entry64.S $kerneldir/build/
    +           cp -a --parents arch/x86/boot/string.h $kerneldir/build/ +           cp -a --parents arch/x86/boot/string.c $kerneldir/build/ +           cp -a --parents arch/x86/boot/ctype.h $kerneldir/build/ +       fi
    +    )
    +
    +    # Make sure the Makefile and version.h have a matching timestamp so that
    +    # external modules can be built
    +    touch -r $kerneldir/build/Makefile $kerneldir/build/include/generated/uapi/linux/version.h
    +
    +    # Copy .config to include/config/auto.conf so "make prepare" is unnecessary.
    +    cp $kerneldir/build/.config $kerneldir/build/include/config/auto.conf
    +
    +    chown -R root:root ${D}
      }
    +
      # Ensure we don't race against "make scripts" during cpio do_install[lockfiles] = "${TMPDIR}/kernel-scripts.lock"

    -PACKAGES = "kernel-devsrc"
    -FILES_${PN} = "${KERNEL_SRC_PATH}"
    -RDEPENDS_${PN} = "bc"
    +FILES_${PN} = "${KERNEL_BUILD_ROOT} ${KERNEL_SRC_PATH}" +FILES_${PN}-dbg += "${KERNEL_BUILD_ROOT}*/build/scripts/*/.debug/*" +RDEPENDS_${PN} = "bc glibc-utils"
    --
    2.5.0