kernel: restore scripts in the sysroot
cover kernel: restore scripts in the sysroot
Richard/Saul,
3rd time is a charm, and I finally have the right incantation to build the scripts in:
tmp/sysroots/<machine>/usr/src/kernel/scripts/mod
This works in my environment, for my test cases, but I don't have Khem's exact external module build case handy.
Khem: if you can try this patch and ack it if it works, that would be fantastic.
Cheers,
Bruce
p.s. I'm out of the office for the next week, but will track this via my gmail account.
The following changes since commit 72ca7801f3b733f7fc63326c1d8c98a3c32a519f:
linux-yocto/3.10: integrate lxc kernel feature and 32 bit x86 config (2013-10-08 16:44:54 -0400)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib zedd/kernel-oe
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=zedd/kernel-oe
Bruce Ashfield (1):
kernel: restore scripts in the sysroot
meta/classes/kernel.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)
Signature · 1 line
1.8.1.2
PATCH 1/1 kernel: restore scripts in the sysroot
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
--- meta/classes/kernel.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/classes/kernel.bbclass +11 -0
@@ -292,6 +292,17 @@ kernel_do_install() { } do_install[prefuncs] += "package_get_auto_pr" + +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; then + ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} + sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }
Signature · 1 line
1.8.1.2
Quoted 26 lines · Bruce Ashfield
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/classes/kernel.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)meta/classes/kernel.bbclass +4
-0@@ -292,6 +292,17 @@ kernel_do_install() { } do_install[prefuncs] += "package_get_auto_pr" + +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; then
meta/classes/kernel.bbclass kernel_do_install() { · line 308
Quoted 10 lines
meta/classes/kernel.bbclass +7
-0+ ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} + sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }
This sstate_postinst fails for me when the compiler isn't already present in the sysroot.
Also, I notice that the environment and parameters passed to oe_runmake are not the same as those used by kernel_do_compile etc.
Patches to follow.
Mike.
PATCH 1/2 kernel.bbclass: Ensure compiler is available for do_populate_sysroot_setscene
kernelscripts_sstate_postinst uses the compiler (and therefore binutils) to generate the kernel scripts so it must declare a dependency on them. The ones in DEPENDS don't apply to do_populate_sysroot_setscene.
--- meta/classes/kernel.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/kernel.bbclass +2 -0
@@ -312,6 +312,8 @@ kernelscripts_sstate_postinst () { fi } +do_populate_sysroot_setscene[depends] += "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot virtual/${TARGET_PREFIX}binutils:do_populate_sysroot" + python sysroot_stage_all () { oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}")) }
Signature · 1 line
1.7.10.4
PATCH 2/2 kernel.bbclass: Call make with correct variables in kernelscripts_sstate_postinst
Ensure that the environment variable and command line arguments that are used when calling make in kernel_do_compile are used in kernelscripts_sstate_postinst.
--- meta/classes/kernel.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +2 -1
@@ -304,10 +304,11 @@ do_install[prefuncs] += "package_get_auto_pr" SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" kernelscripts_sstate_postinst () { + unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; then ( cd ${STAGING_KERNEL_DIR} - oe_runmake scripts + oe_runmake scripts CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} ) fi }
Signature · 1 line
1.7.10.4
Re: [PATCH 2/2] kernel.bbclass: Call make with correct variables in kernelscripts_sstate_postinst
Quoted 3 lines · Mike Crowe
Ensure that the environment variable and command line arguments that are used when calling make in
kernel_do_compileare used inkernelscripts_sstate_postinst.
What issue was this causing for you ? The scripts are native, and hence should just be using the build machines compiler, linker and don't need the kernel args.
So I'm not convinced that we need this patch, without seeing a tangible issue that was triggered.
Bruce
Quoted 30 lines
Signed-off-by Mike Crowe <[email protected]>--- meta/classes/kernel.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)meta/classes/kernel.bbclass +2
-2@@ -304,10 +304,11 @@ do_install[prefuncs] += "package_get_auto_pr" SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" kernelscripts_sstate_postinst () { + unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; then ( cd ${STAGING_KERNEL_DIR} - oe_runmake scripts + oe_runmake scripts CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} ) fi } --1.7.10.4
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Re: [PATCH 2/2] kernel.bbclass: Call make with correct variables in kernelscripts_sstate_postinst
Quoted 11 lines · Bruce Ashfield
Quoted 3 lines · Mike Crowe
Ensure that the environment variable and command line arguments that are used when calling make in
kernel_do_compileare used inkernelscripts_sstate_postinst.What issue was this causing for you ? The scripts are native, and hence should just be using the build machines compiler, linker and don't need the kernel args.
So I'm not convinced that we need this patch, without seeing a tangible issue that was triggered.
I think for some arches at least, something targetish is being cleaned and then needing to be rebuilt.
I don't want to have to add dependencies for the toolchain so we need to look into this further.
Mike: which arch was this on and do you have the compile error?
Cheers,
Richard
Quoted 41 lines · Mike Crowe
Quoted 26 lines · Bruce Ashfield
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/classes/kernel.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)meta/classes/kernel.bbclass +4
-0@@ -292,6 +292,17 @@ kernel_do_install() { } do_install[prefuncs] += "package_get_auto_pr" + +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; thenmeta/classes/kernel.bbclass kernel_do_install() { · line 308
Quoted 10 lines
meta/classes/kernel.bbclass +7
-0+ ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} + sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }This
sstate_postinstfails for me when the compiler isn't already present in the sysroot.
Quoted 2 lines
Also, I notice that the environment and parameters passed to
oe_runmakeare not the same as those used bykernel_do_compileetc.
Also note that module.bbclass already does "make scripts" before do_compile() so out-of-tree modules should already have access to all the files that they need.
If we're going to make a policy decision that the kernel is responsible for making scripts then I guess that's fine (modulo the implementation problems that Mike describes) but in that case the code in module{-base}.bbclass is redundant and ought to be removed.
p.
Re: [PATCH 1/2] kernel.bbclass: Ensure compiler is available for do_populate_sysroot_setscene
In reply to Mike CroweQuoted 14 lines · Mike Crowe
kernelscripts_sstate_postinstuses the compiler (and therefore binutils) to generate the kernel scripts so it must declare a dependency on them. The ones in DEPENDS don't apply todo_populate_sysroot_setscene.Signed-off-by Mike Crowe <[email protected]>--- meta/classes/kernel.bbclass | 2 ++ 1 file changed, 2 insertions(+)meta/classes/kernel.bbclass +0
-0
meta/classes/kernel.bbclass kernelscripts_sstate_postinst () { · line 315
Quoted 4 lines
meta/classes/kernel.bbclass +1
-0@@ -312,6 +312,8 @@ kernelscripts_sstate_postinst () { fi } +do_populate_sysroot_setscene[depends] += "virtual/${TARGET_PREFIX}gcc:do_populate_sysroot virtual/${TARGET_PREFIX}binutils:do_populate_sysroot"
This makes sense, we have a bug about this that I was never able to reproduce:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5390
Bruce
Quoted 11 lines
meta/classes/kernel.bbclass +1
-1+ python sysroot_stage_all () { oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}")) } --1.7.10.4
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Re: [PATCH 2/2] kernel.bbclass: Call make with correct variables in kernelscripts_sstate_postinst
In reply to Bruce AshfieldQuoted 11 lines · Bruce Ashfield
Quoted 3 lines · Mike Crowe
Ensure that the environment variable and command line arguments that are used when calling make in
kernel_do_compileare used inkernelscripts_sstate_postinst.What issue was this causing for you ? The scripts are native, and hence should just be using the build machines compiler, linker and don't need the kernel args.
So I'm not convinced that we need this patch, without seeing a tangible issue that was triggered.
Well, for us v2.6.39 appears to be using the target compiler for something. At least, I get a lot of:
DEBUG: Executing shell function kernelscripts_sstate_postinst
NOTE: make scripts
/bin/sh: mipsel-oe-linux-gcc: command not found /bin/sh: mipsel-oe-linux-gcc: command not found /bin/sh: mipsel-oe-linux-gcc: command not found /bin/sh: mipsel-oe-linux-gcc: command not found
make: mipsel-oe-linux-gcc: Command not found /bin/sh: mipsel-oe-linux-gcc: command not found /bin/sh: mipsel-oe-linux-gcc: command not found
which led me to the previous patch to ensure that the compiler is available.
Thanks.
Mike.
Quoted 42 lines · Phil Blundell
Quoted 41 lines · Mike Crowe
Quoted 26 lines · Bruce Ashfield
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/classes/kernel.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)meta/classes/kernel.bbclass +4
-0@@ -292,6 +292,17 @@ kernel_do_install() { } do_install[prefuncs] += "package_get_auto_pr" + +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; thenmeta/classes/kernel.bbclass kernel_do_install() { · line 308
Quoted 10 lines
meta/classes/kernel.bbclass +7
-0+ ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} + sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }This
sstate_postinstfails for me when the compiler isn't already present in the sysroot.
Quoted 6 lines
Quoted 2 lines
Also, I notice that the environment and parameters passed to
oe_runmakeare not the same as those used bykernel_do_compileetc.Also note that module.bbclass already does "make scripts" before
do_compile()so out-of-tree modules should already have access to all the files that they need.
Ah, but they didn't. Otherwise we wouldn't have mucked about in the routines. Khem was building out of the sysroot and the support scripts weren't present, something which we were able to consistently reproduce.
Perhaps the whole problem was just a misordering of the tasks. I'll take another look.
But yes, we need one way or the other .. not both. I'd prefer to not fiddle with sstate, so I'll go back and see why the module.bbclass code wasn't working for the reported error.
Bruce
Quoted 13 lines
If we're going to make a policy decision that the kernel is responsible for making scripts then I guess that's fine (modulo the implementation problems that Mike describes) but in that case the code in module{-base}.bbclass is redundant and ought to be removed.
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Re: [PATCH 2/2] kernel.bbclass: Call make with correct variables in kernelscripts_sstate_postinst
In reply to Richard PurdieQuoted 10 lines · Richard Purdie
Quoted 9 lines · Bruce Ashfield
Quoted 3 lines · Mike Crowe
Ensure that the environment variable and command line arguments that are used when calling make in
kernel_do_compileare used inkernelscripts_sstate_postinst.What issue was this causing for you ? The scripts are native, and hence should just be using the build machines compiler, linker and don't need the kernel args.
Quoted 5 lines
Quoted 2 lines
So I'm not convinced that we need this patch, without seeing a tangible issue that was triggered.
I think for some arches at least, something targetish is being cleaned and then needing to be rebuilt.
Right. I suppose that if you are installing a mips->mips compiler, then the target toolchain is required. That really wasn't the intention of this change, so it likely isn't properly restricted to the native build case (Where native is a x86 development machine).
But with Phil pointing out the module.bbclass 'make scripts' code again, I'd like to figure out why it wasn't working for the original bug report and try to drop all of this mess.
Bruce
Quoted 10 lines
I don't want to have to add dependencies for the toolchain so we need to look into this further.
Mike: which arch was this on and do you have the compile error?
Cheers,
Richard
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Re: [PATCH 2/2] kernel.bbclass: Call make with correct variables in kernelscripts_sstate_postinst
In reply to Mike CroweQuoted 10 lines · Mike Crowe
Quoted 6 lines · Bruce Ashfield
What issue was this causing for you ? The scripts are native, and hence should just be using the build machines compiler, linker and don't need the kernel args.
So I'm not convinced that we need this patch, without seeing a tangible issue that was triggered.
Well, for us v2.6.39 appears to be using the target compiler for something. At least, I get a lot of:
I did "make V=1 scripts" in a kernel devshell and it seems to be using the cross-compiler to build scripts/mod/empty.o. It's not entirely obvious that this file actually needs to be in the sysroot, but it seems that patching the kernel to prevent it getting built would be a bit of a maintenance headache.
p.
Adding Khem.
Khem .. see below.
Quoted 54 lines · Bruce Ashfield
Quoted 42 lines · Phil Blundell
Quoted 41 lines · Mike Crowe
Quoted 26 lines · Bruce Ashfield
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
Signed-off-by Bruce Ashfield <[email protected]>--- meta/classes/kernel.bbclass | 11 +++++++++++ 1 file changed, 11 insertions(+)meta/classes/kernel.bbclass +4
-0@@ -292,6 +292,17 @@ kernel_do_install() { } do_install[prefuncs] += "package_get_auto_pr" + +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; thenmeta/classes/kernel.bbclass kernel_do_install() { · line 308
Quoted 10 lines
meta/classes/kernel.bbclass +7
-0+ ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} + sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }This
sstate_postinstfails for me when the compiler isn't already present in the sysroot.Quoted 6 lines
Quoted 2 lines
Also, I notice that the environment and parameters passed to
oe_runmakeare not the same as those used bykernel_do_compileetc.Also note that module.bbclass already does "make scripts" before
do_compile()so out-of-tree modules should already have access to all the files that they need.Ah, but they didn't. Otherwise we wouldn't have mucked about in the routines. Khem was building out of the sysroot and the support scripts weren't present, something which we were able to consistently reproduce.
Quoted 6 lines
Perhaps the whole problem was just a misordering of the tasks. I'll take another look.
But yes, we need one way or the other .. not both. I'd prefer to not fiddle with sstate, so I'll go back and see why the module.bbclass code wasn't working for the reported error.
Khem: I wasn't working from a bugzilla when making these changes, so I can't find your original reproducer for the missing recordmcount script.
Do you recall what it was ? I'm revisiting this and would like to understand why the make scripts in module-base.bbclass wasn't properly restoring the support scripts for your build.
I'm leaning towards reverting the whole mess, but without the reproducing case, I can't be sure that I'm not breaking you again.
Bruce
Quoted 23 lines
Bruce
Quoted 6 lines
If we're going to make a policy decision that the kernel is responsible for making scripts then I guess that's fine (modulo the implementation problems that Mike describes) but in that case the code in module{-base}.bbclass is redundant and ought to be removed.
Quoted 7 lines
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core--
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 70 lines · Bruce Ashfield
Adding Khem.
Khem .. see below.
Quoted 54 lines · Bruce Ashfield
Quoted 42 lines · Phil Blundell
Quoted 41 lines · Mike Crowe
Quoted 33 lines · Bruce Ashfield
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
Signed-off-by Bruce Ashfield <[email protected]>---
meta/classes/kernel.bbclass | 11 +++++++++++1 file changed, 11 insertions(+)meta/classes/kernel.bbclass +0
-0@@ -292,6 +292,17 @@ kernel_do_install() { }do_install[prefuncs] += "
package_get_auto_pr"meta/classes/kernel.bbclass +11
-0+ +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; then + ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} +Quoted 3 lines
sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }This
sstate_postinstfails for me when the compiler isn't already present in the sysroot.Quoted 6 lines
Quoted 2 lines
Also, I notice that the environment and parameters passed to
oe_runmakeare not the same as those used bykernel_do_compileetc.Also note that module.bbclass already does "make scripts" before
do_compile()so out-of-tree modules should already have access to all the files that they need.Ah, but they didn't. Otherwise we wouldn't have mucked about in the routines. Khem was building out of the sysroot and the support scripts weren't present, something which we were able to consistently reproduce.
Quoted 6 lines
Perhaps the whole problem was just a misordering of the tasks. I'll take another look.
But yes, we need one way or the other .. not both. I'd prefer to not fiddle with sstate, so I'll go back and see why the module.bbclass code wasn't working for the reported error.
Khem: I wasn't working from a bugzilla when making these changes, so I can't find your original reproducer for the missing recordmcount script.
Quoted 6 lines
Do you recall what it was ? I'm revisiting this and would like to understand why the make scripts in module-base.bbclass wasn't properly restoring the support scripts for your build.
I'm leaning towards reverting the whole mess, but without the reproducing case, I can't be sure that I'm not breaking you again.
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that
# inherit this recipe and override do_compile() should be sure to call
# do_make_scripts() or ensure the scripts are built independently.
#
do_make_scripts() {
unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
-C ${STAGING_KERNEL_DIR} scripts
}so it expects that, do_make_scripts is explicitly called by external module recipes
and my recipes did override module_do_compile task but not do_compile like below
module_do_compile() {
oe_runmake
}
so, is comment wrong there should it say module_do_compile instead ?
will it work with sstate always ?
it will be OK to revert it if thats the case.
Quoted 32 lines
Bruce
Quoted 23 lines
Bruce
Quoted 6 lines
If we're going to make a policy decision that the kernel is responsible for making scripts then I guess that's fine (modulo the implementation problems that Mike describes) but in that case the code in module{-base}.bbclass is redundant and ought to be removed.
Quoted 7 lines
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core--
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 25 lines · Khem Raj
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }so it expects that,
do_make_scriptsis explicitly called by external module recipesand my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}
Quoted 5 lines
so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
it will be OK to revert it if thats the case.
Did you inherit module-base or module? I think the wording applies to module and not module-base. I think the function used to be in module and was moved along with the comment which is now incorrect.
Cheers,
Richard
Quoted 77 lines · Khem Raj
Quoted 74 lines · Bruce Ashfield
Adding Khem.
Khem .. see below.
Quoted 54 lines · Bruce Ashfield
Quoted 42 lines · Phil Blundell
Quoted 41 lines · Mike Crowe
Quoted 33 lines · Bruce Ashfield
When building against the sysroot, out of tree modules can require modpost and other utilities normally found in the kernel's scripts directory. For the kernel source in the staging dir, these scripts have been removed to avoid mixing archiectures when packaging kernel-dev (among other things).
Rather than further complicate the kernel's install rule, or its packaging, we can restore the scripts by building them in the kernel staging directory after the sstate is installed, making them available to packages that need them.
Signed-off-by Bruce Ashfield <[email protected]>---
meta/classes/kernel.bbclass | 11 +++++++++++1 file changed, 11 insertions(+)meta/classes/kernel.bbclass +0
-0@@ -292,6 +292,17 @@ kernel_do_install() { }do_install[prefuncs] += "
package_get_auto_pr"meta/classes/kernel.bbclass +11
-0+ +SSTATEPOSTINSTFUNCS += "kernelscripts_sstate_postinst" +kernelscripts_sstate_postinst () { + if [ "${BB_CURRENTTASK}" = "populate_sysroot" -o "${BB_CURRENTTASK}" = "populate_sysroot_setscene" ]; then + ( + cd ${STAGING_KERNEL_DIR} + oe_runmake scripts + ) + fi +} +Quoted 3 lines
sysroot_stage_all_append() { sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH} }This
sstate_postinstfails for me when the compiler isn't already present in the sysroot.Quoted 6 lines
Quoted 2 lines
Also, I notice that the environment and parameters passed to
oe_runmakeare not the same as those used bykernel_do_compileetc.Also note that module.bbclass already does "make scripts" before
do_compile()so out-of-tree modules should already have access to all the files that they need.Ah, but they didn't. Otherwise we wouldn't have mucked about in the routines. Khem was building out of the sysroot and the support scripts weren't present, something which we were able to consistently reproduce.
Quoted 6 lines
Perhaps the whole problem was just a misordering of the tasks. I'll take another look.
But yes, we need one way or the other .. not both. I'd prefer to not fiddle with sstate, so I'll go back and see why the module.bbclass code wasn't working for the reported error.
Khem: I wasn't working from a bugzilla when making these changes, so I can't find your original reproducer for the missing recordmcount script.
Do you recall what it was ? I'm revisiting this and would like to understand why the make scripts in module-base.bbclass wasn't properly restoring the support scripts for your build.
Quoted 7 lines
Quoted 2 lines
I'm leaning towards reverting the whole mess, but without the reproducing case, I can't be sure that I'm not breaking you again.
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
Right.
Quoted 13 lines
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }
Quoted 2 lines
so it expects that,
do_make_scriptsis explicitly called by external module recipes
Exactly. And I had windriver bug with the same symptoms as yours. It was a package
that built its own modules, and hence never called this either.
Quoted 11 lines
and my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
it will be OK to revert it if thats the case.
I'll come up with something that works in all cases. The sstate fix is better from the point
of view that it is transparent to the recipes, and they don't need to do anything special
for the support.
So my proposal is this:
- fix the new bug at hand by making the sstate change depend on the
toolchain (I agree
that patching the scripts to not reference the target toolchain
isn't a good idea since not
all kernels get the rix).
- remove the modules-base.bbclass scripts recreation, so we only have
one scripts
restore in the tree.
Alternatively, recipes need to be fixed to call the modules-base.bbclass routine to restore
scripts, and I think it is obvious that won't work in all cases.
I'll wait for comments/concensus before sending a new patch (which needs to be tested
on all the cases), but in the meantime, the patches to the list to fix the sstate solution
are good to be applied.
Cheers,
Bruce
Quoted 34 lines
Quoted 33 lines
Bruce
Quoted 20 lines
Bruce
Quoted 6 lines
If we're going to make a policy decision that the kernel is responsible for making scripts then I guess that's fine (modulo the implementation problems that Mike describes) but in that case the code in module{-base}.bbclass is redundant and ought to be removed.
Quoted 7 lines
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-coreQuoted 3 lines
--
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"--
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 29 lines · Richard Purdie
Quoted 27 lines · Khem Raj
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }so it expects that,
do_make_scriptsis explicitly called by external module recipesand my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}so, is comment wrong there should it say
module_do_compileinstead ?
Quoted 8 lines
Quoted 3 lines
will it work with sstate always ?
it will be OK to revert it if thats the case.
Did you inherit module-base or module? I think the wording applies to module and not module-base. I think the function used to be in module and was moved along with the comment which is now incorrect.
inherit module
Quoted 5 lines
Cheers,
Richard
Quoted 19 lines · Khem Raj
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }so it expects that,
do_make_scriptsis explicitly called by external module recipes
Quoted 7 lines
and my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}so, is comment wrong there should it say
module_do_compileinstead ?
The comment is slightly wrong, yes. For correct results you need to either:
1. inherit module (not module-base), which does:
addtask make_scripts after do_patch before do_compile
and will ensure that the scripts are built before your module is compiled without you needing to do anything else. Or...
2. inherit module-base and arrange to call do_make_scripts() explicitly from your own recipe.
p.
Quoted 23 lines · Bruce Ashfield
Exactly. And I had windriver bug with the same symptoms as yours. It was a package
that built its own modules, and hence never called this either.Quoted 11 lines
and my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
it will be OK to revert it if thats the case.
I'll come up with something that works in all cases. The sstate fix is better from the point
of view that it is transparent to the recipes, and they don't need to do anything special
for the support.
Quoted 7 lines
So my proposal is this:
- fix the new bug at hand by making the sstate change depend on the
toolchain (I agree
that patching the scripts to not reference the target toolchain
isn't a good idea since not
all kernels get the rix).
Can we please not do this? Adding in toolchain dependencies into the sstate code whilst possible, is going to massively complicate the dependency chains and is a last possible resort. I bought that argument in the useradd case since there are horrible issues there. We don't have that issue here.
In kernel terms, its safer/easier to hack the kernel makefiles than expecting sstate to work well with dependencies like that embedded in it.
Quoted 7 lines
- remove the modules-base.bbclass scripts recreation, so we only have
one scripts
restore in the tree.Alternatively, recipes need to be fixed to call the modules-base.bbclass routine to restore
scripts, and I think it is obvious that won't work in all cases.
Which cases won't that work in?
As far as I'm concerned, people using module-base are taking a loaded gun and are expected to use it safely (which means calling do_make_scripts somehow). People wanting a safer ride can use module which adds the appropriate task for them.
The comments in the bbclass files do need fixing but that is trivial to sort out.
Quoted 4 lines
I'll wait for comments/concensus before sending a new patch (which needs to be tested
on all the cases), but in the meantime, the patches to the list to fix the sstate solution are good to be applied.
I will respectfully disagree ;-).
Cheers,
Richard
Quoted 3 lines · Bruce Ashfield
Alternatively, recipes need to be fixed to call the modules-base.bbclass routine to restore
scripts, and I think it is obvious that won't work in all cases.
Admittedly I'm no expert, but this is not obvious to me. Why can't those recipes just be fixed to do the right thing?
p.
Quoted 33 lines · Khem Raj
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Khem Raj
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }so it expects that,
do_make_scriptsis explicitly called by external module recipesand my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
it will be OK to revert it if thats the case.
Did you inherit module-base or module? I think the wording applies to module and not module-base. I think the function used to be in module and was moved along with the comment which is now incorrect.
inherit module
So in that case there is:
addtask make_scripts after do_patch before do_compile do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock" do_make_scripts[deptask] = "do_populate_sysroot"
which forces the make_scripts task to run before compile. I don't see how that could therefore fail? :/
Do you have a copy of the exact log? I'm wondering if this is another variant of the bitbake dependency bug I just tracked down
(http://git.yoctoproject.org/cgit.cgi/poky-contrib/patch/?id=7b49d336c4672f3dfa78e1c3f1f5c7384264a118)
Cheers,
Richard
Quoted 32 lines · Richard Purdie
Quoted 23 lines · Bruce Ashfield
Exactly. And I had windriver bug with the same symptoms as yours. It was a package
that built its own modules, and hence never called this either.Quoted 11 lines
and my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
it will be OK to revert it if thats the case.
I'll come up with something that works in all cases. The sstate fix is better from the point
of view that it is transparent to the recipes, and they don't need to do anything special
for the support.Quoted 7 lines
So my proposal is this:
- fix the new bug at hand by making the sstate change depend on the
toolchain (I agree
that patching the scripts to not reference the target toolchain
isn't a good idea since not
all kernels get the rix).
Quoted 6 lines
Can we please not do this? Adding in toolchain dependencies into the sstate code whilst possible, is going to massively complicate the dependency chains and is a last possible resort. I bought that argument in the useradd case since there are horrible issues there. We don't have that issue here.
Works for me. I just wonder if there's another way to handle things more gracefully ? i.e. somehow check if the toolchain isn't available and warn, otherwise continue making the scripts ?
Quoted 3 lines
In kernel terms, its safer/easier to hack the kernel makefiles than expecting sstate to work well with dependencies like that embedded in it.
It is pretty simple to make the change. Just making it available everywhere is the trick.
Quoted 9 lines
Quoted 8 lines
- remove the modules-base.bbclass scripts recreation, so we only have
one scripts
restore in the tree.Alternatively, recipes need to be fixed to call the modules-base.bbclass routine to restore
scripts, and I think it is obvious that won't work in all cases.
Quoted 6 lines
Which cases won't that work in?
As far as I'm concerned, people using module-base are taking a loaded gun and are expected to use it safely (which means calling
do_make_scriptssomehow). People wanting a safer ride can use module which adds the appropriate task for them.
I've got people not using any of the above .. yes, I know they are evil too :)
Quoted 10 lines
The comments in the bbclass files do need fixing but that is trivial to sort out.
Quoted 4 lines
I'll wait for comments/concensus before sending a new patch (which needs to be tested
on all the cases), but in the meantime, the patches to the list to fix the sstate solution are good to be applied.I will respectfully disagree ;-).
No patch from me for this then. That's why i waited, I figured it wouldn't be immediate agreement.
Bruce
Quoted 5 lines
Cheers,
Richard
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 7 lines · Phil Blundell
Quoted 3 lines · Bruce Ashfield
Alternatively, recipes need to be fixed to call the modules-base.bbclass routine to restore
scripts, and I think it is obvious that won't work in all cases.Admittedly I'm no expert, but this is not obvious to me. Why can't those recipes just be fixed to do the right thing?
Alas, people with legacy build environments that they've wrapped and ported to bitbake .. they either can't or won't change, so I'm stuck at least trying to detect them.
That wasn't the primary driver for this though, if that case breaks, they'll live and so will i :)
I think we are converging to removing the sstate scripts restore, which is fine with me a this point, as long as Khem isn't broken.
Bruce
Quoted 4 lines
p.
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Khem Raj
Quoted 37 lines · Richard Purdie
Quoted 25 lines · Khem Raj
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }so it expects that,
do_make_scriptsis explicitly called by external module recipesand my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}Quoted 5 lines
so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
it will be OK to revert it if thats the case.
Did you inherit module-base or module? I think the wording applies to module and not module-base. I think the function used to be in module and was moved along with the comment which is now incorrect.
Quoted 8 lines
Quoted 1 line
inherit module
So in that case there is:
addtask
make_scriptsafterdo_patchbeforedo_compiledo_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock" do_make_scripts[deptask] = “do_populate_sysroot”
yes I see.
Quoted 4 lines
which forces the
make_scriptstask to run before compile. I don't see how that could therefore fail? :/Do you have a copy of the exact log?
Not anymore, it was sometime ago, the logs are unfortunately recycled.
Quoted 4 lines
I'm wondering if this is another
variant of the bitbake dependency bug I just tracked down
(http://git.yoctoproject.org/cgit.cgi/poky-contrib/patch/?id=7b49d336c4672f3dfa78e1c3f1f5c7384264a118)
ah very likely, I think I will try this fix and revert the kernel.bbclass fix locally and see if I still am able to reproduce this issue. In any case I think the kernel.bbclass may be abandoned since I now think it was a wrong fix although it did fix the problem
Quoted 4 lines
Cheers,
Richard
Quoted 63 lines · Khem Raj
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Khem Raj
Quoted 37 lines · Richard Purdie
Quoted 25 lines · Khem Raj
Well reproducer case is that build from sstate but such that an external module needs to be rebuilt if external module also comes from sstate then it all is fine. Its only when everything is coming from sstate except this external module which needs to be rebuilt then you see this problem.
now, I see the code in module-base.class
#
# Ensure the hostprogs are available for module compilation. Modules that# inherit this recipe and override do_compile() should be sure to call # do_make_scripts() or ensure the scripts are built independently. # do_make_scripts() { unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS make CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \ -C ${STAGING_KERNEL_DIR} scripts }so it expects that,
do_make_scriptsis explicitly called by external module recipesand my recipes did override
module_do_compiletask but notdo_compilelike below
module_do_compile(){
oe_runmake
}Quoted 5 lines
so, is comment wrong there should it say
module_do_compileinstead ?will it work with sstate always ?
it will be OK to revert it if thats the case.
Did you inherit module-base or module? I think the wording applies to module and not module-base. I think the function used to be in module and was moved along with the comment which is now incorrect.
Quoted 8 lines
Quoted 1 line
inherit module
So in that case there is:
addtask
make_scriptsafterdo_patchbeforedo_compiledo_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock" do_make_scripts[deptask] = “do_populate_sysroot”yes I see.
Quoted 4 lines
which forces the
make_scriptstask to run before compile. I don't see how that could therefore fail? :/Do you have a copy of the exact log?
Not anymore, it was sometime ago, the logs are unfortunately recycled.
Quoted 9 lines
Quoted 4 lines
I'm wondering if this is another
variant of the bitbake dependency bug I just tracked down
(http://git.yoctoproject.org/cgit.cgi/poky-contrib/patch/?id=7b49d336c4672f3dfa78e1c3f1f5c7384264a118)ah very likely, I think I will try this fix and revert the kernel.bbclass fix locally and see if I still am able to reproduce this issue. In any case I think the kernel.bbclass may be abandoned since I now think it was a wrong fix although it did fix the problem
Agreed.
Since you have control over your recipes, you don't have to worry about some of the more evil ones that build, and inside their own build head reference the kernel-staging tree and attempt to build modules. Just getting the right inheritance should fix it up.
The other case is a different concern, and even then, including module, and overriding the do_compile() should work to get the scripts restored and not just attempt to launch a build.
I'm going to be in Transit tomorrow, so won't be all that responsive, but my vote is now to simply revert the sstate scripts changes once Khem confirms that he is ok for his builds.
Cheers,
Bruce
Quoted 5 lines
Quoted 4 lines
Cheers,
Richard
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"