Add dummy tools to help identify needed dependencies
v3 cover Add dummy tools to help identify needed dependencies
After the introduction of RSS, I still found it hard to get dependencies on some common tools that are typically installed on the build host correct. Using the wrong version of tools like pkg-config, gdbus-codegen and dbus-binding-tool can cause build failures.
To circumvent this, I created dummy versions of the tools that always fail and placed them in the scripts directory. Thus, if the real tool has not been installed in the RSS, the dummy version is used and the build fails. For good measures I even output a message that says what needs to be corrected in the recipe.
Included is also an update to linux-libc-headers as it apparently needs to inherit pkgconfig, which the introduction of the dummy version of pkg-config showed.
PATCHv2: Also inherit pkgconfig in module.bbclass since pkg-config is needed for the make_scripts task.
PATCHv3: Add a missing Signed-off-by footer.
//Peter
The following changes since commit 8f1ed1758748e4c00eaa6b4b3528e79f36d62c19:
bitbake: fetch2: add initial Amazon AWS S3 fetcher (2017-03-03 00:19:37 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib pkj/dummy_tools
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/dummy_tools
Peter Kjellerstedt (5):
module.bbclass: Add inherit of pkgconfig
linux-libc-headers: Add inherit of pkgconfig scripts/dbus-binding-tool: Add a dummy version that always fails scripts/gdbus-codegen: Add a dummy version that always fails scripts/pkg-config: Add a dummy version that always fails
meta/classes/module.bbclass | 2 +- .../linux-libc-headers/linux-libc-headers.inc | 2 +- scripts/dbus-binding-tool | 11 +++++++++++ scripts/gdbus-codegen | 11 +++++++++++ scripts/pkg-config | 16 ++++++++++++++++ 5 files changed, 40 insertions(+), 2 deletions(-) create mode 100755 scripts/dbus-binding-tool create mode 100755 scripts/gdbus-codegen create mode 100755 scripts/pkg-config
Signature · 1 line
2.9.0
PATCH v3 1/5 module.bbclass: Add inherit of pkgconfig
This is needed for the make_scripts task.
--- meta/classes/module.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/module.bbclass +1 -1
@@ -1,4 +1,4 @@ -inherit module-base kernel-module-split +inherit module-base kernel-module-split pkgconfig addtask make_scripts after do_prepare_recipe_sysroot before do_compile do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
Signature · 1 line
2.9.0
PATCH v3 2/5 linux-libc-headers: Add inherit of pkgconfig
In reply to Peter Kjellerstedtpkg-config is used by the kernel build system when creating the configuration tools.
--- meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc +1 -1
@@ -42,7 +42,7 @@ python __anonymous () { d.setVar("HEADER_FETCH_VER", "2.6") } -inherit kernel-arch +inherit kernel-arch pkgconfig KORG_ARCHIVE_COMPRESSION ?= "xz"
Signature · 1 line
2.9.0
PATCH v3 3/5 scripts/dbus-binding-tool: Add a dummy version that always fails
In reply to Peter KjellerstedtThis is intended to catch missing dependencies on the real version.
--- scripts/dbus-binding-tool | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100755 scripts/dbus-binding-tool
scripts/dbus-binding-tool new file +11 -0
@@ -0,0 +1,11 @@ +#!/bin/sh + +# This is a dummy version that always fails. It is intended to catch +# missing dependencies on the real version. + +myname=${0##*/} + +echo "The correct version of $myname is not being used!" >&2 +echo "Make sure the recipe DEPENDS on 'dbus-glib-native'." >&2 + +exit 1
Signature · 1 line
2.9.0
PATCH v3 4/5 scripts/gdbus-codegen: Add a dummy version that always fails
In reply to Peter KjellerstedtThis is intended to catch missing dependencies on the real version.
--- scripts/gdbus-codegen | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100755 scripts/gdbus-codegen
scripts/gdbus-codegen new file +11 -0
@@ -0,0 +1,11 @@ +#!/bin/sh + +# This is a dummy version that always fails. It is intended to catch +# missing dependencies on the real version. + +myname=${0##*/} + +echo "The correct version of $myname is not being used!" >&2 +echo "Make sure the recipe DEPENDS on 'glib-2.0-native'." >&2 + +exit 1
Signature · 1 line
2.9.0
PATCH v3 5/5 scripts/pkg-config: Add a dummy version that always fails
In reply to Peter KjellerstedtThis is intended to catch missing dependencies on the real version.
--- scripts/pkg-config | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 scripts/pkg-config
scripts/pkg-config new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/sh + +# This is a dummy version that always fails. It is intended to catch +# missing dependencies on the real version. + +myname=${0##*/} + +echo "The correct version of $myname is not being used!" >&2 +echo "Make sure the recipe inherits 'pkgconfig'." >&2 + +# Return something that will make the compiler barf. This is for when +# pkg-config is used directly in a Makefile in a construct like: +# CFLAGS += $(shell pkg-config --cflags glib-2.0) +echo "The-correct-version-of-$myname-is-not-being-used" + +exit 1
Signature · 1 line
2.9.0
Hi
2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt <[email protected]>:
Quoted 10 lines
To circumvent this, I created dummy versions of the tools that always fail and placed them in the scripts directory. Thus, if the real tool has not been installed in the RSS, the dummy version is used and the build fails. For good measures I even output a message that says what needs to be corrected in the recipe.
Included is also an update to linux-libc-headers as it apparently needs to inherit pkgconfig, which the introduction of the dummy version of pkg-config showed.
I tried the whole series, i.e. applied on top of openembedded-core, 20c6e87 and bitbake 6fe07ed.
The resulting build failed in linux-libc-headers because it picked up the dummy script instead of the real pkg-config binary which actually is installed in the RSS.
run.do_configure has a PATH setting with first ...meta/scripts and only then the paths to the different RSS directories.
Changing that order in meta/conf/bitbake.conf line 450 fixed this for me, however I'm unsure of posible sideeffects of that reordering.
Max
Quoted 31 lines
-----Original Message-----
From: Max Krummenacher [mailto:[email protected]]
Sent: den 3 mars 2017 17:24
To: Peter Kjellerstedt
Cc: OE-core
Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify needed dependenciesHi
2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt
<[email protected]>:Quoted 10 lines
To circumvent this, I created dummy versions of the tools that always fail and placed them in the scripts directory. Thus, if the real tool has not been installed in the RSS, the dummy version is used and the build fails. For good measures I even output a message that says what needs to be corrected in the recipe.
Included is also an update to linux-libc-headers as it apparently needs to inherit pkgconfig, which the introduction of the dummy version of pkg-config showed.
I tried the whole series, i.e. applied on top of openembedded-core, 20c6e87 and bitbake 6fe07ed.
The resulting build failed in linux-libc-headers because it picked up the dummy script instead of the real pkg-config binary which actually is installed in the RSS.
Quoted 7 lines
run.do_configure has a PATH setting with first ...meta/scripts and only then the paths to the different RSS directories.
Changing that order in
meta/conf/bitbake.confline 450 fixed this for me, however I'm unsure of posible sideeffects of that reordering.Max
Bah, you are correct. That is what I get for not testing exactly what I sent for integration. Shame on me.
Anyway, I see a couple of possible solutions:
* Put the scripts in bitbake/bin instead. This should work as it is only added to $PATH by oe-buildenv-internal (contrary to ${COREBASE}/scripts which is added to $PATH both by oe-buildenv-internal and bitbake.conf). However, I do not like this idea as it feels wrong to put them there.
* Put the scripts in a subdirectory of ${COREBASE}/scripts, e.g., ${COREBASE}/scripts/blacklisted, and add that path to $PATH in bitbake.conf, but at the other end of it so that the RSS paths appear before it. This sounds a lot more appealing to me.
* Implement the whitelist solution Patrick Ohly mentioned instead. Although this is probably the best solution, prioritizing it for 2.3 does not sound likely to happen.
I have implemented solution two above (properly tested this time), and will send that along in a little while.
//Peter
Quoted 71 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Peter Kjellerstedt
Sent: den 3 mars 2017 20:53
To: Max Krummenacher
Cc: OE-core
Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify needed dependenciesQuoted 8 lines
-----Original Message-----
From: Max Krummenacher [mailto:[email protected]]
Sent: den 3 mars 2017 17:24
To: Peter Kjellerstedt
Cc: OE-core
Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify needed dependenciesQuoted 6 lines
Hi
2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt
<[email protected]>:Quoted 1 line
To circumvent this, I created dummy versions of the tools that
always
Quoted 1 line
Quoted 1 line
fail and placed them in the scripts directory. Thus, if the real
tool
Quoted 1 line
Quoted 1 line
has not been installed in the RSS, the dummy version is used and
the
Quoted 1 line
Quoted 1 line
build fails. For good measures I even output a message that says
what
Quoted 14 lines
Quoted 6 lines
needs to be corrected in the recipe.
Included is also an update to linux-libc-headers as it apparently needs to inherit pkgconfig, which the introduction of the dummy version of pkg-config showed.
I tried the whole series, i.e. applied on top of openembedded-core, 20c6e87 and bitbake 6fe07ed.
The resulting build failed in linux-libc-headers because it picked up the dummy script instead of the real pkg-config binary which actually is installed in the RSS.
Quoted 7 lines
run.do_configure has a PATH setting with first ...meta/scripts and only then the paths to the different RSS directories.
Changing that order in
meta/conf/bitbake.confline 450 fixed this for me, however I'm unsure of posible sideeffects of that reordering.Max
Bah, you are correct. That is what I get for not testing exactly what I sent for integration. Shame on me.
Anyway, I see a couple of possible solutions:
* Put the scripts in bitbake/bin instead. This should work as it is only added to $PATH by oe-buildenv-internal (contrary to ${COREBASE}/scripts which is added to $PATH both by oe-buildenv-internal and bitbake.conf). However, I do not like this idea as it feels wrong to put them there.
* Put the scripts in a subdirectory of ${COREBASE}/scripts, e.g., ${COREBASE}/scripts/blacklisted, and add that path to $PATH in bitbake.conf, but at the other end of it so that the RSS paths appear before it. This sounds a lot more appealing to me.
* Implement the whitelist solution Patrick Ohly mentioned instead. Although this is probably the best solution, prioritizing it for 2.3 does not sound likely to happen.
Quoted 4 lines
I have implemented solution two above (properly tested this time), and will send that along in a little while.
//Peter
Richard,
since I see that you have integrated/staged the two patches that add inherits of pkgconfig, but not the patches that add the dummy commands, I assume you have some reservations to these patches. What are your take on the subject of blacklisting vs whitelisting the commands from the build host? I know that having these dummy commands in place helped me a great deal when updating our recipes to build correctly with RSS.
//Peter
Quoted 76 lines · Peter Kjellerstedt
Quoted 71 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Peter Kjellerstedt
Sent: den 3 mars 2017 20:53
To: Max Krummenacher
Cc: OE-core
Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify needed dependenciesQuoted 8 lines
-----Original Message-----
From: Max Krummenacher [mailto:[email protected]]
Sent: den 3 mars 2017 17:24
To: Peter Kjellerstedt
Cc: OE-core
Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify needed dependenciesQuoted 6 lines
Hi
2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt
<[email protected]>:Quoted 1 line
To circumvent this, I created dummy versions of the tools that
always
Quoted 1 line
Quoted 1 line
fail and placed them in the scripts directory. Thus, if the real
tool
Quoted 1 line
Quoted 1 line
has not been installed in the RSS, the dummy version is used and
the
Quoted 1 line
Quoted 1 line
build fails. For good measures I even output a message that says
what
Quoted 14 lines
Quoted 6 lines
needs to be corrected in the recipe.
Included is also an update to linux-libc-headers as it apparently needs to inherit pkgconfig, which the introduction of the dummy version of pkg-config showed.
I tried the whole series, i.e. applied on top of openembedded-core, 20c6e87 and bitbake 6fe07ed.
The resulting build failed in linux-libc-headers because it picked up the dummy script instead of the real pkg-config binary which actually is installed in the RSS.
Quoted 7 lines
run.do_configure has a PATH setting with first ...meta/scripts and only then the paths to the different RSS directories.
Changing that order in
meta/conf/bitbake.confline 450 fixed this for me, however I'm unsure of posible sideeffects of that reordering.Max
Bah, you are correct. That is what I get for not testing exactly what I sent for integration. Shame on me.
Anyway, I see a couple of possible solutions:
* Put the scripts in bitbake/bin instead. This should work as it is only added to $PATH by oe-buildenv-internal (contrary to ${COREBASE}/scripts which is added to $PATH both by oe-buildenv-internal and bitbake.conf). However, I do not like this idea as it feels wrong to put them there.
* Put the scripts in a subdirectory of ${COREBASE}/scripts, e.g., ${COREBASE}/scripts/blacklisted, and add that path to $PATH in bitbake.conf, but at the other end of it so that the RSS paths appear before it. This sounds a lot more appealing to me.
* Implement the whitelist solution Patrick Ohly mentioned instead. Although this is probably the best solution, prioritizing it for 2.3 does not sound likely to happen.Quoted 4 lines
I have implemented solution two above (properly tested this time), and will send that along in a little while.
//Peter
Quoted 8 lines
Richard,
since I see that you have integrated/staged the two patches that add inherits of pkgconfig, but not the patches that add the dummy commands, I assume you have some reservations to these patches. What are your take on the subject of blacklisting vs whitelisting the commands from the build host? I know that having these dummy commands in place helped me a great deal when updating our recipes to build correctly with RSS.
perhaps if we have a way to containerize the binaries/scripts used from buildhost would be comprehensive but hard to tool in may be blacklisting is contained in that we can weed out the known problems but if we can sanitize and create a comprehensive list of needed stuff from buildhost would be cool, it will probably help the reproducability cause as well.
Quoted 8 lines
//Peter
Signature · 4 lines
_______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 7 lines · Peter Kjellerstedt
since I see that you have integrated/staged the two patches that add inherits of pkgconfig, but not the patches that add the dummy commands, I assume you have some reservations to these patches. What are your take on the subject of blacklisting vs whitelisting the commands from the build host? I know that having these dummy commands in place helped me a great deal when updating our recipes to build correctly with RSS.
FWIW I wasn't entirely keen on how the blacklisting had to be very careful about how PATH was manipulated, which is why I merged to my staging branch the fixes but not the dummy commands.
If the majority of people agree that this should be merged I don't have a strong opinion either way.
Ross
Quoted 12 lines · Ross
On 8 March 2017 at 09:43, Peter Kjellerstedt <[email protected] <mailto:[email protected]>> wrote:
since I see that you have integrated/staged the two patches that add inherits of pkgconfig, but not the patches that add the dummy commands, I assume you have some reservations to these patches. What are your take on the subject of blacklisting vs whitelisting the commands from the build host? I know that having these dummy commands in place helped me a great deal when updating our recipes to build correctly with RSS.
Quoted 6 lines
FWIW I wasn't entirely keen on how the blacklisting had to be very careful about how PATH was manipulated, which is why I merged to my staging branch the fixes but not the dummy commands.
If the majority of people agree that this should be merged I don't have a strong opinion either way.
The scripts would be very useful for people to avoid host-contamination issues.
If we could build up a set of these types of tools (and associated 'please do the following' behaviors) I think it would be very useful. (Starting with these three items will probably catch the majority of things.)
--Mark
Quoted 3 lines
Ross
Quoted 15 lines · Ross
Quoted 12 lines · Peter Kjellerstedt
since I see that you have integrated/staged the two patches that add
inherits of pkgconfig, but not the patches that add the dummy commands,
I assume you have some reservations to these patches. What are your take on the subject of blacklisting vs whitelisting the commands from
the build host? I know that having these dummy commands in place helped
me a great deal when updating our recipes to build correctly with RSS.
Quoted 6 lines
FWIW I wasn't entirely keen on how the blacklisting had to be very careful about how PATH was manipulated, which is why I merged to my staging branch the fixes but not the dummy commands.
If the majority of people agree that this should be merged I don't have a strong opinion either way.
Ultimately I think there is a better solution than this but I haven't had time to look at that as yet.
I do worry that it makes assumptions about PATH that may or may not be valid everywhere, equally I do see it could be useful.
So I also have mixed feelings...
Cheers,
Richard
Quoted 23 lines · Richard Purdie
Quoted 15 lines · Ross
Quoted 12 lines · Peter Kjellerstedt
since I see that you have integrated/staged the two patches that add
inherits of pkgconfig, but not the patches that add the dummy commands,
I assume you have some reservations to these patches. What are your take on the subject of blacklisting vs whitelisting the commands from
the build host? I know that having these dummy commands in place helped
me a great deal when updating our recipes to build correctly with RSS.Quoted 6 lines
FWIW I wasn't entirely keen on how the blacklisting had to be very careful about how PATH was manipulated, which is why I merged to my staging branch the fixes but not the dummy commands.
If the majority of people agree that this should be merged I don't have a strong opinion either way.
Quoted 7 lines
Ultimately I think there is a better solution than this but I haven't had time to look at that as yet.
I do worry that it makes assumptions about PATH that may or may not be valid everywhere, equally I do see it could be useful.
So I also have mixed feelings...
I got many complains about my PNBLACKLISTs saying that the issues weren't reproducible elsewhere and in the end it was in 90% one of these commonly installed tools like pkgconfig being used from the host.
So I agree that PATH changes are tricky and that whitelisting would be even better than this, but still I would prefer to get this merged as it will help to identify and fix most common issues and changing this to whitelist would be easier in future.
My jenkins buildhost is quite minimal (that's why it detected so many issues with RSS, but I'll include the rest of this series in next run to see if it will find even more).
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 49 lines · Martin Jansa
Quoted 38 lines · Richard Purdie
Quoted 29 lines · Ross
Quoted 18 lines · Peter Kjellerstedt
since I see that you have integrated/staged the two patches that
add
inherits of pkgconfig, but not the patches that add the dummy commands,
I assume you have some reservations to these patches. What are your
take on the subject of blacklisting vs whitelisting the commands
from
the build host? I know that having these dummy commands in place
helped
me a great deal when updating our recipes to build correctly with
RSS.FWIW I wasn't entirely keen on how the blacklisting had to be very
careful about how PATH was manipulated, which is why I merged to my
staging branch the fixes but not the dummy commands.Quoted 3 lines
If the majority of people agree that this should be merged I don't
have a strong opinion either way.Ultimately I think there is a better solution than this but I haven't
had time to look at that as yet.Quoted 5 lines
I do worry that it makes assumptions about PATH that may or may not be
valid everywhere, equally I do see it could be useful.So I also have mixed feelings...
I got many complains about my PNBLACKLISTs saying that the issues weren't reproducible elsewhere and in the end it was in 90% one of these commonly installed tools like pkgconfig being used from the host.
Quoted 4 lines
So I agree that PATH changes are tricky and that whitelisting would be even better than this, but still I would prefer to get this merged as it will help to identify and fix most common issues and changing this to whitelist would be easier in future.
I decided to see how hard adding a whitelist would be. I came up with the patch below. Admittedly it changes both bitbake and OE-Core, injects itself into OE-Core's layer.conf and is generally pretty horrible in some ways but equally it does let me start to run builds. It crashes and burns somewhere after libtool-native and the list of tools clearly needs work but it does show what we could do.
I'm therefore torn on Peter's blacklist, or whether its worth a push to the whitelist now...
How badly do people dislike the patch below?
From: Richard Purdie <[email protected]>
Subject: bitbake/oe-core: Filter contents of PATH
bitbake/lib/bb/utils.py +0 -0
@@ -1526,3 +1526,13 @@ class LogCatcher(logging.Handler):
self.messages.append(bb.build.logformatter.format(record)) def contains(self, message):
return (message in self.messages)
+
+def setup_native_bindir(dest, toolsvar, d):
+ tools = d.getVar(toolsvar).split()
+ path = os.environ.get("PATH")
+ mkdirhier(dest)
+ for tool in tools:
+ desttool = os.path.join(dest, tool)
+ if not os.path.exists(desttool):
+ srctool = which(path, tool)
+ os.symlink(srctool, desttool)
meta/conf/layer.conf +0 -0
@@ -59,3 +59,14 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
oprofile->virtual/kernel \
"
+NATIVETOOLS = " \
+ bash sh cut sed gcc ld git rm install which find xargs cat true mktemp \
+ grep tar gzip touch cp mv basename dirname tr getopt sort awk head tail \
+ mkdir patch uniq perl python chmod python3 ar strip expr ls make as \
+ ranlib egrep echo chown cpio tee wc wget bzip2 stat date rmdir od diff \
+ md5sum unlzma dd chrpath file pod2man gunzip python2.7 ln g++ [ \
+ taskset \
+"
+
+DUMMY := "${@bb.utils.setup_native_bindir('${TOPDIR}/nativetools', 'NATIVETOOLS', d)}" +PATH = "${TOPDIR}/nativetools"
Quoted 37 lines · Richard Purdie
How badly do people dislike the patch below?
From: Richard Purdie <[email protected]>
Subject: bitbake/oe-core: Filter contents of PATHSigned-off-by Richard Purdie <[email protected]>bitbake/lib/bb/utils.py +0
-0@@ -1526,3 +1526,13 @@ class LogCatcher(logging.Handler): self.messages.append(bb.build.logformatter.format(record))def
contains(self, message):
return (message in self.messages)bitbake/lib/bb/utils.py +10
-0+ +def setup_native_bindir(dest, toolsvar, d): + tools = d.getVar(toolsvar).split() + path = os.environ.get("PATH") + mkdirhier(dest) + for tool in tools: + desttool = os.path.join(dest, tool) + if not os.path.exists(desttool): + srctool = which(path, tool) + os.symlink(srctool, desttool)meta/conf/layer.conf +0
-0@@ -59,3 +59,14 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \ oprofile->virtual/kernel \"
+NATIVETOOLS = " \
+ bash sh cut sed gcc ld git rm install which find xargs cat true mktemp \
+ grep tar gzip touch cp mv basename dirname tr getopt sort awk
Quoted 8 lines
head tail \
+ mkdir patch uniq perl python chmod python3 ar strip expr ls make as \
+ ranlib egrep echo chown cpio tee wc wget bzip2 stat date rmdir od diff \
+ md5sum unlzma dd chrpath file pod2man gunzip python2.7 ln g++ [ \
+ taskset \
Adding "false true uname test hostname nm objdump objcopy cmp printf env" gets this going much further. taskset is only above as I have local patches in my build which use it for other experiments and I added it just to get things working.
Cheers,
Richard