Warn user on missing optional dependencies
cover Warn user on missing optional dependencies
Following series aim to help developers (maybe with help of autobuilders) to find missing optional dependencies. The principle is to list shared libraries needed for each ELF found in $TARGET_DIR and check if it does not missing in dependency list.
- Patches 1 and 2 implement this feature
- Patch 3 is a proposal to make packages-file-list.txt compatible with top level parallelize
- Last 3 patches are fixes for errors I found for my configuration
Notice, I also work on reproducible builds. I am going to send my work soon. Using it and fixing errors found by current series, I am able to do reproducible builds with TLP enabled (and without per package staging directory).
J?r?me Pouiller (6):
check-shlibs-deps: new script to check shared library dependencies
pkg-generic: add check_shlibs_deps hooks
infra: fix 'packages-file-list.txt' with TLP
ntp: fix missing optional dependencies
xterm: depend on libXinerama if appropriate
xserver_xorg-server: fix dependency with dbus
package/ntp/ntp.mk | 14 ++ package/pkg-generic.mk | 24 +++ .../xserver_xorg-server/xserver_xorg-server.mk | 4 +- package/xterm/xterm.mk | 7 + support/scripts/check-shlibs-deps | 172 +++++++++++++++++++++ 5 files changed, 219 insertions(+), 2 deletions(-) create mode 100755 support/scripts/check-shlibs-deps
Signature · 1 line
2.9.3
PATCH 1/6 check-shlibs-deps: new script to check shared library dependencies
Add a script that show or check binary dependencies based on linked shared libraries.
To do that, it scan NEEDED entries in ELF header and get corresponding package from packages-file-list.txt.
It have 3 modes:
check-shlibs-deps -b OUTDIR
Scan $TARGET_DIR and display found dependencies for all ELF files
check-shlibs-deps -b OUTDIR -p PACKAGE
Display found dependencies for PACKAGE
check-shlibs-deps -b OUTDIR -p PACKAGE -d DEP1,DEP2,...
Display missing dependencies for PACKAGE
Unfortunately, `packages-file-list.txt' is not properly filled when Top Level Parallelization is enabled. Therefore, check-shlibs-deps does not (yet) work with it.
--- support/scripts/check-shlibs-deps | 172 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100755 support/scripts/check-shlibs-deps
support/scripts/check-shlibs-deps new file +172 -0
@@ -0,0 +1,172 @@ +#!/usr/bin/env python + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +# Copyright (C) 2016 by Jerome Pouiller <[email protected]> +# Inspired from size-stats and check-host-rpath scripts + +import os +import re +import subprocess +import argparse + +ignored_deps = [ "unknown", "glibc", "uclibc", "musl" ] + +# Add an entry in dictionnaries pkgsdict and filesdict +def add_file(pkgsdict, filesdict, abspath, pkg): + if not os.path.exists(abspath): + return + #if abspath in filesdict and filesdict[abspath] != pkg: + # print("WARNING: %s is owned by %s, but overwritten by %s" % + # (abspath, filesdict[abspath], pkg)) + filesdict[abspath] = pkg + if not pkg in pkgsdict: + pkgsdict[pkg] = set() + pkgsdict[pkg].add(abspath) + +# Build dictionnaries from "build/packages-file-list.txt" +def build_dicts(builddir): + pkgsdict = {} + filesdict = {} + with open(os.path.join(builddir, "build", "packages-file-list.txt")) as filelistf: + for line in filelistf.readlines(): + pkg, fpath = line.split(",", 1) + fpath = fpath.strip() + fpath = os.path.join(builddir, "target", fpath) + fpath = os.path.normpath(os.path.relpath(fpath)) + add_file(pkgsdict, filesdict, fpath, pkg) + return filesdict, pkgsdict + +# Return package associated to a file +def get_package(filesdict, fpath): + if not fpath in filesdict: + #print("WARNING: %s is not part of any package" % fpath) + # Do not flood user with warning messages. Especially host-gcc-final + # does not declare its files and produce many warnings. + filesdict[fpath] = "unknown" + return filesdict[fpath] + +# Return list of libraries linked with a binary +def get_shared_libs(builddir, binary): + libs = set() + # Host readelf seems able to read foreign binaries (tested with arm/glibc and arm/uclibc) + pipe = subprocess.Popen([ "readelf", "-d", binary ], stdout=subprocess.PIPE) + for line in pipe.stdout: + match = re.match("^.* \(NEEDED\) .*Shared library: \[(.+)\]$", line) + if match: + libname = match.group(1) + # Support case where "/lib" s a symlink to "/usr/lib" + lpaths = set() + for dir in [ "usr/lib", "lib" ]: + lpaths.add(os.path.relpath(os.path.realpath(os.path.join(builddir, "target", dir, libname)))) + found = 0 + for file in lpaths: + if os.path.exists(file): + found += 1 + libs.add(file) + #if found == 0: + # # FIXME: Also take into account RPATH in order to find missed libraries + # print("WARNING: %s depends on %s but it was not found on filesystem" % (binary, libname)) + if found > 1: + print("BUG: %s depends on %s but it was found multiple time on filesystem" % (binary, libname)) + return libs + +# Return a list a dependencies for a list of ELF files +def build_package_deps(builddir, filesdict, bins): + pkgdeps = { } + for binary in bins: + shlibs = get_shared_libs(builddir, binary) + for sh in shlibs: + pkg = get_package(filesdict, binary) + if not pkg in pkgdeps: + pkgdeps[pkg] = set() + dep = get_package(filesdict, sh) + if not dep in ignored_deps and dep != pkg: + pkgdeps[pkg].add(dep) + return pkgdeps + +# Filter ELF files from a list of files +def filter_elf(builddir, files): + bins = set() + pipe = subprocess.Popen([ "file" ] + list(files), stdout=subprocess.PIPE) + for line in pipe.stdout: + match = re.match("^([^:]+): +ELF ", line) + if match: + bins.add(match.group(1)); + return bins + +# Return files found in "target/" +def build_file_list(builddir): + files = set() + for dirpath, _, filelist in os.walk(os.path.join(builddir, "target")): + for f in filelist: + file = os.path.join(dirpath, f) + file = os.path.relpath(os.path.realpath(file)) + if not os.path.islink(file): + files.add(file) + return files + +def main(builddir, package, deps): + filesdict, pkgsdict = build_dicts(builddir) + if package and not package in pkgsdict: + print("'%s' is an unkown package" % package) + exit(0) + + if package: + file_list = pkgsdict[package] + else: + file_list = build_file_list(builddir) + # print("List of files to check:\n %s" % "\n ".join(sorted(file_list))) + bins = filter_elf(builddir, file_list) + # print("List of binaries to check:\n %s" % "\n ".join(sorted(bins))) + pkgdeps = build_package_deps(builddir, filesdict, sorted(bins)) + error = 0 + for p, pdeps in sorted(pkgdeps.items()): + if not deps == None: + for d in pdeps: + if not d in sorted(deps): + print("%s: missed dependency to %s" % (p, d)) + error += 1 + else: + print("%s: %s" % (p, " ".join(sorted(pdeps)))) + return error + + +parser = argparse.ArgumentParser(description='Show or check binary dependencies based on linked shared libraries') + +parser.add_argument("--builddir", '-b', metavar="BUILDDIR", required=True, + help="Buildroot output directory") +parser.add_argument("--package", '-p', metavar="PACKAGE", + help="Check only PACKAGE (else, show dpendencies of all binairies)") +parser.add_argument("--deps", '-d', metavar="DEP1,DEP2,...", nargs='?', default="", + help="Report errors if found dependencies are not a subset of '--deps'. '-p' is mandatory with this option") +parser.add_argument('-w', action='store_true', + help="Do not return non zero when dependency is missing") +args = parser.parse_args() +if not args.package and args.deps: + print("ERROR: cannot use --deps wihout --package") + exit(1) + +if args.deps == "": + deps = None +elif args.deps == None: + deps = [] +else: + deps = args.deps.split(",") + +ret = main(args.builddir, args.package, deps) +if not args.w: + exit(ret) +exit(0)
Signature · 1 line
2.9.3
PATCH 1/6 check-shlibs-deps: new script to check shared library dependencies
Hi J?r?me, all,
Quoted 71 lines · J?r?me Pouiller
Add a script that show or check binary dependencies based on linked shared libraries.
To do that, it scan NEEDED entries in ELF header and get corresponding package from packages-file-list.txt.
It have 3 modes:
check-shlibs-deps -b OUTDIR
Scan $TARGET_DIR and display found dependencies for all ELF filescheck-shlibs-deps -b OUTDIR -p PACKAGE
Display found dependencies for PACKAGEcheck-shlibs-deps -b OUTDIR -p PACKAGE -d DEP1,DEP2,...
Display missing dependencies for PACKAGEUnfortunately, `packages-file-list.txt' is not properly filled when Top Level Parallelization is enabled. Therefore, check-shlibs-deps does not (yet) work with it.
Signed-off-by J?r?me Pouiller <[email protected]>--- support/scripts/check-shlibs-deps | 172 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100755 support/scripts/check-shlibs-depssupport/scripts/check-shlibs-deps new file +38
-0@@ -0,0 +1,172 @@ +#!/usr/bin/env python + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +# Copyright (C) 2016 by Jerome Pouiller <[email protected]> +# Inspired from size-stats and check-host-rpath scripts + +import os +import re +import subprocess +import argparse + +ignored_deps = [ "unknown", "glibc", "uclibc", "musl" ] + +# Add an entry in dictionnaries pkgsdict and filesdict +def add_file(pkgsdict, filesdict, abspath, pkg): + if not os.path.exists(abspath): + return + #if abspath in filesdict and filesdict[abspath] != pkg: + # print("WARNING: %s is owned by %s, but overwritten by %s" % + # (abspath, filesdict[abspath], pkg)) + filesdict[abspath] = pkg + if not pkg in pkgsdict: + pkgsdict[pkg] = set() + pkgsdict[pkg].add(abspath) +
support/scripts/check-shlibs-deps line 48
Quoted 10 lines
support/scripts/check-shlibs-deps new file +10
-0+# Build dictionnaries from "build/packages-file-list.txt" +def build_dicts(builddir): + pkgsdict = {} + filesdict = {} + with open(os.path.join(builddir, "build", "packages-file-list.txt")) as filelistf: + for line in filelistf.readlines(): + pkg, fpath = line.split(",", 1) + fpath = fpath.strip() + fpath = os.path.join(builddir, "target", fpath) + fpath = os.path.normpath(os.path.relpath(fpath))
I wonder why relpath is needed here? since it is resloving relative path from the current location, i.e. BR's top directory. BTW, is it really the BR's top dir. you want to use here as start path to compute the relative path, not $(O)? How does this behave with out-of-tree build?
Quoted 7 lines
support/scripts/check-shlibs-deps new file +7
-0+ add_file(pkgsdict, filesdict, fpath, pkg) + return filesdict, pkgsdict + +# Return package associated to a file +def get_package(filesdict, fpath): + if not fpath in filesdict: + #print("WARNING: %s is not part of any package" % fpath)
support/scripts/check-shlibs-deps line 65
Quoted 10 lines
support/scripts/check-shlibs-deps new file +10
-0+ # Do not flood user with warning messages. Especially host-gcc-final + # does not declare its files and produce many warnings. + filesdict[fpath] = "unknown" + return filesdict[fpath] + +# Return list of libraries linked with a binary +def get_shared_libs(builddir, binary): + libs = set() + # Host readelf seems able to read foreign binaries (tested with arm/glibc and arm/uclibc) + pipe = subprocess.Popen([ "readelf", "-d", binary ], stdout=subprocess.PIPE)
Side note:
There is this project [1] out-there that looks nice, but I wonder if it is worthwhile to add/embedded another python modules dependency to BR.
support/scripts/check-shlibs-deps line 73
Quoted 8 lines
support/scripts/check-shlibs-deps new file +8
-0+ for line in pipe.stdout: + match = re.match("^.* \(NEEDED\) .*Shared library: \[(.+)\]$", line) + if match: + libname = match.group(1) + # Support case where "/lib" s a symlink to "/usr/lib" + lpaths = set() + for dir in [ "usr/lib", "lib" ]: + lpaths.add(os.path.relpath(os.path.realpath(os.path.join(builddir, "target", dir, libname))))
ditto for relpath/start path.
support/scripts/check-shlibs-deps line 86
Quoted 13 lines
support/scripts/check-shlibs-deps new file +13
-0+ found = 0 + for file in lpaths: + if os.path.exists(file): + found += 1 + libs.add(file) + #if found == 0: + # # FIXME: Also take into account RPATH in order to find missed libraries + # print("WARNING: %s depends on %s but it was not found on filesystem" % (binary, libname)) + if found > 1: + print("BUG: %s depends on %s but it was found multiple time on filesystem" % (binary, libname)) + return libs + +# Return a list a dependencies for a list of ELF files
s/a list a/a list of/
support/scripts/check-shlibs-deps line 88
Quoted 2 lines
support/scripts/check-shlibs-deps new file +2
-0+def build_package_deps(builddir, filesdict, bins): + pkgdeps = { }
s/{ }/{}/
Quoted 7 lines
support/scripts/check-shlibs-deps new file +7
-0+ for binary in bins: + shlibs = get_shared_libs(builddir, binary) + for sh in shlibs: + pkg = get_package(filesdict, binary) + if not pkg in pkgdeps: + pkgdeps[pkg] = set() + dep = get_package(filesdict, sh)
support/scripts/check-shlibs-deps line 105
Quoted 10 lines
support/scripts/check-shlibs-deps new file +10
-0+ if not dep in ignored_deps and dep != pkg: + pkgdeps[pkg].add(dep) + return pkgdeps + +# Filter ELF files from a list of files +def filter_elf(builddir, files): + bins = set() + pipe = subprocess.Popen([ "file" ] + list(files), stdout=subprocess.PIPE) + for line in pipe.stdout: + match = re.match("^([^:]+): +ELF ", line)
ditto pyelftools
support/scripts/check-shlibs-deps line 116
Quoted 11 lines
support/scripts/check-shlibs-deps new file +11
-0+ if match: + bins.add(match.group(1)); + return bins + +# Return files found in "target/" +def build_file_list(builddir): + files = set() + for dirpath, _, filelist in os.walk(os.path.join(builddir, "target")): + for f in filelist: + file = os.path.join(dirpath, f) + file = os.path.relpath(os.path.realpath(file))
ditto for relpath/start path.
Quoted 11 lines
support/scripts/check-shlibs-deps new file +11
-0+ if not os.path.islink(file): + files.add(file) + return files + +def main(builddir, package, deps): + filesdict, pkgsdict = build_dicts(builddir) + if package and not package in pkgsdict: + print("'%s' is an unkown package" % package) + exit(0) + + if package:
support/scripts/check-shlibs-deps line 137
Quoted 10 lines
support/scripts/check-shlibs-deps new file +10
-0+ file_list = pkgsdict[package] + else: + file_list = build_file_list(builddir) + # print("List of files to check:\n %s" % "\n ".join(sorted(file_list))) + bins = filter_elf(builddir, file_list) + # print("List of binaries to check:\n %s" % "\n ".join(sorted(bins))) + pkgdeps = build_package_deps(builddir, filesdict, sorted(bins)) + error = 0 + for p, pdeps in sorted(pkgdeps.items()): + if not deps == None:
s/deps == None/deps is None/
or simply: if deps:
Quoted 18 lines
support/scripts/check-shlibs-deps new file +18
-0+ for d in pdeps: + if not d in sorted(deps): + print("%s: missed dependency to %s" % (p, d)) + error += 1 + else: + print("%s: %s" % (p, " ".join(sorted(pdeps)))) + return error + + +parser = argparse.ArgumentParser(description='Show or check binary dependencies based on linked shared libraries') + +parser.add_argument("--builddir", '-b', metavar="BUILDDIR", required=True, + help="Buildroot output directory") +parser.add_argument("--package", '-p', metavar="PACKAGE", + help="Check only PACKAGE (else, show dpendencies of all binairies)") +parser.add_argument("--deps", '-d', metavar="DEP1,DEP2,...", nargs='?', default="", + help="Report errors if found dependencies are not a subset of '--deps'. '-p' is mandatory with this option") +parser.add_argument('-w', action='store_true',
support/scripts/check-shlibs-deps line 165
Quoted 10 lines
support/scripts/check-shlibs-deps new file +10
-0+ help="Do not return non zero when dependency is missing") +args = parser.parse_args() +if not args.package and args.deps: + print("ERROR: cannot use --deps wihout --package") + exit(1) + +if args.deps == "": + deps = None +elif args.deps == None: + deps = []
What is the difference between "deps = None" and "deps = []"?
support/scripts/check-shlibs-deps line 172
Quoted 7 lines
support/scripts/check-shlibs-deps new file +7
-0+else: + deps = args.deps.split(",") + +ret = main(args.builddir, args.package, deps) +if not args.w: + exit(ret) +exit(0)
Please, put all the main code under a 'if __name__ == "__main__":' block.
Quoted 7 lines
--
2.9.3_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
[1] https://github.com/eliben/pyelftools
Regards,
Signature · 1 line
Samuel
PATCH 1/6 check-shlibs-deps: new script to check shared library dependencies
Hello Samuel,
Quoted 1 line · On Monday 6 February 2017 22:04:23 CET Samuel Martin wrote:
On Mon, Nov 14, 2016 at 2:22 PM, J?r?me Pouiller <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 8 lines
support/scripts/check-shlibs-deps new file +8
-0+# Build dictionnaries from "build/packages-file-list.txt" +def build_dicts(builddir): + pkgsdict = {} + filesdict = {} + with open(os.path.join(builddir, "build", "packages-file-list.txt")) as filelistf: + for line in filelistf.readlines(): + pkg, fpath = line.split(",", 1) + fpath = fpath.strip()
Quoted 8 lines
support/scripts/check-shlibs-deps line 48
Quoted 2 lines
support/scripts/check-shlibs-deps new file +2
-0+ fpath = os.path.join(builddir, "target", fpath) + fpath = os.path.normpath(os.path.relpath(fpath))I wonder why relpath is needed here? since it is resloving relative path from the current location, i.e. BR's top directory. BTW, is it really the BR's top dir. you want to use here as start path to compute the relative path, not $(O)? How does this behave with out-of-tree build?
In subsequent case, I need to call realpath() in order to resolve symlinks. realpath() return an absolute paths, but I prefer to convert them in relative paths because they are smaller in error messages.
In case above, relpath is necessary because 'builddir' may be absolute.
Quoted 14 lines
Quoted 14 lines
support/scripts/check-shlibs-deps new file +14
-0+ add_file(pkgsdict, filesdict, fpath, pkg) + return filesdict, pkgsdict + +# Return package associated to a file +def get_package(filesdict, fpath): + if not fpath in filesdict: + #print("WARNING: %s is not part of any package" % fpath) + # Do not flood user with warning messages. Especially host-gcc-final + # does not declare its files and produce many warnings. + filesdict[fpath] = "unknown" + return filesdict[fpath] + +# Return list of libraries linked with a binary +def get_shared_libs(builddir, binary):
Quoted 8 lines
support/scripts/check-shlibs-deps line 65
Quoted 3 lines
support/scripts/check-shlibs-deps new file +3
-0+ libs = set() + # Host readelf seems able to read foreign binaries (tested with arm/glibc and arm/uclibc) + pipe = subprocess.Popen([ "readelf", "-d", binary ], stdout=subprocess.PIPE)Side note:
There is this project [1] out-there that looks nice, but I wonder if it is worthwhile to add/embedded another python modules dependency to BR.
Interesting. (But not enough interesting to rewrite this script :) )
Quoted 12 lines
Quoted 11 lines
(quoted diff) +2
-0+ for line in pipe.stdout: + match = re.match("^.* \(NEEDED\) .*Shared library: \[(.+)\]$",line)
support/scripts/check-shlibs-deps new file +6
-0+ if match: + libname = match.group(1) + # Support case where "/lib" s a symlink to "/usr/lib" + lpaths = set() + for dir in [ "usr/lib", "lib" ]: +
lpaths.add(os.path.relpath(os.path.realpath(os.path.join(builddir, "target", dir, libname))))>ditto for relpath/start path.
[...]
Quoted 10 lines
support/scripts/check-shlibs-deps line 116
Quoted 8 lines
support/scripts/check-shlibs-deps new file +4
-0+# Return files found in "target/" +def build_file_list(builddir): + files = set() + for dirpath, _, filelist in os.walk(os.path.join(builddir,"target")):
support/scripts/check-shlibs-deps new file +3
-0+ for f in filelist: + file = os.path.join(dirpath, f) + file = os.path.relpath(os.path.realpath(file))ditto for relpath/start path.
[...]
Quoted 4 lines
support/scripts/check-shlibs-deps line 137
Quoted 2 lines
support/scripts/check-shlibs-deps new file +2
-0+ for p, pdeps in sorted(pkgdeps.items()): + if not deps == None:s/deps == None/deps is None/
ok
Quoted 1 line
or simply: if deps:
I do want to make a difference between [] and None (see below). So I think "if deps:" does not work.
[...]
Quoted 13 lines
Quoted 13 lines
support/scripts/check-shlibs-deps new file +13
-0+parser = argparse.ArgumentParser(description='Show or check binary dependencies based on linked shared libraries') + +parser.add_argument("--builddir", '-b', metavar="BUILDDIR", required=True, + help="Buildroot output directory") +parser.add_argument("--package", '-p', metavar="PACKAGE", + help="Check only PACKAGE (else, show dpendencies of all binairies)") +parser.add_argument("--deps", '-d', metavar="DEP1,DEP2,...", nargs='?', default="", + help="Report errors if found dependencies are not a subset of '--deps'. Pass "". '-p' is mandatory with this option") +parser.add_argument('-w', action='store_true', + help="Do not return non zero when dependency is missing") +args = parser.parse_args() +if not args.package and args.deps: + print("ERROR: cannot use --deps wihout --package")
Quoted 8 lines
support/scripts/check-shlibs-deps line 165
Quoted 6 lines
support/scripts/check-shlibs-deps new file +6
-0+ exit(1) + +if args.deps == "": + deps = None +elif args.deps == None: + deps = []What is the difference between "deps = None" and "deps = []"?
'None' means user don't to check dependencies (-d is not used) while '[]' means package shouldn't have dependencies.
However, my current implementation is broken. Following command does not work as expected:
check-shlibs-deps -b OUTDIR -p PACKAGE -d ""
I will change that.
Quoted 3 lines
Quoted 3 lines
support/scripts/check-shlibs-deps new file +3
-0+else: + deps = args.deps.split(",") +
[...]
Signature · 3 lines
J?r?me Pouiller, Sysmic Embedded Linux specialist http://www.sysmic.fr
PATCH 1/6 check-shlibs-deps: new script to check shared library dependencies
In reply to Jérôme PouillerHello J?r?me,
Quoted 2 lines · J?r?me Pouiller
Add a script that show or check binary dependencies based on linked shared libraries.
Since you now received some feedback, I've marked patches 1/6 and 2/6 as "Changes Requested" in patchwork. Could you address the comments and submit an updated version?
I think this is a very useful thing, and we'd like to have it in Buildroot.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 2/6 pkg-generic: add check_shlibs_deps hooks
In reply to Jérôme PouillerCall check-shlibs-deps after each package. Errors reported by this hooks mean the package could compile differently depending of build order (= missing optional dependency).
Currently, missed dependencies are only logged to $(BUILD_DIR)/missing-dependencies.txt'. In future, we may remove -w' in order to make these errors fatal.
In order to get correct results from step_check_shlibs_deps, we need to compute recursive dependencies for package. get_recursive_dependencies is brittle, don't try to add spaces or line breaks inside its definition.
A good idea to find many missed optional dependencies would be to use this script with LTP. Unfortunately, check-shlibs-deps does not (yet) work with it.
--- package/pkg-generic.mk | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
package/pkg-generic.mk +20 -0
@@ -110,6 +110,26 @@ define step_check_build_dir endef GLOBAL_INSTRUMENTATION_HOOKS += step_check_build_dir +# Compute dependencies for a set of packages recursively +# $(1): list of package to check +# $(2): list of package already processed. Only used for internal recursion. +# Keep empty for normal usage. +# Example usage: +# $(call get_recursive_dependencies,util-linux,) +define get_recursive_dependencies +$(sort $(foreach d,$(1),$(if $(filter $(d),$(2)),$(d),$(call get_recursive_dependencies,$(d) $($(call UPPERCASE,$(d))_DEPENDENCIES),$(d) $(2))))) +endef + +define step_check_shlibs_deps + $(if $(filter install-target,$(2)),\ + $(if $(filter end,$(1)),support/scripts/check-shlibs-deps \ + -b $(CONFIG_DIR) -p $(3) -w \ + -d $(subst $(space),$(comma),$(call get_recursive_dependencies,$(3),)) \ + | tee -a $(BUILD_DIR)/missing-dependencies.txt + )) +endef +GLOBAL_INSTRUMENTATION_HOOKS += step_check_shlibs_deps + # User-supplied script ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),) define step_user
Signature · 1 line
2.9.3
PATCH 2/6 pkg-generic: add check_shlibs_deps hooks
Hi J?r?me, all,
Quoted 25 lines · J?r?me Pouiller
Call check-shlibs-deps after each package. Errors reported by this hooks mean the package could compile differently depending of build order (= missing optional dependency).
Currently, missed dependencies are only logged to
$(BUILD_DIR)/missing-dependencies.txt'. In future, we may remove-w' in order to make these errors fatal.In order to get correct results from
step_check_shlibs_deps, we need to compute recursive dependencies for package.get_recursive_dependenciesis brittle, don't try to add spaces or line breaks inside its definition.A good idea to find many missed optional dependencies would be to use this script with LTP. Unfortunately, check-shlibs-deps does not (yet) work with it.
Signed-off-by J?r?me Pouiller <[email protected]>--- package/pkg-generic.mk | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)package/pkg-generic.mk +0
-0
package/pkg-generic.mk define step_check_build_dir · line 118
Quoted 9 lines
package/pkg-generic.mk +6
-0@@ -110,6 +110,26 @@ define step_check_build_dir endef GLOBAL_INSTRUMENTATION_HOOKS += step_check_build_dir +# Compute dependencies for a set of packages recursively +# $(1): list of package to check +# $(2): list of package already processed. Only used for internal recursion. +# Keep empty for normal usage. +# Example usage: +# $(call get_recursive_dependencies,util-linux,)
You could introduce a inner-get_recursive_dependencies that takes 2 arguments and keep the user API get_recursive_dependencies only taking 1 argument, which just calls inner-get_recursive_dependencies with an empty 2nd argument.
package/pkg-generic.mk define step_check_build_dir · line 126
Quoted 8 lines
package/pkg-generic.mk +8
-0+define get_recursive_dependencies +$(sort $(foreach d,$(1),$(if $(filter $(d),$(2)),$(d),$(call get_recursive_dependencies,$(d) $($(call UPPERCASE,$(d))_DEPENDENCIES),$(d) $(2))))) +endef + +define step_check_shlibs_deps + $(if $(filter install-target,$(2)),\ + $(if $(filter end,$(1)),support/scripts/check-shlibs-deps \ + -b $(CONFIG_DIR) -p $(3) -w \
Here $(CONFIG_DIR) looks dubious... Why do you use it instead of $(O)?
Quoted 12 lines
package/pkg-generic.mk +6
-1+ -d $(subst $(space),$(comma),$(call get_recursive_dependencies,$(3),)) \ + | tee -a $(BUILD_DIR)/missing-dependencies.txt + )) +endef +GLOBAL_INSTRUMENTATION_HOOKS += step_check_shlibs_deps + # User-supplied script ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),) define step_user --2.9.3
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Regards,
Signature · 1 line
Samuel
PATCH 3/6 infra: fix 'packages-file-list.txt' with TLP
In reply to Jérôme PouillerUntil now, `$(BUILD_DIR)/packages-file-list.txt' was not filled properly when top level parallelization is enabled. Therefore, all scripts that rely on packages-file-list.txt did not work.
In order to fix it,this patch place target installation task in a critical section.
--- package/pkg-generic.mk | 4 ++++ 1 file changed, 4 insertions(+)
package/pkg-generic.mk +4 -0
@@ -62,6 +62,9 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # files currently installed in the target. Note that the MD5 is also # stored, in order to identify if the files are overwritten. define step_pkg_size_start + while ! flock $(BUILD_DIR) -c "[ ! -e $(BUILD_DIR)/.target_lock ] && touch $(BUILD_DIR)/.target_lock"; do \ + sleep 0.5; \ + done (cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ $($(PKG)_DIR)/.br_filelist_before endef
@@ -78,6 +81,7 @@ define step_pkg_size_end while read hash file ; do \ echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \ done + rm $(BUILD_DIR)/.target_lock endef define step_pkg_size
Signature · 1 line
2.9.3
PATCH 3/6 infra: fix 'packages-file-list.txt' with TLP
Hello,
Adding Gustavo in Cc. Gustavo, you are working on TLP support. Could you comment on the below patch?
See also my comments below.
Quoted 17 lines · J?r?me Pouiller
Until now, `$(BUILD_DIR)/packages-file-list.txt' was not filled properly when top level parallelization is enabled. Therefore, all scripts that rely on packages-file-list.txt did not work.
In order to fix it,this patch place target installation task in a critical section.
Signed-off-by J?r?me Pouiller <[email protected]>--- package/pkg-generic.mk | 4 ++++ 1 file changed, 4 insertions(+)package/pkg-generic.mk +0
-0
package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 67
Quoted 6 lines
package/pkg-generic.mk +3
-0@@ -62,6 +62,9 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # files currently installed in the target. Note that the MD5 is also # stored, in order to identify if the files are overwritten. define step_pkg_size_start + while ! flock $(BUILD_DIR) -c "[ ! -e $(BUILD_DIR)/.target_lock ] && touch $(BUILD_DIR)/.target_lock"; do \ + sleep 0.5; \ + done
I personally don't really like this retry loop around flock, but since package-file-list.txt is a global file, I don't really see how to do otherwise. Unless storing a per-package file with just the time/date of the start/end of each step for this package, and then have a post-processing logic at the very end of the build that regroups all those per-package files into a single global file, ordering the entries by their timestamp. Don't know if it's really better.
package/pkg-generic.mk define step_pkg_size_end · line 87
Quoted 11 lines
package/pkg-generic.mk +1
-0(cd $(TARGET_DIR) ; find . -type f -print0 | xargs -0 md5sum) | sort > \ $($(PKG)_DIR)/.br_filelist_before endef@@ -78,6 +81,7 @@ define step_pkg_size_end while read hash file ; do \ echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \ done + rm $(BUILD_DIR)/.target_lock endef define step_pkg_size
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 3/6 infra: fix 'packages-file-list.txt' with TLP
Hello Thomas,
Quoted 31 lines · On Thursday 9 February 2017 23:24:29 CET Thomas Petazzoni wrote:
Hello,
Adding Gustavo in Cc. Gustavo, you are working on TLP support. Could you comment on the below patch?
See also my comments below.
Quoted 23 lines · J?r?me Pouiller
Until now, `$(BUILD_DIR)/packages-file-list.txt' was not filled properly when top level parallelization is enabled. Therefore, all scripts that rely on packages-file-list.txt did not work.
In order to fix it,this patch place target installation task in a critical section.
Signed-off-by J?r?me Pouiller <[email protected]>---
package/pkg-generic.mk | 4 ++++ 1 file changed, 4 insertions(+)package/pkg-generic.mk +0
-0@@ -62,6 +62,9 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # files currently installed in the target. Note that the MD5 is also # stored, in order to identify if the files are overwritten. define step_pkg_size_start
Quoted 11 lines
package/pkg-generic.mk GLOBAL_INSTRUMENTATION_HOOKS += step_time · line 69
Quoted 3 lines
package/pkg-generic.mk +2
-0+ while ! flock $(BUILD_DIR) -c "[ ! -e $(BUILD_DIR)/.target_lock ] && touch $(BUILD_DIR)/.target_lock"; do \ + sleep 0.5; \ + doneI personally don't really like this retry loop around flock, but since package-file-list.txt is a global file, I don't really see how to do otherwise. Unless storing a per-package file with just the time/date of the start/end of each step for this package, and then have a post-processing logic at the very end of the build that regroups all those per-package files into a single global file, ordering the entries by their timestamp. Don't know if it's really better.
I think that any tool involving parallel processing need one day or another to protect code inside a critical section. Even if we find another way to solve current problem, I am pretty sure we will need it later, anyway.
However, I think I am going to modify my patch in order to provide a generic mutex implementation that would be enabled iff TLP is enable.
I wonder where I should place lock file. In add, tools like fakedate, check-shlibs-deps and other QA tools could also generate log or temporary files. Currently, we place all of them in build/ without any specific rules. Maybe we should have a naming policy for these files? (prefix them with "BR_"?) or place them elsewhere than in build/?
[...]
Signature · 3 lines
J?r?me Pouiller, Sysmic Embedded Linux specialist http://www.sysmic.fr
PATCH 3/6 infra: fix 'packages-file-list.txt' with TLP
In reply to Jérôme PouillerHello,
Quoted 8 lines · J?r?me Pouiller
Until now, `$(BUILD_DIR)/packages-file-list.txt' was not filled properly when top level parallelization is enabled. Therefore, all scripts that rely on packages-file-list.txt did not work.
In order to fix it,this patch place target installation task in a critical section.
Signed-off-by J?r?me Pouiller <[email protected]>
Arnout said:
"""
I don't think this is the proper approach. The proper approach is
map/reduce, i.e. generate the pieces in per-package files and collect them together in a final gathering stage. Obviously, this requires a per-package target first, but that's anyway the way we want to go.
I thought someone already made that comment but I can't find it in that thread...
"""
So I've marked this patch as Changes Requested in patchwork.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 4/6 ntp: fix missing optional dependencies
In reply to Jérôme Pouillerntpq and ntpdc may depends on libedit and libcap.
$ arm-linux-readelf -d ./usr/bin/ntpdc | grep NEEDED 0x00000001 (NEEDED) Shared library: [libcap.so.2] 0x00000001 (NEEDED) Shared library: [libm.so.6] 0x00000001 (NEEDED) Shared library: [libedit.so.0] 0x00000001 (NEEDED) Shared library: [libncursesw.so.6] 0x00000001 (NEEDED) Shared library: [libssl.so.1.0.0] 0x00000001 (NEEDED) Shared library: [libcrypto.so.1.0.0] 0x00000001 (NEEDED) Shared library: [libpthread.so.0] 0x00000001 (NEEDED) Shared library: [libc.so.6]
However, build order with these libraries is not defined.
In order to keep things simple, we enforce build order even if ntpq/ntpdc are not selected.
--- package/ntp/ntp.mk | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
package/ntp/ntp.mk +14 -0
@@ -28,6 +28,20 @@ else NTP_CONF_OPTS += --without-crypto --disable-openssl-random endif +ifeq ($(BR2_PACKAGE_LIBCAP),y) +NTP_CONF_OPTS += --enable-linuxcaps +NTP_DEPENDENCIES += libcap +else +NTP_CONF_OPTS += --disable-linuxcaps +endif + +ifeq ($(BR2_PACKAGE_LIBEDIT),y) +NTP_CONF_OPTS += --with-lineeditlibs=edit +NTP_DEPENDENCIES += libedit +else +NTP_CONF_OPTS += --with-lineeditlibs="" +endif + ifeq ($(BR2_PACKAGE_NTP_NTPSNMPD),y) NTP_CONF_OPTS += \ --with-net-snmp-config=$(STAGING_DIR)/usr/bin/net-snmp-config
Signature · 1 line
2.9.3
PATCH 4/6 ntp: fix missing optional dependencies
Hello,
Quoted 12 lines · J?r?me Pouiller
ntpq and ntpdc may depends on libedit and libcap.
$ arm-linux-readelf -d ./usr/bin/ntpdc | grep NEEDED 0x00000001 (NEEDED) Shared library: [libcap.so.2] 0x00000001 (NEEDED) Shared library: [libm.so.6] 0x00000001 (NEEDED) Shared library: [libedit.so.0] 0x00000001 (NEEDED) Shared library: [libncursesw.so.6] 0x00000001 (NEEDED) Shared library: [libssl.so.1.0.0] 0x00000001 (NEEDED) Shared library: [libcrypto.so.1.0.0] 0x00000001 (NEEDED) Shared library: [libpthread.so.0] 0x00000001 (NEEDED) Shared library: [libc.so.6]
Quoted 6 lines
However, build order with these libraries is not defined.
In order to keep things simple, we enforce build order even if ntpq/ntpdc are not selected.
Signed-off-by J?r?me Pouiller <[email protected]>
I've applied your patch, after doing one small change, see below.
package/ntp/ntp.mk else · line 42
Quoted 5 lines
package/ntp/ntp.mk +5
-0+ifeq ($(BR2_PACKAGE_LIBEDIT),y) +NTP_CONF_OPTS += --with-lineeditlibs=edit +NTP_DEPENDENCIES += libedit +else +NTP_CONF_OPTS += --with-lineeditlibs=""
I've replaced this by --without-lineeditlibs, which has the same effect but is more commonly used in Buildroot. I did check that with the libedit package installed, but forcing --without-lineeditlibs really causes ntp to ignore libedit, which proves that it does work.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 5/6 xterm: depend on libXinerama if appropriate
In reply to Jérôme Pouillerxterm may depends on libXinerama.so:
$ arm-linux-readelf -d ./usr/bin/xterm | grep NEEDED 0x00000001 (NEEDED) Shared library: [libXft.so.2] 0x00000001 (NEEDED) Shared library: [libfontconfig.so.1] 0x00000001 (NEEDED) Shared library: [libXmu.so.6] 0x00000001 (NEEDED) Shared library: [libXaw.so.7] 0x00000001 (NEEDED) Shared library: [libXt.so.6] 0x00000001 (NEEDED) Shared library: [libX11.so.6] 0x00000001 (NEEDED) Shared library: [libXinerama.so.1] 0x00000001 (NEEDED) Shared library: [libXpm.so.4] 0x00000001 (NEEDED) Shared library: [libICE.so.6] 0x00000001 (NEEDED) Shared library: [libncursesw.so.6] 0x00000001 (NEEDED) Shared library: [libc.so.6]
However, build order with libXinerama is not defined.
--- package/xterm/xterm.mk | 7 +++++++ 1 file changed, 7 insertions(+)
package/xterm/xterm.mk +7 -0
@@ -22,4 +22,11 @@ else XTERM_CONF_OPTS += --disable-freetype endif +ifeq ($(BR2_PACKAGE_XLIB_LIBXINERAMA),y) +XTERM_DEPENDENCIES += xlib_libXinerama +XTERM_CONF_OPTS += --with-xinerama +else +XTERM_CONF_OPTS += --without-xinerama +endif + $(eval $(autotools-package))
Signature · 1 line
2.9.3
PATCH 6/6 xserver_xorg-server: fix dependency with dbus
In reply to Jérôme PouillerIf dbus and udev are enabled. Xorg link with both:
$ arm-linux-readelf -d usr/bin/X | grep NEEDED 0x00000001 (NEEDED) Shared library: [libdbus-1.so.3] 0x00000001 (NEEDED) Shared library: [libpthread.so.0] 0x00000001 (NEEDED) Shared library: [libudev.so.1] [...]
--- package/x11r7/xserver_xorg-server/xserver_xorg-server.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
package/x11r7/xserver_xorg-server/xserver_xorg-server.mk +2 -2
@@ -169,12 +169,12 @@ XSERVER_XORG_SERVER_CONF_OPTS += --enable-config-udev-kms else XSERVER_XORG_SERVER_CONF_OPTS += --disable-config-udev-kms endif -else +endif + ifeq ($(BR2_PACKAGE_DBUS),y) XSERVER_XORG_SERVER_DEPENDENCIES += dbus XSERVER_XORG_SERVER_CONF_OPTS += --enable-config-dbus endif -endif ifeq ($(BR2_PACKAGE_FREETYPE),y) XSERVER_XORG_SERVER_DEPENDENCIES += freetype
Signature · 1 line
2.9.3
Thomas Petazzoni
AppliedPATCH 6/6 xserver_xorg-server: fix dependency with dbus
Hello,
Quoted 8 lines · J?r?me Pouiller
If dbus and udev are enabled. Xorg link with both:
$ arm-linux-readelf -d usr/bin/X | grep NEEDED 0x00000001 (NEEDED) Shared library: [libdbus-1.so.3] 0x00000001 (NEEDED) Shared library: [libpthread.so.0] 0x00000001 (NEEDED) Shared library: [libudev.so.1] [...]
Quoted 4 lines
Signed-off-by J?r?me Pouiller <[email protected]>--- package/x11r7/xserver_xorg-server/xserver_xorg-server.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com