check and convert file depends on bash, perl and python

If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't have bash, python or perl, then warn.

This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the its "Requires" section if it is a perl script, but ipk or deb can't depend on file such as "/usr/bin/perl", they can only depend on pkg, so they would know nothing about perl, then there would be depend issues.

This check can help us figure out the depends, it nearly has no effect on performance when enabled.

[YOCTO #1662]

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
  1 file changed, 30 insertions(+), 2 deletions(-)

meta/classes/insane.bbclass +0 -0

index 106ace7..a14e668 100644

meta/classes/insane.bbclass +29 -1

  QADEPENDS_class-native = ""
  QADEPENDS_class-nativesdk = ""
  QA_SANE = "True"

@@ -29,7 +29,7 @@ QA_SANE = "True"
  WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
              textrel already-stripped incompatible-license files-invalid \
              installed-vs-shipped compile-host-path install-host-path \
-            pn-overrides infodir \
+            pn-overrides infodir file-depends \
              "
  ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
              perms dep-cmp pkgvarcheck perm-config perm-line perm-link \

@@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf, messages):
                  trimmed = path.replace(os.path.join (d.getVar("PKGDEST", True), name), "")
                  messages.append("Symlink %s in %s points to TMPDIR" % (trimmed, name))
 
+QAPATHTEST[file-depends] = "package_qa_check_file_depends"
+def package_qa_check_file_depends(path, name, d, elf, messages):
+    """
+    Check that the package doesn't miss the RDEPENDS on perl, python and bash
+    """
+
+    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
+        return
+
+    import subprocess
+    import re
+
+    # The file command may return:
+    # - Perl or perl, Python or python.
+    # - "Bourne-Again shell" script or "bash" for bash script
+    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
+                "Bourne-Again shell": "bash"}
+
+    file_output = bb.process.Popen(["file", "-b", path], stdout=subprocess.PIPE).stdout.read()
+    for p in prog_maps:
+        if re.search('%s .*script, ASCII text executable' % p, file_output):
+            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name, True) or "")
+            if prog_maps[p] not in rdepends:
+                if prog_maps[p] == "python" and "python-core" in rdepends:
+                    continue
+                messages.append("%s is a %s script, but %s is not in RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
+                break
+
  def package_qa_check_license(workdir, d):
      """
      Check for changes in the license files

Quoted 23 lines · Robert Yang

If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't have bash, python or perl, then warn.

This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the its "Requires" section if it is a perl script, but ipk or deb can't depend on file such as "/usr/bin/perl", they can only depend on pkg, so they would know nothing about perl, then there would be depend issues.

This check can help us figure out the depends, it nearly has no effect on performance when enabled.

[YOCTO #1662]

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
  1 file changed, 30 insertions(+), 2 deletions(-)

meta/classes/insane.bbclass +0 -0

index 106ace7..a14e668 100644

meta/classes/insane.bbclass line 22

Quoted 5 lines

meta/classes/insane.bbclass +1 -1

@@ -19,7 +19,7 @@
 
  # unsafe-references-in-binaries requires prelink-rtld from
  # prelink-native, but we don't want this DEPENDS for -native builds
-QADEPENDS = "prelink-native"
+QADEPENDS = "prelink-native file-native"

If the perfile dependency processing is enabled, run in: package_do_filedeps

The system will already have gathered the information needed for the QA processing. You can look at:

# Collect perfile run-time dependency metadata # Output:
# FILERPROVIDESFLIST_pkg - list of all files w/ deps # FILERPROVIDES_filepath_pkg - per file dep #
# FILERDEPENDSFLIST_pkg - list of all files w/ deps # FILERDEPENDS_filepath_pkg - per file dep

variables for the per-file dependencies and then warn if a dependency is not already declared in the manual RDEPENDS -- and/or promote them to real dependencies automatically.

This should be faster then iterating through and calling file directly.

--Mark

Quoted 54 lines

meta/classes/insane.bbclass +1 -1

  QADEPENDS_class-native = ""
  QADEPENDS_class-nativesdk = ""
  QA_SANE = "True"

@@ -29,7 +29,7 @@ QA_SANE = "True"
  WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
              textrel already-stripped incompatible-license files-invalid \
              installed-vs-shipped compile-host-path install-host-path \
-            pn-overrides infodir \
+            pn-overrides infodir file-depends \
              "
  ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
              perms dep-cmp pkgvarcheck perm-config perm-line perm-link \

@@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d,

elf, messages):
trimmed = path.replace(os.path.join (d.getVar("PKGDEST",
True), name), "")
messages.append("Symlink %s in %s points to TMPDIR" %
(trimmed, name))

meta/classes/insane.bbclass +19 -0

+QAPATHTEST[file-depends] = "package_qa_check_file_depends"
+def package_qa_check_file_depends(path, name, d, elf, messages):
+    """
+    Check that the package doesn't miss the RDEPENDS on perl, python and bash
+    """
+
+    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
+        return
+
+    import subprocess
+    import re
+
+    # The file command may return:
+    # - Perl or perl, Python or python.
+    # - "Bourne-Again shell" script or "bash" for bash script
+    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
+                "Bourne-Again shell": "bash"}
+
+    file_output = bb.process.Popen(["file", "-b", path],

stdout=subprocess.PIPE).stdout.read()

meta/classes/insane.bbclass +3 -0

+    for p in prog_maps:
+        if re.search('%s .*script, ASCII text executable' % p, file_output):
+            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name,

True) or "")

meta/classes/insane.bbclass +4 -0

+            if prog_maps[p] not in rdepends:
+                if prog_maps[p] == "python" and "python-core" in rdepends:
+                    continue
+                messages.append("%s is a %s script, but %s is not in

RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))

meta/classes/insane.bbclass +2 -0

+                break
+
  def package_qa_check_license(workdir, d):
      """
      Check for changes in the license files

+ prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}

Quoted 1 line · Robert Yang

+ prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}

Use something like d.expand("${bindir}/perl") instead of hard-coding paths.

Ross

Quoted 1 line · Robert Yang

+ prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}

Use something like d.expand("${bindir}/perl") instead of hard-coding paths.

Ross

Hello,

Found that eglibc-utils/usr/bin/tzselect is a bash script, should we add:

RDEPENDS_eglibc-utils += "bash" ?

The problem is that this would cause bash installed to meta-toolchain, the dependencies chain is:

meta-toolchain -> eglibc-utils -> bash

Maybe we should only warn rather than add the RDEPENDS atm, please ?

// Robert

Quoted 8 lines · Ross

Quoted 1 line · Robert Yang

+ prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}

Use something like d.expand("${bindir}/perl") instead of hard-coding paths.

Ross