Skip to contentSkip to conversation

insane/package: refactor packaging sanity tests

PATCH Superseded Robert Yang 3 participants 6 messages May 13, 2013 – May 15, 2013

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover insane/package: refactor packaging sanity tests

The following changes since commit c5c1517726aa103a3cdb60abda95e28997cac7c4:

image-vmdk.bbclass: disable the NOISO = "1" (2013-05-12 09:37:21 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/qa
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/qa

Robert Yang (3):
insane/package: let package.bbclass inherit insane.bbclass defaultsetup.conf: remove INHERIT_INSANE insane/package: refactor packaging sanity tests

 meta/classes/insane.bbclass        |   39 ++++++++++++++++++++++------------
 meta/classes/package.bbclass       |   41 ++++++++++++++++++++++++------------
 meta/conf/distro/defaultsetup.conf |    3 +--
 3 files changed, 54 insertions(+), 29 deletions(-)
Signature · 1 line
1.7.10.4
#1 lore raw

PATCH 1/3 insane/package: let package.bbclass inherit insane.bbclass

RP's comment:
"What we're trying to do is move everything to use a standard mechanism for reporting issues of this type (do_package). With insane.bbclass, you can elect whether a given type of error is a warning or error and fails the task."

* The package.bbclass had used package_qa_handle_error() which is from insane.bbclass, and we will use it for handling other warnings and errors, so let package.bbclass inherit insane.bbclass, this change will make the insane as a requirement (always included).

* Change the "PACKAGEFUNCS ?=" to "+=", otherwise there would be an error like:
Exception: variable SUMMARY references itself!

This is because we let package.bbclass inherit insane.bbclass, and PACKAGEFUNCS has been set in insane.bbclass, so the "PACKAGEFUNCS ?=" will set nothing, then the "emit_pkgdata" doesn't run which will cause this error.

* Add a QA_SANE variable in insane.bbclass, once the error type is ERROR_QA, it will fail the task and stop the build.

[YOCTO #3190]
[YOCTO #4396]

---
 meta/classes/insane.bbclass  |    6 ++++--
 meta/classes/package.bbclass |    5 ++++-
 2 files changed, 8 insertions(+), 3 deletions(-)
meta/classes/insane.bbclass +4 -2
index fb18022..c3e4b1e 100644
@@ -17,7 +17,6 @@
 #   files under exec_prefix
 
 
-inherit package
 PACKAGE_DEPENDS += "${QADEPENDS}"
 PACKAGEFUNCS += " do_package_qa "
 
@@ -26,6 +25,7 @@ PACKAGEFUNCS += " do_package_qa "
 QADEPENDS = "prelink-native"
 QADEPENDS_class-native = ""
 QADEPENDS_class-nativesdk = ""
+QA_SANE = "True"
 
 #
 # dictionary for elf headers
@@ -133,6 +133,7 @@ def package_qa_handle_error(error_class, error_msg, d):
     package_qa_write_error(error_msg, d)
     if error_class in (d.getVar("ERROR_QA", True) or "").split():
         bb.error("QA Issue: %s" % error_msg)
+        d.setVar("QA_SANE", False)
         return False
     else:
         bb.warn("QA Issue: %s" % error_msg)
@@ -821,7 +822,8 @@ python do_package_qa () {
     if 'libdir' in d.getVar("ALL_QA", True).split():
         package_qa_check_libdir(d)
 
-    if not walk_sane or not rdepends_sane or not deps_sane:
+    qa_sane = d.getVar("QA_SANE", True)
+    if not walk_sane or not rdepends_sane or not deps_sane or not qa_sane:
         bb.fatal("QA run found fatal errors. Please consider fixing them.")
     bb.note("DONE with PACKAGE QA")
 }
meta/classes/package.bbclass +4 -1
index 36b3ae5..e9a324e 100644
@@ -42,6 +42,9 @@ inherit packagedata
 inherit prserv
 inherit chrpath
 
+# Need the package_qa_handle_error() in insane.bbclass
+inherit insane
+
 PKGD    = "${WORKDIR}/package"
 PKGDEST = "${WORKDIR}/packages-split"
 
@@ -1814,7 +1817,7 @@ PACKAGESPLITFUNCS ?= " \
                 package_do_split_locales \
                 populate_packages"
 # Functions which process metadata based on split packages
-PACKAGEFUNCS ?= " \
+PACKAGEFUNCS += " \
                 package_fixsymlinks \
                 package_name_hook \
                 package_do_filedeps \
Signature · 1 line
1.7.10.4
#2 lore raw

Re: [PATCH 1/3] insane/package: let package.bbclass inherit insane.bbclass

Quoted 6 lines · Robert Yang

RP's comment:
"What we're trying to do is move everything to use a standard mechanism for reporting issues of this type (do_package). With insane.bbclass, you can elect whether a given type of error is a warning or error and fails the task."

Quoted 9 lines

* The package.bbclass had used package_qa_handle_error() which is from insane.bbclass, and we will use it for handling other warnings and errors, so let package.bbclass inherit insane.bbclass, this change will make the insane as a requirement (always included).

* Change the "PACKAGEFUNCS ?=" to "+=", otherwise there would be an error like:
Exception: variable SUMMARY references itself!

Robert,

Did you test this with a multilib build? I am seeing this error with your change set on the Autobuilder.

Below is from the world build, you can see more examples of this failure in the nightly-multilib

http://autobuilder.yoctoproject.org:8011/builders/nightly-multilib/builds/138/steps/Building%20Images/logs/stdio

Sau!

Quoted 135 lines

ERROR: Error executing a python function in /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-world/build/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_3.8.bb:

The stack trace of python calls that resulted in this exception/failure was:
File: 'write_specfile', lineno: 505, function: <module>

     0501:
     0502:    specfile.close()
     0503:
     0504:
 *** 0505:write_specfile(d)
     0506:

File: 'write_specfile', lineno: 209, function: write_specfile
0205: conffiles = (localdata.getVar('CONFFILES', True) or "").split()
0206:
0207: splitname = strip_multilib(pkgname, d)
0208:

 *** 0209:        splitsummary = (localdata.getVar('SUMMARY', True) or localdata.getVar('DESCRIPTION', True) or ".")
     0210:        splitversion = (localdata.getVar('PKGV', True) or "").replace('-', '+')
     0211:        splitrelease = (localdata.getVar('PKGR', True) or "")
     0212:        splitepoch   = (localdata.getVar('PKGE', True) or "")
     0213:        splitlicense = (localdata.getVar('LICENSE', True) or "")

File: '/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-world/build/bitbake/lib/bb/data_smart.py', lineno: 503, function: getVar
0499: value = self.getVarFlag(var, "_content", False, noweakdefault)
0500:
0501: # Call expand() separately to make use of the expand cache
0502: if expand and value:
*** 0503: return self.expand(value, var)
0504: return value
0505:
0506: def renameVar(self, key, newkey, **loginfo):
0507: """
File: '/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-world/build/bitbake/lib/bb/data_smart.py', lineno: 336, function: expand
0332:
0333: return varparse
0334:
0335: def expand(self, s, varname = None):
*** 0336: return self.expandWithRefs(s, varname).value
0337:
0338:
0339: def finalize(self, parent = False):
0340: """Performs final steps upon the datastore, including application of overrides"""
File: '/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-world/build/bitbake/lib/bb/data_smart.py', lineno: 326, function: expandWithRefs
0322: raise
0323: except bb.parse.SkipPackage:
0324: raise
0325: except Exception as exc:
*** 0326: raise ExpansionError(varname, s, exc)
0327:
0328: varparse.value = s
0329:
0330: if varname:
Exception: ExpansionError: Failure expanding variable SUMMARY, expression was ${SUMMARY} - Debugging files which triggered exception Exception: variable SUMMARY references itself!

ERROR: Function failed: write_specfile
ERROR: Logfile of failure stored in: /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-world/build/build/tmp/work/x86-pokymllib32-linux/lib32-linux-libc-headers/3.8-r0/temp/log.do_package_write_rpm.32203

This is because we let package.bbclass inherit insane.bbclass, and PACKAGEFUNCS has been set in insane.bbclass, so the "PACKAGEFUNCS ?=" will set nothing, then the "emit_pkgdata" doesn't run which will cause this error.

* Add a QA_SANE variable in insane.bbclass, once the error type is ERROR_QA, it will fail the task and stop the build.

[YOCTO #3190]
[YOCTO #4396]

---
  meta/classes/insane.bbclass  |    6 ++++--
  meta/classes/package.bbclass |    5 ++++-
  2 files changed, 8 insertions(+), 3 deletions(-)
meta/classes/insane.bbclass +4 -2
index fb18022..c3e4b1e 100644
@@ -17,7 +17,6 @@
  #   files under exec_prefix
 
 
-inherit package
  PACKAGE_DEPENDS += "${QADEPENDS}"
  PACKAGEFUNCS += " do_package_qa "
 
@@ -26,6 +25,7 @@ PACKAGEFUNCS += " do_package_qa "
  QADEPENDS = "prelink-native"
  QADEPENDS_class-native = ""
  QADEPENDS_class-nativesdk = ""
+QA_SANE = "True"
 
  #
  # dictionary for elf headers
@@ -133,6 +133,7 @@ def package_qa_handle_error(error_class, error_msg, d):
      package_qa_write_error(error_msg, d)
      if error_class in (d.getVar("ERROR_QA", True) or "").split():
          bb.error("QA Issue: %s" % error_msg)
+        d.setVar("QA_SANE", False)
          return False
      else:
          bb.warn("QA Issue: %s" % error_msg)
@@ -821,7 +822,8 @@ python do_package_qa () {
      if 'libdir' in d.getVar("ALL_QA", True).split():
          package_qa_check_libdir(d)
 
-    if not walk_sane or not rdepends_sane or not deps_sane:
+    qa_sane = d.getVar("QA_SANE", True)
+    if not walk_sane or not rdepends_sane or not deps_sane or not qa_sane:
          bb.fatal("QA run found fatal errors. Please consider fixing them.")
      bb.note("DONE with PACKAGE QA")
  }
meta/classes/package.bbclass +4 -1
index 36b3ae5..e9a324e 100644
@@ -42,6 +42,9 @@ inherit packagedata
  inherit prserv
  inherit chrpath
 
+# Need the package_qa_handle_error() in insane.bbclass
+inherit insane
+
  PKGD    = "${WORKDIR}/package"
  PKGDEST = "${WORKDIR}/packages-split"
 
@@ -1814,7 +1817,7 @@ PACKAGESPLITFUNCS ?= " \
                  package_do_split_locales \
                  populate_packages"
  # Functions which process metadata based on split packages
-PACKAGEFUNCS ?= " \
+PACKAGEFUNCS += " \
                  package_fixsymlinks \
                  package_name_hook \
                  package_do_filedeps \
#3 lore raw

PATCH 2/3 defaultsetup.conf: remove INHERIT_INSANE

In reply to Robert Yang

The insane has been inherited by package.bbclass and becomes a requirement, so we can remove it from defaultsetup.conf.

Note:
You can decide whether to take this patch or not.

[YOCTO #3190]
[YOCTO #4396]

---
 meta/conf/distro/defaultsetup.conf |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/conf/distro/defaultsetup.conf +1 -2
index be28510..5557350 100644
@@ -17,7 +17,6 @@ CACHE = "${TMPDIR}/cache/${TCMODE}-${TCLIBC}${@['', '/' + str(d.getVar('MACHINE'
 
 USER_CLASSES ?= ""
 PACKAGE_CLASSES ?= "package_ipk"
-INHERIT_INSANE ?= "insane"
 INHERIT_DISTRO ?= "debian devshell sstate license"
-INHERIT += "${PACKAGE_CLASSES} ${USER_CLASSES} ${INHERIT_INSANE} ${INHERIT_DISTRO}"
+INHERIT += "${PACKAGE_CLASSES} ${USER_CLASSES} ${INHERIT_DISTRO}"
 
Signature · 1 line
1.7.10.4
#4 lore raw

PATCH 3/3 insane/package: refactor packaging sanity tests

In reply to Robert Yang

Refactor packaging sanity tests from package.bbclass to insane.bbclass so that the message can respect WARN_QA (print the warning message and go on the task) and ERROR_QA (print the error message and fail the task).

- For the bb.warn(), give it a message name and add it to WARN_QA, then use package_qa_handle_error() to handle it.

- For the bb.error(), give it a message name and add it to ERROR_QA, then use package_qa_handle_error() to handle it.

- All the bb.warn() and bb.error() have been replaced in package.bbclass.

- A few bb.warn() and bb.error() in insane.bbclass have been kept since they can not be replaced or doesn't have to, for example the bb.error() in package_qa_check_license(), it will print the error message and then invoke bb.fatal() to fail the task, I think that we don't have to replace it with package_qa_handle_error().

- Put all the WARN_QA and ERROR_QA in one line, so that they can be redefined by the user easily.

[YOCTO #3190]
[YOCTO #4396]

---
 meta/classes/insane.bbclass  |   33 ++++++++++++++++++++++-----------
 meta/classes/package.bbclass |   36 +++++++++++++++++++++++-------------
 2 files changed, 45 insertions(+), 24 deletions(-)
meta/classes/insane.bbclass +22 -11
index c3e4b1e..a6c0142 100644
@@ -27,6 +27,20 @@ QADEPENDS_class-native = ""
 QADEPENDS_class-nativesdk = ""
 QA_SANE = "True"
 
+# Elect whether a given type of error is a warning or error, they may
+# have been set by other files.
+WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
+            textrel already-stripped incompatible-license files-invalid \
+            installed-vs-shipped compile-host-path install-host-path \
+            pn-overrides \
+            "
+ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch la2 pkgconfig la \
+            perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
+            split-strip packages-list pkgv-undefined var-undefined \
+            "
+
+ALL_QA = "${WARN_QA} ${ERROR_QA}"
+
 #
 # dictionary for elf headers
 #
@@ -111,12 +125,6 @@ def package_qa_get_machine_dict():
         }
 
 
-# Currently not being used by default "desktop"
-WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi textrel"
-ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch la2 pkgconfig la perms dep-cmp pkgvarcheck"
-
-ALL_QA = "${WARN_QA} ${ERROR_QA}"
-
 def package_qa_clean_path(path,d):
     """ Remove the common prefix from the path. In this case it is the TMPDIR"""
     return path.replace(d.getVar('TMPDIR',True),"")
@@ -758,8 +766,9 @@ python do_package_qa () {
     if os.path.exists(compilelog):
         statement = "grep -e 'CROSS COMPILE Badness:' -e 'is unsafe for cross-compilation' %s > /dev/null" % compilelog
         if subprocess.call(statement, shell=True) == 0:
-            bb.warn("%s: The compile log indicates that host include and/or library paths were used.\n \
-        Please check the log '%s' for more information." % (pkg, compilelog))
+            msg = "%s: The compile log indicates that host include and/or library paths were used.\n \
+        Please check the log '%s' for more information." % (pkg, compilelog)
+            package_qa_handle_error("compile-host-path", msg, d)
 
     # Check the install log for host contamination
     installlog = os.path.join(logdir,"log.do_install")
@@ -767,8 +776,9 @@ python do_package_qa () {
     if os.path.exists(installlog):
         statement = "grep -e 'CROSS COMPILE Badness:' -e 'is unsafe for cross-compilation' %s > /dev/null" % installlog
         if subprocess.call(statement, shell=True) == 0:
-            bb.warn("%s: The install log indicates that host include and/or library paths were used.\n \
-        Please check the log '%s' for more information." % (pkg, installlog))
+            msg = "%s: The install log indicates that host include and/or library paths were used.\n \
+        Please check the log '%s' for more information." % (pkg, installlog)
+            package_qa_handle_error("install-host-path", msg, d)
 
     # Scan the packages...
     pkgdest = d.getVar('PKGDEST', True)
@@ -912,7 +922,8 @@ python () {
     overrides = d.getVar('OVERRIDES', True).split(':')
     pn = d.getVar('PN', True)
     if pn in overrides:
-        bb.warn('Recipe %s has PN of "%s" which is in OVERRIDES, this can result in unexpected behaviour.' % (d.getVar("FILE", True), pn))
+        msg = 'Recipe %s has PN of "%s" which is in OVERRIDES, this can result in unexpected behaviour.' % (d.getVar("FILE", True), pn)
+        package_qa_handle_error("pn-overrides", msg, d)
 
     issues = []
     if (d.getVar('PACKAGES', True) or "").split():
meta/classes/package.bbclass +23 -13
index e9a324e..2515565 100644
@@ -504,7 +504,8 @@ python fixup_perms () {
             elif len(lsplit) == 8:
                 self._setdir(lsplit[0], lsplit[1], lsplit[2], lsplit[3], lsplit[4], lsplit[5], lsplit[6], lsplit[7])
             else:
-                bb.error("Fixup Perms: invalid config line %s" % line)
+                msg = "Fixup Perms: invalid config line %s" % line
+                package_qa_handle_error("perm-config", msg, d)
                 self.path = None
                 self.link = None
 
@@ -639,7 +640,8 @@ python fixup_perms () {
                 if len(lsplit) == 0:
                     continue
                 if len(lsplit) != 8 and not (len(lsplit) == 3 and lsplit[1].lower() == "link"):
-                    bb.error("Fixup perms: %s invalid line: %s" % (conf, line))
+                    msg = "Fixup perms: %s invalid line: %s" % (conf, line)
+                    package_qa_handle_error("perm-line", msg, d)
                     continue
                 entry = fs_perms_entry(d.expand(line))
                 if entry and entry.path:
@@ -668,7 +670,8 @@ python fixup_perms () {
             target = os.path.join(os.path.dirname(origin), link)
             ptarget = os.path.join(os.path.dirname(dir), link)
         if os.path.exists(target):
-            bb.error("Fixup Perms: Unable to correct directory link, target already exists: %s -> %s" % (dir, ptarget))
+            msg = "Fixup Perms: Unable to correct directory link, target already exists: %s -> %s" % (dir, ptarget)
+            package_qa_handle_error("perm-link", msg, d)
             continue
 
         # Create path to move directory to, move it, and then setup the symlink
@@ -738,7 +741,8 @@ python split_and_strip_files () {
         ret, result = oe.utils.getstatusoutput("file '%s'" % path)
 
         if ret:
-            bb.error("split_and_strip_files: 'file %s' failed" % path)
+            msg = "split_and_strip_files: 'file %s' failed" % path
+            package_qa_handle_error("split-strip", msg, d)
             return type
 
         # Not stripped
@@ -803,7 +807,8 @@ python split_and_strip_files () {
                     elf_file = isELF(file)
                     if elf_file & 1:
                         if elf_file & 2:
-                            bb.warn("File '%s' from %s was already stripped, this will prevent future debugging!" % (file[len(dvar):], pn))
+                            msg = "File '%s' from %s was already stripped, this will prevent future debugging!" % (file[len(dvar):], pn)
+                            package_qa_handle_error("already-stripped", msg, d)
                             continue
                         # Check if it's a hard link to something else
                         if s.st_nlink > 1:
@@ -929,9 +934,11 @@ python populate_packages () {
 
     for pkg in packages.split():
         if d.getVar('LICENSE_EXCLUSION-' + pkg, True):
-            bb.warn("%s has an incompatible license. Excluding from packaging." % pkg)
+            msg = "%s has an incompatible license. Excluding from packaging." % pkg
+            package_qa_handle_error("incompatible-license", msg, d)
         if pkg in package_list:
-            bb.error("%s is listed in PACKAGES multiple times, this leads to packaging errors." % pkg)
+            msg = "%s is listed in PACKAGES multiple times, this leads to packaging errors." % pkg
+            package_qa_handle_error("packages-list", msg, d)
         else:
             package_list.append(pkg)
     d.setVar('PACKAGES', ' '.join(package_list))
@@ -945,7 +952,8 @@ python populate_packages () {
 
         filesvar = d.getVar('FILES_%s' % pkg, True) or ""
         if "//" in filesvar:
-            bb.warn("FILES variable for package %s contains '//' which is invalid. Attempting to fix this but you should correct the metadata.\n" % pkg)
+            msg = "FILES variable for package %s contains '//' which is invalid. Attempting to fix this but you should correct the metadata.\n" % pkg
+            package_qa_handle_error("files-invalid", msg, d)
             filesvar.replace("//", "/")
         files = filesvar.split()
         for file in files:
@@ -1024,12 +1032,12 @@ python populate_packages () {
 
     if unshipped != []:
         msg = pn + ": Files/directories were installed but not shipped"
-        if "installed_vs_shipped" in (d.getVar('INSANE_SKIP_' + pn, True) or "").split():
-            bb.note("Package %s skipping QA tests: installed_vs_shipped" % pn)
+        if "installed-vs-shipped" in (d.getVar('INSANE_SKIP_' + pn, True) or "").split():
+            bb.note("Package %s skipping QA tests: installed-vs-shipped" % pn)
         else:
             for f in unshipped:
                 msg = msg + "\n  " + f
-            package_qa_handle_error("installed_vs_shipped", msg, d)
+            package_qa_handle_error("installed-vs-shipped", msg, d)
 }
 populate_packages[dirs] = "${D}"
 
@@ -1319,7 +1327,8 @@ python package_do_shlibs() {
 
     ver = d.getVar('PKGV', True)
     if not ver:
-        bb.error("PKGV not defined")
+        msg = "PKGV not defined"
+        package_qa_handle_error("pkgv-undefined", msg, d)
         return
 
     pkgdest = d.getVar('PKGDEST', True)
@@ -1854,7 +1863,8 @@ python do_package () {
     pn = d.getVar('PN', True)
 
     if not workdir or not outdir or not dest or not dvar or not pn:
-        bb.error("WORKDIR, DEPLOY_DIR, D, PN and PKGD all must be defined, unable to package")
+        msg = "WORKDIR, DEPLOY_DIR, D, PN and PKGD all must be defined, unable to package"
+        package_qa_handle_error("var-undefined", msg, d)
         return
 
     bb.build.exec_func("package_get_auto_pr", d)
Signature · 1 line
1.7.10.4
#5 lore raw
In reply to Robert Yang
Quoted 8 lines · Robert Yang

The following changes since commit c5c1517726aa103a3cdb60abda95e28997cac7c4:

image-vmdk.bbclass: disable the NOISO = "1" (2013-05-12 09:37:21 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/qa
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/qa

Looks good, it probably solves also at least partially:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4085

Quoted 18 lines

Robert Yang (3):
insane/package: let package.bbclass inherit insane.bbclass defaultsetup.conf: remove INHERIT_INSANE insane/package: refactor packaging sanity tests

 meta/classes/insane.bbclass        |   39 ++++++++++++++++++++++------------
 meta/classes/package.bbclass       |   41 ++++++++++++++++++++++++------------
 meta/conf/distro/defaultsetup.conf |    3 +--
 3 files changed, 54 insertions(+), 29 deletions(-)
Signature · 7 lines
1.7.10.4


_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]