Skip to contentSkip to conversation

fixes for package_qa_walk() and buildpaths

PATCH In review Robert Yang 4 participants 18 messages Oct 8, 2015 – Oct 13, 2015

lore
Share & tools
lore
#0 lore raw

cover fixes for package_qa_walk() and buildpaths

The following changes since commit 4038970f8ce27ac0d7a0afe2cdaa9a65108dfff5:

os-release: fix do_compile() when RPM signing is enabled (2015-10-07 00:08:07 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/buildpaths
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/buildpaths

Robert Yang (8):
insane.bbclass: do_package_qa depends on WARN_QA and ERROR_QA insane.bbclass: make package_qa_walk() can print all the messages insane.bbclass: buildpaths: ignore .pyc files insane.bbclass: skip buildpath qa check when set
INHIBIT_PACKAGE_STRIP
insane.bbclass: fix for ignoring debug files package.bbclass: strip archive files autotools.bbclass: use relative path to run configure script glibc-initial.inc: use relative path to run configure

 meta/classes/autotools.bbclass            |    4 +++-
 meta/classes/insane.bbclass               |   32 ++++++++++++++++++++++++-----
 meta/classes/package.bbclass              |   18 +++++++++++++++-
 meta/recipes-core/glibc/glibc-initial.inc |    2 +-
 4 files changed, 48 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/8 insane.bbclass: do_package_qa depends on WARN_QA and ERROR_QA

do_package_qa() should rerun when WARN_QA or ERROR_QA changed

[YOCTO #8445]

---
 meta/classes/insane.bbclass |    1 +
 1 file changed, 1 insertion(+)
meta/classes/insane.bbclass +1 -0
index 05e31a7..61db3e1 100644
@@ -1113,6 +1113,7 @@ python do_package_qa () {
 }
 
 do_package_qa[rdeptask] = "do_packagedata"
+do_package_qa[vardeps] += "${WARN_QA} ${ERROR_QA}"
 addtask do_package_qa after do_packagedata do_package before do_build
 
 SSTATETASKS += "do_package_qa"
Signature · 1 line
1.7.9.5
#2 lore raw

PATCH 2/8 insane.bbclass: make package_qa_walk() can print all the messages

In reply to Robert Yang

* If more than one files have the same QA issue, package_qa_walk() can only print the last one, others are overrided, for example:
messages["host-user-contaminated"] = "foo1" messages["host-user-contaminated"] = "foo2"
Only foo2 will be printed, this patch fixes the issue.

* Remove unused parameter "path" from package_qa_walk()
The path is a useless parameter in package_qa_walk() since it has been redined inside.

[YOCTO #8436]

---
 meta/classes/insane.bbclass |   20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
meta/classes/insane.bbclass +17 -3
index 61db3e1..e3a8cd7 100644
@@ -749,8 +749,15 @@ def package_qa_check_staged(path,d):
 
     return sane
 
+# Merge src dict's content into dest
+def package_qa_merge_dict(dest, src):
+    if len(dest) == len(src) and dest != src:
+        for w in src:
+            if dest[w] != src[w]:
+                dest[w] = src[w] + '\n' + dest[w]
+
 # Walk over all files in a directory and call func
-def package_qa_walk(path, warnfuncs, errorfuncs, skip, package, d):
+def package_qa_walk(warnfuncs, errorfuncs, skip, package, d):
     import oe.qa
 
     #if this will throw an exception, then fix the dict above
@@ -766,9 +773,17 @@ def package_qa_walk(path, warnfuncs, errorfuncs, skip, package, d):
             except:
                 elf = None
             for func in warnfuncs:
+                warnings_orig = warnings.copy()
                 func(path, package, d, elf, warnings)
+                # warnings[foo] = "foo1" might be overrided by
+                # warnings [foo] = "foo2", check and merge.
+                package_qa_merge_dict(warnings, warnings_orig)
             for func in errorfuncs:
+                errors_orig = errors.copy()
                 func(path, package, d, elf, errors)
+                # errors[foo] = "foo1" might be overrided by
+                # errors [foo] = "foo2", check and merge.
+                package_qa_merge_dict(errors, errors_orig)
 
     for w in warnings:
         package_qa_handle_error(w, warnings[w], d)
@@ -1094,8 +1109,7 @@ python do_package_qa () {
             package_qa_handle_error("pkgname",
                     "%s doesn't match the [a-z0-9.+-]+ regex" % package, d)
 
-        path = "%s/%s" % (pkgdest, package)
-        if not package_qa_walk(path, warnchecks, errorchecks, skip, package, d):
+        if not package_qa_walk(warnchecks, errorchecks, skip, package, d):
             walk_sane  = False
         if not package_qa_check_rdepends(package, pkgdest, skip, taskdeps, packages, d):
             rdepends_sane = False
Signature · 1 line
1.7.9.5
#3 lore raw

PATCH 3/8 insane.bbclass: buildpaths: ignore .pyc files

In reply to Robert Yang

The .pyc files usually contain build path like:

i586-poky-linux-gcc -m32 -march=i586 --sysroot=/buildarea/lyang1/test_p/tmp/sysroots/qemux86 And:
_PYTHON_PROJECT_BASE=/buildarea/lyang1/test_p/tmp/work/i586-poky-linux/python/2.7.9-r1/build

They are safe, so ignore them.

This can reduce 1065 QA issues for buildpaths in a world build.

[YOCTO #8446]

---
 meta/classes/insane.bbclass |    4 ++++
 1 file changed, 4 insertions(+)
meta/classes/insane.bbclass +4 -0
index e3a8cd7..664dfec 100644
@@ -576,6 +576,10 @@ def package_qa_check_buildpaths(path, name, d, elf, messages):
     if path.find(".debug") != -1:
         return
 
+    # Ignore .pyc files, not interesting
+    if path.endswith(".pyc"):
+        return
+
     # Ignore symlinks
     if os.path.islink(path):
         return
Signature · 1 line
1.7.9.5
#4 lore raw

PATCH 4/8 insane.bbclass: skip buildpath qa check when set INHIBIT_PACKAGE_STRIP

In reply to Robert Yang

There are build paths in the file when set INHIBIT_PACKAGE_STRIP.

[YOCTO #8458]

---
 meta/classes/insane.bbclass |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
meta/classes/insane.bbclass +5 -2
index 664dfec..d6f7388 100644
@@ -572,8 +572,11 @@ def package_qa_check_buildpaths(path, name, d, elf, messages):
     """
     Check for build paths inside target files and error if not found in the whitelist
     """
-    # Ignore .debug files, not interesting
-    if path.find(".debug") != -1:
+    # Ignore:
+    # - .debug files
+    # - which sets INHIBIT_PACKAGE_STRIP
+    if path.find(".debug") != -1 or \
+        d.getVar('INHIBIT_PACKAGE_STRIP', True) == "1":
         return
 
     # Ignore .pyc files, not interesting
Signature · 1 line
1.7.9.5
#5 lore raw

PATCH 5/8 insane.bbclass: fix for ignoring debug files

In reply to Robert Yang

Some recipes put debug files in "usr/src/debug/" rather than ".debug".

---
 meta/classes/insane.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/insane.bbclass +1 -1
index d6f7388..947d7ac 100644
@@ -575,7 +575,7 @@ def package_qa_check_buildpaths(path, name, d, elf, messages):
     # Ignore:
     # - .debug files
     # - which sets INHIBIT_PACKAGE_STRIP
-    if path.find(".debug") != -1 or \
+    if path.find(".debug") != -1 or path.find("/src/debug/") != -1 or \
         d.getVar('INHIBIT_PACKAGE_STRIP', True) == "1":
         return
 
Signature · 1 line
1.7.9.5
#6 lore raw

PATCH 6/8 package.bbclass: strip archive files

In reply to Robert Yang

For saving space and removing build path in .a files.

[YOCTO #8457]

---
 meta/classes/package.bbclass |   18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +17 -1
index 09230b5..583848f 100644
@@ -870,12 +870,25 @@ python split_and_strip_files () {
         return type
 
 
+    def isArchive(path):
+        ret, result = oe.utils.getstatusoutput("file \"%s\"" % path.replace("\"", "\\\""))
+        if ret:
+            msg = "split_and_strip_files: 'file %s' failed" % path
+            package_qa_handle_error("split-strip", msg, d)
+            return False
+
+        if "current ar archive" in result:
+            return True
+        else:
+            return False
+
     #
     # First lets figure out all of the files we may have to process ... do this only once!
     #
     elffiles = {}
     symlinks = {}
     kernmods = []
+    archives = []
     inodes = {}
     libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir", True))
     baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir", True))
@@ -886,6 +899,9 @@ python split_and_strip_files () {
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
                     continue
+                if file.endswith('.a') and isArchive(file):
+                    archives.append(file)
+                    continue
 
                 # Skip debug files
                 if debugappend and file.endswith(debugappend):
@@ -1019,7 +1035,7 @@ python split_and_strip_files () {
             elf_file = int(elffiles[file])
             #bb.note("Strip %s" % file)
             sfiles.append((file, elf_file, strip))
-        for f in kernmods:
+        for f in kernmods + archives:
             sfiles.append((f, 16, strip))
 
         oe.utils.multiprocess_exec(sfiles, oe.package.runstrip)
Signature · 1 line
1.7.9.5
#7 lore raw

PATCH 7/8 autotools.bbclass: use relative path to run configure script

In reply to Robert Yang

The absolute path (/path/to/configure) caused VPATH in Makefile to be an absolute path, and then it will be in elf files, use relative path to run configure can fix the problem.

This will reduce 3127 buildpaths QA issues in a world buld (8957 -> 5830).

[YOCTO #8437]

---
 meta/classes/autotools.bbclass |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes/autotools.bbclass +3 -1
index 078f58f..c40e5f4 100644
@@ -79,7 +79,9 @@ CONFIGUREOPT_DEPTRACK ?= "--disable-dependency-tracking"
 
 
 oe_runconf () {
-	cfgscript="${S}/configure"
+	# Use relative path for S to avoid buildpaths in files
+	relative_s=`python -c "import os; print os.path.relpath('${S}', '.')"`
+	cfgscript="$relative_s/configure"
 	if [ -x "$cfgscript" ] ; then
 		bbnote "Running $cfgscript ${CONFIGUREOPTS} ${EXTRA_OECONF} $@"
 		set +e
Signature · 1 line
1.7.9.5
#8 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 7 lines · Robert Yang

oe_runconf () {

meta/classes/autotools.bbclass +2 -1
-       cfgscript="${S}/configure"
+       # Use relative path for S to avoid buildpaths in files
+       relative_s=`python -c "import os; print os.path.relpath('${S}',

'.')"`
+ cfgscript="$relative_s/configure"

Why not just use inline python here, rather than calling out to an external python process at task execution time?

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#9 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 11 lines · Christopher Larson
Quoted 7 lines · Robert Yang

oe_runconf () {

meta/classes/autotools.bbclass +2 -1
-       cfgscript="${S}/configure"
+       # Use relative path for S to avoid buildpaths in files
+       relative_s=`python -c "import os; print os.path.relpath('${S}',

'.')"`
+ cfgscript="$relative_s/configure"

Quoted 2 lines

Why not just use inline python here, rather than calling out to an external python process at task execution time?

Oh, nevermind, of course oe_runconf can be run from any number of paths :)

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#10 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 11 lines · Christopher Larson

+ # Use relative path for S to avoid buildpaths in files

Quoted 7 lines
Quoted 4 lines

+ relative_s=`python -c "import os; print os.path.relpath('${S}',
'.')"`
+ cfgscript="$relative_s/configure"

Why not just use inline python here, rather than calling out to an external python process at task execution time?

Oh, nevermind, of course oe_runconf can be run from any number of paths :)

Why not use realpath?

ross@flashheart ~/Yocto/poky/meta/recipes-devtools/qemu (mut)

$ realpath --relative-to=. ~/Yocto/poky/meta/recipes-support/mpfr/
../../recipes-support/mpfr

Ross

#11 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Hi Chris and Ross,

Quoted 10 lines · Ross

On 8 October 2015 at 22:00, Christopher Larson <[email protected] <mailto:[email protected]>> wrote:

+ # Use relative path for S to avoid buildpaths in files
+ relative_s=`python -c "import os; print
os.path.relpath('${S}', '.')"`
+ cfgscript="$relative_s/configure"

Quoted 5 lines

Why not just use inline python here, rather than calling out to an external python process at task execution time?

Oh, nevermind, of course oe_runconf can be run from any number of paths :)

Yes, "inline python" is executed at parsing time, but we need it be a building time path.

Quoted 7 lines

Why not use realpath?

ross@flashheart ~/Yocto/poky/meta/recipes-devtools/qemu (mut)

$ realpath --relative-to=. ~/Yocto/poky/meta/recipes-support/mpfr/
../../recipes-support/mpfr

I'm afraid that realpath is not generally existed on the build host ?

// Robert

Quoted 2 lines

Ross

#12 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 2 lines · Robert Yang

I'm afraid that realpath is not generally existed on the build host ?

It's part of coreutils on my Debian stable box. I guess the real question is "does it exist on Centos 6?". [looks]. Hmm, it's not present on that.

Ross

#13 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 9 lines · Ross

On 9 October 2015 at 02:28, Robert Yang <[email protected] <mailto:[email protected]>> wrote:

I'm afraid that realpath is not generally existed on the build host ?

It's part of coreutils on my Debian stable box. I guess the real question is "does it exist on Centos 6?". [looks]. Hmm, it's not present on that.

Ubuntu 14.03 desktop doesn't have it, either.

// Robert

Quoted 2 lines

Ross

#14 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 10 lines · On Friday 09 October 2015 19:30:07 Robert Yang wrote:
Quoted 8 lines · Ross

On 9 October 2015 at 02:28, Robert Yang <[email protected]

<mailto:[email protected]>> wrote:
I'm afraid that realpath is not generally existed on the build host ?

It's part of coreutils on my Debian stable box. I guess the real question is "does it exist on Centos 6?". [looks]. Hmm, it's not present on that.

Ubuntu 14.03 desktop doesn't have it, either.

Does readlink -f suffice ?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#15 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 7 lines · Paul Eggleton
Quoted 6 lines · On Friday 09 October 2015 19:30:07 Robert Yang wrote:
Quoted 5 lines · Ross

On 9 October 2015 at 02:28, Robert Yang <[email protected]

<mailto:[email protected]>> wrote:
I'm afraid that realpath is not generally existed on the build host ?

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

It's part of coreutils on my Debian stable box. I guess the real question is "does it exist on Centos 6?". [looks]. Hmm, it's not present on that.

Ubuntu 14.03 desktop doesn't have it, either.

Does readlink -f suffice ?

Hi Paul,

We deed change an absolute path to relative path here, readlink -f does the opposite ?

// Robert

Quoted 4 lines

Cheers,
Paul

#16 lore raw

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

Quoted 14 lines · On Tuesday 13 October 2015 15:17:45 Robert Yang wrote:
Quoted 13 lines · Paul Eggleton
Quoted 12 lines · On Friday 09 October 2015 19:30:07 Robert Yang wrote:
Quoted 10 lines · Ross

On 9 October 2015 at 02:28, Robert Yang <[email protected]

<mailto:[email protected]>> wrote:
I'm afraid that realpath is not generally existed on the build host ?

It's part of coreutils on my Debian stable box. I guess the real question
is "does it exist on Centos 6?". [looks]. Hmm, it's not present on that.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Ubuntu 14.03 desktop doesn't have it, either.

Does readlink -f suffice ?

Hi Paul,

We deed change an absolute path to relative path here, readlink -f does the opposite ?

Ah, my misunderstanding - sorry for the noise.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#17 lore raw

PATCH 8/8 glibc-initial.inc: use relative path to run configure

In reply to Robert Yang

Avoid potential build path in output files.

---
 meta/recipes-core/glibc/glibc-initial.inc |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/glibc/glibc-initial.inc +1 -1
index 3500848..a09e8a9 100644
@@ -11,7 +11,7 @@ TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TCBOOTSTRAP}"
 do_configure () {
 	(cd ${S} && gnu-configize) || die "failure in running gnu-configize"
 	find ${S} -name "configure" | xargs touch
-	${S}/configure --host=${TARGET_SYS} --build=${BUILD_SYS} \
+	../git/configure --host=${TARGET_SYS} --build=${BUILD_SYS} \
 		--prefix=/usr \
 		--without-cvs --disable-sanity-checks \
 		--with-headers=${STAGING_DIR_TARGET}${includedir} \
Signature · 1 line
1.7.9.5