sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

PATCH In review Richard Purdie 3 participants 7 messages Nov 16, 2012

Share & tools

Richard Purdie

author

2012-11-16 15:46 UTC #0 lore raw

PATCH 1/1 sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

This is a first attempt at logic to determine when a sstate dependency needs to be installed and when it does not. Its a start at the logic and errs on the side of caution, as it gets wider testing, we can refine the logic as needed.

This code should allow a significant performance speedup to certain workflows, for example "bitbake xxx-image -c rootfs" will not populate the target sysroot.

Signed-off-by Richard Purdie <[email protected]>

---

meta/classes/sstate.bbclass +52 -0

index c085c62..470e0ee 100644
@@ -620,3 +620,55 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     return ret
 
+BB_SETSCENE_DEPVALID = "setscene_depvalid"
+
+def setscene_depvalid(task, taskdependees, notneeded, d):
+    # taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME]
+    # task is included in taskdependees too
+
+    bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task])))
+
+    def isNative(x):
+        return x.endswith("-native")
+    def isNativeCross(x):
+        return x.endswith("-native") or x.endswith("-cross") or x.endswith("-cross-initial")
+    def isSafeDep(x):
+        if x in ["quilt-native", "autoconf-native", "automake-native", "gnu-config-native", "libtool-native", "pkgconfig-native", "gcc-cross", "binutils-cross"]:
+            return True
+        return False
+
+    # We can skip these "safe" dependencies since the aren't runtime dependencies, just build time
+    if isSafeDep(taskdependees[task][0]) and taskdependees[task][1] == "do_populate_sysroot":
+        return True
+
+    # We only need to trigger populate_lic through direct dependencies
+    if taskdependees[task][1] == "do_populate_lic":
+        return True
+
+    for dep in taskdependees:
+        bb.debug(2, "  considering dependency: %s" % (str(taskdependees[dep])))
+        if task == dep:
+            continue
+        if dep in notneeded:
+            continue
+        # do_package_write_* and do_package doesn't need do_package
+        if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+            continue
+        # do_package_write_* and do_package doesn't need do_populate_sysroot
+        if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+            continue
+        # Native/Cross packages don't exist and are noexec anyway
+        if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+            continue
+        # Native/Cross populate_sysroot need their dependencies
+        if isNativeCross(taskdependees[task][0]) and isNativeCross(taskdependees[dep][0]) and taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
+            return False
+        # Target populate_sysroot do not need their dependencies
+        if taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
+            continue
+
+        # Safe fallthrough default
+        bb.debug(2, " Default setscene dependency fall through due to dependency: %s" % (str(taskdependees[dep])))
+        return False
+    return True
+

Otavio Salvador

2012-11-16 16:12 UTC #1 lore raw

Quoted 5 lines · Richard Purdie

This is a first attempt at logic to determine when a sstate dependency needs
to be installed and when it does not. Its a start at the logic and errs on the

s/errs/errors/

Quoted 59 lines

side of caution, as it gets wider testing, we can refine the logic as needed.
This code should allow a significant performance speedup to certain workflows, for
example "bitbake xxx-image -c rootfs" will not populate the target sysroot.
Signed-off-by Richard Purdie <[email protected]>
---
meta/classes/sstate.bbclass +0 -0
index c085c62..470e0ee 100644
@@ -620,3 +620,55 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash,
sq_hashfn, d):
return ret
meta/classes/sstate.bbclass +4 -0
+BB_SETSCENE_DEPVALID = "setscene_depvalid"
+
+def setscene_depvalid(task, taskdependees, notneeded, d):
+    # taskdependees is a dict of tasks which depend on task, each being a
3 item list of [PN, TASKNAME, FILENAME]
meta/classes/sstate.bbclass +3 -0
+    # task is included in taskdependees too
+
+    bb.debug(2, "Considering setscene task: %s" %
(str(taskdependees[task])))
meta/classes/sstate.bbclass +5 -0
+
+    def isNative(x):
+        return x.endswith("-native")
+    def isNativeCross(x):
+        return x.endswith("-native") or x.endswith("-cross") or
x.endswith("-cross-initial")
meta/classes/sstate.bbclass +2 -0
+    def isSafeDep(x):
+        if x in ["quilt-native", "autoconf-native", "automake-native",
"gnu-config-native", "libtool-native", "pkgconfig-native", "gcc-cross", "binutils-cross"]:
meta/classes/sstate.bbclass +4 -0
+            return True
+        return False
+
+    # We can skip these "safe" dependencies since the aren't runtime
dependencies, just build time
+ if isSafeDep(taskdependees[task][0]) and taskdependees[task][1] ==
"do_populate_sysroot":
meta/classes/sstate.bbclass +8 -0
+        return True
+
+    # We only need to trigger populate_lic through direct dependencies
+    if taskdependees[task][1] == "do_populate_lic":
+        return True
+
+    for dep in taskdependees:
+        bb.debug(2, "  considering dependency: %s" %
(str(taskdependees[dep])))
meta/classes/sstate.bbclass +6 -0
+        if task == dep:
+            continue
+        if dep in notneeded:
+            continue
+        # do_package_write_* and do_package doesn't need do_package
+        if taskdependees[task][1] == "do_package" and

Quoted 4 lines

taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+ continue

I understand why you does this but it would be better if we had a central place to have the do_package_write_$pkg recorded or when adding new type this could be forgotten.

Quoted 27 lines

+ # do_package_write_* and do_package doesn't need
do_populate_sysroot
+ if taskdependees[task][1] == "do_populate_sysroot" and
taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
meta/classes/sstate.bbclass +3 -0
+            continue
+        # Native/Cross packages don't exist and are noexec anyway
+        if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1]
in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
meta/classes/sstate.bbclass +3 -0
+            continue
+        # Native/Cross populate_sysroot need their dependencies
+        if isNativeCross(taskdependees[task][0]) and
isNativeCross(taskdependees[dep][0]) and taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
meta/classes/sstate.bbclass +3 -0
+            return False
+        # Target populate_sysroot do not need their dependencies
+        if taskdependees[task][1] == 'do_populate_sysroot' and
taskdependees[dep][1] == 'do_populate_sysroot':
meta/classes/sstate.bbclass +4 -0
+            continue
+
+        # Safe fallthrough default
+        bb.debug(2, " Default setscene dependency fall through due to
dependency: %s" % (str(taskdependees[dep])))
meta/classes/sstate.bbclass +3 -0
+        return False
+    return True
+

Signature · 3 lines

Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Burton, Ross

2012-11-16 16:23 UTC #2 lore raw

Quoted 3 lines · Otavio Salvador

Quoted 1 line
Its a start at the logic and errs on the
s/errs/errors/

http://www.thefreedictionary.com/errs+on+the+side

Ross

Otavio Salvador

2012-11-16 16:35 UTC #3 lore raw

Quoted 7 lines · Ross

Quoted 3 lines · Otavio Salvador
Quoted 1 line
Its a start at the logic and errs on the
s/errs/errors/
http://www.thefreedictionary.com/errs+on+the+side

Cool!

Signature · 3 lines

Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Richard Purdie

author

2012-11-16 16:26 UTC #4 lore raw

In reply to Otavio Salvador

Quoted 8 lines · Otavio Salvador

On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie <[email protected]> wrote:
This is a first attempt at logic to determine when a sstate dependency needs
to be installed and when it does not. Its a start at the logic and errs on the
s/errs/errors/

It is actually meant to be errs :)

Quoted 11 lines

+ # do_package_write_* and do_package doesn't need
do_package
+ if taskdependees[task][1] == "do_package" and
taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+ continue
I understand why you does this but it would be better if we had a central place to have the do_package_write_$pkg recorded or when adding new type this could be forgotten.

Its a nice idea but I can't see any practical way to do this at the moment.

Cheers,

Richard

Otavio Salvador

2012-11-16 16:36 UTC #5 lore raw

Quoted 20 lines · Richard Purdie

Quoted 8 lines · Otavio Salvador
On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie <[email protected]> wrote:
This is a first attempt at logic to determine when a sstate dependency needs
to be installed and when it does not. Its a start at the logic and errs on the
s/errs/errors/
It is actually meant to be errs :)
Quoted 8 lines
+ # do_package_write_* and do_package doesn't need
do_package
+ if taskdependees[task][1] == "do_package" and
taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+ continue

Quoted 7 lines

Quoted 3 lines
I understand why you does this but it would be better if we had a central place to have the do_package_write_$pkg recorded or when adding new type this could be forgotten.
Its a nice idea but I can't see any practical way to do this at the moment.

We might hava a variable that provide all package backends names? Like:

PACKAGE_BACKENDS = "ipk deb rpm"

Signature · 3 lines

Otavio Salvador                             O.S. Systems
E-mail: [email protected]  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Richard Purdie

author

2012-11-16 16:45 UTC #6 lore raw

Quoted 17 lines · Otavio Salvador

On Fri, Nov 16, 2012 at 2:12 PM, Richard Purdie
> + # do_package_write_* and do_package doesn't
need
> do_package
> + if taskdependees[task][1] == "do_package"
and
> taskdependees[dep][1] in ['do_package',
> 'do_package_write_deb', 'do_package_write_ipk',
> 'do_package_write_rpm']:
> + continue
>
> I understand why you does this but it would be better if we had a
> central place to have the do_package_write_$pkg recorded or when
> adding new type this could be forgotten.

Quoted 8 lines

Its a nice idea but I can't see any practical way to do this at the
moment.
We might hava a variable that provide all package backends names? Like:
PACKAGE_BACKENDS = "ipk deb rpm"

That works for some usages but not where the expressions are in calls to things like addtask.

I think there are other problems worth spending time on rather that this which at best will probably cover 50% of the usages.

Cheers,

Richard