Skip to contentSkip to conversation

fix archiver missing tasks between do_unpack and do_patch

PATCH Superseded Hongxu Jia 2 participants 3 messages Dec 3, 2014 – Dec 4, 2014

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Hongxu Jia

author
#0 lore raw

cover fix archiver missing tasks between do_unpack and do_patch

The following changes since commit b8631416f12b8a904ce3deb036f9d5ce632937b0:

package_regex.inc: Update REGEX'es in order to find upstream versions correctly (2014-11-28 14:03:02 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/fix-archiver
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-archiver

Hongxu Jia (1):
archiver: execute the probable tasks between do_unpack and do_patch

 meta/classes/archiver.bbclass | 81 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 74 insertions(+), 7 deletions(-)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH 1/1 archiver: execute the probable tasks between do_unpack and do_patch

While archiver inherited, we edit a recipe (such as gzip) to insert four tasks between do_patch and do_unpack:
...
addtask test1 after do_unpack before do_patch addtask test2 after do_unpack before do_test1 addtask test3 after do_test2 before do_test1 addtask test4 after do_test2 before do_test1
...

While building the recipe, the archiver will missing these four task in do_unpack_and_patch. Because it is hardcoded to execute do_unpach and do_patch, did not consider the probable tasks between them.

We make use of the value of _task_deps which provided by metadata to compute the probable tasks between do_unpack and do_patch and execute them.

[Yocto #7018]

---
 meta/classes/archiver.bbclass | 81 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 74 insertions(+), 7 deletions(-)
meta/classes/archiver.bbclass +74 -7
index b598aa3..6672204 100644
@@ -269,6 +269,75 @@ def create_diff_gz(d, src_orig, src, ar_outdir):
     subprocess.call(diff_cmd, shell=True)
     bb.utils.remove(src_patched, recurse=True)
 
+# It helps to compute task dependencies between begin and end.
+# Here is an example, in a recipe we insert four tasks between do_patch
+# and do_unpack:
+# addtask test1 after do_unpack before do_patch
+# addtask test2 after do_unpack before do_test1
+# addtask test3 after do_test2 before do_test1
+# addtask test4 after do_test2 before do_test1
+# which equals in metadata:
+# _task_deps = {
+#   'parents': {
+#     'do_patch': ['do_test1', 'do_unpack'],
+#     'do_test1': ['do_test4', 'do_test3', 'do_test2', 'do_unpack']
+#     'do_test2': ['do_unpack'],
+#     'do_test3': ['do_test2'],
+#     'do_test4': ['do_test2']
+#   }
+# }
+#
+# We want to list the order of task dependencies:
+# ['do_unpack', 'do_test2', 'do_test3', 'do_test4', 'do_test1', 'do_patch']
+def list_sorted_tasks(begin, end, d):
+    task_deps = d.getVar('_task_deps', True)
+    parent_graph = construct_parent_graph(begin, end, d)
+
+    # Sort tasks according to the number of parent tasks incrementally
+    return [x[0] for x in sorted(parent_graph.iteritems(), key = lambda d:len(d[1]))]
+
+# List all parents for each tasks, such as:
+# From
+# _task_deps = {
+#   'parents': {
+#     'do_patch': ['do_test1', 'do_unpack'],
+#     'do_test1': ['do_test4', 'do_test3', 'do_test2', 'do_unpack']
+#     'do_test2': ['do_unpack'],
+#     'do_test3': ['do_test2'],
+#     'do_test4': ['do_test2']
+#   }
+# }
+# to the parent graph we counstruct:
+# parent_graph = {
+#   'do_patch': ['do_test1', 'do_test2', 'do_test3', 'do_test4', 'do_unpack'],
+#   'do_test1': ['do_test4', 'do_test3', 'do_test2', 'do_unpack'],
+#   'do_test2': ['do_unpack'],
+#   'do_test3': ['do_test2', 'do_unpack'],
+#   'do_test4': ['do_test2', 'do_unpack'],
+#   'do_unpack': []
+# }
+#
+# We are not care about circualar dependency, the bitbake parser will do the
+# checking.
+def construct_parent_graph(begin, end, d):
+    task_deps = d.getVar('_task_deps', True)
+    def list_parents(task):
+        if task == begin:
+            return []
+        parents = task_deps['parents'][task]
+        for ptask in task_deps['parents'][task]:
+            parents += [x for x in list_parents(ptask) if x not in parents]
+        return parents
+
+    parent_graph = dict()
+    # All the tasks we need listed in the end's parent
+    end_parent = list_parents(end)
+    parent_graph[end] = end_parent
+    for task in end_parent:
+        parent_graph[task] = list_parents(task)
+
+    return parent_graph
+
 # Run do_unpack and do_patch
 python do_unpack_and_patch() {
     if d.getVarFlag('ARCHIVER_MODE', 'src', True) not in \
@@ -286,13 +355,11 @@ python do_unpack_and_patch() {
     # do_patch required 'B' existed).
     bb.utils.mkdirhier(d.getVar('B', True))
 
-    # The kernel source is ready after do_validate_branches
-    if bb.data.inherits_class('kernel-yocto', d):
-        bb.build.exec_func('do_unpack', d)
-        bb.build.exec_func('do_kernel_checkout', d)
-        bb.build.exec_func('do_validate_branches', d)
-    else:
-        bb.build.exec_func('do_unpack', d)
+    tasks = list_sorted_tasks('do_unpack', 'do_patch', d)
+    bb.note('execute %s in do_unpack_and_patch' % tasks)
+    # Do not execute 'do_patch' here
+    for task in tasks[0:-1]:
+        bb.build.exec_func(task, d)
 
     # Save the original source for creating the patches
     if d.getVarFlag('ARCHIVER_MODE', 'diff', True) == '1':
Signature · 1 line
1.9.1
#2 lore raw

Re: [PATCH 1/1] archiver: execute the probable tasks between do_unpack and do_patch

Quoted 16 lines · Hongxu Jia

While archiver inherited, we edit a recipe (such as gzip) to insert four tasks between do_patch and do_unpack:
...
addtask test1 after do_unpack before do_patch addtask test2 after do_unpack before do_test1 addtask test3 after do_test2 before do_test1 addtask test4 after do_test2 before do_test1
...

While building the recipe, the archiver will missing these four task in do_unpack_and_patch. Because it is hardcoded to execute do_unpach and do_patch, did not consider the probable tasks between them.

Quoted 4 lines

We make use of the value of _task_deps which provided by metadata to compute the probable tasks between do_unpack and do_patch and execute them.

_task_deps is not provided by the metadata, its a bitbake internal variable and it should not be being used in public API like this. The bitbake maintainers reserve the right to change structure and existence of _task_deps freely. So this patch *cannot* merge in this form.

I do appreciate the problem you're trying to solve and that its a hard one to deal with.

We should probably add some API to bb.build which allows the code to query information about tasks. I would prefer to only expose the information we need (the parents) rather than the whole contents of _task_deps since that may change over time. The main reason that structure exists at all is a performance optimisation inside the core of bitbake.

Thinking about this further, have you looked at BB_TASKDEPDATA? You may well be able to extract the information you need from that and that *is* public API.

Cheers,

Richard