fix archiver missing tasks between do_unpack and do_patch
cover fix archiver missing tasks between do_unpack and do_patch
The following changes since commit b8631416f12b8a904ce3deb036f9d5ce632937b0:
package_regex.inc: Update REGEX'es in order to find upstream versions correctly (2014-11-28 14:03:02 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/fix-archiver
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-archiver
Hongxu Jia (1):
archiver: execute the probable tasks between do_unpack and do_patch
meta/classes/archiver.bbclass | 81 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 74 insertions(+), 7 deletions(-)
Signature · 1 line
1.9.1
PATCH 1/1 archiver: execute the probable tasks between do_unpack and do_patch
While archiver inherited, we edit a recipe (such as gzip) to insert four tasks between do_patch and do_unpack:
...
addtask test1 after do_unpack before do_patch addtask test2 after do_unpack before do_test1 addtask test3 after do_test2 before do_test1 addtask test4 after do_test2 before do_test1
...
While building the recipe, the archiver will missing these four task in do_unpack_and_patch. Because it is hardcoded to execute do_unpach and do_patch, did not consider the probable tasks between them.
We make use of the value of _task_deps which provided by metadata to compute the probable tasks between do_unpack and do_patch and execute them.
[Yocto #7018]
--- meta/classes/archiver.bbclass | 81 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 74 insertions(+), 7 deletions(-)
meta/classes/archiver.bbclass +74 -7
@@ -269,6 +269,75 @@ def create_diff_gz(d, src_orig, src, ar_outdir): subprocess.call(diff_cmd, shell=True) bb.utils.remove(src_patched, recurse=True) +# It helps to compute task dependencies between begin and end. +# Here is an example, in a recipe we insert four tasks between do_patch +# and do_unpack: +# addtask test1 after do_unpack before do_patch +# addtask test2 after do_unpack before do_test1 +# addtask test3 after do_test2 before do_test1 +# addtask test4 after do_test2 before do_test1 +# which equals in metadata: +# _task_deps = { +# 'parents': { +# 'do_patch': ['do_test1', 'do_unpack'], +# 'do_test1': ['do_test4', 'do_test3', 'do_test2', 'do_unpack'] +# 'do_test2': ['do_unpack'], +# 'do_test3': ['do_test2'], +# 'do_test4': ['do_test2'] +# } +# } +# +# We want to list the order of task dependencies: +# ['do_unpack', 'do_test2', 'do_test3', 'do_test4', 'do_test1', 'do_patch'] +def list_sorted_tasks(begin, end, d): + task_deps = d.getVar('_task_deps', True) + parent_graph = construct_parent_graph(begin, end, d) + + # Sort tasks according to the number of parent tasks incrementally + return [x[0] for x in sorted(parent_graph.iteritems(), key = lambda d:len(d[1]))] + +# List all parents for each tasks, such as: +# From +# _task_deps = { +# 'parents': { +# 'do_patch': ['do_test1', 'do_unpack'], +# 'do_test1': ['do_test4', 'do_test3', 'do_test2', 'do_unpack'] +# 'do_test2': ['do_unpack'], +# 'do_test3': ['do_test2'], +# 'do_test4': ['do_test2'] +# } +# } +# to the parent graph we counstruct: +# parent_graph = { +# 'do_patch': ['do_test1', 'do_test2', 'do_test3', 'do_test4', 'do_unpack'], +# 'do_test1': ['do_test4', 'do_test3', 'do_test2', 'do_unpack'], +# 'do_test2': ['do_unpack'], +# 'do_test3': ['do_test2', 'do_unpack'], +# 'do_test4': ['do_test2', 'do_unpack'], +# 'do_unpack': [] +# } +# +# We are not care about circualar dependency, the bitbake parser will do the +# checking. +def construct_parent_graph(begin, end, d): + task_deps = d.getVar('_task_deps', True) + def list_parents(task): + if task == begin: + return [] + parents = task_deps['parents'][task] + for ptask in task_deps['parents'][task]: + parents += [x for x in list_parents(ptask) if x not in parents] + return parents + + parent_graph = dict() + # All the tasks we need listed in the end's parent + end_parent = list_parents(end) + parent_graph[end] = end_parent + for task in end_parent: + parent_graph[task] = list_parents(task) + + return parent_graph + # Run do_unpack and do_patch python do_unpack_and_patch() { if d.getVarFlag('ARCHIVER_MODE', 'src', True) not in \
@@ -286,13 +355,11 @@ python do_unpack_and_patch() { # do_patch required 'B' existed). bb.utils.mkdirhier(d.getVar('B', True)) - # The kernel source is ready after do_validate_branches - if bb.data.inherits_class('kernel-yocto', d): - bb.build.exec_func('do_unpack', d) - bb.build.exec_func('do_kernel_checkout', d) - bb.build.exec_func('do_validate_branches', d) - else: - bb.build.exec_func('do_unpack', d) + tasks = list_sorted_tasks('do_unpack', 'do_patch', d) + bb.note('execute %s in do_unpack_and_patch' % tasks) + # Do not execute 'do_patch' here + for task in tasks[0:-1]: + bb.build.exec_func(task, d) # Save the original source for creating the patches if d.getVarFlag('ARCHIVER_MODE', 'diff', True) == '1':
Signature · 1 line
1.9.1
Re: [PATCH 1/1] archiver: execute the probable tasks between do_unpack and do_patch
Quoted 16 lines · Hongxu Jia
While archiver inherited, we edit a recipe (such as gzip) to insert four tasks between
do_patchanddo_unpack:
...
addtask test1 afterdo_unpackbeforedo_patchaddtask test2 afterdo_unpackbeforedo_test1addtask test3 afterdo_test2beforedo_test1addtask test4 afterdo_test2beforedo_test1
...While building the recipe, the archiver will missing these four task in
do_unpack_and_patch. Because it is hardcoded to executedo_unpachanddo_patch, did not consider the probable tasks between them.
Quoted 4 lines
We make use of the value of
_task_depswhich provided by metadata to compute the probable tasks betweendo_unpackanddo_patchand execute them.
_task_deps is not provided by the metadata, its a bitbake internal variable and it should not be being used in public API like this. The bitbake maintainers reserve the right to change structure and existence of _task_deps freely. So this patch *cannot* merge in this form.
I do appreciate the problem you're trying to solve and that its a hard one to deal with.
We should probably add some API to bb.build which allows the code to query information about tasks. I would prefer to only expose the information we need (the parents) rather than the whole contents of _task_deps since that may change over time. The main reason that structure exists at all is a performance optimisation inside the core of bitbake.
Thinking about this further, have you looked at BB_TASKDEPDATA? You may well be able to extract the information you need from that and that *is* public API.
Cheers,
Richard