update.py: several fixes
cover update.py: several fixes
The following changes since commit 5cfdfdca8b9200b5a6e4aa14661b14059a799310:
rrs_upstream_history.py: fix set_regexes function (2018-06-06 11:25:46 +1200)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/li_recs
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/li_recs
Robert Yang (4):
update_layer.py: avoid calling setup_core_layer_sys_path() when
--initial
utils.py: fix checkout_repo when no HEAD
update.py: add layers when RECOMMENDS isn't satisfied update.py: check whether branch existed when nocheckout
layerindex/update.py | 71 +++++++++++++++++++++++++++++++--------------- layerindex/update_layer.py | 22 +++++++------- layerindex/utils.py | 25 ++++++++++------ 3 files changed, 76 insertions(+), 42 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/4 update_layer.py: avoid calling setup_core_layer_sys_path() when --initial
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]This is because core_layerbranch is not in database yet for completely new branch, so it is None and we will get the error. Avoid calling setup_core_layer_sys_path() when "update_layer.py --initial" will fix the problem.
And also only add core layer's sys path when it is present, since core layer may not be added yet for completely new branch.
--- layerindex/update_layer.py | 19 ++++++++++--------- layerindex/utils.py | 9 +++++---- 2 files changed, 15 insertions(+), 13 deletions(-)
layerindex/update_layer.py +10 -9
@@ -426,15 +426,6 @@ def main(): # why won't they just fix that?!) tinfoil.config_data.setVar('LICENSE', '') - # Set up for recording patch info - utils.setup_core_layer_sys_path(settings, branch.name) - skip_patches = False - try: - import oe.recipeutils - except ImportError: - logger.warn('Failed to find lib/oe/recipeutils.py in layers - patch information will not be collected') - skip_patches = True - layerconfparser = layerconfparse.LayerConfParse(logger=logger, tinfoil=tinfoil) layer_config_data = layerconfparser.parse_layer(layerdir) if not layer_config_data:
@@ -447,6 +438,16 @@ def main(): for i in ["BBFILE_COLLECTIONS", "LAYERVERSION", "LAYERDEPENDS", "LAYERRECOMMENDS"]: print('%s = "%s"' % (i, utils.get_layer_var(layer_config_data, i, logger))) sys.exit(0) + + # Set up for recording patch info + utils.setup_core_layer_sys_path(settings, branch.name) + skip_patches = False + try: + import oe.recipeutils + except ImportError: + logger.warn('Failed to find lib/oe/recipeutils.py in layers - patch information will not be collected') + skip_patches = True + utils.add_dependencies(layerbranch, layer_config_data, logger=logger) utils.add_recommends(layerbranch, layer_config_data, logger=logger) layerbranch.save()
layerindex/utils.py +5 -4
@@ -371,10 +371,11 @@ def setup_core_layer_sys_path(settings, branchname): """ core_layer = get_layer(settings.CORE_LAYER_NAME) core_layerbranch = core_layer.get_layerbranch(branchname) - core_urldir = core_layer.get_fetch_dir() - core_repodir = os.path.join(settings.LAYER_FETCH_DIR, core_urldir) - core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir) - sys.path.insert(0, os.path.join(core_layerdir, 'lib')) + if core_layerbranch: + core_urldir = core_layer.get_fetch_dir() + core_repodir = os.path.join(settings.LAYER_FETCH_DIR, core_urldir) + core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir) + sys.path.insert(0, os.path.join(core_layerdir, 'lib')) def run_command_interruptible(cmd): """
Signature · 1 line
2.7.4
PATCH 2/4 utils.py: fix checkout_repo when no HEAD
In reply to Robert YangFixed:
$ git clone <url> warning: remote HEAD refers to nonexistent ref, unable to checkout. $ git rev-parse HEAD HEAD fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]'
Catch the error and avoid that.
And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is default for git reset, so they are the same, but the later one reports error when remote HEAD doesn't exist:
$ git reset --hard HEAD fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree. [snip]
$ git reset --hard No errors.
--- layerindex/utils.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
layerindex/utils.py +12 -4
@@ -217,17 +217,25 @@ def checkout_repo(repodir, commit, logger, force=False): if force: currentref = '' else: - currentref = runcmd("git rev-parse HEAD", repodir, logger=logger).strip() + try: + # The "git rev-parse HEAD" returns "fatal: ambiguous argument 'HEAD'" + # when a repo is unable to check out after git clone: + # git clone <url> + # warning: remote HEAD refers to nonexistent ref, unable to checkout. + # So check and avoid that + currentref = runcmd("git rev-parse HEAD", repodir, logger=logger).strip() + except Exception as esc: + logger.warn(esc) + currentref = '' if currentref != commit: # Reset in case there are added but uncommitted changes - runcmd("git reset --hard HEAD", repodir, logger=logger) + runcmd("git reset --hard", repodir, logger=logger) # Drop any untracked files in case these cause problems (either because # they will exist in the revision we're checking out, or will otherwise # interfere with operation, e.g. stale pyc files) runcmd("git clean -qdfx", repodir, logger=logger) # Now check out the revision - runcmd("git checkout %s" % commit, - repodir, logger=logger) + runcmd("git checkout %s" % commit, repodir, logger=logger) def checkout_layer_branch(layerbranch, repodir, logger=None): branchname = layerbranch.branch.name
Signature · 1 line
2.7.4
PATCH 3/4 update.py: add layers when RECOMMENDS isn't satisfied
In reply to Robert YangWhen layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
--- layerindex/update.py | 67 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 46 insertions(+), 21 deletions(-)
layerindex/update.py +46 -21
@@ -299,6 +299,7 @@ def main(): last_rev = {} failed_layers = {} for branch in branches: + failed_layers[branch] = [] # If layer_A depends(or recommends) on layer_B, add layer_B before layer_A deps_dict_all = {} layerquery_sorted = []
@@ -399,28 +400,34 @@ def main(): deps = re.search("^LAYERDEPENDS = \"(.*)\"", output, re.M).group(1) or '' recs = re.search("^LAYERRECOMMENDS = \"(.*)\"", output, re.M).group(1) or '' - deps_dict = utils.explode_dep_versions2(bitbakepath, deps + ' ' + recs) - if len(deps_dict) == 0: + deps_dict = utils.explode_dep_versions2(bitbakepath, deps) + recs_dict = utils.explode_dep_versions2(bitbakepath, recs) + if not (deps_dict or recs_dict): # No depends, add it firstly layerquery_sorted.append(layer) collections.add((col, ver)) continue - deps_dict_all[layer] = {'requires': deps_dict, 'collection': col, 'version': ver} + deps_dict_all[layer] = {'deps': deps_dict, \ + 'recs': recs_dict, \ + 'collection': col, \ + 'version': ver} # Move deps_dict_all to layerquery_sorted orderly - logger.info("Sorting layers for branch %s" % branch) + if deps_dict_all: + logger.info("Sorting layers for branch %s" % branch) while True: deps_dict_all_copy = deps_dict_all.copy() for layer, value in deps_dict_all_copy.items(): - for req_col, req_ver_list in value['requires'].copy().items(): - matched = False - if req_ver_list: - req_ver = req_ver_list[0] - else: - req_ver = None - if utils.is_deps_satisfied(req_col, req_ver, collections): - del(value['requires'][req_col]) - if not value['requires']: + for deps_recs in ('deps', 'recs'): + for req_col, req_ver_list in value[deps_recs].copy().items(): + matched = False + if req_ver_list: + req_ver = req_ver_list[0] + else: + req_ver = None + if utils.is_deps_satisfied(req_col, req_ver, collections): + del(value[deps_recs][req_col]) + if not (value['deps'] or value['recs']): # All the depends are in collections: del(deps_dict_all[layer]) layerquery_sorted.append(layer)
@@ -429,15 +436,32 @@ def main(): if not len(deps_dict_all): break - # If nothing changed after a run then some dependencies couldn't be resolved + finished = True + # If nothing changed after a run, drop recs and try again if operator.eq(deps_dict_all_copy, deps_dict_all): + for layer, value in deps_dict_all.items(): + if value['recs'] and not value['deps']: + # Add it if recs isn't satisfied only. + logger.warn('Adding %s without LAYERRECOMMENDS...' % layer.name) + del(deps_dict_all[layer]) + layerquery_sorted.append(layer) + collections.add((value['collection'], value['version'])) + failed_msg = '%s: Added without LAYERRECOMMENDS' % layer.name + failed_layers[branch].append(failed_msg) + finished = False + break + if not finished: + continue logger.warning("Cannot find required collections on branch %s:" % branch) - layer_names = [] for layer, value in deps_dict_all.items(): - logger.error('%s: %s' % (layer.name, value['requires'])) - layer_names.append(layer.name) + logger.warn('%s: LAYERDEPENDS: %s LAYERRECOMMENDS: %s' % (layer.name, value['deps'], value['recs'])) + if value['deps']: + failed_layers[branch].append('%s: Failed to add since LAYERDEPENDS is not satisfied' % layer.name) + else: + # Should never come here + logger.error("Unexpected errors when sorting layers") + sys.exit(1) logger.warning("Known collections on branch %s: %s" % (branch, collections)) - failed_layers[branch] = layer_names break for layer in layerquery_sorted:
@@ -479,10 +503,11 @@ def main(): logger.info('Update interrupted, exiting') sys.exit(254) if failed_layers: - print() for branch, err_msg_list in failed_layers.items(): - logger.error("Failed layers on branch %s: %s" % (branch, " ".join(err_msg_list))) - print() + if err_msg_list: + print() + logger.error("Issues found on branch %s:\n %s" % (branch, "\n ".join(err_msg_list))) + print() finally: utils.unlock_file(lockfile)
Signature · 1 line
2.7.4
PATCH 4/4 update.py: check whether branch existed when nocheckout
In reply to Robert YangFixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout.
--- layerindex/update.py | 4 ++-- layerindex/update_layer.py | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-)
layerindex/update.py +2 -2
@@ -345,10 +345,10 @@ def main(): repo = git.Repo(repodir) assert repo.bare == False try: + # Always get origin/branchname, so it raises error when branch doesn't exist when nocheckout + topcommit = repo.commit('origin/%s' % branchname) if options.nocheckout: topcommit = repo.commit('HEAD') - else: - topcommit = repo.commit('origin/%s' % branchname) except: if newbranch: logger.info("Skipping update of layer %s - branch %s doesn't exist" % (layer.name, branchdesc))
layerindex/update_layer.py +1 -2
@@ -362,10 +362,9 @@ def main(): # Collect repo info repo = git.Repo(repodir) assert repo.bare == False + topcommit = repo.commit('origin/%s' % branchname) if options.nocheckout: topcommit = repo.commit('HEAD') - else: - topcommit = repo.commit('origin/%s' % branchname) tinfoil = None tempdir = None
Signature · 1 line
2.7.4
Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
Hi Robert,
Quoted 6 lines · CEST Robert Yang
Fixed:
Assume there is no master branch in hello layer:$ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't existThis is correct since hello layer doesn't have master branch, but when --nocheckout:
Quoted 8 lines
$ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip>This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout.
This fix doesn't seem right. The intention of --nocheckout is to not check anything out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct to get the commit from the origin in this case.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
Quoted 15 lines · CEST Paul Eggleton
Hi Robert,
Quoted 12 lines · CEST Robert Yang
Fixed:
Assume there is no master branch in hello layer:$ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't existThis is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip>
Quoted 6 lines
Quoted 2 lines
This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout.
This fix doesn't seem right. The intention of --nocheckout is to not check anything out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct to get the commit from the origin in this case.
Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
Quoted 16 lines · Paul Eggleton
Quoted 15 lines · CEST Paul Eggleton
Hi Robert,
Quoted 12 lines · CEST Robert Yang
Fixed:
Assume there is no master branch in hello layer:$ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't existThis is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip>
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout.
This fix doesn't seem right. The intention of --nocheckout is to not check anything out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct to get the commit from the origin in this case.
Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it.
Do I need update commit message, please ?
// Robert
Quoted 4 lines
Cheers,
Paul
Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
Quoted 18 lines · CEST Robert Yang
Quoted 17 lines · Paul Eggleton
Quoted 16 lines · CEST Paul Eggleton
Quoted 14 lines · CEST Robert Yang
Fixed:
Assume there is no master branch in hello layer:$ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't existThis is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip>This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
This fix doesn't seem right. The intention of --nocheckout is to not check anything out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct to get the commit from the origin in this case.
Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it.
Do I need update commit message, please ?
No, it's OK, I'll merge as-is.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout
In reply to Paul EggletonHi Paul,
Quoted 15 lines · Paul Eggleton
Hi Robert,
Quoted 12 lines · CEST Robert Yang
Fixed:
Assume there is no master branch in hello layer:$ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't existThis is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout [snip] INFO: Sorting layers for branch mater: WARNING: Cannot find required collections on branch master: WARNING: hello: LAYERDEPENDS: <snip>
Quoted 6 lines
Quoted 2 lines
This is incorrect, this patch fixed the problem, now it skips it since the branch doesn't exists when --nocheckout.
This fix doesn't seem right. The intention of --nocheckout is to not check anything out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct to get the commit from the origin in this case.
It still uses HEAD when nocheckout, please see the code:
# Always get origin/branchname, so it raises error when branch doesn't exist when nocheckout
topcommit = repo.commit('origin/%s' % branchname)
if options.nocheckout:
topcommit = repo.commit('HEAD')
I just want to make sure origin/branchname is existed, and raise error when it doesn't exist (update.py catches the error), it still uses HEAD when --nocheckout.
// Robert
Quoted 4 lines
Cheers,
Paul