Some misc changes/fixes..
cover Some misc changes/fixes..
A few misc changes/fixes. The first two are well tested. However, I suspect the 3/3 may be incorrect and I've labeled it an RFC due to this.
1/3 - '.' wasn't allowed in branch names w/o an error. This turned out to be a fairly simple fix.
2/3 - For people who want to use 'poky' repository and not bitbake + openembedded-core. I've tested this locally in both configurations.
3/3 - When I was testing, my local git mirror is broken up with directories that are called 'git.openembedded.org' and 'git.yoctoproject.org' due to this, the system was matching and locking out the edit layer vcs_web_url submissions... so I tried to make it better.. but I'm not sure it's right.
Mark Hatle (3):
layerindex/urls.py: Allow branches with a '.' in the name update.py: Allow bitbake to live in a subdirectory of a repository
editlayer: Be more specific on the searches
docker/settings.py | 3 +++ layerindex/bulkchange.py | 8 +++++++- layerindex/layerconfparse.py | 8 +++++++- layerindex/tools/import_layer.py | 8 ++++---- layerindex/tools/import_wiki_layers.py | 13 ++++++++++--- layerindex/update.py | 14 +++++++++++--- layerindex/update_layer.py | 6 +++++- layerindex/urls.py | 12 ++++++------ settings.py | 3 +++ templates/layerindex/editlayer.html | 8 ++++---- 10 files changed, 60 insertions(+), 23 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/3 layerindex/urls.py: Allow branches with a '.' in the name
Without this change the system will fail parsing various URL components
--- layerindex/urls.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
layerindex/urls.py +6 -6
@@ -107,7 +107,7 @@ urlpatterns = [ BulkChangeDeleteView.as_view( template_name='layerindex/deleteconfirm.html'), name="bulk_change_delete"), - url(r'^branch/(?P<branch>[-\w]+)/', + url(r'^branch/(?P<branch>[-.\w]+)/', include('layerindex.urls_branch')), url(r'^updates/$', UpdateListView.as_view(
@@ -146,17 +146,17 @@ urlpatterns = [ ClassicRecipeDetailView.as_view( template_name='layerindex/classicrecipedetail.html'), name='classic_recipe'), - url(r'^comparison/recipes/(?P<branch>[-\w]+)/$', + url(r'^comparison/recipes/(?P<branch>[-.\w]+)/$', ClassicRecipeSearchView.as_view( template_name='layerindex/classicrecipes.html'), name='comparison_recipe_search'), - url(r'^comparison/search-csv/(?P<branch>[-\w]+)/$', + url(r'^comparison/search-csv/(?P<branch>[-.\w]+)/$', ClassicRecipeSearchView.as_view( template_name='layerindex/classicrecipes_csv.txt', paginate_by=0, content_type='text/csv'), name='comparison_recipe_search_csv'), - url(r'^comparison/stats/(?P<branch>[-\w]+)/$', + url(r'^comparison/stats/(?P<branch>[-.\w]+)/$', ClassicRecipeStatsView.as_view( template_name='layerindex/classicstats.html'), name='comparison_recipe_stats'),
@@ -185,11 +185,11 @@ urlpatterns = [ url(r'^stoptask/(?P<task_id>[-\w]+)/$', task_stop_view, name='task_stop'), - url(r'^ajax/layerchecklist/(?P<branch>[-\w]+)/$', + url(r'^ajax/layerchecklist/(?P<branch>[-.\w]+)/$', LayerCheckListView.as_view( template_name='layerindex/layerchecklist.html'), name='layer_checklist'), - url(r'^ajax/classchecklist/(?P<branch>[-\w]+)/$', + url(r'^ajax/classchecklist/(?P<branch>[-.\w]+)/$', BBClassCheckListView.as_view( template_name='layerindex/classchecklist.html'), name='class_checklist'),
Signature · 1 line
2.17.1
PATCH 2/3 update.py: Allow bitbake to live in a subdirectory of a repository
In reply to Mark HatleAdd a new BITBAKE_PATH to the settings file to specify the path within the BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined repository, such as poky, that contains bitbake, openembedded-core and other layers.
This change also changes the default path, in the fetch directory, for the bitbake checkout. It no longer uses the path 'bitbake', but instead uses the same URL processing as the layer fetching.
There is a side effect that, when using a shared fetch, the branch of the layer will be used instead of the specified bitbake branch. Generally this is a reasonable compromise, since in a combined repository bitbake and openembedded-core component should already match.
--- docker/settings.py | 3 +++ layerindex/bulkchange.py | 8 +++++++- layerindex/layerconfparse.py | 8 +++++++- layerindex/update.py | 14 +++++++++++--- layerindex/update_layer.py | 6 +++++- settings.py | 3 +++ 6 files changed, 36 insertions(+), 6 deletions(-)
docker/settings.py +3 -0
@@ -244,6 +244,9 @@ TEMP_BASE_DIR = "/tmp" # Fetch URL of the BitBake repository for the update script BITBAKE_REPO_URL = "git://git.openembedded.org/bitbake" +# Path within the BITBAKE_REPO_URL, usually empty +BITBAKE_PATH = "" + # Core layer to be used by the update script for basic BitBake configuration CORE_LAYER_NAME = "openembedded-core"
layerindex/bulkchange.py +7 -1
@@ -98,7 +98,13 @@ def main(): branch = utils.get_branch('master') fetchdir = settings.LAYER_FETCH_DIR - bitbakepath = os.path.join(fetchdir, 'bitbake') + + import layerindex.models import LayerItem + bitbakeitem = LayerItem() + bitbakeitem.vcs_url = settings.BITBAKE_REPO_URL + bitbakepath = os.path.join(fetchdir, bitbakeitem.get_fetch_dir()) + if settings.BITBAKE_PATH: + bitbakepath = os.path.join(bitbakepath, settings.BITBAKE_PATH) if not os.path.exists(bitbakepath): sys.stderr.write("Unable to find bitbake checkout at %s" % bitbakepath)
layerindex/layerconfparse.py +7 -1
@@ -20,7 +20,13 @@ class LayerConfParse: if not bitbakepath: fetchdir = settings.LAYER_FETCH_DIR - bitbakepath = os.path.join(fetchdir, 'bitbake') + + from layerindex.models import LayerItem + bitbakeitem = LayerItem() + bitbakeitem.vcs_url = settings.BITBAKE_REPO_URL + bitbakepath = os.path.join(fetchdir, bitbakeitem.get_fetch_dir()) + if settings.BITBAKE_PATH: + bitbakepath = os.path.join(bitbakepath, settings.BITBAKE_PATH) self.bbpath = bitbakepath # Set up BBPATH.
layerindex/update.py +11 -3
@@ -268,8 +268,6 @@ def main(): logger.error("Layer index lock timeout expired") sys.exit(1) try: - bitbakepath = os.path.join(fetchdir, 'bitbake') - if not options.nofetch: # Make sure oe-core is fetched since recipe parsing requires it layerquery_core = LayerItem.objects.filter(comparison=False).filter(name=settings.CORE_LAYER_NAME)
@@ -285,7 +283,17 @@ def main(): if layer.vcs_url not in allrepos: allrepos[layer.vcs_url] = (repodir, urldir, fetchdir, layer.name) # Add bitbake - allrepos[settings.BITBAKE_REPO_URL] = (bitbakepath, "bitbake", fetchdir, "bitbake") + if settings.BITBAKE_REPO_URL not in allrepos: + bitbakeitem = LayerItem() + bitbakeitem.vcs_url = settings.BITBAKE_REPO_URL + bitbakeurldir = bitbakeitem.get_fetch_dir() + bitbakepath = os.path.join(fetchdir, bitbakeurldir) + allrepos[settings.BITBAKE_REPO_URL] = (bitbakepath, bitbakeurldir, fetchdir, "bitbake") + + (bitbakepath, _, _, _) = allrepos[settings.BITBAKE_REPO_URL] + if settings.BITBAKE_PATH: + bitbakepath = os.path.join(bitbakepath, settings.BITBAKE_PATH) + # Parallel fetching pool = multiprocessing.Pool(int(settings.PARALLEL_JOBS)) for url in allrepos:
layerindex/update_layer.py +5 -1
@@ -300,7 +300,11 @@ def main(): logger.error("Please set LAYER_FETCH_DIR in settings.py") sys.exit(1) - bitbakepath = os.path.join(fetchdir, 'bitbake') + bitbakeitem = LayerItem() + bitbakeitem.vcs_url = settings.BITBAKE_REPO_URL + bitbakepath = os.path.join(fetchdir, bitbakeitem.get_fetch_dir()) + if settings.BITBAKE_PATH: + bitbakepath = os.path.join(bitbakepath, settings.BITBAKE_PATH) layer = utils.get_layer(options.layer) urldir = layer.get_fetch_dir()
settings.py +3 -0
@@ -239,6 +239,9 @@ TEMP_BASE_DIR = "/tmp" # Fetch URL of the BitBake repository for the update script BITBAKE_REPO_URL = "git://git.openembedded.org/bitbake" +# Path within the BITBAKE_REPO_URL, usually empty +BITBAKE_PATH = "" + # Core layer to be used by the update script for basic BitBake configuration CORE_LAYER_NAME = "openembedded-core"
Signature · 1 line
2.17.1
PATCH 3/3 RFC: editlayer: Be more specific on the searches
In reply to Mark HatleJust because git.yoctoproject.org is in the URL, doesn't mean we can or should force the vcs_web_url to be a specific value. If it starts withgit://git.yoctoproject.org then we can do this. git.openembedded.org already did this.
This also changes github, gitlab and bitbucket references.
--- layerindex/tools/import_layer.py | 8 ++++---- layerindex/tools/import_wiki_layers.py | 13 ++++++++++--- templates/layerindex/editlayer.html | 8 ++++---- 3 files changed, 18 insertions(+), 11 deletions(-)
layerindex/tools/import_layer.py +4 -4
@@ -36,27 +36,27 @@ def set_vcs_fields(layer, repoval): layer.vcs_web_tree_base_url = 'http://cgit.openembedded.org/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_file_base_url = 'http://cgit.openembedded.org/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_commit_url = 'http://cgit.openembedded.org/' + reponame + '/commit/?id=%hash%' - elif 'git.yoctoproject.org/' in repoval: + elif repoval.startswith('git://git.yoctoproject.org/'): reponame = re.sub('^.*/', '', repoval) layer.vcs_web_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame layer.vcs_web_tree_base_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_file_base_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_commit_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/commit/?id=%hash%' - elif 'github.com/' in repoval: + elif repoval.startswith('git://github.com/') or repoval.startswith('http://github.com/') or repoval.startswith('https://github.com/'): reponame = re.sub('^.*github.com/', '', repoval) reponame = re.sub('.git$', '', reponame) layer.vcs_web_url = 'http://github.com/' + reponame layer.vcs_web_tree_base_url = 'http://github.com/' + reponame + '/tree/%branch%/' layer.vcs_web_file_base_url = 'http://github.com/' + reponame + '/blob/%branch%/' layer.vcs_web_commit_url = 'http://github.com/' + reponame + '/commit/%hash%' - elif 'gitlab.com/' in repoval: + elif repoval.startswith('git://gitlab.com/') or repoval.startswith('http://gitlab.com/') or repoval.startswith('https://gitlab.com/'): reponame = re.sub('^.*gitlab.com/', '', repoval) reponame = re.sub('.git$', '', reponame) layer.vcs_web_url = 'http://gitlab.com/' + reponame layer.vcs_web_tree_base_url = 'http://gitlab.com/' + reponame + '/tree/%branch%/' layer.vcs_web_file_base_url = 'http://gitlab.com/' + reponame + '/blob/%branch%/' layer.vcs_web_commit_url = 'http://gitlab.com/' + reponame + '/commit/%hash%' - elif 'bitbucket.org/' in repoval: + elif repoval.startswith('git://bitbucket.org/') or repoval.startswith('http://bitbucket.org/') or repoval.startswith('https://bitbucket.org/'): reponame = re.sub('^.*bitbucket.org/', '', repoval) reponame = re.sub('.git$', '', reponame) layer.vcs_web_url = 'http://bitbucket.org/' + reponame
layerindex/tools/import_wiki_layers.py +10 -3
@@ -100,20 +100,27 @@ def main(): layer.vcs_web_tree_base_url = 'http://cgit.openembedded.org/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_file_base_url = 'http://cgit.openembedded.org/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_commit_url = 'http://cgit.openembedded.org/' + reponame + '/commit/?id=%hash%' - elif 'git.yoctoproject.org/' in repoval: + elif repoval.startswith('git://git.yoctoproject.org/'): reponame = re.sub('^.*/', '', repoval) layer.vcs_web_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame layer.vcs_web_tree_base_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_file_base_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/tree/%path%?h=%branch%' layer.vcs_web_commit_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/commit/?id=%hash%' - elif 'github.com/' in repoval: + elif repoval.startswith('git://github.com/') or repoval.startswith('http://github.com/') or repoval.startswith('https://github.com/'): reponame = re.sub('^.*github.com/', '', repoval) reponame = re.sub('.git$', '', reponame) layer.vcs_web_url = 'http://github.com/' + reponame layer.vcs_web_tree_base_url = 'http://github.com/' + reponame + '/tree/%branch%/' layer.vcs_web_file_base_url = 'http://github.com/' + reponame + '/blob/%branch%/' layer.vcs_web_commit_url = 'http://github.com/' + reponame + '/commit/%hash%' - elif 'bitbucket.org/' in repoval: + elif repoval.startswith('git://gitlab.com/') or repoval.startswith('http://gitlab.com/') or repoval.startswith('https://gitlab.com/'): + reponame = re.sub('^.*gitlab.com/', '', repoval) + reponame = re.sub('.git$', '', reponame) + layer.vcs_web_url = 'http://gitlab.com/' + reponame + layer.vcs_web_tree_base_url = 'http://gitlab.com/' + reponame + '/tree/%branch%/' + layer.vcs_web_file_base_url = 'http://gitlab.com/' + reponame + '/blob/%branch%/' + layer.vcs_web_commit_url = 'http://gitlab.com/' + reponame + '/commit/%hash%' + elif repoval.startswith('git://bitbucket.org/') or repoval.startswith('http://bitbucket.org/') or repoval.startswith('https://bitbucket.org/'): reponame = re.sub('^.*bitbucket.org/', '', repoval) reponame = re.sub('.git$', '', reponame) layer.vcs_web_url = 'http://bitbucket.org/' + reponame
templates/layerindex/editlayer.html +4 -4
@@ -204,7 +204,7 @@ this.vcs_web_commit_url = 'http://cgit.openembedded.org/' + reponame + '/commit/?id=%hash%' this.vcs_web_type = 'cgit' } - else if( repoval.indexOf('git.yoctoproject.org/') > -1 ) { + else if( repoval.startsWith('git://git.yoctoproject.org/') ) { reponame = repoval.replace(/^.*\//, '') this.vcs_web_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame this.vcs_web_tree_base_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/tree/%path%?h=%branch%'
@@ -212,7 +212,7 @@ this.vcs_web_commit_url = 'http://git.yoctoproject.org/cgit/cgit.cgi/' + reponame + '/commit/?id=%hash%' this.vcs_web_type = 'cgit' } - else if( repoval.indexOf('github.com/') > -1 ) { + else if( repoval.startsWith('git://github.com/') ) { reponame = repoval.replace(/^.*github.com\//, '') reponame = reponame.replace(/.git$/, '') this.vcs_web_url = 'http://github.com/' + reponame
@@ -221,7 +221,7 @@ this.vcs_web_commit_url = 'http://github.com/' + reponame + '/commit/%hash%/' this.vcs_web_type = '(custom)' } - else if( repoval.indexOf('gitlab.com/') > -1 ) { + else if( repoval.startsWith('git://gitlab.com/') ) { reponame = repoval.replace(/^.*gitlab.com\//, '') reponame = reponame.replace(/.git$/, '') this.vcs_web_url = 'http://gitlab.com/' + reponame
@@ -230,7 +230,7 @@ this.vcs_web_commit_url = 'http://gitlab.com/' + reponame + '/commit/%hash%/' this.vcs_web_type = '(custom)' } - else if( repoval.indexOf('bitbucket.org/') > -1 ) { + else if( repoval.startsWith('git://bitbucket.org/') ) { reponame = repoval.replace(/^.*bitbucket.org\//, '') reponame = reponame.replace(/.git$/, '') this.vcs_web_url = 'http://bitbucket.org/' + reponame
Signature · 1 line
2.17.1
Hi Mark
Quoted 1 line · NZDT Mark Hatle
A few misc changes/fixes. The first two are well tested. However, I
suspect
Quoted 10 lines
the 3/3 may be incorrect and I've labeled it an RFC due to this.
1/3 - '.' wasn't allowed in branch names w/o an error. This turned out to be a fairly simple fix.
2/3 - For people who want to use 'poky' repository and not bitbake + openembedded-core. I've tested this locally in both configurations.
3/3 - When I was testing, my local git mirror is broken up with directories that are called 'git.openembedded.org' and
'git.yoctoproject.org'
Quoted 8 lines
due to this, the system was matching and locking out the edit layer
vcs_web_urlsubmissions... so I tried to make it better.. but I'm not sure it's right.Mark Hatle (3):
layerindex/urls.py: Allow branches with a '.' in the nameupdate.py: Allow bitbake to live in a subdirectory of a repository
editlayer: Be more specific on the searches
Thanks, I merged these - I did make a minor change to 2/3 to allow it to work if BITBAKE_PATH doesn't appear in settings.py (I try to do this when adding new settings in case the user doesn't add the default to their edited settings file).
Cheers
Paul
Signature · 2 lines
Paul Eggleton Intel System Software Products