Skip to contentSkip to conversation

bitbake-layers: add a ability to query layer dependencies from layer index

PATCH Superseded Chong Lu 3 participants 10 messages Jan 23, 2015 – Feb 9, 2015

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Chong Lu

author
#0 lore raw

v3 cover bitbake-layers: add a ability to query layer dependencies from layer index

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:

maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

 bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf     |   6 ++
 2 files changed, 196 insertions(+)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH v3 1/2 bitbake.conf: Add two variables for layer index

Add BITBAKE_LAYERINDEX_URL variable that bitbake-layers can use to find layer index. Add LAYER_FETCH_DIR variable that bitbake-layers can use to specify fetch directory.

[YOCTO #5348]

---
 meta/conf/bitbake.conf | 6 ++++++
 1 file changed, 6 insertions(+)
meta/conf/bitbake.conf +6 -0
index d22e9e8..a8f4dc9 100644
@@ -549,6 +549,12 @@ SELECTED_OPTIMIZATION[vardeps] += "FULL_OPTIMIZATION DEBUG_OPTIMIZATION"
 BUILD_OPTIMIZATION = "-O2 -pipe"
 
 ##################################################################
+# The OE layer index.
+##################################################################
+BITBAKE_LAYERINDEX_URL ??= "http://layers.openembedded.org/layerindex/api/"
+LAYER_FETCH_DIR ??= "${COREBASE}"
+
+##################################################################
 # Download locations and utilities.
 ##################################################################
 
Signature · 1 line
1.9.1
author
#2 lore raw

PATCH v3 2/2 bitbake-layers: add a ability to query layer dependencies from layer index

In reply to Chong Lu

Add a command to query layer dependencies from layer index. Fetch layer and its dependency layers and add them into conf/bblayers.conf.

[YOCTO #5348]

---
 bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 190 insertions(+)
bitbake/bin/bitbake-layers +190 -0
index 9879498..222f685 100755
@@ -27,6 +27,8 @@ import sys
 import fnmatch
 from collections import defaultdict
 import re
+import httplib, urlparse, json
+import subprocess
 
 bindir = os.path.dirname(__file__)
 topdir = os.path.dirname(bindir)
@@ -157,6 +159,194 @@ usage: remove-layer <layerdir>
                 sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
 
 
+    def get_json_data(self, apiurl):
+        proxy_settings = os.environ.get("http_proxy", None)
+        conn = None
+        _parsedurl = urlparse.urlparse(apiurl)
+        path = _parsedurl.path
+        query = _parsedurl.query
+        def parse_url(url):
+            parsedurl = urlparse.urlparse(url)
+            if parsedurl.netloc[0] == '[':
+                host, port = parsedurl.netloc[1:].split(']', 1)
+                if ':' in port:
+                    port = port.rsplit(':', 1)[1]
+                else:
+                    port = None
+            else:
+                if parsedurl.netloc.count(':') == 1:
+                    (host, port) = parsedurl.netloc.split(":")
+                else:
+                    host = parsedurl.netloc
+                    port = None
+            return (host, 80 if port is None else int(port))
+
+        if proxy_settings is None:
+            host, port = parse_url(apiurl)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", path + "?" + query)
+        else:
+            host, port = parse_url(proxy_settings)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", apiurl)
+
+        r = conn.getresponse()
+        if r.status != 200:
+            raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason))
+        return json.loads(r.read())
+
+
+    def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies, selfname=False):
+        def layeritems_info_id(items_name, layeritems):
+            litems_id = ""
+            for li in layeritems:
+                if li['name'] == items_name:
+                    litems_id = li['id']
+                    break
+            if litems_id:
+                return litems_id
+
+        def layerbranches_info(items_id, layerbranches):
+            lbranch = {}
+            for lb in layerbranches:
+                # branch is master.
+                if lb['layer'] == items_id and lb['branch'] == 1:
+                    lbranch['id'] = lb['id']
+                    lbranch['vcs_subdir'] = lb['vcs_subdir']
+                    break
+            if not lbranch['id']:
+                logger.error("The id of layerBranches is not found.")
+                return
+            else:
+                return lbranch
+
+        def layerdependencies_info(lb_id, layerdependencies):
+            ld_deps = []
+            for ld in layerdependencies:
+                if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps:
+                    ld_deps.append(ld['dependency'])
+            if not ld_deps:
+                logger.error("The dependency of layerDependencies is not found.")
+                return
+            else:
+                return ld_deps
+
+        def layeritems_info_name_subdir(items_id, layeritems):
+            litems = {}
+            for li in layeritems:
+                if li['id'] == items_id:
+                    litems['vcs_url'] = li['vcs_url']
+                    litems['name'] = li['name']
+                    break
+            return litems
+
+        if selfname:
+            selfid = layeritems_info_id(layername, layeritems)
+            selfsubdir = layerbranches_info(selfid, layerbranches)['vcs_subdir']
+            selfurl = layeritems_info_name_subdir(selfid, layeritems)['vcs_url']
+            if selfsubdir and selfurl:
+                return selfurl, selfsubdir
+            else:
+                logger.error("Can NOT get %s git repo and subdir" % layername)
+                return
+        ldict = {}
+        itemsid = layeritems_info_id(layername, layeritems)
+        if not itemsid:
+            return layername, None
+        lbid = layerbranches_info(itemsid, layerbranches)['id']
+        for dependency in layerdependencies_info(lbid, layerdependencies):
+            lname = layeritems_info_name_subdir(dependency, layeritems)['name']
+            lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url']
+            lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir']
+            ldict[lname] = lurl, lsubdir
+        return None, ldict
+
+
+    def get_fetch_layer(self, fetchdir, url, subdir):
+        layername = self.get_layer_name(url)
+        if os.path.splitext(layername)[1] == '.git':
+            layername = os.path.splitext(layername)[0]
+        repodir = os.path.join(fetchdir, layername)
+        layerdir = os.path.join(repodir, subdir)
+        if not os.path.exists(repodir):
+            logger.plain("Cloning into '%s'..." % repodir)
+            result = subprocess.call('git clone -q %s %s' % (url, repodir), shell = True)
+            if result:
+                logger.error("Failed to download %s" % url)
+            else:
+                return layername, layerdir
+        elif os.path.exists(layerdir):
+            return layername, layerdir
+        else:
+            logger.error("%s is not in %s" % (url, subdir))
+
+
+    def do_show_layer_deps(self, args):
+        """Find layer dependencies from layer index. Fetch it and its dependency layers. Add them to conf/bblayers.conf.
+
+usage: show-layer-deps [-a] <layername,...>
+
+Options:
+  -a   fetch layer and add to conf/bblayers.conf
+"""
+        fetch_add = False
+        layernames = ""
+        for arg in args.split():
+            if arg == '-a':
+                fetch_add = True
+            elif not arg.startswith('-'):
+                layernames = arg
+            else:
+                sys.stderr.write("show-layer-deps: invalid option %s\n" % arg)
+                self.do_help('')
+                return
+        if not layernames:
+            sys.stderr.write("Please specify layer name.\n")
+            return
+        self.init_bbhandler()
+        apiurl = self.bbhandler.config_data.getVar('BITBAKE_LAYERINDEX_URL', True)
+        if not apiurl:
+            logger.error("Can NOT get BITBAKE_LAYERINDEX_URL.")
+        apilinks = self.get_json_data(apiurl)
+        layeritems = self.get_json_data(apilinks['layerItems'])
+        layerbranches = self.get_json_data(apilinks['layerBranches'])
+        layerdependencies = self.get_json_data(apilinks['layerDependencies'])
+        invaluenames = []
+        repourls = []
+        display = True
+        for layername in layernames.split(','):
+            if not layername == "meta":
+                invaluename, layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies)
+                if layerdict:
+                    repourls.append(self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies, selfname=True))
+                    for layer in layerdict:
+                        if display:
+                            logger.plain("%s  %s  %s  %s" % ("Layer".ljust(20), "Dependencies".ljust(20), "Git repository".ljust(55), "Subdirectory"))
+                            logger.plain('=' * 115)
+                            display = False
+                        logger.plain("%s  %s  %s  %s" % (layername.ljust(20), layer.ljust(20), layerdict[layer][0].ljust(55), layerdict[layer][1]))
+                        if not layer == "openembedded-core" and not (layerdict[layer][0], layerdict[layer][1]) in repourls:
+                            repourls.append((layerdict[layer][0], layerdict[layer][1]))
+                if invaluename and not invaluename in invaluenames:
+                    invaluenames.append(invaluename)
+        for invaluename in invaluenames:
+            logger.warn("%s is not found in layer index." % invaluename)
+        if fetch_add and repourls:
+            fetchdir = self.bbhandler.config_data.getVar('LAYER_FETCH_DIR', True)
+            if not fetchdir:
+                logger.error("Can NOT get LAYER_FETCH_DIR.")
+                return
+            if not os.path.exists(fetchdir):
+                os.makedirs(fetchdir)
+            for repourl, subdir in repourls:
+                name, layerdir = self.get_fetch_layer(fetchdir, repourl, subdir)
+                if subdir:
+                    logger.plain("Add \"%s\" to conf/bblayers.conf" % subdir)
+                else:
+                    logger.plain("Add \"%s\" to conf/bblayers.conf" % name)
+                self.do_add_layer(layerdir)
+
+
     def version_str(self, pe, pv, pr = None):
         verstr = "%s" % pv
         if pr:
Signature · 1 line
1.9.1
#3 lore raw
Quoted 38 lines · Chong Lu

Add a command to query layer dependencies from layer index. Fetch layer and its dependency layers and add them into conf/bblayers.conf.

[YOCTO #5348]

---
 bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 190 insertions(+)
bitbake/bin/bitbake-layers +13 -0
index 9879498..222f685 100755
@@ -27,6 +27,8 @@ import sys
 import fnmatch
 from collections import defaultdict
 import re
+import httplib, urlparse, json
+import subprocess
 
 bindir = os.path.dirname(__file__)
 topdir = os.path.dirname(bindir)
@@ -157,6 +159,194 @@ usage: remove-layer <layerdir>
                 sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
 
 
+    def get_json_data(self, apiurl):
+        proxy_settings = os.environ.get("http_proxy", None)
+        conn = None
+        _parsedurl = urlparse.urlparse(apiurl)
+        path = _parsedurl.path
+        query = _parsedurl.query
+        def parse_url(url):
+            parsedurl = urlparse.urlparse(url)
+            if parsedurl.netloc[0] == '[':
+                host, port = parsedurl.netloc[1:].split(']', 1)
+                if ':' in port:

bitbake/bin/bitbake-layers usage: remove-layer <layerdir> · line 182

Quoted 10 lines
bitbake/bin/bitbake-layers +10 -0
+                    port = port.rsplit(':', 1)[1]
+                else:
+                    port = None
+            else:
+                if parsedurl.netloc.count(':') == 1:
+                    (host, port) = parsedurl.netloc.split(":")
+                else:
+                    host = parsedurl.netloc
+                    port = None
+            return (host, 80 if port is None else int(port))

well yea, except that does not help code-reuse to fix the wrong splitport patterns used elsewhere in bitbake, unfortunately..

author
#4 lore raw
In reply to Chong Lu

ping

//Chong

Quoted 20 lines · Chong Lu

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:

   maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

  bitbake/bin/bitbake-layers | 190 +++++++++++++++++++++++++++++++++++++++++++++
  meta/conf/bitbake.conf     |   6 ++
  2 files changed, 196 insertions(+)
#5 lore raw
In reply to Chong Lu

Hi Chong,

Quoted 7 lines · On Friday 23 January 2015 13:28:32 Chong Lu wrote:

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:

maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

Quoted 8 lines

are available in the git repository at:

git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

Apologies for the delay in getting back to you. This is definitely improving, some feedback though:

* I think the new subcommand would make more sense if it were switched around so that it was fetching by default (and renamed as appropriate). Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what it would fetch).

* I think the layer index URL specified in the OE variable should be to the root rather than to /api - instead have that suffix added by the code that does the API calls. This makes it usable as a URL for the index for presenting to the user if needed.

* Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

* Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

* By using -q with git you lose the status output which can be very useful given how long it can take to fetch some layers e.g. meta-oe, so can you drop this?

* Can you add a check to see if the layer has already been fetched before fetching it and skip the fetch if it does (but not skip the add)?

* If I run "bitbake-layers show-layer-deps meta-security" I get an error:
ERROR: Can NOT get meta-security git repo and subdir
(The code currently expects there to be a subdir, but this is an optional field, not all layers have a subdirectory, meta-security is one example.)

* I think you may be able to get away with skipping the initial parse by passing config_only=True to init_bbhandler(). If you can, this will save a bit of time each time you run the command.

Thanks,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw
Quoted 7 lines · On Monday 02 February 2015 16:05:52 Paul Eggleton wrote:

Hi Chong,

Quoted 4 lines · On Friday 23 January 2015 13:28:32 Chong Lu wrote:

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit

35c9fa0588ed8e88b541a6c80cc1517324616cea:

Quoted 38 lines
Quoted 11 lines

maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

Apologies for the delay in getting back to you. This is definitely improving, some feedback though:

* I think the new subcommand would make more sense if it were switched around so that it was fetching by default (and renamed as appropriate). Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what it would fetch).

* I think the layer index URL specified in the OE variable should be to the root rather than to /api - instead have that suffix added by the code that does the API calls. This makes it usable as a URL for the index for presenting to the user if needed.

* Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

* Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

* By using -q with git you lose the status output which can be very useful given how long it can take to fetch some layers e.g. meta-oe, so can you drop this?

* Can you add a check to see if the layer has already been fetched before fetching it and skip the fetch if it does (but not skip the add)?

Quoted 8 lines

* If I run "bitbake-layers show-layer-deps meta-security" I get an error:
ERROR: Can NOT get meta-security git repo and subdir
(The code currently expects there to be a subdir, but this is an optional field, not all layers have a subdirectory, meta-security is one example.)

* I think you may be able to get away with skipping the initial parse by passing config_only=True to init_bbhandler(). If you can, this will save a bit of time each time you run the command.

I just noticed another issue - dependencies-of-dependencies aren't handled here either; e.g. meta-security's dependencies have just been updated so that it now depends on meta-networking, and meta-networking depends on meta-python but that's not listed when I run bitbake-layers show-layer-deps meta-security. I think maybe this is a deficiency in the layer index API - it would be faster if there was one call we could make to get the entire list of dependencies, URLs and subdirectories for the entire dependency chain.

I know this is probably further than you expected to go with this bug, but is that API enhancement something you could look into adding?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
author
#8 lore raw
In reply to Paul Eggleton
Quoted 26 lines · Paul Eggleton

Hi Chong,

Quoted 7 lines · On Friday 23 January 2015 13:28:32 Chong Lu wrote:

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:

   maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)
Quoted 8 lines

are available in the git repository at:

git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

Apologies for the delay in getting back to you. This is definitely improving, some feedback though:

* I think the new subcommand would make more sense if it were switched around so that it was fetching by default (and renamed as appropriate). Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what it would fetch).

Quoted 7 lines

* I think the layer index URL specified in the OE variable should be to the root rather than to /api - instead have that suffix added by the code that does the API calls. This makes it usable as a URL for the index for presenting to the user if needed.

* Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

I don't understand why does uname? Or, would you give me a example?

Quoted 3 lines

* Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

And I don't understand this one """

Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

"""
Would you explain in details?

Best Regards
Chong

Quoted 20 lines

* By using -q with git you lose the status output which can be very useful given how long it can take to fetch some layers e.g. meta-oe, so can you drop this?

* Can you add a check to see if the layer has already been fetched before fetching it and skip the fetch if it does (but not skip the add)?

* If I run "bitbake-layers show-layer-deps meta-security" I get an error:
ERROR: Can NOT get meta-security git repo and subdir
(The code currently expects there to be a subdir, but this is an optional field, not all layers have a subdirectory, meta-security is one example.)

* I think you may be able to get away with skipping the initial parse by passing config_only=True to init_bbhandler(). If you can, this will save a bit of time each time you run the command.

Thanks,
Paul

#9 lore raw

Hi Chong,

Quoted 6 lines · On Friday 06 February 2015 14:52:14 Chong Lu wrote:
Quoted 5 lines · Paul Eggleton
Quoted 4 lines · On Friday 23 January 2015 13:28:32 Chong Lu wrote:

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit

35c9fa0588ed8e88b541a6c80cc1517324616cea:

Quoted 27 lines
Quoted 27 lines
Quoted 8 lines
   maintainers: Update for non-maintained recipes (2015-01-20 21:39:41
   +0000)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerind
ex

Quoted 4 lines

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

Apologies for the delay in getting back to you. This is definitely improving, some feedback though:

* I think the new subcommand would make more sense if it were switched around so that it was fetching by default (and renamed as appropriate). Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what it would fetch).

* I think the layer index URL specified in the OE variable should be to the
root rather than to /api - instead have that suffix added by the code that does the API calls. This makes it usable as a URL for the index for presenting to the user if needed.

Quoted 4 lines
Quoted 2 lines

* Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

I don't understand why does uname? Or, would you give me a example?

I'm not sure I understand the question - but FWIW the variables I'm talking about renaming are currently named BITBAKE_LAYERINDEX_URL and LAYER_FETCH_DIR. The reason I suggest naming them BBLAYERS_LAYERINDEX_URL and BBLAYERS_FETCH_DIR respectively is it is more closely related to the area where the values are actually used. Is that what you were asking about?

Quoted 10 lines
Quoted 2 lines

* Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

And I don't understand this one """

Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

"""
Would you explain in details?

You're adding defaults for these variables in bitbake.conf. We should mention above those defaults what these variables are actually being used for. i.e.:

##################################################################
+# Settings used by bitbake-layers.
+##################################################################
+BBLAYERS_LAYERINDEX_URL ??= "http://layers.openembedded.org/layerindex/api/" +BBLAYERS_FETCH_DIR ??= "${COREBASE}"
+
+##################################################################

Quoted 7 lines
Quoted 7 lines

* By using -q with git you lose the status output which can be very useful given how long it can take to fetch some layers e.g. meta-oe, so can you drop this?

* Can you add a check to see if the layer has already been fetched before fetching it and skip the fetch if it does (but not skip the add)?

Quoted 8 lines
Quoted 8 lines

* If I run "bitbake-layers show-layer-deps meta-security" I get an error:
ERROR: Can NOT get meta-security git repo and subdir
(The code currently expects there to be a subdir, but this is an optional field, not all layers have a subdirectory, meta-security is one example.)

* I think you may be able to get away with skipping the initial parse by passing config_only=True to init_bbhandler(). If you can, this will save a bit of time each time you run the command.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#7 lore raw
In reply to Paul Eggleton
Quoted 68 lines · On Tuesday 03 February 2015 10:12:19 Paul Eggleton wrote:
Quoted 7 lines · On Monday 02 February 2015 16:05:52 Paul Eggleton wrote:

Hi Chong,

Quoted 4 lines · On Friday 23 January 2015 13:28:32 Chong Lu wrote:

Change since V2:
re-split for IPv6 URL parsing

The following changes since commit

35c9fa0588ed8e88b541a6c80cc1517324616cea:

Quoted 40 lines
Quoted 8 lines

maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerind
ex

Quoted 4 lines

Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer index

Apologies for the delay in getting back to you. This is definitely improving, some feedback though:

* I think the new subcommand would make more sense if it were switched around so that it was fetching by default (and renamed as appropriate). Perhaps "layerindex-fetch" (with -n to avoid fetching and just report what it would fetch).

* I think the layer index URL specified in the OE variable should be to the
root rather than to /api - instead have that suffix added by the code that does the API calls. This makes it usable as a URL for the index for presenting to the user if needed.

* Can you rename the two variables to BBLAYERS_LAYERINDEX_URL and BBLAYERS_FETCH_DIR (sorry I didn't think of this earlier).

* Please mention bitbake-layers in the comment above the settings in bitbake.conf so it's clear where these variables are used.

* By using -q with git you lose the status output which can be very useful given how long it can take to fetch some layers e.g. meta-oe, so can you drop this?

* Can you add a check to see if the layer has already been fetched before fetching it and skip the fetch if it does (but not skip the add)?

Quoted 8 lines

* If I run "bitbake-layers show-layer-deps meta-security" I get an error:
ERROR: Can NOT get meta-security git repo and subdir
(The code currently expects there to be a subdir, but this is an optional field, not all layers have a subdirectory, meta-security is one example.)

* I think you may be able to get away with skipping the initial parse by passing config_only=True to init_bbhandler(). If you can, this will save a bit of time each time you run the command.

I just noticed another issue - dependencies-of-dependencies aren't handled here either; e.g. meta-security's dependencies have just been updated so that it now depends on meta-networking, and meta-networking depends on meta-python but that's not listed when I run bitbake-layers show-layer-deps meta-security. I think maybe this is a deficiency in the layer index API - it would be faster if there was one call we could make to get the entire list of dependencies, URLs and subdirectories for the entire dependency chain.

Quoted 2 lines

I know this is probably further than you expected to go with this bug, but is that API enhancement something you could look into adding?

Yet another concern - we need to deal with layer branches properly, i.e. we can't just assume master or otherwise people using stable branches will have compatibility issues. In many cases we won't be able to rely upon the OE- Core/poky branch being the matching name either, so it might have to be something we require the user to specify on the command line.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre