bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
PATCH 1/2 bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
Added code which takes a list of directories in COLLECTIONS, in priority order (highest to lowest), and uses those to populate BBFILES, BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
The default COLLECTIONS is based on BBPATH, so you can now specify a sane BBPATH, not bother setting up the collections variables or BBFILES, and still likely be able to do a build.
--- conf/bitbake.conf | 1 + conf/collections.inc | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 0 deletions(-) create mode 100644 conf/collections.inc
conf/bitbake.conf +1 -0
@@ -620,6 +620,7 @@ OVERRIDES = "local:${MACHINE}:${DISTRO}:${TARGET_OS}:${TARGET_ARCH}:build-${BUIL # Include the rest of the config files. ################################################################## +require conf/collections.inc include conf/site.conf include conf/auto.conf include conf/local.conf
conf/collections.inc new file +55 -0
55 lines not on this page · raw message
Signature · 1 line
1.5.2.1
PATCH 2/2 collections.inc: add support for collection archives.
From: Chris Larson <[email protected]>
Adds the ability to specify an archive in COLLECTIONS. Any archives of a
supported format (default tar.gz/tgz/tar.X, tar.bz2/tbz/tbz2, zip/jar) will be
unpacked into COLLECTIONS_UNPACKDIR and used from there. The default value of COLLECTIONS_UNPACKDIR is ${TMPDIR}/collections.
--- conf/collections.inc | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 73 insertions(+), 0 deletions(-)
conf/collections.inc +73 -0
@@ -1,8 +1,72 @@ # Take a list of directories in COLLECTIONS, in priority order (highest to # lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS, # BBFILE_PATTERN_*, and BBFILE_PRIORITY_*. +# +# Specifying an archive in COLLECTIONS is also supported. Any archives of a +# supported format will be unpacked into COLLECTIONS_UNPACKDIR and used from +# there. COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}" +COLLECTIONS_UNPACKDIR = "${TMPDIR}/collections" + +def collection_unpack(collection, name, d): + """ Unpack a collection archive and return the path to it. """ + import bb + import os + from md5 import md5 + + handlers = { + ("tar"): "tar x --no-same-owner -f %s", + ("tar.gz", "tgz", "tar.Z"): "tar xz --no-same-owner -f %s", + ("tar.bz2", "tbz", "tbz2"): "tar xj --no-same-owner -f %s", + ("zip", "jar"): "unzip -q -o %s", + } + + outpath = os.path.join(d.getVar("COLLECTIONS_UNPACKDIR", 1), name) + + try: + collectiondata = open(collection, "r").read() + except IOError: + bb.fatal("Unable to open %s to calculate md5 sum" % collection) + + md5obj = md5() + md5obj.update(collectiondata) + md5sum = md5obj.hexdigest() + + md5file = os.path.join(outpath, "md5") + if os.path.exists(md5file): + try: + oldmd5sum = open(md5file).read() + except IOError: + pass + else: + if oldmd5sum == md5sum: + bb.debug(1, "Using existing %s for collection %s" % (outpath, name)) + return outpath + + bb.note("Removing old unpacked collection at %s" % outpath) + os.system("rm -rf %s" % outpath) + + try: + cmd = (cmd for (exts, cmd) in handlers.iteritems() + for e in exts + if collection.endswith(e)).next() + cmd = "cd %s && PATH=\"%s\" %s" % (outpath, d.getVar("PATH", 1), cmd) + except StopIteration: + bb.fatal("Unable to find unpack handler for %s" % collection) + + if not os.path.isdir(outpath): + os.makedirs(outpath) + + bb.note("Unpacking %s to %s/" % (collection, outpath)) + ret = os.system(cmd % collection) + if ret != 0: + bb.fatal("Unable to unpack %s" % collection) + + md5out = open(md5file, "w") + md5out.write(md5sum) + md5out.close() + return outpath def collections_setup(d): """ Populate collection and bbfiles metadata from the COLLECTIONS var. """
@@ -38,6 +102,15 @@ def collections_setup(d): if not name: bb.fatal("Unable to determine collection name for %s" % collection) + if not os.path.isdir(collection): + del collectionmap[collection] + unpacked = collection_unpack(collection, name, d) + if unpacked: + collection = unpacked + collectionmap[collection] = name + else: + bb.fatal("Unable to unpack collection %s" % collection) + setifunset("BBFILE_PATTERN_%s" % name, "^%s/" % collection) setifunset("BBFILE_PRIORITY_%s" % name, str(priority))
Signature · 1 line
1.5.2.1
Quoted 9 lines · Chris Larson
Added code which takes a list of directories in COLLECTIONS, in priority order (highest to lowest), and uses those to populate BBFILES,
BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.The default COLLECTIONS is based on BBPATH, so you can now specify a sane BBPATH, not bother setting up the collections variables or BBFILES, and still likely be able to do a build.
I haven't had time to test the patch, but I really like the idea of this. Also the archives part as well. This will make setup so much easier for people wanting to overlay. And improve the ability for OE to be shipped in setup scripts.
Graeme
Quoted 10 lines · Graeme Gregory
Quoted 9 lines · Chris Larson
Added code which takes a list of directories in COLLECTIONS, in priority order (highest to lowest), and uses those to populate BBFILES,
BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.The default COLLECTIONS is based on BBPATH, so you can now specify a sane BBPATH, not bother setting up the collections variables or BBFILES, and still likely be able to do a build.
Quoted 4 lines
I haven't had time to test the patch, but I really like the idea of this. Also the archives part as well. This will make setup so much easier for people wanting to overlay. And improve the ability for OE to be shipped in setup scripts.
+1 - support the idea!
I'll try to test the patches on my overlay...
Signature · 1 line
Denys
Quoted 11 lines · Denys Dmytriyenko
Quoted 10 lines · Graeme Gregory
Quoted 9 lines · Chris Larson
Added code which takes a list of directories in COLLECTIONS, in priority order (highest to lowest), and uses those to populate BBFILES,
BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.The default COLLECTIONS is based on BBPATH, so you can now specify a sane BBPATH, not bother setting up the collections variables or BBFILES, and still likely be able to do a build.
Quoted 8 lines
Quoted 4 lines
I haven't had time to test the patch, but I really like the idea of this. Also the archives part as well. This will make setup so much easier for people wanting to overlay. And improve the ability for OE to be shipped in setup scripts.
+1 - support the idea!
I'll try to test the patches on my overlay...
+1
Signature · 3 lines
Otavio Salvador O.S. Systems E-mail: [email protected] http://www.ossystems.com.br Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
conf/collections.inc line 4
Quoted 4 lines
conf/collections.inc new file +4
-0+# Take a list of directories in COLLECTIONS, in priority order (highest to +# lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS, +# BBFILE_PATTERN_*, and BBFILE_PRIORITY_*. +
I like this idea, but I think that this *.inc file needs a larger comment block, e.g. not only stating what it does, but also why that would be handy. A typical use case or maybe a problem-solution section.
I think when more of the *.inc and *.bbclass files get such in-file comments it'll be easier for people skimming throught the code to grasp the big picture.
Quoted 10 lines · Holger Schurig
conf/collections.inc line 4
Quoted 4 lines
conf/collections.inc new file +4
-0+# Take a list of directories in COLLECTIONS, in priority order (highest to +# lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS, +# BBFILE_PATTERN_*, and BBFILE_PRIORITY_*. +I like this idea, but I think that this *.inc file needs a larger comment block, e.g. not only stating what it does, but also why that would be handy. A typical use case or maybe a problem-solution section.
Quoted 3 lines
I think when more of the *.inc and *.bbclass files get such in-file comments it'll be easier for people skimming throught the code to grasp the big picture.
Arguably, it could apply to everyone's use cases. It's more convenient to use than BBFILES even when you aren't relying on bitbake's collections support to make it prefer the same recipe in one collection over another, imo.
Signature · 7 lines
Chris Larson clarson at kergoth dot com clarson at mvista dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Software Engineer MontaVista Software, Inc.
Quoted 1 line · On Thursday 02 April 2009 02:27:17 Chris Larson wrote:
COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}"
Why do we need a new variable here? Can't we just use the equivalent BBPATH.split() ?? Or is there a use-case where it makes sense to have BBPATH != COLLECTIONS ?
Quoted 6 lines · Holger Schurig
Quoted 1 line · On Thursday 02 April 2009 02:27:17 Chris Larson wrote:
COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}"
Why do we need a new variable here? Can't we just use the equivalent
BBPATH.split()?? Or is there a use-case where it makes sense to have BBPATH != COLLECTIONS ?
Yes, there are a number of possible cases. For one, consider someone with a ~/oe, under which they have multiple work areas, each with their own trees.. say, poky, a local company specific metadata repo, and upstream oe. Also consider that ~/oe is in BBPATH, so that ~/oe/conf/site.conf can be used for those trees. This is exactly the setup I have on a few of my workstations. If I were to let it use the BBPATH based setup always, I'd always have *all* the recipes from *all* those trees being parsed. Not optimal. In addition, the followup patch to facilitate archive based collections wouldn't be feasable in that case.
Signature · 7 lines
Chris Larson clarson at kergoth dot com clarson at mvista dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Software Engineer MontaVista Software, Inc.