Add oe.fetch, rework fetch/unpack deps from SRC_URI
cover Add oe.fetch, rework fetch/unpack deps from SRC_URI
From: Chris Larson <[email protected]>
This was intended to clean up and consolidate the code in base.bbclass which sets up dependencies conditionally based upon the contents of SRC_URI. To get there, I decided to incorporate and rewrite some Uri handling code I had written previously. This is much more convenient to use than urldata and SRC_URI, and the caching can be easily re-implemented if we move it into bitbake at some point (e.g. create an lru cache decorator, inject its construction/sync into the standard caching spots where cache/signature code caching is done, if necessary). Finally, this makes .zip files pull in unzip-native.
Pull URL: https://github.com/kergoth/oe-core
Branch: unpack-fetch-deps
Browse: https://github.com/kergoth/oe-core/commits/unpack-fetch-deps
Thanks,
Chris Larson <[email protected]>
---
Chris Larson (3):
oe.fetch: add module with uri convenience code Clean up how we handle unpack/fetch deps from SRC_URI
base: when using zip files, depend on unzip-native
meta/classes/base.bbclass | 58 ++++++++++++++++-------------- meta/conf/bitbake.conf | 1 + meta/lib/oe/fetch.py | 87 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+), 27 deletions(-) create mode 100644 meta/lib/oe/fetch.py
Signature · 1 line
1.7.4.1
PATCH 1/3 oe.fetch: add module with uri convenience code
From: Chris Larson <[email protected]>
--- meta/conf/bitbake.conf | 1 + meta/lib/oe/fetch.py | 87 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 0 deletions(-) create mode 100644 meta/lib/oe/fetch.py
meta/conf/bitbake.conf +1 -0
@@ -527,6 +527,7 @@ AUTOREV = "${@bb.fetch2.get_autorev(d)}" SRCPV = "${@bb.fetch2.get_srcrev(d)}" SRC_URI = "file://${FILE}" +SRC_URI[type] = "list" # We can choose which provider of fake root privileges to use # default is fakeroot but in Poky we use pseudo
meta/lib/oe/fetch.py new file +87 -0
87 lines not on this page · raw message
Signature · 1 line
1.7.4.1
PATCH 2/3 Clean up how we handle unpack/fetch deps from SRC_URI
In reply to Chris LarsonFrom: Chris Larson <[email protected]>
--- meta/classes/base.bbclass | 56 +++++++++++++++++++++++--------------------- 1 files changed, 29 insertions(+), 27 deletions(-)
meta/classes/base.bbclass +29 -27
@@ -254,9 +254,37 @@ do_build () { : } + +def set_uri_depends(d): + import oe.fetch + + unpack_depends = set() + fetch_depends = set() + for uri in oe.fetch.src_uri(d): + if uri.scheme == 'git': + fetch_depends.add('git-native') + elif uri.scheme == 'hg': + fetch_depends.add('mercurial-native') + elif uri.scheme == 'osc': + fetch_depends.add('osc-native') + + root, ext = os.path.splitext(uri.path) + if ext == '.xz': + unpack_depends.add('xz-native') + + fdepends = d.getVarFlag('do_fetch', 'depends') or '' + fdepends += ' '.join('%s:do_populate_sysroot' % f for f in fetch_depends) + d.setVarFlag('do_fetch', 'depends', fdepends) + + udepends = d.getVarFlag('do_unpack', 'depends') or '' + udepends += ' '.join('%s:do_populate_sysroot' % u for u in unpack_depends) + d.setVarFlag('do_unpack', 'depends', udepends) + python () { import exceptions, string + set_uri_depends(d) + # If PRINC is set, try and increase the PR value by the amount specified princ = bb.data.getVar('PRINC', d, True) if princ:
@@ -341,25 +369,6 @@ python () { if use_nls != None: bb.data.setVar('USE_NLS', use_nls, d) - # Git packages should DEPEND on git-native - srcuri = bb.data.getVar('SRC_URI', d, 1) - if "git://" in srcuri: - depends = bb.data.getVarFlag('do_fetch', 'depends', d) or "" - depends = depends + " git-native:do_populate_sysroot" - bb.data.setVarFlag('do_fetch', 'depends', depends, d) - - # Mercurial packages should DEPEND on mercurial-native - elif "hg://" in srcuri: - depends = bb.data.getVarFlag('do_fetch', 'depends', d) or "" - depends = depends + " mercurial-native:do_populate_sysroot" - bb.data.setVarFlag('do_fetch', 'depends', depends, d) - - # OSC packages should DEPEND on osc-native - elif "osc://" in srcuri: - depends = bb.data.getVarFlag('do_fetch', 'depends', d) or "" - depends = depends + " osc-native:do_populate_sysroot" - bb.data.setVarFlag('do_fetch', 'depends', depends, d) - # bb.utils.sha256_file() will fail if hashlib isn't present, so we fallback # on shasum-native. We need to ensure that it is staged before we fetch. if bb.data.getVar('PN', d, True) != "shasum-native":
@@ -370,13 +379,6 @@ python () { depends = depends + " shasum-native:do_populate_sysroot" bb.data.setVarFlag('do_fetch', 'depends', depends, d) - # *.xz should depends on xz-native for unpacking - # Not endswith because of "*.patch.xz;patch=1". Need bb.decodeurl in future - if '.xz' in srcuri: - depends = bb.data.getVarFlag('do_unpack', 'depends', d) or "" - depends = depends + " xz-native:do_populate_sysroot" - bb.data.setVarFlag('do_unpack', 'depends', depends, d) - # 'multimachine' handling mach_arch = bb.data.getVar('MACHINE_ARCH', d, 1) pkg_arch = bb.data.getVar('PACKAGE_ARCH', d, 1)
@@ -432,7 +434,7 @@ def check_gcc3(data): for gcc3 in gcc3_versions.split(): if check_app_exists(gcc3, data): return gcc3 - + return False addtask cleanall after do_clean
Signature · 1 line
1.7.4.1
PATCH 3/3 base: when using zip files, depend on unzip-native
In reply to Chris LarsonFrom: Chris Larson <[email protected]>
--- meta/classes/base.bbclass | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
meta/classes/base.bbclass +3 -1
@@ -269,7 +269,9 @@ def set_uri_depends(d): fetch_depends.add('osc-native') root, ext = os.path.splitext(uri.path) - if ext == '.xz': + if ext == '.zip': + unpack_depends.add('unzip-native') + elif ext == '.xz': unpack_depends.add('xz-native') fdepends = d.getVarFlag('do_fetch', 'depends') or ''
Signature · 1 line
1.7.4.1
Quoted 9 lines · Chris Larson
This was intended to clean up and consolidate the code in base.bbclass which sets up dependencies conditionally based upon the contents of
SRC_URI. To get there, I decided to incorporate and rewrite some Uri handling code I had written previously. This is much more convenient to use than urldata andSRC_URI, and the caching can be easily re-implemented if we move it into bitbake at some point (e.g. create an lru cache decorator, inject its construction/sync into the standard caching spots where cache/signature code caching is done, if necessary). Finally, this makes .zip files pull in unzip-native.
Ugh, ignore this patch, at least for now. The url handling code has some problems with our nonstandard url parameters. *rolls eyes and mutters*. Will send the unpack/patch deps cleanup separately.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics