Skip to contentSkip to conversation

Add oe.fetch, rework fetch/unpack deps from SRC_URI

PATCH In review Chris Larson 1 participant 5 messages Apr 5, 2011 – Apr 6, 2011

lore
Share & tools
lore
#0 lore raw

cover Add oe.fetch, rework fetch/unpack deps from SRC_URI

From: Chris Larson <[email protected]>

This was intended to clean up and consolidate the code in base.bbclass which sets up dependencies conditionally based upon the contents of SRC_URI. To get there, I decided to incorporate and rewrite some Uri handling code I had written previously. This is much more convenient to use than urldata and SRC_URI, and the caching can be easily re-implemented if we move it into bitbake at some point (e.g. create an lru cache decorator, inject its construction/sync into the standard caching spots where cache/signature code caching is done, if necessary). Finally, this makes .zip files pull in unzip-native.

Pull URL: https://github.com/kergoth/oe-core
Branch: unpack-fetch-deps
Browse: https://github.com/kergoth/oe-core/commits/unpack-fetch-deps

Thanks,
Chris Larson <[email protected]>
---

Chris Larson (3):
oe.fetch: add module with uri convenience code Clean up how we handle unpack/fetch deps from SRC_URI
base: when using zip files, depend on unzip-native

 meta/classes/base.bbclass |   58 ++++++++++++++++--------------
 meta/conf/bitbake.conf    |    1 +
 meta/lib/oe/fetch.py      |   87 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 119 insertions(+), 27 deletions(-)
 create mode 100644 meta/lib/oe/fetch.py
Signature · 1 line
1.7.4.1
#1 lore raw

PATCH 1/3 oe.fetch: add module with uri convenience code

From: Chris Larson <[email protected]>

---
 meta/conf/bitbake.conf |    1 +
 meta/lib/oe/fetch.py   |   87 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 88 insertions(+), 0 deletions(-)
 create mode 100644 meta/lib/oe/fetch.py
meta/conf/bitbake.conf +1 -0
index 27b8a6b..27b4f74 100644
@@ -527,6 +527,7 @@ AUTOREV = "${@bb.fetch2.get_autorev(d)}"
 SRCPV = "${@bb.fetch2.get_srcrev(d)}"
 
 SRC_URI = "file://${FILE}"
+SRC_URI[type] = "list"
 
 # We can choose which provider of fake root privileges to use
 # default is fakeroot but in Poky we use pseudo
meta/lib/oe/fetch.py new file +87 -0

87 lines not on this page · raw message

Signature · 1 line
1.7.4.1
#2 lore raw

PATCH 2/3 Clean up how we handle unpack/fetch deps from SRC_URI

In reply to Chris Larson

From: Chris Larson <[email protected]>

---
 meta/classes/base.bbclass |   56 +++++++++++++++++++++++---------------------
 1 files changed, 29 insertions(+), 27 deletions(-)
meta/classes/base.bbclass +29 -27
index 9c9fc7c..0a1aebf 100644
@@ -254,9 +254,37 @@ do_build () {
 	:
 }
 
+
+def set_uri_depends(d):
+    import oe.fetch
+
+    unpack_depends = set()
+    fetch_depends = set()
+    for uri in oe.fetch.src_uri(d):
+	if uri.scheme == 'git':
+	    fetch_depends.add('git-native')
+	elif uri.scheme == 'hg':
+	    fetch_depends.add('mercurial-native')
+	elif uri.scheme == 'osc':
+	    fetch_depends.add('osc-native')
+
+	root, ext = os.path.splitext(uri.path)
+	if ext == '.xz':
+	    unpack_depends.add('xz-native')
+
+    fdepends = d.getVarFlag('do_fetch', 'depends') or ''
+    fdepends += ' '.join('%s:do_populate_sysroot' % f for f in fetch_depends)
+    d.setVarFlag('do_fetch', 'depends', fdepends)
+
+    udepends = d.getVarFlag('do_unpack', 'depends') or ''
+    udepends += ' '.join('%s:do_populate_sysroot' % u for u in unpack_depends)
+    d.setVarFlag('do_unpack', 'depends', udepends)
+
 python () {
     import exceptions, string
 
+    set_uri_depends(d)
+
     # If PRINC is set, try and increase the PR value by the amount specified
     princ = bb.data.getVar('PRINC', d, True)
     if princ:
@@ -341,25 +369,6 @@ python () {
     if use_nls != None:
         bb.data.setVar('USE_NLS', use_nls, d)
 
-    # Git packages should DEPEND on git-native
-    srcuri = bb.data.getVar('SRC_URI', d, 1)
-    if "git://" in srcuri:
-        depends = bb.data.getVarFlag('do_fetch', 'depends', d) or ""
-        depends = depends + " git-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_fetch', 'depends', depends, d)
-
-    # Mercurial packages should DEPEND on mercurial-native
-    elif "hg://" in srcuri:
-        depends = bb.data.getVarFlag('do_fetch', 'depends', d) or ""
-        depends = depends + " mercurial-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_fetch', 'depends', depends, d)
-
-    # OSC packages should DEPEND on osc-native
-    elif "osc://" in srcuri:
-        depends = bb.data.getVarFlag('do_fetch', 'depends', d) or ""
-        depends = depends + " osc-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_fetch', 'depends', depends, d)
-
     # bb.utils.sha256_file() will fail if hashlib isn't present, so we fallback
     # on shasum-native.  We need to ensure that it is staged before we fetch.
     if bb.data.getVar('PN', d, True) != "shasum-native":
@@ -370,13 +379,6 @@ python () {
             depends = depends + " shasum-native:do_populate_sysroot"
             bb.data.setVarFlag('do_fetch', 'depends', depends, d)
 
-    # *.xz should depends on xz-native for unpacking
-    # Not endswith because of "*.patch.xz;patch=1". Need bb.decodeurl in future
-    if '.xz' in srcuri:
-        depends = bb.data.getVarFlag('do_unpack', 'depends', d) or ""
-        depends = depends + " xz-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_unpack', 'depends', depends, d)
-
     # 'multimachine' handling
     mach_arch = bb.data.getVar('MACHINE_ARCH', d, 1)
     pkg_arch = bb.data.getVar('PACKAGE_ARCH', d, 1)
@@ -432,7 +434,7 @@ def check_gcc3(data):
 	for gcc3 in gcc3_versions.split():
 		if check_app_exists(gcc3, data):
 			return gcc3
-	
+
 	return False
 
 addtask cleanall after do_clean
Signature · 1 line
1.7.4.1
#3 lore raw

PATCH 3/3 base: when using zip files, depend on unzip-native

In reply to Chris Larson

From: Chris Larson <[email protected]>

---
 meta/classes/base.bbclass |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
meta/classes/base.bbclass +3 -1
index 0a1aebf..7a11c9a 100644
@@ -269,7 +269,9 @@ def set_uri_depends(d):
 	    fetch_depends.add('osc-native')
 
 	root, ext = os.path.splitext(uri.path)
-	if ext == '.xz':
+	if ext == '.zip':
+	    unpack_depends.add('unzip-native')
+	elif ext == '.xz':
 	    unpack_depends.add('xz-native')
 
     fdepends = d.getVarFlag('do_fetch', 'depends') or ''
Signature · 1 line
1.7.4.1
#4 lore raw
In reply to Chris Larson
Quoted 9 lines · Chris Larson

This was intended to clean up and consolidate the code in base.bbclass which sets up dependencies conditionally based upon the contents of SRC_URI.  To get there, I decided to incorporate and rewrite some Uri handling code I had written previously.  This is much more convenient to use than urldata and SRC_URI, and the caching can be easily re-implemented if we move it into bitbake at some point (e.g. create an lru cache decorator, inject its construction/sync into the standard caching spots where cache/signature code caching is done, if necessary).  Finally, this makes .zip files pull in unzip-native.

Ugh, ignore this patch, at least for now. The url handling code has some problems with our nonstandard url parameters. *rolls eyes and mutters*. Will send the unpack/patch deps cleanup separately.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics