fix do_ar_original (multiple git sources, absolute path)

meta-oic's iotivity_1.1.1.bb recently started using destsuffix=${S} for different git source URLs. This breaks randomly (depending on task scheduling) when archive.bbclass and ARCHIVER_MODE[src] = "original" are active, because the archiver then unpacks again into ${S} in parallel to the running build.

While fixing that it was noticed that the archiver also did not archive all upstream sources because it used the same .tar.gz filename for all git repos.

The following changes since commit 1e13d1fd22186af5544e7248dc12635cd2f2e08b:

meta-environment: ensure corret TOOLCHAIN_CONFIGSITE_NOCACHE value (2016-09-26 10:05:53 +0100)

are available in the git repository at:

git://github.com/pohly/openembedded-core do_ar_original
https://github.com/pohly/openembedded-core/tree/do_ar_original

Patrick Ohly (2):
archiver.bbclass: ignore unpack sub-directories in do_ar_original archive.bbclass: fix do_ar_original archiving of multiple source repos

 meta/classes/archiver.bbclass | 37 +++++++++++++++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)

2.1.4

Support for absolute paths in the "subdir" parameter was recently added (bitbake rev: c3873346c6fa). The git fetcher has supported absolute paths in "destsuffix" already before.

When the path is absolute as in destsuffix=${S}/foobar, the tmpdir used by do_ar_original gets ignored, which breaks:
- source code archiving (tmpdir is empty)
- compilation due to race conditions (for example, ${S} getting modified by do_ar_original while do_compile runs)

To solve this, these parameters get removed from URLs before instantiating the fetcher for them.

This is done unconditionally also for relative paths, because these paths are not useful when archiving the original source (upstream source does not have them, they only get used by the recipe during compilation).

---
 meta/classes/archiver.bbclass | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

index 2f3b278..5f9c91d 100644
@@ -132,7 +132,25 @@ python do_ar_original() {
 
     ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
     bb.note('Archiving the original source...')
-    fetch = bb.fetch2.Fetch([], d)
+    urls = d.getVar("SRC_URI", True).split()
+    # destsuffix (git fetcher) and subdir (everything else) are allowed to be
+    # absolute paths (for example, destsuffix=${S}/foobar).
+    # That messes with unpacking inside our tmpdir below, because the fetchers
+    # will then unpack in that directory and completely ignore the tmpdir.
+    # That breaks parallel tasks relying on ${S}, like do_compile.
+    #
+    # To solve this, we remove these parameters from all URLs.
+    # We do this even for relative paths because it makes the content of the
+    # archives more useful (no extra paths that are only used during
+    # compilation).
+    for i, url in enumerate(urls):
+        decoded = bb.fetch2.decodeurl(url)
+        for param in ('destsuffix', 'subdir'):
+            if param in decoded[5]:
+                del decoded[5][param]
+        encoded = bb.fetch2.encodeurl(decoded)
+        urls[i] = encoded
+    fetch = bb.fetch2.Fetch(urls, d)
     for url in fetch.urls:
         local = fetch.localpath(url).rstrip("/");
         if os.path.isfile(local):

2.1.4

When a recipe uses more than one source which isn't a plain file (for example, multiple git repos), then do_ar_original created the source archives using the same filename and thus only archived one source.

The "name" parameter is used as file suffix to create unique names for each source, leading to archives following this pattern:
deploy/${TARGET_SYS}/${PF}/${PF}[-<name>].tar.gz.

The ${PF} part is a bit redundant, which may or may not be desirable. The patch is more localized this way (no need to modify create_tarball()).

For example, meta-oic's iotivity_1.1.1.bb uses:

   url_iotivity = "git://github.com/iotivity/iotivity.git"
   branch_iotivity = "1.1-rel"
   SRC_URI = "${url_iotivity};destsuffix=${S};branch=${branch_iotivity};protocol=http;"

   url_tinycbor = "git://github.com/01org/tinycbor.git"
   SRC_URI += "${url_tinycbor};name=tinycbor;destsuffix=${S}/extlibs/tinycbor/tinycbor;protocol=http"

   url_hippomocks = "git://github.com/dascandy/hippomocks.git"
   SRC_URI += "${url_hippomocks};name=hippomocks;destsuffix=${S}/extlibs/hippomocks-master;protocol=http"
   SRC_URI += "file://hippomocks_mips_patch"

   url_gtest = "http://pkgs.fedoraproject.org/repo/pkgs/gtest/gtest-1.7.0.zip/2d6ec8ccdf5c46b05ba54a9fd1d130d7/gtest-1.7.0.zip"
   SRC_URI += "${url_gtest};name=gtest;subdir=${BP}/extlibs/gtest"

   url_sqlite = "http://www.sqlite.org/2015/sqlite-amalgamation-3081101.zip"
   SRC_URI += "${url_sqlite};name=sqlite3;subdir=${BP}/extlibs/sqlite3;unpack=false"

These now get archived in deploy/sources/*/iotivity-1.1.1-r2/ as:

gtest-1.7.0.zip                      iotivity-1.1.1-r2-recipe.tar.gz    sqlite-amalgamation-3081101.zip
hippomocks_mips_patch                iotivity-1.1.1-r2.tar.gz
iotivity-1.1.1-r2-hippomocks.tar.gz  iotivity-1.1.1-r2-tinycbor.tar.gz

---
 meta/classes/archiver.bbclass | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

index 5f9c91d..1d8e863 100644
@@ -151,6 +151,7 @@ python do_ar_original() {
         encoded = bb.fetch2.encodeurl(decoded)
         urls[i] = encoded
     fetch = bb.fetch2.Fetch(urls, d)
+    tarball_suffix = {}
     for url in fetch.urls:
         local = fetch.localpath(url).rstrip("/");
         if os.path.isfile(local):

@@ -158,7 +159,21 @@ python do_ar_original() {
         elif os.path.isdir(local):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR', True))
             fetch.unpack(tmpdir, (url,))
-            create_tarball(d, tmpdir + '/.', '', ar_outdir)
+            # To handle recipes with more than one source, we add the "name"
+            # URL parameter as suffix. We treat it as an error when
+            # there's more than one URL without a name, or a name gets reused.
+            # This is an additional safety net, in practice the name has
+            # to be set when using the git fetcher, otherwise SRCREV cannot
+            # be set separately for each URL.
+            params = bb.fetch2.decodeurl(url)[5]
+            name = params.get('name', '')
+            if name in tarball_suffix:
+                if not name:
+                    bb.fatal("Cannot determine archive names for original source because 'name' URL parameter is unset in more than one URL. Add it to at least one of these: %s %s" % (tarball_suffix[name], url))
+                else:
+                    bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (tarball_suffix[name], url))
+            tarball_suffix[name] = url
+            create_tarball(d, tmpdir + '/.', name, ar_outdir)
 
     # Emit patch series files for 'original'
     bb.note('Writing patch series files...')

2.1.4