Skip to contentSkip to conversation

sstate fetch improvements

PATCH In review Christopher Larson 2 participants 7 messages Jun 10, 2013

lore
Share & tools
lore

Christopher Larson

author
#0 lore raw

cover sstate fetch improvements

From: Christopher Larson <[email protected]>

The following changes since commit b4f208f418d18f2a4e78a56bebacef481061d917:

tar: don't mv tar for nativesdk (2013-06-10 18:06:33 +0100)

are available in the git repository at:

https://github.com/kergoth/oe-core.git improve-sstate-fetch

Christopher Larson (3):
sstate.bbclass: kill MIRRORS and FILESPATH to speed up fetching sstate.bbclass: fetch archive+siginfo in a single pstaging_fetch sstate.bbclass: brute force silence fetch errors

 meta/classes/sstate.bbclass | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)
Signature · 1 line
1.8.3
author
#1 lore raw

PATCH 1/3 sstate.bbclass: kill MIRRORS and FILESPATH to speed up fetching

From: Christopher Larson <[email protected]>

The default FILESPATH isn't really of use, as we don't expect to find sstate archives buried in layer recipe directories, and the default MIRRORS is intended for use for fetching SRC_URI, not sstate.

Signed-off-by Christopher Larson <[email protected]>
---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/sstate.bbclass +2 -0
index e74c642..3aa4371 100644
@@ -508,6 +508,8 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
     bb.mkdirhier(dldir)
 
+    localdata.delVar('MIRRORS')
+    localdata.delVar('FILESPATH')
     localdata.setVar('DL_DIR', dldir)
     localdata.setVar('PREMIRRORS', mirrors)
     localdata.setVar('SRC_URI', srcuri)
Signature · 1 line
1.8.3
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] sstate.bbclass: kill MIRRORS and FILESPATH to speed up fetching

Quoted 9 lines · Christopher Larson

From: Christopher Larson <[email protected]>

The default FILESPATH isn't really of use, as we don't expect to find sstate
archives buried in layer recipe directories, and the default MIRRORS is intended for use for fetching SRC_URI, not sstate.

Signed-off-by Christopher Larson <[email protected]>
Reviewed-by Otavio Salvador <[email protected]>
Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
author
#3 lore raw

PATCH 2/3 sstate.bbclass: fetch archive+siginfo in a single pstaging_fetch

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

This avoids unnecessary duplication of setup. The only visible change in behavior will be the case if siginfo exists and the archive does not, in which case it'll redownload both, but this doesn't seem unresasonable to me, particularly since the archive is downloaded first, making this case particularly unlikely.

Signed-off-by Christopher Larson <[email protected]>
---
 meta/classes/sstate.bbclass | 32 +++++++++++++++-----------------
 1 file changed, 15 insertions(+), 17 deletions(-)
meta/classes/sstate.bbclass +15 -17
index 3aa4371..28dc312 100644
@@ -224,8 +224,6 @@ def sstate_installpkg(ss, d):
 
     if not os.path.exists(sstatepkg):
         pstaging_fetch(sstatefetch, sstatepkg, d)
-    if not os.path.exists(sstatepkg + ".siginfo"):
-        pstaging_fetch(sstatefetch + ".siginfo", sstatepkg + ".siginfo", d)
 
     if not os.path.isfile(sstatepkg):
         bb.note("Staging package %s does not exist" % sstatepkg)

@@ -504,30 +502,30 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
     bb.data.update_data(localdata)
 
     dldir = localdata.expand("${SSTATE_DIR}")
-    srcuri = "file://" + sstatefetch
-
     bb.mkdirhier(dldir)
 
     localdata.delVar('MIRRORS')
     localdata.delVar('FILESPATH')
     localdata.setVar('DL_DIR', dldir)
     localdata.setVar('PREMIRRORS', mirrors)
-    localdata.setVar('SRC_URI', srcuri)
 
     # Try a fetch from the sstate mirror, if it fails just return and
     # we will build the package
-    try:
-        fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
-        fetcher.download()        
-
-        # Need to optimise this, if using file:// urls, the fetcher just changes the local path
-        # For now work around by symlinking
-        localpath = bb.data.expand(fetcher.localpath(srcuri), localdata)
-        if localpath != sstatepkg and os.path.exists(localpath) and not os.path.exists(sstatepkg):
-            os.symlink(localpath, sstatepkg)
-
-    except bb.fetch2.BBFetchException:
-        pass
+    for srcuri in ['file://{0}'.format(sstatefetch),
+                   'file://{0}.siginfo'.format(sstatefetch)]:
+        localdata.setVar('SRC_URI', srcuri)
+        try:
+            fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher.download()
+
+            # Need to optimise this, if using file:// urls, the fetcher just changes the local path
+            # For now work around by symlinking
+            localpath = bb.data.expand(fetcher.localpath(srcuri), localdata)
+            if localpath != sstatepkg and os.path.exists(localpath) and not os.path.exists(sstatepkg):
+                os.symlink(localpath, sstatepkg)
+
+        except bb.fetch2.BBFetchException:
+            break
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
Signature · 1 line
1.8.3
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] sstate.bbclass: fetch archive+siginfo in a single pstaging_fetch

Quoted 11 lines · Christopher Larson

From: Christopher Larson <[email protected]>

This avoids unnecessary duplication of setup. The only visible change in behavior will be the case if siginfo exists and the archive does not, in which
case it'll redownload both, but this doesn't seem unresasonable to me, particularly since the archive is downloaded first, making this case particularly unlikely.

Signed-off-by Christopher Larson <[email protected]>
Reviewed-by Otavio Salvador <[email protected]>
Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750
author
#5 lore raw

PATCH 3/3 sstate.bbclass: brute force silence fetch errors

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

bb.fetch2.Fetcher.download() prints any fetch errors that come from the underlying fetcher, rather than letting the caller do so, which means there's no way to silence the fetch error messages without manipulating the loggers. So we do so. The reason for doing this is that the user doesn't care if we failed to fetch an sstate archive, the failure mode is just to rerun the real task anyway.

Signed-off-by Christopher Larson <[email protected]>
---
 meta/classes/sstate.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes/sstate.bbclass +5 -0
index 28dc312..5b9d86e 100644
@@ -491,6 +491,7 @@ def sstate_package(ss, d):
 
 def pstaging_fetch(sstatefetch, sstatepkg, d):
     import bb.fetch2
+    import logging
 
     # Only try and fetch if the user has configured a mirror
     mirrors = d.getVar('SSTATE_MIRRORS', True)

@@ -514,6 +515,8 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
     for srcuri in ['file://{0}'.format(sstatefetch),
                    'file://{0}.siginfo'.format(sstatefetch)]:
         localdata.setVar('SRC_URI', srcuri)
+        oldlevel = bb.fetch2.logger.level
+        bb.fetch2.logger.setLevel(logging.CRITICAL)
         try:
             fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
             fetcher.download()

@@ -526,6 +529,8 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
         except bb.fetch2.BBFetchException:
             break
+        finally:
+            bb.fetch2.logger.setLevel(oldlevel)
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
Signature · 1 line
1.8.3
Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] sstate.bbclass: brute force silence fetch errors

Quoted 12 lines · Christopher Larson

From: Christopher Larson <[email protected]>

bb.fetch2.Fetcher.download() prints any fetch errors that come from the underlying fetcher, rather than letting the caller do so, which means there's
no way to silence the fetch error messages without manipulating the loggers.
So we do so. The reason for doing this is that the user doesn't care if we failed to fetch an sstate archive, the failure mode is just to rerun the real
task anyway.

Quoted 2 lines
Signed-off-by Christopher Larson <[email protected]>
Reviewed-by Otavio Salvador <[email protected]>
Signature · 3 lines
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750