Skip to contentSkip to conversation

Bitbake 1.28-next

PATCH In review Armin Kuster 2 participants 6 messages Sep 16, 2016 – Sep 23, 2016

lore
Share & tools
lore
#0 lore raw

cover Bitbake 1.28-next

Please consider these changes for 1.28
to fix bitbake-selftest failures.

Richard Purdie (1):
bb/tests/fetch: Update cups url

Ross Burton (2):
lib/bb/tests/fetch: remove URL that doesn't exist anymore fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()

 lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
 lib/bb/tests/fetch.py |  7 ++---
 2 files changed, 53 insertions(+), 26 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/3 bb/tests/fetch: Update cups url

From: Richard Purdie <[email protected]>

Update the upstream url used for testing cups versions after upstream website changes.

minor fixup to apply

[Bitbake upstream: 5f06041d4936fc22297945bbbad7020bfa9083c6 ]
---
 lib/bb/tests/fetch.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/bb/tests/fetch.py +2 -2
index 94173c1..73eba8c 100644
@@ -692,8 +692,8 @@ class FetchLatestVersionTest(FetcherTest):
             : "5.0",
         ("xserver-xorg", "http://xorg.freedesktop.org/releases/individual/xserver/xorg-server-1.15.1.tar.bz2", "", "")
             : "1.15.1",
-        # packages with valid REGEX_URI and REGEX
-        ("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "http://www.cups.org/software.php", "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
+        # packages with valid UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX
+        ("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "https://github.com/apple/cups/releases", "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
             : "2.0.0",
         ("db", "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz", "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html", "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
             : "6.1.19",
Signature · 1 line
2.7.4
#2 lore raw

PATCH 2/3 lib/bb/tests/fetch: remove URL that doesn't exist anymore

In reply to Armin Kuster

From: Ross Burton <[email protected]>

The CUPS ipptool URL we were checking now redirects to github where the tarball isn't present, so remove it from the test suite.

(cherry picked from commit 4b50895fb3462b21e3874a2e99c363c8d05e89e6)
---
 lib/bb/tests/fetch.py | 1 -
 1 file changed, 1 deletion(-)
lib/bb/tests/fetch.py +0 -1
index 73eba8c..7a08be0 100644
@@ -726,7 +726,6 @@ class FetchLatestVersionTest(FetcherTest):
 
 class FetchCheckStatusTest(FetcherTest):
     test_wget_uris = ["http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2",
-                      "http://www.cups.org/software/ipptool/ipptool-20130731-linux-ubuntu-i686.tar.gz",
                       "http://www.cups.org/",
                       "http://downloads.yoctoproject.org/releases/sato/sato-engine-0.1.tar.gz",
                       "http://downloads.yoctoproject.org/releases/sato/sato-engine-0.2.tar.gz",
Signature · 1 line
2.7.4
#3 lore raw

PATCH 3/3 fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()

In reply to Armin Kuster

From: Ross Burton <[email protected]>

The core change here is to fall back to GET requests if HEAD is rejected in the checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github archives). This meant removing the monkey patch that the default method was GET and adding a fixed redirect handler that doesn't reset to GET.

Also, change the way the opener is constructed from an if/elif cluster to a conditionally constructed list.

(cherry picked from commit 6ec70d5d2e330b41b932b0a655b838a5f37df01e)
---
 lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
 lib/bb/tests/fetch.py |  2 ++
 2 files changed, 51 insertions(+), 23 deletions(-)
lib/bb/fetch2/wget.py +49 -23
index bd2a897..0ff5b2b 100644
@@ -234,38 +234,64 @@ class Wget(FetchMethod):
 
             return exported
 
-        def head_method(self):
-            return "HEAD"
-
+        class HTTPMethodFallback(urllib2.BaseHandler):
+            """
+            Fallback to GET if HEAD is not allowed (405 HTTP error)
+            """
+            def http_error_405(self, req, fp, code, msg, headers):
+                fp.read()
+                fp.close()
+
+                newheaders = dict((k,v) for k,v in req.headers.items()
+                                  if k.lower() not in ("content-length", "content-type"))
+                return self.parent.open(urllib2.Request(req.get_full_url(),
+                                                        headers=newheaders,
+                                                        origin_req_host=req.get_origin_req_host(),
+                                                        unverifiable=True))
+
+            """
+            Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403
+            Forbidden when they actually mean 405 Method Not Allowed.
+            """
+            http_error_403 = http_error_405
+
+            """
+            Some servers (e.g. FusionForge) returns 406 Not Acceptable when they
+            actually mean 405 Method Not Allowed.
+            """
+            http_error_406 = http_error_405
+
+        class FixedHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
+            """
+            urllib2.HTTPRedirectHandler resets the method to GET on redirect,
+            when we want to follow redirects using the original method.
+            """
+            def redirect_request(self, req, fp, code, msg, headers, newurl):
+                newreq = urllib2.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
+                newreq.get_method = lambda: req.get_method()
+                return newreq
         exported_proxies = export_proxies(d)
 
+        handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
+        if export_proxies:
+            handlers.append(urllib2.ProxyHandler())
+        handlers.append(CacheHTTPHandler())
         # XXX: Since Python 2.7.9 ssl cert validation is enabled by default
         # see PEP-0476, this causes verification errors on some https servers
         # so disable by default.
         import ssl
-        ssl_context = None
         if hasattr(ssl, '_create_unverified_context'):
-            ssl_context = ssl._create_unverified_context()
-
-        if exported_proxies == True and ssl_context is not None:
-            opener = urllib2.build_opener(urllib2.ProxyHandler, CacheHTTPHandler,
-                    urllib2.HTTPSHandler(context=ssl_context))
-        elif exported_proxies == False and ssl_context is not None:
-            opener = urllib2.build_opener(CacheHTTPHandler,
-                    urllib2.HTTPSHandler(context=ssl_context))
-        elif exported_proxies == True and ssl_context is None:
-            opener = urllib2.build_opener(urllib2.ProxyHandler, CacheHTTPHandler)
-        else:
-            opener = urllib2.build_opener(CacheHTTPHandler)
-
-        urllib2.Request.get_method = head_method
-        urllib2.install_opener(opener)
-
-        uri = ud.url.split(";")[0]
+            handlers.append(urllib2.HTTPSHandler(context=ssl._create_unverified_context()))
+        opener = urllib2.build_opener(*handlers)
 
         try:
-            urllib2.urlopen(uri)
-        except:
+            uri = ud.url.split(";")[0]
+            r = urllib2.Request(uri)
+            r.get_method = lambda: "HEAD"
+            opener.open(r)
+        except urllib2.URLError as e:
+            # debug for now to avoid spamming the logs in e.g. remote sstate searches
+            logger.debug(2, "checkstatus() urlopen failed: %s" % e)
             return False
         return True
 
lib/bb/tests/fetch.py +2 -0
index 7a08be0..3b74de0 100644
@@ -737,6 +737,8 @@ class FetchCheckStatusTest(FetcherTest):
                       "ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz",
                       "ftp://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz",
                       "ftp://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz",
+                      # GitHub releases are hosted on Amazon S3, which doesn't support HEAD
+                      "https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz"
                       ]
 
     if os.environ.get("BB_SKIP_NETTESTS") == "yes":
Signature · 1 line
2.7.4
#4 lore raw
In reply to Armin Kuster

sheesh. I thought I was working on bitbake used for jethro, looks like I was off. patches are still applicable.

Quoted 15 lines · Armin Kuster

Please consider these changes for 1.28
to fix bitbake-selftest failures.

Richard Purdie (1):
bb/tests/fetch: Update cups url

Ross Burton (2):
lib/bb/tests/fetch: remove URL that doesn't exist anymore fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()

  lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
  lib/bb/tests/fetch.py |  7 ++---
  2 files changed, 53 insertions(+), 26 deletions(-)
author
#5 lore raw

cover Bitbake 1.28-next

In reply to Armin Kuster

Ping?

-------- Forwarded Message --------
Subject: [PATCH 0/3] Bitbake 1.28-next
Date: Fri, 16 Sep 2016 16:36:41 -0700
From: Armin Kuster <[email protected]>
To: [email protected], [email protected]

Please consider these changes for 1.28
to fix bitbake-selftest failures.

Richard Purdie (1):
bb/tests/fetch: Update cups url

Ross Burton (2):
lib/bb/tests/fetch: remove URL that doesn't exist anymore fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()

 lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
 lib/bb/tests/fetch.py |  7 ++---
 2 files changed, 53 insertions(+), 26 deletions(-)
Signature · 1 line
2.7.4