Skip to contentSkip to conversation

fetch2: Add new mirror syntax to simplify mirror specifications

PATCH In review Richard Purdie 4 participants 12 messages Jun 21, 2012 – Jun 28, 2012

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fetch2: Add new mirror syntax to simplify mirror specifications

When writing mirror specifications, the current regexp syntax can be awkward and hard to get it to do what you want. For example, extracting the 'basename' of a repository:

PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"

can now become:

PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"

which is much clearer. A MIRRORNAME substitution is also added which contains an encoded form of both host and path. One of the problems with the existing regexp syntax is you couldn't access HOST information from PATH and vice-versa which is an issue this patch also addresses.

Tests for the new syntax are also added.

---

bitbake/lib/bb/fetch2/__init__.py +11 -2
index 1b99a0f..e6f55f8 100644
@@ -182,7 +182,7 @@ def encodeurl(decoded):
 
     return url
 
-def uri_replace(ud, uri_find, uri_replace, d):
+def uri_replace(ud, uri_find, uri_replace, replacements, d):
     if not ud.url or not uri_find or not uri_replace:
         logger.error("uri_replace: passed an undefined value, not replacing")
         return None
@@ -213,6 +213,8 @@ def uri_replace(ud, uri_find, uri_replace, d):
             if not uri_replace_decoded[loc]:
                 result_decoded[loc] = ""    
             else:
+                for k in replacements:
+                    uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
                 #bb.note("%s %s %s" % (i, uri_replace_decoded[loc], uri_decoded[loc]))
                 result_decoded[loc] = re.sub(i, uri_replace_decoded[loc], uri_decoded[loc])
             if loc == 2:
@@ -485,13 +487,20 @@ def build_mirroruris(origud, mirrors, ld):
     uris = []
     uds = []
 
+    replacements = {}
+    replacements["TYPE"] = origud.type
+    replacements["HOST"] = origud.host
+    replacements["PATH"] = origud.path
+    replacements["BASENAME"] = origud.path.split("/")[-1]
+    replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.')
+
     def adduri(uri, ud, uris, uds):
         for line in mirrors:
             try:
                 (find, replace) = line
             except ValueError:
                 continue
-            newuri = uri_replace(ud, find, replace, ld)
+            newuri = uri_replace(ud, find, replace, replacements, ld)
             if not newuri or newuri in uris or newuri == origud.url:
                 continue
             uris.append(newuri)
bitbake/lib/bb/tests/fetch.py +11 -3
index 9961343..1477fab 100644
@@ -48,7 +48,14 @@ class FetcherTest(unittest.TestCase):
         ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://www.apache.org/dist", "http://archive.apache.org/dist")
             : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
         ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
-            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2"
+            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
+        ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
+            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
+            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") 
+            : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+
         #Renaming files doesn't work
         #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
         #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
@@ -143,8 +150,9 @@ class FetcherTest(unittest.TestCase):
         for k, v in self.replaceuris.items():
             ud = bb.fetch.FetchData(k[0], self.d)
             ud.setup_localpath(self.d)
-            newuris = bb.fetch2.uri_replace(ud, k[1], k[2], self.d)
-            self.assertEqual(newuris, v)
+            mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
+            newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
+            self.assertEqual(newuris, [v])
 
     def test_urilist1(self):
         fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
#1 lore raw

(PRE)MIRRORS doesn't work with https?$:// anymore Was: [PATCH] fetch2: Add new mirror syntax to simplify mirror specifications

Quoted 6 lines · Richard Purdie

When writing mirror specifications, the current regexp syntax can be awkward and hard to get it to do what you want. For example, extracting the 'basename' of a repository:

PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"

Quoted 8 lines

can now become:

PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"

which is much clearer. A MIRRORNAME substitution is also added which contains an encoded form of both host and path. One of the problems with the existing regexp syntax is you couldn't access HOST information from PATH and vice-versa which is an issue this patch also addresses.

One of this fetch2 patches break https?:// in MIRRORS

It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
https?$://.*/.* http://sources.openembedded.org/

I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch log shorter) on libcap (which doesn't exist on upstream URL).

PREMIRRORS_append () {
cvs://.*/.* http://build.shr-project.org/sources/
svn://.*/.* http://build.shr-project.org/sources/
git://.*/.* http://build.shr-project.org/sources/
hg://.*/.* http://build.shr-project.org/sources/
bzr://.*/.* http://build.shr-project.org/sources/
https?$://.*/.* http://build.shr-project.org/sources/
}

bitbake d316f28ed725ff40daa8771c1aa224ac46d5b224:
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished

bitbake e6ff1d4bab43fdcd8af1230f1d54615f53c1978e
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2
...and after trying Upstream and all MIRRORS (none will match) it fails to fetch it..

Removing '$' doesn't work too:
PREMIRRORS_append () {
https?://.*/.* http://build.shr-project.org/sources/
}
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2

Using http$ doesn't work too
EBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream

And without any regexp with different PREMIRRORS for http and https it also does weird things (looks like trying to resolve returned PREMIRROR again) PREMIRRORS_append () {
http://.*/.* http://build.shr-project.org/sources/
https://.*/.* http://build.shr-project.org/https-sources/
}

changed libcap SRC_URI to https:
-SRC_URI = "${KERNELORG_MIRROR}/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2" +SRC_URI = "https://foo.org/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"

DEBUG: Trying PREMIRRORS
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
NOTE: fetch http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Mirror fetch failure for url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 (original url: https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2)
DEBUG: Fetcher failure: Fetch command export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2' failed with exit code 8, output:
STDOUT:
STDERR: http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2:

2012-06-26 19:23:37 ERROR 404: Not Found.

NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished

Cheers,

Quoted 88 lines

Tests for the new syntax are also added.

---

bitbake/lib/bb/fetch2/__init__.py +11 -2
index 1b99a0f..e6f55f8 100644
@@ -182,7 +182,7 @@ def encodeurl(decoded):
 
     return url
 
-def uri_replace(ud, uri_find, uri_replace, d):
+def uri_replace(ud, uri_find, uri_replace, replacements, d):
     if not ud.url or not uri_find or not uri_replace:
         logger.error("uri_replace: passed an undefined value, not replacing")
         return None
@@ -213,6 +213,8 @@ def uri_replace(ud, uri_find, uri_replace, d):
             if not uri_replace_decoded[loc]:
                 result_decoded[loc] = ""    
             else:
+                for k in replacements:
+                    uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
                 #bb.note("%s %s %s" % (i, uri_replace_decoded[loc], uri_decoded[loc]))
                 result_decoded[loc] = re.sub(i, uri_replace_decoded[loc], uri_decoded[loc])
             if loc == 2:
@@ -485,13 +487,20 @@ def build_mirroruris(origud, mirrors, ld):
     uris = []
     uds = []
 
+    replacements = {}
+    replacements["TYPE"] = origud.type
+    replacements["HOST"] = origud.host
+    replacements["PATH"] = origud.path
+    replacements["BASENAME"] = origud.path.split("/")[-1]
+    replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.')
+
     def adduri(uri, ud, uris, uds):
         for line in mirrors:
             try:
                 (find, replace) = line
             except ValueError:
                 continue
-            newuri = uri_replace(ud, find, replace, ld)
+            newuri = uri_replace(ud, find, replace, replacements, ld)
             if not newuri or newuri in uris or newuri == origud.url:
                 continue
             uris.append(newuri)
bitbake/lib/bb/tests/fetch.py +11 -3
index 9961343..1477fab 100644
@@ -48,7 +48,14 @@ class FetcherTest(unittest.TestCase):
         ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://www.apache.org/dist", "http://archive.apache.org/dist")
             : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
         ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
-            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2"
+            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
+        ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
+            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
+            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") 
+            : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
+
         #Renaming files doesn't work
         #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
         #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
@@ -143,8 +150,9 @@ class FetcherTest(unittest.TestCase):
         for k, v in self.replaceuris.items():
             ud = bb.fetch.FetchData(k[0], self.d)
             ud.setup_localpath(self.d)
-            newuris = bb.fetch2.uri_replace(ud, k[1], k[2], self.d)
-            self.assertEqual(newuris, v)
+            mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
+            newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
+            self.assertEqual(newuris, [v])
 
     def test_urilist1(self):
         fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)

_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#2 lore raw

Re: (PRE)MIRRORS doesn't work with https?$:// anymore Was: [PATCH] fetch2: Add new mirror syntax to simplify mirror specifications

Quoted 25 lines · Martin Jansa
Quoted 6 lines · Richard Purdie

When writing mirror specifications, the current regexp syntax can be awkward and hard to get it to do what you want. For example, extracting the 'basename' of a repository:

PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"

Quoted 8 lines

can now become:

PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"

which is much clearer. A MIRRORNAME substitution is also added which contains an encoded form of both host and path. One of the problems with the existing regexp syntax is you couldn't access HOST information from PATH and vice-versa which is an issue this patch also addresses.

One of this fetch2 patches break https?:// in MIRRORS

It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
https?$://.*/.* http://sources.openembedded.org/

I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch log shorter) on libcap (which doesn't exist on upstream URL).

Quoted 8 lines

PREMIRRORS_append () {
cvs://.*/.* http://build.shr-project.org/sources/
svn://.*/.* http://build.shr-project.org/sources/
git://.*/.* http://build.shr-project.org/sources/
hg://.*/.* http://build.shr-project.org/sources/
bzr://.*/.* http://build.shr-project.org/sources/
https?$://.*/.* http://build.shr-project.org/sources/
}

This is almost certainly a result of:

http://git.openembedded.org/bitbake/commit/?id=604df1b25cf114e083f52917df2df64e01279c25

since I didn't believe anyone was using the regexp syntax in the base url type. The bug I wanted to fix was where if you did:

https://.*/.* file:///some/path/

which seems reasonable enough, it would access:

files:///some/path/

which is clearly not what anyone would want.

I'm not sure what to do about fixing this...

Cheers,

Richard

#3 lore raw

Re: (PRE)MIRRORS doesn't work with https?$:// anymore Was: [PATCH] fetch2: Add new mirror syntax to simplify mirror specifications

Quoted 44 lines · Richard Purdie
Quoted 25 lines · Martin Jansa
Quoted 6 lines · Richard Purdie

When writing mirror specifications, the current regexp syntax can be awkward and hard to get it to do what you want. For example, extracting the 'basename' of a repository:

PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"

Quoted 8 lines

can now become:

PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"

which is much clearer. A MIRRORNAME substitution is also added which contains an encoded form of both host and path. One of the problems with the existing regexp syntax is you couldn't access HOST information from PATH and vice-versa which is an issue this patch also addresses.

One of this fetch2 patches break https?:// in MIRRORS

It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
https?$://.*/.* http://sources.openembedded.org/

I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch log shorter) on libcap (which doesn't exist on upstream URL).

Quoted 8 lines

PREMIRRORS_append () {
cvs://.*/.* http://build.shr-project.org/sources/
svn://.*/.* http://build.shr-project.org/sources/
git://.*/.* http://build.shr-project.org/sources/
hg://.*/.* http://build.shr-project.org/sources/
bzr://.*/.* http://build.shr-project.org/sources/
https?$://.*/.* http://build.shr-project.org/sources/
}

This is almost certainly a result of:

http://git.openembedded.org/bitbake/commit/?id=604df1b25cf114e083f52917df2df64e01279c25

since I didn't believe anyone was using the regexp syntax in the base url type. The bug I wanted to fix was where if you did:

https://.*/.* file:///some/path/

Quoted 7 lines

which seems reasonable enough, it would access:

files:///some/path/

which is clearly not what anyone would want.

I'm not sure what to do about fixing this...

oe-core mirrors.bbclass is using that..
http://git.openembedded.org/openembedded-core/tree/meta/classes/mirrors.bbclass
so maybe someone should fix mirrors.bbclass if this is expected behavior with new bitbake..

Cheers,

Quoted 7 lines

Cheers,

Richard

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#4 lore raw

PATCH 20/20 fetch: allow regexps in mirror protocol

In reply to Richard Purdie

Last mirror rewrite caused a regression not accepting

   .*://.*/.* file://${DL_DIR}/../local/

like specifications anymore. Patch restores old behavior by using regexp matching when checking protocol.

---
 lib/bb/fetch2/__init__.py |    2 +-

1 Datei geändert, 1 Zeile hinzugefügt(+), 1 Zeile entfernt(-)

lib/bb/fetch2/__init__.py +1 -1
index 6f3d88c..75ce01b 100644
@@ -207,7 +207,7 @@ def uri_replace(ud, uri_find, uri_replace, d):
         elif loc == 0:
             # Principle of least surprise. We could end up with https matching against http and 
             # generating "files://" urls if we use the regexp engine below.
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
                 return None
             result_decoded[loc] = uri_replace_decoded[loc]
         elif (re.match(i, uri_decoded[loc])):
Signature · 1 line
1.7.10.2
#5 lore raw

Re: [OE-core] [PATCH 20/20] fetch: allow regexps in mirror protocol

Quoted 26 lines · Enrico Scholz

Last mirror rewrite caused a regression not accepting

   .*://.*/.* file://${DL_DIR}/../local/

like specifications anymore. Patch restores old behavior by using regexp matching when checking protocol.

---
 lib/bb/fetch2/__init__.py |    2 +-

1 Datei geändert, 1 Zeile hinzugefügt(+), 1 Zeile entfernt(-)

lib/bb/fetch2/__init__.py +1 -2
index 6f3d88c..75ce01b 100644
@@ -207,7 +207,7 @@ def uri_replace(ud, uri_find, uri_replace, d):
         elif loc == 0:
             # Principle of least surprise. We could end up with https matching against http and
             # generating "files://" urls if we use the regexp engine below.
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
                 return None
             result_decoded[loc] = uri_replace_decoded[loc]
         elif (re.match(i, uri_decoded[loc])):
--
Quoted 1 line

1.7.10.2

that certainly seems to have solved my issue with using
"own-mirrors".

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#6 lore raw

Re: [PATCH 20/20] fetch: allow regexps in mirror protocol

In reply to Enrico Scholz
Quoted 17 lines · Enrico Scholz

Last mirror rewrite caused a regression not accepting

   .*://.*/.* file://${DL_DIR}/../local/

like specifications anymore. Patch restores old behavior by using regexp matching when checking protocol.

---
 lib/bb/fetch2/__init__.py |    2 +-

1 Datei geändert, 1 Zeile hinzugefügt(+), 1 Zeile entfernt(-)

lib/bb/fetch2/__init__.py +0 -0
index 6f3d88c..75ce01b 100644

lib/bb/fetch2/__init__.py def uri_replace(ud, uri_find, uri_replace, d): · line 213

Quoted 8 lines
lib/bb/fetch2/__init__.py +1 -1
@@ -207,7 +207,7 @@ def uri_replace(ud, uri_find, uri_replace, d):
         elif loc == 0:
             # Principle of least surprise. We could end up with https matching against http and 
             # generating "files://" urls if we use the regexp engine below.
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
                 return None
             result_decoded[loc] = uri_replace_decoded[loc]
         elif (re.match(i, uri_decoded[loc])):

I'm not picking on you (Enrico) here, there is a problem with my change and we probably should revert my change in this area.

However the above patch fails "bitbake-selftest" and introduces a regression. I wrote regression tests for the fetcher for a reason.

The trouble is its sees:

https://.*/.* file:///someplace/

but returns a url like files:// which the fetcher doesn't understand. Suggestions on how to fix this are welcome.

Cheers,

Richard

#7 lore raw

Re: [PATCH 20/20] fetch: allow regexps in mirror protocol

Quoted 8 lines · Richard Purdie

lib/bb/fetch2/__init__.py def uri_replace(ud, uri_find, uri_replace, d): · line 213

Quoted 7 lines

# Principle of least surprise. We could end up with https matching against http and # generating "files://" urls if we use the regexp engine below.

lib/bb/fetch2/__init__.py +1 -1
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
                 return None
             result_decoded[loc] = uri_replace_decoded[loc]
         elif (re.match(i, uri_decoded[loc])):
Quoted 5 lines

I'm not picking on you (Enrico) here, there is a problem with my change and we probably should revert my change in this area.

However the above patch fails "bitbake-selftest" and introduces a regression.

selftest fails here in

| ERROR: test_gitfetch_premirror3 (bb.tests.fetch.FetcherTest)
| CalledProcessError: Command 'git clone git://git.openembedded.org/bitbake /tmp/tmpt_dDVD/sourcemirror.git 2> /dev/null' returned non-zero exit status 128

and

| FAIL: test_urilist2 (bb.tests.fetch.FetcherTest)
| AssertionError: Lists differ: ['file:///somepath/downloads/b... != ['file:///someotherpath/downlo...
| - ['file:///somepath/downloads/bitbake-1.0.tar.gz',
| -  'file:///someotherpath/downloads/bitbake-1.0.tar.gz']
| ? ^
| 
| + ['file:///someotherpath/downloads/bitbake-1.0.tar.gz']
| ? ^

(I am some days behind current bitbake master, so these might be fixed in the meantime)

Nothing which seems to be related to the patch.

Quoted 6 lines

The trouble is its sees:

https://.*/.* file:///someplace/

but returns a url like files:// which the fetcher doesn't understand. Suggestions on how to fix this are welcome.

I do not see how my patch can cause such a rewrite. The troublesome 're.sub()' is done in a branch for loc != 0 only and won't be applied on the protocol part.

Enrico

#8 lore raw

Re: [PATCH 20/20] fetch: allow regexps in mirror protocol

Quoted 12 lines · Enrico Scholz
Quoted 2 lines

lib/bb/fetch2/__init__.py def uri_replace(ud, uri_find, uri_replace, d): · line 210

Quoted 2 lines
lib/bb/fetch2/__init__.py +1 -1
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
| FAIL: test_urilist2 (bb.tests.fetch.FetcherTest)
| AssertionError: Lists differ: ['file:///somepath/downloads/b... != ['file:///someotherpath/downlo...
| - ['file:///somepath/downloads/bitbake-1.0.tar.gz',
| -  'file:///someotherpath/downloads/bitbake-1.0.tar.gz']
| ? ^
| 
| + ['file:///someotherpath/downloads/bitbake-1.0.tar.gz']
| ? ^
Quoted 1 line

Nothing which seems to be related to the patch.

I have to correct me... the new line should be

| + if not re.match(i + '$', uri_decoded[loc]):

to match the complete protocol.

Enrico

#9 lore raw

Re: [PATCH 20/20] fetch: allow regexps in mirror protocol

Quoted 14 lines · Enrico Scholz
Quoted 12 lines · Enrico Scholz
Quoted 2 lines

lib/bb/fetch2/__init__.py def uri_replace(ud, uri_find, uri_replace, d): · line 210

Quoted 2 lines
lib/bb/fetch2/__init__.py +1 -1
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
| FAIL: test_urilist2 (bb.tests.fetch.FetcherTest)
| AssertionError: Lists differ: ['file:///somepath/downloads/b... != ['file:///someotherpath/downlo...
| - ['file:///somepath/downloads/bitbake-1.0.tar.gz',
| -  'file:///someotherpath/downloads/bitbake-1.0.tar.gz']
| ? ^
| 
| + ['file:///someotherpath/downloads/bitbake-1.0.tar.gz']
| ? ^
Quoted 7 lines
Quoted 1 line

Nothing which seems to be related to the patch.

I have to correct me... the new line should be

| + if not re.match(i + '$', uri_decoded[loc]):

to match the complete protocol.

Agreed, this is going to make most sense. I've merged a patch which reverts my original change and adds something like this (but checks the pattern doesn't end with this already).

Hopefully this resolves the issues everyone was having.

Cheers,

Richard

#10 lore raw

Re: [PATCH 20/20] fetch: allow regexps in mirror protocol

In reply to Enrico Scholz
Quoted 13 lines · Enrico Scholz
Quoted 11 lines · Richard Purdie

lib/bb/fetch2/__init__.py def uri_replace(ud, uri_find, uri_replace, d): · line 213

Quoted 7 lines

# Principle of least surprise. We could end up with https matching against http and # generating "files://" urls if we use the regexp engine below.

lib/bb/fetch2/__init__.py +1 -1
-            if i != uri_decoded[loc]:
+            if not re.match(i, uri_decoded[loc]):
                 return None
             result_decoded[loc] = uri_replace_decoded[loc]
         elif (re.match(i, uri_decoded[loc])):

I'm not picking on you (Enrico) here, there is a problem with my change and we probably should revert my change in this area.

Quoted 7 lines
Quoted 2 lines

However the above patch fails "bitbake-selftest" and introduces a regression.

selftest fails here in

| ERROR: test_gitfetch_premirror3 (bb.tests.fetch.FetcherTest)
| CalledProcessError: Command 'git clone git://git.openembedded.org/bitbake /tmp/tmpt_dDVD/sourcemirror.git 2> /dev/null' returned non-zero exit status 128

I've pushed a fix for this issue. That one was a bug in the test code (which needed a chdir adding since cwd from the previous test was removed causing the error).

Cheers,

Richard

#11 lore raw

Re: (PRE)MIRRORS doesn't work with https?$:// anymore Was: [bitbake-devel] [PATCH] fetch2: Add new mirror syntax to simplify mirror specifications

In reply to Richard Purdie
Quoted 1 line

I'm not sure what to do about fixing this...

should be trivial by regexp matching the protocol part instead of comparing it with '='. See my parallel posting for a patch.

Enrico