sanity: improve mirrors check

PATCH Superseded Christopher Larson 4 participants 12 messages Aug 25, 2014 – Aug 27, 2014

lore

Share & tools

lore

Christopher Larson

author

2014-08-25 22:47 UTC #0 lore raw

cover sanity: improve mirrors check

From: Christopher Larson <[email protected]>

The following changes since commit 217aa4d9802609d2c3628c8751a27a5d25900898:

autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:25:54 +0100)

are available in the git repository at:

[email protected]:kergoth/openembedded-core improve-sanity-mirrors-check

for you to fetch changes up to 47b5636f4c87979aecfbcb743235a9e010bedd23:

sanity: refactor mirrors checks to be more pythonic (2014-08-25 15:45:19 -0700)

----------------------------------------------------------------
Christopher Larson (3):
sanity: handle both \n and \n in mirror vars
sanity: fix support for regex schemes in mirrors check
sanity: refactor mirrors checks to be more pythonic

 meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 31 deletions(-)

Signature · 1 line

1.8.3.4

Christopher Larson

author

2014-08-25 22:47 UTC #1 lore raw

PATCH 1/3 sanity: handle both \n and \n in mirror vars

From: Christopher Larson <[email protected]>

Signed-off-by Christopher Larson <[email protected]>

---
 meta/classes/sanity.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/sanity.bbclass +1 -1

index dbcc26b..7cfc4be 100644
@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
         for mir in mirros:
             mir_list = mir.split()
             # Should be two members.

Signature · 1 line

1.8.3.4

Christopher Larson

2014-08-25 22:50 UTC #2 lore raw

Re: [PATCH 1/3] sanity: handle both \n and \n in mirror vars

Quoted 4 lines · Christopher Larson

From: Christopher Larson <[email protected]>
Signed-off-by Christopher Larson <[email protected]>

Bah, typo in the subject, fixing. :)

-Chris

Christopher Larson

author

2014-08-25 22:47 UTC #3 lore raw

PATCH 2/3 sanity: fix support for regex schemes in mirrors check

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

Signed-off-by Christopher Larson <[email protected]>

---
 meta/classes/sanity.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

meta/classes/sanity.bbclass +10 -2

index 7cfc4be..f655d8b 100644
@@ -754,8 +754,9 @@ def check_sanity_everybuild(status, d):
         status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.")
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
+    import re
     mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \
+    protocols = ['http://', 'ftp://', 'file://', 'https://', \
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:

@@ -767,12 +768,19 @@ def check_sanity_everybuild(status, d):
                 bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
                     % (mir_type, mir.strip(), len(mir_list)))
             elif len(mir_list) == 2:
+                decoded = bb.fetch2.decodeurl(mir_list[0])
+                try:
+                    pattern_scheme = re.compile(decoded[0])
+                except re.error as exc:
+                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
+                    continue
+
                 # Each member should start with protocols
                 valid_protocol_0 = False
                 valid_protocol_1 = False
                 file_absolute = True
                 for protocol in protocols:
-                    if not valid_protocol_0 and mir_list[0].startswith(protocol):
+                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
                         valid_protocol_0 = True
                     if not valid_protocol_1 and mir_list[1].startswith(protocol):
                         valid_protocol_1 = True

Signature · 1 line

1.8.3.4

Christopher Larson

author

2014-08-25 22:47 UTC #4 lore raw

PATCH 3/3 sanity: refactor mirrors checks to be more pythonic

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

- Use clearer variable names
- Use variable unpacking to reference elements by name rather than index
- Sacrifice a small amount of time (iterate over protocols twice per entry rather than once) for clarity: use readable generator expressions with any() rather than maintaining state.

Signed-off-by Christopher Larson <[email protected]>

---
 meta/classes/sanity.bbclass | 73 ++++++++++++++++++++++-----------------------
 1 file changed, 35 insertions(+), 38 deletions(-)

meta/classes/sanity.bbclass +35 ~~-38~~

index f655d8b..5be5efb 100644
@@ -755,44 +755,41 @@ def check_sanity_everybuild(status, d):
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
     import re
-    mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', \
-        'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
-        'bzr://', 'cvs://']
-    for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
-        for mir in mirros:
-            mir_list = mir.split()
-            # Should be two members.
-            if len(mir_list) not in [0, 2]:
-                bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
-                    % (mir_type, mir.strip(), len(mir_list)))
-            elif len(mir_list) == 2:
-                decoded = bb.fetch2.decodeurl(mir_list[0])
-                try:
-                    pattern_scheme = re.compile(decoded[0])
-                except re.error as exc:
-                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
-                    continue
-
-                # Each member should start with protocols
-                valid_protocol_0 = False
-                valid_protocol_1 = False
-                file_absolute = True
-                for protocol in protocols:
-                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
-                        valid_protocol_0 = True
-                    if not valid_protocol_1 and mir_list[1].startswith(protocol):
-                        valid_protocol_1 = True
-                        # The file:// must be an absolute path.
-                        if protocol == 'file://' and not mir_list[1].startswith('file:///'):
-                            file_absolute = False
-                    if valid_protocol_0 and valid_protocol_1:
-                        break
-                if not (valid_protocol_0 and valid_protocol_1):
-                    bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip()))
-                if not file_absolute:
-                    bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip()))
+    mirror_vars = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
+    protocols = ['http', 'ftp', 'file', 'https', \
+                 'git', 'gitsm', 'hg', 'osc', 'p4', 'svk', 'svn', \
+                 'bzr', 'cvs']
+    for mirror_var in mirror_vars:
+        mirrors = (d.getVar(mirror_var, True) or '').replace('\\n', '\n').split('\n')
+        for mirror_entry in mirrors:
+            mirror_entry = mirror_entry.strip()
+            if not mirror_entry:
+                # ignore blank lines
+                continue
+
+            try:
+                pattern, mirror = mirror_entry.split()
+            except ValueError:
+                bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip()))
+                continue
+
+            decoded = bb.fetch2.decodeurl(pattern)
+            try:
+                pattern_scheme = re.compile(decoded[0])
+            except re.error as exc:
+                bb.warn('Invalid scheme regex (%s) in %s; %s' % (pattern, mirror_var, mirror_entry))
+                continue
+
+            if not any(pattern_scheme.match(protocol) for protocol in protocols):
+                bb.warn('Invalid protocol (%s) in %s: %s' % (decoded[0], mirror_var, mirror_entry))
+                continue
+
+            if not any(mirror.startswith(protocol + '://') for protocol in protocols):
+                bb.warn('Invalid protocol in %s: %s' % (mirror_var, mirror_entry))
+                continue
+
+            if mirror.startswith('file://') and not mirror.startswith('file:///'):
+                bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mirror_var, mirror_entry))
 
     # Check that TMPDIR hasn't changed location since the last time we were run
     tmpdir = d.getVar('TMPDIR', True)

Signature · 1 line

1.8.3.4

Christopher Larson

author

2014-08-25 22:57 UTC #5 lore raw

v2 cover sanity: improve mirrors check

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

v2: fixed subject line of first commit to mention \\n

The following changes since commit 217aa4d9802609d2c3628c8751a27a5d25900898:

autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:25:54 +0100)

are available in the git repository at:

[email protected]:kergoth/openembedded-core improve-sanity-mirrors-check

for you to fetch changes up to 8fa6adaaa363a88c888f2afab2f0117b199a54c0:

sanity: refactor mirrors checks to be more pythonic (2014-08-25 15:50:53 -0700)

----------------------------------------------------------------
Christopher Larson (3):
sanity: handle both \n and \\n in mirror vars
sanity: fix support for regex schemes in mirrors check
sanity: refactor mirrors checks to be more pythonic

 meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 31 deletions(-)

Signature · 1 line

1.8.3.4

Christopher Larson

author

2014-08-25 22:57 UTC #6 lore raw

PATCH v2 1/3 sanity: handle both \n and \\n in mirror vars

From: Christopher Larson <[email protected]>

Signed-off-by Christopher Larson <[email protected]>

---
 meta/classes/sanity.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/sanity.bbclass +1 -1

index dbcc26b..7cfc4be 100644
@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
         for mir in mirros:
             mir_list = mir.split()
             # Should be two members.

Signature · 1 line

1.8.3.4

Martin Jansa

2014-08-26 10:51 UTC #7 lore raw

Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

Quoted 12 lines · Christopher Larson

From: Christopher Larson <[email protected]>
Signed-off-by Christopher Larson <[email protected]>
---
 meta/classes/sanity.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/sanity.bbclass +0 -0
index dbcc26b..7cfc4be 100644

meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 762

Quoted 5 lines

meta/classes/sanity.bbclass +1 -1

@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')

Would you mind fixing the typo in variable name as well?

Quoted 11 lines

meta/classes/sanity.bbclass +0 -0

         for mir in mirros:
             mir_list = mir.split()
             # Should be two members.

Signature · 7 lines

1.8.3.4

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]

Martin Jansa

2014-08-26 10:59 UTC #8 lore raw

Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

Quoted 8 lines · Martin Jansa

Quoted 7 lines · Christopher Larson
From: Christopher Larson <[email protected]>
Signed-off-by Christopher Larson <[email protected]>
---
 meta/classes/sanity.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Quoted 12 lines

meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 762

Quoted 10 lines

meta/classes/sanity.bbclass +1 -1

index dbcc26b..7cfc4be 100644
@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')

Would you mind fixing the typo in variable name as well?

Nevermind, it's refactored in 3/3, sorry for noise.

Quoted 14 lines

Quoted 11 lines

meta/classes/sanity.bbclass +0 -0

         for mir in mirros:
             mir_list = mir.split()
             # Should be two members.

Signature · 7 lines

1.8.3.4

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]

Koen Kooi

Tested-by on v2 1/3

2014-08-27 11:50 UTC #9 lore raw

Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

In reply to Christopher Larson

Op 26 aug. 2014, om 00:57 heeft Christopher Larson <[email protected]> het volgende geschreven:

Quoted 3 lines

From: Christopher Larson <[email protected]>
Signed-off-by Christopher Larson <[email protected]>

Tested-by Koen Kooi <[email protected]>

Quoted 26 lines

---
meta/classes/sanity.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/sanity.bbclass +1 -1

index dbcc26b..7cfc4be 100644
@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
        'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
        'bzr://', 'cvs://']
    for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
        for mir in mirros:
            mir_list = mir.split()
            # Should be two members.

Signature · 7 lines

1.8.3.4

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Christopher Larson

author

2014-08-25 22:57 UTC #10 lore raw

PATCH v2 2/3 sanity: fix support for regex schemes in mirrors check

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

Signed-off-by Christopher Larson <[email protected]>

---
 meta/classes/sanity.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

meta/classes/sanity.bbclass +10 -2

index 7cfc4be..f655d8b 100644
@@ -754,8 +754,9 @@ def check_sanity_everybuild(status, d):
         status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.")
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
+    import re
     mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \
+    protocols = ['http://', 'ftp://', 'file://', 'https://', \
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:

@@ -767,12 +768,19 @@ def check_sanity_everybuild(status, d):
                 bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
                     % (mir_type, mir.strip(), len(mir_list)))
             elif len(mir_list) == 2:
+                decoded = bb.fetch2.decodeurl(mir_list[0])
+                try:
+                    pattern_scheme = re.compile(decoded[0])
+                except re.error as exc:
+                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
+                    continue
+
                 # Each member should start with protocols
                 valid_protocol_0 = False
                 valid_protocol_1 = False
                 file_absolute = True
                 for protocol in protocols:
-                    if not valid_protocol_0 and mir_list[0].startswith(protocol):
+                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
                         valid_protocol_0 = True
                     if not valid_protocol_1 and mir_list[1].startswith(protocol):
                         valid_protocol_1 = True

Signature · 1 line

1.8.3.4

Christopher Larson

author

2014-08-25 22:57 UTC #11 lore raw

PATCH v2 3/3 sanity: refactor mirrors checks to be more pythonic

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

Signed-off-by Christopher Larson <[email protected]>

---
 meta/classes/sanity.bbclass | 73 ++++++++++++++++++++++-----------------------
 1 file changed, 35 insertions(+), 38 deletions(-)

meta/classes/sanity.bbclass +35 ~~-38~~

index f655d8b..5be5efb 100644
@@ -755,44 +755,41 @@ def check_sanity_everybuild(status, d):
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
     import re
-    mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', \
-        'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
-        'bzr://', 'cvs://']
-    for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
-        for mir in mirros:
-            mir_list = mir.split()
-            # Should be two members.
-            if len(mir_list) not in [0, 2]:
-                bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
-                    % (mir_type, mir.strip(), len(mir_list)))
-            elif len(mir_list) == 2:
-                decoded = bb.fetch2.decodeurl(mir_list[0])
-                try:
-                    pattern_scheme = re.compile(decoded[0])
-                except re.error as exc:
-                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
-                    continue
-
-                # Each member should start with protocols
-                valid_protocol_0 = False
-                valid_protocol_1 = False
-                file_absolute = True
-                for protocol in protocols:
-                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
-                        valid_protocol_0 = True
-                    if not valid_protocol_1 and mir_list[1].startswith(protocol):
-                        valid_protocol_1 = True
-                        # The file:// must be an absolute path.
-                        if protocol == 'file://' and not mir_list[1].startswith('file:///'):
-                            file_absolute = False
-                    if valid_protocol_0 and valid_protocol_1:
-                        break
-                if not (valid_protocol_0 and valid_protocol_1):
-                    bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip()))
-                if not file_absolute:
-                    bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip()))
+    mirror_vars = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
+    protocols = ['http', 'ftp', 'file', 'https', \
+                 'git', 'gitsm', 'hg', 'osc', 'p4', 'svk', 'svn', \
+                 'bzr', 'cvs']
+    for mirror_var in mirror_vars:
+        mirrors = (d.getVar(mirror_var, True) or '').replace('\\n', '\n').split('\n')
+        for mirror_entry in mirrors:
+            mirror_entry = mirror_entry.strip()
+            if not mirror_entry:
+                # ignore blank lines
+                continue
+
+            try:
+                pattern, mirror = mirror_entry.split()
+            except ValueError:
+                bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip()))
+                continue
+
+            decoded = bb.fetch2.decodeurl(pattern)
+            try:
+                pattern_scheme = re.compile(decoded[0])
+            except re.error as exc:
+                bb.warn('Invalid scheme regex (%s) in %s; %s' % (pattern, mirror_var, mirror_entry))
+                continue
+
+            if not any(pattern_scheme.match(protocol) for protocol in protocols):
+                bb.warn('Invalid protocol (%s) in %s: %s' % (decoded[0], mirror_var, mirror_entry))
+                continue
+
+            if not any(mirror.startswith(protocol + '://') for protocol in protocols):
+                bb.warn('Invalid protocol in %s: %s' % (mirror_var, mirror_entry))
+                continue
+
+            if mirror.startswith('file://') and not mirror.startswith('file:///'):
+                bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mirror_var, mirror_entry))
 
     # Check that TMPDIR hasn't changed location since the last time we were run
     tmpdir = d.getVar('TMPDIR', True)

Signature · 1 line

1.8.3.4