Robert Yang : sanity.bbclass: check the format of MIRRORS
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · On Sat, Aug 23, 2014 at 10:02:23PM +0000, [email protected] wrote:
Module: openembedded-core.git Branch: master Commit: c8c213bb25b137cf70ba8ce9a45e60065d926735 URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=c8c213bb25b137cf70ba8ce9a45e60065d926735
Quoted 7 lines
Author: Robert Yang <[email protected]> Date: Fri Aug 22 01:31:27 2014 -0700sanity.bbclass: check the format of MIRRORS
Check the format of MIRRORS, PREMIRRORS and
SSTATE_MIRRORS:
* Each mirror shoudl contain two memebers.
shoudl -> should
Quoted 2 lines
* The local "file://" url must use absolute path (file:///).
* The protocol must in protocols list.
must "be"? seems like something is missing
Quoted 13 lines
Signed-off-by Robert Yang <[email protected]>Signed-off-by Richard Purdie <[email protected]>---
meta/classes/sanity.bbclass | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)meta/classes/sanity.bbclass +0
-0
meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 762
Quoted 10 lines
meta/classes/sanity.bbclass +7
-0@@ -753,6 +753,39 @@ def check_sanity_everybuild(status, d): if oeroot.find(' ') != -1: status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.") + # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS + mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] + protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \ + 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ + 'bzr://', 'cvs://'] + for mir_type in mir_types: + mirros = (d.getVar(mir_type, True) or '').split('\\n')
Is the "\n" (followed by actuall line break) really required as separator?
openembedded-core/meta/classes/own-mirrors.bbclass doesn't separate the entries with "\n".
and bitbake code seems to handle them both:
def mirror_from_string(data):
return [ i.split() for i in (data or "").replace('\\n','\n').split('\n') if i ]If you want to unify the format to enforce "\\n" then I think it would be better to show the warning in bitbake fetcher, not in stanity.bbclass
Also the list of supported protocols would be IMHO better in bitbake fetcher module.
Quoted 35 lines
meta/classes/sanity.bbclass +26
-0+ for mir in mirros: + mir_list = mir.split() + # Should be two members. + if len(mir_list) not in [0, 2]: + bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ + % (mir_type, mir.strip(), len(mir_list))) + elif len(mir_list) == 2: + # Each member should start with protocols + valid_protocol_0 = False + valid_protocol_1 = False + file_absolute = True + for protocol in protocols: + if not valid_protocol_0 and mir_list[0].startswith(protocol): + valid_protocol_0 = True + if not valid_protocol_1 and mir_list[1].startswith(protocol): + valid_protocol_1 = True + # The file:// must be an absolute path. + if protocol == 'file://' and not mir_list[1].startswith('file:///'): + file_absolute = False + if valid_protocol_0 and valid_protocol_1: + break + if not (valid_protocol_0 and valid_protocol_1): + bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip())) + if not file_absolute: + bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip())) + # Check that TMPDIR hasn't changed location since the last time we were run tmpdir = d.getVar('TMPDIR', True) checkfile = os.path.join(tmpdir, "saved_tmpdir")Signature · 5 lines
-- _______________________________________________ Openembedded-commits mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-commits
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 14 lines · Martin Jansa
Quoted 5 lines · On Sat, Aug 23, 2014 at 10:02:23PM +0000, [email protected] wrote:
Module: openembedded-core.git Branch: master Commit: c8c213bb25b137cf70ba8ce9a45e60065d926735 URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=c8c213bb25b137cf70ba8ce9a45e60065d926735Quoted 7 lines
Author: Robert Yang <[email protected]> Date: Fri Aug 22 01:31:27 2014 -0700sanity.bbclass: check the format of MIRRORS
Check the format of MIRRORS, PREMIRRORS and
SSTATE_MIRRORS:
* Each mirror shoudl contain two memebers.
Quoted 7 lines
shoudl -> should
Quoted 2 lines
* The local "file://" url must use absolute path (file:///).
* The protocol must in protocols list.must "be"? seems like something is missing
Sorry for the typos.
Quoted 26 lines
Quoted 13 lines
Signed-off-by Robert Yang <[email protected]>Signed-off-by Richard Purdie <[email protected]>---
meta/classes/sanity.bbclass | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)meta/classes/sanity.bbclass +0
-0meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 762
Quoted 10 lines
meta/classes/sanity.bbclass +7
-0@@ -753,6 +753,39 @@ def check_sanity_everybuild(status, d): if oeroot.find(' ') != -1: status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.") + # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS + mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] + protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \ + 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ + 'bzr://', 'cvs://'] + for mir_type in mir_types: + mirros = (d.getVar(mir_type, True) or '').split('\\n')Is the "\n" (followed by actuall line break) really required as separator?
Quoted 6 lines
openembedded-core/meta/classes/own-mirrors.bbclass doesn't separate the entries with "\n".
and bitbake code seems to handle them both:
defmirror_from_string(data):return [ i.split() for i in (data or "").replace('\\n','\n').split('\n') if i ]
I think that "\n" is required, otherwise it can't split it as the code shows, and I had tried that it didn't work without the "\n" at the end of the line (except the last line, but we really need add the "\n" to the last line even it works, in case of _append).
I will update the own-mirrors.bbclass after more investigations.
Quoted 5 lines
If you want to unify the format to enforce "\\n" then I think it would be better to show the warning in bitbake fetcher, not in stanity.bbclass
Also the list of supported protocols would be IMHO better in bitbake fetcher module.
Do it in the fetcher sounds fine to me, but we have to do more work, for example, make sure that only check the various MIRRORS once rather than checked in every do_fetch.
// Robert
Quoted 37 lines
Quoted 36 lines
meta/classes/sanity.bbclass +26
-1+ for mir in mirros: + mir_list = mir.split() + # Should be two members. + if len(mir_list) not in [0, 2]: + bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ + % (mir_type, mir.strip(), len(mir_list))) + elif len(mir_list) == 2: + # Each member should start with protocols + valid_protocol_0 = False + valid_protocol_1 = False + file_absolute = True + for protocol in protocols: + if not valid_protocol_0 and mir_list[0].startswith(protocol): + valid_protocol_0 = True + if not valid_protocol_1 and mir_list[1].startswith(protocol): + valid_protocol_1 = True + # The file:// must be an absolute path. + if protocol == 'file://' and not mir_list[1].startswith('file:///'): + file_absolute = False + if valid_protocol_0 and valid_protocol_1: + break + if not (valid_protocol_0 and valid_protocol_1): + bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip())) + if not file_absolute: + bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip())) + # Check that TMPDIR hasn't changed location since the last time we were run tmpdir = d.getVar('TMPDIR', True) checkfile = os.path.join(tmpdir, "saved_tmpdir") --_______________________________________________
Openembedded-commits mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-commits
Quoted 54 lines · Robert Yang
Quoted 14 lines · Martin Jansa
Quoted 5 lines · On Sat, Aug 23, 2014 at 10:02:23PM +0000, [email protected] wrote:
Module: openembedded-core.git Branch: master Commit: c8c213bb25b137cf70ba8ce9a45e60065d926735 URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=c8c213bb25b137cf70ba8ce9a45e60065d926735Quoted 7 lines
Author: Robert Yang <[email protected]> Date: Fri Aug 22 01:31:27 2014 -0700sanity.bbclass: check the format of MIRRORS
Check the format of MIRRORS, PREMIRRORS and
SSTATE_MIRRORS:
* Each mirror shoudl contain two memebers.Quoted 7 lines
shoudl -> should
Quoted 2 lines
* The local "file://" url must use absolute path (file:///).
* The protocol must in protocols list.must "be"? seems like something is missing
Sorry for the typos.
Quoted 26 lines
Quoted 13 lines
Signed-off-by Robert Yang <[email protected]>Signed-off-by Richard Purdie <[email protected]>---
meta/classes/sanity.bbclass | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)meta/classes/sanity.bbclass +0
-0meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 762
Quoted 10 lines
meta/classes/sanity.bbclass +7
-0@@ -753,6 +753,39 @@ def check_sanity_everybuild(status, d): if oeroot.find(' ') != -1: status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.") + # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS + mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] + protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \ + 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ + 'bzr://', 'cvs://'] + for mir_type in mir_types: + mirros = (d.getVar(mir_type, True) or '').split('\\n')Is the "\n" (followed by actuall line break) really required as separator?
Quoted 8 lines
Quoted 6 lines
openembedded-core/meta/classes/own-mirrors.bbclass doesn't separate the entries with "\n".
and bitbake code seems to handle them both:
defmirror_from_string(data):return [ i.split() for i in (data or "").replace('\\n','\n').split('\n') if i ]I think that "\n" is required, otherwise it can't split it as the code shows,
I think the code shows that it splits it by normal line feed, "\\n" are replaced with "\n" so they probably just add extra line feed which is ignored because of empty line it produces.
On the other hand "\\n" is probably still useful when appending to it as multiline variable (with \ at the end of line) - then the line feeds aren't preserved when parsing and \\n is needed.
Quoted 7 lines
and I had tried that it didn't work without the "\n" at the end of the line (except the last line, but we really need add the "\n" to the last line even it works, in case of
_append).I will update the own-mirrors.bbclass after more investigations.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
If you want to unify the format to enforce "\\n" then I think it would be better to show the warning in bitbake fetcher, not in stanity.bbclass
Also the list of supported protocols would be IMHO better in bitbake fetcher module.
Do it in the fetcher sounds fine to me, but we have to do more work, for example, make sure that only check the various MIRRORS once rather than checked in every
do_fetch.
On the other hand it will show the warning in all projects which can possibly use bitbake and it will make easier to ensure that checked conditions are consistent with the code.
Quoted 40 lines
// Robert
Quoted 37 lines
Quoted 36 lines
meta/classes/sanity.bbclass +26
-1+ for mir in mirros: + mir_list = mir.split() + # Should be two members. + if len(mir_list) not in [0, 2]: + bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ + % (mir_type, mir.strip(), len(mir_list))) + elif len(mir_list) == 2: + # Each member should start with protocols + valid_protocol_0 = False + valid_protocol_1 = False + file_absolute = True + for protocol in protocols: + if not valid_protocol_0 and mir_list[0].startswith(protocol): + valid_protocol_0 = True + if not valid_protocol_1 and mir_list[1].startswith(protocol): + valid_protocol_1 = True + # The file:// must be an absolute path. + if protocol == 'file://' and not mir_list[1].startswith('file:///'): + file_absolute = False + if valid_protocol_0 and valid_protocol_1: + break + if not (valid_protocol_0 and valid_protocol_1): + bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip())) + if not file_absolute: + bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip())) + # Check that TMPDIR hasn't changed location since the last time we were run tmpdir = d.getVar('TMPDIR', True) checkfile = os.path.join(tmpdir, "saved_tmpdir") --_______________________________________________
Openembedded-commits mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-commits
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 64 lines · Martin Jansa
Quoted 54 lines · Robert Yang
Quoted 14 lines · Martin Jansa
Quoted 5 lines · On Sat, Aug 23, 2014 at 10:02:23PM +0000, [email protected] wrote:
Module: openembedded-core.git Branch: master Commit: c8c213bb25b137cf70ba8ce9a45e60065d926735 URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=c8c213bb25b137cf70ba8ce9a45e60065d926735Quoted 7 lines
Author: Robert Yang <[email protected]> Date: Fri Aug 22 01:31:27 2014 -0700sanity.bbclass: check the format of MIRRORS
Check the format of MIRRORS, PREMIRRORS and
SSTATE_MIRRORS:
* Each mirror shoudl contain two memebers.Quoted 7 lines
shoudl -> should
Quoted 2 lines
* The local "file://" url must use absolute path (file:///).
* The protocol must in protocols list.must "be"? seems like something is missing
Sorry for the typos.
Quoted 26 lines
Quoted 13 lines
Signed-off-by Robert Yang <[email protected]>Signed-off-by Richard Purdie <[email protected]>---
meta/classes/sanity.bbclass | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)meta/classes/sanity.bbclass +0
-0meta/classes/sanity.bbclass def check_sanity_everybuild(status, d): · line 762
Quoted 10 lines
meta/classes/sanity.bbclass +7
-0@@ -753,6 +753,39 @@ def check_sanity_everybuild(status, d): if oeroot.find(' ') != -1: status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.") + # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS + mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] + protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \ + 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ + 'bzr://', 'cvs://'] + for mir_type in mir_types: + mirros = (d.getVar(mir_type, True) or '').split('\\n')Is the "\n" (followed by actuall line break) really required as separator?
Quoted 8 lines
Quoted 6 lines
openembedded-core/meta/classes/own-mirrors.bbclass doesn't separate the entries with "\n".
and bitbake code seems to handle them both:
defmirror_from_string(data):return [ i.split() for i in (data or "").replace('\\n','\n').split('\n') if i ]I think that "\n" is required, otherwise it can't split it as the code shows,
Quoted 7 lines
I think the code shows that it splits it by normal line feed, "\\n" are replaced with "\n" so they probably just add extra line feed which is ignored because of empty line it produces.
On the other hand "\\n" is probably still useful when appending to it as multiline variable (with \ at the end of line) - then the line feeds aren't preserved when parsing and \\n is needed.
Yes, without the "\n", it's not easy to set:
PREMIRRORS += "git://.*/.* file://path/to/downloads/ \n \
svn://.*/.* file://path/to/downloads/ \n \
"
They would be treated as one mirror, but they are two.
Quoted 13 lines
Quoted 13 lines
and I had tried that it didn't work without the "\n" at the end of the line (except the last line, but we really need add the "\n" to the last line even it works, in case of
_append).I will update the own-mirrors.bbclass after more investigations.
Quoted 5 lines
If you want to unify the format to enforce "\\n" then I think it would be better to show the warning in bitbake fetcher, not in stanity.bbclass
Also the list of supported protocols would be IMHO better in bitbake fetcher module.
Quoted 7 lines
Quoted 3 lines
Do it in the fetcher sounds fine to me, but we have to do more work, for example, make sure that only check the various MIRRORS once rather than checked in every
do_fetch.On the other hand it will show the warning in all projects which can possibly use bitbake and it will make easier to ensure that checked conditions are consistent with the code.
I think that you mean the project which doesn't have sanity.bbclass ? Then I'm glad to add it to bitbake if RP is OK with that.
// Robert
Quoted 42 lines
Quoted 41 lines
// Robert
Quoted 37 lines
Quoted 36 lines
meta/classes/sanity.bbclass +26
-1+ for mir in mirros: + mir_list = mir.split() + # Should be two members. + if len(mir_list) not in [0, 2]: + bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ + % (mir_type, mir.strip(), len(mir_list))) + elif len(mir_list) == 2: + # Each member should start with protocols + valid_protocol_0 = False + valid_protocol_1 = False + file_absolute = True + for protocol in protocols: + if not valid_protocol_0 and mir_list[0].startswith(protocol): + valid_protocol_0 = True + if not valid_protocol_1 and mir_list[1].startswith(protocol): + valid_protocol_1 = True + # The file:// must be an absolute path. + if protocol == 'file://' and not mir_list[1].startswith('file:///'): + file_absolute = False + if valid_protocol_0 and valid_protocol_1: + break + if not (valid_protocol_0 and valid_protocol_1): + bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip())) + if not file_absolute: + bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip())) + # Check that TMPDIR hasn't changed location since the last time we were run tmpdir = d.getVar('TMPDIR', True) checkfile = os.path.join(tmpdir, "saved_tmpdir") --_______________________________________________
Openembedded-commits mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-commits