svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
PATCH v2 1/1 svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user. It is up to the user to determine if these are necessary for the recipe. They may disable the warning by adding "externals=nowarn" to the url.
In the future we would like to parse this list and see if the items are already in the SRC_URI for that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.
Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
---
v2:
Refactored the test case to use --non-interactive option to try to avoid the controlling terminal failure.
Update the warning message to give more information, as well as a way to disable the message if the recipe has already verified it is ok.
v1:
This was done in response to the patch from Krzysztof Zawadzki and the resulting discussion to make this the default instead of optional.
lib/bb/fetch2/svn.py | 19 +++++++++++++++ lib/bb/tests/fetch.py | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)
lib/bb/fetch2/svn.py +19 -0
@@ -91,6 +91,13 @@ class Svn(FetchMethod): svncmd = "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else: suffix = "" + + # externals may be either 'allowed' or 'nowarn', but not both. Allowed + # will not issue a warning, but will log to the debug buffer what has likely + # been downloaded by SVN. + if not ("externals" in ud.parm and ud.parm["externals"] == "allowed"): + options.append("--ignore-externals") + if ud.revision: options.append("-r %s" % ud.revision) suffix = "@%s" % (ud.revision)
@@ -136,6 +143,18 @@ class Svn(FetchMethod): bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) + if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"): + # Warn the user if this had externals (won't catch them all) + output = runfetchcmd("svn propget svn:externals", d, workdir=ud.moddir) + if output: + if "--ignore-externals" in svnfetchcmd.split(): + bb.warn("%s contains svn:externals." % ud.url) + bb.warn("These should be added to the recipe SRC_URI as necessary.") + bb.warn("svn fetch has ignored externals:\n%s" % output) + bb.warn("To disable this warning add ';externals=nowarn' to the url.") + else: + bb.debug(1, "svn repository has externals:\n%s" % output) + scmdata = ud.parm.get("scmdata", "") if scmdata == "keep": tar_flags = ""
lib/bb/tests/fetch.py +67 -0
73 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Quoted 18 lines · Mark Hatle
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user. It is up to the user to determine if these are necessary for the recipe. They may disable the warning by adding "externals=nowarn" to the url.
In the future we would like to parse this list and see if the items are already in the
SRC_URIfor that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
Quoted 5 lines
Signed-off-by Mark Hatle <[email protected]>---
v2:
Refactored the test case to use --non-interactive option to try to avoid the controlling terminal failure.
I think this is working better but it leaves us with a second issue, not all our autobuilder workers have subversion installed:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/177
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/176
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/175
(a pass:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/180)
I did specifically test on debian 9 where we saw the problem last time and that issue is resolved and passes now.
The question now is whether we want to install subversion on all our workers, whether we download and install some kind of standalone test tarball for this test or whether we disable the test by default...
Cheers,
Richard
Quoted 35 lines · Richard Purdie
Quoted 18 lines · Mark Hatle
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user. It is up to the user to determine if these are necessary for the recipe. They may disable the warning by adding "externals=nowarn" to the url.
In the future we would like to parse this list and see if the items are already in the
SRC_URIfor that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
Quoted 5 lines
Signed-off-by Mark Hatle <[email protected]>---
v2:
Refactored the test case to use --non-interactive option to try to avoid the controlling terminal failure.I think this is working better but it leaves us with a second issue, not all our autobuilder workers have subversion installed:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/177
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/176
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/175(a pass:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/180)
Quoted 6 lines
I did specifically test on debian 9 where we saw the problem last time and that issue is resolved and passes now.
The question now is whether we want to install subversion on all our workers, whether we download and install some kind of standalone test tarball for this test or whether we disable the test by default...
Is there a concept of a skipped test within bitbake-selftest? It would be easy enough for me to check if 'svn' and 'svnadmin' were available in the environment and simply skip them.
Then it's up to you guys to figure out if this is a test you want to run on all of the builders by installing svn or a standalone test tarball as you suggested.
--Mark
Quoted 6 lines
Cheers,
Richard
Quoted 17 lines · Mark Hatle
Quoted 9 lines · Richard Purdie
I did specifically test on debian 9 where we saw the problem last time
and that issue is resolved and passes now.The question now is whether we want to install subversion on all our
workers, whether we download and install some kind of standalone test
tarball for this test or whether we disable the test by default...Is there a concept of a skipped test within bitbake-selftest? It would be easy
enough for me to check if 'svn' and 'svnadmin' were available in the environment
and simply skip them.
Quoted 4 lines
Then it's up to you guys to figure out if this is a test you want to run on all
of the builders by installing svn or a standalone test tarball as you suggested.
Its standard python unitttest which supports skipping so yes, I'd agree this is the best way forward.
Cheers,
Richard