svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
RFC PATCH 1/1 svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user.
In the future we would like to parse this list and see if the items are already in the SRC_URI for that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.
Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
---
This was done in response to the patch from Krzysztof Zawadzki and the resulting discussion to make this the default instead of optional.
lib/bb/fetch2/svn.py | 12 +++++++++ lib/bb/tests/fetch.py | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+)
lib/bb/fetch2/svn.py +12 -0
@@ -91,6 +91,10 @@ class Svn(FetchMethod): svncmd = "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else: suffix = "" + + if not ("externals" in ud.parm and ud.parm["externals"] == "allowed"): + options.append("--ignore-externals") + if ud.revision: options.append("-r %s" % ud.revision) suffix = "@%s" % (ud.revision)
@@ -136,6 +140,14 @@ class Svn(FetchMethod): bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) + # Warn the user if this had externals (won't catch them all) + output = runfetchcmd("svn propget svn:externals", d, workdir=ud.moddir) + if output: + if "--ignore-externals" in svnfetchcmd.split(): + bb.warn("svn repository has ignored externals:\n%s" % output) + else: + bb.debug(1, "svn repository has externals:\n%s" % output) + scmdata = ud.parm.get("scmdata", "") if scmdata == "keep": tar_flags = ""
lib/bb/tests/fetch.py +67 -0
73 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Quoted 12 lines · Mark Hatle
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user.
In the future we would like to parse this list and see if the items are already in the
SRC_URIfor that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.
Quoted 8 lines
Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
Signed-off-by Mark Hatle <[email protected]>---
This was done in response to the patch from Krzysztof Zawadzki and the resulting discussion to make this the default instead of optional.
Patch looks good to me but the autobuilder didn't like it:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/465/steps/7/logs/step1d
Cheers,
Richard
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Mark Hatle
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user.
In the future we would like to parse this list and see if the items are already in the
SRC_URIfor that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
Quoted 8 lines
Quoted 4 lines
Signed-off-by Mark Hatle <[email protected]>---
This was done in response to the patch from Krzysztof Zawadzki and the resulting discussion to make this the default instead of optional.Patch looks good to me but the autobuilder didn't like it:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/465/steps/7/logs/step1d
What is the host system type on the autobuilder? Everything I see in the traceback is as intended. So it's likely a version or argument change in a different version of SVN.
--Mark
Quoted 4 lines
Cheers,
Richard
Quoted 23 lines · Mark Hatle
Quoted 22 lines · Richard Purdie
Quoted 20 lines · Mark Hatle
Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.
By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user.
In the future we would like to parse this list and see if the items are already in the
SRC_URIfor that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.
Signed-off-by Mark Hatle <[email protected]>---
This was done in response to the patch from Krzysztof Zawadzki and the resulting discussion to make this the default instead of optional.
Quoted 7 lines
Quoted 3 lines
Patch looks good to me but the autobuilder didn't like it:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/465/steps/7/logs/step1d
What is the host system type on the autobuilder? Everything I see in the traceback is as intended. So it's likely a version or argument change in a different version of SVN.
That one was debian9.
rpurdie@debian9-ty-2:~$ svn --version svn, version 1.9.5 (r1770682) compiled Jul 21 2018, 02:35:40 on x86_64-pc-linux-gnu
Worryingly I looked on debian9-ty-1 first and subversion isn't installed there at all. In builds we'd build subversion-native.
This could get quite tricky to handle...
Cheers,
Richard