Skip to contentSkip to conversation

svn.py: Stop SVN from directly pulling from an external layer w/o fetcher

PATCH New Mark Hatle 1 participant 1 message May 17, 2019

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

PATCH v3 1/1 svn.py: Stop SVN from directly pulling from an external layer w/o fetcher

Add a new option to the svn fetcher url "externals=allowed". This will allow a user to enable svn co w/ externals. However, this does avoid the fetcher, network access and mirror systems.

By default we no longer allow externals in the checkout. This ensures a deterministic download. The system does attempt to identify SVN repos that have externals enabled, and will warn the user. It is up to the user to determine if these are necessary for the recipe. They may disable the warning by adding "externals=nowarn" to the url.

In the future we would like to parse this list and see if the items are already in the SRC_URI for that recipe, but with SVN being in limited use these days that extra work is likely not worth the trouble.

Add test cases that generated a local SVN tree, with an external source set to github bitbake in svn format. One test case checks that externals are ignored, and one checks that they in downloaded.

---
v3:
Add a new skipIfNoSvn() function to the SVNTest class. This will skip the tests if SVN is not available for testing.

v2:
Refactored the test case to use --non-interactive option to try to avoid the controlling terminal failure.

Update the warning message to give more information, as well as a way to disable the message if the recipe has already verified it is ok.

v1:
This was done in response to the patch from Krzysztof Zawadzki and the resulti$ discussion to make this the default instead of optional.

 lib/bb/fetch2/svn.py  | 19 ++++++++++++
 lib/bb/tests/fetch.py | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+)
lib/bb/fetch2/svn.py +19 -0
index baeb0e7..59ce931 100644
@@ -91,6 +91,13 @@ class Svn(FetchMethod):
             svncmd = "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module)
         else:
             suffix = ""
+
+            # externals may be either 'allowed' or 'nowarn', but not both.  Allowed
+            # will not issue a warning, but will log to the debug buffer what has likely
+            # been downloaded by SVN.
+            if not ("externals" in ud.parm and ud.parm["externals"] == "allowed"):
+                options.append("--ignore-externals")
+
             if ud.revision:
                 options.append("-r %s" % ud.revision)
                 suffix = "@%s" % (ud.revision)
@@ -136,6 +143,18 @@ class Svn(FetchMethod):
                 bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
                 runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
 
+            if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
+                # Warn the user if this had externals (won't catch them all)
+                output = runfetchcmd("svn propget svn:externals", d, workdir=ud.moddir)
+                if output:
+                    if "--ignore-externals" in svnfetchcmd.split():
+                        bb.warn("%s contains svn:externals." % ud.url)
+                        bb.warn("These should be added to the recipe SRC_URI as necessary.")
+                        bb.warn("svn fetch has ignored externals:\n%s" % output)
+                        bb.warn("To disable this warning add ';externals=nowarn' to the url.")
+                    else:
+                        bb.debug(1, "svn repository has externals:\n%s" % output)
+
             scmdata = ud.parm.get("scmdata", "")
             if scmdata == "keep":
                 tar_flags = ""
lib/bb/tests/fetch.py +80 -0

86 lines not on this page · raw message

Signature · 1 line
1.8.3.1