bitbake: fetch: Fix BB_FETCH_PREMIRRORONLY for git mirror tarballs
PATCH 1/2 bitbake: fetch: Fix BB_FETCH_PREMIRRORONLY for git mirror tarballs
From: Julian Haller <[email protected]>
When invoking the original git fetcher after downloading a mirror tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch commands targeting the upstream source being executed silently. Ensure setting BB_NO_NETWORK before invoking the original fetcher. While this was only observed for git, setting this in general for all fetcher types makes sense at this location.
--- bitbake/lib/bb/fetch2/__init__.py | 3 +++ 1 file changed, 3 insertions(+)
bitbake/lib/bb/fetch2/__init__.py +3 -0
@@ -1092,6 +1092,9 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): # If that tarball is a local file:// we need to provide a symlink to it dldir = ld.getVar("DL_DIR") + if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")): + ld.setVar("BB_NO_NETWORK", "1") + if origud.mirrortarballs and os.path.basename(ud.localpath) in origud.mirrortarballs and os.path.basename(ud.localpath) != os.path.basename(origud.localpath): # Create donestamp in old format to avoid triggering a re-download if ud.donestamp:
Signature · 1 line
2.43.0
PATCH 2/2 bitbake: tests/fetch: Fix git PREMIRRORONLY test
From: Julian Haller <[email protected]>
Using a shallow clone to simluate an outdated git mirror tarball does not work in the intended way. A shallow clone already contains the latest commit which can hide certain fetcher behavior. Simulate an outdated mirror tarball, as the test titles indicate, by removing the newer commits from the mirror.
--- bitbake/lib/bb/tests/fetch.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
bitbake/lib/bb/tests/fetch.py +4 -3
@@ -3290,7 +3290,8 @@ class FetchPremirroronlyNetworkTest(FetcherTest): import shutil self.mirrorname = "git2_git.yoctoproject.org.fstests.tar.gz" os.makedirs(self.clonedir) - self.git("clone --bare --shallow-since=\"01.01.2013\" {}".format(self.recipe_url), self.clonedir) + self.git("clone --bare {}".format(self.recipe_url), self.clonedir) + self.git("update-ref HEAD 15413486df1f5a5b5af699b6f3ba5f0984e52a9f", self.gitdir) bb.process.run('tar -czvf {} .'.format(os.path.join(self.mirrordir, self.mirrorname)), cwd = self.gitdir) shutil.rmtree(self.clonedir)
@@ -3298,7 +3299,7 @@ class FetchPremirroronlyNetworkTest(FetcherTest): def test_mirror_tarball_updated(self): self.make_git_repo() ## Upstream commit is in the mirror - self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec") + self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f") fetcher = bb.fetch.Fetch([self.recipe_url], self.d) fetcher.download()
@@ -3306,7 +3307,7 @@ class FetchPremirroronlyNetworkTest(FetcherTest): def test_mirror_tarball_outdated(self): self.make_git_repo() ## Upstream commit not in the mirror - self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f") + self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec") fetcher = bb.fetch.Fetch([self.recipe_url], self.d) with self.assertRaises(bb.fetch2.NetworkAccess): fetcher.download()
Signature · 1 line
2.43.0
Quoted 19 lines · Julian Haller via lists.openembedded.org
From: Julian Haller <[email protected]>
When invoking the original git fetcher after downloading a mirror tarball,
BB_FETCH_PREMIRRORONLYis ignored. This leads to git fetch commands targeting the upstream source being executed silently. Ensure settingBB_NO_NETWORKbefore invoking the original fetcher. While this was only observed for git, setting this in general for all fetcher types makes sense at this location.
---
bitbake/lib/bb/fetch2/__init__.py | 3 +++1 file changed, 3 insertions(+)bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -1092,6 +1092,9 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): # If that tarball is a local file:// we need to provide a symlink to it dldir = ld.getVar("DL_DIR")
Quoted 7 lines
+ if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
+ld.setVar("BB_NO_NETWORK", "1")
+
if origud.mirrortarballs andos.path.basename(ud.localpath)in origud.mirrortarballs andos.path.basename(ud.localpath)!=os.path.basename(origud.localpath):
# Create donestamp in old format to avoid triggering a re-download
if ud.donestamp:
This is nearly right but not quite. You need to ensure BB_NO_NETWORK doesn't leak back into the parent datastore. If you had multiple urls in SRC_URI, you'd see the network was potentially completely disabled after the first url.
You'll see that in the try_mirrors() function which calls try_mirror_url, there is a:
ld = d.createCopy()
which is effectively how we can isolate datastores.
You code probably needs to look like:
if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
ld = ld.createCopy() ld.setVar("BB_NO_NETWORK", "1")
Cheers,
Richard