Skip to contentSkip to conversation

bitbake: fetch: Fix BB_FETCH_PREMIRRORONLY for git mirror tarballs

PATCH Superseded [email protected] 2 participants 3 messages Mar 31, 2025 – Apr 3, 2025

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 bitbake: fetch: Fix BB_FETCH_PREMIRRORONLY for git mirror tarballs

From: Julian Haller <[email protected]>

When invoking the original git fetcher after downloading a mirror tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch commands targeting the upstream source being executed silently. Ensure setting BB_NO_NETWORK before invoking the original fetcher. While this was only observed for git, setting this in general for all fetcher types makes sense at this location.

---
 bitbake/lib/bb/fetch2/__init__.py | 3 +++
 1 file changed, 3 insertions(+)
bitbake/lib/bb/fetch2/__init__.py +3 -0
index fed296097f..76f1440026 100644
@@ -1092,6 +1092,9 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
         # If that tarball is a local file:// we need to provide a symlink to it
         dldir = ld.getVar("DL_DIR")
 
+        if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
+            ld.setVar("BB_NO_NETWORK", "1")
+
         if origud.mirrortarballs and os.path.basename(ud.localpath) in origud.mirrortarballs and os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
             # Create donestamp in old format to avoid triggering a re-download
             if ud.donestamp:
Signature · 1 line
2.43.0
#1 lore raw

PATCH 2/2 bitbake: tests/fetch: Fix git PREMIRRORONLY test

From: Julian Haller <[email protected]>

Using a shallow clone to simluate an outdated git mirror tarball does not work in the intended way. A shallow clone already contains the latest commit which can hide certain fetcher behavior. Simulate an outdated mirror tarball, as the test titles indicate, by removing the newer commits from the mirror.

---
 bitbake/lib/bb/tests/fetch.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
bitbake/lib/bb/tests/fetch.py +4 -3
index 65339d1bb1..8de6862482 100644
@@ -3290,7 +3290,8 @@ class FetchPremirroronlyNetworkTest(FetcherTest):
         import shutil
         self.mirrorname = "git2_git.yoctoproject.org.fstests.tar.gz"
         os.makedirs(self.clonedir)
-        self.git("clone --bare --shallow-since=\"01.01.2013\" {}".format(self.recipe_url), self.clonedir)
+        self.git("clone --bare {}".format(self.recipe_url), self.clonedir)
+        self.git("update-ref HEAD 15413486df1f5a5b5af699b6f3ba5f0984e52a9f", self.gitdir)
         bb.process.run('tar -czvf {} .'.format(os.path.join(self.mirrordir, self.mirrorname)), cwd =  self.gitdir)
         shutil.rmtree(self.clonedir)
 
@@ -3298,7 +3299,7 @@ class FetchPremirroronlyNetworkTest(FetcherTest):
     def test_mirror_tarball_updated(self):
         self.make_git_repo()
         ## Upstream commit is in the mirror
-        self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec")
+        self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f")
         fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
         fetcher.download()
 
@@ -3306,7 +3307,7 @@ class FetchPremirroronlyNetworkTest(FetcherTest):
     def test_mirror_tarball_outdated(self):
         self.make_git_repo()
         ## Upstream commit not in the mirror
-        self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f")
+        self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec")
         fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
         with self.assertRaises(bb.fetch2.NetworkAccess):
             fetcher.download()
Signature · 1 line
2.43.0
#2 lore raw
In reply to [email protected]
Quoted 19 lines · Julian Haller via lists.openembedded.org

From: Julian Haller <[email protected]>

When invoking the original git fetcher after downloading a mirror tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch commands targeting the upstream source being executed silently. Ensure setting BB_NO_NETWORK before invoking the original fetcher. While this was only observed for git, setting this in general for all fetcher types makes sense at this location.
---
 bitbake/lib/bb/fetch2/__init__.py | 3 +++

 1 file changed, 3 insertions(+)
bitbake/lib/bb/fetch2/__init__.py +0 -0
index fed296097f..76f1440026 100644
@@ -1092,6 +1092,9 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
          # If that tarball is a local file:// we need to provide a symlink to it
          dldir = ld.getVar("DL_DIR")
Quoted 7 lines


+        if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
+            ld.setVar("BB_NO_NETWORK", "1")
+
         if origud.mirrortarballs and os.path.basename(ud.localpath) in origud.mirrortarballs and os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
             # Create donestamp in old format to avoid triggering a re-download
             if ud.donestamp:

This is nearly right but not quite. You need to ensure BB_NO_NETWORK doesn't leak back into the parent datastore. If you had multiple urls in SRC_URI, you'd see the network was potentially completely disabled after the first url.

You'll see that in the try_mirrors() function which calls try_mirror_url, there is a:

ld = d.createCopy()

which is effectively how we can isolate datastores.

You code probably needs to look like:

if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
ld = ld.createCopy() ld.setVar("BB_NO_NETWORK", "1")

Cheers,

Richard