Skip to contentSkip to conversation

fetch2: give every FetchData a default unpack_tracer

PATCH In review Siva Balasubramanian 2 participants 3 messages Jul 3

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fetch2: give every FetchData a default unpack_tracer

Fetch() sets an unpack_tracer attribute on the FetchData objects it manages, but the per-mirror FetchData objects created in build_mirroruris() are constructed directly and never get one. When a mirror is used for a git recipe that needs Git LFS, git.py's download() performs a checkout (via Git.unpack()) on the mirror's FetchData to materialise the LFS objects, and that unpack path dereferences ud.unpack_tracer, failing with:

AttributeError: 'FetchData' object has no attribute 'unpack_tracer'

so PREMIRRORS/MIRRORS fetching is broken for git-lfs sources.

Fix this at the source by initialising unpack_tracer to a DummyUnpackTracer in FetchData.__init__(), so the attribute always exists. Fetch() still overrides it with the real (possibly user-configured via BB_UNPACK_TRACER_CLASS) tracer for the URLs it manages; the mirror FetchData objects only perform an internal, throwaway checkout that should not be traced anyway.

Add a MirrorUriTest regression test asserting the mirror FetchData objects carry an unpack_tracer.

[YOCTO #15948]

---
 lib/bb/fetch2/__init__.py |  6 ++++++
 lib/bb/tests/fetch.py     | 11 +++++++++++
 2 files changed, 17 insertions(+)
lib/bb/fetch2/__init__.py +6 -0
index ce7456b60..28d3d50d4 100644
@@ -1325,6 +1325,12 @@ class FetchData(object):
         self.mirrortarballs = []
         self.basename = None
         self.basepath = None
+        # Default to a no-op tracer. Fetch() replaces this with the real
+        # (possibly user-configured) unpack tracer for the URLs it manages,
+        # but FetchData objects created elsewhere (e.g. the per-mirror ones
+        # built in build_mirroruris()) must still have the attribute so that
+        # fetcher unpack methods invoked on them do not raise AttributeError.
+        self.unpack_tracer = DummyUnpackTracer()
         (self.type, self.host, self.path, self.user, self.pswd, self.parm) = decodeurl(d.expand(url))
         self.date = self.getSRCDate(d)
         self.url = url
lib/bb/tests/fetch.py +11 -0
index ebc80aa8c..f9c7e3116 100644
@@ -530,6 +530,17 @@ class MirrorUriTest(FetcherTest):
         uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d)
         self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', 'file:///someotherpath/downloads/bitbake-1.0.tar.gz'])
 
+    def test_mirror_uds_have_unpack_tracer(self):
+        # The per-mirror FetchData objects must carry an unpack_tracer, otherwise
+        # fetcher unpack methods (e.g. the git-lfs checkout done for a mirror)
+        # raise AttributeError. See YOCTO #15948.
+        fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
+        mirrors = bb.fetch2.mirror_from_string(self.mirrorvar)
+        uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d)
+        self.assertTrue(uds)
+        for ud in uds:
+            self.assertTrue(hasattr(ud, "unpack_tracer"))
+
     def test_urilist2(self):
         # Catch https:// -> files:// bug
         fetcher = bb.fetch.FetchData("https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
Signature · 1 line
2.34.1
#1 lore raw
Quoted 19 lines · Siva Balasubramanian via lists.openembedded.org

Fetch() sets an unpack_tracer attribute on the FetchData objects it manages, but the per-mirror FetchData objects created in build_mirroruris() are constructed directly and never get one. When a mirror is used for a git recipe that needs Git LFS, git.py's download() performs a checkout (via Git.unpack()) on the mirror's FetchData to materialise the LFS objects, and that unpack path dereferences ud.unpack_tracer, failing with:

    AttributeError: 'FetchData' object has no attribute 'unpack_tracer'

so PREMIRRORS/MIRRORS fetching is broken for git-lfs sources.

Fix this at the source by initialising unpack_tracer to a DummyUnpackTracer in FetchData.__init__(), so the attribute always exists. Fetch() still overrides it with the real (possibly user-configured via BB_UNPACK_TRACER_CLASS) tracer for the URLs it manages; the mirror FetchData objects only perform an internal, throwaway checkout that should not be traced anyway.

Quoted 8 lines

Add a MirrorUriTest regression test asserting the mirror FetchData objects carry an unpack_tracer.

[YOCTO #15948]

Wasn't this already fixed by:

https://git.openembedded.org/bitbake/commit/?id=f0c9cf8d3885c5b1c2ba448f064421dae476fcd0

?

We don't want to add a dummy one everywhere, only add one when needed as the patch avove does...

Cheers,

Richard

#2 lore raw

Thanks Richard, you're right, this is already fixed in master by f0c9cf8d ("fetch2: make unpack_tracer available when fetching mirror URLs"). My checkout was behind and I missed it. Please drop this patch

Quoted 44 lines · Richard Purdie
Quoted 19 lines · Siva Balasubramanian via lists.openembedded.org

Fetch() sets an unpack_tracer attribute on the FetchData objects it manages, but the per-mirror FetchData objects created in build_mirroruris() are constructed directly and never get one. When a mirror is used for a git recipe that needs Git LFS, git.py's download() performs a checkout (via Git.unpack()) on the mirror's FetchData to materialise the LFS objects, and that unpack path dereferences ud.unpack_tracer, failing with:

AttributeError: 'FetchData' object has no attribute 'unpack_tracer'

so PREMIRRORS/MIRRORS fetching is broken for git-lfs sources.

Fix this at the source by initialising unpack_tracer to a DummyUnpackTracer in FetchData.__init__(), so the attribute always exists. Fetch() still overrides it with the real (possibly user-configured via BB_UNPACK_TRACER_CLASS) tracer for the URLs it manages; the mirror FetchData objects only perform an internal, throwaway checkout that should not be traced anyway.

Quoted 8 lines

Add a MirrorUriTest regression test asserting the mirror FetchData objects carry an unpack_tracer.

[YOCTO #15948]

Wasn't this already fixed by:

https://git.openembedded.org/bitbake/commit/?id=f0c9cf8d3885c5b1c2ba448f064421dae476fcd0

?

We don't want to add a dummy one everywhere, only add one when needed as the patch avove does...

Cheers,

Richard

Signature · 1 line
Sivakumar B