fetch2: give every FetchData a default unpack_tracer
PATCH 1/1 fetch2: give every FetchData a default unpack_tracer
Fetch() sets an unpack_tracer attribute on the FetchData objects it manages, but the per-mirror FetchData objects created in build_mirroruris() are constructed directly and never get one. When a mirror is used for a git recipe that needs Git LFS, git.py's download() performs a checkout (via Git.unpack()) on the mirror's FetchData to materialise the LFS objects, and that unpack path dereferences ud.unpack_tracer, failing with:
AttributeError: 'FetchData' object has no attribute 'unpack_tracer'
so PREMIRRORS/MIRRORS fetching is broken for git-lfs sources.
Fix this at the source by initialising unpack_tracer to a DummyUnpackTracer in FetchData.__init__(), so the attribute always exists. Fetch() still overrides it with the real (possibly user-configured via BB_UNPACK_TRACER_CLASS) tracer for the URLs it manages; the mirror FetchData objects only perform an internal, throwaway checkout that should not be traced anyway.
Add a MirrorUriTest regression test asserting the mirror FetchData objects carry an unpack_tracer.
[YOCTO #15948]
--- lib/bb/fetch2/__init__.py | 6 ++++++ lib/bb/tests/fetch.py | 11 +++++++++++ 2 files changed, 17 insertions(+)
lib/bb/fetch2/__init__.py +6 -0
@@ -1325,6 +1325,12 @@ class FetchData(object): self.mirrortarballs = [] self.basename = None self.basepath = None + # Default to a no-op tracer. Fetch() replaces this with the real + # (possibly user-configured) unpack tracer for the URLs it manages, + # but FetchData objects created elsewhere (e.g. the per-mirror ones + # built in build_mirroruris()) must still have the attribute so that + # fetcher unpack methods invoked on them do not raise AttributeError. + self.unpack_tracer = DummyUnpackTracer() (self.type, self.host, self.path, self.user, self.pswd, self.parm) = decodeurl(d.expand(url)) self.date = self.getSRCDate(d) self.url = url
lib/bb/tests/fetch.py +11 -0
@@ -530,6 +530,17 @@ class MirrorUriTest(FetcherTest): uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', 'file:///someotherpath/downloads/bitbake-1.0.tar.gz']) + def test_mirror_uds_have_unpack_tracer(self): + # The per-mirror FetchData objects must carry an unpack_tracer, otherwise + # fetcher unpack methods (e.g. the git-lfs checkout done for a mirror) + # raise AttributeError. See YOCTO #15948. + fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d) + mirrors = bb.fetch2.mirror_from_string(self.mirrorvar) + uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d) + self.assertTrue(uds) + for ud in uds: + self.assertTrue(hasattr(ud, "unpack_tracer")) + def test_urilist2(self): # Catch https:// -> files:// bug fetcher = bb.fetch.FetchData("https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
Signature · 1 line
2.34.1
Quoted 19 lines · Siva Balasubramanian via lists.openembedded.org
Fetch()sets anunpack_tracerattribute on the FetchData objects it manages, but the per-mirror FetchData objects created inbuild_mirroruris()are constructed directly and never get one. When a mirror is used for a git recipe that needs Git LFS, git.py'sdownload()performs a checkout (viaGit.unpack()) on the mirror's FetchData to materialise the LFS objects, and that unpack path dereferences ud.unpack_tracer, failing with:AttributeError: 'FetchData' object has no attribute '
unpack_tracer'so PREMIRRORS/MIRRORS fetching is broken for git-lfs sources.
Fix this at the source by initialising
unpack_tracerto a DummyUnpackTracer inFetchData.__init__(), so the attribute always exists.Fetch()still overrides it with the real (possibly user-configured viaBB_UNPACK_TRACER_CLASS) tracer for the URLs it manages; the mirror FetchData objects only perform an internal, throwaway checkout that should not be traced anyway.
Quoted 8 lines
Add a MirrorUriTest regression test asserting the mirror FetchData objects carry an
unpack_tracer.Reported-by Oliver Feilner <[email protected]>[YOCTO #15948]
Signed-off-by Siva Balasubramanian <[email protected]>
Wasn't this already fixed by:
https://git.openembedded.org/bitbake/commit/?id=f0c9cf8d3885c5b1c2ba448f064421dae476fcd0
?
We don't want to add a dummy one everywhere, only add one when needed as the patch avove does...
Cheers,
Richard
Thanks Richard, you're right, this is already fixed in master by f0c9cf8d ("fetch2: make unpack_tracer available when fetching mirror URLs"). My checkout was behind and I missed it. Please drop this patch
Quoted 44 lines · Richard Purdie
Quoted 19 lines · Siva Balasubramanian via lists.openembedded.org
Fetch()sets anunpack_tracerattribute on the FetchData objects it manages, but the per-mirror FetchData objects created inbuild_mirroruris()are constructed directly and never get one. When a mirror is used for a git recipe that needs Git LFS, git.py'sdownload()performs a checkout (viaGit.unpack()) on the mirror's FetchData to materialise the LFS objects, and that unpack path dereferences ud.unpack_tracer, failing with:AttributeError: 'FetchData' object has no attribute '
unpack_tracer'so PREMIRRORS/MIRRORS fetching is broken for git-lfs sources.
Fix this at the source by initialising
unpack_tracerto a DummyUnpackTracer inFetchData.__init__(), so the attribute always exists.Fetch()still overrides it with the real (possibly user-configured viaBB_UNPACK_TRACER_CLASS) tracer for the URLs it manages; the mirror FetchData objects only perform an internal, throwaway checkout that should not be traced anyway.Quoted 8 lines
Add a MirrorUriTest regression test asserting the mirror FetchData objects carry an
unpack_tracer.Reported-by Oliver Feilner <[email protected]>[YOCTO #15948]
Signed-off-by Siva Balasubramanian <[email protected]>Wasn't this already fixed by:
https://git.openembedded.org/bitbake/commit/?id=f0c9cf8d3885c5b1c2ba448f064421dae476fcd0
?
We don't want to add a dummy one everywhere, only add one when needed as the patch avove does...
Cheers,
Richard
Signature · 1 line
Sivakumar B