Skip to contentSkip to conversation

archiver.bbclass: archive shallow git sources which have no clone

PATCH Superseded [email protected] 2 participants 2 messages Oct 30

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 archiver.bbclass: archive shallow git sources which have no clone

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can use the gitshallow tarball (ud.fullshallow) without creating the clone: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. In this case, the source was silently skipped and not archived at all.

Treat this case like a directory source: unpack() can unpack the shallow tarball. Use the same check as unpack() (clonedir_need_update()), so that the case of an out of date clone, where unpack() also uses the shallow tarball, is handled as well.

The shallow tarball only contains the repository, so the unpacked tree has a .git with all the (compressed) objects, which would double the size of the archive. Drop it. The .git of a clone has no objects either (clone -s only references them via objects/info/alternates), so neither archive contains a usable repository.

AI-Generated: Claude Sonnet 5.5

---
 meta/classes/archiver.bbclass            | 14 +++++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 41 ++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 1 deletion(-)
meta/classes/archiver.bbclass +13 -1
index 0400d6e373..06ce6443ee 100644
@@ -204,12 +204,24 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]
         local = fetch.localpath(url).rstrip("/");
+        # A shallow git fetch which used the shallow tarball has no up to date
+        # clone, which is what localpath() points to. unpack() then uses the
+        # shallow tarball, using the same check as here.
+        shallow = getattr(ud, 'shallow', False) and os.path.isfile(ud.fullshallow) \
+                  and ud.method.clonedir_need_update(ud, ld)
         if os.path.isfile(local):
             shutil.copy(local, ar_outdir)
-        elif os.path.isdir(local):
+        elif os.path.isdir(local) or shallow:
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
+            if shallow:
+                # The unpacked shallow tarball has a .git with all the objects,
+                # which would ship the sources twice. Drop it: the .git of a
+                # clone has no objects either (clone -s), so neither archive
+                # contains a usable repository.
+                bb.utils.remove(os.path.join(ud.destdir, '.git'), recurse=True)
             # To handle recipes with more than one source, we add the "name"
             # URL parameter as suffix. We treat it as an error when
             # there's more than one URL without a name, or a name gets reused.
meta/lib/oeqa/selftest/cases/archiver.py +41 -0

53 lines not on this page · raw message

Signature · 1 line
2.43.0
#1 lore raw
Quoted 16 lines · On Fri, 2026-10-30 at 14:22 +0100, [email protected] wrote:

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can use the gitshallow tarball (ud.fullshallow) without creating the clone: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. In this case, the source was silently skipped and not archived at all.

Treat this case like a directory source: unpack() can unpack the shallow tarball. Use the same check as unpack() (clonedir_need_update()), so that the case of an out of date clone, where unpack() also uses the shallow tarball, is handled as well.

Quoted 5 lines

The shallow tarball only contains the repository, so the unpacked tree has a .git with all the (compressed) objects, which would double the size of the archive. Drop it. The .git of a clone has no objects either (clone -s only references them via objects/info/alternates), so neither archive contains a usable repository.

Hi Etienne,

I think there are two changes here, which should be separate commits:

1) If the localpath doesn't exist yet because the fetch used a shallow mirror tarball, treat it as a directory as fetch.unpack() will populate the directory.

2) Remove the .git directory to save space and give a clean archive tarball. Should this also be done for non-shallow git clones? If the .git directory doesn't contain objects anyway then it's just confusing and still wastes a little space.

We should still catch the case where both isfile() and isdir() return false and raise a fatal error in that case. So I'd like to see a v2 for your other patch as well. It's better to send them as a combined v2 series.

Best regards,

Signature · 1 line
Paul Barker