Skip to contentSkip to conversation

archiver.bbclass: archive shallow git sources which have no clone

PATCH In review [email protected] 2 participants 5 messages Nov 17 – Nov 20

lore
Share & tools
lore
#0 lore raw

v3 cover archiver.bbclass: archive shallow git sources which have no clone

From: Etienne Cordonnier <[email protected]>

As feedback to the v2 of this patch, it was requested to modify the fetcher to ensure that localpath() exists when do_fetch completes, in order to avoid the need
for special handling in the archiver bbclass.

This v3 series addresses this feedback, and requires the bitbake change "fetch/git: return the shallow tarball as localpath when there is no clone"

Etienne Cordonnier (3):
archiver.bbclass: archive shallow git sources which have no clone archiver.bbclass: remove .git from original git source archives archiver.bbclass: fail on sources which cannot be archived

 meta/classes/archiver.bbclass            | 23 +++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 91 ++++++++++++++++++++++++
 2 files changed, 112 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v3 1/3 archiver.bbclass: archive shallow git sources which have no clone

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath was the clone in DL_DIR/git2, which a shallow fetch (BB_GIT_SHALLOW) does not create when it uses the gitshallow tarball: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. The source was then silently skipped and not archived at all.

The git fetcher now returns the shallow tarball as localpath in this case. Copying it would make the archive differ depending on how the source was fetched, so unpack git sources like a clone, which uses the shallow tarball.

AI-Generated: Claude Sonnet 5.5

---
 meta/classes/archiver.bbclass            |  9 ++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +7 -2
index 0400d6e373..d1a8dadec7 100644
@@ -204,10 +204,15 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]
+        is_git = isinstance(ud.method, bb.fetch.git.Git)
         local = fetch.localpath(url).rstrip("/");
-        if os.path.isfile(local):
+        # The localpath of a git source is the shallow tarball, instead of the
+        # clone, when the fetch used the shallow tarball only. Unpack it like
+        # a clone instead of archiving the tarball itself.
+        if os.path.isfile(local) and not is_git:
             shutil.copy(local, ar_outdir)
-        elif os.path.isdir(local):
+        elif os.path.isdir(local) or (is_git and os.path.isfile(local)):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
             # To handle recipes with more than one source, we add the "name"
meta/lib/oeqa/selftest/cases/archiver.py +48 -0

60 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw
Quoted 27 lines · ecordonnier

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath was the clone in DL_DIR/git2, which a shallow fetch (BB_GIT_SHALLOW) does not create when it uses the gitshallow tarball: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. The source was then silently skipped and not archived at all.

The git fetcher now returns the shallow tarball as localpath in this case. Copying it would make the archive differ depending on how the source was fetched, so unpack git sources like a clone, which uses the shallow tarball.

AI-Generated: Claude Sonnet 5.5

---
 meta/classes/archiver.bbclass            |  9 ++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +0 -0
index 0400d6e373..d1a8dadec7 100644

meta/classes/archiver.bbclass python do_ar_original() { · line 208

Quoted 5 lines
meta/classes/archiver.bbclass +2 -0
@@ -204,10 +204,15 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]
+        is_git = isinstance(ud.method, bb.fetch.git.Git)

Hi Etienne,

These changes are definitely moving in the right direction, but I don't think they're quite ready to merge yet.

Here we are still special casing particular fetchers, which we shouldn't need to do.

I think we just need two cases:

1) If the path returned by localpath() is a file, copy it.

2) Otherwise, call fetch.unpack() into a tmpdir, regardless of whether the path returned by localpath() exists or not. Raise an error if the tmpdir is still empty after the unpack.

Best regards,

Signature · 1 line
Paul Barker
#3 lore raw

PATCH v3 2/3 archiver.bbclass: remove .git from original git source archives

In reply to [email protected]

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original unpacks git sources and archives the unpacked tree, including its .git. This .git is of no use in the archive:

- For a clone, unpack() uses clone -s, so .git has no objects, only a reference to the clone in DL_DIR via objects/info/alternates. It is confusing and still wastes a little space.
- For a shallow tarball, .git has all the (compressed) objects, which duplicates the sources and doubles the size of the archive.

Remove it, as well as the .git files of submodules. Keep it with nocheckout (which bareclone implies), where the repository is what gets unpacked.

AI-Generated: Claude Opus 5.5

---
 meta/classes/archiver.bbclass            | 12 +++++++
 meta/lib/oeqa/selftest/cases/archiver.py | 43 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
meta/classes/archiver.bbclass +12 -0
index d1a8dadec7..d22213be33 100644
@@ -215,6 +215,18 @@ python do_ar_original() {
         elif os.path.isdir(local) or (is_git and os.path.isfile(local)):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
+            if is_git and not ud.nocheckout:
+                # The .git of a clone has no objects (clone -s only references
+                # them via objects/info/alternates), and the one of a shallow
+                # tarball duplicates the sources, so it is of no use in the
+                # archive. This includes the .git files of submodules. Keep it
+                # with nocheckout (and bareclone), where it is the source.
+                for root, dirs, files in os.walk(ud.destdir):
+                    if '.git' in dirs:
+                        dirs.remove('.git')
+                        bb.utils.remove(os.path.join(root, '.git'), recurse=True)
+                    if '.git' in files:
+                        os.remove(os.path.join(root, '.git'))
             # To handle recipes with more than one source, we add the "name"
             # URL parameter as suffix. We treat it as an error when
             # there's more than one URL without a name, or a name gets reused.
meta/lib/oeqa/selftest/cases/archiver.py +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.43.0
#4 lore raw

PATCH v3 3/3 archiver.bbclass: fail on sources which cannot be archived

In reply to [email protected]

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory. Any other source is silently skipped, and the source archive is incomplete without any notice. This used to happen for shallow git fetches without a clone, which is fixed by commit "archiver.bbclass: archive shallow git sources which have no clone".

No fetcher has a valid case for a localpath which is neither a file nor a directory after do_fetch, so make it a fatal error.

AI-Generated: Claude Opus 5.5

---
 meta/classes/archiver.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/archiver.bbclass +2 -0
index d22213be33..a54882eeb8 100644
@@ -248,6 +248,8 @@ python do_ar_original() {
                         bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (name, tarball_suffix[name], url))
             tarball_suffix[name] = url
             create_tarball(d, tmpdir + '/.', name, ar_outdir)
+        else:
+            bb.fatal("Cannot archive %s: its local path %s is neither a file nor a directory" % (url, local))
 
     # Emit patch series files for 'original'
     bb.note('Writing patch series files...')
Signature · 1 line
2.43.0