Skip to contentSkip to conversation

*** archiver.bbclass: archive shallow git sources which have no clone ***

PATCH Superseded [email protected] 2 participants 5 messages Oct 30 – Nov 9

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover *** archiver.bbclass: archive shallow git sources which have no clone ***

From: Etienne Cordonnier <[email protected]>

*** BLURB HERE ***

Etienne Cordonnier (3):
archiver.bbclass: archive shallow git sources which have no clone archiver.bbclass: remove .git from original git source archives archiver.bbclass: fail on sources which cannot be archived

v2:
- Combined the two v1 patches into one series, as requested.
- Split "archive shallow git sources which have no clone" into 1/3 (archive the shallow source) and 2/3 (remove .git), and 2/3 now removes .git for all git sources, including the .git files of submodules (kept for nocheckout/bareclone).
- 3/3 ("warn about sources which are not archived" in v1) now raises a fatal error.

 meta/classes/archiver.bbclass            | 22 +++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 91 ++++++++++++++++++++++++
 2 files changed, 112 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 1/3 archiver.bbclass: archive shallow git sources which have no clone

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can use the gitshallow tarball (ud.fullshallow) without creating the clone: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. In this case, the source was silently skipped and not archived at all.

Treat this case like a directory source, as unpack() populates the directory from the shallow tarball. Use the same check as unpack() (clonedir_need_update()), so that the case of an out of date clone, where unpack() also uses the shallow tarball, is handled as well.

AI-Generated: Claude Sonnet 5.5

---
 meta/classes/archiver.bbclass            |  8 +++-
 meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)
meta/classes/archiver.bbclass +7 -1
index 0400d6e373..f6cca35186 100644
@@ -204,10 +204,16 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]
         local = fetch.localpath(url).rstrip("/");
+        # A shallow git fetch which used the shallow tarball has no up to date
+        # clone, which is what localpath() points to. unpack() then uses the
+        # shallow tarball, using the same check as here.
+        shallow = getattr(ud, 'shallow', False) and os.path.isfile(ud.fullshallow) \
+                  and ud.method.clonedir_need_update(ud, ld)
         if os.path.isfile(local):
             shutil.copy(local, ar_outdir)
-        elif os.path.isdir(local):
+        elif os.path.isdir(local) or shallow:
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
             # To handle recipes with more than one source, we add the "name"
meta/lib/oeqa/selftest/cases/archiver.py +48 -0

60 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v2 1/3] archiver.bbclass: archive shallow git sources which have no clone

Quoted 32 lines · ecordonnier

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can use the gitshallow tarball (ud.fullshallow) without creating the clone: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. In this case, the source was silently skipped and not archived at all.

Treat this case like a directory source, as unpack() populates the directory from the shallow tarball. Use the same check as unpack() (clonedir_need_update()), so that the case of an out of date clone, where unpack() also uses the shallow tarball, is handled as well.

AI-Generated: Claude Sonnet 5.5

---
 meta/classes/archiver.bbclass            |  8 +++-
 meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)
meta/classes/archiver.bbclass +1 -0
index 0400d6e373..f6cca35186 100644
@@ -204,10 +204,16 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]

meta/classes/archiver.bbclass python do_ar_original() { · line 216

Quoted 10 lines
meta/classes/archiver.bbclass +6 -1
         local = fetch.localpath(url).rstrip("/");
+        # A shallow git fetch which used the shallow tarball has no up to date
+        # clone, which is what localpath() points to. unpack() then uses the
+        # shallow tarball, using the same check as here.
+        shallow = getattr(ud, 'shallow', False) and os.path.isfile(ud.fullshallow) \
+                  and ud.method.clonedir_need_update(ud, ld)
         if os.path.isfile(local):
             shutil.copy(local, ar_outdir)
-        elif os.path.isdir(local):
+        elif os.path.isdir(local) or shallow:

Hi Etienne,

One question we had during the patch review call was whether the current fetcher behaviour makes sense. Usually, localpath() exists once do_fetch has completed, but in the case where we restore from a shallow mirror tarball that isn't true. That is probably the thing we should fix.

So, instead of this approach, could we modify the fetcher to ensure that localpath() exists when do_fetch completes? That would avoid the need for special handling in the archiver bbclass.

Best regards,

Signature · 1 line
Paul Barker
#3 lore raw

PATCH v2 2/3 archiver.bbclass: remove .git from original git source archives

In reply to [email protected]

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original unpacks git sources and archives the unpacked tree, including its .git. This .git is of no use in the archive:

- For a clone, unpack() uses clone -s, so .git has no objects, only a reference to the clone in DL_DIR via objects/info/alternates. It is confusing and still wastes a little space.
- For a shallow tarball, .git has all the (compressed) objects, which duplicates the sources and doubles the size of the archive.

Remove it, as well as the .git files of submodules. Keep it with nocheckout (which bareclone implies), where the repository is what gets unpacked.

AI-Generated: Claude Opus 5.5

---
 meta/classes/archiver.bbclass            | 12 +++++++
 meta/lib/oeqa/selftest/cases/archiver.py | 43 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
meta/classes/archiver.bbclass +12 -0
index f6cca35186..a6b5706abd 100644
@@ -216,6 +216,18 @@ python do_ar_original() {
         elif os.path.isdir(local) or shallow:
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
+            if isinstance(ud.method, bb.fetch.git.Git) and not ud.nocheckout:
+                # The .git of a clone has no objects (clone -s only references
+                # them via objects/info/alternates), and the one of a shallow
+                # tarball duplicates the sources, so it is of no use in the
+                # archive. This includes the .git files of submodules. Keep it
+                # with nocheckout (and bareclone), where it is the source.
+                for root, dirs, files in os.walk(ud.destdir):
+                    if '.git' in dirs:
+                        dirs.remove('.git')
+                        bb.utils.remove(os.path.join(root, '.git'), recurse=True)
+                    if '.git' in files:
+                        os.remove(os.path.join(root, '.git'))
             # To handle recipes with more than one source, we add the "name"
             # URL parameter as suffix. We treat it as an error when
             # there's more than one URL without a name, or a name gets reused.
meta/lib/oeqa/selftest/cases/archiver.py +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.43.0
#4 lore raw

PATCH v2 3/3 archiver.bbclass: fail on sources which cannot be archived

In reply to [email protected]

From: Etienne Cordonnier <[email protected]>

In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory. Any other source is silently skipped, and the source archive is incomplete without any notice. Until commit "archiver.bbclass: archive shallow git sources which have no clone", this happened for shallow git fetches, whose localpath is the clone in DL_DIR/git2, which is not created when the shallow tarball is used.

No fetcher has a valid case for a localpath which is neither a file nor a directory after do_fetch, so make it a fatal error.

AI-Generated: Claude Opus 5.5

---
 meta/classes/archiver.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/archiver.bbclass +2 -0
index a6b5706abd..ec62b89400 100644
@@ -249,6 +249,8 @@ python do_ar_original() {
                         bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (name, tarball_suffix[name], url))
             tarball_suffix[name] = url
             create_tarball(d, tmpdir + '/.', name, ar_outdir)
+        else:
+            bb.fatal("Cannot archive %s: its local path %s is neither a file nor a directory" % (url, local))
 
     # Emit patch series files for 'original'
     bb.note('Writing patch series files...')
Signature · 1 line
2.43.0