*** archiver.bbclass: archive shallow git sources which have no clone ***
v2 cover *** archiver.bbclass: archive shallow git sources which have no clone ***
From: Etienne Cordonnier <[email protected]>
*** BLURB HERE ***
Etienne Cordonnier (3):
archiver.bbclass: archive shallow git sources which have no clone archiver.bbclass: remove .git from original git source archives archiver.bbclass: fail on sources which cannot be archived
v2:
- Combined the two v1 patches into one series, as requested.
- Split "archive shallow git sources which have no clone" into 1/3 (archive the shallow source) and 2/3 (remove .git), and 2/3 now removes .git for all git sources, including the .git files of submodules (kept for nocheckout/bareclone).
- 3/3 ("warn about sources which are not archived" in v1) now raises a fatal error.
meta/classes/archiver.bbclass | 22 +++++- meta/lib/oeqa/selftest/cases/archiver.py | 91 ++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.0
PATCH v2 1/3 archiver.bbclass: archive shallow git sources which have no clone
From: Etienne Cordonnier <[email protected]>
In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can use the gitshallow tarball (ud.fullshallow) without creating the clone: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. In this case, the source was silently skipped and not archived at all.
Treat this case like a directory source, as unpack() populates the directory from the shallow tarball. Use the same check as unpack() (clonedir_need_update()), so that the case of an out of date clone, where unpack() also uses the shallow tarball, is handled as well.
AI-Generated: Claude Sonnet 5.5
--- meta/classes/archiver.bbclass | 8 +++- meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-)
meta/classes/archiver.bbclass +7 -1
@@ -204,10 +204,16 @@ python do_ar_original() { fetch = bb.fetch.Fetch(urls, ld) tarball_suffix = {} for url in fetch.urls: + ud = fetch.ud[url] local = fetch.localpath(url).rstrip("/"); + # A shallow git fetch which used the shallow tarball has no up to date + # clone, which is what localpath() points to. unpack() then uses the + # shallow tarball, using the same check as here. + shallow = getattr(ud, 'shallow', False) and os.path.isfile(ud.fullshallow) \ + and ud.method.clonedir_need_update(ud, ld) if os.path.isfile(local): shutil.copy(local, ar_outdir) - elif os.path.isdir(local): + elif os.path.isdir(local) or shallow: tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR')) fetch.unpack(tmpdir, (url,)) # To handle recipes with more than one source, we add the "name"
meta/lib/oeqa/selftest/cases/archiver.py +48 -0
60 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v2 1/3] archiver.bbclass: archive shallow git sources which have no clone
Quoted 32 lines · ecordonnier
From: Etienne Cordonnier <[email protected]>
In the original mode,
do_ar_originalarchives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can use the gitshallow tarball (ud.fullshallow) without creating the clone: since bitbake commit457288b2f("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. In this case, the source was silently skipped and not archived at all.Treat this case like a directory source, as
unpack()populates the directory from the shallow tarball. Use the same check asunpack()(clonedir_need_update()), so that the case of an out of date clone, whereunpack()also uses the shallow tarball, is handled as well.AI-Generated: Claude Sonnet 5.5
Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes/archiver.bbclass | 8 +++- meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-)meta/classes/archiver.bbclass +1
-0@@ -204,10 +204,16 @@ python do_ar_original() { fetch = bb.fetch.Fetch(urls, ld) tarball_suffix = {} for url in fetch.urls: + ud = fetch.ud[url]
meta/classes/archiver.bbclass python do_ar_original() { · line 216
Quoted 10 lines
meta/classes/archiver.bbclass +6
-1local = fetch.localpath(url).rstrip("/"); + # A shallow git fetch which used the shallow tarball has no up to date + # clone, which is what localpath() points to. unpack() then uses the + # shallow tarball, using the same check as here. + shallow = getattr(ud, 'shallow', False) and os.path.isfile(ud.fullshallow) \ + and ud.method.clonedir_need_update(ud, ld) if os.path.isfile(local): shutil.copy(local, ar_outdir) - elif os.path.isdir(local): + elif os.path.isdir(local) or shallow:
Hi Etienne,
One question we had during the patch review call was whether the current fetcher behaviour makes sense. Usually, localpath() exists once do_fetch has completed, but in the case where we restore from a shallow mirror tarball that isn't true. That is probably the thing we should fix.
So, instead of this approach, could we modify the fetcher to ensure that localpath() exists when do_fetch completes? That would avoid the need for special handling in the archiver bbclass.
Best regards,
Signature · 1 line
Paul Barker
PATCH v2 2/3 archiver.bbclass: remove .git from original git source archives
In reply to [email protected]From: Etienne Cordonnier <[email protected]>
In the original mode, do_ar_original unpacks git sources and archives the unpacked tree, including its .git. This .git is of no use in the archive:
- For a clone, unpack() uses clone -s, so .git has no objects, only a reference to the clone in DL_DIR via objects/info/alternates. It is confusing and still wastes a little space.
- For a shallow tarball, .git has all the (compressed) objects, which duplicates the sources and doubles the size of the archive.
Remove it, as well as the .git files of submodules. Keep it with nocheckout (which bareclone implies), where the repository is what gets unpacked.
AI-Generated: Claude Opus 5.5
--- meta/classes/archiver.bbclass | 12 +++++++ meta/lib/oeqa/selftest/cases/archiver.py | 43 ++++++++++++++++++++++++ 2 files changed, 55 insertions(+)
meta/classes/archiver.bbclass +12 -0
@@ -216,6 +216,18 @@ python do_ar_original() { elif os.path.isdir(local) or shallow: tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR')) fetch.unpack(tmpdir, (url,)) + if isinstance(ud.method, bb.fetch.git.Git) and not ud.nocheckout: + # The .git of a clone has no objects (clone -s only references + # them via objects/info/alternates), and the one of a shallow + # tarball duplicates the sources, so it is of no use in the + # archive. This includes the .git files of submodules. Keep it + # with nocheckout (and bareclone), where it is the source. + for root, dirs, files in os.walk(ud.destdir): + if '.git' in dirs: + dirs.remove('.git') + bb.utils.remove(os.path.join(root, '.git'), recurse=True) + if '.git' in files: + os.remove(os.path.join(root, '.git')) # To handle recipes with more than one source, we add the "name" # URL parameter as suffix. We treat it as an error when # there's more than one URL without a name, or a name gets reused.
meta/lib/oeqa/selftest/cases/archiver.py +43 -0
49 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v2 3/3 archiver.bbclass: fail on sources which cannot be archived
In reply to [email protected]From: Etienne Cordonnier <[email protected]>
In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory. Any other source is silently skipped, and the source archive is incomplete without any notice. Until commit "archiver.bbclass: archive shallow git sources which have no clone", this happened for shallow git fetches, whose localpath is the clone in DL_DIR/git2, which is not created when the shallow tarball is used.
No fetcher has a valid case for a localpath which is neither a file nor a directory after do_fetch, so make it a fatal error.
AI-Generated: Claude Opus 5.5
--- meta/classes/archiver.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/archiver.bbclass +2 -0
@@ -249,6 +249,8 @@ python do_ar_original() { bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (name, tarball_suffix[name], url)) tarball_suffix[name] = url create_tarball(d, tmpdir + '/.', name, ar_outdir) + else: + bb.fatal("Cannot archive %s: its local path %s is neither a file nor a directory" % (url, local)) # Emit patch series files for 'original' bb.note('Writing patch series files...')
Signature · 1 line
2.43.0