archiver.bbclass: archive shallow git sources which have no clone
v3 cover archiver.bbclass: archive shallow git sources which have no clone
From: Etienne Cordonnier <[email protected]>
As feedback to the v2 of this patch, it was requested to modify the fetcher to ensure that localpath() exists when do_fetch completes, in order to avoid the need
for special handling in the archiver bbclass.
This v3 series addresses this feedback, and requires the bitbake change "fetch/git: return the shallow tarball as localpath when there is no clone"
Etienne Cordonnier (3):
archiver.bbclass: archive shallow git sources which have no clone archiver.bbclass: remove .git from original git source archives archiver.bbclass: fail on sources which cannot be archived
meta/classes/archiver.bbclass | 23 +++++- meta/lib/oeqa/selftest/cases/archiver.py | 91 ++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
PATCH v3 1/3 archiver.bbclass: archive shallow git sources which have no clone
From: Etienne Cordonnier <[email protected]>
In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath was the clone in DL_DIR/git2, which a shallow fetch (BB_GIT_SHALLOW) does not create when it uses the gitshallow tarball: since bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. The source was then silently skipped and not archived at all.
The git fetcher now returns the shallow tarball as localpath in this case. Copying it would make the archive differ depending on how the source was fetched, so unpack git sources like a clone, which uses the shallow tarball.
AI-Generated: Claude Sonnet 5.5
--- meta/classes/archiver.bbclass | 9 ++++- meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +7 -2
@@ -204,10 +204,15 @@ python do_ar_original() { fetch = bb.fetch.Fetch(urls, ld) tarball_suffix = {} for url in fetch.urls: + ud = fetch.ud[url] + is_git = isinstance(ud.method, bb.fetch.git.Git) local = fetch.localpath(url).rstrip("/"); - if os.path.isfile(local): + # The localpath of a git source is the shallow tarball, instead of the + # clone, when the fetch used the shallow tarball only. Unpack it like + # a clone instead of archiving the tarball itself. + if os.path.isfile(local) and not is_git: shutil.copy(local, ar_outdir) - elif os.path.isdir(local): + elif os.path.isdir(local) or (is_git and os.path.isfile(local)): tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR')) fetch.unpack(tmpdir, (url,)) # To handle recipes with more than one source, we add the "name"
meta/lib/oeqa/selftest/cases/archiver.py +48 -0
60 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 27 lines · ecordonnier
From: Etienne Cordonnier <[email protected]>
In the original mode,
do_ar_originalarchives a source if the fetcher's localpath is a file, or a directory (which is unpacked and tarred). For git, localpath was the clone in DL_DIR/git2, which a shallow fetch (BB_GIT_SHALLOW) does not create when it uses the gitshallow tarball: since bitbake commit457288b2f("fetch2/git: Add support for fast initial shallow fetch") this is what a shallow fetch does by default, and it also happens when the shallow tarball is fetched from a mirror. The source was then silently skipped and not archived at all.The git fetcher now returns the shallow tarball as localpath in this case. Copying it would make the archive differ depending on how the source was fetched, so unpack git sources like a clone, which uses the shallow tarball.
AI-Generated: Claude Sonnet 5.5
Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes/archiver.bbclass | 9 ++++- meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 2 deletions(-)meta/classes/archiver.bbclass +0
-0
meta/classes/archiver.bbclass python do_ar_original() { · line 208
Quoted 5 lines
meta/classes/archiver.bbclass +2
-0@@ -204,10 +204,15 @@ python do_ar_original() { fetch = bb.fetch.Fetch(urls, ld) tarball_suffix = {} for url in fetch.urls: + ud = fetch.ud[url] + is_git = isinstance(ud.method, bb.fetch.git.Git)
Hi Etienne,
These changes are definitely moving in the right direction, but I don't think they're quite ready to merge yet.
Here we are still special casing particular fetchers, which we shouldn't need to do.
I think we just need two cases:
1) If the path returned by localpath() is a file, copy it.
2) Otherwise, call fetch.unpack() into a tmpdir, regardless of whether the path returned by localpath() exists or not. Raise an error if the tmpdir is still empty after the unpack.
Best regards,
Signature · 1 line
Paul Barker
PATCH v3 2/3 archiver.bbclass: remove .git from original git source archives
In reply to [email protected]From: Etienne Cordonnier <[email protected]>
In the original mode, do_ar_original unpacks git sources and archives the unpacked tree, including its .git. This .git is of no use in the archive:
- For a clone, unpack() uses clone -s, so .git has no objects, only a reference to the clone in DL_DIR via objects/info/alternates. It is confusing and still wastes a little space.
- For a shallow tarball, .git has all the (compressed) objects, which duplicates the sources and doubles the size of the archive.
Remove it, as well as the .git files of submodules. Keep it with nocheckout (which bareclone implies), where the repository is what gets unpacked.
AI-Generated: Claude Opus 5.5
--- meta/classes/archiver.bbclass | 12 +++++++ meta/lib/oeqa/selftest/cases/archiver.py | 43 ++++++++++++++++++++++++ 2 files changed, 55 insertions(+)
meta/classes/archiver.bbclass +12 -0
@@ -215,6 +215,18 @@ python do_ar_original() { elif os.path.isdir(local) or (is_git and os.path.isfile(local)): tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR')) fetch.unpack(tmpdir, (url,)) + if is_git and not ud.nocheckout: + # The .git of a clone has no objects (clone -s only references + # them via objects/info/alternates), and the one of a shallow + # tarball duplicates the sources, so it is of no use in the + # archive. This includes the .git files of submodules. Keep it + # with nocheckout (and bareclone), where it is the source. + for root, dirs, files in os.walk(ud.destdir): + if '.git' in dirs: + dirs.remove('.git') + bb.utils.remove(os.path.join(root, '.git'), recurse=True) + if '.git' in files: + os.remove(os.path.join(root, '.git')) # To handle recipes with more than one source, we add the "name" # URL parameter as suffix. We treat it as an error when # there's more than one URL without a name, or a name gets reused.
meta/lib/oeqa/selftest/cases/archiver.py +43 -0
49 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v3 3/3 archiver.bbclass: fail on sources which cannot be archived
In reply to [email protected]From: Etienne Cordonnier <[email protected]>
In the original mode, do_ar_original archives a source if the fetcher's localpath is a file, or a directory. Any other source is silently skipped, and the source archive is incomplete without any notice. This used to happen for shallow git fetches without a clone, which is fixed by commit "archiver.bbclass: archive shallow git sources which have no clone".
No fetcher has a valid case for a localpath which is neither a file nor a directory after do_fetch, so make it a fatal error.
AI-Generated: Claude Opus 5.5
--- meta/classes/archiver.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/archiver.bbclass +2 -0
@@ -248,6 +248,8 @@ python do_ar_original() { bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (name, tarball_suffix[name], url)) tarball_suffix[name] = url create_tarball(d, tmpdir + '/.', name, ar_outdir) + else: + bb.fatal("Cannot archive %s: its local path %s is neither a file nor a directory" % (url, local)) # Emit patch series files for 'original' bb.note('Writing patch series files...')
Signature · 1 line
2.43.0