Implement support for shallow git mirror tarballs

From: Christopher Larson <[email protected]>

This adds support to the git fetcher for fetching and generating mirror tarballs of shallow git repositories, with the specified revisions / depths.

This implements support for shallow mirror tarballs, not shallow clones. Supporting shallow clones directly would be rather more problematic, as we'd need to hardcode the depth between branch HEAD and the SRCREV, and that depth would change as the branch is updated. Sadly, git's remote protocol is extremely limited.

When BB_GIT_SHALLOW is enabled, we will always attempt to fetch a shallow mirror tarball. If the shallow mirror tarball cannot be fetched, it will try to fetch the full mirror tarball and use that.

Multiple variables exist to exert control over what revisions and refs are kept at what are not.

BB_GIT_SHALLOW: enable/disable shallow support as a whole, boolean
BB_GIT_SHALLOW_DEPTH: specify commit depth for the included refs. defaults to 1 (the top commit alone). This can be set to 0 or the empty string.
BB_GIT_SHALLOW_DEPTH_<name>: specify commit depth for a specific named url/branch
BB_GIT_SHALLOW_REVS: specific revisions whose history should be removed, beyond that specified by refs+depth
BB_GIT_SHALLOW_TRIM_REFS: by default, unused refs/branches are removed. set this to 0 or the empty string to keep all refs around. this is required for linux-yocto kernel repositories, otherwise the branch checking done by the kernel scripts will fail
BB_GIT_SHALLOW_EXTRA_REFS: additional refs to keep beyond those referenced in SRC_URI, when TRIM_REFS is enabled
BB_GENERATE_SHALLOW_TARBALLS: explicitly control creation of shallow mirror tarballs. this defaults to enabled when BB_GENERATE_MIRROR_TARBALLS is enabled

Example usage:

BB_GIT_SHALLOW ?= "1"
BB_GIT_SHALLOW_DEPTH ?= "1"

# Usage in a recipe with multiple named uris or multiple named branches BB_GIT_SHALLOW_DEPTH_doc = ""

# Remove the upstream history from our kernel repo, keeping our own BB_GIT_SHALLOW_REVS_pn-linux-mel_mx6 = "v3.14" BB_GIT_SHALLOW_DEPTH_pn-linux-mel_mx6 = ""

# Keep the branches in linux-yocto repositories BB_GIT_SHALLOW_TRIM_REFS_pn-linux-yocto = "0"

Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.

The following changes since commit 309f5907a3661821e041ed14645b5d165007b058:

bitbake: Switch to post release version (2016-04-29 07:41:34 +0100)

are available in the git repository at:

https://github.com/kergoth/bitbake shallow-simplify
https://github.com/kergoth/bitbake/tree/shallow-simplify

Christopher Larson (3):
bb.fetch: simplify mirror tarball handling in try_mirror_url bb.fetch: support ud.mirrortarballs
bb.fetch.git: add support for shallow mirror tarballs

 lib/bb/fetch2/__init__.py |  27 ++---
 lib/bb/fetch2/git.py      | 260 ++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 255 insertions(+), 32 deletions(-)

2.8.0

This branch has been updated after rebasing post-python3-switch, so should be ready for potential merge again. A couple things worth noting that I failed to mention in the original post:

The shallow mirror tarballs are never unpacked into DL_DIR -- I didn't want to create another dir structure there when there wouldn't be much benefit, and I definitely didn't want anything shallow inside of ${DL_DIR}/git2/ to avoid confusion and state problems, so when a shallow tarball is in play, the download() method will return early and do nothing beyond the fetch of the shallow tarball, and then the unpack method will extract it directly.

It's also worth noting that a current clone is preferred to either tarball, a shallow tarball is preferred to an out of date clone, and a missing clone will use either tarball.

I'd also like to clarify the usage of BB_GIT_SHALLOW_REVS. In a normal use of shallow, the history removed is that beyond a certain depth of a certain branch, so you end up with N commits left over, and everything else removed. If BB_GIT_SHALLOW_DEPTH isn't set and BB_GIT_SHALLOW_REVS is set, instead of only keeping a certain depth of history, we're only *removing* a particular subset of the history. The main use case is to get more control over what history is available to a customer when shipping sources, so e.g. you can strip out the bits available in public repositories and just keep the bits they can't easily re-fetch.

Richard, think you'll be able to find time to take a look at this? As long as BB_GIT_SHALLOW isn't set, there's almost no impact at all. The only change that isn't conditional upon this is the change in logic in bb.fetch to simplify the tarball handling, so the risk is actually quite low.

Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics