fetch/git: add reference option
PATCH 1/1 fetch/git: add reference option
Adds a option to the git fetcher which can be used to specify a local reference for a given SRC_URI. Which is then fed into the clone command as "--reference-if-able" which make the option fault tolerant if the directory doesn't exist.
AI-Generated: OpenAI Codex
--- .../bitbake-user-manual-fetching.rst | 7 +++++++ lib/bb/fetch2/git.py | 12 +++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +7 -0
@@ -398,6 +398,13 @@ This fetcher supports the following parameters: preserve the local cache carefully for future use. The default value for this parameter is "0". +- *"reference":* Specifies a local directory to use as a reference + repository. When set, the fetcher passes the directory to ``git clone`` + with ``--reference-if-able``. Git uses the reference repository's + objects when possible, reducing the amount of object data copied or + fetched. If the reference directory cannot be used, Git continues + cloning without it. + - *"nobranch":* Tells the fetcher to not check the SHA validation for the branch when set to "1". The default is "0". Set this option for the recipe that refers to the commit that is valid for any namespace
lib/bb/fetch2/git.py +11 -1
@@ -22,6 +22,10 @@ Supported SRC_URI options are: remind fetcher to preserve local cache carefully for future use. The default value is "0", set rebaseable=1 for rebaseable git repo. +- reference + Use the specified local directory as a reference repository by passing it + to git clone with --reference-if-able. + - nocheckout Don't checkout source code when unpacking. set this option for the recipe who has its own routine to checkout code.
@@ -195,6 +199,12 @@ class Git(FetchMethod): if ud.bareclone: ud.cloneflags.append("--mirror") + ud.reference_args = [] + ud.reference = ud.parm.get("reference") + if ud.reference: + ud.reference_args = ["--reference-if-able", ud.reference] + ud.cloneflags.extend(ud.reference_args) + ud.shallow_skip_fast = False ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1" ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split()
@@ -436,7 +446,7 @@ class Git(FetchMethod): objects = os.path.join(repourl_path, 'objects') if os.path.isdir(objects) and not os.path.islink(objects): repourl = repourl_path - clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror', repourl, ud.clonedir, '--progress'] + clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror'] + ud.reference_args + [repourl, ud.clonedir, '--progress'] if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler = GitProgressHandler(d)
Signature · 1 line
2.44.0
Quoted 5 lines · Daniel Walker via lists.openembedded.org
Adds a option to the git fetcher which can be used to specify a local reference for a given
SRC_URI. Which is then fed into the clone command as "--reference-if-able" which make the option fault tolerant if the directory doesn't exist.
We tried this in the past (maybe 10 years ago now) to chunk the kernel into different sets of blobs and do a set of references for optimal sharing, repacking, etc.
In the end, it was discarded as unnecessary complexity.
I'm still of the opinion that is the case. What problem is this solving that justifies the complexity ?
the patch doesn't show a real world use case, what the SRC_URI would look like, where the referenced clone would be, what would happen if the referenced repository was somehow cleaned, or any number of different interactions (what happens if shallow is specified, etc, etc).
Bruce
Quoted 85 lines
AI-Generated: OpenAI Codex
Signed-off-by Daniel Walker <[email protected]>Change-Id Ib5eb685ddfb48fa19f1c65cfc468894806f99e83--- .../bitbake-user-manual-fetching.rst | 7 +++++++ lib/bb/fetch2/git.py | 12 +++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +7
-0@@ -398,6 +398,13 @@ This fetcher supports the following parameters: preserve the local cache carefully for future use. The default value for this parameter is "0". +- *"reference":* Specifies a local directory to use as a reference + repository. When set, the fetcher passes the directory to ``git clone`` + with ``--reference-if-able``. Git uses the reference repository's + objects when possible, reducing the amount of object data copied or + fetched. If the reference directory cannot be used, Git continues + cloning without it. + - *"nobranch":* Tells the fetcher to not check the SHA validation for the branch when set to "1". The default is "0". Set this option for the recipe that refers to the commit that is valid for any namespacelib/bb/fetch2/git.py +2
-0@@ -22,6 +22,10 @@ Supported SRC_URI options are: remind fetcher to preserve local cache carefully for future use. The default value is "0", set rebaseable=1 for rebaseable git repo. +- reference + Use the specified local directory as a reference repository by passingit
lib/bb/fetch2/git.py +2
-0+ to git clone with --reference-if-able. + - nocheckout Don't checkout source code when unpacking. set this option for therecipe
who has its own routine to checkout code.lib/bb/fetch2/git.py +6
-0@@ -195,6 +199,12 @@ class Git(FetchMethod): if ud.bareclone: ud.cloneflags.append("--mirror") + ud.reference_args = [] + ud.reference = ud.parm.get("reference") + if ud.reference: + ud.reference_args = ["--reference-if-able", ud.reference] + ud.cloneflags.extend(ud.reference_args) + ud.shallow_skip_fast = False ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1" ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or"").split()
lib/bb/fetch2/git.py +0
-1@@ -436,7 +446,7 @@ class Git(FetchMethod): objects = os.path.join(repourl_path, 'objects') if os.path.isdir(objects) and not os.path.islink(objects): repourl = repourl_path - clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror',repourl, ud.clonedir, '--progress']
+clone_cmd= ud.basecmd + ['clone', '--bare', '--mirror'] +
ud.reference_args + [repourl, ud.clonedir, '--progress']
ifud.proto.lower()!= 'file':
bb.fetch2.check_network_access(d, clone_cmd, ud.url)
progresshandler = GitProgressHandler(d)
--
2.44.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#19794):
https://lists.openembedded.org/g/bitbake-devel/message/19794
Mute This Topic: https://lists.openembedded.org/mt/119994628/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 15 lines · Bruce Ashfield
Quoted 5 lines · Daniel Walker via lists.openembedded.org
Adds a option to the git fetcher which can be used to specify a local reference for a given
SRC_URI. Which is then fed into the clone command as "--reference-if-able" which make the option fault tolerant if the directory doesn't exist.We tried this in the past (maybe 10 years ago now) to chunk the kernel into different sets of blobs and do a set of references for optimal sharing, repacking, etc.
In the end, it was discarded as unnecessary complexity.
Quoted 9 lines
I'm still of the opinion that is the case. What problem is this solving that justifies the complexity ?
the patch doesn't show a real world use case, what the
SRC_URIwould look like, where the referenced clone would be, what would happen if the referenced repository was somehow cleaned, or any number of different interactions (what happens if shallow is specified, etc, etc).
At Cisco we have settings which modify the SRC_URI to become the local repo.
For example,
KERNEL_REPO:pn-linux-cisco = "git:///local/git/cache/;protocol=file"
This works to use a local cache of the repo, but it messes up the state cache so objects can't be used by others.
If we use "--reference-if-able" then the git host and location remains the same, but the local copy can speed up the clone.
For down stream users, like Cisco, who work in a complex environment it's helpful to have flexibility.
The reference is not harmful that I know of, it's only used if it can be used.
Daniel