Skip to contentSkip to conversation

fetch/git: add reference option

PATCH In review Daniel Walker 2 participants 3 messages Jun 26

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fetch/git: add reference option

Adds a option to the git fetcher which can be used to specify a local reference for a given SRC_URI. Which is then fed into the clone command as "--reference-if-able" which make the option fault tolerant if the directory doesn't exist.

AI-Generated: OpenAI Codex

Change-Id Ib5eb685ddfb48fa19f1c65cfc468894806f99e83
---
 .../bitbake-user-manual-fetching.rst                 |  7 +++++++
 lib/bb/fetch2/git.py                                 | 12 +++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)
doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +7 -0
index b96018a4e..6e4fc0800 100644
@@ -398,6 +398,13 @@ This fetcher supports the following parameters:
    preserve the local cache carefully for future use. The default value
    for this parameter is "0".
 
+-  *"reference":* Specifies a local directory to use as a reference
+   repository. When set, the fetcher passes the directory to ``git clone``
+   with ``--reference-if-able``. Git uses the reference repository's
+   objects when possible, reducing the amount of object data copied or
+   fetched. If the reference directory cannot be used, Git continues
+   cloning without it.
+
 -  *"nobranch":* Tells the fetcher to not check the SHA validation for
    the branch when set to "1". The default is "0". Set this option for
    the recipe that refers to the commit that is valid for any namespace
lib/bb/fetch2/git.py +11 -1
index 0fbd85441..df1c4415f 100644
@@ -22,6 +22,10 @@ Supported SRC_URI options are:
    remind fetcher to preserve local cache carefully for future use.
    The default value is "0", set rebaseable=1 for rebaseable git repo.
 
+- reference
+   Use the specified local directory as a reference repository by passing it
+   to git clone with --reference-if-able.
+
 - nocheckout
    Don't checkout source code when unpacking. set this option for the recipe
    who has its own routine to checkout code.
@@ -195,6 +199,12 @@ class Git(FetchMethod):
         if ud.bareclone:
             ud.cloneflags.append("--mirror")
 
+        ud.reference_args = []
+        ud.reference = ud.parm.get("reference")
+        if ud.reference:
+            ud.reference_args = ["--reference-if-able", ud.reference]
+            ud.cloneflags.extend(ud.reference_args)
+
         ud.shallow_skip_fast = False
         ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1"
         ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split()
@@ -436,7 +446,7 @@ class Git(FetchMethod):
                 objects = os.path.join(repourl_path, 'objects')
                 if os.path.isdir(objects) and not os.path.islink(objects):
                     repourl = repourl_path
-            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror', repourl, ud.clonedir, '--progress']
+            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror'] + ud.reference_args + [repourl, ud.clonedir, '--progress']
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
Signature · 1 line
2.44.0
#1 lore raw
Quoted 5 lines · Daniel Walker via lists.openembedded.org

Adds a option to the git fetcher which can be used to specify a local reference for a given SRC_URI. Which is then fed into the clone command as "--reference-if-able" which make the option fault tolerant if the directory doesn't exist.

We tried this in the past (maybe 10 years ago now) to chunk the kernel into different sets of blobs and do a set of references for optimal sharing, repacking, etc.

In the end, it was discarded as unnecessary complexity.

I'm still of the opinion that is the case. What problem is this solving that justifies the complexity ?

the patch doesn't show a real world use case, what the SRC_URI would look like, where the referenced clone would be, what would happen if the referenced repository was somehow cleaned, or any number of different interactions (what happens if shallow is specified, etc, etc).

Bruce

Quoted 85 lines

AI-Generated: OpenAI Codex

Change-Id Ib5eb685ddfb48fa19f1c65cfc468894806f99e83
---
 .../bitbake-user-manual-fetching.rst                 |  7 +++++++
 lib/bb/fetch2/git.py                                 | 12 +++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)
doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +7 -0
index b96018a4e..6e4fc0800 100644
@@ -398,6 +398,13 @@ This fetcher supports the following parameters:
    preserve the local cache carefully for future use. The default value
    for this parameter is "0".
 
+-  *"reference":* Specifies a local directory to use as a reference
+   repository. When set, the fetcher passes the directory to ``git clone``
+   with ``--reference-if-able``. Git uses the reference repository's
+   objects when possible, reducing the amount of object data copied or
+   fetched. If the reference directory cannot be used, Git continues
+   cloning without it.
+
 -  *"nobranch":* Tells the fetcher to not check the SHA validation for
    the branch when set to "1". The default is "0". Set this option for
    the recipe that refers to the commit that is valid for any namespace
lib/bb/fetch2/git.py +2 -0
index 0fbd85441..df1c4415f 100644
@@ -22,6 +22,10 @@ Supported SRC_URI options are:
    remind fetcher to preserve local cache carefully for future use.
    The default value is "0", set rebaseable=1 for rebaseable git repo.
 
+- reference
+   Use the specified local directory as a reference repository by passing

it

lib/bb/fetch2/git.py +2 -0
+   to git clone with --reference-if-able.
+
 - nocheckout
    Don't checkout source code when unpacking. set this option for the

recipe
who has its own routine to checkout code.

lib/bb/fetch2/git.py +6 -0
@@ -195,6 +199,12 @@ class Git(FetchMethod):
         if ud.bareclone:
             ud.cloneflags.append("--mirror")
 
+        ud.reference_args = []
+        ud.reference = ud.parm.get("reference")
+        if ud.reference:
+            ud.reference_args = ["--reference-if-able", ud.reference]
+            ud.cloneflags.extend(ud.reference_args)
+
         ud.shallow_skip_fast = False
         ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1"
         ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or

"").split()

lib/bb/fetch2/git.py +0 -1
@@ -436,7 +446,7 @@ class Git(FetchMethod):
                 objects = os.path.join(repourl_path, 'objects')
                 if os.path.isdir(objects) and not os.path.islink(objects):
                     repourl = repourl_path
-            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror',

repourl, ud.clonedir, '--progress']
+ clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror'] +
ud.reference_args + [repourl, ud.clonedir, '--progress']
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd, ud.url)
progresshandler = GitProgressHandler(d)
--
2.44.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#19794):
https://lists.openembedded.org/g/bitbake-devel/message/19794
Mute This Topic: https://lists.openembedded.org/mt/119994628/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#2 lore raw
Quoted 15 lines · Bruce Ashfield
Quoted 5 lines · Daniel Walker via lists.openembedded.org

Adds a option to the git fetcher which can be used to specify a local reference for a given SRC_URI. Which is then fed into the clone command as "--reference-if-able" which make the option fault tolerant if the directory doesn't exist.

We tried this in the past (maybe 10 years ago now) to chunk the kernel into different sets of blobs and do a set of references for optimal sharing, repacking, etc.

In the end, it was discarded as unnecessary complexity.

Quoted 9 lines

I'm still of the opinion that is the case. What problem is this solving that justifies the complexity ?

the patch doesn't show a real world use case, what the SRC_URI would look like, where the referenced clone would be, what would happen if the referenced repository was somehow cleaned, or any number of different interactions (what happens if shallow is specified, etc, etc).

At Cisco we have settings which modify the SRC_URI to become the local repo.

For example,

KERNEL_REPO:pn-linux-cisco = "git:///local/git/cache/;protocol=file"

This works to use a local cache of the repo, but it messes up the state cache so objects can't be used by others.

If we use "--reference-if-able" then the git host and location remains the same, but the local copy can speed up the clone.

For down stream users, like Cisco, who work in a complex environment it's helpful to have flexibility.

The reference is not harmful that I know of, it's only used if it can be used.

Daniel