Skip to contentSkip to conversation

buildman: Use git worktrees instead of git clones when possible

PATCH New Alper Nebi Yasak 2 participants 4 messages Sep 3, 2020 – Sep 5, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones of the source repository, but keeps updating the older clones of the repository that might already exist. These worktrees share "everything except working directory specific files such as HEAD, index, etc." with the source repository. See the git-worktree(1) manual page for more information.

If git-worktree isn't available, silently falls back to cloning the repository.

---
I used 'git worktree list's return code, since 'git worktree' returns non-zero (prints usage) even if it's available.

This does fall back to git clone on my Ubuntu 14.04 amd64 VM, but all the builds instantly give an error about the Makefile even before this patch. The buildman parts are working as far as I can tell.

Changes in v2:
- Fall back to cloning if git-worktree isn't available
- Add a gitutil.CheckWorktreeIsAvailable(git_dir) function
- Refactor the _PrepareThread changes
- Make _PrepeareThread's setup_git argument accept 'clone' or 'worktree'
- Some comment and docstring changes

v1: https://patchwork.ozlabs.org/project/uboot/list/?series=199060

 tools/buildman/builder.py   | 48 ++++++++++++++++++++++++++++++-------
 tools/buildman/func_test.py |  2 ++
 tools/patman/gitutil.py     | 42 ++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 8 deletions(-)
tools/buildman/builder.py +40 -8
index dbb75b35c1..c93946842a 100644
@@ -1541,41 +1541,73 @@ class Builder:
         """Prepare the working directory for a thread.
 
         This clones or fetches the repo into the thread's work directory.
+        Optionally, it can create a linked working tree of the repo in the
+        thread's work directory instead.
 
         Args:
             thread_num: Thread number (0, 1, ...)
-            setup_git: True to set up a git repo clone
+            setup_git:
+               'clone' to set up a git clone
+               'worktree' to set up a git worktree
         """
         thread_dir = self.GetThreadDir(thread_num)
         builderthread.Mkdir(thread_dir)
         git_dir = os.path.join(thread_dir, '.git')
 
-        # Clone the repo if it doesn't already exist
-        # TODO(sjg at chromium): Perhaps some git hackery to symlink instead, so
-        # we have a private index but uses the origin repo's contents?
+        # Create a worktree or a git repo clone for this thread if it
+        # doesn't already exist
         if setup_git and self.git_dir:
             src_dir = os.path.abspath(self.git_dir)
-            if os.path.exists(git_dir):
+            if os.path.isdir(git_dir):
+                # This is a clone of the src_dir repo, we can keep using
+                # it but need to fetch from src_dir.
                 Print('\rFetching repo for thread %d' % thread_num,
                       newline=False)
                 gitutil.Fetch(git_dir, thread_dir)
                 terminal.PrintClear()
-            else:
+            elif os.path.isfile(git_dir):
+                # This is a worktree of the src_dir repo, we don't need to
+                # create it again or update it in any way.
+                pass
+            elif os.path.exists(git_dir):
+                # Don't know what could trigger this, but we probably
+                # can't create a git worktree/clone here.
+                raise ValueError('Git dir %s exists, but is not a file '
+                                 'or a directory.' % git_dir)
+            elif setup_git == 'worktree':
+                Print('\rChecking out worktree for thread %d' % thread_num,
+                      newline=False)
+                gitutil.AddWorktree(src_dir, thread_dir)
+                terminal.PrintClear()
+            elif setup_git == 'clone' or setup_git == True:
                 Print('\rCloning repo for thread %d' % thread_num,
                       newline=False)
                 gitutil.Clone(src_dir, thread_dir)
                 terminal.PrintClear()
+            else:
+                raise ValueError("Can't setup git repo with %s." % setup_git)
 
     def _PrepareWorkingSpace(self, max_threads, setup_git):
         """Prepare the working directory for use.
 
-        Set up the git repo for each thread.
+        Set up the git repo for each thread. Creates a linked working tree
+        if git-worktree is available, or clones the repo if it isn't.
 
         Args:
             max_threads: Maximum number of threads we expect to need.
-            setup_git: True to set up a git repo clone
+            setup_git: True to set up a git worktree or a git clone
         """
         builderthread.Mkdir(self._working_dir)
+        if setup_git and self.git_dir:
+            src_dir = os.path.abspath(self.git_dir)
+            if gitutil.CheckWorktreeIsAvailable(src_dir):
+                setup_git = 'worktree'
+                # If we previously added a worktree but the directory for it
+                # got deleted, we need to prune its files from the repo so
+                # that we can check out another in its place.
+                gitutil.PruneWorktrees(src_dir)
+            else:
+                setup_git = 'clone'
         for thread in range(max_threads):
             self._PrepareThread(thread, setup_git)
 
tools/buildman/func_test.py +2 -0
index 418677f9cc..3dd2e6ee5b 100644
@@ -319,6 +319,8 @@ class TestFunctional(unittest.TestCase):
             return command.CommandResult(return_code=0)
         elif sub_cmd == 'checkout':
             return command.CommandResult(return_code=0)
+        elif sub_cmd == 'worktree':
+            return command.CommandResult(return_code=0)
 
         # Not handled, so abort
         print('git', git_args, sub_cmd, args)
tools/patman/gitutil.py +42 -0

48 lines not on this page · raw message

Signature · 1 line
2.28.0
#1 lore raw

PATCH v2 1/1 buildman: Use git worktrees instead of git clones when possible

Quoted 27 lines · Alper Nebi Yasak

This patch makes buildman create linked working trees instead of clones of the source repository, but keeps updating the older clones of the repository that might already exist. These worktrees share "everything except working directory specific files such as HEAD, index, etc." with the source repository. See the git-worktree(1) manual page for more information.

If git-worktree isn't available, silently falls back to cloning the repository.

---
I used 'git worktree list's return code, since 'git worktree' returns non-zero (prints usage) even if it's available.

This does fall back to git clone on my Ubuntu 14.04 amd64 VM, but all the builds instantly give an error about the Makefile even before this patch. The buildman parts are working as far as I can tell.

Changes in v2:
- Fall back to cloning if git-worktree isn't available
- Add a gitutil.CheckWorktreeIsAvailable(git_dir) function
- Refactor the _PrepareThread changes
- Make _PrepeareThread's setup_git argument accept 'clone' or 'worktree'
- Some comment and docstring changes

Quoted 6 lines

v1: https://patchwork.ozlabs.org/project/uboot/list/?series=199060

 tools/buildman/builder.py   | 48 ++++++++++++++++++++++++++++++-------
 tools/buildman/func_test.py |  2 ++
 tools/patman/gitutil.py     | 42 ++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 8 deletions(-)
#2 lore raw

PATCH v2 1/1 buildman: Use git worktrees instead of git clones when possible

Quoted 29 lines · Simon Glass
Quoted 27 lines · Alper Nebi Yasak

This patch makes buildman create linked working trees instead of clones of the source repository, but keeps updating the older clones of the repository that might already exist. These worktrees share "everything except working directory specific files such as HEAD, index, etc." with the source repository. See the git-worktree(1) manual page for more information.

If git-worktree isn't available, silently falls back to cloning the repository.

---
I used 'git worktree list's return code, since 'git worktree' returns non-zero (prints usage) even if it's available.

This does fall back to git clone on my Ubuntu 14.04 amd64 VM, but all the builds instantly give an error about the Makefile even before this patch. The buildman parts are working as far as I can tell.

Changes in v2:
- Fall back to cloning if git-worktree isn't available
- Add a gitutil.CheckWorktreeIsAvailable(git_dir) function
- Refactor the _PrepareThread changes
- Make _PrepeareThread's setup_git argument accept 'clone' or 'worktree'
- Some comment and docstring changes

Quoted 8 lines
Quoted 6 lines

v1: https://patchwork.ozlabs.org/project/uboot/list/?series=199060

 tools/buildman/builder.py   | 48 ++++++++++++++++++++++++++++++-------
 tools/buildman/func_test.py |  2 ++
 tools/patman/gitutil.py     | 42 ++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 8 deletions(-)
#3 lore raw

PATCH v2 1/1 buildman: Use git worktrees instead of git clones when possible

Quoted 29 lines · Simon Glass
Quoted 27 lines · Alper Nebi Yasak

This patch makes buildman create linked working trees instead of clones of the source repository, but keeps updating the older clones of the repository that might already exist. These worktrees share "everything except working directory specific files such as HEAD, index, etc." with the source repository. See the git-worktree(1) manual page for more information.

If git-worktree isn't available, silently falls back to cloning the repository.

---
I used 'git worktree list's return code, since 'git worktree' returns non-zero (prints usage) even if it's available.

This does fall back to git clone on my Ubuntu 14.04 amd64 VM, but all the builds instantly give an error about the Makefile even before this patch. The buildman parts are working as far as I can tell.

Changes in v2:
- Fall back to cloning if git-worktree isn't available
- Add a gitutil.CheckWorktreeIsAvailable(git_dir) function
- Refactor the _PrepareThread changes
- Make _PrepeareThread's setup_git argument accept 'clone' or 'worktree'
- Some comment and docstring changes

Quoted 8 lines
Quoted 6 lines

v1: https://patchwork.ozlabs.org/project/uboot/list/?series=199060

 tools/buildman/builder.py   | 48 ++++++++++++++++++++++++++++++-------
 tools/buildman/func_test.py |  2 ++
 tools/patman/gitutil.py     | 42 ++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 8 deletions(-)

Applied to u-boot-dm/next, thanks!