Skip to contentSkip to conversation

buildman: Use git worktrees instead of git clones

PATCH New Alper Nebi Yasak 2 participants 4 messages Sep 2, 2020 – Sep 3, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 buildman: Use git worktrees instead of git clones

This patch makes buildman create linked working trees instead of clones of the source repository, but keeps updating the older clones of the repository that might already exist. These worktrees share "everything except working directory specific files such as HEAD, index, etc." with the source repository. See the git-worktree(1) manual page for more information.

---
The git-worktree is available since about 2015 (git v2.5), but its manual page mentions this in its BUGS section:

Multiple checkout in general is still experimental, and the support for submodules is incomplete. It is NOT recommended to make multiple checkouts of a superproject.

I think it'll be fine since U-Boot doesn't use submodules, but still wanted to let you know.

 tools/buildman/builder.py   | 36 ++++++++++++++++++++++++------------
 tools/buildman/func_test.py |  2 ++
 tools/patman/gitutil.py     | 28 ++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 12 deletions(-)
tools/buildman/builder.py +24 -12
index dbb75b35c1..9cb65c15a8 100644
@@ -1540,31 +1540,38 @@ class Builder:
     def _PrepareThread(self, thread_num, setup_git):
         """Prepare the working directory for a thread.
 
-        This clones or fetches the repo into the thread's work directory.
+        This adds a git worktree for the repo into the thread's work
+        directory or fetches the repo into a clone that might already
+        exist there.
 
         Args:
             thread_num: Thread number (0, 1, ...)
-            setup_git: True to set up a git repo clone
+            setup_git: True to set up a git worktree
         """
         thread_dir = self.GetThreadDir(thread_num)
         builderthread.Mkdir(thread_dir)
         git_dir = os.path.join(thread_dir, '.git')
 
-        # Clone the repo if it doesn't already exist
-        # TODO(sjg at chromium): Perhaps some git hackery to symlink instead, so
-        # we have a private index but uses the origin repo's contents?
+        # Create a worktree for this thread if it (or a git repo clone)
+        # doesn't already exist
         if setup_git and self.git_dir:
             src_dir = os.path.abspath(self.git_dir)
-            if os.path.exists(git_dir):
+            if not os.path.exists(git_dir):
+                Print('\rChecking out worktree for thread %d' % thread_num,
+                      newline=False)
+                gitutil.AddWorktree(src_dir, thread_dir)
+                terminal.PrintClear()
+            elif os.path.isdir(git_dir):
+                # Older versions cloned the src_dir repo, we can keep using
+                # the clones but need to fetch from src_dir.
                 Print('\rFetching repo for thread %d' % thread_num,
                       newline=False)
                 gitutil.Fetch(git_dir, thread_dir)
                 terminal.PrintClear()
-            else:
-                Print('\rCloning repo for thread %d' % thread_num,
-                      newline=False)
-                gitutil.Clone(src_dir, thread_dir)
-                terminal.PrintClear()
+            elif os.path.isfile(git_dir):
+                # This is a worktree of the src_dir repo, we don't need to
+                # create it again.
+                pass
 
     def _PrepareWorkingSpace(self, max_threads, setup_git):
         """Prepare the working directory for use.
@@ -1573,9 +1580,14 @@ class Builder:
 
         Args:
             max_threads: Maximum number of threads we expect to need.
-            setup_git: True to set up a git repo clone
+            setup_git: True to set up a git worktree
         """
         builderthread.Mkdir(self._working_dir)
+        if setup_git and self.git_dir:
+            # If we previously added a worktree but the directory for it
+            # got deleted, we need to prune its files from the repo so
+            # that we can check out another in its place.
+            gitutil.PruneWorktrees(os.path.abspath(self.git_dir))
         for thread in range(max_threads):
             self._PrepareThread(thread, setup_git)
 
tools/buildman/func_test.py +2 -0
index 418677f9cc..3dd2e6ee5b 100644
@@ -319,6 +319,8 @@ class TestFunctional(unittest.TestCase):
             return command.CommandResult(return_code=0)
         elif sub_cmd == 'checkout':
             return command.CommandResult(return_code=0)
+        elif sub_cmd == 'worktree':
+            return command.CommandResult(return_code=0)
 
         # Not handled, so abort
         print('git', git_args, sub_cmd, args)
tools/patman/gitutil.py +28 -0
index 192d8e69b3..354884a79d 100644
@@ -259,6 +259,34 @@ def Fetch(git_dir=None, work_tree=None):
     if result.return_code != 0:
         raise OSError('git fetch: %s' % result.stderr)
 
+def AddWorktree(git_dir, output_dir, commit_hash=None):
+    """Create and checkout a new git worktree for this build
+
+    Args:
+        git_dir: The repository to checkout the worktree from
+        output_dir: Path for the new worktree
+        commit_hash: Commit hash to checkout
+    """
+    # We need to pass --detach to avoid creating a new branch
+    pipe = ['git', '--git-dir', git_dir, 'worktree', 'add', '.', '--detach']
+    if commit_hash:
+        pipe.append(commit_hash)
+    result = command.RunPipe([pipe], capture=True, cwd=output_dir,
+                             capture_stderr=True)
+    if result.return_code != 0:
+        raise OSError('git worktree add: %s' % result.stderr)
+
+def PruneWorktrees(git_dir):
+    """Remove administrative files for deleted worktrees
+
+    Args:
+        git_dir: The repository whose deleted worktrees should be pruned
+    """
+    pipe = ['git', '--git-dir', git_dir, 'worktree', 'prune']
+    result = command.RunPipe([pipe], capture=True, capture_stderr=True)
+    if result.return_code != 0:
+        raise OSError('git worktree prune: %s' % result.stderr)
+
 def CreatePatches(branch, start, count, ignore_binary, series):
     """Create a series of patches from the top of the current branch.
 
Signature · 1 line
2.28.0
#1 lore raw

PATCH 1/1 buildman: Use git worktrees instead of git clones

Hi Alper,

Quoted 13 lines · Alper Nebi Yasak

This patch makes buildman create linked working trees instead of clones of the source repository, but keeps updating the older clones of the repository that might already exist. These worktrees share "everything except working directory specific files such as HEAD, index, etc." with the source repository. See the git-worktree(1) manual page for more information.

---
The git-worktree is available since about 2015 (git v2.5), but its manual page mentions this in its BUGS section:

Quoted 6 lines

Multiple checkout in general is still experimental, and the support for submodules is incomplete. It is NOT recommended to make multiple checkouts of a superproject.

I think it'll be fine since U-Boot doesn't use submodules, but still wanted to let you know.

I'd really like to do this as it cuts the disk space alot, particularly with a 30-core machine. I tried to do something similar with symlinks when I first wrote buildman, but it did not seem to be possible without copying files and doing things under the hood that I didn't like.

It looks like that version of git was not in Ubuntu 14.04 but appeared in 16.04. I've just done some patches to make patman work in 14.04, so I'd prefer to keep buildman working for that also.

So I think we should have an option to keep the old behaviour, or perhaps detect when 'git worktree' is not available and fall back.

Quoted 6 lines
 tools/buildman/builder.py   | 36 ++++++++++++++++++++++++------------
 tools/buildman/func_test.py |  2 ++
 tools/patman/gitutil.py     | 28 ++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 12 deletions(-)

Regards,
Simon

#2 lore raw

PATCH 1/1 buildman: Use git worktrees instead of git clones

Quoted 3 lines · Simon Glass

It looks like that version of git was not in Ubuntu 14.04 but appeared in 16.04. I've just done some patches to make patman work in 14.04, so I'd prefer to keep buildman working for that also.

I'll keep that in mind. Looks like Ubuntu 14.04 has python v3.4 and git v1.9. I'm used to more recent versions so maybe I'll take the time to set up a VM to test things in.

Quoted 2 lines

So I think we should have an option to keep the old behaviour, or perhaps detect when 'git worktree' is not available and fall back.

I can check whether 'git worktree list' succeeds, it should have no effect on the repo but would give a 'not a git command' error if things wouldn't work; in which case buildman could fall back to cloning. I'll implement and send a v2 with something like that.

#3 lore raw

PATCH 1/1 buildman: Use git worktrees instead of git clones

Hi Alper,

Quoted 10 lines · Alper Nebi Yasak
Quoted 3 lines · Simon Glass

It looks like that version of git was not in Ubuntu 14.04 but appeared in 16.04. I've just done some patches to make patman work in 14.04, so I'd prefer to keep buildman working for that also.

I'll keep that in mind. Looks like Ubuntu 14.04 has python v3.4 and git v1.9. I'm used to more recent versions so maybe I'll take the time to set up a VM to test things in.

Quoted 7 lines
Quoted 2 lines

So I think we should have an option to keep the old behaviour, or perhaps detect when 'git worktree' is not available and fall back.

I can check whether 'git worktree list' succeeds, it should have no effect on the repo but would give a 'not a git command' error if things wouldn't work; in which case buildman could fall back to cloning. I'll implement and send a v2 with something like that.

OK thanks. If it helps, on 14.04:

$ git worktree
git: 'worktree' is not a git command. See 'git --help'.
echo $?
1

Regards,
Simon