Skip to content

grep: re-enable threads when cached, w/ parallel inflation

Matheus Tavares git 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4object-store: add lock to read_object_file_extended()object-store.h packfile.c sha1-file.c+67 -50 replies
2/4grep: allow locks to be enabled individuallybuiltin/grep.c grep.c grep.h+9 -50 replies
3/4grep: disable grep_read_mutex when possiblebuiltin/grep.c+16 -30 replies
4/4grep: re-enable threads in some non-worktree casesDocumentation/git-grep.txt builtin/grep.c+14 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [GSoC][PATCH 0/4] grep: re-enable threads when cached, w/ parallel inflation

This series focuses on allowing parallel access to zlib inflation and using that to perform a faster git-grep in the non-worktree case.

Threads were disabled for this case at 53b8d93 ("grep: disable threading in non-worktree case", 12-12-2011), due to performance drops.

However, by allowing threads to perform inflation in parallel, we can regain the speedup. This is a good hotspot for parallelism as some test cases[1] showed that it can account for up to 48% of execution time. And besides that, inflation tasks are already independent of each other.

As a result, grepping 'abcd[02]' ("Regex 1") and '(static|extern) (int|double) \*' ("Regex 2") at chromium's repository[2], I got (means of 30 executions):

     Threads |   Regex 1  |  Regex 2
    ---------|------------|-----------
        1    |  17.3557s  | 20.8410s
        2    |   9.7170s  | 11.2415s
        8    |   6.1723s  |  6.9378s

As a reference, just enabling threads in the non-worktree case, without parallel inflation, I got:

     Threads |   Regex 1  |  Regex 2
    ---------|------------|-----------
        1    |  17.1359s  | 20.8306s
        2    |  14.5036s  | 15.4172s
        8    |  13.6304s  | 13.8659s

For now, the optimization is not supported when --textconv or --recurse-submodules are used, but I hope to send another patchset for that still during GSoC. We may also try to allow even more parallelism, refining the added 'obj_read_lock'.

[1]: https://matheustavares.gitlab.io/posts/week-6-working-at-zlib-inflation#multithreading-zlib-inflation
[2]: chromium’s repo at commit 03ae96f (“Add filters testing at DSF=2”,

     04-06-2019), after a 'git gc' execution.

travis build: https://travis-ci.org/matheustavares/git/builds/570255029

Matheus Tavares (4):
object-store: add lock to read_object_file_extended()
grep: allow locks to be enabled individually
grep: disable grep_read_mutex when possible
grep: re-enable threads in some non-worktree cases

 Documentation/git-grep.txt | 12 ++++++++
 builtin/grep.c             | 22 +++++++++++---
 grep.c                     |  4 +--
 grep.h                     |  8 +++--
 object-store.h             |  4 +++
 packfile.c                 |  7 +++++
 sha1-file.c                | 61 ++++++++++++++++++++++++++++++++++----
 7 files changed, 105 insertions(+), 13 deletions(-)
Signature · 1 line
2.22.0