rev-cache

SUGGESTED FOR 'PU':

Traversing objects is currently very costly, as every commit and tree must be loaded and parsed. Much time and energy could be saved by caching metadata and topological info in an efficient, easily accessible manner. Furthermore, this could improve git's interfacing potential, by providing a condensed summary of a repository's commit tree.

This is a series to implement such a revision caching mechanism, aptly named rev-cache. The series will provide:
- a core API to manipulate and traverse caches
- an integration into the internal revision walker
- a porcelain front-end providing access to users and (shell) applications
- a series of tests to verify/demonstrate correctness
- documentation of the API, porcelain and core concepts

In cold starts rev-cache has sped up packing and walking by a factor of 4, and over twice that on warm starts. The mechanism is minimally intrusive: most of the changes take place in seperate files, and only a handful of git's existing functions are modified.

Slides from the presentation I gave at GitTogether'09 give a good overview of the mechanism, usage and performance of rev-cache. (well, they complement the documentation well. they're a bit bare without the dialogue...) You can find them at http://www.flickr.com/photos/sirnot/sets/72157623652754819/

The patchset is structured so that each patchfile represents a fully functional, incremental development of rev-cache. Unfortunately the addition of several files makes for some obscenely large patchfiles. I apologize and regret that this is somewhat unavoidable; my only consolation is that patchfiles are less 'patches' and more additions of files.

Hope you find this useful.

- Nick

---
Hi everyone. I'm sorry about the (very) belated patchset -- University has a tendency of being very absorbing. It's spring break now though, so I've finally got around to finalizing rev-cache! The only significant change from last time (besides the bugfixes) is how structures are read/written -- excessive care is now taken to ensure a standardized file format.

This time round I've also got some swanky slides I made for GitTogether. Hehe.

Documentation/git-rev-cache.txt       |  194 +++
Documentation/technical/rev-cache.txt |  634 +++++++++
Makefile                              |    2 +
builtin.h                             |    1 +
builtin/gc.c                          |    9 +
builtin/rev-cache.c                   |  339 +++++
command-list.txt                      |    1 +
commit.c                              |   36 +-
git.c                                 |    1 +
list-objects.c                        |   46 +-
object.h                              |    3 +-
rev-cache.c                           | 2468 +++++++++++++++++++++++++++++++++
rev-cache.h                           |  123 ++
revision.c                            |   90 +-
revision.h                            |   44 +-
t/t6019-rev-cache-list.sh             |  263 ++++
16 files changed, 4227 insertions(+), 27 deletions(-)

create mode 100644 Documentation/git-rev-cache.txt create mode 100644 Documentation/technical/rev-cache.txt create mode 100644 builtin/rev-cache.c
create mode 100644 rev-cache.c
create mode 100644 rev-cache.h
create mode 100644 t/t6019-rev-cache-list.sh

2010/4/6 Junio C Hamano <[email protected]>:

Yes, I agree. I'll have a look.

Björn Gustavsson, Erlang/OTP, Ericsson AB

In case people are wondering what the message Björn is responding to said...

From: Junio C Hamano <[email protected]>
Subject: Re: [PATCH 0/7 (v5)] rev-cache
Date: Mon, 05 Apr 2010 23:49:57 -0700
References: <[email protected]>
Message-ID: <[email protected]>

Björn, I thought your "apply --whitespace=fix" patch was supposed to be able to cope with this series whose [2/7] introduces a new file with a trailing blank line "t/t6019-rev-cache-list.sh" (which is correctly fixed not to have that extra blank line at the end), and then a subsequent patch in the series [3/7] still tries to apply a patch assuming that the extra blank line was still there.

But that is apparently not the case. I think your series dealt only for the cases where the additional contents came _after_ the fixed-and-gone blank lines, and never dealt with the case where such lines appear in the context.

The message didn't go to vger because I botched while editing To/Cc lines.