diffcore-rename: support rename cache
PATCH 1/2 diffcore-rename: support rename cache
This patch teaches diffcore_rename() to look into $GIT_DIR/rename-cache and make use of it to recreate diff_filepair. With proper cache, there should be no available entry for estimation after exact matching.
Rename caching is per commit. I don't think abitrary tree-tree caching is worth it.
$GIT_DIR/rename-cache spans out like $GIT_DIR/objects. Each file corresponds to one commit. Its content consists of lines like this
<Destination SHA-1> <SPC> <Source SHA-1> <SPC> <Score in decimal> <NL>
This can be used to:
- Make --find-copies-harder pratically usable for moderate-size repositories. The first "git show" on a linux kernel commit was 5.3 sec, it then went down to 0.13 sec.
- Give git-svn a chance to (locally) import explicit renames from Subversion
- People may correct rename results for better diff, if automatic rename detection is not good enough.
--- diff.h | 2 + diffcore-rename.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++- log-tree.c | 2 + t/t4030-rename-cache.sh | 55 ++++++++++++++++++ 4 files changed, 199 insertions(+), 2 deletions(-) create mode 100755 t/t4030-rename-cache.sh
diff.h +2 -0
@@ -110,6 +110,8 @@ struct diff_options {add_remove_fn_t add_remove;diff_format_fn_t format_callback;void *format_callback_data; + +struct commit *commit; };enum color_diff {
diffcore-rename.c +140 -2
183 lines not on this page · raw message
log-tree.c +2 -0
@@ -518,6 +518,7 @@ int log_tree_commit(struct rev_info *opt, struct commit *commit) log.commit = commit; log.parent =NULL ; opt->loginfo = &log; + opt->diffopt.commit = commit; shown =log_tree_diff (opt, commit, &log);if (!shown && opt->loginfo && opt->always_show_header) {
@@ -527,5 +528,6 @@ int log_tree_commit(struct rev_info *opt, struct commit *commit) } opt->loginfo =NULL ;maybe_flush_or_die (stdout,"stdout" ); + opt->diffopt.commit =NULL ;return shown; }
t/t4030-rename-cache.sh new file +55 -0
55 lines not on this page · raw message
Signature · 1 line
1.6.0.3.802.g47c38
PATCH 2/2 diffcore-rename: add config option to allow to cache renames
If diff.cacherenames is true, then renames will be cached to $GIT_DIR/rename-cache. By default, it will not overwrite existing cache. Add --refresh-cache to overwrite.
---
if git-svn is going to use this, then perharps we should add a rule to prevent overwriting certain cache files with .keep files, so that git-svn generated cache does not get lost
Documentation/config.txt | 5 ++++ Documentation/diff-options.txt | 5 ++++ diff.c | 12 +++++++++++ diff.h | 2 + diffcore-rename.c | 41 ++++++++++++++++++++++++++++++++++++++++ t/t4030-rename-cache.sh | 27 ++++++++++++++++++++++++++ 6 files changed, 92 insertions(+), 0 deletions(-)
Documentation/config.txt +5 -0
@@ -630,6 +630,11 @@ diff.renames:: will enable basic rename detection. If set to "copies" or "copy", it will detect copies, as well. +diff.cacherenames:: + Tells git to automatically cache renames when detected. The + cache resides in $GIT_DIR/rename-cache, which is used by git + if exists. + fetch.unpackLimit:: If the number of objects fetched over the git native transfer is below this
Documentation/diff-options.txt +5 -0
@@ -102,6 +102,11 @@ endif::git-format-patch[] Turn off rename detection, even when the configuration file gives the default to do so. +--refresh-rename-cache:: + By default, when git finds a cached version of a commit, it + will not overwrite the cache. This option makes git overwrite + old cache or create a new one. + --check:: Warn if changes introduce trailing whitespace or an indent that uses a space before a tab. Exits with
diff.c +12 -0
@@ -26,6 +26,7 @@ int diff_use_color_default = -1;static const char *external_diff_cmd_cfg;int diff_auto_refresh_index =1 ;static int diff_mnemonic_prefix; +static int diff_cache_renames;static char diff_colors[][COLOR_MAXLEN ] = {"\033[m" ,/* reset */
@@ -109,6 +110,11 @@ int git_diff_basic_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"diff.cacherenames" )) { + diff_cache_renames =git_config_bool (var, value); +return 0 ; + } +if (!prefixcmp (var,"diff.color." ) || !prefixcmp (var,"color.diff." )) {int slot =parse_diff_color_slot (var,11 );if (!value)
@@ -2248,6 +2254,8 @@ int diff_setup_done(struct diff_options *options)if (options->detect_rename && options->rename_limit <0 ) options->rename_limit = diff_rename_limit_default; +if (options->detect_rename) + options->cache_renames = diff_cache_renames;if (options->setup &DIFF_SETUP_USE_CACHE ) {if (!active_cache)/* read-cache does not die even when it fails
@@ -2415,6 +2423,10 @@ int diff_opt_parse(struct diff_options *options, const char **av, int ac)DIFF_OPT_SET (options,RELATIVE_NAME ); options->prefix = arg +11 ; } +else if (!strcmp (arg,"--refresh-rename-cache" )) { + options->cache_renames =1 ; + options->refresh_rename_cache =1 ; + }/* xdiff options */ else if (!strcmp (arg,"-w" ) || !strcmp (arg,"--ignore-all-space" ))
diff.h +2 -0
@@ -85,6 +85,8 @@ struct diff_options {int pickaxe_opts;int rename_score;int rename_limit; +int cache_renames; +int refresh_rename_cache;int warn_on_too_large_rename;int dirstat_percent;int setup;
diffcore-rename.c +41 -0
53 lines not on this page · raw message
t/t4030-rename-cache.sh +27 -0
@@ -52,4 +52,31 @@ test_expect_success 'load create pair cache' ' test_cmp expected result ' +cat >expected <<EOF +f2ad6c76f0115a6ba5b00456a849810e7ec0af20 0000000000000000000000000000000000000000 0 +78981922613b2afb6025042ff6bd878ac1994e85 78981922613b2afb6025042ff6bd878ac1994e85 60000 +EOF +test_expect_success 'save rename cache' ' + P=.git/rename-cache/$(git rev-parse HEAD|sed "s,\(..\)\(.*\),\1/\2,") && + rm -r .git/rename-cache + git config diff.cacherenames true + git show --summary -C -M --find-copies-harder > /dev/null + test_cmp expected $P +' + +test_expect_success 'subsequent command does not change cache' ' + P=.git/rename-cache/$(git rev-parse HEAD|sed "s,\(..\)\(.*\),\1/\2,") && + echo corrupted >> $P + ! test_cmp expected $P && + git show --summary -C -M --find-copies-harder HEAD > /dev/null && + ! test_cmp expected $P +' + +test_expect_success 'overwrite cache with --refresh-rename-cache' ' + P=.git/rename-cache/$(git rev-parse HEAD|sed "s,\(..\)\(.*\),\1/\2,") && + ! test_cmp expected $P && + git show --summary -C -M --find-copies-harder --refresh-rename-cache HEAD > /dev/null && + test_cmp expected $P +' + test_done
Signature · 1 line
1.6.0.3.802.g47c38
Quoted 4 lines · Nguy???n Thái Ng???c Duy
This patch teaches
diffcore_rename()to look into $GIT_DIR/rename-cache and make use of it to recreatediff_filepair. With proper cache, there should be no available entry for estimation after exact matching.
This is something I have thought about in the past, good to see that implemented :)
Quoted 2 lines
Rename caching is per commit. I don't think abitrary tree-tree caching is worth it.
That could be a nice complement to my directory-rename patch.
Quoted 6 lines · Yann Dirson
Quoted 4 lines · Nguy???n Thái Ng???c Duy
This patch teaches
diffcore_rename()to look into $GIT_DIR/rename-cache and make use of it to recreatediff_filepair. With proper cache, there should be no available entry for estimation after exact matching.
Quoted 7 lines
This is something I have thought about in the past, good to see that implemented :)
Quoted 2 lines
Rename caching is per commit. I don't think abitrary tree-tree caching is worth it.
That could be a nice complement to my directory-rename patch.
Has anybody thought about interaction between that caching and pathspec limited operation?
Quoted 11 lines · Junio C Hamano
Quoted 9 lines · Yann Dirson
Quoted 4 lines · Nguy???n Thái Ng???c Duy
This patch teaches
diffcore_rename()to look into $GIT_DIR/rename-cache and make use of it to recreatediff_filepair. With proper cache, there should be no available entry for estimation after exact matching.This is something I have thought about in the past, good to see that implemented :)
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Rename caching is per commit. I don't think abitrary tree-tree caching is worth it.
That could be a nice complement to my directory-rename patch.
Has anybody thought about interaction between that caching and pathspec
limited operation?
I didn't. But I think all out-of-pathspec diff pairs are removed before it reaches diffcore_rename() so the cache has nothing to do with it (except it still loads full cache for a commit).
Signature · 1 line
Duy
Quoted 18 lines · Nguyen Thai Ngoc Duy
Quoted 17 lines · Junio C Hamano
Quoted 13 lines · Yann Dirson
Quoted 4 lines · Nguy???n Thái Ng???c Duy
This patch teaches
diffcore_rename()to look into $GIT_DIR/rename-cache and make use of it to recreatediff_filepair. With proper cache, there should be no available entry for estimation after exact matching.This is something I have thought about in the past, good to see that implemented :)
Quoted 2 lines
Rename caching is per commit. I don't think abitrary tree-tree caching is worth it.
That could be a nice complement to my directory-rename patch.
Quoted 7 lines
Quoted 3 lines
Has anybody thought about interaction between that caching and pathspec
limited operation?I didn't. But I think all out-of-pathspec diff pairs are removed before it reaches
diffcore_rename()so the cache has nothing to do with it (except it still loads full cache for a commit).
Well, it could be that an out-of-pathspec pair would have a better score than an in-pathspec one. Maybe cache recording should be turned off when doing pathspec limitation ?
Quoted 3 lines · Yann Dirson
Well, it could be that an out-of-pathspec pair would have a better score than an in-pathspec one. Maybe cache recording should be turned off when doing pathspec limitation ?
One thing I notice is that the cache works at the level of "here is the best rename for this commit." Maybe it could go down a level and say "here is the inexact rename score between these blobs". Then you would still find the best score between two blobs each time, but save the really computationally intensive part (which is comparing the actual _content_ of the blobs).
That should work in the face of path limiting or any other option, because it is caching something immutable: this is the similarity score between two pieces of content. And then you get arbitrary tree-to-tree speedups for free, since such a cache would be valid for every commit.
The downsides are:
- your cache is potentially bigger, since you are caching the score of every pair you look at, instead of just "good" pairs (OTOH, you are not doing a per-commit cache, which helps reduce the size)
- you can still "lie" about a score to pre-seed imported SVN renames, but such lying will actually apply to all commits.
-Peff
Quoted 7 lines · Jeff King
Quoted 3 lines · Yann Dirson
Well, it could be that an out-of-pathspec pair would have a better score than an in-pathspec one. Maybe cache recording should be turned off when doing pathspec limitation ?
Quoted 10 lines
One thing I notice is that the cache works at the level of "here is the
best rename for this commit." Maybe it could go down a level and say "here is the inexact rename score between these blobs". Then you would still find the best score between two blobs each time, but save the really computationally intensive part (which is comparing the actual _content_ of the blobs).
That should work in the face of path limiting or any other option, because it is caching something immutable: this is the similarity score between two pieces of content. And then you get arbitrary tree-to-tree speedups for free, since such a cache would be valid for every commit.
I did that and realized the cost was not from each diff, in --find-copies-harder case, but from the number of diffs you had to do. Even with exact matching on linux-2.6.git, it could take significant time (it was about 5 minutes in no-cache case, 1 minute without exact match cache, and less than 1 sec if everything is cached).
Quoted 6 lines
The downsides are:
- your cache is potentially bigger, since you are caching the score of every pair you look at, instead of just "good" pairs (OTOH, you are not doing a per-commit cache, which helps reduce the size)
It is huge if you accidentially add --find-copies-harder to your command, considering that every new file will be compared against every files in tree (about 25k).
Quoted 2 lines
- you can still "lie" about a score to pre-seed imported SVN renames, but such lying will actually apply to all commits.
Signature · 1 line
Duy
Quoted 9 lines · Nguyen Thai Ngoc Duy
Quoted 5 lines
The downsides are:
- your cache is potentially bigger, since you are caching the score of every pair you look at, instead of just "good" pairs (OTOH, you are not doing a per-commit cache, which helps reduce the size)
It is huge if you accidentially add --find-copies-harder to your command, considering that every new file will be compared against every files in tree (about 25k).
Hmm, yeah. I was thinking you might be able to do some kind of cut-off on the caching (i.e., don't bother storing anything that didn't come close). But you can't safely assume that because an entry isn't there, it isn't worth seeing (since it might also just not have been computed yet). You could still organize by commit, and then each commit is either fully computed or not. But then you still have a pathspec problem.
One thing you could do is just compute the rename score between all pairs, even if a pathspec is given, limit it to values over "0.5" (or something low, but that eliminates the totally uninteresting cases), and then store that as the complete cache for that commit (or tree pair, if you want to support that).
Then you would have the full information and could do an arbitrary pathspec limit on it. If you wanted to set the rename threshold below 0.5, then we would have to recompute without the cache (but in practice, that should be rare).
The real downside is that you pay for the whole-tree detection when you have asked for a pathspec (but only the first time, after which you can always generate from cache).
Just thinking out loud...
-Peff
Quoted 23 lines · Yann Dirson
Quoted 22 lines · Nguyen Thai Ngoc Duy
Quoted 20 lines · Junio C Hamano
Quoted 6 lines · Yann Dirson
Quoted 4 lines · Nguy???n Thái Ng???c Duy
This patch teaches
diffcore_rename()to look into $GIT_DIR/rename-cache and make use of it to recreatediff_filepair. With proper cache, there should be no available entry for estimation after exact matching.Quoted 7 lines
This is something I have thought about in the past, good to see that implemented :)
Quoted 2 lines
Rename caching is per commit. I don't think abitrary tree-tree caching is worth it.
That could be a nice complement to my directory-rename patch.
Has anybody thought about interaction between that caching and pathspec
limited operation?
Quoted 8 lines
Quoted 3 lines
I didn't. But I think all out-of-pathspec diff pairs are removed before it reaches
diffcore_rename()so the cache has nothing to do with it (except it still loads full cache for a commit).Well, it could be that an out-of-pathspec pair would have a better
score than an in-pathspec one. Maybe cache recording should be turned off when doing pathspec limitation ?
Right, recording should be turned off or something. Let me see..
Signature · 1 line
Duy