Skip to contentSkip to conversation

Detection of directory renames

PATCH Superseded Yann Dirson 3 participants 11 messages Jun 15, 2016 Documentation

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v7 cover Detection of directory renames

Changes since v6.1:
* now using a sorted list for bulkmove_candidates, improved lookup code
* added a bit of doc
* fixed decl-after-stmt and code style issues (Junio)
* fixed signedness of i_am_not_single bitfield (sparse)
* removed useless filtering of debug messages in tests
* avoid using "git update-index" where "git mv" and friends are more readable
* added a testcase for move of a subdir, fixed the code
* more naming normalisation
* added short note in commit message about implementation
* added Thanks-to's to commit messages (hope I did not forget anyone)
* added more possible future uses to commit message
* moved list of optimisation opportunities from FIXME's to commit message, leaving only a handful of things as real FIXME tags (purposely kept as C99 comments)

Yann Dirson (3):
Introduce bulk-move detection in diffcore.
Add testcases for the --detect-bulk-moves diffcore flag. [WIP] Allow hiding renames of individual files involved in a
directory rename.

 Documentation/diff-options.txt   |    4 +
 Documentation/gitdiffcore.txt    |   12 +
 diff-lib.c                       |    6 +-
 diff.c                           |   21 ++-
 diff.h                           |    6 +
 diffcore-rename.c                |  418 ++++++++++++++++++++++++++++++++++++-
 diffcore.h                       |    2 +
 t/t4046-diff-rename-factorize.sh |  296 +++++++++++++++++++++++++++
 tree-diff.c                      |    4 +-
 9 files changed, 753 insertions(+), 16 deletions(-)
 create mode 100755 t/t4046-diff-rename-factorize.sh
Signature · 1 line
1.7.2.3
#1 lore raw

PATCH v7 1/3 Introduce bulk-move detection in diffcore.

This feature tries to group together files moving from and to identical directories - the most common case being directory renames.

It is activated by the new --detect-bulk-moves diffcore flag.

This is only the first step, adding the basic functionnality and adding support to raw diff output (and it is not supported in unified-diff output yet)

It is implemented as a new pass in diffcore-rename, occuring after the file renames get detected, grouping those renames looking like a move of a full directory into some other place.

The output of raw diff is displayed as "Rnnn a/* b/". Those cannot be confused with renames of files named "whatever/*" with a literal star character, from the full-zero SHA1's.

Possible optimisations to this code include:
* avoid use of i_am_not_single by using a separate list
* use a more informative prefixcmp to avoid strcmp calls eg. in discard_if_outside()
* optimize for bulk insertions (avoid useless successive memmove's)

Other future developements to be made on top of this include:
* extension of unified-diff format to express this
* detect inexact bulk-moves (where some files were not moved, or were moved to a different place) - problem of computing a similarity score
* display as such the special case of directory move/rename
* application of such new diffs: issue a conflict, or just a warning ?
* teach git-svn (and others ?) to make use of that flag
* handle new conflict type "bulk-move/add"
* detect "directory splits" as well
* use inexact dir renames to bump score of below-threshold renames from/to same locations
* support other types of bluk-grouping, like prefixes (see eg. kernel 5d1e859c), and maybe config-specified patterns
* add yours here

This patch has been improved by the following contributions:
- Jonathan Nieder: better implementation of copy_dirname()
- Jonathan Nieder: portable implementation of memrchr() in another patch
- Junio C Hamano: split individual renames hiding under control of another flag
- Junio C Hamano: coding style issues
- Ævar Arnfjörð Bjarmason: Don't use C99 comments.
- Jonathan Nieder: just too many other helpful suggestions to list them all

Thanks-to: Jonathan Nieder <[email protected]>
Thanks-to: Junio C Hamano <[email protected]>
Thanks-to: Ævar Arnfjörð Bjarmason <[email protected]>

---
 Documentation/diff-options.txt |    4 +
 Documentation/gitdiffcore.txt  |   12 ++
 diff-lib.c                     |    6 +-
 diff.c                         |   14 ++-
 diff.h                         |    3 +
 diffcore-rename.c              |  354 ++++++++++++++++++++++++++++++++++++++--
 diffcore.h                     |    1 +
 tree-diff.c                    |    4 +-
 8 files changed, 383 insertions(+), 15 deletions(-)
Documentation/diff-options.txt +4 -0
index e954af0..2d99511 100644
@@ -243,6 +243,10 @@ endif::git-log[]
 	delete/add pair to be a rename if more than 90% of the file
 	hasn't changed.
 
+--detect-bulk-moves::
+	Detect bulk move of all files of a directory into a
+	different one.
+
 -C[<n>]::
 	Detect copies as well as renames.  See also `--find-copies-harder`.
 	If `n` is specified, it has the same meaning as for `-M<n>`.
Documentation/gitdiffcore.txt +12 -0
index 6af29a4..edeb159 100644
@@ -175,6 +175,18 @@ the expense of making it slower.  Without `\--find-copies-harder`,
 'git diff-{asterisk}' commands can detect copies only if the file that was
 copied happened to have been modified in the same changeset.
 
+Bulk move of all files of a directory into a different one can get
+detected using the `\--detect-bulk-moves` option.  This adds an
+additional pass on top of the results of per-file rename detection.
+They are reported with NULL SHA1 id, in addition to the file renames:
+
+------------------------------------------------
+:040000 040000 0000000... 0000000... R100	a/*	b/
+:100644 100644 0123456... 1234567... R90	a/file0	b/file3
+:100644 100644 0123456... 1234567... R100	a/file1	b/file1
+:100644 100644 0123456... 1234567... R100	a/file2	b/file2
+------------------------------------------------
+
 
 diffcore-merge-broken: For Putting "Complete Rewrites" Back Together
 --------------------------------------------------------------------
diff-lib.c +4 -2
index 392ce2b..5ec3ddc 100644
@@ -208,7 +208,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)
 						    ce_option, &dirty_submodule);
 		if (!changed && !dirty_submodule) {
 			ce_mark_uptodate(ce);
-			if (!DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER))
+			if (!DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER) &&
+			    !DIFF_OPT_TST(&revs->diffopt, DETECT_BULK_MOVES))
 				continue;
 		}
 		oldmode = ce->ce_mode;
@@ -338,7 +339,8 @@ static int show_modified(struct rev_info *revs,
 
 	oldmode = old->ce_mode;
 	if (mode == oldmode && !hashcmp(sha1, old->sha1) && !dirty_submodule &&
-	    !DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER))
+	    !DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER) &&
+	    !DIFF_OPT_TST(&revs->diffopt, DETECT_BULK_MOVES))
 		return 0;
 
 	diff_change(&revs->diffopt, oldmode, mode,
diff.c +13 -1
index 71efa8e..1d88281 100644
@@ -3188,6 +3188,11 @@ int diff_opt_parse(struct diff_options *options, const char **av, int ac)
 		DIFF_OPT_SET(options, REVERSE_DIFF);
 	else if (!strcmp(arg, "--find-copies-harder"))
 		DIFF_OPT_SET(options, FIND_COPIES_HARDER);
+	else if (!strcmp(arg, "--detect-bulk-moves")) {
+		DIFF_OPT_SET(options, DETECT_BULK_MOVES);
+		if (!options->detect_rename)
+			options->detect_rename = DIFF_DETECT_RENAME;
+	}
 	else if (!strcmp(arg, "--follow"))
 		DIFF_OPT_SET(options, FOLLOW_RENAMES);
 	else if (!strcmp(arg, "--color"))
@@ -3466,7 +3471,14 @@ static void diff_flush_raw(struct diff_filepair *p, struct diff_options *opt)
 	if (p->status == DIFF_STATUS_COPIED ||
 	    p->status == DIFF_STATUS_RENAMED) {
 		const char *name_a, *name_b;
-		name_a = p->one->path;
+		if (p->is_bulkmove) {
+			/* append "*" to the first dirname */
+			char buf[PATH_MAX];
+			char *next = memccpy(buf, p->one->path, '\0', PATH_MAX);
+			next[-1] = '*'; *next = '\0';
+			name_a = buf;
+		} else
+			name_a = p->one->path;
 		name_b = p->two->path;
 		strip_prefix(opt->prefix_length, &name_a, &name_b);
 		write_name_quoted(name_a, opt->file, inter_name_termination);
diff.h +3 -0
index 1fd44f5..b0d6fa6 100644
@@ -78,6 +78,7 @@ typedef struct strbuf *(*diff_prefix_fn_t)(struct diff_options *opt, void *data)
 #define DIFF_OPT_IGNORE_UNTRACKED_IN_SUBMODULES (1 << 25)
 #define DIFF_OPT_IGNORE_DIRTY_SUBMODULES (1 << 26)
 #define DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG (1 << 27)
+#define DIFF_OPT_DETECT_BULK_MOVES  (1 << 28)
 
 #define DIFF_OPT_TST(opts, flag)    ((opts)->flags & DIFF_OPT_##flag)
 #define DIFF_OPT_SET(opts, flag)    ((opts)->flags |= DIFF_OPT_##flag)
@@ -265,6 +266,8 @@ extern void diffcore_fix_diff_index(struct diff_options *);
 "  -C            detect copies.\n" \
 "  --find-copies-harder\n" \
 "                try unchanged files as candidate for copy detection.\n" \
+"  --detect-bulk-moves\n" \
+"                detect wholesale directory renames.\n" \
 "  -l<n>         limit rename attempts up to <n> paths.\n" \
 "  -O<file>      reorder diffs according to the <file>.\n" \
 "  -S<string>    find filepair whose only one side contains the string.\n" \
diffcore-rename.c +343 -11

416 lines not on this page · raw message

diffcore.h +1 -0
index b8f1fde..6dab95b 100644
@@ -69,6 +69,7 @@ struct diff_filepair {
 	unsigned broken_pair : 1;
 	unsigned renamed_pair : 1;
 	unsigned is_unmerged : 1;
+	unsigned is_bulkmove : 1;
 };
 #define DIFF_PAIR_UNMERGED(p) ((p)->is_unmerged)
 
tree-diff.c +3 -1
index cd659c6..5d9f123 100644
@@ -49,7 +49,9 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2, const
 		show_entry(opt, "+", t2, base, baselen);
 		return 1;
 	}
-	if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER) && !hashcmp(sha1, sha2) && mode1 == mode2)
+	if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER) &&
+	    !DIFF_OPT_TST(opt, DETECT_BULK_MOVES) &&
+	    !hashcmp(sha1, sha2) && mode1 == mode2)
 		return 0;
 
 	/*
Signature · 1 line
1.7.2.3
#2 lore raw

Re: [PATCH v7 1/3] Introduce bulk-move detection in diffcore.

Quoted 3 lines · Yann Dirson

The output of raw diff is displayed as "Rnnn a/* b/". Those cannot be confused with renames of files named "whatever/*" with a literal star character, from the full-zero SHA1's.

I do not particularly like this asterisk here. It is ambiguous with the case where you renamed a directory whose name is '*' below a/ to a new directory whose name is 'b', isn't it? Using 0{40} as a differentiator like you did is probably a good idea, but then I do not see a sound reason you would need nor want that asterisk.

Documentation/gitdiffcore.txt the expense of making it slower. Without `\--find-copies-harder`, · line 187

Quoted 10 lines
Documentation/gitdiffcore.txt +10 -0
+Bulk move of all files of a directory into a different one can get
+detected using the `\--detect-bulk-moves` option.  This adds an
+additional pass on top of the results of per-file rename detection.
+They are reported with NULL SHA1 id, in addition to the file renames:
+
+------------------------------------------------
+:040000 040000 0000000... 0000000... R100	a/*	b/
+:100644 100644 0123456... 1234567... R90	a/file0	b/file3
+:100644 100644 0123456... 1234567... R100	a/file1	b/file1
+:100644 100644 0123456... 1234567... R100	a/file2	b/file2

This is kind of interesting. Let's address two issues that should be uncontroversial:

(1) please do not use a/ and b/, as the reader can easily be confused and mistakenly think you are referring to the pre-/post- indicator we use when writing textual diffs (e.g. "diff --git a/frotz b/frotz"). You are illustrating rename between a directory to another directory, both of which are tracked paths.

(2) The last two lines are R100 but with different object names between pre- and post-images, which is wrong and unnecessarily puzzling. Please make them match.

These obvious two complaints behind us, there is one more interesting thing in the above, which is _not_ a complaint.

What about renaming of a/file0 to b/file3? Is this part of "all files from directory A moved to directory B"? IOW, is the goal of this series to use the "A/* -> B/" to label the change as bulk directory rename, if the preimage has A/{1,2,3} and the postimage has their moved contents in B/{one,two,three}?

I am wondering about the utility of such an extra information. If there were no "a/file0 -> b/file3" entry in the example, I would imagine that we could use this "a/* -> b/" information to move "a/file5" to "b/file5" when rebasing this patch to apply to a different preimage that had files other than file{1,2} in directory "a", and I would further imagine that might be a wonderful thing.

But if the new "a/* -> b/" hint only gives "things from A/ have migrated to B/ but I can give no information on what name they took under their new home", that may probably reduce the utility of this feature.

I dunno.

#3 lore raw

Re: [PATCH v7 1/3] Introduce bulk-move detection in diffcore.

Quoted 11 lines · Junio C Hamano
Quoted 3 lines · Yann Dirson

The output of raw diff is displayed as "Rnnn a/* b/". Those cannot be confused with renames of files named "whatever/*" with a literal star character, from the full-zero SHA1's.

I do not particularly like this asterisk here. It is ambiguous with the case where you renamed a directory whose name is '*' below a/ to a new directory whose name is 'b', isn't it? Using 0{40} as a differentiator like you did is probably a good idea, but then I do not see a sound reason you would need nor want that asterisk.

It all boils down to one of the original motivations of this patch:
allowing exporters like git-svn to provide the right level of information on directory renames, instead of only telling svn that the git-svn user has renamed all individual files separately.

It is only later that I realized that the larger picture was about bulk moves, and directory renames were only a subset of those. So to be able here to provide the right level of information to git-svn, we need a separate syntax for the special case of dir renaming - and it appeared natural to me to use something as close as possible to the usual syntax used for the operation:

git mv foo bar => rename directory "foo" to "bar"
git mv foo/* bar/ => move all contents of "foo" to pre-existing "bar"

(I would still add trailing slashed to the first form for the sake of ambiguity, obviously).

Now I'm not 100% happy with using 0{40} as differentiator - eg. we may some day find out that it is indeed useful to output the tree hashes there.

OTOH, the quoting rules for diff output are quite minimalist, I don't know whether adding "*" as a character that requires quotes around the filename would be acceptable: then we'll be able to differentiate "foo/*" from "foo/"* or "foo"/* (quotes here for emphasis, for interaction with other possible special chars) - the current code would have a strong bias towards "foo/"*, but neither seem very sexy to me (but since they would only appear in "pathologic" situations, do we care ?)

[off-topic] while playing with (un)quoted filenames I noticed spaces do not cause quoting, which can cause its own problems, like (not) making a visual distinction between file "foo" and file "foo " in diff output - not that it would be good practice, but someone can always shoot itself in the foot, and it appears we have not thought yet on how to help him not to shoot himself in this particular way.

Quoted 5 lines
Quoted 5 lines
Documentation/gitdiffcore.txt +5 -0
+Bulk move of all files of a directory into a different one can get
+detected using the `\--detect-bulk-moves` option.  This adds an
+additional pass on top of the results of per-file rename detection.
+They are reported with NULL SHA1 id, in addition to the file renames:
+
Quoted 8 lines
Quoted 5 lines

+------------------------------------------------
+:040000 040000 0000000... 0000000... R100 a/* b/
+:100644 100644 0123456... 1234567... R90 a/file0 b/file3
+:100644 100644 0123456... 1234567... R100 a/file1 b/file1
+:100644 100644 0123456... 1234567... R100 a/file2 b/file2

This is kind of interesting. Let's address two issues that should be uncontroversial:

OK.

Quoted 5 lines

These obvious two complaints behind us, there is one more interesting thing in the above, which is _not_ a complaint.

What about renaming of a/file0 to b/file3? Is this part of "all files from directory A moved to directory B"?

The move+rename is "part of" the bulk rename in that the file is moved from/two the same dirs as its siblings, but I acknowledge that it is more than that.

Quoted 4 lines

IOW, is the goal of this series
to use the "A/* -> B/" to label the change as bulk directory rename, if the preimage has A/{1,2,3} and the postimage has their moved contents in B/{one,two,three}?

Yes. But --hide-bulk-move-details would not hide them, as they would not be strictly included in the bulk move. Desite their name change, they are however a confirmation that the contents of A/ was move to B/.

Quoted 6 lines

I am wondering about the utility of such an extra information. If there were no "a/file0 -> b/file3" entry in the example, I would imagine that we could use this "a/* -> b/" information to move "a/file5" to "b/file5" when rebasing this patch to apply to a different preimage that had files other than file{1,2} in directory "a", and I would further imagine that might be a wonderful thing.

I imagined that as well, and that situation would not be a problem:
since "a/file0 -> b/file3" would be there in the rebased patch, "apply" would be able to spot the possible conflict.

OTOH, I had the vision of "merge does automatic moves" when starting this project, but got convinced on-list that there are always cases where the "automatic move" on merge would be wrong, and that we should report a conflict instead.

That would mostly shift the problem to: how do we provide support for easily solving this type of problem (regardless of how we want to represent them, which may not be trivial either).

Quoted 3 lines

But if the new "a/* -> b/" hint only gives "things from A/ have migrated to B/ but I can give no information on what name they took under their new home", that may probably reduce the utility of this feature.

It would still mean that "ALL things from A/ have migrated", and does not imply anything more, just like the fact that a file was renamed does not prevent us to say that while moving it was also modified.

Signature · 1 line
Yann
#4 lore raw

Bulk move and some of its close relatives

Quoted 2 lines · Yann Dirson

It is only later that I realized that the larger picture was about bulk moves, and directory renames were only a subset of those.

I had a couple of thoughts about this dir-rename -> bulk-moves shift today while riding to/from work. So here are some of the fresh air I got...

Is a bulk-move finally so special ? There is a close relative to it which may be of interest too: bulk deletion. If we implement detection of bulk deletion, we can build on it to get:

* detection of a new type of conflict: addition in a branch of a new file in a dir which is deleted in the other branch (quite similar to the one we were talking about, of a new file in a dir that got moved in other branch)
* split a good part of the current bulk-move patch in a standalone patch, which is something I wondered how to do
* probably a much better base for directory-split detection

Then, since a bulk-move is made of a bulk deletion, and addition somewhere else of those deleted files, it became tempting to contemplate the concept of "bulk addition". At first sight it seems awkward, I confess, especially "bulk addition into a preexisting dir", which is a bit hard to define with precision. OTOH, "bulk addition of a brand new dir" could bring goodies as well:

* new conflict type: creation in both branches of a new directory with same name

That confict type made me think of another one. Some of you may have noticed in the v7 commit message a suggestion for future developments labelled "support other types of bluk-grouping, like prefixes, and maybe config-specified patterns". Now if we get the capability of specifying how we define a "group of files" with something other than the directory hierarchy (let's say eg. that in t/ every "t[0-9]{4}-" is a prefix), the "bulk-add/bulk-add" conflict generated by 2 tests with same numeric ID would be detected as a conflict.

Another grouping example would be by file suffix - source-code reorgs eg. bulk-moving header files into their own dir are not rare either.

Does that sound sane ?

Signature · 1 line
Yann
#5 lore raw

Re: Bulk move and some of its close relatives

Oh, and another I forgot to mention is detection of bulk copies. It is somewhat tricky: while --find-copies-harder will properly work in combination with --detect-bulk-copies, for the standard case we will need to use find-copies-harder for those cases where not all of the copied files have been modified.

For this, maybe it could help to store the unmodified src files in a separate list to avoid pref degradation for the common case (similar to my plans to avoid the need of the i_am_not_single flag in the current bulkmove series by using a separte list for dst files).

Signature · 1 line
Yann
#6 lore raw

Re: [PATCH v7 1/3] Introduce bulk-move detection in diffcore.

In reply to Yann Dirson
Quoted 3 lines · Yann Dirson

OTOH, the quoting rules for diff output are quite minimalist, I don't know whether adding "*" as a character that requires quotes around the filename would be acceptable.

I suspect that would be an unacceptable entry to a slippery slope.

Quoted 20 lines
Quoted 4 lines

IOW, is the goal of this series
to use the "A/* -> B/" to label the change as bulk directory rename, if the preimage has A/{1,2,3} and the postimage has their moved contents in B/{one,two,three}?

Yes. But --hide-bulk-move-details would not hide them, as they would not be strictly included in the bulk move. Desite their name change, they are however a confirmation that the contents of A/ was move to B/.

Quoted 6 lines

I am wondering about the utility of such an extra information. If there were no "a/file0 -> b/file3" entry in the example, I would imagine that we could use this "a/* -> b/" information to move "a/file5" to "b/file5" when rebasing this patch to apply to a different preimage that had files other than file{1,2} in directory "a", and I would further imagine that might be a wonderful thing.

I imagined that as well, and that situation would not be a problem:
since "a/file0 -> b/file3" would be there in the rebased patch, "apply" would be able to spot the possible conflict.

Quoted 6 lines

OTOH, I had the vision of "merge does automatic moves" when starting this project, but got convinced on-list that there are always cases where the "automatic move" on merge would be wrong, and that we should report a conflict instead.

That would mostly shift the problem to...

That would mostly make this patch not worth worrying about, wouldn't it? What's the point of spending extra cycles to say "many things have moved in the same direction" without turning it into a usable information?

#7 lore raw

Re: [PATCH v7 1/3] Introduce bulk-move detection in diffcore.

Quoted 7 lines · Junio C Hamano
Quoted 3 lines · Yann Dirson

OTOH, the quoting rules for diff output are quite minimalist, I don't know whether adding "*" as a character that requires quotes around the filename would be acceptable.

I suspect that would be an unacceptable entry to a slippery slope.

OK. I have thought of only 2 remaining options:

* keep using 0{40} sha1 as differentiator
=> will prevent us to ever use this sha1 for something useful
* introducing a new change-type letter
=> another slippery slope, with at least bulk removals and bulk copies in sight, with already too few letters to keep them meaningful to human reader
* introducing a "bulk" flag
=> needs extension of the raw format
Would could want to extend the <letter><score> group, for things like "RB100" (bulk rename), "CB090" (imperfect bulk copy), "DB096" (imperfect bulk removal, meaning there were some files left in source directory)

Quoted 20 lines
Quoted 20 lines
Quoted 4 lines

IOW, is the goal of this series
to use the "A/* -> B/" to label the change as bulk directory rename, if the preimage has A/{1,2,3} and the postimage has their moved contents in B/{one,two,three}?

Yes. But --hide-bulk-move-details would not hide them, as they would not be strictly included in the bulk move. Desite their name change, they are however a confirmation that the contents of A/ was move to B/.

Quoted 6 lines

I am wondering about the utility of such an extra information. If there were no "a/file0 -> b/file3" entry in the example, I would imagine that we could use this "a/* -> b/" information to move "a/file5" to "b/file5" when rebasing this patch to apply to a different preimage that had files other than file{1,2} in directory "a", and I would further imagine that might be a wonderful thing.

I imagined that as well, and that situation would not be a problem:
since "a/file0 -> b/file3" would be there in the rebased patch, "apply" would be able to spot the possible conflict.

Quoted 8 lines
Quoted 6 lines

OTOH, I had the vision of "merge does automatic moves" when starting this project, but got convinced on-list that there are always cases where the "automatic move" on merge would be wrong, and that we should report a conflict instead.

That would mostly shift the problem to...

That would mostly make this patch not worth worrying about, wouldn't it?

(assuming "that" refers to not doing automatic moves on merge)

No, at least the "avoid hiding useful info among less-relevant stuff" provided by the human-targetted --hide-bulk-move-details would stay regardless of what we decide here.

Quoted 2 lines

What's the point of spending extra cycles to say "many things have moved in the same direction" without turning it into a usable information?

If we issue a conflict, we have already gone farther than where we are now. Deciding to have this particular type of conflict auto-resolved in the way useful to a wider audience could be done (having "adds" follow the bulkmove), with obviously the option of not auto-resolving, and possibly, the option of auto-resolving "adds" as they are (ie. keep current behaviour).

But my feeling is "automatically solving the conflict" would be be only marginally different from applying a patch generated with (what in current series is) --hide-bulk-move-details where you cannot know which individual files were moved in the patch author's tree, and has no info to decide if his own added files are at the right place or need to follow the move (which I understood to be your point in [1]).

Or did I miss your point ?

[1] http://marc.info/?l=git&m=122610146506413

Signature · 1 line
Yann
#8 lore raw

PATCH v7 2/3 Add testcases for the --detect-bulk-moves diffcore flag.

In reply to Yann Dirson

This notably includes a couple of tests for cases known not to be working correctly yet.

This patch has been improved by the following contributions:
- Jonathan Nieder: reworked style of test script
- Jonathan Nieder: use "git commit" in test instead of only plumbing, and use test_tick
- Sverre Rabbelier: anonymize hashes

Thanks-to: Jonathan Nieder <[email protected]>
Thanks-to: Sverre Rabbelier <[email protected]>

---
 t/t4046-diff-rename-factorize.sh |  296 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 296 insertions(+), 0 deletions(-)
 create mode 100755 t/t4046-diff-rename-factorize.sh
t/t4046-diff-rename-factorize.sh new file +296 -0
new file mode 100755
index 0000000..88e5b5b
@@ -0,0 +1,296 @@
+#!/bin/sh
+#
+# Copyright (c) 2008,2010 Yann Dirson
+# Copyright (c) 2005 Junio C Hamano
+#
+
+# TODO for dir renames:
+# * two dirs or more moving all their files to a single dir
+# * simultaneous bulkmove and rename
+
+test_description='Test rename factorization in diff engine.
+
+'
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/diff-lib.sh
+
+test_expect_success 'setup' '
+	git commit --allow-empty -m "original empty commit"
+
+	mkdir a &&
+	printf "Line %s\n" 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >a/path0 &&
+	sed <a/path0 >a/path1 s/Line/Record/ &&
+	sed <a/path0 >a/path2 s/Line/Stuff/ &&
+	sed <a/path0 >a/path3 s/Line/Blurb/ &&
+
+	git update-index --add a/path* &&
+	test_tick &&
+	git commit -m "original set of files"
+
+	: rename the directory &&
+	git mv a b
+'
+test_expect_success 'diff-index --detect-bulk-moves after directory move.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/*	b/
+	:100644 100644 X X R#	a/path0	b/path0
+	:100644 100644 X X R#	a/path1	b/path1
+	:100644 100644 X X R#	a/path2	b/path2
+	:100644 100644 X X R#	a/path3	b/path3
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup non-100% rename' '
+	echo "Line 16" >>b/path0 &&
+	git mv b/path2 b/2path &&
+	git rm -f b/path3 &&
+	echo anything >b/path100 &&
+	git add b/path100
+'
+test_expect_success 'diff-index --detect-bulk-moves after content changes.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/*	b/
+	:100644 000000 X X D#	a/path3
+	:100644 100644 X X R#	a/path2	b/2path
+	:100644 100644 X X R#	a/path0	b/path0
+	:100644 100644 X X R#	a/path1	b/path1
+	:000000 100644 X X A#	b/path100
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup bulk move that is not directory move' '
+	git reset -q --hard &&
+
+	mkdir c &&
+	(
+		for i in 0 1 2; do
+			cp a/path$i c/apath$i || exit
+		done
+	) &&
+	git update-index --add c/apath* &&
+	test_tick &&
+	git commit -m "first set of changes" &&
+
+	git mv c/* a/
+'
+test_expect_success 'diff-index --detect-bulk-moves without full-dir rename.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	c/*	a/
+	:100644 100644 X X R#	c/apath0	a/apath0
+	:100644 100644 X X R#	c/apath1	a/apath1
+	:100644 100644 X X R#	c/apath2	a/apath2
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup bulk move with new file in source dir' '
+	echo > c/anotherpath "How much wood?" &&
+	git update-index --add c/another*
+'
+test_expect_success 'diff-index --detect-bulk-moves with new file in source dir.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	c/*	a/
+	:100644 100644 X X R#	c/apath0	a/apath0
+	:100644 100644 X X R#	c/apath1	a/apath1
+	:100644 100644 X X R#	c/apath2	a/apath2
+	:000000 100644 X X A#	c/anotherpath
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup bulk move with interfering copy' '
+	rm c/anotherpath &&
+	git update-index --remove c/anotherpath &&
+	mkdir b &&
+	cp a/apath0 b/apath9 &&
+	echo >> a/apath0 "more" &&
+	git update-index --add a/apath0 b/apath9
+'
+# scores select the "wrong" one as "moved" (only a suboptimal detection)
+test_expect_failure 'diff-index --detect-bulk-moves with interfering copy.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	c/*	a/
+	:100644 100644 X X R#	c/apath0	a/apath0
+	:100644 100644 X X R#	c/apath1	a/apath1
+	:100644 100644 X X R#	c/apath2	a/apath2
+	:100644 100644 X X C#	c/apath0	b/apath9
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup bulk move to toplevel' '
+	git reset -q --hard &&
+	git mv c/* .
+'
+test_expect_success 'diff-index --detect-bulk-moves bulk move to toplevel.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	c/*	./
+	:100644 100644 X X R#	c/apath0	apath0
+	:100644 100644 X X R#	c/apath1	apath1
+	:100644 100644 X X R#	c/apath2	apath2
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup move including a subdir, with some content changes' '
+	git reset -q --hard &&
+	mv c a/ &&
+	git update-index --add --remove a/c/* c/apath0 c/apath1 c/apath2 &&
+	test_tick &&
+	git commit -m "move as subdir" &&
+
+	git mv a b &&
+	echo foo >>b/c/apath0 &&
+	git update-index --add b/c/apath*
+'
+test_expect_success 'diff-index --detect-bulk-moves on a move including a subdir.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/*	b/
+	:040000 040000 X X R#	a/c/*	b/c/
+	:100644 100644 X X R#	a/c/apath0	b/c/apath0
+	:100644 100644 X X R#	a/c/apath1	b/c/apath1
+	:100644 100644 X X R#	a/c/apath2	b/c/apath2
+	:100644 100644 X X R#	a/path0	b/path0
+	:100644 100644 X X R#	a/path1	b/path1
+	:100644 100644 X X R#	a/path2	b/path2
+	:100644 100644 X X R#	a/path3	b/path3
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup move of only a subdir' '
+	git reset -q --hard &&
+	git mv a/c a/d &&
+	: rename a subdirectory of a/.
+'
+test_expect_success 'moving a subdir only' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/c/*	a/d/
+	:100644 100644 X X R#	a/c/apath0	a/d/apath0
+	:100644 100644 X X R#	a/c/apath1	a/d/apath1
+	:100644 100644 X X R#	a/c/apath2	a/d/apath2
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup move without a subdir' '
+	git reset -q --hard &&
+	mkdir b &&
+	: rename files in the directory but not subdir. &&
+	git mv a/path* b/
+'
+test_expect_success 'moving files but not subdirs is not mistaken for dir move' '
+	cat >expected <<-EOF &&
+	:100644 100644 X X R#	a/path0	b/path0
+	:100644 100644 X X R#	a/path1	b/path1
+	:100644 100644 X X R#	a/path2	b/path2
+	:100644 100644 X X R#	a/path3	b/path3
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup move of files and subdirs to different places' '
+	git reset -q --hard &&
+	git mv a/c b &&
+	git mv a d
+'
+test_expect_success 'moving subdirs into one dir and files into another is not mistaken for dir move' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/c/*	b/
+	:100644 100644 X X R#	a/c/apath0	b/apath0
+	:100644 100644 X X R#	a/c/apath1	b/apath1
+	:100644 100644 X X R#	a/c/apath2	b/apath2
+	:100644 100644 X X R#	a/path0	d/path0
+	:100644 100644 X X R#	a/path1	d/path1
+	:100644 100644 X X R#	a/path2	d/path2
+	:100644 100644 X X R#	a/path3	d/path3
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+# the same with different ordering
+test_expect_success 'setup move of files and subdirs to different places' '
+	git mv d 0
+'
+test_expect_success 'moving subdirs into one dir and files into another is not mistaken for dir move' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/c/*	b/
+	:100644 100644 X X R#	a/path0	0/path0
+	:100644 100644 X X R#	a/path1	0/path1
+	:100644 100644 X X R#	a/path2	0/path2
+	:100644 100644 X X R#	a/path3	0/path3
+	:100644 100644 X X R#	a/c/apath0	b/apath0
+	:100644 100644 X X R#	a/c/apath1	b/apath1
+	:100644 100644 X X R#	a/c/apath2	b/apath2
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_expect_success 'setup move of dir with only subdirs' '
+	git reset -q --hard &&
+	mkdir a/b &&
+	mv a/path* a/b/ &&
+	git update-index --add --remove a/path0 a/path1 a/path2 a/path3 a/b/path* &&
+	test_tick &&
+	git commit -m "move all toplevel files down one level" &&
+
+	git mv a z
+'
+# TODO: only a suboptimal non-detection
+test_expect_failure 'moving a dir with no direct children files' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	a/*	z/
+	:040000 040000 X X R#	a/b/*	z/b/
+	:040000 040000 X X R#	a/c/*	z/c/
+	:100644 100644 X X R#	a/b/path0	z/b/path0
+	:100644 100644 X X R#	a/b/path1	z/b/path1
+	:100644 100644 X X R#	a/b/path2	z/b/path2
+	:100644 100644 X X R#	a/b/path3	z/b/path3
+	:100644 100644 X X R#	a/c/apath0	z/c/apath0
+	:100644 100644 X X R#	a/c/apath1	z/c/apath1
+	:100644 100644 X X R#	a/c/apath2	z/c/apath2
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+# now test moving all files from toplevel into subdir (does not hides file moves) (needs consensus on syntax)
+# Note: this is a special case of move of a dir into one of its own subdirs, which in
+# turn is a variant of new files/dirs being added into a dir after all its contents
+# are moved away
+
+test_expect_success 'setup move from toplevel to subdir' '
+	git reset -q --hard HEAD~3 &&
+	mv a/* . &&
+	git update-index --add --remove a/path0 a/path1 a/path2 a/path3 path* &&
+	test_tick &&
+	git commit -m "move all files to toplevel" &&
+
+	mkdir z &&
+	git mv path* z/
+'
+test_expect_success '--detect-bulk-moves everything from toplevel.' '
+	cat >expected <<-EOF &&
+	:040000 040000 X X R#	./*	z/
+	:100644 100644 X X R#	path0	z/path0
+	:100644 100644 X X R#	path1	z/path1
+	:100644 100644 X X R#	path2	z/path2
+	:100644 100644 X X R#	path3	z/path3
+	EOF
+	git diff-index --detect-bulk-moves HEAD >current &&
+	compare_diff_raw expected current
+'
+
+test_done
Signature · 1 line
1.7.2.3
#9 lore raw

PATCH v7 3/3 Allow hiding renames of individual files involved in a directory rename.

In reply to Yann Dirson

Once has identified groups of bulk-moved files, and then
the --hide-bulk-move-details flag hides those of the individual renames which carry no other information (further name change, or content changes).

This makes it much easier to a human reader to spot content changes in a commit that also moves a whole subtree.

Important note: unified diff output is not currently useful, since the "bulk move" headers are not yet added by --detect-bulk-moves, but the redundant renames are really removed.

---
 diff.c            |    7 +++++
 diff.h            |    3 ++
 diffcore-rename.c |   68 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 diffcore.h        |    1 +
 4 files changed, 76 insertions(+), 3 deletions(-)
diff.c +7 -0
index 1d88281..3cc7a55 100644
@@ -3193,6 +3193,13 @@ int diff_opt_parse(struct diff_options *options, const char **av, int ac)
 		if (!options->detect_rename)
 			options->detect_rename = DIFF_DETECT_RENAME;
 	}
+	else if (!strcmp(arg, "--hide-bulk-move-details")) {
+		DIFF_OPT_SET(options, HIDE_DIR_RENAME_DETAILS);
+		if (!DIFF_OPT_TST(options, DETECT_BULK_MOVES))
+			DIFF_OPT_SET(options, DETECT_BULK_MOVES);
+		if (!options->detect_rename)
+			options->detect_rename = DIFF_DETECT_RENAME;
+	}
 	else if (!strcmp(arg, "--follow"))
 		DIFF_OPT_SET(options, FOLLOW_RENAMES);
 	else if (!strcmp(arg, "--color"))
diff.h +3 -0
index b0d6fa6..7f132d0 100644
@@ -79,6 +79,7 @@ typedef struct strbuf *(*diff_prefix_fn_t)(struct diff_options *opt, void *data)
 #define DIFF_OPT_IGNORE_DIRTY_SUBMODULES (1 << 26)
 #define DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG (1 << 27)
 #define DIFF_OPT_DETECT_BULK_MOVES  (1 << 28)
+#define DIFF_OPT_HIDE_DIR_RENAME_DETAILS (1 << 29)
 
 #define DIFF_OPT_TST(opts, flag)    ((opts)->flags & DIFF_OPT_##flag)
 #define DIFF_OPT_SET(opts, flag)    ((opts)->flags |= DIFF_OPT_##flag)
@@ -268,6 +269,8 @@ extern void diffcore_fix_diff_index(struct diff_options *);
 "                try unchanged files as candidate for copy detection.\n" \
 "  --detect-bulk-moves\n" \
 "                detect wholesale directory renames.\n" \
+"  --hide-bulk-move-details\n" \
+"                hide renames of individual files in a directory rename.\n" \
 "  -l<n>         limit rename attempts up to <n> paths.\n" \
 "  -O<file>      reorder diffs according to the <file>.\n" \
 "  -S<string>    find filepair whose only one side contains the string.\n" \
diffcore-rename.c +65 -3

96 lines not on this page · raw message

diffcore.h +1 -0
index 6dab95b..a4eb8e1 100644
@@ -69,6 +69,7 @@ struct diff_filepair {
 	unsigned broken_pair : 1;
 	unsigned renamed_pair : 1;
 	unsigned is_unmerged : 1;
+	unsigned uninteresting_rename : 1;
 	unsigned is_bulkmove : 1;
 };
 #define DIFF_PAIR_UNMERGED(p) ((p)->is_unmerged)
Signature · 1 line
1.7.2.3
#10 lore raw

RFC PATCH 1/1 Add --detect-bulk-moves output to unidiff format.

In reply to Yann Dirson

---

This patch is a complement to the v7 "Detection of directory renames" series. It produces output like the sample below (taken from kernel tree). Does that feel right ? Is it worth moving the output production to a separate func filling an strbuf ?

|commit 86c6d4d0acc64543a485c11e197d0bd2c5ae0bb2
|Author: Ralf Baechle <[email protected]>
|Date:   Thu Aug 5 13:25:55 2010 +0100
|
|    MIPS: PNX833x: Move code one directory level up.
|    
|    It was sharing the nxp directory but no code with pnx8550 and will fit
|    better into the new platform makefile scheme, if moved.
|    
|    Signed-off-by: Ralf Baechle <[email protected]>
|
|diff --git --detect-bulk-moves arch/mips/nxp/pnx833x/common/ arch/mips/pnx833x/common/
|similarity index 100%
|bulk move from arch/mips/nxp/pnx833x/common/*
|bulk move to arch/mips/pnx833x/common/
|diff --git --detect-bulk-moves arch/mips/nxp/pnx833x/stb22x/ arch/mips/pnx833x/stb22x/
|similarity index 100%
|bulk move from arch/mips/nxp/pnx833x/stb22x/*
|bulk move to arch/mips/pnx833x/stb22x/
|diff --git a/arch/mips/Makefile b/arch/mips/Makefile
|index 2ba20d9..0556bc4 100644

[...]

 diff.c |   48 +++++++++++++++++++++++++++++++++++++-----------
 1 files changed, 37 insertions(+), 11 deletions(-)
diff.c +37 -11
index 1d88281..8f2f67f 100644
@@ -2659,6 +2659,7 @@ static void run_diff_cmd(const char *pgm,
 	const char *xfrm_msg = NULL;
 	int complete_rewrite = (p->status == DIFF_STATUS_MODIFIED) && p->score;
 	int must_show_header = 0;
+	int use_color;
 
 	if (!DIFF_OPT_TST(o, ALLOW_EXTERNAL))
 		pgm = NULL;
@@ -2668,14 +2669,36 @@ static void run_diff_cmd(const char *pgm,
 			pgm = drv->external;
 	}
 
+	/*
+	 * don't use colors when the header is intended for an
+	 * external diff driver
+	 */
+	use_color = DIFF_OPT_TST(o, COLOR_DIFF) && !pgm;
+
+	if (p->is_bulkmove) {
+		const char *set = diff_get_color(use_color, DIFF_METAINFO);
+		const char *reset = diff_get_color(use_color, DIFF_RESET);
+		struct strbuf *msgbuf;
+		char *line_prefix = "";
+
+		if (o->output_prefix) {
+			msgbuf = o->output_prefix(o, o->output_prefix_data);
+			line_prefix = msgbuf->buf;
+		}
+		fprintf(o->file, "%s%sdiff --git --detect-bulk-moves %s %s%s\n",
+			line_prefix, set, one->path, two->path, reset);
+		fprintf(o->file, "%s%ssimilarity index %d%%%s\n",
+			line_prefix, set, similarity_index(p), reset);
+		fprintf(o->file, "%s%sbulk move from %s*%s\n",
+			line_prefix, set, one->path, reset);
+		fprintf(o->file, "%s%sbulk move to %s%s\n",
+			line_prefix, set, two->path, reset);
+		return;
+	}
+
 	if (msg) {
-		/*
-		 * don't use colors when the header is intended for an
-		 * external diff driver
-		 */
 		fill_metainfo(msg, name, other, one, two, o, p,
-			      &must_show_header,
-			      DIFF_OPT_TST(o, COLOR_DIFF) && !pgm);
+			      &must_show_header, use_color);
 		xfrm_msg = msg->len ? msg->buf : NULL;
 	}
 
@@ -2748,8 +2771,10 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
 		return;
 	}
 
-	diff_fill_sha1_info(one);
-	diff_fill_sha1_info(two);
+	if (!p->is_bulkmove) {
+		diff_fill_sha1_info(one);
+		diff_fill_sha1_info(two);
+	}
 
 	if (!pgm &&
 	    DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two) &&
@@ -3529,9 +3554,10 @@ static void diff_flush_patch(struct diff_filepair *p, struct diff_options *o)
 	if (diff_unmodified_pair(p))
 		return;
 
-	if ((DIFF_FILE_VALID(p->one) && S_ISDIR(p->one->mode)) ||
-	    (DIFF_FILE_VALID(p->two) && S_ISDIR(p->two->mode)))
-		return; /* no tree diffs in patch format */
+	if (!p->is_bulkmove &&
+	    ((DIFF_FILE_VALID(p->one) && S_ISDIR(p->one->mode)) ||
+	     (DIFF_FILE_VALID(p->two) && S_ISDIR(p->two->mode))))
+		return; /* no tree diffs in patch format, except for bulk moves */
 
 	run_diff(p, o);
 }
Signature · 1 line
1.7.2.3