Filter-branch's "move tree to subdirectory" example fails with BSD sed?

The "Move tree to subdirectory" example in the 'git filter-branch' manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04. I'm using git version 1.7.0.4, which happened to be the version installed on the Ubuntu VM I had laying around. I think it's a difference between 'sed' on the two systems.

The example:

git filter-branch --index-filter \
        'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Demonstrating the problem is with sed:
git init "test"
cd "test"
mkdir -p subdirA/subdirB
echo content > subdirA/subdirB/file
git add .
git commit -m "initial commit"
git ls-files -s | sed "s-\t\"*-&newsubdir/-"

On Mac 10.7.3 the final command outputs:
100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0 subdirA/subdirB/file

On Ubuntu 10.04:
100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0 newsubdir/subdirA/subdirB/file

I can solve my immediate problem using Ubuntu, but is there a way we could rewrite the example to work on both systems? I'm afraid differences between BSD and GNU 'sed' are a bit beyond me.

--
Christopher Tiwald

Am 14.04.2012 18:00, schrieb Christopher Tiwald:

Perhaps a literal TAB instead of \t makes the example work?

It would be difficult, though, to write this down in the manual in an unambiguous way.

-- Hannes

Yes, I suspect that is the problem, too.

Maybe it would be simpler to just use perl:

git filter-branch --index-filter '
git ls-files -s |

     perl -pe "s{\t\"?}{$&newsubdir/}" |
     GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&

mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
' HEAD

I also think tweaking the indentation and line breaks as I did above makes it a little more readable. If we are using perl, we could also just use "-z" to get rid of the funny quote handling.

I wish we could also get rid of the temporary index. It exists solely for the side effect of removing the existing entries (otherwise, you get both "foo" and "newsubdir/foo" in the resulting history). It would be nice if there was some flag to update-index to say "before you do anything, clear the existing index" (or I guess, "don't bother reading the existing index"). And then result could look like:

  git filter-branch --index-filter '
    git ls-files -sz |
    perl -0pe "s{\t}{\tnewsubdir/}" |
    git update-index --from-scratch -z --index-info
  ' HEAD

which is IMHO much easier to read and understand.

-Peff

This just discards existing entries from the index, which can be useful if you are rewriting all entries with "--index-info" or similar.

---
I tried to make something like:

  git update-index --from-scratch --index-info

work by avoiding reading all entries in the first place. However, update-index actually processes its arguments sequentially, so we _must_ read the index before we start processing arguments. But because it's sequential, a "clear" operation makes sense, since you clear, then add new entries.

 Documentation/git-update-index.txt |    3 +++
 builtin/update-index.c             |   10 ++++++++++
 2 files changed, 13 insertions(+)

index a3081f4..47f0ae6 100644
@@ -71,6 +71,9 @@ OPTIONS
 --cacheinfo <mode> <object> <path>::
 	Directly insert the specified info into the index.
 
+--clear::
+	Discard all existing entries from the index.
+
 --index-info::
         Read index information from stdin.
 

index a6a23fa..559dfae 100644
@@ -645,6 +645,13 @@ static int cacheinfo_callback(struct parse_opt_ctx_t *ctx,
 	return 0;
 }
 
+static int clear_callback(const struct option *opt,
+			  const char *arg, int unset)
+{
+	discard_cache();
+	return 0;
+}
+
 static int stdin_cacheinfo_callback(struct parse_opt_ctx_t *ctx,
 			      const struct option *opt, int unset)
 {

@@ -774,6 +781,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 			"add entries from standard input to the index",
 			PARSE_OPT_NONEG | PARSE_OPT_NOARG,
 			(parse_opt_cb *) stdin_cacheinfo_callback},
+		{OPTION_CALLBACK, 0, "clear", NULL, NULL,
+			"drop all index entries", PARSE_OPT_NONEG | PARSE_OPT_NOARG,
+			clear_callback},
 		{OPTION_LOWLEVEL_CALLBACK, 0, "unresolve", &has_errors, NULL,
 			"repopulate stages #2 and #3 for the listed paths",
 			PARSE_OPT_NONEG | PARSE_OPT_NOARG,

1.7.9.6.8.g992e5

At the very least I can give this patch series a go this evening. I'm in the process of combining 40 highly-intertwined, nested repositories into one, and should be able to test it under the conditions outlined in the updated example.

--
Christopher Tiwald

This worked great for me, at least in the case of adding trees to subdirectories.
Fwiw, I found the newer, patched example a little easier to parse, but I also had
spent a while figuring out what the original version did.

--
Christopher Tiwald

Over the years, this simple example has ended up quite hard to read because of the number of special cases that must be handled. Let's simplify it a bit:

1. Use the new "index-info --clear" to avoid the need for a temporary index.

2. Use "-z" and "perl -0" to avoid dealing with quoting issues. As a bonus, using perl means that "\t" will work consistently in regexps (the previous example using sed was reported to fail on OS X).

3. Change the indentation to keep one logical unit per line and avoid extra backslash-escaping.

---
 Documentation/git-filter-branch.txt |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

index 0f2f117..3df138f 100644
@@ -358,11 +358,11 @@ git filter-branch ... D..H --not C
 To move the whole tree into a subdirectory, or remove it from there:
 
 ---------------------------------------------------------------
-git filter-branch --index-filter \
-	'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
-		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
-			git update-index --index-info &&
-	 mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD
+git filter-branch --index-filter '
+	git ls-files -sz |
+	perl -0pe "s{\t}{\tnewsubdir/}" |
+	git update-index -z --clear --index-info
+' HEAD
 ---------------------------------------------------------------
 
 

1.7.9.6.8.g992e5

Wouldn't "git read-tree --prefix=newsubdir/" suffice without the pipeline? I.e.

        git filter-branch --index-filter '
		rm -f "$GIT_INDEX_FILE"
                git read-tree --prefix=newsubdir/ "$GIT_COMMIT"
	' HEAD

Yeah, I think it does (at least it makes sense to me, and worked on a simple test case). I think I might use "git read-tree --empty" instead of "rm", as it is a little more obvious what is going on. But other than that, it is much more readable.

It is a slight shame not to show an "ls-files | update-index" example, because that is the most general form. But since this form is quite often cut-and-paste by people, I think simple is better here. If we want to show a more complex example, then we should add it separately.

-Peff