RFC: Proposing git-filter-repo for inclusion in git.git
Subsystems (2)
Hi everyone,
git-filter-repo[1] is a tool for rewriting or restructuring git history. The recent Git Rev News article[2] on filter-repo introduced it and explained how it is much more versatile than git-filter-branch or BFG Repo Cleaner, and with speed comparable to BFG. (I probably also should have gone into more detail on how it is also much safer than filter-branch).
I propose merging git-filter-repo.git into git.git. There was some previous discussion at [3], but the tool was incomplete and still under heavy development back then.
Questions, comments, or concerns with this proposal? Alternative proposals? If inclusion is acceptable, are there any other tasks that need to be completed first?
Thanks,
Elijah
[1] https://github.com/newren/git-filter-repo
or
https://github.com/newren/git-filter-repo/blob/master/Documentation/git-filter-repo.txt
[2] https://git.github.io/rev_news/2019/08/21/edition-54/#an-introduction-to-git-filter-repo--written-by-elijah-newren
[3] https://public-inbox.org/git/CABPp-BFC--s+D0ijRkFCRxP5Lxfi+__YF4EdxkpO5z+GoNW7Gg@mail.gmail.com/
Quoted 3 lines · Elijah Newren
Questions, comments, or concerns with this proposal? Alternative proposals? If inclusion is acceptable, are there any other tasks that need to be completed first?
I do not want a discussion to begin with a Devil's Advocate response, but anyway...
Are we planning to go to all batteries included approach? I have a feeling that there are other tools (hello, "git imerge") that equally deserve attention by Git users; are we in the business of absorbing them all? How big a project will our tree become, and how much more activity would have to be haneld by the readership of the Git mailing list?
I'd rather see us shed non-core tools we already have (e.g. git-svn, cvs import/export) out of git.git and have them as independent projects. But that may be just me.
The benefits I see in the "leave as much as possible out" approach are (not in particular order):
- Distributed development and integration. The release schedule of Git-core does not have to be constrained by the readiness of non-core part.
- Choice of development tools, language, etc. The core part will stay in C, but peripheral tools do not have to be constrained by our choice. Those who run their own project can choose what they want to use and how they want to structure their development process.
The primary downside I would want to avoid from absorbing non-core stuff is that such peripheral tools (git-cvsimport, I am looking at you) can go stale when their development stalls, and nobody would be bold enough to suggest removing them.
Quoted 17 lines · Junio C Hamano
Quoted 3 lines · Elijah Newren
Questions, comments, or concerns with this proposal? Alternative proposals? If inclusion is acceptable, are there any other tasks that need to be completed first?
I do not want a discussion to begin with a Devil's Advocate response, but anyway...
Are we planning to go to all batteries included approach? I have a feeling that there are other tools (hello, "git imerge") that equally deserve attention by Git users; are we in the business of absorbing them all? How big a project will our tree become, and how much more activity would have to be haneld by the readership of the Git mailing list?
Quoted 3 lines
I'd rather see us shed non-core tools we already have (e.g. git-svn, cvs import/export) out of git.git and have them as independent projects. But that may be just me.
Ooh, if you're going to open this door, then a proposal I assumed would be shot down but which I'd be just about as happy with is:
* Remove git-filter-branch from git.git. Mention in the release
notes where people can go to get it.[1]
filter-branch is not merely a slow or difficult-to-use tool, it's one that *fosters* mistakes by making it hard to get things right in several different ways. Granted, people exercise extra caution using filter-branch because they know they need to, but there are so many gotchas that they're likely to accidentally mess something up. Those mess-ups are not always discovered immediately, and by then it's nearly cast into stone (rewriting being something you want to do very rarely).
For as long as git-filter-branch is part of git.git and other tools are outside, people will take that as a strong endorsement from us for filter-branch and use it regardless of how much other education exists...and that causes problems.
Thoughts?
Elijah
[1] We'd still have to decide where to put it. If no one else wants to do it, I could include it in git-filter-repo with the promise that it's there for backward compatibility for those that still need the tool, even if I recommend folks use filter-repo instead.
Quoted 6 lines · Elijah Newren
Ooh, if you're going to open this door, then a proposal I assumed would be shot down but which I'd be just about as happy with is:
* Remove git-filter-branch from git.git. Mention in the release
notes where people can go to get it.[1]
...
Yup, I think, especially given now filter-repo exists and is well known by Git users, this is a good move in the mid to longer term.
Quoted 4 lines
For as long as git-filter-branch is part of git.git and other tools are outside, people will take that as a strong endorsement from us for filter-branch and use it regardless of how much other education exists...and that causes problems.
Exactly.
cover Remove git-filter-branch from git.git; host it elsewhere
Following up on the suggestion to make git.git smaller and shed non-core tools, here's an RFC series to do so with git-filter-branch. This series first removes dependencies on git-filter-branch (of which there were very few), and then deletes git-filter-branch itself in the final commit.
I'm more than happy to consider alternate places for the filter-branch history (I had considered just merging it in with git-filter-repo), but for now I just made it available here:
https://github.com/newren/git-filter-branch
The rewrite above contains the history of the files deleted in Patch 5, plus a one-time copy of relevant build files (Makefiles, test-lib.sh, etc. -- I didn't want the whole history of these), and then touchups to streamline the build files and make them all work in this standalone repo.
Some highlevel notes on the patches:
* Patches 1&2: are good cleanups & performance wins regardless of whether the rest of the series is taken
* Patch 3: an attempt to improve i18n situation for external scripts, but discovered to not be necessary/useful for git-filter-branch specifically
* Patch 4:
* If we are good with deleting git-filter-branch now and just noting it in the release notes, then patch 4 could be simplified; there's no need to update git-filter-branch.txt in that case.
* If, however, we want to do some external messaging for an additional release cycle or two before moving git-filter-branch out of git.git, this patch will help us until then to at least avoid recommending a tool which will likely mangle user's data in unexpected ways. But it'd be really helpful if folks could review and opine on the BFG stuff if so.
* Patch 5: actually deletes git-filter-branch, its tests, and documentation.
Elijah Newren (5):
t6006: simplify and optimize empty message test
t3427: accelerate this test by using fast-export and fast-import
git-sh-i18n: work with external scripts
Recommend git-filter-repo instead of git-filter-branch in
documentation
Remove git-filter-branch, it is now external to git.git
.gitignore | 1 - Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 481 -------------------- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 2 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 4 +- Documentation/githooks.txt | 7 +- Makefile | 1 - command-list.txt | 1 - contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 662 ---------------------------- git-sh-i18n.sh | 7 +- t/perf/p7000-filter-branch.sh | 24 - t/t3427-rebase-subtree.sh | 32 +- t/t6006-rev-list-format.sh | 5 +- t/t7003-filter-branch.sh | 505 --------------------- t/t7009-filter-branch-null-sha1.sh | 55 --- t/t9902-completion.sh | 12 +- 19 files changed, 63 insertions(+), 1773 deletions(-) delete mode 100644 Documentation/git-filter-branch.txt delete mode 100755 git-filter-branch.sh delete mode 100755 t/perf/p7000-filter-branch.sh delete mode 100755 t/t7003-filter-branch.sh delete mode 100755 t/t7009-filter-branch-null-sha1.sh
Signature · 1 line
2.23.0.5.g775ebaa2a0
RFC PATCH 1/5 t6006: simplify and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of time ./t6006-rev-list-format.sh) by about 11% on Linux and by 13% on Mac.
--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
t/t6006-rev-list-format.sh +2 -3
@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Signature · 1 line
2.23.0.5.g775ebaa2a0
Re: [RFC PATCH 1/5] t6006: simplify and optimize empty message test
Quoted 7 lines · Elijah Newren
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Quoted 5 lines
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of
time ./t6006-rev-list-format.sh) by about 11% on Linux and by 13% on Mac.
Wow! A good cleanup to include, regardless of other concerns.
Quoted 12 lines
Signed-off-by Elijah Newren <[email protected]>--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)t/t6006-rev-list-format.sh +0
-0@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' '
t/t6006-rev-list-format.sh test_expect_success 'reflog identity' ' · line 508
Quoted 10 lines
t/t6006-rev-list-format.sh +2
-3test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
LGTM.
-Stolee
RFC PATCH 2/5 t3427: accelerate this test by using fast-export and fast-import
In reply to Elijah Newrenfast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):
Linux: 4.305s -> 3.684s (~17% speedup)
Mac: 10.128s -> 7.038s (~30% speedup)
--- t/t3427-rebase-subtree.sh | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-)
t/t3427-rebase-subtree.sh +24 -8
@@ -42,7 +42,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-preserve-merges-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master4"
@@ -53,7 +55,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-preserve-merges-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "files_subtree/master5"
@@ -64,7 +68,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-keep-empty-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -75,7 +81,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-keep-empty-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -86,7 +94,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' ' reset_rebase && git checkout -b rebase-keep-empty-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
@@ -96,7 +106,9 @@ test_expect_failure REBASE_P \ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' reset_rebase && git checkout -b rebase-onto-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -106,7 +118,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' reset_rebase && git checkout -b rebase-onto-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -115,7 +129,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' test_expect_failure 'Rebase -Xsubtree --onto empty commit' ' reset_rebase && git checkout -b rebase-onto-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
Signature · 1 line
2.23.0.5.g775ebaa2a0
Re: [RFC PATCH 2/5] t3427: accelerate this test by using fast-export and fast-import
Quoted 9 lines · Elijah Newren
fast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via
time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):Linux: 4.305s -> 3.684s (~17% speedup) Mac: 10.128s -> 7.038s (~30% speedup)
Again, impressive speedup!
Quoted 10 lines
Signed-off-by Elijah Newren <[email protected]>--- t/t3427-rebase-subtree.sh | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-)t/t3427-rebase-subtree.sh +0
-0
t/t3427-rebase-subtree.sh test_expect_failure REBASE_P \ · line 48
Quoted 8 lines
t/t3427-rebase-subtree.sh +3
-1@@ -42,7 +42,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-preserve-merges-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty &&
There's a lot of repetition in these changes. Is this a good time to introduce a helper method? This trio of commands happens to be difficult to parse, so I'd rather do it just once.
Quoted 80 lines
t/t3427-rebase-subtree.sh +21
-7git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master4"@@ -53,7 +55,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-preserve-merges-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "files_subtree/master5"@@ -64,7 +68,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-keep-empty-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"@@ -75,7 +81,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-keep-empty-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"@@ -86,7 +94,9 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' ' reset_rebase && git checkout -b rebase-keep-empty-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"@@ -96,7 +106,9 @@ test_expect_failure REBASE_P \ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' reset_rebase && git checkout -b rebase-onto-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"@@ -106,7 +118,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' reset_rebase && git checkout -b rebase-onto-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"@@ -115,7 +129,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' test_expect_failure 'Rebase -Xsubtree --onto empty commit' ' reset_rebase && git checkout -b rebase-onto-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
RFC PATCH 3/5 git-sh-i18n: work with external scripts
In reply to Elijah NewrenScripts external to git could source $(git --exec-path)/git-sh-setup (as we document in Documentation/git-sh-setup.txt). This will in turn source git-sh-i18n, which will setup some handy internationalization infrastructure. However, git-sh-i18n hardcodes the TEXTDOMAIN, meaning that anyone using this infrastructure will only get translations that are shipped with git. Allow the external scripts to specify their own translation domain but otherwise use our infrastructure for accessing translations.
My original plan was to have git-filter-branch be the first testcase using this feature, with a goal of minimizing the number of changes that needed to be made to it when I moved it out of git.git. However, I realized after creating this patch that no strings in git-filter-branch are translated. However, the generalization could be useful if we move other tools from git.git to an external location.
--- git-sh-i18n.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
git-sh-i18n.sh +6 -1
@@ -5,7 +5,12 @@ # # Export the TEXTDOMAIN* data that we need for Git -TEXTDOMAIN=git +if test -z "$TEXTDOMAIN_OVERRIDE" +then + TEXTDOMAIN=git +else + TEXTDOMAIN="$TEXTDOMAIN_OVERRIDE" +fi export TEXTDOMAIN if test -z "$GIT_TEXTDOMAINDIR" then
Signature · 1 line
2.23.0.5.g775ebaa2a0
Re: [RFC PATCH 3/5] git-sh-i18n: work with external scripts
Quoted 9 lines · Elijah Newren
Scripts external to git could source $(git --exec-path)/git-sh-setup (as we document in
Documentation/git-sh-setup.txt). This will in turn source git-sh-i18n, which will setup some handy internationalization infrastructure. However, git-sh-i18n hardcodes the TEXTDOMAIN, meaning that anyone using this infrastructure will only get translations that are shipped with git. Allow the external scripts to specify their own translation domain but otherwise use our infrastructure for accessing translations.
Quoted 6 lines
My original plan was to have git-filter-branch be the first testcase using this feature, with a goal of minimizing the number of changes that needed to be made to it when I moved it out of git.git. However, I realized after creating this patch that no strings in git-filter-branch are translated. However, the generalization could be useful if we move other tools from git.git to an external location.
Hm. Maybe extract it from this series for the YAGNI principle?
A noble goal, but in my opinion I'd prefer to see this be exercised by something immediately.
Thanks,
-Stolee
RFC PATCH 4/5 Recommend git-filter-repo instead of git-filter-branch in documentation
In reply to Elijah Newrenfilter-branch suffers from a huge number of pitfalls that can result in incorrectly rewritten history, and many of the problems can easily go undetected until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to the filter-branch manpage about this and recommand that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.
--- Documentation/git-fast-export.txt | 6 ++--- Documentation/git-filter-branch.txt | 42 ++++++++--------------------- Documentation/git-gc.txt | 17 ++++++------ Documentation/git-rebase.txt | 2 +- Documentation/git-replace.txt | 10 +++---- Documentation/git-svn.txt | 4 +-- Documentation/githooks.txt | 7 ++--- contrib/svn-fe/svn-fe.txt | 4 +-- 8 files changed, 36 insertions(+), 56 deletions(-)
Documentation/git-fast-export.txt +3 -3
@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------
Documentation/git-filter-branch.txt +11 -31
54 lines not on this page · raw message
Documentation/git-gc.txt +8 -9
@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is using
Documentation/git-rebase.txt +1 -1
@@ -832,7 +832,7 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like `filter-repo`. The easy case
Documentation/git-replace.txt +5 -5
@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +linkgit:git-filter-repo[1], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a
@@ -148,8 +148,8 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] +linkgit:git-filter-repo[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1]
Documentation/git-svn.txt +2 -2
@@ -769,9 +769,9 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git +reports, and archives. If you plan to eventually migrate from SVN to Git and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows +linkgit:git-filter-repo[1] instead. filter-repo also allows reformatting of metadata for ease-of-reading and rewriting authorship info for non-"svn.authorsFile" users.
Documentation/githooks.txt +4 -3
@@ -425,9 +425,10 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do +not call it!). Its first argument denotes the command it was invoked +by: currently one of `amend` or `rebase`. Further command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in the
contrib/svn-fe/svn-fe.txt +2 -2
@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS
@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
Signature · 1 line
2.23.0.5.g775ebaa2a0
Re: [RFC PATCH 4/5] Recommend git-filter-repo instead of git-filter-branch in documentation
Quoted 79 lines · Elijah Newren
filter-branch suffers from a huge number of pitfalls that can result in incorrectly rewritten history, and many of the problems can easily go undetected until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to the filter-branch manpage about this and recommand that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.Signed-off-by Elijah Newren <[email protected]>--- Documentation/git-fast-export.txt | 6 ++--- Documentation/git-filter-branch.txt | 42 ++++++++--------------------- Documentation/git-gc.txt | 17 ++++++------ Documentation/git-rebase.txt | 2 +- Documentation/git-replace.txt | 10 +++---- Documentation/git-svn.txt | 4 +-- Documentation/githooks.txt | 7 ++--- contrib/svn-fe/svn-fe.txt | 4 +-- 8 files changed, 36 insertions(+), 56 deletions(-)Documentation/git-fast-export.txt +3
-3@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------Documentation/git-filter-branch.txt +0
-0@@ -16,6 +16,17 @@ SYNOPSIS [--original <namespace>] [-d <directory>] [-f | --force] [--state-branch <branch>] [--] [<rev-list options>...]
Documentation/git-filter-branch.txt SYNOPSIS · line 27
Quoted 10 lines
Documentation/git-filter-branch.txt +9
-0+WARNING +------- +'git filter-branch' has a litany of gotchas that can and will cause +history to be rewritten incorrectly (in addition to abysmal +performance). These issues cannot be backward compatibly fixed and as +such, its use is not recommended. Please use an alternative history +filtering tool such as 'git filter-repo'. If you still need to use +'git filter-branch', please carefully read the "Safety" section of +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/
Is it possible to present this URL as a hyperlink with a succinct description? Maybe 'carefully read [the "Safety" section of this message on the Git mailing list](url).' (I'm using Markdown notation here as I don't know the equivalent for our docs.)
Quoted 168 lines
Documentation/git-filter-branch.txt +2
-31+and avoid as many of the pitfalls listed there as reasonably possible. + DESCRIPTION ----------- Lets you rewrite Git revision history by rewriting the branches mentioned@@ -445,37 +456,6 @@ warned. (or if your git-gc is not new enough to support arguments to `--prune`, use `git repack -ad; git prune` instead). -NOTES ------ - -git-filter-branch allows you to make complex shell-scripted rewrites -of your Git history, but you probably don't need this flexibility if -you're simply _removing unwanted data_ like large files or passwords. -For those operations you may want to consider -http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner], -a JVM-based alternative to git-filter-branch, typically at least -10-50x faster for those use-cases, and with quite different -characteristics: - -* Any particular version of a file is cleaned exactly _once_. The BFG, - unlike git-filter-branch, does not give you the opportunity to - handle a file differently based on where or when it was committed - within your history. This constraint gives the core performance - benefit of The BFG, and is well-suited to the task of cleansing bad - data - you don't care _where_ the bad data is, you just want it - _gone_. - -* By default The BFG takes full advantage of multi-core machines, - cleansing commit file-trees in parallel. git-filter-branch cleans - commits sequentially (i.e. in a single-threaded manner), though it - _is_ possible to write filters that include their own parallelism, - in the scripts executed against each commit. - -* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options] - are much more restrictive than git-filter branch, and dedicated just - to the tasks of removing unwanted data- e.g: - `--strip-blobs-bigger-than 1M`. - GIT --- Part of the linkgit:git[1] suiteDocumentation/git-gc.txt +8
-9@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is usingDocumentation/git-rebase.txt +1
-1@@ -832,7 +832,7 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like `filter-repo`. The easy caseDocumentation/git-replace.txt +5
-5@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +linkgit:git-filter-repo[1], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a@@ -148,8 +148,8 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] +linkgit:git-filter-repo[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1]Documentation/git-svn.txt +2
-2@@ -769,9 +769,9 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git +reports, and archives. If you plan to eventually migrate from SVN to Git and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows +linkgit:git-filter-repo[1] instead. filter-repo also allows reformatting of metadata for ease-of-reading and rewriting authorship info for non-"svn.authorsFile" users.Documentation/githooks.txt +4
-3@@ -425,9 +425,10 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do +not call it!). Its first argument denotes the command it was invoked +by: currently one of `amend` or `rebase`. Further command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in thecontrib/svn-fe/svn-fe.txt +2
-2@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
Re: [RFC PATCH 4/5] Recommend git-filter-repo instead of git-filter-branch in documentation
Quoted 17 lines · Derrick Stolee
Documentation/git-filter-branch.txt SYNOPSIS · line 27
Quoted 9 lines · Elijah Newren
Documentation/git-filter-branch.txt +9
-0+WARNING +------- +'git filter-branch' has a litany of gotchas that can and will cause +history to be rewritten incorrectly (in addition to abysmal +performance). These issues cannot be backward compatibly fixed and as +such, its use is not recommended. Please use an alternative history +filtering tool such as 'git filter-repo'. If you still need to use +'git filter-branch', please carefully read the "Safety" section of +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/Is it possible to present this URL as a hyperlink with a succinct description? Maybe 'carefully read [the "Safety" section of this message on the Git mailing list](url).' (I'm using Markdown notation here as I don't know the equivalent for our docs.)
Looks like the syntax is
URL[description]
e.g.
https://public-inbox.org/git/[email protected][the
"Safety" section of yadda yadda]
I'll fix that up.
RFC PATCH 5/5 Remove git-filter-branch, it is now external to git.git
In reply to Elijah Newren--- .gitignore | 1 - Documentation/git-filter-branch.txt | 461 ------------------- Makefile | 1 - command-list.txt | 1 - git-filter-branch.sh | 662 ---------------------------- t/perf/p7000-filter-branch.sh | 24 - t/t7003-filter-branch.sh | 505 --------------------- t/t7009-filter-branch-null-sha1.sh | 55 --- t/t9902-completion.sh | 12 +- 9 files changed, 6 insertions(+), 1716 deletions(-) delete mode 100644 Documentation/git-filter-branch.txt delete mode 100755 git-filter-branch.sh delete mode 100755 t/perf/p7000-filter-branch.sh delete mode 100755 t/t7003-filter-branch.sh delete mode 100755 t/t7009-filter-branch-null-sha1.sh
.gitignore +0 -1
@@ -63,7 +63,6 @@ /git-fast-import /git-fetch /git-fetch-pack -/git-filter-branch /git-fmt-merge-msg /git-for-each-ref /git-format-patch
Documentation/git-filter-branch.txt deleted +0 -461
461 lines not on this page · raw message
Makefile +0 -1
@@ -607,7 +607,6 @@ unexport CDPATH SCRIPT_SH += git-bisect.sh SCRIPT_SH += git-difftool--helper.sh -SCRIPT_SH += git-filter-branch.sh SCRIPT_SH += git-merge-octopus.sh SCRIPT_SH += git-merge-one-file.sh SCRIPT_SH += git-merge-resolve.sh
command-list.txt +0 -1
@@ -90,7 +90,6 @@ git-fast-export ancillarymanipulators git-fast-import ancillarymanipulators git-fetch mainporcelain remote git-fetch-pack synchingrepositories -git-filter-branch ancillarymanipulators git-fmt-merge-msg purehelpers git-for-each-ref plumbinginterrogators git-format-patch mainporcelain
git-filter-branch.sh deleted +0 -662
662 lines not on this page · raw message
t/perf/p7000-filter-branch.sh deleted +0 -24
@@ -1,24 +0,0 @@ -#!/bin/sh - -test_description='performance of filter-branch' -. ./perf-lib.sh - -test_perf_default_repo -test_checkout_worktree - -test_expect_success 'mark bases for tests' ' - git tag -f tip && - git tag -f base HEAD~100 -' - -test_perf 'noop filter' ' - git checkout --detach tip && - git filter-branch -f base..HEAD -' - -test_perf 'noop prune-empty' ' - git checkout --detach tip && - git filter-branch -f --prune-empty base..HEAD -' - -test_done
t/t7003-filter-branch.sh deleted +0 -505
505 lines not on this page · raw message
t/t7009-filter-branch-null-sha1.sh deleted +0 -55
55 lines not on this page · raw message
t/t9902-completion.sh +6 -6
@@ -28,10 +28,10 @@ complete () # # (2) A test makes sure that common subcommands are included in the # completion for "git <TAB>", and a plumbing is excluded. "add", -# "filter-branch" and "ls-files" are listed for this. +# "rebase" and "ls-files" are listed for this. -GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files' -GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch' +GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files' +GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase' . "$GIT_BUILD_DIR/contrib/completion/git-completion.bash"
@@ -1392,12 +1392,12 @@ test_expect_success 'basic' ' # built-in grep -q "^add \$" out && # script - grep -q "^filter-branch \$" out && + grep -q "^rebase \$" out && # plumbing ! grep -q "^ls-files \$" out && - run_completion "git f" && - ! grep -q -v "^f" out + run_completion "git r" && + ! grep -q -v "^r" out ' test_expect_success 'double dash "git" itself' '
Signature · 1 line
2.23.0.5.g775ebaa2a0
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
In reply to Elijah NewrenQuoted 18 lines · Elijah Newren
Following up on the suggestion to make git.git smaller and shed non-core tools, here's an RFC series to do so with git-filter-branch. This series first removes dependencies on git-filter-branch (of which there were very few), and then deletes git-filter-branch itself in the final commit.
I'm more than happy to consider alternate places for the filter-branch history (I had considered just merging it in with git-filter-repo), but for now I just made it available here:
https://github.com/newren/git-filter-branchThe rewrite above contains the history of the files deleted in Patch 5, plus a one-time copy of relevant build files (Makefiles,
test-lib.sh, etc. -- I didn't want the whole history of these), and then touchups to streamline the build files and make them all work in this standalone repo.
Quoted 4 lines
Some highlevel notes on the patches:
* Patches 1&2: are good cleanups & performance wins regardless of whether the rest of the series is taken
I agree! These are great. I just had a nit about extracting a helper instead of copy-pasting the same three lines in multiple tests.
Quoted 3 lines
* Patch 3: an attempt to improve i18n situation for external scripts, but discovered to not be necessary/useful for git-filter-branch specifically
I'm not sure this is super-important now, but could be saved for a later date, when it is important.
Quoted 10 lines
* Patch 4:
* If we are good with deleting git-filter-branch now and just noting it in the release notes, then patch 4 could be simplified; there's no need to update git-filter-branch.txt in that case.
* If, however, we want to do some external messaging for an additional release cycle or two before moving git-filter-branch out of git.git, this patch will help us until then to at least avoid recommending a tool which will likely mangle user's data in unexpected ways. But it'd be really helpful if folks could review and opine on the BFG stuff if so.
I think this is a good step, and should be taken even if we never plan to take Patch 5.
Quoted 2 lines
* Patch 5: actually deletes git-filter-branch, its tests, and documentation.
This is the one where others need to chime in with opinions. I think this one can only be taken if we have a concrete plan about how to support the tool _somehow_, even if it is "go download the script from this place; it may have broken since we last tested it."
Yes, we want to strongly recommend that people use newer, better tools. That's not always something users can accept. Having the tool live somewhere that is accessible can appease some users for a while, and it can decay and die a slow death there.
Thanks,
-Stolee
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Quoted 57 lines · Derrick Stolee
Quoted 18 lines · Elijah Newren
Following up on the suggestion to make git.git smaller and shed non-core tools, here's an RFC series to do so with git-filter-branch. This series first removes dependencies on git-filter-branch (of which there were very few), and then deletes git-filter-branch itself in the final commit.
I'm more than happy to consider alternate places for the filter-branch history (I had considered just merging it in with git-filter-repo), but for now I just made it available here:
https://github.com/newren/git-filter-branchThe rewrite above contains the history of the files deleted in Patch 5, plus a one-time copy of relevant build files (Makefiles,
test-lib.sh, etc. -- I didn't want the whole history of these), and then touchups to streamline the build files and make them all work in this standalone repo.Quoted 4 lines
Some highlevel notes on the patches:
* Patches 1&2: are good cleanups & performance wins regardless of whether the rest of the series is taken
I agree! These are great. I just had a nit about extracting a helper instead of copy-pasting the same three lines in multiple tests.
Quoted 3 lines
* Patch 3: an attempt to improve i18n situation for external scripts, but discovered to not be necessary/useful for git-filter-branch specifically
I'm not sure this is super-important now, but could be saved for a later date, when it is important.
Quoted 10 lines
* Patch 4:
* If we are good with deleting git-filter-branch now and just noting it in the release notes, then patch 4 could be simplified; there's no need to update git-filter-branch.txt in that case.
* If, however, we want to do some external messaging for an additional release cycle or two before moving git-filter-branch out of git.git, this patch will help us until then to at least avoid recommending a tool which will likely mangle user's data in unexpected ways. But it'd be really helpful if folks could review and opine on the BFG stuff if so.I think this is a good step, and should be taken even if we never plan to take Patch 5.
Quoted 2 lines
* Patch 5: actually deletes git-filter-branch, its tests, and documentation.
This is the one where others need to chime in with opinions. I think this one can only be taken if we have a concrete plan about how to support the tool _somehow_, even if it is "go download the script from this place; it may have broken since we last tested it."
Quoted 4 lines
Yes, we want to strongly recommend that people use newer, better tools. That's not always something users can accept. Having the tool live somewhere that is accessible can appease some users for a while, and it can decay and die a slow death there.
Perhaps I should add some more words about the separate repo I created; even though it wasn't one of the five patches in this series it actually represented the lionshare of the work before I submitted this. Anyway, it has a Makefile which supports the normal 'test', 'doc', 'clean', 'install' (and variants), and 'dist' (and variants). The 'test' target will run all the filter-branch tests taken from the git.git testsuite (i.e. t7003 and t7009) without requiring a version of git built inside that separate repo, 'doc' will build both html and manpages, etc. It doesn't look at any config.mak* files and has stripped out lots of stuff from the main repo, but it's relatively minimal and self-contained beyond an assumption that a normal copy of git has been installed somehow already. Given how infrequently filter-branch has needed fixes in the past, and the fact that it is pretty good about relying on plumbing rather than porcelain, I suspect it might actually be a pretty light maintenance load to keep it running for a good long time.
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
In reply to Elijah NewrenQuoted 4 lines · Elijah Newren
Some highlevel notes on the patches:
* Patches 1&2: are good cleanups & performance wins regardless of whether the rest of the series is taken
Agreed. Though weren't we moving away from pipes in tests because failures could go unnoticed? (I haven't been paying too much attention, though)
Quoted 2 lines
* Patch 5: actually deletes git-filter-branch, its tests, and documentation.
Given how long we've had git-filter-branch, I suggest we keep it around but add a warning at runtime notifying users of it's impending removal. Such warning should remain for >=10 years or whatever an distro support cycle is, nowadays.
And there should probably be a way to disable the warning via git-config if it's too annoying.
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
But I agree that filter-branch isn't useful and certainly shouldn't be encouraged/promoted.
Yet there's probably still users which ARE happy with it, that will never hit the edge cases and problems it poses; and will never read release notes. And said users are probably getting git from a slow-moving distro, so it'd be a disservice to them if they lost a tool they depend on without any warning.
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Eric Wong <[email protected]> writes:
[...]
Quoted 4 lines
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
Nah, discovering it is simple. Just Google for "git change author". That eventually leads to a script that uses "git filter-branch --env-filter" to get the job done, and I'm afraid it is spread all over the world.
See, e.g.:
https://help.github.com/en/articles/changing-author-info
Quoted 2 lines
But I agree that filter-branch isn't useful and certainly shouldn't be encouraged/promoted.
Well, is there more suitable way to change author for a (large) set of commits then?
Quoted 5 lines
Yet there's probably still users which ARE happy with it, that will never hit the edge cases and problems it poses; and will never read release notes. And said users are probably getting git from a slow-moving distro, so it'd be a disservice to them if they lost a tool they depend on without any warning.
Personally, I'm far from happy with it, but I have no clue how to substitute it in the job above. Anybody?
-- Sergey
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Quoted 11 lines · Sergey Organov
Eric Wong <[email protected]> writes:
[...]
Quoted 4 lines
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
Quoted 7 lines
Nah, discovering it is simple. Just Google for "git change author". That eventually leads to a script that uses "git filter-branch --env-filter" to get the job done, and I'm afraid it is spread all over the world.
See, e.g.:
Side note: Is the goal to "fix names and email addresses in this repository"? If so, this guide fails: it doesn't update tagger names or email addresses. Indeed, filter-branch doesn't provide a way to do that. (Not to mention other problems like not updating references to commit hashes in commit messages when it busy rewriting everything.)
Quoted 5 lines
Quoted 2 lines
But I agree that filter-branch isn't useful and certainly shouldn't be encouraged/promoted.
Well, is there more suitable way to change author for a (large) set of commits then?
I would say yes, use git filter-repo (note that this thread started with me proposing filter-repo for inclusion in git.git -- and getting suggestions that we should remove stuff instead of adding more stuff). I'm biased, but I think it's much better at this particular job as well:
You can create a mailmap file and pass it to the --mailmap option to git-filter-repo.
Or, if you prefer (perhaps you don't like git's mailmap format as used by shortlog and now log, or perhaps you really want to be able to do regex replacement or something), you can use the --name-callback or --email-callback to work on those fields more directly.
Or, if you prefer (e.g. you want to handle author vs. committer vs. tagger differently), you can use the --commit-callback and --tag-callback filters.
As an added bonus, filter-repo will also perform the rewrite far faster than filter-branch (and rewrite commit hashes in commit messages as alluded to above).
Quoted 8 lines
Quoted 5 lines
Yet there's probably still users which ARE happy with it, that will never hit the edge cases and problems it poses; and will never read release notes. And said users are probably getting git from a slow-moving distro, so it'd be a disservice to them if they lost a tool they depend on without any warning.
Personally, I'm far from happy with it, but I have no clue how to substitute it in the job above. Anybody?
The start of this thread where I proposed git filter-repo for inclusion in git[1] had links to documentation and comparisons to other tools and such. You may find those links helpful; if not, let me know what needs to be fixed in the documentation.
Elijah
[1] https://public-inbox.org/git/CABPp-BEr8LVM+yWTbi76hAq7Moe1hyp2xqxXfgVV4_teh_9skA@mail.gmail.com/
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Quoted 18 lines · Elijah Newren
Quoted 17 lines · Sergey Organov
Eric Wong <[email protected]> writes:
[...]
Quoted 4 lines
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
Nah, discovering it is simple. Just Google for "git change author". That eventually leads to a script that uses "git filter-branch --env-filter" to get the job done, and I'm afraid it is spread all over the world.
See, e.g.:
Quoted 7 lines
Side note: Is the goal to "fix names and email addresses in this repository"? If so, this guide fails: it doesn't update tagger names or email addresses. Indeed, filter-branch doesn't provide a way to do that. (Not to mention other problems like not updating references to commit hashes in commit messages when it busy rewriting everything.)
No. Maybe the original goal was like that, by I, personally, use modified version of this to change my "Author" credentials from "internal" to "public" in branches that I'm going to send upstream, so the actual aim is to change e-mail of particular Author from a@b to c@d in all the commits in a (feature) branch.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
But I agree that filter-branch isn't useful and certainly shouldn't be encouraged/promoted.
Quoted 8 lines
Quoted 2 lines
Well, is there more suitable way to change author for a (large) set of commits then?
I would say yes, use git filter-repo (note that this thread started with me proposing filter-repo for inclusion in git.git -- and getting suggestions that we should remove stuff instead of adding more stuff). I'm biased, but I think it's much better at this particular job as well:
Well, I don't want to change the entire repo, and I don't immediately see how to do it with git filter-repo. Is it at all possible?
Quoted 8 lines
You can create a mailmap file and pass it to the --mailmap option to git-filter-repo.
Or, if you prefer (perhaps you don't like git's mailmap format as used by shortlog and now log, or perhaps you really want to be able to do regex replacement or something), you can use the --name-callback or --email-callback to work on those fields more directly.
Quoted 7 lines
Or, if you prefer (e.g. you want to handle author vs. committer vs. tagger differently), you can use the --commit-callback and --tag-callback filters.
As an added bonus, filter-repo will also perform the rewrite far faster than filter-branch (and rewrite commit hashes in commit messages as alluded to above).
These things are nice to have indeed, but it always changes the entire repo, right? If so, it's not a suitable substitute for git-filter-branch for particular job at hand.
Actually, I'd rather expect some support for this in "git rebase", being git history editing/reshaping tool, but it looks like it only has it in the form that is very difficult to automate.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Yet there's probably still users which ARE happy with it, that will never hit the edge cases and problems it poses; and will never read release notes. And said users are probably getting git from a slow-moving distro, so it'd be a disservice to them if they lost a tool they depend on without any warning.
Quoted 7 lines
Quoted 2 lines
Personally, I'm far from happy with it, but I have no clue how to substitute it in the job above. Anybody?
The start of this thread where I proposed git filter-repo for inclusion in git[1] had links to documentation and comparisons to other tools and such. You may find those links helpful; if not, let me know what needs to be fixed in the documentation.
Thank you for the references, I find it a very nice tool to have!
Pity it's not an entire substitute for git filter-branch.
-- Sergey
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Hi Sergey,
Quoted 29 lines · Sergey Organov
Quoted 18 lines · Elijah Newren
Quoted 17 lines · Sergey Organov
Eric Wong <[email protected]> writes:
[...]
Quoted 4 lines
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
Nah, discovering it is simple. Just Google for "git change author". That eventually leads to a script that uses "git filter-branch --env-filter" to get the job done, and I'm afraid it is spread all over the world.
See, e.g.:
Quoted 7 lines
Side note: Is the goal to "fix names and email addresses in this repository"? If so, this guide fails: it doesn't update tagger names or email addresses. Indeed, filter-branch doesn't provide a way to do that. (Not to mention other problems like not updating references to commit hashes in commit messages when it busy rewriting everything.)
Quoted 5 lines
No. Maybe the original goal was like that, by I, personally, use modified version of this to change my "Author" credentials from "internal" to "public" in branches that I'm going to send upstream, so the actual aim is to change e-mail of particular Author from a@b to c@d in all the commits in a (feature) branch.
There's an interesting usecase I hadn't heard of or thought of before. Quick question to see if I'm understanding correctly: "all commits in a branch" or "all commits *unique* to a branch"?
(Perhaps the only commits with the author you want to change are among the commits that are unique to that branch and so the distinction doesn't matter, but it wasn't clear from the description.)
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
But I agree that filter-branch isn't useful and certainly shouldn't be encouraged/promoted.
Well, is there more suitable way to change author for a (large) set of commits then?
Quoted 8 lines
Quoted 5 lines
I would say yes, use git filter-repo (note that this thread started with me proposing filter-repo for inclusion in git.git -- and getting suggestions that we should remove stuff instead of adding more stuff). I'm biased, but I think it's much better at this particular job as well:
Well, I don't want to change the entire repo, and I don't immediately see how to do it with git filter-repo. Is it at all possible?
Yes, it is possible. filter-repo has a hidden --refs argument defaulting to --all; you could instead set it to e.g. origin/master..master.
--refs is the only hidden option in filter-repo. I know it may look funny that I spent a bunch of effort to create the --reference-excluded-parents option to fast-export explicitly so that it would be possible to do partial history rewrites like this, and then to hide and avoid documenting this option (though I did hint that it existed in the documentation if you search for "Partial-repo filtering"), but there was a few reasons for this:
* mixing old and new history for most rewrites that
filter-branch/filter-repo/bfg/etc are used for can really mess things up and make it hard to recover from. I don't like trying to clean up repos with accidental duplicate copies of most commits in the repo, and I suspect others like it even less. So, anything that makes it easier to make such mistakes needs to have a really good rationale in order for me to expose it.
* The only usecases I knew of for partial repo filtering prior to
your email were (1) side-stepping insanely slow execution time of poor filtering tools like filter-branch, and (2) performing operations better suited to git-rebase anyway (e.g. the --signoff option to rebase did not exist once upon a time and so folks could have used filter-branch to fake it, but using rebase is the better way to make this change). And, even after your email, I'm not sure that has changed though, as noted below.
Quoted 12 lines
Quoted 12 lines
You can create a mailmap file and pass it to the --mailmap option to git-filter-repo.
Or, if you prefer (perhaps you don't like git's mailmap format as used by shortlog and now log, or perhaps you really want to be able to do regex replacement or something), you can use the --name-callback or --email-callback to work on those fields more directly.
Or, if you prefer (e.g. you want to handle author vs. committer vs. tagger differently), you can use the --commit-callback and --tag-callback filters.
Quoted 7 lines
Quoted 3 lines
As an added bonus, filter-repo will also perform the rewrite far faster than filter-branch (and rewrite commit hashes in commit messages as alluded to above).
These things are nice to have indeed, but it always changes the entire repo, right? If so, it's not a suitable substitute for git-filter-branch for particular job at hand.
It *defaults* to changing the entire repo. You may also be interested to note that two of the demos in contrib/filter-repo-demos[1] explicitly make use of partial history filtering -- one of the two being the filter-branch reimplementation (I created a script that reimplemented filter-branch on top of filter-repo and made it accept the exact same flags as filter-branch. That script passes all the filter-branch regression tests from git.git and it's much faster than filter-branch, though it's still glacially slow compared to filter-repo, and has all the safety problems that filter-branch does).
[1] https://github.com/newren/git-filter-repo/tree/master/contrib/filter-repo-demos
Quoted 3 lines
Actually, I'd rather expect some support for this in "git rebase", being git history editing/reshaping tool, but it looks like it only has it in the form that is very difficult to automate.
I agree that git rebase would be the better choice here; I typically feel it's the better choice for rewrites of recent history. I think it provides just what you need:
git rebase --exec="git commit --amend --reset-author -C HEAD" $UPSTREAM
(Assuming, of course, that you've either set the right environment variables or set user.name and user.email to the new values you want so that commit's --reset-author flag can reset to the *new* author.)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Yet there's probably still users which ARE happy with it, that will never hit the edge cases and problems it poses; and will never read release notes. And said users are probably getting git from a slow-moving distro, so it'd be a disservice to them if they lost a tool they depend on without any warning.
Personally, I'm far from happy with it, but I have no clue how to substitute it in the job above. Anybody?
Quoted 8 lines
Quoted 4 lines
The start of this thread where I proposed git filter-repo for inclusion in git[1] had links to documentation and comparisons to other tools and such. You may find those links helpful; if not, let me know what needs to be fixed in the documentation.
Thank you for the references, I find it a very nice tool to have!
Pity it's not an entire substitute for git filter-branch.
Au contraire, I believe it is. :-)
Thanks for the interesting usecase. It sounds like we both think this one happens to be better solved by rebase, and the command snippet I provided above should show you to use rebase to solve it. However, if you come up with any others where partial repo filtering makes sense, I'm always willing to reconsider my decision to make the --refs argument hidden; it may just mean adding more warnings, but it might also involve changing other defaults (e.g. the automatic repacking/pruning). I'd need concrete usecases to know for sure how I'd want to handle it.
Hope that helps,
Elijah
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Hi Elijah,
Quoted 12 lines · Elijah Newren
Hi Sergey,
Quoted 9 lines · Sergey Organov
Quoted 6 lines · Elijah Newren
Quoted 5 lines · Sergey Organov
Eric Wong <[email protected]> writes:
[...]
[...]
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Side note: Is the goal to "fix names and email addresses in this repository"? If so, this guide fails: it doesn't update tagger names or email addresses. Indeed, filter-branch doesn't provide a way to do that. (Not to mention other problems like not updating references to commit hashes in commit messages when it busy rewriting everything.)
Quoted 5 lines
No. Maybe the original goal was like that, by I, personally, use modified version of this to change my "Author" credentials from "internal" to "public" in branches that I'm going to send upstream, so the actual aim is to change e-mail of particular Author from a@b to c@d in all the commits in a (feature) branch.
Quoted 7 lines
There's an interesting usecase I hadn't heard of or thought of before. Quick question to see if I'm understanding correctly: "all commits in a branch" or "all commits *unique* to a branch"?
(Perhaps the only commits with the author you want to change are among the commits that are unique to that branch and so the distinction doesn't matter, but it wasn't clear from the description.)
Yes, this is exactly the case for me, as I'm changing entirely linear topic branch that is going to become patch series to send out. No complications.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines
Quoted 3 lines
But I agree that filter-branch isn't useful and certainly shouldn't be encouraged/promoted.
Well, is there more suitable way to change author for a (large) set of commits then?
I would say yes, use git filter-repo (note that this thread started with me proposing filter-repo for inclusion in git.git -- and getting suggestions that we should remove stuff instead of adding more stuff). I'm biased, but I think it's much better at this particular job as well:
Quoted 6 lines
Quoted 2 lines
Well, I don't want to change the entire repo, and I don't immediately see how to do it with git filter-repo. Is it at all possible?
Yes, it is possible. filter-repo has a hidden --refs argument defaulting to --all; you could instead set it to e.g. origin/master..master.
Cool!
Quoted 9 lines
--refs is the only hidden option in filter-repo. I know it may look funny that I spent a bunch of effort to create the --reference-excluded-parents option to fast-export explicitly so that it would be possible to do partial history rewrites like this, and then to hide and avoid documenting this option (though I did hint that it existed in the documentation if you search for "Partial-repo filtering"), but there was a few reasons for this:
Quoted 15 lines
* mixing old and new history for most rewrites that
filter-branch/filter-repo/bfg/etc are used for can really mess things up and make it hard to recover from. I don't like trying to clean up repos with accidental duplicate copies of most commits in the repo, and I suspect others like it even less. So, anything that makes it easier to make such mistakes needs to have a really good rationale in order for me to expose it.
* The only usecases I knew of for partial repo filtering prior to
your email were (1) side-stepping insanely slow execution time of poor filtering tools like filter-branch, and (2) performing operations better suited to git-rebase anyway (e.g. the --signoff option to rebase did not exist once upon a time and so folks could have used filter-branch to fake it, but using rebase is the better way to make this change). And, even after your email, I'm not sure that has changed though, as noted below.
Yeah, I share your worries.
[...]
Quoted 8 lines
Quoted 3 lines
Actually, I'd rather expect some support for this in "git rebase", being git history editing/reshaping tool, but it looks like it only has it in the form that is very difficult to automate.
I agree that git rebase would be the better choice here; I typically feel it's the better choice for rewrites of recent history. I think it provides just what you need:
Quoted 5 lines
git rebase --exec="git commit --amend --reset-author -C HEAD" $UPSTREAM(Assuming, of course, that you've either set the right environment variables or set user.name and user.email to the new values you want so that commit's --reset-author flag can reset to the *new* author.)
This should do the trick for me most of times, thanks a lot for the clue!
However, the script that I'm using doesn't change _all_ the authors, it only changes those that match particular specific author specified in the script. I didn't yet actually need this feature, but I can well imagine it's probable that I will have commits by other author(s) in the branch and I won't want to attribute their job to myself.
Hmm... That said, using the generic "--exec" to "git rebase" I could probably come-up with a script that will check the Author of the latest commit and will choose to either rewrite it or not. Nothing terribly complex.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 9 lines
Quoted 6 lines
Yet there's probably still users which ARE happy with it, that will never hit the edge cases and problems it poses; and will never read release notes. And said users are probably getting git from a slow-moving distro, so it'd be a disservice to them if they lost a tool they depend on without any warning.
Personally, I'm far from happy with it, but I have no clue how to substitute it in the job above. Anybody?
The start of this thread where I proposed git filter-repo for inclusion in git[1] had links to documentation and comparisons to other tools and such. You may find those links helpful; if not, let me know what needs to be fixed in the documentation.
Quoted 5 lines
Quoted 3 lines
Thank you for the references, I find it a very nice tool to have!
Pity it's not an entire substitute for git filter-branch.
Au contraire, I believe it is. :-)
I take your word for it :-)
Quoted 10 lines
Thanks for the interesting usecase. It sounds like we both think this one happens to be better solved by rebase, and the command snippet I provided above should show you to use rebase to solve it. However, if you come up with any others where partial repo filtering makes sense, I'm always willing to reconsider my decision to make the --refs argument hidden; it may just mean adding more warnings, but it might also involve changing other defaults (e.g. the automatic repacking/pruning). I'd need concrete usecases to know for sure how I'd want to handle it.
OK, thanks a lot! Doesn't seem to be necessary for now due to the rebase trick you've suggested.
Quoted 2 lines
Hope that helps,
Sure it does!
-- Sergey
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
In reply to Elijah NewrenHi Elijah,
Quoted 32 lines · Elijah Newren
Hi Sergey,
Quoted 29 lines · Sergey Organov
Quoted 25 lines · Elijah Newren
Quoted 11 lines · Sergey Organov
Eric Wong <[email protected]> writes:
[...]
Quoted 4 lines
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
Quoted 7 lines
Nah, discovering it is simple. Just Google for "git change author". That eventually leads to a script that uses "git filter-branch --env-filter" to get the job done, and I'm afraid it is spread all over the world.
See, e.g.:
Side note: Is the goal to "fix names and email addresses in this repository"? If so, this guide fails: it doesn't update tagger names or email addresses. Indeed, filter-branch doesn't provide a way to do that. (Not to mention other problems like not updating references to commit hashes in commit messages when it busy rewriting everything.)
Quoted 7 lines
Quoted 5 lines
No. Maybe the original goal was like that, by I, personally, use modified version of this to change my "Author" credentials from "internal" to "public" in branches that I'm going to send upstream, so the actual aim is to change e-mail of particular Author from a@b to c@d in all the commits in a (feature) branch.
There's an interesting usecase I hadn't heard of or thought of before.
I'll throw in another use case that's kinda related: extracting the history of one file (or subdirectory).
In my most recent instance of this, I wanted to publish the script I used to use for submitting patch series to the Git mailing list, maintaining tags for iterations and generating cover letters from branch descriptions and interdiffs (this script eventually became GitGitGadget,
https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0
demonstrates this).
To do that, I ran a git filter-branch in the repository where I track all the scripts I deem unsuitable for public consumption, to remove all files but mail-patch-series.sh, then pushed it to
https://github.com/dscho/mail-patch-series
Please note that most crucially, I wanted to rewrite a newly-created branch, and only that branch.
Could I have done the same using git fast-export, filtering the output with a Perl script, then passing it to git fast-import? Sure, I was really tempted to do that. In the end, it took less of _my_ time to just let git filter-branch do its work with a not-too-complicated index filter.
In another instance, a long, long time ago, I needed to restart a repository which had included way too many files for its own good, then rename the old repository and start with a fresh master that contained but a single commit whose tree was identical to the previous master's tip commit. I simply grafted that commit, ran git filter-branch and had precisely what I needed.
I would be _delighted_ if these kinds of use case (rewriting a branch, or even just a commit range) became more of a first-class citizen with git filter-repo.
Thanks,
Dscho
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Hi Dscho,
Quoted 44 lines · Johannes Schindelin
Hi Elijah,
Quoted 38 lines · Elijah Newren
Hi Sergey,
Quoted 34 lines · Sergey Organov
Quoted 18 lines · Elijah Newren
Quoted 17 lines · Sergey Organov
Eric Wong <[email protected]> writes:
[...]
Quoted 4 lines
AFAIK, filter-branch is not causing support headaches for any git developers today. With so many commands in git, it's unlikely newbies will ever get around to discover it :) So I think think we should be in any rush to remove it.
Nah, discovering it is simple. Just Google for "git change author". That eventually leads to a script that uses "git filter-branch --env-filter" to get the job done, and I'm afraid it is spread all over the world.
See, e.g.:
Quoted 7 lines
Side note: Is the goal to "fix names and email addresses in this repository"? If so, this guide fails: it doesn't update tagger names or email addresses. Indeed, filter-branch doesn't provide a way to do that. (Not to mention other problems like not updating references to commit hashes in commit messages when it busy rewriting everything.)
No. Maybe the original goal was like that, by I, personally, use modified version of this to change my "Author" credentials from "internal" to "public" in branches that I'm going to send upstream, so the actual aim is to change e-mail of particular Author from a@b to c@d in all the commits in a (feature) branch.
Quoted 4 lines
Quoted 1 line
There's an interesting usecase I hadn't heard of or thought of before.
I'll throw in another use case that's kinda related: extracting the history of one file (or subdirectory).
Thanks for sending these along! I do have some comments, and a bunch of questions...
Quoted 12 lines
In my most recent instance of this, I wanted to publish the script I used to use for submitting patch series to the Git mailing list, maintaining tags for iterations and generating cover letters from branch descriptions and interdiffs (this script eventually became GitGitGadget,
https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0
demonstrates this).To do that, I ran a
git filter-branchin the repository where I track all the scripts I deem unsuitable for public consumption, to remove all files butmail-patch-series.sh, then pushed it to
https://github.com/dscho/mail-patch-series
Quoted 8 lines
Please note that most crucially, I wanted to rewrite a newly-created branch, and only that branch.
Could I have done the same using
git fast-export, filtering the output with a Perl script, then passing it togit fast-import? Sure, I was really tempted to do that. In the end, it took less of _my_ time to just letgit filter-branchdo its work with a not-too-complicated index filter.
Why a perl script? Shouldn't
git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quiet
do the trick? And it's probably simpler and shorter than the index filter you used.
That said, yeah it'd be nice to get automatic rewriting of commit hashes in commit messages and other niceties from filter-repo (e.g. future automatic reattaching of notes to the rewritten commits). Some questions:
* What's the backup strategy in case you specify the wrong filters
(e.g. you have a typo in the pathnames)? filter-repo encourages folks to make a clone and then filter the fresh clone, because if anything goes awry, you can just delete and restart. (I am heavily opposed to the refs/original/ backup mechanism used by filter-branch, for multiple reasons.) Is your safety stance just "If I mess up it's my own fault; do the rewrite?" Or are you okay with cloning before filtering?
* If you're okay with cloning before filtering...then is there an
issue with rewriting all branches, and just pushing the one you need? (Is there an issue with "this branch is small, the others are huge, and filter-branch is slow -- so rewriting one branch saves me lots of time"? Or are there other issues at play too?)
* What if the user has auxiliary information for the branch in other
refs? For example, git-notes pointing at any of the commits, or tags in the history of the branch that might be relevant, or perhaps even replace refs in combination with GIT_NO_REPLACE_OBJECTS=1? Is this an "I don't care, toss that stuff and just rewrite just this branch?"
* filter-repo by default creates new replace references so that you
can refer to new commit IDs using old (unabbreviated) commit IDs. Would that be considered helpful for this usecase? unhelpful? irrelevant, since you'll just push the branch you want somewhere and nuke the temporary clone?
I'm not by any means ruling out the possibility of documenting --refs and adjusting the defaults when it is used so the user can just run something like
git filter-repo --path $PATH --refs $MYBRANCH
but I feel like I need to understand answers to questions like the above ones so that I can know how to phrase warnings and adjust defaults and update the documentation.
Quoted 6 lines
In another instance, a long, long time ago, I needed to restart a repository which had included way too many files for its own good, then rename the old repository and start with a fresh
masterthat contained but a single commit whose tree was identical to the previousmaster's tip commit. I simply grafted that commit, rangit filter-branchand had precisely what I needed.
filter-repo supports grafts and replace objects, the same as filter-branch. (Although, technically, I didn't have to do a thing to support it; fast-export does the special handling of rewriting based on grafts and replace objects.) So, I'd say this is fully supported.
Side question: the git-replace documents suggest that the graft file is deprecated. Are there any timeframes or plans for phasing out beyond the git-replace manpage existing? Should I avoid documenting the graft file support in filter-repo? Should I include examples using not just git-replace but also using the graft file?
Quoted 3 lines
I would be _delighted_ if these kinds of use case (rewriting a branch, or even just a commit range) became more of a first-class citizen with
git filter-repo.
I've got all the pieces for supporting a single branch or a commit range (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23 mybranch'), but the defaults (error out unless in a bare repo, move refs/remotes/origin/* to refs/heads/*, disconnect origin remote, expire reflogs & repack & prune, create new replace references so folks can access new commits using old commit IDs) may be somewhat friction-filled for this usecase. Those defaults other than the new replace refs happen to all be turned off with the combination of --force and --target, so, assuming turning them off is what you need, you could cheat and just specify 'git filter-repo --force --target . --refs $MYBRANCH' today and perhaps get what you want, but that's a really non-intuitive command line that is way too ugly to recommend. And I don't want to tie myself to '--target .' being the magic sauce in the future either.
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Hi Elijah,
Quoted 25 lines · Elijah Newren
Quoted 13 lines · Johannes Schindelin
[...]
In my most recent instance of this, I wanted to publish the script I used to use for submitting patch series to the Git mailing list, maintaining tags for iterations and generating cover letters from branch descriptions and interdiffs (this script eventually became GitGitGadget,
https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0
demonstrates this).To do that, I ran a
git filter-branchin the repository where I track all the scripts I deem unsuitable for public consumption, to remove all files butmail-patch-series.sh, then pushed it to
https://github.com/dscho/mail-patch-seriesQuoted 8 lines
Please note that most crucially, I wanted to rewrite a newly-created branch, and only that branch.
Could I have done the same using
git fast-export, filtering the output with a Perl script, then passing it togit fast-import? Sure, I was really tempted to do that. In the end, it took less of _my_ time to just letgit filter-branchdo its work with a not-too-complicated index filter.
Quoted 4 lines
Why a perl script? Shouldn't
git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quietdo the trick? And it's probably simpler and shorter than the index filter you used.
Does that not keep the full $PATH? I wanted the resulting branch to have the file in the top-level directory.
Quoted 5 lines
That said, yeah it'd be nice to get automatic rewriting of commit hashes in commit messages and other niceties from filter-repo (e.g. future automatic reattaching of notes to the rewritten commits). Some questions:
Quoted 8 lines
* What's the backup strategy in case you specify the wrong filters
(e.g. you have a typo in the pathnames)? filter-repo encourages folks to make a clone and then filter the fresh clone, because if anything goes awry, you can just delete and restart. (I am heavily opposed to the refs/original/ backup mechanism used by filter-branch, for multiple reasons.) Is your safety stance just "If I mess up it's my own fault; do the rewrite?" Or are you okay with cloning before filtering?
Please note that the refs/original/ refs should not have been written at all anymore, not after reflogs were introduced.
Incidentally, that is my answer to your question: the reflog is my backup.
Quoted 5 lines
* If you're okay with cloning before filtering...then is there an
issue with rewriting all branches, and just pushing the one you need? (Is there an issue with "this branch is small, the others are huge, and filter-branch is slow -- so rewriting one branch saves me lots of time"? Or are there other issues at play too?)
I am not okay with cloning before filtering.
First of all, it is wasteful.
Second of all, in my case it would have been *particularly* wasteful because the repository in question also has quite a few quite large blobs (hysterical raisins, don't ask).
Quoted 5 lines
* What if the user has auxiliary information for the branch in other
refs? For example, git-notes pointing at any of the commits, or tags in the history of the branch that might be relevant, or perhaps even replace refs in combination with GIT_NO_REPLACE_OBJECTS=1? Is this an "I don't care, toss that stuff and just rewrite just this branch?"
In my case: there are no notes. The only time when I make heavy use of notes is in GitGitGadget. I don't use that feature otherwise.
Quoted 5 lines
* filter-repo by default creates new replace references so that you
can refer to new commit IDs using old (unabbreviated) commit IDs. Would that be considered helpful for this usecase? unhelpful? irrelevant, since you'll just push the branch you want somewhere and nuke the temporary clone?
I definitely did not need that mapping in all of my git filter-branch use cases.
Of course, I can see how it can come in handy in other circumstances, just not in the ones I experienced so far.
Quoted 7 lines
I'm not by any means ruling out the possibility of documenting --refs and adjusting the defaults when it is used so the user can just run something like
git filter-repo --path $PATH --refs $MYBRANCH
but I feel like I need to understand answers to questions like the above ones so that I can know how to phrase warnings and adjust defaults and update the documentation.
In all the scenarios where I used git filter-branch (some dozen per year, so not all *that* many), I needed to rewrite one particular branch, typically a freshly-created one. I never, ever ever needed to rewrite all the refs in the repository. Not once ;-)
Quoted 12 lines
Quoted 6 lines
In another instance, a long, long time ago, I needed to restart a repository which had included way too many files for its own good, then rename the old repository and start with a fresh
masterthat contained but a single commit whose tree was identical to the previousmaster's tip commit. I simply grafted that commit, rangit filter-branchand had precisely what I needed.filter-repo supports grafts and replace objects, the same as filter-branch. (Although, technically, I didn't have to do a thing to support it; fast-export does the special handling of rewriting based on grafts and replace objects.) So, I'd say this is fully supported.
Quoted 5 lines
Side question: the git-replace documents suggest that the graft file is deprecated. Are there any timeframes or plans for phasing out beyond the git-replace manpage existing? Should I avoid documenting the graft file support in filter-repo? Should I include examples using not just git-replace but also using the graft file?
I had meant to prepare a patch series to remove grafts support that Junio could carry in pu until the time he considers it appropriate to merge to master, but it seems that this task fell under the rag.
The deprecation itself has been introduced in tags/v2.18.0-rc0~54^2~4, i.e. it is official as of Git v2.18.0, which was released in mid-June last year.
My personal gut feeling is that we should let it simmer for another year before removing support for the grafts file (and we may want to update the label "grafted" when git log shows a shallow commit before we remove that support for grafts).
So I'll not work on that patch for now.
Quoted 4 lines
Quoted 3 lines
I would be _delighted_ if these kinds of use case (rewriting a branch, or even just a commit range) became more of a first-class citizen with
git filter-repo.
Quoted 14 lines
I've got all the pieces for supporting a single branch or a commit range (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23 mybranch'), but the defaults (error out unless in a bare repo, move refs/remotes/origin/* to refs/heads/*, disconnect origin remote, expire reflogs & repack & prune, create new replace references so folks can access new commits using old commit IDs) may be somewhat friction-filled for this usecase. Those defaults other than the new replace refs happen to all be turned off with the combination of --force and --target, so, assuming turning them off is what you need, you could cheat and just specify 'git filter-repo --force --target . --refs $MYBRANCH' today and perhaps get what you want, but that's a really non-intuitive command line that is way too ugly to recommend. And I don't want to tie myself to '--target .' being the magic sauce in the future either.
I agree. I would love for my use cases to become more of first-class citizens. Maybe --branch <branch> could serve as the knob?
What I also found really helpful in git filter-branch is that it was possible to pass one-liner shell scripts directly to the command, giving a lot of freedom about the transformations. I understand that Python makes it hard to write spaghetti-code one-liners, so you cannot really pass the snippet in via the command-line, but I hope there is a way to script things in git filter-repo?
Ciao,
Dscho
Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere
Hi Dscho,
Quoted 30 lines · Johannes Schindelin
Hi Elijah,
Quoted 25 lines · Elijah Newren
Quoted 21 lines · Johannes Schindelin
[...]
In my most recent instance of this, I wanted to publish the script I used to use for submitting patch series to the Git mailing list, maintaining tags for iterations and generating cover letters from branch descriptions and interdiffs (this script eventually became GitGitGadget,
https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0
demonstrates this).To do that, I ran a
git filter-branchin the repository where I track all the scripts I deem unsuitable for public consumption, to remove all files butmail-patch-series.sh, then pushed it to
https://github.com/dscho/mail-patch-seriesPlease note that most crucially, I wanted to rewrite a newly-created branch, and only that branch.
Could I have done the same using
git fast-export, filtering the output with a Perl script, then passing it togit fast-import? Sure, I was really tempted to do that. In the end, it took less of _my_ time to just letgit filter-branchdo its work with a not-too-complicated index filter.
Quoted 7 lines
Quoted 4 lines
Why a perl script? Shouldn't
git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quietdo the trick? And it's probably simpler and shorter than the index filter you used.
Does that not keep the full
$PATH? I wanted the resulting branch to have the file in the top-level directory.
Ah, gotcha; I read your original description to suggest that the script was already at the toplevel.
Quoted 14 lines
Quoted 5 lines
That said, yeah it'd be nice to get automatic rewriting of commit hashes in commit messages and other niceties from filter-repo (e.g. future automatic reattaching of notes to the rewritten commits). Some questions:
Quoted 8 lines
* What's the backup strategy in case you specify the wrong filters
(e.g. you have a typo in the pathnames)? filter-repo encourages folks to make a clone and then filter the fresh clone, because if anything goes awry, you can just delete and restart. (I am heavily opposed to the refs/original/ backup mechanism used by filter-branch, for multiple reasons.) Is your safety stance just "If I mess up it's my own fault; do the rewrite?" Or are you okay with cloning before filtering?
Quoted 5 lines
Please note that the
refs/original/refs should not have been written at all anymore, not after reflogs were introduced.Incidentally, that is my answer to your question: the reflog is my backup.
The reflog is great, but while it works in your special case please note that:
1. Anyone filtering a subset of refs more in number than one may have some difficulty restoring correctly (they have to look in several reflogs, and can't script restoring from all older reflog versions).
2. Few folks have core.logAllRefUpdates set to 'always', meaning they'll lack a backup for some refs if the reflog is relied upon.
3. If the filter specifies only keeping a list of files that happen to not exist within one of the branches (perhaps a filename was typo'ed) and if pruning empty commits, then the branch can be deleted, and git doesn't have a mechanism for deleting a branch without deleting its reflog as far as I know.
Point 1 is kind of minor, but points 2 and 3 are showstoppers in regards to me recommending the reflogs as a reliable recovery mechanism after general filtering operations, and this is true for either filter-branch or filter-repo. (That said, I can definitely allow people to choose their risks and just provide some here-be-dragons warnings.)
Quoted 43 lines
Quoted 5 lines
* If you're okay with cloning before filtering...then is there an
issue with rewriting all branches, and just pushing the one you need? (Is there an issue with "this branch is small, the others are huge, and filter-branch is slow -- so rewriting one branch saves me lots of time"? Or are there other issues at play too?)I am not okay with cloning before filtering.
First of all, it is wasteful.
Second of all, in my case it would have been *particularly* wasteful because the repository in question also has quite a few quite large blobs (hysterical raisins, don't ask).
Quoted 5 lines
* What if the user has auxiliary information for the branch in other
refs? For example, git-notes pointing at any of the commits, or tags in the history of the branch that might be relevant, or perhaps even replace refs in combination with GIT_NO_REPLACE_OBJECTS=1? Is this an "I don't care, toss that stuff and just rewrite just this branch?"In my case: there are no notes. The only time when I make heavy use of notes is in GitGitGadget. I don't use that feature otherwise.
Quoted 5 lines
* filter-repo by default creates new replace references so that you
can refer to new commit IDs using old (unabbreviated) commit IDs. Would that be considered helpful for this usecase? unhelpful? irrelevant, since you'll just push the branch you want somewhere and nuke the temporary clone?I definitely did not need that mapping in all of my
git filter-branchuse cases.Of course, I can see how it can come in handy in other circumstances, just not in the ones I experienced so far.
Quoted 7 lines
I'm not by any means ruling out the possibility of documenting --refs and adjusting the defaults when it is used so the user can just run something like
git filter-repo --path $PATH --refs $MYBRANCH
but I feel like I need to understand answers to questions like the above ones so that I can know how to phrase warnings and adjust defaults and update the documentation.
Quoted 4 lines
In all the scenarios where I used
git filter-branch(some dozen per year, so not all *that* many), I needed to rewrite one particular branch, typically a freshly-created one. I never, ever ever needed to rewrite all the refs in the repository. Not once ;-)
Thanks for answering all these and providing the extra context. Very helpful.
Quoted 26 lines
Quoted 12 lines
Quoted 6 lines
In another instance, a long, long time ago, I needed to restart a repository which had included way too many files for its own good, then rename the old repository and start with a fresh
masterthat contained but a single commit whose tree was identical to the previousmaster's tip commit. I simply grafted that commit, rangit filter-branchand had precisely what I needed.filter-repo supports grafts and replace objects, the same as filter-branch. (Although, technically, I didn't have to do a thing to support it; fast-export does the special handling of rewriting based on grafts and replace objects.) So, I'd say this is fully supported.
Quoted 5 lines
Side question: the git-replace documents suggest that the graft file is deprecated. Are there any timeframes or plans for phasing out beyond the git-replace manpage existing? Should I avoid documenting the graft file support in filter-repo? Should I include examples using not just git-replace but also using the graft file?
I had meant to prepare a patch series to remove
graftssupport that Junio could carry inpuuntil the time he considers it appropriate to merge tomaster, but it seems that this task fell under the rag.The deprecation itself has been introduced in tags/v2.18.0-rc0~54^2~4, i.e. it is official as of Git v2.18.0, which was released in mid-June last year.
Quoted 6 lines
My personal gut feeling is that we should let it simmer for another year before removing support for the
graftsfile (and we may want to update the label "grafted" whengit logshows a shallow commit before we remove that support forgrafts).So I'll not work on that patch for now.
Thanks for the extra history.
Quoted 19 lines
Quoted 4 lines
Quoted 3 lines
I would be _delighted_ if these kinds of use case (rewriting a branch, or even just a commit range) became more of a first-class citizen with
git filter-repo.Quoted 14 lines
I've got all the pieces for supporting a single branch or a commit range (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23 mybranch'), but the defaults (error out unless in a bare repo, move refs/remotes/origin/* to refs/heads/*, disconnect origin remote, expire reflogs & repack & prune, create new replace references so folks can access new commits using old commit IDs) may be somewhat friction-filled for this usecase. Those defaults other than the new replace refs happen to all be turned off with the combination of --force and --target, so, assuming turning them off is what you need, you could cheat and just specify 'git filter-repo --force --target . --refs $MYBRANCH' today and perhaps get what you want, but that's a really non-intuitive command line that is way too ugly to recommend. And I don't want to tie myself to '--target .' being the magic sauce in the future either.
Quoted 2 lines
I agree. I would love for my use cases to become more of first-class citizens. Maybe
--branch <branch>could serve as the knob?
I think I can put something together to make your usecases better. I dislike --branch, though, because:
* It suggests it doesn't work for tags or other things outside of refs/heads/*
* It suggests that revision ranges are unwelcome.
So, I'd prefer a more generic --refs which can potentially take multiple arguments, e.g. any of
--refs mybranch
--refs mytag
--refs HEAD~5..mybranch
--refs ^origin/master ^origin/other-feature mybranch1 mybranch2
Quoted 6 lines
What I also found really helpful in
git filter-branchis that it was possible to pass one-liner shell scripts directly to the command, giving a lot of freedom about the transformations. I understand that Python makes it hard to write spaghetti-code one-liners, so you cannot really pass the snippet in via the command-line, but I hope there is a way to script things ingit filter-repo?
I agree, having the ability to use a programming language snippet or even full script for special cases is really nice. You can totally do that filter-repo, at three different levels:
1. Light-control for easier cases: You can use the command line arguments --filename-callback, --message-callback, --name-callback, --email-callback, or --refname-callback and provide python snippets (usually one-liners) that return a new value. Most of these are for editing fields shared across multiple object types (e.g. the name callback will be used to edit author && committer && tagger names). However, the filename callback allows both editing the filename and also filtering based upon filename (you can return the original filename, OR a new name, OR you can return None to state that you want files with that name filtered out of commits).
2. Moderate-control: You can use the command line arguments --blob-callback, --commit-callback, --tag-callback, or --reset-callback and provide python snippets (possibly one-liners but more likely to be complex enough that you want newlines) that modify these fast-import-stream objects. These provide more control but tend to be slightly more work. (For example, if you want to rename branches, you need to worry about three callbacks: commit && tag && reset; by comparison, you'd only need to use the refname callback from lighter control. Also, if you're worried about filtering based on filenames, you'll need to dig the filenames (and modes and change types) out of the list commit.file_changes).
3. All-in: You can write a python script that imports filter-repo as a python module and (among other things) set up your own functions/classes as callbacks. You have to do a bit more setup to specify the options you are running with, list how many export and import processes you want to run with, name all your callbacks, etc., but it allows you to do anything from just providing a slightly more involved callback up to and including creating your own filtering tool with a totally different user interface while still leveraging filter-repo's capability. There are multiple examples provided along that range too (including bfg-ish and filter-lamely/filter-branch-ish.)
For more details about all of these, see:
https://github.com/newren/git-filter-repo/blob/a6a6a1b0/README.md#callbacks
https://github.com/newren/git-filter-repo/blob/a6a6a1b0/README.md#using-filter-repo-as-a-library
v2 cover Warn about git-filter-branch usage and avoid it
In reply to Elijah NewrenHere's a series that shifts the focus slightly to warning about git-filter-branch usage and avoiding it ourselves. I have retained patch 4 but left it marked as RFC for further discussion. It appears that folks generally seem to agree the first three patches are good to include now -- assuming my small fixes correctly address their requests and suggestions.
Changes since v1 (full range-diff below):
* I might have had a little fun with a thesaurus (just trying to give reviewers something small to smile about...)
* addressed feedback from Eric and Stolee, as detailed below
* [Patch 2] factored out some common code
* [Patch 3] fixed links in asciidoc documentation to make them more readable in both manpages and html format
* [Patch 3] added a warning blurb to git-filter-branch itself
In particular, it'd be helpful if people could take a look at the changes to git-filter-branch.sh in patch 3 and comment on whether an environment variable is fine or if we should make it a config setting or something.
Elijah Newren (4):
t6006: simplify and optimize empty message test
t3427: accelerate this test by using fast-export and fast-import Recommend git-filter-repo instead of git-filter-branch [RFC] Remove git-filter-branch, it is now external to git.git
.gitignore | 1 - Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 481 -------------------- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 2 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 4 +- Documentation/githooks.txt | 7 +- Makefile | 1 - command-list.txt | 1 - contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 662 ---------------------------- t/perf/p7000-filter-branch.sh | 24 - t/t3427-rebase-subtree.sh | 22 +- t/t6006-rev-list-format.sh | 5 +- t/t7003-filter-branch.sh | 505 --------------------- t/t7009-filter-branch-null-sha1.sh | 55 --- t/t9902-completion.sh | 12 +- 18 files changed, 47 insertions(+), 1772 deletions(-) delete mode 100644 Documentation/git-filter-branch.txt delete mode 100755 git-filter-branch.sh delete mode 100755 t/perf/p7000-filter-branch.sh delete mode 100755 t/t7003-filter-branch.sh delete mode 100755 t/t7009-filter-branch-null-sha1.sh
Range-diff:
1: 7ddbeea2ca = 1: 7ddbeea2ca t6006: simplify and optimize empty message test 2: 0172ca771e < -: ---------- t3427: accelerate this test by using fast-export and fast-import 3: b814cc7b65 < -: ---------- git-sh-i18n: work with external scripts -: ---------- > 2: f18bd7a609 t3427: accelerate this test by using fast-export and fast-import 4: dcec36d113 ! 3: 7008c16984 Recommend git-filter-repo instead of git-filter-branch in documentation
@@ Metadata
Author: Elijah Newren <[email protected]>
## Commit message ##
- Recommend git-filter-repo instead of git-filter-branch in documentation
+ Recommend git-filter-repo instead of git-filter-branch
- filter-branch suffers from a huge number of pitfalls that can result in
- incorrectly rewritten history, and many of the problems can easily go
- undetected until the new repository is in use. This can result in
- problems ranging from an even messier history than what led folks to
- filter-branch in the first place, to data loss or corruption. These
- issues cannot be backward compatibly fixed, so add a warning to the
- filter-branch manpage about this and recommand that another tool (such
- as filter-repo) be used instead.
+ filter-branch suffers from a deluge of disguised dangers that disfigure
+ history rewrites (i.e. deviate from the deliberate changes). Many of
+ these problems are unobtrusive and can easily go undiscovered until the
+ new repository is in use. This can result in problems ranging from an
+ even messier history than what led folks to filter-branch in the first
+ place, to data loss or corruption. These issues cannot be backward
+ compatibly fixed, so add a warning to both filter-branch and its manpage
+ recommending that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending
@@ Documentation/git-filter-branch.txt: SYNOPSIS
+WARNING
+-------
-+'git filter-branch' has a litany of gotchas that can and will cause
-+history to be rewritten incorrectly (in addition to abysmal
-+performance). These issues cannot be backward compatibly fixed and as
-+such, its use is not recommended. Please use an alternative history
-+filtering tool such as 'git filter-repo'. If you still need to use
-+'git filter-branch', please carefully read the "Safety" section of -+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/ -+and avoid as many of the pitfalls listed there as reasonably possible.
++'git filter-branch' has a plethora of pitfalls that can produce non-obvious
++manglings of the intended history rewrite (and can leave you with little
++time to investigate such problems since it has such abysmal performance).
++These safety and performance issues cannot be backward compatibly fixed and
++as such, its use is not recommended. Please use an alternative history
++filtering tool such as https://github.com/newren/git-filter-repo/[git
++filter-repo]. If you still need to use 'git filter-branch', please
++carefully read the "Safety" section of the message on the Git mailing list ++https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing ++the land mines of filter-branch] and vigilantly avoid as many of the
++hazards listed there as reasonably possible.
+
DESCRIPTION
-----------
@@ contrib/svn-fe/svn-fe.txt: The exit status does not reflect whether an error was
-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),
+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),
https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
+
+ ## git-filter-branch.sh (mode change 100755 => 100644) ##
+@@ git-filter-branch.sh: set_ident () {
+ finish_ident COMMITTER
+ }
+
++if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \
++ -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then
++ cat <<EOF
++WARNING: git-filter-branch has a glut of gotchas generating mangled history
++ rewrites. Please use an alternative filtering tool such as 'git
++ filter-repo' (https://github.com/newren/git-filter-repo/) instead.
++ See the filter-branch manual page for more details; to squelch
++ this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.
++
++EOF
++ sleep 5
++fi
++
+ USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>]
+ [--tree-filter <command>] [--index-filter <command>]
+ [--parent-filter <command>] [--msg-filter <command>]
5: 9dec8e06ee ! 4: ff3e04e558 Remove git-filter-branch, it is now external to git.git
@@ Metadata
## Commit message ##
Remove git-filter-branch, it is now external to git.git
+ git-filter-branch still exists, still has the same regression tests,
+ etc., but it is now being tracked in a separate repo that users will
+ need to download separately.
+
## .gitignore ##
@@ Documentation/git-filter-branch.txt (deleted)
-
-WARNING
--------
--'git filter-branch' has a litany of gotchas that can and will cause
--history to be rewritten incorrectly (in addition to abysmal
--performance). These issues cannot be backward compatibly fixed and as
--such, its use is not recommended. Please use an alternative history
--filtering tool such as 'git filter-repo'. If you still need to use
--'git filter-branch', please carefully read the "Safety" section of --https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/ --and avoid as many of the pitfalls listed there as reasonably possible.
+-'git filter-branch' has a plethora of pitfalls that can produce non-obvious
+-manglings of the intended history rewrite (and can leave you with little
+-time to investigate such problems since it has such abysmal performance).
+-These safety and performance issues cannot be backward compatibly fixed and
+-as such, its use is not recommended. Please use an alternative history
+-filtering tool such as https://github.com/newren/git-filter-repo/[git
+-filter-repo]. If you still need to use 'git filter-branch', please
+-carefully read the "Safety" section of the message on the Git mailing list +-https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing +-the land mines of filter-branch] and vigilantly avoid as many of the
+-hazards listed there as reasonably possible.
-
-DESCRIPTION
------------
@@ git-filter-branch.sh (deleted)
- finish_ident COMMITTER
-}
-
+-if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \
+- -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then
+- cat <<EOF
+-WARNING: git-filter-branch has a glut of gotchas generating mangled history
+- rewrites. Please use an alternative filtering tool such as 'git
+- filter-repo' (https://github.com/newren/git-filter-repo/) instead.
+- See the filter-branch manual page for more details; to squelch
+- this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.
+-
+-EOF
+- sleep 5
+-fi
+-
-USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>]
- [--tree-filter <command>] [--index-filter <command>]
- [--parent-filter <command>] [--msg-filter <command>]
Signature · 1 line
2.23.0.3.gcc10030edf.dirty
PATCH v2 1/4 t6006: simplify and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of time ./t6006-rev-list-format.sh) by about 11% on Linux and by 13% on Mac.
--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
t/t6006-rev-list-format.sh +2 -3
@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Signature · 1 line
2.23.0.3.gcc10030edf.dirty
PATCH v2 2/4 t3427: accelerate this test by using fast-export and fast-import
In reply to Elijah Newrenfast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):
Linux: 4.305s -> 3.684s (~17% speedup)
Mac: 10.128s -> 7.038s (~30% speedup)
--- t/t3427-rebase-subtree.sh | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
t/t3427-rebase-subtree.sh +14 -8
@@ -11,6 +11,12 @@ commit_message() { git log --pretty=format:%s -1 "$1" } +extract_files_subtree() { + git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet +} + test_expect_success 'setup' ' test_commit README && mkdir files &&
@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-preserve-merges-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master4"
@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-preserve-merges-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "files_subtree/master5"
@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-keep-empty-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-keep-empty-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' ' reset_rebase && git checkout -b rebase-keep-empty-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' reset_rebase && git checkout -b rebase-onto-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' reset_rebase && git checkout -b rebase-onto-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' test_expect_failure 'Rebase -Xsubtree --onto empty commit' ' reset_rebase && git checkout -b rebase-onto-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
Signature · 1 line
2.23.0.3.gcc10030edf.dirty
Re: [PATCH v2 2/4] t3427: accelerate this test by using fast-export and fast-import
t/t3427-rebase-subtree.sh commit_message() { · line 11
Quoted 12 lines · Elijah Newren
fast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via
time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):[...]Signed-off-by Elijah Newren <[email protected]>---
t/t3427-rebase-subtree.sh +1
-0@@ -11,6 +11,12 @@ commit_message() { +extract_files_subtree() {
Style nit: add space before opening '('
(However, commit_message() function just above this doesn't follow that style, so...)
t/t3427-rebase-subtree.sh commit_message() { · line 17
Quoted 3 lines
t/t3427-rebase-subtree.sh +3
-0+ git fast-export --no-data HEAD -- files_subtree/ \ + | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \ + | git fast-import --force --quiet
This would be a bit less noisy if you ended each line with the pipe operator, allowing you to drop the backslashes:
git fast-export --no-data HEAD -- files_subtree/ |
sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" |
git fast-import --force --quiett/t3427-rebase-subtree.sh commit_message() { · line 18
Quoted 1 line
t/t3427-rebase-subtree.sh +1
-0+}
Not sure any of this is worth a re-roll.
PATCH v2 3/4 Recommend git-filter-repo instead of git-filter-branch
In reply to Elijah Newrenfilter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). Many of these problems are unobtrusive and can easily go undiscovered until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to both filter-branch and its manpage recommending that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.
--- Documentation/git-fast-export.txt | 6 ++-- Documentation/git-filter-branch.txt | 45 +++++++++-------------------- Documentation/git-gc.txt | 17 +++++------ Documentation/git-rebase.txt | 2 +- Documentation/git-replace.txt | 10 +++---- Documentation/git-svn.txt | 4 +-- Documentation/githooks.txt | 7 +++-- contrib/svn-fe/svn-fe.txt | 4 +-- git-filter-branch.sh | 13 +++++++++ 9 files changed, 52 insertions(+), 56 deletions(-) mode change 100755 => 100644 git-filter-branch.sh
Documentation/git-fast-export.txt +3 -3
@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------
Documentation/git-filter-branch.txt +14 -31
57 lines not on this page · raw message
Documentation/git-gc.txt +8 -9
@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is using
Documentation/git-rebase.txt +1 -1
@@ -832,7 +832,7 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like `filter-repo`. The easy case
Documentation/git-replace.txt +5 -5
@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +linkgit:git-filter-repo[1], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a
@@ -148,8 +148,8 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] +linkgit:git-filter-repo[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1]
Documentation/git-svn.txt +2 -2
@@ -769,9 +769,9 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git +reports, and archives. If you plan to eventually migrate from SVN to Git and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows +linkgit:git-filter-repo[1] instead. filter-repo also allows reformatting of metadata for ease-of-reading and rewriting authorship info for non-"svn.authorsFile" users.
Documentation/githooks.txt +4 -3
@@ -425,9 +425,10 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do +not call it!). Its first argument denotes the command it was invoked +by: currently one of `amend` or `rebase`. Further command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in the
contrib/svn-fe/svn-fe.txt +2 -2
@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS
@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
git-filter-branch.sh +13 -0
@@ -83,6 +83,19 @@ set_ident () { finish_ident COMMITTER } +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then + cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1. + +EOF + sleep 5 +fi + USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>] [--tree-filter <command>] [--index-filter <command>] [--parent-filter <command>] [--msg-filter <command>]
Signature · 1 line
2.23.0.3.gcc10030edf.dirty
Re: [PATCH v2 3/4] Recommend git-filter-repo instead of git-filter-branch
Quoted 10 lines · Elijah Newren
filter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). [...]
Signed-off-by Elijah Newren <[email protected]>---
Documentation/git-filter-branch.txt +3
-0@@ -16,6 +16,20 @@ SYNOPSIS +WARNING +------- +'git filter-branch' has a plethora of pitfalls that can produce non-obvious
Documentation/git-filter-branch.txt SYNOPSIS · line 28
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+manglings of the intended history rewrite (and can leave you with little +time to investigate such problems since it has such abysmal performance). +These safety and performance issues cannot be backward compatibly fixed and +as such, its use is not recommended. Please use an alternative history +filtering tool such as https://github.com/newren/git-filter-repo/[git +filter-repo]. If you still need to use 'git filter-branch', please +carefully read the "Safety" section of the message on the Git mailing list +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing +the land mines of filter-branch] and vigilantly avoid as many of the +hazards listed there as reasonably possible.
Is there a good reason to not simply copy the "Safety" section from that email directly into this document so that readers don't have to go chasing down the link (especially those who are reading documentation offline)?
Documentation/git-rebase.txt Hard case: The changes are not the same.:: · line 835
Quoted 7 lines
Documentation/git-rebase.txt +1
-1@@ -832,7 +832,7 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like `filter-repo`.
Do we want a clickable link to filter-repo here?
Documentation/git-replace.txt pending objects. · line 150
Quoted 9 lines
Documentation/git-replace.txt +3
-0@@ -123,10 +123,10 @@ The following format are available: +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +linkgit:git-filter-repo[1], among other git commands, can be used to@@ -148,8 +148,8 @@ pending objects. linkgit:git-hash-object[1] linkgit:git-rebase[1] +linkgit:git-filter-repo[1]
Are these 'linkgit:' references to filter-repo going to be meaningful if that tool is not incorporated into the Git project proper? Perhaps use a generic clickable link instead.
Same comment applies to other 'linkgit:' invocations in the remainder of the patch.
git-filter-branch.sh
Quoted 3 lines
git-filter-branch.sh +0
-0
Why lose the executable bit?
git-filter-branch.sh set_ident () { · line 84
Quoted 3 lines
git-filter-branch.sh +2
-0@@ -83,6 +83,19 @@ set_ident () { +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then
If this script didn't already have a mix of styles, I'd say something about modern style being:
if test -z "$FILTER_BRANCH_SQUELCH_WARNING" &&
test -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS"
then
...
figit-filter-branch.sh set_ident () { · line 93
Quoted 6 lines
git-filter-branch.sh +6
-0+ cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.
The "and pause" threw me. There's more than a bit of ambiguity surrounding it. Perhaps drop it?
Re: [PATCH v2 3/4] Recommend git-filter-repo instead of git-filter-branch
Quoted 19 lines · Eric Sunshine
Quoted 17 lines · Elijah Newren
filter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). [...]
Signed-off-by Elijah Newren <[email protected]>---
Documentation/git-filter-branch.txt +10
-0@@ -16,6 +16,20 @@ SYNOPSIS +WARNING +------- +'git filter-branch' has a plethora of pitfalls that can produce non-obvious +manglings of the intended history rewrite (and can leave you with little +time to investigate such problems since it has such abysmal performance). +These safety and performance issues cannot be backward compatibly fixed and +as such, its use is not recommended. Please use an alternative history +filtering tool such as https://github.com/newren/git-filter-repo/[git +filter-repo]. If you still need to use 'git filter-branch', please +carefully read the "Safety" section of the message on the Git mailing list
Quoted 8 lines
Documentation/git-filter-branch.txt SYNOPSIS · line 31
Quoted 3 lines
Documentation/git-filter-branch.txt +3
-0+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing +the land mines of filter-branch] and vigilantly avoid as many of the +hazards listed there as reasonably possible.Is there a good reason to not simply copy the "Safety" section from that email directly into this document so that readers don't have to go chasing down the link (especially those who are reading documentation offline)?
Makes sense, I can include it. However, saying e.g. "the git-filter-branch manpage is missing..." or "the git-filter-branch manpage actually documents <crazy buggy behavior> as expected" feels really weird to include on the git-filter-branch manpage. I'll try to touch it up.
Quoted 9 lines
Documentation/git-rebase.txt Hard case: The changes are not the same.:: · line 835
Quoted 7 lines
Documentation/git-rebase.txt +1
-1@@ -832,7 +832,7 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like `filter-repo`.Do we want a clickable link to
filter-repohere?
I guess it can't hurt.
Quoted 14 lines
Documentation/git-replace.txt pending objects. · line 150
Quoted 9 lines
Documentation/git-replace.txt +3
-0@@ -123,10 +123,10 @@ The following format are available: +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +linkgit:git-filter-repo[1], among other git commands, can be used to@@ -148,8 +148,8 @@ pending objects. linkgit:git-hash-object[1] linkgit:git-rebase[1] +linkgit:git-filter-repo[1]Are these 'linkgit:' references to
filter-repogoing to be meaningful if that tool is not incorporated into the Git project proper? Perhaps use a generic clickable link instead.
Quoted 2 lines
Same comment applies to other 'linkgit:' invocations in the remainder of the patch.
I'm fixing them up.
Quoted 5 lines
git-filter-branch.sh
Quoted 3 lines
git-filter-branch.sh +0
-0Why lose the executable bit?
Whoops. Did some rebasing and fixups, then continued editing my buffer of the file after one of the rebases, realized the file was deleted (because of the final patch in the series), moved the file out of the way and rebased again and copied the file back into place, and forgot to check the filemode.
Quoted 20 lines
git-filter-branch.sh set_ident () { · line 84
Quoted 3 lines
git-filter-branch.sh +2
-0@@ -83,6 +83,19 @@ set_ident () { +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; thenIf this script didn't already have a mix of styles, I'd say something about modern style being:
if test -z "$FILTER_BRANCH_SQUELCH_WARNING" && test -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" then ... figit-filter-branch.sh set_ident () { · line 93
Quoted 6 lines
git-filter-branch.sh +6
-0+ cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.
Quoted 2 lines
The "and pause" threw me. There's more than a bit of ambiguity surrounding it. Perhaps drop it?
Sure, will do.
RFC PATCH v2 4/4 Remove git-filter-branch, it is now external to git.git
In reply to Elijah Newrengit-filter-branch still exists, still has the same regression tests, etc., but it is now being tracked in a separate repo that users will need to download separately.
--- .gitignore | 1 - Documentation/git-filter-branch.txt | 464 ------------------- Makefile | 1 - command-list.txt | 1 - git-filter-branch.sh | 675 ---------------------------- t/perf/p7000-filter-branch.sh | 24 - t/t7003-filter-branch.sh | 505 --------------------- t/t7009-filter-branch-null-sha1.sh | 55 --- t/t9902-completion.sh | 12 +- 9 files changed, 6 insertions(+), 1732 deletions(-) delete mode 100644 Documentation/git-filter-branch.txt delete mode 100644 git-filter-branch.sh delete mode 100755 t/perf/p7000-filter-branch.sh delete mode 100755 t/t7003-filter-branch.sh delete mode 100755 t/t7009-filter-branch-null-sha1.sh
.gitignore +0 -1
@@ -63,7 +63,6 @@ /git-fast-import /git-fetch /git-fetch-pack -/git-filter-branch /git-fmt-merge-msg /git-for-each-ref /git-format-patch
Documentation/git-filter-branch.txt deleted +0 -464
464 lines not on this page · raw message
Makefile +0 -1
@@ -607,7 +607,6 @@ unexport CDPATH SCRIPT_SH += git-bisect.sh SCRIPT_SH += git-difftool--helper.sh -SCRIPT_SH += git-filter-branch.sh SCRIPT_SH += git-merge-octopus.sh SCRIPT_SH += git-merge-one-file.sh SCRIPT_SH += git-merge-resolve.sh
command-list.txt +0 -1
@@ -90,7 +90,6 @@ git-fast-export ancillarymanipulators git-fast-import ancillarymanipulators git-fetch mainporcelain remote git-fetch-pack synchingrepositories -git-filter-branch ancillarymanipulators git-fmt-merge-msg purehelpers git-for-each-ref plumbinginterrogators git-format-patch mainporcelain
git-filter-branch.sh deleted +0 -675
675 lines not on this page · raw message
t/perf/p7000-filter-branch.sh deleted +0 -24
@@ -1,24 +0,0 @@ -#!/bin/sh - -test_description='performance of filter-branch' -. ./perf-lib.sh - -test_perf_default_repo -test_checkout_worktree - -test_expect_success 'mark bases for tests' ' - git tag -f tip && - git tag -f base HEAD~100 -' - -test_perf 'noop filter' ' - git checkout --detach tip && - git filter-branch -f base..HEAD -' - -test_perf 'noop prune-empty' ' - git checkout --detach tip && - git filter-branch -f --prune-empty base..HEAD -' - -test_done
t/t7003-filter-branch.sh deleted +0 -505
505 lines not on this page · raw message
t/t7009-filter-branch-null-sha1.sh deleted +0 -55
55 lines not on this page · raw message
t/t9902-completion.sh +6 -6
@@ -28,10 +28,10 @@ complete () # # (2) A test makes sure that common subcommands are included in the # completion for "git <TAB>", and a plumbing is excluded. "add", -# "filter-branch" and "ls-files" are listed for this. +# "rebase" and "ls-files" are listed for this. -GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files' -GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch' +GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files' +GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase' . "$GIT_BUILD_DIR/contrib/completion/git-completion.bash"
@@ -1392,12 +1392,12 @@ test_expect_success 'basic' ' # built-in grep -q "^add \$" out && # script - grep -q "^filter-branch \$" out && + grep -q "^rebase \$" out && # plumbing ! grep -q "^ls-files \$" out && - run_completion "git f" && - ! grep -q -v "^f" out + run_completion "git r" && + ! grep -q -v "^r" out ' test_expect_success 'double dash "git" itself' '
Signature · 1 line
2.23.0.3.gcc10030edf.dirty
v3 cover Warn about git-filter-branch usage and avoid it
In reply to Elijah NewrenHere's a series warns about git-filter-branch usage and avoids it ourselves.
Changes since v2 (full range-diff below):
* [Patch 2] testcase syntax cleanups
* [Patch 3] fixed "linkgit:" references to filter-repo to be url links (or footnotes)
* [Patch 3] fixed the mode on filter-branch.sh (oops) and dropped the ambiguous "and pause". Linkified "filter-repo" in a place where there was no link.
* [Patch 3] As suggested by Eric (Sunshine), just make the manpage and directly include the safety and performance sections of the referenced email (the performance section was referenced by the safety section). Being included directly in the manpage should help with folks reading the documentation offline. Anyway, the text is really long, so it took a while to format it nicely, recheck for typos, reword based on the fact that it'll be in the manpage (because it's weird to have the manpage refer to itself), etc.
* [Patch 4] Dropped almost all the original patch 4; only including the bits about t9902-completion.sh. Removed the RFC label, since that one piece should be good for including now.
Elijah Newren (4):
t6006: simplify and optimize empty message test
t3427: accelerate this test by using fast-export and fast-import Recommend git-filter-repo instead of git-filter-branch
t9902: use a non-deprecated command for testing
Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 302 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 13 ++ t/t3427-rebase-subtree.sh | 24 ++- t/t6006-rev-list-format.sh | 5 +- t/t9902-completion.sh | 12 +- 12 files changed, 339 insertions(+), 77 deletions(-)
Range-diff:
1: 7ddbeea2ca = 1: 7ddbeea2ca t6006: simplify and optimize empty message test
2: f18bd7a609 ! 2: e1e63189c1 t3427: accelerate this test by using fast-export and fast-import
@@ Commit message
Signed-off-by: Elijah Newren <[email protected]>
## t/t3427-rebase-subtree.sh ##
-@@ t/t3427-rebase-subtree.sh: commit_message() {
+@@ t/t3427-rebase-subtree.sh: This test runs git rebase and tests the subtree strategy.
+ . ./test-lib.sh
+ . "$TEST_DIRECTORY"/lib-rebase.sh
+
+-commit_message() {
++commit_message () {
git log --pretty=format:%s -1 "$1"
}
-+extract_files_subtree() {
-+ git fast-export --no-data HEAD -- files_subtree/ \
-+ | sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
-+ | git fast-import --force --quiet
++extract_files_subtree () {
++ git fast-export --no-data HEAD -- files_subtree/ |
++ sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" |
++ git fast-import --force --quiet
+}
+
test_expect_success 'setup' '3: 7008c16984 ! 3: 59c7446927 Recommend git-filter-repo instead of git-filter-branch
@@ Documentation/git-filter-branch.txt: SYNOPSIS
+as such, its use is not recommended. Please use an alternative history
+filtering tool such as https://github.com/newren/git-filter-repo/[git
+filter-repo]. If you still need to use 'git filter-branch', please
-+carefully read the "Safety" section of the message on the Git mailing list
++carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land
++mines of filter-branch, and then vigilantly avoid as many of the hazards
++listed there as reasonably possible.
++
+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing
-+the land mines of filter-branch] and vigilantly avoid as many of the
-+hazards listed there as reasonably possible.
++the land mines of filter-branch]
+
DESCRIPTION
-----------
@@ Documentation/git-filter-branch.txt: warned.
- are much more restrictive than git-filter branch, and dedicated just
- to the tasks of removing unwanted data- e.g:
- --strip-blobs-bigger-than 1M.
--
++[[PERFORMANCE]]
++PERFORMANCE
++-----------
++
++The performance of filter-branch is glacially slow; its design makes it
++impossible for a backward-compatible implementation to ever be fast:
++
++* In editing files, git-filter-branch by design checks out each and
++every commit as it existed in the original repo. If your repo has 10\^5
++files and 10\^5 commits, but each commit only modifies 5 files, then
++git-filter-branch will make you do 10\^10 modifications, despite only
++having (at most) 5*10^5 unique blobs.
++
++* If you try and cheat and try to make filter-branch only work on
++files modified in a commit, then two things happen
++
++ . you run into problems with deletions whenever the user is simply
++ trying to rename files (because attempting to delete files that
++ don't exist looks like a no-op; it takes some chicanery to remap
++ deletes across file renames when the renames happen via arbitrary
++ user-provided shell)
++
++ . even if you succeed at the map-deletes-for-renames chicanery, you
++ still technically violate backward compatibility because users are
++ allowed to filter files in ways that depend upon topology of commits
++ instead of filtering solely based on file contents or names (though
++ I have never seen any user ever do this).
++
++* Even if you don't need to edit files but only want to e.g. rename or
++remove some and thus can avoid checking out each file (i.e. you can use ++--index-filter), you still are passing shell snippets for your filters.
++This means that for every commit, you have to have a prepared git repo
++where users can run git commands. That's a lot of setup. It also means
++you have to fork at least one process to run the user-provided shell
++snippet, and odds are that the user's shell snippet invokes lots of
++commands in some long pipeline, so you will have lots and lots of forks.
++For every. single. commit. That's a massive amount of overhead to
++rename a few files.
++
++* filter-branch is written in shell, which is kind of slow. Naturally,
++it makes sense to want to rewrite that in some other language. However,
++filter-branch documentation states that several additional shell
++functions are provided for users to call, e.g. 'map', 'skip_commit', ++'git_commit_non_empty_tree'. If filter-branch itself isn't a shell
++script, then in order to make those shell functions available to the
++users' shell snippets you have to prepend the shell definitions of these
++functions to every one of the users' shell snippets and thus make these
++special shell functions be parsed with each and every commit.
++
++* filter-branch provides a --setup option which is a shell snippet that
++can be sourced to make shell functions and variables available to all
++other filters. If filter-branch is a shell script, it can simply eval
++this shell snippet once at the beginning. If you try to fix performance
++by making filter-branch not be a shell script, then you have to prepend
++the setup shell snippet to all other filters and parse it with every
++single commit.
++
++* filter-branch writes lots of files to $workdir/../map/ to keep a
++mapping of commits, which it uses pruning commits and remapping to
++ancestors and the map() command more generally. Other files like
++$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,
++$tempdir/tree-state are all created internally too. It is possible
++(though strongly discouraged) that users could have accessed any of
++these directly. Users even had a pointer to follow in the form of
++Documentation that the 'map' command existed, which naturally uses the ++$workdir/../map/* files. So, even if you don't have to edit files, for
++strict backward compatibility you need to still write a bunch of files
++to disk somewhere and keep them updated for every commit. You can claim
++it was an implementation detail that users should not have depended
++upon, but the truth is they've had a decade where they could so. So, if
++you want full compatibility, it has to be there. Besides, the
++regression tests depend on at least one of these details, specifying an
++--index-filter that reaches down and grabs backup-refs from $tempdir,
++and thus provides resourceful users who do google searches an example
++that there are files there for them to read and grab and use. (And if
++you want to pass the existing regression tests, you have to at least put
++the backup-refs file there even if it's irrelevant to your
++implementation otherwise.)
++
++All of that said, performance of filter-branch could be improved by
++reimplementing it in a non-shell language and taking a couple small
++liberties with backward compatibility (such as having it only run
++filters on files changed within each commit). filter-repo provides a
++demo script named ++https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely] ++which does exactly that and which passes all the git-filter-branch
++regression tests. It's much faster than git-filter-branch, though it
++suffers from all the same safety issues as git-filter-branch, and is
++still glacially slow compared to
++https://github.com/newren/git-filter-repo/[git filter-repo].
++
++[[SAFETY]]
++SAFETY
++------
++
++filter-branch is riddled with gotchas resulting in various ways to
++easily corrupt repos or end up with a mess worse than what you started
++with:
++
++* Someone can have a set of "working and tested filters" which they
++document or provide to a coworker, who then runs them on a different OS
++where the same commands are not working/tested (some examples in the
++git-filter-branch manpage are also affected by this). BSD vs. GNU
++userland differences can really bite. If you're lucky, you get ugly
++error messages spewed. But just as likely, the commands either don't do
++the filtering requested, or silently corrupt making some unwanted
++change. The unwanted change may only affect a few commits, so it's not
++necessarily obvious either. (The fact that problems won't necessarily
++be obvious means they are likely to go unnoticed until the rewritten
++history is in use for quite a while, at which point it's really hard to
++justify another flag-day for another rewrite.)
++
++* Filenames with spaces (which are rare) are often mishandled by shell
++snippets since they cause problems for shell pipelines. Not everyone is
++familiar with find -print0, xargs -0, ls-files -z, etc. Even people who
++are familiar with these may assume such needs are not relevant because
++someone else renamed any such files in their repo back before the person
++doing the filtering joined the project. And, often, even those familiar
++with handling arguments with spaces my not do so just because they
++aren't in the mindset of thinking about everything that could possibly
++go wrong.
++
++* Non-ascii filenames (which are rare) can be silently removed despite
++being in a desired directory. The desire to select paths to keep often
++use pipelines like git ls-files | grep -v ^WANTED_DIR/ | xargs git rm.
++ls-files will only quote filenames if needed so folks may not notice
++that one of the files didn't match the regex, again until it's much too
++late. Yes, someone who knows about core.quotePath can avoid this
++(unless they have other special characters like \t, \n, or "), and
++people who use ls-files -z with something other than grep can avoid
++this, but that doesn't mean they will.
++
++* Similarly, when moving files around, one can find that filenames with
++non-ascii or special characters end up in a different directory, one
++that includes a double quote character. (This is technically the same
++issue as above with quoting, but perhaps an interesting different way
++that it can and has manifested as a problem.)
++
++* It's far too easy to accidentally mix up old and new history. It's
++still possible with any tool, but filter-branch almost invites it. If
++we're lucky, the only downside is users getting frustrated that they
++don't know how to shrink their repo and remove the old stuff. If we're
++unlucky, they merge old and new history and end up with multiple
++"copies" of each commit, some of which have unwanted or sensitive files
++and others which don't. This comes about in multiple different ways:
++
++ ** the default to only doing a partial history rewrite ('--all' is not
++ the default and over 80% of the examples in the manpage don't use
++ it)
++
++ ** the fact that there's no automatic post-run cleanup
++
++ ** the fact that --tag-name-filter (when used to rename tags) doesn't
++ remove the old tags but just adds new ones with the new name (this
++ manpage has documented this for a long time so it's presumably not
++ a "bug" even though it feels like it)
++
++ ** the fact that little educational information is provided to inform
++ users of the ramifications of a rewrite and how to avoid mixing old
++ and new history. For example, this man page discusses how users
++ need to understand that they need to rebase their changes for all
++ their branches on top of new history (or delete and reclone), but
++ that's only one of multiple concerns to consider. See the
++ "DISCUSSION" section of the git filter-repo manual page for more
++ details.
++
++* Annotated tags can be accidentally converted to lightweight tags, due
++to either of two issues:
++
++ . Someone can do a history rewrite, realize they messed up, restore
++ from the backups in refs/original/, and then redo their
++ filter-branch command. (The backup in refs/original/ is not a real
++ backup; it dereferences tags first.)
++
++ . Running filter-branch with either --tags or --all in your <rev-list
++ options>. In order to retain annotated tags as annotated, you must
++ use --tag-name-filter (and must not have restored from
++ refs/original/ in a previously botched rewrite).
++
++* Any commit messages that specify an encoding will become corrupted
++by the rewrite; filter-branch ignores the encoding, takes the original
++bytes, and feeds it to commit-tree without telling it the proper
++encoding. (This happens whether or not --msg-filter is used, though I
++suspect --msg-filter provides additional ways to really mess things
++up).
++
++* Commit messages (even if they are all UTF-8) by default become
++corrupted due to not being updated -- any references to other commit
++hashes in commit messages will now refer to no-longer-extant commits.
++
++* There are no facilities for helping users find what unwanted crud they
++should delete, which means they are much more likely to have incomplete
++or partial cleanups that sometimes result in confusion and people
++wasting time trying to understand. (For example, folks tend to just
++look for big files to delete instead of big directories or extensions,
++and once they do so, then sometime later folks using the new repository
++who are going through history will notice a build artifact directory
++that has some files but not others, or a cache of dependencies
++(node_modules or similar) which couldn't have ever been functional since
++it's missing some files.)
++
++* If --prune-empty isn't specified, then the filtering process can
++create hoards of confusing empty commits
++
++* If --prune-empty is specified, then intentionally placed empty
++commits from before the filtering operation are also pruned instead of
++just pruning commits that became empty due to filtering rules.
++
++* If --prune empty is specified, sometimes empty commits are missed
++and left around anyway (a somewhat rare bug, but it happens...)
++
++* A minor issue, but users who have a goal to update all names and
++emails in a repository may be led to --env-filter which will only update
++authors and committers, missing taggers.
++
++* If the user provides a --tag-name-filter that maps multiple tags to
++the same name, no warning or error is provided; filter-branch simply
++overwrites each tag in some undocumented pre-defined order resulting in
++only one tag at the end. If you try to "fix" this bug in filter-branch
++and make it error out and warn the user instead, one of the
++filter-branch regression tests will fail. (So, if you are trying to
++make a backward compatible reimplementation you have to add extra code
++to detect collisions and make sure that only the lexicographically last
++one is rewritten to avoid fast-import from seeing both since fast-import
++will naturally do the sane thing and error out if told to write the same
++tag more than once.)
++
++Also, the poor performance of filter-branch often leads to safety issues:
++
++* Coming up with the correct shell snippet to do the filtering you want
++is sometimes difficult unless you're just doing a trivial modification
++such as deleting a couple files. People have often come to me for help,
++so I should be practiced and an expert, but even for fairly simple cases
++I still sometimes take over 10 minutes and several iterations to get
++the right commands -- and that's assuming they are working on a tiny
++repository. Unfortunately, people often learn if the snippet is right
++or wrong by trying it out, but the rightness or wrongness can vary
++depending on special circumstances (spaces in filenames, non-ascii
++filenames, funny author names or emails, invalid timezones, presence of
++grafts or replace objects, etc.), meaning they may have to wait a long
++time, hit an error, then restart. The performance of filter-branch is
++so bad that this cycle is painful, reducing the time available to
++carefully re-check (to say nothing about what it does to the patience of
++the person doing the rewrite even if they do technically have more time
++available). This problem is extra compounded because errors from broken
++filters may not be shown for a long time and/or get lost in a sea of
++output. Even worse, broken filters often just result in silent
++incorrect rewrites.
++
++* To top it all off, even when users finally find working commands, they
++naturally want to share them. But they may be unaware that their repo
++didn't have some special cases that someone else's does. So, when
++someone else with a different repository runs the same commands, they
++get hit by the problems above. Or, the user just runs commands that
++really were vetted for special cases, but they run it on a different OS
++where it doesn't work, as noted above.
+
GIT
---
- Part of the linkgit:git[1] suite
## Documentation/git-gc.txt ##
@@ Documentation/git-gc.txt: NOTES
@@ Documentation/git-rebase.txt: Hard case: The changes are not the same.::
--interactive to omit, edit, squash, or fixup commits; or if the upstream used one of commit --amend, reset, or
- filter-branch.
-+ a full history rewriting command like filter-repo.
++ a full history rewriting command like
++ https://github.com/newren/git-filter-repo[filter-repo].
The easy case
@@ Documentation/git-replace.txt: The following format are available:
-replacement objects from existing objects. The --edit option can
-also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and
-+linkgit:git-filter-repo[1], among other git commands, can be used to ++https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to
+create replacement objects from existing objects. The --edit option
+can also be used with 'git replace' to create a replacement object by
editing an existing object.
@@ Documentation/git-replace.txt: pending objects.
linkgit:git-hash-object[1]
-linkgit:git-filter-branch[1]
linkgit:git-rebase[1]
-+linkgit:git-filter-repo[1]
linkgit:git-tag[1]
linkgit:git-branch[1]
linkgit:git-commit[1]
+ linkgit:git-var[1]
+ linkgit:git[1]
++https://github.com/newren/git-filter-repo[git-filter-repo]
+
+ GIT
+ ---
## Documentation/git-svn.txt ##
@@ Documentation/git-svn.txt: option for (hopefully) obvious reasons.
@@ Documentation/git-svn.txt: option for (hopefully) obvious reasons.
This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug
-reports and archives. If you plan to eventually migrate from SVN to Git
-+reports, and archives. If you plan to eventually migrate from SVN to Git
- and are certain about dropping SVN history, consider
+-and are certain about dropping SVN history, consider
-linkgit:git-filter-branch[1] instead. filter-branch also allows
-+linkgit:git-filter-repo[1] instead. filter-repo also allows
- reformatting of metadata for ease-of-reading and rewriting authorship
- info for non-"svn.authorsFile" users.
+-reformatting of metadata for ease-of-reading and rewriting authorship +-info for non-"svn.authorsFile" users.
++reports, and archives. If you plan to eventually migrate from SVN to ++Git and are certain about dropping SVN history, consider ++https://github.com/newren/git-filter-repo[git-filter-repo] instead. ++filter-repo also allows reformatting of metadata for ease-of-reading ++and rewriting authorship info for non-"svn.authorsFile" users.
+ svn.useSvmProps::
+ svn-remote.<name>.useSvmProps::
## Documentation/githooks.txt ##
@@ Documentation/githooks.txt: post-rewrite
@@ Documentation/githooks.txt: post-rewrite
-linkgit:git-rebase[1]; currently git filter-branch does 'not' call -it!). Its first argument denotes the command it was invoked by:
-currently one of amend or rebase. Further command-dependent
+-arguments may be passed in the future.
+linkgit:git-rebase[1]; however, full-history (re)writing tools like
-+linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do -+not call it!). Its first argument denotes the command it was invoked -+by: currently one of amend or rebase. Further command-dependent
- arguments may be passed in the future. ++linkgit:git-fast-import[1] or ++https://github.com/newren/git-filter-repo[git-filter-repo] typically ++do not call it!). Its first argument denotes the command it was ++invoked by: currently one of amend or rebase. Further ++command-dependent arguments may be passed in the future.
The hook receives a list of the rewritten commits on stdin, in the
+ format
## contrib/svn-fe/svn-fe.txt ##
@@ contrib/svn-fe/svn-fe.txt: line. This line has the form git-svn-id: URL@REVNO UUID.
@@ contrib/svn-fe/svn-fe.txt: The exit status does not reflect whether an error was
+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),
https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
- ## git-filter-branch.sh (mode change 100755 => 100644) ##
+ ## git-filter-branch.sh ##
@@ git-filter-branch.sh: set_ident () {
finish_ident COMMITTER
}
@@ git-filter-branch.sh: set_ident () {
+ rewrites. Please use an alternative filtering tool such as 'git
+ filter-repo' (https://github.com/newren/git-filter-repo/) instead.
+ See the filter-branch manual page for more details; to squelch
-+ this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.
++ this warning, set FILTER_BRANCH_SQUELCH_WARNING=1.
+
+EOF
+ sleep 5
4: ff3e04e558 < -: ---------- Remove git-filter-branch, it is now external to git.git -: ---------- > 4: 1dbca82408 t9902: use a non-deprecated command for testing
Signature · 1 line
2.23.0.3.g59c7446927.dirty
PATCH v3 1/4 t6006: simplify and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of time ./t6006-rev-list-format.sh) by about 11% on Linux and by 13% on Mac.
--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
t/t6006-rev-list-format.sh +2 -3
@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Signature · 1 line
2.23.0.3.g59c7446927.dirty
PATCH v3 2/4 t3427: accelerate this test by using fast-export and fast-import
In reply to Elijah Newrenfast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):
Linux: 4.305s -> 3.684s (~17% speedup)
Mac: 10.128s -> 7.038s (~30% speedup)
--- t/t3427-rebase-subtree.sh | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
t/t3427-rebase-subtree.sh +15 -9
@@ -7,10 +7,16 @@ This test runs git rebase and tests the subtree strategy. . ./test-lib.sh . "$TEST_DIRECTORY"/lib-rebase.sh -commit_message() { +commit_message () { git log --pretty=format:%s -1 "$1" } +extract_files_subtree () { + git fast-export --no-data HEAD -- files_subtree/ | + sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" | + git fast-import --force --quiet +} + test_expect_success 'setup' ' test_commit README && mkdir files &&
@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-preserve-merges-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master4"
@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-preserve-merges-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "files_subtree/master5"
@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-keep-empty-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-keep-empty-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' ' reset_rebase && git checkout -b rebase-keep-empty-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' reset_rebase && git checkout -b rebase-onto-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' reset_rebase && git checkout -b rebase-onto-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' test_expect_failure 'Rebase -Xsubtree --onto empty commit' ' reset_rebase && git checkout -b rebase-onto-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
Signature · 1 line
2.23.0.3.g59c7446927.dirty
PATCH v3 3/4 Recommend git-filter-repo instead of git-filter-branch
In reply to Elijah Newrenfilter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). Many of these problems are unobtrusive and can easily go undiscovered until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to both filter-branch and its manpage recommending that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.
--- Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 302 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 13 ++ 9 files changed, 316 insertions(+), 59 deletions(-)
Documentation/git-fast-export.txt +3 -3
@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------
Documentation/git-filter-branch.txt +272 -30
314 lines not on this page · raw message
Documentation/git-gc.txt +8 -9
@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is using
Documentation/git-rebase.txt +2 -1
@@ -832,7 +832,8 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like + https://github.com/newren/git-filter-repo[`filter-repo`]. The easy case
Documentation/git-replace.txt +5 -5
@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a
@@ -148,13 +148,13 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1] linkgit:git-var[1] linkgit:git[1] +https://github.com/newren/git-filter-repo[git-filter-repo] GIT ---
Documentation/git-svn.txt +5 -5
@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git -and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows -reformatting of metadata for ease-of-reading and rewriting authorship -info for non-"svn.authorsFile" users. +reports, and archives. If you plan to eventually migrate from SVN to +Git and are certain about dropping SVN history, consider +https://github.com/newren/git-filter-repo[git-filter-repo] instead. +filter-repo also allows reformatting of metadata for ease-of-reading +and rewriting authorship info for non-"svn.authorsFile" users. svn.useSvmProps:: svn-remote.<name>.useSvmProps::
Documentation/githooks.txt +6 -4
@@ -425,10 +425,12 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent -arguments may be passed in the future. +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or +https://github.com/newren/git-filter-repo[git-filter-repo] typically +do not call it!). Its first argument denotes the command it was +invoked by: currently one of `amend` or `rebase`. Further +command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in the format
contrib/svn-fe/svn-fe.txt +2 -2
@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS
@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
git-filter-branch.sh +13 -0
@@ -83,6 +83,19 @@ set_ident () { finish_ident COMMITTER } +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then + cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1. + +EOF + sleep 5 +fi + USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>] [--tree-filter <command>] [--index-filter <command>] [--parent-filter <command>] [--msg-filter <command>]
Signature · 1 line
2.23.0.3.g59c7446927.dirty
Re: [PATCH v3 3/4] Recommend git-filter-repo instead of git-filter-branch
Quoted 11 lines · Elijah Newren
filter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). [...]
Signed-off-by Elijah Newren <[email protected]>---
Documentation/git-filter-branch.txt +5
-0@@ -16,6 +16,22 @@ SYNOPSIS +WARNING +------- +'git filter-branch' has a plethora of pitfalls that can produce non-obvious +manglings of the intended history rewrite (and can leave you with little +time to investigate such problems since it has such abysmal performance).
Documentation/git-filter-branch.txt SYNOPSIS · line 30
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+These safety and performance issues cannot be backward compatibly fixed and +as such, its use is not recommended. Please use an alternative history +filtering tool such as https://github.com/newren/git-filter-repo/[git +filter-repo]. If you still need to use 'git filter-branch', please +carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land +mines of filter-branch, and then vigilantly avoid as many of the hazards +listed there as reasonably possible. + +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing +the land mines of filter-branch]
This stray link looks like leftover gunk from the previous revision.
Documentation/git-filter-branch.txt warned. · line 468
Quoted 4 lines
Documentation/git-filter-branch.txt +4
-0+PERFORMANCE +----------- + +The performance of filter-branch is glacially slow; its design makes it
The rest of this document spells it git-filter-branch or 'git filter-branch', not plain filter-branch.
Documentation/git-filter-branch.txt warned. · line 478
Quoted 8 lines
Documentation/git-filter-branch.txt +8
-0+* In editing files, git-filter-branch by design checks out each and +every commit as it existed in the original repo. If your repo has 10\^5 +files and 10\^5 commits, but each commit only modifies 5 files, then +git-filter-branch will make you do 10\^10 modifications, despite only +having (at most) 5*10^5 unique blobs. + +* If you try and cheat and try to make filter-branch only work on +files modified in a commit, then two things happen
s/filter-branch/git-&/
Documentation/git-filter-branch.txt warned. · line 490
Quoted 12 lines
Documentation/git-filter-branch.txt +12
-0+ + . you run into problems with deletions whenever the user is simply + trying to rename files (because attempting to delete files that + don't exist looks like a no-op; it takes some chicanery to remap + deletes across file renames when the renames happen via arbitrary + user-provided shell) + + . even if you succeed at the map-deletes-for-renames chicanery, you + still technically violate backward compatibility because users are + allowed to filter files in ways that depend upon topology of commits + instead of filtering solely based on file contents or names (though + I have never seen any user ever do this).
Maybe avoid first-person:
... contents or names (though this has not been observed in the wild).
Quoted 7 lines
Documentation/git-filter-branch.txt +7
-0+* filter-branch is written in shell, which is kind of slow. Naturally, +it makes sense to want to rewrite that in some other language. However, +filter-branch documentation states that several additional shell +functions are provided for users to call, e.g. 'map', 'skip_commit', +'git_commit_non_empty_tree'. If filter-branch itself isn't a shell +script, then in order to make those shell functions available to the +users' shell snippets you have to prepend the shell definitions of these
Documentation/git-filter-branch.txt warned. · line 519
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+functions to every one of the users' shell snippets and thus make these +special shell functions be parsed with each and every commit. + +* filter-branch provides a --setup option which is a shell snippet that +can be sourced to make shell functions and variables available to all +other filters. If filter-branch is a shell script, it can simply eval +this shell snippet once at the beginning. If you try to fix performance +by making filter-branch not be a shell script, then you have to prepend +the setup shell snippet to all other filters and parse it with every +single commit.
Even though they made sense in the context of the original email message, these two bullet points may not belong in the man page since someone reading the man page is doing so to learn about git-filter-branch usage, not because he or she is thinking about re-implementing it. It might make sense, however, to collapse these points to some general statement about shell being slow and process startup being costly.
Also, these bullet points and others below need a s/filter-branch/git-&/.
Quoted 1 line
+* filter-branch writes lots of files to $workdir/../map/ to keep a
Should that path have three dots "..." instead of two ".."?
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+mapping of commits, which it uses pruning commits and remapping to +ancestors and the map() command more generally. Other files like +$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads, +$tempdir/tree-state are all created internally too. It is possible +(though strongly discouraged) that users could have accessed any of +these directly. Users even had a pointer to follow in the form of +Documentation that the 'map' command existed, which naturally uses the +$workdir/../map/* files. So, even if you don't have to edit files, for +strict backward compatibility you need to still write a bunch of files +to disk somewhere and keep them updated for every commit. You can claim
Documentation/git-filter-branch.txt warned. · line 541
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+it was an implementation detail that users should not have depended +upon, but the truth is they've had a decade where they could so. So, if +you want full compatibility, it has to be there. Besides, the +regression tests depend on at least one of these details, specifying an +--index-filter that reaches down and grabs backup-refs from $tempdir, +and thus provides resourceful users who do google searches an example +that there are files there for them to read and grab and use. (And if +you want to pass the existing regression tests, you have to at least put +the backup-refs file there even if it's irrelevant to your +implementation otherwise.)
As with the earlier comment, this bullet point is aimed at someone thinking about re-implementing the command; it sounds out of place in the "Performance" section of the man page. However, it does make sense to mention all the files git-filter-branch creates since that can have an impact on performance. So, perhaps this section can be collapsed so it just talks about that.
Documentation/git-filter-branch.txt warned. · line 553
Quoted 11 lines
Documentation/git-filter-branch.txt +11
-0+All of that said, performance of filter-branch could be improved by +reimplementing it in a non-shell language and taking a couple small +liberties with backward compatibility (such as having it only run +filters on files changed within each commit). filter-repo provides a +demo script named +https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely] +which does exactly that and which passes all the git-filter-branch +regression tests. It's much faster than git-filter-branch, though it +suffers from all the same safety issues as git-filter-branch, and is +still glacially slow compared to +https://github.com/newren/git-filter-repo/[git filter-repo].
This paragraph could be collapsed to say merely that, for those with existing tooling relying upon git-filter-branch, filter-repo's "filter-lamely" provides a drop-in replacement with somewhat improved performance and a few caveats.
Taking the above comments into consideration, here is a possible rewrite of the final three bullet points and the closing paragraph:
* filter-branch is written in shell, which is kind of slow, and it potentially can run many other commands which can slow down its operation significantly, especially on platforms for which process startup is costly.
* filter-branch writes lots of files to $workdir/.../map/ to keep a mapping of commits, which it uses for pruning commits and remapping to ancestors and for the map() command more generally. Other files like $tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads, $tempdir/tree-state are created internally too. Such file creation can be costly in general, but especially on platforms with slow filesystems.
The tool https://github.com/newren/git-filter-repo/[git
filter-repo] is an alternative to git-filter-branch which does not suffer from these performance problems or the safety problems (mentioned below). For those with existing tooling which relies upon git-filter-branch, 'git repo-filter' also provides
https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],
a drop-in git-filter-branch replacement (with a few caveats).
Documentation/git-filter-branch.txt warned. · line 586
Quoted 4 lines
Documentation/git-filter-branch.txt +4
-0+SAFETY +------ + +* Non-ascii filenames (which are rare) can be silently removed despite
Perhaps drop "(which are rare)" to make this sound more formal and less like an email message.
Comment below also are intended to make the prose sound a bit more formal.
Documentation/git-filter-branch.txt warned. · line 589
Quoted 3 lines
Documentation/git-filter-branch.txt +3
-0+being in a desired directory. The desire to select paths to keep often +use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. +ls-files will only quote filenames if needed so folks may not notice
s/ls-files/git-&/
Documentation/git-filter-branch.txt warned. · line 604
Quoted 9 lines
Documentation/git-filter-branch.txt +9
-0+that one of the files didn't match the regex, again until it's much too +late. Yes, someone who knows about core.quotePath can avoid this +(unless they have other special characters like \t, \n, or "), and +people who use ls-files -z with something other than grep can avoid +this, but that doesn't mean they will. + +* It's far too easy to accidentally mix up old and new history. It's +still possible with any tool, but filter-branch almost invites it. If +we're lucky, the only downside is users getting frustrated that they
s/we're//
Quoted 1 line
+don't know how to shrink their repo and remove the old stuff. If we're
s/we're//
Documentation/git-filter-branch.txt warned. · line 612
Quoted 7 lines
Documentation/git-filter-branch.txt +7
-0+unlucky, they merge old and new history and end up with multiple +"copies" of each commit, some of which have unwanted or sensitive files +and others which don't. This comes about in multiple different ways: + + ** the default to only doing a partial history rewrite ('--all' is not + the default and over 80% of the examples in the manpage don't use + it)
Maybe just shorten this to:
('--all is not the default, and few examples show it)
Documentation/git-filter-branch.txt warned. · line 619
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+ ** the fact that there's no automatic post-run cleanup + + ** the fact that --tag-name-filter (when used to rename tags) doesn't + remove the old tags but just adds new ones with the new name (this + manpage has documented this for a long time so it's presumably not + a "bug" even though it feels like it)
Perhaps drop the final parenthetical comment.
Quoted 11 lines
Documentation/git-filter-branch.txt +11
-0+ ** the fact that little educational information is provided to inform + users of the ramifications of a rewrite and how to avoid mixing old + and new history. For example, this man page discusses how users + need to understand that they need to rebase their changes for all + their branches on top of new history (or delete and reclone), but + that's only one of multiple concerns to consider. See the + "DISCUSSION" section of the git filter-repo manual page for more + details. + +* Annotated tags can be accidentally converted to lightweight tags, due +to either of two issues:
Documentation/git-filter-branch.txt warned. · line 641
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+ + . Someone can do a history rewrite, realize they messed up, restore + from the backups in refs/original/, and then redo their + filter-branch command. (The backup in refs/original/ is not a real + backup; it dereferences tags first.) + + . Running filter-branch with either --tags or --all in your <rev-list + options>. In order to retain annotated tags as annotated, you must + use --tag-name-filter (and must not have restored from + refs/original/ in a previously botched rewrite).
Should these bullet points use "**" rather than "."?
Documentation/git-filter-branch.txt warned. · line 648
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+* Any commit messages that specify an encoding will become corrupted +by the rewrite; filter-branch ignores the encoding, takes the original +bytes, and feeds it to commit-tree without telling it the proper +encoding. (This happens whether or not --msg-filter is used, though I +suspect --msg-filter provides additional ways to really mess things +up).
Perhaps shorten simply to:
(This happens whether or not --msg-filter is used.)
Documentation/git-filter-branch.txt warned. · line 689
Quoted 11 lines
Documentation/git-filter-branch.txt +11
-0+* If the user provides a --tag-name-filter that maps multiple tags to +the same name, no warning or error is provided; filter-branch simply +overwrites each tag in some undocumented pre-defined order resulting in +only one tag at the end. If you try to "fix" this bug in filter-branch +and make it error out and warn the user instead, one of the +filter-branch regression tests will fail. (So, if you are trying to +make a backward compatible reimplementation you have to add extra code +to detect collisions and make sure that only the lexicographically last +one is rewritten to avoid fast-import from seeing both since fast-import +will naturally do the sane thing and error out if told to write the same +tag more than once.)
Maybe drop everything from "If you try to 'fix'..." to the end of paragraph.
Quoted 11 lines
Documentation/git-filter-branch.txt +11
-0+Also, the poor performance of filter-branch often leads to safety issues: + +* Coming up with the correct shell snippet to do the filtering you want +is sometimes difficult unless you're just doing a trivial modification +such as deleting a couple files. People have often come to me for help, +so I should be practiced and an expert, but even for fairly simple cases +I still sometimes take over 10 minutes and several iterations to get +the right commands -- and that's assuming they are working on a tiny +repository. Unfortunately, people often learn if the snippet is right +or wrong by trying it out, but the rightness or wrongness can vary +depending on special circumstances (spaces in filenames, non-ascii
Documentation/git-filter-branch.txt warned. · line 711
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+filenames, funny author names or emails, invalid timezones, presence of +grafts or replace objects, etc.), meaning they may have to wait a long +time, hit an error, then restart. The performance of filter-branch is +so bad that this cycle is painful, reducing the time available to +carefully re-check (to say nothing about what it does to the patience of +the person doing the rewrite even if they do technically have more time +available). This problem is extra compounded because errors from broken +filters may not be shown for a long time and/or get lost in a sea of +output. Even worse, broken filters often just result in silent +incorrect rewrites.
Drop the "People have often come to me..." sentence from this paragraph.
Re: [PATCH v3 3/4] Recommend git-filter-repo instead of git-filter-branch
Hi Eric,
Thanks for the careful and thoughtful review.
Quoted 17 lines · Eric Sunshine
Quoted 15 lines · Elijah Newren
filter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). [...]
Signed-off-by Elijah Newren <[email protected]>---
Documentation/git-filter-branch.txt +9
-0@@ -16,6 +16,22 @@ SYNOPSIS +WARNING +------- +'git filter-branch' has a plethora of pitfalls that can produce non-obvious +manglings of the intended history rewrite (and can leave you with little +time to investigate such problems since it has such abysmal performance). +These safety and performance issues cannot be backward compatibly fixed and +as such, its use is not recommended. Please use an alternative history +filtering tool such as https://github.com/newren/git-filter-repo/[git +filter-repo]. If you still need to use 'git filter-branch', please
Quoted 8 lines
Documentation/git-filter-branch.txt SYNOPSIS · line 33
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land +mines of filter-branch, and then vigilantly avoid as many of the hazards +listed there as reasonably possible. + +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing +the land mines of filter-branch]This stray link looks like leftover gunk from the previous revision.
Ugh, indeed.
Quoted 9 lines
Documentation/git-filter-branch.txt warned. · line 468
Quoted 5 lines
Documentation/git-filter-branch.txt +4
-0+PERFORMANCE +----------- + +The performance of filter-branch is glacially slow; its design makes itThe rest of this document spells it git-filter-branch or 'git filter-branch', not plain filter-branch.
Quoted 10 lines
Documentation/git-filter-branch.txt warned. · line 478
Quoted 8 lines
Documentation/git-filter-branch.txt +8
-0+* In editing files, git-filter-branch by design checks out each and +every commit as it existed in the original repo. If your repo has 10\^5 +files and 10\^5 commits, but each commit only modifies 5 files, then +git-filter-branch will make you do 10\^10 modifications, despite only +having (at most) 5*10^5 unique blobs. + +* If you try and cheat and try to make filter-branch only work on +files modified in a commit, then two things happens/filter-branch/git-&/
I can fix these up.
Quoted 16 lines
Documentation/git-filter-branch.txt warned. · line 490
Quoted 13 lines
Documentation/git-filter-branch.txt +12
-0+ + . you run into problems with deletions whenever the user is simply + trying to rename files (because attempting to delete files that + don't exist looks like a no-op; it takes some chicanery to remap + deletes across file renames when the renames happen via arbitrary + user-provided shell) + + . even if you succeed at the map-deletes-for-renames chicanery, you + still technically violate backward compatibility because users are + allowed to filter files in ways that depend upon topology of commits + instead of filtering solely based on file contents or names (though + I have never seen any user ever do this).Maybe avoid first-person:
Quoted 2 lines
... contents or names (though this has not been observed in the wild).
Thanks for providing alternative wording.
Quoted 18 lines
Quoted 7 lines
Documentation/git-filter-branch.txt +7
-0+* filter-branch is written in shell, which is kind of slow. Naturally, +it makes sense to want to rewrite that in some other language. However, +filter-branch documentation states that several additional shell +functions are provided for users to call, e.g. 'map', 'skip_commit', +'git_commit_non_empty_tree'. If filter-branch itself isn't a shell +script, then in order to make those shell functions available to the +users' shell snippets you have to prepend the shell definitions of theseDocumentation/git-filter-branch.txt warned. · line 519
Quoted 10 lines
Documentation/git-filter-branch.txt +10
-0+functions to every one of the users' shell snippets and thus make these +special shell functions be parsed with each and every commit. + +* filter-branch provides a --setup option which is a shell snippet that +can be sourced to make shell functions and variables available to all +other filters. If filter-branch is a shell script, it can simply eval +this shell snippet once at the beginning. If you try to fix performance +by making filter-branch not be a shell script, then you have to prepend +the setup shell snippet to all other filters and parse it with every +single commit.
Quoted 7 lines
Even though they made sense in the context of the original email message, these two bullet points may not belong in the man page since someone reading the man page is doing so to learn about git-filter-branch usage, not because he or she is thinking about re-implementing it. It might make sense, however, to collapse these points to some general statement about shell being slow and process startup being costly.
Hmm. I see where you're coming from, but the performance section isn't really user actionable stuff anyway; it's just a warning. And I have repeatedly seen over the years the question brought up on the list of "Can we make filter-branch fast by making it a builtin?" (Or "Can't _you_ make filter-branch fast by rewriting it in C?")
I could try to reword it so that there's some general statement about shell being slow and process startup being costly, and then add these two items as sub-bullets to try to stave off that obvious but misguided question from coming up. Or maybe I just add a reference to the original email?
Quoted 1 line
Also, these bullet points and others below need a s/filter-branch/git-&/.
Thanks, will fix.
Quoted 3 lines
Quoted 1 line
+* filter-branch writes lots of files to $workdir/../map/ to keep a
Should that path have three dots "..." instead of two ".."?
No, it's a literal parent directory reference. Users have access to $workdir; it's where their commands run. There is no name for the parent of that directory, other than by appending '/..' to wherever they are. Maybe if I had spelled it as $(pwd)/../map/ it would be better?
Or maybe I don't need to name the files at all; does it really matter to the user?
Quoted 19 lines
Quoted 19 lines
Documentation/git-filter-branch.txt +19
-0+mapping of commits, which it uses pruning commits and remapping to +ancestors and the map() command more generally. Other files like +$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads, +$tempdir/tree-state are all created internally too. It is possible +(though strongly discouraged) that users could have accessed any of +these directly. Users even had a pointer to follow in the form of +Documentation that the 'map' command existed, which naturally uses the +$workdir/../map/* files. So, even if you don't have to edit files, for +strict backward compatibility you need to still write a bunch of files +to disk somewhere and keep them updated for every commit. You can claim +it was an implementation detail that users should not have depended +upon, but the truth is they've had a decade where they could so. So, if +you want full compatibility, it has to be there. Besides, the +regression tests depend on at least one of these details, specifying an +--index-filter that reaches down and grabs backup-refs from $tempdir, +and thus provides resourceful users who do google searches an example +that there are files there for them to read and grab and use. (And if +you want to pass the existing regression tests, you have to at least put +the backup-refs file there even if it's irrelevant to your
Quoted 8 lines
Quoted 1 line
+implementation otherwise.)
As with the earlier comment, this bullet point is aimed at someone thinking about re-implementing the command; it sounds out of place in the "Performance" section of the man page. However, it does make sense to mention all the files git-filter-branch creates since that can have an impact on performance. So, perhaps this section can be collapsed so it just talks about that.
I think there's both a how-performance-affects-user component and a component addressing the common incorrect question/statement/thought that filter-branch performance could just be fixed by making it a builtin. But splitting this may make sense. And maybe the portions addressing making-it-a-builtin-wouldn't-fix-it could be a short sentence with a link to the original email for more details.
Quoted 8 lines
Quoted 8 lines
Documentation/git-filter-branch.txt +8
-0+All of that said, performance of filter-branch could be improved by +reimplementing it in a non-shell language and taking a couple small +liberties with backward compatibility (such as having it only run +filters on files changed within each commit). filter-repo provides a +demo script named +https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely] +which does exactly that and which passes all the git-filter-branch +regression tests. It's much faster than git-filter-branch, though it
Quoted 8 lines
Documentation/git-filter-branch.txt warned. · line 553
Quoted 3 lines
Documentation/git-filter-branch.txt +3
-0+suffers from all the same safety issues as git-filter-branch, and is +still glacially slow compared to +https://github.com/newren/git-filter-repo/[git filter-repo].This paragraph could be collapsed to say merely that, for those with existing tooling relying upon git-filter-branch, filter-repo's "filter-lamely" provides a drop-in replacement with somewhat improved performance and a few caveats.
Sounds good.
Quoted 2 lines
Taking the above comments into consideration, here is a possible rewrite of the final three bullet points and the closing paragraph:
Oh, sweet, thanks for providing this. I really like the simplicity of your suggested wording in general; it will be really helpful in rewording. I do have a nitpick with each one, though...
Quoted 4 lines
* filter-branch is written in shell, which is kind of slow, and it potentially can run many other commands which can slow down its operation significantly, especially on platforms for which process startup is costly.
Even if it's not the emphasis you intended, I'm worried this makes it sound as if filter-branch performance is only bad on Windows or Mac. Compared to invoking a function (even in a bytecode interpreted language), creating and running another process is slow on any platform.
Quoted 7 lines
* filter-branch writes lots of files to $workdir/.../map/ to keep a mapping of commits, which it uses for pruning commits and remapping to ancestors and for the
map()command more generally. Other files like $tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads, $tempdir/tree-state are created internally too. Such file creation can be costly in general, but especially on platforms with slow filesystems.
Again, it may not have been your intended emphasis, but I think this may be read as singling out slow filesystems, and make people think the performance problems from this bullet point only affects some OSes. Filesystems are part of the problem. Disks being slow is part of the problem. But it's not all of it. I guess part of what really gets me with these is that they represent forced synchronization (e.g. the kernel has to flush the data upon close() to make sure any other processes can see the file contents and all further filtering is blocked waiting for this to finish). By way of comparison, in filter-repo I have to both write data to fast-import and read back information from fast-import (in order to find out the new commit names, for example). When I did the straightforward thing of writing a commit, writing a 'get-mark' directive, and then reading the answer, it ruined performance. So I had to be a bit smarter and defer reading back the resulting sha1. There's no room for anything similarly clever in filter-branch; writing these files out is a synchronization point that is needed before the user's filter can be eval'ed.
Quoted 7 lines
The tool https://github.com/newren/git-filter-repo/[git
filter-repo] is an alternative to git-filter-branch which does not suffer from these performance problems or the safety problems (mentioned below). For those with existing tooling which relies upon git-filter-branch, 'git repo-filter' also provides
https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],
a drop-in git-filter-branch replacement (with a few caveats).
This suggests filter-lamely doesn't suffer from performance or safety problems, which is very misleading. filter-lamely doesn't improve the safety story at all and only ameliorates the performance problems somewhat.
Quoted 7 lines
Documentation/git-filter-branch.txt warned. · line 586
Quoted 4 lines
Documentation/git-filter-branch.txt +4
-0+SAFETY +------ + +* Non-ascii filenames (which are rare) can be silently removed despitePerhaps drop "(which are rare)" to make this sound more formal and less like an email message.
Makes sense; and I'm guessing I should also drop it from the bullet point above this one.
I'll stop commenting on the individual comments since there's not much to say with most of them other than they look like obviously good suggestions...
Quoted 34 lines
Comment below also are intended to make the prose sound a bit more formal.
Documentation/git-filter-branch.txt warned. · line 589
Quoted 3 lines
Documentation/git-filter-branch.txt +3
-0+being in a desired directory. The desire to select paths to keep often +use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. +ls-files will only quote filenames if needed so folks may not notices/ls-files/git-&/
Documentation/git-filter-branch.txt warned. · line 604
Quoted 9 lines
Documentation/git-filter-branch.txt +9
-0+that one of the files didn't match the regex, again until it's much too +late. Yes, someone who knows about core.quotePath can avoid this +(unless they have other special characters like \t, \n, or "), and +people who use ls-files -z with something other than grep can avoid +this, but that doesn't mean they will. + +* It's far too easy to accidentally mix up old and new history. It's +still possible with any tool, but filter-branch almost invites it. If +we're lucky, the only downside is users getting frustrated that theys/we're//
Quoted 1 line
+don't know how to shrink their repo and remove the old stuff. If we're
s/we're//
Documentation/git-filter-branch.txt warned. · line 612
Quoted 7 lines
Documentation/git-filter-branch.txt +7
-0+unlucky, they merge old and new history and end up with multiple +"copies" of each commit, some of which have unwanted or sensitive files +and others which don't. This comes about in multiple different ways: + + ** the default to only doing a partial history rewrite ('--all' is not + the default and over 80% of the examples in the manpage don't use + it)Maybe just shorten this to:
Quoted 1 line
('--all is not the default, and few examples show it)
I know I said I'd not comment unless I disagreed, but I just wanted to say thanks so much for providing concrete suggestions in so many places. It's *very* helpful.
Quoted 24 lines
Documentation/git-filter-branch.txt warned. · line 619
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+ ** the fact that there's no automatic post-run cleanup + + ** the fact that --tag-name-filter (when used to rename tags) doesn't + remove the old tags but just adds new ones with the new name (this + manpage has documented this for a long time so it's presumably not + a "bug" even though it feels like it)Perhaps drop the final parenthetical comment.
Quoted 15 lines
Documentation/git-filter-branch.txt +15
-0+ ** the fact that little educational information is provided to inform + users of the ramifications of a rewrite and how to avoid mixing old + and new history. For example, this man page discusses how users + need to understand that they need to rebase their changes for all + their branches on top of new history (or delete and reclone), but + that's only one of multiple concerns to consider. See the + "DISCUSSION" section of the git filter-repo manual page for more + details. + +* Annotated tags can be accidentally converted to lightweight tags, due +to either of two issues: + + . Someone can do a history rewrite, realize they messed up, restore + from the backups in refs/original/, and then redo their + filter-branch command. (The backup in refs/original/ is not a real
Quoted 8 lines
Documentation/git-filter-branch.txt warned. · line 641
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+ backup; it dereferences tags first.) + + . Running filter-branch with either --tags or --all in your <rev-list + options>. In order to retain annotated tags as annotated, you must + use --tag-name-filter (and must not have restored from + refs/original/ in a previously botched rewrite).Should these bullet points use "**" rather than "."?
I guess it could but the "either of two issues" above it made me think of numbering them. I also had a couple sub-bullets in the performance section that were numbered. But I guess it is slightly weird coming so close after another section that used un-numbered subbullets. I guess I'll just make them all un-numbered.
Quoted 11 lines
Documentation/git-filter-branch.txt warned. · line 648
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+* Any commit messages that specify an encoding will become corrupted +by the rewrite; filter-branch ignores the encoding, takes the original +bytes, and feeds it to commit-tree without telling it the proper +encoding. (This happens whether or not --msg-filter is used, though I +suspect --msg-filter provides additional ways to really mess things +up).Perhaps shorten simply to:
(This happens whether or not --msg-filter is used.)
Quoted 13 lines
Documentation/git-filter-branch.txt warned. · line 689
Quoted 11 lines
Documentation/git-filter-branch.txt +11
-0+* If the user provides a --tag-name-filter that maps multiple tags to +the same name, no warning or error is provided; filter-branch simply +overwrites each tag in some undocumented pre-defined order resulting in +only one tag at the end. If you try to "fix" this bug in filter-branch +and make it error out and warn the user instead, one of the +filter-branch regression tests will fail. (So, if you are trying to +make a backward compatible reimplementation you have to add extra code +to detect collisions and make sure that only the lexicographically last +one is rewritten to avoid fast-import from seeing both since fast-import +will naturally do the sane thing and error out if told to write the same +tag more than once.)Maybe drop everything from "If you try to 'fix'..." to the end of paragraph.
Or just replace that long section you highlight with a parenthetical comment, "(a git-filter-branch regression test requires this.)"
Quoted 15 lines
Quoted 15 lines
Documentation/git-filter-branch.txt +15
-0+Also, the poor performance of filter-branch often leads to safety issues: + +* Coming up with the correct shell snippet to do the filtering you want +is sometimes difficult unless you're just doing a trivial modification +such as deleting a couple files. People have often come to me for help, +so I should be practiced and an expert, but even for fairly simple cases +I still sometimes take over 10 minutes and several iterations to get +the right commands -- and that's assuming they are working on a tiny +repository. Unfortunately, people often learn if the snippet is right +or wrong by trying it out, but the rightness or wrongness can vary +depending on special circumstances (spaces in filenames, non-ascii +filenames, funny author names or emails, invalid timezones, presence of +grafts or replace objects, etc.), meaning they may have to wait a long +time, hit an error, then restart. The performance of filter-branch is +so bad that this cycle is painful, reducing the time available to
Quoted 8 lines
Documentation/git-filter-branch.txt warned. · line 711
Quoted 6 lines
Documentation/git-filter-branch.txt +6
-0+carefully re-check (to say nothing about what it does to the patience of +the person doing the rewrite even if they do technically have more time +available). This problem is extra compounded because errors from broken +filters may not be shown for a long time and/or get lost in a sea of +output. Even worse, broken filters often just result in silent +incorrect rewrites.Drop the "People have often come to me..." sentence from this paragraph.
Thanks again for the careful reading and many suggestions!
PATCH v3 4/4 t9902: use a non-deprecated command for testing
In reply to Elijah Newrent9902 had a list of three random porcelain commands as a sanity check, one of which was filter-branch. Since we are recommending people not use filter-branch, let's update this test to use rebase instead of filter-branch.
--- t/t9902-completion.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
t/t9902-completion.sh +6 -6
@@ -28,10 +28,10 @@ complete () # # (2) A test makes sure that common subcommands are included in the # completion for "git <TAB>", and a plumbing is excluded. "add", -# "filter-branch" and "ls-files" are listed for this. +# "rebase" and "ls-files" are listed for this. -GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files' -GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch' +GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files' +GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase' . "$GIT_BUILD_DIR/contrib/completion/git-completion.bash"
@@ -1392,12 +1392,12 @@ test_expect_success 'basic' ' # built-in grep -q "^add \$" out && # script - grep -q "^filter-branch \$" out && + grep -q "^rebase \$" out && # plumbing ! grep -q "^ls-files \$" out && - run_completion "git f" && - ! grep -q -v "^f" out + run_completion "git r" && + ! grep -q -v "^r" out ' test_expect_success 'double dash "git" itself' '
Signature · 1 line
2.23.0.3.g59c7446927.dirty
v4 cover Warn about git-filter-branch usage and avoid it
In reply to Elijah NewrenHere's a series warns about git-filter-branch usage and avoids it ourselves.
Changes since v3
* Incorporated Eric's detailed feedback on the git-filter-branch manpage, some notes:
* s/filter-branch/git-&/ (and similar for ls-files)
* Multiple sections removed (and existing sections had a number of sentences removed)
* I ended up not linking to the original html, but just added a small "Side Note" in a sub-bullet to address how fixing the written-in-shell attribute of git-filter-branch would do less than proponents expect.
* ...and lots of other miscellaneous wording fixes and cleanups
* The full range-diff is below, but it's kinda hard to read due to line wrapping and such.
Elijah Newren (4):
t6006: simplify and optimize empty message test
t3427: accelerate this test by using fast-export and fast-import Recommend git-filter-repo instead of git-filter-branch
t9902: use a non-deprecated command for testing
Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 272 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 13 ++ t/t3427-rebase-subtree.sh | 24 ++- t/t6006-rev-list-format.sh | 5 +- t/t9902-completion.sh | 12 +- 12 files changed, 309 insertions(+), 77 deletions(-)
Range-diff:
1: 7ddbeea2ca = 1: 7ddbeea2ca t6006: simplify and optimize empty message test
2: e1e63189c1 = 2: e1e63189c1 t3427: accelerate this test by using fast-export and fast-import
3: 59c7446927 ! 3: ed6505584f Recommend git-filter-repo instead of git-filter-branch
@@ Documentation/git-filter-branch.txt: SYNOPSIS
+carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land
+mines of filter-branch, and then vigilantly avoid as many of the hazards
+listed there as reasonably possible.
-+
-+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing
-+the land mines of filter-branch]
+
DESCRIPTION
-----------
@@ Documentation/git-filter-branch.txt: warned.
+PERFORMANCE
+-----------
+
-+The performance of filter-branch is glacially slow; its design makes it
++The performance of git-filter-branch is glacially slow; its design makes it
+impossible for a backward-compatible implementation to ever be fast:
+
+* In editing files, git-filter-branch by design checks out each and
@@ Documentation/git-filter-branch.txt: warned.
+git-filter-branch will make you do 10\^10 modifications, despite only
+having (at most) 5*10^5 unique blobs.
+
-+* If you try and cheat and try to make filter-branch only work on
++* If you try and cheat and try to make git-filter-branch only work on
+files modified in a commit, then two things happen
+
-+ . you run into problems with deletions whenever the user is simply
-+ trying to rename files (because attempting to delete files that
-+ don't exist looks like a no-op; it takes some chicanery to remap
-+ deletes across file renames when the renames happen via arbitrary
-+ user-provided shell)
++ ** you run into problems with deletions whenever the user is simply
++ trying to rename files (because attempting to delete files that
++ don't exist looks like a no-op; it takes some chicanery to remap
++ deletes across file renames when the renames happen via arbitrary
++ user-provided shell)
+
-+ . even if you succeed at the map-deletes-for-renames chicanery, you
-+ still technically violate backward compatibility because users are
-+ allowed to filter files in ways that depend upon topology of commits
-+ instead of filtering solely based on file contents or names (though
-+ I have never seen any user ever do this).
++ ** even if you succeed at the map-deletes-for-renames chicanery, you
++ still technically violate backward compatibility because users are
++ allowed to filter files in ways that depend upon topology of
++ commits instead of filtering solely based on file contents or names
++ (though this has not been observed in the wild).
+
+* Even if you don't need to edit files but only want to e.g. rename or
+remove some and thus can avoid checking out each file (i.e. you can use
+--index-filter), you still are passing shell snippets for your filters.
+This means that for every commit, you have to have a prepared git repo
-+where users can run git commands. That's a lot of setup. It also means
-+you have to fork at least one process to run the user-provided shell
-+snippet, and odds are that the user's shell snippet invokes lots of
-+commands in some long pipeline, so you will have lots and lots of forks.
-+For every. single. commit. That's a massive amount of overhead to
-+rename a few files.
-+
-+* filter-branch is written in shell, which is kind of slow. Naturally,
-+it makes sense to want to rewrite that in some other language. However,
-+filter-branch documentation states that several additional shell
-+functions are provided for users to call, e.g. 'map', 'skip_commit',
-+'git_commit_non_empty_tree'. If filter-branch itself isn't a shell
-+script, then in order to make those shell functions available to the
-+users' shell snippets you have to prepend the shell definitions of these
-+functions to every one of the users' shell snippets and thus make these
-+special shell functions be parsed with each and every commit.
-+
-+* filter-branch provides a --setup option which is a shell snippet that
-+can be sourced to make shell functions and variables available to all
-+other filters. If filter-branch is a shell script, it can simply eval
-+this shell snippet once at the beginning. If you try to fix performance
-+by making filter-branch not be a shell script, then you have to prepend
-+the setup shell snippet to all other filters and parse it with every
-+single commit.
-+
-+* filter-branch writes lots of files to $workdir/../map/ to keep a
-+mapping of commits, which it uses pruning commits and remapping to
-+ancestors and the map() command more generally. Other files like
-+$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,
-+$tempdir/tree-state are all created internally too. It is possible
-+(though strongly discouraged) that users could have accessed any of
-+these directly. Users even had a pointer to follow in the form of
-+Documentation that the 'map' command existed, which naturally uses the
-+$workdir/../map/* files. So, even if you don't have to edit files, for
-+strict backward compatibility you need to still write a bunch of files
-+to disk somewhere and keep them updated for every commit. You can claim
-+it was an implementation detail that users should not have depended
-+upon, but the truth is they've had a decade where they could so. So, if
-+you want full compatibility, it has to be there. Besides, the
-+regression tests depend on at least one of these details, specifying an
-+--index-filter that reaches down and grabs backup-refs from $tempdir,
-+and thus provides resourceful users who do google searches an example
-+that there are files there for them to read and grab and use. (And if
-+you want to pass the existing regression tests, you have to at least put
-+the backup-refs file there even if it's irrelevant to your
-+implementation otherwise.)
-+
-+All of that said, performance of filter-branch could be improved by
-+reimplementing it in a non-shell language and taking a couple small
-+liberties with backward compatibility (such as having it only run
-+filters on files changed within each commit). filter-repo provides a
-+demo script named
-+https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely]
-+which does exactly that and which passes all the git-filter-branch
-+regression tests. It's much faster than git-filter-branch, though it
-+suffers from all the same safety issues as git-filter-branch, and is
-+still glacially slow compared to
-+https://github.com/newren/git-filter-repo/[git filter-repo].
++where those filters can be run. That's a significant setup.
++
++* Further, several additional files are created or updated per commit by
++git-filter-branch. Some of these are for supporting the convenience
++functions provided by git-filter-branch (such as map()), while others
++are for keeping track of internal state (but could have also been
++accessed by user filters; one of git-filter-branch's regression tests
++does so). This essentially amounts to using the filesystem as an IPC
++mechanism between git-filter-branch and the user-provided filters.
++Disks tend to be a slow IPC mechanism, and writing these files also
++effectively represents a forced synchronization point between separate
++processes that we hit with every commit.
++
++* The user-provided shell commands will likely involve a pipeline of
++commands, resulting in the creation of many processes per commit.
++Creating and running another process takes a widely varying amount of
++time between operating systems, but on any platform it is very slow
++relative to invoking a function.
++
++* git-filter-branch itself is written in shell, which is kind of slow.
++This is the one performance issue that could be backward-compatibly
++fixed, but compared to the above problems that are intrinsic to the
++design of git-filter-branch, the language of the tool itself is a
++relatively minor issue.
++
++ ** Side note: Unfortunately, people tend to fixate on the
++ written-in-shell aspect and periodically ask if git-filter-branch
++ could be rewritten in another language to fix the performance
++ issues. Not only does that ignore the bigger intrinsic problems
++ with the design, it'd help less than you'd expect: if
++ git-filter-branch itself were not shell, then the convenience
++ functions (map(), skip_commit(), etc) and the `--setup` argument
++ could no longer be executed once at the beginning of the program
++ but would instead need to be prepended to every user filter (and
++ thus re-executed with every commit).
++
++The https://github.com/newren/git-filter-repo/[git filter-repo] tool is
++an alternative to git-filter-branch which does not suffer from these
++performance problems or the safety problems (mentioned below). For those
++with existing tooling which relies upon git-filter-branch, 'git
++repo-filter' also provides
++https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],
++a drop-in git-filter-branch replacement (with a few caveats). While
++filter-lamely suffers from all the same safety issues as
++git-filter-branch, it at least ameloriates the performance issues a
++little.
+
+[[SAFETY]]
+SAFETY
+------
+
-+filter-branch is riddled with gotchas resulting in various ways to
++git-filter-branch is riddled with gotchas resulting in various ways to
+easily corrupt repos or end up with a mess worse than what you started
+with:
+
@@ Documentation/git-filter-branch.txt: warned.
+history is in use for quite a while, at which point it's really hard to
+justify another flag-day for another rewrite.)
+
-+* Filenames with spaces (which are rare) are often mishandled by shell
-+snippets since they cause problems for shell pipelines. Not everyone is
-+familiar with find -print0, xargs -0, ls-files -z, etc. Even people who
-+are familiar with these may assume such needs are not relevant because
++* Filenames with spaces are often mishandled by shell snippets since
++they cause problems for shell pipelines. Not everyone is familiar with
++find -print0, xargs -0, git-ls-files -z, etc. Even people who are
++familiar with these may assume such needs are not relevant because
+someone else renamed any such files in their repo back before the person
+doing the filtering joined the project. And, often, even those familiar
+with handling arguments with spaces my not do so just because they
+aren't in the mindset of thinking about everything that could possibly
+go wrong.
+
-+* Non-ascii filenames (which are rare) can be silently removed despite
-+being in a desired directory. The desire to select paths to keep often
-+use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.
-+ls-files will only quote filenames if needed so folks may not notice
-+that one of the files didn't match the regex, again until it's much too
-+late. Yes, someone who knows about core.quotePath can avoid this
-+(unless they have other special characters like \t, \n, or "), and
-+people who use ls-files -z with something other than grep can avoid
-+this, but that doesn't mean they will.
++* Non-ascii filenames can be silently removed despite being in a desired
++directory. The desire to select paths to keep often use pipelines like
++`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will
++only quote filenames if needed so folks may not notice that one of the
++files didn't match the regex, again until it's much too late. Yes,
++someone who knows about core.quotePath can avoid this (unless they have
++other special characters like \t, \n, or "), and people who use ls-files
++-z with something other than grep can avoid this, but that doesn't mean
++they will.
+
+* Similarly, when moving files around, one can find that filenames with
+non-ascii or special characters end up in a different directory, one
@@ Documentation/git-filter-branch.txt: warned.
+that it can and has manifested as a problem.)
+
+* It's far too easy to accidentally mix up old and new history. It's
-+still possible with any tool, but filter-branch almost invites it. If
-+we're lucky, the only downside is users getting frustrated that they
-+don't know how to shrink their repo and remove the old stuff. If we're
-+unlucky, they merge old and new history and end up with multiple
-+"copies" of each commit, some of which have unwanted or sensitive files
-+and others which don't. This comes about in multiple different ways:
++still possible with any tool, but git-filter-branch almost invites it.
++If lucky, the only downside is users getting frustrated that they don't
++know how to shrink their repo and remove the old stuff. If unlucky,
++they merge old and new history and end up with multiple "copies" of each
++commit, some of which have unwanted or sensitive files and others which
++don't. This comes about in multiple different ways:
+
+ ** the default to only doing a partial history rewrite ('--all' is not
-+ the default and over 80% of the examples in the manpage don't use
-+ it)
++ the default and few examples show it)
+
+ ** the fact that there's no automatic post-run cleanup
+
+ ** the fact that --tag-name-filter (when used to rename tags) doesn't
-+ remove the old tags but just adds new ones with the new name (this
-+ manpage has documented this for a long time so it's presumably not
-+ a "bug" even though it feels like it)
++ remove the old tags but just adds new ones with the new name
+
+ ** the fact that little educational information is provided to inform
+ users of the ramifications of a rewrite and how to avoid mixing old
@@ Documentation/git-filter-branch.txt: warned.
+* Annotated tags can be accidentally converted to lightweight tags, due
+to either of two issues:
+
-+ . Someone can do a history rewrite, realize they messed up, restore
-+ from the backups in refs/original/, and then redo their
-+ filter-branch command. (The backup in refs/original/ is not a real
-+ backup; it dereferences tags first.)
++ ** Someone can do a history rewrite, realize they messed up, restore
++ from the backups in refs/original/, and then redo their
++ git-filter-branch command. (The backup in refs/original/ is not a
++ real backup; it dereferences tags first.)
+
-+ . Running filter-branch with either --tags or --all in your <rev-list
-+ options>. In order to retain annotated tags as annotated, you must
-+ use --tag-name-filter (and must not have restored from
-+ refs/original/ in a previously botched rewrite).
++ ** Running git-filter-branch with either --tags or --all in your
++ <rev-list options>. In order to retain annotated tags as
++ annotated, you must use --tag-name-filter (and must not have
++ restored from refs/original/ in a previously botched rewrite).
+
+* Any commit messages that specify an encoding will become corrupted
-+by the rewrite; filter-branch ignores the encoding, takes the original
++by the rewrite; git-filter-branch ignores the encoding, takes the original
+bytes, and feeds it to commit-tree without telling it the proper
-+encoding. (This happens whether or not --msg-filter is used, though I
-+suspect --msg-filter provides additional ways to really mess things
-+up).
++encoding. (This happens whether or not --msg-filter is used.)
+
+* Commit messages (even if they are all UTF-8) by default become
+corrupted due to not being updated -- any references to other commit
@@ Documentation/git-filter-branch.txt: warned.
+authors and committers, missing taggers.
+
+* If the user provides a --tag-name-filter that maps multiple tags to
-+the same name, no warning or error is provided; filter-branch simply
++the same name, no warning or error is provided; git-filter-branch simply
+overwrites each tag in some undocumented pre-defined order resulting in
-+only one tag at the end. If you try to "fix" this bug in filter-branch
-+and make it error out and warn the user instead, one of the
-+filter-branch regression tests will fail. (So, if you are trying to
-+make a backward compatible reimplementation you have to add extra code
-+to detect collisions and make sure that only the lexicographically last
-+one is rewritten to avoid fast-import from seeing both since fast-import
-+will naturally do the sane thing and error out if told to write the same
-+tag more than once.)
++only one tag at the end. (A git-filter-branch regression test requires
++this.)
+
-+Also, the poor performance of filter-branch often leads to safety issues:
++Also, the poor performance of git-filter-branch often leads to safety issues:
+
+* Coming up with the correct shell snippet to do the filtering you want
+is sometimes difficult unless you're just doing a trivial modification
-+such as deleting a couple files. People have often come to me for help,
-+so I should be practiced and an expert, but even for fairly simple cases
-+I still sometimes take over 10 minutes and several iterations to get
-+the right commands -- and that's assuming they are working on a tiny
-+repository. Unfortunately, people often learn if the snippet is right
-+or wrong by trying it out, but the rightness or wrongness can vary
-+depending on special circumstances (spaces in filenames, non-ascii
-+filenames, funny author names or emails, invalid timezones, presence of
-+grafts or replace objects, etc.), meaning they may have to wait a long
-+time, hit an error, then restart. The performance of filter-branch is
-+so bad that this cycle is painful, reducing the time available to
-+carefully re-check (to say nothing about what it does to the patience of
-+the person doing the rewrite even if they do technically have more time
-+available). This problem is extra compounded because errors from broken
-+filters may not be shown for a long time and/or get lost in a sea of
-+output. Even worse, broken filters often just result in silent
-+incorrect rewrites.
++such as deleting a couple files. Unfortunately, people often learn if
++the snippet is right or wrong by trying it out, but the rightness or
++wrongness can vary depending on special circumstances (spaces in
++filenames, non-ascii filenames, funny author names or emails, invalid
++timezones, presence of grafts or replace objects, etc.), meaning they
++may have to wait a long time, hit an error, then restart. The
++performance of git-filter-branch is so bad that this cycle is painful,
++reducing the time available to carefully re-check (to say nothing about
++what it does to the patience of the person doing the rewrite even if
++they do technically have more time available). This problem is extra
++compounded because errors from broken filters may not be shown for a
++long time and/or get lost in a sea of output. Even worse, broken
++filters often just result in silent incorrect rewrites.
+
+* To top it all off, even when users finally find working commands, they
+naturally want to share them. But they may be unaware that their repo
4: 1dbca82408 = 4: ca8e124cb3 t9902: use a non-deprecated command for testing
5: 762d63d6a5 < -: ---------- Remove git-filter-branch, it is now external to git.gitSignature · 1 line
2.23.0.38.g892688c90e
PATCH v4 1/4 t6006: simplify and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of time ./t6006-rev-list-format.sh) by about 11% on Linux and by 13% on Mac.
--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
t/t6006-rev-list-format.sh +2 -3
@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Signature · 1 line
2.23.0.38.g892688c90e
Re: [PATCH v4 1/4] t6006: simplify and optimize empty message test
Hi Elijah,
Quoted 5 lines · Elijah Newren
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of
time ./t6006-rev-list-format.sh) by about 11% on Linux and by 13% on Mac.
A similar effect can be observed on my Windows laptop: from 25s to 21s, i.e. ~15%.
Thanks,
Dscho
PATCH v4 2/4 t3427: accelerate this test by using fast-export and fast-import
In reply to Elijah Newrenfast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):
Linux: 4.305s -> 3.684s (~17% speedup)
Mac: 10.128s -> 7.038s (~30% speedup)
--- t/t3427-rebase-subtree.sh | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
t/t3427-rebase-subtree.sh +15 -9
@@ -7,10 +7,16 @@ This test runs git rebase and tests the subtree strategy. . ./test-lib.sh . "$TEST_DIRECTORY"/lib-rebase.sh -commit_message() { +commit_message () { git log --pretty=format:%s -1 "$1" } +extract_files_subtree () { + git fast-export --no-data HEAD -- files_subtree/ | + sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" | + git fast-import --force --quiet +} + test_expect_success 'setup' ' test_commit README && mkdir files &&
@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-preserve-merges-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master4"
@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-preserve-merges-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "files_subtree/master5"
@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-keep-empty-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-keep-empty-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' ' reset_rebase && git checkout -b rebase-keep-empty-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' reset_rebase && git checkout -b rebase-onto-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' reset_rebase && git checkout -b rebase-onto-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' test_expect_failure 'Rebase -Xsubtree --onto empty commit' ' reset_rebase && git checkout -b rebase-onto-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
Signature · 1 line
2.23.0.38.g892688c90e
Re: [PATCH v4 2/4] t3427: accelerate this test by using fast-export and fast-import
Hi Elijah,
Quoted 9 lines · Elijah Newren
fast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on two laptops that I have access to (measured via
time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):Linux: 4.305s -> 3.684s (~17% speedup) Mac: 10.128s -> 7.038s (~30% speedup)
This patch seems to accelerate t3427 on my Windows laptop, too, from ~1m37s to ~1m17s, i.e. ~20%.
Ciao,
Dscho
PATCH v4 3/4 Recommend git-filter-repo instead of git-filter-branch
In reply to Elijah Newrenfilter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). Many of these problems are unobtrusive and can easily go undiscovered until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to both filter-branch and its manpage recommending that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.
--- Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 272 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 13 ++ 9 files changed, 286 insertions(+), 59 deletions(-)
Documentation/git-fast-export.txt +3 -3
@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------
Documentation/git-filter-branch.txt +242 -30
284 lines not on this page · raw message
Documentation/git-gc.txt +8 -9
@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is using
Documentation/git-rebase.txt +2 -1
@@ -832,7 +832,8 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like + https://github.com/newren/git-filter-repo[`filter-repo`]. The easy case
Documentation/git-replace.txt +5 -5
@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a
@@ -148,13 +148,13 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1] linkgit:git-var[1] linkgit:git[1] +https://github.com/newren/git-filter-repo[git-filter-repo] GIT ---
Documentation/git-svn.txt +5 -5
@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git -and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows -reformatting of metadata for ease-of-reading and rewriting authorship -info for non-"svn.authorsFile" users. +reports, and archives. If you plan to eventually migrate from SVN to +Git and are certain about dropping SVN history, consider +https://github.com/newren/git-filter-repo[git-filter-repo] instead. +filter-repo also allows reformatting of metadata for ease-of-reading +and rewriting authorship info for non-"svn.authorsFile" users. svn.useSvmProps:: svn-remote.<name>.useSvmProps::
Documentation/githooks.txt +6 -4
@@ -425,10 +425,12 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent -arguments may be passed in the future. +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or +https://github.com/newren/git-filter-repo[git-filter-repo] typically +do not call it!). Its first argument denotes the command it was +invoked by: currently one of `amend` or `rebase`. Further +command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in the format
contrib/svn-fe/svn-fe.txt +2 -2
@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS
@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
git-filter-branch.sh +13 -0
@@ -83,6 +83,19 @@ set_ident () { finish_ident COMMITTER } +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then + cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1. + +EOF + sleep 5 +fi + USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>] [--tree-filter <command>] [--index-filter <command>] [--parent-filter <command>] [--msg-filter <command>]
Signature · 1 line
2.23.0.38.g892688c90e
PATCH v4 4/4 t9902: use a non-deprecated command for testing
In reply to Elijah Newrent9902 had a list of three random porcelain commands as a sanity check, one of which was filter-branch. Since we are recommending people not use filter-branch, let's update this test to use rebase instead of filter-branch.
--- t/t9902-completion.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
t/t9902-completion.sh +6 -6
@@ -28,10 +28,10 @@ complete () # # (2) A test makes sure that common subcommands are included in the # completion for "git <TAB>", and a plumbing is excluded. "add", -# "filter-branch" and "ls-files" are listed for this. +# "rebase" and "ls-files" are listed for this. -GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files' -GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch' +GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files' +GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase' . "$GIT_BUILD_DIR/contrib/completion/git-completion.bash"
@@ -1392,12 +1392,12 @@ test_expect_success 'basic' ' # built-in grep -q "^add \$" out && # script - grep -q "^filter-branch \$" out && + grep -q "^rebase \$" out && # plumbing ! grep -q "^ls-files \$" out && - run_completion "git f" && - ! grep -q -v "^f" out + run_completion "git r" && + ! grep -q -v "^r" out ' test_expect_success 'double dash "git" itself' '
Signature · 1 line
2.23.0.38.g892688c90e
v5 cover Warn about git-filter-branch usage and avoid it
In reply to Elijah NewrenIt's been about 5 days with no further feedback, other than some timings from Dscho for Windows showing that my fixes help there too. So, I did one last re-read, made a couple small wording tweaks, and am resending as ready for inclusion.
Changes since v4:
* Included the windows timings from Dscho in the commit messages for the first two perf patches
* A few slight wording tweaks to the manpage
Elijah Newren (4):
t6006: simplify and optimize empty message test
t3427: accelerate this test by using fast-export and fast-import Recommend git-filter-repo instead of git-filter-branch
t9902: use a non-deprecated command for testing
Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 13 ++ t/t3427-rebase-subtree.sh | 24 ++- t/t6006-rev-list-format.sh | 5 +- t/t9902-completion.sh | 12 +- 12 files changed, 310 insertions(+), 77 deletions(-)
Range-diff:
1: 7ddbeea2ca ! 1: ccea0e5846 t6006: simplify and optimize empty message test
@@ Commit message
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of `time
- ./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on
- Mac.
+ ./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and
+ about 15% on Windows.
2: e1e63189c1 ! 2: 6d73135006 t3427: accelerate this test by using fast-export and fast-import
@@ Commit message
fast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on
- systems with a slow fork. Timings from before and after on two laptops
- that I have access to (measured via time ./t3427-rebase-subtree.sh,
- i.e. including everything in this test -- not just the filter-branch or
- fast-export/fast-import pair):
+ systems with a slow fork. Timings from before and after on a few
+ laptops that I or others measured on (measured via `time
+ ./t3427-rebase-subtree.sh`, i.e. including everything in this test --
+ not just the filter-branch or fast-export/fast-import pair):
- Linux: 4.305s -> 3.684s (~17% speedup)
- Mac: 10.128s -> 7.038s (~30% speedup)
+ Linux: 4.305s -> 3.684s (~17% speedup)
+ Mac: 10.128s -> 7.038s (~30% speedup)
+ Windows: 1m 37s -> 1m 17s (~26% speedup)
3: ed6505584f ! 3: 2f225c8697 Recommend git-filter-repo instead of git-filter-branch
@@ Documentation/git-filter-branch.txt: warned.
+document or provide to a coworker, who then runs them on a different OS +where the same commands are not working/tested (some examples in the +git-filter-branch manpage are also affected by this). BSD vs. GNU
-+userland differences can really bite. If you're lucky, you get ugly -+error messages spewed. But just as likely, the commands either don't do -+the filtering requested, or silently corrupt making some unwanted -+change. The unwanted change may only affect a few commits, so it's not -+necessarily obvious either. (The fact that problems won't necessarily -+be obvious means they are likely to go unnoticed until the rewritten -+history is in use for quite a while, at which point it's really hard to -+justify another flag-day for another rewrite.)
++userland differences can really bite. If lucky, error messages are ++spewed. But just as likely, the commands either don't do the filtering ++requested, or silently corrupt by making some unwanted change. The ++unwanted change may only affect a few commits, so it's not necessarily ++obvious either. (The fact that problems won't necessarily be obvious ++means they are likely to go unnoticed until the rewritten history is in ++use for quite a while, at which point it's really hard to justify ++another flag-day for another rewrite.)
+
+* Filenames with spaces are often mishandled by shell snippets since +they cause problems for shell pipelines. Not everyone is familiar with +find -print0, xargs -0, git-ls-files -z, etc. Even people who are
-+familiar with these may assume such needs are not relevant because ++familiar with these may assume such flags are not relevant because
+someone else renamed any such files in their repo back before the person
-+doing the filtering joined the project. And, often, even those familiar -+with handling arguments with spaces my not do so just because they ++doing the filtering joined the project. And often, even those familiar ++with handling arguments with spaces may not do so just because they
+aren't in the mindset of thinking about everything that could possibly +go wrong.
+
+* Non-ascii filenames can be silently removed despite being in a desired
-+directory. The desire to select paths to keep often use pipelines like ++directory. Keeping only wanted paths is often done using pipelines like
+git ls-files | grep -v ^WANTED_DIR/ | xargs git rm. ls-files will
-+only quote filenames if needed so folks may not notice that one of the -+files didn't match the regex, again until it's much too late. Yes, -+someone who knows about core.quotePath can avoid this (unless they have -+other special characters like \t, \n, or "), and people who use ls-files -+-z with something other than grep can avoid this, but that doesn't mean -+they will.
++only quote filenames if needed, so folks may not notice that one of the ++files didn't match the regex (at least not until it's much too late). ++Yes, someone who knows about core.quotePath can avoid this (unless they ++have other special characters like \t, \n, or "), and people who use ++ls-files -z with something other than grep can avoid this, but that ++doesn't mean they will.
+
+* Similarly, when moving files around, one can find that filenames with +non-ascii or special characters end up in a different directory, one
@@ Documentation/git-filter-branch.txt: warned.
+the same name, no warning or error is provided; git-filter-branch simply +overwrites each tag in some undocumented pre-defined order resulting in +only one tag at the end. (A git-filter-branch regression test requires
-+this.)
++this surprising behavior.)
+
-+Also, the poor performance of git-filter-branch often leads to safety issues:
++Also, the poor performance of git-filter-branch often leads to safety ++issues:
+
+* Coming up with the correct shell snippet to do the filtering you want +is sometimes difficult unless you're just doing a trivial modification
4: ca8e124cb3 = 4: 048eba375b t9902: use a non-deprecated command for testing
Signature · 1 line
2.23.0.39.gf92d9de5c3
PATCH v5 1/4 t6006: simplify and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of time ./t6006-rev-list-format.sh) by about 11% on Linux, 13% on Mac, and about 15% on Windows.
--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
t/t6006-rev-list-format.sh +2 -3
@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Signature · 1 line
2.23.0.39.gf92d9de5c3
Re: [PATCH v5 1/4] t6006: simplify and optimize empty message test
Quoted 6 lines · Elijah Newren
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
The result of filter-branch seems to have one empty line as the body (i.e. "echo X; git cat-file commit A; echo Y" will show two blank lines between the committer line and Y), while "--allow-empty-message" does not leave any body (i.e. the same will give you only one blank line there).
Was this test verifying the right thing in the first place, I have to wonder.
IOW,
git commit --allow-empty --cleanup=verbatim -m "$LF" &&
would be more faithful conversion of the original (and hopefully just as performant).
Quoted 5 lines
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of
time ./t6006-rev-list-format.sh) by about 11% on Linux, 13% on Mac, and about 15% on Windows.
Quite an improvement ;-)
Quoted 22 lines
Signed-off-by Elijah Newren <[email protected]>--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)t/t6006-rev-list-format.sh +2
-3@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Re: [PATCH v5 1/4] t6006: simplify and optimize empty message test
Quoted 10 lines · Junio C Hamano
Quoted 6 lines · Elijah Newren
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be empty. This test was written this way because the --allow-empty-message option to git commit did not exist at the time. Simplify this test and avoid the need to invoke filter-branch by just using --allow-empty-message when creating the commit.
Quoted 5 lines
The result of filter-branch seems to have one empty line as the body (i.e. "echo X; git cat-file commit A; echo Y" will show two blank lines between the committer line and Y), while "--allow-empty-message" does not leave any body (i.e. the same will give you only one blank line there).
Ah, good catch. I checked out the commit before 1fb5fdd25f0 ("rev-list: fix --pretty=oneline with empty message", 2010-03-21), to try and see the error before that testcase was introduced. I tried it on a repo with both an actual empty commit message, and one with a commit message consisting solely of a newline. Both styles exhibited the bug that the testcase was introduced to guard against.
Quoted 9 lines
Was this test verifying the right thing in the first place, I have to wonder.
IOW,
git commit --allow-empty --cleanup=verbatim -m "$LF" &&would be more faithful conversion of the original (and hopefully just as performant).
Yeah, it'd be a more faithful conversion of the original, though the original didn't match the testcase description nor the commit message (it claimed it was testing with an empty message). Also, in terms of future proofing, any code changes are more likely to omit a needed trailing LF if the commit message doesn't have one than if it does, so I think it's a more robust test with this change.
I can update the commit message to explain this, or, if you prefer, I could duplicate the testcase and tweak the second as you suggest so we test both with and without the LF. What's your preference?
Quoted 30 lines
Quoted 5 lines
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of
time ./t6006-rev-list-format.sh) by about 11% on Linux, 13% on Mac, and about 15% on Windows.Quite an improvement ;-)
Quoted 22 lines
Signed-off-by Elijah Newren <[email protected]>--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)t/t6006-rev-list-format.sh +2
-3@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --allow-empty-message && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Re: [PATCH v5 1/4] t6006: simplify and optimize empty message test
Quoted 6 lines · Elijah Newren
Ah, good catch. I checked out the commit before
1fb5fdd25f0("rev-list: fix --pretty=oneline with empty message", 2010-03-21), to try and see the error before that testcase was introduced. I tried it on a repo with both an actual empty commit message, and one with a commit message consisting solely of a newline. Both styles exhibited the bug that the testcase was introduced to guard against.
That's a good thing to know to decide what is a reasonable thing to do here.
As we are creating two commits, perhaps adding one with and another without the extra blank line may give us more diversity, and explaining why we are adding two slightly different one (i.e. because the original bug was there for both shapes of commits) would help us not wasting the time we already spent discussing this change ;-)
Of course, we can alternatively just keep the patch as-is and update the explanation as to why we are testing with commits different from the original when we are supposed to be making this change for performance reasons (i.e. the symptom manifests either way, so why not using the form that is easier to create?).
Thanks for working on this ;-)
PATCH v5 2/4 t3427: accelerate this test by using fast-export and fast-import
In reply to Elijah Newrenfast-export and fast-import can easily handle the simple rewrite that was being done by filter-branch, and should be significantly faster on systems with a slow fork. Timings from before and after on a few laptops that I or others measured on (measured via time ./t3427-rebase-subtree.sh, i.e. including everything in this test -- not just the filter-branch or fast-export/fast-import pair):
Linux: 4.305s -> 3.684s (~17% speedup)
Mac: 10.128s -> 7.038s (~30% speedup)
Windows: 1m 37s -> 1m 17s (~26% speedup)
--- t/t3427-rebase-subtree.sh | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
t/t3427-rebase-subtree.sh +15 -9
@@ -7,10 +7,16 @@ This test runs git rebase and tests the subtree strategy. . ./test-lib.sh . "$TEST_DIRECTORY"/lib-rebase.sh -commit_message() { +commit_message () { git log --pretty=format:%s -1 "$1" } +extract_files_subtree () { + git fast-export --no-data HEAD -- files_subtree/ | + sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" | + git fast-import --force --quiet +} + test_expect_success 'setup' ' test_commit README && mkdir files &&
@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-preserve-merges-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master4"
@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-preserve-merges-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "files_subtree/master5"
@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' ' reset_rebase && git checkout -b rebase-keep-empty-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' ' reset_rebase && git checkout -b rebase-keep-empty-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \ 'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' ' reset_rebase && git checkout -b rebase-keep-empty-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' reset_rebase && git checkout -b rebase-onto-4 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' ' test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' reset_rebase && git checkout -b rebase-onto-5 master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' ' test_expect_failure 'Rebase -Xsubtree --onto empty commit' ' reset_rebase && git checkout -b rebase-onto-empty master && - git filter-branch --prune-empty -f --subdirectory-filter files_subtree && + extract_files_subtree && git commit -m "Empty commit" --allow-empty && git rebase -Xsubtree=files_subtree --onto files-master master && verbose test "$(commit_message HEAD)" = "Empty commit"
Signature · 1 line
2.23.0.39.gf92d9de5c3
Re: [PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import
t/t3427-rebase-subtree.sh This test runs git rebase and tests the subtree strategy. · line 18
Quoted 5 lines · Elijah Newren
t/t3427-rebase-subtree.sh +5
-0+extract_files_subtree () { + git fast-export --no-data HEAD -- files_subtree/ | + sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" | + git fast-import --force --quiet +}
Clever, if a bit filthy ;-). We expect to see something like
M 100644 dead...beef files_subtree/bar M 100755 c0f.....fee files_subtree/foo
in the --no-data output, and the assumption here is that 40-hex followed by " files_subtree/" would never appear anywhere in the stream other than these tree dump, so the sed script can rewrite the above to
M 100644 dead...beef bar M 100755 c0f.....fee foo
by getting rid of the leading directory name (plus the slash at the end).
Thanks.
Re: [PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import
Quoted 7 lines · Junio C Hamano
t/t3427-rebase-subtree.sh This test runs git rebase and tests the subtree strategy. · line 18
Quoted 5 lines · Elijah Newren
t/t3427-rebase-subtree.sh +5
-0+extract_files_subtree () { + git fast-export --no-data HEAD -- files_subtree/ | + sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" | + git fast-import --force --quiet +}
This change has obvious interactions with Dscho's d51b771d ("t3427:
move the filter-branch invocation into the setup case", 2019-07-31) that is still in flight, but in a good way. There only needs a single callsite for the above helper function after that step.
I think I'll discard this step from the "move us closer to deprecate filter-branch" topic, and ask you and Dscho to work together to have it or its moral equivalent included as part of js/rebase-r-strategy topic.
Thanks.
Re: [PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import
Quoted 17 lines · Junio C Hamano
Quoted 7 lines · Junio C Hamano
t/t3427-rebase-subtree.sh This test runs git rebase and tests the subtree strategy. · line 18
Quoted 5 lines · Elijah Newren
t/t3427-rebase-subtree.sh +5
-0+extract_files_subtree () { + git fast-export --no-data HEAD -- files_subtree/ | + sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" | + git fast-import --force --quiet +}This change has obvious interactions with Dscho's
d51b771d("t3427:
move thefilter-branchinvocation into thesetupcase", 2019-07-31) that is still in flight, but in a good way. There only needs a single callsite for the above helper function after that step.
Quoted 4 lines
I think I'll discard this step from the "move us closer to deprecate filter-branch" topic, and ask you and Dscho to work together to have it or its moral equivalent included as part of js/rebase-r-strategy topic.
Sounds good. I'll resubmit it separately as a patch on top of his topic.
PATCH v5 3/4 Recommend git-filter-repo instead of git-filter-branch
In reply to Elijah Newrenfilter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). Many of these problems are unobtrusive and can easily go undiscovered until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to both filter-branch and its manpage recommending that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.
--- Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 13 ++ 9 files changed, 287 insertions(+), 59 deletions(-)
Documentation/git-fast-export.txt +3 -3
@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------
Documentation/git-filter-branch.txt +243 -30
285 lines not on this page · raw message
Documentation/git-gc.txt +8 -9
@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is using
Documentation/git-rebase.txt +2 -1
@@ -832,7 +832,8 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like + https://github.com/newren/git-filter-repo[`filter-repo`]. The easy case
Documentation/git-replace.txt +5 -5
@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a
@@ -148,13 +148,13 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1] linkgit:git-var[1] linkgit:git[1] +https://github.com/newren/git-filter-repo[git-filter-repo] GIT ---
Documentation/git-svn.txt +5 -5
@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git -and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows -reformatting of metadata for ease-of-reading and rewriting authorship -info for non-"svn.authorsFile" users. +reports, and archives. If you plan to eventually migrate from SVN to +Git and are certain about dropping SVN history, consider +https://github.com/newren/git-filter-repo[git-filter-repo] instead. +filter-repo also allows reformatting of metadata for ease-of-reading +and rewriting authorship info for non-"svn.authorsFile" users. svn.useSvmProps:: svn-remote.<name>.useSvmProps::
Documentation/githooks.txt +6 -4
@@ -425,10 +425,12 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent -arguments may be passed in the future. +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or +https://github.com/newren/git-filter-repo[git-filter-repo] typically +do not call it!). Its first argument denotes the command it was +invoked by: currently one of `amend` or `rebase`. Further +command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in the format
contrib/svn-fe/svn-fe.txt +2 -2
@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS
@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
git-filter-branch.sh +13 -0
@@ -83,6 +83,19 @@ set_ident () { finish_ident COMMITTER } +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then + cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1. + +EOF + sleep 5 +fi + USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>] [--tree-filter <command>] [--index-filter <command>] [--parent-filter <command>] [--msg-filter <command>]
Signature · 1 line
2.23.0.39.gf92d9de5c3
Re: [PATCH v5 3/4] Recommend git-filter-repo instead of git-filter-branch
git-filter-branch.sh set_ident () { · line 87
Quoted 10 lines · Elijah Newren
git-filter-branch.sh +2
-0@@ -83,6 +83,19 @@ set_ident () { finish_ident COMMITTER } +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then
This is probably the only place where [] instead of "test" is used in our shell scripts.
if test -z "$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" then
...
git-filter-branch.sh set_ident () { · line 97
Quoted 10 lines
git-filter-branch.sh +10
-0+ cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1. + +EOF + sleep 5 +fi
This should say it is "sleeping while showing the message and can safely be killed before starting to do any harm"; alternatively it should lose the "sleep". The user would have fear against typing ^C to get out of a bulk history rewrite command, and the message itself is making the fear worse. If your goal is to discourage its use, then it would be a good idea to make it clear when it is safe to kill it before going and studying the alternative. Otherwise, the sleep does not help that much---the main complaint is that filter branch is too slow, so the user has plenty of time to read the message anyway, right? ;-)
Quoted 3 lines
git-filter-branch.sh +0
-0USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>] [--tree-filter <command>] [--index-filter <command>] [--parent-filter <command>] [--msg-filter <command>]
Re: [PATCH v5 3/4] Recommend git-filter-repo instead of git-filter-branch
Quoted 17 lines · Junio C Hamano
git-filter-branch.sh set_ident () { · line 87
Quoted 10 lines · Elijah Newren
git-filter-branch.sh +2
-0@@ -83,6 +83,19 @@ set_ident () { finish_ident COMMITTER } +if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \ + -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; thenThis is probably the only place where [] instead of "test" is used in our shell scripts.
Quoted 3 lines
if test -z "$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" then
...
Yeah, git-filter-branch.sh has approximately twice as many uses of [] than "test", so it seemed in line with its coding style. I can switch it over.
Quoted 13 lines
git-filter-branch.sh set_ident () { · line 97
Quoted 10 lines
git-filter-branch.sh +10
-0+ cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Please use an alternative filtering tool such as 'git + filter-repo' (https://github.com/newren/git-filter-repo/) instead. + See the filter-branch manual page for more details; to squelch + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1. + +EOF + sleep 5 +fiThis should say it is "sleeping while showing the message and can safely be killed before starting to do any harm"; alternatively it
Quoted 8 lines
should lose the "sleep". The user would have fear against typing ^C to get out of a bulk history rewrite command, and the message itself is making the fear worse. If your goal is to discourage its use, then it would be a good idea to make it clear when it is safe to kill it before going and studying the alternative. Otherwise, the sleep does not help that much---the main complaint is that filter branch is too slow, so the user has plenty of time to read the message anyway, right? ;-)
Makes sense; will fix.
PATCH v5 4/4 t9902: use a non-deprecated command for testing
In reply to Elijah Newrent9902 had a list of three random porcelain commands as a sanity check, one of which was filter-branch. Since we are recommending people not use filter-branch, let's update this test to use rebase instead of filter-branch.
--- t/t9902-completion.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
t/t9902-completion.sh +6 -6
@@ -28,10 +28,10 @@ complete () # # (2) A test makes sure that common subcommands are included in the # completion for "git <TAB>", and a plumbing is excluded. "add", -# "filter-branch" and "ls-files" are listed for this. +# "rebase" and "ls-files" are listed for this. -GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files' -GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch' +GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files' +GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase' . "$GIT_BUILD_DIR/contrib/completion/git-completion.bash"
@@ -1392,12 +1392,12 @@ test_expect_success 'basic' ' # built-in grep -q "^add \$" out && # script - grep -q "^filter-branch \$" out && + grep -q "^rebase \$" out && # plumbing ! grep -q "^ls-files \$" out && - run_completion "git f" && - ! grep -q -v "^f" out + run_completion "git r" && + ! grep -q -v "^r" out ' test_expect_success 'double dash "git" itself' '
Signature · 1 line
2.23.0.39.gf92d9de5c3
v6 cover Warn about git-filter-branch usage and avoid it
In reply to Elijah NewrenChanges since v5 (full range-diff below):
* Dropped patch 3 (which was rebased on top of js/rebase-r-strategy and submitted separately)[1]
* Updated t6006 to include both an empty commit message and a commit message with just a line feed
* Made the two small tweaks Junio suggested to git-filter-branch.sh
[1] https://public-inbox.org/git/[email protected]/
Elijah Newren (3):
t6006: simplify, fix, and optimize empty message test Recommend git-filter-repo instead of git-filter-branch
t9902: use a non-deprecated command for testing
Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 14 ++ t/t6006-rev-list-format.sh | 5 +- t/t9902-completion.sh | 12 +- 11 files changed, 296 insertions(+), 68 deletions(-)
Range-diff:
1: ccea0e5846 ! 1: d5370568a4 t6006: simplify and optimize empty message test
@@ Metadata
Author: Elijah Newren <[email protected]>
## Commit message ##
- t6006: simplify and optimize empty message test
+ t6006: simplify, fix, and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite
- the commit messages to be empty. This test was written this way because
- the --allow-empty-message option to git commit did not exist at the
- time. Simplify this test and avoid the need to invoke filter-branch by
- just using --allow-empty-message when creating the commit.
+ the commit messages to be "empty". This test was introduced in commit
+ 1fb5fdd25f0 ("rev-list: fix --pretty=oneline with empty message",
+ 2010-03-21) and written this way because the --allow-empty-message
+ option to git commit did not exist at the time.
+
+ However, the filter-branch invocation used differed slightly from
+ --allow-empty-message in that it would have a commit message consisting
+ solely of a single newline, and as such was not testing what the
+ original commit intended to test. Since both a truly empty commit
+ message and a commit message with a single linefeed could trigger the
+ original bug, modify the test slightly to include an example of each.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall
@@ t/t6006-rev-list-format.sh: test_expect_success 'reflog identity' '
- git commit -m "dummy" --allow-empty &&
- git commit -m "dummy" --allow-empty &&
- git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. &&
-+ git commit --allow-empty --allow-empty-message &&
++ git commit --allow-empty --cleanup=verbatim -m "$LF" &&
+ git commit --allow-empty --allow-empty-message &&
git rev-list --oneline HEAD >test.txt &&
test_line_count = 5 test.txt &&
2: 6d73135006 < -: ---------- t3427: accelerate this test by using fast-export and fast-import 3: 2f225c8697 ! 2: 8635410b88 Recommend git-filter-repo instead of git-filter-branch
@@ git-filter-branch.sh: set_ident () {
finish_ident COMMITTER
}
-+if [ -z "$FILTER_BRANCH_SQUELCH_WARNING" -a \
-+ -z "$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ]; then
++if test -z "$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" ++then
+ cat <<EOF
+WARNING: git-filter-branch has a glut of gotchas generating mangled history
-+ rewrites. Please use an alternative filtering tool such as 'git
-+ filter-repo' (https://github.com/newren/git-filter-repo/) instead.
-+ See the filter-branch manual page for more details; to squelch
-+ this warning, set FILTER_BRANCH_SQUELCH_WARNING=1.
-+
++ rewrites. Hit Ctrl-C before proceeding to abort, then use an
++ alternative filtering tool such as 'git filter-repo'
++ (https://github.com/newren/git-filter-repo/) instead. See the
++ filter-branch manual page for more details; to squelch this warning,
++ set FILTER_BRANCH_SQUELCH_WARNING=1.
+EOF
-+ sleep 5
++ sleep 10
++ printf "Proceeding with filter-branch...\n\n"
+fi
+
USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>]
4: 048eba375b = 3: 19edb94ec2 t9902: use a non-deprecated command for testing
Signature · 1 line
2.23.0.3.g19edb94ec2
PATCH v6 1/3 t6006: simplify, fix, and optimize empty message test
Test t6006.71 ("oneline with empty message") was creating two commits with simple commit messages, and then running filter-branch to rewrite the commit messages to be "empty". This test was introduced in commit 1fb5fdd25f0 ("rev-list: fix --pretty=oneline with empty message", 2010-03-21) and written this way because the --allow-empty-message option to git commit did not exist at the time.
However, the filter-branch invocation used differed slightly from --allow-empty-message in that it would have a commit message consisting solely of a single newline, and as such was not testing what the original commit intended to test. Since both a truly empty commit message and a commit message with a single linefeed could trigger the original bug, modify the test slightly to include an example of each.
Despite only being one piece of the 71st test and there being 73 tests overall, this small change to just this one test speeds up the overall execution time of t6006 (as measured by the best of 3 runs of time ./t6006-rev-list-format.sh) by about 11% on Linux, 13% on Mac, and about 15% on Windows.
--- t/t6006-rev-list-format.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
t/t6006-rev-list-format.sh +2 -3
@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' ' ' test_expect_success 'oneline with empty message' ' - git commit -m "dummy" --allow-empty && - git commit -m "dummy" --allow-empty && - git filter-branch --msg-filter "sed -e s/dummy//" HEAD^^.. && + git commit --allow-empty --cleanup=verbatim -m "$LF" && + git commit --allow-empty --allow-empty-message && git rev-list --oneline HEAD >test.txt && test_line_count = 5 test.txt && git rev-list --oneline --graph HEAD >testg.txt &&
Signature · 1 line
2.23.0.3.g19edb94ec2
PATCH v6 2/3 Recommend git-filter-repo instead of git-filter-branch
In reply to Elijah Newrenfilter-branch suffers from a deluge of disguised dangers that disfigure history rewrites (i.e. deviate from the deliberate changes). Many of these problems are unobtrusive and can easily go undiscovered until the new repository is in use. This can result in problems ranging from an even messier history than what led folks to filter-branch in the first place, to data loss or corruption. These issues cannot be backward compatibly fixed, so add a warning to both filter-branch and its manpage recommending that another tool (such as filter-repo) be used instead.
Also, update other manpages that referenced filter-branch. Several of these needed updates even if we could continue recommending filter-branch, either due to implying that something was unique to filter-branch when it applied more generally to all history rewriting tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because something about filter-branch was used as an example despite other more commonly known examples now existing. Reword these sections to fix these issues and to avoid recommending filter-branch.
Finally, remove the section explaining BFG Repo Cleaner as an alternative to filter-branch. I feel somewhat bad about this, especially since I feel like I learned so much from BFG that I put to good use in filter-repo (which is much more than I can say for filter-branch), but keeping that section presented a few problems:
* In order to recommend that people quit using filter-branch, we need to provide them a recomendation for something else to use that can handle all the same types of rewrites. To my knowledge, filter-repo is the only such tool. So it needs to be mentioned.
* I don't want to give conflicting recommendations to users
* If we recommend two tools, we shouldn't expect users to learn both and pick which one to use; we should explain which problems one can solve that the other can't or when one is much faster than the other.
* BFG and filter-repo have similar performance
* All filtering types that BFG can do, filter-repo can also do. In fact, filter-repo comes with a reimplementation of BFG named bfg-ish which provides the same user-interface as BFG but with several bugfixes and new features that are hard to implement in BFG due to its technical underpinnings.
While I could still mention both tools, it seems like I would need to provide some kind of comparison and I would ultimately just say that filter-repo can do everything BFG can, so ultimately it seems that it is just better to remove that section altogether.
--- Documentation/git-fast-export.txt | 6 +- Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++--- Documentation/git-gc.txt | 17 +- Documentation/git-rebase.txt | 3 +- Documentation/git-replace.txt | 10 +- Documentation/git-svn.txt | 10 +- Documentation/githooks.txt | 10 +- contrib/svn-fe/svn-fe.txt | 4 +- git-filter-branch.sh | 14 ++ 9 files changed, 288 insertions(+), 59 deletions(-)
Documentation/git-fast-export.txt +3 -3
@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped into 'git fast-import'. You can use it as a human-readable bundle replacement (see -linkgit:git-bundle[1]), or as a kind of an interactive -'git filter-branch'. - +linkgit:git-bundle[1]), or as a format that can be edited before being +fed to 'git fast-import' in order to do history rewrites (an ability +relied on by tools like 'git filter-repo'). OPTIONS -------
Documentation/git-filter-branch.txt +243 -30
285 lines not on this page · raw message
Documentation/git-gc.txt +8 -9
@@ -115,15 +115,14 @@ NOTES ----- 'git gc' tries very hard not to delete objects that are referenced -anywhere in your repository. In -particular, it will keep not only objects referenced by your current set -of branches and tags, but also objects referenced by the index, -remote-tracking branches, refs saved by 'git filter-branch' in -refs/original/, reflogs (which may reference commits in branches -that were later amended or rewound), and anything else in the refs/* namespace. -If you are expecting some objects to be deleted and they aren't, check -all of those locations and decide whether it makes sense in your case to -remove those references. +anywhere in your repository. In particular, it will keep not only +objects referenced by your current set of branches and tags, but also +objects referenced by the index, remote-tracking branches, notes saved +by 'git notes' under refs/notes/, reflogs (which may reference commits +in branches that were later amended or rewound), and anything else in +the refs/* namespace. If you are expecting some objects to be deleted +and they aren't, check all of those locations and decide whether it +makes sense in your case to remove those references. On the other hand, when 'git gc' runs concurrently with another process, there is a risk of it deleting an object that the other process is using
Documentation/git-rebase.txt +2 -1
@@ -832,7 +832,8 @@ Hard case: The changes are not the same.:: This happens if the 'subsystem' rebase had conflicts, or used `--interactive` to omit, edit, squash, or fixup commits; or if the upstream used one of `commit --amend`, `reset`, or - `filter-branch`. + a full history rewriting command like + https://github.com/newren/git-filter-repo[`filter-repo`]. The easy case
Documentation/git-replace.txt +5 -5
@@ -123,10 +123,10 @@ The following format are available: CREATING REPLACEMENT OBJECTS ---------------------------- -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and -linkgit:git-rebase[1], among other git commands, can be used to create -replacement objects from existing objects. The `--edit` option can -also be used with 'git replace' to create a replacement object by +linkgit:git-hash-object[1], linkgit:git-rebase[1], and +https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to +create replacement objects from existing objects. The `--edit` option +can also be used with 'git replace' to create a replacement object by editing an existing object. If you want to replace many blobs, trees or commits that are part of a
@@ -148,13 +148,13 @@ pending objects. SEE ALSO -------- linkgit:git-hash-object[1] -linkgit:git-filter-branch[1] linkgit:git-rebase[1] linkgit:git-tag[1] linkgit:git-branch[1] linkgit:git-commit[1] linkgit:git-var[1] linkgit:git[1] +https://github.com/newren/git-filter-repo[git-filter-repo] GIT ---
Documentation/git-svn.txt +5 -5
@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons. + This option is NOT recommended as it makes it difficult to track down old references to SVN revision numbers in existing documentation, bug -reports and archives. If you plan to eventually migrate from SVN to Git -and are certain about dropping SVN history, consider -linkgit:git-filter-branch[1] instead. filter-branch also allows -reformatting of metadata for ease-of-reading and rewriting authorship -info for non-"svn.authorsFile" users. +reports, and archives. If you plan to eventually migrate from SVN to +Git and are certain about dropping SVN history, consider +https://github.com/newren/git-filter-repo[git-filter-repo] instead. +filter-repo also allows reformatting of metadata for ease-of-reading +and rewriting authorship info for non-"svn.authorsFile" users. svn.useSvmProps:: svn-remote.<name>.useSvmProps::
Documentation/githooks.txt +6 -4
@@ -425,10 +425,12 @@ post-rewrite This hook is invoked by commands that rewrite commits (linkgit:git-commit[1] when called with `--amend` and -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call -it!). Its first argument denotes the command it was invoked by: -currently one of `amend` or `rebase`. Further command-dependent -arguments may be passed in the future. +linkgit:git-rebase[1]; however, full-history (re)writing tools like +linkgit:git-fast-import[1] or +https://github.com/newren/git-filter-repo[git-filter-repo] typically +do not call it!). Its first argument denotes the command it was +invoked by: currently one of `amend` or `rebase`. Further +command-dependent arguments may be passed in the future. The hook receives a list of the rewritten commits on stdin, in the format
contrib/svn-fe/svn-fe.txt +2 -2
@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`. The resulting repository will generally require further processing to put each project in its own repository and to separate the history -of each branch. The 'git filter-branch --subdirectory-filter' command +of each branch. The 'git filter-repo --subdirectory-filter' command may be useful for this purpose. BUGS
@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected. SEE ALSO -------- -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1), +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1), https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
git-filter-branch.sh +14 -0
@@ -83,6 +83,20 @@ set_ident () { finish_ident COMMITTER } +if test -z "$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS" +then + cat <<EOF +WARNING: git-filter-branch has a glut of gotchas generating mangled history + rewrites. Hit Ctrl-C before proceeding to abort, then use an + alternative filtering tool such as 'git filter-repo' + (https://github.com/newren/git-filter-repo/) instead. See the + filter-branch manual page for more details; to squelch this warning, + set FILTER_BRANCH_SQUELCH_WARNING=1. +EOF + sleep 10 + printf "Proceeding with filter-branch...\n\n" +fi + USAGE="[--setup <command>] [--subdirectory-filter <directory>] [--env-filter <command>] [--tree-filter <command>] [--index-filter <command>] [--parent-filter <command>] [--msg-filter <command>]
Signature · 1 line
2.23.0.3.g19edb94ec2
PATCH v6 3/3 t9902: use a non-deprecated command for testing
In reply to Elijah Newrent9902 had a list of three random porcelain commands as a sanity check, one of which was filter-branch. Since we are recommending people not use filter-branch, let's update this test to use rebase instead of filter-branch.
--- t/t9902-completion.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
t/t9902-completion.sh +6 -6
@@ -28,10 +28,10 @@ complete () # # (2) A test makes sure that common subcommands are included in the # completion for "git <TAB>", and a plumbing is excluded. "add", -# "filter-branch" and "ls-files" are listed for this. +# "rebase" and "ls-files" are listed for this. -GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files' -GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch' +GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files' +GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase' . "$GIT_BUILD_DIR/contrib/completion/git-completion.bash"
@@ -1392,12 +1392,12 @@ test_expect_success 'basic' ' # built-in grep -q "^add \$" out && # script - grep -q "^filter-branch \$" out && + grep -q "^rebase \$" out && # plumbing ! grep -q "^ls-files \$" out && - run_completion "git f" && - ! grep -q -v "^f" out + run_completion "git r" && + ! grep -q -v "^r" out ' test_expect_success 'double dash "git" itself' '
Signature · 1 line
2.23.0.3.g19edb94ec2
Quoted 11 lines · Elijah Newren
* Remove git-filter-branch from git.git. Mention in the release
notes where people can go to get it.[1]filter-branch is not merely a slow or difficult-to-use tool, it's one that *fosters* mistakes by making it hard to get things right in several different ways. Granted, people exercise extra caution using filter-branch because they know they need to, but there are so many gotchas that they're likely to accidentally mess something up. Those mess-ups are not always discovered immediately, and by then it's nearly cast into stone (rewriting being something you want to do very rarely).
Is it possible to turn git-filter-branch into a fast, compatible, and (maybe) safe wrapper for git-filter-repo? That would "fix" filter-branch and (if done carefully) not break existing uses.
It could also spew warnings to recommend safer switches.
Stability is a major reason I use git, the Linux kernel, and why I distrust+avoid desktop/GUI software. Removing "unsafe" features, even with good intentions, inevitably leads to frustrated users.
Hi Eric!
Quoted 14 lines · Eric Wong
Quoted 11 lines · Elijah Newren
* Remove git-filter-branch from git.git. Mention in the release
notes where people can go to get it.[1]filter-branch is not merely a slow or difficult-to-use tool, it's one that *fosters* mistakes by making it hard to get things right in several different ways. Granted, people exercise extra caution using filter-branch because they know they need to, but there are so many gotchas that they're likely to accidentally mess something up. Those mess-ups are not always discovered immediately, and by then it's nearly cast into stone (rewriting being something you want to do very rarely).
Quoted 3 lines
Is it possible to turn git-filter-branch into a fast, compatible, and (maybe) safe wrapper for git-filter-repo? That would "fix" filter-branch and (if done carefully) not break existing uses.
Ooh, what an interesting question. I can probably ramble on a LOT longer than you expected about this...
== Short answer ==
It is certainly possible to reimplement git-filter-branch on top of git-filter-repo, though slightly differently than how you appear to be suggesting here. In doing so, you can provide the same user interface and have it be perfectly compatible. In fact, I've already created such a thing -- except that I took a few small liberties with compatibility (and documented each), primarily to improve the speed -- and I can use it instead of git-filter-branch to pass the git.git testsuite. You can see it here:
https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely
HOWEVER, it is NOT possible at all to make such a thing be fast or safe. Not even close. The performance and safety and are not accidents of the implementation, but are baked into the design from top to bottom and cannot be fixed without breaking backward compatibility in lots and lots of different ways. For the really curious, I'll provide a possibly non-comprehensive list of why you can't fix performance or safety if you require any compatibility at all (and maybe throw in one or two things that could be backward-compatibly fixed in filter-branch, such as the commit encoding disaster):
== Long answer ==
Performance:
* In editing files, git-filter-branch by design checks out each and
every commit as it existed in the original repo. If your repo has 10^5 files and 10^5 commits, but each commit only modifies 5 files, then git-filter-branch will make you do 10^10 modifications, despite only having (at most) 5*10^5 unique blobs.
* If you try and cheat and try to make filter-branch only work on
files modified in a commit, then two things happen (1) you run into problems with deletions whenever the user is simply trying to rename files (because attempting to delete files that don't exist looks like a no-op; it takes some chicanery and work to remap deletes across file renames when the renames happen via arbitrary user-provided shell), and (2) even if you succeed at the map-deletes-for-renames chicanery (as I believe I did in my reimplementation), you still technically violate backward compatibility because users are allowed to filter files in ways that depend upon topology of commits instead of filtering solely based on file contents or names (though I have never seen any user ever do this).
* Even if you don't need to edit files but only want to e.g. rename
or remove some and thus can avoid checking out each file (i.e. you can use --index-filter), you still are passing shell snippets for your filters. This means that for every commit, you have to have a prepared git repo where users can run git commands. That's a lot of setup. It also means you have to fork at least one process to run the user-provided shell snippet, and odds are that the user's shell snippet invokes lots of commands in some long pipeline, so you will have lots and lots of forks. For every. single. commit. That's a massive amount of overhead to rename a few files.
* filter-branch is written in shell, which is kind of slow.
Naturally, it makes sense to want to rewrite that in some other language. However, filter-branch documentation states that several additional shell functions are provided for users to call, e.g. 'map', 'skip_commit', 'git_commit_non_empty_tree', If filter-branch itself isn't a shell script, then in order to make those shell functions available to the users' shell snippets you have to prepend the shell definitions of these functions to every one of the users' shell snippets and thus make these special shell functions be parsed with each and every commit.
* filter-branch provides a --setup option which is a shell snippet
that can be sourced to make shell functions and variables available to all other filters. If filter-branch is a shell script, it can simply eval this shell snippet once at the beginning. If you try to fix performance by making filter-branch not be a shell script, then you have to prepend the setup shell snippet to all other filters and parse it with every single commit.
* git-filter-branch writes lots of files to $workdir/../map/ to keep
a mapping of commits, which it uses for the map() command it provides. Other files like $tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads, $tempdir/tree-state are all created internally too -- and users could have accessed any of these. Users even had a pointer to follow in the form of Documentation that the 'map' command existed, which naturally uses the $workdir/../map/* files. So, even if you don't have to edit files, for strict backward compatibility you need to still write a bunch of files to disk somewhere and keep them updated for every commit. You can claim it was an implementation detail that users should not have depended upon, but the truth is they've had a decade where they could so. So, if you want full compatibility, it has to be there. Besides, the regression tests depend on at least one of these details, specifying an --index-filter that reaches down and grabs backup-refs from $tempdir, and thus provides resourceful users who do google searches an example that there are files there for them to read and grab and use. (And if you want to pass the existing regression tests, you have to at least put the backup-refs file there even if it's irrelevant to your implementation otherwise.)
Safety:
filter-branch is riddled with gotchas resulting in various ways to easily corrupt repos or end up with a mess worse than what you started with:
* Someone can have a set of "working and tested filters" which they document or provide to a coworker, who then runs them on a different OS where the same commands are not working/tested (even the git-filter-branch manpage is guilty here). BSD vs. GNU userland differences can really bite. If you're lucky, you get ugly error messages spewed. But just as likely, the commands either don't do the filtering requested, or silently corrupt making some unwanted change. The unwanted change may only affect a few commits, so it's not necessarily obvious either. (The fact that problems won't necessarily be obvious means they are likely to go unnoticed until the rewritten history is in use for quite a while, at which point it's really hard to justify another flag-day for another rewrite.)
* filenames with spaces (which are rare) are often mishandled by
shell snippets since they cause problems for shell pipelines. Yes, I know find -print0, xargs -0, ls-files -z, etc. Not everyone does. And even if they do, they may assume it's not relevant because someone else renamed any such files in their repo back before the person doing the filtering joined the project (or maybe they are just being lazy and not thinking about everything that could go wrong).
* non-ascii filenames (which are rare) can be silently removed
despite being in a desired directory (the desire to select paths to keep often use pipelines like 'git ls-files | grep -v ^WANTED_DIR/ | xargs git rm". ls-files will only quote filenames if needed so folks may not notice that one of the files didn't match the regex, again until it's much too late. Yes, someone who knows about core.quotePath can avoid this (unless they have other special characters like \t, \n, or "), and people who use ls-files -z can avoid this, but that doesn't mean they will).
* Similarly, when moving files around, one can find that filenames
with non-ascii or special characters end up in a different directory, one that includes a double quote character. (This is technically the same issue as above with quoting, but perhaps an interesting different way that it can and has manifested as a problem.)
* It's far too easy to accidentally mix up old and new history.
It's still possible with any tool, but filter-branch almost invites it. If we're lucky, the only downside is users getting frustrated that they don't know how to shrink their repo and remove the old stuff. If we're unlucky, they merge old and new history and end up with multiple "copies" of each commit, some of which have unwanted or sensitive files and others which don't. This comes about in multiple different ways: the default to only doing a partial history rewrite ('--all' is not the default and over 80% of the examples in the manpage don't use it), the fact that there's no automatic post-run cleanup, the fact that --tag-name-filter (when used to rename tags) doesn't remove the old tags but just adds new ones with the new name (the manpage documents this so it's presumably not a "bug" even though it feels like it), and the fact that little educational information is provided to inform users of the ramifications of a rewrite and how to avoid mixing old and new history (e.g. not only do other users need to understand that they need to rebase their changes for all their branches on top of new history (or delete and reclone), but they also need to manually delete all their tags before refetching, any references that were on any shared servers (e.g. the central repo folks push to) that weren't part of the rewrite need to be deleted, and if the shared server has any locked-down refs such as refs/changes/, refs/pull/, or refs/merge-requests/ then people need to exercise extra caution; since none of these are fool-proof, someone should probably also add some server-side hooks to prevent folks from accidentally re-pushing old history, or make use of special facilities (such as gerrit's ban-commit command) to prevent it.)
* annotated tags can be accidentally converted to lightweight tags.
The first way this happens is folks do a rewrite, realize they messed up, restore from the backups in refs/original/, and then redo their filter-branch command. (The backup in refs/original/ is not a real backup; it dereferences tags first.) Another way this happens is despite passing --tags or --all on the command line, filter-branch dereferences the tag for them. The documentation does not make it all that clear that in order to retain annotated tags as annotated, you must use --tag-name-filter (and must not have restored from refs/original/ in a previously botched rewrite).
* Any commit messages that specify an encoding will become corrupted
by the rewrite; filter-branch ignores the encoding, takes the original bytes, and feeds it to commit-tree without telling it the proper encoding. (This happens whether or not --msg-filter is used, though I suspect --msg-filter provides additional ways to really mess things up).
* commit messages (even if they are all UTF-8) by default become
corrupted due to not being updated -- any references to other commit hashes in commit messages will now refer to no-longer-extant commits.
* no facilities for helping users find what unwanted crud they
should delete means they are much more likely to have incomplete or partial cleanups that sometimes result in confusion and people wasting time trying to understand (e.g. folks tend to just look for big files to delete instead of big directories or extensions, and once they do so, then sometime later folks using the new repository who are going through history will notice a build artifact directory that has some files but not others, or a cache of dependencies (node_modules or similar) which couldn't have ever been functional since it's missing some files)
* if --prune-empty isn't specified, then the filtering process can
create hoards of confusing empty commits
* if --prune-empty is specified, then intentionally placed empty
commits from before the filtering operation are also pruned instead of just pruning commits that became empty due to filtering rules.
* if --prune empty is specified, sometimes empty commits are missed
and left around anyway (probably just a bug, but...).
Also, performance and safety combine:
* Coming up with the correct shell snippet to do the filtering you
want is sometimes difficult unless you're just doing a trivial modification such as deleting a couple files. People have often come to me for help, so I should be practiced and an expert, but even for fairly simple cases I still sometimes taken over 10 minutes and several iterations to get the right commands -- and that's assuming they are working on a tiny repository. Unfortunately, people often learn if the snippet is right or wrong by trying it out, but the rightness or wrongness can vary depending on special circumstances (spaces in filenames, non-ascii filenames, funny author names or emails, invalid timezones, presence of grafts or replace objects, etc.), meaning they may have to wait a long time, hit an error, then restart. The performance of filter-branch is so bad that this cycle is painful, reducing the time available to carefully re-check (to say nothing about what it does to the patience of the person doing the rewrite even if they do technically have more time available). This problem is extra compounded because errors from broken filters may not be shown for a long time and/or get lost in a sea of output. Even worse, broken filters often just result in silent incorrect rewrites.
* To top it all off, even when users finally find working commands,
they naturally want to share them. But they may be unaware that their repo didn't have some special cases that someone else's does. So, when someone else with a different repository runs the same commands, they get hit by the problems above. Or, the user just runs commands that really were vetted for special cases, but they run it on a different OS where it doesn't work, as noted above.
== End of long answer ==
Summary of above: Anything compatible with git-filter-branch will be slower than molasses and extraordinarily unsafe.
Quoted 1 line
It could also spew warnings to recommend safer switches.
Ooh, I can take a crack at that right now: "For safety, don't use --tree-filter, --index-filter, --commit-filter, --tag-name-filter, --prune-empty, or (obviously) --force and _always_ use '--all' (and nothing else) for <rev-list options>. For performance, don't use --tree-filter, --index-filter, --commit-filter, --tag-name-filter, --setup, --env-filter, or --msg-filter. Also, don't depend refs/original/ stuff since that's unsafe (tag dereferencing), making --original useless. If you follow all these suggestions, -d is useless too. Oh, and I forgot to include --parent-filter among the bad-for-performance cases because you should have been using git-replace(1) instead of it for some time now, So that leaves us with our subset that could theoretically be made safe and performant:
--subdirectory-filter."
Turns out, filter-repo does support this exact flag, so if you're willing to restrict yourself to this subset, then filter-repo IS a drop-in replacement. :-)
And if you want not just recommendations of flags to avoid, bug flags you can use, then, again, see filter-repo for flags you can use.
Quoted 4 lines
Stability is a major reason I use git, the Linux kernel, and why I distrust+avoid desktop/GUI software. Removing "unsafe" features, even with good intentions, inevitably leads to frustrated users.
I did not and would not suggest deleting git-filter-branch. I suggested removing it from git.git and putting it elsewhere AND telling people where that elsewhere is. That elsewhere might be git-filter-repo, it could be a different repo, or it could even be my alternative faster (but still way too slow) re-implementation of filter-branch.
Hope that helps,
Elijah
On Fri, Aug 23, 2019 at 11:06 AM Elijah Newren <[email protected]> wrote:
<snip>
Quoted 1 line
Safety:
<snip>
Ooh, and another one I remembered just after hitting 'send':
* If the user provides a --tag-name-filter that maps multiple tags
to the same name, no warning or error is provided; filter-branch simply overwrites each tag in some undocumented pre-defined order (lexicographic) resulting in only one tag at the end. A regression test will fail if you attempt to error out and warn the user, so if you are trying to make a backward compatible reimplementation you have to add extra code to detect collisions and make sure that only the lexicographically last one is rewritten. (fast-import will naturally error out if told to write the same tag more than once, so you have to avoid triggering it.)
<snip>
Quoted 2 lines
Summary of above: Anything compatible with git-filter-branch will be slower than molasses and extraordinarily unsafe.
<snip.
Hi Elijah,
Quoted 15 lines · Elijah Newren
Quoted 14 lines · Eric Wong
Quoted 11 lines · Elijah Newren
* Remove git-filter-branch from git.git. Mention in the release
notes where people can go to get it.[1]filter-branch is not merely a slow or difficult-to-use tool, it's one that *fosters* mistakes by making it hard to get things right in several different ways. Granted, people exercise extra caution using filter-branch because they know they need to, but there are so many gotchas that they're likely to accidentally mess something up. Those mess-ups are not always discovered immediately, and by then it's nearly cast into stone (rewriting being something you want to do very rarely).
Quoted 8 lines
Quoted 3 lines
Is it possible to turn git-filter-branch into a fast, compatible, and (maybe) safe wrapper for git-filter-repo? That would "fix" filter-branch and (if done carefully) not break existing uses.
Ooh, what an interesting question. I can probably ramble on a LOT longer than you expected about this...
[...]
FWIW if anybody cares about my opinion: I would be totally fine with integrating git-filter-repo into git.git, have it there for a major version or two, then patch git filter-branch to spew out a deprecation warning, and then remove that latter command a major version (or two) later.
Ciao,
Dscho
Quoted 5 lines · Johannes Schindelin
FWIW if anybody cares about my opinion: I would be totally fine with integrating git-filter-repo into git.git, have it there for a major version or two, then patch
git filter-branchto spew out a deprecation warning, and then remove that latter command a major version (or two) later.
Yup, that's just the usual deprecate then delete sequence. The compatibility wrapper brought up in the discussion earlier would be a big plus ;-)
Quoted 18 lines · Elijah Newren
Quoted 17 lines · Junio C Hamano
Quoted 3 lines · Elijah Newren
Questions, comments, or concerns with this proposal? Alternative proposals? If inclusion is acceptable, are there any other tasks that need to be completed first?
I do not want a discussion to begin with a Devil's Advocate response, but anyway...
Are we planning to go to all batteries included approach? I have a feeling that there are other tools (hello, "git imerge") that equally deserve attention by Git users; are we in the business of absorbing them all? How big a project will our tree become, and how much more activity would have to be haneld by the readership of the Git mailing list?
Quoted 3 lines
Quoted 3 lines
I'd rather see us shed non-core tools we already have (e.g. git-svn, cvs import/export) out of git.git and have them as independent projects. But that may be just me.
Yes please! Let's make the repo smaller.
Quoted 6 lines
Ooh, if you're going to open this door, then a proposal I assumed would be shot down but which I'd be just about as happy with is:
* Remove git-filter-branch from git.git. Mention in the release
notes where people can go to get it.[1]
[snip]
Quoted 5 lines
[1] We'd still have to decide where to put it. If no one else wants to do it, I could include it in git-filter-repo with the promise that it's there for backward compatibility for those that still need the tool, even if I recommend folks use filter-repo instead.
May I recommend an idea, which may be silly?
We could strip these "extra" tools out of git.git and place them in their own repos. The hope would be that they could build on their own and have their own test suites.
Then, Git distributors could pick and choose the components they bundle with Git. Dscho would know more about this sort of thing as he distributes MinGit, which strips these things out already.
The biggest question is: how do we make sure that as git.git moves forward that we don't break the ecosystem? Maybe we create a new, larger repo that contains all of these subrepos? This would give the community more experience dogfooding our own repo-splitting tools.
Personally, I like the idea of 'git subtree' over something like 'git submodule'. Using 'git subtree' may mean that tools like 'git-svn' that may be hard to split into a completely independent repo could live primarily in the meta repo with a source dependence on the included git.git subtree.
This "meta-git.git" repo could then be more flexible in adding new tools like git-filter-repo or even git-lfs and friends. Again, distributors could select a subset to include, but we would have one place to run CI builds and make sure the tools are not obviously breaking as git.git updates.
Thanks,
-Stolee
Quoted 10 lines · Junio C Hamano
I do not want a discussion to begin with a Devil's Advocate response, but anyway...
Are we planning to go to all batteries included approach? I have a feeling that there are other tools (hello, "git imerge") that equally deserve attention by Git users; are we in the business of absorbing them all? How big a project will our tree become, and how much more activity would have to be haneld by the readership of the Git mailing list?
Quoted 3 lines
I'd rather see us shed non-core tools we already have (e.g. git-svn, cvs import/export) out of git.git and have them as independent projects. But that may be just me.
I like the general line of thinking here, but let me Devil's Advocate your Devil's Advocate:
- having separate repos and release schedules means that the dependency changes need to be coordinated. E.g., if a feature in git-filter-repo needs a new feature in git-core, then the feature needs to land in git-core first, then filter-repo needs to decide how to handle older versions. Whereas in the same repo, they can generally assume to move forward atomically.
- some of the non-core stuff helps test coverage for the core parts of the system. E.g., what bugs might we find in fast-import that are only triggered by the filter-repo test suite? Similarly, the scripted tools in git.git often serve as canaries for backwards-incompatible changes to the plumbing.
Something like the meta-git.git that Stolee proposed would help with that. But then we may end up dealing with other people's messes, which is one of the things we'd try to avoid with such a split.
-Peff