Skip to contentSkip to conversation

filter-branch performance

DISCUSSION Henning Moll 4 participants 7 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

Hi,

i am runningthis command

git filter-branch --env-filter 'export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL" GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME" GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"' --prune-empty --tag-name-filter cat -- --all

in a repository which i copied to /dev/shm before. According to "top", the git process only consumes about 5 percent of the CPU. The load is between 0.70 and 1.00.

I assume that there is a lot of process forking going on. Could that be the cause?

Any ideas how to further improve?

Best regards
Henning

#1 lore raw
Quoted 7 lines · Henning Moll

i am runningthis command

git filter-branch --env-filter 'export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL" GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME" GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"' --prune-empty --tag-name-filter cat -- --all

Quoted 6 lines

in a repository which i copied to /dev/shm before. According to "top", the git process only consumes about 5 percent of the CPU. The load is between 0.70 and 1.00.

I assume that there is a lot of process forking going on. Could that be the cause?

Yes. filter-branch is a shell scripts, and it is probably running multiple git commands per commit it is filtering.

Quoted 1 line

Any ideas how to further improve?

In your case you are not touching the tree contents at all. Last time I looked into this, I believe that filter-branch always loaded the index for each commit, even if no --index-filter is being used. So teaching filter-branch to optimize this case would be one strategy.

Another is to try using "git fast-export | git fast-import", and munging the data stream in between. That's may be more work, depending how fancy you want to get with accurate parsing (look into fast-export's --no-data, which omits blob data; that should make things faster and make hacky context-less parsing less likely to cause problems).

-Peff

#2 lore raw
Quoted 8 lines · Jeff King
Quoted 2 lines · Henning Moll

I assume that there is a lot of process forking going on. Could that be the cause?

Yes. filter-branch is a shell scripts, and it is probably running multiple git commands per commit it is filtering.

Quoted 1 line

Any ideas how to further improve?

Depending on how much time you can sink into improving the performance (versus just allowing the process to run to completion), you could also look into a non-forking solution, as well as not bothering to load the commit trees. To me non-forking means putting everything into the JVM by using JGit, like the BFG does, though libgit2 might also be an option.

Changing the BFG's code to do the transformation in your script is absolutely trivial - define a commit-node cleaner like this:

object SetCommitterToAuthor extends CommitNodeCleaner {
override def fixer(kit: CommitNodeCleaner.Kit) = c =>
c.copy(committer = c.author) // PersonIdent class holds name, email & time
}

...trivial if you don't mind compiling Scala with SBT that is, and I'm sure some people do! A DSL for non-Scala people to define their own BFG scripts would be good, I must get on that some day.

The BFG is generally faster than filter-branch for 3 reasons:

1. No forking - everything stays in the JVM process
2. Embarrassingly parallel algorithm makes good use of multi-core machines
3. Memoization means no Git object (file or folder) is cleaned more than once

In the case of your problem, only the first factor will be noticeably helpful. Unfortunately commits do need to be cleaned sequentially, as their hashes depend on the hashes of their parents, and filter-branch doesn't clean /commits/ more than once, the way it does with files or folders - so the last 2 reasons in the list won't be significant.

For your specific use case tho', the fact that BFG doesn't load the file tree at all unless it needs to clean it will also help.

I decided to knock up an egregious hack in the BFG to see what performance would be like. I ran it against a fairly large repo
(https://github.com/bfg-repo-cleaner-demos/intellij-community-original),
100k commits, stored in /dev/shm, and used the SetCommitterToAuthor code above. The BFG run completed in 31.7 seconds, you can see the resulting repo here:

https://github.com/rtyley/intellij-community-set-committer-to-author

I started running the same test some time ago using filter-branch, unfortunately that test has not completed yet - the BFG appears to be substantially faster.

Before:

$ git cat-file -p b02bf46c4e93c2e8570910cdd68eb6f4ce21ff81
tree 7a412e49ecdbd966d7efe5fe746ff3ea3b6067d1
parent 8794219e3e84aed3cc8af926ffd74beafa51fb6b
author peter <[email protected]> 1370854045 +0200
committer peter <[email protected]> 1370854098 +0200

After:

$ git cat-file -p 3adb7b2a5c87320a5a028b6a59a7132c75a6e91c
tree 7a412e49ecdbd966d7efe5fe746ff3ea3b6067d1
parent 5efcdb551789b0d0bb541de9325f09521c5fbcb6
author peter <[email protected]> 1370854045 +0200
committer peter <[email protected]> 1370854045 +0200 <- time fixed

The relevant code is in:
https://github.com/rtyley/bfg-repo-cleaner/compare/set-committer-to-author

#3 lore raw
Quoted 7 lines · Roberto Tyley

Depending on how much time you can sink into improving the performance (versus just allowing the process to run to completion), you could also look into a non-forking solution, as well as not bothering to load the commit trees. To me non-forking means putting everything into the JVM by using JGit, like the BFG does, though libgit2 might also be an option.

Quoted 8 lines

Changing the BFG's code to do the transformation in your script is absolutely trivial - define a commit-node cleaner like this:

object SetCommitterToAuthor extends CommitNodeCleaner {
override def fixer(kit: CommitNodeCleaner.Kit) = c =>
c.copy(committer = c.author) // PersonIdent class holds name, email & time
}

Thanks. I _almost_ mentioned BFG in the original email, but I didn't think it could do arbitrary fixes like this. Can you monkey-patch in arbitrary code, or do you have to rebuild all of BFG to include the snippet above?

Quoted 3 lines

...trivial if you don't mind compiling Scala with SBT that is, and I'm sure some people do! A DSL for non-Scala people to define their own BFG scripts would be good, I must get on that some day.

That would be cool. Even if the DSL was just Java, if you could do something like:

  vi fix.java
  javac fix.java
  bfg --filter=fix.class

that would be very useful (and I am probably showing my lack of Java chops by getting the compilation command or filenames wrong :) ).

Quoted 3 lines

I started running the same test some time ago using filter-branch, unfortunately that test has not completed yet - the BFG appears to be substantially faster.

No fair if you didn't run filter-branch on a PC and BFG on a Raspberry Pi. You have to give us a fighting chance. :)

-Peff

#4 lore raw
Quoted 11 lines · Jeff King
Quoted 5 lines · Roberto Tyley

object SetCommitterToAuthor extends CommitNodeCleaner {
override def fixer(kit: CommitNodeCleaner.Kit) = c =>
c.copy(committer = c.author) // PersonIdent class holds name, email & time
}

Thanks. I _almost_ mentioned BFG in the original email, but I didn't think it could do arbitrary fixes like this. Can you monkey-patch in arbitrary code, or do you have to rebuild all of BFG to include the snippet above?

Well, I publish a bfg-library jar to Maven Central, so you don't need to rebuild that:

http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22bfg-library_2.11%22

...in principle you can write a Java/Groovy/whatever project that calls that jar (your entry point would be com.madgag.git.bfg.cleaner.RepoRewriter) - tho' to be honest, I can't swear to how /friendly/ the API would be to call from non-Scala-land though, as I haven't tried it.

Incidentally, if people want to try compiling this monkey-patched BFG at home, this is how you'd do it:

* Install SBT - http://www.scala-sbt.org/download.html (or 'brew install sbt' for Mac OS X)
* git clone https://github.com/rtyley/bfg-repo-cleaner.git --branch set-committer-to-author
* cd bfg-repo-cleaner
* sbt "bfg/run --no-blob-protection"

There will be a lot of automated downloading of dependencies, and compilation will be slow the first time around, but at least there aren't that many steps. I do realise that being Scala/JVM based makes working on the BFG a bit of a specialist activity at the moment!

Quoted 6 lines
Quoted 2 lines

A DSL for non-Scala people to define their own BFG scripts would be good, I must get on that some day.

That would be cool. Even if the DSL was just Java, if you could do something like:

Quoted 6 lines
  vi fix.java
  javac fix.java
  bfg --filter=fix.class

that would be very useful (and I am probably showing my lack of Java chops by getting the compilation command or filenames wrong :) ).

Your syntax is right :) I'll give it some thought.

Quoted 6 lines
Quoted 3 lines

I started running the same test some time ago using filter-branch, unfortunately that test has not completed yet - the BFG appears to be substantially faster.

No fair if you didn't run filter-branch on a PC and BFG on a Raspberry Pi. You have to give us a fighting chance. :)

I guess I made that rod for my own back :) http://youtu.be/Ir4IHzPhJuI
for those who haven't seen it.

#5 lore raw
In reply to Roberto Tyley
Quoted 11 lines · Roberto Tyley

The BFG is generally faster than filter-branch for 3 reasons:

1. No forking - everything stays in the JVM process
2. Embarrassingly parallel algorithm makes good use of multi-core machines
3. Memoization means no Git object (file or folder) is cleaned more than once

In the case of your problem, only the first factor will be noticeably helpful. Unfortunately commits do need to be cleaned sequentially, as their hashes depend on the hashes of their parents, and filter-branch doesn't clean /commits/ more than once, the way it does with files or folders - so the last 2 reasons in the list won't be significant.

Just this part. If your history is bushy, you should be able to rewrite histories of merged branches in parallel up to the point they are merged---rewriting of the merge commit of course has to wait until all the branches have been rewritten, though.

#6 lore raw
Quoted 14 lines · Junio C Hamano
Quoted 11 lines · Roberto Tyley

The BFG is generally faster than filter-branch for 3 reasons:

1. No forking - everything stays in the JVM process
2. Embarrassingly parallel algorithm makes good use of multi-core machines
3. Memoization means no Git object (file or folder) is cleaned more than once

In the case of your problem, only the first factor will be noticeably helpful. Unfortunately commits do need to be cleaned sequentially, as their hashes depend on the hashes of their parents, and filter-branch doesn't clean /commits/ more than once, the way it does with files or folders - so the last 2 reasons in the list won't be significant.

Quoted 4 lines

Just this part. If your history is bushy, you should be able to rewrite histories of merged branches in parallel up to the point they are merged---rewriting of the merge commit of course has to wait until all the branches have been rewritten, though.

That's true, and the bfg does take advantage of that parallelism, so as well as point 1, point 2 will provide some benefit if history is bushy enough :)