Skip to contentSkip to conversation

Truncating and cleaning a imported git repositary to make it more usable

PATCH New Alex Bennee 7 participants 14 messages Jun 15, 2016 Documentation

lore
Share & tools
lore
#0 lore raw

Hi,

I've succeeded in porting our whole CVS repository into git (with plenty of swap, cvsps is very memory hungry). Now we have a reference historical repository I need to think about creating one we actually use for day to day use (the .git is currently in the order of 650Mb which is a little heavy for pushing around DSL links willy nilly).

What I need to achieve is fairly simple.

1. Find the common ancestor point for all the product branches I want to keep in the repo.

Can anyone point me to the magic invocation that would allow this? git-merge-base can find the common point for two commits, do I just have to iteratively use that?

I'm currently working over a terminal link so it would be dead handy if there was a visual way of showing what gitk --all does on the command line.

2. Create a new repo, starting at the common ancestor point

2(a). I'd need to remove the orphaned old CVS modules which are still
around but not referenced any more.

3. Create the product branches and import ANCESTOR..BRANCH_HEAD into them

I'm not sure if I could pull directly into the new repo here. I could do it by doing a:

old_repo: git-format-patch ANCESTOR..BRANCH_HEAD
new_repo: git-am ../old_repo/0*

But I'm open to newer cleaner ways of dealing with this.

Has anyone done this before? Are there cleaner ways of dealing with this?

--
Alex, homepage: http://www.bennee.com/~alex/
When one knows women one pities men, but when one studies men, one excuses women. -- Horne Tooke

#1 lore raw
Quoted 5 lines · Alex Bennee

I've succeeded in porting our whole CVS repository into git (with plenty of swap, cvsps is very memory hungry). Now we have a reference historical repository I need to think about creating one we actually use for day to day use (the .git is currently in the order of 650Mb which is a little heavy for pushing around DSL links willy nilly).

You used git-repack and git-gc, right? If not, the repo will be bigger than it should be.

Quoted 10 lines

1. Find the common ancestor point for all the product branches I want to keep in the repo.

Can anyone point me to the magic invocation that would allow this? git-merge-base can find the common point for two commits, do I just have to iteratively use that?

I'm currently working over a terminal link so it would be dead handy if there was a visual way of showing what gitk --all does on the command line.

"git show-branch -a" probably does what you want.

Quoted 7 lines

2. Create a new repo, starting at the common ancestor point

2(a). I'd need to remove the orphaned old CVS modules which are still
around but not referenced any more.

3. Create the product branches and import ANCESTOR..BRANCH_HEAD into them

You probably want to look at git-filter-branch, particularly the --parent-filter option.

Note that 2(a) would be easiest if you just did it before importing from CVS at all, since it's easy to remove old modules from CVS just by moving the files out of the way.

Have fun,

Avery

#2 lore raw
Quoted 9 lines · Avery Pennarun
Quoted 5 lines · Alex Bennee

I've succeeded in porting our whole CVS repository into git (with plenty of swap, cvsps is very memory hungry). Now we have a reference historical repository I need to think about creating one we actually use for day to day use (the .git is currently in the order of 650Mb which is a little heavy for pushing around DSL links willy nilly).

You used git-repack and git-gc, right? If not, the repo will be bigger than it should be.

Oh yes. Unfortunately for 7 years of development of an embedded platform there is a lot of cruft in the CVS repo :-(.

However as it needs to be a drop in replacement for the current repo we can't just drop all the history and start from a fresh repo.

Quoted 4 lines
Quoted 2 lines

1. Find the common ancestor point for all the product branches I want to keep in the repo.

"git show-branch -a" probably does what you want.

Cool. Or in fact git-show-branch --merge-base seems to be the baby.

I'm sure it's been mentioned before but the plethora of git commands can make it hard to find the commands you want even
if your sure there is some way of doing it :-)

Quoted 8 lines
Quoted 7 lines

2. Create a new repo, starting at the common ancestor point

2(a). I'd need to remove the orphaned old CVS modules which are still
around but not referenced any more.

3. Create the product branches and import ANCESTOR..BRANCH_HEAD into them

Quoted 6 lines

You probably want to look at git-filter-branch, particularly the --parent-filter option.

Note that 2(a) would be easiest if you just did it before importing from CVS at all, since it's easy to remove old modules from CVS just by moving the files out of the way.

Trouble is looking at the current repo I'm not sure which modules are orphans (at the common anscestor point) and which have been added since.

Of course now I can easily go back in time (easier than in CVS) I can build up this list.

Quoted 1 line

Have fun,

For certain values of fun ;-)

Signature · 1 line
Alex, homepage: http://www.bennee.com/~alex/
#3 lore raw
In reply to Avery Pennarun
Quoted 9 lines · Avery Pennarun
Quoted 7 lines · Alex Bennee

2. Create a new repo, starting at the common ancestor point

2(a). I'd need to remove the orphaned old CVS modules which are still
around but not referenced any more.

3. Create the product branches and import ANCESTOR..BRANCH_HEAD into them

Quoted 6 lines

You probably want to look at git-filter-branch, particularly the --parent-filter option.

Note that 2(a) would be easiest if you just did it before importing from CVS at all, since it's easy to remove old modules from CVS just by moving the files out of the way.

Ahh, having investigated further it seems the repo contains two CVS modules which account for about 1/3 of the checkout and
are not needed (as they are standalone and separate). When I do the final import from CVS I can skip them. Having said that I'll need to run that over the weekend as cvsps takes up so much memory and swaps a lot.

I tried what I think is the correct usage of git-filter-branch but I get:

ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
'rm -rf big_dira big_dirb' dev-branch
warning: refname 'dev-branch' is ambiguous.
Which ref do you want to rewrite?

Any ideas?

Signature · 1 line
Alex, homepage: http://www.bennee.com/~alex/
#4 lore raw
Quoted 4 lines · Alex Bennee
 ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
 'rm -rf big_dira big_dirb' dev-branch
 warning: refname 'dev-branch' is ambiguous.
 Which ref do you want to rewrite?

Ignore me. Not sure why but:

git-checkout dev-branch
git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' HEAD

works

Signature · 1 line
Alex, homepage: http://www.bennee.com/~alex/
#5 lore raw
Quoted 4 lines · Alex Bennee
Quoted 4 lines
 ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
 'rm -rf big_dira big_dirb' dev-branch
 warning: refname 'dev-branch' is ambiguous.
 Which ref do you want to rewrite?

You have two refs called dev-branch. Probably both a tag and a branch (i.e., refs/heads/dev-branch and refs/tags/dev-branch). You can disambiguate by saying "heads/dev-branch").

Quoted 4 lines

Ignore me. Not sure why but:

git-checkout dev-branch
git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' HEAD

git-checkout prefers branches to tags[1], so there is no ambiguity. It puts the full refname (refs/heads/dev-branch) into your HEAD, so there is no ambiguity when saying "HEAD".

[1] Actually, the upcoming builtin-checkout no longer behaves this way, and will complain about the ambiguity.

-Peff

#6 lore raw
Quoted 2 lines · Jeff King

[1] Actually, the upcoming builtin-checkout no longer behaves this way, and will complain about the ambiguity.

To clarify: it is simply a warning; it will still switch to the branch.

-Peff

#7 lore raw
In reply to Alex Bennee
Quoted 11 lines · Alex Bennee
Quoted 4 lines · Alex Bennee
 ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
 'rm -rf big_dira big_dirb' dev-branch
 warning: refname 'dev-branch' is ambiguous.
 Which ref do you want to rewrite?

Ignore me. Not sure why but:

git-checkout dev-branch
git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' HEAD

You really shouldn't do it this way, unless you do it on a ramdisk. Better use an --index-filter. This is modeled after the last example in the man page (and, of course, untested):

git filter-branch --index-filter \
        'git ls-files -s |
	 grep -v "	big_dira" |
	 grep -v "	big_dirb" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

where the space before big_dir is actually a literal TAB!

-- Hannes

#8 lore raw

PATCH 1/1 Documentation: Another example for git-filter-branch

Quoted 7 lines · Johannes Sixt
Quoted 2 lines · Alex Bennee

git-checkout dev-branch
git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' HEAD

You really shouldn't do it this way, unless you do it on a ramdisk. Better use an --index-filter. This is modeled after the last example in the man page (and, of course, untested):

I missed the implications of the last example...

Quoted 9 lines
git filter-branch --index-filter \
        'git ls-files -s |
	 grep -v "	big_dira" |
	 grep -v "	big_dirb" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

where the space before big_dir is actually a literal TAB!

Good example. Maybe we should add the following (because plenty of examples is always a good thing IMHO)?

Quoted 1 line

From 341b480bad4ed9f99a54dc66ba20b0cead4594b5 Mon Sep 17 00:00:00 2001

From: Alex Bennee <[email protected]>
Date: Fri, 28 Mar 2008 18:00:20 +0000
Subject: [PATCH] Update git filter-branch examples for cleaning directories

I had to do this to a tree I imported from CVS which had a lot of cruft in it which was taking ages. Luckily Johannes Sixt submitted this scriplet on the list before I finished which makes things clearer.

---
 Documentation/git-filter-branch.txt |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)
Documentation/git-filter-branch.txt +21 -0
index 543a1cf..0e4b581 100644
@@ -184,6 +184,27 @@ git filter-branch --index-filter 'git update-index --remove filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+If you need to remove whole directories it is tempting to use "rm -rf"
+in the --tree-filter form, however this will generate a lot of disk IO
+on big trees. Hence it's more preferable to use the index filter. For
+example:
+
+--------------------------------------------------------------------------
+git filter-branch --index-filter \
+       'git ls-files -s |
+        grep -v "      big_dira" |
+	grep -v "      big_dirb" |
+		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
+                git update-index --index-info &&
+	mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
+--------------------------------------------------------------------------
+
+NB: The grep is matching literal TABs from the output of "git
+ls-files". Try <ctrl-v><tab> if your shell keeps giving you a directory
+listing every time you hit tab.
+
+
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:
Signature · 10 lines
1.5.5.rc0.6.gdeda.dirty





--
Alex, homepage: http://www.bennee.com/~alex/
We reject: kings, presidents, and voting. We believe in: rough consensus
and working code. -- Dave Clark
#9 lore raw
In reply to Johannes Sixt
Quoted 17 lines · Johannes Sixt
Quoted 11 lines · Alex Bennee
Quoted 4 lines · Alex Bennee
 ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
 'rm -rf big_dira big_dirb' dev-branch
 warning: refname 'dev-branch' is ambiguous.
 Which ref do you want to rewrite?

Ignore me. Not sure why but:

git-checkout dev-branch
git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' HEAD

You really shouldn't do it this way, unless you do it on a ramdisk. Better use an --index-filter. This is modeled after the last example in the man page (and, of course, untested):

Quoted 7 lines
git filter-branch --index-filter \
        'git ls-files -s |
	 grep -v "	big_dira" |
	 grep -v "	big_dirb" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

git filter-branch --index-filter 'git rm -r --cached big_dira big_dirb' should be working nowadays.

Mike

#10 lore raw
In reply to Johannes Sixt
Quoted 13 lines · Johannes Sixt

Alex Bennee schrieb:
You really shouldn't do it this way, unless you do it on a ramdisk. Better use an --index-filter. This is modeled after the last example in the man page (and, of course, untested):

git filter-branch --index-filter \
'git ls-files -s |
grep -v " big_dira" |
grep -v " big_dirb" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
git update-index --index-info &&
mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Quoted 1 line

where the space before big_dir is actually a literal TAB!

Hmm thats odd. Despite having successfully run the filter my repo is still the same size. git-fsck --full shows loads of dangling commits but I'm guessing because the repo is packed they don't get dropped. Is it possible to repack the whole repo or do you have to manually unpack the packs and re-pack?

Signature · 1 line
Alex, homepage: http://www.bennee.com/~alex/
#11 lore raw
Quoted 5 lines · Alex Bennee

Hmm thats odd. Despite having successfully run the filter my repo is still the same size. git-fsck --full shows loads of dangling commits but I'm guessing because the repo is packed they don't get dropped. Is it possible to repack the whole repo or do you have to manually unpack the packs and re-pack?

Yes. Run git repack -a -d -f or create a local clone using the file protocol:

   git clone file://$(pwd)/this that

(If you don't use the file protocol, git clone just hard-links the repository, and you again don't get the reduced size.)

-- Hannes

#12 lore raw
In reply to Alex Bennee
Quoted 15 lines · Alex Bennee
Quoted 14 lines · Johannes Sixt

Alex Bennee schrieb:

You really shouldn't do it this way, unless you do it on a ramdisk. Better use an --index-filter. This is modeled after the last example in the man page (and, of course, untested):

git filter-branch --index-filter \
'git ls-files -s |
grep -v " big_dira" |
grep -v " big_dirb" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
git update-index --index-info &&
mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Quoted 7 lines
Quoted 1 line

where the space before big_dir is actually a literal TAB!

Hmm thats odd. Despite having successfully run the filter my repo is still the same size. git-fsck --full shows loads of dangling commits but I'm guessing because the repo is packed they don't get dropped. Is it possible to repack the whole repo or do you have to manually unpack the packs and re-pack?

Did you pruned the repo ("git gc --prune" or "git prune")?

Signature · 3 lines
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
#13 lore raw
In reply to Alex Bennee
Quoted 2 lines · Alex Bennee

I've succeeded in porting our whole CVS repository into git (with plenty of swap, cvsps is very memory hungry). [...]

It sounds like you are doing a one-time conversion. If so, please consider using cvs2svn [1], which can also convert to git [2]. cvsps-based tools are known to have problems with most nontrivial CVS repositories (the contents of the output repository simply don't agree with those of the CVS original). IMO, the only reason to use a cvsps-based tool is if you really need incremental conversions *and* are willing to live with data loss.

Before you trust your new repository, please make some checks. As a minimum, make sure that the contents of each tag and branch tip revision is the same as the contents that you get if you check out of CVS. (I bet they don't!)

Michael

[1] http://cvs2svn.tigris.org
[2] http://cvs2svn.tigris.org/cvs2git.html