How to deal with historic tar-balls

I have a number of older projects that I want to bring into a git repository. They predate a lot of the popular scm systems, so they are primarily a collection of tarballs today.

I'm fairly new to git so I have a couple questions related to this:

- What is the best approach for bringing them in? Do I just create a repository, then unpack the files, commit them, clean out the directory unpack the next tarball, and repeat until everything is loaded?

- Do I need to pay special attention to files that are renamed/removed from version to version?

- If the timestamps change on a file but the actual content does not, will git treat it as a non-change once it realizes the content hasn't changed?

- Last, if after loading the repository I find another version of the files that predates those I've loaded, or are intermediate between two commits I've already loaded, is there a way to go say that commit B is actually the ancestor of commit C? (i.e. a->c becomes a->b->c if you were to visualize the commit timeline or do diffs) Or do I just reload the tarballs in order to achieve this?

All replies appreciated!

There is a script which will import sources from multiple tarballs, creating a commit with the contents of each tarball. It's in the git repository under contrib/fast-import/import-tars.perl.

tom

Am 01.01.2012 01:27 schrieb Tomas Carnecky:

@tom: True. I didn't know about that script, but it should work.

@nn6eumtr: Basically your workflow is perfect. But let me give you some explanation:

git init
foreach archive in *.tar; do
tar xf $archive
git add --all .
git commit -m "Added $archive"
# now remove everything except for the .git directory # with regular shell commands (rm -rf *). Also remove # any dot-files (and the tarball itself, if it's in the # current directory).
done

Notice the '--all' switch to 'git add': Normally, 'git add .' adds all files that match the given pattern '.', i.e. all files in the current directory (and below, it's recursive). The '--all' switch together with the pattern '.' adds or updates all files already known to git *AND* adds the files not yet known *AND* removes the files that are no longer in the working tree. That's exactly what you want.

Consider archive1.tar with files A, B, C:

git add --all . # will add A, B, and C

Now remove A, B, C, and unpack archive2.tar. Assume it has files B, C, D. A was deleted, B was changed, C is unchanged, D is new.

git add --all . # will remove A, add B, leave C, add D.

git will notice that C hasn't changed its content (timestamp doesn't matter).

Without the '--all' switch, git would simply add B and D.

There is no problem re-arranging the history after your import (see "git rebase --help", especially the --interactive section), but then you probably will have conflicts and have to resolve them. I'd suggest to re-start the import instead.

Please note that "for archive in *.tar" will pick the tarballs in lexicographical order. That might not be your intention.

HTH,
Dirk

From: "Tomas Carnecky" <[email protected]> Sent: Sunday, January 01, 2012 12:27 AM

I'm doing a similar thing with a set of zip files. I grouped mine into batches for easier checking and putting on to separate branches. Planning your branch requirements is probably the biggest task, and will depend on how you hope to use the new repo.

Essentially yes; Obviously if you have an organisation in mind then you can introduce maintenance branches etc as you develop the import.

If it is simply to create a nice history that isn't really looked at, then a simple linear model is OK. If you need to keep old mintenance versions and obtain diffs with newer versions then look at the various branching models and populate appropriately.

No; It is only if you use those file dates to determine the implied date for the commit.

Correct. In fact git doesn't record the time stamps anyway. It simply records the content, and structure, of the snapshot.

You can use 'grafts' as a mechanism to re-arrange the commit order, and/or join partial repos, and then use git filter-branch to re-write the lot as a single cohesive repo. But this 'hack' does re-write all the commit SHA1 values, so you should minimise the number of times that happens...

It is worth capturing your import sequence as a script so that you can wash / rinse / repeat as often as needed to get a result you like.

I wasn't aware of those scripts. I'll be having a look at the zip import script for my needs.

My extra problem is that almost all my zips have an extra top level directory that changes its name for every zip (but some don't..). The TLD changes confuses the git rename detection if I don't remove them before committing. Fortunately it's an internal development project with no formal releases so creating the history is a bit of a personal project which doesn't affect ongoing development (which is the crunch question for fidelity of the repo you create).

Philip

From: "Philip Oakley" <[email protected]>: Sunday, January 01, 2012 6:30 PM

I had a look at the script but Python isn't part of the Msysgit install, so the example wouldn't run.

Also I couldn't see how the "fast_import.write(" method was being created - my ignorance of Python? Otherwise I could look at scripting it.

Philip

Am 02.01.2012 11:07 schrieb Philip Oakley:

Philip,

I'm not a Python guy, but I think fast_import.write() writes sth. to whatever the popen() call in line 24 returned:

  fast_import = popen('git fast-import --quiet', 'w')

I guess it returns a filehandle and 'git fast-import' reads its data from stdin. My guess is, that -- instead of writing to that pipe -- you could as well write everything to a temporary file and finally call

git fast-import < $tempfile

But that's only a guess.

Dirk

Many thanks - That explanation works for me. I just hadn't seen the association.
Philip

From: "Dirk Süsserott" <[email protected]>: Monday, January 02, 2012 6:26 PM

From: "Philip Oakley" <[email protected]> Sent: Sunday, January 01, 2012 6:30 PM

<snip>

Is there a mechanism for either having fast-import respect a .gitignore, or determining if a given file/path should be ignored?
My zips contain a lot of compile by-products that should be excluded from the repo.

The git-rm manpage contains instructions under the "vendor code drop"
section on how to do this. I imagine you will want to do each one
manually instead of queueing them up in a script because you are likely going to want to do appropriate clean up of the working tree in each iteration before committing. This is where you would review renames/removes with git-status before you git-add and git-commit. Also, if you are tracking permissions in git (the executable bit) then you will want to filter out any noise generated by frivolous permissions changes between the tarball contents.

In regard to inserting tarballs into the history that depends on when you think you plan on doing that. You are only going to be able to do that before the history is published (made "public" for other repos to pull down). Otherwise you will be rewriting published history which is a big no-no (see git-rebase manpage). I suggest you do your homework and order them properly before you start because that will be less work.
If you still find that you missed something then you can use
interactive git-rebase to insert. I'm assuming a single "master" branch with linear history is your desired end result. If you want to create maintenance branches showing release history then you will definitely need to do your homework first (see gitworkflow manpage).

If you venture into rebase territory by rewriting history (inserting missed tarballs in between older commits) you will need to be sure to review your automatic merge resolutions. Git only generates merge-conflicts on same-file-same-line conflicts. It will auto-merge same-file-different-line changes.

You also need to ask yourself if you really need a history of all those versions. To exaggerate, if all you really need is the current state then you need to ask yourself if it's worth the effort to record the previous states. Maybe what you want is something in-between (a happy medium).

In regard to the 'start-over' method of inserting missed tarballs you would just git-reset --hard to the commit you want to insert on-top-of, add the tarball, and then re-apply the subsequent tarballs. If you are doing cleanup between commits then the rebase or cherry-pick of the already cleaned-up subsequent commits from the "old-branch" (previous attempt) onto the 'do-over' branch will likely be easier. (You can just do 'git branch old-branch' on your branch before the git-reset --hard (do-over) and that will give you a "backup copy" of the "previous attempt" called "old-branch" that you can salvage already-done-work from by using rebase or cherry-pick.)

Hope this helps.

v/r,
neal

Thanks for the response, there is lots of good information there.

One clarification - can you track renames in git? I tried using git mv but from the status output it looks like it deleted the old file and added the new file. I was expecting it to record some sort of indicator of the name change, instead it looks like a short-cut for delete & add, the docs aren't clear if that is the case.

On 1/5/2012 10:25 AM, Neal Kreitzinger wrote:
...

...

(note: top-posting is not advised.)
You are exactly right in your observation: git-mv is only a shortcut for 'remove old then add new'. Git does not explicitly track "renames". It can detect renames easily in the cases where you really just renamed the file and left the contents the same. Git tracks content (and trees) as opposed to files (and file names). Git stores the 'blob', aka 'contents' of files in the object store. So if you have 30 files with different names and the exact same contents in your work tree they are stored as a single blob in the .git/objects "object store". If some of your "renames" are really "I renamed it and then I modified it" then git will have a harder time detecting the "rename" depending on how much you modified it. In such cases what you really did is arguably not a "rename" anyway. You can record your "renames" manually in your commit message if appropriate. If you have 5 minutes you can watch this video from the 15:00 min to 20:59 min marks to get an explanation of git-mv and rename-detection: http://www.youtube.com/watch?v=j45cs5_nY2k (youtube searchstring: 'git google tech talks', result: 'contributing with git'.)

v/r,
neal

Git only stores snapshots; so for an ordinary (non-merge, non-root) commit, you have the "before" (parent) and "after" (commit's) snapshot. Everything is generated on the fly from that, including diffs, heuristic rename detection, pickaxe, ...

To apply rename detection when diffing (e.g. in diff, log, show, format-patch), use the -M flag.

Thomas Rast
trast@{inf,student}.ethz.ch