playing with git, and netdev/libata-dev trees

Hopefully, this email can quick-start some people on git.

One of the things Linus's new 'git' tool allows me to do is make public the 50+ repositories that were previously only available on my local workstation. This should make it a lot easier for developers to see precisely what I have merged, and makes generating follow-up patches a whole lot easier.

When I merge a patch for drivers/net/forcedeth.c, I merge it into a brand new 'forcedeth' repository, a peer to the 40+ other such repository. Under BitKeeper, I made these repositories available merged together into one big "netdev-2.6" repository because it was too time consuming to make the individual 50+ trees publicly available. With git, developers have direct access to the individual trees.

I thought I would write up a quick guide describing how to mess around with the netdev and libata-dev trees, and with git in general.

1) installing git

git requires bootstrapping, since you must have git installed in order to check out git.git (git repo), and linux-2.6.git (kernel repo). I have put together a bootstrap tarball of today's git repository.

Download tarball from:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-20050526.tar.bz2

tarball build-deps: zlib, libcurl

install tarball: unpack && make && sudo make prefix=/usr/local install

jgarzik helper scripts, not in official git distribution:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-switch-tree
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-new-branch
http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-changes-script

After reading the rest of this document, come back and update your copy of git to the latest:
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/git.git

2) download a linux kernel tree for the very first time

mkdir -p linux-2.6/.git
cd linux-2.6
rsync -a --delete --verbose --stats --progress \ rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/ \ <- word-wrapped backslash; sigh
.git/

3) download latest changes to on-disk local tree

cd linux-2.6
git-pull-script \
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

4) check out files from the git repository into the working directory

cd linux-2.6
git-read-tree -m HEAD && git-checkout-cache -q -f -u -a

5) check in your own modifications (e.g. apply a patch)

# go to repo
cd linux-2.6

# make some modifications
patch -sp1 < /tmp/my.patch
diffstat -p1 < /tmp/my.patch

# NOTE: add '--add' and/or '--remove' if files were added or removed git-update-cache <list of all files changed>

# commit changes
GIT_AUTHOR_NAME="John Doe"		\
   GIT_AUTHOR_EMAIL="[email protected]"	\
   GIT_COMMITTER_NAME="Jeff Garzik"	\
   GIT_COMMITTER_EMAIL="[email protected]"	\
   git-commit-tree `git-write-tree`	\
   -p $(cat .git/HEAD )			\
   < changelog.txt			\

6) List all changes in working dir, in diff format.

git-diff-cache -p HEAD

7) List all changesets (i.e. show each cset's description text) in local tree that are not present in remote tree.

cd my-kernel-tree-2.6
git-changes-script -L ../linux-2.6 | less

8) List all changesets:

git-whatchanged

9) apply all patches in a Berkeley mbox-format file

First, download and add to your PATH Linus's git tools:
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/git-tools.git

cd my-kernel-tree-2.6
dotest /path/to/mbox # yes, Linus has no taste in naming useful scripts

10) don't forget to download tags from time to time.

git-pull-script only downloads sha1-indexed object data, and the requested remote head. This misses updates to the .git/refs/tags/ and .git/refs/heads directories. It is advisable to update your kernel .git directories periodically with a full rsync command, to make sure you got everything:

cd linux-2.6
rsync -a --delete --verbose --stats --progress \ rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/ \ <- word-wrapped backslash; sigh
.git/

11) [jg-specific] list all branches found in netdev-2.6 or libata-dev trees.

Download rsync://rsync.kernel.org/pub/scm/linux/kernel/git/jgarzik/netdev-2.6.git
or
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev.git

cd netdev-2.6
ls .git/refs/heads/

{ these are the current netdev-2.6 branches }

12) [jg-specific] make desired branch current in working directory

git-switch-tree $branch

13) [jg-specific] create a new branch, and make it current

git-new-branch $branch

14) [jg-specific] examine which branch is current

ls -l .git/HEAD

15) undo all local modifications (same as checkout):

git-read-tree -m HEAD && git-checkout-cache -q -f -u -a

Thanks for this, I'm starting to get up to speed on git now...

Two questions,

1. when you want to publish your tree what do you do? just rsync it onto kernel.org?
2. When you are taking things from your queue for Linus do you create another tree and merge your branches into it or what?

Dave.

Basically. I copy the attached script into each repo, customize the script for the upload destination.

When I publish the tree, I just cd to the toplevel dir on my local workstation, and run "./push"

Not quite sure what you're asking, but I'll attempt to answer anyway :)

When I prepare a submission for Linus, I will pull several branches into a single 'for-linus' conglomeration branch. I do this with the following incantation, for each branch I am pushing upstream:

git-resolve-script $(cat .git/HEAD) $(cat .git/refs/heads/$branch) \
pwd $branch

(Note that I modified my git-resolve-script to take an additional $4 argument, which causes the string " branch $4" to be added to the merge cset's commit msg)

After I take care of all the merge conflicts[1] from the git-resolve-script run, I give the big pile of mush a final build and test, and then run "./push".

For the "please pull" email, I use diffstat, git-diff-tree, git-changes-script and the newly written git-shortlog.

Jeff

[1] I'm still scared of conflicts in the merge process. Simple and automatic merging works just fine, like it did under BitKeeper. But if there are conflicts that cause git-pull-script/git-resolve-script to bail, then I bail as well: I export a patch, run patch(1), and then handle the merge the Old Fashioned Way(tm) by looking at .rej files.

I really wish somebody would write a merge helper for git that places the conflicts side-by-side in the code [in the working dir]. BitKeeper and CVS both presented conflicts to you in this manner.

The "I resolved this conflict, now let's continue where we left off" process is still quite raw in git. I suppose this is something that is left for others to script, above the plumbing, but hey...

#!/bin/sh

rsync -e ssh --verbose --delete --stats --progress -az .git/ master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/misc-2.6.git

Yes that's what I'm asking, mainly the pulling of multiple trees into one tree for giving to Linus, for Andrew I'm quite happy to have him pull multiple HEADs from the one tree assuming I don't have many interdependencies between trees...

Say I want something like this one tree with

drm-2.6 - HEAD <- linus tree
- drm-via < a via driver
- drm-initmap
- drm-savage <- a savage driver that depends on
the drm-initmap tree

How would I construct such a beast, how does it work out from where to branch, can I branch a branch for something like drm-savage so I can say send Linus the initmap branch and then have -mm pulling the savage one...

if that makes any sense :-)

Dave.

You used to use BitKeeper, right?

Well, git works just fine for creating and maintaining separate repos, where each repo is its own branch. Dependent branches are handled simply by pulling new changes from the parent (drm-initmap) repo into the child (drm-savage) repo.

If you just have a few branches, such as your example above, rather than the 50+ branches I have, then I would recommend _ignoring_ the advice in my git howto, and instead create a new branch by cloning:

	cd /repos
	mkdir drm-via
	cp -a linux-2.6/.git drm-via
	rm -rf drm-via/.git/objects
	cp -al linux-2.6/.git/objects drm-via/.git

Once created by the above recipes, or similar, the directory will function as a repo just like with BK. When you wish to update it from upstream, update your local linux-2.6 repo and then

	cd /repos/drm-via
	git-pull-script /repos/linux-2.6/.git

or to pull new changes from initmap into savage:

	cd /repos/drm-initmap
	# { check in some changes }
	cd /repos/drm-savage
	git-pull-script /repos/drm-initmap/.git

Does all that make sense? It's really quite similar to BK, if you ignore the "[jg-specific]" portions of the git howto I wrote.

Jeff

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think the quick-start is great. Definitely needed to get people up-to-speed with git.

A tarball of git will work, but it's a big bootstrap, and will need periodic updating.

I'm curious whether we couldn't make a git-bootstrap that contained a significantly stripped-down version that did nothing other than bootstrap git. A single program/file (perl script perhaps?) that...

mkdir git
cd git
rsync -rl rsync://rsync.kernel.org/pub/scm/git/git.git/ .git echo rsync://rsync.kernel.org/pub/scm/git/git.git > .git/branches/origin beginning with .git/HEAD, start extracting files

...and nothing more. Then, we could tell people to just download git-bootstrap and run it to create an up-to-date git repo.

Frank
- --
Frank Sorenson - KD7TZK
Systems Manager, Computer Science Department
Brigham Young University
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFClYaKaI0dwg4A47wRAp1MAKCcGK8vTWtw1gnTCjFbMFpbZkSO8QCff5RE NC8Z7RVHFP4qcbKRMSJ2rzg=
=fXsr
-----END PGP SIGNATURE-----

I don't really see the problem with an initial install from a tarball. If the tarball is a direct pull of the git.git tree .git/ and all then all anyone has to do to keep it up to date is either grab a copy of cogito, or set up their own git based scripts.

James Purser
http://k-sit.com