Updated Kernel Hacker's guide to git

I refreshed my git intro/cookbook for kernel hackers, at http://linux.yyz.us/git-howto.html

This describes most of the commands I use in day-to-day kernel hacking.
Let me know if there are glaring errors or missing key commands.

Jeff

Hi Jeff !

Thanks for this update, it was my most useful source of inspiration when I started with git.

I very often use "git-format-patch -k -m" to produce individual patches that I delay, merge in other branches, or even in other trees with "git-am -k -3". I believe it was Davem who suggested this a while ago, and I agree it's very convenient to maintain a patch collection (and sometimes to clean them up).

Also, I think that for beginners, you have not insisted enough on the fact that they should not modify the master branch, but that they should immediately create their own branch before any local changes.

I got caught by this when I started, and had trouble playing with the origin branch to try to fix my mistakes.

Overall it's a good tutorial anyway.

Cheers,
Willy

Hi.

Thanks for the work! I'd suggest also saying how to repack and cleanup. Could also be a good idea to go through the steps for uploading to master.kernel.org or elsewhere?

Regards,

Nigel

Yes, I should mention repacking. When you say cleanup, what specifically do you mean?

Yes, push should be mentioned at the very least.

Jeff

Hi.

Oh, I was just thinking of the related commands - prune-packed, count-objects, fsck-objects and so on. (I know repack does prune-packed when you use -d, but it might be handy to mention it anyway... or not :>)

Nigel

Jeff Garzik <[email protected]> :

o 'git whatchanged shnortz' can probably be replaced with 'git log -- schnortz' so there is one command less to remember.

o "Display changes since last git-update-index:"
Fine but you have not told the reader what git-update-index is.

Ueimor

Very nice, thanks! A couple of remarks from an absolute git newbie:

1. I heard "git am" is supposed to supersede apply-mbox

2. What I often have problems with is - what to do if git spits at me a bunch of conflict messages after a seemingly safe pull or similar. Don't know if you want to cover those points but "git troubleshooting" would definitely be a valuable document.

Thanks
Guennadi
---
Guennadi Liakhovetski

Hey, that's pretty neat. Glad you told me, this should improve my workflow a bit.

Agreed.

Jeff

Very nice.

A bit on how to revert a commit and how to rebase a branch would make it even nicer :)

Thank you for a very good document, Jeff.

Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Looks nice, starting to look it over.

Notes:

Getting started:

There are RPM packages available (I think they are for latest Fedora; in case of doubt get the latest SRPM and build yourself, sometimes the distros lag /way/ behind). There are also Debian packages there, dunno about those.

Basic tasks:

'git pull' should be enough, no need to give the URL each time. It is useful to tell people how to get "nonofficial" branches (via URL + branches) too.

Miscellaneous debris:

'git pull' has gotten tags each time for me?

Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                    Fono: +56 32 2654431
Universidad Tecnica Federico Santa Maria             +56 32 2654239
Casilla 110-V, Valparaiso, Chile               Fax:  +56 32 2797513

Another year, another update! :)

The kernel hacker's guide to git has received some updates:

http://linux.yyz.us/git-howto.html

This includes all the input sent to me in the past several months, as well as a few new tips and tricks I use on a regular basis.

In general, this document is designed to be a quick-start cookbook, and not a comprehensive introduction.

Merry Christmas and Happy Holidays to all!

Jeff

there's one issue i have with this document, and that's that i wish it more carefully distinguished between regular git "user" tasks, and git "developer" tasks.

i may be mistaken, but it would seem that a lot of folks are going to be what i call basic users, who only want to update their git tree, check the logs, check the status and so on. and if they start to get ambitious, they might make some changes to the tree, do a diff, and submit a patch. but in the beginning, they won't be making commits or switching branches, etc.

in short, i can see the value of something like a "getting started with git as a basic user" tutorial. does such a thing exist?

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry Waterloo, Ontario, CANADA

http://crashcourse.ca
========================================================================

hmmm. There's the tutorial linked at the bottom of the page, which in turn links to http://www.kernel.org/pub/software/scm/git/docs/everyday.html

git is a developer's tool, so I sorta targetted that audience. I definitely agree that is not only git audience...

Jeff

just to be clear, i'm not complaining about the quality of the document above, but when i got started with git, what i really wanted was a list of what i (as a simple, non-developer user) could do once i cloned a repository.

to that end, i put together my own little reference list of git commands. for example, i collected ways to examine my repository -- git commands like branch, tag, log/shortlog, what-changed, show, grep, blame, that sort of thing. exactly the kind of stuff a new user might want to know about, even without the ability to change anything.

just my $0.02.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry Waterloo, Ontario, CANADA

http://crashcourse.ca
========================================================================

Could you perhaps publish your reference list as kind of a christmas gift to all basic users like me?

cu
Dieter

ps.: sorry for sending this twice, messed up recipients.

if you give me a day or two (or three), i may put an updated version of that up on my wiki.

rday

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry Waterloo, Ontario, CANADA

http://crashcourse.ca
========================================================================

Here are three out of four things which I do frequently with git repos:
I look at

- commits and blobs in other people's trees with gitweb,

- commits in a local tree with gitk,

- specific changes to source code with qgit, using it as "git blame" GUI.

(The fourth thing is feeding a driver subsystem git tree at kernel.org using a minimum number of git commands. Everything else which I do with git I do so infrequently that I have to reread manuals all the time.)

Stefan Richter
-=====-=-=== ==-- =-===
http://arcgraph.de/sr/

FYI, i'm typing in my own reference list as we speak here:

http://www.crashcourse.ca/wiki/index.php/Git

still quite a bit to go, but you can get the overall idea. new sections should be appearing there as the morning progresses.

rday

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry Waterloo, Ontario, CANADA

http://crashcourse.ca
========================================================================

Jeff, very good! I like it. Thank you! ;-)

Merry Christmas, kernel hackers!

Best wishes!

one minor note:

i would suggest using:

$ git shortlog master..HEAD

instead of

$ git log master..HEAD | git shortlog

to avoid unnecessary complexity :)

thanks,
- VMiklos

It says

"""Don't forget to download tags from time to time.

git pull only downloads sha1-indexed object data, and the requested remote head. This misses updates to the .git/refs/tags/ and .git/refs/heads/ directories. For tags, run git fetch --tags $URL."""

But when I do git pull on a simple tracking tree (e.g. git-clone torvalds/linux-2.6.git; git pull;) it automatically grabs new tags.

A while ago the default behavior of git pull was changed to fetch all tags which point to objects that can be reached from any of the tracked heads.

Old behaviour:  Option --tags was needed to fetch tags at all.  Current
behavior:  Option --tags forces to download all tags and the objects

they point to. Option --no-tags works like the old default behavior.

Readers of Kernel Hackers' Guide to git will most certainly have a recent enough version of git so that the "download_tags" subsection can be removed without replacement.

Stefan Richter
-=====-=-=== ==-- =====
http://arcgraph.de/sr/

All correct.

That "A while ago" is quite a while ago, though. IIRC it was added very early in 2006, which is eons ago in git timescale.

This does not work in all cases. When I retrieve the latest kernel, it downloads the tags:

	cd /spare/repo/linux-2.6
	git pull

but when I pull those changes into another local repo, the tags do -not- follow the objects:

	cd /spare/repo/misc-2.6
	git checkout master
	git pull ../linux-2.6
	git fetch --tags ../linux-2.6	# still required to this day

Regards,

Jeff

I guess this is because /spare/repo/misc-2.6 does not have branches of /spare/repo/linux-2.6 configured as remote (tracking) branches.

Stefan Richter
-=====-==--- -==- ====-
http://arcgraph.de/sr/

Unfortunately tags are not copied in all cases. To this day, I still have to 'git fetch --tags', generally when pulling from one local repo into another. It's annoying that tags don't follow objects, when pulled.

Jeff

Hi,

May I suggest adding some stuff about "git bisect", or at least links to other documentation about it, in this guide?

Especially, you may want to add an example about how to automate testing using "git bisect run" as you suggest other to do that:

http://www.ussg.iu.edu/hypermail/linux/kernel/0804.1/0633.html

Thanks in advance,
Christian.

I have some comments on the contents, though I need to warn, that I've been following git development for about year and a half now, and I am not a kernel hacker, so my comments may have wrong bias.

IMHO, syntax 'git diff master HEAD' is preferable, in order to avoid confusion with 'git log master..HEAD' usage, which has quite different meaning, roughly expressible as

git diff git merge-base master HEAD HEAD

for getting one big diff of the changes in HEAD since master)

This seems incorrect to me, as 'git diff master HEAD' compares two revisions, while 'git diff HEAD' compares working tree with HEAD revision.

Besides, expression 'HEAD branch' is misleading, because HEAD is not a branch by itself, but rather a link to the latest state of the current branch.

Thanks for doing this. I've referred to your previous page rather often as I grope around trying to learn git and hack a vendor driver for submittal into the mainline kernel.

One thing that baffled me was how to use git to create a "kitchen sink" diff that would produce my entire driver suitable for submittal to lkml for review. This probably isn't needed very often, but for new driver submittals it's important to know how to do it. Francois Romieu showed me how (assume the new driver branch is named "driver"):

$ git diff $(git merge-base master driver)..driver

As a beginner, this command continues to be utterly non-intuitive to me, but it works. There may be other ways to do it, too.

The point is, I think you should add instructions on your cookbook that address how to produce such a "kitchen sink" diff if you're submitting a brand new driver to lkml. (Obviously I don't know what such a diff is actually called.)

Jay

You inflict upon yourself all sorts of pain if you keep updating 'master', but don't merge that into 'driver'. Typically you want to rebase after updating master:

git checkout driver
git rebase master
# build and test
git prune

or merge master into your current branch:

git checkout driver
git pull . master
# build and test

That way, you are GUARANTEED that

git diff master..driver

will result in a diff that you can send upstream.

One moral of this story, as (I think) Linus mentioned, don't update 'master' too frequently. That's one key lesson of distributed programming. Unless the upstream kernel has a key API change or bug fix you need, just pretend the outside world does not exist, and hack away on your driver.

Jeff

There is a nicer way to do it with 1.4.x git -- note the 3 dots:

$ git diff master...driver

cheers,

martin

Careful.

I think Jay was looking at this kind of ancestry graph:

         *---*---*---* driver
        /

--o---o---x---x---x---x master

There might be quite a few merges on either side, but the point is '*' are not yet 'in', and 'o' and 'x' are already in the 'upstream (but 'x' are not in Jay's driver yet).

The three dots would give both '*' and 'x'; I do not think that is what Jay wants. A submitter to mainline usually wants only '*' commits.

I've always thought that 'submission' is supposed to be done as a series of patches, in which case a reasonable way would be to do:

git format-patch -n master driver

If on the other hand a single roll-up patch is desired, I think the most reasonable thing to do is to first merge the tip of the master to the tip of driver, resolve all the conflicts as needed, and take the diff between the 'master' and the result:

         *---*---*---*---y driver (y is the test merge)
        /               / 

--o---o---x---x---x---x master

git checkout driver
git merge master
... resolve conflicts if any, then "git commit" git diff master

This diff by definition should apply cleanly to the tip of 'master' and would result in the source that contains the updates for the driver.

When you are done, it would be advisable to do:

	git reset --hard HEAD^

to remove that 'y' merge, unless the merge involved a true conflict resolution.

Be careful. This ONLY works if you don't have criss-cross merges etc, so you need to be somewhat careful about it. If you end up having a complex merge history between the origin branch and your development tree, things will stop working

So it's actually worth understanding what the above does.

Here's the picture to keep in mind: you started off with something like this:

	<- older			newer ->

	..--A---B---C---D

where the top commit was D, and you started doing your own work off there. However, the tree you are tracking (doing "git fetch origin" or somethng) ALSO continues, so soon enough you'll have a history that looks like

	<- older			newer ->

	..--A---B---C---D---E---F---G---H <-"origin"
	                 \
	                  --X---Y---Z <- your "driver" branch"

where your work is the "X-Y-Z" sequence.

How, the reason you must NOT do

git diff origin..driver

or something like that, is that it will LITERALLY do a diff between those two heads. And while that is a perfectly normal diff, it's not what you
want: it's the diff between the up-stream work and yours, and it will show that a lot of work has _not_ happened in your branch (all the E-F-G-H stuff), and then you've added some work (X-Y-Z).

What you obviously WANT to have is the "diff" between "D" (which was the point where you started) and "Z" (which is where you are now. That's what you want to send up-stream.

Now, the way people used to do this under CVS, was to simply _remember_ D. Preferably by doing a tag at the point where they started working. Then you can always diff against that tag.

You can do that in git too, of course, but it would just be stupid. Because git actually _knows_ the history, so you can just ask git

"What is the most recent commit that both 'origin' and 'driver'
have in common?"

the answer, of course, is the very commit you want: D. And how do you ask for that "most recent common commit"? That's right: it's called the "merge base", because it's also the thing you'd use as the base in a bog-standard three-way merge.

So "git merge-base origin driver" just returns "D", without you having had to tag it or remember it at all.

So when you do

	git diff $(git merge-base origin driver)..driver

you're literally asking for the diff between your current top of the "driver" branch, and the place where you diverged from "origin".

Now, why do I mention the fact that THIS DOES NOT WORK if you do merges and you no longer have a linear tree? Think about it. If you actually ever pull on the up-stream and create a merge in your development tree, the commit history graph will now look something like this:

	<- older			newer ->

	..--A---B---C---D---E---F---G---H <-"origin"
	                 \        \
	                  --X---Y--M--Z <- your "driver" branch"

where the "M" commit is because you merged some new work from origin.

But see what that did to the most recent commit? Now, the most recent commit is no longer "D". In fact, it's not even your "merge" commit, if you think about it (even though you might naively think so), because that merge commit doesn't even exist in "origin", so clearly it cannot be the most recent common commit.

The most recent commit in common is "F". So now,

	git diff $(git merge-base origin driver)..driver

will give the difference between _F_ and your current tip Z.

The thing is, THIS IS STILL THE RIGHT THING TO DO. It's magic. Since you did a merge in M, all the things up until F have been merged into Z too, so you actually NO LONGER want to diff against D. Nor do you want to diff against M (since that would mean that you would not see the work you did in X and Y).

By doing a diff against F (and your own merge having done the right thing), you actually end up getting the "union" of work for X-Y-Z when you do that magic command line. If you had done it against D, you'd have seen the work in E and F that the merge M brought in - but you don't want that, because you want just the work YOU did, not anything in "origin".

So what you want is actually exactly again the most recent common commit.

HOWEVER. While it magically "just works" in things like this, it doesn't actually work for the really complex cases. If I've merged an eariler version of your work AND you did a merge from me too, you can have a criss-cross merge where there simply isn't one "most recent common commit" at all any more, but two or more. And then you really do need to do more.

This is one reason why I suggest developers not merge from me very often:
if your development tree is just a straight line (ie you only had that one merge), you can't ever even get in that situation, and perhaps as importantly, if/when I pull from you, the result will also tend to look more understandable to people who look at the history.

But another way to avoid a criss-cross merge is actually to never let me see your tree at all, and then you can merge with me as often as you damn well like. And in that case, the magic

	git diff $(git merge-base origin driver)..driver

is always going to give you exactly what you want. You can use merges to keep up-to-date, and always see what you want to send off by email upstream. The history will look like crap (you'll end up having a ton of merges in your tree), but if you aren't going to publicise your history anyway (just the diff end result), why would you care?

(And yes, you can actually do "git diff origin...driver" with _three_ dots. I'm not even going to explain _why_ that works, because that's a whole other kettle of fish, and is actually a magic special case in builtin-diff.c. So feel free to use it because it's short and sweet, but the long format is actually easier to explain what it actually MEANS).

Linus