Extend mailmap functionality

v2: changes since v1
--------------------
* Folded in documentation fixup from patch 4 into patch 3.

This patch series extends the mailmap functionality to:
1) Allow the mailmap file in any location (also outside repo)
2) Enable mailmap to match on both Name and Email

So, why would this be a good thing?

2) Lets you replace both name and email of an author/committer, based on a name and/or email. So, should you have done commits with faulty address, or if an old email simply isn't valid anymore, you can add a mapping for that to replace it. So, the old style mapping is Proper Name <[email protected]>

while this patch series adds support for
Proper Name <[email protected]> <[email protected]> Proper Name <[email protected]> Commit Name <[email protected]>

1) Lets you keep a private mailmap file, which is not distributed with your repository.

This extended mapping is necessary when a company wants to have their repositories open to the public, but needs to protect the identities of the developers. It enables you to only show nicks and standardized emails, like 'Dev123 <[email protected]>' in the public repo, but by using an private mailmap file, map the name back to
'John Doe <[email protected]>' inside the company.

Patch serie applies cleanly on master branch, and test run shows no regressions.

Marius Storm-Olsen (4):
Add log.mailmap as configurational option for mailmap location Add find_insert_index, insert_at_index and clear_func functions to
string_list
Add map_user() and clear_mailmap() to mailmap
Change current mailmap usage to do matching on both name and email of
author/committer.

 Documentation/config.txt         |    9 ++
 Documentation/git-shortlog.txt   |   61 +++++++++---
 Documentation/pretty-formats.txt |    2 +
 builtin-blame.c                  |   52 ++++++----
 builtin-shortlog.c               |   25 ++++-
 cache.h                          |    1 +
 config.c                         |   10 ++
 mailmap.c                        |  207 ++++++++++++++++++++++++++++++++++----
 mailmap.h                        |    4 +
 pretty.c                         |   59 ++++++-----
 string-list.c                    |   43 +++++++-
 string-list.h                    |    9 ++
 t/t4203-mailmap.sh               |  152 ++++++++++++++++++++++++++++
 13 files changed, 541 insertions(+), 93 deletions(-)
 create mode 100755 t/t4203-mailmap.sh

I do not find the "necessary" argument very convincing nor I find the particular use case sane. You may want to do things that way, but I do not know if it is the best way to go about it.

The new mapping however brings in more flexibility, and there may be other use cases where people benefit from that flexibility. I am slightly in favor than neutral to this new feature.

Junio C Hamano said the following on 02.02.2009 04:03:

Well, this is the use-case for my company. We have open-source repos, which are the exact same repos which the company's developers work on, not a modified mirror repo.
However, laws in some countries requires that employees information be kept away from the public, should they so wish. So, for our developers to work on the exact same repo as the open-source community, we need a way to map both Names and Emails back to the original author, inside the company without using a different repo.

Also, the company commit policy might be such that the email on the commit is always the companys <[email protected]>. This might be so that if someone finds a bug to your commit, they send the bug report to the companys bug-tracker instead of the individual email address. However, that breaks the old mailmap system, since it requires unique email addresses to map to a name.

You could use this feature in git.git itself, where for example
Junio C Hamano <[email protected]>
Junio C Hamano <[email protected]> <[email protected]>
(or any of the others in the .mailmap file with X number of different email addresses), to ensure that you are always associated with one specific email address.

.marius [@trolltech.com]
'if you know what you're doing, it's not research'

It sure does, I use git-svn for a project hosted on Google Code, and the e-mail addresses are totally borked (e.g., "gmail-account-id" <gmail-account-id@the-svn-repo-uuid>). Not only that, but some developers (including me I must admit) have a rather obscure gmail-account-id, which is uses as the user's name for the commit message. It would be very nice if I could provide a mapping from "borked svn id's" to "normal, sensible e-mail addresses, including real names", a feature which this series seems to add :).

Cheers,

Sverre Rabbelier

This allows us to override a repo mailmap file, or to use mailmap files elsewhere than the repository root.

Should the log.mailmap file not be found, it falls back to ".mailmap".

---
 Documentation/config.txt       |    9 +++++++
 Documentation/git-shortlog.txt |    3 +-
 builtin-blame.c                |    2 +-
 builtin-shortlog.c             |    3 +-
 cache.h                        |    1 +
 config.c                       |   10 +++++++
 mailmap.c                      |    9 ++++++-
 t/t4203-mailmap.sh             |   53 ++++++++++++++++++++++++++++++++++++++++
 8 files changed, 86 insertions(+), 4 deletions(-)
 create mode 100755 t/t4203-mailmap.sh

index e2b8775..02c7b4c 100644
@@ -1012,6 +1012,15 @@ log.showroot::
 	Tools like linkgit:git-log[1] or linkgit:git-whatchanged[1], which
 	normally hide the root commit will now show it. True by default.
 
+log.mailmap::
+	Overrides the default location of the mailmap file. The default
+	mailmap location is .mailmap in the root of the repository.
+	The new location of the mailmap file may be in a repository
+	subdirectory, or somewhere outside of the repository itself.
+	If git can't find the file, it will try to load the default
+	mailmap location instead.
+	See linkgit:git-shortlog[1] and linkgit:git-blame[1].
+
 man.viewer::
 	Specify the programs that may be used to display help in the
 	'man' format. See linkgit:git-help[1].

index 8f7c0e2..cacbeea 100644
@@ -48,7 +48,8 @@ OPTIONS
 FILES
 -----
 
-If a file `.mailmap` exists at the toplevel of the repository,
+If a file `.mailmap` exists at the toplevel of the repository, or at the
+location pointed to by the log.mailmap configuration option,
 it is used to map an author email address to a canonical real name. This
 can be used to coalesce together commits by the same person where their
 name was spelled differently (whether with the same email address or

index aae14ef..4b63775 100644
@@ -2394,7 +2394,7 @@ parse_done:
 		die("reading graft file %s failed: %s",
 		    revs_file, strerror(errno));
 
-	read_mailmap(&mailmap, ".mailmap", NULL);
+	read_mailmap(&mailmap, NULL, NULL);
 
 	if (!incremental)
 		setup_pager();

index 5f9f3f0..04832a8 100644
@@ -219,7 +219,7 @@ void shortlog_init(struct shortlog *log)
 {
 	memset(log, 0, sizeof(*log));
 
-	read_mailmap(&log->mailmap, ".mailmap", &log->common_repo_prefix);
+	read_mailmap(&log->mailmap, NULL, &log->common_repo_prefix);
 
 	log->list.strdup_strings = 1;
 	log->wrap = DEFAULT_WRAPLEN;

@@ -248,6 +248,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix)
 	struct parse_opt_ctx_t ctx;
 
 	prefix = setup_git_directory_gently(&nongit);
+	git_config(git_default_config, NULL);
 	shortlog_init(&log);
 	init_revisions(&rev, prefix);
 	parse_options_start(&ctx, argc, argv, PARSE_OPT_KEEP_DASHDASH |

index 45e713e..3eef7ea 100644
@@ -867,6 +867,7 @@ extern int user_ident_explicitly_given;
 
 extern const char *git_commit_encoding;
 extern const char *git_log_output_encoding;
+extern const char *git_log_mailmap;
 
 /* IO helper functions */
 extern void maybe_flush_or_die(FILE *, const char *);

index 790405a..9ebcbbe 100644
@@ -565,6 +565,13 @@ static int git_default_branch_config(const char *var, const char *value)
 	return 0;
 }
 
+static int git_default_log_config(const char *var, const char *value)
+{
+	if (!strcmp(var, "log.mailmap"))
+		return git_config_string(&git_log_mailmap, var, value);
+	return 0;
+}
+
 int git_default_config(const char *var, const char *value, void *dummy)
 {
 	if (!prefixcmp(var, "core."))

@@ -579,6 +586,9 @@ int git_default_config(const char *var, const char *value, void *dummy)
 	if (!prefixcmp(var, "branch."))
 		return git_default_branch_config(var, value);
 
+	if (!prefixcmp(var, "log."))
+		return git_default_log_config(var, value);
+
 	if (!strcmp(var, "pager.color") || !strcmp(var, "color.pager")) {
 		pager_use_color = git_config_bool(var,value);
 		return 0;

index 88fc6f3..32eab6c 100644
@@ -2,11 +2,18 @@
 #include "string-list.h"
 #include "mailmap.h"
 
+const char *git_log_mailmap;
 int read_mailmap(struct string_list *map, const char *filename, char **repo_abbrev)
 {
 	char buffer[1024];
-	FILE *f = fopen(filename, "r");
+	FILE *f = NULL;
 
+	if (filename)
+		f = fopen(filename, "r");
+	if (f == NULL && git_log_mailmap)
+		f = fopen(git_log_mailmap, "r");
+	if (f == NULL)
+		f = fopen(".mailmap", "r");
 	if (f == NULL)
 		return 1;
 	while (fgets(buffer, sizeof(buffer), f) != NULL) {

1.6.1.2.257.g34f62

string_list_find_insert_index() and string_list_insert_at_index() enables you to see if an item is in the string_list, and to insert at the appropriate index in the list, if not there. This is usefull if you need to manipulate an existing item, if present, and insert a new item if not.

Future mailmap code will use this construct to enable complex (old_name, old_email) -> (new_name, new_email) lookups.

The string_list_clear_func() allows to call a custom cleanup function on each item in a string_list, which is useful is the util member points to a complex structure.

---
 string-list.c |   43 +++++++++++++++++++++++++++++++++++++++----
 string-list.h |    9 +++++++++
 2 files changed, 48 insertions(+), 4 deletions(-)

index ddd83c8..15e14cf 100644
@@ -26,10 +26,10 @@ static int get_entry_index(const struct string_list *list, const char *string,
 }
 
 /* returns -1-index if already exists */
-static int add_entry(struct string_list *list, const char *string)
+static int add_entry(int insert_at, struct string_list *list, const char *string)
 {
-	int exact_match;
-	int index = get_entry_index(list, string, &exact_match);
+	int exact_match = 0;
+	int index = insert_at != -1 ? insert_at : get_entry_index(list, string, &exact_match);
 
 	if (exact_match)
 		return -1 - index;

@@ -53,7 +53,13 @@ static int add_entry(struct string_list *list, const char *string)
 
 struct string_list_item *string_list_insert(const char *string, struct string_list *list)
 {
-	int index = add_entry(list, string);
+	return string_list_insert_at_index(-1, string, list);
+}
+
+struct string_list_item *string_list_insert_at_index(int insert_at,
+						     const char *string, struct string_list *list)
+{
+	int index = add_entry(insert_at, list, string);
 
 	if (index < 0)
 		index = -1 - index;

@@ -68,6 +74,16 @@ int string_list_has_string(const struct string_list *list, const char *string)
 	return exact_match;
 }
 
+int string_list_find_insert_index(const struct string_list *list, const char *string,
+				  int negative_existing_index)
+{
+	int exact_match;
+	int index = get_entry_index(list, string, &exact_match);
+	if (exact_match)
+		index = -1 - (negative_existing_index ? index : 0);
+	return index;
+}
+
 struct string_list_item *string_list_lookup(const char *string, struct string_list *list)
 {
 	int exact_match, i = get_entry_index(list, string, &exact_match);

@@ -94,6 +110,25 @@ void string_list_clear(struct string_list *list, int free_util)
 	list->nr = list->alloc = 0;
 }
 
+void string_list_clear_func(struct string_list *list, string_list_clear_func_t clearfunc)
+{
+	if (list->items) {
+		int i;
+		if (clearfunc) {
+			for (i = 0; i < list->nr; i++)
+				clearfunc(list->items[i].util, list->items[i].string);
+		}
+		if (list->strdup_strings) {
+			for (i = 0; i < list->nr; i++)
+				free(list->items[i].string);
+		}
+		free(list->items);
+	}
+	list->items = NULL;
+	list->nr = list->alloc = 0;
+}
+
+
 void print_string_list(const char *text, const struct string_list *p)
 {
 	int i;

index 4d6a705..d32ba05 100644
@@ -15,9 +15,18 @@ struct string_list
 void print_string_list(const char *text, const struct string_list *p);
 void string_list_clear(struct string_list *list, int free_util);
 
+/* Use this function to call a custom clear function on each util pointer */
+/* The string associated with the util pointer is passed as the second argument */
+typedef void (*string_list_clear_func_t)(void *p, const char *str);
+void string_list_clear_func(struct string_list *list, string_list_clear_func_t clearfunc);
+
 /* Use these functions only on sorted lists: */
 int string_list_has_string(const struct string_list *list, const char *string);
+int string_list_find_insert_index(const struct string_list *list, const char *string,
+				  int negative_existing_index);
 struct string_list_item *string_list_insert(const char *string, struct string_list *list);
+struct string_list_item *string_list_insert_at_index(int insert_at,
+						     const char *string, struct string_list *list);
 struct string_list_item *string_list_lookup(const char *string, struct string_list *list);
 
 /* Use these functions only on unsorted lists: */

1.6.1.2.257.g34f62

map_user() allows to lookup and replace both email and name of a user, based on a new style mailmap file. The possible mailmap definitions now are:

  proper_name <commit_email>                             // Old style
  proper_name <proper_email> <commit_email>              // New style
  proper_name <proper_email> commit_name <commit_email>  // New style

map_email() operates the same as before, with the exception that it also will to try to match on a name passed in through the name return buffer.

clear_mailmap() is needed to now clear the more complex mailmap structure.

---
 Documentation/git-shortlog.txt |   58 +++++++++---
 mailmap.c                      |  198 ++++++++++++++++++++++++++++++++++++----
 mailmap.h                      |    4 +
 3 files changed, 227 insertions(+), 33 deletions(-)

index cacbeea..d64dabf 100644
@@ -50,20 +50,29 @@ FILES
 
 If a file `.mailmap` exists at the toplevel of the repository, or at the
 location pointed to by the log.mailmap configuration option,
-it is used to map an author email address to a canonical real name. This
-can be used to coalesce together commits by the same person where their
-name was spelled differently (whether with the same email address or
-not).
-
-Each line in the file consists, in this order, of the canonical real name
-of an author, whitespace, and an email address (enclosed by '<' and '>')
-to map to the name. Use hash '#' for comments, either on their own line,
-or after the email address.
-
-A canonical name may appear in more than one line, associated with
-different email addresses, but it doesn't make sense for a given address
-to appear more than once (if that happens, a later line overrides the
-earlier ones).
+it is used to map author and committer names and email addresses to
+canonical real names and email addresses.
+This can be used to coalesce together commits by the same person where
+their name and/or email address was spelled differently.
+
+In the simple form, each line in the file consists of the canonical real name
+of an author, whitespace, and an email address used in the commit
+(enclosed by '<' and '>') to map to the name. Thus, looks like this
+--
+	Proper Name <[email protected]>
+--
+
+The more complex forms are
+--
+	Proper Name <[email protected]> <[email protected]>
+--
+which allows mailmap to replace both the name and the email of a commit
+matching the specified commit email address. And
+--
+	Proper Name <[email protected]> Commit Name <[email protected]>
+--
+which allows mailmap to replace both the name and the email of a commit
+matching the specified commit name and email address.
 
 So, for example, if your history contains commits by two authors, Jane
 and Joe, whose names appear in the repository under several forms:

@@ -86,6 +95,27 @@ Jane Doe <jane@desktop.(none)>
 Joe R. Developer <[email protected]>
 ------------
 
+Now, suppose your repository contains commits from the following authors:
+
+------------
+nick1 <[email protected]>
+nick2 <[email protected]>
+nick2 <[email protected]>
+santa <[email protected]>
+claus <[email protected]>
+------------
+
+Then, you might want a `.mailmap` file looking like:
+------------
+Some Dude <[email protected]>         nick1 <[email protected]>
+Other Author <[email protected]>   nick2 <[email protected]>
+Other Author <[email protected]>         <[email protected]>
+Santa Claus <[email protected]> <[email protected]>
+------------
+
+Use hash '#' for comments, either on their own line, or after the email address.
+
+
 Author
 ------
 Written by Jeff Garzik <[email protected]>

index 6e48f83..1b5ae69 100644
@@ -2,6 +2,10 @@
 #define MAILMAP_H
 
 int read_mailmap(struct string_list *map, const char *filename, char **repo_abbrev);
+void clear_mailmap(struct string_list *map);
+
 int map_email(struct string_list *mailmap, const char *email, char *name, int maxlen);
+int map_user(struct string_list *mailmap,
+	     char *email, int maxlen_email, char *name, int maxlen_name);
 
 #endif

1.6.1.2.257.g34f62

---
 Documentation/pretty-formats.txt |    2 +
 builtin-blame.c                  |   50 ++++++++++++-------
 builtin-shortlog.c               |   22 +++++++--
 pretty.c                         |   59 ++++++++++++-----------
 t/t4203-mailmap.sh               |   99 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 180 insertions(+), 52 deletions(-)

index 3d87d3e..28808b7 100644
@@ -103,6 +103,7 @@ The placeholders are:
 - '%an': author name
 - '%aN': author name (respecting .mailmap)
 - '%ae': author email
+- '%aE': author email (respecting .mailmap)
 - '%ad': author date (format respects --date= option)
 - '%aD': author date, RFC2822 style
 - '%ar': author date, relative

@@ -111,6 +112,7 @@ The placeholders are:
 - '%cn': committer name
 - '%cN': committer name (respecting .mailmap)
 - '%ce': committer email
+- '%cE': committer email (respecting .mailmap)
 - '%cd': committer date
 - '%cD': committer date, RFC2822 style
 - '%cr': committer date, relative

index 04832a8..19975ce 100644
@@ -40,6 +40,7 @@ static void insert_one_record(struct shortlog *log,
 	char *buffer, *p;
 	struct string_list_item *item;
 	char namebuf[1024];
+	char emailbuf[1024];
 	size_t len;
 	const char *eol;
 	const char *boemail, *eoemail;

@@ -51,7 +52,19 @@ static void insert_one_record(struct shortlog *log,
 	eoemail = strchr(boemail, '>');
 	if (!eoemail)
 		return;
-	if (!map_email(&log->mailmap, boemail+1, namebuf, sizeof(namebuf))) {
+
+	/* copy author name to namebuf, to support matching on both name and email */
+	memcpy(namebuf, author, boemail - author);
+	len = boemail - author;
+	while(len > 0 && isspace(namebuf[len-1]))
+		len--;
+	namebuf[len] = 0;
+
+	/* copy email name to emailbuf, to allow email replacement as well */
+	memcpy(emailbuf, boemail+1, eoemail - boemail);
+	emailbuf[eoemail - boemail - 1] = 0;
+
+	if (!map_user(&log->mailmap, emailbuf, sizeof(emailbuf), namebuf, sizeof(namebuf))) {
 		while (author < boemail && isspace(*author))
 			author++;
 		for (len = 0;

@@ -67,8 +80,8 @@ static void insert_one_record(struct shortlog *log,
 
 	if (log->email) {
 		size_t room = sizeof(namebuf) - len - 1;
-		int maillen = eoemail - boemail + 1;
-		snprintf(namebuf + len, room, " %.*s", maillen, boemail);
+		int maillen = strlen(emailbuf);
+		snprintf(namebuf + len, room, " <%.*s>", maillen, emailbuf);
 	}
 
 	item = string_list_insert(namebuf, &log->list);

@@ -321,6 +334,5 @@ void shortlog_output(struct shortlog *log)
 
 	log->list.strdup_strings = 1;
 	string_list_clear(&log->list, 1);
-	log->mailmap.strdup_strings = 1;
-	string_list_clear(&log->mailmap, 1);
+	clear_mailmap(&log->mailmap);
 }

1.6.1.2.257.g34f62

What's with these overlong lines? You do not have them in your other patches.

Junio C Hamano said the following on 02.02.2009 04:03:

Sorry, that was my editor which wrapped them visibly at column 70, and I forgot to insert manual line-breaks before saving. I'll correct it for the next version.

.marius [@trolltech.com]
'if you know what you're doing, it's not research'

I think the new feature is not so well conceived.

Should it always be a wholesale override, or should it also support augmenting the repository version with a private copy?

Junio C Hamano said the following on 02.02.2009 04:01:

Sure, I can rewrite it to be augmenting, rather than overriding.
I assume that the normal .mailmap file should be parsed first, then the log.mailmap one?

Sure, but I left it in to allow "old-style" usage. Just in case the were use-cases for not using the log.mailmap one. I can nuke the arguments if you don't want them anymore.

.marius [@trolltech.com]
'if you know what you're doing, it's not research'

Augmenting one would behave that way. I was more wondering if people would want to be able to choose either, perhaps from the command line option or something.

What I think would be sensible, if we really want to make this feature flexible, would be to introduce a command line option (and perhaps environment variable) that takes the usual precidence (the command line that specifies the mailmap file, then the environment and then finally the log.mailmap config variable). And if we go that route, that "usuall NULL" parameter would be where the callers pass the filename they got from their command line.

But if you feel it is overengineering, I would not disagree. In such a case, however, I do not think there is a reason for one particular caller to pass some custom value there, just to be inconsistent from others.

Junio C Hamano said the following on 02.02.2009 08:56:

Oh, I see. *ponder* maybe, though you could say that about any option in the .git/config file, right?

I'm not sure of the use case of the command line option. In which case would you want to only use the mailmap for that one command? It doesn't normally affect your git commands, so it doesn't hurt to just set the log.mailmap option. The environment use case would be just like setting it in your ~/.gitconfig, other than you can have a different one for each console, I guess.

Now, if we extended the mailmap feature even further, to report the mappings from the rev-list code ((optional of course)), so that any log viewer would show the mapped information; _then_ I would consider the command-line and environment variables as mandatory. Since, then you might want the feature off by default, and only use the mappings when you need to figure out a breakage, thus need a quick way to enable it.

So, unless anyone raises their hand that they need command-line/environment ways of setting the mailmap file used, I'll leave it as is for now. Ok?

.marius [@trolltech.com]
'if you know what you're doing, it's not research'

I was more thinking along the line of how .gitignore works, accumulating exclude patterns we find from .gitignore files along the way while descending into the directory hierarchy, and all the while honoring what the user added in .git/info/excludes.

Yeah, that is exactly why I suggested you to say that is overengineered and is not useful ;-)

Now you mention it, it might be wonderful if "git log --author" honored the mailmap file, but I do not think command line nor environment do not have much to do with it.

In any case, it is an independent issue from how mailmap maching and rewriting should work, which was the main point fo your series.

My point was that you should drop the second always-NULL parameter if the plan is not to have any command line filename. And I agree with your reasoning that a command line filename is not useful, so...