'git mailinfo' whitespace bug

PATCH New Linus Torvalds 4 participants 5 messages Jun 15, 2016 – Aug 13, 2016

lore

Share & tools

lore

Linus Torvalds

author

2016-06-15 22:48 UTC #0 lore raw

'git mailinfo' removes the whitespace from the beginning of the email body, but it does it incorrectly.

In particular, some people use indented paragraphs, like this:

Four-score and Four score and seven years ago our fathers
brought forth, upon this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.

Now we are engaged in a great civil war, testing whether that
nation, or any nation so conceived, and so dedicated, can long endure. We are met here on a great battlefield of that war. We have come to dedicate a portion of it as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this.

...

and mailinfo will not just remove empty lines from the beginning of the email body, it will also remove the _first_ indentation (but not any others). Which makes the whole thing come out wrong.

I bisected it, and this bug was introduced almost two years ago. In commit 3b6121f69b2 ("git-mailinfo: use strbuf's instead of fixed buffers"), to be exact. I'm pretty sure the bug is that handle_commit_msg() was changed to use 'strbuf_ltrim()' for the 'still_looking' case.

Before commit 3b6121f69b2, it would create a new variable that had the trimmed results ("char *cp = line;"), after that commit it would just trim the line itself. Which is correct for the case of it being a header, but if it's the first non-header line, it's wrong.

Linus

Lukas Sandström

2016-06-15 22:48 UTC #1 lore raw

PATCH 1/1 mailinfo: don't trim whitespace in the commit message

Previously any whitespace in the beginning of the first line of the commit message was trimmed, destroying any paragraph indentation. Move the whitespace trimming to check_header() instead, and preserve all commit message lines as-is in handle_commit_msg().

Signed-off-by Lukas Sandström <[email protected]>

---

Quoted 27 lines · Linus Torvalds

'git mailinfo' removes the whitespace from the beginning of the email body, but it does it incorrectly.
In particular, some people use indented paragraphs, like this:
Four-score and Four score and seven years ago our fathers
brought forth, upon this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
Now we are engaged in a great civil war, testing whether that
nation, or any nation so conceived, and so dedicated, can long endure. We are met here on a great battlefield of that war. We have come to dedicate a portion of it as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this.
   ...
and mailinfo will not just remove empty lines from the beginning of the email body, it will also remove the _first_ indentation (but not any others). Which makes the whole thing come out wrong.
I bisected it, and this bug was introduced almost two years ago. In commit 3b6121f69b2 ("git-mailinfo: use strbuf's instead of fixed buffers"), to be exact. I'm pretty sure the bug is that handle_commit_msg() was changed to use 'strbuf_ltrim()' for the 'still_looking' case.

Quoted 5 lines

Before commit 3b6121f69b2, it would create a new variable that had the trimmed results ("char *cp = line;"), after that commit it would just trim the line itself. Which is correct for the case of it being a header, but if it's the first non-header line, it's wrong.

This patch should fix it. Note that there was a test-case explicitly checking for this "trim first line" behaviour.

/Lukas

 builtin-mailinfo.c |   41 ++++++++++++++++++++++-------------------
 t/t5100/msg0015    |    2 +-
 2 files changed, 23 insertions(+), 20 deletions(-)

builtin-mailinfo.c +22 ~~-19~~

index a50ac22..954dc11 100644
@@ -283,11 +283,15 @@ static inline int cmp_header(const struct strbuf *line, const char *hdr)
 			line->buf[len] == ':' && isspace(line->buf[len + 1]);
 }
 
-static int check_header(const struct strbuf *line,
+static int check_header(const struct strbuf *line_in,
 				struct strbuf *hdr_data[], int overwrite)
 {
 	int i, ret = 0, len;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf sb = STRBUF_INIT, sb2 = STRBUF_INIT, *line = &sb2;
+
+	strbuf_addbuf(line, line_in);
+	strbuf_ltrim(line);
+
 	/* search for the interesting parts */
 	for (i = 0; header[i]; i++) {
 		int len = strlen(header[i]);

@@ -339,6 +343,7 @@ static int check_header(const struct strbuf *line,
 
 check_header_out:
 	strbuf_release(&sb);
+	strbuf_release(&sb2);
 	return ret;
 }

@@ -773,27 +778,25 @@ static int is_scissors_line(const struct strbuf *line)
 
 static int handle_commit_msg(struct strbuf *line)
 {
-	static int still_looking = 1;
+	static int first_msg_line_found = 0;
+	size_t cnt;
 
 	if (!cmitmsg)
-		return 0;
-
-	if (still_looking) {
-		strbuf_ltrim(line);
-		if (!line->len)
+		return 0; /* FIXME: shouldn't this be: return 1? */
+
+	if (!first_msg_line_found) {
+		if (use_inbody_headers)
+			if (check_header(line, s_hdr_data, 0))
+				return 0;
+		/* Check if the first line is all whitespace */
+		for (cnt = 0; isspace(line->buf[cnt]); cnt++)
+			; /* nothing */
+		if (line->len == cnt)
+			/* Ignore the first line if it's only whitespace */
 			return 0;
+		first_msg_line_found = 1;
 	}
 
-	if (use_inbody_headers && still_looking) {
-		still_looking = check_header(line, s_hdr_data, 0);
-		if (still_looking)
-			return 0;
-	} else
-		/* Only trim the first (blank) line of the commit message
-		 * when ignoring in-body headers.
-		 */
-		still_looking = 0;
-
 	/* normalize the log message to UTF-8. */
 	if (metainfo_charset)
 		convert_to_utf8(line, charset.buf);

@@ -804,7 +807,7 @@ static int handle_commit_msg(struct strbuf *line)
 			die_errno("Could not rewind output message file");
 		if (ftruncate(fileno(cmitmsg), 0))
 			die_errno("Could not truncate output message file at scissors");
-		still_looking = 1;
+		first_msg_line_found = 0;
 
 		/*
 		 * We may have already read "secondary headers"; purge

t/t5100/msg0015 +1 -1

index 9577238..4abb3d5 100644
@@ -1,2 +1,2 @@
-- a list
+  - a list
   - of stuff

Signature · 1 line

1.6.6.1

Junio C Hamano

2016-06-15 22:48 UTC #2 lore raw

In reply to Linus Torvalds

Quoted 9 lines · Linus Torvalds

I bisected it, and this bug was introduced almost two years ago. In commit 3b6121f69b2 ("git-mailinfo: use strbuf's instead of fixed buffers"), to be exact. I'm pretty sure the bug is that handle_commit_msg() was changed to use 'strbuf_ltrim()' for the 'still_looking' case.
Before commit 3b6121f69b2, it would create a new variable that had the trimmed results ("char *cp = line;"), after that commit it would just trim the line itself. Which is correct for the case of it being a header, but if it's the first non-header line, it's wrong.

True; trimming the body is obviously wrong.

But when is it correct to ltrim a header line? It means we are going to accept a header (or header-looking line in body) that is indented. I don't know why 87ab799 (builtin-mailinfo.c 2007-03-12) was coded that way.

 builtin-mailinfo.c |    3 +--
 t/t5100/msg0015    |    2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

builtin-mailinfo.c +1 -2

index a50ac22..ce2ef6b 100644
@@ -779,8 +779,7 @@ static int handle_commit_msg(struct strbuf *line)
 		return 0;
 
 	if (still_looking) {
-		strbuf_ltrim(line);
-		if (!line->len)
+		if (!line->len || (line->len == 1 && line->buf[0] == '\n'))
 			return 0;
 	}

t/t5100/msg0015 +1 -1

index 9577238..4abb3d5 100644
@@ -1,2 +1,2 @@
-- a list
+  - a list
   - of stuff

Don Zickus

2016-06-15 22:48 UTC #3 lore raw

Quoted 12 lines · Junio C Hamano

Quoted 9 lines · Linus Torvalds
I bisected it, and this bug was introduced almost two years ago. In commit 3b6121f69b2 ("git-mailinfo: use strbuf's instead of fixed buffers"), to be exact. I'm pretty sure the bug is that handle_commit_msg() was changed to use 'strbuf_ltrim()' for the 'still_looking' case.
Before commit 3b6121f69b2, it would create a new variable that had the trimmed results ("char *cp = line;"), after that commit it would just trim the line itself. Which is correct for the case of it being a header, but if it's the first non-header line, it's wrong.

Quoted 5 lines

True; trimming the body is obviously wrong.
But when is it correct to ltrim a header line? It means we are going to accept a header (or header-looking line in body) that is indented. I don't know why 87ab799 (builtin-mailinfo.c 2007-03-12) was coded that way.

In regards to 87ab799, I just deleted and pasted it from the function handle_inbody_header (which you can see from that commit). The original code for those lines came from ae448e3854d8b6e7e37aa88fa3917f5dd97f3210. Perhaps I misused it when I moved it.

Your patch belows seems to make sense for what its worth.

Cheers,
Don

Quoted 28 lines


 builtin-mailinfo.c |    3 +--
 t/t5100/msg0015    |    2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

builtin-mailinfo.c +1 -2

index a50ac22..ce2ef6b 100644
@@ -779,8 +779,7 @@ static int handle_commit_msg(struct strbuf *line)
 		return 0;
 
 	if (still_looking) {
-		strbuf_ltrim(line);
-		if (!line->len)
+		if (!line->len || (line->len == 1 && line->buf[0] == '\n'))
 			return 0;
 	}

t/t5100/msg0015 +1 -1

index 9577238..4abb3d5 100644
@@ -1,2 +1,2 @@
-- a list
+  - a list
   - of stuff

Junio C Hamano

2016-08-13 23:25 UTC #4 lore raw

Quoted 7 lines · Don Zickus

Quoted 3 lines
But when is it correct to ltrim a header line? It means we are going to accept a header (or header-looking line in body) that is indented. I don't know why 87ab799 (builtin-mailinfo.c 2007-03-12) was coded that way.
In regards to 87ab799, I just deleted and pasted it from the function handle_inbody_header (which you can see from that commit). The original code for those lines came from ae448e3854d8b6e7e37aa88fa3917f5dd97f3210.

Thanks for clarifying. The one in ae448e3 (mailinfo: ignore blanks after in-body headers., 2006-06-17) was about removing blank lines before the in-body headers begin, and never about removing indentation of the first in-body header. Admittedly, it was also being lenient and skipped over lines that are not empty but all whitespaces, and if we apply the quoted patch we will be retroactively tightening the rule, but I somehow do not think people would care.

Quoted 11 lines

 builtin-mailinfo.c |    3 +--
 t/t5100/msg0015    |    2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

builtin-mailinfo.c +0 -0

index a50ac22..ce2ef6b 100644
@@ -779,8 +779,7 @@ static int handle_commit_msg(struct strbuf *line)
 		return 0;

Quoted 6 lines

builtin-mailinfo.c static int handle_commit_msg(struct strbuf *line) · line 784
Quoted 6 lines
builtin-mailinfo.c +1 -2
 	if (still_looking) {
-		strbuf_ltrim(line);
-		if (!line->len)
+		if (!line->len || (line->len == 1 && line->buf[0] == '\n'))
 			return 0;
 	}

We probably could do something like

	if (still_looking) {
        	if (strspn(line->buf, " \t\n") == line->len)
			return 0;
	}

to keep people with blank but not empty lines before the first in-body header happy. I somehow don't think it is worth it, but what would I know...