Skip to contentSkip to conversation

git bug: rebase fatal failure

PATCH In review Tommy Thorn 8 participants 21 messages Jun 15, 2016 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi,

I have large-ish repository (143 MB) where git rebase failed unexpectedly (see below). If anyone is interested in looking at this, I have left a copy of the repository at http://thorn.ws/git-fatal-rebase-error.tar.gz

Tommy
PS: Please add me in CC as I'm no longer on this list.

$ git --version
git version 1.5.5.1.93.ge3f3e
$ git rebase cacao my-cacao-build
First, rewinding head to replay your work on top of it...
Applying Import binutils-2.18
.dotest/patch:16878: trailing whitespace.

.dotest/patch:17923: trailing whitespace.
0. Additional Definitions.
.dotest/patch:18024: trailing whitespace.
Version.
.dotest/patch:18094: trailing whitespace.
//
.dotest/patch:18099: trailing whitespace.
//
fatal: corrupt patch at line 260912
Patch failed at 0001.

When you have resolved this problem run "git rebase --continue". If you would prefer to skip this patch, instead run "git rebase --skip". To restore the original branch and stop rebasing run "git rebase --abort".

#1 lore raw

Hi,

Quoted 4 lines · Tommy Thorn

I have large-ish repository (143 MB) where git rebase failed unexpectedly (see below). If anyone is interested in looking at this, I have left a copy of the repository at
http://thorn.ws/git-fatal-rebase-error.tar.gz

I can reproduce. It seems that our diff machinery generates a file that adds a file with 10668 lines, but says 10669 lines in the hunk header.

It is a .info file, so I suspect strange things going on with non-printable ASCII characters.

I'm on it,
Dscho

#2 lore raw

Hi,

Quoted 2 lines · Johannes Schindelin

It is a .info file, so I suspect strange things going on with non-printable ASCII characters.

And indeed, this is the culprit:

-- snip --
BFD Index
*********

^@^H[index^@^H]
* Menu:

-- snap --

Note the NUL characters?

Now, the thing is: git format-patch still outputs it correctly. But git-rebase pipes the output to git-am, which in turn calls git-mailsplit, which suppresses that line.

Will keep you posted,
Dscho

#3 lore raw

PATCH 1/1 mailsplit and mailinfo: gracefully handle NUL characters

The function fgets() has a big problem with NUL characters: it reads them, but nobody will know if the NUL comes from the file stream, or was appended at the end of the line.

So implement a custom read_line() function.

Noticed by Tommy Thorn.

---

Sorry for the binary patch: the file t5100/nul contains NUL characters, obviously.

BTW I do not know how much fgetc() instead of fgets() slows down things, but I expect both to be equally fast because they are both buffered, right?

 builtin-mailinfo.c  |   24 +++++++++++++-----------
 builtin-mailsplit.c |   27 +++++++++++++++++++++++----
 builtin.h           |    1 +
 t/t5100-mailinfo.sh |    9 +++++++++
 t/t5100/nul         |  Bin 0 -> 91 bytes
 5 files changed, 46 insertions(+), 15 deletions(-)
 create mode 100644 t/t5100/nul
builtin-mailinfo.c +13 -11
index 11f154b..f0c4209 100644
@@ -641,7 +641,7 @@ static void decode_transfer_encoding(char *line, unsigned linesize)
 	}
 }
 
-static int handle_filter(char *line, unsigned linesize);
+static int handle_filter(char *line, unsigned linesize, int linelen);
 
 static int find_boundary(void)
 {
@@ -669,7 +669,7 @@ again:
 					"can't recover\n");
 			exit(1);
 		}
-		handle_filter(newline, sizeof(newline));
+		handle_filter(newline, sizeof(newline), strlen(newline));
 
 		/* skip to the next boundary */
 		if (!find_boundary())
@@ -759,14 +759,14 @@ static int handle_commit_msg(char *line, unsigned linesize)
 	return 0;
 }
 
-static int handle_patch(char *line)
+static int handle_patch(char *line, int len)
 {
-	fputs(line, patchfile);
+	fwrite(line, 1, len, patchfile);
 	patch_lines++;
 	return 0;
 }
 
-static int handle_filter(char *line, unsigned linesize)
+static int handle_filter(char *line, unsigned linesize, int linelen)
 {
 	static int filter = 0;
 
@@ -779,7 +779,7 @@ static int handle_filter(char *line, unsigned linesize)
 			break;
 		filter++;
 	case 1:
-		if (!handle_patch(line))
+		if (!handle_patch(line, linelen))
 			break;
 		filter++;
 	default:
@@ -794,6 +794,7 @@ static void handle_body(void)
 	int rc = 0;
 	static char newline[2000];
 	static char *np = newline;
+	int len = strlen(line);
 
 	/* Skip up to the first boundary */
 	if (content_top->boundary) {
@@ -807,7 +808,8 @@ static void handle_body(void)
 			/* flush any leftover */
 			if ((transfer_encoding == TE_BASE64)  &&
 			    (np != newline)) {
-				handle_filter(newline, sizeof(newline));
+				handle_filter(newline, sizeof(newline),
+						strlen(newline));
 			}
 			if (!handle_boundary())
 				return;
@@ -824,7 +826,7 @@ static void handle_body(void)
 
 			/* binary data most likely doesn't have newlines */
 			if (message_type != TYPE_TEXT) {
-				rc = handle_filter(line, sizeof(newline));
+				rc = handle_filter(line, sizeof(line), len);
 				break;
 			}
 
@@ -841,7 +843,7 @@ static void handle_body(void)
 					/* should be sitting on a new line */
 					*(++np) = 0;
 					op++;
-					rc = handle_filter(newline, sizeof(newline));
+					rc = handle_filter(newline, sizeof(newline), np - newline);
 					np = newline;
 				}
 			} while (*op != 0);
@@ -851,12 +853,12 @@ static void handle_body(void)
 			break;
 		}
 		default:
-			rc = handle_filter(line, sizeof(newline));
+			rc = handle_filter(line, sizeof(line), len);
 		}
 		if (rc)
 			/* nothing left to filter */
 			break;
-	} while (fgets(line, sizeof(line), fin));
+	} while ((len = read_line_with_nul(line, sizeof(line), fin)));
 
 	return;
 }
builtin-mailsplit.c +23 -4
index 46b27cd..021dc16 100644
@@ -45,6 +45,25 @@ static int is_from_line(const char *line, int len)
 /* Could be as small as 64, enough to hold a Unix "From " line. */
 static char buf[4096];
 
+/* We cannot use fgets() because our lines can contain NULs */
+int read_line_with_nul(char *buf, int size, FILE *in)
+{
+	int len = 0, c;
+
+	for (;;) {
+		c = fgetc(in);
+		buf[len++] = c;
+		if (c == EOF || c == '\n' || len + 1 >= size)
+			break;
+	}
+
+	if (c == EOF)
+		len--;
+	buf[len] = '\0';
+
+	return len;
+}
+
 /* Called with the first line (potentially partial)
  * already in buf[] -- normally that should begin with
  * the Unix "From " line.  Write it into the specified
@@ -70,19 +89,19 @@ static int split_one(FILE *mbox, const char *name, int allow_bare)
 	 * "From " and having something that looks like a date format.
 	 */
 	for (;;) {
-		int is_partial = (buf[len-1] != '\n');
+		int is_partial = len && buf[len-1] != '\n';
 
-		if (fputs(buf, output) == EOF)
+		if (fwrite(buf, 1, len, output) != len)
 			die("cannot write output");
 
-		if (fgets(buf, sizeof(buf), mbox) == NULL) {
+		len = read_line_with_nul(buf, sizeof(buf), mbox);
+		if (len == 0) {
 			if (feof(mbox)) {
 				status = 1;
 				break;
 			}
 			die("cannot read mbox");
 		}
-		len = strlen(buf);
 		if (!is_partial && !is_bare && is_from_line(buf, len))
 			break; /* done with one message */
 	}
builtin.h +1 -0
index c630d5b..d0a0ead 100644
@@ -9,6 +9,7 @@ extern const char git_usage_string[];
 extern void list_common_cmds_help(void);
 extern void help_unknown_cmd(const char *cmd);
 extern void prune_packed_objects(int);
+extern int read_line_with_nul(char *buf, int size, FILE *file);
 
 extern int cmd_add(int argc, const char **argv, const char *prefix);
 extern int cmd_annotate(int argc, const char **argv, const char *prefix);
t/t5100-mailinfo.sh +9 -0
index d6c55c1..5a4610b 100755
@@ -25,4 +25,13 @@ do
 		diff ../t5100/info$mail info$mail"
 done
 
+test_expect_success 'respect NULs' '
+
+	git mailsplit -d3 -o. ../t5100/nul &&
+	cmp ../t5100/nul 001 &&
+	(cat 001 | git mailinfo msg patch) &&
+	test 4 = $(wc -l < patch)
+
+'
+
 test_done
t/t5100/nul new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..3d40691787b855cc0133514a19052492eb853d21
GIT binary patch
Signature · 1 line
1.5.5.1.425.g5f464.dirty

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

two quibbles of no great importance ...

Johannes Schindelin suggested:

Quoted 5 lines

The function fgets() has a big problem with NUL characters: it reads them, but nobody will know if the NUL comes from the file stream, or was appended at the end of the line.

So implement a custom read_line() function.

                        ^^^^^^^^^^^
                        read_line_with_nul()

meaning read part or all of one line which may contain NULs.

Quoted 16 lines

[ ... ]

builtin-mailsplit.c +7 -0
index 46b27cd..021dc16 100644
@@ -45,6 +45,25 @@ static int is_from_line(const char *line, int len)
 /* Could be as small as 64, enough to hold a Unix "From " line. */
 static char buf[4096];
 
+/* We cannot use fgets() because our lines can contain NULs */
+int read_line_with_nul(char *buf, int size, FILE *in)
+{
+     int len = 0, c;
+
+     for (;;) {
+             c = fgetc(in);

builtin-mailsplit.c static int is_from_line(const char *line, int len) · line 64

Quoted 10 lines
builtin-mailsplit.c +10 -0
+             buf[len++] = c;
+             if (c == EOF || c == '\n' || len + 1 >= size)
+                     break;
+     }
+
+     if (c == EOF)
+             len--;
+     buf[len] = '\0';
+
+     return len;

when fgetc(3) — why not use getc(3)? — returns EOF it is pointlessly stored in buf[] (as a 'char'!), len's advanced, and then the storage and advancing
are undone. isn't that a bit silly? untested:

	assert(2 <= size);
	do {
		if ((c = getc(in)) == EOF)
			break;
	} while (((buf[len++] = c) != '\n' && len+1 < size);
	buf[len] = '\0'

	return len;

I'd tend to write this in terms of pointers,
something along the lines (untested):

	char	*p, *endp;

	assert(1 <= size);
	p    = buf;
	endp = p + (size-1);
	while (p < endp) {
		if ((c = getc(in)) == EOF || (*p++ = c) == '\n')
			break;
	}
	*p = '\0';

	return p - buf;
Quoted 2 lines
builtin-mailsplit.c +2 -0
+
+}
Signature · 4 lines
"How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools." |      http://www.stopesso.com
#6 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Quoted 15 lines · Brian Foster

I'd tend to write this in terms of pointers,
something along the lines (untested):

	char	*p, *endp;

	assert(1 <= size);
	p    = buf;
	endp = p + (size-1);
	while (p < endp) {
		if ((c = getc(in)) == EOF || (*p++ = c) == '\n')
			break;
	}
	*p = '\0';

	return p - buf;

Leave optimization to the compiler. Using pointer arithmetic more often than not screws up loop optimization and strength reduction.

Signature · 1 line
David Kastrup
#7 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

In reply to Brian Foster

Hi,

Quoted 9 lines · Brian Foster

Johannes Schindelin suggested:

Quoted 5 lines

The function fgets() has a big problem with NUL characters: it reads them, but nobody will know if the NUL comes from the file stream, or was appended at the end of the line.

So implement a custom read_line() function.

                        ^^^^^^^^^^^
                        read_line_with_nul()

meaning read part or all of one line which may contain NULs.

Right.

Quoted 20 lines
Quoted 20 lines

[ ... ]

builtin-mailsplit.c +11 -0
index 46b27cd..021dc16 100644
@@ -45,6 +45,25 @@ static int is_from_line(const char *line, int len)
 /* Could be as small as 64, enough to hold a Unix "From " line. */
 static char buf[4096];
 
+/* We cannot use fgets() because our lines can contain NULs */
+int read_line_with_nul(char *buf, int size, FILE *in)
+{
+     int len = 0, c;
+
+     for (;;) {
+             c = fgetc(in);
+             buf[len++] = c;
+             if (c == EOF || c == '\n' || len + 1 >= size)
+                     break;
+     }
Quoted 8 lines

builtin-mailsplit.c static int is_from_line(const char *line, int len) · line 64

Quoted 6 lines
builtin-mailsplit.c +6 -0
+
+     if (c == EOF)
+             len--;
+     buf[len] = '\0';
+
+     return len;

when fgetc(3) — why not use getc(3)? -

Because mailsplit can read from a file, too.

Quoted 3 lines

returns EOF it is pointlessly stored in buf[] (as a 'char'!), len's advanced, and then the storage and advancing are undone. isn't that a bit silly?

I left it at that, because it is a rare case, the buffer has to be accessed with the trailing NUL anyway, and I think it is worth to have this function quite readable. I, for one, am pretty certain that I understand what this function does, and how, in 6 months from now, without any additional documentation.

Quoted 10 lines

untested:

	assert(2 <= size);
	do {
		if ((c = getc(in)) == EOF)
			break;
	} while (((buf[len++] = c) != '\n' && len+1 < size);
	buf[len] = '\0'

	return len;

... except this is unreadable at best ;-)

Quoted 15 lines

I'd tend to write this in terms of pointers,
something along the lines (untested):

	char	*p, *endp;

	assert(1 <= size);
	p    = buf;
	endp = p + (size-1);
	while (p < endp) {
		if ((c = getc(in)) == EOF || (*p++ = c) == '\n')
			break;
	}
	*p = '\0';

	return p - buf;

Again, I think this is too cuddled. You have to think about every second line, and that makes for stupid mistakes with this developer.

Ciao,
Dscho

#8 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Brian Foster

when fgetc(3) — why not use getc(3)? -

Because mailsplit can read from a file, too.

Huh? getc != getchar

Signature · 1 line
David Kastrup
#9 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

In reply to Johannes Schindelin
Quoted 3 lines · Johannes Schindelin

BTW I do not know how much fgetc() instead of fgets() slows down things, but I expect both to be equally fast because they are both buffered, right?

In my experience, fgetc() is pretty fantastically slow because you have a function call for every byte (and, I gather, modern libc does thread locking for every fgetc). It's usually much faster to fread() into a buffer and then access the buffer. I don't know if that's appropriate (or matters) here, though.

Have fun,

Avery

#10 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Quoted 10 lines · Avery Pennarun
Quoted 3 lines · Johannes Schindelin

BTW I do not know how much fgetc() instead of fgets() slows down things, but I expect both to be equally fast because they are both buffered, right?

In my experience, fgetc() is pretty fantastically slow because you have a function call for every byte (and, I gather, modern libc does thread locking for every fgetc). It's usually much faster to fread() into a buffer and then access the buffer. I don't know if that's appropriate (or matters) here, though.

Is getc an option?

Signature · 1 line
David Kastrup
#11 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

In reply to Avery Pennarun

Hi,

Quoted 9 lines · Avery Pennarun
Quoted 3 lines · Johannes Schindelin

BTW I do not know how much fgetc() instead of fgets() slows down things, but I expect both to be equally fast because they are both buffered, right?

In my experience, fgetc() is pretty fantastically slow because you have a function call for every byte (and, I gather, modern libc does thread locking for every fgetc). It's usually much faster to fread() into a buffer and then access the buffer.

Hmpf. I hoped to get more definitive information here. Especially given that fgetc() is nothing more than a glorified fread() into a buffer, and then access the buffer.

Well, at least you kind of pointed me to the _unlocked() function family.

Ciao,
Dscho

#12 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Quoted 5 lines · Johannes Schindelin

Hmpf. I hoped to get more definitive information here. Especially given
that fgetc() is nothing more than a glorified fread() into a buffer, and then access the buffer.

Well, at least you kind of pointed me to the _unlocked() function family.

Point taken.

/tmp $ for d in test1 test2 test3 test3u; do echo -n "$d: "; /usr/bin/time ./$d </dev/zero; done
test1: 0.09user 0.05system 0:00.14elapsed 94%CPU
test2: 2.50user 0.05system 0:02.54elapsed 100%CPU
test3: 2.48user 0.06system 0:02.53elapsed 100%CPU
test3u: 1.05user 0.05system 0:01.10elapsed 99%CPU

fread is about 18x faster than fgetc(). getc() is the same speed as fgetc(). getc_unlocked() is definitely faster than getc, but still at least 7x slower than fread().

And if you think *that* sucks, you should try "c << cin" in C++ :)

Source code below.

Have fun,

Avery

=== test1.c ===
#include <stdio.h>

int main()

{
    char buf[1024];
    int i;
    for (i = 0; i < 102400; i++)
        fread(buf, 1, sizeof(buf), stdin);
}

=== test2.c ===
#include <stdio.h>

int main()

{
    int i;
    for (i = 0; i < 1024*102400; i++)
        fgetc(stdin);
}

=== test3.c ===
#include <stdio.h>

int main()

{
    int i;
    for (i = 0; i < 1024*102400; i++)
        getc(stdin);
}

=== test3u.c ===
#include <stdio.h>

int main()

{
    int i;
    for (i = 0; i < 1024*102400; i++)
        getc_unlocked(stdin);
}
#13 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Hi,

Quoted 16 lines · Avery Pennarun
Quoted 5 lines · Johannes Schindelin

Hmpf. I hoped to get more definitive information here. Especially given
that fgetc() is nothing more than a glorified fread() into a buffer, and then access the buffer.

Well, at least you kind of pointed me to the _unlocked() function family.

Point taken.

/tmp $ for d in test1 test2 test3 test3u; do echo -n "$d: "; /usr/bin/time ./$d </dev/zero; done
test1: 0.09user 0.05system 0:00.14elapsed 94%CPU
test2: 2.50user 0.05system 0:02.54elapsed 100%CPU
test3: 2.48user 0.06system 0:02.53elapsed 100%CPU
test3u: 1.05user 0.05system 0:01.10elapsed 99%CPU

Quoted 3 lines

fread is about 18x faster than fgetc(). getc() is the same speed as fgetc(). getc_unlocked() is definitely faster than getc, but still at least 7x slower than fread().

Well, my question was more about fgetc() vs fgets().

If you feel like it, you might benchmark this patch:

-- snipsnap --

builtin-mailsplit.c +20 -1
index 021dc16..5d8defd 100644
@@ -45,13 +45,32 @@ static int is_from_line(const char *line, int len)
 /* Could be as small as 64, enough to hold a Unix "From " line. */
 static char buf[4096];
 
+/*
+ *  This is an ugly hack to avoid fgetc(), which is slow, as it is locking.
+ *  The argument "in" must be the same for all calls to this function!
+ */
+static int fast_fgetc(FILE *in)
+{
+	static char buf[4096];
+	static int offset = 0, len = 0;
+
+	if (offset >= len) {
+		len = fread(buf, 1, sizeof(buf), in);
+		offset = 0;
+		if (!len && feof(in))
+			return EOF;
+	}
+
+	return buf[offset++];
+}
+
 /* We cannot use fgets() because our lines can contain NULs */
 int read_line_with_nul(char *buf, int size, FILE *in)
 {
 	int len = 0, c;
 
 	for (;;) {
-		c = fgetc(in);
+		c = fast_fgetc(in);
 		buf[len++] = c;
 		if (c == EOF || c == '\n' || len + 1 >= size)
 			break;
#14 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Quoted 7 lines · Johannes Schindelin
Quoted 3 lines · Avery Pennarun

fread is about 18x faster than fgetc(). getc() is the same speed as fgetc(). getc_unlocked() is definitely faster than getc, but still at least 7x slower than fread().

Well, my question was more about fgetc() vs fgets(). If you feel like it, you might benchmark this patch:

Wouldn't it be better to improve the implementation of getc() in glibc instead?

getc() is meant to be the fast version of fgetc(), and if it isn't (anymore), then the library needs fixing.

Signature · 2 lines
Sincerely,                                                          [email protected]
           Stephen R. van den Berg.
#15 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Hi,

Quoted 11 lines · Stephen R. van den Berg
Quoted 7 lines · Johannes Schindelin
Quoted 3 lines · Avery Pennarun

fread is about 18x faster than fgetc(). getc() is the same speed as fgetc(). getc_unlocked() is definitely faster than getc, but still at least 7x slower than fread().

Well, my question was more about fgetc() vs fgets(). If you feel like it, you might benchmark this patch:

Wouldn't it be better to improve the implementation of getc() in glibc instead?

Sure, because glibc is used on Windows, AIX, Solaris, etc.

Quoted 2 lines

getc() is meant to be the fast version of fgetc(), and if it isn't (anymore), then the library needs fixing.

fgetc() has to work reliably in a threaded environment, too. So I guess that it does all kinds of locks that slow it down, but it is not something to be "fixed".

Hth,
Dscho

#16 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

In reply to Johannes Schindelin

builtin-mailsplit.c static int is_from_line(const char *line, int len) · line 53

Quoted 7 lines · Johannes Schindelin
builtin-mailsplit.c +6 -0
+/*
+ *  This is an ugly hack to avoid fgetc(), which is slow, as it is locking.
+ *  The argument "in" must be the same for all calls to this function!
+ */
+static int fast_fgetc(FILE *in)
+{

Looks great to me, but shouldn't you add an "inline" for this one? Also, maybe a double the buffer size.

Tommy

#17 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Hi,

Quoted 11 lines · Tommy Thorn

builtin-mailsplit.c static int is_from_line(const char *line, int len) · line 53

Quoted 7 lines · Johannes Schindelin
builtin-mailsplit.c +6 -0
+/*
+ *  This is an ugly hack to avoid fgetc(), which is slow, as it is locking.
+ *  The argument "in" must be the same for all calls to this function!
+ */
+static int fast_fgetc(FILE *in)
+{

Looks great to me, but shouldn't you add an "inline" for this one? Also, maybe a double the buffer size.

No. This is an ugly hack, and not meant for application.

If that is substantially faster than the fgetc() version (and I want this be tested in a _real-world_ scenario, i.e. not the fgetc() alone, but a real mailsplit and a real mailinfo on a huge patch, with all three
versions: fgets(), fgetc() and fast_fgetc())), then I would prefer having something like

	struct line_reader {
		FILE *in;
		char buffer[4096];
		int offset, int len;
		char line[1024];
		int linelen;
	};

and corresponding functions to read lines in that setting. Maybe it would even be better to have line be a strbuf, but I am not so sure on that.

Let's see what the tests show. Would you do them, please? "git format-patch --stdout bla..blub | /usr/bin/time git mailsplit -o." three times in succession should give you a good hint on the runtime.

Ciao,
Dscho

#18 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

In reply to Johannes Schindelin
Quoted 5 lines · Johannes Schindelin

The function fgets() has a big problem with NUL characters: it reads them, but nobody will know if the NUL comes from the file stream, or was appended at the end of the line.

So implement a custom read_line() function.

Looking at what handle_body() does for TE_BASE64 and TE_QP cases, I have to wonder if this is enough. The loop seems to stop at (*op == NUL) which follows an old assumption that each line is terminated with NUL, not the new assumption you introduced that each line's length is kept in local variable len.

#19 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Hi,

Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Johannes Schindelin

The function fgets() has a big problem with NUL characters: it reads them, but nobody will know if the NUL comes from the file stream, or was appended at the end of the line.

Quoted 7 lines
Quoted 1 line

So implement a custom read_line() function.

Looking at what handle_body() does for TE_BASE64 and TE_QP cases, I have to wonder if this is enough. The loop seems to stop at (*op == NUL) which follows an old assumption that each line is terminated with NUL, not the new assumption you introduced that each line's length is kept in local variable len.

Of course! But does BASE64 and QP contain NULs? After all, even my custom read_line_with_nul() function adds a NUL IIRC.

Well, the biggest problem here is my lack of time. I thought I would give Tommy a patch which kinda works, and he would actually hold through to brush it up until it shines and gets into git.git, because it is not _my_ itch.

Hmmmmmmm.

Ciao,
Dscho

#20 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Quoted 7 lines · Johannes Schindelin
Quoted 5 lines

Looking at what handle_body() does for TE_BASE64 and TE_QP cases, I have to wonder if this is enough. The loop seems to stop at (*op == NUL) which follows an old assumption that each line is terminated with NUL, not the new assumption you introduced that each line's length is kept in local variable len.

Of course! But does BASE64 and QP contain NULs?

The loop in question iterates over bytes _after_ decoding these encoded lines, and a typical reason you would encode the payload is because it contains something not safe over e-mail transfer, e.g. NUL.

I think decode_transfer_encoding() also needs to become safe against NULs in the payload.

#21 lore raw

Re: [PATCH] mailsplit and mailinfo: gracefully handle NUL characters

Hi,

Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Johannes Schindelin
Quoted 5 lines

Looking at what handle_body() does for TE_BASE64 and TE_QP cases, I have to wonder if this is enough. The loop seems to stop at (*op == NUL) which follows an old assumption that each line is terminated with NUL, not the new assumption you introduced that each line's length is kept in local variable len.

Quoted 8 lines
Quoted 1 line

Of course! But does BASE64 and QP contain NULs?

The loop in question iterates over bytes _after_ decoding these encoded lines, and a typical reason you would encode the payload is because it contains something not safe over e-mail transfer, e.g. NUL.

I think decode_transfer_encoding() also needs to become safe against NULs in the payload.

Okay, I missed that.

Ciao,
Dscho