Skip to contentSkip to conversation

improving svn-fe error handling

PATCH In review Jonathan Nieder 1 participant 6 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

cover improving svn-fe error handling

Hi,

Currently malformed streams (e.g., ending early) and input errors when reading a dump file can cause svn-fe to dereference a NULL pointer or otherwise act in a confusing way. Especially since incremental imports work now, I think a saner behavior is to error out with a clear error message; the operator can then fix the problem and resume. This series is an attempt in that direction.

Still to do:
- add tests (reading /dev/full? how else to provoke an input error?)
- change buffer_read_string to disallow short reads, to simplify

I am sending it out now because patches 1, 2, and 3 make API changes that I find convenient, so I'd be interested in your thoughts before I commit too heavily to a bad idea.

Patch 1 introduces buffer_ferror, to check if errors were encountered reading from the line_buffer. Unlike buffer_deinit, this can be used after any operation so it can be used early, when errno is still valid.

Patches 2 and 3 make operations that can be partially completed still return some indication of success or failure.

Patch 4 is an example application, using the new API to make svn-fe a little better at detecting malformed dump files.

Thoughts?
Jonathan Nieder (4):
vcs-svn: allow input errors to be detected promptly
vcs-svn: make buffer_skip_bytes return length read
vcs-svn: make buffer_copy_bytes return length read
vcs-svn: improve reporting of input errors

 vcs-svn/fast_export.c   |   13 +++++++++-
 vcs-svn/line_buffer.c   |   36 ++++++++++++++++++-------------
 vcs-svn/line_buffer.h   |    6 +++-
 vcs-svn/line_buffer.txt |    3 +-
 vcs-svn/svndump.c       |   53 +++++++++++++++++++++++++++++++++++++---------
 5 files changed, 80 insertions(+), 31 deletions(-)
#1 lore raw

PATCH 1/4 vcs-svn: allow input errors to be detected promptly

Date: Sun, 10 Oct 2010 21:51:21 -0500

The line_buffer library silently flags input errors until buffer_deinit time; unfortunately, by that point usually errno is invalid. Expose the error flag so callers can check for and report errors early for easy debugging.

	some_error_prone_operation(...);
	if (buffer_ferror())
		return error("input error: %s", strerror(errno));

---
Has been in use for a while, but still I'd be interested to know if this is sane.

 vcs-svn/line_buffer.c |    5 +++++
 vcs-svn/line_buffer.h |    1 +
 2 files changed, 6 insertions(+), 0 deletions(-)
vcs-svn/line_buffer.c +5 -0
index 1543567..1b5ac8a 100644
@@ -35,6 +35,11 @@ int buffer_deinit(void)
 	return err;
 }
 
+int buffer_ferror(void)
+{
+	return ferror(infile);
+}
+
 /* Read a line without trailing newline. */
 char *buffer_read_line(void)
 {
vcs-svn/line_buffer.h +1 -0
index 9c78ae1..5a19873 100644
@@ -3,6 +3,7 @@
 
 int buffer_init(const char *filename);
 int buffer_deinit(void);
+int buffer_ferror(void);
 char *buffer_read_line(void);
 char *buffer_read_string(uint32_t len);
 void buffer_copy_bytes(uint32_t len);
Signature · 1 line
1.7.2.3.554.gc9b5c.dirty
#2 lore raw

PATCH 2/4 vcs-svn: make buffer_skip_bytes return length read

In reply to Jonathan Nieder

Date: Sun, 10 Oct 2010 21:44:21 -0500

Currently there is no way to detect when input ended if it ended early during buffer_skip_bytes. Tell the calling program how many bytes were actually skipped for easier debugging.

Existing callers will still ignore early EOF.

---
 vcs-svn/line_buffer.c   |   13 +++++++------
 vcs-svn/line_buffer.h   |    2 +-
 vcs-svn/line_buffer.txt |    3 ++-
 3 files changed, 10 insertions(+), 8 deletions(-)
vcs-svn/line_buffer.c +7 -6
index 1b5ac8a..58e076f 100644
@@ -86,14 +86,15 @@ void buffer_copy_bytes(uint32_t len)
 	}
 }
 
-void buffer_skip_bytes(uint32_t len)
+uint32_t buffer_skip_bytes(uint32_t nbytes)
 {
-	uint32_t in;
-	while (len > 0 && !feof(infile) && !ferror(infile)) {
-		in = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;
-		in = fread(byte_buffer, 1, in, infile);
-		len -= in;
+	uint32_t done = 0;
+	while (done < nbytes && !feof(infile) && !ferror(infile)) {
+		uint32_t len = nbytes - done;
+		uint32_t in = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;
+		done += fread(byte_buffer, 1, in, infile);
 	}
+	return done;
 }
 
 void buffer_reset(void)
vcs-svn/line_buffer.h +1 -1
index 5a19873..b9dd929 100644
@@ -7,7 +7,7 @@ int buffer_ferror(void);
 char *buffer_read_line(void);
 char *buffer_read_string(uint32_t len);
 void buffer_copy_bytes(uint32_t len);
-void buffer_skip_bytes(uint32_t len);
+uint32_t buffer_skip_bytes(uint32_t len);
 void buffer_reset(void);
 
 #endif
vcs-svn/line_buffer.txt +2 -1
index 8906fb1..a08ad8a 100644
@@ -52,7 +52,8 @@ Functions
 
 `buffer_skip_bytes`::
 	Discards `len` bytes from the input stream (stopping early
-	if necessary because of an error or eof).
+	if necessary because of an error or eof).  Return value is
+	the number of bytes successfully read.
 
 `buffer_reset`::
 	Deallocates non-static buffers.
Signature · 1 line
1.7.2.3.554.gc9b5c.dirty
#3 lore raw

PATCH 3/4 vcs-svn: make buffer_copy_bytes return length read

In reply to Jonathan Nieder

Currently buffer_copy_bytes does not report to its caller whether it encountered an early end of file.

Add a return value representing the number of bytes read (but not the number of bytes copied). This way all three unusual conditions can be distinguished: input error with buffer_ferror, output error with ferror(outfile), input error or early end of input by checking the return value.

---
Intuitive?

 vcs-svn/line_buffer.c |   18 +++++++++---------
 vcs-svn/line_buffer.h |    3 ++-
 2 files changed, 11 insertions(+), 10 deletions(-)
vcs-svn/line_buffer.c +9 -9
index 58e076f..36f177c 100644
@@ -71,19 +71,19 @@ char *buffer_read_string(uint32_t len)
 	return ferror(infile) ? NULL : s;
 }
 
-void buffer_copy_bytes(uint32_t len)
+uint32_t buffer_copy_bytes(uint32_t nbytes)
 {
-	uint32_t in;
-	while (len > 0 && !feof(infile) && !ferror(infile)) {
-		in = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;
+	uint32_t done = 0;
+	while (done < nbytes && !feof(infile) && !ferror(infile)) {
+		uint32_t len = nbytes - done;
+		uint32_t in = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;
 		in = fread(byte_buffer, 1, in, infile);
-		len -= in;
+		done += in;
 		fwrite(byte_buffer, 1, in, stdout);
-		if (ferror(stdout)) {
-			buffer_skip_bytes(len);
-			return;
-		}
+		if (ferror(stdout))
+			return done + buffer_skip_bytes(nbytes - done);
 	}
+	return done;
 }
 
 uint32_t buffer_skip_bytes(uint32_t nbytes)
vcs-svn/line_buffer.h +2 -1
index b9dd929..7766636 100644
@@ -6,7 +6,8 @@ int buffer_deinit(void);
 int buffer_ferror(void);
 char *buffer_read_line(void);
 char *buffer_read_string(uint32_t len);
-void buffer_copy_bytes(uint32_t len);
+/* Returns number of bytes read (not necessarily written). */
+uint32_t buffer_copy_bytes(uint32_t len);
 uint32_t buffer_skip_bytes(uint32_t len);
 void buffer_reset(void);
 
Signature · 1 line
1.7.2.3.554.gc9b5c.dirty
#4 lore raw

PATCH 4/4 vcs-svn: improve reporting of input errors

In reply to Jonathan Nieder

Catch input errors and exit early enough to print a reasonable diagnosis based on errno.

---
Patches 1 and 2 first appeared in the svndiff0 series[1] (and that is the application I have in mind in sending them now).

This patch #4 is just a demo. Untested, I'm afraid. I would be very interested in ideas for automatically testing it --- how to easily stimulate error paths?

Good night,
Jonathan

[1] http://thread.gmane.org/gmane.comp.version-control.git/158731

 vcs-svn/fast_export.c |   13 ++++++++++-
 vcs-svn/svndump.c     |   53 ++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 53 insertions(+), 13 deletions(-)
vcs-svn/fast_export.c +11 -2
index 256a052..dc0ae4e 100644
@@ -62,14 +62,23 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log,
 	printf("progress Imported commit %d.\n\n", revision);
 }
 
+static void die_short_read(void)
+{
+	if (buffer_ferror())
+		die_errno("error reading dump file");
+	die("invalid dump: unexpected end of file");
+}
+
 void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len)
 {
 	if (mode == REPO_MODE_LNK) {
 		/* svn symlink blobs start with "link " */
-		buffer_skip_bytes(5);
 		len -= 5;
+		if (buffer_skip_bytes(5) != 5)
+			die_short_read();
 	}
 	printf("blob\nmark :%d\ndata %d\n", mark, len);
-	buffer_copy_bytes(len);
+	if (buffer_copy_bytes(len) != len)
+		die_short_read();
 	fputc('\n', stdout);
 }
vcs-svn/svndump.c +42 -11

103 lines not on this page · raw message

Signature · 1 line
1.7.2.3.554.gc9b5c.dirty
#5 lore raw

Re: [PATCH 4/4] vcs-svn: improve reporting of input errors

Quoted 3 lines · Jonathan Nieder
vcs-svn/svndump.c +0 -0
@@ -107,20 +107,37 @@ static void init_keys(void)

[...]

vcs-svn/svndump.c static void init_keys(void) · line 140

Quoted 5 lines
vcs-svn/svndump.c +2 -0
 		} else if (!strncmp(t, "V ", 2)) {
 			len = atoi(&t[2]);
 			val = buffer_read_string(len);
+			if (!t)
+				die_short_read();

This should have said "if (!val)". Sorry for the confusion.