sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

PATCH In review René Scharfe 4 participants 27 messages Jun 15, 2016

Share & tools

René Scharfe

author

2016-06-15 23:01 UTC #0 lore raw

PATCH 1/2 sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Signed-off-by Rene Scharfe <[email protected]>

---
 sha1_file.c | 41 +++++++++++++++--------------------------
 1 file changed, 15 insertions(+), 26 deletions(-)

sha1_file.c +15 ~~-26~~

index 34d527f..0c3cada 100644
@@ -1177,48 +1177,36 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&

@@ -1226,7 +1214,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}

@@ -1237,13 +1225,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)

Signature · 1 line

2.0.0

René Scharfe

author

2016-06-15 23:01 UTC #1 lore raw

PATCH 2/4 sha1_file: use strncmp for string comparison

Avoid overrunning the existing pack name (p->pack_name, a C string) in the case that the new path is longer by using strncmp instead of memcmp for comparing. While at it, replace the magic constant 4 with a strlen call to document its meaning.

Signed-off-by Rene Scharfe <[email protected]>

---
 sha1_file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

sha1_file.c +2 -1

index 0c3cada..72b8fcb 100644
@@ -1206,7 +1206,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				if (!strncmp(path.buf, p->pack_name,
+					     path.len - strlen(".idx")))
 					break;
 			}
 			if (p == NULL &&

Signature · 1 line

2.0.0

Duy Nguyen

2016-06-15 23:01 UTC #2 lore raw

In reply to René Scharfe

sha1_file.c static void report_pack_garbage(struct string_list *list) · line 1193

Quoted 12 lines · René Scharfe

sha1_file.c +4 -4

-       sprintf(path, "%s/pack", objdir);
-       len = strlen(path);
-       dir = opendir(path);
+       strbuf_addstr(&path, objdir);
+       strbuf_addstr(&path, "/pack");
+       dir = opendir(path.buf);
        if (!dir) {
                if (errno != ENOENT)
                        error("unable to open object pack directory: %s: %s",
-                             path, strerror(errno));
+                             path.buf, strerror(errno));
                return;

Memory leak. The rest looks good.

Signature · 1 line

Duy

René Scharfe

author

2016-06-15 23:01 UTC #3 lore raw

PATCH v2 1/2 sha1_file: replace PATH_MAX buffer wirh strbuf in, prepare_packed_git_one()

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Helped-by Duy Nguyen <[email protected]>

Signed-off-by Rene Scharfe <[email protected]>

---
@Duy: Thanks for catching the missing strbuf_release in the opendir error case.

 sha1_file.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)

sha1_file.c +16 ~~-26~~

index 34d527f..394fa45 100644
@@ -1177,48 +1177,37 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
+		strbuf_release(&path);
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&

@@ -1226,7 +1215,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}

@@ -1237,13 +1226,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)

Signature · 1 line

2.0.0

Jeff King

2016-06-15 23:01 UTC #4 lore raw

Re: [PATCH v2 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in, prepare_packed_git_one()

Quoted 3 lines · René Scharfe

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Yay. Safer, and the end result is much more readable.

-Peff

René Scharfe

author

2016-06-15 23:01 UTC #5 lore raw

PATCH v2 2/2 sha1_file: use strncmp for string comparison

In reply to René Scharfe

Signed-off-by Rene Scharfe <[email protected]>

---
No changes from intial round.

 sha1_file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

sha1_file.c +2 -1

index 394fa45..8adab14 100644
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				if (!strncmp(path.buf, p->pack_name,
+					     path.len - strlen(".idx")))
 					break;
 			}
 			if (p == NULL &&

Signature · 1 line

2.0.0

Jeff King

2016-06-15 23:01 UTC #6 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Quoted 4 lines · René Scharfe

Avoid overrunning the existing pack name (p->pack_name, a C string) in the case that the new path is longer by using strncmp instead of memcmp for comparing. While at it, replace the magic constant 4 with a strlen call to document its meaning.

An unwritten assumption with this code is that we have "foo.idx" and we want to make sure that we are matching "foo.pack" from the existing pack list. Both before and after your patch, I think we would match "foobar.pack". It's probably not a big deal, as we don't expect random junk in the pack directory, but I wonder if it would be better to be explicit, like:

/*
* like ends_with, but return the truncated size of str via
* the "len" parameter.
*/
int strip_suffix(const char *str, const char *suffix, size_t *len)

  {
	size_t suflen = strlen(suffix);
	*len = strlen(str);

	if (len < suflen)
		return 0;
	else if (strcmp(str + len - suflen, suffix))
		return 0;
	else {
		*len -= suflen;
		return 1;
	}

}

int idx_matches_pack(const char *idx, const char *pack)[

  {
	size_t idx_len, pack_len;
	if (!strip_suffix(idx, ".idx", &idx_len) ||
	    !strip_suffix(pack, ".pack", &pack_len))
		return 0;
	if (idx_len != pack_len)
		return 0;
	return !memcmp(idx, pack, idx_len);

}

You'd perhaps want to split idx_matches_pack across the loop below (our idx is actually a strbuf, so you can reuse path->len to avoid a strlen, and you do not have to verify idx_len each time through the loop).

I think strip_suffix would have other uses, too. It's a more featureful version of ends_with, as skip_prefix is to starts_with. E.g., builtin/remote.c:config_read_branches could use it to avoid some magic numbers.

sha1_file.c static void prepare_packed_git_one(char *objdir, int local) · line 1208

Quoted 7 lines

sha1_file.c +0 -0

index 394fa45..8adab14 100644
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */

If we don't follow my suggestion above, we still have this has_extension. This is a reimplementation of ends_with, isn't it? We can probably drop it and just use ends_with.

-Peff

Jeff King

2016-06-15 23:01 UTC #7 lore raw

cover add strip_suffix as an alternative to ends_with

Here's a series to do for ends_with what the recent skip_prefix series did for starts_with. Namely: drop some magic numbers and repeated string literals, and hopefully make things more readable.

The first patch is René's patch 1/2, with the leak fix from Duy and typo fixes in the commit message from me. The rest are new, and replace the changes to prepare_packed_git_one done elsewhere in the thread.

[1/9]: sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one() [2/9]: add strip_suffix function
[3/9]: implement ends_with via strip_suffix
[4/9]: replace has_extension with ends_with
[5/9]: use strip_suffix instead of ends_with in simple cases
[6/9]: index-pack: use strip_suffix to avoid magic numbers
[7/9]: strbuf: implement strbuf_strip_suffix
[8/9]: verify-pack: use strbuf_strip_suffix
[9/9]: prepare_packed_git_one: refactor duplicate-pack check

-Peff

Jeff King

2016-06-15 23:01 UTC #8 lore raw

PATCH 1/9 sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()

From: René Scharfe <[email protected]>

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Helped-by Duy Nguyen <[email protected]>

Signed-off-by Rene Scharfe <[email protected]>

Signed-off-by Jeff King <[email protected]>

---
 sha1_file.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)

sha1_file.c +16 ~~-26~~

index 34d527f..394fa45 100644
@@ -1177,48 +1177,37 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
+		strbuf_release(&path);
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&

@@ -1226,7 +1215,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}

@@ -1237,13 +1226,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #9 lore raw

PATCH 2/9 add strip_suffix function

In reply to Jeff King

Many callers of ends_with want to not only find out whether a string has a suffix, but want to also strip it off. Doing that separately has two minor problems:

1. We often run over the string twice (once to find the suffix, and then once more to find its length to subtract the suffix length).

2. We have to specify the suffix length again, which means either a magic number, or repeating ourselves with strlen("suffix").

Just as we have skip_prefix to avoid these cases with starts_with, we can add a strip_suffix to avoid them with ends_with.

Note that we add two forms of strip_suffix here: one that takes a string, with the resulting length as an out-parameter; and one that takes a pointer/length pair, and reuses the length as an out-parameter. The latter is more efficient when the caller already has the length (e.g., when using strbufs), but it can be easy to confuse the two, as they take the same number and types of parameters.

For that reason, the "mem" form puts its length parameter next to the buffer (since they are a pair), and the string form puts it at the end (since it is an out-parameter). The compiler can notice when you get the order wrong, which should help prevent writing one when you meant the other.

Signed-off-by Jeff King <[email protected]>

---
I hope the word "strip" is OK, as it does not actually NUL-terminate (doing so would make it unusable for many cases). Between the comment below and the "const" in the parameter, I think it should be pretty clear that it does not touch the string. And I could not think of a better word.

 git-compat-util.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

git-compat-util.h +27 -0

index b6f03b3..d044c42 100644
@@ -358,6 +358,33 @@ static inline const char *skip_prefix(const char *str, const char *prefix)
 	return NULL;
 }
 
+/*
+ * If buf ends with suffix, return 1 and subtract the length of the suffix
+ * from *len. Otherwise, return 0 and leave *len untouched.
+ */
+static inline int strip_suffix_mem(const char *buf, size_t *len,
+				   const char *suffix)
+{
+	size_t suflen = strlen(suffix);
+	if (*len < suflen || memcmp(buf + (*len - suflen), suffix, suflen))
+		return 0;
+	*len -= suflen;
+	return 1;
+}
+
+/*
+ * If str ends with suffix, return 1 and set *len to the size of the string
+ * without the suffix. Otherwise, return 0 and set *len to the size of the
+ * string.
+ *
+ * Note that we do _not_ NUL-terminate str to the new length.
+ */
+static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
+{
+	*len = strlen(str);
+	return strip_suffix_mem(str, len, suffix);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ

Signature · 1 line

2.0.0.566.gfe3e6b2

Junio C Hamano

2016-06-15 23:01 UTC #10 lore raw

Re: [PATCH 2/9] add strip_suffix function

Quoted 5 lines · Jeff King

For that reason, the "mem" form puts its length parameter next to the buffer (since they are a pair), and the string form puts it at the end (since it is an out-parameter). The compiler can notice when you get the order wrong, which should help prevent writing one when you meant the other.

Very sensible consideration. I like commits that careful thinking behind them shows through them.

Quoted 7 lines

Signed-off-by Jeff King <[email protected]>
---
I hope the word "strip" is OK, as it does not actually NUL-terminate (doing so would make it unusable for many cases). Between the comment below and the "const" in the parameter, I think it should be pretty clear that it does not touch the string. And I could not think of a better word.

All other words I can think of offhand, trim, chomp, etc., hint shortening of the input string, and by definition shortening of a string implies NUL-termination.

The "mem" variant deals with a counted string, however, so its shortening implies NUL-termination a lot less [*1*] and should be fine.

If we want to avoid implying NUL-termination, the only way to do so would be to use wording that does not hint shortening. At least for the C-string variant, which is measuring the length of the basename part (i.e. basename $str $suffix) without touching anything else, e.g. basename_length("hello.c", ".c", &len), but at the same time you want to make it a boolean to signal if the string ends with the suffix, so perhaps has_suffix("hello.c", ".c", &len)?

[Footnote]

*1* ... but not entirely, because we often NUL-terminate even
counted strings (the buffer returned from read_sha1_file() and the payload of strbuf are two examples).

Quoted 41 lines

 git-compat-util.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

git-compat-util.h +27 -0

index b6f03b3..d044c42 100644
@@ -358,6 +358,33 @@ static inline const char *skip_prefix(const char *str, const char *prefix)
 	return NULL;
 }
 
+/*
+ * If buf ends with suffix, return 1 and subtract the length of the suffix
+ * from *len. Otherwise, return 0 and leave *len untouched.
+ */
+static inline int strip_suffix_mem(const char *buf, size_t *len,
+				   const char *suffix)
+{
+	size_t suflen = strlen(suffix);
+	if (*len < suflen || memcmp(buf + (*len - suflen), suffix, suflen))
+		return 0;
+	*len -= suflen;
+	return 1;
+}
+
+/*
+ * If str ends with suffix, return 1 and set *len to the size of the string
+ * without the suffix. Otherwise, return 0 and set *len to the size of the
+ * string.
+ *
+ * Note that we do _not_ NUL-terminate str to the new length.
+ */
+static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
+{
+	*len = strlen(str);
+	return strip_suffix_mem(str, len, suffix);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ

Jeff King

2016-06-15 23:01 UTC #11 lore raw

Re: [PATCH 2/9] add strip_suffix function

Quoted 10 lines · Junio C Hamano

Quoted 5 lines · Jeff King
For that reason, the "mem" form puts its length parameter next to the buffer (since they are a pair), and the string form puts it at the end (since it is an out-parameter). The compiler can notice when you get the order wrong, which should help prevent writing one when you meant the other.
Very sensible consideration. I like commits that careful thinking behind them shows through them.

I would like to take credit for advanced thinking, but I actually did what felt natural, and only noticed the "compiler will tell you when you are wrong" effect when I got it wrong while writing a later patch in the series. :)

Quoted 7 lines

If we want to avoid implying NUL-termination, the only way to do so would be to use wording that does not hint shortening. At least for the C-string variant, which is measuring the length of the basename part (i.e. basename $str $suffix) without touching anything else, e.g. basename_length("hello.c", ".c", &len), but at the same time you want to make it a boolean to signal if the string ends with the suffix, so perhaps has_suffix("hello.c", ".c", &len)?

I think that invites some confusion with "ends_with", which is the same thing (but just does not take the "len" parameter). We could just add this feature to ends_with, and ask callers who do not care to pass NULL, but that makes those call sites uglier.

Having had a day to mull it over, and having read your email, I think I still prefer strip_suffix.

-Peff

Junio C Hamano

2016-06-15 23:01 UTC #12 lore raw

Re: [PATCH 2/9] add strip_suffix function

Quoted 2 lines · Jeff King

Having had a day to mull it over, and having read your email, I think I still prefer strip_suffix.

That's OK. I was only trying to help you explore different avenues, without having strong preference myself either way.

Jeff King

2016-06-15 23:01 UTC #13 lore raw

PATCH 3/9 implement ends_with via strip_suffix

In reply to Jeff King

The ends_with function is essentially a simplified version of strip_suffix, in which we throw away the stripped length. Implementing it as an inline on top of strip_suffix has two advantages:

1. We save a bit of duplicated code.

2. The suffix is typically a string literal, and we call strlen on it. By making the function inline, many compilers can replace the strlen call with a constant.

Signed-off-by Jeff King <[email protected]>

---
 git-compat-util.h | 7 ++++++-
 strbuf.c          | 9 ---------
 2 files changed, 6 insertions(+), 10 deletions(-)

git-compat-util.h +6 -1

index d044c42..4cfde49 100644
@@ -347,7 +347,6 @@ extern void set_error_routine(void (*routine)(const char *err, va_list params));
 extern void set_die_is_recursing_routine(int (*routine)(void));
 
 extern int starts_with(const char *str, const char *prefix);
-extern int ends_with(const char *str, const char *suffix);
 
 static inline const char *skip_prefix(const char *str, const char *prefix)
 {

@@ -385,6 +384,12 @@ static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
 	return strip_suffix_mem(str, len, suffix);
 }
 
+static inline int ends_with(const char *str, const char *suffix)
+{
+	size_t len;
+	return strip_suffix(str, suffix, &len);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ

strbuf.c +0 -9

index ac62982..99dbeba 100644
@@ -11,15 +11,6 @@ int starts_with(const char *str, const char *prefix)
 			return 0;
 }
 
-int ends_with(const char *str, const char *suffix)
-{
-	int len = strlen(str), suflen = strlen(suffix);
-	if (len < suflen)
-		return 0;
-	else
-		return !strcmp(str + len - suflen, suffix);
-}
-
 /*
  * Used as the default ->buf value, so that people can always assume
  * buf is non NULL and ->buf is NUL terminated even for a freshly

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #14 lore raw

PATCH 4/9 replace has_extension with ends_with

In reply to Jeff King

These two are almost the same function, with the exception that has_extension only matches if there is content before the suffix. So ends_with(".exe", ".exe") is true, but has_extension would not be.

This distinction does not matter to any of the callers, though, and we can just replace uses of has_extension with ends_with. We prefer the "ends_with" name because it is more generic, and there is nothing about the function that requires it to be used for file extensions.

Signed-off-by Jeff King <[email protected]>

---
 builtin/index-pack.c  |  4 ++--
 builtin/verify-pack.c |  4 ++--
 git-compat-util.h     |  7 -------
 help.c                |  2 +-
 refs.c                |  4 ++--
 sha1_file.c           | 10 +++++-----
 6 files changed, 12 insertions(+), 19 deletions(-)

builtin/index-pack.c +2 -2

index 18f57de..46376b6 100644
@@ -1603,7 +1603,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 		die(_("--fix-thin cannot be used without --stdin"));
 	if (!index_name && pack_name) {
 		int len = strlen(pack_name);
-		if (!has_extension(pack_name, ".pack"))
+		if (!ends_with(pack_name, ".pack"))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
 		index_name_buf = xmalloc(len);

@@ -1613,7 +1613,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	}
 	if (keep_msg && !keep_name && pack_name) {
 		int len = strlen(pack_name);
-		if (!has_extension(pack_name, ".pack"))
+		if (!ends_with(pack_name, ".pack"))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
 		keep_name_buf = xmalloc(len);

builtin/verify-pack.c +2 -2

index 66cd2df..2fd29ce 100644
@@ -27,9 +27,9 @@ static int verify_one_pack(const char *path, unsigned int flags)
 	 * normalize these forms to "foo.pack" for "index-pack --verify".
 	 */
 	strbuf_addstr(&arg, path);
-	if (has_extension(arg.buf, ".idx"))
+	if (ends_with(arg.buf, ".idx"))
 		strbuf_splice(&arg, arg.len - 3, 3, "pack", 4);
-	else if (!has_extension(arg.buf, ".pack"))
+	else if (!ends_with(arg.buf, ".pack"))
 		strbuf_add(&arg, ".pack", 5);
 	argv[2] = arg.buf;

git-compat-util.h +0 -7

index 4cfde49..8cf79ae 100644
@@ -594,13 +594,6 @@ static inline size_t xsize_t(off_t len)
 	return (size_t)len;
 }
 
-static inline int has_extension(const char *filename, const char *ext)
-{
-	size_t len = strlen(filename);
-	size_t extlen = strlen(ext);
-	return len > extlen && !memcmp(filename + len - extlen, ext, extlen);
-}
-
 /* in ctype.c, for kwset users */
 extern const char tolower_trans_tbl[256];

help.c +1 -1

index b266b09..372728f 100644
@@ -156,7 +156,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 			continue;
 
 		entlen = strlen(de->d_name) - prefix_len;
-		if (has_extension(de->d_name, ".exe"))
+		if (ends_with(de->d_name, ".exe"))
 			entlen -= 4;
 
 		add_cmdname(cmds, de->d_name + prefix_len, entlen);

refs.c +2 -2

index dc45774..e31adfd 100644
@@ -1162,7 +1162,7 @@ static void read_loose_refs(const char *dirname, struct ref_dir *dir)
 
 		if (de->d_name[0] == '.')
 			continue;
-		if (has_extension(de->d_name, ".lock"))
+		if (ends_with(de->d_name, ".lock"))
 			continue;
 		strbuf_addstr(&refname, de->d_name);
 		refdir = *refs->name

@@ -3233,7 +3233,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data
 
 		if (de->d_name[0] == '.')
 			continue;
-		if (has_extension(de->d_name, ".lock"))
+		if (ends_with(de->d_name, ".lock"))
 			continue;
 		strbuf_addstr(name, de->d_name);
 		if (stat(git_path("logs/%s", name->buf), &st) < 0) {

sha1_file.c +5 -5

index 394fa45..93b794f 100644
@@ -1204,7 +1204,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_setlen(&path, dirnamelen);
 		strbuf_addstr(&path, de->d_name);
 
-		if (has_extension(de->d_name, ".idx")) {
+		if (ends_with(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
 				if (!memcmp(path.buf, p->pack_name, path.len - 4))

@@ -1222,10 +1222,10 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (!report_garbage)
 			continue;
 
-		if (has_extension(de->d_name, ".idx") ||
-		    has_extension(de->d_name, ".pack") ||
-		    has_extension(de->d_name, ".bitmap") ||
-		    has_extension(de->d_name, ".keep"))
+		if (ends_with(de->d_name, ".idx") ||
+		    ends_with(de->d_name, ".pack") ||
+		    ends_with(de->d_name, ".bitmap") ||
+		    ends_with(de->d_name, ".keep"))
 			string_list_append(&garbage, path.buf);
 		else
 			report_garbage("garbage found", path.buf);

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #15 lore raw

PATCH 5/9 use strip_suffix instead of ends_with in simple cases

In reply to Jeff King

When stripping a suffix like:

if (ends_with(str, "foo"))

	buf = xmemdupz(str, strlen(str) - 3);

we can instead use strip_suffix to avoid the constant 3, which must match the literal "foo" (we sometimes use strlen("foo") instead, but that means we are repeating ourselves). The example above becomes:

if (strip_suffix(str, "foo", &len))

	buf = xmemdupz(str, len);

This also saves a strlen(), since we calculate the string length when detecting the suffix.

Note that in some cases we also switch from xstrndup to xmemdupz, which saves a further strlen call.

Signed-off-by Jeff King <[email protected]>

---
 builtin/remote.c | 13 +++++++------
 builtin/repack.c |  5 ++---
 connected.c      |  6 +++---
 help.c           |  5 ++---
 4 files changed, 14 insertions(+), 15 deletions(-)

builtin/remote.c +7 -6

index c9102e8..7c2999e 100644
@@ -265,16 +265,17 @@ static int config_read_branches(const char *key, const char *value, void *cb)
 		struct string_list_item *item;
 		struct branch_info *info;
 		enum { REMOTE, MERGE, REBASE } type;
+		size_t key_len;
 
 		key += 7;
-		if (ends_with(key, ".remote")) {
-			name = xstrndup(key, strlen(key) - 7);
+		if (strip_suffix(key, ".remote", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = REMOTE;
-		} else if (ends_with(key, ".merge")) {
-			name = xstrndup(key, strlen(key) - 6);
+		} else if (strip_suffix(key, ".merge", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = MERGE;
-		} else if (ends_with(key, ".rebase")) {
-			name = xstrndup(key, strlen(key) - 7);
+		} else if (strip_suffix(key, ".rebase", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = REBASE;
 		} else
 			return 0;

builtin/repack.c +2 -3

index ff2216a..a77e743 100644
@@ -83,16 +83,15 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)
 	DIR *dir;
 	struct dirent *e;
 	char *fname;
-	size_t len;
 
 	if (!(dir = opendir(packdir)))
 		return;
 
 	while ((e = readdir(dir)) != NULL) {
-		if (!ends_with(e->d_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(e->d_name, ".pack", &len))
 			continue;
 
-		len = strlen(e->d_name) - strlen(".pack");
 		fname = xmemdupz(e->d_name, len);
 
 		if (!file_exists(mkpath("%s/%s.keep", packdir, fname)))

connected.c +3 -3

index be0253e..dae9c99 100644
@@ -31,6 +31,7 @@ static int check_everything_connected_real(sha1_iterate_fn fn,
 	unsigned char sha1[20];
 	int err = 0, ac = 0;
 	struct packed_git *new_pack = NULL;
+	size_t base_len;
 
 	if (fn(cb_data, sha1))
 		return err;

@@ -38,10 +39,9 @@ static int check_everything_connected_real(sha1_iterate_fn fn,
 	if (transport && transport->smart_options &&
 	    transport->smart_options->self_contained_and_connected &&
 	    transport->pack_lockfile &&
-	    ends_with(transport->pack_lockfile, ".keep")) {
+	    strip_suffix(transport->pack_lockfile, ".keep", &base_len)) {
 		struct strbuf idx_file = STRBUF_INIT;
-		strbuf_addstr(&idx_file, transport->pack_lockfile);
-		strbuf_setlen(&idx_file, idx_file.len - 5); /* ".keep" */
+		strbuf_add(&idx_file, transport->pack_lockfile, base_len);
 		strbuf_addstr(&idx_file, ".idx");
 		new_pack = add_packed_git(idx_file.buf, idx_file.len, 1);
 		strbuf_release(&idx_file);

help.c +2 -3

index 372728f..97567c4 100644
@@ -145,7 +145,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 	len = buf.len;
 
 	while ((de = readdir(dir)) != NULL) {
-		int entlen;
+		size_t entlen;
 
 		if (!starts_with(de->d_name, prefix))
 			continue;

@@ -156,8 +156,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 			continue;
 
 		entlen = strlen(de->d_name) - prefix_len;
-		if (ends_with(de->d_name, ".exe"))
-			entlen -= 4;
+		strip_suffix(de->d_name, ".exe", &entlen);
 
 		add_cmdname(cmds, de->d_name + prefix_len, entlen);
 	}

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #16 lore raw

PATCH 6/9 index-pack: use strip_suffix to avoid magic numbers

In reply to Jeff King

We also switch to using strbufs, which lets us avoid the potentially dangerous combination of a manual malloc followed by a strcpy.

Signed-off-by Jeff King <[email protected]>

---
 builtin/index-pack.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)

builtin/index-pack.c +14 ~~-15~~

index 46376b6..d4b77fd 100644
@@ -1505,7 +1505,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	const char *curr_index;
 	const char *index_name = NULL, *pack_name = NULL;
 	const char *keep_name = NULL, *keep_msg = NULL;
-	char *index_name_buf = NULL, *keep_name_buf = NULL;
+	struct strbuf index_name_buf = STRBUF_INIT,
+		      keep_name_buf = STRBUF_INIT;
 	struct pack_idx_entry **idx_objects;
 	struct pack_idx_option opts;
 	unsigned char pack_sha1[20];

@@ -1602,24 +1603,22 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	if (fix_thin_pack && !from_stdin)
 		die(_("--fix-thin cannot be used without --stdin"));
 	if (!index_name && pack_name) {
-		int len = strlen(pack_name);
-		if (!ends_with(pack_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(pack_name, ".pack", &len))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
-		index_name_buf = xmalloc(len);
-		memcpy(index_name_buf, pack_name, len - 5);
-		strcpy(index_name_buf + len - 5, ".idx");
-		index_name = index_name_buf;
+		strbuf_add(&index_name_buf, pack_name, len);
+		strbuf_addstr(&index_name_buf, ".idx");
+		index_name = index_name_buf.buf;
 	}
 	if (keep_msg && !keep_name && pack_name) {
-		int len = strlen(pack_name);
-		if (!ends_with(pack_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(pack_name, ".pack", &len))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
-		keep_name_buf = xmalloc(len);
-		memcpy(keep_name_buf, pack_name, len - 5);
-		strcpy(keep_name_buf + len - 5, ".keep");
-		keep_name = keep_name_buf;
+		strbuf_add(&keep_name_buf, pack_name, len);
+		strbuf_addstr(&keep_name_buf, ".idx");
+		keep_name = keep_name_buf.buf;
 	}
 	if (verify) {
 		if (!index_name)

@@ -1667,8 +1666,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	else
 		close(input_fd);
 	free(objects);
-	free(index_name_buf);
-	free(keep_name_buf);
+	strbuf_release(&index_name_buf);
+	strbuf_release(&keep_name_buf);
 	if (pack_name == NULL)
 		free((void *) curr_pack);
 	if (index_name == NULL)

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #17 lore raw

PATCH 7/9 strbuf: implement strbuf_strip_suffix

In reply to Jeff King

You can almost get away with just calling "strip_suffix_mem" on a strbuf's buf and len fields. But we also need to move the NUL-terminator to satisfy strbuf's invariants. Let's provide a convenience wrapper that handles this.

Signed-off-by Jeff King <[email protected]>

---
I called strbuf_setlen here because it seemed sensible to use that as an opaque building block, but we are violating the invariant here for a moment by setting sb->len for a moment. I think that is fine, as this is a strbuf function, and can violate the invariant for a moment if it wants to.

But if we care, we can keep a separate "size_t len" variable, and strbuf_setlen to that.

Or we can make it less opaque, and replace the setlen with "sb->buf[sb->len] = 0".

 strbuf.h | 9 +++++++++
 1 file changed, 9 insertions(+)

strbuf.h +9 -0

index e9ad03e..ec11742 100644
@@ -49,6 +49,15 @@ extern int strbuf_reencode(struct strbuf *sb, const char *from, const char *to);
 extern void strbuf_tolower(struct strbuf *sb);
 extern int strbuf_cmp(const struct strbuf *, const struct strbuf *);
 
+static inline int strbuf_strip_suffix(struct strbuf *sb, const char *suffix)
+{
+	if (strip_suffix_mem(sb->buf, &sb->len, suffix)) {
+		strbuf_setlen(sb, sb->len);
+		return 1;
+	} else
+		return 0;
+}
+
 /*
  * Split str (of length slen) at the specified terminator character.
  * Return a null-terminated array of pointers to strbuf objects

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #18 lore raw

PATCH 8/9 verify-pack: use strbuf_strip_suffix

In reply to Jeff King

In this code, we try to convert both "foo.idx" and "foo" into "foo.pack". By stripping the suffix, we can avoid a confusing use of strbuf_splice, and make it clear that both cases are adding ".pack" to the end.

Signed-off-by Jeff King <[email protected]>

---
 builtin/verify-pack.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

builtin/verify-pack.c +3 -4

index 2fd29ce..972579f 100644
@@ -27,10 +27,9 @@ static int verify_one_pack(const char *path, unsigned int flags)
 	 * normalize these forms to "foo.pack" for "index-pack --verify".
 	 */
 	strbuf_addstr(&arg, path);
-	if (ends_with(arg.buf, ".idx"))
-		strbuf_splice(&arg, arg.len - 3, 3, "pack", 4);
-	else if (!ends_with(arg.buf, ".pack"))
-		strbuf_add(&arg, ".pack", 5);
+	if (strbuf_strip_suffix(&arg, ".idx") ||
+	    !ends_with(arg.buf, ".pack"))
+		strbuf_addstr(&arg, ".pack");
 	argv[2] = arg.buf;
 
 	memset(&index_pack, 0, sizeof(index_pack));

Signature · 1 line

2.0.0.566.gfe3e6b2

Jeff King

2016-06-15 23:01 UTC #19 lore raw

PATCH 9/9 prepare_packed_git_one: refactor duplicate-pack check

In reply to Jeff King

When we are reloading the list of packs, we check whether a particular pack has been loaded. This is slightly tricky, because we load packs based on the presence of their ".idx" files, but record the name of the matching ".pack" file. Therefore we want to compare their bases.

The existing code stripped off ".idx" from a file we found, then compared that whole base length to strings containing the ".pack" version. This meant we could end up comparing bytes past what the ".pack" string contained, if the ".idx" file name was much longer.

In practice, it worked OK because memcmp would end up seeing a difference in the two strings and would return early before hitting the full length. However, memcmp may sometimes read extra bytes past a difference (e.g., because it is comparing 64-bit words), or is even free to compare in reverse order.

Furthermore, our memcmp made no guarantees that we matched the whole pack name, up to ".pack". So "foo.idx" would match "foo-bar.pack", which is wrong (but does not typically happen, because our pack names have a fixed size).

We can fix both issues, avoid magic numbers, and document that we expect to compare against a string with ".pack" by using strip_suffix.

Signed-off-by Jeff King <[email protected]>

---
This is a teeny bit less efficient than it could be, because we are verifying our assumption at run-time that each pack name ends in ".pack". I'd venture to say if we cared about efficiency here, the low hanging fruit would be to avoid the O(n^2) loop to find duplicate pack names in the first place.

 sha1_file.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

sha1_file.c +7 -2

index 93b794f..129a4c5 100644
@@ -1197,6 +1197,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
 		struct packed_git *p;
+		size_t base_len;
 
 		if (is_dot_or_dotdot(de->d_name))
 			continue;

@@ -1204,10 +1205,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_setlen(&path, dirnamelen);
 		strbuf_addstr(&path, de->d_name);
 
-		if (ends_with(de->d_name, ".idx")) {
+		base_len = path.len;
+		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				size_t len;
+				if (strip_suffix(p->pack_name, ".pack", &len) &&
+				    len == base_len &&
+				    !memcmp(p->pack_name, path.buf, len))
 					break;
 			}
 			if (p == NULL &&

Signature · 1 line

2.0.0.566.gfe3e6b2

Junio C Hamano

2016-06-15 23:01 UTC #20 lore raw

Re: [PATCH 0/9] add strip_suffix as an alternative to ends_with

In reply to Jeff King

Quoted 8 lines · Jeff King

Here's a series to do for ends_with what the recent skip_prefix series did for starts_with. Namely: drop some magic numbers and repeated string literals, and hopefully make things more readable.
The first patch is René's patch 1/2, with the leak fix from Duy and typo fixes in the commit message from me. The rest are new, and replace the changes to prepare_packed_git_one done elsewhere in the thread.

Quoted 9 lines

[1/9]: sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one() [2/9]: add strip_suffix function
[3/9]: implement ends_with via strip_suffix
[4/9]: replace has_extension with ends_with
[5/9]: use strip_suffix instead of ends_with in simple cases
[6/9]: index-pack: use strip_suffix to avoid magic numbers
[7/9]: strbuf: implement strbuf_strip_suffix
[8/9]: verify-pack: use strbuf_strip_suffix
[9/9]: prepare_packed_git_one: refactor duplicate-pack check

Looked very sensible. Thanks all ;-)

René Scharfe

author

2016-06-15 23:01 UTC #21 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

In reply to Jeff King

Am 30.06.2014 15:43, schrieb Jeff King:

Quoted 11 lines · René Scharfe

Quoted 5 lines
Avoid overrunning the existing pack name (p->pack_name, a C string) in the case that the new path is longer by using strncmp instead of memcmp for comparing. While at it, replace the magic constant 4 with a strlen call to document its meaning.
An unwritten assumption with this code is that we have "foo.idx" and we want to make sure that we are matching "foo.pack" from the existing pack list. Both before and after your patch, I think we would match "foobar.pack".

Yes, indeed.

Quoted 3 lines

It's probably not a big deal, as we don't expect random junk in the pack directory, but I wonder if it would be better to be explicit, like:

<snip>

Here's a simpler approach:

-- >8 --
Subject: [PATCH v2 3/2] sha1_file: more precise packname matching

Consider the full length of the already loaded pack names when checking for duplicates.

Signed-off-by Rene Scharfe <[email protected]>

---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

sha1_file.c +4 -2

index b7ad6c1..a13fbbd 100644
@@ -1206,10 +1206,12 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
+			size_t len = path.len - strlen(".idx");
+
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!strncmp(path.buf, p->pack_name,
-					     path.len - strlen(".idx")))
+				if (!strncmp(p->pack_name, path.buf, len) &&
+				    !strcmp(p->pack_name + len, ".pack"))
 					break;
 			}
 			if (p == NULL &&

Signature · 1 line

2.0.0

Jeff King

2016-06-15 23:01 UTC #22 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Quoted 7 lines · René Scharfe

Quoted 3 lines
It's probably not a big deal, as we don't expect random junk in the pack directory, but I wonder if it would be better to be explicit, like:
<snip>
Here's a simpler approach:

I agree that solves the problem. However, I'm about to post an alternative series that also replaces has_extension with strip_suffix, which I think ends up a bit nicer.

-Peff

Duy Nguyen

2016-06-15 23:01 UTC #23 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

In reply to Jeff King

Quoted 11 lines · Jeff King

sha1_file.c static void prepare_packed_git_one(char *objdir, int local) · line 1208
Quoted 7 lines
sha1_file.c +0 -0
index 394fa45..8adab14 100644
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
              if (has_extension(de->d_name, ".idx")) {
                      /* Don't reopen a pack we already have. */
If we don't follow my suggestion above, we still have this has_extension. This is a reimplementation of ends_with, isn't it? We can probably drop it and just use ends_with.

This calls for another patch if we just want to kill has_extension() in favor of ends_with(). There are 12 call sites of it.

Signature · 1 line

Duy

Jeff King

2016-06-15 23:01 UTC #24 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Quoted 13 lines · Duy Nguyen

Quoted 11 lines · Jeff King
sha1_file.c static void prepare_packed_git_one(char *objdir, int local) · line 1208
Quoted 7 lines
sha1_file.c +0 -0
index 394fa45..8adab14 100644
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
              if (has_extension(de->d_name, ".idx")) {
                      /* Don't reopen a pack we already have. */
If we don't follow my suggestion above, we still have this has_extension. This is a reimplementation of ends_with, isn't it? We can probably drop it and just use ends_with.

Quoted 2 lines

This calls for another patch if we just want to kill has_extension() in favor of ends_with(). There are 12 call sites of it.

Yes. Some of those would want to become ends_with, and some would actually want to become strip_suffix. I'm working up a series now.

-Peff

René Scharfe

author

2016-06-15 23:01 UTC #25 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Am 30.06.2014 16:22, schrieb Jeff King:

Quoted 15 lines · Duy Nguyen

Quoted 14 lines · Jeff King
Quoted 11 lines
sha1_file.c static void prepare_packed_git_one(char *objdir, int local) · line 1208
Quoted 7 lines
sha1_file.c +0 -0
index 394fa45..8adab14 100644
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
               if (has_extension(de->d_name, ".idx")) {
                       /* Don't reopen a pack we already have. */
If we don't follow my suggestion above, we still have this has_extension. This is a reimplementation of ends_with, isn't it? We can probably drop it and just use ends_with.

Quoted 5 lines

Quoted 2 lines
This calls for another patch if we just want to kill has_extension() in favor of ends_with(). There are 12 call sites of it.
Yes. Some of those would want to become ends_with, and some would actually want to become strip_suffix. I'm working up a series now.

NB: has_extension is almost the same as ends_with, but it also checks if the string is longer than just the extension:

	ends_with("x", "x") => 1
	has_extension("x", "x") => 0

René

Jeff King

2016-06-15 23:01 UTC #26 lore raw

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Quoted 19 lines · René Scharfe

Am 30.06.2014 16:22, schrieb Jeff King:
Quoted 18 lines · Duy Nguyen
Quoted 16 lines · Jeff King
Quoted 11 lines
sha1_file.c static void prepare_packed_git_one(char *objdir, int local) · line 1208
Quoted 7 lines
sha1_file.c +0 -0
index 394fa45..8adab14 100644
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
              if (has_extension(de->d_name, ".idx")) {
                      /* Don't reopen a pack we already have. */
If we don't follow my suggestion above, we still have this has_extension. This is a reimplementation of ends_with, isn't it? We can probably drop it and just use ends_with.
This calls for another patch if we just want to kill has_extension() in favor of ends_with(). There are 12 call sites of it.

Quoted 8 lines

Quoted 2 lines
Yes. Some of those would want to become ends_with, and some would actually want to become strip_suffix. I'm working up a series now.
NB: has_extension is almost the same as ends_with, but it also checks if the string is longer than just the extension:
	ends_with("x", "x") => 1
	has_extension("x", "x") => 0

Thanks, I didn't notice that. I don't think the distinction is important in any callers, but I'll double check.

-Peff