Skip to contentSkip to conversation

Fetching mass of objects at once

PATCH In review Petr Baudis 1 participant 7 messages Jun 15, 2016 Documentation

lore
Share & tools
lore

Petr Baudis

author
#0 lore raw

cover Fetching mass of objects at once

Hi,

this series implements support for fetching many objects at once with
git-local-fetch and git-http-fetch, greatly helping Cogito's performance. It would be of course nice to have this in Git 1.4.2, but it's probably too late for that.

I didn't bother to convert git-ssh-fetch since I think noone uses that
anymore anyway (why would they...), or only some legacy users do.

I will followup with a patch for Cogito to take advantage of this. It's
now roughly on par with Git in cloning speed.

author
#1 lore raw

PATCH 1/4 Make pull() take some implicit data as explicit arguments

Currently it's a bit weird that pull() takes a single argument describing the commit but takes the write_ref from a global variable. This makes it take that as a parameter as well, which might be nicer for the libification in the future, but especially it will make for nicer code when we implement pull()ing multiple commits at once.

Signed-off-by Petr Baudis <[email protected]>

---

 fetch.c       |    6 ++----
 fetch.h       |   11 ++++-------
 http-fetch.c  |    4 ++--
 local-fetch.c |    4 ++--
 ssh-fetch.c   |    4 ++--
 5 files changed, 12 insertions(+), 17 deletions(-)
fetch.c +2 -4
index 989d7a4..3255cc6 100644
@@ -8,9 +8,6 @@ #include "tag.h"
 #include "blob.h"
 #include "refs.h"
 
-const char *write_ref = NULL;
-const char *write_ref_log_details = NULL;
-
 int get_tree = 0;
 int get_history = 0;
 int get_all = 0;

@@ -213,7 +210,8 @@ static int mark_complete(const char *pat
 	return 0;
 }
 
-int pull(char *target)
+int pull(char *target, const char *write_ref,
+         const char *write_ref_log_details)
 {
 	struct ref_lock *lock = NULL;
 	unsigned char sha1[20];
fetch.h +4 -7
index 841bb1a..7bda355 100644
@@ -22,12 +22,6 @@ extern void prefetch(unsigned char *sha1
  */
 extern int fetch_ref(char *ref, unsigned char *sha1);
 
-/* If set, the ref filename to write the target value to. */
-extern const char *write_ref;
-
-/* If set additional text will appear in the ref log. */
-extern const char *write_ref_log_details;
-
 /* Set to fetch the target tree. */
 extern int get_tree;
 

@@ -46,6 +40,9 @@ extern int get_recover;
 /* Report what we got under get_verbosely */
 extern void pull_say(const char *, const char *);
 
-extern int pull(char *target);
+/* If write_ref is set, the ref filename to write the target value to. */
+/* If write_ref_log_details is set, additional text will appear in the ref log. */
+extern int pull(char *target, const char *write_ref,
+		const char *write_ref_log_details);
 
 #endif /* PULL_H */
http-fetch.c +2 -2
index dc286b7..963d439 100644
@@ -1216,6 +1216,7 @@ int fetch_ref(char *ref, unsigned char *
 
 int main(int argc, char **argv)
 {
+	const char *write_ref = NULL;
 	char *commit_id;
 	char *url;
 	char *path;

@@ -1250,7 +1251,6 @@ int main(int argc, char **argv)
 	}
 	commit_id = argv[arg];
 	url = argv[arg + 1];
-	write_ref_log_details = url;
 
 	http_init();
 

@@ -1268,7 +1268,7 @@ int main(int argc, char **argv)
 			alt->path_len = strlen(path);
 	}
 
-	if (pull(commit_id))
+	if (pull(commit_id, write_ref, url))
 		rc = 1;
 
 	http_cleanup();
local-fetch.c +2 -2
index 65a803a..308ed00 100644
@@ -204,6 +204,7 @@ static const char local_pull_usage[] =
  */
 int main(int argc, char **argv)
 {
+	const char *write_ref = NULL;
 	char *commit_id;
 	int arg = 1;
 

@@ -240,9 +241,8 @@ int main(int argc, char **argv)
 		usage(local_pull_usage);
 	commit_id = argv[arg];
 	path = argv[arg + 1];
-	write_ref_log_details = path;
 
-	if (pull(commit_id))
+	if (pull(commit_id, write_ref, path))
 		return 1;
 
 	return 0;
ssh-fetch.c +2 -2
index a8a6cfb..aef3aa4 100644
@@ -123,6 +123,7 @@ static const char ssh_fetch_usage[] =
   " [-c] [-t] [-a] [-v] [--recover] [-w ref] commit-id url";
 int main(int argc, char **argv)
 {
+	const char *write_ref = NULL;
 	char *commit_id;
 	char *url;
 	int arg = 1;

@@ -159,7 +160,6 @@ int main(int argc, char **argv)
 	}
 	commit_id = argv[arg];
 	url = argv[arg + 1];
-	write_ref_log_details = url;
 
 	if (setup_connection(&fd_in, &fd_out, prog, url, arg, argv + 1))
 		return 1;

@@ -167,7 +167,7 @@ int main(int argc, char **argv)
 	if (get_version())
 		return 1;
 
-	if (pull(commit_id))
+	if (pull(commit_id, write_ref, url))
 		return 1;
 
 	return 0;
author
#2 lore raw

PATCH 2/4 Make pull() support fetching multiple targets at once

In reply to Petr Baudis

pull() now takes an array of arguments instead of just one of each kind. Currently, no users use the new capability, but that'll change.

Signed-off-by Petr Baudis <[email protected]>

---

 fetch.c       |   78 +++++++++++++++++++++++++++++++++------------------------
 fetch.h       |    2 +
 http-fetch.c  |    2 +
 local-fetch.c |    2 +
 ssh-fetch.c   |    2 +
 5 files changed, 49 insertions(+), 37 deletions(-)
fetch.c +45 -33
index 3255cc6..281df61 100644
@@ -210,55 +210,67 @@ static int mark_complete(const char *pat
 	return 0;
 }
 
-int pull(char *target, const char *write_ref,
+int pull(int targets, char **target, const char **write_ref,
          const char *write_ref_log_details)
 {
-	struct ref_lock *lock = NULL;
-	unsigned char sha1[20];
+	struct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));
+	unsigned char *sha1 = xmalloc(targets * 20);
 	char *msg;
 	int ret;
+	int i;
 
 	save_commit_buffer = 0;
 	track_object_refs = 0;
-	if (write_ref) {
-		lock = lock_ref_sha1(write_ref, NULL, 0);
-		if (!lock) {
-			error("Can't lock ref %s", write_ref);
-			return -1;
+
+	for (i = 0; i < targets; i++) {
+		if (!write_ref[i])
+			continue;
+
+		lock[i] = lock_ref_sha1(write_ref[i], NULL, 0);
+		if (!lock[i]) {
+			error("Can't lock ref %s", write_ref[i]);
+			goto unlock_and_fail;
 		}
 	}
 
 	if (!get_recover)
 		for_each_ref(mark_complete);
 
-	if (interpret_target(target, sha1)) {
-		error("Could not interpret %s as something to pull", target);
-		if (lock)
-			unlock_ref(lock);
-		return -1;
+	for (i = 0; i < targets; i++) {
+		if (interpret_target(target[i], &sha1[20 * i])) {
+			error("Could not interpret %s as something to pull", target[i]);
+			goto unlock_and_fail;
+		}
+		if (process(lookup_unknown_object(&sha1[20 * i])))
+			goto unlock_and_fail;
 	}
-	if (process(lookup_unknown_object(sha1))) {
-		if (lock)
-			unlock_ref(lock);
-		return -1;
+
+	if (loop())
+		goto unlock_and_fail;
+
+	if (write_ref_log_details) {
+		msg = xmalloc(strlen(write_ref_log_details) + 12);
+		sprintf(msg, "fetch from %s", write_ref_log_details);
+	} else {
+		msg = NULL;
 	}
-	if (loop()) {
-		if (lock)
-			unlock_ref(lock);
-		return -1;
+	for (i = 0; i < targets; i++) {
+		if (!write_ref[i])
+			continue;
+		ret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : "fetch (unknown)");
+		lock[i] = NULL;
+		if (ret)
+			goto unlock_and_fail;
 	}
+	if (msg)
+		free(msg);
 
-	if (write_ref) {
-		if (write_ref_log_details) {
-			msg = xmalloc(strlen(write_ref_log_details) + 12);
-			sprintf(msg, "fetch from %s", write_ref_log_details);
-		}
-		else
-			msg = NULL;
-		ret = write_ref_sha1(lock, sha1, msg ? msg : "fetch (unknown)");
-		if (msg)
-			free(msg);
-		return ret;
-	}
 	return 0;
+
+
+unlock_and_fail:
+	for (i = 0; i < targets; i++)
+		if (lock[i])
+			unlock_ref(lock[i]);
+	return -1;
 }
fetch.h +1 -1
index 7bda355..75e48af 100644
@@ -42,7 +42,7 @@ extern void pull_say(const char *, const
 
 /* If write_ref is set, the ref filename to write the target value to. */
 /* If write_ref_log_details is set, additional text will appear in the ref log. */
-extern int pull(char *target, const char *write_ref,
+extern int pull(int targets, char **target, const char **write_ref,
 		const char *write_ref_log_details);
 
 #endif /* PULL_H */
http-fetch.c +1 -1
index 963d439..bc67db1 100644
@@ -1268,7 +1268,7 @@ int main(int argc, char **argv)
 			alt->path_len = strlen(path);
 	}
 
-	if (pull(commit_id, write_ref, url))
+	if (pull(1, &commit_id, &write_ref, url))
 		rc = 1;
 
 	http_cleanup();
local-fetch.c +1 -1
index 308ed00..eb19f1a 100644
@@ -242,7 +242,7 @@ int main(int argc, char **argv)
 	commit_id = argv[arg];
 	path = argv[arg + 1];
 
-	if (pull(commit_id, write_ref, path))
+	if (pull(1, &commit_id, &write_ref, path))
 		return 1;
 
 	return 0;
ssh-fetch.c +1 -1
index aef3aa4..6e16568 100644
@@ -167,7 +167,7 @@ int main(int argc, char **argv)
 	if (get_version())
 		return 1;
 
-	if (pull(commit_id, write_ref, url))
+	if (pull(1, &commit_id, &write_ref, url))
 		return 1;
 
 	return 0;
author
#3 lore raw

PATCH 3/4 Teach git-local-fetch the --stdin switch

In reply to Petr Baudis

This makes it possible to fetch many commits (refs) at once, greatly speeding up cg-clone.

Signed-off-by Petr Baudis <[email protected]>

---

 Documentation/git-local-fetch.txt |    6 ++++++
 fetch.c                           |   40 +++++++++++++++++++++++++++++++++++++
 fetch.h                           |    6 ++++++
 local-fetch.c                     |   32 ++++++++++++++++++++----------
 4 files changed, 74 insertions(+), 10 deletions(-)
Documentation/git-local-fetch.txt +6 -0
index 87abec1..bd19820 100644
@@ -29,6 +29,12 @@ OPTIONS
         Writes the commit-id into the filename under $GIT_DIR/refs/<filename> on
         the local end after the transfer is complete.
 
+--stdin::
+	Instead of a commit id on the commandline (which is not expected in this
+	case), 'git-local-fetch' excepts lines on stdin in the format
+
+		<commit-id>['\t'<filename-as-in--w>]
+
 Author
 ------
 Written by Junio C Hamano <[email protected]>
fetch.c +40 -0
index 281df61..2151c7b 100644
@@ -7,6 +7,7 @@ #include "tree-walk.h"
 #include "tag.h"
 #include "blob.h"
 #include "refs.h"
+#include "strbuf.h"
 
 int get_tree = 0;
 int get_history = 0;

@@ -210,6 +211,45 @@ static int mark_complete(const char *pat
 	return 0;
 }
 
+int pull_targets_stdin(char ***target, const char ***write_ref)
+{
+	int targets = 0, targets_alloc = 0;
+	struct strbuf buf;
+	*target = NULL; *write_ref = NULL;
+	strbuf_init(&buf);
+	while (1) {
+		char *rf_one = NULL;
+		char *tg_one;
+
+		read_line(&buf, stdin, '\n');
+		if (buf.eof)
+			break;
+		tg_one = buf.buf;
+		rf_one = strchr(tg_one, '\t');
+		if (rf_one)
+			*rf_one++ = 0;
+
+		if (targets >= targets_alloc) {
+			targets_alloc = targets_alloc ? targets_alloc * 2 : 64;
+			*target = xrealloc(*target, targets_alloc * sizeof(**target));
+			*write_ref = xrealloc(*write_ref, targets_alloc * sizeof(**write_ref));
+		}
+		(*target)[targets] = strdup(tg_one);
+		(*write_ref)[targets] = rf_one ? strdup(rf_one) : NULL;
+		targets++;
+	}
+	return targets;
+}
+
+void pull_targets_free(int targets, char **target, const char **write_ref)
+{
+	while (targets--) {
+		free(target[targets]);
+		if (write_ref[targets])
+			free((char *) write_ref[targets]);
+	}
+}
+
 int pull(int targets, char **target, const char **write_ref,
          const char *write_ref_log_details)
 {
fetch.h +6 -0
index 75e48af..be48c6f 100644
@@ -40,6 +40,12 @@ extern int get_recover;
 /* Report what we got under get_verbosely */
 extern void pull_say(const char *, const char *);
 
+/* Load pull targets from stdin */
+extern int pull_targets_stdin(char ***target, const char ***write_ref);
+
+/* Free up loaded targets */
+extern void pull_targets_free(int targets, char **target, const char **write_ref);
+
 /* If write_ref is set, the ref filename to write the target value to. */
 /* If write_ref_log_details is set, additional text will appear in the ref log. */
 extern int pull(int targets, char **target, const char **write_ref,
local-fetch.c +22 -10
index eb19f1a..84b68a6 100644
@@ -8,8 +8,9 @@ #include "fetch.h"
 static int use_link = 0;
 static int use_symlink = 0;
 static int use_filecopy = 1;
+static int commits_on_stdin = 0;
 
-static char *path; /* "Remote" git repository */
+static const char *path; /* "Remote" git repository */
 
 void prefetch(unsigned char *sha1)
 {

@@ -194,7 +195,7 @@ int fetch_ref(char *ref, unsigned char *
 }
 
 static const char local_pull_usage[] =
-"git-local-fetch [-c] [-t] [-a] [-v] [-w filename] [--recover] [-l] [-s] [-n] commit-id path";
+"git-local-fetch [-c] [-t] [-a] [-v] [-w filename] [--recover] [-l] [-s] [-n] [--stdin] commit-id path";
 
 /* 
  * By default we only use file copy.

@@ -202,10 +203,11 @@ static const char local_pull_usage[] =
  * If -s is specified, then a symlink is attempted.
  * If -n is _not_ specified, then a regular file-to-file copy is done.
  */
-int main(int argc, char **argv)
+int main(int argc, const char **argv)
 {
-	const char *write_ref = NULL;
-	char *commit_id;
+	int commits;
+	const char **write_ref = NULL;
+	char **commit_id;
 	int arg = 1;
 
 	setup_git_directory();

@@ -230,20 +232,30 @@ int main(int argc, char **argv)
 		else if (argv[arg][1] == 'v')
 			get_verbosely = 1;
 		else if (argv[arg][1] == 'w')
-			write_ref = argv[++arg];
+			write_ref = &argv[++arg];
 		else if (!strcmp(argv[arg], "--recover"))
 			get_recover = 1;
+		else if (!strcmp(argv[arg], "--stdin"))
+			commits_on_stdin = 1;
 		else
 			usage(local_pull_usage);
 		arg++;
 	}
-	if (argc < arg + 2)
+	if (argc < arg + 2 - commits_on_stdin)
 		usage(local_pull_usage);
-	commit_id = argv[arg];
-	path = argv[arg + 1];
+	if (commits_on_stdin) {
+		commits = pull_targets_stdin(&commit_id, &write_ref);
+	} else {
+		commit_id = (char **) &argv[arg++];
+		commits = 1;
+	}
+	path = argv[arg];
 
-	if (pull(1, &commit_id, &write_ref, path))
+	if (pull(commits, commit_id, write_ref, path))
 		return 1;
 
+	if (commits_on_stdin)
+		pull_targets_free(commits, commit_id, write_ref);
+
 	return 0;
 }
author
#4 lore raw

PATCH 4/4 Teach git-http-fetch the --stdin switch

In reply to Petr Baudis

Speeds up things quite a lot when fetching tags with Cogito.

Signed-off-by Petr Baudis <[email protected]>

---

 Documentation/git-http-fetch.txt |    8 ++++++-
 http-fetch.c                     |   45 ++++++++++++++++++++++++--------------
 2 files changed, 36 insertions(+), 17 deletions(-)
Documentation/git-http-fetch.txt +7 -1
index bc1a132..bea522e 100644
@@ -8,7 +8,7 @@ git-http-fetch - downloads a remote git 
 
 SYNOPSIS
 --------
-'git-http-fetch' [-c] [-t] [-a] [-d] [-v] [-w filename] [--recover] <commit> <url>
+'git-http-fetch' [-c] [-t] [-a] [-d] [-v] [-w filename] [--recover] [--stdin] <commit> <url>
 
 DESCRIPTION
 -----------

@@ -32,6 +32,12 @@ commit-id::
 -w <filename>::
         Writes the commit-id into the filename under $GIT_DIR/refs/<filename> on
         the local end after the transfer is complete.
+ 
+--stdin::
+	Instead of a commit id on the commandline (which is not expected in this
+	case), 'git-http-fetch' excepts lines on stdin in the format
+
+		<commit-id>['\t'<filename-as-in--w>]
 
 Author
 ------
http-fetch.c +29 -16

120 lines not on this page · raw message

author
#5 lore raw

Re: [PATCH 3/4] Teach git-local-fetch the --stdin switch

Dear diary, on Thu, Jul 27, 2006 at 11:56:19PM CEST, I got a letter where Petr Baudis <[email protected]> said that...

Review

Quoted 3 lines

+--stdin::

(quoted diff) +2 -0
+	Instead of a commit id on the commandline (which is not expected in this
+	case), 'git-local-fetch' excepts lines on stdin in the format

                                 ^^^^^^^

Oops, should read "expects" in both patches - thanks, alp!

I blame all those tiny bugs lured into the room by light and then, well, bugging you. (Real-world bugs, but they can transform.)

Signature · 4 lines
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam
author
#6 lore raw
In reply to Petr Baudis

Dear diary, on Thu, Jul 27, 2006 at 11:53:26PM CEST, I got a letter where Petr Baudis <[email protected]> said that...

Quoted 2 lines

I will followup with a patch for Cogito to take advantage of this. It's
now roughly on par with Git in cloning speed.

cg-Xfetchprogress +1 -1
index 53bcbd3..4a272db 100755
@@ -67,7 +67,7 @@ sub getline {
 	if (m#^(link|symlink|copy|got) ([a-f0-9]{2})([a-f0-9]{38})#) {
 		$object = "$2/$3";
 
-	} elsif (m#^(walk) ([a-f0-9]{2})([a-f0-9]{38})#) {
+	} elsif (m#^(ref|walk) ([a-f0-9]{2})([a-f0-9]{38})#) {
 		return 1; # redundant information
 
 	# rsync
cg-fetch +31 -34

101 lines not on this page · raw message

Signature · 4 lines
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam