Allow fetching from multiple repositories at once

This patch enables fetching multiple repositories at once over the Git protocol (and SSH, and locally if git-fetch-pack is your cup of coffee there). This is done especially for the xorg people who have tons of repositories and dislike pulls much slower than they were used to with CVS. I'm eager to hear how this affects the situation.

It's kind of "superproject" thing, basically, taking reverse approach than the subproject ideas. However, in practice I think it can be used for subprojects quite well and perhaps if I find some spare time during the day I will add the lightweight subproject support to Cogito, using this.

So, you need some kind of porcelain for this. The idea is that instead of telling git-fetch-pack a single repository, you pass multiple --repo= parameters and refs always "belong" to the latest mentioned repository; when outputting the new ref values, the appropriate repo is mentioned near each ref. In order for this to be useful, on your local side you should share the objects database in some way - either using alternates (then you must fetch to an object database reachable from everywhere) or symlinked object databases.

You still need to pass git-fetch-pack some URL in addition to the repositories - it is used only for git_connect(), the purpose is that repositories must be local directories so if you want to talk remote, you need to do something like

	git-fetch-pack git://kernel.org/pub/scm/git/git.git --repo=/pub/scm/git/git.git master next --repo=/pub/scm/cogito/cogito.git master

The implementation is, well... rather hackish to say the least. :-) We can do something nicer if it turns out to be a boost huge enough in reality too.

---

 cache.h       |    7 ++++--
 connect.c     |   11 ++++++---
 daemon.c      |    9 +++++---
 fetch-pack.c  |   58 +++++++++++++++++++++++++++++++++----------------
 path.c        |    5 ++++
 peek-remote.c |    4 ++-
 send-pack.c   |    4 ++-
 sha1_file.c   |    6 +++--
 upload-pack.c |   68 +++++++++++++++++++++++++++++++++++++++++++++++++--------
 9 files changed, 127 insertions(+), 45 deletions(-)

index 457d1d0..ea2f5e8 100644
@@ -225,6 +225,7 @@ int git_config_perm(const char *var, con
 int adjust_shared_perm(const char *path);
 int safe_create_leading_directories(char *path);
 char *enter_repo(char *path, int strict);
+int security_repo_check(int export_all_trees);
 
 /* Read and unpack a sha1 file into memory, write memory to a sha1 file */
 extern int sha1_object_info(const unsigned char *, char *, unsigned long *);

@@ -298,6 +299,7 @@ extern struct alternate_object_database 
 	char base[FLEX_ARRAY]; /* more */
 } *alt_odb_list;
 extern void prepare_alt_odb(void);
+extern int link_alt_odb_entry(const char * entry, int len, const char * relative_base, int depth, int allow_self);
 
 extern struct packed_git {
 	struct packed_git *next;

@@ -325,6 +327,7 @@ struct ref {
 	unsigned char new_sha1[20];
 	unsigned char force;
 	struct ref *peer_ref; /* when renaming */
+	char *repo; /* when doing multi-repo operations */
 	char name[FLEX_ARRAY]; /* more */
 };
 

@@ -334,11 +337,11 @@ #define REF_TAGS	(1u << 2)
 
 extern int git_connect(int fd[2], char *url, const char *prog);
 extern int finish_connect(pid_t pid);
-extern int path_match(const char *path, int nr, char **match);
+extern int path_match(const char *path, const char *repo, int nr, char **match, char **match_repo);
 extern int match_refs(struct ref *src, struct ref *dst, struct ref ***dst_tail,
 		      int nr_refspec, char **refspec, int all);
 extern int get_ack(int fd, unsigned char *result_sha1);
-extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match, unsigned int flags);
+extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match, char *repo, unsigned int flags);
 extern int server_supports(const char *feature);
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1);

index 4422a0d..63f095c 100644
@@ -45,9 +45,8 @@ static int check_ref(const char *name, i
  */
 struct ref **get_remote_heads(int in, struct ref **list,
 			      int nr_match, char **match,
-			      unsigned int flags)
+			      char *repo, unsigned int flags)
 {
-	*list = NULL;
 	for (;;) {
 		struct ref *ref;
 		unsigned char old_sha1[20];

@@ -74,11 +73,12 @@ struct ref **get_remote_heads(int in, st
 
 		if (!check_ref(name, name_len, flags))
 			continue;
-		if (nr_match && !path_match(name, nr_match, match))
+		if (nr_match && !path_match(name, NULL, nr_match, match, NULL))
 			continue;
 		ref = xcalloc(1, sizeof(*ref) + len - 40);
 		memcpy(ref->old_sha1, old_sha1, 20);
 		memcpy(ref->name, buffer + 41, len - 40);
+		ref->repo = repo;
 		*list = ref;
 		list = &ref->next;
 	}

@@ -112,7 +112,8 @@ int get_ack(int fd, unsigned char *resul
 	die("git-fetch_pack: expected ACK/NAK, got '%s'", line);
 }
 
-int path_match(const char *path, int nr, char **match)
+int path_match(const char *path, const char *repo, int nr, char **match,
+               char **match_repo)
 {
 	int i;
 	int pathlen = strlen(path);

@@ -121,6 +122,8 @@ int path_match(const char *path, int nr,
 		char *s = match[i];
 		int len = strlen(s);
 
+		if (match_repo && repo != match_repo[i])
+			continue;
 		if (!len || len > pathlen)
 			continue;
 		if (memcmp(path + pathlen - len, s, len))

index e4ec676..179f2fe 100644
@@ -233,6 +233,7 @@ static int upload(char *dir)
 {
 	/* Timeout as string */
 	char timeout_buf[64];
+	char checkexport_buf[64];
 	const char *path;
 
 	loginfo("Request for '%s'", dir);

@@ -250,8 +251,7 @@ static int upload(char *dir)
 	 * path_ok() uses enter_repo() and does whitelist checking.
 	 * We only need to make sure the repository is exported.
 	 */
-
-	if (!export_all_trees && access("git-daemon-export-ok", F_OK)) {
+	if (!security_repo_check(export_all_trees)) {
 		logerror("'%s': repository not exported.", path);
 		errno = EACCES;
 		return -1;

@@ -265,8 +265,11 @@ static int upload(char *dir)
 
 	snprintf(timeout_buf, sizeof timeout_buf, "--timeout=%u", timeout);
 
+	strcpy(checkexport_buf, "--check-export");
+	if (export_all_trees) checkexport_buf[1] = '\0'; /* XXX */
+
 	/* git-upload-pack only ever reads stuff, so this is safe */
-	execl_git_cmd("upload-pack", "--strict", timeout_buf, ".", NULL);
+	execl_git_cmd("upload-pack", "--strict", timeout_buf, checkexport_buf, ".", NULL);
 	return -1;
 }
 

index db8905f..3021164 100644
@@ -243,6 +243,11 @@ char *enter_repo(char *path, int strict)
 	return NULL;
 }
 
+int security_repo_check(int export_all_trees)
+{
+	return (export_all_trees || access("git-daemon-export-ok", F_OK));
+}
+
 int adjust_shared_perm(const char *path)
 {
 	struct stat st;

index 2b30980..7d8ae3c 100644
@@ -9,9 +9,9 @@ static const char *exec = "git-upload-pa
 
 static int peek_remote(int fd[2], unsigned flags)
 {
-	struct ref *ref;
+	struct ref *ref = NULL;
 
-	get_remote_heads(fd[0], &ref, 0, NULL, flags);
+	get_remote_heads(fd[0], &ref, 0, NULL, NULL, flags);
 	packet_flush(fd[1]);
 
 	while (ref) {

index 10bc8bc..24c47c9 100644
@@ -232,14 +232,14 @@ static int receive_status(int in)
 
 static int send_pack(int in, int out, int nr_refspec, char **refspec)
 {
-	struct ref *ref;
+	struct ref *ref = NULL;
 	int new_refs;
 	int ret = 0;
 	int ask_for_status_report = 0;
 	int expect_status_report = 0;
 
 	/* No funny business with the matcher */
-	remote_tail = get_remote_heads(in, &remote_refs, 0, NULL, REF_NORMAL);
+	remote_tail = get_remote_heads(in, &remote_refs, 0, NULL, NULL, REF_NORMAL);
 	get_local_heads();
 
 	/* Does the other end support the reporting? */

index 8f279d8..1ca39e1 100644
@@ -229,7 +229,7 @@ static void read_info_alternates(const c
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static int link_alt_odb_entry(const char * entry, int len, const char * relative_base, int depth)
+int link_alt_odb_entry(const char * entry, int len, const char * relative_base, int depth, int allow_self)
 {
 	struct stat st;
 	const char *objdir = get_object_directory();

@@ -279,7 +279,7 @@ static int link_alt_odb_entry(const char
 			return -1;
 		}
 	}
-	if (!memcmp(ent->base, objdir, pfxlen)) {
+	if (!allow_self && !memcmp(ent->base, objdir, pfxlen)) {
 		free(ent);
 		return -1;
 	}

@@ -325,7 +325,7 @@ static void link_alt_odb_entries(const c
 						relative_base, last);
 			} else {
 				link_alt_odb_entry(last, cp - last,
-						relative_base, depth);
+						relative_base, depth, 0);
 			}
 		}
 		while (cp < ep && *cp == sep)

Dear diary, on Fri, Jul 28, 2006 at 07:44:21AM CEST, I got a letter where Petr Baudis <[email protected]> said that...

This is a simple "porcelain" for it:

-->8--

#!/usr/bin/perl
use warnings;
use strict;

# Remember that you must ensure the obj database is shared - either symlink it

# or setup alternates!
#my $remoteurl = 'git://git.kernel.org/pub/scm/git/git.git';
#my %config = (
#	'/pub/scm/git/git.git' => {
#		'next' => {
#			'/home/xpasky/q/gg' => 'origin'
#		},
#		'master' => {
#			'/home/xpasky/q/gg' => 'origin2',
#			'/home/xpasky/q/gg2' => 'origin'
#		}
#	},
#	'/pub/scm/cogito/cogito-doc.git' => {
#		master => {
#			'/home/xpasky/q/gg2' => 'origin2'
#		}
#	}
#);
my $remoteurl = 'puturlofremotehosthere';
my %config = (
	'remoterepo1' => {
		branch1 => {
			'localrepo1' => 'origin'
		},
		branch2 => {
			'localrepo1' => 'origin2',
			'localrepo2' => 'origin'
		}
	},
	'remoterepo2' => {
		master => {
			'localrepo2' => 'origin2'
		}
	}
);

my @args = ($remoteurl);

foreach my $repo (keys %config) {
	push (@args, '--repo='.$repo);
	foreach my $branch (keys %{$config{$repo}}) {
		push (@args, $branch);
	}
}

open (F, '-|', 'git-fetch-pack', @args) or die "$!";

while (<F>) {
	chomp;
	split / /, $_;
	my ($sha, $ref, $repo) = @_;
	$ref =~ s#^refs/heads/##;
	foreach my $lrepo (keys %{$config{$repo}->{$ref}}) {
		system("GIT_DIR=$lrepo git-update-ref $config{$repo}->{$ref}->{$lrepo} $sha");
	}
}

close (F);

				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam

Dear diary, on Fri, Jul 28, 2006 at 07:44:21AM CEST, I got a letter where Petr Baudis <[email protected]> said that...

				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam

This is quite heavyweight.

I'll take a look at it eventually ;-), but I'd like to take a deeper look at merge-recursive by Johannes and Alex first. Perhaps I can get to it over the weekend, but Saturday is already booked for friends' wedding, so it might take a while.

Hi,

So the scenario is: one remote repository (probably shared), and multiple local repositories, all tracking different branches?

So, why not setup a single local (master) repository, setup all the other repos with the local master as alternate, and write a simple script which first fetches all branches into the master, and then pulls into the other local repos from that master?

The beauty of it is: you can still pull/push directly from the remote repo, if you want.

Ciao,
Dscho

Hi,

Dear diary, on Fri, Jul 28, 2006 at 10:51:56AM CEST, I got a letter where Johannes Schindelin <[email protected]> said that...

Nope, the scenario is many remote repositories and any number of local
repositories. Look at http://gitweb.freedesktop.org/ - as far as I know, many people have many (most?) of those repositories cloned and fetching updates is just a huge pain. This makes no assumptions about the number of local repositories, you can even fetch into various branch of a single local repository - only if you fetch into multiple repositories, all of them must see the objects you fetched.

The alternative of squashing all the remote repositories into a single
one is probably not very attractive since you get a huge branches tree instead, varying hooks will get impractical, it won't look good in gitweb, git clone will by default clone all the stuff, you will need to impose tag namespaces, permissions might get tricky and so on and so on.

This should let you do that too.

				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam

Hi,

Ah, that clarifies it for me.

You misunderstood me: I was talking about a (hidden) local multiplexer.

But in the scenario you painted, wouldn't it be practical to have one consolidated repo _in addition_ to the small ones? Of course, the refs in the consolidated one would have to be prefixed with the name of the repo they come from.

Of course. But the solution I proposed does not need a patch, and _still_ lets you do said operations. Just wanted to point that out.

Ciao,
Dscho

Hi!

Dear diary, on Fri, Jul 28, 2006 at 04:13:28PM CEST, I got a letter where Johannes Schindelin <[email protected]> said that...

Yes; I just wanted to cover another obvious alternative. :-)

You still have problems with tags, for example, and pushing may get tricky. It's also non-intuitive for users (I'm seeing some repositories in gitweb but fetching from something completely different, and look at those weird things I have to do with the branches, or git will clone _all_ the xorg projects at once for me, eeek).

Of course you could hide all that elaborately in the porcelain but I think this turns to be eventually much less-impact solution which is significantly easier to maintain for the repository admins, intuitive for users and also simpler for porcelains supporting at least light-weight subprojects.

				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam