Skip to contentSkip to conversation

New type of remote helpers

RFC In review Tomas Carnecky 4 participants 21 messages Jun 15, 2016 Kbuild

lore
Share & tools
lore
#0 lore raw

My work has the goal of making interaction with foreign SCMs more natural. The work that was done on remote helpers is the right direction. But the 'import' and 'export' commands are the wrong approach I think. The problem I have with 'import' is that updating the refs is left up to the remote helper (or git-fast-import). So you lose the nice output from ls-remote/fetch: non-ff and other warnings etc. I slightly modified how the remote helpers (and fast-import) work, now they behave exactly like 'core' git when fetching: Git tells the remote helper to fetch some refs, the helper does that and creates a pack and git then updates the refs (or not, depending on fast-forward etc). To test this approach I created a simple remote helper for svn.

The work is available in the 'remote-helper-fixups' branch at
http://github.com/wereHamster/git. No end-user or technical documentation is available as of yet. The svn remote helper is only an example how remote helpers can use the new commands. You can see it as a 'technology preview', use with caution.

Here is an example session of how I think foreign SCMs should integrate with git. For each command I explain what changes were needed. This is not just some theoretical output. This is a real, working example. I did not modify any of the output text.

$ git ls-remote svn::/Volumes/Dump/Source/Mirror/Transmission/
r1017 (impure)                            trunk
r919 (impure)                             branches/nat-traversal
r480 (impure)                             branches/0.6

Git learned to understand version numbers from foreign SCMs. Git displays those as 'impure' because it knows that version exists but does not know yet which git commit that version maps to.

$ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/
*:refs/remotes/svn/*
From svn::/Volumes/Dump/Source/Mirror/Transmission
 * [new branch]      trunk      -> svn/trunk
 * [new branch]      branches/nat-traversal -> svn/branches/nat-traversal
 * [new branch]      branches/0.6 -> svn/branches/0.6

Git tells the remote helper to 'fetch r1017 trunk'. The remote helper does that, creates the pack and then tells git that it imported r1017 as commit c5fed7ec. This is done with a new reply to the 'fetch' command:
'map r1017 c5fed7ec'. The remote helper can use that to inform core git as which git commit the impure ref was imported. Git can then update the refs. At no point does the remote helper manipulate refs directly.

The pack is created by a heavily modified git-fast-import. The existing fast-import not only creates the pack but also updates the refs. This is no longer desired as git is in charge of updating the refs. My modified fast-import works like this: After creating a commit, it writes it's git object name to stdout. That way the remote helper can figure out as which git commits the svn revisions were imported and relay that back to core git using the above described 'map' reply.

$ git show --show-notes=svn svn/trunk
commit c5fed7ecc318363523d3ea2045e1c16a378bb10c
Author: livings124 <livings124@localhost>
Date:   Wed Oct 18 13:57:19 2006 +0000

more traditional toolbar icons for those afraid of change

Notes (svn):

    b4697c4a-7d4c-4a30-bd92-6745580d73b3/trunk@1017

The svn helper needs to be able to map svn revisions to git commits. git-svn does this by adding the 'git-svn-id' line to each commit message. I'm using git notes for that and it seems to work just fine. The note contains the repo UUID, path within the repo and revision.

There was a challenge how to update the notes ref (refs/notes/svn). As with fast-import, I did not want the remote helper to do it. Neither the remote helper nor fast-import should be writing any refs. But core git can only update refs which were discovered during transport->fetch(). I modified the remote helper 'fetch' command and the transport->fetch() function to return an optional list of refs. These are the refs that the remote helper wants to update but which should not be presented to the user (because these are internally used refs, such as my svn notes).

So the whole session between git and my svn remote helper looks like this:

Quoted 1 line

list

< :r1017 trunk

Quoted 1 line

fetch :r1017 trunk

[helper creates the pack including history up to r1017 and associated svn notes]
< map r1017 <commit corresponding to r1017>
< silent refs/notes/svn <new commit which stores the updated svn notes>

$ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/
*:refs/remotes/svn/*
From svn::/Volumes/Dump/Source/Mirror/Transmission
   c5fed7e..228eaf3  trunk      -> svn/trunk
$ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/
*:refs/remotes/svn/*
From svn::/Volumes/Dump/Source/Mirror/Transmission
   228eaf3..207e5e5  trunk      -> svn/trunk
 * [new branch]      branches/scrape -> svn/branches/scrape
 * [new branch]      branches/multitracker -> svn/branches/multitracker
 * [new branch]      branches/io -> svn/branches/io

Updating the svn branches works like expected. The remote helper automatically detects which branches it already imported (by going through all refs and the attached svn notes) and creates a new pack with the new commits. New branches are also detected. The svn notes are updated accordingly.

tom

#1 lore raw

PATCH 1/6 Remote helper: accept ':<value> <name>' as a response to 'list'

Parse <value> as the remote-specific revision indicator. The ref is first stored as 'impure', meaning that it doesn't have any representation within git. Only after the remote helper fetches that version into git, it can tell us which git object (SHA1) that revision maps to. That is done with the new reply 'map <value> <sha1>' to the 'list' command.

---
 builtin/ls-remote.c |    7 ++++++-
 cache.h             |    2 +-
 remote.c            |    8 +++++++-
 transport-helper.c  |   41 +++++++++++++++++++++++++++++++++++++++--
 4 files changed, 53 insertions(+), 5 deletions(-)
builtin/ls-remote.c +6 -1
index 97eed40..23dd0d2 100644
@@ -109,7 +109,12 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 			continue;
 		if (!tail_match(pattern, ref->name))
 			continue;
-		printf("%s	%s\n", sha1_to_hex(ref->old_sha1), ref->name);
+		if (ref->impure) {
+			int len = strlen(ref->impure) + strlen(" (impure)");
+			printf("%s (impure)%*s  %s\n", ref->impure, 40 - len, " ", ref->name);
+		} else {
+			printf("%s      %s\n", sha1_to_hex(ref->old_sha1), ref->name);
+		}
 	}
 	return 0;
 }
cache.h +1 -1
index 2ef2fa3..23b43a6 100644
@@ -905,7 +905,7 @@ struct ref {
 	struct ref *next;
 	unsigned char old_sha1[20];
 	unsigned char new_sha1[20];
-	char *symref;
+	char *symref, *impure;
 	unsigned int force:1,
 		merge:1,
 		nonfastforward:1,
remote.c +7 -1
index 9143ec7..c355f11 100644
@@ -907,6 +907,7 @@ static struct ref *copy_ref(const struct ref *ref)
 	memcpy(cpy, ref, sizeof(struct ref) + len + 1);
 	cpy->next = NULL;
 	cpy->symref = ref->symref ? xstrdup(ref->symref) : NULL;
+	cpy->impure = ref->impure ? xstrdup(ref->impure) : NULL;
 	cpy->remote_status = ref->remote_status ? xstrdup(ref->remote_status) : NULL;
 	cpy->peer_ref = copy_ref(ref->peer_ref);
 	return cpy;
@@ -931,6 +932,7 @@ static void free_ref(struct ref *ref)
 	free_ref(ref->peer_ref);
 	free(ref->remote_status);
 	free(ref->symref);
+	free(ref->impure);
 	free(ref);
 }
 
@@ -1453,7 +1455,11 @@ int resolve_remote_symref(struct ref *ref, struct ref *list)
 		return 0;
 	for (; list; list = list->next)
 		if (!strcmp(ref->symref, list->name)) {
-			hashcpy(ref->old_sha1, list->old_sha1);
+			if (list->impure) {
+				ref->impure = xstrdup(list->impure);
+			} else {
+				hashcpy(ref->old_sha1, list->old_sha1);
+			}
 			return 0;
 		}
 	return 1;
transport-helper.c +39 -2

75 lines not on this page · raw message

Signature · 1 line
1.7.3.37.gb6088b
#4 lore raw

PATCH 2/6 Allow more than one keepfile in the transport

In reply to Tomas Carnecky

Git itself creates only one packfile per fetch. But other transports may chose to create more than one (those that use fast-import for example). Use an array to keep track of the pack lockfiles.

---
 transport-helper.c |    5 +----
 transport.c        |   27 ++++++++++++++++++++++-----
 transport.h        |   11 ++++++++++-
 3 files changed, 33 insertions(+), 10 deletions(-)
transport-helper.c +1 -4
index 0fe886e..dcaaa89 100644
@@ -362,10 +362,7 @@ static int fetch_with_fetch(struct transport *transport,
 
 		if (!prefixcmp(buf.buf, "lock ")) {
 			const char *name = buf.buf + 5;
-			if (transport->pack_lockfile)
-				warning("%s also locked %s", data->name, name);
-			else
-				transport->pack_lockfile = xstrdup(name);
+			transport_keep(transport, name);
 		} else if (!prefixcmp(buf.buf, "map ")) {
 			map_impure_ref(nr_heads, to_fetch, buf.buf + 4);
 		}
transport.c +22 -5
index 4dba6f8..df2baa7 100644
@@ -518,6 +518,7 @@ static int fetch_refs_via_pack(struct transport *transport,
 	struct fetch_pack_args args;
 	int i;
 	struct ref *refs_tmp = NULL;
+	char *keepfile = NULL;
 
 	memset(&args, 0, sizeof(args));
 	args.uploadpack = data->options.uploadpack;
@@ -541,7 +542,7 @@ static int fetch_refs_via_pack(struct transport *transport,
 
 	refs = fetch_pack(&args, data->fd, data->conn,
 			  refs_tmp ? refs_tmp : transport->remote_refs,
-			  dest, nr_heads, heads, &transport->pack_lockfile);
+			  dest, nr_heads, heads, &keepfile);
 	close(data->fd[0]);
 	close(data->fd[1]);
 	if (finish_connect(data->conn))
@@ -556,6 +557,10 @@ static int fetch_refs_via_pack(struct transport *transport,
 	free(origh);
 	free(heads);
 	free(dest);
+
+	if (keepfile)
+		transport_keep(transport, keepfile);
+
 	return (refs ? 0 : -1);
 }
 
@@ -1116,11 +1121,15 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)
 
 void transport_unlock_pack(struct transport *transport)
 {
-	if (transport->pack_lockfile) {
-		unlink_or_warn(transport->pack_lockfile);
-		free(transport->pack_lockfile);
-		transport->pack_lockfile = NULL;
+	int i;
+
+	for (i = 0; i < transport->keep_nr; ++i) {
+		char *keepfile = (char *) transport->keep[i];
+		unlink_or_warn(keepfile);
+		free(keepfile);
 	}
+
+	transport->keep_nr = 0;
 }
 
 int transport_connect(struct transport *transport, const char *name,
@@ -1137,6 +1146,7 @@ int transport_disconnect(struct transport *transport)
 	int ret = 0;
 	if (transport->disconnect)
 		ret = transport->disconnect(transport);
+	free(transport->keep);
 	free(transport);
 	return ret;
 }
@@ -1188,3 +1198,10 @@ char *transport_anonymize_url(const char *url)
 literal_copy:
 	return xstrdup(url);
 }
+
+void transport_keep(struct transport *transport, const char *keepfile)
+{
+	int nr = transport->keep_nr + 1;
+	ALLOC_GROW(transport->keep, nr, transport->keep_alloc);
+	transport->keep[transport->keep_nr++] = keepfile;
+}
transport.h +10 -1
index c59d973..6320d28 100644
@@ -78,7 +78,14 @@ struct transport {
 	 * use. disconnect() releases these resources.
 	 **/
 	int (*disconnect)(struct transport *connection);
-	char *pack_lockfile;
+
+	/** The transport can create zero or more pack files which need to be
+	 * kept until we can update the refs. This array holds the names of the
+	 * keep files which we have to delete once the refs are updated.
+	 **/
+	const char **keep;
+	int keep_nr, keep_alloc;
+
 	signed verbose : 3;
 	/**
 	 * Transports should not set this directly, and should use this
@@ -165,4 +172,6 @@ int transport_refs_pushed(struct ref *ref);
 void transport_print_push_status(const char *dest, struct ref *refs,
 		  int verbose, int porcelain, int *nonfastforward);
 
+void transport_keep(struct transport *transport, const char *keepfile);
+
 #endif
Signature · 1 line
1.7.3.37.gb6088b
#6 lore raw

PATCH 3/6 Allow the transport fetch command to add additional refs

In reply to Tomas Carnecky

The fetch transport command (in particular in remote helpers) may need to create or update additional refs which are used internally by the helper, but which it doesn't want to present to the user. Those refs are refered to as 'silent' throughout the code because git should be silent about their presence, but yet process those just like the other refs.

Example use case:
Remote helpers such as those for svn may chose to save the Git SHA1 -> Subversion revision mapping as notes attached to the commits (as opposed to strings in the commit message itself). The helper would need to update the notes on each fetch, but the user should not be bothered by the presence of that ref. The remote helper can update the notes tree through fast-import and then inform Git core that it should silently update the notes ref.

---
 builtin/clone.c    |   11 +++++++----
 builtin/fetch.c    |   15 ++++++++++++---
 transport-helper.c |   32 ++++++++++++++++++++++++++++----
 transport.c        |   16 ++++++++++------
 transport.h        |    6 ++++--
 5 files changed, 61 insertions(+), 19 deletions(-)
builtin/clone.c +7 -4
index 19ed640..78355b6 100644
@@ -348,13 +348,16 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	return local_refs;
 }
 
-static void write_remote_refs(const struct ref *local_refs)
+static void write_remote_refs(const struct ref *local_refs, struct ref *silent)
 {
 	const struct ref *r;
 
 	for (r = local_refs; r; r = r->next)
 		add_extra_ref(r->peer_ref->name, r->old_sha1, 0);
 
+	for (r = silent; r; r = r->next)
+		add_extra_ref(r->name, r->old_sha1, 0);
+
 	pack_refs(PACK_REFS_ALL);
 	clear_extra_refs();
 }
@@ -369,7 +372,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	const struct ref *refs, *remote_head;
 	const struct ref *remote_head_points_at;
 	const struct ref *our_head_points_at;
-	struct ref *mapped_refs;
+	struct ref *mapped_refs, *silent = NULL;
 	struct strbuf key = STRBUF_INIT, value = STRBUF_INIT;
 	struct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;
 	struct transport *transport = NULL;
@@ -542,14 +545,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		refs = transport_get_remote_refs(transport);
 		if (refs) {
 			mapped_refs = wanted_peer_refs(refs, refspec);
-			transport_fetch_refs(transport, mapped_refs);
+			transport_fetch_refs(transport, mapped_refs, &silent);
 		}
 	}
 
 	if (refs) {
 		clear_extra_refs();
 
-		write_remote_refs(mapped_refs);
+		write_remote_refs(mapped_refs, silent);
 
 		remote_head = find_ref_by_name(refs, "HEAD");
 		remote_head_points_at =
builtin/fetch.c +12 -3
index 6fc5047..71db090 100644
@@ -313,7 +313,7 @@ static int update_local_ref(struct ref *ref,
 }
 
 static int store_updated_refs(const char *raw_url, const char *remote_name,
-		struct ref *ref_map)
+		struct ref *ref_map, struct ref *silent)
 {
 	FILE *fp;
 	struct commit *commit;
@@ -411,6 +411,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 				fprintf(stderr, " %s\n", note);
 		}
 	}
+
+	/* Also update the silent refs. */
+	for (rm = silent; rm; rm = rm->next) {
+		snprintf(note, 1024, "note here"); /* TODO */
+		update_local_ref(rm, rm->name, note);
+	}
+
 	free(url);
 	fclose(fp);
 	if (rc & STORE_REF_ERROR_DF_CONFLICT)
@@ -496,13 +503,15 @@ static int quickfetch(struct ref *ref_map)
 
 static int fetch_refs(struct transport *transport, struct ref *ref_map)
 {
+	struct ref *silent = NULL;
+
 	int ret = quickfetch(ref_map);
 	if (ret)
-		ret = transport_fetch_refs(transport, ref_map);
+		ret = transport_fetch_refs(transport, ref_map, &silent);
 	if (!ret)
 		ret |= store_updated_refs(transport->url,
 				transport->remote->name,
-				ref_map);
+				ref_map, silent);
 	transport_unlock_pack(transport);
 	return ret;
 }
transport-helper.c +28 -4
index dcaaa89..c0133ca 100644
@@ -330,8 +330,29 @@ static void map_impure_ref(int nr_heads, struct ref **to_fetch, char *map)
 	}
 }
 
+/* `buf` points to the 'silent' response from the helper. Parse it and
+ * add the ref to the `silent` list. */
+static void add_silent_ref(struct strbuf *buf, struct ref **silent)
+{
+	if (!silent)
+		return;
+
+	struct ref *ref;
+	char *eon = strchr(buf->buf + 7, ' ');
+	if (!eon)
+		warning("Malformed helper response: %s", buf->buf);
+
+	*eon = '\0';
+	ref = alloc_ref(buf->buf + 7);
+	get_sha1_hex(eon + 1, ref->new_sha1);
+	
+	ref->next = *silent;
+	*silent = ref;
+}
+
 static int fetch_with_fetch(struct transport *transport,
-			    int nr_heads, struct ref **to_fetch)
+			    int nr_heads, struct ref **to_fetch,
+			    struct ref **silent)
 {
 	struct helper_data *data = transport->data;
 	int i;
@@ -365,6 +386,8 @@ static int fetch_with_fetch(struct transport *transport,
 			transport_keep(transport, name);
 		} else if (!prefixcmp(buf.buf, "map ")) {
 			map_impure_ref(nr_heads, to_fetch, buf.buf + 4);
+		} else if (!prefixcmp(buf.buf, "silent ")) {
+			add_silent_ref(&buf, silent);
 		}
 		else if (!buf.len)
 			break;
@@ -559,14 +582,15 @@ static int connect_helper(struct transport *transport, const char *name,
 }
 
 static int fetch(struct transport *transport,
-		 int nr_heads, struct ref **to_fetch)
+		 int nr_heads, struct ref **to_fetch,
+		 struct ref **silent)
 {
 	struct helper_data *data = transport->data;
 	int i, count;
 
 	if (process_connect(transport, 0)) {
 		do_take_over(transport);
-		return transport->fetch(transport, nr_heads, to_fetch);
+		return transport->fetch(transport, nr_heads, to_fetch, silent);
 	}
 
 	count = 0;
@@ -578,7 +602,7 @@ static int fetch(struct transport *transport,
 		return 0;
 
 	if (data->fetch)
-		return fetch_with_fetch(transport, nr_heads, to_fetch);
+		return fetch_with_fetch(transport, nr_heads, to_fetch, silent);
 
 	if (data->import)
 		return fetch_with_import(transport, nr_heads, to_fetch);
transport.c +10 -6
index df2baa7..eaab276 100644
@@ -253,7 +253,8 @@ static struct ref *get_refs_via_rsync(struct transport *transport, int for_push)
 }
 
 static int fetch_objs_via_rsync(struct transport *transport,
-				int nr_objs, struct ref **to_fetch)
+				int nr_objs, struct ref **to_fetch,
+				struct ref **silent)
 {
 	struct strbuf buf = STRBUF_INIT;
 	struct child_process rsync;
@@ -428,7 +429,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, int for_pus
 }
 
 static int fetch_refs_from_bundle(struct transport *transport,
-			       int nr_heads, struct ref **to_fetch)
+			       int nr_heads, struct ref **to_fetch,
+			       struct ref **silent)
 {
 	struct bundle_transport_data *data = transport->data;
 	return unbundle(&data->header, data->fd);
@@ -508,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 }
 
 static int fetch_refs_via_pack(struct transport *transport,
-			       int nr_heads, struct ref **to_fetch)
+			       int nr_heads, struct ref **to_fetch,
+			       struct ref **silent)
 {
 	struct git_transport_data *data = transport->data;
 	char **heads = xmalloc(nr_heads * sizeof(*heads));
@@ -1083,7 +1086,8 @@ const struct ref *transport_get_remote_refs(struct transport *transport)
 	return transport->remote_refs;
 }
 
-int transport_fetch_refs(struct transport *transport, struct ref *refs)
+int transport_fetch_refs(struct transport *transport, struct ref *refs,
+				struct ref **silent)
 {
 	int rc;
 	int nr_heads = 0, nr_alloc = 0, nr_refs = 0;
@@ -1113,9 +1117,9 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)
 			heads[nr_heads++] = rm;
 	}
 
-	rc = transport->fetch(transport, nr_heads, heads);
-
+	rc = transport->fetch(transport, nr_heads, heads, silent);
 	free(heads);
+
 	return rc;
 }
 
transport.h +4 -2
index 6320d28..22daf60 100644
@@ -52,7 +52,7 @@ struct transport {
 	 * get_refs_list(), it should set the old_sha1 fields in the
 	 * provided refs now.
 	 **/
-	int (*fetch)(struct transport *transport, int refs_nr, struct ref **refs);
+	int (*fetch)(struct transport *transport, int refs_nr, struct ref **refs, struct ref **silent);
 
 	/**
 	 * Push the objects and refs. Send the necessary objects, and
@@ -149,7 +149,9 @@ int transport_push(struct transport *connection,
 
 const struct ref *transport_get_remote_refs(struct transport *transport);
 
-int transport_fetch_refs(struct transport *transport, struct ref *refs);
+int transport_fetch_refs(struct transport *transport, struct ref *refs,
+			struct ref **silent);
+
 void transport_unlock_pack(struct transport *transport);
 int transport_disconnect(struct transport *transport);
 char *transport_anonymize_url(const char *url);
Signature · 1 line
1.7.3.37.gb6088b
#8 lore raw

PATCH 4/6 Rename get_mode() to decode_tree_mode() and export it

In reply to Tomas Carnecky

Other sources (fast-import-helper.c) may want to use this function to parse trees.

---
 tree-walk.c |    4 ++--
 tree-walk.h |    2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
tree-walk.c +2 -2
index a9bbf4e..5f51f4e 100644
@@ -3,7 +3,7 @@
 #include "unpack-trees.h"
 #include "tree.h"
 
-static const char *get_mode(const char *str, unsigned int *modep)
+const char *decode_tree_mode(const char *str, unsigned int *modep)
 {
 	unsigned char c;
 	unsigned int mode = 0;
@@ -28,7 +28,7 @@ static void decode_tree_entry(struct tree_desc *desc, const char *buf, unsigned
 	if (size < 24 || buf[size - 21])
 		die("corrupt tree file");
 
-	path = get_mode(buf, &mode);
+	path = decode_tree_mode(buf, &mode);
 	if (!path || !*path)
 		die("corrupt tree file");
 	len = strlen(path) + 1;
tree-walk.h +2 -0
index 7e3e0b5..6bbde1c 100644
@@ -13,6 +13,8 @@ struct tree_desc {
 	unsigned int size;
 };
 
+const char *decode_tree_mode(const char *str, unsigned int *modep);
+
 static inline const unsigned char *tree_entry_extract(struct tree_desc *desc, const char **pathp, unsigned int *modep)
 {
 	*pathp = desc->entry.path;
Signature · 1 line
1.7.3.37.gb6088b
#10 lore raw

PATCH 5/6 Introduce the git fast-import-helper

In reply to Tomas Carnecky

The g-f-i-h is a heavily modified (and simplified where possible) copy of git-fast-import. It has a few very important changes which make it suitable to be used in the new generation of remote helpers.

1) It does not update refs itself. Instead, for each 'mark' it sees, it writes the SHA1 of the corresponding git object to stdout. The remote helper can read this data and pass it along to core git for example.

2) It does not read/write mark files itself. Managing the marks is now up to the application which uses g-f-i-h. To support that, a new command was added: 'mark <name> <sha1>'. It can be used to feed g-f-i-h with existing marks from earlier sessions. Also, marks can now be arbitrary strings and not just numbers. This allows remote helpers to use for example whole revision strings (r42 for svn or mercurial changeset IDs).

3) Memory management has been significantly simplified. No more pools and custom allocators. It uses plain malloc/free. Uses struct hash_table instead of custom data structures. This may make it a bit slower than the original, but on the other hand it reduces the complexity of the source code.

---
 .gitignore           |    1 +
 Makefile             |    1 +
 fast-import-helper.c | 2201 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 2203 insertions(+), 0 deletions(-)
 create mode 100644 fast-import-helper.c
.gitignore +1 -0
index 20560b8..c8aa8c7 100644
@@ -42,6 +42,7 @@
 /git-describe
 /git-fast-export
 /git-fast-import
+/git-fast-import-helper
 /git-fetch
 /git-fetch--tool
 /git-fetch-pack
Makefile +1 -0
index 8a56b9a..f8a9c40 100644
@@ -402,6 +402,7 @@ EXTRA_PROGRAMS =
 PROGRAMS += $(EXTRA_PROGRAMS)
 
 PROGRAM_OBJS += fast-import.o
+PROGRAM_OBJS += fast-import-helper.o
 PROGRAM_OBJS += imap-send.o
 PROGRAM_OBJS += shell.o
 PROGRAM_OBJS += show-index.o
fast-import-helper.c new file +2201 -0

2,201 lines not on this page · raw message

Signature · 1 line
1.7.3.37.gb6088b
#11 lore raw

Re: [PATCH 5/6] Introduce the git fast-import-helper

Quoted 1 line · Tomas Carnecky
 fast-import-helper.c | 2201 ++++++++++++++++++++++++++++++++++++++++++++++++++

Aaa! Could someone send a diff of this against the usual fast-import.c?

I'm sure you won't be surprised to hear I am not optimistic about the long-term maintainability of two separate fast-import implementations. If we get usability improvements by patching the existing fast-import, that would be much better.

#12 lore raw

Re: [PATCH 5/6] Introduce the git fast-import-helper

Quoted 10 lines · Jonathan Nieder
Quoted 1 line · Tomas Carnecky
 fast-import-helper.c | 2201 ++++++++++++++++++++++++++++++++++++++++++++++++++

Aaa! Could someone send a diff of this against the usual fast-import.c?

I'm sure you won't be surprised to hear I am not optimistic about the long-term maintainability of two separate fast-import implementations. If we get usability improvements by patching the existing fast-import, that would be much better.

I agree. But when I started hacking on fast-import it seemed easier to completely rip out parts that I didn't need and then add the few bits that I needed. I only need two new things from fast-import:
1) support non-numeric marks (and even this is maybe not strictly
required)
2) dump the mark->sha1 mapping immediately after creating the object
(I heard there is a patch somewhere that does just that)
All other changes are not needed. Though I think there are a few things which could be ported back to fast-import.c. I'll try to see which changes make sense to be backported and will post patches.

tom

#13 lore raw

Re: [PATCH 5/6] Introduce the git fast-import-helper

Heya,

Quoted 3 lines · Tomas Carnecky

I agree. But when I started hacking on fast-import it seemed easier to completely rip out parts that I didn't need and then add the few bits that I needed.

Sure, that would indeed be easier, but so much less maintainable :).

Quoted 3 lines

I only need two new things from fast-import:
 1) support non-numeric marks (and even this is maybe not strictly required)

If this can be avoided, or worked around somehow, it would be a boon to performance. The current marks implementation uses a hash table index by the mark number, which is O(1), very efficient.

Quoted 2 lines

 2) dump the mark->sha1 mapping immediately after creating the object (I heard there is a patch somewhere that does just that)

Why do you need that? Wouldn't the "write created object name to stdout" not be sufficient?

Quoted 3 lines

All other changes are not needed. Though I think there are a few things which could be ported back to fast-import.c. I'll try to see which changes make sense to be backported and will post patches.

I'd be very interested in those :).

Signature · 3 lines
Cheers,

Sverre Rabbelier
#14 lore raw

Re: [PATCH 5/6] Introduce the git fast-import-helper

Quoted 7 lines · Sverre Rabbelier
Quoted 3 lines

I only need two new things from fast-import:
1) support non-numeric marks (and even this is maybe not strictly
required)

If this can be avoided, or worked around somehow, it would be a boon to performance. The current marks implementation uses a hash table index by the mark number, which is O(1), very efficient.

I also use a hash table (struct hash_table from hash.h). It's indexed by the atom. So it's about equally fast as the existing one but uses slightly more memory. I measured the speed and fih is about 5% slower than fi. Also, I found out that setting max_packfile to 32MB makes the import much faster (from 10 minutes down to 3m to import the sources of git itself).

Quoted 5 lines
Quoted 2 lines

2) dump the mark->sha1 mapping immediately after creating the object
(I heard there is a patch somewhere that does just that)

Why do you need that? Wouldn't the "write created object name to stdout" not be sufficient?

I do: fprintf(stdout, "mark :%s %s\n", mark, sha1_to_hex(sha1)); One reason why not just write the plain hash is because that's the same syntax as the fih accepts in its input. This way you can do:

  $ ( cat marks; cat fast-export-stream ) | git fast-import-helper >> marks
and can restart at any time. Also, making the output a bit more
structured allows it to be easily extended in the future.

tom

#16 lore raw

PATCH 6/6 Add git-remote-svn

In reply to Tomas Carnecky

This is an experimental git remote helper for svn repositories. It uses the new git fast-import-helper. It only works with local svn repos (not over network). It uses notes to save the git commit -> svn revision mapping (refs/notes/svn).

This remote helper serves as a technology preview of what the new type of remote helpers can do.

It assumes that the svn repo uses the standard layout (trunk, branches).

---
 .gitignore        |    1 +
 Makefile          |    1 +
 git-remote-svn.py |  408 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 410 insertions(+), 0 deletions(-)
 create mode 100644 git-remote-svn.py
.gitignore +1 -0
index c8aa8c7..0a6011c 100644
@@ -114,6 +114,7 @@
 /git-remote-ftp
 /git-remote-ftps
 /git-remote-testgit
+/git-remote-svn
 /git-repack
 /git-replace
 /git-repo-config
Makefile +1 -0
index f8a9c40..eb959c9 100644
@@ -387,6 +387,7 @@ SCRIPT_PERL += git-send-email.perl
 SCRIPT_PERL += git-svn.perl
 
 SCRIPT_PYTHON += git-remote-testgit.py
+SCRIPT_PYTHON += git-remote-svn.py
 
 SCRIPTS = $(patsubst %.sh,%,$(SCRIPT_SH)) \
 	  $(patsubst %.perl,%,$(SCRIPT_PERL)) \
git-remote-svn.py new file +408 -0

408 lines not on this page · raw message

Signature · 1 line
1.7.3.37.gb6088b
#18 lore raw
In reply to Tomas Carnecky

Heya,

Quoted 4 lines · Tomas Carnecky

My work has the goal of making interaction with foreign SCMs more natural. The work that was done on remote helpers is the right direction. But the 'import' and 'export' commands are the wrong approach I think.

I'm not convinced that they are, but we'll see.

Quoted 3 lines

The problem I have with 'import' is that updating the refs is left up to the remote helper (or git-fast-import). So you lose the nice output from ls-remote/fetch: non-ff and other warnings etc.

This is a good point, and I like the patches addressing this.

Quoted 5 lines

I slightly
modified how the remote helpers (and fast-import) work, now they behave exactly like 'core' git when fetching: Git tells the remote helper to fetch some refs, the helper does that and creates a pack and git then updates the refs (or not, depending on fast-forward etc).

Again, these patches I like, and would like to see them included. I'll probably pick them up and send them out as part of my next git-remote-hg reroll if nothing happens with them.

Quoted 2 lines

To test this
approach I created a simple remote helper for svn.

I guess it suffices as a POC, but I'd have preferred to see collaboration with the people working on git-remote-svn instead (cc-ed).

Quoted 8 lines
$ git ls-remote svn::/Volumes/Dump/Source/Mirror/Transmission/
r1017 (impure)                            trunk
r919 (impure)                             branches/nat-traversal
r480 (impure)                             branches/0.6

Git learned to understand version numbers from foreign SCMs. Git displays those as 'impure' because it knows that version exists but does not know yet which git commit that version maps to.

Very interesting. This is a useful feature, I approve.

Quoted 6 lines
$ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/
*:refs/remotes/svn/*
From svn::/Volumes/Dump/Source/Mirror/Transmission
 * [new branch]      trunk      -> svn/trunk
 * [new branch]      branches/nat-traversal -> svn/branches/nat-traversal
 * [new branch]      branches/0.6 -> svn/branches/0.6

Interesting, if you do 'git remote add svn svn::/Volumes/Dump/Source/Mirror/Transmission/' and then do 'git fetch svn', do you get (more or less) the same output?

Quoted 6 lines

Git tells the remote helper to 'fetch r1017 trunk'.  The remote helper does that, creates the pack and then tells git that it imported r1017 as commit c5fed7ec. This is done with a new reply to the 'fetch' command:
'map r1017 c5fed7ec'. The remote helper can use that to inform core git as which git commit the impure ref was imported. Git can then update the refs. At no point does the remote helper manipulate refs directly.

I love this. Very elegant.

Quoted 3 lines

The pack is created by a heavily modified git-fast-import. The existing fast-import not only creates the pack but also updates the refs. This is no longer desired as git is in charge of updating the refs.

NAK. I object against forking git fast-import just for this purpose. I'd much rather just modify git fast-import to learn to learn not update refs, which should be easy enough.

Quoted 5 lines

My modified
fast-import works like this: After creating a commit, it writes it's git object name to stdout. That way the remote helper can figure out as which git commits the svn revisions were imported and relay that back to core git using the above described 'map' reply.

Work has been underway to teach git fast-import to do just this (courtesy of Jonathan), no need to fork git fast-import to achieve that.

Quoted 9 lines
$ git show --show-notes=svn svn/trunk
commit c5fed7ecc318363523d3ea2045e1c16a378bb10c
Author: livings124 <livings124@localhost>
Date:   Wed Oct 18 13:57:19 2006 +0000

   more traditional toolbar icons for those afraid of change

Notes (svn):
   b4697c4a-7d4c-4a30-bd92-6745580d73b3/trunk@1017

Very nice! I'm definitely stealing the note-generating code for git-remote-hg.

Quoted 4 lines

The svn helper needs to be able to map svn revisions to git commits. git-svn does this by adding the 'git-svn-id' line to each commit message. I'm using git notes for that and it seems to work just fine. The note contains the repo UUID, path within the repo and revision.

Very elegant.

Quoted 8 lines

There was a challenge how to update the notes ref (refs/notes/svn). As with fast-import, I did not want the remote helper to do it. Neither the remote helper nor fast-import should be writing any refs. But core git can only update refs which were discovered during transport->fetch(). I modified the remote helper 'fetch' command and the transport->fetch() function to return an optional list of refs. These are the refs that the remote helper wants to update but which should not be presented to the user (because these are internally used refs, such as my svn notes).

Nice.

Quoted 8 lines

So the whole session between git and my svn remote helper looks like this:

Quoted 1 line

list

< :r1017 trunk

Quoted 1 line

fetch :r1017 trunk

[helper creates the pack including history up to r1017 and associated svn notes]
< map r1017 <commit corresponding to r1017>
< silent refs/notes/svn <new commit which stores the updated svn notes>

Looks good.

Quoted 5 lines
$ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/
*:refs/remotes/svn/*
From svn::/Volumes/Dump/Source/Mirror/Transmission
  c5fed7e..228eaf3  trunk      -> svn/trunk
Quoted 7 lines
$ git fetch svn::/Volumes/Dump/Source/Mirror/Transmission/
*:refs/remotes/svn/*
From svn::/Volumes/Dump/Source/Mirror/Transmission
  228eaf3..207e5e5  trunk      -> svn/trunk
 * [new branch]      branches/scrape -> svn/branches/scrape
 * [new branch]      branches/multitracker -> svn/branches/multitracker
 * [new branch]      branches/io -> svn/branches/io

Note to other reviewers: the repository was updated in between successive calls to 'git fetch', see below.

Quoted 5 lines

Updating the svn branches works like expected. The remote helper automatically detects which branches it already imported (by going through all refs and the attached svn notes) and creates a new pack with the new commits. New branches are also detected. The svn notes are updated accordingly.

I assume this only works for regular svn repositories? I guess it doesn't really matter to the rest of the series, since the 'git-remote-svn' helper is more of a POC I think?

Most of your extensions to the helper protocol make sense. However, after re-reading your series I think we _should_ keep the 'import' and 'export' command, so that helpers don't have to invoke 'git fast-import' or 'git fast-import' themselves. I suspect it will be more efficient than your approach as well. Speed _is_ a very important concern, to have decent support for foreign
remotes, imports/exports should be as fast as possible. Also, this series doesn't address pushing back to the foreign scm, which is very convenient through the 'export' command.

Either way, thank you very much for working on this!

Signature · 3 lines
Cheers,

Sverre Rabbelier
#19 lore raw

Hi,

Quoted 8 lines · Sverre Rabbelier
Quoted 2 lines · Tomas Carnecky

To test this
approach I created a simple remote helper for svn.

I guess it suffices as a POC, but I'd have preferred to see collaboration with the people working on git-remote-svn instead (cc-ed).

Just a quick note: if this approach gets a working remote helper in the hands of users faster, I'm all for it.

My only concern is the name: if it is not compatible the planned remote helper from the summer of code project, they should probably get different names. Correct me if I'm wrong, but the main differences are:

- this is scripted and uses local svn working copy operations; the soc project is in C and uses remote access ("replay")

- this uses the nice ls-remote output etc. Ram, do you think this would be easy to use for remote-svn?

So, not many differences. Maybe we can standardize the interface and consider them alternate implementations?

#20 lore raw

Hi Tom and Jonathan,

Quoted 6 lines · Jonathan Nieder
Quoted 5 lines · Sverre Rabbelier
Quoted 2 lines · Tomas Carnecky

To test this
approach I created a simple remote helper for svn.

Quoted 6 lines
Quoted 3 lines

I guess it suffices as a POC, but I'd have preferred to see collaboration with the people working on git-remote-svn instead (cc-ed).

Just a quick note: if this approach gets a working remote helper in the hands of users faster, I'm all for it.

First off, great work on the fast-import and the remote-helper! I am very impressed with the results.

Quoted 7 lines

My only concern is the name: if it is not compatible the planned remote helper from the summer of code project, they should probably get different names. Correct me if I'm wrong, but the main differences are:

- this is scripted and uses local svn working copy operations; the soc project is in C and uses remote access ("replay")

Yes, the name definitely needs to be changed. Maybe name it something along the lines of "local-svn"?

Quoted 2 lines

- this uses the nice ls-remote output etc. Ram, do you think this would be easy to use for remote-svn?

This is quite awesome. Yeah, I suppose we can use it for remote-svn as well.

Quoted 2 lines

So, not many differences. Maybe we can standardize the interface and consider them alternate implementations?

This helper can't be merged in until Tom's changes to fast-import are ported to the current fast-import. I just hope that those changes to fast-import don't conflict with the changes git-remote-svn will need. Frankly, I'd rather we work towards a common goal.

-- Ram

#15 lore raw

Re: [PATCH 5/6] Introduce the git fast-import-helper

In reply to Tomas Carnecky

Heya,

Quoted 6 lines · Tomas Carnecky

I also use a hash table (struct hash_table from hash.h). It's indexed by the atom. So it's about equally fast as the existing one but uses slightly more memory. I measured the speed and fih is about 5% slower than fi. Also, I found out that setting max_packfile to 32MB makes the import much faster (from 10 minutes down to 3m to import the sources of git itself).

That's curious.

A 5% increase is significant. Imagine importing netbeans, (which is actually a use case of git-remote-hg), which takes about 4 hours. A 5% slowdown means the process will take more than 10 minutes longer.

Quoted 6 lines

I do: fprintf(stdout, "mark :%s %s\n", mark, sha1_to_hex(sha1)); One reason why not just write the plain hash is because that's the same syntax as the fih accepts in its input. This way you can do:

 $ ( cat marks; cat fast-export-stream ) | git fast-import-helper >> marks
and can restart at any time. Also, making the output a bit more
structured allows it to be easily extended in the future.

I don't see much benefit tbh, if we want to do something like that it could (relatively) easy be added to regular git fast-import with a feature. So you'd start the stream with "feature new-marks-format" and only then follow up with "feature import-marks=...". Ditto on the commandline, `git fast-import --new-marks-format --import-marks=...". If it turns out to be very useful/popular it can be made the default after warning for a full release first.

Signature · 3 lines
Cheers,

Sverre Rabbelier
#2 lore raw

Re: [PATCH 1/6] Remote helper: accept ':<value> <name>' as a response to 'list'

In reply to Tomas Carnecky

Hi,

Quoted 4 lines · Tomas Carnecky

The ref is first stored
as 'impure', meaning that it doesn't have any representation within git. Only after the remote helper fetches that version into git, it can tell us which git object (SHA1) that revision maps to.

Hmm.

The existing ls-remote output looks something like

<object id> HEAD
<object id> <refname>
<object id> <refname>
...
<object id> <tag refname>
<object id> <tag refname>^{}
...

In particular, each line has a 40-character object id, a tab character, and then something like a refname.

If we want to extend that, wouldn't we need to do something like

	0000...0000	<refname>
	0000...0000	<refname> is r11

to avoid breaking people's scripts? For example, I wouldn't be surprised if scripts are relying on the following two properties:

- each object id is 40 characters (e.g., the old fetch--tool in contrib/examples relies on this)
- each object id contains no whitespace

simply because they are convenient to script with.

#5 lore raw

Re: [PATCH 2/6] Allow more than one keepfile in the transport

In reply to Tomas Carnecky
Quoted 1 line · Tomas Carnecky

Use an array to keep track of the pack lockfiles.

Should probably use a string_list (which is the same thing :)) to save some code.

transport-helper.c static int fetch_with_fetch(struct transport *transport, · line 365

Quoted 11 lines
transport-helper.c +1 -4
@@ -362,10 +362,7 @@ static int fetch_with_fetch(struct transport *transport,
 
 		if (!prefixcmp(buf.buf, "lock ")) {
 			const char *name = buf.buf + 5;
-			if (transport->pack_lockfile)
-				warning("%s also locked %s", data->name, name);
-			else
-				transport->pack_lockfile = xstrdup(name);
+			transport_keep(transport, name);

Won't buf.buf be released before the lockfile needs to be read?

So I suspect the strdup is necessary.

#7 lore raw

Re: [PATCH 3/6] Allow the transport fetch command to add additional refs

In reply to Tomas Carnecky
Quoted 6 lines · Tomas Carnecky

Remote helpers such as those for svn may chose to save the Git SHA1 -> Subversion revision mapping as notes attached to the commits (as opposed to strings in the commit message itself). The helper would need to update the notes on each fetch, but the user should not be bothered by the presence of that ref. The remote helper can update the notes tree through fast-import and then inform Git core that it should silently update the notes ref.

What is the advantage of this approach over having fast-import update the notes ref (but not the others) on its own?

Curious,
Jonathan

#9 lore raw

Re: [PATCH 4/6] Rename get_mode() to decode_tree_mode() and export it

In reply to Tomas Carnecky
Quoted 2 lines · Tomas Carnecky

Other sources (fast-import-helper.c) may want to use this function to parse trees.

Sounds like a reasonable idea, but you've left out what the function does.

So:

get_mode() parses an octal "100644 " string as might appear in a tree object. Returns a pointer to the first character after the mode string, or NULL on error.

#17 lore raw

Re: [PATCH 6/6] Add git-remote-svn

In reply to Tomas Carnecky
Quoted 2 lines · Tomas Carnecky

This is an experimental git remote helper for svn repositories. It uses the new git fast-import-helper.

Not sure how to review this without some sense of how the fast-import-helper works.

Quoted 2 lines

It uses notes to save the git commit -> svn revision mapping (refs/notes/svn).

Good. :)

Caching the mapping back in the other direction would be very simple. A simple array of commit hashes indexed by svn revision number would not take up much space in memory and could easily be randomly accessed on disk.

#3 lore raw

Re: [PATCH 1/6] Remote helper: accept ':<value> <name>' as a response to 'list'

In reply to Jonathan Nieder

Heya,

Quoted 9 lines · Jonathan Nieder

In particular, each line has a 40-character object id, a tab character, and then something like a refname.

If we want to extend that, wouldn't we need to do something like

       0000...0000     <refname>
       0000...0000     <refname> is r11

to avoid breaking people's scripts?

I don't know... I'd suspect that people's scripts are going to break regardless, but if this is being used in scripts I agree we should make it as easy as possible for scripters to parse this output.

Signature · 3 lines
Cheers,

Sverre Rabbelier