Skip to contentSkip to conversation

git-pack-objects: Automatically pack annotated tags if object was packed

PATCH In review Shawn O. Pearce 4 participants 8 messages Jun 15, 2016 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 git-pack-objects: Automatically pack annotated tags if object was packed

The new option "--auto-follow-tags" allows the caller to request that any annotated tag be included into the packfile if the object the tag references was also included as part of the packfile.

This option can be useful on the server side of a native git transport, where the server knows what commits it is including into a packfile to update the client. If new annotated tags have been introduced then we can also include them in the packfile, saving the client from needing to request them through a second connection.

This change only introduces the backend option and provides a test. Protocol extensions to make this useful in fetch-pack/upload-pack are still necessary to activate the logic during transport.

---
 Documentation/git-pack-objects.txt |    5 ++
 builtin-pack-objects.c             |   24 +++++++++-
 t/t5305-autofollow-tag.sh          |   84 ++++++++++++++++++++++++++++++++++++
 3 files changed, 111 insertions(+), 2 deletions(-)
 create mode 100755 t/t5305-autofollow-tag.sh
Documentation/git-pack-objects.txt +5 -0
index 5c1bd3b..0ba9cbe 100644
@@ -170,6 +170,11 @@ base-name::
 	length, this option typically shrinks the resulting
 	packfile by 3-5 per-cent.
 
+--auto-follow-tags::
+	Automatically include annotated tags if the object they
+	reference was included in the resulting packfile.  This
+	can be useful to send new tags to native git clients.
+
 --threads=<n>::
 	Specifies the number of threads to spawn when searching for best
 	delta matches.  This requires that pack-objects be compiled with
builtin-pack-objects.c +22 -2
index 2799e68..8b5d90a 100644
@@ -15,6 +15,7 @@
 #include "revision.h"
 #include "list-objects.h"
 #include "progress.h"
+#include "refs.h"
 
 #ifdef THREADED_DELTA_SEARCH
 #include "thread-utils.h"
@@ -27,7 +28,8 @@ git-pack-objects [{ -q | --progress | --all-progress }] \n\
 	[--window=N] [--window-memory=N] [--depth=N] \n\
 	[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \n\
 	[--threads=N] [--non-empty] [--revs [--unpacked | --all]*] [--reflog] \n\
-	[--stdout | base-name] [--keep-unreachable] [<ref-list | <object-list]";
+	[--stdout | base-name] [--auto-follow-tags] [--keep-unreachable] \n\
+	[<ref-list | <object-list]";
 
 struct object_entry {
 	struct pack_idx_entry idx;
@@ -63,7 +65,7 @@ static struct pack_idx_entry **written_list;
 static uint32_t nr_objects, nr_alloc, nr_result, nr_written;
 
 static int non_empty;
-static int no_reuse_delta, no_reuse_object, keep_unreachable;
+static int no_reuse_delta, no_reuse_object, keep_unreachable, auto_tags;
 static int local;
 static int incremental;
 static int allow_ofs_delta;
@@ -1630,6 +1632,18 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,
 #define ll_find_deltas(l, s, w, d, p)	find_deltas(l, &s, w, d, p)
 #endif
 
+static int add_ref_tag(const char *path, const unsigned char *sha1, int flag, void *cb_data)
+{
+	unsigned char peeled[20];
+
+	if (!prefixcmp(path, "refs/tags/") && /* is a tag? */
+	    !peel_ref(path, peeled)        && /* peelable? */
+	    !is_null_sha1(peeled)          && /* annotated tag? */
+	    locate_object_entry(peeled))      /* object packed? */
+		add_object_entry(sha1, OBJ_TAG, NULL, 0);
+	return 0;
+}
+
 static void prepare_pack(int window, int depth)
 {
 	struct object_entry **delta_list;
@@ -2033,6 +2047,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 			keep_unreachable = 1;
 			continue;
 		}
+		if (!strcmp("--auto-follow-tags", arg)) {
+			auto_tags = 1;
+			continue;
+		}
 		if (!strcmp("--unpacked", arg) ||
 		    !prefixcmp(arg, "--unpacked=") ||
 		    !strcmp("--reflog", arg) ||
@@ -2109,6 +2127,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		rp_av[rp_ac] = NULL;
 		get_object_list(rp_ac, rp_av);
 	}
+	if (auto_tags && nr_result)
+		for_each_ref(add_ref_tag, NULL);
 	stop_progress(&progress_state);
 
 	if (non_empty && !nr_result)
t/t5305-autofollow-tag.sh new file +84 -0

84 lines not on this page · raw message

Signature · 1 line
1.5.4.3.529.gb25fb
#1 lore raw
Quoted 3 lines · Shawn O. Pearce

The new option "--auto-follow-tags" allows the caller to request that any annotated tag be included into the packfile if the object the tag references was also included as part of the packfile.

Wouldn't "auto-include-tag" a better name for this option?

Nicolas

#2 lore raw
Quoted 7 lines · Nicolas Pitre
Quoted 3 lines · Shawn O. Pearce

The new option "--auto-follow-tags" allows the caller to request that any annotated tag be included into the packfile if the object the tag references was also included as part of the packfile.

Wouldn't "auto-include-tag" a better name for this option?

Ooooh. Indeed it would!

I shall respin this series based around this better name.

Signature · 1 line
Shawn.
#3 lore raw
Quoted 11 lines · Shawn O. Pearce
Quoted 6 lines · Nicolas Pitre
Quoted 3 lines · Shawn O. Pearce

The new option "--auto-follow-tags" allows the caller to request that any annotated tag be included into the packfile if the object the tag references was also included as part of the packfile.

Wouldn't "auto-include-tag" a better name for this option?

Ooooh. Indeed it would!

I shall respin this series based around this better name.

More bikeshedding: Why are you calling it "auto-"? If I tell the program to "--include-tags", I expect it to happen automatically.

-- Hannes

#4 lore raw
Quoted 7 lines · Johannes Sixt
Quoted 6 lines · Shawn O. Pearce
Quoted 5 lines · Nicolas Pitre
Quoted 3 lines · Shawn O. Pearce

The new option "--auto-follow-tags" allows the caller to request that any annotated tag be included into the packfile if the object the tag references was also included as part of the packfile.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Wouldn't "auto-include-tag" a better name for this option?

Ooooh. Indeed it would!

I shall respin this series based around this better name.

More bikeshedding: Why are you calling it "auto-"? If I tell the program to "--include-tags", I expect it to happen automatically.

Because its more automatic than manually including tags? I don't know.

Junio, if you haven't already merged this topic into someplace that I can't rewrite the series for a _3rd_ time, I might be willing to respin it without the "auto-" prefix. We are unlikely to ever have a reason to support "--manually-include-tags". Stone tablet based computing stopped being fun back in the '50s.

Signature · 1 line
Shawn.
#5 lore raw
In reply to Shawn O. Pearce

builtin-pack-objects.c int cmd_pack_objects(int argc, const char **argv, const char *prefix) · line 2131

Quoted 2 lines · Shawn O. Pearce
builtin-pack-objects.c +2 -0
+	if (auto_tags && nr_result)
+		for_each_ref(add_ref_tag, NULL);

There's for_each_tag_ref() that does the path-based filter internally, in a possibly more efficient way, and avoids open-coding the test for whether this is a tag sort of ref.

Aside from that, it looks good to me.

-Daniel
*This .sig left intentionally blank*

#6 lore raw
Quoted 8 lines · Daniel Barkalow

builtin-pack-objects.c int cmd_pack_objects(int argc, const char **argv, const char *prefix) · line 2131

Quoted 2 lines · Shawn O. Pearce
builtin-pack-objects.c +2 -0
+	if (auto_tags && nr_result)
+		for_each_ref(add_ref_tag, NULL);

There's for_each_tag_ref() that does the path-based filter internally, in a possibly more efficient way, and avoids open-coding the test for whether this is a tag sort of ref.

Yea, I know about that call, but I didn't use it because I was trying to use peel_ref() for its optimized peeled caching in a packed-refs file.

To use peel_ref() you have to pass the full ref name ("refs/tags/v1.0") and not a partial ref name ("v1.0") such as for_each_tag_ref() will return to the callback function. So I'd have to actually do extra work to reconstruct a ref name, and possibly an invalid ref name at that if for_each_tag_ref() ever looked outside of the refs/tags/ namespace.

Today there is no performance benefit to for_each_tag_ref() over for_each_ref(), but using for_each_ref() lets me use peel_ref(), which is a performance improvement if the ref in question came from the packed-refs file.

So.... do we keep things as-is or try to optimize peel_ref() (or some new variant of it?!?) for use in for_each_tag_ref() just on the off chance that we somehow can optimize for_each_tag_ref to avoid scanning refs/heads/? Note that I did look into trying to do that optimization to for_each_tag_ref() earlier last week and concluded it wasn't easily possible right now, and wasn't likely to be worth the development costs.

Signature · 1 line
Shawn.
#7 lore raw
Quoted 10 lines · Shawn O. Pearce
Quoted 8 lines · Daniel Barkalow

builtin-pack-objects.c int cmd_pack_objects(int argc, const char **argv, const char *prefix) · line 2131

Quoted 2 lines · Shawn O. Pearce
builtin-pack-objects.c +2 -0
+	if (auto_tags && nr_result)
+		for_each_ref(add_ref_tag, NULL);

There's for_each_tag_ref() that does the path-based filter internally, in a possibly more efficient way, and avoids open-coding the test for whether this is a tag sort of ref.

Quoted 3 lines

Yea, I know about that call, but I didn't use it because I was trying to use peel_ref() for its optimized peeled caching in a packed-refs file.

Ah, okay. I hadn't noticed that it didn't pass on the full name. Using for_each_ref() is better, then, I think.

-Daniel
*This .sig left intentionally blank*