Compile-time extensions for list-object-filter

Adds an extension: option to list-object-filters, these are implemented by static libraries that must be compiled into Git. The Makefile argument FILTER_EXTENSIONS makes it easier to compile these extensions into a custom build of Git. When no custom filter-extensions are supplied, Git works as normal.

Andrew Olsen (4):
Compile-time extensions for list-object-filter
Makefile for list-object-filter extensions
Sample list-object-filter extensions
Documentation for list-object-filter extensions

 .gitignore                                    |   1 +
 Documentation/config/uploadpack.txt           |   7 +-
 Documentation/rev-list-options.txt            |   4 +
 Makefile                                      |  35 +++-
 compat/vcbuild/README                         |   5 +-
 config.mak.uname                              |   6 +-
 contrib/buildsystems/CMakeLists.txt           |   7 +
 contrib/filter-extensions/README.txt          | 153 ++++++++++++++++++
 contrib/filter-extensions/rand/.gitignore     |   2 +
 contrib/filter-extensions/rand/Makefile       |  28 ++++
 contrib/filter-extensions/rand/rand.c         | 103 ++++++++++++
 contrib/filter-extensions/rand_cpp/.gitignore |   2 +
 contrib/filter-extensions/rand_cpp/Makefile   |  34 ++++
 .../rand_cpp/adapter_functions.c              |   6 +
 .../rand_cpp/adapter_functions.h              |  10 ++
 contrib/filter-extensions/rand_cpp/rand.cpp   | 103 ++++++++++++
 generate-list-objects-filter-extensions.sh    |  53 ++++++
 list-objects-filter-extensions.h              | 107 ++++++++++++
 list-objects-filter-options.c                 |  47 ++++++
 list-objects-filter-options.h                 |   6 +
 list-objects-filter.c                         |  84 ++++++++++
 21 files changed, 793 insertions(+), 10 deletions(-)
 create mode 100644 contrib/filter-extensions/README.txt
 create mode 100644 contrib/filter-extensions/rand/.gitignore
 create mode 100644 contrib/filter-extensions/rand/Makefile
 create mode 100644 contrib/filter-extensions/rand/rand.c
 create mode 100644 contrib/filter-extensions/rand_cpp/.gitignore
 create mode 100644 contrib/filter-extensions/rand_cpp/Makefile
 create mode 100644 contrib/filter-extensions/rand_cpp/adapter_functions.c
 create mode 100644 contrib/filter-extensions/rand_cpp/adapter_functions.h
 create mode 100644 contrib/filter-extensions/rand_cpp/rand.cpp
 create mode 100755 generate-list-objects-filter-extensions.sh
 create mode 100644 list-objects-filter-extensions.h

base-commit: e0a2f5cbc585657e757385ad918f167f519cfb96
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1031%2Fkoordinates%2Flist-objects-filter-extensions-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1031/koordinates/list-objects-filter-extensions-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1031

gitgitgadget

From: Andrew Olsen <[email protected]>

Adds an extension:<custom-filter> option to list-object-filters, these are implemented by static libraries that must be compiled into Git. C code changes only - Makefile changes follow.

---
 .gitignore                                 |   1 +
 generate-list-objects-filter-extensions.sh |  53 ++++++++++
 list-objects-filter-extensions.h           | 107 +++++++++++++++++++++
 list-objects-filter-options.c              |  47 +++++++++
 list-objects-filter-options.h              |   6 ++
 list-objects-filter.c                      |  84 ++++++++++++++++
 6 files changed, 298 insertions(+)
 create mode 100755 generate-list-objects-filter-extensions.sh
 create mode 100644 list-objects-filter-extensions.h

index 311841f9bed..3564cb01ad7 100644
@@ -190,6 +190,7 @@
 /gitweb/static/gitweb.min.*
 /config-list.h
 /command-list.h
+/list-objects-filter-extensions.c
 *.tar.gz
 *.dsc
 *.deb

index da5b6737e27..df3e360324e 100644
@@ -15,6 +15,7 @@ enum list_objects_filter_choice {
 	LOFC_TREE_DEPTH,
 	LOFC_SPARSE_OID,
 	LOFC_OBJECT_TYPE,
+	LOFC_EXTENSION,
 	LOFC_COMBINE,
 	LOFC__COUNT /* must be last */
 };

@@ -58,6 +59,11 @@ struct list_objects_filter_options {
 	unsigned long tree_exclude_depth;
 	enum object_type object_type;
 
+	/* LOFC_EXTENSION values */
+
+	char *extension_name;
+	char *extension_value;
+
 	/* LOFC_COMBINE values */
 
 	/* This array contains all the subfilters which this filter combines. */

gitgitgadget

From: Andrew Olsen <[email protected]>

Custom list-object-filter extensions can be compiled into Git using the FILTER_EXTENSIONS Makefile argument.

---
 Makefile                            | 35 +++++++++++++++++++++++++++--
 compat/vcbuild/README               |  5 +++--
 config.mak.uname                    |  6 ++---
 contrib/buildsystems/CMakeLists.txt |  7 ++++++
 4 files changed, 46 insertions(+), 7 deletions(-)

index 429c276058d..0b1d0be81a5 100644
@@ -471,6 +471,11 @@ all::
 # directory, and the JSON compilation database 'compile_commands.json' will be
 # created at the root of the repository.
 #
+# Define FILTER_EXTENSIONS to a space-separated list of static library plugins
+# that implement the filter-object-list extension API. Each of this filter
+# extensions will then be available in addition to the builtin ones such as
+# "blob:limit" and "object:type". See contrib/filter-extensions/README.txt
+#
 # Define DEVELOPER to enable more compiler warnings. Compiler version
 # and family are auto detected, but could be overridden by defining
 # COMPILER_FEATURES (see config.mak.dev). You can still set

@@ -824,6 +829,7 @@ XDIFF_LIB = xdiff/lib.a
 
 GENERATED_H += command-list.h
 GENERATED_H += config-list.h
+GENERATED_C += list-objects-filter-extensions.c
 
 LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
 	$(FIND) . \

@@ -916,6 +922,7 @@ LIB_OBJS += levenshtein.o
 LIB_OBJS += line-log.o
 LIB_OBJS += line-range.o
 LIB_OBJS += linear-assignment.o
+LIB_OBJS += list-objects-filter-extensions.o
 LIB_OBJS += list-objects-filter-options.o
 LIB_OBJS += list-objects-filter.o
 LIB_OBJS += list-objects.o

@@ -2116,6 +2123,19 @@ ifdef DEFAULT_HELP_FORMAT
 BASIC_CFLAGS += -DDEFAULT_HELP_FORMAT='"$(DEFAULT_HELP_FORMAT)"'
 endif
 
+ifneq ($(FILTER_EXTENSIONS),)
+FILTER_EXT_PATHS = $(dir $(FILTER_EXTENSIONS))
+
+$(FILTER_EXTENSIONS): $(FILTER_EXT_PATHS)
+	$(QUIET_SUBDIR0)$(@D) $(QUIET_SUBDIR1) \
+		ALL_CFLAGS='$(subst ','\'',$(ALL_CFLAGS))' \
+		ALL_LDFLAGS='$(subst ','\'',$(ALL_LDFLAGS))' \
+		PROFILE_DIR='$(subst ','\'',$(PROFILE_DIR))' \
+		$(@F)
+
+GITLIBS += $(FILTER_EXTENSIONS)
+endif
+
 PAGER_ENV_SQ = $(subst ','\'',$(PAGER_ENV))
 PAGER_ENV_CQ = "$(subst ",\",$(subst \,\\,$(PAGER_ENV)))"
 PAGER_ENV_CQ_SQ = $(subst ','\'',$(PAGER_ENV_CQ))

@@ -2222,7 +2242,7 @@ git.sp git.s git.o: EXTRA_CPPFLAGS = \
 	'-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \
 	'-DGIT_INFO_PATH="$(infodir_relative_SQ)"'
 
-git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS)
+git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS) $(EXTENSION_LIBS)
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
 		$(filter %.o,$^) $(LIBS)
 

@@ -2261,6 +2281,10 @@ command-list.h: $(wildcard Documentation/git*.txt)
 		$(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \
 		command-list.txt >$@+ && mv $@+ $@
 
+list-objects-filter-extensions.c: generate-list-objects-filter-extensions.sh GIT-BUILD-OPTIONS
+	$(QUIET_GEN)$(SHELL_PATH) ./generate-list-objects-filter-extensions.sh \
+		$(FILTER_EXTENSIONS) > $@+ && mv $@+ $@
+
 SCRIPT_DEFINES = $(SHELL_PATH_SQ):$(DIFF_SQ):$(GIT_VERSION):\
 	$(localedir_SQ):$(NO_CURL):$(USE_GETTEXT_SCHEME):$(SANE_TOOL_PATH_SQ):\
 	$(gitwebdir_SQ):$(PERL_PATH_SQ):$(SANE_TEXT_GREP):$(PAGER_ENV):\

@@ -2612,6 +2636,7 @@ $(LIB_FILE): $(LIB_OBJS)
 $(XDIFF_LIB): $(XDIFF_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
+
 export DEFAULT_EDITOR DEFAULT_PAGER
 
 Documentation/GIT-EXCLUDED-PROGRAMS: FORCE

@@ -2857,6 +2882,9 @@ ifdef RUNTIME_PREFIX
 	@echo RUNTIME_PREFIX=\'true\' >>$@+
 else
 	@echo RUNTIME_PREFIX=\'false\' >>$@+
+endif
+ifdef FILTER_EXTENSIONS
+	@echo FILTER_EXTENSIONS=\''$(subst ','\'',$(subst ','\'',$(FILTER_EXTENSIONS)))'\' >>$@+
 endif
 	@if cmp $@+ $@ >/dev/null 2>&1; then $(RM) $@+; else mv $@+ $@; fi
 

@@ -3241,7 +3269,7 @@ clean: profile-clean coverage-clean cocciclean
 	$(RM) $(HCC)
 	$(RM) -r bin-wrappers $(dep_dirs) $(compdb_dir) compile_commands.json
 	$(RM) -r po/build/
-	$(RM) *.pyc *.pyo */*.pyc */*.pyo $(GENERATED_H) $(ETAGS_TARGET) tags cscope*
+	$(RM) *.pyc *.pyo */*.pyc */*.pyo $(GENERATED_H) $(GENERATED_C) $(ETAGS_TARGET) tags cscope*
 	$(RM) -r .dist-tmp-dir .doc-tmp-dir
 	$(RM) $(GIT_TARNAME).tar.gz
 	$(RM) $(htmldocs).tar.gz $(manpages).tar.gz

@@ -3256,6 +3284,9 @@ endif
 ifndef NO_TCLTK
 	$(MAKE) -C gitk-git clean
 	$(MAKE) -C git-gui clean
+endif
+ifneq ($(FILTER_EXTENSIONS),)
+	$(foreach FP,$(FILTER_EXTENSIONS),$(MAKE) -C $(dir $(FP)) clean && ) true
 endif
 	$(RM) GIT-VERSION-FILE GIT-CFLAGS GIT-LDFLAGS GIT-BUILD-OPTIONS
 	$(RM) GIT-USER-AGENT GIT-PREFIX

index 51fb083dbbe..5e39022eade 100644
@@ -92,8 +92,9 @@ The Steps of Build Git with VS2008
    the git operations.
 
 3. Inside Git's directory run the command:
-       make command-list.h config-list.h
-   to generate the header file needed to compile git.
+       make command-list.h config-list.h list-objects-filter-extensions.c
+   to generate those source files that are not included in the repo, but
+   instead are automatically generated from other files.
 
 4. Then either build Git with the GNU Make Makefile in the Git projects
    root

index 76516aaa9a5..405e7d91e7a 100644
@@ -735,9 +735,9 @@ vcxproj:
 	 echo '</Project>') >git-remote-http/LinkOrCopyRemoteHttp.targets
 	git add -f git/LinkOrCopyBuiltins.targets git-remote-http/LinkOrCopyRemoteHttp.targets
 
-	# Add command-list.h and config-list.h
-	$(MAKE) MSVC=1 SKIP_VCPKG=1 prefix=/mingw64 config-list.h command-list.h
-	git add -f config-list.h command-list.h
+	# Add command-list.h, config-list.h list-objects-filter-extensions.c
+	$(MAKE) MSVC=1 SKIP_VCPKG=1 prefix=/mingw64 config-list.h command-list.h list-objects-filter-extensions.c
+	git add -f config-list.h command-list.h list-objects-filter-extensions.c
 
 	# Add scripts
 	rm -f perl/perl.mak

index 171b4124afe..60627a2892f 100644
@@ -624,6 +624,13 @@ if(NOT EXISTS ${CMAKE_BINARY_DIR}/config-list.h)
 			OUTPUT_FILE ${CMAKE_BINARY_DIR}/config-list.h)
 endif()
 
+if(NOT EXISTS ${CMAKE_BINARY_DIR}/list-objects-filter-extensions.c)
+	message("Generating list-objects-filter-extensions.c")
+	execute_process(COMMAND ${SH_EXE} ${CMAKE_SOURCE_DIR}/generate-list-objects-filter-extensions.sh
+			WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+			OUTPUT_FILE ${CMAKE_BINARY_DIR}/list-objects-filter-extensions.c)
+endif()
+
 include_directories(${CMAKE_BINARY_DIR})
 
 #build

gitgitgadget

This can be squashed to previous patch.

An old man doll... just what I always wanted! - Clara

From: Andrew Olsen <[email protected]>

Basic filter extension example which filters to a random subset of blobs, and another example which shows how to do the same in C++ and how to link in another library required by a filter extension. Documentation changes follow.

---
 contrib/filter-extensions/rand/.gitignore     |   2 +
 contrib/filter-extensions/rand/Makefile       |  28 +++++
 contrib/filter-extensions/rand/rand.c         | 103 ++++++++++++++++++
 contrib/filter-extensions/rand_cpp/.gitignore |   2 +
 contrib/filter-extensions/rand_cpp/Makefile   |  34 ++++++
 .../rand_cpp/adapter_functions.c              |   6 +
 .../rand_cpp/adapter_functions.h              |  10 ++
 contrib/filter-extensions/rand_cpp/rand.cpp   | 103 ++++++++++++++++++
 8 files changed, 288 insertions(+)
 create mode 100644 contrib/filter-extensions/rand/.gitignore
 create mode 100644 contrib/filter-extensions/rand/Makefile
 create mode 100644 contrib/filter-extensions/rand/rand.c
 create mode 100644 contrib/filter-extensions/rand_cpp/.gitignore
 create mode 100644 contrib/filter-extensions/rand_cpp/Makefile
 create mode 100644 contrib/filter-extensions/rand_cpp/adapter_functions.c
 create mode 100644 contrib/filter-extensions/rand_cpp/adapter_functions.h
 create mode 100644 contrib/filter-extensions/rand_cpp/rand.cpp

new file mode 100644
index 00000000000..9eca6c88cf2
@@ -0,0 +1,2 @@
+*.a
+*.o

new file mode 100644
index 00000000000..267221ee952
@@ -0,0 +1,28 @@
+# Run this via `FILTER_EXTENSIONS=contrib/filter-extensions/rand/rand.a make`
+# from the main git directory. That way we inherit useful variables.
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_CC       = @echo '   ' CC $@;
+	QUIET_AR       = @echo '   ' AR $@;
+endif
+endif
+
+FILTER_STATIC_LIB = rand.a
+
+all: $(FILTER_STATIC_LIB)
+ifeq ($(MAKELEVEL),0)
+	$(error "Run via parent git make")
+endif
+	@:
+
+$(FILTER_STATIC_LIB): rand.o
+	$(QUIET_AR)$(AR) $(ARFLAGS) $@ $^
+
+rand.o: rand.c
+	$(QUIET_CC)$(CC) -c $(ALL_CFLAGS) $<
+
+clean:
+	$(RM) $(FILTER_STATIC_LIB) rand.o
+
+.PHONY: all clean

new file mode 100644
index 00000000000..9eca6c88cf2
@@ -0,0 +1,2 @@
+*.a
+*.o

new file mode 100644
index 00000000000..278121e3d5a
@@ -0,0 +1,34 @@
+# Run this via `FILTER_EXTENSIONS=contrib/filter-extensions/rand_cpp/rand_cpp.a make`
+# from the main git directory. That way we inherit useful variables.
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_CC       = @echo '   ' CC $@;
+	QUIET_CXX       = @echo '   ' CXX $@;
+	QUIET_AR       = @echo '   ' AR $@;
+endif
+endif
+
+FILTER_STATIC_LIB = rand_cpp.a
+
+ALL_CXXFLAGS += -std=c++11
+
+all: $(FILTER_STATIC_LIB)
+ifeq ($(MAKELEVEL),0)
+	$(error "Run via parent git make")
+endif
+	@:
+
+$(FILTER_STATIC_LIB): rand.o adapter_functions.o
+	$(QUIET_AR)$(AR) $(ARFLAGS) $@ $^
+
+rand.o: rand.cpp
+	$(QUIET_CXX)$(CXX) -c $(ALL_CFLAGS) $(ALL_CXXFLAGS) $<
+
+adapter_functions.o: adapter_functions.c
+	$(QUIET_CC)$(CC) -c $(ALL_CFLAGS) $<
+
+clean:
+	$(RM) $(FILTER_STATIC_LIB) rand.o
+
+.PHONY: all clean

new file mode 100644
index 00000000000..0d9d2a2aa96
@@ -0,0 +1,6 @@
+#include "../../../git-compat-util.h"
+#include "../../../object.h"
+
+char *obj_to_hex_oid(struct object *obj) {
+    return oid_to_hex(&obj->oid);
+}

new file mode 100644
index 00000000000..1150c21a258
@@ -0,0 +1,10 @@
+#ifndef RAND_CPP_ADAPTER_FUNCTIONS_H
+#define RAND_CPP_ADAPTER_FUNCTIONS_H
+
+struct object;
+
+uint64_t getnanotime(void);
+
+char *obj_to_hex_oid(struct object *obj);
+
+#endif /* RAND_CPP_ADAPTER_FUNCTIONS_H */

gitgitgadget

From: Andrew Olsen <[email protected]>

Explains how to develop a custom extension for list-objects-filter behavior, and how to compile it into a custom build of Git using the FILTER_EXTENSIONS Makefile argument.

---
 Documentation/config/uploadpack.txt  |   7 +-
 Documentation/rev-list-options.txt   |   4 +
 contrib/filter-extensions/README.txt | 153 +++++++++++++++++++++++++++
 3 files changed, 161 insertions(+), 3 deletions(-)
 create mode 100644 contrib/filter-extensions/README.txt

index 32fad5bbe81..b2ef2421a6d 100644
@@ -66,9 +66,10 @@ uploadpackfilter.allow::
 uploadpackfilter.<filter>.allow::
 	Explicitly allow or ban the object filter corresponding to
 	`<filter>`, where `<filter>` may be one of: `blob:none`,
-	`blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`.
-	If using combined filters, both `combine` and all of the nested
-	filter kinds must be allowed. Defaults to `uploadpackfilter.allow`.
+	`blob:limit`, `tree`, `sparse:oid`, `combine`, or a named filter extension
+	`extension:<name>`. If using combined filters, both `combine` and all of
+	the nested filter kinds must be allowed. Defaults to
+	`uploadpackfilter.allow`.
 
 uploadpackfilter.tree.maxDepth::
 	Only allow `--filter=tree:<n>` when `<n>` is no more than the value of

index b7bd27e1713..d7a317f0aa1 100644
@@ -914,6 +914,10 @@ Note that the form '--filter=sparse:path=<path>' that wants to read
 from an arbitrary path on the filesystem has been dropped for security
 reasons.
 +
+The form '--filter=extension:<name>[=<arg>]' uses a compile-time extension
+to implement a named filter. Filter extensions may take an argument string
+which is passed via `<arg>`.
++
 Multiple '--filter=' flags can be specified to combine filters. Only
 objects which are accepted by every filter are included.
 +

gitgitgadget

Having skimmed this and the added documentation I think what's really missing is a "why"? What concrete use-case is this going to serve?

I.e. what is an extension you have in mind that's useful, but not so useful as to even suggest it for inclusion in git.git before coming up with this plug-in API mechanism?

Also, for such plug-ins the license is going to be GPL-v2 too I assume? But that aspect isn't covered at all.

I don't see why this series is useful (use cases?).

An old man doll... just what I always wanted! - Clara

Good point - sorry I sent this out without accompanying explanation. I'm still learning about contributing to Git.

The filter extension that I want to implement is a spatial filter - it will return blobs that store geometries that intersect with a given geometry, eg, "only return blobs in North America". This is useful to us at kartproject.org, "distributed version control for geospatial data", which is built on Git. But safe to say that this functionality is not generally useful to Git users.

However, the idea we have is that there will be others who want to implement custom filters also - perhaps like the spatial filter, these could be domain-specific filters that are not useful to most Git users, but allow for a custom Git to be more powerful when storing data from a particular domain. We could just fork git and do what we want with the fork, but defining a plugin interface makes it possible for us to keep using Git at master, instead of maintaining a fork indefinitely.

My colleague Robert Coup coded this up once already as a plugin library interface that could be loaded at runtime, and I've been tasked with rewriting it as a compile-time interface, which he thought was "more likely" (but of course not guaranteed) to be accepted as a worthwhile change to Git. He's unfortunately on the other side of the world to me and not working today, but I hope when he reappears he'll be able to say something more in defence of this idea, and perhaps give a history of the reasoning for this particular solution.

Regarding licenses: the sample extensions I'm contributing will be covered by Git's GPL-v2 (I assume), if they make it into the Git repository. Any other extensions that may be written by third party authors and are maintained elsewhere could be licensed as those authors see fit, as long as they take care not to violate the terms of Git's GPL-v2 when they distribute the extension or Git and the extension together. I could add a link to the GPL-v2 in the README warning developers to check it before distributing any kind of extension to Git. I'm not a lawyer and wouldn't want to give more specific advice than that.

That's interesting. I think you're probably right that returning blobs by a GEO filter is not going to be generally useful to git (I assume it's aware of some specially-encoded blobs?), but a mechanism that enables that might be...

While it would be easier for you it would leave this project stuck maintaining a C API interface, and indeed your documentation suggests that not only should users use the narrow C API provided here, but any arbitrary internal structs in git.git.

Personally I'm not per-se opposed to such a thing, but I think that we should really be considering and trying something like the clean/smudge hook interface first rather than a full C API.

This seems like a perfect fit for such an IPC interface, i.e. we'd have a hook to register custom filters, and when it came to filtering objects git would communicate with that hook, which in turn would query data with something like the "git cat-file --batch" interface.

I believe you've misunderstood how the GPL works, those third party authors would not be free to license their plugins as they see fit. The reason the LGPL license exists is to allow what you're describing, but git uses the full GPL v2.

See https://en.wikipedia.org/wiki/GPL_linking_exception and
https://www.gnu.org/licenses/gpl-faq.html#LinkingWithGPL

The project you've linked to even has a GPL linking exception of its own (but git.git does not):
https://github.com/koordinates/kart/blob/2934f2b951d61233cbaab9ff627aa3c8cbfb82bc/COPYING#L7-L16

Hi all,

Sorry, life got in the way at an unfortunate moment. And it should very much be tagged "RFC" — thanks Ævar and Bagas for reading. Here's the additional background you could have used earlier on — I've bundled it together, but I'll happily follow up specific questions individually. I've CCed in a couple of other people who might find it interesting too.

So Andrew & my motivation here is to provide some specialised filtering at clone/fetch time. In Kart[1] datasets are organised (simplistically) by primary key, but for spatial data we want to provide an orthogonal spatial extent filter which isn't part of the tree path, so we can't reuse the work done in the sparse filters. For a fetch obviously the server-side will require support for any indexing and ultimately deciding whether a particular blob should be part of the tree or not.

In the original filter implementation [2], various "profiles" were alluded to as a case where the server operator might know a lot more about how the developer would want to use the repository than the client does, and a named profile for the server to interpret would be a reasonably clean approach. Referred again in [3]. Sparse filters, subject to the performance issues hopefully being improved by the cone-mode changes, cater to a lot of them. The existing built-in filters are fairly simple and there's a relatively simple interface for them to implement, extending them seems like a reasonable approach to me — potentially allowing people doing interesting things with partial clones to take it and run in a general way without too much effort.

So the key element to clarify/understand for this proposal is that the main change to Git is the ability to use --filter=extension:<name>[=<param>] which passes through to git-upload-pack on the server side, to rev-list, which looks up / validates the filter name/parameter and applies it. So if you want to offer a custom filter, you build & set it up on the server and any Git client (if this is merged) can make use of it without any additional code.

Wrt IPC, my very first proof of concept used an external process that rev-list launched, passed a series of oids/types via stdin, receiving yes/no responses via stdout. Even after quite a lot of OS-specific efforts to optimise the data flow across the pipes it was slow for non-trivial sized repositories (where it matters) — essentially boiling down to too much context switching between processes. Reorganising the existing filtering approach to do batching with deferred responses and parallelising the filtering into threads seemed like an awful lot of effort for potentially little gain, in a niche use case.

Moving it in-process made it perform well: CPU use moves into the "deciding whether this object is in or out" phase rather than burning it in IPC & context-switching. I did build up a basic runtime-loadable plugin approach, but there was a reasonable amount of the internal git API that the filters need/touched (even things like hash sizes add a pile of complexity to it) unless it was reduced back to passing oids+types. My approach for plugins was basically "could I potentially implement the existing filters?" Without more of the git API I don't think this would be feasible. Plus Git would have to agree on and support a public ABI going forward, which for a potentially niche use case didn't seem reasonable to propose.

Hence compile time: simpler; no ABI issues; the internal API doesn't change that much wrt things that filters are likely to do — if someone creates a plugin then it's on them to keep it building across git upgrades on their server; platform support is simpler; and if others find exciting uses for it then a runtime-loadable plugin API is always possible in future. And only the server ever needs any custom binaries.

Licensing — yes, any filters would need to be GPL-licensed since they're compiled with Git. Only the server operator needs to concern themselves with complying with this (& associated licensing for any external libraries/etc a plugin might need) since that's where the plugin code is linked & runs. With the usual issue around internal use within an organisation not qualifying as "distribution" under the GPL. FWIW, for Kart we'll be GPL-licensing the server-side spatial filter plugin code for anyone who's interested.

Hope this clarifies a bit.

Rob :)

[1] https://kartproject.org — building on Git to version geospatial datasets. Not sure if the videos ever got released (thanks Covid), but
I did a talk at Git Merge 2020 on it when we released the first alpha.
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/