Skip to contentSkip to conversation

attr: convert to new threadsafe API

PATCH In review Stefan Beller 2 participants 9 messages Oct 12, 2016 – Oct 19, 2016 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 attr: convert to new threadsafe API

This revamps the API of the attr subsystem to be thread safe. Before we had the question and its results in one struct type. The typical usage of the API was

    static struct git_attr_check *check;

    if (!check)
        check = git_attr_check_initl("text", NULL);

    git_check_attr(path, check);
    act_on(check->value[0]);

This has a couple of issues when it comes to thread safety:

* the initialization is racy in this implementation. To make it thread safe, we need to acquire a mutex, such that only one thread is executing the code in git_attr_check_initl. As we do not want to introduce a mutex at each call site, this is best done in the attr code. However to do so, we need to have access to the check variable, i.e. the API has to look like

    git_attr_check_initl(struct git_attr_check*, ...);

Then one of the threads calling git_attr_check_initl will acquire the mutex and init the check, while all other threads will wait on the mutex just to realize they're late to the party and they'll return with no work done.

* While the check for attributes to be questioned only need to be initalized once as that part will be read only after its initialisation, the answer may be different for each path. Because of that we need to decouple the check and the answer, such that each thread can obtain an answer for the path it is currently processing.

This commit changes the API and adds locking in git_attr_check_initl that provides the thread safety.

The usage of the new API will be:

    /*
     * The initl call will thread-safely check whether the
     * struct git_attr_check has been initialized. We only
     * want to do the initialization work once, hence we do
     * that work inside a thread safe environment.
     */
    static struct git_attr_check *check;
    git_attr_check_initl(&check, "text", NULL);

    /*
     * Resize internal data structures in the result to match
     * the size of `check`:
     */
    struct git_attr_result *result = GIT_ATTR_RESULT_INIT;
    git_attr_result_init(&result, check);

    /* Perform the check and act on it: */
    git_check_attr(path, check, &result);
    act_on(check->value[0]);

    /*
     * Free the result.
     * The check is static, so doesn't require free'ing
     */
    git_attr_result_clear(&result);

---

* Do not use the Double Check Locking Pattern
* The callers are all thread safe now (i.e. no static result)
* So this is roughly what we want for the first version of the conversion?

Thanks,
Stefan

 Documentation/technical/api-gitattributes.txt |  93 +++++++++++++------
 archive.c                                     |  18 ++--
 attr.c                                        | 125 +++++++++++++++++---------
 attr.h                                        |  80 +++++++++++------
 builtin/check-attr.c                          |  31 ++++---
 builtin/pack-objects.c                        |  20 +++--
 convert.c                                     |  43 +++++----
 ll-merge.c                                    |  28 +++---
 userdiff.c                                    |  24 +++--
 ws.c                                          |  17 ++--
 10 files changed, 314 insertions(+), 165 deletions(-)
Documentation/technical/api-gitattributes.txt +64 -29
index 92fc32a..ac97244 100644
@@ -8,6 +8,18 @@ attributes to set of paths.
 Data Structure
 --------------
 
+extern struct git_attr *git_attr(const char *);
+
+/*
+ * Return the name of the attribute represented by the argument.  The
+ * return value is a pointer to a null-delimited string that is part
+ * of the internal data structure; it should not be modified or freed.
+ */
+extern const char *git_attr_name(const struct git_attr *);
+
+extern int attr_name_valid(const char *name, size_t namelen);
+extern void invalid_attr_name_message(struct strbuf *, const char *, int);
+
 `struct git_attr`::
 
 	An attribute is an opaque object that is identified by its name.
@@ -16,15 +28,17 @@ Data Structure
 	of no interest to the calling programs.  The name of the
 	attribute can be retrieved by calling `git_attr_name()`.
 
-`struct git_attr_check_elem`::
-
-	This structure represents one attribute and its value.
-
 `struct git_attr_check`::
 
-	This structure represents a collection of `git_attr_check_elem`.
+	This structure represents a collection of `struct git_attrs`.
 	It is passed to `git_check_attr()` function, specifying the
-	attributes to check, and receives their values.
+	attributes to check, and receives their values into a corresponding
+	`struct git_attr_result`.
+
+`struct git_attr_result`::
+
+	This structure represents a collection of results to its
+	corresponding `struct git_attr_check`, that has the same order.
 
 
 Attribute Values
@@ -53,19 +67,30 @@ value of the attribute for the path.
 Querying Specific Attributes
 ----------------------------
 
-* Prepare `struct git_attr_check` using git_attr_check_initl()
+* Prepare a `struct git_attr_check` using `git_attr_check_initl()`
   function, enumerating the names of attributes whose values you are
   interested in, terminated with a NULL pointer.  Alternatively, an
-  empty `struct git_attr_check` can be prepared by calling
-  `git_attr_check_alloc()` function and then attributes you want to
-  ask about can be added to it with `git_attr_check_append()`
-  function.
-
-* Call `git_check_attr()` to check the attributes for the path.
-
-* Inspect `git_attr_check` structure to see how each of the
-  attribute in the array is defined for the path.
-
+  empty `struct git_attr_check` as allocated by git_attr_check_alloc()
+  can be prepared by calling `git_attr_check_alloc()` function and
+  then attributes you want to ask about can be added to it with
+  `git_attr_check_append()` function.
+  `git_attr_check_initl()` is thread safe, i.e. you can call it
+  from different threads at the same time; when check determines
+  the initialzisation is still needed, the threads will use a
+  single global mutex to perform the initialization just once, the
+  others will wait on the the thread to actually perform the
+  initialization.
+
+* Prepare a `struct git_attr_result` using `git_attr_result_init()`
+  for the check struct. The call to initialize the result is not thread
+  safe, because different threads need their own thread local result
+  anyway.
+
+* Call `git_check_attr()` to check the attributes for the path,
+  the returned `git_attr_result` contains the result.
+
+* Inspect the returned `git_attr_result` structure to see how
+  each of the attribute in the array is defined for the path.
 
 Example
 -------
@@ -89,15 +114,20 @@ static void setup_check(void)
 
 ------------
 	const char *path;
+	struct git_attr_result *result;
 
 	setup_check();
-	git_check_attr(path, check);
+	result = git_check_attr(path, check);
 ------------
 
-. Act on `.value` member of the result, left in `check->check[]`:
+The `result` must not be free'd as it is owned by the attr subsystem.
+It is reused by the same thread, so a subsequent call to git_check_attr
+in the same thread will overwrite the result.
+
+. Act on `result->value[]`:
 
 ------------
-	const char *value = check->check[0].value;
+	const char *value = result->value[0];
 
 	if (ATTR_TRUE(value)) {
 		The attribute is Set, by listing only the name of the
@@ -123,6 +153,8 @@ the first step in the above would be different.
 static struct git_attr_check *check;
 static void setup_check(const char **argv)
 {
+	if (check)
+		return; /* already done */
 	check = git_attr_check_alloc();
 	while (*argv) {
 		struct git_attr *attr = git_attr(*argv);
@@ -138,17 +170,20 @@ Querying All Attributes
 
 To get the values of all attributes associated with a file:
 
-* Prepare an empty `git_attr_check` structure by calling
-  `git_attr_check_alloc()`.
+* Setup a local variables on the stack for both the question
+  `struct git_attr_check` as well as the result `struct git_attr_result`.
+  Zero them out via their respective _INIT macro.
 
-* Call `git_all_attrs()`, which populates the `git_attr_check`
-  with the attributes attached to the path.
+* Call `git_all_attrs()`
 
-* Iterate over the `git_attr_check.check[]` array to examine
-  the attribute names and values.  The name of the attribute
-  described by a  `git_attr_check.check[]` object can be retrieved via
-  `git_attr_name(check->check[i].attr)`.  (Please note that no items
+* Iterate over the `git_attr_check.attr[]` array to examine the
+  attribute names.  The name of the attribute described by a
+  `git_attr_check.attr[]` object can be retrieved via
+  `git_attr_name(check->attr[i])`.  (Please note that no items
   will be returned for unset attributes, so `ATTR_UNSET()` will return
   false for all returned `git_array_check` objects.)
+  The respective value for an attribute can be found in the same
+  index position in of `git_attr_result`.
 
-* Free the `git_array_check` by calling `git_attr_check_free()`.
+* Clear the local variables by calling `git_attr_check_clear()` and
+  `git_attr_result_clear()`.
archive.c +12 -6
index 11e3951..15849a8 100644
@@ -107,10 +107,12 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		void *context)
 {
 	static struct strbuf path = STRBUF_INIT;
+	static struct git_attr_check *check;
+
 	struct archiver_context *c = context;
 	struct archiver_args *args = c->args;
 	write_archive_entry_fn_t write_entry = c->write_entry;
-	static struct git_attr_check *check;
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
 	const char *path_without_prefix;
 	int err;
 
@@ -124,12 +126,16 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		strbuf_addch(&path, '/');
 	path_without_prefix = path.buf + args->baselen;
 
-	if (!check)
-		check = git_attr_check_initl("export-ignore", "export-subst", NULL);
-	if (!git_check_attr(path_without_prefix, check)) {
-		if (ATTR_TRUE(check->check[0].value))
+	git_attr_check_initl(&check, "export-ignore", "export-subst", NULL);
+	git_attr_result_init(&result, check);
+
+	if (!git_check_attr(path_without_prefix, check, &result)) {
+		if (ATTR_TRUE(result.value[0])) {
+			git_attr_result_clear(&result);
 			return 0;
-		args->convert = ATTR_TRUE(check->check[1].value);
+		}
+		args->convert = ATTR_TRUE(result.value[1]);
+		git_attr_result_clear(&result);
 	}
 
 	if (S_ISDIR(mode) || S_ISGITLINK(mode)) {
attr.c +84 -41

243 lines not on this page · raw message

attr.h +54 -26

104 lines not on this page · raw message

builtin/check-attr.c +20 -11
index ec61476..5e78d5d 100644
@@ -24,13 +24,17 @@ static const struct option check_attr_options[] = {
 	OPT_END()
 };
 
-static void output_attr(struct git_attr_check *check, const char *file)
+static void output_attr(struct git_attr_check *check,
+			struct git_attr_result *result, const char *file)
 {
 	int j;
 	int cnt = check->check_nr;
 
+	if (check->check_nr != result->check_nr)
+		die("BUG: confused check and result internally");
+
 	for (j = 0; j < cnt; j++) {
-		const char *value = check->check[j].value;
+		const char *value = result->value[j];
 
 		if (ATTR_TRUE(value))
 			value = "set";
@@ -44,11 +48,11 @@ static void output_attr(struct git_attr_check *check, const char *file)
 			       "%s%c" /* attrname */
 			       "%s%c" /* attrvalue */,
 			       file, 0,
-			       git_attr_name(check->check[j].attr), 0, value, 0);
+			       git_attr_name(check->attr[j]), 0, value, 0);
 		} else {
 			quote_c_style(file, NULL, stdout, 0);
 			printf(": %s: %s\n",
-			       git_attr_name(check->check[j].attr), value);
+			       git_attr_name(check->attr[j]), value);
 		}
 	}
 }
@@ -59,16 +63,21 @@ static void check_attr(const char *prefix,
 {
 	char *full_path =
 		prefix_path(prefix, prefix ? strlen(prefix) : 0, file);
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
+	struct git_attr_check local_check = GIT_ATTR_CHECK_INIT;
+
 	if (check != NULL) {
-		if (git_check_attr(full_path, check))
-			die("git_check_attr died");
-		output_attr(check, file);
+		git_attr_result_init(&result, check);
+		git_check_attr(full_path, check, &result);
 	} else {
-		check = git_attr_check_alloc();
-		git_all_attrs(full_path, check);
-		output_attr(check, file);
-		git_attr_check_free(check);
+		git_all_attrs(full_path, &local_check, &result);
+		check = &local_check;
 	}
+
+	output_attr(check, &result, file);
+
+	git_attr_result_clear(&result);
+	git_attr_check_clear(&local_check);
 	free(full_path);
 }
 
builtin/pack-objects.c +13 -7
index 3918c07..14b764a 100644
@@ -900,13 +900,19 @@ static int no_try_delta(const char *path)
 {
 	static struct git_attr_check *check;
 
-	if (!check)
-		check = git_attr_check_initl("delta", NULL);
-	if (git_check_attr(path, check))
-		return 0;
-	if (ATTR_FALSE(check->check[0].value))
-		return 1;
-	return 0;
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
+	int ret = 0;
+
+	git_attr_check_initl(&check, "delta", NULL);
+	git_attr_result_init(&result, check);
+
+	if (!git_check_attr(path, check, &result)) {
+		if (ATTR_FALSE(result.value[0]))
+			ret = 1;
+	}
+
+	git_attr_result_clear(&result);
+	return ret;
 }
 
 /*
convert.c +21 -22

91 lines not on this page · raw message

ll-merge.c +18 -10
index bc6479c..3abc77a 100644
@@ -355,6 +355,7 @@ int ll_merge(mmbuffer_t *result_buf,
 {
 	static struct git_attr_check *check;
 	static const struct ll_merge_options default_opts;
+	static struct git_attr_result result;
 	const char *ll_driver_name = NULL;
 	int marker_size = DEFAULT_CONFLICT_MARKER_SIZE;
 	const struct ll_merge_driver *driver;
@@ -368,13 +369,15 @@ int ll_merge(mmbuffer_t *result_buf,
 		normalize_file(theirs, path);
 	}
 
-	if (!check)
-		check = git_attr_check_initl("merge", "conflict-marker-size", NULL);
+	if (!check) {
+		git_attr_check_initl(&check, "merge", "conflict-marker-size", NULL);
+		git_attr_result_init(&result, check);
+	}
 
-	if (!git_check_attr(path, check)) {
-		ll_driver_name = check->check[0].value;
-		if (check->check[1].value) {
-			marker_size = atoi(check->check[1].value);
+	if (!git_check_attr(path, check, &result)) {
+		ll_driver_name = result.value[0];
+		if (result.value[1]) {
+			marker_size = atoi(result.value[1]);
 			if (marker_size <= 0)
 				marker_size = DEFAULT_CONFLICT_MARKER_SIZE;
 		}
@@ -394,14 +397,19 @@ int ll_merge(mmbuffer_t *result_buf,
 int ll_merge_marker_size(const char *path)
 {
 	static struct git_attr_check *check;
+
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
 	int marker_size = DEFAULT_CONFLICT_MARKER_SIZE;
 
-	if (!check)
-		check = git_attr_check_initl("conflict-marker-size", NULL);
-	if (!git_check_attr(path, check) && check->check[0].value) {
-		marker_size = atoi(check->check[0].value);
+	git_attr_check_initl(&check, "conflict-marker-size", NULL);
+	git_attr_result_init(&result, check);
+
+	if (!git_check_attr(path, check, &result) && result.value[0]) {
+		marker_size = atoi(result.value[0]);
 		if (marker_size <= 0)
 			marker_size = DEFAULT_CONFLICT_MARKER_SIZE;
 	}
+	git_attr_result_clear(&result);
+
 	return marker_size;
 }
userdiff.c +17 -7
index 46dfd32..62b1ed6 100644
@@ -264,20 +264,30 @@ struct userdiff_driver *userdiff_find_by_path(const char *path)
 {
 	static struct git_attr_check *check;
 
-	if (!check)
-		check = git_attr_check_initl("diff", NULL);
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
+	const char *value;
+
 	if (!path)
 		return NULL;
-	if (git_check_attr(path, check))
+
+	git_attr_check_initl(&check, "diff", NULL);
+	git_attr_result_init(&result, check);
+
+	if (git_check_attr(path, check, &result)) {
+		git_attr_result_clear(&result);
 		return NULL;
+	}
+
+	value = result.value[0];
+	git_attr_result_clear(&result);
 
-	if (ATTR_TRUE(check->check[0].value))
+	if (ATTR_TRUE(value))
 		return &driver_true;
-	if (ATTR_FALSE(check->check[0].value))
+	if (ATTR_FALSE(value))
 		return &driver_false;
-	if (ATTR_UNSET(check->check[0].value))
+	if (ATTR_UNSET(value))
 		return NULL;
-	return userdiff_find_by_name(check->check[0].value);
+	return userdiff_find_by_name(value);
 }
 
 struct userdiff_driver *userdiff_get_textconv(struct userdiff_driver *driver)
ws.c +11 -6
index bb3270c..5e30fef 100644
@@ -73,15 +73,20 @@ unsigned parse_whitespace_rule(const char *string)
 
 unsigned whitespace_rule(const char *pathname)
 {
-	static struct git_attr_check *attr_whitespace_rule;
+	static struct git_attr_check *check;
 
-	if (!attr_whitespace_rule)
-		attr_whitespace_rule = git_attr_check_initl("whitespace", NULL);
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
+	const char *value;
+	int r;
 
-	if (!git_check_attr(pathname, attr_whitespace_rule)) {
-		const char *value;
+	git_attr_check_initl(&check, "whitespace", NULL);
+	git_attr_result_init(&result, check);
 
-		value = attr_whitespace_rule->check[0].value;
+	r = git_check_attr(pathname, check, &result);
+	value = result.value[0];
+	git_attr_result_clear(&result);
+
+	if (!r) {
 		if (ATTR_TRUE(value)) {
 			/* true (whitespace) */
 			unsigned all_rule = ws_tab_width(whitespace_rule_cfg);
Signature · 1 line
2.10.1.432.g8a36cd8.dirty
#1 lore raw

Documentation/technical/api-gitattributes.txt static void setup_check(void) · line 120

Quoted 9 lines · Stefan Beller
Documentation/technical/api-gitattributes.txt +2 -1
@@ -89,15 +114,20 @@ static void setup_check(void)
 
 ------------
 	const char *path;
+	struct git_attr_result *result;
 
 	setup_check();
-	git_check_attr(path, check);
+	result = git_check_attr(path, check);

This looks stale by a few revisions of the other parts of the patch?

Quoted 28 lines
archive.c +5 -5
index 11e3951..15849a8 100644
@@ -107,10 +107,12 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		void *context)
 {
 	static struct strbuf path = STRBUF_INIT;
+	static struct git_attr_check *check;
+
 	struct archiver_context *c = context;
 	struct archiver_args *args = c->args;
 	write_archive_entry_fn_t write_entry = c->write_entry;
-	static struct git_attr_check *check;
+	struct git_attr_result result = GIT_ATTR_RESULT_INIT;
 	const char *path_without_prefix;
 	int err;
 
@@ -124,12 +126,16 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		strbuf_addch(&path, '/');
 	path_without_prefix = path.buf + args->baselen;
 
-	if (!check)
-		check = git_attr_check_initl("export-ignore", "export-subst", NULL);
-	if (!git_check_attr(path_without_prefix, check)) {
-		if (ATTR_TRUE(check->check[0].value))
+	git_attr_check_initl(&check, "export-ignore", "export-subst", NULL);
+	git_attr_result_init(&result, check);

archive.c static int write_archive_entry(const unsigned char *sha1, const char *base, · line 139

Quoted 10 lines
archive.c +7 -1
+
+	if (!git_check_attr(path_without_prefix, check, &result)) {
+		if (ATTR_TRUE(result.value[0])) {
+			git_attr_result_clear(&result);
 			return 0;
-		args->convert = ATTR_TRUE(check->check[1].value);
+		}
+		args->convert = ATTR_TRUE(result.value[1]);
+		git_attr_result_clear(&result);
 	}

This is exactly what I meant by "can we avoid alloc/free of result in leaf function when we _know_ how many attributes we are interested in already, which is the majority of the case?".

Starting with a simple but unoptimized internal implementation of the attr subsystem is one thing (which is good). Exposing an API that cannot be optimally implemented later without changing the callers is another (which is bad).

By encapsulating each element into "struct git_attr_result", we can extend the API without changing the API user [*1*].

But I do not think of a way to allow an efficient implementation later unless the source of the API user somehow says "this user is only interested in this many attributes", like having this

	struct git_attr_result result[2];

(because this caller only wants "ignore" and "subst") on the API user's side [*2*]. Without such a clue (like the patch above, that only says "there is a structure called 'result'"), I do not think of a way to avoid dynamic allocation on the API implementation side.

All the other callers in the patch (pack-objects, convert, ll-merge, etc.) seem to share the exact same pattern. Each of the leaf functions knows a fixed set of attributes it is interested in, the caller iterates over many paths and makes calls to these leaf functions, and it is a waste to pay alloc/free overhead for each and every iteration when we know how many elements result needs to store.

[Footnote]

*1* Would we need a wrapping struct around the array of results? If

    that is the case, we may need something ugly like this on the
    API user side:

	GIT_ATTR_RESULT_TYPE(2) result = {2,};

    with something like the following on the API implementation
    side:

        #define GIT_ATTR_RESULT_TYPE(n) \
            struct { \
                    int num_slots; \
                    const char *value[n]; \
            }

        struct git_attr_result {
                int num_slots;
                const char *value[FLEX_ARRAY];
        };
        git_attr_result_init(void *result_, struct git_attr_check *check)
        {
                struct git_attr_result *result = result_;

                assert(result->num_slots, check->num_attrs);
                ...
        }                
        git_check_attr(const char *path,
                       struct git_attr_check *check,
                       void *result_)
        {                       
                struct git_attr_result *result = result_;

                assert(result->num_slots, check->num_attrs);
                for (i = 0; i < check_num_attrs; i++)
                        result->value[i] = ... found value ...;
        }

*2* Or the uglier

GIT_ATTR_RESULT_TYPE(2) result = {2,};

I can see why the "check" side would benefit from a structure that contains an array, but I do not see why "result" side would want to, so I am hoping that we won't have to do this uglier variant and just go with the simple "array of resulting values".

#2 lore raw
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Stefan Beller
Documentation/technical/api-gitattributes.txt +1 -0
@@ -89,15 +114,20 @@ static void setup_check(void)
 
 ------------
      const char *path;
+     struct git_attr_result *result;
 
Quoted 5 lines

Documentation/technical/api-gitattributes.txt static void setup_check(void) · line 120

Quoted 3 lines
Documentation/technical/api-gitattributes.txt +1 -1
      setup_check();
-     git_check_attr(path, check);
+     result = git_check_attr(path, check);

This looks stale by a few revisions of the other parts of the patch?

I'll update the documentation.

Quoted 35 lines
Quoted 35 lines
archive.c +9 -6
index 11e3951..15849a8 100644
@@ -107,10 +107,12 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
              void *context)
 {
      static struct strbuf path = STRBUF_INIT;
+     static struct git_attr_check *check;
+
      struct archiver_context *c = context;
      struct archiver_args *args = c->args;
      write_archive_entry_fn_t write_entry = c->write_entry;
-     static struct git_attr_check *check;
+     struct git_attr_result result = GIT_ATTR_RESULT_INIT;
      const char *path_without_prefix;
      int err;
 
@@ -124,12 +126,16 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
              strbuf_addch(&path, '/');
      path_without_prefix = path.buf + args->baselen;
 
-     if (!check)
-             check = git_attr_check_initl("export-ignore", "export-subst", NULL);
-     if (!git_check_attr(path_without_prefix, check)) {
-             if (ATTR_TRUE(check->check[0].value))
+     git_attr_check_initl(&check, "export-ignore", "export-subst", NULL);
+     git_attr_result_init(&result, check);
+
+     if (!git_check_attr(path_without_prefix, check, &result)) {
+             if (ATTR_TRUE(result.value[0])) {
+                     git_attr_result_clear(&result);
                      return 0;
-             args->convert = ATTR_TRUE(check->check[1].value);
Quoted 8 lines

archive.c static int write_archive_entry(const unsigned char *sha1, const char *base, · line 139

Quoted 4 lines
archive.c +3 -0
+             }
+             args->convert = ATTR_TRUE(result.value[1]);
+             git_attr_result_clear(&result);
      }

This is exactly what I meant by "can we avoid alloc/free of result in leaf function when we _know_ how many attributes we are interested in already, which is the majority of the case?".

We can avoid that. For now I settled with an implementation that has no "answer" type, but uses a bare "const char *result[FLEX_ARRAY];", or rather a const char **.

Quoted 8 lines

Starting with a simple but unoptimized internal implementation of the attr subsystem is one thing (which is good). Exposing an API that cannot be optimally implemented later without changing the callers is another (which is bad).

By encapsulating each element into "struct git_attr_result", we can extend the API without changing the API user [*1*].

Oh I see.

So instead of a raw string we want to have

struct git_attr_result {
    const char *value;
};

just to have it extensible. Makes sense. I'll redo that.

Quoted 6 lines

But I do not think of a way to allow an efficient implementation later unless the source of the API user somehow says "this user is only interested in this many attributes", like having this

        struct git_attr_result result[2];
    const char *result[2] = {NULL, NULL};

as of now would be

    struct git_attr_result result[2];

but we'd lose the ability to set them to NULL easily. Probably not needed.

Quoted 6 lines

(because this caller only wants "ignore" and "subst") on the API user's side [*2*]. Without such a clue (like the patch above, that only says "there is a structure called 'result'"), I do not think of a way to avoid dynamic allocation on the API implementation side.

Quoted 8 lines

All the other callers in the patch (pack-objects, convert, ll-merge, etc.) seem to share the exact same pattern. Each of the leaf functions knows a fixed set of attributes it is interested in, the caller iterates over many paths and makes calls to these leaf functions, and it is a waste to pay alloc/free overhead for each and every iteration when we know how many elements result needs to store.

Right.

Quoted 41 lines

[Footnote]

*1* Would we need a wrapping struct around the array of results? If

    that is the case, we may need something ugly like this on the
    API user side:

        GIT_ATTR_RESULT_TYPE(2) result = {2,};

    with something like the following on the API implementation
    side:

        #define GIT_ATTR_RESULT_TYPE(n) \
            struct { \
                    int num_slots; \
                    const char *value[n]; \
            }

        struct git_attr_result {
                int num_slots;
                const char *value[FLEX_ARRAY];
        };
        git_attr_result_init(void *result_, struct git_attr_check *check)
        {
                struct git_attr_result *result = result_;

                assert(result->num_slots, check->num_attrs);
                ...
        }
        git_check_attr(const char *path,
                       struct git_attr_check *check,
                       void *result_)
        {
                struct git_attr_result *result = result_;

                assert(result->num_slots, check->num_attrs);
                for (i = 0; i < check_num_attrs; i++)
                        result->value[i] = ... found value ...;
        }
Quoted 8 lines

*2* Or the uglier

GIT_ATTR_RESULT_TYPE(2) result = {2,};

I can see why the "check" side would benefit from a structure that contains an array, but I do not see why "result" side would want to, so I am hoping that we won't have to do this uglier variant and just go with the simple "array of resulting values".

So I currently have the "array of resulting values", but not wrapped.

Do we expect to get more than the values out of the attr system?

#3 lore raw
Quoted 2 lines · Junio C Hamano

so I am hoping that we won't have to do this uglier variant

---8<--- attr.h:
...

struct git_attr_result {
    int check_nr;
    /* Whether is was statically allocated and cannot be resized. */
    int static_alloc;
    const char *value[FLEX_ARRAY];
};

/* create a pointer pointing to a git_attr_result with a given size: */

#define GIT_ATTR_RESULT_INIT_FOR(name, size) \
    struct { \
        int check_nr; \
        int static_alloc; \
        const char *value[size]; \
    } local_##name; \
    struct git_attr_result *name = \
        (struct git_attr_result *)&(local_##name); \
    local_##name.static_alloc = 1;

...
---8<--- e.g. ws.c:

    static struct git_attr_check *attr_whitespace_rule;
    GIT_ATTR_RESULT_INIT_FOR(result, 1);

    git_attr_check_initl(&attr_whitespace_rule, "whitespace", NULL);

    if (!git_check_attr(pathname, attr_whitespace_rule, result)) {
        if (ATTR_TRUE(result->value[0])) {
            ...
        } else if (ATTR_FALSE(result->value[0])) {
            ...
#4 lore raw
In reply to Junio C Hamano
Quoted 1 line · Junio C Hamano

*1* Would we need a wrapping struct around the array of results?

By the way, I do see a merit on the "check" side (tl;dr: but I do not think "result" needs it, hence I do not see the need for the "ugly" variants).

Take "archive" for example. For each path, it wants to see the attribute "export-ignore" to decide if it is to be omitted. In addition, the usual set of attributes used to smudge blobs into the working tree representation are inspected by the convert.c API as part of its implementation of convert_to_working_tree(). This program has at least two sets of <"check", "result"> that are used by two git_check_attr() callsites that are unaware of each other.

One of the optimizations we discussed is to trim down the attr-stack (which caches the attributes read from .gitattributes files that are in effect for the "last" directory that has the path for which attrbiutes are queried for) by reading/keeping only the entries that affect the attributes the caller is interested in. But when there are multiple callsites that are interested in different sets of attributes, we obviously cannot do such an optimization without taking too much cache-invalidation hit. Because these callsites are not unaware of each other, I do not think we can say "keep the entries that affects the union of all active callsites" very easily, even if it were possible.

But we could tie this cache to "check", which keeps a constant subset of attributes that the caller is interested in (i.e. each callsite would keep its own cache that is useful for its query). While we are single-threaded, "struct git_attr_check" being a wrapping struct around the array of "what attributes are of interest?" is a good place to add that per-check attr-stack cache. When we go multi-threaded, the attr-stack cache must become per-thread, and needs to be moved to per-thread storage, and such a per-thread storage would have multiple attr-stack, one per "check" instance (i.e. looking up the attr-stack may have to say "who/what thread am I?" to first go to the thread-local storage for the current thread, where a table of pointers to attr-stacks is kept and from there, index into that table to find the attr-stack that corresponds to the particular "check"). We could use the address of "check" as the key into this table, but "struct git_attr_check" that wraps the array gives us another option to allocate a small consecutive integer every time initl() creates a new "check" and use it as the index into that attr-stack table, as that integer index can be in the struct that wraps the array of wanted attributes.

Note. none of the above is a suggestion to do the attr caching the way exactly described. The above is primarily to illustrate how a wrapping struct may give us future flexibility without affecting a single line of code in the user of API.

It may turn out that we do not need to have anything other than the array of wanted attributes in the "check" struct, but unlike "result", "check" is shared across threads, and do not have to live directly on the stack, so we can prepare for flexibility.

I do not foresee a similar need for wrapping struct for "result", and given that we do want to keep the option of having them directly on the stack, I am inclined to say we shouldn't introduce one.

If we were still to do the wrapping for result, I would say that basing it around the FLEX_ARRAY idiom, i.e.

Quoted 4 lines
        struct git_attr_result {
                int num_slots;
                const char *value[FLEX_ARRAY];
        };

is a horrible idea. It would be less horrible if it were

	struct git_attr_result {
		int num_slots;
		const char **value;
	};

then make the API user write via a convenience macro something like this

	const char *result_values[NUM_ATTRS_OF_INTEREST];
	struct git_attr_result result = {
		ARRAY_SIZE(result_values), &result_values
	};                

instead. That way, at least the side that implements git_check_attr() would not have to be type-unsafe like the example of ugliness in the message I am following-up on.

#5 lore raw
Quoted 7 lines · Junio C Hamano
Quoted 1 line · Junio C Hamano

*1* Would we need a wrapping struct around the array of results?

By the way, I do see a merit on the "check" side (tl;dr: but I do not think "result" needs it, hence I do not see the need for the "ugly" variants).

So we'd rather go with const char **result instead of our own new struct there. Ok, got it.

Quoted 64 lines

Take "archive" for example. For each path, it wants to see the attribute "export-ignore" to decide if it is to be omitted. In addition, the usual set of attributes used to smudge blobs into the working tree representation are inspected by the convert.c API as part of its implementation of convert_to_working_tree(). This program has at least two sets of <"check", "result"> that are used by two git_check_attr() callsites that are unaware of each other.

One of the optimizations we discussed is to trim down the attr-stack (which caches the attributes read from .gitattributes files that are in effect for the "last" directory that has the path for which attrbiutes are queried for) by reading/keeping only the entries that affect the attributes the caller is interested in. But when there are multiple callsites that are interested in different sets of attributes, we obviously cannot do such an optimization without taking too much cache-invalidation hit. Because these callsites are not unaware of each other, I do not think we can say "keep the entries that affects the union of all active callsites" very easily, even if it were possible.

But we could tie this cache to "check", which keeps a constant subset of attributes that the caller is interested in (i.e. each callsite would keep its own cache that is useful for its query). While we are single-threaded, "struct git_attr_check" being a wrapping struct around the array of "what attributes are of interest?" is a good place to add that per-check attr-stack cache. When we go multi-threaded, the attr-stack cache must become per-thread, and needs to be moved to per-thread storage, and such a per-thread storage would have multiple attr-stack, one per "check" instance (i.e. looking up the attr-stack may have to say "who/what thread am I?" to first go to the thread-local storage for the current thread, where a table of pointers to attr-stacks is kept and from there, index into that table to find the attr-stack that corresponds to the particular "check"). We could use the address of "check" as the key into this table, but "struct git_attr_check" that wraps the array gives us another option to allocate a small consecutive integer every time initl() creates a new "check" and use it as the index into that attr-stack table, as that integer index can be in the struct that wraps the array of wanted attributes.

Note. none of the above is a suggestion to do the attr caching the way exactly described. The above is primarily to illustrate how a wrapping struct may give us future flexibility without affecting a single line of code in the user of API.

It may turn out that we do not need to have anything other than the array of wanted attributes in the "check" struct, but unlike "result", "check" is shared across threads, and do not have to live directly on the stack, so we can prepare for flexibility.

I do not foresee a similar need for wrapping struct for "result", and given that we do want to keep the option of having them directly on the stack, I am inclined to say we shouldn't introduce one.

If we were still to do the wrapping for result, I would say that basing it around the FLEX_ARRAY idiom, i.e.

Quoted 4 lines
        struct git_attr_result {
                int num_slots;
                const char *value[FLEX_ARRAY];
        };
Quoted 6 lines

is a horrible idea. It would be less horrible if it were

        struct git_attr_result {
                int num_slots;
                const char **value;
        };

So const char** but with an additional number of slots, all we do would be to compare this number of slots to the checks number of slots and die("BUG:..."), which is just a burden and no help.

Quoted 4 lines

then make the API user write via a convenience macro something like this

Quoted 8 lines
        const char *result_values[NUM_ATTRS_OF_INTEREST];
        struct git_attr_result result = {
                ARRAY_SIZE(result_values), &result_values
        };

instead. That way, at least the side that implements git_check_attr() would not have to be type-unsafe like the example of ugliness in the message I am following-up on.

Ok I will reroll with the const char** instead of the macro stuff that I came up with,
(that would be type safe though uglier than the pure variant).

#6 lore raw
Quoted 8 lines · Stefan Beller
Quoted 5 lines

By the way, I do see a merit on the "check" side (tl;dr: but I do not think "result" needs it, hence I do not see the need for the "ugly" variants).

So we'd rather go with const char **result instead of our own new struct there. Ok, got it.

I do not think you got it. I am talking about wrapping struct around an array of element,
not each element in the array. IOW

Quoted 7 lines
Quoted 7 lines

If we were still to do the wrapping for result, I would say that basing it around the FLEX_ARRAY idiom, i.e.

Quoted 4 lines
        struct git_attr_result {
                int num_slots;
                const char *value[FLEX_ARRAY];
        };

the structure around the array of elements (value) that allows us to have something other than value[] in it. That is what I said "I do not see the need for".

It is perfectly fine future-proofing to have

struct git_attr_result_value {
const char *value;
};

and have the users of API declare

struct git_attr_result value[5];

or whatever. That way we could fatten the structure later if we wanted to without having to update the users of API, and there is no downside.

Having wrapping strut around the array does have a huge downside, and that is what I said I see no need for.

#7 lore raw
Quoted 34 lines · Junio C Hamano
Quoted 8 lines · Stefan Beller
Quoted 5 lines

By the way, I do see a merit on the "check" side (tl;dr: but I do not think "result" needs it, hence I do not see the need for the "ugly" variants).

So we'd rather go with const char **result instead of our own new struct there. Ok, got it.

I do not think you got it. I am talking about wrapping struct around an array of element,
not each element in the array. IOW

Quoted 7 lines
Quoted 7 lines

If we were still to do the wrapping for result, I would say that basing it around the FLEX_ARRAY idiom, i.e.

Quoted 4 lines
        struct git_attr_result {
                int num_slots;
                const char *value[FLEX_ARRAY];
        };

the structure around the array of elements (value) that allows us to have something other than value[] in it. That is what I said "I do not see the need for".

It is perfectly fine future-proofing to have

struct git_attr_result_value {
const char *value;
};

and have the users of API declare

Quoted 7 lines

struct git_attr_result value[5];

or whatever. That way we could fatten the structure later if we wanted to without having to update the users of API, and there is no downside.

Having wrapping strut around the array does have a huge downside, and that is what I said I see no need for.

I am not sure if I see the upside on wrapping a single value except for its future proofness, i.e. what if we want to transport information that is valid for all values, e.g. an error code or that the result check was done lazily (for non lazy you would need to do X) or ...

IOW I would expect there to be more use cases for information regarding all the values and not each value enhanced by a thing.

We could just repeat the thing in each of the 5 values in

Quoted 1 line

struct git_attr_result value[5];

though.

I'll go with that.

#8 lore raw
Quoted 2 lines · Stefan Beller

I am not sure if I see the upside on wrapping a single value except for its future proofness,

I do not see anything other than future-proofing, either. If we need to touch all the code that uses the attributes to update the API, I'd prefer to avoid having to do that again in the future.