cat-file: reuse ref-filter logic
cover cat-file: reuse ref-filter logic
This patch series makes cat-file reuse ref-filter logic. At the same time, some performance optimizations have been carried out. It's last version is here:
https://lore.kernel.org/git/[email protected]/#t
It seems that zh/ref-filter-raw-data is still hovering in the next branch (Because git is rc2) So I now want to show some recent performance optimizations first.
Change from last version:
1. Use free_global_resource() to avoid memory leaks.
2. Skip parse_object_buffer() which bring 12.5% performance optimization.
3. Merge two for loop in grab_person() which bring 2% performance
optimization.
4. Remove strlen from find_subpos.
5. Introducing xstrvfmt_len() and xstrfmt_len().
6. Remove second parsing in format_ref_array_item() which bring 1.9% performance optimization
7. Introduction ref_filter_slopbuf to instread xstrdup("").
8. Add deref member to struct used_atom to simplify the logic of the program.
9. Introduce symref_atom_parser() to make the program logic more concise.
10. Use switch/case instread of if/else to increase the readability of the code.
11. Reuse finnal buffer which bring 2% performance optimization.
12. Add need_get_object_info flag to reduce memory comparing.
This is the result of the performance test after I did some optimization:
Test upstream/master this tree
------------------------------------------------------------------------------------
1006.2: cat-file --batch-check 0.08(0.07+0.00) 0.09(0.08+0.01) +12.5% 1006.3: cat-file --batch-check with atoms 0.06(0.04+0.02) 0.08(0.06+0.02) +33.3% 1006.4: cat-file --batch 0.49(0.46+0.02) 0.50(0.47+0.03) +2.0% 1006.5: cat-file --batch with atoms 0.47(0.45+0.01) 0.49(0.47+0.02) +4.3%
We can see that the performance of the current patch of git cat-file --batch is very close to upstream/master. The optimization of git cat-file --batch-check does not seem obvious, because its optimization degree will be affected by noise, which may appear in the range of +12.5% to +50.0%. From an optimistic point of view, the execution time of git cat-file --batch-check itself is relatively short, the optimization is of course not obvious.
As GSOC is about to end, this patch series is estimated to be adjusted for some time, I can only wish this patch can be accepted in the future.
Note: The previous part of this patch series is the duplicate content belonging to zh/ref-filter-raw-data.
ZheNing Hu (27):
[GSOC] ref-filter: add obj-type check in grab contents [GSOC] ref-filter: add %(raw) atom
[GSOC] ref-filter: --format=%(raw) support --perl
[GSOC] ref-filter: use non-const ref_format in *_atom_parser() [GSOC] ref-filter: add %(rest) atom
[GSOC] ref-filter: pass get_object() return value to their callers [GSOC] ref-filter: introduce free_ref_array_item_value() function [GSOC] ref-filter: add cat_file_mode to ref_format
[GSOC] ref-filter: modify the error message and value in get_object [GSOC] cat-file: add has_object_file() check
[GSOC] cat-file: change batch_objects parameter name [GSOC] cat-file: create p1006-cat-file.sh
[GSOC] cat-file: reuse ref-filter logic
[GSOC] cat-file: reuse err buf in batch_object_write() [GSOC] cat-file: re-implement --textconv, --filters options [GSOC] ref-filter: remove grab_oid() function
[GSOC] ref-filter: performance optimization by skip
parse_object_buffer
[GSOC] ref-filter: use atom_type and merge two for loop in grab_person [GSOC] ref-filter: remove strlen from find_subpos
[GSOC] ref-filter: introducing xstrvfmt_len() and xstrfmt_len() [GSOC] ref-filter: remove second parsing in format_ref_array_item [GSOC] ref-filter: introduction ref_filter_slopbuf
[GSOC] ref-filter: add deref member to struct used_atom [GSOC] ref-filter: introduce symref_atom_parser()
[GSOC] ref-filter: use switch case instread of if else [GSOC] ref-filter: reuse finnal buffer if no stack need [GSOC] ref-filter: add need_get_object_info flag to struct expand_data
Documentation/git-cat-file.txt | 6 + Documentation/git-for-each-ref.txt | 9 + builtin/branch.c | 2 + builtin/cat-file.c | 275 +++------ builtin/for-each-ref.c | 3 +- builtin/tag.c | 4 +- builtin/verify-tag.c | 2 + quote.c | 17 + quote.h | 1 + ref-filter.c | 902 +++++++++++++++++++---------- ref-filter.h | 30 +- strbuf.c | 21 + strbuf.h | 6 + t/perf/p1006-cat-file.sh | 28 + t/t1006-cat-file.sh | 239 ++++++++ t/t3203-branch-output.sh | 4 + t/t6300-for-each-ref.sh | 235 ++++++++ t/t6301-for-each-ref-errors.sh | 2 +- t/t7004-tag.sh | 4 + t/t7030-verify-tag.sh | 4 + 20 files changed, 1283 insertions(+), 511 deletions(-) create mode 100755 t/perf/p1006-cat-file.sh
base-commit: 5d213e46bb7b880238ff5ea3914e940a50ae9369
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1016%2Fadlternative%2Fcat-file-reuse-ref-filter-logic-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1016/adlternative/cat-file-reuse-ref-filter-logic-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1016
Signature · 1 line
gitgitgadget
PATCH 2/27 ref-filter: add %(raw) atom
From: ZheNing Hu <[email protected]>
Add new formatting option %(raw), which will print the raw object data without any changes. It will help further to migrate all cat-file formatting logic from cat-file to ref-filter.
The raw data of blob, tree objects may contain '\0', but most of the logic in ref-filter depends on the output of the atom being text (specifically, no embedded NULs in it).
E.g. quote_formatting() use strbuf_addstr() or *._quote_buf() add the data to the buffer. The raw data of a tree object is 100644 one\0..., only the 100644 one will be added to the buffer, which is incorrect.
Therefore, we need to find a way to record the length of the atom_value's member s. Although strbuf can already record the string and its length, if we want to replace the type of atom_value's member s with strbuf, many places in ref-filter that are filled with dynamically allocated mermory in v->s are not easy to replace. At the same time, we need to check if v->s == NULL in populate_value(), and strbuf cannot easily distinguish NULL and empty strings, but c-style "const char *" can do it. So add a new member in struct atom_value: s_size, which can record raw object size, it can help us add raw object data to the buffer or compare two buffers which contain raw object data.
Note that --format=%(raw) cannot be used with --python, --shell, --tcl, and --perl because if the binary raw data is passed to a variable in such languages, these may not support arbitrary binary data in their string variable type.
--- Documentation/git-for-each-ref.txt | 9 ++ ref-filter.c | 143 +++++++++++++++---- t/t6300-for-each-ref.sh | 216 +++++++++++++++++++++++++++++ 3 files changed, 340 insertions(+), 28 deletions(-)
Documentation/git-for-each-ref.txt +9 -0
@@ -235,6 +235,15 @@ and `date` to extract the named component. For email fields (`authoremail`, without angle brackets, and `:localpart` to get the part before the `@` symbol out of the trimmed email. +The raw data in an object is `raw`. + +raw:size:: + The raw data size of the object. + +Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`, +`--perl` because such language may not support arbitrary binary data in their +string variable type. + The message in a commit or a tag object is `contents`, from which `contents:<part>` can be used to extract various parts out of:
ref-filter.c +115 -28
292 lines not on this page · raw message
t/t6300-for-each-ref.sh +216 -0
258 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH 1/27 ref-filter: add obj-type check in grab contents
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Only tag and commit objects use grab_sub_body_contents() to grab object contents in the current codebase. We want to teach the function to also handle blobs and trees to get their raw data, without parsing a blob (whose contents looks like a commit or a tag) incorrectly as a commit or a tag. So it's needed to pass a struct expand_data *data instread of only void *buf to both grab_sub_body_contents() and grab_values() to be able to check the object type.
Skip the block of code that is specific to handling commits and tags early when the given object is of a wrong type to help later addition to handle other types of objects in this function.
--- ref-filter.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
ref-filter.c +15 -9
@@ -1357,11 +1357,12 @@ static void append_lines(struct strbuf *out, const char *buf, unsigned long size }/* See grab_values */ -static void grab_sub_body_contents (struct atom_value *val,int deref,void *buf) +static void grab_sub_body_contents (struct atom_value *val,int deref,struct expand_data *data) {int i;const char *subpos =NULL , *bodypos =NULL , *sigpos =NULL ;size_t sublen =0 , bodylen =0 , nonsiglen =0 , siglen =0 ; +void *buf = data->content;for (i =0 ; i < used_atom_cnt; i++) {struct used_atom *atom = &used_atom[i];
@@ -1372,10 +1373,13 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, void *buf)continue ;if (deref) name++; -if (strcmp (name,"body" ) && - !starts_with (name,"subject" ) && - !starts_with (name,"trailers" ) && - !starts_with (name,"contents" )) + +if ((data->type !=OBJ_TAG && + data->type !=OBJ_COMMIT ) || + (strcmp (name,"body" ) && + !starts_with (name,"subject" ) && + !starts_with (name,"trailers" ) && + !starts_with (name,"contents" )))continue ;if (!subpos) find_subpos(buf,
@@ -1439,17 +1443,19 @@ static void fill_missing_values(struct atom_value *val)* pointed at by the ref itself; otherwise it is the object the * ref (which is a tag) refers to. */ -static void grab_values (struct atom_value *val,int deref,struct object *obj,void *buf) +static void grab_values (struct atom_value *val,int deref,struct object *obj,struct expand_data *data) { +void *buf = data->content; +switch (obj->type) {case OBJ_TAG :grab_tag_values (val, deref, obj); -grab_sub_body_contents (val, deref, buf); +grab_sub_body_contents (val, deref, data);grab_person ("tagger" , val, deref, buf);break ;case OBJ_COMMIT :grab_commit_values (val, deref, obj); -grab_sub_body_contents (val, deref, buf); +grab_sub_body_contents (val, deref, data);grab_person ("author" , val, deref, buf);grab_person ("committer" , val, deref, buf);break ;
@@ -1679,7 +1685,7 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **objreturn strbuf_addf_ret (err,-1 ,_ ("parse_object_buffer failed on %s for %s" ),oid_to_hex (&oi->oid), ref->refname); } -grab_values (ref->value, deref, *obj, oi->content); +grab_values (ref->value, deref, *obj, oi); }grab_common_values (ref->value, deref, oi);
Signature · 1 line
gitgitgadget
PATCH 3/27 ref-filter: --format=%(raw) support --perl
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Because the perl language can handle binary data correctly, add the function perl_quote_buf_with_len(), which can specify the length of the data and prevent the data from being truncated at '\0' to help --format="%(raw)" support --perl.
--- Documentation/git-for-each-ref.txt | 4 ++-- quote.c | 17 +++++++++++++++++ quote.h | 1 + ref-filter.c | 15 +++++++++++---- t/t6300-for-each-ref.sh | 19 +++++++++++++++++-- 5 files changed, 48 insertions(+), 8 deletions(-)
Documentation/git-for-each-ref.txt +2 -2
@@ -241,8 +241,8 @@ raw:size:: The raw data size of the object. Note that `--format=%(raw)` can not be used with `--python`, `--shell`, `--tcl`, -`--perl` because such language may not support arbitrary binary data in their -string variable type. +because such language may not support arbitrary binary data in their string +variable type. The message in a commit or a tag object is `contents`, from which `contents:<part>` can be used to extract various parts out of:
quote.c +17 -0
@@ -471,6 +471,23 @@ void perl_quote_buf(struct strbuf *sb, const char *src)strbuf_addch (sb, sq); } +void perl_quote_buf_with_len (struct strbuf *sb,const char *src,size_t len) +{ +const char sq ='\'' ; +const char bq ='\\' ; +const char *c = src; +const char *end = src + len; + +strbuf_addch (sb, sq); +while (c != end) { +if (*c == sq || *c == bq) +strbuf_addch (sb, bq); +strbuf_addch (sb, *c); + c++; + } +strbuf_addch (sb, sq); +} +void python_quote_buf (struct strbuf *sb,const char *src) {const char sq ='\'' ;
quote.h +1 -0
@@ -95,6 +95,7 @@ char *quote_path(const char *in, const char *prefix, struct strbuf *out, unsigne/* quoting as a string literal for other languages */ void perl_quote_buf (struct strbuf *sb,const char *src); +void perl_quote_buf_with_len (struct strbuf *sb,const char *src,size_t len);void python_quote_buf (struct strbuf *sb,const char *src);void tcl_quote_buf (struct strbuf *sb,const char *src);void basic_regex_quote_buf (struct strbuf *sb,const char *src);
ref-filter.c +11 -4
@@ -747,7 +747,10 @@ static void quote_formatting(struct strbuf *s, const char *str, ssize_t len, intsq_quote_buf (s, str);break ;case QUOTE_PERL : -perl_quote_buf (s, str); +if (len <0 ) +perl_quote_buf (s, str); +else +perl_quote_buf_with_len (s, str, len);break ;case QUOTE_PYTHON :python_quote_buf (s, str);
@@ -1010,10 +1013,14 @@ int verify_ref_format(struct ref_format *format) at =parse_ref_filter_atom (format, sp +2 , ep, &err);if (at <0 )die ("%s" , err.buf); -if (format->quote_style && used_atom[at].atom_type ==ATOM_RAW && - used_atom[at].u.raw_data.option ==RAW_BARE ) + +if ((format->quote_style ==QUOTE_PYTHON || + format->quote_style ==QUOTE_SHELL || + format->quote_style ==QUOTE_TCL ) && + used_atom[at].atom_type ==ATOM_RAW && + used_atom[at].u.raw_data.option ==RAW_BARE )die (_ ("--format=%.*s cannot be used with" -"--python, --shell, --tcl, --perl" ), (int )(ep - sp -2 ), sp +2 ); +"--python, --shell, --tcl" ), (int )(ep - sp -2 ), sp +2 ); cp = ep +1 ;if (skip_prefix (used_atom[at].name,"color:" , &color))
t/t6300-for-each-ref.sh +17 -2
@@ -915,8 +915,23 @@ test_expect_success '%(raw) with --tcl must fail' ' test_must_fail git for-each-ref --format="%(raw)" --tcl ' -test_expect_success '%(raw) with --perl must fail' ' - test_must_fail git for-each-ref --format="%(raw)" --perl +test_expect_success '%(raw) with --perl' ' + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/blob1 --perl | perl >actual && + cmp blob1 actual && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/blob3 --perl | perl >actual && + cmp blob3 actual && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/blob8 --perl | perl >actual && + cmp blob8 actual && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/myblobs/first --perl | perl >actual && + cmp one actual && + git cat-file tree refs/mytrees/first > expected && + git for-each-ref --format="\$name= %(raw); +print \"\$name\"" refs/mytrees/first --perl | perl >actual && + cmp expected actual ' test_expect_success '%(raw) with --shell must fail' '
Signature · 1 line
gitgitgadget
PATCH 4/27 ref-filter: use non-const ref_format in *_atom_parser()
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Use non-const ref_format in *_atom_parser(), which can help us modify the members of ref_format in *_atom_parser().
--- builtin/tag.c | 2 +- ref-filter.c | 44 ++++++++++++++++++++++---------------------- ref-filter.h | 4 ++-- 3 files changed, 25 insertions(+), 25 deletions(-)
builtin/tag.c +1 -1
@@ -146,7 +146,7 @@ static int verify_tag(const char *name, const char *ref, const struct object_id *oid, void *cb_data) { int flags; -const struct ref_format *format = cb_data; +struct ref_format *format = cb_data; flags =GPG_VERIFY_VERBOSE ;if (format->format)
ref-filter.c +22 -22
164 lines not on this page · raw message
ref-filter.h +2 -2
@@ -116,7 +116,7 @@ void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);void ref_sorting_set_sort_flags_all (struct ref_sorting *sorting,unsigned int mask,int on);/* Based on the given format and quote_style, fill the strbuf */ int format_ref_array_item (struct ref_array_item *info, -const struct ref_format *format, +struct ref_format *format,struct strbuf *final_buf,struct strbuf *error_buf);/* Parse a single sort specifier and add it to the list */
@@ -137,7 +137,7 @@ void setup_ref_filter_porcelain_msg(void);* name must be a fully qualified refname. */ void pretty_print_ref (const char *name,const struct object_id *oid, -const struct ref_format *format); +struct ref_format *format);/* * Push a single ref onto the array; this can be used to construct your own
Signature · 1 line
gitgitgadget
PATCH 7/27 ref-filter: introduce free_ref_array_item_value() function
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
When we use ref_array_item which is not dynamically allocated and want to free the space of its member "value" after the end of use, free_array_item() does not meet our needs, because it tries to free ref_array_item itself and its member "symref".
Introduce free_ref_array_item_value() for freeing ref_array_item value. It will be called internally by free_array_item(), and it will help cat-file --batch free ref_array_item's value memory later.
--- ref-filter.c | 11 ++++++++--- ref-filter.h | 2 ++ 2 files changed, 10 insertions(+), 3 deletions(-)
ref-filter.c +8 -3
@@ -2300,16 +2300,21 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,return 0 ; } -/* Free memory allocated for a ref_array_item */ -static void free_array_item (struct ref_array_item *item) +void free_ref_array_item_value (struct ref_array_item *item) { -free ((char *)item->symref);if (item->value) {int i;for (i =0 ; i < used_atom_cnt; i++)free ((char *)item->value[i].s);free (item->value); } +} + +/* Free memory allocated for a ref_array_item */ +static void free_array_item (struct ref_array_item *item) +{ +free ((char *)item->symref); +free_ref_array_item_value (item);free (item); }
ref-filter.h +2 -0
@@ -111,6 +111,8 @@ struct ref_format {int filter_refs (struct ref_array *array,struct ref_filter *filter,unsigned int type);/* Clear all memory allocated to ref_array */ void ref_array_clear (struct ref_array *array); +/* Free ref_array_item's value */ +void free_ref_array_item_value (struct ref_array_item *item);/* Used to verify if the given format is correct and to parse out the used atoms */ int verify_ref_format (struct ref_format *format);/* Sort the given ref_array as per the ref_sorting provided */
Signature · 1 line
gitgitgadget
PATCH 5/27 ref-filter: add %(rest) atom
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
%(rest) is a atom used for cat-file batch mode, which can split the input lines at the first whitespace boundary, all characters before that whitespace are considered to be the object name; characters after that first run of whitespace (i.e., the "rest" of the line) are output in place of the %(rest) atom.
In order to let "cat-file --batch=%(rest)" use the ref-filter interface, add %(rest) atom for ref-filter.
Introduce the reject_atom() to reject the atom %(rest) for "git for-each-ref", "git branch", "git tag" and "git verify-tag".
--- ref-filter.c | 25 +++++++++++++++++++++++++ ref-filter.h | 5 ++++- t/t3203-branch-output.sh | 4 ++++ t/t6300-for-each-ref.sh | 4 ++++ t/t7004-tag.sh | 4 ++++ t/t7030-verify-tag.sh | 4 ++++ 6 files changed, 45 insertions(+), 1 deletion(-)
ref-filter.c +25 -0
@@ -157,6 +157,7 @@ enum atom_type {ATOM_IF ,ATOM_THEN ,ATOM_ELSE , + ATOM_REST, };/*
@@ -560,6 +561,15 @@ static int if_atom_parser(struct ref_format *format, struct used_atom *atom,return 0 ; } +static int rest_atom_parser (struct ref_format *format,struct used_atom *atom, +const char *arg,struct strbuf *err) +{ +if (arg) +return strbuf_addf_ret (err,-1 ,_ ("%%(rest) does not take arguments" )); + format->use_rest =1 ; +return 0 ; +} +static int head_atom_parser (struct ref_format *format,struct used_atom *atom,const char *arg,struct strbuf *unused_err) {
@@ -616,6 +626,7 @@ static struct { [ATOM_IF] = { "if", SOURCE_NONE, FIELD_STR, if_atom_parser }, [ATOM_THEN] = { "then", SOURCE_NONE }, [ATOM_ELSE] = { "else", SOURCE_NONE }, + [ATOM_REST] = { "rest", SOURCE_NONE, FIELD_STR, rest_atom_parser },/* * Please update $__git_ref_fieldlist in git-completion.bash * when you add new atoms
@@ -993,6 +1004,11 @@ static const char *find_next(const char *cp)return NULL ; } +static int reject_atom (enum atom_type atom_type) +{ +return atom_type ==ATOM_REST ; +} +/* * Make sure the format string is well formed, and parse out * the used atoms.
@@ -1013,6 +1029,8 @@ int verify_ref_format(struct ref_format *format) at =parse_ref_filter_atom (format, sp +2 , ep, &err);if (at <0 )die ("%s" , err.buf); +if (reject_atom (used_atom[at].atom_type)) +die (_ ("this command reject atom %%(%.*s)" ), (int )(ep - sp -2 ), sp +2 ); if ((format->quote_style == QUOTE_PYTHON || format->quote_style == QUOTE_SHELL ||
@@ -1932,6 +1950,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) v->handler = else_atom_handler; v->s =xstrdup ("" );continue ; + }else if (atom_type ==ATOM_REST ) { +if (ref->rest) + v->s =xstrdup (ref->rest); +else + v->s =xstrdup ("" ); +continue ; }else continue ;
@@ -2149,6 +2173,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,FLEX_ALLOC_STR (ref, refname, refname);oidcpy (&ref->objectname, oid); + ref->rest =NULL ;return ref; }
ref-filter.h +4 -1
@@ -38,6 +38,7 @@ struct ref_sorting {struct ref_array_item {struct object_id objectname; +const char *rest;int flag;unsigned int kind;const char *symref;
@@ -76,14 +77,16 @@ struct ref_format {* verify_ref_format() afterwards to finalize. */ const char *format; +const char *rest;int quote_style; +int use_rest;int use_color;/* Internal state to ref-filter */ int need_color_reset_at_eol; }; -#define REF_FORMAT_INIT { NULL, 0, -1 } +#define REF_FORMAT_INIT { .use_color = -1 }/* Macros for checking --merged and --no-merged options */ #define _OPT_MERGED_NO_MERGED (option, filter, h) \
t/t3203-branch-output.sh +4 -0
@@ -340,6 +340,10 @@ test_expect_success 'git branch --format option' ' test_cmp expect actual ' +test_expect_success 'git branch with --format=%(rest) must fail' ' + test_must_fail git branch --format="%(rest)" >actual +' + test_expect_success 'worktree colors correct' ' cat >expect <<-EOF && * <GREEN>(HEAD detached from fromtag)<RESET>
t/t6300-for-each-ref.sh +4 -0
@@ -1211,6 +1211,10 @@ test_expect_success 'basic atom: head contents:trailers' ' test_cmp expect actual.clean ' +test_expect_success 'basic atom: rest must fail' ' + test_must_fail git for-each-ref --format="%(rest)" refs/heads/main +' + test_expect_success 'trailer parsing not fooled by --- line' ' git commit --allow-empty -F - <<-\EOF && this is the subject
t/t7004-tag.sh +4 -0
@@ -1998,6 +1998,10 @@ test_expect_success '--format should list tags as per format given' ' test_cmp expect actual ' +test_expect_success 'git tag -l with --format="%(rest)" must fail' ' + test_must_fail git tag -l --format="%(rest)" "v1*" +' + test_expect_success "set up color tests" ' echo "<RED>v1.0<RESET>" >expect.color && echo "v1.0" >expect.bare &&
t/t7030-verify-tag.sh +4 -0
@@ -194,6 +194,10 @@ test_expect_success GPG 'verifying tag with --format' ' test_cmp expect actual ' +test_expect_success GPG 'verifying tag with --format="%(rest)" must fail' ' + test_must_fail git verify-tag --format="%(rest)" "fourth-signed" +' + test_expect_success GPG 'verifying a forged tag with --format should fail silently' ' test_must_fail git verify-tag --format="tagname : %(tag)" $(cat forged1.tag) >actual-forged && test_must_be_empty actual-forged
Signature · 1 line
gitgitgadget
PATCH 6/27 ref-filter: pass get_object() return value to their callers
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Because in the refactor of git cat-file --batch later, oid_object_info_extended() in get_object() will be used to obtain the info of an object with it's oid. When the object cannot be obtained in the git repository, cat-file --batch expects to output "<oid> missing" and continue the next oid query instead of letting Git exit. In other error conditions, Git should exit normally. So we can achieve this function by passing the return value of get_object().
--- ref-filter.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
ref-filter.c +11 -6
@@ -1825,6 +1825,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) {struct object *obj;int i; +int ret;struct object_info empty =OBJECT_INFO_INIT ;CALLOC_ARRAY (ref->value, used_atom_cnt);
@@ -1981,8 +1982,9 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) oi.oid = ref->objectname; -if (get_object (ref,0 , &obj, &oi, err)) -return -1 ; + ret =get_object (ref,0 , &obj, &oi, err); +if (ret) +return ret;/* * If there is no atom that wants to know about tagged
@@ -2014,8 +2016,10 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom, struct atom_value **v, struct strbuf *err) {if (!ref->value) { -if (populate_value (ref, err)) -return -1 ; +int ret =populate_value (ref, err); + +if (ret) +return ret;fill_missing_values (ref->value); } *v = &ref->value[atom];
@@ -2592,6 +2596,7 @@ int format_ref_array_item(struct ref_array_item *info, {const char *cp, *sp, *ep;struct ref_formatting_state state =REF_FORMATTING_STATE_INIT ; +int ret; state.quote_style = format->quote_style;push_stack_element (&state.stack);
@@ -2604,10 +2609,10 @@ int format_ref_array_item(struct ref_array_item *info,if (cp < sp)append_literal (cp, sp, &state); pos =parse_ref_filter_atom (format, sp +2 , ep, error_buf); -if (pos <0 ||get_ref_atom_value (info, pos, &atomv, error_buf) || +if (pos <0 || (ret =get_ref_atom_value (info, pos, &atomv, error_buf)) || atomv->handler (atomv, &state, error_buf)) {pop_stack_element (&state.stack); -return -1 ; +return ret ? ret :-1 ; } }if (*cp) {
Signature · 1 line
gitgitgadget
PATCH 8/27 ref-filter: add cat_file_mode to ref_format
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Add cat_file_mode member to struct ref_format. When cat-file --batch use ref-filter logic later, it can help us reject atoms in verify_ref_format() which cat-file cannot use, e.g. %(refname), %(push), `%(upstream)"...
Add batch_test_atom() to t/t1006-cat-file.sh and add check for cat-file --batch, this can help us clearly show which atoms cat-file accepts and which atoms it rejects.
--- ref-filter.c | 23 ++++- ref-filter.h | 1 + t/t1006-cat-file.sh | 226 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 246 insertions(+), 4 deletions(-)
ref-filter.c +19 -4
@@ -1004,9 +1004,24 @@ static const char *find_next(const char *cp)return NULL ; } -static int reject_atom (enum atom_type atom_type) -{ -return atom_type ==ATOM_REST ; +static int reject_atom (int cat_file_mode,enum atom_type atom_type) +{ +if (!cat_file_mode) +return atom_type ==ATOM_REST ; + +/* cat_file_mode */ +switch (atom_type) { +case ATOM_FLAG : +case ATOM_HEAD : +case ATOM_PUSH : +case ATOM_REFNAME : +case ATOM_SYMREF : +case ATOM_UPSTREAM : +case ATOM_WORKTREEPATH : +return 1 ; +default : +return 0 ; + } }/*
@@ -1029,7 +1044,7 @@ int verify_ref_format(struct ref_format *format) at =parse_ref_filter_atom (format, sp +2 , ep, &err);if (at <0 )die ("%s" , err.buf); -if (reject_atom (used_atom[at].atom_type)) +if (reject_atom (format->cat_file_mode, used_atom[at].atom_type))die (_ ("this command reject atom %%(%.*s)" ), (int )(ep - sp -2 ), sp +2 ); if ((format->quote_style == QUOTE_PYTHON ||
ref-filter.h +1 -0
@@ -78,6 +78,7 @@ struct ref_format {*/ const char *format;const char *rest; +int cat_file_mode;int quote_style;int use_rest;int use_color;
t/t1006-cat-file.sh +226 -0
230 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH 9/27 ref-filter: modify the error message and value in get_object
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Let get_object() return 1 and print "<oid> missing" instead of returning -1 and printing "missing object <oid> for <refname>" if oid_object_info_extended() unable to find the data corresponding to oid. When cat-file --batch use ref-filter logic later it can help format_ref_array_item() just report that the object is missing without letting Git exit.
--- ref-filter.c | 4 ++-- t/t6301-for-each-ref-errors.sh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
ref-filter.c +2 -2
@@ -1766,8 +1766,8 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj }if (oid_object_info_extended (the_repository, &oi->oid, &oi->info,OBJECT_INFO_LOOKUP_REPLACE )) -return strbuf_addf_ret (err,-1 ,_ ("missing object %s for %s" ), -oid_to_hex (&oi->oid), ref->refname); +return strbuf_addf_ret (err,1 ,_ ("%s missing" ), +oid_to_hex (&oi->oid));if (oi->info.disk_sizep && oi->disk_size <0 )BUG ("Object size is less than zero." );
t/t6301-for-each-ref-errors.sh +1 -1
@@ -41,7 +41,7 @@ test_expect_success 'Missing objects are reported correctly' ' r=refs/heads/missing && echo $MISSING >.git/$r && test_when_finished "rm -f .git/$r" && - echo "fatal: missing object $MISSING for $r" >missing-err && + echo "fatal: $MISSING missing" >missing-err && test_must_fail git for-each-ref 2>err && test_cmp missing-err err && (
Signature · 1 line
gitgitgadget
PATCH 10/27 cat-file: add has_object_file() check
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Use has_object_file() in batch_one_object() to check whether the input object exists. This can help us reject the missing oid when we let cat-file --batch use ref-filter logic later.
--- builtin/cat-file.c | 7 +++++++ 1 file changed, 7 insertions(+)
builtin/cat-file.c +7 -0
@@ -428,6 +428,13 @@ static void batch_one_object(const char *obj_name,return ; } +if (!has_object_file (&data->oid)) { +printf ("%s missing\n" , + obj_name ? obj_name :oid_to_hex (&data->oid)); +fflush (stdout); +return ; + } +batch_object_write (obj_name, scratch, opt, data); }
Signature · 1 line
gitgitgadget
PATCH 11/27 cat-file: change batch_objects parameter name
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Because later cat-file reuses ref-filter logic that will add parameter "const struct option *options" to batch_objects(), the two synonymous parameters of "opt" and "options" may confuse readers, so change batch_options parameter of batch_objects() from "opt" to "batch".
--- builtin/cat-file.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
builtin/cat-file.c +10 -10
@@ -495,7 +495,7 @@ static int batch_unordered_packed(const struct object_id *oid,return batch_unordered_object (oid, data); } -static int batch_objects (struct batch_options *opt) +static int batch_objects (struct batch_options *batch) {struct strbuf input =STRBUF_INIT ;struct strbuf output =STRBUF_INIT ;
@@ -503,8 +503,8 @@ static int batch_objects(struct batch_options *opt)int save_warning;int retval =0 ; -if (!opt->format) - opt->format ="%(objectname) %(objecttype) %(objectsize)" ; +if (!batch->format) + batch->format ="%(objectname) %(objecttype) %(objectsize)" ;/* * Expand once with our special mark_query flag, which will prime the
@@ -513,20 +513,20 @@ static int batch_objects(struct batch_options *opt)*/ memset (&data,0 ,sizeof (data)); data.mark_query =1 ; -strbuf_expand (&output, opt->format, expand_format, &data); +strbuf_expand (&output, batch->format, expand_format, &data); data.mark_query =0 ;strbuf_release (&output); -if (opt->cmdmode) +if (batch->cmdmode) data.split_on_whitespace =1 ;/* * If we are printing out the object, then always fill in the type, * since we will want to decide whether or not to stream. */ -if (opt->print_contents) +if (batch->print_contents) data.info.typep = &data.type; -if (opt->all_objects) { +if (batch->all_objects) {struct object_cb_data cb;struct object_info empty =OBJECT_INFO_INIT ;
@@ -536,11 +536,11 @@ static int batch_objects(struct batch_options *opt)if (has_promisor_remote ())warning ("This repository uses promisor remotes. Some objects may not be loaded." ); - cb.opt = opt; + cb.opt = batch; cb.expand = &data; cb.scratch = &output; -if (opt->unordered) { +if (batch->unordered) {struct oidset seen =OIDSET_INIT ; cb.seen = &seen;
@@ -590,7 +590,7 @@ static int batch_objects(struct batch_options *opt) data.rest = p; } -batch_one_object (input.buf, &output, opt, &data); +batch_one_object (input.buf, &output, batch, &data); }strbuf_release (&input);
Signature · 1 line
gitgitgadget
PATCH 12/27 cat-file: create p1006-cat-file.sh
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Create p1006-cat-file.sh to provide performance testing for git cat-file --batch and git cat-file --batch-check. This will help us compare the performance changes after we let cat-file reuse the ref-filter logic.
--- t/perf/p1006-cat-file.sh | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 t/perf/p1006-cat-file.sh
t/perf/p1006-cat-file.sh new file +28 -0
@@ -0,0 +1,28 @@ +#!/bin/sh + +test_description='Basic sort performance tests' +. ./perf-lib.sh + +test_perf_default_repo + +test_expect_success 'setup' ' + git rev-list --all >rla +' + +test_perf 'cat-file --batch-check' ' + git cat-file --batch-check <rla +' + +test_perf 'cat-file --batch-check with atoms' ' + git cat-file --batch-check="%(objectname) %(objecttype)" <rla +' + +test_perf 'cat-file --batch' ' + git cat-file --batch <rla +' + +test_perf 'cat-file --batch with atoms' ' + git cat-file --batch="%(objectname) %(objecttype)" <rla +' + +test_done
Signature · 1 line
gitgitgadget
PATCH 13/27 cat-file: reuse ref-filter logic
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
In order to let cat-file use ref-filter logic, let's do the following:
1. Change the type of member format in struct batch_options to ref_format, we will pass it to ref-filter later.
2. Let batch_objects() add atoms to format, and use verify_ref_format() to check atoms.
3. Use format_ref_array_item() in batch_object_write() to get the formatted data corresponding to the object. If the return value of format_ref_array_item() is equals to zero, use batch_write() to print object data; else if the return value is less than zero, use die() to print the error message and exit; else if return value is greater than zero, only print the error message, but don't exit.
4. Use free_ref_array_item_value() to free ref_array_item's value, and use free_global_resource() to free global memory allocated resource.
Most of the atoms in for-each-ref --format are now supported, such as %(tree), %(parent), %(author), %(tagger), %(if), %(then), %(else), %(end). But these atoms will be rejected:%(refname), %(symref), %(upstream), %(push), %(worktreepath), %(flag), %(HEAD), because these atoms are unique to those objects that pointed to by a ref, "for-each-ref"'s family can naturally use these atoms, but not all objects are pointed to be a ref, so "cat-file" will not be able to use them.
The performance for git cat-file --batch-all-objects --batch-check on the Git repository itself with performance testing tool hyperfine changes from 616.7 ms ± 8.9 ms to 758.7 ms ± 16.4 ms.
The performance for `git cat-file --batch-all-objects --batch
Quoted 1 line
/dev/null` on the Git repository itself with performance testing
tool time change from "25.26s user 0.30s system 98% cpu 25.840 total" to "28.79s user 0.83s system 99% cpu 29.829 total".
The reasons for the performance degradation are as follows:
1. There are a lot of data copies in the logic of ref-filter. 2, In order to be able to support more useful formats, complex data structure and parsing process are used in ref-filter.
Later patches will gradually deal with the problem of performance degradation.
--- Documentation/git-cat-file.txt | 6 + builtin/cat-file.c | 245 +++++++-------------------------- ref-filter.c | 20 ++- ref-filter.h | 2 + t/t1006-cat-file.sh | 229 +++++++++++++++--------------- 5 files changed, 191 insertions(+), 311 deletions(-)
Documentation/git-cat-file.txt +6 -0
@@ -226,6 +226,12 @@ newline. The available atoms are: after that first run of whitespace (i.e., the "rest" of the line) are output in place of the `%(rest)` atom. +Note that most of the atoms in `for-each-ref --format` are now supported, +such as `%(tree)`, `%(parent)`, `%(author)`, `%(tagger)`, `%(if)`, +`%(then)`, `%(else)`, `%(end)`. But these atoms will be rejected: +`%(refname)`, `%(symref)`, `%(upstream)`, `%(push)`, `%(worktreepath)`, +`%(flag)`, `%(HEAD)`. See linkgit:git-for-each-ref[1]. + If no format is specified, the default format is `%(objectname) %(objecttype) %(objectsize)`.
builtin/cat-file.c +49 -196
344 lines not on this page · raw message
ref-filter.c +13 -7
@@ -2333,16 +2333,11 @@ static void free_array_item(struct ref_array_item *item)free (item); } -/* Free all memory allocated for ref_array */ -void ref_array_clear (struct ref_array *array) +/* Free global memory allocated resource */ +void free_global_resource (void ) {int i; -for (i =0 ; i < array->nr; i++) -free_array_item (array->items[i]); -FREE_AND_NULL (array->items); - array->nr = array->alloc =0 ; -for (i =0 ; i < used_atom_cnt; i++) {struct used_atom *atom = &used_atom[i];if (atom->atom_type ==ATOM_HEAD )
@@ -2359,6 +2354,17 @@ void ref_array_clear(struct ref_array *array) ref_to_worktree_map.worktrees =NULL ; } } +/* Free all memory allocated for ref_array */ +void ref_array_clear (struct ref_array *array) +{ +int i; + +for (i =0 ; i < array->nr; i++) +free_array_item (array->items[i]); +FREE_AND_NULL (array->items); + array->nr = array->alloc =0 ; +free_global_resource (); +}#define EXCLUDE_REACHED 0#define INCLUDE_REACHED 1
ref-filter.h +2 -0
@@ -112,6 +112,8 @@ struct ref_format {int filter_refs (struct ref_array *array,struct ref_filter *filter,unsigned int type);/* Clear all memory allocated to ref_array */ void ref_array_clear (struct ref_array *array); +/* Free global memory allocated resource */ +void free_global_resource (void );/* Free ref_array_item's value */ void free_ref_array_item_value (struct ref_array_item *item);/* Used to verify if the given format is correct and to parse out the used atoms */
t/t1006-cat-file.sh +121 -108
261 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH 14/27 cat-file: reuse err buf in batch_object_write()
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Reuse the err buffer in batch_object_write(), as the buffer scratch does. This will reduce the overhead of multiple allocations of memory of the err buffer.
--- builtin/cat-file.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
builtin/cat-file.c +14 -8
@@ -212,35 +212,36 @@ static void batch_write(struct batch_options *opt, const void *data, int len)static void batch_object_write (const char *obj_name,struct strbuf *scratch, +struct strbuf *err,struct batch_options *opt,struct expand_data *data) {int ret; -struct strbuf err =STRBUF_INIT ;struct ref_array_item item = { data->oid, data->rest };strbuf_reset (scratch); +strbuf_reset (err); - ret =format_ref_array_item (&item, &opt->format, scratch, &err); + ret =format_ref_array_item (&item, &opt->format, scratch, err);if (ret <0 ) -die ("%s\n" , err.buf); +die ("%s\n" , err->buf);if (ret) {/* ret > 0 means when the object corresponding to oid * cannot be found in format_ref_array_item(), we only print * the error message. */ -printf ("%s\n" , err.buf); +printf ("%s\n" , err->buf);fflush (stdout); }else {strbuf_addch (scratch,'\n' );batch_write (opt, scratch->buf, scratch->len); }free_ref_array_item_value (&item); -strbuf_release (&err); }static void batch_one_object (const char *obj_name,struct strbuf *scratch, +struct strbuf *err,struct batch_options *opt,struct expand_data *data) {
@@ -294,7 +295,7 @@ static void batch_one_object(const char *obj_name,return ; } -batch_object_write (obj_name, scratch, opt, data); +batch_object_write (obj_name, scratch, err, opt, data); }struct object_cb_data {
@@ -302,13 +303,14 @@ struct object_cb_data {struct expand_data *expand;struct oidset *seen;struct strbuf *scratch; +struct strbuf *err; };static int batch_object_cb (const struct object_id *oid,void *vdata) {struct object_cb_data *data = vdata;oidcpy (&data->expand->oid, oid); -batch_object_write (NULL , data->scratch, data->opt, data->expand); +batch_object_write (NULL , data->scratch, data->err, data->opt, data->expand);return 0 ; }
@@ -364,6 +366,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio {struct strbuf input =STRBUF_INIT ;struct strbuf output =STRBUF_INIT ; +struct strbuf err =STRBUF_INIT ;struct strbuf format =STRBUF_INIT ;struct expand_data data;int save_warning;
@@ -392,6 +395,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optio cb.opt = batch; cb.expand = &data; cb.scratch = &output; + cb.err = &err;if (batch->unordered) {struct oidset seen =OIDSET_INIT ;
@@ -416,6 +420,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optiostrbuf_release (&format);strbuf_release (&output);free_global_resource (); +strbuf_release (&err);return 0 ; }
@@ -444,13 +449,14 @@ static int batch_objects(struct batch_options *batch, const struct option *optio data.rest = p; } -batch_one_object (input.buf, &output, batch, &data); +batch_one_object (input.buf, &output, &err, batch, &data); }strbuf_release (&format);strbuf_release (&input);strbuf_release (&output);free_global_resource (); +strbuf_release (&err); warn_on_object_refname_ambiguity = save_warning;return retval; }
Signature · 1 line
gitgitgadget
PATCH 15/27 cat-file: re-implement --textconv, --filters options
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
After cat-file reuses the ref-filter logic, we re-implement the functions of --textconv and --filters options.
Add members cat_file_cmdmode to struct ref_array_item, so that struct batch_option member cmdmode will be passed to ref-filter, and then ref-filter will take use of it to filter the content of the object in get_object().
Use actual_oi to record the real expand_data: it may point to the original oi or the act_oi processed by textconv_object() or convert_to_working_tree(). grab_values() will grab the contents of actual_oi and grab_common_values() to grab the contents of origin oi, this ensures that %(objectsize) still uses the size of the unfiltered data.
--- builtin/cat-file.c | 2 +- ref-filter.c | 37 +++++++++++++++++++++++++++++++++++-- ref-filter.h | 1 + 3 files changed, 37 insertions(+), 3 deletions(-)
builtin/cat-file.c +1 -1
@@ -217,7 +217,7 @@ static void batch_object_write(const char *obj_name, struct expand_data *data) {int ret; -struct ref_array_item item = { data->oid, data->rest }; +struct ref_array_item item = { data->oid, data->rest, opt->cmdmode };strbuf_reset (scratch);strbuf_reset (err);
ref-filter.c +35 -2
@@ -1,3 +1,4 @@ +#define USE_THE_INDEX_COMPATIBILITY_MACROS #include "builtin.h" #include "cache.h" #include "parse-options.h"
@@ -1759,6 +1760,9 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj {/* parse_object_buffer() will set eaten to 0 if free() will be needed */ int eaten =1 ; +struct expand_data *actual_oi = oi; +struct expand_data act_oi = {0 }; +if (oi->info.contentp) {/* We need to know that to use parse_object_buffer properly */ oi->info.sizep = &oi->size;
@@ -1772,19 +1776,47 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **objBUG ("Object size is less than zero." );if (oi->info.contentp) { - *obj =parse_object_buffer (the_repository, &oi->oid, oi->type, oi->size, oi->content, &eaten); +if ((ref->cat_file_cmdmode =='c' || ref->cat_file_cmdmode =='w' ) && !ref->rest) +return strbuf_addf_ret (err,-1 ,_ ("missing path for '%s'" ), +oid_to_hex (&act_oi.oid)); +if (oi->type ==OBJ_BLOB ) { +if (ref->cat_file_cmdmode =='c' ) { + act_oi = *oi; +if (textconv_object (the_repository, + ref->rest,0100644 , &act_oi.oid, +1 , (char **)(&act_oi.content), &act_oi.size)) + actual_oi = &act_oi; + }else if (ref->cat_file_cmdmode =='w' ) { +struct strbuf strbuf =STRBUF_INIT ; +struct checkout_metadata meta; + act_oi = *oi; + +init_checkout_metadata (&meta,NULL ,NULL , &act_oi.oid); +if (!convert_to_working_tree (&the_index, ref->rest, act_oi.content, act_oi.size, &strbuf, &meta)) +die ("could not convert '%s' %s" , +oid_to_hex (&oi->oid), ref->rest); + act_oi.size = strbuf.len; + act_oi.content =strbuf_detach (&strbuf,NULL ); + actual_oi = &act_oi; + } + } + *obj =parse_object_buffer (the_repository, &actual_oi->oid, actual_oi->type, actual_oi->size, actual_oi->content, &eaten);if (!*obj) {if (!eaten) +free (actual_oi->content); +if (actual_oi != oi)free (oi->content);return strbuf_addf_ret (err,-1 ,_ ("parse_object_buffer failed on %s for %s" ),oid_to_hex (&oi->oid), ref->refname); } -grab_values (ref->value, deref, *obj, oi); +grab_values (ref->value, deref, *obj, actual_oi); }grab_common_values (ref->value, deref, oi);if (!eaten)free (oi->content); +if (actual_oi != oi) +free (actual_oi->content);return 0 ; }
@@ -2193,6 +2225,7 @@ static struct ref_array_item *new_ref_array_item(const char *refname,FLEX_ALLOC_STR (ref, refname, refname);oidcpy (&ref->objectname, oid); ref->rest =NULL ; + ref->cat_file_cmdmode =0 ;return ref; }
ref-filter.h +1 -0
@@ -39,6 +39,7 @@ struct ref_sorting {struct ref_array_item {struct object_id objectname;const char *rest; +int cat_file_cmdmode;int flag;unsigned int kind;const char *symref;
Signature · 1 line
gitgitgadget
PATCH 16/27 ref-filter: remove grab_oid() function
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Because "atom_type == ATOM_OBJECTNAME" implies the condition of starts_with(name, "objectname"), "atom_type == ATOM_TREE" implies the condition of starts_with(name, "tree"), so the check for starts_with(name, field) in grab_oid() is redundant.
So Remove the grab_oid() from ref-filter, to reduce repeated check.
--- ref-filter.c | 26 +++++++++----------------- 1 file changed, 9 insertions(+), 17 deletions(-)
ref-filter.c +9 -17
@@ -1081,16 +1081,6 @@ static const char *do_grab_oid(const char *field, const struct object_id *oid, } } -static int grab_oid (const char *name,const char *field,const struct object_id *oid, -struct atom_value *v,struct used_atom *atom) -{ -if (starts_with (name, field)) { - v->s =xstrdup (do_grab_oid (field, oid, atom)); -return 1 ; - } -return 0 ; -} -/* See grab_values */ static void grab_common_values (struct atom_value *val,int deref,struct expand_data *oi) {
@@ -1116,8 +1106,9 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_ } }else if (atom_type ==ATOM_DELTABASE ) v->s =xstrdup (oid_to_hex (&oi->delta_base_oid)); -else if (atom_type ==ATOM_OBJECTNAME && deref) -grab_oid (name,"objectname" , &oi->oid, v, &used_atom[i]); +else if (atom_type ==ATOM_OBJECTNAME && deref) { + v->s =xstrdup (do_grab_oid ("objectname" , &oi->oid, &used_atom[i])); + } } }
@@ -1158,9 +1149,10 @@ static void grab_commit_values(struct atom_value *val, int deref, struct objectcontinue ;if (deref) name++; -if (atom_type ==ATOM_TREE && -grab_oid (name,"tree" ,get_commit_tree_oid (commit), v, &used_atom[i])) +if (atom_type ==ATOM_TREE ) { + v->s =xstrdup (do_grab_oid ("tree" ,get_commit_tree_oid (commit), &used_atom[i]));continue ; + }if (atom_type ==ATOM_NUMPARENT ) { v->value =commit_list_count (commit->parents); v->s =xstrfmt ("%lu" , (unsigned long )v->value);
@@ -1965,9 +1957,9 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) v->s =xstrdup (buf +1 ); }continue ; - }else if (!deref && atom_type ==ATOM_OBJECTNAME && -grab_oid (name,"objectname" , &ref->objectname, v, atom)) { -continue ; + }else if (!deref && atom_type ==ATOM_OBJECTNAME ) { + v->s =xstrdup (do_grab_oid ("objectname" , &ref->objectname, atom)); +continue ; }else if (atom_type ==ATOM_HEAD ) {if (atom->u.head && !strcmp (ref->refname, atom->u.head)) v->s =xstrdup ("*" );
Signature · 1 line
gitgitgadget
PATCH 17/27 ref-filter: performance optimization by skip parse_object_buffer
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
When we are using some atoms such as %(raw), %(objectname), %(objecttype), we don't need to parse the content of the object, so we can skip parse_object_buffer() for performance optimization.
It is worth noting that in these cases, we still need to call parse_object_buffer() for parsing:
1. The atom type is one of %(tag), %(type), %(object), %(tree), %(numparent) or %(parent).
2. The type of the object is tag and the atom need to be dereferenced e.g. %(*objecttype).
With this patch, git cat-file --batch has a 12.5% performance improvement. But there is still a certain gap withgit cat-file --batch before using the logic of ref-filter. git cat-file --batch is 10% worse, git cat-file --batch-check is 30% worse.
--- ref-filter.c | 53 ++++++++++++++++++++++++++++++++++++++-------------- ref-filter.h | 5 +++-- 2 files changed, 42 insertions(+), 16 deletions(-)
ref-filter.c +39 -14
@@ -1025,6 +1025,20 @@ static int reject_atom(int cat_file_mode, enum atom_type atom_type) } } +static int need_parse_buffer (enum atom_type atom_type) { +switch (atom_type) { +case ATOM_TAG : +case ATOM_TYPE : +case ATOM_OBJECT : +case ATOM_TREE : +case ATOM_NUMPARENT : +case ATOM_PARENT : +return 1 ; +default : +return 0 ; + } +} +/* * Make sure the format string is well formed, and parse out * the used atoms.
@@ -1045,6 +1059,8 @@ int verify_ref_format(struct ref_format *format) at =parse_ref_filter_atom (format, sp +2 , ep, &err);if (at <0 )die ("%s" , err.buf); +if (need_parse_buffer (used_atom[at].atom_type)) + format->can_skip_parse_buffer =0 ;if (reject_atom (format->cat_file_mode, used_atom[at].atom_type))die (_ ("this command reject atom %%(%.*s)" ), (int )(ep - sp -2 ), sp +2 );
@@ -1532,14 +1548,16 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, s {void *buf = data->content; -switch (obj->type) { +switch (data->type) {case OBJ_TAG : -grab_tag_values (val, deref, obj); +if (obj) +grab_tag_values (val, deref, obj);grab_sub_body_contents (val, deref, data);grab_person ("tagger" , val, deref, buf);break ;case OBJ_COMMIT : -grab_commit_values (val, deref, obj); +if (obj) +grab_commit_values (val, deref, obj);grab_sub_body_contents (val, deref, data);grab_person ("author" , val, deref, buf);grab_person ("committer" , val, deref, buf);
@@ -1792,16 +1810,23 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **obj actual_oi = &act_oi; } } - *obj =parse_object_buffer (the_repository, &actual_oi->oid, actual_oi->type, actual_oi->size, actual_oi->content, &eaten); -if (!*obj) { -if (!eaten) -free (actual_oi->content); -if (actual_oi != oi) -free (oi->content); -return strbuf_addf_ret (err,-1 ,_ ("parse_object_buffer failed on %s for %s" ), -oid_to_hex (&oi->oid), ref->refname); +if (ref->can_skip_parse_buffer && + ((!deref && + (!need_tagged || oi->type !=OBJ_TAG )) || + deref)) { +grab_values (ref->value, deref,NULL , actual_oi); + }else { + *obj =parse_object_buffer (the_repository, &actual_oi->oid, actual_oi->type, actual_oi->size, actual_oi->content, &eaten); +if (!*obj) { +if (!eaten) +free (actual_oi->content); +if (actual_oi != oi) +free (oi->content); +return strbuf_addf_ret (err,-1 ,_ ("parse_object_buffer failed on %s for %s" ), +oid_to_hex (&oi->oid), ref->refname); + } +grab_values (ref->value, deref, *obj, actual_oi); } -grab_values (ref->value, deref, *obj, actual_oi); }grab_common_values (ref->value, deref, oi);
@@ -2029,7 +2054,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)* If there is no atom that wants to know about tagged * object, we are done. */ -if (!need_tagged || (obj->type !=OBJ_TAG )) +if (!need_tagged || (oi.type !=OBJ_TAG ))return 0 ;/*
@@ -2651,7 +2676,7 @@ int format_ref_array_item(struct ref_array_item *info, state.quote_style = format->quote_style;push_stack_element (&state.stack); - + info->can_skip_parse_buffer = format->can_skip_parse_buffer;for (cp = format->format; *cp && (sp =find_next (cp)); cp = ep +1 ) {struct atom_value *atomv;int pos;
ref-filter.h +3 -2
@@ -41,6 +41,7 @@ struct ref_array_item {const char *rest;int cat_file_cmdmode;int flag; +int can_skip_parse_buffer;unsigned int kind;const char *symref;struct commit *commit;
@@ -83,12 +84,12 @@ struct ref_format {int quote_style;int use_rest;int use_color; - +int can_skip_parse_buffer;/* Internal state to ref-filter */ int need_color_reset_at_eol; }; -#define REF_FORMAT_INIT { .use_color = -1 } +#define REF_FORMAT_INIT { .use_color = -1, .can_skip_parse_buffer = 1 }/* Macros for checking --merged and --no-merged options */ #define _OPT_MERGED_NO_MERGED (option, filter, h) \
Signature · 1 line
gitgitgadget
PATCH 18/27 ref-filter: use atom_type and merge two for loop in grab_person
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
grab_person() uses string for atom type comparison, this is very inefficient, we can use atom_type to replace it. At the same time, we can merge the two for loops in grab_person() into one, this can reduce unnecessary traversal of the used_atom list.
With this patch, git cat-file --batch has a 2% performance improvement.
--- ref-filter.c | 57 +++++++++++++++++----------------------------------- 1 file changed, 18 insertions(+), 39 deletions(-)
ref-filter.c +18 -39
@@ -1307,63 +1307,42 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam }/* See grab_values */ -static void grab_person (const char *who,struct atom_value *val,int deref,void *buf) +static void grab_person (enum atom_type type,struct atom_value *val,int deref,void *buf) {int i; +const char *who = valid_atom[type].name;int wholen =strlen (who);const char *wholine =NULL ;for (i =0 ; i < used_atom_cnt; i++) {const char *name = used_atom[i].name; +enum atom_type atom_type = used_atom[i].atom_type;struct atom_value *v = &val[i];if (!!deref != (*name =='*' ))continue ;if (deref) name++; -if (strncmp (who, name, wholen)) -continue ; -if (name[wholen] !=0 && -strcmp (name + wholen,"name" ) && - !starts_with (name + wholen,"email" ) && - !starts_with (name + wholen,"date" )) +if ((atom_type < type || atom_type > type +3 ) && +/* +* For a tag or a commit object, if "creator" or "creatordate" is +* requested, do something special. +*/ + ((atom_type !=ATOM_CREATOR && atom_type !=ATOM_CREATORDATE ) || + ((atom_type ==ATOM_CREATOR || atom_type ==ATOM_CREATORDATE ) && + type !=ATOM_TAGGER && type !=ATOM_COMMITTER )))continue ;if (!wholine) wholine =find_wholine (who, wholen, buf);if (!wholine)return ;/* no point looking for it */ -if (name[wholen] ==0 ) +if (atom_type == type || atom_type ==ATOM_CREATOR ) v->s =copy_line (wholine); -else if (!strcmp (name + wholen,"name" )) +else if (atom_type == type +1 ) v->s =copy_name (wholine); -else if (starts_with (name + wholen,"email" )) +else if (atom_type == type +2 ) v->s =copy_email (wholine, &used_atom[i]); -else if (starts_with (name + wholen,"date" )) -grab_date (wholine, v, name); - } - -/* -* For a tag or a commit object, if "creator" or "creatordate" is -* requested, do something special. -*/ -if (strcmp (who,"tagger" ) &&strcmp (who,"committer" )) -return ;/* "author" for commit object is not wanted */ -if (!wholine) - wholine =find_wholine (who, wholen, buf); -if (!wholine) -return ; -for (i =0 ; i < used_atom_cnt; i++) { -const char *name = used_atom[i].name; -enum atom_type atom_type = used_atom[i].atom_type; -struct atom_value *v = &val[i]; -if (!!deref != (*name =='*' )) -continue ; -if (deref) - name++; - -if (atom_type ==ATOM_CREATORDATE ) +else if (atom_type == type +3 || atom_type ==ATOM_CREATORDATE )grab_date (wholine, v, name); -else if (atom_type ==ATOM_CREATOR ) - v->s =copy_line (wholine); } }
@@ -1553,14 +1532,14 @@ static void grab_values(struct atom_value *val, int deref, struct object *obj, sif (obj)grab_tag_values (val, deref, obj);grab_sub_body_contents (val, deref, data); -grab_person ("tagger" , val, deref, buf); +grab_person (ATOM_TAGGER , val, deref, buf);break ;case OBJ_COMMIT :if (obj)grab_commit_values (val, deref, obj);grab_sub_body_contents (val, deref, data); -grab_person ("author" , val, deref, buf); -grab_person ("committer" , val, deref, buf); +grab_person (ATOM_AUTHOR , val, deref, buf); +grab_person (ATOM_COMMITTER , val, deref, buf);break ;case OBJ_TREE :/* grab_tree_values(val, deref, obj, buf, sz); */
Signature · 1 line
gitgitgadget
PATCH 19/27 ref-filter: remove strlen from find_subpos
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
find_subpos() needs to use strlen() to get the length of the object content, but in the caller of find_subpos(): grab_sub_body_contents(), we can already get the length of the object content through data->size. So add a size_t buf_size in find_subpos(), and pass data->size to it. This can lead to weak performance optimizations.
--- ref-filter.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
ref-filter.c +6 -6
@@ -1346,7 +1346,7 @@ static void grab_person(enum atom_type type, struct atom_value *val, int deref, } } -static void find_subpos(const char *buf, +static void find_subpos(const char *buf, size_t buf_size, const char **sub, size_t *sublen, const char **body, size_t *bodylen, size_t *nonsiglen,
@@ -1354,12 +1354,12 @@ static void find_subpos(const char *buf, {struct strbuf payload =STRBUF_INIT ;struct strbuf signature =STRBUF_INIT ; +const char *begin = buf;const char *eol; -const char *end = buf +strlen (buf);const char *sigstart;/* parse signature first; we might not even have a subject line */ -parse_signature (buf, end - buf, &payload, &signature); +parse_signature (buf, buf_size, &payload, &signature);/* skip past header until we hit empty line */ while (*buf && *buf !='\n' ) {
@@ -1372,7 +1372,7 @@ static void find_subpos(const char *buf,while (*buf =='\n' ) buf++; *sig =strbuf_detach (&signature, siglen); - sigstart = buf +parse_signed_buffer (buf,strlen (buf)); + sigstart = buf +parse_signed_buffer (buf, buf_size - (buf - begin));/* subject is first non-empty line */ *sub = buf;
@@ -1395,7 +1395,7 @@ static void find_subpos(const char *buf,while (*buf =='\n' || *buf =='\r' ) buf++; *body = buf; - *bodylen =strlen (buf); + *bodylen = buf_size - (buf - begin); *nonsiglen = sigstart - buf; }
@@ -1462,7 +1462,7 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp !starts_with(name, "contents")))continue ;if (!subpos) -find_subpos (buf, +find_subpos (buf, data->size, &subpos, &sublen, &bodypos, &bodylen, &nonsiglen, &sigpos, &siglen);
Signature · 1 line
gitgitgadget
PATCH 20/27 ref-filter: introducing xstrvfmt_len() and xstrfmt_len()
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
xstrfmt(), xstrvfmt(), they are functions used to add formatt strings to a dynamic memory. Because their implementation uses strbuf to store its buffer and its length, so we can easily pass its length to the caller, so create xstrvfmt_len() and xstrfmt_len() which can fetch the length of the buffer through the parameter len. This can reduce the cost of calculating the length of the string later through strlen().
--- ref-filter.c | 50 +++++++++++++++++++++++++------------------------- strbuf.c | 21 +++++++++++++++++++++ strbuf.h | 6 ++++++ 3 files changed, 52 insertions(+), 25 deletions(-)
ref-filter.c +25 -25
@@ -1115,10 +1115,10 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_else if (atom_type ==ATOM_OBJECTSIZE ) {if (used_atom[i].u.objectsize.option ==O_SIZE_DISK ) { v->value = oi->disk_size; - v->s =xstrfmt ("%" PRIuMAX, (uintmax_t )oi->disk_size); + v->s =xstrfmt_len (&v->s_size,"%" PRIuMAX, (uintmax_t )oi->disk_size); }else if (used_atom[i].u.objectsize.option ==O_SIZE ) { v->value = oi->size; - v->s =xstrfmt ("%" PRIuMAX , (uintmax_t )oi->size); + v->s =xstrfmt_len (&v->s_size,"%" PRIuMAX , (uintmax_t )oi->size); } }else if (atom_type ==ATOM_DELTABASE ) v->s =xstrdup (oid_to_hex (&oi->delta_base_oid));
@@ -1171,7 +1171,7 @@ static void grab_commit_values(struct atom_value *val, int deref, struct object }if (atom_type ==ATOM_NUMPARENT ) { v->value =commit_list_count (commit->parents); - v->s =xstrfmt ("%lu" , (unsigned long )v->value); + v->s =xstrfmt_len (&v->s_size,"%lu" , (unsigned long )v->value); }else if (atom_type ==ATOM_PARENT ) {struct commit_list *parents;
@@ -1449,7 +1449,7 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp v->s =xmemdupz (buf, buf_size); v->s_size = buf_size; }else if (atom->u.raw_data.option ==RAW_LENGTH ) { - v->s =xstrfmt ("%" PRIuMAX, (uintmax_t )buf_size); + v->s =xstrfmt_len (&v->s_size,"%" PRIuMAX, (uintmax_t )buf_size); }continue ; }
@@ -1476,7 +1476,7 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct exp }else if (atom->u.contents.option ==C_BODY_DEP ) v->s =xmemdupz (bodypos, bodylen);else if (atom->u.contents.option ==C_LENGTH ) - v->s =xstrfmt ("%" PRIuMAX, (uintmax_t )strlen (subpos)); + v->s =xstrfmt_len (&v->s_size,"%" PRIuMAX, (uintmax_t )strlen (subpos));else if (atom->u.contents.option ==C_BODY ) v->s =xmemdupz (bodypos, nonsiglen);else if (atom->u.contents.option ==C_SIG )
@@ -1645,56 +1645,56 @@ static const char *show_ref(struct refname_atom *atom, const char *refname) }static void fill_remote_ref_details (struct used_atom *atom,const char *refname, -struct branch *branch,const char **s) +struct branch *branch,struct atom_value *v) {int num_ours, num_theirs;if (atom->u.remote_ref.option ==RR_REF ) - *s =show_ref (&atom->u.remote_ref.refname, refname); + v->s =show_ref (&atom->u.remote_ref.refname, refname);else if (atom->u.remote_ref.option ==RR_TRACK ) {if (stat_tracking_info (branch, &num_ours, &num_theirs,NULL , atom->u.remote_ref.push,AHEAD_BEHIND_FULL ) <0 ) { - *s =xstrdup (msgs.gone); + v->s =xstrdup (msgs.gone); }else if (!num_ours && !num_theirs) - *s =xstrdup ("" ); + v->s =xstrdup ("" );else if (!num_ours) - *s =xstrfmt (msgs.behind, num_theirs); + v->s =xstrfmt_len (&v->s_size, msgs.behind, num_theirs);else if (!num_theirs) - *s =xstrfmt (msgs.ahead, num_ours); + v->s =xstrfmt_len (&v->s_size, msgs.ahead, num_ours);else - *s =xstrfmt (msgs.ahead_behind, + v->s=xstrfmt_len (&v->s_size, msgs.ahead_behind, num_ours, num_theirs); -if (!atom->u.remote_ref.nobracket && *s[0 ]) { -const char *to_free = *s; - *s =xstrfmt ("[%s]" , *s); +if (!atom->u.remote_ref.nobracket && v->s[0 ]) { +const char *to_free = v->s; + v->s =xstrfmt_len (&v->s_size,"[%s]" , v->s);free ((void *)to_free); } }else if (atom->u.remote_ref.option ==RR_TRACKSHORT ) {if (stat_tracking_info (branch, &num_ours, &num_theirs,NULL , atom->u.remote_ref.push,AHEAD_BEHIND_FULL ) <0 ) { - *s =xstrdup ("" ); + v->s =xstrdup ("" );return ; }if (!num_ours && !num_theirs) - *s =xstrdup ("=" ); + v->s =xstrdup ("=" );else if (!num_ours) - *s =xstrdup ("<" ); + v->s =xstrdup ("<" );else if (!num_theirs) - *s =xstrdup (">" ); + v->s =xstrdup (">" );else - *s =xstrdup ("<>" ); + v->s =xstrdup ("<>" ); }else if (atom->u.remote_ref.option ==RR_REMOTE_NAME ) {int explicit;const char *remote = atom->u.remote_ref.push ?pushremote_for_branch (branch, &explicit) :remote_for_branch (branch, &explicit); - *s =xstrdup (explicit ? remote :"" ); + v->s =xstrdup (explicit ? remote :"" ); }else if (atom->u.remote_ref.option ==RR_REMOTE_REF ) {const char *merge; merge =remote_ref_for_branch (branch, atom->u.remote_ref.push); - *s =xstrdup (merge ? merge :"" ); + v->s =xstrdup (merge ? merge :"" ); }else BUG ("unhandled RR_* enum" ); }
@@ -1922,7 +1922,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) refname =branch_get_upstream (branch,NULL );if (refname) -fill_remote_ref_details (atom, refname, branch, &v->s); +fill_remote_ref_details (atom, refname, branch, v);else v->s =xstrdup ("" );continue ;
@@ -1943,7 +1943,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) }/* We will definitely re-init v->s on the next line. */ free ((char *)v->s); -fill_remote_ref_details (atom, refname, branch, &v->s); +fill_remote_ref_details (atom, refname, branch, v);continue ; }else if (atom_type ==ATOM_COLOR ) { v->s =xstrdup (atom->u.color);
@@ -2006,7 +2006,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)if (!deref) v->s =xstrdup (refname);else - v->s =xstrfmt ("%s^{}" , refname); + v->s =xstrfmt_len (&v->s_size,"%s^{}" , refname);free ((char *)refname); }
strbuf.c +21 -0
@@ -982,6 +982,15 @@ char *xstrvfmt(const char *fmt, va_list ap)return strbuf_detach (&buf,NULL ); } +char *xstrvfmt_len (ssize_t *len,const char *fmt,va_list ap) +{ +struct strbuf buf =STRBUF_INIT ; +strbuf_vaddf (&buf, fmt, ap); +if (len) + *len = buf.len; +return strbuf_detach (&buf,NULL ); +} +char *xstrfmt (const char *fmt, ...) {va_list ap;
@@ -994,6 +1003,18 @@ char *xstrfmt(const char *fmt, ...)return ret; } +char *xstrfmt_len (ssize_t *len,const char *fmt, ...) +{ +va_list ap; +char *ret; + +va_start (ap, fmt); + ret =xstrvfmt_len (len ,fmt, ap); +va_end (ap); + +return ret; +} +void strbuf_addftime (struct strbuf *sb,const char *fmt,const struct tm *tm,int tz_offset,int suppress_tz_name) {
strbuf.h +6 -0
@@ -738,4 +738,10 @@ char *xstrvfmt(const char *fmt, va_list ap); __attribute__((format (printf,1 ,2 )))char *xstrfmt (const char *fmt, ...); + +__attribute__((format (printf,2 ,0 ))) +char *xstrvfmt_len (ssize_t *len,const char *fmt,va_list ap); +__attribute__((format (printf,2 ,3 ))) +char *xstrfmt_len (ssize_t *len,const char *fmt, ...); +#endif /* STRBUF_H */
Signature · 1 line
gitgitgadget
PATCH 21/27 ref-filter: remove second parsing in format_ref_array_item
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
We parsed format string in verify_ref_format() and stored the parsed atom in used_atom array. But in format_ref_array_item() we have another round of parsing format string. This affects performance.
Introducing the struct parsed_atom_list which can save the current atom's start and end address in format string and its index in used_atom. All parsed_atom_list entry are linked together in the form of linked list, and the head node of the linked list is stored in struct ref_format. We can use clear_parsed_atom_list() to clear the nodes on the linked list.
This can bring 1.9% performance improvement for git cat-file --batch.
--- builtin/branch.c | 2 ++ builtin/cat-file.c | 3 +++ builtin/for-each-ref.c | 3 ++- builtin/tag.c | 2 ++ builtin/verify-tag.c | 2 ++ ref-filter.c | 45 +++++++++++++++++++++++++++++++++--------- ref-filter.h | 11 +++++++++++ 7 files changed, 58 insertions(+), 10 deletions(-)
builtin/branch.c +2 -0
@@ -459,6 +459,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortinstrbuf_release (&err);strbuf_release (&out);ref_array_clear (&array); +clear_parsed_atom_list (&format->parsed_atom_head);free (to_free); }
@@ -678,6 +679,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)memset (&filter,0 ,sizeof (filter)); filter.kind =FILTER_REFS_BRANCHES ; filter.abbrev =-1 ; +INIT_LIST_HEAD (&format.parsed_atom_head);if (argc ==2 && !strcmp (argv[1 ],"-h" ))usage_with_options (builtin_branch_usage, options);
builtin/cat-file.c +3 -0
@@ -421,6 +421,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optiostrbuf_release (&output);free_global_resource ();strbuf_release (&err); +clear_parsed_atom_list (&batch->format.parsed_atom_head);return 0 ; }
@@ -457,6 +458,7 @@ static int batch_objects(struct batch_options *batch, const struct option *optiostrbuf_release (&output);free_global_resource ();strbuf_release (&err); +clear_parsed_atom_list (&batch->format.parsed_atom_head); warn_on_object_refname_ambiguity = save_warning;return retval; }
@@ -534,6 +536,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix) batch.buffer_output =-1 ; batch.format.cat_file_mode =1 ; +INIT_LIST_HEAD (&batch.format.parsed_atom_head); argc =parse_options (argc, argv, prefix, options, cat_file_usage,0 );if (opt) {
builtin/for-each-ref.c +2 -1
@@ -53,8 +53,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)memset (&array,0 ,sizeof (array));memset (&filter,0 ,sizeof (filter)); - format.format ="%(objectname) %(objecttype)\t%(refname)" ; +INIT_LIST_HEAD (&format.parsed_atom_head);git_config (git_default_config,NULL );
@@ -96,6 +96,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)ref_array_clear (&array);free_commit_list (filter.with_commit);free_commit_list (filter.no_commit); +clear_parsed_atom_list (&format.parsed_atom_head);UNLEAK (sorting);return 0 ; }
builtin/tag.c +2 -0
@@ -78,6 +78,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,strbuf_release (&output);ref_array_clear (&array);free (to_free); +clear_parsed_atom_list (&format->parsed_atom_head);return 0 ; }
@@ -493,6 +494,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)memset (&filter,0 ,sizeof (filter)); filter.lines =-1 ; opt.sign =-1 ; +INIT_LIST_HEAD (&format.parsed_atom_head); argc =parse_options (argc, argv, prefix, options, git_tag_usage,0 );
builtin/verify-tag.c +2 -0
@@ -39,6 +39,7 @@ int cmd_verify_tag(int argc, const char **argv, const char *prefix)OPT_END () }; +INIT_LIST_HEAD (&format.parsed_atom_head);git_config (git_verify_tag_config,NULL ); argc = parse_options(argc, argv, prefix, verify_tag_options,
@@ -73,5 +74,6 @@ int cmd_verify_tag(int argc, const char **argv, const char *prefix)if (format.format)pretty_print_ref (name, &oid, &format); } +clear_parsed_atom_list (&format.parsed_atom_head);return had_error; }
ref-filter.c +36 -9
87 lines not on this page · raw message
ref-filter.h +11 -0
@@ -73,6 +73,13 @@ struct ref_filter { verbose; }; +struct parsed_atom_list { +const char *beg; +const char *end; +int at; +struct list_head list; +}; +struct ref_format {/* * Set these to define the format; make sure you call
@@ -87,6 +94,7 @@ struct ref_format {int can_skip_parse_buffer;/* Internal state to ref-filter */ int need_color_reset_at_eol; +struct list_head parsed_atom_head; };#define REF_FORMAT_INIT { .use_color = -1, .can_skip_parse_buffer = 1 }
@@ -118,6 +126,9 @@ void ref_array_clear(struct ref_array *array);void free_global_resource (void );/* Free ref_array_item's value */ void free_ref_array_item_value (struct ref_array_item *item); +/* Clear the parsed_atom_list in ref_format*/ +void clear_parsed_atom_list (struct list_head *parsed_atom_head); +/* Used to verify if the given format is correct and to parse out the used atoms */ int verify_ref_format (struct ref_format *format);/* Sort the given ref_array as per the ref_sorting provided */
Signature · 1 line
gitgitgadget
PATCH 22/27 ref-filter: introduction ref_filter_slopbuf
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
xstrdup("") calls strdup(""), which in glibc will call malloc(1), this means that there will have unwanted memory allocations.
Introduce the global one-byte array ref_filter_slopbuf, which learn from strbuf_slopbuf. let those places which need fill strdup("") now fill ref_filter_slopbuf. This can reduce some unnecessary memory allocation.
--- ref-filter.c | 53 ++++++++++++++++++++++++++-------------------------- 1 file changed, 27 insertions(+), 26 deletions(-)
ref-filter.c +27 -26
@@ -25,6 +25,8 @@#include "hashmap.h" #include "strvec.h" +char ref_filter_slopbuf[1 ]; + static struct ref_msg { const char *gone; const char *ahead;
@@ -1225,7 +1227,7 @@ static const char *copy_name(const char *buf)if (!strncmp (cp," <" ,2 ))return xmemdupz (buf, cp - buf); } -return xstrdup ("" ); +return ref_filter_slopbuf; }static const char *copy_email (const char *buf,struct used_atom *atom)
@@ -1233,7 +1235,7 @@ static const char *copy_email(const char *buf, struct used_atom *atom)const char *email =strchr (buf,'<' );const char *eoemail;if (!email) -return xstrdup ("" ); +return ref_filter_slopbuf;switch (atom->u.email_option.option) {case EO_RAW : eoemail =strchr (email,'>' );
@@ -1255,7 +1257,7 @@ static const char *copy_email(const char *buf, struct used_atom *atom) }if (!eoemail) -return xstrdup ("" ); +return ref_filter_slopbuf;return xmemdupz (email, eoemail - email); }
@@ -1309,7 +1311,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam v->value = timestamp;return ; bad: - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf; v->value =0 ; }
@@ -1519,7 +1521,7 @@ static void fill_missing_values(struct atom_value *val)for (i =0 ; i < used_atom_cnt; i++) {struct atom_value *v = &val[i];if (v->s ==NULL ) - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf; } }
@@ -1594,7 +1596,7 @@ static const char *lstrip_ref_components(const char *refname, int len)switch (*start++) {case '\0' :free ((char *)to_free); -return xstrdup ("" ); +return ref_filter_slopbuf;case '/' : remaining--;break ;
@@ -1632,7 +1634,7 @@ static const char *rstrip_ref_components(const char *refname, int len)char *p =strrchr (start,'/' );if (p ==NULL ) {free ((char *)to_free); -return xstrdup ("" ); +return ref_filter_slopbuf; }else p[0 ] ='\0' ; }
@@ -1663,7 +1665,7 @@ static void fill_remote_ref_details(struct used_atom *atom, const char *refname, AHEAD_BEHIND_FULL) < 0) { v->s =xstrdup (msgs.gone); } else if (!num_ours && !num_theirs) - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf; else if (!num_ours) v->s = xstrfmt_len(&v->s_size, msgs.behind, num_theirs); else if (!num_theirs)
@@ -1680,7 +1682,7 @@ static void fill_remote_ref_details(struct used_atom *atom, const char *refname,if (stat_tracking_info (branch, &num_ours, &num_theirs,NULL , atom->u.remote_ref.push,AHEAD_BEHIND_FULL ) <0 ) { - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;return ; }if (!num_ours && !num_theirs)
@@ -1739,7 +1741,7 @@ char *get_head_description(void)static const char *get_symref (struct used_atom *atom,struct ref_array_item *ref) {if (!ref->symref) -return xstrdup ("" ); +return ref_filter_slopbuf;else return show_ref (&atom->u.refname, ref->symref); }
@@ -1861,7 +1863,7 @@ static char *get_worktree_path(const struct used_atom *atom, const struct ref_ar e =hashmap_get (&(ref_to_worktree_map.map), &entry, ref->refname);if (!e) -return xstrdup ("" ); +return ref_filter_slopbuf; lookup_result = container_of(e, struct ref_to_worktree_entry, ent);
@@ -1884,7 +1886,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) ref->symref =resolve_refdup (ref->refname,RESOLVE_REF_READING ,NULL ,NULL );if (!ref->symref) - ref->symref =xstrdup ("" ); + ref->symref = ref_filter_slopbuf; }/* Fill in specials first */
@@ -1912,7 +1914,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)if (ref->kind ==FILTER_REFS_BRANCHES ) v->s =get_worktree_path (atom, ref);else - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else if (atom_type ==ATOM_SYMREF )
@@ -1922,7 +1924,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)/* only local branches may have an upstream */ if (!skip_prefix (ref->refname,"refs/heads/" , &branch_name)) { - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; } branch =branch_get (branch_name);
@@ -1931,11 +1933,11 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)if (refname)fill_remote_ref_details (atom, refname, branch, v);else - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else if (atom_type ==ATOM_PUSH && atom->u.remote_ref.push) {const char *branch_name; - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;if (!skip_prefix (ref->refname,"refs/heads/" , &branch_name))continue ;
@@ -1948,8 +1950,6 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)if (!refname)continue ; } -/* We will definitely re-init v->s on the next line. */ -free ((char *)v->s);fill_remote_ref_details (atom, refname, branch, v);continue ; }else if (atom_type ==ATOM_COLOR ) {
@@ -1962,7 +1962,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)if (ref->flag &REF_ISPACKED ) cp =copy_advance (cp,",packed" );if (cp == buf) - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;else { *cp ='\0' ; v->s =xstrdup (buf +1 );
@@ -1979,33 +1979,33 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)continue ; }else if (atom_type ==ATOM_ALIGN ) { v->handler = align_atom_handler; - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else if (atom_type ==ATOM_END ) { v->handler = end_atom_handler; - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else if (atom_type ==ATOM_IF ) {const char *s;if (skip_prefix (name,"if:" , &s)) v->s =xstrdup (s);else - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf; v->handler = if_atom_handler;continue ; }else if (atom_type ==ATOM_THEN ) { v->handler = then_atom_handler; - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else if (atom_type ==ATOM_ELSE ) { v->handler = else_atom_handler; - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else if (atom_type ==ATOM_REST ) {if (ref->rest) v->s =xstrdup (ref->rest);else - v->s =xstrdup ("" ); + v->s = ref_filter_slopbuf;continue ; }else continue ;
@@ -2356,7 +2356,8 @@ void free_ref_array_item_value(struct ref_array_item *item)if (item->value) {int i;for (i =0 ; i < used_atom_cnt; i++) -free ((char *)item->value[i].s); +if (item->value[i].s != ref_filter_slopbuf) +free ((char *)item->value[i].s);free (item->value); } }
Signature · 1 line
gitgitgadget
PATCH 23/27 ref-filter: add deref member to struct used_atom
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
In the original logic, the dereference flag '*' is stored at the beginning of used_atom[i].name. (e.g. '*objectname:short') This leads us to make some check on used_atom[i].name[0] to know if we the atom needs to be dereferenced.
So add deref flag to struct used_atom used to indicate whether the atom needs to be dereferenced and let used_atom record only the remaining part (i.e. 'objectname:short')
This can simplify the logic of the program.
--- ref-filter.c | 55 ++++++++++++++++++++-------------------------------- 1 file changed, 21 insertions(+), 34 deletions(-)
ref-filter.c +21 -34
@@ -174,6 +174,7 @@ enum atom_type {* array. */ static struct used_atom { + unsigned int deref : 1; enum atom_type atom_type; const char *name; cmp_type type;
@@ -321,7 +322,7 @@ static int objecttype_atom_parser(struct ref_format *format, struct used_atom *a {if (arg)return strbuf_addf_ret (err,-1 ,_ ("%%(objecttype) does not take arguments" )); -if (*atom->name =='*' ) +if (atom->deref) oi_deref.info.typep = &oi_deref.type;else oi.info.typep = &oi.type;
@@ -333,13 +334,13 @@ static int objectsize_atom_parser(struct ref_format *format, struct used_atom *a {if (!arg) { atom->u.objectsize.option =O_SIZE ; -if (*atom->name =='*' ) +if (atom->deref) oi_deref.info.sizep = &oi_deref.size;else oi.info.sizep = &oi.size; }else if (!strcmp (arg,"disk" )) { atom->u.objectsize.option =O_SIZE_DISK ; -if (*atom->name =='*' ) +if (atom->deref) oi_deref.info.disk_sizep = &oi_deref.disk_size;else oi.info.disk_sizep = &oi.disk_size;
@@ -353,7 +354,7 @@ static int deltabase_atom_parser(struct ref_format *format, struct used_atom *at {if (arg)return strbuf_addf_ret (err,-1 ,_ ("%%(deltabase) does not take arguments" )); -if (*atom->name =='*' ) +if (atom->deref) oi_deref.info.delta_base_oid = &oi_deref.delta_base_oid;else oi.info.delta_base_oid = &oi.delta_base_oid;
@@ -675,10 +676,13 @@ static int parse_ref_filter_atom(struct ref_format *format,const char *sp;const char *arg;int i, at, atom_len; +int deref =0 ; sp = atom; -if (*sp =='*' && sp < ep) +if (*sp =='*' && sp < ep) { sp++;/* deref */ + deref =1 ; + }if (ep <= sp)return strbuf_addf_ret (err,-1 ,_ ("malformed field name: %.*s" ), (int )(ep-atom), atom);
@@ -695,7 +699,7 @@ static int parse_ref_filter_atom(struct ref_format *format,/* Do we have the atom already used elsewhere? */ for (i =0 ; i < used_atom_cnt; i++) {int len =strlen (used_atom[i].name); -if (len == ep - atom && !memcmp (used_atom[i].name, atom, len)) +if (len == ep - sp && !memcmp (used_atom[i].name, sp, len))return i; }
@@ -719,17 +723,18 @@ static int parse_ref_filter_atom(struct ref_format *format, used_atom_cnt++;REALLOC_ARRAY (used_atom, used_atom_cnt); used_atom[at].atom_type = i; - used_atom[at].name =xmemdupz (atom, ep - atom); + used_atom[at].deref = deref; + used_atom[at].name =xmemdupz (sp, ep - sp); used_atom[at].type = valid_atom[i].cmp_type; used_atom[at].source = valid_atom[i].source;if (used_atom[at].source ==SOURCE_OBJ ) { -if (*atom =='*' ) +if (deref) oi_deref.info.contentp = &oi_deref.content;else oi.info.contentp = &oi.content; }if (arg) { - arg = used_atom[at].name + (arg - atom) +1 ; + arg = used_atom[at].name + (arg - sp) +1 ;if (!*arg) {/* * Treat empty sub-arguments list as NULL (i.e.,
@@ -741,7 +746,7 @@ static int parse_ref_filter_atom(struct ref_format *format,memset (&used_atom[at].u,0 ,sizeof (used_atom[at].u));if (valid_atom[i].parser && valid_atom[i].parser (format, &used_atom[at], arg, err))return -1 ; -if (*atom =='*' ) +if (deref) need_tagged =1 ;if (i ==ATOM_SYMREF ) need_symref =1 ;
@@ -1112,13 +1117,10 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_int i;for (i =0 ; i < used_atom_cnt; i++) { -const char *name = used_atom[i].name;enum atom_type atom_type = used_atom[i].atom_type;struct atom_value *v = &val[i]; -if (!!deref != (*name =='*' )) +if (!!deref != used_atom[i].deref)continue ; -if (deref) - name++;if (atom_type ==ATOM_OBJECTTYPE ) v->s =xstrdup (type_name (oi->type));else if (atom_type ==ATOM_OBJECTSIZE ) {
@@ -1144,13 +1146,10 @@ static void grab_tag_values(struct atom_value *val, int deref, struct object *obstruct tag *tag = (struct tag *) obj;for (i =0 ; i < used_atom_cnt; i++) { -const char *name = used_atom[i].name;enum atom_type atom_type = used_atom[i].atom_type;struct atom_value *v = &val[i]; -if (!!deref != (*name =='*' )) +if (!!deref != used_atom[i].deref)continue ; -if (deref) - name++;if (atom_type ==ATOM_TAG ) v->s =xstrdup (tag->tag);else if (atom_type ==ATOM_TYPE && tag->tagged)
@@ -1167,13 +1166,10 @@ static void grab_commit_values(struct atom_value *val, int deref, struct objectstruct commit *commit = (struct commit *) obj;for (i =0 ; i < used_atom_cnt; i++) { -const char *name = used_atom[i].name;enum atom_type atom_type = used_atom[i].atom_type;struct atom_value *v = &val[i]; -if (!!deref != (*name =='*' )) +if (!!deref != used_atom[i].deref)continue ; -if (deref) - name++;if (atom_type ==ATOM_TREE ) { v->s =xstrdup (do_grab_oid ("tree" ,get_commit_tree_oid (commit), &used_atom[i]));continue ;
@@ -1327,10 +1323,8 @@ static void grab_person(enum atom_type type, struct atom_value *val, int deref,const char *name = used_atom[i].name;enum atom_type atom_type = used_atom[i].atom_type;struct atom_value *v = &val[i]; -if (!!deref != (*name =='*' )) +if (!!deref != used_atom[i].deref)continue ; -if (deref) - name++; if ((atom_type < type || atom_type > type + 3) &&/* * For a tag or a commit object, if "creator" or "creatordate" is
@@ -1446,10 +1440,8 @@ static void grab_sub_body_contents(struct atom_value *val, int deref, struct expstruct atom_value *v = &val[i];enum atom_type atom_type = atom->atom_type; -if (!!deref != (*name =='*' )) +if (!!deref != used_atom[i].deref)continue ; -if (deref) - name++;if (atom_type ==ATOM_RAW ) {unsigned long buf_size = data->size;
@@ -1895,7 +1887,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)enum atom_type atom_type = atom->atom_type;const char *name = used_atom[i].name;struct atom_value *v = &ref->value[i]; -int deref =0 ; +int deref = atom->deref;const char *refname;struct branch *branch =NULL ;
@@ -1903,11 +1895,6 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) v->handler = append_atom; v->atom = atom; -if (*name =='*' ) { - deref =1 ; - name++; - } -if (atom_type ==ATOM_REFNAME ) refname =get_refname (atom, ref);else if (atom_type ==ATOM_WORKTREEPATH ) {
Signature · 1 line
gitgitgadget
PATCH 24/27 ref-filter: introduce symref_atom_parser()
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
parse_ref_filter_atom() checks whether the type of the atom is ATOM_SYMREF, this is not necessary for other atoms. So add the symref_atom_parser() specifically to parse the atom, and set need_symref in it. This can make the program logic more concise.
--- ref-filter.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
ref-filter.c +8 -3
@@ -481,6 +481,13 @@ static int person_email_atom_parser(struct ref_format *format, struct used_atomreturn 0 ; } +static int symref_atom_parser (struct ref_format *format,struct used_atom *atom, +const char *arg,struct strbuf *err) +{ + need_symref =1 ; +return refname_atom_parser_internal (&atom->u.refname, arg, atom->name, err); +} +static int refname_atom_parser (struct ref_format *format,struct used_atom *atom,const char *arg,struct strbuf *err) {
@@ -620,7 +627,7 @@ static struct { [ATOM_RAW] = { "raw", SOURCE_OBJ, FIELD_STR, raw_atom_parser }, [ATOM_UPSTREAM] = { "upstream", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser }, [ATOM_PUSH] = { "push", SOURCE_NONE, FIELD_STR, remote_ref_atom_parser }, - [ATOM_SYMREF] = { "symref", SOURCE_NONE, FIELD_STR, refname_atom_parser }, + [ATOM_SYMREF] = { "symref", SOURCE_NONE, FIELD_STR, symref_atom_parser }, [ATOM_FLAG] = { "flag", SOURCE_NONE }, [ATOM_HEAD] = { "HEAD", SOURCE_NONE, FIELD_STR, head_atom_parser }, [ATOM_COLOR] = { "color", SOURCE_NONE, FIELD_STR, color_atom_parser },
@@ -748,8 +755,6 @@ static int parse_ref_filter_atom(struct ref_format *format,return -1 ;if (deref) need_tagged =1 ; -if (i ==ATOM_SYMREF ) - need_symref =1 ;return at; }
Signature · 1 line
gitgitgadget
PATCH 25/27 ref-filter: use switch case instread of if else
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
In the original code, the check of the atom type is through if/else conditional check, using switch/case can allow the compiler to reduce unnecessary branch checks and increase the readability of the code.
--- ref-filter.c | 163 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 109 insertions(+), 54 deletions(-)
ref-filter.c +109 -54
@@ -1126,9 +1126,11 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_struct atom_value *v = &val[i];if (!!deref != used_atom[i].deref)continue ; -if (atom_type ==ATOM_OBJECTTYPE ) +switch (atom_type) { +case ATOM_OBJECTTYPE : { v->s =xstrdup (type_name (oi->type)); -else if (atom_type ==ATOM_OBJECTSIZE ) { +break ; + }case ATOM_OBJECTSIZE : {if (used_atom[i].u.objectsize.option ==O_SIZE_DISK ) { v->value = oi->disk_size; v->s =xstrfmt_len (&v->s_size,"%" PRIuMAX, (uintmax_t )oi->disk_size);
@@ -1136,10 +1138,20 @@ static void grab_common_values(struct atom_value *val, int deref, struct expand_ v->value = oi->size; v->s =xstrfmt_len (&v->s_size,"%" PRIuMAX , (uintmax_t )oi->size); } - }else if (atom_type ==ATOM_DELTABASE ) +break ; + } +case ATOM_DELTABASE : { v->s =xstrdup (oid_to_hex (&oi->delta_base_oid)); -else if (atom_type ==ATOM_OBJECTNAME && deref) { - v->s =xstrdup (do_grab_oid ("objectname" , &oi->oid, &used_atom[i])); +break ; + } +case ATOM_OBJECTNAME : { +if (deref) + v->s =xstrdup (do_grab_oid ("objectname" , &oi->oid, &used_atom[i])); +break ; + } +default : { +break ; + } } } }
@@ -1154,13 +1166,22 @@ static void grab_tag_values(struct atom_value *val, int deref, struct object *obenum atom_type atom_type = used_atom[i].atom_type;struct atom_value *v = &val[i];if (!!deref != used_atom[i].deref) -continue ; -if (atom_type ==ATOM_TAG ) +break ; +switch (atom_type) { +case ATOM_TAG : v->s =xstrdup (tag->tag); -else if (atom_type ==ATOM_TYPE && tag->tagged) - v->s =xstrdup (type_name (tag->tagged->type)); -else if (atom_type ==ATOM_OBJECT && tag->tagged) - v->s =xstrdup (oid_to_hex (&tag->tagged->oid)); +break ; +case ATOM_TYPE : +if (tag->tagged) + v->s =xstrdup (type_name (tag->tagged->type)); +break ; +case ATOM_OBJECT : +if (tag->tagged) + v->s =xstrdup (oid_to_hex (&tag->tagged->oid)); +break ; +default : +break ; + } } }
@@ -1175,15 +1196,17 @@ static void grab_commit_values(struct atom_value *val, int deref, struct objectstruct atom_value *v = &val[i];if (!!deref != used_atom[i].deref)continue ; -if (atom_type ==ATOM_TREE ) { +switch (atom_type) { +case ATOM_TREE : { v->s =xstrdup (do_grab_oid ("tree" ,get_commit_tree_oid (commit), &used_atom[i])); -continue ; +break ; } -if (atom_type ==ATOM_NUMPARENT ) { +case ATOM_NUMPARENT : { v->value =commit_list_count (commit->parents); v->s =xstrfmt_len (&v->s_size,"%lu" , (unsigned long )v->value); +break ; } -else if (atom_type ==ATOM_PARENT ) { +case ATOM_PARENT : {struct commit_list *parents;struct strbuf s =STRBUF_INIT ;for (parents = commit->parents; parents; parents = parents->next) {
@@ -1193,6 +1216,11 @@ static void grab_commit_values(struct atom_value *val, int deref, struct objectstrbuf_addstr (&s,do_grab_oid ("parent" , oid, &used_atom[i])); } v->s =strbuf_detach (&s,NULL ); +break ; + } +default : { +break ; + } } } }
@@ -1900,24 +1928,40 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) v->handler = append_atom; v->atom = atom; -if (atom_type ==ATOM_REFNAME ) +switch (atom_type) { +case ATOM_REFNAME : { refname =get_refname (atom, ref); -else if (atom_type ==ATOM_WORKTREEPATH ) { +if (!deref) + v->s =xstrdup (refname); +else + v->s =xstrfmt_len (&v->s_size,"%s^{}" , refname); +free ((char *)refname); +break ; + } +case ATOM_WORKTREEPATH : {if (ref->kind ==FILTER_REFS_BRANCHES ) v->s =get_worktree_path (atom, ref);else v->s = ref_filter_slopbuf; -continue ; +break ; } -else if (atom_type ==ATOM_SYMREF ) +case ATOM_SYMREF : { refname =get_symref (atom, ref); -else if (atom_type ==ATOM_UPSTREAM ) { +if (!deref) + v->s =xstrdup (refname); +else + v->s =xstrfmt_len (&v->s_size,"%s^{}" , refname); +if (refname != ref_filter_slopbuf) +free ((char *)refname); +break ; + } +case ATOM_UPSTREAM : {const char *branch_name;/* only local branches may have an upstream */ if (!skip_prefix (ref->refname,"refs/heads/" , &branch_name)) { v->s = ref_filter_slopbuf; -continue ; +break ; } branch =branch_get (branch_name);
@@ -1926,13 +1970,17 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)fill_remote_ref_details (atom, refname, branch, v); else v->s = ref_filter_slopbuf; -continue ; - }else if (atom_type ==ATOM_PUSH && atom->u.remote_ref.push) { +break ; + } +case ATOM_PUSH : {const char *branch_name; + +if (!atom->u.remote_ref.push) +break ; v->s = ref_filter_slopbuf;if (!skip_prefix (ref->refname,"refs/heads/" , &branch_name)) -continue ; +break ; branch =branch_get (branch_name);if (atom->u.remote_ref.push_remote)
@@ -1940,14 +1988,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)else { refname =branch_get_push (branch,NULL );if (!refname) -continue ; +break ; }fill_remote_ref_details (atom, refname, branch, v); -continue ; - }else if (atom_type ==ATOM_COLOR ) { +break ; + } +case ATOM_COLOR : { v->s =xstrdup (atom->u.color);continue ; - }else if (atom_type ==ATOM_FLAG ) { + } +case ATOM_FLAG : {char buf[256 ], *cp = buf;if (ref->flag &REF_ISSYMREF ) cp =copy_advance (cp,",symref" );
@@ -1959,54 +2009,59 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) *cp ='\0' ; v->s =xstrdup (buf +1 ); } -continue ; - }else if (!deref && atom_type ==ATOM_OBJECTNAME ) { - v->s =xstrdup (do_grab_oid ("objectname" , &ref->objectname, atom)); -continue ; - }else if (atom_type ==ATOM_HEAD ) { +break ; + } +case ATOM_OBJECTNAME : { +if (!deref) + v->s =xstrdup (do_grab_oid ("objectname" , &ref->objectname, atom)); +break ; + } +case ATOM_HEAD : {if (atom->u.head && !strcmp (ref->refname, atom->u.head)) v->s =xstrdup ("*" );else v->s =xstrdup (" " ); -continue ; - }else if (atom_type ==ATOM_ALIGN ) { +break ; + } +case ATOM_ALIGN : { v->handler = align_atom_handler; v->s = ref_filter_slopbuf; -continue ; - }else if (atom_type ==ATOM_END ) { +break ; + } +case ATOM_END : { v->handler = end_atom_handler; v->s = ref_filter_slopbuf; -continue ; - }else if (atom_type ==ATOM_IF ) { +break ; + } +case ATOM_IF : {const char *s;if (skip_prefix (name,"if:" , &s)) v->s =xstrdup (s);else v->s = ref_filter_slopbuf; v->handler = if_atom_handler; -continue ; - }else if (atom_type ==ATOM_THEN ) { +break ; + } +case ATOM_THEN : { v->handler = then_atom_handler; v->s = ref_filter_slopbuf; -continue ; - }else if (atom_type ==ATOM_ELSE ) { +break ; + } +case ATOM_ELSE : { v->handler = else_atom_handler; v->s = ref_filter_slopbuf; -continue ; - }else if (atom_type ==ATOM_REST ) { +break ; + } +case ATOM_REST : {if (ref->rest) v->s =xstrdup (ref->rest);else v->s = ref_filter_slopbuf; -continue ; - }else -continue ; - -if (!deref) - v->s =xstrdup (refname); -else - v->s =xstrfmt_len (&v->s_size,"%s^{}" , refname); -free ((char *)refname); +break ; + } +default : +break ; + } }for (i =0 ; i < used_atom_cnt; i++) {
Signature · 1 line
gitgitgadget
PATCH 27/27 ref-filter: add need_get_object_info flag to struct expand_data
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
In populate_value() we use an empty struct object_info compare with oi and oi_deref, which determine if we can return early, this is actually very inefficient.
So add a need_get_object_info flag to struct expand_data, which used to indicate whether we need call get_object() or we can return early.
--- ref-filter.c | 46 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 14 deletions(-)
ref-filter.c +32 -14
@@ -77,6 +77,7 @@ static struct ref_trailer_buf { } ref_trailer_buf = {STRING_LIST_INIT_NODUP, STRBUF_INIT, STRBUF_INIT}; static struct expand_data { + unsigned int need_get_object_info : 1; struct object_id oid; enum object_type type; unsigned long size;
@@ -322,10 +323,13 @@ static int objecttype_atom_parser(struct ref_format *format, struct used_atom *a {if (arg)return strbuf_addf_ret (err,-1 ,_ ("%%(objecttype) does not take arguments" )); -if (atom->deref) +if (atom->deref) { oi_deref.info.typep = &oi_deref.type; -else + oi_deref.need_get_object_info =1 ; + }else { oi.info.typep = &oi.type; + oi.need_get_object_info =1 ; + }return 0 ; }
@@ -334,16 +338,23 @@ static int objectsize_atom_parser(struct ref_format *format, struct used_atom *a {if (!arg) { atom->u.objectsize.option =O_SIZE ; -if (atom->deref) +if (atom->deref) { oi_deref.info.sizep = &oi_deref.size; -else + oi_deref.need_get_object_info =1 ; + }else { oi.info.sizep = &oi.size; + oi.need_get_object_info =1 ; + } }else if (!strcmp (arg,"disk" )) { atom->u.objectsize.option =O_SIZE_DISK ; -if (atom->deref) +if (atom->deref) { oi_deref.info.disk_sizep = &oi_deref.disk_size; -else + oi_deref.need_get_object_info =1 ; + } +else { oi.info.disk_sizep = &oi.disk_size; + oi.need_get_object_info =1 ; + } }else return strbuf_addf_ret (err,-1 ,_ ("unrecognized %%(objectsize) argument: %s" ), arg);return 0 ;
@@ -354,10 +365,14 @@ static int deltabase_atom_parser(struct ref_format *format, struct used_atom *at {if (arg)return strbuf_addf_ret (err,-1 ,_ ("%%(deltabase) does not take arguments" )); -if (atom->deref) +if (atom->deref) { oi_deref.info.delta_base_oid = &oi_deref.delta_base_oid; -else + oi_deref.need_get_object_info =1 ; + + }else { oi.info.delta_base_oid = &oi.delta_base_oid; + oi.need_get_object_info =1 ; + }return 0 ; }
@@ -735,10 +750,13 @@ static int parse_ref_filter_atom(struct ref_format *format, used_atom[at].type = valid_atom[i].cmp_type; used_atom[at].source = valid_atom[i].source;if (used_atom[at].source ==SOURCE_OBJ ) { -if (deref) +if (deref) { oi_deref.info.contentp = &oi_deref.content; -else + oi_deref.need_get_object_info =1 ; + }else { oi.info.contentp = &oi.content; + oi.need_get_object_info =1 ; + } }if (arg) { arg = used_atom[at].name + (arg - sp) +1 ;
@@ -1910,7 +1928,6 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)struct object *obj;int i;int ret; -struct object_info empty =OBJECT_INFO_INIT ;CALLOC_ARRAY (ref->value, used_atom_cnt);
@@ -2078,10 +2095,11 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err) oid_to_hex(&ref->objectname), ref->refname); } -if (need_tagged) +if (need_tagged) { oi.info.contentp = &oi.content; -if (!memcmp (&oi.info, &empty,sizeof (empty)) && - !memcmp (&oi_deref.info, &empty,sizeof (empty))) + oi.need_get_object_info =1 ; + } +if (!oi.need_get_object_info && !oi_deref.need_get_object_info)return 0 ;
Signature · 1 line
gitgitgadget
PATCH 26/27 ref-filter: reuse finnal buffer if no stack need
In reply to ZheNing Hu via GitGitGadgetFrom: ZheNing Hu <[email protected]>
Because finnal_buf will be cleared before every time we call format_ref_array_item(). So we actually add content to an empty strbuf, We can add the object's data directly to this finnal_buffer instead of adding objects' data to state.stack->output first, then copying to finnal_buf.
So add a can_reuse_final_buffer flag to struct ref_format to check if we are use %(align), %(end), %(if), %(then), %(else). if not, we can reuse the buf of finnal_buf.
This can bring 2% performance improvement for
git cat-file --batch.
--- ref-filter.c | 36 +++++++++++++++++++++++++++++------- ref-filter.h | 3 ++- 2 files changed, 31 insertions(+), 8 deletions(-)
ref-filter.c +29 -7
@@ -1083,6 +1083,13 @@ int verify_ref_format(struct ref_format *format)if (reject_atom (format->cat_file_mode, used_atom[at].atom_type))die (_ ("this command reject atom %%(%.*s)" ), (int )(ep - sp -2 ), sp +2 ); +if (used_atom[at].atom_type ==ATOM_ALIGN || + used_atom[at].atom_type ==ATOM_END || + used_atom[at].atom_type ==ATOM_IF || + used_atom[at].atom_type ==ATOM_THEN || + used_atom[at].atom_type ==ATOM_ELSE ) + format->can_reuse_final_buffer =0 ; + if ((format->quote_style == QUOTE_PYTHON || format->quote_style == QUOTE_SHELL || format->quote_style == QUOTE_TCL) &&
@@ -2710,7 +2717,14 @@ int format_ref_array_item(struct ref_array_item *info,int ret; state.quote_style = format->quote_style; -push_stack_element (&state.stack); +if (format->can_reuse_final_buffer) { +struct ref_formatting_stack *s =xmalloc (sizeof (struct ref_formatting_stack )); + s->output = *final_buf; + s->prev = state.stack; + state.stack = s; + }else { +push_stack_element (&state.stack); + } info->can_skip_parse_buffer = format->can_skip_parse_buffer; cp = format->format;
@@ -2722,9 +2736,10 @@ int format_ref_array_item(struct ref_array_item *info,if (cp < e->beg -2 )append_literal (cp, e->beg -2 , &state); -if ((ret =get_ref_atom_value (info, e->at, &atomv, error_buf)) || - atomv->handler (atomv, &state, error_buf)) { -pop_stack_element (&state.stack); +if (((ret =get_ref_atom_value (info, e->at, &atomv, error_buf)) || + atomv->handler (atomv, &state, error_buf))) { +if (!format->can_reuse_final_buffer) +pop_stack_element (&state.stack);return ret ? ret :-1 ; } cp = e->end +1 ;
@@ -2739,16 +2754,23 @@ int format_ref_array_item(struct ref_array_item *info,struct atom_value resetv =ATOM_VALUE_INIT ; resetv.s =GIT_COLOR_RESET ;if (append_atom (&resetv, &state, error_buf)) { -pop_stack_element (&state.stack); +if (!format->can_reuse_final_buffer) +pop_stack_element (&state.stack);return -1 ; } }if (state.stack->prev) { +assert (!format->can_reuse_final_buffer);pop_stack_element (&state.stack);return strbuf_addf_ret (error_buf,-1 ,_ ("format: %%(end) atom missing" )); } -strbuf_addbuf (final_buf, &state.stack->output); -pop_stack_element (&state.stack); +if (format->can_reuse_final_buffer) { + *final_buf = state.stack->output; +free (state.stack); + }else { +strbuf_addbuf (final_buf, &state.stack->output); +pop_stack_element (&state.stack); + }return 0 ; }
ref-filter.h +2 -1
@@ -94,10 +94,11 @@ struct ref_format {int can_skip_parse_buffer;/* Internal state to ref-filter */ int need_color_reset_at_eol; +int can_reuse_final_buffer;struct list_head parsed_atom_head; }; -#define REF_FORMAT_INIT { .use_color = -1, .can_skip_parse_buffer = 1 } +#define REF_FORMAT_INIT { .use_color = -1, .can_skip_parse_buffer = 1, .can_reuse_final_buffer = 1 }/* Macros for checking --merged and --no-merged options */ #define _OPT_MERGED_NO_MERGED (option, filter, h) \
Signature · 1 line
gitgitgadget