json-writer V8
v8 cover json-writer V8
From: Jeff Hostetler <[email protected]>
Here is V8 of my json-writer patches. Please replace the existing V5/V6/V7 version of the jh/json-writer branch with this one.
This version uses perl rather than python to test the generated JSON.
Jeff Hostetler (2):
json_writer: new routines to create data in JSON format
json-writer: t0019: add perl unit test
Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 274 +++++++++++++++++++++ t/t0019/parse_json.perl | 52 ++++ 6 files changed, 1432 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json.perl
Signature · 1 line
2.9.3
PATCH v8 1/2 json_writer: new routines to create data in JSON format
From: Jeff Hostetler <[email protected]>
Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.
The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.
--- Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 236 ++++++++++++++++++ 5 files changed, 1342 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees
@@ -820,6 +821,7 @@ LIB_OBJS += hashmap.o LIB_OBJS += help.o LIB_OBJS += hex.o LIB_OBJS += ident.o +LIB_OBJS += json-writer.o LIB_OBJS += kwset.o LIB_OBJS += levenshtein.o LIB_OBJS += line-log.o
json-writer.c new file +419 -0
419 lines not on this page · raw message
json-writer.h new file +113 -0
113 lines not on this page · raw message
t/helper/test-json-writer.c new file +572 -0
572 lines not on this page · raw message
t/t0019-json-writer.sh new file +236 -0
236 lines not on this page · raw message
Signature · 1 line
2.9.3
PATCH v8 2/2 json-writer: t0019: add perl unit test
In reply to [email protected]From: Jeff Hostetler <[email protected]>
Test json-writer output using perl script.
--- t/t0019-json-writer.sh | 38 ++++++++++++++++++++++++++++++++++++ t/t0019/parse_json.perl | 52 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 t/t0019/parse_json.perl
t/t0019-json-writer.sh +38 -0
@@ -233,4 +233,42 @@ test_expect_success 'inline array with no members' ' test_cmp expect actual ' +# As a sanity check, ask Perl to parse our generated JSON and recursively +# dump the resulting data in sorted order. Confirm that that matches our +# expectations. +test_expect_success 'parse JSON using Perl' ' + cat >expect <<-\EOF && + row[0].a abc + row[0].b 42 + row[0].sub1 hash + row[0].sub1.c 3.14 + row[0].sub1.d 1 + row[0].sub1.sub2 array + row[0].sub1.sub2[0] 0 + row[0].sub1.sub2[1] hash + row[0].sub1.sub2[1].g 0 + row[0].sub1.sub2[1].h 1 + row[0].sub1.sub2[2] null + EOF + test-json-writer >output.json \ + @object \ + @object-string a abc \ + @object-int b 42 \ + @object-object "sub1" \ + @object-double c 2 3.140 \ + @object-true d \ + @object-array "sub2" \ + @array-false \ + @array-object \ + @object-int g 0 \ + @object-int h 1 \ + @end \ + @array-null \ + @end \ + @end \ + @end && + perl "$TEST_DIRECTORY"/t0019/parse_json.perl <output.json >actual && + test_cmp expect actual +' + test_done
t/t0019/parse_json.perl new file +52 -0
52 lines not on this page · raw message
Signature · 1 line
2.9.3
Re: [PATCH v8 2/2] json-writer: t0019: add perl unit test
t/t0019/parse_json.perl line 4
Quoted 10 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:
Test json-writer output using perl script.
Signed-off-by Jeff Hostetler <[email protected]>---
t/t0019/parse_json.perl +4
-0@@ -0,0 +1,52 @@ +#!/usr/bin/perl +use strict; +use warnings; +use JSON;
This new script is going to have to be protected by a prerequisite since the JSON module is not part of the standard Perl installation, thus will not necessarily be installed everywhere (it isn't on any of my machines, for instance). Something like:
test_lazy_prereq PERLJSON '
perl -MJSON -e "exit 0"
'
which would be used like this:
test_expect_success PERLJSON 'parse JSON using Perl' '
...
'Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to [email protected]Quoted 13 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:
Add a series of
jw_routines and "structjson_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.[...]Signed-off-by Jeff Hostetler <[email protected]>---
t/t0019-json-writer.sh +5
-0@@ -0,0 +1,236 @@ +test_expect_success 'simple object' ' + cat >expect <<-\EOF && + {"a":"abc","b":42,"c":3.14,"d":true,"e":false,"f":null} + EOF + test-json-writer >actual \
t/t0019-json-writer.sh line 15
Quoted 10 lines
t/t0019-json-writer.sh +10
-0+ @object \ + @object-string a abc \ + @object-int b 42 \ + @object-double c 2 3.140 \ + @object-true d \ + @object-false e \ + @object-null f \ + @end && + test_cmp expect actual +'
To make it easier on people writing these tests, it might be nice for this to be less noisy by getting rid of "@" and "\". To get rid of "\", the test program could grab its script commands from stdin (one instruction per line) rather than from argv[]. For instance:
test-json-writer >actual <<-\EOF &&
object
object-string a abc
...
end
EOFNot a big deal, and certainly not worth a re-roll.
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to [email protected]Am 07.06.2018 um 16:12 schrieb [email protected]:
Quoted 40 lines
From: Jeff Hostetler <[email protected]>
Add a series of
jw_routines and "structjson_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.
Helped-by René Scharfe <[email protected]>Helped-by Wink Saville <[email protected]>Helped-by Ramsay Jones <[email protected]>Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 236 ++++++++++++++++++ 5 files changed, 1342 insertions(+)create mode 100644
json-writer.c
create mode 100644json-writer.h
create mode 100644t/helper/test-json-writer.c
create mode 100755t/t0019-json-writer.shMakefile +0
-0
Makefile TEST_PROGRAMS_NEED_X += test-fake-ssh · line 671
Quoted 6 lines
Makefile +1
-0@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer> TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees
This doesn't apply cleanly on master, because most test helpers have been combined into a single binary to reduce their disk footprint and link times. See efd71f8913 (t/helper: add an empty test-tool program) for the overall rationale.
test-json-writer could become a built-in as well, I think. You can see e.g. in c932a5ff28 (t/helper: merge test-string-list into test-tool) what needs to be done to convert a helper.
René
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to [email protected]Am 07.06.2018 um 16:12 schrieb [email protected]:
Review
Quoted 8 lines
From: Jeff Hostetler <[email protected]>
(quoted diff) +7
-0+/* + * Add comma if we have already seen a member at this level. + */ +static inline void maybe_add_comma(struct json_writer *jw) +{ + if (!jw->open_stack.len) + return;
This is impossible. maybe_add_comma() is only called directly after assert_in_object() and assert_in_object(), which abort if open_stack is empty.
Review
Quoted 4 lines
(quoted diff) +4
-0+ if (jw->first_stack.buf[jw->first_stack.len - 1] == '1') + jw->first_stack.buf[jw->first_stack.len - 1] = '0'; + else + strbuf_addch(&jw->json, ',');
You only need a binary flag to indicate if a comma is needed, not a stack. We need a comma at the current level if and only if we already wrote a child item. If we finish a level then we do need a comma at the previous level because we just wrote a sub-object or sub-array there. And that should cover all cases. Am I missing something?
I get a sense of déjà vu, by the way. :)
Here's what I mean:
--- json-writer.c | 17 ++++++----------- json-writer.h | 13 +++++-------- 2 files changed, 11 insertions(+), 19 deletions(-)
json-writer.c +6 -11
@@ -5,7 +5,7 @@ void jw_init(struct json_writer *jw) {strbuf_reset (&jw->json);strbuf_reset (&jw->open_stack); -strbuf_reset (&jw->first_stack); + jw->need_comma =0 ; jw->pretty =0 ; }
@@ -13,7 +13,6 @@ void jw_release(struct json_writer *jw) {strbuf_release (&jw->json);strbuf_release (&jw->open_stack); -strbuf_release (&jw->first_stack); }/*
@@ -69,7 +68,7 @@ static inline void begin(struct json_writer *jw, char ch_open, int pretty)strbuf_addch (&jw->json, ch_open);strbuf_addch (&jw->open_stack, ch_open); -strbuf_addch (&jw->first_stack,'1' ); + jw->need_comma =0 ; }/*
@@ -99,12 +98,10 @@ static inline void assert_in_array(const struct json_writer *jw)*/ static inline void maybe_add_comma (struct json_writer *jw) { -if (!jw->open_stack.len) -return ; -if (jw->first_stack.buf[jw->first_stack.len -1 ] =='1' ) - jw->first_stack.buf[jw->first_stack.len -1 ] ='0' ; -else +if (jw->need_comma)strbuf_addch (&jw->json,',' ); +else + jw->need_comma =1 ; } static inline void fmt_double(struct json_writer *jw, int precision,
@@ -397,8 +394,6 @@ void jw_end(struct json_writer *jw)char ch_open;int len; -if (jw->open_stack.len != jw->first_stack.len) -BUG ("jwon-writer: open_ and first_ stacks out of sync" );if (!jw->open_stack.len)BUG ("json-writer: too many jw_end(): '%s'" , jw->json.buf);
@@ -406,7 +401,7 @@ void jw_end(struct json_writer *jw) ch_open = jw->open_stack.buf[len];strbuf_setlen (&jw->open_stack, len); -strbuf_setlen (&jw->first_stack, len); + jw->need_comma =1 ;if (jw->pretty)strbuf_addch (&jw->json,'\n' );
json-writer.h +5 -8
@@ -59,18 +59,15 @@ struct json_writerstruct strbuf open_stack;/* -* Another simple stack of the currently open array and object -* forms. This is used in parallel to "open_stack" (same length). -* It contains a string of '1' and '0' where '1' indicates that -* the next child-element seen will be the first child (and does -* not need a leading comma). +* Indicates if the next child item needs a leading comma. +* Only the first item of arrays and objects doesn't need one. */ -struct strbuf first_stack; + unsigned int need_comma:1; -int pretty; + unsigned int pretty:1; }; -#define JSON_WRITER_INIT { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT, 0 } +#define JSON_WRITER_INIT { STRBUF_INIT, STRBUF_INIT, 0, 0 }void jw_init (struct json_writer *jw);void jw_release (struct json_writer *jw);
Signature · 1 line
2.17.1
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to Eric SunshineQuoted 30 lines · Eric Sunshine
Quoted 13 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:
Add a series of
jw_routines and "structjson_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.[...]Signed-off-by Jeff Hostetler <[email protected]>---
t/t0019-json-writer.sh +5
-0@@ -0,0 +1,236 @@ +test_expect_success 'simple object' ' + cat >expect <<-\EOF && + {"a":"abc","b":42,"c":3.14,"d":true,"e":false,"f":null} + EOF + test-json-writer >actual \t/t0019-json-writer.sh line 15
Quoted 10 lines
t/t0019-json-writer.sh +10
-0+ @object \ + @object-string a abc \ + @object-int b 42 \ + @object-double c 2 3.140 \ + @object-true d \ + @object-false e \ + @object-null f \ + @end && + test_cmp expect actual +'To make it easier on people writing these tests, it might be nice for this to be less noisy by getting rid of "@" and "\". To get rid of "\", the test program could grab its script commands from stdin (one instruction per line) rather than from argv[]. For instance:
Quoted 9 lines
test-json-writer >actual <<-\EOF && object object-string a abc ... end EOFNot a big deal, and certainly not worth a re-roll.
I hadn't thought about doing it that way. Might be a little easier to use. Let me take a look and see if it would be much work to switch.
Thanks
Jeff
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to [email protected]Am 07.06.2018 um 16:12 schrieb [email protected]:
Quoted 10 lines
Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 236 ++++++++++++++++++ 5 files changed, 1342 insertions(+)create mode 100644
json-writer.c
create mode 100644json-writer.h
create mode 100644t/helper/test-json-writer.c
create mode 100755t/t0019-json-writer.sh
$ git grep 'static inline' '*json*' json-writer.c:static inline void indent_pretty(struct json_writer *jw) json-writer.c:static inline void begin(struct json_writer *jw, char ch_open, int pretty) json-writer.c:static inline void assert_in_object(const struct json_writer *jw, const char *key) json-writer.c:static inline void assert_in_array(const struct json_writer *jw) json-writer.c:static inline void maybe_add_comma(struct json_writer *jw) json-writer.c:static inline void fmt_double(struct json_writer *jw, int precision, json-writer.c:static inline void object_common(struct json_writer *jw, const char *key) json-writer.c:static inline void array_common(struct json_writer *jw) json-writer.c:static inline void assert_is_terminated(const struct json_writer *jw) t/helper/test-json-writer.c:static inline int scripted(int argc, const char **argv) t/helper/test-json-writer.c:static inline int my_usage(void)
Do you need all those inline keywords? I'd rather leave the decision about inlining to the compiler and (via optimization flags) the user as much as possible. Not a biggie, but the high frequency of that word made me blink..
René
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
Quoted 25 lines · René Scharfe
Am 07.06.2018 um 16:12 schrieb [email protected]:
Quoted 10 lines
Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 236 ++++++++++++++++++ 5 files changed, 1342 insertions(+)create mode 100644
json-writer.c
create mode 100644json-writer.h
create mode 100644t/helper/test-json-writer.c
create mode 100755t/t0019-json-writer.sh$ git grep 'static inline' '*json*' json-writer.c:static inline void indent_pretty(struct json_writer *jw) json-writer.c:static inline void begin(struct json_writer *jw, char ch_open, int pretty) json-writer.c:static inline void assert_in_object(const struct json_writer *jw, const char *key) json-writer.c:static inline void assert_in_array(const struct json_writer *jw) json-writer.c:static inline void maybe_add_comma(struct json_writer *jw) json-writer.c:static inline void fmt_double(struct json_writer *jw, int precision, json-writer.c:static inline void object_common(struct json_writer *jw, const char *key) json-writer.c:static inline void array_common(struct json_writer *jw) json-writer.c:static inline void assert_is_terminated(const struct json_writer *jw) t/helper/test-json-writer.c:static inline int scripted(int argc, const char **argv) t/helper/test-json-writer.c:static inline int my_usage(void)
Quoted 7 lines
Do you need all those inline keywords? I'd rather leave the decision about inlining to the compiler and (via optimization flags) the user as much as possible. Not a biggie, but the high frequency of that word made me blink..
René
I was just trying to match the patterns I found elsewhere in the code.
$ grep "static inline" *.c builtin/*.c | wc -l
111But yeah, it's no big deal. I can remove them.
Jeff
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to René ScharfeQuoted 4 lines · René Scharfe
Am 07.06.2018 um 16:12 schrieb [email protected]:
Quoted 3 lines
From: Jeff Hostetler <[email protected]>
Add a series of
jw_routines and "structjson_writer" structure to compose
[...]
Quoted 10 lines
Quoted 4 lines
TEST_PROGRAMS_NEED_X+= test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer>TEST_PROGRAMS_NEED_X+= test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X+= test-line-buffer
TEST_PROGRAMS_NEED_X+= test-match-treesThis doesn't apply cleanly on master, because most test helpers have been combined into a single binary to reduce their disk footprint and link times. See
efd71f8913(t/helper: add an empty test-tool program) for the overall rationale.
Quoted 6 lines
test-json-writer could become a built-in as well, I think. You can see e.g. in
c932a5ff28(t/helper: merge test-string-list into test-tool) what needs to be done to convert a helper.René
You're right, the test helper framework changed since I started this patch series. I was trying to keep the same parent commit as my V1 to make it easier to compare, but that's not working out so well.
I'll move it forward to the current master and fix it up.
Thanks,
Jeff
Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format
In reply to René ScharfeQuoted 2 lines · René Scharfe
Am 07.06.2018 um 16:12 schrieb [email protected]:
Quoted 1 line
From: Jeff Hostetler <[email protected]>
[...]
Quoted 11 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ if (jw->first_stack.buf[jw->first_stack.len - 1] == '1') + jw->first_stack.buf[jw->first_stack.len - 1] = '0'; + else + strbuf_addch(&jw->json, ',');You only need a binary flag to indicate if a comma is needed, not a stack. We need a comma at the current level if and only if we already wrote a child item. If we finish a level then we do need a comma at the previous level because we just wrote a sub-object or sub-array there. And that should cover all cases. Am I missing something?
Quoted 1 line
I get a sense of déjà vu, by the way. :)
[...]
Yes, your way is simpler. I'll update and re-roll. (And yes, we did discuss this earlier. I found a problem with my first version where it wouldn't handle empty sub-items, but you found the missing piece.)
Thanks
Jeff
Re: [PATCH v8 2/2] json-writer: t0019: add perl unit test
In reply to Eric SunshineQuoted 21 lines · Eric Sunshine
t/t0019/parse_json.perl line 4
Quoted 10 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:
Test json-writer output using perl script.
Signed-off-by Jeff Hostetler <[email protected]>---
t/t0019/parse_json.perl +4
-0@@ -0,0 +1,52 @@ +#!/usr/bin/perl +use strict; +use warnings; +use JSON;This new script is going to have to be protected by a prerequisite since the JSON module is not part of the standard Perl installation, thus will not necessarily be installed everywhere (it isn't on any of my machines, for instance). Something like:
test_lazy_prereqPERLJSON '
perl -MJSON -e "exit 0"
'
Quoted 6 lines
which would be used like this:
test_expect_success PERLJSON 'parse JSON using Perl' ' ... '
will do. thanks!
Jeff