Skip to contentSkip to conversation

json-writer V8

PATCH Superseded [email protected] 3 participants 13 messages Jun 7, 2018 – Jun 11, 2018 Kbuild

v8 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v8 cover json-writer V8

From: Jeff Hostetler <[email protected]>

Here is V8 of my json-writer patches. Please replace the existing V5/V6/V7 version of the jh/json-writer branch with this one.

This version uses perl rather than python to test the generated JSON.

Jeff Hostetler (2):
json_writer: new routines to create data in JSON format
json-writer: t0019: add perl unit test

 Makefile                    |   2 +
 json-writer.c               | 419 ++++++++++++++++++++++++++++++++
 json-writer.h               | 113 +++++++++
 t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 274 +++++++++++++++++++++
 t/t0019/parse_json.perl     |  52 ++++
 6 files changed, 1432 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh
 create mode 100644 t/t0019/parse_json.perl
Signature · 1 line
2.9.3
#1 lore raw

PATCH v8 1/2 json_writer: new routines to create data in JSON format

From: Jeff Hostetler <[email protected]>

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.

The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.

---
 Makefile                    |   2 +
 json-writer.c               | 419 ++++++++++++++++++++++++++++++++
 json-writer.h               | 113 +++++++++
 t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 236 ++++++++++++++++++
 5 files changed, 1342 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
index a1d8775..4ae6946 100644
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
 TEST_PROGRAMS_NEED_X += test-genrandom
 TEST_PROGRAMS_NEED_X += test-hashmap
 TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
 TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
 TEST_PROGRAMS_NEED_X += test-line-buffer
 TEST_PROGRAMS_NEED_X += test-match-trees
@@ -820,6 +821,7 @@ LIB_OBJS += hashmap.o
 LIB_OBJS += help.o
 LIB_OBJS += hex.o
 LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
 LIB_OBJS += kwset.o
 LIB_OBJS += levenshtein.o
 LIB_OBJS += line-log.o
json-writer.c new file +419 -0

419 lines not on this page · raw message

json-writer.h new file +113 -0

113 lines not on this page · raw message

t/helper/test-json-writer.c new file +572 -0

572 lines not on this page · raw message

t/t0019-json-writer.sh new file +236 -0

236 lines not on this page · raw message

Signature · 1 line
2.9.3
#10 lore raw

PATCH v8 2/2 json-writer: t0019: add perl unit test

In reply to [email protected]

From: Jeff Hostetler <[email protected]>

Test json-writer output using perl script.

---
 t/t0019-json-writer.sh  | 38 ++++++++++++++++++++++++++++++++++++
 t/t0019/parse_json.perl | 52 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
 create mode 100644 t/t0019/parse_json.perl
t/t0019-json-writer.sh +38 -0
index c9c2e23..fd61fe4 100755
@@ -233,4 +233,42 @@ test_expect_success 'inline array with no members' '
 	test_cmp expect actual
 '
 
+# As a sanity check, ask Perl to parse our generated JSON and recursively
+# dump the resulting data in sorted order.  Confirm that that matches our
+# expectations.
+test_expect_success 'parse JSON using Perl' '
+	cat >expect <<-\EOF &&
+	row[0].a abc
+	row[0].b 42
+	row[0].sub1 hash
+	row[0].sub1.c 3.14
+	row[0].sub1.d 1
+	row[0].sub1.sub2 array
+	row[0].sub1.sub2[0] 0
+	row[0].sub1.sub2[1] hash
+	row[0].sub1.sub2[1].g 0
+	row[0].sub1.sub2[1].h 1
+	row[0].sub1.sub2[2] null
+	EOF
+	test-json-writer >output.json \
+		@object \
+			@object-string a abc \
+			@object-int b 42 \
+			@object-object "sub1" \
+				@object-double c 2 3.140 \
+				@object-true d \
+				@object-array "sub2" \
+					@array-false \
+					@array-object \
+						@object-int g 0 \
+						@object-int h 1 \
+					@end \
+					@array-null \
+				@end \
+			@end \
+		@end &&
+	perl "$TEST_DIRECTORY"/t0019/parse_json.perl <output.json >actual &&
+	test_cmp expect actual
+'
+
 test_done
t/t0019/parse_json.perl new file +52 -0

52 lines not on this page · raw message

Signature · 1 line
2.9.3
#11 lore raw

Re: [PATCH v8 2/2] json-writer: t0019: add perl unit test

t/t0019/parse_json.perl line 4

Quoted 10 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:

Test json-writer output using perl script.

---

t/t0019/parse_json.perl +4 -0
@@ -0,0 +1,52 @@
+#!/usr/bin/perl
+use strict;
+use warnings;
+use JSON;

This new script is going to have to be protected by a prerequisite since the JSON module is not part of the standard Perl installation, thus will not necessarily be installed everywhere (it isn't on any of my machines, for instance). Something like:

test_lazy_prereq PERLJSON '
perl -MJSON -e "exit 0"
'

which would be used like this:

    test_expect_success PERLJSON 'parse JSON using Perl' '
        ...
    '
#2 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to [email protected]
Quoted 13 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

---

t/t0019-json-writer.sh +5 -0
@@ -0,0 +1,236 @@
+test_expect_success 'simple object' '
+       cat >expect <<-\EOF &&
+       {"a":"abc","b":42,"c":3.14,"d":true,"e":false,"f":null}
+       EOF
+       test-json-writer >actual \

t/t0019-json-writer.sh line 15

Quoted 10 lines
t/t0019-json-writer.sh +10 -0
+               @object \
+                       @object-string a abc \
+                       @object-int b 42 \
+                       @object-double c 2 3.140 \
+                       @object-true d \
+                       @object-false e \
+                       @object-null f \
+               @end &&
+       test_cmp expect actual
+'

To make it easier on people writing these tests, it might be nice for this to be less noisy by getting rid of "@" and "\". To get rid of "\", the test program could grab its script commands from stdin (one instruction per line) rather than from argv[]. For instance:

    test-json-writer >actual <<-\EOF &&
        object
            object-string a abc
            ...
        end
    EOF

Not a big deal, and certainly not worth a re-roll.

#4 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to [email protected]

Am 07.06.2018 um 16:12 schrieb [email protected]:

Quoted 40 lines

From: Jeff Hostetler <[email protected]>

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.

The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.

---
  Makefile                    |   2 +
  json-writer.c               | 419 ++++++++++++++++++++++++++++++++
  json-writer.h               | 113 +++++++++
  t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
  t/t0019-json-writer.sh      | 236 ++++++++++++++++++
  5 files changed, 1342 insertions(+)

create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh

Makefile +0 -0
index a1d8775..4ae6946 100644

Makefile TEST_PROGRAMS_NEED_X += test-fake-ssh · line 671

Quoted 6 lines
Makefile +1 -0
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
 TEST_PROGRAMS_NEED_X += test-genrandom
 TEST_PROGRAMS_NEED_X += test-hashmap
 TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer>  TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
 TEST_PROGRAMS_NEED_X += test-line-buffer
 TEST_PROGRAMS_NEED_X += test-match-trees

This doesn't apply cleanly on master, because most test helpers have been combined into a single binary to reduce their disk footprint and link times. See efd71f8913 (t/helper: add an empty test-tool program) for the overall rationale.

test-json-writer could become a built-in as well, I think. You can see e.g. in c932a5ff28 (t/helper: merge test-string-list into test-tool) what needs to be done to convert a helper.

René

#6 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to [email protected]

Am 07.06.2018 um 16:12 schrieb [email protected]:

Review

Quoted 8 lines

From: Jeff Hostetler <[email protected]>

(quoted diff) +7 -0
+/*
+ * Add comma if we have already seen a member at this level.
+ */
+static inline void maybe_add_comma(struct json_writer *jw)
+{
+	if (!jw->open_stack.len)
+		return;

This is impossible. maybe_add_comma() is only called directly after assert_in_object() and assert_in_object(), which abort if open_stack is empty.

Review

Quoted 4 lines
(quoted diff) +4 -0
+	if (jw->first_stack.buf[jw->first_stack.len - 1] == '1')
+		jw->first_stack.buf[jw->first_stack.len - 1] = '0';
+	else
+		strbuf_addch(&jw->json, ',');

You only need a binary flag to indicate if a comma is needed, not a stack. We need a comma at the current level if and only if we already wrote a child item. If we finish a level then we do need a comma at the previous level because we just wrote a sub-object or sub-array there. And that should cover all cases. Am I missing something?

I get a sense of déjà vu, by the way. :)

Here's what I mean:

---
 json-writer.c | 17 ++++++-----------
 json-writer.h | 13 +++++--------
 2 files changed, 11 insertions(+), 19 deletions(-)
json-writer.c +6 -11
index f35ce1912a..14a4e89188 100644
@@ -5,7 +5,7 @@ void jw_init(struct json_writer *jw)
 {
 	strbuf_reset(&jw->json);
 	strbuf_reset(&jw->open_stack);
-	strbuf_reset(&jw->first_stack);
+	jw->need_comma = 0;
 	jw->pretty = 0;
 }
 
@@ -13,7 +13,6 @@ void jw_release(struct json_writer *jw)
 {
 	strbuf_release(&jw->json);
 	strbuf_release(&jw->open_stack);
-	strbuf_release(&jw->first_stack);
 }
 
 /*
@@ -69,7 +68,7 @@ static inline void begin(struct json_writer *jw, char ch_open, int pretty)
 	strbuf_addch(&jw->json, ch_open);
 
 	strbuf_addch(&jw->open_stack, ch_open);
-	strbuf_addch(&jw->first_stack, '1');
+	jw->need_comma = 0;
 }
 
 /*
@@ -99,12 +98,10 @@ static inline void assert_in_array(const struct json_writer *jw)
  */
 static inline void maybe_add_comma(struct json_writer *jw)
 {
-	if (!jw->open_stack.len)
-		return;
-	if (jw->first_stack.buf[jw->first_stack.len - 1] == '1')
-		jw->first_stack.buf[jw->first_stack.len - 1] = '0';
-	else
+	if (jw->need_comma)
 		strbuf_addch(&jw->json, ',');
+	else
+		jw->need_comma = 1;
 }
 
 static inline void fmt_double(struct json_writer *jw, int precision,
@@ -397,8 +394,6 @@ void jw_end(struct json_writer *jw)
 	char ch_open;
 	int len;
 
-	if (jw->open_stack.len != jw->first_stack.len)
-		BUG("jwon-writer: open_ and first_ stacks out of sync");
 	if (!jw->open_stack.len)
 		BUG("json-writer: too many jw_end(): '%s'", jw->json.buf);
 
@@ -406,7 +401,7 @@ void jw_end(struct json_writer *jw)
 	ch_open = jw->open_stack.buf[len];
 
 	strbuf_setlen(&jw->open_stack, len);
-	strbuf_setlen(&jw->first_stack, len);
+	jw->need_comma = 1;
 
 	if (jw->pretty)
 		strbuf_addch(&jw->json, '\n');
json-writer.h +5 -8
index af9c2612f8..c437462ba8 100644
@@ -59,18 +59,15 @@ struct json_writer
 	struct strbuf open_stack;
 
 	/*
-	 * Another simple stack of the currently open array and object
-	 * forms.  This is used in parallel to "open_stack" (same length).
-	 * It contains a string of '1' and '0' where '1' indicates that
-	 * the next child-element seen will be the first child (and does
-	 * not need a leading comma).
+	 * Indicates if the next child item needs a leading comma.
+	 * Only the first item of arrays and objects doesn't need one.
 	 */
-	struct strbuf first_stack;
+	unsigned int need_comma:1;
 
-	int pretty;
+	unsigned int pretty:1;
 };
 
-#define JSON_WRITER_INIT { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT, 0 }
+#define JSON_WRITER_INIT { STRBUF_INIT, STRBUF_INIT, 0, 0 }
 
 void jw_init(struct json_writer *jw);
 void jw_release(struct json_writer *jw);
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to Eric Sunshine
Quoted 30 lines · Eric Sunshine
Quoted 13 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

---

t/t0019-json-writer.sh +5 -0
@@ -0,0 +1,236 @@
+test_expect_success 'simple object' '
+       cat >expect <<-\EOF &&
+       {"a":"abc","b":42,"c":3.14,"d":true,"e":false,"f":null}
+       EOF
+       test-json-writer >actual \

t/t0019-json-writer.sh line 15

Quoted 10 lines
t/t0019-json-writer.sh +10 -0
+               @object \
+                       @object-string a abc \
+                       @object-int b 42 \
+                       @object-double c 2 3.140 \
+                       @object-true d \
+                       @object-false e \
+                       @object-null f \
+               @end &&
+       test_cmp expect actual
+'

To make it easier on people writing these tests, it might be nice for this to be less noisy by getting rid of "@" and "\". To get rid of "\", the test program could grab its script commands from stdin (one instruction per line) rather than from argv[]. For instance:

Quoted 9 lines
     test-json-writer >actual <<-\EOF &&
         object
             object-string a abc
             ...
         end
     EOF

Not a big deal, and certainly not worth a re-roll.

I hadn't thought about doing it that way. Might be a little easier to use. Let me take a look and see if it would be much work to switch.

Thanks
Jeff

#8 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to [email protected]

Am 07.06.2018 um 16:12 schrieb [email protected]:

Quoted 10 lines
  Makefile                    |   2 +
  json-writer.c               | 419 ++++++++++++++++++++++++++++++++
  json-writer.h               | 113 +++++++++
  t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
  t/t0019-json-writer.sh      | 236 ++++++++++++++++++
  5 files changed, 1342 insertions(+)

create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh

$ git grep 'static inline' '*json*'
json-writer.c:static inline void indent_pretty(struct json_writer *jw)
json-writer.c:static inline void begin(struct json_writer *jw, char ch_open, int pretty)
json-writer.c:static inline void assert_in_object(const struct json_writer *jw, const char *key)
json-writer.c:static inline void assert_in_array(const struct json_writer *jw)
json-writer.c:static inline void maybe_add_comma(struct json_writer *jw)
json-writer.c:static inline void fmt_double(struct json_writer *jw, int precision,
json-writer.c:static inline void object_common(struct json_writer *jw, const char *key)
json-writer.c:static inline void array_common(struct json_writer *jw)
json-writer.c:static inline void assert_is_terminated(const struct json_writer *jw)
t/helper/test-json-writer.c:static inline int scripted(int argc, const char **argv)
t/helper/test-json-writer.c:static inline int my_usage(void)

Do you need all those inline keywords? I'd rather leave the decision about inlining to the compiler and (via optimization flags) the user as much as possible. Not a biggie, but the high frequency of that word made me blink..

René

#9 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

Quoted 25 lines · René Scharfe

Am 07.06.2018 um 16:12 schrieb [email protected]:

Quoted 10 lines
   Makefile                    |   2 +
   json-writer.c               | 419 ++++++++++++++++++++++++++++++++
   json-writer.h               | 113 +++++++++
   t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
   t/t0019-json-writer.sh      | 236 ++++++++++++++++++
   5 files changed, 1342 insertions(+)

create mode 100644 json-writer.c
create mode 100644 json-writer.h
create mode 100644 t/helper/test-json-writer.c
create mode 100755 t/t0019-json-writer.sh

$ git grep 'static inline' '*json*'
json-writer.c:static inline void indent_pretty(struct json_writer *jw)
json-writer.c:static inline void begin(struct json_writer *jw, char ch_open, int pretty)
json-writer.c:static inline void assert_in_object(const struct json_writer *jw, const char *key)
json-writer.c:static inline void assert_in_array(const struct json_writer *jw)
json-writer.c:static inline void maybe_add_comma(struct json_writer *jw)
json-writer.c:static inline void fmt_double(struct json_writer *jw, int precision,
json-writer.c:static inline void object_common(struct json_writer *jw, const char *key)
json-writer.c:static inline void array_common(struct json_writer *jw)
json-writer.c:static inline void assert_is_terminated(const struct json_writer *jw)
t/helper/test-json-writer.c:static inline int scripted(int argc, const char **argv)
t/helper/test-json-writer.c:static inline int my_usage(void)
Quoted 7 lines

Do you need all those inline keywords? I'd rather leave the decision about inlining to the compiler and (via optimization flags) the user as much as possible. Not a biggie, but the high frequency of that word made me blink..

René

I was just trying to match the patterns I found elsewhere in the code.

    $ grep "static inline" *.c builtin/*.c | wc -l
    111

But yeah, it's no big deal. I can remove them.

Jeff

#5 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to René Scharfe
Quoted 4 lines · René Scharfe

Am 07.06.2018 um 16:12 schrieb [email protected]:

Quoted 3 lines

From: Jeff Hostetler <[email protected]>

Add a series of jw_ routines and "struct json_writer" structure to compose

[...]

Quoted 10 lines
Quoted 4 lines

TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer> TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
TEST_PROGRAMS_NEED_X += test-line-buffer
TEST_PROGRAMS_NEED_X += test-match-trees

This doesn't apply cleanly on master, because most test helpers have been combined into a single binary to reduce their disk footprint and link times. See efd71f8913 (t/helper: add an empty test-tool program) for the overall rationale.

Quoted 6 lines

test-json-writer could become a built-in as well, I think. You can see e.g. in c932a5ff28 (t/helper: merge test-string-list into test-tool) what needs to be done to convert a helper.

René

You're right, the test helper framework changed since I started this patch series. I was trying to keep the same parent commit as my V1 to make it easier to compare, but that's not working out so well.

I'll move it forward to the current master and fix it up.

Thanks,
Jeff

#7 lore raw

Re: [PATCH v8 1/2] json_writer: new routines to create data in JSON format

In reply to René Scharfe
Quoted 2 lines · René Scharfe

Am 07.06.2018 um 16:12 schrieb [email protected]:

Quoted 1 line

From: Jeff Hostetler <[email protected]>

[...]

Quoted 11 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+	if (jw->first_stack.buf[jw->first_stack.len - 1] == '1')
+		jw->first_stack.buf[jw->first_stack.len - 1] = '0';
+	else
+		strbuf_addch(&jw->json, ',');

You only need a binary flag to indicate if a comma is needed, not a stack. We need a comma at the current level if and only if we already wrote a child item. If we finish a level then we do need a comma at the previous level because we just wrote a sub-object or sub-array there. And that should cover all cases. Am I missing something?

Quoted 1 line

I get a sense of déjà vu, by the way. :)

[...]

Yes, your way is simpler. I'll update and re-roll. (And yes, we did discuss this earlier. I found a problem with my first version where it wouldn't handle empty sub-items, but you found the missing piece.)

Thanks
Jeff

#12 lore raw

Re: [PATCH v8 2/2] json-writer: t0019: add perl unit test

In reply to Eric Sunshine
Quoted 21 lines · Eric Sunshine

t/t0019/parse_json.perl line 4

Quoted 10 lines · On Thu, Jun 7, 2018 at 10:12 AM, <[email protected]> wrote:

Test json-writer output using perl script.

---

t/t0019/parse_json.perl +4 -0
@@ -0,0 +1,52 @@
+#!/usr/bin/perl
+use strict;
+use warnings;
+use JSON;

This new script is going to have to be protected by a prerequisite since the JSON module is not part of the standard Perl installation, thus will not necessarily be installed everywhere (it isn't on any of my machines, for instance). Something like:

test_lazy_prereq PERLJSON '
perl -MJSON -e "exit 0"
'

Quoted 6 lines

which would be used like this:

     test_expect_success PERLJSON 'parse JSON using Perl' '
         ...
     '

will do. thanks!
Jeff