Skip to contentSkip to conversation

routines to generate JSON data

PATCH Superseded [email protected] 7 participants 14 messages Mar 26, 2018 – Jun 4, 2018 Kbuild

v4 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 routines to generate JSON data

From: Jeff Hostetler <[email protected]>

This is version 4 of my JSON data format routines.

This version adds a "pretty" formatted output. I consider this to be mainly for debugging, but worth keeping available in release builds.

I simplified the stack-level tracing as suggested by René Scharfe and hinted at by Peff.

I converted the _double() routines to take an integer precision rather than a format specification and build a known-to-be-good format string to minimize the __attribute__(...) issues raised by René Scharfe.

It fixes the PRIuMAX and "void inline" compiler warnings on OSX that were reported by Wink Saville and Ramsay Jones. And resolved the "sparse" warnings repoted by Ramsay Jones.

And I updated the commit message and header file documnetation to address the JSON-like (Unicode limitations) mentioned by Jonathan Nieder.

Jeff Hostetler (1):
json_writer: new routines to create data in JSON format

 Makefile                    |   2 +
 json-writer.c               | 395 +++++++++++++++++++++++++++++
 json-writer.h               |  92 +++++++
 t/helper/test-json-writer.c | 590 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 253 +++++++++++++++++++
 5 files changed, 1332 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh
Signature · 1 line
2.9.3
#1 lore raw

PATCH v4 1/1 json_writer: new routines to create data in JSON format

From: Jeff Hostetler <[email protected]>

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.

The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.

---
 Makefile                    |   2 +
 json-writer.c               | 395 +++++++++++++++++++++++++++++
 json-writer.h               |  92 +++++++
 t/helper/test-json-writer.c | 590 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 253 +++++++++++++++++++
 5 files changed, 1332 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
index 1a9b23b..57f58e6 100644
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
 TEST_PROGRAMS_NEED_X += test-genrandom
 TEST_PROGRAMS_NEED_X += test-hashmap
 TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
 TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
 TEST_PROGRAMS_NEED_X += test-line-buffer
 TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o
 LIB_OBJS += help.o
 LIB_OBJS += hex.o
 LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
 LIB_OBJS += kwset.o
 LIB_OBJS += levenshtein.o
 LIB_OBJS += line-log.o
json-writer.c new file +395 -0

395 lines not on this page · raw message

json-writer.h new file +92 -0

92 lines not on this page · raw message

t/helper/test-json-writer.c new file +590 -0

590 lines not on this page · raw message

t/t0019-json-writer.sh new file +253 -0

253 lines not on this page · raw message

Signature · 1 line
2.9.3
#2 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 23 lines · On 26/03/18 15:31, [email protected] wrote:

From: Jeff Hostetler <[email protected]>

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.

The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.

Quoted 12 lines
---
 Makefile                    |   2 +
 json-writer.c               | 395 +++++++++++++++++++++++++++++
 json-writer.h               |  92 +++++++
 t/helper/test-json-writer.c | 590 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 253 +++++++++++++++++++
 5 files changed, 1332 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh

[snip]

Thanks, this version fixes all issues I had (with the compilation and sparse warnings).

[Was using UINT64_C(0xffffffffffffffff) a problem on windows?]

ATB,
Ramsay Jones

#3 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 8 lines · Ramsay Jones
Quoted 1 line · On 26/03/18 15:31, [email protected] wrote:

From: Jeff Hostetler <[email protected]>

[snip]

Thanks, this version fixes all issues I had (with the compilation and sparse warnings).

[Was using UINT64_C(0xffffffffffffffff) a problem on windows?]

Thanks for the confirmation.

I was building on Linux. I haven't tried using UINT64_C() for anything, but I'll keep that in mind next time.

Thanks,
Jeff

#4 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

In reply to Ramsay Jones
Quoted 9 lines · Ramsay Jones
Quoted 2 lines · On 26/03/18 15:31, [email protected] wrote:

From: Jeff Hostetler <[email protected]>

[snip]

Thanks, this version fixes all issues I had (with the compilation and sparse warnings).

[Was using UINT64_C(0xffffffffffffffff) a problem on windows?]

BTW, I forgot to mention that you had some whitespace problems with this patch, viz:

  $ git log --oneline -1 --check master-json
  ab643d838 (master-json) json_writer: new routines to create data in JSON format
  t/helper/test-json-writer.c:280: trailing whitespace.
  + */ 
  t/t0019-json-writer.sh:179: indent with spaces.
  +        "g": 0,
  t/t0019-json-writer.sh:180: indent with spaces.
  +        "h": 1
  $ 

ATB,
Ramsay Jones

#5 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 8 lines · Ramsay Jones
Quoted 5 lines · Ramsay Jones
Quoted 2 lines · On 26/03/18 15:31, [email protected] wrote:

From: Jeff Hostetler <[email protected]>

[snip]

Quoted 7 lines
Quoted 4 lines

Thanks, this version fixes all issues I had (with the compilation and sparse warnings).

[Was using UINT64_C(0xffffffffffffffff) a problem on windows?]

BTW, I forgot to mention that you had some whitespace problems with this patch, viz:

I ran "make sparse" on this and it did not complain about any of this.

What command do you run to get these messages?
I know they appear as red in diffs (and I usually double check that), but I had not seen a command to complain about them like this.

Quoted 10 lines
   $ git log --oneline -1 --check master-json
   ab643d838 (master-json) json_writer: new routines to create data in JSON format
   t/helper/test-json-writer.c:280: trailing whitespace.
   + */
   t/t0019-json-writer.sh:179: indent with spaces.
   +        "g": 0,
   t/t0019-json-writer.sh:180: indent with spaces.
   +        "h": 1
   $

the leading spaces are required in this case.
the pretty json output contains 8 spaces for that sub-structure not a tab. is there a preferred way to denote this in the test script?

Jeff

#6 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 5 lines

the leading spaces are required in this case.
the pretty json output contains 8 spaces for that sub-structure not a tab. is there a preferred way to denote this in the test script?

Jeff

I've used "git diff --check" which I got from Documentation/SubmittingPatches.

-- Wink

#7 lore raw

RE: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 8 lines · Wink Saville
Quoted 5 lines

the leading spaces are required in this case.
the pretty json output contains 8 spaces for that sub-structure not a tab. is there a preferred way to denote this in the test script?

Jeff

I've used "git diff --check" which I got from
Documentation/SubmittingPatches.

While I have not done this in the git suite, my own suites use something along the lines of the following when I need (and have to validate) exact spacing at the beginning of lines in expected output:

cat <<-EOF | sed "1,\$s/_/ /g" >expect
{ level1 ...
________{ level2
Etc.
EOF

Providing you don't use _ elsewhere in the output. It's a bit hacky but works.

Cheers,
Randall

-- Brief whoami:
NonStop developer since approximately 211288444200000000 UNIX developer since approximately 421664400
-- In my real life, I talk too much.

#9 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

In reply to Jeff Hostetler
Quoted 14 lines · Jeff Hostetler
Quoted 11 lines · Ramsay Jones
Quoted 8 lines · Ramsay Jones
Quoted 2 lines · On 26/03/18 15:31, [email protected] wrote:

From: Jeff Hostetler <[email protected]>

[snip]

Thanks, this version fixes all issues I had (with the compilation and sparse warnings).

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

[Was using UINT64_C(0xffffffffffffffff) a problem on windows?]

BTW, I forgot to mention that you had some whitespace problems with this patch, viz:

I ran "make sparse" on this and it did not complain about any of this.

No, sparse doesn't check for whitespace issues.

Quoted 1 line

What command do you run to get these messages?

'git am' told me when I applied the patch from the ML, and ...

Quoted 2 lines

I know they appear as red in diffs (and I usually double check that), but I had not seen a command to complain about them like this.

... since I applied the patch yesterday, it was easier to demonstrate the issue today using this command:

Quoted 10 lines
Quoted 9 lines
   $ git log --oneline -1 --check master-json
   ab643d838 (master-json) json_writer: new routines to create data in JSON format
   t/helper/test-json-writer.c:280: trailing whitespace.
   + */
   t/t0019-json-writer.sh:179: indent with spaces.
   +        "g": 0,
   t/t0019-json-writer.sh:180: indent with spaces.
   +        "h": 1
   $
Quoted 3 lines

the leading spaces are required in this case.
the pretty json output contains 8 spaces for that sub-structure not a tab. is there a preferred way to denote this in the test script?

OK, I should have looked at the output more closely! :(

Only the trailing whitespace in test-json-writer.c needs to be addressed then.

ATB,
Ramsay Jones

#10 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

In reply to Ramsay Jones
Quoted 12 lines · Ramsay Jones

BTW, I forgot to mention that you had some whitespace problems with this patch, viz:

  $ git log --oneline -1 --check master-json
  ab643d838 (master-json) json_writer: new routines to create data in JSON format
  t/helper/test-json-writer.c:280: trailing whitespace.
  + */ 
  t/t0019-json-writer.sh:179: indent with spaces.
  +        "g": 0,
  t/t0019-json-writer.sh:180: indent with spaces.
  +        "h": 1
  $ 

Yes, and the here-doc that shows expected output looked somewhat old-fashioned without using <<- indent.

Writing it in a way like this might be a reasonable workaround for "indent with spaces", which is only about the leading blank used to indent the line:

<--HT-->sed -e "s/^\|//" <<-\EOF

<--HT-->|  ...
<--HT-->|        "g": 0,
<--HT-->|        "h": 0,
<--HT-->|  ...

<--HT-->EOF

That is, (1) Use the same number of HT as the line that begins the here-doc to indent the expected output; (2) but explicitly mark the left-most column with '|' or something; and then (3) write 8 or more spaces liberally as needed to reproduce the expected output.

I called it a "workaround", as another possibility is to loosen the whitespace rule in t/.gitattributes; we _could_ declare that indent with spaces is OK in these scripts as they contain many expected output examples in here-doc form.

The trailing whitespace after closing C-comment is simply a style violation that is not excuable (I think I've dealt with it when I queued the patch); it should just be fixed.

Thanks.

#8 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

In reply to Randall S. Becker
Quoted 7 lines · Randall S. Becker

cat <<-EOF | sed "1,\$s/_/ /g" >expect
{ level1 ...
________{ level2
Etc.
EOF

Providing you don't use _ elsewhere in the output. It's a bit hacky but works.

The trick is especially useful if your golden master has mixture of SP and HT, as you can write the source in a way that helps readers understand what is going on, and turn that with sed into what the machines would want to see. When used that way, I'd say it is not "bit hacky but works", but actually is a beautiful solution ;-).

#11 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

In reply to [email protected]

json-writer.c line 81

Quoted 4 lines · On Mon, Mar 26, 2018 at 4:31 PM, <[email protected]> wrote:
json-writer.c new file +4 -0
+static inline void assert_in_array(const struct json_writer *jw)
+{
+       if (!jw->open_stack.len)
+               die("json-writer: array: missing jw_array_begin()");

When you reroll, please consider marking all these string for translation with _(), unless these are for machine consumption.

Quoted 2 lines
json-writer.c new file +2 -0
+       if (jw->open_stack.buf[jw->open_stack.len - 1] != '[')
+               die("json-writer: array: not in array");
Signature · 1 line
Duy
#12 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 8 lines · Duy Nguyen

json-writer.c line 81

Quoted 4 lines · On Mon, Mar 26, 2018 at 4:31 PM, <[email protected]> wrote:
json-writer.c new file +4 -0
+static inline void assert_in_array(const struct json_writer *jw)
+{
+       if (!jw->open_stack.len)
+               die("json-writer: array: missing jw_array_begin()");

When you reroll, please consider marking all these string for translation with _(), unless these are for machine consumption.

Actually, these are all basically asserts, I think. So ideally they'd be BUG()s and not translated.

-Peff

#13 lore raw

Re: [PATCH v4] json_writer: new routines to create data in JSON format

Quoted 11 lines · Jeff King
Quoted 8 lines · Duy Nguyen

json-writer.c line 81

Quoted 4 lines · On Mon, Mar 26, 2018 at 4:31 PM, <[email protected]> wrote:
json-writer.c new file +4 -0
+static inline void assert_in_array(const struct json_writer *jw)
+{
+       if (!jw->open_stack.len)
+               die("json-writer: array: missing jw_array_begin()");

When you reroll, please consider marking all these string for translation with _(), unless these are for machine consumption.

Quoted 2 lines

Actually, these are all basically asserts, I think. So ideally they'd be BUG()s and not translated.

Yes, these are basically asserts. I'll convert them to BUG's and send up another version this week.

Thanks
Jeff