routines to generate JSON data
PATCH v2 1/1 routines to generate JSON data
From: Jeff Hostetler <[email protected]>
This is version 2 of my JSON data format routines. This version addresses the non-utf8 questions raised on V1.
It includes a new "struct json_writer" which is used to guide the accumulation of JSON data -- knowing whether an object or array is currently being composed. This allows error checking during construction.
It also allows construction of nested structures using an inline model (in addition to the original bottom-up composition).
The test helper has been updated to include both the original unit tests and a new scripting API to allow individual tests to be written directly in our t/t*.sh shell scripts.
TODO
====
I still don't know what to do about the Unicode/UTF-8 questions that were raised WRT strings. Pathnames on Linux can be any sequence of 8bit characters -- this is likely to be UTF-8 on modern systems. Pathnames on Windows are UCS2/UTF-16 in the filesystem and we always convert to/from UTF-8 when moving between git data structures and IO calls.
There are few other fields (like author name) that we may want to log which may or may not be, but that is beyond our control. Even localized error messages may be problematic if they include other fields.
So, I'm not sure we have a route to get UTF-8-clean data out of Git, and if we do it is beyond the scope of this patch series.
So I think for our uses here, defining this as "JSON-like" is probably the best answer. We write the strings as we received them (from the file system, the index, or whatever). These strings are properly escaped WRT double quotes, backslashes, and control characters, so we shouldn't have an issue with decoders getting out of sync -- only with them rejecting non-UTF-8 sequences.
We could blindly \uXXXX encode each of the hi-bit characters, if that would help the parsers, but I don't want to do that right now.
WRT binary data, I had not intended using this for binary data. And without knowing what kinds or quantity of binary data we might use it for, I'd like to ignore this for now.
Jeff Hostetler (1):
json_writer: new routines to create data in JSON format
Makefile | 2 + json-writer.c | 321 +++++++++++++++++++++++++++++++++ json-writer.h | 86 +++++++++ t/helper/test-json-writer.c | 420 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 102 +++++++++++ 5 files changed, 931 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh
Signature · 1 line
2.9.3
PATCH v2 1/1 json_writer: new routines to create data in JSON format
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
--- Makefile | 2 + json-writer.c | 321 +++++++++++++++++++++++++++++++++ json-writer.h | 86 +++++++++ t/helper/test-json-writer.c | 420 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 102 +++++++++++ 5 files changed, 931 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o LIB_OBJS += help.o LIB_OBJS += hex.o LIB_OBJS += ident.o +LIB_OBJS += json-writer.o LIB_OBJS += kwset.o LIB_OBJS += levenshtein.o LIB_OBJS += line-log.o
json-writer.c new file +321 -0
321 lines not on this page · raw message
json-writer.h new file +86 -0
86 lines not on this page · raw message
t/helper/test-json-writer.c new file +420 -0
420 lines not on this page · raw message
t/t0019-json-writer.sh new file +102 -0
102 lines not on this page · raw message
Signature · 1 line
2.9.3
Re: [PATCH v2] json_writer: new routines to create data in JSON format
Quoted 3 lines · [email protected] writes:
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
And the point of having capability to write JSON data in our codebase is...?
json-writer.c line 5
Quoted 11 lines
json-writer.c new file +5
-0@@ -0,0 +1,321 @@ +#include "cache.h" +#include "json-writer.h" + +static char g_ch_open[2 ] = {'{' ,'[' }; +static char g_ch_close[2 ] = {'}' ,']' };
What's "g_" prefix?
json-writer.c line 14
Quoted 9 lines
json-writer.c new file +9
-0+ +/* +* Append JSON-quoted version of the given string to 'out'. +*/ +static void append_quoted_string (struct strbuf *out,const char *in) +{ +strbuf_addch (out,'"' ); +for (/**/ ; *in; in++) { +unsigned char c = (unsigned char )*in;
It is clear enough to lose /**/, i.e.
for (; *in; in++) {but for this one. I wonder if
unsigned char c;
strbuf_addch(out, '"');
while ((c = *in++) != '\0') {...
is easier to follow, though.
json-writer.c line 43
Quoted 6 lines
json-writer.c new file +6
-0+static inline void begin (struct json_writer *jw,int is_array) +{ +ALLOC_GROW (jw->levels, jw->nr +1 , jw->alloc); + + jw->levels[jw->nr].level_is_array = !!is_array; + jw->levels[jw->nr].level_is_empty =1 ;
An element of this array is a struct that represents a level, and everybody who accesses an element of that type knows it is talking about a level by the field that has the array being named as .levels[] (also [*1*]). In such a context, it is a bit too loud to name the fields with level_$blah. IOW,
struct json_writer_level
{
unsigned is_array : 1;
unsigned is_empty : 1;
};json-writer.c line 32
Quoted 12 lines
json-writer.c new file +12
-0+struct json_writer_level +{ +unsigned level_is_array :1 ; +unsigned level_is_empty :1 ; +}; + +struct json_writer +{ +struct json_writer_level *levels; +int nr, alloc; +struct strbuf json; +};
[Footnote]
*1* I personally prefer to call an array of things "thing[]", not
"things[]", because then you can refer to an individual element e.g. "thing[4]" and read it as "the fourth thing".
Unless the code often treats an array as a whole, that is, in which case, things[] is OK as you'll be calling the whole thing with the plural name (e.g. call that function and give all the things by passing things[]).
In this case, one level instance is an element of a stack, and the code would be accessing one level at a time most of the time, so "writer.level[4].is_empty" would read more naturally than "writer.levels[4].level_is_empty".
Re: [PATCH v2] json_writer: new routines to create data in JSON format
Quoted 9 lines · Junio C Hamano
Quoted 3 lines · [email protected] writes:
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
And the point of having capability to write JSON data in our codebase is...?
Quoted 13 lines
json-writer.c line 5
Quoted 11 lines
json-writer.c new file +5
-0@@ -0,0 +1,321 @@ +#include "cache.h" +#include "json-writer.h" + +static char g_ch_open[2 ] = {'{' ,'[' }; +static char g_ch_close[2 ] = {'}' ,']' };What's "g_" prefix?
Global.
Sorry, very old habits.
Quoted 14 lines
json-writer.c line 14
Quoted 9 lines
json-writer.c new file +9
-0+ +/* +* Append JSON-quoted version of the given string to 'out'. +*/ +static void append_quoted_string (struct strbuf *out,const char *in) +{ +strbuf_addch (out,'"' ); +for (/**/ ; *in; in++) { +unsigned char c = (unsigned char )*in;It is clear enough to lose /**/, i.e.
for (; *in; in++) {
Quoted 8 lines
but for this one. I wonder if
unsigned char c; strbuf_addch(out, '"'); while ((c = *in++) != '\0') {...
is easier to follow, though.
either way is fine. will fix.
Quoted 13 lines
json-writer.c line 43
Quoted 6 lines
json-writer.c new file +6
-0+static inline void begin (struct json_writer *jw,int is_array) +{ +ALLOC_GROW (jw->levels, jw->nr +1 , jw->alloc); + + jw->levels[jw->nr].level_is_array = !!is_array; + jw->levels[jw->nr].level_is_empty =1 ;An element of this array is a struct that represents a level, and everybody who accesses an element of that type knows it is talking about a level by the field that has the array being named as .levels[] (also [*1*]). In such a context, it is a bit too loud to name the fields with level_$blah. IOW,
Quoted 5 lines
struct json_writer_level
{
unsigned is_array : 1;
unsigned is_empty : 1;
};make sense. will fix.
Quoted 24 lines
json-writer.c line 32
Quoted 12 lines
json-writer.c new file +12
-0+struct json_writer_level +{ +unsigned level_is_array :1 ; +unsigned level_is_empty :1 ; +}; + +struct json_writer +{ +struct json_writer_level *levels; +int nr, alloc; +struct strbuf json; +};[Footnote]
*1* I personally prefer to call an array of things "thing[]", not
"things[]", because then you can refer to an individual element e.g. "thing[4]" and read it as "the fourth thing".Unless the code often treats an array as a whole, that is, in which case, things[] is OK as you'll be calling the whole thing with the plural name (e.g. call that function and give all the things by passing things[]).
Quoted 4 lines
In this case, one level instance is an element of a stack, and the code would be accessing one level at a time most of the time, so "writer.level[4].is_empty" would read more naturally than "writer.levels[4].level_is_empty".
yeah, that makes sense.
Thanks
Jeff
Re: [PATCH v2] json_writer: new routines to create data in JSON format
In reply to [email protected]Am 21.03.2018 um 20:28 schrieb [email protected]:
Quoted 61 lines
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 2 + json-writer.c | 321 +++++++++++++++++++++++++++++++++ json-writer.h | 86 +++++++++ t/helper/test-json-writer.c | 420 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 102 +++++++++++ 5 files changed, 931 insertions(+)create mode 100644
json-writer.c
create mode 100644json-writer.h
create mode 100644t/helper/test-json-writer.c
create mode 100755t/t0019-json-writer.shMakefile +2
-0@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o LIB_OBJS += help.o LIB_OBJS += hex.o LIB_OBJS += ident.o +LIB_OBJS += json-writer.o LIB_OBJS += kwset.o LIB_OBJS += levenshtein.o LIB_OBJS += line-log.ojson-writer.c new file +18
-0@@ -0,0 +1,321 @@ +#include "cache.h" +#include "json-writer.h" + +static char g_ch_open[2 ] = {'{' ,'[' }; +static char g_ch_close[2 ] = {'}' ,']' }; + +/* +* Append JSON-quoted version of the given string to 'out'. +*/ +static void append_quoted_string (struct strbuf *out,const char *in) +{ +strbuf_addch (out,'"' ); +for (/**/ ; *in; in++) { +unsigned char c = (unsigned char )*in; +if (c =='"' ) +strbuf_add (out,"\\\"" ,2 ); +else if (c =='\\' ) +strbuf_add (out,"\\\\" ,2 );
json-writer.c line 28
Quoted 10 lines
json-writer.c new file +10
-0+else if (c =='\n' ) +strbuf_add (out,"\\n" ,2 ); +else if (c =='\r' ) +strbuf_add (out,"\\r" ,2 ); +else if (c =='\t' ) +strbuf_add (out,"\\t" ,2 ); +else if (c =='\f' ) +strbuf_add (out,"\\f" ,2 ); +else if (c =='\b' ) +strbuf_add (out,"\\b" ,2 );
Using strbuf_addstr() here would result in the same object code (its strlen() call is inlined for constants) and avoid having to specify the redundant length 2.
Quoted 99 lines
json-writer.c new file +99
-0+else if (c <0x20 ) +strbuf_addf (out,"\\u%04x" , c); +else +strbuf_addch (out, c); + } +strbuf_addch (out,'"' ); +} + + +static inline void begin (struct json_writer *jw,int is_array) +{ +ALLOC_GROW (jw->levels, jw->nr +1 , jw->alloc); + + jw->levels[jw->nr].level_is_array = !!is_array; + jw->levels[jw->nr].level_is_empty =1 ; + +strbuf_addch (&jw->json, g_ch_open[!!is_array]); + + jw->nr++; +} + +/* +* Assert that we have an open object at this level. +*/ +static void inline assert_in_object (const struct json_writer *jw,const char *key) +{ +if (!jw->nr) +BUG ("object: missing jw_object_begin(): '%s'" , key); +if (jw->levels[jw->nr -1 ].level_is_array) +BUG ("object: not in object: '%s'" , key); +} + +/* +* Assert that we have an open array at this level. +*/ +static void inline assert_in_array (const struct json_writer *jw) +{ +if (!jw->nr) +BUG ("array: missing jw_begin()" ); +if (!jw->levels[jw->nr -1 ].level_is_array) +BUG ("array: not in array" ); +} + +/* +* Add comma if we have already seen a member at this level. +*/ +static void inline maybe_add_comma (struct json_writer *jw) +{ +if (jw->levels[jw->nr -1 ].level_is_empty) + jw->levels[jw->nr -1 ].level_is_empty =0 ; +else +strbuf_addch (&jw->json,',' ); +} + +/* +* Assert that the given JSON object or JSON array has been properly +* terminated. (Has closing bracket.) +*/ +static void inline assert_is_terminated (const struct json_writer *jw) +{ +if (jw->nr) +BUG ("object: missing jw_end(): '%s'" , jw->json.buf); +} + +void jw_object_begin (struct json_writer *jw) +{ +begin (jw,0 ); +} + +void jw_object_string (struct json_writer *jw,const char *key,const char *value) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); +append_quoted_string (&jw->json, value); +} + +void jw_object_int (struct json_writer *jw,const char *key,int value) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addf (&jw->json,":%d" , value); +} + +void jw_object_uint64 (struct json_writer *jw,const char *key,uint64_t value) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addf (&jw->json,":%" PRIuMAX, value); +} + +void jw_object_double (struct json_writer *jw,const char *fmt, +const char *key,double value)
json-writer.c line 137
Quoted 10 lines
json-writer.c new file +10
-0+{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +if (!fmt || !*fmt) + fmt ="%f" ; + +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); +strbuf_addf (&jw->json, fmt, value);
Hmm. Can compilers check such a caller-supplied format matches the value's type? (I don't know how to specify a format attribute for GCC and Clang for this function.)
Quoted 30 lines
json-writer.c new file +30
-0+} + +void jw_object_true (struct json_writer *jw,const char *key) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addstr (&jw->json,":true" ); +} + +void jw_object_false (struct json_writer *jw,const char *key) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addstr (&jw->json,":false" ); +} + +void jw_object_null (struct json_writer *jw,const char *key) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addstr (&jw->json,":null" ); +} + +void jw_object_sub(struct json_writer *jw, const char *key,
json-writer.c line 177
Quoted 10 lines
json-writer.c new file +10
-0+ const struct json_writer *value) +{ +assert_is_terminated (value); + +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); +strbuf_addstr (&jw->json, value->json.buf);
strbuf_addbuf() would be a better fit here -- it avoids a strlen() call and NUL handling issues.
Quoted 13 lines
json-writer.c new file +13
-0+} + +void jw_object_inline_begin_object (struct json_writer *jw,const char *key) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); + +jw_object_begin (jw); +} +
json-writer.c line 200
Quoted 10 lines
json-writer.c new file +10
-0+void jw_object_inline_begin_array (struct json_writer *jw,const char *key) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); + +jw_array_begin (jw); +}
Those duplicate calls in the last ten functions feel mind-numbing. A helper function for adding comma, key and colon might be a good idea.
Quoted 30 lines
json-writer.c new file +30
-0+ +void jw_array_begin (struct json_writer *jw) +{ +begin (jw,1 ); +} + +void jw_array_string (struct json_writer *jw,const char *value) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +append_quoted_string (&jw->json, value); +} + +void jw_array_int (struct json_writer *jw,int value) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +strbuf_addf (&jw->json,"%d" , value); +} + +void jw_array_uint64 (struct json_writer *jw,uint64_t value) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +strbuf_addf (&jw->json,"%" PRIuMAX, value); +} +
json-writer.c line 240
Quoted 10 lines
json-writer.c new file +10
-0+void jw_array_double (struct json_writer *jw,const char *fmt,double value) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +if (!fmt || !*fmt) + fmt ="%f" ; + +strbuf_addf (&jw->json, fmt, value); +}
Same question/quibbling about format checking again.
Quoted 23 lines
json-writer.c new file +23
-0+ +void jw_array_true (struct json_writer *jw) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +strbuf_addstr (&jw->json,"true" ); +} + +void jw_array_false (struct json_writer *jw) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +strbuf_addstr (&jw->json,"false" ); +} + +void jw_array_null (struct json_writer *jw) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +strbuf_addstr (&jw->json,"null" );
json-writer.c line 273
Quoted 10 lines
json-writer.c new file +10
-0+} + +void jw_array_sub (struct json_writer *jw,const struct json_writer *value) +{ +assert_is_terminated (value); + +assert_in_array (jw); +maybe_add_comma (jw); + +strbuf_addstr (&jw->json, value->json.buf);
strbuf_addbuf() again.
Quoted 89 lines
json-writer.c new file +48
-0+} + + +void jw_array_argc_argv (struct json_writer *jw,int argc,const char **argv) +{ +int k; + +for (k =0 ; k < argc; k++) +jw_array_string (jw, argv[k]); +} + +void jw_array_argv (struct json_writer *jw,const char **argv) +{ +while (*argv) +jw_array_string (jw, *argv++); +} + +void jw_array_inline_begin_object (struct json_writer *jw) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +jw_object_begin (jw); +} + +void jw_array_inline_begin_array (struct json_writer *jw) +{ +assert_in_array (jw); +maybe_add_comma (jw); + +jw_array_begin (jw); +} + +int jw_is_terminated (const struct json_writer *jw) +{ +return !jw->nr; +} + +void jw_end (struct json_writer *jw) +{ +if (!jw->nr) +BUG ("too many jw_end(): '%s'" , jw->json.buf); + + jw->nr--; + +strbuf_addch (&jw->json, + g_ch_close[jw->levels[jw->nr].level_is_array]); +}json-writer.h new file +35
-0@@ -0,0 +1,86 @@ +#ifndef JSON_WRITER_H +#define JSON_WRITER_H + +/* +* JSON data structures are defined at: +* http://json.org/ +* http://www.ietf.org/rfc/rfc7159.txt +* +* The JSON-writer API allows one to build JSON data structures using a +* simple wrapper around a "struct strbuf" buffer. It is intended as a +* simple API to build output strings; it is not intended to be a general +* object model for JSON data. In particular, it does not re-order keys +* in an object (dictionary), it does not de-dup keys in an object, and +* it does not allow lookup or parsing of JSON data. +* +* All string values (both keys and string r-values) are properly quoted +* and escaped if they contain special characters. +* +* These routines create compact JSON data (with no unnecessary whitespace, +* newlines, or indenting). If you get an unexpected response, verify +* that you're not expecting a pretty JSON string. +* +* Both "JSON objects" (aka sets of k/v pairs) and "JSON array" can be +* constructed using a 'begin append* end' model. +* +* Nested objects and arrays can either be constructed bottom up (by +* creating sub object/arrays first and appending them to the super +* object/array) -or- by building them inline in one pass. This is a +* personal style and/or data shape choice. +* +* See t/helper/test-json-writer.c for various usage examples. +*/ + +struct json_writer_level +{
json-writer.h line 45
Quoted 10 lines
json-writer.h new file +10
-0+ unsigned level_is_array : 1; + unsigned level_is_empty : 1; +}; + +struct json_writer +{ +struct json_writer_level *levels; +int nr, alloc; +struct strbuf json; +};
A simpler and probably more compact representation of is_array would be a strbuf with one char per level, e.g. '[' for an array and '{' for an object (or ']' and '}').
I don't understand the need to track emptiness per level. Only the top level array/object can ever be empty, can it?
Quoted 41 lines
json-writer.h new file +41
-0+ +#define JSON_WRITER_INIT { NULL, 0, 0, STRBUF_INIT } + +void jw_object_begin (struct json_writer *jw); +void jw_array_begin (struct json_writer *jw); + +void jw_object_string (struct json_writer *jw,const char *key, +const char *value); +void jw_object_int (struct json_writer *jw,const char *key,int value); +void jw_object_uint64 (struct json_writer *jw,const char *key,uint64_t value); +void jw_object_double (struct json_writer *jw,const char *fmt, +const char *key,double value); +void jw_object_true (struct json_writer *jw,const char *key); +void jw_object_false (struct json_writer *jw,const char *key); +void jw_object_null (struct json_writer *jw,const char *key); +void jw_object_sub (struct json_writer *jw,const char *key, +const struct json_writer *value); + +void jw_object_inline_begin_object (struct json_writer *jw,const char *key); +void jw_object_inline_begin_array (struct json_writer *jw,const char *key); + + + +void jw_array_string (struct json_writer *jw,const char *value); +void jw_array_int (struct json_writer *jw,int value); +void jw_array_uint64 (struct json_writer *jw,uint64_t value); +void jw_array_double (struct json_writer *jw,const char *fmt,double value); +void jw_array_true (struct json_writer *jw); +void jw_array_false (struct json_writer *jw); +void jw_array_null (struct json_writer *jw); +void jw_array_sub (struct json_writer *jw,const struct json_writer *value); +void jw_array_argc_argv (struct json_writer *jw,int argc,const char **argv); +void jw_array_argv (struct json_writer *jw,const char **argv); + +void jw_array_inline_begin_object (struct json_writer *jw); +void jw_array_inline_begin_array (struct json_writer *jw); + +int jw_is_terminated (const struct json_writer *jw); +void jw_end (struct json_writer *jw); + +#endif /* JSON_WRITER_H */
Re: [PATCH v2] json_writer: new routines to create data in JSON format
Quoted 68 lines · René Scharfe
Am 21.03.2018 um 20:28 schrieb [email protected]:
Quoted 67 lines
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
Signed-off-by Jeff Hostetler <[email protected]>--- Makefile | 2 + json-writer.c | 321 +++++++++++++++++++++++++++++++++ json-writer.h | 86 +++++++++ t/helper/test-json-writer.c | 420 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 102 +++++++++++ 5 files changed, 931 insertions(+)create mode 100644
json-writer.c
create mode 100644json-writer.h
create mode 100644t/helper/test-json-writer.c
create mode 100755t/t0019-json-writer.shMakefile +2
-0@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o LIB_OBJS += help.o LIB_OBJS += hex.o LIB_OBJS += ident.o +LIB_OBJS += json-writer.o LIB_OBJS += kwset.o LIB_OBJS += levenshtein.o LIB_OBJS += line-log.ojson-writer.c new file +24
-0@@ -0,0 +1,321 @@ +#include "cache.h" +#include "json-writer.h" + +static char g_ch_open[2 ] = {'{' ,'[' }; +static char g_ch_close[2 ] = {'}' ,']' }; + +/* +* Append JSON-quoted version of the given string to 'out'. +*/ +static void append_quoted_string (struct strbuf *out,const char *in) +{ +strbuf_addch (out,'"' ); +for (/**/ ; *in; in++) { +unsigned char c = (unsigned char )*in; +if (c =='"' ) +strbuf_add (out,"\\\"" ,2 ); +else if (c =='\\' ) +strbuf_add (out,"\\\\" ,2 ); +else if (c =='\n' ) +strbuf_add (out,"\\n" ,2 ); +else if (c =='\r' ) +strbuf_add (out,"\\r" ,2 ); +else if (c =='\t' ) +strbuf_add (out,"\\t" ,2 );
Quoted 8 lines
json-writer.c line 28
Quoted 4 lines
json-writer.c new file +4
-0+else if (c =='\f' ) +strbuf_add (out,"\\f" ,2 ); +else if (c =='\b' ) +strbuf_add (out,"\\b" ,2 );Using
strbuf_addstr()here would result in the same object code (itsstrlen()call is inlined for constants) and avoid having to specify the redundant length 2.
sure. thanks.
Quoted 8 lines
Quoted 8 lines
json-writer.c new file +8
-0+else if (c <0x20 ) +strbuf_addf (out,"\\u%04x" , c); +else +strbuf_addch (out, c); + } +strbuf_addch (out,'"' ); +} +
[...]
Quoted 9 lines
Quoted 9 lines
json-writer.c new file +9
-0+ +void jw_object_double (struct json_writer *jw,const char *fmt, +const char *key,double value) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); + +if (!fmt || !*fmt) + fmt ="%f" ;
Quoted 8 lines
json-writer.c line 137
Quoted 4 lines
json-writer.c new file +4
-0+ +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); +strbuf_addf (&jw->json, fmt, value);Hmm. Can compilers check such a caller-supplied format matches the value's type? (I don't know how to specify a format attribute for GCC and Clang for this function.)
I'll look into this. thanks.
Quoted 1 line
Quoted 1 line
+}
[...]
Quoted 7 lines
Quoted 7 lines
json-writer.c new file +7
-0+ +void jw_object_sub (struct json_writer *jw,const char *key, +const struct json_writer *value) +{ +assert_is_terminated (value); + +assert_in_object (jw, key);
Quoted 8 lines
json-writer.c line 177
Quoted 5 lines
json-writer.c new file +5
-0+maybe_add_comma (jw); + +append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); +strbuf_addstr (&jw->json, value->json.buf);
strbuf_addbuf()would be a better fit here -- it avoids astrlen()call and NUL handling issues.
sure. thanks. i always forget about _addbuf().
Quoted 7 lines
Quoted 7 lines
json-writer.c new file +7
-0+} + +void jw_object_inline_begin_array (struct json_writer *jw,const char *key) +{ +assert_in_object (jw, key); +maybe_add_comma (jw); +
Quoted 8 lines
json-writer.c line 200
Quoted 5 lines
json-writer.c new file +5
-0+append_quoted_string (&jw->json, key); +strbuf_addch (&jw->json,':' ); + +jw_array_begin (jw); +}Those duplicate calls in the last ten functions feel mind-numbing. A helper function for adding comma, key and colon might be a good idea.
I'll see if I can add another macro to do the dirty work here.
[...]
Quoted 56 lines
Quoted 41 lines
json-writer.h new file +35
-0@@ -0,0 +1,86 @@ +#ifndef JSON_WRITER_H +#define JSON_WRITER_H + +/* +* JSON data structures are defined at: +* http://json.org/ +* http://www.ietf.org/rfc/rfc7159.txt +* +* The JSON-writer API allows one to build JSON data structures using a +* simple wrapper around a "struct strbuf" buffer. It is intended as a +* simple API to build output strings; it is not intended to be a general +* object model for JSON data. In particular, it does not re-order keys +* in an object (dictionary), it does not de-dup keys in an object, and +* it does not allow lookup or parsing of JSON data. +* +* All string values (both keys and string r-values) are properly quoted +* and escaped if they contain special characters. +* +* These routines create compact JSON data (with no unnecessary whitespace, +* newlines, or indenting). If you get an unexpected response, verify +* that you're not expecting a pretty JSON string. +* +* Both "JSON objects" (aka sets of k/v pairs) and "JSON array" can be +* constructed using a 'begin append* end' model. +* +* Nested objects and arrays can either be constructed bottom up (by +* creating sub object/arrays first and appending them to the super +* object/array) -or- by building them inline in one pass. This is a +* personal style and/or data shape choice. +* +* See t/helper/test-json-writer.c for various usage examples. +*/ + +struct json_writer_level +{json-writer.h line 45
Quoted 10 lines
json-writer.h new file +10
-0+ unsigned level_is_array : 1; + unsigned level_is_empty : 1; +}; + +struct json_writer +{ +struct json_writer_level *levels; +int nr, alloc; +struct strbuf json; +};A simpler and probably more compact representation of
is_arraywould be a strbuf with one char per level, e.g. '[' for an array and '{' for an object (or ']' and '}').
Quoted 2 lines
I don't understand the need to track emptiness per level. Only the top level array/object can ever be empty, can it?
My expectation was that any sub-object or sub-array could be empty. That is, this should be valid (and the JSON parser in Python allows):
{"a":{}, "b":[], "c":[[]], "d":[{}]}Jeff
Re: [PATCH v2] json_writer: new routines to create data in JSON format
Am 23.03.2018 um 20:55 schrieb Jeff Hostetler:
Quoted 17 lines
Quoted 17 lines
json-writer.h line 45
Quoted 12 lines
json-writer.h new file +12
-0+struct json_writer_level +{ + unsigned level_is_array : 1; + unsigned level_is_empty : 1; +}; + +struct json_writer +{ + struct json_writer_level *levels; + int nr, alloc; + struct strbuf json; +};A simpler and probably more compact representation of
is_arraywould be a strbuf with one char per level, e.g. '[' for an array and '{' for an object (or ']' and '}').
Quoted 7 lines
Quoted 2 lines
I don't understand the need to track emptiness per level. Only the top level array/object can ever be empty, can it?
My expectation was that any sub-object or sub-array could be empty. That is, this should be valid (and the JSON parser in Python allows):
{"a":{}, "b":[], "c":[[]], "d":[{}]}
Sure, but the emptiness of finished arrays and objects doesn't matter for the purposes of error checking, comma setting or closing. At most one of them is empty *and* unclosed while writing the overall JSON object -- the last one opened:
{
{"a":{
{"a":{}, "b":[
{"a":{}, "b":[], "c":[
{"a":{}, "b":[], "c":[[
{"a":{}, "b":[], "c":[[]], "d":[
{"a":{}, "b":[], "c":[[]], "d":[{Any of the earlier written arrays/objects are either closed or contain at least a half-done sub-array/object, which makes them non-empty.
René
Re: [PATCH v2] json_writer: new routines to create data in JSON format
Quoted 39 lines · René Scharfe
Am 23.03.2018 um 20:55 schrieb Jeff Hostetler:
Quoted 17 lines
Quoted 17 lines
json-writer.h line 45
Quoted 12 lines
json-writer.h new file +12
-0+struct json_writer_level +{ + unsigned level_is_array : 1; + unsigned level_is_empty : 1; +}; + +struct json_writer +{ + struct json_writer_level *levels; + int nr, alloc; + struct strbuf json; +};A simpler and probably more compact representation of
is_arraywould be a strbuf with one char per level, e.g. '[' for an array and '{' for an object (or ']' and '}').Quoted 7 lines
Quoted 2 lines
I don't understand the need to track emptiness per level. Only the top level array/object can ever be empty, can it?
My expectation was that any sub-object or sub-array could be empty. That is, this should be valid (and the JSON parser in Python allows):
{"a":{}, "b":[], "c":[[]], "d":[{}]}
Sure, but the emptiness of finished arrays and objects doesn't matter for the purposes of error checking, comma setting or closing. At most one of them is empty *and* unclosed while writing the overall JSON object -- the last one opened:
{ {"a":{ {"a":{}, "b":[ {"a":{}, "b":[], "c":[ {"a":{}, "b":[], "c":[[ {"a":{}, "b":[], "c":[[]], "d":[ {"a":{}, "b":[], "c":[[]], "d":[{
Quoted 5 lines
Any of the earlier written arrays/objects are either closed or contain at least a half-done sub-array/object, which makes them non-empty.
René
good point. i'll revisit. thanks.
Jeff
Quoted 10 lines · On Wed, Mar 21, 2018 at 07:28:26PM +0000, [email protected] wrote:
It includes a new "struct
json_writer" which is used to guide the accumulation of JSON data -- knowing whether an object or array is currently being composed. This allows error checking during construction.It also allows construction of nested structures using an inline model (in addition to the original bottom-up composition).
The test helper has been updated to include both the original unit tests and a new scripting API to allow individual tests to be written directly in our
t/t*.shshell scripts.
Thanks for all of this. The changes look quite sensible to me (I do still suspect we could do the "first_item" thing without having to allocate, but I really like the assertions you were able to put in).
Quoted 6 lines
So I think for our uses here, defining this as "JSON-like" is probably the best answer. We write the strings as we received them (from the file system, the index, or whatever). These strings are properly escaped WRT double quotes, backslashes, and control characters, so we shouldn't have an issue with decoders getting out of sync -- only with them rejecting non-UTF-8 sequences.
Yeah, I think I've come to the same conclusion. My main goal in raising it now was to see if there was some other format we might use before we go too far down the JSON road. But as far as I can tell there really isn't another good option.
Quoted 3 lines
WRT binary data, I had not intended using this for binary data. And without knowing what kinds or quantity of binary data we might use it for, I'd like to ignore this for now.
Yeah, I don't have any plans here either. I was thinking more about things like author names and file paths.
-Peff
Quoted 10 lines · On Wed, Mar 21 2018, [email protected] wrote:
So, I'm not sure we have a route to get UTF-8-clean data out of Git, and if we do it is beyond the scope of this patch series.
So I think for our uses here, defining this as "JSON-like" is probably the best answer. We write the strings as we received them (from the file system, the index, or whatever). These strings are properly escaped WRT double quotes, backslashes, and control characters, so we shouldn't have an issue with decoders getting out of sync -- only with them rejecting non-UTF-8 sequences.
Quoted 6 lines
We could blindly \uXXXX encode each of the hi-bit characters, if that would help the parsers, but I don't want to do that right now.
WRT binary data, I had not intended using this for binary data. And without knowing what kinds or quantity of binary data we might use it for, I'd like to ignore this for now.
I agree we should just ignore this problem for now given the immediate use-case.