routines to generate JSON data
PATCH v3 1/1 routines to generate JSON data
From: Jeff Hostetler <[email protected]>
This is version 3 of my JSON data format routines.
This version addresses the variable name changes in [v2] and adds additional test cases. I also changed the BUG() calls to die() to help with testing.
The json-writer routines can be used generate structured data in a JSON-like format. I say "JSON-like" because we don't enforce the Unicode/UTF-8 requirement [3,4] on string values. This was discussed on the mailing list in the [v1] and [v2] threads, but to summarize here: Git doesn't know if various fields, such as Unix pathnames and author names, are Unicode or just 8-bit character data, so Git would not know how to properly encode such fields and the consumer of such output would not know these strings were encoded (once or twice). So, until we have a pressing need to generate proper Unicode data, we avoid it for now.
The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we might want to use them in other commands such as status.
[v1] https://public-inbox.org/git/[email protected]/
[v2] https://public-inbox.org/git/[email protected]/
[3] http://www.ietf.org/rfc/rfc7159.txt
[4] http://json.org/
Jeff Hostetler (1):
json_writer: new routines to create data in JSON format
Makefile | 2 + json-writer.c | 321 +++++++++++++++++++++++++++++++++ json-writer.h | 86 +++++++++ t/helper/test-json-writer.c | 420 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 213 ++++++++++++++++++++++ 5 files changed, 1042 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh
Signature · 1 line
2.9.3
PATCH v3 1/1 json_writer: new routines to create data in JSON format
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
--- Makefile | 2 + json-writer.c | 321 +++++++++++++++++++++++++++++++++ json-writer.h | 86 +++++++++ t/helper/test-json-writer.c | 420 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 213 ++++++++++++++++++++++ 5 files changed, 1042 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
@@ -662,6 +662,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees
@@ -815,6 +816,7 @@ LIB_OBJS += hashmap.o LIB_OBJS += help.o LIB_OBJS += hex.o LIB_OBJS += ident.o +LIB_OBJS += json-writer.o LIB_OBJS += kwset.o LIB_OBJS += levenshtein.o LIB_OBJS += line-log.o
json-writer.c new file +321 -0
321 lines not on this page · raw message
json-writer.h new file +86 -0
86 lines not on this page · raw message
t/helper/test-json-writer.c new file +420 -0
420 lines not on this page · raw message
t/t0019-json-writer.sh new file +213 -0
213 lines not on this page · raw message
Signature · 1 line
2.9.3
Re: [PATCH v3] json_writer: new routines to create data in JSON format
Quoted 5 lines · [email protected] wrote:
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
Signed-off-by Jeff Hostetler <[email protected]>
If I understand the cover letter correctly, this is a JSON-like format, not actual JSON. Am I understanding correctly?
What are the requirements for consuming this output? Will a typical JSON library be able to handle it without trouble? If not, is there some subset of cases where a typical JSON library is able to handle it without trouble?
Can you say a little about the motivation here? (I know you already put some of that in the cover letter, but since that doesn't become part of permanent history, it doesn't help the audience that matters.)
This would also be easier to review if there were an application of it in the same series. It's fine to send an RFC like this without such an application, but I think we should hold off on merging it until we have one. Having an application makes review much easier since we can see how the API works in practice, how well the approach fits what users need, etc.
Thanks and hope that helps,
Jonathan
Re: [PATCH v3] json_writer: new routines to create data in JSON format
Quoted 8 lines · Jonathan Nieder
Quoted 5 lines · [email protected] wrote:
From: Jeff Hostetler <[email protected]>
Add basic routines to generate data in JSON format.
Signed-off-by Jeff Hostetler <[email protected]>
Quoted 7 lines
If I understand the cover letter correctly, this is a JSON-like format, not actual JSON. Am I understanding correctly?
What are the requirements for consuming this output? Will a typical JSON library be able to handle it without trouble? If not, is there some subset of cases where a typical JSON library is able to handle it without trouble?
I'll add text to the commit message and the .h file explaining the Unicode limitation in the next reroll.
Quoted 4 lines
Can you say a little about the motivation here? (I know you already put some of that in the cover letter, but since that doesn't become part of permanent history, it doesn't help the audience that matters.)
I'll add a note about that to the commit message.
Thanks.
Quoted 7 lines
This would also be easier to review if there were an application of it in the same series. It's fine to send an RFC like this without such an application, but I think we should hold off on merging it until we have one. Having an application makes review much easier since we can see how the API works in practice, how well the approach fits what users need, etc.
That's fine. I'm planning to push up another patch series next week that builds upon this, so hopefully that will help.
Quoted 3 lines
Thanks and hope that helps,
Jonathan
Thanks,
Jeff
Hi,
Quoted 6 lines · [email protected] wrote:
From: Jeff Hostetler <[email protected]>
This is version 3 of my JSON data format routines.
This version addresses the variable name changes in [v2] and adds additional test cases. I also changed the
BUG()calls todie()to help with testing.
Can the information below go in the commit message?
Usually to avoid that kind of problem, I don't send a cover letter in one-patch series. Information that I don't want to make part of permanent history (like the above) can go after the three-dash line to ensure it doesn't go in the commit message.
Thanks,
Jonathan
Quoted 18 lines
The json-writer routines can be used generate structured data in a JSON-like format. I say "JSON-like" because we don't enforce the Unicode/UTF-8 requirement [3,4] on string values. This was discussed on the mailing list in the [v1] and [v2] threads, but to summarize here: Git doesn't know if various fields, such as Unix pathnames and author names, are Unicode or just 8-bit character data, so Git would not know how to properly encode such fields and the consumer of such output would not know these strings were encoded (once or twice). So, until we have a pressing need to generate proper Unicode data, we avoid it for now.
The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we might want to use them in other commands such as status.
[v1] https://public-inbox.org/git/[email protected]/
[v2] https://public-inbox.org/git/[email protected]/
[3] http://www.ietf.org/rfc/rfc7159.txt
[4] http://json.org/
Quoted 3 lines · On 23/03/18 16:29, [email protected] wrote:
From: Jeff Hostetler <[email protected]>
This is version 3 of my JSON data format routines.
I have not looked at v3 yet - the patch below is against the version in 'pu' @3284f940c (presumably that would be v2).
The version in 'pu' broke my build on Linux, but not on cygwin - which was a bit of a surprise. That is, until I saw the warnings and remembered that I have this in my config.mak on Linux, but not on cygwin:
ifneq ($(CC),clang) CFLAGS += -Wold-style-declaration CFLAGS += -Wno-pointer-to-int-cast CFLAGS += -Wsystem-headers endif
... and the warnings were:
$ diff nout pout 1c1 < GIT_VERSION = 2.17.0.rc1.317.g4a561d2cc ---
Quoted 1 line
GIT_VERSION= 2.17.0.rc1.445.g3284f940c
29a30
Quoted 1 line
CC commit-graph.o
73a75,87
Quoted 5 lines
CC json-writer.o
json-writer.c:53:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
static void inlineassert_in_object(const struct json_writer *jw, const char *key)^
json-writer.c:64:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
Quoted 8 lines
static void inline
assert_in_array(const struct json_writer *jw)
^
json-writer.c:75:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
static void inlinemaybe_add_comma(struct json_writer *jw)
^
json-writer.c:87:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
static void inlineassert_is_terminated(const struct json_writer *jw)
^
83a98
Quoted 1 line
CC ls-refs.o
...
$
The '-Wold-style-declaration' gcc warning flag is not a standard project flag, and I can't quite remember why I have it set, so I guess you could just ignore it. However, all other 'static inline' functions in the project have the inline keyword before the return type, so ... ;-)
Also, sparse spewed 40 warnings for t/helper/test-json-writer.c, which were mainly about file-local symbols, but had a couple of 'constant is so large ...', like so:
$ grep warning psp-out | head -8 t/helper/test-json-writer.c:45:46: warning: constant 0xffffffffffffffff is so big it is unsigned long t/helper/test-json-writer.c:108:40: warning: constant 0xffffffffffffffff is so big it is unsigned long t/helper/test-json-writer.c:4:12: warning: symbol 'expect_obj1' was not declared. Should it be static? t/helper/test-json-writer.c:5:12: warning: symbol 'expect_obj2' was not declared. Should it be static? t/helper/test-json-writer.c:6:12: warning: symbol 'expect_obj3' was not declared. Should it be static? t/helper/test-json-writer.c:7:12: warning: symbol 'expect_obj4' was not declared. Should it be static? t/helper/test-json-writer.c:8:12: warning: symbol 'expect_obj5' was not declared. Should it be static? t/helper/test-json-writer.c:10:20: warning: symbol 'obj1' was not declared. Should it be static? $
I decided to use the UINT64_C(v) macro from <stdint.h>, which is a C99 feature, and will (hopefully) not be a problem.
ATB,
Ramsay Jones
-- >8 --
Subject: [PATCH] json-writer: fix up gcc and sparse warnings
--- json-writer.c | 8 ++--- t/helper/test-json-writer.c | 80 ++++++++++++++++++++++----------------------- 2 files changed, 44 insertions(+), 44 deletions(-)
json-writer.c +4 -4
@@ -50,7 +50,7 @@ static inline void begin(struct json_writer *jw, int is_array)/* * Assert that we have an open object at this level. */ -static void inline assert_in_object (const struct json_writer *jw,const char *key) +static inline void assert_in_object (const struct json_writer *jw,const char *key) {if (!jw->nr)BUG ("object: missing jw_object_begin(): '%s'" , key);
@@ -61,7 +61,7 @@ static void inline assert_in_object(const struct json_writer *jw, const char *ke/* * Assert that we have an open array at this level. */ -static void inline assert_in_array (const struct json_writer *jw) +static inline void assert_in_array (const struct json_writer *jw) {if (!jw->nr)BUG ("array: missing jw_begin()" );
@@ -72,7 +72,7 @@ static void inline assert_in_array(const struct json_writer *jw)/* * Add comma if we have already seen a member at this level. */ -static void inline maybe_add_comma (struct json_writer *jw) +static inline void maybe_add_comma (struct json_writer *jw) {if (jw->levels[jw->nr -1 ].level_is_empty) jw->levels[jw->nr -1 ].level_is_empty =0 ;
@@ -84,7 +84,7 @@ static void inline maybe_add_comma(struct json_writer *jw)* Assert that the given JSON object or JSON array has been properly * terminated. (Has closing bracket.) */ -static void inline assert_is_terminated (const struct json_writer *jw) +static inline void assert_is_terminated (const struct json_writer *jw) {if (jw->nr)BUG ("object: missing jw_end(): '%s'" , jw->json.buf);
t/helper/test-json-writer.c +40 -40
183 lines not on this page · raw message
Signature · 1 line
2.16.0
Quoted 10 lines · Ramsay Jones
Quoted 3 lines · On 23/03/18 16:29, [email protected] wrote:
From: Jeff Hostetler <[email protected]>
This is version 3 of my JSON data format routines.
I have not looked at v3 yet - the patch below is against the version in 'pu' @3284f940c (presumably that would be v2).
I built my v3 on Linux and didn't see any warnings. I'll add your extra CFLAGS and fix anything that I find and send up a v4 shortly.
Thanks.
Jeff
Quoted 316 lines
The version in 'pu' broke my build on Linux, but not on cygwin - which was a bit of a surprise. That is, until I saw the warnings and remembered that I have this in my config.mak on Linux, but not on cygwin:
ifneq ($(CC),clang) CFLAGS += -Wold-style-declaration CFLAGS += -Wno-pointer-to-int-cast CFLAGS += -Wsystem-headers endif... and the warnings were:
$ diff nout pout 1c1 < GIT_VERSION = 2.17.0.rc1.317.g4a561d2cc ---Quoted 1 line
GIT_VERSION= 2.17.0.rc1.445.g3284f940c29a30
Quoted 1 line
CC commit-graph.o
73a75,87
Quoted 5 lines
CC json-writer.o
json-writer.c:53:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
static void inlineassert_in_object(const struct json_writer *jw, const char *key)^
json-writer.c:64:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]Quoted 8 lines
static void inline
assert_in_array(const struct json_writer *jw)
^
json-writer.c:75:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
static void inlinemaybe_add_comma(struct json_writer *jw)
^
json-writer.c:87:1: warning: ‘inline’ is not at beginning of declaration [-Wold-style-declaration]
static void inlineassert_is_terminated(const struct json_writer *jw)
^83a98
Quoted 1 line
CC ls-refs.o
... $The '-Wold-style-declaration' gcc warning flag is not a standard project flag, and I can't quite remember why I have it set, so I guess you could just ignore it. However, all other 'static inline' functions in the project have the inline keyword before the return type, so ... ;-)
Also, sparse spewed 40 warnings for
t/helper/test-json-writer.c, which were mainly about file-local symbols, but had a couple of 'constant is so large ...', like so:$ grep warning psp-out | head -8 t/helper/test-json-writer.c:45:46: warning: constant 0xffffffffffffffff is so big it is unsigned long t/helper/test-json-writer.c:108:40: warning: constant 0xffffffffffffffff is so big it is unsigned long t/helper/test-json-writer.c:4:12: warning: symbol 'expect_obj1' was not declared. Should it be static? t/helper/test-json-writer.c:5:12: warning: symbol 'expect_obj2' was not declared. Should it be static? t/helper/test-json-writer.c:6:12: warning: symbol 'expect_obj3' was not declared. Should it be static? t/helper/test-json-writer.c:7:12: warning: symbol 'expect_obj4' was not declared. Should it be static? t/helper/test-json-writer.c:8:12: warning: symbol 'expect_obj5' was not declared. Should it be static? t/helper/test-json-writer.c:10:20: warning: symbol 'obj1' was not declared. Should it be static? $I decided to use the
UINT64_C(v)macro from <stdint.h>, which is a C99 feature, and will (hopefully) not be a problem.ATB,
Ramsay Jones-- >8 --
Subject: [PATCH] json-writer: fix up gcc and sparse warningsSigned-off-by Ramsay Jones <[email protected]>--- json-writer.c | 8 ++--- t/helper/test-json-writer.c | 80 ++++++++++++++++++++++----------------------- 2 files changed, 44 insertions(+), 44 deletions(-)json-writer.c +4
-4@@ -50,7 +50,7 @@ static inline void begin(struct json_writer *jw, int is_array)/* * Assert that we have an open object at this level. */ -static void inline assert_in_object (const struct json_writer *jw,const char *key) +static inline void assert_in_object (const struct json_writer *jw,const char *key) {if (!jw->nr)BUG ("object: missing jw_object_begin(): '%s'" , key);@@ -61,7 +61,7 @@ static void inline assert_in_object(const struct json_writer *jw, const char *ke/* * Assert that we have an open array at this level. */ -static void inline assert_in_array (const struct json_writer *jw) +static inline void assert_in_array (const struct json_writer *jw) {if (!jw->nr)BUG ("array: missing jw_begin()" );@@ -72,7 +72,7 @@ static void inline assert_in_array(const struct json_writer *jw)/* * Add comma if we have already seen a member at this level. */ -static void inline maybe_add_comma (struct json_writer *jw) +static inline void maybe_add_comma (struct json_writer *jw) {if (jw->levels[jw->nr -1 ].level_is_empty) jw->levels[jw->nr -1 ].level_is_empty =0 ;@@ -84,7 +84,7 @@ static void inline maybe_add_comma(struct json_writer *jw)* Assert that the given JSON object or JSON array has been properly * terminated. (Has closing bracket.) */ -static void inline assert_is_terminated (const struct json_writer *jw) +static inline void assert_is_terminated (const struct json_writer *jw) {if (jw->nr)BUG ("object: missing jw_end(): '%s'" , jw->json.buf);t/helper/test-json-writer.c +40
-40@@ -1,20 +1,20 @@#include "cache.h" #include "json-writer.h" -const char *expect_obj1 ="{\"a\":\"abc\",\"b\":42,\"c\":true}" ; -const char *expect_obj2 ="{\"a\":-1,\"b\":2147483647,\"c\":0}" ; -const char *expect_obj3 ="{\"a\":0,\"b\":4294967295,\"c\":18446744073709551615}" ; -const char *expect_obj4 ="{\"t\":true,\"f\":false,\"n\":null}" ; -const char *expect_obj5 ="{\"abc\\tdef\":\"abc\\\\def\"}" ; +static const char *expect_obj1 ="{\"a\":\"abc\",\"b\":42,\"c\":true}" ; +static const char *expect_obj2 ="{\"a\":-1,\"b\":2147483647,\"c\":0}" ; +static const char *expect_obj3 ="{\"a\":0,\"b\":4294967295,\"c\":18446744073709551615}" ; +static const char *expect_obj4 ="{\"t\":true,\"f\":false,\"n\":null}" ; +static const char *expect_obj5 ="{\"abc\\tdef\":\"abc\\\\def\"}" ; -struct json_writer obj1 =JSON_WRITER_INIT ; -struct json_writer obj2 =JSON_WRITER_INIT ; -struct json_writer obj3 =JSON_WRITER_INIT ; -struct json_writer obj4 =JSON_WRITER_INIT ; -struct json_writer obj5 =JSON_WRITER_INIT ; +static struct json_writer obj1 =JSON_WRITER_INIT ; +static struct json_writer obj2 =JSON_WRITER_INIT ; +static struct json_writer obj3 =JSON_WRITER_INIT ; +static struct json_writer obj4 =JSON_WRITER_INIT ; +static struct json_writer obj5 =JSON_WRITER_INIT ; -void make_obj1 (void ) +static void make_obj1 (void ) {jw_object_begin (&obj1); {@@ -25,7 +25,7 @@ void make_obj1(void)jw_end (&obj1); } -void make_obj2 (void ) +static void make_obj2 (void ) {jw_object_begin (&obj2); {@@ -36,18 +36,18 @@ void make_obj2(void)jw_end (&obj2); } -void make_obj3 (void ) +static void make_obj3 (void ) {jw_object_begin (&obj3); {jw_object_uint64 (&obj3,"a" ,0 );jw_object_uint64 (&obj3,"b" ,0xffffffff ); -jw_object_uint64 (&obj3,"c" ,0xffffffffffffffff ); +jw_object_uint64 (&obj3,"c" ,UINT64_C (0xffffffffffffffff )); }jw_end (&obj3); } -void make_obj4 (void ) +static void make_obj4 (void ) {jw_object_begin (&obj4); {@@ -58,7 +58,7 @@ void make_obj4(void)jw_end (&obj4); } -void make_obj5 (void ) +static void make_obj5 (void ) {jw_object_begin (&obj5); {@@ -67,17 +67,17 @@ void make_obj5(void)jw_end (&obj5); } -const char *expect_arr1 ="[\"abc\",42,true]" ; -const char *expect_arr2 ="[-1,2147483647,0]" ; -const char *expect_arr3 ="[0,4294967295,18446744073709551615]" ; -const char *expect_arr4 ="[true,false,null]" ; +static const char *expect_arr1 ="[\"abc\",42,true]" ; +static const char *expect_arr2 ="[-1,2147483647,0]" ; +static const char *expect_arr3 ="[0,4294967295,18446744073709551615]" ; +static const char *expect_arr4 ="[true,false,null]" ; -struct json_writer arr1 =JSON_WRITER_INIT ; -struct json_writer arr2 =JSON_WRITER_INIT ; -struct json_writer arr3 =JSON_WRITER_INIT ; -struct json_writer arr4 =JSON_WRITER_INIT ; +static struct json_writer arr1 =JSON_WRITER_INIT ; +static struct json_writer arr2 =JSON_WRITER_INIT ; +static struct json_writer arr3 =JSON_WRITER_INIT ; +static struct json_writer arr4 =JSON_WRITER_INIT ; -void make_arr1 (void ) +static void make_arr1 (void ) {jw_array_begin (&arr1); {@@ -88,7 +88,7 @@ void make_arr1(void)jw_end (&arr1); } -void make_arr2 (void ) +static void make_arr2 (void ) {jw_array_begin (&arr2); {@@ -99,18 +99,18 @@ void make_arr2(void)jw_end (&arr2); } -void make_arr3 (void ) +static void make_arr3 (void ) {jw_array_begin (&arr3); {jw_array_uint64 (&arr3,0 );jw_array_uint64 (&arr3,0xffffffff ); -jw_array_uint64 (&arr3,0xffffffffffffffff ); +jw_array_uint64 (&arr3,UINT64_C (0xffffffffffffffff )); }jw_end (&arr3); } -void make_arr4 (void ) +static void make_arr4 (void ) {jw_array_begin (&arr4); {@@ -121,12 +121,12 @@ void make_arr4(void)jw_end (&arr4); } -char *expect_nest1 = +static char *expect_nest1 ="{\"obj1\":{\"a\":\"abc\",\"b\":42,\"c\":true},\"arr1\":[\"abc\",42,true]}" ; -struct json_writer nest1 =JSON_WRITER_INIT ; +static struct json_writer nest1 =JSON_WRITER_INIT ; -void make_nest1 (void ) +static void make_nest1 (void ) {jw_object_begin (&nest1); {@@ -136,12 +136,12 @@ void make_nest1(void)jw_end (&nest1); } -char *expect_inline1 = +static char *expect_inline1 ="{\"obj1\":{\"a\":\"abc\",\"b\":42,\"c\":true},\"arr1\":[\"abc\",42,true]}" ; -struct json_writer inline1 =JSON_WRITER_INIT ; +static struct json_writer inline1 =JSON_WRITER_INIT ; -void make_inline1 (void ) +static void make_inline1 (void ) {jw_object_begin (&inline1); {@@ -163,11 +163,11 @@ void make_inline1(void)jw_end (&inline1); } -char *expect_inline2 = +static char *expect_inline2 ="[[1,2],[3,4],{\"a\":\"abc\"}]" ; -struct json_writer inline2 =JSON_WRITER_INIT ; +static struct json_writer inline2 =JSON_WRITER_INIT ; -void make_inline2 (void ) +static void make_inline2 (void ) {jw_array_begin (&inline2); {@@ -193,7 +193,7 @@ void make_inline2(void) } -void cmp (const char *test,const struct json_writer *jw,const char *exp) +static void cmp (const char *test,const struct json_writer *jw,const char *exp) {if (!strcmp (jw->json.buf, exp))return ;@@ -209,7 +209,7 @@ void cmp(const char *test, const struct json_writer *jw, const char *exp)* Run some basic regression tests with some known patterns. * These tests also demonstrate how to use the jw_ API. */ -int unit_tests (void ) +static int unit_tests (void ) {t (obj1);t (obj2);