json-writer V7
v7 cover json-writer V7
From: Jeff Hostetler <[email protected]>
Here is V7 of my json-writer patches. Please replace the existing V5/V6 version of jh/json-writer branch with this one.
This version cleans up the die()-vs-BUG() issue that Duy mentioned recently. It also fixes a formatting bug when composing empty sub-objects/-arrays.
It also includes a new Python-based test to consume the generated JSON.
I plan to use the json-writer routines in a followup telemetry patch series.
Jeff Hostetler (2):
json_writer: new routines to create data in JSON format
json-writer: t0019: add Python unit test
Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 274 +++++++++++++++++++++ t/t0019/parse_json_1.py | 35 +++ 6 files changed, 1415 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh create mode 100644 t/t0019/parse_json_1.py
Signature · 1 line
2.9.3
PATCH v7 1/2 json_writer: new routines to create data in JSON format
From: Jeff Hostetler <[email protected]>
Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.
The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.
The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.
--- Makefile | 2 + json-writer.c | 419 ++++++++++++++++++++++++++++++++ json-writer.h | 113 +++++++++ t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++ t/t0019-json-writer.sh | 236 ++++++++++++++++++ 5 files changed, 1342 insertions(+) create mode 100644 json-writer.c create mode 100644 json-writer.h create mode 100644 t/helper/test-json-writer.c create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh TEST_PROGRAMS_NEED_X += test-genrandom TEST_PROGRAMS_NEED_X += test-hashmap TEST_PROGRAMS_NEED_X += test-index-version +TEST_PROGRAMS_NEED_X += test-json-writer TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash TEST_PROGRAMS_NEED_X += test-line-buffer TEST_PROGRAMS_NEED_X += test-match-trees
@@ -820,6 +821,7 @@ LIB_OBJS += hashmap.o LIB_OBJS += help.o LIB_OBJS += hex.o LIB_OBJS += ident.o +LIB_OBJS += json-writer.o LIB_OBJS += kwset.o LIB_OBJS += levenshtein.o LIB_OBJS += line-log.o
json-writer.c new file +419 -0
419 lines not on this page · raw message
json-writer.h new file +113 -0
113 lines not on this page · raw message
t/helper/test-json-writer.c new file +572 -0
572 lines not on this page · raw message
t/t0019-json-writer.sh new file +236 -0
236 lines not on this page · raw message
Signature · 1 line
2.9.3
PATCH v7 2/2 json-writer: t0019: add Python unit test
In reply to [email protected]From: Jeff Hostetler <[email protected]>
Test json-writer output using Python.
--- t/t0019-json-writer.sh | 38 ++++++++++++++++++++++++++++++++++++++ t/t0019/parse_json_1.py | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 t/t0019/parse_json_1.py
t/t0019-json-writer.sh +38 -0
@@ -233,4 +233,42 @@ test_expect_success 'inline array with no members' ' test_cmp expect actual ' +# As a sanity check, ask Python to parse our generated JSON. Let Python +# recursively dump the resulting dictionary in sorted order. Confirm that +# that matches our expectations. +test_expect_success PYTHON 'parse JSON using Python' ' + cat >expect <<-\EOF && + a abc + b 42 + sub1 dict + sub1.c 3.14 + sub1.d True + sub1.sub2 list + sub1.sub2[0] False + sub1.sub2[1] dict + sub1.sub2[1].g 0 + sub1.sub2[1].h 1 + sub1.sub2[2] None + EOF + test-json-writer >output.json \ + @object \ + @object-string a abc \ + @object-int b 42 \ + @object-object "sub1" \ + @object-double c 2 3.140 \ + @object-true d \ + @object-array "sub2" \ + @array-false \ + @array-object \ + @object-int g 0 \ + @object-int h 1 \ + @end \ + @array-null \ + @end \ + @end \ + @end && + python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual && + test_cmp expect actual +' + test_done
t/t0019/parse_json_1.py new file +35 -0
@@ -0,0 +1,35 @@ +import os +import sys +import json + +def dump_item(label_input, v): + if type(v) is dict: + print("%s dict" % (label_input)) + dump_dict(label_input, v) + elif type(v) is list: + print("%s list" % (label_input)) + dump_list(label_input, v) + else: + print("%s %s" % (label_input, v)) + +def dump_list(label_input, list_input): + ix = 0 + for v in list_input: + label = ("%s[%d]" % (label_input, ix)) + dump_item(label, v) + ix += 1 + return + +def dump_dict(label_input, dict_input): + for k in sorted(dict_input.iterkeys()): + v = dict_input[k] + if (len(label_input) > 0): + label = ("%s.%s" % (label_input, k)) + else: + label = k + dump_item(label, v) + return + +for line in sys.stdin: + data = json.loads(line) + dump_dict("", data)
Signature · 1 line
2.9.3
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4
-0+# As a sanity check, ask Python to parse our generated JSON. Let Python +# recursively dump the resulting dictionary in sorted order. Confirm that +# that matches our expectations. +test_expect_success PYTHON 'parse JSON using Python' '
[...]
t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270
Quoted 1 line
t/t0019-json-writer.sh +1
-0+ python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&
Would this be better using $PYTHON_PATH rather than hard-coding python as the command?
Signature · 4 lines
Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sometimes I think I understand everything, then I regain consciousness.
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
Quoted 10 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4
-0+# As a sanity check, ask Python to parse our generated JSON. Let Python +# recursively dump the resulting dictionary in sorted order. Confirm that +# that matches our expectations. +test_expect_success PYTHON 'parse JSON using Python' '[...]
t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270
Quoted 1 line
t/t0019-json-writer.sh +1
-0+ python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&Would this be better using $PYTHON_PATH rather than hard-coding python as the command?
Probably. We may want to go the same route as we did for perl in a0e0ec9f7d (t: provide a perl() function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.
That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.
-Peff
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
Quoted 17 lines · Jeff King
Quoted 10 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4
-0+# As a sanity check, ask Python to parse our generated JSON. Let Python +# recursively dump the resulting dictionary in sorted order. Confirm that +# that matches our expectations. +test_expect_success PYTHON 'parse JSON using Python' '[...]
t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270
Quoted 1 line
t/t0019-json-writer.sh +1
-0+ python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&Would this be better using $PYTHON_PATH rather than hard-coding python as the command?
Probably. We may want to go the same route as we did for perl in
a0e0ec9f7d(t: provide aperl()function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.
Quoted 7 lines
That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.
Perl works too. (I don't know perl, but I'm told it does work for stuff like this. :-)
I'll take a stab at it.
Jeff
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
In reply to Jeff KingQuoted 13 lines · Jeff King
Quoted 10 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4
-0+# As a sanity check, ask Python to parse our generated JSON. Let Python +# recursively dump the resulting dictionary in sorted order. Confirm that +# that matches our expectations. +test_expect_success PYTHON 'parse JSON using Python' '[...]
t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270
Quoted 1 line
t/t0019-json-writer.sh +1
-0+ python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&Would this be better using $PYTHON_PATH rather than hard-coding python as the command?
Quoted 6 lines
Probably. We may want to go the same route as we did for perl in
a0e0ec9f7d(t: provide aperl()function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and
Hmm, not quite the _first_ use:
$ git grep PYTHON_PATH -- t t/lib-git-p4.sh: (cd / && "$PYTHON_PATH" -c 'import time; print(int(time.time()))') t/lib-git-p4.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/marshal-dump.py" t/t9020-remote-svn.sh:export PATH PYTHON_PATH GIT_BUILD_DIR t/t9020-remote-svn.sh:exec "$PYTHON_PATH" "$GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py" "$@" t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/k_smush.py" <"$cli/k-text-k" >cli-k-text-k-smush && t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/ko_smush.py" <"$cli/k-text-ko" >cli-k-text-ko-smush && t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/gendouble.py" >%double.png && t/t9810-git-p4-rcs.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_k.py" <"$git/$file" >"$scrub" && t/t9810-git-p4-rcs.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_ko.py" <"$git/$file" >"$scrub" && $
I don't run the p4 or svn tests, so ... :-D
Quoted 4 lines
really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.
I agree that fewer languages is (generally) a good idea.
ATB,
Ramsay Jones
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
Ramsay Jones wrote:
[...]
Quoted 1 line
I don't run the p4 or svn tests, so ... :-D
Heh, lucky you. :)
I try to run them all as part of the fedora builds since they cover much more than I'd ever use. That's the main reason I noticed the bare python. That would trip me up when it came time to build on a near-future fedora where python isn't installed by default and I only wanted to require python3 for the build/runtime scripts.
Quoted 7 lines · Jeff King
Quoted 4 lines
really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.
I agree that fewer languages is (generally) a good idea.
Yep, that's certainly even better and if Jeff H. can use perl relatively easily (or one of the many perl folks here can help with that part of the test), that's great. The best way to solve many problems is avoid having them. :)
Thanks,
Signature · 3 lines
Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Chaos, panic, and disorder - my job is done here.
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
Quoted 4 lines · Todd Zullinger
Ramsay Jones wrote:
[...]Quoted 1 line
I don't run the p4 or svn tests, so ... :-D
Quoted 8 lines
Heh, lucky you. :)
I try to run them all as part of the fedora builds since they cover much more than I'd ever use. That's the main reason I noticed the bare python. That would trip me up when it came time to build on a near-future fedora where python isn't installed by default and I only wanted to require python3 for the build/runtime scripts.
I'm not actually sure those svn bits involving python are usable by real users. The main tool that people us for svn interop is git-svn, which is written in perl.
There was an effort to have a real "git-remote-svn" helper (so that you could just seamlessly "git clone svn://..." instead of using the "git svn" wrapper.
So we have the work in vcs-svn, which gets built as part of a regular compile. But AFAICT it is only used for git-remote-testsvn, t/helper/test-svn-fe, and contrib/svn-fe. None of which have seen any substantive work since 2012[1].
I suppose it's possible somebody could be using "git clone testsvn://" in the wild, but the name would hopefully warn them off. I have no idea how usable that work is in practice.
-Peff
[1] Browsing "git log", most of the commits are just tree-wide cleanups, or fixing some compilation or dependency error. The last "real" commit seems to be around 8e43a1d010 (remote-svn: add incremental import, 2012-09-19).
I wonder if it is worth dropping this experiment as incomplete and unmaintained. People have obviously spent time dealing with the code for various fixups, but I think the whole thing may essentially just be dead code. Or maybe people really are using contrib/svn-fe. Like I said, I have no clue if this stuff is even usable, but we certainly haven't packaged it to be seen by most users.
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
In reply to Ramsay JonesQuoted 9 lines · Ramsay Jones
Quoted 6 lines
Probably. We may want to go the same route as we did for perl in
a0e0ec9f7d(t: provide aperl()function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and
Hmm, not quite the _first_ use:
Quoted 11 lines
$ git grep PYTHON_PATH -- t t/lib-git-p4.sh: (cd / && "$PYTHON_PATH" -c 'import time; print(int(time.time()))') t/lib-git-p4.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/marshal-dump.py" t/t9020-remote-svn.sh:export PATH PYTHON_PATH GIT_BUILD_DIR t/t9020-remote-svn.sh:exec "$PYTHON_PATH" "$GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py" "$@" t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/k_smush.py" <"$cli/k-text-k" >cli-k-text-k-smush && t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/ko_smush.py" <"$cli/k-text-ko" >cli-k-text-ko-smush && t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/gendouble.py" >%double.png && t/t9810-git-p4-rcs.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_k.py" <"$git/$file" >"$scrub" && t/t9810-git-p4-rcs.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_ko.py" <"$git/$file" >"$scrub" && $
OK, the first for a feature that is not already written in python (leading into my second claim that python is used only for fringe commands ;) ).
Though maybe I am wrong that the remote-svn stuff requires python. I thought it did, but poking around, it looks like it's all C, and just the "svnrdump_sim" helper is python.
At any rate, I think the point still stands that perl is our main scripting language. I'd rather keep us to that unless there's a good reason not to.
-Peff
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
Quoted 3 lines · Jeff King
Though maybe I am wrong that the remote-svn stuff requires python. I thought it did, but poking around, it looks like it's all C, and just the "
svnrdump_sim" helper is python.
I think I was getting this mixed up with the git_remote_helpers python work, which was removed long ago in ae34ac126f (git_remote_helpers:
remove little used Python library, 2013-09-07).
Note that it really changes much, but I was curious enough to dig down.
-Peff
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
In reply to Jeff KingQuoted 23 lines · Jeff King
Quoted 9 lines · Ramsay Jones
Quoted 6 lines
Probably. We may want to go the same route as we did for perl in
a0e0ec9f7d(t: provide aperl()function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and
Hmm, not quite the _first_ use:
Quoted 11 lines
$ git grep PYTHON_PATH -- t t/lib-git-p4.sh: (cd / && "$PYTHON_PATH" -c 'import time; print(int(time.time()))') t/lib-git-p4.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/marshal-dump.py" t/t9020-remote-svn.sh:export PATH PYTHON_PATH GIT_BUILD_DIR t/t9020-remote-svn.sh:exec "$PYTHON_PATH" "$GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py" "$@" t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/k_smush.py" <"$cli/k-text-k" >cli-k-text-k-smush && t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/ko_smush.py" <"$cli/k-text-ko" >cli-k-text-ko-smush && t/t9802-git-p4-filetype.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/gendouble.py" >%double.png && t/t9810-git-p4-rcs.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_k.py" <"$git/$file" >"$scrub" && t/t9810-git-p4-rcs.sh: "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_ko.py" <"$git/$file" >"$scrub" && $
Quoted 7 lines
OK, the first for a feature that is not already written in python (leading into my second claim that python is used only for fringe commands ;) ).
Though maybe I am wrong that the remote-svn stuff requires python. I thought it did, but poking around, it looks like it's all C, and just the "
svnrdump_sim" helper is python.
Heh, I was a bit tired last night, so although I knew that I required python to be installed to run the test-suite, I could not remember why. As soon as I went to bed, ... :)
I recently installed Ubuntu 18.04, so that I could get a heads up on any possible problems later this month when I install Linux Mint 19. In order to get the test-suite to run, I had to set 'PYTHON_PATH = /usr/bin/python3' in my config.mak file. (yes, I could have set NO_PYTHON, but you get the idea).
Ubuntu 18.04 no longer installs python2 by default (it has ported to python3), but presumably you can still install it as '/usr/bin/python' (I didn't check).
In any event, it was t9020-remote-svn.sh that was failing which, despite the name, does not depend on 'svn'. As you already noted, it does run svnrdump_sim.py and the git-remote-testsvn.
Quoted 3 lines
At any rate, I think the point still stands that perl is our main scripting language. I'd rather keep us to that unless there's a good reason not to.
Agreed. And I see it has come to pass ... :-D
ATB,
Ramsay Jones
Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test
In reply to Todd ZullingerQuoted 11 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4
-0+# As a sanity check, ask Python to parse our generated JSON. Let Python +# recursively dump the resulting dictionary in sorted order. Confirm that +# that matches our expectations. +test_expect_success PYTHON 'parse JSON using Python' '[...]
t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270
Quoted 1 line
t/t0019-json-writer.sh +1
-0+ python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&Would this be better using $PYTHON_PATH rather than hard-coding python as the command?
good catch!
thanks
Jeff