Skip to contentSkip to conversation

json-writer V7

PATCH Superseded [email protected] 4 participants 13 messages Jun 5, 2018 – Jun 7, 2018 Kbuild

v7 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v7 cover json-writer V7

From: Jeff Hostetler <[email protected]>

Here is V7 of my json-writer patches. Please replace the existing V5/V6 version of jh/json-writer branch with this one.

This version cleans up the die()-vs-BUG() issue that Duy mentioned recently. It also fixes a formatting bug when composing empty sub-objects/-arrays.

It also includes a new Python-based test to consume the generated JSON.

I plan to use the json-writer routines in a followup telemetry patch series.

Jeff Hostetler (2):
json_writer: new routines to create data in JSON format
json-writer: t0019: add Python unit test

 Makefile                    |   2 +
 json-writer.c               | 419 ++++++++++++++++++++++++++++++++
 json-writer.h               | 113 +++++++++
 t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 274 +++++++++++++++++++++
 t/t0019/parse_json_1.py     |  35 +++
 6 files changed, 1415 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh
 create mode 100644 t/t0019/parse_json_1.py
Signature · 1 line
2.9.3
#1 lore raw

PATCH v7 1/2 json_writer: new routines to create data in JSON format

From: Jeff Hostetler <[email protected]>

Add a series of jw_ routines and "struct json_writer" structure to compose JSON data. The resulting string data can then be output by commands wanting to support a JSON output format.

The json-writer routines can be used to generate structured data in a JSON-like format. We say "JSON-like" because we do not enforce the Unicode (usually UTF-8) requirement on string fields. Internally, Git does not necessarily have Unicode/UTF-8 data for most fields, so it is currently unclear the best way to enforce that requirement. For example, on Linx pathnames can contain arbitrary 8-bit character data, so a command like "status" would not know how to encode the reported pathnames. We may want to revisit this (or double encode such strings) in the future.

The initial use for the json-writer routines is for generating telemetry data for executed Git commands. Later, we may want to use them in other commands, such as status.

---
 Makefile                    |   2 +
 json-writer.c               | 419 ++++++++++++++++++++++++++++++++
 json-writer.h               | 113 +++++++++
 t/helper/test-json-writer.c | 572 ++++++++++++++++++++++++++++++++++++++++++++
 t/t0019-json-writer.sh      | 236 ++++++++++++++++++
 5 files changed, 1342 insertions(+)
 create mode 100644 json-writer.c
 create mode 100644 json-writer.h
 create mode 100644 t/helper/test-json-writer.c
 create mode 100755 t/t0019-json-writer.sh
Makefile +2 -0
index a1d8775..4ae6946 100644
@@ -666,6 +666,7 @@ TEST_PROGRAMS_NEED_X += test-fake-ssh
 TEST_PROGRAMS_NEED_X += test-genrandom
 TEST_PROGRAMS_NEED_X += test-hashmap
 TEST_PROGRAMS_NEED_X += test-index-version
+TEST_PROGRAMS_NEED_X += test-json-writer
 TEST_PROGRAMS_NEED_X += test-lazy-init-name-hash
 TEST_PROGRAMS_NEED_X += test-line-buffer
 TEST_PROGRAMS_NEED_X += test-match-trees
@@ -820,6 +821,7 @@ LIB_OBJS += hashmap.o
 LIB_OBJS += help.o
 LIB_OBJS += hex.o
 LIB_OBJS += ident.o
+LIB_OBJS += json-writer.o
 LIB_OBJS += kwset.o
 LIB_OBJS += levenshtein.o
 LIB_OBJS += line-log.o
json-writer.c new file +419 -0

419 lines not on this page · raw message

json-writer.h new file +113 -0

113 lines not on this page · raw message

t/helper/test-json-writer.c new file +572 -0

572 lines not on this page · raw message

t/t0019-json-writer.sh new file +236 -0

236 lines not on this page · raw message

Signature · 1 line
2.9.3
#2 lore raw

PATCH v7 2/2 json-writer: t0019: add Python unit test

In reply to [email protected]

From: Jeff Hostetler <[email protected]>

Test json-writer output using Python.

---
 t/t0019-json-writer.sh  | 38 ++++++++++++++++++++++++++++++++++++++
 t/t0019/parse_json_1.py | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 73 insertions(+)
 create mode 100644 t/t0019/parse_json_1.py
t/t0019-json-writer.sh +38 -0
index c9c2e23..951cd89 100755
@@ -233,4 +233,42 @@ test_expect_success 'inline array with no members' '
 	test_cmp expect actual
 '
 
+# As a sanity check, ask Python to parse our generated JSON.  Let Python
+# recursively dump the resulting dictionary in sorted order.  Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '
+	cat >expect <<-\EOF &&
+	a abc
+	b 42
+	sub1 dict
+	sub1.c 3.14
+	sub1.d True
+	sub1.sub2 list
+	sub1.sub2[0] False
+	sub1.sub2[1] dict
+	sub1.sub2[1].g 0
+	sub1.sub2[1].h 1
+	sub1.sub2[2] None
+	EOF
+	test-json-writer >output.json \
+		@object \
+			@object-string a abc \
+			@object-int b 42 \
+			@object-object "sub1" \
+				@object-double c 2 3.140 \
+				@object-true d \
+				@object-array "sub2" \
+					@array-false \
+					@array-object \
+						@object-int g 0 \
+						@object-int h 1 \
+					@end \
+					@array-null \
+				@end \
+			@end \
+		@end &&
+	python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&
+	test_cmp expect actual
+'
+
 test_done
t/t0019/parse_json_1.py new file +35 -0
new file mode 100644
index 0000000..9d928a3
@@ -0,0 +1,35 @@
+import os
+import sys
+import json
+
+def dump_item(label_input, v):
+    if type(v) is dict:
+        print("%s dict" % (label_input))
+        dump_dict(label_input, v)
+    elif type(v) is list:
+        print("%s list" % (label_input))
+        dump_list(label_input, v)
+    else:
+        print("%s %s" % (label_input, v))
+
+def dump_list(label_input, list_input):
+    ix = 0
+    for v in list_input:
+        label = ("%s[%d]" % (label_input, ix))
+        dump_item(label, v)
+        ix += 1
+    return
+              
+def dump_dict(label_input, dict_input):
+    for k in sorted(dict_input.iterkeys()):
+        v = dict_input[k]
+        if (len(label_input) > 0):
+            label = ("%s.%s" % (label_input, k))
+        else:
+            label = k
+        dump_item(label, v)
+    return
+
+for line in sys.stdin:
+    data = json.loads(line)
+    dump_dict("", data)
Signature · 1 line
2.9.3
#3 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4 -0
+# As a sanity check, ask Python to parse our generated JSON.  Let Python
+# recursively dump the resulting dictionary in sorted order.  Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '

[...]

t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270

Quoted 1 line
t/t0019-json-writer.sh +1 -0
+	python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&

Would this be better using $PYTHON_PATH rather than hard-coding python as the command?

Signature · 4 lines
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes I think I understand everything, then I regain
consciousness.
#4 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

Quoted 10 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4 -0
+# As a sanity check, ask Python to parse our generated JSON.  Let Python
+# recursively dump the resulting dictionary in sorted order.  Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '

[...]

t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270

Quoted 1 line
t/t0019-json-writer.sh +1 -0
+	python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&

Would this be better using $PYTHON_PATH rather than hard-coding python as the command?

Probably. We may want to go the same route as we did for perl in a0e0ec9f7d (t: provide a perl() function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.

That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.

-Peff

#12 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Todd Zullinger
Quoted 11 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4 -0
+# As a sanity check, ask Python to parse our generated JSON.  Let Python
+# recursively dump the resulting dictionary in sorted order.  Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '

[...]

t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270

Quoted 1 line
t/t0019-json-writer.sh +1 -0
+	python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&

Would this be better using $PYTHON_PATH rather than hard-coding python as the command?

good catch!
thanks
Jeff

#5 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Jeff King
Quoted 17 lines · Jeff King
Quoted 10 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4 -0
+# As a sanity check, ask Python to parse our generated JSON.  Let Python
+# recursively dump the resulting dictionary in sorted order.  Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '

[...]

t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270

Quoted 1 line
t/t0019-json-writer.sh +1 -0
+	python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&

Would this be better using $PYTHON_PATH rather than hard-coding python as the command?

Probably. We may want to go the same route as we did for perl in a0e0ec9f7d (t: provide a perl() function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.

Quoted 7 lines

That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.

Perl works too. (I don't know perl, but I'm told it does work for stuff like this. :-)

I'll take a stab at it.

Jeff

#6 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Jeff King
Quoted 13 lines · Jeff King
Quoted 10 lines · Todd Zullinger
Quoted 4 lines · [email protected] wrote:
t/t0019-json-writer.sh +4 -0
+# As a sanity check, ask Python to parse our generated JSON.  Let Python
+# recursively dump the resulting dictionary in sorted order.  Confirm that
+# that matches our expectations.
+test_expect_success PYTHON 'parse JSON using Python' '

[...]

t/t0019-json-writer.sh test_expect_success 'inline array with no members' ' · line 270

Quoted 1 line
t/t0019-json-writer.sh +1 -0
+	python "$TEST_DIRECTORY"/t0019/parse_json_1.py <output.json >actual &&

Would this be better using $PYTHON_PATH rather than hard-coding python as the command?

Quoted 6 lines

Probably. We may want to go the same route as we did for perl in a0e0ec9f7d (t: provide a perl() function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.

That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and

Hmm, not quite the _first_ use:

$ git grep PYTHON_PATH -- t
t/lib-git-p4.sh:        (cd / && "$PYTHON_PATH" -c 'import time; print(int(time.time()))')
t/lib-git-p4.sh:        "$PYTHON_PATH" "$TRASH_DIRECTORY/marshal-dump.py"
t/t9020-remote-svn.sh:export PATH PYTHON_PATH GIT_BUILD_DIR
t/t9020-remote-svn.sh:exec "$PYTHON_PATH" "$GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py" "$@"
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/k_smush.py" <"$cli/k-text-k" >cli-k-text-k-smush &&
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/ko_smush.py" <"$cli/k-text-ko" >cli-k-text-ko-smush &&
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/gendouble.py" >%double.png &&
t/t9810-git-p4-rcs.sh:  "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_k.py" <"$git/$file" >"$scrub" &&
t/t9810-git-p4-rcs.sh:  "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_ko.py" <"$git/$file" >"$scrub" &&
$ 

I don't run the p4 or svn tests, so ... :-D

Quoted 4 lines

really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.

I agree that fewer languages is (generally) a good idea.

ATB,
Ramsay Jones

#7 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

Ramsay Jones wrote:
[...]

Quoted 1 line

I don't run the p4 or svn tests, so ... :-D

Heh, lucky you. :)

I try to run them all as part of the fedora builds since they cover much more than I'd ever use. That's the main reason I noticed the bare python. That would trip me up when it came time to build on a near-future fedora where python isn't installed by default and I only wanted to require python3 for the build/runtime scripts.

Quoted 7 lines · Jeff King
Quoted 4 lines

really the only user in the whole code base outside of a few fringe commands). Leaving aside any perl vs python flame-war, I think there's value in keeping the number of languages limited when there's not a compelling reason to do otherwise.

I agree that fewer languages is (generally) a good idea.

Yep, that's certainly even better and if Jeff H. can use perl relatively easily (or one of the many perl folks here can help with that part of the test), that's great. The best way to solve many problems is avoid having them. :)

Thanks,

Signature · 3 lines
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Chaos, panic, and disorder - my job is done here.
#9 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Ramsay Jones
Quoted 9 lines · Ramsay Jones
Quoted 6 lines

Probably. We may want to go the same route as we did for perl in a0e0ec9f7d (t: provide a perl() function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.

That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and

Hmm, not quite the _first_ use:

Quoted 11 lines
$ git grep PYTHON_PATH -- t
t/lib-git-p4.sh:        (cd / && "$PYTHON_PATH" -c 'import time; print(int(time.time()))')
t/lib-git-p4.sh:        "$PYTHON_PATH" "$TRASH_DIRECTORY/marshal-dump.py"
t/t9020-remote-svn.sh:export PATH PYTHON_PATH GIT_BUILD_DIR
t/t9020-remote-svn.sh:exec "$PYTHON_PATH" "$GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py" "$@"
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/k_smush.py" <"$cli/k-text-k" >cli-k-text-k-smush &&
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/ko_smush.py" <"$cli/k-text-ko" >cli-k-text-ko-smush &&
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/gendouble.py" >%double.png &&
t/t9810-git-p4-rcs.sh:  "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_k.py" <"$git/$file" >"$scrub" &&
t/t9810-git-p4-rcs.sh:  "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_ko.py" <"$git/$file" >"$scrub" &&
$ 

OK, the first for a feature that is not already written in python (leading into my second claim that python is used only for fringe commands ;) ).

Though maybe I am wrong that the remote-svn stuff requires python. I thought it did, but poking around, it looks like it's all C, and just the "svnrdump_sim" helper is python.

At any rate, I think the point still stands that perl is our main scripting language. I'd rather keep us to that unless there's a good reason not to.

-Peff

#8 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Todd Zullinger
Quoted 4 lines · Todd Zullinger

Ramsay Jones wrote:
[...]

Quoted 1 line

I don't run the p4 or svn tests, so ... :-D

Quoted 8 lines

Heh, lucky you. :)

I try to run them all as part of the fedora builds since they cover much more than I'd ever use. That's the main reason I noticed the bare python. That would trip me up when it came time to build on a near-future fedora where python isn't installed by default and I only wanted to require python3 for the build/runtime scripts.

I'm not actually sure those svn bits involving python are usable by real users. The main tool that people us for svn interop is git-svn, which is written in perl.

There was an effort to have a real "git-remote-svn" helper (so that you could just seamlessly "git clone svn://..." instead of using the "git svn" wrapper.

So we have the work in vcs-svn, which gets built as part of a regular compile. But AFAICT it is only used for git-remote-testsvn, t/helper/test-svn-fe, and contrib/svn-fe. None of which have seen any substantive work since 2012[1].

I suppose it's possible somebody could be using "git clone testsvn://" in the wild, but the name would hopefully warn them off. I have no idea how usable that work is in practice.

-Peff

[1] Browsing "git log", most of the commits are just tree-wide cleanups, or fixing some compilation or dependency error. The last "real" commit seems to be around 8e43a1d010 (remote-svn: add incremental import, 2012-09-19).

I wonder if it is worth dropping this experiment as incomplete and unmaintained. People have obviously spent time dealing with the code for various fixups, but I think the whole thing may essentially just be dead code. Or maybe people really are using contrib/svn-fe. Like I said, I have no clue if this stuff is even usable, but we certainly haven't packaged it to be seen by most users.

#10 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Jeff King
Quoted 3 lines · Jeff King

Though maybe I am wrong that the remote-svn stuff requires python. I thought it did, but poking around, it looks like it's all C, and just the "svnrdump_sim" helper is python.

I think I was getting this mixed up with the git_remote_helpers python work, which was removed long ago in ae34ac126f (git_remote_helpers:
remove little used Python library, 2013-09-07).

Note that it really changes much, but I was curious enough to dig down.

-Peff

#11 lore raw

Re: [PATCH v7 2/2] json-writer: t0019: add Python unit test

In reply to Jeff King
Quoted 23 lines · Jeff King
Quoted 9 lines · Ramsay Jones
Quoted 6 lines

Probably. We may want to go the same route as we did for perl in a0e0ec9f7d (t: provide a perl() function which uses $PERL_PATH, 2013-10-28) so that test writers don't have to remember this.

That said, I wonder if it would be hard to simply do the python bits here in perl. This is the first use of python in our test scripts (and

Hmm, not quite the _first_ use:

Quoted 11 lines
$ git grep PYTHON_PATH -- t
t/lib-git-p4.sh:        (cd / && "$PYTHON_PATH" -c 'import time; print(int(time.time()))')
t/lib-git-p4.sh:        "$PYTHON_PATH" "$TRASH_DIRECTORY/marshal-dump.py"
t/t9020-remote-svn.sh:export PATH PYTHON_PATH GIT_BUILD_DIR
t/t9020-remote-svn.sh:exec "$PYTHON_PATH" "$GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py" "$@"
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/k_smush.py" <"$cli/k-text-k" >cli-k-text-k-smush &&
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/ko_smush.py" <"$cli/k-text-ko" >cli-k-text-ko-smush &&
t/t9802-git-p4-filetype.sh:             "$PYTHON_PATH" "$TRASH_DIRECTORY/gendouble.py" >%double.png &&
t/t9810-git-p4-rcs.sh:  "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_k.py" <"$git/$file" >"$scrub" &&
t/t9810-git-p4-rcs.sh:  "$PYTHON_PATH" "$TRASH_DIRECTORY/scrub_ko.py" <"$git/$file" >"$scrub" &&
$ 
Quoted 7 lines

OK, the first for a feature that is not already written in python (leading into my second claim that python is used only for fringe commands ;) ).

Though maybe I am wrong that the remote-svn stuff requires python. I thought it did, but poking around, it looks like it's all C, and just the "svnrdump_sim" helper is python.

Heh, I was a bit tired last night, so although I knew that I required python to be installed to run the test-suite, I could not remember why. As soon as I went to bed, ... :)

I recently installed Ubuntu 18.04, so that I could get a heads up on any possible problems later this month when I install Linux Mint 19. In order to get the test-suite to run, I had to set 'PYTHON_PATH = /usr/bin/python3' in my config.mak file. (yes, I could have set NO_PYTHON, but you get the idea).

Ubuntu 18.04 no longer installs python2 by default (it has ported to python3), but presumably you can still install it as '/usr/bin/python' (I didn't check).

In any event, it was t9020-remote-svn.sh that was failing which, despite the name, does not depend on 'svn'. As you already noted, it does run svnrdump_sim.py and the git-remote-testsvn.

Quoted 3 lines

At any rate, I think the point still stands that perl is our main scripting language. I'd rather keep us to that unless there's a good reason not to.

Agreed. And I see it has come to pass ... :-D

ATB,
Ramsay Jones