Skip to contentSkip to conversation

fstests: add basic json output support

RFC PATCH In review Qu Wenruo 3 participants 3 messages Dec 15, 2022 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

RFC PATCH 1/1 fstests: add basic json output support

Although the current result files "check.log" and "check.time" is enough for human to read, it's not that easy to parse.

Thus this patch will introduce a json output to "$RESULT_BASE/check.json".

The example output would look like this:

  {
      "section": "(none)",
      "fstype": "btrfs",
      "start_time": 1671103264,
      "arch": "x86_64",
      "kernel": "6.1.0-rc8-custom+",
      "results": [
          {
              "testcase": "btrfs/001",
              "status": "pass",
              "start_time": 1671103264,
              "end_time": 1671103266
          },
          {
              "testcase": "btrfs/006",
              "status": "pass",
              "start_time": 1671103266,
              "end_time": 1671103268
          },
          {
              "testcase": "btrfs/007",
              "status": "pass",
              "start_time": 1671103268,
              "end_time": 1671103271
          }
      ]
  }

Which should make later parsing much easier.

---
Reason for RFC:

- Not crash safe
If one test case caused a crash, the "check.json" file will be an invalid one, missing the closing "] }" string.

- Is json really a good choice?
It may be much easier to convert to a web page, but we will still need to parse and handle the result using another languages anyway, like to determine a regression.

Another alternative is .csv, and it can be much easier to handle. (pure "echo >> $output", no need to handle the comma rule). But for .csv, we may waste a lot of columes for things like "arch", "kernel", "section".

- No good way to handle old results.
Unlike check.log, which can save old results without any problem, json can not allow multiple top-level json objects.

Thus currently old "check.json" will be saved to "check.json.old", further older results will be lost.

---
 check       |  32 ++++++++++++--
 common/json | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 145 insertions(+), 4 deletions(-)
 create mode 100644 common/json
check +28 -4
index d2e51296..cdb7f793 100755
@@ -364,6 +364,11 @@ if ! . ./common/rc; then
 	exit 1
 fi
 
+if ! . ./common/json; then
+	echo "check: failed to source common/json"
+	exit 1
+fi
+
 if [ -n "$subdir_xfile" ]; then
 	for d in $SRC_GROUPS $FSTYP; do
 		[ -f $SRC_DIR/$d/$subdir_xfile ] || continue
@@ -576,6 +581,11 @@ _stash_fail_loop_files() {
 _stash_test_status() {
 	local test_seq="$1"
 	local test_status="$2"
+	local start_epoch="$3"
+	local end_epoch="$4"
+
+	_json_add_one_result "${REPORT_DIR}/check.json" \
+		"$test_seq" "$test_status"  "$start_epoch" "$end_epoch"
 
 	if $do_report && [[ $test_status != "expunge" ]]; then
 		_make_testcase_report "$section" "$test_seq" \
@@ -804,6 +814,19 @@ function run_section()
 	seqres="$check"
 	_check_test_fs
 
+	local json_out
+	local json_section
+	if $OPTIONS_HAVE_SECTIONS; then
+		mkdir -p "$RESULT_BASE/$section"
+		json_out="$RESULT_BASE/$section/check.json"
+		json_section="$section"
+	else
+		mkdir -p "$RESULT_BASE"
+		json_out="$RESULT_BASE/check.json"
+		json_section=""
+	fi
+
+	_json_start "$json_out" "$(_full_fstyp_details)" "$json_section"
 	loop_status=()	# track rerun-on-failure state
 	local tc_status ix
 	local -a _list=( $list )
@@ -863,7 +886,7 @@ function run_section()
 		tc_status="pass"
 		if [ ! -f $seq ]; then
 			echo " - no such test?"
-			_stash_test_status "$seqnum" "$tc_status"
+			_stash_test_status "$seqnum" "non-exist"
 			continue
 		fi
 
@@ -936,7 +959,7 @@ function run_section()
 				      echo -n "	$seqnum -- "
 			cat $seqres.notrun
 			tc_status="notrun"
-			_stash_test_status "$seqnum" "$tc_status"
+			_stash_test_status "$seqnum" "$tc_status" "$start" "$stop"
 
 			# Unmount the scratch fs so that we can wipe the scratch
 			# dev state prior to the next test run.
@@ -991,7 +1014,7 @@ function run_section()
 		if [ ! -f $seq.out ]; then
 			_dump_err "no qualified output"
 			tc_status="fail"
-			_stash_test_status "$seqnum" "$tc_status"
+			_stash_test_status "$seqnum" "$tc_status" "$start" "$stop"
 			continue;
 		fi
 
@@ -1027,11 +1050,12 @@ function run_section()
 				rm -f $seqres.hints
 			fi
 		fi
-		_stash_test_status "$seqnum" "$tc_status"
+		_stash_test_status "$seqnum" "$tc_status" "$start" "$stop"
 	done
 
 	sect_stop=`_wallclock`
 	interrupt=false
+	_json_end "$json_out"
 	_wrapup
 	interrupt=true
 	echo
common/json new file +117 -0

117 lines not on this page · raw message

Signature · 1 line
2.38.0
#1 lore raw

Hi Qu,

Quoted 2 lines · Qu Wenruo

Although the current result files "check.log" and "check.time" is enough for human to read, it's not that easy to parse.

Have you looked at the existing junit XML based report types, available via "check -R xunit ..."? junit is standardized, parsable and supported by tools such as:
- https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
- https://github.com/weiwei/junitparser
- https://ddiss.github.io/online-junit-parser/
- https://plugins.jenkins.io/junit/

Quoted 38 lines

Thus this patch will introduce a json output to "$RESULT_BASE/check.json".

The example output would look like this:

  {
      "section": "(none)",
      "fstype": "btrfs",
      "start_time": 1671103264,
      "arch": "x86_64",
      "kernel": "6.1.0-rc8-custom+",
      "results": [
          {
              "testcase": "btrfs/001",
              "status": "pass",
              "start_time": 1671103264,
              "end_time": 1671103266
          },
          {
              "testcase": "btrfs/006",
              "status": "pass",
              "start_time": 1671103266,
              "end_time": 1671103268
          },
          {
              "testcase": "btrfs/007",
              "status": "pass",
              "start_time": 1671103268,
              "end_time": 1671103271
          }
      ]
  }

Which should make later parsing much easier.

---
Reason for RFC:

Quoted 8 lines

- Not crash safe
If one test case caused a crash, the "check.json" file will be an invalid one, missing the closing "] }" string.

- Is json really a good choice?
It may be much easier to convert to a web page, but we will still need to parse and handle the result using another languages anyway, like to determine a regression.

I'm not opposed to adding an extra json report type, but I really think it should be plumbed into the existing common/report API.

Quoted 4 lines

Another alternative is .csv, and it can be much easier to handle. (pure "echo >> $output", no need to handle the comma rule). But for .csv, we may waste a lot of columes for things like "arch", "kernel", "section".

My preference for any new output formats, especially if they're intended for parsing, is that they're based on an existing standard/tool. E.g.
https://testanything.org .

Cheers, David

#2 lore raw
Quoted 7 lines · David Disseldorp

Hi Qu,

Quoted 2 lines · Qu Wenruo

Although the current result files "check.log" and "check.time" is enough for human to read, it's not that easy to parse.

Quoted 7 lines

Have you looked at the existing junit XML based report types, available via "check -R xunit ..."? junit is standardized, parsable and supported by tools such as:
- https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
- https://github.com/weiwei/junitparser
- https://ddiss.github.io/online-junit-parser/
- https://plugins.jenkins.io/junit/

Oh, that's way better, the existing reporting facility is really what I need.

Quoted 51 lines
Quoted 39 lines

Thus this patch will introduce a json output to "$RESULT_BASE/check.json".

The example output would look like this:

   {
       "section": "(none)",
       "fstype": "btrfs",
       "start_time": 1671103264,
       "arch": "x86_64",
       "kernel": "6.1.0-rc8-custom+",
       "results": [
           {
               "testcase": "btrfs/001",
               "status": "pass",
               "start_time": 1671103264,
               "end_time": 1671103266
           },
           {
               "testcase": "btrfs/006",
               "status": "pass",
               "start_time": 1671103266,
               "end_time": 1671103268
           },
           {
               "testcase": "btrfs/007",
               "status": "pass",
               "start_time": 1671103268,
               "end_time": 1671103271
           }
       ]
   }

Which should make later parsing much easier.

---
Reason for RFC:

Quoted 8 lines

- Not crash safe
If one test case caused a crash, the "check.json" file will be an invalid one, missing the closing "] }" string.

- Is json really a good choice?
It may be much easier to convert to a web page, but we will still need to parse and handle the result using another languages anyway, like to determine a regression.

I'm not opposed to adding an extra json report type, but I really think it should be plumbed into the existing common/report API.

Quoted 8 lines
Quoted 4 lines

Another alternative is .csv, and it can be much easier to handle. (pure "echo >> $output", no need to handle the comma rule). But for .csv, we may waste a lot of columes for things like "arch", "kernel", "section".

My preference for any new output formats, especially if they're intended for parsing, is that they're based on an existing standard/tool. E.g.
https://testanything.org .

That's a much better unified protocol, I'll definitely look into it.

Thanks for all the feedbacks!
Qu

Quoted 2 lines

Cheers, David