Skip to contentSkip to conversation

fstests: report: add arch and kernel version info into testsuite attributes

PATCH In review Qu Wenruo 4 participants 5 messages Dec 16, 2022 – Dec 20, 2022 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/1 fstests: report: add arch and kernel version info into testsuite attributes

Although "testcase" tags contain the "timestamp" element, for day-0 testing it can be hard to relate the timestamp to the tested kernel version.

Thus this patch will add a "kernel" element to the "testcase" tag, to indicate the kernel version we're running.
Paired with CONFIG_LOCALVERSION_AUTO=y config, it will easily show the kernel commit we're testing.

Since we're here, also add a "arch" element, as there are more and more aarch64 boards (From RK3399 to Apple M1) able to finish fstests in an acceptable duration, we can no longer assume x86_64 as our only platform.

---
 common/report | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
common/report +1 -1
index 4a747f8d..92586527 100644
@@ -49,7 +49,7 @@ _xunit_make_section_report()
 		date_time=$(date +"%F %T")
 	fi
 	local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
-	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
+	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" arch=\"$(uname -m)\" kernel=\"$(uname -r)\""
 	echo "<testsuite name=\"xfstests\" $stats  $hw_info >" >> $REPORT_DIR/result.xml
 
 	# Properties
Signature · 1 line
2.38.0
#1 lore raw
Quoted 24 lines · Qu Wenruo

Although "testcase" tags contain the "timestamp" element, for day-0 testing it can be hard to relate the timestamp to the tested kernel version.

Thus this patch will add a "kernel" element to the "testcase" tag, to indicate the kernel version we're running.
Paired with CONFIG_LOCALVERSION_AUTO=y config, it will easily show the kernel commit we're testing.

Since we're here, also add a "arch" element, as there are more and more aarch64 boards (From RK3399 to Apple M1) able to finish fstests in an acceptable duration, we can no longer assume x86_64 as our only platform.

---
 common/report | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
common/report +0 -0
index 4a747f8d..92586527 100644

common/report _xunit_make_section_report() · line 53

Quoted 6 lines
common/report +1 -1
@@ -49,7 +49,7 @@ _xunit_make_section_report()
 		date_time=$(date +"%F %T")
 	fi
 	local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
-	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
+	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" arch=\"$(uname -m)\" kernel=\"$(uname -r)\""
 	echo "<testsuite name=\"xfstests\" $stats  $hw_info >" >> $REPORT_DIR/result.xml

The original commit that added this report format was f9fde7db2f ("report: Add xunit format report generator"). Dmitry Monakhov's commit message points out that the xml being emitted was "xunit/junit":

    Footnotes:
    [1] https://xunit.github.io/docs/format-xml-v2.html
    [2] http://help.catchsoftware.com/display/ET/JUnit+Format

The first link is now dead, but the second link contains enough information to find the current junit xml format:

[1] https://raw.githubusercontent.com/windyroad/JUnit-Schema/master/JUnit.xsd

Note that the xunit project appears to have diverged their report format:

[2] https://xunit.net/docs/format-xml-v2

(Or perhaps there were multiple things called xunit?)

Either way, it's pretty obvious from common/report code that the "xunit" code is still emitting junit xml files. I think it's important that fstests should continue to follow that schema, so that these files can be fed into test dashboards (yes I have one) that consume this file format.

Regrettably, the schema does not provide for @arch or @kernel attributes hanging off the <testsuite> element, so it's not a good idea to add things that a strict parser could reject.

That said, I think it's important to record the architecture and kernel. Probably even more attributes than that. The junit xml schema provides for arbitrary string properties to be attached to reports; would you mind putting these there instead?

(I want to add a few more properties now that people have started talking about reporting again... ;))

	# Properties
	echo -e "\t<properties>" >> $REPORT_DIR/result.xml
	echo -e "\t\t<property name=\"arch\" value=\"$(uname -m)\"/>" >> $REPORT_DIR/result.xml
	echo -e "\t\t<property name=\"kernel\" value=\"$(uname -r)\"/>" >> $REPORT_DIR/result.xml
	for p in "${REPORT_ENV_LIST[@]}"; do
		_xunit_add_property "$p"
	done
	echo -e "\t</properties>" >> $REPORT_DIR/result.xml

--D

Quoted 5 lines
common/report +0 -0
 	# Properties
Signature · 1 line
2.38.0
#2 lore raw
Quoted 43 lines · Darrick J. Wong
Quoted 24 lines · Qu Wenruo

Although "testcase" tags contain the "timestamp" element, for day-0 testing it can be hard to relate the timestamp to the tested kernel version.

Thus this patch will add a "kernel" element to the "testcase" tag, to indicate the kernel version we're running.
Paired with CONFIG_LOCALVERSION_AUTO=y config, it will easily show the kernel commit we're testing.

Since we're here, also add a "arch" element, as there are more and more aarch64 boards (From RK3399 to Apple M1) able to finish fstests in an acceptable duration, we can no longer assume x86_64 as our only platform.

---
  common/report | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
common/report +0 -0
index 4a747f8d..92586527 100644

common/report _xunit_make_section_report() · line 53

Quoted 6 lines
common/report +1 -1
@@ -49,7 +49,7 @@ _xunit_make_section_report()
  		date_time=$(date +"%F %T")
  	fi
  	local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
-	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
+	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" arch=\"$(uname -m)\" kernel=\"$(uname -r)\""
  	echo "<testsuite name=\"xfstests\" $stats  $hw_info >" >> $REPORT_DIR/result.xml

The original commit that added this report format was f9fde7db2f ("report: Add xunit format report generator"). Dmitry Monakhov's commit message points out that the xml being emitted was "xunit/junit":

     Footnotes:
     [1] https://xunit.github.io/docs/format-xml-v2.html
     [2] http://help.catchsoftware.com/display/ET/JUnit+Format

The first link is now dead, but the second link contains enough information to find the current junit xml format:

Quoted 6 lines

[1] https://raw.githubusercontent.com/windyroad/JUnit-Schema/master/JUnit.xsd

Note that the xunit project appears to have diverged their report format:

[2] https://xunit.net/docs/format-xml-v2

Great we have something solid to follow.

Quoted 8 lines

(Or perhaps there were multiple things called xunit?)

Either way, it's pretty obvious from common/report code that the "xunit" code is still emitting junit xml files. I think it's important that fstests should continue to follow that schema, so that these files can be fed into test dashboards (yes I have one) that consume this file format.

Totally agreed.

Quoted 4 lines

Regrettably, the schema does not provide for @arch or @kernel attributes hanging off the <testsuite> element, so it's not a good idea to add things that a strict parser could reject.

I'm totally fine with the properties method, but still curious why some parsers would even reject elements with unknown attributes?

Isn't the idea of XML (or JSON) to be flex for newer attributes?

Quoted 9 lines

That said, I think it's important to record the architecture and kernel. Probably even more attributes than that. The junit xml schema provides for arbitrary string properties to be attached to reports; would you mind putting these there instead?

(I want to add a few more properties now that people have started talking about reporting again... ;))

Quoted 8 lines
	# Properties
	echo -e "\t<properties>" >> $REPORT_DIR/result.xml
	echo -e "\t\t<property name=\"arch\" value=\"$(uname -m)\"/>" >> $REPORT_DIR/result.xml
	echo -e "\t\t<property name=\"kernel\" value=\"$(uname -r)\"/>" >> $REPORT_DIR/result.xml
	for p in "${REPORT_ENV_LIST[@]}"; do
		_xunit_add_property "$p"
	done
	echo -e "\t</properties>" >> $REPORT_DIR/result.xml

Sounds pretty good, thanks for all the awesome advices! Qu

Quoted 8 lines

--D

Quoted 5 lines
common/report +0 -0
  	# Properties
Signature · 1 line
2.38.0
#3 lore raw
Quoted 66 lines · Qu Wenruo
Quoted 43 lines · Darrick J. Wong
Quoted 24 lines · Qu Wenruo

Although "testcase" tags contain the "timestamp" element, for day-0 testing it can be hard to relate the timestamp to the tested kernel version.

Thus this patch will add a "kernel" element to the "testcase" tag, to indicate the kernel version we're running.
Paired with CONFIG_LOCALVERSION_AUTO=y config, it will easily show the kernel commit we're testing.

Since we're here, also add a "arch" element, as there are more and more aarch64 boards (From RK3399 to Apple M1) able to finish fstests in an acceptable duration, we can no longer assume x86_64 as our only platform.

---
  common/report | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
common/report +0 -0
index 4a747f8d..92586527 100644

common/report _xunit_make_section_report() · line 53

Quoted 6 lines
common/report +1 -1
@@ -49,7 +49,7 @@ _xunit_make_section_report()
  		date_time=$(date +"%F %T")
  	fi
  	local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
-	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
+	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" arch=\"$(uname -m)\" kernel=\"$(uname -r)\""
  	echo "<testsuite name=\"xfstests\" $stats  $hw_info >" >> $REPORT_DIR/result.xml

The original commit that added this report format was f9fde7db2f ("report: Add xunit format report generator"). Dmitry Monakhov's commit message points out that the xml being emitted was "xunit/junit":

     Footnotes:
     [1] https://xunit.github.io/docs/format-xml-v2.html
     [2] http://help.catchsoftware.com/display/ET/JUnit+Format

The first link is now dead, but the second link contains enough information to find the current junit xml format:

Quoted 6 lines

[1] https://raw.githubusercontent.com/windyroad/JUnit-Schema/master/JUnit.xsd

Note that the xunit project appears to have diverged their report format:

[2] https://xunit.net/docs/format-xml-v2

Great we have something solid to follow.

Quoted 8 lines

(Or perhaps there were multiple things called xunit?)

Either way, it's pretty obvious from common/report code that the "xunit" code is still emitting junit xml files. I think it's important that fstests should continue to follow that schema, so that these files can be fed into test dashboards (yes I have one) that consume this file format.

Totally agreed.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Regrettably, the schema does not provide for @arch or @kernel attributes hanging off the <testsuite> element, so it's not a good idea to add things that a strict parser could reject.

I'm totally fine with the properties method, but still curious why some parsers would even reject elements with unknown attributes?

Isn't the idea of XML (or JSON) to be flex for newer attributes?

Yes, and the idea of xml schemas is to declare exactly what we're going to spit out, and in a format where automated tools can hold us to what we've promised. :)

Note that xml allows for namespaced elements. If we ever want to extend the format by adding our own elements, we prefix them with our own namespace and the validation tools won't trip over them.

Though in the end I found enough bugs in the junit xml specification that I decided it'd be easier to create a new schema file that reflects exactly what common/report creates.

Quoted 18 lines
Quoted 9 lines

That said, I think it's important to record the architecture and kernel. Probably even more attributes than that. The junit xml schema provides for arbitrary string properties to be attached to reports; would you mind putting these there instead?

(I want to add a few more properties now that people have started talking about reporting again... ;))

Quoted 8 lines
	# Properties
	echo -e "\t<properties>" >> $REPORT_DIR/result.xml
	echo -e "\t\t<property name=\"arch\" value=\"$(uname -m)\"/>" >> $REPORT_DIR/result.xml
	echo -e "\t\t<property name=\"kernel\" value=\"$(uname -r)\"/>" >> $REPORT_DIR/result.xml
	for p in "${REPORT_ENV_LIST[@]}"; do
		_xunit_add_property "$p"
	done
	echo -e "\t</properties>" >> $REPORT_DIR/result.xml
Quoted 1 line

Sounds pretty good, thanks for all the awesome advices!

I spent the day working on exporting a lot more test machine config elements, so please have a look at the RFC series that I'm going to send in a little bit.

--D

Quoted 9 lines

Qu

Quoted 7 lines

--D

Quoted 4 lines
common/report +0 -0
  	# Properties
Signature · 1 line
2.38.0
#4 lore raw

For what it's worth, here is how I add properties to the xunit file. I do it after the fact using a tiny python script:

https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/update_properties_xunit

This way I can dump a lot of interesting information about the test run into the XML file. (See attached for the beginning of a results.xml file generated by gce-xfstests.)

I also have a handy script which allows me to run separate file system configs in separate VM's, and then combine the xunuit files together:

https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/merge_xunit
https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/combine_xunit

I'm a **big** fan of this approach, since it allows me to take about 24 hours worth of test runs, shard them across 12 different VM's, so that it akes about 2.5 hours of wall clock time to do a full ext4 test run.

After all, cloud VM time is cheap; software engineering time is expensive. :-)

- Ted

<?xml version="1.0" encoding="utf-8"?>
<testsuite name="xfstests" failures="6" skipped="160" tests="1045" time="14724" hostname="xfstests-ltm- 20220825074228-aa" timestamp="2022-08-25T11:55:34">

        <properties>


                <property name="CMDLINE" value="--instance-name xfstests-ltm-20220825074228-aa --gce-zo

ne us-west1-a --gs-bucket gce-xfstests --kernel gs://gce-xfstests/kernel.deb --bucket-subdir results -- no-email -c xfs/4k -g auto --local-ssd-nvme --image-project gce-xfstests"/>
<property name="FSTESTIMG" value="gce-xfstests/xfstests-202208242313"/> <property name="FSTESTPRJ" value="gce-xfstests"/>
<property name="KERNEL" value="kernel 6.0.0-rc2-xfstests #756 SMP PREEMPT_DYNAMIC Mon

 Aug 22 22:04:33 EDT 2022 x86_64"/>
                <property name="FSTESTVER" value="blktests      4e07b0c (Fri, 15 Jul 2022 14:40:03 +090
0)"/>
                <property name="FSTESTVER" value="fio           fio-3.31 (Tue, 9 Aug 2022 14:41:25 -060
0)"/>
                <property name="FSTESTVER" value="fsverity      v1.5 (Sun, 6 Feb 2022 10:59:13 -0800)"/
Quoted 1 line
                <property name="FSTESTVER" value="ima-evm-utils v1.3.2 (Wed, 28 Oct 2020 13:18:08 -0400
)"/>
                <property name="FSTESTVER" value="nvme-cli      v1.16 (Thu, 11 Nov 2021 13:09:06 -0800)
"/>
                <property name="FSTESTVER" value="quota         v4.05-43-gd2256ac (Fri, 17 Sep 2021 14:
04:16 +0200)"/>
                <property name="FSTESTVER" value="util-linux    v2.38.1 (Thu, 4 Aug 2022 11:06:21 +0200
)"/>
                <property name="FSTESTVER" value="xfsprogs      v5.19.0 (Fri, 12 Aug 2022 13:45:01 -050
0)"/>
                <property name="FSTESTVER" value="xfstests-bld  bb566bcf (Wed, 24 Aug 2022 23:07:24 -04
00)"/>
                <property name="FSTESTVER" value="xfstests      v2022.08.21-8-g289f50f8b (Sun, 21 Aug 2
022 15:21:34 -0400)"/>
                <property name="FSTESTVER" value="zz_build-distro       bullseye"/>
                <property name="FSTESTCFG" value="xfs/4k"/>
                <property name="FSTESTSET" value="-g auto"/>
                <property name="FSTESTEXC" value=""/>
                <property name="FSTESTOPT" value="aex"/>
                <property name="MNTOPTS" value=""/>
                <property name="CPUS" value="2"/>
                <property name="MEM" value="7680"/>
                <property name="DMI_MEM" value="7680 MB (Max capacity)"/>
                <property name="GCE ID" value="8073487017178750853"/>
                <property name="MACHINE TYPE" value="n1-standard-2"/>
                <property name="TESTRUNID" value="ltm-20220825074228-aa"/>
                <property name="TESTCFG" value="xfs/4k"/>
        </properties>