Skip to contentSkip to conversation

softfloat tests based on berkeley's testfloat

PATCH Superseded Emilio G. Cota 2 participants 10 messages Sep 8, 2018 – Sep 10, 2018

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover softfloat tests based on berkeley's testfloat

A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html

- Convert copy_qemu_to_soft80 to qemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- Fix Makefile to include .d files so that dependencies are properly tracked
- Update commit log

Grab this from:
https://github.com/cota/qemu/tree/fp-test-v2

Thanks,

Emilio

#1 lore raw

PATCH v2 1/2 gitmodules: add berkeley's softfloat + testfloat version 3

These are BSD-licensed so we can add them as submodules.

---
 .gitmodules                   | 6 ++++++
 tests/fp/berkeley-softfloat-3 | 1 +
 tests/fp/berkeley-testfloat-3 | 1 +
 3 files changed, 8 insertions(+)
 create mode 160000 tests/fp/berkeley-softfloat-3
 create mode 160000 tests/fp/berkeley-testfloat-3
.gitmodules +6 -0
index d108478e0a..165c7c5286 100644
@@ -43,3 +43,9 @@
 [submodule "roms/u-boot-sam460ex"]
 	path = roms/u-boot-sam460ex
 	url = git://git.qemu.org/u-boot-sam460ex.git
+[submodule "tests/fp/testfloat"]
+	path = tests/fp/berkeley-testfloat-3
+	url = git://github.com/ucb-bar/berkeley-testfloat-3
+[submodule "tests/fp/berkeley-softfloat-3"]
+	path = tests/fp/berkeley-softfloat-3
+	url = git://github.com/ucb-bar/berkeley-softfloat-3
tests/fp/berkeley-softfloat-3 new file +1 -0
new file mode 160000
index 0000000000..b64af41c32
@@ -0,0 +1 @@
+Subproject commit b64af41c3276f97f0e181920400ee056b9c88037
tests/fp/berkeley-testfloat-3 new file +1 -0
new file mode 160000
index 0000000000..06b20075dd
@@ -0,0 +1 @@
+Subproject commit 06b20075dd3c1a5d0dd007a93643282832221612
Signature · 1 line
2.17.1
Acked-by on v2 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH v2 1/2] gitmodules: add berkeley's softfloat + testfloat version 3

Quoted 16 lines · Emilio G. Cota

These are BSD-licensed so we can add them as submodules.

---
 .gitmodules                   | 6 ++++++
 tests/fp/berkeley-softfloat-3 | 1 +
 tests/fp/berkeley-testfloat-3 | 1 +
 3 files changed, 8 insertions(+)
 create mode 160000 tests/fp/berkeley-softfloat-3
 create mode 160000 tests/fp/berkeley-testfloat-3
.gitmodules +0 -0
index d108478e0a..165c7c5286 100644

.gitmodules line 51

Quoted 9 lines
.gitmodules +6 -0
@@ -43,3 +43,9 @@
 [submodule "roms/u-boot-sam460ex"]
 	path = roms/u-boot-sam460ex
 	url = git://git.qemu.org/u-boot-sam460ex.git
+[submodule "tests/fp/testfloat"]
+	path = tests/fp/berkeley-testfloat-3
+	url = git://github.com/ucb-bar/berkeley-testfloat-3
+[submodule "tests/fp/berkeley-softfloat-3"]
+	path = tests/fp/berkeley-softfloat-3
+	url = git://github.com/ucb-bar/berkeley-softfloat-3

I think generally when we add submodules we setup a mirror at
git://git.qemu.org so we are only at the whims of our own infrastructure. See https://wiki.qemu.org/AdminContacts

Nevertheless:

Quoted 13 lines
tests/fp/berkeley-softfloat-3 new file +1 -0
new file mode 160000
index 0000000000..b64af41c32
@@ -0,0 +1 @@
+Subproject commit b64af41c3276f97f0e181920400ee056b9c88037
tests/fp/berkeley-testfloat-3 new file +0 -0
new file mode 160000
index 0000000000..06b20075dd

tests/fp/berkeley-testfloat-3 line 1

Quoted 1 line
tests/fp/berkeley-testfloat-3 new file +1 -0
@@ -0,0 +1 @@
+Subproject commit 06b20075dd3c1a5d0dd007a93643282832221612

--
Alex Bennée

#3 lore raw

PATCH v2 2/2 tests: add floating point tests

In reply to Emilio G. Cota

By leveraging berkeley's softfloat and testfloat.

fp-test.c is derived from testfloat's testsoftfloat.c. To ease the tracking of upstream changes to the latter file, fp-test.c keeps the original camel-case variable naming, and includes most new code via wrap.inc.c.

Most changes to the original code are simple style changes, although a couple of not-so-subtle modifications have been made (noted with XXX in the code), namely:

- We do not test ROUND_ODD, since not all of our primitives support it (e.g. fp16)

- Do not test !exact in round-to-integer, since it is not implemented in QEMU (this flag was added to softfloat v3).

---
 configure              |    2 +
 tests/fp/platform.h    |   41 ++
 tests/fp/fp-test.c     | 1052 ++++++++++++++++++++++++++++++++++++++++
 tests/fp/wrap.inc.c    |  600 +++++++++++++++++++++++
 tests/Makefile.include |    3 +
 tests/fp/.gitignore    |    1 +
 tests/fp/Makefile      |  591 ++++++++++++++++++++++
 7 files changed, 2290 insertions(+)
 create mode 100644 tests/fp/platform.h
 create mode 100644 tests/fp/fp-test.c
 create mode 100644 tests/fp/wrap.inc.c
 create mode 100644 tests/fp/.gitignore
 create mode 100644 tests/fp/Makefile
configure +2 -0
index 58862d2ae8..b02da8c0b4 100755
@@ -7451,12 +7451,14 @@ fi
 
 # build tree in object directory in case the source is not in the current directory
 DIRS="tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/qapi-schema tests/tcg/xtensa tests/qemu-iotests tests/vm"
+DIRS="$DIRS tests/fp"
 DIRS="$DIRS docs docs/interop fsdev scsi"
 DIRS="$DIRS pc-bios/optionrom pc-bios/spapr-rtas pc-bios/s390-ccw"
 DIRS="$DIRS roms/seabios roms/vgabios"
 FILES="Makefile tests/tcg/Makefile qdict-test-data.txt"
 FILES="$FILES tests/tcg/cris/Makefile tests/tcg/cris/.gdbinit"
 FILES="$FILES tests/tcg/lm32/Makefile tests/tcg/xtensa/Makefile po/Makefile"
+FILES="$FILES tests/fp/Makefile"
 FILES="$FILES pc-bios/optionrom/Makefile pc-bios/keymaps"
 FILES="$FILES pc-bios/spapr-rtas/Makefile"
 FILES="$FILES pc-bios/s390-ccw/Makefile"
tests/fp/platform.h new file +41 -0

41 lines not on this page · raw message

tests/fp/fp-test.c new file +1052 -0

1,052 lines not on this page · raw message

tests/fp/wrap.inc.c new file +600 -0

600 lines not on this page · raw message

tests/Makefile.include +3 -0
index 87c81d1dcc..363f133101 100644
@@ -657,6 +657,9 @@ tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y)
 tests/test-bufferiszero$(EXESUF): tests/test-bufferiszero.o $(test-util-obj-y)
 tests/atomic_add-bench$(EXESUF): tests/atomic_add-bench.o $(test-util-obj-y)
 
+tests/fp/%:
+	$(MAKE) -C $(dir $@) $(notdir $@)
+
 tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
 	hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\
 	hw/core/bus.o \
tests/fp/.gitignore new file +1 -0
new file mode 100644
index 0000000000..8d45d18ac4
@@ -0,0 +1 @@
+fp-test
tests/fp/Makefile new file +591 -0

591 lines not on this page · raw message

Signature · 1 line
2.17.1
#4 lore raw

Re: [Qemu-devel] [PATCH v2 2/2] tests: add floating point tests

Quoted 27 lines · Emilio G. Cota

By leveraging berkeley's softfloat and testfloat.

fp-test.c is derived from testfloat's testsoftfloat.c. To ease the tracking of upstream changes to the latter file, fp-test.c keeps the original camel-case variable naming, and includes most new code via wrap.inc.c.

Most changes to the original code are simple style changes, although a couple of not-so-subtle modifications have been made (noted with XXX in the code), namely:

- We do not test ROUND_ODD, since not all of our primitives support it (e.g. fp16)

- Do not test !exact in round-to-integer, since it is not implemented in QEMU (this flag was added to softfloat v3).

---
 configure              |    2 +
 tests/fp/platform.h    |   41 ++
 tests/fp/fp-test.c     | 1052 ++++++++++++++++++++++++++++++++++++++++
 tests/fp/wrap.inc.c    |  600 +++++++++++++++++++++++
 tests/Makefile.include |    3 +
 tests/fp/.gitignore    |    1 +
 tests/fp/Makefile      |  591 ++++++++++++++++++++++
 7 files changed, 2290 insertions(+)

<snip>

tests/Makefile.include tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) · line 662

Quoted 11 lines
tests/Makefile.include +3 -0
index 87c81d1dcc..363f133101 100644
@@ -657,6 +657,9 @@ tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y)
 tests/test-bufferiszero$(EXESUF): tests/test-bufferiszero.o $(test-util-obj-y)
 tests/atomic_add-bench$(EXESUF): tests/atomic_add-bench.o $(test-util-obj-y)
 
+tests/fp/%:
+	$(MAKE) -C $(dir $@) $(notdir $@)
+

This isn't enough to cause the build to be included in make check or indeed be run. Perhaps that should be included in a new patch in the series?

tests/Makefile.include tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) · line 665

Quoted 3 lines
tests/Makefile.include +0 -0
 tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
 	hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\
 	hw/core/bus.o \

<snip>

--
Alex Bennée

#5 lore raw

Re: [Qemu-devel] [PATCH v2 2/2] tests: add floating point tests

Quoted 9 lines · Alex Bennée
Quoted 8 lines · Emilio G. Cota
tests/Makefile.include +0 -0
index 87c81d1dcc..363f133101 100644
@@ -657,6 +657,9 @@ tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y)
 tests/test-bufferiszero$(EXESUF): tests/test-bufferiszero.o $(test-util-obj-y)
 tests/atomic_add-bench$(EXESUF): tests/atomic_add-bench.o $(test-util-obj-y)
 
Quoted 7 lines

tests/Makefile.include tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) · line 662

Quoted 3 lines
tests/Makefile.include +3 -0
+tests/fp/%:
+	$(MAKE) -C $(dir $@) $(notdir $@)
+

This isn't enough to cause the build to be included in make check or indeed be run. Perhaps that should be included in a new patch in the series?

Yes I figured we should fix the reported errors first -- otherwise we'll knowingly break the integration tests.

Emilio

#6 lore raw
In reply to Emilio G. Cota
Quoted 9 lines · Emilio G. Cota

A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html

- Convert copy_qemu_to_soft80 to qemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- Fix Makefile to include .d files so that dependencies are properly tracked
- Update commit log

Just some general comments:

- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining our fp-test.c in close alignment to softfloat unless we expect much upstreaming of changes.
- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:

tests/fp/fp-test.c - 53.5 % coverage 43.3 % branch coverage fpu/softfloat.c - 32.5 % coverage 25.1 % branch coverage

But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test fFOO_BAR.

Quoted 7 lines

Grab this from:
https://github.com/cota/qemu/tree/fp-test-v2

Thanks,

Emilio

--
Alex Bennée

#7 lore raw
Quoted 13 lines · Alex Bennée
Quoted 9 lines · Emilio G. Cota

A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html

- Convert copy_qemu_to_soft80 to qemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- Fix Makefile to include .d files so that dependencies are properly tracked
- Update commit log

Quoted 5 lines

Just some general comments:

- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining our fp-test.c in close alignment to softfloat unless we expect much upstreaming of changes.

I don't think there'll be any, to be honest. They have a git repo but I doubt they'll take any patches. This was just a minimum attempt to get some tests working (I don't have a lot of time to work on this)

Quoted 9 lines

- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:

tests/fp/fp-test.c - 53.5 % coverage 43.3 % branch coverage fpu/softfloat.c - 32.5 % coverage 25.1 % branch coverage

But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test fFOO_BAR.

Yes right now to get better coverage you need to run it several times. After a few runs I got it to 58%, but still without testing some rounding modes. So considering we're coming from 0% coverage, I'd say coverage is pretty good!

But yes, we should add a flag to just test -all.

There are some functions that we are not testing yet though, but that can be fixed over time (we can add more tests to fp-test even though they're not in testfloat, such as testing flush-to-zero/ denormals-are-zero, or the muladd variants that we have).

I have basically no time left to work on this. What do you think about the following plan?

1. Have our own clones (forks) of testfloat/softfloat in qemu servers.
2. Add those as submodules
3. Add fp-test with as you said an -all flag that reports all errors to get decent coverage.
4. Add hardfloat patches, with tests. This requires a small change to testfloat:
https://github.com/cota/berkeley-testfloat-3/commit/ca9fa2ba05

I'd then leave adding further tests to increase coverage and fixing the existing bugs (prior to hardfloat) to someone else with more time/resources.

Thanks,

Emilio

#8 lore raw
Quoted 30 lines · Emilio G. Cota
Quoted 13 lines · Alex Bennée
Quoted 9 lines · Emilio G. Cota

A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html

- Convert copy_qemu_to_soft80 to qemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- Fix Makefile to include .d files so that dependencies are properly tracked
- Update commit log

Quoted 5 lines

Just some general comments:

- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining our fp-test.c in close alignment to softfloat unless we expect much upstreaming of changes.

I don't think there'll be any, to be honest. They have a git repo but I doubt they'll take any patches. This was just a minimum attempt to get some tests working (I don't have a lot of time to work on this)

Quoted 6 lines

- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:

tests/fp/fp-test.c - 53.5 % coverage 43.3 % branch coverage fpu/softfloat.c - 32.5 % coverage 25.1 % branch coverage

Quoted 8 lines
Quoted 3 lines

But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test fFOO_BAR.

Yes right now to get better coverage you need to run it several times. After a few runs I got it to 58%, but still without testing some rounding modes. So considering we're coming from 0% coverage, I'd say coverage is pretty good!

I agree ;-)

Quoted 8 lines

But yes, we should add a flag to just test -all.

There are some functions that we are not testing yet though, but that can be fixed over time (we can add more tests to fp-test even though they're not in testfloat, such as testing flush-to-zero/ denormals-are-zero, or the muladd variants that we have).

Quoted 7 lines

I have basically no time left to work on this. What do you think about the following plan?

1. Have our own clones (forks) of testfloat/softfloat in qemu servers.
2. Add those as submodules
3. Add fp-test with as you said an -all flag that reports all errors to get decent coverage.

Works for me.

Quoted 3 lines

4. Add hardfloat patches, with tests. This requires a small change to testfloat:
https://github.com/cota/berkeley-testfloat-3/commit/ca9fa2ba05

It seems a shame to have our mirror of testfloat diverge but if it can't be avoided I guess it will have to do. Couldn't we just zero the HW flags in the wrapped up function? I may be misremembering but don't we fall-back to softfloat code anyway if we have reached any limits that trigger exception flags?

Quoted 3 lines

I'd then leave adding further tests to increase coverage and fixing the existing bugs (prior to hardfloat) to someone else with more time/resources.

Agreed. I should have time to chase down the softfloat regressions once I'm back from Connect after next week.

Quoted 4 lines

Thanks,

Emilio

--
Alex Bennée

#9 lore raw
Quoted 66 lines · Alex Bennée
Quoted 30 lines · Emilio G. Cota
Quoted 13 lines · Alex Bennée
Quoted 9 lines · Emilio G. Cota

A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html

- Convert copy_qemu_to_soft80 to qemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- Fix Makefile to include .d files so that dependencies are properly tracked
- Update commit log

Quoted 5 lines

Just some general comments:

- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining our fp-test.c in close alignment to softfloat unless we expect much upstreaming of changes.

I don't think there'll be any, to be honest. They have a git repo but I doubt they'll take any patches. This was just a minimum attempt to get some tests working (I don't have a lot of time to work on this)

Quoted 6 lines

- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:

tests/fp/fp-test.c - 53.5 % coverage 43.3 % branch coverage fpu/softfloat.c - 32.5 % coverage 25.1 % branch coverage

Quoted 8 lines
Quoted 3 lines

But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test fFOO_BAR.

Yes right now to get better coverage you need to run it several times. After a few runs I got it to 58%, but still without testing some rounding modes. So considering we're coming from 0% coverage, I'd say coverage is pretty good!

I agree ;-)

Quoted 8 lines

But yes, we should add a flag to just test -all.

There are some functions that we are not testing yet though, but that can be fixed over time (we can add more tests to fp-test even though they're not in testfloat, such as testing flush-to-zero/ denormals-are-zero, or the muladd variants that we have).

Quoted 7 lines

I have basically no time left to work on this. What do you think about the following plan?

1. Have our own clones (forks) of testfloat/softfloat in qemu servers.
2. Add those as submodules
3. Add fp-test with as you said an -all flag that reports all errors to get decent coverage.

Works for me.

Quoted 3 lines

4. Add hardfloat patches, with tests. This requires a small change to testfloat:
https://github.com/cota/berkeley-testfloat-3/commit/ca9fa2ba05

Quoted 5 lines

It seems a shame to have our mirror of testfloat diverge but if it can't be avoided I guess it will have to do. Couldn't we just zero the HW flags in the wrapped up function? I may be misremembering but don't we fall-back to softfloat code anyway if we have reached any limits that trigger exception flags?

testfloat compares the result of the ops as well as the resulting flags. It zeroes the exception flags register before computing the "golden" result/flags. The hardfloat code only kicks in when inexact is set, so we need to add the above patch to testfloat so that we can control the initial flags from outside. With this patch + a patch on the fp-test side to set the initial flags, we can choose whether to test hardfloat or not.

About your second question: yes, for corner cases we defer to softfloat. But to reach those corner cases we first have to reach the hardfloat code, which requires inexact to be set.

Quoted 6 lines
Quoted 3 lines

I'd then leave adding further tests to increase coverage and fixing the existing bugs (prior to hardfloat) to someone else with more time/resources.

Agreed. I should have time to chase down the softfloat regressions once I'm back from Connect after next week.

Nice! I'll try to have something ready later today.

Emilio