softfloat tests based on berkeley's testfloat
v2 cover softfloat tests based on berkeley's testfloat
A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html
- Convert copy_qemu_to_soft80 to qemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- Fix Makefile to include .d files so that dependencies are properly tracked
- Update commit log
Grab this from:
https://github.com/cota/qemu/tree/fp-test-v2
Thanks,
Emilio
PATCH v2 1/2 gitmodules: add berkeley's softfloat + testfloat version 3
These are BSD-licensed so we can add them as submodules.
--- .gitmodules | 6 ++++++ tests/fp/berkeley-softfloat-3 | 1 + tests/fp/berkeley-testfloat-3 | 1 + 3 files changed, 8 insertions(+) create mode 160000 tests/fp/berkeley-softfloat-3 create mode 160000 tests/fp/berkeley-testfloat-3
.gitmodules +6 -0
@@ -43,3 +43,9 @@ [submodule "roms/u-boot-sam460ex"] path = roms/u-boot-sam460ex url = git://git.qemu.org/u-boot-sam460ex.git +[submodule "tests/fp/testfloat"] + path = tests/fp/berkeley-testfloat-3 + url = git://github.com/ucb-bar/berkeley-testfloat-3 +[submodule "tests/fp/berkeley-softfloat-3"] + path = tests/fp/berkeley-softfloat-3 + url = git://github.com/ucb-bar/berkeley-softfloat-3
tests/fp/berkeley-softfloat-3 new file +1 -0
@@ -0,0 +1 @@ +Subproject commit b64af41c3276f97f0e181920400ee056b9c88037
tests/fp/berkeley-testfloat-3 new file +1 -0
@@ -0,0 +1 @@ +Subproject commit 06b20075dd3c1a5d0dd007a93643282832221612
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 1/2] gitmodules: add berkeley's softfloat + testfloat version 3
Quoted 16 lines · Emilio G. Cota
These are BSD-licensed so we can add them as submodules.
Signed-off-by Emilio G. Cota <[email protected]>--- .gitmodules | 6 ++++++ tests/fp/berkeley-softfloat-3 | 1 + tests/fp/berkeley-testfloat-3 | 1 + 3 files changed, 8 insertions(+) create mode 160000 tests/fp/berkeley-softfloat-3 create mode 160000 tests/fp/berkeley-testfloat-3.gitmodules +0
-0
.gitmodules line 51
Quoted 9 lines
.gitmodules +6
-0@@ -43,3 +43,9 @@ [submodule "roms/u-boot-sam460ex"] path = roms/u-boot-sam460ex url = git://git.qemu.org/u-boot-sam460ex.git +[submodule "tests/fp/testfloat"] + path = tests/fp/berkeley-testfloat-3 + url = git://github.com/ucb-bar/berkeley-testfloat-3 +[submodule "tests/fp/berkeley-softfloat-3"] + path = tests/fp/berkeley-softfloat-3 + url = git://github.com/ucb-bar/berkeley-softfloat-3
I think generally when we add submodules we setup a mirror atgit://git.qemu.org so we are only at the whims of our own infrastructure. See https://wiki.qemu.org/AdminContacts
Nevertheless:
Quoted 13 lines
tests/fp/berkeley-softfloat-3 new file +1
-0@@ -0,0 +1 @@ +Subproject commit b64af41c3276f97f0e181920400ee056b9c88037tests/fp/berkeley-testfloat-3 new file +0
-0
tests/fp/berkeley-testfloat-3 line 1
Quoted 1 line
tests/fp/berkeley-testfloat-3 new file +1
-0@@ -0,0 +1 @@ +Subproject commit 06b20075dd3c1a5d0dd007a93643282832221612
--
Alex Bennée
PATCH v2 2/2 tests: add floating point tests
In reply to Emilio G. CotaBy leveraging berkeley's softfloat and testfloat.
fp-test.c is derived from testfloat's testsoftfloat.c. To ease the tracking of upstream changes to the latter file, fp-test.c keeps the original camel-case variable naming, and includes most new code via wrap.inc.c.
Most changes to the original code are simple style changes, although a couple of not-so-subtle modifications have been made (noted with XXX in the code), namely:
- We do not test ROUND_ODD, since not all of our primitives support it (e.g. fp16)
- Do not test !exact in round-to-integer, since it is not implemented in QEMU (this flag was added to softfloat v3).
--- configure | 2 + tests/fp/platform.h | 41 ++ tests/fp/fp-test.c | 1052 ++++++++++++++++++++++++++++++++++++++++ tests/fp/wrap.inc.c | 600 +++++++++++++++++++++++ tests/Makefile.include | 3 + tests/fp/.gitignore | 1 + tests/fp/Makefile | 591 ++++++++++++++++++++++ 7 files changed, 2290 insertions(+) create mode 100644 tests/fp/platform.h create mode 100644 tests/fp/fp-test.c create mode 100644 tests/fp/wrap.inc.c create mode 100644 tests/fp/.gitignore create mode 100644 tests/fp/Makefile
configure +2 -0
@@ -7451,12 +7451,14 @@ fi # build tree in object directory in case the source is not in the current directory DIRS="tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/qapi-schema tests/tcg/xtensa tests/qemu-iotests tests/vm" +DIRS="$DIRS tests/fp" DIRS="$DIRS docs docs/interop fsdev scsi" DIRS="$DIRS pc-bios/optionrom pc-bios/spapr-rtas pc-bios/s390-ccw" DIRS="$DIRS roms/seabios roms/vgabios" FILES="Makefile tests/tcg/Makefile qdict-test-data.txt" FILES="$FILES tests/tcg/cris/Makefile tests/tcg/cris/.gdbinit" FILES="$FILES tests/tcg/lm32/Makefile tests/tcg/xtensa/Makefile po/Makefile" +FILES="$FILES tests/fp/Makefile" FILES="$FILES pc-bios/optionrom/Makefile pc-bios/keymaps" FILES="$FILES pc-bios/spapr-rtas/Makefile" FILES="$FILES pc-bios/s390-ccw/Makefile"
tests/fp/platform.h new file +41 -0
41 lines not on this page · raw message
tests/fp/fp-test.c new file +1052 -0
1,052 lines not on this page · raw message
tests/fp/wrap.inc.c new file +600 -0
600 lines not on this page · raw message
tests/Makefile.include +3 -0
@@ -657,6 +657,9 @@ tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) tests/test-bufferiszero$(EXESUF): tests/test-bufferiszero.o $(test-util-obj-y) tests/atomic_add-bench$(EXESUF): tests/atomic_add-bench.o $(test-util-obj-y) +tests/fp/%: + $(MAKE) -C $(dir $@) $(notdir $@) + tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \ hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\ hw/core/bus.o \
tests/fp/.gitignore new file +1 -0
@@ -0,0 +1 @@ +fp-test
tests/fp/Makefile new file +591 -0
591 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [Qemu-devel] [PATCH v2 2/2] tests: add floating point tests
Quoted 27 lines · Emilio G. Cota
By leveraging berkeley's softfloat and testfloat.
fp-test.cis derived from testfloat'stestsoftfloat.c. To ease the tracking of upstream changes to the latter file,fp-test.ckeeps the original camel-case variable naming, and includes most new code viawrap.inc.c.Most changes to the original code are simple style changes, although a couple of not-so-subtle modifications have been made (noted with XXX in the code), namely:
- We do not test
ROUND_ODD, since not all of our primitives support it (e.g. fp16)- Do not test !exact in round-to-integer, since it is not implemented in QEMU (this flag was added to softfloat v3).
Signed-off-by Emilio G. Cota <[email protected]>--- configure | 2 + tests/fp/platform.h | 41 ++ tests/fp/fp-test.c | 1052 ++++++++++++++++++++++++++++++++++++++++ tests/fp/wrap.inc.c | 600 +++++++++++++++++++++++ tests/Makefile.include | 3 + tests/fp/.gitignore | 1 + tests/fp/Makefile | 591 ++++++++++++++++++++++ 7 files changed, 2290 insertions(+)
<snip>
tests/Makefile.include tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) · line 662
Quoted 11 lines
tests/Makefile.include +3
-0@@ -657,6 +657,9 @@ tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) tests/test-bufferiszero$(EXESUF): tests/test-bufferiszero.o $(test-util-obj-y) tests/atomic_add-bench$(EXESUF): tests/atomic_add-bench.o $(test-util-obj-y) +tests/fp/%: + $(MAKE) -C $(dir $@) $(notdir $@) +
This isn't enough to cause the build to be included in make check or indeed be run. Perhaps that should be included in a new patch in the series?
tests/Makefile.include tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) · line 665
Quoted 3 lines
tests/Makefile.include +0
-0tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \ hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\ hw/core/bus.o \
<snip>
--
Alex Bennée
Re: [Qemu-devel] [PATCH v2 2/2] tests: add floating point tests
Quoted 9 lines · Alex Bennée
Quoted 8 lines · Emilio G. Cota
tests/Makefile.include +0
-0@@ -657,6 +657,9 @@ tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) tests/test-bufferiszero$(EXESUF): tests/test-bufferiszero.o $(test-util-obj-y) tests/atomic_add-bench$(EXESUF): tests/atomic_add-bench.o $(test-util-obj-y)
Quoted 7 lines
tests/Makefile.include tests/qht-bench$(EXESUF): tests/qht-bench.o $(test-util-obj-y) · line 662
Quoted 3 lines
tests/Makefile.include +3
-0+tests/fp/%: + $(MAKE) -C $(dir $@) $(notdir $@) +This isn't enough to cause the build to be included in make check or indeed be run. Perhaps that should be included in a new patch in the series?
Yes I figured we should fix the reported errors first -- otherwise we'll knowingly break the integration tests.
Emilio
Quoted 9 lines · Emilio G. Cota
A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html- Convert
copy_qemu_to_soft80toqemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- FixMakefileto include .d files so that dependencies are properly tracked
- Update commit log
Just some general comments:
- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining our fp-test.c in close alignment to softfloat unless we expect much upstreaming of changes.
- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:
tests/fp/fp-test.c - 53.5 % coverage 43.3 % branch coverage fpu/softfloat.c - 32.5 % coverage 25.1 % branch coverage
But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test fFOO_BAR.
Quoted 7 lines
Grab this from:
https://github.com/cota/qemu/tree/fp-test-v2Thanks,
Emilio
--
Alex Bennée
Quoted 13 lines · Alex Bennée
Quoted 9 lines · Emilio G. Cota
A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html- Convert
copy_qemu_to_soft80toqemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- FixMakefileto include .d files so that dependencies are properly tracked
- Update commit log
Quoted 5 lines
Just some general comments:
- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining ourfp-test.cin close alignment to softfloat unless we expect much upstreaming of changes.
I don't think there'll be any, to be honest. They have a git repo but I doubt they'll take any patches. This was just a minimum attempt to get some tests working (I don't have a lot of time to work on this)
Quoted 9 lines
- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:
tests/fp/fp-test.c- 53.5 % coverage 43.3 % branch coveragefpu/softfloat.c- 32.5 % coverage 25.1 % branch coverageBut maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test
fFOO_BAR.
Yes right now to get better coverage you need to run it several times. After a few runs I got it to 58%, but still without testing some rounding modes. So considering we're coming from 0% coverage, I'd say coverage is pretty good!
But yes, we should add a flag to just test -all.
There are some functions that we are not testing yet though, but that can be fixed over time (we can add more tests to fp-test even though they're not in testfloat, such as testing flush-to-zero/ denormals-are-zero, or the muladd variants that we have).
I have basically no time left to work on this. What do you think about the following plan?
1. Have our own clones (forks) of testfloat/softfloat in qemu servers.
2. Add those as submodules
3. Add fp-test with as you said an -all flag that reports all errors to get decent coverage.
4. Add hardfloat patches, with tests. This requires a small change to testfloat:
https://github.com/cota/berkeley-testfloat-3/commit/ca9fa2ba05
I'd then leave adding further tests to increase coverage and fixing the existing bugs (prior to hardfloat) to someone else with more time/resources.
Thanks,
Emilio
Quoted 30 lines · Emilio G. Cota
Quoted 13 lines · Alex Bennée
Quoted 9 lines · Emilio G. Cota
A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html- Convert
copy_qemu_to_soft80toqemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- FixMakefileto include .d files so that dependencies are properly tracked
- Update commit logQuoted 5 lines
Just some general comments:
- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining ourfp-test.cin close alignment to softfloat unless we expect much upstreaming of changes.I don't think there'll be any, to be honest. They have a git repo but I doubt they'll take any patches. This was just a minimum attempt to get some tests working (I don't have a lot of time to work on this)
Quoted 6 lines
- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:
tests/fp/fp-test.c- 53.5 % coverage 43.3 % branch coveragefpu/softfloat.c- 32.5 % coverage 25.1 % branch coverage
Quoted 8 lines
Quoted 3 lines
But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test
fFOO_BAR.Yes right now to get better coverage you need to run it several times. After a few runs I got it to 58%, but still without testing some rounding modes. So considering we're coming from 0% coverage, I'd say coverage is pretty good!
I agree ;-)
Quoted 8 lines
But yes, we should add a flag to just test -all.
There are some functions that we are not testing yet though, but that can be fixed over time (we can add more tests to fp-test even though they're not in testfloat, such as testing flush-to-zero/ denormals-are-zero, or the muladd variants that we have).
Quoted 7 lines
I have basically no time left to work on this. What do you think about the following plan?
1. Have our own clones (forks) of testfloat/softfloat in qemu servers.
2. Add those as submodules
3. Add fp-test with as you said an -all flag that reports all errors to get decent coverage.
Works for me.
Quoted 3 lines
4. Add hardfloat patches, with tests. This requires a small change to testfloat:
https://github.com/cota/berkeley-testfloat-3/commit/ca9fa2ba05
It seems a shame to have our mirror of testfloat diverge but if it can't be avoided I guess it will have to do. Couldn't we just zero the HW flags in the wrapped up function? I may be misremembering but don't we fall-back to softfloat code anyway if we have reached any limits that trigger exception flags?
Quoted 3 lines
I'd then leave adding further tests to increase coverage and fixing the existing bugs (prior to hardfloat) to someone else with more time/resources.
Agreed. I should have time to chase down the softfloat regressions once I'm back from Connect after next week.
Quoted 4 lines
Thanks,
Emilio
--
Alex Bennée
Quoted 66 lines · Alex Bennée
Quoted 30 lines · Emilio G. Cota
Quoted 13 lines · Alex Bennée
Quoted 9 lines · Emilio G. Cota
A few fixes since yesterday's v1:
https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg00884.html- Convert
copy_qemu_to_soft80toqemu_to_soft80, just like the other conversion functions
- Set fp-test as the program name as reported by itself
- FixMakefileto include .d files so that dependencies are properly tracked
- Update commit logQuoted 5 lines
Just some general comments:
- I think this is a better way to go than the IBM test suite
- I'm ambivalent about maintaining ourfp-test.cin close alignment to softfloat unless we expect much upstreaming of changes.I don't think there'll be any, to be honest. They have a git repo but I doubt they'll take any patches. This was just a minimum attempt to get some tests working (I don't have a lot of time to work on this)
Quoted 6 lines
- the coverage seems a bit low. Rebuilding everything with --enable-gcov and running -all1 -all2 I get:
tests/fp/fp-test.c- 53.5 % coverage 43.3 % branch coveragefpu/softfloat.c- 32.5 % coverage 25.1 % branch coverageQuoted 8 lines
Quoted 3 lines
But maybe I didn't pass enough options to fp-test? I could really do with a --just-run-everything-and-summarise-failing-functions option so I can then go through in more detail with fp-test
fFOO_BAR.Yes right now to get better coverage you need to run it several times. After a few runs I got it to 58%, but still without testing some rounding modes. So considering we're coming from 0% coverage, I'd say coverage is pretty good!
I agree ;-)
Quoted 8 lines
But yes, we should add a flag to just test -all.
There are some functions that we are not testing yet though, but that can be fixed over time (we can add more tests to fp-test even though they're not in testfloat, such as testing flush-to-zero/ denormals-are-zero, or the muladd variants that we have).
Quoted 7 lines
I have basically no time left to work on this. What do you think about the following plan?
1. Have our own clones (forks) of testfloat/softfloat in qemu servers.
2. Add those as submodules
3. Add fp-test with as you said an -all flag that reports all errors to get decent coverage.Works for me.
Quoted 3 lines
4. Add hardfloat patches, with tests. This requires a small change to testfloat:
https://github.com/cota/berkeley-testfloat-3/commit/ca9fa2ba05
Quoted 5 lines
It seems a shame to have our mirror of testfloat diverge but if it can't be avoided I guess it will have to do. Couldn't we just zero the HW flags in the wrapped up function? I may be misremembering but don't we fall-back to softfloat code anyway if we have reached any limits that trigger exception flags?
testfloat compares the result of the ops as well as the resulting flags. It zeroes the exception flags register before computing the "golden" result/flags. The hardfloat code only kicks in when inexact is set, so we need to add the above patch to testfloat so that we can control the initial flags from outside. With this patch + a patch on the fp-test side to set the initial flags, we can choose whether to test hardfloat or not.
About your second question: yes, for corner cases we defer to softfloat. But to reach those corner cases we first have to reach the hardfloat code, which requires inexact to be set.
Quoted 6 lines
Quoted 3 lines
I'd then leave adding further tests to increase coverage and fixing the existing bugs (prior to hardfloat) to someone else with more time/resources.
Agreed. I should have time to chase down the softfloat regressions once I'm back from Connect after next week.
Nice! I'll try to have something ready later today.
Emilio