Skip to contentSkip to conversation

tests: Remove tools_test

PATCH In review Petri Latvala 5 participants 7 messages Sep 12, 2018 – Sep 14, 2018 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests: Remove tools_test

The test is basically just testing if the tools work properly with little relevance to testing if the kernel works properly. If the purpose was to test the kernel (or hardware), actual tests will be better suited for the purpose. If the purpose was to test if the tools work, sanity checks somewhere else for all tools instead of just two is better suited for the purpose.

In a nutshell, tools_test as such is fairly useless and has thus far only revealed problems in testing setups (incorrect paths etc) instead of problems in kernel or hardware.

Cc Daniel Vetter <[email protected]>
Cc Arkadiusz Hiler <[email protected]>
---
 tests/Makefile.sources |   1 -
 tests/meson.build      |   1 -
 tests/tools_test.c     | 143 -----------------------------------------
 3 files changed, 145 deletions(-)
 delete mode 100644 tests/tools_test.c
tests/Makefile.sources +0 -1
index c84933f1..59a9d51f 100644
@@ -227,7 +227,6 @@ TESTS_progs = \
 	syncobj_basic \
 	syncobj_wait \
 	template \
-	tools_test \
 	vgem_basic \
 	vgem_slow \
 	$(NULL)
tests/meson.build +0 -1
index 17deb945..f31dc29b 100644
@@ -201,7 +201,6 @@ test_progs = [
 	'syncobj_basic',
 	'syncobj_wait',
 	'template',
-	'tools_test',
 	'vc4_create_bo',
 	'vc4_dmabuf_poll',
 	'vc4_label_bo',
tests/tools_test.c deleted +0 -143

143 lines not on this page · raw message

Signature · 6 lines
2.18.0

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Acked-by on v1 1/1
#1 lore raw
Quoted 7 lines · Petri Latvala

The test is basically just testing if the tools work properly with little relevance to testing if the kernel works properly. If the purpose was to test the kernel (or hardware), actual tests will be better suited for the purpose. If the purpose was to test if the tools work, sanity checks somewhere else for all tools instead of just two is better suited for the purpose.

Quoted 7 lines

In a nutshell, tools_test as such is fairly useless and has thus far only revealed problems in testing setups (incorrect paths etc) instead of problems in kernel or hardware.

Cc Daniel Vetter <[email protected]>
Cc Arkadiusz Hiler <[email protected]>

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#2 lore raw
Quoted 8 lines · Arkadiusz Hiler
Quoted 7 lines · Petri Latvala

The test is basically just testing if the tools work properly with little relevance to testing if the kernel works properly. If the purpose was to test the kernel (or hardware), actual tests will be better suited for the purpose. If the purpose was to test if the tools work, sanity checks somewhere else for all tools instead of just two is better suited for the purpose.

Quoted 8 lines
Quoted 7 lines

In a nutshell, tools_test as such is fairly useless and has thus far only revealed problems in testing setups (incorrect paths etc) instead of problems in kernel or hardware.

Cc Daniel Vetter <[email protected]>
Cc Arkadiusz Hiler <[email protected]>

Hm, not terribly happy about this. We have blown up the tools in trivial ways before, that's why I wrote these. And yes it's incomplete coverage. But we're also not going around deleting other igt tests, just because they're not yet fully covering a given area.

So maybe add a FIXME here instead that there's work to do? -Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#3 lore raw
Quoted 16 lines · Daniel Vetter
Quoted 15 lines · Arkadiusz Hiler
Quoted 14 lines · Petri Latvala

The test is basically just testing if the tools work properly with little relevance to testing if the kernel works properly. If the purpose was to test the kernel (or hardware), actual tests will be better suited for the purpose. If the purpose was to test if the tools work, sanity checks somewhere else for all tools instead of just two is better suited for the purpose.

In a nutshell, tools_test as such is fairly useless and has thus far only revealed problems in testing setups (incorrect paths etc) instead of problems in kernel or hardware.

Cc Daniel Vetter <[email protected]>
Cc Arkadiusz Hiler <[email protected]>
Quoted 6 lines
Quoted 1 line

Hm, not terribly happy about this. We have blown up the tools in trivial ways before, that's why I wrote these. And yes it's incomplete coverage. But we're also not going around deleting other igt tests, just because they're not yet fully covering a given area.

I think there's value in ensuring we don't break the tools from an ABI perspective. But at the same time I don't think we need to test the tools for every patch posted on the list. I.e. tool testing should be separated from ABI testing.

So here's an idea. Extend the tools with a test subcommand or --test parameter, and have the tools themselves check that the ABI they need works. Centralize that information in the tools.

For example, 'intel_reg dump' is a ridiculously slow and potentially dangerous way to test if intel_reg works. I've overlooked that in tools_test; I would never have added something like that in intel_reg.

BR,
Jani.

Signature · 5 lines
Jani Nikula, Intel Open Source Graphics Center
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#4 lore raw
Quoted 29 lines · Jani Nikula
Quoted 16 lines · Daniel Vetter
Quoted 15 lines · Arkadiusz Hiler
Quoted 14 lines · Petri Latvala

The test is basically just testing if the tools work properly with little relevance to testing if the kernel works properly. If the purpose was to test the kernel (or hardware), actual tests will be better suited for the purpose. If the purpose was to test if the tools work, sanity checks somewhere else for all tools instead of just two is better suited for the purpose.

In a nutshell, tools_test as such is fairly useless and has thus far only revealed problems in testing setups (incorrect paths etc) instead of problems in kernel or hardware.

Cc Daniel Vetter <[email protected]>
Cc Arkadiusz Hiler <[email protected]>
Quoted 6 lines
Quoted 1 line

Hm, not terribly happy about this. We have blown up the tools in trivial ways before, that's why I wrote these. And yes it's incomplete coverage. But we're also not going around deleting other igt tests, just because they're not yet fully covering a given area.

I think there's value in ensuring we don't break the tools from an ABI perspective. But at the same time I don't think we need to test the tools for every patch posted on the list. I.e. tool testing should be separated from ABI testing.

Quoted 7 lines

So here's an idea. Extend the tools with a test subcommand or --test parameter, and have the tools themselves check that the ABI they need works. Centralize that information in the tools.

For example, 'intel_reg dump' is a ridiculously slow and potentially dangerous way to test if intel_reg works. I've overlooked that in tools_test; I would never have added something like that in intel_reg.

It's not so much the kernel abi, but libdrm that we've blown up. Or is that what you mean?

But yeah, a more systematic approach to testing this stuff could be good. A also honestly wonder how much use the reg dumpers still have, since we never managed to upstream Damien's gen9+ reg dump tooling afaik :-/ -Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#5 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,1/1] tests: Remove tools_test

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/1] tests: Remove tools_test
URL : https://patchwork.freedesktop.org/series/49527/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4807 -> IGTPW_1826 =

== Summary - SUCCESS ==

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/49527/revisions/1/mbox/

== Known issues ==

Here are the changes found in IGTPW_1826 that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@gem_exec_suspend@basic-s3:
      fi-kbl-soraka:      NOTRUN -> INCOMPLETE (fdo#107556, fdo#107774)

    
    ==== Possible fixes ====

    igt@gem_exec_suspend@basic-s3:
      fi-blb-e6850:       INCOMPLETE (fdo#107718) -> PASS

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
      fi-cfl-8109u:       INCOMPLETE (fdo#106070) -> PASS

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b:
      fi-byt-clapper:     FAIL (fdo#103191, fdo#107362) -> PASS

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c:
      fi-bxt-dsi:         INCOMPLETE (fdo#103927) -> PASS

fdo#103191 https://bugs.freedesktop.org/show_bug.cgi?id=103191
fdo#103927 https://bugs.freedesktop.org/show_bug.cgi?id=103927
fdo#106070 https://bugs.freedesktop.org/show_bug.cgi?id=106070
fdo#107362 https://bugs.freedesktop.org/show_bug.cgi?id=107362
fdo#107556 https://bugs.freedesktop.org/show_bug.cgi?id=107556
fdo#107718 https://bugs.freedesktop.org/show_bug.cgi?id=107718
fdo#107774 https://bugs.freedesktop.org/show_bug.cgi?id=107774

== Participating hosts (46 -> 42) ==

Additional (1): fi-kbl-soraka
Missing (5): fi-ctg-p8600 fi-ilk-m540 fi-byt-squawks fi-bsw-cyan fi-hsw-4200u

== Build changes ==

* IGT: IGT_4639 -> IGTPW_1826

CI_DRM_4807: 55b148b84b254f61adbfeb89c4f6674664f01c46 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1826: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1826/
IGT_4639: c7fa2ea9fbce87206474748100b825558eebe08e @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Testlist changes ==

-igt@tools_test@sysfs_l3_parity
-igt@tools_test@tools_test

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1826/issues.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#6 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for series starting with [i-g-t,1/1] tests: Remove tools_test

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/1] tests: Remove tools_test
URL : https://patchwork.freedesktop.org/series/49527/
State : success

== Summary ==

= CI Bug Log - changes from IGT_4639_full -> IGTPW_1826_full =

== Summary - WARNING ==

Minor unknown changes coming with IGTPW_1826_full need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in IGTPW_1826_full, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://patchwork.freedesktop.org/api/1.0/series/49527/revisions/1/mbox/

== Possible new issues ==

Here are the unknown changes that may have been introduced in IGTPW_1826_full:

=== IGT changes ===

    ==== Warnings ====

    igt@kms_vblank@pipe-b-wait-idle-hang:
      shard-snb:          PASS -> SKIP +3

== Known issues ==

Here are the changes found in IGTPW_1826_full that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@gem_exec_await@wide-contexts:
      shard-apl:          PASS -> FAIL (fdo#106680)

    igt@kms_available_modes_crc@available_mode_test_crc:
      shard-snb:          PASS -> FAIL (fdo#106641)

    igt@kms_cursor_legacy@cursor-vs-flip-varying-size:
      shard-hsw:          PASS -> FAIL (fdo#103355)

    igt@kms_plane@plane-panning-bottom-right-suspend-pipe-c-planes:
      shard-kbl:          PASS -> INCOMPLETE (fdo#103665)

    igt@kms_setmode@basic:
      shard-glk:          PASS -> FAIL (fdo#99912)

    igt@testdisplay:
      shard-glk:          PASS -> INCOMPLETE (k.org#198133, fdo#107093, fdo#103359)

    
    ==== Possible fixes ====

    igt@kms_cursor_legacy@2x-long-cursor-vs-flip-atomic:
      shard-hsw:          FAIL (fdo#105767) -> PASS

    igt@kms_plane_multiple@atomic-pipe-a-tiling-x:
      shard-snb:          FAIL (fdo#103166) -> PASS

    igt@kms_rotation_crc@sprite-rotation-180:
      shard-snb:          FAIL (fdo#103925) -> PASS

    igt@kms_setmode@basic:
      shard-apl:          FAIL (fdo#99912) -> PASS
      shard-kbl:          FAIL (fdo#99912) -> PASS

    igt@perf_pmu@all-busy-check-all:
      shard-snb:          INCOMPLETE (fdo#105411) -> PASS

fdo#103166 https://bugs.freedesktop.org/show_bug.cgi?id=103166
fdo#103355 https://bugs.freedesktop.org/show_bug.cgi?id=103355
fdo#103359 https://bugs.freedesktop.org/show_bug.cgi?id=103359
fdo#103665 https://bugs.freedesktop.org/show_bug.cgi?id=103665
fdo#103925 https://bugs.freedesktop.org/show_bug.cgi?id=103925
fdo#105411 https://bugs.freedesktop.org/show_bug.cgi?id=105411
fdo#105767 https://bugs.freedesktop.org/show_bug.cgi?id=105767
fdo#106641 https://bugs.freedesktop.org/show_bug.cgi?id=106641
fdo#106680 https://bugs.freedesktop.org/show_bug.cgi?id=106680
fdo#107093 https://bugs.freedesktop.org/show_bug.cgi?id=107093
fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912
k.org#198133 https://bugzilla.kernel.org/show_bug.cgi?id=198133

== Participating hosts (5 -> 5) ==

No changes in participating hosts

== Build changes ==

* IGT: IGT_4639 -> IGTPW_1826
* Linux: CI_DRM_4806 -> CI_DRM_4807

CI_DRM_4806: feeccde66999c5e87be3550f2159e5d7eeb61c67 @ git://anongit.freedesktop.org/gfx-ci/linux
CI_DRM_4807: 55b148b84b254f61adbfeb89c4f6674664f01c46 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1826: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1826/
IGT_4639: c7fa2ea9fbce87206474748100b825558eebe08e @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1826/shards.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev