Skip to contentSkip to conversation

intel-ci: Document meta.testlist

PATCH In review Petri Latvala 3 participants 8 messages Mar 13, 2018 – Mar 22, 2018

lore
Share & tools
lore

Petri Latvala

author
#0 lore raw

PATCH 1/3 intel-ci: Document meta.testlist

Signed-off-by Petri Latvala <[email protected]>
Cc Marta Lofstedt <[email protected]>
---
 tests/intel-ci/README | 8 ++++++++
 1 file changed, 8 insertions(+)
tests/intel-ci/README +8 -0
index 9c08e243..4d7a97eb 100644
@@ -35,3 +35,11 @@ test per feature.
 
 The string "basic" in a test name means the test probably belongs in
 this list.
+
+=============
+meta.testlist
+=============
+
+Meta-testlist contains tests that are hardcoded to generate various
+different test results. It is used to test CI systems for proper
+result reporting, notification systems, etc.
Signature · 6 lines
2.14.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
author
#1 lore raw

PATCH 2/3 intel-ci: Document purpose of fast-feedback.testlist

Original text was written before shard runs existed. Removed the criteria for testlist contents; The list is still strictly controlled, but additions/removals will be considered case-by-case.

Signed-off-by Petri Latvala <[email protected]>
---
 tests/intel-ci/README | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
tests/intel-ci/README +8 -14
index 4d7a97eb..2ccbbb1a 100644
@@ -21,20 +21,14 @@ ask a maintainer (see MAINTAINERS).
 fast-feedback.testlist
 ======================
 
-Fast-feedback is the testlist that is run for Patchwork and Trybot
-testing. It is also run on various drm-intel-* branches.
-
-The results of CI runs on drm-tip can be found in
-https://intel-gfx-ci.01.org/CI/
-
-Time limit: Full run must complete in 15 minutes on all tested
-platforms.
-
-Feature coverage: Should test all working features, with at least 1
-test per feature.
-
-The string "basic" in a test name means the test probably belongs in
-this list.
+Fast-feedback contains tests that roughly tests a wide coverage of
+features in a short time. If the tests in fast-feedback report no
+regressions, the (almost) full suite of IGT tests is executed for
+proper testing. Results of a fast-feedback test round should only be
+considered to mean that the kernel is not obviously broken.
+
+The results of CI runs on drm-tip and various other kernel trees can
+be found in https://intel-gfx-ci.01.org/CI/
 
 =============
 meta.testlist
Signature · 6 lines
2.14.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Reviewed-by on v1 2/3
#2 lore raw

Re: [igt-dev] [PATCH i-g-t 2/3] intel-ci: Document purpose of fast-feedback.testlist

Reviewed-by Marta Lofstedt <[email protected]>
Quoted 51 lines

-----Original Message-----
From: igt-dev [mailto:[email protected]] On Behalf Of Petri Latvala
Sent: Tuesday, March 13, 2018 12:22 PM
To: [email protected]
Subject: [igt-dev] [PATCH i-g-t 2/3] intel-ci: Document purpose of fast- feedback.testlist

Original text was written before shard runs existed. Removed the criteria for testlist contents; The list is still strictly controlled, but additions/removals will be considered case-by-case.

Signed-off-by Petri Latvala <[email protected]>
---
 tests/intel-ci/README | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
tests/intel-ci/README index renamed from tests/intel-ci/README +0 -0

4d7a97eb..2ccbbb1a 100644

tests/intel-ci/README +0 -1
@@ -21,20 +21,14 @@ ask a maintainer (see MAINTAINERS).
 fast-feedback.testlist
 ======================
 
-Fast-feedback is the testlist that is run for Patchwork and Trybot -testing. It

is also run on various drm-intel-* branches.

tests/intel-ci/README +0 -2
-
-The results of CI runs on drm-tip can be found in -https://intel-gfx-

ci.01.org/CI/

tests/intel-ci/README +0 -4
-
-Time limit: Full run must complete in 15 minutes on all tested -platforms.
-
-Feature coverage: Should test all working features, with at least 1 -test per

feature.

tests/intel-ci/README +8 -3
-
-The string "basic" in a test name means the test probably belongs in -this list.
+Fast-feedback contains tests that roughly tests a wide coverage of
+features in a short time. If the tests in fast-feedback report no
+regressions, the (almost) full suite of IGT tests is executed for
+proper testing. Results of a fast-feedback test round should only be
+considered to mean that the kernel is not obviously broken.
+
+The results of CI runs on drm-tip and various other kernel trees can be
+found in https://intel-gfx-ci.01.org/CI/
 
 =============
 meta.testlist
--

2.14.1

Quoted 4 lines

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

author
#3 lore raw

PATCH 3/3 intel-ci: Add full-suite test round blacklist

In reply to Petri Latvala

The list of regular expressions for tests that are not to be run in sharded full test suite rounds, copied from
https://intel-gfx-ci.01.org/blacklist.txt

Signed-off-by Petri Latvala <[email protected]>
---
 tests/intel-ci/README        |  7 +++++
 tests/intel-ci/blacklist.txt | 75 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)
 create mode 100644 tests/intel-ci/blacklist.txt
tests/intel-ci/README +7 -0
index 2ccbbb1a..e3289933 100644
@@ -30,6 +30,13 @@ considered to mean that the kernel is not obviously broken.
 The results of CI runs on drm-tip and various other kernel trees can
 be found in https://intel-gfx-ci.01.org/CI/
 
+=============
+blacklist.txt
+=============
+
+This file contains regular expressions (one per line) for tests that
+are not to be executed in full suite test rounds.
+
 =============
 meta.testlist
 =============
tests/intel-ci/blacklist.txt new file +75 -0

75 lines not on this page · raw message

Signature · 6 lines
2.14.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Reviewed-by on v1 3/3
#4 lore raw

Re: [igt-dev] [PATCH i-g-t 3/3] intel-ci: Add full-suite test round blacklist

Reviewed-by Marta Lofstedt <[email protected]>
Quoted 120 lines

-----Original Message-----
From: igt-dev [mailto:[email protected]] On Behalf Of Petri Latvala
Sent: Tuesday, March 13, 2018 12:22 PM
To: [email protected]
Subject: [igt-dev] [PATCH i-g-t 3/3] intel-ci: Add full-suite test round blacklist

The list of regular expressions for tests that are not to be run in sharded full test suite rounds, copied from https://intel-gfx-ci.01.org/blacklist.txt

Signed-off-by Petri Latvala <[email protected]>
---
 tests/intel-ci/README        |  7 +++++
 tests/intel-ci/blacklist.txt | 75
++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)
 create mode 100644 tests/intel-ci/blacklist.txt
tests/intel-ci/README index renamed from tests/intel-ci/README +0 -0

2ccbbb1a..e3289933 100644

tests/intel-ci/README +0 -0
@@ -30,6 +30,13 @@ considered to mean that the kernel is not obviously

broken.
The results of CI runs on drm-tip and various other kernel trees can be
found in https://intel-gfx-ci.01.org/CI/

+=============

tests/intel-ci/README +6 -0
+blacklist.txt
+=============
+
+This file contains regular expressions (one per line) for tests that
+are not to be executed in full suite test rounds.
+
 =============
 meta.testlist
 =============
tests/intel-ci/blacklist.txt new file renamed from tests/intel-ci/blacklist.txt +0 -0

mode 100644 index 00000000..7ca313ac

tests/intel-ci/blacklist.txt new file +75 -1
@@ -0,0 +1,75 @@
+igt@meta_test(@.*)?
+###############################################
+# GEM
+###############################################
+igt@drv_selftest(@.*)?
+igt@drm_mm(@.*)?
+igt@gem_busy@.*hang.*
+igt@gem_close_race@(?!.*basic).*
+igt@gem_concurrent_blit(@.*)?
+igt@gem_cs_prefetch(@.*)?
+igt@gem_ctx_create@(?!.*basic).*
+igt@gem_ctx_exec@(?!.*basic).*
+igt@gem_ctx_switch@(?!.*basic).*
+igt@gem_ctx_thrash(@.*)?
+igt@gem_evict_alignment(@.*)?
+igt@gem_evict_everything(@.*)?
+igt@gem_exec_alignment@(?!.*single).*
+igt@gem_exec_fence@(?!.*basic).*
+igt@gem_exec_flush@(?!.*basic).*
+igt@gem_exec_gttfill@(?!.*basic).*
+igt@gem_exec_latency(@.*)?
+igt@gem_exec_lut_handle(@.*)?
+igt@gem_exec_nop@(?!.*(basic|signal-all)).*
+igt@gem_exec_reloc@(?!.*basic).*
+igt@gem_exec_suspend@(?!.*basic).*
+igt@gem_exec_whisper@(?!normal$).*
+igt@gem_fd_exhaustion(@.*)?
+igt@gem_fence_upload(@.*)?
+igt@gem_gtt_hog(@.*)?
+igt@gem_gtt_speed(@.*)?
+igt@gem_hangcheck_forcewake(@.*)?
+igt@gem_lut_handle(@.*)?
+igt@gem_mmap_gtt@.*(huge|swap|clflush).*
+igt@gem_mmap@.*(swap|huge).*
+igt@gem_mocs_settings@.*(suspend|hibernate).*
+igt@gem_pin(@.*)?
+igt@gem_pread_after_blit(@.*)?
+igt@gem_read_read_speed(@.*)?
+igt@gem_reloc_overflow(@.*)?
+igt@gem_reloc_vs_gpu(@.*)?
+igt@gem_render_linear_blits@(?!.*basic).*
+igt@gem_render_tiled_blits@(?!.*basic).*
+igt@gem_reset_stats(@.*)?
+igt@gem_ringfill@(?!.*basic).*
+igt@gem_ring_sync_copy(@.*)?
+igt@gem_ring_sync_loop(@.*)?
+igt@gem_seqno_wrap(@.*)?
+igt@gem_shrink@(?!reclaim$).*
+igt@gem_softpin@.*(hang|S4).*
+igt@gem_spin_batch(@.*)?
+igt@gem_stolen@.*hibernate.*
+igt@gem_storedw_batches_loop(@.*)?
+igt@gem_storedw_loop(@.*)?
+igt@gem_streaming_writes(@.*)?
+igt@gem_sync@(?!.*basic).*
+igt@gem_tiled_swapping@(?!non-threaded).*
+igt@gem_userptr_blits@(major|minor|forked|mlocked|swapping).*
+igt@gem_wait@.*hang.*
+igt@gem_write_read_ring_switch(@.*)?
+###############################################
+# Broadcom
+###############################################
+igt@vc4_.*
+###############################################
+# AMD
+###############################################
+igt@amdgpu/.*
+###############################################
+# Hibernate: not configured correctly on CI
+###############################################
+igt@.*@.*-hibernate
+igt@gem_softpin@noreloc-s4
+igt@gem_workarounds@hibernate-.*
+igt@gem_exec_suspend@basic-s4$
+igt@gem_ctx_isolation@.*-s4
--

2.14.1

Quoted 4 lines

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Reviewed-by on v1 1/3
#5 lore raw
In reply to Petri Latvala
Reviewed-by Marta Lofstedt <[email protected]>
Quoted 30 lines

-----Original Message-----
From: Latvala, Petri
Sent: Tuesday, March 13, 2018 12:22 PM
To: [email protected]
Cc: Latvala, Petri <[email protected]>; Lofstedt, Marta <[email protected]>
Subject: [PATCH i-g-t 1/3] intel-ci: Document meta.testlist

Signed-off-by Petri Latvala <[email protected]>
Cc Marta Lofstedt <[email protected]>
---
 tests/intel-ci/README | 8 ++++++++
 1 file changed, 8 insertions(+)
tests/intel-ci/README index renamed from tests/intel-ci/README +0 -0

9c08e243..4d7a97eb 100644

tests/intel-ci/README +8 -1
@@ -35,3 +35,11 @@ test per feature.
 
 The string "basic" in a test name means the test probably belongs in  this list.
+
+=============
+meta.testlist
+=============
+
+Meta-testlist contains tests that are hardcoded to generate various
+different test results. It is used to test CI systems for proper result
+reporting, notification systems, etc.
--
Quoted 1 line

2.14.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

CI
#6 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,1/3] intel-ci: Document meta.testlist

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/3] intel-ci: Document meta.testlist
URL : https://patchwork.freedesktop.org/series/39845/
State : success

== Summary ==

IGT patchset tested on top of latest successful build 89b915f5aa465d5c3498b170b1572fae20460491 tests/pm_rpm: Don't try to create an X-tiled ARGB8888 framebuffer

with latest DRM-Tip kernel build CI_DRM_3916
b27e0c360d42 drm-tip: 2018y-03m-13d-09h-56m-18s UTC integration manifest

No testlist changes.

fi-bdw-5557u     total:288  pass:267  dwarn:0   dfail:0   fail:0   skip:21  time:429s
fi-bdw-gvtdvm    total:288  pass:264  dwarn:0   dfail:0   fail:0   skip:24  time:438s
fi-blb-e6850     total:288  pass:223  dwarn:1   dfail:0   fail:0   skip:64  time:384s
fi-bsw-n3050     total:288  pass:242  dwarn:0   dfail:0   fail:0   skip:46  time:534s
fi-bwr-2160      total:288  pass:183  dwarn:0   dfail:0   fail:0   skip:105 time:298s
fi-bxt-dsi       total:288  pass:258  dwarn:0   dfail:0   fail:0   skip:30  time:512s
fi-bxt-j4205     total:288  pass:259  dwarn:0   dfail:0   fail:0   skip:29  time:507s
fi-byt-j1900     total:288  pass:253  dwarn:0   dfail:0   fail:0   skip:35  time:509s
fi-byt-n2820     total:288  pass:249  dwarn:0   dfail:0   fail:0   skip:39  time:512s
fi-cfl-8700k     total:288  pass:260  dwarn:0   dfail:0   fail:0   skip:28  time:414s
fi-cfl-s2        total:288  pass:262  dwarn:0   dfail:0   fail:0   skip:26  time:588s
fi-elk-e7500     total:288  pass:229  dwarn:0   dfail:0   fail:0   skip:59  time:431s
fi-gdg-551       total:288  pass:179  dwarn:0   dfail:0   fail:1   skip:108 time:315s
fi-glk-1         total:288  pass:260  dwarn:0   dfail:0   fail:0   skip:28  time:536s
fi-hsw-4770      total:288  pass:261  dwarn:0   dfail:0   fail:0   skip:27  time:404s
fi-ilk-650       total:288  pass:228  dwarn:0   dfail:0   fail:0   skip:60  time:420s
fi-ivb-3520m     total:288  pass:259  dwarn:0   dfail:0   fail:0   skip:29  time:468s
fi-ivb-3770      total:288  pass:255  dwarn:0   dfail:0   fail:0   skip:33  time:435s
fi-kbl-7500u     total:288  pass:263  dwarn:1   dfail:0   fail:0   skip:24  time:477s
fi-kbl-7567u     total:288  pass:268  dwarn:0   dfail:0   fail:0   skip:20  time:470s
fi-kbl-r         total:288  pass:261  dwarn:0   dfail:0   fail:0   skip:27  time:518s
fi-pnv-d510      total:288  pass:222  dwarn:1   dfail:0   fail:0   skip:65  time:642s
fi-skl-6260u     total:288  pass:268  dwarn:0   dfail:0   fail:0   skip:20  time:439s
fi-skl-6600u     total:288  pass:261  dwarn:0   dfail:0   fail:0   skip:27  time:532s
fi-skl-6700hq    total:288  pass:262  dwarn:0   dfail:0   fail:0   skip:26  time:540s
fi-skl-6700k2    total:288  pass:264  dwarn:0   dfail:0   fail:0   skip:24  time:502s
fi-skl-6770hq    total:288  pass:268  dwarn:0   dfail:0   fail:0   skip:20  time:490s
fi-skl-guc       total:288  pass:260  dwarn:0   dfail:0   fail:0   skip:28  time:426s
fi-skl-gvtdvm    total:288  pass:265  dwarn:0   dfail:0   fail:0   skip:23  time:436s
fi-snb-2520m     total:288  pass:248  dwarn:0   dfail:0   fail:0   skip:40  time:541s
fi-snb-2600      total:288  pass:248  dwarn:0   dfail:0   fail:0   skip:40  time:406s

Blacklisted hosts:
fi-cnl-drrs total:285 pass:254 dwarn:0 dfail:1 fail:2 skip:27

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1115/issues.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

CI
#7 lore raw

[igt-dev] ✗ Fi.CI.IGT: warning for series starting with [i-g-t,1/3] intel-ci: Document meta.testlist

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/3] intel-ci: Document meta.testlist
URL : https://patchwork.freedesktop.org/series/39845/
State : warning

== Summary ==

---- Possible new issues:

Test kms_cursor_crc:

        Subgroup cursor-128x128-suspend:
                pass       -> SKIP       (shard-snb)

---- Known issues:

Test drv_suspend:

        Subgroup debugfs-reader:
                pass       -> SKIP       (shard-snb) fdo#102365
Test gem_eio:
        Subgroup in-flight-external:
                pass       -> INCOMPLETE (shard-apl) fdo#105341
Test kms_flip:
        Subgroup 2x-plain-flip-fb-recreate-interruptible:
                pass       -> FAIL       (shard-hsw) fdo#100368
Test kms_plane:
        Subgroup plane-panning-bottom-right-suspend-pipe-a-planes:
                pass       -> FAIL       (shard-apl) fdo#103375 +1
Test kms_rotation_crc:
        Subgroup primary-rotation-180:
                pass       -> FAIL       (shard-snb) fdo#103925
Test kms_setmode:
        Subgroup basic:
                pass       -> FAIL       (shard-apl) fdo#99912

fdo#102365 https://bugs.freedesktop.org/show_bug.cgi?id=102365
fdo#105341 https://bugs.freedesktop.org/show_bug.cgi?id=105341
fdo#100368 https://bugs.freedesktop.org/show_bug.cgi?id=100368
fdo#103375 https://bugs.freedesktop.org/show_bug.cgi?id=103375
fdo#103925 https://bugs.freedesktop.org/show_bug.cgi?id=103925
fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912

shard-apl        total:3424 pass:1806 dwarn:1   dfail:0   fail:8   skip:1607 time:12764s
shard-hsw        total:3473 pass:1776 dwarn:1   dfail:0   fail:3   skip:1692 time:11969s
shard-snb        total:3473 pass:1362 dwarn:1   dfail:0   fail:4   skip:2106 time:7137s

Blacklisted hosts:
shard-kbl total:3442 pass:1917 dwarn:18 dfail:0 fail:10 skip:1496 time:9658s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1115/shards.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev