Skip to contentSkip to conversation

scripts/verify-blacklist: Script for checking blacklist files

PATCH New Petri Latvala 3 participants 5 messages Jun 15, 2020 – Jun 16, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/2 scripts/verify-blacklist: Script for checking blacklist files

tests/intel-ci/blacklist*.txt files can collect bitrot unless there's an easy way to check for lines that are no longer needed due to the tests being renamed or removed. Therefore, a script just for that.

v2: Use long options for readability, exit with 1 if something found

v3: Verify manually against the list of all tests instead of trying

    with igt_runner's --include-tests
---
 scripts/verify-blacklist.sh | 54 +++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100755 scripts/verify-blacklist.sh
scripts/verify-blacklist.sh new file +54 -0
new file mode 100755
index 00000000..93dca495
@@ -0,0 +1,54 @@
+#!/bin/sh
+
+# Verify that all entries in a blacklist file are still valid
+
+usage() {
+    echo "Usage: $0 <path-to-igt-runner> <test-binary-directory> <blacklist-file>"
+    echo
+    echo " path-to-igt-runner: For example build/runner/igt_runner"
+    echo " test-binary-directory: For example build/tests"
+    echo " blacklist-file: For example tests/intel-ci/blacklist.txt"
+    exit 2
+}
+
+if [ $# -ne 3 ]; then
+    usage
+fi
+
+RUNNER="$1"
+BINDIR="$2"
+BLFILE="$3"
+
+if [ ! -x "$RUNNER" ]; then
+    echo "$RUNNER not found"
+    echo
+    usage
+fi
+
+if [ ! -f "$BINDIR/test-list.txt" ]; then
+    echo "$BINDIR doesn't look like a test-binary directory"
+    echo
+    usage
+fi
+
+if [ ! -f "$BLFILE" ]; then
+    echo "$BLFILE not found"
+    echo
+    usage
+fi
+
+STATUS=0
+
+TESTLIST="$("$RUNNER" --list-all "$BINDIR")"
+
+cat "$BLFILE" | while read line; do
+    blentry=$(echo "$line" | sed 's/#.*//' | tr -d '[:space:]')
+    if [ "$blentry" = "" ]; then continue; fi
+
+    if ! (echo "$TESTLIST" | grep -Pq "$blentry") >/dev/null 2>/dev/null; then
+	echo Useless blacklist entry: "$blentry"
+	STATUS=1
+    fi
+done
+
+exit $STATUS
Signature · 6 lines
2.20.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw

PATCH 2/2 gitlab-ci: Verify blacklist files with verify-blacklist.sh

Cc Arkadiusz Hiler <[email protected]>

---

Now finished in 47 seconds in gitlab.

 .gitlab-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)
.gitlab-ci.yml +6 -0
index c5a6bd9e..d7fdbfde 100644
@@ -252,6 +252,12 @@ test:list-undocumented-tests:
     paths:
       - undocumented_tests.txt
 
+test:verify-blacklists:
+  dependencies:
+    - build:tests-fedora
+  stage: test
+  script: for bl in tests/intel-ci/blacklist{,-pre-merge}.txt; do scripts/verify-blacklist.sh build/runner/igt_runner build/tests "$bl" || exit 1; done
+
 ################### DEPLOY #########################
 
 pages:
Signature · 6 lines
2.20.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Reviewed-by on v1 2/2
#2 lore raw

Re: [igt-dev] [PATCH i-g-t 2/2] gitlab-ci: Verify blacklist files with verify-blacklist.sh

Quoted 29 lines · Petri Latvala
Cc Arkadiusz Hiler <[email protected]>

---

Now finished in 47 seconds in gitlab.

 .gitlab-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)
.gitlab-ci.yml +6 -0
index c5a6bd9e..d7fdbfde 100644
@@ -252,6 +252,12 @@ test:list-undocumented-tests:
     paths:
       - undocumented_tests.txt
 
+test:verify-blacklists:
+  dependencies:
+    - build:tests-fedora
+  stage: test
+  script: for bl in tests/intel-ci/blacklist{,-pre-merge}.txt; do scripts/verify-blacklist.sh build/runner/igt_runner build/tests "$bl" || exit 1; done
+
 ################### DEPLOY #########################
 
 pages:
Signature · 1 line
2.20.1

For both patches,

We still should do something about the assume implicit ^$ that are not there.
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#3 lore raw

[igt-dev] ✗ Fi.CI.BAT: failure for series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files
URL : https://patchwork.freedesktop.org/series/78368/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_8626 -> IGTPW_4675
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with IGTPW_4675 absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in IGTPW_4675, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/index.html

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in IGTPW_4675:

### IGT changes ###

#### Possible regressions ####

* igt@gem_exec_parallel@engines@fds:
- fi-tgl-u2: [PASS][1] -> [INCOMPLETE][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-tgl-u2/igt@gem_exec_parallel@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-tgl-u2/igt@gem_exec_parallel@[email protected]

Known issues
------------

Here are the changes found in IGTPW_4675 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_exec_suspend@basic-s0:
- fi-apl-guc: [PASS][3] -> [INCOMPLETE][4] ([i915#1242])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-apl-guc/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-apl-guc/igt@[email protected]

* igt@i915_module_load@reload:
- fi-byt-j1900: [PASS][5] -> [DMESG-WARN][6] ([i915#1982])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-byt-j1900/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-byt-j1900/igt@[email protected]
- fi-byt-n2820: [PASS][7] -> [DMESG-WARN][8] ([i915#1982])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-byt-n2820/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-byt-n2820/igt@[email protected]

* igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic:
- fi-bsw-kefka: [PASS][9] -> [DMESG-WARN][10] ([i915#1982]) +1 similar issue
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-bsw-kefka/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-bsw-kefka/igt@[email protected]

* igt@kms_flip@basic-flip-vs-wf_vblank@c-edp1:
- fi-icl-u2: [PASS][11] -> [DMESG-WARN][12] ([i915#1982]) +1 similar issue
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-icl-u2/igt@kms_flip@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-icl-u2/igt@kms_flip@[email protected]

#### Possible fixes ####

* igt@kms_cursor_legacy@basic-flip-before-cursor-atomic:
- fi-icl-guc: [DMESG-WARN][13] ([i915#1982]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-icl-guc/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-icl-guc/igt@[email protected]

* igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1:
- fi-icl-u2: [DMESG-WARN][15] ([i915#1982]) -> [PASS][16] +2 similar issues
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-icl-u2/igt@kms_flip@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-icl-u2/igt@kms_flip@[email protected]

* igt@kms_flip@basic-plain-flip@d-dsi1:
- {fi-tgl-dsi}: [DMESG-WARN][17] ([i915#1982]) -> [PASS][18]
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-tgl-dsi/igt@kms_flip@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-tgl-dsi/igt@kms_flip@[email protected]

#### Warnings ####

* igt@gem_exec_suspend@basic-s3:
- fi-kbl-x1275: [DMESG-WARN][19] ([i915#62] / [i915#92] / [i915#95]) -> [DMESG-WARN][20] ([i915#1982] / [i915#62] / [i915#92] / [i915#95])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-kbl-x1275/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-kbl-x1275/igt@[email protected]

* igt@kms_flip@basic-flip-vs-modeset@a-dp1:
- fi-kbl-x1275: [DMESG-WARN][21] ([i915#62] / [i915#92]) -> [DMESG-WARN][22] ([i915#62] / [i915#92] / [i915#95]) +5 similar issues
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-kbl-x1275/igt@kms_flip@[email protected]
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-kbl-x1275/igt@kms_flip@[email protected]

* igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
- fi-kbl-x1275: [DMESG-WARN][23] ([i915#62] / [i915#92] / [i915#95]) -> [DMESG-WARN][24] ([i915#62] / [i915#92]) +4 similar issues
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-kbl-x1275/igt@[email protected]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-kbl-x1275/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[i915#1242]: https://gitlab.freedesktop.org/drm/intel/issues/1242
[i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
[i915#62]: https://gitlab.freedesktop.org/drm/intel/issues/62
[i915#92]: https://gitlab.freedesktop.org/drm/intel/issues/92
[i915#95]: https://gitlab.freedesktop.org/drm/intel/issues/95

Participating hosts (44 -> 41)
------------------------------

Additional (3): fi-whl-u fi-bdw-5557u fi-skl-6600u
Missing (6): fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-byt-clapper fi-bdw-samus

Build changes
-------------

* CI: CI-20190529 -> None
* IGT: IGT_5710 -> IGTPW_4675

CI-20190529: 20190529
CI_DRM_8626: 563d95505a36489373782ca98d67897d4b338f94 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_4675: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/index.html
IGT_5710: f524eee47930601ad7b4cba9d40c26d68dc7d250 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/index.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#4 lore raw

[igt-dev] ✗ GitLab.Pipeline: warning for series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files
URL : https://patchwork.freedesktop.org/series/78368/
State : warning

== Summary ==

Did not get list of undocumented tests for this run, something is wrong!

Other than that, pipeline status: FAILED.

see https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/pipelines/160705 for the overview.

build-containers:build-debian-arm64 has failed (https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/-/jobs/3098121):
Using docker image sha256:594aa868d31ee3304dee8cae8a3433c89a6fcfcf6c7d420c04cce22f60147176 for registry.freedesktop.org/wayland/ci-templates/buildah:2019-08-13.0 ... section_end:1592218945:prepare_executor
section_start:1592218945:prepare_script
Preparing environment
Running on runner-j4Lrg1oF-project-3185-concurrent-1 via gst-htz-1...
section_end:1592218947:prepare_script
section_start:1592218947:get_sources
Getting source from Git repository
Fetching changes...
Initialized empty Git repository in /builds/gfx-ci/igt-ci-tags/.git/
Created fresh repository.
warning: redirecting to https://gitlab-ci-hetzner.freedesktop.org/gfx-ci/igt-ci-tags.git/
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
fatal: the remote end hung up unexpectedly
section_end:1592219009:get_sources
section_start:1592219009:upload_artifacts_on_failure
Uploading artifacts for failed job
section_end:1592219011:upload_artifacts_on_failure
ERROR: Job failed: exit code 1

build-containers:build-debian-armhf has failed (https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/-/jobs/3098120):
Using docker image sha256:594aa868d31ee3304dee8cae8a3433c89a6fcfcf6c7d420c04cce22f60147176 for registry.freedesktop.org/wayland/ci-templates/buildah:2019-08-13.0 ... section_end:1592218945:prepare_executor
section_start:1592218945:prepare_script
Preparing environment
Running on runner-z2CifdYy-project-3185-concurrent-0 via gst-htz-3...
section_end:1592218946:prepare_script
section_start:1592218946:get_sources
Getting source from Git repository
Fetching changes...
Initialized empty Git repository in /builds/gfx-ci/igt-ci-tags/.git/
Created fresh repository.
warning: redirecting to https://gitlab-ci-hetzner.freedesktop.org/gfx-ci/igt-ci-tags.git/
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
fatal: the remote end hung up unexpectedly
section_end:1592219009:get_sources
section_start:1592219009:upload_artifacts_on_failure
Uploading artifacts for failed job
section_end:1592219010:upload_artifacts_on_failure
ERROR: Job failed: exit code 1

== Logs ==

For more details see: https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/pipelines/160705
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev