Skip to contentSkip to conversation

CI: Make pytest export JUnitXML

PATCH In review Tom Rini 2 participants 3 messages Jun 27, 2024 – Jul 4, 2024

lore
Share & tools
lore

Tom Rini

author
#0 lore raw

PATCH 1/1 CI: Make pytest export JUnitXML

Both GitLab and Azure (and other CI systems) have native support for displaying JUnitXML test report results. The pytest framework that we use can generate these reports. Change our CI tests so that they will generate these reports and then have the respective CI platform pick them up. We write to different locations because of where each CI is (and isn't) able to easily pass things along.

---
 .azure-pipelines.yml | 12 +++++++++++-
 .gitlab-ci.yml       |  5 ++++-
 2 files changed, 15 insertions(+), 2 deletions(-)
.azure-pipelines.yml +11 -1
index c43bb51066a5..fff78785ee92 100644
@@ -287,7 +287,7 @@ stages:
           export PATH=/opt/qemu/bin:/tmp/uboot-test-hooks/bin:\${PATH}
           export PYTHONPATH=/tmp/uboot-test-hooks/py/travis-ci
           # "\${var:+"-k \$var"}" expands to "" if \$var is empty, "-k \$var" if not
-          ./test/py/test.py -ra -o cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} \${TEST_PY_ID} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR"
+          ./test/py/test.py -ra -o cache_dir="\$UBOOT_TRAVIS_BUILD_DIR"/.pytest_cache --bd \${TEST_PY_BD} \${TEST_PY_ID} \${TEST_PY_TEST_SPEC:+"-k \${TEST_PY_TEST_SPEC}"} --build-dir "\$UBOOT_TRAVIS_BUILD_DIR" --report-dir "\$UBOOT_TRAVIS_BUILD_DIR" --junitxml=\$(System.DefaultWorkingDirectory)/results.xml
           # the below corresponds to .gitlab-ci.yml "after_script"
           rm -rf /tmp/uboot-test-hooks /tmp/venv
           EOF
@@ -371,6 +371,7 @@ stages:
           docker run "$@" --device /dev/fuse:/dev/fuse \
                          -v $PWD:$(work_dir) \
                          -v $(Pipeline.Workspace):$(Pipeline.Workspace) \
+                         -v $(System.DefaultWorkingDirectory):$(System.DefaultWorkingDirectory) \
                          -e WORK_DIR="${WORK_DIR}" \
                          -e TEST_PY_BD="${TEST_PY_BD}" \
                          -e TEST_PY_ID="${TEST_PY_ID}" \
@@ -378,6 +379,10 @@ stages:
                          -e OVERRIDE="${OVERRIDE}" \
                          -e BUILD_ENV="${BUILD_ENV}" $(ci_runner_image) \
                          $(Pipeline.Workspace)/testsh/test.sh
+      - task: PublishTestResults@2
+        inputs:
+          testResultsFormat: 'JUnit'
+          testResultsFiles: 'results.xml'
 
 - stage: test_py_qemu
   jobs:
@@ -495,6 +500,7 @@ stages:
           docker run "$@" --device /dev/fuse:/dev/fuse \
                          -v $PWD:$(work_dir) \
                          -v $(Pipeline.Workspace):$(Pipeline.Workspace) \
+                         -v $(System.DefaultWorkingDirectory):$(System.DefaultWorkingDirectory) \
                          -e WORK_DIR="${WORK_DIR}" \
                          -e TEST_PY_BD="${TEST_PY_BD}" \
                          -e TEST_PY_ID="${TEST_PY_ID}" \
@@ -503,6 +509,10 @@ stages:
                          -e BUILD_ENV="${BUILD_ENV}" $(ci_runner_image) \
                          $(Pipeline.Workspace)/testsh/test.sh
         retryCountOnTaskFailure: 2 # QEMU may be too slow, etc.
+      - task: PublishTestResults@2
+        inputs:
+          testResultsFormat: 'JUnit'
+          testResultsFiles: 'results.xml'
 
 - stage: world_build
   jobs:
.gitlab-ci.yml +4 -1
index 165f765a8332..a6c5040239ae 100644
@@ -41,7 +41,7 @@ stages:
       fi
 
   after_script:
-    - cp -v /tmp/${TEST_PY_BD}/*.{html,css} .
+    - cp -v /tmp/${TEST_PY_BD}/*.{html,css,xml} .
     - rm -rf /tmp/uboot-test-hooks /tmp/venv
   script:
     # If we've been asked to use clang only do one configuration.
@@ -83,11 +83,14 @@ stages:
       ./test/py/test.py -ra --bd ${TEST_PY_BD} ${TEST_PY_ID}
         ${TEST_PY_TEST_SPEC:+"-k ${TEST_PY_TEST_SPEC}"}
         --build-dir "$UBOOT_TRAVIS_BUILD_DIR"
+        --junitxml=/tmp/${TEST_PY_BD}/results.xml
   artifacts:
     when: always
     paths:
       - "*.html"
       - "*.css"
+    reports:
+      junit: results.xml
     expire_in: 1 week
 
 .world_build:
Signature · 1 line
2.34.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Tom Rini

Both GitLab and Azure (and other CI systems) have native support for displaying JUnitXML test report results. The pytest framework that we use can generate these reports. Change our CI tests so that they will generate these reports and then have the respective CI platform pick them up. We write to different locations because of where each CI is (and isn't) able to easily pass things along.

Quoted 6 lines
---
 .azure-pipelines.yml | 12 +++++++++++-
 .gitlab-ci.yml       |  5 ++++-
 2 files changed, 15 insertions(+), 2 deletions(-)

Nice!

author
#2 lore raw
In reply to Tom Rini
Quoted 8 lines · Tom Rini

Both GitLab and Azure (and other CI systems) have native support for displaying JUnitXML test report results. The pytest framework that we use can generate these reports. Change our CI tests so that they will generate these reports and then have the respective CI platform pick them up. We write to different locations because of where each CI is (and isn't) able to easily pass things along.

[...]

Applied to u-boot/master, thanks!

Signature · 1 line
Tom