Skip to contentSkip to conversation

ci: azure: Update to use stages

PATCH In review Tom Rini 4 participants 13 messages Jan 12, 2022 – Jan 13, 2022

lore
Share & tools
lore

Tom Rini

author
#0 lore raw

PATCH 1/4 ci: azure: Update to use stages

Follow what we do in GitLab CI where we break the jobs up in to stages such that if earlier and often quicker sanity tests fail we don't run everything else.

---
 .azure-pipelines.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
.azure-pipelines.yml +7 -1
index 670bbc0e1636..d97115668167 100644
@@ -9,7 +9,9 @@ variables:
   container_option: -u 0
   work_dir: /u
 
-jobs:
+stages:
+- stage: testsuites
+  jobs:
   - job: tools_only_windows
     displayName: 'Ensure host tools build for Windows'
     pool:
@@ -199,6 +201,8 @@ jobs:
           export PATH=/opt/gcc-11.1.0-nolibc/arm-linux-gnueabi/bin:$PATH
           test/nokia_rx51_test.sh
 
+- stage: test_py
+  jobs:
   - job: test_py
     displayName: 'test.py'
     pool:
@@ -381,6 +385,8 @@ jobs:
           # Some tests using libguestfs-tools need the fuse device to run
           docker run "$@" --device /dev/fuse:/dev/fuse -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh
 
+- stage: world_build
+  jobs:
   - job: build_the_world
     displayName: 'Build the World'
     pool:
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 2/4 CI, Dockerfile: Update to latest "focal" tag

Bring us to the focial-20220105 tag and rebuild our images on top of this.

---
 .azure-pipelines.yml    | 2 +-
 .gitlab-ci.yml          | 2 +-
 tools/docker/Dockerfile | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
.azure-pipelines.yml +1 -1
index d97115668167..c0f72a811363 100644
@@ -2,7 +2,7 @@ variables:
   windows_vm: windows-2019
   ubuntu_vm: ubuntu-18.04
   macos_vm: macOS-10.15
-  ci_runner_image: trini/u-boot-gitlab-ci-runner:focal-20211006-14Nov2021
+  ci_runner_image: trini/u-boot-gitlab-ci-runner:focal-20220105-10Jan2022
   # Add '-u 0' options for Azure pipelines, otherwise we get "permission
   # denied" error when it tries to "useradd -m -u 1001 vsts_azpcontainer",
   # since our $(ci_runner_image) user is not root.
.gitlab-ci.yml +1 -1
index d06cca45fd04..4c44c01e7bf5 100644
@@ -2,7 +2,7 @@
 
 # Grab our configured image.  The source for this is found at:
 # https://source.denx.de/u-boot/gitlab-ci-runner
-image: trini/u-boot-gitlab-ci-runner:focal-20211006-14Nov2021
+image: trini/u-boot-gitlab-ci-runner:focal-20220105-10Jan2022
 
 # We run some tests in different order, to catch some failures quicker.
 stages:
tools/docker/Dockerfile +1 -1
index fb422e758813..f19e618ffba3 100644
@@ -2,7 +2,7 @@
 # This Dockerfile is used to build an image containing basic stuff to be used
 # to build U-Boot and run our test suites.
 
-FROM ubuntu:focal-20211006
+FROM ubuntu:focal-20220105
 MAINTAINER Tom Rini <[email protected]>
 LABEL Description=" This image is for building U-Boot inside a container"
 
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

Quoted 2 lines · Tom Rini

Bring us to the focial-20220105 tag and rebuild our images on top of

s/focial/focal

author
#3 lore raw

Re: [PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

Quoted 5 lines · Fabio Estevam
Quoted 2 lines · Tom Rini

Bring us to the focial-20220105 tag and rebuild our images on top of

s/focial/focal

Whoops, thanks.

Signature · 1 line
Tom
author
#4 lore raw

Re: [PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

In reply to Tom Rini
Quoted 4 lines · Tom Rini

Bring us to the focal-20220105 tag and rebuild our images on top of this.

Applied to u-boot/master, thanks!

Signature · 1 line
Tom
author
#5 lore raw

PATCH 3/4 binman: Have faked binaries live in the output directory.

In reply to Tom Rini

In general, and for Azure specifically, we need to have files created in the output directory and cannot assume a writable source directory. Rework the faked blob support to put the faked binary in to the output directory and then stop the test from deleting the now non-existent file.

Cc Heiko Thiery <[email protected]>
Cc Simon Glass <[email protected]>
---
 tools/binman/etype/blob.py | 4 +++-
 tools/binman/ftest.py      | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)
tools/binman/etype/blob.py +3 -1
index 65ebb2ecf4d8..ca8023b1a0b2 100644
@@ -39,11 +39,13 @@ class Entry_blob(Entry):
 
     def ObtainContents(self):
         if self.allow_fake and not pathlib.Path(self._filename).is_file():
+            self._filename = tools.GetOutputFilename(self._filename)
             with open(self._filename, "wb") as out:
                 out.truncate(1024)
             self.faked = True
+        else:
+            self._filename = self.GetDefaultFilename()
 
-        self._filename = self.GetDefaultFilename()
         self._pathname = tools.GetInputFilename(self._filename,
             self.external and self.section.GetAllowMissing())
         # Allow the file to be missing
tools/binman/ftest.py +0 -1
index a9b7880f362f..d03ce6f05f8d 100644
@@ -4675,7 +4675,6 @@ class TestFunctional(unittest.TestCase):
         err = stderr.getvalue()
         self.assertRegex(err,
                          "Image '.*' has faked external blobs and is non-functional: .*")
-        os.remove('binman_faking_test_blob')
 
     def testVersion(self):
         """Test we can get the binman version"""
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH 3/4] binman: Have faked binaries live in the output directory.

Hi Tom,

Quoted 7 lines · Tom Rini

In general, and for Azure specifically, we need to have files created in the output directory and cannot assume a writable source directory. Rework the faked blob support to put the faked binary in to the output directory and then stop the test from deleting the now non-existent file.

Quoted 7 lines
Cc Heiko Thiery <[email protected]>
Cc Simon Glass <[email protected]>
---
 tools/binman/etype/blob.py | 4 +++-
 tools/binman/ftest.py      | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)

Please see this:

https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/

Regards,
Simon

author
#7 lore raw

Re: [PATCH 3/4] binman: Have faked binaries live in the output directory.

Quoted 18 lines · Simon Glass

Hi Tom,

Quoted 7 lines · Tom Rini

In general, and for Azure specifically, we need to have files created in the output directory and cannot assume a writable source directory. Rework the faked blob support to put the faked binary in to the output directory and then stop the test from deleting the now non-existent file.

Quoted 7 lines
Cc Heiko Thiery <[email protected]>
Cc Simon Glass <[email protected]>
---
 tools/binman/etype/blob.py | 4 +++-
 tools/binman/ftest.py      | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)
Quoted 3 lines

Please see this:

https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/

Ah, I thought you might have had it fixed but I didn't read well enough. My only concern is I would like to get Azure passing again ASAP. Should I just grab that patch by itself then?

Signature · 1 line
Tom
#8 lore raw

Re: [PATCH 3/4] binman: Have faked binaries live in the output directory.

Hi Tom,

Quoted 20 lines · Tom Rini
Quoted 18 lines · Simon Glass

Hi Tom,

Quoted 14 lines · Tom Rini

In general, and for Azure specifically, we need to have files created in the output directory and cannot assume a writable source directory. Rework the faked blob support to put the faked binary in to the output directory and then stop the test from deleting the now non-existent file.

Cc Heiko Thiery <[email protected]>
Cc Simon Glass <[email protected]>
---
 tools/binman/etype/blob.py | 4 +++-
 tools/binman/ftest.py      | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)
Quoted 7 lines
Quoted 3 lines

Please see this:

https://patchwork.ozlabs.org/project/uboot/patch/[email protected]/

Ah, I thought you might have had it fixed but I didn't read well enough. My only concern is I would like to get Azure passing again ASAP. Should I just grab that patch by itself then?

Yes, but I just sent v2 rebased to master (two patches).

Regards,
Simon

author
#9 lore raw

PATCH 4/4 .readthedocs.yml: Set our requirements file

In reply to Tom Rini

We provide a requirements.txt file for doc building, but had not been configuring readthedocs to know where it is.

Cc Heinrich Schuchardt <[email protected]>
---
 .readthedocs.yml | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
.readthedocs.yml +4 -5
index 44949ea239d8..c6f47b3e07e9 100644
@@ -12,8 +12,7 @@ sphinx:
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: []
 
-# Optionally set the version of Python and requirements required to build your docs
-# python:
-#   version: 3.7
-#   install:
-#     - requirements: docs/requirements.txt
+# Explicitly set the requirements file
+python:
+  install:
+    - requirements: doc/sphinx/requirements.txt
Signature · 1 line
2.25.1
#10 lore raw

Re: [PATCH 4/4] .readthedocs.yml: Set our requirements file

Quoted 15 lines · Tom Rini

We provide a requirements.txt file for doc building, but had not been configuring readthedocs to know where it is.

Cc Heinrich Schuchardt <[email protected]>

---
.readthedocs.yml | 9 ++++----- > 1 file changed, 4 insertions(+), 5 deletions(-)

.readthedocs.yml +0 -0
index 44949ea239d8..c6f47b3e07e9 100644
@@ -12,8 +12,7 @@ sphinx:
  # Optionally build your docs in additional formats such as PDF and ePub
  formats: []

.readthedocs.yml sphinx: · line 18

Quoted 10 lines
.readthedocs.yml +4 -5
 
-# Optionally set the version of Python and requirements required to build your docs
-# python:
-#   version: 3.7
-#   install:
-#     - requirements: docs/requirements.txt
+# Explicitly set the requirements file
+python:
+  install:
+    - requirements: doc/sphinx/requirements.txt

I could not get it running without install python3-six:

# .readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

build:
os: "ubuntu-20.04"
apt_packages:
- python3-six
tools:

    python: "3.9"

# Build documentation in the docs/ directory with Sphinx sphinx:
configuration: doc/conf.py

# Optionally build your docs in additional formats such as PDF and ePub
formats: []

python:
install:
- requirements: doc/sphinx/requirements.txt

Best regards

Heinrich

author
#11 lore raw
In reply to Tom Rini
Quoted 8 lines · Tom Rini

Follow what we do in GitLab CI where we break the jobs up in to stages such that if earlier and often quicker sanity tests fail we don't run everything else.

---
 .azure-pipelines.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

I want to note that in my testing this didn't change the overall run-time of the CI job.

Signature · 1 line
Tom
author
#12 lore raw
In reply to Tom Rini
Quoted 5 lines · Tom Rini

Follow what we do in GitLab CI where we break the jobs up in to stages such that if earlier and often quicker sanity tests fail we don't run everything else.

Applied to u-boot/master, thanks!

Signature · 1 line
Tom