Skip to contentSkip to conversation

gitlab-ci: Add a job to build EDK2 firmware binaries

PATCH Superseded Philippe Mathieu-Daudé 2 participants 11 messages Jan 6, 2020 – Jan 7, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover gitlab-ci: Add a job to build EDK2 firmware binaries

We provide the EDK2 firmware binaries in pc-bios/. When we update the roms/edk2/ submodule, we need to rebuild the firmware binaries.
To avoid the burden on a single developer, this series add a GitLab job to generate reproducible EDK2 firmware binaries.

The job is only on specific conditions:
- roms/edk2/ is updated
- the branch or tag start with 'edk2'
- 'edk2' appears in last commit description

Using ccache this job takes 32minutes on a GitLab free runner.

The proposed procedure to update the EDK2 submodule is:

- add a commit to update roms/edk2/ submodule
- push to gitlab
- download generated artifacts (only available if job succeed)
- unzip the firmware binaries from the artifacts archive
- test the binaries
- add a commit with the binaries, referencing the ci job url.

Example of job that built the edk2-stable201905 firmwares:
https://gitlab.com/philmd/qemu/-/jobs/395017298

The first patch is already reviewed, but is a prerequisite to use the Ubuntu docker image to build, so I included it.

Regards,

Phil.

Philippe Mathieu-Daudé (3):
roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian gitlab-ci.yml: Add a job to build EDK2 firmware binaries gitlab-ci-edk2.yml: Use ccache

 .gitlab-ci-edk2.yml | 48 +++++++++++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml      |  3 +++
 MAINTAINERS         |  3 ++-
 roms/edk2-funcs.sh  |  3 +++
 4 files changed, 56 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci-edk2.yml
Signature · 1 line
2.21.1
#1 lore raw

PATCH 1/3 roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian

The Debian (based) distributions currently provides 2 ARM toolchains, documented as [1]:

* The ARM EABI (armel) port targets a range of older 32-bit ARM devices, particularly those used in NAS hardware and a variety of *plug computers.
* The newer ARM hard-float (armhf) port supports newer, more powerful 32-bit devices using version 7 of the ARM architecture specification.

For various reasons documented in [2], the EDK2 project suggests to use the softfloat toolchain (named 'armel' by Debian).

Force the softfloat cross toolchain prefix on Debian distributions.

[1] https://www.debian.org/ports/arm/#status
[2] https://github.com/tianocore/edk2/commit/41203b9a

---
v3: fixed s/hard/float/ typo (Laszlo)

---
 roms/edk2-funcs.sh | 3 +++
 1 file changed, 3 insertions(+)
roms/edk2-funcs.sh +3 -0
index 3f4485b201..cd6e4f2c82 100644
@@ -112,6 +112,9 @@ qemu_edk2_get_cross_prefix()
      ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
     # no cross-compiler needed
     :
+  elif ( [ -e /etc/debian_version ] && [ "$gcc_arch" == arm ] ); then
+    # force soft-float cross-compiler on Debian
+    printf 'arm-linux-gnueabi-'
   else
     printf '%s-linux-gnu-\n' "$gcc_arch"
   fi
Signature · 1 line
2.21.1
#2 lore raw

Re: [PATCH 1/3] roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian

Quoted 15 lines · Philippe Mathieu-Daudé

The Debian (based) distributions currently provides 2 ARM toolchains, documented as [1]:

* The ARM EABI (armel) port targets a range of older 32-bit ARM devices, particularly those used in NAS hardware and a variety of *plug computers.
* The newer ARM hard-float (armhf) port supports newer, more powerful 32-bit devices using version 7 of the ARM architecture specification.

For various reasons documented in [2], the EDK2 project suggests to use the softfloat toolchain (named 'armel' by Debian).

Force the softfloat cross toolchain prefix on Debian distributions.

Quoted 7 lines

[1] https://www.debian.org/ports/arm/#status
[2] https://github.com/tianocore/edk2/commit/41203b9a

---
v3: fixed s/hard/float/ typo (Laszlo)

s/float/soft/, but OK otherwise :)

Thanks
Laszlo

Quoted 19 lines
---
 roms/edk2-funcs.sh | 3 +++
 1 file changed, 3 insertions(+)
roms/edk2-funcs.sh +3 -0
index 3f4485b201..cd6e4f2c82 100644
@@ -112,6 +112,9 @@ qemu_edk2_get_cross_prefix()
      ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
     # no cross-compiler needed
     :
+  elif ( [ -e /etc/debian_version ] && [ "$gcc_arch" == arm ] ); then
+    # force soft-float cross-compiler on Debian
+    printf 'arm-linux-gnueabi-'
   else
     printf '%s-linux-gnu-\n' "$gcc_arch"
   fi
#3 lore raw

PATCH 2/3 gitlab-ci.yml: Add a job to build EDK2 firmware binaries

In reply to Philippe Mathieu-Daudé

Add a GitLab job to build the EDK2 firmware binaries. This job is only built when the roms/edk2/ submodule is updated, when a git-ref starts with 'edk2' or when the last commit contains 'EDK2'.

GitLab CI generates an artifacts.zip file containing the firmware binaries.

With edk2-stable201905, the job took 40 minutes 26 seconds, the artifacts.zip takes 10MiB.

---
 .gitlab-ci-edk2.yml | 37 +++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml      |  3 +++
 MAINTAINERS         |  3 ++-
 3 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci-edk2.yml
.gitlab-ci-edk2.yml new file +37 -0
new file mode 100644
index 0000000000..abfaf52874
@@ -0,0 +1,37 @@
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # ... the branch/tag starts with 'edk2'
+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # last commit description contains 'EDK2'
+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: ubuntu:16.04 # Use Ubuntu Xenial
+ before_script: # Install packages requiered to build EDK2
+ - apt-get update --quiet --quiet
+ - DEBIAN_FRONTEND=noninteractive
+   apt-get install --assume-yes --no-install-recommends --quiet --quiet
+     build-essential
+     ca-certificates
+     dos2unix
+     gcc-aarch64-linux-gnu
+     gcc-arm-linux-gnueabi
+     git
+     iasl
+     make
+     nasm
+     python
+     uuid-dev
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2
+ - git -C roms/edk2 submodule update --init
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)
.gitlab-ci.yml +3 -0
index ebcef0ebe9..f799246047 100644
@@ -1,3 +1,6 @@
+include:
+  - local: '/.gitlab-ci-edk2.yml'
+
 before_script:
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
MAINTAINERS +2 -1
index 8571327881..22a1fd5824 100644
@@ -2346,6 +2346,7 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
+F: .gitlab-ci-edk2.yml
 
 Usermode Emulation
 ------------------
@@ -2689,7 +2690,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
 GitLab Continuous Integration
 M: Thomas Huth <[email protected]>
 S: Maintained
-F: .gitlab-ci.yml
+F: .gitlab-ci*.yml
 
 Guest Test Compilation Support
 M: Alex Bennée <[email protected]>
Signature · 1 line
2.21.1
#4 lore raw

Re: [PATCH 2/3] gitlab-ci.yml: Add a job to build EDK2 firmware binaries

Quoted 32 lines · Philippe Mathieu-Daudé

Add a GitLab job to build the EDK2 firmware binaries. This job is only built when the roms/edk2/ submodule is updated, when a git-ref starts with 'edk2' or when the last commit contains 'EDK2'.

GitLab CI generates an artifacts.zip file containing the firmware binaries.

With edk2-stable201905, the job took 40 minutes 26 seconds, the artifacts.zip takes 10MiB.

---
  .gitlab-ci-edk2.yml | 37 +++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml      |  3 +++
  MAINTAINERS         |  3 ++-
  3 files changed, 42 insertions(+), 1 deletion(-)

create mode 100644 .gitlab-ci-edk2.yml

.gitlab-ci-edk2.yml new file +7 -0
new file mode 100644
index 0000000000..abfaf52874
@@ -0,0 +1,37 @@
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # ... the branch/tag starts with 'edk2'
+   when: always

.gitlab-ci-edk2.yml line 17

Quoted 10 lines
.gitlab-ci-edk2.yml new file +10 -0
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # last commit description contains 'EDK2'
+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: ubuntu:16.04 # Use Ubuntu Xenial
+ before_script: # Install packages requiered to build EDK2

typo "required"

Quoted 53 lines
.gitlab-ci-edk2.yml new file +20 -0
+ - apt-get update --quiet --quiet
+ - DEBIAN_FRONTEND=noninteractive
+   apt-get install --assume-yes --no-install-recommends --quiet --quiet
+     build-essential
+     ca-certificates
+     dos2unix
+     gcc-aarch64-linux-gnu
+     gcc-arm-linux-gnueabi
+     git
+     iasl
+     make
+     nasm
+     python
+     uuid-dev
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2
+ - git -C roms/edk2 submodule update --init
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)
.gitlab-ci.yml +3 -0
index ebcef0ebe9..f799246047 100644
@@ -1,3 +1,6 @@
+include:
+  - local: '/.gitlab-ci-edk2.yml'
+
  before_script:
   - apt-get update -qq
   - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
MAINTAINERS +2 -1
index 8571327881..22a1fd5824 100644
@@ -2346,6 +2346,7 @@ F: roms/edk2
  F: roms/edk2-*
  F: tests/data/uefi-boot-images/
  F: tests/uefi-test-tools/
+F: .gitlab-ci-edk2.yml
 
  Usermode Emulation
  ------------------
@@ -2689,7 +2690,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
  GitLab Continuous Integration
  M: Thomas Huth <[email protected]>
  S: Maintained
-F: .gitlab-ci.yml
+F: .gitlab-ci*.yml
 
  Guest Test Compilation Support
  M: Alex Bennée <[email protected]>
#5 lore raw

Re: [PATCH 2/3] gitlab-ci.yml: Add a job to build EDK2 firmware binaries

In reply to Philippe Mathieu-Daudé
Quoted 4 lines · Philippe Mathieu-Daudé

Add a GitLab job to build the EDK2 firmware binaries. This job is only built when the roms/edk2/ submodule is updated, when a git-ref starts with 'edk2' or when the last commit contains 'EDK2'.

keyword "or"; okay.

Quoted 21 lines

GitLab CI generates an artifacts.zip file containing the firmware binaries.

With edk2-stable201905, the job took 40 minutes 26 seconds, the artifacts.zip takes 10MiB.

---
 .gitlab-ci-edk2.yml | 37 +++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml      |  3 +++
 MAINTAINERS         |  3 ++-
 3 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci-edk2.yml
.gitlab-ci-edk2.yml new file +0 -0
new file mode 100644
index 0000000000..abfaf52874

.gitlab-ci-edk2.yml line 6

Quoted 6 lines
.gitlab-ci-edk2.yml new file +6 -0
@@ -0,0 +1,37 @@
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # ... the branch/tag starts with 'edk2'

(1) can you add "or" in the comment here?

.gitlab-ci-edk2.yml line 8

Quoted 2 lines
.gitlab-ci-edk2.yml new file +2 -0
+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # last commit description contains 'EDK2'

(2) ditto

Quoted 15 lines
.gitlab-ci-edk2.yml new file +15 -0
+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: ubuntu:16.04 # Use Ubuntu Xenial
+ before_script: # Install packages requiered to build EDK2
+ - apt-get update --quiet --quiet
+ - DEBIAN_FRONTEND=noninteractive
+   apt-get install --assume-yes --no-install-recommends --quiet --quiet
+     build-essential
+     ca-certificates
+     dos2unix

.gitlab-ci-edk2.yml line 33

Quoted 10 lines
.gitlab-ci-edk2.yml new file +10 -0
+     gcc-aarch64-linux-gnu
+     gcc-arm-linux-gnueabi
+     git
+     iasl
+     make
+     nasm
+     python
+     uuid-dev
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2

yes, this is needed; qemu users are used to updating top-level submodules (which is why we didn't try to automate that away in the edk2 build stuff)

.gitlab-ci-edk2.yml line 34

Quoted 1 line
.gitlab-ci-edk2.yml new file +1 -0
+ - git -C roms/edk2 submodule update --init

(3) but this should not be necessary. See the "submodules" target in "roms/Makefile.edk2".

.gitlab-ci-edk2.yml line 37

Quoted 3 lines
.gitlab-ci-edk2.yml new file +3 -0
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)

Process substitution is a nifty feature, but perhaps we can do without it, for simplicity. (I realize this is bash-only; I just like to minimize the use of non-portable features if there is a portable replacement that is also simple.)

Redirections are processed in the order they appear on the command line [1], *after* stdout/stdin is redirected for pipelining [2]:

[1]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07

"If more than one redirection operator is specified with a command, the order of evaluation is from beginning to end."

[2]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02

"The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command"

(4) Therefore, the following should work:

  make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log \
  | tee -a edk2-stderr.log >&2

Untested, of course :)

Looks OK otherwise.

Thanks!
Laszlo

Quoted 33 lines
.gitlab-ci.yml +3 -0
index ebcef0ebe9..f799246047 100644
@@ -1,3 +1,6 @@
+include:
+  - local: '/.gitlab-ci-edk2.yml'
+
 before_script:
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
MAINTAINERS +2 -1
index 8571327881..22a1fd5824 100644
@@ -2346,6 +2346,7 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
+F: .gitlab-ci-edk2.yml
 
 Usermode Emulation
 ------------------
@@ -2689,7 +2690,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
 GitLab Continuous Integration
 M: Thomas Huth <[email protected]>
 S: Maintained
-F: .gitlab-ci.yml
+F: .gitlab-ci*.yml
 
 Guest Test Compilation Support
 M: Alex Bennée <[email protected]>
#6 lore raw

Re: [PATCH 2/3] gitlab-ci.yml: Add a job to build EDK2 firmware binaries

Quoted 69 lines · Laszlo Ersek
Quoted 4 lines · Philippe Mathieu-Daudé

Add a GitLab job to build the EDK2 firmware binaries. This job is only built when the roms/edk2/ submodule is updated, when a git-ref starts with 'edk2' or when the last commit contains 'EDK2'.

keyword "or"; okay.

Quoted 21 lines

GitLab CI generates an artifacts.zip file containing the firmware binaries.

With edk2-stable201905, the job took 40 minutes 26 seconds, the artifacts.zip takes 10MiB.

---
  .gitlab-ci-edk2.yml | 37 +++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml      |  3 +++
  MAINTAINERS         |  3 ++-
  3 files changed, 42 insertions(+), 1 deletion(-)

create mode 100644 .gitlab-ci-edk2.yml

.gitlab-ci-edk2.yml new file +0 -0
new file mode 100644
index 0000000000..abfaf52874

.gitlab-ci-edk2.yml line 6

Quoted 6 lines
.gitlab-ci-edk2.yml new file +6 -0
@@ -0,0 +1,37 @@
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # ... the branch/tag starts with 'edk2'

(1) can you add "or" in the comment here?

.gitlab-ci-edk2.yml line 8

Quoted 2 lines
.gitlab-ci-edk2.yml new file +2 -0
+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # last commit description contains 'EDK2'

(2) ditto

Quoted 15 lines
.gitlab-ci-edk2.yml new file +15 -0
+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: ubuntu:16.04 # Use Ubuntu Xenial
+ before_script: # Install packages requiered to build EDK2
+ - apt-get update --quiet --quiet
+ - DEBIAN_FRONTEND=noninteractive
+   apt-get install --assume-yes --no-install-recommends --quiet --quiet
+     build-essential
+     ca-certificates
+     dos2unix

.gitlab-ci-edk2.yml line 33

Quoted 10 lines
.gitlab-ci-edk2.yml new file +10 -0
+     gcc-aarch64-linux-gnu
+     gcc-arm-linux-gnueabi
+     git
+     iasl
+     make
+     nasm
+     python
+     uuid-dev
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2
Quoted 8 lines

yes, this is needed; qemu users are used to updating top-level submodules (which is why we didn't try to automate that away in the edk2 build stuff)

.gitlab-ci-edk2.yml line 34

Quoted 1 line
.gitlab-ci-edk2.yml new file +1 -0
+ - git -C roms/edk2 submodule update --init

(3) but this should not be necessary. See the "submodules" target in "roms/Makefile.edk2".

Hmm build fails without it:
https://gitlab.com/philmd/qemu/-/jobs/395644357#L436

The 'test -d edk2/.git' might be not enough?

Quoted 26 lines

.gitlab-ci-edk2.yml line 37

Quoted 3 lines
.gitlab-ci-edk2.yml new file +3 -0
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)

Process substitution is a nifty feature, but perhaps we can do without it, for simplicity. (I realize this is bash-only; I just like to minimize the use of non-portable features if there is a portable replacement that is also simple.)

Redirections are processed in the order they appear on the command line [1], *after* stdout/stdin is redirected for pipelining [2]:

[1]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07

"If more than one redirection operator is specified with a command, the order of evaluation is from beginning to end."

[2]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02

"The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command"

Quoted 6 lines

(4) Therefore, the following should work:

   make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log \
   | tee -a edk2-stderr.log >&2

Untested, of course :)

This works like charm :>

Quoted 1 line

Looks OK otherwise.

Thanks for the review!

Quoted 3 lines

Thanks!
Laszlo

#7 lore raw

Re: [PATCH 2/3] gitlab-ci.yml: Add a job to build EDK2 firmware binaries

Quoted 78 lines · Philippe Mathieu-Daudé
Quoted 77 lines · Laszlo Ersek
Quoted 4 lines · Philippe Mathieu-Daudé

Add a GitLab job to build the EDK2 firmware binaries. This job is only built when the roms/edk2/ submodule is updated, when a git-ref starts with 'edk2' or when the last commit contains 'EDK2'.

keyword "or"; okay.

Quoted 27 lines

GitLab CI generates an artifacts.zip file containing the firmware binaries.

With edk2-stable201905, the job took 40 minutes 26 seconds, the artifacts.zip takes 10MiB.

---
  .gitlab-ci-edk2.yml | 37 +++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml      |  3 +++
  MAINTAINERS         |  3 ++-

  3 files changed, 42 insertions(+), 1 deletion(-)

  create mode 100644 .gitlab-ci-edk2.yml

.gitlab-ci-edk2.yml new file +6 -0
new file mode 100644
index 0000000000..abfaf52874
@@ -0,0 +1,37 @@
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # ... the branch/tag starts
Quoted 1 line

with 'edk2'

(1) can you add "or" in the comment here?

Quoted 3 lines

+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # last commit description contains 'EDK2'

(2) ditto

Quoted 17 lines

+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: ubuntu:16.04 # Use Ubuntu Xenial
+ before_script: # Install packages requiered to build EDK2
+ - apt-get update --quiet --quiet
+ - DEBIAN_FRONTEND=noninteractive
+   apt-get install --assume-yes --no-install-recommends --quiet --quiet +     build-essential
+     ca-certificates
+     dos2unix
+     gcc-aarch64-linux-gnu
+     gcc-arm-linux-gnueabi

Quoted 8 lines

+     git
+     iasl
+     make
+     nasm
+     python
+     uuid-dev
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2

yes, this is needed; qemu users are used to updating top-level submodules (which is why we didn't try to automate that away in the edk2 build stuff)

Quoted 1 line

+ - git -C roms/edk2 submodule update --init

Quoted 7 lines
Quoted 2 lines

(3) but this should not be necessary. See the "submodules" target in "roms/Makefile.edk2".

Hmm build fails without it:
https://gitlab.com/philmd/qemu/-/jobs/395644357#L436

The 'test -d edk2/.git' might be not enough?

Huh, possibly a regression from commit f3e330e3c319 ("roms/Makefile.edk2: don't pull in submodules when building from tarball", 2019-10-07)?

Anyway, let me jump forward to your latest posting.

Thanks!
Laszlo

Quoted 47 lines
Quoted 30 lines
Quoted 5 lines

+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)

Process substitution is a nifty feature, but perhaps we can do without it, for simplicity. (I realize this is bash-only; I just like to minimize the use of non-portable features if there is a portable replacement that is also simple.)

Redirections are processed in the order they appear on the command line [1], *after* stdout/stdin is redirected for pipelining [2]:

[1]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07

"If more than one redirection operator is specified with a command, the order of evaluation is from beginning to end."

[2]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02

"The standard input, standard output, or both of a command shall be considered to be assigned by the pipeline before any redirection specified by redirection operators that are part of the command"

Quoted 6 lines

(4) Therefore, the following should work:

   make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log \
   | tee -a edk2-stderr.log >&2

Untested, of course :)

This works like charm :>

Quoted 1 line

Looks OK otherwise.

Thanks for the review!

Quoted 3 lines

Thanks!
Laszlo

#8 lore raw

PATCH 3/3 gitlab-ci-edk2.yml: Use ccache

In reply to Philippe Mathieu-Daudé

By using ccache we reduce the job duration from
40 minutes 26 seconds to 32 minutes 6 seconds.

  Running after script...
  $ ccache --show-stats
  cache hit (direct)                  6604
  files in cache                     12090
  cache size                         335.5 MB

For now downloading this cache takes 16 seconds, archiving it 44 seconds.

---
 .gitlab-ci-edk2.yml | 11 +++++++++++
 1 file changed, 11 insertions(+)
.gitlab-ci-edk2.yml +11 -0
index abfaf52874..329ba24f20 100644
@@ -14,12 +14,19 @@ build-edk2:
    - edk2-stdout.log
    - edk2-stderr.log
  image: ubuntu:16.04 # Use Ubuntu Xenial
+ variables:
+   CCACHE_DIR: ${CI_PROJECT_DIR}/.ccache
+ cache: # Use the same cache for all EDK2 jobs
+   key: ubuntu16.04-edk2-ccache
+   paths:
+   - ${CCACHE_DIR}
  before_script: # Install packages requiered to build EDK2
  - apt-get update --quiet --quiet
  - DEBIAN_FRONTEND=noninteractive
    apt-get install --assume-yes --no-install-recommends --quiet --quiet
      build-essential
      ca-certificates
+     ccache
      dos2unix
      gcc-aarch64-linux-gnu
      gcc-arm-linux-gnueabi
@@ -29,9 +36,13 @@ build-edk2:
      nasm
      python
      uuid-dev
+ - export PATH=/usr/lib/ccache:$PATH
+ - ccache --zero-stats
  script: # Clone the required submodules and build EDK2
  - git submodule update --init roms/edk2
  - git -C roms/edk2 submodule update --init
  - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
  - echo "=== Using ${JOBS} simultaneous jobs ==="
  - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)
+ after_script:
+ - ccache --show-stats
Signature · 1 line
2.21.1
#9 lore raw

Re: [PATCH 3/3] gitlab-ci-edk2.yml: Use ccache

Quoted 46 lines · Philippe Mathieu-Daudé

By using ccache we reduce the job duration from
40 minutes 26 seconds to 32 minutes 6 seconds.

  Running after script...
  $ ccache --show-stats
  cache hit (direct)                  6604
  files in cache                     12090
  cache size                         335.5 MB

For now downloading this cache takes 16 seconds, archiving it 44 seconds.

---
 .gitlab-ci-edk2.yml | 11 +++++++++++
 1 file changed, 11 insertions(+)
.gitlab-ci-edk2.yml +7 -0
index abfaf52874..329ba24f20 100644
@@ -14,12 +14,19 @@ build-edk2:
    - edk2-stdout.log
    - edk2-stderr.log
  image: ubuntu:16.04 # Use Ubuntu Xenial
+ variables:
+   CCACHE_DIR: ${CI_PROJECT_DIR}/.ccache
+ cache: # Use the same cache for all EDK2 jobs
+   key: ubuntu16.04-edk2-ccache
+   paths:
+   - ${CCACHE_DIR}
  before_script: # Install packages requiered to build EDK2
  - apt-get update --quiet --quiet
  - DEBIAN_FRONTEND=noninteractive
    apt-get install --assume-yes --no-install-recommends --quiet --quiet
      build-essential
      ca-certificates
+     ccache
      dos2unix
      gcc-aarch64-linux-gnu
      gcc-arm-linux-gnueabi
@@ -29,9 +36,13 @@ build-edk2:
      nasm
      python
      uuid-dev

.gitlab-ci-edk2.yml build-edk2: · line 48

Quoted 10 lines
.gitlab-ci-edk2.yml +4 -0
+ - export PATH=/usr/lib/ccache:$PATH
+ - ccache --zero-stats
  script: # Clone the required submodules and build EDK2
  - git submodule update --init roms/edk2
  - git -C roms/edk2 submodule update --init
  - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
  - echo "=== Using ${JOBS} simultaneous jobs ==="
  - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)
+ after_script:
+ - ccache --show-stats

I suggest dropping this patch. (In the first place: thank you for making this a separate patch!)

I'm not a fan of ccache, to be honest. I've seen obscure failures with it in the past. Also, the edk2 build system is a complicated beast in itself; let's not compose that with another opaque thing. I'm especially not fond of caching artifacts between multiple edk2 jobs.

For speeding up my builds, I used to use distcc instead; it worked better than ccache (using multiple machines in my home). But I abandoned even that, after a while.

I certainly don't intend to nack this patch -- if others really like (and trust) ccache, they are welcome to ack. I'm just not a fan of it.

Thanks,
Laszlo

#10 lore raw

Re: [PATCH 3/3] gitlab-ci-edk2.yml: Use ccache

Quoted 66 lines · Laszlo Ersek
Quoted 46 lines · Philippe Mathieu-Daudé

By using ccache we reduce the job duration from
40 minutes 26 seconds to 32 minutes 6 seconds.

   Running after script...
   $ ccache --show-stats
   cache hit (direct)                  6604
   files in cache                     12090
   cache size                         335.5 MB

For now downloading this cache takes 16 seconds, archiving it 44 seconds.

---
  .gitlab-ci-edk2.yml | 11 +++++++++++
  1 file changed, 11 insertions(+)
.gitlab-ci-edk2.yml +7 -0
index abfaf52874..329ba24f20 100644
@@ -14,12 +14,19 @@ build-edk2:
     - edk2-stdout.log
     - edk2-stderr.log
   image: ubuntu:16.04 # Use Ubuntu Xenial
+ variables:
+   CCACHE_DIR: ${CI_PROJECT_DIR}/.ccache
+ cache: # Use the same cache for all EDK2 jobs
+   key: ubuntu16.04-edk2-ccache
+   paths:
+   - ${CCACHE_DIR}
   before_script: # Install packages requiered to build EDK2
   - apt-get update --quiet --quiet
   - DEBIAN_FRONTEND=noninteractive
     apt-get install --assume-yes --no-install-recommends --quiet --quiet
       build-essential
       ca-certificates
+     ccache
       dos2unix
       gcc-aarch64-linux-gnu
       gcc-arm-linux-gnueabi
@@ -29,9 +36,13 @@ build-edk2:
       nasm
       python
       uuid-dev

.gitlab-ci-edk2.yml build-edk2: · line 48

Quoted 10 lines
.gitlab-ci-edk2.yml +4 -0
+ - export PATH=/usr/lib/ccache:$PATH
+ - ccache --zero-stats
   script: # Clone the required submodules and build EDK2
   - git submodule update --init roms/edk2
   - git -C roms/edk2 submodule update --init
   - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
   - echo "=== Using ${JOBS} simultaneous jobs ==="
   - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)
+ after_script:
+ - ccache --show-stats

I suggest dropping this patch. (In the first place: thank you for making this a separate patch!)

I'm not a fan of ccache, to be honest. I've seen obscure failures with it in the past. Also, the edk2 build system is a complicated beast in itself; let's not compose that with another opaque thing. I'm especially not fond of caching artifacts between multiple edk2 jobs.

Quoted 6 lines

For speeding up my builds, I used to use distcc instead; it worked better than ccache (using multiple machines in my home). But I abandoned even that, after a while.

I certainly don't intend to nack this patch -- if others really like (and trust) ccache, they are welcome to ack. I'm just not a fan of it.

I understand. I'll still include it in v2, tagged 'RFC' (or NOTFORMERGE) so Thomas can look at it, and it gets saved on the mailing list archive.