.gitlab-ci.yml: run our runtime tests

PATCH New Arnout Vandecappelle (Essensium/Mind) 2 participants 7 messages Jul 2, 2017 Kbuild

lore

Share & tools

lore

Arnout Vandecappelle (Essensium/Mind)

author

2017-07-02 16:13 UTC #0 lore raw

PATCH 1/3 .gitlab-ci.yml: run our runtime tests

From: Thomas Petazzoni <[email protected]>

This commit improves our .gitlab-ci.yml logic to execute our runtime tests located in support/testing/. To do so, this commit:

- Adds more Debian packages to be installed, namely the nose2 and pexpect packages needed by the runtime testing infrastructure, as well as the necessary Qemu emulators

- The description of how to run the runtime tests. Each test is executed as a separate Gitlab CI job, so that the status of each test is easily visible in the Gitlab CI web interface.

- The Makefile is improved to auto-generate .gitlab-ci.yml from .gitlab-ci.yml.in, like we're doing for defconfigs. Since the dependencies of .gitlab-ci.yml are no longer correct, we abandon them and instead make it a PHONY target.

Signed-off-by Thomas Petazzoni <[email protected]>

[Arnout: simplify .gitlab-ci.in a little, removing redundant stuff;
make .gitlab-ci.yml a PHONY target]

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 .gitlab-ci.yml    | 33 +++++++++++++++++++++++++++++++++
 .gitlab-ci.yml.in |  8 ++++++++
 Makefile          |  7 +++++--
 3 files changed, 46 insertions(+), 2 deletions(-)

.gitlab-ci.yml +33 -0

index eda8af4997..7acf4b620e 100644
@@ -13,6 +13,7 @@ before_script:
     - apt-get install -y -qq --no-install-recommends
         build-essential locales bc ca-certificates file rsync gcc-multilib
         git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
+        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
     # To be able to generate a toolchain with locales, enable one UTF-8 locale
     - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
     - /usr/sbin/locale-gen

@@ -57,6 +58,13 @@ check-DEVELOPERS:
             - output/build/build-time.log
             - output/build/packages-file-list.txt
 
+.runtime_test: &runtime_test
+    script: ./support/testing/run-tests -o test-output/ -d test-dl/ ${CI_BUILD_NAME}
+    artifacts:
+        when: always
+        expire_in: 2 weeks
+        paths:
+            - test-output/*.log
 acmesystems_aria_g25_128mb_defconfig: *defconfig
 acmesystems_aria_g25_256mb_defconfig: *defconfig
 acmesystems_arietta_g25_128mb_defconfig: *defconfig

@@ -209,3 +217,28 @@ zynq_microzed_defconfig: *defconfig
 zynq_zc706_defconfig: *defconfig
 zynq_zed_defconfig: *defconfig
 zynq_zybo_defconfig: *defconfig
+tests.core.test_post_scripts.TestPostScripts: *runtime_test
+tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
+tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
+tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
+tests.core.test_timezone.TestNoTimezone: *runtime_test
+tests.fs.test_ext.TestExt2: *runtime_test
+tests.fs.test_ext.TestExt2r1: *runtime_test
+tests.fs.test_ext.TestExt3: *runtime_test
+tests.fs.test_ext.TestExt4: *runtime_test
+tests.fs.test_iso9660.TestIso9660Grub2External: *runtime_test
+tests.fs.test_iso9660.TestIso9660Grub2Internal: *runtime_test
+tests.fs.test_iso9660.TestIso9660GrubExternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660GrubInternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660SyslinuxExternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660SyslinuxInternal: *runtime_test
+tests.fs.test_jffs2.TestJffs2: *runtime_test
+tests.fs.test_squashfs.TestSquashfs: *runtime_test
+tests.fs.test_ubi.TestUbi: *runtime_test
+tests.fs.test_yaffs2.TestYaffs2: *runtime_test
+tests.package.test_dropbear.TestDropbear: *runtime_test
+tests.package.test_python.TestPythonBase: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainLinaroArm: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainSourceryArmv4: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainSourceryArmv5: *runtime_test
+tests.toolchain.test_external.TestExternalToolchainSourceryArmv7: *runtime_test

.gitlab-ci.yml.in +8 -0

index 24a1888c1e..c69f9de99c 100644
@@ -13,6 +13,7 @@ before_script:
     - apt-get install -y -qq --no-install-recommends
         build-essential locales bc ca-certificates file rsync gcc-multilib
         git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
+        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
     # To be able to generate a toolchain with locales, enable one UTF-8 locale
     - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
     - /usr/sbin/locale-gen

@@ -57,3 +58,10 @@ check-DEVELOPERS:
             - output/build/build-time.log
             - output/build/packages-file-list.txt
 
+.runtime_test: &runtime_test
+    script: ./support/testing/run-tests -o test-output/ -d test-dl/ ${CI_BUILD_NAME}
+    artifacts:
+        when: always
+        expire_in: 2 weeks
+        paths:
+            - test-output/*.log

Makefile +5 -2

index 811158295a..36231fa9d6 100644
@@ -1105,8 +1105,11 @@ release:
 print-version:
 	@echo $(BR2_VERSION_FULL)
 
-.gitlab-ci.yml: .gitlab-ci.yml.in configs/*_defconfig
-	(cd configs; LC_ALL=C ls -1 *_defconfig) | sed 's/$$/: *defconfig/' | cat $< - > $@
+.PHONY: .gitlab-ci.yml
+.gitlab-ci.yml: .gitlab-ci.yml.in
+	cp $< $@
+	(cd configs; LC_ALL=C ls -1 *_defconfig) | sed 's/$$/: *defconfig/' >> $@
+	./support/testing/run-tests -l 2>&1 | sed -r -e '/^test_run \((.*)\).*/!d; s//\1: *runtime_test/' | LC_ALL=C sort >> $@
 
 include docs/manual/manual.mk
 -include $(foreach dir,$(BR2_EXTERNAL_DIRS),$(dir)/docs/*/*.mk)

Signature · 1 line

2.13.2

Arnout Vandecappelle (Essensium/Mind)

author

2017-07-02 16:13 UTC #1 lore raw

PATCH 2/3 Add Dockerfile for CI

For Gitlab-CI, we want to avoid re-generating the minimal install to be able to run tests all the time. So let's create a docker image that we can post on Docker Hub and then pull.

For the time being, this is just what we need for running our CI. Later we can produce something that is also useful for users.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 support/docker/Dockerfile | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/docker/Dockerfile

support/docker/Dockerfile new file +30 -0

new file mode 100644
index 0000000000..d45b23dc23
@@ -0,0 +1,30 @@
+# This Dockerfile generates the docker image that gets used by Gitlab CI
+# To build it:
+#   sudo docker build -t buildroot/base support/docker
+#   sudo docker push buildroot/base
+
+FROM debian:stable
+LABEL maintainer="Buildroot mailing list <[email protected]>" \
+      vendor="Buildroot" \
+description="Container with everything needed to run Buildroot"
+
+# Setup environment
+ENV DEBIAN_FRONTEND noninteractive
+
+RUN dpkg --add-architecture i386
+# The container has no package lists, so need to update first
+RUN apt-get update -y -qq
+RUN apt-get install -y -qq --no-install-recommends \
+    build-essential cmake libc6:i386 gcc-multilib \
+    bc ca-certificates file locales rsync \
+    cvs bzr git mercurial subversion wget \
+    cpio unzip \
+    libncurses5-dev \
+    python-nose2 python-pexpect qemu-system-arm qemu-system-x86
+RUN apt-get -q -y autoremove
+RUN apt-get -q -y clean
+
+# To be able to generate a toolchain with locales, enable one UTF-8 locale
+RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
+RUN /usr/sbin/locale-gen
+

Signature · 1 line

2.13.2

Thomas Petazzoni

2017-07-02 21:46 UTC #2 lore raw

PATCH 2/3 Add Dockerfile for CI

Hello,

Quoted 4 lines · Arnout Vandecappelle (Essensium/Mind)

For Gitlab-CI, we want to avoid re-generating the minimal install to be able to run tests all the time. So let's create a docker image that we can post on Docker Hub and then pull.

Quoted 8 lines

For the time being, this is just what we need for running our CI. Later we can produce something that is also useful for users.
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/docker/Dockerfile | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/docker/Dockerfile

Applied to master, thanks. Will you provide an update to the Buildroot manual, documenting this next to the Vagrant stuff?

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Arnout Vandecappelle (Essensium/Mind)

author

2017-07-02 16:13 UTC #3 lore raw

PATCH 3/3 .gitlab-ci.yml: use the Buildroot CI image published on Docker Hub

In reply to Arnout Vandecappelle (Essensium/Mind)

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 .gitlab-ci.yml    | 14 +-------------
 .gitlab-ci.yml.in | 14 +-------------
 2 files changed, 2 insertions(+), 26 deletions(-)

.gitlab-ci.yml +1 ~~-13~~

index 7acf4b620e..3539b7e439 100644
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/ci
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'

.gitlab-ci.yml.in +1 ~~-13~~

index c69f9de99c..3088677e67 100644
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/base
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'

Signature · 1 line

2.13.2

Arnout Vandecappelle (Essensium/Mind)

author

2017-07-02 16:24 UTC #4 lore raw

PATCH v2 1/1 .gitlab-ci.yml: use the Buildroot CI image published on Docker Hub

In reply to Arnout Vandecappelle (Essensium/Mind)

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
v2: Forgot to update .gitlab-ci.yml after the final changes

---
 .gitlab-ci.yml    | 14 +-------------
 .gitlab-ci.yml.in | 14 +-------------
 2 files changed, 2 insertions(+), 26 deletions(-)

.gitlab-ci.yml +1 ~~-13~~

index 7acf4b620e..3ae7e5db64 100644
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/base
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'

.gitlab-ci.yml.in +1 ~~-13~~

index c69f9de99c..3088677e67 100644
@@ -4,19 +4,7 @@
 # It needs to be regenerated every time a defconfig is added, using
 # "make .gitlab-ci.yml".
 
-image: debian:stable
-
-before_script:
-    - dpkg --add-architecture i386
-    # The container has no package lists, so need to update first
-    - apt-get update -qq
-    - apt-get install -y -qq --no-install-recommends
-        build-essential locales bc ca-certificates file rsync gcc-multilib
-        git bzr cvs mercurial subversion libc6:i386 unzip wget cpio
-        python-nose2 python-pexpect qemu-system-arm qemu-system-x86
-    # To be able to generate a toolchain with locales, enable one UTF-8 locale
-    - sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen
-    - /usr/sbin/locale-gen
+image: buildroot/base
 
 .defconfig_script: &defconfig_script
     - echo 'Configure Buildroot'

Signature · 1 line

2.13.2

Thomas Petazzoni

Applied

2017-07-02 21:46 UTC #5 lore raw

PATCH v2 1/1 .gitlab-ci.yml: use the Buildroot CI image published on Docker Hub

Hello,

Quoted 7 lines · Arnout Vandecappelle (Essensium/Mind)

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
v2: Forgot to update .gitlab-ci.yml after the final changes
---
 .gitlab-ci.yml    | 14 +-------------
 .gitlab-ci.yml.in | 14 +-------------
 2 files changed, 2 insertions(+), 26 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Thomas Petazzoni

Applied

2017-07-02 21:45 UTC #6 lore raw

PATCH 1/3 .gitlab-ci.yml: run our runtime tests

In reply to Arnout Vandecappelle (Essensium/Mind)

Hello,

Quoted 19 lines · Arnout Vandecappelle (Essensium/Mind)

From: Thomas Petazzoni <[email protected]>
This commit improves our .gitlab-ci.yml logic to execute our runtime tests located in support/testing/. To do so, this commit:
- Adds more Debian packages to be installed, namely the nose2 and pexpect packages needed by the runtime testing infrastructure, as well as the necessary Qemu emulators
- The description of how to run the runtime tests. Each test is executed as a separate Gitlab CI job, so that the status of each test is easily visible in the Gitlab CI web interface.
- The Makefile is improved to auto-generate .gitlab-ci.yml from .gitlab-ci.yml.in, like we're doing for defconfigs. Since the dependencies of .gitlab-ci.yml are no longer correct, we abandon them and instead make it a PHONY target.
Signed-off-by Thomas Petazzoni <[email protected]>

Quoted 8 lines

[Arnout: simplify .gitlab-ci.in a little, removing redundant stuff;
make .gitlab-ci.yml a PHONY target]
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 .gitlab-ci.yml    | 33 +++++++++++++++++++++++++++++++++
 .gitlab-ci.yml.in |  8 ++++++++
 Makefile          |  7 +++++--
 3 files changed, 46 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com