Travis enhancements

RFC PATCH Superseded Petr Vorel 3 participants 11 messages Dec 1, 2017 – Dec 6, 2017

lore

Share & tools

lore

Petr Vorel

author

2017-12-01 15:46 UTC #0 lore raw

v2 cover Travis enhancements

TL;DR
See https://travis-ci.org/pevik-travis/ltp/builds/310089523

Now we have 11 jobs testing 7 compilers in 3 different ways (mostly normal builds, 2x 32-bit cross-compile builds, 1x out-of-tree build, 1x build with minimal dependencies).

It seems to be rule running only 5 containers at a time, so 11th build slowes building about 4 min (18 min vs. 22 min). Which build would you remove?

I decided to specify dependencies in file (saves some duplicity). These files can be used for builds on Debian/Ubuntu (as we re

Changes v1->v2:
* Dropped compilers:
Dropped: gcc 4.6, 4.7, 4.8 and clang 3.5, 3.8
* Added different types of builds (32-bit cross-compile builds, out-of-tree build)
* Better handling of header and library dependencies (minimal build and build with full support).
* Minor changes.

Notes what I learned:
* Travis needs sudo for using apt, but it does not allow to use it in script (it must be in travis.yml)
* Don't use $INSTALL environment variable (I haven't figured how, but it reported build to be failed).

Petr Vorel (5):
travis: Add build script and use it in travis
travis: Drop old compilers, add new ones
travis: Add 32-bit cross-compile builds
travis: Add out-of-tree build
travis: Install all dependencies + add build with minimal dependencies

 .travis.packages_i386   |   9 ++++
 .travis.packages_native |  18 ++++++++
 .travis.yml             |  87 ++++++++++++++++++++++++++-----------
 build.sh                | 111 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 201 insertions(+), 24 deletions(-)
 create mode 100644 .travis.packages_i386
 create mode 100644 .travis.packages_native
 create mode 100755 build.sh

Signature · 1 line

2.15.0

Petr Vorel

author

2017-12-01 15:46 UTC #1 lore raw

RFC PATCH v2 1/5 travis: Add build script and use it in travis

This script handles native build, 32-bit cross-compile build and out-of-tree build.

Script is for travis build, but can be used for local builds as well.

For usage run
./build.sh -h

Signed-off-by Petr Vorel <[email protected]>

---
 .travis.yml |   2 +-
 build.sh    | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 112 insertions(+), 1 deletion(-)
 create mode 100755 build.sh

.travis.yml +1 -1

index d937f9dcf..f2d51f131 100644
@@ -72,4 +72,4 @@ notifications:
     email:
         secure: "b/xcA/K5OyQvPPnd0PRahTH5LJu8lgz8goGHvhXpHo+ZPsPgTDXNFo5cX9fSOkMuFKeoW8iGl0wOgK2+ptc8mbYDw277K4RFIHRHeV/KIoE1EzjQnEFiL8J0oHCAvDj12o0AXeriTyY9gICXKbR31Br6Zh5eKViDJe2OAGeHeDU="
 
-script: make autotools && ./configure --prefix $HOME/ltp --with-open-posix-testsuite --with-realtime-testsuite && make -j$(getconf _NPROCESSORS_ONLN) && make -j$(getconf _NPROCESSORS_ONLN) install
+script: ./build.sh

build.sh new file +111 -0

111 lines not on this page · raw message

Signature · 1 line

2.15.0

Li Wang

2017-12-04 09:55 UTC #2 lore raw

RFC PATCH v2 1/5 travis: Add build script and use it in travis

Quoted 35 lines · Petr Vorel

This script handles native build, 32-bit cross-compile build and out-of-tree build.

Script is for travis build, but can be used for local builds as well.

For usage run
./build.sh -h

Signed-off-by Petr Vorel <[email protected]>

---
 .travis.yml |   2 +-
 build.sh    | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 112 insertions(+), 1 deletion(-)
 create mode 100755 build.sh

.travis.yml +1 -1

index d937f9dcf..f2d51f131 100644
@@ -72,4 +72,4 @@ notifications:
     email:
         secure: "b/xcA/K5OyQvPPnd0PRahTH5LJu8lgz8goGHvhXpHo+ZPsPgTDXNFo5cX9fSOkMuFKeoW8iGl0wOgK2+ptc8mbYDw277K4RFIHRHeV/KIoE1EzjQnEFiL8J0oHCAvDj12o0AXeriTyY9gICXKbR31Br6Zh5eKViDJe2OAGeHeDU="
 
-script: make autotools && ./configure --prefix $HOME/ltp --with-open-posix-testsuite --with-realtime-testsuite && make -j$(getconf _NPROCESSORS_ONLN) && make -j$(getconf _NPROCESSORS_ONLN) install
+script: ./build.sh

build.sh new file +4 -0

new file mode 100755
index 000000000..9e057f73b
@@ -0,0 +1,111 @@
+#!/bin/sh
+# Copyright (c) 2017 Petr Vorel <[email protected]>
+# Script for travis builds.
+#

build.sh line 14

Quoted 10 lines

build.sh new file +10 -0

+# TODO: Implement comparison of installed files. List of installed files can
+# be used only for local builds as Travis currently doesn't support sharing
+# file between jobs, see
+# https://github.com/travis-ci/travis-ci/issues/6054
+
+set -e
+
+PREFIX="$HOME/ltp"
+
+CONFIGURE_OPTS_IN_TREE="--with-open-posix-testsuite --with-realtime-testsuite --prefix=$PREFIX"

Here you set --prefix to current ltp path, why not "/opt/ltp"?

And, I get some installing errors with this configuration for build in tree.

# ./build.sh
[...]
install: ‘/root/ltp/testcases/realtime/perf/latency/disknoise.sh’ and
‘/root/ltp/testcases/realtime/perf/latency/disknoise.sh’ are the same
file
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/cpunoise2000.sh] Error 1
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/pthread_cond_latency]
Error 1
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/disknoise.sh] Error 1
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/cpunoise.sh] Error 1
make[4]: Leaving directory `/root/ltp/testcases/realtime/perf/latency'
make[3]: *** [install] Error 2
make[3]: Leaving directory `/root/ltp/testcases/realtime/perf'
make[2]: *** [install] Error 2
make[2]: Leaving directory `/root/ltp/testcases/realtime'
make[1]: *** [install] Error 2
make[1]: Leaving directory `/root/ltp/testcases'
make: *** [testcases-install] Error 2
make: *** Waiting for unfinished jobs....
[...]

Test Environment
==============

# uname -rm
4.15.0-rc2 x86_64

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
--enable-shared --enable-threads=posix --enable-checking=release
--with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto
--enable-plugin --enable-initfini-array --disable-libgcj
--with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install
--enable-gnu-indirect-function --with-tune=generic
--with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)

Signature · 2 lines

Li Wang
[email protected]

Petr Vorel

author

2017-12-04 11:24 UTC #3 lore raw

RFC PATCH v2 1/5 travis: Add build script and use it in travis

Hi Li,

Quoted 5 lines

build.sh line 14
Quoted 3 lines
build.sh new file +3 -0
+PREFIX="$HOME/ltp"
+
+CONFIGURE_OPTS_IN_TREE="--with-open-posix-testsuite --with-realtime-testsuite --prefix=$PREFIX"
Here you set --prefix to current ltp path, why not "/opt/ltp"?

Permissions (using containers => no sudo). We could have sudo, but it would be without containers (=slow).
See
https://docs.travis-ci.com/user/reference/overview/

Quoted 14 lines

And, I get some installing errors with this configuration for build in tree.

# ./build.sh
[...]
install: ‘/root/ltp/testcases/realtime/perf/latency/disknoise.sh’ and
‘/root/ltp/testcases/realtime/perf/latency/disknoise.sh’ are the same
file
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/cpunoise2000.sh] Error 1
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/pthread_cond_latency]
Error 1
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/disknoise.sh] Error 1
make[4]: *** [/root/ltp/testcases/realtime/perf/latency/cpunoise.sh] Error 1
make[4]: Leaving directory `/root/ltp/testcases/realtime/perf/latency'
make[3]: *** [install] Error 2

Quoted 8 lines

make[3]: Leaving directory `/root/ltp/testcases/realtime/perf'
make[2]: *** [install] Error 2
make[2]: Leaving directory `/root/ltp/testcases/realtime'
make[1]: *** [install] Error 2
make[1]: Leaving directory `/root/ltp/testcases'
make: *** [testcases-install] Error 2
make: *** Waiting for unfinished jobs....
[...]

Github clones into different directory:
git clone --depth=50 --branch=master https://github.com/linux-test-project/ltp.git linux-test-project/ltp OK, you're building in the same directory as the prefix is set. I'll fix it so we can use it for local builds as well.
BTW: I was thinking before using /tmp, but as whole directory has ~600MB, I'll use different name under $HOME).

Thanks for reporting it!

Kind regards,
Petr

Petr Vorel

author

2017-12-01 15:46 UTC #4 lore raw

RFC PATCH v2 2/5 travis: Drop old compilers, add new ones

In reply to Petr Vorel

Dropped: gcc 4.6, 4.7, 4.8 and clang 3.5, 3.8
Added: gcc 6, 7 and clang 4.0, 5.0

+ simplify code for clang 3.9 as distro trusty is default in travis since August 2017, see:
https://docs.travis-ci.com/user/reference/overview/

Signed-off-by Petr Vorel <[email protected]>

---
 .travis.yml | 49 ++++++++++++++++---------------------------------
 1 file changed, 16 insertions(+), 33 deletions(-)

.travis.yml +16 ~~-33~~

index f2d51f131..8aea22d01 100644
@@ -2,27 +2,6 @@ language: c
 
 matrix:
     include:
-        - os: linux
-          compiler: gcc-4.6
-          addons:
-              apt:
-                  sources: ['ubuntu-toolchain-r-test']
-                  packages: ['gcc-4.6']
-
-        - os: linux
-          compiler: gcc-4.7
-          addons:
-              apt:
-                  sources: ['ubuntu-toolchain-r-test']
-                  packages: ['gcc-4.7']
-
-        - os: linux
-          compiler: gcc-4.8
-          addons:
-              apt:
-                  sources: ['ubuntu-toolchain-r-test']
-                  packages: ['gcc-4.8']
-
         - os: linux
           compiler: gcc-4.9
           addons:

@@ -45,28 +24,32 @@ matrix:
                   packages: ['gcc-6']
 
         - os: linux
-          compiler: clang-3.5
+          compiler: gcc-7
+          addons:
+              apt:
+                  sources: ['ubuntu-toolchain-r-test']
+                  packages: ['gcc-7']
+
+        - os: linux
+          compiler: clang-3.9
           addons:
               apt:
-                  sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-precise-3.5']
-                  packages: ['clang-3.5']
+                  sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-trusty-3.9']
+                  packages: ['clang-3.9']
 
         - os: linux
-          compiler: clang-3.8
+          compiler: clang-4.0
           addons:
               apt:
-                  sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-precise-3.8']
-                  packages: ['clang-3.8']
+                  sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-trusty-4.0']
+                  packages: ['clang-4.0']
 
         - os: linux
-          compiler: clang-3.9
+          compiler: clang-5.0
           addons:
               apt:
-                  sources:
-                      - sourceline: "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main"
-                        key_url: "http://apt.llvm.org/llvm-snapshot.gpg.key"
-                      - 'ubuntu-toolchain-r-test'
-                  packages: ['clang-3.9']
+                  sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-trusty-5.0']
+                  packages: ['clang-5.0']
 
 notifications:
     email:

Signature · 1 line

2.15.0

Petr Vorel

author

2017-12-01 15:46 UTC #5 lore raw

RFC PATCH v2 3/5 travis: Add 32-bit cross-compile builds

In reply to Petr Vorel

Choosen gcc 4.9 and 6 (to use old and some quite new version).

Signed-off-by Petr Vorel <[email protected]>

---
 .travis.yml | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

.travis.yml +25 -1

index 8aea22d01..3ca37e47a 100644
@@ -3,6 +3,7 @@ language: c
 matrix:
     include:
         - os: linux
+          env: BUILD="native"
           compiler: gcc-4.9
           addons:
               apt:

@@ -10,6 +11,7 @@ matrix:
                   packages: ['gcc-4.9']
 
         - os: linux
+          env: BUILD="native"
           compiler: gcc-5
           addons:
               apt:

@@ -17,6 +19,7 @@ matrix:
                   packages: ['gcc-5']
 
         - os: linux
+          env: BUILD="native"
           compiler: gcc-6
           addons:
               apt:

@@ -24,6 +27,7 @@ matrix:
                   packages: ['gcc-6']
 
         - os: linux
+          env: BUILD="native"
           compiler: gcc-7
           addons:
               apt:

@@ -31,6 +35,7 @@ matrix:
                   packages: ['gcc-7']
 
         - os: linux
+          env: BUILD="native"
           compiler: clang-3.9
           addons:
               apt:

@@ -38,6 +43,7 @@ matrix:
                   packages: ['clang-3.9']
 
         - os: linux
+          env: BUILD="native"
           compiler: clang-4.0
           addons:
               apt:

@@ -45,14 +51,32 @@ matrix:
                   packages: ['clang-4.0']
 
         - os: linux
+          env: BUILD="native"
           compiler: clang-5.0
           addons:
               apt:
                   sources: ['ubuntu-toolchain-r-test', 'llvm-toolchain-trusty-5.0']
                   packages: ['clang-5.0']
 
+        - os: linux
+          env: BUILD="32"
+          compiler: gcc-4.9
+          addons:
+              apt:
+                  sources: ['ubuntu-toolchain-r-test']
+                  packages: ['gcc-4.9', 'gcc-4.9-multilib', 'linux-libc-dev:i386']
+
+        - os: linux
+          env: BUILD="32"
+          compiler: gcc-6
+          addons:
+              apt:
+                  sources: ['ubuntu-toolchain-r-test']
+                  packages: ['gcc-6', 'gcc-6-multilib', 'linux-libc-dev:i386']
+
 notifications:
     email:
         secure: "b/xcA/K5OyQvPPnd0PRahTH5LJu8lgz8goGHvhXpHo+ZPsPgTDXNFo5cX9fSOkMuFKeoW8iGl0wOgK2+ptc8mbYDw277K4RFIHRHeV/KIoE1EzjQnEFiL8J0oHCAvDj12o0AXeriTyY9gICXKbR31Br6Zh5eKViDJe2OAGeHeDU="
 
-script: ./build.sh
+before_install:
+script: ./build.sh $BUILD

Signature · 1 line

2.15.0

Petr Vorel

author

2017-12-01 15:46 UTC #6 lore raw

RFC PATCH v2 4/5 travis: Add out-of-tree build

In reply to Petr Vorel

Signed-off-by Petr Vorel <[email protected]>

---
 .travis.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

.travis.yml +8 -0

index 3ca37e47a..cc5627412 100644
@@ -74,6 +74,14 @@ matrix:
                   sources: ['ubuntu-toolchain-r-test']
                   packages: ['gcc-6', 'gcc-6-multilib', 'linux-libc-dev:i386']
 
+        - os: linux
+          env: BUILD="out"
+          compiler: gcc-7
+          addons:
+              apt:
+                  sources: ['ubuntu-toolchain-r-test']
+                  packages: ['gcc-7']
+
 notifications:
     email:
         secure: "b/xcA/K5OyQvPPnd0PRahTH5LJu8lgz8goGHvhXpHo+ZPsPgTDXNFo5cX9fSOkMuFKeoW8iGl0wOgK2+ptc8mbYDw277K4RFIHRHeV/KIoE1EzjQnEFiL8J0oHCAvDj12o0AXeriTyY9gICXKbR31Br6Zh5eKViDJe2OAGeHeDU="

Signature · 1 line

2.15.0

Petr Vorel

author

2017-12-01 15:46 UTC #7 lore raw

RFC PATCH v2 5/5 travis: Install all dependencies + add build with minimal dependencies

In reply to Petr Vorel

Currently travis has installed toolchain with libraries, glibc and kernel headers so some of the packages listed are already installed.

Install all dependencies (headers and libraries).
The only exception is selinux support on 32bit build as as libsepol1-dev:i386 conflict with libsepol1-dev for amd64, thus both cannot be installed.

During reinstallation for build with minimal dependencies we need to filter out some packages as toolchain package depends on it.

For minimal build chosen gcc 6 (relativelly recent gcc version, but not too new).

Signed-off-by Petr Vorel <[email protected]>

---
 .travis.packages_i386   |  9 +++++++++
 .travis.packages_native | 18 ++++++++++++++++++
 .travis.yml             | 44 ++++++++++++++++++++++++++++++++++----------
 3 files changed, 61 insertions(+), 10 deletions(-)
 create mode 100644 .travis.packages_i386
 create mode 100644 .travis.packages_native

.travis.packages_i386 new file +9 -0

new file mode 100644
index 000000000..725329041
@@ -0,0 +1,9 @@
+libacl1:i386
+libaio1:i386
+libcap2:i386
+libc6-dev-i386
+libc6:i386
+libkeyutils1:i386
+libnuma1:i386
+libssl-dev:i386
+libtirpc1:i386

.travis.packages_native new file +18 -0

new file mode 100644
index 000000000..d9e1a2acc
@@ -0,0 +1,18 @@
+libacl1
+libacl1-dev
+libaio-dev
+libaio1
+libcap-dev
+libcap2
+libc6
+libc6-dev
+libkeyutils-dev
+libkeyutils1
+libmm-dev
+libnuma-dev
+libnuma1
+libselinux1-dev
+libsepol1-dev
+libssl-dev
+libtirpc1
+linux-libc-dev

.travis.yml +34 ~~-10~~

111 lines not on this page · raw message

Signature · 1 line

2.15.0

Cyril Hrubis

2017-12-04 16:44 UTC #8 lore raw

v2 cover Travis enhancements

In reply to Petr Vorel

Hi!

Quoted 2 lines

It seems to be rule running only 5 containers at a time, so 11th build slowes building about 4 min (18 min vs. 22 min). Which build would you remove?

Well we do the build without any additional libs with gcc-7, we may as well switch it to clang-3.9 and drop the build with clang-3.9 with the additional libraries, that way we wouldn't lost much coverage.

Signature · 2 lines

Cyril Hrubis
[email protected]

Petr Vorel

author

2017-12-05 18:58 UTC #9 lore raw

v2 cover Travis enhancements

Hi Cyril,

Quoted 6 lines

Quoted 2 lines
It seems to be rule running only 5 containers at a time, so 11th build slowes building about 4 min (18 min vs. 22 min). Which build would you remove?
Well we do the build without any additional libs with gcc-7, we may as well switch it to clang-3.9 and drop the build with clang-3.9 with the additional libraries, that way we wouldn't lost much coverage.

gcc-7 is used for out-of-tree (gcc-6 was used in v3 for build with minimal dependencies) OK clang-3.9 is now used only for build with minimal dependencies and it's the only build of this type.
https://travis-ci.org/pevik-travis/ltp/builds/312014219

BTW: I'm planning to add also arm32 cross-compile build in near future. To keep only 10 jobs, I'll replace gcc-6 normal with all dependencies by it.

Kind regards,
Petr

Cyril Hrubis

2017-12-06 10:27 UTC #10 lore raw

v2 cover Travis enhancements

Hi!

Quoted 3 lines

BTW: I'm planning to add also arm32 cross-compile build in near future. To keep only 10 jobs, I'll replace gcc-6 normal with all dependencies by it.

Sounds good to me.

Signature · 2 lines

Cyril Hrubis
[email protected]