Resolve several travis-ci issues

PATCH In review Ilya Shipitsin 2 participants 5 messages Jan 28, 2017

Share & tools

Ilya Shipitsin

author

2017-01-28 08:06 UTC #0 lore raw

PATCH 1/1 Resolve several travis-ci issues

MBEDTLS_VERSION, OPENSSL_VERSION were defined twice - in both
.travis.yml and .travis/build-deps.sh files, the last one
defined OPENSSL_VERSION via nonexistent OPENSSL_VERION variable, which lead us to use openssl-1.0.1 instead of openssl-1.0.2, I removed variable definition from build-deps.sh

"cache: [ apt: true ]" is not a travis supported option, it was introduced by mistake, I removed it

LD_LIBRARY_PATH was defined for the entire test run, it includes custom openssl build, which was picked by "wget", so "wget" could not verify SSL cert at https://www.openssl.org sometimes. We do not want wget to pick our custom LD_LIBRARY_PATH, so I moved that variable to "script" section

LD_LIBRARY_PATH was defined for both linux and osx environments, for the second DYLD_LIBRARY_PATH must be defined instead

v2: Upgrade openssl, mbedtls to the most recent versions

---
 .travis.yml           | 8 ++++----
 .travis/build-deps.sh | 2 --
 2 files changed, 4 insertions(+), 6 deletions(-)

.travis.yml +4 -4

index 369db97..9741d6d 100644
@@ -12,13 +12,12 @@ env:
   global:
     - JOBS=3
     - PREFIX="${HOME}/opt"
-    - MBEDTLS_VERSION="2.2.1"
+    - MBEDTLS_VERSION="2.4.0"
     - MBEDTLS_CFLAGS="-I${PREFIX}/include"
     - MBEDTLS_LIBS="-L${PREFIX}/lib -lmbedtls -lmbedx509 -lmbedcrypto"
-    - OPENSSL_VERSION="1.0.1t"
+    - OPENSSL_VERSION="1.0.2k"
     - OPENSSL_CFLAGS="-I${PREFIX}/include"
     - OPENSSL_LIBS="-L${PREFIX}/lib -lssl -lcrypto"
-    - LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"
 
 matrix:
   include:

@@ -63,7 +62,6 @@ addons:
       - linux-libc-dev
 
 cache:
-  apt: true
   ccache: true
   directories:
   - download-cache

@@ -77,6 +75,8 @@ install:
   - .travis/build-deps.sh > build-deps.log 2>&1 || (cat build-deps.log && exit 1)
 
 script:
+  - if [ "${TRAVIS_OS_NAME}" = "linux" ]; then export LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
+  - if [ "${TRAVIS_OS_NAME}" = "osx"   ]; then export DYLD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
   - autoreconf -vi
   - ./configure --with-crypto-library="${SSLLIB}" ${EXTRA_CONFIG} || (cat config.log && exit 1)
   - make -j$JOBS

.travis/build-deps.sh +0 -2

index bda54ee..3ffba0b 100755
@@ -2,8 +2,6 @@
 set -eux
 
 # Set defaults
-MBEDTLS_VERSION="${MBEDTLS_VERSION:-2.2.1}"
-OPENSSL_VERSION="${OPENSSL_VERION:-1.0.2h}"
 PREFIX="${PREFIX:-${HOME}/opt}"
 
 download_mbedtls () {

Signature · 1 line

2.5.5

Илья Шипицин

author

2017-01-28 08:07 UTC #1 lore raw

this one is final v2 patch

2017-01-28 13:06 GMT+05:00 Ilya Shipitsin <[email protected]...>:

Quoted 82 lines

MBEDTLS_VERSION, OPENSSL_VERSION were defined twice - in both
.travis.yml and .travis/build-deps.sh files, the last one
defined OPENSSL_VERSION via nonexistent OPENSSL_VERION variable, which lead us to use openssl-1.0.1 instead of openssl-1.0.2, I removed variable definition from build-deps.sh
"cache: [ apt: true ]" is not a travis supported option, it was introduced by mistake, I removed it
LD_LIBRARY_PATH was defined for the entire test run, it includes custom openssl build, which was picked by "wget", so "wget" could not verify SSL cert at https://www.openssl.org sometimes. We do not want wget to pick our custom LD_LIBRARY_PATH, so I moved that variable to "script" section
LD_LIBRARY_PATH was defined for both linux and osx environments, for the second DYLD_LIBRARY_PATH must be defined instead
v2: Upgrade openssl, mbedtls to the most recent versions
---
 .travis.yml           | 8 ++++----
 .travis/build-deps.sh | 2 --
 2 files changed, 4 insertions(+), 6 deletions(-)
.travis.yml +3 -4
index 369db97..9741d6d 100644
@@ -12,13 +12,12 @@ env:
   global:
     - JOBS=3
     - PREFIX="${HOME}/opt"
-    - MBEDTLS_VERSION="2.2.1"
+    - MBEDTLS_VERSION="2.4.0"
     - MBEDTLS_CFLAGS="-I${PREFIX}/include"
     - MBEDTLS_LIBS="-L${PREFIX}/lib -lmbedtls -lmbedx509 -lmbedcrypto"
-    - OPENSSL_VERSION="1.0.1t"
+    - OPENSSL_VERSION="1.0.2k"
     - OPENSSL_CFLAGS="-I${PREFIX}/include"
     - OPENSSL_LIBS="-L${PREFIX}/lib -lssl -lcrypto"
-    - LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"
 
 matrix:
   include:
@@ -63,7 +62,6 @@ addons:
       - linux-libc-dev
 
 cache:
-  apt: true
   ccache: true
   directories:
   - download-cache
@@ -77,6 +75,8 @@ install:
   - .travis/build-deps.sh > build-deps.log 2>&1 || (cat build-deps.log &&
 exit 1)
 
 script:
+  - if [ "${TRAVIS_OS_NAME}" = "linux" ]; then export
LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
+ - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then export
DYLD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
- autoreconf -vi
- ./configure --with-crypto-library="${SSLLIB}" ${EXTRA_CONFIG} ||
(cat config.log && exit 1)
- make -j$JOBS
.travis/build-deps.sh +0 -3
index bda54ee..3ffba0b 100755
@@ -2,8 +2,6 @@
 set -eux
 
 # Set defaults
-MBEDTLS_VERSION="${MBEDTLS_VERSION:-2.2.1}"
-OPENSSL_VERSION="${OPENSSL_VERION:-1.0.2h}"
 PREFIX="${PREFIX:-${HOME}/opt}"
 
 download_mbedtls () {
--
2.5.5

Илья Шипицин

author

2017-01-28 10:18 UTC #2 lore raw

forgot to mention,
this patch might be applied to both master and release/2.4 branches

2017-01-28 13:07 GMT+05:00 Илья Шипицин <[email protected]...>:

Quoted 87 lines

this one is final v2 patch
2017-01-28 13:06 GMT+05:00 Ilya Shipitsin <[email protected]...>:
Quoted 82 lines
MBEDTLS_VERSION, OPENSSL_VERSION were defined twice - in both
.travis.yml and .travis/build-deps.sh files, the last one
defined OPENSSL_VERSION via nonexistent OPENSSL_VERION variable, which lead us to use openssl-1.0.1 instead of openssl-1.0.2, I removed variable definition from build-deps.sh
"cache: [ apt: true ]" is not a travis supported option, it was introduced by mistake, I removed it
LD_LIBRARY_PATH was defined for the entire test run, it includes custom openssl build, which was picked by "wget", so "wget" could not verify SSL cert at https://www.openssl.org sometimes. We do not want wget to pick our custom LD_LIBRARY_PATH, so I moved that variable to "script" section
LD_LIBRARY_PATH was defined for both linux and osx environments, for the second DYLD_LIBRARY_PATH must be defined instead
v2: Upgrade openssl, mbedtls to the most recent versions
---
 .travis.yml           | 8 ++++----
 .travis/build-deps.sh | 2 --
 2 files changed, 4 insertions(+), 6 deletions(-)
.travis.yml +3 -4
index 369db97..9741d6d 100644
@@ -12,13 +12,12 @@ env:
   global:
     - JOBS=3
     - PREFIX="${HOME}/opt"
-    - MBEDTLS_VERSION="2.2.1"
+    - MBEDTLS_VERSION="2.4.0"
     - MBEDTLS_CFLAGS="-I${PREFIX}/include"
     - MBEDTLS_LIBS="-L${PREFIX}/lib -lmbedtls -lmbedx509 -lmbedcrypto"
-    - OPENSSL_VERSION="1.0.1t"
+    - OPENSSL_VERSION="1.0.2k"
     - OPENSSL_CFLAGS="-I${PREFIX}/include"
     - OPENSSL_LIBS="-L${PREFIX}/lib -lssl -lcrypto"
-    - LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"
 
 matrix:
   include:
@@ -63,7 +62,6 @@ addons:
       - linux-libc-dev
 
 cache:
-  apt: true
   ccache: true
   directories:
   - download-cache
@@ -77,6 +75,8 @@ install:
   - .travis/build-deps.sh > build-deps.log 2>&1 || (cat build-deps.log
 && exit 1)
 
 script:
+  - if [ "${TRAVIS_OS_NAME}" = "linux" ]; then export
LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
+ - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then export
DYLD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
- autoreconf -vi
- ./configure --with-crypto-library="${SSLLIB}" ${EXTRA_CONFIG} ||
(cat config.log && exit 1)
- make -j$JOBS
.travis/build-deps.sh +0 -3
index bda54ee..3ffba0b 100755
@@ -2,8 +2,6 @@
 set -eux
 
 # Set defaults
-MBEDTLS_VERSION="${MBEDTLS_VERSION:-2.2.1}"
-OPENSSL_VERSION="${OPENSSL_VERION:-1.0.2h}"
 PREFIX="${PREFIX:-${HOME}/opt}"
 
 download_mbedtls () {
--
2.5.5

Selva Nair

2017-01-28 20:29 UTC #3 lore raw

In reply to Ilya Shipitsin

Hi,

Quoted 15 lines · Ilya Shipitsin

MBEDTLS_VERSION, OPENSSL_VERSION were defined twice - in both
.travis.yml and .travis/build-deps.sh files, the last one
defined OPENSSL_VERSION via nonexistent OPENSSL_VERION variable, which lead us to use openssl-1.0.1 instead of openssl-1.0.2, I removed variable definition from build-deps.sh
"cache: [ apt: true ]" is not a travis supported option, it was introduced by mistake, I removed it
LD_LIBRARY_PATH was defined for the entire test run, it includes custom openssl build, which was picked by "wget", so "wget" could not verify SSL cert at https://www.openssl.org sometimes. We do not want wget to pick our custom LD_LIBRARY_PATH, so I moved that variable to "script" section

Quoted 9 lines

LD_LIBRARY_PATH was defined for both linux and osx environments, for the second DYLD_LIBRARY_PATH must be defined instead
v2: Upgrade openssl, mbedtls to the most recent versions
---
 .travis.yml           | 8 ++++----
 .travis/build-deps.sh | 2 --
 2 files changed, 4 insertions(+), 6 deletions(-)

Moving down the LD_LIBRARY_PATH definition makes sense -- we need it (more on that below) in the env to run the test executables, but do not want it to modify the library search path for installed programs used for package updates etc.

script:

Quoted 4 lines

+ - if [ "${TRAVIS_OS_NAME}" = "linux" ]; then export LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
+ - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then export
DYLD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi

On this platform DYLD_LIBRARY_PATH is what matters, so I think we should be appending to it instead of redefining it in terms of LD_LIBRARY_PATH. That is,

DYLD_LIBRARY_PATH="${PREFIX}/lib:${DYLD_LIBRARY_PATH}"

While this patch is good enough for travis builds, there is a more general issue here. I think its the job of libtool to set -rpath properly (and/or use a wrapper) so that uninstalled test programs pick up the library it was linked with. Else a developer building using custom libs gets wrong libraries loaded while running tests. A proper fix will eliminate the need for tweaking LD_LIBRARY_PATH in the first place. But my libtool foo is virtually nil to suggest a proper solution.

Selva

Илья Шипицин

author

2017-01-28 21:13 UTC #4 lore raw

2017-01-29 1:29 GMT+05:00 Selva Nair <[email protected]...>:

Quoted 42 lines

Hi,
Quoted 15 lines · Ilya Shipitsin
MBEDTLS_VERSION, OPENSSL_VERSION were defined twice - in both
.travis.yml and .travis/build-deps.sh files, the last one
defined OPENSSL_VERSION via nonexistent OPENSSL_VERION variable, which lead us to use openssl-1.0.1 instead of openssl-1.0.2, I removed variable definition from build-deps.sh
"cache: [ apt: true ]" is not a travis supported option, it was introduced by mistake, I removed it
LD_LIBRARY_PATH was defined for the entire test run, it includes custom openssl build, which was picked by "wget", so "wget" could not verify SSL cert at https://www.openssl.org sometimes. We do not want wget to pick our custom LD_LIBRARY_PATH, so I moved that variable to "script" section
Quoted 9 lines
LD_LIBRARY_PATH was defined for both linux and osx environments, for the second DYLD_LIBRARY_PATH must be defined instead
v2: Upgrade openssl, mbedtls to the most recent versions
---
 .travis.yml           | 8 ++++----
 .travis/build-deps.sh | 2 --
 2 files changed, 4 insertions(+), 6 deletions(-)
Moving down the LD_LIBRARY_PATH definition makes sense -- we need it (more on that below) in the env to run the test executables, but do not want it to modify the library search path for installed programs used for package updates etc.
script:
Quoted 4 lines
+ - if [ "${TRAVIS_OS_NAME}" = "linux" ]; then export LD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi
+ - if [ "${TRAVIS_OS_NAME}" = "osx" ]; then export
DYLD_LIBRARY_PATH="${PREFIX}/lib:${LD_LIBRARY_PATH}"; fi

Quoted 6 lines

On this platform DYLD_LIBRARY_PATH is what matters, so I think we should be appending to it instead of redefining it in terms of LD_LIBRARY_PATH. That is,
DYLD_LIBRARY_PATH="${PREFIX}/lib:${DYLD_LIBRARY_PATH}"

oops...

I will send "v3" tomorrow

Quoted 9 lines

While this patch is good enough for travis builds, there is a more general issue here. I think its the job of libtool to set -rpath properly (and/or use a wrapper) so that uninstalled test programs pick up the library it was linked with. Else a developer building using custom libs gets wrong libraries loaded while running tests. A proper fix will eliminate the need for tweaking LD_LIBRARY_PATH in the first place. But my libtool foo is virtually nil to suggest a proper solution.

I think that "move down" is a quick and dirty fix rather than proper solution. We'll resolve it ... when I will have an idea how to do that.

https://travis-ci.org/OpenVPN/openvpn/jobs/195893699#L1372-L1377

also, we should compile openssl more carefully, we do need it for cmocka tests, and currently cmocka is not happy about our openssl build (I guess we should compare "configure" params of shipped openssl and our custom openssl)

Quoted 3 lines

Selva