travis: enhancements for build (plus a meson fix)
cover travis: enhancements for build (plus a meson fix)
This series improves the travis integration a bit, and adds the ability for the meson system to auto-detect the number of cores.
Aaron Conole (3):
travis: enable ccache
travis: add a distinguisher to the 'extra' builds app/test/meson: auto detect number of cores
.travis.yml | 19 +++++++++---------- app/test/meson.build | 24 +++++++++++++++++++++--- 2 files changed, 30 insertions(+), 13 deletions(-)
Signature · 1 line
2.19.1
PATCH 1/3 travis: enable ccache
We try to make the planet happy travis builds dpdk.
Also, the 'sudo' flag was recently deprecated. Drop it.
--- .travis.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
.travis.yml +1 -2
@@ -1,4 +1,5 @@ language: c +cache: ccache compiler: - gcc - clang
@@ -24,8 +25,6 @@ extra_packages: &extra_packages before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh -sudo: false - env: - DEF_LIB="static" - DEF_LIB="shared"
Signature · 1 line
2.19.1
Re: [dpdk-dev] [PATCH 1/3] travis: enable ccache
Quoted 30 lines · Aaron Conole
We try to make the planet happy travis builds dpdk.
Also, the 'sudo' flag was recently deprecated. Drop it.
Suggested-by Luca Boccassi <[email protected]>Signed-off-by Aaron Conole <[email protected]>--- .travis.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-).travis.yml +1
-3@@ -1,4 +1,5 @@ language: c +cache: ccache compiler: - gcc - clang@@ -24,8 +25,6 @@ extra_packages: &extra_packages before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh -sudo: false - env: - DEF_LIB="static" - DEF_LIB="shared" --
Quoted 2 lines
2.19.1
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH 1/3] travis: enable ccache
In reply to Aaron ConoleQuoted 26 lines · Aaron Conole
We try to make the planet happy travis builds dpdk.
Also, the 'sudo' flag was recently deprecated. Drop it.
Suggested-by: Luca Boccassi <
[email protected]Quoted 1 line
Signed-off-by: Aaron Conole <
[email protected]Quoted 1 line
--- .travis.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-).travis.yml +1
-0@@ -1,4 +1,5 @@ language: c +cache: ccache compiler: - gcc - clang
.travis.yml extra_packages: &extra_packages · line 30
Quoted 8 lines
.travis.yml +0
-2@@ -24,8 +25,6 @@ extra_packages: &extra_packages before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh -sudo: false - env: - DEF_LIB="static" - DEF_LIB="shared"
Signature · 2 lines
Kind regards, Luca Boccassi
PATCH 2/3 travis: add a distinguisher to the 'extra' builds
In reply to Aaron ConoleThis helps in two ways:
1. When looking at travis page for dpdk, it's a visual distinction
2. For ccache support, the build IDs include the 'env', so we get unique cache data.
--- .travis.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
.travis.yml +8 -8
@@ -45,49 +45,49 @@ matrix: apt: packages: - *aarch64_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: clang addons: apt:
Signature · 1 line
2.19.1
Re: [dpdk-dev] [PATCH 2/3] travis: add a distinguisher to the 'extra' builds
Quoted 74 lines · Aaron Conole
This helps in two ways:
1. When looking at travis page for dpdk, it's a visual distinction2. For ccache support, the build IDs include the 'env', so we get unique cache data.
Signed-off-by Aaron Conole <[email protected]>--- .travis.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-).travis.yml +8
-9@@ -45,49 +45,49 @@ matrix: apt: packages: - *aarch64_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: clang addons: apt: --
Quoted 2 lines
2.19.1
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH 2/3] travis: add a distinguisher to the 'extra' builds
In reply to Aaron ConoleQuoted 76 lines · Aaron Conole
This helps in two ways:
1. When looking at travis page for dpdk, it's a visual distinction2. For ccache support, the build IDs include the 'env', so we get
unique
cache data.Signed-off-by: Aaron Conole <
[email protected]Quoted 1 line
--- .travis.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-).travis.yml +3
-3@@ -45,49 +45,49 @@ matrix: apt: packages: - *aarch64_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false"EXTRA_PACKAGES=1
compiler: gcc
addons:
apt:
packages:
-*extra_packages.travis.yml +1
-1- - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false"EXTRA_PACKAGES=1
compiler: gcc
addons:
apt:
packages:
-*extra_packages.travis.yml +3
-3- - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false"EXTRA_PACKAGES=1
compiler: clang
addons:
apt:
packages:
-*extra_packages.travis.yml +1
-1- - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false"
Quoted 5 lines
EXTRA_PACKAGES=1
compiler: clang
addons:
apt:
Signature · 2 lines
Kind regards, Luca Boccassi
PATCH 3/3 app/test/meson: auto detect number of cores
In reply to Aaron ConoleThe arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
---
Conflicts with http://patches.dpdk.org/patch/50850/
app/test/meson.build | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
app/test/meson.build +21 -3
@@ -344,17 +344,32 @@ if get_option('tests') timeout_seconds = 600 timeout_seconds_fast = 10 + # Retreive the number of CPU cores + num_cores = run_command('lscpu', '-p=cpu').stdout().strip().split('\n')[-1] + num_cores_arg = '-l 0-' + num_cores + + test_args = [num_cores_arg, '-n 4'] foreach arg : fast_parallel_test_names - test(arg, dpdk_test, - env : ['DPDK_TEST=' + arg], - args : ['-c f','-n 4', '--file-prefix=@0@'.format(arg)], + if host_machine.system() == 'linux' + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args + + ['--file-prefix=@0@'.format(arg)], + timeout : timeout_seconds_fast, + suite : 'fast-tests') + else + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, suite : 'fast-tests') + endif endforeach foreach arg : fast_non_parallel_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, is_parallel : false, suite : 'fast-tests')
@@ -363,6 +378,7 @@ if get_option('tests') foreach arg : perf_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'perf-tests')
@@ -371,6 +387,7 @@ if get_option('tests') foreach arg : driver_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'driver-tests')
@@ -379,6 +396,7 @@ if get_option('tests') foreach arg : dump_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'debug-tests')
Signature · 1 line
2.19.1
Re: [dpdk-dev] [PATCH 3/3] app/test/meson: auto detect number of cores
Quoted 23 lines · Aaron Conole
The arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
Signed-off-by Aaron Conole <[email protected]>---
Conflicts with http://patches.dpdk.org/patch/50850/app/test/meson.build | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)app/test/meson.build +0
-0
app/test/meson.build if get_option('tests') · line 348
Quoted 5 lines
app/test/meson.build +1
-0@@ -344,17 +344,32 @@ if get_option('tests') timeout_seconds = 600 timeout_seconds_fast = 10 + # Retreive the number of CPU cores
nit: Retrieve
Little concern here on the approach of getting the max available cpu index. If we have non contiguous cpus (let's say hyper threading is disabled), this won't work.
But we can just assume this won't happen for non regression setups (vms).
+ num_cores = run_command('lscpu',
Quoted 2 lines
'-p=cpu').stdout().strip().split('\n')[-1]
lscpu is a linux command afaik.
Maybe for FreeBSD:
root@freebsd-10:~ # sysctl dev.cpu |cut -d . -f 3 |sort |tail -1 3
Not sure if FreeBSD ensures that the keys names/objects won't change accross the versions.
+ num_cores_arg = '-l 0-' + num_cores
Quoted 1 line
+
Quoted 55 lines
app/test/meson.build +1
-3+ test_args = [num_cores_arg, '-n 4'] foreach arg : fast_parallel_test_names - test(arg, dpdk_test, - env : ['DPDK_TEST=' + arg], - args : ['-c f','-n 4', '--file-prefix=@0@'.format(arg)],
app/test/meson.build +5
-0+ if host_machine.system() == 'linux' + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args + + ['--file-prefix=@0@'.format(arg)],
app/test/meson.build +11
-1+ timeout : timeout_seconds_fast, + suite : 'fast-tests') + else + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, suite : 'fast-tests') + endif endforeach foreach arg : fast_non_parallel_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, is_parallel : false, suite : 'fast-tests')@@ -363,6 +378,7 @@ if get_option('tests') foreach arg : perf_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'perf-tests')@@ -371,6 +387,7 @@ if get_option('tests') foreach arg : driver_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'driver-tests')@@ -379,6 +396,7 @@ if get_option('tests') foreach arg : dump_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'debug-tests') --
Quoted 2 lines
2.19.1
The rest looks good to me.
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH 3/3] app/test/meson: auto detect number of cores
Quoted 36 lines · David Marchand
On Thu, Apr 11, 2019 at 9:52 PM Aaron Conole <[1][email protected]> wrote:
The arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'. Also include a fix to a reported issue with running on FreeBSD.Signed-off-by Aaron Conole <[2][email protected]>---
Conflicts with [3]http://patches.dpdk.org/patch/50850/
app/test/meson.build | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
diff --git a/app/test/meson.build b/app/test/meson.build index867cc5863..1010bfbc8100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -344,17 +344,32 @@ ifget_option('tests')
timeout_seconds= 600timeout_seconds_fast= 10
+ # Retreive the number of CPU coresnit: Retrieve
Little concern here on the approach of getting the max available cpu index.
If we have non contiguous cpus (let's say hyper threading is disabled), this won't work.
But we can just assume this won't happen for non regression setups (vms).+
num_cores= run_command('lscpu',
'-p=cpu').stdout().strip().split('\n')[-1]
Quoted 7 lines
lscpu is a linux command afaik.
Maybe for FreeBSD:
root@freebsd-10:~ # sysctl dev.cpu |cut -d . -f 3 |sort |tail -1 3
Not sure if FreeBSD ensures that the keys names/objects won't change accross the versions.
Very similar thoughs/concerns here. I think doing this per OS is probably safer in the long term - even though lscpu is available for FreeBSD as an extra package.
My suggestion: for FreeBSD, get the number of CPUs using
"/sbin/sysctl -n hw.ncpu"
For Linux, rather than using a "0-N" range, use output from "cat /sys/devices/system/cpu/present", which will ensure that it works even in the non-contiguous CPU numbering case. [Though I suspect we get non-contiguous numbers only in the case a core or two has been hotplugged out, I think - disabling hyperthreading won't leave gaps, just fewer cores]
/Bruce
Re: [dpdk-dev] [PATCH 3/3] app/test/meson: auto detect number of cores
In reply to Aaron ConoleQuoted 14 lines · Aaron Conole
The arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
Signed-off-by: Aaron Conole <
[email protected]Quoted 1 line
Quoted 8 lines
---
Conflicts with
http://patches.dpdk.org/patch/50850/app/test/meson.build | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
With the provision that the changes suggested by Bruce and David are applied,
Signature · 2 lines
Kind regards, Luca Boccassi
v2 cover travis: enhancements for build (plus a meson fix)
In reply to Aaron ConoleThis series improves the travis integration a bit, and adds the ability for the meson system to auto-detect the number of cores.
v1->v2 (all patch 3/3 changes):
* Fix a spelling mistake
* Add support for FreeBSD
* Include a default fallback
* Use a more robust core-mask argument source (rather than lscpu)
Aaron Conole (3):
travis: enable ccache
travis: add a distinguisher to the 'extra' builds app/test/meson: auto detect number of cores
.travis.yml | 19 +++++++++---------- app/test/meson.build | 35 ++++++++++++++++++++++++++++++++--- 2 files changed, 41 insertions(+), 13 deletions(-)
Signature · 1 line
2.19.1
PATCH v2 1/3 travis: enable ccache
We try to make the planet happy travis builds dpdk.
Also, the 'sudo' flag was recently deprecated. Drop it.
--- .travis.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
.travis.yml +1 -2
@@ -1,4 +1,5 @@ language: c +cache: ccache compiler: - gcc - clang
@@ -24,8 +25,6 @@ extra_packages: &extra_packages before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh -sudo: false - env: - DEF_LIB="static" - DEF_LIB="shared"
Signature · 1 line
2.19.1
PATCH v2 2/3 travis: add a distinguisher to the 'extra' builds
In reply to Aaron ConoleThis helps in two ways:
1. When looking at travis page for dpdk, it's a visual distinction
2. For ccache support, the build IDs include the 'env', so we get unique cache data.
--- .travis.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
.travis.yml +8 -8
@@ -45,49 +45,49 @@ matrix: apt: packages: - *aarch64_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: gcc addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" + - env: DEF_LIB="static" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="static" OPTS="-Denable_kmods=false" + - env: DEF_LIB="static" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: clang addons: apt: packages: - *extra_packages - - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" + - env: DEF_LIB="shared" OPTS="-Denable_kmods=false" EXTRA_PACKAGES=1 compiler: clang addons: apt:
Signature · 1 line
2.19.1
PATCH v2 3/3 app/test/meson: auto detect number of cores
In reply to Aaron ConoleThe arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
---
v2:
* Fix a spelling mistake
* Add support for FreeBSD
* Include a default fallback
* Use a more robust core-mask argument source (rather than lscpu)
Conflicts with http://patches.dpdk.org/patch/50850/
app/test/meson.build | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)
app/test/meson.build +32 -3
@@ -344,17 +344,43 @@ if get_option('tests') timeout_seconds = 600 timeout_seconds_fast = 10 + # Retrieve the number of CPU cores, defaulting to 4. + num_cores = '0-3' + if host_machine.system() == 'linux' + num_cores = run_command('cat', + '/sys/devices/system/cpu/present' + ).stdout().strip() + elif host_machine.system() == 'freebsd' + snum_cores = run_command('/sbin/sysctl', '-n', + 'hw.ncpu').stdout().strip() + inum_cores = snum_cores.to_int() - 1 + num_cores = '0-@0@'.format(inum_cores) + endif + + num_cores_arg = '-l ' + num_cores + + test_args = [num_cores_arg, '-n 4'] foreach arg : fast_parallel_test_names - test(arg, dpdk_test, - env : ['DPDK_TEST=' + arg], - args : ['-c f','-n 4', '--file-prefix=@0@'.format(arg)], + if host_machine.system() == 'linux' + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args + + ['--file-prefix=@0@'.format(arg)], + timeout : timeout_seconds_fast, + suite : 'fast-tests') + else + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, suite : 'fast-tests') + endif endforeach foreach arg : fast_non_parallel_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, is_parallel : false, suite : 'fast-tests')
@@ -363,6 +389,7 @@ if get_option('tests') foreach arg : perf_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'perf-tests')
@@ -371,6 +398,7 @@ if get_option('tests') foreach arg : driver_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'driver-tests')
@@ -379,6 +407,7 @@ if get_option('tests') foreach arg : dump_test_names test(arg, dpdk_test, env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds, is_parallel : false, suite : 'debug-tests')
Signature · 1 line
2.19.1
Re: [dpdk-dev] [PATCH v2 3/3] app/test/meson: auto detect number of cores
Quoted 40 lines · Aaron Conole
The arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
Signed-off-by Aaron Conole <[email protected]>Reviewed-by David Marchand <[email protected]>Acked-by Luca Boccassi <[email protected]>---
v2:
* Fix a spelling mistake
* Add support for FreeBSD
* Include a default fallback
* Use a more robust core-mask argument source (rather than lscpu)Conflicts with http://patches.dpdk.org/patch/50850/
app/test/meson.build | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)app/test/meson.build +6
-0@@ -344,17 +344,43 @@ if get_option('tests') timeout_seconds = 600 timeout_seconds_fast = 10 + # Retrieve the number of CPU cores, defaulting to 4. + num_cores = '0-3' + if host_machine.system() == 'linux' + num_cores = run_command('cat', + '/sys/devices/system/cpu/present' + ).stdout().strip()
app/test/meson.build if get_option('tests') · line 362
Quoted 10 lines
app/test/meson.build +10
-0+ elif host_machine.system() == 'freebsd' + snum_cores = run_command('/sbin/sysctl', '-n', + 'hw.ncpu').stdout().strip() + inum_cores = snum_cores.to_int() - 1 + num_cores = '0-@0@'.format(inum_cores) + endif + + num_cores_arg = '-l ' + num_cores + + test_args = [num_cores_arg, '-n 4']
This -n 4 parameter can be dropped. Four is the default setting IIRC. I also wonder are the parameters coming through to the app correctly, generally meson does not work well with parameters with spaces in them - I'd expect the "-l" and the num_cores values to be separated in the array. I also think num_cores_arg value could be dropped too.
If it works though, I'm ok to keep as-is though.
Quoted 9 lines
app/test/meson.build +5
-3foreach arg : fast_parallel_test_names - test(arg, dpdk_test, - env : ['DPDK_TEST=' + arg], - args : ['-c f','-n 4', '--file-prefix=@0@'.format(arg)], + if host_machine.system() == 'linux' + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args + + ['--file-prefix=@0@'.format(arg)],
app/test/meson.build if get_option('tests') · line 378
Quoted 10 lines
app/test/meson.build +7
-0+ timeout : timeout_seconds_fast, + suite : 'fast-tests') + else + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args, timeout : timeout_seconds_fast, suite : 'fast-tests') + endif endforeach
While this is needed now, I think in the medium term we should have the "file-prefix" flag being a warning rather than a hard-error on FreeBSD. [i.e. keep this, but we should fix it in 19.08 to be shorter]
/Bruce
Re: [dpdk-dev] [PATCH v2 3/3] app/test/meson: auto detect number of cores
Quoted 45 lines · Bruce Richardson
Quoted 44 lines · Aaron Conole
The arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
Signed-off-by Aaron Conole <[email protected]>Reviewed-by David Marchand <[email protected]>Acked-by Luca Boccassi <[email protected]>---
v2:
* Fix a spelling mistake
* Add support for FreeBSD
* Include a default fallback
* Use a more robust core-mask argument source (rather than lscpu)Conflicts with http://patches.dpdk.org/patch/50850/
app/test/meson.build | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)app/test/meson.build +10
-0@@ -344,17 +344,43 @@ if get_option('tests') timeout_seconds = 600 timeout_seconds_fast = 10 + # Retrieve the number of CPU cores, defaulting to 4. + num_cores = '0-3' + if host_machine.system() == 'linux' + num_cores = run_command('cat', + '/sys/devices/system/cpu/present' + ).stdout().strip() + elif host_machine.system() == 'freebsd' + snum_cores = run_command('/sbin/sysctl', '-n', + 'hw.ncpu').stdout().strip() + inum_cores = snum_cores.to_int() - 1
Quoted 8 lines
app/test/meson.build if get_option('tests') · line 362
Quoted 6 lines
app/test/meson.build +6
-0+ num_cores = '0-@0@'.format(inum_cores) + endif + + num_cores_arg = '-l ' + num_cores + + test_args = [num_cores_arg, '-n 4']This -n 4 parameter can be dropped. Four is the default setting IIRC.
For another patch. I thought about doing it with this one, but I'd rather keep the changes a little bit traceable. If you think I should resubmit with it dropped, I will.
Quoted 6 lines
I also wonder are the parameters coming through to the app correctly, generally meson does not work well with parameters with spaces in them - I'd expect the "-l" and the
num_coresvalues to be separated in the array. I also thinknum_cores_argvalue could be dropped too.If it works though, I'm ok to keep as-is though.
It does work. Actually, I needed to change because on some of the VM setups (including the one used by Travis) the '-c f' arg errors because it wants 4 cores, and only 2 exist. Either way, this patch doesn't change the spacing being passed with args :)
Quoted 15 lines
Quoted 15 lines
app/test/meson.build +11
-3foreach arg : fast_parallel_test_names - test(arg, dpdk_test, - env : ['DPDK_TEST=' + arg], - args : ['-c f','-n 4', '--file-prefix=@0@'.format(arg)], + if host_machine.system() == 'linux' + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args + + ['--file-prefix=@0@'.format(arg)], + timeout : timeout_seconds_fast, + suite : 'fast-tests') + else + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args,
Quoted 8 lines
app/test/meson.build if get_option('tests') · line 378
Quoted 4 lines
app/test/meson.build +1
-0timeout : timeout_seconds_fast, suite : 'fast-tests') + endif endforeachWhile this is needed now, I think in the medium term we should have the "file-prefix" flag being a warning rather than a hard-error on FreeBSD. [i.e. keep this, but we should fix it in 19.08 to be shorter]
Agreed. Probably a good cleanup in the future.
Quoted 1 line
/Bruce
Bruce Richardson
Acked-byRe: [dpdk-dev] [PATCH v2 3/3] app/test/meson: auto detect number of cores
Quoted 96 lines · Aaron Conole
Quoted 45 lines · Bruce Richardson
Quoted 44 lines · Aaron Conole
The arguments being passed will cause failures on laptops that have, for instance, 2 cores only. Most of the tests don't require more than a single core. Some require multiple cores (but those tests should be modified to 'SKIP' when the correct number of cores aren't available).
The unit test results shouldn't be impacted by this change, but it allows for a future enhancement to pass flags such as '--no-huge'.
Also include a fix to a reported issue with running on FreeBSD.
Signed-off-by Aaron Conole <[email protected]>Reviewed-by David Marchand <[email protected]>Acked-by Luca Boccassi <[email protected]>---
v2:
* Fix a spelling mistake
* Add support for FreeBSD
* Include a default fallback
* Use a more robust core-mask argument source (rather than lscpu)Conflicts with http://patches.dpdk.org/patch/50850/
app/test/meson.build | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)app/test/meson.build +10
-0@@ -344,17 +344,43 @@ if get_option('tests') timeout_seconds = 600 timeout_seconds_fast = 10 + # Retrieve the number of CPU cores, defaulting to 4. + num_cores = '0-3' + if host_machine.system() == 'linux' + num_cores = run_command('cat', + '/sys/devices/system/cpu/present' + ).stdout().strip() + elif host_machine.system() == 'freebsd' + snum_cores = run_command('/sbin/sysctl', '-n', + 'hw.ncpu').stdout().strip() + inum_cores = snum_cores.to_int() - 1Quoted 8 lines
app/test/meson.build if get_option('tests') · line 362
Quoted 6 lines
app/test/meson.build +6
-0+ num_cores = '0-@0@'.format(inum_cores) + endif + + num_cores_arg = '-l ' + num_cores + + test_args = [num_cores_arg, '-n 4']This -n 4 parameter can be dropped. Four is the default setting IIRC.
For another patch. I thought about doing it with this one, but I'd rather keep the changes a little bit traceable. If you think I should resubmit with it dropped, I will.
Quoted 6 lines
I also wonder are the parameters coming through to the app correctly, generally meson does not work well with parameters with spaces in them - I'd expect the "-l" and the
num_coresvalues to be separated in the array. I also thinknum_cores_argvalue could be dropped too.If it works though, I'm ok to keep as-is though.
It does work. Actually, I needed to change because on some of the VM setups (including the one used by Travis) the '-c f' arg errors because it wants 4 cores, and only 2 exist. Either way, this patch doesn't change the spacing being passed with args :)
Quoted 15 lines
Quoted 15 lines
app/test/meson.build +11
-3foreach arg : fast_parallel_test_names - test(arg, dpdk_test, - env : ['DPDK_TEST=' + arg], - args : ['-c f','-n 4', '--file-prefix=@0@'.format(arg)], + if host_machine.system() == 'linux' + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args + + ['--file-prefix=@0@'.format(arg)], + timeout : timeout_seconds_fast, + suite : 'fast-tests') + else + test(arg, dpdk_test, + env : ['DPDK_TEST=' + arg], + args : test_args,Quoted 8 lines
app/test/meson.build if get_option('tests') · line 378
Quoted 4 lines
app/test/meson.build +1
-0timeout : timeout_seconds_fast, suite : 'fast-tests') + endif endforeachWhile this is needed now, I think in the medium term we should have the "file-prefix" flag being a warning rather than a hard-error on FreeBSD. [i.e. keep this, but we should fix it in 19.08 to be shorter]
Quoted 2 lines
Agreed. Probably a good cleanup in the future.
Fair responses to all comments.
Thomas Monjalon
Applied12/04/2019 18:21, Aaron Conole:
Quoted 9 lines
This series improves the travis integration a bit, and adds the ability for the meson system to auto-detect the number of cores.
v1->v2(all patch 3/3 changes):
* Fix a spelling mistake
* Add support for FreeBSD
* Include a default fallback
* Use a more robust core-mask argument source (rather than lscpu)
Quoted 4 lines
Aaron Conole (3):
travis: enable ccache
travis: add a distinguisher to the 'extra' builds app/test/meson: auto detect number of cores
Applied, thanks
For info, we can also cache the build directory with
cache:
directories:
- path/to/meson/build/dir
See https://docs.travis-ci.com/user/caching/#arbitrary-directories.