make-flight job filter cleanups
cover make-flight job filter cleanups
The mechanisms for omitting certain archs from jobs are inconsistently applied to build vs test jobs.
This series tries to tidy that up somewhat.
I've check the output of standalone-generate-dump-flight-runvars at each step (i.e. covering the main make-flight).
I also did a before and after the entire flight check for the branches handled by make-distro-flight and there was no difference (as expected).
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
PATCH 1/5 mg-adjust-flight-makexrefs: Make a nop in Standalone mode
It is not useful to try and drop jobs based on scrobbling around in the standalone history.
This makes it possible to "./standalone make-flight xen-unstable-smoke" and fixes standalone-generate-dump-flight-runvars
--- mg-adjust-flight-makexrefs | 7 +++++++ 1 file changed, 7 insertions(+)
mg-adjust-flight-makexrefs +7 -0
@@ -42,6 +42,13 @@ done if [ $# -le 2 ]; then badusage; fi +. ./cri-getconfig + +jobdb=`getconfig JobDB` +if [ "$jobdb" = Standalone ]; then + exit 0 +fi + flight=$1; shift keepjobs=$1; shift
Signature · 1 line
2.6.1
PATCH 2/5 mfi-common: Drop armhf build jobs on older linux-* branches
In reply to Ian Campbelltest_matrix_branch_filter_callback was recently updated to exclude testing of linux-3.10 (54f237784d4b) and 3.14 (b0c5663a03e7), however this only excludes test and not build jobs so we were still trying to build the arm kernel there.
This changes the build job filtering to be in sync with the test job filtering, the net result is to remove build-armhf* from linux-3.10 and linux-3.14.
A subsequent patch will try and combine those two filters into one to prevent this skew happening again.
--- mfi-common | 2 ++ 1 file changed, 2 insertions(+)
mfi-common +2 -0
@@ -94,6 +94,8 @@ create_build_jobs () { case "$branch" in linux-3.0) continue;; linux-3.4) continue;; + linux-3.10) continue;; + linux-3.14) continue;; linux-mingo-tip-master) continue;; linux-*) ;; qemu-upstream-4.2-testing) continue;;
Signature · 1 line
2.6.1
PATCH 3/5 make-*flight: Reorder.
In reply to Ian CampbellI just got tripped up again by putting a build job filter definition after the call to create_build_jobs. Reorder the make-*flight scripts to reduce the probability of me doing so any more times.
The general order of these scripts is now:
- job filter callbacks
- test job creation
- top-level code which drives the process.
No change to the output of standalone-generate-dump-flight-runvars.
--- make-distros-flight | 24 ++++++++++++------------ make-flight | 20 ++++++++++---------- 2 files changed, 22 insertions(+), 22 deletions(-)
make-distros-flight +12 -12
@@ -50,18 +50,6 @@ job_create_build_filter_callback () { return 0 } -if [ x$buildflight = x ]; then - - WANT_XEND=false REVISION_LINUX_OLD=disable - - create_build_jobs - -else - - bfi=$buildflight. - -fi - job_create_test_filter_callback () { if [ "$xenarch" = "i386" ]; then return 1; fi return 0
@@ -201,6 +189,18 @@ test_matrix_do_one () { done } +if [ x$buildflight = x ]; then + + WANT_XEND=false REVISION_LINUX_OLD=disable + + create_build_jobs + +else + + bfi=$buildflight. + +fi + test_matrix_iterate echo $flight
make-flight +10 -10
@@ -58,16 +58,6 @@ job_create_build_filter_callback () { return 0 } -if [ x$buildflight = x ]; then - - create_build_jobs - -else - - bfi=$buildflight. - -fi - job_create_test_filter_callback () { local job=$1; shift local recipe=$1; shift
@@ -674,6 +664,16 @@ test_matrix_do_one () { do_pvgrub_tests } +if [ x$buildflight = x ]; then + + create_build_jobs + +else + + bfi=$buildflight. + +fi + test_matrix_iterate echo $flight
Signature · 1 line
2.6.1
PATCH 4/5 make-flight: consolidate branch_filter_callback for builds and tests
In reply to Ian CampbellCurrently we have a test_matrix_branch_filter_callback which filters jobs based on the $xenarch and $branch and a separate more adhoc filter inline for the build jobs.
This has lead to things getting out of sync in the past (e.g. recently we dropped armhf tests from the linux-3.10 and -3.14 branches but not the build jobs).
Add a new build_matrix_branch_filter_callback and for make-flight cause this and test_matrix_branch_filter_callback to use a common helper.
The adhoc filtering in the build loop remains and will be tidied up next.
For make-distros-flight just add a nop build filter alongside the test filter.
No change to the output of standalone-generate-dump-flight-runvars.
--- make-distros-flight | 4 ++++ make-flight | 14 ++++++++++++-- mfi-common | 2 ++ 3 files changed, 18 insertions(+), 2 deletions(-)
make-distros-flight +4 -0
@@ -55,6 +55,10 @@ job_create_test_filter_callback () { return 0 } +build_matrix_branch_filter_callback () { + : +} + test_matrix_branch_filter_callback () { : }
make-flight +12 -2
@@ -133,8 +133,9 @@ job_create_test_filter_callback () { return 0; } -test_matrix_branch_filter_callback () { - case "$xenarch" in +arch_branch_filter_callback () { + local arch=$1 + case "$arch" in armhf) case "$branch" in linux-3.0) return 1;;
@@ -154,7 +155,16 @@ test_matrix_branch_filter_callback () { esac ;; esac + return 0 +} +build_matrix_branch_filter_callback () { + if ! arch_branch_filter_callback $arch ; then return 1 ; fi + return 0 +} + +test_matrix_branch_filter_callback () { + if ! arch_branch_filter_callback $xenarch ; then return 1 ; fi return 0 }
mfi-common +2 -0
@@ -89,6 +89,8 @@ create_build_jobs () { if [ "x$arch" = xdisable ]; then continue; fi + build_matrix_branch_filter_callback || continue + case "$arch" in armhf) case "$branch" in
Signature · 1 line
2.6.1
PATCH 5/5 make-flight: move in-lined branch vs arch filtering into callbacks
In reply to Ian CampbellNo change to the output of standalone-generate-dump-flight-runvars
The inlined xenbranch vs arch filters remain where they are since they are common (in that they reflect the addition and removal of arches) and apply equally to all make-*-flight.
--- make-flight | 3 +++ mfi-common | 17 ----------------- 2 files changed, 3 insertions(+), 17 deletions(-)
make-flight +3 -0
@@ -147,6 +147,9 @@ arch_branch_filter_callback () { qemu-upstream-4.2-testing) return 1;; qemu-upstream-4.3-testing) return 1;; qemu-upstream-4.4-testing) return 1;; + rumpuserxen) return 1;; + seabios) return 1;; + ovmf) return 1;; esac ;; i386|amd64)
mfi-common +0 -17
@@ -93,20 +93,6 @@ create_build_jobs () { case "$arch" in armhf) - case "$branch" in - linux-3.0) continue;; - linux-3.4) continue;; - linux-3.10) continue;; - linux-3.14) continue;; - linux-mingo-tip-master) continue;; - linux-*) ;; - qemu-upstream-4.2-testing) continue;; - qemu-upstream-4.3-testing) continue;; - qemu-upstream-4.4-testing) continue;; - rumpuserxen) continue;; - seabios) continue;; - ovmf) continue;; - esac case "$xenbranch" in xen-3.*-testing) continue;; xen-4.0-testing) continue;;
@@ -122,9 +108,6 @@ create_build_jobs () { " ;; *) - case "$branch" in - linux-arm-xen) continue;; - esac pvops_kernel=" tree_linux=$TREE_LINUX revision_linux=${REVISION_LINUX:-${DEFAULT_REVISION_LINUX}}
Signature · 1 line
2.6.1
Re: [PATCH OSSTEST 1/5] mg-adjust-flight-makexrefs: Make a nop in Standalone mode
In reply to Ian CampbellIan Campbell writes ("[PATCH OSSTEST 1/5] mg-adjust-flight-makexrefs: Make a nop in Standalone mode"):
Quoted 5 lines
It is not useful to try and drop jobs based on scrobbling around in the standalone history.
This makes it possible to "./standalone make-flight xen-unstable-smoke" and fixes standalone-generate-dump-flight-runvars
Re: [PATCH OSSTEST 2/5] mfi-common: Drop armhf build jobs on older linux-* branches
In reply to Ian CampbellIan Campbell writes ("[PATCH OSSTEST 2/5] mfi-common: Drop armhf build jobs on older linux-* branches"):
Quoted 11 lines
test_matrix_branch_filter_callbackwas recently updated to exclude testing of linux-3.10 (54f237784d4b) and 3.14 (b0c5663a03e7), however this only excludes test and not build jobs so we were still trying to build the arm kernel there.This changes the build job filtering to be in sync with the test job filtering, the net result is to remove build-armhf* from linux-3.10 and linux-3.14.
A subsequent patch will try and combine those two filters into one to prevent this skew happening again.
Re: [PATCH OSSTEST 3/5] make-*flight: Reorder.
In reply to Ian CampbellIan Campbell writes ("[PATCH OSSTEST 3/5] make-*flight: Reorder."):
Quoted 8 lines
I just got tripped up again by putting a build job filter definition after the call to
create_build_jobs. Reorder the make-*flight scripts to reduce the probability of me doing so any more times.The general order of these scripts is now:
- job filter callbacks
- test job creation
- top-level code which drives the process.
Re: [PATCH OSSTEST 5/5] make-flight: move in-lined branch vs arch filtering into callbacks
In reply to Ian CampbellIan Campbell writes ("[PATCH OSSTEST 5/5] make-flight: move in-lined branch vs arch filtering into callbacks"):
Quoted 5 lines
No change to the output of standalone-generate-dump-flight-runvars
The inlined xenbranch vs arch filters remain where they are since they are common (in that they reflect the addition and removal of arches) and apply equally to all make-*-flight.