osstest: create a local binary FreeBSD package repository
cover osstest: create a local binary FreeBSD package repository
Hello,
In order to reliably run FreeBSD test with the development branch (aka HEAD) osstest needs it's custom binary package repository that contains the packages built against the specific version of FreeBSD under test. FreeBSD HEAD doesn't have any ABI guarantees, and as such binary packages are tied to the base system used to build them.
This series introduces a new job to the FreeBSD specific flight, that builds a local binary package repository against the FreeBSD version under test. Note that this repository only contains the dependencies required to build Xen. The output of the package building job anointed together with the installer as a pair, so they are used in conjunction.
Note that the package building job is (like the installer building job) only used by the FReeBSD specific flight, the output however will be consumed by other flights, just like the FreeBSD installer.
The runvar changes for a FreeBSD flight are:
+build-amd64-freebsd-packages all_host_os freebsd +build-amd64-freebsd-packages all_hostflags PropEq:Firmware:bios:bios +build-amd64-freebsd-packages arch amd64 +build-amd64-freebsd-packages freebsdbuildjob build-amd64-freebsd +build-amd64-freebsd freebsdpackagesbuildjob 133315.build-amd64-freebsd-packages +build-amd64-freebsd-again freebsdpackagesbuildjob build-amd64-freebsd-packages +build-amd64-freebsd-packages freebsdpackagesbuildjob 133315.build-amd64-freebsd-packages +build-amd64-xen-freebsd freebsdpackagesbuildjob build-amd64-freebsd-packages +build-amd64-freebsd-packages host_hostflags arch-amd64,purpose-build +build-amd64-freebsd-packages recipe_skipbuildprep true +build-amd64-freebsd-packages recipe_testinstall true
+build-amd64-freebsd-packages revision_freebsdports +build-amd64-freebsd-packages svnrevision_freebsdports 483590
+build-amd64-freebsd-packages svntree_freebsdports https://svn.FreeBSD.org/ports/head
+build-amd64-freebsd-packages tree_freebsdports git://github.com/freebsd/freebsd-ports.git
And for a xen-unstable flight:
+test-amd64-amd64-examine freebsdpackagesbuildjob 133315.build-amd64-freebsd-packages
The following links show the result of a successful FreeBSD flight including the binary package repository build:
http://logs.test-lab.xenproject.org/osstest/logs/133321/
Note that before applying the series the output of the above flight should be anointed, so follow up flights can used this output as the initial seed:
$ ./mg-anoint anoint "freebsd build master amd64" 133321 build-amd64-freebsd \
"freebsd-packages build master amd64" 133321 build-amd64-freebsd-packagesI've pushed the patch series to my git repo:
git://xenbits.xen.org/people/royger/osstest.git freebsd-pkg
Thanks, Roger.
Roger Pau Monne (6):
osstest: introduce a helper to stash a whole directory
osstest: introduce a helper to create a weblink to a directory
osstest: allow to perform multiple anoints in the same transaction
osstest: introduce a helper to get the svn revision of a git commit
osstest: introduce a script to build a FreeBSD package repository
osstest: use a locally built pkg repository for FreeBSD
Osstest/TestSupport.pm | 32 +++++++- ap-common | 6 ++ ap-fetch-version | 19 ++++- cr-daily-branch | 92 ++++++++++++++------- cri-common | 35 ++++++-- make-freebsd-flight | 14 ++-- mfi-common | 94 ++++++++++++++++----- mg-anoint | 167 ++++++++++++++++++++------------------ sg-run-job | 9 +- ts-build-prep-freebsd | 43 ++++++++++ ts-freebsd-build-packages | 145 +++++++++++++++++++++++++++++++++ ts-freebsd-host-install | 13 --- 12 files changed, 506 insertions(+), 163 deletions(-) create mode 100755 ts-freebsd-build-packages
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/6 osstest: introduce a helper to stash a whole directory
Without compressing it.
--- Osstest/TestSupport.pm | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +18 -1
@@ -30,6 +30,7 @@ use Osstest; use Osstest::Logtailer; use File::Copy; use File::Basename; +use File::Path; use IO::Handle; use Carp; use Digest::SHA;
@@ -90,7 +91,7 @@ BEGIN { dir_identify_vcs build_url_vcs build_clone built_stash built_stash_file built_stash_debugfile - built_compress_stashed + built_compress_stashed built_stash_dir hg_dir_revision git_dir_revision vcs_dir_revision store_revision store_vcs_revision git_massage_url
@@ -1695,6 +1696,22 @@ sub built_stash_file ($$$$;$) { store_runvar("path_$item", $stashleaf); } +sub built_stash_dir ($$$$;$) { + my ($ho, $builddir, $item, $dname, $optional) = @_; + my $files = target_cmd_output($ho, <<END, 300); +cd $builddir/$dname +find -L . -type f +END + + foreach my $file (split /\n/, $files) { + $file =~ s,^./,,; + my $stashleaf = "$stash/build/$item/$file"; + + mkpath(dirname($stashleaf)); + target_getfile($ho, 300, "$builddir/$dname/$file", $stashleaf); + } + store_runvar("path_$item", "build/$item"); +} sub built_compress_stashed($) { my ($path) = @_;
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 1/6] osstest: introduce a helper to stash a whole directory
Roger Pau Monne writes ("[Xen-devel] [PATCH 1/6] osstest: introduce a helper to stash a whole directory"):
Quoted 1 line
Without compressing it.
You've open-coded a recursive directory copy. Is rsync available on $ho at this point ? I think maybe it could be ...
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 1/6] osstest: introduce a helper to stash a whole directory
Quoted 5 lines · Ian Jackson
Roger Pau Monne writes ("[Xen-devel] [PATCH 1/6] osstest: introduce a helper to stash a whole directory"):
Quoted 1 line
Without compressing it.
You've open-coded a recursive directory copy. Is rsync available on $ho at this point ? I think maybe it could be ...
D'oh, yes, rsync could be made available on $ho at this point.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 2/6 osstest: introduce a helper to create a weblink to a directory
In reply to Roger Pau Monne--- Osstest/TestSupport.pm | 13 +++++++++++++ 1 file changed, 13 insertions(+)
Osstest/TestSupport.pm +13 -0
@@ -125,6 +125,7 @@ BEGIN { toolstack guest_create guest_create_paused await_webspace_fetch_byleaf create_webfile + create_weblink file_link_contents get_timeout setup_netboot_di setup_netboot_local host_netboot_file subst_netboot_template setup_netboot_memdisk
@@ -2739,6 +2740,18 @@ sub create_webfile ($$$) { return $wf_url; } +sub create_weblink ($$$) { + my ($ho, $tail, $target) = @_; + my $wf_rhs= hostnamepath($ho)."_".$tail; + my $wf_common= $c{WebspaceCommon}.$wf_rhs; + my $wf_url= $c{WebspaceUrl}.$wf_common; + my $wf_file= $c{WebspaceFile}.$wf_common; + + unlink $wf_file; + symlink $target, $wf_file or die "$wf_file $!"; + return $wf_url; +} + #---------- netboot handling ---------- sub file_link_contents ($$$) {
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 2/6] osstest: introduce a helper to create a weblink to a directory
Roger Pau Monne writes ("[Xen-devel] [PATCH 2/6] osstest: introduce a helper to create a weblink to a directory"):
Osstest/TestSupport.pm sub create_webfile ($$$) { · line 2752
Quoted 10 lines
Osstest/TestSupport.pm +10
-0+sub create_weblink ($$$) { + my ($ho, $tail, $target) = @_; + my $wf_rhs= hostnamepath($ho)."_".$tail; + my $wf_common= $c{WebspaceCommon}.$wf_rhs; + my $wf_url= $c{WebspaceUrl}.$wf_common; + my $wf_file= $c{WebspaceFile}.$wf_common; + + unlink $wf_file; + symlink $target, $wf_file or die "$wf_file $!"; + return $wf_url;
Most of this is the start of create_webfile. Can you factor that out ?
I have three suggestions for the shape:
my ($wf_file, $wf_url) = prepare_create_webfile($ho, $tail);
symlink $target, $wf_file or die "$wf_file $!";
return $wf_url;}
or split create_webfile into create_web_fsobject which takes a subref, and the call to file_link_contents:
create_web_fsobject($ho, $tail, sub {
my ($wf_file) = @_;
symlink $target, $wf_file or die "$wf_file $!";
});}
or the same but make passing a subref as $contents legal for create_webfile,
sub create_webfile ($$$) {
my .... $contents_spec);
# $contents as for file_link_contents, or a subref
# which will be called as $contents_spec($wf_file,$wf_url);
...
if (ref($contents_spec) ne 'CODE') {
my $contents = $contents_spec;
$contents_spec = sub {
my ($wf_file) = @_;
file_link_contents($wf_file, $contents, "webspace-$wf_rhs");
};
}Take your pick, or do something similar ?
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 3/6 osstest: allow to perform multiple anoints in the same transaction
In reply to Roger Pau MonneIn the same way allow to perform several fetches in the same retrieve transaction.
Further patches will anoint a FreeBSD image and a binary ports tree in the same transaction, and it's required to do it in the same transaction in order to avoid inconsistencies when fetching them.
Note that most of the changes in this patch is code movement in order to place the database accessors inside of a loop that iterates over the input parameters.
--- mg-anoint | 167 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 88 insertions(+), 79 deletions(-)
mg-anoint +88 -79
@@ -10,12 +10,12 @@ # # ./mg-anoint destroy REFKEY # -# ./mg-anoint anoint [ANOINT-OPTION...] REFKEY FLIGHT JOB +# ./mg-anoint anoint [ANOINT-OPTION...] REFKEY FLIGHT JOB [REFKEY FLIGHT JOB...] # ANOINT-OPTIONs are: # --allow-blessed=BLESSING,... default is from `prepare' # --allow-job-status=STATUS,... default is only `pass' # -# ./mg-anoint retrieve [--tolerate-unprepared] REFKEY +# ./mg-anoint retrieve [--tolerate-unprepared] REFKEY [REFKEY...] # => FLIGHT JOB # if nothing anointed yet, prints nothing and exits 0 # if anointment not prepared, fails
@@ -189,8 +189,7 @@ sub cmd_anoint { die "unknown option $_ ?"; } } - die unless @ARGV==3; - my ($refkey, $flight, $job) = @ARGV; + die unless @ARGV%3==0; fail_unless_can_anoint(); prep_queries();
@@ -228,69 +227,74 @@ END db_retry($dbh_tests, [], sub { @o = (); - $task_q->execute($refkey); - - # find the task row (ie, the anointment kind) - my ($task, $refinfo) = $task_q->fetchrow_array(); - die "no such anointment kind \`$refkey' (no prepare?)\n" - unless defined $task; - my %params; - foreach (split /\s+/, $refinfo) { - die unless m/=/; - $params{$`} = $'; - } - my %blessings; - $blessings{$_}++ foreach - grep /./, - (split /,/, $params{blessings}), - (split /,/, $allow_blessed); - - my %jobstatus; - $jobstatus{pass}++; - $jobstatus{$_}++ foreach grep /./, split /,/, $allow_jobstatus; - - # check the to-be-anointed flight's blessing - $newflight_q->execute($flight); - my $frow = $newflight_q->fetchrow_hashref(); - die "flight $flight missing" unless $frow; - die "flight $flight not started" unless defined $frow->{started}; - - # check the job status - $newjob_q->execute($flight, $job); - my ($jstatus) = $newjob_q->fetchrow_array(); - die "job $flight.$job missing" unless defined $jstatus; - die "job $flight.$job status $jstatus" unless $jobstatus{$jstatus}; - - push @o, "flight $flight blessed $frow->{blessing}". - " started ".show_abs_time($frow->{started}); - - die "flight $flight blessing $frow->{blessing}". - " (not $params{blessings} / $allow_blessed)" - unless $blessings{ $frow->{blessing} }; - - # check to-be-annointed flight is most recent - $mostrecent_q->execute($task); - my $mostrecent = $mostrecent_q->fetchrow_hashref(); - die "flight $flight not newer than $mostrecent->{flight}" - unless $frow->{started} > ($mostrecent->{started} // 0); - - # expire old anointments - $count_q->execute($task); - my ($current) = $count_q->fetchrow_array(); - my $want_delete = ($current+1) - $params{keep}; - push @o, "anointment $refkey: currently $current anointed"; - if ($want_delete > 0) { - $todelete_q->execute($task, $want_delete); - while (my $d = $todelete_q->fetchrow_hashref()) { - push @o, " expiring $d->{flight}.$d->{job} [/$d->{shareix}]". - " started ".show_abs_time($d->{started}); - $delete_res_q->execute($task, $d->{flight}, $d->{shareix}); - } - } - # at last! - $insert_q->execute($flight,$task,$task,$job); - push @o, "anointed $flight.$job"; + for (my $i=0; $i < @ARGV; $i+=3) { + my ($refkey, $flight, $job) = @ARGV[$i..$i+2]; + + $task_q->execute($refkey); + + # find the task row (ie, the anointment kind) + my ($task, $refinfo) = $task_q->fetchrow_array(); + die "no such anointment kind \`$refkey' (no prepare?)\n" + unless defined $task; + my %params; + foreach (split /\s+/, $refinfo) { + die unless m/=/; + $params{$`} = $'; + } + my %blessings; + $blessings{$_}++ foreach + grep /./, + (split /,/, $params{blessings}), + (split /,/, $allow_blessed); + + my %jobstatus; + $jobstatus{pass}++; + $jobstatus{$_}++ foreach grep /./, split /,/, $allow_jobstatus; + + # check the to-be-anointed flight's blessing + $newflight_q->execute($flight); + my $frow = $newflight_q->fetchrow_hashref(); + die "flight $flight missing" unless $frow; + die "flight $flight not started" unless defined $frow->{started}; + + # check the job status + $newjob_q->execute($flight, $job); + my ($jstatus) = $newjob_q->fetchrow_array(); + die "job $flight.$job missing" unless defined $jstatus; + die "job $flight.$job status $jstatus" unless $jobstatus{$jstatus}; + + push @o, "flight $flight blessed $frow->{blessing}". + " started ".show_abs_time($frow->{started}); + + die "flight $flight blessing $frow->{blessing}". + " (not $params{blessings} / $allow_blessed)" + unless $blessings{ $frow->{blessing} }; + + # check to-be-annointed flight is most recent + $mostrecent_q->execute($task); + my $mostrecent = $mostrecent_q->fetchrow_hashref(); + die "flight $flight not newer than $mostrecent->{flight}" + unless $frow->{started} > ($mostrecent->{started} // 0); + + # expire old anointments + $count_q->execute($task); + my ($current) = $count_q->fetchrow_array(); + my $want_delete = ($current+1) - $params{keep}; + push @o, "anointment $refkey: currently $current anointed"; + if ($want_delete > 0) { + $todelete_q->execute($task, $want_delete); + while (my $d = $todelete_q->fetchrow_hashref()) { + push @o, " expiring $d->{flight}.$d->{job} [/$d->{shareix}]". + " started ".show_abs_time($d->{started}); + $delete_res_q->execute($task, $d->{flight}, $d->{shareix}); + } + } + + # at last! + $insert_q->execute($flight,$task,$task,$job); + push @o, "anointed $flight.$job"; + } }); pr_o(); }
@@ -301,26 +305,31 @@ sub cmd_retrieve { shift @ARGV; $tolerate_unprepared = 1; } - die unless @ARGV==1; + die unless @ARGV>=1; die if $ARGV[0] =~ m/^-/; - my ($refkey) = @ARGV; empty_unless_can_anoint(); prep_queries(); db_retry($dbh_tests, [], sub { @o = (); - $task_q->execute($refkey); - my ($task) = $task_q->fetchrow_array(); - die "no such anointment kind \`$refkey'" - unless defined $task or $tolerate_unprepared; - - $mostrecent_q->execute($task); - my $row = $mostrecent_q->fetchrow_hashref(); - if ($row) { - push @o, "$row->{flight} $row->{job}"; - } else { - print STDERR "warning: nothing anointed $refkey\n"; + + foreach my $refkey (@ARGV) { + $task_q->execute($refkey); + my ($task) = $task_q->fetchrow_array(); + die "no such anointment kind \`$refkey'" + unless defined $task or $tolerate_unprepared; + + $mostrecent_q->execute($task); + my $row = $mostrecent_q->fetchrow_hashref(); + if ($row) { + push @o, "$row->{flight} $row->{job}"; + } else { + # Push an error line so the number of output lines + # matches the number of REFKEYs requested. + push @o, "ERROR"; + print STDERR "warning: nothing anointed $refkey\n"; + } } }); pr_o();
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 3/6] osstest: allow to perform multiple anoints in the same transaction
Roger Pau Monne writes ("[Xen-devel] [PATCH 3/6] osstest: allow to perform multiple anoints in the same transaction"):
Quoted 3 lines
Note that most of the changes in this patch is code movement in order to place the database accessors inside of a loop that iterates over the input parameters.
Sorry to be *really* annoying, but is it possible to put the code motion in a separate patch ? Move it into a subroutine, I guess.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 4/6 osstest: introduce a helper to get the svn revision of a git commit
In reply to Roger Pau MonneThis only works when the svn revision is stored as a git note with the format 'revision=<revision number>'.
Such conversion is required in order to bootstrap a FreeBSD system without relying on external package repositories. FreeBSD base system only contains a subversion client (no git client), and thus in order to fetch the ports repository (that contain the external packages build makefiles) svn must be used.
--- cri-common | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-)
cri-common +27 -8
@@ -38,12 +38,10 @@ besteffort_repo () { cached_repo "$1" '[fetch=try]' } -repo_tree_rev_fetch_git () { - local treename=$1 - local remoteurl=$2 - local remotetag=$3 - local localtag=$4 +repo_get_realurl () { + local remoteurl=$1 local proxy=`getconfig GitCacheProxy` + case $remoteurl in $proxy*) local realurl="$remoteurl" ;;
@@ -52,14 +50,35 @@ repo_tree_rev_fetch_git () { *) local realurl="$remoteurl" ;; esac - if ! test -d $repos/$treename; then - git clone --bare $realurl $repos/$treename >&2 - fi + + echo $realurl +} + +repo_tree_rev_fetch_git () { + local treename=$1 + local remoteurl=$2 + local remotetag=$3 + local localtag=$4 + local realurl=`repo_get_realurl $remoteurl` + cd $repos/$treename git fetch -f $realurl $remotetag:$localtag >&2 git rev-parse $localtag^0 } +repo_tree_git2svn_rev () { + local treename=$1 + local remoteurl=$2 + local gitrev=$3 + local realurl=`repo_get_realurl $remoteurl` + + cd $repos/$treename + git fetch -f $realurl refs/notes/commits:refs/notes/commits >&2 + git notes show $gitrev | \ + sed -n 's/^.*revision=\([0-9][0-9]*\).*$/\1/p' +} + + select_prevxenbranch () { prevxenbranch=`./cri-getprevxenbranch $xenbranch` }
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit
Roger Pau Monne writes ("[Xen-devel] [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit"):
Quoted 2 lines
This only works when the svn revision is stored as a git note with the format 'revision=<revision number>'.
Wow. This is pretty ugly.
Quoted 5 lines
Such conversion is required in order to bootstrap a FreeBSD system without relying on external package repositories. FreeBSD base system only contains a subversion client (no git client), and thus in order to fetch the ports repository (that contain the external packages build makefiles) svn must be used.
git notes have some different way of travelling than commits, don't they ? Where is this git note coming from and how do we know it is the right note, IYSWIM ?
Aside from that, please break the refactoring (in this case, the breaking out of repo_get_realurl) into a separate NFC patch.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit
Quoted 5 lines · Ian Jackson
Roger Pau Monne writes ("[Xen-devel] [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit"):
Quoted 2 lines
This only works when the svn revision is stored as a git note with the format 'revision=<revision number>'.
Wow. This is pretty ugly.
Indeed :(.
Quoted 10 lines
Quoted 6 lines
Such conversion is required in order to bootstrap a FreeBSD system without relying on external package repositories. FreeBSD base system only contains a subversion client (no git client), and thus in order to fetch the ports repository (that contain the external packages build makefiles) svn must be used.
git notes have some different way of travelling than commits, don't they ? Where is this git note coming from and how do we know it is the right note, IYSWIM ?
I'm not an expert on this, but I think notes are always stored in a separate branch on the same repo? In the FreeBSD case at least it's git/refs/notes.
Quoted 2 lines
Aside from that, please break the refactoring (in this case, the breaking out of
repo_get_realurl) into a separate NFC patch.
Sure!
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit
Roger Pau Monne writes ("Re: [Xen-devel] [PATCH 4/6] osstest: introduce a helper to get the svn revision of a git commit"):
Quoted 8 lines · Ian Jackson
Quoted 3 lines
git notes have some different way of travelling than commits, don't they ? Where is this git note coming from and how do we know it is the right note, IYSWIM ?
I'm not an expert on this, but I think notes are always stored in a separate branch on the same repo? In the FreeBSD case at least it's git/refs/notes.
OK, so what are git's rules for where it fetches notes from ? Which notes does it look at ? (Do you understand why I am asking these questions?)
Quoted 4 lines
Quoted 2 lines
Aside from that, please break the refactoring (in this case, the breaking out of
repo_get_realurl) into a separate NFC patch.Sure!
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 5/6 osstest: introduce a script to build a FreeBSD package repository
In reply to Roger Pau MonneThe building of the binary packages itself is done with the poudriere tool, that given a set of port names generates a binary package repository with the requested packages and all it's dependencies.
Add a packages build job in the FreeBSD flight. Note that the binary packages generated are tied to the base system used to build them when not using a FreeBSD stable branch, stable branches guarantee ABI stability.
Fetching the ports tree that contain the Makefiles to build the ports is slightly more complicated than what would expected, since the FreeBSD base system doesn't contain a git client the fetching is done form the svn repository using the svnlite tool from the base system. This is required in order to assure that bootstrapping the binary repository doesn't depend on any external tools not found in the base system.
--- ap-common | 6 ++ ap-fetch-version | 19 +++-- cr-daily-branch | 57 +++++++++------ make-freebsd-flight | 8 ++- sg-run-job | 9 ++- ts-freebsd-build-packages | 145 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 215 insertions(+), 29 deletions(-) create mode 100755 ts-freebsd-build-packages
ap-common +6 -0
@@ -41,6 +41,11 @@ : ${PUSH_TREE_FREEBSD:=$XENBITS:/home/xen/git/freebsd.git} : ${BASE_TREE_FREEBSD:=git://xenbits.xen.org/freebsd.git} +: ${TREE_FREEBSD_PORTS:=git://github.com/freebsd/freebsd-ports.git} +: ${TREE_FREEBSD_PORTS_SVN:=https://svn.FreeBSD.org/ports/head} +: ${PUSH_TREE_FREEBSD_PORTS:=$XENBITS:/home/xen/git/freebsd-ports.git} +: ${BASE_TREE_FREEBSD_PORTS:=git://xenbits.xen.org/freebsd-ports.git} + : ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git} : ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git} : ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git}
@@ -88,6 +93,7 @@ fi : ${LOCALREV_OVMF:=daily-cron.$branch} : ${LOCALREV_XTF:=daily-cron.$branch} : ${LOCALREV_FREEBSD:=daily-cron.$branch} +: ${LOCALREV_FREEBSD_PORTS:=daily-cron.$branch} : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
ap-fetch-version +15 -4
@@ -108,11 +108,22 @@ ovmf) ;; freebsd-*) branchcore=${branch#freebsd-} - if [ "x$branchcore" != "xmaster" ]; then - branchcore="stable/$branchcore" + subbranch=${branchcore#*--} + branchcore=${branchcore%--"$subbranch"} + + if [ "$subbranch" = "$branchcore" ]; then + if [ "$branchcore" != master ]; then + branchcore="stable/$branchcore" + fi + repo_tree_rev_fetch_git freebsd \ + $TREE_FREEBSD $branchcore $LOCALREV_FREEBSD + else + if [ "$subbranch" != ports ]; then + exit 1 + fi + repo_tree_rev_fetch_git freebsd-ports \ + $TREE_FREEBSD_PORTS master $LOCALREV_FREEBSD_PORTS fi - repo_tree_rev_fetch_git freebsd \ - $TREE_FREEBSD $branchcore $LOCALREV_FREEBSD ;; osstest) if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
cr-daily-branch +36 -21
73 lines not on this page · raw message
make-freebsd-flight +5 -3
@@ -38,13 +38,15 @@ job_create_build_filter_callback () { for arch in "$arches"; do set_freebsd_runvars - create_freebsd_build_job build-$arch-freebsd - # Create an identical job that's going to use the build output from - # the previous one. + # Create a job to build the packages against the new world. freebsd_runvars="$freebsd_runvars freebsdbuildjob=build-$arch-freebsd \ recipe_testinstall=true" + create_freebsd_pkg_build_job build-$arch-freebsd-packages + + # Create an identical job that's going to use the build output from + # the previous one. create_freebsd_build_job build-$arch-freebsd-again # Create a Xen build job that's going to use the output from the first
sg-run-job +8 -1
@@ -716,6 +716,7 @@ proc need-hosts/build-rumprun {} { return BUILD_LINUX } proc need-hosts/build-xtf {} { return BUILD_LINUX } proc need-hosts/build-freebsd {} { return BUILD_FREEBSD } proc need-hosts/build-xen-freebsd {} { return BUILD_FREEBSD } +proc need-hosts/build-pkg-freebsd {} { return BUILD_FREEBSD } proc run-job/build {} { run-ts . = ts-xen-build
@@ -746,6 +747,10 @@ proc run-job/build-freebsd {} { run-ts . = ts-freebsd-build } +proc run-job/build-pkg-freebsd {} { + run-ts . = ts-freebsd-build-packages +} + proc run-job/build-xen-freebsd {} { run-ts . = ts-xen-build-freebsd + host }
@@ -768,7 +773,9 @@ proc prepare-build-host-freebsd {} { global jobinfo if {[recipe-flag testinstall]} { set broken fail } { set broken broken } run-ts $broken host-install(*) ts-freebsd-host-install - run-ts . host-build-prep ts-build-prep-freebsd + if {![recipe-flag skipbuildprep]} { + run-ts . host-build-prep ts-build-prep-freebsd + } } proc need-hosts/coverity {} { return BUILD_LINUX }
ts-freebsd-build-packages new file +145 -0
145 lines not on this page · raw message
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository
Roger Pau Monne writes ("[Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository"):
Quoted 3 lines
The building of the binary packages itself is done with the poudriere tool, that given a set of port names generates a binary package repository with the requested packages and all it's dependencies.
...
Quoted 3 lines
ap-common | 6 ++ ap-fetch-version | 19 +++-- cr-daily-branch | 57 +++++++++------
I want to talk separately about the cr-daily-branch and related things.
Also, patches like this are usually more conveniently split up. That way I can ack the less difficult parts separately.
Quoted 3 lines
ts-freebsd-build-packages | 145 ++++++++++++++++++++++++++++++++++++++ sg-run-job | 9 ++- make-freebsd-flight | 8 ++-
At the very least split these three out. But maybe you want to split them into two or three.
I'll read these now...
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository
In reply to Roger Pau MonneRoger Pau Monne writes ("[Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository"):
Quoted 13 lines
make-freebsd-flight +0
-3@@ -38,13 +38,15 @@ job_create_build_filter_callback () { for arch in "$arches"; do set_freebsd_runvars - create_freebsd_build_job build-$arch-freebsd - # Create an identical job that's going to use the build output from - # the previous one.
make-freebsd-flight job_create_build_filter_callback () { · line 52
Quoted 10 lines
make-freebsd-flight +5
-0+ # Create a job to build the packages against the new world. freebsd_runvars="$freebsd_runvars freebsdbuildjob=build-$arch-freebsd \ recipe_testinstall=true" + create_freebsd_pkg_build_job build-$arch-freebsd-packages + + # Create an identical job that's going to use the build output from + # the previous one. create_freebsd_build_job build-$arch-freebsd-again # Create a Xen build job that's going to use the output from the first
This looks OK.
make-freebsd-flight proc prepare-build-host-freebsd {} { · line 777
Quoted 7 lines
make-freebsd-flight +2
-1@@ -768,7 +773,9 @@ proc prepare-build-host-freebsd {} { global jobinfo if {[recipe-flag testinstall]} { set broken fail } { set broken broken } run-ts $broken host-install(*) ts-freebsd-host-install - run-ts . host-build-prep ts-build-prep-freebsd + if {![recipe-flag skipbuildprep]} { + run-ts . host-build-prep ts-build-prep-freebsd
What's this for ? Oh, I see.
I notice that none of your freebsd build jobs pass any share- hostflag so they always use a fresh installation. Is that necessary ?
Quoted 10 lines
make-freebsd-flight +0
-0proc need-hosts/coverity {} { return BUILD_LINUX }ts-freebsd-build-packages new file +3
-0@@ -0,0 +1,145 @@ +#!/usr/bin/perl -w +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2019 Citrix Inc.
...
ts-freebsd-build-packages line 25
Quoted 3 lines
ts-freebsd-build-packages new file +3
-0+# Consumes the following input runvars: +# svnrevision_freebsdports: ports svn revision id to use. +# svntree_freebsdports ports svn tree to fetch the source code from.
More regular in osstest terms would be
tree_freebsdports
revision_freebsdports
treevcs_freebsdports=svn
But I guess svn is sufficiently unlike what osstest expects out of a vcs that this is not feasible, and it is better to do it this way.
Quoted 6 lines
ts-freebsd-build-packages new file +6
-0+sub checkout () { + my $u = URI->new($c{HttpProxy}); + my $host = $u->host; + my $port = $u->port; + prepbuilddirs(); +
ts-freebsd-build-packages line 64
Quoted 10 lines
ts-freebsd-build-packages new file +10
-0+ logm("Checkout ports tree from svn"); + target_cmd_build($ho, 4000, $builddir, <<END); +cd $builddir +rm -rf ports +# svn ignores HTTP_PROXY envvar +svnlite checkout --config-option servers:global:http-proxy-host=$host \\ + --config-option servers:global:http-proxy-port=$port \\ + --trust-server-cert \\ + $r{"svntree_freebsdports"} \\ + -r $r{"svnrevision_freebsdports"} ports
Will this work to cache the checkout ? All of this says http but I assume it's really https ? Typically, https clients expect to do the TLS themselves but I think you're using our squid mitm and that's what "--trust-server-cert" is doing ?
Rather than "--trust-server-cert" which disables TLS's own mitm protection it would be rather better to inject the osstest mitm squid cert into the testbed, but that may be difficult, and the risk is only from internal things between the build (test) box and the proxy.
ts-freebsd-build-packages line 103
Quoted 4 lines
ts-freebsd-build-packages new file +4
-0+sub create_jail() { + my $src_prefix = $r{"freebsd_distpath"} || + get_stashed("path_freebsddist", $r{"freebsdbuildjob"}); + my $dst_prefix = "/root/sets";
Do we need a jail for this ? We have a whole baremetal OS install whose entire purpose is to do this build ...
Quoted 1 line
+logm("FreeBSD packages built successful");
^ly :-)
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository
Quoted 27 lines · Ian Jackson
Roger Pau Monne writes ("[Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository"):
Quoted 13 lines
make-freebsd-flight +0
-3@@ -38,13 +38,15 @@ job_create_build_filter_callback () { for arch in "$arches"; do set_freebsd_runvars - create_freebsd_build_job build-$arch-freebsd - # Create an identical job that's going to use the build output from - # the previous one.make-freebsd-flight job_create_build_filter_callback () { · line 52
Quoted 10 lines
make-freebsd-flight +5
-0+ # Create a job to build the packages against the new world. freebsd_runvars="$freebsd_runvars freebsdbuildjob=build-$arch-freebsd \ recipe_testinstall=true" + create_freebsd_pkg_build_job build-$arch-freebsd-packages + + # Create an identical job that's going to use the build output from + # the previous one. create_freebsd_build_job build-$arch-freebsd-again # Create a Xen build job that's going to use the output from the firstThis looks OK.
Quoted 9 lines
make-freebsd-flight proc prepare-build-host-freebsd {} { · line 777
Quoted 7 lines
make-freebsd-flight +2
-1@@ -768,7 +773,9 @@ proc prepare-build-host-freebsd {} { global jobinfo if {[recipe-flag testinstall]} { set broken fail } { set broken broken } run-ts $broken host-install(*) ts-freebsd-host-install - run-ts . host-build-prep ts-build-prep-freebsd + if {![recipe-flag skipbuildprep]} { + run-ts . host-build-prep ts-build-prep-freebsdWhat's this for ? Oh, I see.
The job that creates the package repository cannot use build-prep because the packages are not yet built.
Quoted 2 lines
I notice that none of your freebsd build jobs pass any share- hostflag so they always use a fresh installation. Is that necessary ?
Hm, I don't think so. build-amd64-xen-freebsd and build-amd64-freebsd-again could share a host. I need to take a look at how to do this, I could send this as a separate fix for the existing jobs.
Quoted 11 lines
Quoted 10 lines
make-freebsd-flight +0
-0proc need-hosts/coverity {} { return BUILD_LINUX }ts-freebsd-build-packages new file +3
-0@@ -0,0 +1,145 @@ +#!/usr/bin/perl -w +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2019 Citrix Inc....
Quoted 10 lines
ts-freebsd-build-packages line 25
Quoted 3 lines
ts-freebsd-build-packages new file +3
-0+# Consumes the following input runvars: +# svnrevision_freebsdports: ports svn revision id to use. +# svntree_freebsdports ports svn tree to fetch the source code from.More regular in osstest terms would be
tree_freebsdports
revision_freebsdports
treevcs_freebsdports=svn
But I guess svn is sufficiently unlike what osstest expects out of a vcs that this is not feasible, and it is better to do it this way.
I don't really have an opinion, I somehow assumed that using the same format might interfere with things like bisection, so I've decided to pass the git revision using tree_freebsdports &c and the svn revision using the newly introduced flags.
Quoted 10 lines
Quoted 10 lines
ts-freebsd-build-packages new file +10
-0+sub checkout () { + my $u = URI->new($c{HttpProxy}); + my $host = $u->host; + my $port = $u->port; + prepbuilddirs(); + + logm("Checkout ports tree from svn"); + target_cmd_build($ho, 4000, $builddir, <<END); +cd $builddir +rm -rf ports
Quoted 8 lines
ts-freebsd-build-packages line 64
Quoted 6 lines
ts-freebsd-build-packages new file +6
-0+# svn ignores HTTP_PROXY envvar +svnlite checkout --config-option servers:global:http-proxy-host=$host \\ + --config-option servers:global:http-proxy-port=$port \\ + --trust-server-cert \\ + $r{"svntree_freebsdports"} \\ + -r $r{"svnrevision_freebsdports"} portsWill this work to cache the checkout ?
I think so? Would https somehow prevent the caching?
Quoted 2 lines
All of this says http but I
assume it's really https ?
AFAIK svn uses the http-proxy options for both http and https.
Quoted 3 lines
Typically, https clients expect to do the
TLS themselves but I think you're using our squid mitm and that's what "--trust-server-cert" is doing ?
I can't really remember why I've added this option, but I'm quite sure it was failing without it. As you say the proxy is acting as a mitm, so that's likely why trust-server-cert is required.
Quoted 4 lines
Rather than "--trust-server-cert" which disables TLS's own mitm protection it would be rather better to inject the osstest mitm squid cert into the testbed, but that may be difficult, and the risk is only from internal things between the build (test) box and the proxy.
I can look into this, but at the end of day this is all internal, so I'm not sure there's a lot of risk here.
Quoted 7 lines
ts-freebsd-build-packages line 103
Quoted 4 lines
ts-freebsd-build-packages new file +4
-0+sub create_jail() { + my $src_prefix = $r{"freebsd_distpath"} || + get_stashed("path_freebsddist", $r{"freebsdbuildjob"}); + my $dst_prefix = "/root/sets";Do we need a jail for this ? We have a whole baremetal OS install whose entire purpose is to do this build ...
Yes, that's how the repository package builder (poudriere) works, it requires a jail to do the package building. In our case it's not so important, but I assume this is mostly done to always use a clean install, so that currently installed packages on the system don't interfere with package building.
Quoted 2 lines
Quoted 1 line
+logm("FreeBSD packages built successful");
^ly :-)
Thanks.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository
Roger Pau Monne writes ("Re: [Xen-devel] [PATCH 5/6] osstest: introduce a script to build a FreeBSD package repository"):
Quoted 8 lines · Ian Jackson
Quoted 2 lines
I notice that none of your freebsd build jobs pass any share- hostflag so they always use a fresh installation. Is that necessary ?
Hm, I don't think so. build-amd64-xen-freebsd and build-amd64-freebsd-again could share a host. I need to take a look at how to do this, I could send this as a separate fix for the existing jobs.
Sure. It's achieved by putting share-SOMETHING in (any of) the hostflags runvars. SOMETHING needs to include the value of every setting which makes a difference, except the osstest revision (which is added implictly). Jobs with identical SOMETHING can share.
Quoted 11 lines
Quoted 10 lines
ts-freebsd-build-packages line 25
Quoted 3 lines
ts-freebsd-build-packages new file +3
-0+# Consumes the following input runvars: +# svnrevision_freebsdports: ports svn revision id to use. +# svntree_freebsdports ports svn tree to fetch the source code from.More regular in osstest terms would be
tree_freebsdports
revision_freebsdports
treevcs_freebsdports=svn
But I guess svn is sufficiently unlike what osstest expects out of a vcs that this is not feasible, and it is better to do it this way.
Quoted 4 lines
I don't really have an opinion, I somehow assumed that using the same format might interfere with things like bisection, so I've decided to pass the git revision using
tree_freebsdports&c and the svn revision using the newly introduced flags.
Well, svn is awkward to cache and to use and probably we couldn't get the bisector to work with it, indeed. So err I guess I'm saying leave it as it is.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
ts-freebsd-build-packages new file +4
-0+sub create_jail() { + my $src_prefix = $r{"freebsd_distpath"} || + get_stashed("path_freebsddist", $r{"freebsdbuildjob"}); + my $dst_prefix = "/root/sets";
Quoted 8 lines
Quoted 2 lines
Do we need a jail for this ? We have a whole baremetal OS install whose entire purpose is to do this build ...
Yes, that's how the repository package builder (poudriere) works, it requires a jail to do the package building. In our case it's not so important, but I assume this is mostly done to always use a clean install, so that currently installed packages on the system don't interfere with package building.
Fair enough.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 6/6 osstest: use a locally built pkg repository for FreeBSD
In reply to Roger Pau MonneThis removes the dependency on the official pkg repository, which is dangerous when not testing stable branches, since the ABI of the development branch is not stable, and thus it's easy for packages to get out of sync, or for osstest to test an old FreeBSD version that has an ABI different than the one used to build the official packages.
The output of the package build test is tested together with the newly built image, and if there are no regressions both are anointed in lockstep in order to prevent temporary discrepancies between the installer and the package repository.
Note that in order to bootstrap the first run it's possible to manually set the package repository to use with an environment variable:
- FREEBSD_PACKAGES_<arch>_BUILDJOB: points to the flight.job that contains the binary package repository.
- FREEBSD_PACKAGES: points to a local directory that contains the binary repository.
It's also possible to set the directory that contains the package repository in the configuration file using FreeBSDPackages.
--- cr-daily-branch | 35 +++++++++++---- make-freebsd-flight | 10 +++-- mfi-common | 94 +++++++++++++++++++++++++++++++---------- ts-build-prep-freebsd | 43 +++++++++++++++++++ ts-freebsd-host-install | 13 ------ 5 files changed, 147 insertions(+), 48 deletions(-)
cr-daily-branch +26 -9
@@ -321,15 +321,26 @@ freebsd) esac IFS=$'\n' +count=0 for anointed in \ - `./mg-anoint list-prepared "freebsd build $freebsd_branch *"`; do + `./mg-anoint list-prepared "freebsd* build $freebsd_branch *"`; do # Retrieve previous successful FreeBSD build for each arch. freebsd_arch=${anointed##* } - freebsd_envvar="FREEBSD_${freebsd_arch^^}_BUILDJOB" + freebsd_name=${anointed%% *} + freebsd_name=${freebsd_name/-/_} + freebsd_envvar="${freebsd_name^^}_${freebsd_arch^^}_BUILDJOB" if [ "x${!freebsd_envvar}" = "x" ]; then - flight_job=`./mg-anoint retrieve "$anointed"` - export ${freebsd_envvar}=${flight_job/ /.} + envvars[$count]="$freebsd_envvar" + refkeys[$count]="$anointed" + count=$((count+1)) + fi +done +count=0 +for flight_job in `./mg-anoint retrieve ${refkeys[@]}`; do + if [ "$flight_job" != "ERROR" ]; then + export ${envvars[$count]}=${flight_job/ /.} fi + count=$((count+1)) done unset IFS
@@ -542,17 +553,23 @@ freebsd-*) [ "x$OSSTEST_BLESSING" == "xreal" ]; then IFS=$'\n' for anointed in `./mg-anoint list-prepared \ - "freebsd build $freebsd_branch *"`; do + "freebsd* build $freebsd_branch *"`; do # Update anointed versions # NB: failure to update an anointed build for a specific arch # should not be fatal, and it's not an issue if one of the # arches gets slightly out of sync with the other ones. freebsd_arch=${anointed##* } - if ./mg-anoint anoint "$anointed" \ - $flight build-$freebsd_arch-freebsd; then - echo "Anointed artifacts from build-$freebsd_arch-freebsd" - fi + freebsd_name=${anointed%% *} + # Rely on the fact that the job suffix is the same as the + # anointment refkey. Ie: + # refkey: freebsd job: build-<arch>-freebsd + # refkey: freebsd-packages job: build-<arch>-freebsd-packages + anoint="$anoint \"$anointed\" $flight \ + build-$freebsd_arch-$freebsd_name" done + if ./mg-anoint anoint $anoint; then + echo "Anointed build artifacts from flight" + fi unset IFS fi ;;
make-freebsd-flight +6 -4
@@ -45,12 +45,14 @@ for arch in "$arches"; do recipe_testinstall=true" create_freebsd_pkg_build_job build-$arch-freebsd-packages - # Create an identical job that's going to use the build output from - # the previous one. + # Create a build job that going to use the output of both the jobs + # above in order to test the newly built FreeBSD and packages + freebsd_runvars="$freebsd_runvars \ + freebsdpackagesbuildjob=build-$arch-freebsd-packages" create_freebsd_build_job build-$arch-freebsd-again - # Create a Xen build job that's going to use the output from the first - # FreeBSD build job. + # Create a Xen build job that's going to use the output from the + # FreeBSD build jobs. create_xen_build_job build-$arch-xen-freebsd build-xen-freebsd \ host_hostflags=arch-$arch,purpose-build all_host_os=freebsd \ $freebsd_runvars
mfi-common +72 -22
112 lines not on this page · raw message
ts-build-prep-freebsd +43 -0
55 lines not on this page · raw message
ts-freebsd-host-install +0 -13
@@ -194,19 +194,6 @@ sysrc sendmail_submit_enable=NO sysrc sendmail_outbound_enable=NO sysrc sendmail_msp_queue_enable=NO -# Set proxy for the pkg manager -mkdir -p /usr/local/etc/ -cat << ENDPKG >> /usr/local/etc/pkg.conf -pkg_env: { http_proxy = $c{HttpProxy} } -default_always_yes: true -assume_always_yes: true -ENDPKG - -# Bootstap the package manager -export HTTP_PROXY=$c{HttpProxy} -export ASSUME_ALWAYS_YES=yes -pkg bootstrap - # Allow root user login and setup ssh keys chsh -s /bin/sh root echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
Signature · 7 lines
2.17.2 (Apple Git-113) _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD
Roger Pau Monne writes ("[Xen-devel] [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD"):
Quoted 6 lines
This removes the dependency on the official pkg repository, which is dangerous when not testing stable branches, since the ABI of the development branch is not stable, and thus it's easy for packages to get out of sync, or for osstest to test an old FreeBSD version that has an ABI different than the one used to build the official packages.
I realise this is a bit late to be saying this, but had you considered making the packages build a different step in the same job ? That might make a lot of this go away...
Quoted 7 lines
IFS=$'\n'
cr-daily-branch +2
-1+count=0 for anointed in \ - `./mg-anoint list-prepared "freebsd build $freebsd_branch *"`; do + `./mg-anoint list-prepared "freebsd* build $freebsd_branch *"`; do # Retrieve previous successful FreeBSD build for each arch. freebsd_arch=${anointed##* }
cr-daily-branch freebsd) · line 335
Quoted 10 lines
cr-daily-branch +6
-3- freebsd_envvar="FREEBSD_${freebsd_arch^^}_BUILDJOB" + freebsd_name=${anointed%% *} + freebsd_name=${freebsd_name/-/_} + freebsd_envvar="${freebsd_name^^}_${freebsd_arch^^}_BUILDJOB" if [ "x${!freebsd_envvar}" = "x" ]; then - flight_job=`./mg-anoint retrieve "$anointed"` - export ${freebsd_envvar}=${flight_job/ /.} + envvars[$count]="$freebsd_envvar" + refkeys[$count]="$anointed" + count=$((count+1))
You don't need this counter. You can just say
envvars=()
...
envvars+=("$freebsd_envvar")
etc.
cr-daily-branch freebsd) · line 343
Quoted 8 lines
cr-daily-branch +7
-0+ fi +done +count=0 +for flight_job in `./mg-anoint retrieve ${refkeys[@]}`; do + if [ "$flight_job" != "ERROR" ]; then + export ${envvars[$count]}=${flight_job/ /.} fi + count=$((count+1))
I think you do need count here, if you do this as two loops. But:
Why not do this retrieve, and set the env vars, inside the first loop ? I think that would avoid having to accumulate a data structure full of information in shell variables at all (and shell is not very good at this kind of thing).
Quoted 8 lines
cr-daily-branch +1
-1@@ -542,17 +553,23 @@ freebsd-*) [ "x$OSSTEST_BLESSING" == "xreal" ]; then IFS=$'\n' for anointed in `./mg-anoint list-prepared \ - "freebsd build $freebsd_branch *"`; do + "freebsd* build $freebsd_branch *"`; do # Update anointed versions # NB: failure to update an anointed build for a specific arch
cr-daily-branch freebsd-*) · line 564
Quoted 10 lines
cr-daily-branch +3
-4# should not be fatal, and it's not an issue if one of the # arches gets slightly out of sync with the other ones. freebsd_arch=${anointed##* } - if ./mg-anoint anoint "$anointed" \ - $flight build-$freebsd_arch-freebsd; then - echo "Anointed artifacts from build-$freebsd_arch-freebsd" - fi + freebsd_name=${anointed%% *} + # Rely on the fact that the job suffix is the same as the + # anointment refkey. Ie:
I don't think you mean "Rely on the fact". Rather, "by definition, from the way the flight is constructed, the intended..." ?
cr-daily-branch freebsd-*) · line 568
Quoted 4 lines
cr-daily-branch +4
-0+ # refkey: freebsd job: build-<arch>-freebsd + # refkey: freebsd-packages job: build-<arch>-freebsd-packages + anoint="$anoint \"$anointed\" $flight \ + build-$freebsd_arch-$freebsd_name"
Maybe use an array variable for anount, and then you can avoid the shell \" quoting.
Quoted 5 lines
mfi-common +0
-0@@ -156,7 +156,6 @@ set_freebsd_runvars () {
...
mfi-common set_freebsd_runvars () { · line 175
Quoted 10 lines
mfi-common +10
-0+ # Check if the packages are provided externally, or else assume they + # are provided by the same flight as the installer binaries. + local pkgpath=`getconfig "FreeBSDPackages"` + counter=0 + IFS=$'\n' + for flightjob in `./mg-anoint retrieve --tolerate-unprepared \ + "freebsd build master $arch" \ + "freebsd-packages build master $arch"`; do + if [ $counter -eq 0 ]; then + # Anointed FreeBSD installer
I don't much like this code, but I'm having trouble saying what I think should be done instead.
I don't much like the $counter -eq 0 approach. Maybe some of it should go into a function ?
./mg-anoint retrieve ... >tmpfile
if freebsd_want_anointed <tmpfile '' Dist ...
but not sure what the function should be.
Quoted 7 lines
mfi-common +7
-0+ local envvar="FREEBSD_${arch^^}_BUILDJOB" + local distpath=`getconfig "FreeBSDDist"` + if [ -n "${!envvar}" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsdbuildjob=${!envvar}" + elif [ -n "$FREEBSD_DIST" ] && [ -n "$FREEBSD_VERSION" ]; then + freebsd_runvars="$freebsd_runvars \
mfi-common set_freebsd_runvars () { · line 192
Quoted 10 lines
mfi-common +10
-0+ freebsd_distpath=$FREEBSD_DIST/$arch \ + freebsd_version=$FREEBSD_VERSION" + elif [ -n "$distpath" ]; then + local version=`getconfig "FreeBSDVersion"` + freebsd_runvars="$freebsd_runvars \ + freebsd_distpath=$distpath/$arch \ + freebsd_version=$version" + elif [ "$flightjob" != "ERROR" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsdbuildjob=${flightjob/ /.}"
There seems like a lot of repetition here. For example, FREEBSD_DIST overrides FreeBSDDist but /$arch is appended in two places. Maybe ${FREEBSD_DIST- ... something ... } would be better ?
It is difficult to see the wood for the trees, particularly with the constant repetition of
freebsd_runvars="$freebsd_runvars \
which could be avoided by having this fragment set a local variable containing the things to be added.
Quoted 6 lines
mfi-common +6
-0+ elif [ $counter -eq 1 ]; then + # Anointed package repository + local envvar="FREEBSD_PACKAGES_${arch^^}_BUILDJOB" + local pkgpath=`getconfig "FreeBSDPackages"` + if [ -n "${!envvar}" ]; then + freebsd_runvars="$freebsd_runvars \
mfi-common set_freebsd_runvars () { · line 209
Quoted 10 lines
mfi-common +10
-0+ freebsdpackagesbuildjob=${!envvar}" + elif [ -n "$FREEBSD_PACKAGES" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsd_packages=$FREEBSD_PACKAGES/$arch" + elif [ -n "$pkgpath" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsd_packages=$pkgpath/$arch" + elif [ "$flightjob" != "ERROR" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsdpackagesbuildjob=${flightjob/ /.}"
This feels very similar to the code above, although it lacks the special handling for the version.
The rest of this looks plausible.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD
Quoted 11 lines · Ian Jackson
Roger Pau Monne writes ("[Xen-devel] [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD"):
Quoted 6 lines
This removes the dependency on the official pkg repository, which is dangerous when not testing stable branches, since the ABI of the development branch is not stable, and thus it's easy for packages to get out of sync, or for osstest to test an old FreeBSD version that has an ABI different than the one used to build the official packages.
I realise this is a bit late to be saying this, but had you considered making the packages build a different step in the same job ? That might make a lot of this go away...
Do you mean to build the packages in build-prep instead of relying on having a custom binary repository?
I could do that, but it's going to take some time. Also doing the package build in build-prep would require fetching the svn ports repository for each build-prep instance.
IIRC the package building job takes a non-trivial amount of time (2-3h IIRC), because it has to build gcc (for SeaBIOS) and python, perl...
Quoted 14 lines
Quoted 14 lines
IFS=$'\n'
cr-daily-branch +5
-4+count=0 for anointed in \ - `./mg-anoint list-prepared "freebsd build $freebsd_branch *"`; do + `./mg-anoint list-prepared "freebsd* build $freebsd_branch *"`; do # Retrieve previous successful FreeBSD build for each arch. freebsd_arch=${anointed##* } - freebsd_envvar="FREEBSD_${freebsd_arch^^}_BUILDJOB" + freebsd_name=${anointed%% *} + freebsd_name=${freebsd_name/-/_} + freebsd_envvar="${freebsd_name^^}_${freebsd_arch^^}_BUILDJOB" if [ "x${!freebsd_envvar}" = "x" ]; then - flight_job=`./mg-anoint retrieve "$anointed"` - export ${freebsd_envvar}=${flight_job/ /.}
Quoted 8 lines
cr-daily-branch freebsd) · line 335
Quoted 3 lines
cr-daily-branch +3
-0+ envvars[$count]="$freebsd_envvar" + refkeys[$count]="$anointed" + count=$((count+1))You don't need this counter. You can just say
envvars=()
...
envvars+=("$freebsd_envvar")
Oh, thanks!
Quoted 11 lines
cr-daily-branch freebsd) · line 343
Quoted 8 lines
cr-daily-branch +7
-0+ fi +done +count=0 +for flight_job in `./mg-anoint retrieve ${refkeys[@]}`; do + if [ "$flight_job" != "ERROR" ]; then + export ${envvars[$count]}=${flight_job/ /.} fi + count=$((count+1))I think you do need count here, if you do this as two loops. But:
Quoted 4 lines
Why not do this retrieve, and set the env vars, inside the first loop ? I think that would avoid having to accumulate a data structure full of information in shell variables at all (and shell is not very good at this kind of thing).
Yes, I think I can do it as you suggest.
Quoted 22 lines
Quoted 8 lines
cr-daily-branch +1
-1@@ -542,17 +553,23 @@ freebsd-*) [ "x$OSSTEST_BLESSING" == "xreal" ]; then IFS=$'\n' for anointed in `./mg-anoint list-prepared \ - "freebsd build $freebsd_branch *"`; do + "freebsd* build $freebsd_branch *"`; do # Update anointed versions # NB: failure to update an anointed build for a specific archcr-daily-branch freebsd-*) · line 564
Quoted 10 lines
cr-daily-branch +3
-4# should not be fatal, and it's not an issue if one of the # arches gets slightly out of sync with the other ones. freebsd_arch=${anointed##* } - if ./mg-anoint anoint "$anointed" \ - $flight build-$freebsd_arch-freebsd; then - echo "Anointed artifacts from build-$freebsd_arch-freebsd" - fi + freebsd_name=${anointed%% *} + # Rely on the fact that the job suffix is the same as the + # anointment refkey. Ie:I don't think you mean "Rely on the fact". Rather, "by definition, from the way the flight is constructed, the intended..." ?
Quoted 7 lines
cr-daily-branch freebsd-*) · line 568
Quoted 4 lines
cr-daily-branch +4
-0+ # refkey: freebsd job: build-<arch>-freebsd + # refkey: freebsd-packages job: build-<arch>-freebsd-packages + anoint="$anoint \"$anointed\" $flight \ + build-$freebsd_arch-$freebsd_name"Maybe use an array variable for anount, and then you can avoid the shell \" quoting.
Please bear with me, but can you elaborate on this?
Quoted 17 lines
Quoted 5 lines
mfi-common +0
-0@@ -156,7 +156,6 @@ set_freebsd_runvars () {...
mfi-common set_freebsd_runvars () { · line 175
Quoted 10 lines
mfi-common +10
-0+ # Check if the packages are provided externally, or else assume they + # are provided by the same flight as the installer binaries. + local pkgpath=`getconfig "FreeBSDPackages"` + counter=0 + IFS=$'\n' + for flightjob in `./mg-anoint retrieve --tolerate-unprepared \ + "freebsd build master $arch" \ + "freebsd-packages build master $arch"`; do + if [ $counter -eq 0 ]; then + # Anointed FreeBSD installer
Quoted 8 lines
I don't much like this code, but I'm having trouble saying what I think should be done instead.
I don't much like the $counter -eq 0 approach. Maybe some of it should go into a function ?
./mg-anoint retrieve ... >tmpfile
iffreebsd_want_anointed<tmpfile '' Dist ...
but not sure what the function should be.
Yes, the counter stuff is nasty. I guess I could change the output of mg-anoint so it signals which output line belongs to which input parameter, but there's always going to be multiple lines, and I wanted to avoid having to change much of mg-anoint.
Quoted 19 lines
Quoted 8 lines
mfi-common +7
-0+ local envvar="FREEBSD_${arch^^}_BUILDJOB" + local distpath=`getconfig "FreeBSDDist"` + if [ -n "${!envvar}" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsdbuildjob=${!envvar}" + elif [ -n "$FREEBSD_DIST" ] && [ -n "$FREEBSD_VERSION" ]; then + freebsd_runvars="$freebsd_runvars \mfi-common set_freebsd_runvars () { · line 192
Quoted 10 lines
mfi-common +10
-0+ freebsd_distpath=$FREEBSD_DIST/$arch \ + freebsd_version=$FREEBSD_VERSION" + elif [ -n "$distpath" ]; then + local version=`getconfig "FreeBSDVersion"` + freebsd_runvars="$freebsd_runvars \ + freebsd_distpath=$distpath/$arch \ + freebsd_version=$version" + elif [ "$flightjob" != "ERROR" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsdbuildjob=${flightjob/ /.}"
Quoted 3 lines
There seems like a lot of repetition here. For example,
FREEBSD_DISToverrides FreeBSDDist but /$arch is appended in two places. Maybe ${FREEBSD_DIST- ... something ... } would be better ?
OK, let me try to remove some of the duplication here.
Quoted 5 lines
It is difficult to see the wood for the trees, particularly with the constant repetition of
freebsd_runvars="$freebsd_runvars \
which could be avoided by having this fragment set a local variable containing the things to be added.
Hm, indeed. Using a local variable and then appending it when done would make it clearer.
Quoted 17 lines
Quoted 6 lines
mfi-common +6
-0+ elif [ $counter -eq 1 ]; then + # Anointed package repository + local envvar="FREEBSD_PACKAGES_${arch^^}_BUILDJOB" + local pkgpath=`getconfig "FreeBSDPackages"` + if [ -n "${!envvar}" ]; then + freebsd_runvars="$freebsd_runvars \mfi-common set_freebsd_runvars () { · line 209
Quoted 10 lines
mfi-common +10
-0+ freebsdpackagesbuildjob=${!envvar}" + elif [ -n "$FREEBSD_PACKAGES" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsd_packages=$FREEBSD_PACKAGES/$arch" + elif [ -n "$pkgpath" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsd_packages=$pkgpath/$arch" + elif [ "$flightjob" != "ERROR" ]; then + freebsd_runvars="$freebsd_runvars \ + freebsdpackagesbuildjob=${flightjob/ /.}"
Quoted 2 lines
This feels very similar to the code above, although it lacks the special handling for the version.
Maybe I can see about factoring some of this into a helper, but there are slight differences in both if branches that I'm not sure can be factored out.
Why I don't start by fixing the repetition of:
freebsd_runvars="$freebsd_runvars \... and we take it from there?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD
Roger Pau Monne writes ("Re: [Xen-devel] [PATCH 6/6] osstest: use a locally built pkg repository for FreeBSD"):
Quoted 7 lines · Ian Jackson
Quoted 3 lines
I realise this is a bit late to be saying this, but had you considered making the packages build a different step in the same job ? That might make a lot of this go away...
Do you mean to build the packages in build-prep instead of relying on having a custom binary repository?
No. Maybe I am confused. I thought your usual flight was
1 install anointed freebsd
2 build this freebsd
3+ build this package repo
4 install this freebsd (from step 2)
5 rebuild this freebsd (for testing that the build didn't break)
6 rebuild this package repo (") 7+ install this package repoo
8+ build xen
My question is why 2/3 and 5/6 are different jobs. If you made 2+3 a single job (with 2 and 3 being separate steps) then there would only need to be a single anointment.
Quoted 2 lines
IIRC the package building job takes a non-trivial amount of time (2-3h IIRC), because it has to build gcc (for SeaBIOS) and python, perl...
You could make the package building job optional if you only want to do it some of the time.
Quoted 9 lines
Quoted 7 lines
cr-daily-branch freebsd-*) · line 568
Quoted 4 lines
cr-daily-branch +4
-0+ # refkey: freebsd job: build-<arch>-freebsd + # refkey: freebsd-packages job: build-<arch>-freebsd-packages + anoint="$anoint \"$anointed\" $flight \ + build-$freebsd_arch-$freebsd_name"Maybe use an array variable for anount, and then you can avoid the shell \" quoting.
Please bear with me, but can you elaborate on this?
Roughly,
anoint=()
....
anoint+=("$anointed" $flight build-..)
...
./mg-anoint "${anoint[@]}"Note that the \" \" construct has gone, because there is now no additional layer of shell dequoting.
Quoted 5 lines
Quoted 3 lines
There seems like a lot of repetition here. For example,
FREEBSD_DISToverrides FreeBSDDist but /$arch is appended in two places. Maybe ${FREEBSD_DIST- ... something ... } would be better ?OK, let me try to remove some of the duplication here.
Thanks.
Another suggestion I mentioned IRL which I wanted to write down was:
maybe have mg-anoint have a reporting mode where it prints something suitable for shell `eval'.
Quoted 6 lines
Quoted 2 lines
This feels very similar to the code above, although it lacks the special handling for the version.
Maybe I can see about factoring some of this into a helper, but there are slight differences in both if branches that I'm not sure can be factored out.
Mmmm.
Quoted 2 lines
Why I don't start by fixing the repetition of:
freebsd_runvars="$freebsd_runvars \... and we take it from there?
Sure, let's see what you come up with.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel