RFC Running benchmarks via OSSTest

Hello Everyone,

This is something I've been working in the latest few days, and it's about making it possible to run benchmarks via OSSTest. I've (of course) tested it in standalone mode, but nothing forbids to integrate it in our push gate, as we were saying at the hackathon. Of course, there are a lot of things to be decided. Where to run benchmarks, which benchmarks, how frequently, what to do with the results, etc, but it's a start! :-P

So, in standalone mode, after applying the patches, you can do the following, and you'll have a PV guest running unixbench for a while, and the results fetched for you on the OSSTest standalone controller stash area:

# Retrieve the benchmark archive
./mg-unixbench-download
# Reset, create the benchmarking flight, install the host and build Xen and Linux ./standalone-reset -t bench
./standalone run-job -R -h ghoul3 build-amd64 # consider --lvextendmax=XXGB, if necessary
./standalone run-job -h ghoul3 build-amd64-pvops
# Or set-paths --if build happened already-- like this:
#./standalone set-paths build-amd64
#./standalone set-paths build-amd64-pvops
# Install Xen on the host
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-xen-install ./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-host-reboot # Install (and start) the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-debian-install ./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-debian-fixup debian ./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-guest-start debian # Prepare the benchmarking environment in the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-prep debian # Run the benchmark in the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-run debian # Retrieve the benchmark results:
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-reslts debian

A few jobs are being created. In case you want to use one, instead of invoking all the ts-s, run the following:

# Retrieve the benchmark archive
./mg-unixbench-download
# Reset, create the benchmarking flight, install the host and build Xen and Linux ./standalone-reset -t bench
./standalone run-job -R -h ghoul3 build-amd64 # consider --lvextendmax=XXGB, if necessary
./standalone run-job -h ghoul3 build-amd64-pvops
# Run the benchmarking job:
./standalone run-job -h ghoul3 bench-unixbench-amd64-amd64-xl-credit2

Almost all the loginc to use an HVM guest are there, a just need a few fixups in ts-debian-hvm-install that I'll submit as a separate patch.

I tested both the ways in standalone mode, and it seems to be working here.

And that's it... So, what do you think about it? IanC? IanJ?

I now it's still a sort of a prototype, but I'd love to hear your thoughts before proceeding any further.

One thing about the last two patches. They are there to show how the code will look like if we take a slightly different path than what's outlined in the first part of the series. The idea is, in fact, to have the mg-xxx-download and the ts-xxx-{prep,run,reslts} scripts to be more generic, accept one (or more) benchmark name(s) as parameter and prepare the target (host or guest) for them. That would make the single scripts slightly more complex, but will avoid having to have 4 scripts (downloading, preparing, running and fetching results) for each new benchmark we want to introduce support for.

Personally, I think I prefer them to be generic, i.e., something similar to what they look like after all the series is applied, as compared to how they look only until patch 7. This is exacly an example of something I'd like to hear your thoughts about before going forward. :-)

The series is also available here:

git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest-RFC

Regards,
Dario
---
Dario Faggioli (9):
mg-unixbench-download: new script for downloading the unixbench archive
ts-unixbench-prep: prep the environment for running unixbench
ts-unixbench-run: kick off the benchmark on the target
ts-unixbench-reslts: for retrieving the results
sg-run-job: new recipe for a unixbench job
make-bench-flight: to create a benchmarking flight
make-flight: introduce a new -t option
mg-unixbench-download: make it generic
ts-unixbench-prep: make it generic

 ap-common             |    8 ++++
 make-bench-flight     |   92 ++++++++++++++++++++++++++++++++++++++++++++++
 mg-unixbench-download |   49 ++++++++++++++++++++++++
 sg-run-job            |   28 ++++++++++++++
 standalone-reset      |    9 +++-
 ts-unixbench-prep     |   99 +++++++++++++++++++++++++++++++++++++++++++++++++
 ts-unixbench-reslts   |   56 ++++++++++++++++++++++++++++
 ts-unixbench-run      |   45 ++++++++++++++++++++++
 8 files changed, 383 insertions(+), 3 deletions(-)
 create mode 100755 make-bench-flight
 create mode 100755 mg-unixbench-download
 create mode 100755 ts-unixbench-prep
 create mode 100755 ts-unixbench-reslts
 create mode 100755 ts-unixbench-run

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)