Running benchmarks via OSSTest

Hello everyone,

This is a highly reworked and much more mature version of this old RFC series:

http://lists.xen.org/archives/html/xen-devel/2014-06/msg03429.html

It is about integrating benchmarking capabilities into OSSTest. The series is a lot bigger, because it is now capable of doing a lot more stuff. :-) All the comments I got during review of the RFC have been addressed, of course... More comments are of course welcome!

The series is available here:

git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest
http://xenbits.xen.org/gitweb/?p=people/dariof/osstest.git;a=shortlog;h=refs/heads/benchmarking-with-osstest

So, about the patches. Patches 01 to 04, as well as patch 20, are preliminary changes and fixes, necessary for the rest of the series to work properly, but they are general enough to be considered separately from the series, if interesting.

Patches 05 to 12 make it possible to run an instance of the Unixbench (XXX) benchmark via OSSTest. Management scripts, test scripts, jobs and recipes are provided to download the benchmark, prepare the guest (PV or HVM), build the benchmark in the guest, run it, and retrieve and process (include some super fancy plots! :-D) the results.

To do so, do as follows (in standalone mode):

  $ ./standalone-reset -t bench
  $ ./mg-unixbench-download
  $ ./standalone run-job -h ultralisk bench-unixbench-amd64-credit-pv-4vcpus-4096ram

One thing about patch 12. I tried, but I can't tell how I should invoke ./standalone in order to test the new parameter I'm adding... Can someone help with this? IanC?

To have an idea of how the 'output' of one of this jobs looks like, have a look here:

http://xenbits.xen.org/people/dariof/osstest-bench/bench-kernbench-amd64-credit-hvm-4vcpus-4096ram.log
http://xenbits.xen.org/people/dariof/osstest-bench/bench-kernbench-amd64-credit-hvm-4vcpus-4096ram/
http://xenbits.xen.org/people/dariof/osstest-bench/bench-kernbench-amd64-credit-hvm-4vcpus-4096ram/debianhvm.guest.osstest--kernbench-n4-PLOT.png

Or here:

http://xenbits.xen.org/people/dariof/osstest-bench/bench-unixbench-amd64-credit-pv-4vcpus-4096ram.log
http://xenbits.xen.org/people/dariof/osstest-bench/bench-unixbench-amd64-credit-pv-4vcpus-4096ram/
http://xenbits.xen.org/people/dariof/osstest-bench/bench-unixbench-amd64-credit-pv-4vcpus-4096ram/debian.guest.osstest--unixbench-i1-c4-PLOT.png

Patches 13 to 19 does the very same, but for Kernbench (XXX):

  $ ./mg-kernbench-download
  $ ./standalone run-job -h ultralisk bench-kernbench-amd64-credit-hvm-4vcpus-4096ram

Patches 21 to 27 aims at allowing running a benchmark both in a guest and on baremetal, to investigate the performances loss due to the virtualization overhead. To do that, for Unixbench and Kernbench, one just issue the following commands (in standalone mode):

  $ ./standalone run-job -h ultralisk bench-hostcmp-unixbench-amd64-credit
  $ ./standalone run-job -h ultralisk bench-hostcmp-kernbench-amd64-credit2

To have an idea of how the 'output' of one of these jobs looks like, have a look here:

http://xenbits.xen.org/people/dariof/osstest-bench/bench-hostcmp-kernbench-amd64-credit.log
http://xenbits.xen.org/people/dariof/osstest-bench/bench-hostcmp-kernbench-amd64-credit/
http://xenbits.xen.org/people/dariof/osstest-bench/bench-hostcmp-kernbench-amd64-credit/bench-hostcmp-kernbench-amd64-credit-PLOT.png

TODOs:
* in patch 02/27, I haven't actually checked whether also grub1 and/or uboot needs fixing
* When comparing performance, I think it would be best to always use the same kernel, in host and guests. This happens by default for PV guest, while it does not for HVM guests. This of course can be solved pretty easily, and I plan to do it either in future posting of this series, or with a follow-up patch.

Regards,
Dario
---
Dario Faggioli (27):
ts-devbian-hvm-install: prune "cdrom:" from install sources Osstest/Debian.pm: fix identifying a Linux baremetal grub2 entry Guest setup: allow the amount of RAM to be a runvar Osstest/TestSupport.pm: Introduce target_getfile_[root_]stash()
mg-unixbench-download: new script for downloading the unixbench archive
ts-unixbench-build: prep the environment for running unixbench
ts-unixbench-run: kick off the benchmark on the target
ts-unixbench-reslts: for retrieving the results
ts-unixbench-reslts: process and plot bench results
sg-run-job: recipes for the unixbench jobs
make-bench-flight: to create a benchmarking flight
standalone-reset: introduce a new -t option
mg-kernbench-download: new script for downloading kernbench
ts-kernbench-build: prep the environment for running kernbench
ts-kernbench-run: kick off the benchmark on the target
ts-unixbench-reslts: retrieve and stash kernbench results
ts-kernbench-reslts: process and plot bench results
sg-run-job: recipes for the kernbench jobs
make-bench-flight: create kernbench jobs Osstest/TestSupport.pm: read hosts' hardware characteristics ts-bench-hostcmp-guest-prep: new script ts-bench-hostcmp-host-prep: new script ts-bench-hostcmp-host-reset: new script
Recipes and jobs for running unixbench both on host and guest
ts-bench-hostcmp-post: add plotting facilities
Kernbench perf comparison between host and guest
ts-bench-hostcmp-post: add plotting facilities

 Osstest/Benchmarking.pm     |  210 +++++++++++++++++++++++++++++++++++++++++++
 Osstest/Debian.pm           |   54 +++++++----
 Osstest/TestSupport.pm      |  166 +++++++++++++++++++++++++++++++++-
 README                      |   36 +++++++
 cr-daily-branch             |    3 -
 make-bench-flight           |  153 +++++++++++++++++++++++++++++++
 mg-host-hw-specs            |   35 +++++++
 mg-kernbench-download       |   49 ++++++++++
 mg-unixbench-download       |   40 ++++++++
 sg-run-job                  |  102 ++++++++++++++++++++-
 standalone                  |    7 +
 standalone-reset            |    9 +-
 ts-bench-hostcmp-guest-prep |   87 ++++++++++++++++++
 ts-bench-hostcmp-host-prep  |   74 +++++++++++++++
 ts-bench-hostcmp-post       |   81 +++++++++++++++++
 ts-debian-fixup             |    3 +
 ts-debian-hvm-install       |    3 -
 ts-kernbench-build          |   86 ++++++++++++++++++
 ts-kernbench-reslts         |   75 +++++++++++++++
 ts-kernbench-run            |   63 +++++++++++++
 ts-unixbench-build          |   77 ++++++++++++++++
 ts-unixbench-reslts         |   84 +++++++++++++++++
 ts-unixbench-run            |   67 ++++++++++++++
 ts-xen-install              |   38 --------
 24 files changed, 1534 insertions(+), 68 deletions(-)
 create mode 100644 Osstest/Benchmarking.pm
 create mode 100755 make-bench-flight
 create mode 100755 mg-host-hw-specs
 create mode 100755 mg-kernbench-download
 create mode 100755 mg-unixbench-download
 create mode 100755 ts-bench-hostcmp-guest-prep
 create mode 100755 ts-bench-hostcmp-host-prep
 create mode 100755 ts-bench-hostcmp-post
 create mode 100755 ts-kernbench-build
 create mode 100755 ts-kernbench-reslts
 create mode 100755 ts-kernbench-run
 create mode 100755 ts-unixbench-build
 create mode 100755 ts-unixbench-reslts
 create mode 100755 ts-unixbench-run

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)