Skip to contentSkip to conversation

XTF: add micro benchmarks

PATCH New Norbert Manthey 1 participant 5 messages Mar 23, 2020

lore
Share & tools
lore
#0 lore raw

cover XTF: add micro benchmarks

Dear all,

I added a benchmark category to XTF, and added functions to measure time in the guest. Finally, I added a first micro benchmark that measures the time to call a specified hypercall, and print the average time the hypercall takes.

The added category should be useful to implement further micro benchmarks. I already implemented a few more that I will publish once the environment is agreed on.

Best,
Norbert

Norbert Manthey (4):
categories: add benchmark
time: add stubs
time: provide measurement template
perf: measure MMUEXT_MARK_SUPER test

 build/common.mk                               |   2 +-
 build/files.mk                                |   1 +
 common/time.c                                 | 279 ++++++++++++++++++
 include/xtf/time.h                            |  81 +++++
 tests/perf-PV-MMUEXT_MARK_SUPER-noop/Makefile |   9 +
 tests/perf-PV-MMUEXT_MARK_SUPER-noop/main.c   |  80 +++++
 xtf-runner                                    |   2 +-
 7 files changed, 452 insertions(+), 2 deletions(-)
 create mode 100644 common/time.c
 create mode 100644 include/xtf/time.h
 create mode 100644 tests/perf-PV-MMUEXT_MARK_SUPER-noop/Makefile
 create mode 100644 tests/perf-PV-MMUEXT_MARK_SUPER-noop/main.c
Signature · 20 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879




_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/4 categories: add benchmark

As XTF allows to write tests that interact with the hypervisor, we would like to use this capability to implement micro benchmarks, so that we can measure the performance impact of modifications to the hypervisor.

This change introduces a category benchmark, which can be used as container for tests of this kind.

---
 build/common.mk | 2 +-
 xtf-runner      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
build/common.mk +1 -1
@@ -1,4 +1,4 @@
-ALL_CATEGORIES     := special functional xsa utility in-development
+ALL_CATEGORIES     := special functional xsa utility in-development benchmark
 
 ALL_ENVIRONMENTS   := pv64 pv32pae hvm64 hvm32pae hvm32pse hvm32
 
xtf-runner +1 -1
@@ -43,7 +43,7 @@ def exit_code(state):
 
 # All test categories
 default_categories     = set(("functional", "xsa"))
-non_default_categories = set(("special", "utility", "in-development"))
+non_default_categories = set(("special", "utility", "in-development", "benchmark"))
 all_categories         = default_categories | non_default_categories
 
 # All test environments
Signature · 20 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879




_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH 2/4 time: add stubs

In reply to Norbert Manthey

To measure how long a certain interaction takes, we need time primitives. This commit introduces these primitives, so that future tests can use the gettimeofday function to retrieve the current time.

---
 build/files.mk     |   1 +
 common/time.c      | 203 +++++++++++++++++++++++++++++++++++++++++++++
 include/xtf/time.h |  66 +++++++++++++++
 3 files changed, 270 insertions(+)
 create mode 100644 common/time.c
 create mode 100644 include/xtf/time.h
build/files.mk +1 -0
@@ -16,6 +16,7 @@ obj-perarch += $(ROOT)/common/libc/vsnprintf.o
 obj-perarch += $(ROOT)/common/report.o
 obj-perarch += $(ROOT)/common/setup.o
 obj-perarch += $(ROOT)/common/xenbus.o
+obj-perarch += $(ROOT)/common/time.o
 
 obj-perenv += $(ROOT)/arch/x86/decode.o
 obj-perenv += $(ROOT)/arch/x86/desc.o
common/time.c new file +203 -0

203 lines not on this page · raw message

include/xtf/time.h new file +66 -0

66 lines not on this page · raw message

Signature · 20 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879




_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#3 lore raw

PATCH 3/4 time: provide measurement template

In reply to Norbert Manthey

The added function measure_performance allows to measure the run time of a function, by computing the average time it takes to call that function a given number of retries. The measured total time is returned in nano seconds. Furthermore, the value is printed via printk in a fixed format, to allow processing the output further.

This format is, where average-time provides ns with ps granularity:

perf test_name <average-time> ns

---
 common/time.c      | 76 ++++++++++++++++++++++++++++++++++++++++++++++
 include/xtf/time.h | 15 +++++++++
 2 files changed, 91 insertions(+)
common/time.c +76 -0
@@ -192,6 +192,82 @@ void msleep(uint64_t t)
     mspin_sleep(t);
 }
 
+unsigned long cycles2nsec(uint64_t cycles)
+{
+    return scale_delta(cycles,
+            shared_info.vcpu_info[0].time.tsc_to_system_mul,
+            shared_info.vcpu_info[0].time.tsc_shift);
+}
+
+unsigned long measure_performance(const char* test_name,
+                                  const char* function_name,
+                                  function_under_test_t call,
+                                  unsigned long measure_seconds,
+                                  unsigned long callibration_calls,
+                                  int print_times)
+{
+    unsigned long start, end;  // time stamps for before and after the execution
+    unsigned long calibration_ns, total_measured_ns, avg_ns, avg_ps_fraction;
+    unsigned long counter;
+    unsigned long measure_calls;
+    int rc = 0;
+
+
+    /* Calibrate by measuring time for given amount of callibration calls*/
+    printk("Calibrate %s by calling %lu times\n", function_name,
+                                                  callibration_calls);
+    start = rdtscp();
+    for(counter = 0; counter < callibration_calls; ++ counter)
+    {
+        rc = call();
+    }
+    end = rdtscp();
+
+    /* Calculate the total number in nano seconds */
+    calibration_ns = cycles2nsec(end - start);
+
+    /* Calculate number of calls for about measure_seconds based on
+       (callibration_calls / calibration_ns) equals
+       (measure_calls / (measure_seconds * 1000 * 1000 * 1000)) */
+    measure_calls = ((measure_seconds * 1000UL * 1000UL * 1000UL)
+                     / calibration_ns) * callibration_calls;
+
+    printk("Measure %s by calling %lu times\n", function_name, measure_calls);
+
+    /* Perform all calls, measure start and end time */
+    start = rdtscp();
+    for(counter = 0; counter < measure_calls; ++ counter)
+    {
+        rc = call();
+    }
+    end = rdtscp();
+
+    /* Calculate the total number in nano seconds */
+    total_measured_ns = cycles2nsec(end - start);
+    avg_ns = total_measured_ns / measure_calls;
+    avg_ps_fraction = (total_measured_ns / (measure_calls/1000)) % 1000;
+
+    if(print_times)
+    {
+        /* Show the result of the last query */
+        printk("%s's last return value: %d\n", function_name, rc);
+
+        /* Print average time and total time */
+        printk("Avg %s call time: avg: %ld.%s%ld ns total: %ld ns\n",
+               function_name, avg_ns,
+               avg_ps_fraction < 10 ? "00" : (avg_ps_fraction < 100 ? "0" : ""),
+               avg_ps_fraction, total_measured_ns);
+
+        /* Print performance value */
+        printk("perf %s %ld.%s%ld ns\n", test_name, avg_ns,
+               avg_ps_fraction < 10 ? "00" : (avg_ps_fraction < 100 ? "0" : ""),
+               avg_ps_fraction);
+    }
+
+    /* Return average run time in pico seconds */
+    return avg_ns * 1000 + avg_ps_fraction;
+}
+
 /*
  * Local variables:
  * mode: C
include/xtf/time.h +15 -0
@@ -53,6 +53,21 @@ int gettimeofday(struct timeval *tp);
 /* This returns the current epoch time */
 #define NOW() current_time()
 
+unsigned long cycles2nsec(uint64_t cycles);
+
+/* Signature of a function to be called for measurement */
+typedef int (*function_under_test_t)(void);
+
+/* Measure the time it takes to call the passed function. Measure for a given
+   amount of time after calibrating for a given amount of calls. Returns the
+   average run time of the measure call in pico seconds. */
+unsigned long measure_performance(const char* test_name,
+                                  const char* function_name,
+                                  function_under_test_t call,
+                                  unsigned long measure_seconds,
+                                  unsigned long callibration_calls,
+                                  int print_times);
+
 #endif /* XTF_TIME_H */
 
 /*
Signature · 20 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879




_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

PATCH 4/4 perf: measure MMUEXT_MARK_SUPER test

In reply to Norbert Manthey

A first simple test is to call a hypercall in a tight loop. To measure implementation aspects of the hypervisor, we picked a hypercall that is not implemented and hence results in a no-op, namely the hypercall mmuext_op with the command MMUEXT_MARK_SUPER.

The test calibrates the execution time for 1000 calls to the hypercall, and next calculates the number of calls to take about 5 minutes.

---
 tests/perf-PV-MMUEXT_MARK_SUPER-noop/Makefile |  9 +++
 tests/perf-PV-MMUEXT_MARK_SUPER-noop/main.c   | 80 +++++++++++++++++++
 2 files changed, 89 insertions(+)
 create mode 100644 tests/perf-PV-MMUEXT_MARK_SUPER-noop/Makefile
 create mode 100644 tests/perf-PV-MMUEXT_MARK_SUPER-noop/main.c
tests/perf-PV-MMUEXT_MARK_SUPER-noop/Makefile new file +9 -0
new file mode 100644
@@ -0,0 +1,9 @@
+include $(ROOT)/build/common.mk
+
+NAME      := perf-PV-MMUEXT_MARK_SUPER-noop
+CATEGORY  := benchmark
+TEST-ENVS := pv64
+
+obj-perenv += main.o
+
+include $(ROOT)/build/gen.mk
tests/perf-PV-MMUEXT_MARK_SUPER-noop/main.c new file +80 -0

80 lines not on this page · raw message

Signature · 20 lines
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879




_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel