Skip to contentSkip to conversation

Instruction counting instrumentation for ARM, 2nd version

PATCH In review Sami Kiminki 3 participants 6 messages Jun 12, 2009 – Jun 29, 2009 1 message not in the archive Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Instruction counting instrumentation for ARM, 2nd version

Hi,

Attached is the second version of Timo Töyry's instruction counting patch for arm-linux-user targets. Applies to QEMU 0.10.4. To try this, just add -instrcount to your regular qemu-arm command line. See [1] for description on how this is implemented.

The patch should now be complete for ARMv6, VFP and Thumb. However, Thumb2, NEON, and more advanced VFP instrumentation is still missing due to lacking public documentation from ARM. Measured overhead for instruction counting using this approach is between 5-35%, and overhead when the patch is applied but instruction counting disabled is very close to 0%. See below for details.

Instruction counting can be used to choosing the most appropriate CPU variant for an embedded application. It can also be used as a part of a more detailed performance analysis, as instruction mix reflects the performance of the code.

There are also other ways to implement instruction counting (see responses to our previous patch [1]). As to our approach, I believe this is the cleanest way to implement this specific instrumentation. However, from the general point of view, I'm not so sure. I'd like to ask for more comments and options.

Two main reasons for us for to construct this patch: First, we needed the functionality. Second, we're considering using QEMU as a platform for other instrumentation work too, so we thought to start with something simple enough.

What we would really like to see is a more general instrumentation framework, not just instruction counting and not just for ARM targets. The reason for this is that many measurements are so much easier to do in emulated/simulated environment than in real HW (e.g. instruction counting) and doing them in QEMU is way faster than using an ISA interpreter such as Valgrind. Some other instrumentation examples that come to my mind are cache usage efficiency analysis, branch profiling, and naturally instruction and memory access tracing.

A specific bonus of such framework would be that it could be used for the profiling part of possible future optimizations in code translation. Consider IA-32 EL (Intel's x86 emulator for Itanium).

Anyways, I'd like to know if there's a general interest in this kind of framework, and if so, how we should proceed to get it implemented into mainline.

Regards,
Sami Kiminki
Embedded Software Group / Helsinki University of Technology

References:
[1] http://lists.gnu.org/archive/html/qemu-devel/2009-05/msg00922.html

Overhead measurements
=====================
(Core 2 Q6600, QEMU compiled with gcc 4.3.3 and benchmarks with gcc 4.3.2 (arm1176jzf-s), average of 5 runs)

Vanilla QEMU:
- dhrystone: 1.709 s
- dhrystone thumb: 2.185 s
- mibench/jpeg/cjpeg: 0.453 s
- mibench/mad: 0.822 s

Patched QEMU, instruction counting disabled:
- dhrystone: 1.700 s (-0.5%)
- dhrystone thumb: 2.172 s (-0.6%)
- mibench/jpeg/cjpeg: 0.456 s (+0.7%)
- mibench/mad: 0.820 s (-0.2%)

Patched QEMU, instruction counting enabled:
- dhrystone: 1.882 s (+10%)
- dhrystone thumb: 2.300 s (+5%)
- mibench/jpeg/cjpeg: 0.513 s (+13%)
- mibench/mad: 1.078 s (+31%)

Makefile +4 -0
diff -urp qemu-0.10.4/Makefile qemu-0.10.4-instrcount/Makefile
@@ -179,6 +179,10 @@ libqemu_common.a: $(OBJS)
 # USER_OBJS is code used by qemu userspace emulation
 USER_OBJS=cutils.o  cache-utils.o
 
+# instruction count instrumentation
+USER_OBJS+=instrumentation.o
+
+
 libqemu_user.a: $(USER_OBJS)
 
 ######################################################################
Makefile.target +3 -1
diff -urp qemu-0.10.4/Makefile.target qemu-0.10.4-instrcount/Makefile.target
@@ -208,8 +208,10 @@ ifeq ($(findstring s390, $(TARGET_ARCH) 
 LIBOBJS+=s390-dis.o
 endif
 
-# libqemu
+# instrumentation support
+LIBOBJS+=instrumentation.o
 
+# libqemu
 libqemu.a: $(LIBOBJS)
 
 translate.o: translate.c cpu.h

Only in qemu-0.10.4-instrcount: instrumentation.c
Only in qemu-0.10.4-instrcount: instrumentation.h

linux-user/main.c +13 -2
diff -urp qemu-0.10.4/linux-user/main.c qemu-0.10.4-instrcount/linux-user/main.c
@@ -35,6 +35,8 @@
 
 #include "envlist.h"
 
+#include "instrumentation.h"
+
 #define DEBUG_LOGFILE "/tmp/qemu.log"
 
 char *exec_path;
@@ -2199,6 +2201,12 @@ static void usage(void)
            "-p pagesize  set the host page size to 'pagesize'\n"
            "-strace      log system calls\n"
            "\n"
+#ifdef TARGET_ARM
+           "Other:\n"
+           "-instrcount             Count instructions\n"
+           "-instrcountlog=qemu_log Output counters to qemu-log file\n"
+           "\n"
+#endif
            "Environment variables:\n"
            "QEMU_STRACE       Print system calls and arguments similar to the\n"
            "                  'strace' program.  Enable by setting to any value.\n"
@@ -2341,8 +2349,11 @@ int main(int argc, char **argv, char **e
             (void) envlist_unsetenv(envlist, "LD_PRELOAD");
         } else if (!strcmp(r, "strace")) {
             do_strace = 1;
-        } else
-        {
+        } else if (!strcmp(r, "instrcount")) {
+            instrumentation_count_instructions = 1;
+        } else if (!strcmp(r, "instrcountlog=qemu_log")) {
+            instrumentation_count_instructions_log = 1;
+        } else {
             usage();
         }
     }
linux-user/syscall.c +66 -0

90 lines not on this page · raw message

target-arm/cpu.h +344 -0

360 lines not on this page · raw message

target-arm/translate.c +745 -36

1,923 lines not on this page · raw message

#1 lore raw
Quoted 8 lines · Sami Kiminki

What we would really like to see is a more general instrumentation framework, not just instruction counting and not just for ARM targets. The reason for this is that many measurements are so much easier to do in emulated/simulated environment than in real HW (e.g. instruction counting) and doing them in QEMU is way faster than using an ISA interpreter such as Valgrind. Some other instrumentation examples that come to my mind are cache usage efficiency analysis, branch profiling, and naturally instruction and memory access tracing.

Valgrind isn't an interpreter. It's a JIT compiler like QEMU's TCG, so are you sure these things are way faster in QEMU?

What might work better would be to share some code from Valgrind/Cachegrind.

-- Jamie

#2 lore raw
Quoted 10 lines · Jamie Lokier
Quoted 8 lines · Sami Kiminki

What we would really like to see is a more general instrumentation framework, not just instruction counting and not just for ARM targets. The reason for this is that many measurements are so much easier to do in emulated/simulated environment than in real HW (e.g. instruction counting) and doing them in QEMU is way faster than using an ISA interpreter such as Valgrind. Some other instrumentation examples that come to my mind are cache usage efficiency analysis, branch profiling, and naturally instruction and memory access tracing.

Quoted 2 lines

Valgrind isn't an interpreter. It's a JIT compiler like QEMU's TCG, so are you sure these things are way faster in QEMU?

No, not anymore. Thanks for correcting ;)

However, what would still speak in favor of having some instrumentation support in QEMU, is measuring things at system-level. I assume this is pretty high in embedded system developers' wish-list.

Quoted 1 line

What might work better would be to share some code from Valgrind/Cachegrind.

Obviously.

- Sami

#3 lore raw
In reply to Sami Kiminki

On Fri, Jun 12, 2009 at 12:56 PM, Sami Kiminki<[email protected]> wrote:
[...]

Quoted 6 lines

There are also other ways to implement instruction counting (see responses to our previous patch [1]). As to our approach, I believe this is the cleanest way to implement this specific instrumentation. However, from the general point of view, I'm not so sure. I'd like to ask for more comments and options.

Apart from the fact your patch doesn't contain instrumentation.[ch] it is bogus in several places and shows why I think the approach is probably not the best.

E.g.:

@@ -6300,6 +6809,7 @@ static void disas_arm_insn(CPUState * en

                         if (insn & (1 << 20)) {
                             gen_helper_mark_exclusive(cpu_env, cpu_T[1]);
                             switch (op1) {
+                            instr_count_inc(ARM_INSTRUCTION_LDREX);
                             case 0: /* ldrex */
                                 tmp = gen_ld32(addr, IS_USER(s));
                                 break;

This is misplaced.

- if (op1 & 2)

+            if (op1 & 2) {
                 gen_helper_double_saturate(tmp2, tmp2);

- if (op1 & 1)
+ if (op1 & 1) instr_count_inc(ARM_INSTRUCTION_QDSUB);
+ else instr_count_inc(ARM_INSTRUCTION_QDADD);
+ }

+            if (op1 & 1) {
                 gen_helper_sub_saturate(tmp, tmp, tmp2);
-            else
+                instr_count_inc(ARM_INSTRUCTION_QSUB);
+            }
+            else {
                 gen_helper_add_saturate(tmp, tmp, tmp2);

+ instr_count_inc(ARM_INSTRUCTION_QADD);
+ }

Here your two last instr_count_inc will overwrite the first two ones. So you don't distinguish between QDADD/QDSUB and QADD/QSUB.

I also would like to see a more general instrumentation framework but your approach is too intrusive and so it's too easy to introduce hard to detect bugs.

Cheers,

Laurent

#6 lore raw

Re: [Qemu-devel] [PATCH] Instruction counting instrumentation for ARM, 2nd version & 2nd try

In reply to a message not in the archive

Apparently I succeeded in forgetting a switch when diffing and 2 files were missing. Attached is a new attempt.

- Sami

Quoted 7 lines · Laurent Desnogues

Hi Sami,

instruction.[ch] are not in your patch it seems :)

Cheers,

Laurent

Makefile +4 -0
diff -Nurp qemu-0.10.4/Makefile qemu-0.10.4-instrcount/Makefile
@@ -179,6 +179,10 @@ libqemu_common.a: $(OBJS)
 # USER_OBJS is code used by qemu userspace emulation
 USER_OBJS=cutils.o  cache-utils.o
 
+# instruction count instrumentation
+USER_OBJS+=instrumentation.o
+
+
 libqemu_user.a: $(USER_OBJS)
 
 ######################################################################
Makefile.target +3 -1
diff -Nurp qemu-0.10.4/Makefile.target qemu-0.10.4-instrcount/Makefile.target
@@ -208,8 +208,10 @@ ifeq ($(findstring s390, $(TARGET_ARCH) 
 LIBOBJS+=s390-dis.o
 endif
 
-# libqemu
+# instrumentation support
+LIBOBJS+=instrumentation.o
 
+# libqemu
 libqemu.a: $(LIBOBJS)
 
 translate.o: translate.c cpu.h
instrumentation.c +37 -0
diff -Nurp qemu-0.10.4/instrumentation.c qemu-0.10.4-instrcount/instrumentation.c
@@ -0,0 +1,37 @@
+/*
+ * instrument.c
+ *
+ *  Created on: May 14, 2009
+ *      Author: ttoyry
+ */
+
+#include "instrumentation.h"
+unsigned int instrumentation_count_instructions = 0;
+unsigned int instrumentation_count_instructions_log = 0;
+
+/*
+static instr_counter_offsets instr_offsets;
+
+static inline void instr_count_inc_init(uint32_t offset,  int instr)
+{
+    if (!instrumentation_count_instructions) return;
+    instr_offsets.cpustate_offset = offset;
+    TCGv tmp = new_tmp();
+    tcg_gen_ld_i32(tmp, cpu_env, instr_offsets.cpustate_offset + sizeof(uint32_t) * instr);
+    instr_offsets.tcg_offset[0] = gen_opparam_ptr - 1;
+    tcg_gen_addi_i32(tmp, tmp, 1);
+    tcg_gen_st_i32(tmp, cpu_env, instr_offsets.cpustate_offset + sizeof(uint32_t) * instr);
+    instr_offsets.tcg_offset[1] = gen_opparam_ptr - 1;
+    dead_tmp(tmp);
+}
+
+ Increment instruction counter
+static inline void instr_count_inc(int instr)
+{
+    if (!instrumentation_count_instructions) return;
+    *(instr_offsets.tcg_offset[0]) = instr_offsets.cpustate_offset + sizeof(uint32_t) * instr;
+    *(instr_offsets.tcg_offset[1]) = instr_offsets.cpustate_offset + sizeof(uint32_t) * instr;
+}
+*/
+
+
instrumentation.h +47 -0

47 lines not on this page · raw message

linux-user/main.c +13 -2
diff -Nurp qemu-0.10.4/linux-user/main.c qemu-0.10.4-instrcount/linux-user/main.c
@@ -35,6 +35,8 @@
 
 #include "envlist.h"
 
+#include "instrumentation.h"
+
 #define DEBUG_LOGFILE "/tmp/qemu.log"
 
 char *exec_path;
@@ -2199,6 +2201,12 @@ static void usage(void)
            "-p pagesize  set the host page size to 'pagesize'\n"
            "-strace      log system calls\n"
            "\n"
+#ifdef TARGET_ARM
+           "Other:\n"
+           "-instrcount             Count instructions\n"
+           "-instrcountlog=qemu_log Output counters to qemu-log file\n"
+           "\n"
+#endif
            "Environment variables:\n"
            "QEMU_STRACE       Print system calls and arguments similar to the\n"
            "                  'strace' program.  Enable by setting to any value.\n"
@@ -2341,8 +2349,11 @@ int main(int argc, char **argv, char **e
             (void) envlist_unsetenv(envlist, "LD_PRELOAD");
         } else if (!strcmp(r, "strace")) {
             do_strace = 1;
-        } else
-        {
+        } else if (!strcmp(r, "instrcount")) {
+            instrumentation_count_instructions = 1;
+        } else if (!strcmp(r, "instrcountlog=qemu_log")) {
+            instrumentation_count_instructions_log = 1;
+        } else {
             usage();
         }
     }
linux-user/syscall.c +66 -0

90 lines not on this page · raw message

target-arm/cpu.h +344 -0

360 lines not on this page · raw message

target-arm/translate.c +745 -36

1,923 lines not on this page · raw message

#4 lore raw

PATCH 1/1 Instruction counting instrumentation for ARM v3

In reply to Laurent Desnogues

Hi,

Attached is a version of Timo Töyry's patch which should fix the issues pointed out below. I'm not expecting this to get committed, but thought to share it anyway, should anyone have use for it.

- Sami

Quoted 47 lines · Laurent Desnogues

On Fri, Jun 12, 2009 at 12:56 PM, Sami Kiminki<[email protected]> wrote:
[...]

Quoted 6 lines

There are also other ways to implement instruction counting (see responses to our previous patch [1]). As to our approach, I believe this is the cleanest way to implement this specific instrumentation. However, from the general point of view, I'm not so sure. I'd like to ask for more comments and options.

Apart from the fact your patch doesn't contain instrumentation.[ch] it is bogus in several places and shows why I think the approach is probably not the best.

E.g.:

(quoted diff) +1 -0
@@ -6300,6 +6809,7 @@ static void disas_arm_insn(CPUState * en
                         if (insn & (1 << 20)) {
                             gen_helper_mark_exclusive(cpu_env, cpu_T[1]);
                             switch (op1) {
+                            instr_count_inc(ARM_INSTRUCTION_LDREX);
                             case 0: /* ldrex */
                                 tmp = gen_ld32(addr, IS_USER(s));
                                 break;

This is misplaced.

(quoted diff) +10 -3
-            if (op1 & 2)
+            if (op1 & 2) {
                 gen_helper_double_saturate(tmp2, tmp2);
-            if (op1 & 1)
+                if (op1 & 1) instr_count_inc(ARM_INSTRUCTION_QDSUB);
+                else instr_count_inc(ARM_INSTRUCTION_QDADD);
+            }
+            if (op1 & 1) {
                 gen_helper_sub_saturate(tmp, tmp, tmp2);
-            else
+                instr_count_inc(ARM_INSTRUCTION_QSUB);
+            }
+            else {
                 gen_helper_add_saturate(tmp, tmp, tmp2);
+                instr_count_inc(ARM_INSTRUCTION_QADD);
+            }

Here your two last instr_count_inc will overwrite the first two ones. So you don't distinguish between QDADD/QDSUB and QADD/QSUB.

Quoted 7 lines

I also would like to see a more general instrumentation framework but your approach is too intrusive and so it's too easy to introduce hard to detect bugs.

Cheers,

Laurent

Makefile +4 -0
diff -Nurp qemu-0.10.4/Makefile qemu-0.10.4-instrcount/Makefile
@@ -179,6 +179,10 @@ libqemu_common.a: $(OBJS)
 # USER_OBJS is code used by qemu userspace emulation
 USER_OBJS=cutils.o  cache-utils.o
 
+# instruction count instrumentation
+USER_OBJS+=instrumentation.o
+
+
 libqemu_user.a: $(USER_OBJS)
 
 ######################################################################
Makefile.target +3 -1
diff -Nurp qemu-0.10.4/Makefile.target qemu-0.10.4-instrcount/Makefile.target
@@ -208,8 +208,10 @@ ifeq ($(findstring s390, $(TARGET_ARCH) 
 LIBOBJS+=s390-dis.o
 endif
 
-# libqemu
+# instrumentation support
+LIBOBJS+=instrumentation.o
 
+# libqemu
 libqemu.a: $(LIBOBJS)
 
 translate.o: translate.c cpu.h
instrumentation.c +37 -0
diff -Nurp qemu-0.10.4/instrumentation.c qemu-0.10.4-instrcount/instrumentation.c
@@ -0,0 +1,37 @@
+/*
+ * instrument.c
+ *
+ *  Created on: May 14, 2009
+ *      Author: ttoyry
+ */
+
+#include "instrumentation.h"
+unsigned int instrumentation_count_instructions = 0;
+unsigned int instrumentation_count_instructions_log = 0;
+
+/*
+static instr_counter_offsets instr_offsets;
+
+static inline void instr_count_inc_init(uint32_t offset,  int instr)
+{
+    if (!instrumentation_count_instructions) return;
+    instr_offsets.cpustate_offset = offset;
+    TCGv tmp = new_tmp();
+    tcg_gen_ld_i32(tmp, cpu_env, instr_offsets.cpustate_offset + sizeof(uint32_t) * instr);
+    instr_offsets.tcg_offset[0] = gen_opparam_ptr - 1;
+    tcg_gen_addi_i32(tmp, tmp, 1);
+    tcg_gen_st_i32(tmp, cpu_env, instr_offsets.cpustate_offset + sizeof(uint32_t) * instr);
+    instr_offsets.tcg_offset[1] = gen_opparam_ptr - 1;
+    dead_tmp(tmp);
+}
+
+ Increment instruction counter
+static inline void instr_count_inc(int instr)
+{
+    if (!instrumentation_count_instructions) return;
+    *(instr_offsets.tcg_offset[0]) = instr_offsets.cpustate_offset + sizeof(uint32_t) * instr;
+    *(instr_offsets.tcg_offset[1]) = instr_offsets.cpustate_offset + sizeof(uint32_t) * instr;
+}
+*/
+
+
instrumentation.h +47 -0

47 lines not on this page · raw message

linux-user/main.c +13 -2
diff -Nurp qemu-0.10.4/linux-user/main.c qemu-0.10.4-instrcount/linux-user/main.c
@@ -35,6 +35,8 @@
 
 #include "envlist.h"
 
+#include "instrumentation.h"
+
 #define DEBUG_LOGFILE "/tmp/qemu.log"
 
 char *exec_path;
@@ -2199,6 +2201,12 @@ static void usage(void)
            "-p pagesize  set the host page size to 'pagesize'\n"
            "-strace      log system calls\n"
            "\n"
+#ifdef TARGET_ARM
+           "Other:\n"
+           "-instrcount             Count instructions\n"
+           "-instrcountlog=qemu_log Output counters to qemu-log file\n"
+           "\n"
+#endif
            "Environment variables:\n"
            "QEMU_STRACE       Print system calls and arguments similar to the\n"
            "                  'strace' program.  Enable by setting to any value.\n"
@@ -2341,8 +2349,11 @@ int main(int argc, char **argv, char **e
             (void) envlist_unsetenv(envlist, "LD_PRELOAD");
         } else if (!strcmp(r, "strace")) {
             do_strace = 1;
-        } else
-        {
+        } else if (!strcmp(r, "instrcount")) {
+            instrumentation_count_instructions = 1;
+        } else if (!strcmp(r, "instrcountlog=qemu_log")) {
+            instrumentation_count_instructions_log = 1;
+        } else {
             usage();
         }
     }
linux-user/syscall.c +66 -0

90 lines not on this page · raw message

target-arm/cpu.h +350 -0

366 lines not on this page · raw message

target-arm/translate.c +765 -36

1,967 lines not on this page · raw message