RFC Trace updates and plugin RFC

Hi,

This series is a combination of general clean-ups to logging and tracing and a RFC for extending the tracepoint API by the addition of plugins. An example usage would be:

./aarch64-linux-user/qemu-aarch64 -d plugin,nochain -plugin file=trace/plugins/hotblocks.so,args=5 ./aarch64-linux-user/tests/sha1 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6
trace/plugins/hotblocks.so: collected 858d entries in the hash table

    pc: 0x00000000400260 (64004 hits) 4257 ns between returns
    pc: 0x00000000419280 (64004 hits) 4256 ns between returns
    pc: 0x00000000419290 (64003 hits) 4063 ns between returns
    pc: 0x000000004192d4 (64002 hits) 4063 ns between returns
    pc: 0x000000004192f0 (64002 hits) 4062 ns between returns

After reviewing Pavel's series I wondered if we could re-use the trace-point API as potential hooks for a plugin API. Trace points already exist as a series of interesting places in QEMU exposing information that can be used for analysis. By re-using them we avoid potential duplication of concerns. Adding new hook points becomes a simple case of adding a new trace point.

This also allows us to simplify some of the facilities provided to the plugins. I currently envision two types of plugin:

Filter plugins

These are plugins that sit at a trace point and return true/false to the trace code. This would short-circuit the trace from continuing, essentially filtering out unwanted information. The final data is still exported the usual way using trace APIs many output modes.

Analysis plugins

These are plugins that aggregate data in real-time to provide some sort of analysis as they go. They may or may not pass on the trace events to the other trace sub-systems. The example hotblocks plugin is such and example.

While the first set up plugins can re-use the existing trace infrastructure to output information the second type needs a reporting mechanism. As a result I have added a plugin_status() function which can either report on the monitor or in linux-user be called on program exit. In both cases the function simply returns a string and leaves the details of where it gets output to QEMU itself.

=Further Work=

==Known Limitations==

Currently there is only one hook allowed per trace event. We could make this more flexible or simply just error out if two plugins try and hook to the same point. What are the expectations of running multiple plugins hooking into the same point in QEMU?

==TCG Hooks==

Thanks to Lluís' work the trace API already splits up TCG events into translation time and exec time events and provides the machinery for hooking a trace helper into the translation stream. Currently that helper is unconditionally added although perhaps we could expand the call convention a little for TCG events to allow the translation time event to filter out planting the execution time helper?

Any serious analysis tool should allow for us to track all memory accesses so I think the guest_mem_before trace point should probably be split into guest_mem_before_store and guest_mem_after_load. We could go the whole hog and add potential trace points for start/end of all memory operations.

I did start down this road but the code quickly got ugly when dealing with our macrofied access functions. I plan to fish out the de-macrofication of softmmu series I posted earlier this year:

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03403.html

And see if that can be applied to all our access functions.

===Instruction Tracing===

Pavel's series had a specific hook for instrumenting individual instructions. I have not yet added it to this series but I think it be done in a slightly cleaner way now we have the ability to insert TCG ops into the instruction stream. If we add a tracepoint for post instruction generation which passes a buffer with the instruction translated and method to insert a helper before or after the instruction. This would avoid exposing the cpu_ldst macros to the plugins.

So what do people think? Could this be a viable way to extend QEMU with plugins?

Alex.

Aaron Lindsay (1):
trace: add linux-user plugin support

Alex Bennée (19):
util/log: allow -dfilter to stack
util/log: add qemu_dfilter_append_range()
linux-user: add -dfilter progtext shortcut
trace: enable the exec_tb trace events
trace: keep a count of trace-point hits
trace: show trace point counts in the monitor accel/tcg/cputlb: convert tlb_flush debugging into trace events accel/tcg/cputlb: convert remaining tlb_debug() to trace events
trace: suppress log output of trace points
qom/cpu: add a cpu_exit trace event
trace: expose a plugin fn pointer in TraceEvent
configure: expose a plugin to the trace-backends
tracetool: generate plugin snippets
trace: add infrastructure for building plugins
hmp: expose status of plugins to the monitor
linux-user: allow dumping of plugin status at end of run
plugins: add an example hotblocks plugin
plugins: add hotness summary to hotblocks
plugin: add tlbflush stats plugin

Pavel Dovgalyuk (1):
trace: add support for plugin infrastructure

 Makefile                            |   5 +-
 accel/tcg/cputlb.c                  |  99 +++++-----
 accel/tcg/trace-events              |  30 ++-
 configure                           |   6 +
 hmp-commands-info.hx                |  17 ++
 include/plugins/plugins.h           |  14 ++
 include/qemu/log.h                  |   2 +
 include/qemu/plugins.h              |  19 ++
 linux-user/exit.c                   |  12 ++
 linux-user/main.c                   |  31 ++-
 monitor.c                           |  19 +-
 qapi/trace.json                     |   3 +-
 qemu-options.hx                     |  10 +
 qom/cpu.c                           |   3 +
 qom/trace-events                    |   4 +
 scripts/tracetool/backend/plugin.py |  79 ++++++++
 scripts/tracetool/backend/simple.py |   2 +
 tests/test-logging.c                |  14 ++
 trace/Makefile.objs                 |  25 +++
 trace/control-internal.h            |   6 +
 trace/event-internal.h              |   4 +
 trace/plugins.c                     | 154 +++++++++++++++
 trace/plugins/Makefile.plugins      |  32 ++++
 trace/plugins/hotblocks/hotblocks.c | 100 ++++++++++
 trace/plugins/tlbflush/tlbflush.c   | 283 ++++++++++++++++++++++++++++
 trace/qmp.c                         |   1 +
 util/log.c                          |  34 +++-
 vl.c                                |  26 ++-
 28 files changed, 962 insertions(+), 72 deletions(-)
 create mode 100644 include/plugins/plugins.h
 create mode 100644 include/qemu/plugins.h
 create mode 100644 scripts/tracetool/backend/plugin.py
 create mode 100644 trace/plugins.c
 create mode 100644 trace/plugins/Makefile.plugins
 create mode 100644 trace/plugins/hotblocks/hotblocks.c
 create mode 100644 trace/plugins/tlbflush/tlbflush.c

2.17.1