Rewrite jevents program in python
Ian Rogers
reviewerv7 cover Rewrite jevents program in python
New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.
The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.
A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.
v7. Adds Tested-by from John Garry <[email protected]>. v6. Adds Tested-by from Zhengjun Xing <[email protected]>
and Thomas Richter <[email protected]>. Fixes issues spotted by Jiri Olsa <[email protected]>, jsmn.c wasn't deleted and the empty pmu-events.c didn't pass the pmu-events test. It also adds a missing mkdir which is necessary before creating pmu-events.c and tweaks the quiet display code in the empty case to display a gen.
v5. Adds a 2>/dev/null as suggested by David Laight
<[email protected]>.
v4. Fixes the build for systems with python2 installed by adding a
patch that makes python3 the preferred python (Reported-by: John Garry <[email protected]>). It also fixes a bash-ism in the jevents-test.sh and fixes the handling of an empty string for a metric BriefDescription as one was added for sapphirerapids in the metric Execute.
v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the generated pmu-events.c isn't altered at all by this change. Adds the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't introduced until 3.6.
v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the build for python not being present or too old (Suggested-by: Peter Zijlstra <[email protected]>/John Garry <[email protected]>).
Ian Rogers (4):
perf python: Prefer python3
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c
tools/perf/Makefile.config | 27 +- tools/perf/Makefile.perf | 16 +- tools/perf/pmu-events/Build | 17 +- tools/perf/pmu-events/empty-pmu-events.c | 158 +++ tools/perf/pmu-events/jevents.c | 1342 ---------------------- tools/perf/pmu-events/jevents.py | 409 +++++++ tools/perf/pmu-events/jsmn.c | 352 ------ tools/perf/pmu-events/jsmn.h | 68 -- tools/perf/pmu-events/json.c | 162 --- tools/perf/pmu-events/json.h | 39 - 10 files changed, 604 insertions(+), 1986 deletions(-) create mode 100644 tools/perf/pmu-events/empty-pmu-events.c delete mode 100644 tools/perf/pmu-events/jevents.c create mode 100755 tools/perf/pmu-events/jevents.py delete mode 100644 tools/perf/pmu-events/jsmn.c delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h
Signature · 1 line
2.37.0.rc0.161.g10f37bed90-goog
Ian Rogers
reviewerPATCH v7 1/4 perf python: Prefer python3
From: Ian Rogers <[email protected]>
The PYTHON_AUTO code orders the preference for the PYTHON command to be python3, python and then python2. python3 makes a more logical preference as python2 is no longer supported:
https://www.python.org/doc/sunset-python-2/
Reorder the priority of the PYTHON command to be python2, python and then python3.
--- tools/perf/Makefile.config | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
tools/perf/Makefile.config +4 -4
@@ -241,15 +241,15 @@ endif # Try different combinations to accommodate systems that only have # python[2][3]-config in weird combinations in the following order of # priority from lowest to highest: -# * python3-config -# * python-config # * python2-config as per pep-0394. +# * python-config +# * python3-config # * $(PYTHON)-config (If PYTHON is user supplied but PYTHON_CONFIG isn't) # PYTHON_AUTO := python-config -PYTHON_AUTO := $(if $(call get-executable,python3-config),python3-config,$(PYTHON_AUTO)) -PYTHON_AUTO := $(if $(call get-executable,python-config),python-config,$(PYTHON_AUTO)) PYTHON_AUTO := $(if $(call get-executable,python2-config),python2-config,$(PYTHON_AUTO)) +PYTHON_AUTO := $(if $(call get-executable,python-config),python-config,$(PYTHON_AUTO)) +PYTHON_AUTO := $(if $(call get-executable,python3-config),python3-config,$(PYTHON_AUTO)) # If PYTHON is defined but PYTHON_CONFIG isn't, then take $(PYTHON)-config as if it was the user # supplied value for PYTHON_CONFIG. Because it's "user supplied", error out if it doesn't exist.
Signature · 1 line
2.37.0.rc0.161.g10f37bed90-goog
Ian Rogers
reviewerPATCH v7 2/4 perf jevents: Add python converter script
In reply to Ian RogersFrom: Ian Rogers <[email protected]>
jevents.c is large, has a dependency on an old forked version of jsmn, and is challenging to work upon. A lot of jevents.c's complexity comes from needing to write json and csv parsing from first principles. In contrast python has this functionality in standard libraries and is already a build pre-requisite for tools like asciidoc (that builds all of the perf man pages).
Introduce jevents.py that produces identical output to jevents.c. Add a test that runs both converter tools and validates there are no output differences. The test can be invoked with a phony build target like:
make -C tools/perf jevents-py-test
The python code deliberately tries to replicate the behavior of jevents.c so that the output matches and transitioning tools shouldn't introduce regressions. In some cases the code isn't as elegant as hoped, but fixing this can be done as follow up.
--- tools/perf/Makefile.perf | 6 + tools/perf/pmu-events/jevents-test.sh | 33 +++ tools/perf/pmu-events/jevents.py | 409 ++++++++++++++++++++++++++ 3 files changed, 448 insertions(+) create mode 100755 tools/perf/pmu-events/jevents-test.sh create mode 100755 tools/perf/pmu-events/jevents.py
tools/perf/Makefile.perf +6 -0
@@ -669,6 +669,12 @@ $(JEVENTS_IN): FORCE $(JEVENTS): $(JEVENTS_IN) $(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@ +JEVENTS_PY := pmu-events/jevents.py +JEVENTS_PY_TEST := pmu-events/jevents-test.sh +.PHONY: jevents-py-test +jevents-py-test: $(JEVENTS) + $(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch + $(PMU_EVENTS_IN): $(JEVENTS) FORCE $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events
tools/perf/pmu-events/jevents-test.sh new file +33 -0
@@ -0,0 +1,33 @@ +#!/bin/sh +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +# Validate that the legacy jevents and jevents.py produce identical output. +set -e + +JEVENTS="$1" +JEVENTS_PY="$2" +ARCH_PATH="$3" +JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c) +JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c) + +cleanup() { + rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" + trap - exit term int +} +trap cleanup exit term int + +for path in "$ARCH_PATH"/* +do + arch=$(basename $path) + if [ "$arch" = "test" ] + then + continue + fi + echo "Checking architecture: $arch" + echo "Generating using jevents.c" + "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED" + echo "Generating using jevents.py" + "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED" + echo "Diffing" + diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" +done +cleanup
tools/perf/pmu-events/jevents.py new file +409 -0
409 lines not on this page · raw message
Signature · 1 line
2.37.0.rc0.161.g10f37bed90-goog
Ian Rogers
reviewerPATCH v7 3/4 perf jevents: Switch build to use jevents.py
In reply to Ian RogersFrom: Ian Rogers <[email protected]>
Generate pmu-events.c using jevents.py rather than the binary built from jevents.c. Add a new config variable NO_JEVENTS that is set when there is no architecture json or an appropriate python interpreter isn't present. When NO_JEVENTS is defined the file pmu-events/empty-pmu-events.c is copied and used as the pmu-events.c file.
--- tools/perf/Makefile.config | 19 +++ tools/perf/Makefile.perf | 1 + tools/perf/pmu-events/Build | 13 +- tools/perf/pmu-events/empty-pmu-events.c | 158 +++++++++++++++++++++++ 4 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
tools/perf/Makefile.config +19 -0
@@ -889,6 +889,25 @@ else endif endif +ifneq ($(NO_JEVENTS),1) + ifeq ($(wildcard pmu-events/arch/$(SRCARCH)/mapfile.csv),) + NO_JEVENTS := 1 + endif +endif +ifneq ($(NO_JEVENTS),1) + NO_JEVENTS := 0 + ifndef PYTHON + $(warning No python interpreter disabling jevent generation) + NO_JEVENTS := 1 + else + # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016. + JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1" if(sys.version_info.major >= 3 and sys.version_info.minor >= 6) else "0")' 2> /dev/null) + ifneq ($(JEVENTS_PYTHON_GOOD), 1) + $(warning Python interpreter too old (older than 3.6) disabling jevent generation) + NO_JEVENTS := 1 + endif + endif +endif ifndef NO_LIBBFD ifeq ($(feature-libbfd), 1)
tools/perf/Makefile.perf +1 -0
@@ -655,6 +655,7 @@ JEVENTS := $(OUTPUT)pmu-events/jevents JEVENTS_IN := $(OUTPUT)pmu-events/jevents-in.o PMU_EVENTS_IN := $(OUTPUT)pmu-events/pmu-events-in.o +export NO_JEVENTS export JEVENTS
tools/perf/pmu-events/Build +11 -2
@@ -9,10 +9,19 @@ JSON = $(shell [ -d $(JDIR) ] && \ JDIR_TEST = pmu-events/arch/test JSON_TEST = $(shell [ -d $(JDIR_TEST) ] && \ find $(JDIR_TEST) -name '*.json') +JEVENTS_PY = pmu-events/jevents.py # # Locate/process JSON files in pmu-events/arch/ # directory and create tables in pmu-events.c. # -$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS) - $(Q)$(call echo-cmd,gen)$(JEVENTS) $(SRCARCH) pmu-events/arch $(OUTPUT)pmu-events/pmu-events.c $(V) + +ifeq ($(NO_JEVENTS),1) +$(OUTPUT)pmu-events/pmu-events.c: pmu-events/empty-pmu-events.c + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)cp $< $@ +else +$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS_PY) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $(JEVENTS_PY) $(SRCARCH) pmu-events/arch $@ +endif
tools/perf/pmu-events/empty-pmu-events.c new file +158 -0
158 lines not on this page · raw message
Signature · 1 line
2.37.0.rc0.161.g10f37bed90-goog
Ian Rogers
reviewerPATCH v7 4/4 perf jevents: Remove jevents.c
In reply to Ian RogersFrom: Ian Rogers <[email protected]>
Remove files and build rules.
Remove test for comparing with jevents.py as there is no longer a binary to compare with.
--- tools/perf/Makefile.perf | 21 +- tools/perf/pmu-events/Build | 4 - tools/perf/pmu-events/jevents-test.sh | 33 - tools/perf/pmu-events/jevents.c | 1342 ------------------------- tools/perf/pmu-events/jsmn.c | 352 ------- tools/perf/pmu-events/jsmn.h | 68 -- tools/perf/pmu-events/json.c | 162 --- tools/perf/pmu-events/json.h | 39 - 8 files changed, 2 insertions(+), 2019 deletions(-) delete mode 100755 tools/perf/pmu-events/jevents-test.sh delete mode 100644 tools/perf/pmu-events/jevents.c delete mode 100644 tools/perf/pmu-events/jsmn.c delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h
tools/perf/Makefile.perf +2 -19
@@ -651,32 +651,15 @@ strip: $(PROGRAMS) $(OUTPUT)perf PERF_IN := $(OUTPUT)perf-in.o -JEVENTS := $(OUTPUT)pmu-events/jevents -JEVENTS_IN := $(OUTPUT)pmu-events/jevents-in.o - PMU_EVENTS_IN := $(OUTPUT)pmu-events/pmu-events-in.o export NO_JEVENTS -export JEVENTS - build := -f $(srctree)/tools/build/Makefile.build dir=. obj $(PERF_IN): prepare FORCE $(Q)$(MAKE) $(build)=perf -$(JEVENTS_IN): FORCE - $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=jevents - -$(JEVENTS): $(JEVENTS_IN) - $(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@ - -JEVENTS_PY := pmu-events/jevents.py -JEVENTS_PY_TEST := pmu-events/jevents-test.sh -.PHONY: jevents-py-test -jevents-py-test: $(JEVENTS) - $(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch - -$(PMU_EVENTS_IN): $(JEVENTS) FORCE +$(PMU_EVENTS_IN): FORCE $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events $(OUTPUT)perf: $(PERFLIBS) $(PERF_IN) $(PMU_EVENTS_IN) $(LIBTRACEEVENT_DYNAMIC_LIST)
@@ -1096,7 +1079,7 @@ clean:: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clea $(call QUIET_CLEAN, core-objs) $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive $(OUTPUT)perf-iostat $(LANG_BINDINGS) $(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete $(Q)$(RM) $(OUTPUT).config-detected - $(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32 $(OUTPUT)pmu-events/jevents $(OUTPUT)$(LIBJVMTI).so + $(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32 $(OUTPUT)$(LIBJVMTI).so $(call QUIET_CLEAN, core-gen) $(RM) *.spec *.pyc *.pyo */*.pyc */*.pyo $(OUTPUT)common-cmds.h TAGS tags cscope* $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)FEATURE-DUMP $(OUTPUT)util/*-bison* $(OUTPUT)util/*-flex* \ $(OUTPUT)util/intel-pt-decoder/inat-tables.c \ $(OUTPUT)tests/llvm-src-{base,kbuild,prologue,relocation}.c \
tools/perf/pmu-events/Build +0 -4
@@ -1,7 +1,3 @@ -hostprogs := jevents - -jevents-y += json.o jsmn.o jevents.o -HOSTCFLAGS_jevents.o = -I$(srctree)/tools/include pmu-events-y += pmu-events.o JDIR = pmu-events/arch/$(SRCARCH) JSON = $(shell [ -d $(JDIR) ] && \
tools/perf/pmu-events/jevents-test.sh deleted +0 -33
@@ -1,33 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) -# Validate that the legacy jevents and jevents.py produce identical output. -set -e - -JEVENTS="$1" -JEVENTS_PY="$2" -ARCH_PATH="$3" -JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c) -JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c) - -cleanup() { - rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" - trap - exit term int -} -trap cleanup exit term int - -for path in "$ARCH_PATH"/* -do - arch=$(basename $path) - if [ "$arch" = "test" ] - then - continue - fi - echo "Checking architecture: $arch" - echo "Generating using jevents.c" - "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED" - echo "Generating using jevents.py" - "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED" - echo "Diffing" - diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" -done -cleanup
tools/perf/pmu-events/jevents.c deleted +0 -1342
1,342 lines not on this page · raw message
tools/perf/pmu-events/jsmn.c deleted +0 -352
352 lines not on this page · raw message
tools/perf/pmu-events/jsmn.h deleted +0 -68
68 lines not on this page · raw message
tools/perf/pmu-events/json.c deleted +0 -162
162 lines not on this page · raw message
tools/perf/pmu-events/json.h deleted +0 -39
@@ -1,39 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef JSON_H -#define JSON_H 1 - -#include "jsmn.h" - -jsmntok_t *parse_json (const char *fn,char **map,size_t *size,int *len); -void free_json (char *map,size_t size,jsmntok_t *tokens); -int json_line (char *map,jsmntok_t *t); -const char *json_name (jsmntok_t *t); -int json_streq (char *map,jsmntok_t *t,const char *s); -int json_len (jsmntok_t *t); - -extern int verbose; - -#include <stdbool.h> - -extern int eprintf (int level,int var,const char *fmt, ...); -#define pr_fmt (fmt) fmt - -#define pr_err (fmt, ...) \ - eprintf(0, verbose, pr_fmt(fmt), ##__VA_ARGS__) - -#define pr_info (fmt, ...) \ - eprintf(1, verbose, pr_fmt(fmt), ##__VA_ARGS__) - -#define pr_debug (fmt, ...) \ - eprintf(2, verbose, pr_fmt(fmt), ##__VA_ARGS__) - -#ifndef roundup -#define roundup (x, y) ( \ -{ \ - const typeof(y) __y = y; \ - (((x) + (__y - 1)) / __y) * __y; \ -} \ -) -#endif - -#endif
Signature · 1 line
2.37.0.rc0.161.g10f37bed90-goog
Quoted 14 lines · Ian Rogers
New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update
jsmn.cgiven its forked nature.The changes here switch from
jevents.cto a rewrite in python calledjevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.
Quoted 4 lines
A challenge with this code is in avoiding regressions. For this reason the
jevents.pyproduces identical output tojevents.c, validated with a test script and build target.
Patchset looks good to me.
Thanks,
Kajol Jain
Quoted 52 lines
v7. Adds Tested-by from John Garry <[email protected]>. v6. Adds Tested-by from Zhengjun Xing <[email protected]>
and Thomas Richter <[email protected]>. Fixes issues spotted by Jiri Olsa <[email protected]>,jsmn.cwasn't deleted and the emptypmu-events.cdidn't pass the pmu-events test. It also adds a missing mkdir which is necessary before creatingpmu-events.cand tweaks the quiet display code in the empty case to display a gen.
v5. Adds a 2>/dev/null as suggested by David Laight
<[email protected]>.
v4. Fixes the build for systems with python2 installed by adding a
patch that makes python3 the preferred python (Reported-by: John Garry <[email protected]>). It also fixes a bash-ism in thejevents-test.shand fixes the handling of an empty string for a metric BriefDescription as one was added for sapphirerapids in the metric Execute.
v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the generatedpmu-events.cisn't altered at all by this change. Adds thejevents.cExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't introduced until 3.6.
v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the build for python not being present or too old (Suggested-by: Peter Zijlstra <[email protected]>/John Garry <[email protected]>).Ian Rogers (4):
perf python: Prefer python3
perf jevents: Add python converter script
perf jevents: Switch build to usejevents.py
perf jevents: Removejevents.ctools/perf/Makefile.config | 27 +- tools/perf/Makefile.perf | 16 +- tools/perf/pmu-events/Build | 17 +- tools/perf/pmu-events/empty-pmu-events.c | 158 +++ tools/perf/pmu-events/jevents.c | 1342 ---------------------- tools/perf/pmu-events/jevents.py | 409 +++++++ tools/perf/pmu-events/jsmn.c | 352 ------ tools/perf/pmu-events/jsmn.h | 68 -- tools/perf/pmu-events/json.c | 162 --- tools/perf/pmu-events/json.h | 39 - 10 files changed, 604 insertions(+), 1986 deletions(-) create mode 100644 tools/perf/pmu-events/empty-pmu-events.c delete mode 100644 tools/perf/pmu-events/jevents.c create mode 100755 tools/perf/pmu-events/jevents.py delete mode 100644 tools/perf/pmu-events/jsmn.c delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h