Rewrite jevents program in python
v4 cover Rewrite jevents program in python
New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.
The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.
A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.
v4. Fixes the build for systems with python2 installed by adding a
patch that makes python3 the preferred python (Reported-by: John Garry <[email protected]>). It also fixes a bash-ism in the jevents-test.sh and fixes the handling of an empty string for a metric BriefDescription as one was added for sapphirerapids in the metric Execute.
v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the generated pmu-events.c isn't altered at all by this change. Adds the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't introduced until 3.6.
v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the build for python not being present or too old (Suggested-by: Peter Zijlstra <[email protected]>/John Garry <[email protected]>).
Ian Rogers (4):
perf python: Prefer python3
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c
tools/perf/Makefile.config | 27 +- tools/perf/Makefile.perf | 16 +- tools/perf/pmu-events/Build | 15 +- tools/perf/pmu-events/empty-pmu-events.c | 21 + tools/perf/pmu-events/jevents.c | 1342 ---------------------- tools/perf/pmu-events/jevents.py | 394 +++++++ tools/perf/pmu-events/jsmn.h | 68 -- tools/perf/pmu-events/json.c | 162 --- tools/perf/pmu-events/json.h | 39 - 9 files changed, 450 insertions(+), 1634 deletions(-) create mode 100644 tools/perf/pmu-events/empty-pmu-events.c delete mode 100644 tools/perf/pmu-events/jevents.c create mode 100755 tools/perf/pmu-events/jevents.py delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h
Signature · 1 line
2.34.1
PATCH v4 1/4 perf python: Prefer python3
The PYTHON_AUTO code orders the preference for the PYTHON command to be python3, python and then python2. python3 makes a more logical preference as python2 is no longer supported:
https://www.python.org/doc/sunset-python-2/
Reorder the priority of the PYTHON command to be python2, python and then python3.
--- tools/perf/Makefile.config | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
tools/perf/Makefile.config +4 -4
@@ -241,15 +241,15 @@ endif # Try different combinations to accommodate systems that only have # python[2][3]-config in weird combinations in the following order of # priority from lowest to highest: -# * python3-config -# * python-config # * python2-config as per pep-0394. +# * python-config +# * python3-config # * $(PYTHON)-config (If PYTHON is user supplied but PYTHON_CONFIG isn't) # PYTHON_AUTO := python-config -PYTHON_AUTO := $(if $(call get-executable,python3-config),python3-config,$(PYTHON_AUTO)) -PYTHON_AUTO := $(if $(call get-executable,python-config),python-config,$(PYTHON_AUTO)) PYTHON_AUTO := $(if $(call get-executable,python2-config),python2-config,$(PYTHON_AUTO)) +PYTHON_AUTO := $(if $(call get-executable,python-config),python-config,$(PYTHON_AUTO)) +PYTHON_AUTO := $(if $(call get-executable,python3-config),python3-config,$(PYTHON_AUTO)) # If PYTHON is defined but PYTHON_CONFIG isn't, then take $(PYTHON)-config as if it was the user # supplied value for PYTHON_CONFIG. Because it's "user supplied", error out if it doesn't exist.
Signature · 1 line
2.34.1
PATCH v4 2/4 perf jevents: Add python converter script
In reply to Ian Rogersjevents.c is large, has a dependency on an old forked version of jsmn, and is challenging to work upon. A lot of jevents.c's complexity comes from needing to write json and csv parsing from first principles. In contrast python has this functionality in standard libraries and is already a build pre-requisite for tools like asciidoc (that builds all of the perf man pages).
Introduce jevents.py that produces identical output to jevents.c. Add a test that runs both converter tools and validates there are no output differences. The test can be invoked with a phony build target like:
make -C tools/perf jevents-py-test
The python code deliberately tries to replicate the behavior of jevents.c so that the output matches and transitioning tools shouldn't introduce regressions. In some cases the code isn't as elegant as hoped, but fixing this can be done as follow up.
--- tools/perf/Makefile.perf | 6 + tools/perf/pmu-events/jevents-test.sh | 33 +++ tools/perf/pmu-events/jevents.py | 394 ++++++++++++++++++++++++++ 3 files changed, 433 insertions(+) create mode 100755 tools/perf/pmu-events/jevents-test.sh create mode 100755 tools/perf/pmu-events/jevents.py
tools/perf/Makefile.perf +6 -0
@@ -669,6 +669,12 @@ $(JEVENTS_IN): FORCE $(JEVENTS): $(JEVENTS_IN) $(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@ +JEVENTS_PY := pmu-events/jevents.py +JEVENTS_PY_TEST := pmu-events/jevents-test.sh +.PHONY: jevents-py-test +jevents-py-test: $(JEVENTS) + $(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch + $(PMU_EVENTS_IN): $(JEVENTS) FORCE $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events
tools/perf/pmu-events/jevents-test.sh new file +33 -0
@@ -0,0 +1,33 @@ +#!/bin/sh +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +# Validate that the legacy jevents and jevents.py produce identical output. +set -e + +JEVENTS="$1" +JEVENTS_PY="$2" +ARCH_PATH="$3" +JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c) +JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c) + +cleanup() { + rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" + trap - exit term int +} +trap cleanup exit term int + +for path in "$ARCH_PATH"/* +do + arch=$(basename $path) + if [ "$arch" = "test" ] + then + continue + fi + echo "Checking architecture: $arch" + echo "Generating using jevents.c" + "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED" + echo "Generating using jevents.py" + "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED" + echo "Diffing" + diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" +done +cleanup
tools/perf/pmu-events/jevents.py new file +394 -0
394 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v4 3/4 perf jevents: Switch build to use jevents.py
In reply to Ian RogersGenerate pmu-events.c using jevents.py rather than the binary built from jevents.c. Add a new config variable NO_JEVENTS that is set when there is no architecture json or an appropriate python interpreter isn't present. When NO_JEVENTS is defined the file pmu-events/empty-pmu-events.c is copied and used as the pmu-events.c file.
--- tools/perf/Makefile.config | 19 +++++++++++++++++++ tools/perf/Makefile.perf | 1 + tools/perf/pmu-events/Build | 11 +++++++++-- tools/perf/pmu-events/empty-pmu-events.c | 21 +++++++++++++++++++++ 4 files changed, 50 insertions(+), 2 deletions(-) create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
tools/perf/Makefile.config +19 -0
@@ -889,6 +889,25 @@ else endif endif +ifneq ($(NO_JEVENTS),1) + ifeq ($(wildcard pmu-events/arch/$(SRCARCH)/mapfile.csv),) + NO_JEVENTS := 1 + endif +endif +ifneq ($(NO_JEVENTS),1) + NO_JEVENTS := 0 + ifndef PYTHON + $(warning No python interpreter disabling jevent generation) + NO_JEVENTS := 1 + else + # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016. + JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1" if(sys.version_info.major >= 3 and sys.version_info.minor >= 6) else "0")') + ifneq ($(JEVENTS_PYTHON_GOOD), 1) + $(warning Python interpreter too old (older than 3.6) disabling jevent generation) + NO_JEVENTS := 1 + endif + endif +endif ifndef NO_LIBBFD ifeq ($(feature-libbfd), 1)
tools/perf/Makefile.perf +1 -0
@@ -655,6 +655,7 @@ JEVENTS := $(OUTPUT)pmu-events/jevents JEVENTS_IN := $(OUTPUT)pmu-events/jevents-in.o PMU_EVENTS_IN := $(OUTPUT)pmu-events/pmu-events-in.o +export NO_JEVENTS export JEVENTS
tools/perf/pmu-events/Build +9 -2
@@ -9,10 +9,17 @@ JSON = $(shell [ -d $(JDIR) ] && \ JDIR_TEST = pmu-events/arch/test JSON_TEST = $(shell [ -d $(JDIR_TEST) ] && \ find $(JDIR_TEST) -name '*.json') +JEVENTS_PY = pmu-events/jevents.py # # Locate/process JSON files in pmu-events/arch/ # directory and create tables in pmu-events.c. # -$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS) - $(Q)$(call echo-cmd,gen)$(JEVENTS) $(SRCARCH) pmu-events/arch $(OUTPUT)pmu-events/pmu-events.c $(V) + +ifeq ($(NO_JEVENTS),1) +$(OUTPUT)pmu-events/pmu-events.c: pmu-events/empty-pmu-events.c + $(Q)cp $< $@ +else +$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS_PY) + $(Q)$(call echo-cmd,gen)$(PYTHON) $(JEVENTS_PY) $(SRCARCH) pmu-events/arch $@ +endif
tools/perf/pmu-events/empty-pmu-events.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* An empty pmu-events.c file used when there is no architecture json files in +* arch or when the jevents.py script cannot be run. +*/ +#include "pmu-events/pmu-events.h" + +const struct pmu_events_map pmu_events_map[] = { + { + .cpuid =0 , + .version =0 , + .type =0 , + .table =0 , + }, +}; + +const struct pmu_sys_events pmu_sys_event_tables[] = { + { + .table =0 + }, +};
Signature · 1 line
2.34.1
RE: [PATCH v4 3/4] perf jevents: Switch build to use jevents.py
From: Ian Rogers
Quoted 8 lines
Sent: 15 June 2022 04:05
Generate
pmu-events.cusingjevents.pyrather than the binary built fromjevents.c. Add a new config variableNO_JEVENTSthat is set when there is no architecture json or an appropriate python interpreter isn't present. WhenNO_JEVENTSis defined the filepmu-events/empty-pmu-events.cis copied and used as thepmu-events.cfile.
...
Quoted 3 lines
(quoted diff) +2
-0+ # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016. + JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1" if(sys.version_info.major >= 3and sys.version_info.minor >= 6) else "0")')
You probably want a "2>/dev/null" in there.
I also think I remember seeing a which python in one of these patches. 'which' is a bourne shell script that is trying to emulate a csh builtin. It can't actually work, and ISTR one of the distros might be trying to remove it.
In a traditional bourne shell you should use type python, the posix equivalent (which any shell written in the last 30 years out to get right) is command -V python.
Both type and command have to be shell builtins.
Although I suspect some shells get command -V wrong.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
PATCH v4 4/4 perf jevents: Remove jevents.c
In reply to Ian RogersRemove files and build rules.
Remove test for comparing with jevents.py as there is no longer a binary to compare with.
--- tools/perf/Makefile.perf | 21 +- tools/perf/pmu-events/Build | 4 - tools/perf/pmu-events/jevents-test.sh | 33 - tools/perf/pmu-events/jevents.c | 1342 ------------------------- tools/perf/pmu-events/jsmn.h | 68 -- tools/perf/pmu-events/json.c | 162 --- tools/perf/pmu-events/json.h | 39 - 7 files changed, 2 insertions(+), 1667 deletions(-) delete mode 100755 tools/perf/pmu-events/jevents-test.sh delete mode 100644 tools/perf/pmu-events/jevents.c delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h
tools/perf/Makefile.perf +2 -19
@@ -651,32 +651,15 @@ strip: $(PROGRAMS) $(OUTPUT)perf PERF_IN := $(OUTPUT)perf-in.o -JEVENTS := $(OUTPUT)pmu-events/jevents -JEVENTS_IN := $(OUTPUT)pmu-events/jevents-in.o - PMU_EVENTS_IN := $(OUTPUT)pmu-events/pmu-events-in.o export NO_JEVENTS -export JEVENTS - build := -f $(srctree)/tools/build/Makefile.build dir=. obj $(PERF_IN): prepare FORCE $(Q)$(MAKE) $(build)=perf -$(JEVENTS_IN): FORCE - $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=jevents - -$(JEVENTS): $(JEVENTS_IN) - $(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@ - -JEVENTS_PY := pmu-events/jevents.py -JEVENTS_PY_TEST := pmu-events/jevents-test.sh -.PHONY: jevents-py-test -jevents-py-test: $(JEVENTS) - $(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch - -$(PMU_EVENTS_IN): $(JEVENTS) FORCE +$(PMU_EVENTS_IN): FORCE $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events $(OUTPUT)perf: $(PERFLIBS) $(PERF_IN) $(PMU_EVENTS_IN) $(LIBTRACEEVENT_DYNAMIC_LIST)
@@ -1096,7 +1079,7 @@ clean:: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clea $(call QUIET_CLEAN, core-objs) $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive $(OUTPUT)perf-iostat $(LANG_BINDINGS) $(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete $(Q)$(RM) $(OUTPUT).config-detected - $(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32 $(OUTPUT)pmu-events/jevents $(OUTPUT)$(LIBJVMTI).so + $(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32 $(OUTPUT)$(LIBJVMTI).so $(call QUIET_CLEAN, core-gen) $(RM) *.spec *.pyc *.pyo */*.pyc */*.pyo $(OUTPUT)common-cmds.h TAGS tags cscope* $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)FEATURE-DUMP $(OUTPUT)util/*-bison* $(OUTPUT)util/*-flex* \ $(OUTPUT)util/intel-pt-decoder/inat-tables.c \ $(OUTPUT)tests/llvm-src-{base,kbuild,prologue,relocation}.c \
tools/perf/pmu-events/Build +0 -4
@@ -1,7 +1,3 @@ -hostprogs := jevents - -jevents-y += json.o jsmn.o jevents.o -HOSTCFLAGS_jevents.o = -I$(srctree)/tools/include pmu-events-y += pmu-events.o JDIR = pmu-events/arch/$(SRCARCH) JSON = $(shell [ -d $(JDIR) ] && \
tools/perf/pmu-events/jevents-test.sh deleted +0 -33
@@ -1,33 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) -# Validate that the legacy jevents and jevents.py produce identical output. -set -e - -JEVENTS="$1" -JEVENTS_PY="$2" -ARCH_PATH="$3" -JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c) -JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c) - -cleanup() { - rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" - trap - exit term int -} -trap cleanup exit term int - -for path in "$ARCH_PATH"/* -do - arch=$(basename $path) - if [ "$arch" = "test" ] - then - continue - fi - echo "Checking architecture: $arch" - echo "Generating using jevents.c" - "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED" - echo "Generating using jevents.py" - "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED" - echo "Diffing" - diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED" -done -cleanup
tools/perf/pmu-events/jevents.c deleted +0 -1342
1,342 lines not on this page · raw message
tools/perf/pmu-events/jsmn.h deleted +0 -68
68 lines not on this page · raw message
tools/perf/pmu-events/json.c deleted +0 -162
162 lines not on this page · raw message
tools/perf/pmu-events/json.h deleted +0 -39
@@ -1,39 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef JSON_H -#define JSON_H 1 - -#include "jsmn.h" - -jsmntok_t *parse_json (const char *fn,char **map,size_t *size,int *len); -void free_json (char *map,size_t size,jsmntok_t *tokens); -int json_line (char *map,jsmntok_t *t); -const char *json_name (jsmntok_t *t); -int json_streq (char *map,jsmntok_t *t,const char *s); -int json_len (jsmntok_t *t); - -extern int verbose; - -#include <stdbool.h> - -extern int eprintf (int level,int var,const char *fmt, ...); -#define pr_fmt (fmt) fmt - -#define pr_err (fmt, ...) \ - eprintf(0, verbose, pr_fmt(fmt), ##__VA_ARGS__) - -#define pr_info (fmt, ...) \ - eprintf(1, verbose, pr_fmt(fmt), ##__VA_ARGS__) - -#define pr_debug (fmt, ...) \ - eprintf(2, verbose, pr_fmt(fmt), ##__VA_ARGS__) - -#ifndef roundup -#define roundup (x, y) ( \ -{ \ - const typeof(y) __y = y; \ - (((x) + (__y - 1)) / __y) * __y; \ -} \ -) -#endif - -#endif
Signature · 1 line
2.34.1