Skip to contentSkip to conversation

Rewrite jevents program in python

PATCH Superseded Ian Rogers 2 participants 10 messages May 27, 2022 – Jun 15, 2022 perf events

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

v3 cover Rewrite jevents program in python

New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.

The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.

A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.

v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the generated pmu-events.c isn't altered at all by this change. Adds the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't introduced until 3.6.

v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the build for python not being present or too old (Suggested-by: Peter Zijlstra <[email protected]>/John Garry <[email protected]>).

Ian Rogers (3):
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c

 tools/perf/Makefile.config               |   19 +
 tools/perf/Makefile.perf                 |   16 +-
 tools/perf/pmu-events/Build              |   15 +-
 tools/perf/pmu-events/empty-pmu-events.c |   21 +
 tools/perf/pmu-events/jevents.c          | 1342 ----------------------
 tools/perf/pmu-events/jevents.py         |  392 +++++++
 tools/perf/pmu-events/jsmn.h             |   68 --
 tools/perf/pmu-events/json.c             |  162 ---
 tools/perf/pmu-events/json.h             |   39 -
 9 files changed, 444 insertions(+), 1630 deletions(-)
 create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
 delete mode 100644 tools/perf/pmu-events/jevents.c
 create mode 100755 tools/perf/pmu-events/jevents.py
 delete mode 100644 tools/perf/pmu-events/jsmn.h
 delete mode 100644 tools/perf/pmu-events/json.c
 delete mode 100644 tools/perf/pmu-events/json.h
Signature · 1 line
2.36.1.255.ge46751e96f-goog
author reviewer
#1 lore raw

PATCH v3 1/3 perf jevents: Add python converter script

jevents.c is large, has a dependency on an old forked version of jsmn, and is challenging to work upon. A lot of jevents.c's complexity comes from needing to write json and csv parsing from first principles. In contrast python has this functionality in standard libraries and is already a build pre-requisite for tools like asciidoc (that builds all of the perf man pages).

Introduce jevents.py that produces identical output to jevents.c. Add a test that runs both converter tools and validates there are no output differences. The test can be invoked with a phony build target like:

make -C tools/perf jevents-py-test

The python code deliberately tries to replicate the behavior of jevents.c so that the output matches and transitioning tools shouldn't introduce regressions. In some cases the code isn't as elegant as hoped, but fixing this can be done as follow up.

---
 tools/perf/Makefile.perf              |   6 +
 tools/perf/pmu-events/jevents-test.sh |  33 +++
 tools/perf/pmu-events/jevents.py      | 392 ++++++++++++++++++++++++++
 3 files changed, 431 insertions(+)
 create mode 100755 tools/perf/pmu-events/jevents-test.sh
 create mode 100755 tools/perf/pmu-events/jevents.py
tools/perf/Makefile.perf +6 -0
index 8f738e11356d..1e29c8936f71 100644
@@ -669,6 +669,12 @@ $(JEVENTS_IN): FORCE
 $(JEVENTS): $(JEVENTS_IN)
 	$(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@
 
+JEVENTS_PY	:=  pmu-events/jevents.py
+JEVENTS_PY_TEST	:=  pmu-events/jevents-test.sh
+.PHONY: jevents-py-test
+jevents-py-test: $(JEVENTS)
+	$(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch
+
 $(PMU_EVENTS_IN): $(JEVENTS) FORCE
 	$(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events
 
tools/perf/pmu-events/jevents-test.sh new file +33 -0
new file mode 100755
index 000000000000..195ec83630ad
@@ -0,0 +1,33 @@
+#!/bin/sh
+# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+# Validate that the legacy jevents and jevents.py produce identical output.
+set -e
+
+JEVENTS="$1"
+JEVENTS_PY="$2"
+ARCH_PATH="$3"
+JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c)
+JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c)
+
+cleanup() {
+  rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED"
+  trap - exit term int
+}
+trap cleanup exit term int
+
+for path in "$ARCH_PATH"/*
+do
+  arch=$(basename $path)
+  if [[ "$arch" == "test" ]]
+  then
+    continue
+  fi
+  echo "Checking architecture: $arch"
+  echo "Generating using jevents.c"
+  "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED"
+  echo "Generating using jevents.py"
+  "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED"
+  echo "Diffing"
+  diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED"
+done
+cleanup
tools/perf/pmu-events/jevents.py new file +392 -0

392 lines not on this page · raw message

Signature · 1 line
2.36.1.255.ge46751e96f-goog
author reviewer
#2 lore raw

PATCH v3 2/3 perf jevents: Switch build to use jevents.py

In reply to Ian Rogers

Generate pmu-events.c using jevents.py rather than the binary built from jevents.c. Add a new config variable NO_JEVENTS that is set when there is no architecture json or an appropriate python interpreter isn't present. When NO_JEVENTS is defined the file pmu-events/empty-pmu-events.c is copied and used as the pmu-events.c file.

---
 tools/perf/Makefile.config               | 19 +++++++++++++++++++
 tools/perf/Makefile.perf                 |  1 +
 tools/perf/pmu-events/Build              | 11 +++++++++--
 tools/perf/pmu-events/empty-pmu-events.c | 21 +++++++++++++++++++++
 4 files changed, 50 insertions(+), 2 deletions(-)
 create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
tools/perf/Makefile.config +19 -0
index 73e0762092fe..9538d737d655 100644
@@ -889,6 +889,25 @@ else
   endif
 endif
 
+ifneq ($(NO_JEVENTS),1)
+  ifeq ($(wildcard pmu-events/arch/$(SRCARCH)/mapfile.csv),)
+    NO_JEVENTS := 1
+  endif
+endif
+ifneq ($(NO_JEVENTS),1)
+  NO_JEVENTS := 0
+  ifndef PYTHON
+    $(warning No python interpreter disabling jevent generation)
+    NO_JEVENTS := 1
+  else
+    # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
+    JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1" if(sys.version_info.major >= 3 and sys.version_info.minor >= 6) else "0")')
+    ifneq ($(JEVENTS_PYTHON_GOOD), 1)
+      $(warning Python interpreter too old (older than 3.6) disabling jevent generation)
+      NO_JEVENTS := 1
+    endif
+  endif
+endif
 
 ifndef NO_LIBBFD
   ifeq ($(feature-libbfd), 1)
tools/perf/Makefile.perf +1 -0
index 1e29c8936f71..dc6b177ac1de 100644
@@ -655,6 +655,7 @@ JEVENTS       := $(OUTPUT)pmu-events/jevents
 JEVENTS_IN    := $(OUTPUT)pmu-events/jevents-in.o
 
 PMU_EVENTS_IN := $(OUTPUT)pmu-events/pmu-events-in.o
+export NO_JEVENTS
 
 export JEVENTS
 
tools/perf/pmu-events/Build +9 -2
index a055dee6a46a..5e459e1c7a85 100644
@@ -9,10 +9,17 @@ JSON		=  $(shell [ -d $(JDIR) ] &&				\
 JDIR_TEST	=  pmu-events/arch/test
 JSON_TEST	=  $(shell [ -d $(JDIR_TEST) ] &&			\
 			find $(JDIR_TEST) -name '*.json')
+JEVENTS_PY	=  pmu-events/jevents.py
 
 #
 # Locate/process JSON files in pmu-events/arch/
 # directory and create tables in pmu-events.c.
 #
-$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS)
-	$(Q)$(call echo-cmd,gen)$(JEVENTS) $(SRCARCH) pmu-events/arch $(OUTPUT)pmu-events/pmu-events.c $(V)
+
+ifeq ($(NO_JEVENTS),1)
+$(OUTPUT)pmu-events/pmu-events.c: pmu-events/empty-pmu-events.c
+	$(Q)cp $< $@
+else
+$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS_PY)
+	$(Q)$(call echo-cmd,gen)$(PYTHON) $(JEVENTS_PY) $(SRCARCH) pmu-events/arch $@
+endif
tools/perf/pmu-events/empty-pmu-events.c new file +21 -0
new file mode 100644
index 000000000000..08fee615e015
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * An empty pmu-events.c file used when there is no architecture json files in
+ * arch or when the jevents.py script cannot be run.
+ */
+#include "pmu-events/pmu-events.h"
+
+const struct pmu_events_map pmu_events_map[] = {
+	{
+		.cpuid = 0,
+		.version = 0,
+		.type = 0,
+		.table = 0,
+	},
+};
+
+const struct pmu_sys_events pmu_sys_event_tables[] = {
+	{
+		.table = 0
+	},
+};
Signature · 1 line
2.36.1.255.ge46751e96f-goog
author reviewer
#3 lore raw

PATCH v3 3/3 perf jevents: Remove jevents.c

In reply to Ian Rogers

Remove files and build rules.
Remove test for comparing with jevents.py as there is no longer a binary to compare with.

---
 tools/perf/Makefile.perf              |   21 +-
 tools/perf/pmu-events/Build           |    4 -
 tools/perf/pmu-events/jevents-test.sh |   33 -
 tools/perf/pmu-events/jevents.c       | 1342 -------------------------
 tools/perf/pmu-events/jsmn.h          |   68 --
 tools/perf/pmu-events/json.c          |  162 ---
 tools/perf/pmu-events/json.h          |   39 -
 7 files changed, 2 insertions(+), 1667 deletions(-)
 delete mode 100755 tools/perf/pmu-events/jevents-test.sh
 delete mode 100644 tools/perf/pmu-events/jevents.c
 delete mode 100644 tools/perf/pmu-events/jsmn.h
 delete mode 100644 tools/perf/pmu-events/json.c
 delete mode 100644 tools/perf/pmu-events/json.h
tools/perf/Makefile.perf +2 -19
index dc6b177ac1de..8f0b1fb39984 100644
@@ -651,32 +651,15 @@ strip: $(PROGRAMS) $(OUTPUT)perf
 
 PERF_IN := $(OUTPUT)perf-in.o
 
-JEVENTS       := $(OUTPUT)pmu-events/jevents
-JEVENTS_IN    := $(OUTPUT)pmu-events/jevents-in.o
-
 PMU_EVENTS_IN := $(OUTPUT)pmu-events/pmu-events-in.o
 export NO_JEVENTS
 
-export JEVENTS
-
 build := -f $(srctree)/tools/build/Makefile.build dir=. obj
 
 $(PERF_IN): prepare FORCE
 	$(Q)$(MAKE) $(build)=perf
 
-$(JEVENTS_IN): FORCE
-	$(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=jevents
-
-$(JEVENTS): $(JEVENTS_IN)
-	$(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@
-
-JEVENTS_PY	:=  pmu-events/jevents.py
-JEVENTS_PY_TEST	:=  pmu-events/jevents-test.sh
-.PHONY: jevents-py-test
-jevents-py-test: $(JEVENTS)
-	$(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch
-
-$(PMU_EVENTS_IN): $(JEVENTS) FORCE
+$(PMU_EVENTS_IN): FORCE
 	$(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events
 
 $(OUTPUT)perf: $(PERFLIBS) $(PERF_IN) $(PMU_EVENTS_IN) $(LIBTRACEEVENT_DYNAMIC_LIST)
@@ -1096,7 +1079,7 @@ clean:: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clea
 	$(call QUIET_CLEAN, core-objs)  $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive $(OUTPUT)perf-iostat $(LANG_BINDINGS)
 	$(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete
 	$(Q)$(RM) $(OUTPUT).config-detected
-	$(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32 $(OUTPUT)pmu-events/jevents $(OUTPUT)$(LIBJVMTI).so
+	$(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32 $(OUTPUT)$(LIBJVMTI).so
 	$(call QUIET_CLEAN, core-gen)   $(RM)  *.spec *.pyc *.pyo */*.pyc */*.pyo $(OUTPUT)common-cmds.h TAGS tags cscope* $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)FEATURE-DUMP $(OUTPUT)util/*-bison* $(OUTPUT)util/*-flex* \
 		$(OUTPUT)util/intel-pt-decoder/inat-tables.c \
 		$(OUTPUT)tests/llvm-src-{base,kbuild,prologue,relocation}.c \
tools/perf/pmu-events/Build +0 -4
index 5e459e1c7a85..82e07be354fb 100644
@@ -1,7 +1,3 @@
-hostprogs := jevents
-
-jevents-y	+= json.o jsmn.o jevents.o
-HOSTCFLAGS_jevents.o	= -I$(srctree)/tools/include
 pmu-events-y	+= pmu-events.o
 JDIR		=  pmu-events/arch/$(SRCARCH)
 JSON		=  $(shell [ -d $(JDIR) ] &&				\
tools/perf/pmu-events/jevents-test.sh deleted +0 -33
deleted file mode 100755
index 195ec83630ad..000000000000
@@ -1,33 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
-# Validate that the legacy jevents and jevents.py produce identical output.
-set -e
-
-JEVENTS="$1"
-JEVENTS_PY="$2"
-ARCH_PATH="$3"
-JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c)
-JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c)
-
-cleanup() {
-  rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED"
-  trap - exit term int
-}
-trap cleanup exit term int
-
-for path in "$ARCH_PATH"/*
-do
-  arch=$(basename $path)
-  if [[ "$arch" == "test" ]]
-  then
-    continue
-  fi
-  echo "Checking architecture: $arch"
-  echo "Generating using jevents.c"
-  "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED"
-  echo "Generating using jevents.py"
-  "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED"
-  echo "Diffing"
-  diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED"
-done
-cleanup
tools/perf/pmu-events/jevents.c deleted +0 -1342

1,342 lines not on this page · raw message

tools/perf/pmu-events/jsmn.h deleted +0 -68

68 lines not on this page · raw message

tools/perf/pmu-events/json.c deleted +0 -162

162 lines not on this page · raw message

tools/perf/pmu-events/json.h deleted +0 -39
deleted file mode 100644
index fbcd5a0590ad..000000000000
@@ -1,39 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef JSON_H
-#define JSON_H 1
-
-#include "jsmn.h"
-
-jsmntok_t *parse_json(const char *fn, char **map, size_t *size, int *len);
-void free_json(char *map, size_t size, jsmntok_t *tokens);
-int json_line(char *map, jsmntok_t *t);
-const char *json_name(jsmntok_t *t);
-int json_streq(char *map, jsmntok_t *t, const char *s);
-int json_len(jsmntok_t *t);
-
-extern int verbose;
-
-#include <stdbool.h>
-
-extern int eprintf(int level, int var, const char *fmt, ...);
-#define pr_fmt(fmt)	fmt
-
-#define pr_err(fmt, ...) \
-	eprintf(0, verbose, pr_fmt(fmt), ##__VA_ARGS__)
-
-#define pr_info(fmt, ...) \
-	eprintf(1, verbose, pr_fmt(fmt), ##__VA_ARGS__)
-
-#define pr_debug(fmt, ...) \
-	eprintf(2, verbose, pr_fmt(fmt), ##__VA_ARGS__)
-
-#ifndef roundup
-#define roundup(x, y) (                                \
-{                                                      \
-        const typeof(y) __y = y;                       \
-        (((x) + (__y - 1)) / __y) * __y;               \
-}                                                      \
-)
-#endif
-
-#endif
Signature · 1 line
2.36.1.255.ge46751e96f-goog
#4 lore raw
In reply to Ian Rogers
Quoted 14 lines · Ian Rogers

New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.

The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.

Quoted 4 lines

A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.

Hi Ian,

I still see this:

...
Makefile.config:906: Python interpreter too old (older than 3.6) disabling jevent generation Makefile.config:939: Old version of libbfd/binutils things like PE executable profiling will not be available Makefile.config:1127: No openjdk development package found, please install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel

Auto-detecting system features:

...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ on  ]
...        disassembler-four-args: [ on  ]

make[3]: Nothing to be done for 'install_headers'. john@localhost:~/acme/tools/perf> python --version Python 3.6.12

I need to figure out what is going wrong...

Thanks,
John

Quoted 35 lines

v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the generated pmu-events.c isn't altered at all by this change. Adds the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't introduced until 3.6.

v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the build for python not being present or too old (Suggested-by: Peter Zijlstra <[email protected]>/John Garry <[email protected]>).

Ian Rogers (3):
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c

  tools/perf/Makefile.config               |   19 +
  tools/perf/Makefile.perf                 |   16 +-
  tools/perf/pmu-events/Build              |   15 +-
  tools/perf/pmu-events/empty-pmu-events.c |   21 +
  tools/perf/pmu-events/jevents.c          | 1342 ----------------------
  tools/perf/pmu-events/jevents.py         |  392 +++++++
  tools/perf/pmu-events/jsmn.h             |   68 --
  tools/perf/pmu-events/json.c             |  162 ---
  tools/perf/pmu-events/json.h             |   39 -
  9 files changed, 444 insertions(+), 1630 deletions(-)

create mode 100644 tools/perf/pmu-events/empty-pmu-events.c delete mode 100644 tools/perf/pmu-events/jevents.c create mode 100755 tools/perf/pmu-events/jevents.py delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h

author reviewer
#5 lore raw
Quoted 57 lines · John Garry
Quoted 14 lines · Ian Rogers

New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.

The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.

Quoted 4 lines

A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.

Hi Ian,

I still see this:

...
Makefile.config:906: Python interpreter too old (older than 3.6) disabling jevent generation
Makefile.config:939: Old version of libbfd/binutils things like PE executable profiling will not be available Makefile.config:1127: No openjdk development package found, please install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel

Auto-detecting system features:

...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ on  ]
...        disassembler-four-args: [ on  ]
Quoted 5 lines

make[3]: Nothing to be done for 'install_headers'. john@localhost:~/acme/tools/perf> python --version Python 3.6.12

I need to figure out what is going wrong...

Thanks for testing John! Could you try adding something like this:

```
diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
index 9538d737d655..c23f39e21a70 100644
--- a/tools/perf/Makefile.config
+++ b/tools/perf/Makefile.config
@@ -901,6 +901,8 @@ ifneq ($(NO_JEVENTS),1)
    NO_JEVENTS := 1
  else
    # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
+    PYTHON_VERSION=$(shell $(PYTHON) -c 'import sys;print(sys.version_info)')
+    $(error $(PYTHON_VERSION))
    JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1"
if(sys.version_info.major >=
3 and sys.version_info.minor >= 6) else "0")')
    ifneq ($(JEVENTS_PYTHON_GOOD), 1)
      $(warning Python interpreter too old (older than 3.6) disabling
jevent generation)
```

You can also specify which python you want on the make command line, for example adding PYTHON=which python at the end of your make command.

Thanks,
Ian

Quoted 39 lines

Thanks,
John

Quoted 35 lines

v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the generated pmu-events.c isn't altered at all by this change. Adds the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't introduced until 3.6.

v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the build for python not being present or too old (Suggested-by: Peter Zijlstra <[email protected]>/John Garry <[email protected]>).

Ian Rogers (3):
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c

  tools/perf/Makefile.config               |   19 +
  tools/perf/Makefile.perf                 |   16 +-
  tools/perf/pmu-events/Build              |   15 +-
  tools/perf/pmu-events/empty-pmu-events.c |   21 +
  tools/perf/pmu-events/jevents.c          | 1342 ----------------------
  tools/perf/pmu-events/jevents.py         |  392 +++++++
  tools/perf/pmu-events/jsmn.h             |   68 --
  tools/perf/pmu-events/json.c             |  162 ---
  tools/perf/pmu-events/json.h             |   39 -
  9 files changed, 444 insertions(+), 1630 deletions(-)

create mode 100644 tools/perf/pmu-events/empty-pmu-events.c delete mode 100644 tools/perf/pmu-events/jevents.c create mode 100755 tools/perf/pmu-events/jevents.py delete mode 100644 tools/perf/pmu-events/jsmn.h delete mode 100644 tools/perf/pmu-events/json.c delete mode 100644 tools/perf/pmu-events/json.h

#6 lore raw
Quoted 76 lines · Ian Rogers
Quoted 55 lines · John Garry
Quoted 14 lines · Ian Rogers

New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.

The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.

Quoted 4 lines

A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.

Hi Ian,

I still see this:

...
Makefile.config:906: Python interpreter too old (older than 3.6) disabling jevent generation
Makefile.config:939: Old version of libbfd/binutils things like PE executable profiling will not be available Makefile.config:1127: No openjdk development package found, please install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel

Auto-detecting system features:

...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ on  ]
...        disassembler-four-args: [ on  ]
Quoted 5 lines

make[3]: Nothing to be done for 'install_headers'. john@localhost:~/acme/tools/perf> python --version Python 3.6.12

I need to figure out what is going wrong...

Thanks for testing John! Could you try adding something like this:
```

tools/perf/Makefile.config +2 -0
index 9538d737d655..c23f39e21a70 100644
@@ -901,6 +901,8 @@ ifneq ($(NO_JEVENTS),1)
     NO_JEVENTS := 1
   else
     # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
+    PYTHON_VERSION=$(shell $(PYTHON) -c 'import sys;print(sys.version_info)')
+    $(error $(PYTHON_VERSION))
     JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1"

if(sys.version_info.major >=
3 and sys.version_info.minor >= 6) else "0")')

Quoted 4 lines
     ifneq ($(JEVENTS_PYTHON_GOOD), 1)
       $(warning Python interpreter too old (older than 3.6) disabling

jevent generation)
```

That gives:

Makefile.config:459: No libdw DWARF unwind found, Please install elfutils-devel/libdw-dev >= 0.158 and/or set LIBDW_DIR Makefile.config:464: No libdw.h found or old libdw.h found or elfutils is older than 0.138, disables dwarf support. Please install new elfutils-devel/libdw-dev Makefile.config:617: DWARF support is off, BPF prologue is disabled
Makefile.config:667: No debug_frame support found in libunwind-aarch64
Makefile.config:738: No debug_frame support found in libunwind
Makefile.config:906: *** sys.version_info(major=2, minor=7, micro=13, releaselevel='final', serial=0).Stop.
Makefile.perf:239: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2

john@debian:~/linux/tools/perf$ python --version
Python 3.6.9
john@debian:~/linux/tools/perf$
Quoted 3 lines

You can also specify which python you want on the make command line, for example adding PYTHON=which python at the end of your make command.

diff -u tools/perf/trace/beauty/include/linux/socket.h include/linux/socket.h Makefile.config:259: *** which python-config not found. Stop. Makefile.perf:239: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2
john@debian:~/linux/tools/perf$

JFYI, this is my alternatives:

sudo update-alternatives --config python
There are 3 choices for the alternative python (providing /usr/bin/python).

Selection Path Priority Status
------------------------------------------------------------

  0            /usr/local/bin/python3.6   50        auto mode
  1            /usr/bin/python2.7         1         manual mode
  2            /usr/bin/python3.5         2         manual mode
* 3            /usr/local/bin/python3.6   50        manual mode

Press <enter> to keep the current choice[*], or type selection number:

Can someone else try this? Arnaldo?

Sorry for being useless with this....

author reviewer
#7 lore raw
Quoted 122 lines · John Garry
Quoted 76 lines · Ian Rogers
Quoted 55 lines · John Garry
Quoted 14 lines · Ian Rogers

New architectures bring new complexity, such as Intel's hybrid models. jevents provides an alternative to specifying events in the kernel and exposing them through sysfs, however, it is difficult to work with. For example, an error in the json input would yield an error message but no json file or location. It is also a challenge to update jsmn.c given its forked nature.

The changes here switch from jevents.c to a rewrite in python called jevents.py. This means there is a build time dependency on python, but such a dependency already exists for asciidoc (used to generate perf's man pages). If the build detects that python isn't present or is older than version 3.6 (released Dec. 2016) then an empty file is substituted for the generated one.

Quoted 4 lines

A challenge with this code is in avoiding regressions. For this reason the jevents.py produces identical output to jevents.c, validated with a test script and build target.

Hi Ian,

I still see this:

...
Makefile.config:906: Python interpreter too old (older than 3.6) disabling jevent generation
Makefile.config:939: Old version of libbfd/binutils things like PE executable profiling will not be available Makefile.config:1127: No openjdk development package found, please install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel

Auto-detecting system features:

...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ on  ]
...        disassembler-four-args: [ on  ]
Quoted 5 lines

make[3]: Nothing to be done for 'install_headers'. john@localhost:~/acme/tools/perf> python --version Python 3.6.12

I need to figure out what is going wrong...

Thanks for testing John! Could you try adding something like this:
```

tools/perf/Makefile.config +2 -0
index 9538d737d655..c23f39e21a70 100644
@@ -901,6 +901,8 @@ ifneq ($(NO_JEVENTS),1)
     NO_JEVENTS := 1
   else
     # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
+    PYTHON_VERSION=$(shell $(PYTHON) -c 'import sys;print(sys.version_info)')
+    $(error $(PYTHON_VERSION))
     JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1"

if(sys.version_info.major >=
3 and sys.version_info.minor >= 6) else "0")')

Quoted 4 lines
     ifneq ($(JEVENTS_PYTHON_GOOD), 1)
       $(warning Python interpreter too old (older than 3.6) disabling

jevent generation)
```

That gives:

Makefile.config:459: No libdw DWARF unwind found, Please install elfutils-devel/libdw-dev >= 0.158 and/or set LIBDW_DIR Makefile.config:464: No libdw.h found or old libdw.h found or elfutils is older than 0.138, disables dwarf support. Please install new elfutils-devel/libdw-dev
Makefile.config:617: DWARF support is off, BPF prologue is disabled Makefile.config:667: No debug_frame support found in libunwind-aarch64 Makefile.config:738: No debug_frame support found in libunwind Makefile.config:906: *** sys.version_info(major=2, minor=7, micro=13, releaselevel='final', serial=0).Stop.
Makefile.perf:239: recipe for target 'sub-make' failed make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2

john@debian:~/linux/tools/perf$ python --version
Python 3.6.9
john@debian:~/linux/tools/perf$
Quoted 3 lines

You can also specify which python you want on the make command line, for example adding PYTHON=which python at the end of your make command.

diff -u tools/perf/trace/beauty/include/linux/socket.h include/linux/socket.h
Makefile.config:259: *** which python-config not found. Stop. Makefile.perf:239: recipe for target 'sub-make' failed make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2 john@debian:~/linux/tools/perf$

JFYI, this is my alternatives:

sudo update-alternatives --config python
There are 3 choices for the alternative python (providing /usr/bin/python).

Quoted 8 lines

Selection Path Priority Status
------------------------------------------------------------

   0            /usr/local/bin/python3.6   50        auto mode
   1            /usr/bin/python2.7         1         manual mode
   2            /usr/bin/python3.5         2         manual mode

* 3 /usr/local/bin/python3.6 50 manual mode

Press <enter> to keep the current choice[*], or type selection number:

Thanks Gary,

Perhaps it relates to dev packages. Perhaps, apt-get install python-dev-is-python3 which should resolve the symlinks. I wonder that you have python dev for python2 but python3 installed without the dev. We should be able to make a Makefile test for this.

Thanks,
Ian

Quoted 5 lines

Can someone else try this? Arnaldo?

Sorry for being useless with this....

#8 lore raw
Quoted 22 lines · Ian Rogers
Quoted 22 lines

diff -u tools/perf/trace/beauty/include/linux/socket.h include/linux/socket.h
Makefile.config:259: *** which python-config not found. Stop. Makefile.perf:239: recipe for target 'sub-make' failed make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2 john@debian:~/linux/tools/perf$

JFYI, this is my alternatives:

sudo update-alternatives --config python
There are 3 choices for the alternative python (providing /usr/bin/python).

Selection Path Priority Status
------------------------------------------------------------

    0            /usr/local/bin/python3.6   50        auto mode
    1            /usr/bin/python2.7         1         manual mode
    2            /usr/bin/python3.5         2         manual mode

* 3 /usr/local/bin/python3.6 50 manual mode

Quoted 7 lines
Quoted 1 line

Press <enter> to keep the current choice[*], or type selection number:

Thanks Gary,

Perhaps it relates to dev packages. Perhaps, apt-get install python-dev-is-python3 which should resolve the symlinks. I wonder that you have python dev for python2 but python3 installed without the dev. We should be able to make a Makefile test for this.

So I was trying another system here as a sanity test (as my main dev box is not working either). And this other system is debian and only supports python up to 3.5, so that is why I have /usr/local/bin/python3.6 as an alt, i.e. I downloaded and built myself.

Anyway, I do have python3-dev:

john@debian:~/linux/tools/perf$ sudo apt-get install python3-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-dev is already the newest version (3.5.3-1).
author reviewer
#9 lore raw
Quoted 37 lines · John Garry
Quoted 22 lines · Ian Rogers
Quoted 22 lines

diff -u tools/perf/trace/beauty/include/linux/socket.h include/linux/socket.h
Makefile.config:259: *** which python-config not found. Stop. Makefile.perf:239: recipe for target 'sub-make' failed make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2 john@debian:~/linux/tools/perf$

JFYI, this is my alternatives:

sudo update-alternatives --config python
There are 3 choices for the alternative python (providing /usr/bin/python).

Selection Path Priority Status
------------------------------------------------------------

    0            /usr/local/bin/python3.6   50        auto mode
    1            /usr/bin/python2.7         1         manual mode
    2            /usr/bin/python3.5         2         manual mode

* 3 /usr/local/bin/python3.6 50 manual mode

Quoted 7 lines
Quoted 1 line

Press <enter> to keep the current choice[*], or type selection number:

Thanks Gary,

Perhaps it relates to dev packages. Perhaps, apt-get install python-dev-is-python3 which should resolve the symlinks. I wonder that you have python dev for python2 but python3 installed without the dev. We should be able to make a Makefile test for this.

So I was trying another system here as a sanity test (as my main dev box is not working either). And this other system is debian and only supports python up to 3.5, so that is why I have /usr/local/bin/python3.6 as an alt, i.e. I downloaded and built myself.

Quoted 8 lines

Anyway, I do have python3-dev:

john@debian:~/linux/tools/perf$ sudo apt-get install python3-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-dev is already the newest version (3.5.3-1).

It turned out to be something simple. The build is set up to prefer python2 over python3, so if you have both you get python2. I've fixed this now in v4 (PTAL):
https://lore.kernel.org/lkml/[email protected]/

I think we should remove python2 build support as it is confusing and python2 doesn't support setuptools meaning this patch fails:
https://lore.kernel.org/lkml/[email protected]/
and without that patch python3 generates deprecated module warnings.

Thanks,
Ian