Skip to contentSkip to conversation

perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py

PATCH Applied Ian Rogers 2 participants 2 messages Jul 22 – Jul 26 perf events

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

PATCH 1/1 perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py

Currently, jevents.py parses hundreds of JSON event and metric files sequentially across all CPU architectures during Kbuild startup, taking ~3.5 seconds of single-core execution time.

Refactor jevents.py to pre-populate its internal JSON AST cache in parallel across all available CPU cores using ProcessPoolExecutor. First gather all the paths with ftw and collect_json, then spawn _parallel_read_json_events that starts workers to just read the json events. Define the worker process initializer _init_worker so that _arch_std_events is available under spawn multiprocessing semantics.

This accelerates the JSON parsing phase by over 10x (from ~3.0s down to ~290ms), reducing overall jevents.py execution time by 3.5x (from ~3.56s down to ~1.03s).

Assisted-by Gemini:gemini-3.1-pro-preview

---
This change was originally part of a larger set of build speed ups, but was taken out to focus on more important fixes. Mailing now separately. The original series at v7 with this change is:
https://lore.kernel.org/lkml/[email protected]/

Note, the change assumes Python 3.9 for type annotations. The parallel execution APIs were added in Python 3.7. This means the python code should meet the minimum build requirements in:
https://www.kernel.org/doc/Documentation/Changes

---
 tools/perf/pmu-events/jevents.py | 36 ++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)
tools/perf/pmu-events/jevents.py +32 -4
index 5c15485d695a..860027bb71b2 100755
@@ -464,8 +464,8 @@ class JsonEvent:
     return f'{make_comment(s)}\t{{ { _bcs.offsets[s] } }},\n'
 
 
-@lru_cache(maxsize=None)
-def read_json_events(path: str, topic: str) -> Sequence[JsonEvent]:
+_json_cache = {}
+def _read_json_events_impl(path: str, topic: str) -> Sequence[JsonEvent]:
   """Read json events from the specified file."""
   try:
     events = json.load(open(path), object_hook=JsonEvent)
@@ -481,12 +481,16 @@ def read_json_events(path: str, topic: str) -> Sequence[JsonEvent]:
   if updates:
     for event in events:
       if event.metric_name in updates:
-        # print(f'Updated {event.metric_name} from\n"{event.metric_expr}"\n'
-        #       f'to\n"{updates[event.metric_name]}"')
         event.metric_expr = updates[event.metric_name]
 
   return events
 
+def read_json_events(path: str, topic: str) -> Sequence[JsonEvent]:
+  key = (path, topic)
+  if key not in _json_cache:
+    _json_cache[key] = _read_json_events_impl(path, topic)
+  return _json_cache[key]
+
 def preprocess_arch_std_files(archpath: str) -> None:
   """Read in all architecture standard events."""
   global _arch_std_events
@@ -1446,6 +1450,14 @@ const char *describe_metricgroup(const char *group)
 }
 """)
 
+def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]:
+  path, topic = task
+  return path, topic, _read_json_events_impl(path, topic)
+
+def _init_worker(std_events: dict) -> None:
+  global _arch_std_events
+  _arch_std_events = std_events
+
 def main() -> None:
   global _args
 
@@ -1524,9 +1536,25 @@ struct pmu_table_entry {
     raise IOError(f'Missing architecture directory \'{_args.arch}\'')
 
   archs.sort()
+  import concurrent.futures
+  tasks = []
+  def collect_json(parents: Sequence[str], item: os.DirEntry) -> None:
+    if len(parents) == 0:
+      return
+    if item.is_file() and item.name.endswith('.json') and not item.name.endswith('metricgroups.json'):
+      tasks.append((item.path, get_topic(item.name)))
+
   for arch in archs:
     arch_path = f'{_args.starting_dir}/{arch}'
     preprocess_arch_std_files(arch_path)
+    ftw(arch_path, [], collect_json)
+
+  with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor:
+    for path, topic, events in executor.map(_parallel_read_json_events, tasks):
+      _json_cache[(path, topic)] = events
+
+  for arch in archs:
+    arch_path = f'{_args.starting_dir}/{arch}'
     ftw(arch_path, [], preprocess_one_file)
 
   assert _bcs is not None
Signature · 1 line
2.55.0.229.g6434b31f56-goog
maintainer Applied
#1 lore raw
Quoted 12 lines · Ian Rogers

Currently, jevents.py parses hundreds of JSON event and metric files sequentially across all CPU architectures during Kbuild startup, taking ~3.5 seconds of single-core execution time.

Refactor jevents.py to pre-populate its internal JSON AST cache in parallel across all available CPU cores using ProcessPoolExecutor. First gather all the paths with ftw and collect_json, then spawn _parallel_read_json_events that starts workers to just read the json events. Define the worker process initializer _init_worker so that _arch_std_events is available under spawn multiprocessing semantics.

Quoted 1 line

[...]

Applied to perf-tools-next, thanks!

Best regards,
Namhyung