Skip to contentSkip to conversation

perf jevents: Limit the number of JSON reading workers

PATCH Applied Sandipan Das 4 participants 4 messages Sep 17 – Sep 25 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf jevents: Limit the number of JSON reading workers

The ProcessPoolExecutor is created without max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaust RLIMIT_NOFILE as seen below.

File ".../multiprocessing/popen_fork.py", line 65, in _launch
child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

The error is reported, but jevents.py does not exit. The workers that did start stay blocked waiting for work and are never joined, so the build eventually hangs. Other targets keep building in the meantime, leaving the traceback several lines above the point where the build stalls.

Limit the pool instead of sizing it against the descriptor limit, which would mean relying on how many descriptors concurrent.futures needs for each worker. At least for recent AMD EPYC processors, reading the JSON files does not need many workers to be fast and things typically stop scaling beyond 32 workers. Hence, a small upper bound keeps the speedup and stays well inside any usable limit.

Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")
---
 tools/perf/pmu-events/jevents.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
tools/perf/pmu-events/jevents.py +10 -1
index 860027bb71b2..d81fe6757915 100755
@@ -1450,6 +1450,13 @@ const char *describe_metricgroup(const char *group)
 }
 """)
 
+# Upper bound on the number of workers reading JSON files in parallel.
+_max_parallel_workers = 32
+
+def _parallel_worker_count(num_tasks: int) -> int:
+  return max(1, min(getattr(os, 'process_cpu_count', os.cpu_count)() or 1,
+                    _max_parallel_workers, num_tasks))
+
 def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]:
   path, topic = task
   return path, topic, _read_json_events_impl(path, topic)
@@ -1549,7 +1556,9 @@ struct pmu_table_entry {
     preprocess_arch_std_files(arch_path)
     ftw(arch_path, [], collect_json)
 
-  with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor:
+  with concurrent.futures.ProcessPoolExecutor(max_workers=_parallel_worker_count(len(tasks)),
+                                              initializer=_init_worker,
+                                              initargs=(_arch_std_events,)) as executor:
     for path, topic, events in executor.map(_parallel_read_json_events, tasks):
       _json_cache[(path, topic)] = events
 
Signature · 1 line
2.53.0
#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines

The ProcessPoolExecutor is created without max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaust RLIMIT_NOFILE as seen below.

File ".../multiprocessing/popen_fork.py", line 65, in _launch
child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

The error is reported, but jevents.py does not exit. The workers that did start stay blocked waiting for work and are never joined, so the build eventually hangs. Other targets keep building in the meantime, leaving the traceback several lines above the point where the build stalls.

Limit the pool instead of sizing it against the descriptor limit, which would mean relying on how many descriptors concurrent.futures needs for each worker. At least for recent AMD EPYC processors, reading the JSON files does not need many workers to be fast and things typically stop scaling beyond 32 workers. Hence, a small upper bound keeps the speedup and stays well inside any usable limit.

Quoted 2 lines
Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer Reviewed-by on v1 1/1
#2 lore raw
In reply to Sandipan Das
Quoted 24 lines · Sandipan Das

The ProcessPoolExecutor is created without max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaust RLIMIT_NOFILE as seen below.

File ".../multiprocessing/popen_fork.py", line 65, in _launch
child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

The error is reported, but jevents.py does not exit. The workers that did start stay blocked waiting for work and are never joined, so the build eventually hangs. Other targets keep building in the meantime, leaving the traceback several lines above the point where the build stalls.

Limit the pool instead of sizing it against the descriptor limit, which would mean relying on how many descriptors concurrent.futures needs for each worker. At least for recent AMD EPYC processors, reading the JSON files does not need many workers to be fast and things typically stop scaling beyond 32 workers. Hence, a small upper bound keeps the speedup and stays well inside any usable limit.

Quoted 2 lines
Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")

Thanks,
Ian

Quoted 36 lines
---
 tools/perf/pmu-events/jevents.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
tools/perf/pmu-events/jevents.py +10 -2
index 860027bb71b2..d81fe6757915 100755
@@ -1450,6 +1450,13 @@ const char *describe_metricgroup(const char *group)
 }
 """)
 
+# Upper bound on the number of workers reading JSON files in parallel.
+_max_parallel_workers = 32
+
+def _parallel_worker_count(num_tasks: int) -> int:
+  return max(1, min(getattr(os, 'process_cpu_count', os.cpu_count)() or 1,
+                    _max_parallel_workers, num_tasks))
+
 def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]:
   path, topic = task
   return path, topic, _read_json_events_impl(path, topic)
@@ -1549,7 +1556,9 @@ struct pmu_table_entry {
     preprocess_arch_std_files(arch_path)
     ftw(arch_path, [], collect_json)
 
-  with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor:
+  with concurrent.futures.ProcessPoolExecutor(max_workers=_parallel_worker_count(len(tasks)),
+                                              initializer=_init_worker,
+                                              initargs=(_arch_std_events,)) as executor:
     for path, topic, events in executor.map(_parallel_read_json_events, tasks):
       _json_cache[(path, topic)] = events
 
--

2.53.0

maintainer Applied
#3 lore raw
Quoted 6 lines · Ian Rogers
Quoted 5 lines · Sandipan Das

The ProcessPoolExecutor is created without max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaust RLIMIT_NOFILE as seen below.

<SNIP>

Quoted 3 lines
Quoted 1 line
Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")

Thanks, applied to perf-tools-next, for v7.4.

- Arnaldo