perf jevents: Limit the number of JSON reading workers
PATCH 1/1 perf jevents: Limit the number of JSON reading workers
The ProcessPoolExecutor is created without max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaust RLIMIT_NOFILE as seen below.
File ".../multiprocessing/popen_fork.py", line 65, in _launch
child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files
The error is reported, but jevents.py does not exit. The workers that did start stay blocked waiting for work and are never joined, so the build eventually hangs. Other targets keep building in the meantime, leaving the traceback several lines above the point where the build stalls.
Limit the pool instead of sizing it against the descriptor limit, which would mean relying on how many descriptors concurrent.futures needs for each worker. At least for recent AMD EPYC processors, reading the JSON files does not need many workers to be fast and things typically stop scaling beyond 32 workers. Hence, a small upper bound keeps the speedup and stays well inside any usable limit.
--- tools/perf/pmu-events/jevents.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
tools/perf/pmu-events/jevents.py +10 -1
@@ -1450,6 +1450,13 @@ const char *describe_metricgroup(const char *group) } """) +# Upper bound on the number of workers reading JSON files in parallel. +_max_parallel_workers = 32 + +def _parallel_worker_count(num_tasks: int) -> int: + return max(1, min(getattr(os, 'process_cpu_count', os.cpu_count)() or 1, + _max_parallel_workers, num_tasks)) + def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]: path, topic = task return path, topic, _read_json_events_impl(path, topic)
@@ -1549,7 +1556,9 @@ struct pmu_table_entry { preprocess_arch_std_files(arch_path) ftw(arch_path, [], collect_json) - with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor: + with concurrent.futures.ProcessPoolExecutor(max_workers=_parallel_worker_count(len(tasks)), + initializer=_init_worker, + initargs=(_arch_std_events,)) as executor: for path, topic, events in executor.map(_parallel_read_json_events, tasks): _json_cache[(path, topic)] = events
Signature · 1 line
2.53.0
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines
The ProcessPoolExecutor is created without
max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaustRLIMIT_NOFILEas seen below.File "
.../multiprocessing/popen_fork.py", line 65, in_launch
child_r,parent_w=os.pipe()
OSError: [Errno 24] Too many open filesThe error is reported, but
jevents.pydoes not exit. The workers that did start stay blocked waiting for work and are never joined, so the build eventually hangs. Other targets keep building in the meantime, leaving the traceback several lines above the point where the build stalls.Limit the pool instead of sizing it against the descriptor limit, which would mean relying on how many descriptors concurrent.futures needs for each worker. At least for recent AMD EPYC processors, reading the JSON files does not need many workers to be fast and things typically stop scaling beyond 32 workers. Hence, a small upper bound keeps the speedup and stays well inside any usable limit.
Quoted 2 lines
Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")Signed-off-by Sandipan Das <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 24 lines · Sandipan Das
The ProcessPoolExecutor is created without
max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaustRLIMIT_NOFILEas seen below.File "
.../multiprocessing/popen_fork.py", line 65, in_launch
child_r,parent_w=os.pipe()
OSError: [Errno 24] Too many open filesThe error is reported, but
jevents.pydoes not exit. The workers that did start stay blocked waiting for work and are never joined, so the build eventually hangs. Other targets keep building in the meantime, leaving the traceback several lines above the point where the build stalls.Limit the pool instead of sizing it against the descriptor limit, which would mean relying on how many descriptors concurrent.futures needs for each worker. At least for recent AMD EPYC processors, reading the JSON files does not need many workers to be fast and things typically stop scaling beyond 32 workers. Hence, a small upper bound keeps the speedup and stays well inside any usable limit.
Quoted 2 lines
Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")Signed-off-by Sandipan Das <[email protected]>
Thanks,
Ian
Quoted 36 lines
--- tools/perf/pmu-events/jevents.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)tools/perf/pmu-events/jevents.py +10
-2@@ -1450,6 +1450,13 @@ const char *describe_metricgroup(const char *group) } """) +# Upper bound on the number of workers reading JSON files in parallel. +_max_parallel_workers = 32 + +def _parallel_worker_count(num_tasks: int) -> int: + return max(1, min(getattr(os, 'process_cpu_count', os.cpu_count)() or 1, + _max_parallel_workers, num_tasks)) + def _parallel_read_json_events(task: Tuple[str, str]) -> Tuple[str, str, Sequence[JsonEvent]]: path, topic = task return path, topic, _read_json_events_impl(path, topic)@@ -1549,7 +1556,9 @@ struct pmu_table_entry { preprocess_arch_std_files(arch_path) ftw(arch_path, [], collect_json) - with concurrent.futures.ProcessPoolExecutor(initializer=_init_worker, initargs=(_arch_std_events,)) as executor: + with concurrent.futures.ProcessPoolExecutor(max_workers=_parallel_worker_count(len(tasks)), + initializer=_init_worker, + initargs=(_arch_std_events,)) as executor: for path, topic, events in executor.map(_parallel_read_json_events, tasks): _json_cache[(path, topic)] = events --2.53.0
Quoted 6 lines · Ian Rogers
Quoted 5 lines · Sandipan Das
The ProcessPoolExecutor is created without
max_workers, so concurrent.futures defaults to one worker per CPU and forks all of them when the first task is submitted. Each worker costs the parent a few file descriptors. For systems with high core-count processors like the AMD EPYC 9996, the pool can easily exhaustRLIMIT_NOFILEas seen below.
<SNIP>
Quoted 3 lines
Quoted 1 line
Fixes eaab2eb09dc2 ("perf pmu-events: Parallelize JSON and metric pre-computation in jevents.py")Reviewed-by Ian Rogers <[email protected]>
Thanks, applied to perf-tools-next, for v7.4.
- Arnaldo