perf build: prevent "argument list too long" error
PATCH 1/1 perf build: prevent "argument list too long" error
Due to a recent change, building perf may result in a build error when it is trying to "prune orphans". The file list passed to "rm" may exceed what the shell can handle. The build will then abort with an error like this:
TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]: /bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217: prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2
Processing the arguments via "xargs", instead of passing the list of files directly to "rm" via the shell, prevents this issue.
--- tools/perf/pmu-events/Build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/pmu-events/Build +2 -1
@@ -214,7 +214,8 @@ ifneq ($(strip $(ORPHAN_FILES)),) quiet_cmd_rm = RM $^ prune_orphans: $(ORPHAN_FILES) - $(Q)$(call echo-cmd,rm)rm -f $^ + # The list of files can be long. Use xargs to prevent issues. + $(Q)$(call echo-cmd,rm)echo "$^" | xargs rm -f JEVENTS_DEPS += prune_orphans endif
Signature · 1 line
2.52.0
Quoted 14 lines · Markus Mayer
Due to a recent change, building perf may result in a build error when it is trying to "prune orphans". The file list passed to "rm" may exceed what the shell can handle. The build will then abort with an error like this:
TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]:/bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217:prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2
Quoted 5 lines
Processing the arguments via "xargs", instead of passing the list of files directly to "rm" via the shell, prevents this issue.
Fixes 36a1b0061a5 (perf build: Reduce pmu-events related copying and mkdirs)Signed-off-by Markus Mayer <[email protected]>
As with:
https://lore.kernel.org/lkml/[email protected]/
I'm not able to reproduce the problem but the change makes sense.
Thanks,
Ian
Quoted 21 lines
--- tools/perf/pmu-events/Build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)tools/perf/pmu-events/Build +2
-2@@ -214,7 +214,8 @@ ifneq ($(strip $(ORPHAN_FILES)),) quiet_cmd_rm = RM $^ prune_orphans: $(ORPHAN_FILES) - $(Q)$(call echo-cmd,rm)rm -f $^ + # The list of files can be long. Use xargs to prevent issues. + $(Q)$(call echo-cmd,rm)echo "$^" | xargs rm -f JEVENTS_DEPS += prune_orphans endif --2.52.0
Quoted 11 lines · Ian Rogers
Quoted 8 lines · Markus Mayer
TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]:/bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217:prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2
Quoted 5 lines
As with:
https://lore.kernel.org/lkml/[email protected]/
I'm not able to reproduce the problem but the change makes sense.Reviewed-by Ian Rogers <[email protected]>
Thanks for the review. Not sure what makes my setup different from yours. FWIW, it's an ARM64 cross-compile. I had the build system dump the contents of $(ORPHAN_FILES) into a file for me. This is how many files it was trying to pass via the shell:
$ wc -l json_files.txt 634 json_files.txt
Maybe you have fewer orphans to prune or your shell has a higher the argument limit. Whatever the case, both our automated nightly builds and manual builds have been running into this issue.
Regards,
-Markus
Ian Rogers
reviewerQuoted 24 lines · Markus Mayer
Quoted 11 lines · Ian Rogers
Quoted 8 lines · Markus Mayer
TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]:/bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217:prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2Quoted 5 lines
As with:
https://lore.kernel.org/lkml/[email protected]/
I'm not able to reproduce the problem but the change makes sense.Reviewed-by Ian Rogers <[email protected]>Thanks for the review. Not sure what makes my setup different from yours. FWIW, it's an ARM64 cross-compile. I had the build system dump the contents of $(ORPHAN_FILES) into a file for me. This is how many files it was trying to pass via the shell:
Quoted 6 lines
$ wc -l json_files.txt 634 json_files.txtMaybe you have fewer orphans to prune or your shell has a higher the argument limit. Whatever the case, both our automated nightly builds and manual builds have been running into this issue.
Yes, building an x86 version and then cross-compiling for ARM would create a large set of orphan files requiring cleanup. Trying that with a giant output directory path I couldn't trigger a breakage, but I think I just have a very generous MAX_ARGS.
Thanks,
Ian
Quoted 2 lines
Regards,
-Markus
Quoted 13 lines · Markus Mayer
Due to a recent change, building perf may result in a build error when it is trying to "prune orphans". The file list passed to "rm" may exceed what the shell can handle. The build will then abort with an error like this:
TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]:/bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217:prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2
Quoted 5 lines
Processing the arguments via "xargs", instead of passing the list of files directly to "rm" via the shell, prevents this issue.
Fixes 36a1b0061a5 (perf build: Reduce pmu-events related copying and mkdirs)Signed-off-by Markus Mayer <[email protected]>
Thanks, applied to perf-tools, for v7.0.
- Arnaldo
Quoted 20 lines
--- tools/perf/pmu-events/Build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)tools/perf/pmu-events/Build +2
-1@@ -214,7 +214,8 @@ ifneq ($(strip $(ORPHAN_FILES)),) quiet_cmd_rm = RM $^ prune_orphans: $(ORPHAN_FILES) - $(Q)$(call echo-cmd,rm)rm -f $^ + # The list of files can be long. Use xargs to prevent issues. + $(Q)$(call echo-cmd,rm)echo "$^" | xargs rm -f JEVENTS_DEPS += prune_orphans endifSignature · 1 line
2.52.0