Skip to contentSkip to conversation

perf build: prevent "argument list too long" error

PATCH Applied Markus Mayer 3 participants 5 messages Mar 3 – Mar 4 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf build: prevent "argument list too long" error

Due to a recent change, building perf may result in a build error when it is trying to "prune orphans". The file list passed to "rm" may exceed what the shell can handle. The build will then abort with an error like this:

TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]: /bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217: prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2

Processing the arguments via "xargs", instead of passing the list of files directly to "rm" via the shell, prevents this issue.

Fixes 36a1b0061a5 (perf build: Reduce pmu-events related copying and mkdirs)
---
 tools/perf/pmu-events/Build | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/pmu-events/Build +2 -1
index 63c65788d442..dc5f94862a3b 100644
@@ -214,7 +214,8 @@ ifneq ($(strip $(ORPHAN_FILES)),)
 quiet_cmd_rm  = RM      $^
 
 prune_orphans: $(ORPHAN_FILES)
-	$(Q)$(call echo-cmd,rm)rm -f $^
+	# The list of files can be long. Use xargs to prevent issues.
+	$(Q)$(call echo-cmd,rm)echo "$^" | xargs rm -f
 
 JEVENTS_DEPS += prune_orphans
 endif
Signature · 1 line
2.52.0
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 14 lines · Markus Mayer

Due to a recent change, building perf may result in a build error when it is trying to "prune orphans". The file list passed to "rm" may exceed what the shell can handle. The build will then abort with an error like this:

TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]: /bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217: prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2

Quoted 5 lines

Processing the arguments via "xargs", instead of passing the list of files directly to "rm" via the shell, prevents this issue.

Fixes 36a1b0061a5 (perf build: Reduce pmu-events related copying and mkdirs)

As with:
https://lore.kernel.org/lkml/[email protected]/
I'm not able to reproduce the problem but the change makes sense.

Thanks,
Ian

Quoted 21 lines
---
 tools/perf/pmu-events/Build | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/pmu-events/Build +2 -2
index 63c65788d442..dc5f94862a3b 100644
@@ -214,7 +214,8 @@ ifneq ($(strip $(ORPHAN_FILES)),)
 quiet_cmd_rm  = RM      $^
 
 prune_orphans: $(ORPHAN_FILES)
-       $(Q)$(call echo-cmd,rm)rm -f $^
+       # The list of files can be long. Use xargs to prevent issues.
+       $(Q)$(call echo-cmd,rm)echo "$^" | xargs rm -f
 
 JEVENTS_DEPS += prune_orphans
 endif
--

2.52.0

#2 lore raw
Quoted 11 lines · Ian Rogers
Quoted 8 lines · Markus Mayer

TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]: /bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217: prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2

Quoted 5 lines

As with:
https://lore.kernel.org/lkml/[email protected]/
I'm not able to reproduce the problem but the change makes sense.

Thanks for the review. Not sure what makes my setup different from yours. FWIW, it's an ARM64 cross-compile. I had the build system dump the contents of $(ORPHAN_FILES) into a file for me. This is how many files it was trying to pass via the shell:

$ wc -l json_files.txt
634 json_files.txt

Maybe you have fewer orphans to prune or your shell has a higher the argument limit. Whatever the case, both our automated nightly builds and manual builds have been running into this issue.

Regards,
-Markus

reviewer
#3 lore raw
Quoted 24 lines · Markus Mayer
Quoted 11 lines · Ian Rogers
Quoted 8 lines · Markus Mayer

TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]: /bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217: prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2

Quoted 5 lines

As with:
https://lore.kernel.org/lkml/[email protected]/
I'm not able to reproduce the problem but the change makes sense.

Thanks for the review. Not sure what makes my setup different from yours. FWIW, it's an ARM64 cross-compile. I had the build system dump the contents of $(ORPHAN_FILES) into a file for me. This is how many files it was trying to pass via the shell:

Quoted 6 lines
$ wc -l json_files.txt
634 json_files.txt

Maybe you have fewer orphans to prune or your shell has a higher the argument limit. Whatever the case, both our automated nightly builds and manual builds have been running into this issue.

Yes, building an x86 version and then cross-compiling for ARM would create a large set of orphan files requiring cleanup. Trying that with a giant output directory path I couldn't trigger a breakage, but I think I just have a very generous MAX_ARGS.

Thanks,
Ian

Quoted 2 lines

Regards,
-Markus

maintainer Applied
#4 lore raw
In reply to Markus Mayer
Quoted 13 lines · Markus Mayer

Due to a recent change, building perf may result in a build error when it is trying to "prune orphans". The file list passed to "rm" may exceed what the shell can handle. The build will then abort with an error like this:

TEST [...]/arm64/build/linux-custom/tools/perf/pmu-events/metric_test.log
make[5]: /bin/sh: Argument list too long
make[5]: *** [pmu-events/Build:217: prune_orphans] Error 127
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile.perf:773: [...]/tools/perf/pmu-events/pmu-events-in.o] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:289: sub-make] Error 2

Quoted 5 lines

Processing the arguments via "xargs", instead of passing the list of files directly to "rm" via the shell, prevents this issue.

Fixes 36a1b0061a5 (perf build: Reduce pmu-events related copying and mkdirs)

Thanks, applied to perf-tools, for v7.0.

- Arnaldo

Quoted 20 lines
---
 tools/perf/pmu-events/Build | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/pmu-events/Build +2 -1
index 63c65788d442..dc5f94862a3b 100644
@@ -214,7 +214,8 @@ ifneq ($(strip $(ORPHAN_FILES)),)
 quiet_cmd_rm  = RM      $^
 
 prune_orphans: $(ORPHAN_FILES)
-	$(Q)$(call echo-cmd,rm)rm -f $^
+	# The list of files can be long. Use xargs to prevent issues.
+	$(Q)$(call echo-cmd,rm)echo "$^" | xargs rm -f
 
 JEVENTS_DEPS += prune_orphans
 endif
Signature · 1 line
2.52.0