Skip to contentSkip to conversation

kbuild: make clean rule robust against too long argument error

PATCH In review Masahiro Yamada 2 participants 4 messages Jun 17, 2023 – Jun 19, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kbuild: make clean rule robust against too long argument error

Commit cd968b97c492 ("kbuild: make built-in.a rule robust against too long argument error") made a build rule robust against "Argument list too long" error.

Eugeniu Rosca reported the same error occurred when cleaning an external module.

The $(obj)/ prefix can be a very long path for external modules.

Apply a similar solution to 'make clean'.

---

 scripts/Makefile.clean | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/Makefile.clean +2 -1
index 3649900696dd..235408a44f90 100644
@@ -37,8 +37,9 @@ __clean-files   := $(wildcard $(addprefix $(obj)/, $(__clean-files)))
 
 # ==========================================================================
 
+# Use xargs to make this robust against "Argument list too long" error
 quiet_cmd_clean = CLEAN   $(obj)
-      cmd_clean = rm -rf $(__clean-files)
+      cmd_clean = printf '$(obj)/%s ' $(patsubst $(obj)/%,%,$(__clean-files)) | xargs rm -rf
 
 __clean: $(subdir-ymn)
 ifneq ($(strip $(__clean-files)),)
Signature · 1 line
2.39.2
Reviewed-by Tested-by on v1 1/1
#1 lore raw

Hello Yamada-san,

Many thanks for your feedback and for your patch. It indeed addresses the issue I have reported in [1].

Quoted 8 lines · Masahiro Yamada

Commit cd968b97c492 ("kbuild: make built-in.a rule robust against too long argument error") made a build rule robust against "Argument list too long" error.

Eugeniu Rosca reported the same error occurred when cleaning an external module.

The $(obj)/ prefix can be a very long path for external modules.

Confirmed. I am seeing an instance of $(obj) being 150 characters long, due to an out-of-tree module deeply buried in a specific Yocto build.

In the current vanilla version of 'make clean' (w/o this patch), the $(obj) prefix is applied to each and every file being removed, dramatically increasing the strlen of arguments passed to 'rm -rf'.

Quoted 15 lines

Apply a similar solution to 'make clean'.

---

 scripts/Makefile.clean | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/Makefile.clean +0 -0
index 3649900696dd..235408a44f90 100644

scripts/Makefile.clean __clean-files := $(wildcard $(addprefix $(obj)/, $(__clean-files))) · line 40

Quoted 4 lines
scripts/Makefile.clean +1 -0
@@ -37,8 +37,9 @@ __clean-files   := $(wildcard $(addprefix $(obj)/, $(__clean-files)))
 
 # ==========================================================================
 
+# Use xargs to make this robust against "Argument list too long" error

Please, correct me if I am wrong, but it looks like the magic/brilliance is in the 'patsubst' function, since below version also fails for me:

NOK: cmd_clean = printf '%s ' $(__clean-files) | xargs rm -rf

When running 'make clean' for my particular out-of-tree kernel module, 'patsubst' is able to decrease the total number of characters passed to the shell's argument list from ~204k to ~47k, which elegantly prevents the "Argument list too long" error.

Quoted 9 lines
scripts/Makefile.clean +1 -1
 quiet_cmd_clean = CLEAN   $(obj)
-      cmd_clean = rm -rf $(__clean-files)
+      cmd_clean = printf '$(obj)/%s ' $(patsubst $(obj)/%,%,$(__clean-files)) | xargs rm -rf
 
 __clean: $(subdir-ymn)
 ifneq ($(strip $(__clean-files)),)
Signature · 1 line
2.39.2

Since it fixes the problem reported in [1] (much appreciated):

[1] https://lore.kernel.org/linux-kbuild/20230616194505.GA27753@lxhi-065/

Signature · 2 lines
Best regards,
Eugeniu Rosca
#2 lore raw
Quoted 39 lines · Eugeniu Rosca

Hello Yamada-san,

Many thanks for your feedback and for your patch. It indeed addresses the issue I have reported in [1].

Quoted 8 lines · Masahiro Yamada

Commit cd968b97c492 ("kbuild: make built-in.a rule robust against too long argument error") made a build rule robust against "Argument list too long" error.

Eugeniu Rosca reported the same error occurred when cleaning an external module.

The $(obj)/ prefix can be a very long path for external modules.

Confirmed. I am seeing an instance of $(obj) being 150 characters long, due to an out-of-tree module deeply buried in a specific Yocto build.

In the current vanilla version of 'make clean' (w/o this patch), the $(obj) prefix is applied to each and every file being removed, dramatically increasing the strlen of arguments passed to 'rm -rf'.

Quoted 16 lines

Apply a similar solution to 'make clean'.

---

 scripts/Makefile.clean | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/Makefile.clean +0 -0
index 3649900696dd..235408a44f90 100644
@@ -37,8 +37,9 @@ __clean-files   := $(wildcard $(addprefix $(obj)/, $(__clean-files)))
 
Quoted 8 lines

scripts/Makefile.clean __clean-files := $(wildcard $(addprefix $(obj)/, $(__clean-files))) · line 40

Quoted 3 lines
scripts/Makefile.clean +1 -0
 # ==========================================================================
 
+# Use xargs to make this robust against "Argument list too long" error

Please, correct me if I am wrong, but it looks like the magic/brilliance is in the 'patsubst' function, since below version also fails for me:

NOK: cmd_clean = printf '%s ' $(__clean-files) | xargs rm -rf

Right.
Now, 'printf' (instead of 'rm') failed with the too long argument list.

GNU Make does not have the length limit, but shell does. So, the full-path list must be passed via stdout instead of the command line.

The comment might be confusing.

I will repeat the same comment written in scripts/Makefile.build:

# To make this rule robust against "Argument list too long" error,

  # remove $(obj)/ prefix, and restore it by a shell command.
Signature · 2 lines
Best Regards
Masahiro Yamada
#3 lore raw

Hello Yamada-san,

Quoted 1 line · Masahiro Yamada

On Sun, Jun 18, 2023 at 5:07 AM Eugeniu Rosca <[email protected]> wrote:

[snip]

Quoted 13 lines
Quoted 4 lines

Please, correct me if I am wrong, but it looks like the magic/brilliance is in the 'patsubst' function, since below version also fails for me:

NOK: cmd_clean = printf '%s ' $(__clean-files) | xargs rm -rf

Right.
Now, 'printf' (instead of 'rm') failed with the too long argument list.

GNU Make does not have the length limit, but shell does. So, the full-path list must be passed via stdout instead of the command line.

Quoted 6 lines

The comment might be confusing.

I will repeat the same comment written in scripts/Makefile.build:

# To make this rule robust against "Argument list too long" error,

  # remove $(obj)/ prefix, and restore it by a shell command.

Sounds good to me. No open questions from my end :)

Signature · 2 lines
Best regards,
Eugeniu Rosca