make headers_install fail when path is too long

This problem has probably already been reported, but I don't find the fix in a kernel.org tree and I don't understand why.

Quoted 2 lines

This problem has probably already been reported, but I don't find the fix in a kernel.org tree and I don't understand why.

First time I see this bug described - but I may missed it here at kbuild. If a patch exists then please include the patch in your mail so we can see it and comment on it.

Quoted 2 lines · Nicolas Dichtel

This problem has probably already been reported, but I don't find the fix in a kernel.org tree and I don't understand why.

First time I see this bug described - but I may missed it here at kbuild.

If a patch exists then please include the patch in your mail so we can see it and comment on it.

And make sure to have the attribution correct if you consider it to be applied.

Quoted 6 lines · Sam Ravnborg

Quoted 3 lines · Nicolas Dichtel

This problem has probably already been reported, but I don't find the fix
in a kernel.org tree and I don't understand why.

First time I see this bug described - but I may missed it here at kbuild.

It's probably why I didn't find this on a linux kernel mailing list ;-)

Quoted 6 lines

If a patch exists then please include the patch in your mail so we can see it and comment on it.

And make sure to have the attribution correct if you consider it to be applied.

I'm not the author of the patch, I've found it on the web. Here it is:

From cb9c811b8f23277de95dc687e87c6859308e68e6 Mon Sep 17 00:00:00 2001
From: Bruce Ashfield <[email protected]>
Date: Wed, 6 Mar 2013 17:23:51 +0100
Subject: [PATCH] scripts/Makefile.headersinst: install headers from scratch
file

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation
with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

By dumping the input files to a scratch file and using xargs to read the input list from the scratch file, we can avoid blowing out the maximum argument size and install headers in a long path name environment.

Signed-off-by Bruce Ashfield <[email protected]>

---
  scripts/Makefile.headersinst | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

scripts/Makefile.headersinst +1 -1

index 25f216a..c481986 100644
@@ -71,7 +71,7 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
  quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
                              file$(if $(word 2, $(all-files)),s))
        cmd_install = \
-        $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \
+        xargs $(PERL) $< $(installdir) $(SRCARCH) <
 $(INSTALL_HDR_PATH)/.input-files; \
          for F in $(wrapper-files); do                                   \
                  echo "\#include <asm-generic/$$F>" >

$(installdir)/$$F;    \
          done;                                                           \

scripts/Makefile.headersinst +2 -0

@@ -100,7 +100,9 @@ targets += $(install-file)
  $(install-file): scripts/headers_install.pl $(input-files) FORCE
      $(if $(unwanted),$(call cmd,remove),)
      $(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
+    @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files
      $(call if_changed,install)
+    @rm $(INSTALL_HDR_PATH)/.input-files
 
  else
  __headerscheck: $(subdirs) $(check-file)

- $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \

+ @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files

Dne 6.3.2013 17:27, Nicolas Dichtel napsal(a):

Quoted 1 line

- $(PERL) $< $(installdir) $(SRCARCH) $(input-files); \

[...]

Quoted 1 line

+ @echo $(input-files) > $(INSTALL_HDR_PATH)/.input-files

Are you sure this is a reliable fix? What make does is to spawn

/bin/sh -c 'echo <long list of files> > usr/include/.input-files'

here. So I guess that it works for you just because "sh -c" is shorter than "sh -c 'perl scripts/headers_install.pl...'".

Michal

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible path, I have tree input-files list, one per path.

Signed-off-by Nicolas Dichtel<[email protected]>

---

I come back to this issue. Here is another proposal to fix this pb. Comments are welcome.

  scripts/Makefile.headersinst | 20 ++++++++++++++------
  scripts/headers_install.pl   |  8 ++++----
  2 files changed, 18 insertions(+), 10 deletions(-)

scripts/Makefile.headersinst +14 -6

index 477d137..fbadfc3 100644
@@ -47,18 +47,24 @@ header-y      := $(filter-out $(generic-y), $(header-y))
  all-files     := $(header-y) $(genhdr-y) $(wrapper-files)
  output-files  := $(addprefix $(installdir)/, $(all-files))
 
-input-files   := $(foreach hdr, $(header-y), \
+input-files1  := $(foreach hdr, $(header-y), \
  		   $(if $(wildcard $(srcdir)/$(hdr)), \
-			$(wildcard $(srcdir)/$(hdr)), \
+			$(wildcard $(srcdir)/$(hdr))) \
+		   )
+input-files1-name := $(notdir $(input-files1))
+input-files2  := $(foreach hdr, $(header-y), \
+		   $(if  $(wildcard $(srcdir)/$(hdr)),, \
  			$(if $(wildcard $(oldsrcdir)/$(hdr)), \
  				$(wildcard $(oldsrcdir)/$(hdr)), \
  				$(error Missing UAPI file $(srcdir)/$(hdr))) \
-		   )) \
-		 $(foreach hdr, $(genhdr-y), \
+		   ))
+input-files2-name := $(notdir $(input-files2))
+input-files3  := $(foreach hdr, $(genhdr-y), \
  		   $(if	$(wildcard $(gendir)/$(hdr)), \
  			$(wildcard $(gendir)/$(hdr)), \
  			$(error Missing generated UAPI file $(gendir)/$(hdr)) \
  		   ))
+input-files3-name := $(notdir $(input-files3))
 
  # Work out what needs to be removed
  oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))

@@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
  quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
                              file$(if $(word 2, $(all-files)),s))
        cmd_install = \
-        $(PERL) $<  $(installdir) $(SRCARCH) $(input-files); \
+        $(PERL) $<  $(installdir) $(SRCARCH) $(srcdir) $(input-files1-name); \
+        $(PERL) $<  $(installdir) $(SRCARCH) $(oldsrcdir) $(input-files2-name); \
+        $(PERL) $<  $(installdir) $(SRCARCH) $(gendir) $(input-files3-name); \
          for F in $(wrapper-files); do                                   \
                  echo "\#include<asm-generic/$$F>">  $(installdir)/$$F;    \
          done;                                                           \

@@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file)
  	@:
 
  targets += $(install-file)
-$(install-file): scripts/headers_install.pl $(input-files) FORCE
+$(install-file): scripts/headers_install.pl $(input-files1) $(input-files2) $(input-files3) FORCE
  	$(if $(unwanted),$(call cmd,remove),)
  	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
  	$(call if_changed,install)

scripts/headers_install.pl +4 -4

index 581ca99..87d9700 100644
@@ -3,7 +3,7 @@
  # headers_install prepare the listed header files for use in
  # user space and copy the files to their destination.
  #
-# Usage: headers_install.pl readdir installdir arch [files...]
+# Usage: headers_install.pl readdir installdir arch srcdir [files...]
  # installdir: dir to install the files to
  # arch:       current architecture
  #             arch is used to force a reinstallation when the arch

@@ -17,7 +17,7 @@
 
  use strict;
 
-my ($installdir, $arch, @files) = @ARGV;
+my ($installdir, $arch, $srcdir, @files) = @ARGV;
 
  my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__";
 

@@ -27,8 +27,8 @@ foreach my $filename (@files) {
 
  	my $tmpfile = "$installdir/$file.tmp";
 
-	open(my $in, '<', $filename)
-	    or die "$filename: $!\n";
+	open(my $in, '<', "$srcdir/$filename")
+	    or die "$srcdir/$filename: $!\n";
  	open(my $out, '>', $tmpfile)
  	    or die "$tmpfile: $!\n";
  	while (my $line =<$in>) {

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible path, I have tree input-files list, one per path.

Signed-off-by Nicolas Dichtel <[email protected]>

---

I come back to this issue. Here is another proposal to fix this pb. Comments are welcome.

 scripts/Makefile.headersinst | 20 ++++++++++++++------
 scripts/headers_install.pl   |  8 ++++----

Quoted 17 lines · Nicolas Dichtel

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible path, I have tree input-files list, one per path.

Signed-off-by Nicolas Dichtel <[email protected]>

---

Quoted 5 lines

I come back to this issue. Here is another proposal to fix this pb. Comments are welcome.

 scripts/Makefile.headersinst | 20 ++++++++++++++------
 scripts/headers_install.pl   |  8 ++++----

In the kbuild tree the perl script has been replaced by a shell script. You should base you patch on the kbuild-tree (or -next).

Quoted 21 lines · Sam Ravnborg

Quoted 20 lines · Nicolas Dichtel

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible path, I have tree input-files list, one per path.

Signed-off-by Nicolas Dichtel <[email protected]>

---

I come back to this issue. Here is another proposal to fix this pb. Comments are welcome.

Quoted 5 lines

Quoted 2 lines

  scripts/Makefile.headersinst | 20 ++++++++++++++------
  scripts/headers_install.pl   |  8 ++++----

In the kbuild tree the perl script has been replaced by a shell script. You should base you patch on the kbuild-tree (or -next).

Yes. Otherwise, the patch looks good. At least until the number of userspace headers explodes :-).

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible paths, I have tree input-files list, one per path.

Signed-off-by Nicolas Dichtel <[email protected]>

---

v2: rebase patch on linux-next

  scripts/Makefile.headersinst | 20 ++++++++++++++------
  scripts/headers_install.sh   |  7 +++++--
  2 files changed, 19 insertions(+), 8 deletions(-)

scripts/Makefile.headersinst +14 -6

index 182084d..8ccf830 100644
@@ -47,18 +47,24 @@ header-y      := $(filter-out $(generic-y), $(header-y))
  all-files     := $(header-y) $(genhdr-y) $(wrapper-files)
  output-files  := $(addprefix $(installdir)/, $(all-files))
 
-input-files   := $(foreach hdr, $(header-y), \
+input-files1  := $(foreach hdr, $(header-y), \
  		   $(if $(wildcard $(srcdir)/$(hdr)), \
-			$(wildcard $(srcdir)/$(hdr)), \
+			$(wildcard $(srcdir)/$(hdr))) \
+		   )
+input-files1-name := $(notdir $(input-files1))
+input-files2  := $(foreach hdr, $(header-y), \
+		   $(if  $(wildcard $(srcdir)/$(hdr)),, \
  			$(if $(wildcard $(oldsrcdir)/$(hdr)), \
  				$(wildcard $(oldsrcdir)/$(hdr)), \
  				$(error Missing UAPI file $(srcdir)/$(hdr))) \
-		   )) \
-		 $(foreach hdr, $(genhdr-y), \
+		   ))
+input-files2-name := $(notdir $(input-files2))
+input-files3  := $(foreach hdr, $(genhdr-y), \
  		   $(if	$(wildcard $(gendir)/$(hdr)), \
  			$(wildcard $(gendir)/$(hdr)), \
  			$(error Missing generated UAPI file $(gendir)/$(hdr)) \
  		   ))
+input-files3-name := $(notdir $(input-files3))
 
  # Work out what needs to be removed
  oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))

@@ -72,7 +78,9 @@ printdir = $(patsubst $(INSTALL_HDR_PATH)/%/,%,$(dir $@))
  quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
                              file$(if $(word 2, $(all-files)),s))
        cmd_install = \
-        $(CONFIG_SHELL) $< $(installdir) $(input-files); \
+        $(CONFIG_SHELL) $< $(installdir) $(srcdir) $(input-files1-name); \
+        $(CONFIG_SHELL) $< $(installdir) $(oldsrcdir) $(input-files2-name); \
+        $(CONFIG_SHELL) $< $(installdir) $(gendir) $(input-files3-name); \
          for F in $(wrapper-files); do                                   \
                  echo "\#include <asm-generic/$$F>" > $(installdir)/$$F;    \
          done;                                                           \

@@ -98,7 +106,7 @@ __headersinst: $(subdirs) $(install-file)
  	@:
 
  targets += $(install-file)
-$(install-file): scripts/headers_install.sh $(input-files) FORCE
+$(install-file): scripts/headers_install.sh $(input-files1) $(input-files2) $(input-files3) FORCE
  	$(if $(unwanted),$(call cmd,remove),)
  	$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
  	$(call if_changed,install)

scripts/headers_install.sh +5 -2

index 643764f..5de5660 100644
@@ -2,7 +2,7 @@
 
  if [ $# -lt 1 ]
  then
-	echo "Usage: headers_install.sh OUTDIR [FILES...]
+	echo "Usage: headers_install.sh OUTDIR SRCDIR [FILES...]
  	echo
  	echo "Prepares kernel header files for use by user space, by removing"
  	echo "all compiler.h definitions and #includes, removing any"

@@ -10,6 +10,7 @@ then
  	echo "asm/inline/volatile keywords."
  	echo
  	echo "OUTDIR: directory to write each userspace header FILE to."
+	echo "SRCDIR: source directory where files are picked."
  	echo "FILES:  list of header files to operate on."
 
  	exit 1

@@ -19,6 +20,8 @@ fi
 
  OUTDIR="$1"
  shift
+SRCDIR="$1"
+shift
 
  # Iterate through files listed on command line
 

@@ -34,7 +37,7 @@ do
  		-e 's/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g' \
  		-e 's/(^|[ \t(])(inline|asm|volatile)([ \t(]|$)/\1__\2__\3/g' \
  		-e 's@#(ifndef|define|endif[ \t]*/[*])[ \t]*_UAPI@#\1 @' \
-		"$i" > "$OUTDIR/$FILE.sed" || exit 1
+		"$SRCDIR/$i" > "$OUTDIR/$FILE.sed" || exit 1
  	scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ "$OUTDIR/$FILE.sed" \
  		> "$OUTDIR/$FILE"
  	[ $? -gt 1 ] && exit 1

Quoted 11 lines · Nicolas Dichtel

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Quoted 2 lines

Because there is three possible paths, I have tree input-files list, one per path.

This version still works for my test case.

Tested-by Bruce Ashfield <[email protected]>

Quoted 3 lines

Signed-off-by Nicolas Dichtel <[email protected]>

---

Quoted 15 lines · Bruce Ashfield

Quoted 13 lines · Nicolas Dichtel

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible paths, I have tree input-files list, one per path.

Quoted 7 lines

This version still works for my test case.

Tested-by Bruce Ashfield <[email protected]>

Quoted 3 lines

Signed-off-by Nicolas Dichtel <[email protected]>

---

Should I do something more for this patch? It seems not included.

Quoted 18 lines · Nicolas Dichtel

Quoted 17 lines · Bruce Ashfield

Quoted 7 lines · Nicolas Dichtel

If headers_install is executed from a deep/long directory structure, the shell's maximum argument length can be execeeded, which breaks the operation with:

| make[2]: execvp: /bin/sh: Argument list too long | make[2]: ***

Quoted 6 lines

Instead of passing each files name with the entire path, I give only the file name without the source path and give this path as a new argument to headers_install.pl.

Because there is three possible paths, I have tree input-files list, one per path.

This version still works for my test case.

Quoted 6 lines

Quoted 5 lines

Tested-by Bruce Ashfield <[email protected]>

Quoted 3 lines

Signed-off-by Nicolas Dichtel <[email protected]>

---

Should I do something more for this patch? It seems not included.

ping