Skip to contentSkip to conversation

scripts: headers_install: avoid some reprocessing

PATCH In review David Disseldorp 3 participants 8 messages Jan 7 – Jan 29 Kbuild

lore
Share & tools
lore
#0 lore raw

v2 cover scripts: headers_install: avoid some reprocessing

scripts/headers_install.sh is relatively inefficient: it parses/modifies headers via unifdef and three sed invocations, with sed output post-processed via grep in one case (user-space CONFIG_ leaks).

These patches attempt to improve simplicity and efficiency by merging two sed calls and processing user-space CONFIG_ leak filters inline.

Output is unaffected, aside from one minor error string change:
error: $INFILE: leak CONFIG_X to user-space
becomes...
error: $INFILE:CONFIG_X leak to user-space

License and CONFIG leak checks have been manually tested via error injection.

Changes since v1 RFC:
- accept $INFILE prefixes from e.g. make headers_install O=build_dir as reported by Oliver Sang / kernel test robot
- drop extra config filter variable and do it all inline in sed

David Disseldorp (2):
scripts: headers_install: filter ignored configs via sed
scripts: headers_install: avoid extra sed call for license check

 scripts/headers_install.sh | 74 ++++++++++++++++----------------------
 1 file changed, 31 insertions(+), 43 deletions(-)
#1 lore raw

PATCH v2 1/2 scripts: headers_install: filter ignored configs via sed

The sed script currently prints any CONFIG_ entries carried in installed headers. A subsequent shell script parses this output to check whether the found CONFIG_ values should be ignored or not.

Drop the unnecessary sed output post-processing and instead skip over ignored CONFIG_ values as part of initial processing.

INFILE may carry a prefix for e.g. make headers_install O=build_dir . Therefore, don't anchor the <file-name>:<CONFIG-option> match against the beginning of INFILE.

---
 scripts/headers_install.sh | 55 +++++++++++++-------------------------
 1 file changed, 19 insertions(+), 36 deletions(-)
scripts/headers_install.sh +19 -36
index 0e4e939efc940..2181abd1c9b70 100755
@@ -42,7 +42,7 @@ scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ $TMPFILE > $OUTFILE
 [ $? -gt 1 ] && exit 1
 
 # Remove /* ... */ style comments, and find CONFIG_ references in code
-configs=$(sed -e '
+sed -e '
 :comment
 	s:/\*[^*][^*]*:/*:
 	s:/\*\*\**\([^/]\):/*\1:
@@ -53,48 +53,31 @@ configs=$(sed -e '
 	N
 	b comment
 :print
+	# The entries in the following list do not result in an error.
+	# Please do not add a new entry. This list is only for existing ones.
+	# The list will be reduced gradually, and deleted eventually.
+	#
+	# The format is s@<file-name>:<CONFIG-option>\n@@ in each line.
+	s@arch/arc/include/uapi/asm/swab.h:CONFIG_ARC_HAS_SWAPE\n@@
+	s@arch/arm/include/uapi/asm/ptrace.h:CONFIG_CPU_ENDIAN_BE8\n@@
+	s@arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_NO\n@@
+	s@arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_SUPPORT\n@@
+	s@arch/x86/include/uapi/asm/auxvec.h:CONFIG_IA32_EMULATION\n@@
+	s@arch/x86/include/uapi/asm/auxvec.h:CONFIG_X86_64\n@@
+
+	# Jump if any of the above filters applied, otherwise error out.
+	t check
+	s@^\(.*\)\n.*@error: \1 leak to user-space@
 	P
-	D
+	Q2
 :check
-	s:^\(CONFIG_[[:alnum:]_]*\):\1\n:
+	s@^\(CONFIG_[[:alnum:]_]*\)@'"$INFILE"':\1\n@
 	t print
 	s:^[[:alnum:]_][[:alnum:]_]*::
 	s:^[^[:alnum:]_][^[:alnum:]_]*::
 	t check
 	d
-' $OUTFILE)
-
-# The entries in the following list do not result in an error.
-# Please do not add a new entry. This list is only for existing ones.
-# The list will be reduced gradually, and deleted eventually. (hopefully)
-#
-# The format is <file-name>:<CONFIG-option> in each line.
-config_leak_ignores="
-arch/arc/include/uapi/asm/swab.h:CONFIG_ARC_HAS_SWAPE
-arch/arm/include/uapi/asm/ptrace.h:CONFIG_CPU_ENDIAN_BE8
-arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_NO
-arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_SUPPORT
-arch/x86/include/uapi/asm/auxvec.h:CONFIG_IA32_EMULATION
-arch/x86/include/uapi/asm/auxvec.h:CONFIG_X86_64
-"
-
-for c in $configs
-do
-	leak_error=1
-
-	for ignore in $config_leak_ignores
-	do
-		if echo "$INFILE:$c" | grep -q "$ignore$"; then
-			leak_error=
-			break
-		fi
-	done
-
-	if [ "$leak_error" = 1 ]; then
-		echo "error: $INFILE: leak $c to user-space" >&2
-		exit 1
-	fi
-done
+' $OUTFILE >&2 || exit 1
 
 rm -f $TMPFILE
 trap - EXIT
Signature · 1 line
2.51.0
#2 lore raw

Re: [PATCH v2 1/2] scripts: headers_install: filter ignored configs via sed

Hi David,

Quoted 5 lines · David Disseldorp

The sed script currently prints any CONFIG_ entries carried in installed headers. A subsequent shell script parses this output to check whether the found CONFIG_ values should be ignored or not.
Drop the unnecessary sed output post-processing and instead skip over ignored CONFIG_ values as part of initial processing.

The exlusion logic is going to go away completely soon:
https://lore.kernel.org/lkml/20260112-headers_install-config-leak-v2-0-4d8084444603@linutronix.de/
(My series is probably going through the asm-generic tree)

Our patches will conflict.

Thomas

#3 lore raw

Re: [PATCH v2 1/2] scripts: headers_install: filter ignored configs via sed

Hi Thomas,

Quoted 9 lines · Thomas Weißschuh

Hi David,

Quoted 5 lines · David Disseldorp

The sed script currently prints any CONFIG_ entries carried in installed headers. A subsequent shell script parses this output to check whether the found CONFIG_ values should be ignored or not.
Drop the unnecessary sed output post-processing and instead skip over ignored CONFIG_ values as part of initial processing.

Quoted 5 lines

The exlusion logic is going to go away completely soon:
https://lore.kernel.org/lkml/20260112-headers_install-config-leak-v2-0-4d8084444603@linutronix.de/
(My series is probably going through the asm-generic tree)

Our patches will conflict.

Looks good, I'm happy to drop this patch in favour of your removals. Patch v2 2/2 should still be relevant and apply relatively cleanly.

Thanks, David

#4 lore raw

PATCH v2 2/2 scripts: headers_install: avoid extra sed call for license check

In reply to David Disseldorp

headers_install runs a sed script to check that any GPL variant SPDX-License-Identifier line carries a "WITH Linux-syscall-note". A subsequent sed invocation then handles removal of a few things that aren't desired in installed headers (e.g. __attribute_const__).

Combine these two sed scripts to avoid re-processing the same file. License check errors, as opposed to write errors, are indicated via a special sed exit status of 9.

---
 scripts/headers_install.sh | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
scripts/headers_install.sh +12 -7
index 2181abd1c9b70..ef952cbbb8fde 100755
@@ -23,20 +23,25 @@ TMPFILE=$OUTFILE.tmp
 
 trap 'rm -f $OUTFILE $TMPFILE' EXIT
 
-# SPDX-License-Identifier with GPL variants must have "WITH Linux-syscall-note"
-if [ -n "$(sed -n -e "/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/!p}" $INFILE)" ]; then
-	echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2
-	exit 1
-fi
-
+# returns 9 if GPL SPDX-License-Identifier omits "WITH Linux-syscall-note"
 sed -E -e '
+	/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/! Q9}
 	s/([[:space:](])(__user|__force|__iomem)[[:space:]]/\1/g
 	s/__attribute_const__([[:space:]]|$)/\1/g
 	s@^#include <linux/compiler.h>@@
 	s/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g
 	s/(^|[[:space:](])(inline|asm|volatile)([[:space:](]|$)/\1__\2__\3/g
 	s@#(ifndef|define|endif[[:space:]]*/[*])[[:space:]]*_UAPI@#\1 @
-' $INFILE > $TMPFILE || exit 1
+' $INFILE > $TMPFILE
+case $? in
+9)
+	echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2
+	exit 1
+	;;
+1)
+	exit 1
+	;;
+esac
 
 scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ $TMPFILE > $OUTFILE
 [ $? -gt 1 ] && exit 1
Signature · 1 line
2.51.0
maintainer
#5 lore raw

Re: [PATCH v2 2/2] scripts: headers_install: avoid extra sed call for license check

[Cc+=nathan]

Quoted 21 lines · David Disseldorp

headers_install runs a sed script to check that any GPL variant SPDX-License-Identifier line carries a "WITH Linux-syscall-note". A subsequent sed invocation then handles removal of a few things that aren't desired in installed headers (e.g. __attribute_const__).

Combine these two sed scripts to avoid re-processing the same file. License check errors, as opposed to write errors, are indicated via a special sed exit status of 9.

---
 scripts/headers_install.sh | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
scripts/headers_install.sh +0 -0
index 2181abd1c9b70..ef952cbbb8fde 100755
@@ -23,20 +23,25 @@ TMPFILE=$OUTFILE.tmp
 
 trap 'rm -f $OUTFILE $TMPFILE' EXIT

scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 28

Quoted 10 lines
scripts/headers_install.sh +2 -6
 
-# SPDX-License-Identifier with GPL variants must have "WITH Linux-syscall-note"
-if [ -n "$(sed -n -e "/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/!p}" $INFILE)" ]; then
-	echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2
-	exit 1
-fi
-
+# returns 9 if GPL SPDX-License-Identifier omits "WITH Linux-syscall-note"
 sed -E -e '
+	/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/! Q9}

thanks, I like the idea of combining the sed calls. According to the manual 'Q' (or 'q' with argument) is a GNU extension (non-POSIX).

May we expect GNU sed on all system running headers_install?

Quoted 7 lines
scripts/headers_install.sh +0 -1
 	s/([[:space:](])(__user|__force|__iomem)[[:space:]]/\1/g
 	s/__attribute_const__([[:space:]]|$)/\1/g
 	s@^#include <linux/compiler.h>@@
 	s/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g
 	s/(^|[[:space:](])(inline|asm|volatile)([[:space:](]|$)/\1__\2__\3/g
 	s@#(ifndef|define|endif[[:space:]]*/[*])[[:space:]]*_UAPI@#\1 @
-' $INFILE > $TMPFILE || exit 1

scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 44

Quoted 10 lines
scripts/headers_install.sh +10 -0
+' $INFILE > $TMPFILE
+case $? in
+9)
+	echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2
+	exit 1
+	;;
+1)
+	exit 1
+	;;
+esac

This silently ignores all other exit codes. What about this:

9)
...

	;;
0)	;;
*)
	exit 1
	;;

esac

Kind regards,
Nicolas

#6 lore raw

Re: [PATCH v2 2/2] scripts: headers_install: avoid extra sed call for license check

Thanks for the review Nicolas...

Quoted 38 lines · Nicolas Schier

[Cc+=nathan]

Quoted 21 lines · David Disseldorp

headers_install runs a sed script to check that any GPL variant SPDX-License-Identifier line carries a "WITH Linux-syscall-note". A subsequent sed invocation then handles removal of a few things that aren't desired in installed headers (e.g. __attribute_const__).

Combine these two sed scripts to avoid re-processing the same file. License check errors, as opposed to write errors, are indicated via a special sed exit status of 9.

---
 scripts/headers_install.sh | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
scripts/headers_install.sh +0 -0
index 2181abd1c9b70..ef952cbbb8fde 100755
@@ -23,20 +23,25 @@ TMPFILE=$OUTFILE.tmp
 
 trap 'rm -f $OUTFILE $TMPFILE' EXIT

scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 28

Quoted 10 lines
scripts/headers_install.sh +2 -6
 
-# SPDX-License-Identifier with GPL variants must have "WITH Linux-syscall-note"
-if [ -n "$(sed -n -e "/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/!p}" $INFILE)" ]; then
-	echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2
-	exit 1
-fi
-
+# returns 9 if GPL SPDX-License-Identifier omits "WITH Linux-syscall-note"
 sed -E -e '
+	/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/! Q9}  

thanks, I like the idea of combining the sed calls. According to the manual 'Q' (or 'q' with argument) is a GNU extension (non-POSIX).

Quoted 1 line

May we expect GNU sed on all system running headers_install?

Good catch.
I suspect there might already be some GNU sed specific code, based on LKL's explicit use of gsed for builds on BSD hosts:
https://github.com/lkl/linux/commit/84f39699a6d1013d1866b8637977804e515d8d7e
I don't have any proof for mainline though (yet).

Quoted 20 lines
Quoted 7 lines
scripts/headers_install.sh +0 -1
 	s/([[:space:](])(__user|__force|__iomem)[[:space:]]/\1/g
 	s/__attribute_const__([[:space:]]|$)/\1/g
 	s@^#include <linux/compiler.h>@@
 	s/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g
 	s/(^|[[:space:](])(inline|asm|volatile)([[:space:](]|$)/\1__\2__\3/g
 	s@#(ifndef|define|endif[[:space:]]*/[*])[[:space:]]*_UAPI@#\1 @
-' $INFILE > $TMPFILE || exit 1

scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 44

Quoted 10 lines
scripts/headers_install.sh +10 -0
+' $INFILE > $TMPFILE
+case $? in
+9)
+	echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2
+	exit 1
+	;;
+1)
+	exit 1
+	;;
+esac  

This silently ignores all other exit codes. What about this:

Quoted 8 lines

9)
...

	;;
0)	;;
*)
	exit 1
	;;

esac

Ack, I see that 2 and 4 returns are also possible. Your fix looks good to me.
I'll squash it in and resend if I find any other GNU sed specific code. Otherwise this change can probably be dropped.

Cheers, David

#7 lore raw
In reply to David Disseldorp

Here are the perf (perf stat -r 10) numbers I see with / without the v2 patches applied atop v6.19-rc4:

  $ cat bench.sh 
  touch scripts/headers_install.sh
  make O=/tmp/build_dir headers_install
  $ cat bench_j8.sh 
  touch scripts/headers_install.sh
  make -j8 O=/tmp/build_dir headers_install

| bench.sh | bench_j8.sh
--------+--------------------+---------------------
before | 41.176 (+-0.28%) | 2.58672 (+-0.28%)
--------+--------------------+---------------------
after | 34.811 (+-1.35%) | 2.33096 (+-0.39%)
--------+--------------------+---------------------