scripts: headers_install: avoid some reprocessing
v2 cover scripts: headers_install: avoid some reprocessing
scripts/headers_install.sh is relatively inefficient: it parses/modifies headers via unifdef and three sed invocations, with sed output post-processed via grep in one case (user-space CONFIG_ leaks).
These patches attempt to improve simplicity and efficiency by merging two sed calls and processing user-space CONFIG_ leak filters inline.
Output is unaffected, aside from one minor error string change:
error: $INFILE: leak CONFIG_X to user-space
becomes...
error: $INFILE:CONFIG_X leak to user-space
License and CONFIG leak checks have been manually tested via error injection.
Changes since v1 RFC:
- accept $INFILE prefixes from e.g. make headers_install O=build_dir as reported by Oliver Sang / kernel test robot
- drop extra config filter variable and do it all inline in sed
David Disseldorp (2):
scripts: headers_install: filter ignored configs via sed
scripts: headers_install: avoid extra sed call for license check
scripts/headers_install.sh | 74 ++++++++++++++++---------------------- 1 file changed, 31 insertions(+), 43 deletions(-)
PATCH v2 1/2 scripts: headers_install: filter ignored configs via sed
The sed script currently prints any CONFIG_ entries carried in installed headers. A subsequent shell script parses this output to check whether the found CONFIG_ values should be ignored or not.
Drop the unnecessary sed output post-processing and instead skip over ignored CONFIG_ values as part of initial processing.
INFILE may carry a prefix for e.g. make headers_install O=build_dir . Therefore, don't anchor the <file-name>:<CONFIG-option> match against the beginning of INFILE.
--- scripts/headers_install.sh | 55 +++++++++++++------------------------- 1 file changed, 19 insertions(+), 36 deletions(-)
scripts/headers_install.sh +19 -36
@@ -42,7 +42,7 @@ scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ $TMPFILE > $OUTFILE [ $? -gt 1 ] && exit 1 # Remove /* ... */ style comments, and find CONFIG_ references in code -configs=$(sed -e ' +sed -e ' :comment s:/\*[^*][^*]*:/*: s:/\*\*\**\([^/]\):/*\1:
@@ -53,48 +53,31 @@ configs=$(sed -e ' N b comment :print + # The entries in the following list do not result in an error. + # Please do not add a new entry. This list is only for existing ones. + # The list will be reduced gradually, and deleted eventually. + # + # The format is s@<file-name>:<CONFIG-option>\n@@ in each line. + s@arch/arc/include/uapi/asm/swab.h:CONFIG_ARC_HAS_SWAPE\n@@ + s@arch/arm/include/uapi/asm/ptrace.h:CONFIG_CPU_ENDIAN_BE8\n@@ + s@arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_NO\n@@ + s@arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_SUPPORT\n@@ + s@arch/x86/include/uapi/asm/auxvec.h:CONFIG_IA32_EMULATION\n@@ + s@arch/x86/include/uapi/asm/auxvec.h:CONFIG_X86_64\n@@ + + # Jump if any of the above filters applied, otherwise error out. + t check + s@^\(.*\)\n.*@error: \1 leak to user-space@ P - D + Q2 :check - s:^\(CONFIG_[[:alnum:]_]*\):\1\n: + s@^\(CONFIG_[[:alnum:]_]*\)@'"$INFILE"':\1\n@ t print s:^[[:alnum:]_][[:alnum:]_]*:: s:^[^[:alnum:]_][^[:alnum:]_]*:: t check d -' $OUTFILE) - -# The entries in the following list do not result in an error. -# Please do not add a new entry. This list is only for existing ones. -# The list will be reduced gradually, and deleted eventually. (hopefully) -# -# The format is <file-name>:<CONFIG-option> in each line. -config_leak_ignores=" -arch/arc/include/uapi/asm/swab.h:CONFIG_ARC_HAS_SWAPE -arch/arm/include/uapi/asm/ptrace.h:CONFIG_CPU_ENDIAN_BE8 -arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_NO -arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_SUPPORT -arch/x86/include/uapi/asm/auxvec.h:CONFIG_IA32_EMULATION -arch/x86/include/uapi/asm/auxvec.h:CONFIG_X86_64 -" - -for c in $configs -do - leak_error=1 - - for ignore in $config_leak_ignores - do - if echo "$INFILE:$c" | grep -q "$ignore$"; then - leak_error= - break - fi - done - - if [ "$leak_error" = 1 ]; then - echo "error: $INFILE: leak $c to user-space" >&2 - exit 1 - fi -done +' $OUTFILE >&2 || exit 1 rm -f $TMPFILE trap - EXIT
Signature · 1 line
2.51.0
Re: [PATCH v2 1/2] scripts: headers_install: filter ignored configs via sed
Hi David,
Quoted 5 lines · David Disseldorp
The sed script currently prints any
CONFIG_entries carried in installed headers. A subsequent shell script parses this output to check whether the foundCONFIG_values should be ignored or not.
Drop the unnecessary sed output post-processing and instead skip over ignoredCONFIG_values as part of initial processing.
The exlusion logic is going to go away completely soon:
https://lore.kernel.org/lkml/20260112-headers_install-config-leak-v2-0-4d8084444603@linutronix.de/
(My series is probably going through the asm-generic tree)
Our patches will conflict.
Thomas
Re: [PATCH v2 1/2] scripts: headers_install: filter ignored configs via sed
Hi Thomas,
Quoted 9 lines · Thomas Weißschuh
Hi David,
Quoted 5 lines · David Disseldorp
The sed script currently prints any
CONFIG_entries carried in installed headers. A subsequent shell script parses this output to check whether the foundCONFIG_values should be ignored or not.
Drop the unnecessary sed output post-processing and instead skip over ignoredCONFIG_values as part of initial processing.
Quoted 5 lines
The exlusion logic is going to go away completely soon:
https://lore.kernel.org/lkml/20260112-headers_install-config-leak-v2-0-4d8084444603@linutronix.de/
(My series is probably going through the asm-generic tree)Our patches will conflict.
Looks good, I'm happy to drop this patch in favour of your removals. Patch v2 2/2 should still be relevant and apply relatively cleanly.
Thanks, David
PATCH v2 2/2 scripts: headers_install: avoid extra sed call for license check
In reply to David Disseldorpheaders_install runs a sed script to check that any GPL variant SPDX-License-Identifier line carries a "WITH Linux-syscall-note". A subsequent sed invocation then handles removal of a few things that aren't desired in installed headers (e.g. __attribute_const__).
Combine these two sed scripts to avoid re-processing the same file. License check errors, as opposed to write errors, are indicated via a special sed exit status of 9.
--- scripts/headers_install.sh | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
scripts/headers_install.sh +12 -7
@@ -23,20 +23,25 @@ TMPFILE=$OUTFILE.tmp trap 'rm -f $OUTFILE $TMPFILE' EXIT -# SPDX-License-Identifier with GPL variants must have "WITH Linux-syscall-note" -if [ -n "$(sed -n -e "/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/!p}" $INFILE)" ]; then - echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2 - exit 1 -fi - +# returns 9 if GPL SPDX-License-Identifier omits "WITH Linux-syscall-note" sed -E -e ' + /SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/! Q9} s/([[:space:](])(__user|__force|__iomem)[[:space:]]/\1/g s/__attribute_const__([[:space:]]|$)/\1/g s@^#include <linux/compiler.h>@@ s/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g s/(^|[[:space:](])(inline|asm|volatile)([[:space:](]|$)/\1__\2__\3/g s@#(ifndef|define|endif[[:space:]]*/[*])[[:space:]]*_UAPI@#\1 @ -' $INFILE > $TMPFILE || exit 1 +' $INFILE > $TMPFILE +case $? in +9) + echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2 + exit 1 + ;; +1) + exit 1 + ;; +esac scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__ $TMPFILE > $OUTFILE [ $? -gt 1 ] && exit 1
Signature · 1 line
2.51.0
Nicolas Schier
maintainerRe: [PATCH v2 2/2] scripts: headers_install: avoid extra sed call for license check
[Cc+=nathan]
Quoted 21 lines · David Disseldorp
headers_installruns a sed script to check that any GPL variant SPDX-License-Identifier line carries a "WITH Linux-syscall-note". A subsequent sed invocation then handles removal of a few things that aren't desired in installed headers (e.g.__attribute_const__).Combine these two sed scripts to avoid re-processing the same file. License check errors, as opposed to write errors, are indicated via a special sed exit status of 9.
Signed-off-by David Disseldorp <[email protected]>--- scripts/headers_install.sh | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)scripts/headers_install.sh +0
-0@@ -23,20 +23,25 @@ TMPFILE=$OUTFILE.tmp trap 'rm -f $OUTFILE $TMPFILE' EXIT
scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 28
Quoted 10 lines
scripts/headers_install.sh +2
-6-# SPDX-License-Identifier with GPL variants must have "WITH Linux-syscall-note" -if [ -n "$(sed -n -e "/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/!p}" $INFILE)" ]; then - echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2 - exit 1 -fi - +# returns 9 if GPL SPDX-License-Identifier omits "WITH Linux-syscall-note" sed -E -e ' + /SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/! Q9}
thanks, I like the idea of combining the sed calls. According to the manual 'Q' (or 'q' with argument) is a GNU extension (non-POSIX).
May we expect GNU sed on all system running headers_install?
Quoted 7 lines
scripts/headers_install.sh +0
-1s/([[:space:](])(__user|__force|__iomem)[[:space:]]/\1/g s/__attribute_const__([[:space:]]|$)/\1/g s@^#include <linux/compiler.h>@@ s/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g s/(^|[[:space:](])(inline|asm|volatile)([[:space:](]|$)/\1__\2__\3/g s@#(ifndef|define|endif[[:space:]]*/[*])[[:space:]]*_UAPI@#\1 @ -' $INFILE > $TMPFILE || exit 1
scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 44
Quoted 10 lines
scripts/headers_install.sh +10
-0+' $INFILE > $TMPFILE +case $? in +9) + echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2 + exit 1 + ;; +1) + exit 1 + ;; +esac
This silently ignores all other exit codes. What about this:
9)
...
;; 0) ;; *) exit 1 ;;
esac
Kind regards,
Nicolas
Re: [PATCH v2 2/2] scripts: headers_install: avoid extra sed call for license check
Thanks for the review Nicolas...
Quoted 38 lines · Nicolas Schier
[Cc+=nathan]
Quoted 21 lines · David Disseldorp
headers_installruns a sed script to check that any GPL variant SPDX-License-Identifier line carries a "WITH Linux-syscall-note". A subsequent sed invocation then handles removal of a few things that aren't desired in installed headers (e.g.__attribute_const__).Combine these two sed scripts to avoid re-processing the same file. License check errors, as opposed to write errors, are indicated via a special sed exit status of 9.
Signed-off-by David Disseldorp <[email protected]>--- scripts/headers_install.sh | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)scripts/headers_install.sh +0
-0@@ -23,20 +23,25 @@ TMPFILE=$OUTFILE.tmp trap 'rm -f $OUTFILE $TMPFILE' EXITscripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 28
Quoted 10 lines
scripts/headers_install.sh +2
-6-# SPDX-License-Identifier with GPL variants must have "WITH Linux-syscall-note" -if [ -n "$(sed -n -e "/SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/!p}" $INFILE)" ]; then - echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2 - exit 1 -fi - +# returns 9 if GPL SPDX-License-Identifier omits "WITH Linux-syscall-note" sed -E -e ' + /SPDX-License-Identifier:.*GPL-/{/WITH Linux-syscall-note/! Q9}thanks, I like the idea of combining the sed calls. According to the manual 'Q' (or 'q' with argument) is a GNU extension (non-POSIX).
Quoted 1 line
May we expect GNU sed on all system running
headers_install?
Good catch.
I suspect there might already be some GNU sed specific code, based on LKL's explicit use of gsed for builds on BSD hosts:
https://github.com/lkl/linux/commit/84f39699a6d1013d1866b8637977804e515d8d7e
I don't have any proof for mainline though (yet).
Quoted 20 lines
Quoted 7 lines
scripts/headers_install.sh +0
-1s/([[:space:](])(__user|__force|__iomem)[[:space:]]/\1/g s/__attribute_const__([[:space:]]|$)/\1/g s@^#include <linux/compiler.h>@@ s/(^|[^a-zA-Z0-9])__packed([^a-zA-Z0-9_]|$)/\1__attribute__((packed))\2/g s/(^|[[:space:](])(inline|asm|volatile)([[:space:](]|$)/\1__\2__\3/g s@#(ifndef|define|endif[[:space:]]*/[*])[[:space:]]*_UAPI@#\1 @ -' $INFILE > $TMPFILE || exit 1scripts/headers_install.sh TMPFILE=$OUTFILE.tmp · line 44
Quoted 10 lines
scripts/headers_install.sh +10
-0+' $INFILE > $TMPFILE +case $? in +9) + echo "error: $INFILE: missing \"WITH Linux-syscall-note\" for SPDX-License-Identifier" >&2 + exit 1 + ;; +1) + exit 1 + ;; +esacThis silently ignores all other exit codes. What about this:
Quoted 8 lines
9)
...;; 0) ;; *) exit 1 ;;esac
Ack, I see that 2 and 4 returns are also possible. Your fix looks good to me.
I'll squash it in and resend if I find any other GNU sed specific code. Otherwise this change can probably be dropped.
Cheers, David
Here are the perf (perf stat -r 10) numbers I see with / without the v2 patches applied atop v6.19-rc4:
$ cat bench.sh touch scripts/headers_install.sh make O=/tmp/build_dir headers_install
$ cat bench_j8.sh touch scripts/headers_install.sh make -j8 O=/tmp/build_dir headers_install
| bench.sh | bench_j8.sh
--------+--------------------+---------------------
before | 41.176 (+-0.28%) | 2.58672 (+-0.28%)
--------+--------------------+---------------------
after | 34.811 (+-1.35%) | 2.33096 (+-0.39%)
--------+--------------------+---------------------