lib/string: shrink lib/string.i via IWYU
Subsystems (5)
PATCH 1/1 lib/string: shrink lib/string.i via IWYU
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to lib/string.c resulted in a preprocessed size of lib/string.i from 26371 lines to 5232 lines (-80%).
If there are any problems with the output of the tool please raise an issue on the github.
---
--- lib/string.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
lib/string.c +7 -7
@@ -16,16 +16,16 @@#define __NO_FORTIFY#include <linux/types.h> +#include <asm/bitsperlong.h> +#include <asm/bug.h> +#include <asm/errno.h> +#include <asm/rwonce.h> +#include <linux/linkage.h> +#include <linux/stddef.h> +#include <vdso/limits.h> #include <linux/string.h> #include <linux/ctype.h> -#include <linux/kernel.h> -#include <linux/export.h> -#include <linux/bug.h> -#include <linux/errno.h> -#include <linux/slab.h> -#include <asm/unaligned.h> -#include <asm/byteorder.h> #include <asm/word-at-a-time.h> #include <asm/page.h>
---
base-commit: 33cc938e65a98f1d29d0a18403dbbee050dcad9a
change-id: 20231204-libstringheader-e238e2af5eec
Best regards,
Signature · 1 line
Tanzir Hasan <[email protected]>
Andrew Morton
maintainerQuoted 5 lines · On Tue, 05 Dec 2023 20:58:53 +0000 [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
Quoted 7 lines
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).If there are any problems with the output of the tool please raise an issue on the github.
Issue:
lib/string.c line 19
Quoted 7 lines
lib/string.c +1
-0@@ -16,16 +16,16 @@#define __NO_FORTIFY#include <linux/types.h> +#include <asm/bitsperlong.h>
The preferred way to import bit-fiddling stuff is to include <linux/bits.h>. Under the hood this may include asm/bitsperlong.h. Or it may not, depending on Kconfig settings (particularly architecture).
Quoted 18 lines · Andrew Morton
Quoted 5 lines · On Tue, 05 Dec 2023 20:58:53 +0000 [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.Quoted 7 lines
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).If there are any problems with the output of the tool please raise an issue on the github.
Issue:
Quoted 7 lines
lib/string.c line 19
Quoted 7 lines
lib/string.c +1
-0@@ -16,16 +16,16 @@#define __NO_FORTIFY#include <linux/types.h> +#include <asm/bitsperlong.h>
Thanks for the quick feedback. For additional context, Tanzir is my intern working on getting IWYU working on the kernel. Welcome Tanzir to LKML!
Quoted 5 lines
The preferred way to import bit-fiddling stuff is to include <
linux/bits.h>. Under the hood this may includeasm/bitsperlong.h. Or it may not, depending onKconfigsettings (particularly architecture).
Just triple checking my understanding; it looks like include/linux/bits.h unconditionally includes asm/bitsperlong.h (which is implemented per arch) most of which seem to include asm-generic/bitsperlong.h.
include/linux/bits.h also defines a few macros (BIT_MASK, BIT_WORD, BITS_PER_BYTE, GENMASK, etc). If lib/string.c is not using any of those, why can't we go straight to #including asm/bitsperlong.h? That should resolve to the arch specific impl which may include asm-generic/bitsperlong.h?
Signature · 2 lines
Thanks, ~Nick Desaulniers
Andrew Morton
maintainerQuoted 11 lines · On Tue, 5 Dec 2023 13:14:16 -0800 Nick Desaulniers <[email protected]>…
Quoted 5 lines
The preferred way to import bit-fiddling stuff is to include <
linux/bits.h>. Under the hood this may includeasm/bitsperlong.h. Or it may not, depending onKconfigsettings (particularly architecture).Just triple checking my understanding; it looks like
include/linux/bits.hunconditionally includesasm/bitsperlong.h(which is implemented per arch) most of which seem to includeasm-generic/bitsperlong.h.
Quoted 5 lines
include/linux/bits.halso defines a few macros (BIT_MASK,BIT_WORD,BITS_PER_BYTE, GENMASK, etc). Iflib/string.cis not using any of those, why can't we go straight to #includingasm/bitsperlong.h? That should resolve to the arch specific impl which may includeasm-generic/bitsperlong.h?
It's just a general rule. If the higher-level include is present, use that. Because of the above, plus I guess things might change in the future.
We've been getting better about irregular asm/include files.
But bits.h is a poor example. A better case to study is spinlock.h. If this tool recommended including asm/spinlock.h then that won't work on any architecture which doesn't implement SMP (there is no arch/nios2/include/asm/spinlock.h).
Quoted 2 lines · On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
How does it account for arch- and config-dependent indirect includes?
In particular, on sh this patch breaks, since there word-at-a-time.h does not contain an include of kernel.h, even though it uses REPEAT_BYTE. This is a very simple case (they really ought to include kernel.h, same as all other instances of word-at-a-time.h), but I would expect arseloads of more subtle breakage in anything less trivial.
And I'm not at all sure that there's no config-dependent breakage as well - this had been caught by quick make allmodconfig + make lib/string.o on a bunch of architectures; the graph of indirects includes (as well as the set of symbols needed for given header) is very much config-dependent.
Quoted 5 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <asm/uaccess.h> *anywhere* outside of linux/uaccess.h is a bad idea.
Quoted 2 lines
If there are any problems with the output of the tool please raise an issue on the github.
Sorry, no. Your tool, your workflow, of course, but I don't do github issues. You are welcome to the contents of this reply, but I'm not using browser-based UI without very strong reasons; this one isn't.
Quoted 20 lines · Andrew Morton
Quoted 11 lines · On Tue, 5 Dec 2023 13:14:16 -0800 Nick Desaulniers <[email protected]>…
Quoted 5 lines
The preferred way to import bit-fiddling stuff is to include <
linux/bits.h>. Under the hood this may includeasm/bitsperlong.h. Or it may not, depending onKconfigsettings (particularly architecture).Just triple checking my understanding; it looks like
include/linux/bits.hunconditionally includesasm/bitsperlong.h(which is implemented per arch) most of which seem to includeasm-generic/bitsperlong.h.Quoted 5 lines
include/linux/bits.halso defines a few macros (BIT_MASK,BIT_WORD,BITS_PER_BYTE, GENMASK, etc). Iflib/string.cis not using any of those, why can't we go straight to #includingasm/bitsperlong.h? That should resolve to the arch specific impl which may includeasm-generic/bitsperlong.h?
Quoted 3 lines
It's just a general rule. If the higher-level include is present, use that. Because of the above, plus I guess things might change in the future.
Hmm...how does one know that linux/bits.h is the higher-level include of asm/bitsperlong.h?
Do we mention these conventions anywhere under Documentation?
Quoted 7 lines
We've been getting better about irregular asm/include files.
But
bits.his a poor example. A better case to study isspinlock.h. If this tool recommended includingasm/spinlock.hthen that won't work on any architecture which doesn't implement SMP (there is noarch/nios2/include/asm/spinlock.h).
The tooling Tanzir is working on does wrap IWYU, and does support such mapping (of 'low level' to 'high level' headers; more so, if it recommends X you can override to suggest Y instead).
arch/nios/ also doesn't provide a bug.h, which this patch is suggesting we include directly. I guess the same goes for asm/rwonce.h.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 7 lines · Nick Desaulniers
The tooling Tanzir is working on does wrap IWYU, and does support such mapping (of 'low level' to 'high level' headers; more so, if it recommends X you can override to suggest Y instead).
arch/nios/also doesn't provide abug.h, which this patch is suggesting we include directly. I guess the same goes forasm/rwonce.h.
See include/asm-generic/Kbuild:
mandatory-y += bug.h
...
mandatory-y += rwonce.h
IOW, sh will have asm/bug.h and as/rwonce.h copied from asm-generic.
Still, includes of asm/*.h had been a massive headache historically and breeding more of those shouldn't be overdone.
More painful problem is arch- and config-dependent stuff, though...
Andy Shevchenko
reviewerQuoted 8 lines · On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).
Quoted 6 lines
If there are any problems with the output of the tool please raise an issue on the github.
Signed-off-by Tanzir Hasan <[email protected]>
Blank lines are not allowed in tag block. Unless Link: is not supposed to be a tag (which looks quite strange in that case).
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 13 lines · Al Viro
Quoted 2 lines · On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
How does it account for arch- and config-dependent indirect includes?
In particular, on sh this patch breaks, since there
word-at-a-time.hdoes not contain an include ofkernel.h, even though it usesREPEAT_BYTE. This is a very simple case (they really ought to includekernel.h, same as all other instances ofword-at-a-time.h), but I would expect arseloads of more subtle breakage in anything less trivial.
Quoted 4 lines
And I'm not at all sure that there's no config-dependent breakage as well - this had been caught by quick make allmodconfig + make
lib/string.oon a bunch of architectures; the graph of indirects includes (as well as the set of symbols needed for given header) is very much config-dependent.
We're sending these to Kees to stage in branch flowing into linux-next in order for the patches to get soak time in linux-next; it's not possible to test every possible randconfig, but with enough soak time and the bots chewing on linux-next, I think we can get to a certain level of confidence.
We'll ramp up the amount of testing we're doing locally as well. (I did quite a few randconfigs locally in a loop; didn't test all architectures) We can probably fetch the kernel.org toolchains for very extensive testing.
https://mirrors.edge.kernel.org/pub/tools/crosstool/
Quoted 10 lines
Quoted 6 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.
It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
Quoted 7 lines
Quoted 3 lines
If there are any problems with the output of the tool please raise an issue on the github.
Sorry, no. Your tool, your workflow, of course, but I don't do github issues. You are welcome to the contents of this reply, but I'm not using browser-based UI without very strong reasons; this one isn't.
No problem; Tanzir, please drop that part of the commit message in future patches.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Andy Shevchenko
reviewerQuoted 17 lines · Nick Desaulniers
Quoted 16 lines · Andrew Morton
Quoted 14 lines · On Tue, 5 Dec 2023 13:14:16 -0800 Nick Desaulniers <[email protected]>…
Quoted 3 lines
The preferred way to import bit-fiddling stuff is to include <
linux/bits.h>. Under the hood this may includeasm/bitsperlong.h. Or it may not, depending onKconfigsettings (particularly architecture).Just triple checking my understanding; it looks like
include/linux/bits.hunconditionally includesasm/bitsperlong.h(which is implemented per arch) most of which seem to includeasm-generic/bitsperlong.h.
include/linux/bits.halso defines a few macros (BIT_MASK,BIT_WORD,BITS_PER_BYTE, GENMASK, etc). Iflib/string.cis not using any of those, why can't we go straight to #includingasm/bitsperlong.h? That should resolve to the arch specific impl which may includeasm-generic/bitsperlong.h?
Quoted 8 lines
Quoted 3 lines
It's just a general rule. If the higher-level include is present, use that. Because of the above, plus I guess things might change in the future.
Hmm...how does one know that
linux/bits.his the higher-level include ofasm/bitsperlong.h?Do we mention these conventions anywhere under Documentation?
Unfortunately it comes with an experience. The problem here is the absence of Documentation for the headers that guarantee inclusion of other headers. In general linux/* are preferred over asm/* but sometimes things are complicated (when some asm ones include linux ones via architectural code and circling around).
Quoted 7 lines
Quoted 6 lines
We've been getting better about irregular asm/include files.
But
bits.his a poor example. A better case to study isspinlock.h. If this tool recommended includingasm/spinlock.hthen that won't work on any architecture which doesn't implement SMP (there is noarch/nios2/include/asm/spinlock.h).
Quoted 7 lines
The tooling Tanzir is working on does wrap IWYU, and does support such mapping (of 'low level' to 'high level' headers; more so, if it recommends X you can override to suggest Y instead).
arch/nios/also doesn't provide abug.h, which this patch is suggesting we include directly. I guess the same goes forasm/rwonce.h.
Have you checked Ingo Molnar's gigantic series (2k+ patches) for the header hell clean up? Perhaps we need to apply that first.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 3 lines · Al Viro
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.
Let me elaborate a bit:
Consider e.g. a situation when we have linux/foo.h pulling asm/foo.h; among the things declared there is foo_bar(), which is identical in 18 instances of asm/foo.h (out of current 21).
At some point we notice that; the obvious approach would be to have 3 unusual architectures have their asm/foo.h define _ARCH_HAS_ODD_FOO_BAR and put the common variant in linux/foo.h under ifndef _ARCH_HAS_ODD_FOO_BAR.
Except that any file that pulls asm/foo.h will get broken by that. And something like your tool might convert the users of linux/foo.h to asm/foo.h - all it takes is using only the primitives that currently happen to be in asm/foo.h on all architectures.
This is not an artifical example, BTW - look at the history of uaccess.h and checksum.h...
Quoted 11 lines · Al Viro
Quoted 7 lines · Nick Desaulniers
The tooling Tanzir is working on does wrap IWYU, and does support such mapping (of 'low level' to 'high level' headers; more so, if it recommends X you can override to suggest Y instead).
arch/nios/also doesn't provide abug.h, which this patch is suggesting we include directly. I guess the same goes forasm/rwonce.h.
Quoted 6 lines
See
include/asm-generic/Kbuild:
mandatory-y +=bug.h
...
mandatory-y +=rwonce.hIOW, sh will have
asm/bug.handas/rwonce.hcopied from asm-generic.
Ah, right, Ard had mentioned this file to me once. And mandatory-y is documented in Documentation/kbuild/makefiles.rst.
Ok, so we can use include/asm-generic/Kbuild to know that if the .h files listed there are recommended to include as asm/*.h, then we should replace that recommendation with linux/*.h. Our tooling has something where we can automate that.
Quoted 5 lines
Still, includes of asm/*.h had been a massive headache historically and breeding more of those shouldn't be overdone.
More painful problem is arch- and config-dependent stuff, though...
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 38 lines · Nick Desaulniers
Quoted 13 lines · Al Viro
Quoted 2 lines · On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
How does it account for arch- and config-dependent indirect includes?
In particular, on sh this patch breaks, since there
word-at-a-time.hdoes not contain an include ofkernel.h, even though it usesREPEAT_BYTE. This is a very simple case (they really ought to includekernel.h, same as all other instances ofword-at-a-time.h), but I would expect arseloads of more subtle breakage in anything less trivial.Quoted 4 lines
And I'm not at all sure that there's no config-dependent breakage as well - this had been caught by quick make allmodconfig + make
lib/string.oon a bunch of architectures; the graph of indirects includes (as well as the set of symbols needed for given header) is very much config-dependent.We're sending these to Kees to stage in branch flowing into linux-next in order for the patches to get soak time in linux-next; it's not possible to test every possible randconfig, but with enough soak time and the bots chewing on linux-next, I think we can get to a certain level of confidence.
We'll ramp up the amount of testing we're doing locally as well. (I did quite a few randconfigs locally in a loop; didn't test all architectures) We can probably fetch the kernel.org toolchains for very extensive testing.
https://mirrors.edge.kernel.org/pub/tools/crosstool/Quoted 7 lines
Quoted 6 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).
Quoted 6 lines
Quoted 3 lines
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
General rule, NEVER include asm/*.h, there should be a include/*.h instead that works. So much so that checkpatch.pl should catch this, right?
But of course, it doesn't always hold true, there are a few minor exceptions, but they are rare.
thanks,
greg k-h
Andy Shevchenko
reviewerQuoted 2 lines · Nick Desaulniers
Quoted 1 line · Al Viro
On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).
Quoted 6 lines
Quoted 3 lines
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
The mandatory ones can be used, but not all of them. In some cases you even must include asm and not linux (unaligned.h, byteorder.h, maybe others...).
As I told, it comes with experience, we lack of the respective documentation (or file which is good for automation checks, like with IWYU).
Signature · 2 lines
With Best Regards, Andy Shevchenko
Andy Shevchenko
reviewer+Cc: Jonathan, who tried IWYU with kernel a year or so ago.
Quoted 48 lines · On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).If there are any problems with the output of the tool please raise an issue on the github.
Signed-off-by Tanzir Hasan <[email protected]>---
--- lib/string.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)lib/string.c +7
-7@@ -16,16 +16,16 @@#define __NO_FORTIFY#include <linux/types.h> +#include <asm/bitsperlong.h> +#include <asm/bug.h> +#include <asm/errno.h> +#include <asm/rwonce.h> +#include <linux/linkage.h> +#include <linux/stddef.h> +#include <vdso/limits.h> #include <linux/string.h> #include <linux/ctype.h> -#include <linux/kernel.h> -#include <linux/export.h> -#include <linux/bug.h> -#include <linux/errno.h> -#include <linux/slab.h> -#include <asm/unaligned.h> -#include <asm/byteorder.h> #include <asm/word-at-a-time.h> #include <asm/page.h>
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 4 lines · Andy Shevchenko
Have you checked Ingo Molnar's gigantic series (2k+ patches) for the header hell clean up? Perhaps we need to apply that first.
We missed you at plumbers. Tanzir gave a talk on this, with the raison d'etre of this work was to see if we could help automate Ingo's work.
https://www.youtube.com/watch?v=eFq_oqLiXPM
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 19 lines · Andy Shevchenko
Quoted 2 lines · Nick Desaulniers
Quoted 1 line · Al Viro
On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).Quoted 6 lines
Quoted 3 lines
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
Quoted 7 lines
The mandatory ones can be used, but not all of them. In some cases you even must include asm and not linux (
unaligned.h,byteorder.h, maybe others...).As I told, it comes with experience, we lack of the respective documentation (or file which is good for automation checks, like with IWYU).
I have an unpublished Linux Best Known Practices txt file with this "rule" and some other info that I have collected over a few years. I wouldn't say that it's up to date. Anyway, it's copied below FWIW.
Signature · 97 lines
~Randy --- ================================= Linux Kernel Best Known Practices ================================= N.B.: This is an evolving document. No need to init variables in bss to 0. bss is always cleared to 0. User interfaces should use explicit types (and hence type sizes) instead of generic types like "long" or "unsigned int". E.g. Also see Documentation/process/botching-up-ioctls.rst Unused fields of API structures or flags bits should be checked for value 0 on input and warn if they are not 0. submit-checklist Rule #1. use the kmalloc() helpers for variable-size array allocations. Generic drivers should not (do not) #include header files from <asm-generic>. Only arch code and arch-specific drivers do that. Don't use <asm/header.h> in generic driver code. Instead use some <linux/header.h> that then #includes <asm/header.h>. asm/ headers are implementation; linux/ headers are API. Use __maybe_unused to driver suspend/resume functions instead of surrounding them with #ifdef CONFIG_PM_SLEEP / #endif. Many API families provide full stub support for when their Kconfig option is disabled. This means that these APIs can still be used with COMPILE_TEST. Examples of these API families are: CONFIG_OF, <more>. CONFIG_namespace belongs to Kconfig files. A driver or filesystem or whatever should not have a local macro (or variable!) whose name begins with "CONFIG_". see Documentation/process/submit-checklist.rst run scripts/checkpatch.pl use Documentation/core-api/printk-formats.rst Don't introduce any new build errors or warnings Yes, that means that you need to check for warnings when you build. Yes, that includes Documentation or kernel-doc warnings or errors (now that we have those down to zero). Do NOT use stack space for DMA. Don't use 'extern' on function prototypes. (deprecated) Don't depend on Makefile order to control builtin module/driver init calls. Use initcall levels instead. In a Kconfig dependency list, by convention, please keep "|| COMPILE_TEST" last in the list. Shell scripts should use /bin/sh instead of another shell as much as possible. When making a patch such as a coding-style patch that should have no effect on the produced code, do a before/after comparison of the binary files to show that they are the same. If they are not the same, explain why not. Most scripts in the kernel source tree try to avoid bashisms. There may be some exceptions to this, such as bpf-related scripts and perf-related scripts. Don't use BIT() macros in UAPI headers. Either open-code bit fields or use _BITUL(). "sigcontext is accessible to user; do *NOT* assume that sigreturn will find what the kernel had written there" (Viro) Please just use whole numbers for patch versions, there's no need for .X for this type of thing as our tools do not know how to catch that, and it makes no sense. Just keep bumping the number please. (gkh) C99 style comments are OK in source code except in UAPI header files. They may be used with cstd=C90, which does not accept // comments. TODO: Should we say something about __delay() not being available in all arch-es so some other delay function is preferable. [maybe add something like this to checkpatch?] #####
Quoted 8 lines · Greg KH
Quoted 6 lines · Nick Desaulniers
Quoted 4 lines · Al Viro
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.
Quoted 6 lines
Quoted 2 lines
It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
General rule, NEVER include asm/*.h, there should be a
include/*.hinstead that works. So much so that checkpatch.pl should catch this, right?
ah, shoot, I was showing Tanzir how to use b4 for patch development, and forgot to check this. Indeed it does.
I can see how the check works (scripts/checkpatch.pl L5881). Decoding that will probably help us improve the tooling.
Quoted 3 lines
But of course, it doesn't always hold true, there are a few minor exceptions, but they are rare.
$ grep -r \\#include lib | grep asm
shows quite a few exceptions, and just in lib/.
For example, lib/math/int_log.c includes asm/bug.h. Is that a case where lib/math/int_log.c should be #include 'ing linux/bug.h rather than asm/bug.h?
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 19 lines · Andy Shevchenko
Quoted 2 lines · Nick Desaulniers
Quoted 1 line · Al Viro
On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).Quoted 6 lines
Quoted 3 lines
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
Quoted 7 lines
The mandatory ones can be used, but not all of them. In some cases you even must include asm and not linux (
unaligned.h,byteorder.h, maybe others...).As I told, it comes with experience, we lack of the respective documentation (or file which is good for automation checks, like with IWYU).
It would certainly be nice to have such information in the tree; "where should I pick $SYMBOL from?" is something one needs to find out often enough. To a large extent it's covered by "where in include/*.h do we have it defined?", but that's not all there is to it. E.g. "get_user() => use linux/uaccess.h".
There's also stuff like "$SYMBOL should not be used outside of arch/* and include/*, better use $OTHER_SYMBOL", etc.
Quoted 29 lines · Al Viro
Quoted 19 lines · Andy Shevchenko
Quoted 2 lines · Nick Desaulniers
Quoted 1 line · Al Viro
On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).Quoted 6 lines
Quoted 3 lines
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
Quoted 7 lines
The mandatory ones can be used, but not all of them. In some cases you even must include asm and not linux (
unaligned.h,byteorder.h, maybe others...).As I told, it comes with experience, we lack of the respective documentation (or file which is good for automation checks, like with IWYU).
Quoted 8 lines
It would certainly be nice to have such information in the tree; "where should I pick $SYMBOL from?" is something one needs to find out often enough. To a large extent it's covered by "where in
include/*.hdo we have it defined?", but that's not all there is to it. E.g. "get_user()=> uselinux/uaccess.h".There's also stuff like "$SYMBOL should not be used outside of
arch/*andinclude/*, better use $OTHER_SYMBOL", etc.
That's basically one of the tables we maintain.
https://github.com/ClangBuiltLinux/IWYUScripts/blob/main/symbol.imp
Of course, such a table is a living document; whether it resides in tree or out of tree eventually I don't particularly care either way. There are outstanding questions like "is it even possible to autogenerate such a table (vs manual curation)" and "are the ones we've coded so far correct?" I don't expect to have all of the answers with the initial implementation, but instead to collect feedback from kernel devs and iterate from there. Very helpful responses in this thread so far; we appreciate it.
I suspect that folks that have played with IWYU in the past encountered great difficulty since these override tables are poorly documented, and the out of the box defaults assume more of a userspace layout for common definitions (which our script which wraps IWYU disables).
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 7 lines · Randy Dunlap
I have an unpublished Linux Best Known Practices txt file with this "rule" and some other info that I have collected over a few years. I wouldn't say that it's up to date. Anyway, it's copied below FWIW.
Generic drivers should not (do not) #include header files from <asm-generic>. Only arch code and arch-specific drivers do that.
Ok, that confirms my understanding. We had picked up on that convention, but had missed the one below.
Quoted 4 lines
Don't use <
asm/header.h> in generic driver code. Instead use some <linux/header.h> that then #includes <asm/header.h>. asm/ headers are implementation; linux/ headers are API.
Ah, yeah, that's what we missed.
---
You should publish you doc somewhere, it can be helpful to refer back to them when you're on a different machine or to point someone else to.
I keep notes like this myself for debugging LLVM:
https://github.com/ClangBuiltLinux/llvm-dev-conf-2020/issues/3 (and recently gcc: https://github.com/ClangBuiltLinux/llvm-dev-conf-2020/issues/4)
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 8 lines · Al Viro
It would certainly be nice to have such information in the tree; "where should I pick $SYMBOL from?" is something one needs to find out often enough. To a large extent it's covered by "where in
include/*.hdo we have it defined?", but that's not all there is to it. E.g. "get_user()=> uselinux/uaccess.h".There's also stuff like "$SYMBOL should not be used outside of
arch/*andinclude/*, better use $OTHER_SYMBOL", etc.
Speaking of...
arch/arm64/include/asm/asm-prototypes.h:18:#include <asm/uaccess.h> arch/powerpc/include/asm/inst.h:8:#include <asm/uaccess.h> arch/powerpc/kvm/book3s_xive_native.c:16:#include <asm/uaccess.h> arch/powerpc/mm/book3s64/radix_pgtable.c:31:#include <asm/uaccess.h> arch/riscv/kernel/sys_riscv.c:15:#include <asm/uaccess.h> arch/xtensa/include/asm/asm-prototypes.h:10:#include <asm/uaccess.h> include/linux/uaccess.h:11:#include <asm/uaccess.h> tools/testing/crypto/chacha20-s390/test-cipher.c:8:#include <asm/uaccess.h>
Only one of those is legitimate (linux/uaccess.h); asm-prototypes.h ones are borderline, but probably should be pulling linux/uaccess.h, everything in *.c is really wrong. Powerpc asm/inst.h is the really unpleasant one here - it's pulled by quite a few other places, including several asm/*.h. Hell knows; might be worth splitting the {__,}{get_user_instr,copy_inst_from_kernel_nofault}() off to
a separate header (asm/inst-uaccess.h?), to be included only by the files that use any of those (that would be arch/powerpc/kernel/{align,hw_breakpoint_constraints,module_32,traps,vecemu}.c and arch/powerpc/kernel/trace/{ftrace,ftrace_64_pg}.c - not a single header among them). That header would pull linux/uaccess.h and asm/inst.h wouldn't need any uaccess at all; might spill into explicit includes of linux/uaccess.h in some of the places in arch/powerpc that pull asm/inst.h, directly or not...
Quoted 26 lines · Nick Desaulniers
Quoted 8 lines · Greg KH
Quoted 6 lines · Nick Desaulniers
Quoted 4 lines · Al Viro
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.Quoted 6 lines
Quoted 2 lines
It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
General rule, NEVER include asm/*.h, there should be a
include/*.hinstead that works. So much so that checkpatch.pl should catch this, right?ah, shoot, I was showing Tanzir how to use
b4for patch development, and forgot to check this. Indeed it does.I can see how the check works (
scripts/checkpatch.plL5881). Decoding that will probably help us improve the tooling.Quoted 3 lines
But of course, it doesn't always hold true, there are a few minor exceptions, but they are rare.
Quoted 7 lines
$ grep -r \\#include lib | grep asmshows quite a few exceptions, and just in
lib/.For example,
lib/math/int_log.cincludesasm/bug.h. Is that a case wherelib/math/int_log.cshould be #include 'inglinux/bug.hrather thanasm/bug.h?
Probably yes, but we don't normally go back and take coding style fixes for old files like this as it doesn't make much sense to do so.
But, if you are cleaning up the headers for large portions with the goal of faster builds, that's a good reason.
Good luck!
greg k-h
Quoted 12 lines · Greg KH
Quoted 11 lines
Quoted 3 lines
But of course, it doesn't always hold true, there are a few minor exceptions, but they are rare.
$ grep -r \\#include lib | grep asmshows quite a few exceptions, and just in
lib/.For example,
lib/math/int_log.cincludesasm/bug.h. Is that a case wherelib/math/int_log.cshould be #include 'inglinux/bug.hrather thanasm/bug.h?
Quoted 5 lines
Probably yes, but we don't normally go back and take coding style fixes for old files like this as it doesn't make much sense to do so.
But, if you are cleaning up the headers for large portions with the goal of faster builds, that's a good reason.
FWIW, the most common (by far - about 13% of such includes, over drivers/, fs/, mm/, net/ and sound/) is asm/unaligned.h.
The next are asm/io.h (10%), asm/byteorder.h (6%), asm/irq.h (5%), asm/div64.h (3%), asm/page.h and asm/dma.h (2% each). The rest is below that (the next is a bit over 1%).
In fs/* unaligned.h is again the top one (37 out of 139), followed by byteorder.h (30 out of 139), div64.h (12), page.h (10) and then comes random shite - I do not believe that fs/coda/psdev.c might have a legitimate reason to pull asm/io.h, for one thing...
Quoted 13 lines · Al Viro
Quoted 12 lines · Greg KH
Quoted 11 lines
Quoted 3 lines
But of course, it doesn't always hold true, there are a few minor exceptions, but they are rare.
$ grep -r \\#include lib | grep asmshows quite a few exceptions, and just in
lib/.For example,
lib/math/int_log.cincludesasm/bug.h. Is that a case wherelib/math/int_log.cshould be #include 'inglinux/bug.hrather thanasm/bug.h?
Quoted 8 lines
Quoted 5 lines
Probably yes, but we don't normally go back and take coding style fixes for old files like this as it doesn't make much sense to do so.
But, if you are cleaning up the headers for large portions with the goal of faster builds, that's a good reason.
FWIW, the most common (by far - about 13% of such includes, over
drivers/,fs/,mm/,net/andsound/) isasm/unaligned.h.
Why the hell is unaligned.h in asm/*, anyway?
We have 3 variants: arc, parisc and generic (== everything else). Both arc and parisc instances have an explicit include of asm-generic/unaligned.h (i.e. the generic variant).
On arc there's also misaligned_fixup() extern or stub, with exactly one user (in arch/arc/kernel/traps.c). On parisc there are externs for handle_unaligned() and check_unaligned() (3 call sites, all in arch/parisc/kernel/traps.c).
How about we take those into arch/{arc,parisc}/kernel/unaligned.h, slap #include "unaligned.h" into their traps.c and unaligned.c (callers and definitions resp.) and strip those from asm/unaligned.h? At that point we can remove arch/{arc,parisc}/asm/unaligned.h - everything will pick include/asm-generic/unaligned.h.
Then the next cycle we ask Linus to run the following:
for i in git grep -l -w asm/unaligned.h; do
sed -i -e "s/asm\/unaligned.h/linux\/unaligned.h/" $i
done
git mv include/asm-generic/unaligned.h include/linux/unaligned.h sed -i -e "/unaligned.h/d" include/asm-generic/Kbuild right before releasing -rc1 and asm/unaligned.h is gone...
Completely untested delta (for the non-automatic parts, that is) follows:
------------------
arc, parisc: get rid of private asm/unaligned.h
Declarations local to arch/*/kernel/*.c are better off *not* in a public header - arch/{arc,parisc}/kernel/unaligned.h is just fine for those bits.
With that done these asm/unaligned.h instances are reduced to include of asm-generic/unaligned.h and can be removed - unaligned.h is in mandatory-y in include/asm-generic/Kbuild.
---
arch/arc/include/asm/unaligned.h deleted +0 -27
@@ -1,27 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* -* Copyright (C) 2004, 2007-2010, 2011-2012 Synopsys, Inc. (www.synopsys.com) -*/ - -#ifndef _ASM_ARC_UNALIGNED_H -#define _ASM_ARC_UNALIGNED_H - -/* ARC700 can't handle unaligned Data accesses. */ - -#include <asm-generic/unaligned.h> -#include <asm/ptrace.h> - -#ifdef CONFIG_ARC_EMUL_UNALIGNED -int misaligned_fixup (unsigned long address,struct pt_regs *regs, -struct callee_regs *cregs); -#else -static inline int -misaligned_fixup (unsigned long address,struct pt_regs *regs, -struct callee_regs *cregs) -{ -/* Not fixed */ -return 1 ; -} -#endif - -#endif /* _ASM_ARC_UNALIGNED_H */
arch/arc/kernel/traps.c +1 -0
@@ -20,6 +20,7 @@#include <asm/setup.h> #include <asm/unaligned.h> #include <asm/kprobes.h> +#include "unaligned.h" void die (const char *str,struct pt_regs *regs,unsigned long address) {
arch/arc/kernel/unaligned.c +1 -0
@@ -12,6 +12,7 @@#include <linux/ptrace.h> #include <linux/uaccess.h> #include <asm/disasm.h> +#include "unaligned.h" #ifdef CONFIG_CPU_BIG_ENDIAN #define BE 1
arch/arc/kernel/unaligned.h new file +16 -0
@@ -0,0 +1,16 @@ +struct pt_regs ; +struct callee_regs ; + +#ifdef CONFIG_ARC_EMUL_UNALIGNED +int misaligned_fixup (unsigned long address,struct pt_regs *regs, +struct callee_regs *cregs); +#else +static inline int +misaligned_fixup (unsigned long address,struct pt_regs *regs, +struct callee_regs *cregs) +{ +/* Not fixed */ +return 1 ; +} +#endif +
arch/parisc/include/asm/unaligned.h deleted +0 -11
@@ -1,11 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_PARISC_UNALIGNED_H -#define _ASM_PARISC_UNALIGNED_H - -#include <asm-generic/unaligned.h> - -struct pt_regs ; -void handle_unaligned (struct pt_regs *regs); -int check_unaligned (struct pt_regs *regs); - -#endif /* _ASM_PARISC_UNALIGNED_H */
arch/parisc/kernel/traps.c +2 -0
@@ -47,6 +47,8 @@#include <linux/kgdb.h> #include <linux/kprobes.h> +#include "unaligned.h" +#if defined(CONFIG_LIGHTWEIGHT_SPINLOCK_CHECK )#include <asm/spinlock.h> #endif
arch/parisc/kernel/unaligned.c +1 -0
@@ -15,6 +15,7 @@#include <asm/unaligned.h> #include <asm/hardirq.h> #include <asm/traps.h> +#include "unaligned.h" /* #define DEBUG_UNALIGNED 1 */
arch/parisc/kernel/unaligned.h new file +3 -0
@@ -0,0 +1,3 @@ +struct pt_regs ; +void handle_unaligned (struct pt_regs *regs); +int check_unaligned (struct pt_regs *regs);
Quoted 40 lines · Al Viro
Quoted 13 lines · Al Viro
Quoted 12 lines · Greg KH
Quoted 11 lines
Quoted 3 lines
But of course, it doesn't always hold true, there are a few minor exceptions, but they are rare.
$ grep -r \\#include lib | grep asmshows quite a few exceptions, and just in
lib/.For example,
lib/math/int_log.cincludesasm/bug.h. Is that a case wherelib/math/int_log.cshould be #include 'inglinux/bug.hrather thanasm/bug.h?Quoted 8 lines
Quoted 5 lines
Probably yes, but we don't normally go back and take coding style fixes for old files like this as it doesn't make much sense to do so.
But, if you are cleaning up the headers for large portions with the goal of faster builds, that's a good reason.
FWIW, the most common (by far - about 13% of such includes, over
drivers/,fs/,mm/,net/andsound/) isasm/unaligned.h.Why the hell is
unaligned.hin asm/*, anyway?We have 3 variants: arc, parisc and generic (== everything else). Both arc and parisc instances have an explicit include of
asm-generic/unaligned.h(i.e. the generic variant).On arc there's also
misaligned_fixup()extern or stub, with exactly one user (inarch/arc/kernel/traps.c). On parisc there are externs forhandle_unaligned()andcheck_unaligned()(3 call sites, all inarch/parisc/kernel/traps.c).How about we take those into
arch/{arc,parisc}/kernel/unaligned.h, slap #include "unaligned.h" into theirtraps.candunaligned.c(callers and definitions resp.) and strip those fromasm/unaligned.h? At that point we can removearch/{arc,parisc}/asm/unaligned.h- everything will pickinclude/asm-generic/unaligned.h.
Quoted 7 lines
Then the next cycle we ask Linus to run the following:
for i ingit grep -l -w asm/unaligned.h; dosed -i -e "s/asm\/unaligned.h/linux\/unaligned.h/" $idone
git mvinclude/asm-generic/unaligned.hinclude/linux/unaligned.hsed -i -e "/unaligned.h/d"include/asm-generic/Kbuildright before releasing -rc1 andasm/unaligned.his gone...
Please do, it's really annoying and would be great to fix up.
thanks,
greg k-h
kernel test robot
build botkernel test robot noticed the following build errors:
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on 33cc938e65a98f1d29d0a18403dbbee050dcad9a]
url: https://github.com/intel-lab-lkp/linux/commits/tanzirh-google-com/lib-string-shrink-lib-string-i-via-IWYU/20231206-050121 base: 33cc938e65a98f1d29d0a18403dbbee050dcad9a patch link: https://lore.kernel.org/r/20231205-libstringheader-v1-1-7f9c573053a7%40gmail.com patch subject: [PATCH] lib/string: shrink lib/string.i via IWYU config: sh-allmodconfig (https://download.01.org/0day-ci/archive/20231206/[email protected]/config) compiler: sh4-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from lib/string.c:29:
lib/string.c: In function 'strscpy':
>> arch/sh/include/asm/word-at-a-time.h:15:36: error: implicit declaration of function 'REPEAT_BYTE' [-Werror=implicit-function-declaration]
15 | #define WORD_AT_A_TIME_CONSTANTS { REPEAT_BYTE(0x01), REPEAT_BYTE(0x80) }
| ^~~~~~~~~~~
lib/string.c:124:49: note: in expansion of macro 'WORD_AT_A_TIME_CONSTANTS'
124 | const struct word_at_a_time constants = WORD_AT_A_TIME_CONSTANTS;
| ^~~~~~~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/REPEAT_BYTE +15 arch/sh/include/asm/word-at-a-time.h
cba8df4be3bdf1 Paul Mundt 2012-06-04 14cba8df4be3bdf1 Paul Mundt 2012-06-04 @15 #define WORD_AT_A_TIME_CONSTANTS { REPEAT_BYTE(0x01), REPEAT_BYTE(0x80) } cba8df4be3bdf1 Paul Mundt 2012-06-04 16
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 2 lines · Nick Desaulniers
Hmm...how does one know that
linux/bits.his the higher-level include ofasm/bitsperlong.h?
I think this the wrong way of thinking. In general we should always avoid including asm/ headers unless there is no other way. No other way is not formalized right now, but I think the answer is roughly if no linux/ headers includes that asm header. This will probably give a few wrong results at the moment, but it's probably worth fixing them up.
Andy Shevchenko
reviewerQuoted 1 line · Nick Desaulniers
On Tue, Dec 5, 2023 at 1:59 PM Greg KH <[email protected]> wrote:
...
Quoted 3 lines
For example,
lib/math/int_log.cincludesasm/bug.h. Is that a case wherelib/math/int_log.cshould be #include 'inglinux/bug.hrather thanasm/bug.h?
For the C-files it's fine to include linux/* versions of asm/* ones, for the _headers_ I would prefer to be strictly minimalistic, because of possible header dependency hell. If something is defined in asm/* and some header wants to use that, it's better to include asm/*.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Andy Shevchenko
reviewerQuoted 4 lines · Al Viro
Quoted 3 lines · Andy Shevchenko
Quoted 2 lines · Nick Desaulniers
Quoted 1 line · Al Viro
On Tue, Dec 05, 2023 at 08:58:53PM +0000, [email protected] wrote:
...
Quoted 21 lines
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.
This change to
lib/string.cresulted in a preprocessed size oflib/string.ifrom 26371 lines to 5232 lines (-80%).Quoted 6 lines
Quoted 3 lines
It also breeds includes of asm/*.h, by the look of the output, which is not a good thing in general ;-/ E.g. #include <
asm/uaccess.h> *anywhere* outside oflinux/uaccess.his a bad idea.It's not clear to me when it's ok to #include <asm/*.h>. Is there a convention here that I'm missing?
Quoted 7 lines
The mandatory ones can be used, but not all of them. In some cases you even must include asm and not linux (
unaligned.h,byteorder.h, maybe others...).As I told, it comes with experience, we lack of the respective documentation (or file which is good for automation checks, like with IWYU).
Quoted 8 lines
It would certainly be nice to have such information in the tree; "where should I pick $SYMBOL from?" is something one needs to find out often enough. To a large extent it's covered by "where in
include/*.hdo we have it defined?", but that's not all there is to it. E.g. "get_user()=> uselinux/uaccess.h".There's also stuff like "$SYMBOL should not be used outside of
arch/*andinclude/*, better use $OTHER_SYMBOL", etc.
Precisely! That's what many developers will benefit from!
Signature · 2 lines
With Best Regards, Andy Shevchenko
Andy Shevchenko
reviewerQuoted 13 lines · kernel test robot
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on
33cc938e65a98f1d29d0a18403dbbee050dcad9a]url: https://github.com/intel-lab-lkp/linux/commits/tanzirh-google-com/lib-string-shrink-lib-string-i-via-IWYU/20231206-050121 base: 33cc938e65a98f1d29d0a18403dbbee050dcad9a patch link: https://lore.kernel.org/r/20231205-libstringheader-v1-1-7f9c573053a7%40gmail.com patch subject: [PATCH] lib/string: shrink lib/string.i via IWYU config: sh-allmodconfig (https://download.01.org/0day-ci/archive/20231206/[email protected]/config) compiler: sh4-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)
...
Quoted 4 lines
lib/string.c:124:49: note: in expansion of macro 'WORD_AT_A_TIME_CONSTANTS'124 | const struct word_at_a_time constants = WORD_AT_A_TIME_CONSTANTS;
cba8df4be3bdf1Paul Mundt 2012-06-04 @15 #defineWORD_AT_A_TIME_CONSTANTS{REPEAT_BYTE(0x01),REPEAT_BYTE(0x80)}
REPEAT_BYTE() shouldn't be even in kernel.h to begin with!
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 16 lines · Al Viro
Quoted 7 lines · Nick Desaulniers
The tooling Tanzir is working on does wrap IWYU, and does support such mapping (of 'low level' to 'high level' headers; more so, if it recommends X you can override to suggest Y instead).
arch/nios/also doesn't provide abug.h, which this patch is suggesting we include directly. I guess the same goes forasm/rwonce.h.See
include/asm-generic/Kbuild:
mandatory-y +=bug.h
...
mandatory-y +=rwonce.h
Quoted 6 lines
IOW, sh will have
asm/bug.handas/rwonce.hcopied from asm-generic.Still, includes of asm/*.h had been a massive headache historically and breeding more of those shouldn't be overdone.
More painful problem is arch- and config-dependent stuff, though...
Ah, it looks like include/uapi/asm-generic/Kbuild also makes use of this pattern using mandatory-y.
So I think we can handle this as a two step translation. We can tell the tooling to 'nevermind recommending X, always replace recommendations for X with Y <for raisins>', so:
1. any recommendation to use asm-generic/foo.h should be replaced with asm/foo.h (always, based on those 2 Kbuild files, could autogenerate)
2. some recommendations to use asm/foo.h should be replaced with linux/foo.h (not necessarily codified anywhere; depends on if there is a linux/foo.h, will manually curate this list for now)
Orthogonal but some places in tree can be cleaned up to include linux/foo.h rather than asm/foo.h.
Does this sound like an improvement to my mental model of the conventions used for kernel header inclusion within the linux kernel?
Tanzir nearly has the above done (for 1, 2 we will probably need to iterate on more). We've also beefed up our local testing to test more architectures. I expect Tanzir to send a v2 of this patch (as a series, with the fix for ARCH=sh) later this week, if the above seems correct.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 38 lines · Nick Desaulniers
Quoted 15 lines · Al Viro
Quoted 7 lines · Nick Desaulniers
The tooling Tanzir is working on does wrap IWYU, and does support such mapping (of 'low level' to 'high level' headers; more so, if it recommends X you can override to suggest Y instead).
arch/nios/also doesn't provide abug.h, which this patch is suggesting we include directly. I guess the same goes forasm/rwonce.h.See
include/asm-generic/Kbuild:
mandatory-y +=bug.h
...
mandatory-y +=rwonce.hQuoted 6 lines
IOW, sh will have
asm/bug.handas/rwonce.hcopied from asm-generic.Still, includes of asm/*.h had been a massive headache historically and breeding more of those shouldn't be overdone.
More painful problem is arch- and config-dependent stuff, though...
Ah, it looks like
include/uapi/asm-generic/Kbuildalso makes use of this pattern using mandatory-y.So I think we can handle this as a two step translation. We can tell the tooling to 'nevermind recommending X, always replace recommendations for X with Y <for raisins>', so:
1. any recommendation to useasm-generic/foo.hshould be replaced withasm/foo.h(always, based on those 2Kbuildfiles, could autogenerate)
2. some recommendations to useasm/foo.hshould be replaced withlinux/foo.h(not necessarily codified anywhere; depends on if there is alinux/foo.h, will manually curate this list for now)Orthogonal but some places in tree can be cleaned up to include
linux/foo.hrather thanasm/foo.h.
Quoted 8 lines
Does this sound like an improvement to my mental model of the conventions used for kernel header inclusion within the linux kernel?
Tanzir nearly has the above done (for 1, 2 we will probably need to iterate on more). We've also beefed up our local testing to test more architectures. I expect Tanzir to send a v2 of this patch (as a series, with the fix for ARCH=sh) later this week, if the above seems correct.
Whatever you choose, please make sure it gets documented, so we won't go another round in the future for something similar.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 13 lines · Greg KH
Quoted 5 lines
slap #include "
unaligned.h" into theirtraps.candunaligned.c(callers and definitions resp.) and strip those fromasm/unaligned.h? At that point we can removearch/{arc,parisc}/asm/unaligned.h- everything will pickinclude/asm-generic/unaligned.h.Quoted 7 lines
Then the next cycle we ask Linus to run the following:
for i ingit grep -l -w asm/unaligned.h; dosed -i -e "s/asm\/unaligned.h/linux\/unaligned.h/" $idone
git mvinclude/asm-generic/unaligned.hinclude/linux/unaligned.hsed -i -e "/unaligned.h/d"include/asm-generic/Kbuildright before releasing -rc1 andasm/unaligned.his gone...
Quoted 1 line
Please do, it's really annoying and would be great to fix up.
FWIW, turns out that we have several places in drivers/* that pullasm-generic/unaligned.h. IMO that's completely wrong - not just in this case (it's a matter of trivially adjusting the script), but I think that as a matter of policy we should *NOT* have includes of asm-generic/*.h anywhere in drivers/ fs/ io_uring/ kernel/ mm/ net/ sound/
Current situation (outside of arch, include, scripts and tools,
with asm-generic/unaligned.h cases already taken out):
drivers/android/binderfs.c:32:#include <uapi/asm-generic/errno-base.h> drivers/clk/microchip/clk-mpfs-ccc.c:7:#include "asm-generic/errno-base.h" drivers/firmware/arm_scmi/shmem.c:13:#include <asm-generic/bug.h> drivers/irqchip/irq-ti-sci-inta.c:24:#include <asm-generic/msi.h> drivers/mtd/nand/raw/ingenic/ingenic_ecc.h:9:#include <uapi/asm-generic/errno-base.h> drivers/platform/x86/hp/hp-bioscfg/passwdobj-attributes.c:10:#include <asm-generic/posix_types.h> io_uring/uring_cmd.c:10:#include <uapi/asm-generic/ioctls.h> lib/trace_readwrite.c:10:#include <asm-generic/io.h>
mm/damon/vaddr.c:10:#include <asm-generic/mman-common.h> net/sunrpc/xprtrdma/verbs.c:58:#include <asm-generic/barrier.h> rust/uapi/uapi_helper.h:9:#include <uapi/asm-generic/ioctl.h>
That's it. asm-generic/errno-base.h cases - just use linux/errno.h and be done with it. Anyone who goes "I've got ENOENT and ENOMEM in my driver, but those two are covered by errno-base.h and I can be clever and use just that" is wrong.
asm-generic/mman-common.h - and linux/mman.h would not serve why, exactly? uapi/linux/linux/mman.h, if one really feels like it...
asm-generic/ioctls.h in io_uring - definitely wrong; SIOCINQ is what it's brought for, but that's in sockios.h. It expands to FIONREAD, which *is* in ioctls.h, but...
arch/alpha/include/uapi/asm/ioctls.h:11:#define FIONREAD _IOR('f', 127, int)
arch/mips/include/uapi/asm/ioctls.h:64:#define FIONREAD 0x467f
arch/parisc/include/uapi/asm/ioctls.h:35:#define FIONREAD 0x541B
arch/powerpc/include/uapi/asm/ioctls.h:11:#define FIONREAD _IOR('f', 127, int)
arch/sh/include/uapi/asm/ioctls.h:11:#define FIONREAD _IOR('f', 127, int)
arch/sparc/include/uapi/asm/ioctls.h:101:#define FIONREAD _IOR('f', 127, int)
arch/xtensa/include/uapi/asm/ioctls.h:23:#define FIONREAD _IOR('f', 127, int)
include/uapi/asm-generic/ioctls.h:46:#define FIONREAD 0x541B
See the problem? On mips the value is not 0x541B - it's 0x467F; on alpha, powerpc and sparc it's 0x4004667F, on sh and xtensa - 0x8004667F (different expansions of _IOR).
The only thing that has any business including asm-generic/ioctls.h is arch/*/include/uapi/asm/ioctls.h - if it feels like doing so. Anywhere else it ought to be asm/ioctls.h.
asm-generic/msi.h - looks bogus; there's an include of linux/msi.h several lines above, and that pulls asm/msi.h. Which makes it either a no-op or a build bug (the latter - if that driver can be built on x86 with CONFIG_GENERIC_MSI_IRQ defined;
typedef struct irq_alloc_info msi_alloc_info_t;
in asm/msi.h vs.
typedef struct msi_alloc_info {
struct msi_desc *desc;
irq_hw_number_t hwirq;
unsigned long flags;
union {
unsigned long ul;
void *ptr;
} scratchpad[NUM_MSI_ALLOC_SCRATCHPAD_REGS];
} msi_alloc_info_t;in asm-generic/msi.h).
Hell knows about the rest, but they all look very dubious...
Quoted 1 line · Al Viro
drivers/firmware/arm_scmi/shmem.c:13:#include <
asm-generic/bug.h>
Should just use linux/bug.h and be done with that.
Quoted 1 line
drivers/platform/x86/hp/hp-bioscfg/passwdobj-attributes.c:10:#include <
asm-generic/posix_types.h>
Completely pointless; not to mention that none of the types defined there are used anywhere in that file, the previous line pulls their private header, which explicitly pulls linux/types.h.
Quoted 1 line
lib/trace_readwrite.c:10:#include <
asm-generic/io.h>
Yeccchhh... This is just plain wrong.
What happens here is that hooks are added to writeb() et.al., to allow ftrace to play with those. By default these are empty inlines; with CONFIG_TRACE_MMIO_ACCESS they are real function calls, the functions living in lib/trace_readwrite.c
asm-generic/io.h is pulled by all asm/io.h instances, so that's where the externs went. That would've made sense, except that asm-generic/io.h is used as a backstop for architectures that had not bothered to define e.g. readl() of their own. And *that* is where the calls of those hooks had gone. IOW, if architecture has readl()/writeb()/whatnot of its own, it won't get those hooks at all.
It smells like a conversion in progress, stalled after the first (and only) architecture where that thing is selectable. In effect, it's arm64-specific.
Quoted 1 line
net/sunrpc/xprtrdma/verbs.c:58:#include <
asm-generic/barrier.h>
Bogus, same as the include of asm/bitops.h immediately before that line (the latter would've blown up if we hadn't already pulled linux/bitops.h - which needs asm/barrier.h and pulls it, TYVM).
Quoted 1 line
rust/uapi/uapi_helper.h:9:#include <
uapi/asm-generic/ioctl.h>
To the rust crowd, that... It's wrong for e.g. powerpc - uapi/asm/ioctl.h in there does pull asm-generic/ioctl.h, but only after
#define _IOC_SIZEBITS 13
#define _IOC_DIRBITS 3
#define _IOC_NONE 1U #define _IOC_READ 2U #define _IOC_WRITE 4U
which means that trying to use asm-generic/ioctl.h directly will yield the wrong numbers out of _IOC() et.al.
So...
in arch headers (..../asm/.../*.h and similar):
OK, that's what asm-generic is for
in asm-generic headers themselves (..../asm-generic/.../*.h):
OK
in linker scripts (*/*.lds{,.S}):
asm-generic/vmlinux.lds.h is fine in those (and nowhere else)
in */*audit*.c:
asm-generic/audit_*.h is OK there (ugly, but...)
in drivers,fs,init,io_uring,ipc,kernel,mm,net,sound,virt and probably rust:
100% bollocks, not a single valid use
in lib/trace_readwrite.c:
asm-generic/io.h is an exception; complicated story.
That leaves several instances in arch/, tools/ and include/linux, plus some oddities in makefiles, scripts, etc. And include/linux ones are also not obviously correct - e.g. linux/bug.h pulls asm/bug.h, then (under ifdef CONFIG_BUG) asm-generic/bug.h. AFAICS on all architectures we have asm/bug.h pulling asm-generic/bug.h (the ones that don't have asm/bug.h of their own get it generated with just such an include in it). So do we need that include in linux/bug.h these days?