Skip to contentSkip to conversation

shrink lib/string.i via IWYU

PATCH Superseded [email protected] 6 participants 15 messages Dec 14, 2023 – Jan 11, 2024 1 message not in the archive

Subsystems (3)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover shrink lib/string.i via IWYU

This patch series changes the include list of string.c to minimize the preprocessing size. The patch series intends to remove kernel.h which causes problems in the sh architecture because kernel.h is not properly declared in asm/word-at-a-time.h

---

---
Changes in v2:
- Transformed into a patch series
- Changed asm inclusions to linux inclusions
- added a patch to sh
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Tanzir Hasan (2):
sh: Added kernel.h to word-at-a-time
lib/string: shrink lib/string.i via IWYU

 arch/sh/include/asm/word-at-a-time.h |  1 +
 lib/string.c                         | 14 +++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)

---
base-commit: 33cc938e65a98f1d29d0a18403dbbee050dcad9a
change-id: 20231204-libstringheader-e238e2af5eec

Best regards,

Signature · 1 line
Tanzir Hasan <[email protected]>
#1 lore raw

PATCH v2 1/2 sh: Added kernel.h to word-at-a-time

This patch includes linux/kernel.h in asm/word-at-a-time.h for the sh architecture. WORD_AT_A_TIME_CONSTANTS depends on kernel.h. Making this implicit dependancy explicit allows for later improvements in the lib/string.c inclusion list.

---
 arch/sh/include/asm/word-at-a-time.h | 1 +
 1 file changed, 1 insertion(+)
arch/sh/include/asm/word-at-a-time.h +1 -0
index 4aa398455b94..f680f5f1d626 100644
@@ -5,6 +5,7 @@
 #ifdef CONFIG_CPU_BIG_ENDIAN
 # include <asm-generic/word-at-a-time.h>
 #else
+#include <linux/kernel.h>
 /*
  * Little-endian version cribbed from x86.
  */
Signature · 1 line
2.43.0.472.g3155946c3a-goog
#11 lore raw

PATCH v2 2/2 lib/string: shrink lib/string.i via IWYU

In reply to [email protected]

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to lib/string.c resulted in a preprocessed size of lib/string.i from 26371 lines to 5259 lines (-80%) for the x86 defconfig.

---
 lib/string.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
lib/string.c +7 -7
index be26623953d2..7fe1acefb1a1 100644
@@ -16,16 +16,16 @@
 
 #define __NO_FORTIFY
 #include <linux/types.h>
-#include <linux/string.h>
-#include <linux/ctype.h>
-#include <linux/kernel.h>
-#include <linux/export.h>
+#include <linux/bits.h>
 #include <linux/bug.h>
 #include <linux/errno.h>
-#include <linux/slab.h>
-
+#include <asm/rwonce.h>
+#include <linux/linkage.h>
+#include <linux/stddef.h>
+#include <vdso/limits.h>
+#include <linux/string.h>
+#include <linux/ctype.h>
 #include <asm/unaligned.h>
-#include <asm/byteorder.h>
 #include <asm/word-at-a-time.h>
 #include <asm/page.h>
 
Signature · 1 line
2.43.0.472.g3155946c3a-goog
#2 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

In reply to [email protected]
Quoted 6 lines · On Thu, Dec 14, 2023 at 09:06:12PM +0000, [email protected] wrote:

This patch includes linux/kernel.h in asm/word-at-a-time.h for the sh architecture. WORD_AT_A_TIME_CONSTANTS depends on kernel.h. Making this implicit dependancy explicit allows for later improvements in the lib/string.c inclusion list.

This patch is missing your Signed-off-by: tag.

Also, please double-check your email configs: your full name is missing from your emails (it's just "[email protected]"):
https://lore.kernel.org/lkml/[email protected]/

But otherwise, the change itself looks fine. :)

Signature · 1 line
Kees Cook
Reviewed-by on v2 2/2
#12 lore raw

Re: [PATCH v2 2/2] lib/string: shrink lib/string.i via IWYU

In reply to [email protected]
Quoted 11 lines · On Thu, Dec 14, 2023 at 09:06:13PM +0000, [email protected] wrote:

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to lib/string.c resulted in a preprocessed size of lib/string.i from 26371 lines to 5259 lines (-80%) for the x86 defconfig.

nit: Please drop this blank line to keep your tags together.

Quoted 1 line

I'm glad to see such a big difference with just a little header tweaking. I look forward to more like this!

Signature · 1 line
Kees Cook
#3 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

In reply to Kees Cook
Quoted 4 lines · Kees Cook

Also, please double-check your email configs: your full name is missing from your emails (it's just "[email protected]"):
https://lore.kernel.org/lkml/[email protected]/

This is the issue related to use of our internal mailer with b4. Konstantin fixed this in b4 a while ago, but I suspect the version of b4 we're getting from apt still does not contain the fix.

Or perhaps we need to switch to pyenv and pypi to install a newer version of b4. Tanzir (sits right next to me) and reports his version of b4 is:

0.12.3

On pypi, looks like there's a 0.12.4
https://pypi.org/project/b4/#history
Looking at those release dates, I'm pretty sure Konstantine fixed this particular issue in the 0.12.4 release.

(I made this mistake too recently, with my latest commit you picked up)

Debian stable says it has 0.12.0 https://packages.debian.org/stable/b4
Debian unstable says it has 0.12.4 https://packages.debian.org/unstable/b4
IDK where 0.12.3 is coming from. Perhaps our internal mirrors are lagging behind.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#4 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

In reply to [email protected]
Quoted 6 lines · On Thu, Dec 14, 2023 at 09:06:12PM +0000, [email protected] wrote:

This patch includes linux/kernel.h in asm/word-at-a-time.h for the sh architecture. WORD_AT_A_TIME_CONSTANTS depends on kernel.h. Making this implicit dependancy explicit allows for later improvements in the lib/string.c inclusion list.

You forgot your SoB, but...

...

Quoted 4 lines

#ifdef CONFIG_CPU_BIG_ENDIAN

 # include <asm-generic/word-at-a-time.h>
 #else
+#include <linux/kernel.h>

I highly discourage from doing that. Instead, split what is needed to the separate (new) header and include that one.

Quoted 3 lines

/*
* Little-endian version cribbed from x86.
*/

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#13 lore raw

Re: [PATCH v2 2/2] lib/string: shrink lib/string.i via IWYU

In reply to [email protected]
Quoted 5 lines · On Thu, Dec 14, 2023 at 09:06:13PM +0000, [email protected] wrote:

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

Quoted 7 lines

This change to lib/string.c resulted in a preprocessed size of lib/string.i from 26371 lines to 5259 lines (-80%) for the x86 defconfig.

Tag block should not have blank line(s).

Quoted 10 lines

#include <linux/types.h>

lib/string.c +1 -6
-#include <linux/string.h>
-#include <linux/ctype.h>
-#include <linux/kernel.h>
-#include <linux/export.h>
+#include <linux/bits.h>
 #include <linux/bug.h>
 #include <linux/errno.h>
-#include <linux/slab.h>
-

lib/string.c line 30

Quoted 10 lines
lib/string.c +6 -1
+#include <asm/rwonce.h>
+#include <linux/linkage.h>
+#include <linux/stddef.h>
+#include <vdso/limits.h>
+#include <linux/string.h>
+#include <linux/ctype.h>
 #include <asm/unaligned.h>
-#include <asm/byteorder.h>
 #include <asm/word-at-a-time.h>
 #include <asm/page.h>

My gosh, this becomes unreadable!

Please make sure you go with groups of headers from more generic to more particular, i.e.

linux/*

asm/*

vdso/*

BUT, why the heck vdso? It should be rather uapi/ or regular linux/.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#6 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

In reply to a message not in the archive
Quoted 2 lines · Tanzir Hasan
Quoted 1 line · Andy Shevchenko

On Thu, Dec 14, 2023 at 09:06:12PM +0000, [email protected] wrote:

...

Quoted 11 lines
Quoted 4 lines
Quoted 1 line

+#include <linux/kernel.h>

I highly discourage from doing that. Instead, split what is needed to the separate (new) header and include that one.

I think it would make the most sense to do this in a separate patch. What word-at-a-time.h needs from kernel.h is REPEAT_BYTE and to my knowledge, almost every other version of word-at-a-time.h includes kernel.h gets this by including kernel.h. A future change could be removing REPEAT_BYTE out of kernel.h

Just create a patch that either moves that macro (along with upper_*() and lower_*() APIs) to a more distinguishable header
(maybe bytes.h or words.h or wordpart.h, etc) and use it in your case and fix others.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#7 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

Quoted 19 lines · Andy Shevchenko
Quoted 2 lines · Tanzir Hasan
Quoted 1 line · Andy Shevchenko

On Thu, Dec 14, 2023 at 09:06:12PM +0000, [email protected] wrote:

...

Quoted 11 lines
Quoted 4 lines
Quoted 1 line

+#include <linux/kernel.h>

I highly discourage from doing that. Instead, split what is needed to the separate (new) header and include that one.

I think it would make the most sense to do this in a separate patch. What word-at-a-time.h needs from kernel.h is REPEAT_BYTE and to my knowledge, almost every other version of word-at-a-time.h includes kernel.h gets this by including kernel.h. A future change could be removing REPEAT_BYTE out of kernel.h

Quoted 4 lines

Just create a patch that either moves that macro (along with upper_*() and lower_*() APIs) to a more distinguishable header
(maybe bytes.h or words.h or wordpart.h, etc) and use it in your case and fix others.

Andy,
These are good suggestions and we should do them...

...and Tanzir only has 3 weeks left of his internship. I don't want him to get bogged down chasing build regressions from modifying the headers themselves. I think what's best for him from here through the remainder of his internship is to stay focused on applying suggestions from IWYU to just modify the #include list of .c files, and not start splitting .h files. Splitting the .h files will be the next step, and is made easier by having the codebase not have so many indirect includes (via IWYU), but we need time to soak header changes, and time Tanzir does not have. Please can we keep the suggestions focused on whether the modifications to the header includes (and the tangential cleanups) are correct?

While REPEAT_BYTE has a manageable number of users, upper_* and lower_* have significantly more; I worry about moving those causing regressions. We can move them, but such changes would need significantly more soak time than this series IMO. Tanzir is also working on statistical analysis; I suspect if he analyzes include/linux/kernel.h, he can comment on whether the usage of REPEAT_BYTE is correlated with the usage of upper_* and lower_* in order to inform whether they should be grouped together or not.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#8 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

Quoted 8 lines

While REPEAT_BYTE has a manageable number of users, upper_* and lower_* have significantly more; I worry about moving those causing regressions. We can move them, but such changes would need significantly more soak time than this series IMO. Tanzir is also working on statistical analysis; I suspect if he analyzes include/linux/kernel.h, he can comment on whether the usage of REPEAT_BYTE is correlated with the usage of upper_* and lower_* in order to inform whether they should be grouped together or not.

Removing REPEAT_BYTE is manageable and I have already moved it. I will be pushing a patch that moves just that into another file called wordpart.h. There are too many instances of the other functions for it to make sense to remove them all in this patch.

Best,
Tanzir

#10 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

In reply to Nick Desaulniers
Quoted 5 lines · Nick Desaulniers
Quoted 3 lines · Andy Shevchenko
Quoted 2 lines · Tanzir Hasan
Quoted 1 line · Andy Shevchenko

On Thu, Dec 14, 2023 at 09:06:12PM +0000, [email protected] wrote:

...

Quoted 20 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
Quoted 1 line

+#include <linux/kernel.h>

I highly discourage from doing that. Instead, split what is needed to the separate (new) header and include that one.

I think it would make the most sense to do this in a separate patch. What word-at-a-time.h needs from kernel.h is REPEAT_BYTE and to my knowledge, almost every other version of word-at-a-time.h includes kernel.h gets this by including kernel.h. A future change could be removing REPEAT_BYTE out of kernel.h

Quoted 4 lines

Just create a patch that either moves that macro (along with upper_*() and lower_*() APIs) to a more distinguishable header
(maybe bytes.h or words.h or wordpart.h, etc) and use it in your case and fix others.

Andy,
These are good suggestions and we should do them...

Quoted 11 lines

...and Tanzir only has 3 weeks left of his internship. I don't want him to get bogged down chasing build regressions from modifying the headers themselves. I think what's best for him from here through the remainder of his internship is to stay focused on applying suggestions from IWYU to just modify the #include list of .c files, and not start splitting .h files. Splitting the .h files will be the next step, and is made easier by having the codebase not have so many indirect includes (via IWYU), but we need time to soak header changes, and time Tanzir does not have. Please can we keep the suggestions focused on whether the modifications to the header includes (and the tangential cleanups) are correct?

Understood. Can we add a comment like

/* FIXME: replace with a proper header to avoid dependency hell */ #include <linux/kernel.h>

?

Quoted 3 lines

While REPEAT_BYTE has a manageable number of users, upper_* and lower_* have significantly more; I worry about moving those causing regressions.

If you look at how I did similar in the past, I back included new header into kernel.h. Not the pretty solution, but allows to split in the new code.

Quoted 1 line

We can move them, but such changes would need significantly more soak time

s/significantly//

But I got your point, see above.

Quoted 4 lines

than this series IMO. Tanzir is also working on statistical analysis; I suspect if he analyzes include/linux/kernel.h, he can comment on whether the usage of REPEAT_BYTE is correlated with the usage of upper_* and lower_* in order to inform whether they should be grouped together or not.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#9 lore raw

Re: [PATCH v2 1/2] sh: Added kernel.h to word-at-a-time

In reply to Tanzir Hasan
Quoted 10 lines · Tanzir Hasan
Quoted 8 lines

While REPEAT_BYTE has a manageable number of users, upper_* and lower_* have significantly more; I worry about moving those causing regressions. We can move them, but such changes would need significantly more soak time than this series IMO. Tanzir is also working on statistical analysis; I suspect if he analyzes include/linux/kernel.h, he can comment on whether the usage of REPEAT_BYTE is correlated with the usage of upper_* and lower_* in order to inform whether they should be grouped together or not.

Removing REPEAT_BYTE is manageable and I have already moved it.

Removing? You mean switching to something else in all those headers?

Quoted 4 lines

I will
be pushing a patch that moves just that into another file called wordpart.h. There are too many instances of the other functions for it to make sense to remove them all in this patch.

Okay, let's see the proposal (patch) code then!

Signature · 2 lines
With Best Regards,
Andy Shevchenko
Tested-by on v2 2/2
#14 lore raw

Re: [PATCH v2 2/2] lib/string: shrink lib/string.i via IWYU

In reply to [email protected]

Hi,

Quoted 9 lines · On Thu, Dec 14, 2023 at 09:06:13PM +0000, [email protected] wrote:

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to lib/string.c resulted in a preprocessed size of lib/string.i from 26371 lines to 5259 lines (-80%) for the x86 defconfig.

Tanzir, I wonder if you could include some of the symbol names used for some of these more esoteric headers. Let me describe what I mean:

Andy talks about "why <vdso/...>" and perhaps some comments (in your patch message, not in the source itself) about which symbols are being used from these headers would serve useful. I believe IWYU can generate this information and should clear up some confusion or lead to better suggestions from reviewers if we understand why a header is being included.

At any rate, this builds for me doing randconfigs on x86_64 with these KCONFIG_SEEDs
1: 0x3DD9D136
2: 0xB4440EE4
3: 0x98778270
4: 0x8C237F26
5: 0x244F8A64
6: 0x5A5C5E5C
7: 0xA77896BC
8: 0x9B5FF0D5
9: 0x24F23F6A
10: 0x35C0A107

I applied your patch on top of 5db8752c3b81bd33.

Quoted 37 lines
---
 lib/string.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
lib/string.c +7 -8
index be26623953d2..7fe1acefb1a1 100644
@@ -16,16 +16,16 @@
 
 #define __NO_FORTIFY
 #include <linux/types.h>
-#include <linux/string.h>
-#include <linux/ctype.h>
-#include <linux/kernel.h>
-#include <linux/export.h>
+#include <linux/bits.h>
 #include <linux/bug.h>
 #include <linux/errno.h>
-#include <linux/slab.h>
-
+#include <asm/rwonce.h>
+#include <linux/linkage.h>
+#include <linux/stddef.h>
+#include <vdso/limits.h>
+#include <linux/string.h>
+#include <linux/ctype.h>
 #include <asm/unaligned.h>
-#include <asm/byteorder.h>
 #include <asm/word-at-a-time.h>
 #include <asm/page.h>
 
 
--
Quoted 2 lines

2.43.0.472.g3155946c3a-goog

Thanks
Justin

#15 lore raw

Re: [PATCH v2 2/2] lib/string: shrink lib/string.i via IWYU

Quoted 14 lines · Justin Stitt
Quoted 9 lines · On Thu, Dec 14, 2023 at 09:06:13PM +0000, [email protected] wrote:

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list changing indirect includes to direct includes.
IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to lib/string.c resulted in a preprocessed size of lib/string.i from 26371 lines to 5259 lines (-80%) for the x86 defconfig.

Tanzir, I wonder if you could include some of the symbol names used for some of these more esoteric headers. Let me describe what I mean:

Quoted 6 lines

Andy talks about "why <vdso/...>" and perhaps some comments (in your patch message, not in the source itself) about which symbols are being used from these headers would serve useful. I believe IWYU can generate this information and should clear up some confusion or lead to better suggestions from reviewers if we understand why a header is being included.

If there aren't any other objections, I'd like to pick this up in my tree after the merge window closes.

Quoted 14 lines

At any rate, this builds for me doing randconfigs on x86_64 with these KCONFIG_SEEDs
1: 0x3DD9D136
2: 0xB4440EE4
3: 0x98778270
4: 0x8C237F26
5: 0x244F8A64
6: 0x5A5C5E5C
7: 0xA77896BC
8: 0x9B5FF0D5
9: 0x24F23F6A
10: 0x35C0A107

Quoted 3 lines

I applied your patch on top of 5db8752c3b81bd33.

Thanks for the testing!

-Kees

Quoted 41 lines
Quoted 37 lines
---
 lib/string.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
lib/string.c +7 -8
index be26623953d2..7fe1acefb1a1 100644
@@ -16,16 +16,16 @@
 
 #define __NO_FORTIFY
 #include <linux/types.h>
-#include <linux/string.h>
-#include <linux/ctype.h>
-#include <linux/kernel.h>
-#include <linux/export.h>
+#include <linux/bits.h>
 #include <linux/bug.h>
 #include <linux/errno.h>
-#include <linux/slab.h>
-
+#include <asm/rwonce.h>
+#include <linux/linkage.h>
+#include <linux/stddef.h>
+#include <vdso/limits.h>
+#include <linux/string.h>
+#include <linux/ctype.h>
 #include <asm/unaligned.h>
-#include <asm/byteorder.h>
 #include <asm/word-at-a-time.h>
 #include <asm/page.h>
 
 
--
Quoted 2 lines

2.43.0.472.g3155946c3a-goog

Thanks
Justin

Signature · 1 line
Kees Cook