Skip to contentSkip to conversation

x86/vdso: shrink vdso/extable.i via IWYU

PATCH Superseded Tanzir Hasan 3 participants 7 messages Dec 28, 2023 – Jan 5, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/vdso: shrink vdso/extable.i via IWYU

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list, changing indirect includes to direct includes. IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to vdso/extable.c resulted in a preprocessed size of vdso/extable.i from 64332 lines to 45377 lines (-27%) for the x86 defconfig.

---
 arch/x86/entry/vdso/extable.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/entry/vdso/extable.c +6 -2
index afcf5b65beef..8221231917ec 100644
@@ -1,10 +1,14 @@
 // SPDX-License-Identifier: GPL-2.0
-#include <linux/err.h>
 #include <linux/mm.h>
+#include <linux/sched.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
 #include <asm/current.h>
-#include <asm/traps.h>
+#include <asm/trapnr.h>
 #include <asm/vdso.h>
 
+struct pt_regs;
+
 struct vdso_exception_table_entry {
 	int insn, fixup;
 };

---
base-commit: f5837722ffecbbedf1b1dbab072a063565f0dad1
change-id: 20231228-extable-18f095e0aeba

Best regards,

Signature · 1 line
Tanzir Hasan <[email protected]>
maintainer
#1 lore raw
Quoted 9 lines · Tanzir Hasan

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list, changing indirect includes to direct includes. IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to vdso/extable.c resulted in a preprocessed size of vdso/extable.i from 64332 lines to 45377 lines (-27%) for the x86 defconfig.

Maybe I'm missing something but this sounds like the build-time "improvement" is so minute that you have to look at preprocessed sizes.

Do you have any, more "noticeable" numbers to justify this churn?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#2 lore raw

Hi Boris,

Quoted 4 lines

Maybe I'm missing something but this sounds like the build-time "improvement" is so minute that you have to look at preprocessed sizes.

Do you have any, more "noticeable" numbers to justify this churn?

These changes are mainly targeted at reducing preprocessing size which is why it was brought up. You're not wrong about build times. The changes in build time are small. To build the original file as is, it takes .69 seconds on average on my
devices. This changes it to be .55 seconds on average so ~17%.

The intention with these changes is to add up the impact of these small changes over time to improve build time and also use more direct inclusions where possible.

Best,
Tanzir

maintainer
#3 lore raw
Quoted 3 lines · Tanzir Hasan

The intention with these changes is to add up the impact of these small changes over time to improve build time and also use more direct inclusions where possible.

The problem I see with such "changes" is that it'll be a never-ending stream of them because over time, headers get reworked, moved, changed, split, etc and one would have to do such reorganization again.

And I'd understand it if the benefits were higher than what you're seeing. But right now it looks like unnecessary churn.

Dunno, perhaps one could do those in one single patch for, say, on arch/x86/ or so, and show a before and after wrt numbers. And then do it again in a couple of releases, when it becomes necessary again.

Or one could run that tool on patches which haven't been applied yet, patches on lkml.

Patches which touch headers or add #include directives and then reply with an automatic report to them, saying something along the lines of

"your patch has a suboptimal include - if you include this and that, it'll improve build time by this and that"

and then people would incorporate those changes and right then and there.

But let's see what the others think first.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#4 lore raw
Quoted 9 lines · Borislav Petkov
Quoted 3 lines · Tanzir Hasan

The intention with these changes is to add up the impact of these small changes over time to improve build time and also use more direct inclusions where possible.

The problem I see with such "changes" is that it'll be a never-ending stream of them because over time, headers get reworked, moved, changed, split, etc and one would have to do such reorganization again.

By being more precise in what's necessary via making more specific use of includes (making "indirect" includes into "direct" includes) in the .c files, it will allow us to more easily refactor the header files themselves. Otherwise changing the .h files may necessitate updating multiple .c files to fix this anyways first.

And yes, these inclusion lists will change over time. As you say below, they can be cleaned up again every couple of releases. Though right now most files have never been cleaned up at all. If smaller translation units like arch/x86/entry/vdso/extable.c don't change often, I don't expect churn to those files even between multiple releases.

Quoted 2 lines

And I'd understand it if the benefits were higher than what you're seeing. But right now it looks like unnecessary churn.

Over time, they will add up. But not if we reject the patches as unnecessary churn.

Quoted 2 lines

Dunno, perhaps one could do those in one single patch for, say, on arch/x86/ or so, and show a before and after wrt numbers.

Maybe the overall numbers are interesting, but landing one patch that updates numerous .c files' inclusion lists seems error prone. I suspect it's more likely that a more incremental approach of smaller patches allows progress if there are any issues; a build failure/mistake doesn't block the whole thing from landing.

Overall numbers can also be collected after the fact.

Quoted 2 lines

And then do it
again in a couple of releases, when it becomes necessary again.

That's the use case I had in mind. Though I suspect the initial run of this tooling will result in the most changes, as some files in tree are hardly touched between releases. For those, I don't expect any automated tooling to be churning those files after the initial cleaning.

Quoted 11 lines

Or one could run that tool on patches which haven't been applied yet, patches on lkml.

Patches which touch headers or add #include directives and then reply with an automatic report to them, saying something along the lines of

"your patch has a suboptimal include - if you include this and that, it'll improve build time by this and that"

and then people would incorporate those changes and right then and there.

Yeah, the idea is that the tooling results in repeatable processes by others. One could imagine build bots running this, or integrating it into checkpatch, or git presubmit hooks, or w/e.

But Tanzir's internship ends Friday, so he won't be building such a service by then. Something for me to keep in mind though for sure.

Quoted 1 line

But let's see what the others think first.

I have some feedback on the v1; I'll post it inline in another reply. --
Thanks,
~Nick Desaulniers

maintainer
#5 lore raw
Quoted 4 lines · Nick Desaulniers

By being more precise in what's necessary via making more specific use of includes (making "indirect" includes into "direct" includes) in the .c files, it will allow us to more easily refactor the header files themselves.

I know, but you have to realize that header files are not a static thing
- we do "refactor" them all the time. So it's not like we'll refactor them and they'll be done. No, we'll refactor them again later.

Quoted 3 lines

And yes, these inclusion lists will change over time. As you say below, they can be cleaned up again every couple of releases. Though right now most files have never been cleaned up at all.

Yap, the need for some tool or facility to keep include files proper and optimal is indisputable. There's a reason it is called the "Include Hell".

Quoted 2 lines

Over time, they will add up. But not if we reject the patches as unnecessary churn.

There's the other side of that coin: if we keep applying those, it'll turn into a never-ending stream of silly fixes. Like spellchecking and whitespace cleanups. And maintainers will start ignoring them.

Quoted 7 lines

Maybe the overall numbers are interesting, but landing one patch that updates numerous .c files' inclusion lists seems error prone. I suspect it's more likely that a more incremental approach of smaller patches allows progress if there are any issues; a build failure/mistake doesn't block the whole thing from landing.

Overall numbers can also be collected after the fact.

You could do a single patch *set* and say, before and after, we get these improvements. And the whole set goes in in one fell swoop.

Quoted 8 lines
Quoted 2 lines

And then do it
again in a couple of releases, when it becomes necessary again.

That's the use case I had in mind. Though I suspect the initial run of this tooling will result in the most changes, as some files in tree are hardly touched between releases. For those, I don't expect any automated tooling to be churning those files after the initial cleaning.

That's why I said "when it becomes necessary". :)

Quoted 3 lines

Yeah, the idea is that the tooling results in repeatable processes by others. One could imagine build bots running this, or integrating it into checkpatch, or git presubmit hooks, or w/e.

Yap, that would be best. Because it'll stop from the whole include hell from even ensuing in the first place.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#6 lore raw
In reply to Tanzir Hasan
Quoted 21 lines · Tanzir Hasan

This diff uses an open source tool include-what-you-use (IWYU) to modify the include list, changing indirect includes to direct includes. IWYU is implemented using the IWYUScripts github repository which is a tool that is currently undergoing development. These changes seek to improve build times.

This change to vdso/extable.c resulted in a preprocessed size of vdso/extable.i from 64332 lines to 45377 lines (-27%) for the x86 defconfig.

---
 arch/x86/entry/vdso/extable.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/entry/vdso/extable.c +0 -0
index afcf5b65beef..8221231917ec 100644

arch/x86/entry/vdso/extable.c line 2

Quoted 3 lines
arch/x86/entry/vdso/extable.c +0 -1
@@ -1,10 +1,14 @@
 // SPDX-License-Identifier: GPL-2.0
-#include <linux/err.h>
 #include <linux/mm.h>

^ I think we can get more aggressive about avoiding linux/mm.h even. I think you could replace linux/mm.h with

#include <linux/mm_types.h> // for mm_struct, mm_struct::(anonymous)
#include <asm/mmu.h> // for mm_context_t

assuming it's ok to include asm/* files in .c files under arch/* (more below).

arch/x86/entry/vdso/extable.c line 11

Quoted 10 lines
arch/x86/entry/vdso/extable.c +6 -1
+#include <linux/sched.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
 #include <asm/current.h>
-#include <asm/traps.h>
+#include <asm/trapnr.h>
 #include <asm/vdso.h>
 
+struct pt_regs;
+

^ This forward declaration shouldn't be necessary and looks wrong; pt_regs is not used as an opaque type in this TU.

What I suspect is happening:
1. debug output from your tooling seems to think pt_regs should come from linux/ptrace.h.
2. your tooling has a rule to replace recommendations for asm/ptrace.h with linux/ptrace.h. I may have misunderstood Al's point in an earlier thread; perhaps it's ok to use asm/*.h in .c files under arch/*/, but linux/*.h should be used everywhere else? If that's the case then perhaps we can break up that table such that asm-generic/*.h -> asm/*.h is one table that's always used, and asm/*.h -> linux/*.h is another table that's conditionally used when the .c file is not under arch/*/?

To avoid pinging maintainers too many times, why don't we do an internal code review for v2 before sending it publicly? I appreciate you making progress by sending a v1 while I was on holiday; for v2, let's review internally first.

Signature · 2 lines
Thanks,
~Nick Desaulniers