Recommendation for activating a deferred module init in the kernel
[Resending due to reject from vger mail server. My apologies if you got this twice ]
Hi,
Quoted 5 lines · Tim Bird
I am working with a product team on bootup time issues. One technique that we are forward-porting from an old kernel (and that I thought I might work on mainlining) is to compile modules statically into the kernel, but defer their initialization until after boot time.
This may sound like a stupid question, but why are you compiling the modules statically?
I mean, it sounds to me like the perfect way to do what you want is to compile the modules dynamically (you can store them in a an in kernel initramfs if you want to keep them attached to the kernel binary for ease of maintenance) and simply call a script that calls modprobe when it's OK to load them?
Sounds a hell of lot simpler to me...
Gilad
Signature · 13 lines
Gilad Ben-Yossef Chief Coffee Drinker Codefidence Ltd. The code is free, your time isn't.(TM) Web: http://codefidence.com Email: [email protected] Office: +972-8-9316883 ext. 201 Fax: +972-8-9316885 Mobile: +972-52-8260388 Q: How many NSA agents does it take to replace a lightbulb? A: dSva7DrYiY24yeTItKyyogFXD5gRuoRqPNQ9v6WCLLywZPINlu!
Quoted 2 lines · Gilad Ben-Yossef
This may sound like a stupid question, but why are you compiling the modules statically?
I wondered that.
One potential reason to avoid modules is that they waste RAM -- you have to allocate an integral number of pages for each one, which means an average of 2KiB wasted for each module you load. Although that isn't much, it's not zero either. It might be possible to optimise that by 'packing' module allocations, if you're careful about it.
Also, on some architectures modules have historically been less efficient because when kernel and module text are too far apart in the virtual address space, you have to insert trampolines for all calls between them. I believe this used to be the case on ARM, but it got fixed by moving stuff around? It's only v850 which still does this AFAICT.
Do we actually manage to free inittext/initdata from modules on all architectures now? The comments in <linux/init.h> suggest not, but I think they lie -- it looks like like kernel/module.c will handle that entirely generically for sections named ".init*" -- it'll allocate space for those sections in a separate module_alloc() call, and free them itself. (Do bear this in mind, if you take the above suggestion about packing in module_alloc().)
So the only real reason I can see to avoid modules in the _current_ kernel would be the wasted RAM, which should be something we can address. Tim, have I missed something? What _were_ your reasons for avoiding modules, and can we do something about those instead of trying to defer the initialisation of in-kernel code?
Delaying init of certain modules seems like a poor man's substitute for a properly multi-threaded startup....
Signature · 1 line
dwmw2
Quoted 3 lines · David Woodhouse
So the only real reason I can see to avoid modules in the _current_ kernel would be the wasted RAM, which should be something we can address. Tim, have I missed something?
... like the time it takes to actually load modules and do the relocations, perhaps? You did say you were doing it to improve boot time. But this is the stuff you were happy to _defer_, so presumably not in your fast path?
If that _is_ a real concern, then given the observation¹ that most systems load exactly the same modules in exactly the same order every time they boot... has anyone looked at doing 'prelink' for modules?
Signature · 8 lines
dwmw2 ¹ which I just pulled out of my arse, admittedly. But I think it's true :) -- To unsubscribe from this list: send the line "unsubscribe linux-embedded" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · David Woodhouse
Quoted 2 lines · Gilad Ben-Yossef
This may sound like a stupid question, but why are you compiling the modules statically?
I wondered that.
Quoted 10 lines
One potential reason to avoid modules is that they waste RAM -- you have to allocate an integral number of pages for each one, which means an average of 2KiB wasted for each module you load. Although that isn't much, it's not zero either. It might be possible to optimise that by 'packing' module allocations, if you're careful about it.
...
So the only real reason I can see to avoid modules in the _current_ kernel would be the wasted RAM, which should be something we can address.
...
You miss the size increase imposed by CONFIG_MODULES=y.
E.g. setting CONFIG_MODULES=y in the arm collie_defconfig will increase the size of vmlinux by 14% (sic).
I haven't investigated why it takes that much space, but stuff like kernel/module.o taking 23kB and each EXPORT_SYMBOL requiring a few bytes simply cannot be completely eliminated.
Quoted 1 line
dwmw2
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 18 lines · Adrian Bunk
Quoted 6 lines · David Woodhouse
Quoted 2 lines · Gilad Ben-Yossef
This may sound like a stupid question, but why are you compiling the modules statically?
I wondered that.
Quoted 10 lines
One potential reason to avoid modules is that they waste RAM -- you have to allocate an integral number of pages for each one, which means an average of 2KiB wasted for each module you load. Although that isn't much, it's not zero either. It might be possible to optimise that by 'packing' module allocations, if you're careful about it.
...
So the only real reason I can see to avoid modules in the _current_ kernel would be the wasted RAM, which should be something we can address.
...
Quoted 8 lines
You miss the size increase imposed by
CONFIG_MODULES=y.E.g. setting
CONFIG_MODULES=yin the armcollie_defconfigwill increase the size of vmlinux by 14% (sic).I haven't investigated why it takes that much space, but stuff like
kernel/module.otaking 23kB and eachEXPORT_SYMBOLrequiring a few bytes simply cannot be completely eliminated.
Sounds like we need a tool that strips out the unneeded symbols, given a list of modules?
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453 Fax: +32 (0)2 700 8622 E-mail: [email protected] Internet: http://www.sony-europe.com/
Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V. Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB
Quoted 24 lines · Geert Uytterhoeven
Quoted 23 lines · Adrian Bunk
Quoted 6 lines · David Woodhouse
Quoted 2 lines · Gilad Ben-Yossef
This may sound like a stupid question, but why are you compiling the modules statically?
I wondered that.
Quoted 10 lines
One potential reason to avoid modules is that they waste RAM -- you have to allocate an integral number of pages for each one, which means an average of 2KiB wasted for each module you load. Although that isn't much, it's not zero either. It might be possible to optimise that by 'packing' module allocations, if you're careful about it.
...
So the only real reason I can see to avoid modules in the _current_ kernel would be the wasted RAM, which should be something we can address.
...You miss the size increase imposed by
CONFIG_MODULES=y.E.g. setting
CONFIG_MODULES=yin the armcollie_defconfigwill increase the size of vmlinux by 14% (sic).
Quoted 6 lines
Quoted 3 lines
I haven't investigated why it takes that much space, but stuff like
kernel/module.otaking 23kB and eachEXPORT_SYMBOLrequiring a few bytes simply cannot be completely eliminated.Sounds like we need a tool that strips out the unneeded symbols, given a list of modules?
And do the --gc-sections step again after that... :)
Signature · 1 line
dwmw2
Quoted 25 lines · David Woodhouse
Quoted 24 lines · Geert Uytterhoeven
Quoted 23 lines · Adrian Bunk
Quoted 6 lines · David Woodhouse
Quoted 2 lines · Gilad Ben-Yossef
This may sound like a stupid question, but why are you compiling the modules statically?
I wondered that.
Quoted 10 lines
One potential reason to avoid modules is that they waste RAM -- you have to allocate an integral number of pages for each one, which means an average of 2KiB wasted for each module you load. Although that isn't much, it's not zero either. It might be possible to optimise that by 'packing' module allocations, if you're careful about it.
...
So the only real reason I can see to avoid modules in the _current_ kernel would be the wasted RAM, which should be something we can address.
...You miss the size increase imposed by
CONFIG_MODULES=y.E.g. setting
CONFIG_MODULES=yin the armcollie_defconfigwill increase the size of vmlinux by 14% (sic).
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I haven't investigated why it takes that much space, but stuff like
kernel/module.otaking 23kB and eachEXPORT_SYMBOLrequiring a few bytes simply cannot be completely eliminated.Sounds like we need a tool that strips out the unneeded symbols, given a list of modules?
And do the --gc-sections step again after that... :)
But even after all optimizations CONFIG_MODULES=y will still cause a significant additional cost [1] when thinking in the dimensions of Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of reductions. For me, this is about 5% of my kernel size" statement in another thread.
Quoted 1 line
dwmw2
cu
Adrian
[1] as I said, kernel/module.o alone takes 23kB
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 5 lines · Adrian Bunk
But even after all optimizations
CONFIG_MODULES=ywill still cause a significant additional cost [1] when thinking in the dimensions of Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of reductions. For me, this is about 5% of my kernel size" statement in another thread.
Yes. If the system in question is currently being built without CONFIG_MODULES, that's fairly much a no-go for my alternative suggestion.
Signature · 1 line
dwmw2
Quoted 17 lines · Adrian Bunk
Quoted 7 lines · David Woodhouse
Quoted 6 lines · Geert Uytterhoeven
Quoted 5 lines · Adrian Bunk
You miss the size increase imposed by
CONFIG_MODULES=y.E.g. setting
CONFIG_MODULES=yin the armcollie_defconfigwill increase the size of vmlinux by 14% (sic).Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I haven't investigated why it takes that much space, but stuff like
kernel/module.otaking 23kB and eachEXPORT_SYMBOLrequiring a few bytes simply cannot be completely eliminated.Sounds like we need a tool that strips out the unneeded symbols, given a list of modules?
And do the --gc-sections step again after that... :)
Quoted 7 lines
But even after all optimizations
CONFIG_MODULES=ywill still cause a significant additional cost [1] when thinking in the dimensions of Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of reductions. For me, this is about 5% of my kernel size" statement in another thread.[1] as I said,
kernel/module.oalone takes 23kB
Which could be freed after the last module has been loaded?
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453 Fax: +32 (0)2 700 8622 E-mail: [email protected] Internet: http://www.sony-europe.com/
Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V. Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB
Quoted 24 lines · Geert Uytterhoeven
Quoted 23 lines · Adrian Bunk
Quoted 7 lines · David Woodhouse
Quoted 6 lines · Geert Uytterhoeven
Quoted 5 lines · Adrian Bunk
You miss the size increase imposed by
CONFIG_MODULES=y.E.g. setting
CONFIG_MODULES=yin the armcollie_defconfigwill increase the size of vmlinux by 14% (sic).Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I haven't investigated why it takes that much space, but stuff like
kernel/module.otaking 23kB and eachEXPORT_SYMBOLrequiring a few bytes simply cannot be completely eliminated.Sounds like we need a tool that strips out the unneeded symbols, given a list of modules?
And do the --gc-sections step again after that... :)
But even after all optimizations
CONFIG_MODULES=ywill still cause a significant additional cost [1] when thinking in the dimensions of Tim's "the 30 or so Linux-tiny patches that I use get me about 110k of reductions. For me, this is about 5% of my kernel size" statement in another thread.
Quoted 3 lines
Quoted 1 line
[1] as I said,
kernel/module.oalone takes 23kBWhich could be freed after the last module has been loaded?
Even if you free the module loading infrastructure afterwards, you still have to carry it in flash for both the kernel part and the user part. And the whole process (having to sort out module loading in user space, etc.) is still more complex than not having to load modules at all. Many embedded designs have fixed hardware configuration and purpose which makes dynamic module loading an unnecessary bloat.
Having the kernel span a parallel thread with a lower priority to initialize drivers could be a much simpler solution in terms of size and speed. This way the init process could run early, and devices would become available eventually just as if corresponding modules were loaded.
The only difficulty is to properly determine how to distinguish between drivers that have to be initialized early in order to mount the root fs and the others that can wait.
Nicolas