Modularizing IOMMUs (devel/iommu-0.4)
Hey Peter, Tomo-san, Jan and Jeremy, and LKML:
I was wondering if I could run this idea by you guys. One of the things that Peter proposed was in some ways to "modularize" the IOMMUs (and potentially later on use that framework to "modularize" other built-in components). For details:
http://lkml.org/lkml/2010/8/2/282 and the follow-on threads.
If you want to look at code right away, I've some beginnings at:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git devel/iommu-0.4 (look at patches past swiotlb-xen: Remove the unnecessary EXPORT_SYMBOL_GPL macros)
The 10,000 ft view is that during bootup, we want to jettison those IOMMUs that are built-in, but actually aren't used b/c the machine does not have it. Once it is identified which one we don't want, we mark those sections as available for the kernel memory subsystem and we can shed some kilobytes.
So the first step is to identify the functions for the IOMMUs. And also group them tightly together. The path I pursued was to stick all of the executable sections in clearly identifiable sections that can be easily identified.
The first idea I had that was to define in vmlinux.ld.S this little macro:
.PAGE_ALIGN(PAGE_SIZE) = .;
.iommu_text: {
PROVIDE_HIDDEN(__iommu_swiotlb_start) := .;
swiotlb*.o(*.text)
PROVIDE_HIDDEN(__iommu_gart_start) := .;
pci-gart_64.o(*.text)
..
} = 0x9090
So essentially renaming .text section of specific object files to be instead .iommu_text section.
Well, that did not work as during linking GCC makes the object files in /tmp, so instead of pci-gart_64.o we get /tmp/f892343.o. So question #1: Do you know of any mechanism to enforce the naming of the object files? Or perhaps another way to identify in the linker script file the <object-file><.text section> ?
Another way, less elegant was to manually enforce each function to be stuck in the .iommu_text section. For that I made a macro: __iommu that would force the function to be stuck in section specific for the IOMMU. So all functions in pci-gart_64.c would be funneled in .iommu.gart.text. For calgary:
.iommu.calgary.text., and so on. This is accomplished by having at the beginning of the the file the name of the IOMMU section, as so:
#define IOMMU_MODULE "gart"
And all of the functions would get stuck in .iommu.gart.text. This actually worked out (hadn't tried to boot yet) and based on the compilation these are the sizes:
1 .iommu.gart.text 00000be4 ..sip. 0000000001436fe2 00636fe2 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
2 .iommu.calgary.text 00000c94 ..snip.. 0000000001437bc6 00637bc6 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
3 .iommu.amdvi.text 00002d74 ..snip.. 000000000143885a 0063885a 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
4 .iommu.dmar.text 00004ff7 ..snip.. 000000000143b5ce 0063b5ce 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
5 .iommu.xen-swiotlb.text 000005f3 ..snip. 00000000014405c5 006405c5 2**0
For those that can't read nativly hexadecimal, here are the sizes:
Xen-SWIOTLB: 1523
GART: 3044 bytes
Calgary: 3220 bytes
AMD V-I: 11636
Intel DMAR: 20471.
Roughly it comes out to 40kB of executable. I haven't run the math, but estimating this it looks as if on AMD machine we can dump about eight pages (VT-D, Calgary, Xen-SWIOTBL), while on Intel with VT-D we can dump six pages (GART, Calgary, AMD VI, Xen-SWIOTLB), and on Intel with non-VT, around eleven pages (all IOMMUs, just stitch with the bare-metal SWIOTLB). Each .iommu.<x>.text would be PAGE_ALIGN-ed of course.
The next stage (not yet written) would have to:
- scan the EXPORT_SYMBOL definitions and see if any of them map within the sections that are to be discarded. If so, either neuter them or don't discard the IOMMU.
- Mark the discarded sections to be freed. And utilize the framework for doing so.
- Give the same treatment as the _stext -> _etext sections have. So setting of the various flags, etc.
But the Question #2 I have is: is it worth it? We can save on average around eight pages (6+7+11/3), which is not exactly earth-shattering. This could be become more generic so in the long run it could jettison more code and not just IOMMUs, but lets start small first.
And then Question #3): Is there a better way?
Quoted 17 lines · Konrad Rzeszutek Wilk
Hey Peter, Tomo-san, Jan and Jeremy, and LKML:
I was wondering if I could run this idea by you guys. One of the things that Peter proposed was in some ways to "modularize" the IOMMUs (and potentially later on use that framework to "modularize" other built-in components). For details:
http://lkml.org/lkml/2010/8/2/282 and the follow-on threads.If you want to look at code right away, I've some beginnings at:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.gitdevel/iommu-0.4 (look at patches past swiotlb-xen: Remove the unnecessaryEXPORT_SYMBOL_GPLmacros)The 10,000 ft view is that during bootup, we want to jettison those IOMMUs that are built-in, but actually aren't used b/c the machine does not have it. Once it is identified which one we don't want, we mark those sections as available for the kernel memory subsystem and we can shed some kilobytes.
Quoted 5 lines
So the first step is to identify the functions for the IOMMUs. And also group them tightly together. The path I pursued was to stick all of the executable sections in clearly identifiable sections that can be easily identified.
I think a much better way to do this is something I discussed with Linus at least 12 years ago, which is allow modules to be built into the kernel, but still be unloadable using the actual module mechanism. This means they are part of the full kernel image and therefore immediately available, but appear to the module system as an already-installed module. Then we can unload modules to free up space.
This would be a handy thing to have in general, and would simplify this kind of stuff tremendously.
-hpa
Quoted 10 lines
Quoted 1 line
Quoted 1 line · Konrad Rzeszutek Wilk
Another way, less elegant was to manually enforce each function to be stuck in the .iommu_text section. For that I made a macro:
__iommuthat would force the function to be stuck in section specific for the IOMMU. So all functions inpci-gart_64.cwould be funneled in .iommu.gart.text. For calgary:
.iommu.calgary.text., and so on. This is accomplished by having at the beginning
of the the file the name of the IOMMU section, as so:
Quoted 3 lines
#define
IOMMU_MODULE"gart"And all of the functions would get stuck in .iommu.gart.text.
For this particular approach - did you consider using objcopy's --rename-section option?
Quoted 1 line
And then Question #3): Is there a better way?
Generally I like hpa's suggestion of using pre-loaded but unloadable modules much better, not the least because in the past I had seen potential uses for such a mechanism in other places of the kernel.
Jan
Quoted 10 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · Konrad Rzeszutek Wilk
Another way, less elegant was to manually enforce each function to be stuck in the .iommu_text section. For that I made a macro:
__iommuthat would force the function to be stuck in section specific for the IOMMU. So all functions inpci-gart_64.cwould be funneled in .iommu.gart.text. For calgary:
.iommu.calgary.text., and so on. This is accomplished by having at the beginning
of the the file the name of the IOMMU section, as so:
Quoted 6 lines
Quoted 3 lines
#define
IOMMU_MODULE"gart"And all of the functions would get stuck in .iommu.gart.text.
For this particular approach - did you consider using objcopy's --rename-section option?
<goes off to look at that>
Quoted 6 lines
Quoted 2 lines
And then Question #3): Is there a better way?
Generally I like hpa's suggestion of using pre-loaded but unloadable modules much better, not the least because in the past I had seen potential uses for such a mechanism in other places of the kernel.
<nods> It does sound like the right option. 12 years ago it was suggested, so.. how come nobody worked on it in the past? Were there some epic battles?
Quoted 2 lines · Konrad Rzeszutek Wilk
<nods> It does sound like the right option. 12 years ago it was suggested, so.. how come nobody worked on it in the past? Were there some epic battles?
No, quite the opposite... deafening silence. However, 12 years ago the module infrastructure looked very different than it does today, with large chunks implemented in user space. These days the module infrastructure is almost entirely in the kernel, which should make it a lot easier to implement.
I thus suspect that it was just too hard when it was first proposed, and then got forgotten about.
-hpa