Slow loading big kernel module in 2.6 on PPC platform
Hi,
I am trying to port several third party kernel modules from kernel 2.4 to 2.6 on a ppc (MPC824x) platform. For small size of modules, it works perfectly in 2.6. But there's one huge kernel module which size is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module starts. I did not notice there's such obvious delay in 2.4 kernel.
Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod. It didn't help. I also tried other things like disabling CONFIG_KALLSYMS and commenting out all the EXPORT_SYMBOLs in that module. Nothing works so far. I've not been able to find any relevant thread about slow loading of big kernel module on PPC platform.
Is this related to the new way of loading kernel module in 2.6 or vmalloc since the kernel module also needs contiguous memory? I am running 2.6.8 kernel on a ppc platform (MPC824x) with 24M bytes memory visible to kernel. Two small kernel modules are loaded first through shell command right after system boots up. And there are about 10M bytes free memory left before loading this big chunk. The memory seems big enough to me and memory is not that fragmented since it just boots up.
Any suggestions?
Thanks a lot,
Wilson Li
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
Quoted 9 lines · Wilson Li
Hi,
I am trying to port several third party kernel modules from kernel 2.4 to 2.6 on a ppc (MPC824x) platform. For small size of modules, it works perfectly in 2.6. But there's one huge kernel module which size is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before
init_modulestarts. I did not notice there's such obvious delay in 2.4 kernel.
I don't think it's a problem with the size. Here is the lspci output after I hacked a Rtc module to use 16 megabytes of data space. It took about 1/4 second to load (time insmod Rtc.ko).
Module Size Used by Rtc 16783748 0 floppy 58964 0 loop 18440 0 parport_pc 28740 1 lp 14472 0 parport 37320 2 parport_pc,lp
[SNIPPED...]
Quoted 7 lines
Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod. It didn't help. I also tried other things like disabling
CONFIG_KALLSYMSand commenting out all theEXPORT_SYMBOLsin that module. Nothing works so far. I've not been able to find any relevant thread about slow loading of big kernel module on PPC platform.
The PPC might be a bit slower, but not as slow as you are seeing. I suspect that you have something that is 'waiting' for initialization.
Quoted 9 lines
Is this related to the new way of loading kernel module in 2.6 or vmalloc since the kernel module also needs contiguous memory? I am running 2.6.8 kernel on a ppc platform (MPC824x) with 24M bytes memory visible to kernel. Two small kernel modules are loaded first through shell command right after system boots up. And there are about 10M bytes free memory left before loading this big chunk. The memory seems big enough to me and memory is not that fragmented since it just boots up.
Quoted 4 lines
Any suggestions?
Thanks a lot,
Wilson Li
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 40 lines · --- "linux-os (Dick Johnson)
Quoted 3 lines · Wilson Li
Hi,
I am trying to port several third party kernel modules from
kernel
Quoted 1 line
2.4 to 2.6 on a ppc (MPC824x) platform. For small size of
modules, it
Quoted 1 line
works perfectly in 2.6. But there's one huge kernel module which
size
Quoted 2 lines
is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before
init_modulestarts.
Quoted 2 lines
I did not notice there's such obvious delay in 2.4 kernel.
I don't think it's a problem with the size. Here is the
lspcioutput after I hacked a Rtc module to use 16 megabytes of data space. It took about 1/4 second to load (time insmod Rtc.ko).Module Size Used by Rtc 16783748 0 floppy 58964 0 loop 18440 0 parport_pc 28740 1 lp 14472 0 parport 37320 2 parport_pc,lp[SNIPPED...]
Quoted 2 lines
Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod.
It
Quoted 1 line
didn't help. I also tried other things like disabling
CONFIG_KALLSYMSQuoted 2 lines
and commenting out all the
EXPORT_SYMBOLsin that module. Nothing works so far. I've not been able to find any relevant thread
Quoted 7 lines
about
Quoted 2 lines
slow loading of big kernel module on PPC platform.
The PPC might be a bit slower, but not as slow as you are seeing. I suspect that you have something that is 'waiting' for initialization.
For debugging, my module init function will print a message first when it gets called. After insmod the module, the console hangs about 90 seconds for loading then my init function gets called and message prints. I even commented out all the initialization code, it still does not help. Anything else I am missing?
Thanks,
Wilson Li
______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/
Quoted 52 lines · Wilson Li
Quoted 40 lines · --- "linux-os (Dick Johnson)
Quoted 3 lines · Wilson Li
Hi,
I am trying to port several third party kernel modules from
kernel
Quoted 1 line
2.4 to 2.6 on a ppc (MPC824x) platform. For small size of
modules, it
Quoted 1 line
works perfectly in 2.6. But there's one huge kernel module which
size
Quoted 2 lines
is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before
init_modulestarts.
Quoted 2 lines
I did not notice there's such obvious delay in 2.4 kernel.
I don't think it's a problem with the size. Here is the
lspcioutput after I hacked a Rtc module to use 16 megabytes of data space. It took about 1/4 second to load (time insmod Rtc.ko).Module Size Used by Rtc 16783748 0 floppy 58964 0 loop 18440 0 parport_pc 28740 1 lp 14472 0 parport 37320 2 parport_pc,lp[SNIPPED...]
Quoted 2 lines
Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod.
It
Quoted 1 line
didn't help. I also tried other things like disabling
CONFIG_KALLSYMSQuoted 2 lines
and commenting out all the
EXPORT_SYMBOLsin that module. Nothing works so far. I've not been able to find any relevant threadQuoted 7 lines
about
Quoted 2 lines
slow loading of big kernel module on PPC platform.
The PPC might be a bit slower, but not as slow as you are seeing. I suspect that you have something that is 'waiting' for initialization.
Quoted 8 lines
For debugging, my module init function will print a message first when it gets called. After insmod the module, the console hangs about 90 seconds for loading then my init function gets called and message prints. I even commented out all the initialization code, it still does not help. Anything else I am missing?
Thanks,
Wilson Li
I have a module I am currently working on. It has some assembly stuff linked with it, so it was easy to modify. In the assembly I allocated 16 megabytes of static storage (using the .space keyword), first in the .data section, then in .rodata, then in .text. The .text section is where code exists. In no case did the module take more than 1/4 second to load. In all cases the size shown by lsmod reflected the enormous size of the module.
Now, that's static storage. If your code uses kmalloc() and friends to allocate a lot of storage when it is being loaded, then that's a different story. FYI, when you see a kernel message on the screen, that's not necessarily when it was "printed". It gets buffered, you know. If you want to time-check when various sections get control, to find out what's eating the time, then put the jiffie count into your kernel messages.
A simple macro using the __FUNCTION__ string and the jiffie count can go a long way towards finding out what's happening.
For instance, I once had a problem with continuous interrupts from a device, that couldn't be cleared, until the device was initialized. That slowed the system to a stand-still until I found that. The fix was easy, do some initialization before attaching the interrupt, at least enough to quiet the board. This board had an empty FPGA, whos bits need to be loaded with a bit-banger to make it work. The pin connected to an interrupt was just whatever-it-was-from-the-factory, before the intelligence was loaded. That system took about a minute for the first kernel message to be printed. Sometimes the system was very quiet <forever> and needed to be kicked.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 57 lines
--- "linux-os (Dick Johnson)" <[email protected]> wrote:
On Wed, 28 Sep 2005, Wilson Li wrote:
I have a module I am currently working on. It has some assembly stuff linked with it, so it was easy to modify. In the assembly I allocated 16 megabytes of static storage (using the .space keyword),
first in the .data section, then in .rodata, then in .text. The .text section is where code exists. In no case did the module take more than 1/4 second to load. In all cases the size shown bylsmod
reflected the enormous size of the module.Now, that's static storage. If your code uses
kmalloc()and friends to allocate a lot of storage when it is being loaded, then that's a different story. FYI, when you see a kernel message on the screen, that's not necessarily when it was "printed". It gets buffered, you know. If you want to time-check when various sections get control, to find out what's eating the time, then put the jiffie count into your kernel messages.A simple macro using the
__FUNCTION__string and the jiffie count can go a long way towards finding out what's happening.For instance, I once had a problem with continuous interrupts from a device, that couldn't be cleared, until the device was initialized. That slowed the system to a stand-still until I found that. The fix was easy, do some initialization before attaching the interrupt, at least enough to quiet the board. This board had an empty FPGA, whos bits need to be loaded with a bit-banger to make it work. The pin connected to an interrupt was just whatever-it-was-from-the-factory, before the intelligence was loaded. That system took about a minute for the first kernel message to be printed. Sometimes the system was very quiet <forever> and needed to be kicked.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Quoted 1 line
Thank you.
Appreciate your advice and help. I finally tracked it down to the function called in init_module() in kernel/module.c.
/* Allow arches to frob section contents and sizes. */ err = module_frob_arch_sections(hdr, sechdrs, secstrings, mod);
which takes most of time (more then 99%) during module loading just as the reply from Semuel earlier.
Wilson Li
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
Quoted 62 lines · Wilson Li
Quoted 57 lines
--- "linux-os (Dick Johnson)" <[email protected]> wrote:
On Wed, 28 Sep 2005, Wilson Li wrote:
I have a module I am currently working on. It has some assembly stuff linked with it, so it was easy to modify. In the assembly I allocated 16 megabytes of static storage (using the .space keyword),
first in the .data section, then in .rodata, then in .text. The .text section is where code exists. In no case did the module take more than 1/4 second to load. In all cases the size shown bylsmod
reflected the enormous size of the module.Now, that's static storage. If your code uses
kmalloc()and friends to allocate a lot of storage when it is being loaded, then that's a different story. FYI, when you see a kernel message on the screen, that's not necessarily when it was "printed". It gets buffered, you know. If you want to time-check when various sections get control, to find out what's eating the time, then put the jiffie count into your kernel messages.A simple macro using the
__FUNCTION__string and the jiffie count can go a long way towards finding out what's happening.For instance, I once had a problem with continuous interrupts from a device, that couldn't be cleared, until the device was initialized. That slowed the system to a stand-still until I found that. The fix was easy, do some initialization before attaching the interrupt, at least enough to quiet the board. This board had an empty FPGA, whos bits need to be loaded with a bit-banger to make it work. The pin connected to an interrupt was just whatever-it-was-from-the-factory, before the intelligence was loaded. That system took about a minute for the first kernel message to be printed. Sometimes the system was very quiet <forever> and needed to be kicked.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.Quoted 1 line
Thank you.
Appreciate your advice and help. I finally tracked it down to the function called in
init_module()inkernel/module.c.
Quoted 8 lines
/* Allow arches to frob section contents and sizes. */ err =
module_frob_arch_sections(hdr, sechdrs, secstrings, mod);which takes most of time (more then 99%) during module loading just as the reply from Semuel earlier.
Wilson Li
Yes. Look at the code in .../arch/ppc/kernel/module.c count_relocs() and you will see a quadratic (n ^2) example of how not to do something!
This probably could be fixed as (n * 2) order, two passes, however in the meantime, you might want to make as many symbols as you can in your code static. Because of the quadratic relationship, a small change in the number of symbols will make a large change in the load-time!
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 8 lines · Wilson Li
Hi,
I am trying to port several third party kernel modules from kernel 2.4 to 2.6 on a ppc (MPC824x) platform. For small size of modules, it works perfectly in 2.6. But there's one huge kernel module which size is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before
init_modulestarts. I did not notice there's such obvious delay in 2.4 kernel.
How big is the module on disk? That is, what is the disk to memory transfer size. Really 2.7MB, or is there a lot of uninitialized storage?
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 9 lines · --- Bill Davidsen
Quoted 3 lines · Wilson Li
Hi,
I am trying to port several third party kernel modules from
kernel
Quoted 1 line
2.4 to 2.6 on a ppc (MPC824x) platform. For small size of
modules, it
Quoted 1 line
works perfectly in 2.6. But there's one huge kernel module which
size
Quoted 10 lines
Quoted 2 lines
is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before
init_modulestarts.
Quoted 1 line
I did not notice there's such obvious delay in 2.4 kernel.
How big is the module on disk? That is, what is the disk to memory transfer size. Really 2.7MB, or is there a lot of uninitialized storage?
Signature · 1 line
-bill davidsen ([email protected])
The original module size on disk is around 3.3M bytes. Here's details of size.
text data bss dec hex filename
2025644 263244 213024 2501912 262d18 mrbig.ko
Thanks,
Wilson Li
______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/
Quoted 6 lines
The original module size on disk is around 3.3M bytes. Here's details of size.
text data bss dec hex filename
2025644 263244 213024 2501912 262d18 mrbig.ko
wow that is a whole lot of GPL code ;)