Skip to contentSkip to conversation

Slow loading big kernel module in 2.6 on PPC platform

DISCUSSION Wilson Li 4 participants 9 messages Sep 28, 2005 – Sep 30, 2005

lore
Share & tools
lore

Wilson Li

author
#0 lore raw

Hi,

I am trying to port several third party kernel modules from kernel 2.4 to 2.6 on a ppc (MPC824x) platform. For small size of modules, it works perfectly in 2.6. But there's one huge kernel module which size is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module starts. I did not notice there's such obvious delay in 2.4 kernel.

Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod. It didn't help. I also tried other things like disabling CONFIG_KALLSYMS and commenting out all the EXPORT_SYMBOLs in that module. Nothing works so far. I've not been able to find any relevant thread about slow loading of big kernel module on PPC platform.

Is this related to the new way of loading kernel module in 2.6 or vmalloc since the kernel module also needs contiguous memory? I am running 2.6.8 kernel on a ppc platform (MPC824x) with 24M bytes memory visible to kernel. Two small kernel modules are loaded first through shell command right after system boots up. And there are about 10M bytes free memory left before loading this big chunk. The memory seems big enough to me and memory is not that fragmented since it just boots up.

Any suggestions?

Thanks a lot,
Wilson Li

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

#1 lore raw
Quoted 9 lines · Wilson Li

Hi,

I am trying to port several third party kernel modules from kernel 2.4 to 2.6 on a ppc (MPC824x) platform. For small size of modules, it works perfectly in 2.6. But there's one huge kernel module which size is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module starts. I did not notice there's such obvious delay in 2.4 kernel.

I don't think it's a problem with the size. Here is the lspci output after I hacked a Rtc module to use 16 megabytes of data space. It took about 1/4 second to load (time insmod Rtc.ko).

Module                  Size  Used by
Rtc                 16783748  0
floppy                 58964  0
loop                   18440  0
parport_pc             28740  1
lp                     14472  0
parport                37320  2 parport_pc,lp

[SNIPPED...]

Quoted 7 lines

Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod. It didn't help. I also tried other things like disabling CONFIG_KALLSYMS and commenting out all the EXPORT_SYMBOLs in that module. Nothing works so far. I've not been able to find any relevant thread about slow loading of big kernel module on PPC platform.

The PPC might be a bit slower, but not as slow as you are seeing. I suspect that you have something that is 'waiting' for initialization.

Quoted 9 lines

Is this related to the new way of loading kernel module in 2.6 or vmalloc since the kernel module also needs contiguous memory? I am running 2.6.8 kernel on a ppc platform (MPC824x) with 24M bytes memory visible to kernel. Two small kernel modules are loaded first through shell command right after system boots up. And there are about 10M bytes free memory left before loading this big chunk. The memory seems big enough to me and memory is not that fragmented since it just boots up.

Quoted 4 lines

Any suggestions?

Thanks a lot,
Wilson Li

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

author
#2 lore raw
Quoted 40 lines · --- "linux-os (Dick Johnson)
Quoted 3 lines · Wilson Li

Hi,

I am trying to port several third party kernel modules from

kernel

Quoted 1 line

2.4 to 2.6 on a ppc (MPC824x) platform. For small size of

modules, it

Quoted 1 line

works perfectly in 2.6. But there's one huge kernel module which

size

Quoted 2 lines

is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module

starts.

Quoted 2 lines

I did not notice there's such obvious delay in 2.4 kernel.

I don't think it's a problem with the size. Here is the lspci output after I hacked a Rtc module to use 16 megabytes of data space. It took about 1/4 second to load (time insmod Rtc.ko).

Module                  Size  Used by
Rtc                 16783748  0
floppy                 58964  0
loop                   18440  0
parport_pc             28740  1
lp                     14472  0
parport                37320  2 parport_pc,lp

[SNIPPED...]

Quoted 2 lines

Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod.

It

Quoted 1 line

didn't help. I also tried other things like disabling

CONFIG_KALLSYMS

Quoted 2 lines

and commenting out all the EXPORT_SYMBOLs in that module. Nothing works so far. I've not been able to find any relevant thread

Quoted 7 lines

about

Quoted 2 lines

slow loading of big kernel module on PPC platform.

The PPC might be a bit slower, but not as slow as you are seeing. I suspect that you have something that is 'waiting' for initialization.

For debugging, my module init function will print a message first when it gets called. After insmod the module, the console hangs about 90 seconds for loading then my init function gets called and message prints. I even commented out all the initialization code, it still does not help. Anything else I am missing?

Thanks,
Wilson Li

______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/

#3 lore raw
Quoted 52 lines · Wilson Li
Quoted 40 lines · --- "linux-os (Dick Johnson)
Quoted 3 lines · Wilson Li

Hi,

I am trying to port several third party kernel modules from

kernel

Quoted 1 line

2.4 to 2.6 on a ppc (MPC824x) platform. For small size of

modules, it

Quoted 1 line

works perfectly in 2.6. But there's one huge kernel module which

size

Quoted 2 lines

is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module

starts.

Quoted 2 lines

I did not notice there's such obvious delay in 2.4 kernel.

I don't think it's a problem with the size. Here is the lspci output after I hacked a Rtc module to use 16 megabytes of data space. It took about 1/4 second to load (time insmod Rtc.ko).

Module                  Size  Used by
Rtc                 16783748  0
floppy                 58964  0
loop                   18440  0
parport_pc             28740  1
lp                     14472  0
parport                37320  2 parport_pc,lp

[SNIPPED...]

Quoted 2 lines

Initially I suspected there might be a problem of the insmod of busybox I was using. I switched to module-init-tools-3.1 insmod.

It

Quoted 1 line

didn't help. I also tried other things like disabling

CONFIG_KALLSYMS

Quoted 2 lines

and commenting out all the EXPORT_SYMBOLs in that module. Nothing works so far. I've not been able to find any relevant thread

Quoted 7 lines

about

Quoted 2 lines

slow loading of big kernel module on PPC platform.

The PPC might be a bit slower, but not as slow as you are seeing. I suspect that you have something that is 'waiting' for initialization.

Quoted 8 lines

For debugging, my module init function will print a message first when it gets called. After insmod the module, the console hangs about 90 seconds for loading then my init function gets called and message prints. I even commented out all the initialization code, it still does not help. Anything else I am missing?

Thanks,
Wilson Li

I have a module I am currently working on. It has some assembly stuff linked with it, so it was easy to modify. In the assembly I allocated 16 megabytes of static storage (using the .space keyword), first in the .data section, then in .rodata, then in .text. The .text section is where code exists. In no case did the module take more than 1/4 second to load. In all cases the size shown by lsmod reflected the enormous size of the module.

Now, that's static storage. If your code uses kmalloc() and friends to allocate a lot of storage when it is being loaded, then that's a different story. FYI, when you see a kernel message on the screen, that's not necessarily when it was "printed". It gets buffered, you know. If you want to time-check when various sections get control, to find out what's eating the time, then put the jiffie count into your kernel messages.

A simple macro using the __FUNCTION__ string and the jiffie count can go a long way towards finding out what's happening.

For instance, I once had a problem with continuous interrupts from a device, that couldn't be cleared, until the device was initialized. That slowed the system to a stand-still until I found that. The fix was easy, do some initialization before attaching the interrupt, at least enough to quiet the board. This board had an empty FPGA, whos bits need to be loaded with a bit-banger to make it work. The pin connected to an interrupt was just whatever-it-was-from-the-factory, before the intelligence was loaded. That system took about a minute for the first kernel message to be printed. Sometimes the system was very quiet <forever> and needed to be kicked.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

author
#4 lore raw
Quoted 57 lines

--- "linux-os (Dick Johnson)" <[email protected]> wrote:

On Wed, 28 Sep 2005, Wilson Li wrote:

I have a module I am currently working on. It has some assembly stuff linked with it, so it was easy to modify. In the assembly I allocated 16 megabytes of static storage (using the .space keyword),
first in the .data section, then in .rodata, then in .text. The .text section is where code exists. In no case did the module take more than 1/4 second to load. In all cases the size shown by lsmod
reflected the enormous size of the module.

Now, that's static storage. If your code uses kmalloc() and friends to allocate a lot of storage when it is being loaded, then that's a different story. FYI, when you see a kernel message on the screen, that's not necessarily when it was "printed". It gets buffered, you know. If you want to time-check when various sections get control, to find out what's eating the time, then put the jiffie count into your kernel messages.

A simple macro using the __FUNCTION__ string and the jiffie count can go a long way towards finding out what's happening.

For instance, I once had a problem with continuous interrupts from a device, that couldn't be cleared, until the device was initialized. That slowed the system to a stand-still until I found that. The fix was easy, do some initialization before attaching the interrupt, at least enough to quiet the board. This board had an empty FPGA, whos bits need to be loaded with a bit-banger to make it work. The pin connected to an interrupt was just whatever-it-was-from-the-factory, before the intelligence was loaded. That system took about a minute for the first kernel message to be printed. Sometimes the system was very quiet <forever> and needed to be kicked.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Quoted 1 line

Thank you.

Appreciate your advice and help. I finally tracked it down to the function called in init_module() in kernel/module.c.

/* Allow arches to frob section contents and sizes. */ err = module_frob_arch_sections(hdr, sechdrs, secstrings, mod);

which takes most of time (more then 99%) during module loading just as the reply from Semuel earlier.

Wilson Li

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

#5 lore raw
Quoted 62 lines · Wilson Li
Quoted 57 lines

--- "linux-os (Dick Johnson)" <[email protected]> wrote:

On Wed, 28 Sep 2005, Wilson Li wrote:

I have a module I am currently working on. It has some assembly stuff linked with it, so it was easy to modify. In the assembly I allocated 16 megabytes of static storage (using the .space keyword),
first in the .data section, then in .rodata, then in .text. The .text section is where code exists. In no case did the module take more than 1/4 second to load. In all cases the size shown by lsmod
reflected the enormous size of the module.

Now, that's static storage. If your code uses kmalloc() and friends to allocate a lot of storage when it is being loaded, then that's a different story. FYI, when you see a kernel message on the screen, that's not necessarily when it was "printed". It gets buffered, you know. If you want to time-check when various sections get control, to find out what's eating the time, then put the jiffie count into your kernel messages.

A simple macro using the __FUNCTION__ string and the jiffie count can go a long way towards finding out what's happening.

For instance, I once had a problem with continuous interrupts from a device, that couldn't be cleared, until the device was initialized. That slowed the system to a stand-still until I found that. The fix was easy, do some initialization before attaching the interrupt, at least enough to quiet the board. This board had an empty FPGA, whos bits need to be loaded with a bit-banger to make it work. The pin connected to an interrupt was just whatever-it-was-from-the-factory, before the intelligence was loaded. That system took about a minute for the first kernel message to be printed. Sometimes the system was very quiet <forever> and needed to be kicked.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips).
Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Quoted 1 line

Thank you.

Appreciate your advice and help. I finally tracked it down to the function called in init_module() in kernel/module.c.

Quoted 8 lines

/* Allow arches to frob section contents and sizes. */ err = module_frob_arch_sections(hdr, sechdrs, secstrings, mod);

which takes most of time (more then 99%) during module loading just as the reply from Semuel earlier.

Wilson Li

Yes. Look at the code in .../arch/ppc/kernel/module.c count_relocs() and you will see a quadratic (n ^2) example of how not to do something!

This probably could be fixed as (n * 2) order, two passes, however in the meantime, you might want to make as many symbols as you can in your code static. Because of the quadratic relationship, a small change in the number of symbols will make a large change in the load-time!

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#6 lore raw
In reply to Wilson Li
Quoted 8 lines · Wilson Li

Hi,

I am trying to port several third party kernel modules from kernel 2.4 to 2.6 on a ppc (MPC824x) platform. For small size of modules, it works perfectly in 2.6. But there's one huge kernel module which size is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module starts. I did not notice there's such obvious delay in 2.4 kernel.

How big is the module on disk? That is, what is the disk to memory transfer size. Really 2.7MB, or is there a lot of uninitialized storage?

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
author
#7 lore raw
Quoted 9 lines · --- Bill Davidsen
Quoted 3 lines · Wilson Li

Hi,

I am trying to port several third party kernel modules from

kernel

Quoted 1 line

2.4 to 2.6 on a ppc (MPC824x) platform. For small size of

modules, it

Quoted 1 line

works perfectly in 2.6. But there's one huge kernel module which

size

Quoted 10 lines
Quoted 2 lines

is about 2.7M bytes (size reported by lsmod after insmod), and it takes about 90 seconds to load this module before init_module

starts.

Quoted 1 line

I did not notice there's such obvious delay in 2.4 kernel.

How big is the module on disk? That is, what is the disk to memory transfer size. Really 2.7MB, or is there a lot of uninitialized storage?

Signature · 1 line
    -bill davidsen ([email protected])

The original module size on disk is around 3.3M bytes. Here's details of size.

text data bss dec hex filename
2025644 263244 213024 2501912 262d18 mrbig.ko

Thanks,
Wilson Li

______________________________________________________
Yahoo! for Good
Donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/

#8 lore raw
Quoted 6 lines

The original module size on disk is around 3.3M bytes. Here's details of size.

text data bss dec hex filename
2025644 263244 213024 2501912 262d18 mrbig.ko

wow that is a whole lot of GPL code ;)