Boot time: Kernel start parallelization issue?
There are some reports [1] [2] looking at the boot time of embedded (ARM?) systems using initcall debug [3]. Both reports seem to show that they have issues with the start up sequence of the kernel being completely single-threaded. In [2] Greg mentions that on a x86 box multi-threads are happening and that there he doesn't see this issue. On the other hand, both reports mention Arjan's async initcall patches [4] to help against the issue. I.e. introducing some parallelization (on ARM) does help, too.
With this, I wonder
- if anybody faces similar issues with single-threaded only kernel start on embedded (ARM?) systems? Or if this is known? Or if there are fixes for this?
- if we somehow should try to 're-activate' Arjan's async initcall patches?
Any ideas?
Many thanks and best regards
Dirk
P.S.: I couldn't find a mail address of Alex Gonzalez, the author of [1], to put him into CC. If anybody knows him, please feel free to add him to this thread. Thanks!
[1] http://www.lindusembedded.com/blog/2010/06/02/measuring-the-boot-time-of-an-embedded-linux-device/
[2] http://thread.gmane.org/gmane.linux.usb.general/41181 (mainly the last mail of this thread: http://article.gmane.org/gmane.linux.usb.general/41619)
Quoted 10 lines · Dirk Behme
There are some reports [1] [2] looking at the boot time of embedded (ARM?) systems using initcall debug [3]. Both reports seem to show that they have issues with the start up sequence of the kernel being completely single-threaded. In [2] Greg mentions that on a x86 box multi-threads are happening and that there he doesn't see this issue. On the other hand, both reports mention Arjan's async initcall patches [4] to help against the issue. I.e. introducing some parallelization (on ARM) does help, too.
With this, I wonder
- if anybody faces similar issues with single-threaded only kernel
start on embedded (ARM?) systems? Or if this is known? Or if there are fixes for this?- if we somehow should try to 're-activate' Arjan's async initcall patches?
those patches are obsoleted by the merged async_schedule() calls.
what kernel are you seeing issues on?
if it's something ancient, you should update first.
Quoted 13 lines · Arjan van de Ven
Quoted 12 lines · Dirk Behme
There are some reports [1] [2] looking at the boot time of embedded (ARM?) systems using initcall debug [3]. Both reports seem to show that they have issues with the start up sequence of the kernel being completely single-threaded. In [2] Greg mentions that on a x86 box multi-threads are happening and that there he doesn't see this issue. On the other hand, both reports mention Arjan's async initcall patches [4] to help against the issue. I.e. introducing some parallelization (on ARM) does help, too.
With this, I wonder
Quoted 8 lines
Quoted 6 lines
- if anybody faces similar issues with single-threaded only kernel start on embedded (ARM?) systems? Or if this is known? Or if there are fixes for this?
- if we somehow should try to 're-activate' Arjan's async initcall patches?
those patches are obsoleted by the merged
async_schedule()calls.
Any link (patch name, git link) to be able to check for async_schedule() calls?
Quoted 1 line
what kernel are you seeing issues on?
[1] talks about 2.6.28, [2] talks about 2.6.34.
Best regards
Dirk
[1] http://www.lindusembedded.com/blog/2010/06/02/measuring-the-boot-time-of-an-embedded-linux-device/
[2] http://thread.gmane.org/gmane.linux.usb.general/41181 (mainly the last mail of this thread: http://article.gmane.org/gmane.linux.usb.general/41619)
Quoted 18 lines · Dirk Behme
Quoted 17 lines · Arjan van de Ven
Quoted 16 lines · Dirk Behme
There are some reports [1] [2] looking at the boot time of embedded (ARM?) systems using initcall debug [3]. Both reports seem to show that they have issues with the start up sequence of the kernel being completely single-threaded. In [2] Greg mentions that on a x86 box multi-threads are happening and that there he doesn't see this issue. On the other hand, both reports mention Arjan's async initcall patches [4] to help against the issue. I.e. introducing some parallelization (on ARM) does help, too.
With this, I wonder
- if anybody faces similar issues with single-threaded only kernel start on embedded (ARM?) systems? Or if this is known? Or if there are fixes for this?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
- if we somehow should try to 're-activate' Arjan's async initcall patches?
those patches are obsoleted by the merged
async_schedule()calls.Any link (patch name, git link) to be able to check for
async_schedule()calls?
Do you talk about
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=22a9d645677feefd402befd02edd59b122289ef1
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4ace92fc112c6069b4fcb95a31d3142d4a43ff2a
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=793180570ff2530d133343ceea85648de5f01b02
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f29d3b23238e1955a8094e038c72546e99308e61
http://lwn.net/Articles/314808/
merged with 2.6.29?
With a recent .37 kernel the usage of async_schedule() seems to be implemented for
./drivers/acpi/battery.c./drivers/s390/block/dasd.c./drivers/base/power/main.c./drivers/ata/libata-core.c./drivers/scsi/sd.c./drivers/md/raid5.c
With this, on an embedded system using none of this, the completely single-threaded start up reported in [2] seems to be reasonable, then?
This would mean that to improve the issues reported in [2], async_schedule() has to be implemented for the sub systems used, there, too? I.e. the USB init?
Quoted 3 lines
Quoted 1 line
what kernel are you seeing issues on?
[1] talks about 2.6.28, [2] talks about 2.6.34.
Thanks
Dirk
Quoted 7 lines
[1]
http://www.lindusembedded.com/blog/2010/06/02/measuring-the-boot-time-of-an-embedded-linux-device/[2] http://thread.gmane.org/gmane.linux.usb.general/41181 (mainly the last mail of this thread:
http://article.gmane.org/gmane.linux.usb.general/41619)
Quoted 9 lines · Dirk Behme
Do you talk about
Quoted 8 lines
http://lwn.net/Articles/314808/
merged with 2.6.29?
With a recent .37 kernel the usage of
async_schedule()seems to be implemented for
yep
Quoted 9 lines
./drivers/acpi/battery.c./drivers/s390/block/dasd.c./drivers/base/power/main.c./drivers/ata/libata-core.c./drivers/scsi/sd.c./drivers/md/raid5.cWith this, on an embedded system using none of this, the completely single-threaded start up reported in [2] seems to be reasonable, then?
quite likely.
Quoted 2 lines
This would mean that to improve the issues reported in [2],
async_schedule()has to be implemented for the sub systems used, there, too? I.e. the USB init?
USB init is asynchronous internal to USB; this was one of the outcomes of a side discussion at the Kernel Summit a year and a half ago or so where Alan Stern visualized the problem... and solved it internal to USB.
Now, there is one 100 msec delay that is inherent to USB (mandatory delay required by the spec for voltage to stabilize) but that shouldn't be in the synchronous path anymore
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
what kernel are you seeing issues on?
[1] talks about 2.6.28, [2] talks about 2.6.34.
yeah 2.6.28 is a lost cause ;-) 2.6.34 should be not too bad.
Now, the one thing that's critical for all of this is that we need to do this data driven. Using async init is not always as easy as it sounds; there are device ordering things that need to be dealt with, and failure cases get much more complex as well.
So the first order of business ALWAYS is to get a bootgraph (from scripts/bootgraph.pl)... to see which things are big on the one hand, but also to see if there is gains to be had. There is absolutely no gain possible (in terms of making something async) if it is pretty much the last big thing in the boot that you are making asynchronous (since the end of kernel boot has to wait for it all anyway); there is only gains for async if the expensive operation, when made async, can run in parallel with something else expensive that comes later in the kernel init.
The bootgraph will show this opportunity / pitfall very clearly......