Skip to contentSkip to conversation

What's taking so long??

DISCUSSION Gary Thomas 6 participants 16 messages Jan 20, 2011 – Feb 8, 2011

lore
Share & tools
lore
#0 lore raw

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build & package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

  compile            1m49s
  deploy             0m16s
  populate_sysroot   0m38s

package 23m20s

Just to compare, using OTS Poky for BeagleBoard:

  compile            5m07s
  deploy             0m36s
  populate_sysroot   0m29s
  package            5m02s

That's a five-fold difference in the time to build & package the kernel.

What am I doing [wrong?] that makes this so?
Would it be better if I switched to RPM like the BB is using?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#1 lore raw
Quoted 15 lines · Gary Thomas

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build & package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

   compile            1m49s
   deploy             0m16s
   populate_sysroot   0m38s

package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:

   compile            5m07s
   deploy             0m36s
   populate_sysroot   0m29s
   package            5m02s

That's a five-fold difference in the time to build & package the kernel.

This is a significant performance impact. Thanks for taking the time to do some profiling.

I've filed a bug http://bugzilla.pokylinux.org/show_bug.cgi?id=666 and we'll get someone looking into that as a matter of priority.

Quoted 4 lines

What am I doing [wrong?] that makes this so?
Would it be better if I switched to RPM like the BB is using?

It looks like the slow paths are in package format independent code, so probably not.

Regards,
Joshua

Signature · 2 lines
Joshua Lock
        Intel Open Source Technology Centre
#2 lore raw
Quoted 32 lines · Joshua Lock
Quoted 15 lines · Gary Thomas

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build& package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

    compile            1m49s
    deploy             0m16s
    populate_sysroot   0m38s

package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:

    compile            5m07s
    deploy             0m36s
    populate_sysroot   0m29s
    package            5m02s

That's a five-fold difference in the time to build& package the kernel.

This is a significant performance impact. Thanks for taking the time to do some profiling.

I've filed a bug http://bugzilla.pokylinux.org/show_bug.cgi?id=666 and we'll get someone looking into that as a matter of priority.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

What am I doing [wrong?] that makes this so?
Would it be better if I switched to RPM like the BB is using?

It looks like the slow paths are in package format independent code, so probably not.

Thanks. I wonder what has changed as this used to be quite snappy!

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
In reply to Joshua Lock

Op 20 jan 2011, om 16:58 heeft Joshua Lock het volgende geschreven:

Quoted 32 lines · Gary Thomas
Quoted 15 lines

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build & package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

  compile            1m49s
  deploy             0m16s
  populate_sysroot   0m38s

package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:

  compile            5m07s
  deploy             0m36s
  populate_sysroot   0m29s
  package            5m02s

That's a five-fold difference in the time to build & package the kernel.

This is a significant performance impact. Thanks for taking the time to do some profiling.

I've filed a bug http://bugzilla.pokylinux.org/show_bug.cgi?id=666 and we'll get someone looking into that as a matter of priority.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

What am I doing [wrong?] that makes this so?
Would it be better if I switched to RPM like the BB is using?

It looks like the slow paths are in package format independent code, so probably not.

From a chat I had with RP 2 weeks ago:

Koen: there's still something slow in bitbake log output
Koen: do_package takes eons (> 10mins) with almost zero IO and CPU
Koen: ctrl-c'ing it shows it's in the installed-but-not-packaged output Richard Purdie: poky's bitbake or upstream?
Koen: I only tried poky bitbake so far
Koen: upstream is slow as well, but I don't recall being so slow
Koen: we talked about it at ELC
Richard Purdie: My memory isn't remembering that
Richard Purdie: This is do_package for the kernel or something large?
Koen: kernel
Koen: but I think anything would trigger it
Koen: it feels like bitbake is only doing N lines of output per second
Koen: effectively throttling it
Koen: and with you have a 4k lines out output....
Richard Purdie: The message handling means thread context switching Richard Purdie: in runqueue.py there is a number 4096 and a 0 on the line below Richard Purdie: try changing the 0 for something like 100000 Richard Purdie: You won't get nice console output but it might run faster Richard Purdie: Was poky running "faster" until recently?
Koen: I think so
Koen: I haven't measured it

Does doing:

Richard Purdie: in runqueue.py there is a number 4096 and a 0 on the line below Richard Purdie: try changing the 0 for something like 100000

help?

regards,

Koen

#4 lore raw
Quoted 66 lines · Koen Kooi

Op 20 jan 2011, om 16:58 heeft Joshua Lock het volgende geschreven:

Quoted 32 lines · Gary Thomas
Quoted 15 lines

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build& package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

   compile            1m49s
   deploy             0m16s
   populate_sysroot   0m38s

package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:

   compile            5m07s
   deploy             0m36s
   populate_sysroot   0m29s
   package            5m02s

That's a five-fold difference in the time to build& package the kernel.

This is a significant performance impact. Thanks for taking the time to do some profiling.

I've filed a bug http://bugzilla.pokylinux.org/show_bug.cgi?id=666 and we'll get someone looking into that as a matter of priority.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

What am I doing [wrong?] that makes this so?
Would it be better if I switched to RPM like the BB is using?

It looks like the slow paths are in package format independent code, so probably not.

From a chat I had with RP 2 weeks ago:

Koen: there's still something slow in bitbake log output
Koen: do_package takes eons (> 10mins) with almost zero IO and CPU
Koen: ctrl-c'ing it shows it's in the installed-but-not-packaged output Richard Purdie: poky's bitbake or upstream?
Koen: I only tried poky bitbake so far
Koen: upstream is slow as well, but I don't recall being so slow
Koen: we talked about it at ELC
Richard Purdie: My memory isn't remembering that
Richard Purdie: This is do_package for the kernel or something large?
Koen: kernel
Koen: but I think anything would trigger it
Koen: it feels like bitbake is only doing N lines of output per second
Koen: effectively throttling it
Koen: and with you have a 4k lines out output....
Richard Purdie: The message handling means thread context switching Richard Purdie: in runqueue.py there is a number 4096 and a 0 on the line below Richard Purdie: try changing the 0 for something like 100000 Richard Purdie: You won't get nice console output but it might run faster Richard Purdie: Was poky running "faster" until recently?
Koen: I think so
Koen: I haven't measured it

Quoted 6 lines

Does doing:

Richard Purdie: in runqueue.py there is a number 4096 and a 0 on the line below Richard Purdie: try changing the 0 for something like 100000

help?

No.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#5 lore raw
Quoted 67 lines · Gary Thomas
Quoted 66 lines · Koen Kooi

Op 20 jan 2011, om 16:58 heeft Joshua Lock het volgende geschreven:

Quoted 32 lines · Gary Thomas
Quoted 15 lines

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build& package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:
compile 1m49s
deploy 0m16s
populate_sysroot 0m38s
package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:
compile 5m07s
deploy 0m36s
populate_sysroot 0m29s
package 5m02s

That's a five-fold difference in the time to build& package the kernel.

This is a significant performance impact. Thanks for taking the time to do some profiling.

I've filed a bug http://bugzilla.pokylinux.org/show_bug.cgi?id=666 and we'll get someone looking into that as a matter of priority.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

What am I doing [wrong?] that makes this so?
Would it be better if I switched to RPM like the BB is using?

It looks like the slow paths are in package format independent code, so probably not.

From a chat I had with RP 2 weeks ago:

Koen: there's still something slow in bitbake log output
Koen: do_package takes eons (> 10mins) with almost zero IO and CPU
Koen: ctrl-c'ing it shows it's in the installed-but-not-packaged output Richard Purdie: poky's bitbake or upstream?
Koen: I only tried poky bitbake so far
Koen: upstream is slow as well, but I don't recall being so slow
Koen: we talked about it at ELC
Richard Purdie: My memory isn't remembering that
Richard Purdie: This is do_package for the kernel or something large?
Koen: kernel
Koen: but I think anything would trigger it
Koen: it feels like bitbake is only doing N lines of output per second
Koen: effectively throttling it
Koen: and with you have a 4k lines out output....
Richard Purdie: The message handling means thread context switching Richard Purdie: in runqueue.py there is a number 4096 and a 0 on the line below Richard Purdie: try changing the 0 for something like 100000 Richard Purdie: You won't get nice console output but it might run faster Richard Purdie: Was poky running "faster" until recently?
Koen: I think so
Koen: I haven't measured it

Quoted 9 lines
Quoted 6 lines

Does doing:

Richard Purdie: in runqueue.py there is a number 4096 and a 0 on the line below Richard Purdie: try changing the 0 for something like 100000

help?

No.

Note: I got tired of waiting for this (I needed to rebuild anyway) and typed ^C. I got this dump which I don't understand:

NOTE: Sending SIGTERM to remaining 1 tasks Warning, worker left partial message: <event>(ilogging LogRecord
p1
(dp2
S'taskpid'
p3
I23606
sS'threadName'
p4
S'MainThread'
p5
sS'name'
p6
S'BitBake'
p7
sS'thread'
p8
I-1216357584
sS'created'
p9
F1295541481.8053031
sS'process'
p10
I23606
sS'processName'
p11
S'MainProcess'
p12
sS'args'
p13
(tsS'module'
p14
S'__init__'
p15
sS'filename'
p16
S'__init__.py'
p17
sS'levelno'
p18
I20
sS'exc_text'
p19
NsS'pathname'
p20 S'/home/local/poky-amltd/bitbake/lib/bb/__init__.py' p21
sS'lineno'
p22
I85
sS'msg'
p23
S' /kernel/arch/arm/plat-s3c64xx/dev-audio.c' p24
sS'exc_info'
p25
NsS'funcName'
p26
S'note'
p27

What does it mean? Maybe it helps understand the problem.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#7 lore raw
In reply to Gary Thomas
Quoted 15 lines · Gary Thomas

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build & package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

   compile            1m49s
   deploy             0m16s
   populate_sysroot   0m38s

package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:

   compile            5m07s
   deploy             0m36s
   populate_sysroot   0m29s
   package            5m02s

That's a five-fold difference in the time to build & package the kernel.

This is a serious and worrying number. I just tried a test where I ran:

bitbake virtual/kernel -c package | ../predate.sh

where predate.sh contains:

#!/bin/bash
while read line ; do

    echo "$(date): ${line}"

done

and I added a bb.error to the package functions interation loop in package.bbclass.

I saw:

Fri Jan 21 13:58:00 GMT 2011: NOTE: Running task 667 of 667 (ID: 14, /media/build2/builds/rptest/meta/recipes-kernel/linux/linux-yocto_git.bb, do_package) Fri Jan 21 13:58:00 GMT 2011: NOTE: package linux-yocto-2.6.37+git0+e3895db37bd65ea97e59b70ea936c84b398d3f3e_0+a9d833fda90e2f1257888a97e092135610b5f259-r15: task do_package: Started Fri Jan 21 13:58:00 GMT 2011: ERROR: Running perform_packagecopy
Fri Jan 21 13:58:02 GMT 2011: ERROR: Running package_do_split_locales
Fri Jan 21 13:58:02 GMT 2011: ERROR: Running populate_packages
Fri Jan 21 13:58:02 GMT 2011: WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program
Fri Jan 21 13:58:02 GMT 2011:
Fri Jan 21 13:58:18 GMT 2011: ERROR: Running package_do_filedeps
Fri Jan 21 13:59:08 GMT 2011: ERROR: Running package_do_shlibs
Fri Jan 21 13:59:10 GMT 2011: ERROR: Running package_do_pkgconfig
Fri Jan 21 13:59:10 GMT 2011: ERROR: Running read_shlibdeps
Fri Jan 21 13:59:11 GMT 2011: ERROR: Running package_depchains
Fri Jan 21 13:59:11 GMT 2011: ERROR: Running emit_pkgdata
Fri Jan 21 13:59:11 GMT 2011: ERROR: Running do_package_qa
Fri Jan 21 14:00:42 GMT 2011: NOTE: package linux-yocto-2.6.37+git0+e3895db37bd65ea97e59b70ea936c84b398d3f3e_0+a9d833fda90e2f1257888a97e092135610b5f259-r15: task do_package: Succeeded Fri Jan 21 14:00:42 GMT 2011: NOTE: Tasks Summary: Attempted 667 tasks of which 615 didn't need to be rerun and 0 failed.

so it did this in 2.6 minutes which doesn't seem unreasonable.

Are you seeing the slow package step in current master and when you made the timings above, were they the only task running or were there others active too?

Cheers,

Richard

#6 lore raw
In reply to Gary Thomas
Quoted 65 lines · Gary Thomas

Note: I got tired of waiting for this (I needed to rebuild anyway) and typed ^C. I got this dump which I don't understand:

NOTE: Sending SIGTERM to remaining 1 tasks Warning, worker left partial message: <event>(ilogging LogRecord
p1
(dp2
S'taskpid'
p3
I23606
sS'threadName'
p4
S'MainThread'
p5
sS'name'
p6
S'BitBake'
p7
sS'thread'
p8
I-1216357584
sS'created'
p9
F1295541481.8053031
sS'process'
p10
I23606
sS'processName'
p11
S'MainProcess'
p12
sS'args'
p13
(tsS'module'
p14
S'__init__'
p15
sS'filename'
p16
S'__init__.py'
p17
sS'levelno'
p18
I20
sS'exc_text'
p19
NsS'pathname'
p20 S'/home/local/poky-amltd/bitbake/lib/bb/__init__.py' p21
sS'lineno'
p22
I85
sS'msg'
p23
S' /kernel/arch/arm/plat-s3c64xx/dev-audio.c' p24
sS'exc_info'
p25
NsS'funcName'
p26
S'note'
p27

Quoted 1 line

What does it mean? Maybe it helps understand the problem.

Its the internal representation of what you see as messages on the bitbake console (output of a pickled LogRecord). The worker didn't complete its queue so there was a partial message, it doesn't help work out where the time is being spent sadly.

Cheers,

Richard

#8 lore raw
In reply to Richard Purdie
Quoted 58 lines · Richard Purdie
Quoted 15 lines · Gary Thomas

My target platform/distro uses IPK and I have a local kernel recipe which is based on the pre linux-yocto recipe that was used by Poky. When I build& package the kernel, I have terrible performance - this is new, it used to be just fine.

Here's what I mean. Building for my kernel and timing the various steps, e.g.
% time bitbake virtual/kernel -c compile

My kernel:

    compile            1m49s
    deploy             0m16s
    populate_sysroot   0m38s

package 23m20s

Quoted 8 lines

Just to compare, using OTS Poky for BeagleBoard:

    compile            5m07s
    deploy             0m36s
    populate_sysroot   0m29s
    package            5m02s

That's a five-fold difference in the time to build& package the kernel.

This is a serious and worrying number. I just tried a test where I ran:

bitbake virtual/kernel -c package | ../predate.sh

where predate.sh contains:

#!/bin/bash
while read line ; do

     echo "$(date): ${line}"

done

and I added a bb.error to the package functions interation loop in package.bbclass.

I saw:

Fri Jan 21 13:58:00 GMT 2011: NOTE: Running task 667 of 667 (ID: 14, /media/build2/builds/rptest/meta/recipes-kernel/linux/linux-yocto_git.bb, do_package) Fri Jan 21 13:58:00 GMT 2011: NOTE: package linux-yocto-2.6.37+git0+e3895db37bd65ea97e59b70ea936c84b398d3f3e_0+a9d833fda90e2f1257888a97e092135610b5f259-r15: task do_package: Started Fri Jan 21 13:58:00 GMT 2011: ERROR: Running perform_packagecopy
Fri Jan 21 13:58:02 GMT 2011: ERROR: Running package_do_split_locales
Fri Jan 21 13:58:02 GMT 2011: ERROR: Running populate_packages
Fri Jan 21 13:58:02 GMT 2011: WARNING:<string>:5: RuntimeWarning: tmpnam is a potential security risk to your program
Fri Jan 21 13:58:02 GMT 2011:
Fri Jan 21 13:58:18 GMT 2011: ERROR: Running package_do_filedeps
Fri Jan 21 13:59:08 GMT 2011: ERROR: Running package_do_shlibs
Fri Jan 21 13:59:10 GMT 2011: ERROR: Running package_do_pkgconfig
Fri Jan 21 13:59:10 GMT 2011: ERROR: Running read_shlibdeps
Fri Jan 21 13:59:11 GMT 2011: ERROR: Running package_depchains
Fri Jan 21 13:59:11 GMT 2011: ERROR: Running emit_pkgdata
Fri Jan 21 13:59:11 GMT 2011: ERROR: Running do_package_qa
Fri Jan 21 14:00:42 GMT 2011: NOTE: package linux-yocto-2.6.37+git0+e3895db37bd65ea97e59b70ea936c84b398d3f3e_0+a9d833fda90e2f1257888a97e092135610b5f259-r15: task do_package: Succeeded Fri Jan 21 14:00:42 GMT 2011: NOTE: Tasks Summary: Attempted 667 tasks of which 615 didn't need to be rerun and 0 failed.

Quoted 5 lines

so it did this in 2.6 minutes which doesn't seem unreasonable.

Are you seeing the slow package step in current master and when you made the timings above, were they the only task running or were there others active too?

This particular test was isolated, just that one recipe. I noticed it before when building a complete image, so running it isolated does not seem to make any difference. There is also nothing else running on my build machine (it's a separate server that is used only for builds)

This is against master as of 2011-01-14 (897530621fabf3c345c302050a5c21e16097a2ba) with my local layers.

What can I look at (alternatively what data can I provide) to help understand this?

Thanks

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#9 lore raw
Quoted 10 lines · Gary Thomas

This particular test was isolated, just that one recipe. I noticed it before when building a complete image, so running it isolated does not seem to make any difference. There is also nothing else running on my build machine (it's a separate server that is used only for builds)

This is against master as of 2011-01-14 (897530621fabf3c345c302050a5c21e16097a2ba) with my local layers.

What can I look at (alternatively what data can I provide) to help understand this?

You could try pulling in:
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ede381d56b180b384fdad98d445e5430819cfade
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=7857834691868b7f48f732ee78d8770f5473ff68
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=fd88588df029604689db9b0e30c55aad68392a5d

and see if they fix the problem. I'm not sure that they would in a single task case but who knows. You could also try adding timings as per my previous email, the change to package.bbclass was simply:

meta/classes/package.bbclass +1 -0
index 8e7fa26..8b0cb96 100644
@@ -1082,6 +1082,7 @@ python package_do_package () {
                return
 
        for f in (bb.data.getVar('PACKAGEFUNCS', d, True) or '').split():
+               bb.error("Running %s" % f)
                bb.build.exec_func(f, d)
 }
 do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK} ${D}"

as I'd like to understand of the 23 minutes, where the time is being spent.

Cheers,

Richard

#10 lore raw
Quoted 34 lines · Richard Purdie
Quoted 10 lines · Gary Thomas

This particular test was isolated, just that one recipe. I noticed it before when building a complete image, so running it isolated does not seem to make any difference. There is also nothing else running on my build machine (it's a separate server that is used only for builds)

This is against master as of 2011-01-14 (897530621fabf3c345c302050a5c21e16097a2ba) with my local layers.

What can I look at (alternatively what data can I provide) to help understand this?

You could try pulling in:
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ede381d56b180b384fdad98d445e5430819cfade
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=7857834691868b7f48f732ee78d8770f5473ff68
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=fd88588df029604689db9b0e30c55aad68392a5d

and see if they fix the problem. I'm not sure that they would in a single task case but who knows. You could also try adding timings as per my previous email, the change to package.bbclass was simply:

meta/classes/package.bbclass +1 -0
index 8e7fa26..8b0cb96 100644
@@ -1082,6 +1082,7 @@ python package_do_package () {
                 return
 
         for f in (bb.data.getVar('PACKAGEFUNCS', d, True) or '').split():
+               bb.error("Running %s" % f)
                 bb.build.exec_func(f, d)
  }
  do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK} ${D}"
Quoted 1 line

as I'd like to understand of the 23 minutes, where the time is being spent.

For my board:
Fri Jan 21 10:12:13 MST 2011: NOTE: Running task 658 of 661 (ID: 11, /home/local/poky-amltd/meta-amltd/packages/linux/linux-am_2.6.32.bb, do_package) Fri Jan 21 10:12:13 MST 2011: NOTE: package linux-am-2.6.32-r0: task do_package: Started
Fri Jan 21 10:12:13 MST 2011: ERROR: Running perform_packagecopy
Fri Jan 21 10:12:27 MST 2011: ERROR: Running package_do_split_locales
Fri Jan 21 10:12:27 MST 2011: ERROR: Running populate_packages
Fri Jan 21 10:12:27 MST 2011: WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program
Fri Jan 21 10:12:27 MST 2011:
Fri Jan 21 10:12:45 MST 2011: NOTE: package linux-am-2.6.32-r0: task do_populate_sysroot: Succeeded
Fri Jan 21 10:28:33 MST 2011: ERROR: Running package_do_filedeps
Fri Jan 21 10:34:16 MST 2011: ERROR: Running package_do_shlibs
Fri Jan 21 10:34:21 MST 2011: ERROR: Running package_do_pkgconfig
Fri Jan 21 10:34:23 MST 2011: ERROR: Running read_shlibdeps
Fri Jan 21 10:34:23 MST 2011: ERROR: Running package_depchains
Fri Jan 21 10:34:23 MST 2011: ERROR: Running emit_pkgdata
Fri Jan 21 10:34:23 MST 2011: ERROR: Running do_package_qa

For the BeagleBoard:
Fri Jan 21 13:32:22 MST 2011: NOTE: Running task 878 of 882 (ID: 17, /home/local/poky-amltd/meta/recipes-kernel/linux/linux-yocto-stable_git.bb, do_package)
Fri Jan 21 13:32:22 MST 2011: NOTE: package linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1: task do_package: Started Fri Jan 21 13:32:23 MST 2011: ERROR: Running perform_packagecopy
Fri Jan 21 13:32:25 MST 2011: ERROR: Running package_do_split_locales
Fri Jan 21 13:32:26 MST 2011: ERROR: Running populate_packages
Fri Jan 21 13:32:27 MST 2011: WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program
Fri Jan 21 13:32:27 MST 2011:
Fri Jan 21 13:32:31 MST 2011: NOTE: package linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1: task do_populate_sysroot: Succeeded Fri Jan 21 13:32:43 MST 2011: ERROR: Running package_do_filedeps
Fri Jan 21 13:35:52 MST 2011: ERROR: Running package_do_shlibs
Fri Jan 21 13:35:54 MST 2011: ERROR: Running package_do_pkgconfig
Fri Jan 21 13:35:54 MST 2011: ERROR: Running read_shlibdeps
Fri Jan 21 13:35:55 MST 2011: ERROR: Running package_depchains
Fri Jan 21 13:35:55 MST 2011: ERROR: Running emit_pkgdata
Fri Jan 21 13:35:55 MST 2011: ERROR: Running do_package_qa

Interesting results. I'm going to run it again, with more messages in the two time consuming steps - populate_packages & package_do_filedeps

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#11 lore raw
Quoted 73 lines · Gary Thomas
Quoted 28 lines · Richard Purdie
Quoted 10 lines · Gary Thomas

This particular test was isolated, just that one recipe. I noticed it before when building a complete image, so running it isolated does not seem to make any difference. There is also nothing else running on my build machine (it's a separate server that is used only for builds)

This is against master as of 2011-01-14 (897530621fabf3c345c302050a5c21e16097a2ba) with my local layers.

What can I look at (alternatively what data can I provide) to help understand this?

You could try pulling in:
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ede381d56b180b384fdad98d445e5430819cfade
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=7857834691868b7f48f732ee78d8770f5473ff68
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=fd88588df029604689db9b0e30c55aad68392a5d

and see if they fix the problem. I'm not sure that they would in a single task case but who knows. You could also try adding timings as per my previous email, the change to package.bbclass was simply:

meta/classes/package.bbclass +0 -0
index 8e7fa26..8b0cb96 100644
@@ -1082,6 +1082,7 @@ python package_do_package () {

return

Quoted 7 lines

for f in (bb.data.getVar('PACKAGEFUNCS', d, True) or '').split():
+ bb.error("Running %s" % f)
bb.build.exec_func(f, d)
}
do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK} ${D}"

as I'd like to understand of the 23 minutes, where the time is being spent.

For my board:
Fri Jan 21 10:12:13 MST 2011: NOTE: Running task 658 of 661 (ID: 11, /home/local/poky-amltd/meta-amltd/packages/linux/linux-am_2.6.32.bb, do_package) Fri Jan 21 10:12:13 MST 2011: NOTE: package linux-am-2.6.32-r0: task do_package: Started
Fri Jan 21 10:12:13 MST 2011: ERROR: Running perform_packagecopy
Fri Jan 21 10:12:27 MST 2011: ERROR: Running package_do_split_locales
Fri Jan 21 10:12:27 MST 2011: ERROR: Running populate_packages
Fri Jan 21 10:12:27 MST 2011: WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program
Fri Jan 21 10:12:27 MST 2011:
Fri Jan 21 10:12:45 MST 2011: NOTE: package linux-am-2.6.32-r0: task do_populate_sysroot: Succeeded
Fri Jan 21 10:28:33 MST 2011: ERROR: Running package_do_filedeps
Fri Jan 21 10:34:16 MST 2011: ERROR: Running package_do_shlibs
Fri Jan 21 10:34:21 MST 2011: ERROR: Running package_do_pkgconfig
Fri Jan 21 10:34:23 MST 2011: ERROR: Running read_shlibdeps
Fri Jan 21 10:34:23 MST 2011: ERROR: Running package_depchains
Fri Jan 21 10:34:23 MST 2011: ERROR: Running emit_pkgdata
Fri Jan 21 10:34:23 MST 2011: ERROR: Running do_package_qa

For the BeagleBoard:
Fri Jan 21 13:32:22 MST 2011: NOTE: Running task 878 of 882 (ID: 17, /home/local/poky-amltd/meta/recipes-kernel/linux/linux-yocto-stable_git.bb, do_package) Fri Jan 21 13:32:22 MST 2011: NOTE: package linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1: task do_package:
Started
Fri Jan 21 13:32:23 MST 2011: ERROR: Running perform_packagecopy
Fri Jan 21 13:32:25 MST 2011: ERROR: Running package_do_split_locales
Fri Jan 21 13:32:26 MST 2011: ERROR: Running populate_packages
Fri Jan 21 13:32:27 MST 2011: WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program
Fri Jan 21 13:32:27 MST 2011:
Fri Jan 21 13:32:31 MST 2011: NOTE: package linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1: task
do_populate_sysroot: Succeeded
Fri Jan 21 13:32:43 MST 2011: ERROR: Running package_do_filedeps
Fri Jan 21 13:35:52 MST 2011: ERROR: Running package_do_shlibs
Fri Jan 21 13:35:54 MST 2011: ERROR: Running package_do_pkgconfig
Fri Jan 21 13:35:54 MST 2011: ERROR: Running read_shlibdeps
Fri Jan 21 13:35:55 MST 2011: ERROR: Running package_depchains
Fri Jan 21 13:35:55 MST 2011: ERROR: Running emit_pkgdata
Fri Jan 21 13:35:55 MST 2011: ERROR: Running do_package_qa

Quoted 3 lines

Interesting results. I'm going to run it again, with more messages in the two time consuming steps - populate_packages & package_do_filedeps

I've isolated it down to this section of code in 'populate_packages':

unshipped = []
for root, dirs, files in os.walk(dvar):
for f in files:
path = os.path.join(root[len(dvar):], f) if ('.' + path) not in seen:
unshipped.append(path)

if unshipped != []:
bb.note("the following files were installed but not shipped in any package:") for f in unshipped:
bb.note(" " + f)

The difference comes from what's in that ".../package" tree. For my kernel:

  $ find /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/package | wc
     8567    8567 1098403
For the OTS BeagleBoard kernel:
  $ find 
/home/local/bb_poky/tmp/work/beagleboard-poky-linux-gnueabi/linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1/package | wc
     1598    1598  364151

I measured this and in the 'for f in unshipped:' loop above, each iteration is taking ~125ms! I have a hard time understanding why, but 8567*125ms = 1070.875sec = 17.847min!

I also found that later on, in 'do_filedeps()', some packages take
an extraordinary amount of time. Looking at a few:
Sat Jan 22 09:17:55 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-image Sat Jan 22 09:17:55 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-image Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-dev Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-dev Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-vmlinux Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-vmlinux Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-misc Sat Jan 22 09:20:36 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-misc Sat Jan 22 09:23:28 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-crc-itu-t Sat Jan 22 09:23:28 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-crc-itu-t Sat Jan 22 09:23:28 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-g-cdc Sat Jan 22 09:23:29 MST 2011: ERROR: do_filedeps: /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh --rpmdeps /home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps --requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-g-cdc

You'll see that most take a few seconds (at most), but the 'kernel-misc'
package is taking 6 minutes by itself! This particular sub-package seems
to be completely different between the two kernels.
Mine:
-rwxrw-r-- 1 gthomas gthomas 8627800 Jan 22 09:25 tmp/deploy/ipk/cobra3530p60/kernel-misc_2.6.32-r0_cobra3530p60.ipk OTS BeagleBoard:
-rwxrw-r-- 1 gthomas gthomas 72014 Jan 22 10:12 tmp/deploy/ipk/beagleboard/kernel-misc_2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1_beagleboard.ipk

I think this boils down to the actual kernel recipe - mine (attached) is based on the older setup and includes "linux.inc" whereas the BeagleBoard was built using a newer recipe which includes "linux-yocto.inc" I tried just changing to use linux-yocto, but there seems to be much different and nothing seemed to build properly anymore. Any comments on this would be appreciated.

At least now I know where the time is being spent...

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

require recipes-kernel/linux/linux.inc

DESCRIPTION = "My Linux kernel"

DEPENDS += "linux-firmware "
PV = "2.6.32"

# Note: defconfig comes from target recipe, as does the machine specific patchset

SRC_URI = "http://www.mlbassoc.com/repos/public/linux-am-${PV}.tar.bz2 \
           file://${MACHINE}.patch \
           file://defconfig \

"
LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7 "

#12 lore raw
Quoted 114 lines · Gary Thomas
Quoted 87 lines · Gary Thomas
Quoted 37 lines · Richard Purdie
Quoted 6 lines · Gary Thomas

This particular test was isolated, just that one recipe. I noticed it before when building a complete image, so running it isolated does not seem to make any difference. There is also nothing else running on my build machine (it's a separate server that is used only for builds)

Quoted 6 lines

This is against master as of 2011-01-14
(897530621fabf3c345c302050a5c21e16097a2ba)
with my local layers.

What can I look at (alternatively what data can I provide) to help understand this?

You could try pulling in:
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=ede381d56b180b384fd
ad98d445e5430819cfade
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=7857834691868b7f48f
732ee78d8770f5473ff68
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=fd88588df029604689d
b9b0e30c55aad68392a5d

and see if they fix the problem. I'm not sure that they would in a single task case but who knows. You could also try adding timings as per my previous email, the change to package.bbclass was simply:

(quoted diff) +0 -0

b/meta/classes/package.bbclass index 8e7fa26..8b0cb96 100644

meta/classes/package.bbclass +0 -0
@@ -1082,6 +1082,7 @@ python package_do_package () { return
 

for f in (bb.data.getVar('PACKAGEFUNCS', d, True) or '').split(): + bb.error("Running %s" % f) bb.build.exec_func(f, d)
}
do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK} ${D}"

Quoted 2 lines

as I'd like to understand of the 23 minutes, where the time is being spent.

For my board:
Fri Jan 21 10:12:13 MST 2011: NOTE: Running task 658 of 661 (ID: 11, /home/local/poky-amltd/meta-amltd/packages/linux/linux-am_2.6.32.bb, do_package) Fri Jan 21 10:12:13 MST 2011: NOTE: package linux-am-2.6.32-r0: task do_package: Started Fri Jan 21 10:12:13 MST 2011: ERROR: Running perform_packagecopy Fri Jan 21 10:12:27 MST 2011:
ERROR: Running package_do_split_locales Fri Jan 21 10:12:27 MST 2011:
ERROR: Running populate_packages Fri Jan 21 10:12:27 MST 2011:
WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program Fri Jan 21 10:12:27 MST 2011:
Fri Jan 21 10:12:45 MST 2011: NOTE: package linux-am-2.6.32-r0: task
do_populate_sysroot: Succeeded Fri Jan 21 10:28:33 MST 2011: ERROR:
Running package_do_filedeps Fri Jan 21 10:34:16 MST 2011: ERROR:
Running package_do_shlibs Fri Jan 21 10:34:21 MST 2011: ERROR:
Running
package_do_pkgconfig Fri Jan 21 10:34:23 MST 2011: ERROR: Running read_shlibdeps Fri Jan 21 10:34:23 MST 2011: ERROR: Running package_depchains Fri Jan 21 10:34:23 MST 2011: ERROR: Running emit_pkgdata Fri Jan 21 10:34:23 MST 2011: ERROR: Running do_package_qa

For the BeagleBoard:
Fri Jan 21 13:32:22 MST 2011: NOTE: Running task 878 of 882 (ID: 17, /home/local/poky-amltd/meta/recipes-kernel/linux/linux-yocto-stable_git.bb,
do_package) Fri Jan 21 13:32:22 MST 2011: NOTE: package linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1:
task do_package: Started Fri Jan 21 13:32:23 MST 2011: ERROR:
Running perform_packagecopy Fri

Jan 21 13:32:25 MST 2011: ERROR: Running package_do_split_locales Fri
Jan 21 13:32:26 MST 2011: ERROR: Running populate_packages Fri Jan 21

13:32:27 MST 2011: WARNING: <string>:5: RuntimeWarning: tmpnam is a potential security risk to your program Fri Jan 21 13:32:27 MST 2011:
Fri Jan 21 13:32:31 MST 2011: NOTE: package linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b7 5_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1: task
do_populate_sysroot: Succeeded
Fri Jan 21 13:32:43 MST 2011: ERROR: Running package_do_filedeps Fri

Jan 21 13:35:52 MST 2011: ERROR: Running package_do_shlibs Fri Jan 21

13:35:54 MST 2011: ERROR: Running package_do_pkgconfig Fri Jan 21 13:35:54 MST 2011: ERROR: Running read_shlibdeps Fri Jan 21 13:35:55 MST 2011: ERROR: Running package_depchains Fri Jan 21 13:35:55 MST 2011: ERROR: Running emit_pkgdata Fri Jan 21 13:35:55 MST 2011:
ERROR:
Running do_package_qa

Quoted 4 lines

Interesting results. I'm going to run it again, with more messages in the two time consuming steps - populate_packages & package_do_filedeps

I've isolated it down to this section of code in 'populate_packages':

unshipped = []
for root, dirs, files in os.walk(dvar):
for f in files:
path = os.path.join(root[len(dvar):], f) if ('.' + path) not in seen:
unshipped.append(path)

if unshipped != []:
bb.note("the following files were installed but not shipped in any package:") for f in unshipped:
bb.note(" " + f)

The difference comes from what's in that ".../package" tree. For my kernel:

   $ find

/home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/package
| wc 8567 8567 1098403
For the OTS BeagleBoard kernel:

Quoted 8 lines
   $ find
/home/local/bb_poky/tmp/work/beagleboard-poky-linux-gnueabi/linux-yocto-stable-2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1/package
Quoted 1 line

wc

      1598    1598  364151

I measured this and in the 'for f in unshipped:' loop above, each iteration is taking ~125ms! I have a hard time understanding why, but 8567*125ms = 1070.875sec = 17.847min!

Hi Richard and Gary,

Kevin and I spent some time investigating this logging issue, and finally we found its cause.

The phenomenon is, if we write a task like:

testlog.bb:

python do_crazylog () {
for i in range(500):
bb.note("xdx: hello world.")
}
addtask crazylog

And run the command: "time bitbake testlog -c crazylog -f", it needs about 68 seconds to print the 500 sentences, which is problematic.

I used "bitbake -P" to do some profiling for this task, and found most time is costed by time.sleep(), and each time it will sleep 0.25 seconds.

Thu Jan 27 16:11:59 2011 profile.log

         922167 function calls (892080 primitive calls) in 69.882 CPU seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall filename:lineno(function)

        1    0.012    0.012   69.887   69.887 /home/dongxiao/poky/bitbake/lib/bb/ui/knotty.py:67(main)
     1391    0.005    0.000   69.829    0.050 /home/dongxiao/poky/bitbake/lib/bb/server/none.py:95(waitEvent)
     1044    0.005    0.000   69.818    0.067 /home/dongxiao/poky/bitbake/lib/bb/server/none.py:121(idle_commands)
      265   66.350    0.250   66.350    0.250 {time.sleep}
      780    0.001    0.000    2.143    0.003 /home/dongxiao/poky/bitbake/lib/bb/cooker.py:156(runCommands)
      780    0.003    0.000    2.142    0.003 /home/dongxiao/poky/bitbake/lib/bb/command.py:91(runAsyncCommand)
      780    0.002    0.000    1.935    0.002 /home/dongxiao/poky/bitbake/lib/bb/cooker.py:738(updateCache)

I bisected the git log and found that the commit introduced this issue is a merge from openembedded.

After this commit, the whole LogRecord object into eventQueue, which contains more information than only level and messages.

Commit: d3a45c7d41a88d79389fc40eb68816e4939fb6f9.

Use logging in the knotty ui, and pass the log record across directly

This kills firing of Msg* events in favor of just passing along LogRecord objects. These objects hold more than just level and message, but can also have exception information, so the UI can decide what to do with that.

As an aside, when using the 'none' server, this results in the log messages in the server being displayed directly via the logging module and the UI's handler, rather than going through the server's event queue. As a result of doing it this way, we have to override the event handlers of the base logger when spawning a worker process, to ensure they log via events rather than directly.

Based on our analysis of the current code, the main steps for logging is: (knotty ui and none server)

1) Each time when bitbake server fork off a task in execute(), it will create pipe between server and forked task.

2) Server will read the pipe to get "events", and then append it to the server eventQueue. Each time it will read 1024 bytes.

3) The UI module (knotty in our case) will wait on getting the event from eventQueue of "none" server, if there are events exist, it will handle the event. Otherwise, it will sleep 0.25 seconds and see there is any event.

Previsouly before the patch, we only put message and level information into eventQueue, thus each time in read(), we can read a bunch of messages in above case (about 10 "xdx: hello world.").

However, after passing the whole object, 1024 bytes could only cover 1, 2 or even zero LogRecord objects. Therefore the server is always "starve" to get event from the pipe, and thus the eventQueue is empty in most cases, which makes knotty ui sleep.

One of the straightforward idea is to change 1024 to be a bigger number, for example, 10240. See following patch.

bitbake/lib/bb/runqueue.py +1 -1
index b9d89ec..196b957 100644
@@ -1637,7 +1637,7 @@ class runQueuePipe():
     def read(self):
         start = len(self.queue)
         try:
-            self.queue = self.queue + self.input.read(1024)
+            self.queue = self.queue + self.input.read(10240)
         except (OSError, IOError):
             pass
         end = len(self.queue)

I retested the above case, and it can get a much better results:

After patch:

real    0m9.994s
user    0m3.410s
sys     0m0.390s

Before patch:

real    1m8.614s
user    0m3.460s
sys     0m0.320s

This is a bottle neck for those cases which have a loop of "bb.note", like Gary found in the package bbclass for kernel recipes.

But for normal "bb.note", it may not benefit a lot. So this logging issue may be not a key point to the current long building time.

But anyhow, I am scheduling a test for this patch to see whether it helps for our common build result, like poky-image-minimal, etc.

Comments and thoughts on it?

Thanks,
Dongxiao

Quoted 96 lines

I also found that later on, in 'do_filedeps()', some packages take an
extraordinary amount of time. Looking at a few:
Sat Jan 22 09:17:55 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-image Sat Jan 22 09:17:55 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-image Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-dev Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-dev Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-vmlinux Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-vmlinux Sat Jan 22 09:17:56 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-misc Sat Jan 22 09:20:36 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-misc Sat Jan 22 09:23:28 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-crc-itu-t Sat Jan 22 09:23:28 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-crc-itu-t Sat Jan 22 09:23:28 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--provides /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-g-cdc Sat Jan 22 09:23:29 MST 2011: ERROR: do_filedeps:
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/perfile_rpmdeps.sh
--rpmdeps
/home/local/p60_poky/tmp/sysroots/i686-linux/usr/lib/rpm/rpmdeps
--requires /home/local/p60_poky/tmp/work/cobra3530p60-poky-linux-gnueabi/linux-am-2.6.32-r0/packages-split/kernel-module-g-cdc

You'll see that most take a few seconds (at most), but the
'kernel-misc'
package is taking 6 minutes by itself! This particular sub-package
seems to be completely different between the two kernels.
Mine:
-rwxrw-r-- 1 gthomas gthomas 8627800 Jan 22 09:25
tmp/deploy/ipk/cobra3530p60/kernel-misc_2.6.32-r0_cobra3530p60.ipk
OTS BeagleBoard:
-rwxrw-r-- 1 gthomas gthomas 72014 Jan 22 10:12
tmp/deploy/ipk/beagleboard/kernel-misc_2.6.34+git0+ed446ecd29e8c0f81ab8630a2db652121eeb0b75_0+35521a5a70316785a67aca1de1d39a7b84c49ccf-r1_beagleboard.ipk

I think this boils down to the actual kernel recipe - mine (attached) is based on the older setup and includes "linux.inc" whereas the BeagleBoard was built using a newer recipe which includes "linux-yocto.inc"
I tried just changing to use linux-yocto, but there seems to be much different and nothing seemed to build properly anymore. Any comments on this would be appreciated.

At least now I know where the time is being spent...

#13 lore raw
Quoted 6 lines

From: Xu, Dongxiao
Sent: Friday, January 28, 2011 3:42 PM

This is a bottle neck for those cases which have a loop of "bb.note", like Gary found in the package bbclass for kernel recipes.

Quoted 8 lines

But for normal "bb.note", it may not benefit a lot. So this logging issue may be not a key point to the current long building time.

But anyhow, I am scheduling a test for this patch to see whether it helps for our common build result, like poky-image-minimal, etc.

Comments and thoughts on it?

Another issue found in the test, which we're not sure whether we did it correctly, that bb.debug is never printed out to the console even with -DDD. Dongxiao checked the server side, that no record is passed from the child process regarding to debug info.

Is there any change to activate the debug message or is it a real issue?

Thanks
Kevin

#15 lore raw
In reply to Xu, Dongxiao
Quoted 131 lines · Dongxiao

Kevin and I spent some time investigating this logging issue, and finally we found its cause.

The phenomenon is, if we write a task like:

testlog.bb:

python do_crazylog () {
for i in range(500):
bb.note("xdx: hello world.")
}
addtask crazylog

And run the command: "time bitbake testlog -c crazylog -f", it needs about 68 seconds to print the 500 sentences, which is problematic.

I used "bitbake -P" to do some profiling for this task, and found most time is costed by time.sleep(), and each time it will sleep 0.25 seconds.

Thu Jan 27 16:11:59 2011 profile.log

         922167 function calls (892080 primitive calls) in 69.882 CPU

seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall
filename:lineno(function)

        1    0.012    0.012   69.887
69.887 /home/dongxiao/poky/bitbake/lib/bb/ui/knotty.py:67(main)
     1391    0.005    0.000   69.829
0.050 /home/dongxiao/poky/bitbake/lib/bb/server/none.py:95(waitEvent)
     1044    0.005    0.000   69.818
0.067 /home/dongxiao/poky/bitbake/lib/bb/server/none.py:121(idle_commands)
      265   66.350    0.250   66.350    0.250 {time.sleep}
      780    0.001    0.000    2.143
0.003 /home/dongxiao/poky/bitbake/lib/bb/cooker.py:156(runCommands)
      780    0.003    0.000    2.142
0.003 /home/dongxiao/poky/bitbake/lib/bb/command.py:91(runAsyncCommand)
      780    0.002    0.000    1.935

0.002 /home/dongxiao/poky/bitbake/lib/bb/cooker.py:738(updateCache)

I bisected the git log and found that the commit introduced this issue is a merge from openembedded.

After this commit, the whole LogRecord object into eventQueue, which contains more information than only level and messages.

Commit: d3a45c7d41a88d79389fc40eb68816e4939fb6f9.

Use logging in the knotty ui, and pass the log record across
directly

This kills firing of Msg* events in favor of just passing along
LogRecord
objects. These objects hold more than just level and message, but
can also
have exception information, so the UI can decide what to do with
that.

As an aside, when using the 'none' server, this results in the log
messages in
the server being displayed directly via the logging module and the
UI's
handler, rather than going through the server's event queue. As a
result of
doing it this way, we have to override the event handlers of the
base logger
when spawning a worker process, to ensure they log via events
rather than
directly.

Based on our analysis of the current code, the main steps for logging
is: (knotty ui and none server)

1) Each time when bitbake server fork off a task in execute(), it will create pipe between server and forked task.

2) Server will read the pipe to get "events", and then append it to the server eventQueue. Each time it will read 1024 bytes.

3) The UI module (knotty in our case) will wait on getting the event from eventQueue of "none" server, if there are events exist, it will handle the event. Otherwise, it will sleep 0.25 seconds and see there is any event.

Previsouly before the patch, we only put message and level information into eventQueue, thus each time in read(), we can read a bunch of messages in above case (about 10 "xdx: hello world.").

However, after passing the whole object, 1024 bytes could only cover 1, 2 or even zero LogRecord objects. Therefore the server is always "starve" to get event from the pipe, and thus the eventQueue is empty in most cases, which makes knotty ui sleep.

One of the straightforward idea is to change 1024 to be a bigger number, for example, 10240. See following patch.

bitbake/lib/bb/runqueue.py +1 -1
index b9d89ec..196b957 100644
@@ -1637,7 +1637,7 @@ class runQueuePipe():
     def read(self):
         start = len(self.queue)
         try:
-            self.queue = self.queue + self.input.read(1024)
+            self.queue = self.queue + self.input.read(10240)
         except (OSError, IOError):
             pass
         end = len(self.queue)

I retested the above case, and it can get a much better results:

After patch:

real    0m9.994s
user    0m3.410s
sys     0m0.390s

Before patch:

real    1m8.614s
user    0m3.460s
sys     0m0.320s

This is a bottle neck for those cases which have a loop of "bb.note", like Gary found in the package bbclass for kernel recipes.

Quoted 7 lines

But for normal "bb.note", it may not benefit a lot. So this logging issue may be not a key point to the current long building time.

But anyhow, I am scheduling a test for this patch to see whether it helps for our common build result, like poky-image-minimal, etc.

Comments and thoughts on it?

Great work in tracking that down. I've merged a version into master but I did increase the read value by another factor of 10 as my local tests showed it to be beneficial. I was nearly tempted to remove the size all together but didn't for the reasons I've outlined in the commit (task scheduling fairness and the risk of a crazy task blocking the server main loop).

I think this could well have a significant speed effect on things in general, we'll soon find out I guess! :)

Cheers,

Richard

#14 lore raw
In reply to Tian, Kevin
Quoted 12 lines · Kevin
Quoted 12 lines

From: Xu, Dongxiao
Sent: Friday, January 28, 2011 3:42 PM

This is a bottle neck for those cases which have a loop of "bb.note", like Gary found in the package bbclass for kernel recipes.

But for normal "bb.note", it may not benefit a lot. So this logging issue may be not a key point to the current long building time.

But anyhow, I am scheduling a test for this patch to see whether it helps for our common build result, like poky-image-minimal, etc.

Quoted 8 lines
Quoted 2 lines

Comments and thoughts on it?

Another issue found in the test, which we're not sure whether we did it correctly, that bb.debug is never printed out to the console even with -DDD. Dongxiao checked the server side, that no record is passed from the child process regarding to debug info.

Is there any change to activate the debug message or is it a real issue?

There was a problem in bitbake which was fixed in:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=d7eebbe9dbf0d790d4af93466f5c27127cae0999

I think this was the likely cause of this observation.

Cheers,

Richard