Skip to contentSkip to conversation

Some data collection and analysis on poky performance

DISCUSSION Qing He 2 participants 2 messages Oct 27, 2010 – Oct 28, 2010

lore
Share & tools
lore

Qing He

author
#0 lore raw

As we know, many of us have experienced slow builds of recent poky, and it also takes larger disk space. This affects user exprience thus is one of our directions in 1.0.

To find the problems leading to performance issues, I tried some profiling on poky builds, below is a very brief summary of the data. I profiled poky-image-minimal of both the current master branch and green release, with similar parameters (4 CPUs) on NHM. Note that both rpm and ipk packages are built for current master branch, while oonly ipk packages are built for green release

I. some stats
1. recipes (including -native)
green release:
recipes built: 76
tasks run: 998
master:
recipes built: 133
tasks run: 1600

2. time
green release:

    real:  28m7s
    user:  57m45s
    sys:    9m41s
    user+sys: 67m26s
  master:
    real:  66m39s
    user: 152m17s
    sys:   27m37s
    user+sys: 179m54s

3. space (haven't analyze though)
green release: 7.8G
master: 26.6G

II. profiling
I tried a brief profile by collecting the time used for every task, so we can scrutinize the result from a microscopic point of view. I'm still looking into the full result, but there's something of immediate attention.

In master, hardly any task consume less than 1.3s, this is quite surprising, since many tasks like do_patch virtually do nothing, while in the green release, these tasks may simply consume 0.1~0.2s. A deeper investigation shows that this large overhead goes to bitbake-runtask, the bitbake config and cache mechanism is executed for every task, considerably increased the time. The overhead introduced solely by this is around 1600*1.3=2080s, approximately 35 minutes (user+sys).

Also, we should count the additional rpm packaging system, do_package_write_rpm costs around 1400s, (excluding 1.3s per task, btw, this is about 50% slower than do_package_write_ipk, in average), that's around 23 minutes.

Roughly considering that the build time is proportional to recipes count, we can try to estimate master build time from green release:

    67 * (76 / 133) + 35 + 23 = 175

very close to the real time consuming (although somewhat too closed...), so possibly the above two are the most significant time consumers in the slowness of current poky

Thanks,
Qing

#1 lore raw
Quoted 6 lines

From: Qing He
Sent: Wednesday, October 27, 2010 5:23 PM

As we know, many of us have experienced slow builds of recent poky, and it also takes larger disk space. This affects user exprience thus is one of our directions in 1.0.

thanks Qing, that's a great start.

Quoted 9 lines

To find the problems leading to performance issues, I tried some profiling on poky builds, below is a very brief summary of the data. I profiled poky-image-minimal of both the current master branch and green release, with similar parameters (4 CPUs) on NHM. Note that both rpm and ipk packages are built for current master branch, while oonly ipk packages are built for green release

Quoted 8 lines

I. some stats
1. recipes (including -native)
green release:
recipes built: 76
tasks run: 998
master:
recipes built: 133
tasks run: 1600

could you get a compare list which recipes have been newly added? I thought that minimal image was seldom changed...

Quoted 13 lines

2. time
green release:

   real:  28m7s
   user:  57m45s
   sys:    9m41s
   user+sys: 67m26s
 master:
   real:  66m39s
   user: 152m17s
   sys:   27m37s
   user+sys: 179m54s
Quoted 3 lines

3. space (haven't analyze though)
green release: 7.8G
master: 26.6G

I recall that RP mentioned debug symbol is enabled in current master, but I'm not sure whether green has done the same. You may double-confirm that part first.

Quoted 8 lines

II. profiling
I tried a brief profile by collecting the time used for every task, so we can scrutinize the result from a microscopic point of view. I'm still looking into the full result, but there's something of immediate attention.

Quoted 7 lines

In master, hardly any task consume less than 1.3s, this is quite surprising, since many tasks like do_patch virtually do nothing, while in the green release, these tasks may simply consume 0.1~0.2s. A deeper investigation shows that this large overhead goes to bitbake-runtask, the bitbake config and cache mechanism is executed for every task, considerably increased the time. The overhead introduced solely by this is around 1600*1.3=2080s, approximately 35 minutes (user+sys).

It's said this change comes from pseudo integration. Now it's time for us to revisit this implementation then. RP can have more insights here.

Quoted 6 lines

Also, we should count the additional rpm packaging system, do_package_write_rpm costs around 1400s, (excluding 1.3s per task, btw, this is about 50% slower than do_package_write_ipk, in average), that's around 23 minutes.

Quoted 7 lines

Roughly considering that the build time is proportional to recipes count, we can try to estimate master build time from green release:

   67 * (76 / 133) + 35 + 23 = 175

very close to the real time consuming (although somewhat too closed...), so possibly the above two are the most significant time consumers in the slowness of current poky

above are all good findings!

Thanks
Kevin