Skip to contentSkip to conversation

nightly-release takes more than 24 hours to build.

DISCUSSION Scott Garman 5 participants 6 messages Nov 1, 2010 – Nov 4, 2010

lore
Share & tools
lore
#0 lore raw

Hello,

Leading up to our 0.9 release, our autobuilder has been building an increasing number of targets for our nightly-release buildset. We've now reached the point that the nightly build takes more than 24 hours to run (> 26 hours, in fact) - which is clearly a problem on a build that we'd like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm, qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference, but it would be misleading as we build the targets in succession (e.g, we start with poky-image-sdk which takes the bulk of the time, and then the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours to build. The question is what we can move out of the nightly build and do on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM. Throwing hardware at the problem is another solution, but not an inexpensive one (we'd be looking at a 4-socket machine filled with quad-cores and 32 GB of RAM).

I'm open to ideas on how to address this issue. QA will be driving a lot of the requirements and I'm especially interested to hear your thoughts.

Scott

#1 lore raw
Quoted 32 lines · Scott Garman

Leading up to our 0.9 release, our autobuilder has been building an increasing number of targets for our nightly-release buildset. We've now reached the point that the nightly build takes more than 24 hours to run (> 26 hours, in fact) - which is clearly a problem on a build that we'd like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm, qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference, but it would be misleading as we build the targets in succession (e.g, we start with poky-image-sdk which takes the bulk of the time, and then the other targets can largely rely on the shared state builds).

Quoted 8 lines

Ideally I think our nightly build should take much less than 24 hours to build. The question is what we can move out of the nightly build and do on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM. Throwing hardware at the problem is another solution, but not an inexpensive one (we'd be looking at a 4-socket machine filled with quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up needing multiple machines and we can split the load between them quite easily. I think there is going to be a second machine needed alongside the existing one regardless of what other optimisations we make.

The changes in development at the moment will have a mixed effect on the autobuilder's load. On the plus side we know there are performance regressions with 0.9 which we're about to investigate. I can think of one change I have in mind which on its own should get the builds back under the 24 hour time.

On the down side there are going to be changes that need increased horsepower from the build machines such as the runtime testing we're close to enabling or enabling builds of world.

Cheers,

Richard

#2 lore raw
Quoted 11 lines

From: [email protected] [mailto:yocto- [email protected]] On Behalf Of Richard Purdie
Sent: Monday, November 01, 2010 4:03 AM

Quoted 5 lines · Scott Garman

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM. Throwing hardware at the problem is another solution, but not an inexpensive one (we'd be looking at a 4-socket machine filled with quad-cores and 32 GB of RAM).

Quoted 4 lines

There doesn't just have to be one build machine, we are going to end up needing multiple machines and we can split the load between them quite easily. I think there is going to be a second machine needed alongside the existing one regardless of what other optimisations we make.

I can buy another server and contribute it to the build effort. I had intended to buy one this quarter to begin hosting yoctoproject.org and a source mirror, but we could offload part of the builds to it as well.

As you know, I want us to get to the root of the problem, which is efficiency of the build process. Since RP has already committed to addressing this, I'm fine with a short-term fix if it will help. :-)

Scott - please put in an order, will tell you the max amount today when we're f2f.

Dave

#3 lore raw
Quoted 4 lines · David C

I can buy another server and contribute it to the build effort. I had intended to buy one this quarter to begin hosting yoctoproject.org and a source mirror, but we could offload part of the builds to it as well.

Great, thanks Dave! :)

I have spec'ed out and initiated an order for our newest server. In case anyone likes to geek out on hardware specs, here they are:

Dell PowerEdge R710
2x Xeon X5680 @ 3.33Ghz
24 GB RAM
6x 1.5 TB hard drives for use in a RAID5 array

While it's not a 4-processor system, the additional resources should be sufficient for us to tackle our resource problems my distributing the workload more evenly.

The system will likely arrive and be set up around the end of the month.

Scott

#4 lore raw
In reply to Richard Purdie
Quoted 50 lines

From: Richard Purdie
Sent: Monday, November 01, 2010 7:03 PM

Quoted 32 lines · Scott Garman

Leading up to our 0.9 release, our autobuilder has been building an increasing number of targets for our nightly-release buildset. We've now reached the point that the nightly build takes more than 24 hours to run (> 26 hours, in fact) - which is clearly a problem on a build that we'd like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm, qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference, but it would be misleading as we build the targets in succession (e.g, we start with poky-image-sdk which takes the bulk of the time, and then the other targets can largely rely on the shared state builds).

Quoted 8 lines

Ideally I think our nightly build should take much less than 24 hours to build. The question is what we can move out of the nightly build and do on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM. Throwing hardware at the problem is another solution, but not an inexpensive one (we'd be looking at a 4-socket machine filled with quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up needing multiple machines and we can split the load between them quite easily. I think there is going to be a second machine needed alongside the existing one regardless of what other optimisations we make.

Quoted 5 lines

The changes in development at the moment will have a mixed effect on the autobuilder's load. On the plus side we know there are performance regressions with 0.9 which we're about to investigate. I can think of one change I have in mind which on its own should get the builds back under the 24 hour time.

In case you didn't note Qing's investigation last week, attach his mail for your reference.

Thanks
Kevin

Quoted 5 lines

On the down side there are going to be changes that need increased horsepower from the build machines such as the runtime testing we're close to enabling or enabling builds of world.

Quoted 8 lines

Cheers,

Richard

_______________________________________________
yocto mailing list
[email protected]
https://lists.pokylinux.org/listinfo/yocto

As we know, many of us have experienced slow builds of recent poky, and it also takes larger disk space. This affects user exprience thus is one of our directions in 1.0.

To find the problems leading to performance issues, I tried some profiling on poky builds, below is a very brief summary of the data. I profiled poky-image-minimal of both the current master branch and green release, with similar parameters (4 CPUs) on NHM. Note that both rpm and ipk packages are built for current master branch, while oonly ipk packages are built for green release

I. some stats
1. recipes (including -native)
green release:
recipes built: 76
tasks run: 998
master:
recipes built: 133
tasks run: 1600

2. time
green release:

    real:  28m7s
    user:  57m45s
    sys:    9m41s
    user+sys: 67m26s
  master:
    real:  66m39s
    user: 152m17s
    sys:   27m37s
    user+sys: 179m54s

3. space (haven't analyze though)
green release: 7.8G
master: 26.6G

II. profiling
I tried a brief profile by collecting the time used for every task, so we can scrutinize the result from a microscopic point of view. I'm still looking into the full result, but there's something of immediate attention.

In master, hardly any task consume less than 1.3s, this is quite surprising, since many tasks like do_patch virtually do nothing, while in the green release, these tasks may simply consume 0.1~0.2s. A deeper investigation shows that this large overhead goes to bitbake-runtask, the bitbake config and cache mechanism is executed for every task, considerably increased the time. The overhead introduced solely by this is around 1600*1.3=2080s, approximately 35 minutes (user+sys).

Also, we should count the additional rpm packaging system, do_package_write_rpm costs around 1400s, (excluding 1.3s per task, btw, this is about 50% slower than do_package_write_ipk, in average), that's around 23 minutes.

Roughly considering that the build time is proportional to recipes count, we can try to estimate master build time from green release:

    67 * (76 / 133) + 35 + 23 = 175

very close to the real time consuming (although somewhat too closed...), so possibly the above two are the most significant time consumers in the slowness of current poky

Thanks,
Qing

_______________________________________________
yocto mailing list
[email protected]
https://lists.pokylinux.org/listinfo/yocto

#5 lore raw
In reply to Scott Garman
Quoted 30 lines

Hello,

Leading up to our 0.9 release, our autobuilder has been building an increasing number of targets for our nightly-release buildset. We've now reached the point that the nightly build takes more than 24 hours to run (> 26 hours, in fact)
- which is clearly a problem on a build that we'd like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm, qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state prebuilds and RPM output at the end of the build.

Quoted 9 lines

I was going to post build times for some of these targets for reference, but it would be misleading as we build the targets in succession (e.g, we start with poky-image-sdk which takes the bulk of the time, and then the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours to build. The question is what we can move out of the nightly build and do on perhaps a weekly basis instead?

How about customizing the nightly build to build part of targets for every day? For example, x86/x86_64 for Monday, atom-pc/emenlow for Tuesday..... etc. And we can schedule a new build task for weekly, which is only triggered on Wednesday(or maybe Tuesday night) for QA weekly testing and with all targets built out.

Quoted 8 lines

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM. Throwing hardware at the problem is another solution, but not an inexpensive one (we'd be looking at a 4-socket machine filled with quad-cores and 32 GB of RAM).

I'm open to ideas on how to address this issue. QA will be driving a lot of the requirements and I'm especially interested to hear your thoughts.

Quoted 7 lines

Scott

_______________________________________________
yocto mailing list
[email protected]
https://lists.pokylinux.org/listinfo/yocto

Best Regards,
Jiajun