Parallel builds with common shared state cache
Hi,
I'm curious is there are any issues with using a common shared state cache directory, outside the build directory, for multiple different image builds that also might be running in parallel to each other? I'm in the process of setting up a new build server and wanted to make sure this wasn't an invalid configuration before proceeding.
We generally build three different images that are all based on the same machine type. Our existing builds, which are still based on Edison, all have their shared state cache dir set to /opt/sstate-cache. While I'm pretty confident there are no issues having the three projects point at the same directory from a content perspective, are there any problems with parallel builds in separate build trees referencing the same shared state cache area at potentially the same time? Are there any general restrictions to using a common shared state cache directory?
Thanks,
Brian
Quoted 5 lines · Brian Karcz
I’m curious is there are any issues with using a common shared state cache directory, outside the build directory, for multiple different image builds that also might be running in parallel to each other? I’m in the process of setting up a new build server and wanted to make sure this wasn’t an invalid configuration before proceeding.
I'm not positive that there are locks around the sstate cache (although there arguably should be if there isn't), but the worst-case scenario is that one build tries to extract a partial sstate archive, fails because it's truncated, and does the build itself. It's probably safe enough: my personal AB shares a sstate between its own builders and my own builds without any issues.
Ross
Hi Ross,
I'm sure the fact that we are still on Edison doesn't increase the odds that any potential lock protection is in place on the cache. Much like your experiences, I haven't seen anything adverse with our previous builds, but the configuration seemed odd and with potential to have problems. I figured since I was in the process of working on the build system and this was right in my face, I'd seek out some input on the configuration's validity.
Thanks,
Brian
-----Original Message-----
From: Burton, Ross [mailto:[email protected]]
Sent: Wednesday, February 05, 2014 9:34 AM
To: Brian Karcz
Cc: [email protected]
Subject: Re: [yocto] Parallel builds with common shared state cache
Quoted 5 lines · Brian Karcz
I’m curious is there are any issues with using a common shared state cache directory, outside the build directory, for multiple different image builds that also might be running in parallel to each other? I’m in the process of setting up a new build server and wanted to make sure this wasn’t an invalid configuration before proceeding.
I'm not positive that there are locks around the sstate cache (although there arguably should be if there isn't), but the worst-case scenario is that one build tries to extract a partial sstate archive, fails because it's truncated, and does the build itself. It's probably safe enough: my personal AB shares a sstate between its own builders and my own builds without any issues.
Ross