Skip to contentSkip to conversation

Add locking on tasks that add package_update_index_ipk

PATCH In review Tom Rini 3 participants 4 messages Jun 5, 2010 – Jun 7, 2010

lore
Share & tools
lore

Tom Rini

author
#0 lore raw

PATCH 1/1 Add locking on tasks that add package_update_index_ipk

We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that is also the lockfile that package_update_index_ipk uses. With enough threads it is possible that one image (or meta-toolchain) is attempting to use the package index while another has only reached the point of generating the index leading to an empty index being seen later on.

classes/rootfs_ipk.bbclass +1 -0
index 9fa1d5e..0303c54 100644
@@ -6,6 +6,7 @@
 #
 
 do_rootfs[depends] += "opkg-native:do_populate_sysroot"
+do_rootfs[lockfiles] = "${DEPLOY_DIR_IPK}.lock"
 
 IPKG_TMP_DIR = "${IMAGE_ROOTFS}-tmp"
 IPKG_ARGS = "-f ${IPKGCONF_TARGET} -o ${IMAGE_ROOTFS} -t

${IPKG_TMP_DIR} ${@base_conditional("PACKAGE_INSTALL_NO_DEPS", "1", "-nodeps", "", d)}"

recipes/meta/canadian-sdk.bb +1 -0
index 47529a2..2160441 100644
@@ -169,5 +169,6 @@ do_populate_sdk() {
 }
 
 do_populate_sdk[nostamp] = "1"
+do_populate_sdk[lockfiles] = "${DEPLOY_DIR_IPK}.lock"
 addtask package_update_index_ipk before do_populate_sdk
 addtask populate_sdk before do_build after do_install
recipes/meta/meta-toolchain.bb +1 -0
index 064fc3b..82918ad 100644
@@ -177,5 +177,6 @@ do_populate_sdk() {
 }
 
 do_populate_sdk[nostamp] = "1"
+do_populate_sdk[lockfiles] = "${DEPLOY_DIR_IPK}.lock"
 addtask package_update_index_ipk before do_populate_sdk
 addtask populate_sdk before do_build after do_install
Signature · 2 lines
Tom Rini
Mentor Graphics Corporation
Acked-by on v1 1/1
#1 lore raw
Quoted 5 lines · Tom Rini

We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that is also the lockfile that package_update_index_ipk uses. With enough threads it is possible that one image (or meta-toolchain) is attempting to use the package index while another has only reached the point of generating the index leading to an empty index being seen later on.

Acked-by Chris Larson <[email protected]>
Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
Acked-by on v1 1/1
#2 lore raw
In reply to Tom Rini

В сообщении от Суббота 05 июня 2010 04:03:59 автор Tom Rini написал:

Quoted 5 lines

We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that is also the lockfile that package_update_index_ipk uses. With enough threads it is possible that one image (or meta-toolchain) is attempting to use the package index while another has only reached the point of generating the index leading to an empty index being seen later on.

Am I right that effectively it makes rootfs creation sequential? It's bad for performance. Probably it's time to fix ipkg-make-index to not touch index files if no new packages are found as was suggested some time ago:

http://patchwork.openembedded.org/patch/744/

But thinking about it a bit more, ipkg-make-index hack won't really fix the issue in this case, as we can have image A with a set of packages, image B with a similar set of packages plus additional package P and then A would do package_update_index_ipk and enter do_rootfs, then P gets completed, package_update_index_ipk for B is started and the package index really needs an update, thus things can break again.

Maybe it's easier to make those indexes per-image for performance.

Well,

Acked-by Roman I Khimov <[email protected]>

for now.

Signature · 3 lines
 http://roman.khimov.ru
mailto: [email protected]
gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3
author
#3 lore raw
Quoted 8 lines · Roman I Khimov

В сообщении от Суббота 05 июня 2010 04:03:59 автор Tom Rini написал:

Quoted 5 lines

We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that is also the lockfile that package_update_index_ipk uses. With enough threads it is possible that one image (or meta-toolchain) is attempting to use the package index while another has only reached the point of generating the index leading to an empty index being seen later on.

Am I right that effectively it makes rootfs creation sequential? It's bad for performance.

Yes, this does introduce a bottleneck at these points.

Quoted 7 lines

Probably it's time to fix ipkg-make-index to not touch index files if no new packages are found as was suggested some time ago:

http://patchwork.openembedded.org/patch/744/

But thinking about it a bit more, ipkg-make-index hack won't really fix the issue in this case, as we can have image A with a set of packages, image B with a similar set of packages plus additional package P and then A would do package_update_index_ipk and enter do_rootfs, then P gets completed, package_update_index_ipk for B is started and the package index really needs an update, thus things can break again.

Maybe it's easier to make those indexes per-image for performance.

Right. The problem is that we really need the index to be per user of the feed. I don't know enough about ipkg to say if we could move the indices to ${WORKDIR} and keep the packages where they are (and then, what do folks that populate a feed need?) but that would solve this particular problem.

Signature · 2 lines
Tom Rini
Mentor Graphics Corporation