locking in do_package_tar

Hey folks,

I would like to introduce locking in do_package_tar. I had a case here were do_package_tar and do_package_ipk where running parallel. Because of this "tar: .: file changed as we read it" occurs. I fixed it by introduce locking in do_package_tar. What do you think about it?

Signature · 12 lines

------- ROAD ...the handyPC Company - - -  ) ) )

Jens Erdmann
Software Development

ROAD GmbH
Bennigsenstr. 14 | 12159 Berlin | Germany
fon: +49 (30) 230069 - 64 | fax: +49 (30) 230069 - 69
url: www.road.de

Amtsgericht Charlottenburg: HRB 96688 B
Managing director: Hans-Peter Constien

classes/package_tar.bbclass +4 -0

index b905e17..47bf2cc 100644
@@ -58,6 +58,7 @@ python do_package_tar () {
 		localdata = bb.data.createCopy(d)
 		pkgdest = bb.data.getVar('PKGDEST', d, 1)
 		root = "%s/%s" % (pkgdest, pkg)
+		lf = bb.utils.lockfile(root + ".lock")
 
 		bb.data.setVar('ROOT', '', localdata)
 		bb.data.setVar('ROOT_%s' % pkg, root, localdata)

@@ -65,6 +66,7 @@ python do_package_tar () {
 
 		overrides = bb.data.getVar('OVERRIDES', localdata)
 		if not overrides:
+			bb.utils.unlockfile(lf)
 			raise bb.build.FuncFailed('OVERRIDES not defined')
 		overrides = bb.data.expand(overrides, localdata)
 		bb.data.setVar('OVERRIDES', '%s:%s' % (overrides, pkg), localdata)

@@ -82,10 +84,12 @@ python do_package_tar () {
 		from glob import glob
 		if not glob('*') + glob('.[!.]*'):
 			bb.note("Not creating empty archive for %s-%s" % (pkg, bb.data.expand('${PV}-${PR}${DISTRO_PR}', d, True)))
+			bb.utils.unlockfile(lf)
 			continue
 		ret = os.system("tar -czf %s %s" % (tarfn, '.'))
 		if ret != 0:
 			bb.error("Creation of tar %s failed." % tarfn)
+		bb.utils.unlockfile(lf)
 }
 
 python () {

_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Quoted 6 lines · Jens Erdmann

Hey folks,

I would like to introduce locking in do_package_tar. I had a case here were do_package_tar and do_package_ipk where running parallel. Because of this "tar: .: file changed as we read it" occurs. I fixed it by introduce locking in do_package_tar. What do you think about it?

Ah, I've been suffering from this problem for quite some time now, when making parallel builds with BB_NUMBER_THREADS. I was trying to debug it and noticed it would fail when kernel recipe would try to do_deploy at the same time another package would try to do_package_stage, so I was digging in the packaged staging direction. Tom Rini even made a patch to add an extra lock in there, which didn't help with my issue...
So, let me try your patch and see if this problem is gone for good! Thanks.

Quoted 8 lines · Denys Dmytriyenko

Quoted 6 lines · Jens Erdmann

Hey folks,

I would like to introduce locking in do_package_tar. I had a case here were do_package_tar and do_package_ipk where running parallel. Because of this "tar: .: file changed as we read it" occurs. I fixed it by introduce locking in do_package_tar. What do you think about it?

Quoted 7 lines

Ah, I've been suffering from this problem for quite some time now, when making parallel builds with BB_NUMBER_THREADS. I was trying to debug it and noticed it would fail when kernel recipe would try to do_deploy at the same time another package would try to do_package_stage, so I was digging in the packaged staging direction. Tom Rini even made a patch to add an extra lock in there, which didn't help with my issue...
So, let me try your patch and see if this problem is gone for good! Thanks.

It may solve some other similar issue, but unfortunately, it didn't seem to work for me, as it failed again the same way with the patch applied:

NOTE: Running task 2011 of 2173 (ID: 953, /OE/arago-oe-dev/recipes/freetype/freetype_2.3.9.bb, do_package_stage) /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/ /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/ /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/ /OE/arago-tmp/work/armv7a-none-linux-gnueabi/freetype-2.3.9-r1/staging-pkg/deploy/ipk/armv7a/
ERROR: function staging_packager failed
ERROR: log data follows (/OE/arago-tmp/work/dm3730-am3715-evm-none-linux-gnueabi/linux-omap-psp-2.6.32-r60+gitr7b8926aa626991fa087b00f6bbc1fb6b0e8269b0/temp/log.staging_packager.19650) | tar: .: file changed as we read it
NOTE: Task failed: /OE/arago-tmp/work/dm3730-am3715-evm-none-linux-gnueabi/linux-omap-psp-2.6.32-r60+gitr7b8926aa626991fa087b00f6bbc1fb6b0e8269b0/temp/log.staging_packager.19650
NOTE: oestats: task failed, see http://tinderbox.openembedded.net/packages/556734/
ERROR: TaskFailed event exception, aborting
ERROR: Build of /OE/arago-oe-dev/recipes/linux/linux-omap-psp_2.6.32.bb do_package_stage failed
ERROR: Task 1009 (/OE/arago-oe-dev/recipes/linux/linux-omap-psp_2.6.32.bb, do_package_stage) failed
NOTE: Waiting for 3 active tasks to finish
NOTE: 1: /OE/arago-oe-dev/recipes/freetype/freetype_2.3.9.bb, do_package_stage (21747)

So, it looks like my problem is actually still packaged staging related, as is fails when 2 recipes run do_package_stage task at the same time (in the example above, it's freetype and linux-omap-psp)...