sstate.bbclass: fix parallel building issue

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Signed-off-by Roy.Li <[email protected]>

---
  meta/classes/sstate.bbclass |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/sstate.bbclass +1 -1

index c1ca54b..3e2fedd 100644
@@ -565,7 +565,7 @@ sstate_create_package () {
  	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
  	# Need to handle empty directories
  	if [ "$(ls -A)" ]; then
-		tar -czf $TFILE *
+		tar --ignore-failed-read -czf $TFILE *
  	else
  		tar -cz --file=$TFILE --files-from=/dev/null
  	fi

Quoted 7 lines · On 08/13/2013 01:20 AM, [email protected] wrote:

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

Quoted 6 lines

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed
as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

I am not sure it's so harmless, what if the file is corrupted, then we have a bad sstate tarball. You have identified the part of the root cause being the hardlink, but what if the file actually does change (which would be a different bug potentially), then your packaging a differet set of macros (in this case) with the sysroot.

Sau!

Quoted 20 lines

Signed-off-by Roy.Li <[email protected]>

---
  meta/classes/sstate.bbclass |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/sstate.bbclass +1 -1

index c1ca54b..3e2fedd 100644
@@ -565,7 +565,7 @@ sstate_create_package () {
      TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
      # Need to handle empty directories
      if [ "$(ls -A)" ]; then
-        tar -czf $TFILE *
+        tar --ignore-failed-read -czf $TFILE *
      else
          tar -cz --file=$TFILE --files-from=/dev/null
      fi

Quoted 14 lines · Saul Wold

Quoted 7 lines · On 08/13/2013 01:20 AM, [email protected] wrote:

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

Quoted 6 lines

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed
as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Quoted 8 lines

I am not sure it's so harmless, what if the file is corrupted, then we have a bad sstate tarball. You have identified the part of the root cause being the hardlink, but what if the file actually does change (which would be a different bug potentially), then your packaging a differet set of macros (in this case) with the sysroot.

Sau!

The file is not corrupted, and the file content is not changed, "tar" said xorg-macros.m4 file is changed, since the number of links of xorg-macros.m4 has changed when other packages is doing configuration and call autotools_copy_aclocal to make a hardlink to ${ACLOCALDIR}

If this fix can be accepted, I will rework the commit header.

Quoted 27 lines · Rongqing Li

Quoted 14 lines · Saul Wold

Quoted 7 lines · On 08/13/2013 01:20 AM, [email protected] wrote:

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

Quoted 6 lines

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed
as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Quoted 8 lines

I am not sure it's so harmless, what if the file is corrupted, then we have a bad sstate tarball. You have identified the part of the root cause being the hardlink, but what if the file actually does change (which would be a different bug potentially), then your packaging a differet set of macros (in this case) with the sysroot.

Sau!

Quoted 6 lines

The file is not corrupted, and the file content is not changed, "tar" said xorg-macros.m4 file is changed, since the number of links of xorg-macros.m4 has changed when other packages is doing configuration and call autotools_copy_aclocal to make a hardlink to ${ACLOCALDIR}

If this fix can be accepted, I will rework the commit header.

I think there is still some other issue.

I haven't seen this on ext4 filesystems, but with reiserfs I was able to reproduce "cp: will not create hard link" issue, e.g.:

do_populate_lic_setscene task failing in sstate_install with
cp: will not create hard link /OE/deploy/licenses/recipe' to directory /OE/deploy/licenses/recipe' (same path)

or
ERROR: Error executing a python function in pn.bb:
CalledProcessError: Command 'cp -afl /OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/* /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi' returned non-zero exit status 1 with output
cp: warning: source file `/OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/pn' specified more than once

cp: will not create hard link /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime' to directory /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime'
cp: will not create hard link /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime-reverse' to directory /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime-reverse'
cp: will not create hard link /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/shlibs' to directory /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/shlibs'

Number of hardlinks is:

$ find pn/1.0/pkgdata -printf "%f/%n/%i\n"
pkgdata/5/190867045
runtime-reverse/2/190867046
pn-dbg/1/190867047
pn-dev/1/190867048
pn-doc/1/190867049
pn/1/190867067
pn-staticdev/1/190867051
pn-locale/1/190867078
runtime/2/190867053
pn-dbg.packaged/1/190867054
pn-dev.packaged/1/190867056
pn-dbg/1/190867057
pn-dev/1/190867058
pn-doc/1/190867059
pn/1/190867060
pn-staticdev/1/190867062
pn.packaged/1/190867063
pn-locale/1/190867064
pn/1/190867065
shlibs/2/190867069

find ~ -xdev -samefile pn/1.0/pkgdata 2>/dev/null
pn/1.0/pkgdata

I'm not sure where the other pkgdata hardlinks came from.

The problem is that I can reproduce it on 1-2 random recipes from few hundreds included in bigger image and even not in every build. After the error is shown it all looks sane, only way to manually reproduce the same error is to really specify source dirs twice:

$ cp -afl \
/OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/* \
/OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/* \
/OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi

shows exactly the same 1 warning and 3 errors.

Quoted 27 lines · Rongqing Li

Quoted 14 lines · Saul Wold

Quoted 7 lines · On 08/13/2013 01:20 AM, [email protected] wrote:

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

Quoted 6 lines

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed
as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Quoted 8 lines

I am not sure it's so harmless, what if the file is corrupted, then we have a bad sstate tarball. You have identified the part of the root cause being the hardlink, but what if the file actually does change (which would be a different bug potentially), then your packaging a differet set of macros (in this case) with the sysroot.

Sau!

Quoted 6 lines

The file is not corrupted, and the file content is not changed, "tar" said xorg-macros.m4 file is changed, since the number of links of xorg-macros.m4 has changed when other packages is doing configuration and call autotools_copy_aclocal to make a hardlink to ${ACLOCALDIR}

If this fix can be accepted, I will rework the commit header.

I think there is still some other issue.

I haven't seen this on ext4 filesystems, but with reiserfs I was able to reproduce "cp: will not create hard link" issue, e.g.:

do_populate_lic_setscene task failing in sstate_install with
cp: will not create hard link /OE/deploy/licenses/recipe' to directory /OE/deploy/licenses/recipe' (same path)

or
ERROR: Error executing a python function in pn.bb:
CalledProcessError: Command 'cp -afl /OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/* /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi' returned non-zero exit status 1 with output
cp: warning: source file `/OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/pn' specified more than once

Quoted 43 lines · Martin Jansa

Quoted 27 lines · Rongqing Li

Quoted 14 lines · Saul Wold

Quoted 7 lines · On 08/13/2013 01:20 AM, [email protected] wrote:

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

Quoted 6 lines

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed
as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Quoted 8 lines

I am not sure it's so harmless, what if the file is corrupted, then we have a bad sstate tarball. You have identified the part of the root cause being the hardlink, but what if the file actually does change (which would be a different bug potentially), then your packaging a differet set of macros (in this case) with the sysroot.

Sau!

Quoted 6 lines

The file is not corrupted, and the file content is not changed, "tar" said xorg-macros.m4 file is changed, since the number of links of xorg-macros.m4 has changed when other packages is doing configuration and call autotools_copy_aclocal to make a hardlink to ${ACLOCALDIR}

If this fix can be accepted, I will rework the commit header.

I think there is still some other issue.

I haven't seen this on ext4 filesystems, but with reiserfs I was able to reproduce "cp: will not create hard link" issue, e.g.:

do_populate_lic_setscene task failing in sstate_install with
cp: will not create hard link /OE/deploy/licenses/recipe' to directory /OE/deploy/licenses/recipe' (same path)

Quoted 4 lines

or
ERROR: Error executing a python function in pn.bb:
CalledProcessError: Command 'cp -afl /OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/* /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi' returned non-zero exit status 1 with output
cp: warning: source file `/OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/pn' specified more than once

This sounds like a race issue in reiserfs to me...

Quoted 49 lines · Richard Purdie

Quoted 47 lines · Martin Jansa

Quoted 27 lines · Rongqing Li

Quoted 14 lines · Saul Wold

Quoted 7 lines · On 08/13/2013 01:20 AM, [email protected] wrote:

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

Quoted 6 lines

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed
as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Quoted 8 lines

I am not sure it's so harmless, what if the file is corrupted, then we have a bad sstate tarball. You have identified the part of the root cause being the hardlink, but what if the file actually does change (which would be a different bug potentially), then your packaging a differet set of macros (in this case) with the sysroot.

Sau!

Quoted 6 lines

The file is not corrupted, and the file content is not changed, "tar" said xorg-macros.m4 file is changed, since the number of links of xorg-macros.m4 has changed when other packages is doing configuration and call autotools_copy_aclocal to make a hardlink to ${ACLOCALDIR}

If this fix can be accepted, I will rework the commit header.

I think there is still some other issue.

I haven't seen this on ext4 filesystems, but with reiserfs I was able to reproduce "cp: will not create hard link" issue, e.g.:

do_populate_lic_setscene task failing in sstate_install with
cp: will not create hard link /OE/deploy/licenses/recipe' to directory /OE/deploy/licenses/recipe' (same path)

or
ERROR: Error executing a python function in pn.bb:
CalledProcessError: Command 'cp -afl /OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/* /OE/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi' returned non-zero exit status 1 with output
cp: warning: source file `/OE/work/armv7a-vfp-neon-oe-linux-gnueabi/pn/1.0/pkgdata/pn' specified more than once

Quoted 1 line

This sounds like a race issue in reiserfs to me...

True, I assume the same until someone else is able to reproduce it on some other filesystem.

Any idea how to confirm this theory at least to add warning in documentation that using reiserfs on build partition is causing random build failures?

OK, But your issue is not related to me.

I can reproduce my issue by two simple script.

Quoted 3 lines · Rongqing Li

OK, But your issue is not related to me.

I can reproduce my issue by two simple script.

If tar is deciding that the file has "changed" just because the link count on the dentry has increased, that sounds like it is probably a bug in tar and ought to be fixed there.

That said, I can't immediately think why autotools_copy_aclocal couldn't use a symlink rather than a hard link which would avoid this whole problem. If the file is in the sysroot then there should be no risk of it going away underneath its user.

p.

Quoted 3 lines · Rongqing Li

OK, But your issue is not related to me.

I can reproduce my issue by two simple script.

If tar is deciding that the file has "changed" just because the link count on the dentry has increased, that sounds like it is probably a bug in tar and ought to be fixed there.

That said, I can't immediately think why autotools_copy_aclocal couldn't use a symlink rather than a hard link which would avoid this whole problem. If the file is in the sysroot then there should be no risk of it going away underneath its user.

Quoted 5 lines · Phil Blundell

Quoted 3 lines · Rongqing Li

OK, But your issue is not related to me.

I can reproduce my issue by two simple script.

Quoted 8 lines

If tar is deciding that the file has "changed" just because the link count on the dentry has increased, that sounds like it is probably a bug in tar and ought to be fixed there.

That said, I can't immediately think why autotools_copy_aclocal couldn't use a symlink rather than a hard link which would avoid this whole problem. If the file is in the sysroot then there should be no risk of it going away underneath its user.

Sadly this doesn't work. We block copy a set of .m4 files from the sysroot. We can be running do_configure of package A whilst package B is de-installed from the sysroot and this leads to files disappearing whilst they're being accessed. Its turned out to be a really awkward problem to fix.

Cheers,

Richard

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 15 lines · Richard Purdie

Quoted 13 lines · Phil Blundell

Quoted 3 lines · Rongqing Li

OK, But your issue is not related to me.

I can reproduce my issue by two simple script.

If tar is deciding that the file has "changed" just because the link count on the dentry has increased, that sounds like it is probably a bug in tar and ought to be fixed there.

That said, I can't immediately think why autotools_copy_aclocal couldn't use a symlink rather than a hard link which would avoid this whole problem. If the file is in the sysroot then there should be no risk of it going away underneath its user.

Quoted 5 lines

Sadly this doesn't work. We block copy a set of .m4 files from the sysroot. We can be running do_configure of package A whilst package B is de-installed from the sysroot and this leads to files disappearing whilst they're being accessed. Its turned out to be a really awkward problem to fix.

Do we need some kind of a read/write lock on accessing those files. (Is this even something that we can do easily though the existing mechanisms?)

Quoted 16 lines · Mark Hatle

Quoted 15 lines · Richard Purdie

Quoted 13 lines · Phil Blundell

Quoted 3 lines · Rongqing Li

OK, But your issue is not related to me.

I can reproduce my issue by two simple script.

If tar is deciding that the file has "changed" just because the link count on the dentry has increased, that sounds like it is probably a bug in tar and ought to be fixed there.

That said, I can't immediately think why autotools_copy_aclocal couldn't use a symlink rather than a hard link which would avoid this whole problem. If the file is in the sysroot then there should be no risk of it going away underneath its user.

Quoted 8 lines

Quoted 5 lines

Sadly this doesn't work. We block copy a set of .m4 files from the sysroot. We can be running do_configure of package A whilst package B is de-installed from the sysroot and this leads to files disappearing whilst they're being accessed. Its turned out to be a really awkward problem to fix.

Do we need some kind of a read/write lock on accessing those files. (Is this even something that we can do easily though the existing mechanisms?)

It would kill performance for no good reason, been there, looked at it...

Cheers,

Richard

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 11 lines

Quoted 3 lines

Do we need some kind of a read/write lock on accessing those files. (Is this
even something that we can do easily though the existing mechanisms?)

It would kill performance for no good reason, been there, looked at it...

Cheers,

Richard

I think reverting the below optimization maybe better than using lock

commit 8c5544c2311b080bb212efb7f6b804db63e125f5

Author: Richard Purdie <[email protected]>
Date:   Thu Oct 11 13:36:53 2012 +0100

scripts/cp-noerror: Try and use hardlinks if possible

Since we generally have lots of copies of the directories created
using this tool, use
hardlinks where possible. This should save a little disk space and
improve performance
slightly.

(From OE-Core rev: bfa11c028c2da093f7b4e6b7b1d611da90ae052f)

Signed-off-by Richard Purdie <[email protected]>

-Roy

Quoted 31 lines · Rongqing Li

Quoted 11 lines

Quoted 3 lines

Do we need some kind of a read/write lock on accessing those files. (Is this
even something that we can do easily though the existing mechanisms?)

It would kill performance for no good reason, been there, looked at it...

Cheers,

Richard

I think reverting the below optimization maybe better than using lock

commit 8c5544c2311b080bb212efb7f6b804db63e125f5

Author: Richard Purdie <[email protected]>
Date:   Thu Oct 11 13:36:53 2012 +0100

scripts/cp-noerror: Try and use hardlinks if possible

Since we generally have lots of copies of the directories created
using this tool, use
hardlinks where possible. This should save a little disk space and
improve performance
slightly.

(From OE-Core rev: bfa11c028c2da093f7b4e6b7b1d611da90ae052f)

Signed-off-by Richard Purdie <[email protected]>

Quoted 3 lines

-Roy

I think the upper commit saves lots of space, but the saved time maybe be ignored.

Sadly this doesn't work. We block copy a set of .m4 files from the sysroot. We can be running do_configure of package A whilst package B is de-installed from the sysroot and this leads to files disappearing whilst they're being accessed. Its turned out to be a really awkward problem to fix.

From: "Roy.Li" <[email protected]>

sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then sstate_create_package will store SSTATE_BUILDDIR into a archive file by tar, but once other packages install the same file into sysroot, the creating the archive file will fail with below error:

DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed as we read it

This kind of error is harmless, use --ignore-failed-read to ignore it.

Signed-off-by Roy.Li <[email protected]>

---
 meta/classes/sstate.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/sstate.bbclass +1 -1

index c1ca54b..3e2fedd 100644
@@ -565,7 +565,7 @@ sstate_create_package () {
 	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
 	# Need to handle empty directories
 	if [ "$(ls -A)" ]; then
-		tar -czf $TFILE *
+		tar --ignore-failed-read -czf $TFILE *
 	else
 		tar -cz --file=$TFILE --files-from=/dev/null
 	fi