opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
If a the number of hard links decreases or increases while creating the tar files used for an ipk package, tar fails with error code 1, if this is the case we ignore the error and continue to create the ipk file
[YOCTO #7933]
--- .../opkg-utils/opkg-utils/tar_ignore_error.patch | 47 ++++++++++++++++++++++ meta/recipes-devtools/opkg-utils/opkg-utils_git.bb | 4 +- 2 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +47 -0
@@ -0,0 +1,47 @@ +If a the number of hard links decreases or increases while creating +the tar files used for an ipk package, tar fails with error code 1: + +| DEBUG: Executing python function do_package_ipk +| tar: ./usr/src/debug/gperf/3.0.4-r0/gperf-3.0.4/src/main.cc: file changed as we read it +NOTE: recipe gperf-3.0.4-r0: task do_package_write_ipk: Failed +ERROR: Task 6539 (recipes-extended/gperf/gperf_3.0.4.bb, do_package_write_ipk) failed with exit code '1' + + +We detect if the error code produced by tar is 1 and in this case ignore it. + +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: Inappropriate + +Signed-off-by: Alejandro Hernandez <[email protected]> + + +Index: git/opkg-build +=================================================================== +--- git.orig/opkg-build ++++ git/opkg-build +@@ -250,8 +250,21 @@ tmp_dir=$dest_dir/IPKG_BUILD.$$ + mkdir $tmp_dir + + echo $CONTROL > $tmp_dir/tarX +-( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . ) +-( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . ) ++ ++ ++# Ignore error code 1, caused by modifying the number of hard links while creating the tar file ++rc=0 ++( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . ) || rc=$? ++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then ++ exit $rc ++fi ++ ++rc=0 ++( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . ) || rc=$? ++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then ++ exit $rc ++fi ++ + rm $tmp_dir/tarX + + echo "2.0" > $tmp_dir/debian-binary
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +3 -1
@@ -10,7 +10,9 @@ PROVIDES += "virtual/update-alternatives" SRCREV = "53274f087565fd45d8452c5367997ba6a682a37a" PV = "0.1.8+git${SRCPV}" -SRC_URI = "git://git.yoctoproject.org/opkg-utils" +SRC_URI = "git://git.yoctoproject.org/opkg-utils " + +SRC_URI_append_class-native = "file://tar_ignore_error.patch" S = "${WORKDIR}/git"
Signature · 1 line
1.8.4.5
Quoted 24 lines · Alejandro Hernandez
If a the number of hard links decreases or increases while creating the tar files used for an ipk package, tar fails with error code 1, if this is the case we ignore the error and continue to create the ipk file
[YOCTO #7933]
Signed-off-by Alejandro Hernandez <[email protected]>--- .../opkg-utils/opkg-utils/tar_ignore_error.patch | 47 ++++++++++++++++++++++ meta/recipes-devtools/opkg-utils/opkg-utils_git.bb | 4 +- 2 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patchmeta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +5
-0@@ -0,0 +1,47 @@ +If a the number of hard links decreases or increases while creating +the tar files used for an ipk package, tar fails with error code 1: + +| DEBUG: Executing python function do_package_ipk +| tar: ./usr/src/debug/gperf/3.0.4-r0/gperf-3.0.4/src/main.cc: file changed as we read it
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch line 15
Quoted 10 lines
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +10
-0+NOTE: recipe gperf-3.0.4-r0: task do_package_write_ipk: Failed +ERROR: Task 6539 (recipes-extended/gperf/gperf_3.0.4.bb, do_package_write_ipk) failed with exit code '1' + + +We detect if the error code produced by tar is 1 and in this case ignore it. + +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: Inappropriate
Typo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build.
Quoted 37 lines
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +32
-0+ +Signed-off-by: Alejandro Hernandez <[email protected]> + + +Index: git/opkg-build +=================================================================== +--- git.orig/opkg-build ++++ git/opkg-build +@@ -250,8 +250,21 @@ tmp_dir=$dest_dir/IPKG_BUILD.$$ + mkdir $tmp_dir + + echo $CONTROL > $tmp_dir/tarX +-( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . ) +-( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . ) ++ ++ ++# Ignore error code 1, caused by modifying the number of hard links while creating the tar file ++rc=0 ++( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . ) || rc=$? ++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then ++ exit $rc ++fi ++ ++rc=0 ++( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . ) || rc=$? ++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then ++ exit $rc ++fi ++ + rm $tmp_dir/tarX + + echo "2.0" > $tmp_dir/debian-binarymeta/recipes-devtools/opkg-utils/opkg-utils_git.bb +0
-0
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb PROVIDES += "virtual/update-alternatives" · line 15
Quoted 7 lines
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +3
-1@@ -10,7 +10,9 @@ PROVIDES += "virtual/update-alternatives" SRCREV = "53274f087565fd45d8452c5367997ba6a682a37a" PV = "0.1.8+git${SRCPV}" -SRC_URI = "git://git.yoctoproject.org/opkg-utils" +SRC_URI = "git://git.yoctoproject.org/opkg-utils " + +SRC_URI_append_class-native = "file://tar_ignore_error.patch"
Add leading space in append not in original SRC_URI variable.
Why is it applied only for native anyway?
Quoted 11 lines
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +0
-0S = "${WORKDIR}/git"Signature · 7 lines
1.8.4.5 -- _______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 9 lines · Martin Jansa
On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +5
-0+ +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: InappropriateTypo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build.
It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using opkg on another environment
Quoted 8 lines
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +3
-1-SRC_URI = "git://git.yoctoproject.org/opkg-utils" +SRC_URI = "git://git.yoctoproject.org/opkg-utils " + +SRC_URI_append_class-native = "file://tar_ignore_error.patch"Add leading space in append not in original
SRC_URIvariable.Why is it applied only for native anyway?
Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when creating an ipk for any other reason it shouldn't be ignored; space was a typo
Quoted 19 lines · Alejandro Hernandez
Quoted 9 lines · Martin Jansa
On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +5
-0+ +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: InappropriateTypo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build.
It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using opkg on another environment
Quoted 6 lines
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +3
-1-SRC_URI = "git://git.yoctoproject.org/opkg-utils" +SRC_URI = "git://git.yoctoproject.org/opkg-utils " + +SRC_URI_append_class-native = "file://tar_ignore_error.patch"Add leading space in append not in original
SRC_URIvariable.
Quoted 4 lines
Quoted 1 line
Why is it applied only for native anyway?
Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when creating an ipk for any other reason it shouldn't be ignored; space was a typo
Why is it happening when using bitbake? This seems like an error to me - one should not be generating the tarball (i.e. packing up the final results) if the set of files being packaged is changing in any way.
Note: I'm pretty sure that error can also happen if the inode changes in many different ways, not just the number of links changing.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Hello,
Quoted 37 lines · Gary Thomas
Quoted 26 lines · Alejandro Hernandez
Quoted 11 lines · Martin Jansa
On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +5
-0+ +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: InappropriateTypo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build.
It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using
opkg on another environmentQuoted 7 lines
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +3
-1-SRC_URI = "git://git.yoctoproject.org/opkg-utils" +SRC_URI = "git://git.yoctoproject.org/opkg-utils " + +SRC_URI_append_class-native = "file://tar_ignore_error.patch"Add leading space in append not in original
SRC_URIvariable.Quoted 8 lines
Quoted 1 line
Why is it applied only for native anyway?
Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when
creating an ipk for any other reason it shouldn't be ignored; space was a typo
Quoted 8 lines
Why is it happening when using bitbake? This seems like an error to me - one should not be generating the tarball (i.e. packing up the final results) if the set of files being packaged is changing in any way.
Note: I'm pretty sure that error can also happen if the inode changes in many different ways, not just the number of links changing.
I fully agree with Gary; this also makes unreproducible the number of shared hard links in a final rootfs which is also important.
This (and dpkg related patch) seems to be covering the real bug.
(Adding Richard so he can also comment on this and to bring his attention to the concerns exposed here)
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 49 lines · Otavio Salvador
Hello,
Quoted 37 lines · Gary Thomas
Quoted 26 lines · Alejandro Hernandez
Quoted 11 lines · Martin Jansa
On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch new file +5
-0+ +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: InappropriateTypo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build.
It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using
opkg on another environmentQuoted 7 lines
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb +3
-1-SRC_URI = "git://git.yoctoproject.org/opkg-utils" +SRC_URI = "git://git.yoctoproject.org/opkg-utils " + +SRC_URI_append_class-native = "file://tar_ignore_error.patch"Add leading space in append not in original
SRC_URIvariable.Quoted 8 lines
Quoted 1 line
Why is it applied only for native anyway?
Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when
creating an ipk for any other reason it shouldn't be ignored; space was a typoQuoted 8 lines
Why is it happening when using bitbake? This seems like an error to me - one should not be generating the tarball (i.e. packing up the final results) if the set of files being packaged is changing in any way.
Note: I'm pretty sure that error can also happen if the inode changes in many different ways, not just the number of links changing.
Quoted 7 lines
I fully agree with Gary; this also makes unreproducible the number of shared hard links in a final rootfs which is also important.
This (and dpkg related patch) seems to be covering the real bug.
(Adding Richard so he can also comment on this and to bring his attention to the concerns exposed here)
The issue here is some of the attempts to decrease disk usage and increase the build speed. To do that we hardlink files in the package directories rather than create individual copies in packages/ and package-split/. There are cases where other parts of the build (e.g. populate-sysroot) can create or remote copies of the hardlinked file which changes the link count.
The files themselves do not change, only the link does and then only rarely. We fixed dpkg a while ago for this issue, this is the same fix being applied to opkg-build. The resulting packages will be the same since the hardlink count is internal to the tarball and not related to the number of original file copies. Since we control the build env, it is safe to do what we're doing here, however I would agree with Alejandro that it isn't something upstream would want to do.
Cheers,
Richard