devtool: modify: correctly handle multiple sources in SRC_URI
PATCH 1/1 devtool: modify: correctly handle multiple sources in SRC_URI
[YOCTO #15162]
When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted
Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it
Here is the stack trace from devtool modify/build bzip2:
NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:
| DEBUG: Executing shell function do_install_ptest_base
| NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest
| sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
| ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
| cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
| cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
| WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/
1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
| WARNING: Backtrace (BB generated script):
| #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
| #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
| #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base
--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +8 -0
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree) + # In case SRC_URI contains multiple sources + # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)
Signature · 1 line
2.41.0
Quoted 55 lines · Julien Stephan
[YOCTO #15162]
When recipes such as bzip2 have several sources in
SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deletedFix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it
Here is the stack trace from devtool modify/build bzip2:
NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:Quoted 10 lines
DEBUG: Executing shell function
do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptestsed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Quoted 3 lines
ln -s
/usr/bin/bzip2<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
Quoted 4 lines
WARNING: Backtrace (BB generated script):
#1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_baseSigned-off-by Julien Stephan <[email protected]>--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)scripts/lib/devtool/standard.py +1
-0@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree) + # In case SRC_URI contains multiple sources
scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697
Quoted 10 lines
scripts/lib/devtool/standard.py +7
-0+ # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)
Looks like a good start although I worry a bit about the hardcoded list of exceptions.
Is it possible to add a test case to the devtool testsuite so this can't regress in future?
Thanks,
Richard
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan
[YOCTO #15162]
When recipes such as bzip2 have several sources in
SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deletedFix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it
Here is the stack trace from devtool modify/build bzip2:
NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:Quoted 10 lines
DEBUG: Executing shell function
do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptestsed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Quoted 3 lines
ln -s
/usr/bin/bzip2<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
Quoted 4 lines
WARNING: Backtrace (BB generated script):
#1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_baseSigned-off-by Julien Stephan <[email protected]>--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)scripts/lib/devtool/standard.py +0
-0@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree)
Quoted 14 lines
scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697
Quoted 11 lines
scripts/lib/devtool/standard.py +8
-0+ # In case SRC_URI contains multiple sources + # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)Looks like a good start although I worry a bit about the hardcoded list of exceptions.
Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?
Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieve SRC_URI and manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?
Quoted 3 lines
Is it possible to add a test case to the devtool testsuite so this can't regress in future?
Sure, I'll do it!
Cheers
Juliem
Quoted 1 line
sure I will add a test for that.
Quoted 3 lines
Thanks,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 79 lines · Julien Stephan via lists.openembedded.org
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan
[YOCTO #15162]
When recipes such as bzip2 have several sources in
SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deletedFix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it
Here is the stack trace from devtool modify/build bzip2:
NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:Quoted 10 lines
DEBUG: Executing shell function
do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptestsed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Quoted 3 lines
ln -s
/usr/bin/bzip2<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
Quoted 4 lines
WARNING: Backtrace (BB generated script):
#1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_baseSigned-off-by Julien Stephan <[email protected]>--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)scripts/lib/devtool/standard.py +0
-0@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree)Quoted 14 lines
scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697
Quoted 11 lines
scripts/lib/devtool/standard.py +8
-0+ # In case SRC_URI contains multiple sources + # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)Looks like a good start although I worry a bit about the hardcoded list of exceptions.
Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?
Quoted 7 lines
Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieveSRC_URIand manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?
Hi Richard,
I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR.
This makes several tests from devtool selftest to fail.
Any suggestions on how I should handle this?
Cheers
Julien
Quoted 26 lines
Quoted 3 lines
Is it possible to add a test case to the devtool testsuite so this can't regress in future?
Sure, I'll do it!
Cheers
JuliemQuoted 1 line
sure I will add a test for that.
Quoted 3 lines
Thanks,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188455): https://lists.openembedded.org/g/openembedded-core/message/188455
Mute This Topic: https://lists.openembedded.org/mt/101621289/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 89 lines · Julien Stephan
Quoted 79 lines · Julien Stephan via lists.openembedded.org
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan
[YOCTO #15162]
When recipes such as bzip2 have several sources in
SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deletedFix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it
Here is the stack trace from devtool modify/build bzip2:
NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:Quoted 10 lines
DEBUG: Executing shell function
do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptestsed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Quoted 3 lines
ln -s
/usr/bin/bzip2<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
Quoted 4 lines
WARNING: Backtrace (BB generated script):
#1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_baseSigned-off-by Julien Stephan <[email protected]>--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)scripts/lib/devtool/standard.py +0
-0@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree)Quoted 14 lines
scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697
Quoted 11 lines
scripts/lib/devtool/standard.py +8
-0+ # In case SRC_URI contains multiple sources + # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)Looks like a good start although I worry a bit about the hardcoded list of exceptions.
Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?Quoted 6 lines
Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieveSRC_URIand manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?
Quoted 7 lines
I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR.
This makes several tests from devtool selftest to fail.Any suggestions on how I should handle this?
There is some work on the bitbake list from Alberto which adds unpack tracing. I'm wondering if by default we should track some really basic form of unpack accesses which would then give you a list of target locations to merge?
Cheers,
Richard
Quoted 97 lines · Richard Purdie
Quoted 95 lines · Julien Stephan
Quoted 79 lines · Julien Stephan via lists.openembedded.org
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan
[YOCTO #15162]
When recipes such as bzip2 have several sources in
SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deletedFix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it
Here is the stack trace from devtool modify/build bzip2:
NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:Quoted 10 lines
DEBUG: Executing shell function
do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptestsed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Quoted 3 lines
ln -s
/usr/bin/bzip2<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
Quoted 4 lines
WARNING: Backtrace (BB generated script):
#1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_baseSigned-off-by Julien Stephan <[email protected]>--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)scripts/lib/devtool/standard.py +0
-0@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree)Quoted 14 lines
scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697
Quoted 11 lines
scripts/lib/devtool/standard.py +8
-0+ # In case SRC_URI contains multiple sources + # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)Looks like a good start although I worry a bit about the hardcoded list of exceptions.
Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?Quoted 6 lines
Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieveSRC_URIand manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR.
This makes several tests from devtool selftest to fail.
Quoted 7 lines
Quoted 1 line
Any suggestions on how I should handle this?
There is some work on the bitbake list from Alberto which adds unpack tracing. I'm wondering if by default we should track some really basic form of unpack accesses which would then give you a list of target locations to merge?
Looks very interesting, let me check this :)
Cheers
Julien
Quoted 7 lines
Cheers,
Richard