Skip to contentSkip to conversation

devtool: modify: correctly handle multiple sources in SRC_URI

PATCH In review Julien Stephan 2 participants 6 messages Sep 27, 2023 – Oct 6, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 devtool: modify: correctly handle multiple sources in SRC_URI

[YOCTO #15162]

When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted

Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it

Here is the stack trace from devtool modify/build bzip2:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:

| DEBUG: Executing shell function do_install_ptest_base
| NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest
| sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
|            ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
| cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| ln -s /usr/bin/bzip2          <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
| cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
| WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/
1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
| WARNING: Backtrace (BB generated script):
|       #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
|       #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
|       #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226

ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +8 -0
index d53fb810071..6c8ad7a6c8c 100644
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             shutil.move(srcsubdir, srctree)
             symlink_oelocal_files_srctree(d,srctree)
 
+            # In case SRC_URI contains multiple sources
+            # they are extracted into the devtool temporary workdir
+            # We need to move them into WORKDIR, otherwise they will be lost
+            for path in os.listdir(os.path.join(tempdir, 'workdir')):
+                if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]:
+                    workdir = d.getVar('WORKDIR')
+                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
+
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
             shutil.copy2(os.path.join(tempdir, '.config'), srctree)
Signature · 1 line
2.41.0
#1 lore raw
Quoted 55 lines · Julien Stephan

[YOCTO #15162]

When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted

Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it

Here is the stack trace from devtool modify/build bzip2:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:

Quoted 10 lines

DEBUG: Executing shell function do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest

sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
           ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
Quoted 3 lines

ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/

1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'

Quoted 4 lines

WARNING: Backtrace (BB generated script):

      #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
      #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
      #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226

ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +1 -0
index d53fb810071..6c8ad7a6c8c 100644
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             shutil.move(srcsubdir, srctree)
             symlink_oelocal_files_srctree(d,srctree)
 
+            # In case SRC_URI contains multiple sources

scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697

Quoted 10 lines
scripts/lib/devtool/standard.py +7 -0
+            # they are extracted into the devtool temporary workdir
+            # We need to move them into WORKDIR, otherwise they will be lost
+            for path in os.listdir(os.path.join(tempdir, 'workdir')):
+                if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]:
+                    workdir = d.getVar('WORKDIR')
+                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
+
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
             shutil.copy2(os.path.join(tempdir, '.config'), srctree)

Looks like a good start although I worry a bit about the hardcoded list of exceptions.

Is it possible to add a test case to the devtool testsuite so this can't regress in future?

Thanks,

Richard

#2 lore raw
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan

[YOCTO #15162]

When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted

Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it

Here is the stack trace from devtool modify/build bzip2:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:

Quoted 10 lines

DEBUG: Executing shell function do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest

sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
           ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
Quoted 3 lines

ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/

1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'

Quoted 4 lines

WARNING: Backtrace (BB generated script):

      #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
      #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
      #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226

ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +0 -0
index d53fb810071..6c8ad7a6c8c 100644
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             shutil.move(srcsubdir, srctree)
             symlink_oelocal_files_srctree(d,srctree)
 
Quoted 14 lines

scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697

Quoted 11 lines
scripts/lib/devtool/standard.py +8 -0
+            # In case SRC_URI contains multiple sources
+            # they are extracted into the devtool temporary workdir
+            # We need to move them into WORKDIR, otherwise they will be lost
+            for path in os.listdir(os.path.join(tempdir, 'workdir')):
+                if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]:
+                    workdir = d.getVar('WORKDIR')
+                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
+
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
             shutil.copy2(os.path.join(tempdir, '.config'), srctree)

Looks like a good start although I worry a bit about the hardcoded list of exceptions.

Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?

Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieve SRC_URI and manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?

Quoted 3 lines

Is it possible to add a test case to the devtool testsuite so this can't regress in future?

Sure, I'll do it!

Cheers
Juliem

Quoted 1 line

sure I will add a test for that.

Quoted 3 lines

Thanks,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 79 lines · Julien Stephan via lists.openembedded.org
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan

[YOCTO #15162]

When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted

Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it

Here is the stack trace from devtool modify/build bzip2:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:

Quoted 10 lines

DEBUG: Executing shell function do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest

sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
           ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
Quoted 3 lines

ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/

1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'

Quoted 4 lines

WARNING: Backtrace (BB generated script):

      #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
      #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
      #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226

ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +0 -0
index d53fb810071..6c8ad7a6c8c 100644
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             shutil.move(srcsubdir, srctree)
             symlink_oelocal_files_srctree(d,srctree)
 
Quoted 14 lines

scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697

Quoted 11 lines
scripts/lib/devtool/standard.py +8 -0
+            # In case SRC_URI contains multiple sources
+            # they are extracted into the devtool temporary workdir
+            # We need to move them into WORKDIR, otherwise they will be lost
+            for path in os.listdir(os.path.join(tempdir, 'workdir')):
+                if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]:
+                    workdir = d.getVar('WORKDIR')
+                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
+
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
             shutil.copy2(os.path.join(tempdir, '.config'), srctree)

Looks like a good start although I worry a bit about the hardcoded list of exceptions.

Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?

Quoted 7 lines

Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieve SRC_URI and manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?

Hi Richard,
I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR.
This makes several tests from devtool selftest to fail.

Any suggestions on how I should handle this?

Cheers
Julien

Quoted 26 lines
Quoted 3 lines

Is it possible to add a test case to the devtool testsuite so this can't regress in future?

Sure, I'll do it!

Cheers
Juliem

Quoted 1 line

sure I will add a test for that.

Quoted 3 lines

Thanks,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#188455): https://lists.openembedded.org/g/openembedded-core/message/188455
Mute This Topic: https://lists.openembedded.org/mt/101621289/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#5 lore raw
Quoted 89 lines · Julien Stephan
Quoted 79 lines · Julien Stephan via lists.openembedded.org
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan

[YOCTO #15162]

When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted

Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it

Here is the stack trace from devtool modify/build bzip2:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:

Quoted 10 lines

DEBUG: Executing shell function do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest

sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
           ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
Quoted 3 lines

ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/

1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'

Quoted 4 lines

WARNING: Backtrace (BB generated script):

      #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
      #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
      #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226

ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +0 -0
index d53fb810071..6c8ad7a6c8c 100644
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             shutil.move(srcsubdir, srctree)
             symlink_oelocal_files_srctree(d,srctree)
 
Quoted 14 lines

scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697

Quoted 11 lines
scripts/lib/devtool/standard.py +8 -0
+            # In case SRC_URI contains multiple sources
+            # they are extracted into the devtool temporary workdir
+            # We need to move them into WORKDIR, otherwise they will be lost
+            for path in os.listdir(os.path.join(tempdir, 'workdir')):
+                if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]:
+                    workdir = d.getVar('WORKDIR')
+                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
+
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
             shutil.copy2(os.path.join(tempdir, '.config'), srctree)

Looks like a good start although I worry a bit about the hardcoded list of exceptions.

Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?

Quoted 6 lines

Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieve SRC_URI and manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?

Quoted 7 lines

I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR.
This makes several tests from devtool selftest to fail.

Any suggestions on how I should handle this?

There is some work on the bitbake list from Alberto which adds unpack tracing. I'm wondering if by default we should track some really basic form of unpack accesses which would then give you a list of target locations to merge?

Cheers,

Richard

#6 lore raw
Quoted 97 lines · Richard Purdie
Quoted 95 lines · Julien Stephan
Quoted 79 lines · Julien Stephan via lists.openembedded.org
Quoted 56 lines · Richard Purdie
Quoted 54 lines · Julien Stephan

[YOCTO #15162]

When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted

Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it

Here is the stack trace from devtool modify/build bzip2:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:

Quoted 10 lines

DEBUG: Executing shell function do_install_ptest_base
NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest

sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
           ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
Quoted 3 lines

ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/

1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'

Quoted 4 lines

WARNING: Backtrace (BB generated script):

      #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
      #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
      #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226

ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +0 -0
index d53fb810071..6c8ad7a6c8c 100644
@@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             shutil.move(srcsubdir, srctree)
             symlink_oelocal_files_srctree(d,srctree)
 
Quoted 14 lines

scripts/lib/devtool/standard.py def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works · line 697

Quoted 11 lines
scripts/lib/devtool/standard.py +8 -0
+            # In case SRC_URI contains multiple sources
+            # they are extracted into the devtool temporary workdir
+            # We need to move them into WORKDIR, otherwise they will be lost
+            for path in os.listdir(os.path.join(tempdir, 'workdir')):
+                if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]:
+                    workdir = d.getVar('WORKDIR')
+                    shutil.move(os.path.join(tempdir, 'workdir', path), workdir)
+
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
             shutil.copy2(os.path.join(tempdir, '.config'), srctree)

Looks like a good start although I worry a bit about the hardcoded list of exceptions.

Hi Richard,
Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources?

Quoted 6 lines

Here are the alternatives I can see:
* encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists
* retrieve SRC_URI and manually build the list of secondary sources to move. This looks a little bit overkill here...
What do you think?

I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR.
This makes several tests from devtool selftest to fail.

Quoted 7 lines
Quoted 1 line

Any suggestions on how I should handle this?

There is some work on the bitbake list from Alberto which adds unpack tracing. I'm wondering if by default we should track some really basic form of unpack accesses which would then give you a list of target locations to merge?

Looks very interesting, let me check this :)

Cheers
Julien

Quoted 7 lines

Cheers,

Richard