Skip to contentSkip to conversation

python: prevent lib32-python depending on python

PATCH In review Martin Jansa 2 participants 2 messages Sep 7, 2020 – Sep 11, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 python: prevent lib32-python depending on python

* it was a bit tricky to find this, because

  $ bitbake -g lib32-python
  $ grep '"python\.' task-depends.dot
  doesn't show anything, but if you set
  COMPATIBLE_HOST_pn-python = "^(?!x86_64).*"
  $ bitbake lib32-python
  will fail with:
    NOTE: Resolving any missing task queue dependencies
    ERROR: Nothing PROVIDES 'python'
    python was skipped: incompatible with host x86_64-oe-linux (not in COMPATIBLE_HOST)
    ERROR: Required build target 'lib32-python' has no buildable providers.
    Missing or unbuildable dependency chain was: ['lib32-python', 'python']

so I was wondering where this dependency was created when it isn't part of task-depends.dot

in the end I've added small debug output in bitbake:

    diff --git a/lib/bb/taskdata.py b/lib/bb/taskdata.py
    index ffbaf362..e7fab3cc 100644
    --- a/lib/bb/taskdata.py
    +++ b/lib/bb/taskdata.py
    @@ -70,6 +70,7 @@ class TaskData:
             """

             task_deps = dataCache.task_deps[fn]
    +        logger.verbose('add_tasks for %s: %s' % (fn, task_deps))

             if fn in self.failed_fns:
                 bb.msg.fatal("TaskData", "Trying to re-add a failed file? Something is broken...")

which showed exactly where the issue was:
NOTE: add_tasks for virtual:multilib:lib32:/OE/meta-python2/recipes-devtools/python/python_2.7.18.bb: {'tasks': ['do_patch', 'do_populate_sysroot', 'do_populate_sysroot_setscene', 'do_prepare_recipe_sysroot', 'do_listtasks', 'do_clean', 'do_checkuri', 'do_fetch', 'do_unpack', 'do_configure', 'do_compile', 'do_install', 'do_build', 'do_cleansstate', 'do_cleanall', 'do_rm_work_all', 'do_rm_work', 'do_build_without_rm_work', 'do_package_qa', 'do_package_qa_setscene', 'do_package', 'do_package_setscene', 'do_packagedata', 'do_packagedata_setscene', 'do_package_write_ipk_setscene', 'do_package_write_ipk', 'do_devshell', 'do_devpyshell', 'do_populate_lic', 'do_populate_lic_setscene', 'do_create_manifest'], 'parents': {'do_patch': ['do_unpack'], 'do_populate_sysroot': ['do_install'], 'do_populate_sysroot_setscene': [], 'do_prepare_recipe_sysroot': ['do_fetch'], 'do_listtasks': [], 'do_clean': [], 'do_checkuri': [], 'do_fetch': [], 'do_unpack': ['do_fetch'], 'do_configure': ['do_prepare_recipe_sysroot', 'do_patch'], 'do_compile': ['do_configure'], 'do_install': ['do_compile'], 'do_build': ['do_rm_work', 'do_populate_lic', 'do_packagedata', 'do_package_qa', 'do_rm_work_all', 'do_populate_sysroot'], 'do_cleansstate': ['do_clean'], 'do_cleanall': ['do_cleansstate'], 'do_rm_work_all': [], 'do_rm_work': ['do_deploy', 'do_package_qa', 'do_package_write_ipk', 'do_packagedata', 'do_populate_lic', 'do_populate_sysroot'], 'do_build_without_rm_work': ['do_deploy', 'do_package_qa', 'do_package_write_ipk', 'do_packagedata', 'do_populate_lic', 'do_populate_sysroot'], 'do_package_qa': ['do_packagedata', 'do_package'], 'do_package_qa_setscene': [], 'do_package': ['do_install'], 'do_package_setscene': [], 'do_packagedata': ['do_package'], 'do_packagedata_setscene': [], 'do_package_write_ipk_setscene': [], 'do_package_write_ipk': ['do_packagedata', 'do_package'], 'do_devshell': ['do_patch', 'do_prepare_recipe_sysroot'], 'do_devpyshell': ['do_patch'], 'do_populate_lic': ['do_patch'], 'do_populate_lic_setscene': [], 'do_create_manifest': []}, 'mcdepends': {}, 'depends': {'do_patch': 'quilt-native:do_populate_sysroot patch-replacement-native:do_populate_sysroot', 'do_populate_sysroot': ' virtual/lib32-i686-oemllib32-linux-binutils:do_populate_sysroot', 'do_prepare_recipe_sysroot': ' virtual/fakeroot-native:do_populate_sysroot', 'do_fetch': ' wget-native:do_populate_sysroot', 'do_unpack': ' xz-native:do_populate_sysroot', 'do_install': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_qa': ' virtual/lib32-i686-oemllib32-linux-binutils:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot', 'do_package': ' virtual/fakeroot-native:do_populate_sysroot rpm-native:do_populate_sysroot dwarfsrcfiles-native:do_populate_sysroot file-native:do_populate_sysroot', 'do_package_setscene': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_write_ipk': ' opkg-utils-native:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot xz-native:do_populate_sysroot', 'do_devshell': ' virtual/fakeroot-native:do_populate_sysroot', 'do_create_manifest': ' python:do_prepare_recipe_sysroot python:do_patch'}, 'rdepends': {}, 'deptask': {'do_prepare_recipe_sysroot': 'do_populate_sysroot', 'do_package': ' do_packagedata'}, 'rdeptask': {'do_package_qa': 'do_packagedata', 'do_package_write_ipk': 'do_packagedata'}, 'recrdeptask': {'do_build': ' do_deploy do_package_write_ipk', 'do_rm_work_all': 'do_rm_work'}, 'recideptask': {}, 'nostamp': {'do_listtasks': '1', 'do_clean': '1', 'do_checkuri': '1', 'do_cleansstate': '1', 'do_cleanall': '1', 'do_devshell': '1', 'do_devpyshell': '1'}, 'fakeroot': {'do_install': '1', 'do_package_qa': '1', 'do_package': '1', 'do_package_setscene': '1', 'do_package_write_ipk': '1'}, 'noexec': {'do_build': '1', 'do_rm_work_all': '1', 'do_build_without_rm_work': '1'}, 'umask': {'do_patch': '022', 'do_populate_sysroot': '022', 'do_unpack': '022', 'do_configure': '022', 'do_compile': '022', 'do_install': '022', 'do_package': '022', 'do_package_write_ipk': '022'}}

* now the fix was simple (the same fix is already in oe-core's python3 recipe,
since https://git.openembedded.org/openembedded-core/commit/?id=02714c105426b0d687620913c1a7401b386428b6)

* and now when I know what task was causing it, it can be shown with bitbake -g as well:

  $ bitbake -g lib32-python -c create_manifest
    NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 43095, PID: 47453
    Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
    Loaded 7076 entries from dependency cache.
    NOTE: Resolving any missing task queue dependencies
    NOTE: PN build list saved to 'pn-buildlist'
    NOTE: Task dependencies saved to 'task-depends.dot'
  $ grep '"python\.' task-depends.dot
"lib32-python.do_create_manifest" -> "python.do_patch"
"lib32-python.do_create_manifest" -> "python.do_prepare_recipe_sysroot"
"python.do_fetch" [label="python do_fetch\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/python_2.7.18.bb"]
"python.do_patch" [label="python do_patch\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/python_2.7.18.bb"]
"python.do_patch" -> "patch-native.do_populate_sysroot"
"python.do_patch" -> "python.do_unpack"
"python.do_patch" -> "quilt-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" [label="python do_prepare_recipe_sysroot\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/python_2.7.18.bb"]
"python.do_prepare_recipe_sysroot" -> "autoconf-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "automake-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "bzip2.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "db.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "gcc-cross-x86_64.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "gcc-runtime.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "gdbm.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "glibc.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "gnu-config-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libffi.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libtool-cross.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libtool-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libxcrypt.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "openssl.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "pkgconfig-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "pseudo-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "python-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "python.do_fetch"
"python.do_prepare_recipe_sysroot" -> "readline.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "sqlite3.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "zlib.do_populate_sysroot"
"python.do_unpack" [label="python do_unpack\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/python_2.7.18.bb"]
"python.do_unpack" -> "python.do_fetch"
"python.do_unpack" -> "xz-native.do_populate_sysroot"

* I've included all these debugging steps, to discuss whether "bitbake -g" should show also the "optional" tasks - no task from python.do_build depends on do_create_manifest tasks, it's only called manually with:
bitbake python -c create_manifest
but still bitbake includes this task dependency when preparing runqueue or maybe bitbake shouldn't include it in the dependency tree when it's not going to call that task anyway

---
 recipes-devtools/python/python_2.7.18.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
recipes-devtools/python/python_2.7.18.bb +2 -2
index be7c6d2..bca2910 100644
@@ -256,5 +256,5 @@ do_create_manifest() {
 addtask do_create_manifest
 
 # Make sure we have native python ready when we create a new manifest
-do_create_manifest[depends] += "python:do_prepare_recipe_sysroot"
-do_create_manifest[depends] += "python:do_patch"
+do_create_manifest[depends] += "${PN}:do_prepare_recipe_sysroot"
+do_create_manifest[depends] += "${PN}:do_patch"
Signature · 1 line
2.25.1
#1 lore raw

Merged. Thank you.
It would be nice to have a test case for this.

Quoted 194 lines · Martin Jansa

* it was a bit tricky to find this, because

  $ bitbake -g lib32-python
  $ grep '"python\.' task-depends.dot
  doesn't show anything, but if you set
  COMPATIBLE_HOST_pn-python = "^(?!x86_64).*"
  $ bitbake lib32-python
  will fail with:
    NOTE: Resolving any missing task queue dependencies
    ERROR: Nothing PROVIDES 'python'
    python was skipped: incompatible with host x86_64-oe-linux (not in
COMPATIBLE_HOST)
    ERROR: Required build target 'lib32-python' has no buildable providers.
    Missing or unbuildable dependency chain was: ['lib32-python', 'python']

so I was wondering where this dependency was created when it isn't part of task-depends.dot

in the end I've added small debug output in bitbake:

    diff --git a/lib/bb/taskdata.py b/lib/bb/taskdata.py
    index ffbaf362..e7fab3cc 100644
    --- a/lib/bb/taskdata.py
    +++ b/lib/bb/taskdata.py
    @@ -70,6 +70,7 @@ class TaskData:
             """

             task_deps = dataCache.task_deps[fn]
    +        logger.verbose('add_tasks for %s: %s' % (fn, task_deps))

             if fn in self.failed_fns:
                 bb.msg.fatal("TaskData", "Trying to re-add a failed file?

Something is broken...")

which showed exactly where the issue was:
NOTE: add_tasks for
virtual:multilib:lib32:/OE/meta-python2/recipes-devtools/python/ python_2.7.18.bb: {'tasks': ['do_patch', 'do_populate_sysroot', 'do_populate_sysroot_setscene', 'do_prepare_recipe_sysroot', 'do_listtasks', 'do_clean', 'do_checkuri', 'do_fetch', 'do_unpack', 'do_configure', 'do_compile', 'do_install', 'do_build', 'do_cleansstate', 'do_cleanall', 'do_rm_work_all', 'do_rm_work', 'do_build_without_rm_work', 'do_package_qa', 'do_package_qa_setscene', 'do_package', 'do_package_setscene', 'do_packagedata', 'do_packagedata_setscene', 'do_package_write_ipk_setscene', 'do_package_write_ipk', 'do_devshell', 'do_devpyshell', 'do_populate_lic', 'do_populate_lic_setscene', 'do_create_manifest'], 'parents': {'do_patch': ['do_unpack'], 'do_populate_sysroot': ['do_install'], 'do_populate_sysroot_setscene': [], 'do_prepare_recipe_sysroot': ['do_fetch'], 'do_listtasks': [], 'do_clean':
[], 'do_checkuri': [], 'do_fetch': [], 'do_unpack': ['do_fetch'], 'do_configure': ['do_prepare_recipe_sysroot', 'do_patch'], 'do_compile':
['do_configure'], 'do_install': ['do_compile'], 'do_build': ['do_rm_work', 'do_populate_lic', 'do_packagedata', 'do_package_qa', 'do_rm_work_all', 'do_populate_sysroot'], 'do_cleansstate': ['do_clean'], 'do_cleanall':
['do_cleansstate'], 'do_rm_work_all': [], 'do_rm_work': ['do_deploy', 'do_package_qa', 'do_package_write_ipk', 'do_packagedata', 'do_populate_lic', 'do_populate_sysroot'], 'do_build_without_rm_work':
['do_deploy', 'do_package_qa', 'do_package_write_ipk', 'do_packagedata', 'do_populate_lic', 'do_populate_sysroot'], 'do_package_qa':
['do_packagedata', 'do_package'], 'do_package_qa_setscene': [], 'do_package': ['do_install'], 'do_package_setscene': [], 'do_packagedata':
['do_package'], 'do_packagedata_setscene': [], 'do_package_write_ipk_setscene': [], 'do_package_write_ipk':
['do_packagedata', 'do_package'], 'do_devshell': ['do_patch', 'do_prepare_recipe_sysroot'], 'do_devpyshell': ['do_patch'], 'do_populate_lic': ['do_patch'], 'do_populate_lic_setscene': [], 'do_create_manifest': []}, 'mcdepends': {}, 'depends': {'do_patch':
'quilt-native:do_populate_sysroot patch-replacement-native:do_populate_sysroot', 'do_populate_sysroot': ' virtual/lib32-i686-oemllib32-linux-binutils:do_populate_sysroot', 'do_prepare_recipe_sysroot': ' virtual/fakeroot-native:do_populate_sysroot', 'do_fetch': ' wget-native:do_populate_sysroot', 'do_unpack': ' xz-native:do_populate_sysroot', 'do_install': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_qa': ' virtual/lib32-i686-oemllib32-linux-binutils:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot', 'do_package': ' virtual/fakeroot-native:do_populate_sysroot rpm-native:do_populate_sysroot dwarfsrcfiles-native:do_populate_sysroot file-native:do_populate_sysroot', 'do_package_setscene': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_write_ipk': ' opkg-utils-native:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot xz-native:do_populate_sysroot', 'do_devshell': ' virtual/fakeroot-native:do_populate_sysroot', 'do_create_manifest': ' python:do_prepare_recipe_sysroot python:do_patch'}, 'rdepends': {}, 'deptask': {'do_prepare_recipe_sysroot':
'do_populate_sysroot', 'do_package': ' do_packagedata'}, 'rdeptask':
{'do_package_qa': 'do_packagedata', 'do_package_write_ipk':
'do_packagedata'}, 'recrdeptask': {'do_build': ' do_deploy do_package_write_ipk', 'do_rm_work_all': 'do_rm_work'}, 'recideptask': {}, 'nostamp': {'do_listtasks': '1', 'do_clean': '1', 'do_checkuri': '1', 'do_cleansstate': '1', 'do_cleanall': '1', 'do_devshell': '1', 'do_devpyshell': '1'}, 'fakeroot': {'do_install': '1', 'do_package_qa':
'1', 'do_package': '1', 'do_package_setscene': '1', 'do_package_write_ipk':
'1'}, 'noexec': {'do_build': '1', 'do_rm_work_all': '1', 'do_build_without_rm_work': '1'}, 'umask': {'do_patch': '022', 'do_populate_sysroot': '022', 'do_unpack': '022', 'do_configure': '022', 'do_compile': '022', 'do_install': '022', 'do_package': '022', 'do_package_write_ipk': '022'}}

* now the fix was simple (the same fix is already in oe-core's python3 recipe,
since
https://git.openembedded.org/openembedded-core/commit/?id=02714c105426b0d687620913c1a7401b386428b6
)

* and now when I know what task was causing it, it can be shown with bitbake -g as well:

  $ bitbake -g lib32-python -c create_manifest
    NOTE: Started PRServer with DBfile:
/OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 43095, PID:
47453
    Loading cache: 100%
|###################################################################################################################################################################################################################################|
Time: 0:00:00
    Loaded 7076 entries from dependency cache.
    NOTE: Resolving any missing task queue dependencies
    NOTE: PN build list saved to 'pn-buildlist'
    NOTE: Task dependencies saved to 'task-depends.dot'
  $ grep '"python\.' task-depends.dot
"lib32-python.do_create_manifest" -> "python.do_patch"
"lib32-python.do_create_manifest" -> "python.do_prepare_recipe_sysroot"
"python.do_fetch" [label="python
do_fetch\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/
python_2.7.18.bb"]
"python.do_patch" [label="python
do_patch\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/
python_2.7.18.bb"]
"python.do_patch" -> "patch-native.do_populate_sysroot"
"python.do_patch" -> "python.do_unpack"
"python.do_patch" -> "quilt-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" [label="python
do_prepare_recipe_sysroot\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/
python_2.7.18.bb"]
"python.do_prepare_recipe_sysroot" -> "autoconf-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "automake-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "bzip2.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "db.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" ->
"gcc-cross-x86_64.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "gcc-runtime.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "gdbm.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "glibc.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" ->
"gnu-config-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libffi.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libtool-cross.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libtool-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "libxcrypt.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "openssl.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" ->
"pkgconfig-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "pseudo-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "python-native.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "python.do_fetch"
"python.do_prepare_recipe_sysroot" -> "readline.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "sqlite3.do_populate_sysroot"
"python.do_prepare_recipe_sysroot" -> "zlib.do_populate_sysroot"
"python.do_unpack" [label="python
do_unpack\n:2.7.18-r0\n/OE/build/oe-core/meta-python2/recipes-devtools/python/
python_2.7.18.bb"]
"python.do_unpack" -> "python.do_fetch"
"python.do_unpack" -> "xz-native.do_populate_sysroot"

* I've included all these debugging steps, to discuss whether "bitbake -g" should
show also the "optional" tasks - no task from python.do_build depends on do_create_manifest tasks, it's only called manually with:
bitbake python -c create_manifest
but still bitbake includes this task dependency when preparing runqueue
or
maybe bitbake shouldn't include it in the dependency tree when it's not going to call that task anyway

---
 recipes-devtools/python/python_2.7.18.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
recipes-devtools/python/python_2.7.18.bb +2 -3
index be7c6d2..bca2910 100644
@@ -256,5 +256,5 @@ do_create_manifest() {
 addtask do_create_manifest
 
 # Make sure we have native python ready when we create a new manifest
-do_create_manifest[depends] += "python:do_prepare_recipe_sysroot"
-do_create_manifest[depends] += "python:do_patch"
+do_create_manifest[depends] += "${PN}:do_prepare_recipe_sysroot"
+do_create_manifest[depends] += "${PN}:do_patch"
--

2.25.1