Robert Yang : perf: fix for rebuilding
I'm still seeing issues when perf is rebuilt
The original issue was:
OSError: [Errno 2] No such file or directory: 'BUILD/work/qemux86-webos-linux/perf/1.0-r9/perf-1.0'
as described here:
http://patches.openembedded.org/patch/94719/
First part of fix is now backported to dizzy:
commit 40aeb23c427e0c56f7073fc54ea63bf0eabbe092
Author: Robert Yang <[email protected]> Date: Wed Mar 11 01:35:38 2015 -0700
license.bbclass: set dirs for do_populate_lic_setscene
but perf is still failing in some builds with the same error when do_configure and do_populate_lic are executed at the same time:
NOTE: recipe perf-1.0-r9: task do_configure: Started
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
ERROR: Build of do_populate_lic failed
ERROR: Traceback (most recent call last):
File "bitbake/lib/bb/build.py", line 497, in exec_task
return _exec_task(fn, task, d, quieterr)
File "bitbake/lib/bb/build.py", line 440, in _exec_task
exec_func(func, localdata)
File "bitbake/lib/bb/build.py", line 212, in exec_func
exec_func_python(func, d, runfile, cwd=adir)
File "bitbake/lib/bb/build.py", line 237, in exec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '....../perf/1.0-r9/perf-1.0'
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 3826 (../oe-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
So is there some 2nd/3rd part of this fix or is it broken in master?
I haven't found any other commit related to this perf configure or populate_lic in master and fido.
Regards,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 32 lines · Martin Jansa
I'm still seeing issues when perf is rebuilt
The original issue was:
OSError: [Errno 2] No such file or directory: 'BUILD/work/qemux86-webos-linux/perf/1.0-r9/perf-1.0'as described here:
http://patches.openembedded.org/patch/94719/First part of fix is now backported to dizzy:
commit40aeb23c427e0c56f7073fc54ea63bf0eabbe092Author: Robert Yang <[email protected]> Date: Wed Mar 11 01:35:38 2015 -0700license.bbclass: set dirs for
do_populate_lic_setscenebut perf is still failing in some builds with the same error when
do_configureanddo_populate_licare executed at the same time:NOTE: recipe perf-1.0-r9: task
do_configure: Started
NOTE: recipe perf-1.0-r9: taskdo_populate_lic: Started
ERROR: Build ofdo_populate_licfailed
ERROR: Traceback (most recent call last):
File "bitbake/lib/bb/build.py", line 497, inexec_task
return_exec_task(fn, task, d, quieterr)
File "bitbake/lib/bb/build.py", line 440, in_exec_task
exec_func(func, localdata)
File "bitbake/lib/bb/build.py", line 212, inexec_func
exec_func_python(func, d, runfile, cwd=adir)
File "bitbake/lib/bb/build.py", line 237, inexec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '....../perf/1.0-r9/perf-1.0'
Quoted 8 lines
NOTE: recipe perf-1.0-r9: task
do_populate_lic: Failed
ERROR: Task 3826 (../oe-core/meta/recipes-kernel/perf/perf.bb,do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: taskdo_configure: SucceededSo is there some 2nd/3rd part of this fix or is it broken in master?
I haven't found any other commit related to this perf configure or
populate_licin master and fido.
It's still broken in master.
Easy way to reproduce this is:
OE qemux86@ ~/build/oe-core $ for i in seq -w 1 100; do bitbake -c cleansstate perf; bitbake -c build perf | tee log.perf.$i; done
On slow machine I got first failure on 60th iteration, on much faster machine I got 4 failures in first 20 iterations.
NOTE: recipe perf-1.0-r9: task do_configure: Started
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
ERROR: Function failed: do_configure (log file is located at /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_configure.10700)
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_configure.10700 Log data follows:
| DEBUG: Executing python function sysroot_cleansstate | DEBUG: Python function sysroot_cleansstate finished | DEBUG: Executing shell function do_configure | mkdir: cannot create directory '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0/': File exists | WARNING: /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/run.do_configure.10700:1 exit 1 from | mkdir /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0/ | ERROR: Function failed: do_configure (log file is located at /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_configure.10700)
NOTE: recipe perf-1.0-r9: task do_configure: Failed
ERROR: Task 5 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_configure) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_populate_lic: Succeeded
NOTE: Tasks Summary: Attempted 833 tasks of which 828 didn't need to be rerun and 1 failed.
Can you please fix it properly instead of removing whole ${B}?
do_configure_prepend () {
# Fix for rebuilding
rm -rf ${B}/
mkdir ${B}/...
I wish there is easier way to undo this commit in .bbappend, we very rarely rebuild perf in the same build directory, but too often we see normal clean builds failing in perf do_configure or do_populate_lic tasks :/.
Regards and thanks for feedback (it's broken for 8 months and no reply after it was reported 3 months ago).
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
PATCH 1/1 Revert "perf: fix for rebuilding"
* This reverts commit 9dafa571ed0a40d21a886dec7704c31150b21942.
* The "fix" is causing more issues then it's fixing and there was no feedback in many months.
* More info:
http://lists.openembedded.org/pipermail/openembedded-core/2015-June/105684.html
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/109870.html
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/110338.html
--- meta/recipes-kernel/perf/perf.bb | 4 ---- 1 file changed, 4 deletions(-)
meta/recipes-kernel/perf/perf.bb +0 -4
@@ -118,10 +118,6 @@ do_install() { } do_configure_prepend () { - # Fix for rebuilding - rm -rf ${B}/ - mkdir ${B}/ - # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit # build, with a 64 bit multilib, the arch won't match and the detection of a
Signature · 1 line
2.5.1
Re: [PATCH][master][fido][dizzy] Revert "perf: fix for rebuilding"
Hi Martin,
Sorry, I'm blocked by mutitlib SDK issues recently, I will fix it sooner. Revert the patch is fine to me if it is urgent.
// Robert
Quoted 30 lines · Martin Jansa
* This reverts commit
9dafa571ed0a40d21a886dec7704c31150b21942.* The "fix" is causing more issues then it's fixing and there was no feedback in many months.
* More info:http://lists.openembedded.org/pipermail/openembedded-core/2015-June/105684.html http://lists.openembedded.org/pipermail/openembedded-core/2015-September/109870.html http://lists.openembedded.org/pipermail/openembedded-core/2015-September/110338.htmlSigned-off-by Martin Jansa <[email protected]>--- meta/recipes-kernel/perf/perf.bb | 4 ---- 1 file changed, 4 deletions(-)meta/recipes-kernel/perf/perf.bb +0
-4@@ -118,10 +118,6 @@ do_install() { } do_configure_prepend () { - # Fix for rebuilding - rm -rf ${B}/ - mkdir ${B}/ - # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit # build, with a 64 bit multilib, the arch won't match and the detection of a
Re: [PATCH][master][fido][dizzy] Revert "perf: fix for rebuilding"
Hi Martin,
I've figured out the reason, this is because the task's default cwd is ${B}, the easier way to fix the problem is use mkdir -p rather than mkdir, autotools.bbclass also has this problem, but I'd like to fix exec_func, I will send a patch to bitbake-devel after more testing.
// Robert
Quoted 46 lines · Robert Yang
Hi Martin,
Sorry, I'm blocked by mutitlib SDK issues recently, I will fix it sooner. Revert the patch is fine to me if it is urgent.
// Robert
Quoted 37 lines · Martin Jansa
* This reverts commit
9dafa571ed0a40d21a886dec7704c31150b21942.* The "fix" is causing more issues then it's fixing and there was no feedback in many months.
* More info:http://lists.openembedded.org/pipermail/openembedded-core/2015-June/105684.html
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/109870.html
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/110338.html
Signed-off-by Martin Jansa <[email protected]>--- meta/recipes-kernel/perf/perf.bb | 4 ---- 1 file changed, 4 deletions(-)meta/recipes-kernel/perf/perf.bb +0
-4@@ -118,10 +118,6 @@ do_install() { } do_configure_prepend () { - # Fix for rebuilding - rm -rf ${B}/ - mkdir ${B}/ - # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a32 bit
# build, with a 64 bit multilib, the arch won't match and the detection
of a
Re: [PATCH][master][fido][dizzy] Revert "perf: fix for rebuilding"
Quoted 5 lines · Robert Yang
I've figured out the reason, this is because the task's default cwd is ${B}, the easier way to fix the problem is use mkdir -p rather than mkdir, autotools.bbclass also has this problem, but I'd like to fix
exec_func, I will send a patch to bitbake-devel after more testing.
Changing the default $B of [dirs] is not something we'll be doing post-M3. I've a branch that changes that default and have fixed all the obvious breakage in oe-core already so this is something on my plan for immediately after 2.1 branches off.
Ross
Re: [PATCH][master][fido][dizzy] Revert "perf: fix for rebuilding"
Quoted 11 lines · Ross
On 16 September 2015 at 08:48, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
I've figured out the reason, this is because the task's default cwd is ${B}, the easier way to fix the problem is use mkdir -p rather than mkdir, autotools.bbclass also has this problem, but I'd like to fix
exec_func, I will send a patch to bitbake-devel after more testing.Changing the default $B of [dirs] is not something we'll be doing post-M3. I've
Thanks, so let's simply use "mkdir -p" to fix the problem, atm ?
Quoted 3 lines
a branch that changes that default and have fixed all the obvious breakage in oe-core already so this is something on my plan for immediately after 2.1 branches off.
I made 2 patches just now, one is for bitbake, and one for insane.bbclass,
maybe we are doing the similar things, I can drop them then.
build.py: default exec_func's cwd to WORKDIR
This can fix a few problems:
- The ${B} was nearly always created in the past after any tak runs, for example, the ${B} exists after do_clean or do_cleansstate (first removed, then created), but ${B} is useless and confused end user in this case, the similar to a lot of prefuncs and postfuncs which also create ${B}, but not used. This patch fixes the problem.
- This can fix race issue when we use the following commands in other tasks:
rm -rf ${B}
mkdir ${B}
such as autotools.bbclass and perf.bb.
When the 'dirs' or 'cleandirs' is not specified by task, which means that they are not important, so default to WORKDIR which is more common and usually existed.
bitbake/lib/bb/build.py +3 -2
@@ -182,8 +182,9 @@ def exec_func(func, d, dirs = None): bb.utils.mkdirhier(adir) adir = dirs[-1] else: - adir = d.getVar('B', True) - bb.utils.mkdirhier(adir) + adir = d.getVar('WORKDIR', True) + if not os.path.exists(adir): + bb.utils.mkdirhier(adir) ispython = flags.get('python')
===================
insane.bbclass: only do_qa_unpack warn when SRC_URI is not null
If SRC_URI is null, then no source is needed to be unpacked, thus no warn.
meta/classes/insane.bbclass +1 -1
@@ -1214,7 +1214,7 @@ python do_qa_unpack() { bb.note("Checking has ${S} been created") s_dir = d.getVar('S', True) - if not os.path.exists(s_dir): + if not os.path.exists(s_dir) and d.getVar('SRC_URI', True): bb.warn('%s: the directory %s (%s) pointed to by the S variable
doesn\'t exist - please set S within the recipe to point to where the source has been unpacked to' % (d.getVar('PN', T
}
// Robert
Quoted 2 lines
Ross
Re: [PATCH][master][fido][dizzy] Revert "perf: fix for rebuilding"
In reply to Robert YangQuoted 7 lines · Robert Yang
Hi Martin,
I've figured out the reason, this is because the task's default cwd is ${B}, the easier way to fix the problem is use mkdir -p rather than mkdir, autotools.bbclass also has this problem, but I'd like to fix
exec_func, I will send a patch to bitbake-devel after more testing.
Unfortunately "mkdir -p" fixes only part of this problem, it still fails quite often when bitbake tries to "os.chdir(cwd)" in populate_lic after do_configure removed ${B} and before do_configure re-created it with "mkdir -p".
So 3 from 50 iterations (if I use the same for cycle to test this) fail with:
NOTE: recipe perf-1.0-r9: task do_configure: Started
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
ERROR: Build of do_populate_lic failed
ERROR: Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 553, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 493, in _exec_task
exec_func(func, localdata)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 227, in exec_func
exec_func_python(func, d, runfile, cwd=adir)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 252, in exec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
or even longer exception:
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
NOTE: recipe perf-1.0-r9: task do_configure: Started
ERROR: Error executing a python function in /OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'sstate_task_postfunc', lineno: 14, function: <module>
0010: sstate_package(shared_state, d)
0011: os.umask(omask)
0012:
0013:
*** 0014:sstate_task_postfunc(d)
0015:
File: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc
0001:
0002:def sstate_task_postfunc(d):
0003: shared_state = sstate_state_fromvars(d) *** 0004: sstate_install(shared_state, d)
0005: for intercept in shared_state['interceptfuncs']:
0006: bb.build.exec_func(intercept, d)
0007: omask = os.umask(002)
0008: if omask != 002:
File: 'sstate.bbclass', lineno: 113, function: sstate_install
0109: if os.path.exists(state[1]):
0110: oe.path.copyhardlinktree(state[1], state[2])
0111:
0112: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
*** 0113: bb.build.exec_func(postinst, d)
0114:
0115: for lock in locks:
0116: bb.utils.unlockfile(lock)
0117:File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 227, function: exec_func
0223: pass
0224:
0225: with bb.utils.fileslocked(lockfiles):
0226: if ispython:
*** 0227: exec_func_python(func, d, runfile, cwd=adir)
0228: else:
0229: exec_func_shell(func, d, runfile, cwd=adir)
0230:
0231:_functionfmt = """File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 252, function: exec_func_python
0248: try:
0249: olddir = os.getcwd()
0250: except OSError:
0251: olddir = None
*** 0252: os.chdir(cwd)
0253:
0254: bb.debug(2, "Executing python function %s" % func)
0255:
0256: try:Exception: OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
ERROR: Function failed: sstate_task_postfunc
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_populate_lic.16588
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
NOTE: Tasks Summary: Attempted 840 tasks of which 835 didn't need to be rerun and 1 failed.
So this revert (or more complete fix) is still needed in master, fido and dizzy.
Regards,
Quoted 50 lines
// Robert
Quoted 46 lines · Robert Yang
Hi Martin,
Sorry, I'm blocked by mutitlib SDK issues recently, I will fix it sooner. Revert the patch is fine to me if it is urgent.
// Robert
Quoted 37 lines · Martin Jansa
* This reverts commit
9dafa571ed0a40d21a886dec7704c31150b21942.* The "fix" is causing more issues then it's fixing and there was no feedback in many months.
* More info:http://lists.openembedded.org/pipermail/openembedded-core/2015-June/105684.html
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/109870.html
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/110338.html
Signed-off-by Martin Jansa <[email protected]>--- meta/recipes-kernel/perf/perf.bb | 4 ---- 1 file changed, 4 deletions(-)meta/recipes-kernel/perf/perf.bb +0
-4@@ -118,10 +118,6 @@ do_install() { } do_configure_prepend () { - # Fix for rebuilding - rm -rf ${B}/ - mkdir ${B}/ - # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a32 bit
# build, with a 64 bit multilib, the arch won't match and the detection
of a
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
RFC PATCH 1/1 perf: don't re-create ${B}
* otherwise there is race-condition between do_configure rm+mkdir and bitbake trying to use it as CWD for do_populate_lic task, which results in errors like this:
NOTE: recipe perf-1.0-r9: task do_configure: Started
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
ERROR: Build of do_populate_lic failed
ERROR: Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 553, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 493, in _exec_task
exec_func(func, localdata)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 227, in exec_func
exec_func_python(func, d, runfile, cwd=adir)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 252, in exec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
or even longer exception:
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
NOTE: recipe perf-1.0-r9: task do_configure: Started
ERROR: Error executing a python function in /OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'sstate_task_postfunc', lineno: 14, function: <module>
0010: sstate_package(shared_state, d)
0011: os.umask(omask)
0012:
0013:
*** 0014:sstate_task_postfunc(d)
0015:
File: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc
0001:
0002:def sstate_task_postfunc(d):
0003: shared_state = sstate_state_fromvars(d) *** 0004: sstate_install(shared_state, d)
0005: for intercept in shared_state['interceptfuncs']:
0006: bb.build.exec_func(intercept, d)
0007: omask = os.umask(002)
0008: if omask != 002:
File: 'sstate.bbclass', lineno: 113, function: sstate_install
0109: if os.path.exists(state[1]):
0110: oe.path.copyhardlinktree(state[1], state[2])
0111:
0112: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
*** 0113: bb.build.exec_func(postinst, d)
0114:
0115: for lock in locks:
0116: bb.utils.unlockfile(lock)
0117:File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 227, function: exec_func
0223: pass
0224:
0225: with bb.utils.fileslocked(lockfiles):
0226: if ispython:
*** 0227: exec_func_python(func, d, runfile, cwd=adir)
0228: else:
0229: exec_func_shell(func, d, runfile, cwd=adir)
0230:
0231:_functionfmt = """File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 252, function: exec_func_python
0248: try:
0249: olddir = os.getcwd()
0250: except OSError:
0251: olddir = None
*** 0252: os.chdir(cwd)
0253:
0254: bb.debug(2, "Executing python function %s" % func)
0255:
0256: try:Exception: OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
ERROR: Function failed: sstate_task_postfunc
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_populate_lic.16588
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
NOTE: Tasks Summary: Attempted 840 tasks of which 835 didn't need to be rerun and 1 failed.
--- meta/recipes-kernel/perf/perf.bb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
meta/recipes-kernel/perf/perf.bb +1 -2
@@ -119,8 +119,7 @@ do_install() { do_configure_prepend () { # Fix for rebuilding - rm -rf ${B}/ - mkdir -p ${B}/ + find ${B} -mindepth 1 -maxdepth 1 -delete # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit
Signature · 1 line
2.5.1
RFC PATCH v2 1/1 cmake.bbclass, perf: don't re-create ${B}
In reply to Martin Jansa* otherwise there is race-condition between do_configure rm+mkdir and bitbake trying to use it as CWD for do_populate_lic task, which results in errors like this:
NOTE: recipe perf-1.0-r9: task do_configure: Started
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
ERROR: Build of do_populate_lic failed
ERROR: Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 553, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 493, in _exec_task
exec_func(func, localdata)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 227, in exec_func
exec_func_python(func, d, runfile, cwd=adir)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 252, in exec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
or even longer exception:
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
NOTE: recipe perf-1.0-r9: task do_configure: Started
ERROR: Error executing a python function in /OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'sstate_task_postfunc', lineno: 14, function: <module>
0010: sstate_package(shared_state, d)
0011: os.umask(omask)
0012:
0013:
*** 0014:sstate_task_postfunc(d)
0015:
File: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc
0001:
0002:def sstate_task_postfunc(d):
0003: shared_state = sstate_state_fromvars(d) *** 0004: sstate_install(shared_state, d)
0005: for intercept in shared_state['interceptfuncs']:
0006: bb.build.exec_func(intercept, d)
0007: omask = os.umask(002)
0008: if omask != 002:
File: 'sstate.bbclass', lineno: 113, function: sstate_install
0109: if os.path.exists(state[1]):
0110: oe.path.copyhardlinktree(state[1], state[2])
0111:
0112: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
*** 0113: bb.build.exec_func(postinst, d)
0114:
0115: for lock in locks:
0116: bb.utils.unlockfile(lock)
0117:File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 227, function: exec_func
0223: pass
0224:
0225: with bb.utils.fileslocked(lockfiles):
0226: if ispython:
*** 0227: exec_func_python(func, d, runfile, cwd=adir)
0228: else:
0229: exec_func_shell(func, d, runfile, cwd=adir)
0230:
0231:_functionfmt = """File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 252, function: exec_func_python
0248: try:
0249: olddir = os.getcwd()
0250: except OSError:
0251: olddir = None
*** 0252: os.chdir(cwd)
0253:
0254: bb.debug(2, "Executing python function %s" % func)
0255:
0256: try:Exception: OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
ERROR: Function failed: sstate_task_postfunc
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_populate_lic.16588
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
NOTE: Tasks Summary: Attempted 840 tasks of which 835 didn't need to be rerun and 1 failed.
--- meta/classes/cmake.bbclass | 3 +-- meta/recipes-kernel/perf/perf.bb | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-)
meta/classes/cmake.bbclass +1 -2
@@ -91,8 +91,7 @@ cmake_do_configure() { fi if [ "${S}" != "${B}" ]; then - rm -rf ${B} - mkdir -p ${B} + find ${B} -mindepth 1 -maxdepth 1 | xargs rm -rf cd ${B} else find ${B} -name CMakeFiles -or -name Makefile -or -name cmake_install.cmake -or -name CMakeCache.txt -delete
meta/recipes-kernel/perf/perf.bb +1 -2
@@ -119,8 +119,7 @@ do_install() { do_configure_prepend () { # Fix for rebuilding - rm -rf ${B}/ - mkdir -p ${B}/ + find ${B} -mindepth 1 -maxdepth 1 | xargs rm -rf # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit
Signature · 1 line
2.5.1
Re: [RFC][PATCHv2] cmake.bbclass, perf: don't re-create ${B}
Quoted 4 lines · Martin Jansa
* otherwise there is race-condition between
do_configurerm+mkdir and bitbake trying to use it as CWD fordo_populate_lictask, which results in errors like this:
It took me a few minutes to understand this race:
Task A executing perf:do_configure and deleting ${B}
Task B finishing perf:do_populate_lic and so running the sstate generation logic and the functions listed in SSTATEPOSTINSTFUNCS, which get executed with the default cwd of ${B}.
I'll note that autotools.bbclass also does rmdir/mkdir ${B} so I'm curious why you see it more in perf and cmake and not everything else.
Would a better long-term fix be to pass cwd=${WORKDIR} to exec_func in the sstate code to stop the functions being executed in ${B} (after vetting the functions, obviously). FWIW I'm the owner of the bug to stop bitbake knowing about ${B} which is leading to exactly this sort of problem.
Ross
Re: [RFC][PATCHv2] cmake.bbclass, perf: don't re-create ${B}
Quoted 7 lines · Ross
Quoted 4 lines · Martin Jansa
* otherwise there is race-condition between
do_configurerm+mkdir and bitbake trying to use it as CWD fordo_populate_lictask, which results in errors like this:
Quoted 6 lines
It took me a few minutes to understand this race:
Task A executing perf:do_configure and deleting ${B}
Task B finishing perf:do_populate_lic and so running the sstate generation logic and the functions listed in SSTATEPOSTINSTFUNCS, which get executed with the default cwd of ${B}.
Thanks for more detailed explanation, sorry if I wasn't clear on this one in my previous e-mails.
Quoted 2 lines
I'll note that autotools.bbclass also does rmdir/mkdir ${B} so I'm curious why you see it more in perf and cmake and not everything else.
perf for some reason shows this most often, most our internal components are using cmake, that's probably cmake was easier to find in our logs, but I didn't grep all of them, maybe there are similar exceptions in autotools based components. I'll send v3 with autotools.bbclass fix as well.
Quoted 4 lines
Would a better long-term fix be to pass cwd=${WORKDIR} to
exec_funcin the sstate code to stop the functions being executed in ${B} (after vetting the functions, obviously). FWIW I'm the owner of the bug to stop bitbake knowing about ${B} which is leading to exactly this sort of problem.
Yes, I agree it would be better, but as you said, such change is too late even for master branch and I need to fix this long-standing issue even in fido and dizzy, so smaller, simpler change will be easier to get backported.
My for cycle finished 469 iterations over night and no exception since
this fix:
OE @ ~/build/oe-core $ grep "ERROR: Task" log.perf.00*
log.perf.0003:ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1' log.perf.0004:ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1' log.perf.0031:ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1' log.perf.0055:ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1' log.perf.0062:ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
Regards,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
RFC PATCH v3 1/1 autotools.bbclass, cmake.bbclass, perf: don't re-create ${B}
In reply to Martin Jansa* otherwise there is race-condition between do_configure rm+mkdir and bitbake trying to use it as CWD for do_populate_lic task, which results in errors like this:
NOTE: recipe perf-1.0-r9: task do_configure: Started
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
ERROR: Build of do_populate_lic failed
ERROR: Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 553, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 493, in _exec_task
exec_func(func, localdata)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 227, in exec_func
exec_func_python(func, d, runfile, cwd=adir)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 252, in exec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
or even longer exception:
NOTE: recipe perf-1.0-r9: task do_populate_lic: Started
NOTE: recipe perf-1.0-r9: task do_configure: Started
ERROR: Error executing a python function in /OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'sstate_task_postfunc', lineno: 14, function: <module>
0010: sstate_package(shared_state, d)
0011: os.umask(omask)
0012:
0013:
*** 0014:sstate_task_postfunc(d)
0015:
File: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc
0001:
0002:def sstate_task_postfunc(d):
0003: shared_state = sstate_state_fromvars(d) *** 0004: sstate_install(shared_state, d)
0005: for intercept in shared_state['interceptfuncs']:
0006: bb.build.exec_func(intercept, d)
0007: omask = os.umask(002)
0008: if omask != 002:
File: 'sstate.bbclass', lineno: 113, function: sstate_install
0109: if os.path.exists(state[1]):
0110: oe.path.copyhardlinktree(state[1], state[2])
0111:
0112: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
*** 0113: bb.build.exec_func(postinst, d)
0114:
0115: for lock in locks:
0116: bb.utils.unlockfile(lock)
0117:File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 227, function: exec_func
0223: pass
0224:
0225: with bb.utils.fileslocked(lockfiles):
0226: if ispython:
*** 0227: exec_func_python(func, d, runfile, cwd=adir)
0228: else:
0229: exec_func_shell(func, d, runfile, cwd=adir)
0230:
0231:_functionfmt = """File: '/OE/build/oe-core/bitbake/lib/bb/build.py', lineno: 252, function: exec_func_python
0248: try:
0249: olddir = os.getcwd()
0250: except OSError:
0251: olddir = None
*** 0252: os.chdir(cwd)
0253:
0254: bb.debug(2, "Executing python function %s" % func)
0255:
0256: try:Exception: OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
ERROR: Function failed: sstate_task_postfunc
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/temp/log.do_populate_lic.16588
NOTE: recipe perf-1.0-r9: task do_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb, do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: task do_configure: Succeeded
NOTE: Tasks Summary: Attempted 840 tasks of which 835 didn't need to be rerun and 1 failed.
--- meta/classes/autotools.bbclass | 3 +-- meta/classes/cmake.bbclass | 3 +-- meta/recipes-kernel/perf/perf.bb | 3 +-- 3 files changed, 3 insertions(+), 6 deletions(-)
meta/classes/autotools.bbclass +1 -2
@@ -104,8 +104,7 @@ autotools_preconfigure() { if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then if [ "${S}" != "${B}" ]; then echo "Previously configured separate build directory detected, cleaning ${B}" - rm -rf ${B} - mkdir -p ${B} + find ${B} -mindepth 1 -maxdepth 1 | xargs rm -rf else # At least remove the .la files since automake won't automatically # regenerate them even if CFLAGS/LDFLAGS are different
meta/classes/cmake.bbclass +1 -2
@@ -91,8 +91,7 @@ cmake_do_configure() { fi if [ "${S}" != "${B}" ]; then - rm -rf ${B} - mkdir -p ${B} + find ${B} -mindepth 1 -maxdepth 1 | xargs rm -rf cd ${B} else find ${B} -name CMakeFiles -or -name Makefile -or -name cmake_install.cmake -or -name CMakeCache.txt -delete
meta/recipes-kernel/perf/perf.bb +1 -2
@@ -119,8 +119,7 @@ do_install() { do_configure_prepend () { # Fix for rebuilding - rm -rf ${B}/ - mkdir -p ${B}/ + find ${B} -mindepth 1 -maxdepth 1 | xargs rm -rf # If building a multlib based perf, the incorrect library path will be # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit
Signature · 1 line
2.5.1
Re: [RFC][PATCHv2] cmake.bbclass, perf: don't re-create ${B}
In reply to Martin JansaQuoted 45 lines · Martin Jansa
* otherwise there is race-condition between
do_configurerm+mkdir and bitbake trying to use it as CWD fordo_populate_lictask, which results in errors like this:NOTE: recipe perf-1.0-r9: task
do_configure: Started
NOTE: recipe perf-1.0-r9: taskdo_populate_lic: Started
ERROR: Build ofdo_populate_licfailed
ERROR: Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 553, inexec_task
return_exec_task(fn, task, d, quieterr)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 493, in_exec_task
exec_func(func, localdata)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 227, inexec_func
exec_func_python(func, d, runfile, cwd=adir)
File "/OE/build/oe-core/bitbake/lib/bb/build.py", line 252, inexec_func_python
os.chdir(cwd)
OSError: [Errno 2] No such file or directory: '/OE/build/oe-core/tmp-glibc/work/qemux86-oe-linux/perf/1.0-r9/perf-1.0'
NOTE: recipe perf-1.0-r9: taskdo_populate_lic: Failed
ERROR: Task 7 (/OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb,do_populate_lic) failed with exit code '1'
NOTE: recipe perf-1.0-r9: taskdo_configure: Succeededor even longer exception:
NOTE: recipe perf-1.0-r9: task
do_populate_lic: Started
NOTE: recipe perf-1.0-r9: taskdo_configure: Started
ERROR: Error executing a python function in /OE/build/oe-core/openembedded-core/meta/recipes-kernel/perf/perf.bb:The stack trace of python calls that resulted in this exception/failure was:
File: 'sstate_task_postfunc', lineno: 14, function: <module>0010: sstate_package(shared_state, d) 0011: os.umask(omask) 0012: 0013: *** 0014:sstate_task_postfunc(d) 0015: File: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc 0001: 0002:def sstate_task_postfunc(d): 0003: shared_state = sstate_state_fromvars(d)*** 0004:
sstate_install(shared_state, d)
0005: for intercept in shared_state['interceptfuncs']:
0006:bb.build.exec_func(intercept, d)
0007: omask =os.umask(002)
0008: if omask != 002:
File: 'sstate.bbclass', lineno: 113, function:sstate_install
Quoted 9 lines
0109: if os.path.exists(state[1]):
0110: oe.path.copyhardlinktree(state[1], state[2])
0111:
0112: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
*** 0113: bb.build.exec_func(postinst, d)
0114:
0115: for lock in locks:
0116: bb.utils.unlockfile(lock)
0117:The longer stack trace is actually really helpful. I believe the code here should inherit the working directory of the current task rather than rely on the default of ${B} which bitbake assigns if there is nothing else.
The trouble is that if the function does specify a dirs flag, it should really use that instead of a supplied parameter and if we use the parameter to exec_func(), it won't do that.
There aren't that many uses of SSTATEPOSTINSTFUNCS in our metadata so the easiest way to fix this short term is probably to go through them and add specific [dirs] options to the functions.
Cheers,
Richard
Re: [RFC][PATCHv2] cmake.bbclass, perf: don't re-create ${B}
Quoted 5 lines · Richard Purdie
The longer stack trace is actually really helpful. I believe the code here should inherit the working directory of the current task rather than rely on the default of ${B} which bitbake assigns if there is nothing else.
Quoted 8 lines
The trouble is that if the function does specify a dirs flag, it should really use that instead of a supplied parameter and if we use the parameter to
exec_func(), it won't do that.There aren't that many uses of SSTATEPOSTINSTFUNCS in our metadata so the easiest way to fix this short term is probably to go through them and add specific [dirs] options to the functions.
There's enough instances of SSTATEPOSTINSTFUNCS dotted around even just oe-core, but they're all going to be using SSTATE_INSTDIR as the working directory. Whilst you have a valid point that the functions should respect [dirs] if set, for the case of sstate post-install hooks I think we should just force the right path so the recipe creators don't have to worry about this.
Ross