rm_work.bbclass: reserve 'image' folder when remove WORKDIR
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 4da3cd3c9adebd3279b376a1f7aa2a0353484ecd.1296023140.git.dongxiao.xu@intel.com
Op 26 jan 2011, om 07:30 heeft [email protected] het volgende geschreven:
Quoted 10 lines
From: Dongxiao Xu <[email protected]>
After the implementation of per machine sysroot,
do_populate_sysrootanddo_packageneed to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine'sdo_populate_sysrootanddo_packagewill fail due to the missing of "image" directory.Thus reserve 'image' folder when doing
rm_work.
Wouldn't adding do_install to sstate be a better idea? OE was designed that it wouldn't depend on WORKDIR after building, so this patch is a step in the wrong direction.
regards,
Koen
Quoted 12 lines · Koen Kooi
Op 26 jan 2011, om 07:30 heeft [email protected] het volgende geschreven:
Quoted 9 lines
From: Dongxiao Xu <[email protected]>
After the implementation of per machine sysroot,
do_populate_sysrootanddo_packageneed to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine'sdo_populate_sysrootanddo_packagewill fail due to the missing of "image" directory.
Quoted 5 lines
Quoted 1 line
Thus reserve 'image' folder when doing
rm_work.Wouldn't adding
do_installto sstate be a better idea? OE was designed that it wouldn't depend on WORKDIR after building, so this patch is a step in the wrong direction.
Hi Koen,
Thanks for your suggestions, however this may not help to solve this issue.
The senario of the above issue is, build one machine firstly, doing rm_work, and then try to build the second machine of the same architecture. Some certain recipe's prebuilt result could not be shared between machines.
After the first machine finishes the build, most contents in ${WORKDIR} has been removed by "rm_work". Prebuilt result could not be shared for certain recipes. For those recipes, the second machine finds that most of the recipe stamp files are already there (do_fetch, do_unpack, do_patch, ...) except do_populate_sysroot and do_package, which contains ${MACHINE} tag, thus it will try to re-run do_populate_sysroot and do_package, which relies on the existence of "image" directory.
So even we add the do_install in sstate, the above issue could not be solved because it is triggered when certain recipe's sstate prebuilt result could *not* be shared.
Thanks,
Dongxiao
Quoted 8 lines
regards,
Koen
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Quoted 10 lines · On Wed, 2011-01-26 at 14:30 +0800, [email protected] wrote:
From: Dongxiao Xu <[email protected]>
After the implementation of per machine sysroot,
do_populate_sysrootanddo_packageneed to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine'sdo_populate_sysrootanddo_packagewill fail due to the missing of "image" directory.Thus reserve 'image' folder when doing
rm_work.
Whilst I understand the intent, this is just putting wallpaper over cracks, sooner or later the crack will reappear, possibly worse than before.
For example if you run "bitbake xxx -c install -f" for a target xxx which has had rm_work run against it, the result will be a nasty error in Poky.
OE's rm_work.bbclass is a little cleverer than ours and has this:
addhandler rmwork_stampfixing_eventhandler
python rmwork_stampfixing_eventhandler() {
if bb.event.getName(e) == "StampUpdate":
for (fn, task) in e.targets:
if task == 'rm_work_all':
continue
stamp = "%s.do_rm_work" % e.stampPrefix[fn]
if os.path.exists(stamp):
dir = "%s.*" % e.stampPrefix[fn]
bb.note("Removing stamps: " + dir)
os.system('rm -f '+ dir)
}which is intended to fix my specific example above. It won't fix the new corner cases we've introduced with machine specific sysroots though. So what do we do to fix this?
I'm actually tempted to tie together rm_work, sstate and the scenequeue. When rm_work wipes out the workdir, it also migrates the stamps into _setscene stamps as if the package had been installed by sstate. We'd then remove the unpack, fetch, patch and so on stamps just leaving the setscene varients. This would mean the above wouldn't be needed and most things should "just work".
So to be specific, we'd take stamps like:
xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack
and end up with:
xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow
Does that sound reasonable?
Then both codepaths use the same underlying code and everything should be fine apart from some hardcoded assumptions about stamp filenames which we'll probably just have to live with for now. The function will need to handle stamps including the taskhash too, e.g.:
xxx-1.4.7-r3.do_unpack.HASHHERE
Cheers,
Richard
Quoted 69 lines · Richard Purdie
Quoted 10 lines · On Wed, 2011-01-26 at 14:30 +0800, [email protected] wrote:
From: Dongxiao Xu <[email protected]>
After the implementation of per machine sysroot,
do_populate_sysrootanddo_packageneed to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine'sdo_populate_sysrootanddo_packagewill fail due to the missing of "image" directory.Thus reserve 'image' folder when doing
rm_work.Whilst I understand the intent, this is just putting wallpaper over cracks, sooner or later the crack will reappear, possibly worse than before.
For example if you run "bitbake xxx -c install -f" for a target xxx which has had
rm_workrun against it, the result will be a nasty error in Poky.OE's rm_work.bbclass is a little cleverer than ours and has this:
addhandler
rmwork_stampfixing_eventhandlerpythonrmwork_stampfixing_eventhandler() { if bb.event.getName(e) == "StampUpdate": for (fn, task) in e.targets: if task == 'rm_work_all': continue stamp = "%s.do_rm_work" % e.stampPrefix[fn] if os.path.exists(stamp): dir = "%s.*" % e.stampPrefix[fn] bb.note("Removing stamps: " + dir) os.system('rm -f '+ dir) }which is intended to fix my specific example above. It won't fix the new corner cases we've introduced with machine specific sysroots though. So what do we do to fix this?
I'm actually tempted to tie together
rm_work, sstate and the scenequeue.
Whenrm_workwipes out the workdir, it also migrates the stamps into_setscenestamps as if the package had been installed by sstate. We'd then remove the unpack, fetch, patch and so on stamps just leaving the setscene varients. This would mean the above wouldn't be needed and most things should "just work".So to be specific, we'd take stamps like:
xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpackand end up with:
xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow
Quoted 8 lines
Does that sound reasonable?
Then both codepaths use the same underlying code and everything should be fine apart from some hardcoded assumptions about stamp filenames which we'll probably just have to live with for now. The function will need to handle stamps including the taskhash too, e.g.:
xxx-1.4.7-r3.do_unpack.HASHHERE
Hi Richard,
I think this way is workable to track down the several rm_work issues.
Could you help to clarify what are "hardcoded assumptions about stamp filenames"?
Also you mentioned we need to handle the HASH stamp, do you mean we need to reserve those HASH stamps and they should not be removed while rm_work?
I just wrote a patch that implement the above senario and now is doing test for it.
Please help to review it to see if I understand your point correctly.
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/distro
There are two patches in the branch, the first one is a fix patch that add "${MACHINE}" tag into stamps of do_populate_sysroot_setscene and do_package_setscene, and the second one is to handle rm_work issue.
Thanks,
Dongxiao
Quoted 4 lines
Cheers,
Richard