Skip to contentSkip to conversation

rm_work.bbclass: reserve 'image' folder when remove WORKDIR

PATCH Koen Kooi 3 participants 4 messages Jan 26, 2011 – Jan 27, 2011 2 messages not in the archive

lore
Share & tools
lore

Koen Kooi

author
#2 lore raw
In reply to a message not in the archive

Op 26 jan 2011, om 07:30 heeft [email protected] het volgende geschreven:

Quoted 10 lines

From: Dongxiao Xu <[email protected]>

After the implementation of per machine sysroot, do_populate_sysroot and do_package need to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine's do_populate_sysroot and do_package will fail due to the missing of "image" directory.

Thus reserve 'image' folder when doing rm_work.

Wouldn't adding do_install to sstate be a better idea? OE was designed that it wouldn't depend on WORKDIR after building, so this patch is a step in the wrong direction.

regards,

Koen

#3 lore raw
Quoted 12 lines · Koen Kooi

Op 26 jan 2011, om 07:30 heeft [email protected] het volgende geschreven:

Quoted 9 lines

From: Dongxiao Xu <[email protected]>

After the implementation of per machine sysroot, do_populate_sysroot and do_package need to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine's do_populate_sysroot and do_package will fail due to the missing of "image" directory.

Quoted 5 lines
Quoted 1 line

Thus reserve 'image' folder when doing rm_work.

Wouldn't adding do_install to sstate be a better idea? OE was designed that it wouldn't depend on WORKDIR after building, so this patch is a step in the wrong direction.

Hi Koen,

Thanks for your suggestions, however this may not help to solve this issue.

The senario of the above issue is, build one machine firstly, doing rm_work, and then try to build the second machine of the same architecture. Some certain recipe's prebuilt result could not be shared between machines.

After the first machine finishes the build, most contents in ${WORKDIR} has been removed by "rm_work". Prebuilt result could not be shared for certain recipes. For those recipes, the second machine finds that most of the recipe stamp files are already there (do_fetch, do_unpack, do_patch, ...) except do_populate_sysroot and do_package, which contains ${MACHINE} tag, thus it will try to re-run do_populate_sysroot and do_package, which relies on the existence of "image" directory.

So even we add the do_install in sstate, the above issue could not be solved because it is triggered when certain recipe's sstate prebuilt result could *not* be shared.

Thanks,
Dongxiao

Quoted 8 lines

regards,

Koen
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#4 lore raw
In reply to a message not in the archive
Quoted 10 lines · On Wed, 2011-01-26 at 14:30 +0800, [email protected] wrote:

From: Dongxiao Xu <[email protected]>

After the implementation of per machine sysroot, do_populate_sysroot and do_package need to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine's do_populate_sysroot and do_package will fail due to the missing of "image" directory.

Thus reserve 'image' folder when doing rm_work.

Whilst I understand the intent, this is just putting wallpaper over cracks, sooner or later the crack will reappear, possibly worse than before.

For example if you run "bitbake xxx -c install -f" for a target xxx which has had rm_work run against it, the result will be a nasty error in Poky.

OE's rm_work.bbclass is a little cleverer than ours and has this:

addhandler rmwork_stampfixing_eventhandler

python rmwork_stampfixing_eventhandler() {
    if bb.event.getName(e) == "StampUpdate":
        for (fn, task) in e.targets:
            if task == 'rm_work_all':
                continue
            stamp = "%s.do_rm_work" % e.stampPrefix[fn]
            if os.path.exists(stamp):
                dir = "%s.*" % e.stampPrefix[fn]
                bb.note("Removing stamps: " + dir)
                os.system('rm -f '+ dir)
}

which is intended to fix my specific example above. It won't fix the new corner cases we've introduced with machine specific sysroots though. So what do we do to fix this?

I'm actually tempted to tie together rm_work, sstate and the scenequeue. When rm_work wipes out the workdir, it also migrates the stamps into _setscene stamps as if the package had been installed by sstate. We'd then remove the unpack, fetch, patch and so on stamps just leaving the setscene varients. This would mean the above wouldn't be needed and most things should "just work".

So to be specific, we'd take stamps like:

xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack

and end up with:

xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow

Does that sound reasonable?

Then both codepaths use the same underlying code and everything should be fine apart from some hardcoded assumptions about stamp filenames which we'll probably just have to live with for now. The function will need to handle stamps including the taskhash too, e.g.:

xxx-1.4.7-r3.do_unpack.HASHHERE

Cheers,

Richard

#5 lore raw
Quoted 69 lines · Richard Purdie
Quoted 10 lines · On Wed, 2011-01-26 at 14:30 +0800, [email protected] wrote:

From: Dongxiao Xu <[email protected]>

After the implementation of per machine sysroot, do_populate_sysroot and do_package need to be re-run if we build two machines of one architecture (in case that sstate prebuilt result could not be reused for certain recipes). But if removing all the WORKDIR contents after the first macine build, the second machine's do_populate_sysroot and do_package will fail due to the missing of "image" directory.

Thus reserve 'image' folder when doing rm_work.

Whilst I understand the intent, this is just putting wallpaper over cracks, sooner or later the crack will reappear, possibly worse than before.

For example if you run "bitbake xxx -c install -f" for a target xxx which has had rm_work run against it, the result will be a nasty error in Poky.

OE's rm_work.bbclass is a little cleverer than ours and has this:

addhandler rmwork_stampfixing_eventhandler python

    rmwork_stampfixing_eventhandler() { if bb.event.getName(e) ==
        "StampUpdate": for (fn, task) in e.targets:
            if task == 'rm_work_all':
                continue
            stamp = "%s.do_rm_work" % e.stampPrefix[fn]
            if os.path.exists(stamp):
                dir = "%s.*" % e.stampPrefix[fn]
                bb.note("Removing stamps: " + dir)
                os.system('rm -f '+ dir) }

which is intended to fix my specific example above. It won't fix the new corner cases we've introduced with machine specific sysroots though. So what do we do to fix this?

I'm actually tempted to tie together rm_work, sstate and the scenequeue.
When rm_work wipes out the workdir, it also migrates the stamps into _setscene stamps as if the package had been installed by sstate. We'd then remove the unpack, fetch, patch and so on stamps just leaving the setscene varients. This would mean the above wouldn't be needed and most things should "just work".

So to be specific, we'd take stamps like:

xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack

and end up with:

xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow

Quoted 8 lines

Does that sound reasonable?

Then both codepaths use the same underlying code and everything should be fine apart from some hardcoded assumptions about stamp filenames which we'll probably just have to live with for now. The function will need to handle stamps including the taskhash too, e.g.:

xxx-1.4.7-r3.do_unpack.HASHHERE

Hi Richard,

I think this way is workable to track down the several rm_work issues.

Could you help to clarify what are "hardcoded assumptions about stamp filenames"?

Also you mentioned we need to handle the HASH stamp, do you mean we need to reserve those HASH stamps and they should not be removed while rm_work?

I just wrote a patch that implement the above senario and now is doing test for it.

Please help to review it to see if I understand your point correctly.

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/distro

There are two patches in the branch, the first one is a fix patch that add "${MACHINE}" tag into stamps of do_populate_sysroot_setscene and do_package_setscene, and the second one is to handle rm_work issue.

Thanks,
Dongxiao

Quoted 4 lines

Cheers,

Richard