Using external source trees with OE-Core
Someone recently asked me about using external source trees with OE-Core. I was aware of srctree.bbclass in OE-Classic and did start looking at it but it has various elements I wasn't so keen on. I therefore wondered if I could improve upon it. I did start from that code base but came up with:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
[patch inline below]
which I've called externalsrc.bbclass since it behaves a bit differently. In particular, all the usual task targets are available.
As an example usage with the above patch applied:
cd ~
tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
(creates libfm-0.1.17 in my homedir)
edit libfm_0.1.17.bb and add:
inherit externalsrc
S = "/home/richard/libfm-0.1.17"
bitbake libfm
and it does what you'd expect. You can also -c clean and it will wipe out WORKDIR but it won't touch ${S}. I picked libfm effectively at random. It won't remove the autoreconf changes from configure but I'm not sure I care much about that.
Admittedly, I did have to fix one Makefile.am which did '-I../' instead of "-I$(srcdir)/../" but that is a bug in libfm.
Obviously this won't work quite as well with some recipes like linux-yocto which manipulate ${S} a lot more but it should work in most cases. I also found it very hard to remove tasks from the anonymous python, we probably need to improve the API for this.
Opinions on including this class?
Cheers,
Richard
From ac8a54c18c33aaab6758c34a3d6093d3d3384cee Mon Sep 17 00:00:00 2001
From: Richard Purdie <[email protected]>
Date: Fri, 24 Feb 2012 12:29:36 +0000
Subject: externalsrc.bbclass: Add class for handling external source trees
This is loosly based upon srctree.bbclass from OE-Classic but with some changes appropriate to OE-Core.
---
meta/classes/externalsrc.bbclass renamed from dev/null +53 -0
@@ -0,0 +1,53 @@ +# Copyright (C) 2012 Linux Foundation +# Author: Richard Purdie +# Some code and influence taken from srctree.bbclass: +# Copyright (C) 2009 Chris Larson <[email protected]> +# Released under the MIT license (see COPYING.MIT for the terms) +# +# externalsrc.bbclass enables use of an existing source tree, usually external to +# the build system to build a piece of software rather than the usual fetch/unpack/patch +# process. +# +# To use, set S to point at the directory you want to use containing the sources +# e.g. S = "/path/to/my/source/tree" +# +# If the class is to work for both target and native versions (or with multilibs/ +# cross or other BBCLASSEXTEND variants), its expected that setting B to point to +# where to place the compiled binaries will work (split source and build directories). +# This is the default but B can be set to S if circumstaces dictate. +# + +SRC_URI = "" +SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch" +B = "${WORKDIR}/${BPN}-${PV}/" + +def remove_tasks(tasks, deltasks, d): + for task in tasks: + deps = d.getVarFlag(task, "deps") + for preptask in deltasks: + if preptask in deps: + deps.remove(preptask) + d.setVarFlag(task, "deps", deps) + # Poking around bitbake internal variables is evil but there appears to be no better way :( + tasklist = d.getVar('__BBTASKS') or [] + for task in deltasks: + d.delVarFlag(task, "task") + if task in tasklist: + tasklist.remove(task) + d.setVar('__BBTASKS', tasklist) + +python () { + tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys()) + covered = d.getVar("SRCTREECOVEREDTASKS", True).split() + + for task in tasks: + if task.endswith("_setscene"): + # sstate is never going to work for external source trees, disable it + covered.append(task) + else: + # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time + d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock") + + remove_tasks(tasks, covered, d) +} +
Quoted 14 lines · Richard Purdie
Someone recently asked me about using external source trees with OE-Core. I was aware of srctree.bbclass in OE-Classic and did start looking at it but it has various elements I wasn't so keen on. I therefore wondered if I could improve upon it. I did start from that code base but came up with:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
[patch inline below]which I've called externalsrc.bbclass since it behaves a bit differently. In particular, all the usual task targets are available.
As an example usage with the above patch applied:
Quoted 8 lines
cd ~
tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
(creates libfm-0.1.17 in my homedir)edit libfm_0.1.17.bb and add:
inherit externalsrc
S = "/home/richard/libfm-0.1.17"
Do I need to edit libfm_0.1.17.bb in-place, or can it be stored outside my layers to be built with bitbake -b, for example?
Does S need to contain an absolute path, or could it be set to a path relative to the location of the recipe?
It would be nice if it was possible to include bitbake recipes in the source trees of my projects, which "just work".
I think this was possible with srctree.bbclass, though I never had it working reliably. I think bitbake -cclean always raised errors.
Regards,
Andreas
Quoted 19 lines · Andreas Oberritter
Quoted 18 lines · Richard Purdie
Someone recently asked me about using external source trees with OE-Core. I was aware of srctree.bbclass in OE-Classic and did start looking at it but it has various elements I wasn't so keen on. I therefore wondered if I could improve upon it. I did start from that code base but came up with:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
[patch inline below]which I've called externalsrc.bbclass since it behaves a bit differently. In particular, all the usual task targets are available.
As an example usage with the above patch applied:
cd ~
tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
(creates libfm-0.1.17 in my homedir)
Quoted 7 lines
Quoted 4 lines
edit libfm_0.1.17.bb and add:
inherit externalsrc
S = "/home/richard/libfm-0.1.17"Do I need to edit libfm_0.1.17.bb in-place, or can it be stored outside my layers to be built with bitbake -b, for example?
You should be able to add a bbappend for this item, in a custom development layer and not modify the original .bb. It will still require the two line configuration change (and a custom layer configuration... but I recommend that anyway.)
Quoted 2 lines
Does S need to contain an absolute path, or could it be set to a path relative to the location of the recipe?
I don't know for this development.... but relative can get very messy, especially when all of the project, work dir, etc can be overridden to create a custom environment for someone. Enforcing an absolute path isn't a bad idea here.
Quoted 2 lines
It would be nice if it was possible to include bitbake recipes in the source trees of my projects, which "just work".
Might be possible in something like base.bbclass to add the inherit if a value is set.. then in local.conf you could do something like:
EXTERNALSRC_libfm = "/home/richard/libfm-0.1.17"
And if it's set, the class could do the reset? That would certainly be easier then a layer.
--Mark
Quoted 10 lines
I think this was possible with srctree.bbclass, though I never had it working reliably. I think bitbake -cclean always raised errors.
Regards,
Andreas_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 30 lines · Richard Purdie
Someone recently asked me about using external source trees with OE-Core. I was aware of srctree.bbclass in OE-Classic and did start looking at it but it has various elements I wasn't so keen on. I therefore wondered if I could improve upon it. I did start from that code base but came up with:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
[patch inline below]which I've called externalsrc.bbclass since it behaves a bit differently. In particular, all the usual task targets are available.
As an example usage with the above patch applied:
cd ~
tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
(creates libfm-0.1.17 in my homedir)edit libfm_0.1.17.bb and add:
inherit externalsrc
S = "/home/richard/libfm-0.1.17"bitbake libfm
and it does what you'd expect. You can also -c clean and it will wipe out WORKDIR but it won't touch ${S}. I picked libfm effectively at random. It won't remove the autoreconf changes from configure but I'm not sure I care much about that.
Quoted 7 lines
Admittedly, I did have to fix one Makefile.am which did '-I../' instead of "-I$(srcdir)/../" but that is a bug in libfm.
Obviously this won't work quite as well with some recipes like linux-yocto which manipulate ${S} a lot more but it should work in most cases. I also found it very hard to remove tasks from the anonymous python, we probably need to improve the API for this.
I was just looking at this myself, and I agree that when using this class, you'd want to drop a few phases that linux-yocto does and just let things run from compile onwards. You aren't using much infrastructure that is provided in this scenario .. but that is the point :)
I like the option that this provides for a development workflow, with some safety and with associated risks. But overall, if used properly .. I see the value.
Cheers,
Bruce
Quoted 85 lines
Opinions on including this class?
Cheers,
Richard
From
ac8a54c18c33aaab6758c34a3d6093d3d3384ceeMon Sep 17 00:00:00 2001
From: Richard Purdie <[email protected]>
Date: Fri, 24 Feb 2012 12:29:36 +0000
Subject: externalsrc.bbclass: Add class for handling external source treesThis is loosly based upon srctree.bbclass from OE-Classic but with some changes appropriate to OE-Core.
Signed-off-by Richard Purdie <[email protected]>---
meta/classes/externalsrc.bbclass renamed from dev/null +53
-0@@ -0,0 +1,53 @@ +# Copyright (C) 2012 Linux Foundation +# Author: Richard Purdie +# Some code and influence taken from srctree.bbclass: +# Copyright (C) 2009 Chris Larson <[email protected]> +# Released under the MIT license (see COPYING.MIT for the terms) +# +# externalsrc.bbclass enables use of an existing source tree, usually external to +# the build system to build a piece of software rather than the usual fetch/unpack/patch +# process. +# +# To use, set S to point at the directory you want to use containing the sources +# e.g. S = "/path/to/my/source/tree" +# +# If the class is to work for both target and native versions (or with multilibs/ +# cross or other BBCLASSEXTEND variants), its expected that setting B to point to +# where to place the compiled binaries will work (split source and build directories). +# This is the default but B can be set to S if circumstaces dictate. +# + +SRC_URI = "" +SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch" +B = "${WORKDIR}/${BPN}-${PV}/" + +def remove_tasks(tasks, deltasks, d): + for task in tasks: + deps = d.getVarFlag(task, "deps") + for preptask in deltasks: + if preptask in deps: + deps.remove(preptask) + d.setVarFlag(task, "deps", deps) + # Poking around bitbake internal variables is evil but there appears to be no better way :( + tasklist = d.getVar('__BBTASKS') or [] + for task in deltasks: + d.delVarFlag(task, "task") + if task in tasklist: + tasklist.remove(task) + d.setVar('__BBTASKS', tasklist) + +python () { + tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys()) + covered = d.getVar("SRCTREECOVEREDTASKS", True).split() + + for task in tasks: + if task.endswith("_setscene"): + # sstate is never going to work for external source trees, disable it + covered.append(task) + else: + # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time + d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock") + + remove_tasks(tasks, covered, d) +} +_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Richard,
We do something very similar to this currently. I've pushed what we are using here:
https://github.com/mattsm/meta-fsl-local-sdk
It would be nice to consolidate all the required bits upstream... =)
-M
Quoted 122 lines · Richard Purdie
Someone recently asked me about using external source trees with OE-Core. I was aware of srctree.bbclass in OE-Classic and did start looking at it but it has various elements I wasn't so keen on. I therefore wondered if I could improve upon it. I did start from that code base but came up with:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/a0&id=ac8a54c18c33aaab6758c34a3d6093d3d3384cee
[patch inline below]which I've called externalsrc.bbclass since it behaves a bit differently. In particular, all the usual task targets are available.
As an example usage with the above patch applied:
cd ~
tar -xvzf $DL_DIR/libfm-0.1.17.tar.gz
(creates libfm-0.1.17 in my homedir)edit libfm_0.1.17.bb and add:
inherit externalsrc
S = "/home/richard/libfm-0.1.17"bitbake libfm
and it does what you'd expect. You can also -c clean and it will wipe out WORKDIR but it won't touch ${S}. I picked libfm effectively at random. It won't remove the autoreconf changes from configure but I'm not sure I care much about that.
Admittedly, I did have to fix one Makefile.am which did '-I../' instead of "-I$(srcdir)/../" but that is a bug in libfm.
Obviously this won't work quite as well with some recipes like linux-yocto which manipulate ${S} a lot more but it should work in most cases. I also found it very hard to remove tasks from the anonymous python, we probably need to improve the API for this.
Opinions on including this class?
Cheers,
Richard
From
ac8a54c18c33aaab6758c34a3d6093d3d3384ceeMon Sep 17 00:00:00 2001
From: Richard Purdie <[email protected]>
Date: Fri, 24 Feb 2012 12:29:36 +0000
Subject: externalsrc.bbclass: Add class for handling external source treesThis is loosly based upon srctree.bbclass from OE-Classic but with some changes appropriate to OE-Core.
Signed-off-by Richard Purdie <[email protected]>---
meta/classes/externalsrc.bbclass renamed from dev/null +53
-0@@ -0,0 +1,53 @@ +# Copyright (C) 2012 Linux Foundation +# Author: Richard Purdie +# Some code and influence taken from srctree.bbclass: +# Copyright (C) 2009 Chris Larson <[email protected]> +# Released under the MIT license (see COPYING.MIT for the terms) +# +# externalsrc.bbclass enables use of an existing source tree, usually external to +# the build system to build a piece of software rather than the usual fetch/unpack/patch +# process. +# +# To use, set S to point at the directory you want to use containing the sources +# e.g. S = "/path/to/my/source/tree" +# +# If the class is to work for both target and native versions (or with multilibs/ +# cross or other BBCLASSEXTEND variants), its expected that setting B to point to +# where to place the compiled binaries will work (split source and build directories). +# This is the default but B can be set to S if circumstaces dictate. +# + +SRC_URI = "" +SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch" +B = "${WORKDIR}/${BPN}-${PV}/" + +def remove_tasks(tasks, deltasks, d): + for task in tasks: + deps = d.getVarFlag(task, "deps") + for preptask in deltasks: + if preptask in deps: + deps.remove(preptask) + d.setVarFlag(task, "deps", deps) + # Poking around bitbake internal variables is evil but there appears to be no better way :( + tasklist = d.getVar('__BBTASKS') or [] + for task in deltasks: + d.delVarFlag(task, "task") + if task in tasklist: + tasklist.remove(task) + d.setVar('__BBTASKS', tasklist) + +python () { + tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys()) + covered = d.getVar("SRCTREECOVEREDTASKS", True).split() + + for task in tasks: + if task.endswith("_setscene"): + # sstate is never going to work for external source trees, disable it + covered.append(task) + else: + # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time + d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock") + + remove_tasks(tasks, covered, d) +} +_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 6 lines · On Fri, 2012-02-24 at 16:55 +0000, McClintock Matthew-B29882 wrote:
We do something very similar to this currently. I've pushed what we are using here:
https://github.com/mattsm/meta-fsl-local-sdk
It would be nice to consolidate all the required bits upstream... =)
I would warn you that there is code in there that is clearly inappropriate for OE-Core.
It would also explain that bug report you sent me which referenced "do_populate_sysroot_post" and I asked what local modifications you had and you said "none" ;-).
Does the patch in this thread take care of your needs regarding the srctree and clean classes?
For cleanobjs, I can see advantages in merging something like that into the core...
Cheers,
Richard
Quoted 10 lines · Richard Purdie
Quoted 6 lines · On Fri, 2012-02-24 at 16:55 +0000, McClintock Matthew-B29882 wrote:
We do something very similar to this currently. I've pushed what we are using here:
https://github.com/mattsm/meta-fsl-local-sdk
It would be nice to consolidate all the required bits upstream... =)
I would warn you that there is code in there that is clearly inappropriate for OE-Core.
No surprise at all...
Quoted 3 lines
It would also explain that bug report you sent me which referenced "
do_populate_sysroot_post" and I asked what local modifications you had and you said "none" ;-).
I don't use this... we use this for internal developers... so the error was not using this...
Quoted 2 lines
Does the patch in this thread take care of your needs regarding the srctree and clean classes?
Not sure, hoping Zhenhua can take a look.
Quoted 2 lines
For cleanobjs, I can see advantages in merging something like that into the core...
....
-M
Quoted 4 lines · Richard Purdie
You can also -c clean and it will wipe
out WORKDIR but it won't touch ${S}. I picked libfm effectively at random. It won't remove the autoreconf changes from configure but I'm not sure I care much about that.
This is one of the things that srctree handled a bit differently. I think we should think about pushing the clean improvement bits into a core class, to make it easy to select slightly different clean behavior. Either that, or add a hook for it. As it is today, it'd be slightly painful to make the recipe both wipe workdir and make clean your source tree.
Signature · 1 line
Christopher Larson
Quoted 3 lines · Richard Purdie
Someone recently asked me about using external source trees with OE-Core. [...]
Opinions on including this class?
I see value.
Our workflow, (based on an ancient branch of oe), uses something similar. When working on a component in context, (as distinct from standalone), the developer sets S to point to his own directory and then just avoids the tasks with awkward side effects like clean and mrproper.
We had problems initially with people committing bb files with S still pointed to the local values. There needs to be a safety catch for this somewhere as this mistake is too easy to make and affects everyone when it happens.
Against my better judgment, someone locally implemented a ~/oe.conf arrangement that developers now use to override S for the components in which they work. That seems to have largely addressed the problem of accidental commits of bb files with bad S values, but has opened a completely different set of problems, of course. I don't think this is the right solution to the accidental commit problem, but I think some sanity check or "best practice" for overriding S is required before including this class.
--rich