non-upstreamed patches
So I've switched back to looking at Yocto again for a project, and the Edison release works fine. Thanks!
I work on GNOME (almost all of it), and I see quite a number of patches in the Yocto tree that I think have been inappropriately categorized as
Upstream-Status: Inappropriate.
For example, recipies-core/glib-2.0/glib-2.0/remove.test.for.qsort_r.patch
We definitely want upstream GLib to work for cross builds. It was my understanding that the better way to handle autoconf checks that require running code was to pass a precomputed cache file:
http://www.gnu.org/s/hello/manual/autoconf/Site-Defaults.html
I think this makes sense to do for fundamental libraries like GLib; hacking up the configure.ac isn't a good long term plan.
Another thing that seems to be proliferating is gtk-doc workarounds. If it isn't working for you guys to --disable-gtk-doc, just tell me why and I'll fix it.
(The -Werror bit there is a total fuckup on our part, I will eventually convince Owen to remove it...)
librsvg/librsvg-2.31.1/doc_Makefile.patch is also TOTALLY appropriate upstream. Why would you think it isn't?
gtk+/gtk+-2.24.6/run-iconcache.patch is another cross build issue that I'm pretty sure we can solve better upstream. gtk+ for example already has an automake conditional:AM_CONDITIONAL(CROSS_COMPILING, test $cross_compiling = yes)
We can just not run this code if that's true.
(And I am totally fine with just copying that conditional into every single GNOME module).
hardcoded_libtool.patch is another one I think we can get upstream in a better way; shouldn't it just be using $LIBTOOL ?
Finally, on the Upstream-Status: pending patches, can you guys please add links to bugzilla?
Thanks!
Op 13 nov. 2011, om 01:57 heeft Colin Walters het volgende geschreven:
Quoted 4 lines
So I've switched back to looking at Yocto again for a project, and the Edison release works fine. Thanks!
I work on GNOME (almost all of it)
Have a look at the GNOME layer as well http://cgit.openembedded.org/meta-openembedded/tree/ :)
regards,
Koen
Quoted 8 lines · Koen Kooi
Op 13 nov. 2011, om 01:57 heeft Colin Walters het volgende geschreven:
Quoted 4 lines
So I've switched back to looking at Yocto again for a project, and the Edison release works fine. Thanks!
I work on GNOME (almost all of it)
Have a look at the GNOME layer as well http://cgit.openembedded.org/meta-openembedded/tree/ :)
What's the high level relationship of this tree versus the Yocto one? The git log looks very similar, are Yocto commits cherry-picked into it?
Op 13 nov. 2011, om 18:16 heeft Colin Walters het volgende geschreven:
Quoted 11 lines · Koen Kooi
Quoted 8 lines
Op 13 nov. 2011, om 01:57 heeft Colin Walters het volgende geschreven:
Quoted 4 lines
So I've switched back to looking at Yocto again for a project, and the Edison release works fine. Thanks!
I work on GNOME (almost all of it)
Have a look at the GNOME layer as well http://cgit.openembedded.org/meta-openembedded/tree/ :)
What's the high level relationship of this tree versus the Yocto one?
There's no such thing as a yocto tree, only a poky tree, which consists of openembedded-core, bitbake and some other stuff
Quoted 1 line
The git log looks very similar, are Yocto commits cherry-picked into it?
No, oe-core generally picks up fixes from here.
Quoted 2 lines · Colin Walters
So I've switched back to looking at Yocto again for a project, and the Edison release works fine. Thanks!
Pleased to hear it! :)
Quoted 3 lines
I work on GNOME (almost all of it), and I see quite a number of patches in the Yocto tree that I think have been inappropriately categorized as
Upstream-Status: Inappropriate.
The Upstream-Status fields are relatively new and I think some of the categorizations are perhaps not quite right or need to be interpreted correctly.
http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations
Quoted 4 lines
For example,
recipies-core/glib-2.0/glib-2.0/remove.test.for.qsort_r.patchWe definitely want upstream GLib to work for cross builds.
Taking this one as an example, someone has run into problems and fixed it in a way that is certainly not appropriate for upstream as the patch stands.
Quoted 7 lines
It was my
understanding that the better way to handle autoconf checks that require running code was to pass a precomputed cache file:
http://www.gnu.org/s/hello/manual/autoconf/Site-Defaults.htmlI think this makes sense to do for fundamental libraries like GLib; hacking up the configure.ac isn't a good long term plan.
Totally agreed. We do have site cache files an a much better way of handling that issue would be to add the value to the site cache. In an ideal world I've have caught that patch and asked it to be a site entry, it looks like its slipped through. I'll certainly take a site file entry and change to drop the patch.
Quoted 9 lines
Another thing that seems to be proliferating is gtk-doc workarounds. If it isn't working for you guys to --disable-gtk-doc, just tell me why and I'll fix it.
(The -Werror bit there is a total fuckup on our part, I will eventually convince Owen to remove it...)
librsvg/librsvg-2.31.1/doc_Makefile.patchis also TOTALLY appropriate upstream. Why would you think it isn't?
This is a mess of our own making :/. You'd probably get a bit of a shock if you look at our gtk-doc recipe. Since we reautoconf files, we need the gtk-doc .m4 file. We just provide a static copy of it. To handle the makefile, we just touch that.
I have a suspicion that EXTRA_DIST exists if you use the gtk-doc makefile and hence appending to it is a good idea and hence that patch truly is inappropriate for upstream.
In many cases we've learnt from our experiences and instead of touching the makefile, we echo EXTRA_DIST = "" to it instead (or similar).
Ultimately I'd like to see a fixed gtk-doc recipe. It was so long ago I last tried that I can't remember what issues we found.
Quoted 6 lines
gtk+/gtk+-2.24.6/run-iconcache.patchis another cross build issue that I'm pretty sure we can solve better upstream. gtk+ for example already has an automake conditional:AM_CONDITIONAL(CROSS_COMPILING, test $cross_compiling = yes)We can just not run this code if that's true.
That would be a nice change to get upstream. The current patch is inappropriate but I'm sure we could create one that could be accepted.
Quoted 5 lines
(And I am totally fine with just copying that conditional into every single GNOME module).
hardcoded_libtool.patch is another one I think we can get upstream in a better way; shouldn't it just be using $LIBTOOL ?
Good question, I think I might tried that and had some kind of problem but if that did would it would be great to see it upstream too.
Quoted 2 lines
Finally, on the Upstream-Status: pending patches, can you guys please add links to bugzilla?
If its been filed there should be links. "Pending" can also mean "needs to be submitted upstream". As I said, this is a relatively new thing, we used to have no information in the patches at all so it is getting better. We're making progress although never as fast as we would in an ideal world! :)
Any help would be appreciated, the reminders above are helpful as there does appear to be some low hanging fruit there. Also the pointers on which ones might make it into gnome upstream are useful!
Cheers,
Richard
Quoted 5 lines · Richard Purdie
Totally agreed. We do have site cache files an a much better way of handling that issue would be to add the value to the site cache. In an ideal world I've have caught that patch and asked it to be a site entry, it looks like its slipped through. I'll certainly take a site file entry and change to drop the patch.
Ok, as I move forward I'll look at draining the patches here. It'd be nice if we could try harder to avoid adding new ones though - that was my main goal with the mail.
Quoted 8 lines
Quoted 4 lines
Another thing that seems to be proliferating is gtk-doc workarounds. If it isn't working for you guys to --disable-gtk-doc, just tell me why and I'll fix it.
This is a mess of our own making :/. You'd probably get a bit of a shock if you look at our gtk-doc recipe. Since we reautoconf files, we need the gtk-doc .m4 file. We just provide a static copy of it. To handle the makefile, we just touch that.
Ah, I see. Have you seen people.gnome.org/~walters/docs/build-api.txt ?
One of the reasons I wrote it is because in GNOME we need ./autogen.sh scripts because there's no easy way to teach the autotools about external dependencies.
What would you think about a patch to autotools.bbclass to just check for autogen.sh and run it (instead of the big mess of autoreconf/etc. hacks) if it exists?
Quoted 11 lines · Colin Walters
Quoted 5 lines · Richard Purdie
Totally agreed. We do have site cache files an a much better way of handling that issue would be to add the value to the site cache. In an ideal world I've have caught that patch and asked it to be a site entry, it looks like its slipped through. I'll certainly take a site file entry and change to drop the patch.
Ok, as I move forward I'll look at draining the patches here. It'd be nice if we could try harder to avoid adding new ones though - that was my main goal with the mail.
Believe me, we're trying *very* hard to get the number of patches down!
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
Another thing that seems to be proliferating is gtk-doc workarounds. If it isn't working for you guys to --disable-gtk-doc, just tell me why and I'll fix it.
This is a mess of our own making :/. You'd probably get a bit of a shock if you look at our gtk-doc recipe. Since we reautoconf files, we need the gtk-doc .m4 file. We just provide a static copy of it. To handle the makefile, we just touch that.
Ah, I see. Have you seen
people.gnome.org/~walters/docs/build-api.txt?
Quoted 7 lines
One of the reasons I wrote it is because in GNOME we need ./autogen.sh scripts because there's no easy way to teach the autotools about external dependencies.
What would you think about a patch to autotools.bbclass to just check for
autogen.shand run it (instead of the big mess of autoreconf/etc. hacks) if it exists?
The number of people who write cross safe autogen.sh scripts which do everything we need is unfortunately low. I do appreciate the gnome community may do better than others and things are improving over time. I'd not yet at a point where I'd trust even 10% of the autogen.sh scripts out there though.
I'm prepared to stick my neck out and state that the reautoconf we do doesn't cause that much of a problem at this point and that class works pretty well. The process for reautoconf'ing a package is pretty standard when you get down to it. We did used to have a lot more hacky recipe specific do_configure functions and we've managed to just have the .bbclass code in most cases now too which is an achievement in itself.
Cheers,
Richard
Quoted 2 lines · Richard Purdie
The number of people who write cross safe
autogen.shscripts which do everything we need is unfortunately low.
Can you elaborate on this? What are some common cross-unsafe problems?
So concretely here with gtk-doc, since we don't actually want to build the documentation (because gtk-doc is native only for the exact same reasons gobject-introspection is, and we've talked about that one).
The GNOME autogen.sh scripts will run "gtkdocize", which pretty much just copies in gtk-doc.make and gtk-doc.m4. We could reimplement that with dummy copies, or build a lobotomized gtk-doc (this will require perl-native but that's about it I think).
I'm experimenting with this now.
Quoted 4 lines
I do appreciate the gnome
community may do better than others and things are improving over time. I'd not yet at a point where I'd trust even 10% of theautogen.shscripts out there though.
The other thing we could do I guess is make it opt-in, like OE_RUN_AUTOGEN=1 or something in individual recipes using autotools.bbclass?
Quoted 6 lines · Colin Walters
Quoted 2 lines · Richard Purdie
The number of people who write cross safe
autogen.shscripts which do everything we need is unfortunately low.Can you elaborate on this? What are some common cross-unsafe problems?
Well, I went and looked at the gnome-common autogen stuff. Its not a cross specific problem in this case but it does seem to reinvent/reimplement pieces of what autoconf/pkgconfig is supposed to do. Compare/check autoconf version? automake version? pkg-config dependencies present? Why? The whole point of autoconf was to avoid having these scripts :(
Examples of cross unsafe issues we've seen are libtool macros not being updated and old versions used for the reconf. Forcibly running configure from the script with options we don't want (or not including options we do need). We do have some autotools* flags we pass in ourselves too since the autotools we're using are slightly tweaked to avoid certain cross bugs. There is also confusion about whether to force copies of file or not (we always do).
Using those scripts just adds so much unpredictability to the system :(.
Quoted 8 lines
So concretely here with gtk-doc, since we don't actually want to build the documentation (because gtk-doc is native only for the exact same reasons gobject-introspection is, and we've talked about that one).
The GNOME
autogen.shscripts will run "gtkdocize", which pretty much just copies in gtk-doc.make and gtk-doc.m4. We could reimplement that with dummy copies, or build a lobotomized gtk-doc (this will require perl-native but that's about it I think).
Our current hack is really the former approach - add dummy copies in. Its just a little incomplete and we should enhance it deal with that so we can run gtkdocize.
Quoted 10 lines
I'm experimenting with this now.
Quoted 4 lines
I do appreciate the gnome
community may do better than others and things are improving over time. I'd not yet at a point where I'd trust even 10% of theautogen.shscripts out there though.The other thing we could do I guess is make it opt-in, like OE_RUN_AUTOGEN=1 or something in individual recipes using autotools.bbclass?
We could do that, yes. What is the advantage over our current class code (which I'll be the first to admit has faults but does seem to work)?
Cheers,
Richard
Quoted 6 lines · Richard Purdie
Well, I went and looked at the gnome-common autogen stuff. Its not a cross specific problem in this case but it does seem to reinvent/reimplement pieces of what autoconf/pkgconfig is supposed to do. Compare/check autoconf version? automake version? pkg-config dependencies present? Why? The whole point of autoconf was to avoid having these scripts :(
They serve a few purposes; one is simply helping people who check out a given module from git know they have compatible versions of the autotools. This used to be a major problem, nowadays it's been a long time since any kind serious of autotools incompatibilities.
The still-relevant issue though is that autoreconf isn't extensible - in GNOME for better or worse we have two things that follow the auto* style (gtk-doc and intltool mainly) where they copy data into your system if building from git, and get shipped in tarballs so you don't need them at build time when building from tarballs.
But we have no way to tell autoreconf to run these tools currently. Maybe the right thing is to extend autoreconf but even if we did that, we'd have to wait some amount of time before assuming everyone had the modified autoreconf.
Quoted 3 lines
Our current hack is really the former approach - add dummy copies in. Its just a little incomplete and we should enhance it deal with that so we can run gtkdocize.
Hmmm...so we'd be patching autotools.bbclass to grep for e.g. GTK_DOC_CHECK in configure.ac and if detected, copy in a dummy gtk-doc.make and gtk-doc.m4?
(And the same for intltool?)
I wouldn't object too much to that I guess, but I'd still prefer the solution where we opt-in to running autogen.sh for GNOME modules which use gtk-doc and/or intltool.
Quoted 9 lines · Colin Walters
Quoted 6 lines · Richard Purdie
Well, I went and looked at the gnome-common autogen stuff. Its not a cross specific problem in this case but it does seem to reinvent/reimplement pieces of what autoconf/pkgconfig is supposed to do. Compare/check autoconf version? automake version? pkg-config dependencies present? Why? The whole point of autoconf was to avoid having these scripts :(
Quoted 4 lines
They serve a few purposes; one is simply helping people who check out a given module from git know they have compatible versions of the autotools. This used to be a major problem, nowadays it's been a long time since any kind serious of autotools incompatibilities.
I certainly remember those days and can appreciate why this kind of thing exists (we have the same problem with the OE codebase in many ways).
Quoted 19 lines
The still-relevant issue though is that autoreconf isn't extensible - in GNOME for better or worse we have two things that follow the auto* style (gtk-doc and intltool mainly) where they copy data into your system if building from git, and get shipped in tarballs so you don't need them at build time when building from tarballs.
But we have no way to tell autoreconf to run these tools currently. Maybe the right thing is to extend autoreconf but even if we did that, we'd have to wait some amount of time before assuming everyone had the modified autoreconf.
Quoted 3 lines
Our current hack is really the former approach - add dummy copies in. Its just a little incomplete and we should enhance it deal with that so we can run gtkdocize.
Hmmm...so we'd be patching autotools.bbclass to grep for e.g.
GTK_DOC_CHECKin configure.ac and if detected, copy in a dummy gtk-doc.make and gtk-doc.m4?
Quoted 5 lines
(And the same for intltool?)
I wouldn't object too much to that I guess, but I'd still prefer the solution where we opt-in to running
autogen.shfor GNOME modules which use gtk-doc and/or intltool.
FWIW, we already do this for intltool:
if grep "^[[:space:]]*[AI][CT]_PROG_INTLTOOL" $CONFIGURE_AC >/dev/null; then bbnote Executing intltoolize --copy --force --automake intltoolize --copy --force --automake fi
so we might even use gtkdocize, but have modified files for it to place into the system. Strange as it might sound I'd actually like to be able to generate the docs if possible. It is completely the same as the introspection case where we need to be able to run target binaries or could a set of -native binaries get us through of most of the difficult bits?
The reason I mention docs is that we generate -doc packages. I have an idea of being able to generate the technical reference that would go alongside a given image. Obviously the gtkdoc pieces would be an important part of that. Creating the -doc packages is part of that, the remaining tricky part would then be indexing them in a sane way that was useful and accessible.
Cheers,
Richard
Quoted 5 lines
Quoted 5 lines · Richard Purdie
Our current hack is really the former approach - add dummy copies in. Its just a little incomplete and we should enhance it deal with that so we can run gtkdocize.
Revisiting this thread - so I've now created in GNOME git a tiny new module "gtk-doc-stub" which contains enough bits to build modules from git with --disable-gtk-doc.
http://git.gnome.org/browse/gtk-doc-stub
It should be enough to add this to Yocto, and then hacking up GNOME modules to remove GTK_DOC_CHECK from configure.ac etc. should be unnecessary. I'm using it now in my self-hosting OS build tool to avoid wading into Docbook dependency hell.