Building one package needs header from another
I'm trying to build python-matplotlib, which is a plotting package for PyGTK, and I need to compile the GDK rendering backend so that I can put plots onto the screen. (This is for a Gumstix, and I'm using their Danny branch metadata.) The backend source file needs to include pygtk/pygtk.h, which is all over the place in the fragments of the target rootfs since I've built that for the target system, but isn't anywhere to be found in the cross-compilation environment, so the compilation fails.
The whole topic of cross-compilation makes me dizzy. How is this situation dealt with? The source tarball for matplotlib (1.1.0) just assumes that this header file already exists on the user's system. Is this a normal thing for which there's a routine solution in Yocto? Or do I have to hack it by copying one of those files from the target rootfs and including it in my own metadata, forcibly putting it somewhere that the compiler can find it?
Signature · 2 lines
Ciao, Paul D. DeRocco Paul mailto:[email protected]
Quoted 7 lines · Paul D. DeRocco
I'm trying to build python-matplotlib, which is a plotting package for PyGTK, and I need to compile the GDK rendering backend so that I can put plots onto the screen. (This is for a Gumstix, and I'm using their Danny branch metadata.) The backend source file needs to include
pygtk/pygtk.h, which is all over the place in the fragments of the target rootfs since I've built that for the target system, but isn't anywhere to be found in the cross-compilation environment, so the compilation fails.
If you've built pygtk then the target sysroot should have the headers in, and for me it does:
ross@melchett /data/poky-master/tmp/sysroots/genericx86
$ find . -name pygtk.h ./usr/include/pygtk-2.0/pygtk/pygtk.h
This is probably a problem with python-matplotlib, can you share the configure and build logs? It's probably looking in the wrong place.
Ross
Quoted 11 lines
From: Burton, Ross [mailto:[email protected]]
If you've built pygtk then the target sysroot should have the headers in, and for me it does:
ross@melchett /data/poky-master/tmp/sysroots/genericx86
$ find . -name pygtk.h ./usr/include/pygtk-2.0/pygtk/pygtk.hThis is probably a problem with python-matplotlib, can you share the configure and build logs? It's probably looking in the wrong place.
Yes, the include files are there. I'm only beginning to grasp a little about how builds work, and that sysroots/overo is the context in which the Gumstix cross tools run. But the appropriate directory, /home/pauld/yocto/build/tmp/sysroots/overo/usr/include/pygtk-2.0, is not on the compiler command line, and that causes the error.
I've been trying to figure out how the setup.py/setup.cfg (and distutils) stuff works. The setup.cfg file lists only one possible option for adding directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate. I tried just setting include_dirs to the proper directory in the [directories] section, but it had no effect.
The configure log shows nothing, but the compile log (which shows the compile error at the end), shows this DEBUG note right at the beginning:
/home/pauld/yocto/build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/python -matplotlib-1.1.0-r3/temp/run.do_compile.14201: line 82:/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python: No such file or directory
basedirlist is: ['/home/pauld/yocto/build/tmp/sysroots/overo/usr/lib']
The message refers to this script in run.do_compile (slightly reformatted):
do_compile() {
BUILD_SYS=i686-linux HOST_SYS=arm-poky-linux-gnueabi \
/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python \
setup.py build || true distutils_do_compile
}
This would suggest that setup.py isn't even being run. Yet when I accidentally put a syntax error into setup.cfg, it barfed. Is setup.cfg read before setup.py is run, by something else?
So there are two questions: why is there no python in that directory? (There is a python-native subdirectory containing python.) And once that's dealt with, how does one add a specific include subdirectory to the command line via setup.cfg?
Or maybe those aren't the questions. I understand about 0.1% of what's going on here.
I'm also curious if distutils is something that is used throughout the bitbake process, or is it something specific to building Python-related stuff? Can the setup.py/setup.cfg mechanism be used in any recipes?
Signature · 2 lines
Ciao, Paul D. DeRocco Paul mailto:[email protected]
Hi Paul,
Quoted 17 lines · On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
Quoted 10 lines
From: Burton, Ross [mailto:[email protected]]
If you've built pygtk then the target sysroot should have the headers in, and for me it does:ross@melchett /data/poky-master/tmp/sysroots/genericx86
$ find . -name pygtk.h ./usr/include/pygtk-2.0/pygtk/pygtk.hThis is probably a problem with python-matplotlib, can you share the configure and build logs? It's probably looking in the wrong place.
Yes, the include files are there. I'm only beginning to grasp a little about how builds work, and that sysroots/overo is the context in which the Gumstix cross tools run. But the appropriate directory,
/home/pauld/yocto/build/tmp/sysroots/overo/usr/include/pygtk-2.0, is not on the compiler command line, and that causes the error.
Quoted 5 lines
I've been trying to figure out how the setup.py/setup.cfg (and distutils) stuff works. The setup.cfg file lists only one possible option for adding directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate.
Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere with similar-behaving configure scripts.
Quoted 19 lines
The configure log shows nothing, but the compile log (which shows the compile error at the end), shows this DEBUG note right at the beginning:
/home/pauld/yocto/build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/python-matplotlib-1.1.0-r3/temp/run.do_compile.14201: line 82:/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python: No such file or directory
basedirlist is: ['/home/pauld/yocto/build/tmp/sysroots/overo/usr/lib']The message refers to this script in run.do_compile (slightly reformatted):
do_compile(){
BUILD_SYS=i686-linux HOST_SYS=arm-poky-linux-gnueabi \/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python \ setup.py build || true
distutils_do_compile
}
Quoted 6 lines
This would suggest that
setup.pyisn't even being run. Yet when I accidentally put a syntax error into setup.cfg, it barfed. Is setup.cfg read beforesetup.pyis run, by something else?So there are two questions: why is there no python in that directory? (There is a python-native subdirectory containing python.)
To be honest, I'm not sure if it is appropriate, but you could inherit pythonnative in the recipe, that might fix this.
Quoted 3 lines
I'm also curious if distutils is something that is used throughout the bitbake process, or is it something specific to building Python-related stuff? Can the setup.py/setup.cfg mechanism be used in any recipes?
distutils is python-specific, so no.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 10 lines
Quoted 9 lines · On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
I've been trying to figure out how the setup.py/setup.cfg (and distutils)
stuff works. The setup.cfg file lists only one possible option for adding
directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate.
Quoted 4 lines
From: Paul Eggleton
Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere with similar-behaving configure scripts.
I think I've figured out what's going on. The whole distutils.bbclass interface to the Python distutils package involves the writing of a setup.py script to perform all the packaging and unpackaging processes. The setup.py script for matplotlib has to deal with the fact that matplotlib depends upon a varying set of libraries depending upon how you configure it. In this case, because I want to build the GTK rendering backend, it depends upon the pygtk library. It cleverly uses the pkg-config command to interrogate each dependent library to find out what compiler options are needed to compile modules that refer to the library, but unfortunately, pygtk-2.0 is not a "library" that can be searched by pkg-config, so the do_compile script coughs up the following nonfatal error message:
pkg-config: looking for pygtk-2.0 gtk+-2.0
* Package pygtk-2.0 was not found in the pkg-config
* search path. Perhaps you should add the directory
* containing `pygtk-2.0.pc' to the PKG_CONFIG_PATH
* environment variable No package 'pygtk-2.0' found
I think I'm just going to have to patch the setup.py file (actually a subsidiary setupext.py file) to hack the additional pygtk-2.0 include subdirectory into it. However, setupext.py and setup.py both already have small patches in the original matplotlib recipe. If I add another patch in my bbappend, will it always be applied after the one in the original recipe? Or should I just replace the patch file with one containing its fixes and my fix?
Signature · 2 lines
Ciao, Paul D. DeRocco Paul mailto:[email protected]
Hi Paul,
Apologies for the delayed reply.
Quoted 34 lines · On Friday 23 August 2013 12:52:54 Paul D. DeRocco wrote:
Quoted 10 lines
Quoted 9 lines · On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
I've been trying to figure out how the setup.py/setup.cfg (and distutils)
stuff works. The setup.cfg file lists only one possible option for adding
directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate.Quoted 4 lines
From: Paul Eggleton
Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere with similar-behaving configure scripts.
I think I've figured out what's going on. The whole distutils.bbclass interface to the Python distutils package involves the writing of a
setup.pyscript to perform all the packaging and unpackaging processes. Thesetup.pyscript for matplotlib has to deal with the fact that matplotlib depends upon a varying set of libraries depending upon how you configure it. In this case, because I want to build the GTK rendering backend, it depends upon the pygtk library. It cleverly uses the pkg-config command to interrogate each dependent library to find out what compiler options are needed to compile modules that refer to the library, but unfortunately, pygtk-2.0 is not a "library" that can be searched by pkg-config, so thedo_compilescript coughs up the following nonfatal error message:pkg-config: looking for pygtk-2.0 gtk+-2.0
* Package pygtk-2.0 was not found in the pkg-config
* search path. Perhaps you should add the directory
* containing `pygtk-2.0.pc' to thePKG_CONFIG_PATH
* environment variable No package 'pygtk-2.0' found
Quoted 3 lines
I think I'm just going to have to patch the
setup.pyfile (actually a subsidiarysetupext.pyfile) to hack the additional pygtk-2.0 include subdirectory into it.
I'm not very familiar with this particular problem, but if that's the only way forward I can't see a huge problem with that. FWIW I have had to do similar hacks in setup.py in python recipes I've written in the past.
Quoted 5 lines
However,
setupext.pyandsetup.pyboth already have
small patches in the original matplotlib recipe. If I add another patch in my bbappend, will it always be applied after the one in the original recipe? Or should I just replace the patch file with one containing its fixes and my fix?
Patches added in a bbappend will always be applied after the ones in the recipe itself, assuming you use SRC_URI += or SRC_URI_append = in your bbappend that is (since it's the order in which patches are listed in SRC_URI that determines the order in which they will be applied).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre