Skip to contentSkip to conversation

Building one package needs header from another

DISCUSSION Paul D. DeRocco 3 participants 6 messages Aug 20, 2013 – Sep 4, 2013

lore
Share & tools
lore
#0 lore raw

I'm trying to build python-matplotlib, which is a plotting package for PyGTK, and I need to compile the GDK rendering backend so that I can put plots onto the screen. (This is for a Gumstix, and I'm using their Danny branch metadata.) The backend source file needs to include pygtk/pygtk.h, which is all over the place in the fragments of the target rootfs since I've built that for the target system, but isn't anywhere to be found in the cross-compilation environment, so the compilation fails.

The whole topic of cross-compilation makes me dizzy. How is this situation dealt with? The source tarball for matplotlib (1.1.0) just assumes that this header file already exists on the user's system. Is this a normal thing for which there's a routine solution in Yocto? Or do I have to hack it by copying one of those files from the target rootfs and including it in my own metadata, forcibly putting it somewhere that the compiler can find it?

Signature · 2 lines
Ciao,               Paul D. DeRocco
Paul                mailto:[email protected] 
#1 lore raw
Quoted 7 lines · Paul D. DeRocco

I'm trying to build python-matplotlib, which is a plotting package for PyGTK, and I need to compile the GDK rendering backend so that I can put plots onto the screen. (This is for a Gumstix, and I'm using their Danny branch metadata.) The backend source file needs to include pygtk/pygtk.h, which is all over the place in the fragments of the target rootfs since I've built that for the target system, but isn't anywhere to be found in the cross-compilation environment, so the compilation fails.

If you've built pygtk then the target sysroot should have the headers in, and for me it does:

ross@melchett /data/poky-master/tmp/sysroots/genericx86

$ find . -name pygtk.h
./usr/include/pygtk-2.0/pygtk/pygtk.h

This is probably a problem with python-matplotlib, can you share the configure and build logs? It's probably looking in the wrong place.

Ross

#2 lore raw
Quoted 11 lines

From: Burton, Ross [mailto:[email protected]]

If you've built pygtk then the target sysroot should have the headers in, and for me it does:

ross@melchett /data/poky-master/tmp/sysroots/genericx86

$ find . -name pygtk.h
./usr/include/pygtk-2.0/pygtk/pygtk.h

This is probably a problem with python-matplotlib, can you share the configure and build logs? It's probably looking in the wrong place.

Yes, the include files are there. I'm only beginning to grasp a little about how builds work, and that sysroots/overo is the context in which the Gumstix cross tools run. But the appropriate directory, /home/pauld/yocto/build/tmp/sysroots/overo/usr/include/pygtk-2.0, is not on the compiler command line, and that causes the error.

I've been trying to figure out how the setup.py/setup.cfg (and distutils) stuff works. The setup.cfg file lists only one possible option for adding directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate. I tried just setting include_dirs to the proper directory in the [directories] section, but it had no effect.

The configure log shows nothing, but the compile log (which shows the compile error at the end), shows this DEBUG note right at the beginning:

/home/pauld/yocto/build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/python -matplotlib-1.1.0-r3/temp/run.do_compile.14201: line 82:
/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python: No such file or directory
basedirlist is: ['/home/pauld/yocto/build/tmp/sysroots/overo/usr/lib']

The message refers to this script in run.do_compile (slightly reformatted):

do_compile() {
BUILD_SYS=i686-linux HOST_SYS=arm-poky-linux-gnueabi \

  /home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python \
      setup.py build || true

distutils_do_compile
}

This would suggest that setup.py isn't even being run. Yet when I accidentally put a syntax error into setup.cfg, it barfed. Is setup.cfg read before setup.py is run, by something else?

So there are two questions: why is there no python in that directory? (There is a python-native subdirectory containing python.) And once that's dealt with, how does one add a specific include subdirectory to the command line via setup.cfg?

Or maybe those aren't the questions. I understand about 0.1% of what's going on here.

I'm also curious if distutils is something that is used throughout the bitbake process, or is it something specific to building Python-related stuff? Can the setup.py/setup.cfg mechanism be used in any recipes?

Signature · 2 lines
Ciao,               Paul D. DeRocco
Paul                mailto:[email protected] 
#3 lore raw

Hi Paul,

Quoted 17 lines · On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
Quoted 10 lines

From: Burton, Ross [mailto:[email protected]]
If you've built pygtk then the target sysroot should have the headers in, and for me it does:

ross@melchett /data/poky-master/tmp/sysroots/genericx86

$ find . -name pygtk.h
./usr/include/pygtk-2.0/pygtk/pygtk.h

This is probably a problem with python-matplotlib, can you share the configure and build logs? It's probably looking in the wrong place.

Yes, the include files are there. I'm only beginning to grasp a little about how builds work, and that sysroots/overo is the context in which the Gumstix cross tools run. But the appropriate directory, /home/pauld/yocto/build/tmp/sysroots/overo/usr/include/pygtk-2.0, is not on the compiler command line, and that causes the error.

Quoted 5 lines

I've been trying to figure out how the setup.py/setup.cfg (and distutils) stuff works. The setup.cfg file lists only one possible option for adding directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate.

Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere with similar-behaving configure scripts.

Quoted 19 lines

The configure log shows nothing, but the compile log (which shows the compile error at the end), shows this DEBUG note right at the beginning:

/home/pauld/yocto/build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/python -matplotlib-1.1.0-r3/temp/run.do_compile.14201: line 82:
/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python: No such file or directory
basedirlist is: ['/home/pauld/yocto/build/tmp/sysroots/overo/usr/lib']

The message refers to this script in run.do_compile (slightly reformatted):

do_compile() {
BUILD_SYS=i686-linux HOST_SYS=arm-poky-linux-gnueabi \

  /home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python \
      setup.py build || true

distutils_do_compile
}

Quoted 6 lines

This would suggest that setup.py isn't even being run. Yet when I accidentally put a syntax error into setup.cfg, it barfed. Is setup.cfg read before setup.py is run, by something else?

So there are two questions: why is there no python in that directory? (There is a python-native subdirectory containing python.)

To be honest, I'm not sure if it is appropriate, but you could inherit pythonnative in the recipe, that might fix this.

Quoted 3 lines

I'm also curious if distutils is something that is used throughout the bitbake process, or is it something specific to building Python-related stuff? Can the setup.py/setup.cfg mechanism be used in any recipes?

distutils is python-specific, so no.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#4 lore raw
Quoted 10 lines
Quoted 9 lines · On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:

I've been trying to figure out how the setup.py/setup.cfg (and distutils)
stuff works. The setup.cfg file lists only one possible option for adding
directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate.

Quoted 4 lines

From: Paul Eggleton

Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere with similar-behaving configure scripts.

I think I've figured out what's going on. The whole distutils.bbclass interface to the Python distutils package involves the writing of a setup.py script to perform all the packaging and unpackaging processes. The setup.py script for matplotlib has to deal with the fact that matplotlib depends upon a varying set of libraries depending upon how you configure it. In this case, because I want to build the GTK rendering backend, it depends upon the pygtk library. It cleverly uses the pkg-config command to interrogate each dependent library to find out what compiler options are needed to compile modules that refer to the library, but unfortunately, pygtk-2.0 is not a "library" that can be searched by pkg-config, so the do_compile script coughs up the following nonfatal error message:

pkg-config: looking for pygtk-2.0 gtk+-2.0
* Package pygtk-2.0 was not found in the pkg-config
* search path. Perhaps you should add the directory
* containing `pygtk-2.0.pc' to the PKG_CONFIG_PATH
* environment variable No package 'pygtk-2.0' found

I think I'm just going to have to patch the setup.py file (actually a subsidiary setupext.py file) to hack the additional pygtk-2.0 include subdirectory into it. However, setupext.py and setup.py both already have small patches in the original matplotlib recipe. If I add another patch in my bbappend, will it always be applied after the one in the original recipe? Or should I just replace the patch file with one containing its fixes and my fix?

Signature · 2 lines
Ciao,               Paul D. DeRocco
Paul                mailto:[email protected] 
#5 lore raw

Hi Paul,

Apologies for the delayed reply.

Quoted 34 lines · On Friday 23 August 2013 12:52:54 Paul D. DeRocco wrote:
Quoted 10 lines
Quoted 9 lines · On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:

I've been trying to figure out how the setup.py/setup.cfg (and distutils)
stuff works. The setup.cfg file lists only one possible option for adding
directories, which is basedirlist, but setting that to foo adds foo/include to the include directories and foo/lib to the library directories, so that's not appropriate.

Quoted 4 lines

From: Paul Eggleton

Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere with similar-behaving configure scripts.

I think I've figured out what's going on. The whole distutils.bbclass interface to the Python distutils package involves the writing of a setup.py script to perform all the packaging and unpackaging processes. The setup.py script for matplotlib has to deal with the fact that matplotlib depends upon a varying set of libraries depending upon how you configure it. In this case, because I want to build the GTK rendering backend, it depends upon the pygtk library. It cleverly uses the pkg-config command to interrogate each dependent library to find out what compiler options are needed to compile modules that refer to the library, but unfortunately, pygtk-2.0 is not a "library" that can be searched by pkg-config, so the do_compile script coughs up the following nonfatal error message:

pkg-config: looking for pygtk-2.0 gtk+-2.0
* Package pygtk-2.0 was not found in the pkg-config
* search path. Perhaps you should add the directory
* containing `pygtk-2.0.pc' to the PKG_CONFIG_PATH
* environment variable No package 'pygtk-2.0' found

Quoted 3 lines

I think I'm just going to have to patch the setup.py file (actually a subsidiary setupext.py file) to hack the additional pygtk-2.0 include subdirectory into it.

I'm not very familiar with this particular problem, but if that's the only way forward I can't see a huge problem with that. FWIW I have had to do similar hacks in setup.py in python recipes I've written in the past.

Quoted 5 lines

However, setupext.py and setup.py both already have
small patches in the original matplotlib recipe. If I add another patch in my bbappend, will it always be applied after the one in the original recipe? Or should I just replace the patch file with one containing its fixes and my fix?

Patches added in a bbappend will always be applied after the ones in the recipe itself, assuming you use SRC_URI += or SRC_URI_append = in your bbappend that is (since it's the order in which patches are listed in SRC_URI that determines the order in which they will be applied).

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre