python-pip: new package, update python-setuptools, enable openssl for host python

Added python-pip - a package manager for python. easy way of adding python packages to buildroot. Updated python-setuptools to version 0.8 - distribute and setuptools have now merged. Enabled ssl in host python config so pip can download packages over https.

---
 package/Config.in                              |  2 +
 package/python-pip/Config.in                   | 20 +++++++++
 package/python-pip/python-pip.mk               | 58 ++++++++++++++++++++++++++
 package/python-setuptools/python-setuptools.mk | 10 ++---
 package/python/python.mk                       |  5 +--
 5 files changed, 87 insertions(+), 8 deletions(-)
 create mode 100644 package/python-pip/Config.in
 create mode 100644 package/python-pip/python-pip.mk

index 6d5ff01..d6678a3 100644
@@ -359,6 +359,8 @@ source "package/python-pyparsing/Config.in"
 source "package/python-serial/Config.in"
 source "package/python-setuptools/Config.in"
 source "package/python-thrift/Config.in"
+comment "custom python modules"
+source "package/python-pip/Config.in"
 endmenu
 endif
 source "package/python3/Config.in"

new file mode 100644
index 0000000..48f3b04
@@ -0,0 +1,20 @@
+config BR2_PACKAGE_PYTHON_PIP
+    bool "python-pip"
+    help
+      A tool for installing and managing Python packages.
+
+      http://www.pip-installer.org
+
+if BR2_PACKAGE_PYTHON_PIP
+
+config BR2_PACKAGE_PYTHON_PIP_MODULES_ADDITIONAL
+	string "Additional modules"
+	help
+	  List of space-separated python modules to install via pip.
+	  See 'pip help install' for available installation methods. 
+	  For repeatable builds, download and save tgz files or clone 
+	  git repos for the components you care about.
+
+	  Example: module-name module-name==1.3.4 /my/module/mymodule.tgz git://github.com/someuser/somemodule.git#v1.2
+
+endif

index cd5ee4d..416abfb 100644
@@ -9,9 +9,9 @@
 # switch back to it.
 # See http://pypi.python.org/packages/source/s/setuptools
 
-PYTHON_SETUPTOOLS_VERSION = 0.6.36
-PYTHON_SETUPTOOLS_SOURCE  = distribute-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
-PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/d/distribute
+PYTHON_SETUPTOOLS_VERSION = 0.8
+PYTHON_SETUPTOOLS_SOURCE  = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
+PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/s/setuptools
 PYTHON_SETUPTOOLS_DEPENDENCIES = python
 
 define HOST_PYTHON_SETUPTOOLS_BUILD_CMDS

@@ -20,7 +20,7 @@ endef
 
 define PYTHON_SETUPTOOLS_BUILD_CMDS
 	(cd $(@D); \
-	PYTHONPATH="/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages" \
+	PYTHONPATH="$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages" \
 	$(HOST_DIR)/usr/bin/python setup.py build)
 endef
 

@@ -32,7 +32,7 @@ endef
 
 define PYTHON_SETUPTOOLS_INSTALL_TARGET_CMDS
 	(cd $(@D); \
-	PYTHONPATH="/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages" \
+	PYTHONPATH="$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages" \
 	$(HOST_DIR)/usr/bin/python setup.py install --executable=/usr/bin/python \
 	--single-version-externally-managed --root=/ --prefix=$(TARGET_DIR)/usr)
 endef

index ecea638..45a0563 100644
@@ -31,8 +31,7 @@ HOST_PYTHON_CONF_OPT += 	\
 	--disable-gdbm		\
 	--disable-bsddb		\
 	--disable-test-modules	\
-	--disable-bz2		\
-	--disable-ssl
+	--disable-bz2		
 
 HOST_PYTHON_MAKE_ENV = \
 	PYTHON_MODULES_INCLUDE=$(HOST_DIR)/usr/include \

@@ -50,7 +49,7 @@ HOST_PYTHON_MAKE = $(MAKE1)
 
 PYTHON_DEPENDENCIES  = host-python libffi
 
-HOST_PYTHON_DEPENDENCIES = host-expat host-zlib
+HOST_PYTHON_DEPENDENCIES = host-expat host-zlib host-openssl
 
 PYTHON_INSTALL_STAGING = YES
 

1.8.1.2

---
 package/python-pip/python-pip.mk | 1 +
 1 file changed, 1 insertion(+)

index 7cf8ab4..9f78d83 100644
@@ -9,6 +9,7 @@ PYTHON_PIP_VERSION = 1.3.1
 PYTHON_PIP_SOURCE = pip-$(PYTHON_PIP_VERSION).tar.gz
 PYTHON_PIP_SITE = https://pypi.python.org/packages/source/p/pip
 PYTHON_PIP_DEPENDENCIES = python python-setuptools host-python-setuptools host-python-pip
+PYTHON_PIP_LICENSE = MIT
 
 # README.rst refers to the file "LICENSE" but it's not included
 

1.8.1.2

Dear Rohan Fletcher,

Thanks for your contribution! A few comments below, to help improve your patches and make them ready for merging in Buildroot.

In this commit description, you're mentioning three totally separate things. This is a very good indication that those three separate things should be in separate patches: adding ssl in host python, bumping setuptools, and finally adding python-pip.

However, your second patch that adds the license to python-pip should be part of the patch adding python-pip itself.

To re-organize your patches, I recommend you to have a look about "git rebase -i", it's really the git killer feature to make your patch series look nice.

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Hello Thomas,

Thanks for your contribution! A few comments below, to help improve

You are welcome. Ok, I will make changes to the patches based upon your comments and submit again. This is the first time I have submitted anything to an open source project.

Don't certain pip modules have native dependencies, that should be

Yes some python modules have native dependencies - pip does not automatically select and compile the dependencies in buildroot - they need to be preselected.

For instance, the python-lxml module requires the libxml2 and libxslt libraries. When these libraries have been preselected in buildroot, the python-lxml module will successfully compile and install to the target.

So far I have installed gevent, gevent-socketio, flask, jinja2, psutil, daemon, passlib, requests, werkzeug, pillow, and feedparser successfully on the target using this method.

I have conducted basic tests of the installed modules on the target python interpreter, with mostly success.

There are tiny issues I am coming across installing modules that require libraries with pip:
- Pillow (a fork of Python Image Library) won't compile with jpeg support
because it expects libjpeg headers to be located at /usr/include instead of $BASE_DIR/output/staging/usr/include.
- The python-magic package can't find libmagic because it relies on
ldconfig cache output to find the library.
- The requests package gives an error
SSL3_GET_SERVER_CERTIFICATE:certificate verify failed, even though I can connect fine to the same website using httplib. Still investigating this one.
- The lxml package must be able to find xslt-config on $PATH.

Some of these python modules that compile against libraries might need to be made into a buildroot package with patches to work 100%. However, pip will cover the rest.

Generally, we don't really like this kind of package that builds other

I can understand that. Its definitely a package addition that will bend the rules of the game.

My inspiration for this package is found under nodejs -> module selection in menuconfig. There is an option to provide a list of additional modules that will be automatically installed by npm, the nodejs package manager.

The reasoning for adding pip to my setup was:
1. Quick and easy - If I want to add another python module, I can just add a name to a list in menuconfig and it will be included.
2. Dependency resolution - By using pip, the (python) dependencies of each python module are automatically retrieved and installed during the build process.

What are your main concerns about breaking the download infrastructure?

In regards to licensing infrastructure, is it possible to create a script that downloads the license metadata from https://pypi.python.org/ and makes it available to "make legal-info"?

Putting STAGING_DIR in the PATH is really bad. Isn't there a way to

Yes, I was wrong. The staging bin directory has binaries for the target platform in it. Oops.

The path modification is now removed. I have tried giving pip an argument which contained --with-xslt-config (supported by lxml), but other modules I install at the same time trigger errors because they don't recognise this option.

Maybe I can create a directory which contains symlinks to only the "-config" scripts from staging, and then stick that directory of symlinks on PATH temporarily. I'm not sure where you stand on such a strategy.

I'm not sure why either. Comment deleted.

I will add a patch for this.

Isn't there a way of installing pip only on the host, use it during the

From your suggestion, I have removed the installation of pip to the target with no resulting errors.

Ok I have tested this again, and I have found I no longer need to change the PYTHONPATH in order to get setuptools to work. So these changes have been reverted as well.

Thanks for the feedback. I will send updated patches through shortly.

Regards,
Rohan Fletcher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130715/445db96f/attachment-0001.html>