Skip to contentSkip to conversation

Python 3, simple, external package support

PATCH Superseded Patrick Gerber 3 participants 14 messages Feb 12, 2013

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 Python 3, simple, external package support

This patch series refactor python support in buildroot to enable the use of from line 'This patch series refactor python support in buildroot to enable the use of external package with both python2 and python3. The packages python-bottle and python-serial have adapted as examples.

To support others external package more work is needed as they are based on python setuptools. It's no more supported by python3 and should be replaced by "Distribute". This should be done in another patch series.

v2: Add short patch title

#1 lore raw

PATCH 1/5 Python menu reorganization

In order to have a better support for python external package make the python menu clearer. Python2 and Python3 are now mutually exclusive

---
 package/Config.in                |   19 +------------
 package/python-generic/Config.in |   56 ++++++++++++++++++++++++++++++++++++++
 package/python/Config.in         |   14 ----------
 package/python3/Config.in        |   14 ----------
 4 files changed, 57 insertions(+), 46 deletions(-)
 create mode 100644 package/python-generic/Config.in
package/Config.in +1 -18
index 2fad94d..b7da22f 100644
@@ -315,24 +315,7 @@ endmenu
 endif
 source "package/microperl/Config.in"
 source "package/php/Config.in"
-source "package/python/Config.in"
-source "package/python3/Config.in"
-if BR2_PACKAGE_PYTHON
-menu "external python modules"
-source "package/python-bottle/Config.in"
-source "package/python-dpkt/Config.in"
-source "package/python-id3/Config.in"
-source "package/python-mad/Config.in"
-source "package/python-meld3/Config.in"
-source "package/python-netifaces/Config.in"
-source "package/python-nfc/Config.in"
-source "package/python-protobuf/Config.in"
-source "package/python-pygame/Config.in"
-source "package/python-pyparsing/Config.in"
-source "package/python-serial/Config.in"
-source "package/python-setuptools/Config.in"
-endmenu
-endif
+source "package/python-generic/Config.in"
 source "package/ruby/Config.in"
 source "package/tcl/Config.in"
 if BR2_PACKAGE_TCL
package/python-generic/Config.in new file +56 -0

56 lines not on this page · raw message

package/python/Config.in +0 -14
index 4089cce..b4fd6c2 100644
@@ -1,17 +1,3 @@
-config BR2_PACKAGE_PYTHON
-	bool "python"
-	depends on BR2_USE_WCHAR
-	# uses fork()
-	depends on BR2_USE_MMU
-	select BR2_PACKAGE_LIBFFI
-	help
-	  The python language interpreter.
-
-	  http://www.python.org/
-
-comment "python requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON
 
 choice
package/python3/Config.in +0 -14
index 5959d31..279f31b 100644
@@ -1,17 +1,3 @@
-config BR2_PACKAGE_PYTHON3
-	bool "python3"
-	depends on BR2_USE_WCHAR
-	# uses fork()
-	depends on BR2_USE_MMU
-	select BR2_PACKAGE_LIBFFI
-	help
-	  The python language interpreter.
-
-	  http://www.python.org/
-
-comment "python3 requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON3
 
 choice
Signature · 1 line
1.7.10.4
#2 lore raw

PATCH 2/5 Add python3 to python simlinks in staging

In reply to Patrick Gerber

When using python3 a symlink "python" is added in staging and point to python3. This is needed to call python setup for external package. A second simlink is added to alias the config-3.3m directory to config

---
 package/python3/python3.mk |    8 ++++++++
 1 file changed, 8 insertions(+)
package/python3/python3.mk +8 -0
index 7551a7b..5974824 100644
@@ -160,6 +160,14 @@ ifneq ($(BR2_PACKAGE_PYTHON),y)
 PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_INSTALL_SYMLINK
 endif
 
+define HOST_PYTHON3_INSTALL_SYMLINK
+	ln -fs python3 $(HOST_DIR)/usr/bin/python
+	ln -fs config-3.3m $(HOST_DIR)/usr/lib/python3.3/config
+endef
+
+HOST_PYTHON3_POST_INSTALL_HOOKS += HOST_PYTHON3_INSTALL_SYMLINK
+
+
 ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
 define PYTHON3_REMOVE_MODULES_FILES
 	for i in `find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) \
Signature · 1 line
1.7.10.4
#3 lore raw

PATCH 2/5 Add python3 to python simlinks in staging

Hi Patrick,

2013/2/12 Patrick Gerber <[email protected]>:

Quoted 1 line

When using python3 a symlink "python" is added in staging and point to

s/staging/host directory/

Quoted 2 lines

python3. This is needed to call python setup for external package. A second simlink is added to alias the config-3.3m directory to config

s/simlink/symlink/

Quoted 11 lines
---
 package/python3/python3.mk |    8 ++++++++
 1 file changed, 8 insertions(+)
package/python3/python3.mk +0 -0
index 7551a7b..5974824 100644

package/python3/python3.mk ifneq ($(BR2_PACKAGE_PYTHON),y) · line 168

Quoted 9 lines
package/python3/python3.mk +6 -0
@@ -160,6 +160,14 @@ ifneq ($(BR2_PACKAGE_PYTHON),y)
 PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_INSTALL_SYMLINK
 endif
 
+define HOST_PYTHON3_INSTALL_SYMLINK
+       ln -fs python3 $(HOST_DIR)/usr/bin/python
+       ln -fs config-3.3m $(HOST_DIR)/usr/lib/python3.3/config
+endef
+
+HOST_PYTHON3_POST_INSTALL_HOOKS += HOST_PYTHON3_INSTALL_SYMLINK

BTW, I'm a bit concerned about the consequencies this may have...

From my experience, having python poiting to python3 in the PATH has some bad effect,
and some patch have been posted some time ago to fix this issue:
http://patchwork.ozlabs.org/patch/208803/

Also, I'm dubious about having python poiting to python3. Though there is nothing really stated, there is usually a consensus:
"python" refers to the
python-2 interpreter if it exists on the system, otherwise it refers to the python-3 one (if existing).

What worries me is that, python (i.e. python 2.6 or 2.7) is currently a mandatory for Buildroot
(see: http://buildroot.org/downloads/manual/manual.html#requirement-mandatory),
and here,
you are overriding it with python3 in the PATH inside Buildroot.

I undersatnd that, when BR2_PACKAGE_PYTHON3_PY_ONLY is enabled, you want to build
all modules using python3, but meanwhile some other build scripts may use 'python' referring
to python2 and may badly failed because they will trigger python3...

I may have missed something... so, enlighten me please.

Regards,

Signature · 1 line
Samuel
#4 lore raw

PATCH 3/5 Add common declaration to ease the support of both version

In reply to Patrick Gerber

Define a new variable called PYTHON_GENERIC_DEPENDENCIES either to "python" or "python3" depending of the selected Python version. This is needed to easily set dependencies for packages that could be used either with Python 2 or Python 3.

---
 package/python-generic/python-generic.mk |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 package/python-generic/python-generic.mk
package/python-generic/python-generic.mk new file +16 -0
new file mode 100644
index 0000000..0cf0e9d
@@ -0,0 +1,16 @@
+#############################################################
+#
+# python-generic
+#
+#############################################################
+# PYTHON_GENERIC_DEPENDENCIES is defined to the selected python 
+# version. 
+
+ifeq ($(BR2_PACKAGE_PYTHON),y)
+	PYTHON_GENERIC_DEPENDENCIES = python
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+	PYTHON_GENERIC_DEPENDENCIES = python3
+endif
+
Signature · 1 line
1.7.10.4
#5 lore raw

PATCH 4/5 python-bottle: Convert package to be used with python2 or python3

In reply to Patrick Gerber
---
 package/python-bottle/Config.in        |    1 +
 package/python-bottle/python-bottle.mk |    2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
package/python-bottle/Config.in +1 -0
index f9b60ca..e6f2f3b 100644
@@ -1,5 +1,6 @@
 config BR2_PACKAGE_PYTHON_BOTTLE
 	bool "python-bottle"
+	depends on (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
 	help
 	  Bottle is a fast, simple and lightweight WSGI micro web-framework
 	  for Python. It is distributed as a single file module and has no
package/python-bottle/python-bottle.mk +1 -1
index 370b31b..d24e0ec 100644
@@ -7,7 +7,7 @@
 PYTHON_BOTTLE_VERSION = 0.11.4
 PYTHON_BOTTLE_SOURCE = bottle-$(PYTHON_BOTTLE_VERSION).tar.gz
 PYTHON_BOTTLE_SITE = http://pypi.python.org/packages/source/b/bottle
-PYTHON_BOTTLE_DEPENDENCIES = python
+PYTHON_BOTTLE_DEPENDENCIES = $(PYTHON_GENERIC_DEPENDENCIES)
 PYTHON_BOTTLE_LICENSE = MIT
 # README.rst refers to the file "LICENSE" but it's not included
 
Signature · 1 line
1.7.10.4
#6 lore raw

PATCH 4/5 python-bottle: Convert package to be used with python2 or python3

Dear Patrick Gerber,

Quoted 12 lines · Patrick Gerber
---
 package/python-bottle/Config.in        |    1 +
 package/python-bottle/python-bottle.mk |    2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
package/python-bottle/Config.in +0 -0
index f9b60ca..e6f2f3b 100644

package/python-bottle/Config.in line 3

Quoted 3 lines
package/python-bottle/Config.in +1 -0
@@ -1,5 +1,6 @@
 config BR2_PACKAGE_PYTHON_BOTTLE
 	bool "python-bottle"
+	depends on (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)

This change is useless, since the python-bottle package is only available when either BR2_PACKAGE_PYTHON or BR2_PACKAGE_PYTHON3 are enabled:

+if (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
+menu "external python modules"
+source "package/python-bottle/Config.in"

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#7 lore raw

PATCH 5/5 python-serial: Convert package to be used with python2 or python3

In reply to Patrick Gerber
---
 package/python-serial/Config.in        |    2 +-
 package/python-serial/python-serial.mk |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
package/python-serial/Config.in +1 -1
index ea090bd..db6a4b1 100644
@@ -1,6 +1,6 @@
 config BR2_PACKAGE_PYTHON_SERIAL
 	bool "python-serial"
-	depends on BR2_PACKAGE_PYTHON
+	depends on (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
 	help
 	  python-serial is a Python library to access serial ports.
 
package/python-serial/python-serial.mk +1 -1
index fb6b31c..6f2bb85 100644
@@ -8,7 +8,7 @@ PYTHON_SERIAL_VERSION = 2.6
 PYTHON_SERIAL_SOURCE  = pyserial-$(PYTHON_SERIAL_VERSION).tar.gz
 PYTHON_SERIAL_SITE    = http://pypi.python.org/packages/source/p/pyserial/
 
-PYTHON_SERIAL_DEPENDENCIES = python
+PYTHON_SERIAL_DEPENDENCIES = $(PYTHON_GENERIC_DEPENDENCIES)
 
 define PYTHON_SERIAL_BUILD_CMDS
 	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
Signature · 1 line
1.7.10.4
#8 lore raw

PATCH v2 1/1 Python 3, simple, external package support

In reply to Patrick Gerber

Dear Patrick Gerber,

Quoted 8 lines · Patrick Gerber

This patch series refactor python support in buildroot to enable the use of from line 'This patch series refactor python support in buildroot to enable the use of external package with both python2 and python3. The packages python-bottle and python-serial have adapted as examples.

To support others external package more work is needed as they are based on python setuptools. It's no more supported by python3 and should be replaced by "Distribute". This should be done in another patch series.

Thanks, but it still doesn't answer the question:

"""
As Samuel pointed it out, this unfortunately may break a number of things in Buildroot. They are packages that depend on host-python, and would only be capable of using an Python 2.x host-python.

Or at least, I would like to have the confirmation that all packages that depend on host-python continue to build normally when host-python is Python 3.

That said, some packages are not really clear about this. For example, Scons uses the Python provided with the distribution, and not the host-python built by Buildroot.

As you can see, this is not a trivial topic.
"""

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
author
#9 lore raw

PATCH v2 1/1 Python 3, simple, external package support

Quoted 27 lines · Thomas Petazzoni

Dear Patrick Gerber,

Quoted 8 lines · Patrick Gerber

This patch series refactor python support in buildroot to enable the use of from line 'This patch series refactor python support in buildroot to enable the use of external package with both python2 and python3. The packages python-bottle and python-serial have adapted as examples.

To support others external package more work is needed as they are based on python setuptools. It's no more supported by python3 and should be replaced by "Distribute". This should be done in another patch series.

Thanks, but it still doesn't answer the question:

"""
As Samuel pointed it out, this unfortunately may break a number of things in Buildroot. They are packages that depend on host-python, and would only be capable of using an Python 2.x host-python.

Or at least, I would like to have the confirmation that all packages that depend on host-python continue to build normally when host-python is Python 3.

That said, some packages are not really clear about this. For example, Scons uses the Python provided with the distribution, and not the host-python built by Buildroot.

Quoted 7 lines

As you can see, this is not a trivial topic.
"""

Best regards,

Thomas

Related with host-python, the only difference is that now the patch add a symlink in HOST_DIR. I agree that this could be a problem.

I could edit the external python package to use "python" or "python3" depending of the selected version. This will not impact host-python at all. Does it seems good to you ?

Patrick

#10 lore raw

PATCH v2 1/1 Python 3, simple, external package support

Dear Patrick,

Quoted 6 lines · Patrick

Related with host-python, the only difference is that now the patch add a symlink in HOST_DIR. I agree that this could be a problem.

I could edit the external python package to use "python" or "python3" depending of the selected version. This will not impact host-python at all. Does it seems good to you ?

No, it doesn't solve the problem of packages that currently depend on host-python, and that expect a Python 2 to be installed.

You seem to be focused only on Python external modules, but you completely forget that there are packages in Buildroot that depend on host-python as their build dependencies, even though those packages are not directly Python related.

For example, are you sure that mesa3d and libxcb will built properly once host-python is Python 3 ?

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
author
#11 lore raw

PATCH v2 1/1 Python 3, simple, external package support

Quoted 19 lines · Thomas Petazzoni

Dear Patrick,

Quoted 6 lines · Patrick

Related with host-python, the only difference is that now the patch add a symlink in HOST_DIR. I agree that this could be a problem.

I could edit the external python package to use "python" or "python3" depending of the selected version. This will not impact host-python at all. Does it seems good to you ?

No, it doesn't solve the problem of packages that currently depend on host-python, and that expect a Python 2 to be installed.

You seem to be focused only on Python external modules, but you completely forget that there are packages in Buildroot that depend on host-python as their build dependencies, even though those packages are not directly Python related.

Quoted 5 lines

For example, are you sure that mesa3d and libxcb will built properly once host-python is Python 3 ?

Thomas

Thomas, I am aware that there is others packages that depends on host-python, mostly x11. But I have not changed the way host-python nor host-python3 are build and used. I don't understand how the changes I have made could break this compared as previously ? If I'm not mistaken, host-python is selected directly by the package (mesa3d for example) when needed, and this is independent of the selected python package, right ?

Patrick

#12 lore raw

PATCH v2 1/1 Python 3, simple, external package support

Dear Patrick,

Quoted 6 lines · Patrick

Thomas, I am aware that there is others packages that depends on host-python, mostly x11. But I have not changed the way host-python nor host-python3 are build and used. I don't understand how the changes I have made could break this compared as previously ? If I'm not mistaken, host-python is selected directly by the package (mesa3d for example) when needed, and this is independent of the selected python package, right ?

True.

So we might end up with two versions of Python installed in $(HOST_DIR): python2 and python3. Hopefully, it should work.

Regarding the patch set, I think I would prefer to rename the current python package to python2, and have the 'python' package be the virtual package that provides either python2 or python3. That's indeed more work, but I find the name "python-generic" not so great.

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
author
#13 lore raw

PATCH v2 1/1 Python 3, simple, external package support

Dear Thomas,

Quoted 11 lines · Thomas Petazzoni

Dear Patrick,

Quoted 6 lines · Patrick

Thomas, I am aware that there is others packages that depends on host-python, mostly x11. But I have not changed the way host-python nor host-python3 are build and used. I don't understand how the changes I have made could break this compared as previously ? If I'm not mistaken, host-python is selected directly by the package (mesa3d for example) when needed, and this is independent of the selected python package, right ?

Quoted 5 lines

True.

So we might end up with two versions of Python installed in $(HOST_DIR): python2 and python3. Hopefully, it should work.

This is already the case if I enable python3, no ?

Quoted 4 lines

Regarding the patch set, I think I would prefer to rename the current python package to python2, and have the 'python' package be the virtual package that provides either python2 or python3. That's indeed more work, but I find the name "python-generic" not so great.

I have tried like you suggest, but then I have to rename python.mk to python2.mk and then any occurrence of python or host-python to python2 and host-python2 in many package dependencies, no ?

But I could rename python-generic to python-common or something like that ?