Skip to contentSkip to conversation

python-setuptools: forkbump and add workaround for shebang

PATCH Superseded Gustavo Zacarias 3 participants 13 messages Apr 13, 2013 – May 6, 2013

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

Switch to distribute fork since it's maintained and fixes a few issues while bumping to version 0.6.36.
There's work upstream in re-merging with setuptools which is now supposedly maintained, see
http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html

---
 .../python-setuptools-add-executable.patch         | 50 ++++++++++++++++++++++
 package/python-setuptools/python-setuptools.mk     |  8 ++--
 2 files changed, 54 insertions(+), 4 deletions(-)
 create mode 100644 package/python-setuptools/python-setuptools-add-executable.patch
package/python-setuptools/python-setuptools-add-executable.patch new file +50 -0
new file mode 100644
index 0000000..a14a3ed
@@ -0,0 +1,50 @@
+Add a new --executable option to distribute so that we can
+force the shebang line in installed python scripts.
+
+Signed-off-by: Gustavo Zacarias <[email protected]>
+
+diff -Nura distribute-0.6.36.orig/setuptools/command/install.py distribute-0.6.36/setuptools/command/install.py
+--- distribute-0.6.36.orig/setuptools/command/install.py	2013-04-13 09:46:23.160823598 -0300
++++ distribute-0.6.36/setuptools/command/install.py	2013-04-13 11:17:28.418841986 -0300
+@@ -6,6 +6,7 @@
+     """Use easy_install to install the package, w/dependencies"""
+
+     user_options = _install.user_options + [
++        ('executable=', 'e', "specify final destination interpreter path"),
+         ('old-and-unmanageable', None, "Try not to use this!"),
+         ('single-version-externally-managed', None,
+             "used by system package builders to create 'flat' eggs"),
+@@ -21,6 +22,7 @@
+
+     def initialize_options(self):
+         _install.initialize_options(self)
++	self.executable = None
+         self.old_and_unmanageable = None
+         self.single_version_externally_managed = None
+         self.no_compile = None  # make DISTUTILS_DEBUG work right!
+diff -Nura distribute-0.6.36.orig/setuptools/command/install_scripts.py distribute-0.6.36/setuptools/command/install_scripts.py
+--- distribute-0.6.36.orig/setuptools/command/install_scripts.py	2013-04-13 09:46:23.160823598 -0300
++++ distribute-0.6.36/setuptools/command/install_scripts.py	2013-04-13 11:24:32.305416400 -0300
+@@ -10,6 +10,13 @@
+     def initialize_options(self):
+         _install_scripts.initialize_options(self)
+         self.no_ep = False
++	self.executable = None
++
++    def finalize_options(self):
++        _install_scripts.finalize_options(self)
++        self.set_undefined_options('install',
++	    ('executable','executable')
++	)
+
+     def run(self):
+         from setuptools.command.easy_install import get_script_args
+@@ -31,6 +38,8 @@
+         )
+         bs_cmd = self.get_finalized_command('build_scripts')
+         executable = getattr(bs_cmd,'executable',sys_executable)
++	if self.executable is not None:
++	    executable = self.executable
+         is_wininst = getattr(
+             self.get_finalized_command("bdist_wininst"), '_is_running', False
+         )
package/python-setuptools/python-setuptools.mk +4 -4
index 25fddb9..64f6ded 100644
@@ -4,9 +4,9 @@
 #
 #############################################################
 
-PYTHON_SETUPTOOLS_VERSION = 0.6c11
-PYTHON_SETUPTOOLS_SOURCE  = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
-PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/s/setuptools
+PYTHON_SETUPTOOLS_VERSION = 0.6.36
+PYTHON_SETUPTOOLS_SOURCE  = distribute-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
+PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/d/distribute
 PYTHON_SETUPTOOLS_DEPENDENCIES = python
 
 define HOST_PYTHON_SETUPTOOLS_BUILD_CMDS
@@ -24,7 +24,7 @@ endef
 
 define PYTHON_SETUPTOOLS_INSTALL_TARGET_CMDS
 	(cd $(@D); PYTHONPATH="$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages"\
-	$(HOST_DIR)/usr/bin/python setup.py install --prefix=$(TARGET_DIR)/usr)
+	$(HOST_DIR)/usr/bin/python setup.py install --executable=/usr/bin/python --single-version-externally-managed --root=/ --prefix=$(TARGET_DIR)/usr)
 endef
 
 $(eval $(generic-package))
Signature · 1 line
1.8.1.5
#1 lore raw

PATCH 2/3 python-bottle: bump version and fix shebang

Bump to version 0.11.6
Add the --executable shebang fixup option to the build phase so that the script ends with a proper #!/usr/bin/python interpeter line set.

---
 package/python-bottle/python-bottle.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-bottle/python-bottle.mk +2 -2
index 370b31b..9df359f 100644
@@ -4,7 +4,7 @@
 #
 #############################################################
 
-PYTHON_BOTTLE_VERSION = 0.11.4
+PYTHON_BOTTLE_VERSION = 0.11.6
 PYTHON_BOTTLE_SOURCE = bottle-$(PYTHON_BOTTLE_VERSION).tar.gz
 PYTHON_BOTTLE_SITE = http://pypi.python.org/packages/source/b/bottle
 PYTHON_BOTTLE_DEPENDENCIES = python
@@ -12,7 +12,7 @@ PYTHON_BOTTLE_LICENSE = MIT
 # README.rst refers to the file "LICENSE" but it's not included
 
 define PYTHON_BOTTLE_BUILD_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
+	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build --executable=/usr/bin/python)
 endef
 
 define PYTHON_BOTTLE_INSTALL_TARGET_CMDS
Signature · 1 line
1.8.1.5
#2 lore raw

PATCH 2/3 python-bottle: bump version and fix shebang

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Gustavo" == Gustavo Zacarias <[email protected]> writes:

Gustavo> Bump to version 0.11.6

Gustavo> Add the --executable shebang fixup option to the build phase Gustavo> so that the script ends with a proper #!/usr/bin/python Gustavo> interpeter line set.

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#3 lore raw

PATCH 3/3 python-serial: fix shebang

In reply to Gustavo Zacarias

Add the --executable shebang fixup option to the build phase so that the script ends with a proper #!/usr/bin/python interpeter line set.

---
 package/python-serial/python-serial.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
package/python-serial/python-serial.mk +2 -2
index fb6b31c..4bfc355 100644
@@ -6,12 +6,12 @@
 
 PYTHON_SERIAL_VERSION = 2.6
 PYTHON_SERIAL_SOURCE  = pyserial-$(PYTHON_SERIAL_VERSION).tar.gz
-PYTHON_SERIAL_SITE    = http://pypi.python.org/packages/source/p/pyserial/
+PYTHON_SERIAL_SITE    = http://pypi.python.org/packages/source/p/pyserial
 
 PYTHON_SERIAL_DEPENDENCIES = python
 
 define PYTHON_SERIAL_BUILD_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
+	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build --executable=/usr/bin/python)
 endef
 
 define PYTHON_SERIAL_INSTALL_TARGET_CMDS
Signature · 1 line
1.8.1.5
#4 lore raw

PATCH 3/3 python-serial: fix shebang

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Gustavo" == Gustavo Zacarias <[email protected]> writes:

Gustavo> Add the --executable shebang fixup option to the build phase Gustavo> so that the script ends with a proper #!/usr/bin/python Gustavo> interpeter line set.

Committed, thanks.

I see we have the same issue with usr/bin/smtpd.py from python itself, but I'm not sure how to fix that (besides running sed after installation).

Any idea?

Signature · 1 line
Bye, Peter Korsgaard
#5 lore raw

PATCH 3/3 python-serial: fix shebang

Quoted 5 lines · Peter Korsgaard

I see we have the same issue with usr/bin/smtpd.py from python itself, but I'm not sure how to fix that (besides running sed after installation).

Any idea?

Some (all?) distributions just remove it, it's a smtp proxy, kind of a sample program, not something we want lingering on the target filesystem IMHO.
Regards.

#6 lore raw

PATCH 3/3 python-serial: fix shebang

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Gustavo" == Gustavo Zacarias <[email protected]> writes:

Quoted 5 lines · Peter Korsgaard
Quoted 5 lines

I see we have the same issue with usr/bin/smtpd.py from python itself, but I'm not sure how to fix that (besides running sed after installation).

Any idea?

Gustavo> Some (all?) distributions just remove it, it's a smtp proxy, Gustavo> kind of a sample program, not something we want lingering on Gustavo> the target filesystem

Ok, thanks. Care to send a patch doing that?

Signature · 1 line
Bye, Peter Korsgaard
#7 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

In reply to Gustavo Zacarias

Hi Gustavo,

I usually don't bother reviewing your patches, but now I found something:

Quoted 30 lines · Gustavo Zacarias

Switch to distribute fork since it's maintained and fixes a few issues while bumping to version 0.6.36.
There's work upstream in re-merging with setuptools which is now supposedly maintained, see
http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html

---
  .../python-setuptools-add-executable.patch         | 50 ++++++++++++++++++++++
  package/python-setuptools/python-setuptools.mk     |  8 ++--
  2 files changed, 54 insertions(+), 4 deletions(-)

create mode 100644 package/python-setuptools/python-setuptools-add-executable.patch

package/python-setuptools/python-setuptools-add-executable.patch new file +11 -0
new file mode 100644
index 0000000..a14a3ed
@@ -0,0 +1,50 @@
+Add a new --executable option to distribute so that we can
+force the shebang line in installed python scripts.
+
+Signed-off-by: Gustavo Zacarias <[email protected]>
+
+diff -Nura distribute-0.6.36.orig/setuptools/command/install.py distribute-0.6.36/setuptools/command/install.py
+--- distribute-0.6.36.orig/setuptools/command/install.py	2013-04-13 09:46:23.160823598 -0300
++++ distribute-0.6.36/setuptools/command/install.py	2013-04-13 11:17:28.418841986 -0300
+@@ -6,6 +6,7 @@
+     """Use easy_install to install the package, w/dependencies"""
+

package/python-setuptools/python-setuptools-add-executable.patch line 21

Quoted 10 lines
package/python-setuptools/python-setuptools-add-executable.patch new file +10 -0
+     user_options = _install.user_options + [
++        ('executable=', 'e', "specify final destination interpreter path"),
+         ('old-and-unmanageable', None, "Try not to use this!"),
+         ('single-version-externally-managed', None,
+             "used by system package builders to create 'flat' eggs"),
+@@ -21,6 +22,7 @@
+
+     def initialize_options(self):
+         _install.initialize_options(self)
++	self.executable = None

Mixing tabs with spaces... Also a few other instances in the rest of
this patch.

Quoted 34 lines
package/python-setuptools/python-setuptools-add-executable.patch new file +29 -0
+         self.old_and_unmanageable = None
+         self.single_version_externally_managed = None
+         self.no_compile = None  # make DISTUTILS_DEBUG work right!
+diff -Nura distribute-0.6.36.orig/setuptools/command/install_scripts.py distribute-0.6.36/setuptools/command/install_scripts.py
+--- distribute-0.6.36.orig/setuptools/command/install_scripts.py	2013-04-13 09:46:23.160823598 -0300
++++ distribute-0.6.36/setuptools/command/install_scripts.py	2013-04-13 11:24:32.305416400 -0300
+@@ -10,6 +10,13 @@
+     def initialize_options(self):
+         _install_scripts.initialize_options(self)
+         self.no_ep = False
++	self.executable = None
++
++    def finalize_options(self):
++        _install_scripts.finalize_options(self)
++        self.set_undefined_options('install',
++	    ('executable','executable')
++	)
+
+     def run(self):
+         from setuptools.command.easy_install import get_script_args
+@@ -31,6 +38,8 @@
+         )
+         bs_cmd = self.get_finalized_command('build_scripts')
+         executable = getattr(bs_cmd,'executable',sys_executable)
++	if self.executable is not None:
++	    executable = self.executable
+         is_wininst = getattr(
+             self.get_finalized_command("bdist_wininst"), '_is_running', False
+         )
package/python-setuptools/python-setuptools.mk +0 -0
index 25fddb9..64f6ded 100644

package/python-setuptools/python-setuptools.mk line 8

Quoted 8 lines
package/python-setuptools/python-setuptools.mk +2 -3
@@ -4,9 +4,9 @@
  #
  #############################################################
 
-PYTHON_SETUPTOOLS_VERSION = 0.6c11
-PYTHON_SETUPTOOLS_SOURCE  = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
-PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/s/setuptools
+PYTHON_SETUPTOOLS_VERSION = 0.6.36
+PYTHON_SETUPTOOLS_SOURCE  = distribute-$(PYTHON_SETUPTOOLS_VERSION).tar.gz

I would add a comment here, e.g.

# "distribute" is a fork of the unmaintained setuptools package. There

# are plans to re-merge it into setuptools; if this happens, we can
# switch back to http://pypi.python.org/packages/source/s/setuptools

package/python-setuptools/python-setuptools.mk endef · line 27

Quoted 10 lines
package/python-setuptools/python-setuptools.mk +2 -1
+PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/d/distribute
  PYTHON_SETUPTOOLS_DEPENDENCIES = python
 
  define HOST_PYTHON_SETUPTOOLS_BUILD_CMDS
@@ -24,7 +24,7 @@ endef
 
  define PYTHON_SETUPTOOLS_INSTALL_TARGET_CMDS
  	(cd $(@D); PYTHONPATH="$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages"\
-	$(HOST_DIR)/usr/bin/python setup.py install --prefix=$(TARGET_DIR)/usr)
+	$(HOST_DIR)/usr/bin/python setup.py install --executable=/usr/bin/python --single-version-externally-managed --root=/ --prefix=$(TARGET_DIR)/usr)

Please split this long line. Also, maybe it's worth explaining why we
want --single-version-externally-managed?

Regards,
Arnout

Quoted 4 lines
package/python-setuptools/python-setuptools.mk +0 -0
  endef
 
  $(eval $(generic-package))
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#8 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

Quoted 2 lines · Arnout Vandecappelle

Mixing tabs with spaces... Also a few other instances in the rest of
this patch.

That's probably my vim settings doing something evil, i don't have a special case for python, i'll fix it.

Quoted 5 lines

I would add a comment here, e.g.

# "distribute" is a fork of the unmaintained setuptools package. There

# are plans to re-merge it into setuptools; if this happens, we can
# switch back to http://pypi.python.org/packages/source/s/setuptools

Fair enough.

Quoted 2 lines

Please split this long line. Also, maybe it's worth explaining why we
want --single-version-externally-managed?

That's for the trick to work, when python builds an egg (zip file) in the build phase not only are we wasting time doing double decompression later on the target but we are also finishing up the package (kind of a mini-install phase) into the egg.
That makes the trick fail (probably solvable by doing --execute in the build itself, didn't try since it's not the way IMHO).
I'll add a note since it's required for other python packages that do use setuptools/distribution.
V2 when Thomas confirms it solves his issues with lava-test, did it for me in my tests.
Regards.

#9 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Gustavo" == Gustavo Zacarias <[email protected]> writes:

Hi,

Quoted 2 lines
Quoted 2 lines

Please split this long line. Also, maybe it's worth explaining why we want --single-version-externally-managed?

Gustavo> That's for the trick to work, when python builds an egg (zip Gustavo> file) in the build phase not only are we wasting time doing Gustavo> double decompression later on the target but we are also Gustavo> finishing up the package (kind of a mini-install phase) into Gustavo> the egg.

Gustavo> That makes the trick fail (probably solvable by doing Gustavo> --execute in the build itself, didn't try since it's not the Gustavo> way IMHO). I'll add a note since it's required for other Gustavo> python packages that do use setuptools/distribution. V2 when Gustavo> Thomas confirms it solves his issues with lava-test, did it Gustavo> for me in my tests.

Did this ever get tested? Thomas, could you send your ack / Gustavo, could you send a v2 fixing the issues pointed out by Arnout?

Signature · 1 line
Bye, Peter Korsgaard
#10 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

Quoted 2 lines · Peter Korsgaard

Did this ever get tested? Thomas, could you send your ack / Gustavo, could you send a v2 fixing the issues pointed out by Arnout?

It's partially solved for lava-test since it's pulling other dependencies by itself - that should probably be solved via new buildroot python packages since it's doing so because they're not found, and it's not nice when a package starts downloading stuff at build/install time.
Yes, a V2 is forthcoming addressing Arnout's comments and a few more fixes/improvements.
You shouldn't have committed the fixes to python-bottle and python-serial since they both depend on the patch to this package, they'll probably fail now.
Regards.

#11 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Gustavo" == Gustavo Zacarias <[email protected]> writes:

Quoted 2 lines · Peter Korsgaard
Quoted 2 lines

Did this ever get tested? Thomas, could you send your ack / Gustavo, could you send a v2 fixing the issues pointed out by Arnout?

Gustavo> It's partially solved for lava-test since it's pulling other Gustavo> dependencies by itself - that should probably be solved via new Gustavo> buildroot python packages since it's doing so because they're not found, Gustavo> and it's not nice when a package starts downloading stuff at Gustavo> build/install time.
Gustavo> Yes, a V2 is forthcoming addressing Arnout's comments and a few more Gustavo> fixes/improvements.

Ok, great.

Gustavo> You shouldn't have committed the fixes to python-bottle and Gustavo> python-serial since they both depend on the patch to this package, Gustavo> they'll probably fail now.

Hmm, odd - They built fine here:

Quoted 1 line
Quoted 1 line
Quoted 1 line

python-bottle 0.11.6 Building

(cd /home/peko/source/buildroot/output/build/python-bottle-0.11.6; /home/peko/source/buildroot/output/host/usr/bin/python setup.py build --executable=/usr/bin/python) running build
running build_py
creating build
creating build/lib
copying bottle.py -> build/lib
running build_scripts
creating build/scripts-2.7
copying and adjusting bottle.py -> build/scripts-2.7
changing mode of build/scripts-2.7/bottle.py from 644 to 755

Signature · 1 line
Bye, Peter Korsgaard
#12 lore raw

PATCH 1/3 python-setuptools: forkbump and add workaround for shebang

Quoted 1 line · Peter Korsgaard

Hmm, odd - They built fine here:

My bad, those are python distutils-based which has the --executable functionality builtin (yes, there's more than one way of building/installing modules in python, yes, it hurts). Regards.