Skip to contentSkip to conversation

tools/python: Pass linker to Python build process

PATCH RESEND In review Elliott Mitchell 3 participants 6 messages Oct 12, 2020 – Oct 16, 2020

lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 tools/python: Pass linker to Python build process

Unexpectedly the environment variable which needs to be passed is $LDSHARED and not $LD. Otherwise Python may find the build ld instead of the host ld.

Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects it can load at runtime, not executables.

This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS to $LDFLAGS which breaks many linkers.

---
This is now the *third* time this has been sent to the list. Mark Pryor has tested and confirms Python cross-building is working. There is one wart left which I'm unsure of the best approach for.

Having looked around a bit, I believe this is a Python 2/3 compatibility issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or $LD variable, whereas Python 3 does have this. Alas this is pointless due to the above (unless you can cause distutils.py to do the final link step separately).

---
 tools/pygrub/Makefile | 9 +++++----
 tools/python/Makefile | 9 +++++----
 2 files changed, 10 insertions(+), 8 deletions(-)
tools/pygrub/Makefile +5 -4
index 3063c4998f..37b2146214 100644
@@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: all
 all: build
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)/$(bindir)
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		 --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force
 	set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \
 	             "`readlink -f $(DESTDIR)/$(bindir)`" != \
tools/python/Makefile +5 -4
index 8d22c03676..b675f5b4de 100644
@@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk
 all: build
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install:
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
 
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		--root="$(DESTDIR)" --force
 
 	$(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN)
Signature · 8 lines
2.20.1


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         [email protected]  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
#1 lore raw
Quoted 16 lines · Elliott Mitchell

Unexpectedly the environment variable which needs to be passed is $LDSHARED and not $LD. Otherwise Python may find the build ld instead of the host ld.

Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects it can load at runtime, not executables.

This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS to $LDFLAGS which breaks many linkers.

---
This is now the *third* time this has been sent to the list. Mark Pryor has tested and confirms Python cross-building is working. There is one wart left which I'm unsure of the best approach for.

Quoted 5 lines

Having looked around a bit, I believe this is a Python 2/3 compatibility issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or $LD variable, whereas Python 3 does have this. Alas this is pointless due to the above (unless you can cause distutils.py to do the final link step separately).

I think this is well-reasoned but I don't have time to figure out and verify the details.

Marek, do you have any comment on this?

Quoted 75 lines
---
 tools/pygrub/Makefile | 9 +++++----
 tools/python/Makefile | 9 +++++----
 2 files changed, 10 insertions(+), 8 deletions(-)
tools/pygrub/Makefile +5 -4
index 3063c4998f..37b2146214 100644
@@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: all
 all: build
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)/$(bindir)
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		 --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force
 	set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \
 	             "`readlink -f $(DESTDIR)/$(bindir)`" != \
tools/python/Makefile +5 -4
index 8d22c03676..b675f5b4de 100644
@@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk
 all: build
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install:
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
 
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		--root="$(DESTDIR)" --force
 
 	$(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN)
Signature · 8 lines
2.20.1


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         [email protected]  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
#2 lore raw
Quoted 17 lines · Wei Liu
Quoted 16 lines · Elliott Mitchell

Unexpectedly the environment variable which needs to be passed is $LDSHARED and not $LD. Otherwise Python may find the build ld instead of the host ld.

Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects it can load at runtime, not executables.

This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS to $LDFLAGS which breaks many linkers.

---
This is now the *third* time this has been sent to the list. Mark Pryor has tested and confirms Python cross-building is working. There is one wart left which I'm unsure of the best approach for.

Quoted 8 lines
Quoted 5 lines

Having looked around a bit, I believe this is a Python 2/3 compatibility issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or $LD variable, whereas Python 3 does have this. Alas this is pointless due to the above (unless you can cause distutils.py to do the final link step separately).

I think this is well-reasoned but I don't have time to figure out and verify the details.

Yes, it looks like distutils in Python 2 was even more limited than the one in Python 3.

Quoted 77 lines

Marek, do you have any comment on this?

Quoted 75 lines
---
 tools/pygrub/Makefile | 9 +++++----
 tools/python/Makefile | 9 +++++----
 2 files changed, 10 insertions(+), 8 deletions(-)
tools/pygrub/Makefile +5 -4
index 3063c4998f..37b2146214 100644
@@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: all
 all: build
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)/$(bindir)
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		 --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force
 	set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \
 	             "`readlink -f $(DESTDIR)/$(bindir)`" != \
tools/python/Makefile +5 -4
index 8d22c03676..b675f5b4de 100644
@@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk
 all: build
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install:
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
 
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		--root="$(DESTDIR)" --force
 
 	$(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN)
Signature · 8 lines
2.20.1


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         [email protected]  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
#3 lore raw
Quoted 8 lines · Marek Marczykowski-G??recki
Quoted 7 lines · Wei Liu
Quoted 5 lines · Elliott Mitchell

Having looked around a bit, I believe this is a Python 2/3 compatibility issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or $LD variable, whereas Python 3 does have this. Alas this is pointless due to the above (unless you can cause distutils.py to do the final link step separately).

Quoted 5 lines
Quoted 2 lines

I think this is well-reasoned but I don't have time to figure out and verify the details.

Yes, it looks like distutils in Python 2 was even more limited than the one in Python 3.

Actually feels like two steps forward, one step back. Separate $LDSHARED, yet $CFLAGS is appended to $LDFLAGS during invocation.

Architecture name is included the output Python extension filename, yet the only way to override requires writing Python code. I've got ideas to workaround this, but they're rather gross.

Signature · 4 lines
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         [email protected]  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
Acked-by on v1 1/1
#4 lore raw
In reply to Elliott Mitchell
Quoted 11 lines · Elliott Mitchell

Unexpectedly the environment variable which needs to be passed is $LDSHARED and not $LD. Otherwise Python may find the build ld instead of the host ld.

Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects it can load at runtime, not executables.

This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS to $LDFLAGS which breaks many linkers.

Quoted 74 lines

---
This is now the *third* time this has been sent to the list. Mark Pryor has tested and confirms Python cross-building is working. There is one wart left which I'm unsure of the best approach for.

Having looked around a bit, I believe this is a Python 2/3 compatibility issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or $LD variable, whereas Python 3 does have this. Alas this is pointless due to the above (unless you can cause distutils.py to do the final link step separately).

---
 tools/pygrub/Makefile | 9 +++++----
 tools/python/Makefile | 9 +++++----
 2 files changed, 10 insertions(+), 8 deletions(-)
tools/pygrub/Makefile +5 -4
index 3063c4998f..37b2146214 100644
@@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: all
 all: build
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)/$(bindir)
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		 --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force
 	set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \
 	             "`readlink -f $(DESTDIR)/$(bindir)`" != \
tools/python/Makefile +5 -4
index 8d22c03676..b675f5b4de 100644
@@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk
 all: build
 
 PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS)
-PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS)
+PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS)
 INSTALL_LOG = build/installed_files.txt
 
 .PHONY: build
 build:
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build
 
 .PHONY: install
 install:
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
 
-	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \
-		setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
+	CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \
+		LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \
+		--record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \
 		--root="$(DESTDIR)" --force
 
 	$(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN)
Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
#5 lore raw
Quoted 8 lines · Marek Marczykowski-Górecki
Quoted 7 lines · Elliott Mitchell

Unexpectedly the environment variable which needs to be passed is $LDSHARED and not $LD. Otherwise Python may find the build ld instead of the host ld.

Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects it can load at runtime, not executables.

Quoted 6 lines
Quoted 4 lines

This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS to $LDFLAGS which breaks many linkers.

Thanks. Applied.

Wei.