Skip to contentSkip to conversation

perf tools: Use Python devtools for version autodetection

PATCH In review James Clark 2 participants 4 messages Mar 9, 2022 – Apr 12, 2022 perf events

lore
Share & tools
lore
#0 lore raw

cover perf tools: Use Python devtools for version autodetection

As mentioned here [1], I was going to try to attempt to improve Python autodetection in the scenario where Python3 devtools are installed alongside the Python2 runtime which is a pretty common scenario on a base system after running "apt install python3-dev".

Thanks
James

[1]: https://lkml.org/lkml/2021/11/30/614

James Clark (1):
perf tools: Use Python devtools for version autodetection rather than
runtime

 tools/perf/Makefile.config | 39 ++++++++++++++++++++++++++------------
 1 file changed, 27 insertions(+), 12 deletions(-)
Signature · 1 line
2.28.0
#1 lore raw

PATCH 1/1 perf tools: Use Python devtools for version autodetection rather than runtime

This fixes the issue where the build will fail if only the Python2 runtime is installed but the Python3 devtools are installed. Currently the workaround is 'make PYTHON=python3'.

Fix it by autodetecting Python based on whether python[x]-config exists rather than just python[x] because both are needed for the build. Then -config is stripped to find the Python runtime.

Testing
=======

* Auto detect links with Python3 when the v3 devtools are installed and only Python 2 runtime is installed
* Auto detect links with Python2 when both devtools are installed
* Sensible warning is printed if no Python devtools are installed
* 'make PYTHON=x' still automatically sets PYTHON_CONFIG=x-config
* 'make PYTHON=x' fails if x-config doesn't exist
* 'make PYTHON=python3' overrides Python2 devtools
* 'make PYTHON=python2' overrides Python3 devtools
* 'make PYTHON_CONFIG=x-config' works
* 'make PYTHON=x PYTHON_CONFIG=x' works
* 'make PYTHON=missing' reports an error
* 'make PYTHON_CONFIG=missing' reports an error

Fixes 79373082fa9d ("perf python: Autodetect python3 binary")
---
 tools/perf/Makefile.config | 39 ++++++++++++++++++++++++++------------
 1 file changed, 27 insertions(+), 12 deletions(-)
tools/perf/Makefile.config +27 -12
index 96ad944ca6a8..b3fbb746e7f0 100644
@@ -239,18 +239,33 @@ ifdef PARSER_DEBUG
 endif
 
 # Try different combinations to accommodate systems that only have
-# python[2][-config] in weird combinations but always preferring
-# python2 and python2-config as per pep-0394. If python2 or python
-# aren't found, then python3 is used.
-PYTHON_AUTO := python
-PYTHON_AUTO := $(if $(call get-executable,python3),python3,$(PYTHON_AUTO))
-PYTHON_AUTO := $(if $(call get-executable,python),python,$(PYTHON_AUTO))
-PYTHON_AUTO := $(if $(call get-executable,python2),python2,$(PYTHON_AUTO))
-override PYTHON := $(call get-executable-or-default,PYTHON,$(PYTHON_AUTO))
-PYTHON_AUTO_CONFIG := \
-  $(if $(call get-executable,$(PYTHON)-config),$(PYTHON)-config,python-config)
-override PYTHON_CONFIG := \
-  $(call get-executable-or-default,PYTHON_CONFIG,$(PYTHON_AUTO_CONFIG))
+# python[2][3]-config in weird combinations in the following order of
+# priority from lowest to highest:
+#   * python3-config
+#   * python-config
+#   * python2-config as per pep-0394.
+#   * $(PYTHON)-config (If PYTHON is user supplied but PYTHON_CONFIG isn't)
+#
+PYTHON_AUTO := python-config
+PYTHON_AUTO := $(if $(call get-executable,python3-config),python3-config,$(PYTHON_AUTO))
+PYTHON_AUTO := $(if $(call get-executable,python-config),python-config,$(PYTHON_AUTO))
+PYTHON_AUTO := $(if $(call get-executable,python2-config),python2-config,$(PYTHON_AUTO))
+
+# If PYTHON is defined but PYTHON_CONFIG isn't, then take $(PYTHON)-config as if it was the user
+# supplied value for PYTHON_CONFIG. Because it's "user supplied", error out if it doesn't exist.
+ifdef PYTHON
+  ifndef PYTHON_CONFIG
+    PYTHON_CONFIG_AUTO := $(call get-executable,$(PYTHON)-config)
+    PYTHON_CONFIG := $(if $(PYTHON_CONFIG_AUTO),$(PYTHON_CONFIG_AUTO),\
+                          $(call $(error $(PYTHON)-config not found)))
+  endif
+endif
+
+# Select either auto detected python and python-config or use user supplied values if they are
+# defined. get-executable-or-default fails with an error if the first argument is supplied but
+# doesn't exist.
+override PYTHON_CONFIG := $(call get-executable-or-default,PYTHON_CONFIG,$(PYTHON_AUTO))
+override PYTHON := $(call get-executable-or-default,PYTHON,$(subst -config,,$(PYTHON_AUTO)))
 
 grep-libs  = $(filter -l%,$(1))
 strip-libs  = $(filter-out -l%,$(1))
Signature · 1 line
2.28.0
#2 lore raw

Re: [PATCH 1/1] perf tools: Use Python devtools for version autodetection rather than runtime

Hi James,

Quoted 11 lines · James Clark

This fixes the issue where the build will fail if only the Python2 runtime is installed but the Python3 devtools are installed. Currently the workaround is 'make PYTHON=python3'.

Fix it by autodetecting Python based on whether python[x]-config exists rather than just python[x] because both are needed for the build. Then -config is stripped to find the Python runtime.

Testing
=======

Quoted 6 lines

* Auto detect links with Python3 when the v3 devtools are installed and only Python 2 runtime is installed
* Auto detect links with Python2 when both devtools are installed
* Sensible warning is printed if no Python devtools are installed
* 'make PYTHON=x' still automatically sets PYTHON_CONFIG=x-config
* 'make PYTHON=x' fails if x-config doesn't exist

If x is a valid python but no x-config is found in the system, the build fails, instead of printing a warning like before. If we use this approach I think [1] in the Makefile is never hit and needs to be cleaned up.

Thanks,
German

[1] https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.config?h=perf/core#n826:

Quoted 63 lines

* 'make PYTHON=python3' overrides Python2 devtools
* 'make PYTHON=python2' overrides Python3 devtools
* 'make PYTHON_CONFIG=x-config' works
* 'make PYTHON=x PYTHON_CONFIG=x' works
* 'make PYTHON=missing' reports an error
* 'make PYTHON_CONFIG=missing' reports an error

Fixes 79373082fa9d ("perf python: Autodetect python3 binary")
---
 tools/perf/Makefile.config | 39 ++++++++++++++++++++++++++------------
 1 file changed, 27 insertions(+), 12 deletions(-)
tools/perf/Makefile.config +27 -12
index 96ad944ca6a8..b3fbb746e7f0 100644
@@ -239,18 +239,33 @@ ifdef PARSER_DEBUG
 endif
 
 # Try different combinations to accommodate systems that only have
-# python[2][-config] in weird combinations but always preferring
-# python2 and python2-config as per pep-0394. If python2 or python
-# aren't found, then python3 is used.
-PYTHON_AUTO := python
-PYTHON_AUTO := $(if $(call get-executable,python3),python3,$(PYTHON_AUTO))
-PYTHON_AUTO := $(if $(call get-executable,python),python,$(PYTHON_AUTO))
-PYTHON_AUTO := $(if $(call get-executable,python2),python2,$(PYTHON_AUTO))
-override PYTHON := $(call get-executable-or-default,PYTHON,$(PYTHON_AUTO))
-PYTHON_AUTO_CONFIG := \
-  $(if $(call get-executable,$(PYTHON)-config),$(PYTHON)-config,python-config)
-override PYTHON_CONFIG := \
-  $(call get-executable-or-default,PYTHON_CONFIG,$(PYTHON_AUTO_CONFIG))
+# python[2][3]-config in weird combinations in the following order of
+# priority from lowest to highest:
+#   * python3-config
+#   * python-config
+#   * python2-config as per pep-0394.
+#   * $(PYTHON)-config (If PYTHON is user supplied but PYTHON_CONFIG isn't)
+#
+PYTHON_AUTO := python-config
+PYTHON_AUTO := $(if $(call get-executable,python3-config),python3-config,$(PYTHON_AUTO))
+PYTHON_AUTO := $(if $(call get-executable,python-config),python-config,$(PYTHON_AUTO))
+PYTHON_AUTO := $(if $(call get-executable,python2-config),python2-config,$(PYTHON_AUTO))
+
+# If PYTHON is defined but PYTHON_CONFIG isn't, then take $(PYTHON)-config as if it was the user
+# supplied value for PYTHON_CONFIG. Because it's "user supplied", error out if it doesn't exist.
+ifdef PYTHON
+  ifndef PYTHON_CONFIG
+    PYTHON_CONFIG_AUTO := $(call get-executable,$(PYTHON)-config)
+    PYTHON_CONFIG := $(if $(PYTHON_CONFIG_AUTO),$(PYTHON_CONFIG_AUTO),\
+                          $(call $(error $(PYTHON)-config not found)))
+  endif
+endif
+
+# Select either auto detected python and python-config or use user supplied values if they are
+# defined. get-executable-or-default fails with an error if the first argument is supplied but
+# doesn't exist.
+override PYTHON_CONFIG := $(call get-executable-or-default,PYTHON_CONFIG,$(PYTHON_AUTO))
+override PYTHON := $(call get-executable-or-default,PYTHON,$(subst -config,,$(PYTHON_AUTO)))
 
 grep-libs  = $(filter -l%,$(1))
 strip-libs  = $(filter-out -l%,$(1))
#3 lore raw

Re: [PATCH 1/1] perf tools: Use Python devtools for version autodetection rather than runtime

Quoted 20 lines · German Gomez

Hi James,

Quoted 11 lines · James Clark

This fixes the issue where the build will fail if only the Python2 runtime is installed but the Python3 devtools are installed. Currently the workaround is 'make PYTHON=python3'.

Fix it by autodetecting Python based on whether python[x]-config exists rather than just python[x] because both are needed for the build. Then -config is stripped to find the Python runtime.

Testing
=======

Quoted 6 lines

* Auto detect links with Python3 when the v3 devtools are installed and only Python 2 runtime is installed
* Auto detect links with Python2 when both devtools are installed
* Sensible warning is printed if no Python devtools are installed
* 'make PYTHON=x' still automatically sets PYTHON_CONFIG=x-config
* 'make PYTHON=x' fails if x-config doesn't exist

Quoted 3 lines

If x is a valid python but no x-config is found in the system, the build fails, instead of printing a warning like before. If we use this approach I think [1] in the Makefile is never hit and needs to be cleaned up.

Sorry, small correction:

The line in the Makefile is never hit when PYTHON is explicitly defined (probably not a big deal).

Still I wanted to point the small change in behaviour with the build warning vs. build fail.