Skip to contentSkip to conversation

perf python: Fix splitting CC into compiler and options

PATCH In review Arnaldo Carvalho de Melo 3 participants 5 messages Dec 22, 2022 perf events

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer
#0 lore raw

PATCH 1/1 perf python: Fix splitting CC into compiler and options

Just fyi, I'm carrying this in the perf tools tree.

- Arnaldo

----

Noticed this build failure on archlinux:base when building with clang:

clang-14: error: optimization flag '-ffat-lto-objects' is not supported [-Werror,-Wignored-optimization-argument]

In tools/perf/util/setup.py we check if clang supports that option, but since commit 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC") this got broken as in the common case where CC="clang":

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cc="clang"
print(cc.split()[0])

clang

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

option="-ffat-lto-objects"
print(str(cc.split()[1:]) + option)

[]-ffat-lto-objects

Quoted 1 line
Quoted 1 line
Quoted 1 line

And then the Popen will call clang with that bogus option name that in turn will not produce the b"unknown argument" or b"is not supported" that this function uses to detect if the option is not available and thus later on clang will be called with an unknown/unsupported option.

Fix it by looking if really there are options in the provided CC variable, and if so override 'cc' with the first token and append the options to the 'option' variable.

Fixes 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC")
Cc Adrian Hunter <[email protected]>
Cc Fangrui Song <[email protected]>
Cc Florian Fainelli <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Keeping <[email protected]>
Cc Khem Raj <[email protected]>
Cc Leo Yan <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Sedat Dilek <[email protected]>
---
 tools/perf/util/setup.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
tools/perf/util/setup.py +11 -2
index 4f265d0222c454e2..c294db713677c0c2 100644
@@ -3,11 +3,20 @@ from subprocess import Popen, PIPE
 from re import sub
 
 cc = getenv("CC")
-cc_is_clang = b"clang version" in Popen([cc.split()[0], "-v"], stderr=PIPE).stderr.readline()
+
+# Check if CC has options, as is the case in yocto, where it uses CC="cc --sysroot..."
+cc_tokens = cc.split()
+if len(cc_tokens) > 1:
+    cc = cc_tokens[0]
+    cc_options = " ".join([str(e) for e in cc_tokens[1:]]) + " "
+else:
+    cc_options = ""
+
+cc_is_clang = b"clang version" in Popen([cc, "-v"], stderr=PIPE).stderr.readline()
 src_feature_tests  = getenv('srctree') + '/tools/build/feature'
 
 def clang_has_option(option):
-    cc_output = Popen([cc.split()[0], str(cc.split()[1:]) + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
+    cc_output = Popen([cc, cc_options + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
     return [o for o in cc_output if ((b"unknown argument" in o) or (b"is not supported" in o))] == [ ]
 
 if cc_is_clang:
Signature · 1 line
2.38.1
#1 lore raw
Quoted 56 lines · Arnaldo Carvalho de Melo

Just fyi, I'm carrying this in the perf tools tree.

- Arnaldo

----

Noticed this build failure on archlinux:base when building with clang:

clang-14: error: optimization flag '-ffat-lto-objects' is not supported [-Werror,-Wignored-optimization-argument]

In tools/perf/util/setup.py we check if clang supports that option, but since commit 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC") this got broken as in the common case where CC="clang":

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cc="clang"
print(cc.split()[0])

clang

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

option="-ffat-lto-objects"
print(str(cc.split()[1:]) + option)

[]-ffat-lto-objects

Quoted 1 line
Quoted 1 line
Quoted 1 line

And then the Popen will call clang with that bogus option name that in turn will not produce the b"unknown argument" or b"is not supported" that this function uses to detect if the option is not available and thus later on clang will be called with an unknown/unsupported option.

Fix it by looking if really there are options in the provided CC variable, and if so override 'cc' with the first token and append the options to the 'option' variable.

Fixes 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC")
Cc Adrian Hunter <[email protected]>
Cc Fangrui Song <[email protected]>
Cc Florian Fainelli <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Keeping <[email protected]>
Cc Khem Raj <[email protected]>
Cc Leo Yan <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Sedat Dilek <[email protected]>
---
 tools/perf/util/setup.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
tools/perf/util/setup.py +0 -0
index 4f265d0222c454e2..c294db713677c0c2 100644

tools/perf/util/setup.py from subprocess import Popen, PIPE · line 10

Quoted 9 lines
tools/perf/util/setup.py +5 -1
@@ -3,11 +3,20 @@ from subprocess import Popen, PIPE
 from re import sub
 
 cc = getenv("CC")
-cc_is_clang = b"clang version" in Popen([cc.split()[0], "-v"], stderr=PIPE).stderr.readline()
+
+# Check if CC has options, as is the case in yocto, where it uses CC="cc --sysroot..."
+cc_tokens = cc.split()
+if len(cc_tokens) > 1:
+    cc = cc_tokens[0]

What if someone is using CC="cache clang"? Then cc is set to cache and the cc_is_clang check below will fail.

Quoted 16 lines
tools/perf/util/setup.py +6 -2
+    cc_options = " ".join([str(e) for e in cc_tokens[1:]]) + " "
+else:
+    cc_options = ""
+
+cc_is_clang = b"clang version" in Popen([cc, "-v"], stderr=PIPE).stderr.readline()
 src_feature_tests  = getenv('srctree') + '/tools/build/feature'
 
 def clang_has_option(option):
-    cc_output = Popen([cc.split()[0], str(cc.split()[1:]) + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
+    cc_output = Popen([cc, cc_options + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
     return [o for o in cc_output if ((b"unknown argument" in o) or (b"is not supported" in o))] == [ ]
 
 if cc_is_clang:
--

2.38.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
#2 lore raw
Quoted 61 lines · Nick Desaulniers
Quoted 58 lines · Arnaldo Carvalho de Melo

Just fyi, I'm carrying this in the perf tools tree.

- Arnaldo

----

Noticed this build failure on archlinux:base when building with clang:

clang-14: error: optimization flag '-ffat-lto-objects' is not supported [-Werror,-Wignored-optimization-argument]

In tools/perf/util/setup.py we check if clang supports that option, but since commit 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC") this got broken as in the common case where CC="clang":

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cc="clang"
print(cc.split()[0])

clang

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

option="-ffat-lto-objects"
print(str(cc.split()[1:]) + option)

[]-ffat-lto-objects

Quoted 1 line
Quoted 1 line
Quoted 1 line

And then the Popen will call clang with that bogus option name that in turn will not produce the b"unknown argument" or b"is not supported" that this function uses to detect if the option is not available and thus later on clang will be called with an unknown/unsupported option.

Fix it by looking if really there are options in the provided CC variable, and if so override 'cc' with the first token and append the options to the 'option' variable.

Fixes 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC")
Cc Adrian Hunter <[email protected]>
Cc Fangrui Song <[email protected]>
Cc Florian Fainelli <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Keeping <[email protected]>
Cc Khem Raj <[email protected]>
Cc Leo Yan <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Sedat Dilek <[email protected]>
---
 tools/perf/util/setup.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
tools/perf/util/setup.py +0 -0
index 4f265d0222c454e2..c294db713677c0c2 100644
@@ -3,11 +3,20 @@ from subprocess import Popen, PIPE
 from re import sub
 
Quoted 10 lines

tools/perf/util/setup.py from subprocess import Popen, PIPE · line 10

Quoted 7 lines
tools/perf/util/setup.py +5 -1
 cc = getenv("CC")
-cc_is_clang = b"clang version" in Popen([cc.split()[0], "-v"], stderr=PIPE).stderr.readline()
+
+# Check if CC has options, as is the case in yocto, where it uses CC="cc --sysroot..."
+cc_tokens = cc.split()
+if len(cc_tokens) > 1:
+    cc = cc_tokens[0]

What if someone is using CC="cache clang"? Then cc is set to cache and the cc_is_clang check below will fail.

s/cache/ccache/g

Quoted 22 lines
Quoted 15 lines
tools/perf/util/setup.py +6 -2
+    cc_options = " ".join([str(e) for e in cc_tokens[1:]]) + " "
+else:
+    cc_options = ""
+
+cc_is_clang = b"clang version" in Popen([cc, "-v"], stderr=PIPE).stderr.readline()
 src_feature_tests  = getenv('srctree') + '/tools/build/feature'
 
 def clang_has_option(option):
-    cc_output = Popen([cc.split()[0], str(cc.split()[1:]) + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
+    cc_output = Popen([cc, cc_options + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
     return [o for o in cc_output if ((b"unknown argument" in o) or (b"is not supported" in o))] == [ ]
 
 if cc_is_clang:
--
Quoted 2 lines

2.38.1

--
Thanks,
~Nick Desaulniers

Signature · 2 lines
Thanks,
~Nick Desaulniers
#3 lore raw
In reply to Nick Desaulniers
Quoted 60 lines · Nick Desaulniers
Quoted 58 lines · Arnaldo Carvalho de Melo

Just fyi, I'm carrying this in the perf tools tree.

- Arnaldo

----

Noticed this build failure on archlinux:base when building with clang:

clang-14: error: optimization flag '-ffat-lto-objects' is not supported [-Werror,-Wignored-optimization-argument]

In tools/perf/util/setup.py we check if clang supports that option, but since commit 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC") this got broken as in the common case where CC="clang":

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cc="clang"
print(cc.split()[0])

clang

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

option="-ffat-lto-objects"
print(str(cc.split()[1:]) + option)

[]-ffat-lto-objects

Quoted 1 line
Quoted 1 line
Quoted 1 line

And then the Popen will call clang with that bogus option name that in turn will not produce the b"unknown argument" or b"is not supported" that this function uses to detect if the option is not available and thus later on clang will be called with an unknown/unsupported option.

Fix it by looking if really there are options in the provided CC variable, and if so override 'cc' with the first token and append the options to the 'option' variable.

Fixes 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC")
Cc Adrian Hunter <[email protected]>
Cc Fangrui Song <[email protected]>
Cc Florian Fainelli <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Keeping <[email protected]>
Cc Khem Raj <[email protected]>
Cc Leo Yan <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Sedat Dilek <[email protected]>
---
 tools/perf/util/setup.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
tools/perf/util/setup.py +0 -0
index 4f265d0222c454e2..c294db713677c0c2 100644
@@ -3,11 +3,20 @@ from subprocess import Popen, PIPE
 from re import sub
 
Quoted 10 lines

tools/perf/util/setup.py from subprocess import Popen, PIPE · line 10

Quoted 7 lines
tools/perf/util/setup.py +5 -1
 cc = getenv("CC")
-cc_is_clang = b"clang version" in Popen([cc.split()[0], "-v"], stderr=PIPE).stderr.readline()
+
+# Check if CC has options, as is the case in yocto, where it uses CC="cc --sysroot..."
+cc_tokens = cc.split()
+if len(cc_tokens) > 1:
+    cc = cc_tokens[0]

What if someone is using CC="cache clang"? Then cc is set to cache and the cc_is_clang check below will fail.

Agreed, but this is a preexisting bug, let's fix this with a follow-up patch,

- Arnaldo

Quoted 19 lines
Quoted 17 lines
tools/perf/util/setup.py +5 -2
+    cc_options = " ".join([str(e) for e in cc_tokens[1:]]) + " "
+else:
+    cc_options = ""
 
+cc_is_clang = b"clang version" in Popen([cc, "-v"], stderr=PIPE).stderr.readline()
 src_feature_tests  = getenv('srctree') + '/tools/build/feature'
 
 def clang_has_option(option):
-    cc_output = Popen([cc.split()[0], str(cc.split()[1:]) + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
+    cc_output = Popen([cc, cc_options + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
     return [o for o in cc_output if ((b"unknown argument" in o) or (b"is not supported" in o))] == [ ]
 
 if cc_is_clang:
--

2.38.1

#4 lore raw
Quoted 75 lines · Arnaldo Carvalho de Melo
Quoted 60 lines · Nick Desaulniers
Quoted 58 lines · Arnaldo Carvalho de Melo

Just fyi, I'm carrying this in the perf tools tree.

- Arnaldo

----

Noticed this build failure on archlinux:base when building with clang:

clang-14: error: optimization flag '-ffat-lto-objects' is not supported [-Werror,-Wignored-optimization-argument]

In tools/perf/util/setup.py we check if clang supports that option, but since commit 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC") this got broken as in the common case where CC="clang":

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

cc="clang"
print(cc.split()[0])

clang

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

option="-ffat-lto-objects"
print(str(cc.split()[1:]) + option)

[]-ffat-lto-objects

Quoted 1 line
Quoted 1 line
Quoted 1 line

And then the Popen will call clang with that bogus option name that in turn will not produce the b"unknown argument" or b"is not supported" that this function uses to detect if the option is not available and thus later on clang will be called with an unknown/unsupported option.

Fix it by looking if really there are options in the provided CC variable, and if so override 'cc' with the first token and append the options to the 'option' variable.

Fixes 3cad53a6f9cdbafa ("perf python: Account for multiple words in CC")
Cc Adrian Hunter <[email protected]>
Cc Fangrui Song <[email protected]>
Cc Florian Fainelli <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Keeping <[email protected]>
Cc Khem Raj <[email protected]>
Cc Leo Yan <[email protected]>
Cc Michael Petlan <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Sedat Dilek <[email protected]>
---
 tools/perf/util/setup.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
tools/perf/util/setup.py +0 -0
index 4f265d0222c454e2..c294db713677c0c2 100644
@@ -3,11 +3,20 @@ from subprocess import Popen, PIPE
 from re import sub
 
Quoted 10 lines

tools/perf/util/setup.py from subprocess import Popen, PIPE · line 10

Quoted 7 lines
tools/perf/util/setup.py +5 -1
 cc = getenv("CC")
-cc_is_clang = b"clang version" in Popen([cc.split()[0], "-v"], stderr=PIPE).stderr.readline()
+
+# Check if CC has options, as is the case in yocto, where it uses CC="cc --sysroot..."
+cc_tokens = cc.split()
+if len(cc_tokens) > 1:
+    cc = cc_tokens[0]

What if someone is using CC="cache clang"? Then cc is set to cache and the cc_is_clang check below will fail.

Quoted 1 line

Agreed, but this is a preexisting bug, let's fix this with a follow-up patch,

Ack; it is.

Quoted 23 lines

- Arnaldo

Quoted 19 lines
Quoted 17 lines
tools/perf/util/setup.py +5 -2
+    cc_options = " ".join([str(e) for e in cc_tokens[1:]]) + " "
+else:
+    cc_options = ""
 
+cc_is_clang = b"clang version" in Popen([cc, "-v"], stderr=PIPE).stderr.readline()
 src_feature_tests  = getenv('srctree') + '/tools/build/feature'
 
 def clang_has_option(option):
-    cc_output = Popen([cc.split()[0], str(cc.split()[1:]) + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
+    cc_output = Popen([cc, cc_options + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
     return [o for o in cc_output if ((b"unknown argument" in o) or (b"is not supported" in o))] == [ ]
 
 if cc_is_clang:
--

2.38.1

Signature · 2 lines
Thanks,
~Nick Desaulniers