perf tools: Fix python extension build

PATCH In review Jiri Olsa 4 participants 5 messages Feb 27, 2016 – Mar 5, 2016 perf events

lore

Share & tools

lore

Jiri Olsa

author

2016-02-27 20:23 UTC #0 lore raw

PATCH 1/1 perf tools: Fix python extension build

Quoted 24 lines · Jiri Olsa

On Fri, Feb 26, 2016 at 05:49:02PM -0500, Josh Boyer wrote:
SNIP
Quoted 14 lines
Quoted 14 lines
./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o
Quoted 5 lines
Quoted 1 line
not sure we want to come up with some 'nicer' solution
I don't think anything under python_ext_build/tmp/ actually matters in any significant way. As long as this doesn't negatively impact something via side-effect, it's probably good enough.

Quoted 8 lines

right, and it's probably generic sollution too
Quoted 2 lines
Will you write up a full patch?
yep, will post it tomorrow
jirka

sending full patch

jirka

---
The util/python-ext-sources file contains sources files for python extension relative to $(srctree)/tools/perf,

Such a file path $(FILE).c is handed over to the python extension build system, which builts final object in $(PYTHON_EXTBUILD)/tmp/$(FILE).o path.

After build is done all files from $(PYTHON_EXTBUILD)lib/ are carried as the result binaries.

Above system fails when we add source file relative to ../lib, which we do for:

../lib/bitmap.c
../lib/find_bit.c
../lib/hweight.c
../lib/rbtree.c

All above objects will be built like:

  $(PYTHON_EXTBUILD)/tmp/../lib/bitmap.c
  $(PYTHON_EXTBUILD)/tmp/../lib/find_bit.c
  $(PYTHON_EXTBUILD)/tmp/../lib/hweight.c
  $(PYTHON_EXTBUILD)/tmp/../lib/rbtree.c

which accidentally happens to be final library path:
$(PYTHON_EXTBUILD)/lib/

Changing setup.py to pass full paths of source files to Extension build class and thus keep all built objects under $(PYTHON_EXTBUILD)tmp directory.

Cc David Ahern <[email protected]>

Cc Namhyung Kim <[email protected]>

Cc Peter Zijlstra <[email protected]>

Reported-by Josh Boyer <[email protected]>

Link http://lkml.kernel.org/n/[email protected] open here

Signed-off-by Jiri Olsa <[email protected]>

---
 tools/perf/util/setup.py | 4 ++++
 1 file changed, 4 insertions(+)

tools/perf/util/setup.py +4 -0

index 1833103768cb..c8680984d2d6 100644
@@ -22,6 +22,7 @@ cflags = getenv('CFLAGS', '').split()
 # switch off several checks (need to be at the end of cflags list)
 cflags += ['-fno-strict-aliasing', '-Wno-write-strings', '-Wno-unused-parameter' ]
 
+src_perf  = getenv('srctree') + '/tools/perf'
 build_lib = getenv('PYTHON_EXTBUILD_LIB')
 build_tmp = getenv('PYTHON_EXTBUILD_TMP')
 libtraceevent = getenv('LIBTRACEEVENT')

@@ -30,6 +31,9 @@ libapikfs = getenv('LIBAPI')
 ext_sources = [f.strip() for f in file('util/python-ext-sources')
 				if len(f.strip()) > 0 and f[0] != '#']
 
+# use full paths with source files
+ext_sources = map(lambda x: '%s/%s' % (src_perf, x) , ext_sources)
+
 perf = Extension('perf',
 		  sources = ext_sources,
 		  include_dirs = ['util/include'],

Signature · 1 line

2.4.3

Josh Boyer

2016-02-29 13:20 UTC #1 lore raw

Quoted 72 lines · Jiri Olsa

Quoted 24 lines · Jiri Olsa
On Fri, Feb 26, 2016 at 05:49:02PM -0500, Josh Boyer wrote:
SNIP
Quoted 14 lines
Quoted 14 lines
./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o
Quoted 5 lines
Quoted 1 line
not sure we want to come up with some 'nicer' solution
I don't think anything under python_ext_build/tmp/ actually matters in any significant way. As long as this doesn't negatively impact something via side-effect, it's probably good enough.
Quoted 8 lines
right, and it's probably generic sollution too
Quoted 2 lines
Will you write up a full patch?
yep, will post it tomorrow
jirka
sending full patch
jirka
---
The util/python-ext-sources file contains sources files for python extension relative to $(srctree)/tools/perf,
Such a file path $(FILE).c is handed over to the python extension build system, which builts final object in $(PYTHON_EXTBUILD)/tmp/$(FILE).o path.
After build is done all files from $(PYTHON_EXTBUILD)lib/ are carried as the result binaries.
Above system fails when we add source file relative to ../lib, which we do for:
../lib/bitmap.c
../lib/find_bit.c
../lib/hweight.c
../lib/rbtree.c
All above objects will be built like:
  $(PYTHON_EXTBUILD)/tmp/../lib/bitmap.c
  $(PYTHON_EXTBUILD)/tmp/../lib/find_bit.c
  $(PYTHON_EXTBUILD)/tmp/../lib/hweight.c
  $(PYTHON_EXTBUILD)/tmp/../lib/rbtree.c
which accidentally happens to be final library path:
$(PYTHON_EXTBUILD)/lib/
Changing setup.py to pass full paths of source files to Extension build class and thus keep all built objects under $(PYTHON_EXTBUILD)tmp directory.

Quoted 6 lines

Cc David Ahern <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Reported-by Josh Boyer <[email protected]>
Link http://lkml.kernel.org/n/[email protected] open here
Signed-off-by Jiri Olsa <[email protected]>

The Reported-by: tag should probably be changed to Jeff Bastian <[email protected]> if there's still time. He reported the original bug. Maybe also include this in stable for 4.2+?

I'll be putting this into Fedora Rawhide today. Thanks!

josh

Quoted 28 lines

---
 tools/perf/util/setup.py | 4 ++++
 1 file changed, 4 insertions(+)

tools/perf/util/setup.py +4 -1

index 1833103768cb..c8680984d2d6 100644
@@ -22,6 +22,7 @@ cflags = getenv('CFLAGS', '').split()
 # switch off several checks (need to be at the end of cflags list)
 cflags += ['-fno-strict-aliasing', '-Wno-write-strings', '-Wno-unused-parameter' ]
 
+src_perf  = getenv('srctree') + '/tools/perf'
 build_lib = getenv('PYTHON_EXTBUILD_LIB')
 build_tmp = getenv('PYTHON_EXTBUILD_TMP')
 libtraceevent = getenv('LIBTRACEEVENT')

@@ -30,6 +31,9 @@ libapikfs = getenv('LIBAPI')
 ext_sources = [f.strip() for f in file('util/python-ext-sources')
                                if len(f.strip()) > 0 and f[0] != '#']
 
+# use full paths with source files
+ext_sources = map(lambda x: '%s/%s' % (src_perf, x) , ext_sources)
+
 perf = Extension('perf',
                  sources = ext_sources,
                  include_dirs = ['util/include'],
--

2.4.3

Arnaldo Carvalho de Melo

maintainer

2016-02-29 14:08 UTC #2 lore raw

Em Mon, Feb 29, 2016 at 08:20:33AM -0500, Josh Boyer escreveu:

Quoted 80 lines · Jiri Olsa

Quoted 72 lines · Jiri Olsa
Quoted 24 lines · Josh Boyer
SNIP
Quoted 14 lines
Quoted 14 lines
./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o
Quoted 5 lines
Quoted 1 line
not sure we want to come up with some 'nicer' solution
I don't think anything under python_ext_build/tmp/ actually matters in any significant way. As long as this doesn't negatively impact something via side-effect, it's probably good enough.
Quoted 8 lines
right, and it's probably generic sollution too
Quoted 2 lines
Will you write up a full patch?
yep, will post it tomorrow
jirka
sending full patch
jirka
---
The util/python-ext-sources file contains sources files for python extension relative to $(srctree)/tools/perf,
Such a file path $(FILE).c is handed over to the python extension build system, which builts final object in $(PYTHON_EXTBUILD)/tmp/$(FILE).o path.
After build is done all files from $(PYTHON_EXTBUILD)lib/ are carried as the result binaries.
Above system fails when we add source file relative to ../lib, which we do for:
../lib/bitmap.c
../lib/find_bit.c
../lib/hweight.c
../lib/rbtree.c
All above objects will be built like:
  $(PYTHON_EXTBUILD)/tmp/../lib/bitmap.c
  $(PYTHON_EXTBUILD)/tmp/../lib/find_bit.c
  $(PYTHON_EXTBUILD)/tmp/../lib/hweight.c
  $(PYTHON_EXTBUILD)/tmp/../lib/rbtree.c
which accidentally happens to be final library path:
$(PYTHON_EXTBUILD)/lib/
Changing setup.py to pass full paths of source files to Extension build class and thus keep all built objects under $(PYTHON_EXTBUILD)tmp directory.
Quoted 6 lines
Cc David Ahern <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Reported-by Josh Boyer <[email protected]>
Link http://lkml.kernel.org/n/[email protected] open here
Signed-off-by Jiri Olsa <[email protected]>

Quoted 3 lines

The Reported-by: tag should probably be changed to Jeff Bastian <[email protected]> if there's still time. He reported the original bug. Maybe also include this in stable for 4.2+?

Sure, I'll use Jeff in the Reported-by: tag and have it CCed to stable,

I also will add a Tested-by: Josh Boyer, ok?

Thanks

Quoted 32 lines

I'll be putting this into Fedora Rawhide today. Thanks!

josh

Quoted 28 lines

---
 tools/perf/util/setup.py | 4 ++++
 1 file changed, 4 insertions(+)

tools/perf/util/setup.py +4 -1

index 1833103768cb..c8680984d2d6 100644
@@ -22,6 +22,7 @@ cflags = getenv('CFLAGS', '').split()
 # switch off several checks (need to be at the end of cflags list)
 cflags += ['-fno-strict-aliasing', '-Wno-write-strings', '-Wno-unused-parameter' ]
 
+src_perf  = getenv('srctree') + '/tools/perf'
 build_lib = getenv('PYTHON_EXTBUILD_LIB')
 build_tmp = getenv('PYTHON_EXTBUILD_TMP')
 libtraceevent = getenv('LIBTRACEEVENT')

@@ -30,6 +31,9 @@ libapikfs = getenv('LIBAPI')
 ext_sources = [f.strip() for f in file('util/python-ext-sources')
                                if len(f.strip()) > 0 and f[0] != '#']
 
+# use full paths with source files
+ext_sources = map(lambda x: '%s/%s' % (src_perf, x) , ext_sources)
+
 perf = Extension('perf',
                  sources = ext_sources,
                  include_dirs = ['util/include'],
--

2.4.3

Josh Boyer

2016-02-29 19:58 UTC #3 lore raw

Quoted 81 lines · Arnaldo Carvalho de Melo

Em Mon, Feb 29, 2016 at 08:20:33AM -0500, Josh Boyer escreveu:
Quoted 80 lines · Jiri Olsa
Quoted 78 lines · Jiri Olsa
Quoted 24 lines · Josh Boyer
SNIP
Quoted 14 lines
Quoted 14 lines
./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o
Quoted 5 lines
Quoted 1 line
not sure we want to come up with some 'nicer' solution
I don't think anything under python_ext_build/tmp/ actually matters in any significant way. As long as this doesn't negatively impact something via side-effect, it's probably good enough.
Quoted 8 lines
right, and it's probably generic sollution too
Quoted 2 lines
Will you write up a full patch?
yep, will post it tomorrow
jirka
sending full patch
jirka
---
The util/python-ext-sources file contains sources files for python extension relative to $(srctree)/tools/perf,
Such a file path $(FILE).c is handed over to the python extension build system, which builts final object in $(PYTHON_EXTBUILD)/tmp/$(FILE).o path.
After build is done all files from $(PYTHON_EXTBUILD)lib/ are carried as the result binaries.
Above system fails when we add source file relative to ../lib, which we do for:
../lib/bitmap.c
../lib/find_bit.c
../lib/hweight.c
../lib/rbtree.c
All above objects will be built like:
  $(PYTHON_EXTBUILD)/tmp/../lib/bitmap.c
  $(PYTHON_EXTBUILD)/tmp/../lib/find_bit.c
  $(PYTHON_EXTBUILD)/tmp/../lib/hweight.c
  $(PYTHON_EXTBUILD)/tmp/../lib/rbtree.c
which accidentally happens to be final library path:
$(PYTHON_EXTBUILD)/lib/
Changing setup.py to pass full paths of source files to Extension build class and thus keep all built objects under $(PYTHON_EXTBUILD)tmp directory.
Cc David Ahern <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Reported-by Josh Boyer <[email protected]>
Link http://lkml.kernel.org/n/[email protected] open here
Signed-off-by Jiri Olsa <[email protected]>

Quoted 7 lines

Quoted 3 lines
The Reported-by: tag should probably be changed to Jeff Bastian <[email protected]> if there's still time. He reported the original bug. Maybe also include this in stable for 4.2+?
Sure, I'll use Jeff in the Reported-by: tag and have it CCed to stable,
I also will add a Tested-by: Josh Boyer, ok?

Yes, that is fine. I tested the build and the resulting python-perf package only has the proper contents. I see you've added this to the formal patch, so thank you. Apologies for the delayed reply.

josh

tip-bot for Jiri Olsa

merge bot

2016-03-05 08:13 UTC #4 lore raw

In reply to Jiri Olsa

perf tools: Fix python extension build

Commit-ID:  67d5268908283c187e0a460048a423256c2fb288
Gitweb:     http://git.kernel.org/tip/67d5268908283c187e0a460048a423256c2fb288
Author:     Jiri Olsa <[email protected]>
AuthorDate: Sat, 27 Feb 2016 21:21:12 +0100
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Mon, 29 Feb 2016 11:18:25 -0300

perf tools: Fix python extension build

The util/python-ext-sources file contains source files required to build the python extension relative to $(srctree)/tools/perf,

Such a file path $(FILE).c is handed over to the python extension build system, which builds the final object in the $(PYTHON_EXTBUILD)/tmp/$(FILE).o path.

After the build is done all files from $(PYTHON_EXTBUILD)lib/ are carried as the result binaries.

Above system fails when we add source file relative to ../lib, which we do for:

../lib/bitmap.c
../lib/find_bit.c
../lib/hweight.c
../lib/rbtree.c

All above objects will be built like:

  $(PYTHON_EXTBUILD)/tmp/../lib/bitmap.c
  $(PYTHON_EXTBUILD)/tmp/../lib/find_bit.c
  $(PYTHON_EXTBUILD)/tmp/../lib/hweight.c
  $(PYTHON_EXTBUILD)/tmp/../lib/rbtree.c

which accidentally happens to be final library path:

  $(PYTHON_EXTBUILD)/lib/

Changing setup.py to pass full paths of source files to Extension build class and thus keep all built objects under $(PYTHON_EXTBUILD)tmp directory.

Reported-by Jeff Bastian <[email protected]>

Signed-off-by Jiri Olsa <[email protected]>

Tested-by Josh Boyer <[email protected]>

Cc David Ahern <[email protected]>

Cc Namhyung Kim <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc [email protected] # v4.2+

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Arnaldo Carvalho de Melo <[email protected]>

---
 tools/perf/util/setup.py | 4 ++++
 1 file changed, 4 insertions(+)

tools/perf/util/setup.py +4 -0

index 1833103..c868098 100644
@@ -22,6 +22,7 @@ cflags = getenv('CFLAGS', '').split()
 # switch off several checks (need to be at the end of cflags list)
 cflags += ['-fno-strict-aliasing', '-Wno-write-strings', '-Wno-unused-parameter' ]
 
+src_perf  = getenv('srctree') + '/tools/perf'
 build_lib = getenv('PYTHON_EXTBUILD_LIB')
 build_tmp = getenv('PYTHON_EXTBUILD_TMP')
 libtraceevent = getenv('LIBTRACEEVENT')

@@ -30,6 +31,9 @@ libapikfs = getenv('LIBAPI')
 ext_sources = [f.strip() for f in file('util/python-ext-sources')
 				if len(f.strip()) > 0 and f[0] != '#']
 
+# use full paths with source files
+ext_sources = map(lambda x: '%s/%s' % (src_perf, x) , ext_sources)
+
 perf = Extension('perf',
 		  sources = ext_sources,
 		  include_dirs = ['util/include'],