perf tools: Share map groups within process
cover perf tools: Share map groups within process
hi,
this patchset moves thread's map_groups to be dynamically allocated and shared within process threads.
The main benefit would be to be able to look up memory map from any thread that belongs to the process.
This implements one of the solution ideas for issue described by Don in following thread:
http://marc.info/?l=linux-kernel&m=139403876017159&w=2
RFC changes:
- added automated test for thread map groups get/put methods
- fix reference count for case described by Namhyung
- rename the mmap-events.c test to mmap-thread-lookup.c
- added PROT_EXEC to mmap call in tests/mmap-thread-lookup.c as it's not implied by default on all archs (Namhyung)
- lazy mg allocation in thread__find_addr_map (Namhyung)
- fix compilation failures (Arnaldo)
also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_mmaps
thanks,
jirka
---
Jiri Olsa (6):
perf tests: Add thread maps lookup automated tests perf tools: Remove thread__find_map function
perf tools: Allocate thread map_groups dynamically perf tools: Add machine pointer into thread struct perf tools: Share process map groups within process threads perf tests: Add map groups sharing with thread object test
tools/perf/Makefile.perf | 2 + tools/perf/arch/x86/tests/dwarf-unwind.c | 9 +++- tools/perf/perf.h | 6 +++ tools/perf/tests/builtin-test.c | 8 ++++ tools/perf/tests/mmap-thread-lookup.c | 233 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/tests/tests.h | 2 + tools/perf/tests/thread-mg-share.c | 110 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/ui/stdio/hist.c | 9 +++- tools/perf/util/event.c | 10 +++- tools/perf/util/machine.c | 8 ++-- tools/perf/util/map.h | 3 +- tools/perf/util/thread.c | 120 +++++++++++++++++++++++++++++++++++++++++----- tools/perf/util/thread.h | 19 ++++---- 13 files changed, 508 insertions(+), 31 deletions(-) create mode 100644 tools/perf/tests/mmap-thread-lookup.c create mode 100644 tools/perf/tests/thread-mg-share.c
PATCH 1/6 perf tests: Add thread maps lookup automated tests
Adding automated test for memory maps lookup within multiple machines threads.
The test creates 4 threads and separated memory maps.
It checks that we could use thread__find_addr_map
function with thread object based on TID to find
memory maps.
Currently map info is reachable only from thread leader. Upcoming patches for map groups sharing allow to find maps from any thread.
--- tools/perf/Makefile.perf | 1 + tools/perf/perf.h | 6 + tools/perf/tests/builtin-test.c | 4 + tools/perf/tests/mmap-thread-lookup.c | 233 ++++++++++++++++++++++++++++++++++ tools/perf/tests/tests.h | 1 + 5 files changed, 245 insertions(+) create mode 100644 tools/perf/tests/mmap-thread-lookup.c
tools/perf/Makefile.perf +1 -0
@@ -414,6 +414,7 @@ ifeq ($(ARCH),x86) LIB_OBJS += $(OUTPUT)tests/dwarf-unwind.o endif endif +LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/perf.h +6 -0
@@ -15,6 +15,9 @@#ifndef __NR_futex# define __NR_futex 240#endif +#ifndef __NR_gettid +# define __NR_gettid 224 +#endif #endif #if defined(__x86_64__)
@@ -29,6 +32,9 @@#ifndef __NR_futex# define __NR_futex 202#endif +#ifndef __NR_gettid +# define __NR_gettid 186 +#endif #endif #ifdef __powerpc__
tools/perf/tests/builtin-test.c +4 -0
@@ -124,6 +124,10 @@ static struct test {#endif #endif { + .desc = "Test mmap thread lookup", + .func = test__mmap_thread_lookup, + }, + { .func = NULL, }, };
tools/perf/tests/mmap-thread-lookup.c new file +233 -0
233 lines not on this page · raw message
tools/perf/tests/tests.h +1 -0
@@ -41,6 +41,7 @@ int test__sample_parsing(void);int test__keep_tracking (void );int test__parse_no_sample_id_all (void );int test__dwarf_unwind (void ); +int test__mmap_thread_lookup (void );#if defined(__x86_64__) || defined(__i386__)#ifdef HAVE_DWARF_UNWIND_SUPPORT
Signature · 1 line
1.8.3.1
PATCH 2/6 perf tools: Remove thread__find_map function
In reply to Jiri OlsaBecause it's not used any more.
--- tools/perf/util/thread.h | 6 ------ 1 file changed, 6 deletions(-)
tools/perf/util/thread.h +0 -6
@@ -44,12 +44,6 @@ void thread__insert_map(struct thread *thread, struct map *map);int thread__fork (struct thread *thread,struct thread *parent,u64 timestamp);size_t thread__fprintf (struct thread *thread,FILE *fp); -static inline struct map *thread__find_map (struct thread *thread, -enum map_type type,u64 addr) -{ -return thread ?map_groups__find (&thread->mg, type, addr) :NULL ; -} -void thread__find_addr_map (struct thread *thread,struct machine *machine,u8 cpumode,enum map_type type,u64 addr,struct addr_location *al);
Signature · 1 line
1.8.3.1
tip-bot for Jiri Olsa
merge bot[tip:perf/core] perf tools: Remove thread__find_map function
perf tools: Remove thread__find_map function
Commit-ID: 8fffdb6821e8ca249f56fba796ef463ac74ab196 Gitweb: http://git.kernel.org/tip/8fffdb6821e8ca249f56fba796ef463ac74ab196 Author: Jiri Olsa <[email protected]> AuthorDate: Tue, 18 Mar 2014 15:46:52 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Tue, 18 Mar 2014 18:17:04 -0300
perf tools: Remove thread__find_map function
Because it's not used any more.
--- tools/perf/util/thread.h | 6 ------ 1 file changed, 6 deletions(-)
tools/perf/util/thread.h +0 -6
@@ -44,12 +44,6 @@ void thread__insert_map(struct thread *thread, struct map *map);int thread__fork (struct thread *thread,struct thread *parent,u64 timestamp);size_t thread__fprintf (struct thread *thread,FILE *fp); -static inline struct map *thread__find_map (struct thread *thread, -enum map_type type,u64 addr) -{ -return thread ?map_groups__find (&thread->mg, type, addr) :NULL ; -} -void thread__find_addr_map (struct thread *thread,struct machine *machine,u8 cpumode,enum map_type type,u64 addr,struct addr_location *al);
PATCH 3/6 perf tools: Allocate thread map_groups dynamically
In reply to Jiri OlsaMoving towards sharing map groups within a process threads.
Because of this we need the map groups to be dynamically allocated. No other functional change is intended in here.
Adding new interface to get/put map groups:
thread__map_groups_get
- allocates map groups for thread
thread__map_groups_put
- frees map groups for thread
--- tools/perf/arch/x86/tests/dwarf-unwind.c | 9 +++- tools/perf/ui/stdio/hist.c | 9 +++- tools/perf/util/event.c | 10 +++- tools/perf/util/machine.c | 6 +-- tools/perf/util/map.h | 1 - tools/perf/util/thread.c | 86 +++++++++++++++++++++++++++----- tools/perf/util/thread.h | 8 ++- 7 files changed, 107 insertions(+), 22 deletions(-)
tools/perf/arch/x86/tests/dwarf-unwind.c +8 -1
@@ -23,7 +23,14 @@ static int sample_ustack(struct perf_sample *sample, sp = (unsigned long ) regs[PERF_REG_X86_SP ]; - map =map_groups__find (&thread->mg,MAP__FUNCTION , (u64 ) sp); +/* +* The generic code (test__dwarf_unwind) synthesizes and +* processes mmap events, so if we get NULL in thread->mg, +* it's bug. +*/ +BUG_ON (!thread->mg); + + map =map_groups__find (thread->mg,MAP__FUNCTION , (u64 ) sp);if (!map) {pr_debug ("failed to get stack map\n" );return -1 ;
tools/perf/ui/stdio/hist.c +8 -1
@@ -495,7 +495,14 @@ print_entries:break ;if (h->ms.map ==NULL && verbose >1 ) { -__map_groups__fprintf_maps (&h->thread->mg, +struct map_groups *mg; + + mg =thread__map_groups_get (h->thread); +if (!mg) { + ret = -ENOMEM ; +break ; + } +__map_groups__fprintf_maps (mg,MAP__FUNCTION , verbose, fp);fprintf (fp,"%.10s end\n" , graph_dotted_line); }
tools/perf/util/event.c +9 -1
@@ -699,7 +699,7 @@ void thread__find_addr_map(struct thread *thread, enum map_type type, u64 addr, struct addr_location *al) { - struct map_groups *mg = &thread->mg; + struct map_groups *mg;bool load_map = false; al->machine = machine;
@@ -719,12 +719,14 @@ void thread__find_addr_map(struct thread *thread, load_map = true; }else if (cpumode ==PERF_RECORD_MISC_USER && perf_host) { al->level ='.' ; + mg =thread__map_groups_get (thread); }else if (cpumode ==PERF_RECORD_MISC_GUEST_KERNEL && perf_guest) { al->level ='g' ; mg = &machine->kmaps; load_map = true; }else if (cpumode ==PERF_RECORD_MISC_GUEST_USER && perf_guest) { al->level ='u' ; + mg =thread__map_groups_get (thread); }else { al->level ='H' ; al->map =NULL ;
@@ -740,6 +742,12 @@ void thread__find_addr_map(struct thread *thread,return ; } + +if (mg ==NULL ) { + al->map =NULL ; +return ; + } + try_again: al->map =map_groups__find (mg, type, al->addr);if (al->map ==NULL ) {
tools/perf/util/machine.c +2 -4
@@ -1046,8 +1046,7 @@ int machine__process_mmap2_event(struct machine *machine,if (map ==NULL ) goto out_problem; -thread__insert_map (thread, map); -return 0 ; +return thread__insert_map (thread, map); out_problem:dump_printf ("problem processing PERF_RECORD_MMAP2, skipping event.\n" );
@@ -1093,8 +1092,7 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *eventif (map ==NULL ) goto out_problem; -thread__insert_map (thread, map); -return 0 ; +return thread__insert_map (thread, map); out_problem:dump_printf ("problem processing PERF_RECORD_MMAP, skipping event.\n" );
tools/perf/util/map.h +0 -1
@@ -202,5 +202,4 @@ struct map *map_groups__find_by_name(struct map_groups *mg, enum map_type type, const char *name);void map_groups__flush (struct map_groups *mg); -#endif /* __PERF_MAP_H */
tools/perf/util/thread.c +74 -12
124 lines not on this page · raw message
tools/perf/util/thread.h +6 -2
@@ -13,7 +13,7 @@ struct thread {struct rb_node rb_node;struct list_head node; }; -struct map_groups mg; +struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;pid_t ppid;
@@ -40,7 +40,7 @@ int thread__set_comm(struct thread *thread, const char *comm, u64 timestamp);int thread__comm_len (struct thread *thread);struct comm *thread__comm (const struct thread *thread);const char *thread__comm_str (const struct thread *thread); -void thread__insert_map (struct thread *thread,struct map *map); +int thread__insert_map (struct thread *thread,struct map *map);int thread__fork (struct thread *thread,struct thread *parent,u64 timestamp);size_t thread__fprintf (struct thread *thread,FILE *fp);
@@ -77,4 +77,8 @@ static inline bool thread__is_filtered(struct thread *thread) return false; } +struct map_groups *thread__map_groups_get (struct thread *thread); + +void thread__map_groups_put (struct thread *thread); +#endif /* __PERF_THREAD_H */
Signature · 1 line
1.8.3.1
PATCH 4/6 perf tools: Add machine pointer into thread struct
In reply to Jiri OlsaNeed machine pointer in thread object, so we could lookup the process thread in following patch.
--- tools/perf/util/machine.c | 2 +- tools/perf/util/thread.c | 4 +++- tools/perf/util/thread.h | 5 ++++- 3 files changed, 8 insertions(+), 3 deletions(-)
tools/perf/util/machine.c +1 -1
@@ -311,7 +311,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,if (!create)return NULL ; - th =thread__new (pid, tid); + th =thread__new (pid, tid, machine);if (th !=NULL ) {rb_link_node (&th->rb_node, parent, p);rb_insert_color (&th->rb_node, &machine->threads);
tools/perf/util/thread.c +3 -1
@@ -8,13 +8,15 @@#include "debug.h" #include "comm.h" -struct thread *thread__new (pid_t pid,pid_t tid) +struct thread *thread__new (pid_t pid,pid_t tid, +struct machine *machine) {char *comm_str;struct comm *comm;struct thread *thread =zalloc (sizeof (*thread));if (thread !=NULL ) { + thread->machine = machine; thread->pid_ = pid; thread->tid = tid; thread->ppid =-1 ;
tools/perf/util/thread.h +4 -1
@@ -7,12 +7,14 @@#include <sys/types.h> #include "symbol.h" #include <strlist.h> +#include "machine.h" struct thread {union {struct rb_node rb_node;struct list_head node; }; +struct machine *machine;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;
@@ -29,7 +31,8 @@ struct thread {struct machine ;struct comm ; -struct thread *thread__new (pid_t pid,pid_t tid); +struct thread *thread__new (pid_t pid,pid_t tid, +struct machine *machine);void thread__delete (struct thread *thread);static inline void thread__exited (struct thread *thread) {
Signature · 1 line
1.8.3.1
PATCH 5/6 perf tools: Share process map groups within process threads
In reply to Jiri OlsaSharing map groups within all process threads. This way there's only one copy of mmap info and it's reachable from any thread within the process.
Adding reference counting for struct map_groups and changing the thread map groups get/put interface.
Adding reference counter 'refcnt' into struct map_groups.
Function thread__map_groups_get now:
- lookup (&create) thread leader and returns its mg, allocates one if needed
- increments reference counter for each copy
Function thread__map_groups_put now:
- decrease map groups reference counter and sets its pointer to NULL
- frees map groups data if it was last copy
--- tools/perf/util/map.h | 2 ++ tools/perf/util/thread.c | 40 +++++++++++++++++++++++++++++++++++----- 2 files changed, 37 insertions(+), 5 deletions(-)
tools/perf/util/map.h +2 -0
@@ -59,6 +59,8 @@ struct map_groups {struct rb_root maps[MAP__NR_TYPES ];struct list_head removed_maps[MAP__NR_TYPES ];struct machine *machine; +/* Used for thread sharing. */ +int refcnt; };static inline struct kmap *map__kmap (struct map *map)
tools/perf/util/thread.c +35 -5
@@ -130,6 +130,13 @@ int thread__insert_map(struct thread *thread, struct map *map)return 0 ; } +static struct thread *thread__get_leader (struct thread *thread) +{ +pid_t pid = thread->pid_; + +return machine__findnew_thread (thread->machine, pid, pid); +} +static struct map_groups *thread__map_groups_alloc (struct thread *thread) {struct map_groups *mg =zalloc (sizeof (*mg));
@@ -137,6 +144,7 @@ static struct map_groups *thread__map_groups_alloc(struct thread *thread)if (mg) {map_groups__init (mg); thread->mg = mg; + mg->refcnt =1 ; }return mg;
@@ -146,17 +154,39 @@ struct map_groups *thread__map_groups_get(struct thread *thread) {struct map_groups *mg = thread->mg; -if (!mg) - mg =thread__map_groups_alloc (thread); +if (!mg) { +struct thread *leader =thread__get_leader (thread); + +if (!leader) +return NULL ; + +if (leader->mg) + mg = leader->mg; +else + mg =thread__map_groups_alloc (leader); + +if (leader != thread) { + thread->mg = mg; + mg->refcnt++; + } + }return mg; }void thread__map_groups_put (struct thread *thread) { -if (thread->mg) { -map_groups__exit (thread->mg); -zfree (&thread->mg); +struct map_groups *mg = thread->mg; + +if (mg) { +BUG_ON (!mg->refcnt); + +if (!--mg->refcnt) { +map_groups__exit (thread->mg); +free (thread->mg); + } + + thread->mg =NULL ; } }
Signature · 1 line
1.8.3.1
PATCH 6/6 perf tests: Add map groups sharing with thread object test
In reply to Jiri OlsaThis test create 2 processes abstractions, with several threads and checks they properly share and maintain map groups info.
--- tools/perf/Makefile.perf | 1 + tools/perf/tests/builtin-test.c | 4 ++ tools/perf/tests/tests.h | 1 + tools/perf/tests/thread-mg-share.c | 110 +++++++++++++++++++++++++++++++++++++ 4 files changed, 116 insertions(+) create mode 100644 tools/perf/tests/thread-mg-share.c
tools/perf/Makefile.perf +1 -0
@@ -415,6 +415,7 @@ LIB_OBJS += $(OUTPUT)tests/dwarf-unwind.o endif endif LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o +LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
@@ -128,6 +128,10 @@ static struct test { .func = test__mmap_thread_lookup, }, { + .desc = "Test thread mg sharing", + .func = test__thread_mg_share, + }, + { .func = NULL, }, };
tools/perf/tests/tests.h +1 -0
@@ -42,6 +42,7 @@ int test__keep_tracking(void);int test__parse_no_sample_id_all (void );int test__dwarf_unwind (void );int test__mmap_thread_lookup (void ); +int test__thread_mg_share (void );#if defined(__x86_64__) || defined(__i386__)#ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-mg-share.c new file +110 -0
110 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Quoted 20 lines · Jiri Olsa
hi,
this patchset moves thread'smap_groupsto be dynamically allocated and shared within process threads.The main benefit would be to be able to look up memory map from any thread that belongs to the process.
This implements one of the solution ideas for issue described by Don in following thread:
http://marc.info/?l=linux-kernel&m=139403876017159&w=2RFC changes:
- added automated test for thread map groups get/put methods
- fix reference count for case described by Namhyung
- rename themmap-events.ctest tommap-thread-lookup.c
- addedPROT_EXECto mmap call intests/mmap-thread-lookup.cas it's not implied by default on all archs (Namhyung)
- lazy mg allocation inthread__find_addr_map(Namhyung)
- fix compilation failures (Arnaldo)
Quoted 3 lines
also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_mmaps
I ran this patchset through my multi-threaded java test (SPECJBB) and saw no issues. The threads seemed to have matched with their pids properly. I even took an old perf.data file (with old synthesized mmap2 events using pids) and the tids from the samples matched the pids correctly (for what it is worth).
So I will add my
I don't know the code well enough to know if it is right or not, but my quick review didn't see anything wrong.
Cheers,
Don
Quoted 40 lines
thanks,
jirkaSigned-off-by Jiri Olsa <[email protected]>Cc Don Zickus <[email protected]>Cc Corey Ashford <[email protected]>Cc David Ahern <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Ingo Molnar <[email protected]>Cc Namhyung Kim <[email protected]>Cc Paul Mackerras <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>---
Jiri Olsa (6):
perf tests: Add thread maps lookup automated tests perf tools: Removethread__find_mapfunction
perf tools: Allocate threadmap_groupsdynamically perf tools: Add machine pointer into thread struct perf tools: Share process map groups within process threads perf tests: Add map groups sharing with thread object testtools/perf/Makefile.perf | 2 + tools/perf/arch/x86/tests/dwarf-unwind.c | 9 +++- tools/perf/perf.h | 6 +++ tools/perf/tests/builtin-test.c | 8 ++++ tools/perf/tests/mmap-thread-lookup.c | 233 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/tests/tests.h | 2 + tools/perf/tests/thread-mg-share.c | 110 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/ui/stdio/hist.c | 9 +++- tools/perf/util/event.c | 10 +++- tools/perf/util/machine.c | 8 ++-- tools/perf/util/map.h | 3 +- tools/perf/util/thread.c | 120 +++++++++++++++++++++++++++++++++++++++++----- tools/perf/util/thread.h | 19 ++++---- 13 files changed, 508 insertions(+), 31 deletions(-) create mode 100644 tools/perf/tests/mmap-thread-lookup.c create mode 100644 tools/perf/tests/thread-mg-share.c
Hi Jiri,
Quoted 11 lines · Jiri Olsa
hi,
this patchset moves thread'smap_groupsto be dynamically allocated and shared within process threads.The main benefit would be to be able to look up memory map from any thread that belongs to the process.
This implements one of the solution ideas for issue described by Don in following thread:
http://marc.info/?l=linux-kernel&m=139403876017159&w=2
Quoted 8 lines
RFC changes:
- added automated test for thread map groups get/put methods
- fix reference count for case described by Namhyung
- rename themmap-events.ctest tommap-thread-lookup.c
- addedPROT_EXECto mmap call intests/mmap-thread-lookup.cas it's not implied by default on all archs (Namhyung)
- lazy mg allocation inthread__find_addr_map(Namhyung)
- fix compilation failures (Arnaldo)
Looks good to me!
Thanks,
Namhyung
Quoted 20 lines · Jiri Olsa
hi,
this patchset moves thread'smap_groupsto be dynamically allocated and shared within process threads.The main benefit would be to be able to look up memory map from any thread that belongs to the process.
This implements one of the solution ideas for issue described by Don in following thread:
http://marc.info/?l=linux-kernel&m=139403876017159&w=2RFC changes:
- added automated test for thread map groups get/put methods
- fix reference count for case described by Namhyung
- rename themmap-events.ctest tommap-thread-lookup.c
- addedPROT_EXECto mmap call intests/mmap-thread-lookup.cas it's not implied by default on all archs (Namhyung)
- lazy mg allocation inthread__find_addr_map(Namhyung)
- fix compilation failures (Arnaldo)
Quoted 3 lines
also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_mmaps
Arnaldo,
I rebased this to your latest perf/core
thanks,
jirka