perf: Fallback to JIT support for mmap'd non-ELF binaries.
PATCH 1/1 perf: Fallback to JIT support for mmap'd non-ELF binaries.
Programs can execute code contained in files that don't conform to the ELF standard. This is common for cross-platform runtimes that support ahead-of-time compilation. In this case, perf is unable to resolve addresses to symbolic names, as the format of the mmap'd file is unknown to perf.
Make perf fallback to JIT support (/tmp/perf-%pid.map) when resolving function names if the file containing the code is not an ELF binary.
--- tools/perf/util/map.c | 1 + tools/perf/util/map.h | 1 + tools/perf/util/symbol.c | 33 +++++++++++++++++++++++++++++++-- 3 files changed, 33 insertions(+), 2 deletions(-)
tools/perf/util/map.c +1 -0
@@ -164,6 +164,7 @@ struct map *map__new(struct machine *machine, u64 start, u64 len, map->ino_generation = ino_gen; map->prot = prot; map->flags = flags; + map->pid = pid;if ((anon || no_dso) && type ==MAP__FUNCTION ) {snprintf (newfilename,sizeof (newfilename),"/tmp/perf-%d.map" , pid);
tools/perf/util/map.h +1 -0
@@ -52,6 +52,7 @@ struct map {struct dso *dso;struct map_groups *groups;atomic_t refcnt; +u32 pid; };struct kmap {
tools/perf/util/symbol.c +31 -2
@@ -1516,7 +1516,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter) }if (!runtime_ss && !syms_ss) - goto out_free; + goto jit_fallback;if (runtime_ss && !syms_ss) { syms_ss = runtime_ss;
@@ -1541,7 +1541,36 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)for (; ss_pos >0 ; ss_pos--)symsrc__destroy (&ss_[ss_pos -1 ]); -out_free: + +jit_fallback: +/* If we couldn't load symbols, fall back to /tmp/perf-%pid.map. */ +if (ret <0 ) { +struct stat st; +const char *origlongname; +snprintf (name,PATH_MAX ,"/tmp/perf-%d.map" , map->pid); + +if (lstat (name, &st) <0 ) + goto out; + +if (st.st_uid && (st.st_uid !=geteuid ())) { +pr_warning ("File %s not owned by current user or root, " +"ignoring it.\n" , name); + goto out; + } + + origlongname = dso->long_name; + dso->long_name = name; + + ret =dso__load_perf_map (dso, map, filter); + + dso->long_name = origlongname; + dso->symtab_type = ret >0 ?DSO_BINARY_TYPE__JAVA_JIT : +DSO_BINARY_TYPE__NOT_FOUND ; + +if (ret >0 ) + map->map_ip = map->unmap_ip = identity__map_ip; + } +free (name);if (ret <0 &&strstr (dso->name," (deleted)" ) !=NULL ) ret =0 ;
Signature · 1 line
1.9.1
Peter Zijlstra
maintainerQuoted 5 lines · Brian Robbins
Programs can execute code contained in files that don't conform to the ELF standard. This is common for cross-platform runtimes that support ahead-of-time compilation. In this case, perf is unable to resolve addresses to symbolic names, as the format of the mmap'd file is unknown to perf.
I'm assuming you're using PECOFF? Why not teach perf to read those?
Surely there's a library that can parse that stuff, it seems to me Wine for one needs fairly decent support for this, also GDB (and other toolchain bits) on Windows targets.
Its just that I was hoping we could move towards getting rid of those /tmp files now that we have much better JIT support.
From: Peter Zijlstra <[email protected]>
Sent: Thursday, November 19, 2015 11:31 AM
To: Brian Robbins
Cc: Ingo Molnar; Arnaldo Carvalho de Melo; [email protected]; Stephane Eranian
Subject: Re: [PATCH] perf: Fallback to JIT support for mmap'd non-ELF binaries.
Quoted 7 lines · Brian Robbins
Quoted 5 lines
Programs can execute code contained in files that don't conform to the ELF standard. This is common for cross-platform runtimes that support ahead-of-time compilation. In this case, perf is unable to resolve addresses to symbolic names, as the format of the mmap'd file is unknown to perf.
Quoted 8 lines
I'm assuming you're using PECOFF? Why not teach perf to read those?
Surely there's a library that can parse that stuff, it seems to me Wine for one needs fairly decent support for this, also GDB (and other toolchain bits) on Windows targets.
Its just that I was hoping we could move towards getting rid of those /tmp files now that we have much better JIT support.
Hi Peter,
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Thanks.
-Brian
Quoted 20 lines · Brian Robbins
From: Peter Zijlstra <[email protected]>
Sent: Thursday, November 19, 2015 11:31 AM
To: Brian Robbins
Cc: Ingo Molnar; Arnaldo Carvalho de Melo; [email protected]; Stephane Eranian
Subject: Re: [PATCH] perf: Fallback to JIT support for mmap'd non-ELF binaries.Quoted 12 lines · Brian Robbins
Quoted 5 lines
Programs can execute code contained in files that don't conform to the ELF standard. This is common for cross-platform runtimes that support ahead-of-time compilation. In this case, perf is unable to resolve addresses to symbolic names, as the format of the mmap'd file is unknown to perf.
I'm assuming you're using PECOFF? Why not teach perf to read those?
Surely there's a library that can parse that stuff, it seems to me Wine for one needs fairly decent support for this, also GDB (and other toolchain bits) on Windows targets.
Quoted 9 lines
Quoted 2 lines
Its just that I was hoping we could move towards getting rid of those /tmp files now that we have much better JIT support.
Hi Peter,
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
You need 4.1 if you have a situation where your JIT may recompile code multiple times. The key difference is that
with 4.1 you can use the same timestamp clock source for both the JIT runtime and the kernel perf_event subsystem.
That makes it possible to correlate samples from the JIT with samples from the hardware. Without that, you may not
get correct symbolization if the code is re-jitted or moved.
Quoted 5 lines
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Thanks.
-Brian
Quoted 30 lines · Stephane Eranian
Quoted 20 lines · Brian Robbins
From: Peter Zijlstra <[email protected]>
Sent: Thursday, November 19, 2015 11:31 AM
To: Brian Robbins
Cc: Ingo Molnar; Arnaldo Carvalho de Melo; [email protected]; Stephane Eranian
Subject: Re: [PATCH] perf: Fallback to JIT support for mmap'd non-ELF binaries.Quoted 12 lines · Brian Robbins
Quoted 5 lines
Programs can execute code contained in files that don't conform to the ELF standard. This is common for cross-platform runtimes that support ahead-of-time compilation. In this case, perf is unable to resolve addresses to symbolic names, as the format of the mmap'd file is unknown to perf.
I'm assuming you're using PECOFF? Why not teach perf to read those?
Surely there's a library that can parse that stuff, it seems to me Wine for one needs fairly decent support for this, also GDB (and other toolchain bits) on Windows targets.
Quoted 9 lines
Quoted 2 lines
Its just that I was hoping we could move towards getting rid of those /tmp files now that we have much better JIT support.
Hi Peter,
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
Quoted 8 lines
You need 4.1 if you have a situation where your JIT may recompile code multiple times. The key difference is that
with 4.1 you can use the same timestamp clock source for both the JIT runtime and the kernelperf_eventsubsystem.
That makes it possible to correlate samples from the JIT with samples from the hardware. Without that, you may not
get correct symbolization if the code is re-jitted or moved.
Thanks Stephane. That's good to know.
Unfortunately, because the scenario here is not actually a JIT, it requires a bit more control around when the data is generated. Specifically, the requirement is that the data be generated offline as opposed to when the process is running. Rather than using virtual addresses, we use offsets from the beginning of the file, and then update the data adding in the base address of the loaded file to get the full virtual address after the data has been collected.
Thus, the scenario is a bit different than a JIT scenario, but can re-use some of the plumbing. Knowing this information, does this scenario sound OK?
I'd be happy to look at seeing what it would take to make this a bit cleaner (and deprecate the tmp file), but am keen to enable the scenario sooner rather than later.
Thanks again for your consideration.
-Brian
Quoted 6 lines
Quoted 5 lines
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Thanks.
-Brian
Peter Zijlstra
maintainerQuoted 3 lines · Brian Robbins
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
*groan* you just had to go and invent yet another executable format, right? :-)
Quoted 4 lines
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
As I think Stephane explained, this is only required if you need to match up kernel and userspace timestamps, which is important for dynamic code generation, less so for static code in a weird format.
So what the new JIT stuff does is online write 'fake' ELF files with symbol sections and (optionally?) dwarf debug info for line numbers.
Since you don't dynamically generate code, you can offline generate these ELF files and redirect the symbol parser bits to that (we already look for debug ELF files in various locations), or...
Quoted 4 lines
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Since, someone somewhere needs to go parse this funny new file format anyhow to either generate /tmp files or fake ELF files or whatever, you might as well put that decoder in perf?
Or just ship these fake ELF files in /usr/lib/debug/ or whatever the 'right' location for the distro at hand is.
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Brian Robbins
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
*groan* you just had to go and invent yet another executable format, right? :-)
Nah, it just has some extra stuff in it that makes this less desirable - it's technically PE, but anyway.
Quoted 9 lines
Quoted 5 lines
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
As I think Stephane explained, this is only required if you need to match up kernel and userspace timestamps, which is important for dynamic code generation, less so for static code in a weird format.
Yes, agreed.
Quoted 13 lines
So what the new JIT stuff does is online write 'fake' ELF files with symbol sections and (optionally?) dwarf debug info for line numbers.
Since you don't dynamically generate code, you can offline generate these ELF files and redirect the symbol parser bits to that (we already look for debug ELF files in various locations), or...
Quoted 4 lines
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Quoted 7 lines
Since, someone somewhere needs to go parse this funny new file format anyhow to either generate /tmp files or fake ELF files or whatever, you might as well put that decoder in perf?
Or just ship these fake ELF files in
/usr/lib/debug/or whatever the 'right' location for the distro at hand is.
This seems like a reasonable approach.
Stephane, are your changes available for public consumption? Last I recall, the patches were still in review.
Thanks.
-Brian
Arnaldo Carvalho de Melo
maintainerEm Fri, Nov 20, 2015 at 01:18:55PM -0800, Brian Robbins escreveu:
Quoted 43 lines · Peter Zijlstra
Quoted 9 lines · Brian Robbins
Quoted 4 lines
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
*groan* you just had to go and invent yet another executable format, right? :-)
Nah, it just has some extra stuff in it that makes this less desirable - it's technically PE, but anyway.
Quoted 9 lines
Quoted 5 lines
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
As I think Stephane explained, this is only required if you need to match up kernel and userspace timestamps, which is important for dynamic code generation, less so for static code in a weird format.
Yes, agreed.
Quoted 17 lines
So what the new JIT stuff does is online write 'fake' ELF files with symbol sections and (optionally?) dwarf debug info for line numbers.
Since you don't dynamically generate code, you can offline generate these ELF files and redirect the symbol parser bits to that (we already look for debug ELF files in various locations), or...
Quoted 4 lines
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Since, someone somewhere needs to go parse this funny new file format anyhow to either generate /tmp files or fake ELF files or whatever, you might as well put that decoder in perf?
Quoted 7 lines
Quoted 2 lines
Or just ship these fake ELF files in
/usr/lib/debug/or whatever the 'right' location for the distro at hand is.This seems like a reasonable approach.
Stephane, are your changes available for public consumption? Last I recall, the patches were still in review.
Its in my queue, AFAIK there were not objections expressed, I'll try to get this reviewed/merged soon.
- Arnaldo
Quoted 20 lines · Brian Robbins
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Brian Robbins
Thank you for the feedback. The file format is similar to PE, but is not identical. So, we would be implementing something very scoped, which doesn't feel right to me.
*groan* you just had to go and invent yet another executable format, right? :-)
Nah, it just has some extra stuff in it that makes this less desirable - it's technically PE, but anyway.
Quoted 6 lines
Quoted 5 lines
I am interested in the new JIT support, however my understanding from the information that I've read is that it requires kernel support in 4.x, though I can't seem to find where I read that. I want to make sure that this works on older kernels (3.x) as well.
Quoted 10 lines
Quoted 3 lines
As I think Stephane explained, this is only required if you need to match up kernel and userspace timestamps, which is important for dynamic code generation, less so for static code in a weird format.
Yes, agreed.
Quoted 4 lines
So what the new JIT stuff does is online write 'fake' ELF files with symbol sections and (optionally?) dwarf debug info for line numbers.
The debug info section is optional and contains the line table info. This can be used should you want to display source code along with assembly.
Quoted 9 lines
Quoted 9 lines
Since you don't dynamically generate code, you can offline generate these ELF files and redirect the symbol parser bits to that (we already look for debug ELF files in various locations), or...
Quoted 4 lines
The reason I went with this approach is because it is simple for runtimes to implement and has no requirement that perf understand the file format. I am open to feedback if there is a preferred solution that would still work for older kernels as well.
Quoted 4 lines
Quoted 4 lines
Since, someone somewhere needs to go parse this funny new file format anyhow to either generate /tmp files or fake ELF files or whatever, you might as well put that decoder in perf?
If you have a new binary format, I think you'd have to write the parser in perf.
Quoted 9 lines
Quoted 3 lines
Or just ship these fake ELF files in
/usr/lib/debug/or whatever the 'right' location for the distro at hand is.This seems like a reasonable approach.
Stephane, are your changes available for public consumption? Last I recall, the patches were still in review.
I will post V8 this week.