Question: perf dso support for /proc/kallsyms
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
map__load(al.map);
dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
`-> data_read_offset()I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data, ssize_t size)
{
if (data_file_size(dso, machine))
return -1;
/* Check the offset sanity. */
if (offset > dso->data.file_size)
return -1;
if (offset + size < offset)
return -1;
return cached_read(dso, machine, offset, data, size);
}Thanks,
Leo Yan
Quoted 2 lines
Now I still don't understand how the dso/map support "
/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.
The way this is supported is that at record time, pseudo mmap records are created for the kernel. But depending on permissions these might not get created. This isn't just an ETM issue, it can happen on Intel too.
What do you see in "perf report -D", do you see a PERF_RECORD_MMAP record for "[kernel.kallsyms]_text" and possibly some others for loadable kernel modules in /lib/modules?
Does it all work if you run perf record as sudo? Or if you do
sudo sysctl kernel.kptr_restrict=0
before you run perf record?
Al
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hi Al,
Quoted 6 lines · Al Grant
Quoted 2 lines
Now I still don't understand how the dso/map support "
/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.The way this is supported is that at record time, pseudo mmap records are created for the kernel. But depending on permissions these might not get created. This isn't just an ETM issue, it can happen on Intel too.
Agree, I also think this is not a specific issue only for Arm platform, this should be one common issue for how to parse kernel symbols with kallsyms file.
Quoted 3 lines
What do you see in "perf report -D", do you see a
PERF_RECORD_MMAPrecord for "[kernel.kallsyms]_text" and possibly some others for loadable kernel modules in/lib/modules?
Yes, I can see PERF_RECORD_MMAP for "[kernel.kallsyms]_text".
0x350 [0x50]: event: 1
.
. ... raw event: size 80 bytes
. 0000: 01 00 00 00 01 00 50 00 ff ff ff ff 00 00 00 00 ......P......... . 0010: 00 00 08 08 00 00 ff ff ff ff f7 f7 ff ff 00 00 ................ . 0020: 00 00 08 08 00 00 ff ff 5b 6b 65 72 6e 65 6c 2e ........[kernel. . 0030: 6b 61 6c 6c 73 79 6d 73 5d 5f 74 65 78 74 00 00 kallsyms]_text.. . 0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x350 [0x50]: PERF_RECORD_MMAP -1/0: [0xffff000008080000(0xfffff7f7ffff) @ 0xffff000008080000]: x [kernel.kallsyms]_text
Below is the infor for checking buildid:
root@debian:~/coresight_test# perf buildid-list 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms] b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-2.27.so 8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso] 25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-gnu/libc-2.27.so root@debian:~/coresight_test# ls ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/ kallsyms
Quoted 5 lines
Does it all work if you run perf record as sudo? Or if you do
sudo sysctl kernel.kptr_restrict=0
before you run perf record?
Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user. I suspected the pointer permission issue so checked with below command:
root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict 0
Thanks,
Leo Yan
Quoted 5 lines
root@debian:~/coresight_test# perf buildid-list 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms] b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld- 2.27.so
Quoted 8 lines
8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso] 25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux- gnu/libc-2.27.so root@debian:~/coresight_test# ls ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/ kallsyms
What's in that last file? I've seen it happen that the copy of kallsyms in ~/.debug has the symbol addresses as zeroes - possibly because it was created when you didn't have permissions. That's really a bug in perf, as cacheing a copy of this file with the addresses zeroed out is kind of pointless. Again, this happens on Intel too.
Then, you can give yourself permissions - but perf's already cached the file and won't update it!
If you delete it, and then rerun perf record (either as sudo or now that you've got kptr_restrict=0) you should see it reappear, with correct kernel addresses.
Perhaps nobody spotted this on Intel because perf report goes directly to /proc/kallsyms. But it would be an issue if you ran a perf report on a perf.data from an older kernel and it had to go to ~/.debug. At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would mean you couldn't symbolicate any addresses.
Al
Quoted 7 lines
Quoted 6 lines
Does it all work if you run perf record as sudo? Or if you do
sudo sysctl kernel.kptr_restrict=0
before you run perf record?
Quoted 8 lines
Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user. I suspected the pointer permission issue so checked with below command:
root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict 0Thanks,
Leo Yan
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 14 lines · Al Grant
Quoted 5 lines
root@debian:~/coresight_test# perf buildid-list 0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms] b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main 26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld- 2.27.soQuoted 8 lines
8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso] 25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so 70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux- gnu/libc-2.27.so root@debian:~/coresight_test# ls ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/ kallsyms
Quoted 1 line
What's in that last file?
I can see the correct symbol infos in this file:
root@debian:~/coresight_test# cat ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/kallsyms | head -20 ffff000008080000 t _head ffff000008080000 T _text ffff000008080040 t pe_header ffff000008080044 t coff_header ffff000008080058 t optional_header ffff000008080070 t extra_header_fields ffff0000080800f8 t section_table ffff000008081000 T do_undefinstr ffff000008081000 t efi_header_end ffff000008081000 T _stext ffff000008081000 T __exception_text_start ffff000008081268 T do_cp15instr ffff000008081380 T do_sysinstr ffff000008081410 T do_debug_exception ffff000008081550 T do_mem_abort ffff000008081600 T do_el0_irq_bp_hardening ffff000008081650 T do_el0_ia_bp_hardening ffff0000080816f8 T do_sp_pc_abort ffff0000080817c4 T __exception_text_end ffff0000080817c8 t bcm2835_handle_irq
Quoted 13 lines
I've seen it happen that the copy of kallsyms
in ~/.debug has the symbol addresses as zeroes - possibly because it was created when you didn't have permissions. That's really a bug in perf, as cacheing a copy of this file with the addresses zeroed out is kind of pointless. Again, this happens on Intel too.Then, you can give yourself permissions - but perf's already cached the file and won't update it!
If you delete it, and then rerun perf record (either as sudo or now that you've got kptr_restrict=0) you should see it reappear, with correct kernel addresses.
Quoted 5 lines
Perhaps nobody spotted this on Intel because perf report goes directly to
/proc/kallsyms. But it would be an issue if you ran a perf report on a perf.data from an older kernel and it had to go to ~/.debug. At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would mean you couldn't symbolicate any addresses.
This is another issue related with permission for accessing kernel pointer values and this will bother much for cross platforms usage (e.g. 'perf record' on Arm platform and 'perf script' on x86 platform).
At my side, I executed all commands on Arm Juno board and simply to say the kallsyms support is broken in perf.
Thanks,
Leo Yan
Hi Leo,
My understanding is that when we decode CoreSight trace - be it on the system that generated it, or off target device on a separate host system, we are using the dso files in .debug/ as these represent the memory layout at the time trace was recorded.
If I look into a recent session copied up to my linux box from DB410, is see ~/.debug/\[kernel.kallsyms\]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
- which seems to me to be the relevant symbol file to use rather than the live one generated by /proc/kallsyms. I don't really want to use the local /proc/kallsyms on my x86 linux box when decoding an ARM
trace captured elsewhere.
So perhaps the problem to be solved is not how to use /proc/kallsyms if no vmlinux is supplied to the script, but ensure that the [kernel.kallsyms] is used?
Regards
Mike
Quoted 49 lines · On Fri, 2 Nov 2018 at 02:55, <[email protected]> wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use/proc/kallsymsfor kernel symbols parsing, as result it will run into below flow:thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()I can observe the function
data_read_offset()returns failure, this is caused by checking the offset sanity "if (offset >dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.Now I still don't understand how the dso/map support "
/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static
ssize_tdata_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data,ssize_tsize){ if (data_file_size(dso, machine)) return -1; /* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size); }Thanks,
Leo Yan
_______________________________________________
CoreSight mailing list
[email protected]
https://lists.linaro.org/mailman/listinfo/coresight
Signature · 3 lines
Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK
Hi Mike,
Quoted 14 lines · Mike Leach
Hi Leo,
My understanding is that when we decode CoreSight trace - be it on the system that generated it, or off target device on a separate host system, we are using the dso files in .debug/ as these represent the memory layout at the time trace was recorded.
If I look into a recent session copied up to my linux box from DB410, is see ~/.debug/\[kernel.kallsyms\]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
- which seems to me to be the relevant symbol file to use rather than the live one generated by/proc/kallsyms. I don't really want to use the local/proc/kallsymson my x86 linux box when decoding an ARM
trace captured elsewhere.
Quoted 3 lines
So perhaps the problem to be solved is not how to use
/proc/kallsymsif no vmlinux is supplied to the script, but ensure that the [kernel.kallsyms] is used?
Yes, this is good point, the subject should be changed to:
"Question: perf dso support for kallsyms".
I think I made a mistake in my previous email, so clarify it:
~/.debug/\[kernel.kallsyms\]/$BUILDID/kallsyms should has higher priority than /proc/kallsyms, by default the perf tool should use kallsyms file under ~/.debug folder. I also tried to manually specify the kallsyms file with the command "perf script --kallsyms ./kallsyms", for both cases perf fails to parse symbols for any kernel address.
Thanks,
Leo Yan
Quoted 20 lines · Leo Yan
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use/proc/kallsymsfor kernel symbols parsing, as result it will run into below flow:thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()I can observe the function
data_read_offset()returns failure, this is caused by checking the offset sanity "if (offset >dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.
Quoted 5 lines
Now I still don't understand how the dso/map support "
/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
Hi Jiri, Arnaldo,
Could you give some suggestion for this question? Thanks!
Quoted 18 lines
static
ssize_tdata_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data,ssize_tsize){ if (data_file_size(dso, machine)) return -1; /* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size); }Thanks,
Leo Yan
Quoted 7 lines · On Fri, Nov 02, 2018 at 10:55:16AM +0800, [email protected] wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use/proc/kallsymsfor kernel symbols parsing, as result it will run into below flow:
yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms
Quoted 5 lines
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
map__load(al.map);
dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
`-> data_read_offset()so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?
thanks,
jirka
Quoted 32 lines
I can observe the function
data_read_offset()returns failure, this is caused by checking the offset sanity "if (offset >dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.Now I still don't understand how the dso/map support "
/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static
ssize_tdata_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data,ssize_tsize){ if (data_file_size(dso, machine)) return -1; /* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size); }Thanks,
Leo Yan
Hi Jiri,
Quoted 12 lines · Jiri Olsa
Quoted 7 lines · On Fri, Nov 02, 2018 at 10:55:16AM +0800, [email protected] wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use/proc/kallsymsfor kernel symbols parsing, as result it will run into below flow:yep, AFAIK if there's no vmlinux found we fallback to
/proc/kallsymsQuoted 1 line
Quoted 8 lines
Quoted 4 lines
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?
So at my side the error is the CoreSight trace decoder fails to generate samples if the sample has kernel address, thus the decoder doesn't generate any kernel sample if use kallsyms as dso.
For more detailed info is: the CoreSight decoder needs firstly to get dso related info by calling dso__data_read_offset() [1], if we use kallsyms then this function always returns failure then this leads the docoder to discard all kernel samples.
Regarding to testing case, I can simply run 'uname' program so it can produce kernel and user space trace data, if I use 'perf script -k vmlinux' then I can see perf generate samples for kernel related traces, but if use 'perf script' then only can see samples for user space.
Thanks,
Leo Yan
Quoted 31 lines
Quoted 31 lines
I can observe the function
data_read_offset()returns failure, this is caused by checking the offset sanity "if (offset >dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.Now I still don't understand how the dso/map support "
/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.
static
ssize_tdata_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data,ssize_tsize){ if (data_file_size(dso, machine)) return -1; /* Check the offset sanity. */ if (offset > dso->data.file_size) return -1; if (offset + size < offset) return -1; return cached_read(dso, machine, offset, data, size); }Thanks,
Leo Yan
Quoted 24 lines · On Thu, Nov 08, 2018 at 04:51:52PM +0800, [email protected] wrote:
Hi Jiri,
Quoted 12 lines · Jiri Olsa
Quoted 7 lines · On Fri, Nov 02, 2018 at 10:55:16AM +0800, [email protected] wrote:
Hi all,
Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use/proc/kallsymsfor kernel symbols parsing, as result it will run into below flow:yep, AFAIK if there's no vmlinux found we fallback to
/proc/kallsymsQuoted 1 line
Quoted 8 lines
Quoted 4 lines
thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al); map__load(al.map); dso__data_read_offset(al.map->dso, machine, offset, buffer, size); `-> data_read_offset()so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?
Quoted 8 lines
So at my side the error is the CoreSight trace decoder fails to generate samples if the sample has kernel address, thus the decoder doesn't generate any kernel sample if use kallsyms as dso.
For more detailed info is: the CoreSight decoder needs firstly to get dso related info by calling
dso__data_read_offset()[1], if we use kallsyms then this function always returns failure then this leads the docoder to discard all kernel samples.
I haven't checked on this code for some time but AFAICS before you call dso__data_read_offset you need to check
dso->data.status != DSO_DATA_STATUS_ERROR
map__load call ahead should take care on setting this and find the source of the data
check the intel_pt code (intel_pt_walk_next_insn) or for example grab_bb in perf script
I guess you need to have either vmlinux or kcore in place to get some data out of it
jirka