Skip to contentSkip to conversation

Question: perf dso support for /proc/kallsyms

DISCUSSION [email protected] 4 participants 11 messages Nov 2, 2018 – Nov 9, 2018

lore
Share & tools
lore
#0 lore raw

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:

  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.

static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data, ssize_t size)

{
        if (data_file_size(dso, machine))
                return -1;

        /* Check the offset sanity. */
        if (offset > dso->data.file_size)
                return -1;

        if (offset + size < offset)
                return -1;

        return cached_read(dso, machine, offset, data, size);
}

Thanks,
Leo Yan

#1 lore raw
Quoted 2 lines

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

The way this is supported is that at record time, pseudo mmap records are created for the kernel. But depending on permissions these might not get created. This isn't just an ETM issue, it can happen on Intel too.

What do you see in "perf report -D", do you see a PERF_RECORD_MMAP record for "[kernel.kallsyms]_text" and possibly some others for loadable kernel modules in /lib/modules?

Does it all work if you run perf record as sudo? Or if you do

sudo sysctl kernel.kptr_restrict=0

before you run perf record?

Al
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#2 lore raw

Hi Al,

Quoted 6 lines · Al Grant
Quoted 2 lines

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

The way this is supported is that at record time, pseudo mmap records are created for the kernel. But depending on permissions these might not get created. This isn't just an ETM issue, it can happen on Intel too.

Agree, I also think this is not a specific issue only for Arm platform, this should be one common issue for how to parse kernel symbols with kallsyms file.

Quoted 3 lines

What do you see in "perf report -D", do you see a PERF_RECORD_MMAP record for "[kernel.kallsyms]_text" and possibly some others for loadable kernel modules in /lib/modules?

Yes, I can see PERF_RECORD_MMAP for "[kernel.kallsyms]_text".

0x350 [0x50]: event: 1
.
. ... raw event: size 80 bytes

.  0000:  01 00 00 00 01 00 50 00 ff ff ff ff 00 00 00 00  ......P.........
.  0010:  00 00 08 08 00 00 ff ff ff ff f7 f7 ff ff 00 00  ................
.  0020:  00 00 08 08 00 00 ff ff 5b 6b 65 72 6e 65 6c 2e  ........[kernel.
.  0030:  6b 61 6c 6c 73 79 6d 73 5d 5f 74 65 78 74 00 00  kallsyms]_text..
.  0040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

0x350 [0x50]: PERF_RECORD_MMAP -1/0: [0xffff000008080000(0xfffff7f7ffff) @ 0xffff000008080000]: x [kernel.kallsyms]_text

Below is the infor for checking buildid:

root@debian:~/coresight_test# perf buildid-list
0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms]
b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main
26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-2.27.so
8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso]
25829a59e21012cfde7850b30a310cd3a58f531c /root/coresight_test/libcstest.so
70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-gnu/libc-2.27.so
root@debian:~/coresight_test# ls ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/
kallsyms
Quoted 5 lines

Does it all work if you run perf record as sudo? Or if you do

sudo sysctl kernel.kptr_restrict=0

before you run perf record?

Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user. I suspected the pointer permission issue so checked with below command:

root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict 
0

Thanks,
Leo Yan

#3 lore raw
Quoted 5 lines
root@debian:~/coresight_test# perf buildid-list
0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms]
b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main
26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-
2.27.so
Quoted 8 lines
8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso]
25829a59e21012cfde7850b30a310cd3a58f531c
/root/coresight_test/libcstest.so
70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-
gnu/libc-2.27.so
root@debian:~/coresight_test# ls
~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/
kallsyms

What's in that last file? I've seen it happen that the copy of kallsyms in ~/.debug has the symbol addresses as zeroes - possibly because it was created when you didn't have permissions. That's really a bug in perf, as cacheing a copy of this file with the addresses zeroed out is kind of pointless. Again, this happens on Intel too.

Then, you can give yourself permissions - but perf's already cached the file and won't update it!

If you delete it, and then rerun perf record (either as sudo or now that you've got kptr_restrict=0) you should see it reappear, with correct kernel addresses.

Perhaps nobody spotted this on Intel because perf report goes directly to /proc/kallsyms. But it would be an issue if you ran a perf report on a perf.data from an older kernel and it had to go to ~/.debug. At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would mean you couldn't symbolicate any addresses.

Al

Quoted 7 lines
Quoted 6 lines

Does it all work if you run perf record as sudo? Or if you do

sudo sysctl kernel.kptr_restrict=0

before you run perf record?

Quoted 8 lines

Yes, tested this on Juno board with Debian rootFS and logined in with 'root' user. I suspected the pointer permission issue so checked with below command:

root@debian:~/coresight_test# cat /proc/sys/kernel/kptr_restrict
0

Thanks,
Leo Yan

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#4 lore raw
Quoted 14 lines · Al Grant
Quoted 5 lines
root@debian:~/coresight_test# perf buildid-list
0242d9154c78df1d8fe1d0512c36a236d0861a18 [kernel.kallsyms]
b8c89e8ba41a2ea486c66a50c29c60d38c34a759 /root/coresight_test/main
26b12a9d1a54ed2b0478cb0203435b76aabab3fb /usr/lib/aarch64-linux-gnu/ld-
2.27.so
Quoted 8 lines
8fca7ed524c9469b065af83bc8a529fe72858f53 [vdso]
25829a59e21012cfde7850b30a310cd3a58f531c
/root/coresight_test/libcstest.so
70512527439ef76c8802a7a2a546bde6a5a6e967 /usr/lib/aarch64-linux-
gnu/libc-2.27.so
root@debian:~/coresight_test# ls
~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/
kallsyms
Quoted 1 line

What's in that last file?

I can see the correct symbol infos in this file:

root@debian:~/coresight_test# cat ~/.debug/\[kernel.kallsyms\]/0242d9154c78df1d8fe1d0512c36a236d0861a18/kallsyms | head -20
ffff000008080000 t _head
ffff000008080000 T _text
ffff000008080040 t pe_header
ffff000008080044 t coff_header
ffff000008080058 t optional_header
ffff000008080070 t extra_header_fields
ffff0000080800f8 t section_table
ffff000008081000 T do_undefinstr
ffff000008081000 t efi_header_end
ffff000008081000 T _stext
ffff000008081000 T __exception_text_start
ffff000008081268 T do_cp15instr
ffff000008081380 T do_sysinstr
ffff000008081410 T do_debug_exception
ffff000008081550 T do_mem_abort
ffff000008081600 T do_el0_irq_bp_hardening
ffff000008081650 T do_el0_ia_bp_hardening
ffff0000080816f8 T do_sp_pc_abort
ffff0000080817c4 T __exception_text_end
ffff0000080817c8 t bcm2835_handle_irq
Quoted 13 lines

I've seen it happen that the copy of kallsyms
in ~/.debug has the symbol addresses as zeroes - possibly because it was created when you didn't have permissions. That's really a bug in perf, as cacheing a copy of this file with the addresses zeroed out is kind of pointless. Again, this happens on Intel too.

Then, you can give yourself permissions - but perf's already cached the file and won't update it!

If you delete it, and then rerun perf record (either as sudo or now that you've got kptr_restrict=0) you should see it reappear, with correct kernel addresses.

Quoted 5 lines

Perhaps nobody spotted this on Intel because perf report goes directly to /proc/kallsyms. But it would be an issue if you ran a perf report on a perf.data from an older kernel and it had to go to ~/.debug. At that point the fact that ~/.debug/[kernel.kallsyms] had zeroes would mean you couldn't symbolicate any addresses.

This is another issue related with permission for accessing kernel pointer values and this will bother much for cross platforms usage (e.g. 'perf record' on Arm platform and 'perf script' on x86 platform).

At my side, I executed all commands on Arm Juno board and simply to say the kallsyms support is broken in perf.

Thanks,
Leo Yan

#5 lore raw
In reply to [email protected]

Hi Leo,

My understanding is that when we decode CoreSight trace - be it on the system that generated it, or off target device on a separate host system, we are using the dso files in .debug/ as these represent the memory layout at the time trace was recorded.

If I look into a recent session copied up to my linux box from DB410, is see ~/.debug/\[kernel.kallsyms\]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
- which seems to me to be the relevant symbol file to use rather than the live one generated by /proc/kallsyms. I don't really want to use the local /proc/kallsyms on my x86 linux box when decoding an ARM
trace captured elsewhere.

So perhaps the problem to be solved is not how to use /proc/kallsyms if no vmlinux is supplied to the script, but ensure that the [kernel.kallsyms] is used?

Regards

Mike

Quoted 49 lines · On Fri, 2 Nov 2018 at 02:55, <[email protected]> wrote:

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:

  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.

static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data, ssize_t size)

{
        if (data_file_size(dso, machine))
                return -1;

        /* Check the offset sanity. */
        if (offset > dso->data.file_size)
                return -1;

        if (offset + size < offset)
                return -1;

        return cached_read(dso, machine, offset, data, size);
}

Thanks,
Leo Yan
_______________________________________________
CoreSight mailing list
[email protected]
https://lists.linaro.org/mailman/listinfo/coresight

Signature · 3 lines
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK
#6 lore raw

Hi Mike,

Quoted 14 lines · Mike Leach

Hi Leo,

My understanding is that when we decode CoreSight trace - be it on the system that generated it, or off target device on a separate host system, we are using the dso files in .debug/ as these represent the memory layout at the time trace was recorded.

If I look into a recent session copied up to my linux box from DB410, is see ~/.debug/\[kernel.kallsyms\]/ee80c1f3a434469f6174e1cf4bb5583d83a013dc/kallsyms
- which seems to me to be the relevant symbol file to use rather than the live one generated by /proc/kallsyms. I don't really want to use the local /proc/kallsyms on my x86 linux box when decoding an ARM
trace captured elsewhere.

Quoted 3 lines

So perhaps the problem to be solved is not how to use /proc/kallsyms if no vmlinux is supplied to the script, but ensure that the [kernel.kallsyms] is used?

Yes, this is good point, the subject should be changed to:
"Question: perf dso support for kallsyms".

I think I made a mistake in my previous email, so clarify it:

~/.debug/\[kernel.kallsyms\]/$BUILDID/kallsyms should has higher priority than /proc/kallsyms, by default the perf tool should use kallsyms file under ~/.debug folder. I also tried to manually specify the kallsyms file with the command "perf script --kallsyms ./kallsyms", for both cases perf fails to parse symbols for any kernel address.

Thanks,
Leo Yan

#7 lore raw
In reply to [email protected]
Quoted 20 lines · Leo Yan

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:

  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.

Quoted 5 lines

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.

Hi Jiri, Arnaldo,

Could you give some suggestion for this question? Thanks!

Quoted 18 lines

static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data, ssize_t size)

{
        if (data_file_size(dso, machine))
                return -1;

        /* Check the offset sanity. */
        if (offset > dso->data.file_size)
                return -1;

        if (offset + size < offset)
                return -1;

        return cached_read(dso, machine, offset, data, size);
}

Thanks,
Leo Yan

#8 lore raw
In reply to [email protected]
Quoted 7 lines · On Fri, Nov 02, 2018 at 10:55:16AM +0800, [email protected] wrote:

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:

yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms

Quoted 5 lines
  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?

thanks,
jirka

Quoted 32 lines

I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.

static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data, ssize_t size)

{
        if (data_file_size(dso, machine))
                return -1;

        /* Check the offset sanity. */
        if (offset > dso->data.file_size)
                return -1;

        if (offset + size < offset)
                return -1;

        return cached_read(dso, machine, offset, data, size);
}

Thanks,
Leo Yan

#9 lore raw

Hi Jiri,

Quoted 12 lines · Jiri Olsa
Quoted 7 lines · On Fri, Nov 02, 2018 at 10:55:16AM +0800, [email protected] wrote:

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:

yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?

So at my side the error is the CoreSight trace decoder fails to generate samples if the sample has kernel address, thus the decoder doesn't generate any kernel sample if use kallsyms as dso.

For more detailed info is: the CoreSight decoder needs firstly to get dso related info by calling dso__data_read_offset() [1], if we use kallsyms then this function always returns failure then this leads the docoder to discard all kernel samples.

Regarding to testing case, I can simply run 'uname' program so it can produce kernel and user space trace data, if I use 'perf script -k vmlinux' then I can see perf generate samples for kernel related traces, but if use 'perf script' then only can see samples for user space.

Thanks,
Leo Yan

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/cs-etm.c#n302

Quoted 31 lines
Quoted 31 lines

I can observe the function data_read_offset() returns failure, this is caused by checking the offset sanity "if (offset > dso->data.file_size)" (I pasted the whole function code at below in case you want to get more context for it), but if perf use "/proc/kallsyms" to load kernel symbols, the variable 'dso->data.file_size' will be set to zero thus the sanity checking always thinks the offset is out of the file size bound.

Now I still don't understand how the dso/map support "/proc/kallsyms" and have no idea to fix this issue, though I spent some time to look into it.

Could you give some suggestion for this? Or even better if you have fixing for this, I am glad to test at my side.

static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
u64 offset, u8 *data, ssize_t size)

{
        if (data_file_size(dso, machine))
                return -1;

        /* Check the offset sanity. */
        if (offset > dso->data.file_size)
                return -1;

        if (offset + size < offset)
                return -1;

        return cached_read(dso, machine, offset, data, size);
}

Thanks,
Leo Yan

#10 lore raw
Quoted 24 lines · On Thu, Nov 08, 2018 at 04:51:52PM +0800, [email protected] wrote:

Hi Jiri,

Quoted 12 lines · Jiri Olsa
Quoted 7 lines · On Fri, Nov 02, 2018 at 10:55:16AM +0800, [email protected] wrote:

Hi all,

Now I found that if use the command 'perf script' for Arm CoreSight trace data, it fails to parse kernel symbols if we don't specify kernel vmlinux
file. So when we don't specify kernel symbol files then perf tool will
roll back to use /proc/kallsyms for kernel symbols parsing, as result it will run into below flow:

yep, AFAIK if there's no vmlinux found we fallback to /proc/kallsyms

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
  thread__find_addr_map(thread, cpumode, MAP__FUNCTION, address, &al);
  map__load(al.map);
  dso__data_read_offset(al.map->dso, machine, offset, buffer, size);
    `-> data_read_offset()

so what is the actual error you see in the perf script? unresolved samples? could you please describe your config and workload?

Quoted 8 lines

So at my side the error is the CoreSight trace decoder fails to generate samples if the sample has kernel address, thus the decoder doesn't generate any kernel sample if use kallsyms as dso.

For more detailed info is: the CoreSight decoder needs firstly to get dso related info by calling dso__data_read_offset() [1], if we use kallsyms then this function always returns failure then this leads the docoder to discard all kernel samples.

I haven't checked on this code for some time but AFAICS before you call dso__data_read_offset you need to check
dso->data.status != DSO_DATA_STATUS_ERROR

map__load call ahead should take care on setting this and find the source of the data

check the intel_pt code (intel_pt_walk_next_insn) or for example grab_bb in perf script

I guess you need to have either vmlinux or kcore in place to get some data out of it

jirka