Symbol resolution when text in hugepages
Hi all,
We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously, everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into /tmp/perf-PID.map file. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.
Thanks
Sri
Quoted 4 lines · Srilakshmi
Hi all,
We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously, everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into
/tmp/perf-PID.mapfile. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.
Hi Sri,
Since Linux 5.4 you can use MADV_HUGEPAGE with madvise for file backed pages. It can still be profitable to use other sources of huge pages so that you don't wait for the hugepage to be formed, in that case if there is a file/memfd_create you can embed the information in the filename. For anonymous pages you can merge them when scanning the executable in perf_event__synthesize_mmap_events, but it can be problematic if the program header is on an anonymous page.
Thanks,
Ian
Quoted 2 lines
Thanks
Sri
A message that is not in the archive (sent off-list or not yet received): [email protected]
From: Ian Rogers <[email protected]>
Sent: Thursday, September 14, 2023 4:26 PM
To: Jayaramappa, Srilakshmi
Cc: [email protected]
Subject: Re: Symbol resolution when text in hugepages
Quoted 4 lines · Srilakshmi
Hi all,
We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously, everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into
/tmp/perf-PID.mapfile. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.
Hi Sri,
Since Linux 5.4 you can use MADV_HUGEPAGE with madvise for file backed pages. It can still be profitable to use other sources of huge pages so that you don't wait for the hugepage to be formed, in that case if there is a file/memfd_create you can embed the information in the filename. For anonymous pages you can merge them when scanning the executable in perf_event__synthesize_mmap_events, but it can be problematic if the program header is on an anonymous page.
Thanks,
Ian
Quoted 2 lines
Thanks
Sri
[resending since my message bounced from the mailing list. Apologies for the spam, Ian]
Thank you for the response, Ian, appreciate it. The pointer to perf_event__synthesize_mmap_events is super useful, we will need to do something like that to find if the address range is backed by hugepages and if so grab the offset from the executable.
For more context - our application first loads and then we remap its text segment to hugepages via the libhugetlbfs. When that happens the memory map doesn't need any offset. But perf is grabbing the maps from the /proc/pid/maps and trying to lookup the address in the binary, so it looks at the wrong place without the offset into the file.
Thanks!
Sri
Quoted 32 lines · Srilakshmi
From: Ian Rogers <[email protected]>
Sent: Thursday, September 14, 2023 4:26 PM
To: Jayaramappa, Srilakshmi
Cc: [email protected]
Subject: Re: Symbol resolution when text in hugepagesQuoted 4 lines · Srilakshmi
Hi all,
We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously, everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into
/tmp/perf-PID.mapfile. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.Hi Sri,
Since Linux 5.4 you can use
MADV_HUGEPAGEwith madvise for file backed pages. It can still be profitable to use other sources of huge pages so that you don't wait for the hugepage to be formed, in that case if there is a file/memfd_create you can embed the information in the filename. For anonymous pages you can merge them when scanning the executable inperf_event__synthesize_mmap_events, but it can be problematic if the program header is on an anonymous page.Thanks,
IanQuoted 2 lines
Thanks
Sri
Quoted 6 lines
[resending since my message bounced from the mailing list. Apologies for the spam, Ian]
Thank you for the response, Ian, appreciate it. The pointer to
perf_event__synthesize_mmap_eventsis super useful, we will need to do something like that to find if the address range is backed by hugepages and if so grab the offset from the executable.For more context - our application first loads and then we remap its text segment to hugepages via the libhugetlbfs. When that happens the memory map doesn't need any offset. But perf is grabbing the maps from the
/proc/pid/mapsand trying to lookup the address in the binary, so it looks at the wrong place without the offset into the file.
Hi Sri,
I think you'd be much better served using MADV_COLLAPSE or MADV_HUGEPAGE, not least because hugetlbfs requires preallocation of huge pages. Using these shouldn't alter perf's symbolization.
Thanks,
Ian
Quoted 2 lines
Thanks!
Sri