bpf: Cache the last valid build_id.
Subsystems (2)
PATCH v2 1/1 bpf: Cache the last valid build_id.
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Tested through "stacktrace_build_id" and "stacktrace_build_id_nmi" in test_progs.
--- kernel/bpf/stackmap.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
kernel/bpf/stackmap.c +11 -1
@@ -132,7 +132,8 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,int i;struct mmap_unlock_irq_work *work =NULL ;bool irq_work_busy =bpf_mmap_unlock_get_irq_work (&work); -struct vm_area_struct *vma; +struct vm_area_struct *vma, *prev_vma =NULL ; +const char *prev_build_id;/* If the irq_work is in use, fall back to report ips. Same * fallback is used for kernel stack (!user) on a stackmap with
@@ -150,6 +151,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }for (i =0 ; i < trace_nr; i++) { +if (range_in_vma (prev_vma, ips[i], ips[i])) { + vma = prev_vma; +memcpy (id_offs[i].build_id, prev_build_id, +BUILD_ID_SIZE_MAX ); + goto build_id_valid; + } vma =find_vma (current->mm, ips[i]);if (!vma ||build_id_parse (vma, id_offs[i].build_id,NULL )) {/* per entry fall back to ips */
@@ -158,9 +165,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,memset (id_offs[i].build_id,0 ,BUILD_ID_SIZE_MAX );continue ; } +build_id_valid: id_offs[i].offset = (vma->vm_pgoff <<PAGE_SHIFT ) + ips[i] - vma->vm_start; id_offs[i].status =BPF_STACK_BUILD_ID_VALID ; + prev_vma = vma; + prev_build_id = id_offs[i].build_id; }bpf_mmap_unlock_mm (work, current->mm); }
Signature · 1 line
2.35.1.473.g83b2b277ed-goog
Quoted 8 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Quoted 6 lines
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>---
LGTM. Can you share performance numbers before and after?
Quoted 46 lines
kernel/bpf/stackmap.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)kernel/bpf/stackmap.c +11
-2@@ -132,7 +132,8 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,int i;struct mmap_unlock_irq_work *work =NULL ;bool irq_work_busy =bpf_mmap_unlock_get_irq_work (&work); -struct vm_area_struct *vma; +struct vm_area_struct *vma, *prev_vma =NULL ; +const char *prev_build_id;/* If the irq_work is in use, fall back to report ips. Same * fallback is used for kernel stack (!user) on a stackmap with @@ -150,6 +151,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }for (i =0 ; i < trace_nr; i++) { +if (range_in_vma (prev_vma, ips[i], ips[i])) { + vma = prev_vma; +memcpy (id_offs[i].build_id, prev_build_id, +BUILD_ID_SIZE_MAX ); + goto build_id_valid; + } vma =find_vma (current->mm, ips[i]);if (!vma ||build_id_parse (vma, id_offs[i].build_id,NULL )) {/* per entry fall back to ips */ @@ -158,9 +165,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,memset (id_offs[i].build_id,0 ,BUILD_ID_SIZE_MAX );continue ; } +build_id_valid: id_offs[i].offset = (vma->vm_pgoff <<PAGE_SHIFT ) + ips[i] - vma->vm_start; id_offs[i].status =BPF_STACK_BUILD_ID_VALID ; + prev_vma = vma; + prev_build_id = id_offs[i].build_id; }bpf_mmap_unlock_mm (work, current->mm); } --2.35.1.473.g83b2b277ed-goog
Quoted 13 lines · Andrii Nakryiko
Quoted 11 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.
Quoted 8 lines
Quoted 3 lines
Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>---
LGTM. Can you share performance numbers before and after?
Acked-by Andrii Nakryiko <[email protected]>
Thanks Andrii.
On a real-world workload, we observed that 66% of cpu cycles in __bpf_get_stackid() were spent on build_id_parse() and find_vma(). This was before.
We haven't evaluated the performance with this patch yet. This optimization seems straightforward, so we plan to upstream it first and then retest.
Quoted 27 lines · Hao Luo
Quoted 13 lines · Andrii Nakryiko
Quoted 11 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Quoted 8 lines
Quoted 3 lines
Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>---
LGTM. Can you share performance numbers before and after?
Acked-by Andrii Nakryiko <[email protected]>Thanks Andrii.
Quoted 7 lines
On a real-world workload, we observed that 66% of cpu cycles in
__bpf_get_stackid()were spent onbuild_id_parse()andfind_vma(). This was before.We haven't evaluated the performance with this patch yet. This optimization seems straightforward, so we plan to upstream it first and then retest.
Ok, once it lands upstream, I'd really appreciate if you can retest and update us with numbers. Thanks!
Quoted 33 lines · Andrii Nakryiko
Quoted 31 lines · Hao Luo
Quoted 13 lines · Andrii Nakryiko
Quoted 11 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Quoted 8 lines
Quoted 3 lines
Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>---
LGTM. Can you share performance numbers before and after?
Acked-by Andrii Nakryiko <[email protected]>Thanks Andrii.
On a real-world workload, we observed that 66% of cpu cycles in
__bpf_get_stackid()were spent onbuild_id_parse()andfind_vma(). This was before.
Quoted 6 lines
Quoted 3 lines
We haven't evaluated the performance with this patch yet. This optimization seems straightforward, so we plan to upstream it first and then retest.
Ok, once it lands upstream, I'd really appreciate if you can retest and update us with numbers. Thanks!
Sure, will do that.
Quoted 8 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Quoted 5 lines
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>
Nice optimization!
Namhyung Kim
Acked-byHello,
Quoted 10 lines · Song Liu
Quoted 8 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Quoted 5 lines
Quoted 5 lines
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>
Thanks,
Namhyung
Quoted 7 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Quoted 5 lines
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>
Thanks,
Pasha
Quoted 9 lines · Pasha Tatashin
Quoted 7 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Quoted 8 lines
Quoted 5 lines
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>Reviewed-by Pasha Tatashin <[email protected]>
An update with performance numbers. Thanks to Blake Jones for collecting the stats:
In a production workload, with BPF probes sampling stack trace, we see the following changes:
- stack_map_get_build_id_offset() is taking 70% of the time of__bpf_get_stackid(); it was 80% before.
- find_get_page() and find_vma() together are taking 75% of the time
of stack_map_get_build_id_offset(); it was 83% before.
Note the call chain is
__bpf_get_stackid()
-> stack_map_get_build_id_offset()
-> find_get_page()
-> find_vma()Quoted 2 lines
Thanks,
Pasha
Quoted 21 lines · Hao Luo
Quoted 9 lines · Pasha Tatashin
Quoted 7 lines · Hao Luo
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
Quoted 8 lines
Quoted 5 lines
Tested through "
stacktrace_build_id" and "stacktrace_build_id_nmi" intest_progs.Suggested-by Greg Thelen <[email protected]>Signed-off-by Hao Luo <[email protected]>Reviewed-by Pasha Tatashin <[email protected]>
Quoted 8 lines
An update with performance numbers. Thanks to Blake Jones for collecting the stats:
In a production workload, with BPF probes sampling stack trace, we see the following changes:
-
stack_map_get_build_id_offset()is taking 70% of the time of__bpf_get_stackid(); it was 80% before.
Great, thanks for following up with updated numbers!
Quoted 13 lines
-
find_get_page()andfind_vma()together are taking 75% of the time
ofstack_map_get_build_id_offset(); it was 83% before.Note the call chain is
__bpf_get_stackid()
->stack_map_get_build_id_offset()-> find_get_page() -> find_vma()Quoted 2 lines
Thanks,
Pasha
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Quoted 8 lines · On Wed, 23 Feb 2022 16:05:31 -0800 you wrote:
For binaries that are statically linked, consecutive stack frames are likely to be in the same VMA and therefore have the same build id. As an optimization for this case, we can cache the previous frame's VMA, if the new frame has the same VMA as the previous one, reuse the previous one's build id. We are holding the MM locks as reader across the entire loop, so we don't need to worry about VMA going away.
[...]
Here is the summary with links:
- [bpf-next,v2] bpf: Cache the last valid build_id.
https://git.kernel.org/bpf/bpf-next/c/ceac059ed4fd
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html