bpf: Better build_id caching in stack_map_get_build_id_offset()
Subsystems (2)
Ihor Solodrai
reviewerPATCH v2 1/1 bpf: Better build_id caching in stack_map_get_build_id_offset()
This patch is a follow up to recent implementation of stack_map_get_build_id_offset_sleepable() [1].
stack_map_get_build_id_offset() and its sleepable variant each cached only the last successfully resolved VMA, with separate bookkeeping in each function. A run of IPs in a VMA with no usable build ID will repeat the lookup for every frame: find_vma() in the non-sleepable path, a VMA lock and a blocking build_id_parse_file() in the sleepable.
Factor the per-call cache into a shared struct stack_map_build_id_cache with two independent slots [2][3], used by both functions:
* resolved - last VMA that produced a build ID (file, build_id and
range), reused to skip the lookup and the parse;
* unresolved - last VMA with no usable build ID (range only), reused to emit a raw IP without another lookup or parse.
Keeping the slots independent means a build-ID-less VMA no longer evicts the last resolved build ID, so a trace alternating between a binary and a region without one stops re-resolving the binary on every return.
The shared lookup tests [vm_start, vm_end), matching the sleepable path; the non-sleepable path previously reused a build ID for ip == vm_end (range_in_vma() is inclusive) and now re-resolves it correctly.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/CAEf4Bza2fRDGhLQoPE-EzM7F34xaEJfi5Exmxb-iWVUN3F06=g@mail.gmail.com/
[3] https://lore.kernel.org/bpf/CAEf4BzZXJFr=1iiVx937ht=4PYQkQHg=eFk810zhMDzXQG3ihw@mail.gmail.com/
---
v1->v2:
* nits in attempt to de-claudify the diff (Alexei)
* factor out local variables in stack_map_build_id_set_from_cache() to make the conditions more readable
* make comments more concise
* I am ignoring the issue flagged by sashiko, since it's unrelated to the patch
v1: https://lore.kernel.org/bpf/[email protected]/
--- kernel/bpf/stackmap.c | 175 +++++++++++++++++++++++++++++------------- 1 file changed, 122 insertions(+), 53 deletions(-)
kernel/bpf/stackmap.c +122 -53
@@ -175,6 +175,88 @@ static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id,memcpy (id->build_id, build_id,BUILD_ID_SIZE_MAX ); } +/* +* Per stack_map_get_build_id_offset() call cache of the last VMA with a build ID +* resolved and the last VMA with no usable build ID. +* Adjacent stack frames tend to land in the same VMA or the same backing file, +* so caching the last result of each kind lets us skip unnecessary VMA lookups +* and build ID parse calls. +* A zero vm_end marks a slot as empty. +* resolved.build_id aliases the id_offs[] entry. +*/ +struct stack_map_build_id_cache { +struct { +struct file *file;/* pinned in the sleepable path; NULL otherwise */ +const unsigned char *build_id; +unsigned long vm_start; +unsigned long vm_end; +unsigned long vm_pgoff; + } resolved; +struct { +unsigned long vm_start; +unsigned long vm_end; + } unresolved; +}; + +/* +* Fill @id from a cached range covering @ip. On a hit this writes @id (resolved +* range -> build ID + offset, unresolved range -> raw ip) and returns 0; on a +* miss it leaves @id untouched and returns -ENOENT. +*/ +static int stack_map_build_id_set_from_cache (struct stack_map_build_id_cache *cache, +struct bpf_stack_build_id *id,u64 ip) +{ +unsigned long vm_start, vm_end, vm_pgoff; +u64 offset; + + vm_start = cache->resolved.vm_start; + vm_end = cache->resolved.vm_end; +if (vm_end && ip >= vm_start && ip < vm_end) { + vm_pgoff = cache->resolved.vm_pgoff; + offset =stack_map_build_id_offset (vm_pgoff, vm_start, ip); +stack_map_build_id_set_valid (id, offset, cache->resolved.build_id); +return 0 ; + } + + vm_start = cache->unresolved.vm_start; + vm_end = cache->unresolved.vm_end; +if (vm_end && ip >= vm_start && ip < vm_end) { +stack_map_build_id_set_ip (id); +return 0 ; + } + +return -ENOENT ; +} + +/* +* Record @vma's build ID as the last resolved one. @file is the pinned backing +* file in the sleepable path (released when evicted), or NULL otherwise. +*/ +static void stack_map_build_id_cache_set_resolved (struct stack_map_build_id_cache *cache, +struct file *file, +const unsigned char *build_id, +unsigned long vm_start, +unsigned long vm_end, +unsigned long vm_pgoff) +{ +if (cache->resolved.file) +fput (cache->resolved.file); + cache->resolved.file = file; + cache->resolved.build_id = build_id; + cache->resolved.vm_start = vm_start; + cache->resolved.vm_end = vm_end; + cache->resolved.vm_pgoff = vm_pgoff; +} + +/* Record [vm_start, vm_end) as a range with no usable build ID. */ +static void stack_map_build_id_cache_set_unresolved (struct stack_map_build_id_cache *cache, +unsigned long vm_start, +unsigned long vm_end) +{ + cache->unresolved.vm_start = vm_start; + cache->unresolved.vm_end = vm_end; +} +struct stack_map_vma_lock {struct vm_area_struct *vma;struct mm_struct *mm;
@@ -244,15 +326,8 @@ static void stack_map_unlock_vma(struct stack_map_vma_lock *lock)static void stack_map_get_build_id_offset_sleepable (struct bpf_stack_build_id *id_offs,u32 trace_nr) { -struct mm_struct *mm = current->mm; -struct stack_map_vma_lock lock = { .mm = mm }; -struct { -struct file *file; -const unsigned char *build_id; -unsigned long vm_start; -unsigned long vm_end; -unsigned long vm_pgoff; - } cache = {}; +struct stack_map_vma_lock lock = { .mm = current->mm }; +struct stack_map_build_id_cache cache = {};unsigned long vm_pgoff, vm_start, vm_end;struct vm_area_struct *vma;struct file *file;
@@ -262,44 +337,40 @@ static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *ifor (u32 i =0 ; i < trace_nr; i++) { ip =READ_ONCE (id_offs[i].ip); -/* -* Range cache fast path: if ip falls within the previously -* resolved VMA range, reuse the cache build_id without -* re-acquiring the VMA lock. -*/ -if (cache.build_id && ip >= cache.vm_start && ip < cache.vm_end) { - offset =stack_map_build_id_offset (cache.vm_pgoff, cache.vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, cache.build_id); +if (!stack_map_build_id_set_from_cache (&cache, &id_offs[i], ip))continue ; - } vma =stack_map_lock_vma (&lock, ip);if (!vma) {stack_map_build_id_set_ip (&id_offs[i]);continue ; } + + vm_pgoff = vma->vm_pgoff; + vm_start = vma->vm_start; + vm_end = vma->vm_end; +if (vma_is_anonymous (vma) || !vma->vm_file) { -stack_map_build_id_set_ip (&id_offs[i]);stack_map_unlock_vma (&lock); +stack_map_build_id_set_ip (&id_offs[i]); +stack_map_build_id_cache_set_unresolved (&cache, vm_start, vm_end);continue ; } file = vma->vm_file; - vm_pgoff = vma->vm_pgoff; - vm_start = vma->vm_start; - vm_end = vma->vm_end; offset =stack_map_build_id_offset (vm_pgoff, vm_start, ip);/* -* Same backing file as previous (e.g. different VMAs -* of the same ELF binary). Reuse the cache build_id. +* Same backing file as the last resolved VMA (another mapping +* of the same ELF binary): reuse its build_id without re-parsing. */ -if (file == cache.file) { +if (file == cache.resolved.file) {stack_map_unlock_vma (&lock); -stack_map_build_id_set_valid (&id_offs[i], offset, cache.build_id); - cache.vm_start = vm_start; - cache.vm_end = vm_end; - cache.vm_pgoff = vm_pgoff; +stack_map_build_id_set_valid (&id_offs[i], offset, + cache.resolved.build_id); + cache.resolved.vm_start = vm_start; + cache.resolved.vm_end = vm_end; + cache.resolved.vm_pgoff = vm_pgoff;continue ; }
@@ -310,21 +381,17 @@ static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *iif (build_id_parse_file (file, id_offs[i].build_id,NULL )) {stack_map_build_id_set_ip (&id_offs[i]);fput (file); +stack_map_build_id_cache_set_unresolved (&cache, vm_start, vm_end);continue ; }stack_map_build_id_set_valid (&id_offs[i], offset, id_offs[i].build_id); -if (cache.file) -fput (cache.file); - cache.file = file; - cache.build_id = id_offs[i].build_id; - cache.vm_start = vm_start; - cache.vm_end = vm_end; - cache.vm_pgoff = vm_pgoff; +stack_map_build_id_cache_set_resolved (&cache, file, id_offs[i].build_id, + vm_start, vm_end, vm_pgoff); } -if (cache.file) -fput (cache.file); +if (cache.resolved.file) +fput (cache.resolved.file); }/*
@@ -343,8 +410,8 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,struct mmap_unlock_irq_work *work =NULL ;bool irq_work_busy =bpf_mmap_unlock_get_irq_work (&work);bool has_user_ctx = user && current && current->mm; -struct vm_area_struct *vma, *prev_vma =NULL ; -const unsigned char *prev_build_id =NULL ; +struct stack_map_build_id_cache cache = {}; +struct vm_area_struct *vma;int i;if (may_fault && has_user_ctx) {
@@ -365,27 +432,29 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,for (i =0 ; i < trace_nr; i++) {u64 ip =READ_ONCE (id_offs[i].ip); -u64 offset; -if (prev_build_id &&range_in_vma (prev_vma, ip, ip)) { - vma = prev_vma; - offset =stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, prev_build_id); +if (!stack_map_build_id_set_from_cache (&cache, &id_offs[i], ip))continue ; - } + vma =find_vma (current->mm, ip);if (!vma ||vma_is_anonymous (vma) ||fetch_build_id (vma, id_offs[i].build_id, may_fault)) { -/* per entry fall back to ips */ +/* per entry fall back to ips; cache build-ID-less range */ stack_map_build_id_set_ip (&id_offs[i]); - prev_vma = vma; - prev_build_id =NULL ; +if (vma) +stack_map_build_id_cache_set_unresolved (&cache, + vma->vm_start, vma->vm_end);continue ; } - offset =stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, id_offs[i].build_id); - prev_vma = vma; - prev_build_id = id_offs[i].build_id; +/* mmap_lock is held for the whole loop, so the cached VMA +* fields stay valid; no file pinning is needed here. +*/ +stack_map_build_id_set_valid (&id_offs[i], +stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip), + id_offs[i].build_id); +stack_map_build_id_cache_set_resolved (&cache,NULL , id_offs[i].build_id, + vma->vm_start, vma->vm_end, + vma->vm_pgoff); }bpf_mmap_unlock_mm (work, current->mm); }
Signature · 1 line
2.54.0
[email protected]
AI reviewThis isn't a bug, but should the comment text start on the line after the opening /* to match the BPF multi-line comment style?
Quoted 4 lines
kernel/bpf/stackmap.c +0
-0
[ ... ]
Quoted 11 lines
kernel/bpf/stackmap.c +2
-2@@ -343,8 +410,8 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,struct mmap_unlock_irq_work *work =NULL ;bool irq_work_busy =bpf_mmap_unlock_get_irq_work (&work);bool has_user_ctx = user && current && current->mm; -struct vm_area_struct *vma, *prev_vma =NULL ; -const unsigned char *prev_build_id =NULL ; +struct stack_map_build_id_cache cache = {}; +struct vm_area_struct *vma;int i;if (may_fault && has_user_ctx) {
[ ... ]
Quoted 24 lines
kernel/bpf/stackmap.c +5
-9@@ -365,27 +432,29 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,for (i =0 ; i < trace_nr; i++) {u64 ip =READ_ONCE (id_offs[i].ip); -u64 offset; -if (prev_build_id &&range_in_vma (prev_vma, ip, ip)) { - vma = prev_vma; - offset =stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, prev_build_id); +if (!stack_map_build_id_set_from_cache (&cache, &id_offs[i], ip))continue ; - } + vma =find_vma (current->mm, ip);if (!vma ||vma_is_anonymous (vma) ||fetch_build_id (vma, id_offs[i].build_id, may_fault)) { -/* per entry fall back to ips */ +/* per entry fall back to ips; cache build-ID-less range */ stack_map_build_id_set_ip (&id_offs[i]); - prev_vma = vma; - prev_build_id =NULL ; +if (vma) + stack_map_build_id_cache_set_unresolved(&cache,
kernel/bpf/stackmap.c static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, · line 451
Quoted 10 lines
kernel/bpf/stackmap.c +4
-4+ vma->vm_start, vma->vm_end);continue ; } - offset =stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, id_offs[i].build_id); - prev_vma = vma; - prev_build_id = id_offs[i].build_id; +/* mmap_lock is held for the whole loop, so the cached VMA +* fields stay valid; no file pinning is needed here. +*/
^^^^
This isn't a bug, but should the comment text start on the line after the opening /* to match the BPF multi-line comment style?
The other multi-line comments added in this patch follow that convention, such as in the new stack_map_build_id_cache structure and the helper functions.
kernel/bpf/stackmap.c static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, · line 460
Quoted 9 lines
kernel/bpf/stackmap.c +6
-0+stack_map_build_id_set_valid (&id_offs[i], +stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip), + id_offs[i].build_id); +stack_map_build_id_cache_set_resolved (&cache,NULL , id_offs[i].build_id, + vma->vm_start, vma->vm_end, + vma->vm_pgoff); }bpf_mmap_unlock_mm (work, current->mm); }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/27446103592
Ihor Solodrai
reviewerQuoted 57 lines · On 6/12/26 3:37 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/stackmap.c +0
-0[ ... ]
Quoted 11 lines
kernel/bpf/stackmap.c +2
-2@@ -343,8 +410,8 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,struct mmap_unlock_irq_work *work =NULL ;bool irq_work_busy =bpf_mmap_unlock_get_irq_work (&work);bool has_user_ctx = user && current && current->mm; -struct vm_area_struct *vma, *prev_vma =NULL ; -const unsigned char *prev_build_id =NULL ; +struct stack_map_build_id_cache cache = {}; +struct vm_area_struct *vma;int i;if (may_fault && has_user_ctx) {[ ... ]
Quoted 24 lines
kernel/bpf/stackmap.c +5
-9@@ -365,27 +432,29 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,for (i =0 ; i < trace_nr; i++) {u64 ip =READ_ONCE (id_offs[i].ip); -u64 offset; -if (prev_build_id &&range_in_vma (prev_vma, ip, ip)) { - vma = prev_vma; - offset =stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, prev_build_id); +if (!stack_map_build_id_set_from_cache (&cache, &id_offs[i], ip))continue ; - } + vma =find_vma (current->mm, ip);if (!vma ||vma_is_anonymous (vma) ||fetch_build_id (vma, id_offs[i].build_id, may_fault)) { -/* per entry fall back to ips */ +/* per entry fall back to ips; cache build-ID-less range */ stack_map_build_id_set_ip (&id_offs[i]); - prev_vma = vma; - prev_build_id =NULL ; +if (vma) + stack_map_build_id_cache_set_unresolved(&cache,kernel/bpf/stackmap.c static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, · line 451
Quoted 10 lines
kernel/bpf/stackmap.c +4
-4+ vma->vm_start, vma->vm_end);continue ; } - offset =stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip); -stack_map_build_id_set_valid (&id_offs[i], offset, id_offs[i].build_id); - prev_vma = vma; - prev_build_id = id_offs[i].build_id; +/* mmap_lock is held for the whole loop, so the cached VMA +* fields stay valid; no file pinning is needed here. +*/ ^^^^
Quoted 2 lines
This isn't a bug, but should the comment text start on the line after the opening /* to match the BPF multi-line comment style?
sigh... Looks like this also was present in v1, and I missed it.
I'll wait in case there is more feedback before making a v3.
Quoted 21 lines
The other multi-line comments added in this patch follow that convention, such as in the new
stack_map_build_id_cachestructure and the helper functions.kernel/bpf/stackmap.c static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, · line 460
Quoted 9 lines
kernel/bpf/stackmap.c +6
-0+stack_map_build_id_set_valid (&id_offs[i], +stack_map_build_id_offset (vma->vm_pgoff, vma->vm_start, ip), + id_offs[i].build_id); +stack_map_build_id_cache_set_resolved (&cache,NULL , id_offs[i].build_id, + vma->vm_start, vma->vm_end, + vma->vm_pgoff); }bpf_mmap_unlock_mm (work, current->mm); }---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/27446103592
Quoted 5 lines · Ihor Solodrai
kernel/bpf/stackmap.c +5
-0+static void stack_map_build_id_cache_set_resolved(struct stack_map_build_id_cache *cache, + struct file *file, + const unsigned char *build_id, + unsigned long vm_start, + unsigned long vm_end,
kernel/bpf/stackmap.c static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id, · line 249
Quoted 10 lines
kernel/bpf/stackmap.c +10
-0+ unsigned long vm_pgoff) +{ +if (cache->resolved.file) +fput (cache->resolved.file); + cache->resolved.file = file; + cache->resolved.build_id = build_id; + cache->resolved.vm_start = vm_start; + cache->resolved.vm_end = vm_end; + cache->resolved.vm_pgoff = vm_pgoff; +}
sorry for broken record.
claude's copy paste doesn't bother you?
It could have been:
struct resolved *res = &cache->resolved;
if (res->file)
fput(res->file);
res->file = file;res->build_id = build_id;res->vm_start = vm_start;res->vm_end = vm_end;res->vm_pgoff = vm_pgoff;
or
cache->resolved = (struct resolved) { file, build_iud, vm_start, emnv_end, vm_pgoff };
and probably other options to make a code easier for humans to read.
You have to explictly tell claude to deduplicate otherwise it keeps copy pasting left and right.
Ihor Solodrai
reviewerQuoted 11 lines · Alexei Starovoitov
Quoted 10 lines · Ihor Solodrai
kernel/bpf/stackmap.c +10
-0+static void stack_map_build_id_cache_set_resolved (struct stack_map_build_id_cache *cache, +struct file *file, +const unsigned char *build_id, +unsigned long vm_start, +unsigned long vm_end, +unsigned long vm_pgoff) +{ +if (cache->resolved.file) +fput (cache->resolved.file); + cache->resolved.file = file;
Quoted 8 lines
kernel/bpf/stackmap.c static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id, · line 249
Quoted 5 lines
kernel/bpf/stackmap.c +5
-0+ cache->resolved.build_id = build_id; + cache->resolved.vm_start = vm_start; + cache->resolved.vm_end = vm_end; + cache->resolved.vm_pgoff = vm_pgoff; +}sorry for broken record.
claude's copy paste doesn't bother you?
tbh, this particular block is fine with me, it's certainly not as bad as big expressions in the conditions that I rewrote in the other helper. I guess I'm just used to lengthy qualified expressions from my java days, so it doesn't bother me much (maybe it should).
I briefly chatted with Eduard off-list about this, and I agree with him that what makes this patch awkward is anonymous structs as cache fields. I'll try something else in v3.
Quoted 11 lines
It could have been:
struct resolved *res =
&cache->resolved;
if (res->file)fput(res->file);
res->file= file;res->build_id=build_id;res->vm_start=vm_start;res->vm_end=vm_end;res->vm_pgoff=vm_pgoff;
Quoted 8 lines
or
cache->resolved= (struct resolved) { file,build_iud,vm_start,emnv_end,vm_pgoff};and probably other options to make a code easier for humans to read.
You have to explictly tell claude to deduplicate otherwise it keeps copy pasting left and right.
I am painfully aware of claude's quirks, yes :)