BPF with deferred stacktrace and sframes?
Hi folks,
Now that deferred stacktrace is upstream and hopefully sframes will soon follow, is there any plans to integrate BPF with this facility?
Would there be a way to register a BPF program to request and receive a user space backtrace when the task goes back to user space?
Basically the API would require a program to register a callback function that retrieves the backtrace at a later time, and then during runtime (in any context) request that function to be called when the currently running task goes back to user space.
I've been Cc'ing the BPF list (and maintainers) on the deferred stack trace work and did get a few comments. But I haven't heard anything about BPF utilizing the feature yet. So I'm asking now if that's in the works?
If not, perhaps this would be a good topic to bring up at LSFMMBPF?
Cheers,
-- Steve
Quoted 6 lines · Steven Rostedt
Hi folks,
Now that deferred stacktrace is upstream and hopefully sframes will soon follow, is there any plans to integrate BPF with this facility?
Quoted 7 lines
Would there be a way to register a BPF program to request and receive a user space backtrace when the task goes back to user space?
Basically the API would require a program to register a callback function that retrieves the backtrace at a later time, and then during runtime (in any context) request that function to be called when the currently running task goes back to user space.
Hey Steven,
So I've been thinking about this on and off, and over the last year my view on how to integrate SFrame stack traces into BPF changed significantly. It now is actually much simpler, IMO, which also means much less complexity for the rest of the kernel.
In short, I think we should only provide a synchronous way to fetch SFrame-based stack traces. We will only allow this to be called from so-called sleepable BPF programs which can take page faults and wait for whatever file contents are necessary to parse SFrame data.
The reason this is sufficient (and even preferable) for BPF is that we now have the ability to utilize task_work_add() infrastructure from the BPF program from any context (including NMI). So even though SFrame-based stack unwinding can only happen from sleepable context, it is possible for the BPF program to set up an asynchronous callback that will run (through task_work "framework") right before the kernel returns to user space. That callback will run in a task context where we can wait for SFrame data to be faulted in, and then we can synchronously do stack unwinding. We return data to the BPF program, and that BPF program will use whatever means necessary to do something useful with that stack trace. This all simplifies API design significantly and we don't need to worry about any caching concerns either.
So what we will need from kernel's sframe support side is just an internal API to get stack trace, similar to get_perf_callchain(). But it will be guaranteed to be called in a nice task context. No NMI, no irq disabled, no nothing.
This shouldn't be a problem, right?
Quoted 6 lines
I've been Cc'ing the BPF list (and maintainers) on the deferred stack trace work and did get a few comments. But I haven't heard anything about BPF utilizing the feature yet. So I'm asking now if that's in the works?
If not, perhaps this would be a good topic to bring up at LSFMMBPF?
Yeah, we definitely can sync up on this during LSFMMBPF.
Quoted 4 lines
Cheers,
-- Steve
Quoted 6 lines · On Mon, 26 Jan 2026 16:02:10 -0800 Andrii Nakryiko <[email protected]>…
So what we will need from kernel's sframe support side is just an internal API to get stack trace, similar to
get_perf_callchain(). But it will be guaranteed to be called in a nice task context. No NMI, no irq disabled, no nothing.This shouldn't be a problem, right?
Not at all. The unwind interface already has a method to get a stack trace at any time, but it's up to the caller to make sure it's safe to do so.
See unwind_user_faultable()
I guess all we need to do is add a perf API to call that?
Quoted 9 lines
Quoted 7 lines
I've been Cc'ing the BPF list (and maintainers) on the deferred stack trace work and did get a few comments. But I haven't heard anything about BPF utilizing the feature yet. So I'm asking now if that's in the works?
If not, perhaps this would be a good topic to bring up at LSFMMBPF?
Yeah, we definitely can sync up on this during LSFMMBPF.
Awesome, I was looking for an excuse to go to Croatia ;-)
-- Steve
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Mon, 26 Jan 2026 16:02:10 -0800 Andrii Nakryiko <[email protected]>…
So what we will need from kernel's sframe support side is just an internal API to get stack trace, similar to
get_perf_callchain(). But it will be guaranteed to be called in a nice task context. No NMI, no irq disabled, no nothing.
Quoted 8 lines
Quoted 1 line
This shouldn't be a problem, right?
Not at all. The unwind interface already has a method to get a stack trace at any time, but it's up to the caller to make sure it's safe to do so.
See
unwind_user_faultable()I guess all we need to do is add a perf API to call that?
Yep, unwind_user_faultable() should work. There is that current->unwind_info use, so we should be careful about nested sleepable BPF programs, but I don't think that's possible right now.
I don't really think we need another API, we should be able to call into this directly. I'm not 100% sure on the best way to expose this through the BPF-specific API, but that's minor.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I've been Cc'ing the BPF list (and maintainers) on the deferred stack trace work and did get a few comments. But I haven't heard anything about BPF utilizing the feature yet. So I'm asking now if that's in the works?
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
If not, perhaps this would be a good topic to bring up at LSFMMBPF?
Yeah, we definitely can sync up on this during LSFMMBPF.
Awesome, I was looking for an excuse to go to Croatia ;-)
heh, you are welcome ;)
Quoted 2 lines
-- Steve
Hi Andrii,
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · On Mon, 26 Jan 2026 16:02:10 -0800 Andrii Nakryiko <[email protected]>…
So what we will need from kernel's sframe support side is just an internal API to get stack trace, similar to
get_perf_callchain(). But it will be guaranteed to be called in a nice task context. No NMI, no irq disabled, no nothing.This shouldn't be a problem, right?
I raised this concern with Steve last week, in the context of off-CPU profiling (e.g. https://www.brendangregg.com/offcpuanalysis.html). I maintain a BPF-based off-CPU profiler that runs out of the sched_switch tracepoint. It acquires both user and kernel stack traces via bpf_get_stackid(), stores their stack trace IDs in task-local storage until that task is rescheduled, and then uses those IDs as parts of a key into another BPF map to store the off-CPU data.
The context switch handler is of course not a safe place to block or page fault, so it wouldn't be able to acquire a synchronous stack trace anymore. (It's possible that my profiler is missing some user stack frames today due to some text being paged out, but presumably paged-out text isn't too performance-relevant anyway.)
So, as far as I can tell, my profiler would need to hook into this new task_work framework to acquire the user stack trace before storing the results into the final BPF map. But a profiled thread might block several times in the kernel before returning to userland - I saw one block 53 times, just looking through one day's profiling data - and I don't see a straightforward way to store data about an unbounded number of blocking events in task-local storage until the thread can wait for a user stack trace.
Do you have advice for how I could handle this under your new proposal?
Blake
Quoted 26 lines · Blake Jones
Hi Andrii,
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · On Mon, 26 Jan 2026 16:02:10 -0800 Andrii Nakryiko <[email protected]>…
So what we will need from kernel's sframe support side is just an internal API to get stack trace, similar to
get_perf_callchain(). But it will be guaranteed to be called in a nice task context. No NMI, no irq disabled, no nothing.This shouldn't be a problem, right?
I raised this concern with Steve last week, in the context of off-CPU profiling (e.g. https://www.brendangregg.com/offcpuanalysis.html). I maintain a BPF-based off-CPU profiler that runs out of the
sched_switchtracepoint. It acquires both user and kernel stack traces viabpf_get_stackid(), stores their stack trace IDs in task-local storage until that task is rescheduled, and then uses those IDs as parts of a key into another BPF map to store the off-CPU data.The context switch handler is of course not a safe place to block or page fault, so it wouldn't be able to acquire a synchronous stack trace anymore. (It's possible that my profiler is missing some user stack frames today due to some text being paged out, but presumably paged-out text isn't too performance-relevant anyway.)
Quoted 9 lines
So, as far as I can tell, my profiler would need to hook into this new
task_workframework to acquire the user stack trace before storing the results into the final BPF map. But a profiled thread might block several times in the kernel before returning to userland - I saw one block 53 times, just looking through one day's profiling data - and I don't see a straightforward way to store data about an unbounded number of blocking events in task-local storage until the thread can wait for a user stack trace.
I'm not quite clear on why blocking 53 times in the kernel before returning to user space matters for capturing user space stack trace? Given we haven't returned to user space, its stack trace will remain the same. You'll need to schedule task_work (see bpf_task_work_schedule_resume()) just once and capture stack trace just once. How you join that asynchronously (from the perspective of the point where you called bpf_task_work_schedule_resume()) captured stack traces will be up to you. You can send it through ringbuf, or store in some map, or whatever makes most sense for how exactly you structured your BPF logic.
This, of course, is not going to be as convenient as synchronous stackmap approach you are using with bpf_get_stackid() (or, similarly, bpf_get_stack() that doesn't rely on BPF map, but is still synchronous), but there is no way to avoid complications due to asynchronous nature of all of this new SFrame-based stack trace capture.
Quoted 3 lines
Do you have advice for how I could handle this under your new proposal?
Blake
Quoted 10 lines · Andrii Nakryiko
I'm not quite clear on why blocking 53 times in the kernel before returning to user space matters for capturing user space stack trace? Given we haven't returned to user space, its stack trace will remain the same. You'll need to schedule
task_work(seebpf_task_work_schedule_resume()) just once and capture stack trace just once. How you join that asynchronously (from the perspective of the point where you calledbpf_task_work_schedule_resume()) captured stack traces will be up to you. You can send it through ringbuf, or store in some map, or whatever makes most sense for how exactly you structured your BPF logic.
Yeah, it's exactly this async joining that I'm referring to.
If the thread blocked just once before returning to userland, I could capture a synchronous kernel stack trace and store its ID (along with the amount of time blocked there and the number of times it blocked there) in a simple structure in task-local storage, e.g.:
struct block_info {
__u32 kstackid;
__u64 sleep_nsec;
__u64 sleep_count;
};and then once the user stack trace ID had been captured via task_work, I could merge it along with the data from "block_info" into the final BPF map. The final BPF map's key is { kstackid, ustackid }, and its value is { sleep_nsec, sleep_count }.
But a thread that blocks multiple times before returning to userland would have to track multiple distinct kernel stack trace IDs and multiple distinct blocking durations before it can join them with the user stack trace ID. And the fixed-size structure used with task-local storage wouldn't be ... up to the task, as it were.
Blake
Quoted 27 lines · Blake Jones
Quoted 10 lines · Andrii Nakryiko
I'm not quite clear on why blocking 53 times in the kernel before returning to user space matters for capturing user space stack trace? Given we haven't returned to user space, its stack trace will remain the same. You'll need to schedule
task_work(seebpf_task_work_schedule_resume()) just once and capture stack trace just once. How you join that asynchronously (from the perspective of the point where you calledbpf_task_work_schedule_resume()) captured stack traces will be up to you. You can send it through ringbuf, or store in some map, or whatever makes most sense for how exactly you structured your BPF logic.Yeah, it's exactly this async joining that I'm referring to.
If the thread blocked just once before returning to userland, I could capture a synchronous kernel stack trace and store its ID (along with the amount of time blocked there and the number of times it blocked there) in a simple structure in task-local storage, e.g.:
struct block_info { __u32 kstackid; __u64 sleep_nsec; __u64 sleep_count; };
Quoted 4 lines
and then once the user stack trace ID had been captured via
task_work, I could merge it along with the data from "block_info" into the final BPF map. The final BPF map's key is { kstackid, ustackid }, and its value is {sleep_nsec,sleep_count}.
I'd probably pre-generate some unique ID (using atomic counters from the BPF side) that will be sent along with the async user stack trace eventually. And then join everything after the fact in user space.
Quoted 8 lines
But a thread that blocks multiple times before returning to userland would have to track multiple distinct kernel stack trace IDs and multiple distinct blocking durations before it can join them with the user stack trace ID. And the fixed-size structure used with task-local storage wouldn't be ... up to the task, as it were.
Blake
Hi Andrii,
Quoted 3 lines · Andrii Nakryiko
I'd probably pre-generate some unique ID (using atomic counters from the BPF side) that will be sent along with the async user stack trace eventually. And then join everything after the fact in user space.
I might be misunderstanding what you're proposing, but I don't see how a single unique ID would solve my problem.
If a thread executes the following kernel function foo():
void bar(object_t *o)
{
mutex_lock(&o->lock);
o->count++;
mutex_unlock(&o->lock);
}
void foo(void)
{
mutex_lock(¤t->foo_lock);
list_for_each_entry(object, ...) {
bar(object); }
mutex_unlock(¤t->foo_lock);
}
and blocked on both mutex acquisitions, it would generate two distinct kernel stack traces in my profiler, corresponding to the two calls to mutex_lock().
If I held onto the first stack trace ID in fixed-sized task-local storage until I could acquire the user stack trace, it (and its associated data) would get overwritten when it came time to take the second stack trace.
If I just used the kernel stack trace ID (and not the user stack trace ID) to index into a non-task-local BPF map, I wouldn't be able to distinguish data from different user stack traces.
If I stored this data in a log, adding the user stack trace ID when it was available, and aggregated the log in userspace, my profiler's memory needs would go up by 50-100x, which would make it unusable.
Could you help me understand what you're proposing?
Thanks.
Blake
Quoted 43 lines · Blake Jones
Hi Andrii,
Quoted 3 lines · Andrii Nakryiko
I'd probably pre-generate some unique ID (using atomic counters from the BPF side) that will be sent along with the async user stack trace eventually. And then join everything after the fact in user space.
I might be misunderstanding what you're proposing, but I don't see how a single unique ID would solve my problem.
If a thread executes the following kernel function
foo():void
bar(object_t *o)
{
mutex_lock(&o->lock);
o->count++;
mutex_unlock(&o->lock);
}void foo(void)
{
mutex_lock(¤t->foo_lock);list_for_each_entry(object, ...) { bar(object);}
mutex_unlock(¤t->foo_lock);
}and blocked on both mutex acquisitions, it would generate two distinct kernel stack traces in my profiler, corresponding to the two calls to
mutex_lock().If I held onto the first stack trace ID in fixed-sized task-local storage until I could acquire the user stack trace, it (and its associated data) would get overwritten when it came time to take the second stack trace.
If I just used the kernel stack trace ID (and not the user stack trace ID) to index into a non-task-local BPF map, I wouldn't be able to distinguish data from different user stack traces.
Quoted 6 lines
If I stored this data in a log, adding the user stack trace ID when it was available, and aggregated the log in userspace, my profiler's memory needs would go up by 50-100x, which would make it unusable.
Could you help me understand what you're proposing?
I'm not really proposing anything. You have a baked-in assumption that you will have a user stack available *synchronously* and that you can use that as some sort of lookup key. This is simply not true with SFrame (and asynchronously captured user space stack traces in general), so you'll have to redesign your approach (or you just say that you only support frame pointer-based synchronous stack traces).
Quoted 3 lines
Thanks.
Blake
Quoted 6 lines · On Fri, 30 Jan 2026 11:50:16 -0800 Andrii Nakryiko <[email protected]>…
I'm not really proposing anything. You have a baked-in assumption that you will have a user stack available *synchronously* and that you can use that as some sort of lookup key. This is simply not true with SFrame (and asynchronously captured user space stack traces in general), so you'll have to redesign your approach (or you just say that you only support frame pointer-based synchronous stack traces).
From my understanding of what Blake's tools are doing is that it records stacktraces (both kernel and user space) when a task schedules out and for how long. This shows how long tasks were off the CPU due to being blocked, or sleeping, and even preemption.
Currently, when a task schedules out and comes back on the CPU, the kernel stack is combined with the user stack and it's put into a hash table where the time off the CPU is added to the hash entry's time. I'm not sure of the details here, but this could be updating BPF maps and such here.
The issue that Blake has is that this cannot be recorded when waking up because only the kernel's stack is known. The user stack isn't known yet.
One solution we discussed was to make the hash key of the kernel tasks and save it in a small array that also stores the time it was off the kernel. If the task schedules out in the same location multiple times before going back to user space, only one entry needs to be updated as the hash of the kernel stack would be the same. When the task goes back to user space, the kernel key is attached to the user stack trace hash key and added to the main table that associates both the kernel and user stacktrace.
But if the task were to schedule out in many locations (Blake has recorded over 50 unique locations where a task could schedule out in the same system call), then it would need to save the time off the CPU for each of those locations. When going back to user space, each of those locations would need to be attached to the current user stacktrace hash key and added to the main table individually.
Note, these temporary kernel hash keys are unique per task, as they haven't been associated with a user stacktrace yet, whereas the global table is for the entire system. That means any storage of these temporary kernel stacktrace keys needs to be done per task.
Blake, did I explain that correctly?
Now the question is, is there a way to easily store these unique kernel stacktraces along with the time the task was off the CPU for those locations and be able to attach them each to the user stacktrace when going back to user space?
-- Steve
Quoted 1 line · Steven Rostedt
Blake, did I explain that correctly?
Yep Steve, your explanation sounds correct to me. If there was a way of storing a non-constant amount of metadata in something like task-local storage, I would have a solution.
What I'm hearing from Andrii is that fixing this is my problem - that "user stack traces can generally be taken synchronously from arbitrary places in the kernel" should be thought of as an implementation detail, not an interface. To me, this seems like an example of
https://www.hyrumslaw.com/ - this has been reliably possible for decades (modulo paged-out text, which again is probably not performance-relevant), and so I do think of it as an interface. When I'm making interface-breaking (or even "interface"-breaking) changes, I view it as my responsibility to try to help out consumers that are broken by it.
So again, I'd love to get some ideas for how I can make this profiler work under SFrames while still aggregating data in the kernel. I strongly suspect that mine is not the only profiler that will be broken by this change.
Blake
Quoted 7 lines · Blake Jones
Quoted 1 line · Steven Rostedt
Blake, did I explain that correctly?
Yep Steve, your explanation sounds correct to me. If there was a way of storing a non-constant amount of metadata in something like task-local storage, I would have a solution.
There isn't (AFAIA), but you can approximate that with a global hashmap keyed by task's pid + whatever the original per-task key you needed.
Quoted 2 lines
What I'm hearing from Andrii is that fixing this is my problem - that
bpf_get_stackid() and bpf_get_stack() are working and will keep working as before. Which, for user space stack traces, means capturing stack traces in a best effort way based on the frame pointer-based unwinding approach (assuming the user space program was compiled with -fno-omit-frame-pointer). If it wasn't, or there are some function calls not preserving frame pointers, those BPF helpers will return garbage or partial stack traces.
Nothing changes about that.
Quoted 5 lines
"user stack traces can generally be taken synchronously from arbitrary places in the kernel" should be thought of as an implementation detail, not an interface. To me, this seems like an example of
https://www.hyrumslaw.com/ - this has been reliably possible for decades (modulo paged-out text, which again is probably not
This has never been reliably possible if you didn't have frame pointers preserved. Are we talking past each other?
Quoted 2 lines
performance-relevant), and so I do think of it as an interface. When I'm making interface-breaking (or even "interface"-breaking) changes,
Point me to "interface breakage", please.
Quoted 2 lines
I view it as my responsibility to try to help out consumers that are broken by it.
Who and how is breaking you by trying to support asynchronous SFrame-based stack traces? Do we remove bpf_get_stackid() or bpf_get_stack()? Do we change how they work?
Or breakage for you is that bpf_get_stack[id]() can't magically support SFrame stack trace capture **synchronously**? SFrame requires paging in file contents, which means page faulting, which means it cannot be supported (not reliably in any case) synchronously from non-sleepable BPF program types (which pretty much means any tracing program type).
Quoted 5 lines
So again, I'd love to get some ideas for how I can make this profiler work under SFrames while still aggregating data in the kernel. I strongly suspect that mine is not the only profiler that will be broken by this change.
Of course existing profilers will have to come up with ways to support **asynchronous** stack trace capture. How to do it will depend on exact data flow and might be completely different in each case. There is no universal answer, and some models (like those expecting to have user stack trace as lookup key in any context) are just incompatible and will have to be rethought.
Quoted 2 lines
Blake
Hi Andrii,
Quoted 7 lines · Andrii Nakryiko
Quoted 3 lines
Yep Steve, your explanation sounds correct to me. If there was a way of storing a non-constant amount of metadata in something like task-local storage, I would have a solution.
There isn't (AFAIA), but you can approximate that with a global hashmap keyed by task's pid + whatever the original per-task key you needed.
My original per-task key did include the kernel stack trace ID, as stored in a global hash map by bpf_get_stackid(); if I understood you right, that's what you suggested in your reply to Steve as well. But I don't see how the task_work handler that gets the user stack trace would be able to find any hash map entries stored under that key, since it would be operating from a different location (and thus also from different stack traces) than the profiler. Could you help me understand this, please?
Regarding the rest of the discussion - what I'm hearing is that the need to handle asynchronous user stack traces is just part and parcel of any decision to switch to SFrames, and it's up to each user to decide how that cost/benefit tradeoff works for them. That makes sense to me.
Blake
Quoted 13 lines · Blake Jones
Hi Andrii,
Quoted 7 lines · Andrii Nakryiko
Quoted 3 lines
Yep Steve, your explanation sounds correct to me. If there was a way of storing a non-constant amount of metadata in something like task-local storage, I would have a solution.
There isn't (AFAIA), but you can approximate that with a global hashmap keyed by task's pid + whatever the original per-task key you needed.
Quoted 8 lines
My original per-task key did include the kernel stack trace ID, as stored in a global hash map by
bpf_get_stackid(); if I understood you right, that's what you suggested in your reply to Steve as well. But I don't see how thetask_workhandler that gets the user stack trace would be able to find any hash map entries stored under that key, since it would be operating from a different location (and thus also from different stack traces) than the profiler. Could you help me understand this, please?
Ok, I think I see the confusion. You need to look up how task_work (and, similarly, timer and wq callbacks) work from BPF usage perspective. You basically get your custom context data passed to you in the same map value, so in this case you can have that entire hashmap value as your context by just embedding task_work field into your hashmap value. You will be given that same exact key and value, so you don't even need to look up anything.
Just look at the BPF selftest for examples (grep for bpf_task_work_schedule_resume), note what the callback gets passed in (map+key+value).
Quoted 6 lines
Regarding the rest of the discussion - what I'm hearing is that the need to handle asynchronous user stack traces is just part and parcel of any decision to switch to SFrames, and it's up to each user to decide how that cost/benefit tradeoff works for them. That makes sense to me.
yep, pretty much
Quoted 2 lines
Blake
Quoted 44 lines · Steven Rostedt
Quoted 6 lines · On Fri, 30 Jan 2026 11:50:16 -0800 Andrii Nakryiko <[email protected]>…
I'm not really proposing anything. You have a baked-in assumption that you will have a user stack available *synchronously* and that you can use that as some sort of lookup key. This is simply not true with SFrame (and asynchronously captured user space stack traces in general), so you'll have to redesign your approach (or you just say that you only support frame pointer-based synchronous stack traces).
From my understanding of what Blake's tools are doing is that it records stacktraces (both kernel and user space) when a task schedules out and for how long. This shows how long tasks were off the CPU due to being blocked, or sleeping, and even preemption.
Currently, when a task schedules out and comes back on the CPU, the kernel stack is combined with the user stack and it's put into a hash table where the time off the CPU is added to the hash entry's time. I'm not sure of the details here, but this could be updating BPF maps and such here.
The issue that Blake has is that this cannot be recorded when waking up because only the kernel's stack is known. The user stack isn't known yet.
One solution we discussed was to make the hash key of the kernel tasks and save it in a small array that also stores the time it was off the kernel. If the task schedules out in the same location multiple times before going back to user space, only one entry needs to be updated as the hash of the kernel stack would be the same. When the task goes back to user space, the kernel key is attached to the user stack trace hash key and added to the main table that associates both the kernel and user stacktrace.
But if the task were to schedule out in many locations (Blake has recorded over 50 unique locations where a task could schedule out in the same system call), then it would need to save the time off the CPU for each of those locations. When going back to user space, each of those locations would need to be attached to the current user stacktrace hash key and added to the main table individually.
Note, these temporary kernel hash keys are unique per task, as they haven't been associated with a user stacktrace yet, whereas the global table is for the entire system. That means any storage of these temporary kernel stacktrace keys needs to be done per task.
Quoted 6 lines
Blake, did I explain that correctly?
Now the question is, is there a way to easily store these unique kernel stacktraces along with the time the task was off the CPU for those locations and be able to attach them each to the user stacktrace when going back to user space?
You can use a global hashtable keyed by pid + kernel stack (either kernel stack id as returned by bpf_get_stackid(); or just embedding the entire stack trace as returned by bpf_get_stack()). You'd need to pre-size your hashtable for total number of unique kernel stack traces across all threads, so I'm not saying it's a perfect solution, but as anything with BPF, there are some trade offs to consider.
The important part here is that the kernel stack trace can be retrieved synchronously, so you'll have its contents right where you request it.
How to join this with user stack trace (if user stack trace will be known later) is up to the implementer, there is no general best way. Pre-generating some ID is one way.
Let me address the rest in reply to Blake's email.
Quoted 2 lines
-- Steve