Skip to contentSkip to conversation

arm64: Add BRBE support for bpf_get_branch_snapshot()

PATCH Superseded Puranjay Mohan 6 participants 15 messages Mar 18 – Apr 13

Subsystems (5)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover arm64: Add BRBE support for bpf_get_branch_snapshot()

v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Rebased on arm64/for-next/core
- Add per-CPU brbe_active flag to guard against UNDEFINED sysreg access on non-BRBE CPUs in heterogeneous big.LITTLE systems.
- Fix pre-existing bug in perf_clear_branch_entry_bitfields() that missed zeroing new_type and priv bitfields, added as a separate patch with Fixes tags (new patch 2).
- Use architecture-specific selftest threshold (#if defined(__aarch64__)) instead of raising the global threshold, to preserve x86 regression detection.

RFC: https://lore.kernel.org/all/[email protected]/
Changes from RFC:
- Fix pre-existing NULL pointer dereference in armv8pmu_sched_task() found by Leo Yan during testing (patch 1)
- Pause BRBE before local_daif_save() to avoid branch pollution from trace_hardirqs_off()
- Use local_daif_save() to prevent pNMI race from counter overflow (Mark Rutland)
- Reuse perf_entry_from_brbe_regset() instead of duplicating register read logic, by making it accept NULL event (Mark Rutland)
- Invalidate BRBE after reading to maintain record contiguity for other consumers (Mark Rutland)
- Adjust selftest wasted_entries threshold for ARM64 (patch 3)
- Tested on ARM FVP with BRBE enabled

This series enables the bpf_get_branch_snapshot() BPF helper on ARM64 by implementing the perf_snapshot_branch_stack static call for ARM's Branch Record Buffer Extension (BRBE).

bpf_get_branch_snapshot() [1] allows BPF programs to capture hardware branch records on-demand from any BPF tracing context. This was previously only available on x86 (Intel LBR) since v5.16. With BRBE available on ARMv9, this series closes the gap for ARM64.

Usage model
-----------

The helper works in conjunction with perf events. The userspace component of the BPF application opens a perf event with PERF_SAMPLE_BRANCH_STACK on each CPU, which configures the hardware to continuously record branches into BRBE (on ARM64) or LBR (on x86). A BPF program attached to a tracepoint, kprobe, or fentry hook can then call bpf_get_branch_snapshot() to snapshot the branch buffer at any point. Without an active perf event, BRBE is not recording and the buffer is empty.

On-demand branch snapshots from BPF are useful for diagnosing which specific code path was taken inside a function. Stack traces only show function boundaries, but branch records reveal the exact sequence of jumps, calls, and returns within a function -- making it possible to identify which specific error check triggered a failure, or which callback implementation was invoked through a function pointer.

For example, retsnoop [2] is a BPF-based tool for non-intrusive mass-tracing of kernel internals. Its LBR mode (--lbr) creates per-CPU perf events with PERF_SAMPLE_BRANCH_STACK and then uses bpf_get_branch_snapshot() in its fentry/fexit BPF programs to capture branch records whenever a traced function returns an error.

Consider debugging a bpf() syscall that returns -EINVAL when creating a BPF map with invalid parameters. Running retsnoop on an ARM64 FVP with BRBE to trace the bpf() syscall and array_map_alloc_check():

  $ retsnoop -e '*sys_bpf' -a 'array_map_alloc_check' --lbr=any \
             -F -k vmlinux --debug full-lbr
  $ simfail bpf-bad-map-max-entries-array  # in another terminal

Output of retsnoop:

--- fentry BPF program (entries #63-#17) ---

[#63-#59] __htab_map_lookup_elem: hash table walk with memcmp (hashtab.c)
[#58] __htab_map_lookup_elem+0x98 -> dump_bpf_prog+0xc850 (hashtab.c:750)
[#57-#55] ... dump_bpf_prog internal branches ...
[#54] dump_bpf_prog+0xcab8 -> bpf_get_current_pid_tgid+0x0 (helpers.c:225)
[#53] bpf_get_current_pid_tgid+0x1c -> dump_bpf_prog+0xcabc (helpers.c:225)
[#52-#51] ... dump_bpf_prog -> __htab_map_lookup_elem ... [#50-#47] __htab_map_lookup_elem: htab_map_hash (jhash2), select_bucket
[#46-#42] lookup_nulls_elem_raw: hash chain walk with memcmp (hashtab.c:717)
[#41] __htab_map_lookup_elem+0x98 -> dump_bpf_prog+0xcaf8 (hashtab.c:750)
[#40-#37] ... dump_bpf_prog -> bpf_ktime_get_ns ...
[#36] bpf_ktime_get_ns+0x10 -> ktime_get_mono_fast_ns+0x0 (helpers.c:178)
[#35-#32] ktime_get_mono_fast_ns: tk_clock_read -> arch_counter_get_cntpct
[#31] ktime_get_mono_fast_ns+0x9c -> bpf_ktime_get_ns+0x14 (timekeeping.c:493)
[#30] bpf_ktime_get_ns+0x18 -> dump_bpf_prog+0xcd50 (helpers.c:178)
[#29-#25] ... dump_bpf_prog internal branches ...
[#24] dump_bpf_prog+0x11b28 -> __bpf_prog_exit_recur+0x0 (trampoline.c:1190)
[#23-#17] __bpf_prog_exit_recur: rcu_read_unlock, migrate_enable (trampoline.c:1195)

--- array_map_alloc_check (entries #16-#12) ---

  [#16] dump_bpf_prog+0x11b38       -> array_map_alloc_check+0x8       (arraymap.c:55)
  [#15] array_map_alloc_check+0x18  -> array_map_alloc_check+0xb8      (arraymap.c:56)
        . bpf_map_attr_numa_node       . bpf_map_attr_numa_node
  [#14] array_map_alloc_check+0xbc  -> array_map_alloc_check+0x20      (arraymap.c:59)
        . bpf_map_attr_numa_node
  [#13] array_map_alloc_check+0x24  -> array_map_alloc_check+0x94      (arraymap.c:64)
  [#12] array_map_alloc_check+0x98  -> dump_bpf_prog+0x11b3c           (arraymap.c:82)

--- fexit trampoline overhead (entries #11-#00) ---

  [#11] dump_bpf_prog+0x11b5c       -> __bpf_prog_enter_recur+0x0      (trampoline.c:1145)
  [#10-#03] __bpf_prog_enter_recur: rcu_read_lock, migrate_disable     (trampoline.c:1146)
  [#02] __bpf_prog_enter_recur+0x114 -> dump_bpf_prog+0x11b60          (trampoline.c:1157)

[#01] dump_bpf_prog+0x11b6c -> dump_bpf_prog+0xd230
[#00] dump_bpf_prog+0xd340 -> arm_brbe_snapshot_branch_stack+0x0 (arm_brbe.c:814)

                   el0t_64_sync+0x168
                   el0t_64_sync_handler+0x98
                   el0_svc+0x28
                   do_el0_svc+0x4c
                   invoke_syscall.constprop.0+0x54
    373us [-EINVAL] __arm64_sys_bpf+0x8
                    __sys_bpf+0x87c
                    map_create+0x120
     95us [-EINVAL] array_map_alloc_check+0x8

The FVP's BRBE buffer has 64 entries (BRBE supports 8, 16, 32, or 64). Of these, entries #63-#17 (47) are consumed by the fentry BPF trampoline that ran before the function, and entries #11-#00 (12) are consumed by the fexit trampoline that runs after. Entry #00 shows the very last branch recorded before BRBE is paused: the call into arm_brbe_snapshot_branch_stack().

The 5 useful entries (#16-#12) show the exact path taken inside array_map_alloc_check(). Record #14 shows a jump from line 56 (bpf_map_attr_numa_node) to line 59 (the if-condition), and #13 shows an immediate jump from line 59 (attr->max_entries == 0) to line 64 (return -EINVAL), skipping lines 60-63. This pinpoints max_entries==0 as the cause -- a diagnosis impossible with stack traces alone.

[1] 856c02dbce4f ("bpf: Introduce helper bpf_get_branch_snapshot")
[2] https://github.com/anakryiko/retsnoop

Puranjay Mohan (4):
perf/arm_pmuv3: Fix NULL pointer dereference in armv8pmu_sched_task()
perf: Fix uninitialized bitfields in
perf_clear_branch_entry_bitfields()
perf/arm64: Add BRBE support for bpf_get_branch_snapshot() selftests/bpf: Adjust wasted entries threshold for ARM64 BRBE

 drivers/perf/arm_brbe.c                       | 79 ++++++++++++++++++-
 drivers/perf/arm_brbe.h                       |  9 +++
 drivers/perf/arm_pmuv3.c                      | 16 +++-
 include/linux/perf_event.h                    |  2 +
 .../bpf/prog_tests/get_branch_snapshot.c      | 13 ++-
 5 files changed, 110 insertions(+), 9 deletions(-)

base-commit: d118f32246fdabfb4f6a3fd2e511dc5e622bc553

Signature · 1 line
2.52.0
#1 lore raw

PATCH v2 1/4 perf/arm_pmuv3: Fix NULL pointer dereference in armv8pmu_sched_task()

This is easily triggered with:

perf record -b -e cycles -a -- ls

which crashes on the first context switch with:

  Unable to handle kernel NULL pointer dereference at virtual address 00[.]
  PC is at armv8pmu_sched_task+0x14/0x50
  LR is at perf_pmu_sched_task+0xac/0x108
  Call trace:
    armv8pmu_sched_task+0x14/0x50 (P)
    perf_pmu_sched_task+0xac/0x108
    __perf_event_task_sched_out+0x6c/0xe0
    prepare_task_switch+0x120/0x268
    __schedule+0x1e8/0x828
    ...

perf_pmu_sched_task() invokes the PMU sched callback with cpc->task_epc, which is NULL when no per-task events exist for this PMU. With CPU-wide branch-stack events, armv8pmu_sched_task() is still registered and dereferences pmu_ctx->pmu unconditionally, causing the crash.

The bug was introduced by commit fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer") which changed the function from using the per-CPU cpu_armpmu pointer (always valid) to dereferencing pmu_ctx->pmu without adding a NULL check.

Add a NULL check for pmu_ctx to avoid the crash.

Fixes fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer")
---
 drivers/perf/arm_pmuv3.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/perf/arm_pmuv3.c +9 -2
index 8014ff766cff..2d097fad9c10 100644
@@ -1074,8 +1074,15 @@ static int armv8pmu_user_event_idx(struct perf_event *event)
 static void armv8pmu_sched_task(struct perf_event_pmu_context *pmu_ctx,
 				struct task_struct *task, bool sched_in)
 {
-	struct arm_pmu *armpmu = to_arm_pmu(pmu_ctx->pmu);
-	struct pmu_hw_events *hw_events = this_cpu_ptr(armpmu->hw_events);
+	struct arm_pmu *armpmu;
+	struct pmu_hw_events *hw_events;
+
+	/* cpc->task_epc is NULL when no per-task events exist for this PMU */
+	if (!pmu_ctx)
+		return;
+
+	armpmu = to_arm_pmu(pmu_ctx->pmu);
+	hw_events = this_cpu_ptr(armpmu->hw_events);
 
 	if (!hw_events->branch_users)
 		return;
Signature · 1 line
2.52.0
Acked-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] perf/arm_pmuv3: Fix NULL pointer dereference in armv8pmu_sched_task()

Quoted 27 lines · On Wed, 18 Mar 2026 10:16:55 -0700 Puranjay Mohan <[email protected]> wrote:

This is easily triggered with:

perf record -b -e cycles -a -- ls

which crashes on the first context switch with:

  Unable to handle kernel NULL pointer dereference at virtual address 00[.]
  PC is at armv8pmu_sched_task+0x14/0x50
  LR is at perf_pmu_sched_task+0xac/0x108
  Call trace:
    armv8pmu_sched_task+0x14/0x50 (P)
    perf_pmu_sched_task+0xac/0x108
    __perf_event_task_sched_out+0x6c/0xe0
    prepare_task_switch+0x120/0x268
    __schedule+0x1e8/0x828
    ...

perf_pmu_sched_task() invokes the PMU sched callback with cpc->task_epc, which is NULL when no per-task events exist for this PMU. With CPU-wide branch-stack events, armv8pmu_sched_task() is still registered and dereferences pmu_ctx->pmu unconditionally, causing the crash.

The bug was introduced by commit fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer") which changed the function from using the per-CPU cpu_armpmu pointer (always valid) to dereferencing pmu_ctx->pmu without adding a NULL check.

Quoted 8 lines

Add a NULL check for pmu_ctx to avoid the crash.

Fixes fa9d27773873 ("perf: arm_pmu: Kill last use of per-CPU cpu_armpmu pointer")
---
 drivers/perf/arm_pmuv3.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
#3 lore raw

Re: [PATCH v2 1/4] perf/arm_pmuv3: Fix NULL pointer dereference in armv8pmu_sched_task()

In reply to Puranjay Mohan

Hi Puranjay,

Quoted 17 lines · Puranjay Mohan

This is easily triggered with:

perf record -b -e cycles -a -- ls

which crashes on the first context switch with:

  Unable to handle kernel NULL pointer dereference at virtual address 00[.]
  PC is at armv8pmu_sched_task+0x14/0x50
  LR is at perf_pmu_sched_task+0xac/0x108
  Call trace:
    armv8pmu_sched_task+0x14/0x50 (P)
    perf_pmu_sched_task+0xac/0x108
    __perf_event_task_sched_out+0x6c/0xe0
    prepare_task_switch+0x120/0x268
    __schedule+0x1e8/0x828
    ...
Quoted 4 lines

perf_pmu_sched_task() invokes the PMU sched callback with cpc->task_epc, which is NULL when no per-task events exist for this PMU. With CPU-wide branch-stack events, armv8pmu_sched_task() is still registered and dereferences pmu_ctx->pmu unconditionally, causing the crash.

Would not this is a bug in perf core layer that it should properly pass &cpc->epc for CPU wide trace?

kernel/events/core.c +2 -1
index 89b40e439717..a8cd0a26effa 100644
@@ -3906,7 +3906,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
        perf_ctx_lock(cpuctx, cpuctx->task_ctx);
        perf_pmu_disable(pmu);
 
-       pmu->sched_task(cpc->task_epc, task, sched_in);
+       pmu->sched_task(cpc->task_epc ? cpc->task_epc : &cpc->epc,
+                       task, sched_in);
 
        perf_pmu_enable(pmu);
        perf_ctx_unlock(cpuctx, cpuctx->task_ctx);

I checked other archs, most of them don't use the parameter *pmu_ctx so the issue never hits.

My understanding is that for CPU-wide (or system-wide) tracing, we still need to invalidate the branch record on a task switch so that a branch stack is recorded within the same context.

Thanks,
Leo

#4 lore raw

Re: [PATCH v2 1/4] perf/arm_pmuv3: Fix NULL pointer dereference in armv8pmu_sched_task()

Quoted 45 lines · Leo Yan

Hi Puranjay,

Quoted 17 lines · Puranjay Mohan

This is easily triggered with:

perf record -b -e cycles -a -- ls

which crashes on the first context switch with:

  Unable to handle kernel NULL pointer dereference at virtual address 00[.]
  PC is at armv8pmu_sched_task+0x14/0x50
  LR is at perf_pmu_sched_task+0xac/0x108
  Call trace:
    armv8pmu_sched_task+0x14/0x50 (P)
    perf_pmu_sched_task+0xac/0x108
    __perf_event_task_sched_out+0x6c/0xe0
    prepare_task_switch+0x120/0x268
    __schedule+0x1e8/0x828
    ...
Quoted 4 lines

perf_pmu_sched_task() invokes the PMU sched callback with cpc->task_epc, which is NULL when no per-task events exist for this PMU. With CPU-wide branch-stack events, armv8pmu_sched_task() is still registered and dereferences pmu_ctx->pmu unconditionally, causing the crash.

Would not this is a bug in perf core layer that it should properly pass &cpc->epc for CPU wide trace?

kernel/events/core.c +2 -1
index 89b40e439717..a8cd0a26effa 100644
@@ -3906,7 +3906,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
        perf_ctx_lock(cpuctx, cpuctx->task_ctx);
        perf_pmu_disable(pmu);
 
-       pmu->sched_task(cpc->task_epc, task, sched_in);
+       pmu->sched_task(cpc->task_epc ? cpc->task_epc : &cpc->epc,
+                       task, sched_in);
 
        perf_pmu_enable(pmu);
        perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
Quoted 6 lines

I checked other archs, most of them don't use the parameter *pmu_ctx so the issue never hits.

My understanding is that for CPU-wide (or system-wide) tracing, we still need to invalidate the branch record on a task switch so that a branch stack is recorded within the same context.

Thanks for your review, and I agree with your reasoning, I will change the patch to use your diff.

Puranjay

#5 lore raw

PATCH v2 2/4 perf: Fix uninitialized bitfields in perf_clear_branch_entry_bitfields()

In reply to Puranjay Mohan

perf_clear_branch_entry_bitfields() zeroes individual bitfields of struct perf_branch_entry but misses the new_type (4 bits) and priv (3 bits) fields. This means any code path that relies on this function to produce a clean entry may expose stale or uninitialised data in these fields to userspace.

The function was introduced by commit bfe4daf850f4 ("perf/core: Add perf_clear_branch_entry_bitfields() helper") specifically to "centralize the initialization to avoid missing a field in case more are added." Unfortunately, the commits that later added new_type and priv to struct perf_branch_entry only updated the UAPI header and did not update this clearing function.

Zero new_type and priv alongside the other bitfields.

Fixes b190bc4ac9e6 ("perf: Extend branch type classification")
Fixes 5402d25aa571 ("perf: Capture branch privilege information")
---
 include/linux/perf_event.h | 2 ++
 1 file changed, 2 insertions(+)
include/linux/perf_event.h +2 -0
index 48d851fbd8ea..d7f39b7e9cda 100644
@@ -1481,6 +1481,8 @@ static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *b
 	br->cycles	= 0;
 	br->type	= 0;
 	br->spec	= PERF_BR_SPEC_NA;
+	br->new_type	= 0;
+	br->priv	= 0;
 	br->reserved	= 0;
 }
 
Signature · 1 line
2.52.0
#6 lore raw

Re: [PATCH v2 2/4] perf: Fix uninitialized bitfields in perf_clear_branch_entry_bitfields()

Quoted 27 lines · Puranjay Mohan

perf_clear_branch_entry_bitfields() zeroes individual bitfields of struct perf_branch_entry but misses the new_type (4 bits) and priv (3 bits) fields. This means any code path that relies on this function to produce a clean entry may expose stale or uninitialised data in these fields to userspace.

The function was introduced by commit bfe4daf850f4 ("perf/core: Add perf_clear_branch_entry_bitfields() helper") specifically to "centralize the initialization to avoid missing a field in case more are added." Unfortunately, the commits that later added new_type and priv to struct perf_branch_entry only updated the UAPI header and did not update this clearing function.

Zero new_type and priv alongside the other bitfields.

Fixes b190bc4ac9e6 ("perf: Extend branch type classification")
Fixes 5402d25aa571 ("perf: Capture branch privilege information")
---
 include/linux/perf_event.h | 2 ++
 1 file changed, 2 insertions(+)
include/linux/perf_event.h +0 -0
index 48d851fbd8ea..d7f39b7e9cda 100644

include/linux/perf_event.h static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *b · line 1487

Quoted 7 lines
include/linux/perf_event.h +2 -0
@@ -1481,6 +1481,8 @@ static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *b
 	br->cycles	= 0;
 	br->type	= 0;
 	br->spec	= PERF_BR_SPEC_NA;
+	br->new_type	= 0;
+	br->priv	= 0;
 	br->reserved	= 0;
 }

We already know this does not work well. Instead, we can define a union for bitfield and use memset to clear it, later we will not bother for this kind of issue anymore.

  struct perf_branch_entry {
          ...
          union {
                  struct {
                          __u64   mispred   :  1, /* target mispredicted */
                                  predicted :  1, /* target predicted */
                                  ...
                                  reserved  : 31;
                  };
                  __u64 bitfields;
          };

};

static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *br)

{
        memset(&br->bitfields, 0, sizeof(br->bitfields));
}

Thanks,
Leo

#7 lore raw

Re: [PATCH v2 2/4] perf: Fix uninitialized bitfields in perf_clear_branch_entry_bitfields()

In reply to Puranjay Mohan

On Wed, Mar 18, 2026 at 10:16:56AM -0700, Puranjay Mohan wrote:

[...]

include/linux/perf_event.h

Quoted 4 lines
include/linux/perf_event.h +0 -0
index 48d851fbd8ea..d7f39b7e9cda 100644

BTW, I found you didn't include correct maintainers for perf core changes. You might need to loop them in the new version.

#8 lore raw

Re: [PATCH v2 2/4] perf: Fix uninitialized bitfields in perf_clear_branch_entry_bitfields()

Quoted 11 lines · Leo Yan

On Wed, Mar 18, 2026 at 10:16:56AM -0700, Puranjay Mohan wrote:

[...]

include/linux/perf_event.h

Quoted 4 lines
include/linux/perf_event.h +0 -0
index 48d851fbd8ea..d7f39b7e9cda 100644

BTW, I found you didn't include correct maintainers for perf core changes. You might need to loop them in the new version.

Yes. And use get_maintainers.pl please. b4 tool will do that for you. That will get everyone including authors of commits in "Fixes". That would be Marc Z in this case.

Rob

#9 lore raw

PATCH v2 3/4 perf/arm64: Add BRBE support for bpf_get_branch_snapshot()

In reply to Puranjay Mohan

Enable the bpf_get_branch_snapshot() BPF helper on ARM64 by implementing the perf_snapshot_branch_stack static call for ARM's Branch Record Buffer Extension (BRBE).

The BPF helper bpf_get_branch_snapshot() allows BPF programs to capture hardware branch records on-demand. This was previously only available on x86 (Intel LBR) but not on ARM64 despite BRBE being available since ARMv9.

BRBE is paused before disabling interrupts because local_irq_save() can trigger trace_hardirqs_off() which performs stack walking and pollutes the branch buffer. The sysreg read/write and ISB used to pause BRBE are branchless, so pausing first avoids this pollution.

All exceptions are masked after pausing BRBE using local_daif_save() to prevent pseudo-NMI from PMU counter overflow from interfering with the snapshot read. A PMU overflow arriving between the pause and local_daif_save() can re-enable BRBE via the interrupt handler; the snapshot detects this by re-checking BRBFCR_EL1.PAUSED and bailing out.

Branch records are read using the existing perf_entry_from_brbe_regset() helper with a NULL event pointer, which bypasses event-specific filtering and captures all recorded branches. The BPF program is responsible for filtering entries based on its own criteria. The BRBE buffer is invalidated after reading to maintain contiguity for other consumers.

On heterogeneous big.LITTLE systems, only some CPUs may implement FEAT_BRBE. The perf_snapshot_branch_stack static call is system-wide, so a per-CPU brbe_active flag is used to prevent BRBE sysreg access on CPUs that do not implement FEAT_BRBE, where such access would be UNDEFINED.

---
 drivers/perf/arm_brbe.c  | 79 +++++++++++++++++++++++++++++++++++++++-
 drivers/perf/arm_brbe.h  |  9 +++++
 drivers/perf/arm_pmuv3.c |  5 ++-
 3 files changed, 90 insertions(+), 3 deletions(-)
drivers/perf/arm_brbe.c +77 -2
index ba554e0c846c..527c2d5ebba6 100644
@@ -8,9 +8,13 @@
  */
 #include <linux/types.h>
 #include <linux/bitmap.h>
+#include <linux/percpu.h>
 #include <linux/perf/arm_pmu.h>
+#include <asm/daifflags.h>
 #include "arm_brbe.h"
 
+static DEFINE_PER_CPU(bool, brbe_active);
+
 #define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT   | \
 				   BRBFCR_EL1_INDIRECT | \
 				   BRBFCR_EL1_RTN      | \
@@ -533,6 +537,8 @@ void brbe_enable(const struct arm_pmu *arm_pmu)
 	/* Finally write SYS_BRBFCR_EL to unpause BRBE */
 	write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
 	/* Synchronization in PMCR write ensures ordering WRT PMU enabling */
+
+	this_cpu_write(brbe_active, true);
 }
 
 void brbe_disable(void)
@@ -544,6 +550,7 @@ void brbe_disable(void)
 	 */
 	write_sysreg_s(BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
 	write_sysreg_s(0, SYS_BRBCR_EL1);
+	this_cpu_write(brbe_active, false);
 }
 
 static const int brbe_type_to_perf_type_map[BRBINFx_EL1_TYPE_DEBUG_EXIT + 1][2] = {
@@ -618,10 +625,10 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 
 	brbe_set_perf_entry_type(entry, brbinf);
 
-	if (!branch_sample_no_cycles(event))
+	if (!event || !branch_sample_no_cycles(event))
 		entry->cycles = brbinf_get_cycles(brbinf);
 
-	if (!branch_sample_no_flags(event)) {
+	if (!event || !branch_sample_no_flags(event)) {
 		/* Mispredict info is available for source only and complete branch records. */
 		if (!brbe_record_is_target_only(brbinf)) {
 			entry->mispred = brbinf_get_mispredict(brbinf);
@@ -803,3 +810,71 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 done:
 	branch_stack->nr = nr_filtered;
 }
+
+/*
+ * Best-effort BRBE snapshot for BPF tracing. Pause BRBE to avoid
+ * self-recording and return 0 if the snapshot state appears disturbed.
+ */
+int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
+{
+	unsigned long flags;
+	int nr_hw, nr_banks, nr_copied = 0;
+	u64 brbidr, brbfcr, brbcr;
+
+	if (!cnt || !__this_cpu_read(brbe_active))
+		return 0;
+
+	/* Pause BRBE first to avoid recording our own branches. */
+	brbfcr = read_sysreg_s(SYS_BRBFCR_EL1);
+	brbcr = read_sysreg_s(SYS_BRBCR_EL1);
+	write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
+	isb();
+
+	/* Block local exception delivery while reading the buffer. */
+	flags = local_daif_save();
+
+	/*
+	 * A PMU overflow before local_daif_save() could have re-enabled
+	 * BRBE, clearing the PAUSED bit. The overflow handler already
+	 * restored BRBE to its correct state, so just bail out.
+	 */
+	if (!(read_sysreg_s(SYS_BRBFCR_EL1) & BRBFCR_EL1_PAUSED)) {
+		local_daif_restore(flags);
+		return 0;
+	}
+
+	brbidr = read_sysreg_s(SYS_BRBIDR0_EL1);
+	if (!valid_brbidr(brbidr))
+		goto out;
+
+	nr_hw = FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr);
+	nr_banks = DIV_ROUND_UP(nr_hw, BRBE_BANK_MAX_ENTRIES);
+
+	for (int bank = 0; bank < nr_banks; bank++) {
+		int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
+		int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
+
+		select_brbe_bank(bank);
+
+		for (int i = 0; i < nr_this_bank; i++) {
+			if (nr_copied >= cnt)
+				goto done;
+
+			if (!perf_entry_from_brbe_regset(i, &entries[nr_copied], NULL))
+				goto done;
+
+			nr_copied++;
+		}
+	}
+
+done:
+	brbe_invalidate();
+out:
+	/* Restore BRBCR before unpausing via BRBFCR, matching brbe_enable(). */
+	write_sysreg_s(brbcr, SYS_BRBCR_EL1);
+	isb();
+	write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
+	local_daif_restore(flags);
+
+	return nr_copied;
+}
drivers/perf/arm_brbe.h +9 -0
index b7c7d8796c86..c2a1824437fb 100644
@@ -10,6 +10,7 @@
 struct arm_pmu;
 struct perf_branch_stack;
 struct perf_event;
+struct perf_branch_entry;
 
 #ifdef CONFIG_ARM64_BRBE
 void brbe_probe(struct arm_pmu *arm_pmu);
@@ -22,6 +23,8 @@ void brbe_disable(void);
 bool brbe_branch_attr_valid(struct perf_event *event);
 void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 				const struct perf_event *event);
+int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries,
+				   unsigned int cnt);
 #else
 static inline void brbe_probe(struct arm_pmu *arm_pmu) { }
 static inline unsigned int brbe_num_branch_records(const struct arm_pmu *armpmu)
@@ -44,4 +47,10 @@ static void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 				       const struct perf_event *event)
 {
 }
+
+static inline int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries,
+						 unsigned int cnt)
+{
+	return 0;
+}
 #endif
drivers/perf/arm_pmuv3.c +4 -1
index 2d097fad9c10..e00c7c47a98d 100644
@@ -1456,8 +1456,11 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name,
 	cpu_pmu->set_event_filter	= armv8pmu_set_event_filter;
 
 	cpu_pmu->pmu.event_idx		= armv8pmu_user_event_idx;
-	if (brbe_num_branch_records(cpu_pmu))
+	if (brbe_num_branch_records(cpu_pmu)) {
 		cpu_pmu->pmu.sched_task		= armv8pmu_sched_task;
+		static_call_update(perf_snapshot_branch_stack,
+				   arm_brbe_snapshot_branch_stack);
+	}
 
 	cpu_pmu->name			= name;
 	cpu_pmu->map_event		= map_event;
Signature · 1 line
2.52.0
#10 lore raw

Re: [PATCH v2 3/4] perf/arm64: Add BRBE support for bpf_get_branch_snapshot()

Quoted 26 lines · Puranjay Mohan

Enable the bpf_get_branch_snapshot() BPF helper on ARM64 by implementing the perf_snapshot_branch_stack static call for ARM's Branch Record Buffer Extension (BRBE).

The BPF helper bpf_get_branch_snapshot() allows BPF programs to capture hardware branch records on-demand. This was previously only available on x86 (Intel LBR) but not on ARM64 despite BRBE being available since ARMv9.

BRBE is paused before disabling interrupts because local_irq_save() can trigger trace_hardirqs_off() which performs stack walking and pollutes the branch buffer. The sysreg read/write and ISB used to pause BRBE are branchless, so pausing first avoids this pollution.

All exceptions are masked after pausing BRBE using local_daif_save() to prevent pseudo-NMI from PMU counter overflow from interfering with the snapshot read. A PMU overflow arriving between the pause and local_daif_save() can re-enable BRBE via the interrupt handler; the snapshot detects this by re-checking BRBFCR_EL1.PAUSED and bailing out.

Branch records are read using the existing perf_entry_from_brbe_regset() helper with a NULL event pointer, which bypasses event-specific filtering and captures all recorded branches. The BPF program is responsible for filtering entries based on its own criteria. The BRBE buffer is invalidated after reading to maintain contiguity for other consumers.

Quoted 4 lines

On heterogeneous big.LITTLE systems, only some CPUs may implement FEAT_BRBE. The perf_snapshot_branch_stack static call is system-wide, so a per-CPU brbe_active flag is used to prevent BRBE sysreg access on CPUs that do not implement FEAT_BRBE, where such access would be UNDEFINED.

Is this something you've seen? IIRC, the existing assumption is all CPUs have FEAT_BRBE or that perf has been limited to those CPUs. It is allowed that the number of records can vary.

Quoted 69 lines
---
 drivers/perf/arm_brbe.c  | 79 +++++++++++++++++++++++++++++++++++++++-
 drivers/perf/arm_brbe.h  |  9 +++++
 drivers/perf/arm_pmuv3.c |  5 ++-
 3 files changed, 90 insertions(+), 3 deletions(-)
drivers/perf/arm_brbe.c +18 -2
index ba554e0c846c..527c2d5ebba6 100644
@@ -8,9 +8,13 @@
  */
 #include <linux/types.h>
 #include <linux/bitmap.h>
+#include <linux/percpu.h>
 #include <linux/perf/arm_pmu.h>
+#include <asm/daifflags.h>
 #include "arm_brbe.h"
 
+static DEFINE_PER_CPU(bool, brbe_active);
+
 #define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT   | \
 				   BRBFCR_EL1_INDIRECT | \
 				   BRBFCR_EL1_RTN      | \
@@ -533,6 +537,8 @@ void brbe_enable(const struct arm_pmu *arm_pmu)
 	/* Finally write SYS_BRBFCR_EL to unpause BRBE */
 	write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
 	/* Synchronization in PMCR write ensures ordering WRT PMU enabling */
+
+	this_cpu_write(brbe_active, true);
 }
 
 void brbe_disable(void)
@@ -544,6 +550,7 @@ void brbe_disable(void)
 	 */
 	write_sysreg_s(BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
 	write_sysreg_s(0, SYS_BRBCR_EL1);
+	this_cpu_write(brbe_active, false);
 }
 
 static const int brbe_type_to_perf_type_map[BRBINFx_EL1_TYPE_DEBUG_EXIT + 1][2] = {
@@ -618,10 +625,10 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 
 	brbe_set_perf_entry_type(entry, brbinf);
 
-	if (!branch_sample_no_cycles(event))
+	if (!event || !branch_sample_no_cycles(event))
 		entry->cycles = brbinf_get_cycles(brbinf);
 
-	if (!branch_sample_no_flags(event)) {
+	if (!event || !branch_sample_no_flags(event)) {
 		/* Mispredict info is available for source only and complete branch records. */
 		if (!brbe_record_is_target_only(brbinf)) {
 			entry->mispred = brbinf_get_mispredict(brbinf);
@@ -803,3 +810,71 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 done:
 	branch_stack->nr = nr_filtered;
 }
+
+/*
+ * Best-effort BRBE snapshot for BPF tracing. Pause BRBE to avoid
+ * self-recording and return 0 if the snapshot state appears disturbed.
+ */
+int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
+{
+	unsigned long flags;
+	int nr_hw, nr_banks, nr_copied = 0;

drivers/perf/arm_brbe.c void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, · line 831

Quoted 10 lines
drivers/perf/arm_brbe.c +10 -0
+	u64 brbidr, brbfcr, brbcr;
+
+	if (!cnt || !__this_cpu_read(brbe_active))
+		return 0;
+
+	/* Pause BRBE first to avoid recording our own branches. */
+	brbfcr = read_sysreg_s(SYS_BRBFCR_EL1);
+	brbcr = read_sysreg_s(SYS_BRBCR_EL1);
+	write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
+	isb();

Is there something that guarantees BRBE is enabled when you enter this function and that it is not disabled in this window? A context switch could disable it unless it's a global event for example.

Quoted 6 lines
drivers/perf/arm_brbe.c +6 -0
+
+	/* Block local exception delivery while reading the buffer. */
+	flags = local_daif_save();
+
+	/*
+	 * A PMU overflow before local_daif_save() could have re-enabled

drivers/perf/arm_brbe.c void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, · line 847

Quoted 10 lines
drivers/perf/arm_brbe.c +10 -0
+	 * BRBE, clearing the PAUSED bit. The overflow handler already
+	 * restored BRBE to its correct state, so just bail out.
+	 */
+	if (!(read_sysreg_s(SYS_BRBFCR_EL1) & BRBFCR_EL1_PAUSED)) {
+		local_daif_restore(flags);
+		return 0;
+	}
+
+	brbidr = read_sysreg_s(SYS_BRBIDR0_EL1);
+	if (!valid_brbidr(brbidr))

This is not possibly true if brbe_active is true. If BRBIDR is not valid, then we would have rejected any event requesting branch stack.

drivers/perf/arm_brbe.c void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, · line 859

Quoted 12 lines
drivers/perf/arm_brbe.c +12 -0
+		goto out;
+
+	nr_hw = FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr);
+	nr_banks = DIV_ROUND_UP(nr_hw, BRBE_BANK_MAX_ENTRIES);
+
+	for (int bank = 0; bank < nr_banks; bank++) {
+		int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
+		int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
+
+		select_brbe_bank(bank);
+
+		for (int i = 0; i < nr_this_bank; i++) {

I don't love all this being duplicated. Perhaps an iterator would help here (and the other copy):

static void next_slot(int *bank, int *bank_idx, int nr_hw)

{
	*bank_idx++;
	if (*bank_idx + (*bank * BRBE_BANK_MAX_ENTRIES) == nr_hw)
		bank = BRBE_MAX_BANKS;
	else if ((*bank_idx % BRBE_BANK_MAX_ENTRIES) == 0) {
		*bank_idx = 0;
		select_brbe_bank(++(*bank));
	}

}

#define for_each_bank_slot(i, nr_hw) \
for (int bank = 0, int i = 0; bank < BRBE_MAX_BANKS; next_slot(&bank, &i))

Then here you just have:

for_each_bank_slot(i, FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr)) {
	...
}

Feel free to come up with better naming. :)

Quoted 21 lines
drivers/perf/arm_brbe.c +21 -0
+			if (nr_copied >= cnt)
+				goto done;
+
+			if (!perf_entry_from_brbe_regset(i, &entries[nr_copied], NULL))
+				goto done;
+
+			nr_copied++;
+		}
+	}
+
+done:
+	brbe_invalidate();
+out:
+	/* Restore BRBCR before unpausing via BRBFCR, matching brbe_enable(). */
+	write_sysreg_s(brbcr, SYS_BRBCR_EL1);
+	isb();
+	write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
+	local_daif_restore(flags);
+
+	return nr_copied;
+}
#11 lore raw

Re: [PATCH v2 3/4] perf/arm64: Add BRBE support for bpf_get_branch_snapshot()

Quoted 28 lines · Rob Herring
Quoted 26 lines · Puranjay Mohan

Enable the bpf_get_branch_snapshot() BPF helper on ARM64 by implementing the perf_snapshot_branch_stack static call for ARM's Branch Record Buffer Extension (BRBE).

The BPF helper bpf_get_branch_snapshot() allows BPF programs to capture hardware branch records on-demand. This was previously only available on x86 (Intel LBR) but not on ARM64 despite BRBE being available since ARMv9.

BRBE is paused before disabling interrupts because local_irq_save() can trigger trace_hardirqs_off() which performs stack walking and pollutes the branch buffer. The sysreg read/write and ISB used to pause BRBE are branchless, so pausing first avoids this pollution.

All exceptions are masked after pausing BRBE using local_daif_save() to prevent pseudo-NMI from PMU counter overflow from interfering with the snapshot read. A PMU overflow arriving between the pause and local_daif_save() can re-enable BRBE via the interrupt handler; the snapshot detects this by re-checking BRBFCR_EL1.PAUSED and bailing out.

Branch records are read using the existing perf_entry_from_brbe_regset() helper with a NULL event pointer, which bypasses event-specific filtering and captures all recorded branches. The BPF program is responsible for filtering entries based on its own criteria. The BRBE buffer is invalidated after reading to maintain contiguity for other consumers.

Quoted 8 lines
Quoted 4 lines

On heterogeneous big.LITTLE systems, only some CPUs may implement FEAT_BRBE. The perf_snapshot_branch_stack static call is system-wide, so a per-CPU brbe_active flag is used to prevent BRBE sysreg access on CPUs that do not implement FEAT_BRBE, where such access would be UNDEFINED.

Is this something you've seen? IIRC, the existing assumption is all CPUs have FEAT_BRBE or that perf has been limited to those CPUs. It is allowed that the number of records can vary.

I just added it as a safeguard, but if you say that it is not possible, I will remove this in the next version.

Quoted 76 lines
Quoted 76 lines
---
 drivers/perf/arm_brbe.c  | 79 +++++++++++++++++++++++++++++++++++++++-
 drivers/perf/arm_brbe.h  |  9 +++++
 drivers/perf/arm_pmuv3.c |  5 ++-
 3 files changed, 90 insertions(+), 3 deletions(-)
drivers/perf/arm_brbe.c +24 -2
index ba554e0c846c..527c2d5ebba6 100644
@@ -8,9 +8,13 @@
  */
 #include <linux/types.h>
 #include <linux/bitmap.h>
+#include <linux/percpu.h>
 #include <linux/perf/arm_pmu.h>
+#include <asm/daifflags.h>
 #include "arm_brbe.h"
 
+static DEFINE_PER_CPU(bool, brbe_active);
+
 #define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT   | \
                                 BRBFCR_EL1_INDIRECT | \
                                 BRBFCR_EL1_RTN      | \
@@ -533,6 +537,8 @@ void brbe_enable(const struct arm_pmu *arm_pmu)
      /* Finally write SYS_BRBFCR_EL to unpause BRBE */
      write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
      /* Synchronization in PMCR write ensures ordering WRT PMU enabling */
+
+     this_cpu_write(brbe_active, true);
 }
 
 void brbe_disable(void)
@@ -544,6 +550,7 @@ void brbe_disable(void)
       */
      write_sysreg_s(BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
      write_sysreg_s(0, SYS_BRBCR_EL1);
+     this_cpu_write(brbe_active, false);
 }
 
 static const int brbe_type_to_perf_type_map[BRBINFx_EL1_TYPE_DEBUG_EXIT + 1][2] = {
@@ -618,10 +625,10 @@ static bool perf_entry_from_brbe_regset(int index, struct perf_branch_entry *ent
 
      brbe_set_perf_entry_type(entry, brbinf);
 
-     if (!branch_sample_no_cycles(event))
+     if (!event || !branch_sample_no_cycles(event))
              entry->cycles = brbinf_get_cycles(brbinf);
 
-     if (!branch_sample_no_flags(event)) {
+     if (!event || !branch_sample_no_flags(event)) {
              /* Mispredict info is available for source only and complete branch records. */
              if (!brbe_record_is_target_only(brbinf)) {
                      entry->mispred = brbinf_get_mispredict(brbinf);
@@ -803,3 +810,71 @@ void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack,
 done:
      branch_stack->nr = nr_filtered;
 }
+
+/*
+ * Best-effort BRBE snapshot for BPF tracing. Pause BRBE to avoid
+ * self-recording and return 0 if the snapshot state appears disturbed.
+ */
+int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsigned int cnt)
+{
+     unsigned long flags;
+     int nr_hw, nr_banks, nr_copied = 0;
+     u64 brbidr, brbfcr, brbcr;
+
+     if (!cnt || !__this_cpu_read(brbe_active))
+             return 0;
+
+     /* Pause BRBE first to avoid recording our own branches. */
Quoted 8 lines

drivers/perf/arm_brbe.c void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, · line 831

Quoted 4 lines
drivers/perf/arm_brbe.c +4 -0
+     brbfcr = read_sysreg_s(SYS_BRBFCR_EL1);
+     brbcr = read_sysreg_s(SYS_BRBCR_EL1);
+     write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1);
+     isb();

Is there something that guarantees BRBE is enabled when you enter this function and that it is not disabled in this window? A context switch could disable it unless it's a global event for example.

The user space program handling this BPF program should create a system wide event in the expected use case. But you are right that it is not guaranteed to be enabled, I will check and return early in that case.

Quoted 12 lines
Quoted 12 lines
drivers/perf/arm_brbe.c +11 -0
+
+     /* Block local exception delivery while reading the buffer. */
+     flags = local_daif_save();
+
+     /*
+      * A PMU overflow before local_daif_save() could have re-enabled
+      * BRBE, clearing the PAUSED bit. The overflow handler already
+      * restored BRBE to its correct state, so just bail out.
+      */
+     if (!(read_sysreg_s(SYS_BRBFCR_EL1) & BRBFCR_EL1_PAUSED)) {
+             local_daif_restore(flags);
Quoted 8 lines

drivers/perf/arm_brbe.c void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, · line 847

Quoted 5 lines
drivers/perf/arm_brbe.c +5 -0
+             return 0;
+     }
+
+     brbidr = read_sysreg_s(SYS_BRBIDR0_EL1);
+     if (!valid_brbidr(brbidr))

This is not possibly true if brbe_active is true. If BRBIDR is not valid, then we would have rejected any event requesting branch stack.

Will remove it in the next version.

Quoted 33 lines

drivers/perf/arm_brbe.c void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, · line 859

Quoted 13 lines
drivers/perf/arm_brbe.c +12 -0
+             goto out;
+
+     nr_hw = FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr);
+     nr_banks = DIV_ROUND_UP(nr_hw, BRBE_BANK_MAX_ENTRIES);
+
+     for (int bank = 0; bank < nr_banks; bank++) {
+             int nr_remaining = nr_hw - (bank * BRBE_BANK_MAX_ENTRIES);
+             int nr_this_bank = min(nr_remaining, BRBE_BANK_MAX_ENTRIES);
+
+             select_brbe_bank(bank);
+
+             for (int i = 0; i < nr_this_bank; i++) {

I don't love all this being duplicated. Perhaps an iterator would help here (and the other copy):

static void next_slot(int *bank, int *bank_idx, int nr_hw)

{
        *bank_idx++;
        if (*bank_idx + (*bank * BRBE_BANK_MAX_ENTRIES) == nr_hw)
                bank = BRBE_MAX_BANKS;
        else if ((*bank_idx % BRBE_BANK_MAX_ENTRIES) == 0) {
                *bank_idx = 0;
                select_brbe_bank(++(*bank));
        }

}

#define for_each_bank_slot(i, nr_hw) \
for (int bank = 0, int i = 0; bank < BRBE_MAX_BANKS; next_slot(&bank, &i))

Quoted 7 lines

Then here you just have:

for_each_bank_slot(i, FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr)) {
        ...
}

Feel free to come up with better naming. :)

Will add this to the next version, thanks

Quoted 22 lines
Quoted 22 lines
drivers/perf/arm_brbe.c +21 -0
+                     if (nr_copied >= cnt)
+                             goto done;
+
+                     if (!perf_entry_from_brbe_regset(i, &entries[nr_copied], NULL))
+                             goto done;
+
+                     nr_copied++;
+             }
+     }
+
+done:
+     brbe_invalidate();
+out:
+     /* Restore BRBCR before unpausing via BRBFCR, matching brbe_enable(). */
+     write_sysreg_s(brbcr, SYS_BRBCR_EL1);
+     isb();
+     write_sysreg_s(brbfcr, SYS_BRBFCR_EL1);
+     local_daif_restore(flags);
+
+     return nr_copied;
+}
#12 lore raw

PATCH v2 4/4 selftests/bpf: Adjust wasted entries threshold for ARM64 BRBE

In reply to Puranjay Mohan

The get_branch_snapshot test checks that bpf_get_branch_snapshot() doesn't waste too many branch entries on infrastructure overhead. The threshold of < 10 was calibrated for x86 where about 7 entries are wasted.

On ARM64, the BPF trampoline generates more branches than x86, resulting in about 13 wasted entries. The overhead comes from the BPF trampoline calling __bpf_prog_enter_recur which on ARM64 makes out-of-line calls to __rcu_read_lock and generates more conditional branches than x86:

 [#12] bpf_testmod_loop_test+0x40    -> bpf_trampoline_...+0x48
 [#11] bpf_trampoline_...+0x68       -> __bpf_prog_enter_recur+0x0
 [#10] __bpf_prog_enter_recur+0x20   -> __bpf_prog_enter_recur+0x118
 [#09] __bpf_prog_enter_recur+0x154  -> __bpf_prog_enter_recur+0x160
 [#08] __bpf_prog_enter_recur+0x164  -> __bpf_prog_enter_recur+0x2c
 [#07] __bpf_prog_enter_recur+0x2c   -> __rcu_read_lock+0x0
 [#06] __rcu_read_lock+0x18          -> __bpf_prog_enter_recur+0x30
 [#05] __bpf_prog_enter_recur+0x9c   -> __bpf_prog_enter_recur+0xf0
 [#04] __bpf_prog_enter_recur+0xf4   -> __bpf_prog_enter_recur+0xa8
 [#03] __bpf_prog_enter_recur+0xb8   -> __bpf_prog_enter_recur+0x100
 [#02] __bpf_prog_enter_recur+0x114  -> bpf_trampoline_...+0x6c
 [#01] bpf_trampoline_...+0x78       -> bpf_prog_...test1+0x0
 [#00] bpf_prog_...test1+0x58        -> arm_brbe_snapshot_branch_stack+0x0

Use an architecture-specific threshold of < 14 for ARM64 to accommodate this overhead while still detecting regressions.

---
 .../selftests/bpf/prog_tests/get_branch_snapshot.c  | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c +9 -4
index 0394a1156d99..8d1a3480767f 100644
@@ -116,13 +116,18 @@ void serial_test_get_branch_snapshot(void)
 
 	ASSERT_GT(skel->bss->test1_hits, 6, "find_looptest_in_lbr");
 
-	/* Given we stop LBR in software, we will waste a few entries.
+	/* Given we stop LBR/BRBE in software, we will waste a few entries.
 	 * But we should try to waste as few as possible entries. We are at
-	 * about 7 on x86_64 systems.
-	 * Add a check for < 10 so that we get heads-up when something
-	 * changes and wastes too many entries.
+	 * about 7 on x86_64 and about 13 on arm64 systems (the arm64 BPF
+	 * trampoline generates more branches than x86_64).
+	 * Add a check so that we get heads-up when something changes and
+	 * wastes too many entries.
 	 */
+#if defined(__aarch64__)
+	ASSERT_LT(skel->bss->wasted_entries, 14, "check_wasted_entries");
+#else
 	ASSERT_LT(skel->bss->wasted_entries, 10, "check_wasted_entries");
+#endif
 
 cleanup:
 	get_branch_snapshot__destroy(skel);
Signature · 1 line
2.52.0
#13 lore raw
In reply to Puranjay Mohan

Hi Catalin, Mark, and Will,

Would you mind taking a look at this patchset when you have a chance?

Thanks,
Puranjay

Quoted 157 lines · Puranjay Mohan

v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Rebased on arm64/for-next/core
- Add per-CPU brbe_active flag to guard against UNDEFINED sysreg access on non-BRBE CPUs in heterogeneous big.LITTLE systems.
- Fix pre-existing bug in perf_clear_branch_entry_bitfields() that missed zeroing new_type and priv bitfields, added as a separate patch with Fixes tags (new patch 2).
- Use architecture-specific selftest threshold (#if defined(__aarch64__)) instead of raising the global threshold, to preserve x86 regression detection.

RFC: https://lore.kernel.org/all/[email protected]/
Changes from RFC:
- Fix pre-existing NULL pointer dereference in armv8pmu_sched_task() found by Leo Yan during testing (patch 1)
- Pause BRBE before local_daif_save() to avoid branch pollution from trace_hardirqs_off()
- Use local_daif_save() to prevent pNMI race from counter overflow (Mark Rutland)
- Reuse perf_entry_from_brbe_regset() instead of duplicating register read logic, by making it accept NULL event (Mark Rutland)
- Invalidate BRBE after reading to maintain record contiguity for other consumers (Mark Rutland)
- Adjust selftest wasted_entries threshold for ARM64 (patch 3)
- Tested on ARM FVP with BRBE enabled

This series enables the bpf_get_branch_snapshot() BPF helper on ARM64 by implementing the perf_snapshot_branch_stack static call for ARM's Branch Record Buffer Extension (BRBE).

bpf_get_branch_snapshot() [1] allows BPF programs to capture hardware branch records on-demand from any BPF tracing context. This was previously only available on x86 (Intel LBR) since v5.16. With BRBE available on ARMv9, this series closes the gap for ARM64.

Usage model
-----------

The helper works in conjunction with perf events. The userspace component of the BPF application opens a perf event with PERF_SAMPLE_BRANCH_STACK on each CPU, which configures the hardware to continuously record branches into BRBE (on ARM64) or LBR (on x86). A BPF program attached to a tracepoint, kprobe, or fentry hook can then call bpf_get_branch_snapshot() to snapshot the branch buffer at any point. Without an active perf event, BRBE is not recording and the buffer is empty.

On-demand branch snapshots from BPF are useful for diagnosing which specific code path was taken inside a function. Stack traces only show function boundaries, but branch records reveal the exact sequence of jumps, calls, and returns within a function -- making it possible to identify which specific error check triggered a failure, or which callback implementation was invoked through a function pointer.

For example, retsnoop [2] is a BPF-based tool for non-intrusive mass-tracing of kernel internals. Its LBR mode (--lbr) creates per-CPU perf events with PERF_SAMPLE_BRANCH_STACK and then uses bpf_get_branch_snapshot() in its fentry/fexit BPF programs to capture branch records whenever a traced function returns an error.

Consider debugging a bpf() syscall that returns -EINVAL when creating a BPF map with invalid parameters. Running retsnoop on an ARM64 FVP with BRBE to trace the bpf() syscall and array_map_alloc_check():

  $ retsnoop -e '*sys_bpf' -a 'array_map_alloc_check' --lbr=any \
             -F -k vmlinux --debug full-lbr
  $ simfail bpf-bad-map-max-entries-array  # in another terminal

Output of retsnoop:

--- fentry BPF program (entries #63-#17) ---

[#63-#59] __htab_map_lookup_elem: hash table walk with memcmp (hashtab.c)
[#58] __htab_map_lookup_elem+0x98 -> dump_bpf_prog+0xc850 (hashtab.c:750)
[#57-#55] ... dump_bpf_prog internal branches ...
[#54] dump_bpf_prog+0xcab8 -> bpf_get_current_pid_tgid+0x0 (helpers.c:225)
[#53] bpf_get_current_pid_tgid+0x1c -> dump_bpf_prog+0xcabc (helpers.c:225)
[#52-#51] ... dump_bpf_prog -> __htab_map_lookup_elem ... [#50-#47] __htab_map_lookup_elem: htab_map_hash (jhash2), select_bucket
[#46-#42] lookup_nulls_elem_raw: hash chain walk with memcmp (hashtab.c:717)
[#41] __htab_map_lookup_elem+0x98 -> dump_bpf_prog+0xcaf8 (hashtab.c:750)
[#40-#37] ... dump_bpf_prog -> bpf_ktime_get_ns ...
[#36] bpf_ktime_get_ns+0x10 -> ktime_get_mono_fast_ns+0x0 (helpers.c:178)
[#35-#32] ktime_get_mono_fast_ns: tk_clock_read -> arch_counter_get_cntpct
[#31] ktime_get_mono_fast_ns+0x9c -> bpf_ktime_get_ns+0x14 (timekeeping.c:493)
[#30] bpf_ktime_get_ns+0x18 -> dump_bpf_prog+0xcd50 (helpers.c:178)
[#29-#25] ... dump_bpf_prog internal branches ...
[#24] dump_bpf_prog+0x11b28 -> __bpf_prog_exit_recur+0x0 (trampoline.c:1190)
[#23-#17] __bpf_prog_exit_recur: rcu_read_unlock, migrate_enable (trampoline.c:1195)

--- array_map_alloc_check (entries #16-#12) ---

  [#16] dump_bpf_prog+0x11b38       -> array_map_alloc_check+0x8       (arraymap.c:55)
  [#15] array_map_alloc_check+0x18  -> array_map_alloc_check+0xb8      (arraymap.c:56)
        . bpf_map_attr_numa_node       . bpf_map_attr_numa_node
  [#14] array_map_alloc_check+0xbc  -> array_map_alloc_check+0x20      (arraymap.c:59)
        . bpf_map_attr_numa_node
  [#13] array_map_alloc_check+0x24  -> array_map_alloc_check+0x94      (arraymap.c:64)
  [#12] array_map_alloc_check+0x98  -> dump_bpf_prog+0x11b3c           (arraymap.c:82)

--- fexit trampoline overhead (entries #11-#00) ---

  [#11] dump_bpf_prog+0x11b5c       -> __bpf_prog_enter_recur+0x0      (trampoline.c:1145)
  [#10-#03] __bpf_prog_enter_recur: rcu_read_lock, migrate_disable     (trampoline.c:1146)
  [#02] __bpf_prog_enter_recur+0x114 -> dump_bpf_prog+0x11b60          (trampoline.c:1157)

[#01] dump_bpf_prog+0x11b6c -> dump_bpf_prog+0xd230
[#00] dump_bpf_prog+0xd340 -> arm_brbe_snapshot_branch_stack+0x0 (arm_brbe.c:814)

                   el0t_64_sync+0x168
                   el0t_64_sync_handler+0x98
                   el0_svc+0x28
                   do_el0_svc+0x4c
                   invoke_syscall.constprop.0+0x54
    373us [-EINVAL] __arm64_sys_bpf+0x8
                    __sys_bpf+0x87c
                    map_create+0x120
     95us [-EINVAL] array_map_alloc_check+0x8

The FVP's BRBE buffer has 64 entries (BRBE supports 8, 16, 32, or 64). Of these, entries #63-#17 (47) are consumed by the fentry BPF trampoline that ran before the function, and entries #11-#00 (12) are consumed by the fexit trampoline that runs after. Entry #00 shows the very last branch recorded before BRBE is paused: the call into arm_brbe_snapshot_branch_stack().

The 5 useful entries (#16-#12) show the exact path taken inside array_map_alloc_check(). Record #14 shows a jump from line 56 (bpf_map_attr_numa_node) to line 59 (the if-condition), and #13 shows an immediate jump from line 59 (attr->max_entries == 0) to line 64 (return -EINVAL), skipping lines 60-63. This pinpoints max_entries==0 as the cause -- a diagnosis impossible with stack traces alone.

[1] 856c02dbce4f ("bpf: Introduce helper bpf_get_branch_snapshot")
[2] https://github.com/anakryiko/retsnoop

Puranjay Mohan (4):
perf/arm_pmuv3: Fix NULL pointer dereference in armv8pmu_sched_task()
perf: Fix uninitialized bitfields in
perf_clear_branch_entry_bitfields()
perf/arm64: Add BRBE support for bpf_get_branch_snapshot() selftests/bpf: Adjust wasted entries threshold for ARM64 BRBE

 drivers/perf/arm_brbe.c                       | 79 ++++++++++++++++++-
 drivers/perf/arm_brbe.h                       |  9 +++
 drivers/perf/arm_pmuv3.c                      | 16 +++-
 include/linux/perf_event.h                    |  2 +
 .../bpf/prog_tests/get_branch_snapshot.c      | 13 ++-
 5 files changed, 110 insertions(+), 9 deletions(-)

base-commit: d118f32246fdabfb4f6a3fd2e511dc5e622bc553 --
2.52.0

maintainer
#14 lore raw
Quoted 3 lines · Puranjay Mohan

Hi Catalin, Mark, and Will,

Would you mind taking a look at this patchset when you have a chance?

Adding Rob and Anshuman, as they wrote the perf driver for BRBE and are the best people to review this stuff.

Will