perf/core: Prepare sample data for BPF
Subsystems (8)
Namhyung Kim
maintainercover perf/core: Prepare sample data for BPF
Hello,
The perf_prepare_sample() is to fill the perf sample data and update the header info before sending it to the ring buffer. But we want to use it for BPF overflow handler so that it can access the sample data to filter relevant ones.
Changes in v2)
* the layout change is merged
* reduce branches using __cond_set (PeterZ)
* add helpers to set dynamic sample data (PeterZ)
* introduce perf_prepare_header() (PeterZ)
* call perf_prepare_sample() before bpf_overflow_handler unconditionally
This means the perf_prepare_handler() can be called more than once. To avoid duplicate work, use the data->sample_flags and save the data size.
I also added a few of helpers to set those information accordingly. But it looks some fields like REGS_USER, STACK_USER and AUX are saved in the perf_prepare_sample() so I didn't add the helpers for them.
After than we can just check the filtered_sample_type flags begin zero to determine if it has more work. In that case, it needs to update the data->type since it's possible to miss when PMU driver sets all required sample flags before calling perf_prepare_sample().
The code is also available at 'perf/prepare-sample-v2' branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks,
Namhyung
Namhyung Kim (8):
perf/core: Save the dynamic parts of sample data size perf/core: Add perf_sample_save_callchain() helper perf/core: Add perf_sample_save_raw_data() helper perf/core: Add perf_sample_save_brstack() helper perf/core: Set data->sample_flags in perf_prepare_sample() perf/core: Do not pass header for sample id init perf/core: Introduce perf_prepare_header() perf/core: Call perf_prepare_sample() before running BPF
arch/powerpc/perf/core-book3s.c | 3 +- arch/s390/kernel/perf_cpum_cf.c | 4 +- arch/s390/kernel/perf_cpum_sf.c | 3 +- arch/s390/kernel/perf_pai_crypto.c | 4 +- arch/s390/kernel/perf_pai_ext.c | 4 +- arch/x86/events/amd/core.c | 6 +- arch/x86/events/amd/ibs.c | 9 +- arch/x86/events/intel/core.c | 6 +- arch/x86/events/intel/ds.c | 24 ++-- include/linux/perf_event.h | 133 +++++++++++++----- kernel/events/core.c | 207 ++++++++++++++++------------- kernel/trace/bpf_trace.c | 6 +- 12 files changed, 236 insertions(+), 173 deletions(-)
base-commit: 9fcad995c6c52cc9791f7ee9f1386a5684055f9c
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Namhyung Kim
maintainerPATCH 1/8 perf/core: Save the dynamic parts of sample data size
The perf sample data can be divided into parts. The event->header_size and event->id_header_size keep the static part of the sample data which is determined by the sample_type flags.
But other parts like CALLCHAIN and BRANCH_STACK are changing dynamically so it needs to see the actual data. In preparation of handling repeated calls for perf_prepare_sample(), it can save the dynamic size to the perf sample data to avoid the duplicate work.
--- include/linux/perf_event.h | 2 ++ kernel/events/core.c | 17 ++++++++++------- 2 files changed, 12 insertions(+), 7 deletions(-)
include/linux/perf_event.h +2 -0
@@ -1103,6 +1103,7 @@ struct perf_sample_data {*/ u64 sample_flags;u64 period; +u64 dyn_size;/* * Fields commonly set by __perf_event_header__init_id(),
@@ -1158,6 +1159,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data,/* remaining struct members initialized in perf_prepare_sample() */ data->sample_flags =PERF_SAMPLE_PERIOD ; data->period = period; + data->dyn_size =0 ;if (addr) { data->addr = addr;
kernel/events/core.c +10 -7
@@ -7593,7 +7593,7 @@ void perf_prepare_sample(struct perf_event_header *header, size += data->callchain->nr; - header->size += size *sizeof (u64); + data->dyn_size += size *sizeof (u64); }if (sample_type &PERF_SAMPLE_RAW ) {
@@ -7619,7 +7619,7 @@ void perf_prepare_sample(struct perf_event_header *header, data->raw =NULL ; } - header->size += size; + data->dyn_size += size; }if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {
@@ -7631,7 +7631,7 @@ void perf_prepare_sample(struct perf_event_header *header, size += data->br_stack->nr *sizeof (struct perf_branch_entry ); } - header->size += size; + data->dyn_size += size; }if (sample_type & (PERF_SAMPLE_REGS_USER |PERF_SAMPLE_STACK_USER ))
@@ -7646,7 +7646,7 @@ void perf_prepare_sample(struct perf_event_header *header, size +=hweight64 (mask) *sizeof (u64); } - header->size += size; + data->dyn_size += size; }if (sample_type &PERF_SAMPLE_STACK_USER ) {
@@ -7671,7 +7671,7 @@ void perf_prepare_sample(struct perf_event_header *header, size +=sizeof (u64) + stack_size; data->stack_user_size = stack_size; - header->size += size; + data->dyn_size += size; }if (filtered_sample_type &PERF_SAMPLE_WEIGHT_TYPE )
@@ -7700,7 +7700,7 @@ void perf_prepare_sample(struct perf_event_header *header, size +=hweight64 (mask) *sizeof (u64); } - header->size += size; + data->dyn_size += size; } if (sample_type & PERF_SAMPLE_PHYS_ADDR &&
@@ -7745,8 +7745,11 @@ void perf_prepare_sample(struct perf_event_header *header, size =perf_prepare_sample_aux (event, data, size);WARN_ON_ONCE (size + header->size >U16_MAX ); - header->size += size; + data->dyn_size += size +sizeof (u64);/* size above */ } + + header->size += data->dyn_size; +/* * If you're adding more sample types here, you likely need to do * something about the overflowing header::size, like repurpose the
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Re: [PATCH 1/8] perf/core: Save the dynamic parts of sample data size
Quoted 11 lines · Namhyung Kim
The perf sample data can be divided into parts. The
event->header_sizeandevent->id_header_sizekeep the static part of the sample data which is determined by thesample_typeflags.But other parts like CALLCHAIN and
BRANCH_STACKare changing dynamically so it needs to see the actual data. In preparation of handling repeated calls forperf_prepare_sample(), it can save the dynamic size to the perf sample data to avoid the duplicate work.Signed-off-by Namhyung Kim <[email protected]>
Namhyung Kim
maintainerPATCH 2/8 perf/core: Add perf_sample_save_callchain() helper
In reply to Namhyung KimWhen it saves the callchain to the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add the perf_sample_save_callchain() helper and convert all call sites.
--- arch/x86/events/amd/ibs.c | 6 ++---- arch/x86/events/intel/ds.c | 12 ++++-------- include/linux/perf_event.h | 16 +++++++++++++++- kernel/events/core.c | 12 ++---------- 4 files changed, 23 insertions(+), 23 deletions(-)
arch/x86/events/amd/ibs.c +2 -4
@@ -1122,10 +1122,8 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)* recorded as part of interrupt regs. Thus we need to use rip from * interrupt regs while unwinding call stack. */ -if (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN ) { - data.callchain =perf_callchain (event, iregs); - data.sample_flags |=PERF_SAMPLE_CALLCHAIN ; - } +if (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN ) +perf_sample_save_callchain (&data, event, iregs); throttle =perf_event_overflow (event, &data, ®s); out:
arch/x86/events/intel/ds.c +4 -8
@@ -1617,10 +1617,8 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,* previous PMI context or an (I)RET happened between the record and * PMI. */ -if (sample_type &PERF_SAMPLE_CALLCHAIN ) { - data->callchain =perf_callchain (event, iregs); - data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; - } +if (sample_type &PERF_SAMPLE_CALLCHAIN ) +perf_sample_save_callchain (data, event, iregs);/* * We use the interrupt regs as a base because the PEBS record does not
@@ -1795,10 +1793,8 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,* previous PMI context or an (I)RET happened between the record and * PMI. */ -if (sample_type &PERF_SAMPLE_CALLCHAIN ) { - data->callchain =perf_callchain (event, iregs); - data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; - } +if (sample_type &PERF_SAMPLE_CALLCHAIN ) +perf_sample_save_callchain (data, event, iregs); *regs = *iregs;/* The ip in basic is EventingIP */
include/linux/perf_event.h +15 -1
@@ -1095,6 +1095,8 @@ int perf_event_read_local(struct perf_event *event, u64 *value,extern u64 perf_event_read_value (struct perf_event *event,u64 *enabled,u64 *running); +extern struct perf_callchain_entry *perf_callchain (struct perf_event *event,struct pt_regs *regs); +struct perf_sample_data {/*
@@ -1167,6 +1169,19 @@ static inline void perf_sample_data_init(struct perf_sample_data *data, } } +static inline void perf_sample_save_callchain (struct perf_sample_data *data, +struct perf_event *event, +struct pt_regs *regs) +{ +int size =1 ; + + data->callchain =perf_callchain (event, regs); + size += data->callchain->nr; + + data->dyn_size += size *sizeof (u64); + data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; +} +/* * Clear all bitfields in the perf_branch_entry. * The to and from fields are not cleared because they are
@@ -1408,7 +1423,6 @@ extern void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, structextern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,u32 init_nr,bool kernel,bool user,u32 max_stack,bool crosstask,bool add_mark); -extern struct perf_callchain_entry *perf_callchain (struct perf_event *event,struct pt_regs *regs);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);
kernel/events/core.c +2 -10
@@ -7585,16 +7585,8 @@ void perf_prepare_sample(struct perf_event_header *header,if (sample_type & (PERF_SAMPLE_IP |PERF_SAMPLE_CODE_PAGE_SIZE )) data->ip =perf_instruction_pointer (regs); -if (sample_type &PERF_SAMPLE_CALLCHAIN ) { -int size =1 ; - -if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN ) - data->callchain =perf_callchain (event, regs); - - size += data->callchain->nr; - - data->dyn_size += size *sizeof (u64); - } +if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN ) +perf_sample_save_callchain (data, event, regs);if (sample_type &PERF_SAMPLE_RAW ) {struct perf_raw_record *raw = data->raw;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Re: [PATCH 2/8] perf/core: Add perf_sample_save_callchain() helper
Quoted 8 lines · Namhyung Kim
When it saves the callchain to the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add the
perf_sample_save_callchain()helper and convert all call sites.Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Namhyung Kim <[email protected]>
Namhyung Kim
maintainerPATCH 3/8 perf/core: Add perf_sample_save_raw_data() helper
In reply to Namhyung KimWhen it saves the raw_data to the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add the perf_sample_save_raw_data() helper and convert all call sites.
--- arch/s390/kernel/perf_cpum_cf.c | 4 +--- arch/s390/kernel/perf_pai_crypto.c | 4 +--- arch/s390/kernel/perf_pai_ext.c | 4 +--- arch/x86/events/amd/ibs.c | 3 +-- include/linux/perf_event.h | 33 +++++++++++++++++++++++++----- kernel/events/core.c | 31 +++++----------------------- kernel/trace/bpf_trace.c | 6 ++---- 7 files changed, 39 insertions(+), 46 deletions(-)
arch/s390/kernel/perf_cpum_cf.c +1 -3
@@ -662,9 +662,7 @@ static int cfdiag_push_sample(struct perf_event *event,if (event->attr.sample_type &PERF_SAMPLE_RAW ) { raw.frag.size = cpuhw->usedss; raw.frag.data = cpuhw->stop; - raw.size = raw.frag.size; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); } overflow =perf_event_overflow (event, &data, ®s);
arch/s390/kernel/perf_pai_crypto.c +1 -3
@@ -362,9 +362,7 @@ static int paicrypt_push_sample(void)if (event->attr.sample_type &PERF_SAMPLE_RAW ) { raw.frag.size = rawsize; raw.frag.data = cpump->save; - raw.size = raw.frag.size; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); } overflow =perf_event_overflow (event, &data, ®s);
arch/s390/kernel/perf_pai_ext.c +1 -3
@@ -451,9 +451,7 @@ static int paiext_push_sample(void)if (event->attr.sample_type &PERF_SAMPLE_RAW ) { raw.frag.size = rawsize; raw.frag.data = cpump->save; - raw.size = raw.frag.size; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); } overflow =perf_event_overflow (event, &data, ®s);
arch/x86/events/amd/ibs.c +1 -2
@@ -1110,8 +1110,7 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs) .data = ibs_data.data, }, }; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); }if (perf_ibs == &perf_ibs_op)
include/linux/perf_event.h +28 -5
@@ -95,6 +95,11 @@ struct perf_raw_record {u32 size; }; +static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag) +{ +return frag->pad <sizeof (u64); +} +/* * branch stack layout: * nr: number of taken branches stored in entries[]
@@ -1182,6 +1187,29 @@ static inline void perf_sample_save_callchain(struct perf_sample_data *data, data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; } +static inline void perf_sample_save_raw_data (struct perf_sample_data *data, +struct perf_raw_record *raw) +{ +struct perf_raw_frag *frag = &raw->frag; +u32 sum =0 ; +int size; + +do { + sum += frag->size; +if (perf_raw_frag_last (frag)) +break ; + frag = frag->next; + }while (1 ); + + size =round_up (sum +sizeof (u32),sizeof (u64)); + raw->size = size -sizeof (u32); + frag->pad = raw->size - sum; + + data->raw = raw; + data->dyn_size += size; + data->sample_flags |=PERF_SAMPLE_RAW ; +} +/* * Clear all bitfields in the perf_branch_entry. * The to and from fields are not cleared because they are
@@ -1690,11 +1718,6 @@ extern void perf_restore_debug_store(void);static inline void perf_restore_debug_store (void ) { }#endif -static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag) -{ -return frag->pad <sizeof (u64); -} -#define perf_output_put (handle, x) perf_output_copy((handle), &(x), sizeof(x))struct perf_pmu_events_attr {
kernel/events/core.c +5 -26
@@ -7588,30 +7588,10 @@ void perf_prepare_sample(struct perf_event_header *header,if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN )perf_sample_save_callchain (data, event, regs); -if (sample_type &PERF_SAMPLE_RAW ) { -struct perf_raw_record *raw = data->raw; -int size; - -if (raw && (data->sample_flags &PERF_SAMPLE_RAW )) { -struct perf_raw_frag *frag = &raw->frag; -u32 sum =0 ; - -do { - sum += frag->size; -if (perf_raw_frag_last (frag)) -break ; - frag = frag->next; - }while (1 ); - - size =round_up (sum +sizeof (u32),sizeof (u64)); - raw->size = size -sizeof (u32); - frag->pad = raw->size - sum; - }else { - size =sizeof (u64); - data->raw =NULL ; - } - - data->dyn_size += size; +if (filtered_sample_type &PERF_SAMPLE_RAW ) { + data->raw =NULL ; + data->dyn_size +=sizeof (u64); + data->sample_flags |=PERF_SAMPLE_RAW ; }if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {
@@ -10127,8 +10107,7 @@ void perf_tp_event(u16 event_type, u64 count, void *record, int entry_size, };perf_sample_data_init (&data,0 ,0 ); - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw);perf_trace_buf_update (record, event_type);
kernel/trace/bpf_trace.c +2 -4
@@ -687,8 +687,7 @@ BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map, }perf_sample_data_init (sd,0 ,0 ); - sd->raw = &raw; - sd->sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (sd, &raw); err =__bpf_perf_event_output (regs, map, flags, sd);
@@ -746,8 +745,7 @@ u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,perf_fetch_caller_regs (regs);perf_sample_data_init (sd,0 ,0 ); - sd->raw = &raw; - sd->sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (sd, &raw); ret =__bpf_perf_event_output (regs, map, flags, sd); out:
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Peter Zijlstra
maintainerRe: [PATCH 3/8] perf/core: Add perf_sample_save_raw_data() helper
Quoted 16 lines · Namhyung Kim
include/linux/perf_event.h +12
-0@@ -1182,6 +1187,29 @@ static inline void perf_sample_save_callchain(struct perf_sample_data *data, data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; } +static inline void perf_sample_save_raw_data (struct perf_sample_data *data, +struct perf_raw_record *raw) +{ +struct perf_raw_frag *frag = &raw->frag; +u32 sum =0 ; +int size; + + do { + sum += frag->size; + if (perf_raw_frag_last(frag)) + break; + frag = frag->next;
include/linux/perf_event.h static inline void perf_sample_save_callchain(struct perf_sample_data *data, · line 1211
Quoted 10 lines
include/linux/perf_event.h +10
-0+ }while (1 ); + + size =round_up (sum +sizeof (u32),sizeof (u64)); + raw->size = size -sizeof (u32); + frag->pad = raw->size - sum; + + data->raw = raw; + data->dyn_size += size; + data->sample_flags |=PERF_SAMPLE_RAW ; +}
This might be a wee big for inline, but I suppose it doesn't matter too much.
Song Liu
maintainerRe: [PATCH 3/8] perf/core: Add perf_sample_save_raw_data() helper
In reply to Namhyung KimQuoted 164 lines · Namhyung Kim
When it saves the
raw_datato the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add theperf_sample_save_raw_data()helper and convert all call sites.Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- arch/s390/kernel/perf_cpum_cf.c | 4 +--- arch/s390/kernel/perf_pai_crypto.c | 4 +--- arch/s390/kernel/perf_pai_ext.c | 4 +--- arch/x86/events/amd/ibs.c | 3 +-- include/linux/perf_event.h | 33 +++++++++++++++++++++++++----- kernel/events/core.c | 31 +++++----------------------- kernel/trace/bpf_trace.c | 6 ++---- 7 files changed, 39 insertions(+), 46 deletions(-)arch/s390/kernel/perf_cpum_cf.c +1
-3@@ -662,9 +662,7 @@ static int cfdiag_push_sample(struct perf_event *event,if (event->attr.sample_type &PERF_SAMPLE_RAW ) { raw.frag.size = cpuhw->usedss; raw.frag.data = cpuhw->stop; - raw.size = raw.frag.size; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); } overflow =perf_event_overflow (event, &data, ®s);arch/s390/kernel/perf_pai_crypto.c +1
-3@@ -362,9 +362,7 @@ static int paicrypt_push_sample(void)if (event->attr.sample_type &PERF_SAMPLE_RAW ) { raw.frag.size = rawsize; raw.frag.data = cpump->save; - raw.size = raw.frag.size; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); } overflow =perf_event_overflow (event, &data, ®s);arch/s390/kernel/perf_pai_ext.c +1
-3@@ -451,9 +451,7 @@ static int paiext_push_sample(void)if (event->attr.sample_type &PERF_SAMPLE_RAW ) { raw.frag.size = rawsize; raw.frag.data = cpump->save; - raw.size = raw.frag.size; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); } overflow =perf_event_overflow (event, &data, ®s);arch/x86/events/amd/ibs.c +1
-2@@ -1110,8 +1110,7 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs) .data = ibs_data.data, }, }; - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw); }if (perf_ibs == &perf_ibs_op)include/linux/perf_event.h +28
-5@@ -95,6 +95,11 @@ struct perf_raw_record {u32 size; }; +static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag) +{ +return frag->pad <sizeof (u64); +} +/* * branch stack layout: * nr: number of taken branches stored in entries[] @@ -1182,6 +1187,29 @@ static inline void perf_sample_save_callchain(struct perf_sample_data *data, data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; } +static inline void perf_sample_save_raw_data (struct perf_sample_data *data, +struct perf_raw_record *raw) +{ +struct perf_raw_frag *frag = &raw->frag; +u32 sum =0 ; +int size; + +do { + sum += frag->size; +if (perf_raw_frag_last (frag)) +break ; + frag = frag->next; + }while (1 ); + + size =round_up (sum +sizeof (u32),sizeof (u64)); + raw->size = size -sizeof (u32); + frag->pad = raw->size - sum; + + data->raw = raw; + data->dyn_size += size; + data->sample_flags |=PERF_SAMPLE_RAW ; +} +/* * Clear all bitfields in the perf_branch_entry. * The to and from fields are not cleared because they are @@ -1690,11 +1718,6 @@ extern void perf_restore_debug_store(void);static inline void perf_restore_debug_store (void ) { }#endif -static __always_inline bool perf_raw_frag_last(const struct perf_raw_frag *frag) -{ -return frag->pad <sizeof (u64); -} -#define perf_output_put (handle, x) perf_output_copy((handle), &(x), sizeof(x))struct perf_pmu_events_attr {kernel/events/core.c +0
-19@@ -7588,30 +7588,10 @@ void perf_prepare_sample(struct perf_event_header *header,if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN )perf_sample_save_callchain (data, event, regs); -if (sample_type &PERF_SAMPLE_RAW ) { -struct perf_raw_record *raw = data->raw; -int size; - -if (raw && (data->sample_flags &PERF_SAMPLE_RAW )) { -struct perf_raw_frag *frag = &raw->frag; -u32 sum =0 ; - -do { - sum += frag->size; -if (perf_raw_frag_last (frag)) -break ; - frag = frag->next; - }while (1 ); - - size =round_up (sum +sizeof (u32),sizeof (u64)); - raw->size = size -sizeof (u32); - frag->pad = raw->size - sum; - }else {
kernel/events/core.c void perf_prepare_sample(struct perf_event_header *header, · line 7595
Quoted 10 lines
kernel/events/core.c +4
-5- size =sizeof (u64); - data->raw =NULL ; - } - - data->dyn_size += size; +if (filtered_sample_type &PERF_SAMPLE_RAW ) { + data->raw =NULL ; + data->dyn_size +=sizeof (u64); + data->sample_flags |=PERF_SAMPLE_RAW ; }
I don't quite follow this change, and the commit log doesn't seem to cover this part.
Quoted 39 lines
kernel/events/core.c +1
-2if (sample_type &PERF_SAMPLE_BRANCH_STACK ) {@@ -10127,8 +10107,7 @@ void perf_tp_event(u16 event_type, u64 count, void *record, int entry_size, };perf_sample_data_init (&data,0 ,0 ); - data.raw = &raw; - data.sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (&data, &raw);perf_trace_buf_update (record, event_type);kernel/trace/bpf_trace.c +2
-5@@ -687,8 +687,7 @@ BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map, }perf_sample_data_init (sd,0 ,0 ); - sd->raw = &raw; - sd->sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (sd, &raw); err =__bpf_perf_event_output (regs, map, flags, sd);@@ -746,8 +745,7 @@ u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,perf_fetch_caller_regs (regs);perf_sample_data_init (sd,0 ,0 ); - sd->raw = &raw; - sd->sample_flags |=PERF_SAMPLE_RAW ; +perf_sample_save_raw_data (sd, &raw); ret =__bpf_perf_event_output (regs, map, flags, sd); out: --2.39.0.314.g84b9a713c41-goog
Namhyung Kim
maintainerRe: [PATCH 3/8] perf/core: Add perf_sample_save_raw_data() helper
Hi Song,
Quoted 13 lines · Song Liu
Quoted 11 lines · Namhyung Kim
When it saves the
raw_datato the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add theperf_sample_save_raw_data()helper and convert all call sites.Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Namhyung Kim <[email protected]>---
[SNIP]
Quoted 32 lines
Quoted 32 lines
kernel/events/core.c +0
-24@@ -7588,30 +7588,10 @@ void perf_prepare_sample(struct perf_event_header *header,if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN )perf_sample_save_callchain (data, event, regs); -if (sample_type &PERF_SAMPLE_RAW ) { -struct perf_raw_record *raw = data->raw; -int size; - -if (raw && (data->sample_flags &PERF_SAMPLE_RAW )) { -struct perf_raw_frag *frag = &raw->frag; -u32 sum =0 ; - -do { - sum += frag->size; -if (perf_raw_frag_last (frag)) -break ; - frag = frag->next; - }while (1 ); - - size =round_up (sum +sizeof (u32),sizeof (u64)); - raw->size = size -sizeof (u32); - frag->pad = raw->size - sum; - }else { - size =sizeof (u64); - data->raw =NULL ; - } - - data->dyn_size += size;
Quoted 8 lines
kernel/events/core.c void perf_prepare_sample(struct perf_event_header *header, · line 7595
Quoted 5 lines
kernel/events/core.c +4
-0+if (filtered_sample_type &PERF_SAMPLE_RAW ) { + data->raw =NULL ; + data->dyn_size +=sizeof (u64); + data->sample_flags |=PERF_SAMPLE_RAW ; }I don't quite follow this change, and the commit log doesn't seem to cover this part.
It's for when the user requested RAW but no actual data. It assumes PMU drivers call perf_sample_save_raw_data() before perf_prepare_sample() if there's RAW data. So we can only handle the 'else' part in the original code.
Thanks,
Namhyung
Song Liu
maintainerRe: [PATCH 3/8] perf/core: Add perf_sample_save_raw_data() helper
Quoted 60 lines · Namhyung Kim
Hi Song,
Quoted 13 lines · Song Liu
Quoted 11 lines · Namhyung Kim
When it saves the
raw_datato the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add theperf_sample_save_raw_data()helper and convert all call sites.Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Namhyung Kim <[email protected]>---
[SNIP]
Quoted 32 lines
Quoted 32 lines
kernel/events/core.c +0
-24@@ -7588,30 +7588,10 @@ void perf_prepare_sample(struct perf_event_header *header,if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN )perf_sample_save_callchain (data, event, regs); -if (sample_type &PERF_SAMPLE_RAW ) { -struct perf_raw_record *raw = data->raw; -int size; - -if (raw && (data->sample_flags &PERF_SAMPLE_RAW )) { -struct perf_raw_frag *frag = &raw->frag; -u32 sum =0 ; - -do { - sum += frag->size; -if (perf_raw_frag_last (frag)) -break ; - frag = frag->next; - }while (1 ); - - size =round_up (sum +sizeof (u32),sizeof (u64)); - raw->size = size -sizeof (u32); - frag->pad = raw->size - sum; - }else { - size =sizeof (u64); - data->raw =NULL ; - } - - data->dyn_size += size;Quoted 8 lines
kernel/events/core.c void perf_prepare_sample(struct perf_event_header *header, · line 7595
Quoted 5 lines
kernel/events/core.c +4
-0+if (filtered_sample_type &PERF_SAMPLE_RAW ) { + data->raw =NULL ; + data->dyn_size +=sizeof (u64); + data->sample_flags |=PERF_SAMPLE_RAW ; }I don't quite follow this change, and the commit log doesn't seem to cover this part.
Quoted 4 lines
It's for when the user requested RAW but no actual data. It assumes PMU drivers call
perf_sample_save_raw_data()beforeperf_prepare_sample()if there's RAW data. So we can only handle the 'else' part in the original code.
Got it. Thanks for the explanation.
Song
Namhyung Kim
maintainerPATCH 4/8 perf/core: Add perf_sample_save_brstack() helper
In reply to Namhyung KimWhen it saves the branch stack to the perf sample data, it needs to update the sample flags and the dynamic size. To make sure this, add the perf_sample_save_brstack() helper and convert all call sites.
--- arch/powerpc/perf/core-book3s.c | 3 +- arch/x86/events/amd/core.c | 6 +-- arch/x86/events/intel/core.c | 6 +-- arch/x86/events/intel/ds.c | 9 ++--- include/linux/perf_event.h | 66 ++++++++++++++++++++------------- kernel/events/core.c | 16 +++----- 6 files changed, 53 insertions(+), 53 deletions(-)
arch/powerpc/perf/core-book3s.c +1 -2
@@ -2313,8 +2313,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,struct cpu_hw_events *cpuhw; cpuhw =this_cpu_ptr (&cpu_hw_events);power_pmu_bhrb_read (event, cpuhw); - data.br_stack = &cpuhw->bhrb_stack; - data.sample_flags |=PERF_SAMPLE_BRANCH_STACK ; +perf_sample_save_brstack (&data, event, &cpuhw->bhrb_stack); } if (event->attr.sample_type & PERF_SAMPLE_DATA_SRC &&
arch/x86/events/amd/core.c +2 -4
@@ -928,10 +928,8 @@ static int amd_pmu_v2_handle_irq(struct pt_regs *regs)if (!x86_perf_event_set_period (event))continue ; -if (has_branch_stack (event)) { - data.br_stack = &cpuc->lbr_stack; - data.sample_flags |=PERF_SAMPLE_BRANCH_STACK ; - } +if (has_branch_stack (event)) +perf_sample_save_brstack (&data, event, &cpuc->lbr_stack);if (perf_event_overflow (event, &data, regs))x86_pmu_stop (event,0 );
arch/x86/events/intel/core.c +2 -4
@@ -3036,10 +3036,8 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)perf_sample_data_init (&data,0 , event->hw.last_period); -if (has_branch_stack (event)) { - data.br_stack = &cpuc->lbr_stack; - data.sample_flags |=PERF_SAMPLE_BRANCH_STACK ; - } +if (has_branch_stack (event)) +perf_sample_save_brstack (&data, event, &cpuc->lbr_stack);if (perf_event_overflow (event, &data, regs))x86_pmu_stop (event,0 );
arch/x86/events/intel/ds.c +3 -6
@@ -1720,10 +1720,8 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event, data->sample_flags |=PERF_SAMPLE_TIME ; } -if (has_branch_stack (event)) { - data->br_stack = &cpuc->lbr_stack; - data->sample_flags |=PERF_SAMPLE_BRANCH_STACK ; - } +if (has_branch_stack (event)) +perf_sample_save_brstack (data, event, &cpuc->lbr_stack); } static void adaptive_pebs_save_regs(struct pt_regs *regs,
@@ -1883,8 +1881,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,if (has_branch_stack (event)) {intel_pmu_store_pebs_lbrs (lbr); - data->br_stack = &cpuc->lbr_stack; - data->sample_flags |=PERF_SAMPLE_BRANCH_STACK ; +perf_sample_save_brstack (data, event, &cpuc->lbr_stack); } }
include/linux/perf_event.h +40 -26
82 lines not on this page · raw message
kernel/events/core.c +5 -11
@@ -7317,7 +7317,7 @@ void perf_output_sample(struct perf_output_handle *handle, }if (sample_type &PERF_SAMPLE_BRANCH_STACK ) { -if (data->sample_flags &PERF_SAMPLE_BRANCH_STACK ) { +if (data->br_stack) {size_t size; size = data->br_stack->nr
@@ -7594,16 +7594,10 @@ void perf_prepare_sample(struct perf_event_header *header, data->sample_flags |=PERF_SAMPLE_RAW ; } -if (sample_type &PERF_SAMPLE_BRANCH_STACK ) { -int size =sizeof (u64);/* nr */ -if (data->sample_flags &PERF_SAMPLE_BRANCH_STACK ) { -if (branch_sample_hw_index (event)) - size +=sizeof (u64); - - size += data->br_stack->nr - *sizeof (struct perf_branch_entry ); - } - data->dyn_size += size; +if (filtered_sample_type &PERF_SAMPLE_BRANCH_STACK ) { + data->br_stack =NULL ; + data->dyn_size +=sizeof (u64); + data->sample_flags |=PERF_SAMPLE_BRANCH_STACK ; }if (sample_type & (PERF_SAMPLE_REGS_USER |PERF_SAMPLE_STACK_USER ))
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Namhyung Kim
maintainerPATCH 5/8 perf/core: Set data->sample_flags in perf_prepare_sample()
In reply to Namhyung KimThe perf_prepare_sample() sets the perf_sample_data according to the attr->sample_type before copying it to the ring buffer. But BPF also wants to access the sample data so it needs to prepare the sample even before the regular path.
That means the perf_prepare_sample() can be called more than once. Set the data->sample_flags consistently so that it can indicate which fields are set already and skip them if sets.
Also update the filtered_sample_type to have the dependent flags to reduce the number of branches.
--- kernel/events/core.c | 85 +++++++++++++++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 20 deletions(-)
kernel/events/core.c +65 -20
@@ -7053,6 +7053,14 @@ static void perf_aux_sample_output(struct perf_event *event,ring_buffer_put (rb); } +/* +* A set of common sample data types saved even for non-sample records +* when event->attr.sample_id_all is set. +*/ +#define PERF_SAMPLE_ID_ALL (PERF_SAMPLE_TID | PERF_SAMPLE_TIME | \ + PERF_SAMPLE_CPU | PERF_SAMPLE_ID | \ + PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_STREAM_ID) + static void __perf_event_header__init_id(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event,
@@ -7561,6 +7569,11 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return callchain ?: &__empty_callchain; } +static __always_inline u64 __cond_set(u64 flags, u64 s, u64 d) +{ +return d * !!(flags & s); +} + void perf_prepare_sample(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event,
@@ -7576,14 +7589,25 @@ void perf_prepare_sample(struct perf_event_header *header, header->misc |=perf_misc_flags (regs);/* -* Clear the sample flags that have already been done by the -* PMU driver. +* Add the sample flags that are dependent to others. And clear the +* sample flags that have already been done by the PMU driver. */ - filtered_sample_type = sample_type & ~data->sample_flags; + filtered_sample_type = sample_type; + filtered_sample_type |=__cond_set (sample_type,PERF_SAMPLE_CODE_PAGE_SIZE , +PERF_SAMPLE_IP ); + filtered_sample_type |=__cond_set (sample_type,PERF_SAMPLE_DATA_PAGE_SIZE | +PERF_SAMPLE_PHYS_ADDR ,PERF_SAMPLE_ADDR ); + filtered_sample_type |=__cond_set (sample_type,PERF_SAMPLE_STACK_USER , +PERF_SAMPLE_REGS_USER ); + filtered_sample_type &= ~data->sample_flags; +__perf_event_header__init_id (header, data, event, filtered_sample_type); + data->sample_flags |= sample_type &PERF_SAMPLE_ID_ALL ; -if (sample_type & (PERF_SAMPLE_IP |PERF_SAMPLE_CODE_PAGE_SIZE )) +if (filtered_sample_type &PERF_SAMPLE_IP ) { data->ip =perf_instruction_pointer (regs); + data->sample_flags |=PERF_SAMPLE_IP ; + }if (filtered_sample_type &PERF_SAMPLE_CALLCHAIN )perf_sample_save_callchain (data, event, regs);
@@ -7600,10 +7624,15 @@ void perf_prepare_sample(struct perf_event_header *header, data->sample_flags |=PERF_SAMPLE_BRANCH_STACK ; } -if (sample_type & (PERF_SAMPLE_REGS_USER |PERF_SAMPLE_STACK_USER )) +if (filtered_sample_type &PERF_SAMPLE_REGS_USER )perf_sample_regs_user (&data->regs_user, regs); -if (sample_type &PERF_SAMPLE_REGS_USER ) { +/* +* It cannot use the filtered_sample_type here as REGS_USER can be set +* by STACK_USER (using __cond_set() above) and we don't want to update +* the dyn_size if it's not requested by users. +*/ +if ((sample_type & ~data->sample_flags) &PERF_SAMPLE_REGS_USER ) {/* regs dump ABI info */ int size =sizeof (u64);
@@ -7613,9 +7642,10 @@ void perf_prepare_sample(struct perf_event_header *header, } data->dyn_size += size; + data->sample_flags |=PERF_SAMPLE_REGS_USER ; } -if (sample_type &PERF_SAMPLE_STACK_USER ) { +if (filtered_sample_type &PERF_SAMPLE_STACK_USER ) {/* * Either we need PERF_SAMPLE_STACK_USER bit to be always * processed as the last one or have additional check added
@@ -7638,23 +7668,30 @@ void perf_prepare_sample(struct perf_event_header *header, data->stack_user_size = stack_size; data->dyn_size += size; + data->sample_flags |=PERF_SAMPLE_STACK_USER ; } -if (filtered_sample_type &PERF_SAMPLE_WEIGHT_TYPE ) +if (filtered_sample_type &PERF_SAMPLE_WEIGHT_TYPE ) { data->weight.full =0 ; + data->sample_flags |=PERF_SAMPLE_WEIGHT_TYPE ; + } -if (filtered_sample_type &PERF_SAMPLE_DATA_SRC ) +if (filtered_sample_type &PERF_SAMPLE_DATA_SRC ) { data->data_src.val =PERF_MEM_NA ; + data->sample_flags |=PERF_SAMPLE_DATA_SRC ; + } -if (filtered_sample_type &PERF_SAMPLE_TRANSACTION ) +if (filtered_sample_type &PERF_SAMPLE_TRANSACTION ) { data->txn =0 ; + data->sample_flags |=PERF_SAMPLE_TRANSACTION ; + } -if (sample_type & (PERF_SAMPLE_ADDR |PERF_SAMPLE_PHYS_ADDR |PERF_SAMPLE_DATA_PAGE_SIZE )) { -if (filtered_sample_type &PERF_SAMPLE_ADDR ) - data->addr =0 ; +if (filtered_sample_type &PERF_SAMPLE_ADDR ) { + data->addr =0 ; + data->sample_flags |=PERF_SAMPLE_ADDR ; } -if (sample_type &PERF_SAMPLE_REGS_INTR ) { +if (filtered_sample_type &PERF_SAMPLE_REGS_INTR ) {/* regs dump ABI info */ int size =sizeof (u64);
@@ -7667,19 +7704,22 @@ void perf_prepare_sample(struct perf_event_header *header, } data->dyn_size += size; + data->sample_flags |=PERF_SAMPLE_REGS_INTR ; } -if (sample_type &PERF_SAMPLE_PHYS_ADDR && - filtered_sample_type &PERF_SAMPLE_PHYS_ADDR ) +if (filtered_sample_type &PERF_SAMPLE_PHYS_ADDR ) { data->phys_addr =perf_virt_to_phys (data->addr); + data->sample_flags |=PERF_SAMPLE_PHYS_ADDR ; + }#ifdef CONFIG_CGROUP_PERF -if (sample_type &PERF_SAMPLE_CGROUP ) { +if (filtered_sample_type &PERF_SAMPLE_CGROUP ) {struct cgroup *cgrp;/* protected by RCU */ cgrp =task_css_check (current, perf_event_cgrp_id,1 )->cgroup; data->cgroup =cgroup_id (cgrp); + data->sample_flags |=PERF_SAMPLE_CGROUP ; }#endif
@@ -7688,13 +7728,17 @@ void perf_prepare_sample(struct perf_event_header *header,* require PERF_SAMPLE_ADDR, kernel implicitly retrieve the data->addr, * but the value will not dump to the userspace. */ -if (sample_type &PERF_SAMPLE_DATA_PAGE_SIZE ) +if (filtered_sample_type &PERF_SAMPLE_DATA_PAGE_SIZE ) { data->data_page_size =perf_get_page_size (data->addr); + data->sample_flags |=PERF_SAMPLE_DATA_PAGE_SIZE ; + } -if (sample_type &PERF_SAMPLE_CODE_PAGE_SIZE ) +if (filtered_sample_type &PERF_SAMPLE_CODE_PAGE_SIZE ) { data->code_page_size =perf_get_page_size (data->ip); + data->sample_flags |=PERF_SAMPLE_CODE_PAGE_SIZE ; + } -if (sample_type &PERF_SAMPLE_AUX ) { +if (filtered_sample_type &PERF_SAMPLE_AUX ) {u64 size; header->size +=sizeof (u64);/* size */
@@ -7712,6 +7756,7 @@ void perf_prepare_sample(struct perf_event_header *header,WARN_ON_ONCE (size + header->size >U16_MAX ); data->dyn_size += size +sizeof (u64);/* size above */ + data->sample_flags |=PERF_SAMPLE_AUX ; } header->size += data->dyn_size;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Peter Zijlstra
maintainerRe: [PATCH 5/8] perf/core: Set data->sample_flags in perf_prepare_sample()
Quoted 8 lines · Namhyung Kim
kernel/events/core.c +2
-0@@ -7053,6 +7053,14 @@ static void perf_aux_sample_output(struct perf_event *event,ring_buffer_put (rb); } +/* +* A set of common sample data types saved even for non-sample records
kernel/events/core.c static void perf_aux_sample_output(struct perf_event *event,
Quoted 10 lines
kernel/events/core.c +7
-0+* when event->attr.sample_id_all is set. +*/ +#define PERF_SAMPLE_ID_ALL (PERF_SAMPLE_TID | PERF_SAMPLE_TIME | \ + PERF_SAMPLE_CPU | PERF_SAMPLE_ID | \ + PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_STREAM_ID) +__perf_event_header__init_id (header, data, event, filtered_sample_type); + data->sample_flags |= sample_type &PERF_SAMPLE_ID_ALL ;
Why not update __perf_event_header__init_id() ?
Namhyung Kim
maintainerRe: [PATCH 5/8] perf/core: Set data->sample_flags in perf_prepare_sample()
Quoted 19 lines · Peter Zijlstra
Quoted 14 lines · Namhyung Kim
kernel/events/core.c +8
-0@@ -7053,6 +7053,14 @@ static void perf_aux_sample_output(struct perf_event *event,ring_buffer_put (rb); } +/* +* A set of common sample data types saved even for non-sample records +* when event->attr.sample_id_all is set. +*/ +#define PERF_SAMPLE_ID_ALL (PERF_SAMPLE_TID | PERF_SAMPLE_TIME | \ + PERF_SAMPLE_CPU | PERF_SAMPLE_ID | \ + PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_STREAM_ID) +
Quoted 4 lines
kernel/events/core.c void perf_prepare_sample(struct perf_event_header *header, · line 7605
Quoted 2 lines
kernel/events/core.c +1
-0__perf_event_header__init_id (header, data, event, filtered_sample_type); + data->sample_flags |= sample_type &PERF_SAMPLE_ID_ALL ;Why not update
__perf_event_header__init_id()?
My concern was the non-sample records which don't check the sample flags at all. So I'd like to skip the write for them.
But that's not a big deal. I can move it if you want.
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 6/8 perf/core: Do not pass header for sample id init
In reply to Namhyung KimThe only thing it does for header in __perf_event_header__init_id() is to update the header size with event->id_header_size. We can do this outside and get rid of the argument for the later change.
--- kernel/events/core.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/events/core.c +7 -7
@@ -7061,13 +7061,11 @@ static void perf_aux_sample_output(struct perf_event *event, PERF_SAMPLE_CPU | PERF_SAMPLE_ID | \ PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_STREAM_ID) -static void __perf_event_header__init_id (struct perf_event_header *header, -struct perf_sample_data *data, +static void __perf_event_header__init_id (struct perf_sample_data *data,struct perf_event *event,u64 sample_type) { data->type = event->attr.sample_type; - header->size += event->id_header_size;if (sample_type &PERF_SAMPLE_TID ) {/* namespace issues */
@@ -7094,8 +7092,10 @@ void perf_event_header__init_id(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event) { -if (event->attr.sample_id_all) -__perf_event_header__init_id (header, data, event, event->attr.sample_type); +if (event->attr.sample_id_all) { + header->size += event->id_header_size; +__perf_event_header__init_id (data, event, event->attr.sample_type); + } } static void __perf_event__output_id_sample(struct perf_output_handle *handle,
@@ -7583,7 +7583,7 @@ void perf_prepare_sample(struct perf_event_header *header,u64 filtered_sample_type; header->type =PERF_RECORD_SAMPLE ; - header->size =sizeof (*header) + event->header_size; + header->size =sizeof (*header) + event->header_size + event->id_header_size; header->misc =0 ; header->misc |=perf_misc_flags (regs);
@@ -7601,7 +7601,7 @@ void perf_prepare_sample(struct perf_event_header *header, PERF_SAMPLE_REGS_USER); filtered_sample_type &= ~data->sample_flags; -__perf_event_header__init_id (header, data, event, filtered_sample_type); +__perf_event_header__init_id (data, event, filtered_sample_type); data->sample_flags |= sample_type &PERF_SAMPLE_ID_ALL ;if (filtered_sample_type &PERF_SAMPLE_IP ) {
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Re: [PATCH 6/8] perf/core: Do not pass header for sample id init
Quoted 6 lines · Namhyung Kim
The only thing it does for header in
__perf_event_header__init_id()is to update the header size withevent->id_header_size. We can do this outside and get rid of the argument for the later change.Signed-off-by Namhyung Kim <[email protected]>
Namhyung Kim
maintainerPATCH 7/8 perf/core: Introduce perf_prepare_header()
In reply to Namhyung KimFactor out perf_prepare_header() so that it can call perf_prepare_sample() without a header if not needed.
Also it checks the filtered_sample_type to avoid duplicate work when perf_prepare_sample() is called twice (or more).
--- arch/s390/kernel/perf_cpum_sf.c | 3 ++- arch/x86/events/intel/ds.c | 3 ++- include/linux/perf_event.h | 16 +++++++++++++- kernel/events/core.c | 38 +++++++++++++++++++++------------ 4 files changed, 43 insertions(+), 17 deletions(-)
arch/s390/kernel/perf_cpum_sf.c +2 -1
@@ -671,7 +671,8 @@ static void cpumsf_output_event_pid(struct perf_event *event,/* Protect callchain buffers, tasks */ rcu_read_lock (); -perf_prepare_sample (&header, data, event, regs); +perf_prepare_sample (data, event, regs); +perf_prepare_header (&header, data, event, regs);if (perf_output_begin (&handle, data, event, header.size)) goto out;
arch/x86/events/intel/ds.c +2 -1
@@ -807,7 +807,8 @@ int intel_pmu_drain_bts_buffer(void)* the sample. */ rcu_read_lock (); -perf_prepare_sample (&header, &data, event, ®s); +perf_prepare_sample (&data, event, ®s); +perf_prepare_header (&header, &data, event, ®s);if (perf_output_begin (&handle, &data, event, header.size * (top - base - skip)))
include/linux/perf_event.h +15 -1
@@ -1250,6 +1250,17 @@ static inline void perf_sample_save_brstack(struct perf_sample_data *data, data->sample_flags |=PERF_SAMPLE_BRANCH_STACK ; } +static inline u32 perf_sample_data_size (struct perf_sample_data *data, +struct perf_event *event) +{ +u32 size =sizeof (struct perf_event_header ); + + size += event->header_size + event->id_header_size; + size += data->dyn_size; + +return size; +} +/* * Clear all bitfields in the perf_branch_entry. * The to and from fields are not cleared because they are
@@ -1271,7 +1282,10 @@ extern void perf_output_sample(struct perf_output_handle *handle,struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event); -extern void perf_prepare_sample (struct perf_event_header *header, +extern void perf_prepare_sample (struct perf_sample_data *data, +struct perf_event *event, +struct pt_regs *regs); +extern void perf_prepare_header (struct perf_event_header *header,struct perf_sample_data *data,struct perf_event *event,struct pt_regs *regs);
kernel/events/core.c +24 -14
@@ -7574,20 +7574,13 @@ static __always_inline u64 __cond_set(u64 flags, u64 s, u64 d)return d * !!(flags & s); } -void perf_prepare_sample (struct perf_event_header *header, -struct perf_sample_data *data, +void perf_prepare_sample (struct perf_sample_data *data,struct perf_event *event,struct pt_regs *regs) {u64 sample_type = event->attr.sample_type;u64 filtered_sample_type; - header->type =PERF_RECORD_SAMPLE ; - header->size =sizeof (*header) + event->header_size + event->id_header_size; - - header->misc =0 ; - header->misc |=perf_misc_flags (regs); -/* * Add the sample flags that are dependent to others. And clear the * sample flags that have already been done by the PMU driver.
@@ -7601,6 +7594,12 @@ void perf_prepare_sample(struct perf_event_header *header, PERF_SAMPLE_REGS_USER); filtered_sample_type &= ~data->sample_flags; +if (filtered_sample_type ==0 ) { +/* Make sure it has the correct data->type for output */ + data->type = event->attr.sample_type; +return ; + } +__perf_event_header__init_id (data, event, filtered_sample_type); data->sample_flags |= sample_type &PERF_SAMPLE_ID_ALL ;
@@ -7653,9 +7652,10 @@ void perf_prepare_sample(struct perf_event_header *header,* up the rest of the sample size. */ u16 stack_size = event->attr.sample_stack_user; +u16 header_size =perf_sample_data_size (data, event);u16 size =sizeof (u64); - stack_size =perf_sample_ustack_size (stack_size, header->size, + stack_size =perf_sample_ustack_size (stack_size, header_size, data->regs_user.regs);/*
@@ -7740,8 +7740,9 @@ void perf_prepare_sample(struct perf_event_header *header,if (filtered_sample_type &PERF_SAMPLE_AUX ) {u64 size; +u16 header_size =perf_sample_data_size (data, event); - header->size +=sizeof (u64);/* size */ + header_size +=sizeof (u64);/* size */ /* * Given the 16bit nature of header::size, an AUX sample can
@@ -7749,17 +7750,25 @@ void perf_prepare_sample(struct perf_event_header *header,* Make sure this doesn't happen by using up to U16_MAX bytes * per sample in total (rounded down to 8 byte boundary). */ - size =min_t (size_t,U16_MAX - header->size, + size =min_t (size_t,U16_MAX - header_size, event->attr.aux_sample_size); size =rounddown (size,8 ); size =perf_prepare_sample_aux (event, data, size); -WARN_ON_ONCE (size + header->size >U16_MAX ); +WARN_ON_ONCE (size + header_size >U16_MAX ); data->dyn_size += size +sizeof (u64);/* size above */ data->sample_flags |=PERF_SAMPLE_AUX ; } +} - header->size += data->dyn_size; +void perf_prepare_header (struct perf_event_header *header, +struct perf_sample_data *data, +struct perf_event *event, +struct pt_regs *regs) +{ + header->type =PERF_RECORD_SAMPLE ; + header->size =perf_sample_data_size (data, event); + header->misc =perf_misc_flags (regs);/* * If you're adding more sample types here, you likely need to do
@@ -7788,7 +7797,8 @@ __perf_event_output(struct perf_event *event,/* protect the callchain buffers */ rcu_read_lock (); -perf_prepare_sample (&header, data, event, regs); +perf_prepare_sample (data, event, regs); +perf_prepare_header (&header, data, event, regs); err =output_begin (&handle, data, event, header.size);if (err)
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Re: [PATCH 7/8] perf/core: Introduce perf_prepare_header()
Quoted 11 lines · Namhyung Kim
Factor out
perf_prepare_header()so that it can callperf_prepare_sample()without a header if not needed.Also it checks the
filtered_sample_typeto avoid duplicate work whenperf_prepare_sample()is called twice (or more).Suggested-by Peter Zijlstr <[email protected]>Signed-off-by Namhyung Kim <[email protected]>
Namhyung Kim
maintainerPATCH 8/8 perf/core: Call perf_prepare_sample() before running BPF
In reply to Namhyung KimAs BPF can access sample data, it needs to populate the data. Also remove the logic to get the callchain specifically as it's covered by the perf_prepare_sample() now.
--- kernel/events/core.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
kernel/events/core.c +1 -7
@@ -10363,13 +10363,7 @@ static void bpf_overflow_handler(struct perf_event *event,rcu_read_lock (); prog =READ_ONCE (event->prog);if (prog) { -if (prog->call_get_stack && - (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN ) && - !(data->sample_flags &PERF_SAMPLE_CALLCHAIN )) { - data->callchain =perf_callchain (event, regs); - data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; - } - +perf_prepare_sample (data, event, regs); ret =bpf_prog_run (prog, &ctx); }rcu_read_unlock ();
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Re: [PATCH 8/8] perf/core: Call perf_prepare_sample() before running BPF
Quoted 8 lines · Namhyung Kim
As BPF can access sample data, it needs to populate the data. Also remove the logic to get the callchain specifically as it's covered by the
perf_prepare_sample()now.
Quoted 26 lines
--- kernel/events/core.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)kernel/events/core.c +1
-8@@ -10363,13 +10363,7 @@ static void bpf_overflow_handler(struct perf_event *event,rcu_read_lock (); prog =READ_ONCE (event->prog);if (prog) { -if (prog->call_get_stack && - (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN ) && - !(data->sample_flags &PERF_SAMPLE_CALLCHAIN )) { - data->callchain =perf_callchain (event, regs); - data->sample_flags |=PERF_SAMPLE_CALLCHAIN ; - } - +perf_prepare_sample (data, event, regs); ret =bpf_prog_run (prog, &ctx); }rcu_read_unlock (); --2.39.0.314.g84b9a713c41-goog
Peter Zijlstra
maintainerQuoted 9 lines · Namhyung Kim
Namhyung Kim (8):
perf/core: Save the dynamic parts of sample data size perf/core: Addperf_sample_save_callchain()helper perf/core: Addperf_sample_save_raw_data()helper perf/core: Addperf_sample_save_brstack()helper perf/core: Setdata->sample_flagsinperf_prepare_sample()perf/core: Do not pass header for sample id init perf/core: Introduceperf_prepare_header()perf/core: Callperf_prepare_sample()before running BPF
Aside from a few small niggles, this looks really good, Thanks!
Quoted 38 lines · Namhyung Kim
Hello,
The
perf_prepare_sample()is to fill the perf sample data and update the header info before sending it to the ring buffer. But we want to use it for BPF overflow handler so that it can access the sample data to filter relevant ones.Changes in v2)
* the layout change is merged
* reduce branches using__cond_set(PeterZ)
* add helpers to set dynamic sample data (PeterZ)
* introduceperf_prepare_header()(PeterZ)
* callperf_prepare_sample()beforebpf_overflow_handlerunconditionallyThis means the
perf_prepare_handler()can be called more than once. To avoid duplicate work, use thedata->sample_flagsand save the data size.I also added a few of helpers to set those information accordingly. But it looks some fields like
REGS_USER,STACK_USERand AUX are saved in theperf_prepare_sample()so I didn't add the helpers for them.After than we can just check the
filtered_sample_typeflags begin zero to determine if it has more work. In that case, it needs to update thedata->typesince it's possible to miss when PMU driver sets all required sample flags before callingperf_prepare_sample().The code is also available at 'perf/prepare-sample-v2' branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThanks,
NamhyungCc Song Liu <[email protected]>
Quoted 9 lines
Namhyung Kim (8):
perf/core: Save the dynamic parts of sample data size perf/core: Addperf_sample_save_callchain()helper perf/core: Addperf_sample_save_raw_data()helper perf/core: Addperf_sample_save_brstack()helper perf/core: Setdata->sample_flagsinperf_prepare_sample()perf/core: Do not pass header for sample id init perf/core: Introduceperf_prepare_header()perf/core: Callperf_prepare_sample()before running BPF
lgtm, I ran the bpf selftests on top of that and it's ok
jirka
Quoted 20 lines
arch/powerpc/perf/core-book3s.c | 3 +- arch/s390/kernel/perf_cpum_cf.c | 4 +- arch/s390/kernel/perf_cpum_sf.c | 3 +- arch/s390/kernel/perf_pai_crypto.c | 4 +- arch/s390/kernel/perf_pai_ext.c | 4 +- arch/x86/events/amd/core.c | 6 +- arch/x86/events/amd/ibs.c | 9 +- arch/x86/events/intel/core.c | 6 +- arch/x86/events/intel/ds.c | 24 ++-- include/linux/perf_event.h | 133 +++++++++++++----- kernel/events/core.c | 207 ++++++++++++++++------------- kernel/trace/bpf_trace.c | 6 +- 12 files changed, 236 insertions(+), 173 deletions(-)base-commit:
9fcad995c6c52cc9791f7ee9f1386a5684055f9cSignature · 1 line
2.39.0.314.g84b9a713c41-goog
Namhyung Kim
maintainerHi Jiri,
Quoted 50 lines · Jiri Olsa
Quoted 38 lines · Namhyung Kim
Hello,
The
perf_prepare_sample()is to fill the perf sample data and update the header info before sending it to the ring buffer. But we want to use it for BPF overflow handler so that it can access the sample data to filter relevant ones.Changes in v2)
* the layout change is merged
* reduce branches using__cond_set(PeterZ)
* add helpers to set dynamic sample data (PeterZ)
* introduceperf_prepare_header()(PeterZ)
* callperf_prepare_sample()beforebpf_overflow_handlerunconditionallyThis means the
perf_prepare_handler()can be called more than once. To avoid duplicate work, use thedata->sample_flagsand save the data size.I also added a few of helpers to set those information accordingly. But it looks some fields like
REGS_USER,STACK_USERand AUX are saved in theperf_prepare_sample()so I didn't add the helpers for them.After than we can just check the
filtered_sample_typeflags begin zero to determine if it has more work. In that case, it needs to update thedata->typesince it's possible to miss when PMU driver sets all required sample flags before callingperf_prepare_sample().The code is also available at 'perf/prepare-sample-v2' branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThanks,
NamhyungCc Song Liu <[email protected]>Quoted 9 lines
Namhyung Kim (8):
perf/core: Save the dynamic parts of sample data size perf/core: Addperf_sample_save_callchain()helper perf/core: Addperf_sample_save_raw_data()helper perf/core: Addperf_sample_save_brstack()helper perf/core: Setdata->sample_flagsinperf_prepare_sample()perf/core: Do not pass header for sample id init perf/core: Introduceperf_prepare_header()perf/core: Callperf_prepare_sample()before running BPF
Quoted 1 line
lgtm, I ran the bpf selftests on top of that and it's ok
Thanks for your review. I'll add your Acked-by and Tested-by in the v3. :)
Thanks,
Namhyung