perf: IRQ-bound performance events
Subsystems (3)
v2 cover perf: IRQ-bound performance events
Hello,
This is version 2 of RFC "perf: IRQ-bound performance events". That is an introduction of IRQ-bound performance events - ones that only count in a context of a hardware interrupt handler. Ingo suggested to extend this functionality to softirq and threaded handlers as well:
[quote]
Looks useful.
I think the main challenges are:
- Creating a proper ABI for all this:
- IRQ numbers alone are probably not specific enough: we'd also want to be more specific to match on handler names - or handler numbers if the handler name is not unique.
- another useful variant would be where IRQ numbers are too specific:
something like 'perf top irq' would be a natural thing to do, to see only overhead in hardirq execution - without limiting it to a specific handler. An 'all irq contexts' wildcard concept?
- Covering softirqs as well. If we handle both hardirqs and softirqs, then we are pretty much feature complete: all major context types that the Linux kernel cares about are covered in instrumentation. For things like networking the softirq overhead is obviously very important, and for example on routers it will do most of the execution.
- Covering threaded IRQs as well, in a similar model. So if someone types 'perf top irq', and some IRQ handlers are running threaded, those should probaby be included as well.
- Making the tooling friendlier: 'perf top irq' would be useful, and accepting handler names would be useful as well.
The runtime overhead of your patches seems to be pretty low: when no IRQ contexts are instrumented then it's a single 'is the list empty' check at context scheduling time. That looks acceptable.
Regarding the ABI and IRQ/softirq context enumeration you are breaking lots of new ground here, because unlike tasks, cgroups and CPUs the IRQ execution contexts do not have a good programmatically accessible namespace (yet). So it has to be thought out pretty well I think, but once we have it, it will be a lovely feature IMO.
Thanks,
Ingo
[/quote]
This RFC version addresses only "Creating a proper ABI for all this" suggestion for kernel side. Each hardware interrupt context performance event is assigned a bitmask where each bit indicates whether the action with the bit's number should be measured or not. A task to convert handler name(s), wildcards etc. to bitmasks to be off-loaded to user level and is not yet supported.
The kernel side implementation revolves around a need to make enabling and disabling performance counters in hardware interrupt context as fast as possible. For this reason a new command PERF_EVENT_IOC_SET_HARDIRQ pre- allocates and initializes a per-CPU array with performance events destined for this IRQ, before the event is started. Once, the action (aka ISR) is called, another pre-allocated per-CPU array gets initialized with events for this action and then submitted to relevant PMUs using a new PMU callback:
void (*start_hardirq)(struct perf_event *events[], int count);
Since the performance events are expected known to PMUs, it should be able to enable the counters in a performance-aware manner. I.e. in the sample patch for Intel PMU this goal is achieved with a single pass thru the 'events' array and a single call to WRMSR instruction.
By contrast with version 1 of this RFC, per-CPU lists are replaced with per-CPU arrays whenever possible. With an assumption there will be normally no more than a dozen of events being counted at a time, it expected to add to cache hit rate when the events are enabled or disabled from the hardware interrupt context.
Besides the original purpose the design accommodates an ability to run the same performance counter for any combination of actions and IRQs, which makes possible a unlimited level of flexibility. This feature is not yet supported with the perf tool, though.
Although the whole idea seems simple, I am not sure if it fits into the current perf design and does not break some ground assumptions. The very purpose of this RFC is to ensure the taken approach is correct.
This RFC interleaves with toggling events introduced some time ago. While addressing a similar problem, it does not appear the toggling events could count on per-action basis, nor to provide a flexibility this RFC assumes. The performance is also a major concern. Perhaps, the two designs could be merged, but at the moment I am not realizing how. Suggestions are very welcomed.
The perf tool update for now is just a hack to make possible kernel side testing. Here is a sample session against IRQ #8, 'rtc0' device:
# ./tools/perf/perf stat -a -e L1-dcache-load-misses:k --hardirq=8 sleep 1
Performance counter stats for 'system wide':
0 L1-dcache-load-misses
1.001190052 seconds time elapsed# ./tools/perf/perf stat -a -e L1-dcache-load-misses:k --hardirq=8 hwclock --test Sat 04 Jan 2014 12:16:36 EST -0.484913 seconds
Performance counter stats for 'system wide':
374 L1-dcache-load-misses
0.485939068 seconds time elapsedThe patchset is against Arnaldo's repo, in "perf/core" branch.
The tree could be found in "pci-next-msi-v5" branch in repo:
https://github.com/a-gordeev/linux.git
Thanks!
Alexander Gordeev (4):
perf/core: IRQ-bound performance events
perf/x86: IRQ-bound performance events
perf/x86/Intel: IRQ-bound performance events
perf/tool: IRQ-bound performance events
arch/x86/kernel/cpu/perf_event.c | 55 +++++- arch/x86/kernel/cpu/perf_event.h | 15 ++ arch/x86/kernel/cpu/perf_event_amd.c | 2 + arch/x86/kernel/cpu/perf_event_intel.c | 72 ++++++- arch/x86/kernel/cpu/perf_event_knc.c | 2 + arch/x86/kernel/cpu/perf_event_p4.c | 2 + arch/x86/kernel/cpu/perf_event_p6.c | 2 + include/linux/irq.h | 10 + include/linux/irqdesc.h | 4 + include/linux/perf_event.h | 24 ++ include/uapi/linux/perf_event.h | 14 ++- kernel/events/Makefile | 2 +- kernel/events/core.c | 142 ++++++++++++- kernel/events/hardirq.c | 370 ++++++++++++++++++++++++++++++++ kernel/irq/handle.c | 7 +- kernel/irq/irqdesc.c | 15 ++ tools/perf/builtin-stat.c | 9 + tools/perf/util/evlist.c | 38 ++++ tools/perf/util/evlist.h | 3 + tools/perf/util/evsel.c | 8 + tools/perf/util/evsel.h | 3 + tools/perf/util/parse-events.c | 24 ++ tools/perf/util/parse-events.h | 1 + 23 files changed, 811 insertions(+), 13 deletions(-) create mode 100644 kernel/events/hardirq.c
Signature · 1 line
1.7.7.6
RFC PATCH v2 3/4 perf/x86/Intel: IRQ-bound performance events
--- arch/x86/kernel/cpu/perf_event.h | 5 ++ arch/x86/kernel/cpu/perf_event_intel.c | 72 +++++++++++++++++++++++++++---- 2 files changed, 67 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/perf_event.h +5 -0
@@ -164,6 +164,11 @@ struct cpu_hw_events {struct perf_guest_switch_msr guest_switch_msrs[X86_PMC_IDX_MAX ];/* +* Intel hardware interrupt context exclude bits +*/ +u64 intel_ctrl_hardirq_mask; + +/* * Intel checkpoint mask */ u64 intel_cp_status;
arch/x86/kernel/cpu/perf_event_intel.c +62 -10
127 lines not on this page · raw message
Signature · 1 line
1.7.7.6
RFC PATCH v2 4/4 perf/tool: IRQ-bound performance events
In reply to Alexander Gordeev--- tools/perf/builtin-stat.c | 9 +++++++++ tools/perf/util/evlist.c | 38 ++++++++++++++++++++++++++++++++++++++ tools/perf/util/evlist.h | 3 +++ tools/perf/util/evsel.c | 8 ++++++++ tools/perf/util/evsel.h | 3 +++ tools/perf/util/parse-events.c | 24 ++++++++++++++++++++++++ tools/perf/util/parse-events.h | 1 + 7 files changed, 86 insertions(+), 0 deletions(-)
tools/perf/builtin-stat.c +9 -0
@@ -292,6 +292,7 @@ static int create_perf_stat_counter(struct perf_evsel *evsel)PERF_FORMAT_TOTAL_TIME_RUNNING ; attr->inherit = !no_inherit; + attr->hardirq =1 ;if (target__has_cpu (&target))return perf_evsel__open_per_cpu (evsel,perf_evsel__cpus (evsel));
@@ -590,6 +591,12 @@ static int __run_perf_stat(int argc, const char **argv)return -1 ; } +if (perf_evlist__apply_hardirq (evsel_list)) { +error ("failed to set hardirq with %d (%s)\n" , errno, +strerror (errno)); +return -1 ; + } +/* * Enable counters and exec the command: */
@@ -1613,6 +1620,8 @@ int cmd_stat(int argc, const char **argv, const char *prefix __maybe_unused) parse_events_option),OPT_CALLBACK (0 ,"filter" , &evsel_list,"filter" ,"event filter" , parse_filter), +OPT_CALLBACK ('h' ,"hardirq" , &evsel_list,"hardirq" , +"stat events on existing hardware IRQ" , parse_hardirq),OPT_BOOLEAN ('i' ,"no-inherit" , &no_inherit,"child tasks do not inherit counters" ), OPT_STRING('p', "pid", &target.pid, "pid",
tools/perf/util/evlist.c +38 -0
@@ -861,6 +861,27 @@ int perf_evlist__apply_filters(struct perf_evlist *evlist)return err; } +int perf_evlist__apply_hardirq (struct perf_evlist *evlist) +{ +struct perf_evsel *evsel; +int err =0 ; +const int ncpus =cpu_map__nr (evlist->cpus), + nthreads =thread_map__nr (evlist->threads); + +list_for_each_entry (evsel, &evlist->entries, node) { +if (evsel->hardirq ==NULL ) +continue ; + + err =perf_evsel__set_hardirq (evsel, ncpus, nthreads, evsel->hardirq); +if (err) +break ; + + evsel->attr.hardirq =1 ; + } + +return err; +} +int perf_evlist__set_filter (struct perf_evlist *evlist,const char *filter) {struct perf_evsel *evsel;
@@ -877,6 +898,23 @@ int perf_evlist__set_filter(struct perf_evlist *evlist, const char *filter)return err; } +int perf_evlist__set_hardirq (struct perf_evlist *evlist, +const struct perf_hardirq_event_disp *hardirq) +{ +struct perf_evsel *evsel; +int err =0 ; +const int ncpus =cpu_map__nr (evlist->cpus), + nthreads =thread_map__nr (evlist->threads); + +list_for_each_entry (evsel, &evlist->entries, node) { + err =perf_evsel__set_hardirq (evsel, ncpus, nthreads, hardirq); +if (err) +break ; + } + +return err; +} +bool perf_evlist__valid_sample_type (struct perf_evlist *evlist) {struct perf_evsel *pos;
tools/perf/util/evlist.h +3 -0
@@ -71,6 +71,8 @@ int perf_evlist__add_newtp(struct perf_evlist *evlist, const char *sys, const char *name, void *handler);int perf_evlist__set_filter (struct perf_evlist *evlist,const char *filter); +int perf_evlist__set_hardirq (struct perf_evlist *evlist, +const struct perf_hardirq_event_disp *hardirq);struct perf_evsel *perf_evlist__find_tracepoint_by_id (struct perf_evlist *evlist,int id);
@@ -136,6 +138,7 @@ static inline void perf_evlist__set_maps(struct perf_evlist *evlist,int perf_evlist__create_maps (struct perf_evlist *evlist,struct target *target);int perf_evlist__apply_filters (struct perf_evlist *evlist); +int perf_evlist__apply_hardirq (struct perf_evlist *evlist);void __perf_evlist__set_leader (struct list_head *list);void perf_evlist__set_leader (struct perf_evlist *evlist);
tools/perf/util/evsel.c +8 -0
@@ -704,6 +704,14 @@ int perf_evsel__set_filter(struct perf_evsel *evsel, int ncpus, int nthreads, (void *)filter); } +int perf_evsel__set_hardirq (struct perf_evsel *evsel,int ncpus,int nthreads, +const struct perf_hardirq_event_disp *disp) +{ +return perf_evsel__run_ioctl (evsel, ncpus, nthreads, +PERF_EVENT_IOC_SET_HARDIRQ , + (void *)disp); +} +int perf_evsel__enable (struct perf_evsel *evsel,int ncpus,int nthreads) { return perf_evsel__run_ioctl(evsel, ncpus, nthreads,
tools/perf/util/evsel.h +3 -0
@@ -59,6 +59,7 @@ struct perf_evsel {struct list_head node;struct perf_event_attr attr;char *filter; +struct perf_hardirq_event_disp *hardirq;struct xyarray *fd;struct xyarray *sample_id;u64 *id;
@@ -169,6 +170,8 @@ void perf_evsel__set_sample_id(struct perf_evsel *evsel,int perf_evsel__set_filter (struct perf_evsel *evsel,int ncpus,int nthreads,const char *filter); +int perf_evsel__set_hardirq (struct perf_evsel *evsel,int ncpus,int nthreads, +const struct perf_hardirq_event_disp *hardirq);int perf_evsel__enable (struct perf_evsel *evsel,int ncpus,int nthreads); int perf_evsel__open_per_cpu(struct perf_evsel *evsel,
tools/perf/util/parse-events.c +24 -0
@@ -987,6 +987,30 @@ int parse_filter(const struct option *opt, const char *str,return 0 ; } +int parse_hardirq (const struct option *opt,const char *str, +int unset__maybe_unused ) +{ +struct perf_evlist *evlist = *(struct perf_evlist **)opt->value; +struct perf_evsel *evsel; +struct perf_hardirq_event_disp *event_disp; + + event_disp = malloc(offsetof(typeof(*event_disp), disp) + +sizeof (event_disp->disp[0 ])); +if (!event_disp) { +fprintf (stderr,"not enough memory to hold hardirq disp\n" ); +return -1 ; + } + + event_disp->nr_disp =1 ; + event_disp->disp[0 ].irq_nr =atoi (str); + event_disp->disp[0 ].actions =-1 ; + +list_for_each_entry (evsel, &evlist->entries, node) + evsel->hardirq = event_disp; + +return 0 ; +} +static const char *const event_type_descriptors[] = {"Hardware event" ,"Software event" ,
tools/perf/util/parse-events.h +1 -0
@@ -33,6 +33,7 @@ extern int parse_events_option(const struct option *opt, const char *str,extern int parse_events (struct perf_evlist *evlist,const char *str);extern int parse_events_terms (struct list_head *terms,const char *str);extern int parse_filter (const struct option *opt,const char *str,int unset); +extern int parse_hardirq (const struct option *opt,const char *str,int unset);#define EVENTS_HELP_MAX (128*1024)
Signature · 1 line
1.7.7.6
RFC PATCH v2 2/4 perf/x86: IRQ-bound performance events
In reply to Alexander Gordeev--- arch/x86/kernel/cpu/perf_event.c | 55 +++++++++++++++++++++++++++++++- arch/x86/kernel/cpu/perf_event.h | 10 ++++++ arch/x86/kernel/cpu/perf_event_amd.c | 2 + arch/x86/kernel/cpu/perf_event_intel.c | 4 ++ arch/x86/kernel/cpu/perf_event_knc.c | 2 + arch/x86/kernel/cpu/perf_event_p4.c | 2 + arch/x86/kernel/cpu/perf_event_p6.c | 2 + include/uapi/linux/perf_event.h | 1 - kernel/events/core.c | 34 ++++++-------------- 9 files changed, 86 insertions(+), 26 deletions(-)
arch/x86/kernel/cpu/perf_event.c +54 -1
@@ -532,15 +532,66 @@ void x86_pmu_enable_all(int added)int idx;for (idx =0 ; idx < x86_pmu.num_counters; idx++) { -struct hw_perf_event *hwc = &cpuc->events[idx]->hw; +struct perf_event *event = cpuc->events[idx]; +struct hw_perf_event *hwc = &event->hw;if (!test_bit (idx, cpuc->active_mask))continue ; +if (is_hardirq_event (event)) +continue ; + +__x86_pmu_enable_event (hwc,ARCH_PERFMON_EVENTSEL_ENABLE ); + } +} + +void x86_pmu_enable_hardirq (struct perf_event *events[],int count) +{ +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); +int idx; + +for (idx =0 ; idx < count; idx++) { +struct perf_event *event = cpuc->events[idx]; +struct hw_perf_event *hwc = &event->hw; + +BUG_ON (!test_bit (idx, cpuc->active_mask)); +BUG_ON (!is_hardirq_event (event)); + +if (event->hw.state) +continue ;__x86_pmu_enable_event (hwc,ARCH_PERFMON_EVENTSEL_ENABLE ); } } +void x86_pmu_disable_hardirq (struct perf_event *events[],int count) +{ +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); +int idx; + +for (idx =0 ; idx < count; idx++) { +struct perf_event *event = events[idx]; + +BUG_ON (!test_bit (idx, cpuc->active_mask)); +BUG_ON (!is_hardirq_event (event)); + +x86_pmu_disable_event (event); + } +} + +void x86_pmu_nop_hardirq (struct perf_event *events[],int count) +{ +} + +static void x86_pmu_start_hardirq (struct perf_event *events[],int count) +{ + x86_pmu.enable_hardirq (events, count); +} + +static void x86_pmu_stop_hardirq (struct perf_event *events[],int count) +{ + x86_pmu.disable_hardirq (events, count); +} +static struct pmu pmu;static inline int is_x86_event (struct perf_event *event)
@@ -1871,6 +1922,8 @@ static struct pmu pmu = { .del = x86_pmu_del, .start = x86_pmu_start, .stop = x86_pmu_stop, + .start_hardirq = x86_pmu_start_hardirq, + .stop_hardirq = x86_pmu_stop_hardirq, .read = x86_pmu_read, .start_txn = x86_pmu_start_txn,
arch/x86/kernel/cpu/perf_event.h +10 -0
@@ -367,6 +367,8 @@ struct x86_pmu {void (*enable_all)(int added);void (*enable)(struct perf_event *);void (*disable)(struct perf_event *); +void (*enable_hardirq)(struct perf_event *[],int ); +void (*disable_hardirq)(struct perf_event *[],int );int (*hw_config)(struct perf_event *event);int (*schedule_events)(struct cpu_hw_events *cpuc,int n,int *assign);unsigned eventsel;
@@ -538,6 +540,8 @@ int x86_pmu_hw_config(struct perf_event *event);void x86_pmu_disable_all (void ); +void x86_pmu_disable_hardirq (struct perf_event *events[],int count); +static inline void __x86_pmu_enable_event (struct hw_perf_event *hwc,u64 enable_mask) {
@@ -550,6 +554,12 @@ static inline void __x86_pmu_enable_event(struct hw_perf_event *hwc,void x86_pmu_enable_all (int added); +void x86_pmu_enable_hardirq (struct perf_event *events[],int count); + +void x86_pmu_nop_hardirq (struct perf_event *events[],int count); + +void x86_pmu_nop_hardirq_void_int (int irq); +int perf_assign_events (struct perf_event **events,int n,int wmin,int wmax,int *assign);int x86_schedule_events (struct cpu_hw_events *cpuc,int n,int *assign);
arch/x86/kernel/cpu/perf_event_amd.c +2 -0
@@ -621,6 +621,8 @@ static __initconst const struct x86_pmu amd_pmu = { .handle_irq = x86_pmu_handle_irq, .disable_all = x86_pmu_disable_all, .enable_all = x86_pmu_enable_all, + .disable_hardirq = x86_pmu_nop_hardirq, + .enable_hardirq = x86_pmu_nop_hardirq, .enable = x86_pmu_enable_event, .disable = x86_pmu_disable_event, .hw_config = amd_pmu_hw_config,
arch/x86/kernel/cpu/perf_event_intel.c +4 -0
@@ -1931,6 +1931,8 @@ static __initconst const struct x86_pmu core_pmu = { .handle_irq = x86_pmu_handle_irq, .disable_all = x86_pmu_disable_all, .enable_all = core_pmu_enable_all, + .disable_hardirq = x86_pmu_nop_hardirq, + .enable_hardirq = x86_pmu_nop_hardirq, .enable = core_pmu_enable_event, .disable = x86_pmu_disable_event, .hw_config = x86_pmu_hw_config,
@@ -2076,6 +2078,8 @@ static __initconst const struct x86_pmu intel_pmu = { .disable_all = intel_pmu_disable_all, .enable_all = intel_pmu_enable_all, .enable = intel_pmu_enable_event, + .disable_hardirq = x86_pmu_nop_hardirq, + .enable_hardirq = x86_pmu_nop_hardirq, .disable = intel_pmu_disable_event, .hw_config = intel_pmu_hw_config, .schedule_events = x86_schedule_events,
arch/x86/kernel/cpu/perf_event_knc.c +2 -0
@@ -289,6 +289,8 @@ static const struct x86_pmu knc_pmu __initconst = { .handle_irq = knc_pmu_handle_irq, .disable_all = knc_pmu_disable_all, .enable_all = knc_pmu_enable_all, + .disable_hardirq = x86_pmu_nop_hardirq, + .enable_hardirq = x86_pmu_nop_hardirq, .enable = knc_pmu_enable_event, .disable = knc_pmu_disable_event, .hw_config = x86_pmu_hw_config,
arch/x86/kernel/cpu/perf_event_p4.c +2 -0
@@ -1286,6 +1286,8 @@ static __initconst const struct x86_pmu p4_pmu = { .handle_irq = p4_pmu_handle_irq, .disable_all = p4_pmu_disable_all, .enable_all = p4_pmu_enable_all, + .disable_hardirq = x86_pmu_nop_hardirq, + .enable_hardirq = x86_pmu_nop_hardirq, .enable = p4_pmu_enable_event, .disable = p4_pmu_disable_event, .eventsel = MSR_P4_BPU_CCCR0,
arch/x86/kernel/cpu/perf_event_p6.c +2 -0
@@ -202,6 +202,8 @@ static __initconst const struct x86_pmu p6_pmu = { .handle_irq = x86_pmu_handle_irq, .disable_all = p6_pmu_disable_all, .enable_all = p6_pmu_enable_all, + .disable_hardirq = x86_pmu_nop_hardirq, + .enable_hardirq = x86_pmu_nop_hardirq, .enable = p6_pmu_enable_event, .disable = p6_pmu_disable_event, .hw_config = x86_pmu_hw_config,
include/uapi/linux/perf_event.h +0 -1
@@ -726,7 +726,6 @@ enum perf_callchain_context {#define PERF_FLAG_FD_NO_GROUP (1U << 0)#define PERF_FLAG_FD_OUTPUT (1U << 1)#define PERF_FLAG_PID_CGROUP (1U << 2)/* pid=cgroup id, per-cpu mode only */ -#define PERF_FLAG_PID_HARDIRQ (1U << 3)/* pid=irq number */ union perf_mem_data_src {__u64 val;
kernel/events/core.c +10 -24
@@ -119,8 +119,7 @@ static int cpu_function_call(int cpu, int (*func) (void *info), void *info)#define PERF_FLAG_ALL (PERF_FLAG_FD_NO_GROUP |\ PERF_FLAG_FD_OUTPUT |\ - PERF_FLAG_PID_CGROUP |\ - PERF_FLAG_PID_HARDIRQ) + PERF_FLAG_PID_CGROUP)/* * branch priv levels that need permission checks
@@ -7028,7 +7027,6 @@ SYSCALL_DEFINE5(perf_event_open,struct fd group = {NULL ,0 };struct task_struct *task =NULL ;struct pmu *pmu; -int hardirq =-1 ;int event_fd;int move_group =0 ;int err;
@@ -7037,27 +7035,6 @@ SYSCALL_DEFINE5(perf_event_open,if (flags & ~PERF_FLAG_ALL )return -EINVAL ; -if ((flags & (PERF_FLAG_PID_CGROUP |PERF_FLAG_PID_HARDIRQ )) == - (PERF_FLAG_PID_CGROUP |PERF_FLAG_PID_HARDIRQ )) -return -EINVAL ; - -/* -* In irq mode, the pid argument is used to pass irq number. -*/ -if (flags &PERF_FLAG_PID_HARDIRQ ) { - hardirq = pid; - pid =-1 ; - } - -/* -* In cgroup mode, the pid argument is used to pass the fd -* opened to the cgroup directory in cgroupfs. The cpu argument -* designates the cpu on which to monitor threads from that -* cgroup. -*/ -if ((flags &PERF_FLAG_PID_CGROUP ) && (pid ==-1 || cpu ==-1 )) -return -EINVAL ; - err =perf_copy_attr (attr_uptr, &attr);if (err)return err;
@@ -7072,6 +7049,15 @@ SYSCALL_DEFINE5(perf_event_open,return -EINVAL ; } +/* +* In cgroup mode, the pid argument is used to pass the fd +* opened to the cgroup directory in cgroupfs. The cpu argument +* designates the cpu on which to monitor threads from that +* cgroup. +*/ +if ((flags &PERF_FLAG_PID_CGROUP ) && (pid ==-1 || cpu ==-1 )) +return -EINVAL ; + event_fd =get_unused_fd ();if (event_fd <0 )return event_fd;
Signature · 1 line
1.7.7.6
RFC PATCH v2 1/4 perf/core: IRQ-bound performance events
In reply to Alexander Gordeev--- include/linux/irq.h | 10 + include/linux/irqdesc.h | 4 + include/linux/perf_event.h | 24 +++ include/uapi/linux/perf_event.h | 15 ++- kernel/events/Makefile | 2 +- kernel/events/core.c | 176 +++++++++++++++++-- kernel/events/hardirq.c | 370 +++++++++++++++++++++++++++++++++++++++ kernel/irq/handle.c | 7 +- kernel/irq/irqdesc.c | 15 ++ 9 files changed, 609 insertions(+), 14 deletions(-) create mode 100644 kernel/events/hardirq.c
include/linux/irq.h +10 -0
@@ -632,6 +632,16 @@ static inline int irq_reserve_irq(unsigned int irq)# define irq_reg_readl (addr) readl(addr)#endif +#ifdef CONFIG_PERF_EVENTS +extern void perf_start_hardirq_events (struct irq_desc *desc,int action_nr); +extern void perf_stop_hardirq_events (struct irq_desc *desc,int action_nr); +#else +static inline void +perf_start_hardirq_events (struct irq_desc *desc,int action_nr) { } +static inline void +perf_stop_hardirq_events (struct irq_desc *desc,int action_nr) { } +#endif +/** * struct irq_chip_regs - register offsets for struct irq_gci * @enable: Enable register offset to reg_base
include/linux/irqdesc.h +4 -0
@@ -12,6 +12,7 @@ struct irq_affinity_notify;struct proc_dir_entry ;struct module ;struct irq_desc ; +struct hardirq_events ;/** * struct irq_desc - interrupt descriptor
@@ -68,6 +69,9 @@ struct irq_desc {struct proc_dir_entry *dir;#endif int parent_irq; +#ifdef CONFIG_PERF_EVENTS +struct hardirq_events __percpu **events; +#endif struct module *owner;const char *name; }____cacheline_internodealigned_in_smp ;
include/linux/perf_event.h +24 -0
@@ -215,6 +215,12 @@ struct pmu {void (*stop) (struct perf_event *event,int flags);/* +* Start/Stop hardware interrupt context counters present on the PMU. +*/ +void (*start_hardirq) (struct perf_event *events[],int count);/* optional */ +void (*stop_hardirq) (struct perf_event *events[],int count);/* optional */ + +/* * Updates the counter value of the event. */ void (*read) (struct perf_event *event);
@@ -313,6 +319,11 @@ struct perf_event {struct list_head sibling_list;/* +* List of hardware interrupt context numbers and actions +*/ +struct list_head hardirq_list; + +/* * We need storage to track the entries in perf_pmu_migrate_context; we * cannot use the event_entry because of RCU and we want to keep the * group in tact which avoids us using the other two entries.
@@ -528,6 +539,12 @@ struct perf_output_handle {int page; }; +struct perf_hardirq_param { +struct list_head list; +int irq; +unsigned long mask; +}; +#ifdef CONFIG_PERF_EVENTS extern int perf_pmu_register (struct pmu *pmu,const char *name,int type);
@@ -635,6 +652,11 @@ static inline int is_software_event(struct perf_event *event)return event->pmu->task_ctx_nr == perf_sw_context; } +static inline bool is_hardirq_event (struct perf_event *event) +{ +return event->attr.hardirq !=0 ; +} +extern struct static_key perf_swevent_enabled[PERF_COUNT_SW_MAX ];extern void __perf_sw_event (u32 ,u64 ,struct pt_regs *,u64 );
@@ -772,6 +794,8 @@ extern void perf_event_enable(struct perf_event *event);extern void perf_event_disable (struct perf_event *event);extern int __perf_event_disable (void *info);extern void perf_event_task_tick (void ); +extern int perf_event_init_hardirq (void *info); +extern int perf_event_term_hardirq (void *info);#else static inline void perf_event_task_sched_in(struct task_struct *prev,
include/uapi/linux/perf_event.h +14 -1
@@ -301,8 +301,9 @@ struct perf_event_attr { exclude_callchain_kernel : 1, /* exclude kernel callchains */ exclude_callchain_user : 1, /* exclude user callchains */ mmap2 : 1, /* include mmap with inode data */ + hardirq : 1, - __reserved_1 : 40; + __reserved_1 : 39;union {__u32 wakeup_events;/* wakeup every n events */
@@ -348,6 +349,7 @@ struct perf_event_attr {#define PERF_EVENT_IOC_SET_OUTPUT _IO ('$', 5)#define PERF_EVENT_IOC_SET_FILTER _IOW('$', 6, char *)#define PERF_EVENT_IOC_ID _IOR('$', 7, __u64 *) +#define PERF_EVENT_IOC_SET_HARDIRQ _IOR('$', 8, __u64 *)enum perf_event_ioc_flags {PERF_IOC_FLAG_GROUP =1U <<0 ,
@@ -724,6 +726,7 @@ enum perf_callchain_context {#define PERF_FLAG_FD_NO_GROUP (1U << 0)#define PERF_FLAG_FD_OUTPUT (1U << 1)#define PERF_FLAG_PID_CGROUP (1U << 2)/* pid=cgroup id, per-cpu mode only */ +#define PERF_FLAG_PID_HARDIRQ (1U << 3)/* pid=irq number */ union perf_mem_data_src {__u64 val;
@@ -812,4 +815,14 @@ struct perf_branch_entry { reserved:60; }; +struct perf_hardirq_disp { +__s32 irq_nr; +__u64 actions; +}; + +struct perf_hardirq_event_disp { +__s32 nr_disp;/* everything if <0 */ +struct perf_hardirq_disp disp[0 ]; +}; +#endif /* _UAPI_LINUX_PERF_EVENT_H */
kernel/events/Makefile +1 -1
@@ -2,7 +2,7 @@ ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_core.o = -pg endif -obj-y := core.o ring_buffer.o callchain.o +obj-y := core.o ring_buffer.o callchain.o hardirq.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o obj-$(CONFIG_UPROBES) += uprobes.o
kernel/events/core.c +165 -11
246 lines not on this page · raw message
kernel/events/hardirq.c new file +370 -0
370 lines not on this page · raw message
kernel/irq/handle.c +6 -1
@@ -133,13 +133,17 @@ irqreturn_t handle_irq_event_percpu(struct irq_desc *desc, struct irqaction *action) { irqreturn_t retval = IRQ_NONE; -unsigned int flags =0 , irq = desc->irq_data.irq; +unsigned int flags =0 , irq = desc->irq_data.irq, action_nr =0 ; do { irqreturn_t res; trace_irq_handler_entry(irq, action); + perf_start_hardirq_events(desc, action_nr); + res = action->handler(irq, action->dev_id); + + perf_stop_hardirq_events(desc, action_nr); trace_irq_handler_exit(irq, action, res); if (WARN_ONCE(!irqs_disabled(),"irq %u handler %pF enabled interrupts\n",
@@ -170,6 +174,7 @@ handle_irq_event_percpu(struct irq_desc *desc, struct irqaction *action) retval |= res; action = action->next; + action_nr++; }while (action);add_interrupt_randomness (irq, flags);
kernel/irq/irqdesc.c +15 -0
@@ -131,6 +131,14 @@ static void free_masks(struct irq_desc *desc)static inline void free_masks (struct irq_desc *desc) { }#endif +#ifdef CONFIG_PERF_EVENTS +extern int perf_alloc_hardirq_events (struct irq_desc *desc); +extern void perf_free_hardirq_events (struct irq_desc *desc); +#else +static inline int perf_alloc_hardirq_events (struct irq_desc *desc) {return 0 ; } +static inline void perf_free_hardirq_events (struct irq_desc *desc) { } +#endif +static struct irq_desc *alloc_desc (int irq,int node,struct module *owner) {struct irq_desc *desc;
@@ -147,6 +155,10 @@ static struct irq_desc *alloc_desc(int irq, int node, struct module *owner)if (alloc_masks (desc, gfp, node)) goto err_kstat; +if (perf_alloc_hardirq_events (desc)) + goto err_masks; + +raw_spin_lock_init (&desc->lock);lockdep_set_class (&desc->lock, &irq_desc_lock_class);
@@ -154,6 +166,8 @@ static struct irq_desc *alloc_desc(int irq, int node, struct module *owner)return desc; +err_masks: +free_masks (desc); err_kstat:free_percpu (desc->kstat_irqs); err_desc:
@@ -171,6 +185,7 @@ static void free_desc(unsigned int irq)delete_irq_desc (irq);mutex_unlock (&sparse_irq_lock); +perf_free_hardirq_events (desc);free_masks (desc);free_percpu (desc->kstat_irqs);kfree (desc);
Signature · 1 line
1.7.7.6
Quoted 6 lines · Alexander Gordeev
Hello,
This is version 2 of RFC "perf: IRQ-bound performance events". That is an introduction of IRQ-bound performance events - ones that only count in a context of a hardware interrupt handler. Ingo suggested to extend this functionality to softirq and threaded handlers as well:
Did you measure the overhead in workloads that do a lot of interrupts? I assume two WRMSR could be a significant part of the cost of small interrupts.
For counting at least it would be likely a lot cheaper to just RDPMC and subtract manually.
The cache miss example below is certainly misleading, as cache misses by interrupts are often a "debt", that is they are forced on whoever is interrupted. I don't think that is a good use of this.
I guess it can be useful for cycles.
-Andi
Quoted 7 lines · Andi Kleen
Quoted 4 lines
This is version 2 of RFC "perf: IRQ-bound performance events". That is an introduction of IRQ-bound performance events - ones that only count in a context of a hardware interrupt handler. Ingo suggested to extend this functionality to softirq and threaded handlers as well:
Did you measure the overhead in workloads that do a lot of interrupts? I assume two WRMSR could be a significant part of the cost of small interrupts.
No, that would be the next step. I hoped first to ensure the way I am intruding into the current perf design is correct.
Quoted 2 lines
For counting at least it would be likely a lot cheaper to just RDPMC and subtract manually.
Sigh, that seems as quite a rework for Intel PMU.
Quoted 3 lines
The cache miss example below is certainly misleading, as cache misses by interrupts are often a "debt", that is they are forced on whoever is interrupted. I don't think that is a good use of this.
May be useless rather than misleading? :) Actually, cache and power use are exactly the data I thought are useful if one wants to check the dependency from the interrupt affinity mask. There was some discussion on this topic some year ago:
On Mon, May 21, 2012 at 08:36:09AM -0700, Linus Torvalds wrote:
"So it may well make perfect sense to allow a mask of CPU's for interrupt delivery, but just make sure that the mask all points to CPU's on the same socket. That would give the hardware some leeway in choosing the actual core - it's very possible that hardware could avoid cores that are running with irq's disabled (possibly improving latecy) or even more likely - avoid cores that are in deeper powersaving modes."
So this RFC is kind of follow-up to come up with necessary tooling.
Quoted 3 lines
I guess it can be useful for cycles.
-Andi
Thanks, Andi!
Signature · 3 lines
Regards, Alexander Gordeev [email protected]
Quoted 6 lines · Alexander Gordeev
Hello,
This is version 2 of RFC "perf: IRQ-bound performance events". That is an introduction of IRQ-bound performance events - ones that only count in a context of a hardware interrupt handler. Ingo suggested to extend this functionality to softirq and threaded handlers as well:
Hi Alexander,
I still strongly think we should use toggle events to achieve that:
https://lkml.org/lkml/2013/9/25/227
This will let us count not just IRQs (-e 'cycles,irq_entry/on=cycles/,irq_exit/off=cycles/') but much more.
The patchset still needs some polishing but I think that's a better diection.
Thanks.
Quoted 8 lines · Frederic Weisbecker
Quoted 6 lines · Alexander Gordeev
Hello,
This is version 2 of RFC "perf: IRQ-bound performance events". That is an introduction of IRQ-bound performance events - ones that only count in a context of a hardware interrupt handler. Ingo suggested to extend this functionality to softirq and threaded handlers as well:
Quoted 4 lines
Hi Alexander,
I still strongly think we should use toggle events to achieve that:
https://lkml.org/lkml/2013/9/25/227
Hi Frederic,
The toggle events would not allow counting per-action in hardware interrupt context. The design I propose provisions any possible combination of actions/ IRQs.
I.e. if we had few drivers on IRQn and few drivers on IRQm we could assign an event to let's say ISR0, ISR2 on IRQn and ISR1 on IRQm.
Moreover, given that hardware context handlers are running with local interrupts disabled and therefore an IRQ-bound event could be enabled/ disabled only from a single handler at a time - we just need to allocate a single hardware counter for any possible combination.
Unless I missing something, neither kernel nor user level for toggle events can meet the features described above.
I think it would be ideal if the two approaches could be converged somehow, but I just do not know how at the moment. I believe the next step is to measure the overhead Andi mentioned. That well might be a showstopper for either or both approaches.
By contrast with the hardware context, the toggle events seem to able monitoring softirq in its current form.
As of the threaded context handlers, I have not come up with the idea how to make it yet, but it does not seem the toggle events are able eigher.
Quoted 3 lines
This will let us count not just IRQs (-e 'cycles,irq_entry/on=cycles/,irq_exit/off=cycles/') but much more.
The patchset still needs some polishing but I think that's a better diection.
Thanks, Frederic.
Quoted 1 line
Thanks.
Signature · 3 lines
Regards, Alexander Gordeev [email protected]
Quoted 11 lines · Alexander Gordeev
Quoted 10 lines · Frederic Weisbecker
Quoted 6 lines · Alexander Gordeev
Hello,
This is version 2 of RFC "perf: IRQ-bound performance events". That is an introduction of IRQ-bound performance events - ones that only count in a context of a hardware interrupt handler. Ingo suggested to extend this functionality to softirq and threaded handlers as well:
Hi Alexander,
Quoted 8 lines
Quoted 2 lines
I still strongly think we should use toggle events to achieve that:
https://lkml.org/lkml/2013/9/25/227Hi Frederic,
The toggle events would not allow counting per-action in hardware interrupt context. The design I propose provisions any possible combination of actions/ IRQs.
I think we could define one event per handler by using tracepoint filters.
Quoted 3 lines
I.e. if we had few drivers on IRQn and few drivers on IRQm we could assign an event to let's say ISR0, ISR2 on IRQn and ISR1 on IRQm.
Yeah that should be possible with tracepoints as well.
Quoted 4 lines
Moreover, given that hardware context handlers are running with local interrupts disabled and therefore an IRQ-bound event could be enabled/ disabled only from a single handler at a time - we just need to allocate a single hardware counter for any possible combination.
Hmm I don't get what you mean here. Why tracepoint defined event don't work in this scenario?
Quoted 11 lines
I think it would be ideal if the two approaches could be converged somehow, but I just do not know how at the moment. I believe the next step is to measure the overhead Andi mentioned. That well might be a showstopper for either or both approaches.
By contrast with the hardware context, the toggle events seem to able monitoring softirq in its current form.
As of the threaded context handlers, I have not come up with the idea how to make it yet, but it does not seem the toggle events are able eigher.
A per task event should do the trick for threaded irqs profiling.
Thanks.