perf_events: add support for sampling taken branches
Subsystems (9)
cover perf_events: add support for sampling taken branches
This patchset adds an important and useful new feature toperf_events: branch stack sampling. In other words, the ability to capture taken branches into each sample.
Statistical sampling of taken branch should not be confused for branch tracing. Not all branches are necessarily captured
Sampling taken branches is important for basic block profiling, statistical call graph, function call counts. Many of those measurements can help drive a compiler optimizer.
The branch stack is a software abstraction which sits on top of the PMU hardware. As such, it is not available on all processors. For now, the patch provides the generic interface and the Intel X86 implementation where it leverages the Last Branch Record (LBR) feature (from Core2 to SandyBridge).
Branch stack sampling is supported for both per-thread and system-wide modes.
It is possible to filter the type and privilege level of branches to sample. The target of the branch is used to determine the privilege level.
For each branch, the source and destination are captured. On some hardware platforms, it may be possible to also extract the target prediction and, in that case, it is also exposed to end users.
The branch stack can record a variable number of taken branches per sample. Those branches are always consecutive in time. The number of branches captured depends on the filtering and the underlying hardware. On Intel Nehalem and later, up to 16 consecutive branches can be captured per sample.
Branch sampling is always coupled with an event. It can be any PMU event but it can't be a SW or tracepoint event.
Branch sampling is requested by setting a new sample_type flag called: PERF_SAMPLE_BRANCH_STACK.
To support branch filtering, we introduce a new field to the perf_event_attr struct: branch_sample_type. We chose NOT to overload the config1, config2 field because those are related to the event encoding. Branch stack is a separate feature which is combined with the event.
The branch_sample_type is a bitmask of possible filters. The following filters are defined (more can be added):
- PERF_SAMPLE_BRANCH_ANY : any control flow change - PERF_SAMPLE_BRANCH_USER : capture branches when target is at user level - PERF_SAMPLE_BRANCH_KERNEL : capture branches when target is at user level
- PERF_SAMPLE_BRANCH_ANY_CALL: capture call branches (incl. syscalls)
- PERF_SAMPLE_BRANCH_ANY_RET : capture return branches (incl. syscall returns)
- PERF_SAMPLE_BRANCH_IND_CALL: capture indirect calls
It is possible to combine filters, e.g., IND_CALL|USER|KERNEL.
When the privilege level is not specified, the branch stack inherits that of the associated event.
Some processors may not offer hardware branch filtering, e.g., Intel Atom. Some may have HW filtering bugs (e.g., Nehalem). The Intel X86 implementation in this patchset also provides a SW branch filter which works on a best effort basis. It can compensate for the lack of LBR filtering. But first and foremost, it helps work around LBR filtering errata. The goal is to only capture the type of branches requested by the user.
It is possible to combine branch stack sampling with PEBS on Intel X86 processors. Depending on the precise_sampling mode, there are certain filterting restrictions. When precise_sampling=1, then there are no filtering restrictions. When precise_sampling > 1, then only ANY|USER|KERNEL filter can be used. This comes from the fact that the kernel uses LBR to compensate for the PEBS off-by-1 skid on the instruction pointer.
To demonstrate how the perf_event branch stack sampling interface works, the patchset also modifies perf record to capture taken branches. Similarly perf report is enhanced to display a histogram of taken branches.
I would like to thank Roberto Vitillo @ LBL for his work on the perf tool for this.
Enough talking, let's take a simple example. Our trivial test program goes like this:
void f2(void)
{}
void f3(void)
{}
void f1(unsigned long n)
{
if (n & 1UL)
f2();
else
f3();
}int main(void)
{
unsigned long i;
for (i=0; i < N; i++)
f1(i);
return 0;
}
$ perf record -b any branchy $ perf report -b # Events: 23K cycles # # Overhead Source Symbol Target Symbol # ........ ................ ................
18.13% [.] f1 [.] main
18.10% [.] main [.] main
18.01% [.] main [.] f1
15.69% [.] f1 [.] f1
9.11% [.] f3 [.] f1
6.78% [.] f1 [.] f3
6.74% [.] f1 [.] f2
6.71% [.] f2 [.] f1 Of the total number of branches captured, 18.13% were from f1() -> main().
Let's make this clearer by filtering the user call branches only:
$ perf record -b any_call -e cycles:u branchy
$ perf report
# Events: 19K cycles
#
# Overhead Source Symbol Target Symbol
# ........ ......................... .........................
#
52.50% [.] main [.] f1
23.99% [.] f1 [.] f3
23.48% [.] f1 [.] f2
0.03% [.] _IO_default_xsputn [.] _IO_new_file_overflow
0.01% [k] _start [k] __libc_start_main Now it is more obvious. %52 of all the captured branches where calls from main() -> f1(). The rest is split 50/50 between f1() -> f2() and f1() -> f3() which is expected given that f1() dispatches based on odd vs. even values of n which is constantly increasing.
Roberto Agostino Vitillo (2):
perf: add support for sampling taken branch to perf record
perf: add support for taken branch sampling to perf report
Stephane Eranian (10):
perf_events: add generic taken branch sampling support
perf_events: add Intel LBR MSR definitions
perf_events: add Intel X86 LBR sharing logic
perf_events: sync branch stack sampling with X86 precise_sampling
perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters
perf_events: implement PERF_SAMPLE_BRANCH for Intel X86
perf_events: add LBR software filter support for Intel X86
perf_events: disable PERF_SAMPLE_BRANCH_* when not supported
perf_events: add hook to flush branch_stack on context switch
perf: add code to support PERF_SAMPLE_BRANCH_STACK
arch/alpha/kernel/perf_event.c | 4 + arch/arm/kernel/perf_event.c | 4 + arch/mips/kernel/perf_event.c | 4 + arch/powerpc/kernel/perf_event.c | 4 + arch/sh/kernel/perf_event.c | 4 + arch/sparc/kernel/perf_event.c | 4 + arch/x86/include/asm/msr-index.h | 7 + arch/x86/kernel/cpu/perf_event.c | 62 +++- arch/x86/kernel/cpu/perf_event_amd.c | 3 + arch/x86/kernel/cpu/perf_event_intel.c | 126 +++++-- arch/x86/kernel/cpu/perf_event_intel_ds.c | 21 +- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 529 ++++++++++++++++++++++++++-- include/linux/perf_event.h | 74 ++++- kernel/events/core.c | 167 +++++++++ kernel/events/hw_breakpoint.c | 6 + tools/perf/Documentation/perf-record.txt | 18 + tools/perf/Documentation/perf-report.txt | 7 + tools/perf/builtin-record.c | 75 ++++ tools/perf/builtin-report.c | 93 +++++- tools/perf/perf.h | 17 + tools/perf/util/annotate.c | 2 +- tools/perf/util/event.h | 1 + tools/perf/util/evsel.c | 10 + tools/perf/util/hist.c | 97 ++++-- tools/perf/util/hist.h | 6 + tools/perf/util/session.c | 72 ++++ tools/perf/util/session.h | 5 + tools/perf/util/sort.c | 348 ++++++++++++++----- tools/perf/util/sort.h | 5 + tools/perf/util/symbol.h | 13 + 30 files changed, 1584 insertions(+), 204 deletions(-)
Signature · 1 line
1.7.4.1
PATCH 1/12 perf_events: add generic taken branch sampling support
This patch adds the ability to sample taken branches to the perf_event interface.
The ability to capture taken branches is very useful for all sorts of analysis. For instance, basic block profiling, call counts, statistical call graph.
This new capability requires hardware assist and as such may not be available on all HW platforms. On Intel X86, it is implemented on top of the Last Branch Record (LBR) facility.
To enable taken branches sampling, the PERF_SAMPLE_BRANCH_STACK bit must be set in attr->sample_type.
Sampled taken branches may be filtered by type and/or priv levels.
The patch adds a new field, called branch_sample_type, to the perf_event_attr structure. It contains a bitmask of filters to apply to the sampled taken branches.
Filters may be implemented in HW. If the HW filter does not exist or is not good enough, some arch may also implement a SW filter.
The following generic filters are currently defined:
- PERF_SAMPLE_USER
only branches whose targets are at the user level
- PERF_SAMPLE_KERNEL
only branches whose targets are at the kernel level
- PERF_SAMPLE_ANY
any type of branches (subject to priv levels filters)
- PERF_SAMPLE_ANY_CALL
any call branches (may incl. syscall on some arch)
- PERF_SAMPLE_ANY_RET
any return branches (may incl. syscall returns on some arch)
- PERF_SAMPLE_IND_CALL
indirect call branches
Obviously filter may be combined. The priv level bits are optional. If not provided, the priv level of the associated event are used. It is possible to collect branches at a priv level different from the associated event.
The number of taken branch records present in each sample may vary based on HW, the type of sampled branches, the executed code. Therefore each sample contains the number of taken branches it contains.
--- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 21 ++++++---- include/linux/perf_event.h | 64 +++++++++++++++++++++++++++- kernel/events/core.c | 58 +++++++++++++++++++++++++ 3 files changed, 133 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_lbr.c +13 -8
@@ -138,9 +138,11 @@ static void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)rdmsrl (x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr); - cpuc->lbr_entries[i].from = msr_lastbranch.from; - cpuc->lbr_entries[i].to = msr_lastbranch.to; - cpuc->lbr_entries[i].flags =0 ; + cpuc->lbr_entries[i].from = msr_lastbranch.from; + cpuc->lbr_entries[i].to = msr_lastbranch.to; + cpuc->lbr_entries[i].mispred =0 ; + cpuc->lbr_entries[i].predicted =0 ; + cpuc->lbr_entries[i].reserved =0 ; } cpuc->lbr_stack.nr = i; }
@@ -161,19 +163,22 @@ static void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)for (i =0 ; i < x86_pmu.lbr_nr; i++) {unsigned long lbr_idx = (tos - i) & mask; -u64 from, to, flags =0 ; +u64 from, to, mis =0 , pred =0 ;rdmsrl (x86_pmu.lbr_from + lbr_idx, from);rdmsrl (x86_pmu.lbr_to + lbr_idx, to);if (lbr_format ==LBR_FORMAT_EIP_FLAGS ) { - flags = !!(from &LBR_FROM_FLAG_MISPRED ); + mis = !!(from &LBR_FROM_FLAG_MISPRED ); + pred= !mis; from = (u64)((((s64 )from) <<1 ) >>1 ); } - cpuc->lbr_entries[i].from = from; - cpuc->lbr_entries[i].to = to; - cpuc->lbr_entries[i].flags = flags; + cpuc->lbr_entries[i].from = from; + cpuc->lbr_entries[i].to = to; + cpuc->lbr_entries[i].mispred = mis; + cpuc->lbr_entries[i].predicted = pred; + cpuc->lbr_entries[i].reserved =0 ; } cpuc->lbr_stack.nr = i; }
include/linux/perf_event.h +62 -2
122 lines not on this page · raw message
kernel/events/core.c +58 -0
76 lines not on this page · raw message
Signature · 1 line
1.7.4.1
Peter Zijlstra
maintainerRe: [PATCH 01/12] perf_events: add generic taken branch sampling support
include/linux/perf_event.h struct perf_raw_record { · line 576
Quoted 9 lines · Stephane Eranian
struct perf_branch_entry { __u64 from; __u64 to;include/linux/perf_event.h +5
-0+struct { +__u64 mispred:1 ,/* target mispredicted */ + predicted:1 ,/* target predicted */ + reserved:62 ; + }; };
Why that anonymous structure?
Re: [PATCH 01/12] perf_events: add generic taken branch sampling support
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Stephane Eranian
struct
perf_branch_entry{
__u64from;
__u64to;
+ struct {
+__u64mispred:1, /* target mispredicted */
+ predicted:1,/* target predicted */
+ reserved:62;
+ };
};
Quoted 2 lines
Why that anonymous structure?
The interface can return more than source/destination, it can also return prediction infos.
Prediction is boolean, thus we only need a couple of bits. The reason there are two bits
and not just one is to disambiguate between: predicted correctly and 'prediction reporting
unsupported'. For instance, prediction is also supported since Nehalem, Core2/Atom do
not have it.
But maybe you're just commenting of the anonymous vs. named struct for that? It is just for convenience. Isn't that the same argument for the anonymous bitfield
in struct perf_event_attr?
Peter Zijlstra
maintainerRe: [PATCH 01/12] perf_events: add generic taken branch sampling support
Quoted 14 lines · Stephane Eranian
Quoted 11 lines · Peter Zijlstra
include/linux/perf_event.h struct perf_raw_record { · line 576
Quoted 9 lines · Stephane Eranian
struct perf_branch_entry { __u64 from; __u64 to;include/linux/perf_event.h +5
-0+struct { +__u64 mispred:1 ,/* target mispredicted */ + predicted:1 ,/* target predicted */ + reserved:62 ; + }; };Quoted 2 lines
Why that anonymous structure?
Quoted 9 lines
The interface can return more than source/destination, it can also return prediction infos.
Prediction is boolean, thus we only need a couple of bits. The reason there are two bits
and not just one is to disambiguate between: predicted correctly and 'prediction reporting
unsupported'. For instance, prediction is also supported since Nehalem, Core2/Atom do
not have it.
Right, I got that.
Quoted 2 lines
But maybe you're just commenting of the anonymous vs. named struct for that?
I don't see the need for any struct, why can't the bitfield live in perf_branch_entry proper?
Quoted 3 lines
It is just for convenience. Isn't that the same argument for the anonymous bitfield
in structperf_event_attr?
But that isn't wrapped in a structure either is it..
I guess I'm asking, what's wrong with:
struct perf_branch_entry {
__u64 from;
__u64 to;
__u64 mispred:1,
predicted:1,
reserved:62;
};?
Re: [PATCH 01/12] perf_events: add generic taken branch sampling support
Quoted 41 lines · Peter Zijlstra
Quoted 14 lines · Stephane Eranian
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Stephane Eranian
struct
perf_branch_entry{
__u64from;
__u64to;
+ struct {
+__u64mispred:1, /* target mispredicted */
+ predicted:1,/* target predicted */
+ reserved:62;
+ };
};Quoted 2 lines
Why that anonymous structure?
Quoted 9 lines
The interface can return more than source/destination, it can also return prediction infos.
Prediction is boolean, thus we only need a couple of bits. The reason there are two bits
and not just one is to disambiguate between: predicted correctly and 'prediction reporting
unsupported'. For instance, prediction is also supported since Nehalem, Core2/Atom do
not have it.Right, I got that.
Quoted 2 lines
But maybe you're just commenting of the anonymous vs. named struct for that?
I don't see the need for any struct, why can't the bitfield live in
perf_branch_entryproper?Quoted 3 lines
It is just for convenience. Isn't that the same argument for the anonymous bitfield
in structperf_event_attr?But that isn't wrapped in a structure either is it..
I guess I'm asking, what's wrong with:
Quoted 8 lines
struct
perf_branch_entry{
__u64 from;
__u64 to;
__u64 mispred:1,
predicted:1,
reserved:62;
};
Ok, I think I missed your original point about the anonymous struct. What you're suggesting above is certainly fine with me. I am not just used to bitfields mixed with other fields in a struct. I'll fix that.
Peter Zijlstra
maintainerRe: [PATCH 01/12] perf_events: add generic taken branch sampling support
In reply to Stephane Eranianinclude/linux/perf_event.h enum perf_event_sample_format { · line 148
Quoted 3 lines · Stephane Eranian
include/linux/perf_event.h +3
-0+enum perf_branch_sample_type { +PERF_SAMPLE_BRANCH_USER =1U <<0 ,/* user level branches */ +PERF_SAMPLE_BRANCH_KERNEL =1U <<1 ,/* kernel level branches */
I just merged a patch-set that adds guest/host muck as well, would it make sense to extend this stuff to support that as well?
Quoted 7 lines
include/linux/perf_event.h +7
-0+PERF_SAMPLE_BRANCH_ANY =1U <<2 ,/* any branch types */ +PERF_SAMPLE_BRANCH_ANY_CALL =1U <<3 ,/* any call branch */ +PERF_SAMPLE_BRANCH_ANY_RETURN =1U <<4 ,/* any return branch */ +PERF_SAMPLE_BRANCH_IND_CALL =1U <<5 ,/* indirect calls */ + + PERF_SAMPLE_BRANCH_MAX = 1U << 6,/* non-ABI */ +}
PATCH 2/12 perf_events: add Intel LBR MSR definitions
In reply to Stephane EranianThis patch adds the LBR definitions for NHM/WSM/SNB and Core. It also adds the definitions for the architected LBR MSR:LBR_SELECT, LBRT_TOS.
--- arch/x86/include/asm/msr-index.h | 7 +++++++ arch/x86/kernel/cpu/perf_event_intel_lbr.c | 18 +++++++++--------- 2 files changed, 16 insertions(+), 9 deletions(-)
arch/x86/include/asm/msr-index.h +7 -0
@@ -56,6 +56,13 @@#define MSR_OFFCORE_RSP_0 0x000001a6#define MSR_OFFCORE_RSP_1 0x000001a7 +#define MSR_LBR_SELECT 0x000001c8 +#define MSR_LBR_TOS 0x000001c9 +#define MSR_LBR_NHM_FROM 0x00000680 +#define MSR_LBR_NHM_TO 0x000006c0 +#define MSR_LBR_CORE_FROM 0x00000040 +#define MSR_LBR_CORE_TO 0x00000060 +#define MSR_IA32_PEBS_ENABLE 0x000003f1#define MSR_IA32_DS_AREA 0x00000600#define MSR_IA32_PERF_CAPABILITIES 0x00000345
arch/x86/kernel/cpu/perf_event_intel_lbr.c +9 -9
@@ -199,25 +199,25 @@ static void intel_pmu_lbr_read(void)static void intel_pmu_lbr_init_core (void ) { x86_pmu.lbr_nr =4 ; - x86_pmu.lbr_tos =0x01c9 ; - x86_pmu.lbr_from =0x40 ; - x86_pmu.lbr_to =0x60 ; + x86_pmu.lbr_tos =MSR_LBR_TOS ; + x86_pmu.lbr_from =MSR_LBR_CORE_FROM ; + x86_pmu.lbr_to =MSR_LBR_CORE_TO ; }static void intel_pmu_lbr_init_nhm (void ) { x86_pmu.lbr_nr =16 ; - x86_pmu.lbr_tos =0x01c9 ; - x86_pmu.lbr_from =0x680 ; - x86_pmu.lbr_to =0x6c0 ; + x86_pmu.lbr_tos =MSR_LBR_TOS ; + x86_pmu.lbr_from =MSR_LBR_NHM_FROM ; + x86_pmu.lbr_to =MSR_LBR_NHM_TO ; }static void intel_pmu_lbr_init_atom (void ) { x86_pmu.lbr_nr =8 ; - x86_pmu.lbr_tos =0x01c9 ; - x86_pmu.lbr_from =0x40 ; - x86_pmu.lbr_to =0x60 ; + x86_pmu.lbr_tos =MSR_LBR_TOS ; + x86_pmu.lbr_from =MSR_LBR_CORE_FROM ; + x86_pmu.lbr_to =MSR_LBR_CORE_TO ; }#endif /* CONFIG_CPU_SUP_INTEL */
Signature · 1 line
1.7.4.1
PATCH 3/12 perf_events: add Intel X86 LBR sharing logic
In reply to Stephane EranianThe Intel LBR on some recent processor is capable of filtering branches by type. The filter is configurable via the LBR_SELECT MSR register.
There are limitation on how this register can be used.
On Nehalem/Westmere, the LBR_SELECT is shared by the two HT threads when HT is on. It is private to each core when HT is off.
On SandyBridge, the LBR_SELECT register is private to each thread when HT is on. It is private to each core when HT is off.
The kernel must manage the sharing of LBR_SELECT. It allows multiple users on the same logical CPU to use LBR_SELECT as long as they program it with the same value. Across sibling CPUs (HT threads), the same restriction applies on NHM/WSM.
This patch implements this sharing logic by leveraging the mechanism put in place for managing the offcore_response shared MSR.
We modify __intel_shared_reg_get_constraints() to cause x86_get_event_constraint() to be called because LBR may be associated with events that may be counter constrained.
--- arch/x86/kernel/cpu/perf_event.c | 8 +++ arch/x86/kernel/cpu/perf_event_intel.c | 76 ++++++++++++++++++++------------ 2 files changed, 55 insertions(+), 29 deletions(-)
arch/x86/kernel/cpu/perf_event.c +8 -0
@@ -62,6 +62,7 @@ enum extra_reg_type {EXTRA_REG_RSP_0 =0 ,/* offcore_response_0 */ EXTRA_REG_RSP_1 =1 ,/* offcore_response_1 */ +EXTRA_REG_LBR =2 ,/* lbr_select */ EXTRA_REG_MAX /* number of entries needed */ };
@@ -118,6 +119,7 @@ struct cpu_hw_events {void *lbr_context;struct perf_branch_stack lbr_stack;struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES ]; +struct er_account *lbr_sel;/* * manage shared (per-core, per-cpu) registers
@@ -284,6 +286,8 @@ struct x86_pmu {/* * Intel LBR */ +u64 lbr_sel_mask;/* valid bits in LBR_SELECT */ +const int *lbr_sel_map;/* lbr_select mappings */ unsigned long lbr_tos, lbr_from, lbr_to;/* MSR base regs */ int lbr_nr;/* hardware stack size */
@@ -720,6 +724,10 @@ static int __x86_pmu_event_init(struct perf_event *event)/* mark unused */ event->hw.extra_reg.idx =EXTRA_REG_NONE ; +/* mark not used */ + event->hw.extra_reg.idx =EXTRA_REG_NONE ; + event->hw.branch_reg.idx =EXTRA_REG_NONE ; +return x86_pmu.hw_config (event); }
arch/x86/kernel/cpu/perf_event_intel.c +47 -29
124 lines not on this page · raw message
Signature · 1 line
1.7.4.1
PATCH 4/12 perf_events: sync branch stack sampling with X86 precise_sampling
In reply to Stephane EranianIf precise sampling is enabled on Intel X86, then perf_event uses PEBS. To correct for the off-by-one error of PEBS, perf_event uses LBR when precise_sample > 1.
On Intel X86 PERF_SAMPLE_BRANCH_STACK is implemented using LBR, therefore both features must be coordinated as they may not configure LBR the same way.
For PEBS, LBR needs to capture all branches at all priv levels. This patch sets this up.
The configuration of PERF_SAMPLE_BRANCH_STACK may not be compatible in which case an error must be returned.
--- arch/x86/kernel/cpu/perf_event.c | 22 ++++++++++++++++++++++ 1 files changed, 22 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/perf_event.c +22 -0
@@ -654,6 +654,7 @@ static int x86_setup_perfctr(struct perf_event *event)static int x86_pmu_hw_config (struct perf_event *event) {if (event->attr.precise_ip) { +u64 *br_type, br_sel;int precise =0 ;/* Support for constant skid */
@@ -667,6 +668,27 @@ static int x86_pmu_hw_config(struct perf_event *event)if (event->attr.precise_ip > precise)return -EOPNOTSUPP ; +/* +* check that PEBS LBR correction does not conflict with +* whatever the user is asking with attr->branch_sample_type +*/ +if (event->attr.precise_ip >1 ) { + + br_type = &event->attr.branch_sample_type; + +if (has_branch_stack (event)) { + br_sel = *br_type &PERF_SAMPLE_BRANCH_ANY ; +if (br_sel !=PERF_SAMPLE_BRANCH_ANY ) +return -EOPNOTSUPP ; + }else { +/* +* For PEBS fixups, we capture all +* the branches at all priv levels +*/ + *br_type =PERF_SAMPLE_BRANCH_ANY + |PERF_SAMPLE_BRANCH_PLM_ALL ; + } + } }/*
Signature · 1 line
1.7.4.1
Peter Zijlstra
maintainerRe: [PATCH 04/12] perf_events: sync branch stack sampling with X86 precise_sampling
Quoted 4 lines · Stephane Eranian
On Intel X86
PERF_SAMPLE_BRANCH_STACKis implemented using LBR, therefore both features must be coordinated as they may not configure LBR the same way.
Differently, you mean? Both wanting the same configuration seems fine.
Re: [PATCH 04/12] perf_events: sync branch stack sampling with X86 precise_sampling
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
On Intel X86
PERF_SAMPLE_BRANCH_STACKis implemented using LBR, therefore both features must be coordinated as they may not configure LBR the same way.Differently, you mean? Both wanting the same configuration seems fine.
No, I meant you can allow LBR + precise_sampling>1 ONLY when users set LBR to record ALL branches and at the same priv levels. In other words,
you're exposing the LBR content used by the fixup code.
One could argue, that if LBR is set to filter certain branches, it may also be okay,
it's just that you won't necessarily get the same number of successful fixups. The
samples are tagged when fixups were successful, so that may also be an viable option. Best effort given the content of the LBR. Depending on the code, that might
be slightly better than dropping to precise_sampling=1 (no fixups).
Peter Zijlstra
maintainerRe: [PATCH 04/12] perf_events: sync branch stack sampling with X86 precise_sampling
Quoted 7 lines · Stephane Eranian
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
On Intel X86
PERF_SAMPLE_BRANCH_STACKis implemented using LBR, therefore both features must be coordinated as they may not configure LBR the same way.
Quoted 6 lines
Quoted 2 lines
Differently, you mean? Both wanting the same configuration seems fine.
No, I meant you can allow LBR + precise_sampling>1 ONLY when users set LBR to record ALL branches and at the same priv levels. In other words,
you're exposing the LBR content used by the fixup code.
Right.
Quoted 8 lines
One could argue, that if LBR is set to filter certain branches, it may also be okay,
it's just that you won't necessarily get the same number of successful fixups. The
samples are tagged when fixups were successful, so that may also be an viable option. Best effort given the content of the LBR. Depending on the code, that might
be slightly better than dropping to precise_sampling=1 (no fixups).
Possible, but I imagine it might surprise some people.
PATCH 5/12 perf_events: add LBR mappings for PERF_SAMPLE_BRANCH filters
In reply to Stephane EranianThis patch adds the mappings from the generic PERF_SAMPLE_BRANCH_* filters to the actual Intel X86 LBR filters, whenever they exist.
The patch also adds a restriction on Intel Atom, whereby only stepping 10 (PineView) and more recent are supported. Older models, do not have a functional LBR (does not freeze on PMU interrupt).
--- arch/x86/kernel/cpu/perf_event_intel.c | 2 +- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 110 +++++++++++++++++++++++++++- 2 files changed, 109 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/perf_event_intel.c +1 -1
@@ -1612,7 +1612,7 @@ static __init int intel_pmu_init(void)memcpy (hw_cache_event_ids, snb_hw_cache_event_ids,sizeof (hw_cache_event_ids)); -intel_pmu_lbr_init_nhm (); +intel_pmu_lbr_init_snb (); x86_pmu.event_constraints = intel_snb_event_constraints; x86_pmu.pebs_constraints = intel_snb_pebs_events;
arch/x86/kernel/cpu/perf_event_intel_lbr.c +108 -2
150 lines not on this page · raw message
Signature · 1 line
1.7.4.1
PATCH 6/12 perf_events: implement PERF_SAMPLE_BRANCH for Intel X86
In reply to Stephane EranianThis patch implements PERF_SAMPLE_BRANCH support for Intel X86 processors. It connects PERF_SAMPLE_BRANCH to the actual LBR.
The patch adds the hooks in the PMU irq handler to save the LBR on counter overflow for both regular and PEBS modes.
--- arch/x86/kernel/cpu/perf_event_intel.c | 35 +++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 10 ++-- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 73 +++++++++++++++++++++++++++- include/linux/perf_event.h | 3 + 4 files changed, 113 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/perf_event_intel.c +35 -0
@@ -727,6 +727,19 @@ static __initconst const u64 atom_hw_cache_event_ids }, }; +static inline bool intel_pmu_needs_lbr_smpl (struct perf_event *event) +{ +/* user explicitly requested branch sampling */ +if (has_branch_stack (event)) +return true; + +/* implicit branch sampling to correct PEBS skid */ +if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip >1 ) +return true; + +return false; +} +static void intel_pmu_disable_all (void ) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);
@@ -876,6 +889,13 @@ static void intel_pmu_disable_event(struct perf_event *event)return ; } +/* +* must disable before any actual event +* because any event may be combined with LBR +*/ +if (intel_pmu_needs_lbr_smpl (event)) +intel_pmu_lbr_disable (event); +if (unlikely (hwc->config_base ==MSR_ARCH_PERFMON_FIXED_CTR_CTRL )) {intel_pmu_disable_fixed (hwc);return ;
@@ -929,6 +949,12 @@ static void intel_pmu_enable_event(struct perf_event *event)intel_pmu_enable_bts (hwc->config);return ; } +/* +* must enabled before any actual event +* because any event may be combined with LBR +*/ +if (intel_pmu_needs_lbr_smpl (event)) +intel_pmu_lbr_enable (event);if (unlikely (hwc->config_base ==MSR_ARCH_PERFMON_FIXED_CTR_CTRL )) {intel_pmu_enable_fixed (hwc);
@@ -1046,6 +1072,9 @@ again: data.period = event->hw.last_period; +if (has_branch_stack (event)) + data.br_stack = &cpuc->lbr_stack; +if (perf_event_overflow (event, &data, regs))x86_pmu_stop (event,0 ); }
@@ -1279,6 +1308,12 @@ static int intel_pmu_hw_config(struct perf_event *event) event->hw.config = alt_config; } +if (intel_pmu_needs_lbr_smpl (event)) { + ret =intel_pmu_setup_lbr_filter (event); +if (ret) +return ret; + } +if (event->attr.type !=PERF_TYPE_RAW )return 0 ;
arch/x86/kernel/cpu/perf_event_intel_ds.c +4 -6
@@ -455,9 +455,6 @@ static void intel_pmu_pebs_enable(struct perf_event *event) cpuc->pebs_enabled |=1ULL << hwc->idx;WARN_ON_ONCE (cpuc->enabled); - -if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip >1 ) -intel_pmu_lbr_enable (event); }static void intel_pmu_pebs_disable (struct perf_event *event)
@@ -470,9 +467,6 @@ static void intel_pmu_pebs_disable(struct perf_event *event)wrmsrl (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); hwc->config |=ARCH_PERFMON_EVENTSEL_INT ; - -if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip >1 ) -intel_pmu_lbr_disable (event); }static void intel_pmu_pebs_enable_all (void )
@@ -586,6 +580,7 @@ static void __intel_pmu_pebs_event(struct perf_event *event,* both formats and we don't use the other fields in this * routine. */ +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);struct pebs_record_core *pebs = __pebs;struct perf_sample_data data;struct pt_regs regs;
@@ -616,6 +611,9 @@ static void __intel_pmu_pebs_event(struct perf_event *event,else regs.flags &= ~PERF_EFLAGS_EXACT ; +if (has_branch_stack (event)) + data.br_stack = &cpuc->lbr_stack; +if (perf_event_overflow (event, &data, ®s))x86_pmu_stop (event,0 ); }
arch/x86/kernel/cpu/perf_event_intel_lbr.c +71 -2
116 lines not on this page · raw message
include/linux/perf_event.h +3 -0
@@ -159,6 +159,9 @@ enum perf_branch_sample_type { (PERF_SAMPLE_BRANCH_USER |\ PERF_SAMPLE_BRANCH_KERNEL) +#define PERF_SAMPLE_BRANCH_ALL \ + (PERF_SAMPLE_BRANCH_PLM_ALL|PERF_SAMPLE_BRANCH_ANY) +/* * The format of the data returned by read() on a perf event fd, * as specified by attr.read_format:
Signature · 1 line
1.7.4.1
Peter Zijlstra
maintainerRe: [PATCH 06/12] perf_events: implement PERF_SAMPLE_BRANCH for Intel X86
Quoted 4 lines · Stephane Eranian
arch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -876,6 +889,13 @@ static void intel_pmu_disable_event(struct perf_event *event)return ; }
arch/x86/kernel/cpu/perf_event_intel.c static void intel_pmu_disable_event(struct perf_event *event) · line 901
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel.c +7
-0+/* +* must disable before any actual event +* because any event may be combined with LBR +*/ +if (intel_pmu_needs_lbr_smpl (event)) +intel_pmu_lbr_disable (event); +if (unlikely (hwc->config_base ==MSR_ARCH_PERFMON_FIXED_CTR_CTRL )) {intel_pmu_disable_fixed (hwc);return ;
I don't get that, since until you disable the counter a PMI could happen (*) you'd need to disable the LBR after you disable the counters.
* doesn't actually happen since we likely have the whole pmu disabled here, but that's another thing ;-)
Peter Zijlstra
maintainerRe: [PATCH 06/12] perf_events: implement PERF_SAMPLE_BRANCH for Intel X86
In reply to Stephane Eranianarch/x86/kernel/cpu/perf_event_intel.c again:
Quoted 3 lines · Stephane Eranian
arch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -1046,6 +1072,9 @@ again:
$ cat ~/.gitconfig
[diff "default"]
xfuncname = "^[[:alpha:]$_].*[^:]$"Cures that. Although there's different ways too..
https://lkml.org/lkml/2011/8/25/613
PATCH 7/12 perf_events: add LBR software filter support for Intel X86
In reply to Stephane EranianThis patch adds an internal sofware filter to complement the (optional) LBR hardware filter.
The software filter is necessary:
- as a substitute when there is no HW LBR filter (e.g., Atom, Core)
- to complement HW LBR filter in case of errata (e.g., Nehalem/Westmere)
- to provide finer grain filtering (e.g., all processors)
Sometimes, the LBR HW filter cannot distinguish between two types of branches. For instance, to capture syscall as CALLS, it is necessary to enable the LBR_FAR filter which will also capture JMP instructions. Thus, a second pass is necessary to filter those out, this is what the SW filter can do.
The SW filter is built on top of the internal x86 disassembler. It is a best effort filter especially for user level code. It is subject to the availability of the text page of the program.
The SW filter is enabled on all Intel X86 processors. It is bypassed when the user is capturing all branches at all priv levels.
--- arch/x86/kernel/cpu/perf_event.c | 1 + arch/x86/kernel/cpu/perf_event_intel_ds.c | 11 - arch/x86/kernel/cpu/perf_event_intel_lbr.c | 323 +++++++++++++++++++++++++++- 3 files changed, 316 insertions(+), 19 deletions(-)
arch/x86/kernel/cpu/perf_event.c +1 -0
@@ -120,6 +120,7 @@ struct cpu_hw_events {struct perf_branch_stack lbr_stack;struct perf_branch_entry lbr_entries[MAX_LBR_ENTRIES ];struct er_account *lbr_sel; +u64 br_sel;/* * manage shared (per-core, per-cpu) registers
arch/x86/kernel/cpu/perf_event_intel_ds.c +0 -11
@@ -485,17 +485,6 @@ static void intel_pmu_pebs_disable_all(void)wrmsrl (MSR_IA32_PEBS_ENABLE ,0 ); } -#include <asm/insn.h> - -static inline bool kernel_ip (unsigned long ip) -{ -#ifdef CONFIG_X86_32 -return ip >PAGE_OFFSET ; -#else -return (long )ip <0 ; -#endif -} -static int intel_pmu_pebs_fixup_ip (struct pt_regs *regs) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);
arch/x86/kernel/cpu/perf_event_intel_lbr.c +315 -8
384 lines not on this page · raw message
Signature · 1 line
1.7.4.1
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);
This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 6 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
You mean:
arch/x86/kernel/cpu/perf_event_intel_ds.c:intel_pmu_pebs_fixup_ip() doesn't use the opcode decoder on user space code?
ISTR Masami telling me they ran fuzzers on it, feeding it bad instructions etc. But maybe he can tell more.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
You mean:
arch/x86/kernel/cpu/perf_event_intel_ds.c:intel_pmu_pebs_fixup_ip() doesn't use the opcode decoder on user space code?
True it does. Ok the second.
I know uprobes is planning it too, so the problem has to be handled eventually.
Still it makes me uncomfortable.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 9 lines · Andi Kleen
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
Quoted 5 lines
Quoted 3 lines
You mean:
arch/x86/kernel/cpu/perf_event_intel_ds.c:intel_pmu_pebs_fixup_ip() doesn't use the opcode decoder on user space code?True it does. Ok the second.
That's what matters.
Quoted 4 lines
I know uprobes is planning it too, so the problem has to be handled eventually.
Still it makes me uncomfortable.
FUD is not how we approach these things - you need to say something specific.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 2 lines · Ingo Molnar
FUD is not how we approach these things - you need to say something specific.
Its Andi, of course its FUD. I mean, he's been telling us to use IRQs for the PMI instead of NMIs for a while now. If we were to do something daft like that this whole feature, which he does like, would be useless/broken.
He really can't seem to make up his mind.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 6 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
Quoted 6 lines
You mean:
arch/x86/kernel/cpu/perf_event_intel_ds.c:intel_pmu_pebs_fixup_ip() doesn't use the opcode decoder on user space code?ISTR Masami telling me they ran fuzzers on it, feeding it bad instructions etc. But maybe he can tell more.
I've tested it by decoding userspace libraries and binaries. It is also easy to test decoding with random binaries too :) I'll try to do that.
Thanks,
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Andi KleenQuoted 7 lines · Andi Kleen
Quoted 2 lines
+
kernel_insn_init(&insn, kaddr);
+insn_get_opcode(&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
This is not new, it's already used for the PEBS fixups and that includes user level fixups, if possible.
We are not executing the instruction here, just decoding it to filter it out from a buffer if necessary.
Quoted 4 lines
-Andi
--
[email protected] -- Speaking for myself only.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 8 lines · Stephane Eranian
Quoted 7 lines · Andi Kleen
Quoted 2 lines
+
kernel_insn_init(&insn, kaddr);
+insn_get_opcode(&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
Quoted 6 lines
This is not new, it's already used for the PEBS fixups and that includes user level fixups, if possible.
We are not executing the instruction here, just decoding it to filter it out from a buffer if necessary.
I would add that in this particular usage, the source address is coming straight from LBR, it's not made up my SW. That means it corresponds to a point where there was a control flow change. But it can certainly be any x86 opcode (not just branches). LBR captures control flow changes due to traps, faults, interrupts.
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 12 lines · Stephane Eranian
Quoted 11 lines · Stephane Eranian
Quoted 7 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
This is not new, it's already used for the PEBS fixups and that includes user level fixups, if possible.
Quoted 8 lines
Quoted 3 lines
We are not executing the instruction here, just decoding it to filter it out from a buffer if necessary.
I would add that in this particular usage, the source address is coming straight from LBR, it's not made up my SW. That means it corresponds to a point where there was a control flow change. But it can certainly be any x86 opcode (not just branches). LBR captures control flow changes due to traps, faults, interrupts.
You could still fuzz it after the cpu passed through and before the kernel reads the LBR. Its a narrow window, but quite feasible.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Stephane Eranian
Quoted 14 lines · Stephane Eranian
Quoted 7 lines · Andi Kleen
Quoted 2 lines
+
kernel_insn_init(&insn, kaddr);
+insn_get_opcode(&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
This is not new, it's already used for the PEBS fixups and that includes user level fixups, if possible.
We are not executing the instruction here, just decoding it to filter it out from a buffer if necessary.
Quoted 9 lines
Quoted 5 lines
I would add that in this particular usage, the source address is coming straight from LBR, it's not made up my SW. That means it corresponds to a point where there was a control flow change. But it can certainly be any x86 opcode (not just branches). LBR captures control flow changes due to traps, faults, interrupts.
You could still fuzz it after the cpu passed through and before the kernel reads the LBR. Its a narrow window, but quite feasible.
Yeah, depending on the depth of the LBR. But then what happens, you decode an instruction that is not what was executed. In this scenario, either that new instruction matches the user LBR filter or it does not. In the latter situation, the LBR record is not copied into the sampling buffer. In the former case, it is included and all you've done is generate a false profile. I think this can happen even without being malicious but just with self-modifying (JIT) code, though the depth of LBR is such that this is very unlikely (it must happen in the last 16 branches before the PMU interrupt).
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 5 lines · Stephane Eranian
Quoted 3 lines
You could still fuzz it after the cpu passed through and before the kernel reads the LBR. Its a narrow window, but quite feasible.
Yeah, depending on the depth of the LBR. But then what happens, you decode an instruction that is not what was executed.
Right, and Andi's concern is that this might cause our instruction decoder to blow up, or worse.
The whole false profile thing isn't really a problem, I mean, that's what you get for poking at your own instruction stream.
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Stephane EranianQuoted 5 lines · Peter Zijlstra
Quoted 2 lines
Yeah, depending on the depth of the LBR. But then what happens, you decode an instruction that is not what was executed.
Right, and Andi's concern is that this might cause our instruction decoder to blow up, or worse.
That is, we have a fair confidence that its capable of correctly decoding correct streams, but I understood from Masami that they haven't actually tried feeding it crap.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines
Yeah, depending on the depth of the LBR. But then what happens, you decode an instruction that is not what was executed.
Right, and Andi's concern is that this might cause our instruction decoder to blow up, or worse.
That is, we have a fair confidence that its capable of correctly decoding correct streams, but I understood from Masami that they haven't actually tried feeding it crap.
OK, I'll do hardening it.
Thanks,
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
PATCH 1/1 x86: Fix insn decoder for longer instruction
Fix x86 insn decoder for hardening against invalid length instructions. This adds length checkings for each byte-read site and if it exceeds MAX_INSN_SIZE, returns immediately. This can happen when decoding user-space binary.
Caller can check whether it happened by checking insn.*.got member is set or not.
---
arch/x86/lib/insn.c | 48 +++++++++++++++++++++++++++++++++++++++++++----- 1 files changed, 43 insertions(+), 5 deletions(-)
arch/x86/lib/insn.c +43 -5
@@ -22,14 +22,23 @@#include <asm/inat.h> #include <asm/insn.h> -#define get_next (t, insn) \ - ({t r; r = *(t*)insn->next_byte; insn->next_byte += sizeof(t); r; }) +/* Verify next sizeof(t) bytes can be on the same instruction */ +#define validate_next (t, insn, n) \ + ((insn)->next_byte + sizeof(t) + n - (insn)->kaddr <= MAX_INSN_SIZE) + +#define __get_next (t, insn) \ + ({ t r = *(t*)insn->next_byte; insn->next_byte += sizeof(t); r; }) + +#define __peek_nbyte_next (t, insn, n) \ + ({ t r = *(t*)((insn)->next_byte + n); r; }) -#define peek_next (t, insn) \ - ({t r; r = *(t*)insn->next_byte; r; }) +#define get_next (t, insn) \ + ({ if (unlikely(!validate_next(t, insn, 0))) goto err_out; __get_next(t, insn); })#define peek_nbyte_next (t, insn, n) \ - ({t r; r = *(t*)((insn)->next_byte + n); r; }) + ({ if (unlikely(!validate_next(t, insn, n))) goto err_out; __peek_nbyte_next(t, insn, n); }) + +#define peek_next (t, insn) peek_nbyte_next(t, insn, 0)/** * insn_init() - initialize struct insn
@@ -158,6 +167,8 @@ vex_end: insn->vex_prefix.got =1 ; prefixes->got =1 ; + +err_out:return ; }
@@ -208,6 +219,9 @@ void insn_get_opcode(struct insn *insn) insn->attr =0 ;/* This instruction is bad */ end: opcode->got =1 ; + +err_out: +return ; }/**
@@ -241,6 +255,9 @@ void insn_get_modrm(struct insn *insn)if (insn->x86_64 &&inat_is_force64 (insn->attr)) insn->opnd_bytes =8 ; modrm->got =1 ; + +err_out: +return ; }
@@ -290,6 +307,9 @@ void insn_get_sib(struct insn *insn) } } insn->sib.got =1 ; + +err_out: +return ; }
@@ -351,6 +371,9 @@ void insn_get_displacement(struct insn *insn) } out: insn->displacement.got =1 ; + +err_out: +return ; }/* Decode moffset16/32/64 */
@@ -373,6 +396,9 @@ static void __get_moffset(struct insn *insn)break ; } insn->moffset1.got = insn->moffset2.got =1 ; + +err_out: +return ; }/* Decode imm v32(Iz) */
@@ -389,6 +415,9 @@ static void __get_immv32(struct insn *insn) insn->immediate.nbytes =4 ;break ; } + +err_out: +return ; }/* Decode imm v64(Iv/Ov) */
@@ -411,6 +440,9 @@ static void __get_immv(struct insn *insn)break ; } insn->immediate1.got = insn->immediate2.got =1 ; + +err_out: +return ; }/* Decode ptr16:16/32(Ap) */
@@ -432,6 +464,9 @@ static void __get_immptr(struct insn *insn) insn->immediate2.value = get_next(unsigned short, insn); insn->immediate2.nbytes =2 ; insn->immediate1.got = insn->immediate2.got =1 ; + +err_out: +return ; }/**
@@ -496,6 +531,9 @@ void insn_get_immediate(struct insn *insn) } done: insn->immediate.got =1 ; + +err_out: +return ; }/**
Re: [PATCH] x86: Fix insn decoder for longer instruction
Quoted 7 lines · Masami Hiramatsu
Fix x86 insn decoder for hardening against invalid length instructions. This adds length checkings for each byte-read site and if it exceeds
MAX_INSN_SIZE, returns immediately. This can happen when decoding user-space binary.Caller can check whether it happened by checking insn.*.got member is set or not.
Thanks.
We really need the /dev/urandom based build-time tester as well.
Thanks,
Ingo
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Masami HiramatsuQuoted 5 lines · Masami Hiramatsu
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Quoted 2 lines
Yeah, depending on the depth of the LBR. But then what happens, you decode an instruction that is not what was executed.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Right, and Andi's concern is that this might cause our instruction decoder to blow up, or worse.
That is, we have a fair confidence that its capable of correctly decoding correct streams, but I understood from Masami that they haven't actually tried feeding it crap.
OK, I'll do hardening it.
Yes, that's a good idea. Could you perhaps add it to the existing build time sanity test we do for the instruction decoder, to feed it a sequence of /dev/urandom input or such?
Thanks,
Ingo
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 15 lines · Ingo Molnar
Quoted 12 lines · Masami Hiramatsu
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines
Yeah, depending on the depth of the LBR. But then what happens, you decode an instruction that is not what was executed.
Right, and Andi's concern is that this might cause our instruction decoder to blow up, or worse.
That is, we have a fair confidence that its capable of correctly decoding correct streams, but I understood from Masami that they haven't actually tried feeding it crap.
Quoted 5 lines
Quoted 1 line
OK, I'll do hardening it.
Yes, that's a good idea. Could you perhaps add it to the existing build time sanity test we do for the instruction decoder, to feed it a sequence of
/dev/urandominput or such?
Ah, nice. Maybe we need another test binary, since current one is just ensuring the output of objdump and decoder is same. anyway it's not so difficult if it feeds random binaries to ensure the decoder doesn't access bad address.
Thank you,
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 4 lines
Ah, nice. Maybe we need another test binary, since current one is just ensuring the output of objdump and decoder is same. anyway it's not so difficult if it feeds random binaries to ensure the decoder doesn't access bad address.
Pure /dev/urandom is not good because it cannot be ever reproduced. Better use a PRNG with random seed from urandom, but print the seed.
In addition to random I would do fuzzing: take an existing stream and just corrupt some bits and groups of bits. This will exercise different paths. In fact fuzzing is probably better than random for most tests.
Not sure it's needed to run on every build though, just checks now and then should be sufficient.
-Andi
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 7 lines · Andi Kleen
Quoted 4 lines
Ah, nice. Maybe we need another test binary, since current one is just ensuring the output of objdump and decoder is same. anyway it's not so difficult if it feeds random binaries to ensure the decoder doesn't access bad address.
Pure
/dev/urandomis not good because it cannot be ever reproduced. Better use a PRNG with random seed from urandom, but print the seed.
Sure,
Quoted 6 lines
In addition to random I would do fuzzing: take an existing stream and just corrupt some bits and groups of bits. This will exercise different paths. In fact fuzzing is probably better than random for most tests.
Not sure it's needed to run on every build though, just checks now and then should be sufficient.
I made a test for that, and I didn't hit any random bytes which can not be decoded after I've tested 100,000,000. (with previous hardening patch)
since the number of the combination is 2^128 (MAX_INSN_SIZE is 16), too huge to check all of them. Of course, the real number of possible combination should be smaller than that.
- Because this decoder don't evaluate but just decode,
we can skip immediates and operands.
- If we hit the non-oprand opcode, mod/rm, sib, or displacement in the middle of byte stream, we can stop trying decode tail bytes.
So I think there is smarter way to cover all possible combination for the decoder than feeding random bytes.
Anyway, the decoder code can evolve in the future, I think it should be put into the linux kernel.
Thank you,
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 10 lines · Masami Hiramatsu
Quoted 7 lines · Andi Kleen
Quoted 4 lines
Ah, nice. Maybe we need another test binary, since current one is just ensuring the output of objdump and decoder is same. anyway it's not so difficult if it feeds random binaries to ensure the decoder doesn't access bad address.
Pure
/dev/urandomis not good because it cannot be ever reproduced. Better use a PRNG with random seed from urandom, but print the seed.Sure,
Obviously the urandom data should be kbuild generated and only cleared on 'make clean' - not regenerated on every run. So if there's a failure the failing urandom data stays around readily.
Thanks,
Ingo
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Masami Hiramatsu
Quoted 5 lines · Andi Kleen
Quoted 4 lines
Ah, nice. Maybe we need another test binary, since current one is just ensuring the output of objdump and decoder is same. anyway it's not so difficult if it feeds random binaries to ensure the decoder doesn't access bad address.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Pure
/dev/urandomis not good because it cannot be ever reproduced. Better use a PRNG with random seed from urandom, but print the seed.Sure,
Obviously the urandom data should be kbuild generated and only cleared on 'make clean' - not regenerated on every run. So if there's a failure the failing urandom data stays around readily.
I've added options for passing a combination of seed and seqno or instruction sequence directly. So if someone find an error message, they can easily reproduce it or report it.
Thank you,
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
RFC PATCH 1/1 x86: Add a sanity test of x86 decoder
In reply to Ingo MolnarAdd a sanity test of x86 insn decoder against the random input. This test is also able to reproduce the bug by passing random-seed and iteration-number, or by passing input while which has invalid byte code.
---
arch/x86/tools/Makefile | 10 +- arch/x86/tools/insn_sanity.c | 253 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+), 1 deletions(-) create mode 100644 arch/x86/tools/insn_sanity.c
arch/x86/tools/Makefile +9 -1
@@ -18,14 +18,22 @@ chkobjdump = $(srctree)/arch/x86/tools/chkobjdump.awk quiet_cmd_posttest = TEST $@ cmd_posttest = ($(OBJDUMP) -v | $(AWK) -f $(chkobjdump)) || $(OBJDUMP) -d -j .text $(objtree)/vmlinux | $(AWK) -f $(distill_awk) | $(obj)/test_get_len $(posttest_64bit) $(posttest_verbose) -posttest: $(obj)/test_get_len vmlinux +quiet_cmd_sanitytest = TEST $@ + cmd_sanitytest = $(obj)/insn_sanity $(posttest_64bit) -m 1000000 + +posttest: $(obj)/test_get_len vmlinux $(obj)/insn_sanity $(call cmd,posttest) + $(call cmd,sanitytest) hostprogs-y := test_get_len +hostprogs-y := insn_sanity # -I needed for generated C source and C source which in the kernel tree. HOSTCFLAGS_test_get_len.o := -Wall -I$(objtree)/arch/x86/lib/ -I$(srctree)/arch/x86/include/ -I$(srctree)/arch/x86/lib/ -I$(srctree)/include/ +HOSTCFLAGS_insn_sanity.o := -Wall -I$(objtree)/arch/x86/lib/ -I$(srctree)/arch/x86/include/ -I$(srctree)/arch/x86/lib/ -I$(srctree)/include/ + # Dependencies are also needed. $(obj)/test_get_len.o: $(srctree)/arch/x86/lib/insn.c $(srctree)/arch/x86/lib/inat.c $(srctree)/arch/x86/include/asm/inat_types.h $(srctree)/arch/x86/include/asm/inat.h $(srctree)/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c +$(obj)/insn_sanity.o: $(srctree)/arch/x86/lib/insn.c $(srctree)/arch/x86/lib/inat.c $(srctree)/arch/x86/include/asm/inat_types.h $(srctree)/arch/x86/include/asm/inat.h $(srctree)/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c
arch/x86/tools/insn_sanity.c new file +253 -0
253 lines not on this page · raw message
Re: [RFC PATCH] x86: Add a sanity test of x86 decoder
Quoted 4 lines · Masami Hiramatsu
Add a sanity test of x86 insn decoder against the random input. This test is also able to reproduce the bug by passing random-seed and iteration-number, or by passing input while which has invalid byte code.
Looks good in general.
a few nitpicking details:
arch/x86/tools/Makefile chkobjdump = $(srctree)/arch/x86/tools/chkobjdump.awk · line 22
Quoted 3 lines
arch/x86/tools/Makefile +2
-1-posttest: $(obj)/test_get_len vmlinux +quiet_cmd_sanitytest = TEST $@ + cmd_sanitytest = $(obj)/insn_sanity $(posttest_64bit) -m 1000000
Just curious, what's the execution time for this? I'd expect milliseconds, but there will also be urandom overhead.
Quoted 13 lines
arch/x86/tools/insn_sanity.c new file +13
-0+#define unlikely (cond) (cond) + +#include <asm/insn.h> +#include <inat.c> +#include <insn.c> + +/* +* Test of instruction analysis against tampering. +* Feed random binary to instruction decoder and ensure not to +* access out-of-instruction-buffer. +*/ + +#define DEFAULT_MAX_ITER 10000
arch/x86/tools/insn_sanity.c line 53
Quoted 10 lines
arch/x86/tools/insn_sanity.c new file +10
-0+#define INSN_NOP 0x90 + +static const char *prog; +static int verbose; +static int x86_64; +static unsigned int seed; +static unsigned long nr_iter; +static unsigned long max_iter =DEFAULT_MAX_ITER ; +static char insn_buf[MAX_INSN_SIZE *2 ]; +static FILE *input_file;
This needs more comments and a bit more vertical structure.
arch/x86/tools/insn_sanity.c line 101
Quoted 4 lines
arch/x86/tools/insn_sanity.c new file +4
-0+static void dump_stream (FILE *fp,const char *msg,struct insn *insn) +{ +int i; +fprintf (fp,"%s:\n code:" , msg);
missing newline.
arch/x86/tools/insn_sanity.c line 127
Quoted 6 lines
arch/x86/tools/insn_sanity.c new file +6
-0+static int get_next_insn (void ) +{ +char buf[256 ] ="" , *tmp; +int i; + + tmp =fgets (buf,256 , input_file);
ARRAY_SIZE().
arch/x86/tools/insn_sanity.c line 132
Quoted 5 lines
arch/x86/tools/insn_sanity.c new file +5
-0+if (tmp ==NULL ||feof (input_file)) +return 0 ; + +for (i =0 ; i <MAX_INSN_SIZE ; i++) { + ((unsigned char *)insn_buf)[i] = (unsigned char )strtoul (tmp, &tmp,16 );
why is this cast needed? Shouldnt insn_buf[] have this type if this is how it's used?
arch/x86/tools/insn_sanity.c line 143
Quoted 3 lines
arch/x86/tools/insn_sanity.c new file +3
-0+{ +int i; +if (nr_iter >= max_iter)
missing newline.
arch/x86/tools/insn_sanity.c line 153
Quoted 2 lines
arch/x86/tools/insn_sanity.c new file +2
-0+for (i =0 ; i <MAX_INSN_SIZE ; i +=2 ) + *(unsigned short *)(&insn_buf[i]) =random () &0xffff ;
this silently assumes that MAX_INSN_SIZE is a multiple of 2. Which is true ... for now.
arch/x86/tools/insn_sanity.c line 222
Quoted 3 lines
arch/x86/tools/insn_sanity.c new file +3
-0+#define insn_complete (insn) \ + (insn.opcode.got && insn.modrm.got && insn.sib.got && \ + insn.displacement.got && insn.immediate.got)
This could move into insn.h (even though only user-space uses it), right?
arch/x86/tools/insn_sanity.c line 232
Quoted 1 line
arch/x86/tools/insn_sanity.c new file +1
-0+while (generate_random_insn ()) {
this loop is really weird: half of it is hidden in generate_random_insn()'s use of nr_iter global variable!
Why not just do it in the old-fashioned way:
for (i = 0; i < max_iter; i++) {
...
}and keep generate_random_insn() loop-invariant?
arch/x86/tools/insn_sanity.c line 244
Quoted 5 lines
arch/x86/tools/insn_sanity.c new file +5
-0+if (insn.next_byte <= insn.kaddr || + insn.kaddr +MAX_INSN_SIZE < insn.next_byte) { +/* Access out-of-range memory */ +dump_stream (stdout,"Find access violation" , &insn); + warnings++;
s/Find/Found ?
arch/x86/tools/insn_sanity.c line 251
Quoted 4 lines
arch/x86/tools/insn_sanity.c new file +4
-0+if (warnings) +fprintf (stdout,"Warning: decoded and checked %d insns with %d warnings (seed:0x%x)\n" , insns, warnings, seed); +else +fprintf (stdout,"Succeed: decoded and checked %d insns (seed:0x%x)\n" , insns, seed);
s/Succeed/Success ?
Also, s/insns/random instructions - there's rarely any good reason to abbreviate in human readable output.
Thanks,
Ingo
Re: [RFC PATCH] x86: Add a sanity test of x86 decoder
Quoted 11 lines · Ingo Molnar
Quoted 4 lines · Masami Hiramatsu
Add a sanity test of x86 insn decoder against the random input. This test is also able to reproduce the bug by passing random-seed and iteration-number, or by passing input while which has invalid byte code.
Looks good in general.
a few nitpicking details:
Thank you for the comments :)
Quoted 7 lines
arch/x86/tools/Makefile chkobjdump = $(srctree)/arch/x86/tools/chkobjdump.awk · line 22
Quoted 4 lines
arch/x86/tools/Makefile +2
-1-posttest: $(obj)/test_get_len vmlinux +quiet_cmd_sanitytest = TEST $@ + cmd_sanitytest = $(obj)/insn_sanity $(posttest_64bit) -m 1000000Just curious, what's the execution time for this? I'd expect milliseconds, but there will also be urandom overhead.
I measured it with time command.
---
Succeed: decoded and checked 1000000 insns (seed:0x73f2b3bb)
real 0m0.152s user 0m0.149s sys 0m0.002s
---
Here, you can see how long it takes. It actually refers /dev/urandom just one time at start, so I guess there is no urandom overhead.
Quoted 17 lines
Quoted 17 lines
arch/x86/tools/insn_sanity.c new file +17
-0+#define unlikely (cond) (cond) + +#include <asm/insn.h> +#include <inat.c> +#include <insn.c> + +/* +* Test of instruction analysis against tampering. +* Feed random binary to instruction decoder and ensure not to +* access out-of-instruction-buffer. +*/ + +#define DEFAULT_MAX_ITER 10000 +#define INSN_NOP 0x90 + +static const char *prog; +static int verbose;
Quoted 8 lines
arch/x86/tools/insn_sanity.c line 53
Quoted 6 lines
arch/x86/tools/insn_sanity.c new file +6
-0+static int x86_64; +static unsigned int seed; +static unsigned long nr_iter; +static unsigned long max_iter =DEFAULT_MAX_ITER ; +static char insn_buf[MAX_INSN_SIZE *2 ]; +static FILE *input_file;This needs more comments and a bit more vertical structure.
OK.
Quoted 6 lines
arch/x86/tools/insn_sanity.c line 101
Quoted 4 lines
arch/x86/tools/insn_sanity.c new file +4
-0+static void dump_stream (FILE *fp,const char *msg,struct insn *insn) +{ +int i; +fprintf (fp,"%s:\n code:" , msg);missing newline.
This prints a header of code sequence, so that we'll see a message like below;
Error message:
code: 00 01 02 03 04 ...
Quoted 8 lines
arch/x86/tools/insn_sanity.c line 127
Quoted 6 lines
arch/x86/tools/insn_sanity.c new file +6
-0+static int get_next_insn (void ) +{ +char buf[256 ] ="" , *tmp; +int i; + + tmp =fgets (buf,256 , input_file);
ARRAY_SIZE().
OK.
Quoted 8 lines
arch/x86/tools/insn_sanity.c line 132
Quoted 5 lines
arch/x86/tools/insn_sanity.c new file +5
-0+if (tmp ==NULL ||feof (input_file)) +return 0 ; + +for (i =0 ; i <MAX_INSN_SIZE ; i++) { + ((unsigned char *)insn_buf)[i] = (unsigned char )strtoul (tmp, &tmp,16 );why is this cast needed? Shouldnt insn_buf[] have this type if this is how it's used?
Yes, the type of insn_buf can be changed.
Quoted 7 lines
arch/x86/tools/insn_sanity.c line 143
Quoted 4 lines
arch/x86/tools/insn_sanity.c new file +3
-0+{ +int i; +if (nr_iter >= max_iter)missing newline.
Quoted 5 lines
arch/x86/tools/insn_sanity.c line 153
Quoted 2 lines
arch/x86/tools/insn_sanity.c new file +2
-0+for (i =0 ; i <MAX_INSN_SIZE ; i +=2 ) + *(unsigned short *)(&insn_buf[i]) =random () &0xffff ;this silently assumes that
MAX_INSN_SIZEis a multiple of 2. Which is true ... for now.
Ah, right. OK, add a line to fill the last byte if needed.
Quoted 6 lines
arch/x86/tools/insn_sanity.c line 222
Quoted 3 lines
arch/x86/tools/insn_sanity.c new file +3
-0+#define insn_complete (insn) \ + (insn.opcode.got && insn.modrm.got && insn.sib.got && \ + insn.displacement.got && insn.immediate.got)This could move into
insn.h(even though only user-space uses it), right?
Right.
Quoted 7 lines
arch/x86/tools/insn_sanity.c line 232
Quoted 2 lines
arch/x86/tools/insn_sanity.c new file +1
-0+while (generate_random_insn ()) {this loop is really weird: half of it is hidden in generate_random_insn()'s use of
nr_iterglobal variable!
Quoted 7 lines
Why not just do it in the old-fashioned way:
for (i = 0; i < max_iter; i++) { ... }and keep
generate_random_insn()loop-invariant?
OK, I'll change this.
Quoted 9 lines
arch/x86/tools/insn_sanity.c line 244
Quoted 6 lines
arch/x86/tools/insn_sanity.c new file +5
-0+if (insn.next_byte <= insn.kaddr || + insn.kaddr +MAX_INSN_SIZE < insn.next_byte) { +/* Access out-of-range memory */ +dump_stream (stdout,"Find access violation" , &insn); + warnings++;s/Find/Found ?
Quoted 6 lines
arch/x86/tools/insn_sanity.c line 251
Quoted 4 lines
arch/x86/tools/insn_sanity.c new file +4
-0+if (warnings) +fprintf (stdout,"Warning: decoded and checked %d insns with %d warnings (seed:0x%x)\n" , insns, warnings, seed); +else +fprintf (stdout,"Succeed: decoded and checked %d insns (seed:0x%x)\n" , insns, seed);s/Succeed/Success ?
Oops...
Quoted 2 lines
Also, s/insns/random instructions - there's rarely any good reason to abbreviate in human readable output.
Agreed.
Thank you!
Quoted 4 lines
Thanks,
Ingo
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [RFC PATCH] x86: Add a sanity test of x86 decoder
Quoted 9 lines · Masami Hiramatsu
Quoted 6 lines
arch/x86/tools/insn_sanity.c line 101
Quoted 4 lines
arch/x86/tools/insn_sanity.c new file +4
-0+static void dump_stream (FILE *fp,const char *msg,struct insn *insn) +{ +int i; +fprintf (fp,"%s:\n code:" , msg);missing newline.
This prints a header of code sequence, so that we'll see a message like below;
I mean, it's missing the standard newline between local variable(s) and the first statement:
static void dump_stream(FILE *fp, const char *msg, struct insn *insn)
{
int i;
fprintf(fp, "%s:\n code:", msg);[...]
We do this for readability.
Thanks,
Ingo
RFC PATCH v2 1/2 x86: Add a sanity test of x86 decoder
Add a sanity test of x86 insn decoder against the random input. This test is also able to reproduce the bug by passing random-seed and iteration-number, or by passing input while which has invalid byte code.
Changes in V2:
- Code cleanup.
- Show how to reproduce the error by insn_sanity test.
---
arch/x86/include/asm/insn.h | 7 + arch/x86/tools/Makefile | 10 +- arch/x86/tools/insn_sanity.c | 275 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 291 insertions(+), 1 deletions(-) create mode 100644 arch/x86/tools/insn_sanity.c
arch/x86/include/asm/insn.h +7 -0
@@ -137,6 +137,13 @@ static inline int insn_is_avx(struct insn *insn)return (insn->vex_prefix.value !=0 ); } +/* Ensure this instruction is decoded completely */ +static inline int insn_complete (struct insn *insn) +{ +return insn->opcode.got && insn->modrm.got && insn->sib.got && + insn->displacement.got && insn->immediate.got; +} +static inline insn_byte_t insn_vex_m_bits (struct insn *insn) {if (insn->vex_prefix.nbytes ==2 )/* 2 bytes VEX */
arch/x86/tools/Makefile +9 -1
@@ -18,14 +18,22 @@ chkobjdump = $(srctree)/arch/x86/tools/chkobjdump.awk quiet_cmd_posttest = TEST $@ cmd_posttest = ($(OBJDUMP) -v | $(AWK) -f $(chkobjdump)) || $(OBJDUMP) -d -j .text $(objtree)/vmlinux | $(AWK) -f $(distill_awk) | $(obj)/test_get_len $(posttest_64bit) $(posttest_verbose) -posttest: $(obj)/test_get_len vmlinux +quiet_cmd_sanitytest = TEST $@ + cmd_sanitytest = $(obj)/insn_sanity $(posttest_64bit) -m 1000000 + +posttest: $(obj)/test_get_len vmlinux $(obj)/insn_sanity $(call cmd,posttest) + $(call cmd,sanitytest) hostprogs-y := test_get_len +hostprogs-y := insn_sanity # -I needed for generated C source and C source which in the kernel tree. HOSTCFLAGS_test_get_len.o := -Wall -I$(objtree)/arch/x86/lib/ -I$(srctree)/arch/x86/include/ -I$(srctree)/arch/x86/lib/ -I$(srctree)/include/ +HOSTCFLAGS_insn_sanity.o := -Wall -I$(objtree)/arch/x86/lib/ -I$(srctree)/arch/x86/include/ -I$(srctree)/arch/x86/lib/ -I$(srctree)/include/ + # Dependencies are also needed. $(obj)/test_get_len.o: $(srctree)/arch/x86/lib/insn.c $(srctree)/arch/x86/lib/inat.c $(srctree)/arch/x86/include/asm/inat_types.h $(srctree)/arch/x86/include/asm/inat.h $(srctree)/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c +$(obj)/insn_sanity.o: $(srctree)/arch/x86/lib/insn.c $(srctree)/arch/x86/lib/inat.c $(srctree)/arch/x86/include/asm/inat_types.h $(srctree)/arch/x86/include/asm/inat.h $(srctree)/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c
arch/x86/tools/insn_sanity.c new file +275 -0
275 lines not on this page · raw message
[tip:perf/core] x86, perf: Add a build-time sanity test to the x86 decoder
Commit-ID: 1ec454baf1245df4fdb5dae728da3363630ce6de Gitweb: http://git.kernel.org/tip/1ec454baf1245df4fdb5dae728da3363630ce6de Author: Masami Hiramatsu <[email protected]> AuthorDate: Thu, 20 Oct 2011 23:01:09 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 10 Nov 2011 12:38:51 +0100
x86, perf: Add a build-time sanity test to the x86 decoder
Add a sanity test of x86 insn decoder against a stream of randomly generated input, at build time.
This test is also able to reproduce any bug that might trigger by allowing the passing of random-seed and iteration-number to the test, or by passing input which has invalid byte code.
Changes in V2:
- Code cleanup.
- Show how to reproduce the error by insn_sanity test.
--- arch/x86/include/asm/insn.h | 7 + arch/x86/tools/Makefile | 10 ++- arch/x86/tools/insn_sanity.c | 275 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 291 insertions(+), 1 deletions(-)
arch/x86/include/asm/insn.h +7 -0
@@ -137,6 +137,13 @@ static inline int insn_is_avx(struct insn *insn)return (insn->vex_prefix.value !=0 ); } +/* Ensure this instruction is decoded completely */ +static inline int insn_complete (struct insn *insn) +{ +return insn->opcode.got && insn->modrm.got && insn->sib.got && + insn->displacement.got && insn->immediate.got; +} +static inline insn_byte_t insn_vex_m_bits (struct insn *insn) {if (insn->vex_prefix.nbytes ==2 )/* 2 bytes VEX */
arch/x86/tools/Makefile +9 -1
@@ -18,14 +18,22 @@ chkobjdump = $(srctree)/arch/x86/tools/chkobjdump.awk quiet_cmd_posttest = TEST $@ cmd_posttest = ($(OBJDUMP) -v | $(AWK) -f $(chkobjdump)) || $(OBJDUMP) -d -j .text $(objtree)/vmlinux | $(AWK) -f $(distill_awk) | $(obj)/test_get_len $(posttest_64bit) $(posttest_verbose) -posttest: $(obj)/test_get_len vmlinux +quiet_cmd_sanitytest = TEST $@ + cmd_sanitytest = $(obj)/insn_sanity $(posttest_64bit) -m 1000000 + +posttest: $(obj)/test_get_len vmlinux $(obj)/insn_sanity $(call cmd,posttest) + $(call cmd,sanitytest) hostprogs-y := test_get_len +hostprogs-y := insn_sanity # -I needed for generated C source and C source which in the kernel tree. HOSTCFLAGS_test_get_len.o := -Wall -I$(objtree)/arch/x86/lib/ -I$(srctree)/arch/x86/include/ -I$(srctree)/arch/x86/lib/ -I$(srctree)/include/ +HOSTCFLAGS_insn_sanity.o := -Wall -I$(objtree)/arch/x86/lib/ -I$(srctree)/arch/x86/include/ -I$(srctree)/arch/x86/lib/ -I$(srctree)/include/ + # Dependencies are also needed. $(obj)/test_get_len.o: $(srctree)/arch/x86/lib/insn.c $(srctree)/arch/x86/lib/inat.c $(srctree)/arch/x86/include/asm/inat_types.h $(srctree)/arch/x86/include/asm/inat.h $(srctree)/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c +$(obj)/insn_sanity.o: $(srctree)/arch/x86/lib/insn.c $(srctree)/arch/x86/lib/inat.c $(srctree)/arch/x86/include/asm/inat_types.h $(srctree)/arch/x86/include/asm/inat.h $(srctree)/arch/x86/include/asm/insn.h $(objtree)/arch/x86/lib/inat-tables.c
arch/x86/tools/insn_sanity.c new file +275 -0
275 lines not on this page · raw message
RFC PATCH v2 2/2 x86: Fix insn decoder for longer instruction
In reply to Ingo MolnarFix x86 insn decoder for hardening against invalid length instructions. This adds length checkings for each byte-read site and if it exceeds MAX_INSN_SIZE, returns immediately. This can happen when decoding unreliable binary.
For example, without this patch,
$ echo 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 |\
./insn_sanity -i -
Error: Found an access violation:
Instruction = {
.prefixes = {
.value = 1711276134, bytes[] = {66, 0, 0, 66},
.got = 1, .nbytes = 16},
.rex_prefix = {
.value = 0, bytes[] = {0, 0, 0, 0},
.got = 1, .nbytes = 0},
.vex_prefix = {
.value = 0, bytes[] = {0, 0, 0, 0},
.got = 1, .nbytes = 0},
.opcode = {
.value = 144, bytes[] = {90, 0, 0, 0},
.got = 1, .nbytes = 1},
...
.attr = 0, .opnd_bytes = 2, .addr_bytes = 4,
.length = 17, .x86_64 = 0, .kaddr = 0x7fffedc67ae0}
You can reproduce this with below command(s);
$ echo 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 | ./insn_sanity -i -
Failure: decoded and checked 1 given instructions with 1 errors (seed:0x0)With this patch;
$ echo 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 |\ ./insn_sanity -i - Success: decoded and checked 1 given instructions with 0 errors (seed:0x0)
Caller can check whether it happened by insn_complete().
---
arch/x86/lib/insn.c | 48 +++++++++++++++++++++++++++++++++++++++++++----- 1 files changed, 43 insertions(+), 5 deletions(-)
arch/x86/lib/insn.c +43 -5
@@ -22,14 +22,23 @@#include <asm/inat.h> #include <asm/insn.h> -#define get_next (t, insn) \ - ({t r; r = *(t*)insn->next_byte; insn->next_byte += sizeof(t); r; }) +/* Verify next sizeof(t) bytes can be on the same instruction */ +#define validate_next (t, insn, n) \ + ((insn)->next_byte + sizeof(t) + n - (insn)->kaddr <= MAX_INSN_SIZE) + +#define __get_next (t, insn) \ + ({ t r = *(t*)insn->next_byte; insn->next_byte += sizeof(t); r; }) + +#define __peek_nbyte_next (t, insn, n) \ + ({ t r = *(t*)((insn)->next_byte + n); r; }) -#define peek_next (t, insn) \ - ({t r; r = *(t*)insn->next_byte; r; }) +#define get_next (t, insn) \ + ({ if (unlikely(!validate_next(t, insn, 0))) goto err_out; __get_next(t, insn); })#define peek_nbyte_next (t, insn, n) \ - ({t r; r = *(t*)((insn)->next_byte + n); r; }) + ({ if (unlikely(!validate_next(t, insn, n))) goto err_out; __peek_nbyte_next(t, insn, n); }) + +#define peek_next (t, insn) peek_nbyte_next(t, insn, 0)/** * insn_init() - initialize struct insn
@@ -158,6 +167,8 @@ vex_end: insn->vex_prefix.got =1 ; prefixes->got =1 ; + +err_out:return ; }
@@ -208,6 +219,9 @@ void insn_get_opcode(struct insn *insn) insn->attr =0 ;/* This instruction is bad */ end: opcode->got =1 ; + +err_out: +return ; }/**
@@ -241,6 +255,9 @@ void insn_get_modrm(struct insn *insn)if (insn->x86_64 &&inat_is_force64 (insn->attr)) insn->opnd_bytes =8 ; modrm->got =1 ; + +err_out: +return ; }
@@ -290,6 +307,9 @@ void insn_get_sib(struct insn *insn) } } insn->sib.got =1 ; + +err_out: +return ; }
@@ -351,6 +371,9 @@ void insn_get_displacement(struct insn *insn) } out: insn->displacement.got =1 ; + +err_out: +return ; }/* Decode moffset16/32/64 */
@@ -373,6 +396,9 @@ static void __get_moffset(struct insn *insn)break ; } insn->moffset1.got = insn->moffset2.got =1 ; + +err_out: +return ; }/* Decode imm v32(Iz) */
@@ -389,6 +415,9 @@ static void __get_immv32(struct insn *insn) insn->immediate.nbytes =4 ;break ; } + +err_out: +return ; }/* Decode imm v64(Iv/Ov) */
@@ -411,6 +440,9 @@ static void __get_immv(struct insn *insn)break ; } insn->immediate1.got = insn->immediate2.got =1 ; + +err_out: +return ; }/* Decode ptr16:16/32(Ap) */
@@ -432,6 +464,9 @@ static void __get_immptr(struct insn *insn) insn->immediate2.value = get_next(unsigned short, insn); insn->immediate2.nbytes =2 ; insn->immediate1.got = insn->immediate2.got =1 ; + +err_out: +return ; }/**
@@ -496,6 +531,9 @@ void insn_get_immediate(struct insn *insn) } done: insn->immediate.got =1 ; + +err_out: +return ; }/**
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Stephane EranianQuoted 3 lines
just with self-modifying (JIT) code, though the depth of LBR is such that this is very unlikely (it must happen in the last 16 branches before the PMU interrupt).
Interrupts can make this window arbitary long.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Stephane EranianQuoted 10 lines · Stephane Eranian
Quoted 7 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_lbr.c static int intel_pmu_setup_lbr_filter(struct perf_event *event) · line 456
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_lbr.c +2
-0+kernel_insn_init (&insn, kaddr); +insn_get_opcode (&insn);This makes me uncomfortable. AFAIK that's the first use of the opcode decoder being used directly for user space. It has a quite large attack surface. Who says it cannot be exploited?
This is not new, it's already used for the PEBS fixups and that includes user level fixups, if possible.
Oops, I've thought existing code is only for kernel. Hmm, I guess this could cause a problem when running 32bit binary on x86-64.
As you know, there are differences between decoding routines of x86-64 and -32. So insn_init() has the third parameter to give a flag for changing it.
However, since the kernel itself runs on the native mode, I've added kernel_insn_init() only for the kernel decoding.
----
/* Init insn for kernel text */
static inline void kernel_insn_init(struct insn *insn, const void *kaddr) {
#ifdef CONFIG_X86_64
insn_init(insn, kaddr, 1);
#else /* CONFIG_X86_32 */
insn_init(insn, kaddr, 0);#endif
}
----
Thus, I strongly recommend you to check the target instruction bitwidth on x86-64 as (probably) below;
if (kernel_ip(addr) || !test_tsk_thread_flag(current, TIF_IA32))
kernel_insn_init(insn, addr); else insn_init(insn, addr, 0);
Thank you,
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 3 lines · Masami Hiramatsu
Oops, I've thought existing code is only for kernel. Hmm, I guess this could cause a problem when running 32bit binary on x86-64.
I queued below.
Quoted 1 line
---
Subject: perf, x86: Fix PEBS instruction unwind
From: Peter Zijlstra <[email protected]>
Date: Fri Oct 07 13:36:40 CEST 2011
Masami spotted that we always try to decode the instruction stream as 64bit instructions when running a 64bit kernel, this doesn't work for ia32-compat proglets.
Use TIF_IA32 to detect if we need to use the 32bit instruction decoder.
--- arch/x86/kernel/cpu/perf_event_intel_ds.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/perf_event_intel_ds.c +5 -1
@@ -493,6 +493,7 @@ static int intel_pmu_pebs_fixup_ip(strucunsigned long from = cpuc->lbr_entries[0 ].from;unsigned long old_to, to = cpuc->lbr_entries[0 ].to;unsigned long ip = regs->ip; +int is_64bit =0 ;/* * We don't need to fixup if the PEBS assist is fault like
@@ -544,7 +545,10 @@ static int intel_pmu_pebs_fixup_ip(struc }else kaddr = (void *)to; -kernel_insn_init (&insn, kaddr); +#ifdef CONFIG_X86_64 + is_64bit =kernel_ip (to) || !test_thread_flag (TIF_IA32 ); +#endif +insn_init (&insn, kaddr, is_64bit);insn_get_length (&insn); to += insn.length; }while (to < ip);
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
arch/x86/kernel/cpu/perf_event_intel_ds.c static int intel_pmu_pebs_fixup_ip(struc · line 549
Quoted 3 lines
arch/x86/kernel/cpu/perf_event_intel_ds.c +2
-1-kernel_insn_init (&insn, kaddr); +#ifdef CONFIG_X86_64 + is_64bit =kernel_ip (to) || !test_thread_flag (TIF_IA32 );
Sorry that's wrong. If anything check the code segment (but see other email)
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
In reply to Masami HiramatsuQuoted 1 line
if (
kernel_ip(addr)|| !test_tsk_thread_flag(current,TIF_IA32))
That's not good, a 64bit process can do 32bit syscalls.
The only fully working way is to check the code segment, unfortunately that requires checking the LDT to be fully correct :-(
In 2.4 we had some code for that if you want to copy, but it disappeared later.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Peter Zijlstra
maintainerRe: [PATCH 07/12] perf_events: add LBR software filter support for Intel X86
Quoted 6 lines · Andi Kleen
Quoted 1 line
if (
kernel_ip(addr)|| !test_tsk_thread_flag(current,TIF_IA32))That's not good, a 64bit process can do 32bit syscalls.
The only fully working way is to check the code segment, unfortunately that requires checking the LDT to be fully correct :-(
I can't really be arsed about people doing daft shit like that. The worst that happens is that their PEBS fixup fails. Honestly if you do crazy things like that you deserve far far worse.
PATCH 8/12 perf_events: disable PERF_SAMPLE_BRANCH_* when not supported
In reply to Stephane EranianPERF_SAMPLE_BRANCH_* is disabled for:
- SW events (sw counters, tracepoints)
- HW breakpoints
- ALL but Intel X86 architecture
- AMD64 processors
--- arch/alpha/kernel/perf_event.c | 4 ++++ arch/arm/kernel/perf_event.c | 4 ++++ arch/mips/kernel/perf_event.c | 4 ++++ arch/powerpc/kernel/perf_event.c | 4 ++++ arch/sh/kernel/perf_event.c | 4 ++++ arch/sparc/kernel/perf_event.c | 4 ++++ arch/x86/kernel/cpu/perf_event_amd.c | 3 +++ kernel/events/core.c | 24 ++++++++++++++++++++++++ kernel/events/hw_breakpoint.c | 6 ++++++ 9 files changed, 57 insertions(+), 0 deletions(-)
arch/alpha/kernel/perf_event.c +4 -0
@@ -685,6 +685,10 @@ static int alpha_pmu_event_init(struct perf_event *event) {int err; +/* does not support taken branch sampling */ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +switch (event->attr.type) {case PERF_TYPE_RAW :case PERF_TYPE_HARDWARE :
arch/arm/kernel/perf_event.c +4 -0
@@ -544,6 +544,10 @@ static int armpmu_event_init(struct perf_event *event) {int err =0 ; +/* does not support taken branch sampling */ +if (has_branch_smpl (event)) +return -EOPNOTSUPP ; +switch (event->attr.type) {case PERF_TYPE_RAW :case PERF_TYPE_HARDWARE :
arch/mips/kernel/perf_event.c +4 -0
@@ -370,6 +370,10 @@ static int mipspmu_event_init(struct perf_event *event) {int err =0 ; +/* does not support taken branch sampling */ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +switch (event->attr.type) {case PERF_TYPE_RAW :case PERF_TYPE_HARDWARE :
arch/powerpc/kernel/perf_event.c +4 -0
@@ -1078,6 +1078,10 @@ static int power_pmu_event_init(struct perf_event *event)if (!ppmu)return -ENOENT ; +/* does not support taken branch sampling */ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +switch (event->attr.type) {case PERF_TYPE_HARDWARE : ev = event->attr.config;
arch/sh/kernel/perf_event.c +4 -0
@@ -309,6 +309,10 @@ static int sh_pmu_event_init(struct perf_event *event) {int err; +/* does not support taken branch sampling */ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +switch (event->attr.type) {case PERF_TYPE_RAW :case PERF_TYPE_HW_CACHE :
arch/sparc/kernel/perf_event.c +4 -0
@@ -1105,6 +1105,10 @@ static int sparc_pmu_event_init(struct perf_event *event)if (atomic_read (&nmi_active) <0 )return -ENODEV ; +/* does not support taken branch sampling */ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +switch (attr->type) {case PERF_TYPE_HARDWARE :if (attr->config >= sparc_pmu->max_events)
arch/x86/kernel/cpu/perf_event_amd.c +3 -0
@@ -132,6 +132,9 @@ static int amd_pmu_hw_config(struct perf_event *event)if (ret)return ret; +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +if (event->attr.type !=PERF_TYPE_RAW )return 0 ;
kernel/events/core.c +24 -0
@@ -5104,6 +5104,12 @@ static int perf_swevent_init(struct perf_event *event)if (event->attr.type !=PERF_TYPE_SOFTWARE )return -ENOENT ; +/* +* no branch sampling for software events +*/ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +switch (event_id) {case PERF_COUNT_SW_CPU_CLOCK :case PERF_COUNT_SW_TASK_CLOCK :
@@ -5207,6 +5213,12 @@ static int perf_tp_event_init(struct perf_event *event)if (event->attr.type !=PERF_TYPE_TRACEPOINT )return -ENOENT ; +/* +* no branch sampling for tracepoint events +*/ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; + err =perf_trace_init (event);if (err)return err;
@@ -5430,6 +5442,12 @@ static int cpu_clock_event_init(struct perf_event *event)if (event->attr.config !=PERF_COUNT_SW_CPU_CLOCK )return -ENOENT ; +/* +* no branch sampling for software events +*/ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +perf_swevent_init_hrtimer (event);return 0 ;
@@ -5502,6 +5520,12 @@ static int task_clock_event_init(struct perf_event *event)if (event->attr.config !=PERF_COUNT_SW_TASK_CLOCK )return -ENOENT ; +/* +* no branch sampling for software events +*/ +if (has_branch_stack (event)) +return -EOPNOTSUPP ; +perf_swevent_init_hrtimer (event);return 0 ;
kernel/events/hw_breakpoint.c +6 -0
@@ -581,6 +581,12 @@ static int hw_breakpoint_event_init(struct perf_event *bp)if (bp->attr.type !=PERF_TYPE_BREAKPOINT )return -ENOENT ; +/* +* no branch sampling for breakpoint events +*/ +if (has_branch_stack (bp)) +return -EOPNOTSUPP ; + err =register_perf_hw_breakpoint (bp);if (err)return err;
Signature · 1 line
1.7.4.1
Peter Zijlstra
maintainerRe: [PATCH 08/12] perf_events: disable PERF_SAMPLE_BRANCH_* when not supported
Quoted 5 lines · Stephane Eranian
PERF_SAMPLE_BRANCH_* is disabled for:
- ALL but Intel X86 architecture
- AMD64 processors
It would be good if you could CC all arch/*/kernel/perf_event* maintainers on this, that way they are aware of the existence of the feature. Maybe one of them can actually support some of this.. who knows ;-)
Robert, does this Fam15 thing have anything useful in this regards?
PATCH 9/12 perf_events: add hook to flush branch_stack on context switch
In reply to Stephane EranianWith branch stack sampling, it is possible to filter by priv levels. In system-wide mode, that means it is possible to capture only user level branches. The builtin SW LBR filter needs to disassemble code based on LBR captured addresses. For that, it needs to know the task the addresses are associated with. Because of context switches, the content of the branch stack buffer may contain addresses from different tasks.
We need a hook on context switch to either flush the branch stack or save it. This patch adds a new hook in struct pmu which is called during context switches. The hook is called only when necessary. That is when a system-wide context has, at least, one event which uses PERF_SAMPLE_BRANCH_STACK. The hook is never called for per-thread context.
In this version, the Intel X86 code simply flushes (reset) the LBR on context switches (fill with zeroes). Those zeroed branches are then filtered out by the SW filter.
--- arch/x86/kernel/cpu/perf_event.c | 31 +++++++----- arch/x86/kernel/cpu/perf_event_intel.c | 13 +++++ include/linux/perf_event.h | 7 ++- kernel/events/core.c | 85 ++++++++++++++++++++++++++++++++ 4 files changed, 123 insertions(+), 13 deletions(-)
arch/x86/kernel/cpu/perf_event.c +19 -12
@@ -268,7 +268,7 @@ struct x86_pmu {void (*cpu_starting)(int cpu);void (*cpu_dying)(int cpu);void (*cpu_dead)(int cpu); - +void (*flush_branch_stack)(void );/* * Intel Arch Perfmon v2+ */
@@ -1827,21 +1827,28 @@ static int x86_pmu_event_init(struct perf_event *event)return err; } +static void x86_pmu_flush_branch_stack (void ) +{ +if (x86_pmu.flush_branch_stack) + x86_pmu.flush_branch_stack (); +} +static struct pmu pmu = { - .pmu_enable = x86_pmu_enable, - .pmu_disable = x86_pmu_disable, + .pmu_enable = x86_pmu_enable, + .pmu_disable = x86_pmu_disable, - .event_init = x86_pmu_event_init, + .event_init = x86_pmu_event_init, - .add = x86_pmu_add, - .del = x86_pmu_del, - .start = x86_pmu_start, - .stop = x86_pmu_stop, - .read = x86_pmu_read, + .add = x86_pmu_add, + .del = x86_pmu_del, + .start = x86_pmu_start, + .stop = x86_pmu_stop, + .read = x86_pmu_read, - .start_txn = x86_pmu_start_txn, - .cancel_txn = x86_pmu_cancel_txn, - .commit_txn = x86_pmu_commit_txn, + .start_txn = x86_pmu_start_txn, + .cancel_txn = x86_pmu_cancel_txn, + .commit_txn = x86_pmu_commit_txn, + .flush_branch_stack = x86_pmu_flush_branch_stack, };/*
arch/x86/kernel/cpu/perf_event_intel.c +13 -0
@@ -1440,6 +1440,18 @@ static void intel_pmu_cpu_dying(int cpu)fini_debug_store_on_cpu (cpu); } +static void intel_pmu_flush_branch_stack (void ) +{ +/* +* Intel LBR does not tag entries with the +* PID of the current task, then we need to +* flush it on ctxsw +* For now, we simply reset it +*/ +if (x86_pmu.lbr_nr) +intel_pmu_lbr_reset (); +} +static __initconst const struct x86_pmu intel_pmu = { .name ="Intel" , .handle_irq = intel_pmu_handle_irq,
@@ -1466,6 +1478,7 @@ static __initconst const struct x86_pmu intel_pmu = { .cpu_prepare = intel_pmu_cpu_prepare, .cpu_starting = intel_pmu_cpu_starting, .cpu_dying = intel_pmu_cpu_dying, + .flush_branch_stack = intel_pmu_flush_branch_stack, };static void intel_clovertown_quirks (void )
include/linux/perf_event.h +6 -1
@@ -732,6 +732,9 @@ struct pmu {* for each successful ->add() during the transaction. */ void (*cancel_txn) (struct pmu *pmu);/* optional */ +/* +* flush branch stack on context-switches (needed in cpu-wide mode) */ +void (*flush_branch_stack) (void ); };/**
@@ -960,7 +963,8 @@ struct perf_event_context {u64 parent_gen;u64 generation;int pin_count; -int nr_cgroups;/* cgroup events present */ +int nr_cgroups;/* cgroup evts */ +int nr_branch_stack;/* branch_stack evt */ struct rcu_head rcu_head; };
@@ -1025,6 +1029,7 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr,extern u64 perf_event_read_value (struct perf_event *event,u64 *enabled,u64 *running); +struct perf_sample_data {u64 type;
kernel/events/core.c +85 -0
127 lines not on this page · raw message
Signature · 1 line
1.7.4.1
PATCH 10/12 perf: add code to support PERF_SAMPLE_BRANCH_STACK
In reply to Stephane EranianThis patch adds:
- ability to parse samples with PERF_SAMPLE_BRANCH_STACK
- sort on branches
- build histograms on branches
--- tools/perf/perf.h | 17 ++ tools/perf/util/annotate.c | 2 +- tools/perf/util/event.h | 1 + tools/perf/util/evsel.c | 10 ++ tools/perf/util/hist.c | 97 ++++++++++--- tools/perf/util/hist.h | 6 + tools/perf/util/session.c | 72 +++++++++ tools/perf/util/session.h | 5 + tools/perf/util/sort.c | 348 +++++++++++++++++++++++++++++++++----------- tools/perf/util/sort.h | 5 + tools/perf/util/symbol.h | 13 ++ 11 files changed, 464 insertions(+), 112 deletions(-)
tools/perf/perf.h +17 -0
@@ -170,6 +170,23 @@ struct ip_callchain { u64 ips[0]; }; +struct branch_flags { +u64 mispred :1 ; +u64 predicted :1 ; +u64 reserved :62 ; +}; + +struct branch_entry { +u64 from; +u64 to; +struct branch_flags flags; +}; + +struct branch_stack { +u64 nr; +struct branch_entry entries[0 ]; +}; +extern bool perf_host, perf_guest;#endif
tools/perf/util/annotate.c +1 -1
@@ -62,7 +62,7 @@ int symbol__inc_addr_samples(struct symbol *sym, struct map *map,pr_debug3 ("%s: addr=%#" PRIx64"\n" , __func__, map->unmap_ip (map, addr)); -if (addr >= sym->end) +if (addr >= sym->end || addr < sym->start)return 0 ; offset = addr - sym->start;
tools/perf/util/event.h +1 -0
@@ -80,6 +80,7 @@ struct perf_sample {u32 raw_size;void *raw_data;struct ip_callchain *callchain; +struct branch_stack *branch_stack; };#define BUILD_ID_SIZE 20
tools/perf/util/evsel.c +10 -0
@@ -472,5 +472,15 @@ int perf_event__parse_sample(const union perf_event *event, u64 type, data->raw_data = (void *) pdata; } +if (type &PERF_SAMPLE_BRANCH_STACK ) { +u64 sz; + + data->branch_stack = (struct branch_stack *)array; + array++;/* nr */ + + sz = data->branch_stack->nr *sizeof (struct branch_entry ); + sz /=sizeof (uint64_t ); + array += sz; + }return 0 ; }
tools/perf/util/hist.c +74 -23
130 lines not on this page · raw message
tools/perf/util/hist.h +6 -0
@@ -39,6 +39,7 @@ enum hist_column {HISTC_COMM ,HISTC_PARENT ,HISTC_CPU , +HISTC_MISPREDICT , HISTC_NR_COLS, /* Last entry */ };
@@ -55,6 +56,11 @@ struct hists {struct hist_entry *__hists__add_entry (struct hists *self,struct addr_location *al,struct symbol *parent,u64 period); +struct hist_entry *__hists__add_branch_entry (struct hists *self, +struct addr_location *al, +struct symbol *sym_parent, +struct branch_info * bi, +u64 period);extern int64_t hist_entry__cmp (struct hist_entry *,struct hist_entry *);extern int64_t hist_entry__collapse (struct hist_entry *,struct hist_entry *); int hist_entry__fprintf(struct hist_entry *self, struct hists *hists,
tools/perf/util/session.c +72 -0
90 lines not on this page · raw message
tools/perf/util/session.h +5 -0
@@ -100,6 +100,11 @@ int __perf_session__process_events(struct perf_session *self,int perf_session__process_events (struct perf_session *self,struct perf_event_ops *event_ops); + +struct branch_info *perf_session__resolve_bstack (struct perf_session *self, +struct thread *thread, +struct branch_stack *bs); + int perf_session__resolve_callchain(struct perf_session *self, struct thread *thread, struct ip_callchain *chain,
tools/perf/util/sort.c +260 -88
422 lines not on this page · raw message
tools/perf/util/sort.h +5 -0
@@ -31,11 +31,14 @@ extern const char *parent_pattern;extern const char default_sort_order[];extern int sort__need_collapse;extern int sort__has_parent; +extern bool sort__branch_mode;extern char *field_sep;extern struct sort_entry sort_comm;extern struct sort_entry sort_dso;extern struct sort_entry sort_sym;extern struct sort_entry sort_parent; +extern struct sort_entry sort_lbr_dso; +extern struct sort_entry sort_lbr_sym;extern enum sort_type sort__first_dimension;/**
@@ -70,6 +73,7 @@ struct hist_entry {struct hist_entry *pair;struct rb_root sorted_chain; }; +struct branch_info *branch_info;struct callchain_root callchain[0 ]; };
@@ -80,6 +84,7 @@ enum sort_type {SORT_SYM ,SORT_PARENT ,SORT_CPU , + SORT_MISPREDICTED, };/*
tools/perf/util/symbol.h +13 -0
@@ -5,6 +5,7 @@#include <stdbool.h> #include <stdint.h> #include "map.h" +#include "../perf.h" #include <linux/list.h> #include <linux/rbtree.h> #include <stdio.h>
@@ -115,6 +116,18 @@ struct map_symbol {bool has_children; }; +struct addr_map_symbol { +struct map *map; +struct symbol *sym; +u64 addr; +}; + +struct branch_info { +struct addr_map_symbol from; +struct addr_map_symbol to; +struct branch_flags flags; +}; +struct addr_location {struct thread *thread;struct map *map;
Signature · 1 line
1.7.4.1
Peter Zijlstra
maintainerRe: [PATCH 10/12] perf: add code to support PERF_SAMPLE_BRANCH_STACK
On Thu, 2011-10-06 at 16:49 +0200, Stephane Eranian wrote:
Is a From: Roberto ..
Line missing here?
Quoted 7 lines
This patch adds:
- ability to parse samples withPERF_SAMPLE_BRANCH_STACK
- sort on branches
- build histograms on branchesSigned-off-by Roberto Agostino Vitillo <[email protected]>Signed-off-by Stephane Eranian <[email protected]>
Re: [PATCH 10/12] perf: add code to support PERF_SAMPLE_BRANCH_STACK
Quoted 6 lines · Peter Zijlstra
On Thu, 2011-10-06 at 16:49 +0200, Stephane Eranian wrote:
Is a From: Roberto ..
Line missing here?
Yes, I can add it back if that's the way the system works.
Quoted 8 lines
Quoted 7 lines
This patch adds:
- ability to parse samples withPERF_SAMPLE_BRANCH_STACK
- sort on branches
- build histograms on branchesSigned-off-by Roberto Agostino Vitillo <[email protected]>Signed-off-by Stephane Eranian <[email protected]>
Peter Zijlstra
maintainerRe: [PATCH 10/12] perf: add code to support PERF_SAMPLE_BRANCH_STACK
Quoted 8 lines · Stephane Eranian
Quoted 6 lines · Peter Zijlstra
On Thu, 2011-10-06 at 16:49 +0200, Stephane Eranian wrote:
Is a From: Roberto ..
Line missing here?
Yes, I can add it back if that's the way the system works.
Please, otherwise it might get attributed to you.
PATCH 11/12 perf: add support for sampling taken branch to perf record
In reply to Stephane EranianThis patch adds a new option to enable taken branch stack sampling, i.e., leverage the PERF_SAMPLE_BRANCH_STACK feature of perf_events.
There is a new option to active this mode: -b.
It is possible to pass a set of filters to select the type of branches to sample.
The following filters are available:
- any : any type of branches
- any_call : any function call or system call
- any_ret : any function return or system call return
- any_ind : any indirect branch
- u: only when the branch target is at the user level
- k: only when the branch target is in the kernel
Filters can be combined by passing a comma separated list to the option:
$ perf record -b any_call,u -e cycles:u branchy
--- tools/perf/Documentation/perf-record.txt | 18 +++++++ tools/perf/builtin-record.c | 75 ++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+), 0 deletions(-)
tools/perf/Documentation/perf-record.txt +18 -0
@@ -148,6 +148,24 @@ an empty cgroup (monitor all the time) using, e.g., -G foo,,bar. Cgroups must ha corresponding events, i.e., they always refer to events defined earlier on the command line. +-b:: +--branch-stack:: +Enable taken branch stack sampling. Each sample captures a series of consecutive +taken branches. The number of branches captured with each sample depends on the +underlying hardware, the type of branches of interested and the executed code. +It is possible to filter the types of branches by enabling filters. The +following filters are defined: any (any type of branches), any_call (any function +call or system call), any_ret (any function return or system call return), any_ind +(any indirect branch), u (only when the branch target is at the user level), k (only when +the branch target is in the kernel). At least one of any, any_call, any_ret, any_ind +must be provided. The privilege levels may be ommitted, in which case, the privilege +levels of the associated event is applied to the branch filter. When sampling on multiple +events, branch stack sampling is enabled for all the sampling events. The sampled branch +type is the same for all events. The privilege levels are adjusted based on those of +the associated event unless specified explicitly with this option. Note that taken +branch sampling may not be available on all processors. The various filters must +be specified as a comma separated list: -b any_ret,u,k + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +75 -0
99 lines not on this page · raw message
Signature · 1 line
1.7.4.1
PATCH 12/12 perf: add support for taken branch sampling to perf report
In reply to Stephane EranianThis patch adds support for taken branch sampling, i.e, the PERF_SAMPLE_BRANCH_STACK feature to perf report. In other words, to display histograms based on taken branches rather than executed instructions addresses.
The new option is called -b and it takes no argument. To generate meaningful output, the perf.data must have been obtained using perf record -b xxx ... where xxx is a branch filter option.
The output shows symbols, modules, sorted by 'who branches where' the most often. The percentages reported in the first column refer to the total number of branches captured and not the usual number of samples.
Here is a quick example.
Here branchy is simple test program which looks as follows:
void f2(void)
{}
void f3(void)
{}
void f1(unsigned long n)
{
if (n & 1UL)
f2();
else
f3();
}int main(void)
{
unsigned long i;
for (i=0; i < N; i++)
f1(i);
return 0;
}
Here is the output captured on Nehalem, if we are
only interested in user level function calls.
$ perf record -b any_call,u -e cycles:u branchy
$ perf report -b --sort=symbol
52.34% [.] main [.] f1
24.04% [.] f1 [.] f3
23.60% [.] f1 [.] f2
0.01% [k] _IO_new_file_xsputn [k] _IO_file_overflow
0.01% [k] _IO_vfprintf_internal [k] _IO_new_file_xsputn
0.01% [k] _IO_vfprintf_internal [k] strchrnul
0.01% [k] __printf [k] _IO_vfprintf_internal
0.01% [k] main [k] __printfAbout half (52%) of the call branches captured are from main() -> f1(). The second half (24%+23%) is split in two equal shares between f1() -> f2(), f1() ->f3(). The output is as expected given the code.
It should be noted, that using -b in perf record does not eliminate information in the perf.data file. Consequently, a typical profile can also be obtained by perf report by simply not using its -b option.
--- tools/perf/Documentation/perf-report.txt | 7 ++ tools/perf/builtin-report.c | 93 +++++++++++++++++++++++++++--- 2 files changed, 91 insertions(+), 9 deletions(-)
tools/perf/Documentation/perf-report.txt +7 -0
@@ -134,6 +134,13 @@ OPTIONS CPUs are specified with -: 0-2. Default is to report samples on all CPUs. +-b:: +--branch-stack:: + Use the addresses of sampled taken branches instead of the instruction + address to build the histograms. To generate meaningful output, the + perf.data file must have been obtained using perf record -b xxx where + xxx is a branch filter option. + SEE ALSO -------- linkperf:perf-stat[1]
tools/perf/builtin-report.c +84 -9
128 lines not on this page · raw message
Signature · 1 line
1.7.4.1
Thanks very nice patchkit (at least from the description, haven't read it yet)
Quoted 6 lines
# Overhead Source Symbol Target Symbol # ........ ................ ................18.13% [.] f1 [.] main 18.10% [.] main [.] main 18.01% [.] main [.] f1
The big problem here is no line numbers (something I miss for other analysis too). I hope that can be fixed so that analysis inside functions becomes possible. Something similar to pprofs line mode.
-Andi
Quoted 3 lines · Andi Kleen
Thanks very nice patchkit (at least from the description, haven't read it yet)
Thanks.
Quoted 4 lines
Quoted 4 lines
# Overhead Source Symbol Target Symbol
# ........ ................ ................
Quoted 8 lines
Quoted 3 lines
18.13% [.] f1 [.] main
18.10% [.] main [.] main
18.01% [.] main [.] f1The big problem here is no line numbers (something I miss for other analysis too). I hope that can be fixed so that analysis inside functions becomes possible. Something similar to pprofs line mode.
You get line numbers with annotate only today so this is not specific to this particular patch. But yeah, it would be nice.
Peter Zijlstra
maintainerThis doesn't apply at all, its against some ancient tree. Please re-spin against a current tip/master or so.
Quoted 4 lines · Peter Zijlstra
This doesn't apply at all, its against some ancient tree. Please re-spin against a current tip/master or so.
It is against Linus' 3.1-rc9 commit 976d167.
I will re-spin tomorrow and cc the arch maintainers too.