x86, perf: Tweak broken BIOS rules during check_hw_exists
Subsystems (2)
PATCH 1/1 x86, perf: Tweak broken BIOS rules during check_hw_exists
I stumbled upon an AMD box that had the BIOS using a hardware counter. Instead of printing out a warning and continuing, it failed and blocked further perf counter usage.
Looking through the history, I found commit a5ebe0ba3dff had tweaked the rules for a xen guest on an almost identical box and now changed the behaviour.
Unfortunately the rules were tweaked incorrectly and will always lead to msr failures even though the msrs are completely fine.
What happens now is in arch/x86/kernel/cpu/perf_event.c::check_hw_exists:
<snip>
for (i = 0; i < x86_pmu.num_counters; i++) {
reg = x86_pmu_config_addr(i);
ret = rdmsrl_safe(reg, &val);
if (ret)
goto msr_fail;
if (val & ARCH_PERFMON_EVENTSEL_ENABLE) {
bios_fail = 1;
val_fail = val;
reg_fail = reg;
}
}<snip>
/*
* Read the current value, change it and read it back to see if it
* matches, this is needed to detect certain hardware emulators
* (qemu/kvm) that don't trap on the MSR access and always return 0s.
*/
reg = x86_pmu_event_addr(0);
^^^^if the first perf counter is enabled, then this routine will always fail because the counter is running. :-(
if (rdmsrl_safe(reg, &val))
goto msr_fail;
val ^= 0xffffUL;
ret = wrmsrl_safe(reg, val);
ret |= rdmsrl_safe(reg, &val_new);
if (ret || val != val_new)
goto msr_fail;The above bios_fail used to be a 'goto' which is why it worked in the past.
Further, most vendors have migrated to using fixed counters to hide their evilness hence this problem rarely shows up now days except on a few old boxes.
I fixed my problem and kept the spirit of the original Xen fix, by recording a safe non-enable register to be used safely for the reading/writing check. Because it is not enabled, this passes on bare metal boxes (like metal), but should continue to throw an msr_fail on Xen guests because the register isn't emulated yet.
Now I get a proper bios_fail error message and Xen should still see their msr_fail message (untested).
--- arch/x86/kernel/cpu/perf_event.c | 16 +++++++++++++++- 1 files changed, 15 insertions(+), 1 deletions(-)
arch/x86/kernel/cpu/perf_event.c +15 -1
@@ -186,6 +186,7 @@ static bool check_hw_exists(void)u64 val, val_fail, val_new= ~0 ;int i, reg, reg_fail, ret =0 ;int bios_fail =0 ; +int reg_safe =-1 ;/* * Check to see if the BIOS enabled any of the counters, if so
@@ -200,6 +201,8 @@ static bool check_hw_exists(void) bios_fail =1 ; val_fail = val; reg_fail = reg; + }else { + reg_safe = i; } }
@@ -218,11 +221,22 @@ static bool check_hw_exists(void) }/* +* If all the counters are enabled, the below test will always +* fail. The tools will also become useless in this scenario. +* Just fail and disable the hardware counters. +*/ + +if (reg_safe ==-1 ) { + reg = reg_safe; + goto msr_fail; + } + +/* * Read the current value, change it and read it back to see if it * matches, this is needed to detect certain hardware emulators * (qemu/kvm) that don't trap on the MSR access and always return 0s. */ - reg =x86_pmu_event_addr (0 ); + reg =x86_pmu_event_addr (reg_safe);if (rdmsrl_safe (reg, &val)) goto msr_fail; val ^=0xffffUL ;
Signature · 1 line
1.7.1
Peter Zijlstra
maintainerQuoted 3 lines · Don Zickus
I stumbled upon an AMD box that had the BIOS using a hardware counter. Instead of printing out a warning and continuing, it failed and blocked further perf counter usage.
Hehe, which was the original behaviour iirc.
Quoted 46 lines
Looking through the history, I found commit
a5ebe0ba3dffhad tweaked the rules for a xen guest on an almost identical box and now changed the behaviour.Unfortunately the rules were tweaked incorrectly and will always lead to msr failures even though the msrs are completely fine.
What happens now is in arch/x86/kernel/cpu/perf_event.c::check_hw_exists:
<snip>
for (i = 0; i < x86_pmu.num_counters; i++) { reg = x86_pmu_config_addr(i); ret = rdmsrl_safe(reg, &val); if (ret) goto msr_fail; if (val & ARCH_PERFMON_EVENTSEL_ENABLE) { bios_fail = 1; val_fail = val; reg_fail = reg; } }<snip>
/* * Read the current value, change it and read it back to see if it * matches, this is needed to detect certain hardware emulators * (qemu/kvm) that don't trap on the MSR access and always return 0s. */ reg = x86_pmu_event_addr(0); ^^^^if the first perf counter is enabled, then this routine will always fail because the counter is running. :-(
if (rdmsrl_safe(reg, &val)) goto msr_fail; val ^= 0xffffUL; ret = wrmsrl_safe(reg, val); ret |= rdmsrl_safe(reg, &val_new); if (ret || val != val_new) goto msr_fail;The above
bios_failused to be a 'goto' which is why it worked in the past.Further, most vendors have migrated to using fixed counters to hide their evilness hence this problem rarely shows up now days except on a few old boxes.
Quoted 8 lines
I fixed my problem and kept the spirit of the original Xen fix, by recording a safe non-enable register to be used safely for the reading/writing check. Because it is not enabled, this passes on bare metal boxes (like metal), but should continue to throw an
msr_failon Xen guests because the register isn't emulated yet.Now I get a proper
bios_failerror message and Xen should still see theirmsr_failmessage (untested).
Thanks!
Quoted 56 lines · Don Zickus
I stumbled upon an AMD box that had the BIOS using a hardware counter. Instead of printing out a warning and continuing, it failed and blocked further perf counter usage.
Looking through the history, I found commit
a5ebe0ba3dffhad tweaked the rules for a xen guest on an almost identical box and now changed the behaviour.Unfortunately the rules were tweaked incorrectly and will always lead to msr failures even though the msrs are completely fine.
What happens now is in arch/x86/kernel/cpu/perf_event.c::check_hw_exists:
<snip>
for (i = 0; i < x86_pmu.num_counters; i++) { reg = x86_pmu_config_addr(i); ret = rdmsrl_safe(reg, &val); if (ret) goto msr_fail; if (val & ARCH_PERFMON_EVENTSEL_ENABLE) { bios_fail = 1; val_fail = val; reg_fail = reg; } }<snip>
/* * Read the current value, change it and read it back to see if it * matches, this is needed to detect certain hardware emulators * (qemu/kvm) that don't trap on the MSR access and always return 0s. */ reg = x86_pmu_event_addr(0); ^^^^if the first perf counter is enabled, then this routine will always fail because the counter is running. :-(
if (rdmsrl_safe(reg, &val)) goto msr_fail; val ^= 0xffffUL; ret = wrmsrl_safe(reg, val); ret |= rdmsrl_safe(reg, &val_new); if (ret || val != val_new) goto msr_fail;The above
bios_failused to be a 'goto' which is why it worked in the past.Further, most vendors have migrated to using fixed counters to hide their evilness hence this problem rarely shows up now days except on a few old boxes.
I fixed my problem and kept the spirit of the original Xen fix, by recording a safe non-enable register to be used safely for the reading/writing check. Because it is not enabled, this passes on bare metal boxes (like metal), but should continue to throw an
msr_failon Xen guests because the register isn't emulated yet.
Quoted 4 lines
Now I get a proper
bios_failerror message and Xen should still see theirmsr_failmessage (untested).Signed-off-by Don Zickus <[email protected]>
Right -- so what was actually broken was the "does this register work" check, which needs a non-enabled register.
Would it make sense to add a comment somewhere in the code saying that you need a disabled event counter for the MSR check to work properly? It's sort of implied but it's not explicit.
Other than that, this looks good to me. I'm not positive I have access to the box I needed this for anymore -- I'll take a look for it next week.
In the mean time:
Quoted 50 lines
--- arch/x86/kernel/cpu/perf_event.c | 16 +++++++++++++++- 1 files changed, 15 insertions(+), 1 deletions(-)arch/x86/kernel/cpu/perf_event.c +15
-1@@ -186,6 +186,7 @@ static bool check_hw_exists(void)u64 val, val_fail, val_new= ~0 ;int i, reg, reg_fail, ret =0 ;int bios_fail =0 ; +int reg_safe =-1 ;/* * Check to see if the BIOS enabled any of the counters, if so @@ -200,6 +201,8 @@ static bool check_hw_exists(void) bios_fail =1 ; val_fail = val; reg_fail = reg; + }else { + reg_safe = i; } }@@ -218,11 +221,22 @@ static bool check_hw_exists(void) }/* +* If all the counters are enabled, the below test will always +* fail. The tools will also become useless in this scenario. +* Just fail and disable the hardware counters. +*/ + +if (reg_safe ==-1 ) { + reg = reg_safe; + goto msr_fail; + } + +/* * Read the current value, change it and read it back to see if it * matches, this is needed to detect certain hardware emulators * (qemu/kvm) that don't trap on the MSR access and always return 0s. */ - reg =x86_pmu_event_addr (0 ); + reg =x86_pmu_event_addr (reg_safe);if (rdmsrl_safe (reg, &val)) goto msr_fail; val ^=0xffffUL ;
George Dunlap
Tested-byQuoted 69 lines · George Dunlap
Quoted 56 lines · Don Zickus
I stumbled upon an AMD box that had the BIOS using a hardware counter. Instead of printing out a warning and continuing, it failed and blocked further perf counter usage.
Looking through the history, I found commit
a5ebe0ba3dffhad tweaked the rules for a xen guest on an almost identical box and now changed the behaviour.Unfortunately the rules were tweaked incorrectly and will always lead to msr failures even though the msrs are completely fine.
What happens now is in arch/x86/kernel/cpu/perf_event.c::check_hw_exists:
<snip>
for (i = 0; i < x86_pmu.num_counters; i++) { reg = x86_pmu_config_addr(i); ret = rdmsrl_safe(reg, &val); if (ret) goto msr_fail; if (val & ARCH_PERFMON_EVENTSEL_ENABLE) { bios_fail = 1; val_fail = val; reg_fail = reg; } }<snip>
/* * Read the current value, change it and read it back to see if it * matches, this is needed to detect certain hardware emulators * (qemu/kvm) that don't trap on the MSR access and always return 0s. */ reg = x86_pmu_event_addr(0); ^^^^if the first perf counter is enabled, then this routine will always fail because the counter is running. :-(
if (rdmsrl_safe(reg, &val)) goto msr_fail; val ^= 0xffffUL; ret = wrmsrl_safe(reg, val); ret |= rdmsrl_safe(reg, &val_new); if (ret || val != val_new) goto msr_fail;The above
bios_failused to be a 'goto' which is why it worked in the past.Further, most vendors have migrated to using fixed counters to hide their evilness hence this problem rarely shows up now days except on a few old boxes.
I fixed my problem and kept the spirit of the original Xen fix, by recording a safe non-enable register to be used safely for the reading/writing check. Because it is not enabled, this passes on bare metal boxes (like metal), but should continue to throw an
msr_failon Xen guests because the register isn't emulated yet.Quoted 4 lines
Now I get a proper
bios_failerror message and Xen should still see theirmsr_failmessage (untested).Signed-off-by Don Zickus <[email protected]>Right -- so what was actually broken was the "does this register work" check, which needs a non-enabled register.
Would it make sense to add a comment somewhere in the code saying that you need a disabled event counter for the MSR check to work properly? It's sort of implied but it's not explicit.
Quoted 6 lines
Other than that, this looks good to me. I'm not positive I have access to the box I needed this for anymore -- I'll take a look for it next week.
In the mean time:
Acked-by George Dunlap <[email protected]>
I managed to track down the machine that had the problem and verify that things still work for me after this patch. So now you can add:
Thanks,
-George
tip-bot for Don Zickus
merge bot[tip:perf/core] perf/x86: Tweak broken BIOS rules during check_hw_exists()
In reply to Don Zickusperf/x86: Tweak broken BIOS rules during check_hw_exists()
Commit-ID: 68ab747604da98f0a0414f197f346ac22888fcee Gitweb: http://git.kernel.org/tip/68ab747604da98f0a0414f197f346ac22888fcee Author: Don Zickus <[email protected]> AuthorDate: Mon, 18 May 2015 15:16:48 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 27 May 2015 09:16:20 +0200
perf/x86: Tweak broken BIOS rules during check_hw_exists()
I stumbled upon an AMD box that had the BIOS using a hardware performance counter. Instead of printing out a warning and continuing, it failed and blocked further perf counter usage.
Looking through the history, I found this commit:
a5ebe0ba3dff ("perf/x86: Check all MSRs before passing hw check")
which tweaked the rules for a Xen guest on an almost identical box and now changed the behaviour.
Unfortunately the rules were tweaked incorrectly and will always lead to MSR failures even though the MSRs are completely fine.
What happens now is in arch/x86/kernel/cpu/perf_event.c::check_hw_exists():
<snip>
for (i = 0; i < x86_pmu.num_counters; i++) {
reg = x86_pmu_config_addr(i);
ret = rdmsrl_safe(reg, &val);
if (ret)
goto msr_fail;
if (val & ARCH_PERFMON_EVENTSEL_ENABLE) {
bios_fail = 1;
val_fail = val;
reg_fail = reg;
}
}<snip>
/*
* Read the current value, change it and read it back to see if it
* matches, this is needed to detect certain hardware emulators
* (qemu/kvm) that don't trap on the MSR access and always return 0s.
*/
reg = x86_pmu_event_addr(0);
^^^^if the first perf counter is enabled, then this routine will always fail because the counter is running. :-(
if (rdmsrl_safe(reg, &val))
goto msr_fail;
val ^= 0xffffUL;
ret = wrmsrl_safe(reg, val);
ret |= rdmsrl_safe(reg, &val_new);
if (ret || val != val_new)
goto msr_fail;The above bios_fail used to be a 'goto' which is why it worked in the past.
Further, most vendors have migrated to using fixed counters to hide their evilness hence this problem rarely shows up now days except on a few old boxes.
I fixed my problem and kept the spirit of the original Xen fix, by recording a safe non-enable register to be used safely for the reading/writing check. Because it is not enabled, this passes on bare metal boxes (like metal), but should continue to throw an msr_fail on Xen guests because the register isn't emulated yet.
Now I get a proper bios_fail error message and Xen should still see their msr_fail message (untested).
--- arch/x86/kernel/cpu/perf_event.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/perf_event.c +15 -1
@@ -190,6 +190,7 @@ static bool check_hw_exists(void)u64 val, val_fail, val_new= ~0 ;int i, reg, reg_fail, ret =0 ;int bios_fail =0 ; +int reg_safe =-1 ;/* * Check to see if the BIOS enabled any of the counters, if so
@@ -204,6 +205,8 @@ static bool check_hw_exists(void) bios_fail =1 ; val_fail = val; reg_fail = reg; + }else { + reg_safe = i; } }
@@ -222,11 +225,22 @@ static bool check_hw_exists(void) }/* +* If all the counters are enabled, the below test will always +* fail. The tools will also become useless in this scenario. +* Just fail and disable the hardware counters. +*/ + +if (reg_safe ==-1 ) { + reg = reg_safe; + goto msr_fail; + } + +/* * Read the current value, change it and read it back to see if it * matches, this is needed to detect certain hardware emulators * (qemu/kvm) that don't trap on the MSR access and always return 0s. */ - reg =x86_pmu_event_addr (0 ); + reg =x86_pmu_event_addr (reg_safe);if (rdmsrl_safe (reg, &val)) goto msr_fail; val ^=0xffffUL ;