x86/rapl: Do not load in a guest
Subsystems (3)
Borislav Petkov
maintainerPATCH 1/1 x86/rapl: Do not load in a guest
From: Borislav Petkov <[email protected]>
qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.
--- arch/x86/kernel/cpu/perf_event_intel_rapl.c | 3 +++ drivers/powercap/intel_rapl.c | 3 +++ 2 files changed, 6 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_rapl.c +3 -0
@@ -711,6 +711,9 @@ static int __init rapl_pmu_init(void)struct x86_pmu_quirk *quirk;int i; +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +return 0 ; +/* * check for Intel processor family 6 */
drivers/powercap/intel_rapl.c +3 -0
@@ -1511,6 +1511,9 @@ static int __init rapl_init(void)int ret =0 ;const struct x86_cpu_id *id; +if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +return -ENODEV ; + id =x86_match_cpu (rapl_ids);if (!id) { pr_err("driver does not support CPU family %d model %d\n",
Signature · 1 line
2.3.5
arch/x86/kernel/cpu/perf_event_intel_rapl.c static int __init rapl_pmu_init(void) · line 716
Quoted 4 lines · On Thu, 3 Dec 2015 19:27:02 +0100 Borislav Petkov <[email protected]> wrote:
arch/x86/kernel/cpu/perf_event_intel_rapl.c +3
-0+if (boot_cpu_has (X86_FEATURE_HYPERVISOR )) +return 0 ; +
or use this?
#define cpu_has_hypervisor boot_cpu_has(X86_FEATURE_HYPERVISOR)
Borislav Petkov
maintainerQuoted 2 lines · Jacob Pan
or use this?
#definecpu_has_hypervisorboot_cpu_has(X86_FEATURE_HYPERVISOR)
No, those are going away:
https://lkml.kernel.org/r/[email protected]
Next on my TODO is killing the rest of them.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Jacob Pan
Acked-byQuoted 5 lines · On Thu, 3 Dec 2015 19:42:41 +0100 Borislav Petkov <[email protected]> wrote:
No, those are going away:
https://lkml.kernel.org/r/[email protected]
Next on my TODO is killing the rest of them.
Fair enough.
Quoted 11 lines · Jacob Pan
Quoted 5 lines · On Thu, 3 Dec 2015 19:42:41 +0100 Borislav Petkov <[email protected]> wrote:
No, those are going away:
https://lkml.kernel.org/r/[email protected]
Next on my TODO is killing the rest of them.
Fair enough.
Acked-by Jacob Pan <[email protected]>
OK, so who's suppose to apply this?
Rafael
Borislav Petkov
maintainerQuoted 1 line · Rafael J. Wysocki
OK, so who's suppose to apply this?
It doesn't matter to me. I was planning on routing it through tip next week if you're fine with it or tip guys could ack it too and you could pick it up. Preference?
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 6 lines · Borislav Petkov
Quoted 1 line · Rafael J. Wysocki
OK, so who's suppose to apply this?
It doesn't matter to me. I was planning on routing it through tip next week if you're fine with it or tip guys could ack it too and you could pick it up. Preference?
Nope. It can go in via tip as far as I'm concerned.
Quoted 4 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.
So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?
Isn't there any better method to detect RAPL support?
So in particular in drivers/powercap/intel_rapl.c there's an enumerated list of CPU models, which is used via a x86_match_cpu() call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.
Thanks,
Ingo
Peter Zijlstra
maintainerQuoted 10 lines · Ingo Molnar
Quoted 4 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.
So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?
Quoted 7 lines
Isn't there any better method to detect RAPL support?
So in particular in
drivers/powercap/intel_rapl.cthere's an enumerated list of CPU models, which is used via ax86_match_cpu()call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.
RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.
Also, yuck @ powercap/intel_rapl.c for doing rdmsr_on_cpu() + wrmsr_on_cpu() all over the place.
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Ingo Molnar
Quoted 4 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.
So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?
Isn't there any better method to detect RAPL support?
Quoted 8 lines
Quoted 5 lines
So in particular in
drivers/powercap/intel_rapl.cthere's an enumerated list of CPU models, which is used via ax86_match_cpu()call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.
Hm, bad - writing to MSRs like that is generally dangerous.
So we should at least provide a central 'is RAPL available' call instead of spreading multiple X86_FEATURE_HYPERVISOR checks.
Thanks,
Ingo
Borislav Petkov
maintainer+ Paolo.
Quoted 10 lines · Ingo Molnar
Quoted 10 lines
Quoted 9 lines
So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?
Isn't there any better method to detect RAPL support?
So in particular in
drivers/powercap/intel_rapl.cthere's an enumerated list of CPU models, which is used via ax86_match_cpu()call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.
Quoted 7 lines
Quoted 2 lines
RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.
Hm, bad - writing to MSRs like that is generally dangerous.
So we should at least provide a central 'is RAPL available' call instead of spreading multiple
X86_FEATURE_HYPERVISORchecks.
Well, looks like someone dropped the ball at the CPUID registrar. Other features have more than one CPUID bit allocated to them, this one doesn't have a single one.
And since there's no CPUID bit, I don't see any other way to detect the RAPL presence. Poking at MSRs is a bad idea.
I wonder if we could go and allocate a bit in the kvm-emulated CPUID leafs which says whether RAPL is supported or not.
Then we can go and check for that leaf on baremetal - if it is not there, we do the vendor + fms check and if it is there, we know we're in a guest and whether the guest supports it or not.
Dunno.
On the one hand, it looks like a bit too much to me.
On the other, it could be useful for other future feature checks where we want baremetal and kvm to be synchronized wrt features and a single method to be used by the kernel for checking features presence works both on baremetal and virt.
Just a thought, anyway...
hpa, thoughts?
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 16 lines · Borislav Petkov
+ Paolo.
Quoted 13 lines · Ingo Molnar
Quoted 12 lines
Quoted 9 lines
So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?
Isn't there any better method to detect RAPL support?
So in particular in
drivers/powercap/intel_rapl.cthere's an enumerated list of CPU models, which is used via ax86_match_cpu()call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.
Quoted 6 lines
Quoted 4 lines
Hm, bad - writing to MSRs like that is generally dangerous.
So we should at least provide a central 'is RAPL available' call instead of spreading multiple
X86_FEATURE_HYPERVISORchecks.Well, looks like someone dropped the ball at the CPUID registrar.
Yup, this is an issue with RAPL.
Quoted 5 lines
And since there's no CPUID bit, I don't see any other way to detect the RAPL presence. Poking at MSRs is a bad idea.
I wonder if we could go and allocate a bit in the kvm-emulated CPUID leafs which says whether RAPL is supported or not.
No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.
Paolo
Quoted 17 lines
Then we can go and check for that leaf on baremetal - if it is not there, we do the vendor + fms check and if it is there, we know we're in a guest and whether the guest supports it or not.
Dunno.
On the one hand, it looks like a bit too much to me.
On the other, it could be useful for other future feature checks where we want baremetal and kvm to be synchronized wrt features and a single method to be used by the kernel for checking features presence works both on baremetal and virt.
Just a thought, anyway...
hpa, thoughts?
Borislav Petkov
maintainerQuoted 3 lines · Paolo Bonzini
No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.
Well, we could not do anything, sure, that's an option too. It would only be the annoying error message. Which is
pr_err("no valid rapl domains found in package %d\n", rp->id);I guess we can tone that down as apparently it is not an error to not have valid rapl domains anymore. Maybe kill it altogether:rapl_detect_topology() will propagate the error and the driver won't load...
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Paolo Bonzini
No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.
Well, we could not do anything, sure, that's an option too. It would only be the annoying error message. Which is
Quoted 6 lines
pr_err("no valid rapl domains found in package %d\n", rp->id);I guess we can tone that down as apparently it is not an error to not have valid rapl domains anymore. Maybe kill it altogether:
rapl_detect_topology()will propagate the error and the driver won't load...
So given than nothing really tells us in a clear way whether RAPL is supported or not on that kernel, it might be better to just centralize the 'detect RAPL' function, and print "x86/rapl: Feature detected" on bootup. That function can also install a synthetic CPUID bit, which all other code could use in a clean fashion.
Since it will be an __init function, there's not much of an overhead argument against it.
This way it becomes part of the CPUID infrastructure - and eventually it might even grow a real CPUID bit in future CPU models.
and we'll have a lot less RAPL detection muck all around. Win-win.
Thanks,
Ingo
Quoted 11 lines · On Fri, 4 Dec 2015 12:53:35 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Paolo Bonzini
No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.
Well, we could not do anything, sure, that's an option too. It would only be the annoying error message. Which is
Quoted 8 lines
Quoted 7 lines
pr_err("no valid rapl domains found in package %d\n",
rp->id);I guess we can tone that down as apparently it is not an error to not have valid rapl domains anymore. Maybe kill it altogether:
rapl_detect_topology()will propagate the error and the driver won't load...
Since RAPL is not architectural, consistency of hw support needs lots of improvement at the least. This error message is valid in other than VM. Domain detection error already propagated.
Quoted 9 lines
So given than nothing really tells us in a clear way whether RAPL is supported or not on that kernel, it might be better to just centralize the 'detect RAPL' function, and print "x86/rapl: Feature detected" on bootup. That function can also install a synthetic CPUID bit, which all other code could use in a clean fashion.
Since it will be an
__initfunction, there's not much of an overhead argument against it.
This is good for the first level RAPL detection. The only way is to base detection on known CPU models.
But I still think hypervisor check is sufficient. I don't there will ever be a use case for VM to control platform level power. A disaster for sure.
Quoted 5 lines
This way it becomes part of the CPUID infrastructure - and eventually it might even grow a real CPUID bit in future CPU models.
and we'll have a lot less RAPL detection muck all around. Win-win.
True, RAPL is not architectural today but it is supported by all Intel CPUs since SNB.
Quoted 3 lines
Thanks,
Ingo
Quoted 3 lines · Jacob Pan
But I still think hypervisor check is sufficient. I don't there will ever be a use case for VM to control platform level power. A disaster for sure.
There isn't just usual VMs. There are partitioning hypervisors for example.
Paolo
Borislav Petkov
maintainerQuoted 2 lines · Jacob Pan
Since RAPL is not architectural, consistency of hw support needs lots of improvement at the least.
It should have received a CPUID bit, no matter if it is architectural or not.
Quoted 2 lines
This error message is valid in other than VM. Domain detection error already propagated.
Huh, didn't I say that?
Quoted 2 lines
This is good for the first level RAPL detection. The only way is to base detection on known CPU models.
...and yet Hannes sees it on a minnowboard. CCed.
The CPU model-based detection is ugly and does not always work, as you see.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 7 lines · On Fri, 4 Dec 2015 19:04:26 +0100 Borislav Petkov <[email protected]> wrote:
Quoted 2 lines
This is good for the first level RAPL detection. The only way is to base detection on known CPU models.
...and yet Hannes sees it on a minnowboard. CCed.
The CPU model-based detection is ugly and does not always work, as you see.
CPU model detection is the first level checking. The error is about no valid domains (e.g. counters not working). So the error on minnowboard board could be a real problem if you expect to use RAPL.
Jacob
Borislav Petkov
maintainerQuoted 1 line · Jacob Pan
CPU model detection is the first level checking.
And in the case of RAPL, the only checking you can do. This is why it should've had a CPUID bit.
Quoted 3 lines
The error is about no valid domains (e.g. counters not working). So the error on minnowboard board could be a real problem if you expect to use RAPL.
Right, and if you need to disable it there, you would need to add a quirk table looking at DMI strings or so. A CPUID bit might've been a bit better if BIOS update would clear it on those boards. Then, sw won't even try to load there.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 6 lines · On Fri, 4 Dec 2015 19:28:07 +0100 Borislav Petkov <[email protected]> wrote:
Quoted 1 line · Jacob Pan
CPU model detection is the first level checking.
And in the case of RAPL, the only checking you can do. This is why it should've had a CPUID bit.
I am 200% with you, all I can say is we are working on it. Look at the pain in the driver for dealing with various quirks.
Quoted 8 lines
Quoted 3 lines
The error is about no valid domains (e.g. counters not working). So the error on minnowboard board could be a real problem if you expect to use RAPL.
Right, and if you need to disable it there, you would need to add a quirk table looking at DMI strings or so. A CPUID bit might've been a bit better if BIOS update would clear it on those boards. Then, sw won't even try to load there.
who is gonna collect all the DMI strings? I don't think this is scalable.
Borislav Petkov
maintainerQuoted 2 lines · Jacob Pan
who is gonna collect all the DMI strings? I don't think this is scalable.
Let's hope only a small number of platforms is affected.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 2 lines · On Fri, 4 Dec 2015 09:22:56 +0100 Peter Zijlstra <[email protected]> wrote:
Also, yuck @
powercap/intel_rapl.cfor doingrdmsr_on_cpu()+wrmsr_on_cpu()all over the place.
Can you please be more specific? is the concern related to the overhead of IPI? I am doing these calls based on MSR CPU scope and consider the fact that access is less frequent.
Thanks,
Jacob
Peter Zijlstra
maintainerQuoted 8 lines · Jacob Pan
Quoted 2 lines · On Fri, 4 Dec 2015 09:22:56 +0100 Peter Zijlstra <[email protected]> wrote:
Also, yuck @
powercap/intel_rapl.cfor doingrdmsr_on_cpu()+wrmsr_on_cpu()all over the place.Can you please be more specific? is the concern related to the overhead of IPI? I am doing these calls based on MSR CPU scope and consider the fact that access is less frequent.
Yeah, its just offensive to do an IPI to read a value, then twiddle a few bits on the value and then IPI again to store the value.
I know its low freq, and that MSR access is slow, but *groan*.
H. Peter Anvin
reviewerQuoted 6 lines · Peter Zijlstra
Quoted 5 lines · Jacob Pan
Quoted 2 lines · On Fri, 4 Dec 2015 09:22:56 +0100 Peter Zijlstra <[email protected]> wrote:
Also, yuck @
powercap/intel_rapl.cfor doingrdmsr_on_cpu()+wrmsr_on_cpu()all over the place.
Quoted 8 lines
Quoted 3 lines
Can you please be more specific? is the concern related to the overhead of IPI? I am doing these calls based on MSR CPU scope and consider the fact that access is less frequent.
Yeah, its just offensive to do an IPI to read a value, then twiddle a few bits on the value and then IPI again to store the value.
I know its low freq, and that MSR access is slow, but *groan*.
Yes, for that it would be better to invoke a common routine to do all the accesses on the target CPU. MSR accesses may be slow, but IPIs are way slower.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.