Skip to contentSkip to conversation

x86/rapl: Do not load in a guest

PATCH In review Borislav Petkov 7 participants 24 messages Dec 3, 2015 – Dec 4, 2015

Subsystems (3)
lore
Share & tools
lore

Borislav Petkov

author maintainer
#0 lore raw

PATCH 1/1 x86/rapl: Do not load in a guest

From: Borislav Petkov <[email protected]>

qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jacob Pan <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/x86/kernel/cpu/perf_event_intel_rapl.c | 3 +++
 drivers/powercap/intel_rapl.c               | 3 +++
 2 files changed, 6 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_rapl.c +3 -0
index ed446bdcbf31..bc60bc1118b4 100644
@@ -711,6 +711,9 @@ static int __init rapl_pmu_init(void)
 	struct x86_pmu_quirk *quirk;
 	int i;
 
+	if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		return 0;
+
 	/*
 	 * check for Intel processor family 6
 	 */
drivers/powercap/intel_rapl.c +3 -0
index cc97f0869791..297a9b5074e2 100644
@@ -1511,6 +1511,9 @@ static int __init rapl_init(void)
 	int ret = 0;
 	const struct x86_cpu_id *id;
 
+	if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		return -ENODEV;
+
 	id = x86_match_cpu(rapl_ids);
 	if (!id) {
 		pr_err("driver does not support CPU family %d model %d\n",
Signature · 1 line
2.3.5
#1 lore raw

arch/x86/kernel/cpu/perf_event_intel_rapl.c static int __init rapl_pmu_init(void) · line 716

Quoted 4 lines · On Thu, 3 Dec 2015 19:27:02 +0100 Borislav Petkov <[email protected]> wrote:
arch/x86/kernel/cpu/perf_event_intel_rapl.c +3 -0
+	if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		return 0;
+

or use this?
#define cpu_has_hypervisor boot_cpu_has(X86_FEATURE_HYPERVISOR)

author maintainer
#2 lore raw
Quoted 2 lines · Jacob Pan

or use this?
#define cpu_has_hypervisor boot_cpu_has(X86_FEATURE_HYPERVISOR)

No, those are going away:

https://lkml.kernel.org/r/[email protected]

Next on my TODO is killing the rest of them.

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
author maintainer
#5 lore raw
Quoted 1 line · Rafael J. Wysocki

OK, so who's suppose to apply this?

It doesn't matter to me. I was planning on routing it through tip next week if you're fine with it or tip guys could ack it too and you could pick it up. Preference?

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#6 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Rafael J. Wysocki

OK, so who's suppose to apply this?

It doesn't matter to me. I was planning on routing it through tip next week if you're fine with it or tip guys could ack it too and you could pick it up. Preference?

Nope. It can go in via tip as far as I'm concerned.

#7 lore raw
In reply to Borislav Petkov
Quoted 4 lines · Borislav Petkov

From: Borislav Petkov <[email protected]>

qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.

So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?

Isn't there any better method to detect RAPL support?

So in particular in drivers/powercap/intel_rapl.c there's an enumerated list of CPU models, which is used via a x86_match_cpu() call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.

Thanks,

Ingo

maintainer
#8 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 4 lines · Borislav Petkov

From: Borislav Petkov <[email protected]>

qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.

So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?

Quoted 7 lines

Isn't there any better method to detect RAPL support?

So in particular in drivers/powercap/intel_rapl.c there's an enumerated list of CPU models, which is used via a x86_match_cpu() call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.

RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.

Also, yuck @ powercap/intel_rapl.c for doing rdmsr_on_cpu() + wrmsr_on_cpu() all over the place.

#9 lore raw
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Ingo Molnar
Quoted 4 lines · Borislav Petkov

From: Borislav Petkov <[email protected]>

qemu/kvm doesn't support RAPL and RAPL doesn't have a CPUID feature bit so check whether we're in a guest instead.

So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?

Isn't there any better method to detect RAPL support?

Quoted 8 lines
Quoted 5 lines

So in particular in drivers/powercap/intel_rapl.c there's an enumerated list of CPU models, which is used via a x86_match_cpu() call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.

RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.

Hm, bad - writing to MSRs like that is generally dangerous.

So we should at least provide a central 'is RAPL available' call instead of spreading multiple X86_FEATURE_HYPERVISOR checks.

Thanks,

Ingo

author maintainer
#10 lore raw

+ Paolo.

Quoted 10 lines · Ingo Molnar
Quoted 10 lines
Quoted 9 lines

So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?

Isn't there any better method to detect RAPL support?

So in particular in drivers/powercap/intel_rapl.c there's an enumerated list of CPU models, which is used via a x86_match_cpu() call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.

Quoted 7 lines
Quoted 2 lines

RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.

Hm, bad - writing to MSRs like that is generally dangerous.

So we should at least provide a central 'is RAPL available' call instead of spreading multiple X86_FEATURE_HYPERVISOR checks.

Well, looks like someone dropped the ball at the CPUID registrar. Other features have more than one CPUID bit allocated to them, this one doesn't have a single one.

And since there's no CPUID bit, I don't see any other way to detect the RAPL presence. Poking at MSRs is a bad idea.

I wonder if we could go and allocate a bit in the kvm-emulated CPUID leafs which says whether RAPL is supported or not.

Then we can go and check for that leaf on baremetal - if it is not there, we do the vendor + fms check and if it is there, we know we're in a guest and whether the guest supports it or not.

Dunno.

On the one hand, it looks like a bit too much to me.

On the other, it could be useful for other future feature checks where we want baremetal and kvm to be synchronized wrt features and a single method to be used by the kernel for checking features presence works both on baremetal and virt.

Just a thought, anyway...

hpa, thoughts?

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#11 lore raw
Quoted 16 lines · Borislav Petkov

+ Paolo.

Quoted 13 lines · Ingo Molnar
Quoted 12 lines
Quoted 9 lines

So when a hypervisor starts supporting RAPL we'll disable the driver erroneously?

Isn't there any better method to detect RAPL support?

So in particular in drivers/powercap/intel_rapl.c there's an enumerated list of CPU models, which is used via a x86_match_cpu() call. That's still not ideal (it does not work on hypervisors for example), but even better would be to detect RAPL support in some other fashion, that does not rely on us statically enumerating CPU models that support it.

RAPL isn't enumerated, the best we could do is attempt to write to one of the writable MSRs and see if that 'works'.

Quoted 6 lines
Quoted 4 lines

Hm, bad - writing to MSRs like that is generally dangerous.

So we should at least provide a central 'is RAPL available' call instead of spreading multiple X86_FEATURE_HYPERVISOR checks.

Well, looks like someone dropped the ball at the CPUID registrar.

Yup, this is an issue with RAPL.

Quoted 5 lines

And since there's no CPUID bit, I don't see any other way to detect the RAPL presence. Poking at MSRs is a bad idea.

I wonder if we could go and allocate a bit in the kvm-emulated CPUID leafs which says whether RAPL is supported or not.

No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.

Paolo

Quoted 17 lines

Then we can go and check for that leaf on baremetal - if it is not there, we do the vendor + fms check and if it is there, we know we're in a guest and whether the guest supports it or not.

Dunno.

On the one hand, it looks like a bit too much to me.

On the other, it could be useful for other future feature checks where we want baremetal and kvm to be synchronized wrt features and a single method to be used by the kernel for checking features presence works both on baremetal and virt.

Just a thought, anyway...

hpa, thoughts?

author maintainer
#12 lore raw
Quoted 3 lines · Paolo Bonzini

No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.

Well, we could not do anything, sure, that's an option too. It would only be the annoying error message. Which is

	pr_err("no valid rapl domains found in package %d\n", rp->id);

I guess we can tone that down as apparently it is not an error to not have valid rapl domains anymore. Maybe kill it altogether:
rapl_detect_topology() will propagate the error and the driver won't load...

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#13 lore raw
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Paolo Bonzini

No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.

Well, we could not do anything, sure, that's an option too. It would only be the annoying error message. Which is

Quoted 6 lines
	pr_err("no valid rapl domains found in package %d\n", rp->id);

I guess we can tone that down as apparently it is not an error to not have valid rapl domains anymore. Maybe kill it altogether:
rapl_detect_topology() will propagate the error and the driver won't load...

So given than nothing really tells us in a clear way whether RAPL is supported or not on that kernel, it might be better to just centralize the 'detect RAPL' function, and print "x86/rapl: Feature detected" on bootup. That function can also install a synthetic CPUID bit, which all other code could use in a clean fashion.

Since it will be an __init function, there's not much of an overhead argument against it.

This way it becomes part of the CPUID infrastructure - and eventually it might even grow a real CPUID bit in future CPU models.

and we'll have a lot less RAPL detection muck all around. Win-win.

Thanks,

Ingo

#14 lore raw
Quoted 11 lines · On Fri, 4 Dec 2015 12:53:35 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Borislav Petkov
Quoted 3 lines · Paolo Bonzini

No, please don't. Why do you need a wrmsr instead of a rdmsr? If there's no RAPL domains, the device doesn't load. On hypervisors, reading random MSRs is generally safe.

Well, we could not do anything, sure, that's an option too. It would only be the annoying error message. Which is

Quoted 8 lines
Quoted 7 lines

pr_err("no valid rapl domains found in package %d\n",
rp->id);

I guess we can tone that down as apparently it is not an error to not have valid rapl domains anymore. Maybe kill it altogether:
rapl_detect_topology() will propagate the error and the driver won't load...

Since RAPL is not architectural, consistency of hw support needs lots of improvement at the least. This error message is valid in other than VM. Domain detection error already propagated.

Quoted 9 lines

So given than nothing really tells us in a clear way whether RAPL is supported or not on that kernel, it might be better to just centralize the 'detect RAPL' function, and print "x86/rapl: Feature detected" on bootup. That function can also install a synthetic CPUID bit, which all other code could use in a clean fashion.

Since it will be an __init function, there's not much of an overhead argument against it.

This is good for the first level RAPL detection. The only way is to base detection on known CPU models.

But I still think hypervisor check is sufficient. I don't there will ever be a use case for VM to control platform level power. A disaster for sure.

Quoted 5 lines

This way it becomes part of the CPUID infrastructure - and eventually it might even grow a real CPUID bit in future CPU models.

and we'll have a lot less RAPL detection muck all around. Win-win.

True, RAPL is not architectural today but it is supported by all Intel CPUs since SNB.

Quoted 3 lines

Thanks,

Ingo

#15 lore raw
Quoted 3 lines · Jacob Pan

But I still think hypervisor check is sufficient. I don't there will ever be a use case for VM to control platform level power. A disaster for sure.

There isn't just usual VMs. There are partitioning hypervisors for example.

Paolo

author maintainer
#16 lore raw
In reply to Jacob Pan
Quoted 2 lines · Jacob Pan

Since RAPL is not architectural, consistency of hw support needs lots of improvement at the least.

It should have received a CPUID bit, no matter if it is architectural or not.

Quoted 2 lines

This error message is valid in other than VM. Domain detection error already propagated.

Huh, didn't I say that?

Quoted 2 lines

This is good for the first level RAPL detection. The only way is to base detection on known CPU models.

...and yet Hannes sees it on a minnowboard. CCed.

The CPU model-based detection is ugly and does not always work, as you see.

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#17 lore raw
Quoted 7 lines · On Fri, 4 Dec 2015 19:04:26 +0100 Borislav Petkov <[email protected]> wrote:
Quoted 2 lines

This is good for the first level RAPL detection. The only way is to base detection on known CPU models.

...and yet Hannes sees it on a minnowboard. CCed.

The CPU model-based detection is ugly and does not always work, as you see.

CPU model detection is the first level checking. The error is about no valid domains (e.g. counters not working). So the error on minnowboard board could be a real problem if you expect to use RAPL.

Jacob

author maintainer
#18 lore raw
Quoted 1 line · Jacob Pan

CPU model detection is the first level checking.

And in the case of RAPL, the only checking you can do. This is why it should've had a CPUID bit.

Quoted 3 lines

The error is about no valid domains (e.g. counters not working). So the error on minnowboard board could be a real problem if you expect to use RAPL.

Right, and if you need to disable it there, you would need to add a quirk table looking at DMI strings or so. A CPUID bit might've been a bit better if BIOS update would clear it on those boards. Then, sw won't even try to load there.

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#19 lore raw
Quoted 6 lines · On Fri, 4 Dec 2015 19:28:07 +0100 Borislav Petkov <[email protected]> wrote:
Quoted 1 line · Jacob Pan

CPU model detection is the first level checking.

And in the case of RAPL, the only checking you can do. This is why it should've had a CPUID bit.

I am 200% with you, all I can say is we are working on it. Look at the pain in the driver for dealing with various quirks.

Quoted 8 lines
Quoted 3 lines

The error is about no valid domains (e.g. counters not working). So the error on minnowboard board could be a real problem if you expect to use RAPL.

Right, and if you need to disable it there, you would need to add a quirk table looking at DMI strings or so. A CPUID bit might've been a bit better if BIOS update would clear it on those boards. Then, sw won't even try to load there.

who is gonna collect all the DMI strings? I don't think this is scalable.

author maintainer
#20 lore raw
Quoted 2 lines · Jacob Pan

who is gonna collect all the DMI strings? I don't think this is scalable.

Let's hope only a small number of platforms is affected.

Signature · 4 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
#21 lore raw
In reply to Peter Zijlstra
Quoted 2 lines · On Fri, 4 Dec 2015 09:22:56 +0100 Peter Zijlstra <[email protected]> wrote:

Also, yuck @ powercap/intel_rapl.c for doing rdmsr_on_cpu() + wrmsr_on_cpu() all over the place.

Can you please be more specific? is the concern related to the overhead of IPI? I am doing these calls based on MSR CPU scope and consider the fact that access is less frequent.

Thanks,

Jacob

maintainer
#22 lore raw
Quoted 8 lines · Jacob Pan
Quoted 2 lines · On Fri, 4 Dec 2015 09:22:56 +0100 Peter Zijlstra <[email protected]> wrote:

Also, yuck @ powercap/intel_rapl.c for doing rdmsr_on_cpu() + wrmsr_on_cpu() all over the place.

Can you please be more specific? is the concern related to the overhead of IPI? I am doing these calls based on MSR CPU scope and consider the fact that access is less frequent.

Yeah, its just offensive to do an IPI to read a value, then twiddle a few bits on the value and then IPI again to store the value.

I know its low freq, and that MSR access is slow, but *groan*.

#23 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Jacob Pan
Quoted 2 lines · On Fri, 4 Dec 2015 09:22:56 +0100 Peter Zijlstra <[email protected]> wrote:

Also, yuck @ powercap/intel_rapl.c for doing rdmsr_on_cpu() + wrmsr_on_cpu() all over the place.

Quoted 8 lines
Quoted 3 lines

Can you please be more specific? is the concern related to the overhead of IPI? I am doing these calls based on MSR CPU scope and consider the fact that access is less frequent.

Yeah, its just offensive to do an IPI to read a value, then twiddle a few bits on the value and then IPI again to store the value.

I know its low freq, and that MSR access is slow, but *groan*.

Yes, for that it would be better to invoke a common routine to do all the accesses on the target CPU. MSR accesses may be slow, but IPIs are way slower.

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.